Robots.txt Explained: Controlling Search Engine Crawlers Properly

The file robots.txt belongs to the basic tools of technical search engine optimization. With it, you can tell search engine crawlers which areas of your website they are allowed to crawl and which not.

However, an incorrect configuration can have significant consequences. In the worst-case scenario, Google will be prevented from crawling important parts or even the entire website.

Just as important is understanding what the robots.txt not possible: It is not a reliable tool for removing web pages from the Google index.

In this article, we explain how a robots.txt is structured, which User-agent, Disallow and Allow mean how you submit a sitemap and which mistakes you should definitely avoid.

Briefly explained: The robots.txt primarily controls which URLs search engine crawlers are allowed to access. It thereby controls the Crawling. Whether a website should be indexed is another question.

What is a robots.txt? #

The robots.txt is a simple text file located in the root directory of a host.

For a website at:

https://example.com

is the file located under:

https://example.com/robots.txt

Search engine crawlers fetch this file and check which crawling rules apply to them.

Where does the robots.txt need to be located? #

The file must be located directly in the root directory of the respective host.

Correct:

https://example.com/robots.txt

A file under:

https://example.com/verzeichnis/robots.txt

isn't the robots.txt for the entire website.

The file name must robots.txt loud.

robots.txt always applies to the respective host #

An important technical feature is the scope of validity.

The file under:

https://example.com/robots.txt

applies to URLs under this host and protocol.

It does not automatically apply to:

https://shop.example.com

If a subdomain requires its own crawling rules, it accordingly requires its own robots.txt under their host.

Different protocols and ports are also considered technically separate.

How is a robots.txt structured? #

For example, a simple file can look like this:

User-agent: *
Disallow: /intern/

Sitemap: https://example.com/sitemap.xml

The most important elements are:

Instruction	Meaning
User-agent	Determine which crawler the following rules apply to
Disallow	Prevents crawling of matching URL paths
Allow	Allows crawling of matching URL paths within corresponding rule groups
Sitemap	Provide the full address of a sitemap

What does User-agent mean? #

With User-agent it is specified which crawler a group of rules is intended for.

The asterisk stands for all crawlers to which this general group applies:

User-agent: *

A rule can also be written specifically for a particular crawler.

For example:

User-agent: Googlebot
Disallow: /beispiel/

This rule group is therefore specifically targeted at Googlebot.

What does disallow mean? #

With Disallow Can you exclude a URL path for the respective crawler from crawling?.

Example:

User-agent: *
Disallow: /intern/

This tells the respective crawlers that URLs under this path should not be crawled.

For example:

https://example.com/intern/

https://example.com/intern/datei.html

https://example.com/intern/unterordner/

What does an empty Disallow statement mean? #

An empty Disallow-Instruction does not block any path.

Example:

User-agent: *
Disallow:

Nothing is excluded from crawling by this for this rule group.

However, if your website is allowed to be fully crawled, such a rule is not strictly necessary. In principle, a website can also be accessed without restricting robots.txt-Rules are crawled.

Block the entire website for crawlers #

The following rule is particularly important:

User-agent: *
Disallow: /

It instructs the targeted crawlers not to crawl the entire website.

Attention: This configuration must not accidentally remain active on a publicly indexable website. Especially after switching from a development or staging environment to the production website, you should robots.txt check.

What does allow mean? #

With Allow a specific path within an otherwise blocked structure can be explicitly allowed for crawling.

A simplified example:

User-agent: *
Disallow: /bereich/
Allow: /bereich/oeffentlich/

This fundamentally blocks the area, while the specifically allowed path is permitted to be crawled.

Allow and Disallow can be used together #

More complex websites may require rules where a larger area is excluded and a specific resource or substructure is allowed again.

Special care should be taken here, as small changes to the URL pattern can have a different effect than expected.

Practical Tip: Hold your robots.txt as simple as possible. You should only use complex rules if there is a specific technical reason for doing so.

robots.txt is not a security feature #

A Disallow-Rule does not protect confidential data.

The file is publicly accessible and its rules can be viewed by anyone.

For example, if you enter the following:

User-agent: *
Disallow: /geheime-dokumente/

then the path is not protected against visitors by that.

Anyone who knows the URL can still access it directly, as long as there is no real access protection in place.

Important: Confidential content must be protected with appropriate access mechanisms. A robots.txt is neither password protection nor access control.

robots.txt and noindex are not the same #

This is the most important difference in the entire article.

robots.txt	noindex
controls crawling	controls indexing
is available as a central text file	is delivered, for example, as a meta robots directive or HTTP header
can prevent the retrieval of a URL	instructs search engines not to index the page
no reliable tool to remove a webpage from Google	intended to exclude an accessible page from the index

Why can a blocked URL still appear on Google? #

Google can also learn of a URL in other ways, for example through links from other websites.

If the URL through robots.txt excluded from crawling, Google is not allowed to normally retrieve their actual content.

Google may still know the URL.

Therefore, the following assumption is false:

Disallow = definitely not on Google

A through robots.txt A blocked URL may under certain circumstances still appear as a URL in the search results.

Why does noindex not work if Google is not allowed to crawl the page? #

Assuming a page contains:

<meta name="robots" content="noindex">

and at the same time blocks the robots.txt Google from retrieving this page.

Then a problem arises:

robots.txt
blocks crawling
        ↓
Google does not fetch the page
        ↓
Google does not see the noindex tag

So that Google can noindexTo be able to recognize the instruction, the crawler must be allowed to access the page in question.

Note: If a publicly accessible page is to be excluded from the Google index, you must not simultaneously prevent Google from accessing the corresponding noindex-Instruction to view.

Can you write noindex in the robots.txt? #

No. Google supports noindex not as a rule within the robots.txt.

Therefore, you should avoid using the following:

User-agent: *
Noindex: /beispiel/

If an HTML page should not be indexed, a robots meta directive can be used, for example:

<meta name="robots" content="noindex">

For other resource types, a corresponding HTTP header can be used depending on the use case.

When is robots.txt useful? #

A robots.txt is particularly useful if you want to control the crawling of certain URL areas.

This can be relevant, for example, for websites that have a large number of technically generated URL variants.

Possible cases are:

certain technical areas
unnecessary search or filter URLs
certain automatically generated URL patterns
crawler access to individual resources
large volumes of irrelevant crawl URLs

However, whether a lock is useful must be assessed for the specific system.

Not every technical URL needs to be blocked #

A long robots.txt is not automatically better than a short one.

For smaller and clearly structured websites, there is often no reason to block numerous directories as a precaution.

Rather, every additional rule increases the risk of accidentally excluding important content or resources.

What about WordPress? #

WordPress websites often possess an automatically or dynamically provided robots.txt.

Depending on the configuration and plugins used, their content may vary.

That is why you shouldn't blindly adopt an unfamiliar „optimal WordPress robots.txt“ from the internet.

First open:

https://deine-domain.ch/robots.txt

and check which rules your website actually delivers.

The WordPress setting for search engine visibility #

WordPress has an option under settings to prevent search engines from indexing the website.

This setting is relevant, for example, during a development phase.

For a publicly accessible website, you should definitely check after going live that an earlier development configuration has not accidentally been left in place.

What technical output results from this can depend on the WordPress version, configuration, and plugins used. The decisive factor is therefore always the actually delivered website.

Handling staging websites correctly #

A development or staging website should normally not appear publicly in search engines.

A pure robots.txt-However, a block is not complete protection against this.

For non-public development environments, real access protection is much more reliable.

For example:

Staging website
↓
Authentication /
password protection
↓
not publicly accessible

This protects you not only against search engine crawling, but also against unwanted public access.

Check robots.txt after a relaunch #

After a website relaunch belongs the robots.txt for the files you should definitely check.

Check in particular:

Is the production website crawlable?

Is Disallow: / active by accident?

Are important areas blocked?

Are important CSS/JS resources accessible?

Is the sitemap address correct?

Do rules still stem from the development environment?

If the URL structure was changed at the same time, you should also check redirects, sitemaps, and the indexing status.

Do not unnecessarily block important CSS and JavaScript files #

Google renders modern websites and often requires CSS, JavaScript, and other resources to do so.

When important resources through the robots.txt being blocked, Google may not be able to process and render a page the way a normal visitor sees it.

Therefore, you should not block CSS and JavaScript directories wholesale just because these files themselves are not intended as normal search results.

Google should be able to understand the page as much like a visitor as possible #

If layout, navigation, or essential content depend on JavaScript or CSS, blocking these resources can make rendering more difficult.

If you encounter problems, you can use the Google Search Console URL Inspection tool to examine how Google processes the page in question.

Find more about URL inspection at Google is not indexing my website: Check causes.

Specify a sitemap in robots.txt #

The robots.txt can point to an XML sitemap.

Example:

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

The sitemap address must be specified in full.

If your website uses a sitemap index, it can be specified accordingly:

Sitemap: https://example.com/sitemap_index.xml

We explain how XML sitemaps work and how to submit them to Google at XML Sitemap: What it does and how to submit it to Google.

Sitemap and robots.txt have opposing tasks #

Simply put, you can remember the difference like this:

XML Sitemap

"Here are important URLs
that you should know."


robots.txt

"You should not crawl
these URL areas."

Therefore, both configurations should logically match each other.

Do not recommend and block a URL at the same time #

If a URL is listed as an important indexable page in your XML sitemap, it should not simultaneously be blocked by robots.txt be excluded from crawling.

Such a configuration produces conflicting signals:

Sitemap:
"This URL is relevant."

robots.txt:
"Do not fetch this URL."

Therefore, always check both sides of the configuration in case of indexing problems.

Use comments in robots.txt #

Comments can be with a # be initiated.

Example:

# Do not crawl the technical section
User-agent: *
Disallow: /intern/

Everything after the # is treated as a comment in this line.

Comments can help document why a certain rule exists in more complex files.

Case sensitivity in URL paths #

For the URL paths in Allow- and Disallow-Rules case sensitivity is relevant.

For example:

/Pictures/

/pictures/

not automatically the same URL pattern.

Write rules that match the URL paths actually used.

Wildcards in robots.txt #

Google supports, among other things, the asterisk in path rules * as a placeholder.

An example:

User-agent: Googlebot
Disallow: /*.pdf

This can be used to capture URL paths based on a pattern.

You should use such rules with caution, because they can affect significantly more URLs than a simple directory rule.

The dollar sign for the end of a URL #

Google also supports $, to mark the end of a URL pattern.

Example:

User-agent: Googlebot
Disallow: /*.pdf$

The pattern targets URLs whose path correspondingly starts with .pdf ends.

A URL with additional characters or parameters following this pattern can be treated differently as a result.

Practical Tip: Wildcards are powerful, but error-prone. For a normal corporate website, you should not use complicated patterns if simple rules achieve the same purpose.

Which rule wins in case of overlaps? #

If several Allow- and DisallowIf multiple rules match a URL, Google evaluates the most specific matching rule based on the length of the matching path.

A simplified example:

User-agent: *
Disallow: /ordner/
Allow: /ordner/oeffentlich/

For a URL under:

https://example.com/ordner/oeffentlich/seite

is the more specific one Allow-Rule authoritative.

Especially with complex rule sets, you should therefore not only look at the order of the lines.

Crawl-delay and Google #

Some robots.txt-Examples on the internet include an instruction like:

Crawl-delay: 10

Google supports crawl-delay in the robots.txt not.

You should therefore not use this instruction as a method to control Googlebot's crawl speed.

Do not needlessly fill robots.txt with foreign rules #

Numerous prefabricated files with long lists of bots and rules can be found on the internet.

You shouldn't adopt such templates blindly.

An external configuration can:

be intended for another CMS
contain outdated rules
block important resources
contain unnecessary bot rules
not match your URL structure
make future errors harder to trace

A short, understandable, and documented file is often the better solution.

robots.txt and AI crawlers #

Other automated systems can also use their own user-agent names and respect the robots exclusion protocol.

Whether and how a particular service interprets crawling rules depends on the respective provider and crawler.

Therefore, you should not assume that a single rule for Googlebot automatically affects all search engine, AI, analytics, and other crawlers.

If you want to specifically control a certain crawler, you should check its current official documentation and user-agent designation.

Check robots.txt publicly in browser #

The simplest initial check is to call the file directly:

https://deine-domain.ch/robots.txt

Then check:

Is the file displayed?

Is the content as expected?

Is there a Disallow: /?

Are important paths blocked?

Is the sitemap correct?

Are there any old rules?

Check robots.txt in Google Search Console #

Google Search Console provides information on the recognized by Google robots.txt Ready.

You can use this, for example, to check which file Google has detected for your website and whether there are any issues when retrieving it.

For a specific URL, the URL Inspection tool is also important if you want to check whether Google can crawl the page.

After a change, don't just check the browser #

If you a robots.txt-changed the rule, you should not only check whether the new file is visible in the browser.

Additionally, check one or more actually affected URLs.

For example:

Rule changed
        ↓
Check robots.txt in browser
        ↓
Identify affected URL
        ↓
Use URL inspection
        ↓
Check crawlability

Google can cache robots.txt #

Google is recalling robots.txt regularly and can cache their content.

Therefore, a change does not have to be taken into account by every crawl in the exact same second.

Once you have fixed a critical block, you should check the current status and give Google time to recrawl and process the change.

What happens if robots.txt is inaccessible? #

The behavior of a crawler also depends on which HTTP status the robots.txt delivers.

A permanently non-existent robots.txt-document is not the same as a temporary server error during retrieval.

Therefore, you should not intentionally experiment with error states to control crawling.

For a standard website, the desired configuration should be clearly and reliably accessible.

We explain the basics of HTTP responses under HTTP status codes explained: 200, 301, 404, 403, and 500.

robots.txt and 404 #

If none robots.txt exists and the corresponding fetch returns a normal „not found“ status, Google generally treats this as if there are no crawling restrictions by a robots.txt.

That doesn't mean you should intentionally create a faulty file.

An existing, understandable configuration is usually simpler for maintenance and diagnosis.

check robots.txt on a new website #

Before launching a new website, you should check the following points:

robots.txt accessible?
        ↓
no accidental
total block?
        ↓
important pages crawlable?
        ↓
important resources crawlable?
        ↓
sitemap address correct?
        ↓
check noindex separately?
        ↓
Search Console set up?
        ↓
test important URLs?

robots.txt after a domain change #

When changing a domain, you should robots.txt consider both in the context of the old and the new domain.

The new website must not accidentally prevent Google from crawling it.

At the same time, necessary redirects of the old URLs must be accessible to Google.

Therefore, do not completely block the old website if Google should still be able to crawl its redirects.

robots.txt and redirects #

When Google removes an old URL because of a robots.txt-rule is not allowed to crawl, this can prevent Google from normally retrieving a redirect set up there.

During a migration, Google should therefore generally be able to crawl relevant old URLs in order to recognize their redirects to the new destinations.

We explain how to correctly use permanent redirects at Set up a 301 redirect: Permanently redirect URLs.

Consider robots.txt and HTTP status codes together #

When dealing with technical SEO issues, you should keep several levels apart:

robots.txt
→ Is Google allowed to crawl the URL?

HTTP status
→ What does the server respond with?

noindex
→ Is the page allowed to be indexed?

Canonical
→ Which URL is the preferred version?

Sitemap
→ Which URL is reported as relevant?

Only the interplay of these signals reveals how a URL is technically configured.

Example: public page should be indexed #

Assuming the following URL is supposed to appear on Google:

https://example.com/ratgeber

A clean configuration could look something like this, simplified:

robots.txt:
Crawling allowed

HTTP:
200 OK

Meta Robots:
no noindex

Canonical:
https://example.com/ratgeber/

Sitemap:
URL included

Internal links:
present

Therefore, there are no obvious technical contradictions at these levels.

Example: Page should not be indexed #

A publicly accessible page should be crawlable by Google, but not appear in search results.

Then the principle could be:

robots.txt:
Crawling allowed

Meta Robots:
noindex

Google can crawl the page that noindex-Recognize instruction and process accordingly.

Example: private area #

A truly confidential area should not merely be through:

Disallow: /privat/

„be protected.

Instead, he needs real access protection.

private content
        ↓
Login / Authentication
        ↓
not publicly accessible

The question of search engine indexing is then only part of the actual access control.

Example: entire website accidentally blocked #

After a relaunch, the file shows:

User-agent: *
Disallow: /

However, the website should be publicly crawled by Google.

Then this rule is a critical configuration error.

After the correction, you should:

fetch robots.txt again
        ↓
check rules
        ↓
check important URLs with Search Console
        ↓
check sitemap
        ↓
monitor indexing progress

Example: URL is in sitemap and blocked at the same time #

Sitemap:

https://example.com/produkt/


robots.txt:

Disallow: /produkt/

If the product page is to be indexed normally, these signals contradict each other.

The solution is not to ask Google to index more frequently, but to fix the technical configuration first.

When should you even change robots.txt? #

Only change the file if you have a specific reason to do so.

For example:

unnecessary crawl area identified

technically generated URL structure
should not be crawled

correct existing incorrect rule

add sitemap address

control targeted crawler

configure migration / relaunch correctly

„I want to improve my SEO“ by itself is not yet a reason for additional Disallow-to integrate rules.

Always check the impact before making changes #

A single short rule can affect thousands of URLs.

From:

Disallow: /shop/filter/

and

Disallow: /shop/

completely different effects can occur.

Therefore, check exactly which URLs are affected by a rule before saving.

Attention: Do not experiment directly on a live website with complex robots.txt-Rules, if you cannot clearly assess their effects.

Common mistakes in robots.txt #

Disallow: / on live website

use robots.txt as noindex

block noindex page at the same time

"protect" confidential content only with robots.txt

block important CSS files

block important JavaScript files

enter incorrect sitemap URL

carry over old staging rules

copy third-party WordPress templates unchecked

use complicated wildcards without necessity

ignore case sensitivity

use Crawl-delay for Google

block old domain during migration even though redirects should be crawled

Checklist: Properly checking robots.txt #

https://your-domain.ch/robots.txt
access
        ↓
File reachable?
        ↓
Correct host?
        ↓
Check user-agent rules
        ↓
Check disallow rules
        ↓
Disallow: / present?
        ↓
Check allow rules
        ↓
Check wildcards
        ↓
Important pages crawlable?
        ↓
Important CSS/JS resources crawlable?
        ↓
Sitemap address correct?
        ↓
Check noindex separately
        ↓
Test important URLs in
Search Console
        ↓
Check again after changes

robots.txt, noindex, and sitemap compared #

Tool	Main task	Typical purpose
robots.txt	Control crawling	Keep crawlers away from certain URL ranges
noindex	Prevent indexing	exclude accessible page from search results
XML Sitemap	Share URLs	Point search engines to relevant URLs

These three mechanisms do not replace each other.

A technically clean website uses them according to their respective task.

Summary #

The robots.txt is a publicly accessible text file in the root directory of a host. It tells search engine crawlers which URL areas they are allowed to crawl and which not.

With User-agent you determine the addressed crawler. Disallow excludes matching paths from crawling while Allow can release more specific paths. In addition, the full address of an XML sitemap can be specified.

The most important distinction is: robots.txt controls crawling and noindex controls indexing. A through robots.txt A blocked URL can still be known to Google and may continue to appear in search results under certain circumstances.

A robots.txt is also not a security feature. Confidential content requires real access protection.

Special caution is required after a relaunch, during domain changes, and with staging websites. An accidental rule like Disallow: / can prevent Google from crawling the entire production website.

You should only use complex rules, wildcards, and long lists of foreign crawlers if there is a specific technical reason for doing so.

A good robots.txt is therefore not as extensive as possible. It is as simple as possible and as specific as necessary – and every restriction it contains has a traceable purpose.