{"id":22916,"date":"2026-08-30T09:36:09","date_gmt":"2026-08-30T07:36:09","guid":{"rendered":"https:\/\/www.curiaweb.ch\/?post_type=docs&p=22916"},"modified":"2026-08-30T09:36:10","modified_gmt":"2026-08-30T07:36:10","password":"","slug":"robots-txt-explained","status":"publish","type":"docs","link":"https:\/\/www.curiaweb.ch\/en\/hilfe\/seo-monitoring\/robots-txt-erklaert\/","title":{"rendered":"Robots.txt Explained: Controlling Search Engine Crawlers Properly"},"content":{"rendered":"

The file robots.txt<\/code> belongs to the basic tools of technical search engine optimization. With it, you can tell search engine crawlers which areas of your website they are allowed to crawl and which not.<\/p>\n\n\n\n

However, an incorrect configuration can have significant consequences. In the worst-case scenario, Google will be prevented from crawling important parts or even the entire website.<\/p>\n\n\n\n

Just as important is understanding what the robots.txt<\/code> not possible: It is not a reliable tool for removing web pages from the Google index.<\/p>\n\n\n\n

In this article, we explain how a robots.txt<\/code> is structured, which User-agent<\/code>, Disallow<\/code> and Allow<\/code> mean how you submit a sitemap and which mistakes you should definitely avoid.<\/p>\n\n\n\n

\n

Briefly explained:<\/strong> The robots.txt<\/code> primarily controls which URLs search engine crawlers are allowed to access. It thereby controls the Crawling<\/strong>. Whether a website should be indexed is another question.<\/p>\n<\/blockquote>\n\n\n\n

What is a robots.txt?<\/h2>\n\n\n\n

The robots.txt<\/code> is a simple text file located in the root directory of a host.<\/p>\n\n\n\n

For a website at:<\/p>\n\n\n\n

\nhttps:\/\/example.com\n<\/div><\/figure>\n\n\n\n

is the file located under:<\/p>\n\n\n\n

https:\/\/example.com\/robots.txt<\/code><\/p>\n\n\n\n

Search engine crawlers fetch this file and check which crawling rules apply to them.<\/p>\n\n\n\n

Where does the robots.txt need to be located?<\/h2>\n\n\n\n

The file must be located directly in the root directory of the respective host.<\/p>\n\n\n\n

Correct:<\/p>\n\n\n\n

https:\/\/example.com\/robots.txt<\/code><\/p>\n\n\n\n

A file under:<\/p>\n\n\n\n

https:\/\/example.com\/verzeichnis\/robots.txt<\/code><\/p>\n\n\n\n

isn't the robots.txt<\/code> for the entire website.<\/p>\n\n\n\n

The file name must robots.txt<\/code> loud.<\/p>\n\n\n\n

robots.txt always applies to the respective host<\/h2>\n\n\n\n

An important technical feature is the scope of validity.<\/p>\n\n\n\n

The file under:<\/p>\n\n\n\n

https:\/\/example.com\/robots.txt<\/code><\/p>\n\n\n\n

applies to URLs under this host and protocol.<\/p>\n\n\n\n

It does not automatically apply to:<\/p>\n\n\n\n

\nhttps:\/\/shop.example.com\n<\/div><\/figure>\n\n\n\n

If a subdomain requires its own crawling rules, it accordingly requires its own robots.txt<\/code> under their host.<\/p>\n\n\n\n

Different protocols and ports are also considered technically separate.<\/p>\n\n\n\n

How is a robots.txt structured?<\/h2>\n\n\n\n

For example, a simple file can look like this:<\/p>\n\n\n\n

User-agent: *\nDisallow: \/intern\/\n\nSitemap: https:\/\/example.com\/sitemap.xml<\/code><\/pre>\n\n\n\n
The most important elements are:<\/p>\n\n\n\n
Instruction<\/th>Meaning<\/th><\/tr><\/thead>
User-agent<\/code><\/td>Determine which crawler the following rules apply to<\/td><\/tr>
Disallow<\/code><\/td>Prevents crawling of matching URL paths<\/td><\/tr>
Allow<\/code><\/td>Allows crawling of matching URL paths within corresponding rule groups<\/td><\/tr>
Sitemap<\/code><\/td>Provide the full address of a sitemap<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
What does User-agent mean?<\/h2>\n\n\n\n
With User-agent<\/code> it is specified which crawler a group of rules is intended for.<\/p>\n\n\n\n
The asterisk stands for all crawlers to which this general group applies:<\/p>\n\n\n\n
User-agent: *<\/code><\/pre>\n\n\n\n
A rule can also be written specifically for a particular crawler.<\/p>\n\n\n\n
For example:<\/p>\n\n\n\n
User-agent: Googlebot\nDisallow: \/beispiel\/<\/code><\/pre>\n\n\n\n
This rule group is therefore specifically targeted at Googlebot.<\/p>\n\n\n\n
What does disallow mean?<\/h2>\n\n\n\n
With Disallow<\/code> Can you exclude a URL path for the respective crawler from crawling?.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
User-agent: *\nDisallow: \/intern\/<\/code><\/pre>\n\n\n\n
This tells the respective crawlers that URLs under this path should not be crawled.<\/p>\n\n\n\n
For example:<\/p>\n\n\n\n
https:\/\/example.com\/intern\/\n\nhttps:\/\/example.com\/intern\/datei.html\n\nhttps:\/\/example.com\/intern\/unterordner\/<\/code><\/pre>\n\n\n\n
What does an empty Disallow statement mean?<\/h2>\n\n\n\n
An empty Disallow<\/code>-Instruction does not block any path.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
User-agent: *\nDisallow:<\/code><\/pre>\n\n\n\n
Nothing is excluded from crawling by this for this rule group.<\/p>\n\n\n\n
However, if your website is allowed to be fully crawled, such a rule is not strictly necessary. In principle, a website can also be accessed without restricting robots.txt<\/code>-Rules are crawled.<\/p>\n\n\n\n
Block the entire website for crawlers<\/h2>\n\n\n\n
The following rule is particularly important:<\/p>\n\n\n\n
User-agent: *\nDisallow: \/<\/code><\/pre>\n\n\n\n
It instructs the targeted crawlers not to crawl the entire website.<\/p>\n\n\n\n
\n
Attention:<\/strong> This configuration must not accidentally remain active on a publicly indexable website. Especially after switching from a development or staging environment to the production website, you should robots.txt<\/code> check.<\/p>\n<\/blockquote>\n\n\n\n
What does allow mean?<\/h2>\n\n\n\n
With Allow<\/code> a specific path within an otherwise blocked structure can be explicitly allowed for crawling.<\/p>\n\n\n\n
A simplified example:<\/p>\n\n\n\n
User-agent: *\nDisallow: \/bereich\/\nAllow: \/bereich\/oeffentlich\/<\/code><\/pre>\n\n\n\n
This fundamentally blocks the area, while the specifically allowed path is permitted to be crawled.<\/p>\n\n\n\n
Allow and Disallow can be used together<\/h2>\n\n\n\n
More complex websites may require rules where a larger area is excluded and a specific resource or substructure is allowed again.<\/p>\n\n\n\n
Special care should be taken here, as small changes to the URL pattern can have a different effect than expected.<\/p>\n\n\n\n
\n
Practical Tip:<\/strong> Hold your robots.txt<\/code> as simple as possible. You should only use complex rules if there is a specific technical reason for doing so.<\/p>\n<\/blockquote>\n\n\n\n
robots.txt is not a security feature<\/h2>\n\n\n\n
A Disallow<\/code>-Rule does not protect confidential data.<\/p>\n\n\n\n
The file is publicly accessible and its rules can be viewed by anyone.<\/p>\n\n\n\n
For example, if you enter the following:<\/p>\n\n\n\n
User-agent: *\nDisallow: \/geheime-dokumente\/<\/code><\/pre>\n\n\n\n
then the path is not protected against visitors by that.<\/p>\n\n\n\n
Anyone who knows the URL can still access it directly, as long as there is no real access protection in place.<\/p>\n\n\n\n
\n
Important:<\/strong> Confidential content must be protected with appropriate access mechanisms. A robots.txt<\/code> is neither password protection nor access control.<\/p>\n<\/blockquote>\n\n\n\n
robots.txt and noindex are not the same<\/h2>\n\n\n\n
This is the most important difference in the entire article.<\/p>\n\n\n\n
robots.txt<\/th>noindex<\/th><\/tr><\/thead>
controls crawling<\/td>controls indexing<\/td><\/tr>
is available as a central text file<\/td>is delivered, for example, as a meta robots directive or HTTP header<\/td><\/tr>
can prevent the retrieval of a URL<\/td>instructs search engines not to index the page<\/td><\/tr>
no reliable tool to remove a webpage from Google<\/td>intended to exclude an accessible page from the index<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Why can a blocked URL still appear on Google?<\/h2>\n\n\n\n
Google can also learn of a URL in other ways, for example through links from other websites.<\/p>\n\n\n\n
If the URL through robots.txt<\/code> excluded from crawling, Google is not allowed to normally retrieve their actual content.<\/p>\n\n\n\n
Google may still know the URL.<\/p>\n\n\n\n
Therefore, the following assumption is false:<\/p>\n\n\n\n
Disallow = definitely not on Google<\/code><\/p>\n\n\n\n
A through robots.txt<\/code> A blocked URL may under certain circumstances still appear as a URL in the search results.<\/p>\n\n\n\n
Why does noindex not work if Google is not allowed to crawl the page?<\/h2>\n\n\n\n
Assuming a page contains:<\/p>\n\n\n\n
<meta name=\"robots\" content=\"noindex\"><\/code><\/pre>\n\n\n\n
and at the same time blocks the robots.txt<\/code> Google from retrieving this page.<\/p>\n\n\n\n
Then a problem arises:<\/p>\n\n\n\n
robots.txt\nblocks crawling\n        \u2193\nGoogle does not fetch the page\n        \u2193\nGoogle does not see the noindex tag<\/code><\/pre>\n\n\n\n
So that Google can noindex<\/code>To be able to recognize the instruction, the crawler must be allowed to access the page in question.<\/p>\n\n\n\n
\n
Note:<\/strong> If a publicly accessible page is to be excluded from the Google index, you must not simultaneously prevent Google from accessing the corresponding noindex<\/code>-Instruction to view.<\/p>\n<\/blockquote>\n\n\n\n
Can you write noindex in the robots.txt?<\/h2>\n\n\n\n
No. Google supports noindex<\/code> not as a rule within the robots.txt<\/code>.<\/p>\n\n\n\n
Therefore, you should avoid using the following:<\/p>\n\n\n\n
User-agent: *\nNoindex: \/beispiel\/<\/code><\/pre>\n\n\n\n
If an HTML page should not be indexed, a robots meta directive can be used, for example:<\/p>\n\n\n\n
<meta name=\"robots\" content=\"noindex\"><\/code><\/pre>\n\n\n\n
For other resource types, a corresponding HTTP header can be used depending on the use case.<\/p>\n\n\n\n
When is robots.txt useful?<\/h2>\n\n\n\n
A robots.txt<\/code> is particularly useful if you want to control the crawling of certain URL areas.<\/p>\n\n\n\n
This can be relevant, for example, for websites that have a large number of technically generated URL variants.<\/p>\n\n\n\n
Possible cases are:<\/p>\n\n\n\n
certain technical areas\nunnecessary search or filter URLs\ncertain automatically generated URL patterns\ncrawler access to individual resources\nlarge volumes of irrelevant crawl URLs<\/code><\/pre>\n\n\n\n
However, whether a lock is useful must be assessed for the specific system.<\/p>\n\n\n\n
Not every technical URL needs to be blocked<\/h2>\n\n\n\n
A long robots.txt<\/code> is not automatically better than a short one.<\/p>\n\n\n\n
For smaller and clearly structured websites, there is often no reason to block numerous directories as a precaution.<\/p>\n\n\n\n
Rather, every additional rule increases the risk of accidentally excluding important content or resources.<\/p>\n\n\n\n
What about WordPress?<\/h2>\n\n\n\n
WordPress websites often possess an automatically or dynamically provided robots.txt<\/code>.<\/p>\n\n\n\n
Depending on the configuration and plugins used, their content may vary.<\/p>\n\n\n\n
That is why you shouldn't blindly adopt an unfamiliar \u201eoptimal WordPress robots.txt\u201c from the internet.<\/p>\n\n\n\n
First open:<\/p>\n\n\n\n
https:\/\/deine-domain.ch\/robots.txt<\/code><\/p>\n\n\n\n
and check which rules your website actually delivers.<\/p>\n\n\n\n
The WordPress setting for search engine visibility<\/h2>\n\n\n\n
WordPress has an option under settings to prevent search engines from indexing the website.<\/p>\n\n\n\n
This setting is relevant, for example, during a development phase.<\/p>\n\n\n\n
For a publicly accessible website, you should definitely check after going live that an earlier development configuration has not accidentally been left in place.<\/p>\n\n\n\n
What technical output results from this can depend on the WordPress version, configuration, and plugins used. The decisive factor is therefore always the actually delivered website.<\/p>\n\n\n\n
Handling staging websites correctly<\/h2>\n\n\n\n
A development or staging website should normally not appear publicly in search engines.<\/p>\n\n\n\n
A pure robots.txt<\/code>-However, a block is not complete protection against this.<\/p>\n\n\n\n
For non-public development environments, real access protection is much more reliable.<\/p>\n\n\n\n
For example:<\/p>\n\n\n\n
Staging website\n\u2193\nAuthentication \/\npassword protection\n\u2193\nnot publicly accessible<\/code><\/pre>\n\n\n\n
This protects you not only against search engine crawling, but also against unwanted public access.<\/p>\n\n\n\n
Check robots.txt after a relaunch<\/h2>\n\n\n\n
After a website relaunch belongs the robots.txt<\/code> for the files you should definitely check.<\/p>\n\n\n\n
Check in particular:<\/p>\n\n\n\n
Is the production website crawlable?\n\nIs Disallow: \/ active by accident?\n\nAre important areas blocked?\n\nAre important CSS\/JS resources accessible?\n\nIs the sitemap address correct?\n\nDo rules still stem from the development environment?<\/code><\/pre>\n\n\n\n
If the URL structure was changed at the same time, you should also check redirects, sitemaps, and the indexing status.<\/p>\n\n\n\n
Do not unnecessarily block important CSS and JavaScript files<\/h2>\n\n\n\n
Google renders modern websites and often requires CSS, JavaScript, and other resources to do so.<\/p>\n\n\n\n
When important resources through the robots.txt<\/code> being blocked, Google may not be able to process and render a page the way a normal visitor sees it.<\/p>\n\n\n\n
Therefore, you should not block CSS and JavaScript directories wholesale just because these files themselves are not intended as normal search results.<\/p>\n\n\n\n
Google should be able to understand the page as much like a visitor as possible<\/h2>\n\n\n\n
If layout, navigation, or essential content depend on JavaScript or CSS, blocking these resources can make rendering more difficult.<\/p>\n\n\n\n
If you encounter problems, you can use the Google Search Console URL Inspection tool to examine how Google processes the page in question.<\/p>\n\n\n\n
Find more about URL inspection at Google is not indexing my website: Check causes<\/a>.<\/p>\n\n\n\n
Specify a sitemap in robots.txt<\/h2>\n\n\n\n
The robots.txt<\/code> can point to an XML sitemap.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
User-agent: *\nDisallow:\n\nSitemap: https:\/\/example.com\/sitemap.xml<\/code><\/pre>\n\n\n\n
The sitemap address must be specified in full.<\/p>\n\n\n\n
If your website uses a sitemap index, it can be specified accordingly:<\/p>\n\n\n\n
Sitemap: https:\/\/example.com\/sitemap_index.xml<\/code><\/pre>\n\n\n\n
We explain how XML sitemaps work and how to submit them to Google at XML Sitemap: What it does and how to submit it to Google<\/a>.<\/p>\n\n\n\n
Sitemap and robots.txt have opposing tasks<\/h2>\n\n\n\n
Simply put, you can remember the difference like this:<\/p>\n\n\n\n
XML Sitemap\n\n\"Here are important URLs\nthat you should know.\"\n\n\nrobots.txt\n\n\"You should not crawl\nthese URL areas.\"<\/code><\/pre>\n\n\n\n
Therefore, both configurations should logically match each other.<\/p>\n\n\n\n
Do not recommend and block a URL at the same time<\/h2>\n\n\n\n
If a URL is listed as an important indexable page in your XML sitemap, it should not simultaneously be blocked by robots.txt<\/code> be excluded from crawling.<\/p>\n\n\n\n
Such a configuration produces conflicting signals:<\/p>\n\n\n\n
Sitemap:\n\"This URL is relevant.\"\n\nrobots.txt:\n\"Do not fetch this URL.\"<\/code><\/pre>\n\n\n\n
Therefore, always check both sides of the configuration in case of indexing problems.<\/p>\n\n\n\n
Use comments in robots.txt<\/h2>\n\n\n\n
Comments can be with a #<\/code> be initiated.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
# Do not crawl the technical section\nUser-agent: *\nDisallow: \/intern\/<\/code><\/pre>\n\n\n\n
Everything after the #<\/code> is treated as a comment in this line.<\/p>\n\n\n\n
Comments can help document why a certain rule exists in more complex files.<\/p>\n\n\n\n
Case sensitivity in URL paths<\/h2>\n\n\n\n
For the URL paths in Allow<\/code>- and Disallow<\/code>-Rules case sensitivity is relevant.<\/p>\n\n\n\n
For example:<\/p>\n\n\n\n
\/Pictures\/\n\n\/pictures\/<\/code><\/pre>\n\n\n\n
not automatically the same URL pattern.<\/p>\n\n\n\n
Write rules that match the URL paths actually used.<\/p>\n\n\n\n
Wildcards in robots.txt<\/h2>\n\n\n\n
Google supports, among other things, the asterisk in path rules *<\/code> as a placeholder.<\/p>\n\n\n\n
An example:<\/p>\n\n\n\n
User-agent: Googlebot\nDisallow: \/*.pdf<\/code><\/pre>\n\n\n\n
This can be used to capture URL paths based on a pattern.<\/p>\n\n\n\n
You should use such rules with caution, because they can affect significantly more URLs than a simple directory rule.<\/p>\n\n\n\n
The dollar sign for the end of a URL<\/h2>\n\n\n\n
Google also supports $<\/code>, to mark the end of a URL pattern.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
User-agent: Googlebot\nDisallow: \/*.pdf$<\/code><\/pre>\n\n\n\n
The pattern targets URLs whose path correspondingly starts with .pdf<\/code> ends.<\/p>\n\n\n\n
A URL with additional characters or parameters following this pattern can be treated differently as a result.<\/p>\n\n\n\n
\n
Practical Tip:<\/strong> Wildcards are powerful, but error-prone. For a normal corporate website, you should not use complicated patterns if simple rules achieve the same purpose.<\/p>\n<\/blockquote>\n\n\n\n
Which rule wins in case of overlaps?<\/h2>\n\n\n\n
If several Allow<\/code>- and Disallow<\/code>If multiple rules match a URL, Google evaluates the most specific matching rule based on the length of the matching path.<\/p>\n\n\n\n
A simplified example:<\/p>\n\n\n\n
User-agent: *\nDisallow: \/ordner\/\nAllow: \/ordner\/oeffentlich\/<\/code><\/pre>\n\n\n\n
For a URL under:<\/p>\n\n\n\n
\nhttps:\/\/example.com\/ordner\/oeffentlich\/seite\n<\/div><\/figure>\n\n\n\n
is the more specific one Allow<\/code>-Rule authoritative.<\/p>\n\n\n\n
Especially with complex rule sets, you should therefore not only look at the order of the lines.<\/p>\n\n\n\n
Crawl-delay and Google<\/h2>\n\n\n\n
Some robots.txt<\/code>-Examples on the internet include an instruction like:<\/p>\n\n\n\n
Crawl-delay: 10<\/code><\/pre>\n\n\n\n
Google supports crawl-delay<\/code> in the robots.txt<\/code> not.<\/p>\n\n\n\n
You should therefore not use this instruction as a method to control Googlebot's crawl speed.<\/p>\n\n\n\n
Do not needlessly fill robots.txt with foreign rules<\/h2>\n\n\n\n
Numerous prefabricated files with long lists of bots and rules can be found on the internet.<\/p>\n\n\n\n
You shouldn't adopt such templates blindly.<\/p>\n\n\n\n
An external configuration can:<\/p>\n\n\n\n
be intended for another CMS\ncontain outdated rules\nblock important resources\ncontain unnecessary bot rules\nnot match your URL structure\nmake future errors harder to trace<\/code><\/pre>\n\n\n\n
A short, understandable, and documented file is often the better solution.<\/p>\n\n\n\n
robots.txt and AI crawlers<\/h2>\n\n\n\n
Other automated systems can also use their own user-agent names and respect the robots exclusion protocol.<\/p>\n\n\n\n
Whether and how a particular service interprets crawling rules depends on the respective provider and crawler.<\/p>\n\n\n\n
Therefore, you should not assume that a single rule for Googlebot<\/code> automatically affects all search engine, AI, analytics, and other crawlers.<\/p>\n\n\n\n
If you want to specifically control a certain crawler, you should check its current official documentation and user-agent designation.<\/p>\n\n\n\n
Check robots.txt publicly in browser<\/h2>\n\n\n\n
The simplest initial check is to call the file directly:<\/p>\n\n\n\n
https:\/\/deine-domain.ch\/robots.txt<\/code><\/p>\n\n\n\n
Then check:<\/p>\n\n\n\n
Is the file displayed?\n\nIs the content as expected?\n\nIs there a Disallow: \/?\n\nAre important paths blocked?\n\nIs the sitemap correct?\n\nAre there any old rules?<\/code><\/pre>\n\n\n\n
Check robots.txt in Google Search Console<\/h2>\n\n\n\n
Google Search Console provides information on the recognized by Google robots.txt<\/code> Ready.<\/p>\n\n\n\n
You can use this, for example, to check which file Google has detected for your website and whether there are any issues when retrieving it.<\/p>\n\n\n\n
For a specific URL, the URL Inspection tool is also important if you want to check whether Google can crawl the page.<\/p>\n\n\n\n
After a change, don't just check the browser<\/h2>\n\n\n\n
If you a robots.txt<\/code>-changed the rule, you should not only check whether the new file is visible in the browser.<\/p>\n\n\n\n
Additionally, check one or more actually affected URLs.<\/p>\n\n\n\n
For example:<\/p>\n\n\n\n
Rule changed\n        \u2193\nCheck robots.txt in browser\n        \u2193\nIdentify affected URL\n        \u2193\nUse URL inspection\n        \u2193\nCheck crawlability<\/code><\/pre>\n\n\n\n
Google can cache robots.txt<\/h2>\n\n\n\n
Google is recalling robots.txt<\/code> regularly and can cache their content.<\/p>\n\n\n\n
Therefore, a change does not have to be taken into account by every crawl in the exact same second.<\/p>\n\n\n\n
Once you have fixed a critical block, you should check the current status and give Google time to recrawl and process the change.<\/p>\n\n\n\n
What happens if robots.txt is inaccessible?<\/h2>\n\n\n\n
The behavior of a crawler also depends on which HTTP status the robots.txt<\/code> delivers.<\/p>\n\n\n\n
A permanently non-existent robots.txt<\/code>-document is not the same as a temporary server error during retrieval.<\/p>\n\n\n\n
Therefore, you should not intentionally experiment with error states to control crawling.<\/p>\n\n\n\n
For a standard website, the desired configuration should be clearly and reliably accessible.<\/p>\n\n\n\n
We explain the basics of HTTP responses under HTTP status codes explained: 200, 301, 404, 403, and 500<\/a>.<\/p>\n\n\n\n
robots.txt and 404<\/h2>\n\n\n\n
If none robots.txt<\/code> exists and the corresponding fetch returns a normal \u201enot found\u201c status, Google generally treats this as if there are no crawling restrictions by a robots.txt.<\/p>\n\n\n\n
That doesn't mean you should intentionally create a faulty file.<\/p>\n\n\n\n
An existing, understandable configuration is usually simpler for maintenance and diagnosis.<\/p>\n\n\n\n
check robots.txt on a new website<\/h2>\n\n\n\n
Before launching a new website, you should check the following points:<\/p>\n\n\n\n
robots.txt accessible?\n        \u2193\nno accidental\ntotal block?\n        \u2193\nimportant pages crawlable?\n        \u2193\nimportant resources crawlable?\n        \u2193\nsitemap address correct?\n        \u2193\ncheck noindex separately?\n        \u2193\nSearch Console set up?\n        \u2193\ntest important URLs?<\/code><\/pre>\n\n\n\n
robots.txt after a domain change<\/h2>\n\n\n\n
When changing a domain, you should robots.txt<\/code> consider both in the context of the old and the new domain.<\/p>\n\n\n\n
The new website must not accidentally prevent Google from crawling it.<\/p>\n\n\n\n
At the same time, necessary redirects of the old URLs must be accessible to Google.<\/p>\n\n\n\n
Therefore, do not completely block the old website if Google should still be able to crawl its redirects.<\/p>\n\n\n\n
robots.txt and redirects<\/h2>\n\n\n\n
When Google removes an old URL because of a robots.txt<\/code>-rule is not allowed to crawl, this can prevent Google from normally retrieving a redirect set up there.<\/p>\n\n\n\n
During a migration, Google should therefore generally be able to crawl relevant old URLs in order to recognize their redirects to the new destinations.<\/p>\n\n\n\n
We explain how to correctly use permanent redirects at Set up a 301 redirect: Permanently redirect URLs<\/a>.<\/p>\n\n\n\n
Consider robots.txt and HTTP status codes together<\/h2>\n\n\n\n
When dealing with technical SEO issues, you should keep several levels apart:<\/p>\n\n\n\n
robots.txt\n\u2192 Is Google allowed to crawl the URL?\n\nHTTP status\n\u2192 What does the server respond with?\n\nnoindex\n\u2192 Is the page allowed to be indexed?\n\nCanonical\n\u2192 Which URL is the preferred version?\n\nSitemap\n\u2192 Which URL is reported as relevant?<\/code><\/pre>\n\n\n\n
Only the interplay of these signals reveals how a URL is technically configured.<\/p>\n\n\n\n
Example: public page should be indexed<\/h2>\n\n\n\n
Assuming the following URL is supposed to appear on Google:<\/p>\n\n\n\n
\nhttps:\/\/example.com\/ratgeber\n<\/div><\/figure>\n\n\n\n
A clean configuration could look something like this, simplified:<\/p>\n\n\n\n
robots.txt:\nCrawling allowed\n\nHTTP:\n200 OK\n\nMeta Robots:\nno noindex\n\nCanonical:\nhttps:\/\/example.com\/ratgeber\/\n\nSitemap:\nURL included\n\nInternal links:\npresent<\/code><\/pre>\n\n\n\n
Therefore, there are no obvious technical contradictions at these levels.<\/p>\n\n\n\n
Example: Page should not be indexed<\/h2>\n\n\n\n
A publicly accessible page should be crawlable by Google, but not appear in search results.<\/p>\n\n\n\n
Then the principle could be:<\/p>\n\n\n\n
robots.txt:\nCrawling allowed\n\nMeta Robots:\nnoindex<\/code><\/pre>\n\n\n\n
Google can crawl the page that noindex<\/code>-Recognize instruction and process accordingly.<\/p>\n\n\n\n
Example: private area<\/h2>\n\n\n\n
A truly confidential area should not merely be through:<\/p>\n\n\n\n
Disallow: \/privat\/<\/code><\/pre>\n\n\n\n
\u201ebe protected.<\/p>\n\n\n\n
Instead, he needs real access protection.<\/p>\n\n\n\n
private content\n        \u2193\nLogin \/ Authentication\n        \u2193\nnot publicly accessible<\/code><\/pre>\n\n\n\n
The question of search engine indexing is then only part of the actual access control.<\/p>\n\n\n\n
Example: entire website accidentally blocked<\/h2>\n\n\n\n
After a relaunch, the file shows:<\/p>\n\n\n\n
User-agent: *\nDisallow: \/<\/code><\/pre>\n\n\n\n
However, the website should be publicly crawled by Google.<\/p>\n\n\n\n
Then this rule is a critical configuration error.<\/p>\n\n\n\n
After the correction, you should:<\/p>\n\n\n\n
fetch robots.txt again\n        \u2193\ncheck rules\n        \u2193\ncheck important URLs with Search Console\n        \u2193\ncheck sitemap\n        \u2193\nmonitor indexing progress<\/code><\/pre>\n\n\n\n
Example: URL is in sitemap and blocked at the same time<\/h2>\n\n\n\n
Sitemap:\n\nhttps:\/\/example.com\/produkt\/\n\n\nrobots.txt:\n\nDisallow: \/produkt\/<\/code><\/pre>\n\n\n\n
If the product page is to be indexed normally, these signals contradict each other.<\/p>\n\n\n\n
The solution is not to ask Google to index more frequently, but to fix the technical configuration first.<\/p>\n\n\n\n
When should you even change robots.txt?<\/h2>\n\n\n\n
Only change the file if you have a specific reason to do so.<\/p>\n\n\n\n
For example:<\/p>\n\n\n\n
unnecessary crawl area identified\n\ntechnically generated URL structure\nshould not be crawled\n\ncorrect existing incorrect rule\n\nadd sitemap address\n\ncontrol targeted crawler\n\nconfigure migration \/ relaunch correctly<\/code><\/pre>\n\n\n\n
\u201eI want to improve my SEO\u201c by itself is not yet a reason for additional Disallow<\/code>-to integrate rules.<\/p>\n\n\n\n
Always check the impact before making changes<\/h2>\n\n\n\n
A single short rule can affect thousands of URLs.<\/p>\n\n\n\n
From:<\/p>\n\n\n\n
Disallow: \/shop\/filter\/<\/code><\/pre>\n\n\n\n
and<\/p>\n\n\n\n
Disallow: \/shop\/<\/code><\/pre>\n\n\n\n
completely different effects can occur.<\/p>\n\n\n\n
Therefore, check exactly which URLs are affected by a rule before saving.<\/p>\n\n\n\n
\n
Attention:<\/strong> Do not experiment directly on a live website with complex robots.txt<\/code>-Rules, if you cannot clearly assess their effects.<\/p>\n<\/blockquote>\n\n\n\n
Common mistakes in robots.txt<\/h2>\n\n\n\n
Disallow: \/ on live website\n\nuse robots.txt as noindex\n\nblock noindex page at the same time\n\n\"protect\" confidential content only with robots.txt\n\nblock important CSS files\n\nblock important JavaScript files\n\nenter incorrect sitemap URL\n\ncarry over old staging rules\n\ncopy third-party WordPress templates unchecked\n\nuse complicated wildcards without necessity\n\nignore case sensitivity\n\nuse Crawl-delay for Google\n\nblock old domain during migration even though redirects should be crawled<\/code><\/pre>\n\n\n\n
Checklist: Properly checking robots.txt<\/h2>\n\n\n\n
https:\/\/your-domain.ch\/robots.txt\naccess\n        \u2193\nFile reachable?\n        \u2193\nCorrect host?\n        \u2193\nCheck user-agent rules\n        \u2193\nCheck disallow rules\n        \u2193\nDisallow: \/ present?\n        \u2193\nCheck allow rules\n        \u2193\nCheck wildcards\n        \u2193\nImportant pages crawlable?\n        \u2193\nImportant CSS\/JS resources crawlable?\n        \u2193\nSitemap address correct?\n        \u2193\nCheck noindex separately\n        \u2193\nTest important URLs in\nSearch Console\n        \u2193\nCheck again after changes<\/code><\/pre>\n\n\n\n
robots.txt, noindex, and sitemap compared<\/h2>\n\n\n\n
Tool<\/th>Main task<\/th>Typical purpose<\/th><\/tr><\/thead>
robots.txt<\/code><\/td>Control crawling<\/td>Keep crawlers away from certain URL ranges<\/td><\/tr>
noindex<\/code><\/td>Prevent indexing<\/td>exclude accessible page from search results<\/td><\/tr>
XML Sitemap<\/td>Share URLs<\/td>Point search engines to relevant URLs<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
These three mechanisms do not replace each other.<\/p>\n\n\n\n
A technically clean website uses them according to their respective task.<\/p>\n\n\n\n
Summary<\/h2>\n\n\n\n
The robots.txt<\/code> is a publicly accessible text file in the root directory of a host. It tells search engine crawlers which URL areas they are allowed to crawl and which not.<\/p>\n\n\n\n
With User-agent<\/code> you determine the addressed crawler. Disallow<\/code> excludes matching paths from crawling while Allow<\/code> can release more specific paths. In addition, the full address of an XML sitemap can be specified.<\/p>\n\n\n\n
The most important distinction is: robots.txt controls crawling and noindex controls indexing.<\/strong> A through robots.txt<\/code> A blocked URL can still be known to Google and may continue to appear in search results under certain circumstances.<\/p>\n\n\n\n
A robots.txt<\/code> is also not a security feature. Confidential content requires real access protection.<\/p>\n\n\n\n
Special caution is required after a relaunch, during domain changes, and with staging websites. An accidental rule like Disallow: \/<\/code> can prevent Google from crawling the entire production website.<\/p>\n\n\n\n
You should only use complex rules, wildcards, and long lists of foreign crawlers if there is a specific technical reason for doing so.<\/p>\n\n\n\n
A good robots.txt is therefore not as extensive as possible. It is as simple as possible and as specific as necessary \u2013 and every restriction it contains has a traceable purpose.<\/strong><\/p>","protected":false},"excerpt":{"rendered":"
Die Datei robots.txt geh\u00f6rt zu den grundlegenden Werkzeugen der technischen Suchmaschinenoptimierung. Mit ihr kannst du Suchmaschinen-Crawlern mitteilen, welche Bereiche deiner Website sie crawlen d\u00fcrfen und welche nicht. Eine falsche Konfiguration kann allerdings erhebliche Auswirkungen haben. Im ung\u00fcnstigsten Fall wird Google daran gehindert, wichtige Teile oder sogar die gesamte Website zu crawlen. Genauso wichtig ist zu […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_joinchat":[],"footnotes":""},"doc_category":[85],"doc_tag":[],"class_list":["post-22916","docs","type-docs","status-publish","hentry","doc_category-seo-monitoring"],"year_month":"2026-09","word_count":3552,"total_views":"6","reactions":{"happy":"0","normal":"0","sad":"0"},"author_info":{"name":"Silvio Mazenauer","author_nicename":"admin-curia","author_url":"https:\/\/www.curiaweb.ch\/en\/author\/admin-curia\/"},"doc_category_info":[{"term_name":"SEO & Monitoring","term_url":"https:\/\/www.curiaweb.ch\/en\/hilfe-kategorie\/seo-monitoring\/"}],"doc_tag_info":[],"knowledge_base_info":[],"knowledge_base_slug":[],"_links":{"self":[{"href":"https:\/\/www.curiaweb.ch\/en\/wp-json\/wp\/v2\/docs\/22916","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.curiaweb.ch\/en\/wp-json\/wp\/v2\/docs"}],"about":[{"href":"https:\/\/www.curiaweb.ch\/en\/wp-json\/wp\/v2\/types\/docs"}],"author":[{"embeddable":true,"href":"https:\/\/www.curiaweb.ch\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.curiaweb.ch\/en\/wp-json\/wp\/v2\/comments?post=22916"}],"version-history":[{"count":1,"href":"https:\/\/www.curiaweb.ch\/en\/wp-json\/wp\/v2\/docs\/22916\/revisions"}],"predecessor-version":[{"id":22918,"href":"https:\/\/www.curiaweb.ch\/en\/wp-json\/wp\/v2\/docs\/22916\/revisions\/22918"}],"wp:attachment":[{"href":"https:\/\/www.curiaweb.ch\/en\/wp-json\/wp\/v2\/media?parent=22916"}],"wp:term":[{"taxonomy":"doc_category","embeddable":true,"href":"https:\/\/www.curiaweb.ch\/en\/wp-json\/wp\/v2\/doc_category?post=22916"},{"taxonomy":"doc_tag","embeddable":true,"href":"https:\/\/www.curiaweb.ch\/en\/wp-json\/wp\/v2\/doc_tag?post=22916"}],"curies":[{"name":"WP","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}