What robots.txt does and what it doesn't

robots.txt tells compliant crawlers which URLs they may fetch. It does not make a URL private, and it does not reliably keep that URL out of search results.

That distinction is the most important thing to understand about the file.

Put it at the root

The file must be available as plain text at the root of the site:

https://example.com/robots.txt

Rules apply only to that scheme, hostname, and port. A file on www.example.com does not automatically control shop.example.com.

A minimal file that allows crawling and points to a sitemap looks like this:

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

User-agent selects a crawler. Disallow names a path it should not fetch. An empty Disallow value blocks nothing.

Block a section, not a secret

To ask all compliant crawlers not to fetch an internal search area:

User-agent: *
Disallow: /search/

Anyone can open the file and see that path. A crawler can ignore it, and a blocked URL may still appear in search results when other pages link to it. Never list passwords, private documents, or admin endpoints as if the file protects them. Use authentication and proper access controls.

Disallow is not noindex

If a public page may be crawled but should not appear in search results, use a robots meta tag:

<meta name="robots" content="noindex">

Or send the equivalent HTTP header for non-HTML content:

X-Robots-Tag: noindex

The crawler must be allowed to fetch the page to see that instruction. Combining Disallow with noindex can therefore prevent the crawler from reading the very directive meant to remove the page.

Password protection or a 404/410 response is more appropriate when content should not be public at all.

Targeting individual crawlers

Rules can name a specific user agent:

User-agent: ExampleBot
Disallow: /

User-agent: *
Disallow: /search/

Crawler-specific support differs. Non-standard directives such as Crawl-delay are not understood by every major search engine, and Noindex inside robots.txt is not part of the standard. Use each search provider’s current controls for crawl rate and indexing.

Test the actual URLs

Small syntax choices can block more than intended. A missing slash, wildcard, or broad directory rule may affect assets needed to render the page.

After editing the file:

  1. Fetch /robots.txt without authentication or redirects to an unrelated host.
  2. Check several URLs you mean to allow and disallow.
  3. Use the search engine’s current testing or inspection tool.
  4. Watch crawl logs for unexpected changes.

I keep robots.txt short. Use it to reduce unwanted crawling and point bots to the sitemap. Use other tools for indexing decisions, and real security for anything that deserves the name.