robots.txt: What It Can and Cannot Do (and the Mistakes That Cost Traffic)
A robots.txt file is a handful of lines of plain text, and it is behind some of the most expensive mistakes in SEO. One stray Disallow: / has taken whole sites out of search results. One careless pattern has hidden the stylesheets a search engine needs to render a page.
Almost every one of those mistakes has the same cause: people believe robots.txt does something it does not.
What it does: controls crawling
robots.txt tells crawlers which URLs they may request. That is the whole job. The format is standardised in RFC 9309, and compliance is voluntary. Googlebot and Bingbot follow it; a hostile scraper will not.
Three things follow from that.
It is not a security tool. The file is public. Anyone can open /robots.txt, so listing /admin/ or /internal-reports/ in it announces exactly where to look. Protect private content with authentication, not with a request to be polite.
It is not an indexing control. A URL blocked from crawling can still appear in search results if other pages link to it, usually as a bare URL with no description, because the crawler was never allowed to read the page. To keep a page out of the index, the page must be crawlable and carry a noindex signal. Blocking it makes the noindex invisible. The robots.txt generator explains this trap in its own notes, and it is worth reading before writing any Disallow rule.
It is not retroactive. Adding a rule stops future requests. It does not remove anything already indexed.
Where it lives, and how it fails
- At the root.
https://example.com/robots.txt. A file in a subdirectory is ignored. - Per origin. The rules apply only to the protocol, host and port that serve the file.
example.com,www.example.comandblog.example.comare three separate hosts and each needs its own. - Size-limited. Google reads the first 500 KiB and ignores the rest, so a very long file can silently lose its last rules.
- Cached. Crawlers do not re-fetch it on every request, so a change can take a while to be seen. Do not expect instant effect.
- Sensitive to errors. A missing file (404) means everything is allowed. A persistent server error (5xx) is treated more cautiously, and Google's documentation describes it holding back crawling while the file is unreadable. A broken robots.txt can be worse than none.
How rules are matched
Rules are grouped under User-agent lines:
User-agent: *
Disallow: /private/
Allow: /private/press-kit/
Three matching rules explain most of the behaviour that surprises people.
A crawler obeys one group, not all of them. It picks the group that matches its name most specifically and ignores the rest. This is the classic bug: add a User-agent: Googlebot group with one extra rule, and Googlebot stops following everything in the User-agent: * group, because it now has a more specific group to obey instead.
The longest matching path wins. In the example above, Allow: /private/press-kit/ is a longer match than Disallow: /private/, so the press kit is crawlable. If two rules match equally, Google applies the least restrictive one.
Matching is by prefix and is case-sensitive. Disallow: /blog blocks /blog, /blog/, /blog-archive and /blogger. Use Disallow: /blog/ to block only the directory. /Admin/ and /admin/ are different paths.
Two wildcards are widely supported: * matches any sequence of characters, and $ anchors the end of the URL. Disallow: /*.pdf$ blocks URLs ending in .pdf.
The mistakes that actually happen
Shipping the staging file to production. Staging sites commonly use Disallow: / to stay out of search. When that file is deployed with the real site, everything is blocked. Make the file environment-aware, and check /robots.txt after every launch.
Blocking CSS, JavaScript and images. Search engines render pages. If the files a page needs to display are blocked, the page may be evaluated as broken or empty. Block crawl-wasteful areas such as internal search results, not the assets pages depend on.
Blocking a page you want de-indexed. As above: the crawler can never see the noindex. Allow the crawl, serve noindex, and only block it later if crawl budget is a real concern.
Expecting canonicals or structured data to work on blocked pages. They are on the page, and the page is not being read.
Writing noindex inside robots.txt. Google stopped honouring it on 1 September 2019, as it announced at the time. The line does nothing.
Relying on Crawl-delay. Google ignores it. Some other crawlers respect it.
A pattern that is broader than intended. Disallow: /*? blocks every URL containing a query string, which can include pagination, filters and real product pages. Test patterns against actual URLs before relying on them.
A relative sitemap URL. The Sitemap: line needs the full absolute URL, including the protocol.
A sensible baseline
For most small sites the correct file is nearly empty:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
That allows everything and tells crawlers where the sitemap is. This site's own file is exactly that. Add Disallow lines only for a specific, understood reason, such as internal search results or a cart.
Testing before you trust it
Reading the file by eye is how mistakes ship. In Google Search Console, the robots.txt report shows how Google fetched and parsed your file, and the URL Inspection tool tells you whether a specific URL is blocked. Google's own robots.txt documentation covers the full syntax.
To build the file in the first place, the robots.txt generator handles per-crawler groups and sitemap lines. If you also need to control what appears in results, the meta tag generator covers the robots meta tag and the title and description that actually show up. Then test a handful of real URLs against the finished file. Five minutes of testing is cheaper than a traffic drop.