Robots.txt Tester

A robots.txt is a list of groups. Each group starts with one or more User-agent lines and is followed by the Allow and Disallow rules for those crawlers.

1. The crawler picks the group whose user-agent matches its own name, ignoring case. If there is none, it uses the * group. It uses only that group: a specific group replaces * completely, it does not add to it.
2. Within the group, every rule whose pattern matches the URL's path and query is considered.
3. The longest matching pattern wins. If an Allow and a Disallow match with the same length, Allow wins.

The order of lines inside a group does not matter, which surprises people used to firewall rules.

Patterns are matched from the start of the path, so they are prefixes unless you say otherwise. Two special characters exist: * matches any run of characters and $ marks the end of the URL.

Paths are case-sensitive: /Admin and /admin are different.

It does not remove pages from search. A blocked URL can still be indexed, as a bare link with no description, if other pages link to it. To keep a page out of the index, leave it crawlable and add a noindex robots meta tag or an X-Robots-Tag header. If you block the page, Google never sees the tag.

It is not security. The file is public, and well-behaved crawlers choose to obey it. Anything private needs authentication.

It no longer supports noindex. Google stopped honouring that line in 2019. If you have one, it does nothing.

Several AI companies publish a crawler name you can address in its own group:

• GPTBot is OpenAI's crawler for training data.
• ClaudeBot is Anthropic's.
• CCBot is Common Crawl's, whose data many models are trained on.
• Google-Extended is a control token, not a separate crawler. Blocking it stops your content being used to train Google's Gemini models, and has no effect on Search.

Compliance is voluntary, though the major operators document that they follow it. To block one of them from the whole site, give it its own group with Disallow: /, as in the sample.