Which robots.txt rules should you use for AI crawlers? Four complete examples

How do the groups actually match?

Consecutive User-agent lines share the rules that follow. A matching named group supplies the applicable rules; the wildcard group is the fallback. Repeated groups for the same product token combine. Moving a second group to the bottom does not turn robots.txt into a last-setting-wins configuration. RFC 9309, section 2.2.1.

Within the applicable rules, the most specific matching path wins, and an equivalent allow/disallow tie favors Allow. Product-token matching is case-insensitive; path matching should be case-sensitive. These examples use simple path prefixes so you can test the intended result without relying on an extension. RFC 9309, section 2.2.2.

That named-group rule creates a common mistake. If the wildcard excludes /search/ and a newly added OAI-SearchBot group only says Allow: /, the named bot no longer receives the wildcard exclusion. Either leave it on the fallback or carry the exclusions into its matching rules.

1. A public archive that permits broad crawler access

This fits an owner who has decided to permit ordinary search and the documented AI collection uses on its public archive. The exclusions avoid two unhelpful crawl spaces. They do not protect a secret: temporary exports containing private information should require authorization or should never be public.

No named provider group is needed here. The wildcard supplies the same default policy to the listed agents that follow it. Adding a long list of redundant allow groups would create more places to forget these exclusions.

User-agent: *
Allow: /
Disallow: /search/
Disallow: /exports/tmp/

Sitemap: https://example.com/sitemap.xml

2. A portfolio or publisher that allows search and declines training

The search agents use the wildcard policy, while GPTBot and ClaudeBot receive a site-wide disallow. OpenAI and Anthropic document those agents for model-development collection. Google-Extended handles the Google uses described in its own reference. OpenAI agent guidance, Anthropic guidance and Google’s crawler reference.

PerplexityBot stays on the wildcard path because Perplexity describes it as a search crawler outside foundation-model training. A provider name alone would be a poor basis for putting it in the training block. Perplexity’s agent definitions.

User-agent: *
Allow: /
Disallow: /search/
Disallow: /preview/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

3. A product site that limits listed AI agents to documentation

Here ordinary search crawlers can access the public site, while the named AI search and user agents are directed to /docs/, /releases/ and /assets/. The longer Allow prefixes reopen those paths beneath Disallow: /. Training collection is declined.

This is useful when public documentation is the intended reference surface and the rest of the product site has fast-changing or low-value crawl spaces. Match the allowed directories to the actual site and its supporting resources. User-fetch exceptions still apply, as explained below.

User-agent: *
Allow: /
Disallow: /search/

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Disallow: /
Allow: /docs/
Allow: /releases/
Allow: /assets/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

4. A publisher that declines the listed AI agents

This declines the named agents while leaving Googlebot and other ordinary search crawlers on the wildcard policy. It is a maintained list, so a newly introduced token needs review. User-triggered fetchers and uncooperative clients require further attention.

The file also leaves Google’s generative Search presentation decision unresolved. Google-Extended does not control AI Overviews; Google now documents a Search generative AI setting in Search Console. Set that separately if the publisher wants to decline the specified Search features. Google’s control documentation.

User-agent: *
Allow: /
Disallow: /search/

User-agent: OAI-SearchBot
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

Which requests can these files fail to stop?

OpenAI says robots rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Anthropic says its bots honor robots directives. Preserve those provider-specific differences when interpreting a request. The named user groups above express the chosen preference; they cannot promise the same behavior from every fetcher. OpenAI, Perplexity, Anthropic.

For a technical deny, use an appropriate edge or application control and verify the response. For private material, require authorization. The robots access-control boundary explains why naming a path in this public file cannot protect it.

How do you validate the file you actually publish?

Fetch the public /robots.txt for each host after deployment and retain its bytes. Test that returned file, because a CDN or hosting setting can alter the origin response. Check an allowed page, a prohibited path and every named token relevant to your choice.

Google’s open-source parser provides a documented single-URL CLI. After building it using the repository README, run the following from its checkout with an absolute path to your saved file. This checks parser interpretation, not whether another provider will make a request or honor the result.

I would finish with the served file, the parser/version and the path results. Then check document delivery and attributed provider visits using the server-log method. A policy file earns its place when its behavior matches the publishing choice that produced it.

bazel run robots_main -- /absolute/path/robots.txt OAI-SearchBot https://example.com/docs/start
bazel run robots_main -- /absolute/path/robots.txt OAI-SearchBot https://example.com/pricing/