AI crawlers: search, training, and access.
Choose crawler access by purpose, then test the policy you actually deployed.
Sulayman Bowles · · Updated
Can I allow AI search discovery while blocking model-training crawlers?
Use separate rules for each provider’s documented search, model-development, and user-requested fetch mechanisms. A training-crawler block does not automatically block the other two. Start with the outcome you want, then merge the appropriate named groups into your existing robots.txt and verify the deployed response. The comparison here retains its July 2026 documentation cutoff; revisit provider definitions before changing policy. Permission to crawl does not guarantee indexing, training, or a citation.
Evidence: OpenAI bot documentation · Anthropic crawler guidance · Perplexity crawler documentation
Questions, with evidence · 4 answers with source links and supporting passages
Read this alongside Build search systems you can verify.
AI search, model development, and user-opened URLs need separate decisions. Blocking a training token does not by itself block a provider’s search crawler or user-requested fetcher. The dated comparison below preserves the provider guidance reviewed for this guide; it is not a newly verified inventory.
Publish each named product token in its own intended /robots.txt group with Allow: / or Disallow: /, and retain the correct Sitemap line. The protocol requires case-insensitive product-token group matching; it separately says path matching should be case sensitive. Use the documented branded spellings without treating token casing as an access boundary.
Choose the outcome, select the groups, merge the existing file, and fetch production before inspecting logs. Robots rules guide cooperating clients; protect private files with authentication or server-side authorization regardless of those rules.
Scope and assumptions
This guide documents published crawler controls as verified on July 14, 2026. A robots.txt rule expresses policy; it does not secure private content, authenticate a bot, or prove indexing, ranking, training exclusion, or answer-system citation.
- Agents compared
- 8 USER AGENTS
- Policies
- 2 COPY-READY
- Last verified
- 2026.07.14
- Primary artifact
- ROBOTS POLICY
Separate search, training, and user requests
Choose the intended outcome before copying a rule. Search discovery and model-development opt-out target different agents; user-requested fetching follows provider-specific behavior. Private files require an enforceable access boundary for every client, not another robots group.
Allow: /Disallow: /Provider-specific choiceAuthentication or edge authorization| Operator outcome | Token group | Rule | Release check |
|---|---|---|---|
| AI answer discovery | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Allow: / | /robots.txt returns the intended groups |
| Model-development opt-out | GPTBot, ClaudeBot | Disallow: / | Release time and provider guidance saved |
| User-opened URLs | ChatGPT-User, Claude-User, Perplexity-User | Provider-specific choice | Request type classified correctly |
| Private files | Every client | Authentication or edge authorization | Unauthenticated request denied |
Compare purposes before tokens
Use the named product token rather than a versioned HTTP header. The table distinguishes search, model-development, and user-request roles at its stated verification date. In particular, user-triggered access may follow different robots behavior; do not infer compliance merely from the provider name.
| Provider | Exact token | Mode | robots.txt | Operator note | Verified |
|---|---|---|---|---|---|
| OpenAI | OAI-SearchBot | Search | Supported | Answer-discovery token | 2026-07-14 |
| OpenAI | GPTBot | Model development | Supported | Training opt-out token | 2026-07-14 |
| OpenAI | ChatGPT-User | User request | May not apply | Not an automatic-index token | 2026-07-14 |
| Anthropic | Claude-SearchBot | Search | Honored | Claude search token | 2026-07-14 |
| Anthropic | ClaudeBot | Model development | Honored | Training token | 2026-07-14 |
| Anthropic | Claude-User | User request | Honored | User-opened URL token | 2026-07-14 |
| Perplexity | PerplexityBot | Search | Supported | Search token; not training | 2026-07-14 |
| Perplexity | Perplexity-User | User request | Generally ignored | User-requested fetch token | 2026-07-14 |
Merge the example with the host’s policy
Choose a template, then reconcile it with existing search-engine, media, path, and sitemap rules. Replace the example sitemap hostname. The broad block names only this guide’s agents; its user-fetch exceptions remain important, and private-path rules do not secure the underlying content.
Allows search-oriented and user-triggered access while blocking the two published training/model-development agents in this guide.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
Sitemap: https://example.com/sitemap.xmlBlocks every AI-related agent named in this guide. No static list can cover every crawler, ChatGPT-User rules may not apply, and Perplexity-User generally ignores robots.txt, so enforce sensitive-content controls at authentication or the server edge.
User-agent: OAI-SearchBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /Check production and retain attribution uncertainty
Fetch every production /robots.txt endpoint and compare its returned groups with the committed file. Inspect current and rotated NGINX logs using the filter below. A token match records a claim; check its address against the provider’s dated JSON ranges before attributing traffic, and retain uncertainty when the evidence is insufficient.
- Expected response: HTTP 200, text/plain, intended group, correct sitemap hostname.
- Useful fields: timestamp, host, path, status, source address, complete User-Agent.
- Interpretation rule: absence from the log does not prove a block succeeded.
Adjust the log path for your server or CDN export. This reads current and rotated gzip logs without changing them.
zgrep -hEi 'OAI-SearchBot|GPTBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' /var/log/nginx/access.log*
curl -sS -D - https://example.com/robots.txt -o /dev/nullKeep the policy claim narrower than the evidence
Test the deployed groups and classify observed traffic with provider-specific guidance and dated ranges. A named-agent rule does not cover every client, prove compliance, or establish indexing, training exclusion, or citation.
Questions, with evidence
The provider comparison retains its July 14, 2026 documentation cutoff. The Google/OpenAI control distinction below was checked against official documentation on October 6, 2026. Consult current provider definitions before deploying a policy; these notes do not verify any host’s live crawler settings.
Can I allow ChatGPT search discovery while blocking GPTBot? — link to this answer
The documented mechanisms have separate purposes: OAI-SearchBot serves search discovery, while GPTBot concerns model development. Their named robots groups can express separate choices. Merge those groups into the existing host policy, preserve path restrictions, and verify the deployed file. Search permission is an eligibility input; it does not establish that your page will be indexed or cited.
Evidence: OpenAI’s crawler definitions
Read the supporting analysis
Test the deployed policy and separate controls
Will a training-crawler block stop a user asking an AI assistant to open a URL? — link to this answer
Do not assume so. User-requested retrieval is a separate mechanism with provider-specific behavior; OpenAI states that robots.txt rules may not apply to ChatGPT-User requests. Blocking a model-development token is therefore not a general access barrier. If the content must be private, require authorization for every client rather than relying on a crawler-purpose label.
Evidence: OpenAI’s ChatGPT-User explanation · Google’s robots.txt limitations
Read the supporting analysis
Protect the resource independently of robots rules
What does an AI crawler visit prove about search visibility? — link to this answer
A retained request can establish that a client reached the measured layer and received a particular response at a particular time. It does not establish indexing, model training, answer selection, a citation, or a human referral. Keep observed requests, attributed identities, and downstream search outcomes as separate measurements; absence from one log is not proof of absence everywhere.
Evidence: OpenAI search-crawler documentation
Read the supporting analysis
Record the environment and scope of a capture
Does blocking Google-Extended block Google's AI search answers? — link to this answer
No. Googlebot controls crawling for Google Search, including AI Overviews and AI Mode. Google-Extended controls specified Gemini training and grounding uses; it does not change Search inclusion or rankings and has no separate HTTP request user agent. To limit content shown in Search, use Google's documented snippet or noindex controls. OpenAI separately defines OAI-SearchBot for ChatGPT search and GPTBot for possible model training. Express each purpose-specific choice in the appropriate robots group, preserving existing path restrictions. Crawl permission does not establish indexing or citation.
Evidence: Google's common crawler and Google-Extended definitions · Google's controls for AI features in Search · OpenAI's crawler definitions
Read the supporting analysis
Match each intended outcome to its control
Sources
- OpenAI crawler documentation
Verified Jul 14, 2026
- Anthropic crawler guidance
Verified Jul 14, 2026
- Perplexity crawler documentation
Verified Jul 14, 2026
- Google robots.txt introduction and limitations
Verified Jul 14, 2026
- OpenAI OAI-SearchBot IP ranges
Verified Jul 14, 2026
- OpenAI GPTBot IP ranges
Verified Jul 14, 2026
- OpenAI ChatGPT-User IP ranges
Verified Jul 14, 2026
- Anthropic bot IP ranges
Verified Jul 14, 2026
- PerplexityBot IP ranges
Verified Jul 14, 2026
- Perplexity-User IP ranges
Verified Jul 14, 2026