AI Crawler Robots.txt Guide: GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot

Search crawler versus training crawler versus user-triggered crawler

A search crawler builds or improves an index used to retrieve and cite web pages in an answer product. A training crawler gathers public content that may contribute to future foundation-model development. A user-triggered crawler or fetcher visits a URL because a person asked a product to retrieve it at that moment.

Those roles have different consequences. Blocking a search crawler can reduce eligibility for that provider's search results. Blocking a training crawler expresses an opt-out from future training collection without necessarily blocking search. Blocking a user-triggered fetcher can prevent a product from opening a page on a user's behalf, but robots.txt behavior is provider-specific.

• Search: OAI-SearchBot, Claude-SearchBot, and PerplexityBot.
• Training or model development: GPTBot and ClaudeBot.
• User-triggered retrieval: ChatGPT-User, Claude-User, and Perplexity-User.

AI crawler comparison table

Use the exact case-sensitive product tokens below in robots.txt. Version numbers in full HTTP User-Agent strings can change, so policy should target the stable token rather than a copied versioned string.

Copy-and-paste robots.txt configurations

These examples cover the named agents in the comparison table. They are not an exhaustive registry of every AI-related crawler on the web. Preserve any existing rules for Googlebot, Bingbot, media crawlers, private paths, and your sitemap when merging a template into a live file.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

Sitemap: https://example.com/sitemap.xml

User-agent: OAI-SearchBot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

What robots.txt cannot accomplish

Robots.txt manages cooperative crawler access. It does not authenticate a requester, create a legal or technical access barrier, remove a URL from every index, or guarantee that previously collected material is deleted. A malicious client can copy a trusted User-Agent token, and compliant providers can interpret user-triggered requests differently from automatic crawling.

Google also warns that a disallowed URL can still appear in search results when other pages link to it. Use authentication for private content, noindex for crawlable pages that should not appear in supporting search engines, and server or WAF rules when you need enforceable traffic controls. Do not block a page in robots.txt and expect a crawler to read a noindex directive on that blocked page.

• It cannot keep secrets private or replace access control.
• It cannot prove a request came from the company named in its User-Agent.
• It cannot guarantee deindexing, ranking, citation, or model-training outcomes.
• It cannot retroactively govern content already fetched under an earlier policy.
• It cannot cover unknown or newly introduced agents unless the policy is reviewed and updated.

Verify crawler behavior in server logs

Start with raw access logs rather than an analytics dashboard: many bot requests do not execute client-side analytics. Record the timestamp, request path, status, response bytes, source IP, host, and full User-Agent. The example below filters a standard NGINX access-log archive for the exact tokens in this guide.

A matching User-Agent is only a claim. Compare the source IP with the provider's current published CIDR manifest, and retain the manifest retrieval time with your evidence. Do not hard-code today's ranges forever; providers update them. Then check whether requests to /robots.txt preceded content fetches, whether disallowed paths stopped receiving compliant automatic-crawler traffic after the provider's stated propagation window, and whether CDN or WAF rules changed the observed result.

• Confirm /robots.txt returns 200 on every relevant host and contains the intended group.
• Filter logs by exact User-Agent token and review status codes, paths, rate, and timestamps.
• Validate the source IP against the linked OpenAI, Anthropic, or Perplexity JSON manifest.
• Separate automatic crawlers from user-triggered fetchers when interpreting compliance.
• Recheck after policy changes and keep a dated snapshot; absence of requests is not proof of blocking.

zgrep -hEi 'OAI-SearchBot|GPTBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' /var/log/nginx/access.log*

curl -sS -D - https://example.com/robots.txt -o /dev/null