AI Crawler Robots.txt Guide: GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot

Copy-ready host rules for eight named product tokens, with exact Allow, Disallow, sitemap, and release checks.

Direct answer: robots.txt GPTBot OAI-SearchBot

Robots.txt can express different crawl preferences for GPTBot, OAI-SearchBot, and other named agents, but those directives govern crawler access only; they do not guarantee indexing, rankings, citations, model inclusion, or retrieval by a user-requested tool.

Original research artifact

A maintained provider-agent matrix and public CSV source map for crawler identities, purposes, and policy boundaries.

What this page adds

Keep search, training, and user-requested retrieval agents distinct and tie every directive to provider documentation and a dated source map.

Related research

Monochrome editorial artwork showing crawler pathways meeting policy and access-control boundaries. — Named crawler rules express policy; authentication and edge controls protect private resources.

Memo Details

Category: CRAWLER POLICY. Author: SULAYMAN BOWLES. Published: 2026.06.19. Read time: 10 MIN. Source count: 10.

Evidence Boundary

This guide documents published crawler controls as verified on July 14, 2026. A robots.txt rule expresses policy; it does not secure private content, authenticate a bot, or prove indexing, ranking, training exclusion, or answer-system citation.

Article Metrics

Agents compared

8 USER AGENTS

Policies

2 COPY-READY

Last verified

2026.07.14

Primary artifact

ROBOTS POLICY

Research Note

/robots.txt is a host-scoped text file. Match the exact case-sensitive product token, then assign Allow: / or Disallow: /. Preserve the correct Sitemap line.

Release sequence: outcome → token → group → file merge → production fetch → access-log check.

Choose the policy outcome before choosing a user-agent

Choose one row before editing the file.

Outcome-to-user-agent decision matrix

Operator outcome	Token group	Rule	Release check
AI answer discovery	OAI-SearchBot, Claude-SearchBot, PerplexityBot	Allow: /	/robots.txt returns the intended groups
Model-development opt-out	GPTBot, ClaudeBot	Disallow: /	Release time and provider guidance saved
User-opened URLs	ChatGPT-User, Claude-User, Perplexity-User	Provider-specific choice	Request type classified correctly
Private files	Every client	Authentication or edge authorization	Unauthenticated request denied

AI crawler comparison table

Target the stable token, not a versioned HTTP header.

Published AI crawler roles and robots.txt behavior, last verified July 14, 2026

Provider	Exact token	Mode	robots.txt	Operator note	Verified
OpenAI	OAI-SearchBot	Search	Supported	Answer-discovery token	2026-07-14
OpenAI	GPTBot	Model development	Supported	Training opt-out token	2026-07-14
OpenAI	ChatGPT-User	User request	May not apply	Not an automatic-index token	2026-07-14
Anthropic	Claude-SearchBot	Search	Honored	Claude search token	2026-07-14
Anthropic	ClaudeBot	Model development	Honored	Training token	2026-07-14
Anthropic	Claude-User	User request	Honored	User-opened URL token	2026-07-14
Perplexity	PerplexityBot	Search	Supported	Search token; not training	2026-07-14
Perplexity	Perplexity-User	User request	Generally ignored	User-requested fetch token	2026-07-14

Copy-and-paste robots.txt configurations

Merge one template with the host’s existing search-engine, media, private-path, and sitemap groups. Replace the example sitemap hostname before release.

Allow AI search but block training — Allows search-oriented and user-triggered access while blocking the two published training/model-development agents in this guide.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

Sitemap: https://example.com/sitemap.xml

Block all AI crawlers — Blocks every AI-related agent named in this guide. No static list can cover every crawler, ChatGPT-User rules may not apply, and Perplexity-User generally ignores robots.txt, so enforce sensitive-content controls at authentication or the server edge.

User-agent: OAI-SearchBot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

Verify deployment with server logs

Fetch each production /robots.txt endpoint and compare the returned group with the committed file. Then run the log filter below against current and rotated NGINX files. A matching token is only a claim, so check the address against the provider’s linked JSON range before attributing the request.

Expected response: HTTP 200, text/plain, intended group, correct sitemap hostname.
Useful fields: timestamp, host, path, status, source address, complete User-Agent.
Interpretation rule: absence from the log does not prove a block succeeded.

Find the named agents in NGINX logs — Adjust the log path for your server or CDN export. This reads current and rotated gzip logs without changing them.

zgrep -hEi 'OAI-SearchBot|GPTBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' /var/log/nginx/access.log*

curl -sS -D - https://example.com/robots.txt -o /dev/null

Thesis

Treat search crawlers, training crawlers, and user-triggered fetchers as separate policy choices, then verify observed traffic with server logs and provider-published IP ranges.

Choose the outcome before the token

Search discovery, model-development access, and user-triggered fetches are separate policy decisions. Encode each by exact agent group, deploy the merged file, then use provider ranges and logs to verify observed traffic without mistaking a user-agent string for identity.