AI Crawler Registry: Search, Training, User Fetchers, and Product Bots
A maintained field guide to the major AI-facing crawler identities, what they are for, how robots.txt applies, and what network evidence can verify them.
AI Crawler Registry: Search, Training, User Fetchers, and Product Bots
The useful unit in an AI crawler registry is not “AI company”; it is a specific web-access identity with a documented purpose, robots behavior, and verifiable network source.
Evidence: OpenAI — Overview of OpenAI Crawlers · Anthropic — ClaudeBot, Claude-User, and Claude-SearchBot guidance · Perplexity — crawler and user-agent documentation
Read this alongside Build search systems you can verify.
“AI crawler” has become too broad to be an operational category. OAI-SearchBot and PerplexityBot exist for search discovery. GPTBot and ClaudeBot are tied to model-development crawling. ChatGPT-User, Claude-User, and Perplexity-User are user-triggered retrieval identities. OAI-AdsBot exists for ad landing-page review. Google-Extended is not even a separate HTTP crawler; it is a robots product token.
A registry should make those differences machine-readable. The minimum fields are provider, token, purpose, automatic or user-triggered status, robots policy semantics, full or sample User-Agent, official IP/range endpoint where available, documentation URL, first-seen date, last-verified date, and notes about product-specific exceptions.
Scope and assumptions
This registry focuses on major publisher-facing identities documented by OpenAI, Anthropic, Perplexity, and Google as of September 23, 2026. It is not a claim that every AI product crawler is listed, and individual providers can change behavior or add agents after the evidence date.
- Providers covered
- 4
- Primary classes
- SEARCH / TRAINING / USER / PRODUCT
- Verification
- OFFICIAL DOCS + NETWORK
- Refresh rule
- DATE EVERY RECORD
Current high-value registry entries
OpenAI: OAI-SearchBot for Search, GPTBot for model-development crawling, ChatGPT-User for user-triggered fetches, OAI-AdsBot for ad landing-page validation. Anthropic: ClaudeBot for model development, Claude-SearchBot for search, Claude-User for user-directed retrieval. Perplexity: PerplexityBot for search and Perplexity-User for user requests.
Google requires a different mental model. Googlebot powers ordinary Search crawling; Google-Extended is a policy token for defined Gemini uses rather than a separate network User-Agent. Google also documents user-triggered fetchers separately.
| Provider | Identity | Primary purpose | Automatic? | Key control |
|---|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT Search discovery | Yes | robots.txt + published IP ranges |
| OpenAI | GPTBot | Model-development crawling | Yes | robots.txt + published IP ranges |
| OpenAI | ChatGPT-User | User-triggered retrieval | No | Access control; robots may not apply |
| OpenAI | OAI-AdsBot | Ad landing-page validation | Triggered by ad submission | Reachability + published IP ranges |
| Anthropic | ClaudeBot | Model-development crawling | Yes | robots.txt + provider network evidence |
| Anthropic | Claude-SearchBot | Search crawling | Yes | robots.txt + provider network evidence |
| Anthropic | Claude-User | User-directed retrieval | No | Provider-documented bot policy |
| Perplexity | PerplexityBot | Search discovery | Yes | robots.txt + published IP ranges |
| Perplexity | Perplexity-User | User-triggered retrieval | No | Generally ignores robots.txt; IP verification |
| Google-Extended | Gemini training/grounding policy token | Not a separate UA | robots.txt product token |
Generate policy from purpose, not brand
A publisher may want search discovery but not automatic model-development crawling. OpenAI and Anthropic expose separate tokens that can express that outcome. A blanket Disallow for every token containing the provider name throws away that granularity.
User-triggered fetchers need another decision because several providers document robots behavior differently for user requests. Private content still belongs behind authentication regardless of crawler policy.
Keep a network-verification column
OpenAI and Perplexity publish product-specific IP JSON files. Google publishes crawler IP data and DNS verification methods. Anthropic publishes bot/network guidance. Store the official endpoint rather than copying a list into a wiki that will age silently.
When ingesting logs, stamp the exact range-file retrieval time used for classification. That makes a historical attribution reproducible even after the provider changes its current network ranges.
What should trigger a registry refresh
Refresh when a provider changes crawler documentation, a new token appears in logs, a known token arrives from previously unseen networks, or a product launches a new search, ads, agent, or user-fetch workflow. A periodic scheduled check catches quieter changes.
Version the registry. Crawler policy is infrastructure: an unrecorded semantic change can alter search visibility, training preference, WAF behavior, and analytics at the same time.
Maintain the registry as operational infrastructure
Crawler names and IP ranges change. A durable registry stores the provider, token, purpose, automatic-versus-user-triggered class, robots semantics, official network source, and last-verified date. Policy rules and log analytics should both derive from that record instead of hand-maintained folklore.
Sources
- OpenAI — Overview of OpenAI Crawlers
Verified Sep 23, 2026
- Anthropic — ClaudeBot, Claude-User, and Claude-SearchBot guidance
Verified Sep 23, 2026
- Perplexity — crawler and user-agent documentation
Verified Sep 23, 2026
- Google — common crawlers and Google-Extended
Verified Sep 23, 2026
- Google — verify crawler and fetcher requests
Verified Sep 23, 2026
Supporting material
- AI crawler robots.txt policy guide
Copy-ready policy patterns for separating search discovery, training crawlers, and user-triggered fetchers.
- robots.txt access-control boundary
Why crawler policy is not authentication or authorization, with RFC 9309 context.