ClaudeBot vs Claude-SearchBot vs Claude-User
Anthropic separates model-development crawling, web-search crawling, and user-directed retrieval into three robots with different purposes.
ClaudeBot vs Claude-SearchBot vs Claude-User
Anthropic’s crawler taxonomy separates training-oriented collection, search indexing, and user-requested retrieval, allowing a publisher to express distinct policies instead of one all-or-nothing Claude decision.
Evidence: Anthropic — ClaudeBot, Claude-User, and Claude-SearchBot guidance
Read this alongside Build search systems you can verify.
Anthropic documents three web robots because “Claude accessed my site” can mean three different things. ClaudeBot collects public web content that may contribute to model development. Claude-SearchBot supports web search. Claude-User fetches content when a person asks Claude to visit or use a page.
The separation is operationally useful. A publisher that is comfortable appearing in Claude search results but does not want future automatic model-development crawling can write different groups instead of blocking every Claude-branded request.
Scope and assumptions
Anthropic’s guidance and network ranges can change. robots.txt is a crawler-policy layer, not authentication. This page reports official documentation as checked September 23, 2026 and does not infer training or citation from a server request.
- Anthropic crawler identities
- 3
- Training
- CLAUDEBOT
- Search
- CLAUDE-SEARCHBOT
- User retrieval
- CLAUDE-USER
ClaudeBot: model-development crawling
Anthropic says ClaudeBot helps improve the utility and safety of its generative models by collecting web content that could potentially contribute to training. Restricting ClaudeBot signals that future material from the site should be excluded from Anthropic’s model-training datasets.
That policy should not be described as a cryptographic guarantee about every historical copy of content. It is a documented crawler-control signal for future collection behavior.
Claude-SearchBot: web-search discovery
Claude-SearchBot is the search-oriented identity. It exists so publishers can treat Claude’s search feature differently from model-development crawling.
As with other search crawlers, a robots allow rule can still be defeated by a WAF, rate limiter, CDN challenge, or network block. Production access has to be checked in server or edge logs.
Claude-User: retrieval at a user’s direction
Claude-User supports fetches initiated by people using Claude. This class is closer to a user-agent proxy than a conventional discovery crawler, so its traffic pattern and policy meaning differ from ClaudeBot.
Anthropic’s 2026 guidance also says its bots respect robots directives and supports the non-standard Crawl-delay extension. That implementation detail should be verified against current docs before relying on it as a permanent standard.
How to classify Anthropic traffic in logs
Store the exact claimed token, source IP, status code, and timestamp. Then compare the source address with Anthropic’s current bot/network information. Treat an unmatched “ClaudeBot” header as unverified rather than silently counting it as Anthropic.
Finally, keep the result scoped to access. A 200 response from a verified crawler demonstrates a successful fetch, not whether the page entered a model dataset or appeared in a Claude answer.
Target the Claude identity that matches the outcome
ClaudeBot is the model-development crawler, Claude-SearchBot supports search, and Claude-User supports user-directed retrieval. Site owners should maintain separate robots groups when they want separate outcomes and verify important traffic against Anthropic’s current published network information.
Sources
- Anthropic — ClaudeBot, Claude-User, and Claude-SearchBot guidance
Verified Sep 23, 2026
Supporting material
- AI crawler robots.txt policy guide
Copy-ready policy patterns for separating search discovery, training crawlers, and user-triggered fetchers.
- robots.txt access-control boundary
Why crawler policy is not authentication or authorization, with RFC 9309 context.