How to Verify an AI Crawler in Server Logs
A User-Agent match is a lead, not proof. Verification should join request logs to provider-published IP ranges, response codes, robots policy, and time-bounded network data.
How to Verify an AI Crawler in Server Logs
Crawler attribution should be probabilistic until network evidence confirms it: parse the claimed User-Agent, validate the source IP against the provider’s current ranges or verification method, then preserve the HTTP response and robots state.
Evidence: OpenAI — Overview of OpenAI Crawlers · Anthropic — ClaudeBot, Claude-User, and Claude-SearchBot guidance · Perplexity — crawler and user-agent documentation
Read this alongside Build search systems you can verify.
Searching access logs for “GPTBot” or “ClaudeBot” is useful, but it is not verification. Any client can send a copied User-Agent string. For crawler analytics, allowlisting, abuse controls, or a public study, the log record should be joined to provider-published network evidence and the actual HTTP response.
The practical data model is small: timestamp, host, path, query string if retained, claimed User-Agent, source IP after trusted-proxy normalization, HTTP status, bytes, edge/WAF action, and a verification field such as verified-range, failed-range, unknown-range, or user-triggered class.
Scope and assumptions
Provider IP ranges can change, proxies can mask client addresses, and some user-triggered fetchers have different robots behavior from automatic crawlers. Network-range matching supports attribution but does not identify the individual end user or prove how fetched content was later used.
- Minimum identity signals
- UA + NETWORK
- Operational signals
- PATH + STATUS + TIME
- Common spoof risk
- USER-AGENT ONLY
- Evidence date
- 2026.09.23
First recover the real client IP
On a site behind Cloudflare, Fastly, a load balancer, or another reverse proxy, the socket peer may be the proxy rather than the crawler. Use the platform’s trusted client-IP field or forwarding header only after configuring a trusted proxy chain; accepting arbitrary X-Forwarded-For headers from the public internet creates another spoofing path.
Preserve the original request metadata long enough to audit a classification error. A derived daily count without the underlying IP, path, and status code is hard to reproduce.
Match the source address to current provider ranges
OpenAI publishes separate JSON IP ranges for OAI-SearchBot, GPTBot, ChatGPT-User, and OAI-AdsBot. Perplexity publishes ranges for PerplexityBot and Perplexity-User. Anthropic publishes bot network information from its crawler guidance. Fetch the current provider file on a schedule instead of hard-coding a list indefinitely.
For Google, the official method supports matching published IP lists or performing reverse-DNS and forward-DNS verification. That is a useful model for any crawler study: provider network evidence should be stronger than the string in the header.
A verified bot can still be blocked
Identity and access are separate fields. Once a request is attributed, record whether robots.txt allowed the automatic crawler and whether the edge actually returned 200, 301, 403, 429, 5xx, or another response. A nominally allowed bot that receives 403 from the WAF is not successfully crawling the page.
Rate limiting often appears as a traffic-shape problem rather than an explicit robots problem. OpenAI’s advertiser guidance specifically points site owners toward 429 responses, firewall logs, bot-mitigation events, and throttling rules when crawler access fails.
Store observations, not just totals
A useful record might read: OAI-SearchBot claimed; source IP matched the current OpenAI SearchBot ranges; GET /research/...; 200 response; robots allowed; first seen 14:04:12Z; verification file fetched at 13:00Z. That statement can be audited later.
Do not infer indexing, citation, model training, or a human user from a successful fetch. The server proves that bytes were requested and served. Product behavior is a separate measurement problem.
Log evidence should survive a spoofed User-Agent
A trustworthy crawler observation names the token, source IP, provider-verification result, URL, timestamp, HTTP status, robots policy in force, and whether a CDN or proxy altered the apparent client address. Anything less is traffic labeling, not identity verification.
Sources
- OpenAI — Overview of OpenAI Crawlers
Verified Sep 23, 2026
- Anthropic — ClaudeBot, Claude-User, and Claude-SearchBot guidance
Verified Sep 23, 2026
- Perplexity — crawler and user-agent documentation
Verified Sep 23, 2026
- Google — verify crawler and fetcher requests
Verified Sep 23, 2026
Supporting material
- AI crawler robots.txt policy guide
Copy-ready policy patterns for separating search discovery, training crawlers, and user-triggered fetchers.
- robots.txt access-control boundary
Why crawler policy is not authentication or authorization, with RFC 9309 context.