Do AI crawlers respect robots.txt? What the evidence shows

What the controlled studies add

A 2025 preprint, Scrapers selectively respect robots.txt directives, studies institutional logs over 40 days and 130 self-declared bots under changing rules. It reports selective compliance and weaker compliance under stricter directives. Its setting and self-declared identities limit how directly those results transfer to a different website or current product.

A July 2026 preprint, Do Generative AI Assistants Respect robots.txt?, tests ten assistants across 200 controlled trials. It combines server observations with secret page codes and varies universal and agent-specific policies. The authors report substantial differences and show that page access can diverge from visible answer correctness.

These studies offer experimental structure, rather than a universal verdict about a provider forever. An assistant configuration, retrieval integration and date are part of the treatment. A later product or a different access route needs its own check. Neither study turns a passive User-Agent count on this site into the same kind of experiment.

Separate a policy promise, an observed request and a proven violation

RFC 9309 defines robots rules as requests to cooperating crawlers, with caching and error handling. It does not make the file an authorization system. A technical deny needs an edge or application control. Protocol and security boundary.

Provider purpose matters too. OpenAI distinguishes automatic crawling from user-triggered ChatGPT-User requests, to which robots rules may not apply. A request associated with an assistant should not automatically be graded as a training crawler. Use the current agent map before defining the test.

A policy promise is the provider’s stated behavior. An observation is what the site records. A violation finding joins a defensible identity to a prohibited request after accounting for the applicable policy and its timing. Keeping those stages separate makes a report more useful to the owner and to anyone investigating it.

What my logs show, and what they leave unknown

The 90-day snapshot contains 26,083 self-identified-bot events and 4,265 suspected-automation events. All identities are unverified. The observer records selected request arrivals through a best-effort pipeline; it does not store returned HTTP status or content.

It also lacks a versioned timeline of the robots bytes served at each request time. Comparing a July request with today’s file would substitute a later policy for the one the client could have read. A request shortly after a policy change also requires attention to caching and propagation.

Consequently, I report the compliance result as unmeasured. That is different from asserting that every request complied, and different from accusing every claimed agent of ignoring a rule. The existing dataset is useful as a queue of cases to verify.

What would make the next test persuasive?

Use an authorized, harmless test surface and archive each policy version with its effective time. Keep page content, public discoverability and client configuration stable enough to interpret the policy change. Test allowed and disallowed conditions with repeated runs instead of judging one screenshot.

Record the request path and method, trusted identity evidence where available, response status and a content-specific marker. Log robots requests and possible cache intervals. Compare the served document with the visible answer separately; either stage can succeed while the other fails.

Predefine the classification: compliant observation, prohibited retrieval with adequate evidence, attempted request denied by the edge, identity unresolved, policy unresolved or no observable attempt. Preserve uncertainty in the denominator. Excluding every ambiguous case silently would make a rate look stronger than the measurement.

How the owner can act before that experiment exists

Keep robots.txt aligned with the publishing decision, and use real authorization for private material. If the immediate issue is load or an unwanted request, inspect the acting edge rule and delivery evidence. You can repair that incident without first proving a platform-wide behavioral claim.

The access-checking guide separates policy, delivery and provider observation. The verification guide covers identity evidence. Together they produce a report that names what was observed and leaves the broader inference proportionate.