Which crawlers support crawl-delay?

The same line produces different expectations

RFC 9309 permits extra records without making them part of its standard rule set. A crawler can therefore implement Allow and Disallow correctly while ignoring Crawl-delay. Check the product token that actually reaches your host; a policy for one provider does not establish another provider’s behavior.

Keep the provider group’s restrictions complete

Bing’s current help and Crawl Control documentation describe support and precedence. Its older window explanation is useful for the semantics: one URL per time window is not necessarily an exact minimum gap between every pair of requests.

Anthropic’s current bot documentation gives a ClaudeBot delay example. Yandex’s notice explicitly retires the directive. Google lists Crawl-delay among unsupported fields. Those differences are why a copied all-bots delay can leave the traffic you care about unchanged.

In this authored configuration, repeat /drafts/ in each named group. Specific groups do not simply inherit the wildcard block. The hub’s group-selection example explains that boundary.

User-agent: *
Disallow: /drafts/

User-agent: Bingbot
Disallow: /drafts/
Crawl-delay: 5

User-agent: ClaudeBot
Disallow: /drafts/
Crawl-delay: 1

Reproduce pacing on a host you control

Download the local-host pacing example, extract it, and run node run.mjs with Node 22 or later. It binds only to 127.0.0.1 on an automatically assigned port. It starts a temporary policy endpoint, reads its Crawl-delay: 1, makes three paced page requests and three unpaced requests, prints the observed arrival gaps, then closes the server.

The paced synthetic client deliberately waits at least the configured delay after each completed response. The unpaced client performs the same fetches without that wait. Both use DelayFixtureBot, not a real provider’s identity. The server applies no throttle, so the unpaced requests can arrive close together even though robots.txt asks for spacing.

This demonstrates an observable client/server boundary on an owned loopback host. It does not test Bingbot, ClaudeBot or production crawl traffic. The strict spacing chosen for this client is not an emulation of Bing’s window algorithm.

unzip robots-crawl-delay-example.zip -d robots-crawl-delay-example
cd robots-crawl-delay-example
node run.mjs

Measure work at the layer that can limit it

For your actual site, first attribute the caller using provider verification methods, then compare request start times, status codes, concurrent requests and origin work. A synthetic User-Agent request checks your routing; it does not establish that a provider violated its published policy.

If the origin needs a hard ceiling, enforce rate, concurrency, quota and cache rules there or at the acting edge. Keep the policy endpoint available so cooperative clients can retrieve its preferences. A robots-file 403 or a permanently broken endpoint is not a reliable substitute for those controls.

For Google, current crawl-rate guidance describes temporary 500, 503 or 429 responses, optional Retry-After with 503/429, and possible Search impact if errors persist. Use that mechanism deliberately rather than assuming Crawl-delay works. If requests changed after a policy outage, inspect robots fetch errors and cached policies before attributing the change to pacing.

Keep crawl-frontier scheduling and durable retries separate from the selected path’s permission result.