Owned local-host Crawl-delay example — October 26, 2026

Requires Node 22+. No packages, accounts, external hosts or provider tokens.
Extract the ZIP and run: node run.mjs

The script binds only to 127.0.0.1 on an automatic port, serves a synthetic
robots.txt for DelayFixtureBot with Crawl-delay: 1, fetches that policy, then
makes three paced and three unpaced page requests to that same server.
The paced mode waits at least one second after each completed response.
The server records monotonic arrival times and prints the measured gaps.
The two later paced arrivals must be at least 1000ms apart or the script fails.
Unpaced mode does not wait; its exact observed gaps depend on your machine.
The server closes on success or failure. There is no persistent process.

This demonstrates a client-controlled pacing choice, not a server throttle.
It does not impersonate or test Bingbot, ClaudeBot, Googlebot or Yandex.
Its minimum-gap scheduler is NOT Bing's one-URL-per-time-window algorithm.
No result establishes production support, compliance, indexing or attribution.

Guide: https://sulayman-bowles.dev/research/ai-crawlers/crawl-delay-provider-support
Current provider references are cited in that guide. Change a temporary copy
when exploring the code; do not point this finite example at another host.
