Robots.txt errors and cached rules

Classify the robots fetch before changing a rule

Every status in this table belongs to GET /robots.txt, not to an article or a private download. A 403 on the policy file does not protect a different URL. Inspect the serving hostname and the redirect destination before deciding what failed.

Follow Google’s documented failure sequence

Google’s robots status and cache specification describes an initial 12-hour crawl pause after a server-side policy fetch failure. It keeps retrying the policy. In the following 30-day period it can use the last good version; without one it assumes no restrictions.

After that prolonged failure, general site availability affects the next step: an otherwise available site can be treated as having no robots file, while broader availability trouble can keep crawling stopped. This is Google’s documented behavior, not a retry algorithm prescribed for every REP client.

The important operational difference is the version: an outage can preserve yesterday’s restrictions after you edited today’s file. Saving the current policy on your laptop does not show that Google retrieved it.

One edit, two copies

Consider this illustrative release: version A disallows /drafts/. At 09:00 you deploy version B with an exception for /drafts/public-guide. At 09:10 an edge configuration starts returning 503 for /robots.txt. A crawler that retained A can continue making decisions from A under its failure policy; a browser that sees B after recovery observes a different time and copy.

Capture the policy body with its retrieval time, final URL, Cache-Control, Age, ETag and Last-Modified when those headers exist. Those headers help explain delivery through caches; their presence alone does not prove what a particular crawler stored. RFC 9111 defines HTTP caching, while REP and the provider govern the policy’s use.

Google normally caches robots.txt for up to 24 hours, can adjust lifetime using max-age, and can retain it longer when refresh is impossible. Do not convert 24 hours into a guaranteed propagation deadline.

curl --silent --show-error --location --max-redirs 10 \
  --dump-header robots.headers \
  --output robots.body \
  --write-out "status=%{http_code} final=%{url_effective}\n" \
  https://example.com/robots.txt

Recover the policy endpoint, then inspect the page

Restore a stable plain-text policy response at /robots.txt, remove unintended challenge or redirect loops, and confirm the required rules survived generation and edge caching. Test representative paths against the exact returned body. The wildcard fixture covers matching separately from retrieval.

Then inspect the affected page. Its own noindex header, canonical, status and content may explain an indexing problem after the policy fetch recovers. Use provider tools and verified server records for a later fetch observation; do not repeatedly change policy merely because a report still shows an older crawl.

If the actual problem is origin overload, use deliberate capacity controls. Google’s crawl-rate guidance describes temporary error responses and their Search consequences; breaking /robots.txt is an imprecise way to manage expensive requests. The crawl-delay support guide compares the available cooperation mechanisms.

Use the crawler directive response chooser when the policy endpoint works but the page’s own indexing response still needs a decision.