Is Cloudflare or your web host blocking AI crawlers?

Which layer is making the decision?

Follow the request from public policy to edge enforcement and then to the application. A robots disallow asks a compliant crawler to refrain from requesting a path. A WAF can stop the HTTP request. An application can return a login form even after the edge passes it.

The layers can disagree without any component being broken. A publisher may intentionally decline training collection while allowing search, or keep a public help page open while requiring a session for account data. Record the intended result before trying to make every indicator green.

Does the edge serve the file you edited?

Fetch /robots.txt from the exact public hostname. Save its bytes, response headers and check time. Compare it with the source version before assuming a recent commit changed the live policy.

Cloudflare’s current robots setting can prepend managed directives to an existing origin file. Its October 8, 2026 documentation describes Bot Preference Sync using Search, Agent and Training policies. Inspect the combined response, especially when a managed group and an origin group name the same crawler. Managed robots.txt behavior.

Current Cloudflare guidance also offers behavior-specific Search, Agent and Training policies. Check the active values on your zone instead of following an old screenshot of a single “Block AI bots” toggle. AI bot policy controls. You may want the training block to remain in place while repairing search access.

What did the denied request receive?

Use an ordinary GET to retain the response body. A HEAD request is a useful header probe, but it omits the document that would tell you whether the client received an article, error page or challenge. The commands in the crawler policy guide provide a first policy probe; the body-saving example below goes further.

• For a denial, find the matching security event using its time, host, path and request ID.
• For a challenge, inspect headers and body; Cloudflare documents cf-mitigated: challenge.
• For 429 responses, inspect the active throttle and request volume.
• For 200 responses, confirm that response.html contains the expected page text.

curl --connect-timeout 10 --max-time 30 --silent --show-error --location \
  --dump-header response.headers --output response.html \
  --write-out 'final_url=%{url_effective} status=%{http_code}\n' \
  https://example.com/public-article/

How would a scoped fix work?

Consider a public service page whose robots policy permits OAI-SearchBot. A later request claims that agent, matches the relevant dated provider network evidence, and receives 403 at the edge. The security event points to a country restriction that runs before the crawler rule. This is the illustrative incident; each field would need evidence in a real case.

The repair starts with the country rule. Decide whether the selected provider should have an exception on that hostname and public path, then change that rule or its intended exception. Keep the original expression and a rollback route. Adding another Allow line to robots.txt would leave the HTTP denial untouched.

Where an exception relies on identity, combine the claimed agent with maintained provider verification evidence or the platform’s documented verified-bot mechanism. A User-Agent string alone is cheap to copy. OpenAI and Perplexity publish product-specific network sources; use the feed for the product you intend to admit.

Cloudflare’s Skip action can target selected security products, and its documentation says it cannot bypass Bot Fight Mode. Check the specific product before designing an exception around Skip. Skip action reference. A blanket bypass for every request containing “bot” would expand the change far beyond the denied public-page request.

What evidence closes the incident?

Repeat the ordinary diagnostic after the change and confirm that the public article arrives. Check an excluded or protected path as well, so the repair does not silently open more than intended. For Cloudflare Challenge Pages, the documented cf-mitigated header helps classify the result; its absence is no substitute for inspecting the body. Challenge response reference.

A request made with curl -A GPTBot still originates from your machine. It can reveal a string-based rule, but it cannot reproduce OpenAI’s source address, fingerprints or request behavior. A provider access claim needs a later provider-attributed event, classified with the server-log method.

If the edge serves a cached article, the origin may have no matching hit. Preserve edge evidence and join logs where possible. If no attributable provider request arrives during the observation window, leave that final check pending and record the working local response.

I would close the incident with a small receipt: the old denial, the rule changed, the new delivered document and the state of the provider check. That gives the owner a repair they can explain, while keeping intentional training and private-content restrictions intact.