Robots.txt Is a Courtesy Layer, Not Access Control

Direct answer: is robots.txt access control

Robots.txt is not access control: it is a publicly readable crawler-coordination protocol that cooperative agents may follow, while authentication, authorization, network policy, and server-side response controls must protect material that cannot be publicly retrieved.

Original research artifact

A layered crawler-policy matrix separating discovery guidance, authentication, authorization, network controls, and response handling.

What this page adds

Show the complete control stack and failure modes instead of repeating that robots.txt is voluntary without an implementation boundary.

Related research

• robots.txt GPTBot OAI-SearchBot
• crawl frontier design
• raw HTML vs rendered DOM

Name the six control layers before writing rules

The first layer is discovery policy: robots.txt communicates path rules to crawlers that implement the protocol. The second is document processing policy: robots meta elements and response headers can ask supporting indexers not to index a fetched resource or not to expose certain features. The third and fourth are authentication and authorization, which establish who or what is making a request and whether that principal may perform the action.

The fifth layer protects service capacity with connection limits, rate limits, quotas, caching, circuit breakers, and workload isolation. The sixth is observation: logs, challenge outcomes, provider-published address ranges, and response evidence used to classify requests after or during access. Observation can inform an enforcement decision, but a log label alone does not protect a route.

Write the desired outcome first. “Do not spend crawler capacity on faceted combinations,” “do not expose this account document publicly,” “allow search discovery but decline a named model-development crawler,” and “keep the origin healthy under bursts” are different requirements. Each belongs to a different combination of layers.

Implement robots semantics as a versioned policy parser

RFC 9309 standardizes the Robots Exclusion Protocol, including user-agent matching, rule matching, access results, and caching. A production parser should be covered by fixtures for casing, group selection, percent-encoding, longest-match behavior, comments, empty rules, Unicode handling, and retrieval failure states. A hand-written substring test is not an adequate policy engine.

Policy is scoped to the service where the file is served. Hostname, scheme, and port distinctions matter operationally. Redirects and cache behavior must be recorded because a stale or cross-host file can produce a policy different from the one an operator inspected in source control. Store the retrieved bytes, effective URL, status, fetch time, parser version, selected group, and matched rule.

A missing, unreachable, or invalid file is not the same observation as an explicit Allow or Disallow. Preserve the protocol-derived result and any conservative local override as separate fields. That distinction lets operators change local risk posture without rewriting what the remote service published.

• Fetch the production file from every relevant scheme, host, and port combination.
• Keep the raw policy artifact and its redirect chain.
• Version parser fixtures against the standard and known edge cases.
• Record the selected group and exact matching rule for each decision.
• Expire cached policy deliberately and expose stale-policy state.

Protect private content even when every crawler rule disappears

The practical test for confidential content is simple: if robots.txt were deleted and the client changed its User-Agent, would the server still deny the request? If not, the content is public. Move the decision to an authorization layer that validates a principal and checks permission for the resource on every request. Default denial is appropriate when no rule grants access.

Do not publish sensitive paths in robots.txt as a substitute for protection. The file is public and can reveal route names. A disallowed URL can also be discovered from links, logs, referrers, sitemaps, or prior crawls. The response itself must avoid returning the protected representation to an unauthorized requester.

Choose HTTP responses according to the application and standard semantics, then test them at the boundary. The important invariant is not a specific status alone; it is that an unauthenticated or unauthorized request cannot obtain the representation through alternate methods, encodings, hosts, cache keys, file variants, or indirect endpoints.

Treat User-Agent as a claim, not an identity

Any client can send a familiar crawler token. The header is still useful for routing logs and selecting a robots group because the protocol depends on crawler identification, but it should not unlock privileged content or bypass expensive-request controls. Security decisions need a stronger authenticated signal or a policy that remains safe when identity is unknown.

Some providers publish address ranges or verification guidance. Use those records as dated attribution evidence, not as a timeless allowlist copied into application code. Retrieve the provider artifact securely, validate its format, record the version or hash, compare the observed address, and define what happens when the feed is stale or unavailable. Network ownership can support classification without proving the purpose of an individual request.

Separate observed, claimed, and attributed identity in logs. Observed fields include address, TLS and request properties, time, host, path, and complete User-Agent. Claimed identity is the token. Attributed identity is a derived conclusion with method and confidence. This prevents dashboards from turning a string match into a verified traffic count.

curl -fsS -D robots.headers https://example.com/robots.txt -o robots.body

curl -sS -o /dev/null -w "%{http_code}\n" \
  -A "ExampleCrawler/1.0" https://example.com/private/account

curl -sS -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0" https://example.com/private/account

Release policy with an evidence matrix

A crawler-policy change can affect discovery, indexing workflows, origin load, data use, and user-triggered retrieval. Review the exact user-agent groups, path rules, existing search-engine rules, sitemap declarations, and inherited edge controls before deployment. Generate the final file from structured policy when several hosts or environments must stay aligned.

After deployment, fetch the public file without relying on a local filesystem copy. Assert status, content type, expected group selection, sitemap hostname, redirect behavior, and cache headers. Exercise representative allowed and disallowed paths with the policy parser, then separately test authorization and rate-control boundaries. A robots test passing says nothing about the private-route test.

Monitor outcomes with bounded language. Logs can show requests carrying selected tokens, response statuses, and rate-control events. They cannot show requests that never reached the measured layer, prove full provider identity from a token, or prove downstream indexing or training decisions. A useful policy report includes those gaps instead of converting silence into compliance.

• Review every existing group before merging a new crawler policy.
• Test the deployed file from outside the origin environment.
• Run parser fixtures against representative site paths.
• Test private resources with no credentials and wrong credentials.
• Measure origin rejections and latency after capacity-policy changes.
• Date every provider-specific identity source and attribution rule.