Robots.txt is not access control.
Choose the control that matches the outcome you need.
Sulayman Bowles · · Updated
Deleting robots.txt should not expose a private document. RFC 9309 explicitly says its rules are not access authorization: the public file asks cooperating crawlers how to request paths on a service. A client that ignores it can still send ordinary HTTP requests. The file supplies no credential, firewall, encryption, or verified identity.
Crawl guidance, index processing, private access, and capacity protection need different controls. Robots directives express preferences to supporting systems; authentication identifies a principal; authorization checks permission; edge and application controls limit work. Logs then provide evidence about requests, not a substitute for enforcement.
In this article
Will robots.txt keep a private page out of search and prevent access?
No. Robots.txt asks cooperating crawlers which paths to request; it does not authenticate a visitor or deny access. Google can still list a disallowed URL discovered elsewhere. For a public page that should not be indexed, use a supported noindex instruction that the crawler can retrieve. For private content, enforce authentication and authorization at the server. Test those boundaries separately, and treat a claimed bot User-Agent as a claim rather than verified identity.
Sources: RFC 9309, security considerations · Google robots.txt limitations · OWASP authorization guidance
Questions and sources · 3 questions
Read this alongside Build search systems you can verify.
Scope and assumptions
The controls in this article describe publisher-side policy and enforcement. A rule does not prove that every client complies, that a provider attributed a request the same way, or that an allowed page will be crawled, indexed, trained on, or cited.
- Control layers
- 06
- Security boundary
- SERVER
- Identity claim
- UNTRUSTED
- Primary artifact
- POLICY MATRIX
The protocol and Google’s guidance
RFC 9309 section 3 distinguishes crawler rules from access authorization. Google’s robots.txt guidance explains the corresponding limits for Search. Both sources were checked on September 10, 2026; other source dates retain their original cutoffs.
| Official source | What it establishes | What it does not establish |
|---|---|---|
| RFC 9309, section 3 | Publishing robots rules does not secure a resource; use an application-layer security control. | That every client follows the rules or that paths in robots.txt are private. |
| Google Search Central: robots.txt introduction | Robots rules manage cooperative crawling; a blocked URL may still appear in search. | A guarantee of deindexing or confidentiality. |
Match the outcome to the layer
First decide whether you need crawler guidance, index-processing preferences, private access, or capacity protection. Use robots.txt, noindex, authentication with authorization, or rate and concurrency limits accordingly; they do not enforce the same boundary.
Discovery policy communicates path rules through robots.txt. Document-processing policy uses meta elements or response headers to ask supporting indexers not to index fetched content or expose certain features. Authentication establishes a principal; authorization checks whether that principal may perform the action.
Capacity controls include connection and rate limits, quotas, caching, circuit breakers, and workload isolation. Observation uses logs, challenges, published address ranges, and response evidence to classify requests. Classification can inform enforcement, but a dashboard label cannot protect a route.
Write requirements in operational terms: avoid crawler work on facets, keep account documents private, permit search discovery while declining a named model-development crawler, or keep the origin healthy under bursts. Each requires its own combination of layers.
For a public page you want removed from Google’s index, do not block crawling before Google can see its noindex directive. Keep the directive retrievable while the removal is processed, then verify the provider’s reported state separately. For a private page, enforce authorization immediately; search-processing preferences cannot substitute for that requirement. Google’s robots limitations.
- 01Guide cooperative crawlers away from pathsrobots.txt
Does not deny access or establish identity
- 02Keep a public response out of supporting indexesnoindex via meta or X-Robots-Tag
Does not make the response private
- 03Keep a resource privateAuthentication plus server-side authorization
Must hold across hosts, methods, caches, and alternate routes
- 04Protect origin capacityRate, concurrency, quota, and cache controls
Does not decide search or training eligibility
| Desired outcome | Primary control | What it changes | Boundary |
|---|---|---|---|
| Guide cooperative crawlers away from paths | robots.txt | Crawler request policy | Does not deny access or establish identity |
| Keep a public response out of supporting indexes | noindex via meta or X-Robots-Tag | Indexer processing after retrieval | Does not make the response private |
| Keep a resource private | Authentication plus server-side authorization | Whether a principal receives the representation | Must hold across hosts, methods, caches, and alternate routes |
| Protect origin capacity | Rate, concurrency, quota, and cache controls | How much work a client may consume | Does not decide search or training eligibility |
Version the robots parser and its observations
RFC 9309 specifies user-agent and rule matching, access results, and caching. Test casing, group selection, percent-encoding, longest matches, comments, empty rules, Unicode, and retrieval failures with fixtures. Substring matching cannot implement that contract.
Scope policy to the serving service, preserving scheme, host, and port distinctions. Record redirects and caches: stale or cross-host files can differ from inspected source control. Save bytes, effective URL, status, retrieval time, parser version, selected group, and matching rule.
Retrieval failure has semantics too. RFC 9309 permits access when robots.txt is unavailable, such as a 404; it requires assumed complete disallow for an unreachable file caused by server or network errors, with specified caching and prolonged-failure provisions. Save the response and resulting policy state. A local “fail closed on every error” choice is a separate override. Protocol access results.
- Fetch the production file from every relevant scheme, host, and port combination.
- Keep the raw policy artifact and its redirect chain.
- Version parser fixtures against the standard and known edge cases.
- Record the selected group and exact matching rule for each decision.
- Expire cached policy deliberately and expose stale-policy state.
Make privacy independent of robots rules
Remove the robots file and change the client’s User-Agent: does the server still deny protected content? If not, that boundary does not make it private. Validate a principal and check resource permission on every request, denying by default when no rule grants access.
Disallowed paths are not secrets. The public file can reveal them, and links, logs, referrers, sitemaps, or prior crawls can expose them independently. Unauthorized requests must not receive the protected representation.
Use HTTP responses appropriate to the application and standard, then test actual access. Status alone is insufficient: alternate methods, encodings, hosts, cache keys, file variants, and indirect endpoints must not yield the representation to an unauthenticated or unauthorized requester.
Test after an authorized user has loaded the resource as well as before. A shared cache that serves that response to an anonymous requester defeats the application’s permission check even if the original request was authorized. Compare the returned representation and cache behavior, not just a status code; a 200 login page and a 200 private document are different outcomes.
| Requirement | Primary control | Supporting controls | Verification |
|---|---|---|---|
| Keep account data private | Authorization | Authentication, cache isolation, audit log | Anonymous and wrong-principal requests denied |
| Reduce duplicate crawling | Robots policy or URL inventory repair | Canonical routes, link cleanup | Production policy and request logs reviewed |
| Remove a public page from an index | Page-level directive or removal workflow | Status and sitemap updates | Fetched response contains intended directive |
| Protect origin capacity | Rate and concurrency controls | Caching, backoff, workload queues | Load and rejection behavior measured |
| Classify a named provider bot | Multi-signal observation | Published ranges and reverse checks when available | Attribution confidence retained |
Separate claimed and attributed identity
Anyone can send a familiar crawler token. User-Agent remains useful for log routing and robots-group selection, but cannot safely grant privileges or bypass expensive-request limits. Authenticate more strongly or keep the policy safe when identity is unknown.
Provider address ranges and verification guidance are dated attribution evidence. Retrieve them securely, validate the format, record version or hash, compare observed addresses, and specify stale or unavailable-feed behavior. Network ownership may support classification without proving a request’s purpose.
Log observed address, TLS and request properties, time, host, path, and full User-Agent separately from the claimed token. Keep attributed identity as a derived conclusion with method and confidence. A string match should not become a verified traffic count.
The public policy response and protected-resource denial should both be tested.
curl -fsS -D robots.headers https://example.com/robots.txt -o robots.body
curl -sS -o /dev/null -w "%{http_code}\n" \
-A "ExampleCrawler/1.0" https://example.com/private/account
curl -sS -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0" https://example.com/private/accountVerify the deployed file and the protected routes
Before changing policy, review all user-agent groups, path rules, search rules, sitemaps, and inherited edge controls. Changes can affect discovery, indexing workflows, origin load, data use, and user-triggered retrieval. Structured generation can keep hosts and environments aligned.
Fetch production rather than inspecting a local copy. Check status, content type, group selection, sitemap host, redirects, and caching. Parse representative allowed and disallowed paths, then test authorization and capacity boundaries independently. Passing robots checks cannot establish private-route protection.
Logs show tokens, statuses, and rate-control events at the measured layer. They omit requests that never reached it, cannot verify full provider identity from a token, and do not reveal indexing or training decisions. Keep these gaps explicit instead of treating silence as compliance.
- Review every existing group before merging a new crawler policy.
- Test the deployed file from outside the origin environment.
- Run parser fixtures against representative site paths.
- Test private resources with no credentials and wrong credentials.
- Measure origin rejections and latency after capacity-policy changes.
- Date every provider-specific identity source and attribution rule.
Test each boundary separately
Fetch the deployed robots policy, then test private access and capacity limits independently. Observed logs can establish which requests and responses reached that layer—not universal compliance, verified provider identity from a token, or downstream use.
Questions, with evidence
These are protocol and control distinctions. A robots rule is crawler guidance; private-access and capacity protection require separate enforcement.
Does Disallow in robots.txt make a document private? — link to this answer
No. The Robots Exclusion Protocol asks cooperating crawlers to avoid requests; it does not authorize users or deny access at the server. A public robots file can also expose the paths it lists. Protect private documents with authentication and resource-level authorization, then test that an unauthorized request cannot receive the representation.
Evidence: RFC 9309, security considerations · OWASP authorization guidance
Read the supporting analysis
Separate access protection from provider-use policy
Can a robots-blocked URL still appear in Google Search? — link to this answer
Yes. Google may learn about a blocked URL from links even when it cannot crawl the page’s content. A noindex directive works only when Google can fetch and read it. For a public page you want excluded, distinguish crawl permission from indexing instructions; for private information, enforce access regardless of crawler cooperation.
Evidence: Google’s robots.txt limitations
Read the supporting analysis
Check what response a client actually receives
Does a Googlebot or AI-bot user-agent prove who made a request? — link to this answer
No. A client can send a familiar token. Retain the observed address and full request separately from an attributed identity, then apply the provider’s verification method and dated address evidence. A token match is useful for routing observations; it should not become permission to read private data or a verified count of that provider’s visits.
Evidence: OpenAI crawler and address documentation · Google crawler verification
Read the supporting analysis
Keep attribution uncertainty in the log review
Sources
- RFC 9309: Robots Exclusion Protocol
Verified Sep 10, 2026
- RFC 9110: HTTP Semantics
Verified Jul 19, 2026
- Google Search: introduction to robots.txt
Verified Sep 10, 2026
- OWASP Authorization Cheat Sheet
Verified Jul 19, 2026
- OpenAI crawler documentation
Verified Jul 19, 2026