Technical SEO
Crawler directives: choose the right control
A fetch rule, an indexing instruction and a permission check operate at different points. Choose the point that owns the result.
Use robots.txt to guide cooperative fetching, robots meta or X-Robots-Tag to control supported indexing and presentation after a fetch, and server-side authorization to deny private access. Use a canonical to nominate an equivalent representative. A crawler directive cannot substitute for every other control.
The decision table below starts with the required result, rather than a tag you happen to be able to edit. The response chooser then shows one concrete implementation. All paths are teaching examples on example.com.
Method & limits
Provider documentation checked October 26, 2026. The response designs and local-host demonstration are authored examples, not measured behavior by a search engine or AI provider.
Which layer owns the result?
A public thank-you page, a private invoice and a duplicate campaign URL can all be described loosely as “do not index this.” Their correct delivery is different. The invoice should never become public; the thank-you page can expose noindex; the campaign URL may belong to the same canonical family as a clean guide.
| Desired outcome | Control and location | Keep fetch available? | What to verify |
|---|---|---|---|
| Reduce crawling of low-value paths | Disallow in /robots.txt | The blocked paths are intentionally not fetched. | Selected group and representative path/query matches. |
| Exclude public HTML from Search | noindex in its head or X-Robots-Tag header | Yes: the indexer must read it. | Returned directive and later indexed state. |
| Exclude a public PDF | X-Robots-Tag: noindex on the PDF response | Yes: an HTML tag cannot be embedded in its head. | Actual PDF GET headers across CDN and origin. |
| Limit search snippets | Supported nosnippet or preview controls in meta/header | Yes. | Correct resource scope; this is not a crawl block. |
| Consolidate equivalent URLs | rel=canonical or an appropriate permanent redirect | Keep the intended representative accessible. | Equivalent content and aligned links/sitemap. |
| Deny private access | Authentication plus resource authorization and safe caching | Only authorized principals receive the content. | Anonymous and wrong-user body checks. |
| Protect origin capacity | Rate/concurrency limits and caching; provider pacing where supported | Apply a deliberate service policy. | Request rates, statuses and expensive origin work. |
Choose an outcome; inspect the response
Choose a case to see its policy and response together. These are independent alternatives, not a single configuration to combine. The public-HTML case deliberately leaves crawl access open so noindex is visible.
For Google, robots meta and X-Robots-Tag can deliver the same supported indexing rules. The robots meta specification defines their scope, user-agent targeting and conflict behavior. If one delivered rule permits indexing but another says noindex, adding more permissive markup does not cancel the restrictive rule.
Allow the fetch; return noindex.
GET /robots.txt → 200
User-agent: *
Disallow:
GET /thanks → 200
<head><meta name="robots" content="noindex"></head>Verify the directive in the response, then Google’s later indexed state.
Read the response the directive belongs to
A robots file governs its serving scheme, host and port. A meta element belongs in the HTML head of the page it controls. An X-Robots-Tag travels with the resource’s HTTP response, which makes it useful for PDFs and other non-HTML content. None of these locations automatically governs every hostname or alternate download route.
Capture the final GET response rather than relying on a CMS setting or a HEAD request alone. Compare response headers and initial HTML; then inspect the rendered head if JavaScript changes it. The raw HTML versus rendered DOM guide keeps those observations separate.
For link following, distinguish a page-level nofollow hint from a robots fetch restriction. For AI collection and search, identify the provider and purpose before choosing a token. The separate AI crawler policy hub owns that purpose-specific decision; a training opt-out is not a universal Search exclusion.
Combine controls only when the sequence works
A crawl block plus noindex is a broken sequence when the block prevents the indexer reading noindex. The indexed-though-blocked repair shows the corrected public response. Canonical plus noindex needs a separate consolidation-versus-exclusion decision.
For capacity, publishing Crawl-delay is a request to supporting clients; it does not impose a server limit. Use the provider-support comparison before deciding whether it helps your actual traffic.
After a change, verify the property the control owns. A correct returned header establishes delivery. A later Search Console observation establishes what Google processed. An anonymous denial establishes a permission boundary only for the tested route and cache state. Keep those checks distinct so one passing check does not stand in for the intended result.
Start with the general robots.txt matching examples when the outcome calls for a cooperative fetch rule.
Sources
- Google: robots.txt introduction
Verified Oct 26, 2026
Crawl policy, indexing limitations and protection alternatives.
- Google: robots meta and X-Robots-Tag
Verified Oct 26, 2026
Directive delivery, resource scope and restrictive-rule combinations.
- Google: canonical URL methods
Verified Oct 26, 2026
Representative selection for equivalent content.
- RFC 9309: security considerations
Verified Oct 26, 2026
Robots exclusion is not access authorization.