Indexed, though blocked by robots.txt

Decide what the URL should do

Start with the page owner’s intended outcome. A search-worthy product accidentally swept into a directory block needs a different response from a public confirmation page that should stay out of Search. Avoid applying a removal recipe to every URL in the warning.

For a private invoice, protect the representation with authentication and authorization first. Making it publicly readable so a crawler can see noindex would create the wrong result. The general robots guide separates permission checks from search controls.

The same noindex needs a reachable response

In the first state below, /receipts/thanks is disallowed. Adding noindex to its template does not help Google if Google cannot fetch that template. In the repaired public-page state, the fetch is permitted and the returned HTML carries the exclusion.

The fixture uses an exact-path exception so other /receipts/ URLs stay blocked. The exception does not cover query variants: /receipts/thanks?order=123 remains a different match. List the real variants, decide their responses, and test them before copying the rule. Never put customer identifiers in a public teaching fixture.

GET /robots.txt → 200
User-agent: *
Disallow: /receipts/

GET /receipts/thanks → 200
<head><meta name="robots" content="noindex"></head>

GET /robots.txt → 200
User-agent: *
Disallow: /receipts/
Allow: /receipts/thanks$

GET /receipts/thanks → 200
<head><meta name="robots" content="noindex"></head>

Verify delivery, then verify Google’s later state

First capture the deployed /robots.txt response for the exact scheme, host and port of the affected URL. Record the final GET status, redirect chain and body. Evaluate the selected crawler group against the actual path and query; a change to the wildcard group may leave a specific Googlebot group untouched.

Next fetch the page. Inspect its X-Robots-Tag header and HTML head, then the rendered document if JavaScript changes the head. For a PDF, use the HTTP header instead of an HTML tag. Google’s noindex documentation requires crawl access for either method.

Run URL Inspection’s live test on the same URL and inspect the returned content. Then use indexed inspection to compare the last crawl and indexing state. The URL Inspection documentation distinguishes current delivery from Google’s stored observation: a live test does not establish removal or successful indexing.

If fresh public delivery is correct but Google’s last crawl predates it, keep the response stable and request a recrawl when appropriate. Record that request separately from its eventual effect. A cached robots policy can also postpone the new fetch behavior.

Keep noindex available for future crawls

Do not reapply Disallow immediately after seeing the page disappear. If the public URL must stay excluded, future crawls need to keep encountering noindex. An actually deleted document can instead return an appropriate not-found or gone response; an urgent search-result removal has a separate temporary removal workflow.

If the real objective is to consolidate equivalent pages, use the canonical and noindex decision before choosing exclusion. A canonical points to a representative; noindex removes this URL. Those are different publishing intentions.

Finish with a small before-and-after record: intended outcome, exact policy match, returned directive, deployment date, and Google’s later crawl/index state. That makes the repair testable without promising a recrawl date.

The robots.txt control-layer guide retains the group-selection example that explains why changing one policy group may leave another untouched.