Robots.txt wildcards and path matching

Read a pattern as a path condition

A rule is not a shell glob over filenames or a regular expression over a full URL. It applies to the path and query of a URL on the service that supplied the policy. A fragment is not sent in the HTTP request. Choose the User-agent group before comparing paths.

A trailing slash is consequential: /drafts/ names a directory prefix, while /drafts also matches /draftsmanship. A final $ makes the endpoint exact. In /*.pdf$, the wildcard can include slashes, but the suffix must end the path-and-query string; report.pdf?download=1 does not end in .pdf.

RFC 9309’s special characters define * and $. Google’s path examples show their use with case and query strings.

Run all 18 decisions locally

Download the executable fixture bundle, extract it, and run node run.mjs with Node 22 or later. It reads only its adjacent robots.txt and cases.csv, prints each expected and actual decision, and exits unsuccessfully if a case differs. It makes no network requests and installs no packages.

The policy, expected results and runner are also available separately. Run from any directory: inputs resolve relative to the script, not your current working directory.

This is a readable reference for the published cases, not a general-purpose robots tester or a certified full REP implementation. It models exact product-token groups, combined equal groups, fallback *, path matching and normalization. It does not implement HTTP retrieval, robots caching or partial user-agent matching. Use your production parser against the same cases, including the retained October 23 policy, before relying on it.

unzip robots-wildcard-fixtures.zip -d robots-wildcard-fixtures
cd robots-wildcard-fixtures
node run.mjs

Two boundaries a substring check loses

The fixture treats /cafe and /%63afe as equivalent because c is an unreserved character. It keeps an encoded slash distinct from a path separator: /folder%2Fsecret does not become /folder/secret. Decode only the octets the protocol says to decode; decoding every escape can invent a directory boundary.

Group selection is independent of that comparison. FixtureBot has two exact named groups whose rules are combined; it does not inherit the fallback group. A result permitted for FixtureBot can therefore be blocked for ExampleBot. See the original two-group worked example rather than copying that whole table here.

The runner is intentionally inspectable. Normalization, selection, matching and precedence are separate functions. That makes a disagreement with an established parser easier to locate than a single “blocked” badge would.

Test the generator and the crawl frontier separately

Add the exact deployed file to your release check, select the real crawler product token and evaluate a handful of representative paths. Include one intended block, one exception, a query variant, a capitalized path and a prefix neighbor. Preserve the returned policy when a release changes its effective result.

A crawler must also decide when that decision expires and what to do after a failed robots fetch. Keep the match result separate from those scheduling states in a crawl-frontier state machine. A correct string match against yesterday’s file can still produce the wrong request today.

For production parser work, Google’s open-source robots parser supplies a separate implementation and test corpus. Passing this small fixture establishes agreement on these inputs, not universal provider compatibility.

A URL that is indexed while blocked by robots.txt needs an intended indexing outcome as well as a correct path match.