# Austin AI crawler policy check — October 21, 2026

The unit is one of 67 published brands in the July 26, 2026 ownership census. The roster includes 66 grounded official URLs on 65 requested origins and one unresolved URL. URLs were taken from the retained source ledger; an origin homepage is used where the supporting official page is an About/policy page, with the derivation recorded. Shared national/franchise sites are not independent local server systems.

One low-rate ordinary research client collected homepages and robots.txt, following at most five redirects, using four workers and at least 1.5 seconds between requests to an origin. The run made 161 request attempts including redirects. Homepage bodies were capped at 1 MiB and robots at 512 KiB. The observation window and exact configuration are in summary.json. No simulated provider-UA probes, real provider-IP tests, JavaScript renders, licensing audit, current ownership refresh or AI answer/citation experiment was performed.

Policy decisions apply to the final homepage path, or the requested path when no redirect was observed. bot-robots-decisions.csv also retains intermediate/requested-target decisions, so its 468 rows are not 468 brands. Group only through brand_id and target_stage. homepage-observations.csv has exactly one final analytic row per brand and six bot-policy fields.

Unknown official URLs, failed/incomplete HTTP observations and unusable policy bodies remain unknown. A readable policy with no applicable restriction yields allowed for the tested path. A 4xx or otherwise unreadable robots response is conservatively unknown in this study even where a crawler protocol could define a fallback. The parser implements the documented subset: exact case-insensitive named product tokens, merged duplicate groups, wildcard fallback, longest path rule, allow on ties, percent/unicode normalization and path wildcards/end anchors. It is not a provider-conformance certification.

48 brands yielded complete HTTP 200 homepage bodies without challenge indicators. 61 had an OAI-SearchBot-allowed rule result; 13 of those 61 did not also yield such a complete 200 homepage response to this research client. That comparison separates policy from delivery; it does not measure what OpenAI's crawler received. The cohort is purposeful, historical and nonrepresentative. Do not infer market share, service quality, company intent or a causal ownership effect.

Ownership classifications retain July 26, 2026. An October fetch grouped by a historical owner is not an October ownership finding. Raw homepage bodies and HTTP headers remain in the private research receipt; saved public parseable robots policies and digests support reproducing the policy calculation.

## Redirect attribution and requested-authority comparison

A-Tex's retained URL redirected to a Canva design editor/share URL. The final-target OAI-SearchBot, Claude-SearchBot and Googlebot disallow is Canva's rule, not a policy attributed to atexpm.com. The requested-authority policy observation there was HTTP 404, classified unknown. Requested-authority totals and final-target totals are separate in policy-views.csv. At requested authorities, GPTBot and ClaudeBot each have 58 allowed / 1 disallowed / 8 unknown; each of the other four tokens has 59 / 0 / 8. Do not describe the final-target counts as brands deliberately blocking a provider.

## Reproduce or repeat

Requires Python 3.9+ and curl 7.70+. Extract into a new directory and keep roster67.json, collect_access.py and robots_rules.py together. Run:

```sh
python3 -m unittest -v test_robots_rules.py test_collect_access.py
python3 collect_access.py --collect --roster ./roster67.json --workers 4 --spacing 1.5 --origin-cap 30
```

The second command makes new network requests into a new run directory. Results can change. The current collector avoids retrying body-cap errors; the baseline made one such retry for each of three capped pages, and all 161 original attempts remain retained privately. The exact original code is under executed/. Its digest matches summary.json; copy the executed collector/parser to a separate new directory under their ordinary names with roster67.json to rerun that version. The current code has a distinct digest in manifest.json. Neither rerun recreates historical network responses. The saved robots bodies and analytic CSVs support offline inspection of this edition.