Crawler evidence
Do Austin home-service websites allow AI crawlers?
Most tested homepage paths were allowed by the saved robots rules. Policy and a delivered page still produced different results.
For 61 of the 67 brands, the saved robots rules allowed OAI-SearchBot to request the tested homepage path. One disallowed it and five remained unknown. But only 48 brands returned a complete HTTP 200 homepage body without challenge indicators to this research client. An allowed rule and a delivered document are different observations.
Thirteen of the 61 OAI-SearchBot-allowed records did not also have that complete 200 response. That is the useful result of the check: inspecting the policy file alone would have missed a delivery question. It is not evidence that OpenAI itself was blocked, because the requests came from this research client’s network.
Method & limits
Observed October 21, 2026, 20:08:13–20:08:53 UTC. One analytic record per brand in the 67-brand July ownership panel; 66 official URLs on 65 requested origins, one unresolved URL. Requests came from an ordinary research client, with no real provider-IP, JavaScript-rendering or AI-citation test. Ownership remains dated July 26. Unknown and incomplete observations are retained.
What the six policy checks found
The table evaluates the saved robots rules for each brand’s final homepage path, or its requested path when no redirect was observed. “Allowed” means no matching restriction in a usable policy under the documented parser. It does not mean that a provider visited, indexed or cited the site.
The training-oriented tokens and search-oriented tokens can receive different rules. The crawler policy guide explains those product distinctions. The exact URL matters: this is a path check, not a certification of every page on a domain.
Redirects change the authority being evaluated. A-Tex’s saved homepage redirect ended at a Canva design editor/share URL; the OAI-SearchBot and Googlebot disallow in that row belongs to Canva, not to atexpm.com. Its requested-origin robots response was 404 and remains unknown in this study. The table therefore describes final targets, and must not be read as a list of brands intentionally blocking crawlers.
| Product token | Allowed | Disallowed | Unknown |
|---|---|---|---|
| GPTBot | 60 | 2 | 5 |
| OAI-SearchBot | 61 | 1 | 5 |
| ClaudeBot | 60 | 2 | 5 |
| Claude-SearchBot | 61 | 1 | 5 |
| PerplexityBot | 62 | 0 | 5 |
| Googlebot | 61 | 1 | 5 |
A successful policy check can precede an incomplete fetch
The ordinary homepage requests produced 51 HTTP 200 statuses, ten 403s, three 404s and three records without a final status. Only 48 of the 51 HTTP 200 bodies were complete under the collection cap. Across all statuses, nine records carried challenge indicators. These categories describe this run and do not establish the response a provider would receive.
Three records without a final status include the unresolved official URL and transport gaps. Their absence is not a negative judgment about the businesses. The downloadable observations preserve status, completeness, error, redirect and challenge fields separately.
A site owner who sees an allowed policy and a denied or incomplete research response has a next diagnostic step: inspect the matching security or delivery event. Changing the robots file again may address the wrong layer. The access-checking guide separates the checks.
The cohort comes from a dated ownership census
The sampling frame is the July 26 Austin home-service census. Its complete intake has 75 records; 67 are published_active. Historical, excluded and unpublished-candidate rows stay outside this study’s brand denominator.
Existing source evidence grounded official URLs for 66 brands. Two brands shared a requested origin, leaving 65 requested origins; some records use national or platform-hosted sites. One brand had no supported official URL and remains in the table as unresolved. Replacing it with a search guess would make the roster look more complete without making the observation more trustworthy.
The preserved owner classifications are historical. They are available for descriptive grouping, but this run does not reverify current ownership or show that an owner caused a policy decision. Shared franchise infrastructure also prevents treating every brand row as an independent server system.
A bounded check with saved policies
The collector used four workers, a minimum 1.5-second spacing between requests to an origin, at most five redirects, finite connection/request timeouts and one transient retry per hop. The run made 161 request attempts including redirects. Homepage bodies were capped at 1 MiB; robots bodies at 512 KiB. No bot-labelled simulation was used.
For a usable saved policy, the evaluator matches named product tokens without case sensitivity, merges repeated matching groups, falls back to the wildcard group and applies path rules with allow winning a tie. The code, parser tests, roster and saved parseable policy bodies are in the research bundle. The implementation is an inspectable protocol subset, not a provider-conformance certificate.
Unreadable, incomplete or failed policy observations remain unknown in this dataset. That is a conservative research classification rather than a claim about every crawler’s error fallback. The method note states the rule and distinguishes the 468 requested/final-target policy rows from the 67 final analytic brand rows.
Use the result to choose the next check
For one of your own public pages, save the policy, the response and the useful content separately. If policy disallows the page, resolve that choice first. If policy allows it but the edge returns a challenge, investigate that response. If the document is delivered, a real provider visit or answer citation still needs its own observation.
The study does not rank contractors, estimate market share, assess licensing or attribute intent. It also does not show that llms.txt changes visibility. What it supplies is a dated, reproducible set of observations that another edition can repeat against the same brand identifiers.
Use the ownership dataset guide to keep brand, platform and historical owner fields distinct when joining a later measurement. A changed response is worth investigating; an unchanged owner label does not prove the owner was freshly checked.
Sources
- Public homepage observations — October 21, 2026
Verified Oct 21, 2026
- Study bundle: roster, policies, parser, tests and configuration
Verified Oct 21, 2026
- Study method and classification boundaries
Verified Oct 21, 2026
- Original Austin ownership relational dataset — July 26, 2026
Verified Jul 26, 2026
- RFC 9309 — Robots Exclusion Protocol
Verified Oct 21, 2026
- OpenAI — crawler product purposes and policy choices
Verified Oct 21, 2026
Supporting material
- 67-brand observations
One analytic row per brand with policy, status, completeness and unknown fields.
- Reproducible study bundle
Roster, policy rows, saved parseable robots policies, collector, parser and tests.
- Method note
Denominators, collection settings and exact limits on the claims.