Atlas

The crawl and its retained record

Atlas normalizes URLs, deduplicates requests, handles robots and sitemaps, and records a bounded crawl. A failed fetch stays distinguishable from a successfully fetched page with no extractable content. Run limits remain part of the context when reviewing coverage.

The Python console combines raw HTML extraction, optional browser rendering, link-graph calculations, and provider collection. Reviewers can filter local runs, compare them, inspect findings, and export reports. The shipped local workflow does not include distributed crawling, hosted delivery, rank tracking, or GA4 integration.

Reusing content without freezing its interpretation

Incremental crawling separates unchanged page bytes from unchanged extracted facts. A 304 can reuse valid cached artifacts; an unchanged 200 can skip parsing when the matching artifact is still valid.

The artifact identity includes the content hash, artifact type, and version token. Changing extractor, schema, or scoring versions forces parsing even if the page body has not changed. Otherwise a faster crawl could silently preserve an old interpretation.

Source facts and effective page facts

The retained July 16 sample compares two successful responses. The static page contains ten quote cards; the JavaScript page contains no source quote cards but embeds ten runtime records. Both retain pagination addresses and lack a source canonical. That evidence calls for a render check before judging visible content.

A separate implementation test exercises an empty raw H1 and an incorrect raw canonical. Rendering supplies one effective H1 and a corrected canonical. Atlas keeps the raw-only findings while avoiding missing-H1, thin-content, and canonical-mismatch findings against the effective page.

Recording collection gaps beside the findings

A source observation and a severity assignment are different records. The sample’s absent canonical is retained without an assigned defect or severity. Effective links feed the graph; missing depth remains unknown.

Optional measurement providers have explicit availability states, including skipped collection when a key is absent. A provider gap must not become a bad website score. The report needs the page state, rule version, and collection limits beside the recommendation so a reviewer can see which evidence was actually available.

Code, captures, and the full study

The repository documents the local console and its current implementation. The July source captures provide a small reproducible reading example; they are not client traffic, a fresh browser-render measurement, or evidence of search gains.