Audit Findings Should Be Derived Records

Direct answer: SEO audit provenance

SEO audit provenance means every finding can be regenerated from immutable observations, rule and policy versions, affected URLs, timestamps, confidence, and reviewer decisions while unknown or incomplete evidence remains a named measurement gap instead of becoming a claim.

Original research artifact

A finding-provenance schema, observation-to-finding lineage table, confidence states, and review transition contract.

What this page adds

Define a derivation and review contract for findings instead of presenting severity labels without their source records.

Related research

• SEO data pipeline
• AI agent evaluation traces
• SQLite web crawler

Separate the six records an audit usually collapses

The artifact is the captured object: response headers and body, rendered DOM, screenshot, log slice, sitemap, configuration, or analytics export. An observation is a typed fact extracted from an artifact, such as status 200, canonical absent in source, or three crawlable inlinks. The observation should retain the artifact pointer, extractor version, capture time, and completeness state.

A rule evaluation records the rule ID and version, input observations, outcome, and parameters. A finding turns one or more evaluations into a scoped claim such as “the product template emits conflicting source and rendered canonicals on 184 reviewed URLs.” Review state records human or automated decisions without overwriting the claim. A recommendation proposes a change, owner, risk, and acceptance test.

These layers have different truth conditions. An artifact can be authentic while incomplete. An observation can correctly describe the artifact while the capture scenario is unrepresentative. A rule can run deterministically and still encode the wrong requirement. A recommendation can be impractical even when the finding is sound. Keeping the layers visible makes disagreement productive.

Give every run, artifact, rule, and claim a stable identifier

W3C PROV distinguishes entities, activities, and agents; OpenLineage similarly represents runs, jobs, inputs, outputs, and event state. A technical audit does not need to reproduce either vocabulary completely to benefit from the structure. The core requirement is that a derived record names the input entities, generating activity, responsible software or reviewer, and time.

Use run IDs for bounded executions, artifact IDs derived from content hashes plus storage identity, observation IDs tied to artifact location and extractor version, and rule IDs separated from rule versions. A finding ID should represent the continuing issue identity while each revision receives a distinct version. This supports comparison without pretending that amended claims are the same record.

Propagate a correlation identifier through fetch, render, extract, evaluate, review, and export events. W3C Trace Context offers a standard mechanism for distributed request correlation, but audit lineage also needs durable domain IDs because a trace may be sampled, short-lived, or split across asynchronous work. Correlation helps locate events; provenance defines what was derived from what.

• Do not use mutable filenames as the only artifact identity.
• Separate rule identity from code deployment identity.
• Retain the exact input set for every evaluation.
• Make review actor and automation identity explicit.
• Carry run and URL identity into every export row.
• Record sampling and missing-event states when telemetry is incomplete.

Make rule evaluation deterministic and versioned

A rule should declare required input fields, eligibility conditions, parameters, outcomes, and gap behavior. “Missing title” is incomplete if the parser failed, the response was not HTML, the render timed out, or the route intentionally returns no document. Evaluate prerequisites first and return not-applicable or insufficient-evidence when they are not satisfied.

Version a rule whenever its logic, threshold, normalization, or required inputs change. Store a human-readable summary and a machine-executable implementation reference. Re-running old artifacts through a new rule should create new evaluations rather than mutating historical results. This makes changes in issue counts explainable.

Determinism requires controlled inputs. Time-dependent checks need a recorded evaluation time. External lookups need captured responses or versioned datasets. Machine-learned classification needs model and prompt or feature versions, confidence, and a review threshold. If the same inputs can produce materially different results, the evaluation must store the trial or sampling contract.

CREATE TABLE rule_evaluation (
  run_id TEXT NOT NULL,
  url_key TEXT NOT NULL,
  rule_id TEXT NOT NULL,
  rule_version TEXT NOT NULL,
  input_set_hash TEXT NOT NULL,
  outcome TEXT NOT NULL CHECK (outcome IN
    ("pass", "fail", "gap", "not_applicable")),
  evidence_json TEXT NOT NULL,
  evaluated_at TEXT NOT NULL,
  PRIMARY KEY (run_id, url_key, rule_id, rule_version, input_set_hash)
);

Represent gaps and confidence without turning them into scores

A missing observation is data. The fetch may have been blocked, the render may have timed out, the parser may not support the content type, the route may require authentication, or the source inventory may be incomplete. Give those states codes and evidence. Do not convert every gap into a failing finding or exclude it from the denominator without disclosure.

Confidence should explain which uncertainty it summarizes. Observation confidence can describe parser certainty; rule confidence can describe classification strength; coverage confidence can describe whether the affected set is complete; recommendation confidence can describe implementation assumptions. One unlabeled percentage collapses those distinct questions.

Use counts with denominators and eligibility definitions. “184 of 210 eligible product URLs failed; 17 additional URLs had incomplete renders” is more useful than “87.6% issue confidence.” The first statement lets a reviewer locate both evidence and gaps. The second sounds precise while hiding what was measured.

Make review and export lossless

Review should append a decision to a finding version: accepted, rejected, needs evidence, duplicate, risk reclassified, scope amended, or recommendation changed. The reviewer needs the claim, affected set, representative examples, gap counts, rule definition, and direct artifact links. A dashboard that shows only the final label forces trust in the pipeline it is supposed to expose.

Exports should preserve stable IDs, not flatten everything into prose. A CSV can carry finding ID, version, rule, affected URL, outcome, confidence class, gap state, artifact reference, review state, owner, and acceptance test. JSON can preserve nested lineage. An executive summary can remain readable while linking back to those records.

A post-fix rerun should generate a new run and new evaluations, then relate them to the accepted finding. Resolution requires both implementation evidence and the affected rule passing across the agreed scope, with new gaps disclosed. Closing a ticket because code merged is a workflow event, not proof that the observed condition changed.

• Never overwrite the evidence set after a finding enters review.
• Store reviewer amendments as new claim versions.
• Keep URL-level rows behind every aggregate.
• Export gap states and denominators with pass/fail counts.
• Link the implementation change and the validating rerun separately.
• Mark a finding resolved only against its acceptance contract.