Audit findings are derived records.
Keep the evidence and reasoning behind each audit claim recoverable.
Sulayman Bowles · · Updated
“Missing canonicals” is the end of a pipeline, not an observation. The audit first captured a response, extracted fields, normalized them, applied a rule, grouped URLs, assessed impact, and proposed a change. Collapsing those steps into a label makes the conclusion difficult to reproduce or revise.
Keep artifacts immutable and observations tied to their fields. Versioned rules produce evaluations; findings group them into scoped claims with confidence and gaps. Append review decisions, then keep recommendations as proposals rather than facts inside the observation.
In this article
What must I save so another person can reproduce an audit finding?
Save the original artifact, a typed observation, the rule version and inputs, the scoped finding, and the reviewer’s decision as separate linked records. Give each layer a stable identifier and timestamp. A later rule or correction should append a new interpretation rather than overwrite what was captured. Keep confidence, unavailable evidence, and recommended actions explicit so an exported finding still explains how it was reached and what it cannot establish.
Sources: W3C provenance model · OpenLineage API specification · OpenLineage facets
Read this alongside Build search systems you can verify.
Scope and assumptions
Provenance makes derivation inspectable; it does not make the observation complete or the rule correct. Findings still require scope, data-quality checks, domain review, and explicit treatment of measurement gaps.
- Record layers
- 06
- Rule identity
- VERSIONED
- History
- APPEND-ONLY
- Primary artifact
- LINEAGE CHAIN
Separate capture, interpretation, and action
An artifact is a response, DOM, screenshot, log slice, sitemap, configuration, or analytics export. An observation extracts a typed fact: status 200, no source canonical, or three crawlable inlinks. Retain its artifact pointer, extractor version, capture time, and completeness.
Evaluations record rule identity and version, inputs, parameters, and outcome. A finding can then claim that a template emits conflicting source and DOM canonicals on 184 reviewed URLs. Reviews preserve decisions without overwriting it; recommendations name a change, owner, risk, and acceptance test.
Each layer can fail differently. Authentic captures may be incomplete; correct extraction may describe an unrepresentative scenario; deterministic rules may encode the wrong requirement; sound findings may lead to impractical recommendations. Separation lets reviewers locate their disagreement.
- 01artifactBytes, hash, capture metadata
- 02observationArtifact pointer, extractor, field value
- 03evaluationRule version, inputs, outcome
- 04findingClaim version and evidence set
- 05reviewActor, time, decision, rationale
- 06recommendationProposed change and acceptance test
| Layer | Example | Immutable core | Can be superseded by |
|---|---|---|---|
| artifact | HTTP response body and headers | Bytes, hash, capture metadata | A new capture, never an overwrite |
| observation | Source canonical is absent | Artifact pointer, extractor, field value | A corrected extractor result |
| evaluation | canonical_presence@2.1 = fail | Rule version, inputs, outcome | A new rule evaluation |
| finding | Template conflict affects 184 URLs | Claim version and evidence set | A reviewed successor finding |
| review | Accepted with scope amendment | Actor, time, decision, rationale | Another append-only review event |
| recommendation | Generate source canonical from route registry | Proposed change and acceptance test | An updated implementation plan |
Identify inputs, activities, and revisions
W3C PROV distinguishes entities, activities, and agents; OpenLineage tracks runs, jobs, inputs, outputs, and event state. An audit need not adopt either whole vocabulary. It needs input identities, the generating activity, responsible software or reviewer, and time.
Identify bounded runs, artifacts by hash and storage identity, and observations by location and extractor version. Separate rule IDs from their versions. Let a finding retain its issue ID while each claim revision receives a distinct version, making amendment visible.
Propagate correlation through fetch, render, extract, evaluate, review, and export. Trace Context can connect distributed requests, but sampled, short-lived, or asynchronous traces still need durable domain IDs. Correlation locates events; provenance records derivation.
- Do not use mutable filenames as the only artifact identity.
- Separate rule identity from code deployment identity.
- Retain the exact input set for every evaluation.
- Make review actor and automation identity explicit.
- Carry run and URL identity into every export row.
- Record sampling and missing-event states when telemetry is incomplete.
Test prerequisites before the finding
Rules need required fields, eligibility, parameters, outcomes, and gap behavior. A failed parser, non-HTML response, incomplete render, or intentionally documentless route cannot automatically become “missing title.” Check prerequisites and return not-applicable or insufficient-evidence when they fail.
Version logic, thresholds, normalization, and required-input changes. Keep a readable summary and executable reference. New rules applied to old artifacts create new evaluations, leaving historical issue counts explainable.
Record evaluation time for clock-dependent checks and captures or dataset versions for external lookups. Learned classification needs model, prompt or feature versions, confidence, and review thresholds. Materially variable results require an explicit trial or sampling contract.
Run a changed extractor against the same artifact before claiming the page changed. If a parser update now discovers the canonical that was present all along, append a corrected observation and superseding evaluation. Keep the original finding’s evidence and review history intact, and record that the earlier interpretation was wrong. A new capture is unnecessary to explain that particular correction.
The composite key prevents a new rule run from overwriting an earlier interpretation.
CREATE TABLE rule_evaluation (
run_id TEXT NOT NULL,
url_key TEXT NOT NULL,
rule_id TEXT NOT NULL,
rule_version TEXT NOT NULL,
input_set_hash TEXT NOT NULL,
outcome TEXT NOT NULL CHECK (outcome IN
("pass", "fail", "gap", "not_applicable")),
evidence_json TEXT NOT NULL,
evaluated_at TEXT NOT NULL,
PRIMARY KEY (run_id, url_key, rule_id, rule_version, input_set_hash)
);Keep gaps in the denominator
Blocked fetches, timed-out renders, unsupported formats, authentication, and incomplete inventories are observations about coverage. Give them codes and evidence. Do not turn every gap into failure or quietly remove it from the denominator.
Name the uncertainty behind confidence: parser certainty, classification strength, affected-set coverage, or implementation assumptions. One unlabeled percentage cannot answer all four questions.
“184 of 210 eligible product URLs failed; 17 additional URLs had incomplete renders” gives two counts but still needs a declared universe: are the 17 part of the 210 or additional to it? State the answer. If they are additional, report 227 in scope, 210 evaluated, 184 failures, 26 passes and 17 gaps. Then 87.6% is the failure rate among evaluated URLs, not “issue confidence” or a fully observed site-wide rate. This is an illustrative reconciliation, not measured audit data.
| Gap | Example cause | Report treatment | Next action |
|---|---|---|---|
| not_captured | URL never entered the bounded run | Outside measured coverage | Review discovery and scope |
| fetch_failed | DNS, timeout, or transport failure | No response-state claim | Bounded retry or operator review |
| render_partial | Completion conditions not met | Present fields only; absence uncertain | Inspect dependencies and rerun |
| unsupported | Parser lacks format or feature | Tooling gap | Add support or exclude explicitly |
| ambiguous | Rule inputs conflict | Review queue, not automatic fail | Inspect artifact and product intent |
Preserve review history through export
Append acceptance, rejection, evidence requests, duplicate decisions, risk changes, scope amendments, and revised recommendations to a finding version. Show reviewers the claim, affected set, examples, gaps, rule, and artifact links. A final label alone conceals the pipeline under review.
Keep stable IDs in exports. CSV can carry finding and rule versions, URL, outcome, confidence class, gap, evidence reference, review state, owner, and acceptance test; JSON can retain nested lineage. Link readable executive summaries to those underlying records.
A reproducible export needs the definition of its affected set, not only example URLs. Retain the query or explicit member IDs and the run/rule/input versions. If a reviewer narrows a finding from every product to one template, append that membership revision. Otherwise an unchanged headline count can hide a completely different population.
After repair, create a fresh run and evaluations linked to the accepted finding. Require implementation evidence and the affected rule passing over the agreed scope, with new gaps disclosed. Ticket closure or merged code does not establish that the condition changed.
- Never overwrite the evidence set after a finding enters review.
- Store reviewer amendments as new claim versions.
- Keep URL-level rows behind every aggregate.
- Export gap states and denominators with pass/fail counts.
- Link the implementation change and the validating rerun separately.
- Mark a finding resolved only against its acceptance contract.
Require evidence of the changed condition
A merged repair is implementation evidence. Resolution needs a new evaluation across the agreed scope, connected to the accepted finding and disclosing new gaps. Keep the earlier evidence and interpretation available for comparison.
Sources
- W3C PROV-O provenance ontology
Verified Jul 19, 2026
- OpenLineage API specification
Verified Jul 19, 2026
- W3C Trace Context Recommendation
Verified Jul 19, 2026
- OpenTelemetry signals documentation
Verified Jul 19, 2026
- OpenLineage facets and extensibility
Verified Jul 19, 2026