Replayable Traces for Evaluating Tool-Using AI Agents

Direct answer: AI agent evaluation traces

AI agent evaluation traces should preserve the task, messages, tool calls, tool results, state transitions, artifacts, policy versions, costs, timing, and grader evidence so a failure can be replayed without pretending a frozen test world predicts the live environment.

Original research artifact

A typed trace-event schema, replay-world manifest, evaluator separation matrix, and promotion evidence checklist.

What this page adds

Join trace completeness, replay limits, independent grading, and promotion evidence in one inspectable evaluation contract.

Related research

• AI agent evaluation harness
• can AI run a business
• SEO audit provenance

Separate the task, trial, and observable trace

A task is the reusable test case: instructions, fixtures, initial environment, permitted actions, success criteria, side-effect limits, budgets, and stop conditions. A trial is one attempt by a specific agent configuration. The trace is the ordered record of observable events produced during that trial. Anthropic uses similar distinctions between task, trial, grader, and transcript in its agent-evaluation guidance.

Give each layer an identifier. A task version changes when fixtures or success criteria change. A trial ID identifies one stochastic attempt with model, prompt, tool, and policy versions. Trace events carry sequence, parent event, time, actor, input reference, output reference, status, and state-delta reference. Correlation IDs can follow calls across services, while trial IDs preserve the evaluation domain.

Capture observable inputs and outputs; do not make hidden chain-of-thought a dependency. Useful evidence includes instructions delivered to the agent, model responses available to the harness, tool schemas, tool calls, tool results, file or database diffs, browser states, policy decisions, errors, tokens, latency, and cost. Sensitive data should be redacted or represented by secure references before traces enter general analytics.

Control the environment or declare what cannot be replayed

A trial depends on more than the agent. APIs change, web pages update, clocks advance, search results move, credentials differ, and shared state leaks between attempts. Isolate each trial in a resettable workspace or account when possible. Seed databases and files from versioned fixtures, freeze or record time inputs, and provide fake external services for deterministic contract tests.

For live dependencies, record the request and response boundary subject to privacy and licensing constraints. A replay can then return the captured response to the same tool call or expose a controlled divergence. This tests harness behavior and downstream decisions, not whether the model will reproduce the same call. Mark live, recorded, simulated, and unavailable dependencies in the trial manifest.

State cleanup is part of correctness. A failed agent may leave a draft, reservation, branch, browser session, or partial database mutation. The harness should snapshot before and after, attribute changes to events, and restore or discard the environment. If cleanup cannot be guaranteed, run in a disposable boundary and treat escape as a critical failure.

• Create one isolated state namespace per trial.
• Pin tool schemas and fixtures with content hashes.
• Classify every external dependency as live, recorded, simulated, or blocked.
• Capture state before, after, and at consequential actions.
• Verify cleanup independently of the agent final message.
• Fail the trial if it acts outside the permitted boundary.

Grade outcomes, process, policy, and side effects separately

A final-answer grader cannot establish that an external action succeeded. Begin with deterministic state graders where the domain permits them: tests pass, expected rows exist, files match a schema, the correct calendar event was created, or no unauthorized message was sent. Add process graders for required or prohibited actions, policy graders for permission boundaries, and resource graders for cost and latency.

Model-based graders are useful when success is semantic or several valid paths exist, but their output is another derived record. Store the grader model, prompt or rubric, evidence supplied, output, and confidence. Calibrate against expert-labeled trajectories and inspect disagreement. AgentRewardBench reports that no single evaluated language-model judge excelled across all included web-agent benchmarks, which is a reason to combine graders rather than appoint one universal judge.

Allow partial and invalid states. A trial can complete the main task but cause an unacceptable side effect; it can fail the outcome because the environment broke; or it can be ungradeable because the trace is incomplete. One scalar score hides those distinctions. Use a result object with outcome, policy, side effects, efficiency, trace integrity, and environment validity.

Report distributions across repeated trials

Model outputs and agent paths vary. One passing demonstration is evidence that success is possible, not an estimate of reliability. Run repeated trials for representative tasks, keep the fixture and configuration fixed, and report pass counts with denominators. Stratify by task family and risk so a large set of easy cases does not hide a failure on the consequential path.

Pair baseline and candidate trials when practical by using the same task versions and environment seeds. Report changes in outcome, policy violations, side effects, cost, and latency. Confidence intervals or resampling can communicate uncertainty, but no statistical method repairs a benchmark whose tasks are ambiguous, contaminated, or graded incorrectly.

Inspect traces behind aggregate movement. A candidate can improve a score by exploiting a grader, skipping an expensive safety check, repeating until a flaky state passes, or shifting failure into an unmeasured side effect. Promotion review should sample wins, losses, disagreements, and unusually cheap or expensive trials.

• Predefine the primary metric and non-negotiable safety metrics.
• Use the same task versions for baseline and candidate comparison.
• Report numerator, denominator, exclusions, and invalid trials.
• Separate task-level variance from repeated attempts on one task.
• Review representative traces from every changed outcome class.
• Keep benchmark edits out of the comparison window unless rebaselined.

Promote the system, not a leaderboard number

An agent configuration includes model, prompts, tool contracts, retrieval, memory, policies, budgets, and harness code. Promote that versioned bundle. A model upgrade can regress because a tool description no longer elicits the intended call; a prompt improvement can increase cost; a stricter policy can reduce nominal success while preventing unsafe completion. The evaluation must preserve those tradeoffs.

Use a staged gate: deterministic contract tests, replay against recorded dependencies, repeated trials in isolated environments, adversarial or policy cases, and a bounded canary where external effects are reversible and observable. Define rollback conditions before the canary. A high average score should never override a critical authorization or destructive-action failure.

The final evidence package should include task manifest, configuration manifest, trial ledger, grade records, aggregate report, selected traces, known gaps, and promotion decision. This makes later debugging possible when production behavior differs. It also keeps the claim narrow: the candidate met the documented gate on the tested distribution at the recorded cutoff.

{
  "trial_id": "trial-0184",
  "task": { "id": "publish-draft", "version": "3" },
  "configuration": { "id": "agent-candidate", "version": "7" },
  "environment": { "fixture": "workspace@sha256:...", "network": "recorded" },
  "trace": { "id": "trace-0184", "integrity": "complete" },
  "graders": ["outcome@2", "policy@5", "side-effects@3"],
  "result": { "outcome": "pass", "policy": "pass", "side_effects": "pass" },
  "cost_usd": 0.42,
  "duration_ms": 18420
}