Replayable traces for AI agents.
Evaluate the execution and its effects, not just the final answer.
Sulayman Bowles · · Updated
An agent can give the right answer after an unsafe action, fail after a useful partial change, or pass because its grader believed the final message instead of inspecting the world. Tool calls, observations, mutations, retries, costs, policy decisions, and final state are part of the result.
Define an initial state and success contract, run isolated trials, and capture observable messages, tool inputs and outputs, state changes, timing, and errors. Combine versioned deterministic and model-based graders. Promote improvements only when repeated outcomes satisfy safety, cost, and regression bounds.
In this article
How do I tell whether an agent improvement is reliable rather than a lucky run?
Define the task, initial environment, and success conditions before running the agent, then retain the actions and resulting state as a replayable trace. Grade the final outcome separately from the path: a correct answer can conceal an unacceptable action. Repeat comparable trials, record cost and failure patterns, and test regressions before promoting a change. One successful transcript cannot establish reliability, and a model-based grader needs its own validation against the behavior you intend to measure.
Sources: Anthropic agent-evaluation methods · AgentRewardBench research · Anthropic statistical evaluation methods
Read this alongside Make AI results inspectable.
Scope and assumptions
A replay can reproduce recorded inputs or simulate an environment boundary; it cannot prove that a stochastic model will emit the same trajectory. Evaluation results apply to the tested agent, tools, models, policies, graders, and environment versions.
- Unit
- TRIAL
- Evidence
- OBSERVABLE TRACE
- Graders
- LAYERED
- Promotion
- CONTROLLED
Define the task, attempt, and evidence
The task specifies instructions, fixtures, initial environment, permitted actions, success, side-effect limits, budgets, and stopping conditions. A trial is one attempt by a particular configuration; its trace orders the observable events. Anthropic’s guidance uses comparable task, trial, grader, and transcript distinctions.
Version tasks when fixtures or criteria change. Identify each stochastic trial with model, prompt, tool, and policy versions. Events retain sequence, parent, time, actor, input and output references, status, and state-delta reference. Correlation IDs connect service calls; trial IDs identify the evaluation attempt.
Use observable evidence rather than hidden chain-of-thought: delivered instructions, harness-visible responses, tool schemas, calls and results, file or database diffs, browser states, policy decisions, errors, tokens, latency, and cost. Redact sensitive data or use secure references before general analytics.
| Record | Stable inputs | Variable outputs | Version boundary |
|---|---|---|---|
| task | Fixture, instruction, allowed actions, graders | None | Change when the test contract changes |
| configuration | Model, prompts, tools, policies, budgets | None | Change for any execution dependency |
| trial | Task and configuration IDs, random seed where applicable | Outcome, cost, duration, terminal state | One attempt only |
| trace event | Sequence and parent context | Message, tool result, state delta, error | Immutable event |
| grade | Grader and rubric version | Score, labels, evidence references | One grader run on one artifact set |
Reset state and label live dependencies
APIs, pages, time, search results, credentials, and shared state can change between trials. Where possible, isolate resettable workspaces or accounts; seed versioned databases and files; freeze or record time; and simulate services for deterministic contract tests.
For live dependencies, capture request and response boundaries within privacy and licensing limits. Recorded responses can test the harness or replay a matching action, but a candidate may ask a different question or take a different path. Match fixtures by the request contract, fail visibly on an unmatched request, and label any simulated fallback. Feeding the next historical response to a different call would test an invented world.
Failed agents can leave drafts, reservations, branches, sessions, or partial database writes. Snapshot before and after, attribute changes to events, and restore or discard state. If cleanup is uncertain, use disposable isolation and treat an escape as critical failure.
- Create one isolated state namespace per trial.
- Pin tool schemas and fixtures with content hashes.
- Classify every external dependency as live, recorded, simulated, or blocked.
- Capture state before, after, and at consequential actions.
- Verify cleanup independently of the agent final message.
- Fail the trial if it acts outside the permitted boundary.
Grade external state and the path separately
Start with deterministic state checks where possible: passing tests, expected rows, valid files, the intended calendar event, or no unauthorized message. Then evaluate required and prohibited steps, permission boundaries, cost, and latency. A final-answer grader cannot establish external success.
Semantic tasks may need model-based graders, whose grades are themselves derived records. Retain model, prompt or rubric, evidence, output, and confidence; calibrate against expert-labeled traces and review disagreement. AgentRewardBench found no evaluated language-model judge excelled across all included web-agent benchmarks, supporting combined graders rather than a universal judge.
Distinguish completion with unacceptable side effects, outcome failure caused by a broken environment, and ungradeable trials with missing traces. Report outcome, policy, side effects, efficiency, trace integrity, and environment validity rather than collapsing them into one scalar.
| Grader | Evidence | Example result | Failure meaning |
|---|---|---|---|
| outcome | Final environment and artifacts | Requested record exists | Task not completed |
| process | Ordered tool events | Required validation ran | Success path violated procedure |
| policy | Actions and authorization context | No external send without approval | Boundary breach |
| side effect | Before/after state diff | Only approved files changed | Collateral mutation |
| efficiency | Tokens, calls, latency, cost | Within declared budget | Operational regression |
| trace integrity | Event sequence and artifact availability | Complete and ordered | Result cannot be fully audited |
Measure reliability over repeated attempts
One successful demonstration establishes possibility, not reliability. Repeat representative tasks with fixed fixtures and configurations, reporting passes and denominators. Stratify task families and risk so many easy successes cannot conceal a consequential failure.
Consider an illustrative baseline that completes 8 of 10 trials and a candidate that completes 9 of 10. The candidate’s extra completion is not an improvement if one of its nine successes sent an unauthorized message. Retain completion and policy outcomes separately, inspect the paired changes, and report the small denominator. A higher pass percentage cannot resolve a failed hard boundary.
Where practical, pair baseline and candidate task versions and seeds. Compare outcomes, policy violations, side effects, cost, and latency. Intervals or resampling can express uncertainty, but cannot repair ambiguous, contaminated, or incorrectly graded tasks.
Inspect the traces behind score changes. Grader exploitation, skipped safety checks, retries until a flaky state passes, or unmeasured side effects can inflate a result. Review wins, losses, disagreements, and unusually cheap or costly attempts before promotion.
- Predefine the primary metric and non-negotiable safety metrics.
- Use the same task versions for baseline and candidate comparison.
- Report numerator, denominator, exclusions, and invalid trials.
- Separate task-level variance from repeated attempts on one task.
- Review representative traces from every changed outcome class.
- Keep benchmark edits out of the comparison window unless rebaselined.
Promote the whole agent configuration
Version model, prompts, tool contracts, retrieval, memory, policies, budgets, and harness code together. A model change may alter tool use; a prompt change may raise cost; stricter policy may lower nominal success while preventing unsafe completion. Keep these tradeoffs visible.
Freeze a held-out evaluation set before the patching agent investigates failures. Development replays can become regression cases, but repeatedly tuning against them turns their score into a measure of fit to known cases. Promotion needs tasks or scenarios the candidate has not used for diagnosis, with the same success and side-effect contracts.
Stage deterministic contract tests, recorded-dependency replay, isolated repeated trials, adversarial or policy cases, and a bounded canary with reversible, observable external effects. Set rollback conditions beforehand. Average scores cannot override critical authorization or destructive-action failures.
Retain task and configuration manifests, trial ledger, grades, aggregate report, selected traces, gaps, and promotion decision. This supports later production diagnosis and a bounded claim: the candidate met that gate on that distribution at the recorded cutoff.
Every aggregate row can resolve to one task, configuration, trace, and grader set.
{
"trial_id": "trial-0184",
"task": { "id": "publish-draft", "version": "3" },
"configuration": { "id": "agent-candidate", "version": "7" },
"environment": { "fixture": "workspace@sha256:...", "network": "recorded" },
"trace": { "id": "trace-0184", "integrity": "complete" },
"graders": ["outcome@2", "policy@5", "side-effects@3"],
"result": { "outcome": "pass", "policy": "pass", "side_effects": "pass" },
"cost_usd": 0.42,
"duration_ms": 18420
}Promote the recorded configuration
Keep tasks, configuration, traces, side effects, graders, and repeated results together. Passing supports the tested bundle on the recorded distribution; replaying its evidence cannot promise that a stochastic model will take the same path again.
Sources
- Anthropic Engineering: demystifying evals for AI agents
Verified Jul 19, 2026
- OpenAI API reference: Evals
Verified Jul 19, 2026
- AgentRewardBench: evaluating automatic evaluations of web-agent trajectories
Verified Jul 19, 2026
- W3C Trace Context Recommendation
Verified Jul 19, 2026
- Anthropic: a statistical approach to model evaluations
Verified Jul 19, 2026