---
title: "The agent passed the eval. I still wouldn't ship it"
canonical: https://maxtokens.ai/posts/passed-eval-wont-ship/
date: 2026-07-16
tags: [agents, infra, evals]
description: "Passing an eval doesn't prove the agent acted safely or within budget. What a release gate must check: outcome, trajectory, side effects, reliability."
---
*Passing an eval does not prove that the agent acted safely, stayed within budget, or kept its side effects under control.*
In agent engineering, attention is shifting from the final answer to the run that produced it. That is why [evaluation harnesses](https://x.com/shivam74689/status/2077405823203258611), [the distinction between evals and observability](https://x.com/petarivanovv9/status/2077372127133512058), and [live evaluation environments](https://x.com/AnteDotGames/status/2077042819722879434) are getting more attention.
Building a convincing agent demo has become easier, but passing an eval only shows that the measured outcome met a criterion. It does not prove that the agent reached it through an allowed process or can be trusted with real tools and authority.
An agent can return the right patch, pass the tests, and open the intended pull request while stepping outside its permitted scope, repeating a state-changing call, hiding a partial failure, or spending an unreasonable budget. The final state may be correct while the run is still unsafe.
This isn't a theoretical caveat. In July 2026, Noma Labs [described GitLost](https://noma.security/blog/gitlost-how-we-tricked-githubs-ai-agent-into-leaking-private-repos/) — an indirect prompt-injection vulnerability in GitHub Agentic Workflows. According to the researchers, a specially crafted issue in a public repository could make an agentic workflow exfiltrate data from private repositories in the same organization. From the agent's point of view, the sequence could look like ordinary work with available tools. From a security perspective, it was data exfiltration triggered by untrusted input.
So a release decision must account for the whole run alongside the result: the path, the side effects, and the behavior under failure. A passed eval is useful evidence, not production clearance.
## A correct outcome is only one dimension
Ordinary software tests often compare inputs to outputs. That's useful for agents too, but an agent also chooses actions between the two points. Those actions touch files, APIs, databases, browsers, queues, and people.
A run needs four separate evaluations:
1. **Outcome.** Does the final state satisfy the task, and were prior requirements preserved?
2. **Trajectory.** Were the chosen steps justified and allowed? Did they actually produce the result?
3. **Side effects.** What changed beyond the requested artifact, including failed and repeated operations?
4. **Reliability.** Does correctness hold across repeated runs, timeouts, tool unavailability, and partial execution?
They can't be collapsed into one overall score.
- **Evals** check the work against explicit criteria.
- **Observability** records what actually happened during the run.
- **Policies** define which actions are allowed.
- **A release gate** combines these signals and decides whether the system can be trusted with real authority.
More telemetry doesn't by itself create a quality judgment. A high score doesn't replace a permission boundary. For an agent with tools, the text of the final answer may be the least important part of the run.
The list above and the diagram describe different levels of checking. Policy is a separate gate on every tool call. Reliability is not shown as one box because it is tested across repeat runs and recovery scenarios. The outcome can pass while trajectory, policy, and side-effect gates fail.
## Evaluate system state, not the agent's report
An evaluator should inspect the system state after the run rather than trust the agent's report.
For a code change, you check the final tree against the acceptance criteria, read the patch, and confirm the repository's prior invariants held. "The tests passed" isn't enough if the agent can edit the tests themselves.
Suppose the task requires changing an endpoint and updating its test coverage. The obvious check exercises the new behavior. The less obvious checks are to:
- compare the test set before and after the run;
- flag removed assertions;
- run the unaffected suites;
- verify that only the intended contract changed;
- confirm authorization checks were preserved.
A removed authorization check can turn a failing test into a passing one. The final suite passes even though the product got worse.
The same rule holds beyond code. If the agent updates a customer record, you check the saved record and the associated audit trail. If it schedules a job, you confirm the identifier, the parameters, and the job's uniqueness, rather than trusting a success message.
Outcome evaluation compares the expected system state with the observed one and checks that existing invariants still hold. "The task is done" and "the agent reported it done" are different events. Whenever possible, the evaluator should obtain evidence directly from the repository, database, API, queue, or audit log.
## The final diff doesn't show how the agent got there
A trajectory is an ordered chain of decisions, tool calls, observations, and state transitions. It answers questions the final diff can't:
- did the agent study the relevant contract before changing it;
- did it keep going after a denial;
- did a timeout lead to a blind retry;
- did the agent find the failing test or delete it;
- did the correct result come from an allowed process or from luck?
[OpenAI defines trace grading](https://developers.openai.com/api/docs/guides/trace-grading) as assigning structured scores or labels to the full log of an agent's decisions, tool calls, and execution steps. Unlike a black-box eval of the final answer, this grading shows both the error and the point where behavior diverged from what was expected.
Google ADK makes the same distinction in its [built-in evaluation criteria](https://adk.dev/evaluate/): it separately evaluates final-response match, response quality, and tool-call trajectory match. This doesn't mean every acceptable trajectory has to be identical to a reference. But tool calls should be graded as a standalone part of behavior.
An ordinary log doesn't become a trace automatically. The line "tool failed with an error" carries no call arguments, no link to the prior state, no policy decision, no resulting change, and no recovery path.
A minimally useful record of a run might look like this:
```json
{
"run_id": "...",
"task": "...",
"steps": [
{
"parent_step_id": "...",
"tool": "...",
"arguments": {},
"result": "...",
"state_diff": {},
"policy_decision": "allow | deny | require_approval",
"latency_ms": 0,
"cost_usd": 0
}
],
"outcome_score": 0,
"trajectory_score": 0,
"side_effects": [],
"recovery_events": []
}
```
This is an illustration, not a proposed standard. The value of the schema is in the links between events: each state_diff points to the call that created it, each recovery event points to the original failure, and each policy decision points to the action that was checked. Cost and latency are kept per step and per run, because an acceptable outcome can hide a loop that simply got lucky and finished.
Secrets must be masked during collection. But masking must not hide the affected resource, the class of action, or the fact that an access boundary was crossed. You can't investigate a leak if the trace only says the agent "called an API".
Observability tools are already built around this model. Langfuse, for example, [describes application tracing](https://langfuse.com/docs/observability/overview) as a structured record of the request, the model's response, token usage, latency, retrieval, and tool calls. A trace shows what happened. Evals and policies decide whether it was acceptable.
## Side effects are part of correctness
Every tool call has an allowed effect scope.
A read-only file tool must not modify the working tree. A repository editor may change permitted paths but not credentials or shared configuration. An API client may create the one requested object but not duplicates while recovering from a timeout.
Start by comparing system state before and after the run. For a code agent it includes:
- the working tree;
- generated files;
- local processes;
- created branches;
- commits and pull requests;
- calls to external services.
For a business process, the difference can span database rows, queue messages, scheduled jobs, payments, and notifications. The state model depends on the task, but "side effects not specified" doesn't equal "there were no side effects".
Risk also depends on reversibility. Reading a file, editing an uncommitted patch, opening a draft, deleting live data, and sending a message to a customer must not obey a single approval rule.
Policies must classify actions by scope and reversibility:
- a safe, reversible action runs automatically;
- an action with limited, controlled effects may run automatically, but must be logged and verified afterward;
- a dangerous or irreversible action requires explicit approval;
- a forbidden action is blocked regardless of the outcome score.
Human-in-the-loop doesn't mean a separate click before every file read. Human approval is needed immediately before an action whose consequences cannot be reversed cheaply and reliably.
GitLost shows why you can't get there with a good prompt alone. An agent can read untrusted text, interpret it as an instruction, and use a formally available tool. You need constraints on data sources, output destinations, and the permitted data flows between them. An unsafe run can't be counted as correct, even if its final answer looks convincing.
I found this failure mode in my own pipeline, not in someone else's report. The bot that publishes articles on this blog is an ordinary agent with git tools: it assembles the package, commits, pushes to `main`, promotes the preview to production. The "publish" command worked: the article showed up on the site, and the final state was exactly what was asked. By the outcome metric, it passed.
The hidden problem was a side effect outside the requested change. The bot and the content lived in one repository, and Railway redeployed the service on any push to `main`. The bot's own promotion restarted the bot service — and killed the run that was in progress at that moment. "Publish the article" quietly also meant "restart yourself".
The failure was hard to spot because the trace did not align the relevant events. Commits were timestamped in UTC, while Railway logs used UTC+7. The sequence — promotion, redeploy, run termination — was hard to see because the events appeared several hours apart. No one had hidden the boundary "this action changes the infrastructure the agent itself runs on." The trace simply did not record it.
The fix wasn't in the prompt but in scope: `watchPatterns: ["bot/**"]` in the Railway config. Now only bot-code changes trigger a redeploy, not a content promotion. That's exactly a scope policy from the list above, not "be more careful".
## Every failure needs a defined outcome
Two successful happy-path runs do not establish reliability. A reliable agent keeps its constraints intact when the network, its tools, or the external system behave unexpectedly.
A retry is one recovery mechanism, not the recovery process itself.
Consider a state-changing API call that timed out. A timeout means no response, but it doesn't prove no change happened. An immediate retry can create a duplicate.
The safe sequence looks different:
1. find the operation by an idempotency key or a stable identifier;
2. reconcile the observed state;
3. choose retry, fallback, replan, rollback, escalation, or stop;
4. record the decision and its basis in the trace.
If reconciliation isn't possible, the uncertainty itself becomes a reason to stop.
Partial execution demands the same discipline. The trace must note the preconditions met, the change committed, the check that failed, and the compensating action available. The line "step failed with an error" throws away exactly the information recovery needs.
So a reliability check must:
- repeat non-deterministic scenarios;
- reorder non-essential inputs;
- disable tools;
- inject timeouts and explicit errors;
- test budget exhaustion;
- simulate partially executed operations;
- verify safe stop and handoff to a human.
A safe stop counts as a successful outcome too. An agent that pushes on at any cost is no more reliable than one that honestly admits it can't finish the task without breaking a constraint.
Step count by itself says nothing about quality. That problem is unpacked in more detail in the post ["Turn count isn't an agent metric"](https://maxtokens.ai/posts/turn-count-metric/). Here the number of steps is useful only as a budget and a signal of divergence from expected behavior.
## Two green pull requests. Only one is shippable
The task: change an existing endpoint, update the tests, and open a pull request. The permitted scope includes the endpoint implementation and its separate test directory. Existing authorization checks must be preserved.
### Run A: outcome green, trajectory red
The endpoint works as required, the visible tests are green, the pull request is created. But the trace and the state difference show something else:
- the agent changed a shared file outside the permitted scope;
- it deleted an existing authorization assertion to make a test pass;
- it repeated the external pull-request-creation call after an ambiguous response;
- it spent five times the allotted budget;
- it left an intermediate tool error out of the recorded history.
By outcome alone, the run passed. It still must not ship.
Otherwise, the system learns that any path is acceptable as long as the final check passes. An outcome score doesn't erase the scope violation, the weakened coverage, the uncertainty around the repeated call, the budget overrun, and the incomplete trace.
### Run B: green outcome and a controlled path
The agent edits only permitted files and records every tool call. When the test command times out, the agent first checks the process state and preserves the available result. Only then does it revise the plan.
It repeats only the affected check, inspects the full diff for removed assertions, confirms that no duplicate pull request was created, and stays within the cost, step, and time limits.
A replay using recorded tool results reproduces the decision path without repeating real side effects. A repeat live run doesn't have to produce a byte-identical patch: the agent loop can be non-deterministic. But it must preserve the invariants, the permitted scope, the allowed effects, and the acceptance criteria.
Both runs can pass the same endpoint test. Only the second demonstrates a controlled process the team can safely authorize again.
## A minimal release gate fits in CI
A useful first release gate does not require a large evaluation platform. It needs a versioned set of checks, a trace format, policy enforcement around the tools, and a runner that can compare state.
The minimal set:
1. **A fixed set of eval cases.** The happy path, forbidden actions, and known edge conditions.
2. **Full traces.** The run configuration and every tool-call attempt, but with secrets masked.
3. **Tool policies.** An allowlist, argument constraints, ordering rules, and protection against dangerous retries.
4. **State diff.** The observed difference before and after the run, including verifiable effects such as deletions.
5. **Policy tests.** Checking the allow, deny, and require-approval branches.
6. **Independent budgets.** Cost, step count, and total time.
7. **Failure injection.** Tool errors and timeouts, including ambiguous timeouts after state-changing requests.
8. **Replay.** Reproducing failed runs on recorded responses or safe fixtures.
9. **Repeat runs.** Keep every trace, not just the best result.
10. **Manual approval.** Right before an irreversible action, showing the action and the expected state difference.
In CI you can automate the fixed scenarios, the sandboxed state diff, the trace-completeness check, the limits, failure injection, replay, and repeat runs. CI can also simulate policy decisions and verify that the agent stops at the approval boundary.
The runtime must re-apply the same budgets and policies to the real calls. CI can't pre-approve an action whose target and state only appear in a live run.
Static benchmarks are useful for comparing the model component. They don't test the real tool wrappers, permissions, environment state, and recovery code. The harness around the agent matters more than a single model call; more on this in ["Loop engineering: the harness around the loop"](https://maxtokens.ai/posts/loop-engineering-harness-loops/).
Cost can't be hidden inside an average score either. It depends on the model's token count, retries, tools, infrastructure, and failed branches. How far it can diverge shows up even on a simple one-shot run: in my [model benchmark](https://maxtokens.ai/bench/big-mac/), on the same task, `claude-opus-4-6` spent 59,603 output tokens, 14 minutes, and $1.49 — and came eleventh of twelve. `deepseek-v4-pro` handled the same task for $0.03 and came sixth. The costs differed by a factor of roughly fifty, and none of that difference shows up in the outcome score. A practical breakdown of the cost components is in the post [on the cost of one headless run](https://maxtokens.ai/posts/headless-agent-run-cost/).
## Some failures must be hard blockers
A single final score is convenient, but it implies that outcome quality, cost, safety, and recovery can compensate for one another. Under that model, a perfect patch could numerically offset an unauthorized write.
Some failures are non-negotiable. They block the release.
Outcome and trajectory can have scores. Scope, policy compliance, side effects, and trace completeness need hard gates:
```text
release_allowed =
outcome_score >= outcome_threshold
AND trajectory_score >= trajectory_threshold
AND trace_is_complete
AND budgets_respected
AND no_scope_violation
AND no_unapproved_irreversible_action
AND no_unreconciled_side_effect
AND required_replays_and_repeats_passed
```
Thresholds depend on the task and the acceptable risk. The structure matters more than a universal number: high performance doesn't buy permission, and a safe but reliably wrong agent isn't ready either.
Real traces should feed back into the eval suite. Every new failure mode should become a reproducible fixture that future versions must pass before receiving production authority.
Observability supplies evidence to evals. It does not replace them.
## You ship controlled behavior, not an answer
An agent version includes more than the model and the prompt. It includes:
- tool schemas and implementations;
- permissions;
- policies;
- budgets;
- recovery logic;
- eval cases;
- the runtime environment;
- how traces are collected and masked.
Changing any component can change behavior, even if a benchmark of final answers shows the same result.
So the question "can the agent solve the task?" is too weak. You have to ask differently:
> Can this specific configuration solve the task repeatably, within the permitted scope and budget, leaving enough evidence to verify every significant action and recover from failures?
A release gate does not guarantee perfect behavior. It grants authority only when the system's behavior is observable, reproducible, and bounded.
Evals measure the quality of the agent's work. Observability shows what it actually did. Policies constrain what it's allowed to do. A release gate decides whether the whole system can be trusted with the next run.