Evaluating an agent

"How do you evaluate an agent that takes 20 steps?"

What it is

Agent evaluation measures whether a multi-step, tool-using system accomplished the task, not whether each step looked reasonable. The primary metric is task-level success rate against a rubric, with three companions: trajectory efficiency (steps and tokens to completion), cost per successful task, and reliability across repeated attempts.

The distinction from single-turn LLM evaluation is structural. A single-turn system has one output you can score. An agent produces a trajectory: a sequence of reasoning, tool calls, observations and corrections, and it can reach the right answer through a bizarre path or the wrong answer through a plausible one.

Commonly confused with per-step accuracy, which is the natural thing to instrument and is close to useless. An agent that gets 95 percent of steps right across 20 steps succeeds $0.95^{20} = 36$ percent of the time if the errors are independent. Per-step accuracy of 95 percent sounds excellent and describes a system that fails two times in three.

The problem it solves

Three things go wrong without task-level measurement.

You optimise the wrong thing. Per-step accuracy rewards an agent that takes safe, unhelpful steps. An agent that calls search twenty times and never acts scores well on step correctness and accomplishes nothing.

You cannot compare architectures. ReAct versus plan-and-execute versus a supervisor with sub-agents produce completely different trajectories, so any step-level metric is comparing different things. Task success is the only metric that is comparable across designs.

You do not see the reliability problem. An agent that succeeds 70 percent of the time is not a 70-percent-good product; it is a product that fails for a third of users, and if the same user retries they may fail again. Single-run metrics hide this completely, which is why the reliability metric below matters so much.

Mechanics

The metric set

Task success rate. Binary or rubric-scored, against a defined end state. The definition of "success" is the hard part and it must be checkable:

# Good: verifiable end state, checked programmatically.
{
  "task": "Refund order 8842 and notify the customer",
  "success_criteria": [
    {"check": "db", "assert": "orders[8842].status == 'refunded'"},
    {"check": "db", "assert": "refunds.exists(order_id=8842, amount=4299)"},
    {"check": "outbox", "assert": "email_sent(to=order.customer_email)"},
    {"check": "db", "assert": "not orders[8843].modified"},   # no collateral damage
  ]
}

# Bad: "the agent handled the refund appropriately". Unfalsifiable, and it
# forces a model judge on something a database query could answer.

The "no collateral damage" assertion is the one teams forget, and it catches the agent that accomplishes the task by doing something destructive alongside it.

Trajectory efficiency. Steps to completion, tokens consumed, wall-clock. Two agents with the same success rate and a 3x difference in steps are not equivalent: one costs three times as much and takes three times as long.

Cost per successful task. The metric that makes the tradeoff honest. A cheaper model with a lower success rate can be more expensive per success, and this number is the only one that says so:

$$\text{cost per success} = \frac{\text{total cost}}{\text{successful tasks}}$$

Reliability across attempts (pass^k). Run the same task $k$ times independently and measure the fraction where all $k$ attempts succeed. This is the metric introduced by τ-bench and it is the one that exposes what single-run evaluation hides: agents that look acceptable at pass@1 degrade sharply as $k$ rises, because success was partly luck.

def pass_hat_k(results_by_task, k):
    """Fraction of tasks that succeed on ALL k independent attempts.
    pass@1 answers 'can it do this?'. pass^k answers 'can I ship it?'"""
    return sum(all(runs[:k]) for runs in results_by_task.values()) / len(results_by_task)

The shape you typically see, and it is worth being able to sketch:

A system whose pass@1 is 0.61 and whose pass^8 is 0.25 is not reliable enough for an autonomous workflow, and only the second number tells you.

Safety and containment metrics, which belong in the same suite:

• Rate of destructive actions taken without confirmation.
• Rate of tool calls outside the granted scope (should be zero, enforced).
• Injection resistance: fraction of adversarial tasks where planted instructions changed behaviour.
• Loop rate: fraction of runs hitting the iteration or token cap.

Building the evaluation set

Between 50 and 200 tasks, and unlike RAG evaluation the labelling cost is in defining checkable end states rather than in labelling documents.

The last two strata are the ones that separate a real suite from a demo. An agent that never gives up produces confident nonsense on impossible tasks, and that is a worse product than one that says it cannot do this.

Environment determinism is the practical hard part. An agent that touches real systems is not repeatable, so you need a sandboxed environment with seeded state that resets between runs. Record-and-replay for external APIs, a fixture database, and a frozen clock. Without this, pass^k is measuring environment variance rather than agent reliability, and you will chase noise for weeks.

Where per-step analysis is still useful

Not as a metric, as a diagnostic. When a task fails, the trajectory tells you where:

Step 1  search_orders("8842")            -> found          ok
Step 2  get_order(8842)                  -> 200            ok
Step 3  calculate_refund(8842)           -> 4299           ok
Step 4  process_refund(8842, 42.99)      -> 200            <- WRONG UNIT
Step 5  send_email(...)                  -> 200            ok
Result  FAIL: refunds.amount == 4299 assertion failed

The failure is a unit error at step 4, and the tool's schema should have prevented it (minor units as an integer, documented in the description). That is a tool design fix, not a prompting fix, and trajectory analysis is what tells you so.

Categorise failures rather than counting them. Typical distribution:

The distribution is the point: most agent failures are tool-design failures, not model failures. Teams reach for a bigger model when the fix is a better tool description, and the trajectory data is what settles that argument.

A worked example

A support agent that can look up orders, issue refunds, and email customers. 120 evaluation tasks in a sandboxed environment with seeded state.

Baseline:

Two readings jump out. Failed runs take twice as many steps, which means the agent flails rather than failing fast, so every failure costs double. And pass^4 of 0.39 against pass@1 of 0.68 means a third of the apparent successes were luck.

Failure analysis on the 38 failures:

• 14: called process_refund before verifying eligibility, so the refund was rejected downstream and the agent did not recover.
• 9: looped between search_orders and get_order when the order id was ambiguous.
• 7: unit errors on amounts (dollars versus cents).
• 5: fabricated an order id when the search returned nothing.
• 3: destructive action without confirmation.

Fixes, all in the tool layer rather than the prompt:

1. process_refund gains a precondition in its schema and rejects with a structured error naming check_eligibility as the required prior call. Fixes 14.
2. Amounts become integer minor units, with the description stating so and an example. Fixes 7.
3. search_orders returns a structured empty result with an explicit "no_match": true rather than an empty list, so "nothing found" is unambiguous. Fixes 5.
4. Cycle detection: identical tool call with identical arguments three times ends the run with a clear failure. Fixes 9, and makes failures cheap instead of expensive.
5. confirm: true on every destructive tool. Fixes 3, structurally.

After:

The model was never changed. That is the headline: a 21-point improvement in pass@1 and a doubling of pass^4 from tool schemas, error messages and a loop guard. It is the single most useful thing to convey about agent evaluation, because the instinct in the room is always "use a better model".

Production evidence

τ-bench (Yao et al., 2024) evaluates agents on tool-agent-user interaction in retail and airline domains, and introduced pass^k as the reliability metric. Its headline finding is the one quoted above: frontier agents that look reasonable at pass@1 degrade substantially as $k$ increases, meaning consistency, not capability, is the barrier to autonomous deployment.

SWE-bench (Jimenez et al., 2023) evaluates agents on real GitHub issues with a verifiable success criterion: does the generated patch make the repository's own tests pass. It is the clearest example of the "checkable end state" principle, because the rubric is a test suite that already existed.

WebArena (Zhou et al., 2023) provides a reproducible, self-hosted web environment with programmatically verifiable task completion, which is the environment-determinism problem solved properly.

Anthropic's published guidance on building effective agents makes the same tool-design argument from the other direction: that tool definitions and their descriptions deserve as much engineering attention as prompts, because the model's behaviour is largely determined by the interface it is given.

The debate

The alternative is process-based evaluation: score the trajectory itself, with a model judging whether each step was reasonable. It has a real advantage on tasks where the outcome is hard to verify programmatically (research, writing, analysis), where "did it do a good job" genuinely is a judgement call.

Its weakness is that it measures plausibility rather than correctness, and it inherits every LLM-judge bias. A trajectory can look excellent and produce the wrong answer; agents are quite good at producing reasonable-looking steps.

The other alternative is online metrics only: task abandonment, escalation to a human, user thumbs. Ground truth, and slow, noisy, unavailable pre-launch, and useless for CI.

My position: outcome-based task success as the primary metric, with checkable end states asserted programmatically wherever possible; pass^k rather than pass@1, because consistency is what determines whether you can ship it; cost per successful task so cheaper-model tradeoffs are honest; and trajectory analysis as a diagnostic for categorising failures rather than as a metric. Process-based judging only where the outcome genuinely cannot be verified, and then with the judge validated against humans.

Outcome-only evaluation is the wrong choice for open-ended tasks with no verifiable end state, and for safety properties, where "it did not do anything destructive this time" is not evidence. Those need explicit adversarial tasks and enforced tool scoping rather than measurement.

Follow-up Q&A

"How do you evaluate an agent that takes 20 steps?" On task-level success, not per-step accuracy, because 95 percent per-step accuracy over 20 steps compounds to about 36 percent task success if errors are independent. I define checkable end states, assert them programmatically, and include a no-collateral-damage assertion. Then trajectory efficiency and cost per successful task, and pass^k rather than pass@1, because an agent that succeeds once in four attempts is not shippable and single-run metrics hide that.

"What is pass^k and why does it matter more than pass@1?" pass@1 asks whether the agent can do the task; pass^k asks whether it does the task on all $k$ independent attempts. τ-bench introduced it and found that frontier agents degrade sharply as $k$ rises, which means apparent success is partly luck. For an autonomous workflow the relevant question is consistency, because a user who retries a failed task and fails again has a broken product, not a probabilistic one.

"Your agent's success rate is 70 percent. What do you do first?" Categorise the 30 percent, because the fix depends entirely on the category and my prior is that most of it is tool design rather than model capability. Wrong tool selected means the description does not say when to call it. Right tool with wrong arguments means the schema is too permissive, so add enums, constraints and examples. Looping means no cycle detection. Fabricated inputs usually mean a tool returns an ambiguous empty result. In the case I worked through, tool-layer fixes alone took pass@1 from 0.68 to 0.89 with no model change.

"How do you make agent evaluation repeatable?" A sandboxed environment with seeded state that resets between runs, record-and-replay for external APIs, and a frozen clock. Without it, pass^k measures environment variance rather than agent reliability and you will chase noise. This is genuinely the expensive part of agent evaluation, and it is why WebArena and SWE-bench are valuable: they solved the environment problem, not just the task problem.

"How do you evaluate safety properties?" Not by measuring their absence in normal runs, because that proves nothing. Explicit adversarial tasks with planted instructions, measuring whether behaviour changed; a count of tool calls outside the granted scope, which should be zero and should be enforced rather than measured; and confirmation gates on destructive actions so the property is structural. The principle: measure what you can, and for the things you cannot measure reliably, constrain them architecturally instead.

Common misconceptions

The biggest is that per-step accuracy is a useful headline metric. It compounds catastrophically over a long trajectory and it rewards agents that take safe, unhelpful steps.

The second is that a low success rate means you need a better model. In most deployments the failure distribution is dominated by tool selection and argument errors, both of which are fixed in the tool schema and description.

The third is that pass@1 is enough. It answers whether the agent can do the task. Shipping requires knowing whether it reliably does, which is pass^k, and the gap between the two is usually large.

Interview delivery note

Say this: "Task-level success rate, not per-step accuracy. Ninety-five percent per step over twenty steps compounds to about thirty-six percent task success, so per-step numbers flatter a system that fails two times in three. I define success as a checkable end state asserted against the environment, including an assertion that nothing else changed. Alongside it: trajectory efficiency, cost per successful task, and pass^k rather than pass@1, because an agent that succeeds once in four attempts isn't shippable and single-run metrics hide that."

The depth signal is what you do with a failure: "per-step analysis is a diagnostic, not a metric. I'd categorise failures, and my prior is that most are tool design rather than model capability: wrong tool selected means the description doesn't say when to call it, wrong arguments means the schema is too permissive. In the case I'm thinking of, tool-layer fixes alone took pass@1 from 0.68 to 0.89 with no model change." That last sentence is the one that lands, because the expected answer in the room is always "use a bigger model".

Further reading

• Yao et al., "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains" (2024), for pass^k and the reliability finding.
• Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (2023), for verifiable success criteria from an existing test suite.
• Zhou et al., "WebArena: A Realistic Web Environment for Building Autonomous Agents" (2023), for reproducible environments.
• Anthropic, "Building Effective Agents", on tool design as the primary determinant of agent behaviour.