Evals as regression tests

A prompt is code. A tool description is code. A model id is configuration. Each of them changes what the platform does, and each ships today, in most agent projects, with no test that can fail. That is the gap this chapter closes. Software earned reliability when it made behavior testable in CI; agent platforms earn it the same way, with evals: golden task suites, graded outcomes, and a pass threshold that blocks a merge. Gate 8 asked whether changing your system prompt could fail a pull request; this chapter is how the answer becomes yes.

Why agent evals are harder than unit tests

A unit test asserts an exact output: add(2, 3) == 5. Agent behavior resists that in two ways, and naming them tells you what the eval machinery must supply.

First, the output is open-ended. "Summarize this incident" has no single correct string; many summaries are good, more are acceptable, some are wrong, and equality testing cannot tell them apart. Second, the unit under test is a trajectory, not a value: an agent can reach a fine answer through a wasteful, wrong, or unsafe path, and a test that checks only the final answer misses that the agent read data it should not have or looped twelve times to get there. Evals must therefore score quality on a rubric rather than assert equality, and they must be able to score the path, using the traces the last chapter made readable, not just the endpoint.

The three pieces

An eval suite for agents is three things:

  • A golden task set. A curated collection of representative tasks with known-good outcomes or rubrics, the agent equivalent of a test fixture. It is the most valuable and least glamorous artifact in the whole eval story: it encodes what "good" means for your domain, no vendor ships it, and it grows every time production surprises you (each real failure becomes a new golden case, exactly as bugs become regression tests). Curating it is the work; everything else is machinery.
  • A grader. Something that scores an outcome against the rubric. For checkable properties (did the extraction produce a numeric price field? did the agent avoid the delete tool?), the grader is plain code, deterministic and free, and you should prefer it wherever the property is expressible. For open-ended quality (is this summary accurate and complete?), the grader is an LLM judge: a model with a rubric reading the output, and often the trajectory. Everything the verification mesh taught about judges applies here, because it is the same tool pointed at a different job.
  • A gate. A threshold that turns scores into a pass/fail: this suite must score at least X, or the merge is blocked. The gate is what makes the suite a test rather than a report you can ignore.

LLM judges, and their calibration

The LLM judge is powerful and treacherous, and the treachery is the part teams skip. A judge is a probabilistic scorer: it is itself an agent output, subject to the same variance, and its absolute numbers mean nothing until you have calibrated it against human labels. The discipline: take a sample of outcomes, have humans score them, have the judge score them, and measure agreement (a statistic like Cohen's kappa answers "does the judge agree with humans more than chance"). A judge that tracks human judgment is trustworthy; one that does not is a random number generator with a rationale, and shipping gates on an uncalibrated judge is worse than no gate, because it manufactures false confidence.

Two practices keep judges honest. Prefer relative to absolute: a judge is far more reliable at "is candidate B better than baseline A" than at "is B an 8.5 out of 10," so build gates that compare a candidate to the current baseline rather than to an abstract bar, which is also exactly what a regression test wants. And use a strong judge: the model portfolio said verification deserves the best model you can afford, and eval grading is verification; a cheap judge that usually agrees is a false economy at the one place accuracy is the product.

Where evals run

The suite lives in three places, at three cadences:

  • In CI, on every change. A prompt, tool, or model change opens a pull request; the golden suite runs; the gate blocks the merge if scores regress. This is the loop that makes the platform's behavior testable, and it is the direct answer to gate 8. Model upgrades run the same gate, which is exactly the candidate-evaluation step the upgrade playbook specified.
  • As a canary, in production. Some regressions only appear against live traffic's messiness, so a small slice of production runs on the candidate with its outcomes scored, catching what the golden set's curation could not anticipate. The managed Evaluations service is a way to buy this continuous-scoring machinery; the golden set and the gate stay yours.
  • On a schedule, against drift. The tracking-model eval from the upgrade playbook runs the suite against the latest model version continuously, so vendor-side drift is a report, not a surprise.

The eval, memory, and mesh trinity

Three of this book's labs are the same idea at different scopes, and seeing that is worth a paragraph. Replay makes a single run reproducible, so a recorded failure becomes a fixture. The memory eval harness makes memory quality a monitored number via plant-disturb-probe-score. This chapter makes the whole agent's behavior gated via golden suites and judges. All three convert a fuzzy quality into a regression-testable number, which is the move that separates an agent platform you can change with confidence from one you change with prayer. A team that has all three can refactor a prompt at 4 p.m. on a Friday; a team with none ships every change as an experiment on its users.

Don't be confused: evals vs the verification mesh. They share the LLM-judge machinery and blur constantly. The mesh (Chapter 26) runs inside a production run, gating individual artifacts before they reach the output, and its verdict is keep-or-kill per claim, right now. Evals run about the system, offline or on a canary, scoring trajectories to decide whether the platform got better or worse this week, and they change nothing about the runs they score. One is a hot-path quality control; the other is a development-time regression test. Confuse them and you either block production traffic on slow offline tooling or trust ungated artifacts because "we have evals."

👉 Next: cost engineering, where the token ledger from Chapter 40 becomes budgets enforced, caches shaped, and a regression gate that fails CI when a change makes every run more expensive.