Design an evaluation pipeline that gates CI
"A prompt change should not reach production without evidence it did not make things worse. Design the pipeline that enforces that."
Step 1: clarify, and name what makes this different (4 minutes)
The requirement sounds like ordinary CI and it is not, for one reason: the system under test is non-deterministic and the correct answer is often not exactly specified.
Ordinary CI assert result == expected
Deterministic. A failure is unambiguous.
LLM CI The output is different every run, several outputs
are equally correct, and "worse" is a statistical
statement about a distribution rather than a
property of a single run.
That means the gate cannot be "did the test pass". It has to be "did the measured quality drop by more than noise", which requires knowing what noise is, and that is the whole design.
The clarifications:
What changes? Prompts, model version, retrieval config, tool
definitions, chunking, temperature. All of them
should gate; teams usually only gate prompts.
What is tested? 15 agents/features across 8 teams, each with its
own eval set.
How fast? A gate that takes 40 minutes will be bypassed.
Assume a target of under 10 minutes for the
blocking tier.
Cost? An eval run is real model spend. 500 cases x 8k
tokens x every PR is a real budget line.
Ground truth? Where does it come from, and how much is there?
This is usually the binding constraint.
The question that decides the design: "Do we have labelled ground truth, or only production traffic? If we have 200 labelled examples, that is a small sample and I need to be honest about what a 3 percent difference means on 200 cases, which is nothing."
Step 2: capacity and statistics (5 minutes)
The statistics are the capacity math here, and getting them right is the point.
Eval set size and detectable effect
With a binary pass/fail metric at ~80% baseline, the standard
error on a proportion is sqrt(p(1-p)/n):
n = 100 -> SE = 4.0% detectable difference ~11% (2.8 x SE)
n = 500 -> SE = 1.8% detectable ~5%
n = 2000 -> SE = 0.9% detectable ~2.5%
*** With 100 cases you cannot detect a 5-point regression. ***
Teams routinely gate on 50 cases and believe they have a gate.
Paired comparison changes this materially
Running old and new on the SAME cases and comparing per-case
removes between-case variance, which is the dominant term.
A paired test on 200 cases can detect what an unpaired test
needs 800 for.
-> ALWAYS run the baseline in the same job, on the same cases,
rather than comparing to a stored number from last week.
Model non-determinism
At temperature 0, most models are still not bit-identical
(batching, kernel non-determinism, provider-side changes).
Measured variance run-to-run on an unchanged prompt is the
NOISE FLOOR, and it must be measured rather than assumed.
-> Run the baseline 3 times on the eval set. The spread is
your noise floor. A change smaller than that is not a
signal, whatever the direction.
Cost and time
500 cases x 8k tokens in, 500 out, x 2 (baseline + candidate)
= 8M input + 500k output tokens per run.
At ~30 PRs/day across 8 teams that is real money, and it is
the reason for the tiered design in step 4.
Time: 500 cases x 2 at 50 concurrent = 20 batches x ~4 s = 80 s.
Fast, IF you parallelise. Serially it is 40 minutes and the
gate gets bypassed.
Three conclusions to state:
- Paired comparison, baseline re-run in the same job. Comparing to a stored number from last week conflates a real regression with provider-side drift.
- The noise floor must be measured, by running the unchanged baseline several times. A gate that fires on differences smaller than its own noise is worse than no gate, because it trains people to override it.
- Parallelise or the gate gets bypassed. This is a social constraint with a technical fix.
Step 3: the metric families
Two families, and reporting only one is the most common mistake. For a RAG or agent system:
RETRIEVAL / PROCESS metrics Did the system find and do the
right things?
recall@k Is the answer-bearing document in the results?
context precision Are retrieved chunks relevant?
tool correctness Did the agent call the right tools in a
reasonable order?
GENERATION / OUTCOME metrics Given what it found, was the
answer good?
faithfulness Is every claim supported by the context?
answer relevance Does it address the question?
task success Did the task actually complete?
Why both are required: an answer can be faithful to retrieved context that did not contain the answer, scoring well on generation while the system failed. And retrieval can be perfect while generation ignores it. A single aggregate score hides which half broke, which is exactly what you need to know when the gate fires.
Scoring mechanisms, in order of preference:
1. PROGRAMMATIC Exact match, schema validity, regex, numeric
tolerance, "did the tool get called".
Cheap, deterministic, zero noise.
*** Use for everything that can be. ***
2. LLM-AS-JUDGE A model grades the output against a rubric.
Necessary for open-ended quality.
Noisy, biased, and needs its own validation.
3. HUMAN Expensive, slow, and the only ground truth for
the judge itself.
LLM-as-judge needs to be treated as a measurement instrument that requires calibration, and saying so unprompted is a strong signal:
Known biases, all documented:
Position bias Prefers the first option in a pairwise
comparison. Fix: run both orders, average.
Verbosity bias Prefers longer answers. Fix: control for
length in the rubric, or measure it.
Self-preference Prefers outputs from its own model family.
Fix: use a different model as judge, or
accept and note it.
Validation: measure the judge's agreement with human labels on a
held-out set. Below ~80% agreement the judge is measuring
something other than what you think, and gating on it is worse
than not gating.
Pairwise comparison is more reliable than absolute scoring. Asking "is A or B better" gets much higher human agreement than asking "rate this 1 to 5", because the second requires the judge to hold a stable internal scale across independent calls, which it does not.
Step 4: the tiered pipeline
One gate cannot be both fast and thorough, so there are three.
TIER 1 SMOKE on every commit, ~30 s, BLOCKING
20-30 cases, programmatic scoring only.
Catches: broken prompt template, invalid tool schema,
a change that makes the system produce nothing.
Zero LLM-judge cost, near-zero noise.
TIER 2 REGRESSION on every PR, ~5 min, BLOCKING
300-500 cases, paired against a baseline re-run in the same job,
mixed programmatic and judge scoring.
Catches: a real quality regression above the noise floor.
Reports per-metric-family so the failure is diagnosable.
TIER 3 DEEP nightly + pre-release, ~2 h, NON-BLOCKING
2,000+ cases, full judge scoring, per-segment breakdowns
(language, query type, difficulty), adversarial and safety sets.
Catches: segment regressions invisible in the aggregate, and
slow drift.
# The gate decision, which is the part worth writing out.
def decide(baseline: Results, candidate: Results,
noise_floor: float) -> Gate:
# Paired per-case comparison: same cases, both systems.
deltas = [c.score - b.score
for b, c in zip(baseline.by_case, candidate.by_case)]
mean_delta = statistics.mean(deltas)
# Bootstrap CI rather than a t-test: scores are often bounded
# and non-normal, and bootstrap makes no distributional
# assumption.
lo, hi = bootstrap_ci(deltas, confidence=0.95)
# 1. Within noise? Not a signal in either direction.
if abs(mean_delta) < noise_floor:
return Gate.pass_("within measured noise floor")
# 2. Confidently worse on the aggregate?
if hi < 0:
return Gate.block(f"regression {mean_delta:.1%}, CI [{lo:.1%},{hi:.1%}]")
# 3. The check people omit: an aggregate improvement can hide
# a severe regression in one segment or one metric family.
for seg, d in per_segment(deltas).items():
if d.mean < -SEGMENT_THRESHOLD and d.ci_high < 0:
return Gate.block(f"segment '{seg}' regressed {d.mean:.1%}")
# 4. Hard failures are absolute, not statistical. One safety
# failure blocks regardless of the aggregate.
if candidate.safety_failures > 0:
return Gate.block(f"{candidate.safety_failures} safety failures")
return Gate.pass_(f"delta {mean_delta:+.1%}")
Check 3 is the one that distinguishes a real pipeline. A change that improves English by 4 percent and destroys Portuguese by 15 percent shows as a net improvement, and shipping it is how a system quietly becomes bad for a minority of users. Segment-level gating is the mechanism that catches it, and it requires the eval set to be segmented deliberately rather than sampled uniformly.
Check 4 matters too: safety and correctness failures are absolute. A statistical gate that permits one prompt-injection success because the aggregate improved is the wrong gate.
Step 5: where the eval set comes from
The hardest and least-discussed part. A gate is only as good as its cases.
1. PRODUCTION TRAFFIC, sampled and labelled
The best source, because it matches the real distribution.
Sample stratified by query type and by outcome, deliberately
over-sampling failures, because uniform sampling of a system
that is 90% correct spends 90% of the labelling budget
confirming what already works.
2. FAILURE HARVESTING
Every production incident, every thumbs-down, every escalation
becomes a permanent test case. This is the highest-value source
and it is nearly free.
*** The rule: a bug is not fixed until its case is in the
eval set. *** That single policy is what makes the suite
grow in the right direction.
3. SYNTHETIC GENERATION
Model-generated variations for coverage of rare cases.
Cheap, and it drifts toward what the model finds easy, so it
must be human-reviewed before entering the blocking tier.
4. ADVERSARIAL
Prompt injection attempts, jailbreaks, out-of-scope questions,
ambiguous queries. Curated, small, and absolute-gated.
Eval set maintenance is an ongoing cost, not a project. Cases go stale when the product changes, ground truth becomes wrong when the underlying data changes, and a suite nobody prunes accumulates cases that fail for reasons unrelated to quality, which is how teams learn to ignore the gate.
And the contamination problem to name: if the eval set is derived from production traffic and the prompts are tuned against it repeatedly, you are overfitting to the eval set. Hold out a portion that is never used for iteration and only for release decisions, and rotate it periodically.
Step 6: the developer experience, which decides adoption
A gate that is slow, flaky or opaque gets bypassed, and a bypassed gate is worse than no gate because it creates false confidence.
When the gate fails, the PR comment must contain:
Overall: BLOCKED. Regression -6.2% (CI -9.1% to -3.4%)
Noise floor: +/-1.8% (measured over 3 baseline runs)
By metric family:
retrieval recall@5 0.84 -> 0.83 (-1.2%, within noise)
faithfulness 0.91 -> 0.79 (-13.2%) <-- the cause
task success 0.77 -> 0.71 (-7.8%)
By segment:
en -4.1%
pt-BR -14.9% <-- worst
fr -2.2%
5 worst regressions, with links to the full traces:
case_0412 "how do I cancel..." 1.0 -> 0.0 [trace]
...
Compare: [side-by-side diff of baseline vs candidate outputs]
Three properties that matter more than the statistics:
The failure is diagnosable. "Quality dropped 6 percent" is not actionable; "faithfulness dropped 13 percent, worst in pt-BR, here are five cases with traces" is.
The noise floor is shown, so a developer can see why a 1 percent difference was not blocked and trust the gate rather than argue with it.
There is a documented override, with a required reason, that is logged and reviewed. An override path that requires justification is used rarely; one that does not exist gets replaced by disabling the gate.
Step 7: failure modes
Flaky gate (fires on noise)
-> The fastest way to destroy trust. Fix by measuring the noise
floor and gating above it, and by using paired comparison so
between-case variance is removed.
Judge model changes underneath you
-> The provider updates the model and every score shifts. PIN
the judge model version, and treat a judge upgrade as its own
change requiring re-validation against human labels.
Eval set overfitting
-> Held-out set never used for iteration, rotated periodically.
Track the gap between the iteration set and the held-out set:
a widening gap is the signal.
Gate too slow -> bypassed
-> Parallelise aggressively (the work is embarrassingly
parallel), tier it, and keep the blocking tier under 10
minutes. This is a social failure with a technical fix.
Cost of running evals
-> Cache baseline results by (eval_set_version, system_version)
so the baseline is not re-run for every PR against the same
base. Sample the regression tier for draft PRs and run in
full on ready-for-review.
Aggregate improvement hiding a segment collapse
-> Segment-level gating. This is the failure that ships.
Ground truth becomes wrong
-> A refund policy changes and 40 cases now have wrong expected
answers. Version the eval set alongside the product, and
review cases when the underlying data changes.
Step 8: what changes at ten times the scale
At 150 teams and 300 PRs a day:
Eval infrastructure becomes a platform. Shared runners, a case registry, judge model management and cost attribution per team, which is the cost attribution problem again.
Caching becomes essential rather than an optimisation. Baseline results keyed by (eval set version, system version, model version) so identical baselines are computed once per day rather than per PR.
Judge cost forces a hierarchy. Programmatic scoring wherever possible, a small fast model as a first-pass judge with escalation to a larger model only for cases near the decision boundary, which is the same escalation pattern as the LLM gateway.
Case provenance and licensing become real. Production-derived cases contain customer data, so the eval store inherits the same access-control requirements as the permissioned RAG design, and that is usually discovered late.
Production evidence
Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023) documents position bias, verbosity bias and self-enhancement bias in LLM judges, and reports that strong judges agree with human preferences at roughly the rate humans agree with each other, which is the basis for both using judges and validating them.
Ragas and DeepEval are the reference open frameworks for the retrieval and generation metric families, and Ragas's separation of faithfulness, answer relevance, context precision and context recall is the standard decomposition.
OpenAI Evals and Anthropic's evaluation guidance both emphasise task-specific eval sets built from real usage over generic benchmarks, and the practice of turning every production failure into a permanent test case.
Chatbot Arena's use of pairwise comparison with Elo rather than absolute scoring is the strongest available evidence that pairwise is the more reliable elicitation format.
Google's SRE practice of error budgets is the structural analogue for the gating policy: a threshold agreed in advance, enforced automatically, with an explicit and logged override path.
The debate
The case for blocking gates: without enforcement, quality regressions ship. Teams mean to check and do not, and the failure is silent because nobody looks at a dashboard after a prompt tweak.
The case for non-blocking (report-only): LLM evals are noisy, and a gate that blocks on noise gets overridden until it is disabled. Reporting the delta and letting a human decide respects that the measurement is imperfect.
The case for production-only evaluation: offline eval sets drift from real traffic, and online metrics (thumbs-down rate, escalation rate, task completion) are what actually matter. Ship behind a flag, measure in production, roll back on signal.
My position: block on the fast programmatic tier and on measured, segment-aware regressions in the second tier, report-only on the deep tier, and pair all of it with online measurement.
The property I would insist on is that the gate's threshold is above its measured noise floor, which means measuring the floor by running the unchanged baseline several times rather than assuming it. A gate that fires on noise is worse than no gate: it trains a team to override reflexively, and then it catches nothing when it matters. That failure is social rather than technical and it is the most common way these systems die.
The second is paired comparison with the baseline re-run in the same job. Comparing to a stored number from last week conflates a real regression with provider-side drift, and pairing removes between-case variance, which is the dominant term. It costs double the model calls and it roughly quarters the sample size you need, so it is cheaper overall.
And segment-level gating, because the aggregate is where regressions hide. A change that improves the majority language by 4 percent and destroys a minority one by 15 shows as a win, and shipping it is how a system quietly becomes unusable for a subset of users who do not complain loudly enough to appear in the aggregate.
Where I would push back on the framing: offline evaluation is a regression gate, not a quality measurement. It tells you whether you made things worse on cases you already know about. It cannot tell you whether the product is good, and a team that ships on green evals without online measurement is measuring the wrong thing. The offline gate and the production signal are complements, and I would build the failure-harvesting loop from production into the eval set as the mechanism that connects them.
Follow-up Q&A
"How is this different from normal CI?" The system is non-deterministic and the correct answer is often not exactly specified, so the gate cannot be "did the test pass". It has to be "did quality drop by more than noise", which means you have to know what noise is. I would measure the noise floor by running the unchanged baseline three times on the eval set and taking the spread, then gate above it. A gate that fires on differences smaller than its own noise gets overridden reflexively and then catches nothing.
"How many eval cases do you need?" More than teams use. With a binary metric around 80 percent, the standard error is about 4 percent at 100 cases, so you cannot detect a 5-point regression at all. At 500 you can detect about 5 percent. But paired comparison changes this materially: running baseline and candidate on the same cases and comparing per case removes between-case variance, which is the dominant term, so 200 paired cases detect what 800 unpaired ones would need. Which is why I always re-run the baseline in the same job rather than comparing to a stored number.
"Why not compare against last week's stored score?" Because it conflates your change with provider-side drift. Models get updated, batching changes numerics, and the baseline moves for reasons that have nothing to do with the PR. Re-running the baseline in the same job on the same cases with the same judge version isolates the change, and it enables the paired statistics that make small eval sets usable.
"What do you measure?" Two families, and reporting only one is the common mistake. Retrieval and process metrics: recall at k, context precision, whether the right tools were called. And generation metrics: faithfulness, relevance, task success. Both, because an answer can be perfectly faithful to context that did not contain the answer, which scores well on generation while the system failed. A single aggregate score hides which half broke, which is exactly what you need when the gate fires.
"LLM-as-judge is noisy. How do you trust it?" Treat it as a measurement instrument that needs calibration. It has documented biases: position bias, so run both orders and average; verbosity bias, so control for length; self-preference, so use a different model family as judge. Then validate it against human labels on a held-out set and measure agreement. Below about 80 percent agreement it is measuring something other than what you think and gating on it is worse than not gating. And pin the judge model version, because a provider update shifts every score.
"Absolute scores or pairwise?" Pairwise where you can. Asking "is A or B better" gets much higher agreement than "rate this one to five", because absolute scoring requires the judge to hold a stable internal scale across independent calls and it does not. Chatbot Arena's use of pairwise with Elo rather than absolute ratings is the strongest evidence for that.
"The aggregate improved but you blocked. Why?" Segment regression. A change that improves English by four percent and destroys Portuguese by fifteen shows as a net win, and shipping it is how a system quietly becomes unusable for a group who do not complain loudly enough to move the aggregate. So the gate checks per segment as well as overall, which requires the eval set to be deliberately segmented rather than uniformly sampled. Safety failures are also absolute rather than statistical: one prompt-injection success blocks regardless of the aggregate.
"Where do the eval cases come from?" Production traffic, stratified and deliberately over-sampling failures, because uniform sampling of a 90-percent-correct system spends most of the labelling budget confirming what already works. Plus failure harvesting, which is the highest-value and nearly free source: every incident, thumbs-down and escalation becomes a permanent case. The policy I would set is that a bug is not fixed until its case is in the eval set, and that one rule is what makes the suite grow in the right direction.
"What about overfitting to the eval set?" Real, and it happens when prompts are tuned against the same cases repeatedly. Hold out a portion that is never used for iteration and only for release decisions, rotate it periodically, and track the gap between the iteration set and the held-out set. A widening gap is the signal that you are tuning to the test rather than improving the system.
"What if the gate is slow?" It gets bypassed, and a bypassed gate is worse than none because it creates false confidence. That is a social failure with a technical fix: the work is embarrassingly parallel, so run 50 cases concurrently and a 500-case paired run is about 80 seconds rather than 40 minutes. Plus tiering: 30 cases of programmatic scoring on every commit, 500 paired on every PR, 2,000 with full judge scoring nightly and non-blocking.
Common misconceptions
"50 test cases is a gate." At 50 cases the standard error swamps any realistic regression. You are measuring noise and calling it quality.
"Compare to the last stored score." That conflates your change with provider drift. Re-run the baseline in the same job.
"One quality number is enough." Retrieval and generation fail differently, and an aggregate hides which one broke.
"An LLM judge is a metric." It is an instrument with known biases that needs validation against human labels and a pinned version.
"Green evals mean the product is good." Offline evaluation is a regression gate on cases you already know about. Online measurement is what tells you whether it works.
Interview delivery note
Name what makes this different immediately: "This looks like CI and it isn't, because the system is non-deterministic and there's often no single correct answer. So the gate can't be 'did the test pass', it has to be 'did quality drop by more than noise'. Which means the first thing I'd build is a measurement of the noise floor: run the unchanged baseline three times and take the spread. A gate that fires below its own noise gets overridden reflexively and then catches nothing."
Then the statistical decision, with numbers: "And I'd run the baseline in the same job on the same cases, paired. Comparing to a stored score from last week conflates my change with provider drift, and pairing removes between-case variance, which is the dominant term. At 80 percent baseline accuracy, unpaired, 100 cases has a 4 percent standard error so you can't see a 5-point regression at all. Paired, 200 cases does what 800 unpaired would."
Volunteer the failure that actually ships: "The check I'd make sure is there is segment-level. A change that improves English by four percent and destroys Portuguese by fifteen shows as a net win, and shipping it is how a system quietly becomes unusable for a group that doesn't complain loudly enough to move the aggregate."
Show you understand the judge is an instrument: "LLM-as-judge has position bias, verbosity bias and self-preference, all documented. So: run both orders and average, use a different model family as judge, pin the version, and validate agreement against human labels. Below about 80 percent agreement it's measuring something else and gating on it is worse than not gating."
Close on the adoption constraint, because it is what decides whether any of this matters: "and the blocking tier has to stay under ten minutes, because a slow gate gets bypassed and a bypassed gate is worse than none. That's a social failure with a technical fix: the work is embarrassingly parallel."
Further reading
- Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023), for judge biases and human-agreement rates.
- The Ragas documentation, for the faithfulness / answer relevance / context precision / context recall decomposition.
- Anthropic's guidance on building evaluations, and OpenAI Evals, for task-specific suites built from real usage.
- Efron and Tibshirani, An Introduction to the Bootstrap, for confidence intervals without distributional assumptions.
- Beyer et al., Site Reliability Engineering, on error budgets, as the structural analogue for an agreed threshold with a logged override.