Regression gates in CI for LLM systems
What it is
A regression gate is an automated check that blocks a change when a measured quality metric drops. For deterministic software this is a test suite. For an LLM system the output is non-deterministic and unbounded, so the gate is statistical, and a statistical gate has a property test suites do not: it has a noise floor.
The four things a usable gate needs:
AN EVAL SET fixed, versioned, representative, and
large enough for the effect size you care
about.
A METRIC computed per example, aggregable, and
validated against human judgment if a
model computes it.
A THRESHOLD above the noise floor, derived rather than
chosen.
A NOISE FLOOR measured, by running the same eval against
the same system repeatedly.
The rule that governs everything else: measure the noise floor before gating on the metric. A gate that fires below its own noise gets overridden reflexively, and a gate that is always overridden catches nothing.
What this is confused with: evaluation as a quality measurement. An offline eval on a fixed set tells you whether you got worse at the things in that set. It does not tell you how good the system is, and treating an eval score as a quality number is how a system with an 87 percent eval score ships a bad user experience.
Also confused: a gate and a benchmark. A benchmark compares systems. A gate compares this change to the previous state of this system, which is a much easier statistical problem and the only one you can run per pull request.
The problem it solves
Without a gate, LLM regressions ship, because nothing fails.
A prompt change to improve tone.
unit tests pass (they test the parsing)
integration tests pass (the API returns 200)
manual spot check looks fine on the three examples
the author tried
Shipped. Two weeks later, support notices that the assistant
has stopped asking clarifying questions on ambiguous
requests, which was the behaviour the previous prompt's third
paragraph produced, and which nobody was testing.
The change was correct for what it aimed at and it broke
something adjacent, which is the characteristic failure of
prompt and model changes.
And the gate that is worse than no gate:
An eval suite of 50 examples, pass rate 86%, gate set at
"must not drop below 85%".
run 1 86%
run 2 82% -> BLOCKED, no code change
run 3 88%
run 4 84% -> BLOCKED, no code change
With 50 examples at p = 0.86, the standard deviation of the
pass rate is sqrt(0.86 x 0.14 / 50) = 4.9 percentage points.
So a 1-point threshold sits well inside one standard
deviation, and the gate fires on nothing.
Within three weeks the team is overriding it by default, and
when a real 9-point regression arrives it is overridden too.
The arithmetic is the whole argument, and it takes ten minutes to do before building anything.
Mechanics
Sizing the eval set
The set size is determined by the effect you want to detect, and it is a standard proportion calculation.
To detect a drop of d in a pass rate around p, with
conventional power:
n ~= 16 x p(1 - p) / d^2 (per arm; here the "arms"
are before and after)
p = 0.85, detect d = 0.05 (5 points):
n = 16 x 0.1275 / 0.0025 = 816
p = 0.85, detect d = 0.10:
n = 16 x 0.1275 / 0.01 = 204
p = 0.85, detect d = 0.02:
n = 16 x 0.1275 / 0.0004 = 5,100
So: ~200 examples detects a 10-point regression, ~800
detects 5 points, and detecting 2 points needs thousands.
A 50-example suite can reliably detect roughly a 20-point
regression, which is a regression you would have noticed
anyway.
This calculation decides whether your suite can do the job, and most suites are built by collecting examples until someone gets bored, which lands around 50.
Measuring the noise floor
Run the ENTIRE suite N times against an UNCHANGED system.
10 runs, same model version, same prompt, same eval set:
88.1, 86.4, 87.9, 85.2, 87.0, 88.6, 86.1, 87.3, 85.9, 87.5
mean 87.0, sd 1.1
The floor is ~1.1 points of sd, so ~2.2 points at two
standard deviations.
A gate threshold of "must not drop more than 1 point" fires
on noise about a third of the time.
A threshold of 3 points is above the floor and detectable.
SOURCES OF THE NOISE, and they do not all go away at
temperature 0:
- sampling, obviously, if temperature > 0
- batching and hardware non-determinism in the serving
stack, which affects even greedy decoding
- mixture-of-experts routing
- provider-side model updates between runs
- an LLM judge's own variance, which compounds
TEMPERATURE 0 REDUCES AND DOES NOT ELIMINATE
NON-DETERMINISM. Do not assume determinism; measure it.
Choosing metrics, hardest-first
Prefer, in order, because each is cheaper and more reliable
than the next:
1. EXACT MATCH / STRUCTURAL ASSERTION
Did it produce valid JSON against the schema? Did it call
the right tool with the right arguments? Did it extract
the right entity? Is the citation a real document id?
Deterministic, free, and it covers more than people
expect.
2. PROGRAMMATIC METRIC
Retrieval recall@k against known-relevant documents. Token
count. Latency. Refusal rate on a safety set. All
computable without a model.
3. LLM-AS-JUDGE
For things only a reader can assess: helpfulness,
faithfulness to the source, tone.
AND IT MUST BE VALIDATED: measure the judge's agreement
with human labels on a sample before you trust it, and
re-measure when you change the judge model.
An unvalidated judge is a metric with unknown bias and
unknown variance, gating your releases.
4. HUMAN REVIEW
On a sample, periodically, not per pull request. This is
what calibrates 3.
Pushing as much as possible into layers 1 and 2 is the single biggest improvement to a gate, because those layers have no noise floor at all: a schema either validates or it does not.
The gate structure
TIER 1: PER PULL REQUEST, fast, hard-fail
- structural validity: schema, tool-call shape, citation
ids resolve
- safety set: must refuse, 100% required, pass/fail
- PII leakage set: 0 leaks required, pass/fail
- a smoke subset of the golden set, ~50 examples, gated
loosely (a large drop only)
Runtime target: under 5 minutes. Cost: small.
TIER 2: ON MERGE OR NIGHTLY, full, soft-fail
- the full golden set (800+), gated at a threshold derived
from the measured noise floor
- the regression set: every past production failure,
accumulated. This one is gated HARD at 100%, because a
previously fixed bug reappearing is unambiguous.
- retrieval metrics, latency, cost per request
Runtime: tens of minutes. Cost: budgeted.
TIER 3: BEFORE A MODEL OR MAJOR PROMPT CHANGE
- human review of a stratified sample
- an A/B or shadow comparison on real traffic
Because an offline eval cannot tell you about the
distribution of real inputs, only about the distribution in
your set.
THE HARD GATES ARE THE PASS/FAIL ONES.
Safety, PII, schema validity and the regression set are
binary and are gated at 100%. The statistical metrics are
gated at a threshold above the noise. Conflating the two is
how a safety check ends up with a tolerance.
The regression set gated at 100 percent is the highest-value single component, because it grows with every production failure and every entry is a bug you have already paid for once.
The eval set itself
THREE SETS, with different jobs:
GOLDEN curated, representative of real usage,
stratified across the input distribution you
care about (intent, language, length, difficulty).
Sampled from production, labelled, versioned.
This is the one whose size the power
calculation determines.
REGRESSION every production failure, added when fixed.
Grows monotonically. Gated at 100%.
This is the cheapest set to build because the
examples arrive on their own.
ADVERSARIAL prompt injection, jailbreaks, edge-case inputs,
inputs designed to elicit the failure mode you
are afraid of. Gated as pass/fail on the
safety-relevant subset.
VERSION ALL THREE, in git, with the labels. An eval set that
changes silently makes every historical comparison
meaningless, and the temptation to "fix" a mislabelled example
after a bad run is exactly the failure.
Pinning, cost and the things that move under you
PIN THE MODEL VERSION. A provider-side update to a model
alias changes your baseline overnight, and the symptom is a
gate firing on a change that did not touch anything. Pin to
a dated snapshot where the provider offers one, and treat a
version bump as a deliberate change with its own tier-3
review.
BUDGET THE COST. 800 examples x 2 calls (generation plus
judge) is 1,600 calls per full run. At a nightly cadence
that is ~48,000 calls a month, which is a real number that
should be on a dashboard.
Reduce it by: a cheaper judge model where the judging task
is easy (and validated separately), caching generations for
unchanged prompts, and running tier 2 on merge rather than
per PR.
RECORD EVERYTHING. Every run stores the model version, prompt
version, eval set version, per-example outputs and the
aggregate. Without per-example outputs you cannot diff two
runs, and diffing two runs is how you find out WHICH
examples regressed, which is the only actionable output.
Storing per-example outputs is what turns "the score dropped 4 points" into "these 31 examples flipped, and 22 of them are multi-turn", and the second statement is the one someone can act on.
A worked example: a gate that blocked everything, then nothing
A support assistant: retrieval over a documentation corpus, a generation step, and a citation requirement. An eval suite existed and was distrusted.
The starting state:
Eval set: 60 examples, collected ad hoc over a year.
Metric: an LLM judge scoring "was this a good answer" 1-5,
averaged.
Gate: block if the mean drops more than 0.1.
Behaviour over three months:
gate fired on 34 of 51 pull requests
of those 34, 31 were overridden
of the 3 that were not overridden, 2 were later found to
be noise
1 real regression was caught
And separately, two real regressions shipped without the
gate firing.
Thirty-one overrides out of thirty-four firings is a gate that has trained everyone to ignore it, and the two regressions that shipped anyway are the predictable consequence.
The measurement, done before changing anything:
NOISE FLOOR: ran the unchanged system through the suite 12
times.
mean judge score 3.81, sd 0.14
The gate threshold was 0.1, which is 0.7 standard
deviations. Under the null it fires roughly half the time,
which matches the observed 34 of 51.
POWER: with 60 examples, the detectable effect at
conventional power is roughly 0.4 on a 1-5 scale, which is a
change nobody would need a gate to notice.
JUDGE VALIDATION, which had never been done:
200 examples labelled by two humans, adjudicated.
Judge-human agreement (Cohen's kappa): 0.31.
That is weak agreement, and it means the gate was
substantially measuring the judge's idiosyncrasies.
Broken down: the judge agreed well on "did it answer the
question" (kappa 0.68) and badly on "was the tone
appropriate" and "was it concise", which it had been
asked to fold into one score.
A composite judge score with kappa 0.31 was the metric gating every release, and the decomposition shows why: it was averaging one thing the judge could assess with two it could not.
The rebuild:
1. SPLIT THE METRIC into what each layer can measure.
STRUCTURAL (tier 1, deterministic, 100% required):
- every citation id resolves to a real document
- the response parses against the schema
- no PII from the retrieved context appears outside a
quoted citation
These caught things nobody had been checking. On the
first run, 4% of responses cited a document id that did
not exist.
PROGRAMMATIC (tier 2):
- retrieval recall@5 against known-relevant docs
- answer contains at least one citation when the
question is answerable from the corpus
- refusal rate on the 40-example unanswerable set
JUDGE (tier 2), NARROWED to the single dimension the
judge could actually assess:
- "does the response answer the question, given this
source?" binary, kappa 0.68
Tone and concision were removed from the gate and moved
to periodic human review, because they were not
measurable at the required reliability.
2. GROW THE SET.
Golden set: 60 -> 840, sampled from production and
stratified by intent, question type, language and
whether the answer is in the corpus.
Power calculation: at p = 0.85 and 840 examples, the
detectable effect is about 5 points, which was the
target.
Regression set: seeded with the 11 known production
failures, and it now grows by 2 to 5 per month.
Adversarial set: 60 prompt-injection and jailbreak
attempts, gated pass/fail.
3. RE-MEASURE THE NOISE FLOOR on the new metric and set.
12 runs: pass rate 84.6%, sd 1.2 points.
Gate threshold set at 3 points (2.5 sd), derived rather
than chosen.
4. TIERED GATES.
PR: structural + safety + adversarial, hard, 3 min
merge: full suite, gate at -3 points, plus regression
set at 100%
model
change: human review of a stratified 100-example sample,
plus a one-week shadow comparison
5. PINNED the model to a dated snapshot, and treated the
provider's next alias update as a tier-3 change.
Splitting the metric by what each layer can reliably measure is the structural fix, and the immediate finding, that 4 percent of responses cited a non-existent document id, was a deterministic check nobody had written because everyone was looking at a judge score.
Six months later:
before after
gate firings per 50 PRs 34 6
overrides 31 1
real regressions caught 1 9
real regressions shipped 2 0
judge-human agreement (kappa) 0.31 0.68
eval set size 60 840 + 23
regression + 60
adversarial
tier-1 runtime n/a 3 min
tier-2 cost n/a ~$180/month
Firings fell from 34 to 6 and real catches rose from 1 to 9, which is the signature of a gate that moved above its noise floor: it fires less and means more.
One thing that was tried and reverted:
An attempt to gate per pull request on the full suite, for
faster feedback.
Runtime: 22 minutes. Cost: ~$6 per run, ~$300/month at the
observed PR rate.
The real problem was not cost, it was that a 22-minute gate
on a 3-point threshold produced a queue and people started
batching changes to amortise it, which made attributing a
regression to a change harder, which was the opposite of the
goal.
Reverted to the tiered arrangement. The lesson recorded: gate
latency changes behaviour, and a slow gate on a statistical
metric encourages exactly the batching that makes the metric
uninterpretable.
Production evidence
OpenAI Evals, Anthropic's evaluation guidance, and frameworks such as Braintrust, LangSmith, Promptfoo and DeepEval all implement the same structure: a versioned dataset, per-example scorers, aggregate comparison against a baseline, and per-example diffing between runs. Their convergence on storing per-example outputs is because run-to-run diffing is the only actionable output.
LLM-as-judge validation against human labels is standard practice in the evaluation literature, including the widely cited MT-Bench and Chatbot Arena work, which reports judge-human agreement rates and documents known judge biases: position bias, verbosity bias and self-preference. An unvalidated judge is a metric with unknown bias, which is the basis for requiring an agreement measurement before gating.
Non-determinism at temperature 0 is documented behaviour in production serving stacks, arising from batching, floating-point non-associativity on parallel hardware, and mixture-of-experts routing. It is why the noise floor must be measured rather than assumed away.
Provider model aliases updating underneath applications is the documented reason providers offer dated snapshot identifiers, and pinning to one is standard practice precisely so that a baseline does not move without a deliberate change.
Sample-size calculations for proportions are standard statistics, and the approximation used here (roughly 16 p(1-p)/d² per arm at conventional power) is the standard rule of thumb; it is the same calculation as the minimum detectable effect used for canary analysis and for A/B tests.
Regression suites seeded from production failures are ordinary software engineering practice, and they transfer directly: every fixed bug becomes a permanent test, and for LLM systems it is the cheapest set to build because the examples arrive on their own.
The debate
Should you gate on LLM outputs at all? Yes, and mostly on the deterministic layers. Structural validity, citation resolution, schema conformance, safety refusals and PII checks have no noise floor and catch a surprising amount: in one system, 4 percent of responses cited a document id that did not exist, which no judge score would have surfaced. The statistical gate is the smaller and more fragile part.
Is an eval score a measure of quality? No. It measures whether you got worse at the things in your set, and treating it as a quality number is how a system with a good score ships a bad experience. The honest framing is that offline evaluation is a regression gate and online measurement is the quality measurement, and the correlation between offline and online deltas is something to measure rather than assume.
Is LLM-as-judge acceptable? For dimensions a human reader can assess and a program cannot, yes, after validating agreement with human labels and on a narrowed question. A composite "was this good" score is usually unvalidatable, and in one case its agreement with humans was kappa 0.31 while a single decomposed dimension reached 0.68. Narrow the question, measure the agreement, and re-measure when the judge model changes.
How big should the eval set be? As big as the effect you want to detect requires, which is a calculation rather than a judgement: roughly 200 examples for a 10-point regression, 800 for 5 points, thousands for 2. A 50-example suite detects roughly a 20-point regression, which is one you would have noticed anyway, and that is why most ad hoc suites cannot do the job they were built for.
Hard gate or soft gate? Both, split by measurability. Safety, PII, schema validity and the regression set are binary and gated at 100 percent; statistical metrics are gated at a threshold above the measured noise. Conflating them produces a safety check with a tolerance, which is not a safety check.
Should the gate run per pull request? The fast deterministic tier, yes. The full statistical suite, no, because a 22-minute gate produces batching, and batching destroys the attribution that makes a regression signal useful. Gate latency changes behaviour, and a slow gate on a noisy metric encourages exactly the behaviour that makes the metric uninterpretable.
Follow-up Q&A
"Why can't you test an LLM system like normal software?"
Because the output is non-deterministic and unbounded, so you cannot assert equality, and the aggregate metric you assert on instead has a noise floor that a test suite does not. That changes the design: you need a fixed versioned eval set sized for the effect you want to detect, a metric validated against human judgment if a model computes it, and a threshold derived from a measured noise floor rather than chosen. A gate that fires below its own noise gets overridden reflexively, and then it catches nothing.
"How do you measure the noise floor?"
Run the entire suite repeatedly against an unchanged system, same model version, same prompt, same set, and take the standard deviation of the aggregate. Ten to twelve runs is usually enough. In one case that gave a standard deviation of 1.2 points, so a threshold of 3 points sat at two and a half standard deviations and was detectable; the previous threshold had been 0.7 standard deviations and fired on about half of all pull requests. Note that temperature zero reduces but does not eliminate non-determinism, because batching, floating-point non-associativity on parallel hardware and expert routing all contribute.
"How large does the eval set need to be?"
It follows from the effect size: roughly 16 times p times one-minus-p divided by the square of the drop you want to detect. At a pass rate around 85 percent, that is about 200 examples for a 10-point regression, about 800 for 5 points, and several thousand for 2 points. A 50-example suite can reliably detect roughly a 20-point regression, which you would have noticed without a gate, and that is why sets assembled until someone got bored do not work.
"When is LLM-as-judge acceptable?"
For a narrow dimension a human reader can assess and a program cannot, after measuring the judge's agreement with human labels. In one system a composite "was this a good answer" score had a Cohen's kappa of 0.31 against adjudicated human labels, so the gate was substantially measuring the judge's idiosyncrasies. Decomposed, the judge agreed well on "does this answer the question given this source" (0.68) and badly on tone and concision, so the first became the gated metric and the other two moved to periodic human review. Re-measure agreement whenever the judge model changes.
"What should be a hard gate versus a statistical one?"
Anything binary is a hard gate at 100 percent: safety refusals, PII leakage, schema validity, citation ids resolving, and the regression set of previously fixed production failures. Anything aggregate is gated at a threshold above the measured noise floor. Conflating the two produces a safety check with a tolerance, which is not a safety check. And pushing as much as possible into the deterministic layer is the biggest single improvement available, because those checks have no noise floor: in one system the first deterministic run found that 4 percent of responses cited a document id that did not exist.
"Why store per-example outputs?"
Because "the score dropped 4 points" is not actionable and "these 31 examples flipped, and 22 of them are multi-turn" is. Diffing two runs at the example level is the only output anyone can work from, and it requires storing the model version, prompt version, eval set version and every generation. It is also what lets you tell a real regression from a noise excursion: a noise excursion flips a scattered set of borderline examples, and a real regression flips a cluster with something in common.
"How do you keep the cost of a gate reasonable?"
Tier it. A fast deterministic tier per pull request in a few minutes, the full statistical suite on merge or nightly, and human review plus a shadow comparison only before a model or major prompt change. Use a cheaper judge model where the judging task is easy and validate it separately, cache generations for unchanged prompts, and put the monthly call count on a dashboard. One team's full nightly suite ran about 1,600 calls per run at roughly $180 a month, and an attempt to run it per pull request cost more in changed behaviour than in money: a 22-minute gate produced batching, and batching destroyed the attribution the gate existed to provide.
Common misconceptions
"An eval score measures quality." It measures whether you got worse at the things in your set. Online measurement is the quality measurement, and the correlation between the two is something to measure.
"Temperature zero makes it deterministic." Batching, floating-point non-associativity and expert routing produce variance even under greedy decoding. Measure the floor rather than assuming it away.
"Fifty examples is a reasonable eval suite." It detects roughly a 20-point regression. Detecting 5 points needs around 800.
"An LLM judge is good enough." Until you have measured its agreement with human labels on the specific question you are asking, it is a metric with unknown bias and unknown variance gating your releases.
"Tighter thresholds catch more." Below the noise floor a tighter threshold catches noise, gets overridden by default, and then misses the real regression too.
"Gate everything on every pull request." A slow statistical gate produces batching, and batching destroys the change attribution that makes the signal useful.
Interview delivery note
Say this verbatim: "Measure the noise floor before you set the threshold. We had a 60-example suite with a 0.1 threshold on a judge score whose standard deviation was 0.14, so it fired on roughly half of all pull requests, was overridden thirty-one times out of thirty-four, and missed two real regressions that shipped anyway." It states the rule and gives the failure it prevents with numbers.
The senior-versus-staff separator is validating the judge before trusting the gate. A senior engineer builds an eval suite with an LLM judge. A staff engineer labels a couple of hundred examples with humans, measures agreement, finds a Cohen's kappa of 0.31 on the composite score, decomposes it to discover the judge agrees at 0.68 on "does this answer the question" and poorly on tone and concision, and moves the unmeasurable dimensions out of the gate entirely. Gating on an unvalidated judge is gating on an instrument with unknown bias.
The second signal is pushing work down into the deterministic layers. Saying "we moved citation resolution, schema validity and PII checks into a hard tier-one gate, and the first run found that four percent of responses cited a document id that did not exist" shows you know that the parts of an LLM system's output which can be checked exactly have no noise floor, and that they catch things no aggregate score surfaces.
Further reading
- OpenAI Evals and the documentation for Braintrust, LangSmith, Promptfoo or DeepEval, for the versioned dataset plus per-example scorer plus run-diff structure.
- The MT-Bench and Chatbot Arena papers, for judge-human agreement measurement and the documented judge biases (position, verbosity, self-preference).
- Standard sample-size calculations for proportions, which are the same arithmetic as the minimum detectable effect used for canary analysis.
- The bake time and minimum detectable effect page, for the same statistical reasoning applied to deployments, and experiment analysis pitfalls for peeking, which applies equally to repeatedly re-running an eval until it passes.