Bake time, minimum detectable effect, and the ritual canary

What it is

The arithmetic that says whether a canary can detect anything, and the name for what you have when it cannot.

MINIMUM DETECTABLE EFFECT (MDE)
  The smallest regression your canary configuration can
  distinguish from noise at a chosen confidence and power.
  A function of sample size, baseline variance, and the
  thresholds you chose.

BAKE TIME
  How long the canary runs. It determines sample size, which
  determines MDE. It is therefore not a matter of taste.

THE RITUAL CANARY
  A canary whose MDE is larger than any regression you care
  about. It runs, it passes, everyone feels safer, and it
  cannot detect anything. This is the majority of canaries.

Commonly confused with a duration convention. "We bake for fifteen minutes" is a statement about statistical power, whether or not the team knows it, and fifteen minutes at 5 percent of a low-traffic service detects nothing.

Also commonly confused with the canary's purpose. A canary that cannot detect a 5 percent latency regression can still detect a crash loop, and that is worth having. The distinction is between hard failure detection, which needs almost no samples, and statistical regression detection, which needs a lot.

The problem it solves

Teams choose bake time by feel and then trust a gate that cannot decide anything.

"We canary at 5% for 10 minutes."

Service does 200 requests/second.
  Canary sample: 200 x 0.05 x 600 = 6,000 requests
  Baseline sample: same

What regression can 6,000 vs 6,000 detect?
  For an error rate moving from 0.5% to X, at 95% confidence
  and 80% power, the detectable difference is roughly 0.9
  percentage points.
  -> It can detect 0.5% -> 1.4%.
  -> It CANNOT detect 0.5% -> 1.0%, which is a doubling of
     the error rate.

*** The gate would pass a release that doubled errors. ***

And the arithmetic gets worse quickly at lower traffic:

Service at 20 requests/second, 5% canary, 10 minutes:
  600 requests each side.
  Detectable error-rate difference: roughly 3 percentage
  points, so 0.5% -> 3.5%.
  -> It cannot detect a SEVENFOLD increase in errors.

Mechanics

The sample size formula

For comparing two proportions (error rates, conversion rates):

$$ n = \frac{(z_{\alpha/2} + z_{\beta})^2 \cdot (p_1(1-p_1) + p_2(1-p_2))}{(p_1 - p_2)^2} $$

z(α/2) = 1.96   for 95% confidence (two-sided)
z(β)   = 0.84   for 80% power
        (0.84 -> 80% power; 1.28 -> 90% power)

Rearranged for MDE, which is the direction you actually want:

  MDE ≈ (z_α/2 + z_β) × sqrt(2 × p(1-p) / n)
      ≈ 2.8 × sqrt(2 × p(1-p) / n)

def mde_proportion(baseline_rate, n_per_group,
                   confidence=0.95, power=0.80):
    """Smallest absolute difference in rate detectable with
    n samples per group."""
    z_a = norm.ppf(1 - (1 - confidence) / 2)   # 1.96
    z_b = norm.ppf(power)                       # 0.84
    p = baseline_rate
    return (z_a + z_b) * math.sqrt(2 * p * (1 - p) / n_per_group)

# The table that should be on the wall:
#   baseline 0.5% error rate
#   n=1,000    MDE = 0.79pp   (0.5% -> 1.29%)
#   n=10,000   MDE = 0.25pp   (0.5% -> 0.75%)
#   n=100,000  MDE = 0.08pp   (0.5% -> 0.58%)
#   n=1,000,000 MDE = 0.02pp  (0.5% -> 0.52%)
#
# *** MDE improves as 1/sqrt(n). To halve it, quadruple the
#     sample. ***

The $1/\sqrt{n}$ relationship is the fact to carry: doubling bake time improves MDE by about 40 percent, and detecting a regression half the size takes four times as long.

For latency, the variance matters more than the rate

For a continuous metric like latency, MDE depends on the
metric's own variability:

  MDE ≈ 2.8 × σ × sqrt(2/n)

  expressed as a fraction of the mean:
  MDE% ≈ 2.8 × CV × sqrt(2/n)

  where CV = σ/μ is the coefficient of variation.

A service with a TIGHT latency distribution (CV = 0.3):
  n = 10,000  ->  MDE ≈ 1.2% of the mean

A service with a HEAVY TAIL (CV = 2.0, common when p99 is
20x the median):
  n = 10,000  ->  MDE ≈ 7.9% of the mean
  n = 100,000 ->  MDE ≈ 2.5%

*** The same bake time gives a 6.6x worse MDE on the
    heavy-tailed service. ***

This is why a bake time copied from another team's runbook is usually wrong: their variance is not yours, and variance drives the answer as strongly as sample size does.

And it is why p99 is much harder to gate than p50. A percentile estimate's variance is higher than the mean's, and for the p99 specifically only about 1 percent of samples are near it, so the effective sample size for the tail is a hundredth of the total.

40,000 requests in the canary.
  Estimating p50: effectively informed by all 40,000.
  Estimating p99: informed by roughly the 400 samples near
                  the 99th percentile.
  -> The p99 comparison has ~1% of the statistical power,
     so its MDE is ~10x worse.

Working backwards: bake time from the regression you care about

This is the correct direction and almost nobody does it.

STEP 1: what is the smallest regression that matters?
  "A 5% latency increase" or "error rate doubling from 0.5%
  to 1%". Ask the question; it forces a real answer.

STEP 2: what is your baseline variance?
  Measure it. CV for latency, current rate for errors.

STEP 3: solve for n.
  n = 2 × (2.8 × CV / MDE_target)^2                for latency
  n = 2 × p(1-p) × (2.8 / MDE_target_abs)^2        for rates

STEP 4: convert n to bake time.
  bake = n / (total_rps × canary_share)

WORKED
  Service: 800 rps, latency CV = 1.4, want to detect a 5%
  latency regression at 95%/80%.

  n = 2 × (2.8 × 1.4 / 0.05)^2 = 2 × (78.4)^2 = 12,293
  At 5% canary share: 800 × 0.05 = 40 rps
  bake = 12,293 / 40 = 307 seconds ≈ 5 minutes.

  Feasible. Now for a 2% regression:
  n = 2 × (2.8 × 1.4 / 0.02)^2 = 76,832
  bake = 76,832 / 40 = 1,920 s = 32 minutes.

  And for a 2% regression on the p99, with ~1% effective
  sample: 100x the samples, so ~53 hours.
  *** Gating the p99 on a 2% regression is not achievable
      in a canary. Say so rather than pretending. ***

That last line is the honest output of the exercise, and it is more useful than a bake time: some things cannot be gated in a canary and need a longer-running A/B test or a different detection mechanism.

n = rps × share × duration

To get more samples you can raise share OR duration, and
they are not equivalent:

RAISE DURATION     Blast radius stays small. Slower feedback.
                   Better for a risky change.
RAISE SHARE        Faster feedback, more users exposed to a
                   bad release. Better for a low-risk change
                   where speed matters.

The multi-stage rollout resolves it: start at 1% to catch
catastrophic failures with minimal exposure, then raise the
share once the hard-failure checks have passed, because by
then the risk is lower and you want the samples.

And there is a load-dependent reason to raise the share regardless of statistics: some failures need real concurrency to appear, so a 25 percent stage catches connection pool exhaustion and lock contention that 5 percent never will.

The ritual canary, and how to recognise one

SIGNS YOU HAVE ONE

  The bake time was chosen because it "felt right" or was
  copied from another team.
  Nobody can state the MDE.
  The canary has never failed, or fails only on hard errors.
  Business metrics are in the gate at low traffic.
  Thresholds have been loosened at least once to stop noise.
  Low-traffic services use the same configuration as
  high-traffic ones.

WHAT TO DO ABOUT IT

  a. Compute the MDE and publish it. "This canary can detect
     a 9% latency regression and nothing smaller" is a useful
     sentence, and often a shocking one.
  b. Keep the hard-failure checks regardless. They need
     almost no samples and they catch crash loops, which is
     genuinely valuable.
  c. For services where the statistics do not work, be
     explicit: the canary is a smoke test, and regression
     detection happens elsewhere, via a longer A/B test or
     via monitoring after rollout with a fast rollback.

Step (b) is the nuance. A canary with a useless MDE is not worthless: detecting a process that crashes on startup needs one sample, not ten thousand. The mistake is believing it does more than that.

A worked example: fixing a fleet of canaries

CONTEXT
  40 services, one shared canary configuration: 5% for 15
  minutes, promote above 95% score.

AUDIT: compute MDE per service.

  service        rps    CV    canary n   MDE (latency)
  ------------------------------------------------------
  api-gateway   4,200   0.9      189,000    0.6%
  search        1,100   1.6       49,500    1.6%
  checkout        380   1.2       17,100    2.5%
  inventory       140   2.1        6,300    7.5%
  reporting        18   3.4          810   35.0%   <--
  admin             4   2.8          180   58.0%   <--

FINDINGS
  Two services have a canary that cannot detect a 35% or 58%
  latency regression. Those are rituals.
  Four services have a usable canary, and only two can
  detect a 2% regression.

ACTIONS
  api-gateway, search:  usable as-is. Tighten thresholds,
                        since the MDE supports it.
  checkout, inventory:  extend the bake to 45 minutes and add
                        a 25% stage. New MDE: 1.4% and 3.8%.
                        Acceptable given a checkout regression
                        of under 4% is unlikely to matter.
  reporting, admin:     the statistics do not work at any
                        feasible bake time. Convert to a smoke
                        test: hard-failure checks only, 5
                        minutes, plus feature flags and a
                        documented fast rollback. Publish that
                        these services have NO regression
                        gate, so nobody believes otherwise.

THE FINDING WORTH REPORTING
  The shared configuration created false confidence on
  low-traffic services and left statistical power unused on
  high-traffic ones. A single canary policy across
  heterogeneous traffic is wrong in both directions
  simultaneously.

Production evidence

The sample size formulas are standard two-sample power analysis (Cohen, Statistical Power Analysis for the Behavioral Sciences), and their application to online experiments is documented across the experimentation literature.

Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments (2020) is the definitive practical treatment of MDE, power and sample size in online settings, including the point that most experiments are underpowered and that teams systematically overestimate what they can detect.

Kohavi et al.'s work at Microsoft documents that the majority of proposed changes have small true effects, which is precisely why MDE matters: a gate that can only detect large effects will pass most real regressions.

Netflix's Kayenta documentation notes the need for sufficient sample size and provides configuration for minimum sample thresholds, which is the tooling acknowledging the problem.

Google's experimentation practice, described in the "Overlapping Experiment Infrastructure" paper (Tang et al., KDD 2010), builds on the same power arithmetic and is the reference for why traffic allocation is a scarce resource that must be budgeted.

The debate

The case for computing MDE: without it you do not know whether your gate works, and the common outcome is a gate that passes real regressions while everyone believes it is catching them. False confidence is worse than no gate, because it substitutes for the manual care that would otherwise happen.

The case against the ceremony: most releases are fine, most regressions that matter are large enough to detect, and computing power analysis per service is work that could go into faster rollback instead. A ten-minute canary that catches crashes plus a two-minute rollback may be a better system than a statistically rigorous forty-minute gate.

The case for longer bakes: more samples, better detection, and slow-developing failures have time to appear.

The case for shorter bakes: deployment frequency is itself a reliability property, and a forty-minute gate on every deploy means fewer, larger, riskier deploys.

My position: compute the MDE, publish it per service, and be explicit about which services have a real gate and which have a smoke test.

The single most valuable artifact is the sentence "this canary can detect a 9 percent latency regression and nothing smaller", published next to the pipeline. It is often shocking, it immediately tells you whether the gate is doing what people think, and it costs an afternoon of arithmetic.

The design error I would call out is a shared canary configuration across heterogeneous services. In the audit above it was simultaneously wrong in both directions: false confidence on low-traffic services whose MDE was 35 to 58 percent, and unused statistical power on high-traffic ones whose thresholds could have been much tighter. One policy across services with three orders of magnitude of traffic difference cannot be right.

The nuance I would hold is that a statistically useless canary is not a worthless canary. Hard-failure detection, crash loops, health check failures, an endpoint returning 500 on every request, needs one sample rather than ten thousand. So the right move for a low-traffic service is not to delete the canary but to relabel it a smoke test and to move regression detection to a mechanism that can actually do it, which is usually a longer A/B test or post-rollout monitoring with a fast rollback.

And I would work the arithmetic backwards, from the regression that matters, rather than forwards from a bake time. Asking "what is the smallest regression we would care about" forces a real answer, and then the bake time is derived rather than chosen. Sometimes the derivation says 53 hours, which means that thing cannot be gated in a canary, and saying so is more useful than a bake time that pretends otherwise.

Where I would push back on a longer bake: deployment frequency is a reliability property. A forty-minute gate on every deploy pushes teams toward fewer, larger, riskier releases, and that trade should be made deliberately rather than by adding time until the statistics look good.

Follow-up Q&A

"How do you choose bake time?" Backwards, from the smallest regression that would matter. Ask that question first, because it forces a real answer, then measure the baseline variance, solve for the sample size, and divide by the canary's request rate. For a service at 800 requests per second with a latency coefficient of variation of 1.4, detecting a five percent regression needs about 12,000 samples per side, which at a five percent canary is five minutes. Detecting two percent needs 77,000, which is 32 minutes. The bake time is derived, not chosen.

"What is MDE and why does it matter?" The smallest regression the configuration can distinguish from noise. It matters because a gate whose MDE is larger than any regression you care about will pass real problems while everyone believes it is catching them, and false confidence is worse than no gate because it substitutes for the manual care that would otherwise happen. The most valuable artifact is publishing it: "this canary can detect a nine percent latency regression and nothing smaller" is often a shocking sentence.

"How does MDE scale with sample size?" As one over the square root of n. So doubling the bake time improves MDE by about 40 percent, and detecting a regression half the size takes four times as long. That relationship is the one to carry, because it tells you immediately that you cannot fix a bad MDE by adding a few minutes.

"Why is p99 so much harder to gate than p50?" Because only about one percent of samples are near the 99th percentile, so the effective sample size for the tail is a hundredth of the total. With 40,000 canary requests, the p50 estimate is informed by all of them and the p99 by roughly 400. That is one percent of the statistical power, so the MDE is about ten times worse. Gating the p99 on a small regression is frequently not achievable in a canary at all, and saying so is better than pretending.

"What's a ritual canary?" One whose MDE exceeds any regression you care about. It runs, it passes, everyone feels safer, and it cannot detect anything. The signs: nobody can state the MDE, the bake time was copied from another team, the canary has never failed except on hard errors, and thresholds have been loosened at least once to stop noise. In an audit I did, two of forty services had canaries that could not detect a 35 and a 58 percent latency regression respectively.

"So delete those canaries?" No, relabel them. A statistically useless canary still detects hard failures, and detecting a process that crashes on startup needs one sample rather than ten thousand. So keep the hard-failure checks, call it a smoke test, publish that those services have no regression gate so nobody believes otherwise, and move regression detection to something that can do it: a longer A/B test, or post-rollout monitoring with a fast rollback and feature flags.

"Raise the bake time or the canary share?" They give the same samples and different risk. Raising duration keeps the blast radius small and slows feedback, which suits a risky change. Raising share speeds feedback and exposes more users, which suits a low-risk one. The multi-stage rollout resolves it: one percent first to catch catastrophic failures with minimal exposure, then raise the share once the hard checks pass. And there is a non-statistical reason to raise share anyway: connection pool exhaustion and lock contention need real concurrency, so a 25 percent stage catches what five percent never will.

"What's wrong with one canary policy for all services?" It is wrong in both directions simultaneously. In the audit, the shared five-percent-for-fifteen-minutes configuration gave false confidence on low-traffic services, whose MDE was tens of percent, and left statistical power unused on high-traffic ones, whose thresholds could have been far tighter. One policy across services spanning three orders of magnitude of traffic cannot be right for any of them.

"Isn't a longer bake always safer?" No, because deployment frequency is itself a reliability property. A forty-minute gate on every deploy pushes teams toward fewer, larger, riskier releases, which is a worse outcome than a shorter gate plus fast rollback. That trade should be made deliberately, rather than by adding minutes until the statistics look acceptable.

Common misconceptions

"Bake time is a convention." It determines sample size, which determines what the gate can detect. It is a statistical parameter whether or not the team treats it as one.

"A canary that passes means the release is safe." It means no regression larger than the MDE was detected. If the MDE is 35 percent, that is almost no information.

"Doubling the bake time doubles the sensitivity." MDE improves as one over root n, so doubling the time improves it by about 40 percent.

"You can gate the p99 like the p50." Only about one percent of samples inform the p99, so its MDE is roughly ten times worse at the same bake time.

"A useless canary should be deleted." It still catches crash loops, which needs one sample. Relabel it a smoke test and move regression detection elsewhere.

Interview delivery note

Reframe the question from duration to detection, because that is the whole point: "Bake time isn't a convention, it's a statistical parameter. It sets the sample size, which sets the minimum detectable effect. So I'd work backwards: what's the smallest regression that would matter, what's the baseline variance, solve for n, divide by the canary's request rate."

Give the arithmetic concretely: "At 800 requests a second with a latency coefficient of variation of 1.4, detecting a five percent regression needs about twelve thousand samples per side, which at a five percent canary is five minutes. Detecting two percent needs seventy-seven thousand, so thirty-two minutes. And MDE improves as one over root n, so detecting a regression half the size takes four times as long."

Volunteer the p99 problem, because it is the one people assume away: "And the p99 is much harder than the p50, because only about one percent of samples are near it. With forty thousand canary requests, the p50 is informed by all of them and the p99 by about four hundred. That's one percent of the power, so ten times the MDE. Gating the p99 on a small regression is often just not achievable in a canary, and I'd say so rather than pretend."

Name the failure mode, which is the memorable part: "The failure mode is the ritual canary: one whose MDE is bigger than anything you care about. It runs, it passes, everyone feels safer, and it can't detect anything. When I audited forty services against a shared five-percent-fifteen-minute policy, two had canaries that couldn't detect a thirty-five and a fifty-eight percent latency regression."

Close with the nuance and the artifact: "But I wouldn't delete those. A useless canary still catches crash loops, and that needs one sample rather than ten thousand. I'd relabel it a smoke test and publish the MDE per service, because 'this canary can detect a nine percent regression and nothing smaller' is the single most useful sentence you can put next to a pipeline."

The Staff & Team Lead Interview Reference