Canary vs A/B testing

What it is

Both split traffic between two versions of a system. They are otherwise different in question, horizon, statistics, decision rule and owner.

A canary asks "is this version safe?" It compares operational metrics (error rate, latency percentiles, saturation) between a small slice of production traffic on the new version and a contemporaneous baseline. The horizon is minutes to hours, the decision is asymmetric (any credible regression means roll back), and it is run by the deployment system.

An A/B test asks "is this change better?" It compares product metrics (conversion, engagement, revenue per user) between randomised user cohorts. The horizon is days to weeks, it requires a pre-registered hypothesis and a power analysis, and it is run by the experimentation platform.

They share one piece of machinery, a traffic splitter, and nothing else. The conflation is not academic: it produces a specific, common and expensive failure, which is a team declaring a feature a winner from a two-day canary readout with no power analysis, on a variant that also happened to be 30 milliseconds slower.

The problem it solves

If you only have canaries, you ship changes that are safe and possibly useless, because nothing measured whether users were better off. If you only have A/B tests, you expose a statistically meaningful share of users to a variant for days before anyone notices it is broken, because product metrics move slowly and operational regressions do not.

The correct arrangement is a pipeline: every A/B variant rides through a canary first. Safety gates before measurement begins. Canary the deploy, then ramp the experiment. If you say only one sentence about this in an interview, say that one.

Mechanics

The differences that follow from the question

The unit of assignment row is the one that quietly ruins experiments. Canaries can legitimately split per request, because operational metrics are per request. Experiments must split per user, because a user flipping between variants mid-session contaminates the measurement and, worse, experiences bugs that neither variant has on its own. Consistent cohorting means hashing a stable user ID into a bucket, not calling a random number generator per request.

Automated canary analysis, concretely

The comparison must be against a contemporaneous baseline cohort of the same size, not against the whole fleet and not against last week.

Against the whole fleet: the fleet has different scale, so percentiles are not comparable, and averaging across 200 instances hides what 2 instances are doing. Against last week: traffic mix differs by day, time and marketing calendar, so you are measuring the day of the week.

So a canary deployment provisions three things: the baseline cohort (old version, freshly started, same instance count as the canary), the canary cohort (new version), and the untouched production fleet. Freshly started matters: comparing a warm fleet against a cold canary attributes JIT warmup and cold caches to the change.

Kayenta-style scoring, which is the model most tools follow:

For each metric in the config:
  collect canary series and baseline series over the analysis window
  classify: Pass / High / Low / Nodata
    (Mann-Whitney U or a similar non-parametric test, with a configured
     tolerance so a trivially different distribution is not flagged)
Aggregate:
  score = weighted fraction of metrics classified Pass
  score >= 95  -> promote to the next step
  75 <= score < 95 -> hold, notify a human
  score < 75   -> roll back automatically

The metric set should be the SLIs (error rate, latency percentiles), saturation signals (CPU, memory, GC pause time, thread pool queue depth) and a small number of business guardrails (checkout starts, search result clicks). Do not include fifty metrics: with a per-metric false positive rate, fifty metrics guarantee a flagged metric on every deploy, and the team learns to override the gate.

Bake time is half the design

A canary schedule is percentage multiplied by duration:

1%   for 30 minutes    ->  smoke, obvious breakage
5%   for 1 hour        ->  statistical signal on error rate
25%  for 2 hours       ->  saturation, GC behaviour, cache warmth
100% (soak overnight)  ->  cron paths, memory leaks, daily batch interactions

Memory leaks, cache warmup effects, connection pool exhaustion and anything triggered by an hourly or daily cron do not appear in a 10-minute window at any traffic percentage. Percentage buys statistical power; duration buys coverage of slow-developing failure modes. Both are needed and they are not interchangeable.

A worked example: the minimum detectable effect

This is the calculation that turns a canary from a ritual into an instrument, and it is the answer to "design a canary for a payments service doing 200 QPS".

Baseline error rate is 0.1 percent. You want to detect a doubling to 0.2 percent. For a two-proportion test at $\alpha = 0.05$ and 80 percent power:

$$n \approx \frac{(z_{\alpha/2} + z_\beta)^2 \left[p_1(1-p_1) + p_2(1-p_2)\right]}{(p_1 - p_2)^2}$$

$$n \approx \frac{(1.96 + 0.84)^2 \left[0.000999 + 0.001996\right]}{(0.001)^2} = \frac{7.84 \times 0.002995}{10^{-6}} \approx 23{,}500 \text{ per arm}$$

At 200 QPS, a 1 percent canary receives 2 requests per second. 23,500 requests takes 3.3 hours. So a 1 percent canary baked for 30 minutes cannot detect a doubling of the error rate. It is not a weak test, it is not a test at all: it will detect a catastrophic failure (50 percent errors need only a handful of requests) and will pass a doubling of a rare failure with high probability.

That is the honest answer, and then the mitigations:

1. Raise the canary percentage. At 25 percent, 23,500 requests takes about 8 minutes. For a payments service the exposure cost of 25 percent is high, so this is a real tradeoff and not a free fix.
2. Compare distributions, not just rates. Latency percentiles converge much faster than a rare binary outcome, and many payment regressions show up as latency before they show up as errors.
3. Shadow traffic first. Mirror 100 percent of traffic to the new version with responses discarded, which gives full-volume signal at zero user exposure. It requires that side effects are suppressed, which for payments is the whole difficulty (see the follow-up below).
4. Synthetic transactions. Drive a known-good payment scenario against the canary at high rate, which manufactures a denominator.
5. Bake longer and accept the risk consciously, with an explicit statement of what you cannot detect. Writing down "this canary detects a 10x error regression within 5 minutes and cannot detect a 2x regression within 3 hours" is an honest artifact, and it is the sort of thing that makes a design review go well.

The general form to state in an interview: know your minimum detectable effect, or your canary is a ritual.

Production evidence

Netflix's Kayenta, the automated canary analysis service integrated with Spinnaker, is the reference implementation of the scoring model above. Netflix's engineering blog describes the design goals directly, including comparing against a contemporaneous baseline rather than historical data, and Kayenta is open source, so the metric classification logic is readable.

Argo Rollouts (CNCF) implements the same pattern in Kubernetes with AnalysisTemplate resources that query Prometheus, Datadog or CloudWatch between steps of a canary and abort the rollout on failure. Flagger does the same for service meshes. Both being widely deployed independent implementations of the same idea is good evidence that it is settled practice.

Microsoft's ring-based deployment model (validation ring, internal users, early adopters, broad, world) is the same blast-radius idea applied to client software where you cannot shift traffic percentages, and it is documented in their DevOps guidance.

On the experimentation side, Microsoft's ExP platform, Netflix's experimentation writing and Airbnb's published work on interleaving and experiment analysis all describe the separation explicitly: the deployment system gates safety and the experimentation platform measures effect. Kohavi, Tang and Xu's Trustworthy Online Controlled Experiments is the canonical text and is where the pitfalls below are catalogued.

The debate

The case for merging them into one system is real: they share a traffic splitter, a cohorting mechanism and a metrics pipeline, and building two of everything is expensive. Several companies do run one platform with two modes.

The case against merging, which I hold: the stopping rules are incompatible. A canary must stop early on evidence of harm; that is its purpose. An experiment that stops early on a favourable interim result is committing the peeking fallacy, and its p-value is meaningless. If both live in one tool with one readout, someone will apply canary reasoning to an experiment, because the canary reasoning is the intuitive one. That is exactly the failure this page exists to prevent.

My position: share the traffic-splitting infrastructure, separate the analysis, the ownership and the vocabulary. Concretely: the deploy pipeline owns percentage ramps and automated rollback on SLIs; the experimentation platform owns user-level assignment, power analysis and readouts; a variant is only eligible for experiment ramp-up after it has passed canary. Different dashboards, different words, different people accountable.

Canary analysis is the wrong tool when traffic is too low for any signal (use shadow traffic and synthetics), when the change is not traffic-serving (a batch job, a schema migration), or when the risk is data corruption rather than request failure, because canaries do not detect slow-burn corruption: by the time you notice, the bad version has been writing for hours and rolling back the code does not roll back the data.

Follow-up Q&A

"A PM says 'the canary shows the feature is winning'. Correct them." Kindly and concretely: "The canary tells us the new version is not breaking anything, which is great news and a different question from whether it is better. Canary metrics are error rate and latency over about an hour, with a rule that says roll back on any regression. To claim a lift on conversion we need a user-level randomised experiment with a pre-registered metric and enough traffic for the effect size we care about, which for a 1 percent lift on this surface is about two weeks. The canary was the safety gate; let me get the experiment set up so we can answer the question you are actually asking." Naming what the canary did prove keeps it collaborative rather than corrective.

"Shadow-test a rewrite of a service that sends emails. Walk the side-effect containment." The mirrored traffic must not send email. Four layers, and I would use all of them: route the shadow deployment to a sandbox SMTP or provider sandbox key so nothing can leave; set a shadow flag in the mirrored request context and make the email client a no-op when it is set; use separate credentials for the shadow deployment so its production email API key does not exist; and diff the intended sends (recipient, template, variables) between old and new rather than the delivered ones, which is the actual verification you want. The same reasoning applies to every non-idempotent downstream: payments, webhooks, push notifications, analytics events and writes to shared state. Also consider read amplification: mirroring doubles the load on shared read dependencies, so a shadow test can be an unintentional load test on your database.

"What can a canary not catch?" Slow-burn data corruption, because the damage accumulates in storage rather than in the response. Failures that only appear at full scale, such as a connection pool that is fine at 5 percent of traffic and exhausted at 100 percent, or a cache hit rate that only degrades once the canary holds a meaningful share. Coordination bugs that require both versions to interact, which the canary period is uniquely likely to trigger and uniquely unable to attribute. And anything with a period longer than the bake, such as a nightly job. Naming these is the difference between using the tool and understanding it.

"Your canary passes but the full rollout regresses. What happened?" Most likely a scale-dependent failure: the canary's 5 percent share did not exhaust a shared resource that 100 percent does. Second most likely, sticky sessions or a cohorting bug meant the canary received unrepresentative traffic, for example only new sessions, or only one region. Third, the bake was shorter than the failure's onset time. The fix for the first is a load test at full scale against the canary build; for the second, verify the actual traffic share and mix against the intended one, because session affinity routinely starves a canary below its configured weight.

"How do you canary a change that only affects 0.1 percent of requests?" You do not, at least not usefully, because the affected population is too small for signal at any exposure below 100 percent. Instead: gate the change behind a feature flag targeted at the affected population specifically, so 100 percent of the relevant traffic is in the experiment; use synthetic traffic that exercises the path deliberately; and rely on the flag as the rollback mechanism, since flag-off is faster than a redeploy.

Common misconceptions

The most common is that a canary is a small A/B test. It is not smaller, it is a different test. The metrics, the horizon, the statistics and the decision rule all differ, and the only shared component is the traffic splitter.

The second is that automated rollback needs human approval. At 3am the automation is the on-call. Roll back automatically and notify a human; requiring approval means the regression runs for however long it takes someone to wake up and read a graph.

The third is comparing the canary to the existing fleet. Different instance counts, different cache warmth and different uptime all skew the comparison. The baseline must be a freshly deployed cohort of the old version at the same size, running at the same time.

Interview delivery note

Say this: "Same mechanism, different question. A canary asks whether the version is safe: operational metrics, minutes to hours, asymmetric decision, automatic rollback, owned by the deploy system. An A/B test asks whether the change is better: product metrics, days to weeks, pre-registered hypothesis and power analysis, owned by the experimentation platform. Every A/B variant rides through a canary first, so safety gates before measurement starts. And I would state the minimum detectable effect up front, because a 1 percent canary on a low-traffic service cannot detect a doubling of a rare error and pretending otherwise is worse than not canarying."

The depth signal is the minimum detectable effect calculation. Do it out loud with real numbers for whatever service is on the whiteboard. Almost nobody does, and it converts a process answer into an engineering one.

Further reading

• Netflix Technology Blog, "Automated Canary Analysis at Netflix with Kayenta" (2018), and the Kayenta repository for the metric classification implementation.
• Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments (2020), especially the chapters on power, peeking and guardrail metrics.
• Argo Rollouts documentation on AnalysisTemplate and progressive delivery steps, as a readable concrete implementation.
• Google, The Site Reliability Workbook, chapter 16, on canarying releases.