What canaries cannot catch
What it is
A canary detects fast, high-frequency, request-scoped, observable regressions in a small population over a short window. Everything outside that description passes a canary cleanly and breaks in production.
A canary sees: a small fraction of traffic
for a short time
on request-scoped signals (error rate, latency,
saturation)
from one version, in isolation
So it is blind to anything that is:
SLOW accumulates over hours or days
RARE executes less often than the bake window allows
EMERGENT only appears at full scale or full concurrency
SILENT produces no error and no latency change
MIXED is a property of v1 and v2 coexisting
ELSEWHERE happens on a client, in a batch job, or downstream
What this is confused with: a bad canary configuration. These are not tuning problems. No choice of percentage, bake time or threshold makes a canary detect a memory leak that manifests at hour six, because the canary's replicas are replaced before hour six.
Also confused: "the canary passed" and "the change is safe." A passing canary is evidence about one class of failure. Treating it as a general safety verdict is how the six categories below reach production with everyone's confidence attached.
The problem it solves
Naming the blind spots is what lets you buy a compensating control for each one, instead of extending the bake time and hoping.
The failure this prevents is the confident rollout:
"Canary was green for 30 minutes at 5%, ACA scored 0.94, we ramped
to 100% at 14:20."
18:40 memory on all replicas crosses the limit within four minutes
of each other. Every pod OOM-kills. Full outage.
The leak was ~40MB/hour. In 30 minutes a canary pod gained 20MB
against a 2GB limit, which is 1% and indistinguishable from noise.
The canary could not have caught it, and nobody had asked whether
it could.
The point of the list is to make "could this class of bug be caught by our canary?" a question that gets asked before the rollout, and to have an answer other than "extend the bake."
Mechanics
1. Slow-burn resource exhaustion
Leaks and accumulations:
heap / off-heap memory
file descriptors, sockets
thread or goroutine growth
connection-pool entries never returned
disk: logs, temp files, unbounded local caches
database connections held by an unreleased transaction
Why the canary misses it: the manifestation time exceeds the bake
window, often by one to three orders of magnitude.
Arithmetic:
leak rate 40 MB/hour, limit 2 GB, baseline usage 800 MB
headroom = 1,200 MB -> time to OOM = 30 hours
Canary bake: 30 minutes -> 20 MB gained -> 1.7% of headroom.
Normal hour-to-hour variation on a JVM with a non-deterministic
GC schedule is comfortably larger than that.
Compensating controls:
- A SOAK stage: one replica running the new build for 24 to 72
hours under real (or mirrored) traffic, with resource trends
monitored. Slower than a canary and it is the only thing that
finds this class.
- Trend-based alerts rather than threshold alerts: alert on
"RSS increasing monotonically for 6h" instead of "RSS > 90%".
- Bounded everything: a heap limit that forces the OOM early in
a non-production soak, container memory limits, connection-pool
maximums, log rotation, cache size caps with eviction.
- Deliberate replica recycling as a mitigation of last resort,
which is a workaround and should be labelled as one.
2. Low-frequency code paths
A canary at p percent for t seconds executes a path that occurs at rate r only occasionally, and the probability is computable.
Total traffic: 200 req/s
Canary share: 1% -> 2 req/s to the canary
Path frequency: 1 in 10,000 -> 0.0002 executions/s
Bake window: 20 min = 1,200 s
Expected executions = 0.0002 * 1200 = 0.24
P(at least one) = 1 - e^-0.24 = 21%
So a 20-minute 1% canary has a roughly one-in-five chance of ever
running that code path, and if it runs it once, one failure is
statistically invisible against 2,400 canary requests.
The paths this describes are exactly the risky ones: error and retry handlers, the fallback when a dependency is down, a rarely used locale or currency, an admin operation, the leap-day branch, the "user has more than 1,000 items" branch, and the month-end path.
Compensating controls:
- SYNTHETIC traffic that deliberately exercises rare paths against
the canary, including forced dependency failures.
- Fault injection during the bake, so the error path executes.
- Contract tests and unit tests for rare paths, since this is the
category where lower-layer testing genuinely wins.
- Route by attribute rather than by percentage: send 100% of a
small, low-risk segment (one locale, internal users, one small
tenant) to the canary, so the rare path's rate within that
segment is normal rather than divided by 100.
The last one is the useful trick and it is under-used. A canary that receives 100 percent of one small country's traffic executes that country's rare paths at their natural rate.
3. Emergent, scale-dependent failures
Some failures are a function of the fraction, not of the code.
CACHE KEY CHANGES.
v2 changes a cache key format. At 1%, v2's misses are a rounding
error and the hit rate looks fine. At 100%, the entire cache is
cold and the backing store receives the full uncached load.
A 95% hit rate becoming 0% is a 20x increase in origin QPS.
CONNECTION POOL / DOWNSTREAM SATURATION.
v2 opens 2 connections per request instead of 1. At 1% the
downstream pool absorbs it. At 100% the pool is exhausted and
every service sharing it degrades.
THUNDERING HERD ON A SHARED RESOURCE.
v2 refreshes a config on a fixed schedule instead of a jittered
one. One canary replica is invisible; 200 replicas hitting the
config service on the same second is an outage.
RETRY AMPLIFICATION.
v2 adds a retry. At 1% it is 2 extra req/s. At 100% during a
downstream blip it triples load exactly when the downstream is
already failing.
Compensating controls:
- Ramp with bake at EACH step (1, 5, 25, 50), not straight to 100.
Most emergent failures show a signal at 25% before they become
an outage at 100%.
- Monitor DOWNSTREAM and SHARED resources during the ramp, not
just the service being deployed. This is the specific gap:
canary analysis almost always scopes metrics to the deploying
service.
- Explicitly review diffs for: cache key changes, connection
lifecycle changes, retry/timeout changes, and scheduled-work
jitter. Those four cover most of this category.
- Load-test the new version at full expected concurrency in a
non-production environment where "full scale" is achievable.
4. Silent correctness failures
Canary analysis watches error rate, latency and saturation. A wrong answer is none of those.
v2 computes a discount incorrectly: 2xx, fast, low CPU.
v2 writes a field in the wrong unit (cents vs dollars): 2xx, fast.
v2 truncates a field at 255 characters: 2xx, fast.
v2 silently drops a message on a parse failure: 2xx, fast, and the
error rate goes DOWN.
Every one of these scores well on a canary. Some corrupt persisted
data, which means the blast radius grows for as long as the change
is live and rollback does not undo it.
Compensating controls:
- SHADOW with response diffing: mirror traffic to v2 and compare
its responses to v1's, rather than discarding them. This is the
control specifically designed for this category. (See shadow
traffic for side-effect containment.)
- Business-metric monitoring on the canary cohort: revenue per
session, conversion, items per order, and message throughput,
not just RED metrics.
- Data-quality assertions: invariants checked continuously
(non-negative totals, referential integrity, distribution
drift on key fields) rather than at write time only.
- For anything that writes, prefer a dark-write comparison over a
canary: v2 computes and logs what it WOULD write, and a job
compares.
"The canary's error rate went down" deserves suspicion, because the most common cause is that v2 stopped reporting something.
5. Mixed-version failures
The canary tests v2. Production during a rollout runs v1 and v2 simultaneously, and that mixture is a third system nobody tested.
- v2 writes a message format v1 cannot parse. The canary's own
messages round-trip fine; v1 consumers fail.
- v2 writes a cache entry v1 misreads (or vice versa).
- v2 takes a lock with different semantics, so v1 and v2 both
believe they hold it.
- v2 changes a database column's meaning; v1 keeps writing the old
meaning, and the data is now ambiguous by row.
- Two versions of a leader-elected component disagree about the
protocol.
This is the category expand-contract exists to prevent (see expand and contract), and it is the argument for treating compatibility as a design requirement rather than a rollout concern.
Compensating controls:
- Explicit N-1 (and often N-2) compatibility testing: run the test
suite with a MIXED deployment, not a uniform one.
- Contract tests between versions, not just between services.
- Never change a field's meaning; add a new field.
- For shared caches: version the key prefix so v1 and v2 cannot
read each other's entries. Costs a cold cache; buys correctness.
6. Effects that happen elsewhere or later
CLIENT-SIDE. A server canary cannot see that v2's response breaks
an app version from eight months ago that 3% of users still run.
ASYNCHRONOUS. v2 enqueues a job; the job runs at 02:00 and fails.
The canary ended at 14:30.
BATCH AND SCHEDULED. Month-end, the nightly reconciliation, the
weekly report. A Tuesday-afternoon canary exercises none of them.
DOWNSTREAM-DELAYED. v2 writes data that a partner ingests daily.
TEMPORAL. The 03:00 traffic shape, the Monday spike, the sale.
Compensating controls:
- Bake across the relevant cycle for changes that touch it. A
change to the nightly job bakes overnight, not for 20 minutes.
- Client-version dimensions on canary metrics, so old clients are
visible as a cohort. (See mobile staged rollout for N-2 support.)
- Separate rollout tracks for scheduled work, with their own
canary run against a subset of jobs.
- A kill switch on anything asynchronous, since rollback of the
deployment does not un-enqueue work.
The summary table worth remembering
Blind spot Manifests Compensating control
-------------------------------------------------------------------
Slow-burn leaks hours-days soak stage, trend alerts, caps
Rare code paths 1 in 10^3-10^5 synthetic traffic, fault
injection, segment-routing
Emergent at scale at 50-100% ramp with bake, watch SHARED
and DOWNSTREAM resources
Silent correctness never (as an shadow + response diff,
error) business metrics, data
invariants
Mixed-version during the expand-contract, mixed-version
rollout test runs, versioned cache keys
Elsewhere/later hours-weeks cycle-length bakes, client
version dimensions, kill
switches
A worked example: four incidents behind green canaries
One year of post-incident reviews at a payments company, filtered to incidents where the change had passed a canary. Four are instructive.
Incident 1: the leak (slow-burn).
Change: a new HTTP client with connection pooling.
Canary: 5% for 30 minutes, ACA score 0.97, all green.
Rolled to 100% at 11:10.
19:45 all 60 replicas OOM within a 4-minute window.
Cause: connections were not returned to the pool on a specific
timeout path. Growth: ~2,200 sockets/hour per replica.
In 30 minutes a canary replica accumulated ~1,100 sockets against
a 65,536 file-descriptor limit. Undetectable.
The 4-minute clustering was itself the signature: every replica
started at the same deploy time and leaked at the same rate, so
they all crossed the limit together. A staggered fleet would have
failed one at a time and been diagnosed hours earlier.
Control added: a 24-hour soak stage for changes touching I/O or
connection lifecycle, plus an alert on monotonic FD growth over 4h.
Synchronised deployment turns a gradual failure into a simultaneous one, which is a second-order argument for staggering replica start times.
Incident 2: the rare path.
Change: refactored the refund handler.
Canary: 1% for 20 minutes.
Refund rate: ~1 in 4,000 requests. Traffic 350 req/s.
Expected refunds seen by the canary:
350 * 0.01 * (1/4000) * 1200 = 1.05
Observed: 1 refund, which succeeded.
At 100%, a specific refund subtype (partial refund on a
multi-currency order, ~1 in 90,000 requests) threw. 41 failed
refunds before the alert fired.
Control added: a synthetic suite that exercises 14 named low-
frequency paths against the canary on every rollout, including
partial and multi-currency refunds. Run time: 90 seconds.
Ninety seconds of synthetic traffic replaced a bake window that would have needed 26 hours to see that path once.
Incident 3: emergent at scale.
Change: switched the idempotency-key cache to include the API
version in the key.
Canary: 10% for 45 minutes. Cache hit rate on the canary: 91%
(baseline 94%). Judged acceptable.
Ramp to 100% at 16:00. Within 90 seconds the idempotency store's
p99 went from 3ms to 900ms and the service shed load.
Cause: at 10%, the canary was reading keys that v1 had already
populated in the OLD format for most requests, and its 3-point hit
rate drop was the genuinely new keys. At 100%, no old-format key
was being refreshed, the entire working set turned over at once,
and the backing store took the full uncached load: roughly 20x its
normal write rate for several minutes.
The signal existed and was on the WRONG SERVICE'S dashboard: the
idempotency store's write rate had risen during the canary. Canary
analysis was scoped to the deploying service only.
Control added: canary analysis now includes named downstream
dependencies' saturation metrics, and cache-key format changes are
flagged in review as requiring a pre-warm or a staged key
migration.
"The signal existed on another team's dashboard" is the single most common shape of this category, and it is fixed by scope rather than by sensitivity.
Incident 4: silent correctness.
Change: moved currency conversion to a new library.
Canary: 5% for 60 minutes. Error rate unchanged. Latency improved
by 4ms. ACA score 0.99. Ramped.
Discovered 9 days later by finance reconciliation: for three
currencies with more than two decimal places, amounts were rounded
to 2dp before conversion rather than after. Average error: 0.3%.
Affected transactions: ~140,000. All returned 200. All were fast.
Control added:
- shadow-with-diff for any change touching money: v2 computes,
v1 serves, a job compares and alerts on any mismatch above a
per-field tolerance.
- a daily invariant job: sum(ledger entries) == sum(transactions)
per currency.
- business metrics on the canary cohort (average order value by
currency), not only RED metrics.
Nine days is the cost of a canary that watches only RED metrics on a system whose failure mode is being wrong.
The programme change that came out of the four:
A deployment risk questionnaire, three questions, answered in the
change description:
1. Could this fail slowly? (I/O, connections, caches, memory,
disk) -> soak stage required
2. Does this touch a path rarer than
1 in 1,000 requests? -> synthetic coverage
required
3. Could this be wrong without being
an error? -> shadow diff or
invariant required
Answering "no" to all three keeps the standard canary path.
Measured effect over the following two quarters: incidents behind
a green canary fell from 11 to 3.
The value was not in any single control, it was in making "what can our canary not see" a required question, which is a two-minute cost on every change and was the only intervention that addressed all four categories.
Production evidence
Netflix's Kayenta / Automated Canary Analysis scores canaries against control on a defined metric set, and Netflix's own writing is explicit that ACA evaluates a bounded set of signals over a bounded window, which is the documented statement of scope rather than a limitation discovered later.
Google's SRE Book describes canarying as detecting a specific class of problem and pairs it with independent controls (staged rollouts across cells and regions, soak periods, and separate verification of data correctness), which is the same argument as this page made at the level of a release process.
Facebook and Microsoft ring models both include an extended internal dogfood ring measured in days before any percentage-based exposure, and the stated reason is precisely that slow and rare failures need calendar time rather than traffic volume.
Kubernetes container memory limits and OOMKill behaviour are the mechanism behind the synchronised-failure signature: replicas deployed together, leaking at the same rate, cross the same limit at nearly the same time.
Consumer-driven contract testing (Pact) and mixed-version test runs are the documented industry answer to the mixed-version category, and they exist because a uniform test deployment cannot represent a rollout.
Shadow traffic with response comparison is used in production by teams migrating critical paths (GitHub's Scientist library is the widely cited implementation of the compare-old-and-new pattern), and it is the control specifically aimed at silent correctness failures.
The debate
Should a longer bake fix this? For slow-burn leaks, a longer bake is genuinely the answer, and a 24-hour soak is a different stage rather than a longer canary. For rare paths, emergent failures and silent correctness, no amount of bake time helps, because the limitation is coverage, concurrency or the metric set rather than duration. Extending the bake is the default response and it addresses one of six categories.
Is shadow traffic with diffing worth the cost? For money, permissions, and anything whose failure mode is being wrong rather than being down, yes. The cost is real: doubled compute for the shadow path, and rigorous side-effect containment, because a shadow that charges cards is worse than no canary. For a stateless read path with visible errors, it is over-engineering.
Should canary analysis include downstream metrics? Yes, and almost no default configuration does. The counter-argument, that a noisy shared dependency will fail canaries for unrelated reasons, is real and is handled by scoring downstream signals as warnings requiring human review rather than as automatic failures. The alternative is the incident-3 shape, where the signal existed on another team's dashboard.
Is segment routing better than percentage routing for canaries? For rare-path coverage, clearly: 100 percent of one small tenant or locale exercises that segment's rare paths at their natural rate, where 1 percent of everything divides every path's rate by 100. The trade is representativeness: one segment is not the population, so it is a complement to percentage ramping rather than a replacement.
Does this argue against canaries? No. It argues against a canary being the only gate, and against the organisational effect of a green canary, which is that it transfers confidence out of proportion to what it measured. The position: keep the canary, add a three-question risk triage, and buy a specific control for whichever category the change is exposed to.
Follow-up Q&A
"What can a canary not catch?"
Six categories. Slow-burn resource exhaustion, because the manifestation time exceeds the bake window by orders of magnitude. Rare code paths, because a 1 percent canary divides every path's frequency by 100. Emergent failures that are a function of the fraction rather than the code, like a cache-key change that looks fine at 1 percent and turns the cache cold at 100. Silent correctness failures, because canary analysis watches errors, latency and saturation and a wrong answer is none of those. Mixed-version failures, because production during a rollout runs v1 and v2 together and the canary tested only v2. And effects that happen elsewhere or later: on old clients, in async jobs, in batch work, on a different cycle.
"Why can't a longer bake fix a rare code path?"
Because the constraint is arithmetic. With 200 requests per second, a 1 percent canary, and a path that occurs once in 10,000 requests, the canary executes that path 0.0002 times per second. A 20-minute bake gives 0.24 expected executions, so a 21 percent chance of seeing it once, and one execution is statistically invisible anyway. Reaching useful coverage needs hours to days. The answers that work are synthetic traffic that exercises the path deliberately, fault injection so error handlers run, or routing 100 percent of one small segment to the canary so the path occurs at its natural rate.
"Give a concrete emergent failure that a canary cannot show."
A cache-key format change. At 10 percent, the canary still reads keys that the 90 percent on v1 keeps populating in the old format, so the hit rate barely moves. At 100 percent nothing refreshes the old format, the entire working set turns over at once, and the backing store takes the full uncached load, which for a 95 percent hit rate is a twentyfold increase in origin traffic. The signal during the canary was a rise in the idempotency store's write rate, which was on a different team's dashboard because canary analysis was scoped to the deploying service.
"How do you catch silent correctness failures?"
Not with RED metrics, because the failure returns 200 quickly. Shadow the traffic and diff v2's responses against v1's, which is the control designed for this. Add business metrics to the canary cohort, average order value by currency, conversion, items per order, rather than only errors and latency. Add continuously checked data invariants, like ledger sums matching transaction sums per currency. And treat a canary error rate that goes down as suspicious, because the usual cause is that the new version stopped reporting something.
"What is the mixed-version blind spot?"
The canary evaluates v2 in isolation, but during a rollout production is running v1 and v2 at once, and that mixture is a third system nobody tested. The failures are v2 writing a message or cache entry v1 cannot read, two versions disagreeing about lock semantics, or a column whose meaning differs by which version wrote the row. Expand-contract exists to prevent this class, and the practical test is running the suite against a mixed deployment rather than a uniform one, plus versioning shared cache key prefixes so the two versions cannot read each other's entries.
"How would you operationalise all this without slowing every change?"
A three-question triage in the change description. Could this fail slowly, meaning it touches I/O, connections, memory, disk or caches, which requires a soak stage. Does it touch a path rarer than about one in a thousand requests, which requires synthetic coverage. Could it be wrong without being an error, which requires a shadow diff or a data invariant. Answering no to all three keeps the standard canary path, so the cost is two minutes on most changes. In one case that questionnaire took incidents behind a green canary from eleven to three over two quarters.
Common misconceptions
"A green canary means the change is safe." It means one class of failure was not detected in a small population over a short window.
"Extend the bake time." That addresses slow-burn leaks and nothing else. Rare paths, emergent scale failures and silent correctness are coverage and metric problems, not duration problems.
"The canary error rate improved, so the change is good." The most common cause of an improving error rate is that the new version stopped reporting an error class.
"Shadow traffic validates correctness." Only if you compare the responses. Discarding them validates that it does not crash and is not slower.
"Canary analysis covers the blast radius." It almost always scopes metrics to the deploying service, so downstream and shared-resource saturation, which is where emergent failures show first, is not in the score.
"Two versions running together is a rollout detail." It is a distinct system configuration with its own failure modes, and it is the one that runs during every deployment.
Interview delivery note
Say this verbatim: "A canary detects fast, frequent, request-scoped, observable regressions. So it is structurally blind to slow leaks, rare paths, failures that only emerge at full fraction, silent correctness bugs, mixed-version interactions, and anything that happens on a client or later. Those are not tuning problems, and the honest response is a specific control per category rather than a longer bake." It converts a vague "canaries have limits" into a checklist.
The senior-versus-staff separator is doing the rare-path arithmetic. A senior engineer says a canary might not exercise unusual code paths. A staff engineer computes it: at 200 requests per second, a 1 percent canary and a path occurring once in 10,000 requests, a 20-minute bake gives 0.24 expected executions and a 21 percent chance of running it at all, so reaching coverage needs hours, which means the answer is synthetic traffic or segment routing rather than a longer window. Turning "might not" into a probability is what makes the conclusion actionable.
The second signal is naming the scoping failure: canary analysis is almost always scoped to the deploying service, so the emergent failures that show up first as downstream or shared-resource saturation are invisible to the score even though the signal exists on someone's dashboard. Proposing that downstream saturation enters the analysis as a human-review warning rather than an automatic fail shows you have thought about the false-positive cost too.
Further reading
- Netflix's writing on Kayenta and automated canary analysis, for the defined metric set and scoring window that this page describes the boundary of.
- Google's SRE Book chapters on release engineering and canarying, for canarying paired with soak periods and independent correctness verification.
- GitHub's Scientist library and its accompanying write-up, for the compare-old-and-new pattern that addresses silent correctness.
- The shadow traffic and expand and contract pages, which are the controls for the silent-correctness and mixed-version categories.
- The bake time and minimum detectable effect page, for the statistical limit on what a canary of a given size can detect at all.