The deployment strategy matrix, and consistent cohorting
What it is
A deployment strategy is the answer to one question: during the change, what does the system look like, and how fast can you get back? Everything else, capacity cost, rollback speed, statistical power, is derived from that.
Strategy During the change Rollback Extra capacity
--------------------------------------------------------------------------
Recreate v1 down, then v2 up redeploy v1 none
(slow)
Rolling v1 and v2 both serving, reverse 0 to maxSurge
proportion shifting rolling
(slow)
Blue/green both fully deployed, one flip the 100%
serving router
(seconds)
Canary v1 serving, v2 on a small shift back ~1 replica
slice, ramping (seconds)
Shadow v1 serving, v2 receives nothing to ~100% of v2
mirrored traffic, responses roll back
discarded
Rings cohorts upgraded in stop the none
expanding waves wave
What this is confused with: canary and A/B testing. A canary asks "is this build healthy" and runs for minutes to hours against operational metrics. An A/B test asks "is this change better" and runs for days to weeks against business metrics with statistical power planning. Same routing machinery, entirely different question, stopping rule and owner. (Covered in canary vs A/B testing.)
Also confused: strategy and release. All six of these are about getting code onto machines. Whether a feature is on is a separate decision made by a flag, which is why the two can and should be decoupled (see deploy vs release).
The problem it solves
Every strategy is buying down a different risk, and picking one without naming the risk is how teams end up with an expensive ritual.
Risk The strategy that addresses it
-----------------------------------------------------------------
"the new build crashes" canary, rings (small blast radius)
"the new build is slower" canary or shadow (needs real load)
"rollback takes 25 minutes and
we are down for all of it" blue/green (flip the router)
"we cannot run two versions at
once" recreate, or blue/green with a
hard cutover
"the bug only appears at scale" rolling or ramped canary, and even
then, see the limits page
"a user sees an inconsistent
experience across requests" consistent cohorting, below
And the failure of picking wrong is concrete:
A 20-replica service running at 85% CPU at peak.
Rolling update, maxUnavailable: 25%, maxSurge: 0.
replicas available during the rollout: 15 of 20
load per replica: 85% * 20/15 = 113%
The rollout browns out the service every time, at peak. The team
concludes "deploys cause latency" and starts deploying only at
night, which makes every incident's fix slower.
The rollout window's capacity arithmetic is the thing nobody does, and it is one line.
Mechanics
The six strategies, with when each is correct
Recreate. Stop everything, start the new version.
Correct when: two versions genuinely cannot coexist. A schema change
that cannot be dual-read, a singleton that takes an exclusive lock, a
protocol change with no negotiation. Also correct for dev and for
batch jobs.
Cost: downtime equal to shutdown + startup + warmup.
Do not reach for it to avoid version-compatibility work, because that work is what makes every other strategy available. Expand-contract exists for exactly this (see expand and contract).
Rolling. Replace replicas in batches.
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25% # extra replicas allowed above desired
maxUnavailable: 0 # NEVER go below desired capacity
The two knobs, and the arithmetic they control, for 20 replicas:
maxSurge 25%, maxUnavailable 0 -> 20 to 25 running. Needs 25%
spare capacity/quota. Safe.
maxSurge 0, maxUnavailable 25% -> 15 to 20 running. Free, and
it removes 25% of capacity
during the rollout.
maxSurge 25%, maxUnavailable 25% -> 15 to 25. Fastest, least safe.
Rule: if the service runs above (1 - maxUnavailable) of its capacity
at peak, maxUnavailable must be 0 and you pay for surge.
Rollback is another rolling update, so recovery time equals rollout time. For a 20-replica service with a 60-second readiness delay and batches of 5, that is about four minutes each way.
Blue/green. Two complete environments; the router points at one.
green (v1) <- 100% traffic
blue (v2) <- 0%, fully deployed, warmed, health-checked
cutover: change the router. Seconds.
rollback: change it back. Seconds.
Correct when: rollback speed is the dominant requirement, and you can
afford double capacity for the cutover window.
The hard part is NOT the router, it is shared state:
- the database is shared, so v2's writes are visible to v1 after a
rollback. Expand-contract is still mandatory.
- caches are shared; v2 may populate entries v1 cannot read.
- in-flight long-lived connections do not move with the router.
"Blue/green means I do not need backwards compatibility" is the expensive misconception, because the datastore does not flip with the router.
Canary. A small slice of production traffic, watched, then ramped.
1% -> observe -> 5% -> observe -> 25% -> 50% -> 100%
Requires: enough traffic for the slice to produce a statistically
usable signal in the bake window, and metrics that distinguish
"this build is bad" from noise. Both are covered in bake time and
minimum detectable effect.
Shadow. Mirror requests to v2, discard the responses.
Catches: crashes, latency regressions, resource use, under REAL
production traffic shape, with zero user risk.
Cannot catch: anything about the response, because nobody sees it.
Requires: side-effect containment, or v2 double-charges every card.
(See shadow traffic.)
Rings. Expand by audience rather than by request percentage.
ring 0 internal / dogfood (hours to days)
ring 1 volunteers, beta channel (days)
ring 2 1% of general population
ring 3 10%
ring 4 100%
Correct when you cannot split per request: mobile and desktop apps,
firmware, agents, anything installed. The unit of exposure is a
device or a user, not a request.
Choosing, in four questions
1. Can v1 and v2 coexist, including in the database, cache and
message formats?
NO -> fix that first (expand-contract), or accept recreate.
YES -> continue.
2. How fast must rollback be?
seconds -> blue/green or a flag
minutes -> canary or rolling
3. Is the change per-request routable?
YES -> canary
NO (installed clients) -> rings
4. Do you have the traffic and the metrics for a canary to mean
anything in the bake window?
NO -> a canary is a ritual. Use rings, longer bakes, or accept
that the signal comes from ring 0 humans rather than from
a dashboard.
Question 4 is the one that gets skipped, and the result is a 20-minute 1% canary on a service receiving 40 requests per second, where the canary sees 48,000 requests and cannot detect anything below a several-percent error-rate change.
Traffic shifting mechanics
Layer Mechanism
------------------------------------------------------------------
DNS weighted records. SLOW and imprecise: client and
resolver caching means "shift to 0%" takes as long
as the longest cached TTL. Fine for regional
failover, wrong for canaries.
Load balancer weighted target groups (ALB), backend weights.
Per-connection, not per-request, if connections
are long-lived.
Service mesh per-request routing with weights and header
matching (Istio VirtualService, Linkerd). The most
precise, and it can route on a cohort header.
Gateway/edge a CDN worker or ingress that computes the cohort
and sets a header or cookie.
Application a feature flag SDK, evaluated in-process.
Per-connection weighting silently breaks canaries for gRPC and HTTP/2, because one long-lived connection carries thousands of requests, so a 1 percent weight on connections is not 1 percent of requests, and a small number of clients get all of the canary.
Consistent cohorting: the part that is actually subtle
Random per-request assignment is wrong whenever a user's experience must be coherent across requests.
User loads the page -> v2 (new API response shape)
User's next XHR -> v1 (old shape)
The SPA, holding v2's JS bundle, gets a v1 response and breaks.
Same failure for: a multi-step form, a paginated list where v2
changed the cursor format, anything cached client-side.
The fix is deterministic hash-based assignment, computed once and stable.
import hashlib
BUCKETS = 10_000
def bucket(unit_id: str, salt: str) -> int:
"""Stable, uniform, stateless assignment of a unit to a bucket.
`salt` differs per rollout so that the same users are not
always in the leading cohort. Without it, your canary
population is a fixed, self-selected sample that bears every
rollout's risk and whose behaviour stops being representative.
"""
h = hashlib.sha256(f"{salt}:{unit_id}".encode()).digest()
return int.from_bytes(h[:8], "big") % BUCKETS
def in_canary(unit_id: str, salt: str, percent: float) -> bool:
return bucket(unit_id, salt) < percent * BUCKETS / 100
Three properties fall out of the threshold form, and all three matter:
1. STATELESS. No lookup, no coordination, no storage. Every service
that sees the unit id computes the same answer.
2. MONOTONIC RAMPS. Going 1% -> 5% keeps everyone who was already in,
because bucket < 100 implies bucket < 500. A user is never
flipped BACK to v1 by a ramp up, which would be the same
inconsistency you were avoiding.
3. INDEPENDENT ROLLOUTS. A different salt gives an independent
sample, so two concurrent rollouts do not overlap
systematically, and no user is permanently the guinea pig.
Choosing the unit is a product decision, not a technical one:
per request infrastructure changes with no user-visible state
(a proxy version, a serialisation library)
per session UI changes that must be coherent within a visit
per user anything the user could notice changing under them,
including across devices
per account/org B2B. Two people in the same company on different
versions will file a bug about each other.
per device installed clients, where the user id may not be
known before login
For B2B products the account is almost always the right unit, and using the user is a recurring source of "why does my colleague's screen look different" support tickets.
Propagating the cohort
The cohort must be computed once at the edge and carried, not recomputed independently by every service, because services disagree the moment one of them has a different salt or a different percentage.
edge/gateway: cohort = bucket(user_id, salt)
set header x-cohort: canary
set cookie __cohort=canary; SameSite=Lax
downstream: route on the header (mesh), and log it on every span
and metric.
Two consequences people miss:
CACHING. If a cached response differs by cohort, the cohort MUST be
in the cache key (or in Vary). Otherwise the first canary user
populates the shared cache with v2 HTML and every v1 user gets it.
This is the single most common way a "1% canary" becomes a 100%
incident.
OBSERVABILITY. If the cohort is not a label on your metrics and a
field on your traces, you cannot compare v1 to v2, which is the
entire purpose of the exercise. Add it before the first canary,
not after the first ambiguous one.
A worked example: a rollout that browned out, and the fix
A checkout service. 20 replicas, ~85 percent CPU at peak, a 45-second warmup (JIT plus cache fill), gRPC from the mobile backend, and an SPA that calls it directly.
What was configured, and what happened:
strategy:
rollingUpdate: { maxSurge: 25%, maxUnavailable: 25% }
Peak deploy, observed:
replicas ready during rollout: 15 (5 terminating, 5 not yet ready)
CPU per remaining replica: 85% * 20/15 = 113%
p99 latency: 240ms -> 1,900ms for ~6 minutes
error rate: 0.02% -> 1.4% (upstream timeouts)
Additionally: new replicas were marked Ready by a readiness probe
that only checked the HTTP port, so they took traffic 45 seconds
before they were warm, and each one's first 200 requests were slow.
Three fixes, in order of effect:
1. maxUnavailable: 0, maxSurge: 25%
Capacity never drops below 20. Requires 25% spare quota during
the rollout, which was 5 pods for ~6 minutes.
-> p99 during rollout: 1,900ms -> 310ms
2. Readiness probe that reflects readiness
The probe now checks that the local cache is populated and one
synthetic transaction has completed. New replicas take traffic
when they can serve it.
-> the residual 310ms spike -> 250ms, i.e. gone
3. A per-request canary, because it was now affordable
With maxUnavailable 0 the rollout no longer had to be fast, so
the team moved to Argo Rollouts with a canary at 5% for 15
minutes before the rolling update proceeded.
The capacity fix was free in engineering terms and cost 25 percent of the fleet for six minutes,
which was a straightforward trade once someone wrote down the 85% x 20/15 = 113% line.
Then the cohorting bug, which the canary surfaced.
The canary was configured as a 5% weight on the ingress, applied
PER REQUEST.
Symptom, within 90 seconds: a 3% rate of "session expired" errors
across ALL users, not just the canary's 5%.
Cause: v2 changed the cart cursor encoding. A user's page load hit
v1, the next XHR hit v2, v2 rejected v1's cursor, and the SPA
logged the user out. Per-request splitting meant every user had a
5% chance per request of crossing versions, so with ~12 requests
per checkout session, P(at least one crossing) = 1 - 0.95^12 = 46%.
A 5 percent canary produced a fault rate near 50 percent of sessions, which is the clearest possible demonstration that the routing unit, not the routing percentage, determines blast radius.
The fix:
Edge computes the cohort once per user and pins it:
cohort = bucket(user_id or device_id, salt="checkout-v2-2024-03")
set header x-cohort, and a __cohort cookie for the SPA
Ingress routes on the header rather than on a weight.
Ramp: 1% -> 5% -> 25% -> 50% -> 100%, monotonic by construction.
Result: crossings went to zero. The 5% canary affected exactly 5%
of users, all of whom stayed on v2.
And one more bug appeared at the 25 percent step, which is the caching consequence:
The CDN cached /api/cart/summary for 30 seconds with a key of
(path, user segment). Cohort was not in the key.
At 25% canary, a canary user's response was cached and served to
control users, whose SPA could not parse it.
Fix: added `Vary: x-cohort` and included the cohort in the CDN cache
key. Hit rate fell from 71% to 68% during the rollout and returned
afterwards.
A 3-point cache hit-rate cost for the duration of a rollout is the correct price, and the alternative was a canary whose blast radius was unbounded by construction.
Final shape:
before after
strategy rolling only cohorted canary -> rolling
maxUnavailable 25% 0
readiness port check warm check
routing unit per request per user, hashed and pinned
cohort in cache key no yes
cohort on telemetry no yes (label + span attribute)
p99 during deploy 1,900ms 250ms
deploys per day 1 (at night) 6 (business hours)
The last line is the outcome that mattered. Deploys stopped being an event, which shortened every subsequent incident's fix time, and that argument, not the latency graph, is what justified the work.
Production evidence
Kubernetes Deployment maxSurge/maxUnavailable are the documented knobs and default to 25
percent each, which means the default configuration reduces capacity by 25 percent during every
rollout. That default is the origin of a large share of "our deploys cause latency" reports.
Argo Rollouts and Flagger both implement canary and blue/green as first-class Kubernetes resources with automated analysis and traffic-provider integrations (Istio, Linkerd, ALB, NGINX), and both expose the ramp-with-bake pattern directly.
Istio's VirtualService supports weighted routing and header-based match rules on the same
resource, which is what makes edge-computed cohort headers routable without every service
participating.
Microsoft's deployment-rings model for Windows and Azure DevOps is the documented reference for audience-based progressive rollout where per-request splitting is impossible, and its ring 0 (internal dogfood) stage is the acknowledgement that some signal comes from humans rather than metrics.
Facebook's Gatekeeper and modern flag platforms (LaunchDarkly, Statsig) all use deterministic hashing of a unit id with a per-flag salt for assignment, and the salt exists specifically so that the same users are not repeatedly in the leading cohort.
Netflix's regional traffic shifting for evacuation uses weighted DNS and edge steering, and their published work on it is also the clearest illustration of why DNS-based shifting is unsuitable for canaries: the shift completes on the timescale of client-side cache expiry, not on the timescale of a bake window.
The debate
Blue/green or canary? Canary for most services, because it gives graduated exposure and a real signal, and blue/green mainly buys rollback speed you can also get from a flag. Blue/green is right when a change cannot be partially deployed (a full-stack cutover, a routing layer change) or when the organisation genuinely cannot tolerate a minutes-long rollback. The double-capacity cost is usually the deciding factor, and it is a real budget line rather than an abstraction.
Is a canary worth it at low traffic? Often not, and this is under-admitted. Below the traffic where the bake window produces a detectable effect, a canary is a delay that feels like diligence. The honest alternatives are a longer bake at a higher percentage, ring 0 human validation, or accepting rolling with fast rollback. Running a ritual canary is worse than not running one, because it consumes the organisation's belief that the rollout was checked.
Should maxUnavailable ever be non-zero? For services with meaningful headroom, yes, it is free
and faster. For anything running above 70 percent of capacity at peak, no, and the arithmetic
decides it rather than a convention.
Per-request or per-user cohorting? Per-user by default for anything a user could notice, because the cost of getting it wrong is not proportional to the canary percentage: a 5 percent per-request canary produced session failures in roughly half of sessions in the worked example. Per-request is correct only for changes with no cross-request state, which is a smaller category than it appears.
Should the cohort be sticky across a ramp down? Yes, and it comes free from the threshold form. The case for breaking it, rebalancing to keep cohorts "clean" for analysis, is an experimentation concern rather than a deployment one, and mixing the two is how a rollback flips users back and forth. Keep deployment cohorts monotonic; let the experimentation platform own its own assignment.
Is DNS-weighted shifting ever acceptable? For regional failover and evacuation, yes. For canaries, no, because the shift's completion time is bounded by client and resolver caching rather than by your control plane, so "roll back now" is a request rather than an action.
Follow-up Q&A
"How do you choose a deployment strategy?"
Four questions. Can the two versions coexist, including in the database, caches and message formats, because if not you either fix that with expand-contract or accept a recreate. How fast must rollback be: seconds means blue/green or a flag, minutes means canary or rolling. Is the change routable per request, because installed clients force rings instead. And do you have the traffic and metrics for a canary to detect anything in the bake window, because below that threshold a canary is a ritual and the alternatives are a longer bake, ring 0 humans, or rolling with fast rollback.
"Why do deploys cause latency spikes?"
Usually the default maxUnavailable: 25%, which removes a quarter of your capacity during the
rollout. A 20-replica service at 85 percent CPU drops to 15 replicas at 113 percent, which browns
out. The fix is maxUnavailable: 0 with maxSurge, paying for spare capacity during the window. The
second cause is a readiness probe that checks the port rather than readiness, so replicas take
traffic before caches are warm or the JIT has compiled, and each new replica serves its first few
hundred requests slowly.
"Why is per-request canary routing dangerous?"
Because the blast radius is not the canary percentage. If a user's requests are independently assigned, a session of twelve requests at a 5 percent canary has a 1 minus 0.95 to the twelfth power, about 46 percent, chance of crossing versions at least once. Any cross-version incompatibility, a changed cursor format, a changed response shape, a client-side cached bundle, then breaks for roughly half of sessions from a 5 percent rollout. The routing unit, not the percentage, determines exposure.
"How does consistent cohorting work, and why the salt?"
Hash the unit id with a per-rollout salt into a large bucket space and compare against a threshold:
bucket(id, salt) < percent. It is stateless, so every service computes the same answer with no
coordination. It is monotonic, so ramping from 1 to 5 percent keeps everyone already in and never
flips a user back. The salt makes each rollout an independent sample, which matters because without
it the same users are always in the leading cohort of every rollout, so they bear all the risk and
their behaviour stops being representative of the population you are measuring.
"What must you do before the first cohorted rollout?"
Two things, both easy to forget and both expensive afterwards. Put the cohort in the cache key or in
Vary, because otherwise a canary response gets cached and served to control users, which turns a 1
percent rollout into a 100 percent incident. And put the cohort on your metrics as a label and on
your traces as an attribute, because comparing v1 to v2 is the entire purpose and you cannot slice
what you did not record.
"When is per-connection traffic weighting wrong?"
For gRPC and HTTP/2, where a single long-lived connection carries thousands of requests. A 1 percent weight on connections is not 1 percent of requests, and a small number of clients receive all of the canary traffic, which both biases the signal and concentrates the risk. Per-request routing through a mesh, or cohort-header matching, is what you need there.
Common misconceptions
"Blue/green means I do not need backwards compatibility." The database, caches and message queues do not flip with the router. Expand-contract is still required.
"A canary limits blast radius to the canary percentage." Only if the routing unit matches the unit of user-visible state. Per-request assignment lets one session cross versions repeatedly.
"The defaults are safe." Kubernetes defaults to 25 percent maxUnavailable, which is a capacity
reduction during every rollout and is unsafe for any service running near its limits.
"A canary is always worth running." Below the traffic needed for detection in the bake window it is a delay that consumes the organisation's belief that something was checked.
"Shadow traffic proves the new version is correct." It proves it does not crash and is not slower. Nobody looks at the responses, and side effects must be contained or it does real damage.
"DNS weighting is a traffic-shifting mechanism." For failover, yes. For canaries, no: rollback completes on the timescale of resolver caches rather than your control plane.
Interview delivery note
Say this verbatim: "The routing unit determines blast radius, not the percentage. A five percent per-request canary on a twelve-request session gives a forty-six percent chance that the session crosses versions, so any cross-version incompatibility breaks half of sessions. Cohort by user with a salted hash, and the ramp is monotonic for free." It is the specific, quantified insight most candidates do not have.
The senior-versus-staff separator is doing the capacity arithmetic out loud. A senior engineer
configures a rolling update. A staff engineer says that a 20-replica service at 85 percent CPU with
maxUnavailable: 25% runs at 113 percent during every rollout, so the default is a guaranteed
brownout at peak, and then names the trade: maxUnavailable: 0 costs 25 percent spare quota for six
minutes and buys business-hours deploys. Converting a config default into one line of arithmetic and
then into a deploy-frequency outcome is the move.
The second signal is naming the two prerequisites nobody sets up first: the cohort in the cache key, and the cohort on the telemetry. Both are trivial before the first rollout and both are found during an incident otherwise, the first as "our 1 percent canary served v2 responses to everyone" and the second as "we could not tell whether the canary was worse."
Further reading
- Kubernetes documentation on Deployment strategies,
maxSurgeandmaxUnavailable, and on readiness probes. - Argo Rollouts and Flagger documentation, for canary and blue/green as declarative resources with analysis and traffic-provider integration.
- Istio
VirtualServicedocumentation, for weighted routing combined with header-based matching. - Microsoft's deployment rings guidance, for audience-based progressive rollout where per-request splitting is impossible.
- The bake time and minimum detectable effect page, which decides whether a canary at a given percentage can detect anything at all.