Drills 61 to 66: deployment and progressive delivery

Six drills, ninety seconds each, out loud. Four of these six are questions where the naive answer creates a new problem, which is the pattern to rehearse: correct the premise, then answer.

The other pattern: half of them turn on a number. A canary at 200 QPS, a 4-second flag flip against a 25-minute redeploy, a rollout percentage that cannot be reduced. Having the number is what separates the answer from the description.

Drill 61. A PM says "the canary shows the feature is winning." Correct it.

Gently, because they are not wrong about wanting to know, they are wrong about which instrument answers it.

A canary asks "is this safe". It runs for minutes, on technical metrics, over a small traffic share, and it defaults to rollback. An A/B test asks "is this better". It runs for weeks, on business metrics, and it defaults to keeping the control. They are different questions with different statistics.

Concretely, the canary cannot tell you the feature is winning for two reasons. Sample size: at five percent of traffic for thirty minutes, a business metric like conversion has enormous variance, and any apparent lift is well inside the noise. And the canary population is not randomised the way an experiment's is, so it may not be representative.

What I'd offer instead: the canary told us it's safe, which is what it's for and which is genuinely good news. Now we run an A/B test to find out whether it's better, and here's the traffic and duration that needs. If we shipped on the canary result we'd be shipping on noise, and the specific risk is that we'd learn the wrong lesson about what works.

And I'd raise that this is exactly why business metrics don't belong in the canary gate: they're too noisy at that sample size, gating on them causes false rollbacks, and the team's response is always to lower the overall threshold, which degrades the whole gate.

Depth signal: the two distinct reasons the canary cannot answer it (sample size and randomisation), and connecting it to why business metrics stay out of the gate.

Full treatment: Canary vs A/B testing and Experimentation design.

Drill 62. Canary schedule for a payments service at 200 QPS. What can you not detect?

At 200 QPS, a five percent canary for ten minutes is six thousand requests each side. So the first thing I'd do is compute what that can detect, because the honest answer is "not much".

For an error rate moving from a 0.5 percent baseline, at ninety-five percent confidence and eighty percent power, six thousand samples per side detects a difference of roughly 0.9 percentage points. So it can catch 0.5 going to 1.4 percent, and it cannot catch 0.5 going to 1.0, which is a doubling of the error rate. For a payments service that is not an acceptable blind spot.

So the schedule I'd propose is staged. One percent for fifteen minutes first, which catches catastrophic failures with minimal exposure and needs almost no samples, since a crash loop shows up in one request. Then twenty-five percent for forty-five minutes, which gets to about 135,000 samples per side and takes the detectable error difference down to around 0.2 percentage points. Then fifty, then a hundred.

And the higher share isn't only about statistics: some failures need real concurrency to appear. A connection pool sized for the full fleet isn't stressed at five percent, and lock contention needs load to manifest.

What I still cannot detect at any feasible bake time: a small p99 regression, because only about one percent of samples inform the p99, so its effective sample size is a hundredth of the total and its detectable effect is roughly ten times worse. I'd say that explicitly rather than pretend the gate covers it, and move p99 regression detection to post-rollout monitoring with a fast rollback.

Depth signal: computing the MDE rather than proposing a schedule, and naming what cannot be gated at all.

Full treatment: Bake time and minimum detectable effect.

Drill 63. Shadow-test a rewrite of a service that sends emails.

The emails are the whole problem, and I'd say that first. Shadowing reads is trivial; shadowing anything with an external side effect is where systems get destroyed, because the naive version sends every customer a second email.

So the containment has to be a design property rather than something improvised in shadow mode. If the email gateway is called directly from scattered call sites there is no seam and this isn't safely possible, and the honest answer is to shadow reads only until the seam exists.

Assuming a seam: I'd inject a recording no-op email sender into the shadow. Not a silent no-op, a recording one, so the shadow tells me "this version would have sent to this address with this template and these variables". That lets me compare the intended side effects rather than only the responses, which is where the interesting differences are in a rewrite.

Then response comparison with normalisation: strip request ids, timestamps and hostnames, sort unordered collections, and bucket the diffs by shape rather than listing instances, because forty thousand diffs of the same kind is one finding.

Two operational things. Shadow ten percent rather than a hundred, because the shadow's database queries hit the same database and its cache writes pollute the same cache, so full mirroring doubles load on every dependency. And propagate a shadow header end to end, because without it the shadow traffic is indistinguishable from real traffic in every downstream service's dashboards and corrupts their error-rate and latency SLIs.

And I'd be clear about what this cannot tell me: whether the rewrite is better. Nobody sees the responses, so there's no user signal at all. It answers "is it safe and does it behave the same".

Depth signal: recording no-ops rather than silent ones, and the shadow header preventing SLI corruption.

Full treatment: Shadow traffic.

Drill 64. A rollback fails because the old version cannot read new cache entries.

The rollback didn't fail because the code was wrong. It failed because the new version left state the old version can't read, and rolling back the code didn't roll back the world.

And with a TTL it doesn't self-heal. A 24-hour TTL means 24 hours of poisoned entries, so the rollback leaves you in a worse state than before it: the new code is gone and its cache entries aren't.

Two fixes, and I'd use both. Version the cache key, so the new version writes session:v3:* and the old reads session:v2:* and they cannot collide. Rollback is then instant and clean, at the cost of a cold cache for the new version, which is a stampede consideration rather than a correctness one. And version the payload with a rule that an unknown version is treated as a cache miss rather than an exception, which downgrades a rollback failure into a performance dip.

But the general fix is the habit, and it's five minutes before every deploy: what state does this version write that the previous one can't read? Database schema, cache entries, serialised sessions, published events, object storage, queue message shapes, feature flag state. Schema migrations get reviewed because they're visible; a serialisation change in a cached object is a detail in a pull request that nobody flags, and it's the one that breaks the rollback.

Depth signal: naming that a TTL means it does not self-heal, and the pre-deploy checklist as the general fix.

Full treatment: Expand and contract.

Drill 65. Mobile app crashes for 2 percent of users at 40 percent rollout.

The first thing I'd say is that I can't roll back. Google Play won't let me decrease a staged rollout percentage, and users who have the build keep it. So halting freezes the affected population at forty percent, it doesn't shrink it, and those users keep crashing until a new build reaches them.

Halt first, before diagnosing, because it costs nothing and stops the exposure growing.

Then get crash-free users alongside crash-free sessions, because the ratio tells me whether it recurs: if users dropped more than sessions, it's a startup or persistent-state path rather than a rare interaction. Then segment: OS version, device, locale, and crucially upgrade versus fresh install. That last one is decisive more often than people expect, because if fresh installs are clean and upgrades crash, it's a migration bug against data written by the previous version, which no internal test could catch since test devices are clean installs.

Then the question that determines everything: is there a server-side kill switch for the affected feature? If yes, flip it and the crash stops for everyone including users already on the bad build, and I've turned a three-day incident into twenty minutes. If no, the only fix is a new build through review, which is days.

Which is why the policy is that every feature ships dark: the binary goes out with the feature flagged off, propagates for days, and is enabled server-side at one, five, twenty-five and a hundred percent. On the server a bad deploy is fixed by rollback; on mobile a bad binary is fixed by a flag or it isn't fixed for days.

And I'd expect the postmortem to find the alert fired late, because staged rollout mathematically dilutes a segmented failure: a crash hitting a hundred percent of one OS version at twenty percent rollout looks like noise in the aggregate. So alerts have to be per OS version and per upgrade path.

Depth signal: the inability to roll back as the framing, the upgrade-versus-fresh-install segmentation, and the alerting dilution.

Full treatment: A mobile crash at 40 percent rollout.

Drill 66. Rename a column across a 3-version compatibility window. Sequence it.

Five deploys, and the reason it's five rather than one is that a rolling update isn't atomic: both versions serve traffic for minutes, so a bare RENAME COLUMN throws on every pod that hasn't been replaced yet. And rollback is a deploy backwards, so the old version has to work against the new schema too.

Deploy one: add the new column, backfill in batches with a trigger keeping both in sync in both directions. The trigger is what makes the middle phase safe, because whichever column a given version writes, both stay correct.

Deploy two: application writes both, reads the old. Rollback is free.

Deploy three: reads the new, still writes both. This is the load-bearing one, because it's the first time the new column matters, so it gets a canary and a completeness check before it, and a rollback here just restores the old read path with no schema change to undo.

Deploy four: stop writing the old column. Then wait a full release cycle, or however long the longest-lived client survives.

Deploy five: drop the trigger and the column.

On the database side specifically, the two things I'd get right: SET lock_timeout on the migration, because the danger isn't the operation's duration, it's that it queues behind a long-running transaction and then every subsequent query queues behind it. And the backfill in five-thousand-row batches with a short sleep, because a single UPDATE over forty million rows holds a long transaction, generates enormous WAL, blocks autovacuum and lags replicas.

And I'd size the ceremony by the rolling-update window rather than by the table. Minutes at production volume justifies five deploys. An internal tool with three users doesn't, and I'd take the lock deliberately and say so.

Depth signal: the trigger for bidirectional sync, deploy three as the load-bearing one, and lock_timeout on the DDL.

Full treatment: Expand and contract and Online schema change.

How to practise these

Four of these six begin by correcting a premise, and the correction is the answer:

Drill 61  "the canary shows it's winning"
          -> a canary answers "is it safe", not "is it
             better", and cannot answer the second at that
             sample size
Drill 62  "propose a canary schedule"
          -> first compute what it can detect, which at
             200 QPS is less than you would want
Drill 64  "the rollback failed"
          -> the rollback worked; the new version left state
             the old one can't read
Drill 65  "roll it back"
          -> you can't. Halting freezes the population, it
             doesn't shrink it

Three tests for your own answer:

1. Did you correct the premise before answering it? In four of six, answering the question as asked produces a wrong answer, and the correction is where the signal is.
2. Did you produce the number? Six thousand samples detecting 0.9 percentage points. Four seconds against twenty-five minutes. Twenty-four hours of poisoned cache entries. These are what make the answer verifiable.
3. Did you name the general habit, not just the specific fix? For drill 64 the specific fix is a versioned cache key and the general one is the five-minute pre-deploy question: what state does this version write that the previous one can't read. The general habit is worth more.

And the delivery note for the whole batch: these are the drills where being the person who says "actually, that instrument can't answer that question" is the point. Doing it gently, with the reason and an alternative, is the difference between correcting a PM usefully and correcting them annoyingly.