Drills 43 to 52: SRE, architecture and delivery

Ten drills, ninety seconds each, out loud. This cluster has a consistent tell: the good answer contains arithmetic and the weak answer contains a practice. "We'd use burn-rate alerting" is a practice. "Two windows because a fast one catches a sudden burn and a slow one catches a leak, at 14.4x and 6x, with a short reset window so it stops firing after the problem passes" is an answer.

The other pattern: several of these are questions where the naive answer creates the problem it was meant to solve. Alerting on every SLO breach creates fatigue. Testing 40 services end-to-end creates a bottleneck. Splitting a team by component creates coordination overhead. Naming that inversion is what lands.

Drill 43. Design burn-rate alerts for a 99.9 percent SLO. Why two windows?

The budget is 0.1 percent over 28 days, which is 43 minutes. Burn rate is how fast you are consuming it relative to the rate that would exactly exhaust it over the window, so a burn rate of one means on track to spend exactly the budget by the end.

Two windows because there are two failure shapes. A sudden total outage burns the budget in hours and needs a page now. A slow leak, an error rate that went from 0.05 to 0.3 percent, burns it over days and would never trip a fast alert, but it will exhaust the budget by the end of the month.

So: 14.4x over one hour, which is 2 percent of the budget in an hour, pages. 6x over six hours, which is 5 percent in six hours, pages. And 1x over three days ticketed rather than paged, because that is the slow leak and nobody needs to wake up for it.

The detail that matters is the short second window on each. A 14.4x alert over one hour also requires the last five minutes to be burning, otherwise it keeps firing for an hour after the incident is resolved. That reset condition is what makes the alerts trustworthy, and without it people learn to ignore them.

And the reason this is better than a threshold alert on error rate: a fixed threshold has no relationship to how much budget you have left. Burn rate alerts on what actually matters, which is whether you are going to run out.

Depth signal: the short reset window, and the framing that a threshold alert has no relationship to the budget.

Full treatment: Burn-rate alerting.

Drill 44. A label explosion took down Prometheus. What happened?

Cardinality, and it kills suddenly rather than gradually. Cardinality is multiplicative across labels, so a metric with fifty services, five methods, eight statuses and twenty endpoints is forty thousand series, and someone adding user_id as a label takes that to eighty billion.

The mechanism: the ingester holds an in-memory inverted index from labels to series, roughly one to three kilobytes per active series, and that grows until it OOMs. And because ingest is sharded by series hash, the bad series are spread evenly across every shard, so every shard dies at the same time rather than one degrading.

The defences in order of when they act. Lint rules in CI rejecting label values drawn from unbounded sources, which is the cheapest place to stop it. metric_relabel_configs at scrape time to drop known-bad labels, which is also the emergency lever during an incident and worth having pre-written. Per-tenant series limits at ingest with a hard reject that names the metric and the offending label. And a weekly cardinality report per team ranked by series count.

The per-tenant limit is the load-bearing one, and the argument is blast radius: without it one team's pull request takes down observability for everyone at exactly the moment everyone needs it. A hard limit makes it one team's problem.

And the thing worth saying to whoever wanted user_id: they want to get from a latency spike to a specific slow request, and the right mechanism is exemplars, a trace id attached to a histogram bucket sample. That answers the actual need without the id becoming a label.

Depth signal: every shard dying simultaneously because of hash sharding, and exemplars as the answer to the underlying request.

Full treatment: Cardinality: the observability cost model.

Drill 45. How do you test 40 microservices without a full environment?

Consumer-driven contract testing. Each consumer declares what it needs from a provider in executable form, its tests run against a mock generated from that contract, and the provider's CI replays every consumer's contract against the real provider. The two sides never run at the same time, so no shared environment is needed.

The arithmetic that makes it tractable: forty services sounds like 1,560 possible pairs, and the real interaction graph is sparse, typically fewer than a hundred actual consumer-provider edges. The work scales with edges, not with the square of the node count.

The mechanism that makes it usable in a pipeline is can-i-deploy: before deploying, the tool checks whether this version's contracts have been verified against the versions currently in the target environment, and blocks if not. That is what turns contract testing from a test suite into a deployment gate.

What it cannot catch is emergent behaviour. Two services can each satisfy their contracts and produce a wrong outcome together, like marking an order shipped before payment settled. So I would keep three to five end-to-end tests for the critical journeys, plus synthetic monitoring in production, and be explicit that contract testing replaces the other ninety-five percent of integration tests rather than all of them.

Depth signal: the sparsity observation, can-i-deploy as the gate, and naming what it cannot catch.

Full treatment: Testing 40 microservices without a full environment.

Drill 46. Deploy vs release: explain the distinction and what it buys you.

Deployment moves bits onto infrastructure. Release exposes behaviour to users. A feature flag decouples them, so you deploy continuously and release deliberately.

The benefit people name first is rollback speed, four seconds for a flag flip against twenty-five minutes for a redeploy, and that is real and it is not the main one. The main benefit is that trunk-based development becomes possible: unfinished work merges behind a disabled flag, which removes long-lived branches and merge hell as a category.

Four flag types with different lifecycles, and conflating them is why flag debt happens. Release flags, days to weeks, deleted after rollout. Operational kill switches, permanent infrastructure. Experiment flags, owned by the A/B platform. And permission or entitlement flags, which are permanent business rules and are not really flags. Only release flags need an expiry policy, and I would enforce it in CI: a release flag older than ninety days fails the build.

And the important part: what does not decouple. Database schema needs expand-contract. Cached and serialised data, because the old code path reads a format it cannot parse unless you version the cache key. Published events, where consumers must deploy before producers. And irreversible external side effects, because you cannot un-send an email with a flag.

Depth signal: trunk-based development as the real benefit, and the list of what does not decouple.

Full treatment: Deploy is not release.

Drill 47. Split this 15-person team. Walk your reasoning.

I would not split by component, which is the obvious answer and the wrong one, because a team owning a component has to coordinate with every other team on any change that crosses components, and most changes do.

The frame I would use is Team Topologies: stream-aligned teams organised around a flow of value, with a platform team providing self-service capabilities, and the measure of a good split is cognitive load. A team should be able to hold its domain in its head, which in practice means owning a bounded context end to end rather than a layer of one.

So for a fifteen-person team I would look for the natural seams in the work, not the code: which changes currently require the fewest people, which parts of the backlog never touch each other, and where are the existing informal sub-teams. Those usually reveal two or three stream-aligned teams of four to six.

The number that matters is how many cross-team handoffs a typical change requires. If a split increases that from zero to two, it has made things worse regardless of how clean the boundaries look on a diagram. And Conway's law says the architecture will follow the split, so I am choosing an architecture whether I intend to or not.

The thing I would not do is split without also deciding what the interaction modes are: collaboration for a period, then X-as-a-service. Leaving that implicit is how you get two teams in permanent collaboration mode, which is the same as one team with extra meetings.

Depth signal: cognitive load as the criterion, handoffs per change as the metric, and naming interaction modes explicitly.

Full treatment: Team Topologies and splitting a team.

Drill 48. When is a modular monolith right, and what forces an extraction?

A modular monolith is right by default, and I would put the burden of proof on extraction rather than on staying.

What you get from the monolith: one deploy, one transaction boundary, refactoring across module boundaries with a compiler checking it, no network between components, and no distributed debugging. The discipline required is enforcing module boundaries in the build, so that modules communicate through defined interfaces rather than reaching into each other's internals.

Three things force an extraction, and only three. Independent scaling, when one component's resource profile is genuinely different, a GPU workload or something needing ten times the memory. Independent deployment, when one component's release cadence is genuinely incompatible, usually because of compliance or because it is on a different risk profile. And team autonomy at a scale where the shared deploy pipeline is a real bottleneck, which is usually above ten teams rather than three.

What does not force it: "microservices are best practice", a desire for clean boundaries which module enforcement gives you anyway, or a slow build, which is a build problem.

Segment's account of consolidating back to a monolith is the useful reference, because their problem was operational surface: a hundred and forty services with shared libraries meant every library update was a hundred and forty deploys, and the coordination cost exceeded the isolation benefit.

Depth signal: exactly three forcing conditions, and putting the burden of proof on extraction.

Full treatment: Modular monolith vs microservices.

Drill 49. Design an experiment: randomisation unit, power, guardrails.

The randomisation unit first, because getting it wrong invalidates the result rather than adding noise. Randomise at the level at which the experience is consistent and the effect operates: request-level only when the change is invisible across requests, user-level for most product changes, and market-level when there is interference.

Interference is the case people miss. In a marketplace, treatment sellers win sales from control sellers, so the measured effect is the treatment's gain plus the control's loss, roughly double the true effect. The fix is randomising by geography so the competition happens within a variant, at a large cost in power because there are far fewer units.

Then power. Sixteen times the variance over the squared effect, so for a three percent baseline conversion detecting a five percent relative lift, about two hundred thousand per variant, which at fifty thousand daily users is eight days, rounded to fourteen for two full weekly cycles. And detecting one percent instead of five needs twenty-five times the sample, so most experiments cannot detect the effect their change actually has.

Then guardrails, and the important part is that they need the opposite error asymmetry from the primary metric. For the primary you control false positives because you do not want a fake win. For a guardrail you are detecting harm, so you control false negatives: a looser alpha and a non-inferiority framing, is it worse by more than two percent rather than is it different.

And one primary metric chosen before the test, plus a sample ratio mismatch check, because expecting fifty-fifty and observing 50.4 over four hundred thousand users is not chance and invalidates the experiment entirely.

Depth signal: interference, the guardrail error asymmetry, and SRM as a hard gate.

Full treatment: Experimentation design.

Drill 50. Explain CQRS and where you would stop on the ladder.

CQRS is a ladder with four rungs, not one thing you adopt, and almost everyone asking about it is imagining rung four and needs rung one.

Rung one is separate command and query handlers, hours of work and no infrastructure. Rung two adds separate read models against the same database, days. Rung three is a separate read store updated in the same transaction, weeks. Rung four makes the projections asynchronous, months, and it permanently changes consistency.

In the case I worked, a team asked for CQRS with event sourcing after a conference talk. Splitting the handlers took an afternoon and moved p99 from 2.4 seconds to 310 milliseconds, because the actual problem was lazy loading on the aggregate. A dedicated read DTO took it to 95. At that point the original complaint was gone and the remaining issues were a read replica for reporting and a search index, which are two different answers.

Rung four is a product decision rather than an engineering one. "Can a customer place an order and not see it in their order list for two seconds" is answered by whoever owns the customer experience, and deciding it yourself because the architecture is more elegant is making a product change without authority.

And I would separate event sourcing explicitly, because the conflation is why teams think this costs months. They are independent, and most teams asking for event sourcing want an audit trail, which an append-only audit table gives without making replay the system's recovery path.

Depth signal: the ladder with measured outcomes at each rung, and rung four as a product decision.

Full treatment: CQRS: the adoption ladder.

Drill 51. Name an untested assumption in your architecture. How would you chaos-test it?

The one I would pick is "our search degrades gracefully when OpenSearch is slow, because the circuit breaker opens and we serve cached popular results". That is in the design document, it is configured, and nothing has executed it since it was written.

The experiment: steady state defined in user-visible terms, so search success rate above 99.5 percent and p99 under 800 milliseconds, not an internal metric. Hypothesis: that holds when I add two seconds of latency to every OpenSearch call. Blast radius: one pod, fifteen minutes. Automated abort conditions, not a human watching a dashboard. And business hours, deliberately, because I want the people who understand the system awake.

When I have run experiments shaped like this, what they find is that the fallback path throws, because it is the least-executed code in the system and a refactor broke it months ago with no test covering it. The circuit breaker worked perfectly and opened onto a broken fallback.

But I would not start here in most organisations. If I inject two seconds of latency and no dashboard moves, I have taken risk and learned nothing, so observability comes first. And latency injection before failure injection, because dependencies rarely die cleanly, they slow down, and systems handle slow far worse than dead.

Depth signal: the abort conditions being automated, business hours with a reason, and observability as a precondition.

Full treatment: Chaos-testing an untested assumption.

Drill 52. You have been down-levelled in an offer. What do you do?

First, find out which of three things happened, because they need different responses: the loop calibrated me lower, the requisition is scoped lower, or it is an anchor. I would reply quickly and warmly, make no decision, and ask that one question without mentioning compensation.

Then the question almost nobody asks: what specifically would have needed to be different? And I would say I want to know either way, whether or not I take the offer, which is true and it makes the answer more honest.

If it is calibration, one written response with three concrete artifacts, naming my own presentation failure if that is what it was, and offering an additional conversation with a staff engineer focused on that gap. That is the highest-conversion move, because it turns my assertion into something they can verify and it is hard to refuse.

If the level holds, two asks. Compensation at the top of the band rather than the middle. And the promotion path, which is the question that actually decides it: who has gone senior to staff on this team, how long did it take, when are the cycles, what work would build the evidence. If the answer is "definitely possible for strong performers", there is no path and I would price the offer as though the level is permanent.

Depth signal: diagnosing which of three situations it is, and the promotion-path answer as the real decision input.

Full treatment: Being down-levelled in an offer.

How to practise these

This is the largest batch and the one where the answers most reward arithmetic. Six of the ten have a number that is the answer: 14.4x and 6x, 43 minutes, 100 edges rather than 1,560, 16σ²/Δ², 2.4 seconds to 310 milliseconds, and 200,000 per variant.

Three tests for your own answer:

1. Did you give a number rather than a practice? "Burn-rate alerting" is a practice. "14.4x over an hour with a five-minute reset window" is an answer, and the reset window is the part that shows you have operated one.
2. Did you name what the technique does not do? Contract testing cannot catch emergent behaviour. A chaos experiment with no observability produces no information. Guardrails control a different error than the primary metric. The boundary is where the credibility is.
3. Did you resist the obvious answer? Splitting a team by component, adopting rung four of CQRS, extracting services because microservices are best practice. Several of these drills are specifically testing whether you take the bait.

And a delivery note for the whole batch: these are the drills where a senior candidate and a staff candidate diverge most visibly, because a senior answer describes the practice correctly and a staff answer says what it costs, what it does not cover, and when not to use it. Every one of the ten has a "and I would not do this when..." available, and including it is the single most reliable upgrade.