The vocabulary of systems, failure and design

What it is

The terms senior engineers use to describe how a system behaves under stress and how a design treats the people who use it. Two families sit on this page because they are the same subject from two ends: the design vocabulary (footgun, pit of success, seam, escape hatch) describes whether a system is easy to use correctly, and the failure vocabulary (backpressure, load shedding, retry storm, brownout) describes what it does when it cannot keep up. A system with sharp edges fails in ways a system with guardrails does not.

Most of these have precise technical referents elsewhere in this book. What this page adds is the conversational use: when to reach for the word, what it commits you to, and how it misfires. A term used loosely by someone who does not know its mechanism is detectable in one follow-up question.

Don't be confused: backpressure and load shedding are opposite responses to the same condition. Both trigger when a consumer cannot keep up. Backpressure propagates the slowness upstream, so the producer slows down and nothing is lost. Load shedding discards work, so throughput is preserved for what remains and something is lost. The choice is not stylistic: backpressure is right when the producer can slow down and the work must not be lost (a batch pipeline, a file upload), and shedding is right when the producer cannot be slowed and lateness is worthless (live requests, a stock feed). Using them as synonyms is the most common way to fail a follow-up on this material.

The problem it solves

Under pressure, the words available to a team determine what it can discuss. A team without "backpressure" reaches for "add a bigger buffer", which is the failure mode the word exists to prevent. A team without "pit of success" argues about whether developers should be more careful, which never works, rather than about whether the default is wrong, which does.

In an interview the job is narrower and sharper. These terms let you describe a failure mode in three words that would otherwise take a paragraph, which buys you time to say something substantive, and they signal that you have operated a system rather than only built one.

Mechanics

Whether the design helps or hurts its user

Footgun: something that makes it easy to accidentally harm yourself. "Letting callers pass an arbitrary timeout here is a footgun." The word carries a specific accusation, that the danger is available by default rather than requiring effort, and that is what makes it different from "this API is complicated". Reach for it when the failure requires no mistake, only inattention.

Sharp edge: a feature that works but is easy to misuse with serious consequences. "The client works, but it has sharp edges around retries." Softer than footgun, and the right word when the danger is real but requires the user to be doing something reasonable-looking.

Pit of success: a design where the easiest, default behaviour is also the correct one. "Make idempotency automatic so callers fall into the pit of success." Rico Mariani's phrase from Microsoft, and it is the constructive counterpart to the two above. It is the single most useful design term on this page for a lead, because it redirects a conversation from "how do we get people to do the right thing" to "why is the right thing not the default", and the second question has answers.

Leaky abstraction: an abstraction whose implementation details still reach the user. "Our repository layer is leaking database semantics." Joel Spolsky's law is that all non-trivial abstractions leak to some degree, so the useful form is never "this leaks" but "this leaks in a way that costs us something specific". An ORM that leaks N+1 query behaviour is a leak that matters; one that leaks connection-pool naming is not.

Escape hatch: a deliberate, supported way to bypass the abstraction when it does not fit. "The framework needs an escape hatch for unusual query patterns." The important word is deliberate. Every abstraction will be bypassed; the choice is whether the bypass is designed, documented and observable, or whether it is a hack that spreads. A platform without escape hatches does not get compliance, it gets shadow infrastructure, which is the argument to make when a platform team proposes to lock something down.

Seam: a place where behaviour can be changed or substituted without editing the surrounding code. Michael Feathers's term from Working Effectively with Legacy Code, and the most useful single concept for talking about untestable systems. "We need a seam around the payment provider before we can test this safely." It reframes "this code is untestable" as a concrete structural request, which someone can actually do.

How you change a system without stopping it

Strangler pattern: incrementally replace a legacy system by routing traffic to new components until the old one is unused, rather than rewriting and switching. "Put the new recommendation API in front and strangle the legacy path incrementally." Martin Fowler's name for it, after the strangler fig. The reason it wins arguments is that it is the only rewrite approach that produces value before it finishes and that can be abandoned halfway without having wasted everything.

Walking skeleton: a minimal end-to-end implementation proving every major component connects, before any of them do anything useful. "Before we build ranking logic, get a walking skeleton from ingestion through to serving." Tracer bullet is the close cousin from The Pragmatic Programmer, emphasising discovery rather than structure: a thin path through the whole system fired early to find out what you do not know. "Build a tracer bullet through Kinesis, embeddings, OpenSearch and the API."

Thin vertical slice: one small feature built end to end, rather than one architectural layer built completely. "Do one thin vertical slice before building the whole data platform." All three of these share a single argument, and it is worth being able to state it directly: integration risk is the risk that does not decompose. You can estimate each component accurately and still be badly wrong about the whole, because the interactions are where the surprises are, and horizontal layer-by-layer construction defers every interaction to the end. This is also the practical content of Gall's law.

What a system does when it cannot keep up

Backpressure: the mechanism by which a slow consumer causes producers to slow down. "The consumer needs to propagate backpressure instead of buffering indefinitely." The failure it prevents is unbounded buffering, and the sentence that gives it teeth is that an unbounded queue does not solve a throughput mismatch, it converts it into a memory failure and adds latency while it does so.

Load shedding: deliberately rejecting lower-priority work to protect the rest. "At 95 percent saturation we shed recommendation refresh requests." The lead-relevant point is that shedding is a decision about priority, so the design question is not "should we shed" but "what is the priority order and who decided it", and that question usually has no owner.

Graceful degradation: continuing with reduced functionality rather than failing entirely. "If personalisation is unavailable, degrade to popular articles." Brownout is the deliberate, temporary version: switching off expensive optional functionality during overload. "During peak we brown out the expensive recommendations."

Fail open and fail closed describe what happens when a dependency is unavailable: does the operation proceed or is it rejected? "For authorisation, we fail closed." This is a genuine decision with no universal answer, and the deciding variable is what the check is protecting against. Security checks fail closed, because an unavailable authoriser must not become an open door. Availability-oriented checks (a recommendation service, a feature flag lookup) fail open. The dangerous case is a check that everyone assumes fails closed and which was implemented to fail open for availability reasons, usually years earlier, usually undocumented, which is a specific and excellent thing to go and check on your own systems.

Defense in depth: multiple independent protective layers, so that any single failure is not sufficient. "Rate limits, authentication, quotas and tenant isolation give us defence in depth." The load-bearing word is independent: three controls that all depend on the same identity service are one control wearing three hats, and saying so is a good way to demonstrate that you understand the term rather than merely using it.

Head-of-line blocking: one slow item delaying unrelated work behind it. "Large jobs are causing head-of-line blocking in the queue." Thundering herd: many clients acting simultaneously, usually after a shared trigger. "Cache expiry causes a thundering herd against Postgres." Retry storm: retries amplifying an outage rather than recovering from it. "Without exponential backoff and jitter, a brief dependency blip becomes a retry storm." Death spiral: a feedback loop where degradation creates load that creates more degradation. "Latency rose, clients retried, retries raised load, and we entered a death spiral."

Those four are a progression and it is worth saying so, because the progression is what makes the story coherent in a postmortem: head-of-line blocking creates latency, latency triggers retries, retries create a herd, the herd creates a death spiral. A candidate who narrates an outage along that chain sounds like someone who has been in one.

The guardrail family

Paved road or golden path: the supported, recommended way of doing something, which is easier than the alternatives rather than merely mandated. "Our deployment template on Kubernetes is the paved road." Guardrails, not gates: prefer automated constraints that catch problems to human approvals that hold things up. "Security should provide guardrails, not gates."

Both encode the same position, which is worth stating as a position rather than a slogan: compliance obtained by making the right thing easy is durable, and compliance obtained by review is a tax that people route around. The honest limit, and you should volunteer it, is that guardrails only work where the constraint is machine-checkable. Judgement calls (is this design sound, is this vendor acceptable) still need people, and pretending otherwise produces automation theatre.

Shift left: move validation earlier, into development and CI. "We shifted schema compatibility checks left into CI." Shift right: validate in production with observability, canaries and experiments, on the argument that some properties only exist under real traffic. "Some reliability properties can only be validated by shifting right." They are complements rather than alternatives, and the mature position is that you shift left everything that is cheaply checkable and shift right everything that is only true in production, which is more than people expect: real load shapes, real data distributions, real client behaviour.

Complexity as a budget

Accidental complexity comes from your implementation choices; essential complexity is inherent to the problem. Brooks's distinction, and the reason it matters is that only one of them is removable. "Half of this workflow is accidental complexity from the framework." "Exactly-once financial posting is essential complexity, and no abstraction removes it."

Complexity budget: the informal limit on how much a team can operate. "Kafka would work, but I am not sure this justifies the complexity budget." Change surface: how much of the system a modification touches. "Can we reduce the change surface by putting the compatibility logic at the boundary?"

The reason to have these as nouns is that they make an otherwise unwinnable argument winnable. "This is too complicated" is an aesthetic objection and loses to a concrete benefit. "This spends complexity budget we are already overdrawn on, and here is the on-call burden that proves it" is a cost, and costs can be weighed.

Worked example

An incident review, narrated twice.

Without the vocabulary: "The database got slow, and then everything got slower, and there were a lot of requests, and eventually it fell over. We restarted things and it recovered. We should probably add some limits."

With it: "A slow query in the reporting path caused head-of-line blocking on the shared connection pool, so unrelated requests queued behind it. Latency crossed the client timeout, so clients retried, and because the retry policy had no backoff or jitter that became a retry storm, roughly tripling offered load at the moment we had least capacity. That is a death spiral: the system's response to overload increased the overload. We recovered by restarting, which worked only because it dropped every in-flight retry.

Three fixes at different levels. Immediately, backoff with jitter on the client, which breaks the amplification. Structurally, a separate connection pool for reporting so the failure domain does not include the serving path, which turns this from an outage into a degraded report. And the real fix is load shedding: at 90 percent pool saturation we should reject reporting queries with a clear error rather than queueing them, because a report that arrives four minutes late is worth nothing and it cost us the serving path. The thing I want to change culturally is that the unbounded queue looked like resilience and was actually the mechanism of the failure."

The second version is not longer by much and is a completely different signal. It names a causal chain rather than a sequence of events, it identifies the amplifying feedback loop explicitly, it separates fixes by time horizon, and it ends with the generalisable lesson. Every one of those moves is enabled by having the words.

Production evidence

Backpressure is a specified part of the Reactive Streams standard (now java.util.concurrent.Flow in the JDK), where request(n) is literally the consumer telling the producer how much it can take. That it needed standardising across Akka, Reactor, RxJava and Vert.x is evidence of how often the naive version was got wrong.

Load shedding at Netflix and Google is documented practice rather than theory. Google's SRE book devotes a chapter to handling overload, covering client-side throttling and criticality levels so that shedding is priority-aware, and Netflix has published on concurrency limits and prioritised load shedding at the edge. The recurring finding in both: the failure of an overloaded system is usually caused by the system's own retry behaviour, which is why adaptive concurrency limits and circuit breakers exist.

Thundering herd mitigation is why cache libraries ship jitter. The standard fixes, randomised TTLs, request coalescing so only one caller recomputes, and probabilistic early expiration, are built into mature caching layers precisely because the naive implementation reliably produces synchronised expiry.

The paved road model is how large platform organisations actually operate. Netflix's "paved road" language is the widely-cited origin, and Spotify's "golden path" is the same idea; in both, the road is optional but supported, and the deal offered to teams is explicit: take the road and get deployment, observability and compliance handled, or leave it and own those yourself. The optionality is the design, not a weakness of it, because a mandatory road gets no feedback about whether it is any good.

Strangler applications are the documented approach for most large legacy migrations that succeeded, and the pattern name comes from Martin Fowler's 2004 article. The counter-evidence is instructive too: the well-known big-bang rewrite failures (the Netscape 6 rewrite is the canonical software-industry example, described in Joel Spolsky's "Things You Should Never Do") are what the pattern exists to avoid.

The debate

Backpressure or load shedding, when both are available? The deciding variable is whether the work retains value when delayed. If it does, propagate backpressure, because dropping work that is still worth doing is pure loss. If it does not, shed, because queueing worthless work consumes capacity that valuable work needs. My default for user-facing request paths is to shed, and for data pipelines is backpressure, and the mistake I see most often is applying the pipeline instinct to a request path: buffering user requests during overload produces a queue of responses nobody is waiting for any more, and the system spends its recovery capacity answering abandoned questions. Say the phrase "we should reject rather than queue, because a late answer here is worth nothing" and you have made the whole argument.

Is "guardrails, not gates" always right? No, and treating it as absolute is a recognisable inexperience tell. Gates are correct where the action is irreversible, rare, and high-consequence: production data deletion, a change to how money moves, granting standing production access. The cost of a gate is proportional to how often it triggers, so a gate on a rare, catastrophic action is cheap and a gate on a daily action is ruinous. The refined position: automate everything checkable into guardrails, then place a small number of gates on the irreversible actions, and be able to name every gate and why it exists. An organisation that cannot enumerate its gates has accumulated them rather than chosen them.

Should every abstraction have an escape hatch? Mostly yes, with one real exception. Escape hatches undermine invariants you actually need to hold: if your platform guarantees that all traffic is authenticated and observable, an escape hatch that bypasses the mesh voids the guarantee, and the guarantee was the product. The resolution is to distinguish invariants from conveniences. Provide escape hatches around conveniences (query patterns, serialisation, deployment shape) and refuse them around invariants, while making the invariant path fast enough that nobody wants out. If people are routing around an invariant, the invariant is either wrong or too expensive, and that is worth learning rather than policing.

Follow-up Q&A

How do you actually implement backpressure across a network boundary? You need an explicit protocol-level mechanism, because TCP's flow control only tells you the socket buffer is full, which is far too late and too coarse. The real options: a credit or request-based protocol (Reactive Streams' request(n), gRPC streaming flow control, RSocket's leasing), a bounded queue with a blocking or rejecting put so that fullness propagates as slowness or as an error, or an explicit rate negotiated between the parties. The anti-pattern is an unbounded queue between services, which converts a throughput mismatch into unbounded latency followed by an out-of-memory failure. If you cannot introduce a real mechanism, a bounded queue that rejects is still far better than an unbounded one that lies.

How do you choose the shedding threshold? Not from CPU, which is the common mistake, because CPU is a lagging and non-linear signal near saturation. Shed on queueing delay or on concurrency measured against a limit, because those rise before throughput collapses. The adaptive approach (Netflix's concurrency-limits work, based on TCP congestion-control ideas like Vegas) measures latency continuously and infers the concurrency limit rather than requiring you to configure one, which matters because the right limit changes with instance type, dependency health and traffic mix. Start with a static limit derived from a load test, then move to adaptive when the static one is visibly wrong, and instrument shed events as a first-class metric so you can tell shedding from failure.

How is graceful degradation different from a circuit breaker? A circuit breaker is a mechanism that stops calling a failing dependency; graceful degradation is the product decision about what to do instead. The breaker gives you a fast failure; degradation is what fills the hole. A circuit breaker without a degradation plan just converts a slow failure into a fast one, which helps your latency and does nothing for your user. The interview-relevant point is that the degradation plan is a product conversation, not a technical one: someone has to decide that popular articles are an acceptable substitute for personalised ones, and that person is usually not an engineer.

Which is the more dangerous default, fail open or fail closed? Fail open, because its failure is silent. A fail-closed system that breaks causes an outage, and everyone finds out in minutes. A fail-open system that breaks keeps serving, and you discover months later that the authorisation check has been returning true since a deploy in March. So my default is to fail closed unless there is a stated availability reason to do otherwise, and to alert loudly on every fail-open event, because the thing that makes fail-open acceptable is knowing it happened. An unmonitored fail-open path is a vulnerability with a deployment date.

When is a walking skeleton a waste of time? When the integration risk is genuinely already retired: you are adding the fourth service to an established platform, the deployment path is known, the interfaces are the same as the last three. Then the skeleton proves something you already know, and the honest thing is to skip it. The test is whether you can name something the skeleton would tell you that you do not currently know. If you cannot, it is ritual. If you can (does the embedding service actually handle our payload size, does the network path between these two VPCs exist), build it this week.

Is technical debt a useful metaphor or a harmful one? Genuinely contested, and worth having a view. It is useful because it captures the interest payment: a shortcut has a recurring cost, not a one-time one, which is why "debt service" is the more precise phrase and why "that integration consumes 20 percent of the team's capacity in debt service" is a sentence that gets funding. It is harmful because it implies the debt was deliberately taken on at a good rate, which flatters most of what gets called debt, and because it suggests any debt can be repaid, when some of it is better characterised as a design you would not choose and cannot afford to replace. My position: keep the metaphor for deliberate, identified shortcuts with a known payoff, and use plainer language ("this design is wrong for what we now need") for the rest, because conflating them lets genuinely bad architecture hide inside a respectable-sounding category.

Common misconceptions

"Backpressure means adding a queue." It means bounding the queue and propagating fullness. An unbounded queue is the absence of backpressure with extra steps.

"Retries improve reliability." Retries improve reliability against independent, transient failures and actively harm it against correlated ones, which is what an overload is. Retries without backoff, jitter and a budget are an availability risk, not a mitigation.

"Defence in depth means more layers." It means more independent layers. Layers sharing a failure mode add cost and complexity while adding no depth, and the shared dependency is usually identity or DNS.

"A leaky abstraction is a bug." All non-trivial abstractions leak. The question is whether the leak costs anything, and the goal is to choose which details leak rather than to eliminate leaking.

"Shift left means shift everything left." Some properties only exist in production, and pretending a staging environment tests them produces false confidence, which is worse than untested. Real traffic shapes, real data skew and real client behaviour are shift-right concerns by nature.

Interview delivery note

The highest-value habit from this page is narrating a failure as a causal chain with a named amplifier, because it is what people who have been in incidents do and it cannot be faked convincingly. "Head-of-line blocking created latency, latency triggered retries, and the retries were the amplifier that turned a slow query into an outage" is one sentence, and it tells the interviewer you have sat through the real thing.

The senior-to-lead separator is which layer your fix lands on. A senior engineer fixes the slow query. A lead fixes the slow query, isolates the failure domain so the next slow query cannot reach the serving path, and then changes the default so that the class does not recur, usually by making the safe thing the easy thing. The last move is the pit-of-success argument applied to your own organisation, and it is what makes a fix durable after you have moved on.

The Staff & Team Lead Interview Reference