Chaos-testing an untested assumption

"Name an untested assumption in your architecture. How would you chaos-test it?"

What it is

Chaos engineering is running controlled experiments against a production or production-like system to test a stated hypothesis about how it behaves under failure. The emphasis is on both words: an experiment has a hypothesis and a control, and "controlled" means a blast radius you chose in advance and can stop.

The formal loop, from the Principles of Chaos Engineering:

1. Define STEADY STATE as a measurable output that indicates normal behaviour
   ("checkout success rate stays above 99.5%"), not an internal metric.
2. Hypothesise that steady state CONTINUES in both the control group and
   the experimental group.
3. Introduce a real-world fault: instance loss, latency, dependency failure,
   region loss, disk fill, clock skew.
4. Try to DISPROVE the hypothesis.
5. The smaller the blast radius, the more you learn per unit of risk.

Commonly confused with "randomly breaking things", which is what makes leadership refuse it. It is the opposite: you cannot run an experiment without a hypothesis, because with no hypothesis there is no result, only an incident.

Also commonly confused with fault injection testing generally. Injecting a fault in a test environment is useful and is not chaos engineering, because the value comes from the parts of production you did not model: real traffic, real data volumes, real timeouts, real dependencies, real people responding.

The problem it solves

Every distributed system contains assumptions that have never been executed.

Not unknown unknowns. Specific, written-down beliefs:

• "If the recommendations service is down we degrade gracefully."
• "The circuit breaker opens after 5 consecutive failures."
• "The read replica takes over within 30 seconds."
• "Our retry policy is bounded and will not amplify."
• "The cache is optional; a cold cache is survivable."

Each of those is in a design document. None of them has been executed since it was written, and the code around them has changed dozens of times. The chance that all of them are still true is low, and you find out during an incident, which is the worst possible time and the most expensive way.

The specific class of bug chaos engineering catches, and nothing else does: failure handling code that is never otherwise executed. Your happy path runs a billion times a day. The catch block that is supposed to serve a fallback runs approximately never, so it silently rotted three refactors ago.

Mechanics

Writing a real experiment

The document is short and every field earns its place.

EXPERIMENT: Recommendation service failure degrades product page gracefully

Assumption under test
  The product page renders without recommendations if the recs service
  is unavailable, within the normal latency budget.

Steady state (user-visible, measured before, during and after)
  product_page_render_success_rate >= 99.5%
  product_page_p99_latency <= 800 ms
  add_to_cart_rate within 2% of the same hour last week

Hypothesis
  Steady state holds when 100% of recommendation calls fail.

Blast radius
  Round 1: 1% of traffic, one pod, staging-like canary segment
  Round 2: 5% of production traffic in one AZ
  Round 3: 100% of one region

Abort conditions (automated, not judgement calls)
  render_success_rate < 99.0%    -> auto-abort
  p99 latency > 1500 ms          -> auto-abort
  any 5xx rate > 0.5%            -> auto-abort
  a human says stop              -> abort

Duration       15 minutes per round
Scheduled      Tuesday 14:00 local, business hours deliberately
Owner          [name]. On-call notified, incident channel open.
Rollback       Remove the fault injection rule. Under 10 seconds.

Two design choices in there are the ones interviewers listen for.

Business hours, deliberately. Running chaos experiments at 3am to minimise impact is exactly backwards: you want the people who understand the system awake, watching, and able to respond. If an experiment is too risky to run at 2pm, it is too risky to run at all, and that is a finding in itself.

Automated abort conditions. A human watching a dashboard is a slow and unreliable circuit breaker. The abort thresholds are code, and they fire in seconds.

The fault menu

What you can actually inject, roughly in ascending order of blast radius:

Latency injection is the highest-value place to start, because it is the most realistic failure (dependencies rarely die, they slow down), it is the one systems handle worst, and it exposes timeout and thread-pool problems that a clean failure never does.

# Envoy / Istio: 100% of calls to recs get 2 s of added delay.
# A clean failure is easy to handle; slowness is what kills you.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
spec:
  hosts: ["recommendations"]
  http:
    - fault:
        delay:
          percentage: { value: 100.0 }
          fixedDelay: 2s
      route:
        - destination: { host: recommendations }

Game days

The experiment tests the system. A game day tests the system plus the humans plus the runbooks plus the alerting, which is the part that actually fails during incidents.

Game day structure (2-3 hours)

  Before   Publish scenario category, not specifics. Confirm the abort
           authority. Confirm nobody is mid-deploy.
  Run      Inject. Do NOT tell responders what you injected.
           Observe: how long to detect, how long to diagnose, whether
           the runbook matched, whether the alert fired at all.
  After    Blameless review. The findings are usually about
           observability and runbooks, not about the code.

The metric that matters is time to detection. In practice a large share of game days find that the fault was never alerted on at all, and someone noticed a graph. That is the finding, and it is worth more than whether the fallback worked.

Maturity, and where to start

Level 0  Fault injection in tests and staging. Cheap, and it catches
         the code that never compiled against a failure path.
Level 1  Game days in production, manual, announced, business hours,
         tiny blast radius.
Level 2  Automated experiments in production, scheduled, auto-abort.
Level 3  Continuous chaos in CI/CD: every deploy runs a small
         experiment suite before promotion.

Do not start at level 2. Starting with automated production chaos in an organisation that has never done a game day is how the practice gets banned after the first incident.

The prerequisites, honestly

Chaos engineering is not the first thing to do. It requires:

• Observability good enough to detect the fault you injected. If you cannot see it, the experiment produces no information.
• A blast radius you can actually control, which means traffic routing you can target, not a global config flag.
• A rollback measured in seconds.
• Organisational agreement, in advance and in writing, that an experiment causing an incident is an acceptable outcome that gets reviewed blamelessly.

Missing any of those, the correct answer is to fix that first and say so, which is itself a strong interview answer.

A worked example

The untested assumption I would name: "Our search service degrades gracefully when OpenSearch is slow, because the circuit breaker opens and we serve cached popular results."

That belief is in the design document, the circuit breaker is configured, and nothing has exercised it since it was written eighteen months ago.

Round 1: the smallest possible test, on one pod.

Fault:      500 ms latency on 100% of OpenSearch calls, one pod, 15 min
Hypothesis: circuit breaker stays CLOSED (500 ms is under the 1 s timeout),
            p99 rises to about 700 ms, success rate unchanged.

Result:     Success rate unchanged. p99 rose to 740 ms. Hypothesis held.
            But: the thread pool queue depth went from 2 to 48, which
            nobody predicted and which is the interesting finding.

Round 2: past the timeout.

Fault:      2 s latency, one pod, 15 min
Hypothesis: calls time out at 1 s, breaker opens after 5 failures,
            cached results served, success rate stays above 99.5%.

Result:     HYPOTHESIS DISPROVED. Three findings:

  1. The breaker opened as designed, in 6 seconds. Good.
  2. The cached-results fallback threw NullPointerException, because a
     refactor four months ago changed the cache value shape and the
     fallback path was never executed in any test. Every request in the
     fallback path returned a 500.
  3. The alert did not fire. The SLO alert is on overall error rate,
     and one pod out of forty is 2.5% of traffic, under the threshold.
     Nobody would have noticed until it was several pods.

Finding 2 is the whole justification for the practice. A catch block that had not executed in production since a refactor, silently broken, sitting behind a circuit breaker that worked perfectly. No test caught it because no test exercised the path with realistic data. It would have surfaced during a real OpenSearch degradation, at which point the "graceful degradation" would have converted a slow search into a total outage.

Finding 3 is the second-most valuable, and it is about observability rather than code: the alerting could not detect a single-pod degradation, which means the blast radius has to grow before anyone knows.

What happens next:

1. Fix the fallback. Add a test that exercises it with the current
   cache value shape.
2. Add a per-pod error rate alert alongside the aggregate SLO alert.
3. Re-run round 2. Hypothesis now holds.
4. Automate round 2 as a weekly experiment with auto-abort, so the
   fallback cannot silently rot again.

Step 4 is the part people skip and it is the one that compounds. A one-off experiment finds one bug. A scheduled experiment converts "the fallback works" from a belief into a continuously verified property.

Production evidence

Netflix's Chaos Monkey (2011) and the later Simian Army established the practice; Netflix's stated goal was making instance failure so routine that the system was built to assume it. Their Chaos Automation Platform (ChAP) is the more instructive system: it runs experiments with a control and an experimental group on a small percentage of traffic and compares them statistically, which is closer to what the practice should look like than random instance killing.

AWS Fault Injection Service is a managed offering for instance, API, network and AZ-level faults with stop conditions tied to CloudWatch alarms. Its existence, and the fact that AWS documents AZ-loss experiments as a supported pattern, is evidence the practice has moved from novelty to expected discipline.

The Principles of Chaos Engineering (principlesofchaos.org, authored by the Netflix team) is the canonical statement of the hypothesis-driven framing, including "minimise blast radius" and "run experiments in production".

Gremlin, Chaos Mesh (CNCF) and LitmusChaos (CNCF) are the tooling ecosystem; Chaos Mesh's fault catalogue is a useful checklist of what is injectable in Kubernetes.

Google's DiRT (Disaster Recovery Testing) programme is the game-day form at scale, and its published accounts emphasise the same finding: the failures discovered are usually in the human and process layer rather than in the code.

The debate

The case for: failure-handling code is the least-executed code in the system and therefore the most likely to be broken. Only production has real traffic, real data and real dependencies, so only production experiments test what actually happens. And turning "we believe the fallback works" into a continuously verified property is a categorical improvement over a design document.

The case against: it is a mature-organisation practice with real prerequisites, and running it without them means causing incidents while learning nothing, because you cannot observe the result. A team with a 40 percent change failure rate and no observability should fix those first; chaos experiments will only find what its incidents are already telling it. And the practice has a reputational failure mode: one badly-scoped experiment that causes a customer-visible outage can get it banned for years.

My position: start at level 0 and level 1 and be honest that most organisations should not be at level 2. Concretely: fault injection in staging first, then announced game days in production during business hours with a 1 percent blast radius and automated abort conditions. Only automate an experiment once it has been run manually and its hypothesis has held twice.

The prerequisite I would not compromise on is observability sufficient to detect the fault you injected. If you inject 2 seconds of latency into a dependency and no dashboard changes, the experiment has produced no information and you have taken risk for nothing. In that situation the correct recommendation is to fix observability first, and saying that is a stronger interview answer than describing a sophisticated chaos programme.

And the thing I would insist on regardless of maturity: latency injection before failure injection, because dependencies rarely die cleanly, they slow down, and systems handle slow far worse than dead.

Follow-up Q&A

"Name an untested assumption and how you'd chaos-test it." I would name a specific belief from a design document, for example "search degrades gracefully when OpenSearch is slow, because the circuit breaker opens and we serve cached popular results". Then I would define steady state in user-visible terms, not internal metrics: search success rate above 99.5 percent and p99 under 800 ms. Hypothesis: steady state holds when OpenSearch calls take 2 seconds. Blast radius: one pod, fifteen minutes, business hours. Automated abort if success rate drops below 99 percent. Then run it and try to disprove the hypothesis.

"Why business hours? Isn't that reckless?" It is the opposite. You want the people who understand the system awake and watching, able to diagnose and respond. Running at 3am minimises the number of customers affected and also minimises the number of people capable of noticing what went wrong, which defeats the purpose. And the test is useful in itself: if an experiment is too risky to run at 2pm, it is too risky to run, and that is a finding about the system rather than about the schedule.

"What do these experiments actually find?" In my experience three things, in roughly this order. Fallback code that has silently rotted, because it is the least-executed code in the system and a refactor broke it with no test covering it. Alerting that cannot detect the failure at a small blast radius, so the aggregate SLO alert stays quiet while one pod is completely broken. And runbooks that no longer match the system. Notice that two of the three are observability findings rather than code findings, which is consistently what game days surface.

"What if leadership won't approve production experiments?" Then I start at level 0, which is fault injection in staging and in tests, and build the case with findings. The argument that works is not "chaos engineering is best practice", it is "here is a specific belief in our design document, here is the experiment, and here is what we found in staging". One broken fallback discovered cheaply is a better argument than any amount of principle. And I would frame the production step as a game day with a one percent blast radius and automated abort, because "announced experiment with a stop button" is a very different proposal from "randomly break production".

"What are the prerequisites?" Four. Observability good enough to detect the fault you injected, otherwise the experiment produces no information. Traffic routing precise enough to control the blast radius, which means a mesh or a flag, not a global config change. A rollback measured in seconds. And written agreement in advance that an experiment causing an incident is an acceptable, blamelessly-reviewed outcome. Missing any of those, fixing it is the higher-priority work.

"Latency or failure injection first?" Latency, always. Dependencies rarely fail cleanly; they slow down. And systems handle slow much worse than dead, because a clean failure trips a circuit breaker while slowness fills thread pools, exhausts connection pools, and propagates backpressure into callers that have no idea why. Most of the interesting findings come from the latency experiments.

Common misconceptions

"It means randomly breaking things." An experiment without a hypothesis is an incident. The hypothesis, the steady-state metric and the abort condition are what make it an experiment.

"You need to start in production." Level 0 fault injection in tests finds real bugs cheaply and builds the organisational case.

"Bigger blast radius means better learning." The Principles say the opposite: minimise blast radius, because the goal is information per unit of risk.

"It replaces testing." It tests what only production has: real traffic, real data, real dependencies, real people. Unit and integration tests still do their job.

"The findings are about code." Most are about alerting and runbooks. Time to detection is the metric that matters most in a game day.

Interview delivery note

Name a specific assumption immediately, because a generic answer here is a weak one: "The one I'd pick is 'our search degrades gracefully when OpenSearch is slow, because the circuit breaker opens and we serve cached popular results'. That's in the design doc, it's configured, and nothing has executed it since it was written."

Then run the experiment structure out loud, since that is what is being scored: "Steady state in user-visible terms, so search success rate above 99.5 percent and p99 under 800 milliseconds, not an internal metric. Hypothesis: that holds when I add two seconds of latency to every OpenSearch call. Blast radius: one pod, fifteen minutes. Automated abort conditions, not a human watching a dashboard. And business hours, deliberately, because I want the people who understand the system awake."

The line that lands hardest is the finding: "and when I've run experiments shaped like this, the thing they find is that the fallback path throws, because it's the least-executed code in the system and a refactor broke it months ago with no test covering it. The circuit breaker worked perfectly and opened onto a broken fallback."

Close with the prerequisite, because it shows judgement rather than enthusiasm: "but I wouldn't start here in most organisations. If I inject two seconds of latency and no dashboard moves, I've taken risk and learned nothing, so observability comes first. And latency injection before failure injection, because dependencies rarely die cleanly, they slow down, and systems handle slow far worse than dead."

Further reading

• Principles of Chaos Engineering (principlesofchaos.org), the canonical statement of the hypothesis-driven method.
• Basiri et al., "Chaos Engineering" (IEEE Software, 2016), the Netflix team's paper, and the ChAP write-ups on the Netflix Tech Blog.
• AWS Fault Injection Service documentation, particularly stop conditions and the AZ-availability-power-interruption scenarios.
• Chaos Mesh (CNCF) documentation, whose fault catalogue doubles as a checklist of what is injectable.
• Rosenthal and Jones, Chaos Engineering: System Resiliency in Practice (2020), for the maturity model and game-day mechanics.