Load testing: open vs closed models, and coordinated omission

What it is

A load test measures how a system behaves under a specified arrival of work. The two things that determine whether the answer means anything are the load model and how latency is recorded, and most load testing gets both wrong in the same direction: it reports a system as healthier than it is.

CLOSED MODEL     N virtual users. Each: send request, WAIT for
                 the response, think, repeat.
                 -> arrival rate is a FUNCTION of response time.
                 -> if the server slows, the load goes DOWN.
                 -> self-limiting, and it cannot reproduce
                    overload.

OPEN MODEL       Requests arrive at a specified rate,
                 independent of response times.
                 -> if the server slows, the queue GROWS.
                 -> models internet traffic, and it is the only
                    model that can reproduce overload and
                    metastable failure.

Coordinated omission is the measurement error that accompanies the closed model: a generator that waits for a response before sending the next request does not send requests during a stall, so the requests that would have been slowest are never measured.

What this is confused with: a benchmark. A benchmark compares implementations under controlled conditions. A load test predicts behaviour under production-shaped arrival, and the two need different rigs: a benchmark can legitimately be closed-loop, and a load test usually cannot.

Also confused: throughput and goodput. Under overload a system can maintain high throughput while doing almost no useful work, because it is completing requests whose clients timed out minutes ago.

The problem it solves

A closed-model test tells you the system is fine right up until production tells you otherwise.

Closed test: 500 virtual users, 1 second think time.

  Healthy:      response time 50ms
                arrival rate = 500 / (0.05 + 1) = 476 req/s
  Degraded:     response time 2s
                arrival rate = 500 / (2 + 1) = 167 req/s

The test AUTOMATICALLY REDUCED THE LOAD by 65% the moment the
system got slower.

Production does not do this. Real users' request rate does not
fall because your p99 went up; if anything it rises, because
they retry.

So the closed test cannot find the interesting failure at all:

The failure you are trying to find:
  arrival 500/s, capacity 480/s
  -> queue grows by 20/s, unbounded
  -> latency grows without limit
  -> clients time out at 5s and retry
  -> arrival becomes 700/s
  -> the system never recovers even when arrival returns to
     400/s, because the queue and the retry backlog persist

That is a METASTABLE FAILURE, and no closed-model test can
produce it, because the model's feedback loop is the exact
opposite of the real one.

And coordinated omission hides the severity of any stall you do produce.

Canonical shape:
  Intended: 100 requests/sec for 200 seconds = 20,000 requests.
  The system is perfect for 100s, then stalls completely for
  100s.

CLOSED-LOOP GENERATOR, uncorrected:
  100s of normal operation -> 10,000 requests at ~1ms
  the stall -> the generator sends ONE request, waits 100s
  Total recorded: 10,001 samples.
  p99   = ~1ms
  p99.99 = ~1ms  (10,001 samples: the single 100s sample is
                  the 99.99th percentile only just)
  Reported: "p99 of 1ms, one outlier."

WHAT ACTUALLY HAPPENED TO USERS:
  10,000 requests SHOULD have arrived during the stall. Each
  would have waited: the first ~100s, the last ~0s, uniformly
  in between.
  Correct percentiles over 20,000 samples:
    p50  ~ 0ms      (the healthy half)
    p90  ~ 80s
    p99  ~ 98s
  Reported honestly: "p99 of 98 seconds."

A 1ms p99 and a 98s p99 from the same event. The difference is
entirely a property of the measurement rig.

"p99 of 1ms" and "p99 of 98 seconds" describing the same outage is the clearest illustration of why the measurement method is not a detail.

Mechanics

Closed model: what it is good for, and Little's Law

Closed model is CORRECT when the real system has a fixed
population that waits:

  - an internal batch system with N worker threads
  - a connection pool with a fixed size
  - a call centre with N agents
  - a fixed fleet of devices polling in a loop

Little's Law describes it exactly:
    N = X * (R + Z)
  N = concurrent users, X = throughput, R = response time,
  Z = think time.

So for N=500, Z=1s:
  R = 50ms   -> X = 500/1.05 = 476/s
  R = 500ms  -> X = 500/1.5  = 333/s
  R = 2s     -> X = 500/3    = 167/s

The relationship is the point: in a closed system throughput
and latency are not independent, and you cannot ask "what
happens at 500 req/s when the system can only do 480" because
the model will not let you.

Open model: specifying arrival, not population

You specify a RATE. The generator sends at that rate regardless
of what the system does.

Two sub-choices that matter:

CONSTANT RATE      exactly N per second, evenly spaced.
                   Unrealistically smooth.
POISSON ARRIVAL    exponentially distributed inter-arrival
                   times with mean 1/N. This is what
                   independent users produce, and it has
                   bursts.

Poisson is meaningfully harder on a system than constant rate at
the same mean, because queueing is driven by burstiness. A
system that holds at a constant 480/s can fail at a Poisson
480/s.

// k6: the distinction is explicit and it is the single most
// important configuration choice in the file.
export const options = {
  scenarios: {
    // CLOSED: a fixed population. Arrival rate falls if the
    // system slows. Use only when modelling a fixed-population
    // system.
    closed: {
      executor: 'constant-vus',
      vus: 500,
      duration: '10m',
    },
    // OPEN: a fixed arrival rate. If the system slows, k6 warns
    // that it cannot start iterations fast enough, which is the
    // signal you are looking for rather than an inconvenience.
    open: {
      executor: 'constant-arrival-rate',
      rate: 500,
      timeUnit: '1s',
      duration: '10m',
      // Must be large enough that the generator is not the
      // bottleneck when the system slows.
      preAllocatedVUs: 2000,
      maxVUs: 8000,
    },
  },
}

The preAllocatedVUs detail is where open-model tests quietly become closed-model tests. If the generator runs out of workers it cannot maintain the arrival rate, and it silently reverts to closed-loop behaviour at exactly the moment the system is degrading. Watch the generator's own "dropped iterations" or "cannot start iteration" counter as a first-class test result, because a test that dropped 40 percent of its intended requests measured a load you never applied.

Tool defaults, since this decides the answer:

JMeter (thread groups)      CLOSED by default. Open requires
                            the Concurrency/Arrivals Thread
                            Group plugins.
Gatling (injection)         supports both; `constantUsersPerSec`
                            is open, `atOnceUsers` + loop is
                            closed.
k6                          both, explicitly, as above.
wrk                         closed.
wrk2                        OPEN, and it corrects for
                            coordinated omission by design.
Vegeta                      open by design (`-rate`).
Locust                      closed by default; open requires
                            configuration.

Correcting coordinated omission

The fix is to measure latency from the intended send time, not the actual send time.

# WRONG: measures service time from when we managed to send.
# During a stall, we send nothing, so we measure nothing.
start = now()
response = send(request)
record(now() - start)

# RIGHT: measure from when the request WAS DUE.
# If the system stalls, the backlog of overdue requests shows
# up as the long latencies they actually represent.
interval = 1.0 / target_rate
due = start_time
while due < end_time:
    sleep_until(due)
    send_start = now()
    response = send(request)
    completed = now()
    # latency is measured against the SCHEDULE, so a request
    # sent 30s late because the generator was blocked records
    # 30s + service time, which is what a real user would have
    # experienced.
    record(completed - due)
    due += interval

HdrHistogram provides the correction directly for cases where you cannot restructure the generator:

// recordValueWithExpectedInterval synthesises the samples that
// coordinated omission would have dropped: if a measured value
// exceeds the expected interval, it back-fills the intermediate
// values that should have been recorded.
histogram.recordValueWithExpectedInterval(latencyNanos, expectedIntervalNanos);

Three practical rules:

1. Prefer a generator that is open by design and corrects
   natively (wrk2, Vegeta, k6 with an arrival-rate executor).
2. Always report the generator's dropped/late-iteration count
   alongside the latency numbers. A test with 12% dropped
   iterations did not run the test you configured.
3. Never report a percentile from a closed-loop run as if it
   described user experience.

The rest of the discipline

FIND THE KNEE, DO NOT VALIDATE A TARGET.
  Ramp arrival rate in steps, hold each step long enough to
  reach steady state, and plot throughput and latency against
  offered load.
  The useful output is the shape:

    offered   completed   p99
    100/s       100/s      40ms
    300/s       300/s      55ms
    450/s       450/s      90ms
    480/s       478/s     280ms   <- the knee
    550/s       470/s    4,200ms  <- past capacity: completed
                                     DROPPED while offered rose
    700/s       310/s     timeout <- collapse, not saturation

  Capacity is the knee (~470/s), not the peak of the offered
  column. And the fact that completed FALLS past the knee is
  the finding: the system is spending capacity on work that
  will not be delivered.

MEASURE GOODPUT, NOT JUST THROUGHPUT.
  goodput = responses delivered within the client's deadline.
  Past the knee, throughput can look flat while goodput goes to
  zero, because every response arrives after its client gave
  up.

WARM UP, THEN MEASURE. JIT compilation, connection pools,
  caches and autoscalers all need time. A 30-second test of a
  JVM service measures the interpreter.

REALISTIC DATA. A test that requests the same key 10 million
  times measures your cache, not your system. Use a key
  distribution matching production (usually Zipfian, not
  uniform).

TEST THE WHOLE PATH. Include TLS, auth, the CDN and the real
  ingress. A test that bypasses the edge does not exercise the
  component most likely to be the constraint.

MEASURE AT THE CLIENT. Server-side timings exclude queueing in
  the accept backlog, TLS, and the network, which is where
  overload shows up first.

CHECK THE GENERATOR. If the load generator saturates its own
  CPU, network or ephemeral ports, you have measured the
  generator. Run it from more than one host and confirm the
  numbers agree.

Reproducing the failures you actually care about

METASTABLE FAILURE: push past the knee, then return to a load
  the system handled comfortably before. If it does not
  recover, you have found a metastable failure, and you can
  only find it in an open model.
  The usual causes: unbounded queues, retries without budgets,
  cache stampedes after an eviction, and connection-pool
  exhaustion that persists.

RETRY AMPLIFICATION: enable the real client retry policy in the
  test. A 3-attempt policy turns a 480/s overload into 1,440/s
  at exactly the wrong moment.

DEPENDENCY DEGRADATION: rather than only loading your service,
  slow a dependency by 500ms and hold the arrival rate. This
  finds the missing timeout and the exhausted pool far more
  reliably than raw load does.

COLD CACHE: run a test immediately after a deploy or a cache
  flush. Steady-state capacity and cold-start capacity can
  differ by an order of magnitude, and deploys produce the
  second one.

A worked example: a service that passed every load test and fell over

A checkout API. Quarterly load testing with JMeter, standing result: "handles 800 req/s with p99 under 200ms, well above our 400 req/s peak." Then a marketing campaign produced 520 req/s and the service was unavailable for 40 minutes, including 25 minutes after traffic had returned to 300 req/s.

What the old test did:

JMeter, 1,000 threads, 1s think time, 15-minute run.

Reported: 800 req/s achieved, p99 190ms.

Reconstructed: at p99 190ms and mean ~60ms,
  X = N / (R + Z) = 1000 / (0.06 + 1) = 943/s offered
  observed 800/s

So the test was operating just below the knee, and every time
the system slowed, the offered rate fell with it. The test
could not push the system past its capacity because the model
would not allow it.

And the p99 of 190ms was a coordinated-omission number: the
threads that were blocked on a slow response were not sending,
so the slow period is represented by a handful of samples
rather than by the hundreds of requests that would have
arrived.

The rebuilt test, open model with corrected measurement:

k6, constant-arrival-rate, preAllocatedVUs 4000, Poisson-ish
arrival via a jittered schedule, latency measured against the
intended send time, dropped-iteration count reported.

Ramp: 200, 300, 400, 450, 500, 550, 600 req/s, 8 minutes each
with a 2-minute warm-up per step.

  offered  completed  p99      dropped iters  goodput (<2s)
  200/s      200/s     48ms         0            200/s
  300/s      300/s     61ms         0            300/s
  400/s      400/s     94ms         0            400/s
  450/s      449/s    210ms         0            449/s
  500/s      462/s  1,900ms       0.2%           351/s   <- knee
  550/s      441/s  6,400ms       4.1%            88/s
  600/s      337/s   timeout      22%              0/s

Real capacity: ~460 req/s, not 800.

The old test overstated capacity by roughly 74 percent, and the campaign's 520 req/s was on the wrong side of a knee the team believed was at 800.

The goodput column is the one that changed the conversation:

At 550 req/s the system was completing 441 requests per second,
which reads as "degraded but working".

Of those 441, only 88 arrived within the client's 2-second
timeout. The other 353 per second were work the system
performed and delivered to a client that had already given up.

80% of the system's capacity, at the moment it most needed it,
was spent producing responses nobody received.

"Throughput 441, goodput 88" is a single line that justifies load shedding to any audience, and it is invisible in a test that reports only throughput and latency.

The metastable failure, reproduced deliberately:

Push to 600 req/s for 5 minutes, then drop to 300 req/s (a load
the system had comfortably handled at step 2).

  t+0    drop to 300 req/s
  t+2m   still timing out
  t+10m  still timing out
  t+25m  still timing out; test aborted

The system did not recover. Causes, found by instrumenting the
rerun:
  - an unbounded internal work queue, holding ~180,000 requests
    whose clients had timed out 4 minutes earlier
  - a 3-attempt client retry policy with no budget, so the
    effective arrival at 300/s nominal was ~900/s
  - the connection pool to the pricing service exhausted, with
    a 30-second checkout timeout, so recovery was gated on
    draining requests that were themselves doomed

This exactly reproduced the 25 minutes of unavailability after traffic returned to normal, which the previous test methodology could not have predicted or explained.

The fixes, and the retest:

1. BOUNDED QUEUE, 2,000 items, with LIFO ordering under
   pressure. LIFO because the newest request is the one whose
   client is still waiting; FIFO under overload serves requests
   whose clients timed out minutes ago.
2. DEADLINE PROPAGATION: a request whose deadline has passed is
   dropped at dequeue without doing the work.
3. RETRY BUDGET: retries capped at 10% of request volume, so a
   3x amplification becomes 1.1x.
4. LOAD SHEDDING at the admission point, rejecting cheaply with
   429 once the queue exceeds a threshold, prioritised so
   checkout beats recommendations.

Retest:
  offered  completed  p99      goodput (<2s)
  500/s      460/s    240ms       460/s
  550/s      461/s    250ms       461/s   (89 shed with 429)
  600/s      459/s    260ms       459/s   (141 shed)
  700/s      458/s    270ms       458/s   (242 shed)

  Then 700 -> 300: fully recovered in 11 seconds.

Capacity is unchanged at ~460/s. What changed is that
everything above capacity is now rejected in ~2ms instead of
consuming a worker for 30 seconds, so goodput equals capacity
instead of collapsing, and recovery is immediate.

Capacity did not improve at all, which is the honest headline: the work was about what happens above capacity, and the measured result is that goodput at 700 req/s went from 0 to 458.

Two things the team got wrong on the first attempt:

1. The first open-model run used preAllocatedVUs of 500 against
   a 600/s target. When the system slowed, k6 could not start
   iterations, dropped 61% of them, and the run reported a p99
   of 320ms, which looked like a pass. The generator had
   silently reverted to closed-loop behaviour.
   Caught only because someone read the dropped-iteration
   counter. It is now a hard failure condition in the test: any
   run with >1% dropped iterations is invalid and does not
   produce a number.

2. The first test used a single product SKU, so the pricing
   cache had a 100% hit rate. With a production-shaped Zipfian
   SKU distribution the cache hit rate was 71% and measured
   capacity fell from 610/s to 460/s.
   The test had been measuring the cache.

Both failures produced a passing result, which is the direction load-test errors always fail, and both were found by checking the rig rather than the system.

Production evidence

Gil Tene's "How NOT to Measure Latency" is the canonical treatment of coordinated omission, including the worked stall example and the argument that most published latency numbers from closed-loop tools are measuring service time under a load the tool reduced itself.

wrk2 was written by Tene specifically to fix this: it drives a constant throughput and measures latency from the intended send time, and its documentation states the correction explicitly. HdrHistogram's recordValueWithExpectedInterval implements the same correction as a library function.

Open versus closed system models are a standard distinction in queueing theory, and the practical consequences for load testing are covered in Schroeder, Wierman and Harchol-Balter's "Open versus Closed: A Cautionary Tale" (NSDI 2006), which shows the two models produce qualitatively different conclusions about scheduling and capacity.

Metastable failures in distributed systems were characterised by Bronson et al. (HotOS 2021, with Meta co-authors), defining the sustaining feedback loops, retries, cache misses and queue growth, that keep a system in a degraded state after the triggering load has gone. Their central point, that the system does not recover when load returns to normal, is only reproducible under open-model load.

Little's Law (Little, 1961) is what makes the closed model's behaviour exactly predictable, and it is why a closed-loop test cannot be pushed past capacity: throughput is determined by population and response time.

LIFO queueing and CoDel-style controlled delay under overload are documented in Meta's published work on their load-shedding infrastructure, with the same reasoning used above: under overload the newest request is the one whose client is still waiting.

k6's arrival-rate executors and dropped-iteration metric, Gatling's injection profiles and JMeter's concurrency plugins all exist because the default thread-based model is closed, which is documented in each tool's own guidance.

The debate

Is the closed model ever right? Yes, when the real system has a fixed waiting population: a connection pool, a fixed device fleet, a bounded worker set, a call centre. The failure is applying it to internet-facing services, where arrival is independent of your response time and the whole point of the test is to find what happens past capacity.

Is coordinated omission overstated? No, and the arithmetic settles it: the same 100-second stall reports as a 1 millisecond p99 or a 98 second p99 depending only on the rig. The reasonable counter-argument is that many teams only want a regression signal, and for that a consistently wrong number is still comparable over time. It stops being acceptable the moment anyone quotes it as a user-experience figure or uses it to set an SLO.

Should load tests run against production? Ideally yes, carefully, because a staging environment with a tenth of the data and a different topology measures a different system. The cost is real risk, and the mitigations are well known: shadow traffic, a small percentage of real capacity, a kill switch, and running at a low-traffic time. A test in an unrepresentative environment gives you a number you cannot use, which is its own failure.

Ramp to a target, or find the knee? Find the knee. A test that validates "we handle 400 req/s" tells you nothing about what happens at 520, and 520 is the number the campaign produced. The shape of the curve past the knee is the actual deliverable, because that is where the design decisions live.

Is capacity the right output at all? It is half of it. Goodput past the knee is the other half, and the pair is what justifies load shedding: throughput 441 with goodput 88 is a system spending 80 percent of its capacity on responses nobody receives, and no single number expresses that.

How realistic must the data be? Realistic enough that cache behaviour matches production, which is usually the dominant factor. A uniform key distribution against a system whose production traffic is Zipfian is measuring a different system, and in one case the difference between a single-SKU test and a production-shaped one was 610 req/s against 460.

Follow-up Q&A

"What is the difference between open and closed load models?"

In a closed model a fixed population of virtual users each waits for a response before sending again, so the arrival rate is a function of response time and falls when the system slows. In an open model requests arrive at a specified rate independent of response times, so when the system slows the queue grows. Internet traffic is open: real users' request rate does not fall because your p99 rose, and if anything it rises because they retry. The consequence is that a closed-model test cannot push a system past capacity, because its feedback loop is the exact opposite of production's, so it cannot find the failure you are testing for.

"What is coordinated omission?"

A measurement error where the generator waits for a response before sending the next request, so during a stall it sends nothing and the requests that would have been slowest are never recorded. The canonical example: 100 requests per second for 200 seconds with a 100-second stall in the middle. A closed-loop generator records 10,000 fast samples plus one 100-second sample, and reports a p99 of about a millisecond. Correctly measured against the intended schedule, the 10,000 requests that should have arrived during the stall waited between 100 seconds and zero, and the p99 is about 98 seconds. Same outage, two answers, and the difference is entirely the rig.

"How do you correct for it?"

Measure latency from the time a request was due according to the schedule, not from when you managed to send it, so a backlog of overdue requests records the delays it actually represents. Use a generator that does this natively, such as wrk2 or Vegeta or k6 with an arrival-rate executor, or apply HdrHistogram's recordValueWithExpectedInterval, which back-fills the samples that omission dropped. And always report the generator's dropped-iteration count, because a run that dropped 12 percent of its intended requests did not apply the load you configured.

"How does an open-model test silently become a closed-model test?"

When the generator runs out of workers. If you configure 600 requests per second but pre-allocate only 500 virtual users, then once responses take longer than a second the generator cannot start iterations fast enough and reverts to closed-loop behaviour at exactly the moment the system is degrading. In one case that produced a 61 percent dropped-iteration rate and a reported p99 of 320 milliseconds that looked like a pass. Treat any run above about 1 percent dropped iterations as invalid and refuse to produce a number from it.

"What is the useful output of a load test?"

The shape of the curve, not a pass against a target. Ramp the arrival rate in steps, hold each to steady state, and plot offered load against completed load, latency and goodput. Capacity is the knee, the point where completed stops tracking offered, and the important finding is usually that completed falls past the knee while offered rises, because the system is spending capacity on work that will not be delivered. In one case a service reported at 800 requests per second by a closed-loop test had a real knee at 460.

"What is goodput and why report it separately?"

Responses delivered within the client's deadline, as opposed to throughput, which counts everything the system completes. Past the knee they diverge sharply: at 550 requests per second one service was completing 441 per second, of which only 88 arrived within the client's two-second timeout, so 80 percent of its capacity at the worst possible moment was producing responses nobody received. That single line justifies load shedding to any audience, and it is invisible in a test that reports only throughput and latency.

"How do you test for a metastable failure?"

Push past the knee, hold, then drop back to a load the system handled comfortably earlier, and see whether it recovers. Only an open model can do this, because a closed model reduces its own load as the system degrades. In one case the system did not recover within 25 minutes at a load it had previously handled at 61 milliseconds p99, and instrumenting the rerun found the three sustaining loops: an unbounded queue holding 180,000 requests whose clients had already timed out, an unbudgeted three-attempt retry policy tripling effective arrival, and an exhausted connection pool with a 30-second timeout.

Common misconceptions

"More virtual users means more load." In a closed model, adding users raises load only until response time rises, at which point the model throttles itself.

"Our p99 from the load test is our p99." If the test was closed-loop and uncorrected, it is a service-time number measured under a load the tool reduced, and it can understate a real stall by four orders of magnitude.

"The system handled 800 req/s, so we have headroom to 800." If the test could not exceed capacity by construction, the number is the load at which the test equilibrated, not the capacity.

"Throughput stayed high, so we degraded gracefully." Check goodput. Completing requests whose clients gave up is indistinguishable from working, on a throughput graph.

"The load generator is not part of the experiment." A saturated generator, exhausted ephemeral ports, or dropped iterations all produce passing results, which is the direction load-test errors always fail.

"A staging environment is close enough." Cache behaviour usually dominates, and a uniform key distribution against a Zipfian production workload measures a different system: in one case 610 req/s against 460.

Interview delivery note

Say this verbatim: "Closed-model load tests reduce their own load when the system slows, so they cannot reproduce overload, and coordinated omission means the requests that would have been slowest are never sent. The same 100-second stall reports as a one-millisecond p99 or a 98-second p99 depending entirely on the rig." One structural claim and one number pair that is impossible to argue with.

The senior-versus-staff separator is reporting goodput alongside throughput. A senior engineer reports capacity and p99. A staff engineer notes that at 550 requests per second the system completed 441 but only 88 arrived within the client's timeout, so 80 percent of capacity at the worst moment was producing responses nobody received, and uses that single line to fund load shedding. It reframes overload from a performance problem into a waste problem, which is a much easier argument.

The second signal is auditing the rig before the system. Treating dropped iterations as a hard failure condition, checking that the generator is not itself saturated, and noticing that a single-SKU test was measuring the cache rather than the service, all catch errors that produce passing results, which is the direction load-testing mistakes reliably fail in.

The Staff & Team Lead Interview Reference