Benchmarking discipline, and what microbenchmarks lie about

What it is

A benchmark is an experiment, and benchmarking discipline is the set of habits that keep the experiment honest: measuring the thing you actually care about, under conditions that resemble production, with enough repetitions to separate signal from noise, and reporting the distribution rather than a single number.

The thing it is confused with is profiling. A profile tells you where time goes inside one execution; a benchmark tells you how long the execution takes and whether one version is faster than another. They answer different questions and fail in different ways. A profile can be perfectly accurate and still point you at code that does not matter, because the profile has no notion of "how often does this path run in production." A benchmark can show a real 30 percent improvement that evaporates on deploy, because the benchmark measured a warm, single-threaded, cache-resident version of a system that in production is cold, concurrent and memory-bound.

The second confusion is benchmark versus load test. A microbenchmark measures one function or one operation in isolation, usually in nanoseconds to microseconds. A load test drives the whole system at a target request rate and measures the response distribution, usually in milliseconds. Most of the disappointing "it was faster in the benchmark" stories come from using the first where the second was required.

The problem it solves

Without discipline, benchmarks produce confident numbers that are wrong, which is strictly worse than no numbers, because a wrong number ends the discussion. The specific failures are mechanical and repeat across languages and teams:

Dead code elimination. You benchmark a pure function whose result you discard. A modern JIT or optimising compiler proves the result is unused, deletes the call, and reports an operation that takes 0.3 nanoseconds. That is roughly one CPU cycle, which is a good hint you measured nothing.

Constant folding. You benchmark hash(input) with input a compile-time constant. The compiler computes the hash once and the loop measures a load from a register.

Warmup and the JIT. On the JVM, a method runs interpreted first, gets profiled, then gets compiled at tier 4 with inlining and loop optimisations informed by the profile it collected. Measurements taken before the transition are measuring the interpreter. The same effect exists in different form for Python (no JIT in CPython, but import and first-call costs dominate short runs), Go (no warmup, but the first allocations grow the heap), and any system with a cache.

Profile pollution. This one is subtle and specific to the JVM. If the same generic method is called from your benchmark with two different concrete types, the JIT sees a bimorphic call site and refuses to inline it. Your benchmark now measures a slower version than production, where the call site is monomorphic. The reverse also happens: your benchmark is monomorphic and production is not.

Coordinated omission. The most damaging one, because it silently deletes exactly the measurements you care about. If your load generator sends a request, waits for the response, and only then sends the next one, then during a 2-second stall it sends no requests at all. The requests that would have been slow were never issued, so they never appear in the histogram. The result is a latency distribution that looks fine while the system is visibly stalling.

Mechanics

The fixes are individually simple and mostly consist of removing the compiler's ability to cheat and the harness's ability to lie.

Consume your results. JMH provides a Blackhole; Go has runtime.KeepAlive and the convention of assigning to a package-level variable. The idea is the same: make the result observable so it cannot be optimised away.

// JMH: the return value is implicitly consumed by the harness.
@Benchmark
public long hashOne(BenchState s) {
    return s.hasher.hash(s.key);      // returned, so not dead
}

// When you produce several values, use the Blackhole explicitly.
@Benchmark
public void hashMany(BenchState s, Blackhole bh) {
    for (String k : s.keys) {
        bh.consume(s.hasher.hash(k));
    }
}

@State(Scope.Benchmark)
public class BenchState {
    // Not final, not a literal: the JIT cannot fold it.
    public Hasher hasher;
    public String key;
    public String[] keys;

    @Setup(Level.Trial)
    public void setup() {
        hasher = new Murmur3Hasher();
        key = "user:" + ThreadLocalRandom.current().nextInt();
        keys = IntStream.range(0, 1024)
                        .mapToObj(i -> "user:" + i)
                        .toArray(String[]::new);
    }
}

Warm up, then measure. The JMH defaults are 5 warmup iterations and 5 measurement iterations, each 10 seconds, in a forked JVM. Forking matters: it isolates each benchmark from profile pollution caused by benchmarks that ran before it in the same JVM.

@Warmup(iterations = 5, time = 10, timeUnit = SECONDS)
@Measurement(iterations = 5, time = 10, timeUnit = SECONDS)
@Fork(value = 3)                       // 3 JVMs; variance across forks is real signal
@BenchmarkMode(Mode.SampleTime)        // gives you a distribution, not just a mean
@OutputTimeUnit(TimeUnit.MICROSECONDS)

Mode.SampleTime deserves emphasis. Mode.AverageTime reports one number, and a mean is nearly useless for latency because the distribution is right-skewed. Sample mode gives percentiles.

Fix coordinated omission by using an open model. An open-model load generator issues requests on a schedule that does not depend on when responses arrive. If the target is 1,000 requests per second, it issues one every millisecond whether or not the previous one came back. wrk2, Gatling and k6 (with arrival-rate executors) work this way; ab and naive closed loops do not.

The correction, if you are stuck with a closed-loop harness, is to record the intended start time rather than the actual one:

# Open-model timing: schedule is fixed in advance.
start = time.perf_counter()
interval = 1.0 / target_rps
for i in range(n):
    intended = start + i * interval
    now = time.perf_counter()
    if now < intended:
        time.sleep(intended - now)
    # If now > intended we are behind: do NOT skip, and measure from `intended`.
    t0 = intended                       # not time.perf_counter()
    send_request()
    record(time.perf_counter() - t0)    # includes the queueing we caused

The difference between t0 = intended and t0 = now is the entire coordinated omission correction. It is two lines and it routinely changes a reported p99 by an order of magnitude.

A worked example: a 4x speedup that was 1.06x

A team replaced a JSON serialiser in a service's response path. The JMH benchmark was clean by most standards: warmed up, forked, results consumed, realistic payloads.

Benchmark                    Mode  Cnt    Score   Error  Units
SerBench.jackson            avgt   15   14.213 ± 0.402  us/op
SerBench.newSerialiser      avgt   15    3.556 ± 0.111  us/op

Four times faster, tight error bars, reproducible. They shipped it behind a flag and measured the service:

                p50      p95      p99
before        18.4ms   61.2ms   142ms
after         17.9ms   59.8ms   139ms

A 2.7 percent improvement at p50. The benchmark was not wrong about serialisation; it was wrong about the share. Serialisation was 14 microseconds out of an 18,400 microsecond request. Amdahl's law caps the achievable speedup at

$$S = \frac{1}{(1-p) + p/s} = \frac{1}{(1 - 0.00077) + 0.00077/4} = 1.0006$$

The measured 2.7 percent was mostly allocation pressure reduction, which the microbenchmark had not measured at all. The lesson is not that the benchmark lied about its own scope. It is that the benchmark answered a question nobody had asked: nobody had first measured what fraction of the request serialisation represented. Five minutes with a profiler would have killed the project before it started, and the correct order is always profile first, then benchmark the thing the profile identified.

There is a second half to this story worth carrying into interviews. The same team later found that the p99 of 142ms was dominated by a synchronous call to an authorisation service. Fixing that took the p99 to 71ms. The lever was never in the code that was easiest to benchmark.

Production evidence

JMH exists because Sun and later Oracle engineers could not trust JVM microbenchmarks, including their own. Aleksey Shipilev's talk "JMH vs. the Benchmarking Traps" walks through dead-code elimination, constant folding and loop unrolling with concrete disassembly, showing benchmarks that report timings for code the JIT deleted. JMH's design (forking, blackholes, @State objects, -prof perfasm to dump the compiled assembly) is a direct response to specific observed failures rather than general caution.

Coordinated omission was named and popularised by Gil Tene (Azul Systems) in "How NOT to Measure Latency," where he showed standard load-testing tools reporting healthy percentiles while the system under test was pausing for seconds. HdrHistogram, his library, includes recordValueWithExpectedInterval() specifically to synthesise the missing samples when a closed-loop harness is unavoidable. wrk2 was written by Tene as a corrected fork of wrk for the same reason.

Go's testing package hard-codes some of this discipline. go test -bench chooses b.N adaptively so each benchmark runs for a minimum duration rather than a fixed iteration count, and benchstat (from golang.org/x/perf) reports geometric means with confidence intervals across repeated runs, refusing to declare a difference that is not statistically distinguishable. -benchmem is on by default in most Go teams' habits because allocation count is more stable than wall time.

Databases publish benchmark reproduction kits for the same reason. ClickHouse's public benchmark suite ships the dataset, the queries and the hardware specification, because the community's experience was that unqualified "X is faster than Y" numbers were unreproducible often enough to be worthless.

The debate

The strongest argument against microbenchmarking discipline is cost. A properly forked JMH suite with 3 forks and 5+5 iterations takes minutes per benchmark, and a suite of 40 benchmarks becomes a nightly job rather than something you run before a commit. Teams respond by cutting warmup, dropping to one fork, or running in-process, and each cut reintroduces a specific failure mode.

The counter-argument, and my position, is that the choice is not between an expensive benchmark and a cheap one, it is between an expensive benchmark and a misleading one. A benchmark you do not trust has negative value: it will be cited in a design review by someone who was not in the room when it was run.

Where I would genuinely skip the discipline: when you are looking for an order of magnitude, not a percentage. If the question is "is this 10 microseconds or 10 milliseconds," a sloppy loop answers it correctly and the traps do not matter at that resolution. The traps matter when you are comparing two implementations that are within 2x of each other, which is exactly when teams most want an answer.

The position to hold as a staff engineer: microbenchmarks are for validating a hypothesis that a profile has already produced, and load tests are for validating that a change survives contact with concurrency, cold caches and real payload distributions. Reversing that order is the single most common way teams spend a quarter optimising something that does not matter. And any latency number reported without saying which percentile it is, and whether the generator was open or closed, should be treated as unmeasured.

Follow-up Q&A

"Your benchmark shows 3.5 microseconds and production shows 14. What are the suspects, in order?"

First, what else is running: the benchmark had a whole core, production shares one with 40 other threads, so you are paying context switches and cache eviction. Second, cache residency: the benchmark's working set was 1,024 keys resident in L2, production's is millions and every access is a memory reference at roughly 100 nanoseconds. Third, monomorphic versus polymorphic call sites: the benchmark had one implementation, production has three behind an interface, so the JIT stopped inlining. Fourth, allocation and GC: the benchmark's garbage fit in the young gen and never got promoted. Fifth, input distribution: benchmark inputs are usually uniform and production's are Zipfian, which changes branch prediction and cache hit rates. I would attack them in that order because that is roughly the order of expected magnitude.

"How many runs do you need?"

Enough for the confidence interval to be narrower than the effect you are claiming. Concretely: if you claim a 5 percent improvement and your run-to-run standard deviation is 4 percent, you need enough samples that the standard error is well under 5 percent, which for a 4 percent SD is roughly 10 or more independent runs. The practical rule I use is that the error bars must not overlap, and if they do, the honest report is "no measurable difference," not "slightly faster." Run the two variants interleaved rather than all of A then all of B, because machine state drifts (thermal throttling, background jobs, page cache) and interleaving turns a systematic bias into noise.

"What is coordinated omission, and how do you tell if you have it?"

It is the systematic loss of the slowest measurements because the load generator stops issuing requests while it waits. The tell is that your reported latency distribution is much narrower than your throughput dips would imply. Concretely: if throughput dropped to zero for 2 seconds and your p99.9 is 40 milliseconds, those numbers are inconsistent, because during the stall there must have been requests that would have taken up to 2 seconds. The fix is an open-model generator (wrk2, Gatling, k6 with constant-arrival-rate) or HdrHistogram's recordValueWithExpectedInterval.

"Why fork the JVM for each benchmark?"

Because the JIT's decisions are made from profile data collected across the whole process lifetime. If benchmark A runs process(String) and benchmark B runs process(Integer) in the same JVM, the call site inside the shared helper becomes bimorphic and neither gets inlined. Whichever runs second (or both) reports a slower number for reasons that have nothing to do with the code under test. Forking gives each benchmark a fresh JVM. Running multiple forks additionally exposes run-to-run variance from things like different heap layouts and different tiered-compilation decisions, which is real variance that a single fork hides.

"When would you not benchmark at all, and just ship it?"

When the change is obviously correct and the risk of the change is lower than the cost of measuring it. Removing an N+1 query that fires 300 times per request does not need a benchmark, it needs a code review. Benchmarks are for the cases where your intuition could plausibly be wrong, and a large class of performance work (removing work entirely) is not in that class.

Common misconceptions

"The mean is the number." Latency distributions are right-skewed and often multi-modal (cache hit versus miss, JIT-compiled versus interpreted, GC pause or not). The mean sits in a valley between modes and describes no actual request. Two systems with identical means can have p99s that differ by 10x. Report percentiles, and be aware that percentiles do not average: you cannot take the mean of two shards' p99s and get the fleet p99. You need the merged histogram, which is why HdrHistogram and DDSketch support mergeable representations.

"I ran it twice and got the same number, so it's stable." Two consecutive runs in the same process share JIT state, page cache and thermal conditions. Stability across runs in the same process is nearly meaningless; stability across forks and across days is the thing to check.

"The benchmark is 4x faster, so the service will be 4x faster." This is Amdahl's law and it is worth doing the arithmetic explicitly before starting the work. See the worked example: a genuine 4x on 0.08 percent of the request is a 1.0006x speedup. Related: the Universal Scalability Law page covers the other direction, where adding capacity makes things worse.

"Production traffic is the best benchmark." It is the best validation, but it is a poor experiment: you cannot control the variables, you cannot repeat it, and the confounders (deploy time, traffic mix, a neighbouring service's incident) are unbounded. Shadow traffic gives you production's input distribution with experimental control, which is the actual best of both.

Interview delivery note

The line to say verbatim: "Before I benchmark anything, I want to know what fraction of the request the thing represents, because Amdahl's law caps the payoff and that arithmetic takes thirty seconds." It signals that you treat performance work as a budgeting problem rather than a craft exercise.

The senior-versus-staff separator on this topic is coordinated omission. A senior engineer will talk about warmup, JIT and consuming results, which is the standard microbenchmark checklist. A staff engineer notices that the load test harness deletes the slow requests, because that is the failure that survives all the way to a customer-visible incident while every dashboard says the service is healthy. If you get a chance to say "was the generator open or closed model," take it.

The second signal is willingness to report a null result. Saying "we measured it, the difference was inside the error bars, so we did not ship it" is a stronger answer than any speedup number, because it demonstrates you were running an experiment rather than building a case.

The Staff & Team Lead Interview Reference