Testing strategy, and the techniques that raise its ceiling
What it is
A testing strategy is a distribution of effort across layers, plus a position on what each layer is for. The named shapes are shorthand for different distributions:
Pyramid (Cohn) many unit, fewer service, few UI
Trophy (Dodds) static, some unit, MOST integration, few E2E
Honeycomb (Spotify) few unit, most integration, few E2E; aimed at
microservices where a "unit" spans a network hop
Ice cream cone mostly manual and E2E, few unit. The
anti-pattern, and the shape a system reaches
by default if nobody chooses
The shapes disagree because they were written about different systems. Cohn's pyramid assumes an integration test is expensive and a unit test is cheap. Testcontainers changed that premise: a test that starts a real Postgres in a container costs a few seconds, not an environment.
And above the distribution question sits a separate one that no ratio answers: every layer tests the cases you thought of. Property-based testing generates cases you did not, and mutation testing tells you whether your tests would notice if the code were wrong.
What this is confused with: coverage as the measure of a strategy. Coverage records which lines executed. A test with no assertions gives full coverage of the code it runs, which is why line coverage of 90 percent routinely coexists with a mutation score of 40 percent.
Also confused: mocks and fakes. A mock encodes your belief about how a collaborator behaves. A fake is a working in-memory implementation of the same interface. The first can be wrong in the same way your code is wrong; the second cannot.
The problem it solves
Choosing a shape without a reason produces a suite that is expensive and does not catch your bugs.
The diagnostic that settles it, and it takes an afternoon:
Take the last 20 production incidents caused by a code change.
For each, ask which layer would have caught it.
Typical finding for a service-oriented backend:
cause count layer
--------------------------------------------------------
contract/schema mismatch with a
dependency 6 contract
behaviour wrong under a real database
(transaction, isolation, constraint) 5 integration w/
a real DB
input the code never anticipated
(unicode, empty, huge, boundary) 4 property-based
pure logic error 2 unit
config/environment 2 deploy-time check
genuine cross-system journey 1 E2E
Eleven of twenty are in two layers most suites under-invest in, and four are in a technique most suites do not have at all. The shape argument is downstream of that table.
And the second failure: a suite that passes while the code is wrong.
def apply_discount(cents: int, pct: int) -> int:
return cents - (cents * pct // 100)
Tests: apply_discount(1000, 10) == 900 PASSES
apply_discount(2000, 25) == 1500 PASSES
Coverage: 100%.
Mutate `-` to `+`: apply_discount(1000,10) = 1100. Test fails.
Mutate `//` to `/`: returns 900.0. Test PASSES (== compares equal).
Mutate `>= ` guards that do not exist... there are none, which is
the actual bug: no rounding policy, no negative-pct guard, no
overflow bound.
Property test: for all cents >= 0, 0 <= pct <= 100
0 <= apply_discount(cents, pct) <= cents
Fails immediately at pct=100, cents=1: floor division gives
1 - 0 = 1, i.e. a 100% discount charges full price on 1 cent.
Neither more coverage nor a different pyramid shape finds that. A generated input does.
Mechanics
Choosing the distribution
Three properties differ per layer, and the strategy is a trade among them:
Layer Cost/test Confidence Failure localisation
-----------------------------------------------------------------
Static (types,
lint) ~0 low exact
Unit (pure) <5ms low exact
Integration
(real DB/broker
via containers) 0.1-3s HIGH good
Contract <100ms high, for exact (which field)
one risk
E2E 5-60s highest POOR ("something
broke")
Manual/exploratory minutes unique n/a
E2E's weakness is localisation, not cost. A failing E2E test tells you the checkout is broken and not which of forty services changed, which is why its value per test falls as system size grows, exactly when teams add more of them.
The committed position:
Static everything. Strict types, lint, no exceptions.
Unit only genuinely pure logic: algorithms, parsers,
money and date arithmetic, permission predicates,
state-machine transitions.
Integration THE BULK, with real dependencies via containers.
A repository test against a real Postgres catches
the constraint, the isolation level and the
migration; against a mock it catches nothing.
Contract one per consumed dependency, generated from a
schema where one exists.
E2E a small capped set of journeys where failure means
rollback. Capped by count, because flake compounds.
Property-based wherever there is an algebraic law: round trips,
invariants, idempotence, ordering, monetary
arithmetic, state machines.
Mutation on the diff, on the modules that matter, surfaced
in review rather than as a global gate.
Fakes over mocks
// MOCK: encodes your belief. If the real client throws on a 429
// rather than returning null, this test proves nothing.
when(pricingClient.quote(any())).thenReturn(null);
// FAKE: a real implementation of the interface, in memory. It can
// enforce the same invariants the real one does.
class FakePricing implements Pricing {
private final Map<String, Long> prices = new HashMap<>();
private int callBudget = 100;
public Quote quote(Request r) {
if (callBudget-- <= 0) throw new RateLimited(); // the real
if (r.sku() == null) throw new IllegalArgumentException();
Long p = prices.get(r.sku());
if (p == null) throw new NotFound(r.sku());
return new Quote(p);
}
}
A fake is written once per collaborator and used by every test, and it is the place to encode the collaborator's real contract, including its failure modes. Mocks distribute those assumptions across hundreds of test methods where they cannot be corrected in one place.
Testcontainers is what makes the integration layer affordable:
@Container
static PostgreSQLContainer<?> pg = new PostgreSQLContainer<>("postgres:16");
// The test now exercises: the real SQL dialect, real constraints,
// real transaction isolation, the real migration, and the real
// driver's type mapping. An in-memory H2 substitute exercises none
// of those and produces its own false failures.
Property-based testing
State a property that must hold for all inputs, let the framework generate them, and let it shrink a failure to a minimal counterexample.
from hypothesis import given, strategies as st
# 1. ROUND TRIP: the most productive property, and the easiest to write.
@given(st.text())
def test_encode_decode_roundtrip(s):
assert decode(encode(s)) == s
# 2. INVARIANT: something true of every output.
@given(st.lists(st.integers()))
def test_sort_invariants(xs):
out = my_sort(xs)
assert len(out) == len(xs)
assert sorted(out) == sorted(xs) # a permutation
assert all(a <= b for a, b in zip(out, out[1:]))
# 3. ORACLE / DIFFERENTIAL: compare against a slow, obviously
# correct implementation, or against the old one during a
# migration. This is the highest-value property in a rewrite.
@given(st.lists(st.integers()), st.integers())
def test_new_matches_old(items, k):
assert new_topk(items, k) == sorted(items, reverse=True)[:k]
# 4. METAMORPHIC: a relationship between outputs, when you cannot
# state the correct output.
@given(st.lists(st.floats(allow_nan=False, allow_infinity=False)))
def test_adding_an_item_cannot_reduce_the_max(xs):
assert max_of(xs + [10**6]) >= max_of(xs) if xs else True
# 5. STATEFUL: generate SEQUENCES of operations against a model.
from hypothesis.stateful import RuleBasedStateMachine, rule
class CacheModel(RuleBasedStateMachine):
def __init__(self):
super().__init__(); self.real = LRUCache(3); self.model = {}
@rule(k=st.text(min_size=1), v=st.integers())
def put(self, k, v):
self.real.put(k, v); self.model[k] = v
@rule(k=st.text(min_size=1))
def get(self, k):
got = self.real.get(k)
# An LRU may evict, so it may return None. It must NEVER
# return a value that was not the last one written.
if got is not None:
assert got == self.model[k]
Shrinking is the feature that makes this practical. A failure on a 400-element list with 14-character strings is reported as a two-element counterexample, so the bug is readable rather than archaeological.
Falsifying example: test_sort_invariants(xs=[0, -1])
The failure mode of property testing is a weak property. assert result is not None passes for
every implementation including a stub. Write the property that would fail if the function were
subtly wrong, not the one that is easy to state.
Where it pays best: parsers and serialisers, money and date arithmetic, caches and data structures, permission and rule engines, protocol encoders, and any rewrite where the old implementation is the oracle.
Mutation testing
Change the source, re-run the tests, and see whether anything notices.
Original: if (balance >= amount) { withdraw(amount); }
Mutants: if (balance > amount) { withdraw(amount); } boundary
if (true) { withdraw(amount); } condition
if (balance >= amount) { } removed call
if (balance <= amount) { withdraw(amount); } negated
Each mutant is run against the suite:
KILLED at least one test fails -> the suite detects this change
SURVIVED all tests pass -> the suite would not notice
this bug
mutation score = killed / (total - equivalent)
The gap between coverage and mutation score is the number that changes minds:
A pricing module, measured:
line coverage 91%
branch coverage 84%
mutation score 38%
Reading: 91% of lines run during the suite, and for 62% of the
semantic changes you could make to them, no test fails. Coverage
was measuring that the tests visit the code, not that they check
it.
The cost is real: N mutants times the suite duration. Three controls make it affordable:
1. INCREMENTAL: mutate only lines changed in the diff. Both PIT and
Stryker support this. A 40-line diff produces perhaps 60
mutants, which is minutes rather than hours.
2. SCOPED: run it on the modules where being wrong is expensive
(money, permissions, safety), not on the whole repository.
3. SURFACED IN REVIEW, not as a gate. A surviving mutant shown as a
review comment ("no test fails if this `>=` becomes `>`") is
actionable. A repository-wide score threshold produces gaming.
Google's published experience is exactly this shape: they surface mutants during code review rather than reporting a score, and they suppress categories of mutant that are uninteresting in practice, because an unfiltered mutation report is mostly noise.
A worked example: a pricing library with 91 percent coverage and four latent bugs
A billing service's pricing module: discounts, tax, proration, multi-currency rounding. 4,200 lines, 310 tests, 91 percent line coverage, no production incidents attributed to it in the previous quarter, which was the reason nobody had looked at it.
Step 1: mutation testing on the module.
mutants generated 1,847
killed 702
survived 1,089
equivalent (manual) 56
mutation score 38.6%
Survivor clusters:
- 312 in boundary conditions (>= vs >, < vs <=) on amount and
quantity thresholds
- 244 in rounding: changing ROUND_HALF_UP to ROUND_DOWN killed
nothing
- 190 in error paths: removing a `throw` killed nothing, because
no test asserted on the exception
- 158 in currency handling: swapping two currency codes in a
lookup killed nothing
"Changing the rounding mode fails no test" is a one-line summary of a suite that does not test the thing the module exists to do.
Step 2: property tests for the algebraic laws.
from decimal import Decimal
from hypothesis import given, strategies as st
money = st.decimals(min_value=Decimal("0"), max_value=Decimal("1e6"),
places=2, allow_nan=False, allow_infinity=False)
pct = st.integers(min_value=0, max_value=100)
@given(money, pct)
def test_discount_bounds(amount, p):
out = apply_discount(amount, p)
assert Decimal("0") <= out <= amount
@given(money, st.lists(pct, min_size=2, max_size=5))
def test_discount_order_does_not_matter(amount, pcts):
# Sequential percentage discounts must be commutative.
import itertools
results = {reduce_discounts(amount, list(o))
for o in itertools.permutations(pcts)}
assert len(results) == 1
@given(money, st.lists(st.integers(1, 12), min_size=1, max_size=12))
def test_proration_sums_to_total(amount, month_lengths):
parts = prorate(amount, month_lengths)
assert sum(parts) == amount # NO CENT MAY BE LOST
Four bugs, found in the first run:
1. test_discount_bounds
Falsifying example: amount=Decimal('0.01'), p=100
A 100% discount on one cent charged one cent. Integer floor
division: 1 - (1*100 // 100) should be 0, but the code applied
a MIN_CHARGE clamp before the discount rather than after.
Blast radius: every sub-dollar line item with a full discount.
2. test_discount_order_does_not_matter
Falsifying example: amount=Decimal('19.99'), pcts=[33, 7]
Applying 33% then 7% gave 12.46; 7% then 33% gave 12.45. Each
step rounded to 2dp. The invoice total depended on the ORDER
discounts were stored in, which was the database's natural
ordering, which changed after a reindex.
This is the class of bug that produces a support ticket nobody
can reproduce.
3. test_proration_sums_to_total
Falsifying example: amount=Decimal('100.00'), month_lengths=[3]*7
Each part rounded independently, so 7 parts of 14.285714 became
7 x 14.29 = 100.03. Three cents created from nothing, on every
7-way split.
Fix: largest-remainder allocation, where the last part absorbs
the residual.
4. A stateful test on the currency converter found that converting
USD -> EUR -> USD lost value monotonically, because both
directions rounded to 2dp. Round-tripping 100 times lost
$0.41 per $100.
Every one of these returns a plausible number and no error, which is the silent-correctness category: no layer of the existing pyramid could have caught them, and coverage was already 91 percent.
Step 3: the resulting distribution change.
before after
unit tests 310 180 (deleted 130 that were
mocked-collaborator
assertions)
integration (real DB
via Testcontainers) 0 64
property tests 0 41
contract (generated
from the schema) 0 9
mutation, incremental
on the diff - enabled, surfaced in review
line coverage 91% 88% (went DOWN)
mutation score 38.6% 79.4%
suite duration 2m10s 6m40s
Coverage fell and confidence rose, which is the clearest available demonstration that the two are different measurements. The 130 deleted unit tests asserted that a mocked collaborator was called with particular arguments; deleting them removed executed lines without removing any check.
Step 4: the operating policy that came out of it.
- Mutation testing runs on the diff for the pricing, entitlements
and tax modules. Surviving mutants appear as review comments.
Median added CI time: 90 seconds.
- Any function with an algebraic law (round trip, commutativity,
conservation, monotonicity) gets a property test, and the law is
named in the test's docstring.
- No new mocks for collaborators that have a fake. The fake is the
one place the collaborator's contract is written down.
Measured over the following two quarters: production defects
attributed to the pricing module went from 7 in the prior two
quarters to 1.
Production evidence
Google's "State of Mutation Testing at Google" (ICSE-SEIP 2018) describes running mutation testing at scale by surfacing mutants during code review rather than as a score, and by suppressing categories of mutant developers consistently judged uninteresting. It is the primary evidence for the incremental, review-surfaced approach.
Hypothesis (Python), QuickCheck (Haskell), jqwik (Java), fast-check (JavaScript) and proptest (Rust) all implement generation plus shrinking, and shrinking to a minimal counterexample is the documented feature that makes generated-input failures debuggable.
Testcontainers runs real dependencies in Docker for the duration of a test, and its adoption is the practical reason the pyramid's cost premise no longer holds: a repository test against a real Postgres, rather than an in-memory substitute with a different SQL dialect, now costs seconds.
Jepsen is the best-known application of generated operation sequences plus a model checker to distributed databases, and its findings, consistently uncovering consistency violations that example-based test suites missed, are the strongest available argument for stateful property testing.
Spotify's "honeycomb" and Kent C. Dodds's "testing trophy" are both published reactions to the pyramid, arguing for an integration-heavy distribution, in Spotify's case explicitly because a microservice's meaningful behaviour spans a network boundary.
AWS's use of formal methods and property-based techniques on S3 and DynamoDB, described in their published work on lightweight formal methods, is the industrial end of the same argument: examples cannot cover a state space, so state the property.
The debate
Pyramid, trophy or honeycomb? The shape follows from where your bugs are and what a unit costs in your system. For a service with heavy I/O and thin pure logic, integration-heavy is correct; for a library of algorithms, unit-heavy is correct. The position: derive it from the escaped-bug table rather than adopting a named shape, and note that the pyramid's cost premise predates containers.
Are mocks ever right? For a collaborator you cannot implement (a third-party SDK with opaque behaviour) or for asserting an interaction that is the point (a payment was attempted exactly once). Otherwise prefer a fake, because a mock puts your assumption about the collaborator in every test that uses it, and a fake puts it in one place where it can be corrected.
Does property testing replace example tests? No, and the pairing is the standard practice: examples document intent and pin known cases, properties explore the space. Keep the falsifying example as a regression test when a property finds a bug, which every mature framework does automatically.
Is mutation testing worth the cost? As an incremental, scoped, review-surfaced signal, yes. As a repository-wide score with a threshold, no, because it is expensive, produces large volumes of uninteresting mutants, and a threshold invites gaming with assertion-free tests that happen to kill mutants. Google's approach exists because the naive version was unusable.
Should coverage be a gate? As a floor to catch entirely untested new code, weakly. As a target, no, and the pricing example is the argument: coverage fell from 91 to 88 percent while the mutation score doubled. A team optimising coverage writes tests that execute code; a team optimising mutation score writes tests that check it.
Do property tests slow CI unacceptably? They are configurable: a low example count in CI and a high one in a nightly run is the standard arrangement. The real cost is authorship, because stating a useful property is harder than writing an example, and a weak property is worse than none because it looks like coverage.
Follow-up Q&A
"How do you choose a testing shape?"
Derive it rather than adopting a name. Take the last twenty production incidents caused by a code change and assign each to the layer that would have caught it. For a service-oriented backend, the answer is usually contract tests and integration tests against real dependencies, with a surprising number in a category no layer covers: inputs nobody anticipated. The pyramid's premise, that integration tests are expensive, predates containers, so the shape it recommends is answering a cost question that has changed.
"Why prefer a fake to a mock?"
A mock encodes your belief about how a collaborator behaves, and if that belief is wrong the test still passes. It also distributes that belief across every test method that uses it, so a correction means editing hundreds of call sites. A fake is a working in-memory implementation of the same interface, written once, where you can encode the real contract including its failure modes: rate limits, not-found, validation. Mocks remain correct for collaborators you cannot implement and for asserting that an interaction happened exactly once.
"What is property-based testing good for, concretely?"
Anything with an algebraic law. Round trips: decode of encode is identity. Invariants: a sort's output is a permutation of its input and is ordered. Oracles: the new implementation matches the old one, which is the highest-value property during a rewrite. Metamorphic relations: adding a larger element cannot reduce the maximum. And stateful sequences checked against a model, which is how Jepsen finds consistency violations. It pays best in parsers, money and date arithmetic, caches, and rule engines. The failure mode is a weak property like asserting the result is not null, which passes for a stub.
"What does shrinking do and why does it matter?"
When a generated input falsifies a property, the framework searches for the smallest input that still fails and reports that. A failure on a 400-element list of long strings becomes a two-element counterexample, so the bug is readable rather than archaeological. Without shrinking, generated-input testing produces failures that take longer to understand than to find, which is why the technique was impractical before frameworks implemented it.
"What does mutation testing tell you that coverage does not?"
Coverage records that a line executed during the suite. Mutation testing changes the line and asks whether any test fails. A pricing module measured at 91 percent line coverage had a mutation score of 38 percent, meaning that for roughly six out of ten semantic changes you could make to code the tests visit, nothing failed. The most compressed reading of that report was that switching the rounding mode from half-up to down killed no test, in a module whose purpose is computing money.
"How do you run mutation testing without it costing hours?"
Three controls. Incremental: mutate only the lines in the diff, which both PIT and Stryker support, so a forty-line change produces perhaps sixty mutants and about ninety seconds. Scoped: run it on the modules where being wrong is expensive, money, permissions, safety, rather than the whole repository. And surfaced in review rather than gated on a score, because a surviving mutant shown as a comment is actionable while a threshold invites gaming. That is the shape Google published after finding the naive version unusable.
Common misconceptions
"Follow the pyramid." Its cost premise, that integration tests are expensive, predates containers. Derive the shape from where your bugs actually are.
"Coverage measures test quality." It measures execution. A suite can visit every line and check none of them, which is what a 91 percent coverage and 38 percent mutation score means.
"Mocking is how you isolate." Mocking is how you encode an assumption in hundreds of places. A fake isolates equally and puts the assumption in one correctable location.
"Property tests are academic." They find rounding, ordering and conservation bugs in production money code within minutes, and the failures they report are minimal counterexamples, not noise.
"A property test is done when it passes." A weak property passes for a stub. The useful question is whether the property would fail if the function were subtly wrong.
"Mutation testing means enforcing a score." That is the version that fails. Incremental, scoped and surfaced in review is the version that works.
Interview delivery note
Say this verbatim: "Coverage tells you a line ran; mutation testing tells you whether any test would notice if it were wrong. One module measured at 91 percent line coverage and a 38 percent mutation score, and the compressed version of that report was that changing the rounding mode from half-up to down failed no test, in a module whose job is computing money." It gives the distinction and the evidence in one breath.
The senior-versus-staff separator is deriving the shape from escaped bugs rather than naming one. A senior engineer argues pyramid versus trophy. A staff engineer takes twenty production incidents, assigns each to the layer that would have caught it, and finds that most were contract mismatches and real-database behaviour, with a cluster in a category no layer covers at all, inputs nobody anticipated, which is what property testing is for. The shape becomes a conclusion instead of a preference.
The second signal is accepting a coverage decrease. Saying "we deleted 130 mocked-collaborator unit tests, coverage fell from 91 to 88 percent and the mutation score went from 38 to 79" shows you understand which number is measuring the thing you care about, and that you are willing to move a visible metric in the wrong direction to do it.
Further reading
- Petrovic and Ivankovic, "State of Mutation Testing at Google" (ICSE-SEIP 2018), for incremental, review-surfaced mutation testing and mutant suppression.
- Hypothesis documentation, particularly on shrinking and stateful testing with
RuleBasedStateMachine. - Jepsen's analyses, as the largest body of evidence for generated operation sequences checked against a model.
- Testcontainers documentation, for the cost change that undermines the pyramid's original premise.
- The frontend testing ratio page, which applies the same reasoning where the layers and costs differ.