Testing 40 microservices without a full environment

"How do you test 40 microservices without a full integration environment?"

What it is

Consumer-driven contract testing. Each consumer declares, in executable form, exactly what it needs from a provider: the requests it makes and the parts of the response it depends on. Those expectations become a contract. The consumer's tests run against a mock built from the contract; the provider's CI replays the contract against the real provider and fails if it no longer satisfies it.

The critical property: the two sides never run at the same time. Consumer tests run in the consumer's pipeline against a stub. Provider verification runs in the provider's pipeline against the real provider. No shared environment, no orchestration, no waiting for other teams.

Commonly confused with schema validation. A schema says the response is well-formed; a contract says this consumer needs these specific fields with these specific semantics. Schema compatibility tells you the shape did not break; a contract tells you nobody's actual usage broke, which is a stronger and more useful statement.

The problem it solves

The instinct with 40 services is to build an environment containing all 40 and run end-to-end tests against it. That fails for reasons that compound:

Combinatorics. Forty services with independent release cadences means the environment is never in a state that will exist in production. You are testing a configuration that no user will encounter.

Flakiness. End-to-end tests across 40 network hops fail for environmental reasons constantly. Once the team learns that red is usually noise, the suite has stopped being a signal, and a suite everyone retries is worse than no suite because it consumes time and confers false confidence.

Serialisation. One environment, 40 teams, so it becomes a booking system. Deploys queue behind each other and lead time is the sum of everyone's queue.

Ownership. When an end-to-end test fails, which team fixes it? In practice it is whoever notices, and eventually nobody notices.

Cost. Forty services with their databases and dependencies, running continuously.

Contract testing replaces the environment with an artifact. Instead of testing integration, you test the interface, and you do it in each team's own pipeline on their own schedule.

Mechanics

The consumer side

// Consumer test. This declares an expectation AND generates the contract.
// Note what is specified and what is not: only the fields this consumer
// actually reads. If we over-specify, we constrain the provider for no reason.
@ExtendWith(PactConsumerTestExt.class)
@PactTestFor(providerName = "order-service")
class OrderClientContractTest {

    @Pact(consumer = "checkout-ui")
    RequestResponsePact orderExists(PactDslWithProvider builder) {
        return builder
            .given("an order 8842 exists in state SHIPPED")   // provider STATE
            .uponReceiving("a request for order 8842")
                .path("/orders/8842").method("GET")
            .willRespondWith()
                .status(200)
                .body(new PactDslJsonBody()
                    .stringType("id", "8842")          // type matcher, not value
                    .stringValue("status", "SHIPPED")  // exact: we branch on it
                    .integerType("total_minor", 4299))
            .toPact();
    }

    @Test
    @PactTestFor(pactMethod = "orderExists")
    void parsesOrder(MockServer mock) {
        var order = new OrderClient(mock.getUrl()).fetch("8842");
        assertEquals("SHIPPED", order.status());
        assertEquals(4299, order.totalMinor());
    }
}

Two details carry most of the value. Matchers over literals: stringType asserts "a string is here", not "exactly this string", so the provider is free to return real data. Over-specifying with literal values is the most common way contract tests become brittle and get abandoned. And provider states (given(...)) are how the provider knows what data to set up before replaying that interaction.

The consumer's pipeline publishes the generated contract to a broker, tagged with the branch and version.

The provider side

// Provider verification. Runs in the PROVIDER's pipeline, against the real
// provider, replaying every consumer's contract. No consumer code involved.
@Provider("order-service")
@PactBroker(url = "https://pact-broker.internal")
class OrderServiceContractVerification {

    @State("an order 8842 exists in state SHIPPED")
    void seedShippedOrder() {
        // Put the provider into the state the consumer's scenario assumed.
        // This is the only coupling, and it is a named string, not a schema.
        testData.insertOrder("8842", Status.SHIPPED, 4299);
    }

    @TestTemplate
    @ExtendWith(PactVerificationInvocationContextProvider.class)
    void verify(PactVerificationContext ctx) { ctx.verifyInteraction(); }
}

If a provider change breaks any consumer's expectation, the provider's build fails, in the provider's own pipeline, before merge. That is the whole point: the feedback lands on the team that made the change, immediately, without a shared environment existing.

The deployment gate

The broker turns contracts into a deployability question:

# Can this version of order-service go to production without breaking
# anything currently deployed there? The broker knows which consumer
# versions are in production and which contracts they published.
pact-broker can-i-deploy \
  --pacticipant order-service --version "$GIT_SHA" \
  --to-environment production

This is the piece that makes it operational rather than academic. It answers "is it safe to deploy this" from recorded facts about what is deployed and what each deployed consumer needs, rather than from a test run in an environment that resembles production.

Bi-directional contracts are the lighter-weight variant: the provider publishes its OpenAPI specification, consumers publish their contracts, and the broker checks compatibility without the provider running verification. Weaker (it verifies the spec, not the implementation) and much cheaper to adopt when the provider team will not write verification tests.

The test pyramid this implies

Contract tests do not stand alone. The distribution that works for 40 services:

Layer	What it covers	Count	Runs
Unit	Logic, algorithms, edge cases	Thousands	Every commit, seconds
Integration (in-process)	Service plus its own database, via Testcontainers	Hundreds	Every commit, minutes
Contract	Every consumer-provider interface	Tens	Every commit, both sides
End-to-end	3 to 5 critical user journeys	A handful	Pre-release, against production-like
Synthetic monitoring	Same journeys, continuously	A handful	Production, forever

The load-bearing claims: contract tests replace the integration portion of the end-to-end suite, not all of it; you keep a small number of end-to-end tests for genuine cross-service journeys; and synthetic monitoring in production is worth more than a large staging suite, because it tests the real thing continuously.

The rule to state: if an end-to-end test would fail only because an interface changed, it should be a contract test. Keep end-to-end for behaviour that genuinely emerges from several services interacting, which is a much smaller set than teams assume.

A worked example

40 services, 12 teams, one shared staging environment. Symptoms: the end-to-end suite takes 90 minutes and is red about 40 percent of the time, staging is booked out days in advance, and two production incidents in the last quarter were interface breakages the suite should have caught but did not, because it had been red for unrelated reasons and nobody looked.

Migration, six months:

Month 1   Stand up a broker. Pick the two most-coupled services and write
          contracts for their three interactions. Deliberately small: the
          goal is to prove the loop, not coverage.

Month 2-4 Expand outward, consumer-driven: each consumer team writes
          contracts for what it needs. 40 services turns out to have
          about 85 real consumer-provider pairs, not the 1,560 the
          combinatorics suggest, because most services talk to few others.

Month 4   Wire `can-i-deploy` into every pipeline as a deploy gate.

Month 5   Delete end-to-end tests that only verified interfaces. The suite
          goes from 140 tests to 6. Runtime from 90 minutes to 7.

Month 6   Staging stops being a booking system, because most teams no
          longer need it. Add synthetic monitoring for the 6 journeys in
          production, running every 5 minutes.

Measured outcome: interface breakages caught pre-merge rather than in staging or production; end-to-end runtime down 92 percent; staging contention eliminated; and deploy lead time down because nobody queues for the environment.

The number worth quoting is the 85 rather than 1,560. Forty services do not interact pairwise; the interaction graph is sparse, and the contract-testing effort scales with the number of real edges rather than with the square of the node count. That fact is what makes this tractable and it is the one that surprises people.

What it did not fix, and I would say so: a bug where two services each satisfied their contracts and the combination produced wrong behaviour, because an order was marked shipped before payment settled. No contract test finds that; it is emergent behaviour and it is exactly what the remaining six end-to-end tests exist for. That honesty is what makes the answer credible rather than a sales pitch.

Production evidence

Pact is the reference implementation of consumer-driven contract testing, with the broker, provider states, can-i-deploy, and bi-directional contracts. Its documentation is the primary source for the mechanics above.

Spring Cloud Contract is the JVM-native alternative, with a producer-driven emphasis: the provider defines contracts and generates consumer stubs. Worth knowing as the counterpoint, because the direction of authorship is the main design difference between the two.

Buf applies the same idea to Protobuf: buf breaking detects breaking schema changes in CI against a baseline, which is contract testing at the schema level for gRPC. Mentioning it signals you know the gRPC world has its own answer.

Martin Fowler's article on consumer-driven contracts (with Ian Robinson) is the canonical statement of the pattern, and the integration test pyramid discussion in his testing material is the source of the "keep a handful of end-to-end tests" guidance.

Google's and Microsoft's published testing guidance both emphasise hermetic tests and small numbers of end-to-end tests for the same reasons: flakiness and maintenance cost scale badly with the number of components under test.

The debate

The case for a full integration environment: it tests the real thing, it catches emergent behaviour that no interface test can, and it is conceptually simple. Regulated environments sometimes require it, and for a small number of services it is genuinely fine.

Its failure is scale, and specifically the flakiness dynamic: at 40 services an end-to-end suite is red often enough that the team stops treating red as information. A suite everyone retries is worse than no suite, because it costs time and confers confidence it has not earned.

The case for schema-only compatibility checking (OpenAPI diff, Protobuf breaking change detection): far cheaper to adopt, no provider verification tests, no broker, and it catches the majority of breakages, which are shape changes.

Its weakness is that it verifies the specification rather than the implementation, and it cannot express semantic expectations. A provider that starts returning status: "SHIPPED_PARTIAL" has not broken the schema and has broken every consumer that branches on status.

My position: contract tests as the integration layer, replacing most of the end-to-end suite; three to five end-to-end tests for genuinely emergent cross-service journeys; synthetic monitoring in production for those same journeys, because that is the version that tests reality. Adopt incrementally, starting with the two most coupled services rather than attempting full coverage. And if a provider team will not write verification tests, take bi-directional contracts rather than nothing.

Contract testing is the wrong investment for a small number of services (below about five, an integration environment is cheaper), when the provider is a third party you cannot run verification against (use recorded interactions and a compatibility check instead), and for genuinely emergent multi-service behaviour, which needs the end-to-end tests you kept.

Follow-up Q&A

"How do you test 40 microservices without a full integration environment?" Consumer-driven contract tests. Each consumer declares what it needs in executable form; the consumer tests run against a mock built from that; the provider's CI replays every consumer's contract against the real provider and fails if it no longer satisfies them. The two sides never run at the same time, so there is no shared environment and no cross-team orchestration. Then a broker plus can-i-deploy as a deploy gate, and three to five end-to-end tests kept for genuinely emergent behaviour, backed by synthetic monitoring in production.

"How is that different from schema validation?" A schema says the response is well-formed. A contract says this consumer reads these specific fields and depends on these specific semantics. A provider that adds an enum value has not broken the schema and has broken every consumer that branches on it exhaustively. Schema checking is much cheaper and catches most breakages, which is why bi-directional contracts (provider publishes OpenAPI, consumers publish contracts, broker checks compatibility) are a reasonable compromise when a provider team will not write verification tests.

"Doesn't this couple the teams?" Less than the alternative, and asymmetrically. The only coupling is the set of provider-state strings, which are named scenarios rather than schemas. The provider learns what consumers actually need, which is information it previously did not have, and a breaking change fails in the provider's own pipeline before merge rather than in someone else's environment a week later. Compare that to a shared environment, where every team's deploy is coupled to every other team's schedule.

"What can't contract testing catch?" Emergent behaviour. Two services can each satisfy their contracts perfectly and produce a wrong outcome in combination, for example marking an order shipped before payment settled. Also performance, saturation, and anything about the interaction of load with correctness. That is precisely why you keep a handful of end-to-end tests and run synthetic monitors in production, and being explicit about the gap is what makes the answer honest.

"How would you migrate an existing end-to-end suite?" Incrementally, and by deletion. Start with the two most coupled services and three interactions, to prove the loop end to end including the broker and the deploy gate. Expand consumer-driven, one consumer team at a time. Then apply the rule: any end-to-end test that would fail only because an interface changed becomes a contract test and gets deleted from the suite. Keep the ones that test genuine cross-service journeys. In practice that takes a 140-test suite to under ten, and the runtime reduction is what buys you the political capital for the rest.

Common misconceptions

The most common is that contract tests replace all integration testing. They replace the interface-verification portion, which is most of it. Emergent behaviour still needs a small end-to-end suite.

The second is that the effort scales with the number of services squared. The interaction graph is sparse: 40 services typically have fewer than 100 real consumer-provider edges, not 1,560, so the work scales with edges.

The third is that a red end-to-end suite is still providing value. Past a certain flakiness rate it provides negative value, because it consumes attention and trains the team to ignore failures, which is exactly when a real breakage slips through.

Interview delivery note

Say this: "Consumer-driven contract tests. Each consumer declares what it actually needs from a provider in executable form, its own tests run against a mock generated from that, and the provider's CI replays every consumer's contract against the real provider. The key property is that the two sides never run at the same time, so there's no shared environment and no cross-team scheduling. Then can-i-deploy as a gate, which answers 'is this version safe to ship' from recorded facts about what's deployed rather than from a test run."

Then the honest boundary, which is what makes it credible: "What contracts can't catch is emergent behaviour: two services can each satisfy their contracts and produce a wrong outcome together. So I'd keep three to five end-to-end tests for genuine cross-service journeys, and I'd put more weight on synthetic monitoring in production than on a large staging suite."

The depth signal is the sparsity observation: "forty services doesn't mean 1,560 pairs, it usually means fewer than a hundred real edges, which is what makes this tractable." And the flakiness argument: "a suite that's red 40 percent of the time has negative value, because it trains the team to ignore red."

The Staff & Team Lead Interview Reference