Shadow traffic and side-effect containment
What it is
Duplicating production requests to a new version whose responses are discarded, so the new version experiences real traffic without affecting any user.
┌──────────────┐
request ────────►│ PRODUCTION │────► response to the user
│ └──────────────┘
│
└─ copy ───►┌──────────────┐
│ SHADOW │────► response DISCARDED
└──────────────┘ (and compared, logged,
or measured)
Commonly confused with a canary. A canary serves real users and its failures are visible; a shadow serves nobody and its failures are invisible. That difference is the entire point and also the entire risk: shadow traffic is safe on the response path and dangerous on the side-effect path.
Also called traffic mirroring, dark launching (though that term also covers flag-gated code paths), and dark traffic.
The three things it is used for, which have different requirements:
LOAD VALIDATION Does the new version handle production
traffic volume and shape? Only needs the
requests, not response comparison.
BEHAVIOURAL DIFF Does the new version produce the same
responses? Needs response capture and
comparison, and a notion of "same" that
tolerates legitimate differences.
PRODUCTION-LIKE Does it work against real data with real
TESTING cardinality, real edge cases and real
malformed input? The value is the input
distribution, which no synthetic test has.
The problem it solves
Staging environments lie, and they lie in specific, predictable ways:
Staging has Production has
--------------------------------------------------------------
1% of the data 100%, with different cardinality and skew
synthetic traffic real traffic, including malformed input,
bots, unusual clients, and the one
customer whose integration is wrong
even load diurnal peaks, spikes, correlated bursts
one region multi-region latency and partial failures
fresh caches warm caches with specific hot keys
The specific class of bug shadow traffic finds: the one that only appears against the real input distribution. A parser that fails on 0.01 percent of requests will never be hit in staging and will be hit 400 times an hour in production. A query that is fast on 10,000 rows and quadratic will be fine in staging.
And the class it finds that nothing else does: performance under the real traffic shape. Load tests use synthetic distributions; shadow traffic uses the actual one, including the cache hit rate that real key distribution produces.
Mechanics
Side-effect containment: the whole problem
Shadowing reads is trivial. Shadowing writes is where systems get destroyed.
Request: POST /orders { items: [...], payment_token: "..." }
Shadowed naively:
-> the shadow charges the customer's card AGAIN
-> the shadow sends a second confirmation email
-> the shadow decrements inventory a second time
-> the shadow publishes a duplicate OrderCreated event,
which every downstream consumer processes
Four containment strategies, in ascending order of cost and fidelity:
1. READ-ONLY SHADOWING
Mirror only idempotent GET requests. Trivially safe.
Covers a lot: for most services reads are the majority of
traffic and most of the latency risk.
Misses: everything about the write path.
2. STUBBED SIDE EFFECTS
The shadow runs the full code path with external calls
replaced by no-ops or recorded stubs.
+ Exercises the real logic and the real data access.
- The stub boundary is code that only runs in shadow mode,
so it is untested code in the shadow path, and a bug
there can leak a real side effect.
3. SHADOW-SPECIFIC DEPENDENCIES
The shadow points at a separate database, a test payment
sandbox, a null email transport.
+ No possibility of a real side effect.
- The shadow database has different data, so behavioural
comparison is invalid and performance is unrepresentative
because cache and index state differ.
4. TRANSACTIONAL ROLLBACK
The shadow runs against the real database inside a
transaction that is always rolled back.
+ Real data, real query plans, real contention.
- Holds locks against production data, which can affect
production. Does not contain non-transactional side
effects (HTTP calls, queue publishes, cache writes).
- Genuinely risky and I would use it rarely.
The design rule that makes this tractable: side effects must be behind an interface that can be swapped, and that interface must exist for reasons other than shadowing. If the payment gateway is called via a direct SDK invocation scattered through the code, there is no seam and shadowing writes is not safely possible.
# The seam is the design, not the shadow config.
class OrderService:
def __init__(self, payments: PaymentGateway,
email: EmailSender, events: EventPublisher):
...
# Production wiring
OrderService(StripeGateway(), SesEmailSender(), KafkaPublisher())
# Shadow wiring: every external effect is explicitly a no-op
# that RECORDS what it would have done, so the comparison can
# assert on intended side effects without performing them.
OrderService(RecordingNoOpGateway(), RecordingNoOpEmail(),
RecordingNoOpPublisher())
Recording no-ops rather than silent ones is the improvement worth making: the shadow then tells you "this version would have charged $40 where production charged $40", which is a behavioural comparison of the side effects themselves rather than only of the responses.
Response comparison, and what "same" means
def compare(prod_response, shadow_response) -> Diff:
p, s = normalise(prod_response), normalise(shadow_response)
return deep_diff(p, s)
def normalise(r):
"""Legitimate differences must be stripped or every
comparison is a diff and the signal is lost."""
r = strip_fields(r, [
"request_id", "trace_id", # per-request identifiers
"server_time", "generated_at", # timestamps
"server_hostname",
"cache_hit", # differs by construction
])
r = sort_unordered_collections(r) # set-valued fields
r = round_floats(r, places=4) # float formatting drift
return r
The normalisation is where the work is, and getting it wrong in either direction is costly: too little and every response diffs so nobody looks at the report, too much and you normalise away the difference you were looking for.
The classification that makes the report usable:
IDENTICAL after normalisation. The bulk, hopefully.
SEMANTIC MATCH different representation, same meaning
(field ordering, an optional field present
in one and absent in the other with the
same default)
DIFFERENT a real behavioural change. Bucket by shape
rather than listing, because 40,000 diffs
of the same kind is one finding.
SHADOW ERROR the shadow threw where production did not.
The highest-value bucket.
PROD ERROR production threw and the shadow did not,
which is a fix and worth confirming.
Bucketing by diff shape rather than by instance is what makes the output actionable. A raw diff log at production volume is unreadable; "3 distinct diff shapes across 41,000 requests" is a finding.
The load question: does the shadow double your dependencies' load?
Yes, and it is frequently forgotten.
Shadowing 100% of traffic to a new version means:
- the shadow's database queries hit the SAME database
- the shadow's cache reads hit the SAME cache (and its
writes pollute it)
- the shadow's downstream calls hit the SAME services
*** Your dependencies now serve 2x the traffic. ***
The mitigations:
SHADOW A FRACTION 10% of traffic still gives a large
sample and adds 10% dependency load.
Usually the right answer.
SEPARATE READ REPLICAS Point the shadow at replicas, so
primary load is unaffected.
RATE LIMIT THE SHADOW Cap its outbound calls, accepting
that it is then not a faithful load
test.
MARK SHADOW TRAFFIC A header propagated through every
call, so downstream services can
rate-limit, log or reject it, and so
it does not pollute their metrics.
The header propagation is the one that matters most operationally, and it needs to be
end-to-end: X-Shadow-Request: true on every hop, so a downstream service's error rate and
latency dashboards can exclude shadow traffic. Without it, shadowing corrupts every
downstream service's SLI, which is a subtle and annoying failure.
What shadow traffic cannot tell you
CANNOT: whether users like it. No user sees the response.
CANNOT: conversion, engagement, or any business metric.
CANNOT: whether the write path is correct end-to-end, unless
you accept real side effects.
CANNOT: how it behaves under a traffic pattern it did not
receive, so it will not surface a Black Friday
failure in March.
CANNOT: interactive or stateful flows easily. A multi-step
checkout shadowed statelessly does not reproduce the
session.
And a subtle one: it cannot tell you about feedback effects. If the new ranking model would change what users click, the shadow sees the old model's click distribution, so any metric that depends on user response is measured under the wrong distribution.
A worked example: replacing a search backend
GOAL: replace an ageing search service with a rewrite.
High risk: the query language, the ranking and the
index are all changing.
PHASE 1: READ-ONLY SHADOW, 10% (two weeks)
Mirror GET /search to the new service. Responses discarded.
Measure: latency distribution, error rate, resource usage.
Findings:
- p99 was 2.4x production's, traced to a missing index on
a field used by 3% of queries. Invisible in staging,
where that field was never populated.
- 0.4% of queries returned a 500, all containing a
specific unicode normalisation form that the parser did
not handle. Zero occurrences in the staging corpus.
PHASE 2: READ-ONLY SHADOW WITH COMPARISON, 100% (three weeks)
Capture both response sets and compare result IDs and order.
Normalisation: strip scores (different scales), strip
request ids, compare the top 20 IDs as an ordered list.
Findings, bucketed:
- 71% identical top-20
- 22% same set, different order -> expected, ranking
changed deliberately
- 6% different set, overlapping -> investigated, mostly
the new analyser handling compounds differently, which
was intended
- 1.1% completely different -> a bug in filter
handling for a specific facet combination, found only
because it was 1.1% of a very large sample
PHASE 3: WRITE PATH, WITH STUBS
Index updates shadowed with a recording no-op writer.
Compare the intended index operations rather than
performing them.
Finding: the new service issued 3.2x the index operations
for the same source events, because a change-detection
optimisation had been dropped in the rewrite. That would
have tripled indexing cost in production and would not have
been visible until the bill arrived.
PHASE 4: CANARY, 1% -> 5% -> 25% -> 100%
Only now do real users see it.
TOTAL: seven weeks of shadow before any user exposure.
The point of the example: each phase found a class of bug the previous one could not. Load shadowing found the index and the parser; comparison found the filter bug and confirmed the intended ranking changes; write shadowing found a cost regression that no functional test would catch. And none of them exposed a user to any of it.
Production evidence
Envoy's request mirroring (request_mirror_policies) implements traffic shadowing at the
proxy layer with a configurable fraction, and it explicitly discards the shadow response and
does not let it affect the primary. Istio exposes the same through its mirror and
mirrorPercentage settings.
GitHub's Scientist library is the reference for the in-process version: run both the old and new code paths, return the old result, and report mismatches. Its design decisions (sampling, ignoring known-acceptable diffs, never letting the candidate's exception surface) are the same containment problem at a smaller scope.
Diffy (Twitter, open-sourced) is a dedicated response-diffing proxy that runs three instances (two of the current version and one candidate) to distinguish real behavioural differences from non-determinism, which is a good answer to the normalisation problem.
AWS's traffic mirroring at the VPC level and Nginx's mirror directive are the
infrastructure-layer versions, and their documentation is consistent that the mirrored
response is discarded.
Netflix's and LinkedIn's published migration practices both describe extended shadow periods before user-facing rollout for high-risk replacements, which is the multi-phase structure above.
The debate
The case for shadow traffic: it is the only way to test against the real input distribution, real data cardinality and real traffic shape without exposing a user to anything. For a high-risk rewrite it finds classes of bug that no staging environment and no synthetic load test can.
The case against: it doubles dependency load, it requires side-effect seams that may not exist, the comparison infrastructure is real work, and it cannot tell you anything about whether users like the change. For an incremental change, a canary is faster and tells you more.
The case for going straight to a canary: a canary exposes a small number of real users and gives you real feedback including business metrics, in minutes rather than weeks. Shadow traffic is weeks of work to learn less.
My position: shadow for high-risk replacements, canary for incremental changes, and never shadow a write path without an explicit side-effect seam.
The discriminator is whether the change is a replacement or an increment. Rewriting a search backend, replacing a database, or swapping a ranking model changes behaviour across the whole input distribution, and the only way to see that before users do is to run it against the whole input distribution. An incremental change does not need seven weeks; it needs a canary.
The rule I would not bend is the side-effect seam is a design property, not a shadow configuration. If the payment gateway is invoked directly from scattered call sites, there is no safe way to shadow the write path, and the honest answer is to shadow reads only until the seam exists. Teams that improvise containment inside the shadow path are writing untested code whose only job is to prevent a catastrophic side effect, which is the worst possible place for a bug.
Two operational details I would insist on. Recording no-ops rather than silent ones, so the shadow reports "this version would have charged $40" and you can compare intended side effects rather than only responses. And an end-to-end shadow header, propagated through every downstream hop, because without it shadow traffic pollutes every downstream service's error rate and latency SLIs, which is subtle, annoying, and erodes trust in the practice.
On volume, I would shadow 10 percent rather than 100 in most cases. It still gives an enormous sample at production traffic, and it adds 10 percent rather than 100 percent to every dependency's load. Full mirroring is for the phase where you specifically need to see the full traffic shape.
Where I would set expectations honestly: shadow traffic cannot tell you whether the change is good. No user sees the response, so there is no conversion, no engagement, no satisfaction signal, and any metric that depends on user response is measured under the old model's distribution. It answers "is it safe and does it behave the same", and the "is it better" question needs an A/B test.
Follow-up Q&A
"What is shadow traffic for?" Testing against the real input distribution, real data cardinality and real traffic shape without exposing any user. Staging lies in predictable ways: one percent of the data with different skew, synthetic traffic without the malformed input and unusual clients, even load rather than diurnal peaks, and cold caches. The specific bug class it catches is the one that only appears against real inputs, like a parser failing on 0.01 percent of requests, which is never hit in staging and is hit four hundred times an hour in production.
"What's the danger?" Side effects. Shadowing reads is trivial; shadowing a POST /orders
naively charges the card again, sends a second email, decrements inventory twice and publishes
a duplicate event that every downstream consumer processes. So the containment strategy is the
design, and the rule I would hold is that the side-effect seam has to be a property of the
code rather than something improvised in shadow mode. If the payment gateway is invoked
directly from scattered call sites, there is no safe way to shadow writes.
"What are the containment options?" Four, in ascending cost. Read-only shadowing, which is trivially safe and covers most traffic. Stubbed side effects, where the real logic runs with external calls replaced, which exercises real data access at the cost of a stub boundary that only runs in shadow mode. Shadow-specific dependencies, which are fully safe and make behavioural comparison invalid because the data differs. And transactional rollback against the real database, which gives real query plans and holds locks against production and does not contain non-transactional effects, so I would use it rarely.
"How do you compare responses?" Normalise first, and that is where the work is: strip request ids, timestamps, hostnames and cache-hit flags, sort unordered collections, round floats. Too little normalisation and every response diffs so nobody reads the report; too much and you normalise away what you were looking for. Then classify into identical, semantic match, different, shadow-error and prod-error, and bucket the differences by shape rather than listing instances, because forty thousand diffs of the same kind is one finding.
"Does shadowing double your load?" Yes, on every dependency, and it is frequently forgotten. The shadow's queries hit the same database, its cache reads hit the same cache and its writes pollute it, and its downstream calls hit the same services. So I would shadow ten percent rather than a hundred in most cases, which still gives an enormous sample and adds ten percent of load. And point it at read replicas where possible.
"What operational detail catches people?" The shadow header, propagated end to end. Without
X-Shadow-Request on every hop, shadow traffic is indistinguishable from real traffic in every
downstream service's dashboards, so it corrupts their error rates and latency SLIs. That is
subtle, annoying to diagnose, and it erodes trust in the practice quickly.
"What can't shadow traffic tell you?" Anything about whether users like it, because no user sees the response. So no conversion, no engagement, no satisfaction. And a subtle one: it cannot measure feedback effects, because if a new ranking model would change what users click, the shadow sees the old model's click distribution, so any metric depending on user response is measured under the wrong distribution. It answers "is it safe and does it behave the same"; "is it better" needs an A/B test.
"When would you use it, and when not?" For a replacement rather than an increment. Rewriting a search backend, swapping a database, replacing a ranking model: those change behaviour across the whole input distribution and the only way to see it before users do is to run against the whole input distribution. For an incremental change, a canary exposes a few real users, gives you business metrics, and takes minutes rather than weeks, so shadowing is weeks of work to learn less.
"Walk me through a real use." A search backend rewrite, four phases. Read-only shadow at ten percent for two weeks found a missing index causing 2.4x p99, and a parser failing on a unicode normalisation form absent from the staging corpus. Full-traffic shadow with response comparison for three weeks found that 71 percent of top-20 result sets were identical, 22 percent were reordered as intended, and 1.1 percent were completely different due to a filter bug. Write-path shadow with recording no-ops found the new service issuing 3.2 times the index operations because a change-detection optimisation had been dropped, which would have tripled indexing cost and shown up only on the bill. Then a canary. Seven weeks before any user exposure, and each phase found a class the previous could not.
Common misconceptions
"Shadow traffic is a safer canary." It is a different tool. A canary gives user feedback and business metrics; a shadow gives none, because nobody sees the response.
"Discarding the response makes it safe." The response path is safe. The side-effect path is where systems get destroyed, and containment is a design property rather than a configuration.
"Mirror 100 percent for the best signal." That doubles load on every dependency. Ten percent is usually plenty and adds a tenth of the load.
"The shadow doesn't affect production." Its queries share the database, its cache writes pollute the cache, and without a propagated header it corrupts every downstream service's SLIs.
"Response comparison is a diff." It is a normalisation problem. Without stripping legitimate differences, every response diffs and the report is worthless.
Interview delivery note
Distinguish it from a canary immediately, because that framing carries the whole answer: "A canary serves real users and its failures are visible. A shadow serves nobody and its failures are invisible, which is the point and also the risk: it's safe on the response path and dangerous on the side-effect path."
Name what it is uniquely good for: "It's the only way to test against the real input distribution. Staging has one percent of the data with different skew and synthetic traffic without the malformed input. A parser that fails on 0.01 percent of requests is never hit in staging and is hit four hundred times an hour in production."
Then go straight to containment, because that is where the difficulty is: "Shadowing reads is trivial. Shadowing a POST charges the card twice, sends a second email and publishes a duplicate event to every consumer. And the containment has to be a design property, not something improvised in shadow mode: if the payment gateway is called directly from scattered call sites, there's no seam and shadowing the write path isn't safely possible."
Two operational details that show you have run it: "I'd use recording no-ops rather than silent ones, so the shadow tells you 'this version would have charged forty dollars' and you can compare the intended side effects, not just responses. And a shadow header propagated end to end, because without it you're corrupting every downstream service's error rate and latency SLIs."
Set the expectation honestly at the end: "and it can't tell you whether the change is good. Nobody sees the response, so there's no conversion or engagement signal, and any metric depending on user response is measured under the old model's distribution. It answers 'is it safe and does it behave the same'. 'Is it better' needs an A/B test."
Further reading
- Envoy's
request_mirror_policiesdocumentation, and Istio'smirrorandmirrorPercentagesettings. - GitHub's Scientist library, for the in-process version and its containment decisions.
- Twitter's Diffy, for response diffing with a second control instance to handle non-determinism.
- Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments, for why shadowing cannot substitute for an experiment.