Experimentation: randomisation unit, power, and guardrails
What it is
The design decisions that determine whether an A/B test measures what it claims to. Three of them dominate, and each has a specific failure mode when chosen wrongly:
RANDOMISATION UNIT What gets assigned to a variant: a
request, a session, a user, an account,
a geography? Wrong choice -> the result
is invalid, not merely noisy.
POWER Can the test detect the effect you care
about, given traffic and duration?
Under-powered -> you conclude "no effect"
when there was one.
GUARDRAILS Metrics that must not regress even if the
primary metric improves. Absent -> you
ship a conversion win that increased
latency and churn.
Commonly confused with the statistics. The statistical test is the easy part; the randomisation unit and the guardrail set are where experiments are invalidated, and they are decisions made before any data is collected.
Also commonly confused with a canary. A canary asks is this safe, over minutes, on technical metrics, defaulting to rollback. An experiment asks is this better, over weeks, on business metrics, defaulting to keeping the control.
The problem it solves
Without experiment discipline, the organisation ships changes based on numbers that are wrong in a consistent direction: optimistic.
The three ways a naive experiment lies:
1. PEEKING. Checking daily and stopping when significant.
With daily checks over two weeks, the false positive rate
rises from the nominal 5% to roughly 30%.
-> One in three "wins" is noise.
2. WRONG RANDOMISATION UNIT. Randomising by request when the
change affects the user experience across requests. Users
see both variants, the effect is diluted, and the variance
calculation is wrong because observations are not
independent.
3. NO GUARDRAILS. The primary metric improved and latency
regressed 15%, or the change cannibalised another surface,
or it improved this week's conversion by degrading
retention.
And the base rate makes it worse. Published experience from large experimentation programmes is that a minority of ideas produce a positive effect, so with a 5 percent false positive rate and a low true-positive base rate, a meaningful fraction of "significant" results are false, before any peeking.
Mechanics
The randomisation unit
The rule: randomise at the level at which the experience is consistent and at which the effect operates.
UNIT USE WHEN BREAKS WHEN
------------------------------------------------------------
request the change is invisible the user could notice
across requests (a backend inconsistency; or
routing change, an infra the effect accumulates
swap) across a session
+ maximum statistical power
+ smallest sample needed
session the change affects a visit the effect persists
but not identity across visits (learning,
habit, a saved setting)
user the change is user-visible users share accounts,
and may persist. THE DEFAULT or the effect spills
for product changes between users
account / the change affects a shared accounts are few, so
workspace workspace (B2B) power is very low
geography / the effect SPILLS between geo units are few and
market users (marketplace supply, heterogeneous, so
social network effects, variance is enormous
pricing)
The failure that invalidates rather than dilutes: interference.
MARKETPLACE EXAMPLE
Test: show sellers a new pricing recommendation.
Randomise by seller.
Treatment sellers lower their prices and win more sales.
Control sellers lose those sales TO the treatment sellers.
Measured effect = treatment gain + control loss.
*** The measured effect is roughly DOUBLE the true effect,
because the control group was harmed by the treatment. ***
Fix: randomise by MARKET (geography, category), so the
competition happens within a variant rather than across
them. Cost: far fewer units, so far less power, so a much
longer test.
Interference is the failure people miss, and it is not a small correction. It applies to marketplaces, social products, anything with shared inventory, and anything where treatment users' behaviour changes what control users see.
Consistent bucketing is a requirement, not a nicety:
def assign(user_id: str, experiment_id: str, split: float) -> str:
# Hash the (user, experiment) pair, NOT just the user.
# Hashing the user alone means the same users land in
# treatment for every experiment, so effects correlate
# across experiments and the "control" population is
# systematically different.
h = hashlib.sha256(f"{experiment_id}:{user_id}".encode()).digest()
bucket = int.from_bytes(h[:8], "big") / 2**64
return "treatment" if bucket < split else "control"
Salting by experiment id is the detail, and hashing the user alone is a real and common bug that makes every experiment's populations correlated.
Power, and the traffic you actually need
n per variant ≈ 16 × σ² / Δ² (for 95% confidence,
80% power, two-sided)
For a PROPORTION (conversion rate):
n ≈ 16 × p(1-p) / Δ²
WORKED: baseline conversion 3%, want to detect a 5% RELATIVE
lift (3% -> 3.15%, so Δ = 0.0015)
n = 16 × 0.03 × 0.97 / (0.0015)^2
= 16 × 0.0291 / 0.00000225
= 206,933 per variant
= 413,866 total
At 50,000 daily users into the experiment:
413,866 / 50,000 = 8.3 days minimum.
Round up to 14 for a full two weekly cycles.
Two rules that follow from that arithmetic:
Run for whole weeks. Behaviour differs by day of week substantially, and a test running Tuesday to Tuesday is fine while one running Tuesday to Friday over-samples weekdays. Always whole multiples of seven days.
Small effects are expensive. Detecting a 1 percent relative lift instead of 5 percent needs 25 times the sample, so 8 days becomes 208 days. Which means most experiments cannot detect the effects most changes actually have, and the honest response is either to bundle changes or to accept that you are only testing for large effects.
Peeking, and the two legitimate fixes
THE PROBLEM
A fixed-horizon test's p-value is valid ONLY at the
pre-specified sample size. Checking repeatedly and stopping
at the first significant result inflates the false positive
rate:
checks actual false positive rate (nominal 5%)
1 5%
2 8%
5 14%
10 19%
daily/14 ~30%
FIX 1: DO NOT LOOK. Pre-register the sample size and check
once. Simple, and organisationally unrealistic:
someone will look.
FIX 2: SEQUENTIAL TESTING. Use a method designed for
continuous monitoring:
- always-valid p-values / mixture sequential
probability ratio test (Optimizely's approach)
- group sequential with alpha spending (O'Brien-
Fleming boundaries)
+ You can look continuously and stop early legitimately.
- Costs roughly 10-25% more samples for the same power,
which is the price of the option to stop early.
Sequential testing is the right default for a platform, because "do not look" is not a policy anyone follows, and building the correct method into the tool removes the temptation rather than relying on discipline.
Guardrails
Metrics that must not regress, checked automatically, regardless of the primary result.
THE STANDARD SET
latency p50, p95, p99 a conversion win that costs
200 ms is often not a win
error rate
crash rate (mobile)
page load / Core Web Vitals
unsubscribe / opt-out rate
support ticket rate
cost per request
BUSINESS GUARDRAILS
revenue per user a conversion lift that lowers
order value is a loss
retention (7d, 28d) the most important and the
slowest to measure
cross-surface cannibalisation did the win come from
elsewhere in the product?
The guardrail statistics are different from the primary metric's, and this is the part that gets done wrong.
PRIMARY METRIC: you want to detect an effect. You control
the FALSE POSITIVE rate (don't claim a win
that isn't real).
GUARDRAIL: you want to detect HARM. You should control
the FALSE NEGATIVE rate (don't miss a
regression that is real).
-> A guardrail should use a LOOSER significance threshold
and a non-inferiority framing: "is it worse by more than
X?" rather than "is it different?"
# Non-inferiority test for a guardrail: we do not need to
# prove it did not change, only that it is not worse by more
# than the tolerated margin.
def guardrail_ok(treat, control, margin=0.02, alpha=0.10):
"""margin: how much regression is tolerable (2%).
alpha 0.10 rather than 0.05: we accept more false
ALARMS in exchange for missing fewer real regressions,
which is the correct asymmetry for harm detection."""
diff = mean(treat) - mean(control)
se = pooled_se(treat, control)
# Upper bound of the confidence interval on the harm.
ci_upper = diff + norm.ppf(1 - alpha) * se
return ci_upper < margin * mean(control)
The metric hierarchy
1. PRIMARY (exactly ONE, chosen before the test)
The metric the change is supposed to move. One, because
testing five metrics at 5% each gives a 23% chance of at
least one false positive.
2. SECONDARY (a small number, for mechanism)
Metrics that explain HOW the primary moved. Not for
decisions.
3. GUARDRAILS (a fixed standard set)
Must not regress. Same set for every experiment, so
nobody chooses them per test.
4. DIAGNOSTIC (unlimited)
Sample ratio, assignment counts, instrumentation health.
Not outcomes; validity checks.
The sample ratio mismatch check in tier 4 is the highest-value validity check available:
Expected 50/50. Observed 50.4/49.6 over 400,000 users.
Chi-squared p-value: 0.0003.
*** That is not chance. Something is broken: bot filtering
differing by variant, a crash in treatment causing users
to disappear, a redirect losing assignments, or
instrumentation dropping events on one side. ***
An SRM invalidates the experiment ENTIRELY. Do not analyse
it, find the bug.
Published experience from large programmes is that SRM appears in a noticeable fraction of experiments and almost always indicates a real bug, which is why it belongs as an automatic check rather than something to remember.
A worked example: a win that was not
EXPERIMENT
Simplify the checkout form from 3 steps to 1.
Primary: checkout completion rate.
Randomised by user, 50/50, two weeks.
RESULT
completion rate: 12.4% -> 13.9% +12.1% relative
p < 0.001
Team declares a win.
WHAT THE GUARDRAILS SAID
revenue per completed order: $58.20 -> $51.40 -11.7%
p99 page latency: 1.2s -> 1.4s +16%
7-day retention: flat (not yet powered)
support tickets/1k orders: 4.1 -> 6.8 +66%
DIAGNOSIS
The single-step form dropped the upsell placement that had
been on step 2. More people completed, each spent less.
revenue per user = 0.139 x 51.40 = $7.14
0.124 x 58.20 = $7.22
*** Revenue per user DECREASED by 1.1%. ***
And the support ticket increase traced to the simplified
form allowing an ambiguous address format that the
three-step version had validated.
DECISION
Do not ship. Iterate: keep the single step, restore the
upsell as an inline module, restore the address validation.
WHAT MADE THIS CATCHABLE
a. Revenue per USER was a guardrail, not just conversion
rate. Conversion is a rate; the business cares about the
product of rate and value.
b. Support tickets were instrumented as a guardrail at all,
which most teams do not do.
c. The primary metric was chosen BEFORE the test, so
nobody could retroactively declare revenue the primary
and call it a loss, or declare conversion the primary
and call it a win. It was fixed.
The generalisable lesson: a rate improvement that is not checked against value per unit is not a business result, and conversion rate is the most commonly mis-used primary metric for exactly this reason.
Production evidence
Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments (2020) is the definitive reference and the source for most of the practices here: the metric hierarchy, guardrails, SRM as a validity check, and the finding that a minority of ideas produce positive effects.
Kohavi et al.'s "Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained" (KDD 2012) documents real cases where naive analysis produced wrong conclusions, including interference and instrumentation effects.
Microsoft's published SRM work ("Diagnosing Sample Ratio Mismatch in Online Controlled Experiments", KDD 2019) documents SRM's frequency and its causes, and is the basis for treating it as a hard validity gate rather than a warning.
Optimizely's Stats Engine implements always-valid p-values via a mixture sequential probability ratio test, and their published explanation of why fixed-horizon p-values break under continuous monitoring is a good reference for the peeking problem.
Airbnb's and Netflix's published experimentation work both document interference in marketplace and content settings and the switch to market-level or time-based randomisation, including the power cost.
Google's "Overlapping Experiment Infrastructure" (Tang et al., KDD 2010) is the reference for running many experiments simultaneously with layered randomisation, which is what makes experiment throughput possible without exhausting traffic.
The debate
The case for rigorous experimentation: most ideas do not work, intuition is a poor guide, and without controlled measurement an organisation ships changes based on numbers that are optimistic in a consistent direction. The discipline is what converts opinion into evidence.
The case against: experiments are slow, most are under-powered for the effects that actually exist, and the ceremony can substitute for product judgement. A team that A/B-tests every button colour is spending traffic and weeks on decisions that do not need evidence.
The case for shipping and monitoring: for reversible, low-risk changes, ship behind a flag, watch the metrics, and roll back if they move. Faster, and it conflates a change with everything else happening that week.
My position: experiment on things that change user behaviour, use the standard guardrail set on everything, and pick the randomisation unit before anything else.
The randomisation unit is the decision I would spend the most time on, because getting it wrong invalidates rather than degrades the result. Randomising by request when the effect operates across a session dilutes the effect and breaks the independence assumption behind the variance calculation. And interference is the case people miss entirely: in a marketplace, treatment sellers win sales from control sellers, so the measured effect is roughly double the true one, and the fix is market-level randomisation at a large cost in power.
Sequential testing should be the platform default, because "do not peek" is not a policy anyone follows and daily checks over two weeks take the false positive rate from 5 percent to about 30. Building always-valid p-values into the tool removes the temptation instead of relying on discipline, at a cost of roughly 10 to 25 percent more samples, which is a good trade.
The asymmetry I would insist on is that guardrails use different statistics from the primary metric. For the primary you control false positives, because you do not want to claim a win that is not real. For a guardrail you are detecting harm, so you should control false negatives: a looser alpha and a non-inferiority framing, "is it worse by more than X" rather than "is it different". Applying the primary metric's threshold to guardrails means missing real regressions, which is exactly backwards.
And one primary metric, chosen before the test. Five metrics at 5 percent each gives a 23 percent chance of a false positive somewhere, and more importantly a fixed primary prevents the retroactive reframing that makes every experiment a win. In the worked example, that discipline is what made "conversion up 12 percent, revenue per user down 1 percent" a clear non-ship rather than an argument.
Where I would push back: most experiments cannot detect the effect the change actually has. Detecting a 1 percent relative lift instead of 5 takes 25 times the sample, so eight days becomes seven months. The honest responses are to bundle changes so the effect is larger, to accept that you are only testing for large effects, or to not experiment on things where the expected effect is below your MDE and use judgement instead.
Follow-up Q&A
"What's the most important design decision?" The randomisation unit, because getting it wrong invalidates the result rather than just adding noise. Randomise at the level at which the experience is consistent and the effect operates. Request-level gives the most power and is only valid when the change is invisible across requests. User-level is the default for product changes. And market-level when there is interference, which is the case people miss.
"What is interference and why does it matter so much?" When the treatment group's behaviour affects the control group's outcomes. In a marketplace, treatment sellers lower prices and win sales from control sellers, so the measured effect is the treatment's gain plus the control's loss, roughly double the true effect. It applies to marketplaces, social products, shared inventory, anything where one user's treatment changes another user's experience. The fix is randomising by market or geography so competition happens within a variant, at a large cost in power because there are far fewer units.
"How much traffic does an experiment need?" Roughly 16 times the variance over the squared effect size. For a 3 percent baseline conversion detecting a 5 percent relative lift, that is about 207,000 per variant, so 414,000 total, which at 50,000 daily users is 8 days, rounded to 14 for two full weekly cycles. And the harsh part: detecting a 1 percent lift instead of 5 needs 25 times the sample, so 8 days becomes 208. Most experiments cannot detect the effect their change actually has.
"Why run for whole weeks?" Because behaviour differs substantially by day of week, so a test running Tuesday to Friday over-samples weekdays and one running Tuesday to Tuesday does not. Always whole multiples of seven days, and it is a cheap discipline that removes a real source of bias.
"What's wrong with checking the results daily?" A fixed-horizon p-value is valid only at the pre-specified sample size. Checking daily over two weeks takes the false positive rate from a nominal 5 percent to roughly 30, so one in three "wins" is noise. The two fixes are pre-registering the sample size and looking once, which nobody actually does, or sequential testing with always-valid p-values, which lets you look continuously and costs 10 to 25 percent more samples. I would make sequential the platform default, because building it into the tool removes the temptation rather than relying on discipline.
"How do guardrails differ from the primary metric?" In what error you control. For the primary you control false positives, because you do not want to claim a win that is not real. For a guardrail you are detecting harm, so you control false negatives: a looser alpha, say 0.10, and a non-inferiority framing, "is it worse by more than two percent" rather than "is it different". Using the primary's threshold on guardrails means missing real regressions, which is exactly the wrong asymmetry.
"What's the highest-value validity check?" Sample ratio mismatch. If you expected 50/50 and observed 50.4/49.6 over 400,000 users, the chi-squared p-value is 0.0003, which is not chance: something is broken, usually bot filtering differing by variant, a crash in treatment causing users to disappear, a redirect losing assignments, or instrumentation dropping events on one side. An SRM invalidates the experiment entirely, so the correct response is not to analyse it but to find the bug. It should be an automatic gate, not something to remember.
"Conversion went up twelve percent. Do you ship?" Not without checking value per unit. In a case I worked, checkout completion went from 12.4 to 13.9 percent, and revenue per completed order fell from $58.20 to $51.40 because the simplified form dropped an upsell placement. Revenue per user was 0.139 times 51.40 against 0.124 times 58.20, so it decreased by about one percent. A rate improvement not checked against value per unit is not a business result, and conversion rate is the most commonly mis-used primary metric for exactly that reason.
"How many metrics should an experiment have?" One primary, chosen before the test. Five metrics at five percent each gives a 23 percent chance of at least one false positive, and more importantly a fixed primary prevents the retroactive reframing that makes every experiment a win. Then a small number of secondaries to explain the mechanism, a fixed standard guardrail set that is the same for every experiment so nobody chooses them per test, and unlimited diagnostics, which are validity checks rather than outcomes.
"When would you not run an experiment?" When the expected effect is below your minimum detectable effect, which is more often than teams assume. Also for changes that are clearly correct (a bug fix, an accessibility improvement), for changes where the ethics of a control group are questionable, and for reversible low-risk changes where shipping behind a flag and watching the metrics is faster. The failure mode on the other side is real too: a team that A/B-tests every button colour is spending weeks and traffic on decisions that do not need evidence.
Common misconceptions
"Randomise by request for more power." Only valid when the change is invisible across requests. Otherwise the effect is diluted and the independence assumption behind the variance calculation is violated.
"Check the results as they come in." Daily checks over two weeks take the false positive rate from 5 to about 30 percent. Use sequential testing or do not look.
"Statistical significance means the change is good." It means the primary moved. Without guardrails, a conversion win that regressed revenue per user, latency and support load looks identical to a real win.
"A guardrail is just another metric." It has an inverted error asymmetry: you are detecting harm, so control false negatives with a looser alpha and a non-inferiority test.
"A small sample ratio imbalance is fine." An SRM that is statistically significant almost always indicates a real bug and invalidates the experiment entirely.
Interview delivery note
Lead with the randomisation unit, because it is the decision that invalidates rather than degrades: "The first thing I'd decide is the randomisation unit, because getting it wrong invalidates the result rather than adding noise. Randomise at the level at which the experience is consistent and the effect operates: request-level only when the change is invisible across requests, user-level for most product changes."
Then the case people miss: "And market-level when there's interference. In a marketplace, treatment sellers win sales from control sellers, so the measured effect is the treatment gain plus the control loss, roughly double the true effect. The fix is randomising by geography so the competition happens inside a variant, and it costs a lot of power because there are far fewer units."
Do the power arithmetic and land on the uncomfortable implication: "For a three percent baseline detecting a five percent relative lift, that's about 207,000 per variant, so eight days at fifty thousand daily users, rounded to fourteen for two full weekly cycles. And detecting one percent instead of five needs twenty-five times the sample, so most experiments can't detect the effect their change actually has."
Give the peeking number, because it is startling: "Checking daily over two weeks takes the false positive rate from five percent to about thirty. So one in three wins is noise. I'd make sequential testing the platform default rather than relying on people not looking, because nobody doesn't look."
The guardrail asymmetry is the depth signal: "and guardrails need different statistics from the primary. For the primary you control false positives; for a guardrail you're detecting harm, so you control false negatives with a looser alpha and a non-inferiority framing. Using the primary's threshold on guardrails means missing real regressions, which is backwards."
Close with the worked case: "Checkout conversion up twelve percent, revenue per completed order down twelve, so revenue per user down one. A rate improvement not checked against value per unit isn't a business result, and conversion rate is the most commonly mis-used primary metric for exactly that reason."
Further reading
- Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments (2020).
- Fabijan et al., "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments" (KDD 2019).
- Tang et al., "Overlapping Experiment Infrastructure: More, Better, Faster Experimentation" (KDD 2010).
- Johari et al., "Always Valid Inference: Continuous Monitoring of A/B Tests" (Optimizely / Stanford), for sequential testing.
- Airbnb's and Netflix's engineering write-ups on interference and market-level randomisation.