Peeking, novelty, interference and switchback tests

What it is

Four ways a correctly implemented A/B test produces a wrong answer. The randomisation can be perfect and the metrics accurate, and the conclusion still be false, because each of these violates an assumption the statistics depend on.

Pitfall        The assumption it breaks
------------------------------------------------------------------
Peeking        the sample size was fixed in advance, so the
               significance threshold means what it says
Novelty /      the treatment effect is stable over the measurement
primacy        window
Interference   a unit's outcome depends only on its own assignment
(SUTVA)        (stable unit treatment value assumption)
Sample ratio   assignment was actually random and complete
mismatch

Switchback testing is not a pitfall, it is the standard remedy for total interference: randomise over time periods applied globally rather than over units.

What this is confused with: statistical power. Power is about detecting a real effect that exists, and it is planned before the test (covered in experimentation design). These four are about believing an effect that is not there, or measuring the wrong one. A perfectly powered test can be ruined by all four.

Also confused: a canary and an experiment. A canary asks whether a build is healthy over minutes; an experiment asks whether a change is better over weeks. Sharing routing machinery does not make the analysis the same (see canary vs A/B testing).

The problem it solves

Each pitfall has a signature failure and they are all common enough to expect:

PEEKING          "we hit significance on day 3, shipped, and the
                 effect vanished in the holdback"

NOVELTY          "the new design won by 8% in week 1 and by 0.4% in
                 week 4, and we had already shipped it"

INTERFERENCE     "treatment sellers got 12% more orders. Marketplace
                 GMV did not move at all."

SRM              "the split was 50/50 but treatment had 47.2% of
                 users, and every metric was significant"

The interference case is the most expensive, because the test result is not merely wrong, it is confidently wrong in the direction the team wanted.

A marketplace ranking change, randomised by BUYER.
Treatment buyers see a ranking that surfaces small sellers.

Treatment: orders per buyer +12%, significant, p < 0.001.

But supply is finite. Treatment buyers bought items that control
buyers would otherwise have bought. The measured lift is partly
REALLOCATION between arms, not creation.

Total marketplace orders after full rollout: +0.3%, inside the
noise band.

The experiment measured cannibalisation and reported it as growth.

Mechanics

1. Peeking, and why it inflates false positives

A fixed-horizon test's p-value is only valid if you look once, at the planned sample size. Each additional look is another chance for random walk to cross the threshold.

Intuition, as an upper bound:
  If k looks were independent, P(at least one significant | no
  effect) = 1 - 0.95^k

    k=1   5.0%
    k=2   9.8%
    k=5  22.6%
    k=10 40.1%

Looks are NOT independent (the sample at look 5 contains the
sample at look 4), so the real inflation is lower than this bound,
but the direction and the mechanism are right: reported values in
the literature for continuous monitoring of a fixed-horizon test
are roughly 8% at 2 looks, 14% at 5, and rising toward certainty
as looks become continuous.

The limiting case is exact: under continuous monitoring of an
unbounded test, the probability of eventually crossing any fixed
threshold approaches 1 even with no true effect. That is the law
of the iterated logarithm, and it is why "just wait for
significance" is not a stopping rule.

Three legitimate remedies:

1. FIXED HORIZON. Compute the sample size in advance from the MDE,
   run to it, look once. Correct, and slow, and it is what people
   defect from.

2. GROUP SEQUENTIAL with alpha spending. Pre-specify K interim
   analyses and spend the total alpha across them.
     O'Brien-Fleming: very strict early, near-nominal at the end.
       Good when early stopping should be reserved for dramatic
       effects.
     Pocock: constant, stricter threshold at every look. Good when
       you want a real chance of stopping early.
   Cost: the final-look threshold is stricter than 0.05, so you
   need slightly more samples for the same power.

3. ALWAYS-VALID INFERENCE (sequential tests, e.g. mixture SPRT, or
   confidence sequences). The p-value or interval is valid at
   EVERY point in time, so continuous monitoring is legitimate by
   construction.
   Cost: less powerful than a fixed-horizon test at the planned
   sample size. You pay for the right to peek.

The honest framing: you can look continuously, or you can have maximum power, and you choose. Optimizely rebuilt its statistics engine around always-valid inference in 2015 for exactly this reason, because their users peeked regardless of instruction.

The guardrail that costs nothing: pre-register the primary metric, the sample size and the stopping rule before the test starts, in the experiment's own record. Most platforms support this and most teams skip it.

2. Novelty and primacy effects

NOVELTY   users react to the CHANGE, not the change's merit. A new
          banner gets clicked because it is new. The effect decays.
PRIMACY   users are worse with the new thing while they relearn.
          The effect improves. Common for navigation changes and
          anything with muscle memory.

Both mean the first days' measurement is not the steady-state effect, and the direction of the error is opposite, which is why "run it longer" is the shared remedy.

The diagnostic is to plot the effect by days since first exposure, not by calendar day:

Effect by user-tenure-in-experiment:

day 1   +8.2%
day 2   +6.1%
day 3   +4.4%
day 5   +2.0%
day 7   +0.9%
day 14  +0.3%   <- this is the effect

Calendar-day plots hide this, because on calendar day 10 you have
a mixture of users on their first day and users on their tenth.
The tenure-aligned view separates them.

Controls:
- Run at least 2 full weekly cycles. Weekday/weekend composition
  alone can move a metric more than the treatment.
- Analyse NEW users separately from EXISTING users. Novelty and
  primacy apply mainly to existing users, who have expectations;
  new users have none, so their curve is the closest thing to the
  steady state you can measure early.
- Keep a long-running HOLDBACK (1 to 5 percent never exposed) so
  you can measure the effect months later. This is the only way to
  detect an effect that decayed after you shipped.

The holdback is the control most organisations lack and the one that answers "did the thing we shipped last quarter actually do anything."

3. Interference: when SUTVA breaks

The stable unit treatment value assumption says a unit's outcome depends only on its own assignment. Four common violations:

FINITE SHARED SUPPLY (marketplaces, ads, inventory)
  Treatment buyers consuming a limited item deprive control buyers.
  The between-arm difference overstates the total effect, sometimes
  by all of it.

NETWORK / SOCIAL EFFECTS
  A treatment user posts more; their control friends see more
  content and engage more. The control arm is contaminated, so the
  measured difference UNDERSTATES the true effect.

SHARED RESOURCES
  Treatment consumes more cache, connections, or a rate-limit
  budget, degrading control's latency. Now control is worse for a
  reason unrelated to the change's merit.

LEARNING SYSTEMS
  Treatment and control share a model that trains on both arms'
  behaviour. Each arm's data changes the other's predictions.

Cluster randomisation is the general remedy: randomise a unit large enough that interference is mostly inside it.

Marketplace  -> randomise by CITY or region
Social       -> randomise by graph cluster (community detection,
                then assign whole clusters)
Ads/budget   -> budget-split: give each arm its own budget and
                inventory pool
Shared model -> train separate models per arm, or accept the bias
                and state it

Cost: variance. With 40 cities instead of 4 million users, your
effective sample size is 40, so the MDE is far larger. A cluster
design usually needs a much bigger effect or a much longer run.

That variance cost is the real reason teams avoid cluster designs, and stating it explicitly is better than pretending a user-level test was valid.

4. Switchback tests

When interference is total, so that no cluster is independent, randomise time instead.

The whole system runs treatment or control, alternating:

  10:00-10:30  control
  10:30-11:00  treatment
  11:00-11:30  treatment
  11:30-12:00  control
  ...

Randomisation unit = (region, time bucket).
Every unit in the system experiences the same condition
simultaneously, so within-period interference is not a
confound: it is part of what you are measuring.

Three design decisions, and each is a trade:

PERIOD LENGTH
  too short  -> CARRYOVER: the previous period's effects persist
                into this one (a dispatch decision made at 10:29
                affects a delivery at 10:45)
  too long   -> fewer periods, so fewer effective samples and
                more time-of-day confounding within a period
  Typical: 30 minutes to a few hours, chosen so the system's
  relaxation time is short relative to the period.

BURN-IN
  Discard the first minutes of each period so carryover from the
  previous condition is excluded from the analysis. Costs sample.

ANALYSIS
  Observations within a period are correlated, so you cannot treat
  each order as an independent sample. Use the PERIOD as the unit
  of analysis, or cluster-robust standard errors clustered on
  (region, period). Treating individual events as independent
  understates the standard error dramatically and manufactures
  significance.

The effective sample size is the number of periods, not the number of events, and that is the single most misapplied part of switchback analysis.

Two weeks, 4 regions, 30-minute periods:
  14 days * 48 periods * 4 regions = 2,688 period-region units
  ...but time-of-day and day-of-week effects mean these are far
  from exchangeable, which is why switchback analyses use
  time-of-day fixed effects or paired designs.

Compare to a user-level test over the same two weeks with 4
million users. The MDE is an order of magnitude worse. Switchback
buys unbiasedness and pays in precision.

5. Sample ratio mismatch, the check that runs first

If the observed split differs from the intended split by more than chance, the experiment is invalid and no metric from it should be read.

from scipy.stats import chisquare

def srm_check(control_n: int, treatment_n: int, expected_ratio=0.5) -> float:
    """Returns a p-value. Below ~0.0005, treat the experiment as invalid.

    An SRM means assignment or logging is broken, which means the
    two arms are not comparable populations, which means every
    metric difference may be selection rather than treatment.
    """
    total = control_n + treatment_n
    expected = [total * (1 - expected_ratio), total * expected_ratio]
    return chisquare([control_n, treatment_n], expected).pvalue

Example: intended 50/50, observed 501,234 control / 498,102 treatment
  p = 0.014.  With 1M users, a 0.3% imbalance is NOT chance.

Common causes:
  - the treatment's extra JS fails on an old browser, so those
    users never fire the exposure event: treatment loses its
    slowest, oldest-device users, and every performance metric
    improves for free
  - a redirect-based implementation loses users on the redirect
  - bot filtering applied after assignment, asymmetrically
  - the assignment hash is not uniform for some id format
  - a crash in treatment removes its worst-affected users from
    the data entirely (survivorship)

Notice the direction: the mechanism that causes the SRM usually
also biases the metric in the treatment's favour.

Run the SRM check automatically on every experiment and refuse to display results when it fires. Microsoft's published guidance treats it as a trustworthiness gate, not a warning.

6. Multiple comparisons

20 metrics at alpha = 0.05, no true effects:
  P(at least one "significant") = 1 - 0.95^20 = 64%

So a dashboard of 20 metrics will show a winner about two times in
three, purely by chance.

Controls:
  - ONE pre-registered primary metric. The decision is made on it.
  - Guardrails (latency, error rate, crash rate) are checked with a
    different rule: they are one-sided harm checks, not
    win conditions.
  - Secondary/exploratory metrics get Benjamini-Hochberg FDR
    control, and are labelled exploratory in the write-up.

A worked example: a marketplace ranking test that shipped to zero

A local-services marketplace. A ranking change intended to surface newer providers.

The first analysis, which was wrong four ways:

Design:   randomise by consumer, 50/50, national.
Duration: 6 days ("we hit significance on day 3, ran three more
          to be safe").
Primary:  bookings per consumer.
Result:   +12.4%, p < 0.001. Also +9% on provider profile views,
          +6% on messages, +14% on new-provider bookings.
Decision: ship.

Post-rollout, total bookings moved +0.4 percent, inside the historical week-over-week noise band.

The review found all four pitfalls, and the largest one was structural:

1. INTERFERENCE (the big one).
   Provider capacity is finite: a plumber can take 3 jobs a day.
   Treatment consumers booking newly-surfaced providers meant
   control consumers found those providers busy. Bookings moved
   BETWEEN arms.
   The measured +12.4% was mostly reallocation.

2. PEEKING.
   Significance was reached on day 3 of a planned 14-day test and
   the test was stopped at 6. The pre-computed sample size for the
   stated 3% MDE was 11 days.

3. NOVELTY.
   Re-analysed by days-since-exposure, the effect was +19% on day
   1 and +5% by day 6, still declining. The 6-day average was
   dominated by early days.

4. MULTIPLE COMPARISONS.
   The write-up listed 24 metrics and highlighted 6 significant
   ones. Under the null, 24 metrics give a 1 - 0.95^24 = 71%
   chance of at least one false positive, and no primary metric
   had been pre-registered.

The redesign, and what each choice cost:

DESIGN: cluster-randomised by metro area.
  38 metros, randomised 19/19, stratified by size so the arms are
  balanced on baseline bookings.

  Cost, stated up front: the effective sample size is 38, not 4.1
  million. Power analysis on historical metro-level variance gave
  an MDE of 4.1% over 4 weeks, against 0.8% for the (invalid)
  user-level design.
  The team accepted measuring less precisely rather than
  precisely measuring the wrong thing.

DURATION: 4 weeks, fixed horizon, pre-registered.
  Primary metric: TOTAL BOOKINGS PER METRO. Deliberately a
  system-level metric, because that is the quantity interference
  was corrupting.

MONITORING: an always-valid sequential test for the guardrails
  only (crash rate, p99 latency, provider complaint rate), so the
  team could stop early for HARM without inflating the primary
  test's alpha. Stopping for harm and stopping for success are
  different decisions and only one of them needs to be
  conservative.

ANALYSIS: tenure-aligned effect curve reported alongside the
  headline, so novelty decay is visible rather than averaged away.

CHECKS: SRM on metro-level exposure counts; Benjamini-Hochberg on
  the 23 secondary metrics, all labelled exploratory.

Result of the redesigned test:

Total bookings per metro:   +1.2%, 95% CI [-0.9%, +3.3%]
                            not significant at the 4.1% MDE
New-provider bookings:      +21%, significant
Established-provider
  bookings:                 -6%, significant

Reading: the change reallocated demand toward new providers
without growing the market. That is a legitimate outcome and
possibly a desirable one (supply-side retention), but it is a
DIFFERENT decision from "this grows bookings by 12%".

The change shipped, with a different justification and a
different success metric: new-provider 90-day retention, which
rose 8 points over the following two quarters.

The redesign did not find a bigger effect, it found the right question, and the organisational outcome was that the feature was still worth shipping for a reason nobody had articulated before.

Then a switchback test, for a different change on the same platform.

Change: a dispatch algorithm deciding which providers see a job
        first. Interference is TOTAL: any provider allocation
        affects every other job, and metro-level clustering is
        not enough because a metro's providers are shared across
        all its jobs.

Design: switchback, 30-minute periods, randomised per
        (metro, period), 3 weeks.
        Burn-in: first 5 minutes of each period discarded, chosen
        from the measured median job-completion time (~4 min) so
        carryover from the previous condition is excluded.

Analysis: period-region as the unit, with time-of-day and
          day-of-week fixed effects, cluster-robust standard
          errors clustered on (metro, period).

A first-pass analysis that treated each JOB as independent gave
p < 10^-9 on a +3% effect. Re-analysed at the period level, the
same data gave p = 0.11. The individual-job analysis had
understated the standard error by roughly an order of magnitude,
because jobs within a period share a condition and a demand
environment.

Final: +2.9% jobs completed, 95% CI [+0.4%, +5.4%], shipped.

"p < 10^-9 became p = 0.11 on the same data" is the switchback lesson in one line: the effective sample size is the number of periods, not the number of events.

Production evidence

Optimizely's Stats Engine (2015) moved their platform to always-valid inference specifically because users peeked at fixed-horizon tests regardless of guidance, and their published explanation of the false-positive inflation from continuous monitoring is the clearest industry statement of the problem.

Microsoft's "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments" (Fabijan et al., KDD 2019) documents SRM as a trustworthiness gate, catalogues its causes, and reports how often it occurs in a mature experimentation platform.

Kohavi, Tang and Xu, "Trustworthy Online Controlled Experiments" is the standard reference and covers peeking, novelty and primacy, SRM, and multiple comparisons, with the Microsoft experience behind each.

DoorDash and Lyft have both published on switchback experimentation for marketplace and dispatch changes, including period-length selection and the requirement to cluster standard errors at the period level, which is the specific analysis error described above.

LinkedIn's and Facebook's published work on network A/B testing describes graph-cluster randomisation for social interference, and both note the substantial variance cost relative to user-level randomisation.

Netflix's writing on long-term holdbacks describes maintaining a never-exposed population to measure the persistence of effects after rollout, which is the documented remedy for a novelty effect you only discover after shipping.

The debate

Should you use sequential tests so you can peek? If your organisation will peek, yes, because a valid procedure that people follow beats an invalid use of a stricter one. The cost is real: at the planned sample size a sequential test is less powerful than a fixed-horizon test, so you detect smaller effects more slowly. The position: always-valid inference for guardrails and for teams that monitor continuously; fixed horizon with pre-registration when the discipline exists.

Is cluster randomisation always right for marketplaces? When interference is material, yes, and the honest cost is a much larger MDE. The counter-argument, that a cluster design cannot detect the small effects the business cares about, is often true, and the correct response is to say so rather than to run a user-level test whose bias is in the direction you want. Measuring the wrong quantity precisely is worse than measuring the right one imprecisely.

Are switchbacks better than cluster randomisation? They handle total interference, which clusters cannot, and they cost precision and require careful period design. Use clusters when interference is mostly local (geography, social groups) and switchbacks when it is global (dispatch, pricing, matching, a shared model).

How long should a test run? At least two full weekly cycles, and long enough that the tenure-aligned effect curve has flattened. The counter-argument is opportunity cost, which is real, and the resolution is a long-running holdback: ship on a two-week read, and keep a small never-exposed population so the decayed steady-state effect is measurable later.

Should you look at many metrics? Look at all of them, decide on one. A pre-registered primary metric plus one-sided guardrail checks plus FDR-controlled exploratory metrics is the arrangement that survives scrutiny, and the alternative, a dashboard of 20 metrics with the significant ones highlighted, produces a false winner about two times in three.

Is an SRM ever ignorable? No. The mechanisms that cause an SRM, treatment users failing to log exposure, redirects dropping users, crashes removing the worst-affected, almost always bias the metric in the treatment's favour. The imbalance is evidence that the arms are different populations, which is a different and worse thing than noise. Refuse to read results.

Follow-up Q&A

"Why is peeking a problem, and what do you do about it?"

A fixed-horizon p-value assumes one look at the planned sample size. Each additional look is another chance for the random walk to cross the threshold, so the false-positive rate inflates: roughly 8 percent at two looks, 14 at five, and under continuous monitoring of an unbounded test the probability of eventually crossing any fixed threshold approaches 1 even with no effect. The remedies are a fixed horizon with a pre-registered sample size, a group-sequential design with alpha spending across pre-specified interim looks, or always-valid inference where the p-value is valid at every point in time. The last one costs power, which is the price of the right to peek.

"How do you tell a novelty effect from a real one?"

Plot the treatment effect against days since that user's first exposure, not against calendar date. A calendar plot mixes users on day one with users on day ten and averages the decay away. If the tenure-aligned curve trends toward zero, it is novelty; if it trends upward from a dip, it is primacy. Also analyse new users separately, since they have no prior expectations and their curve is the closest early proxy for the steady state. And keep a small never-exposed holdback so you can measure the effect months after shipping, which is the only way to catch decay you did not notice in time.

"What is interference and when does it invalidate a test?"

It is a violation of the assumption that a unit's outcome depends only on its own assignment. It happens with finite shared supply, where treatment consumes what control would have consumed; with social networks, where treatment users' behaviour reaches control friends; with shared infrastructure, where treatment's resource use degrades control; and with shared learning systems trained on both arms. The direction differs: shared supply overstates the effect, network spillover understates it. The remedy is to randomise a unit large enough to contain the interference, a metro, a graph cluster, a budget pool, and to accept the variance cost that comes with far fewer effective samples.

"When would you use a switchback test?"

When interference is global rather than local, so no cluster is independent: dispatch, matching, pricing, or anything mediated by a shared pool. The whole system alternates between treatment and control over randomised time periods, so within-period interference is part of what you measure rather than a confound. The design decisions are period length, long enough that carryover from the previous condition decays and short enough to get many periods, a burn-in that discards the first few minutes of each period, and an analysis clustered at the period level.

"What is the most common analysis mistake in a switchback?"

Treating each individual event as an independent observation. Events within a period share a condition and a demand environment, so they are strongly correlated, and analysing them as independent understates the standard error by roughly an order of magnitude. In one case a +3 percent effect computed at the job level gave p below 10 to the minus 9, and the same data analysed at the period level with cluster-robust standard errors gave p equals 0.11. The effective sample size is the number of periods, not the number of events.

"What check should run before you look at any experiment result?"

Sample ratio mismatch. Chi-square the observed arm sizes against the intended split, and if the p-value is very small, treat the experiment as invalid and display nothing. A 0.3 percent imbalance across a million users is not chance. The causes, treatment JavaScript failing on old devices so those users never log exposure, redirects dropping users, crashes removing the worst-affected, asymmetric bot filtering, all remove a non-random slice of the treatment population, and they almost always bias the metrics in the treatment's favour, which is why an SRM is a stop rather than a caveat.

Common misconceptions

"Stop when it reaches significance." That is not a stopping rule for a fixed-horizon test. Under continuous monitoring the false-positive probability climbs toward certainty with no true effect.

"Run it a week, that is long enough." One week cannot separate a novelty effect from a real one, and a single weekly cycle cannot separate weekday composition from treatment.

"Randomising by user is always safest." With shared supply or a social graph, user-level randomisation gives a precise measurement of the wrong quantity, and the bias usually points the way the team hopes.

"More metrics means more evidence." Twenty metrics at alpha 0.05 give a 64 percent chance of at least one false positive with no true effects.

"A switchback with a million events has a million samples." It has as many samples as it has periods, and treating events as independent manufactures significance.

"A small sample-ratio imbalance is just noise." At scale it is not, and the mechanisms that produce it also bias the outcome.

Interview delivery note

Say this verbatim: "Randomising by user in a marketplace measures the wrong thing precisely. Treatment buyers consume finite supply that control buyers would have consumed, so the between-arm difference is partly reallocation. We measured plus twelve percent and shipped to plus point four percent total, and the fix was cluster randomisation by metro with a system-level primary metric, which cost us an MDE of four percent instead of point eight." It names the pitfall, the failure and the price of the fix.

The senior-versus-staff separator is accepting a worse MDE deliberately and saying so. A senior engineer knows interference exists. A staff engineer states the trade out loud: the cluster design has 38 effective samples instead of 4.1 million, so we can only detect a 4.1 percent effect over four weeks, and we are choosing to measure the right quantity imprecisely rather than the wrong one precisely. Pricing your own proposal's downside is what makes it credible.

The second signal is separating stopping for harm from stopping for success. Using an always-valid sequential test on guardrails so the team can abort early for damage, while keeping the primary metric on a pre-registered fixed horizon, shows you understand that the two decisions have different asymmetries and only one of them needs to be conservative.

The Staff & Team Lead Interview Reference