Interleaving and position bias

What it is

Position bias is the observation that users click higher-ranked results more often regardless of relevance. A document at rank 1 gets clicked far more than the same document at rank 5. Click-through rate therefore measures position plus relevance, and any ranking model trained naively on clicks learns to reproduce the ranker that generated the logs.

Interleaving is an online evaluation method that removes position bias by construction. Instead of showing ranker A to one group of users and ranker B to another, it merges both rankings into a single result list shown to every user, attributes each click to whichever ranker contributed that document, and compares the totals. Every user sees both rankers, at comparable positions, on the same query.

The confusion worth clearing: interleaving is not an A/B test with a fancier split. An A/B test compares two populations; interleaving compares two rankers within each impression. That difference is why it needs far less traffic, and also why it cannot measure anything except relative ranker preference.

The problem it solves

Two problems, actually.

Sensitivity. Ranking changes produce small effects on session-level metrics. A genuinely better ranker might move click-through rate by half a percent, and detecting half a percent against the variance of user behaviour needs an enormous sample. Teams end up running one ranking experiment per month, which caps how fast relevance can improve.

Bias. If you evaluate a new ranker on historical click logs, you are scoring it against data generated by the old ranker. Documents the old ranker never showed have no clicks, so they look irrelevant. The new ranker is penalised for disagreeing, and the system converges on the incumbent. This is the feedback loop that makes offline evaluation of rankers untrustworthy.

Mechanics

Team-draft interleaving

The robust algorithm, and the one to describe. It works like picking teams in a playground: the two rankers alternate picks, with a coin flip deciding who picks first at each round, and each document is credited to the ranker that picked it.

import random

def team_draft_interleave(ranking_a, ranking_b, k=10):
    """Merge two rankings, recording which ranker contributed each document.

    Randomising who picks first at every round is what removes position bias:
    over many impressions, each ranker's picks land at each position equally
    often, so position contributes equally to both and cancels in the totals.
    """
    result, team_a, team_b = [], [], []
    ia = ib = 0

    while len(result) < k and (ia < len(ranking_a) or ib < len(ranking_b)):
        a_picks_first = (len(team_a) < len(team_b)) or \
                        (len(team_a) == len(team_b) and random.random() < 0.5)

        if a_picks_first:
            while ia < len(ranking_a) and ranking_a[ia] in result:
                ia += 1                      # skip documents already placed
            if ia < len(ranking_a):
                result.append(ranking_a[ia]); team_a.append(ranking_a[ia]); ia += 1
        else:
            while ib < len(ranking_b) and ranking_b[ib] in result:
                ib += 1
            if ib < len(ranking_b):
                result.append(ranking_b[ib]); team_b.append(ranking_b[ib]); ib += 1

    return result, set(team_a), set(team_b)


def score_impression(clicked_docs, team_a, team_b):
    """One impression contributes +1, -1 or 0. Ties (equal clicks) are
    discarded, which is deliberate: they carry no preference information."""
    a = len(clicked_docs & team_a)
    b = len(clicked_docs & team_b)
    return (a > b) - (a < b)

Aggregate the per-impression outcomes and test whether the win rate differs from 0.5. A binomial test on impressions where the two rankers disagreed is the standard analysis, and the effect size is the preference for A over B.

Balanced interleaving, the earlier algorithm, merged by taking from whichever ranker had contributed fewer documents so far. Radlinski, Kurup and Joachims showed it has a systematic bias: for certain pairs of rankings it credits one ranker more often even when the two are identical in quality. Team draft was the fix. Knowing that balanced interleaving is broken and why is a clean depth signal, because it shows the naive merge is not good enough.

Why it needs so much less traffic

An A/B test measures a between-subjects difference: user population A saw ranker A, population B saw ranker B, and the comparison must clear the variance of user behaviour across two different sets of people.

Interleaving measures a within-impression difference: on this query, for this user, which ranker's documents got clicked. The user is their own control, which removes the largest source of variance in the comparison.

The empirical result reported across multiple published studies is one to two orders of magnitude fewer impressions to reach the same statistical power. That converts a two-week ranking experiment into a two-hour one, and it is the entire business case.

Position bias, and correcting for it in training data

Interleaving fixes online evaluation. It does not fix the training data. For that you need to estimate the propensity: the probability that a document at rank $r$ is examined at all.

Inverse propensity scoring (IPS) reweights each click by the inverse of that probability. A click at rank 10 is rarer, so it counts for more:

$$\hat{\Delta}(\pi) = \frac{1}{|D|}\sum_{d \in D} \frac{c_d \cdot \mathbb{1}[\text{rank}\pi(d) \le k]}{p{r_d}}$$

where $c_d$ is the click and $p_{r_d}$ is the examination propensity at the rank where the document was originally shown. Joachims, Swaminathan and Schnabel proved this estimator is unbiased for the true relevance-based metric given correct propensities.

Getting the propensities is the hard part, and there are two honest ways:

Result randomisation (RandPair). For a small fraction of traffic, swap the document at rank 1 with the document at a randomly chosen rank $r$. Because the swap is random, any difference in click rate between the two positions is position, not relevance. That gives you $p_r / p_1$ directly. It costs a little relevance on the randomised traffic, which is the price of an unbiased estimate.

Intervention harvesting. Exploit the randomisation you already have. If you run multiple rankers (an A/B test, a canary, a multi-armed bandit), the same document appears at different ranks across variants for the same query, and you can estimate propensities from that natural variation without deliberately degrading anything. Cheaper, and it needs enough ranker diversity in the logs to work.

A typical propensity curve looks roughly like this and is worth having in your head as a shape:

Rank	Relative examination probability
1	1.00
2	0.65
3	0.50
5	0.35
10	0.20

The exact numbers are surface-specific and you must estimate your own. The shape (steep drop across the first three, long flat tail) is general.

The other biases in the same family

Naming these unprompted signals that you have thought about click data as a measurement instrument rather than as ground truth:

Selection bias. Documents the ranker never showed have no clicks. IPS corrects for examination given exposure; it cannot correct for never having been exposed.
Trust bias. Users click top results more than examination alone explains, because ranking position is itself a quality signal to them.
Presentation bias. A result with a rich snippet, an image or a badge gets clicked more at the same position.
Novelty and recency effects. A visibly changed interface gets exploratory clicks for a week or two, which contaminates short experiments.

A worked example

A product search team ships a new learning-to-rank model. Offline NDCG@10 on their judgment set improves from 0.412 to 0.438, a 6 percent relative gain. Should they ship it?

The A/B route. Their surface does 400,000 searches a day. The target metric is search-result click-through rate, currently 34 percent. To detect a 1 percent relative change (34 percent to 34.34 percent) at 95 percent confidence and 80 percent power:

$$n \approx \frac{(1.96+0.84)^2\left[p_1(1-p_1)+p_2(1-p_2)\right]}{(p_1-p_2)^2} = \frac{7.84 \times 0.4489}{(0.0034)^2} \approx 304{,}000 \text{ per arm}$$

At 200,000 searches per arm per day, that is about 1.5 days minimum, and in practice a week to cover the weekly cycle and avoid peeking. One experiment per week per surface, and the team has a queue of eleven ranking changes.

The interleaving route. Team-draft interleave the two rankers on the same traffic. With roughly two orders of magnitude better sensitivity, the same comparison resolves in hours rather than days. The readout is a preference: for example, ranker B preferred in 53 percent of impressions where the two disagreed, which against a null of 50 percent is a clear win at this sample size.

And then still run the A/B test. This is the part people get wrong. Interleaving told you B produces more clicks on its documents than A does. It did not tell you whether the change moves revenue, whether it increased latency, whether it hurt the long tail of rare queries, or whether the clicks converted. The workflow that actually works:

Offline on a judgment set: cheap, fast, and directionally useful. Kills the obviously bad ideas.
Interleaving: fast, sensitive, unbiased on ranker preference. Ranks the surviving candidates and kills the ones that are not actually better.
A/B test on the winner: slow, but it measures the business metrics and the guardrails, and it is what the ship decision cites.

The team runs eleven interleaving experiments in the time one A/B test would have taken, then A/B tests the two that won.

Production evidence

Netflix published "Innovating Faster on Personalization Algorithms at Netflix Using Interleaving" (Netflix Technology Blog, 2017), reporting that interleaving required dramatically fewer subscribers to detect a difference between rankers than a conventional A/B test, and describing the two-stage pipeline (interleaving to select candidates, then A/B to measure member-level impact) exactly as above. It is the most citable industrial account.

Radlinski, Kurup and Joachims, "How Does Clickthrough Data Reflect Retrieval Quality?" (CIKM 2008) introduced team-draft interleaving and demonstrated the bias in balanced interleaving.

Chapelle et al., "Large-Scale Validation and Analysis of Interleaved Search Evaluation" (TOIS 2012) validated interleaving against A/B outcomes at scale and is the standard citation for the sensitivity claim.

Joachims, Swaminathan and Schnabel, "Unbiased Learning-to-Rank with Biased Feedback" (WSDM 2017) established the IPS estimator for ranking and the randomisation-based propensity estimation that goes with it. Agarwal et al. followed with intervention harvesting, which estimates propensities from existing ranker diversity instead of deliberate randomisation.

Airbnb, Etsy and Yandex have all published on position bias correction in their ranking pipelines, which is good evidence that this is standard practice in marketplaces rather than a search-engine speciality.

The debate

The alternative is A/B testing only. It measures what you actually care about (revenue, retention, session success) rather than a proxy, it needs no special merging infrastructure, and it cannot produce the confusing situation where interleaving and A/B disagree.

The case against relying on it alone is throughput. If a ranking change takes a week to evaluate, you get 50 experiments a year across all surfaces, which is not enough to make meaningful relevance progress. Interleaving is what makes ranking iteration fast.

The honest limitations of interleaving, which you should volunteer:

It measures relative ranker preference only. It cannot tell you the absolute quality of either ranker, and it cannot measure revenue, retention, or anything session-level.
It requires comparable result lists. If ranker B returns a different result type (a card instead of a link, a generated answer instead of ten blue links), merging is meaningless and you are back to A/B.
Users see a merged list nobody designed. For most surfaces this is fine; for a curated experience it can degrade the product during the experiment.
It is weak for diversity and whole-page changes, because the merge destroys the page composition each ranker intended.

My position: interleaving for ranker selection, A/B for the ship decision, and IPS on the training data regardless. They answer different questions and the mistake is treating them as competitors. If a team can only build one thing, build the A/B platform, because you cannot ship on interleaving alone. If they already have A/B, interleaving is the highest-return next investment for a search or recommendations team.

Interleaving is the wrong tool for testing a new surface, a UI change, a different result type, or anything where the two variants are not both ordered lists of comparable items.

Follow-up Q&A

"Why is interleaving more sensitive than A/B?" Because the comparison is within-impression rather than between-population. In an A/B test the two arms are different users, so the difference between rankers must clear the variance of user behaviour across two populations. In interleaving every user sees both rankers on the same query, so the user is their own control and that variance disappears from the comparison. Published validations report one to two orders of magnitude fewer impressions for the same power.

"How do you correct position bias in training data?" Inverse propensity scoring: weight each click by the inverse of the probability that its position was examined, which makes the estimator unbiased with respect to relevance rather than position. The hard part is estimating propensities, and there are two honest methods. Deliberate randomisation (swap rank 1 with a random rank on a small traffic slice, and read the position effect off the difference), or intervention harvesting (estimate from the natural rank variation you already have across concurrent rankers). The first costs a little relevance; the second needs enough ranker diversity in your logs.

"Your interleaving result and your A/B result disagree. Which do you believe?" Neither, until you understand why. They measure different things: interleaving measures which ranker's documents users prefer, A/B measures what happened to the business. A ranker can win on interleaving and lose on A/B if it surfaces clickable but unsatisfying results (clickbait), if it increased latency, or if the effect on clicks does not translate to conversion. That disagreement is diagnostic information, not noise, and the usual culprit is that the clicked results were not the useful ones. Check downstream metrics per ranker: dwell time, conversion, reformulation rate.

"Why is balanced interleaving not good enough?" Because it has a systematic bias for certain ranking pairs. It merges by taking from whichever ranker has contributed fewer documents so far, and Radlinski et al. showed that for some pairs this credits one ranker disproportionately even when the two rankers are equally good. Team draft fixes it by randomising who picks first at each round, so over many impressions each ranker's picks are distributed identically across positions.

"What can offline NDCG not tell you that these can?" Whether your judgments match your users. Offline evaluation scores against relevance labels, which are somebody's opinion, collected at a point in time, on a query sample that is almost certainly head-heavy. It cannot capture personalisation, freshness, intent that shifts with the news, or the long tail. The standard failure is an offline gain that does not replicate online, and the standard practice is to track offline-online correlation as a metric in its own right: if your offline harness stops predicting online outcomes, the harness needs fixing before the ranker does.

Common misconceptions

The most common is that position bias means "users click the top result more". That is the observation. The bias is that click-through rate confounds position with relevance, so any model trained on raw clicks learns to reproduce the incumbent ranker rather than to improve on it.

The second is that interleaving replaces A/B testing. It replaces A/B testing for ranker selection. The ship decision still needs business metrics and guardrails, and interleaving measures neither.

The third is that IPS needs a model of user behaviour. It needs propensities, and the reliable ways to get them are randomisation or harvesting existing interventions, not assuming a click model.

Interview delivery note

Say this: "Click-through rate confounds position with relevance, so a ranker trained on raw clicks learns to reproduce whatever ranker produced the logs. I fix it in two places. Online, team-draft interleaving: merge both rankings into one list with a coin flip deciding who picks first each round, attribute clicks to the contributing ranker, and compare. Because every user sees both rankers on the same query, it's one to two orders of magnitude more sensitive than an A/B test. Offline, inverse propensity scoring on the training data, with propensities estimated from deliberate rank randomisation on a small traffic slice or harvested from the ranker diversity I already have."

Then land the practical framing: "Interleaving picks the ranker; A/B decides whether to ship it, because interleaving can't see revenue or latency." The depth signal is knowing that balanced interleaving is biased and team draft was the fix, and being able to name where propensities come from rather than waving at "we correct for position".

The Staff & Team Lead Interview Reference