NDCG, MRR, recall@k, and the offline-online gap

What they are

Three metric families that answer different questions, and using the wrong one is how teams optimise the wrong thing for a quarter.

RECALL@k        Of all relevant documents, what fraction did we
                RETRIEVE in the top k?
                A property of retrieval. Order-insensitive.

MRR             1 / (rank of the FIRST relevant result), averaged
                over queries.
                A property of ranking, for known-item search where
                there is one right answer.

NDCG@k          Discounted cumulative gain, normalised by the ideal
                ordering. Rewards putting HIGHLY relevant documents
                HIGH.
                A property of ranking, for graded relevance.

Commonly confused: recall@k measures whether the answer was found, NDCG measures whether it was placed well, and they can move in opposite directions. A reranker can raise NDCG while retrieval recall falls, and reporting only NDCG hides that the system is losing documents it can never recover.

Also commonly confused: precision@k and recall@k. Precision is "of what we returned, how much was relevant"; recall is "of what was relevant, how much did we return". For a search system with a fixed result page, precision@10 and NDCG@10 measure similar things and recall measures something structurally different.

The problem they solve

Without a metric, ranking changes are argued rather than decided, and every change looks like an improvement to the person who made it. The specific failures they prevent:

A change that helps head queries and destroys tail queries. Aggregate click-through rate rises, the team ships, and long-tail search quietly stops working. Segment-level metrics catch it; a single number does not.

A reranker papering over a retrieval failure. If the answer-bearing document was never retrieved, no reranker can surface it. Recall@k measured separately from NDCG is the only way to see that, and it is the single most common measurement gap in RAG and search systems.

Optimising a proxy. Click-through rate is not relevance: it is relevance times attractiveness times position bias. Optimising raw CTR reliably produces clickbait, which is why position-debiased metrics exist.

Mechanics

NDCG, derived

Start from the intuition and build it:

1. GAIN: how good is this document?
   Graded relevance: 3 = perfect, 2 = good, 1 = marginal, 0 = bad.
   Binary works too but wastes information.

2. CUMULATIVE GAIN: sum the gains of the top k.
   CG@5 = 3 + 2 + 0 + 3 + 1 = 9
   Problem: order-insensitive. [3,2,0,3,1] and [0,1,2,3,3]
   score identically, and they are clearly not equally good.

3. DISCOUNTED CG: divide each gain by log2(rank + 1), because
   a user's attention decays with position.

$$ \text{DCG@k} = \sum_{i=1}^{k} \frac{2^{rel_i} - 1}{\log_2(i + 1)} $$

The 2^rel - 1 numerator is the standard "exponential gain"
variant and it is what most implementations use. It makes the
difference between "perfect" and "good" much larger than
between "good" and "marginal", which matches how users
actually experience results.

  rel=3 -> 7    rel=2 -> 3    rel=1 -> 1    rel=0 -> 0

4. NORMALISED: divide by the DCG of the ideal ordering, so
   scores are comparable across queries with different numbers
   of relevant documents.

$$ \text{NDCG@k} = \frac{\text{DCG@k}}{\text{IDCG@k}} $$

A worked example, which is what to be able to produce:

Query returns 5 results with graded relevance [2, 3, 0, 1, 2].

DCG@5:
  i=1: (2^2 - 1)/log2(2) = 3 / 1.000 = 3.000
  i=2: (2^3 - 1)/log2(3) = 7 / 1.585 = 4.416
  i=3: (2^0 - 1)/log2(4) = 0 / 2.000 = 0.000
  i=4: (2^1 - 1)/log2(5) = 1 / 2.322 = 0.431
  i=5: (2^2 - 1)/log2(6) = 3 / 2.585 = 1.161
  DCG@5 = 9.008

Ideal ordering of the same set: [3, 2, 2, 1, 0]
  i=1: 7 / 1.000 = 7.000
  i=2: 3 / 1.585 = 1.893
  i=3: 3 / 2.000 = 1.500
  i=4: 1 / 2.322 = 0.431
  i=5: 0 / 2.585 = 0.000
  IDCG@5 = 10.824

NDCG@5 = 9.008 / 10.824 = 0.832

The interpretation to state: 0.832 means we captured 83 percent of the achievable gain given the documents we retrieved. It says nothing about documents we failed to retrieve, which is exactly why recall must be measured alongside it.

Recall@k, and why it is the ceiling

recall@k = |relevant ∩ retrieved_top_k| / |relevant|

Query has 8 relevant documents in the corpus.
Retrieval returns 100 candidates, containing 5 of them.

recall@100 = 5/8 = 0.625

*** No reranker can exceed NDCG corresponding to those 5. ***
Three relevant documents are unreachable, permanently, for
this query.

Retrieval recall is the ceiling on end-to-end quality, and measuring it separately is the discipline. In a two-stage system the correct instrumentation is:

recall@k_retrieval    Did the candidate generator find them?
NDCG@10_final         Did the reranker order them well?

If NDCG is low and recall is high  -> ranking problem
If NDCG is low and recall is low   -> retrieval problem
If you only measure NDCG           -> you cannot tell, and you
                                      will spend a quarter tuning
                                      the wrong stage

MRR, and when it is the right metric

$$ \text{MRR} = \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{\text{rank}_q} $$

Query 1: first relevant result at position 1  -> 1.000
Query 2: first relevant result at position 3  -> 0.333
Query 3: first relevant result at position 2  -> 0.500
Query 4: no relevant result in top k          -> 0.000
MRR = 0.458

MRR is the right metric for known-item search, where the user is looking for one specific thing: a product code, a person, a document they have seen before, a factual answer. It is the wrong metric for exploratory search, where the user wants a good set, because it ignores everything after the first hit entirely.

Ranking A: [relevant, irrelevant, irrelevant, irrelevant]
Ranking B: [relevant, relevant, relevant, relevant]
MRR: identical (1.0 for both).
NDCG@4: A = 0.51, B = 1.00.

Choosing MRR when the task is exploratory is a real error, and it is common because MRR is easy to compute and easy to explain.

Where the labels come from

The metrics are arithmetic. The judgements are the hard part, and this is where the discussion should go.

HUMAN JUDGEMENT (explicit)
  Raters score (query, document) pairs against a rubric.
  + High quality, graded, no position bias.
  - Expensive: $0.50-3 per judgement. A 500-query set at 20
    documents each is 10,000 judgements.
  - Inter-rater agreement is typically 0.6-0.8 (Cohen's kappa)
    even with a good rubric. That disagreement is the noise
    floor of your offline metric and it must be measured.
  - Raters are not your users and judge topical relevance
    rather than usefulness.

CLICK DATA (implicit)
  + Free, abundant, and reflects real intent.
  - POSITION BIAS: position 1 gets far more clicks than
    position 5 regardless of relevance.
  - PRESENTATION BIAS: clicks reflect the snippet, not the
    document.
  - SELECTION BIAS: you only observe clicks on what you showed,
    so the log is a record of your own ranker's decisions.

COUNTERFACTUAL / DEBIASED CLICKS
  Estimate a position propensity and weight clicks by its
  inverse (IPS). Recovers an approximately unbiased estimate
  from logged data.
  + Cheap, at scale, from data you already have.
  - Requires propensity estimation, and high variance on the
    low-propensity tail.

The position-bias correction, concretely:

Observed CTR by position (typical shape, web search):
  pos 1: 0.31   pos 2: 0.15   pos 3: 0.10   pos 4: 0.07   pos 5: 0.05

If propensity p(pos 1) = 1.0 and p(pos 5) = 0.16, then a click
at position 5 is evidence worth 1/0.16 = 6.25 clicks at
position 1.

IPS-weighted relevance estimate:
  rel_hat(d) = sum over impressions of click(d) / p(position(d))

That is the basis for interleaving and position bias, and the practical point is that an uncorrected CTR comparison between two rankers is not a measurement of relevance, it is a measurement of which ranker put things higher.

The offline-online gap

The failure that costs the most, and the reason to expect it:

Offline NDCG@10:  0.71 -> 0.76   (+7%)
Online CTR:       4.2% -> 4.0%   (-5%)

Ship it? No. And the divergence has identifiable causes.

1. LABEL MISMATCH
   Raters judge topical relevance; users want usefulness.
   A perfectly on-topic 2015 document scores 3 from a rater
   and is useless to a user asking about current behaviour.
   Fix: add freshness and authority to the rubric.

2. EVALUATION SET BIAS
   The judged set was sampled from the OLD ranker's results,
   so a new ranker surfacing genuinely different documents has
   them scored as 0 (unjudged) by default.
   *** This is the most common cause and it is silent. ***
   Fix: pool judgements across both rankers (TREC pooling),
   and treat unjudged documents explicitly rather than as
   irrelevant.

3. SEGMENT EFFECTS
   Aggregate improves; one segment collapses. Head queries up
   4%, tail queries down 12%, and tail is 40% of volume.
   Fix: report per segment always.

4. THE METRIC IS NOT THE OBJECTIVE
   NDCG optimises ordering. The business objective might be
   conversions, task completion or long-term retention, and
   those can diverge from relevance.

5. PRESENTATION CHANGED
   The new ranker surfaces documents with worse titles or
   snippets. Relevance improved; clickability did not.

Cause 2 deserves emphasis because it is silent and it invalidates the comparison entirely. If your judged pool came from ranker A's top 20, then ranker B's genuinely excellent new document at position 1 is unjudged, scored as irrelevant by default, and ranker B measures worse than it is. Pooling judgements from both systems before scoring is the standard fix and it comes from TREC methodology.

The correlation to actually measure: run several ranking changes, record both offline delta and online delta, and compute the correlation across them. If it is weak, the offline metric is not a decision tool and using it as a gate is worse than not having one. A team that has never measured that correlation does not know whether its offline evaluation means anything.

Production evidence

Järvelin and Kekäläinen, "Cumulated gain-based evaluation of IR techniques" (TOIS 2002) is the original NDCG paper, including the argument for a logarithmic discount and for normalisation to make queries comparable.

Joachims et al., "Accurately Interpreting Clickthrough Data as Implicit Feedback" (SIGIR 2005) established position bias empirically with eye-tracking, and its finding that clicks are relative rather than absolute judgements is the foundation of interleaving.

Wang et al., "Learning to Rank with Selection Bias in Personal Search" (SIGIR 2016) and Joachims, Swaminathan and Schnabel, "Unbiased Learning-to-Rank with Biased Feedback" (WSDM 2017) are the primary sources for inverse propensity weighting in ranking.

The TREC evaluation methodology established pooling: judgements are collected over the union of results from all participating systems, precisely to avoid the bias in cause 2. It is the standard fix and it predates the problem's rediscovery in industry by decades.

Bing's and Yandex's published experimentation work documents the offline-online divergence directly and is the basis for the industry practice of treating offline metrics as a filter and online experiments as the decision.

Microsoft's MS MARCO and the BEIR benchmark are the modern public evaluation sets, and BEIR is notable for measuring out-of-domain generalisation, which is where dense retrievers historically underperformed BM25 and which motivated hybrid retrieval.

The debate

The case for offline metrics as the primary gate: they are fast, cheap, reproducible and available before you expose anything to users. You can iterate twenty times a day against NDCG and once a week against an A/B test.

The case for online-only evaluation: offline labels are a proxy for user satisfaction and the proxy drifts. Only real users on real traffic tell you whether the change helped, and every offline metric has failed to predict an online result at some point.

The case for interleaving over A/B: interleaving needs far less traffic to reach significance because it controls within-user, and it directly measures preference between two rankers rather than a downstream aggregate.

My position: offline metrics filter, interleaving decides for ranking changes, and A/B tests measure business impact, in that order. Each is doing a different job and skipping any of them costs you something specific.

Offline is a filter because it is the only thing fast enough to support iteration, and because it catches the changes that are obviously worse before they reach users. Interleaving is the decision for ranking specifically, because it needs roughly an order of magnitude less traffic than an A/B test and controls for position bias by construction. A/B measures the business outcome, which is what actually matters and which interleaving cannot tell you.

The measurement I would insist on before trusting any of it: measure the correlation between offline and online deltas across a set of past changes. A team using offline NDCG as a gate without knowing that correlation is using a number of unknown validity, and the honest outcome of that exercise is sometimes "our offline metric predicts nothing", which is worth knowing.

And I would always report recall@k separately from NDCG, because they answer different questions and their divergence is the single most useful diagnostic in a two-stage system. A quarter spent tuning a reranker when the problem was retrieval is a real and common cost, and one extra metric prevents it.

On labels: pool judgements across both rankers before scoring. The default of treating unjudged documents as irrelevant systematically penalises any ranker that finds genuinely new documents, which is exactly the ranker you were hoping to build. It is silent, it invalidates the comparison, and the fix has been standard TREC practice for thirty years.

Follow-up Q&A

"Walk me through NDCG." Start from cumulative gain, which is just the sum of relevance grades in the top k, and note it is order-insensitive so it cannot distinguish a good ranking from a bad one with the same documents. Add a logarithmic position discount, because attention decays with rank, which gives DCG. Then normalise by the DCG of the ideal ordering so scores are comparable across queries with different numbers of relevant documents. The standard numerator is $2^{rel} - 1$, which makes the gap between "perfect" and "good" much larger than between "good" and "marginal", matching how users experience it.

"Why measure recall@k separately from NDCG?" Because they answer different questions and can move in opposite directions. Recall is whether retrieval found the answer; NDCG is whether ranking placed it well. If the answer-bearing document was never in the candidate set, no reranker can surface it, so retrieval recall is a hard ceiling on end-to-end quality. Reporting only NDCG means that when quality is bad you cannot tell whether to fix retrieval or ranking, and teams routinely spend a quarter tuning the wrong stage.

"When is MRR the right metric?" Known-item search, where there is one right answer: a product code, a person, a specific document. It is wrong for exploratory search, because it ignores everything after the first relevant hit. A ranking of [relevant, irrelevant, irrelevant, irrelevant] and one of [relevant, relevant, relevant, relevant] have identical MRR and very different NDCG, and choosing MRR when the task is exploratory is a real error that happens because MRR is easy to compute and explain.

"Your offline metric improved and the online metric got worse. What happened?" Five candidate causes and I would check them in order. Evaluation set bias first, because it is the most common and it is silent: if the judged pool came from the old ranker's results, then the new ranker's genuinely new documents are unjudged and scored as irrelevant by default. Then segment effects, where the aggregate improved while a large segment collapsed. Then label mismatch, where raters judge topical relevance and users want usefulness, so a perfectly on-topic 2015 document scores well and helps nobody. Then whether the metric is actually the objective. Then presentation, since the new documents may have worse titles.

"How do you fix evaluation set bias?" Pooling, which is standard TREC methodology: collect judgements over the union of results from both systems before scoring either. It is thirty years old and it gets rediscovered constantly. The alternative, treating unjudged as irrelevant, systematically penalises any ranker that surfaces genuinely different documents, which is exactly the ranker you were trying to build.

"Can you just use click-through rate?" Not directly, because CTR is relevance times attractiveness times position bias, and the position term dominates. Position one gets several times the clicks of position five regardless of relevance, so comparing two rankers by raw CTR measures which one put things higher. The corrections are inverse propensity weighting, where a click at position five with propensity 0.16 counts as 6.25 clicks at position one, or interleaving, which controls for position by construction and is why it needs far less traffic than an A/B test.

"How much do you trust human judgements?" They are the best labels available and they have a measurable noise floor: inter-rater agreement is typically 0.6 to 0.8 even with a good rubric, and that disagreement bounds how small a difference your offline metric can detect. I would measure it rather than assume it, and I would be explicit that raters judge topical relevance rather than usefulness, so the rubric needs freshness and authority in it if those matter to users.

"How do you know your offline metric is worth anything?" Measure the correlation between offline and online deltas across a set of past changes. Run ten ranking changes, record both numbers, and compute the correlation. If it is weak, the offline metric is not a decision tool and gating on it is worse than not having one, because it produces false confidence. That is an uncomfortable exercise and most teams have never done it, which means they do not know whether their evaluation means anything.

Common misconceptions

"NDCG measures the system." It measures ranking quality over the documents that were retrieved. Recall measures whether the right documents were retrieved at all.

"Higher CTR means better relevance." CTR conflates relevance, attractiveness and position. An uncorrected CTR comparison measures which ranker put things higher.

"Unjudged documents are irrelevant." That default systematically penalises rankers that find new documents. Pool judgements across systems.

"MRR and NDCG measure the same thing." MRR ignores everything after the first relevant result. For exploratory search that is most of the signal.

"Offline metrics are the gate." They are a filter. Interleaving decides ranking changes and A/B tests measure business impact, and the correlation between offline and online is something to measure rather than assume.

Interview delivery note

Lead with the metric-choice framing, because it is the judgement being tested: "These answer different questions. Recall@k is whether retrieval found it, MRR is where the first relevant result landed, and NDCG is whether the whole ordering is good. And in a two-stage system I'd always report retrieval recall separately from final NDCG, because they can move in opposite directions and their divergence tells you which stage to fix."

Derive NDCG rather than reciting it: "Cumulative gain is just the sum of grades, which is order-insensitive, so add a logarithmic position discount because attention decays with rank. Then normalise by the ideal ordering so queries with different numbers of relevant documents are comparable. And the $2^{rel} - 1$ numerator makes 'perfect versus good' a much bigger gap than 'good versus marginal', which matches how users experience it."

Volunteer the offline-online gap with its most common cause, because that is the operator's answer: "and I'd expect offline and online to diverge sometimes. The cause that's most common and most silent is evaluation set bias: if the judged pool came from the old ranker's results, the new ranker's genuinely new documents are unjudged and scored as zero. So it measures worse than it is, precisely because it's finding things the old one didn't. Pooling judgements across both systems before scoring is the TREC fix and it's thirty years old."

The line that separates someone who has run this: "and before trusting any offline gate I'd measure the correlation between offline and online deltas across past changes. Most teams have never done that, which means they don't know whether their offline metric predicts anything. Sometimes the honest answer is that it doesn't."

The Staff & Team Lead Interview Reference