Hybrid retrieval and reciprocal rank fusion

What it is

Hybrid retrieval runs two or more retrievers over the same corpus and merges their result lists. In practice that is BM25 (lexical, term-matching, sparse) and a dense vector retriever (semantic, embedding-based), because they fail on different queries and their failures are close to uncorrelated.

Reciprocal rank fusion (RRF) is the standard merge. For a document $d$ appearing at rank $r_i(d)$ in each result list $i$:

$$\text{RRF}(d) = \sum_{i} \frac{1}{k + r_i(d)}$$

with $k = 60$ by convention. Note what is absent: the retrievers' scores. RRF uses only rank, which is exactly why it works.

Commonly confused with two neighbours. It is not reranking: fusion merges candidate lists, a reranker rescores a merged list with a more expensive model, and a good pipeline does both in that order. And it is not a weighted score blend, which is the obvious alternative and is worse for a specific reason.

The problem it solves

BM25 and dense retrieval fail on complementary query types.

BM25 fails when the query and document use different words for the same thing: "parental leave" against a document that says "family care absence policy". It has no notion of meaning, only of term overlap weighted by inverse document frequency.

Dense retrieval fails on exact tokens: product codes, error codes, function names, rare proper nouns, version numbers. An embedding model trained on general text maps ERR_CONN_REFUSED_4471 and ERR_CONN_REFUSED_4472 to nearly the same vector, because the difference is one character with no semantic weight. It also degrades on out-of-domain jargon it never saw in training.

Fusing them recovers both. The measured effect is consistent across published benchmarks: hybrid beats either arm alone on heterogeneous query mixes, and the margin is largest exactly where you would expect, on corpora with domain-specific vocabulary.

Mechanics

Why rank rather than score

The obvious merge is to normalise both scores and take a weighted sum. It breaks in three ways, and being able to name them is the answer to "why RRF".

The scales are incomparable. BM25 is unbounded above and depends on document length, term frequency and corpus statistics; a "good" BM25 score is 12 on one corpus and 40 on another. Cosine similarity is bounded in $[-1, 1]$ and typically compressed into a narrow band, often 0.7 to 0.9, for everything remotely relevant. There is no principled conversion.

Min-max normalisation is unstable. It normalises against the candidate set, so the same document gets a different normalised score depending on what else was retrieved. Two queries, one with a strong outlier and one without, produce incomparable normalised scores for identical documents.

Score distributions are query-dependent. A rare-term query produces high BM25 scores across the board; a common-term query produces low ones. So a fixed weight between the two arms is wrong for most queries.

Rank sidesteps all three. Rank 1 means the same thing on every query and every corpus: this retriever's best guess.

The role of $k$

$k = 60$ comes from Cormack, Clarke and Buettcher's 2009 paper, where it was tuned on TREC data and then adopted essentially unchanged by everyone.

It controls how sharply the fusion discriminates between top ranks:

Large $k$ flattens the curve, so a document must rank respectably in several lists to win, rather than winning by topping one. That is the behaviour you want from a fusion: it rewards agreement between retrievers over confidence within one. A small $k$ makes fusion behave like "take whichever retriever is most confident", which throws away the reason to fuse.

Tune $k$ only with an evaluation set. In practice 60 is close enough that the effort is better spent elsewhere.

Implementation

def rrf(result_lists, k=60, weights=None):
    """Fuse ranked lists. `result_lists` is a list of lists of doc ids,
    each already ordered best-first by its own retriever.

    Weights let you express that one retriever is more trustworthy on this
    corpus, which is the honest way to bias the fusion: it stays rank-based,
    so none of the score-scale problems come back.
    """
    weights = weights or [1.0] * len(result_lists)
    scores = {}
    for docs, w in zip(result_lists, weights):
        for rank, doc_id in enumerate(docs, start=1):   # 1-indexed: rank 1 is best
            scores[doc_id] = scores.get(doc_id, 0.0) + w / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)


# The retrieval arms run in parallel; the fusion is microseconds.
lexical = bm25_search(query, size=100)
dense   = vector_search(embed(query), k=100)
fused   = rrf([lexical, dense])[:50]        # 50 into the reranker

Two operational details. Retrieve more than you need from each arm (100 each to produce a top 50), because a document ranked 80th by one retriever and 3rd by the other should still surface. And run the arms concurrently: they are independent and the fusion cannot start until both finish, so latency is the max, not the sum.

In a search engine that supports it natively, this is configuration rather than code:

{
  "retriever": {
    "rrf": {
      "retrievers": [
        { "standard": { "query": { "match": { "text": "parental leave policy" } } } },
        { "knn": { "field": "embedding", "query_vector": [ ... ], "k": 100,
                   "num_candidates": 200 } }
      ],
      "rank_constant": 60,
      "rank_window_size": 100
    }
  }
}

rank_window_size is the per-arm depth and rank_constant is $k$. Getting the window too small is the most common misconfiguration: at 10, a document that one arm ranks 15th can never be recovered.

Where hybrid sits in the funnel

Query
  |-- BM25          -> top 100  --.
  |                                >-- RRF -> top 50 -- cross-encoder -> top 10
  '-- dense (ANN)   -> top 100  --'
       ~30 ms            ~40 ms      ~2 ms        ~80 ms

Fusion is a recall stage: its job is to get the right documents into the candidate set. Reranking is a precision stage: its job is to order them well. Conflating them is the mistake behind "we added a reranker and recall didn't improve", which is correct behaviour, because a reranker cannot conjure a document retrieval never returned.

A worked example

An enterprise policy search over 2 million documents. Evaluation set: 180 questions with a labelled correct source document.

The numbers above are the shape of a typical result rather than a published benchmark, and I would present them that way in an interview. The two structural facts they illustrate are real and reproducible: fusion beats either arm by a wide margin, and reranking does not change recall, only ordering within the retrieved set.

The failure analysis is where the value is. Of the 25 questions RRF still missed:

• 11 were chunking failures: the answer spanned a chunk boundary, so no chunk scored well. Fixed by structure-aware chunking, not by retrieval.
• 7 were vocabulary gaps so severe neither arm helped ("offboarding" against a document titled "leaver process"). Fixed by a synonym list built from query logs.
• 4 were multi-hop: the answer required combining two documents. Fixed by query decomposition or not at all.
• 3 were stale index.

None of those are fixed by tuning $k$. That is the point worth making: once you have hybrid plus RRF, the remaining recall problems are almost never in the fusion, and teams that keep tuning the retriever are optimising the part that already works.

Production evidence

Cormack, Clarke and Buettcher, "Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods" (SIGIR 2009) is the origin, including the $k = 60$ constant and the finding that RRF beat more sophisticated learned fusion methods on TREC data.

Elasticsearch and OpenSearch both ship RRF as a first-class retriever, with rank_constant defaulting to 60 and a configurable per-arm window. Two independent implementations converging on the same default is good evidence it is settled.

Anthropic's contextual retrieval write-up used exactly this stack (contextual embeddings plus contextual BM25, fused, then reranked) and reported that the hybrid combination reduced top-20 retrieval failure rate by about 49 percent versus a dense-only baseline, rising to about 67 percent with reranking added.

BEIR (Thakur et al., NeurIPS 2021) is the benchmark that established the generalisation problem empirically: dense retrievers that beat BM25 in-domain frequently lose to it zero-shot on unfamiliar corpora. That result is the strongest single argument for keeping a lexical arm rather than assuming embeddings have superseded it.

The debate

The credible alternative to RRF is a learned fusion: train a small model on features from both retrievers (scores, ranks, query characteristics) to produce a combined ordering. With enough labelled data it beats RRF, because it can learn that this corpus's dense arm should be trusted more on long queries and less on short ones.

The case against, and why RRF is still the default: it needs training data you usually do not have, it needs retraining when either retriever changes, and the gain over RRF is typically small compared to the gain from fixing chunking or adding a reranker. It is the right investment at large scale with a mature evaluation pipeline, and the wrong first move.

The other alternative is dense only, which is simpler and is what most teams start with. BEIR is the argument against: dense retrievers generalise poorly out-of-domain, and every enterprise corpus is out-of-domain relative to the embedding model's training data.

My position: hybrid with RRF as the default, because it is configuration rather than a model, it has no training data requirement, and it fixes the exact-match failure that dense retrieval cannot. Then a cross-encoder reranker over the fused top 50, because fusion buys recall and reranking buys precision, and you need both. Learned fusion only once you have an evaluation set good enough to prove it helps.

Hybrid is the wrong answer when the corpus is small enough that recall is not the constraint, when latency is so tight that a second retrieval arm does not fit (though the arms are parallel, so this is rarer than people assume), or when the queries are uniformly semantic and there are no identifiers, codes or rare terms in the domain. That last case is unusual: most real corpora have identifiers.

Follow-up Q&A

"Why does RRF beat score normalisation?" Because the scores are not comparable and cannot be made comparable. BM25 is unbounded and corpus-dependent; cosine similarity is bounded and compressed. Min-max normalisation is computed against the candidate set, so the same document normalises differently depending on what else was retrieved. And score distributions vary by query, so a fixed blend weight is wrong for most queries. Rank means the same thing everywhere, which is why using only rank is more robust than using the information you threw away.

"What does $k$ do, and why 60?" It damps the difference between top ranks. At $k = 60$ the contribution of rank 1 is only about 1.15 times that of rank 10, so a document has to rank decently in several lists rather than topping one. That is the behaviour you want from a fusion: reward agreement over single-retriever confidence. 60 comes from the original 2009 paper's tuning on TREC and has been adopted essentially unchanged; tune it only against your own evaluation set.

"You added a reranker and recall didn't improve. Why?" Because it cannot. A reranker reorders the candidate set; if the right document is not in the fused top 50, no amount of rescoring puts it there. Reranking improves precision at small $k$, measured by NDCG@5 or MRR, not recall. If recall is the problem, the fix is upstream: better chunking, a wider retrieval window, an additional retrieval arm, or query expansion.

"How do you tune the balance between the two arms?" With weights on the RRF sum, not by adjusting $k$ per arm, because weighting stays rank-based and therefore avoids reintroducing the score-scale problems. Derive the weights from an evaluation set segmented by query type: if identifier-style queries are 30 percent of traffic and dense hurts them, that is an argument for query routing (send identifier-shaped queries to BM25 alone) rather than a global weight, and routing usually beats weighting.

"How does this interact with filters and access control?" Both arms must apply the same filter, and the dense arm is where it gets expensive: a selective filter sends HNSW off the recall cliff, so you may need exact search over the permitted set on that arm while BM25 handles the filter natively. The failure to watch for is applying the filter to only one arm, which produces a fused list containing documents the user cannot see.

Common misconceptions

The most common is that dense retrieval supersedes BM25. BEIR showed the opposite out-of-domain, and every corpus with product codes, error codes or internal jargon is a case where lexical matching is not optional.

The second is that RRF is a heuristic people use because it is easy. It outperformed more sophisticated learned methods in the paper that introduced it, for the principled reason that ranks are comparable across systems and scores are not.

The third is that fusion and reranking are alternatives. They operate at different stages on different metrics: fusion is a recall stage, reranking is a precision stage, and a pipeline wants both.

Interview delivery note

Say this: "BM25 and dense retrieval fail on different queries, and the failures are close to uncorrelated: BM25 misses paraphrases, dense misses exact tokens like error codes and product identifiers. So I run both and fuse with reciprocal rank fusion, one over k plus rank with k of 60, summed across lists. The important part is that it uses only rank, not score, because BM25 is unbounded and corpus-dependent while cosine is bounded and compressed, and min-max normalisation is computed against the candidate set so the same document normalises differently per query. Rank means the same thing everywhere."

The depth signal is knowing what $k$ does: "a large k flattens the curve, so a document has to rank well in several lists rather than topping one, which is exactly what you want from a fusion." And then the funnel discipline: "fusion is a recall stage and reranking is a precision stage, so if recall is the problem a reranker will not fix it."

Further reading

• Cormack, Clarke and Buettcher, "Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods" (SIGIR 2009).
• Thakur et al., "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models" (NeurIPS 2021), for the out-of-domain result.
• Elasticsearch and OpenSearch documentation on the RRF retriever, rank_constant and rank_window_size.
• Anthropic, "Introducing Contextual Retrieval" (2024), for the measured effect of contextual BM25 plus contextual embeddings plus reranking.