BM25, derived from first principles

What it is

BM25 ("Best Match 25") scores how well a document matches a query, and it is the default lexical ranking function in Lucene, Elasticsearch, OpenSearch, Solr and Vespa. It is a bag-of-words function: it uses term frequencies, document frequencies and document length, and knows nothing about word order or meaning.

$$ \text{score}(D, Q) = \sum_{t \in Q} \text{IDF}(t) \cdot \frac{f(t, D) \cdot (k_1 + 1)}{f(t, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)} $$

Commonly confused with TF-IDF, which it superseded and which it resembles. The two differences are the whole point of BM25 and are derived below: term frequency saturates rather than growing linearly, and document length is normalised with a tunable strength.

Also commonly confused with a similarity metric. BM25 scores are not comparable across queries: a score of 14 on one query and 6 on another says nothing about which document is more relevant, because the scale depends on the query's terms and the corpus. That matters directly when fusing lexical and vector results, which is why RRF fuses by rank rather than by score.

The problem it solves

Start from the naive thing and watch it fail. The obvious relevance signal is how many times the query term appears in the document.

Query: "kubernetes"
Doc A: mentions "kubernetes" 3 times in 200 words   -> tf = 3
Doc B: mentions "kubernetes" 60 times in 40,000 words -> tf = 60
Raw term frequency ranks B above A by 20x.

Doc B is a 40,000-word book index; Doc A is a focused article. Raw term frequency is wrong, and it is wrong in three separate ways that BM25 fixes one at a time.

Mechanics: building the formula, one failure at a time

Failure 1: common words dominate

Query: "the kubernetes scheduler"
"the" appears in every document, often many times.
Summing raw tf, "the" contributes far more than "kubernetes",
so the ranking is driven by the least informative term.

A term that appears everywhere carries no information. Quantify that with inverse document frequency: a term in few documents is more discriminating than one in many.

$$ \text{IDF}(t) = \ln\left(\frac{N - n_t + 0.5}{n_t + 0.5} + 1\right) $$

where $N$ is the number of documents and $n_t$ the number containing $t$.

Corpus: N = 10,000,000

term          n_t          IDF
------------------------------------
"the"         9,900,000    0.010
"kubernetes"     40,000    5.521
"kube-proxy"        800    9.434

"kubernetes" is worth ~550x "the"; "kube-proxy" ~940x.

Why the +0.5 terms and the +1: the 0.5 smoothing comes from the probabilistic retrieval model BM25 derives from (Robertson and Sparck Jones), and it avoids a divide by zero. The +1 inside the logarithm is Lucene's variant and it guarantees IDF is never negative. Without it, a term appearing in more than half the corpus gets a negative IDF, and a document containing it scores lower than one that does not, which is absurd and was a real defect in earlier implementations.

Failure 2: term frequency grows without bound

IDF fixes term weighting across terms. Within a term, raw frequency is still linear:

tf = 1   -> contributes 1 unit
tf = 10  -> contributes 10 units
tf = 100 -> contributes 100 units

Is a document mentioning "kubernetes" 100 times ten times more relevant than one mentioning it 10 times? Clearly not. The first mention tells you the document is about the topic; the hundredth tells you almost nothing new. Relevance should saturate.

BM25's saturation function:

$$ \text{tf-component} = \frac{f \cdot (k_1 + 1)}{f + k_1} $$

With k1 = 1.2:

  f        component      marginal gain
  0          0.00
  1          1.20            +1.20
  2          1.65            +0.45
  3          1.87            +0.22
  5          2.10            +0.13
 10          2.29            +0.06
 20          2.38            +0.02
100          2.42            +0.001
  ∞          2.20 * ...  ->  asymptote at k1 + 1 = 2.2

The first occurrence is worth as much as occurrences 3 through 100
combined.

$k_1$ controls how fast saturation happens. Small $k_1$ saturates quickly (near-binary: does the term appear at all); large $k_1$ approaches linear term frequency. The default of 1.2 is empirical, from TREC evaluations, and it is a good default across a wide range of corpora.

k1 = 0    -> component is constant. Pure binary presence.
k1 = 1.2  -> default. Saturates fast.
k1 = 3.0  -> saturates slowly. Suits corpora where repetition
             genuinely signals aboutness (long technical documents).

Failure 3: long documents win by accident

Saturation helps and does not solve it. A 40,000-word document has more opportunities to contain any term, so it accumulates matches across many query terms.

Normalise by length, but not fully, because two facts are in tension:

A long document is longer because:
  (a) it is verbose about ONE topic  -> should be penalised, since
      its term frequencies are inflated without more relevance
  (b) it genuinely covers MORE       -> should not be penalised,
      since it really does contain more relevant material

BM25 splits the difference with a tunable $b$:

$$ \text{norm} = 1 - b + b \cdot \frac{|D|}{\text{avgdl}} $$

b = 0    -> norm = 1. No length normalisation at all.
b = 1    -> norm = |D|/avgdl. Full normalisation: a document twice
            the average length has its term frequencies effectively
            halved.
b = 0.75 -> the default. Three quarters of the way to full.

avgdl = 500 words.

|D|      b=0.75 norm    effect on the tf component
-----------------------------------------------------
  100      0.40         boosted: short doc, term is concentrated
  500      1.00         neutral, by construction
 2000      2.25         penalised: term is diluted
40000     60.25         heavily penalised

Putting the normaliser in the denominator alongside $k_1$:

$$ \frac{f \cdot (k_1 + 1)}{f + k_1 \cdot \left(1 - b + b\frac{|D|}{\text{avgdl}}\right)} $$

A long document needs proportionally more occurrences to reach the same score, which is exactly the intended behaviour.

The complete function, and a hand-worked example

Corpus:  N = 1,000,000 documents,  avgdl = 300 words
Query:   "kubernetes scheduler"
Params:  k1 = 1.2, b = 0.75

n("kubernetes") = 40,000    IDF = ln((1e6 - 4e4 + 0.5)/(4e4 + 0.5) + 1)
                                = ln(24.0 + 1) = 3.219
n("scheduler")  = 120,000   IDF = ln((1e6 - 1.2e5 + 0.5)/(1.2e5+0.5) +1)
                                = ln(7.33 + 1) = 2.120

DOC A: 150 words. "kubernetes" x 3, "scheduler" x 2
  norm = 1 - 0.75 + 0.75 * (150/300) = 0.25 + 0.375 = 0.625

  kubernetes: 3 * 2.2 / (3 + 1.2*0.625) = 6.6 / 3.75  = 1.760
              x 3.219 = 5.666
  scheduler:  2 * 2.2 / (2 + 1.2*0.625) = 4.4 / 2.75  = 1.600
              x 2.120 = 3.392
  SCORE A = 9.058

DOC B: 3,000 words. "kubernetes" x 12, "scheduler" x 9
  norm = 0.25 + 0.75 * (3000/300) = 0.25 + 7.5 = 7.75

  kubernetes: 12 * 2.2 / (12 + 1.2*7.75) = 26.4 / 21.3 = 1.239
              x 3.219 = 3.989
  scheduler:  9 * 2.2 / (9 + 1.2*7.75)   = 19.8 / 18.3 = 1.082
              x 2.120 = 2.294
  SCORE B = 6.283

DOC A WINS, despite Doc B containing 4x more occurrences of each term.

That is the whole argument for BM25 in one calculation, and it is worth being able to produce on a whiteboard: a focused 150-word document beats a 3,000-word document with four times as many matches, because saturation caps the value of repetition and length normalisation penalises dilution.

Multi-field scoring, and the trap

Real documents have fields: title, body, tags. The obvious approach scores each field and sums.

score = 3.0 * bm25(title, query) + 1.0 * bm25(body, query)

This is most_fields / a weighted sum, and it has a known defect. Each field has its own length normalisation and its own saturation, so a term appearing once in the title and once in the body gets two separate first-occurrence bonuses, which over-rewards documents that mention a term in several fields rather than mentioning it meaningfully in one.

BM25F is the principled fix: combine the term frequencies before applying saturation, with a per-field weight and per-field length normalisation.

$$ \tilde{f}(t, D) = \sum_{\text{fields } s} w_s \cdot \frac{f(t, D_s)}{1 - b_s + b_s \frac{|D_s|}{\text{avgdl}_s}} $$

then apply the saturation function once to $\tilde{f}$.

In Elasticsearch, multi_match with type: most_fields is the naive sum and type: cross_fields approximates BM25F by treating the fields as one combined field. Knowing that cross_fields exists and why is a strong practical signal, because the naive sum is the default people reach for and its failure (a document with the term in five fields beating one that is genuinely about the term) is subtle.

Production evidence

Lucene made BM25 the default similarity in version 6.0 (2016), replacing the classic TF-IDF vector-space model, and the change was based on consistently better relevance across standard collections. Elasticsearch, OpenSearch and Solr inherit it.

Robertson and Zaragoza, "The Probabilistic Relevance Framework: BM25 and Beyond" (2009) is the definitive derivation, showing BM25 as an approximation to the probabilistic relevance model rather than a heuristic. The saturation and length-normalisation terms have a probabilistic justification, not just an empirical one.

The TREC evaluations through the 1990s are where $k_1 \approx 1.2$ and $b \approx 0.75$ come from: they are empirical defaults tuned across many collections, which is why they generalise reasonably and why tuning them per corpus produces modest rather than dramatic gains.

Elasticsearch's explain API exposes every term of the calculation for a given document and query, which is the practical tool for debugging a ranking complaint and is worth naming as the first thing to reach for.

Anserini and BM25 as a baseline in the neural IR literature is telling evidence for its durability: a large fraction of papers proposing neural retrieval report BM25 as the baseline, and for many years it was surprisingly hard to beat on out-of-domain collections, which is the empirical basis for hybrid retrieval rather than pure dense retrieval.

The debate

The case for BM25 as the lexical default: it is fast (a postings-list intersection with a cheap arithmetic score), interpretable (you can explain exactly why a document ranked where it did), needs no training data, and generalises across domains without tuning. On exact-match queries (product codes, error messages, names) it is not merely competitive, it is correct in a way embeddings structurally are not.

The case against, and for dense retrieval: BM25 cannot match a paraphrase. "How do I stop my pods restarting" and "CrashLoopBackOff troubleshooting" share no terms, and BM25 scores that pair at zero. Vocabulary mismatch is the fundamental limitation, and it is exactly what embeddings solve.

The case for learned sparse retrieval (SPLADE, uniCOIL): it keeps the inverted index and its speed while learning term weights and expansions, so it addresses vocabulary mismatch within the sparse framework. Genuinely promising, and it needs training data and a model at index time.

My position: BM25 is one arm of a hybrid, not a legacy baseline to be replaced. The specific reason is that the two methods fail on disjoint query populations. BM25 fails on paraphrase; dense retrieval fails on exact identifiers, because "iPhone 15 Pro 256GB" and "iPhone 15 Pro 128GB" are nearly identical in embedding space and are different products. A commercial corpus has large volumes of both, so fusing them beats either alone by a wide margin, and that is why every serious production search system I have seen runs both.

On tuning: I would not tune $k_1$ and $b$ before fixing analysis and field structure. The defaults are good, the gains from tuning them are typically small, and the large wins in lexical relevance come from getting the analyzer right (stemming, synonyms, the right tokenisation for the language) and from field weighting. Teams tune $k_1$ because it is a visible knob, and it is rarely where the problem is.

The one place I would deviate from defaults deliberately: for a corpus of near-uniform short documents, such as product titles, set $b$ closer to 0, because length normalisation is correcting for a variance that does not exist and it penalises legitimately descriptive titles.

Follow-up Q&A

"Derive BM25 for me." I would build it from the failures of raw term frequency. Raw counts let common words dominate, so multiply by inverse document frequency, which makes a term in 40,000 of 10 million documents worth about 550 times one that appears in nearly all of them. Then raw counts grow linearly, which implies a document mentioning a term 100 times is ten times more relevant than one mentioning it ten times, so saturate: $f(k_1+1)/(f

k_1)$, which asymptotes at $k_1+1$ and makes the first occurrence worth more than occurrences three through a hundred combined. Then long documents accumulate matches by accident, so normalise by length relative to the average, with $b$ controlling how strongly, because a long document might be verbose about one topic or might genuinely cover more.

"What do $k_1$ and $b$ actually do?" $k_1$ is the saturation rate. At zero the score is binary presence; at 1.2 it saturates quickly, so the second and third occurrences add little; at 3 it stays closer to linear, which suits long technical documents where repetition really does signal aboutness. $b$ is the strength of length normalisation: zero is none, one is full, so a document twice the average length has its frequencies effectively halved. The 1.2 and 0.75 defaults are empirical from TREC and they generalise well, which is why tuning them usually yields small gains.

"Why does the first occurrence matter so much?" Because it is the one that tells you the document is about the term. Everything after it is confirmation. With $k_1 = 1.2$, a single occurrence gets 1.20 of a maximum 2.2, so it is more than half the achievable score, and going from ten occurrences to a hundred gains about 0.13. That is the intended behaviour: relevance is not proportional to repetition, and any function that treats it as proportional ranks keyword-stuffed pages above focused ones.

"Can you compare BM25 scores across queries?" No, and this trips people up. The scale depends on the query's IDF values and on the corpus, so 14 on one query and 6 on another tells you nothing about relative relevance. Which is why you cannot threshold on an absolute score to decide "is this a good match", and why fusing lexical and vector results uses reciprocal rank fusion, which combines by rank position, rather than a weighted sum of scores that are not on comparable scales.

"How do you score multiple fields?" Not by summing per-field BM25 scores, which is the default people reach for and which double-counts the first-occurrence bonus: a term in the title and once in the body gets two separate saturation bonuses, over-rewarding documents that mention a term in several fields over one that is genuinely about it. BM25F is the principled fix, combining the per-field frequencies with per-field weights and length normalisation before applying saturation once. In Elasticsearch, cross_fields approximates that and most_fields is the naive sum.

"Where does BM25 fail?" Vocabulary mismatch. "How do I stop my pods restarting" and "CrashLoopBackOff troubleshooting" share no terms and score zero, and no parameter tuning fixes that because the function only sees the terms present. That is exactly what dense retrieval solves, which is why the answer is hybrid rather than replacement. It also has no notion of word order or proximity, so "machine learning" and "learning machine" score identically unless you add a phrase or proximity clause explicitly.

"A user says a document ranked too low. How do you debug it?" The explain API first, because it decomposes the score into per-term IDF, saturation and length normalisation, and the answer is usually visible immediately: the term was stemmed differently than expected, or the document is long and being penalised, or the term is common enough that its IDF is near zero. In my experience the cause is analysis rather than scoring more often than not, a stemmer or tokeniser producing a different term than the query does, which is why I would check the analyzer output before touching $k_1$ or $b$.

Common misconceptions

"BM25 is TF-IDF." It supersedes it with two specific changes: saturating term frequency and tunable length normalisation. Both were responses to concrete failures of the linear form.

"Higher BM25 means more relevant, across queries." Scores are not comparable across queries or corpora. Only the ordering within one query is meaningful.

"Tuning $k_1$ and $b$ is where relevance gains come from." The defaults are good. Analysis, field structure and synonyms are where the large wins are.

"BM25 understands the query." It is bag-of-words. No word order, no proximity, no meaning. Phrase matching is a separate clause you add deliberately.

"BM25 is obsolete now that we have embeddings." It is the correct arm of a hybrid for exact-match queries, which are a large share of commercial traffic, and it remained a stubbornly strong baseline in the neural IR literature for years.

Interview delivery note

Derive it rather than recite it, because the derivation is what demonstrates understanding: "I'd build it from what breaks with raw term frequency. First, common words dominate, so you weight by inverse document frequency. Second, frequency grows linearly, which says a document mentioning a term a hundred times is ten times more relevant than one mentioning it ten times, which is obviously wrong, so you saturate. Third, long documents accumulate matches by accident, so you normalise by length relative to the average."

Give the saturation intuition with a number, because it is the most memorable part: "With $k_1$ at 1.2, the first occurrence is worth 1.2 out of a maximum 2.2, so it's more than half the achievable score, and going from ten occurrences to a hundred buys you about 0.13. The first mention says the document is about the topic; everything after is confirmation."

If you have a whiteboard, do the two-document comparison, because very few candidates can: "A 150-word doc with three mentions beats a 3,000-word doc with twelve, because saturation caps the repetition and length normalisation penalises the dilution. That single calculation is the whole argument for the function."

The practical signals that separate someone who has operated a search system: "scores aren't comparable across queries, which is why hybrid fusion uses reciprocal rank rather than a weighted score sum." And: "for multi-field I'd use cross_fields rather than summing per-field scores, because the naive sum gives a separate first-occurrence bonus per field and over-rewards documents that mention the term everywhere shallowly." And: "when someone complains about a ranking I go to explain first, and the cause is analysis more often than scoring."

The Staff & Team Lead Interview Reference