Cross-encoder and LLM reranking, and the latency budget

What it is

A second-stage scorer that reorders retrieval results by reading the query and each candidate together, which a retriever structurally cannot do.

BI-ENCODER (retrieval)
  encode(query) · encode(document)
  Two independent encodings, one dot product.
  Document side precomputed -> ANN index works.
  No interaction between query terms and document terms.

CROSS-ENCODER (reranking)
  score = model([CLS] query [SEP] document [SEP])
  ONE forward pass over the concatenation, with full
  attention between query and document tokens.
  Nothing precomputable. Every pair is a forward pass.

LLM RERANKER
  A generative model asked to score or order candidates.
  Strongest quality, highest latency and cost, and the
  output needs constraining.

Commonly confused with a better retriever. A reranker cannot recover a document that retrieval never returned, so its ceiling is retrieval recall and reranking a bad candidate set produces a well-ordered bad answer.

Also commonly confused as one technique. Cross-encoders and LLM rerankers differ by roughly an order of magnitude in latency and cost, and choosing between them is a budget decision rather than a quality decision.

The problem it solves

The retrieval model was optimised for the wrong thing, deliberately.

A bi-encoder must place a document's vector in a fixed space
BEFORE seeing any query. So it encodes "what is this document
about" rather than "does this document answer this question".

Query:  "why does my pod get OOMKilled when heap looks fine"
Doc A:  a page about JVM heap tuning
Doc B:  a page about container memory limits counting
        non-heap memory

A bi-encoder scores both highly: both are about memory and
JVMs. A cross-encoder reads the query's "heap looks fine"
against Doc B's "the cgroup limit counts metaspace, thread
stacks and direct buffers" and sees that B ANSWERS the
question while A restates its premise.

That interaction is what the second stage buys, and reported gains from adding a cross-encoder over a dense retriever alone are large on standard benchmarks: it was the finding that made the retrieve-then-rerank pattern standard in neural IR.

Mechanics

The cost model, which decides everything

CROSS-ENCODER, a small model (roughly BERT-base scale)
  ~5 ms per query-document pair unbatched
  Batched on GPU: 50 pairs in ~30-40 ms total

  10M documents:  10M x 5 ms = ~14 hours per query
  50 candidates:  ~35 ms batched
  *** Six orders of magnitude. That gap IS the funnel. ***

LLM RERANKER
  Depends heavily on the shape:
    pointwise (score each doc)     N calls, or one batched
                                   call with N documents in
                                   context
    listwise (order a window)      1 call per window of ~20
    pairwise (compare two)         O(N log N) comparisons.
                                   Best quality, worst cost.
  Realistic: 300 ms to 2 s for a listwise pass over 50
  candidates, and $0.002 to $0.02 per query.

The number to carry: a cross-encoder over 50 candidates is ~35 ms; an LLM reranker over the same 50 is 300 ms to 2 s and costs real money per query. For an interactive search box that difference decides the design.

How many candidates to rerank

Measured, not chosen, and the curve bends.

candidates   NDCG@10   rerank latency
    20         0.712       18 ms
    50         0.741       35 ms
   100         0.749       68 ms
   200         0.752      134 ms

The knee is 50 to 100. Doubling 100 -> 200 buys 0.003 NDCG
for 66 ms, which is a bad trade interactively and a fine one
in an offline pipeline.

And make it adaptive rather than fixed: drop to 20 when the reranker's queue is deep, which is a flag with a measured quality cost rather than an outage.

The full budget, itemised

An interactive search with a 200 ms p99 target:

  query understanding                  2 ms
  query embedding (batched)            4 ms
  lexical retrieval (BM25)            15 ms  ┐ parallel
  dense retrieval (HNSW, ef=64)       25 ms  ┘
  fusion (RRF)                         1 ms
  filtering / business rules           3 ms
  CROSS-ENCODER RERANK (50 docs)      35 ms  <- 40% of the
                                                remaining
                                                budget
  business re-rank + diversity         3 ms
  serialise                            2 ms
  -------------------------------------------
  total                              ~75 ms p50
                                     ~140 ms p99

An LLM reranker in that slot instead:
  300 ms to 2 s -> the budget is gone. Not a tuning problem.

So the LLM reranker is viable when the budget is seconds, not milliseconds: RAG where a generation call follows anyway and adds a second or more, offline pipelines, and low-QPS high-value queries. It is not viable in an interactive search box, and saying so directly is better than proposing it and being asked about latency.

Making LLM reranking work when you do use it

1. LISTWISE, NOT POINTWISE.
   Pointwise scoring asks the model for an absolute
   relevance score per document, which requires a stable
   internal scale across independent calls, and it does not
   have one. Listwise ("order these 20 by relevance to the
   query") is more reliable and is one call instead of 20.

2. SLIDING WINDOW FOR LONG LISTS.
   50 candidates do not fit usefully in one ordering call.
   Order documents 1-20, keep the top 10, slide to include
   11-30, repeat. Roughly N/10 calls, and the top of the
   list gets seen repeatedly, which is where accuracy
   matters.

3. CONSTRAIN THE OUTPUT.
   Ask for document identifiers in order, validate against
   the input set, and fall back to the input order on a
   parse failure or a hallucinated id. A reranker that can
   invent a document is worse than no reranker.

4. POSITION BIAS IS REAL HERE TOO.
   LLMs favour items early in the context. Shuffle the input
   order between windows, or run two orders and combine, and
   measure whether it matters for your model.

5. CACHE AGGRESSIVELY.
   Rerank results keyed by (query, candidate id set). Head
   queries repeat heavily, and a 30 to 50 percent hit rate
   on the top 1 percent of queries removes most of the cost.

Point 3 is not optional. An LLM asked to order ten document ids will occasionally return nine, or eleven, or one that was not in the input, and a reranker that silently drops or invents results is a correctness bug rather than a quality one.

Distillation: the answer when you want both

Train a small cross-encoder to reproduce the LLM reranker's
scores, rather than to reproduce the human labels.

  1. Run the LLM reranker offline over a large query sample.
  2. Use its scores (or its orderings) as training targets.
  3. Train a small cross-encoder to match them.

The student inherits much of the teacher's judgement at
roughly a hundredth of the latency, and the teacher runs
offline where its cost does not matter.

This is the standard resolution of the quality-versus-latency tension, and it is worth naming because the naive framing treats it as a binary choice.

What to do when the reranker is the bottleneck

The degradation ladder, in order:

1. Reduce candidates: 50 -> 20. Measured cost ~0.03 NDCG.
2. Cache more aggressively, and accept staleness.
3. Rerank only for queries where it helps. Head navigational
   queries frequently do not need it: BM25's top result is
   already correct. A cheap classifier on query intent can
   skip reranking for 30 to 40 percent of traffic.
4. Skip the reranker entirely and serve the fused order.
   Measurable quality drop, and a working product.

Step 3 is the under-used one. A query that is an exact product code does not benefit from semantic reranking, and skipping it for that traffic buys latency headroom for the queries that do benefit.

A worked example: choosing between the two

CONTEXT
  A support assistant over 40,000 internal documents.
  Two surfaces:
    (a) a search box in the help centre, p99 target 300 ms
    (b) a RAG answer, where a generation call takes 2 to 4 s

SURFACE (a): SEARCH BOX
  Budget after retrieval and fusion: ~200 ms.
  Cross-encoder over 50: 35 ms. Fits comfortably.
  LLM reranker: 400 ms minimum. Does not fit.
  -> Cross-encoder. And an intent classifier skips reranking
     for exact-title lookups, which are 22% of traffic.

SURFACE (b): RAG ANSWER
  Total budget is seconds because generation dominates.
  An LLM reranker adding 400 ms to a 3 s answer is a 13%
  latency increase for a measurable retrieval-precision gain.
  -> LLM reranker, listwise, sliding window, cached.

THE MEASURED OUTCOME
  Search box: NDCG@10 0.68 -> 0.74 from the cross-encoder,
  at +35 ms.
  RAG: context precision 0.61 -> 0.79 from the LLM reranker,
  and the downstream effect was larger than the retrieval
  metric suggested, because fewer irrelevant chunks in
  context meant fewer confidently wrong answers.

THE OBSERVATION WORTH MAKING
  The same system used different rerankers on different
  surfaces, because the latency budgets differ by an order
  of magnitude. Choosing one reranker for the whole product
  would have been wrong for one surface or the other.

And the second-order effect in the RAG case is the interesting one: improving retrieval precision reduced hallucination more than improving the generation prompt did, because a model given three relevant chunks and seven irrelevant ones will use the irrelevant ones. See diagnosing confidently wrong RAG.

Production evidence

Nogueira and Cho, "Passage Re-ranking with BERT" (2019) established the retrieve-then-rerank pattern with a cross-encoder and reported large gains over BM25 alone, with the cost model that motivated everything after it.

Khattab and Zaharia, "ColBERT" (SIGIR 2020) is the late-interaction middle ground: precompute per-token document embeddings, compute a cheap MaxSim at query time. Better than a bi-encoder, much cheaper than a cross-encoder, and it costs a vector per token in storage.

Sun et al., "Is ChatGPT Good at Search?" (2023) and the RankGPT line of work established listwise LLM reranking with a sliding window, and reported that listwise prompting outperforms pointwise scoring, which is the basis for the listwise recommendation.

Cohere Rerank and similar hosted rerankers are cross-encoder-class models offered as an API, and their published latency figures (tens of milliseconds for tens of documents) are consistent with the cost model above.

The MS MARCO and BEIR leaderboards consistently show cross-encoder reranking as a large gain over retrieval alone across domains, which is the empirical basis for the pattern being standard rather than optional.

Distillation from a cross-encoder teacher into a bi-encoder or smaller cross-encoder is documented across the dense-retrieval literature (for example the work on distilling cross-encoder scores into dual encoders), which is why it is the standard resolution rather than an exotic option.

The debate

The case for a cross-encoder: it is the largest single quality gain available after retrieval, it fits in an interactive latency budget at 50 candidates, it is cheap to run, and it has no output-validity failure mode because it emits a score rather than text.

The case for an LLM reranker: better quality, particularly on queries needing reasoning about the relationship between query and document rather than surface relevance, and no training data required, so it works immediately on a new domain where a cross-encoder would need fine-tuning.

The case for neither: if retrieval recall is the problem, reranking cannot help, and the effort belongs upstream. Measuring retrieval recall separately is what tells you which.

My position: cross-encoder for interactive surfaces, LLM reranker where the budget is seconds and generation already dominates, and distil the LLM into a cross-encoder when you want both.

The budget decides it and the numbers are not close: a cross-encoder over 50 candidates is around 35 milliseconds, an LLM reranker over the same 50 is 300 milliseconds to 2 seconds. For a search box with a 200 millisecond budget, that is not a tuning problem, it is a different design, and proposing an LLM reranker for an interactive surface without acknowledging that is the mistake I would avoid.

The point I would make before either, though, is that a reranker cannot recover what retrieval never returned, so I would measure retrieval recall separately before investing in reranking at all. A team improving their reranker while retrieval recall is 0.6 is spending on the wrong stage, and only the separated metric shows it.

For LLM reranking specifically, the two things I would treat as mandatory rather than refinements: listwise rather than pointwise, because pointwise asks for an absolute score on a scale the model does not stably hold across independent calls; and output validation against the input id set with a fallback to the input order, because a reranker that can drop or invent a document is a correctness bug rather than a quality one, and it will do so occasionally.

And the under-used lever: do not rerank every query. An exact product code or a navigational lookup does not benefit from semantic reranking, and a cheap intent classifier skipping 30 to 40 percent of traffic buys latency headroom for the queries that do benefit. Teams reach for a faster model when the answer is fewer calls.

Where I would push back on the framing of the question: quality versus latency is not a binary choice here. Distillation, running the LLM offline as a teacher and serving a small cross-encoder student, gets most of the quality at a hundredth of the latency, and it is the standard resolution rather than a clever trick.

Follow-up Q&A

"Why do you need a reranker at all?" Because the retriever was optimised for the wrong thing deliberately. A bi-encoder must place a document's vector in a fixed space before seeing any query, so it encodes what the document is about rather than whether it answers this question. A cross-encoder reads both together with full attention, so it can see that one document restates the query's premise and another answers it. That interaction is impossible in a bi-encoder by construction.

"What does it cost?" A small cross-encoder is about five milliseconds per query-document pair unbatched, or roughly 35 milliseconds for 50 candidates batched on GPU. Over ten million documents that same model would be fourteen hours per query, which is the six-order-of-magnitude gap that forces the funnel. An LLM reranker over the same 50 candidates is 300 milliseconds to 2 seconds and costs real money per query.

"So when would you use an LLM reranker?" When the budget is seconds rather than milliseconds, which means RAG where a generation call already takes two to four seconds, offline pipelines, and low-QPS high-value queries. Adding 400 milliseconds to a 3 second answer is a 13 percent latency increase for a real precision gain. Adding it to a 200 millisecond search box is not a tuning problem, it is a different design.

"How many candidates should you rerank?" Measured, and the curve bends around 50 to 100. In the case I worked, 20 gave NDCG 0.712 at 18 milliseconds, 50 gave 0.741 at 35, 100 gave 0.749 at 68, and 200 gave 0.752 at 134. So doubling from 100 to 200 bought 0.003 for 66 milliseconds. I would take 50 and make it adaptive, dropping to 20 under load, which is a flag with a measured quality cost.

"What goes wrong with LLM reranking specifically?" Three things. Pointwise scoring asks for an absolute relevance number on a scale the model does not stably hold across independent calls, so listwise ordering is more reliable and is one call instead of twenty. Position bias, because the model favours items early in the context, so shuffle between windows or run two orders. And output validity: it will occasionally return nine ids instead of ten, or one that was not in the input, so you validate against the input set and fall back to the input order. A reranker that can invent a document is a correctness bug.

"How do you handle more than twenty candidates with a listwise reranker?" A sliding window. Order documents one to twenty, keep the top ten, slide to include eleven to thirty, repeat. Roughly N over ten calls, and the top of the list gets seen repeatedly, which is where accuracy matters most. That is the RankGPT approach and it is the standard way to get listwise quality over a list too long for one call.

"Can you get LLM quality at cross-encoder latency?" Largely, through distillation: run the LLM reranker offline over a large query sample, use its scores or orderings as training targets, and train a small cross-encoder to match them. The student inherits much of the teacher's judgement at about a hundredth of the latency, and the teacher runs offline where cost does not matter. That is the standard resolution and it is why the quality-latency framing is not really binary.

"Your reranker is the bottleneck. What do you drop?" Four steps in order. Reduce candidates from 50 to 20, at a measured cost of around 0.03 NDCG. Cache more aggressively, keyed by query and candidate set, since head queries repeat heavily. Then the under-used one: skip reranking entirely for query types that do not benefit, because an exact product code or a navigational lookup already has BM25's top result correct, and a cheap intent classifier can skip 30 to 40 percent of traffic. Then serve the fused order.

"When is reranking not the answer at all?" When retrieval recall is the problem. A reranker only reorders what retrieval returned, so if the answer-bearing document was never a candidate, no reranker recovers it. That is why I would measure retrieval recall separately from final NDCG before investing here, because a team tuning a reranker while recall is 0.6 is spending on the wrong stage and the aggregate metric will not tell them.

Common misconceptions

"A better reranker fixes bad results." It reorders what retrieval returned. Recall lost upstream is unrecoverable.

"Cross-encoders and LLM rerankers are interchangeable." They differ by an order of magnitude in latency and cost, and that difference decides which surfaces they fit.

"Pointwise LLM scoring works fine." It requires a stable absolute scale across independent calls, which the model does not hold. Listwise is both better and cheaper.

"Rerank everything." Navigational and exact-match queries do not benefit, and skipping them buys headroom for the queries that do.

"It's quality versus latency." Distillation gets most of the quality at a fraction of the latency, and it is the standard answer rather than a trick.

Interview delivery note

Explain what the second stage buys, mechanically, because that justifies the whole cost: "A bi-encoder has to place a document's vector before seeing any query, so it encodes what the document is about rather than whether it answers this question. A cross-encoder reads both together with full attention, so it can tell the difference between a document that restates the query's premise and one that answers it. That interaction is impossible in a bi-encoder by construction."

Then lead with the cost model, because it decides the design: "A small cross-encoder is about five milliseconds a pair, so 50 candidates is around 35 milliseconds batched. An LLM reranker over the same 50 is 300 milliseconds to 2 seconds. For a search box with a 200 millisecond budget that isn't a tuning problem, it's a different design."

Give the candidate-count answer with the curve, because it is a concrete tuning decision: "For how many to rerank, the curve bends around 50 to 100. Going from 100 to 200 bought 0.003 NDCG for 66 milliseconds in the case I worked, which is a bad trade interactively. I'd take 50 and make it adaptive under load."

The two LLM-specific requirements, because they are where implementations fail: "If I am using an LLM reranker, listwise rather than pointwise, because pointwise needs a stable absolute scale the model doesn't hold across independent calls. And output validation against the input id set with a fallback to input order, because it will occasionally return nine ids or one that wasn't there, and a reranker that can invent a document is a correctness bug."

Close on the framing correction: "and I'd resist the quality-versus-latency binary. Distilling the LLM reranker into a small cross-encoder gets most of the quality at a hundredth of the latency, with the teacher running offline where cost doesn't matter. That's the standard answer. Though before any of it I'd measure retrieval recall separately, because a reranker can't recover what retrieval never returned."

The Staff & Team Lead Interview Reference