The multi-stage ranking funnel

What it is

A cascade of stages, each processing fewer items with a more expensive model than the one before it.

   10,000,000 documents
        │  RETRIEVAL          cheap, recall-oriented, ~10 ms
        ▼
        1,000 candidates
        │  FILTERING          business rules, ~2 ms
        ▼
          500 candidates
        │  LIGHT RANKER       cheap model, many features, ~15 ms
        ▼
           50 candidates
        │  HEAVY RANKER       cross-encoder or large model, ~35 ms
        ▼
           20 results
        │  RE-RANK            diversity, business, exploration, ~3 ms
        ▼
        the page

The invariant that defines it: each stage may only shrink the set, so recall lost at stage $n$ is unrecoverable at stage $n+1$. A brilliant reranker over a bad candidate set is worse than a mediocre reranker over a good one, and that asymmetry is why retrieval recall is measured separately from ranking quality.

Commonly confused with an optimisation. The funnel is not a performance tweak applied to a one-model design; it is a consequence of arithmetic. Scoring ten million documents with a cross-encoder is not slow, it is impossible: at roughly 5 ms per query-document pair, ten million pairs is fourteen hours.

Also commonly confused with "retrieval then reranking", which is the two-stage version. The distinction that matters in production is between a light ranker (a model over precomputed features, scoring hundreds of items) and a heavy ranker (a model that jointly encodes query and document, scoring tens).

The problem it solves

The bi-encoder / cross-encoder trade is the reason the funnel exists, and stating it is the cleanest way to justify the whole structure.

BI-ENCODER (two-tower)
  Encode query and document SEPARATELY, compare with a dot product.
  Document vectors are precomputed offline -> an ANN index works.
  Query cost: one encode + one ANN search over millions.
  Quality: no interaction between query and document terms, so it
  cannot represent "this document answers this specific question".

CROSS-ENCODER
  Encode query and document TOGETHER through a transformer, with
  full attention between them.
  Quality: substantially better, because the model sees the
  interaction.
  Cost: nothing can be precomputed. Every (query, document) pair
  is a forward pass.

Arithmetic, 10M documents, a small cross-encoder at ~5 ms/pair:
  10,000,000 x 5 ms = 50,000 seconds = ~14 hours per query.

The same cross-encoder over 50 candidates:
  50 x 5 ms = 250 ms, and batched on GPU, ~35 ms.

That six-order-of-magnitude gap is the funnel. Every stage exists to reduce the set enough that the next stage's model becomes affordable.

Mechanics

Stage 1: retrieval, and why it is several sources

Retrieval optimises recall, not precision. Its job is to guarantee the answer is in the candidate set; ordering it is somebody else's problem.

Source                    Mechanism                     Yield
------------------------------------------------------------------
Lexical (BM25)            inverted index                 200
Dense (bi-encoder)        ANN over embeddings            300
Item-to-item              precomputed co-occurrence      200
Popular in segment        precomputed, cached            100
Recent / fresh            time-ordered                   100

All run in parallel, so latency is the slowest (typically the ANN search), not the sum. The union is deduplicated.

Multiple sources rather than one better source, because they fail on disjoint query populations. BM25 fails on paraphrase; dense retrieval fails on exact identifiers, since "iPhone 15 Pro 256GB" and the 128GB variant are nearly identical in embedding space. Precomputed sources cover cold-start and popularity, which neither learned source handles well.

The measurement that governs this stage:

retrieval recall@1000 = (relevant docs in the candidate set)
                        / (relevant docs in the corpus)

If this is 0.72, then 28% of relevant documents are permanently
unreachable for that query, and no downstream stage recovers them.
*** This number is the ceiling on the entire system. ***

Stage 2: the light ranker

Scores hundreds of candidates with a model over precomputed features, so nothing needs to be encoded at query time.

def light_rank(query_features, candidates, feature_store):
    # Item features come from PROCESS MEMORY, not a network call:
    # 500 candidates x thousands of QPS is not servable remotely.
    item_feats = feature_store.batch_lookup(candidates)   # ~0.5 ms

    # Cross features are computed here because they depend on both
    # sides and therefore cannot be precomputed per item.
    cross = compute_cross(query_features, item_feats)

    X = np.hstack([np.tile(query_features, (len(candidates), 1)),
                   item_feats, cross])
    # ONE batched call. Scoring 500 items individually is 500
    # framework round trips and is commonly 10x slower.
    return model.predict_batch(X)          # GBDT or small MLP

Gradient-boosted trees (LambdaMART, LightGBM) remain the standard choice here and are often better than a neural model on tabular features, because the features are heterogeneous (counts, ratios, categorical ids, text-match scores) and trees handle that without feature engineering that a neural network requires.

Typical feature families:

Query-only        length, language, intent class, is-navigational
Document-only     quality score, freshness, popularity, length
Query-document    BM25 score, title match, embedding cosine,
                  exact-phrase match, field-level matches
User-document     has-seen, category affinity, past interaction
Context           device, time of day, session position

The query-document features are the ones that matter most and the ones that must be computed at query time. Everything else can be precomputed.

Stage 3: the heavy ranker

A cross-encoder over the top 50, where full query-document attention is affordable.

# Pairs are batched into ONE forward pass. The GPU is idle
# scoring one pair at a time, exactly as in LLM decode.
pairs = [(query, doc.text) for doc in top_50]
scores = cross_encoder.predict(pairs, batch_size=50)   # ~35 ms

How many candidates to send is the tuning decision, and it has a measurable shape:

candidates    NDCG@10    latency
    20          0.712      18 ms
    50          0.741      35 ms
   100          0.749      68 ms
   200          0.752     134 ms

The curve bends around 50-100. Doubling from 100 to 200 buys
0.003 NDCG for 66 ms, which is a bad trade in an interactive
system and a fine one in an offline pipeline.

Late interaction (ColBERT) is the middle option worth naming: precompute per-token document embeddings and compute a cheap MaxSim interaction at query time. Much better than a bi-encoder, much cheaper than a cross-encoder, and it costs a lot of storage because you store a vector per token rather than per document.

Stage 4: re-ranking, for what the model cannot express

def rerank(scored, k=20):
    selected, per_cat, per_seller = [], {}, {}
    for item_id, score in scored:
        c, s = meta[item_id].category, meta[item_id].seller
        # A pointwise ranker scores items INDEPENDENTLY, so it
        # cannot express "twenty results from one seller is worse
        # for the user than twelve plus eight others". That is a
        # property of the SET, so it is applied to the set.
        if per_cat.get(c, 0) >= 5:    continue
        if per_seller.get(s, 0) >= 3: continue
        selected.append(item_id)
        per_cat[c] = per_cat.get(c, 0) + 1
        per_seller[s] = per_seller.get(s, 0) + 1
        if len(selected) == k: break
    return selected

Plus exploration slots: a small fraction of positions filled by a policy that deliberately shows items the ranker is uncertain about. Without it the training data becomes a record of the ranker's own past decisions and the effective catalogue narrows over months. See popularity bias and feedback loops.

Training the stages consistently

The subtlety that separates a real answer: the stages must be trained on the distribution they will see in production.

WRONG: train the heavy ranker on (query, random negative) pairs.
       In production it never sees random negatives; it sees the
       top 50 from the light ranker, which are all plausible.
       The model has never had to distinguish "good" from
       "slightly better" and its production behaviour is much
       worse than its offline metrics suggest.

RIGHT: HARD NEGATIVE MINING. Sample negatives from the actual
       output of the preceding stage.
         1. Train stage N on easy negatives.
         2. Run stage N over the training queries.
         3. Take its top results that are NOT relevant.
         4. Retrain stage N+1 using those as negatives.
       Repeat. This is the single largest quality lever in
       multi-stage training and it is routinely skipped.

Distillation is the complementary technique: train the light ranker to reproduce the heavy ranker's scores rather than the raw labels. The light ranker inherits some of the cross-encoder's judgement at a fraction of the cost, which raises the quality of the 50 that reach the expensive stage.

The latency budget

Stage                              p50      p99
--------------------------------------------------
Query understanding                 2 ms     4 ms
Retrieval (5 sources, parallel)    12 ms    28 ms
Dedupe + business filters           2 ms     4 ms
Feature lookup (in process)         1 ms     2 ms
Light ranker (500 items, batched)  14 ms    26 ms
Heavy ranker (50 items, GPU)       33 ms    58 ms
Re-rank + diversity                 3 ms     5 ms
--------------------------------------------------
TOTAL                              67 ms   127 ms

The heavy ranker is roughly half the budget, which makes it the first degradation lever: under load, skip it and serve the light ranker's order, which is a measurable but acceptable quality drop and roughly halves latency.

Production evidence

Covington, Adams and Sargin, "Deep Neural Networks for YouTube Recommendations" (RecSys 2016) is the canonical description of the candidate-generation-then-ranking split, including the argument that the two stages use different models and different features because they solve different problems.

Nogueira and Cho, "Passage Re-ranking with BERT" (2019) established the retrieve-then-cross-encode pattern in neural IR and reported large gains over BM25 alone, with the cost model that motivated everything after it.

Khattab and Zaharia, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT" (SIGIR 2020) is the late-interaction middle ground, and its storage cost (a vector per token) is the documented trade.

Xiong et al., "Approximate Nearest Neighbor Negative Contrastive Learning" (ANCE) (ICLR 2021) is the reference for hard negative mining from the retriever's own output, and its central finding is that negatives sampled from the actual retrieval distribution matter more than model architecture.

Bing's and Baidu's published search architectures both describe three or more ranking stages with increasing cost, which is convergent evidence that two stages is the simplified version rather than the production shape.

Amazon's and Airbnb's published search-ranking work both describe the funnel with business rules and diversity applied after model scoring, which is where those constraints belong because pointwise rankers cannot express them.

The debate

The case for more stages: each stage lets the next use a more expensive model on fewer items, so quality per unit of latency improves. Three or four stages is what large search systems converge on.

The case for fewer stages: every stage is a model to train, monitor, version and debug, and every boundary is a place where the training distribution can diverge from production. Two stages (retrieve, rerank) captures most of the benefit with much less operational surface.

The case for one model: a single end-to-end model has no distribution mismatch between stages and no recall lost at a boundary. It is also arithmetically impossible above a certain corpus size, which is the whole point.

My position: retrieval plus one light ranker plus one heavy ranker, and add stages only when a measured latency or quality bound forces it.

Three stages rather than two because the light ranker earns its place specifically: it lets the heavy ranker see 50 candidates chosen by a model with features rather than 50 chosen by similarity alone, and that materially improves what the expensive stage has to work with. Four stages I would want evidence for.

The property I would insist on regardless of stage count is measuring retrieval recall separately from ranking quality. It is the ceiling on everything downstream, it is the number that tells you which stage to fix, and a team reporting only NDCG cannot distinguish "our ranker is bad" from "our ranker never saw the right document". That distinction is worth a quarter of engineering time.

The technique I would insist on is hard negative mining from the preceding stage's actual output. Training the reranker on random negatives teaches it to distinguish relevant from absurd, when its production job is distinguishing relevant from plausible. It is the single largest quality lever in multi-stage training, it costs a training-pipeline change rather than new infrastructure, and it is routinely skipped.

Where I would push back: do not add a stage to fix a recall problem. If the answer is not in the candidate set, another ranker cannot help, and the instinct to add a smarter reranker when quality is poor is exactly backwards when the cause is retrieval. Measure first.

Follow-up Q&A

"Why not score everything with the best model?" Arithmetic. A cross-encoder at about five milliseconds per query-document pair over ten million documents is fourteen hours per query. Over fifty candidates it is 250 milliseconds, or about 35 batched on GPU. That six-order-of-magnitude gap is the funnel: every stage exists to reduce the set enough that the next stage's model becomes affordable. It is not an optimisation applied to a one-model design, it is a consequence of the cost model.

"What is the difference between a bi-encoder and a cross-encoder?" A bi-encoder encodes query and document separately and compares with a dot product, so document vectors are precomputed offline and an ANN index works, at the cost of no interaction between query and document terms. A cross-encoder encodes them together with full attention, which is substantially better and means nothing can be precomputed, so every pair is a forward pass. The funnel exists because you want the cross-encoder's quality on the few candidates where you can afford it.

"How many candidates should reach the heavy ranker?" Measured, not guessed, and the curve bends. In the shape I have seen, twenty gives NDCG around 0.71 at 18 milliseconds, fifty gives 0.74 at 35, a hundred gives 0.749 at 68, and two hundred gives 0.752 at 134. So doubling from a hundred to two hundred buys 0.003 for 66 milliseconds, which is a bad trade interactively and a fine one offline. I would pick the knee and make the count adaptive under load.

"What is the single biggest mistake in training these?" Training each stage on random negatives. In production the reranker sees the top fifty from the previous stage, all of which are plausible, and if it was trained to distinguish relevant from absurd it has never learned the distinction it actually needs. Hard negative mining fixes it: run the previous stage over the training queries, take its top non-relevant results, and use those as negatives. It is the largest quality lever available and it costs a pipeline change rather than infrastructure.

"Where does diversity go, and why not in the model?" In the re-rank, after scoring, because a pointwise ranker scores items independently and cannot express a property of the set. "Twenty results from one seller is worse for the user than twelve plus eight others" is a statement about the whole page. Listwise models can express it and are much more expensive to train and serve, so the standard answer is a cheap greedy constraint pass after ranking, which is what Amazon's and Airbnb's published architectures do.

"Your quality is poor. Which stage do you fix?" Measure retrieval recall separately from final NDCG, and the two together tell you. High recall and low NDCG is a ranking problem; low recall is a retrieval problem and no reranker can fix it, because the document was never in the set. If you only report NDCG you cannot distinguish those, and the instinct is to add a smarter reranker, which is exactly wrong when the cause is retrieval. That confusion is worth a quarter of wasted engineering.

"What do you drop under load?" The heavy ranker first, because it is roughly half the latency budget and the light ranker's order is already reasonable. Then reduce the candidate count into the light ranker. Then reduce the ANN ef parameter, which trades recall for speed. Each is a flag with a measured quality cost, so the decision under pressure is informed rather than improvised.

"Is there a middle ground between bi-encoder and cross-encoder?" Late interaction, which is ColBERT: precompute per-token document embeddings offline and compute a cheap MaxSim interaction at query time. Much better than a bi-encoder because there is some term-level interaction, much cheaper than a cross-encoder because the document side is precomputed. The cost is storage, since you keep a vector per token rather than per document, which for a large corpus is substantial.

Common misconceptions

"A better reranker fixes bad results." Not if the document was never retrieved. Recall lost at retrieval is unrecoverable, and this is the most consequential misconception in the topic.

"The funnel is an optimisation." It is a consequence of the cost model. The one-model alternative is not slow, it is arithmetically impossible.

"Retrieval should be precise." Retrieval should be recall-oriented. Precision is what the later stages are for, and a retrieval stage tuned for precision throws away documents the reranker could have promoted.

"Train each stage independently on the same data." Each stage must be trained on the distribution it will actually see, which means hard negatives from the preceding stage.

"Diversity is a model objective." Pointwise rankers score independently and cannot express set-level constraints.

Interview delivery note

Justify the structure with the cost arithmetic, because that converts a pattern into a derivation: "The funnel isn't an optimisation, it's forced. A cross-encoder is about five milliseconds per query-document pair, so ten million documents is fourteen hours per query. Over fifty candidates it's thirty-five milliseconds batched. Every stage exists to shrink the set enough that the next stage's model becomes affordable."

State the invariant, because it is the thing that governs every decision: "And the rule is that each stage can only shrink the set, so recall lost at retrieval is unrecoverable. Which is why I'd measure retrieval recall separately from final NDCG: high recall with low NDCG is a ranking problem, low recall is a retrieval problem, and if you only report NDCG you'll spend a quarter tuning the wrong stage."

Volunteer the training subtlety, because it is where the depth is: "The biggest mistake I see is training each stage on random negatives. In production the reranker sees fifty plausible candidates from the previous stage, and if it learned to distinguish relevant from absurd it never learned the distinction it actually needs. Hard negative mining, sampling negatives from the previous stage's own output, is the largest quality lever and it's routinely skipped."

And give a concrete tuning answer rather than a principle: "For how many candidates reach the heavy ranker, the curve bends around fifty to a hundred. Going from a hundred to two hundred bought about 0.003 NDCG for sixty-six milliseconds in the case I worked, which is a bad trade interactively. I'd make it adaptive so it drops under load."

The Staff & Team Lead Interview Reference