Two-tower retrieval and feature-store parity

What it is

Two separate encoders producing vectors in a shared space, compared with a dot product.

    query / user features            item features
            │                              │
     ┌──────▼──────┐                ┌──────▼──────┐
     │ QUERY TOWER │                │ ITEM TOWER  │
     │  (small,    │                │  (can be    │
     │   online)   │                │   large,    │
     │             │                │   offline)  │
     └──────┬──────┘                └──────┬──────┘
            │                              │
        q ∈ R^d                       i ∈ R^d
            │                              │
            └────────► score = q · i ◄─────┘

The asymmetry is the entire point. The item tower runs offline over the whole catalogue, producing vectors that are loaded into an ANN index. The query tower runs once per request on a small feature set. There is no interaction between query and item features inside the model, which is precisely what allows the item side to be precomputed and indexed, and precisely why a two-tower model ranks worse than a cross-encoder.

Commonly confused with a ranking model. It is a retrieval model: its job is to get the right items into a candidate set of a few hundred, not to order them. Evaluating it with NDCG@10 is measuring the wrong thing; recall@100 or recall@1000 is the metric.

Also commonly confused with "just embeddings". The architecture is unremarkable; what makes it work is the loss function and the negative sampling, and that is where the interesting engineering is.

The problem it solves

Retrieval must reduce millions of items to hundreds in single-digit milliseconds. Two families of approach:

INVERTED INDEX (BM25)
  Fast, no training, exact-match strong.
  Cannot match a paraphrase: "shoes for running" and "trainers"
  share no terms.

CROSS-ENCODER
  Best quality, and it cannot be precomputed, so it is
  arithmetically impossible over millions of items.

TWO-TOWER
  Learned semantic matching, precomputable item side, ANN-indexable.
  The compromise that makes learned retrieval affordable.

The compromise is the loss of query-item interaction. A cross-encoder can represent "this document answers this specific question"; a two-tower model can only place the query and the document near each other in a fixed space. That is why the funnel exists: two-tower retrieves, a cross-encoder ranks.

Mechanics

Training: the sampled softmax, and why the correction matters

The natural objective is a softmax over the whole catalogue, which is intractable at ten million items. The standard approach uses in-batch negatives: for each positive (query, item) pair in a batch, the other items in the batch are negatives.

def in_batch_loss(q, i, temperature=0.05):
    """q: (B, d) query vectors, i: (B, d) item vectors.
    The diagonal is the positive pairs; off-diagonal are negatives."""
    logits = (q @ i.T) / temperature          # (B, B)
    labels = torch.arange(len(q))             # positives on the diagonal
    return F.cross_entropy(logits, labels)

This is free negatives at batch size, and it has a specific bias that must be corrected.

Items appear in a batch in proportion to their frequency in the training data, so popular items are sampled as negatives far more often than rare ones. The model therefore learns to push popular items away from queries, which systematically under-ranks exactly the items that are most often relevant.

The logQ correction (Yi et al., 2019) subtracts the log sampling probability from each logit:

def corrected_loss(q, i, item_freq, temperature=0.05):
    logits = (q @ i.T) / temperature
    # Subtract log P(item sampled). Frequently sampled items get a
    # larger subtraction, which cancels the over-penalty.
    logits = logits - torch.log(item_freq).unsqueeze(0)
    return F.cross_entropy(logits, torch.arange(len(q)))

This is the single most important detail in two-tower training and it is routinely omitted. Without it the retrieval model has a built-in anti-popularity bias that fights the popularity bias in the rest of the system, in the wrong direction, and the symptom is that head queries retrieve poorly.

Hard negatives: the second lever

In-batch negatives are easy: a random item from the catalogue is obviously irrelevant to most queries. The model learns to separate relevant from absurd, when its production job is separating relevant from plausible.

Query: "waterproof hiking boots size 44"

EASY NEGATIVE (in-batch):     a coffee grinder
                              The model learns nothing from this
                              after the first few epochs.

HARD NEGATIVE (mined):        "waterproof hiking boots size 42"
                              This is what production looks like,
                              and distinguishing it is the actual task.

# ANCE-style mining: periodically re-index with the CURRENT model
# and sample negatives from its own top results. The negatives
# track the model as it improves, which is why it is done
# iteratively rather than once.
def mine_hard_negatives(model, queries, index, k=200, skip_top=10):
    index.rebuild(model.item_tower(all_items))
    negs = {}
    for q in queries:
        hits = index.search(model.query_tower(q), k=k)
        # Skip the very top, because those are often unlabelled
        # positives rather than true negatives. Sampling them as
        # negatives actively teaches the model to be wrong.
        negs[q.id] = [h for h in hits[skip_top:] if h not in q.positives]
    return negs

skip_top is the detail that matters. The model's highest-scoring non-labelled results are disproportionately unlabelled relevant items, because relevance labels are sparse. Using them as negatives teaches the model that correct answers are wrong, and it degrades quality in a way that is hard to diagnose.

The reported effect of hard negatives is large: in the dense-retrieval literature it is consistently a bigger lever than architecture changes, which is the practical takeaway.

Serving: the asymmetry in practice

# OFFLINE, nightly or on catalogue change.
item_vectors = item_tower.encode(all_items, batch_size=1024)   # GPU batch
ann_index.build(item_vectors, M=32, efConstruction=200)

# ONLINE, per request. Everything here is on the latency budget.
def retrieve(user, context, k=300):
    features = feature_store.get_online(user.id)        # ~2 ms
    q = query_tower(features, context)                  # ~0.3 ms, must be tiny
    return ann_index.search(q, k=k, ef=64)              # ~6 ms

The query tower must be small. It runs on every request, so a large query encoder is a latency cost paid per query while a large item tower is a batch job cost paid once. Asymmetric tower sizes are a legitimate and underused design choice: a 12-layer item tower and a 3-layer query tower is a good trade when query features are few and simple.

Feature-store parity: where these break in production

This is the section that separates people who have shipped one. The model trains on features computed one way and serves on features computed another, and the divergence is silent.

THREE DISTINCT FAILURES, WITH DIFFERENT FIXES

1. POINT-IN-TIME LEAKAGE
   Training on user_purchase_count as it is TODAY, to predict a
   click from three months ago. The feature contains the future.
   Offline metrics look excellent; production is much worse.
   Fix: the feature store must answer "what was this feature's
   value at time T", which means event-time versioning rather
   than a current-value table.

2. IMPLEMENTATION SKEW
   Training computes days_since_last_purchase in SQL over the
   warehouse; serving computes it in Python from Redis. Different
   timezone handling, different null semantics, different
   rounding. They diverge, subtly, and nobody notices.
   Fix: ONE definition compiled to both paths. This is what a
   feature store is actually for, and the reason it is
   infrastructure rather than a convenience.

3. MISSINGNESS MISMATCH
   Training data is backfilled and complete. Production has Redis
   timeouts, cold users and partial failures. A model that never
   saw a null produces garbage rather than a degraded score.
   Fix: inject realistic missingness into training, at the rates
   observed in production. Cheap, and almost never done.

The detection mechanism, which should be built before the model ships:

# Log the EXACT feature vector used at serving time, sampled.
# Then compare distributions against the training set daily.
def check_parity(serving_sample, training_ref):
    for name in training_ref.columns:
        psi = population_stability_index(serving_sample[name],
                                         training_ref[name])
        # PSI > 0.1 is a shift worth investigating;
        # PSI > 0.2 is an alert, not a dashboard.
        if psi > 0.2:
            alert(f"feature {name} PSI={psi:.3f}")

        # Null rate is the one that catches implementation skew
        # fastest, because a broken join shows up here first.
        null_serve = serving_sample[name].isna().mean()
        null_train = training_ref[name].isna().mean()
        if abs(null_serve - null_train) > 0.05:
            alert(f"feature {name} null rate {null_train:.2%} -> {null_serve:.2%}")

Null-rate divergence is the fastest signal in my experience, because a broken join or a timed-out lookup shows up there before it shows up in the distribution of the values that did arrive.

The embedding-freshness problem

The item tower produces vectors offline. The catalogue changes.

Item text changes    -> re-embed that item, upsert the vector.
                        Cheap, incremental, hourly is fine.

MODEL changes        -> EVERY vector is now in a different space.
                        Old and new vectors are not comparable, so
                        a partial rollout produces nonsense scores.

A model version change requires a full re-embed and an atomic index swap, and mixing vectors from two model versions in one index is the failure that produces inexplicably bad results with no error anywhere. The mechanics are the alias-swap pattern:

1. Build index_v2 offline with the new item tower.
2. Verify recall on a held-out set against index_v1.
3. Swap the alias atomically. The query tower version must
   change at the same instant, because a v1 query vector
   searched against a v2 index is meaningless.
4. Keep index_v1 for the rollback window.

Query tower and item tower versions are coupled and must be deployed together, which is worth saying explicitly because they are usually in different services and deploy independently by default.

Production evidence

Yi et al., "Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations" (RecSys 2019) is the Google paper that introduced the logQ correction for in-batch negatives, and its central result is that the correction materially improves retrieval quality, which is why omitting it is a real defect rather than a simplification.

Covington, Adams and Sargin, "Deep Neural Networks for YouTube Recommendations" (RecSys 2016) is the earlier reference for the candidate-generation tower and for serving it via approximate nearest neighbour search.

Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering" (EMNLP 2020) established the dual-encoder for text retrieval with in-batch negatives, and Xiong et al., "ANCE" (ICLR 2021) established iterative hard-negative mining from the model's own index, reporting it as a larger lever than architecture.

Feast and Tecton exist as products specifically because point-in-time correctness and one-definition-two-paths are hard enough to warrant dedicated infrastructure, which is good evidence that the parity section is a real production risk rather than a theoretical one.

Facebook's "Embedding-based Retrieval in Facebook Search" (KDD 2020) documents the two-tower architecture in production search including hard negative mining strategy and the interaction with the downstream ranking stages.

The debate

The case for two-tower retrieval: it is the only learned retrieval that can be precomputed and ANN-indexed, so it is the only way to get semantic matching into the retrieval stage at all. It handles paraphrase and intent, which lexical retrieval cannot.

The case for lexical retrieval instead: no training, no feature store, no embedding freshness problem, no model-version coupling, and it is strictly better on exact-match queries where the dense model puts the 128GB and 256GB variants at near-identical similarity. The operational surface is a fraction of the size.

The case for late interaction (ColBERT): better quality than a two-tower model because there is some term-level interaction, while keeping the document side precomputable. The cost is storage: a vector per token rather than per document.

My position: two-tower as one arm of hybrid retrieval, never as a replacement for lexical, and with the logQ correction and hard negatives treated as mandatory rather than refinements.

The two arms fail on disjoint query populations, which is the concrete reason for hybrid rather than a preference for ensembles: dense retrieval fails on exact identifiers and lexical fails on paraphrase, and a real corpus has large volumes of both.

On training, the two details I would treat as non-negotiable are the logQ correction and iteratively mined hard negatives. Without the correction the model learns an anti-popularity bias, because popular items appear as in-batch negatives in proportion to their frequency, and the symptom is poor retrieval on head queries which is exactly where volume is. Without hard negatives the model has learned to separate relevant from absurd when its production task is separating relevant from plausible, and the offline metrics will not show it because the offline negatives are easy too.

The operational risk I would design for from the start is feature-store parity, because the failure is silent and the usual symptom is "the model was much better offline". Three specific things: point-in-time correct features so training does not leak the future, one feature definition compiled to both paths rather than SQL in training and Python in serving, and realistic missingness injected into training so a Redis timeout produces a degraded score rather than garbage. Then log serving feature vectors and alert on population stability index above 0.2 and on null-rate divergence, which is the fastest signal.

And the deployment coupling: query tower and item tower versions must change together, with a full re-embed and an atomic alias swap. Mixing vectors from two model versions in one index produces meaningless scores with no error anywhere, and because the two towers usually live in different services that deploy independently, it takes a deliberate mechanism to prevent.

Follow-up Q&A

"Why two towers rather than one model?" Because the item side has to be precomputable. There is no interaction between query and item features inside the model, which is exactly what lets you encode the catalogue offline and put the vectors in an ANN index. A model with interaction, a cross-encoder, cannot precompute anything, so every query-item pair is a forward pass and it is arithmetically impossible over millions of items. The cost of the two-tower design is that lost interaction, which is why it retrieves rather than ranks.

"How is it trained?" Sampled softmax with in-batch negatives: for each positive pair in a batch, the other items in the batch are negatives, which gives you free negatives at batch size. The critical detail is the logQ correction: items appear in batches in proportion to their frequency, so popular items are over-sampled as negatives and the model learns to push them away from queries. Subtracting the log sampling probability from each logit cancels that. Without it you have a built-in anti-popularity bias and the symptom is poor retrieval on head queries.

"Why do hard negatives matter so much?" Because in-batch negatives are easy. A random catalogue item is obviously irrelevant to most queries, so after a few epochs the model learns nothing from them, and it has learned to separate relevant from absurd. In production it must separate "waterproof hiking boots size 44" from "size 42", which is a completely different task. Mining negatives from the model's own top results, iteratively as the model improves, is consistently a bigger quality lever than architecture changes in the dense retrieval literature.

"Is there a trap in hard negative mining?" Yes, and it is the reason for skipping the very top results. The model's highest-scoring unlabelled items are disproportionately unlabelled positives, because relevance labels are sparse. Using them as negatives teaches the model that correct answers are wrong, and the degradation is hard to diagnose because nothing errors. So sample from below the top-k, typically skipping the first ten.

"How do you evaluate it?" Recall at 100 or 1000, not NDCG at 10. It is a retrieval model and its job is to get the right items into the candidate set, not to order them. Evaluating it with a ranking metric measures the downstream reranker's job and will mislead you about whether retrieval is the problem. The reason to be strict about this is that retrieval recall is the ceiling on the whole funnel.

"What actually breaks in production?" Feature-store parity, and the symptom is "the model was much better offline". Three distinct causes: point-in-time leakage, where training uses a feature's value as of today to predict a click from three months ago, so the feature contains the future. Implementation skew, where training computes a feature in SQL and serving computes it in Python with different null and timezone semantics. And missingness mismatch, where training data is backfilled and complete while production has timeouts and cold users, so a model that never saw a null produces garbage rather than a degraded score.

"How would you detect that?" Log the exact feature vector used at serving time, sampled, and compare distributions against the training reference daily. Population stability index above 0.1 is worth investigating and above 0.2 is an alert rather than a dashboard. And check null rates separately, because a broken join or a timed-out lookup shows up in the null rate before it shows up in the distribution of the values that did arrive. That is usually the fastest signal.

"What happens when you retrain the model?" Every item vector is in a different space, so old and new vectors are not comparable and a partial rollout produces meaningless scores with no error anywhere. So a model change requires a full re-embed into a new index and an atomic alias swap, with the query tower version changing at the same instant. That coupling is worth enforcing deliberately, because the two towers usually live in different services that deploy independently by default.

"Would you use this instead of BM25?" No, alongside it. They fail on disjoint query populations: dense retrieval cannot distinguish "iPhone 15 Pro 256GB" from the 128GB variant, because they are nearly identical in embedding space and are different products, and BM25 cannot match a paraphrase. A real catalogue has large volumes of both query types, so the answer is hybrid with rank-based fusion, and treating two-tower as a replacement is how you get a system that is worse on the queries that convert.

Common misconceptions

"It's a ranking model." It is retrieval. Evaluate it with recall, not NDCG, and expect a reranker downstream.

"In-batch negatives are enough." They are easy negatives, and without the logQ correction they actively teach an anti-popularity bias.

"Both towers should be the same size." The query tower runs per request and the item tower runs as a batch job. Asymmetric sizing is a legitimate and underused choice.

"Feature stores are a convenience." They exist for point-in-time correctness and one definition across both paths, which are correctness properties.

"You can roll out a new model gradually." Mixing vectors from two model versions in one index is meaningless. Full re-embed, atomic swap, both towers together.

Interview delivery note

Explain the asymmetry first, because it justifies the whole architecture: "Two towers with no interaction between them, and that lack of interaction is the point: it's what lets you encode the catalogue offline and put the vectors in an ANN index. A model with query-item interaction can't precompute anything, so it's arithmetically impossible over millions of items. The cost is that it retrieves rather than ranks."

Give the training detail most candidates miss: "The critical detail in training is the logQ correction. In-batch negatives mean items appear as negatives in proportion to their frequency, so popular items get over-penalised and the model learns an anti-popularity bias. Subtracting the log sampling probability cancels it. Without that, retrieval is worst on head queries, which is where the volume is."

Then hard negatives, with the trap: "And hard negatives mined from the model's own index, iteratively. In-batch negatives are easy: a random catalogue item is obviously irrelevant, so the model learns to separate relevant from absurd when the real task is separating 'size 44' from 'size 42'. The trap is that the model's top unlabelled results are disproportionately unlabelled positives, so you skip the very top or you teach it that correct answers are wrong."

Volunteer the production risk, because it is what shipping one teaches: "And where these actually break is feature-store parity. Point-in-time leakage, where training uses today's feature value to predict a click from three months ago. Implementation skew, where the feature is SQL in training and Python in serving. And missingness, where training data is complete and production has timeouts, so a model that never saw a null produces garbage. I'd log serving feature vectors and alert on null-rate divergence, which is the fastest signal."

The Staff & Team Lead Interview Reference