Design multilingual semantic search at 10,000 QPS

"Design semantic search over a product catalogue in nine languages, serving 10,000 queries per second at a 150 ms p99."

Step 1: clarify (4 minutes)

Six questions, and each one changes the design. Ask them, do not assume.

"Multilingual" means which of three things? They are different systems.

(a) Query and documents in the same language, nine separate corpora
    -> nine independent indexes. Easiest. Often what is meant.
(b) Cross-lingual: a French query must match an English document
    -> one shared embedding space. Much harder, and the interesting case.
(c) Multilingual documents: one product with descriptions in 4 locales
    -> a document-representation question before it is a retrieval one.

Assume (b) with (c), because that is the version worth designing and because it is what a marketplace actually has: a seller writes in Portuguese and a buyer searches in English.

Is it pure semantic, or hybrid? Pure vector search on a product catalogue is a trap: it fails on exact identifiers (SKUs, model numbers, brand names) which are a large share of commercial search traffic. Assume hybrid, and say why.

What is the corpus size and update rate? Assume 50 million products, roughly 2 million updates per day, and a freshness requirement of under 5 minutes for price and availability, under an hour for text.

What is the p99 budget, end to end or retrieval only? Assume 150 ms end to end, which after network and serialisation leaves roughly 100 ms for retrieval and ranking.

Do results need filtering? Yes: category, price range, in-stock, seller country, shipping availability. This is the single most consequential answer in the whole design, because filtered approximate nearest neighbour search has a recall cliff, and a design that ignores it looks fine and returns wrong results in production.

Step 2: capacity math (4 minutes)

Traffic
  10,000 QPS steady, assume 3x peak headroom -> design for 30,000 QPS
  p99 budget 150 ms end to end

Corpus
  50M products x 9 locales, but NOT 450M vectors:
  one product = one multilingual embedding (that is the point of (b))
  -> 50M vectors

Vector memory
  Dense: 50M x 768 dims x 4 bytes (fp32)      = 153 GB
  Quantised to int8:                           = 38 GB
  Product-quantised (m=96, 8 bits)             = 4.8 GB + codebooks

  HNSW graph overhead: M=32 -> ~32 x 2 x 4 bytes x 50M = 12.8 GB

  So: fp32 + graph = ~166 GB   -> too big for one node comfortably
      int8 + graph = ~51 GB    -> fits a 128 GB node with room
      PQ + graph   = ~18 GB    -> fits easily, recall cost to measure

Query embedding cost
  30,000 QPS x 1 embedding each. A small multilingual encoder on GPU
  does ~2,000 queries/sec/GPU at batch 32 -> 15 GPUs, or
  ~4 ms of the latency budget with batching.
  This is a real cost and it is the part people forget.

Sharding
  int8 at 51 GB, replicated 3x for QPS and availability.
  At ~1,500 QPS per replica for HNSW at ef=128 on 16 cores,
  30,000 QPS needs ~20 serving replicas.
  Shard by 4 to keep per-shard latency down: 4 shards x 5 replicas.

The number that decides the architecture: 20 replicas of a 51 GB index. That is a real fleet, and it means the embedding model, the index build and the serving path are three separate systems with three separate scaling stories.

Step 3: the embedding decision

This is where most candidates go straight to "use a multilingual model" and stop. The depth is in what that costs.

Option A: one multilingual model, shared space
  e.g. a multilingual sentence encoder (LaBSE, multilingual-E5, BGE-M3)
  + True cross-lingual: FR query matches EN document natively
  + One index, one model to operate
  - Weaker per-language quality than a monolingual model, typically
  - The "curse of multilinguality": fixed capacity split across
    languages, so low-resource languages suffer most

Option B: nine monolingual models, nine indexes
  + Best per-language quality
  - No cross-lingual matching at all
  - Nine models to serve, version and evaluate. Operationally 9x.
  - A query in an unsupported language has nowhere to go

Option C: translate everything to English, one English model
  + Best-quality model, one index, cheap serving
  - Translation cost at index time (50M documents) and query time
  - Translation errors compound into retrieval errors, invisibly
  - Loses language-specific nuance, especially for product names

Choose A, and say the reason precisely: cross-lingual matching is a requirement (from clarification (b)), and only A provides it without paying translation latency on every query. The quality gap to monolingual is real, and the mitigation is the hybrid retrieval in step 4, where lexical matching recovers exactly the cases dense multilingual models are weakest on: brand names, model numbers and locale-specific terminology.

The detail that shows depth: for case (c), multilingual documents, do not embed each locale separately. Embed the canonical description once (usually the seller's original language) and add a lexical field per locale. Embedding nine translations of the same product creates nine near-duplicate vectors that compete for the same result slots, which measurably hurts diversity and inflates the index by 9x for no recall gain.

Step 4: hybrid retrieval

Query "chaussures de course imperméables taille 42"
    |
    +--> query understanding (2 ms)
    |      language detection, locale hints, filter extraction
    |      ("taille 42" -> size filter, not a text match)
    |
    +--> lexical retrieval (BM25, per-locale analyzers)  ~15 ms
    |      top 200
    |
    +--> dense retrieval (multilingual embedding + HNSW)  ~25 ms
    |      top 200                                   (parallel)
    |
    v
  fusion (RRF)  ~1 ms
    |
    v
  cross-encoder rerank, top 50 -> top 20   ~35 ms on GPU
    |
    v
  business ranking (availability, margin, seller quality)  ~3 ms

Reciprocal Rank Fusion combines the two lists without needing score calibration between them, which matters because BM25 scores and cosine similarities are not comparable:

def rrf(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """k=60 is the value from Cormack et al. (2009) and it is robust;
    it damps the influence of any single list's top position."""
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda kv: -kv[1])

Why hybrid rather than dense alone, stated concretely: a query for "iPhone 15 Pro Max 256GB" is an exact-match problem, and a dense encoder will happily return the 128GB variant with high similarity. A query for "something warm for hiking in winter" is a semantic problem that BM25 cannot touch. A production catalogue has both, in large volumes, and the mix is why hybrid beats either alone by a wide margin on real traffic.

Step 5: the filtering problem, and why it is the hard part

The deep dive. This is the component to volunteer.

The naive approaches both fail:

Post-filter:  retrieve top 100 by vector similarity, then drop the ones
              that fail the filter.
              With a 1% selective filter, you expect ~1 surviving result.
              Recall collapses. Silently.

Pre-filter:   compute the filtered subset, then brute-force search it.
              Correct, and O(n) in the subset. At 500k matching documents
              that is 40+ ms of pure distance computation.

The recall cliff is the thing to name: HNSW graph traversal assumes the graph is connected. Filtering removes nodes, which can disconnect regions of the graph, so the search cannot reach qualifying neighbours even though they exist. Recall does not degrade gracefully; it falls off a cliff at a filter selectivity that depends on the graph parameters.

Filter selectivity   Naive filtered-HNSW recall@10 (illustrative shape)
   50%                 ~0.95
   10%                 ~0.88
    5%                 ~0.70
    1%                 ~0.30   <- the cliff

The design that works, and it is a routing decision rather than one algorithm:

def retrieve(query_vec, filters, index) -> list[str]:
    # Cardinality estimate from the same statistics the query planner
    # would use. This is cheap and it is what makes the routing possible.
    est = estimate_matching_docs(filters)

    if est < 10_000:
        # Small enough to scan exactly. Exact is FASTER than approximate
        # here, and it is exactly right.
        return brute_force_search(query_vec, filters, top_k=200)

    if est < 0.05 * index.size:
        # In the cliff zone. Use a partitioned index if the filter is
        # a partition key, otherwise widen ef and accept the latency.
        if is_partition_key(filters):
            return search_partition(query_vec, filters, ef=256)
        return filtered_hnsw(query_vec, filters, ef=512, top_k=200)

    # Filter is unselective: ordinary filtered traversal is fine.
    return filtered_hnsw(query_vec, filters, ef=128, top_k=200)

And the structural move: partition the index by the highest-cardinality mandatory filter. For a marketplace that is usually country or category. If every query carries a country filter, build one index per country and the filter stops being a filter at all: it becomes index selection, which is free.

ACORN (Patel et al., 2024) is the current research direction worth naming: it predicates the HNSW traversal itself so the graph remains navigable under filtering, rather than filtering the results of an unaware traversal. Some vector databases have shipped variants of this idea.

Step 6: freshness, and the two update paths

Price and availability change constantly; text and embeddings rarely. Do not put them in the same pipeline.

TEXT / EMBEDDING PATH (slow, expensive)
  product text change -> CDC -> embedding service (GPU batch)
                             -> vector upsert -> index
  Latency budget: under 1 hour. Batched, cheap per item.

PRICE / STOCK PATH (fast, cheap)
  price or stock change -> CDC -> attribute store (Redis / doc values)
  Latency budget: under 5 seconds. No re-embedding, no index rebuild.
  Applied at RANKING time, not at retrieval time.

Why this split matters: re-embedding 2 million products a day because their price changed would cost more in GPU time than the entire serving fleet, and it would churn the HNSW graph continuously. Separating them means the expensive path handles roughly 50,000 genuine text changes a day and the cheap path handles the other 1.95 million.

The consequence to state: an out-of-stock product can still be retrieved and is then filtered or demoted at ranking time using fresh attributes. Retrieval works on slow-moving semantics; the fast-moving facts are applied late.

Index rebuild happens weekly, offline, into a new index, promoted by an alias swap. That is the same mechanism as zero-downtime OpenSearch reindexing, and it gives you a tested rollback: swap the alias back.

Step 7: failure modes and degradation

Embedding service down or slow
  -> Fall back to lexical-only retrieval. Quality drops measurably,
     the product still works. This is the single most important
     degradation path and it should be exercised in a game day.

Cross-encoder reranker down
  -> Serve the RRF-fused order. Ranking quality drops, latency improves.
     Circuit-break on the reranker at a p99 threshold, not on errors,
     because slow is the realistic failure.

One vector shard down
  -> Serve from the remaining shards with reduced recall, and mark the
     response degraded. Partial results beat no results for search.

Query is in an unsupported language
  -> Language detection confidence below threshold -> lexical only.

Cache
  -> Head queries are extremely skewed: typically the top 1% of queries
     are 30-50% of traffic. Cache fused results keyed by
     (normalised query, filters, locale) with a 60 s TTL.
     At a 35% hit rate this removes a third of the fleet cost.

The degradation ladder is the answer to "what happens under load": drop the reranker first (35 ms, largest single cost), then reduce ef from 128 to 64 (halves vector search time, costs a few points of recall), then fall back to lexical only. Each step is a flag, and each is reversible.

Step 8: what changes at ten times the scale

At 100,000 QPS and 500 million products:

The embedding fleet becomes the dominant cost. 100k QPS of query embedding is 50+ GPUs purely for encoding queries. The move is a smaller distilled query encoder (a 6-layer student of the 12-layer teacher, distilled to match its embedding space) plus aggressive caching of head-query embeddings, which are extremely repetitive.

Memory forces quantisation. 500M x 768 x 4 bytes is 1.5 TB, so int8 or product quantisation stops being an option and becomes the design. The pattern is a two-stage search: PQ for a coarse top-1000, then rerank those with full-precision vectors fetched from disk or a separate store.

Sharding strategy changes from "for latency" to "for capacity". At 50 million you shard to keep per-shard latency low; at 500 million you shard because it does not fit. Route by partition key where possible so a query touches one shard rather than all of them, because scatter-gather at p99 is governed by the slowest shard and that gets worse with shard count. This is the tail-at-scale problem.

Index build becomes a scheduling problem. Building HNSW over 500 million vectors is hours of compute. It becomes a Spark or Ray job, built shard-parallel, with the alias swap per shard rather than globally.

Production evidence

Pinterest's multilingual search work and Airbnb's published search-ranking architecture both describe the retrieve-then-rerank shape with a cheap first stage and an expensive cross-encoder second stage over a small candidate set, which is the latency structure this design uses.

Cormack, Clarke and Buettcher, "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods" (SIGIR 2009) is the source of RRF and of the k=60 constant, and it is the citation for using it without score calibration.

Patel et al., "ACORN: Performance-Aligned Hybrid Search" (SIGMOD 2024) is the primary source on predicate-aware graph traversal and documents the filtered-search recall cliff directly.

Malkov and Yashunin's HNSW paper (2016) supplies the M and ef parameters and the graph-connectivity assumption that filtering violates.

BGE-M3, multilingual-E5 and LaBSE are the current open multilingual encoders; BGE-M3 is notable here because it produces dense, sparse and multi-vector representations from one model, which collapses part of the hybrid architecture into a single encoder.

Conneau et al., "Unsupervised Cross-lingual Representation Learning at Scale" (2020) documents the curse of multilinguality: fixed model capacity divided across languages, with low-resource languages degrading most, which is the quality cost of choosing option A.

The debate

The case for dense-only: simpler, one index, one model, and modern multilingual encoders handle a lot of what BM25 used to be needed for. Every additional retrieval arm is another system to tune, monitor and keep consistent.

The case for hybrid: exact-match queries are a large fraction of commercial search traffic and dense retrieval is structurally bad at them, because "iPhone 15 Pro 256GB" and "iPhone 15 Pro 128GB" are nearly identical in embedding space and completely different products. The failure is not a small quality regression; it is returning the wrong item.

The case for per-language indexes over one multilingual model: better quality per language, and each language's index can be sized and tuned to its actual traffic, which matters when one locale is 60 percent of queries.

My position: one multilingual dense model plus per-locale BM25, fused with RRF, and partition the index by the mandatory filter. The multilingual model is chosen because cross-lingual matching was a stated requirement and it is the only option that provides it without per-query translation latency. The lexical arm is chosen because it recovers precisely what multilingual dense models are worst at (brand names, model numbers, locale-specific terms), so the two are complementary rather than redundant. RRF is chosen because it needs no score calibration between arms, which is one fewer thing to maintain as either arm changes.

The position I hold most firmly is on filtering: I would design the filter routing before designing the retrieval, because filtered ANN has a recall cliff that does not announce itself. A system that returns 3 results instead of 200 for a selective filter looks like a sparse catalogue rather than like a bug, and teams ship it for months. The routing rule (exact scan under 10,000 candidates, partitioned index where the filter is a partition key, widened ef otherwise) is not sophisticated, and it is the difference between a system that works and one that quietly does not.

What I would push back on is separating the freshness paths being treated as an optimisation. It is structural: re-embedding on price change would cost more in GPU time than the entire serving fleet, and it would churn the graph continuously.

Follow-up Q&A

"Why hybrid rather than dense-only?" Because a product catalogue has two query populations. "Something warm for hiking in winter" is semantic and BM25 cannot touch it. "iPhone 15 Pro Max 256GB" is exact-match, and a dense encoder puts the 128GB variant at near-identical similarity, so it returns the wrong product with confidence. Both populations are large in commercial traffic. RRF fuses them without needing the two score distributions to be comparable, which is why it is the fusion method rather than a weighted sum.

"What breaks when you add filters to vector search?" Recall, and it breaks off a cliff rather than degrading. HNSW traversal assumes a connected graph; filtering removes nodes and can disconnect regions, so the search cannot reach qualifying neighbours that exist. At around 1 percent selectivity, naive filtered search can lose most of its recall. And it is silent: you get fewer, worse results and the system reports success. The design answer is routing by estimated cardinality: exact scan under about 10,000 candidates, because exact is genuinely faster there; a partitioned index where the filter is a partition key; widened ef otherwise.

"How do you handle a French query matching an English document?" One multilingual encoder producing a shared embedding space, which is the only option that does it without translating on the query path. The cost is the curse of multilinguality: fixed capacity split across languages, so per-language quality is below a monolingual model and low-resource languages suffer most. The mitigation is the lexical arm, which is per-locale and recovers the brand names and model numbers where the multilingual model is weakest.

"A product has descriptions in nine locales. Nine vectors?" No. Embed the canonical description once and keep a lexical field per locale. Nine embeddings of the same product are near-duplicates that compete for the same result slots, which hurts result diversity and inflates the index ninefold for no recall gain. The locale-specific signal belongs in the lexical arm, where it is cheap.

"Prices change two million times a day. Do you re-embed?" No, and this is the split that makes the system affordable. Two paths: text and embedding changes go through the GPU pipeline with an hour of latency, which is about 50,000 genuine changes a day; price and stock go to an attribute store with five-second latency and are applied at ranking time, not retrieval. Re-embedding on price change would cost more GPU time than the entire serving fleet and would churn the HNSW graph continuously.

"What's your degradation ladder under load?" Three steps, each a flag. Drop the cross-encoder reranker first, because it is 35 milliseconds of a roughly 80 millisecond budget and the RRF order is already decent. Then reduce ef from 128 to 64, which roughly halves vector search time for a few points of recall. Then fall back to lexical-only, which is a real quality drop and a working product. The embedding service failing should trigger the last one automatically, and that path is worth exercising in a game day, because it is the one that has never run.

"How do you evaluate this?" Two families, and reporting only one is the common mistake. Offline: recall@k and NDCG against a labelled set, plus an explicit retrieval recall measurement separate from ranking quality, because a reranker cannot fix what retrieval never returned. Online: interleaving rather than an A/B test for ranking changes, because interleaving needs far less traffic to reach significance and controls for the position bias that makes click-through comparisons unreliable. And I would segment every metric by locale, because an aggregate improvement that regresses one language is common and invisible in the total.

"Where does the latency actually go?" Roughly: 2 ms query understanding, 25 ms vector search, 15 ms lexical in parallel with it, 1 ms fusion, 35 ms cross-encoder rerank, 3 ms business ranking. The reranker is the largest single item, which is why it is the first thing dropped under load. The query embedding is 4 ms if batched and considerably more if not, which is a common and avoidable mistake: batching query embeddings across concurrent requests is worth doing even at the cost of a couple of milliseconds of queueing delay.

Common misconceptions

"Semantic search replaces keyword search." It complements it. Exact-identifier queries are a large share of commercial traffic and dense retrieval is structurally bad at them.

"Filters are cheap." Filtered ANN has a recall cliff. Filters are the hardest part of this design, not a detail bolted on at the end.

"One vector per locale." Near-duplicate vectors compete for result slots and inflate the index. One canonical embedding plus per-locale lexical fields.

"Recall@k measures the system." It measures retrieval. Report retrieval recall separately from ranking quality, because a reranker cannot recover a document that was never retrieved.

"The embedding model is the whole design." The embedding fleet, the freshness split and the filter routing are each as consequential, and at 10x scale the embedding fleet becomes the dominant cost rather than the index.

Interview delivery note

Clarify the word "multilingual" first, because it is genuinely ambiguous and the distinction changes the whole design: "Before I design anything: does multilingual mean nine separate corpora with same-language queries, or cross-lingual where a French query should match an English document? Those are different systems. I'll assume cross-lingual, because it's the harder and more useful one."

Then commit early to hybrid with a concrete reason: "I'd go hybrid rather than dense-only, and the reason is specific. A query for 'iPhone 15 Pro Max 256GB' is exact match, and a dense encoder puts the 128GB variant at nearly identical similarity, so it confidently returns the wrong product. That's not a quality regression, it's a wrong answer, and those queries are a large share of commercial traffic."

Volunteer the filtering problem before being asked, because it is the depth signal here: "The hard part of this isn't the embedding, it's filtering. Filtered HNSW has a recall cliff, because the graph traversal assumes connectivity and filtering disconnects it. At about one percent selectivity you can lose most of your recall, and it's silent: you get fewer results and the system reports success. So I'd route by estimated cardinality: exact scan under ten thousand candidates, because exact is actually faster there; partitioned index where the filter is a partition key; widened ef otherwise."

The line that shows operational experience: "and I'd split the freshness paths. Text and embeddings go through the GPU pipeline with an hour of latency; price and stock go to an attribute store in five seconds and get applied at ranking time. Re-embedding two million products a day because their price changed would cost more GPU than the whole serving fleet."

The Staff & Team Lead Interview Reference