Multilingual index topology

What it is

The decision about how documents in many languages are physically organised across indexes, shards and analyzers, and what that implies for query routing and scoring.

Three topologies, and the choice has consequences well beyond storage layout:

ONE INDEX, ONE ANALYZER
  All languages in one index with a language-agnostic analyzer
  (typically standard tokenisation, no stemming).
  + Simple. One index, one query, cross-language matching for
    shared tokens (brand names, product codes).
  - No stemming means "running" and "run" do not match in ANY
    language. Relevance is materially worse for morphologically
    rich languages.

ONE INDEX, PER-LANGUAGE FIELDS
  One document, fields like title_en, title_fr, title_de, each
  with its own analyzer.
  + Correct analysis per language, one document, one index.
  - Field explosion: 9 languages x 5 fields = 45 fields per doc.
    Query must know which fields to search.
  - IDF is computed per field, which is correct, but sparse
    fields have unstable IDF.

ONE INDEX PER LANGUAGE
  index_en, index_fr, index_de, each with its own analyzer,
  mapping and shard count.
  + Correct analysis, correct per-language IDF, independent
    scaling and tuning per language.
  - Routing: which index(es) does a query search?
  - Cross-language queries need explicit fan-out.
  - Operational multiplication: 9 indexes to manage, alias,
    reindex and monitor.

Commonly confused with the embedding decision, which is separate and orthogonal: per-language indexes can still share one multilingual embedding space, and one index can hold per-language embeddings. The lexical topology and the vector topology are two decisions, and conflating them produces a design where neither is right.

The problem it solves

Analysis is language-specific and getting it wrong is a large relevance loss, larger than most ranking-model changes.

GERMAN         Compound words. "Donaudampfschifffahrtsgesellschaft"
               must decompose or a search for "Dampfschiff" fails.
               Needs a decompounder with a dictionary.

FINNISH /      Extremely rich morphology. Finnish nouns have ~15
TURKISH        cases; a single lemma has dozens of surface forms.
               Without stemming, recall collapses.

ARABIC         Root-and-pattern morphology, optional diacritics,
               and several orthographic variants of the same
               letter. Needs normalisation plus a light stemmer.

CHINESE /      No whitespace between words. Requires segmentation
JAPANESE       (or n-grams). A wrong segmentation produces wrong
               terms and the failure is silent.

JAPANESE       Three scripts (kanji, hiragana, katakana) plus
               romaji. The same word appears in several scripts
               and must normalise to one form.

ENGLISH        Comparatively simple, which is why systems designed
               English-first get everything else wrong.

A shared analyzer is correct for none of these, and the cost is not subtle: for morphologically rich languages, no stemming can cut recall substantially on the queries that matter.

Mechanics

The per-language index, with routing

PUT /products_de
{
  "settings": {
    "analysis": {
      "filter": {
        "de_decompound": {
          "type": "hyphenation_decompounder",
          "word_list_path": "analysis/de_dictionary.txt",
          "hyphenation_patterns_path": "analysis/de_DR.xml",
          "only_longest_match": true
        },
        "de_stem": { "type": "stemmer", "language": "light_german" }
      },
      "analyzer": {
        "de_text": {
          "tokenizer": "standard",
          "filter": ["lowercase", "de_decompound", "de_stem"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title":  { "type": "text", "analyzer": "de_text" },
      "body":   { "type": "text", "analyzer": "de_text" },
      "vector": { "type": "dense_vector", "dims": 1024,
                  "index": true, "similarity": "cosine" }
    }
  }
}

Note the vector field lives in the same index. That is the hybrid arrangement: the lexical side is per-language and the vector side uses a shared multilingual embedding space, so the vectors in products_de and products_fr are comparable.

Routing: the decision that makes or breaks it

def route(query: str, user_locale: str, content_prefs: list[str]) -> list[str]:
    detected, confidence = detect_language(query)

    # Short queries are unreliable to detect. "Paris" is French,
    # English, German and Spanish. Below a confidence threshold,
    # do NOT trust detection.
    if confidence < 0.7 or len(query.split()) < 2:
        # Fall back to the user's declared locale, which is far
        # more reliable than detecting two words.
        primary = user_locale
    else:
        primary = detected

    indexes = [f"products_{primary}"]

    # Cross-language: search the user's other declared languages
    # too. A bilingual user in Montreal searching in French may
    # want English results, and only they can tell us that.
    for lang in content_prefs:
        if lang != primary:
            indexes.append(f"products_{lang}")

    return indexes

Language detection on short queries is the practical failure mode, and it is worth naming because it is where the topology breaks in production. Detection accuracy on a two-word query is poor: "Paris hotel" is ambiguous across several languages, product names are language-neutral, and code-switching is common in bilingual populations. The user's declared locale is a far stronger signal than detection on a short string, and the design should treat detection as a hint that supplements it rather than a decision that overrides it.

The scoring problem in multi-index search

This is the subtle failure and the one an interviewer is likely to probe.

Search products_en AND products_fr, merge by score.

BM25 IDF is computed PER INDEX:
  "smartphone" in products_en: appears in 40,000 of 2,000,000 docs
                               IDF = ln(2e6/4e4 + 1) = 3.93
  "smartphone" in products_fr: appears in    800 of   120,000 docs
                               IDF = ln(1.2e5/800 + 1) = 5.02

The SAME term is worth 28% more in the French index, purely
because that index is smaller. So French results score higher
for reasons that have nothing to do with relevance.

Three responses:

1. dfs_query_then_fetch (Elasticsearch)
   Gathers global term statistics across shards before scoring.
   Fixes intra-index shard variance; does NOT fix cross-INDEX
   variance, and it costs an extra round trip.

2. Normalise scores per index before merging
   Min-max or z-score per index, then merge. Crude, and it
   discards the absolute signal.

3. Rank-based fusion (RRF)
   Merge by RANK position, not score. Immune to score
   incomparability by construction.
   *** This is the right answer, and it is the same argument
   as fusing lexical with vector results. ***

RRF is the answer for the same reason it is the answer in hybrid retrieval: BM25 scores are not comparable across queries, corpora or indexes, and a fusion method that never compares them cannot be broken by the incomparability.

Where the documents themselves are multilingual

A single product with descriptions in nine locales is a different problem from nine documents in nine languages.

OPTION A: one document per (product, locale)
  9 documents per product, each in its own language index.
  + Clean analysis, clean routing.
  - 9x the document count. Deduplication needed at result time
    so a user does not see the same product nine times.
  - Popularity and interaction signals are split across nine
    documents and must be aggregated back to the product.

OPTION B: one document, per-locale fields
  One document in one index, fields title_en, title_fr, ...
  + One document, so signals aggregate naturally and there is
    no dedup problem.
  - Field explosion, and the query must know which fields.
  - Every document carries every locale's fields, so sparse
    locales bloat the index.

OPTION C: one document, one canonical text, one shared vector
  Index the canonical description lexically in its source
  language, plus ONE multilingual embedding, plus a small
  per-locale keyword field for names.
  + One vector per product, no near-duplicate vectors competing
    for result slots.
  + Cross-lingual matching comes from the shared embedding.
  - Lexical recall in non-canonical languages depends on the
    keyword field being good.

Option C is what I would default to for a marketplace, and the reason is the vector side: embedding nine translations of one product creates nine near-duplicate vectors that compete for the same result slots, which measurably hurts diversity and inflates the index ninefold for no recall gain. The cross-lingual matching should come from the shared embedding space, not from indexing every translation.

Sharding and the size disparity

Realistic locale distribution for a global marketplace:

locale    documents      queries/sec
en        12,000,000        4,200
de         3,400,000          900
fr         2,100,000          640
es         1,900,000          580
pt-BR      1,200,000          410
ja           900,000          380
nl           400,000          110
sv           240,000           70
fi           110,000           35

Per-language indexes let each be sized independently, which is the main operational argument for the topology.

products_en:  12 shards (1M docs each, ~20-40 GB target)
products_de:   3 shards
products_fi:   1 shard

Contrast with one shared index at 12 shards: every query,
including a Finnish one over 110,000 documents, fans out to 12
shards and waits for the slowest. That is the tail-at-scale
problem imposed on a query that could have touched one shard.

Over-sharding small locales is a specific and common mistake. A 110,000-document index on six shards has 18,000 documents per shard, so each shard's work is trivial and the query cost is dominated by scatter-gather coordination. One shard is correct, and the general rule is a target shard size of roughly 20 to 40 GB rather than a fixed shard count.

Production evidence

Elasticsearch's and OpenSearch's language analyzer set ships around 30 language-specific analyzers with different stemmer, stopword and normalisation chains, which is direct evidence that a shared analyzer is not adequate. The German decompounder and the Japanese kuromoji and Chinese smartcn plugins exist because those languages cannot be tokenised by whitespace at all.

Elasticsearch's dfs_query_then_fetch exists specifically because per-shard IDF produces inconsistent scoring, and its documented cost (an extra round trip to gather global term statistics) is the reason it is not the default.

Cormack, Clarke and Buettcher's RRF (SIGIR 2009) is the fusion method, and its property of requiring no score calibration is exactly what makes it correct for merging results across indexes with different corpus statistics.

Multilingual embedding models (LaBSE, multilingual-E5, BGE-M3) place all languages in one vector space, which is what makes option C viable, and Conneau et al.'s work on the "curse of multilinguality" documents the quality cost: fixed model capacity divided across languages, with low-resource languages degrading most.

Elasticsearch's shard-sizing guidance recommends a target of tens of gigabytes per shard rather than a fixed count, which is the basis for sizing each locale index independently.

The debate

The case for one shared index: operational simplicity. One index to manage, alias, reindex and monitor; no routing logic; cross-language matching for free on shared tokens. For a system where most content is in one language and the others are a small tail, the complexity of nine indexes is not repaid.

The case for per-language indexes: correct analysis, correct per-language IDF, independent shard sizing and independent tuning. Analysis quality is a larger relevance lever than most ranking changes, particularly for morphologically rich languages, and a shared analyzer forfeits it entirely.

The case for per-language fields in one index: correct analysis without index multiplication, and signals aggregate naturally to one document.

My position: per-language indexes for the lexical side, one shared multilingual embedding space for the vector side, and RRF to fuse.

The lexical side goes per-index because analysis is where the relevance is, and because per-language indexes let you size, tune and reindex each locale independently, which matters when English is 12 million documents and Finnish is 110,000. Forcing both through the same shard layout means the Finnish query pays a twelve-way scatter-gather for a corpus that fits on one shard.

The vector side stays shared because that is what delivers cross-lingual matching, and because per-locale embeddings of the same product create near-duplicate vectors that compete for result slots. One product, one vector, is the rule I would hold.

RRF for fusion, and the argument is specific rather than aesthetic: BM25 IDF is computed per index, so the same term is worth more in a smaller index purely because that index is smaller. Merging by score therefore systematically favours results from smaller-language indexes. Merging by rank is immune to that by construction, and it is the same reason RRF is correct for hybrid lexical-plus-vector fusion.

Two things I would insist on that are easy to get wrong. Do not trust language detection on short queries: a two-word query is genuinely ambiguous across languages, product names are language-neutral, and code-switching is normal in bilingual populations, so the user's declared locale is the stronger signal and detection supplements it. And do not over-shard small locales: a 110,000-document index on six shards is dominated by scatter-gather coordination, and one shard is correct.

Where I would push back on the premise: if the product is 90 percent one language, do the simple thing. Per-language indexes are the right answer when several locales are substantial, and they are over-engineering when one language dominates and the rest is a long tail that a shared analyzer serves adequately.

Follow-up Q&A

"Which topology would you choose?" Per-language indexes for the lexical side, because analysis is language-specific and it is a bigger relevance lever than most ranking changes, and because it lets each locale be sharded and tuned independently when English is twelve million documents and Finnish is a hundred and ten thousand. One shared multilingual embedding space for the vector side, because that is what gives cross-lingual matching. Then fuse with reciprocal rank fusion. Those are two separate decisions and conflating them produces a design where neither is right.

"Why not one index with a shared analyzer?" Because it forfeits stemming, and for morphologically rich languages that is a large recall loss. Finnish nouns have around fifteen cases, German compounds have to be decomposed or a search for a component fails, and Chinese and Japanese have no whitespace so they need segmentation before there are terms at all. A shared analyzer is correct for none of those, and English-first systems get this wrong because English is the language where it hurts least.

"What breaks when you search multiple indexes and merge?" Scoring, subtly. BM25's IDF is computed per index, so a term appearing in forty thousand of two million English documents has a lower IDF than the same term in eight hundred of a hundred and twenty thousand French documents, by about 28 percent in the example I would work. So French results score higher purely because the French index is smaller, which has nothing to do with relevance. Merging by rank rather than by score, with RRF, is immune to that by construction.

"Doesn't dfs_query_then_fetch fix that?" Only partly. It gathers global term statistics across shards before scoring, which fixes intra-index shard variance, and it does not fix cross-index variance because the indexes have genuinely different corpora. It also costs an extra round trip, which is why it is not the default. RRF is the more robust answer and it costs nothing.

"How do you decide which index to search?" The user's declared locale first, with language detection as a supplement rather than an override. Detection on short queries is unreliable: "Paris hotel" is ambiguous across several languages, product names are language-neutral, and bilingual users code-switch mid-session. So below a confidence threshold, or under about three words, I would trust the locale. And for users with several declared content languages, fan out to those indexes too, because a bilingual user in Montreal may want both French and English results and only they can tell us that.

"A product has descriptions in nine locales. How many documents and how many vectors?" One vector, definitely. Embedding nine translations creates nine near-duplicate vectors that compete for the same result slots, hurts diversity, and inflates the index ninefold for no recall gain. Cross-lingual matching should come from the shared embedding space rather than from indexing every translation. For the lexical side I would index the canonical description in its source language plus a per-locale keyword field for names and terms, and accept that lexical recall in non-canonical languages leans on that field.

"How do you shard this?" By target shard size, roughly twenty to forty gigabytes, not by a fixed count. Twelve shards for English at twelve million documents, three for German, one for Finnish. The mistake I would call out is over-sharding small locales: a hundred and ten thousand documents on six shards is eighteen thousand per shard, so each shard's work is trivial and the query cost is dominated by scatter-gather coordination. That is imposing the tail-at-scale problem on a query that could touch one shard.

"When is per-language over-engineering?" When one language is ninety percent of the corpus and traffic, and the rest is a long tail. Then the operational cost of nine indexes to manage, alias, reindex and monitor is not repaid by better analysis on a small fraction of queries, and per-language fields in one index gets most of the analysis benefit with none of the routing and fusion complexity. I would ask for the locale distribution before choosing.

Common misconceptions

"Multilingual is a tokenisation setting." It is a topology decision with routing, scoring and sharding consequences, and the analyzer is one part of it.

"Language detection solves routing." Detection on short queries is unreliable, and the user's declared locale is a stronger signal.

"Merging results from several indexes is just a sort." IDF is per index, so scores are not comparable and merging by score systematically favours smaller indexes.

"Index every translation as its own vector." Nine near-duplicate vectors compete for result slots and inflate the index for no recall gain. One product, one vector.

"More shards is safer." Over-sharding a small locale makes scatter-gather coordination the dominant cost.

Interview delivery note

Separate the two decisions immediately, because most candidates make them as one: "There are two topology questions here and they're orthogonal. The lexical topology, which is about analysis and IDF, and the vector topology, which is about whether the embedding space is shared. I'd go per-language indexes for the lexical side and one shared multilingual embedding space for the vector side."

Justify the lexical side with the analysis cost: "Per-language because analysis is language-specific and it's a bigger relevance lever than most ranking changes. Finnish nouns have about fifteen cases, German compounds have to be decomposed or a search for a component fails, and Chinese has no whitespace so you need segmentation before you have terms at all. A shared analyzer is correct for none of those."

Volunteer the scoring failure, because it is the subtle one: "And the thing that breaks when you search multiple indexes is scoring. BM25's IDF is per index, so the same term is worth about twenty-eight percent more in a small French index than a large English one, purely because the index is smaller. Merging by score systematically favours the smaller language. So fuse by rank with RRF, which is immune to it by construction."

Two practical signals worth including: "I wouldn't trust language detection on short queries, because two words are genuinely ambiguous and product names are language-neutral, so the user's declared locale is the stronger signal and detection supplements it." And: "and one vector per product, not one per locale. Nine translations of the same product are nine near-duplicate vectors competing for the same result slots."

The Staff & Team Lead Interview Reference