Analyzers per language: stemming, lemmatisation, CJK, transliteration
What it is
The pipeline that turns a string into the terms stored in the index, and the same pipeline applied to the query. If they disagree, nothing matches, and that is the single most common cause of "why does this obviously-relevant document not come back".
THE PIPELINE, in order
CHARACTER FILTERS before tokenisation. Strip HTML, map
characters, normalise unicode forms.
TOKENISER string -> tokens. This is the
language-specific decision that matters
most, because CJK has no whitespace.
TOKEN FILTERS lowercase, stopwords, stemming,
synonyms, ASCII folding, decompounding.
Order matters and is a frequent bug.
Commonly confused with a configuration detail. Analysis is a bigger relevance lever than most ranking changes for morphologically rich languages, because a term that is never produced can never be matched at any score.
Also commonly confused: stemming and lemmatisation are different operations with different failure modes. Stemming chops algorithmically and produces non-words; lemmatisation maps to a dictionary form using part-of-speech context. Stemming is fast and crude; lemmatisation is slower and correct.
The problem it solves
Users do not type the form that is in the document.
document: "running shoes for marathon runners"
query: "run shoe marathon"
WITHOUT ANALYSIS: zero matches. Every term differs.
WITH STEMMING: run/run, shoe/shoe, marathon/marathon.
All three match.
And the failure gets much worse as morphology gets richer:
ENGLISH run, runs, running, ran -> ~4 forms
GERMAN compounds: Donaudampfschifffahrt -> unbounded
FINNISH ~15 noun cases x number x
possessive suffixes -> dozens
TURKISH agglutinative: a single stem takes
many suffixes in sequence -> very many
ARABIC root-and-pattern, plus optional
diacritics and orthographic
variants -> many
For Finnish or Turkish, no stemming is not a small quality loss, it is most of your recall, and that is the argument for per-language analysis rather than a shared analyzer.
Mechanics
Stemming versus lemmatisation
STEMMING (algorithmic, rule-based)
running -> run
better -> better (misses: lemma is "good")
studies -> studi (a non-word, and that is fine
as long as the query stems the
same way)
universe, university -> univers (OVER-STEMMING: two
distinct concepts collapse)
+ Fast, no dictionary, no model.
- Over-stems (merges distinct words) and under-stems
(misses irregular forms).
LEMMATISATION (dictionary + part-of-speech)
running -> run
better -> good (correct)
studies -> study (a real word)
saw -> see OR saw (needs POS: verb or noun)
+ Correct forms, handles irregulars.
- Needs a dictionary and often a POS tagger, so it is
slower and it is a model to maintain.
The practical answer for most systems: light stemming, because aggressive stemming's over-merging costs precision and the marginal recall is small.
Elasticsearch's German options illustrate the spectrum:
german the classic Snowball stemmer, aggressive
light_german less aggressive, fewer over-stems
minimal_german plural handling only
For a product catalogue, light or minimal is usually
better, because over-stemming merges product terms that
users distinguish.
And the technique that captures most of stemming's benefit without its cost: index both. Index the unstemmed form in one field and the stemmed form in another, search both, and boost the exact field. Exact matches outrank stemmed ones and recall is preserved, which is strictly better than choosing.
CJK: there is no whitespace
The hardest case, and the one that breaks systems designed English-first, because tokenisation is not a filter step, it is the whole problem.
CHINESE "北京大学生"
Could segment as:
北京 / 大学生 (Beijing / university student)
北京大学 / 生 (Peking University / student)
Different segmentations, different meanings, and
the wrong one produces terms that never match.
JAPANESE Three scripts in one sentence, plus no spaces:
"東京タワーに行きました"
kanji + katakana + hiragana, and the same word
appears in several scripts.
KOREAN Spaces exist, and agglutination means a "word"
carries several morphemes.
TWO APPROACHES
DICTIONARY-BASED SEGMENTATION
kuromoji (Japanese), smartcn or IK (Chinese), nori
(Korean). A morphological analyser with a dictionary.
+ Correct terms, so precision is good.
- Out-of-vocabulary words segment badly, which for
product names and brands is exactly the words that
matter.
N-GRAMS (bigrams, typically)
"北京大学生" -> 北京, 京大, 大学, 学生
+ No dictionary, no OOV problem, robust.
- Index size grows substantially and precision drops
because meaningless bigrams match.
THE ANSWER MOST PRODUCTION SYSTEMS USE: BOTH.
A dictionary analyser as the primary field, with a
user-dictionary for domain terms, plus an n-gram field
as a fallback for recall. Search both, boost the
dictionary field.
The user dictionary is the operationally important part for a product catalogue: brand and product names are precisely the out-of-vocabulary terms a general dictionary segments wrongly, and adding them is a data pipeline rather than a one-time configuration.
German compounds
"Donaudampfschifffahrtsgesellschaft"
A search for "Dampfschiff" finds nothing without
decompounding.
TWO DECOMPOUNDERS
dictionary_decompounder needs a word list; splits
greedily and produces
spurious splits
hyphenation_decompounder uses hyphenation patterns
plus a word list; better,
and it is what to use
only_longest_match: true important, or you get every
sub-split and the index
bloats
And the failure to watch for: over-splitting produces terms that change meaning. Splitting a brand name into its components makes it match unrelated documents, so a protected-words list is part of a working configuration.
Arabic, and normalisation before anything else
CHARACTER-LEVEL PROBLEMS THAT MUST BE FIXED FIRST
Diacritics (harakat) are optional in writing, so the
same word appears with and without them.
-> arabic_normalization strips them.
Several orthographic variants of the same letter:
alef with and without hamza, ta marbuta vs ha,
alef maksura vs ya.
-> normalisation maps them to one form.
THEN light stemming: arabic_stem or the light_arabic
stemmer, which strips common prefixes and suffixes
without attempting full root extraction.
The ordering is the point: normalise, then stem. Stemming unnormalised text produces different stems for the same word written two ways, which is the whole problem restated.
Transliteration and folding
ASCII FOLDING
café -> cafe, naïve -> naive, Müller -> Muller
So a user typing without accents finds accented
documents.
THE TRAP: in German, ü folds to u, and the correct
transliteration is ue. "Müller" should match both
"Muller" and "Mueller", and plain ASCII folding gives
only the first.
-> german_normalization handles ue/oe/ae correctly, and
it must come BEFORE ascii_folding.
PRESERVE THE ORIGINAL
asciifolding with preserve_original: true indexes both
forms, so an exact accented match can still outrank a
folded one.
TRANSLITERATION ACROSS SCRIPTS
Cyrillic, Greek, Arabic and CJK romanisation, for users
typing a name in the Latin alphabet. Standards differ
(there are several romanisation systems for Japanese and
for Chinese), so pick one and apply it identically at
index and query time, which is the recurring rule.
Token filter order, which is where the bugs are
WRONG
[stemmer, stopwords, lowercase]
Stemming before lowercasing means "Running" and "running"
stem differently. Stopword removal after stemming means
the stopword list has to be in stemmed form, which nobody
remembers.
RIGHT
[char filters]
html_strip
language-specific character normalisation
[tokeniser]
[token filters]
lowercase
stopwords
synonyms <- before stemming, so synonym terms
get stemmed too
decompounder <- German
stemmer
ascii_folding <- last, with preserve_original if
exact matching matters
Synonyms before stemming is the ordering people get wrong: if synonyms are expanded after stemming, the synonym's own terms are never stemmed and will not match stemmed document terms.
The rule that governs everything
INDEX-TIME AND QUERY-TIME ANALYSIS MUST AGREE.
If the document indexed "run" and the query produces
"running", there is no match at any score.
THE EXCEPTIONS, which are deliberate
SYNONYMS at query time only, so adding a synonym does
not require a reindex. Costs query expansion time.
DECOMPOUNDING at index time only, in some
configurations.
Both are legitimate and both must be understood, because
an accidental mismatch is invisible: the query returns
nothing and looks like a data problem.
_analyze is the diagnostic and it is the first thing to reach for:
GET /products/_analyze
{ "field": "title", "text": "laufende Schuhe" }
-> shows exactly which terms were produced.
Run it on the document text and on the query text and
compare. Most "why doesn't this match" questions are
answered in thirty seconds this way.
A worked example: recall collapse in one locale
SYMPTOM
A marketplace's German locale had a null-result rate of
14 percent against 3 percent for English. Same catalogue
shape, same ranking, same query volume per item.
DIAGNOSIS (twenty minutes with _analyze)
The German index used the `standard` analyzer, because
the mapping had been copied from the English index and
only the field names changed.
Query "Dampfschiff" against a document containing
"Donaudampfschifffahrt": no match, because no
decompounding.
Query "Schuhe" against "Schuh": no match, because no
stemming.
Query "Muller" against "Müller": no match, because no
normalisation.
Three separate failures, all invisible in the ranking
metrics, because the documents never entered the
candidate set at all.
THE FIX
A proper German analyzer:
char: html_strip
tokeniser: standard
filters: lowercase, german_normalization,
hyphenation_decompounder (only_longest_match),
german_stopwords, light_german stemmer
Plus a `.exact` sub-field with minimal analysis, searched
alongside and boosted, so exact matches still outrank
stemmed ones.
RESULT
Null-result rate 14% -> 3.4%, in line with English.
NDCG@10 on the German judged set rose substantially, and
none of it was a ranking change.
THE LESSON
The relevance work planned for that quarter was a ranking
model. The actual problem was that a mapping had been
copied and the analyzer never changed, and no ranking
model can score a document that retrieval never returned.
Production evidence
Elasticsearch and OpenSearch ship around thirty language-specific analyzers with different
stemmer, stopword and normalisation chains, which is direct evidence that a shared analyzer is not
adequate. The German decompounder and the kuromoji, smartcn, IK and nori plugins exist
because those languages cannot be tokenised by whitespace at all.
The Snowball stemmer family (Porter's successor project) provides the algorithmic stemmers for most European languages, and its documentation of the light versus aggressive variants is the basis for the light-stemming recommendation.
Lucene's ASCIIFoldingFilter and the language-specific normalisation filters are separate
components precisely because plain folding is wrong for German, and the ordering requirement
(german_normalization before asciifolding) is documented rather than folklore.
Kuromoji's user dictionary support exists because out-of-vocabulary segmentation is the known weakness of dictionary-based CJK analysis, and product and brand names are systematically out-of-vocabulary.
Elasticsearch's _analyze API is documented as the primary debugging tool for exactly the
class of problem in the worked example, and the fact that it exists as a first-class endpoint
reflects how common the index/query mismatch failure is.
The debate
The case for per-language analyzers: analysis determines what can be matched at all, so it caps every downstream ranking improvement. For morphologically rich languages a shared analyzer forfeits most of the recall, and no reranker recovers a document that was never a candidate.
The case for a shared analyzer: one configuration, one index, no routing, and cross-language matching on shared tokens like brand names and product codes. For a corpus that is 90 percent one language, the operational simplicity may be worth the tail's quality.
The case for dense retrieval instead: a multilingual embedding model handles morphology implicitly, so the analyzer question goes away. It also fails on exact identifiers, which is why the lexical arm exists at all.
My position: per-language analyzers with light stemming, an exact sub-field searched alongside,
and _analyze as the first diagnostic for any relevance complaint.
The reason to prioritise analysis over ranking is that it is a ceiling rather than a score: a term that is never produced cannot be matched at any score, so an analyzer bug is invisible in every ranking metric while capping all of them. In the worked example the quarter's planned relevance work was a ranking model, and the actual problem was a copied mapping.
On stemming aggressiveness I would take light rather than classic, because over-stemming merges
words users distinguish (universe and university both becoming univers is the textbook case)
and the marginal recall from aggressive stemming is small. And index both forms: an unstemmed
.exact sub-field searched alongside and boosted gives recall from the stemmed field and precision
from the exact one, which is strictly better than choosing between them.
For CJK the answer is also both: a dictionary analyser with a user dictionary as primary, plus an n-gram field for recall. The user dictionary is the operationally important half for a catalogue, because brand and product names are systematically out-of-vocabulary and those are exactly the queries that convert.
The rule I would state as absolute is that index-time and query-time analysis must agree, with two deliberate exceptions (query-time synonyms so adding one does not require a reindex, and index-time-only decompounding in some setups). An accidental mismatch is invisible: the query returns nothing and reads as a data problem rather than a configuration one.
Where I would push back on a relevance roadmap: run _analyze on the failing query and the
failing document before anything else. It takes thirty seconds, it answers most "why doesn't this
match" questions outright, and it is skipped because it feels too basic for a relevance problem.
Follow-up Q&A
"What does an analyzer actually do?" Three stages. Character filters before tokenisation, for HTML stripping and unicode normalisation. The tokeniser, which is the language-specific decision that matters most because CJK has no whitespace at all. Then token filters: lowercase, stopwords, synonyms, stemming, decompounding, folding. And the rule that governs all of it is that index-time and query-time analysis must produce the same terms, or nothing matches at any score.
"Stemming or lemmatisation?" Stemming for most systems, and light stemming rather than aggressive. Stemming chops algorithmically and produces non-words, which is fine as long as the query stems identically; lemmatisation maps to a real dictionary form using part-of-speech context, which is correct and needs a dictionary and often a tagger. The reason to prefer light stemming is over-merging: aggressive stemmers collapse "universe" and "university" into "univers", and the marginal recall from being aggressive is small against that precision cost.
"Can you avoid choosing?" Yes, and it is what I would do: index both. An unstemmed .exact
sub-field alongside the stemmed field, searched together with the exact field boosted. You get recall
from the stemmed field and precision from the exact one, and exact matches outrank stemmed ones. It
costs index size and it is strictly better than picking one.
"What makes CJK hard?" There is no whitespace, so tokenisation is the entire problem rather than a preprocessing step. And segmentation is genuinely ambiguous: the same Chinese string can segment as "Beijing / university student" or "Peking University / student", and the wrong choice produces terms that never match. Dictionary analysers like kuromoji or IK give correct terms and fail on out-of-vocabulary words, which for a catalogue means brand and product names specifically. So the production answer is a dictionary analyser with a user dictionary, plus an n-gram field as a recall fallback.
"What's special about German?" Compounds. A search for "Dampfschiff" finds nothing inside
"Donaudampfschifffahrtsgesellschaft" without decompounding, and the hyphenation decompounder with a
word list is better than the plain dictionary one, which splits greedily and produces spurious
terms. Set only_longest_match or the index bloats with every sub-split, and keep a protected-words
list, because splitting a brand name into its components makes it match unrelated documents.
"What's the ordering bug people hit?" Synonyms after stemming. If synonyms are expanded after
the stemmer, the synonym's own terms are never stemmed, so they will not match the stemmed terms in
the documents. Synonyms go before stemming. The other common one is stemming before lowercasing, so
"Running" and "running" stem differently. And german_normalization must come before
asciifolding, because plain folding turns ü into u when the correct German transliteration is ue.
"How do you debug 'this obviously relevant document doesn't come back'?" _analyze on the query
text and on the document text, and compare the terms produced. Thirty seconds, and it answers most of
these outright. It gets skipped because it feels too basic for a relevance problem, and in my
experience the cause is analysis rather than scoring more often than not.
"Give me a case where this mattered." A marketplace whose German locale had a 14 percent null-result rate against 3 percent for English. The mapping had been copied from the English index with only field names changed, so German used the standard analyzer. Three separate failures: no decompounding, no stemming, no umlaut normalisation, all invisible in ranking metrics because the documents never entered the candidate set. A proper analyzer took null results to 3.4 percent, and the quarter's planned relevance work had been a ranking model.
"Doesn't dense retrieval make this obsolete?" It handles morphology implicitly, which is real, and it fails on exact identifiers: a dense encoder puts "iPhone 15 Pro 256GB" and the 128GB variant at nearly identical similarity. So the lexical arm still exists in any hybrid system, and while it exists its analysis determines what it can match. Analysis is a ceiling on that arm, and hybrid retrieval means you still care about the ceiling.
Common misconceptions
"Analysis is a configuration detail." It determines what can be matched at all, so it caps every ranking improvement, and an analyzer bug is invisible in ranking metrics.
"Stemming and lemmatisation are the same." One chops algorithmically and produces non-words; the other maps to dictionary forms using part-of-speech. Different cost, different failure modes.
"More aggressive stemming is better recall." It also merges words users distinguish. Light stemming plus an exact sub-field beats aggressive stemming.
"ASCII folding handles German." It turns ü into u when the correct transliteration is ue. Language-specific normalisation must come first.
"You can add synonyms anywhere in the filter chain." After the stemmer, the synonym's terms are never stemmed and never match. Synonyms go before stemming.
Interview delivery note
Frame it as a ceiling rather than a setting, because that is what makes it worth attention: "Analysis determines what can be matched at all, so it caps every ranking improvement above it. A term that's never produced can't be matched at any score, which means an analyzer bug is invisible in every ranking metric while limiting all of them."
Give the morphology argument with the languages: "And it gets much worse than English. Finnish nouns have about fifteen cases, German compounds have to be decomposed or a search for a component finds nothing, and Chinese has no whitespace so you need segmentation before you have terms at all. A shared analyzer is correct for none of those, which is why English-first systems get it wrong."
Give the both-not-either move, twice: "For stemming I'd take light rather than aggressive, because aggressive merges 'universe' and 'university' into 'univers'. And I'd index both: an unstemmed exact sub-field searched alongside and boosted, so I get recall from the stemmed field and precision from the exact one. Same for CJK: a dictionary analyser with a user dictionary as primary, plus an n-gram field for recall, because brand names are systematically out of vocabulary."
Name the absolute rule and the diagnostic: "The rule is that index-time and query-time analysis must
agree, or nothing matches at any score. And the first thing I'd run for any 'why doesn't this match'
is _analyze on both the query and the document text and compare the terms. Thirty seconds, and it
gets skipped because it feels too basic for a relevance problem."
Close with the case, because it makes the priority argument: "In one marketplace the German locale had a fourteen percent null-result rate against three for English, because the mapping had been copied and the analyzer never changed. Three failures, all invisible in ranking metrics. The quarter's planned relevance work had been a ranking model."
Further reading
- The Elasticsearch language analyzers reference, and the
kuromoji,smartcn,IKandnoriplugin documentation. - The Snowball stemmer project, for the algorithmic stemmers and the light versus classic variants.
- Lucene's analysis package documentation, particularly
ASCIIFoldingFilterand the language-specific normalisation filters. - Elasticsearch's
_analyzeAPI reference, which is the diagnostic this topic is really about.