Search, retrieval and ranking
If your background is search, recommendations and retrieval, this is the chapter where you control the room. Most interviewers cannot go deep here, which is an advantage only if you have rehearsed the depth. The two places candidates lose it: hand-waving the evaluation story, and treating vector search as a drop-in replacement for lexical search rather than one arm of a hybrid.
The unprompted signals that land well: naming position bias and how you correct for it, naming popularity feedback loops in a recommender, and naming the recall cliff that filtered ANN search walks off.
What this chapter covers
- [todo] The inverted index, mechanically: postings, skip lists, segments
- [todo] Lucene segments, refresh vs flush vs merge, near-real-time search
- [todo] BM25 derived from first principles
- [todo] Analyzers per language: stemming vs lemmatisation, CJK segmentation, transliteration
- [todo] Multilingual index topology: shared vs per-language vs cross-lingual
- [todo] HNSW internals: M, efConstruction, efSearch, the memory formula, delete cost
- [todo] IVF-PQ vs HNSW vs DiskANN vs ScaNN
- [done] Filtered vector search and the recall cliff
- [todo] Billion-scale sharding, routing and cold start
- [todo] The multi-stage funnel: candidate generation, light ranker, heavy ranker, re-rank
- [todo] Learning to rank: features, training data, and a worked feature set
- [todo] Judgment collection and a training-data pipeline
- [todo] NDCG, MRR, recall@k, and offline-online correlation
- [done] Interleaving and position bias
- [todo] Two-tower retrieval and the feature store parity problem
- [todo] Cold start, exploration and bandits
- [todo] Popularity bias and feedback loops
Source: §6, §44.2, §47.