Design personalised recommendation serving on a 50 ms budget

"Serve personalised recommendations for a homepage, 20,000 requests per second, p99 under 50 milliseconds end to end."

Step 1: clarify (3 minutes)

What is the surface? A homepage carousel of 20 items differs enormously from a "customers also bought" module. Assume a homepage feed: 5 carousels, 20 items each, so 100 items ranked per request.

Is 50 ms the whole page or the recommendation call? Assume the recommendation service call, which means roughly 40 ms of internal budget after network and serialisation.

How many candidate items? Assume a 10 million item catalogue, of which perhaps 2 million are eligible at any moment (in stock, in region, not suppressed).

How fresh must personalisation be? This is the question that separates the designs. Assume within-session: a user who clicks a hiking boot should see hiking content in the next request. That rules out purely precomputed recommendations and forces real-time feature retrieval.

Cold start? Assume 30 percent of daily traffic is logged-out or new, which is typical and which means the fallback path is not an edge case, it is a third of the system.

Step 2: capacity math (4 minutes)

Traffic
  20,000 RPS steady, 3x peak -> design for 60,000 RPS
  Each request: 100 items scored, 5 carousels

Scoring volume
  60,000 RPS x 500 candidates scored = 30M scores/sec
  A small MLP (3 layers, 256 hidden) is ~200k FLOPs per item
  -> 6 TFLOPs/sec of pure ranking compute

  On CPU at ~50 GFLOPs/core effective: 120 cores. Feasible.
  This is why the ranker is small: a 100x bigger model is 12,000 cores.

Feature retrieval
  60,000 RPS x 1 user-feature fetch  = 60,000 reads/sec
  60,000 RPS x 500 item-feature fetch = 30M reads/sec  <-- the problem

  30M item-feature reads/sec is NOT servable from a remote store.
  Redis does ~200k ops/sec/node with pipelining -> 150 nodes.
  This forces item features INTO the ranking process memory.

Item feature memory
  2M eligible items x 256 floats x 4 bytes = 2 GB
  Quantised to fp16:                        = 1 GB
  -> fits comfortably in every serving replica. Broadcast it.

Fleet
  At ~500 RPS per 16-core replica (scoring-bound), 60,000 RPS
  needs ~120 replicas.

The decisive number: 30 million item-feature reads per second. It is the constraint that determines the architecture. Item features must live in the serving process, replicated to every node, refreshed periodically. Only user features are fetched per request, and there is one of those.

That single realisation is the design.

Step 3: the multi-stage funnel

You cannot score 2 million items in 40 ms. Nobody can. The architecture is a funnel where each stage is cheaper per item and worse at ranking than the next.

2,000,000 eligible items
      |
      |  RETRIEVAL  (~8 ms)   multiple parallel sources, cheap
      v
    ~1,000 candidates
      |
      |  FILTERING  (~2 ms)   business rules, seen-list, availability
      v
      ~500 candidates
      |
      |  RANKING    (~15 ms)  the learned model, per-item features
      v
       ~100 scored
      |
      |  RE-RANK    (~4 ms)   diversity, business, exploration
      v
        100 items in 5 carousels

The rule that governs the funnel: each stage may only reduce the set, never expand it, and recall lost at retrieval can never be recovered downstream. A brilliant ranker over a bad candidate set is worse than a mediocre ranker over a good one, which is why retrieval quality is measured separately.

Retrieval: several cheap sources in parallel

Source                      How                        Latency  Yield
--------------------------------------------------------------------
Two-tower ANN               user embedding -> HNSW      6 ms    300
Recent-interaction i2i      precomputed item->item      1 ms    200
Trending in user's segment  precomputed, cached         <1 ms   200
Category affinity           precomputed per user        1 ms    200
Fresh / new items           time-ordered, exploration   <1 ms   100

All five run in parallel and the union is deduplicated. Total latency is the slowest (6 ms), not the sum.

The two-tower model is the only learned source:

# Item tower: run OFFLINE, nightly. Its output is the ANN index.
item_vec = item_tower(item_features)          # 10M items, batch job

# User tower: run PER REQUEST, on ~40 features, and it must be tiny.
user_vec = user_tower(user_features)          # ~0.3 ms on CPU
candidates = hnsw_index.search(user_vec, k=300, ef=64)   # ~6 ms

The asymmetry is the point of the two-tower architecture. The item tower is expensive and offline; the user tower is cheap and online. There is no interaction between user and item features at retrieval time, which is exactly what makes an ANN index possible, and exactly why retrieval is worse at ranking than the ranker is.

Ranking: where the accuracy lives

The ranker can use user-item interaction features, which is why it is better and why it only runs on 500 items.

def rank(user_features, candidates, item_store) -> list[tuple[str, float]]:
    # Item features come from PROCESS MEMORY, not a network call.
    # This is the whole reason the design works at 30M lookups/sec.
    item_feats = item_store.batch_lookup(candidates)      # ~0.5 ms, in-memory

    # Cross features: computed here, not stored. They depend on both sides.
    cross = compute_cross_features(user_features, item_feats)
    #   e.g. category affinity match, price vs user's typical price band,
    #        brand seen before, days since last interaction with seller

    X = np.hstack([np.tile(user_features, (len(candidates), 1)),
                   item_feats, cross])
    scores = model.predict_batch(X)        # one batched call, ~12 ms
    return sorted(zip(candidates, scores), key=lambda t: -t[1])

Batch the whole candidate set into one model call. Scoring 500 items one at a time is 500 Python round trips and framework overhead; scoring them as one matrix is a single GEMM. This is often a 10x difference and it is the most common implementation mistake.

Re-ranking: what the model cannot express

def rerank(scored, k=100):
    selected, seen_categories, seen_sellers = [], {}, {}
    for item_id, score in scored:
        cat, seller = catalog[item_id].category, catalog[item_id].seller
        # Diversity constraints. A pure-relevance list of 20 hiking boots
        # is worse for the user than 12 boots and 8 related items, and no
        # pointwise ranker expresses that, because it scores items
        # independently.
        if seen_categories.get(cat, 0) >= 4:   continue
        if seen_sellers.get(seller, 0) >= 3:   continue
        selected.append(item_id)
        seen_categories[cat] = seen_categories.get(cat, 0) + 1
        seen_sellers[seller] = seen_sellers.get(seller, 0) + 1
        if len(selected) == k: break
    return selected

Plus an exploration slot: a small fraction of positions filled from an epsilon-greedy or Thompson-sampling policy, because a purely exploitative system never learns that a new item is good. Without exploration the training data is generated by the model's own past decisions, and the feedback loop narrows the catalogue over months. That is a slow, invisible failure and it is worth naming.

Step 4: the feature store, and the online/offline split

                      OFFLINE (training)              ONLINE (serving)
User features   ->    warehouse table, point-in-time   Redis, 1 fetch/request
Item features   ->    warehouse table                  in-process, broadcast
Cross features  ->    computed in the training job     computed at request time
Real-time       ->    replayed from the event log      session cache in memory

Training/serving skew is the failure mode that costs the most and shows up as "the model was better offline". Three specific causes, and each has a specific mitigation:

1. Point-in-time correctness. Training on user_purchase_count as it is today, when predicting a click from three months ago, leaks the future. The feature store must answer "what was this feature's value at time T", which means event-time versioning rather than a current-value table.

2. Different code paths. If the training pipeline computes days_since_last_purchase in SQL and the serving path computes it in Python, they will diverge, and the divergence will be subtle. The fix is one definition, compiled to both, which is what a feature store is actually for.

3. Missing-value handling. Training data has features backfilled; production has timeouts and nulls. If the model was never trained on nulls, a Redis timeout produces garbage scores rather than degraded ones. Train with realistic missingness injected, which is cheap and almost never done.

The check that catches all three:

# Log the exact feature vector used to score, at serving time.
# Then compare distributions against the training set daily.
# A PSI (population stability index) above ~0.2 on any feature
# is an alert, not a dashboard.
log_serving_features(request_id, user_id, candidates[:20], X[:20])

Step 5: latency budget, itemised

Component                              p50     p99    Notes
------------------------------------------------------------------
Request parse + auth                    1 ms    2 ms
User feature fetch (Redis, 1 call)      2 ms    6 ms  <- tail risk
User tower forward pass                0.3 ms   1 ms
ANN retrieval (ef=64, k=300)            4 ms    9 ms
Other 4 retrieval sources (parallel)    1 ms    3 ms
Dedupe + filter (in memory)             1 ms    2 ms
Item feature lookup (in process)       0.5 ms   1 ms
Cross feature computation               2 ms    4 ms
Ranker batch inference (500 items)     10 ms   18 ms  <- largest
Re-rank + diversity                     2 ms    4 ms
Serialise + respond                     1 ms    2 ms
------------------------------------------------------------------
TOTAL                                  25 ms   52 ms  <- p99 over budget

p99 is 52 ms and the budget is 50. Do not hand-wave this; fix it, and the fix demonstrates more than the design did.

Fix 1: hedge the user feature fetch.
  Send a second request at p95 (4 ms) and take the first response.
  Costs ~5% more Redis load, removes the tail.
  p99 6 ms -> 3 ms.

Fix 2: cap candidates at 400 instead of 500 when the ranker queue
  depth exceeds a threshold. Adaptive, not static.
  p99 ranker 18 ms -> 14 ms.

Fix 3: precompute cross features that depend only on coarse user
  attributes (segment, price band) rather than on the exact user.
  p99 cross 4 ms -> 2 ms.

New p99: ~43 ms.

Hedged requests are the highest-leverage single technique here, and it is worth saying why: at 60,000 RPS with 5 dependencies, the probability that a given request hits at least one dependency's p99 is high, so the service's p99 is dominated by the tail of its dependencies rather than by its own work. This is Dean and Barroso's "tail at scale" argument, and hedging is their answer.

Step 6: failure modes and degradation

Feature store timeout
  -> Serve with the user features you have, missing values imputed the
     way the model was trained to expect. Do NOT fail the request.

Ranker model unavailable or slow
  -> Fall back to the retrieval order (two-tower similarity), which is
     already a reasonable ranking. Quality drops, latency improves.

ANN index unavailable
  -> Fall back to the precomputed sources: i2i, trending, category
     affinity. These are cheap lookups and cover most of the surface.

Cold user (no history)
  -> Trending in their geography + category by referrer + editorial.
     30% of traffic. Design it first, not last.

Total personalisation failure
  -> Static popular-items list, cached at the edge, per country.
     Every recommendation system needs this and it should be served
     from a CDN so it survives the service being entirely down.

The degradation ladder: drop exploration, then reduce candidates from 500 to 200, then drop the ranker and serve the retrieval order, then serve trending. Four steps, each a flag, each with a measured quality cost so the decision is informed rather than panicked.

Step 7: what changes at ten times the scale

At 200,000 RPS:

The ranker moves to GPU or to a specialised inference server. 300 million scores per second on CPU is 1,200 cores of pure inference, and at that point batching across requests on a GPU (Triton with dynamic batching) is cheaper, at the cost of a few milliseconds of queueing.

Item features stop fitting in every replica. At 20 million eligible items and richer features, the broadcast copy is 40 GB and no longer sits comfortably in every process. The move is a sharded item-feature cache with locality-aware routing, or a smaller learned item representation (an embedding rather than raw features).

The feature store becomes the bottleneck before the model does. One user-feature fetch per request at 200k RPS is 200k reads/sec, which is fine, but the tail is not: at that volume you are hitting the p999 constantly. Colocate the user feature cache with the serving fleet and accept staleness, or push user features into the request from an upstream edge service that already has them.

Model refresh becomes continuous rather than daily. At this scale, a day-old model is measurably worse on fast-moving inventory, and the design shifts to incremental training with hourly deploys, which brings its own problem: model version skew across a fleet mid-rollout, so A/B assignment must be sticky per user rather than per request.

Production evidence

Covington, Adams and Sargin, "Deep Neural Networks for YouTube Recommendations" (RecSys 2016) is the canonical description of the candidate-generation-then-ranking funnel, including the reasoning for why the two stages use different models and different features.

Yi et al., "Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations" (RecSys 2019) is the two-tower reference from Google, including the in-batch negative sampling correction that makes the retrieval tower trainable at scale.

Dean and Barroso, "The Tail at Scale" (CACM 2013) is the source for hedged requests and for why a service fanning out to several dependencies inherits their tails.

Meta's DLRM and its published serving work document the embedding-table memory problem that makes item features the dominant memory cost, and the sharding approaches used when they stop fitting.

Feast and Tecton exist as products specifically because point-in-time correctness and the training/serving skew problem are hard enough to warrant dedicated infrastructure, which is useful evidence that the skew section above is a real risk rather than a theoretical one.

Instagram's and Pinterest's published ranking architectures both describe the retrieval-plus-ranking-plus-rerank shape with diversity constraints applied after scoring, which is where the diversity logic belongs because pointwise rankers score items independently.

The debate

The case for a single large model: funnels lose recall at every stage, and the retrieval stage is optimised for a different objective than the ranker. One end-to-end model avoids the mismatch and is simpler to reason about and to improve.

The case for the funnel: it is the only thing that fits the latency budget. Scoring 2 million items with a model that uses user-item cross features is not a tuning problem, it is arithmetically impossible in 40 milliseconds. The funnel is a consequence of the budget, not a preference.

The case for precomputing everything offline: batch-compute recommendations per user nightly, serve from a key-value store, 2 ms p99, trivially scalable. Many companies do this and it works.

My position: the funnel, with real-time ranking, and precomputed sources as retrieval arms rather than as the answer. Precomputation fails the within-session freshness requirement, which was a stated constraint: a user who just clicked a hiking boot must see hiking content in the next request, and a nightly batch cannot do that. But precomputed results are excellent candidates, which is why three of the five retrieval arms are precomputed lookups.

The decision I would defend hardest is putting item features in process memory rather than in a remote store. It looks like an optimisation and it is structural: 30 million item-feature reads per second is not servable remotely at any reasonable cost, so the choice is between in-process features and a fundamentally different architecture. Since 2 million items times 256 fp16 features is about 1 GB, broadcasting it to every replica is easy, and the whole latency budget then works. Teams that put item features in Redis discover this at load-test time and rearchitect.

The second is hedging the user-feature fetch. At 60,000 RPS across five dependencies, the service's p99 is dominated by its dependencies' tails rather than by its own work, and a hedge costs about 5 percent extra load to remove several milliseconds of p99. That is one of the best trades available in a latency-constrained service and it is under-used.

Where I would push back is on skipping exploration to protect the metric. It always improves short-term engagement and it degrades the system over months, because the training data becomes the model's own past decisions and the effective catalogue narrows. That failure is slow, invisible in daily metrics, and expensive to reverse.

Follow-up Q&A

"Why can't you just score everything?" Arithmetic. Two million eligible items at roughly 200,000 FLOPs per item is 400 GFLOPs per request, and the budget is 40 milliseconds. Even ignoring feature retrieval, that is not a tuning problem. The funnel exists because each stage is cheaper per item and worse at ranking than the next, and the design question is where to put each boundary.

"What's the single most important architectural decision here?" Putting item features in the serving process rather than in a remote store. Sixty thousand requests per second times five hundred candidates is thirty million item-feature reads per second, which is not servable from Redis at any sensible cost. Two million items at 256 fp16 features is about a gigabyte, so it broadcasts to every replica and the lookup becomes half a millisecond of memory access. That one decision is what makes the rest of the budget feasible.

"How do you prevent training/serving skew?" Three specific things. Point-in-time correct features, so training on a three-month-old label uses the feature values as of that moment rather than today's, which otherwise leaks the future. One feature definition compiled to both paths rather than SQL in training and Python in serving. And training with realistic missingness injected, because production has timeouts and nulls that the backfilled training set does not, and a model that has never seen a null produces garbage rather than degraded scores. Then log the exact serving feature vectors and compare distributions daily, alerting on population stability index above about 0.2.

"Your p99 is 52 ms against a 50 ms budget. What do you do?" Hedge the user feature fetch: send a second request at the p95 and take the first answer, which costs about 5 percent more load and removes most of the tail. Then make the candidate count adaptive, dropping from 500 to 400 when the ranker queue is deep. Then precompute the cross features that depend only on coarse user attributes like segment and price band rather than on the exact user. That gets it to about 43 milliseconds with headroom.

"Why hedge rather than just lower the timeout?" A lower timeout converts slow requests into failed ones, so you trade latency for errors and still have to handle the failure. Hedging gets you an answer from whichever replica is not currently slow. The underlying point is that at this fan-out, my p99 is mostly my dependencies' tails rather than my own work, so the fix has to address the tail specifically.

"How do you handle the cold-start third of traffic?" Design it first, not last, because 30 percent is not an edge case. Trending in their geography, category inferred from the referrer or landing page, and editorial or merchandised slots. The important part is that the same funnel serves it, with retrieval arms that need no user history, so there is one code path rather than a neglected fallback that nobody tests.

"Where does diversity live?" In the re-rank, after scoring, and not in the model. A pointwise ranker scores each item independently, so it cannot express "twenty hiking boots is worse for the user than twelve boots and eight related items". That constraint is about the set, so it is applied to the set. Listwise models can express it, and they are much more expensive to train and serve, so the standard answer is a cheap greedy constraint pass after ranking.

"What happens if you skip exploration?" Short-term engagement improves and the system degrades over months. The training data becomes a record of the model's own past decisions, so items the model never showed never get positive signal, and the effective catalogue narrows. It is invisible in daily metrics because each day looks fine, and it is expensive to reverse because you have to re-learn the parts of the catalogue you stopped showing. A few percent of positions on an epsilon-greedy or Thompson-sampling policy is the cheap insurance.

Common misconceptions

"The model is the system." The funnel, the feature store and the latency budget are each as consequential. The ranker is one 15 ms box in a 40 ms diagram.

"Recall lost at retrieval can be fixed by a better ranker." It cannot. Downstream stages only reduce the set. Retrieval recall is measured separately for exactly this reason.

"Feature stores are for convenience." They exist for point-in-time correctness and for having one feature definition, both of which are correctness properties rather than conveniences.

"Diversity is a model objective." Pointwise rankers score items independently and cannot express set-level constraints. Diversity belongs in the re-rank.

"Precomputed recommendations are a lesser design." They are excellent candidate sources and a perfectly good whole answer when within-session freshness is not required. Here it was required, which is why they are arms rather than the answer.

Interview delivery note

Lead with the arithmetic that forces the architecture, because it converts a pattern-matched answer into a derived one: "The number that decides this design is thirty million item-feature reads per second: sixty thousand requests times five hundred candidates. That's not servable from a remote store, so item features have to live in the serving process. Two million items at 256 fp16 features is about a gigabyte, so it broadcasts to every replica. Only user features get fetched per request, and there's one of those."

Then the funnel, with the rule that governs it: "Two million eligible items down to a thousand candidates by retrieval, five hundred after filtering, a hundred after ranking. Each stage is cheaper per item and worse at ranking than the next. And the rule is that recall lost at retrieval can never be recovered, so I'd measure retrieval recall separately from ranking quality."

Itemise the latency budget and then fix it out loud, because that is where the depth is: "Adding it up I get a p99 of about 52 milliseconds against a 50 millisecond budget, so let me fix that. I'd hedge the user feature fetch at the p95, which costs about five percent more Redis load and removes most of the tail. At this fan-out my p99 is mostly my dependencies' tails rather than my own work, which is the tail-at-scale argument."

The line that shows you have operated one of these: "and I'd keep a few percent of positions on exploration even though it costs short-term engagement, because without it the training data becomes the model's own past decisions and the effective catalogue narrows over months. That failure is invisible in daily metrics and expensive to reverse."

The Staff & Team Lead Interview Reference