Design a recommendation system (Netflix, YouTube, TikTok feed)

1. Scope

In scope

• Candidate generation (retrieval) from a large item catalog
• Ranking the retrieved candidates by predicted relevance/engagement
• A feature store serving both training and serving paths consistently
• The feedback loop from user interactions back into retraining

Out of scope

• The actual model training infrastructure (assume a training pipeline exists)
• Cold-start handling for brand-new users/items (worth a follow-up deep dive, not core to this pass)
• Multi-armed-bandit exploration strategies

2. Functional requirements

Must have

• Given a user, return a ranked list of N recommended items in under 200ms p99
• Recommendations reflect the user's recent interaction history
• Already-shown-and-dismissed items are not immediately re-shown

Nice to have

• Explanations for why an item was recommended
• Real-time personalization within a single session (not just from historical data)

3. Non-functional requirements

Latencyp99 < 200ms end-to-end— candidate generation + ranking + re-ranking, all inside the request path

Catalog size10M+ items

QPS~50,000 requests/sec at peak for a large consumer app

Freshnessfeatures updated within minutes, full model retrain daily-to-weekly

4. Back-of-envelope estimation

Assumptions

• 10M item catalog, 100M daily active users
• Average user requests a feed refresh 5x/day
• Candidate generation must narrow 10M items to ~500 in under 20ms

Derived numbers

Total daily requests500M/day— 100M users × 5 refreshes

Average QPS~5,800 req/sec— 500M / 86,400 seconds

Peak QPS~25,000-50,000 req/sec— assuming a 5-10x peak-to-average ratio typical of consumer traffic

5. API design

GET/feed?userId={id}&count={n}

Return a ranked list of n recommended items for the user

POST/feedback

Log an impression, click, or explicit dismissal event

6. High-level design

What each box in the reference diagram is responsible for:

Candidate Generation — Narrows the full catalog to a few hundred plausible candidates fast, using an approximate-nearest-neighbor search over learned embeddings (a two-tower model) or collaborative filtering

Ranking Model — Scores the candidate shortlist using richer, more expensive features (recency, user-item interaction history, item metadata) — can afford to be slow per-item since it only runs on ~500 candidates, not the full catalog

Re-ranker — Applies business rules on top of the model score: diversity (don't show 10 near-identical items), freshness boosting, deduplication against recently-shown items

Feature Store — Single source of truth for features used at both training time (offline, batch) and serving time (online, low-latency) — critical that both paths compute the same feature the same way

Training Pipeline — Periodically retrains both the candidate-generation embeddings and the ranking model using the logged interaction events

7. Data model

Feature Store (online): user_id, item_id → feature_vector, keyed for low-latency point lookups— typically a key-value store like Redis or DynamoDB, not a relational DB

Feature Store (offline): historical (user_id, item_id, features, label, timestamp) rows for training— point-in-time correctness required: a training row must only use features as they existed AT that historical timestamp, not current values

Event Stream: (user_id, item_id, event_type, timestamp) append-only log of impressions/clicks/dismissals

8. Deep dives

Why two stages, not one model scoring everything?

Running a rich, expensive ranking model against all 10M catalog items per request would blow any reasonable latency budget. Candidate generation trades ranking precision for speed — a cheap embedding-similarity search narrows the field to ~500 plausible items in single-digit milliseconds. The expensive ranking model then only has to score those 500, which fits the latency budget. This is the same "cheap filter, expensive re-rank" pattern used in search engines.

Point-in-time correctness — the most common production bug in this architecture

If training data pulls "current" feature values instead of what the feature's value actually was at the time of that historical interaction, the model implicitly learns to use information it wouldn't have had in production (e.g. training on a feature that includes engagement from AFTER the interaction it's predicting). This "feature leakage" produces a model that looks great in offline evaluation and performs far worse in production. A correctly-built feature store versions features by timestamp specifically to prevent this.

How two-tower models are trained: contrastive loss and in-batch negatives

The two-tower model has a user tower and an item tower, each producing an embedding. At serving time, ANN search finds items whose embeddings are close to the user's embedding. But how do you train it? The key is contrastive learning with in-batch negatives. For each (user, item) pair the user actually engaged with (a positive pair), the training objective pulls the two embeddings together and pushes them apart from all other items in the same training batch (the "in-batch negatives"). This works because other items in a large batch are very unlikely to be relevant to this user, so they make cheap, abundant negatives without any explicit negative-labeling step. The loss is typically a softmax over one positive and hundreds of in-batch negatives — similar to noise-contrastive estimation. The practical consequence: batch size matters enormously. Larger batches expose each example to more negatives and produce better-separated embeddings, which directly improves retrieval recall. Teams at YouTube and Pinterest have published that going from small to large batches produces retrieval quality gains equivalent to several months of data collection.

Mitigating popularity-bias feedback loops with inverse propensity scoring

The feedback loop problem: the system recommends popular items → users click popular items → training data is dominated by popular-item interactions → the next model is even more biased toward popular items. Left unchecked, a small fraction of the catalog captures almost all the recommendations, suppressing discovery of niche content. Inverse propensity scoring (IPS) is the most principled fix: each training example is re-weighted by 1/P(item was shown), where P is estimated from the logging policy (e.g., the prior model's score for that item). An interaction with an item that had a 90% chance of being shown carries little new information; an interaction with an item that only had a 5% chance of being shown is much more informative. Downweighting overexposed interactions and upweighting underexposed ones de-biases the training distribution without collecting new data. A complementary technique is exploration bucketing at re-rank time: reserve a small slot in every feed (5-10%) for items selected by exploration policy (e.g., recently uploaded items, or items with high variance in predicted score) rather than the main ranking model. This generates logged interactions across a more diverse item set, continuously expanding the training distribution and keeping the feedback loop from collapsing to a fixed popularity ranking.

9. Bottlenecks & trade-offs

Candidate generation's approximate-nearest-neighbor index becomes the bottleneck first as the catalog grows — mitigate by sharding the index or reducing embedding dimensionality
The feature store's online read path is latency-critical for every request; a slow feature store directly caps the whole pipeline's p99
Popularity bias feedback loop: if the model is trained naively on what users clicked (which is itself biased by what was previously recommended), already-popular items get recommended more, suppressing discovery of new/niche items over time