1. Scope
In scope
- • Real-time CTR scoring for ad candidates within an auction request
- • A feature store serving near-real-time signals (recent CTR, budget pacing)
- • Online/incremental learning to react to fast-shifting ad performance
Out of scope
- • The full ad auction mechanism design (second-price auction theory, etc.) — this doc focuses on the ML scoring component
- • Ad creative generation or targeting rule authoring
- • Fraud/bot-click detection (a related but separate system)
2. Functional requirements
Must have
- • Score every eligible ad candidate for a given auction request with a calibrated click probability
- • Complete all scoring within the auction's latency budget (tens of milliseconds)
- • Incorporate near-real-time signals — an ad's CTR over the last few minutes, not just historical averages
Nice to have
- • Per-user personalized CTR prediction, not just per-ad global CTR
- • Multi-objective scoring (CTR combined with conversion-rate prediction)
3. Non-functional requirements
4. Back-of-envelope estimation
Assumptions
- • 1B ad auction requests/day across a large platform
- • Average of 20 eligible ad candidates scored per auction request
- • Auction-level latency budget of 100ms total, of which CTR scoring gets ~50ms
Derived numbers
5. API design
Given an auction context and a list of ad candidates, return predicted CTR per candidate
Log an observed click, feeding both offline retraining and online incremental updates
Log an ad impression (shown but not necessarily clicked) — the negative-label signal
6. High-level design
What each box in the reference diagram is responsible for:
7. Data model
8. Deep dives
Calibration versus ranking accuracy — why both matter here specifically
In most ML systems, ranking accuracy (does the model correctly order candidates by relevance) is what matters. Here, the RAW predicted probability itself feeds a monetary bid calculation — if the model says 5% CTR but the true rate is 8%, the platform systematically underprices that ad slot. This is why ad-ranking systems evaluate calibration (e.g. via reliability diagrams, comparing predicted vs. observed CTR in probability buckets) as a first-class metric, not just AUC or log-loss.
Why online/incremental learning, not just daily batch retraining
Ad creative and campaign performance can shift within hours — a new creative can start strong and "fatigue" (CTR declining as the same audience sees it repeatedly) well before a daily batch retrain would catch it. Online learning (updating model weights incrementally as new labeled data streams in, rather than retraining from scratch) or at minimum near-real-time feature updates (recent CTR as a feature, refreshed every few minutes) let the system react within the timescale that actually matters for ad performance.
FTRL-Proximal and feature hashing — how CTR models are actually trained at internet scale
A naive logistic regression over ad_id × user_id × context feature crosses would have billions of parameters — far too large to fit in memory or train with standard gradient descent. Two techniques make this tractable, both applied and described at scale in Google's "Ad Click Prediction: a View from the Trenches." First, Follow-The-Regularized-Leader-Proximal (FTRL-Proximal): an online optimizer originating in earlier McMahan et al. optimizer research that processes one example at a time (fitting the streaming nature of ad click data), produces sparse models via L1 regularization (most weights converge to exactly zero and can be dropped), and achieves better per-example convergence than standard stochastic gradient descent for logistic loss. This makes it possible to train continuously on the incoming click stream without storing the entire dataset. Second, feature hashing (the "hashing trick"): instead of maintaining an explicit vocabulary of every ad_id, user_id, and cross-feature, each feature string is hashed to a fixed-size index in a weight vector of, say, 2^26 entries. Hash collisions are acceptable in practice because the model is highly regularized and the hash space is large relative to the number of live features at any one time. Together, FTRL + feature hashing enable training a logistic regression with hundreds of billions of virtual features in a model that fits in a few gigabytes — the combination that powered Google's CTR prediction at scale for years before deep models became latency-viable.
Position bias — click position affects CTR independent of ad relevance, and naively training on raw clicks teaches the wrong thing
Users are far more likely to click an ad shown in position 1 than an identical ad shown in position 5, purely because of where it appeared on the page — not because of any difference in the ad's actual relevance or quality. If you train a CTR model on raw click labels without accounting for this, the model learns to predict "was shown in a good slot" rather than "is actually relevant." This is position bias, and it creates a self-reinforcing loop: ads that were historically shown in top positions accumulate more clicks, the model learns they are more clickable, so they get ranked higher again. The standard fix is to include position as a training feature so the model can learn to separate the position effect from true ad quality — but then zero it out or set it to a neutral value at serving time (before the auction decides position), so the model predicts the click probability as if position were controlled for. A more principled variant uses inverse propensity scoring, treating the probability of being shown in each position as the propensity and reweighting training examples accordingly. Some systems use both: position as a shallow feature in the model (fast, approximate correction) combined with IPS reweighting in the loss function (unbiased correction for the training distribution). Either way, ignoring position bias is one of the most common and costly mistakes in building a first production CTR model.
9. Bottlenecks & trade-offs
- The feature store's read latency directly caps the whole auction's latency budget — this is usually the first thing to optimize (in-memory caching, pre-computation) as traffic grows
- Extreme class imbalance (sub-1% CTR) makes evaluation metric choice critical — accuracy is meaningless here (predicting "never clicks" is >99% accurate and useless); log-loss and AUC-PR are the metrics that actually matter
- Feedback loop risk: the auction only observes outcomes for ads it actually showed, creating a selection-bias problem (the model never learns from ads it never showed) — this is a subtler, ads-specific version of the same feedback-loop issue recommendation systems face