A test split must stay static so that:

You have a trusted benchmark for comparing two models — Which creates the tension: a model updated for a new distribution is being judged on splits from the old one.

A backtest cannot replace a static test split because:

A pipeline bug could have corrupted the recent window it uses — So the static set stays as a sanity check. And doing well on the last hour does not mean doing well in the future.

In shadow deployment, the candidate model's predictions are:

Logged for analysis and never shown to a user — Each request goes to both models, but only the existing model's prediction is served.

The cost of shadow deployment is that it:

Doubles the number of predictions and the inference compute — Risk is low, but you are paying to generate predictions nobody sees.

As of 2017, Microsoft and Google each conducted roughly how many A/B tests annually?

Over 10,000 — Which is why the discipline around running them properly is itself infrastructure.

If users exposed to model A are usually on phones and those exposed to B are usually on desktops, the test is broken because:

You cannot tell whether A is better or whether device type drives the difference — Traffic routing must be truly random. It is one of the two things an A/B test has to get right.

At p = 0.05, repeatedly running the same A/B experiment will report "B is better" about:

5% of the time — Statistical significance is not foolproof, which is why a single significant result is not a decision.

A statistically insignificant A/B result on a large sample most likely means:

There is not much difference, so either model is probably fine — An insignificant result is not a failed test.

Ride-sharing dynamic pricing may require running A/B variants alternately, one per day, because:

One model's predicted prices affect the number of available drivers and riders — When one model's predictions affect the other's environment, simultaneous serving contaminates both arms.

ML-SYSTEMS

advanced

Testing in Production

ML Engineer path · lesson 76 of 80

Offline evaluation cannot tell you a model works. Fan a request out to a shadow deployment, size an A/B test until the detectable lift shrinks, ramp a canary and abort it, redraw a team-draft interleaving list, and watch a bandit reach a verdict in 12,000 samples where A/B testing needed 630,000.

The idea

Tasting a new recipe. You can read it, you can cook it for yourself, and you can serve it to one table before you put it on the menu. Only the last one involves the people who will actually be eating it.

How to use

On the interleaving tab, redraw the draft a few times — the merged list changes, but at every position the item is equally likely to have come from either model. That is the whole guarantee.

Static test splits

They must be static so you have a trusted benchmark — “it'll be hard to compare the test results of two models if they are tested on different test sets.” The catch follows immediately: a model updated for a new distribution should not be evaluated only on test splits drawn from the old one.

Backtests

“The method of testing a predictive model on data from a specific period of time in the past.” After updating the model on the last day's data, test it on the last hour's data, assuming that hour was not part of the update.

Why backtests cannot replace the static split

If a data pipeline bug corrupted the last hour's data, evaluating solely on it is not sufficient — so the static set stays, as a sanity check. And even a clean backtest only tells you about the past: doing well on the last hour does not mean doing well in the future.

“The only way to know whether a model will do well in production is to deploy it.”

Five techniques, ordered by the risk they expose users to

Who runs the tests is part of the pipeline

“A good evaluation pipeline is not only about what tests to run, but also about who should run those tests.” In ML, evaluation is usually owned by the same data scientists who developed the model, tested ad hoc with whichever tests they like. Two problems follow. The process is imbued with biases — data scientists have context most users do not, so they probably will not use the model the way most users will. And the ad hoc nature makes results variable: “One data scientist might perform a set of tests and find that model A is better than model B, while another data scientist might report differently.” The fix is clear pipelines — which tests, in what order, against what promotion thresholds — automated to kick off on every model update, with results reported and reviewed, like CI/CD.

Name the technique

Sort

Sort each description into the testing-in-production technique.

Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

Shadow

Drop items here

A/B test

Drop items here

Canary

Drop items here

Interleaving

Drop items here

Bandit

Drop items here

5 pieces left to place

Test your knowledge

20 questions

Question 1 of 20

Score: 0

The book's distinction between monitoring and testing in production is that testing in production:

The short version

In more depth

Testing in Production

*"If monitoring means passively keeping track of the outputs of whatever model is being used, test in production means proactively choosing which model to produce outputs so that we can evaluate it."*

Why offline evaluation cannot finish

Static test splits must stay static — "it'll be hard to compare the test results of two models if they are tested on different test sets" — so a model updated for a new distribution should not be evaluated only on splits from the old one.

Backtests test "on data from a specific period of time in the past" — the last hour, after updating on the last day. But if a data pipeline bug corrupted the last hour's data, evaluating solely on it isn't sufficient, so the static set stays as a sanity check.

"The only way to know whether a model will do well in production is to deploy it."

Five techniques

Shadow deployment — "might be the safest way to deploy your model or any software update." Route each request to both, serve only the existing model's prediction, log the candidate's. Risk is low, but it doubles the number of predictions and generally doubles inference compute cost.

A/B testing — as of 2017, companies like Microsoft and Google each conduct over 10,000 A/B tests annually. Two things to get right: routing must be truly random (otherwise phone users on A, desktop users on B confounds everything) and enough samples. At p = 0.05, repeating the experiment gives "A is better" 95% of the time and "B is better" 5%. An insignificant result is not a failed test. When predictions affect each other — dynamic pricing changes how many drivers get on the road — run the variants alternately.

Canary release — "slowly rolling out the change to a small subset of users." You do not have to randomize the traffic; a plausible scenario is rolling out to a less critical market first.

Interleaving — Joachims, 2002. Netflix: it "reliably identifies the best algorithms with considerably smaller sample size compared to traditional A/B testing." Team-draft interleaving: for each position, pick A or B with equal probability, and that model takes its top item not already chosen. But "there's no guarantee that user preference will lead to better core metrics."

Bandits — ε-greedy, Thompson Sampling (1933), UCB (Auer, 2002), which "implements optimism in the face of uncertainty." Greg Rafferty: A/B testing needed over 630,000 samples for a 95% confidence interval; Thompson Sampling called a 5% improvement in under 12,000. A/B testing is stateless; bandits are stateful, which is why "bandit algorithms are not widely used in the industry other than at a few big tech companies."

Contextual bandits determine the payout of each action, not of each model. 1,000 items is a 1,000-arm problem where you see feedback only on the top 10 — the partial feedback problem, also known as bandit feedback. The trap: an item with 0 value points could be one that was never shown, so showing only high-value items keeps the same items on top forever.

Who runs the tests

Evaluation is usually owned by the same data scientists who developed the model, tested ad hoc. That is imbued with biases — "they probably won't use the model the way most users will" — and variable: "One data scientist might perform a set of tests and find that model A is better than model B, while another data scientist might report differently." The fix is clear, automated pipelines with promotion thresholds, reported and reviewed like CI/CD.