The book names two reasons production data diverges from training data. The first is that:

Real-world data is virtually infinite while training data is finite — The second is that the real world is not stationary. The divergence can be as minor as a different emoji encoding.

A training set encodes emoji differently from the production input. The resulting failure mode is called:

Train-serving skew — It is not the world changing — it is the two environments never having agreed in the first place.

A person jaywalking on a highway, correctly detected and avoided by a self-driving car, is:

An outlier but not an edge case — Outliers refer to data; edge cases refer to performance. An outlier can cause an edge case, but not all outliers are edge cases.

During inference, an unusual query generally:

Cannot be removed or ignored, so the model has to handle it — You can sometimes transform it, the way Google Search rewrites "mechin learnin" — but mostly you want a model that performs well on unexpected inputs.

Covariate shift is defined as:

P(X) changes while P(Y|X) stays the same — It follows the decomposition P(X,Y) = P(Y|X)P(X).

Which of these is a cause of covariate shift that arises from the learning process itself?

Active learning, which alters the training input distribution — Covariate shift is a by-product of active learning. The other development-time causes are selection bias and artificial alteration of the training data.

A three-bedroom San Francisco apartment that cost $2,000,000 before COVID-19 and $1,500,000 at the beginning of it is an example of:

Concept drift — Same input, different output. The distribution of house features did not move; the conditional distribution of price did.

Because concept drift is often cyclic or seasonal, companies sometimes:

Keep one model for weekdays and another for weekends — Rideshare prices fluctuate weekday against weekend; flight prices rise in holiday seasons.

A credit score system whose range changes from 300-850 to 250-900 is an example of:

Label schema change — Unlike label shift, both P(Y) and P(X|Y) change. It may require relabeling and retraining from scratch.

ML-SYSTEMS

advanced

Data Distribution Shifts & Monitoring

ML Engineer path · lesson 74 of 80

A model performs best the day it is trained. Separate software failures from ML failures, name the shift by which term of the joint distribution moved, break a degenerate feedback loop with a randomisation slider, and watch a detection window too short to see a weekly cycle.

The idea

A map of a city that keeps being rebuilt. The map was accurate the day it was drawn; nothing about it has changed since; and every month it is a little more wrong. The failure is not in the map.

How to use

On the feedback-loops tab, raise the randomization slider and watch two nearly-identical songs stop diverging — that is TikTok's mechanism.

Software system failures

• Dependency failure — especially common when the third party that maintains it no longer exists.
• Deployment failure — deploying the binaries of an older model version, or missing file permissions.
• Hardware failures — CPUs overheating. Cosmic rays cause “soft errors”.
• Downtime or crashing — a hosted server going down.

ML-specific failures

• Production data differing from training data
• Edge cases
• Degenerate feedback loops

“Even though they account for a small portion of failures, they can be more dangerous than non-ML failures as they are hard to detect and fix.”

Why the stationarity assumption is wrong — two reasons

1. Real-world data is multifaceted and virtually infinite, whereas training data is finite and constrained by time, compute and human resources. The divergence can be as minor as a different type of emoji encoding, producing the failure mode called train-serving skew.

2. The real world is not stationary. “In 2019, when people searched for Wuhan, they likely wanted to get travel information, but since COVID-19, when people search for Wuhan, they likely want to know about the place where COVID-19 originated.” Shifts happen suddenly (a competitor changes pricing, you launch in a new region, a celebrity mentions your product), gradually (social norms, cultures, languages, trends), or seasonally (more rideshares in cold snowy winter than in spring).

Before you conclude the world moved

“A large percentage of what might look like data shifts on monitoring dashboards are caused by internal errors” — bugs in the data pipeline, missing values incorrectly imputed, train/inference feature inconsistencies, features standardised with statistics from the wrong subset, the wrong model version, or bugs in the app interface that force users to change their behaviour. The CTO of a monitoring service company estimates 80% of the drifts his service captures are caused by human errors.

Edge cases are not outliers

“Outliers refer to data: an example that differs significantly from other examples. Edge cases refer to performance: an example where a model performs significantly worse than other examples.” An outlier can cause an edge case, but not all outliers are edge cases — a person jaywalking on a highway is an outlier but not an edge case if your self-driving car detects them and responds appropriately.

A self-driving car that drives you safely 99.99% of the time but has a catastrophic accident the other 0.01% may be unusable. For calibration: as of 2019, the US traffic-fatality rate per 100,000 licensed drivers was 15.8, or 0.0158% — so 99.99% is a shade safer than an average human driver. And during development removing outliers helps the model learn better boundaries, while during inference you usually cannot remove or ignore an unusual query.

Key insight

A sudden increase in samples where the model performs badly can indicate the underlying distribution has shifted. Which means edge-case rate is not just a safety metric — it is a leading indicator for the next section.

Name the shift

Sort

Sort each example by which part of the joint distribution moved.

Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

Covariate shift

Drop items here

Label shift

Drop items here

Concept shift

Drop items here

6 pieces left to place

Test your knowledge

20 questions

Question 1 of 20

Score: 0

The CTO of a monitoring service company estimates that the share of drifts his service captures that are caused by human errors is about:

The short version

A map of a city that keeps being rebuilt. The map was accurate the day it was drawn, nothing about it has changed since, and every month it is a little more wrong. The failure is not in the map.

In more depth

Data Distribution Shifts & Monitoring

A consulting firm took six months to build a grocery restocking model. A year later it consistently overestimated demand for some items — which expired — and underestimated others, losing sales, until the inventory team hand-corrected it into uselessness. "Deploying a model isn't the end of the process."

Two kinds of failure

Software system failures: dependency failure (especially common when the third party that maintains it no longer exists) · deployment failure (the binaries of an older model version) · hardware failures (cosmic rays cause "soft errors") · downtime.

ML-specific failures — "even though they account for a small portion of failures, they can be more dangerous than non-ML failures as they're hard to detect and fix."

Why production data diverges

1.Real-world data is virtually infinite; training data is finite. The divergence can be as minor as a different type of emoji encoding — the failure mode is train-serving skew.

2.The real world isn't stationary. "In 2019, when people searched for Wuhan, they likely wanted to get travel information." Shifts come suddenly, gradually, or seasonally.

The CTO of a monitoring service company estimates 80% of the drifts his service captures are caused by human errors.

Edge cases are not outliers: "outliers refer to data... edge cases refer to performance." A person jaywalking on a highway is an outlier but not an edge case if the car responds appropriately. A car that is safe 99.99% of the time may be unusable — for scale, the 2019 US traffic-fatality rate per 100,000 licensed drivers was 15.8, or 0.0158%.

Three shifts

Type	What moves	Decomposition
Covariate shift	P(X) changes, **P(Y	X)** does not
Label shift	P(Y) changes, **P(X	Y)** does not
Concept drift	**P(Y	X) changes, P(X)** does not

Covariate: "an independent variable that can influence the outcome... but which is not of direct interest." Three causes during development: selection bias, artificial alteration of training data, and the learning process itself — active learning produces covariate shift as a by-product.

Concept drift: a three-bedroom San Francisco apartment cost $2,000,000 before COVID-19 and $1,500,000 at the beginning of it. Drifts are often cyclic or seasonal, so companies may keep one model for weekdays and another for weekends.

Label schema change is distinct: both P(Y) and P(X|Y) change — a credit score range of 300-850 becomes 250-900, or NEGATIVE splits into SAD and ANGRY, requiring relabeling and retraining from scratch.

Degenerate feedback loops

"Created when a system's outputs are used to generate the system's future inputs, which, in turn, influence the system's future outputs." Known as exposure bias, popularity bias, filter bubbles and echo chambers.

Detect: aggregate diversity and average coverage of long-tail items; Chia et al. (2021) bucket items by interaction count (bucket 1 under 100, bucket 2 between 100 and 1,000) and measure hit rate per bucket.

Correct: randomization — TikTok assigns each new video a random initial pool of up to hundreds of impressions — and positional features, either a "1st Position" feature set to False at inference, or two models, one for whether the recommendation was seen and one for the click given it was seen.

Detecting shift

Only P(X) can be monitored without knowing Y. Summary statistics first — as of October 2021 even TensorFlow Extended's data validation uses only summary statistics — then two-sample tests. Kolmogorov-Smirnov is nonparametric but only works on one-dimensional data. MMD is "popular in research, but I'm not aware of any company that is using it in the industry." Alibi Detect implements many of them.

The window determines what you can detect: with weekly-cycle data, days 9-14 as the source makes day 15 look like a shift; days 1-14 makes it look normal. Sliding statistics reset each window; cumulative statistics don't, and cumulative accuracy hides a sudden dip between hours 16 and 18.

Monitoring and observability

Four artifacts, in pipeline order: raw inputs → features → predictions → accuracy-related metrics. "The deeper into the pipeline an artifact is, the more transformations it has gone through" — more likely to be a transformation error, but more structured and easier to monitor.

99.99% allows a little over 4 minutes of downtime a month; 99.999% allows only 26 seconds.

Ian Malpass at Etsy: "If it moves, we track it." Badoo was handling 20 billion events a day in 2019. "A request may do 20-30 hops... the hard part might not be in detecting when something happened, but where the problem was."

"Monitoring is inherently passive. You wait for a shift to happen to detect it." Observability assumes internal states can be inferred from external outputs, so you can diagnose without shipping new code.