Age missing for respondents of a particular gender is:

Missing at random — missing because of another observed variable — Gender is observed and explains the omission — which is why deleting those rows deletes a whole group.

The book describes MCAR as:

Very rare — There are usually reasons why certain values are missing, and you should investigate.

Filling missing number-of-children with 0 is a bad idea because it:

Makes missing indistinguishable from childless — In general, avoid filling missing values with possible values.

In the cautionary bug, a frontend stopped asking for age and the pipeline filled it with 0. The failure was that:

The model never saw 0 during training — Nothing errored — the model was simply asked to reason about newborns and produced garbage.

The book reports empirically preferring which scaling range?

[−1, 1] — Standardization is still the choice for normally distributed variables.

Because scaling requires global statistics, at inference time you must:

Reuse the training statistics — Which is exactly why you must retrain often — a scaler fitted a year ago was fitted to last year's world.

The boundary problem with discretization is that:

$34,999 becomes completely different from $35,000 — While $35,000 is treated the same as $100,000. "In practice, I have rarely found discretization to help."

Encoding Amazon brands as integers 0 to 1,999,999 fails because:

It crashes on an unseen brand — And adding an UNKNOWN bucket only moves the problem, since the model never saw UNKNOWN in training.

ML-SYSTEMS

intermediate

Feature Engineering & Data Leakage

ML Engineer path · lesson 70 of 80

Amazon had over two million brands in 2019, and three obvious encodings each fail one step later. Watch the hashing trick absorb an unbounded category space, then toggle scale-before-split to see a statistic carry the test set into training.

The idea

A recipe where the ingredients matter more than the technique. And where one of the ingredients is a taste of the finished dish, which makes every practice run go beautifully and tells you nothing about dinner.

How to use

On the leakage tab, toggle scale-before-split to see how a statistic computed over everything carries the test set into training.

Three kinds of missingness, and the one that makes deletion safe is very rare.

Missing because of the true value itself. Respondents who did not disclose income tend to have higher income than those who did.

Removing these rows removes information — the fact that income is missing is itself a signal.

Deletion

Column deletion drops a variable with too many missing values — but you might remove important information, since married couples are much more likely to be homeowners. Row deletion works when values are MCAR and the affected count is small, such as less than 0.1%. “You do not want to do row deletion if that means 10% of your data samples are removed.”

Imputation

Fill with defaults, or with the mean, median or mode — the median July temperature for a missing July temperature. But avoid filling missing values with possible values: filling missing number-of-children with 0 makes it impossible to distinguish people whose information is missing from people who have no children.

The age-zero bug

A frontend stopped asking for age, so age went missing, and the model filled it with 0 — a value the model never saw during training. It produced garbage. Nothing errored: the pipeline ran, the imputation rule fired as written, and the model was asked to reason about newborns.

Key insight

“There is no perfect way to handle missing values.” Deletion risks losing information and accentuating bias; imputation risks injecting bias, adding noise, or data leakage. The choice is between two known costs, so the useful question is which cost this particular column can afford.

Leak or safe?

Sort

Sort each practice by whether it leaks information from the test set into training.

Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

Leaks

Drop items here

Safe

Drop items here

6 pieces left to place

Test your knowledge

20 questions

Question 1 of 20

Score: 0

Income missing because high earners decline to disclose it is an example of:

The short version

In more depth

Feature Engineering & Data Leakage

"Practical Lessons from Predicting Clicks on Ads at Facebook" (He et al., 2014) claimed having the right features is the most important thing — once you have a workable model, the right features give a bigger boost than clever algorithmic techniques such as hyperparameter tuning.

Missing values — three kinds

Kind	Meaning	The trap
MNAR	Missing because of the true value itself	Removing rows removes information — the fact that income is missing is itself a signal
MAR	Missing because of another observed variable	Removing rows creates bias — drop missing-age and you drop every gender-A respondent
MCAR	No pattern	"This type of missing is very rare."

Row deletion works for MCAR when the affected count is small, such as under 0.1%. Avoid filling missing values with possible values — filling number-of-children with 0 makes missing indistinguishable from childless. Cautionary bug: a frontend stopped asking for age, the pipeline filled it with 0, and the model never saw 0 during training.

"There is no perfect way to handle missing values."

Scaling

x' = (x − min)/(max − min) · to [a,b] · x' = (x − x̄)/σ. "Empirically, I find the range [−1, 1] to work better than [0, 1]." Log transformation for skew.

Two notes that matter later: scaling is a common source of data leakage, and it requires global statistics — so at inference you reuse the training statistics, which means you must retrain often.

Discretization: "In practice, I've rarely found discretization to help." The boundary problem — $34,999 is now completely different from $35,000, which is treated the same as $100,000.

The hashing trick

Amazon had over two million brands in 2019. Three encodings, each failing one step later: integer indices crash; an UNKNOWN bucket means the model never saw UNKNOWN; top-99%-plus-UNKNOWN means 20 new brands in an hour all get treated like unpopular brands.

The hashing trick (popularised by Vowpal Wabbit) fixes the number of encoded values in advance without knowing the category count. An 18-bit space gives 2¹⁸ = 262,144 indices. Collisions are random, so a new brand can collide with any brand — and Booking.com found that even at a 50% collision rate, log-loss performance loss is under 0.5%.

Data leakage

"Data leakage refers to the phenomenon when a form of the label 'leaks' into the set of features used for making predictions, and this same information is not available during inference."

Six causes: random splits of time-correlated data · scaling before splitting · filling missing data with test statistics · duplication before splitting (3.3% of CIFAR-10 and 10% of CIFAR-100 test images have duplicates in training) · group leakage · leakage from the data generation process (the hospital that routed suspected cancer patients to a better scanner).

Five detections: per-feature predictive power (two features can be clean individually and leak together — start date and end date give away tenure) · ablation studies · suspicion of any new feature that helps a lot · care with the test split · subject matter experts.

Good features

Five costs of too many: leakage · overfitting · memory · latency · technical debt.

Facebook's ads team: the top 10 features are responsible for about half the total feature importance, while the last 300 contribute less than 1%.

Generalization: coverage (and the exception — a feature in 1% of data where 99% are POSITIVE is useful) and distribution overlap (DAY_OF_THE_WEEK with train Mon–Sat and test Sun will not generalise).

Interview Questions

Google

References & Further Reading

Stanford MLSys Seminar Episode 5 — Principles of Good Machine Learning Systems DesignYouTube · Stanford MLSys

video

Practical Lessons from Predicting Clicks on Ads at Facebook (He et al., 2014)Meta Research

paper

Feature hashing — scikit-learnscikit-learn Docs

docs

All ML Systems Design topics

Love this visualization?

100% free • No credit card required