Learning Curves

Question 1

A learning curve plots training and validation error against:

Accepted Answer

The training set size, or the training iteration — Evaluate the model at intervals on both sets and plot the results. sklearn's learning_curve() grows the training subset by default.

Question 2

Why does the training error start near zero on a learning curve?

Accepted Answer

Because a model can fit one or two instances perfectly — With only a couple of points there is nothing to disagree with. The error climbs as more instances arrive and the model can no longer pass through all of them.

Question 3

The signature of an underfitting model on a learning curve is:

Accepted Answer

Both curves plateauing close together and fairly high — Close and high. The model is too simple, so neither curve has anywhere good to go.

Question 4

Your learning curves show underfitting. Collecting more training data will:

Accepted Answer

Not help — you need a better model or better features — Geron states it flatly. This is the expensive wrong answer, and the curve shape is what saves you from it.

Question 5

Geron calls which feature of a learning curve "the hallmark of an overfitting model"?

Accepted Answer

A gap between the training and validation curves — The model performs better on data it has seen than on data it has not. That distance is the gap.

Question 6

Your learning curves show overfitting. Feeding the model more training data will:

Accepted Answer

Help — the gap narrows as validation error approaches training error — The opposite of the underfitting case, from the same control. That is precisely why you plot the curves before deciding what to buy.

Question 7

Generalization error decomposes into which three parts?

Accepted Answer

Bias, variance, and irreducible error — An important theoretical result of statistics and machine learning, and the frame the whole chapter rests on.

Question 8

In that decomposition, bias arises from:

Accepted Answer

Wrong assumptions, such as assuming linear data when it is quadratic — A high-bias model is most likely to underfit. Sensitivity to the particular sample is variance, and measurement noise is the irreducible term.

Question 9

Variance, in the bias/variance decomposition, comes from:

Accepted Answer

Excessive sensitivity to small variations in the training data — A model with many degrees of freedom, such as a high-degree polynomial, is likely to have high variance and overfit.

Question 10

The only way to reduce the irreducible error is to:

Accepted Answer

Clean up the data — fix broken sensors, remove outliers — It is the noise in the data itself, so no amount of modelling touches it. It is the floor your validation curve settles onto.

Error against training set size

Read the learning curve

Test your knowledge

A learning curve plots training and validation error against:

Learn these first

Interview Questions

References & Further Reading

Love this visualization?