Precision/Recall Trade-off & PR vs ROC

Question 1

What does a binary classifier actually produce before any class label exists?

Accepted Answer

A score, which is then compared against a threshold — Geron: the classifier computes a score from a decision function, and the threshold decides the class. That is why the threshold is a choice, not a property of the model.

Question 2

Scikit-Learn's SGDClassifier uses which default decision threshold?

Accepted Answer

0 — Its decision_function() returns a raw score, and the default threshold is 0. Models exposing predict_proba() use 0.5 on a probability instead.

Question 3

As the decision threshold is raised, recall…

Accepted Answer

Can only go down — Raising the threshold can only turn positives into negatives, never the reverse, so recall is monotonic — which is why its curve is smooth.

Question 4

Why is the precision curve bumpier than the recall curve?

Accepted Answer

Precision can go down when the threshold is raised — Geron's example: moving the threshold one instance to the right takes precision from 4/5 (80%) down to 3/4 (75%). It usually rises, but not monotonically.

Question 5

Precision is defined as:

Accepted Answer

TP / (TP + FP), the share of predicted positives that were right — Of everything the model called positive, how much of it actually was. TP / (TP + FN) is recall, and (TP + TN) over everything is accuracy.

Question 6

A model makes exactly one positive prediction and gets it right. Its precision is:

Accepted Answer

100%, and the model is nonetheless useless — Precision = 1/1. Geron uses exactly this to explain why precision is never reported alone — it is trivially maximised by refusing to speak.

Question 7

The F1 score is the harmonic mean of precision and recall rather than the arithmetic mean because the harmonic mean:

Accepted Answer

Gives much more weight to low values — So a classifier only scores well when both numbers are high. One terrible value drags F1 down in a way an average would hide.

Question 8

F1 favours classifiers with similar precision and recall. Why is that sometimes the wrong thing to want?

Accepted Answer

Because many real tasks deliberately live at one end of the trade-off — A kid-safe video filter wants precision at the cost of recall; shoplifter detection accepts 30% precision for 99% recall.

Question 9

Geron's response to "let's reach 99% precision" is:

Accepted Answer

"At what recall?" — Any precision is reachable by raising the threshold. The number is meaningless without the recall it cost.

Question 10

Aiming for 90% precision on the MNIST 5-detector left recall at roughly:

Accepted Answer

48% — Precision 0.9000 at a threshold of about 3,370 — and "for many applications, 48% recall wouldn't be great at all."

Scores, and the line drawn through them

The same model, judged two ways

Which error is worse here?

Test your knowledge

What does a binary classifier actually produce before any class label exists?

Learn these first

Interview Questions

References & Further Reading

Love this visualization?