algorithms.live
    ML-SYSTEMS
    intermediate

    Class Imbalance

    ML Engineer path · lesson 69 of 80
    The idea
    A smoke alarm that never goes off. In a building that has never had a fire, it is right every single night, and its accuracy is magnificent. The number you wanted was never accuracy.
    How to use
    On the metrics tab, switch between the two models — identical accuracy — and then flip which class counts as positive to see F1 move from 0.17 to 0.95 on the same predictions.
    Three reasons, and the second one is the one that explains why the problem does not simply go away with more training.
    click to step
    2. The model gets stuck on a simple heuristic
    Always predicting the majority class already gives 99.99% accuracy in the lung-cancer example, and gradient descent finds it very hard to beat — because adding a small amount of randomness to that heuristic makes accuracy worse, so every direction looks downhill.
    It happens in regression too
    Health-care bills are highly skewed: the median bill is low, but the 95th percentile is astronomical. A 100% error on a $250 bill is acceptable; a 100% error on a $10,000 bill is not. You may have to train the model to be better at the 95th percentile even if it reduces the overall metrics — which is a decision, not an optimisation.
    What the research says
    Japkowicz: sensitivity to imbalance increases with the complexity of the problem, and noncomplex, linearly separable problems are unaffected by all levels of class imbalance.
    Ding et al. (2017): very deep neural networks — over 10 layers, in 2017 — performed much better on imbalanced data than shallower ones. And binary imbalance is a much easier problem than multiclass imbalance.
    There is a real counterpoint worth stating: a good model should learn to model that imbalance if that is how the real world looks. The reason special techniques still exist is that developing a model good enough to do so is hard.

    Key insight

    Reason 2 is why this is not a data-volume problem. The majority-class heuristic sits in a broad, flat basin of the loss surface: perturb it slightly and accuracy drops, so gradient descent has no local signal telling it that a far better solution exists somewhere else entirely.

    Three levers for imbalance

    Sort

    Sort each technique by what it changes.

    Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

    Resample the data

    Drop items here

    Change the loss

    Drop items here

    Change the metric

    Drop items here
    6 pieces left to place

    Test your knowledge

    20 questions
    Question 1 of 20
    Score: 0

    Why does gradient descent struggle to escape the majority-class heuristic?

    The short version

    A smoke alarm that never goes off. In a building that has never had a fire, it is right every single night, and its accuracy is magnificent. The number you wanted was never accuracy.

    In more depth

    Class Imbalance

    With 900 NORMAL scans and 100 CANCER scans, a model that always answers NORMAL scores 90% accuracy and catches nothing.

    Three reasons it's hard

    1.Insufficient signal for the minority class — it becomes a few-shot learning problem.
    2.The model gets stuck on a simple heuristic. Always predicting the majority gives 99.99% accuracy in the lung-cancer example, and gradient descent finds it very hard to beat, because a small amount of randomness added to it makes accuracy worse.
    3.Asymmetric costs of error — and an unconfigured loss function treats all samples the same.

    It happens in regression too: health-care bills are highly skewed, and a 100% error on a $250 bill is acceptable; a 100% error on a $10k bill is not.

    Where it comes from

    Inherent to the problem — fraud detection (6.8¢ per $100 of cardholder spending, as of 2018), churn, disease screening, resume screening (98% eliminated at the initial screen), object detection.
    Sampling bias — nearly 85% of all emails are spam (Talos, May 2021), but spam is filtered before reaching a company's database.
    Labeling errors.

    Metrics — Table 4-7

    1,000 scans: 100 CANCER, 900 NORMAL.

    AccuracyPrecisionRecallF1
    Model A (catches 10/100)0.90.50.10.17
    Model B (catches 90/100)0.90.50.90.64

    F1, precision and recall are asymmetric. With CANCER positive, model A's F1 is 0.17; with NORMAL positive, it is 0.95. In scikit-learn's f1_score, pos_label defaults to 1.

    Under heavy imbalance ROC flatters a model: FPR = FP/(FP+TN), and the enormous TN count means a flood of false positives barely moves the curve. Davis and Goadrich (2006) argued for the Precision-Recall Curve, whose precision term has no true-negative cushion.

    Data-level: resampling

    Tomek links (1976) — undersampling: remove the majority-class member of close opposite-class pairs. Clearer boundary, less robust model.
    SMOTE (Chawla et al., 2002) — oversampling by convex combinations of existing minority points.
    Two-phase learning (Lee et al., 2016) — train on resampled data, then finetune on the original.
    Dynamic sampling (Pouyanfar et al., 2018) — oversample low-performing classes, undersample high-performing ones during training.

    Both SMOTE and Tomek links have only been proven effective in low-dimensional data.

    "When you resample your training data, never evaluate your model on resampled data."

    Algorithm-level: change the loss

    Cost-sensitive learning (Elkan, 2001) — a cost matrix C_ij; if POSITIVE→NEGATIVE is twice as costly, C_10 = 2 × C_01. You must define the matrix manually.
    Class-balanced loss — W_i = N ÷ count(class i).
    Focal loss (Lin et al., 2017) — lower probability of being right ⇒ higher weight. The only one that adapts during training.

    Interview Questions

    Google Health
    Stripe
    PayPal
    Amazon
    Stripe
    PayPal
    Mastercard
    Visa
    Amazon
    Meta
    Microsoft
    Intuit
    Meta
    Google
    NVIDIA
    Tesla

    References & Further Reading

    Stanford MLSys Seminar Episode 5 — Principles of Good Machine Learning Systems DesignYouTube · Stanford MLSys
    video
    SMOTE: Synthetic Minority Over-sampling Technique (Chawla et al., 2002)arXiv
    paper
    Focal Loss for Dense Object Detection (Lin et al., 2017)arXiv
    paper
    All ML Systems Design topics

    Love this visualization?

    Sign up to track your progress, earn achievements, use the AI assistant, and access your personal learning dashboard.

    100% free • No credit card required