algorithms.live
    ML-SYSTEMS
    intermediate

    Sampling Training Data

    ML Engineer path · lesson 67 of 80
    The idea
    Polling a city by standing outside one supermarket. You will get a thousand responses by lunchtime, and every one of them is from someone who shops there, at that hour, on a weekday. The sample size was never the problem.
    How to use
    On the rare-class tab, set the class rate to 0.01% and the sample to 1% — then watch how much you have to sample before the class reliably survives.
    Four ways of selecting data where the selection criteria are not based on probability. All four are common, and all four produce samples that are not representative.
    Convenience sampling
    Samples are selected by availability.
    Popular because it is convenient — which is the whole problem.
    These samples are not representative and are riddled with selection biases — and yet in many cases data selection is still driven by convenience. That is not a criticism of individual teams; it is a description of how most datasets come to exist.

    Key insight

    Notice that none of the four is wrong in a way you can see from inside the sample. A convenience sample of a million points looks exactly like a representative sample of a million points, right up until the model meets someone the supermarket queue did not contain.

    Probability or not?

    Sort

    Sort each sampling method by whether every example has a known chance of selection.

    Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

    Nonprobability

    Drop items here

    Probability

    Drop items here
    6 pieces left to place

    Test your knowledge

    20 questions
    Question 1 of 20
    Score: 0

    The book prefers "training data" to "training dataset" because "dataset" denotes a set that is:

    The short version

    Polling a city by standing outside one supermarket. You will get a thousand responses by lunchtime, and every one of them is from someone who shops there, at that hour, on a weekday. The sample size was never the problem.

    In more depth

    Sampling Training Data

    The book says training data, not training dataset, because "dataset" denotes a set that is finite and stationary, and data in production is neither.

    Nonprobability sampling — four criteria

    Convenience — selected by availability.
    Snowball — future samples selected from existing ones (scrape a few Twitter accounts, then everyone they follow).
    Judgment — experts decide what to include.
    Quota — fixed quotas per slice with no randomization (100 responses from each of under-30, 30–60, over-60, regardless of the actual distribution).

    These samples are not representative and are riddled with selection biases. Three real cases: language models trained on Wikipedia, Common Crawl, Reddit; sentiment analysis on IMDB and Amazon reviews, biased toward people willing to review; self-driving data from Phoenix (lax regulations) and the Bay Area — both sunny. Waymo expanded to Kirkland, Washington in 2016 for the rain, and sunny data still dominates.

    Random sampling

    Simple random — equal probability. Rare categories may not appear: a class at 0.01% is unlikely to be picked when you sample 1%.
    Stratified — divide into strata and sample within each. Not always possible — especially for multilabel tasks, where one sample belongs to several groups.
    Weighted — each sample gets a selection weight (A, B, C at 50/30/20% ⇒ 0.5, 0.3, 0.2). Lets you leverage domain expertise and correct distribution mismatch.
    Weighted sampling ≠ sample weights. The first selects what to train on; the second assigns importance inside the loss, and changing it can shift the decision boundary significantly.
    Importance sampling — sample from an easier Q(x) and reweight by P(x)/Q(x). Q can be anything as long as Q(x) > 0 whenever P(x) ≠ 0. Used in policy-based RL.

    Reservoir sampling

    For streaming data of unknown length that won't fit in memory.

    1.Put the first k elements in the reservoir.
    2.For each incoming n-th element, generate random i, 1 ≤ i ≤ n.
    3.If 1 ≤ i ≤ k, replace the i-th reservoir element; else do nothing.

    Two guarantees: every item has an equal probability of selection (k/n), and you can stop at any time with correct probabilities.

    Interview Questions

    Google
    Waymo
    Meta
    Scale AI
    Waymo
    Tesla
    Google
    Meta
    Stripe
    PayPal
    Google
    Amazon
    Google
    Meta
    Confluent
    Databricks

    References & Further Reading

    Stanford MLSys Seminar Episode 5 — Principles of Good Machine Learning Systems DesignYouTube · Stanford MLSys
    video
    Machine Learning Systems DesignChip Huyen
    course
    Reservoir samplingWikipedia
    docs
    All ML Systems Design topics

    Love this visualization?

    Sign up to track your progress, earn achievements, use the AI assistant, and access your personal learning dashboard.

    100% free • No credit card required