Labeling & Weak Supervision
ML Engineer path · lesson 68 of 80Accurate phonetic transcription takes 400 times longer than the speech, so one hour is almost three months of work. Watch three annotators find three different answers, drag a labeling window that quietly undercounts clicks, and see eight hours of labeling functions match a year of hand labels.
Key insight
Karpathy's answer reframes the whole thing. If labeling is an ongoing function rather than a project, the right question is not “how do we get these labelled” but “what does our labeling capability look like in two years” — which leads to very different tooling decisions.
Where do the labels come from?
Sort each description into the labeling approach.
Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.
Hand labeling
Drop items hereWeak supervision
Drop items hereSemi-supervision
Drop items hereActive learning
Drop items hereTest your knowledge
Asked how long his team would need an in-house labeling team, Andrej Karpathy replied:
The short version
Marking exam papers. You can hire markers, which costs money and time and means strangers read every script. Or you can set questions that mark themselves — and most of the craft is in finding which questions those are.
In more depth
Labeling & Weak Supervision
Asked how long he'd need an in-house labeling team, Andrej Karpathy replied: "How long do we need an engineering team for?"
Hand labels — three problems
Slow labeling slows iteration: adding an ANGRY class to a NEGATIVE/POSITIVE sentiment model needs a relabeling pass, and the existing model degrades meanwhile.
Label multiplicity
On one sentence: annotator 1 found 3 entities, annotator 2 found 6, annotator 3 found 4. "The higher the level of domain expertise required, the higher the potential for annotating disagreement." If experts can't agree, what does human-level performance even mean?
Two fixes: a clear problem definition ("pick the entity that comprises the longest substring") and incorporating it into annotator training.
Data lineage: a good model on 100K samples gets another million crowdsourced samples and performance decreases — and you can't fix it once the data is mixed and indistinguishable.
Natural labels
The system evaluates its own predictions: Google Maps ETA, stock prices, and recommender systems (click = POSITIVE, no click after a window = NEGATIVE). Survey of 86 companies: 63% work with tasks that have natural labels.
Feedback loop length — prediction to feedback. Amazon related products: minutes. Articles and videos: hours. Stitch Fix-style clothing: weeks. Fraud detection: a dispute window of one to three months.
Window length is a speed/accuracy trade-off. Twitter Ads (early 2021): most ad clicks happen within five minutes, but some happen hours later — so the label type underestimates the actual click-through rate.
Four techniques (Table 4-2)
| Method | How | Ground truths required? |
|---|---|---|
| Weak supervision | Noisy heuristics generate labels | No, but a few help guide development |
| Semi-supervision | Structural assumptions generate labels | Yes — a small number of seeds |
| Transfer learning | Reuse a model pretrained on another task | No for zero-shot; far fewer for fine-tuning |
| Active learning | Label the samples most useful to the model | Yes |
Snorkel's labeling functions come in four kinds: keyword heuristics, regular expressions, database lookups, and the outputs of other models. They're noisy and may conflict, so you must combine, denoise and reweight them.
Stanford Medicine (Dunnmon et al.): labels from a single radiologist after eight hours of writing labeling functions performed comparably to almost a year of hand labeling.
Interview Questions
References & Further Reading
Love this visualization?
Sign up to track your progress, earn achievements, use the AI assistant, and access your personal learning dashboard.
100% free • No credit card required