ML Systems: An Overview
ML Engineer path · lesson 60 of 80Before the model, a harder question: should this be machine learning at all? Step the five-part definition as a checklist, compare research against production on five axes, and watch one 3,000 ms request drag an average to 390 ms.
Key insight
The clause that fails most often in practice is the fifth. A project can satisfy every other condition and still fail because the production distribution drifts away from the training distribution — which is not a modelling problem and cannot be fixed by a better model.
Should this be ML at all?
Sort each property of a problem by which way it points.
Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.
Points toward ML
Drop items herePoints away from ML
Drop items hereTest your knowledge
Which of these fails the "complex patterns" clause of the definition of ML?
The short version
Hiring a forecaster instead of writing a rule. A rule is right every time and only for the case you wrote it for. A forecaster is usually right, generalises to cases you never considered, needs to be paid continuously, and will quietly get worse as the world changes. Whether that trade is good depends entirely on the problem.
In more depth
ML Systems: An Overview
An ML system is not the algorithm. It is the business requirements, the interface, the data stack, the logic for developing, monitoring and updating models, and the infrastructure.
MLOps is a set of tools and best practices for bringing ML into production. ML systems design takes a system approach to MLOps — considering the system holistically so all components and stakeholders satisfy the objectives.
When to use ML — a five-part checklist
"Machine learning is an approach to (1) learn (2) complex patterns from (3) existing data and use these patterns to make (4) predictions on (5) unseen data."
Patterns are different from distributions: we know a fair die's distribution and there are no patterns in it. Zip code → state is a lookup; rental price from listing characteristics is not.
Four amplifiers: it's repetitive · wrong predictions are cheap · it's at scale · the patterns keep changing.
Three stop conditions: it's unethical · simpler solutions do the trick · it's not cost-effective. Even then, break the problem into components and use ML for some.
Research vs production — Table 1-1
| Research | Production | |
|---|---|---|
| Requirements | SOTA on benchmarks | Different stakeholders, different requirements |
| Computational priority | Fast training, high throughput | Fast inference, low latency |
| Data | Static | Constantly shifting |
| Fairness | Often not a focus | Must be considered |
| Interpretability | Often not a focus | Must be considered |
Stakeholders disagree
The restaurant app with a 10% service fee: ML engineers want a better model, sales wants expensive restaurants, product wants under 100 ms, the platform team wants no updates, the manager wants margin. Model A (most clicked) and Model B (most money) are different models, and neither meets 100 ms. Resolution: decouple objectives — one model per objective, combined at serving time.
Latency
Not like traditional software
Software assumes code and data are separated. ML systems are part code, part data, part artifacts — so you must version the data too, and that's the hard part. Not all samples are equal: a cancerous lung scan among a million normal ones is worth far more.
Interview Questions
References & Further Reading
Related topics
Love this visualization?
Sign up to track your progress, earn achievements, use the AI assistant, and access your personal learning dashboard.
100% free • No credit card required