algorithms.live
    ML-SYSTEMS
    beginner

    ML System Requirements

    ML Engineer path · lesson 61 of 80
    The idea
    A restaurant kitchen judged on how good the food is versus how many people come back. Those are related and they are not the same number, and a chef who optimises the first without ever checking the second eventually finds out the hard way.
    How to use
    Drag the take-rate sliders to see how Netflix turned a recommendation metric into a business one — and note that quality plays, not plays, is the numerator.
    How a short-lived ML project dies
    Data scientists hack ML metrics→Managers see no business movement→The project is killed
    Possibly letting go of the team along the way. The cause is not bad modelling — it is that nobody ever established which business number the ML number was supposed to move.
    Netflix's take-rate — a metric that bridges both
    take-rate = quality plays ÷ recommendations a user sees
    20
    6
    30.0%
    take-rate
    streaming hours ↑, cancellations ↓
    what Netflix found it predicts
    Netflix found that a higher take-rate also results in higher total streaming hours and lower subscription cancellation rates — which is what makes it a good bridge metric. It is measurable by the ML team on every request, and it is demonstrably connected to the thing the business is actually paid for.
    The effect is often ambiguous, and that is normal
    A more personalised model can make customers happier so they spend more — or solve their problems faster so they spend less. Which one happened is usually only answerable by A/B testing, and companies choose the model with better business metrics regardless of whether it has better ML metrics. Worse, in a pipeline — an anomaly model feeding a rules layer feeding human experts feeding a response process — when the process fails it may be impossible to tell whether the ML component had anything to do with it.

    Key insight

    “Magically: possible. Overnight: no.” Google has been investing in ML for decades, and the Algorithmia 2020 survey shows the maturity curve plainly: among companies with models in production for over five years, almost 75% can deploy a model in under 30 days; among those just getting started, 60% take over 30 days.

    The four requirements

    Sort

    Sort each description into the requirement it belongs to.

    Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

    Reliability

    Drop items here

    Scalability

    Drop items here

    Maintainability

    Drop items here

    Adaptability

    Drop items here
    5 pieces left to place

    Test your knowledge

    20 questions
    Question 1 of 20
    Score: 0

    The book describes a common way ML projects die. What is the first step?

    The short version

    A restaurant kitchen judged on how good the food is versus how many people come back. Those are related and they are not the same number, and a chef who optimises the first without ever checking the second eventually finds out the hard way.

    In more depth

    ML System Requirements

    Business vs ML objectives

    Data scientists get excited about improving accuracy from 94% to 94.2%. Most companies don't care unless it moves a business metric. The failure pattern: data scientists hack ML metrics → managers see no business movement → the project is killed.

    Ad CTR prediction and fraud detection are the most popular use cases precisely because ML performance maps easily to business metrics.

    Netflix's take-rate = quality plays ÷ recommendations a user sees. Netflix found a higher take-rate also results in higher total streaming hours and lower subscription cancellation rates.

    The effect is often ambiguous — a more personalised model can make customers spend more or less — so A/B testing decides, and companies pick the model with better business metrics regardless of ML metrics.

    Algorithmia 2020: among companies with models in production over five years, almost 75% can deploy in under 30 days; among beginners, 60% take over 30 days.

    Four requirements

    1.Reliability — "perform the correct function at the desired level of performance even in the face of adversity". Correctness is hard to determine: model.predict() can succeed and be wrong. ML systems fail silently.
    2.Scalability — three axes: complexity (1 GB logistic regression → 16 GB network), traffic (10,000/day → 1–10 million), model count (one startup had 8,000 models for 8,000 customers). Scaling out ≠ scaling up. Autoscaling failures are expensive — Amazon's failed on Prime Day, an hour estimated at $72–99 million.
    3.Maintainability — ML engineers, DevOps engineers and SMEs with different tools. Let each group use the tools they're comfortable with. Version code, data and artifacts; make models reproducible; debug jointly without finger-pointing.
    4.Adaptability — discover improvements and update without service interruption.

    The loop

    The ad-serving workflow runs thirteen steps and returns to step one, with four discoveries along the way — wrong labels, a 99.99% negative class, staleness, and a wrong objective. None is a modelling failure.

    Six phases: project scoping → data engineering → ML model development → deployment → monitoring and continual learning → business analysis → back to scoping.

    Interview Questions

    Netflix
    Meta
    Amazon
    Spotify
    Google
    Microsoft
    Databricks
    Uber
    Amazon
    Uber
    Airbnb
    Databricks
    Google
    Meta
    LinkedIn
    Stripe

    References & Further Reading

    Stanford MLSys Seminar Episode 5 — Principles of Good Machine Learning Systems DesignYouTube · Stanford MLSys
    video
    Machine Learning Systems DesignChip Huyen
    course
    Hidden Technical Debt in Machine Learning Systems (Sculley et al., 2015)NeurIPS
    paper
    All ML Systems Design topics

    Love this visualization?

    Sign up to track your progress, earn achievements, use the AI assistant, and access your personal learning dashboard.

    100% free • No credit card required