Google Search processes roughly how many queries per second?

63,000 — Which is about 227 million an hour — a genuinely unique requirement, and not a useful benchmark for anyone else.

Facebook was handling how much data a day back in 2014?

4 petabytes — Three orders of magnitude above Zillow's 2 TB biggest dataset in 2018.

Which of the four infrastructure layers is the most abstract from a data scientist's point of view?

Storage and compute — And the most commoditized — storage has become so cheap that most companies just store everything.

An instance capable of a million FLOPS running a job at 0.3 million FLOPS has a utilization rate of:

30% — Utilization is achieved throughput over capability, and it is near impossible to reach 100%.

FLOPS is a contentious metric partly because:

Vendors disagree on whether a fused operation counts as one or two — The other reason is that capability is not achieved throughput.

An AWS m5.xlarge has two CPU cores with two threads per core, which is:

Four vCPUs — A vCPU is roughly half a physical core.

The core argument for the public cloud over your own data centers is that:

Data science workloads are bursty, so peak provisioning is wasted — 1,000 cores one day a year and 10 the rest — owned hardware pays for the peak all year.

Synergy Research Group found that in 2020 enterprise spending on cloud infrastructure services grew 35% to almost $130 billion, while data center spending:

Dropped 6% to under $90 billion — The direction of travel, though not the whole story.

ML-SYSTEMS

advanced

MLOps Infrastructure

ML Engineer path · lesson 77 of 80

Most companies are at "reasonable scale" — terabytes, not petabytes. Pick your scale band, climb the four layers, read compute by utilization instead of capability, drag a burstiness slider past the point where owning hardware wins, and separate cron from a scheduler from an orchestrator.

The idea

Roads, water and power for a town. Nobody moves somewhere because of the sewer system, and nobody stays anywhere without one. The right question is never which infrastructure is most impressive, but how many people the town has.

How to use

On the cloud tab, drag the peak-days slider past about 144 days — that is where provisioning your own hardware for the peak starts to beat renting it.

The majority. Fraud detection, price optimization, churn prediction, a recommender. Data "in the order of gigabytes and terabytes, instead of petabytes, a day," and a data science team of 10 to hundreds of engineers — "from a 20-person startup to a company at Zillow's scale, but not at FAAAM scale." The term is credited to Jacopo Tagliabue, "You Do Not Need a Bigger Boat" (2021).

Data per day, on a log scale

Key insight

Naming the band is the decision. At ad hoc scale, buying an ML platform is an expensive way to replace a notebook; at unique-requirements scale, a vendor cannot sell to you at all because of “integration hell”. Everything else in this topic is aimed at the middle.

Which infrastructure layer?

Sort

Sort each piece into its layer of ML infrastructure.

Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

Storage & compute

Drop items here

Resource management

Drop items here

ML platform

Drop items here

Development environment

Drop items here

5 pieces left to place

Test your knowledge

20 questions

Question 1 of 20

Score: 0

The term "reasonable scale" describes companies handling data on the order of:

The short version

In more depth

MLOps Infrastructure

"Bringing ML models to production is an infrastructural problem."

Three scale bands

Ad hoc / single use case — "Jupyter Notebooks, Python, and Pandas would be their best friends."

Unique requirements — self-driving cars responding within milliseconds; Google Search processes 63,000 queries a second, about 227 million an hour. "Google extended their internal cloud infrastructure to the public, resulting in Google Cloud Platform."

Reasonable scale — the majority. Gigabytes and terabytes a day, instead of petabytes; 10 to hundreds of engineers; "from a 20-person startup to a company at Zillow's scale, but not at FAAAM scale." Term credited to Jacopo Tagliabue, "You Do Not Need a Bigger Boat" (2021).

Company	Data per day
Zillow (2018), biggest dataset	2 TB uncompressed
Uber (2018), to their data lake	tens of TB
Facebook (2014)	4 PB

Four layers

Development environment (least abstract) → ML platform → Resource management → Storage and compute (most abstract). Infrastructure is "the set of fundamental facilities and systems that support the sustainable functionality of households and firms."

Compute

Utilization = FLOPS the job runs / FLOPS the unit is capable of. A million-FLOPS instance running 0.3 million is 30% utilization. "It's near impossible to achieve 100% utilization rate. Depending on the hardware backend and the application, the utilization rate of 50% might be considered good or bad."

FLOPS is contentious because vendors disagree on what an operation is, and because capability is not throughput. The benchmark is MLPerf. A typical ML workload requires 4-8 GB of memory; 16 GB is enough for most. A vCPU is about half a physical core — an m5.xlarge has 2 cores × 2 threads = 4 vCPUs. Kubernetes' smallest deployable unit is the pod.

Cloud vs own

*"Imagine if your workloads need 1,000 CPU cores one day of the year and only 10 CPU cores the rest of the year."* "Data science workloads are bursty." But the cloud is elastic, not infinite — AWS EC2's largest instance is X1e, 128 vCPUs and almost 4 TB of memory, at $26.688/hour.

Synergy Research Group, 2020: cloud infrastructure spending +35% to almost $130B; data center spending −6% to under $90B.

a16z: cloud is ~50% of cost of revenue for public software companies; $100B of market value lost across 50 companies.

Dropbox's 2018 S-1: $75M saved over two years, largely from repatriation.

Gartner 2019: 81% of organizations work with two or more public cloud providers — though "nobody in their right mind intends to use multicloud."

Development environment

Ville Tuulos: "If you have time to set up only one piece of infrastructure well, make it the development environment for data scientists."

The escalation: unpinned torch → pin the version · a bug only in Python 3.8 and below → pin Python · an M1 MacBook losing a full day → move to the cloud.

Notebook statefulness cuts both ways: rerun only step 4 after a failure, but "you can run cell 2, 3, then 4 or cell 4, 3, then 2. This makes notebook reproducibility harder." Netflix built Papermill and Commuter.

Four benefits of a cloud dev environment: IT support ("1,000 different local machines" vs one instance type) · remote work · security · and the biggest, it reduces the gap between dev and production. GitHub Codespaces auto-shuts down after 30 minutes idle; 4 vCPUs and 8 GB is about $0.1/hour, ~$73/month if never shut off.

Containers and orchestration

Dockerfile → image → container. Two containers because featurizing is fast but memory-hungry while training is slow and needs less memory — run featurizing on CPU instances and training on GPU instances. Docker Compose manages containers on a single host only; Kubernetes "creates a network for containers to communicate and share resources."

cron cannot express run B if A succeeds and run C if A fails. Schedulers are cron programs that handle dependencies. Orchestrators provision the machines.

**Schedulers are concerned with when to run jobs and what resources are needed. Orchestrators are concerned with where to get those resources.**

Airflow (Airbnb, 2014) is monolithic, with non-parameterized and static DAGs. Prefect fixes the last two. Argo containerizes every step but is YAML-only and Kubernetes-only. Kubeflow Pipelines is built on Argo and is currently more popular. Metaflow uses Python decorators — @batch(gpu=2, memory=16000) on one step and nothing on another — so different steps of one workflow run in different environments.