Exporting a model, also called serialization, saves two things:

The model definition and the parameter values — The definition is the structure — how many hidden layers, how many units. They are usually exported together.

A 2021 Algorithmia study found that among organizations with over 25,000 employees, the share running more than 100 models in production was:

41% — Against the myth that you only ever deploy one or two models at a time.

Booking.com has how many models in production?

150+ — Uber has thousands; Booking.com has 150+.

The term for software degrading over time even when nobody touches it is:

Software rot, also called bit rot — "Software doesn't age like fine wine. It ages poorly." ML adds data distribution shift on top.

The book says the right question is not "how often should I update my models" but:

How often can I update my models — Because the ceiling is almost always infrastructural, not statistical.

In 2015, AWS was deploying roughly once every:

11.7 seconds — Etsy deployed 50 times a day and Netflix thousands of times a day in the same year.

Weibo's iteration cycle for some ML models is about:

10 minutes — With similar numbers reported at Alibaba and ByteDance.

Batch prediction is also known as:

Asynchronous prediction — Online prediction is on-demand prediction, and synchronous prediction when sent over HTTP.

Netflix might generate movie recommendations for all users every:

Four hours — And that period is exactly the responsiveness problem — a new interest cannot surface until the next batch.

ML-SYSTEMS

intermediate

Model Deployment & Prediction Service

ML Engineer path · lesson 73 of 80

Deploying is easy if you ignore all the hard parts. Four myths against the Uber, Booking.com and Algorithmia numbers, batch against online prediction, a compression slider that reproduces the Roblox 7x result, and why the browser is an edge device.

The idea

Cooking to order against cooking a buffet. The buffet is efficient if you know roughly who is coming and what they want, and wasteful if most of the trays go untouched — and it cannot serve anyone who orders something not on it.

How to use

On the batch-or-online tab, drag the daily-active rate to see how much precomputed work a batch recommender throws away.

The myth

“You only deploy one or two ML models at a time.”

A ride-sharing app needs models for ride demand, driver availability, ETA, dynamic pricing, fraudulent transactions and customer churn — and 20 countries times 10 models is 200 models.

Uber has thousands of models in production. At any given moment Google has thousands of models training concurrently with hundreds of billions of parameters. Booking.com has 150+ models. A 2021 Algorithmia study found that among organizations with over 25,000 employees, 41% have more than 100 models in production.

Deployment frequency, back in 2015

Deploying is easy if you ignore all the hard parts

Wrapping predict in a Flask or FastAPI POST endpoint, containerising the dependencies and pushing to a cloud gives a functional deployment in an hour. The hard parts are millisecond latency for millions of users, 99% uptime, alerting the right person immediately, diagnosing what went wrong, and seamlessly shipping the fix. And separating the team that builds from the team that deploys causes high cross-team communication overhead, slow updates and hard debugging.

Key insight

Josh Wills, formerly of Google and Slack: “We are always trying to bring new models into production just as fast as humanly possible.” That is the posture myth 3 is arguing for — update frequency as a capability you build, rather than a schedule you negotiate.

Batch or online prediction?

Sort

Sort each trait into the prediction pattern it describes.

Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

Batch

Drop items here

Online

Drop items here

6 pieces left to place

Test your knowledge

20 questions

Question 1 of 20

Score: 0

Wrapping a predict function in a Flask endpoint, containerizing it and pushing to a cloud provider takes roughly:

The short version

A restaurant choosing between a buffet and a kitchen. The buffet cooks everything in advance — enormous throughput, nothing waits, and it cannot serve a dish nobody anticipated. The kitchen cooks on demand — anything on the menu, but every diner waits for the stove.

In more depth

Model Deployment & Prediction Service

"Deploying is easy if you ignore all the hard parts." Wrapping predict in a Flask or FastAPI endpoint, containerizing and pushing to AWS gives a functional deployment in an hour. The hard parts are millisecond latency for millions of users, 99% uptime, alerting the right person, diagnosing the cause, and shipping the fix seamlessly.

Exporting a model — also called serialization — has two parts, the model definition (how many hidden layers, how many units) and the parameter values. tf.keras.Model.save() produces SavedModel; torch.onnx.export() produces ONNX.

Four myths

1."You only deploy one or two models." A ride-sharing app needs ride demand, driver availability, ETA, dynamic pricing, fraud and churn — 20 countries x 10 models = 200. Uber has thousands in production. Booking.com has 150+. Google has thousands training concurrently with hundreds of billions of parameters. A 2021 Algorithmia study found 41% of organizations with over 25,000 employees run more than 100 models in production.

2."Performance stays the same if we do nothing." "Software doesn't age like fine wine. It ages poorly." The phenomenon is software rot or bit rot; ML adds data distribution shift, so a model performs best right after training.

3."You won't need to update much." "How often should I update my models?" is the wrong question. The right question is "How often can I update my models?" In 2015 Etsy deployed 50 times a day, Netflix thousands, and AWS every 11.7 seconds. Weibo's iteration cycle for some models is 10 minutes.

4."Most ML engineers don't need to worry about scale." Per the Stack Overflow Developer Survey 2019, more than half of respondents worked at a company of at least 100 employees.

Batch vs online

Three modes: batch prediction with batch features · online prediction with batch features · online prediction with batch and streaming features, also called streaming prediction.

	Batch (asynchronous)	Online (on-demand)
Frequency	periodical, e.g. every four hours	as soon as requests come
Useful for	accumulated data, no immediate need — recommender systems	fraud detection
Optimized for	high throughput	low latency

Streaming features are not online features. Online features are any feature used for online prediction, including batch features held in memory — item embeddings are online but not streaming.

Grubhub had 31 million users and 622,000 daily orders in 2020 — at 2% daily logins, batch prediction wastes 98% of the compute.

Compression

Low-rank factorization — SqueezeNets: AlexNet accuracy, 50x fewer parameters. MobileNets: K² + C instead of K²C, an 8-9x reduction at K=3.

Knowledge distillation — DistilBERT: 40% smaller, 97% of understanding retained, 60% faster. Works across architectures; needs a teacher.

Pruning — usually set the least useful parameters to 0, making the network sparse. Over 90% of nonzero parameters can go without compromising accuracy.

Quantization — "the most general and commonly used". 100M parameters at 32 bits = 400 MB. XNOR-Net's authors spun off Xnor.ai, acquired by Apple in early 2020 for a reported $200M.

Roblox, scaling BERT to 1+ billion daily requests on CPUs at 25,000 inferences per second under 20 ms: BERT → DistilBERT → dynamic shape input → quantize. The biggest boost was quantization: 7x lower latency, 8x higher throughput.

Cloud vs edge

Compiling: TPUs shipped February 2018; PyTorch support arrived September 2020. Intermediate representation is the middleman, and generating it is called lowering — "it's not translating because there's no one-to-one mapping."

A Stanford DAWN study found typical NumPy/pandas/TensorFlow workloads run 23 times slower in one thread than hand-optimized code. autoTVM needs about 70 trials to beat cuDNN for ResNet-50 on an NVIDIA TITAN X.

WebAssembly is supported by 93% of devices worldwide (September 2021) but runs 45% (Firefox) to 55% (Chrome) slower than native.