Catastrophic forgetting is:

A network abruptly forgetting old information upon learning new information — One of the two reasons per-sample updating is rare. The other is cost.

The second reason companies avoid per-sample updating is that:

Most hardware backends were designed for batch processing — So it wastes compute and cannot exploit data parallelism.

In the champion-and-challenger pattern, the existing model is:

Never changed directly; a replica is updated instead — And the champion is replaced only if the challenger proves better. A company may run several challengers at once.

The book says continual learning is not about retraining frequency but about:

The manner in which the model is retrained — Which is why daily stateless retraining and daily stateful training are very different things at the same cadence.

Stateful training is called that rather than "stateful retraining" because:

There is no re-training — the model continues from its last state — It is also known as fine-tuning or incremental learning.

Grubhub moved from daily stateless retraining to daily stateful training. Training compute cost fell by a factor of:

45 — And purchase-through rate increased by 20%, at the same daily cadence.

The privacy property of stateful training that the book calls overlooked is that:

A data sample is used only once, so it need never be stored permanently — "Today's let's-keep-track-of-everything practice still makes many companies reluctant to throw away data."

Stateful training is mostly applied to:

Data iteration, where architecture and features stay the same — Model iteration still requires training from scratch, which is what knowledge transfer and model surgery try to avoid.

ML-SYSTEMS

advanced

Continual Learning

ML Engineer path · lesson 75 of 80

Continual learning is not per-sample updating, and it is not about how often you retrain. Slide a micro-batch size, toggle stateless retraining against the stateful training that cut Grubhub 45x, climb the four stages, and price data freshness with the book's own experiment.

The idea

A ship's navigator who takes a new bearing every few minutes and adjusts, against one who recalculates the entire voyage from the home port each morning. Both end up somewhere. Only one of them can respond to a storm.

How to use

On the last tab, move the slider between the three training windows — every one of them is scored on the same December data, which is the whole point of the experiment.

Catastrophic forgetting

“The tendency of a neural network to completely and abruptly forget previously learned information upon learning new information.” One sample at a time is the most aggressive version of that risk.

Cost

Most hardware backends today were designed for batch processing, so updating on one sample at a time wastes compute and cannot exploit data parallelism.

So companies update in micro-batches

Micro-batch size

512 examples

3,906

updates per day at 2M events

2.7

updates per minute

This is the range the book names: after every 512 or 1,024 examples. The optimal number is task dependent.

Champion and challenger

Never change the existing model directly. Replace the champion only if a challenger proves better. A company may have several challengers at once, and handling a failed one is more sophisticated than discarding it.

Why the book avoids two common names

“Online learning” makes people think of online courses, and some use it for the strict per-sample setting — of which continual learning is a generalization. “Continuous learning” is worse: it means either learning with each incoming sample or, from the DevOps side, continuous delivery of ML. “Due to the ambiguity of the term 'continuous learning,' I hope that the community can stay away from this term altogether.”

Four stages of retraining

Sequence

Order the stages a team climbs toward continual learning.

Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

1Drop a step here
2Drop a step here
3Drop a step here
4Drop a step here

4 pieces left to place

Test your knowledge

20 questions

Question 1 of 20

Score: 0

Continual learning, as the book uses the term, means updating the model:

The short version

In more depth

Continual Learning

Two things it is not

It is not per-sample updating. Two reasons: catastrophic forgetting — "the tendency of a neural network to completely and abruptly forget previously learned information upon learning new information" — and cost, since most hardware backends today were designed for batch processing. Companies update in micro-batches, after every 512 or 1,024 examples.

It is not about frequency. "Continual learning isn't about the retraining frequency, but the manner in which the model is retrained."

Champion and challenger: never change the existing model directly. Create a replica, update the replica, replace the champion only if the challenger proves better.

Stateless retraining vs stateful training

	Stateless retraining	Stateful training
What it does	trains from scratch each time	continues from the last checkpoint
Data needed	all data from the last three months	only the last day's
Who does it	what most companies do	Grubhub, TikTok

Grubhub: daily stateless retraining → daily stateful training cut training compute cost 45 times and raised purchase-through rate 20%.

The overlooked property: a data sample is used only once, so it is possible to train without storing data in permanent storage.

Data iteration (same architecture, new data) is where stateful training applies. Model iteration (new feature or new architecture) still needs training from scratch — knowledge transfer (Google, 2015) and model surgery (OpenAI, 2019) try to avoid it, but "I'm not aware of any clear results in the industry."

Three reasons to want it

1.Sudden distribution shifts — a quiet Thursday evening, a big event, and riders switching to a competitor.

2.Rare events — Black Friday happens once a year. In 2019 Alibaba acquired Data Artisans, the team behind Apache Flink, for $103 million, with Singles Day recommendations as the flagship use case.

3.Continuous cold start — Coveo found it is common for an ecommerce site to have more than 70% of their shoppers visit fewer than three times a year. TikTok adapts its recommender to each user within minutes.

"'Why continual learning?' should be rephrased as 'why not continual learning?' Continual learning is a superset of batch learning."

Three challenges

Fresh data access — pull from Kafka or Kinesis before the warehouse. Label computation is the hidden cost: the app registers user A clicked product 32345 at 10:33 p.m. and the system must look back through logs to see whether it was recommended and by what query. Confluent is a $16 billion company as of October 2021.

Evaluation — "the biggest challenge is in making sure that this update is good enough to be deployed." Microsoft's Tay was shut down 16 hours after launch. A payment company needs ~two weeks to see enough fraud transactions, so it can only update every two weeks.

Algorithms — only bites matrix-based and tree-based models updated hourly. Hoeffding Tree variants exist but are not yet widespread. And the feature code must work on partial data, so compute scaling statistics online.

Four stages

1.Manual, stateless retraining — "a vast majority of companies outside the tech industry... are in this stage."

2.Automated retraining — 5 to 10 models in production; "most companies with somewhat mature ML infrastructure are in this stage." Frequency set on gut feeling.

3.Automated, stateful training — needs model lineage, and "no existing model store has this model lineage capacity."

4.Continual learning — time-based, performance-based, volume-based (labeled data up 5%) and drift-based triggers. The holy grail is continual learning plus edge deployment.

How often

Facebook, 2014: weekly → daily retraining of ad CTR cut loss by 1%, enough to switch the pipeline. And the budgeting rule: "if finding a better model architecture requires 100X compute... whereas updating the same model on data from the last three hours requires only 1X... you'll be better off iterating on data."