With most deployment services, the trickier of the two is usually:

Batch prediction — Online prediction means hitting an endpoint; batch prediction means setting up batch jobs and storing the results.

In the debugging story, the DevOps engineer's first problem is that:

The model store does not say which data scientist owns the model — There might be 20 data scientists in the company; who should she ping? Tags are the on-call routing table.

Which of these is NOT one of the four causes the data scientist cannot rule out from the weights alone?

The serving hardware differs from the training hardware — The fourth is a problem in the data processing pipeline itself.

Which two artifacts do most model stores already hold?

Model definition and model parameters — The other six are the ones that would have answered the page.

In a model store, the "data" artifact is typically:

Pointers to where the data is stored, or its name and version — With DVC, this can be the DVC commit that generated the data.

Model generation code should record the number of experiments run, the range of hyperparameters considered, and:

The actual set of hyperparameters the final model used — "In many companies, this process is ad hoc, and data scientists don't even check in their notebooks."

Of the six top MLflow questions on Stack Overflow, how many are about storing and accessing artifacts?

Three — A fair indicator of where the hard part of a model store actually is.

Stitch Fix's in-house model store holds a link to the serialized model, the Python environment, owning-team tags, and:

The Git commit where the model generation code was created — Which makes the "I cannot reproduce it locally" problem resolvable.

ML-SYSTEMS

advanced

ML Platform: Model & Feature Stores

ML Engineer path · lesson 78 of 80

A DevOps engineer is paged and there are twenty data scientists. Rule out the four causes she cannot distinguish from the weights alone, click through the eight artifacts a model store should hold, separate the three unrelated problems called a feature store, and run the build-versus-buy staffing math.

The idea

A library that stores only the text of every book, with no author, no edition, no publication date and no catalogue. Every book is technically present. Nothing is findable, and nothing is checkable.

How to use

On the model store tab, click through the eight artifacts — the first two are what most teams store, and the debugging story on the previous tab is what the other six would have answered.

“Deployment is the most mature among all ML platform components.” Every major cloud has one — AWS SageMaker, GCP Vertex AI, Azure Azure ML, Alibaba Machine Learning Studio — alongside startups such as MLflow Models, Seldon, Cortex and Ray Serve.

The trap in the maturity

“While it’s usually straightforward to do online prediction at a smaller scale with most deployment services, doing batch prediction is usually trickier.”

And a distinction that trips teams up: batching requests for online prediction is not the same thing as batch prediction. Many companies end up running separate pipelines — for example Seldon for online prediction and Databricks for batch prediction.

Two questions to ask of any ML platform tool

Does it work with your cloud provider, or your own data center? “Nobody likes having to adopt a new cloud provider for another tool.”

Open source or managed service? Open source means self-hosting, so less to worry about on data security and privacy, but extra engineering time to maintain. A managed service puts your models and some of your data on their service, “which might not work for certain regulations” — virtual private cloud deployments exist partly to answer that.

Key insight

The open problem in this layer is ensuring model quality before deployment. Which turns into a concrete purchasing criterion: check whether the service makes shadow deployment, canary release and A/B testing easy, because a deployment tool that cannot express those is only half a deployment tool.

Model store or feature store?

Sort

Sort each responsibility into the store that owns it.

Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

Model store

Drop items here

Feature store

Drop items here

6 pieces left to place

Test your knowledge

20 questions

Question 1 of 20

Score: 0

The book calls which ML platform component the most mature?

The short version

A library that stores only the text of every book, with no author, no edition, no publication date and no catalogue. Every book is technically present. Nothing is findable, and nothing is checkable.

In more depth

ML Platform: Model & Feature Stores

The story that justifies the whole layer

Performance drops for a group of inputs. A DevOps engineer is alerted. "There might be 20 data scientists in the company; who should she ping?" She finds the right one, who cannot reproduce it locally. Four candidate causes:

1.The model in production is not the model she has locally — the wrong binary was uploaded.

2.The model is right but the feature list is wrong.

3.Both right, but the featurization code is outdated.

4.All three right, and something is wrong with the data processing pipeline.

And then: "What if that data scientist no longer has access to that notebook, or she has already quit or is on vacation?"

The eight artifacts

Model definition · Model parameters · Featurize and predict functions · Dependencies · Data (pointers, or a DVC commit) · Model generation code ("the number of experiments run, the range of hyperparameters considered, and the actual set the final model used") · Experiment artifacts · Tags (owner, task).

The scattering problem: "model definitions and model parameters might be in S3. Containers... in ECS. Data... in Snowflake. Experiment artifacts... in Weights & Biases. Featurize and prediction functions... in AWS Lambda." Tracked in a README — "but this file can be easily lost."

MLflow is "undoubtedly the most popular model store that isn't associated with a major cloud provider." And: three out of the six top MLflow questions on Stack Overflow are about storing and accessing artifacts in MLflow. Stitch Fix built their own, holding the serialized model, the Python environment, the Git commit where the generation code was created, and the owning team.

Deployment

"Deployment is the most mature among all ML platform components." SageMaker, Vertex AI, Azure ML, Machine Learning Studio; MLflow Models, Seldon, Cortex, Ray Serve.

The catch: *"while it's usually straightforward to do online prediction at a smaller scale with most deployment services, doing batch prediction is usually trickier."* And batching requests for online prediction is not batch prediction — many companies run Seldon for online and Databricks for batch.

Feature store — three unrelated problems

1.Feature management — "Back in 2017, Uber had about 10,000 features across teams!" A catalog with roles and sharing settings. Amundsen (Lyft), DataHub (LinkedIn).

2.Feature computation — cheap features recomputed on demand, expensive features computed once and stored. Acts like a data warehouse.

3.Feature consistency — training/inference skew, made worse when features are defined in Python and production is written in Java or C for performance, so you write the same features twice. "A key selling point of modern feature stores is that they unify the logic for both batch features and streaming features."

Market: took off around 2020. Feast is the most popular open source, strong on batch, not streaming. Tecton is fully managed but "their actual traction is slow because they require deep integration." A January 2022 survey of 95 companies found only about 40% use a feature store — and half of those built their own.

Build versus buy

"If you want to use fully managed Databricks clusters, you probably need only one engineer. However, if you want to host your own Spark Elastic MapReduce clusters, you might need five more people."

Three factors: the stage of the company · what you believe your competitive advantage is ("If it's something we want to be really good at, we'll manage that in-house") · the maturity of available tools, where late entrants hit "integration hell".

"Some people think that building is cheaper than buying, which is not necessarily the case." Plus the cost of innovation — custom infrastructure makes new technology hard to adopt.