algorithms.live

    Design a feature store serving both model training and real-time inference

    The shared infrastructure layer underneath recommendation, fraud detection, and CTR-prediction systems alike — a single source of truth for ML features that serves both slow, cheap batch training and fast, expensive real-time inference from the same underlying definitions.

    AI & Machine Learning
    8 components ~30 min

    1. Scope

    In scope

    • • Feature computation from both batch and streaming sources
    • • Dual storage: an offline store for training, an online store for serving
    • • Point-in-time correctness for the offline/training path
    • • A feature registry for sharing definitions across teams/models

    Out of scope

    • • The model training frameworks that consume the offline store (Spark/TensorFlow/etc. — assumed as given)
    • • Feature engineering itself (choosing what features to build) — this system serves features, it doesn't decide which ones matter

    2. Functional requirements

    Must have

    • • Serve features to a real-time inference request in single-digit milliseconds
    • • Serve historical, point-in-time-correct feature values for training data generation
    • • Guarantee the same feature computes to the same value whether read from the offline or online path (consistency)

    Nice to have

    • • A UI/API for discovering existing features to avoid duplicate work across teams
    • • Automatic staleness monitoring — alert when a feature hasn't been updated within its expected freshness window

    3. Non-functional requirements

    Online read latencyp99 < 10ms— this store sits directly in the inference request path for every consuming model
    Offline read throughputmust support scanning millions of historical rows for a single training job without timing out
    Correctnesspoint-in-time correctness is non-negotiable— a violation silently corrupts every downstream model's training data — the corruption is invisible until you audit the training labels
    Consistencyoffline and online values for the same feature at the same logical time must match exactly— even a small discrepancy between the two paths is a training/serving skew bug that degrades every model consuming that feature

    4. Back-of-envelope estimation

    Assumptions

    • • 10 consuming models across the organization; the highest-volume consumer is an ad-ranking system running ~50,000 inference requests/sec at peak
    • • Each inference request reads ~20 features from the online store (one feature-store read per feature per request)
    • • 100M active entities (users + items) in the system; 500 distinct features tracked per entity; 2 years of history retained in the offline store; each feature value averages ~50 bytes stored

    Derived numbers

    Online store read QPS (single top consumer)~1,000,000 reads/sec— 50,000 inference req/sec × 20 features per request = 1M feature reads/sec from the ad-ranking model alone; the remaining 9 models are lower-volume (recommender, fraud, etc.) and likely add another 100K-400K/sec aggregate — total sits comfortably in the 1-1.5M range
    Offline store size~5 PB— 100M entities × 500 features × 730 days × 50 bytes = 1.825 × 10^15 bytes ≈ 1.8 PB in the hot partition; with redundancy and format overhead (columnar encoding, replication factor ~3) the physical footprint reaches the low-PB range — consistent with "10s of TB to PB" for a 2-year window

    5. API design

    GET/online/features?entityId={id}&features={list}

    Low-latency read of current feature values for a given entity, used at inference time

    POST/offline/training-set

    Generate a point-in-time-correct training dataset for a given set of entities, features, and historical timestamps

    POST/registry/features

    Register a new feature definition (name, computation logic, source, owner)

    6. High-level design

    What each box in the reference diagram is responsible for:

    Feature Pipeline — Computes features from both batch sources (data warehouse tables, recomputed periodically) and streaming sources (Kafka events, computed incrementally) — critically, using the SAME transformation logic for both paths where the feature semantics overlap, to avoid training/serving skew
    Feature Registry — Metadata layer decoupling "what a feature means" (its name, computation logic, owner, freshness SLA) from where it's physically stored — lets multiple teams discover and reuse existing features instead of each building their own version of "user's 7-day purchase count"
    Offline Store — A data-warehouse-backed store (e.g. columnar storage) optimized for large historical scans used to generate training datasets — cheap per byte, not latency-optimized
    Online Store — A key-value store (e.g. Redis, DynamoDB) optimized for single-entity point lookups under 10ms — expensive per byte relative to the offline store, but that's the correct tradeoff for the serving path's latency requirement

    7. Data model

    Offline Store: (entity_id, feature_name, value, event_timestamp) — versioned by timestamp so a query can ask "what was this value AS OF time T"— the event_timestamp (not the write timestamp) is what makes point-in-time queries correct
    Online Store: entity_id → current feature vector, no history retained— only the latest value matters for serving — history is the offline store's job
    Feature Registry: feature_name → {owner, computation_logic_reference, source, freshness_sla, consuming_models}

    8. Deep dives

    Point-in-time correctness, worked through concretely

    Suppose you're building a training example for "did this user click this ad on March 1st." The feature "user's click-through rate over the last 7 days" must be computed using ONLY data from Feb 22-28 — not including March 1st itself (that would leak the label) and not using today's current CTR (which reflects weeks of data the model wouldn't have had access to at prediction time in production). Getting this right at scale — for millions of training examples, each needing a different "as of" timestamp — is what makes feature stores nontrivial infrastructure rather than "just a database." The offline store's timestamp-versioned schema exists specifically to make this query answerable correctly.

    Training/serving skew — the failure mode this whole system exists to prevent

    If the offline feature-computation code and the online feature-computation code are two separate implementations (even logically identical ones, written twice), they WILL eventually drift — a bug fix applied to one path and not the other, a subtle difference in how nulls are handled, a timezone bug. The model then trains on features computed one way and serves on features computed a subtly different way, degrading production performance in a way that's notoriously hard to debug (the model looks fine in offline evaluation, since offline evaluation uses the offline path too). The fix is architectural: share the actual transformation logic between both paths rather than reimplementing it twice, which is exactly what the Feature Pipeline component is for.

    How feature backfill works at scale — and why it's genuinely expensive

    When a team registers a brand-new feature — say, "user's 30-day purchase count, computed with a corrected definition of what counts as a purchase" — the online store can be populated going forward by the live pipeline. But the offline store needs HISTORICAL values of that feature going back potentially years, so that models can be trained on past data as if this feature had always existed. This is backfill: a one-time batch job that reprocesses the entire retention window of the raw source data through the new feature's computation logic and writes the timestamped output into the offline store. At scale, this is a genuinely expensive operation for several reasons. First, it involves scanning years of raw source data — potentially scanning petabytes of a data warehouse for millions of entities over a multi-year retention window. This scan is done via a distributed batch job (Spark being the most common choice), typically partitioned by time bucket so the work can be parallelized across a cluster. Second, the cluster resources needed for backfill compete with production batch jobs, so backfill jobs are usually run during off-peak hours or throttled to avoid starving the pipeline that keeps features fresh. Third, backfill jobs can take hours to days for features over large entity populations with long retention windows — during which time the feature is "partially backfilled" and can't be safely used for training on historical ranges that haven't been filled yet. A common pattern to reduce backfill cost is incremental materialization: store intermediate aggregation results (e.g. per-day or per-week sums) in addition to raw events, so a backfill of a rolling-window feature doesn't have to re-scan every raw event, just aggregate the pre-computed daily buckets. Feature stores like Feast and Tecton expose explicit "materialize_range" APIs that trigger exactly this kind of bounded backfill job, letting teams control how far back a new feature's history extends before they wire it into training.

    Feature versioning and schema evolution — why silent in-place changes are dangerous

    A feature's DEFINITION can need to change over time. The computation logic for "user's 7-day purchase count" might have a bug (a JOIN that double-counts cancelled orders), or the business might redefine what counts as a "purchase" (excluding refunded transactions). The tempting move is to fix the computation in place, let the pipeline silently start writing the corrected values, and move on. This is dangerous for a subtle reason: any model that was trained on the old definition of this feature is now serving on a different distribution than it was trained on, because the online store now writes the corrected values while the model's weights were learned against the old (buggy or differently-defined) values. The performance degradation from this distributional shift can be small enough to look like ordinary noise, making it genuinely hard to attribute the regression to the feature change — especially if the change was made by a different team than the one that owns the consuming model. The standard fix is explicit versioning: a changed feature definition registers a new version in the registry, typically with an explicit version suffix in the name (e.g. `purchase_count_7d_v2`), leaving `purchase_count_7d_v1` intact and unchanged. The registry tracks which models depend on which versions, enabling a controlled migration: teams owning consuming models can schedule their retraining to use the new version on their own timeline, and the old version remains stable (and continues to receive correct writes) until all consuming models have migrated. Only once all dependents have migrated does the old version get deprecated and its pipeline shut down. This explicit-versioning discipline is what separates a mature feature store practice from an ad-hoc one, and it's directly analogous to schema evolution practices in event-driven systems (e.g. Avro schema registry) — the insight is the same: implicit in-place changes to a shared, typed contract that multiple consumers depend on are a reliability hazard.

    9. Bottlenecks & trade-offs

    • The online store's read latency directly caps every consuming model's own latency budget — as more models are onboarded, contention on shared feature-store infrastructure becomes a real operational concern
    • Backfilling a NEW feature's historical values (so it can be used in training on past data) requires reprocessing the batch source history, which can be a genuinely expensive one-time cost at scale
    • The feature registry can become a bottleneck to velocity, not just infrastructure, if governance/review requirements for registering a new feature are too heavy — a real design tension between correctness/reuse and team autonomy

    Sign in to track this guide in your curriculum progress.