Design a real-time fraud detection system for payment transactions

1. Scope

In scope

• Real-time transaction scoring within the payment authorization window
• A hybrid rules-engine plus ML-model architecture
• Real-time (streaming) feature computation for velocity-style signals
• The delayed-feedback retraining problem specific to fraud (chargeback labels arrive late)

Out of scope

• Identity verification / KYC systems (a related but separate concern)
• Post-hoc fraud investigation tooling for the manual review team
• Account-takeover detection specifically (a related but distinct fraud type with different signals)

2. Functional requirements

Must have

• Score every transaction and return approve/decline/review within the payment gateway's authorization window
• Known fraud patterns (e.g. a card used in two geographically impossible locations within minutes) are caught deterministically, not left to model uncertainty
• Ambiguous cases route to a manual review queue rather than an automatic decline, to control the false-positive rate

Nice to have

• Explainability for why a transaction was flagged, to support the manual review team
• Adaptive thresholds per merchant category, since normal transaction patterns vary widely by vertical

3. Non-functional requirements

Latencyp99 < 100ms— a hard constraint from the payment gateway's own authorization timeout

Availability99.99%+— a scoring-system outage either blocks all payments (fail-closed) or lets all transactions through unscored (fail-open) — both are bad, so this system needs real redundancy

Class imbalancefraud is typically well under 0.1% of transactions

Label latencyconfirmed-fraud labels (chargebacks) often arrive days to weeks after the transaction— this delayed feedback fundamentally shapes the retraining strategy

4. Back-of-envelope estimation

Assumptions

• 10M transactions/day for a large payment processor
• Fraud rate of ~0.05% (500 fraud/day out of 10M)
• 100ms total authorization budget, of which fraud scoring gets a fraction (rules engine + model must fit inside it alongside other gateway logic)

Derived numbers

Average QPS~115/sec— 10M / 86,400 seconds

Peak QPS~1,000-2,000/sec— payment traffic has sharp peaks around business hours and shopping events

Daily confirmed fraud volume~500 transactions/day— the actual positive-label volume the model eventually trains on — small relative to total volume, reinforcing the class-imbalance challenge

5. API design

POST/score

Score a transaction in real time, returning a decision (approve/decline/review) and a risk score

POST/labels/chargeback

Ingest a confirmed-fraud label arriving via chargeback, days/weeks after the original transaction

GET/review-queue

Fetch transactions pending manual review

6. High-level design

What each box in the reference diagram is responsible for:

Rules Engine — Deterministic, low-latency checks for known fraud patterns (impossible travel velocity, known-bad card BINs, mismatched billing/shipping country) — runs first because it's cheap and catches the obvious cases without needing model inference

Fraud Model — Handles the harder, ambiguous cases the rules engine doesn't confidently resolve — typically a gradient-boosted-tree model for latency and interpretability reasons, sometimes supplemented with an anomaly-detection model for genuinely novel fraud patterns

Velocity Features — Real-time streaming aggregation (transactions per card per minute/hour, spend velocity) — these signals only exist as streaming computations, since "how many times has this card been used in the last 5 minutes" can't be answered from a batch table

Decision Service — Combines the rules engine and model outputs into a final approve/decline/review decision, applying business-defined thresholds

Manual Review Queue — Ambiguous cases that neither the rules engine nor the model are confident about route here, trading a small latency/labor cost for a lower false-positive-on-decline rate

Retraining Pipeline — Periodically retrains the fraud model using confirmed chargeback labels — unlike most ML systems, this pipeline operates on delayed, asynchronously-arriving labels (days to weeks lag) rather than near-real-time feedback, which shapes both its scheduling and its evaluation strategy

7. Data model

Velocity Features (streaming): card_id, merchant_id → rolling window counters (txn count, total amount) over multiple time windows— requires a streaming aggregation system (e.g. a windowed stream processor), not a plain database

Transaction Log: (transaction_id, card_id, merchant_id, amount, timestamp, decision, model_score)— the training set, joined against chargeback labels as they arrive

Chargeback Labels: (transaction_id, confirmed_fraud: bool, label_timestamp)— arrives asynchronously, days to weeks after the original transaction record

8. Deep dives

Why rules AND a model, not just one or the other

Rules alone can't adapt to novel fraud patterns and require constant manual maintenance as fraudsters adapt. A model alone is slower to react to a NEWLY discovered fraud pattern that a human analyst notices before enough labeled data exists to retrain on it — and is harder to explain/audit for compliance purposes. The hybrid approach gets the best of both: rules give instant, auditable coverage for known patterns and can be hand-authored the moment a new pattern is discovered; the model catches the harder, evolving cases rules can't enumerate.

The delayed-feedback problem and how it shapes retraining

Unlike most ML systems where the true label is known quickly (a user clicked or didn't), fraud labels often only arrive when a customer disputes a charge — days or weeks later. This means: (1) a transaction scored today can't be used as a confirmed training example until its label window closes, so training data always lags real transaction volume; (2) the model must be evaluated with this lag in mind (a model that looks great on last week's data might already be missing patterns from fraud that hasn't been confirmed yet); (3) some fraud is never confirmed via chargeback at all (the cardholder doesn't notice or doesn't bother disputing), meaning the "confirmed fraud" label set systematically undercounts true fraud — a subtlety worth naming explicitly in an interview.

How velocity features are computed at scale — streaming aggregation and the exact-count vs. approximate-count tradeoff

Velocity features — "how many times has this card been charged in the last 5 minutes?" or "what is the total spend velocity for this merchant in the last hour?" — cannot be answered from a batch table; they require a system that maintains rolling-window counts in real time. The standard implementation is a stream processor (Apache Flink is the reference, though Spark Structured Streaming and cloud-managed equivalents are also used) maintaining per-key sliding or tumbling window aggregates. Each incoming transaction event increments a counter keyed by card_id + time_window; the fraud model then point-reads the current counter at inference time, adding only a single KV lookup to the scoring latency budget. At high transaction volume, maintaining exact per-key rolling counts in memory for every active card can become untenable. Two approximate-counting data structures are worth knowing: Count-Min Sketch maintains per-key counts in sub-linear space with a small guaranteed overcount error — but a standard CMS accumulates all-time counts, not windowed ones; computing a windowed velocity signal requires either maintaining a CMS per time bucket and summing recent buckets (e.g. one sketch per minute, querying the last five), or periodically rotating/resetting the sketch. This per-bucket approach is useful when the per-key state doesn't fit in memory and a slight overcount in the velocity signal is acceptable. HyperLogLog estimates the cardinality of a set — "how many distinct merchants has this card visited in the last 24 hours?" — in O(log log n) space, with error around 2–3% (useful for distinct-count velocity signals, not raw-count signals). In practice, most production fraud systems maintain exact per-key counters for the highest-value signals (recent transaction count and total spend by card_id), because the cardinality is manageable with a sharded Redis cluster or distributed KV store, and reserve approximate structures for the highest-cardinality cross-product features (e.g., device × IP × merchant velocity). The partition key is typically card_id with consistent hashing, keeping a given card's counters on a single shard to avoid distributed aggregation latency at read time.

Graceful degradation: why fail-open vs. fail-closed is a false binary, and how circuit-breaker patterns give a third option

The bottlenecks section names fail-open (let all transactions through unscored during a model outage) versus fail-closed (block all transactions) as a genuine business tradeoff with no free option. But framing it as a binary understates what production systems actually do: a well-designed scoring pipeline degrades gracefully to rules-only scoring rather than choosing either extreme. The pattern is a circuit breaker on the ML model service. The gateway (or decision service) tracks the model's recent error rate and p99 latency: if the model times out or its error rate spikes above a threshold (indicating the model pods are unhealthy), the circuit opens and the scoring path falls back to rules-only mode. The decision service then makes approve/decline/review decisions using only the deterministic rules, accepting a controlled, temporary increase in fraud exposure on the ambiguous cases the model normally handles — rather than either blocking all legitimate payments (fail-closed) or approving everything unscored (fail-open). Rules-only fallback catches the clear fraud patterns (impossible travel, known-bad BINs, blocked card status), keeps the payment flow running for the vast majority of legitimate transactions, and produces auditable, logged decisions throughout. The circuit resets as the model service recovers, with gradual traffic ramping to validate health before restoring full scoring. A complementary safeguard is a lightweight fallback model — a much simpler model (logistic regression on a handful of hand-crafted features, serving from a single always-on pod) that runs as a hot standby when the primary model is unhealthy. This narrows the degradation gap between full-model scoring and rules-only, since even a simple model handles the easiest ambiguous cases. The key interview answer is not just naming the circuit-breaker pattern but articulating the full degradation hierarchy: full model → lightweight fallback model → rules only → (if even the rules engine is unhealthy) conservative hard limits per transaction amount, failing closed only on high-value transactions. This graduated response is what "graceful degradation" means in practice for a latency-critical, high-stakes service.

9. Bottlenecks & trade-offs

The velocity-feature streaming aggregation is the most operationally complex piece — it needs to maintain correct rolling-window counts under high write volume with strict latency, and a bug here silently degrades the model's best signal
The manual review queue doesn't scale linearly with transaction volume the way automated scoring does — at high growth, the review-queue capacity becomes a real constraint requiring either more reviewers or a higher model-confidence threshold to route fewer cases there
Fail-open vs fail-closed on a scoring-system outage is a genuine business decision with no free option: failing closed blocks all legitimate transactions during an outage; failing open lets all fraud through unscored — both are worth naming explicitly rather than assuming one is obviously correct