algorithms.live
    AI-ENGINEERING
    advanced

    Speculative Decoding

    AI Engineer path · lesson 105 of 124
    The idea
    A junior drafting a letter for a senior to sign. The senior reads the whole draft in one pass and signs off everything down to the first thing they disagree with, fixes that line themselves, and hands it back. Reading is fast; writing is slow; and the senior's signature means the final letter is exactly what they would have written.
    How to use
    Raise K on the first tab and watch expected tokens per loop rise while acceptance-driven waste rises too — then lower the acceptance rate and see the speedup collapse.
    One iteration: the draft model proposes K tokens sequentially, the target model verifies all K in one pass, accepts the longest left-to-right prefix it agrees with, and then generates one token of its own.
    draft proposest+1t+2t+3t+4target verifiesone parallel passkept✓✓✗✗+1
    The purple +1 is the token the target generates itself after the accepted prefix — it lands on the first rejected position, which is why the loop never yields zero.
    4
    70%
    0.13×
    1.77
    expected accepted prefix (j)
    2.77
    tokens per loop (j+1)
    1.82×
    speedup vs plain decoding
    Each loop costs 4 draft passes at 0.13× plus one target pass — 1.52 target-passes in total — and yields 2.77 tokens. Plain decoding costs one pass per token, so this is 1.82× faster. The output is identical either way.
    The K trade-off
    Larger K means fewer verification calls, and a lower acceptance rate — each additional proposal is conditioned on all the guesses before it, so the probability of the whole prefix surviving decays geometrically. Push K up on the slider with acceptance at 50% and watch the expected prefix stop growing well before K does. There is an optimum, it depends on your workload, and it is worth measuring rather than assuming.

    Key insight

    The loop yields j+1, not j. Even when the target rejects the very first proposal, it still emits the correct token for that position from the verification pass it already ran — so a completely useless draft model costs you the drafting time and nothing else. That asymmetry is what makes the technique safe to switch on.

    Draft and verify

    Sequence

    Order one round of speculative decoding.

    Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

    1. 1Drop a step here
    2. 2Drop a step here
    3. 3Drop a step here
    4. 4Drop a step here
    5. 5Drop a step here
    5 pieces left to place

    Test your knowledge

    20 questions
    Question 1 of 20
    Score: 0

    In speculative decoding, the draft model's job is to:

    The short version

    A junior drafting a letter for a senior to sign. The senior reads the whole draft in one pass and signs off everything down to the first thing they disagree with, fixes that line themselves, and hands it back. Reading is fast; writing is slow; and the senior's signature means the final letter is exactly what they would have written.

    In more depth

    Speculative Decoding

    Generating one token requires one full forward pass, and the passes cannot overlap. Speculative decoding breaks the sequential bottleneck without changing the output.

    The loop

    1.A small draft model generates K tokens sequentially.
    2.The target model verifies all K in one parallel pass.
    3.It accepts the longest left-to-right subsequence it agrees with — say j tokens.
    4.It then generates one extra token itself.

    So the loop yields 1 token if nothing is accepted and K+1 if everything is. Output quality is preserved exactly, because the target signs off on every token.

    Three enabling insights

    1.Verification is parallelisable; generation is sequential. Verifying takes less time than generating — speculative decoding effectively turns the computation profile of decoding into that of prefilling.
    2.Some tokens are easy to predict, so a weak draft model gets them right.
    3.Decoding is memory bandwidth-bound, so there are typically idle FLOPs available for free verification.

    The numbers

    DeepMind trained a 4B draft model of the same architecture for Chinchilla-70B (Chen et al., 2023): the draft generates a token eight times faster — 1.8 ms vs 14.1 ms — reducing overall response latency by more than half without compromising quality. Similar for T5-XXL (Leviathan et al., 2022).

    Implementable in ~50 lines of PyTorch; already in vLLM, TensorRT-LLM and llama.cpp.

    Trade-offs

    Larger K ⇒ fewer verification calls but a lower acceptance rate.
    Acceptance rates are domain-dependent — higher for structured text like code.
    The draft model should ideally share the same vocabulary and tokenizer.

    Four variants

    VariantExtra costNote
    Draft modelA second modelThe classic form
    Inference with reference (Yang et al., 2023)NothingDrafts from the input; 2× speedup where context and output overlap
    Lookahead / Jacobi (Fu et al., 2024)No second modelSame decoder; regenerate only failed tokens
    Medusa (Cai et al., 2024)Training the headsk-th head predicts x(t+k+1); tree-based attention verification; NVIDIA claimed up to 1.9× on Llama 3.1

    Interview Questions

    Google DeepMind
    NVIDIA
    Anthropic
    Meta
    NVIDIA
    Google
    OpenAI
    NVIDIA
    Google
    Microsoft
    Anthropic
    NVIDIA
    Amazon

    References & Further Reading

    Speculative Decoding ExplainedYouTube
    video
    Accelerating Large Language Model Decoding with Speculative Sampling (Chen et al., 2023)arXiv
    paper
    Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (Cai et al., 2024)arXiv
    paper
    phoenixtype/ai-engineering-interview-prepGitHub
    docs
    All AI Engineering topics

    Love this visualization?

    Sign up to track your progress, earn achievements, use the AI assistant, and access your personal learning dashboard.

    100% free • No credit card required