LLM Inference Optimization

Question 1

What does the KV cache store during autoregressive LLM generation?

Accepted Answer

The key and value tensors cached from prior tokens — Without the KV cache, computing attention for the n-th token requires recomputing K and V for all n-1 previous tokens — O(n²) total. The cache stores past K/V so only the new token's K/V are computed, reducing each step to O(n).

Question 2

KV cache memory consumption grows with…

Accepted Answer

Batch size, sequence length, layer count, and model dimension — KV cache size = 2 × B × S × L × H × M — batch size, sequence length, number of layers, model dimension, and bytes per value. Doubling context length doubles cache memory and so does doubling the batch, which is why very long contexts at high concurrency are the memory wall. A Google paper measured 3 TB of KV cache for a 500B+ model at batch size 512 and context 2,048 — three times the size of the model weights.

Question 3

Continuous batching improves throughput over static batching by…

Accepted Answer

Inserting new requests at each generation step — Static batching causes short requests to wait for the longest one in the batch, wasting GPU cycles. Continuous (iteration-level) batching fills freed slots immediately, achieving near-100% utilization.

Question 4

INT8 weight quantization reduces model memory by approximately…

Accepted Answer

A 2× reduction, halving the bits per weight — FP16 uses 16 bits per weight; INT8 uses 8 bits — a 2× size reduction. Combined with faster INT8 tensor core ops on modern GPUs, this typically yields 1.5–2.5× throughput improvement with <1% quality drop on most tasks.

Question 5

Speculative decoding reduces latency by…

Accepted Answer

Draft model speculates, target verifies in parallel — The large model's memory bandwidth is the bottleneck for single-request latency (not FLOPS). Speculative decoding amortizes a single large-model forward pass over multiple tokens, effectively multiplying throughput when the acceptance rate is high.

Question 6

PagedAttention (used in vLLM) solves which KV cache problem?

Accepted Answer

It eliminates KV cache fragmentation via non-contiguous paged blocks — Without PagedAttention, each request pre-allocates a contiguous KV cache block for its maximum length, wasting memory for short sequences. PagedAttention allocates non-contiguous "pages," used only as needed — dramatically improving GPU memory utilization.

Question 7

LLM inference is typically "memory-bandwidth bound" rather than "compute bound" for single requests. This means…

Accepted Answer

The bottleneck is reading model weights from GPU memory, not arithmetic compute — For small batch sizes (e.g. a single user request), the GPU arithmetic units spend most of their time waiting for weights to load from VRAM. Quantization reduces weight size (faster loads); speculative decoding processes multiple tokens per load cycle.

Question 8

TTFT (Time To First Token) and TPOT (Time Per Output Token) measure what?

Accepted Answer

Time to first token (prefill latency) vs time per output token (decode latency) — TTFT is dominated by the prefill (processing the full prompt in one parallel pass). TPOT is the cost per decode step — dominated by memory bandwidth to load weights. Optimizing them requires different techniques (batched prefill vs. continuous batching/speculative decoding).

Question 9

INT4 quantization (e.g. GPTQ, AWQ) compared to INT8 offers…

Accepted Answer

Smaller model size with a larger quality tradeoff than INT8 — INT4 quantizes to 4 bits per weight — 4× smaller than FP16. The quality hit is larger than INT8 but often acceptable for chat tasks. AWQ (Activation-aware Weight Quantization) and GPTQ use calibration data to minimize degradation.

Question 10

Speculative decoding requires that the draft and target models…

Accepted Answer

Share the same vocabulary for verification — Verification requires comparing the draft token's probability under the target distribution. For this to work, the models must share a vocabulary and tokenizer so tokens are directly comparable.

Precision	Size	Speed	Quality loss
FP32	4×	1×	none
FP16/BF16	2×	1.5–2×	minimal
INT8 (W8A8)	1×	2–3×	<1% on most benchmarks
INT4 (GPTQ/AWQ)	0.5×	3–4×	~2–5% on hard tasks

Key insight

Prefill or decode?

Test your knowledge

What does the KV cache store during autoregressive LLM generation?

The short version

In more depth

LLM Inference Optimization

KV Cache

Continuous Batching

Quantization

Speculative Decoding

Interview Questions

References & Further Reading

Related topics

Love this visualization?