algorithms.live
    AI-ENGINEERING
    advanced

    Quantization & Memory Math

    AI Engineer path · lesson 99 of 124
    The idea
    Measuring a plank. You can record it to the millimetre or to the nearest centimetre, and you can allow for planks up to two metres or up to a kilometre. Those are two separate decisions — precision and range — and a tape measure has a fixed number of digits to split between them.
    How to use
    Use the training calculator to set every parameter trainable and then drop to 1B, which is roughly what LoRA does — watch the optimizer overhead collapse.
    Every float format spends one bit on the sign. The rest is split between range — the exponent bits, how wide a span of values it can express — and precision, the significand bits, how exactly a number is represented.
    FP6464b · 8BFP3232b · 4BTF3219b · 4BBF1616b · 2BFP1616b · 2BINT88b · 1BINT44b · 0.5Bsignrange (exponent)precision (significand)
    FP16half precision
    5
    range bits
    10
    precision bits
    2
    bytes per value
    More precise than BF16 in the middle of its range, and it overflows to infinity much sooner.
    The same number, three ways
    FP32 valueFP16BF16TF32
    0.01234567890.01234436035156250.01232910.01234436035
    1234.567891235.0 (0.035% change)1232.0 (0.208% change)1234.0
    123456.789INF123392.0123456.0
    The last row is the one to remember. Out-of-bound values round to infinity, and FP16 runs out of range at around 123,456 while BF16 — the same 16 bits — handles it, because BF16 spends 8 bits on range against FP16's 5.

    Key insight

    BF16 and FP16 are the same width and are not interchangeable. BF16 trades precision for range, which is what training needs — gradients span many orders of magnitude, and a gradient that overflows to infinity destroys a step in a way a slightly imprecise one does not.

    Order by bytes per parameter

    Sequence

    Order these formats from most memory per parameter to least.

    Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

    1. 1Drop a step here
    2. 2Drop a step here
    3. 3Drop a step here
    4. 4Drop a step here
    4 pieces left to place

    Test your knowledge

    20 questions
    Question 1 of 20
    Score: 0

    In a floating-point format, the exponent bits determine:

    The short version

    Measuring a plank. You can record it to the millimetre or to the nearest centimetre, and you can allow for planks up to two metres or up to a kilometre. Those are two separate decisions — precision and range — and a tape measure has a fixed number of digits to split between them.

    In more depth

    Quantization & Memory Math

    Whether a model runs on your hardware is arithmetic. Two formulas.

    Numerical representations

    Every float spends 1 bit on the sign; the rest splits between range (exponent bits) and precision (significand bits).

    FormatBitsBytesRangePrecisionNote
    FP646481152Default in NumPy/pandas; rare in neural nets
    FP32324823The old train-and-serve standard
    TF32194810NVIDIA, for GPUs — actually 19 bits, not 32
    BF1616287Google, for TPUs
    FP16162510Half precision
    INT8 / INT48 / 41 / 0.5——Fixed point

    BF16 and FP16 are the same width and not interchangeable. 123456.789 → FP16 gives INF (out-of-bound values round to infinity); BF16 gives 123392.0.

    Inference memory

    N × M × 1.2

    Activations and KV vectors grow with sequence length and batch size; ~20% is a working assumption.

    13B at 2 bytes = 26 GB weights ⇒ ~31.2 GB total
    70B at 2 bytes = 140 GB just for weights

    Training memory

    weights + activations + gradients + optimizer states. The last two are charged per trainable parameter.

    OptimizerStates per trainable parameter
    Vanilla SGD0
    Momentum1
    Adam2
    13B, Adam, 3 values (1 gradient + 2 states), 2 bytes ⇒ 78 GB — on top of 26 GB of weights.
    With only 1B trainable ⇒ 6 GB. This gap is why PEFT exists.
    Activations can dwarf weight memory; the fix is gradient checkpointing — 10× larger models at a 20% increase in computation time.

    How low can you go

    PTQ is by far the most common. Weight quantization is more common than activation quantization (more stable, less accuracy loss).

    8 bits — LLM.int8() (Dettmers et al., 2022)
    4 bits — QLoRA (Dettmers et al., 2023); NVIDIA Blackwell (2024) supports 4-bit float
    3.5 bits average — Apple (2024), mixing 2-bit and 4-bit on device
    1.58 bits — BitNet b1.58 (Ma et al., 2024), comparable to 16-bit Llama 2 up to 3.9B

    The smallest IEEE-conforming float is 4-bit. And you cannot go below 1 bit per value.

    Two traps

    1.Loading a model in the wrong format. Llama 2 shipped in BF16; teams loaded it in FP16 and got much worse quality, silently.
    2.Assuming lower precision means lower latency. Format conversion overhead means reduced precision doesn't always reduce latency.

    Interview Questions

    NVIDIA
    Meta
    Amazon
    Hugging Face
    Google
    NVIDIA
    Meta
    Hugging Face
    NVIDIA
    Meta
    Microsoft
    Character.AI
    NVIDIA
    Amazon
    Apple

    References & Further Reading

    Deep Dive into LLMs like ChatGPTYouTube · Andrej Karpathy
    video
    QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023)arXiv
    paper
    Making Deep Learning Go Brrrr From First PrinciplesHorace He
    article
    phoenixtype/ai-engineering-interview-prepGitHub
    docs
    All AI Engineering topics

    Love this visualization?

    Sign up to track your progress, earn achievements, use the AI assistant, and access your personal learning dashboard.

    100% free • No credit card required