algorithms.live
    AI-ENGINEERING
    advanced

    AI Accelerators

    AI Engineer path · lesson 102 of 124
    The idea
    A kitchen. A CPU is one brilliant chef who can cook anything in sequence; a GPU is a thousand line cooks who can only chop, but all at once. Neither matters if the pantry is down the hall — most of the time the kitchen is not waiting on knives, it is waiting on ingredients.
    How to use
    On the MFU and MBU tab, set the model size and precision and watch bandwidth utilisation move — halving bytes per parameter halves the bandwidth the same token rate demands.
    The lack of sufficiently powerful computers was one of the contributing factors to the first AI winter in the 1970s. AlexNet in 2012 is commonly credited as the first paper to successfully train a neural network on GPUs — before that you would have needed thousands of CPUs.
    CPU
    A few powerful cores — typically up to 64 for high-end consumer machines. Built for high single-thread performance: the operating system, I/O, and anything that has to happen in order.
    few, large
    GPU
    Thousands of smaller, less powerful cores optimised for parallel work. The operation that constitutes most ML workloads is matrix multiplication, which is highly parallelisable — so weak and numerous wins.
    many, small
    The accelerator zoo
    Training and general
    AMD GPUs · Google TPU (Tensor Processing Unit) · Intel Habana Gaudi · Graphcore IPU · Groq LPU (Language Processing Unit) · Cerebras WSE
    Inference-specialised
    Apple Neural Engine · AWS Inferentia · Meta MTIA · and for the edge, Google Edge TPU and NVIDIA Jetson Xavier
    Why inference gets its own chips. Training needs more memory (backpropagation keeps activations around) and higher precision, and it optimises for throughput. Inference optimises for latency. So inference chips are built for lower precision and faster memory access rather than large memory capacity — a genuinely different design target, not a cheaper version of the same one.
    Compute primitives: scalars, vectors, tensors
    GPUs traditionally supported vector operations; modern ones include tensor cores. TPUs were designed with tensor operations as their primary primitive from the start. This is why the H100 table below quotes “Tensor Core” figures — the general-purpose path is much slower, and a workload that cannot use the tensor cores is not getting the chip you paid for.

    Key insight

    Inference can exceed the cost of training. Desislavov et al. (2023) put it at up to 90% of the machine learning costs for deployed AI systems — which is the whole reason this chapter exists, and the reason a separate class of inference chips is worth designing.

    Which number is the bottleneck?

    Sort

    Sort each workload by what limits it on the accelerator.

    Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

    Compute-bound (FLOPS)

    Drop items here

    Memory-bandwidth-bound

    Drop items here
    6 pieces left to place

    Test your knowledge

    20 questions
    Question 1 of 20
    Score: 0

    A GPU beats a CPU on ML workloads mainly because:

    The short version

    A kitchen. A CPU is one brilliant chef who can cook anything in sequence; a GPU is a thousand line cooks who can only chop, but all at once. Neither matters if the pantry is down the hall — most of the time the kitchen is not waiting on knives, it is waiting on ingredients.

    In more depth

    AI Accelerators

    Every latency and cost number traces back to two hardware quantities: how fast the chip does arithmetic, and how fast it moves bytes.

    CPU vs GPU

    CPU — a few powerful cores, typically up to 64 on high-end consumer machines. High single-thread performance.
    GPU — thousands of smaller, weaker cores. Most ML work is matrix multiplication, which is highly parallelisable.
    Inference chips are a different design target, not a cheaper GPU: training needs more memory (backpropagation) and higher precision and optimises throughput; inference optimises latency, so inference chips favour lower precision and faster memory access over large capacity. Desislavov et al. (2023): inference can be up to 90% of ML costs for deployed systems.

    Which ceiling are you under?

    From the Roofline paper (Williams et al., 2009). The deciding quantity is arithmetic intensity — operations per byte of memory access.

    PhaseBoundFixed by
    Prefill (input, processed in parallel)Compute-boundMore FLOP/s
    Decode (one output token at a time)Memory bandwidth-boundMore bandwidth, or fewer bytes

    This single fact explains three things that look unrelated: quantization speeds up generation despite doing no less arithmetic; batching is nearly free during decode; and speculative decoding works because verification consumes idle FLOPs.

    The three memory levels

    LevelBandwidthCapacity
    GPU on-chip SRAMoften exceeding 10 TB/stypically 40 MB or under
    GPU HBM256 GB/s – over 1.5 TB/s~24–80 GB
    CPU DRAM25–50 GB/s16–64 GB, or a TB+

    PyTorch and TensorFlow do not expose fine-grained memory control — hence CUDA, Triton, ROCm.

    MFU and MBU

    nvidia-smi's utilization measures time active, not efficiency, and is near-useless.

    MFU = observed throughput ÷ theoretical max at peak FLOP/s. High when compute-bound. Above 50% is good for training.
    MBU = achieved ÷ peak bandwidth, where achieved = params × bytes/param × tokens/s. High when bandwidth-bound.

    Worked example: 7B in FP16 at 100 tokens/s = 7B × 2 × 100 = 700 GB/s; on an A100-80GB at a theoretical 2 TB/s, MBU = 70%.

    Power

    A100 has 54B transistors; H100 has 80B. An H100 at peak for a year uses ~7,000 kWh against a US household's 10,000 kWh. Maximum power draw is roughly 1.1–1.5× TDP.

    Interview Questions

    NVIDIA
    Google
    Anthropic
    Meta
    NVIDIA
    Google
    Microsoft
    NVIDIA
    OpenAI
    Meta
    Amazon
    Google
    Microsoft
    Meta

    References & Further Reading

    LLM Inference Lecture: Roofline Analysis for GPU (arithmetic intensity, compute and memory bound)YouTube
    video
    Making Deep Learning Go Brrrr From First PrinciplesHorace He
    article
    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al., 2022)arXiv
    paper
    phoenixtype/ai-engineering-interview-prepGitHub
    docs
    All AI Engineering topics

    Love this visualization?

    Sign up to track your progress, earn achievements, use the AI assistant, and access your personal learning dashboard.

    100% free • No credit card required