algorithms.live
    DL
    intermediate

    Batch Normalization

    algorithms.live

    Batch

    10

    Learnable Parameters

    1.50

    Rescales the normalised output. γ=1 = no change.

    0.50

    Shifts the output. β=0 = centred at zero.

    Mode

    Current Statistics
    Batch μ6.99
    Batch σ3.19
    Active μ6.99
    Active σ3.19

    Internal Covariate Shift — Layer Distributions

    Without BatchNorm
    Layer 1
    -1.26.4
    Layer 2
    1.215.1
    Layer 3
    1.120.4

    Each layer sees a different distribution → later layers must constantly readjust

    With BatchNorm
    Layer 1
    -2.11.6
    Layer 2
    -1.81.8
    Layer 3
    -2.51.3

    All layers receive a stable ~N(0,1) distribution → training is fast and predictable

    Batch Values → After BN (γ=1.5, β=0.5)

    2.0
    -0.7
    0.5
    -0.2
    -0.9
    1.8
    -0.7
    -1.8
    2.4
    2.5
    Normalised to μ≈0.5, σ≈1.5 (from γ and β)

    The Transformation

    Step 1 — Normalise
    x^i=xi−μBσB2+ε\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \varepsilon}}

    Subtracts batch mean, divides by batch std. Output: ~N(0, 1).

    Step 2 — Affine Transform
    yi=γ x^i+βy_i = \gamma\,\hat{x}_i + \beta

    γ=1.50 and β=0.50 are learned. The network can recover any distribution it needs.

    γ and β are learnable
    Unlike the normalisation step (fixed math), γ and β are trained via backprop just like weights. At γ=1, β=0 this is an identity transform. The network diverges from there only if it helps.
    Training ≠ Inference
    Training uses the current batch's μ and σ. Inference uses exponential moving averages accumulated over all training batches — so a single example can be normalised without needing a batch.
    Why it speeds training
    Stable inputs mean gradients are better conditioned. You can use 10× larger learning rates without diverging. Fewer steps needed → faster convergence.
    Mild regularisation
    Batch statistics vary across batches — this injects small random noise into activations, similar in effect to dropout. Models trained with BN often need less dropout.

    Normalise a batch

    Sequence

    Order the batch-norm computation.

    Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

    1. 1Drop a step here
    2. 2Drop a step here
    3. 3Drop a step here
    4. 4Drop a step here
    5. 5Drop a step here
    5 pieces left to place

    Test your knowledge

    20 questions
    Question 1 of 20
    Score: 0

    Batch normalization normalizes…

    Learn these first

    Interview Questions

    Google
    Meta
    Amazon
    Apple
    Google
    OpenAI
    Anthropic
    Meta
    Google
    Meta
    Microsoft
    NVIDIA
    Google Health
    Microsoft
    NVIDIA
    Intuitive Surgical
    Meta AI
    Google
    Microsoft
    NVIDIA

    References & Further Reading

    Batch Normalization — EXPLAINED!YouTube
    video
    Batch normalizationWikipedia
    article
    Batch Normalization (Ioffe & Szegedy, 2015)arXiv
    paper
    phoenixtype/ai-engineering-interview-prepGitHub
    docs

    Love this visualization?

    Sign up to track your progress, earn achievements, use the AI assistant, and access your personal learning dashboard.

    100% free • No credit card required