Optimizers: SGD, Momentum, RMSprop, Adam

Question 1

MOMENTUM improves plain SGD by…

Accepted Answer

Accumulating a velocity — Momentum adds a fraction of the previous update, building speed in consistent directions and damping oscillations across narrow valleys.

Question 2

AdaGrad adapts the learning rate by…

Accepted Answer

Scaling each rate by accumulated squared gradients — Parameters with large past gradients get smaller steps; it suits sparse features but its ever-growing denominator can shrink rates too much.

Question 3

AdaGrad's main weakness is that its learning rate…

Accepted Answer

Monotonically decays toward near zero over time — Because it accumulates ALL squared gradients, the effective rate keeps shrinking; RMSprop fixes this with a moving average.

Question 4

RMSprop addresses AdaGrad's decay by…

Accepted Answer

Using an exponentially moving average of squares — The moving average forgets old gradients, keeping per-parameter rates responsive and preventing them from vanishing.

Question 5

Adam combines which two ideas?

Accepted Answer

Momentum and adaptive (RMSprop-style) scaling — Adam tracks an exponential average of gradients (momentum) and of squared gradients (adaptive scaling), making it robust and a common default.

Question 6

Adam applies 'bias correction' to its moment estimates because…

Accepted Answer

Moving averages start at zero, biased low early — Initializing moments at zero underestimates them initially; dividing by (1−βᵗ) corrects this bias for accurate early steps.

Question 7

The first moment in Adam estimates the gradient's…

Accepted Answer

The gradient's mean, or direction — The first moment is an exponential average of gradients — the momentum term capturing the typical update direction.

Question 8

The second moment in Adam estimates the gradient's…

Accepted Answer

The gradient's uncentered variance — The second moment tracks per-parameter squared-gradient magnitude, giving smaller steps to high-variance parameters.

Question 9

Why do practitioners sometimes prefer plain SGD (with momentum) over Adam?

Accepted Answer

SGD often generalizes noticeably better overall — Adam converges fast but can settle in sharper minima; well-tuned SGD+momentum frequently yields better test accuracy in vision.

Question 10

AdamW differs from Adam by…

Accepted Answer

Decoupling weight decay from the gradient update — AdamW applies weight decay separately rather than mixing it into the adaptive gradient, improving regularization — now standard for training transformers.

Controls

Settings

Analysis

What does each optimiser track?

Test your knowledge

MOMENTUM improves plain SGD by…

Learn these first

Interview Questions

References & Further Reading

Love this visualization?