Gradient Descent

Question 1

What is the "Learning Rate"?

Accepted Answer

The step size taken at each iteration — The learning rate is a hyperparameter that controls how much to change the model in response to the estimated error each time the model weights are updated.

Question 2

Gradient descent minimizes a loss function by repeatedly…

Accepted Answer

Stepping in the negative gradient direction — The gradient points toward steepest increase, so moving opposite it (θ ← θ − η∇L) decreases the loss fastest locally.

Question 3

The learning rate η determines…

Accepted Answer

The step size of each update — η scales how far you move along the negative gradient each iteration — the single most important optimization hyperparameter.

Question 4

If the learning rate is TOO LARGE, gradient descent may…

Accepted Answer

Overshoot the minimum and diverge or oscillate — Big steps can jump past the minimum and bounce or blow up, so the loss fails to settle.

Question 5

If the learning rate is TOO SMALL, gradient descent will…

Accepted Answer

Converge extremely, very slowly overall — Tiny steps creep toward the minimum, wasting many iterations and potentially stalling on flat regions.

Question 6

BATCH gradient descent computes the gradient using…

Accepted Answer

The entire training set at each step — Full-batch GD uses all data per update — accurate but slow and memory-heavy for large datasets.

Question 7

STOCHASTIC gradient descent (SGD) updates using…

Accepted Answer

A single — SGD estimates the gradient from one example, giving fast, noisy updates that iterate quickly and can escape shallow minima.

Question 8

MINI-BATCH gradient descent is popular because it…

Accepted Answer

Balances stable gradient estimates with frequent updates — Mini-batches (e.g. 32–512) give reasonably accurate gradients with efficient vectorized/GPU computation — the standard in deep learning.

Question 9

On a CONVEX loss, gradient descent (with a suitable η) converges to…

Accepted Answer

The overall global minimum — Convex surfaces have a single minimum, so gradient descent reliably reaches the global optimum (e.g. linear/logistic regression).

Question 10

On NON-CONVEX deep-network losses, gradient descent may…

Accepted Answer

Settle in a local minimum or saddle region — Deep losses have many critical points; SGD's noise and good initialization usually find solutions that generalize well, even if not globally optimal.

Gradient Descent

Diagnose the learning rate

Test your knowledge

What is the "Learning Rate"?

Interview Questions

References & Further Reading

Love this visualization?