Vanishing & Exploding Gradients

Question 1

The vanishing gradients problem means that during backpropagation:

Accepted Answer

Gradients shrink as they descend, so lower layers barely update — Training "never converges to a good solution" because the lower layers effectively never learn.

Question 2

Exploding gradients surface most often in which architecture?

Accepted Answer

Recurrent neural networks — The repeated multiplication through time is the same compounding problem running in the other direction.

Question 3

Both problems are facets of what more general issue?

Accepted Answer

Unstable gradients, where layers learn at widely different speeds — The shared symptom is that the network does not train as one system.

Question 4

Which two culprits did Glorot and Bengio identify in 2010?

Accepted Answer

The sigmoid activation and initialization from a normal distribution with standard deviation 1 — That pairing makes each layer's output variance far exceed its input variance, until the activation saturates.

Question 5

With that pairing, going forward through the layers the variance:

Accepted Answer

Keeps increasing until the activation saturates — And once the sigmoid saturates at 0 or 1, its derivative is extremely close to zero, so the backward pass carries nothing.

Question 6

Why does the sigmoid shrink gradients even when the network is well initialised?

Accepted Answer

Because its derivative peaks at 0.25, so every layer multiplies by at most a quarter — Over twelve layers that bound alone is a factor of sixteen million. Initialization fixes the forward variance, not a derivative below one.

Question 7

Why does tanh behave slightly better than sigmoid in deep networks?

Accepted Answer

It has a mean of 0 rather than 0.5 — The sigmoid pushes a systematic offset through every layer on top of the variance growth. Tanh's derivative also peaks at 1 rather than 0.25.

Question 8

Glorot initialization aims to keep which quantity equal across a layer?

Accepted Answer

The variance of the outputs and the variance of the inputs — And, flowing backwards, the gradients should have equal variance before and after the layer.

Question 9

Why is fanavg used rather than fanin or fanout alone?

Accepted Answer

Because both conditions cannot hold unless fan-in equals fan-out — Glorot and Bengio proposed the average as "a good compromise that has proven to work very well in practice".

Question 10

Glorot initialization draws weights with what normal variance?

Accepted Answer

1 / fanavg — For the uniform version, r = √(3σ²), which works out to √(3/fanavg).

Measured through a real forward and backward pass

Vanishing or exploding?

Test your knowledge

The vanishing gradients problem means that during backpropagation:

Learn these first

Interview Questions

References & Further Reading

Love this visualization?