Backpropagation

Question 1

Backpropagation calculates the gradient of the loss function with respect to:

Accepted Answer

The weights and biases of every layer — Backpropagation computes the gradient of the loss function with respect to the weights (and biases) of the network.

Question 2

Backpropagation computes…

Accepted Answer

The gradient of loss w.r.t. every weight — It efficiently calculates how much each weight contributed to the loss, so gradient descent can update them.

Question 3

Backpropagation is fundamentally an application of the…

Accepted Answer

The chain rule, through the network's layers — It chains local derivatives layer by layer (dL/dw = product of intermediate derivatives) to get each weight's gradient.

Question 4

Training one step proceeds as…

Accepted Answer

Forward pass then backward pass — The forward pass produces predictions and the loss; the backward pass sends the error gradient back through the layers.

Question 5

Why is backpropagation EFFICIENT compared to computing each gradient numerically?

Accepted Answer

It reuses shared intermediates in one pass — Reverse-mode autodiff computes all parameter gradients in O(network size), versus O(parameters × forward passes) for finite differences.

Question 6

During the backward pass, gradients flow…

Accepted Answer

From the output back toward the input — The loss gradient is propagated backward, each layer transforming the gradient from its output to its input and weights.

Question 7

Backpropagation relies on caching which values from the forward pass?

Accepted Answer

The intermediate activations — Local gradients depend on forward activations, so frameworks store them — the main reason training uses more memory than inference.

Question 8

The 'vanishing gradient' problem in backprop occurs when…

Accepted Answer

Small local derivatives shrink gradients — Saturating activations (sigmoid/tanh) have small derivatives; chaining many in deep nets makes early-layer gradients vanish — addressed by ReLU, residuals, normalization.

Question 9

The 'exploding gradient' problem is when…

Accepted Answer

Gradients grow uncontrollably large — Products of large derivatives blow up; gradient clipping and careful initialization keep updates stable.

Question 10

After backpropagation computes gradients, the weights are updated by…

Accepted Answer

An optimizer — Backprop only produces gradients; an optimizer applies them (θ ← θ − η·gradient) to actually learn.

Backpropagation

One forward-backward pass

Test your knowledge

Backpropagation calculates the gradient of the loss function with respect to:

Learn these first

Interview Questions

References & Further Reading

Love this visualization?