The PyTorch Training Loop

Question 1

What does calling backward() on a tensor do to the .grad attribute of every variable used to compute it?

Accepted Answer

Adds the new gradient to whatever is already there — Accumulation is deliberate — it is what lets you sum gradients across several mini-batches — but it means each iteration must clear the slate itself.

Question 2

Géron highlights accumulation as a difference from which frameworks?

Accepted Answer

TensorFlow, where computing the gradients simply returns them — In those, computing the gradients of f with respect to x returns the values without affecting x itself.

Question 3

If you forget to zero the gradients each iteration, what happens?

Accepted Answer

The updates are simply wrong, with no explicit error raised — Géron: "Since there won’t be any explicit error, just low performance (and perhaps infinite or NaN values), this issue may be hard to debug."

Question 4

Why is the gradient descent step wrapped in a torch.no_grad() context?

Accepted Answer

To keep the parameter update itself out of the computation graph — The update is arithmetic on the parameters, not part of the model, and PyTorch will raise on an in-place operation against a tracked leaf tensor.

Question 5

Does resetting the gradients with w.grad.zero_() need its own no_grad() context?

Accepted Answer

No — the gradient tensor has requires_grad=False already — The context is needed for the parameter update, not for clearing the gradient that update just consumed.

Question 6

What does the detach() method return?

Accepted Answer

A new tensor off the graph that shares the same memory — Because they share memory, modifying the detached tensor also modifies the original — which is exactly how x_detached -= learning_rate * x.grad updates x.

Question 7

When should you prefer no_grad() over detach()?

Accepted Answer

When a whole block of code should not be tracked — detach() is handy for one tensor; no_grad() covers everything inside the block, so Géron calls it generally preferable for inference.

Question 8

Why is torch.no_grad() recommended during inference?

Accepted Answer

It saves memory and time by not building the graph — A separate concern from eval() mode, which is what actually switches off dropout and changes batch-norm behaviour.

Question 9

Applying an in-place operation such as x += 1 to a leaf tensor with requires_grad=True causes:

Accepted Answer

A RuntimeError, as there is nowhere to store the graph — Géron gives x.cos_() and x += 1 as examples that both fail for this reason.

Question 10

In this snippet — z = t.exp(); z += 1; z.backward() — why does the last line fail?

Accepted Answer

The value the backward pass needed was overwritten — The error reads "one of the variables needed for gradient computation has been modified by an in-place operation". The fix is z = z + 1, which creates a new tensor.

Six lines, and what each one defends against

Write the training loop

Test your knowledge

What does calling backward() on a tensor do to the .grad attribute of every variable used to compute it?

Learn these first

Interview Questions

References & Further Reading

Related topics

Love this visualization?