Loss Landscape 3D

Question 1

What is a global minimum?

Accepted Answer

The absolute lowest value of the loss function — The global minimum is the point in the parameter space where the loss function has the lowest possible value.

Question 2

A 'loss landscape' visualizes…

Accepted Answer

Loss as a function of the model's parameters — It plots how loss changes across parameter space; training is the process of descending this surface toward low-loss regions.

Question 3

Linear/logistic regression have CONVEX loss landscapes, meaning…

Accepted Answer

A single global minimum — Convexity guarantees one global minimum, so gradient descent reliably finds the best solution regardless of initialization.

Question 4

Deep neural network loss landscapes are…

Accepted Answer

Highly non-convex, with many minima and saddles — Stacking non-linear layers creates a complex high-dimensional surface; optimization must navigate it without guarantees of global optimality.

Question 5

A SADDLE POINT is a location where…

Accepted Answer

Where the gradient is zero but curvature mixes — Saddles (more common than local minima in high dimensions) have zero gradient but aren't minima; momentum/SGD noise help escape them.

Question 6

FLAT minima are often associated with…

Accepted Answer

Better generalization than sharp minima — Flat regions are robust to small parameter perturbations and data shifts, which empirically correlates with better test performance.

Question 7

SHARP minima tend to…

Accepted Answer

Generalize worse — In sharp basins, tiny shifts (e.g. train→test distribution) cause large loss increases, often hurting generalization.

Question 8

Why is high DIMENSIONALITY important to understanding deep loss landscapes?

Accepted Answer

With millions of parameters, true minima are rare — In high dimensions a critical point is a minimum only if curvature is positive in ALL directions — unlikely — so saddles dominate, which SGD escapes.

Question 9

SKIP/residual connections (ResNets) affect the loss landscape by…

Accepted Answer

Smoothing it and easing optimization overall — Residual connections create smoother, more navigable landscapes (and better gradient flow), which is why they enable training hundreds of layers.

Question 10

Batch normalization tends to make the loss landscape…

Accepted Answer

Smoother, allowing higher rates — By normalizing activations, batch norm reduces sharp curvature, stabilizing and speeding optimization.

Simulation

Convex (Bowl)

Landscape Settings

Start Position

What is this point on the surface?

Test your knowledge

What is a global minimum?

Learn these first

Interview Questions

References & Further Reading

Love this visualization?