Regularization in Weight Space 3D

Question 1

Viewed in WEIGHT SPACE, L2 regularization constrains the weights to lie within a…

Accepted Answer

Circular ball, sphere in 3-D — The L2 penalty ‖w‖² defines a circular/spherical constraint region; the solution balances the loss against staying inside this ball.

Question 2

L1 regularization's constraint region is a…

Accepted Answer

Diamond — The L1 ball is a diamond/cross-polytope; its sharp corners lie on the axes, which is geometrically why L1 yields sparse (zeroed) weights.

Question 3

Why does L1 regularization produce SPARSE weights while L2 does not?

Accepted Answer

Loss contours are more likely to touch the L1 diamond at an axis-aligned corner — The diamond's corners on the axes make it probable the optimum sets some coordinates to 0; the smooth L2 circle has no such corners, so weights merely shrink.

Question 4

Increasing the regularization strength λ makes the constraint region…

Accepted Answer

Smaller, pushing the allowed weight region closer to the origin — A bigger λ tightens the allowed weight region, shrinking weights more — increasing bias and reducing overfitting.

Question 5

L2 regularization shrinks weights…

Accepted Answer

Smoothly toward zero without driving any weight to exactly zero — L2 (ridge) reduces all weights gradually but keeps them non-zero, distributing influence across correlated features.

Question 6

Ridge regression refers to linear regression with which penalty?

Accepted Answer

L2 — Ridge adds λ‖w‖² to the least-squares loss, stabilizing solutions and handling multicollinearity by shrinking coefficients.

Question 7

Lasso regression uses which penalty and is valued for…

Accepted Answer

L1; automatic feature selection by driving irrelevant weights to zero — Lasso's L1 penalty zeros out unimportant coefficients, performing embedded feature selection.

Question 8

The 'regularization path' shows how coefficients change as…

Accepted Answer

The regularization strength λ is swept from large to small values — Plotting weights vs λ reveals when each feature enters (becomes non-zero) — informative for Lasso feature selection.

Question 9

Geometrically, the regularized solution is where…

Accepted Answer

The loss contour first touches the boundary of the constraint region — The optimum balances minimizing loss against the penalty; it's the point where the elliptical loss contours meet the L1/L2 region.

Question 10

Weight decay during training is the practical implementation of…

Accepted Answer

L2 penalty on the weights, applied as a proportional shrinkage each step — Subtracting a fraction of each weight every step corresponds to adding an L2 penalty to the objective.

Training

Parameters

Display

What regularisation adds

Test your knowledge

Viewed in WEIGHT SPACE, L2 regularization constrains the weights to lie within a…

Learn these first

Interview Questions

References & Further Reading

Love this visualization?