Activation Functions

Question 1

What is the main purpose of an activation function?

Accepted Answer

To introduce non-linearity into layer outputs — Without non-linear activation functions, a neural network, no matter how many layers it has, would behave just like a single-layer perceptron (linear model).

Question 2

Which activation function suffers from the "dying ReLU" problem?

Accepted Answer

ReLU — ReLU neurons can become inactive and only output 0 for any input if the weights update in a certain way, effectively "dying".

Question 3

The primary purpose of a non-linear activation function is to…

Accepted Answer

Let the network model non-linear relations between features — Without non-linearity, any depth of linear layers is equivalent to a single linear layer; activations give networks their expressive power.

Question 4

The sigmoid activation σ(x) = 1/(1+e⁻ˣ) outputs values in…

Accepted Answer

(0, 1) — Sigmoid squashes inputs to (0,1), historically used for probabilities/gates but prone to saturation and vanishing gradients.

Question 5

A major drawback of sigmoid (and tanh) in deep networks is…

Accepted Answer

Saturation causes vanishing gradients in deep layers — When inputs are large in magnitude, the derivative is near zero; chaining many such layers shrinks gradients to nothing.

Question 6

tanh is often preferred over sigmoid for hidden layers because it is…

Accepted Answer

zero-centered, with outputs ranging symmetrically from −1 to 1 — Zero-centered outputs keep activations balanced around 0, improving optimization compared to sigmoid's all-positive output.

Question 7

ReLU is defined as…

Accepted Answer

max(0, x) specifically — ReLU passes positive values unchanged and zeros out negatives — cheap to compute and largely free of vanishing gradients for x>0.

Question 8

Why did ReLU largely replace sigmoid/tanh in deep networks?

Accepted Answer

It avoids vanishing gradients, is cheap — ReLU's constant gradient of 1 for x>0 keeps gradients flowing through deep stacks, enabling much faster, deeper training.

Question 9

The 'dying ReLU' problem is when neurons…

Accepted Answer

They get stuck always outputting zero — If a neuron's input stays negative, ReLU outputs 0 with 0 gradient, so it never recovers. Leaky ReLU/ELU give a small negative slope to prevent this.

Question 10

Leaky ReLU differs from ReLU by…

Accepted Answer

A small non-zero slope for negatives — A small negative-side gradient keeps neurons alive, mitigating the dying-ReLU problem while retaining ReLU's benefits.

Activation Functions Interactive Explorer

Where does this activation belong?

Test your knowledge

What is the main purpose of an activation function?

Interview Questions

References & Further Reading

Love this visualization?