RNN Architecture (Unrolled)

Question 1

Why do RNNs use the same weights at every time step?

Accepted Answer

To generalize learned patterns across any position in the sequence — Weight sharing enables RNNs to apply learned patterns regardless of position in the sequence. A pattern learned at position 3 can be recognized at position 100.

Question 2

What causes the vanishing gradient problem in RNNs?

Accepted Answer

Multiplying the gradient by the recurrent weight matrix at every step — During backpropagation through time (BPTT), gradients are multiplied by W_hh at each step. If the spectral norm of W_hh is less than 1, gradients vanish exponentially.

Question 3

Which RNN architecture is best for sentiment analysis?

Accepted Answer

Many-to-One — Many-to-One processes an entire sequence and produces a single output (the sentiment label). This matches sentiment analysis where we read a review and output one classification.

Question 4

'Unrolling' an RNN means…

Accepted Answer

Copying the cell once per time step and chaining hidden states sequentially — The unrolled view shows the same cell applied at each step, with the hidden state passed along — clarifying how sequences are processed and trained.

Question 5

In the unrolled RNN, the weights at each time step are…

Accepted Answer

Identical across every time step — Parameter sharing across time means one set of weights handles any sequence length and learns position-independent dynamics.

Question 6

The hidden state flows…

Accepted Answer

Forward through time, carrying accumulated context to each successive step — hₜ is computed from xₜ and hₜ₋₁ and passed to step t+1, so information accumulates along the sequence.

Question 7

A 'many-to-many' RNN configuration is used for…

Accepted Answer

Sequence labeling and translation tasks — Outputs at multiple steps suit sequence labeling and seq2seq tasks; other configs include many-to-one and one-to-many.

Question 8

A 'one-to-many' RNN is used for…

Accepted Answer

Generating an output sequence from a single seed input — A single input seeds a sequence of outputs, as in caption generation or music generation.

Question 9

Training the unrolled RNN uses…

Accepted Answer

Backpropagation through time with gradients accumulated across all steps — Because weights are shared, BPTT accumulates each step's gradient contribution before updating the single weight set.

Question 10

The recurrence relation of a simple RNN is roughly…

Accepted Answer

h_t = tanh(Wxh·x_t + Whh·h_{t-1}), blending input with prior hidden state — The new hidden state mixes the current input and previous state through learned weight matrices and a non-linearity.

RNN Architecture Visualizer

Architecture Type

Parameters

Display Options

Step 1 Computation

Forward Pass

Weight Sharing

Vanishing Gradient

Unroll a recurrent step

Test your knowledge

Why do RNNs use the same weights at every time step?

Learn these first

Interview Questions

References & Further Reading

Love this visualization?