Self-Attention Mechanism

Question 1

Self-attention lets each token in a sequence…

Accepted Answer

Attend to all other tokens in the sequence — Each position computes a weighted combination of all positions, so its representation incorporates relevant context from the whole sequence.

Question 2

The three vectors derived from each token in attention are…

Accepted Answer

Query, Key, and Value — Each token is projected into a Query (what it's looking for), Key (what it offers), and Value (its content) to compute attention.

Question 3

Attention WEIGHTS are computed by…

Accepted Answer

The softmax of query-key similarity scores — The dot product of a query with each key gives compatibility scores; softmax turns them into weights that sum to 1.

Question 4

The attention output for a token is…

Accepted Answer

A weighted sum of the Value vectors — Each token gathers information by averaging the Values of all tokens, weighted by how much it attends to each.

Question 5

Scaled dot-product attention divides the scores by √dₖ to…

Accepted Answer

Prevent products from causing tiny gradients — Scaling by the square root of the key dimension keeps the softmax inputs at a reasonable magnitude, stabilizing gradients.

Question 6

A major advantage of attention over RNNs is that it…

Accepted Answer

Processes positions in parallel — Attention relates any two positions in one step (O(1) path length), enabling parallel training and capturing long-distance relationships RNNs struggle with.

Question 7

MULTI-HEAD attention is used to…

Accepted Answer

Attend to different relationships in parallel — Multiple heads learn distinct attention patterns (syntax, coreference, etc.), and their outputs are concatenated and projected.

Question 8

Because attention is permutation-invariant, transformers add ___ to encode word order.

Accepted Answer

Positional encodings — Attention alone ignores order, so positional information is injected (sinusoidal or learned) so the model knows token positions.

Question 9

The computational complexity of self-attention with sequence length n is…

Accepted Answer

O(n squared), every token attends fully — All-pairs attention scales quadratically with sequence length — the motivation for efficient/sparse attention variants for long contexts.

Question 10

CAUSAL (masked) self-attention, used in GPT-style decoders, prevents a token from…

Accepted Answer

Attending to any future tokens in the sequence — A mask zeros out future positions so each token predicts the next using only preceding context — essential for autoregressive generation.

Self-Attention Mechanism

"sat" attending to all tokens

Attention Pattern Analysis

Linguistic Pattern

Scaled dot-product attention

Test your knowledge

Self-attention lets each token in a sequence…

Learn these first

Interview Questions

References & Further Reading

Love this visualization?