Transformer Explainer

Question 1

Transformer Explainer visualizes how a GPT-style model…

Accepted Answer

Predicts the next token given the preceding tokens — GPT models are autoregressive: they output a probability distribution over the vocabulary for the next token at each step.

Question 2

The first step when text enters the model is…

Accepted Answer

Tokenization — Text is tokenized to IDs, each ID looks up an embedding, and positional information is added before the transformer layers.

Question 3

A GPT model generates text…

Accepted Answer

One token at a time, feeding each result back as new context — Each predicted token is appended to the context, and the model predicts the next, repeating to produce a sequence.

Question 4

The model's final layer outputs LOGITS, which are converted to next-token probabilities by…

Accepted Answer

Softmax to normalize all logits into a probability distribution — Softmax turns the raw scores over all vocabulary tokens into a probability distribution for sampling the next token.

Question 5

GPT uses CAUSAL (masked) attention so that each token…

Accepted Answer

Can only attend to itself and tokens that came before it — Masking future positions enforces left-to-right generation: predictions depend only on already-generated context.

Question 6

The 'temperature' parameter controls…

Accepted Answer

How peaked or flat the output distribution is at each sampling step — Dividing logits by temperature before softmax sharpens (low T) or flattens (high T) the distribution, trading coherence for creativity.

Question 7

GREEDY decoding picks…

Accepted Answer

Always takes the highest-probability token at each generation step — Greedy always takes the argmax, giving deterministic but sometimes repetitive/bland text; sampling adds diversity.

Question 8

Top-p (nucleus) sampling restricts choices to…

Accepted Answer

The minimal prefix of top tokens whose cumulative mass reaches p — Nucleus sampling samples from the dynamically-sized 'nucleus' of most-probable tokens, balancing quality and diversity.

Question 9

Each transformer block contains…

Accepted Answer

Multi-head attention and a feed-forward layer, linked by residuals and layer norms — Stacking these blocks builds the model's depth; attention mixes information across tokens and the MLP processes each position.

Question 10

Self-attention lets a token's representation incorporate…

Accepted Answer

Weighted signals from contextually relevant positions in the sequence — Attention weights determine how much each previous token contributes, building context-aware representations.

One transformer block

Test your knowledge

Transformer Explainer visualizes how a GPT-style model…

Learn these first

Interview Questions

References & Further Reading

Love this visualization?