How LLMs and transformers work, visualized
Follow one prompt through a language model: split into tokens, embedded, mixed by attention layer after layer, and decoded one token at a time.
If neural networks are new, read the perceptron and backpropagation topics on the deep learning hub first.
Text in
- Tokenization Strategies (Word vs Subword vs BPE)Type any sentence and switch between Word, Subword (BERT-style), Character, and BPE (GPT-style) tokenization to compare how each method splits the same text, then see how each token maps to a vocabulary ID.Beginner15m
- Tokenization & Encoding (Tiktokenizer)Interactive tokenizer for OpenAI models - see how text is broken into tokens and encoded. Perfect for understanding GPT tokenization.Beginner15m
- Text Representation (OHE, BOW, N-gram, TF-IDF, Word2Vec)Master ALL fundamental NLP techniques: One-Hot Encoding, Bag of Words, N-grams, TF-IDF, and Word2Vec (CBOW & Skip-gram). See how text becomes numbers!Beginner25m
- Word Embeddings & AnalogiesSee how words become points in a vector space where similar meanings cluster together, and how vector arithmetic captures relationships like king - man + woman ≈ queen.Beginner15m
The transformer
- Self-Attention MechanismInteractive visualization of how self-attention works in transformer models.Advanced25m
- Transformer ExplainerLearn how Transformer models like GPT work with a live GPT-2 model running in your browser. Interact with attention maps and see token predictions in real-time.Advanced35m
- Mixture of Experts (MoE)Discover how Mixtral achieves massive scale efficiently, and how GPT-4 is rumored to use a similar mixture-of-experts approach! Visualize the Mixture of Experts architecture: gating networks, top-K routing, sparse activation, and load balancing. The secret behind modern trillion-parameter models.Advanced30m
Text out
- Sampling & Decoding ParametersReshape a real next-token distribution with temperature, cut its tail with top-k or top-p, then sample repeatedly to feel how the knobs trade determinism for diversity — and why T=0 still isn't fully reproducible.Beginner15m
- LLM Inference OptimizationUnderstand how production LLM serving achieves low latency and high throughput — explore KV cache mechanics, continuous batching, quantization tradeoffs, and speculative decoding side by side.Advanced22m
- KV Cache & Attention OptimizationWatch the KV cache turn quadratic recomputation into linear, then work the 2 × B × S × L × H × M formula that makes it 54 GB for a single Llama 2 13B batch — and see how Character.AI stacked three techniques to cut theirs by over twenty times.Advanced21m
- Speculative DecodingLet a small draft model guess K tokens ahead and have the real model check them all in one pass. Drive the accept-reject loop, see why the verification is close to free, and reproduce DeepMind's Chinchilla-70B result — half the latency, identical output.Advanced19m