Tokenization & Encoding (Tiktokenizer)

Question 1

Tokenization is the process of…

Accepted Answer

Splitting text into smaller units called tokens — Before an LLM sees text, it's broken into tokens (words, subwords, or characters), each mapped to an integer ID.

Question 2

Modern LLMs typically use ___ tokenization.

Accepted Answer

Subword — Subword tokenization balances vocabulary size against coverage, representing rare/unknown words as sequences of known subword pieces.

Question 3

Byte-Pair Encoding (BPE) builds its vocabulary by…

Accepted Answer

Iteratively merging the most frequent adjacent symbol pair — BPE starts from characters and repeatedly merges the most common pairs into new tokens, learning a subword vocabulary from data.

Question 4

A key advantage of subword tokenization is handling…

Accepted Answer

Out-of-vocabulary words by splitting them into known subword pieces — Unknown words (e.g. 'tokenization') split into familiar subwords, avoiding a hard OOV failure while keeping the vocabulary bounded.

Question 5

Each token is converted into a number called a…

Accepted Answer

A token ID integer — The tokenizer maps each token to a unique integer ID, which then indexes into the model's embedding table.

Question 6

After tokenization, token IDs are turned into…

Accepted Answer

Dense embedding vectors — Each ID retrieves a learned dense embedding; these vectors are the actual input to the transformer layers.

Question 7

CHARACTER-level tokenization has very small vocabulary but…

Accepted Answer

Produces very long sequences — Splitting into characters means many tokens per word, lengthening sequences and making it harder to model long-range structure.

Question 8

WORD-level tokenization keeps sequences short but…

Accepted Answer

Requires a very large vocabulary and fails on any unseen word — Whole-word vocabularies are enormous and still miss novel words; subword tokenization is the practical middle ground.

Question 9

Subword tokenization is the 'sweet spot' because it balances…

Accepted Answer

Vocabulary size against sequence length and coverage of unseen words — It keeps the vocabulary moderate, sequences reasonably short, and gracefully represents any word — the best of word- and character-level approaches.

Question 10

In API pricing and context limits, token COUNT matters because…

Accepted Answer

Cost and context-window limits are both measured in tokens — LLM usage is billed per token and context length is capped in tokens, so tokenization directly affects cost and how much text fits.

Text to model input

Test your knowledge

Tokenization is the process of…

Interview Questions

References & Further Reading

Love this visualization?