Tokenization Strategies (Word vs Subword vs BPE)

Question 1

Why must text be tokenized before a neural network can process it?

Accepted Answer

Neural networks operate on numbers, so text must first be split into discrete units and mapped to numeric IDs — Neural networks compute over numeric tensors, not raw text, so tokenization plus a vocabulary lookup is the first step that turns a sentence into numbers the model can consume.

Question 2

Word-level tokenization splits text into:

Accepted Answer

Whole words, using spaces/punctuation as boundaries — Word-level tokenization treats each whitespace/punctuation-delimited word as one token.

Question 3

A major drawback of pure word-level tokenization is:

Accepted Answer

Any word absent from the training vocabulary maps to an unknown-token entry, and vocabulary size explodes because every distinct surface form requires its own table entry — Because every distinct word form (including typos, rare inflections, and new words) needs its own vocabulary entry, word-level vocabularies grow huge and still can't cover every word a model will ever see.

Question 4

Character-level tokenization splits text into:

Accepted Answer

Individual letters and symbols, each split into its own separate token — Character-level tokenization treats each individual character as one token.

Question 5

A tradeoff of character-level tokenization compared to word-level is:

Accepted Answer

It has a tiny vocabulary but produces much longer sequences, making patterns harder to learn — A character vocabulary is small (just the alphabet plus punctuation), but a sentence turns into many more tokens, making it harder for a model to learn long-range word-level patterns.

Question 6

Subword tokenization (e.g., WordPiece, used by BERT) aims to:

Accepted Answer

Balance vocabulary size and sequence length by splitting rare/unknown words into meaningful smaller pieces while keeping common words whole — Subword methods keep frequent words as single tokens but break rare or unseen words into smaller, reusable pieces, getting the best of both word-level and character-level tokenization.

Question 7

If the word "unbelievable" is not in a WordPiece vocabulary, a subword tokenizer would most likely:

Accepted Answer

Split it into recognizable pieces like "un", "##believ", "##able" — Subword tokenizers fall back to smaller known pieces rather than giving up, so an unfamiliar word gets decomposed into subword fragments the model has seen before.

Question 8

BPE (Byte Pair Encoding) builds its vocabulary by:

Accepted Answer

Iteratively merging the most frequent adjacent pair of symbols in the training corpus — BPE starts from individual characters and repeatedly merges the most frequent adjacent pair into a new symbol, building up a vocabulary of increasingly large, frequent subword units.

Question 9

BPE was originally invented for:

Accepted Answer

General-purpose data compression, later repurposed as a subword tokenization method for natural language text — Byte Pair Encoding was introduced in 1994 as a text compression algorithm, and was repurposed decades later as a subword tokenization scheme for NLP.

Question 10

GPT models use a byte-level variant of BPE mainly because it:

Accepted Answer

Can tokenize any Unicode text without ever producing unknown-token fallbacks, because every possible byte value is included in the base vocabulary from the start — Because every possible byte value is already in the base vocabulary, byte-level BPE can represent any Unicode input, avoiding unknown-token fallbacks entirely.

Input

Configuration

Actions

Which tokenizer has this problem?

Test your knowledge

Why must text be tokenized before a neural network can process it?

Interview Questions

References & Further Reading

Love this visualization?