Text Representation (OHE, BOW, N-gram, TF-IDF, Word2Vec)

Question 1

What is One-Hot Encoding?

Accepted Answer

Binary vectors with exactly one 1 — One-Hot Encoding represents categorical variables (like words) as binary vectors where exactly one element is 1 and all others are 0.

Question 2

One-Hot Encoding represents a word as…

Accepted Answer

A sparse vector with a single 1 in its position — OHE gives each vocabulary word a unique basis vector; it's simple but high-dimensional and encodes no similarity between words.

Question 3

A key weakness of One-Hot Encoding is that…

Accepted Answer

Every word is equally distant from every other — Every one-hot vector is orthogonal to every other, so 'cat' and 'dog' are no closer than 'cat' and 'car' — no meaning is captured.

Question 4

Bag-of-Words (BoW) represents a document as…

Accepted Answer

A vector where each element counts word frequency — BoW tallies how often each vocabulary word appears, discarding order — simple and effective for many classification tasks.

Question 5

The main information BoW LOSES is…

Accepted Answer

The order and co-occurrence context of words — Because it only counts words, BoW treats 'dog bites man' and 'man bites dog' identically.

Question 6

N-grams (e.g. bigrams) help by capturing…

Accepted Answer

Adjacent word sequences (word order) — Bigrams/trigrams retain short-range order ('not good' vs 'good'), partially recovering context BoW discards — at the cost of a larger feature space.

Question 7

TF-IDF weights a term by its frequency in a document AND…

Accepted Answer

How uncommon the word is across all documents — TF-IDF boosts terms frequent in a document but rare overall, downweighting common words like 'the' that carry little discriminative signal.

Question 8

The IDF (inverse document frequency) factor is HIGH for words that are…

Accepted Answer

Appears in very few documents (discriminative) — IDF = log(N / documents containing the term); rare, informative words get high IDF, common words get low IDF.

Question 9

Word2Vec produces word representations that are…

Accepted Answer

Dense, low-dimensional vectors capturing similarity — Word2Vec learns dense vectors capturing semantic/syntactic similarity, so related words cluster in the embedding space.

Question 10

A famous property of Word2Vec embeddings is that they support…

Accepted Answer

Linear relationships encoding word analogies — Linear relationships in the embedding space encode analogical structure, demonstrating that meaning is captured geometrically.

Technique	Dimension	Sparsity	Semantics	Best For
OHE	\|Vocab\|	Extremely Sparse	None	Categorical features
BOW	\|Vocab\|	Very Sparse	None	Document classification
N-grams	up to \|Vocab\|^N (in practice, # of observed N-grams)	Extremely Sparse	Local context	Text generation, LMs
TF-IDF	\|Vocab\|	Sparse	Importance	Search, similarity
Word2Vec	100-300	Dense	Rich semantics	Modern NLP, transfer learning

Text Representation Techniques

Input Documents (one per line)

One-Hot Encoding (OHE)

Key Properties:

Technique Comparison

Order by how much meaning is kept

Test your knowledge

What is One-Hot Encoding?

Interview Questions

References & Further Reading

Love this visualization?