Word Embeddings & Analogies

Question 1

What is a word embedding?

Accepted Answer

A dense, real-valued vector that captures a word's meaning based on its usage context — Word embeddings are dense vectors (typically 50-300+ dimensions) learned so that words used in similar contexts end up near each other in the vector space.

Question 2

The distributional hypothesis behind word embeddings can be summarized as:

Accepted Answer

A word's meaning is inferred from the words that appear around it — "You shall know a word by the company it keeps" (Firth, 1957) - words appearing in similar contexts tend to have similar meanings, which is what embedding models exploit.

Question 3

In the CBOW (Continuous Bag-of-Words) architecture, the model is trained to:

Accepted Answer

Predict a center word given its surrounding context words — CBOW takes context words as input and predicts the missing center word - the opposite of Skip-gram.

Question 4

In the Skip-gram architecture, the model is trained to:

Accepted Answer

Predict the surrounding context words given a center word — Skip-gram flips CBOW: given a center word, it predicts the words likely to appear around it.

Question 5

Why do "king" and "queen" end up close together in embedding space while "king" and "banana" do not?

Accepted Answer

They appear in similar contexts during training — royalty-related words cluster together — Training on large text corpora exposes the model to countless sentences where "king" and "queen" appear in similar surrounding words (royalty, court, etc.), pulling their vectors together.

Question 6

The famous analogy "king - man + woman ≈ queen" demonstrates that:

Accepted Answer

Vector arithmetic in embedding space can capture semantic relationships like gender — The offset between "man" and "woman" approximates a "gender direction" in the space; adding that same offset to "king" lands near "queen," showing relationships are encoded as consistent directions.

Question 7

Cosine similarity between two word vectors measures:

Accepted Answer

The angle between the two vectors, ignoring their magnitude — Cosine similarity = dot product divided by the product of magnitudes, so it captures direction (semantic orientation) rather than vector length.

Question 8

A key limitation of classic Word2Vec/GloVe embeddings is that:

Accepted Answer

Each word gets one vector, so homonyms like "bank" conflate all their senses — Classic embeddings are "static" - one vector per word type regardless of context - so polysemous words are conflated. Contextual models like BERT fix this by producing a different vector per occurrence.

Question 9

How do contextual embeddings (e.g., from BERT) differ from Word2Vec embeddings?

Accepted Answer

The same word token gets a different vector depending on the surrounding sentence — Contextual models generate a vector for each occurrence of a word based on its surrounding sentence, so "bank" in "river bank" differs from "bank" in "savings bank."

Question 10

Real-world word embeddings (Word2Vec, GloVe) are typically how many dimensions, versus the 2D used for visualization?

Accepted Answer

Typically 50–300 dimensions (768+ for transformer-based models) — Classic embeddings commonly use 50-300 dimensions (BERT/GPT-style contextual embeddings often use 768+), far more than the 2D projection used for human-readable plots.

Interaction

Legend

Analysis

Complete the analogy

Test your knowledge

What is a word embedding?

Interview Questions

References & Further Reading

Love this visualization?