Evaluation Metrics: Perplexity & Similarity

Question 1

Perplexity is defined as:

Accepted Answer

Two raised to the cross-entropy — Perplexity exponentiates cross-entropy, converting bits into the effective number of equally-likely choices the model was deciding between.

Question 2

A model assigns probability 0.5 to the token that actually came next. The cross-entropy in bits is:

Accepted Answer

1.0 — Cross-entropy is −log₂(0.5), which is exactly 1 bit — the cost of one binary decision.

Question 3

A perplexity of 1 would mean the model is:

Accepted Answer

Completely certain and correct at every token — Perplexity 1 corresponds to zero cross-entropy — probability 1 on the observed token every time. Perplexity equal to vocabulary size means nothing was learned.

Question 4

Suspiciously low perplexity on a public benchmark is evidence of:

Accepted Answer

Possible training-data contamination — If the model finds the benchmark unusually unsurprising, the most likely explanation is that it has seen it before — which invalidates the score.

Question 5

Perplexity is not comparable between two models when they differ in:

Accepted Answer

Tokenizer — Perplexity is normalised per token, so models that split text into different tokens are measuring on different units and the numbers cannot be lined up.

Question 6

Reference: "The refund arrives in three to five business days." Output: "You should see the money back within 3-5 working days." Exact match returns:

Accepted Answer

0, since the strings differ — Exact match is binary on the normalised strings. It is a false negative here, which is the characteristic weakness of the metric on paraphrase.

Question 7

Which task is exact match genuinely the best metric for?

Accepted Answer

Classifying a support ticket into one of eight intents — Classification has exactly one correct label, so anything fuzzier than exact comparison would hide real errors behind partial credit.

Question 8

Unigram precision in a ROUGE-style score is computed as:

Accepted Answer

Overlapping tokens divided by output length — Precision asks how much of what the model said appeared in the reference; recall divides the same overlap by the reference length instead.

Question 9

A correct but verbose answer padded with pleasantries will score:

Accepted Answer

High on recall and low on precision — All the reference content is present so recall is high, while the extra filler inflates the denominator of precision and drags it down.

Question 10

Inserting the word "not" into an otherwise identical answer affects lexical overlap how?

Accepted Answer

It barely changes overlap at all — One extra token out of a dozen leaves overlap almost unchanged, so the metric scores an inverted meaning as nearly perfect. This is a false positive that ships bugs.

Key insight

What is this metric measuring?

Test your knowledge

Perplexity is defined as:

The short version

In more depth

Evaluation Metrics: Perplexity & Similarity

Language-modelling metrics

Similarity metrics

Choosing

Interview Questions

References & Further Reading

Related topics

Love this visualization?