Comparative Evaluation & Leaderboards

Question 1

Comparative evaluation asks graders to:

Accepted Answer

Choose which of two responses is better — Pairwise choice removes the scale, which is where most inter-grader disagreement lives, and produces far more consistent judgements.

Question 2

The main problem with asking graders to score responses out of ten is:

Accepted Answer

Scales are inconsistent between graders and drift over a session — One person's 7 is another's 5, and the same person shifts over time — so absolute scores carry a lot of noise that pairwise judgements avoid.

Question 3

In the Elo expected-score formula, a rating difference of 0 gives an expected score of:

Accepted Answer

0.5 — Equal ratings predict an even match, so each side is expected to score half a point — the coin-flip baseline the rest of the curve departs from.

Question 4

A 100-point Elo gap corresponds to the stronger model winning approximately:

Accepted Answer

64% of the time — The logistic curve with a 400-point scale puts a 100-point gap at about 64%, which is a visible but far from decisive advantage.

Question 5

Elo ratings are described as zero-sum because:

Accepted Answer

Every point one model gains, its opponent loses — The update moves both sides by equal and opposite amounts, which is what makes the scale meaningful across a whole population of models.

Question 6

The main practical difference between Elo and Bradley–Terry is that Bradley–Terry:

Accepted Answer

Fits all recorded comparisons at once and is order-independent — Elo is incremental and therefore order-dependent; Bradley–Terry is a maximum-likelihood fit over the whole record and yields proper confidence intervals.

Question 7

Roughly how many pairwise votes are needed to separate a 55% win rate from chance?

Accepted Answer

About 400 — At around 400 votes the 95% interval on 55% clears 50%. A 52% edge needs several thousand, which is why close comparisons are so often under-powered.

Question 8

A paired experimental design means:

Accepted Answer

Both models answer the identical prompt set — Running both models on the same prompts removes prompt-difficulty variance from the comparison, which sharply reduces the sample size needed.

Question 9

Twelve wins out of twenty comparisons should be interpreted as:

Accepted Answer

Consistent with the two models being equally good — At n = 20 the confidence interval around 60% comfortably contains 50%, so the result does not distinguish the models at all.

Question 10

Position bias in pairwise evaluation refers to graders favouring:

Accepted Answer

Whichever response is shown first — Order effects are large for both human and model graders, but they run in opposite directions: AI judges show first-position bias and favour the answer shown first, while humans show recency bias and favour the one they see last. Either way, sides must be randomised per item.

Key insight

How an arena leaderboard is built

Test your knowledge

Comparative evaluation asks graders to:

The short version

In more depth

Comparative Evaluation & Leaderboards

From comparisons to a ranking

Sample size is the thing everyone gets wrong

What it hides

Interview Questions

References & Further Reading

Related topics

Love this visualization?