Model Selection

Question 1

The first stage of a model-selection funnel should be:

Accepted Answer

Hard gates that disqualify outright — Gates are binary and cheap to apply, so they stop you spending evaluation effort on models you were never permitted to use.

Question 2

Which of these is a hard gate rather than a ranking criterion?

Accepted Answer

Data residency requirements — Residency is binary — either the data may go there or it may not. The others are continuous and belong to later stages.

Question 3

Public benchmarks should be used primarily to:

Accepted Answer

Build the shortlist — Public numbers are measured on someone else's prompt distribution and possibly on data the model saw, so they select candidates rather than winners.

Question 4

The only funnel stage whose result transfers directly to production is:

Accepted Answer

Your own evaluation — Your eval measures your task on your distribution. It is also the stage teams drop when they are in a hurry, which is why surprises arrive later.

Question 5

A large gap between a public benchmark score and a freshly-written variant of the same test indicates:

Accepted Answer

Training-data contamination — If the model does far better on the published items than on equivalent new ones, it is recalling answers rather than solving problems.

Question 6

When frontier models all score 92–95% on a benchmark, that benchmark is:

Accepted Answer

Saturated, so remaining differences are largely noise — At the top of a saturated benchmark the remaining points are mostly label noise and ambiguous items, so differences there are not differences.

Question 7

Why can two papers reporting the same benchmark not be comparable?

Accepted Answer

Prompt template, shot count, decoding and scoring script all move the number — Those setup choices shift published numbers by several points, so the same benchmark name often does not mean the same experiment.

Question 8

The strongest argument for a hosted API on a brand-new project is:

Accepted Answer

Fastest iteration with no fixed cost or ops burden — Early on the dominant uncertainty is what the product should be, so you optimise for iteration speed and avoid committing fixed cost.

Question 9

Which factor most favours self-hosting open weights?

Accepted Answer

Data that cannot leave your network — Residency is a constraint no API terms can satisfy. Spiky traffic and small teams both argue the other way.

Question 10

API cost and self-hosting cost cross exactly once because:

Accepted Answer

One is purely variable and the other is mostly fixed — A line through the origin and a nearly flat line have exactly one intersection — that point is the breakeven volume.

Key insight

Narrow the field

Test your knowledge

The first stage of a model-selection funnel should be:

The short version

In more depth

Model Selection

The four stages

API vs open weights

Reading a benchmark

Breakeven

Interview Questions

References & Further Reading

Related topics

Love this visualization?