Model Routing & Cost Engineering

Question 1

The main reason "just use a smaller model" is often a worse cost lever than routing:

Accepted Answer

It drops quality on the hard tail — Traffic is a distribution. Routing lets you match model capacity to the shape of each request.

Question 2

You are running an all-big policy. 60% of traffic is trivial (short, factual). Cheapest immediate change without quality regression?

Accepted Answer

Route trivial traffic to a small model — The easy majority is exactly the population a small model can handle well. Heuristic routing captures the savings without touching the hard tail.

Question 3

A confidence cascade tries small then escalates. What is the strongest cheap escalation signal?

Accepted Answer

Top-token logprob as an uncertainty signal — Logprobs come back with every call; low values on the answer span correlate with the small model being unsure — a clean threshold to escalate on.

Question 4

On a slice of traffic the small model succeeds only 40% of the time. Compared to a heuristic that routes this slice directly to the appropriate tier, the cascade:

Accepted Answer

Costs slightly more because every escalated request also pays for the failed small-model attempt — Each escalation pays the small-model cost even when it fails. On slices where the small model rarely succeeds, heuristic routing directly to the right tier avoids that wasted first-rung cost. Note the baseline: against always calling the big model, the cascade still wins comfortably at 40% — 0.20 + 0.60 x 8.00 = $5.00 per 1M tokens against $8.00. It is only a perfect shape-based router that beats it, and only by the wasted first-rung cost.

Question 5

You measure blended average quality at 0.94, cost down 60%. Later, complaints spike from expert users. Most likely cause?

Accepted Answer

Hard-bucket regression hidden in the average — Averages hide route-specific regressions. Always chart per-route quality, especially on the hard tail where the small share of traffic lives.

Question 6

Best framing for interviews when asked about cutting LLM cost:

Accepted Answer

Route, then cache, then fine-tune — Levers by ROI: routing → caching / compression → fine-tuning. Fine-tuning is the last resort because of dataset and ops overhead.

Question 7

Which escalation signal is unreliable under structured-output masking (JSON-mode / grammar-guided decoding)?

Accepted Answer

Top-token logprob distorted by grammar-guided decoding masks — Masking distorts logprobs. For structured outputs, use a verifier or length-based escalation instead.

Question 8

A router assigns model tiers by shape (length, code presence, keywords). What breaks that heuristic soonest?

Accepted Answer

Traffic drift shifting the distribution the heuristic was fitted on — Shape-based routing is a snapshot of past traffic. Re-calibrate against per-route quality regularly.

Question 9

Two models cost $0.20 vs $8.00 per 1M tokens. What is the "escalation break-even" rate above which a cascade to big is worse than always-big?

Accepted Answer

About 97.5% escalation rate — Break-even: c_small + r·c_big = c_big, so r = 1 − c_small/c_big = 1 − 0.20/8.00 ≈ 97.5%. Because the small model is so much cheaper, a cascade stays cheaper than always-big until escalation rate is extremely high — nearly every request has to escalate before always-big wins.

Question 10

A verifier is a lightweight model used to score generator output against retrieved evidence. In a cascade its role is:

Accepted Answer

Score answer support to decide escalation — Verifiers give a semantic signal that logprobs miss — especially in RAG, where "sounds right" and "is supported" diverge.

Key insight

Route a request by cost

Test your knowledge

The main reason "just use a smaller model" is often a worse cost lever than routing:

The short version

In more depth

Model routing & cost engineering

Four common policies

Escalation signals

Why routing beats picking a smaller model outright

Instrument per-route

Interview Questions

References & Further Reading

Related topics

Love this visualization?