Sampling & Decoding Parameters

Question 1

Where does temperature apply in the decoding pipeline?

Accepted Answer

To logits before softmax, dividing by T — Standard formulation: p = softmax(z / T). T scales the logits, changing the distribution before any truncation or sampling — it does not change the model itself.

Question 2

A support-bot pipeline uses top-k = 40 with T = 1.4 and users complain the bot occasionally answers with off-topic tokens. What is the most direct fix?

Accepted Answer

Lower temperature, or use top-p — A large fixed k combined with high T lets rare tokens into the sample pool. Lowering T sharpens the distribution; nucleus (top-p) trims however many tokens fall outside the mass cutoff, which adapts per step.

Question 3

Under greedy decoding (T = 0), the same prompt to the same model version returns two subtly different outputs. Most likely cause?

Accepted Answer

Floating-point/MoE routing effects — Greedy is deterministic given identical logits, but logits themselves depend on batch composition (kernel reductions, MoE routing) and the current server build. Test for structural equivalence, not string equality.

Question 4

Which task most obviously calls for T = 0?

Accepted Answer

Extracting a field into a structured reply — Extraction has one right answer. Sampling variance is pure risk — you want the argmax token every time.

Question 5

What is the essential difference between top-k and top-p sampling?

Accepted Answer

Top-k is fixed count; top-p is cumulative — Top-k is a hard count; top-p (nucleus) adapts to the shape of the distribution — small when it is peaked, larger when it is flat.

Question 6

The provider returns per-token logprobs. Which use of them is most valuable in production?

Accepted Answer

Reading label probability as confidence — Where the provider exposes them, logprobs come free with generation. They serve as confidence for classification, low-confidence triggers for escalating to a bigger model or a human, and heuristics for flagging hallucinated spans. Many providers expose none, or only the top handful of tokens per position.

Question 7

A code-generation product wants faithful copies of a boilerplate 90% of the time and mild variety 10%. Which single-knob tweak fits best?

Accepted Answer

T≈0.2, top-p 0.9 — A tiny amount of temperature plus nucleus sampling gives near-greedy determinism while permitting acceptable variants when the model is uncertain between equally good completions.

Question 8

Self-consistency (Wang et al.) samples k reasoning paths and takes the majority answer. What decoding settings make this work?

Accepted Answer

T ≈ 0.7 with top-p, sampling k independent paths — You need genuine diversity across paths for the vote to be meaningful; T=0 collapses all k samples to the same path.

Question 9

A user reports the model is “too creative and keeps making up API names.” Which single change is most likely to help immediately?

Accepted Answer

Lower temperature/tighten top-p — Fabrication under high T often just means low-probability tokens are being sampled. Sharpening the distribution is the cheapest first fix; grounding via retrieval is the next lever.

Question 10

A colleague sets temperature to 0.9, top-p to 0.95, and top-k to 50. In what order are these applied?

Accepted Answer

temperature first, then top-k and top-p — Temperature is applied first — the logits are divided by T — so it changes which tokens survive truncation. The two truncations then differ: top-k keeps the k highest logits and runs softmax over only those, which is exactly how it saves computation, while top-p needs the full softmax first and then keeps the smallest set whose cumulative probability reaches p. Either way you sample from the renormalized remainder.

Key insight

What each decoding knob actually does

Test your knowledge

Where does temperature apply in the decoding pipeline?

Interview Questions

References & Further Reading

Related topics

Love this visualization?