LLM Ensembling

LLM Ensembling

A single model call is one opinion. Ensembling spends more compute on the same request — instead of picking one model to save cost (routing), it combines multiple outputs to raise quality and robustness.

Self-consistency (majority vote)

Sample the same model N times at temperature > 0 and take the majority answer. Works when there's a single checkable answer (math, logic, classification). Cost scales Nx; accuracy gains are front-loaded and plateau — most of the lift shows up in the first 5-10 samples.

Mixture-of-Agents (MoA)

Multiple "proposer" models each draft an independent answer; an "aggregator" model reads all drafts and synthesizes one final, improved answer. Best for open-ended generation where there's no clean vote. Risk: the aggregator can inherit a blind spot shared by every proposer, and it's one more model call (often a larger, pricier one) on the critical path.

Best-of-N + reranking

Generate N candidates from one policy; a reward/verifier model scores each and the top-scoring one is kept. Cost scales linearly with N; quality gains are sub-linear with clear diminishing returns. The reward model itself is a risk — a miscalibrated one can confidently pick a worse candidate.

Weighted / stacked ensembling

Combine multiple models' output scores as a weighted average rather than a simple vote. Fixed weights can be hand-set (e.g. trust a stronger model more); a learned stacking layer fits the weights from held-out data instead. Down-weighting a confident-but-wrong outlier model prevents it from dominating the combined decision.

Multi-agent debate

Several agents answer independently, then see each other's reasoning across multiple rounds and may revise their own answer, converging (or not) on a consensus. Distinct from agent orchestration (a supervisor delegating subtasks to workers) — debate has no delegation, just iterative critique of answers to the same question. Critically, round 1's majority can be confidently wrong; the value is in the rounds after, where peer critique corrects it. Production systems need a round budget and a fallback (majority-after-N-rounds, or escalate to a human) for cases that never converge.