LLM-as-a-Judge
AI Engineer path · lesson 88 of 124Watch judge biases flip a winner, compare absolute/pairwise/reference scoring, tour the full bias catalog with mitigations, and calibrate the judge against humans before trusting it.
Refunds: within 30 days.
Regarding refunds, customers are generally able to request one within a window of approximately thirty days from purchase, subject to standard terms.
Key insight
An LLM judge scales evaluation but isn’t neutral: just swapping the order can flip the winner (position bias), and length alone inflates scores (verbosity bias). Average over both orders, normalize length, anchor the rubric, and validate against humans before trusting it.
Build a judge you can trust
Order the steps that make LLM-as-a-judge defensible.
Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.
- 1Drop a step here
- 2Drop a step here
- 3Drop a step here
- 4Drop a step here
- 5Drop a step here
- 6Drop a step here
Test your knowledge
Which is a well-known bias of LLM judges?
The short version
An LLM judge is a quick but impressionable contest referee: it tends to favor whoever went first and whoever talked longest. Swap the order or trim the waffle and the verdict can change — so you control for those before trusting the score.
Interview Questions
References & Further Reading
Love this visualization?
Sign up to track your progress, earn achievements, use the AI assistant, and access your personal learning dashboard.
100% free • No credit card required