How it works
LLM-as-judge replaces a hand-written assertion with a scoring model. You hand a second model three things — the input, the candidate output, and a rubric — and it returns a judgment: a pass/fail, a score on a scale, or a preference between two candidates. The pattern comes in a few shapes:
- Pointwise scoring: rate one output against a rubric (1–5, or pass/fail per criterion).
- Pairwise comparison: pick the better of two outputs. It sidesteps the judge's poorly-calibrated sense of absolute scale, so it tends to be more stable.
- Reference-guided: grade against a gold answer or a checklist of required facts.
The rubric is the real artifact. A vague prompt — "is this good?" — produces noisy, un-auditable grades. A decomposed rubric — "does the answer cite a source for each claim; is the tone neutral; does it refuse out-of-scope requests?" — produces judgments you can inspect. You constrain the judge to emit a structured shape, a per-criterion verdict plus a short reason, so the grade is legible and machine-readable rather than a floating number.
Because the judge is itself a stochastic model, you calibrate it against a small human-labeled set before you trust it, and you re-check that calibration whenever you swap the judge model. An uncalibrated judge is just a second opinion you have not verified.
Why it matters in an agent harness
Agents produce open-ended output — plans, tool arguments, prose, refusals — that deterministic checks cannot fully cover. A verifier can confirm JSON parses or a test suite passes; it cannot tell you the agent chose a reasonable next step or stayed in scope. LLM-as-judge fills that gap, and in a harness it earns its place when it:
- Gates transitions: a judge scores an output before the loop advances, acting like an approval gate you do not have to staff by hand.
- Powers regression evals: every prompt or model change runs against a scored suite, so you catch quality drops before they ship instead of in production.
- Grades trajectories, not just final answers: judging the sequence of steps catches an agent that reached a correct answer through reckless or expensive tool calls.
- Scales oversight: it turns the operator's carried judgment — the subjective review only you used to do — into a repeatable check that runs on every job.
The payoff is observability. A judge that emits per-criterion reasons converts "the output felt worse this week" into a specific, tracked signal you can trend and alert on.
LLM-as-judge vs deterministic verifier
They are not competitors; they cover different failure classes, and mixing them up is how eval suites rot.
| LLM-as-judge | Deterministic verifier | |
|---|---|---|
| Checks | Subjective, open-ended quality | Exact, rule-expressible facts |
| Result | Probabilistic, needs calibration | Reproducible, trustworthy by construction |
| Cost | Token cost + latency per grade | Near-free, fast |
| Failure mode | Bias, drift, gameability | Can only assert what you can encode |
The design rule: assert everything you can encode with a deterministic verifier, and reserve the judge for what genuinely resists encoding. If a check can be a regex or a unit test, it should never be a model call. Every criterion you move from the judge to a verifier is one less source of nondeterminism in your eval.
The Rifty take
We treat a judge as an instrument that must itself be under test, not as an oracle. It optimizes for coverage of the qualities that resist rules — and we accept the tradeoff that its grades are probabilistic, so we hold it to a calibration set and watch for drift the same way we watch the system it grades. The boundary we enforce: a judge advises and flags; it does not silently gate an irreversible action. Anything the judge could get wrong sits behind a deterministic check or a reversible step, so a bad grade costs a re-run, not a rollback.
Common failure modes
- Position and verbosity bias: judges favor the first option shown, longer answers, and their own writing style. Randomize order in pairwise runs and score for concision explicitly.
- Self-preference: a model grades outputs from its own family more generously. Use a different judge model than the one under evaluation where you can.
- Rubric drift: the prompt says one thing, the judge grades another. Spot-check judge reasons against labels regularly, not once.
- Uncalibrated trust: shipping a judge without a human-labeled baseline means you are trending a number you never validated.
- Gaming: once a judge gates the loop, the generator learns to satisfy the judge rather than the goal. Rotate held-out criteria and keep a human in the loop on a sample.
- Silent model swaps: changing the judge model invalidates prior scores. Version the judge and re-baseline on every change.
Frequently asked questions
When should I use LLM-as-judge instead of a deterministic verifier?
Use a deterministic verifier for anything you can encode as a rule — parsing, schema, test passes, exact matches — because it is free and reproducible. Reserve LLM-as-judge for open-ended qualities that resist encoding: tone, helpfulness, faithfulness, or whether a plan is reasonable. Assert what you can; judge only the rest.
How do I keep an LLM judge from being gamed once it gates my agent loop?
A generator optimized against a fixed judge learns to satisfy the judge, not the goal. Rotate held-out criteria the generator never sees, keep a deterministic check behind any irreversible action, and sample the judge's outputs for human review. Treat a gating judge as adversarially exploitable, not as a settled oracle.
Do I need to calibrate an LLM judge?
Yes. A judge is a stochastic model, so its grades are only trustworthy once you compare them against a small human-labeled set and measure agreement. Re-run that calibration whenever you change the judge model or rubric. An uncalibrated judge is an unverified second opinion, not an evaluation.
What biases affect LLM-as-judge scores?
The common ones are position bias (favoring the first option), verbosity bias (favoring longer answers), and self-preference (grading its own model family generously). Mitigate by randomizing option order in pairwise runs, scoring concision explicitly, and using a judge model from a different family than the system under test.
Should the judge grade the final answer or the whole trajectory?
Grade both, but weight the trajectory in an agent harness. A final-answer-only judge misses agents that reach a correct result through reckless, expensive, or out-of-scope tool calls. Scoring the sequence of steps catches process failures that a correct output would otherwise hide from your evals.