How it works
An evaluation harness is the runner, not the tests. You give it a set of cases — an input, the context an agent would actually see, and a way to decide whether the result is acceptable — and it executes all of them against a specific version of your system, then reports a score.
The loop is small and repeatable:
- Fix the inputs. Freeze known-good prompts, tool responses, and starting state so the same case means the same thing every run.
- Run the agent. Execute the real loop, or a trajectory replay, against the version under test.
- Judge the output. Apply a check: an exact assertion, a deterministic verifier (did the code compile, did the row get written), or a model-based judge scored against a rubric.
- Aggregate. Roll individual results into a pass rate, and diff that rate against the last known-good version.
Because an agent's output is non-deterministic, checks are tolerant by design. You rarely assert one true string; you assert a property held, a side effect happened, or a rubric was satisfied. The harness's job is to make that judgment cheap to repeat, so the marginal cost of re-checking a hundred cases after a one-line prompt edit is near zero. That cheapness is the whole point — it converts "I think this is better" into a measured claim.
Why it matters in an agent harness
Every knob on an agentic system is coupled and invisible. Change a model, reorder a tool description, tighten a system prompt, and you have silently altered behavior across every path the agent can take. Without an evaluation harness, you find out from users.
The harness gives you three things the rest of the harness depends on:
- A regression floor. A change cannot ship if the pass rate drops below the bar. This is what lets you upgrade models or refactor prompts without holding your breath.
- Legible change. A diff in pass rate, broken down by case, tells you what got worse — not just that something did. That maps a fuzzy quality complaint onto specific, reproducible failures.
- Bounded iteration. With a fast eval loop, you can try ten prompt variants in an afternoon and keep the one that measurably wins, instead of arguing from anecdotes.
It also contains failure. A case that once embarrassed you in production becomes a permanent tripwire. The harness is where you deposit hard-won lessons so the system cannot forget them — every fixed bug earns a case, and the suite is the memory of everything that has ever gone wrong.
Evaluation harness vs a single eval
These get used interchangeably and shouldn't be. The distinction changes what you build.
| Single eval | Evaluation harness | |
|---|---|---|
| Unit | One case: input + check | The runner over the whole set |
| Answers | "Did this pass?" | "Did the system regress?" |
| Owns | A correctness claim | Scoring, aggregation, the bar, CI wiring |
| Grows by | Writing a better check | Absorbing new production failures |
An eval is a data point. The harness is the instrument and the bar it's read against. You write evals continuously; you build the harness once and then feed it. Confusing the two leads to teams with a hundred clever checks and no gate — cases that nobody runs together and no threshold that blocks a bad change.
The Rifty take
We treat the evaluation harness as a control surface, not a report card. Its value is that it can stop a change, so we wire it as a gate and hold the line on pass rate rather than admiring dashboards. We prefer deterministic verifiers wherever a check can be exact, and reserve model-based judges for genuinely fuzzy qualities — anchored to a rubric and validated against human labels, never trusted blind. The tradeoff we accept: a slower, narrower suite of real regressions beats a broad synthetic one that scores high and catches nothing.
Common failure modes
- No gate. A suite that runs but never blocks a deploy is documentation, not a harness. Wire it to fail the build.
- Synthetic-only cases. Made-up inputs pass while real user patterns break. Seed the suite from production failures, parks, and bug reports.
- Ungrounded judges. A model grading itself drifts and can be gamed. Pin it to a rubric, check it against labeled data, and use exact verifiers where you can.
- Bar erosion. Quietly lowering the threshold to make a change ship. Treat the bar as a versioned decision, not a nuisance.
- Stale fixtures. Frozen tool responses rot as real APIs change, so the harness passes on a world that no longer exists. Re-capture fixtures on a schedule.
- Averaging away disasters. A high mean pass rate can hide a small set of catastrophic cases. Track the worst cases, not just the aggregate.
Frequently asked questions
How is an evaluation harness different from unit tests?
An evaluation harness scores non-deterministic agent behavior against a bar, where unit tests assert exact outputs on deterministic code. Because an agent's output varies run to run, the harness uses tolerant checks — assertions, deterministic verifiers, or model-based judges — and tracks pass rates and trajectories rather than a single true-or-false result.
How many eval cases do I need to start?
Start with the failures you already have. Turn every bug, park, or embarrassing output into a case, and you will have a useful harness within a day. A small, real, regression-driven suite beats a large synthetic one; coverage grows as production surfaces new failure modes worth freezing.
Should evaluations block deployment?
Yes, treat the harness as a gate for the changes it can judge. Wire it into CI so a prompt, model, or tool change cannot ship if pass rate drops below the bar. Reserve human review for genuinely new behavior the suite has never scored, not for known-good cases.
Can I trust an LLM to grade its own outputs?
Sometimes, but anchor it. Model-based judges drift and can be gamed, so pin them to a rubric, validate them against a human-labeled set, and prefer deterministic verifiers wherever a check can be exact. Use the judge for fuzzy qualities like tone or completeness, not for facts a script can confirm.
What makes a harness useful versus just noisy?
Groundedness and a gate. A useful harness runs real production-derived cases, judges them with checks you trust, holds a versioned bar that can block a bad change, and reports which cases moved. A noisy one runs synthetic cases nobody acts on and reports a mean that hides the disasters.