How it works
A regression eval starts as a failure you already paid for. Something broke — a tool got called with the wrong argument, a prompt drifted, a guardrail let a bad output through — you diagnosed it, fixed it, and then you froze the case. Freezing means capturing the exact input, the relevant context, and the outcome that now counts as correct, then filing it in a registry the harness runs on every change.
The loop is deliberately mechanical:
- A failure is observed and root-caused.
- The triggering input and its expected outcome are captured verbatim.
- The case is registered with an ID and an assertion — an exact match, a deterministic verifier, or a model-based check for fuzzier outputs.
- The full registry runs on every change to prompts, models, tools, or orchestration.
- A reintroduced failure trips the assertion and blocks or flags the change.
The assertion is the hard part. For structured outputs you can assert exact values. For open-ended agent behavior you assert on a trajectory property or use an LLM-as-judge with a fixed rubric — accepting that the check is now itself probabilistic. What makes it a regression eval, not a general one, is provenance: every case earns its place by having failed once. The registry is append-mostly; you retire a case only when the behavior it guards is intentionally removed.
Why it matters in an agent harness
Agents are nondeterministic and their behavior is coupled to things you change constantly — model versions, prompt wording, tool descriptions, context assembly. A fix in one place silently breaks a behavior three hops away. Manual spot-checking cannot cover that surface, and "it worked when I tried it" is not evidence when the same input can produce different trajectories.
Regression evals convert incidents into permanent coverage. Each production failure becomes a test that can never regress unnoticed, which means your reliability ratchets in one direction. This is what makes model upgrades and prompt refactors survivable: you can swap a model and read, from the registry, exactly which earned behaviors held and which broke. It also localizes blame — a failing case names the specific behavior and the change that reintroduced it, so debugging starts from a known symptom instead of a vague report. In a harness that runs mostly unattended, this is one of the few controls that lets you move fast without trusting your own optimism.
Regression eval vs general eval
Both measure quality, but they answer different questions and belong at different points in the loop.
| Dimension | Regression eval | General eval |
|---|---|---|
| Origin | A specific past failure | A capability or quality target you want to measure |
| Question | "Did a fixed bug come back?" | "How good is the system at X?" |
| Cases | Frozen, provenance-tagged | Broad, representative, often sampled |
| Failure meaning | A hard block — known-bad behavior returned | A score to track and improve |
| Growth | Grows one incident at a time | Curated as a benchmark set |
The distinction changes an operating decision: a regression failure should usually gate the change, while a general-eval score movement is a signal you weigh. Conflating them either turns every score dip into a false alarm or lets real reintroductions slip through as "just a lower number."
The Rifty take
We treat every incident as a debt that is only repaid when it becomes a frozen case in the registry. A fix without a regression eval is a fix on credit — it will silently unwind. We optimize for a reliability ratchet: coverage that only accumulates, so the harness gets harder to break as it ages. The tradeoff we accept is up-front friction on every fix and a registry that must be maintained rather than a one-time benchmark you run and forget.
Common failure modes
- Asserting on brittle surface detail. Pinning exact phrasing on an open-ended output makes the case fail on harmless variation. Assert on the property that actually broke.
- No provenance. A case without a linked incident is impossible to judge later — you can't tell whether a failure means a real regression or a stale expectation.
- Silent skips. Cases that error out and get counted as "passed" give false confidence. Treat a non-run as a failure.
- Frozen-in a bug. If you capture the buggy output as expected, the eval defends the wrong behavior. Verify the expected outcome at capture time.
- Registry rot. Never retiring cases for genuinely removed behavior leaves permanent red that trains the team to ignore failures.
- Judge drift. When the assertion is a model-based check, the judge model or rubric changing quietly redefines pass/fail. Version the rubric and pin the judge.
Frequently asked questions
How is a regression eval different from a regression test in normal software?
The idea is identical — freeze a fixed input and expected outcome to catch a returning bug — but agent outputs are nondeterministic, so exact-match assertions often don't work. You assert on trajectory properties or use a rubric-driven judge instead, accepting that the check itself becomes probabilistic.
When should I add a regression eval?
Add one whenever you diagnose and fix a real failure. Capture the triggering input and the now-correct outcome before the incident is closed. Treat a fix without a frozen case as unfinished work; the eval is what stops the same failure from silently returning after a later change.
How do I write assertions for open-ended agent outputs?
Assert on the specific property that failed, not the whole output. Use a deterministic verifier when structure allows — a required field, a forbidden tool call, a value range. For fuzzy behavior, use an LLM-as-judge with a fixed, versioned rubric, and pin the judge model so pass/fail doesn't quietly drift.
Do regression evals replace general evals or benchmarks?
No — they answer a different question. General evals measure how good the system is at a capability and produce scores you track. Regression evals check whether a known-fixed failure returned and usually gate the change. You want both: broad benchmarks for quality, a growing registry for reliability.
How large should the regression registry get?
It grows one incident at a time and should mostly only grow. Retire a case only when the behavior it guards is intentionally removed. If run cost becomes a problem, shard by change type or run in parallel — but never skip cases silently, since a non-run should count as a failure.