How it works
A negative control starts with a known-bad case and an expected rejection. The case should violate one defined requirement while keeping unrelated variables as stable as practical. If an evaluator is meant to detect unsupported completion claims, for example, the control might present a plausible success message paired with a trace showing that the required write never occurred.
A useful control loop is:
- Name the behavior the evaluator must reject.
- Construct a case that exhibits that behavior unambiguously.
- Record the expected verdict and the reason for it.
- Run the case through the same evaluator path used in production.
- Treat acceptance, inconsistent rejection, or rejection for the wrong reason as a signal that the evaluator needs investigation.
The last distinction matters. A control has limited value if it fails only because of an incidental formatting error. The evaluator may still miss the behavior the control was designed to test. Good negative controls are narrow enough to diagnose a particular detection boundary, but realistic enough to exercise the actual parsing, state assembly, and judgment path.
Negative controls should also travel with evaluator changes. A rewritten prompt, model change, schema revision, or preprocessing step can silently alter what the evaluator notices.
Why it matters in an agent harness
An agent harness relies on evaluators to decide whether work may continue, be retried, require approval, or be rejected. Those decisions can affect files, messages, credentials, external systems, and durable state. Testing only successful cases shows that an evaluator can approve acceptable work. It does not show that the evaluator can stop unacceptable work.
Negative controls make this missing half observable. They can reveal an evaluator that always returns success, overweights polished prose, ignores tool results, or loses critical evidence during context assembly. They also help separate model quality from harness quality. An agent may produce a bad result, but the operational failure becomes more serious when the surrounding system accepts it as valid.
The most valuable controls represent failures that could cross a control boundary. Examples include:
- A completion claim without evidence that the side effect occurred.
- A valid-looking result produced outside the permitted scope.
- An approval record referring to a different proposed action.
- A recovery step that reports success while leaving authoritative state unchanged.
- A citation or trace reference that does not support the associated claim.
Controls should cover both outputs and trajectories. A final artifact can look correct even when the agent used prohibited credentials, exceeded an execution limit, or bypassed an approval gate. Conversely, a noisy trajectory may still produce a valid result. The evaluator needs controls aligned with the layer it governs.
Negative controls also improve failure containment. When a known-bad case begins passing after a change, operators have a concrete reason to stop promotion or narrow authority. Without that signal, verifier drift can remain hidden until a production incident exposes it.
Negative control vs regression eval
| Concept | Primary question | Typical input | Failure signal |
|---|---|---|---|
| Negative control | Can the evaluator reject this known-bad condition? | A deliberately invalid result or trajectory | The evaluator accepts it or rejects it for an unrelated reason |
| Regression eval | Did system behavior worsen relative to an established expectation? | A suite of representative tasks and expected outcomes | A meaningful score, behavior, or contract moves outside its accepted range |
A negative control can be part of a regression suite, but the terms are not interchangeable. Regression evals measure stability across broader behavior. Negative controls specifically test the rejection boundary. This distinction changes suite design: aggregate performance can remain stable while a critical known-bad case starts passing.
The Rifty take
We treat rejection behavior as part of an evaluator's contract, not as optional test coverage. We prefer a small set of sharply targeted controls tied to consequential failure modes over a large collection of ambiguous bad examples. If a control cannot explain what boundary it tests, it is not yet a useful control.
Implementation checks
- Give each control one primary violation and one expected rejection reason.
- Run controls through production-equivalent parsing, context assembly, and evaluator routing.
- Verify the verdict and rationale; a rejection for the wrong reason may hide the target defect.
- Include trajectory-level controls when permissions, approvals, or tool use matter.
- Version controls alongside prompts, schemas, policies, and model-routing changes.
- Keep controls deterministic where possible, and repeat stochastic judgments enough to expose instability.
- Fail the evaluator release when a high-consequence negative control is accepted.
- Review controls when the harness gains new tools, authority, state, or recovery paths.
Frequently asked questions
How is a negative control different from an ordinary failing test?
A negative control is designed to validate the evaluator's rejection boundary, while an ordinary failing test may simply expose any defect. The control begins with a known-bad condition and an expected reason for rejection, making it useful for detecting permissive, misdirected, or inconsistent evaluation.
Should a negative control test the final output or the agent trace?
It should test whichever layer the evaluator governs, and often both are needed. Output controls catch invalid artifacts or unsupported claims. Trace controls catch prohibited tools, missing approvals, excessive authority, or unverified side effects that a plausible final answer can conceal.
Can an LLM-based evaluator use negative controls?
Yes, but the control should be repeated when judgment is stochastic and should check more than the verdict label. Record whether the evaluator identifies the intended violation. A model that rejects inconsistently, or for irrelevant reasons, is not reliably enforcing the target boundary.
How many negative controls does an evaluator need?
Use enough controls to cover distinct, consequential rejection boundaries rather than targeting an arbitrary count. Start with failures that could authorize harmful side effects, create false completion, or corrupt durable state. Add controls when new tools, permissions, workflows, or observed incidents introduce materially different failure modes.
What should happen when a negative control unexpectedly passes?
Treat the result as an evaluator failure until the cause is understood. Check context assembly, parsing, routing, prompts, schemas, and model changes before widening acceptance. For a high-consequence boundary, block promotion or reduce the agent's authority rather than treating the case as ordinary score noise.