How it works
Verifier drift begins when a verifier and its target stop sharing the same contract. The target may change its output schema, execution path, side effects, timing, or definition of completion. The verifier may retain an older assumption about one of those properties. Both components can remain internally correct while their interaction becomes wrong.
A typical drift sequence is:
- A verifier is written against an observed behavior or explicit contract.
- The runtime, tool, prompt, model, or workflow changes.
- The verifier is not updated, versioned, or exercised against the new behavior.
- Its verdict still looks authoritative because the check completed normally.
The result can be a false rejection, such as failing a valid run because a field moved, or a false acceptance, such as approving completion based on an obsolete status flag. The second case is more dangerous because it turns an evaluator into evidence for a condition it no longer proves.
Verifier drift is not merely model non-determinism. A verifier can tolerate variable trajectories while enforcing stable invariants. Drift occurs when those invariants, their observation points, or their interpretation no longer match the system being evaluated. The key diagnostic question is therefore not whether the output changed, but whether the check still measures the property it claims to measure.
Why it matters in an agent harness
An agent harness depends on verifiers to convert uncertain execution into bounded operational decisions. A verifier may decide whether to retry, stop, publish, request approval, release a lock, commit state, or run a compensating action. When that verifier drifts, the harness can apply the wrong control action with high confidence.
False rejection wastes work and can destabilize a loop. A correct result may be retried until the iteration budget is exhausted, handed to another agent unnecessarily, or escalated to a person with a misleading failure report. Repeated retries can also create new side effects unless the operation is idempotent.
False acceptance weakens containment. An agent may report that a file was written because the verifier checks only the agent’s response, not the filesystem. A workflow may mark a publication complete because it sees a CMS identifier, even though the public URL is unavailable. A permission check may validate the requested operation but miss the actual scope used at runtime. In each case, the verdict is syntactically valid and operationally false.
Drift also damages observability. Operators see disagreement between traces, state, and verdicts but cannot tell whether the worker or the evaluator is wrong. That ambiguity makes recovery slower and can contaminate subsequent decisions if an incorrect verdict is written into authoritative workflow state.
The practical response is to treat a verifier as a versioned dependency of the behavior it evaluates. Its inputs, observation point, invariants, and allowed outcomes should be explicit. Changes to the runtime contract should exercise the verifier with positive cases, negative controls, and known boundary cases. For consequential actions, verification should inspect the resulting state or side effect rather than trusting the agent’s description of what happened.
Verifier Drift vs Contract Drift
The two often appear together, but they identify different repair points.
| Condition | What changed | Primary repair |
|---|---|---|
| Verifier drift | The relationship between a check and the behavior it judges | Update or replace the verifier, its observation point, or its fixtures |
| Contract drift | A producer or consumer no longer follows their agreed interface | Restore compatibility, migrate both sides, or version the contract |
A contract change can cause verifier drift when the verifier is one of the consumers left behind. But verifier drift can also occur without a formal contract change. For example, a check may rely on an undocumented log phrase, timing assumption, or intermediate state that was never guaranteed. Conversely, contract drift between two runtime components may break execution before any verifier runs.
This distinction changes incident handling. If the runtime violates a still-valid contract, repairing the verifier would hide the defect. If the runtime intentionally adopted a new contract and the verifier remained stale, rolling back correct runtime behavior may preserve the wrong control model. The investigation must identify which invariant is authoritative before changing either side.
The Rifty take
We treat a verifier’s verdict as a claim with its own evidence boundary, not as ground truth merely because it came from a check. We optimize for verifiers that inspect durable outcomes and stable behavioral invariants. We accept some maintenance cost to keep those checks versioned alongside the contracts and control decisions they govern.
Common failure modes
- Checking the agent’s completion message instead of the resulting external state.
- Binding a verifier to incidental details such as log text, step count, ordering, or latency.
- Updating a schema or workflow without running the verifier’s positive and negative fixtures.
- Allowing an unknown or unreadable outcome to collapse into success.
- Reusing recorded fixtures after their permissions, tools, or runtime assumptions have changed.
- Treating every failed verdict as worker failure and automatically retrying the same operation.
- Testing only valid outputs, so a verifier that accepts everything appears healthy.
- Sharing one verifier across versioned contracts without recording which version it evaluated.
- Writing the verdict into authoritative state without preserving the observed evidence and verifier version.
Frequently asked questions
How can I detect verifier drift in production?
Detect verifier drift by comparing verdicts with independently observed outcomes and by replaying known positive, negative, and boundary cases against the deployed verifier. Preserve the verifier version, runtime contract version, inputs, observations, and decision so disagreements can be traced to the worker, the check, or their interface.
Is verifier drift the same as a flaky evaluation?
No. A flaky evaluation produces inconsistent verdicts under materially equivalent conditions, while verifier drift can produce the same wrong verdict consistently. Flakiness points toward unstable execution or measurement. Drift points toward a stable check whose assumptions, observation point, or interpretation no longer matches the behavior being judged.
Can verifier drift cause both false failures and false completion?
Yes. A stale verifier can reject correct work because it expects an obsolete representation, or approve incorrect work because it observes a proxy that no longer proves completion. False completion usually carries greater operational risk because downstream systems may commit state or trigger side effects from an unsupported success verdict.
Who should own verifier updates when runtime behavior changes?
Ownership should follow the control contract rather than a team boundary. Whoever changes an evaluated behavior must identify affected verifiers, while the owner of the consequential decision must confirm that the revised evidence remains sufficient. Recording both responsibilities prevents runtime changes from silently weakening approval, completion, or recovery checks.