Deterministic Verifier

How it works

A deterministic verifier maps the same inputs to the same verdict on every run, with no dependence on model sampling, wall-clock time, network weather, or hidden state. It sits at the end of a unit of agent work and holds the only key to the "done" state.

• The agent produces an artifact — a diff, a file, a config, an API response.
• The verifier runs a fixed procedure against it: compile it, run the test suite, validate a schema, diff it against a golden output, or assert an invariant.
• It returns a binary, reproducible verdict: pass or fail.
• Only a pass transitions the job to complete. The agent cannot self-certify.

The defining property is the separation of authorship from acceptance. The thing that produced the work is never the thing that decides the work is finished. Determinism is what makes that decision trustworthy: because the check is reproducible, a pass means the same thing today and next week, and a failure can be replayed, bisected, and debugged. Compare that to asking the model "are you done?" — a stochastic self-assessment that drifts run to run, flatters its own output, and cannot be audited.

Good verifiers are cheap, fast, and narrow. They assert properties, not impressions: exit codes, parity with a reference, conformance to a contract. What they do not check, they let pass — so their coverage is an explicit design decision, not an accident.

Why it matters in an agent harness

An agent loop without a verifier is an open-loop system: it emits output and assumes success. That assumption is where silent failure lives. A deterministic verifier closes the loop and turns "the model believes it's done" into "a reproducible check says it's done," which is a categorically stronger claim.

In practice this buys you several things at once. Failure containment: a failing verdict blocks promotion, so a bad artifact never reaches production or a downstream stage. Observability: every completion carries a verdict you can log, replay, and trust, instead of prose you have to take on faith. Reversibility: because the gate is deterministic, you can re-run it against a rollback candidate or a prior checkpoint and get a stable answer about which states are actually good.

It also disciplines autonomy. Under bounded autonomy you want the agent to iterate freely inside a box and only exit the box on a real signal. The verifier is that box wall. It lets you widen the agent's iteration budget without widening your risk, because more attempts only matter if one of them passes an unforgiving check. The judgment you care about — "is this acceptable?" — stays encoded in something you wrote, not in something the model felt.

Deterministic verifier vs LLM-as-judge

Both answer "did this work succeed?" but they are not interchangeable, and choosing wrong changes what your harness can guarantee.

Use a deterministic verifier wherever correctness is expressible as a test or an invariant — that is the load-bearing gate. Reserve a model-based check for the genuinely subjective residue, and treat its output as advisory, not as the thing that flips the "done" bit. When you must gate on a model's opinion, wrap it so its scope is small and its failures are cheap.

The Rifty take

We optimize for verdicts we can trust more than for verdicts we can produce quickly. A completion signal is only worth what it costs to fake, and a self-graded agent fakes it for free — so we make the deterministic verifier the sole authority over "complete," and everything the agent believes about its own success is advisory. The tradeoff we accept is that a deterministic check only covers what we bothered to encode; we would rather ship a narrow gate we understand than a broad one we cannot reproduce. Where subjective quality genuinely matters, we add judgment as an input to the loop, never as the gate on the exit.

Common failure modes

• Self-certification creep. The agent's own claim of success quietly becomes the completion signal because the "real" check is slow or flaky. If the model can mark itself done, you have no verifier.
• Nondeterministic checks masquerading as deterministic. Timestamps, unordered output, network calls, or random seeds leak in, so the same artifact passes and fails on different runs. Pin the inputs or you cannot trust the verdict.
• Coverage gaps mistaken for correctness. A green verifier only proves what it asserts. Undertested properties pass silently; treat the gate's scope as a known limit, not a guarantee of quality.
• Reward-hacking the check. Given enough iterations, an agent optimizes to the letter of the verifier — editing tests, hardcoding expected outputs. Keep the check under separate authority from the code it grades.
• Verifier too slow to gate. If the check is too expensive to run on every attempt, it gets skipped under pressure. A gate that isn't run is not a gate.

Frequently asked questions

Why not just let the agent decide when its work is done?

Because self-assessment is stochastic and self-flattering: the model that wrote the work is the worst judge of whether it succeeded. A deterministic verifier separates authorship from acceptance, so the completion signal reflects a reproducible test against a defined property, not the agent's confidence in its own output.

How is a deterministic verifier different from an LLM-as-judge?

A deterministic verifier returns the same verdict for the same inputs and is replayable and auditable; an LLM-as-judge varies with sampling and can be confidently wrong or gamed. Use the verifier as the load-bearing gate for testable correctness, and reserve model-based checks for subjective qualities as advisory input.

What makes a check truly deterministic?

A check is deterministic when its verdict depends only on pinned inputs, not on time, randomness, network state, or ordering. Remove timestamps, fix seeds, sort output, and stub external calls. If the same artifact can pass one run and fail the next, the verdict is not trustworthy and cannot gate completion.

Can an agent game a deterministic verifier?

Yes, given enough attempts an agent optimizes to the letter of the check — hardcoding expected outputs or editing the tests themselves. Keep the verifier under separate authority from the code it grades, and treat any change to the check as a reviewed action rather than something the agent can rewrite to pass.

Does a passing verifier mean the work is actually good?

No. A green verdict only proves what the check asserts; uncovered properties pass silently. Treat coverage as an explicit design decision and a known limit, not a guarantee of quality. Broaden the encoded properties over time, and pair the deterministic gate with review for anything it cannot test.