How it works
The pattern splits one job into two roles that never collapse into each other:
- Generate. A model produces a candidate output for the task.
- Evaluate. A separate evaluator scores that candidate against an explicit target — a rubric, a test suite, a schema, a numeric threshold, or a reference answer.
- Decide. If the score clears the bar, the loop returns the output. If not, the evaluator's feedback is fed back to the generator as the next instruction.
- Bound. The loop terminates on a passing score or on an iteration/token budget, whichever comes first.
The evaluator can be deterministic (a compiler, a test runner, a schema validator, a linter) or model-based (an LLM judging against a rubric). Deterministic evaluators give a hard, repeatable gate; model-based evaluators cover fuzzy targets like tone or completeness but introduce their own variance. The strength of the pattern is not the number of retries — it is that the thing deciding "good enough" is a distinct component with its own contract, not the same context that produced the work. That separation is what lets you inspect, swap, or tighten the standard without touching the generator.
Why it matters in an agent harness
An agent that only generates has no internal notion of "done" beyond stopping. Evaluator-Optimizer gives the harness a termination condition it can reason about and a signal it can log.
- Control. The loop stops on a criterion you defined, not on the model's sense of completion. You own the exit condition.
- Observability. Every iteration emits a score and a rationale. A trajectory of scores tells you whether the agent is converging, oscillating, or stuck — far more legible than a single final answer.
- Failure containment. Pairing the evaluator with an iteration budget caps cost and prevents a degenerate loop from grinding forever on an unsatisfiable target.
- Reversibility. Because the evaluator gates release, weak output is caught before it reaches a side effect. The gate sits between generation and any irreversible action.
The risk to design around is a weak evaluator. A generator will optimize toward whatever the evaluator rewards, so a shallow rubric produces output that games the rubric. The evaluator is the specification; treat its quality as the ceiling on the loop's quality.
Evaluator-Optimizer vs Reflection
Both improve output through iteration, but they differ in where the judgment lives — and that changes your design.
| Aspect | Evaluator-Optimizer | Reflection |
|---|---|---|
| Judge | A separate component with its own contract | The same model critiquing itself in-context |
| Target | Explicit rubric, tests, or threshold | Implicit, model-internal |
| Auditability | Scores and rationale are external artifacts | Buried in the model's reasoning trace |
| Boundable | Naturally, via score plus budget | Harder; self-critique can loop on itself |
Use reflection when the target is hard to externalize and cheap self-correction helps. Reach for Evaluator-Optimizer when you need an auditable gate, a hard pass/fail, or an evaluator you can strengthen independently of the generator.
The Rifty take
We treat the evaluator as the real artifact. A generator is easy to swap; a precise, versioned evaluator is the thing that encodes the standard, so we invest there and keep it separate on purpose. We prefer a deterministic verifier wherever the target admits one, and reserve model-based judging for genuinely fuzzy criteria — accepting its variance as a cost, not pretending it away. And we always bound the loop: an optimizer without an iteration budget is a way to spend money slowly, not a control.
Implementation checks
- Version the evaluator and its rubric; a silent change to the standard silently changes every output.
- Set an explicit iteration budget and define the fallback when it is exhausted — return best-scoring candidate, escalate, or fail closed.
- Prefer a deterministic verifier when the target has one; fall back to a model judge only for fuzzy criteria, and measure its agreement with ground truth.
- Log the full score trajectory, not just the final result, so you can see convergence versus oscillation.
- Guard against reward-gaming: spot-check high-scoring outputs to confirm the evaluator rewards substance, not surface features.
- Keep the evaluator's inputs isolated from the generator's — a judge that shares the generator's context inherits its blind spots.
Frequently asked questions
How is Evaluator-Optimizer different from Reflection?
Reflection has a model critique its own output inside one context; Evaluator-Optimizer externalizes the judge into a separate component with its own rubric or tests. That separation makes the standard auditable, swappable, and easy to bound, whereas self-critique can loop on its own blind spots without an external gate.
Should the evaluator be a model or a deterministic check?
Prefer a deterministic verifier — tests, schema, compiler, threshold — whenever the target admits one, because it gives a hard, repeatable gate. Reserve model-based judging for fuzzy criteria like tone or completeness, accept its variance as a real cost, and measure its agreement against ground truth.
How do I stop the loop from running forever?
Bound it with an explicit iteration or token budget alongside the passing threshold, and define the fallback when the budget is spent: return the best-scoring candidate, escalate to a human, or fail closed. An optimizer without a budget is a way to burn cost on an unsatisfiable target.
What is the main failure mode of this pattern?
A weak evaluator. The generator optimizes toward whatever the evaluator rewards, so a shallow rubric yields output that games the rubric while looking good. The evaluator is effectively the specification, so its quality sets the ceiling on the loop's quality — invest there first.
Where does the evaluator gate sit relative to side effects?
Place the gate between generation and any irreversible action. Because the evaluator decides release, weak output is caught before it writes, deploys, or sends. That positioning is what makes the pattern useful for reversibility and failure containment, not just quality improvement.