How it works
The evaluator-optimizer pattern turns observed agent behavior into controlled harness changes. The evaluator examines completed runs against explicit criteria. The optimizer uses those findings to select an intervention. The intervention is then tested before it becomes the new baseline.
A practical loop looks like this:
- Capture the run’s inputs, decisions, tool calls, outputs, state changes, and outcomes.
- Evaluate that evidence against task-specific checks, including deterministic rules where possible and model-based judgment where necessary.
- Classify the failure. Common classes include missing context, weak instructions, excessive authority, unsuitable tools, incorrect stopping behavior, and poor recovery.
- Rank candidate harness changes by expected benefit, risk, cost, and reversibility.
- Apply one bounded change in an isolated version of the harness.
- Replay representative cases and run regression evaluations.
- Adopt, revise, or reject the change based on the resulting evidence.
The optimizer need not be an autonomous model. It can be a human operator, a deterministic policy, an agent that proposes patches, or a combination of all three. What matters is the contract between evaluation and change: findings must map to a defined intervention surface, and adoption must depend on evidence rather than intuition alone.
Why it matters in an agent harness
Agent failures rarely belong to the model alone. A run can fail because the harness supplied stale context, exposed the wrong tool, granted excessive permissions, omitted a stop condition, or resumed from ambiguous state. Replacing the model or rewriting the prompt may hide the symptom without correcting the mechanism.
The evaluator-optimizer pattern makes improvement attributable. When each iteration changes one bounded surface, you can ask whether the result came from a new prompt, a narrower permission scope, a different retrieval policy, or a stronger verifier. That attribution is essential when model outputs are non-deterministic. Without it, several simultaneous changes can produce an apparent improvement that cannot be reproduced or explained.
The pattern also protects reversibility. Candidate changes should be versioned, tested outside the production baseline, and tied to the evaluation evidence that motivated them. If a change improves average completion quality but weakens failure containment, the operator can reject or roll it back. Optimization is therefore constrained by control requirements, not reduced to maximizing one score.
A useful evaluator covers more than final-answer quality. It can inspect the execution trajectory for unauthorized tool attempts, redundant calls, missing approval checks, premature stopping, state corruption, or unsafe recovery behavior. These observations reveal failures that a polished final response would conceal.
The pattern does not justify unconstrained self-modification. Letting an agent edit its own harness and immediately run the result collapses evaluation, authorization, and execution into one authority domain. Safer designs separate proposal from adoption. High-impact changes pass deterministic checks, regression cases, permission review, or an approval gate before they affect production work.
Evaluator-optimizer pattern vs feedback loop
Every evaluator-optimizer pattern is a feedback loop, but many feedback loops lack the structure needed for reliable harness improvement.
| Design question | Evaluator-optimizer pattern | General feedback loop |
|---|---|---|
| What produces the signal? | Explicit evaluation of recorded behavior | Any downstream response or observation |
| What changes? | A defined harness surface | Potentially any part of the system |
| How is a change selected? | Ranked intervention based on diagnosed failure | May be implicit or reactive |
| How is adoption controlled? | Regression checks and a bounded release decision | Not necessarily specified |
| Can the change be explained or reversed? | Expected through versioning and attribution | Depends on the implementation |
The distinction changes the design when the system modifies prompts, tools, permissions, or control logic. A loose feedback loop may reinforce whatever signal is easiest to measure. An evaluator-optimizer loop requires the operator to define what counts as improvement, which changes are permitted, and which invariants must not regress.
The Rifty take
We optimize for attributable improvement, not continuous change. A harness modification should answer a diagnosed failure, stay inside an explicit control boundary, and survive regression evaluation before adoption. We accept slower iteration when the alternative is an opaque system that cannot explain which change helped or be restored to a known state.
Common failure modes
- Optimizing a single success score while ignoring permissions, recovery, cost, or failure containment.
- Evaluating only final outputs and missing harmful or wasteful execution trajectories.
- Changing prompts, tools, context, and model routing together, which destroys attribution.
- Letting the same agent propose, approve, and deploy its own high-impact harness changes.
- Training the optimizer against a narrow case set until it overfits known examples.
- Using model-based evaluation without stable rubrics, calibration cases, or deterministic checks for hard constraints.
- Promoting a change without saving the previous baseline, evaluation evidence, and rollback path.
- Treating evaluator disagreement as noise instead of evidence that the acceptance criteria are underspecified.
- Allowing easy-to-measure signals to displace the operator’s actual definition of safe and useful work.
Frequently asked questions
Does the optimizer have to be an AI agent?
No. The optimizer can be a human operator, deterministic policy, model-based proposer, or mixed workflow. Its defining role is to convert evaluation findings into ranked, bounded harness changes. Separating proposal, approval, and adoption is usually more important than whether a model participates.
What should the evaluator inspect?
The evaluator should inspect both outcomes and execution behavior. Useful evidence includes task results, tool calls, permission decisions, state transitions, approval events, stopping behavior, recovery attempts, and resource use. The exact checks should reflect the harness contract rather than a generic notion of answer quality.
Can this pattern run automatically in production?
It can, but adoption authority should match the change’s blast radius. Low-risk parameter adjustments may pass through automated regression gates. Changes to permissions, tools, approval policies, or recovery logic deserve stronger controls, versioned rollout, and often explicit operator approval before reaching production.
How do you know whether an optimization actually helped?
Compare the candidate against a versioned baseline using representative cases, regression cases, and invariant checks. Change one meaningful surface at a time when possible. An improvement is credible only when the evidence identifies the changed mechanism and shows that important safety and control properties did not regress.