Agent Evaluation

How it works

Agent evaluation runs an agent against a fixed task set and scores the outcome with repeatable checks, instead of watching one run succeed and calling it done. Agents are stochastic and multi-step, so a single passing demo tells you almost nothing about the next thousand runs.

The loop:

• Build a task set. Pair each input with a way to judge success: an expected output, a deterministic verifier, or a rubric for open-ended work.
• Run and capture. Execute the agent and record the full trajectory — tool calls, intermediate state, retries, and final result — not just the last message.
• Score each run. Use exact match or a deterministic check where the answer is checkable; use a model-based judge where it isn't.
• Aggregate. Repeat across seeds and inputs to get a success rate and a distribution, not an anecdote.
• Gate on the number. Block changes that regress the score; promote the ones that move it.

The output is a metric you can track over time, attach to a commit, and defend. Two properties matter most: the set is fixed enough to compare runs against each other, and the scoring is cheap enough to run on every change.

Why it matters in an agent harness

A harness exists to keep an agent controllable and reversible. Evaluation is how you know a control actually works instead of hoping it does. Without evals, every prompt tweak, model swap, or new tool is a blind edit — you ship it, it looks fine in one trace, and you learn about the regression from a user weeks later.

With evals wired into the harness, several things become tractable:

• Regression defense. A change that quietly breaks 12% of tasks shows up as a number, before it ships.
• Change confidence. You can swap a model or refactor a loop and let the score, not vibes, decide whether it was an improvement.
• Failure localization. Trajectory-level scoring tells you where the agent went wrong — a bad tool call, a dropped constraint — not merely that the final answer missed.
• Bounded autonomy. Evals are the evidence that lets you widen an agent's permissions. You grant more scope to behaviors you have measured and hold the line on the ones you haven't.

Evaluation turns "the agent seems to work" into a claim you can stand behind.

Agent evaluation vs model evaluation

Model (benchmark) evaluation asks whether a model is capable. Agent evaluation asks whether your system — model plus prompt, tools, memory, and loop — does the job. The distinction changes what you test and where you invest.

A model that tops a public benchmark can still fail your agent evals because your tools are wrong, your loop stalls, or your prompt leaks. Optimize the thing you actually ship.

The Rifty take

We treat evals as a harness component, not a research afterthought — the same standing as a rollback path or a permission boundary. We optimize for a small, honest task set that runs on every change over a large suite that runs never. We accept that model-based scoring is noisy and imperfect; a slightly fuzzy signal you check constantly beats a precise one you check quarterly. An agent's scope should never exceed what its evals cover.

Common failure modes

• Demo-driven confidence. One good trace stands in for the whole test, and the variance across runs is never measured.
• Overfitting to the set. You tune until the fixed tasks pass and the agent gets worse everywhere else. Rotate and expand the set.
• Scoring only the final answer. A correct output from a broken trajectory — wrong tool, got lucky — passes silently until it doesn't.
• Judge drift. A model-based grader changes behavior across versions and moves your score with no change to the agent. Pin it and spot-check its grades.
• Evals that never run. A suite too slow or expensive to run on every change is documentation, not a gate.
• No failure taxonomy. A single pass or fail hides whether failures are tool errors, constraint violations, or hallucinations — each needs a different fix.

Frequently asked questions

How is agent evaluation different from testing?

Agent evaluation is statistical where testing is deterministic. A unit test asserts one exact result; an eval runs a stochastic, multi-step agent across many inputs and reports a success rate. You still use deterministic checks inside evals, but the top-level signal is a distribution, not a single pass or fail.

How many tasks do I need to start?

Start with ten to thirty tasks that represent your real failure modes, not a giant suite. A small set you run on every change beats a large one you run rarely. Add a task each time production surfaces a new failure, so the suite grows toward the cases that actually break.

When should I use an LLM-as-judge versus a deterministic check?

Use a deterministic verifier whenever the answer is checkable — an exact value, a passing build, a valid schema. Reserve model-based judging for genuinely open-ended output like summaries or explanations, and pin the judge model plus spot-check its grades, because a drifting judge silently moves your score.

Should I score the final output or the whole trajectory?

Score both. Outcome scoring tells you whether the agent succeeded; trajectory scoring tells you why it succeeded or failed — which tool call broke, which constraint it dropped. Trajectory-level detail is what makes a failing eval actionable instead of just a red number you cannot explain.

How do evals support giving an agent more autonomy?

Evals are the evidence that justifies scope. You widen an agent's permissions for behaviors you have measured and held steady, and you keep the unmeasured ones gated. Without a link from coverage to scope, more autonomy is just untested risk you have not priced yet.