How it works
A behavioral evaluation inspects how an agent performed a task, not only what it produced. Its input is a recorded trajectory: model decisions, tool calls, arguments, permission requests, state transitions, retries, approvals, and stop behavior. The evaluator compares those events with explicit expectations for safe and competent operation.
A typical evaluation loop is:
- Capture a trace from a real, simulated, or replayed task.
- Normalize unstable details such as timestamps or generated identifiers.
- Assert required, permitted, and prohibited behaviors.
- Grade the trajectory with deterministic checks, model-based judgment, or both.
- Save a failing case as a regression fixture and rerun it after harness changes.
The strongest checks describe observable contracts. An agent may be required to read before writing, request approval before an irreversible action, remain within a tool budget, or stop after an authorization failure. Other checks allow several valid paths but grade whether the chosen path stayed within the control boundary.
Because agent behavior is non-deterministic, one run rarely proves reliability. Repeated runs can reveal whether a failure is systematic, intermittent, or sensitive to context. The evaluation should preserve enough evidence to explain the grade rather than reducing a complex trajectory to an unsupported score.
Why it matters in an agent harness
A correct final artifact can conceal a dangerous execution path. An agent might produce the requested file after reading unrelated secrets, retrying a destructive operation, bypassing an approval gate, or exhausting far more tools than the task justified. Output-only evaluation misses these failures because it observes the destination but not the route.
Behavioral evaluation makes harness policies testable. Permission boundaries, approval rules, iteration limits, recovery behavior, and stop conditions become assertions over recorded events. This connects architectural intent to evidence. If a policy exists only in a prompt or design document, a later model, tool description, or context change can weaken it without producing an obvious output defect.
It also improves failure containment. A production incident can be reduced to the smallest trajectory that reproduces the unsafe behavior, then added to the evaluation suite. The team can test a proposed fix against that case and check whether the fix creates a different failure, such as excessive refusal or unnecessary approval requests.
The main limitation is observability. An evaluator cannot grade actions the harness did not record, and ambiguous traces encourage ambiguous judgments. Event schemas should preserve tool identity, arguments, results, authority, state changes, and the reason a control allowed or denied an action. Sensitive values can be redacted, but the operational meaning must remain legible.
Behavioral evaluation vs outcome evaluation
The distinction changes what evidence you collect and what regressions you can catch.
| Axis | Behavioral evaluation | Outcome evaluation |
|---|---|---|
| Primary question | Did the agent operate correctly? | Was the resulting artifact acceptable? |
| Evidence | Actions, decisions, tool calls, state transitions | Final answer, file, score, or task result |
| Best at detecting | Policy violations, unsafe paths, wasteful loops, faulty recovery | Incorrect, incomplete, or low-quality outputs |
| Main blind spot | Unrecorded behavior and poorly specified policies | Hidden execution failures behind a good result |
Neither replaces the other. A safe trajectory can still produce a poor result, while a strong result can come from an unacceptable trajectory. Production evaluation should keep the grades separate so that success on one axis cannot silently compensate for failure on the other.
The Rifty take
We treat behavior as part of the deliverable because an agent’s route determines its blast radius, recoverability, and operating cost. We accept some evaluation complexity to keep control policies visible and regression-testable. A favorable final output never overrides a clear permission, approval, or containment failure.
Common failure modes
- Grading only the final response while ignoring tool calls and state changes.
- Writing vague criteria such as “acted safely” without observable pass conditions.
- Treating one successful stochastic run as evidence that a regression is fixed.
- Using a model-based judge where a deterministic policy check would be clearer.
- Recording tool names but omitting arguments, authority, results, or denial reasons.
- Letting a combined score hide a critical control violation behind strong output quality.
- Building fixtures around incidental trace details that change without affecting behavior.
- Saving incident traces without converting them into repeatable regression cases.
Frequently asked questions
What should a behavioral evaluation record?
Record the events needed to reconstruct the agent’s operating path: decisions, tool calls and arguments, results, permission context, approvals, retries, state transitions, and stop reasons. Redact sensitive values where necessary, but preserve enough structure to determine what authority was used and why each consequential action occurred.
Should behavioral evaluations use deterministic checks or an LLM judge?
Use deterministic checks for crisp contracts such as tool limits, required approvals, prohibited calls, and state-transition rules. Use a model-based judge for behavior that requires contextual interpretation. Keep its evidence and rubric visible, and never let a probabilistic grade override an unambiguous control-policy violation.
How many times should a behavioral evaluation run?
Run it enough times to expose meaningful variation in the agent’s trajectory, with the count chosen according to risk and cost. A deterministic harness invariant may need a small regression set, while a stochastic planning behavior needs repeated trials. Report the distribution of failures rather than selecting a favorable run.
How does an incident become a behavioral regression test?
Reduce the incident to the smallest task, context, permissions, and tool responses that still reproduce the failure. Define the prohibited or required observable behavior, retain the relevant trace evidence, and rerun the case after changes. Avoid freezing incidental identifiers or wording that does not affect the underlying control failure.