How it works
Trajectory evaluation treats an agent run as an ordered record, not a single output. You capture the full trajectory — the initial task, each model decision, every tool call and its arguments, the observation each call returned, and any retries or recovery — then grade the steps rather than the endpoint.
A typical loop:
- Record the run as structured steps, usually as
(state, action, observation)triples with timestamps and IDs. - Define what a good step looks like: was the tool choice justified, were the arguments correct, did the agent read the observation before acting again, did it recover after an error.
- Score each step or span. Use deterministic checks where you can (did the call succeed, did it stay in scope) and an LLM-as-judge or a human rubric where the judgment is subjective.
- Aggregate into path-level metrics: wrong-tool rate, redundant calls, steps-to-completion, recovery success, and whether the agent touched anything outside its intended surface.
You can grade against a reference trajectory — did the run match a known-good path — or reference-free, asking only whether each step was defensible on its own terms. The output is not pass or fail on the answer. It is a map of where the agent's reasoning and actions held and where they broke.
Why it matters in an agent harness
Final-answer scoring hides how the agent got there. Two runs can return the same answer while one took a clean three-step path and the other made nine calls, wrote to the wrong file, and recovered by luck. In a harness, the path is what touches production, so the path is what you have to grade.
Trajectory evaluation is how you connect evals to the controls you actually run:
- Failure containment. You catch runs that reached the right answer through an unsafe route — a destructive call that happened to be reversible this time — before that route fails on a different input.
- Observability. A graded trajectory turns an opaque run into a legible sequence you can inspect, replay, and diff against a prior version.
- Permissions and scope. Step-level scoring surfaces when an agent reaches for tools or arguments outside its intended surface, which a correct final answer would otherwise mask.
- Regression detection. When you change a prompt, model, or tool, path metrics show behavioral drift — more retries, new detours — even when pass rates look flat.
The payoff is that you debug behavior, not just outcomes. You can point at the exact step where the loop went wrong and fix the harness, the tool description, or the budget instead of guessing.
Trajectory evaluation vs outcome evaluation
Both are needed; they answer different questions, and confusing them leads to agents that pass on paper and misbehave in production.
| Outcome evaluation | Trajectory evaluation | |
|---|---|---|
| Grades | The final answer or state | The steps, tool calls, and recovery |
| Answers | Did it work | How did it work, and was that safe |
| Catches | Wrong results | Unsafe routes, wasted calls, brittle recovery |
| Cost | Cheap, one judgment | Higher, many judgments per run |
| Best for | Acceptance gates | Debugging and behavioral regression |
Outcome evaluation is your gate. Trajectory evaluation is your microscope. Use the outcome to decide whether a run is acceptable, and the trajectory to understand and control how acceptable runs are produced.
The Rifty take
We grade the path, because the path is the thing that touches your systems. An agent that reaches the right answer through an unsafe or wasteful route has an unsafe route in production — the correct answer just hid it this time. We optimize for legible, defensible steps over clever shortcuts, and we accept the extra cost of scoring more of the run to buy that visibility. A trajectory you cannot inspect is a trajectory you cannot bound.
Common failure modes
- Grading only the endpoint. If you never look at steps, unsafe and wasteful paths pass as long as the answer is right — until the input changes and the path fails alone.
- Rewarding path-matching over judgment. Scoring only against one reference trajectory punishes valid alternative routes and trains agents to be brittle rather than adaptable.
- Unlogged runs. You cannot evaluate a trajectory you did not record. Missing tool arguments or observations reduce the eval to guesswork.
- Judge without a rubric. An LLM-as-judge with no concrete step criteria drifts and rates inconsistently; write down what a good step is before you automate scoring.
- Aggregating away the signal. A single path score hides where the run broke. Keep step-level detail so you can point at the failing decision, not just a lower number.
- No deterministic anchors. Leaning entirely on model-based scoring makes the eval itself non-reproducible; anchor what you can with deterministic checks.
Frequently asked questions
How is trajectory evaluation different from outcome evaluation?
Outcome evaluation grades the final answer; trajectory evaluation grades the path — the decisions, tool calls, and recovery that produced it. Use outcomes as an acceptance gate and trajectories as a debugging microscope. Both matter: two runs can share an answer while taking very different, and differently safe, routes to reach it.
Do I need a reference trajectory to score a run?
No. Reference-based scoring checks whether a run matched a known-good path, which is precise but punishes valid alternatives. Reference-free scoring asks only whether each step was defensible on its own terms, which is more flexible. Most harnesses use both: references for critical paths, reference-free judgment elsewhere.
How do I keep an LLM-as-judge consistent when grading steps?
Write a concrete rubric before automating: define what a justified tool choice, correct argument, and successful recovery look like. Anchor everything you can with deterministic checks — did the call succeed, did it stay in scope — and reserve the judge for genuinely subjective steps. A judge without a rubric drifts.
What do I need to log to make trajectory evaluation possible?
Record the run as ordered steps: the task, each model decision, every tool call with its arguments, the observation returned, and any retries or errors. Missing arguments or observations reduce the eval to guesswork. Structured, replayable logs are the prerequisite — you cannot grade a trajectory you did not capture.
Which path metrics are worth tracking?
Track wrong-tool rate, redundant or unnecessary calls, steps-to-completion, recovery success after errors, and whether the agent touched anything outside its intended tool surface. These surface behavioral drift after a prompt or model change even when pass rates look flat, and they point at the exact step to fix.