How it works
An execution trace records a run as a sequence of causally related events. Each event should say what occurred, when it occurred, which component initiated it, what state or input it used, and what outcome followed. The useful unit is not a raw log line. It is a structured event that can be connected to the event before and after it.
A typical trace follows this path:
- The harness records the run identity, initial request, applicable policy, and starting state.
- Each model step records its inputs or durable references to them, the resulting decision, and relevant routing metadata.
- Each tool invocation records the requested operation, permission context, arguments, result, and error status.
- Control events capture approvals, denials, retries, interrupts, limits, checkpoints, and stop conditions.
- The terminal event records the final outcome and whether any external effects remain uncertain.
Trace events need stable identifiers and parent-child links. Those links let an operator distinguish a retry from a new action, connect a tool result to the decision that consumed it, and follow work delegated to another agent. Sensitive values may need redaction, but redaction should be explicit. Removing data without marking the gap makes the trace look more complete than it is.
Why it matters in an agent harness
An agent can produce a plausible final answer after taking a poor path. It can also fail after making several correct decisions. Looking only at the outcome hides both cases. The execution trace exposes the path, which is where many harness failures actually live.
For control, the trace shows whether the harness enforced its boundaries. An operator can verify that an approval preceded a write, that credentials stayed within scope, or that an execution limit stopped further work. If the system claims a policy was applied but no corresponding control event exists, that claim should be treated as unverified.
For reversibility, the trace identifies external effects in order. That does not make every action reversible, but it gives a recovery process the information needed to choose rollback, compensation, or escalation. A trace should distinguish a confirmed effect from an attempted action with an unknown outcome. Retrying the latter without checking can duplicate work.
For observability, traces make distributed agent behavior legible across model calls, tools, workers, and checkpoints. Metrics can reveal that failure rates rose. Logs can expose local errors. A trace connects those signals to one run and preserves causality well enough to diagnose the responsible path.
For evaluation, traces let you judge behavior rather than only outputs. You can test whether the agent selected an appropriate tool, sought approval at the right boundary, repeated retrieval unnecessarily, or reached the answer through unsupported reasoning. This supports regression checks on operating behavior even when model outputs vary.
Trace collection has costs. Detailed payloads increase storage, privacy exposure, and review load. Overly sparse traces leave operators guessing. I prefer structured events for consequential decisions and effects, with references to larger artifacts, rather than indiscriminate capture of every token and internal detail.
Execution trace vs audit trail
The two records can share events, but they serve different operating decisions.
| Axis | Execution trace | Audit trail |
|---|---|---|
| Primary question | How did this run unfold? | Who or what changed a governed resource? |
| Scope | Model steps, tool calls, routing, retries, and control events | Security- or compliance-relevant actions and state changes |
| Detail | Diagnostic and causal | Durable, selective, and accountability-focused |
| Typical use | Debugging, evaluation, replay analysis, and recovery | Review of authority, access, approvals, and material effects |
A trace should not be treated as an audit trail merely because it contains timestamps. An audit record usually needs stronger guarantees around retention, identity, integrity, and access. Conversely, an audit trail may omit intermediate reasoning and failed attempts that are essential for diagnosing an agent run. Use a shared event model when practical, but define each record's guarantees separately.
The Rifty take
We optimize execution traces for causal reconstruction, not maximal data capture. A good trace makes consequential decisions, authority boundaries, and external effects inspectable while accepting that some sensitive or low-value detail should remain outside the record. If a gap prevents a confident reconstruction, the trace should name that gap instead of implying certainty.
Implementation checks
- Give every run, step, attempt, and delegated child stable identifiers.
- Preserve parent-child and retry relationships between events.
- Record policy decisions alongside the policy version or durable reference used.
- Capture tool intent, permission scope, outcome, and external effect status separately.
- Mark redacted, missing, sampled, or truncated data explicitly.
- Distinguish failure, cancellation, interruption, timeout, and unknown outcome.
- Keep trace emission outside the agent's discretion for events the harness must observe.
- Test whether an operator can reconstruct one failed run without consulting ephemeral process state.
- Restrict trace access because prompts, tool arguments, and results may contain sensitive data.
- Verify that tracing failures surface visibly rather than silently producing an incomplete record.
Frequently asked questions
What should an execution trace record?
An execution trace should record run and step identifiers, inputs or durable references, model and routing decisions, tool invocations, permission checks, retries, checkpoints, external effects, errors, and the terminal outcome. It should also mark redaction, truncation, sampling, or missing events so operators can judge the record's completeness.
Is an execution trace the same as a log?
No. Logs are usually component-local messages intended for diagnosis, while an execution trace links structured events across one run in causal order. Logs can contribute evidence to a trace, but timestamps alone do not establish which decision caused a tool call, retry, state change, or final outcome.
Can an execution trace make an agent run replayable?
Not by itself. A trace can supply the ordered decisions, calls, and state references needed for replay analysis, but faithful execution replay also depends on preserved inputs, tool behavior, environment state, model versions, and nondeterministic outputs. The trace should identify unavailable dependencies instead of promising exact reproduction.
How much detail should an execution trace retain?
Retain enough detail to reconstruct consequential decisions, control checks, and external effects without capturing data indiscriminately. Large payloads can be stored separately and referenced by stable identifiers. Apply explicit redaction and access controls, then test whether the remaining record still supports diagnosis, evaluation, and recovery.