How it works
Execution replay starts with an execution journal detailed enough to reconstruct the path an agent took. A final transcript is insufficient. The harness must capture the inputs presented at each step, assembled context, model and tool requests, returned values, state transitions, approvals, errors, and identifiers for external operations. It should also retain relevant prompt, policy, tool, and workflow versions.
A replay then follows a declared mode:
- Load the recorded workflow state and versioned configuration.
- Restore or substitute recorded responses for nondeterministic dependencies.
- advance through the original decisions and tool boundaries.
- Compare reconstructed events with the recorded execution.
- Stop, branch, or resume according to the recovery policy.
The critical boundary is side effects. A replay that sends the same email, charges the same account, or writes the same record is not merely diagnostic. It is a second execution. Tool calls should therefore be stubbed, read from recorded results, protected by idempotency keys, or routed into an isolated environment. If a live call is necessary, the harness should mark the divergence and require the permissions appropriate to a new action.
Replay fidelity is always bounded. Changed external state, unavailable model versions, rotating credentials, timing, and stochastic outputs can prevent exact reproduction. A useful replay records those differences instead of claiming equivalence.
Why it matters in an agent harness
Agent failures rarely fit into a final-output field. The visible answer may be wrong because retrieval supplied stale material, context assembly omitted an instruction, a tool returned partial data, an approval was evaluated against the wrong state, or the loop continued after its stop condition. Execution replay makes that sequence inspectable as an operating process rather than a single model response.
That improves several engineering outcomes. Debugging becomes evidence-led because an operator can locate the first meaningful divergence. Evaluation becomes more precise because a changed prompt, policy, model, or tool can be tested against recorded cases without recreating every external dependency. Incident review gains a causal record of what the harness knew and which authority it exercised at each step.
Replay also supports recovery, but only when paired with authoritative state. A harness may resume after the last durable checkpoint, reuse completed read-only work, and avoid repeating confirmed writes. If the outcome of a prior tool call is unknown, replay cannot safely infer success or failure. The workflow needs an explicit reconciliation step before continuing.
The control benefit comes from separating observation from action. An operator should be able to inspect an execution with no production permissions. Reissuing a call, branching from an earlier state, or applying a compensating action is a different operation with a fresh authorization decision. Without that separation, a debugging feature quietly becomes a privileged execution path.
Replay is not free. Detailed journals increase storage, retention, privacy, and secret-handling obligations. The right design records enough semantic state to explain and recover the workflow while redacting credentials and avoiding unnecessary sensitive payloads.
Execution replay vs execution trace
An execution trace is evidence about a run. Execution replay is a procedure that consumes that evidence to reconstruct, compare, or continue the run. Treating them as interchangeable leads teams to collect readable logs that cannot restore state or to build a replay command that repeats unsafe effects.
| Design axis | Execution trace | Execution replay |
|---|---|---|
| Primary purpose | Explain the observed path | Reconstruct or continue the path |
| Required material | Events, relationships, timing, and attribution | Trace data plus restorable state, versions, dependency behavior, and side-effect controls |
| Default authority | Read-only inspection | Read-only unless a separate policy authorizes new actions |
| Main failure risk | Missing or misleading evidence | Duplicate effects or false claims of equivalence |
A trace can stand alone for observability. Replay requires stronger contracts around state identity, ordering, idempotency, permissions, and divergence reporting. If those contracts are absent, call the result a trace viewer or a best-effort simulation rather than a replay system.
The Rifty take
We optimize replay for legibility and safe recovery, not theatrical determinism. A useful system shows which facts were recorded, which dependencies were simulated, and where the reconstructed path diverged. We accept that some executions cannot be reproduced exactly, but we do not accept hidden live effects or an unmarked approximation presented as the original run.
Common failure modes
- Recording model text but omitting assembled context, policy versions, tool arguments, or state transitions.
- Reissuing write operations during diagnosis without idempotency keys, isolation, or explicit approval.
- Assuming the same prompt will reproduce the same decision despite changed models, tools, data, or timing.
- Restoring a checkpoint while ignoring external operations completed after that checkpoint.
- Treating an unknown tool outcome as a failure and retrying an operation that may already have succeeded.
- Allowing replay to inherit production credentials even when the operator only requested inspection.
- Redacting secrets inconsistently, leaving credentials or sensitive payloads in journals and exported traces.
- Substituting recorded dependency responses without marking that the replay is simulated rather than live.
- Comparing only final outputs and missing an unsafe trajectory that happened to produce an acceptable answer.
- Retaining detailed replay data indefinitely without a defined access, deletion, and privacy policy.
Frequently asked questions
Does execution replay require deterministic model output?
No. Replay can substitute recorded model responses, rerun the model and report divergences, or combine both approaches. Exact token reproduction is often less important than reconstructing the workflow state, authority, and tool boundaries. The replay mode must state which components are recorded, simulated, or executed live.
How do you prevent replay from repeating side effects?
Default replay to read-only behavior and intercept effectful tool calls. Return recorded results, route calls to an isolated environment, or require idempotency keys and explicit authorization before live execution. Unknown prior outcomes need reconciliation; blindly retrying them can duplicate an operation that already completed.
What must an execution journal record for replay?
Record workflow and step identifiers, ordered inputs, assembled context, decisions, tool arguments and results, approvals, errors, state transitions, external operation identifiers, and relevant configuration versions. Credentials should be referenced or redacted rather than copied. The journal must preserve causal order, not merely timestamped log lines.
Can execution replay be used to resume a failed workflow?
Yes, when the harness has authoritative resumable state and knows the status of earlier side effects. Resume from a durable checkpoint, reconcile uncertain operations, and authorize any new actions under current policy. A diagnostic reconstruction alone is not enough to guarantee that continuing the workflow is safe.