How it works
A transcript replay starts with a recorded interaction: system instructions, user messages, assistant responses, tool requests, tool results, and relevant harness events. The harness uses that record as test input, but replay does not necessarily mean copying every message back into a model unchanged. Engineers first choose which parts are fixed and which parts are under test.
A useful replay usually follows this sequence:
- Select a transcript and identify the decision or failure being examined.
- Reconstruct the context visible at the chosen replay boundary.
- Freeze unstable dependencies where possible, including retrieved documents, tool results, timestamps, and external state.
- Change one deliberate variable, such as a prompt, memory policy, model, or context-compaction rule.
- Run the agent inside a controlled tool and permission surface.
- Compare the new trajectory and outcome with the recorded run.
The comparison should cover more than the final answer. Inspect which tools were selected, what arguments were formed, where approvals occurred, how much context was consumed, and whether the agent stopped for the right reason.
Transcript replay reduces variation; it does not remove it. Model sampling, provider changes, hidden state, and live dependencies can still produce different behavior. The replay artifact therefore needs to state what was frozen, what remained live, and which differences the evaluator considers meaningful.
Why it matters in an agent harness
Production failures often depend on the exact context that reached the agent. A summary omitted a constraint. Memory introduced stale information. A tool result appeared before an approval instruction. A compaction step retained the task but discarded the operator's boundary. Reading the final response rarely reveals those causes.
Transcript replay turns that history into a reusable test case. After changing context assembly or system instructions, you can rerun a known difficult interaction and ask whether the change corrected the target behavior without breaking another control. A collection of representative transcripts can become a regression set for harness changes.
Replay also improves failure containment. Potentially dangerous tool calls can be intercepted, stubbed, or redirected to a sandbox while preserving the decision point that produced them. This lets an engineer examine authorization, confused-deputy, or excessive-permission failures without repeating the original side effect. Any live call should use an explicitly bounded permission scope and a fresh idempotency strategy.
The method has limits. A transcript captures what was recorded, not every fact that influenced execution. Missing retrieval results, unlogged policy decisions, mutable external resources, or absent model settings can make a replay misleading. Treat incomplete provenance as a test limitation, not as evidence that the harness is stable.
The strongest use is comparative. Run the same transcript against a baseline and a candidate change, then evaluate explicit properties: policy compliance, tool choice, argument safety, recovery behavior, output quality, and stop conditions. A single improved response is weak evidence. Repeated improvement across relevant transcripts, without regressions on control properties, supports a more defensible engineering decision.
Transcript replay vs execution replay
The distinction changes how safely and faithfully you can investigate a run. Transcript replay focuses on the conversation and reconstructed model context. Execution replay focuses on reproducing the wider workflow, including state transitions and tool effects.
| Axis | Transcript replay | Execution replay |
|---|---|---|
| Primary input | Recorded messages and context events | Recorded workflow events, state, and invocations |
| Main question | How does the agent behave under this context? | Can the system reproduce this execution path? |
| External effects | Usually stubbed, frozen, or isolated | May be simulated or re-executed under controls |
| Best suited to | Prompt, memory, model, and context regressions | Recovery, idempotency, orchestration, and state-machine faults |
| Fidelity requirement | Accurate reconstruction of model-visible context | Accurate reconstruction of state and dependency behavior |
Use transcript replay when the variable under test is primarily cognitive or contextual. Use execution replay when correctness depends on workflow state, retries, ordering, or side effects. Complex incidents may require both, but combining them without labeling the boundary makes the result hard to interpret.
The Rifty take
We treat transcript replay as a controlled comparison, not a promise of determinism. We optimize for legible differences: the replay record should show what changed, what stayed fixed, and which control property was evaluated. We accept some simulation loss to prevent a diagnostic run from repeating unsafe or irreversible actions.
Common failure modes
- Replaying only user and assistant text while omitting system instructions, tool results, approvals, or compaction events.
- Leaving retrieval, clocks, remote files, or mutable APIs live without recording that source of variation.
- Changing the prompt, model, memory policy, and evaluator together, then attributing the result to one cause.
- Comparing only final answers and missing unsafe tool selection, malformed arguments, or a broken stop condition.
- Allowing replayed tool calls to reach production resources with the original credentials or identifiers.
- Treating one successful rerun as proof that a stochastic failure has been fixed.
- Scoring behavior without stating the expected control property or acceptable range of variation.
- Preserving transcripts without the metadata needed to reconstruct context order, model settings, and harness policy.
- Calling an incomplete reconstruction a replay without marking missing events and resulting evaluation limits.
Frequently asked questions
Does transcript replay produce the same agent response every time?
No. Transcript replay holds selected inputs stable, but model sampling, provider changes, hidden dependencies, and live tool behavior can still vary. Use it to compare defined behavioral properties across runs, not to demand byte-for-byte output unless the entire evaluated path is designed to be deterministic.
What must a transcript contain to be replayable?
A useful transcript contains ordered model-visible messages, system instructions, tool requests and results, relevant memory or retrieval inputs, and harness decisions such as approvals or compaction. It should also identify model settings and unresolved external dependencies. Missing data must be recorded because it limits what the replay can establish.
Should replayed tool calls execute against live systems?
Usually not. Stub, freeze, or sandbox tool calls when the test concerns model or context behavior. If live execution is necessary, use bounded permissions, isolated resources, fresh identifiers, and explicit side-effect controls. A diagnostic replay should not reproduce the incident it was created to investigate.
How should engineers evaluate a transcript replay?
Define the expected property before running it, then compare the baseline and candidate across tool choice, arguments, policy compliance, recovery, stop conditions, and final output. Record which inputs were fixed and which remained variable. Multiple representative replays provide stronger evidence than one favorable response.
When is execution replay more appropriate than transcript replay?
Use execution replay when the suspected fault depends on workflow state, event ordering, retries, idempotency, or external side effects. Use transcript replay when the main variable is prompts, memory, context assembly, or model behavior. An incident spanning both layers may need separate tests at each boundary.