How it works
Durable execution treats an agent run as a sequence of recorded steps rather than a single in-memory process. Each meaningful action—a tool call, a model turn, a state transition—is written to a durable store, along with enough metadata to resume it. The control loop typically:
- Persists the run's state and the position of the next step before advancing.
- Wraps each side-effecting action so its result is recorded and can be replayed.
- Detects orphaned or crashed runs and re-queues them.
- Resumes from the last committed checkpoint, replaying recorded results instead of re-executing completed work.
The engine that owns this—often called a durable control plane—separates what the agent decided from where the process happens to be running. Because state lives outside the worker, any worker can pick up a run. The hard part is not storage; it is making replay safe. A completed side effect must not fire twice, so steps are made idempotent or their outputs are memoized. This is a property you build into the harness, not something the model provides.
Why it matters in an agent harness
Long agent work fails in boring ways: a deploy restarts the worker, a node is preempted, a rate limit stalls a job for an hour. Without durability, every one of those events discards hours of reasoning and leaves half-applied changes behind.
Durable execution converts those failures from data loss into a pause. Concretely it buys you:
- Resumability with a known position. You stop, inspect, and resume at a specific step instead of restarting the trajectory.
- Failure containment. A crash costs one step's retry, not the whole run.
- Observability. The persisted step log doubles as your audit trail and your evaluation input.
- Safe operations. You can redeploy the harness mid-run without killing in-flight work, because the loop is reconstructed from durable state rather than memory.
This is what lets an operator run a fleet of long jobs without babysitting each one. The engine holds the position; the human is not the recovery mechanism.
Durable Execution vs Checkpointing
They are often conflated, but the distinction changes how you design the loop. Checkpointing is saving state. Durable execution is the guarantee that a run resumes correctly from that state—including that no completed side effect runs again.
| Checkpointing | Durable execution | |
|---|---|---|
| Primary concern | Capturing a snapshot of state | Correct resumption of the whole run |
| Scope | A point-in-time value | The step sequence and its scheduling |
| Side effects | Not addressed | Must be idempotent or memoized on replay |
| Failure it solves | Losing computed context | Losing progress and double-applying actions |
A harness can checkpoint context yet still re-issue a payment on restart. Durable execution is the property that closes that gap.
The Rifty take
We treat durability as something we engineer into the loop, not a feature we hope the model has. We optimize for resumability over raw throughput, and we accept the overhead of recording every side-effecting step and enforcing idempotency—because the alternative is silent loss and double-applied actions. A run you cannot resume from a known step is a run you cannot trust to leave unattended.
Common failure modes
- Non-idempotent replay. A resumed run re-executes a completed side effect—sends the email twice, writes the row twice—because the step was recorded after the effect, not made idempotent.
- State that isn't actually durable. Progress lives in worker memory or a local cache, so a restart still loses it.
- Checkpointing without resumption logic. State is saved but nothing reconstructs the loop position, so recovery restarts from the top.
- Orphan detection gaps. Crashed runs are never re-queued and just stall, indistinguishable from healthy long-running ones.
- Unbounded replay cost. Long trajectories replay expensive recorded results on every resume because nothing compacts or pins committed prefixes.
- Coupling decisions to infrastructure. The agent's next step depends on a specific worker or in-process handle, so state can't move and any worker can't pick the run up.
Frequently asked questions
Is durable execution something the model provides?
No. It is an engineered property of the harness, not a model capability. The model decides the next step; the harness persists state, records each side-effecting action, and reconstructs the run on restart. Swapping to a stronger model does not make a run resumable—only your control plane can.
How is durable execution different from checkpointing?
Checkpointing saves state; durable execution guarantees a run resumes correctly from that state, including that completed side effects do not run twice. You can checkpoint context and still re-issue a payment on restart. Durable execution is the property—idempotent replay plus resumption logic—that closes that gap.
Why do side effects need to be idempotent?
Because resumption replays the recorded step sequence. If a step performed a side effect but the worker crashed before committing that fact, a naive resume re-runs it. Making steps idempotent, or memoizing their outputs, ensures replay is safe—so a restart continues the run instead of duplicating actions.
What breaks first when a harness lacks durable execution?
Long runs. A routine deploy, preemption, or stalled rate limit restarts the worker and discards hours of reasoning and half-applied changes. Short jobs tolerate this; multi-hour agent trajectories do not. Without durability, the human becomes the recovery mechanism, which does not scale to a fleet.
Does durable execution slow the agent down?
It adds overhead—persisting state and recording each side-effecting step before advancing. That is a deliberate trade: you accept lower raw throughput for resumability and containment. For long-running or unattended work the trade is strongly positive, because the alternative is silent progress loss and duplicated actions.