How it works
A durable checkpoint turns workflow progress into recoverable system state. The harness writes the checkpoint to storage that survives process exits, worker replacement, and machine restarts. That record identifies the run, the completed boundary, relevant inputs and outputs, and enough execution metadata to decide what may happen next.
A typical checkpoint cycle is:
- Load the latest committed checkpoint for the run.
- Validate its schema, workflow version, and referenced artifacts.
- Reconcile any operation whose outcome is still unknown.
- Execute the next incomplete step under an idempotency or deduplication rule.
- Persist the resulting state atomically before advancing the workflow pointer.
The important word is committed. A log line saying that a step started is not proof that it completed. Likewise, an output file is not necessarily authoritative unless the workflow records its identity, validation status, and relationship to the run.
Checkpoint boundaries should follow meaningful recovery units. Persisting after every token or minor calculation creates overhead without useful control. Persisting only at the end leaves expensive work and external side effects exposed to repetition. I usually place a boundary after a validated artifact, a confirmed external mutation, or a decision that changes later permissions or routing.
Why it matters in an agent harness
Agent runs fail in ordinary ways: workers restart, tools time out, credentials expire, operators interrupt execution, and models produce invalid artifacts. Without a durable checkpoint, the harness must either restart blindly or reconstruct state from partial logs and side effects. Both choices weaken control.
A reliable checkpoint improves several engineering properties at once. It makes interruption reversible at the execution level because the run can stop without discarding all prior work. It improves observability by giving the operator an authoritative statement of what the harness believes is complete. It also contains failure by preventing an upstream retry from automatically replaying every downstream action.
This matters most around non-repeatable effects. Suppose an agent prepares a CMS update, sends it, and crashes before recording success. On restart, the harness cannot safely assume either success or failure. The checkpoint must represent that operation as an unknown outcome and force reconciliation against the external system. Treating it as incomplete may duplicate the update. Treating it as complete may silently lose it.
Durability does not make arbitrary execution safe. A checkpoint is useful only when paired with explicit state transitions, stable operation identities, scoped permissions, and recovery rules. It should also preserve the provenance needed to evaluate resumed work. If the model, prompt, tool contract, or source artifact changed, the harness needs a policy for accepting, migrating, or invalidating the old state.
Durable checkpoint vs checkpoint
A checkpoint is any recorded recovery boundary. A durable checkpoint adds a survival and authority requirement: the record remains available after the execution environment disappears and is trusted by the workflow to govern resumption.
| Design question | Checkpoint | Durable checkpoint |
|---|---|---|
| Storage | May exist only in process memory or local ephemeral storage | Persists outside the failed execution context |
| Restart behavior | Often supports retry within one session | Supports recovery after process, worker, or host loss |
| Authority | May be diagnostic or advisory | Acts as committed workflow state |
| Integrity | Can be an informal snapshot | Requires validation, identity, and a defined commit boundary |
| Side effects | May not distinguish attempted from confirmed work | Must preserve completed, incomplete, and unknown outcomes |
The distinction changes architecture. An in-memory checkpoint can reduce repeated computation during a healthy session. It cannot support a claim of resumable execution after the session is gone.
The Rifty take
We optimize for checkpoints that are authoritative enough to constrain the next action, not merely detailed enough to explain the last one. We accept some persistence and reconciliation overhead because repeated side effects and ambiguous recovery are more expensive. If a run cannot prove an operation completed, its state should remain visibly unknown until the harness resolves it.
Common failure modes
- Writing the workflow pointer before the corresponding artifact or side effect is durably confirmed.
- Recording only
startedandcompleted, with no explicit state for an unknown outcome. - Resuming from a checkpoint without validating its schema, workflow version, or artifact references.
- Treating logs, transcripts, or model narration as authoritative workflow state.
- Repeating external mutations without an idempotency key, deduplication rule, or reconciliation step.
- Saving model context while omitting deterministic state such as permissions, selected tools, budgets, and approval decisions.
- Using checkpoint data across changed prompts or tool contracts without an invalidation or migration policy.
- Persisting so frequently that checkpoint overhead dominates execution, or so rarely that recovery repeats expensive work.
- Marking a stage complete because one output exists when required sibling outputs failed or were never validated.
- Allowing resumed execution to inherit broader credentials than the original step was authorized to use.
Frequently asked questions
What should a durable checkpoint contain?
A durable checkpoint should contain the run identity, committed workflow position, validated input and output references, state-transition status, operation identities, and recovery-relevant policy data. It should store enough deterministic state to choose the next safe action without depending on the failed worker’s memory or the model’s narration.
When should an agent harness write a durable checkpoint?
Write a durable checkpoint after a meaningful recovery boundary: a validated artifact, a confirmed external mutation, an approval decision, or a routing change that affects later execution. The checkpoint should be committed only when its referenced state is durable, internally consistent, and safe to treat as completed during resumption.
Does a durable checkpoint prevent duplicate side effects?
No. A durable checkpoint reduces duplicate work, but it cannot by itself prove what happened during a crash between an external side effect and the checkpoint commit. The harness still needs idempotency keys, external reconciliation, or compensating actions for operations whose outcomes can become ambiguous.
How should a harness handle an incompatible checkpoint?
The harness should reject, migrate, or explicitly invalidate an incompatible checkpoint according to a defined recovery contract. It should not silently fill missing fields or resume under changed assumptions. Workflow versions, artifact schemas, tool contracts, and permission state are common compatibility boundaries that require validation.