Checkpointed execution

How it works

Checkpointed execution divides a run into recoverable segments. At selected boundaries, the harness persists enough authoritative state to determine what finished, what remains, and what may have changed outside the harness. Recovery starts from that record instead of reconstructing progress from a transcript or rerunning the workflow from the beginning.

A typical control loop is:

1. Load the latest valid checkpoint and verify that it belongs to the current run and workflow version.
2. Reconstruct the next executable state from recorded inputs, outputs, decisions, and completion markers.
3. Execute the next bounded unit of work.
4. Resolve its outcome, including any external side effect, before marking the unit complete.
5. Write and validate a new checkpoint, then advance the authoritative workflow state.

The boundary matters more than the save frequency. A checkpoint taken before a tool call cannot prove that the call did not happen. One written after a local result but before an external system confirms a change can leave an unknown outcome. The harness therefore needs explicit rules for commit order, idempotency keys, outcome reconciliation, and checkpoint validation.

A checkpoint should contain operational state, not merely conversation text. The transcript may explain what the agent said, but recoverable state must tell the runner what it is allowed and required to do next.

Why it matters in an agent harness

Agent runs fail for ordinary reasons: processes restart, providers become unavailable, execution limits are reached, credentials expire, or an operator interrupts work. Without checkpoints, recovery usually means replaying the whole run or asking the model to infer progress from logs. Both approaches weaken control. Replay can repeat expensive or destructive actions, while inference can mistake an intention for a completed effect.

Checkpointed execution turns recovery into a harness decision. The runner can inspect a declared state transition, validate its prerequisites, and resume within the same permission and budget boundaries. This improves several engineering properties:

• Reversibility: the operator can return to a known boundary, provided external effects are reversible or have compensating actions.
• Observability: the current stage, completed work, pending work, and unresolved outcomes are represented as state rather than buried in prose.
• Failure containment: a failed segment can be retried without reopening unrelated completed steps.
• Evaluation: runs can be compared at stable boundaries, making it easier to identify where trajectories diverged.
• Permission control: resumed work can revalidate credentials and authority instead of inheriting stale access implicitly.

Checkpointing does not make side effects safe by itself. If an agent submits a payment, creates a deployment, or sends a message, persisting “tool call started” is not enough. The harness must distinguish not started, started, confirmed, failed, and unknown outcomes. An unknown outcome should trigger reconciliation against the external system before retry. Otherwise, checkpointing can make duplicate actions more systematic rather than less likely.

State also needs a compatibility boundary. If prompts, tools, schemas, or policies change after a checkpoint is written, blindly resuming may combine old assumptions with new behavior. A practical checkpoint records the workflow version and either supports an explicit migration or fails visibly.

Checkpointed execution vs durable execution

The terms overlap, but they answer different design questions. Checkpointed execution describes how progress is captured and resumed. Durable execution describes the broader guarantee that a workflow can survive process loss, waiting periods, and infrastructure interruptions while preserving its intended semantics.

A system may save checkpoints yet still be non-durable. For example, it may persist model outputs but lose wake-up schedules, retry counters, approval state, or the outcome of an external action. Conversely, a durable workflow engine may use event history or another recovery mechanism instead of exposing explicit checkpoints. Choose based on the recovery contract, not the label.

The Rifty take

We optimize for checkpoints that make the next action mechanically decidable. We accept some storage and orchestration overhead to avoid asking a model to reconstruct authoritative state from narrative history. A checkpoint boundary is only useful when uncertain side effects remain visible and recovery cannot silently cross a changed permission or workflow boundary.

Implementation checks

• Define which state is authoritative and which records are diagnostic only.
• Place checkpoints around bounded units with clear entry and completion conditions.
• Record workflow, schema, prompt, and tool versions needed for compatibility checks.
• Persist inputs, validated outputs, completion markers, budgets, and approval state.
• Represent unknown external outcomes explicitly and reconcile them before retrying.
• Use idempotency controls or compensating actions for repeatable side effects.
• Validate checkpoint integrity before resuming and fail visibly on incompatible state.
• Recheck permission scope, credentials, execution limits, and stop conditions at resume time.
• Test interruption before, during, and after each consequential tool call.
• Confirm that replaying recovery logic cannot duplicate a completed external action.

Frequently asked questions

What should an agent checkpoint contain?

An agent checkpoint should contain the authoritative workflow state needed to choose the next action: validated inputs and outputs, completed-step markers, pending work, budgets, approvals, permission context, version identifiers, and unresolved side effects. Logs and transcripts can supplement this state, but they should not be the only recovery mechanism.

How often should a harness create checkpoints?

Create checkpoints at meaningful recovery boundaries rather than at an arbitrary time interval. Good boundaries surround expensive work, consequential tool calls, approvals, and stages with stable outputs. More checkpoints reduce repeated computation, but they also increase persistence, validation, migration, and side-effect coordination costs.

Does checkpointed execution prevent duplicate tool calls?

No. Checkpointed execution records progress, but duplicate prevention requires an explicit side-effect protocol. Use idempotency keys where available, persist external operation identifiers, distinguish confirmed from unknown outcomes, and reconcile ambiguous calls before retrying. A poorly placed checkpoint can preserve uncertainty without resolving it.

Can a run resume after its tools or prompts change?

It can resume safely only when the harness can establish compatibility. Record the relevant workflow, schema, prompt, policy, and tool versions in the checkpoint. If the new system cannot interpret the old state under the same control contract, migrate it explicitly or stop with a visible incompatibility error.

Is a transcript enough to resume an agent run?

A transcript is usually insufficient because it mixes intentions, observations, and model-generated claims without proving which state transitions committed. It is useful diagnostic evidence, but the runner needs structured completion markers, validated outputs, permission state, and side-effect outcomes to decide the next action without relying on model interpretation.