How it works
Checkpointing turns an agent run from a single volatile process into a sequence of durable, resumable states. After each completed step, the harness writes enough state to durable storage that the run can be reconstructed without redoing prior work.
A typical loop looks like this:
- Execute a step — a model turn, a tool call, or a batch of both.
- Reduce the result into the run's canonical state: transcript, scratch memory, tool outputs, cursor position, and any external side effects already committed.
- Write that state to durable storage under a monotonically increasing checkpoint ID.
- Only then advance the loop. On restart, load the latest valid checkpoint and continue from there.
The unit that matters is the boundary. A checkpoint is a promise that everything before it is done and recorded, and everything after it has not started. Good checkpoints are taken at step boundaries where the work is genuinely settled — not mid-tool-call, not mid-write. What you persist is a design decision: full state is simplest to reason about; an event log you replay is cheaper to store but forces you to make replay deterministic. Either way, the checkpoint is the contract the harness resumes against.
Why it matters in an agent harness
Long agent runs fail for boring reasons: a preempted spot instance, an OOM, a rate limit that trips a retry storm, a deploy that restarts the worker. Without checkpoints, every one of those failures costs you the entire run — the tokens, the wall-clock, and the tool side effects you have to reconcile by hand.
Checkpointing buys you four concrete things:
- Recoverability. A crash re-queues the job and it resumes from the last saved point instead of page one. Hours-long drains survive their own infrastructure.
- Observability. A checkpoint stream is an audit trail. You can inspect the exact state the agent held before a bad decision, replay from it, and diff outcomes.
- Controllability. If you can persist and rehydrate state, you can pause. Human-in-the-loop review, an approval gate, or a model swap becomes a clean stop-and-resume rather than a fork.
- Cost containment. You never pay twice for the same completed step, and a runaway loop is bounded by the last checkpoint rather than the whole run.
The subtle payoff is that checkpointing makes an agent interruptible on purpose. An agent you can stop safely at any step boundary is an agent you can actually operate.
Checkpointing vs Rollback
These are often conflated because they use the same saved state, but they answer different questions. Checkpointing is about forward recovery — resume where you left off. Rollback is about reversal — undo to a prior point and take a different path.
| Checkpointing | Rollback | |
|---|---|---|
| Intent | Resume forward from last good state | Return to an earlier state deliberately |
| Trigger | Crash, preemption, planned pause | Bad decision, failed eval, wrong branch |
| Side effects | Assumed already committed | Must be compensated or undone |
| Direction | Continue the trajectory | Discard and re-attempt the trajectory |
The distinction changes design. Checkpointing lets you skip already-committed side effects on resume, so it wants idempotent steps. Rollback has to reverse side effects, so it needs compensating actions or a fully reversible tool surface. Build checkpointing and you get resume; you only get rollback if you also design for reversibility.
The Rifty take
We treat the checkpoint boundary as the atomic unit of an agent's life, not an optimization bolted on at the end. We optimize for step boundaries that are honest — a checkpoint is only taken when the side effects it claims are truly durable — and we accept the extra write cost and the discipline of idempotent steps to get it. The boundary you can resume from is also the boundary you can pause at, review at, and reason about. An agent that cannot be checkpointed cannot be operated at length; it can only be launched and hoped over.
Common failure modes
- Checkpointing before the side effect commits. You persist "sent the email" but the send hadn't flushed. On resume the step is skipped and the effect is lost. Order the write after the commit, or make the effect idempotent so replay is safe.
- Non-idempotent resume. The run resumes and re-executes a step whose side effect was already applied, so you double-charge, double-post, or double-file. Key every external action so a repeat is a no-op.
- State that outgrows storage. The transcript grows unbounded and each checkpoint gets more expensive until writes dominate the run. Pair checkpointing with compaction so you persist a bounded, meaningful state.
- Silent schema drift. You change the state shape, then an old checkpoint won't load. Version the checkpoint format and refuse to resume against a shape you can't parse rather than resuming into corruption.
- Checkpointing mid-step. Saving in the middle of a tool call captures a state no clean resume exists for. Only checkpoint at settled boundaries.
- Trusting the write without reading it back. A checkpoint you never verified may be truncated. Validate on load and fall back to the prior valid checkpoint when the latest is unreadable.
Frequently asked questions
When should an agent take a checkpoint?
Take a checkpoint at settled step boundaries — after a model turn or tool call whose side effects are fully committed, and before the next step starts. Never checkpoint mid-tool-call. The right cadence balances write cost against how much work you can afford to lose on a crash.
What state does a checkpoint need to include?
Enough to reconstruct the run: the transcript or message history, scratch memory, tool outputs, the loop cursor, and a record of side effects already committed. You can persist full state for simplicity or an event log you replay for smaller storage, provided replay is deterministic and idempotent.
How is checkpointing different from durable execution?
Durable execution is the broader guarantee that a workflow survives failures and resumes exactly-once; checkpointing is the concrete mechanism underneath it. Durable execution engines checkpoint step results and replay history to rebuild state. You can hand-roll checkpointing without a full durable-execution framework, but the framework standardizes it.
Does checkpointing give me rollback for free?
No. Checkpointing lets you resume forward from saved state; rollback requires reversing side effects to return to an earlier point. Resume assumes committed effects stay committed, so it wants idempotent steps. Rollback needs compensating actions or a reversible tool surface. Build reversibility separately if you need to undo.
How do I make resume safe when steps have side effects?
Make each external action idempotent — key it so a repeat is a no-op — and order the checkpoint write after the effect commits, not before. On resume, skip completed steps by their recorded keys. This prevents the classic double-charge or lost-effect bugs when a run restarts mid-trajectory.