How it works
A recovery contract turns restart behavior into an explicit state machine. It defines what the harness must persist before and after each consequential transition, then maps every recoverable state to a permitted next action. The contract should cover more than model context. It needs the execution identity, input and policy versions, completed steps, tool-call receipts, side-effect identifiers, retry counts, approval records, and the next safe transition.
A typical recovery sequence is:
- Load the last authoritative state rather than reconstructing progress from a transcript.
- Classify any in-flight operation as completed, failed, not started, or unknown.
- Reconcile unknown outcomes with the target system before issuing another write.
- Resume an idempotent step, apply a compensating action, request a new approval, or stop according to policy.
- Record the recovery decision and its evidence in the execution journal.
The important boundary is the commit point. State recorded before an external side effect may prove intent but not completion. State recorded after it may prove completion only if the external system returned a durable identifier or can be queried. A useful contract therefore specifies both persistence timing and reconciliation behavior. “Retry on failure” is not a recovery contract because interruption can occur after the effect succeeded but before the harness observed the result.
Why it matters in an agent harness
Agent runs cross several stateful surfaces: model turns, tool calls, approval gates, files, queues, databases, and external services. An interruption can leave these surfaces disagreeing about what happened. Without a recovery contract, the harness often guesses from partial logs. That can duplicate a payment-like action, overwrite later work, reuse expired approval, or skip a step that never completed.
The contract contains that uncertainty. It gives the orchestrator a finite set of recovery decisions instead of letting the model improvise one from conversational context. This improves control because restart behavior remains policy-driven. It improves reversibility because irreversible operations can require receipts, narrower commit boundaries, or defined compensating actions. It improves observability because operators can distinguish an ordinary retry from reconciliation after an unknown outcome.
Recovery also affects permissions. A prior approval should not automatically authorize every resumed action. The contract must state whether approval attaches to a run, a specific tool call, a bounded effect, or a time-limited scope. If the resumed plan changes the target, arguments, or expected blast radius, the old approval may no longer apply.
The engineering tradeoff is additional durable state and more explicit adapters around side-effecting tools. That cost is justified where duplicate or missing effects matter. For read-only retrieval, restarting the step may be sufficient. For writes, notifications, deployments, or credentialed operations, recovery needs stronger evidence than a generated transcript or a model’s claim that the work was probably completed.
Recovery Contract vs checkpoint
A checkpoint records state. A recovery contract explains what that state means after interruption and which transitions are safe. The distinction changes whether persistence alone is treated as resilience.
| Concern | Checkpoint | Recovery contract |
|---|---|---|
| Primary purpose | Preserve a snapshot | Govern restart behavior |
| Side effects | May record that a call was attempted | Defines how completion and unknown outcomes are reconciled |
| Approvals | May store an approval record | Defines its scope and validity after resume |
| Retry behavior | Supplies a place to restart | Specifies when retry, compensation, escalation, or termination is allowed |
| Correctness question | “What state did we save?” | “What may safely happen next?” |
A checkpoint is therefore an input to recovery, not the complete policy. Several checkpoints can still be unsafe if none captures external receipts, policy versions, or the boundary between planned and committed effects.
The Rifty take
We treat interruption as a normal execution state, not an exceptional path that can be delegated back to the model. We optimize for evidence-backed resume decisions and accept extra persistence around consequential effects. When the outcome cannot be established, the boundary is clear: reconcile or stop rather than retry blindly.
Common failure modes
- Persisting the transcript while omitting authoritative workflow state, tool receipts, or policy versions.
- Marking a step complete before its external effect has a durable confirmation.
- Retrying an unknown write without an idempotency key or reconciliation query.
- Treating every exception as proof that no side effect occurred.
- Reusing approval after the target, arguments, permission scope, or execution plan changed.
- Resuming from a checkpoint created under incompatible code, prompts, schemas, or policies.
- Defining compensation without testing whether it is permitted, bounded, and independently observable.
- Hiding recovery behind a generic success status instead of recording the interruption and decision.
- Allowing the model to select a recovery path that the control plane has not authorized.
Frequently asked questions
What state should a recovery contract require the harness to persist?
Persist enough authoritative state to decide the next safe transition: execution identity, versioned inputs and policies, completed steps, tool arguments, side-effect receipts, idempotency keys, retry counters, approval scope, errors, and unresolved outcomes. Model transcripts can aid diagnosis, but they should not substitute for workflow state.
How should a harness handle a tool call with an unknown outcome?
Reconcile the outcome before retrying. Query the target system using a durable operation identifier, idempotency key, or expected-state check. If completion still cannot be established, the contract should require escalation, compensation where valid, or termination. It should not let the model infer success from context.
Does a resumed run need a new human approval?
It needs a new approval when the original authorization no longer covers the resumed effect. The recovery contract should bind approval to explicit targets, arguments, permission scope, policy version, and validity conditions. A restart alone need not invalidate approval, but a changed action or wider blast radius should.
Is idempotency enough to make an agent run recoverable?
No. Idempotency can make repeated operations produce one effective result, but recovery also requires durable state, outcome reconciliation, approval handling, version compatibility, retry limits, and safe terminal states. Some effects are not idempotent, and even an idempotent call can fail in ways that require operator-visible diagnosis.
When is a lightweight recovery contract sufficient?
A lightweight contract is sufficient when work is read-only, inexpensive to repeat, and free of consequential external effects. It can persist inputs, progress, retry limits, and errors. Once a run writes data, sends messages, changes permissions, deploys code, or consumes approvals, the contract needs stronger reconciliation and evidence.