Failure isolation

How it works

Failure isolation divides an agentic workflow into units that can fail without taking the rest of the system with them. A useful boundary controls more than process execution. It also limits state changes, credentials, retries, budgets, and the authority to trigger downstream work.

A typical isolation sequence is:

1. The harness starts a unit with explicit inputs, permissions, and execution limits.
2. The unit performs work against private or versioned intermediate state.
3. Its outputs are checked before they become authoritative workflow state.
4. Success commits the accepted result. Failure records the error and leaves unrelated units intact.
5. Recovery retries, resumes, compensates, or rejects only the affected unit.

The boundary might surround one tool invocation, an agent loop, a worker, or an entire workflow stage. The right size depends on where side effects become expensive or ambiguous.

Isolation is incomplete if components still share uncontrolled mutable state, broad credentials, or a retry policy that can repeat external writes. It also fails when an orchestrator treats a partial result as complete. Containment therefore requires explicit status, durable attribution, and a commit rule. The harness must know what ran, what changed, and whether the outcome is safe to expose downstream.

Why it matters in an agent harness

Agent failures rarely stay confined by accident. A malformed retrieval result can enter context, shape a plan, trigger the wrong tool, and then be copied into later outputs. A timed-out write can be retried even though the external system accepted the first request. A compromised worker can use shared credentials to reach resources beyond its task. Each case turns a local uncertainty into a wider operational problem.

Failure isolation limits that propagation. It gives the harness a place to stop and make a bounded decision. Instead of asking whether the entire run succeeded, the operator can ask which unit failed, which effects escaped, and which units remain valid.

That produces several concrete engineering benefits:

• Control: the harness can interrupt or reject one branch without discarding independent work.
• Reversibility: isolated writes can be staged, rolled back, or paired with compensating actions.
• Observability: errors retain a clear relationship to inputs, permissions, tool calls, and state transitions.
• Permission containment: scoped credentials prevent one failing component from exercising unrelated authority.
• Evaluation: a failed unit can be replayed or compared without rerunning the full workflow.
• Recovery: checkpoints and idempotent operations reduce how much work must be repeated.

The boundary must align with real side effects. Splitting code into separate functions does little if every function mutates the same record or uses the same unrestricted token. Conversely, isolating every trivial step can add orchestration cost and make the execution trace harder to follow. I usually place the strongest boundaries before irreversible writes, across trust changes, and wherever an unknown outcome could make a retry dangerous.

Failure isolation also changes how partial success is represented. A fan-out should not be declared healthy because one worker returned a usable result. The orchestrator needs an explicit acceptance policy: which failures are tolerable, which invalidate the aggregate, and which indicate a provider or control-plane problem that must block further execution.

Failure isolation vs fault tolerance

Failure isolation and fault tolerance solve related but different problems. Isolation contains a fault. Fault tolerance keeps an acceptable service or workflow operating despite that fault. A system can have one without the other.

This distinction changes retry and fallback design. A retry may improve tolerance, but it weakens isolation if it repeats a non-idempotent write. A fallback may keep work moving, but it is unsafe if the run still reports normal success while evidence quality or provider coverage has collapsed. Continued execution is only useful when the degraded path is explicit, observable, and acceptable under the run contract.

The Rifty take

We optimize for small, legible blast radii rather than uninterrupted motion at any cost. We accept some orchestration overhead to preserve attribution, scoped authority, and recovery options. If the harness cannot distinguish a contained failure from a valid result, the boundary is not doing useful engineering work.

Implementation checks

• Identify the smallest unit that can create an external or authoritative state change.
• Give each unit explicit inputs, outputs, permissions, execution limits, and terminal states.
• Keep intermediate state private or versioned until validation passes.
• Scope credentials to the resources and operations required by that unit.
• Record tool calls and side effects with per-call attribution.
• Treat timeouts after writes as unknown outcomes, not automatic failures safe to retry.
• Require idempotency keys, reconciliation, or compensating actions where repetition is possible.
• Define how fan-out failures affect the aggregate result; do not let one survivor mask systemic failure.
• Make every degraded path visible in status, warnings, and run artifacts.
• Test containment by forcing a worker crash, malformed output, permission denial, timeout, and partial provider outage.
• Verify that recovery resumes from the last trusted state without replaying unrelated side effects.

Frequently asked questions

Where should I place a failure-isolation boundary?

Place the boundary where authority, trust, or recovery cost changes. Strong candidates include external writes, model-to-tool transitions, fan-out workers, and stages that publish authoritative state. The boundary should be small enough to contain damage but large enough to preserve a coherent unit that can be evaluated, retried, or rejected.

Are retries a form of failure isolation?

No. Retries are a recovery or fault-tolerance mechanism, while failure isolation limits propagation. A retry can undermine isolation when the previous attempt may have completed an external write. Use idempotency, reconciliation, or an explicit unknown-outcome state before repeating an operation whose effects cannot be confirmed.

Does running each agent in a separate process provide enough isolation?

Not by itself. Process separation can contain crashes and memory corruption, but agents may still share credentials, mutable workflow state, retry queues, or unrestricted tools. Effective isolation covers execution, permissions, state publication, budgets, and error handling, with explicit rules for what may cross the boundary.

How should a harness report partial failures?

Report partial failure as a first-class result with affected units, confirmed side effects, unknown outcomes, and the aggregate acceptance decision. Do not convert it into success merely because some branches completed. Operators need to see whether the failure was expected content variance, a contained task error, or a systemic provider problem.