How it works
Fault injection testing introduces a controlled failure at a known point in an agent workflow, then inspects the resulting state and recovery path. The injected fault might interrupt a model call, reject a tool invocation, time out a worker, corrupt a non-authoritative response, or terminate execution between an external side effect and its acknowledgement. The objective is not to make the model fail creatively. It is to test whether the surrounding harness behaves correctly when a dependency or execution step does not complete normally.
A useful test follows a short control loop:
- Establish the authoritative pre-fault state and expected invariants.
- Select an execution boundary where failure has a meaningful consequence.
- Inject one explicit fault without changing unrelated conditions.
- Observe state transitions, emitted records, retries, and external effects.
- Verify that the harness stops, resumes, retries, or compensates as specified.
- Repeat the case around adjacent boundaries to expose timing-sensitive gaps.
The test needs an oracle stronger than “the run eventually finished.” It should check which work was committed, whether any outcome remains unknown, whether repeated execution is safe, and whether the operator can reconstruct what happened. A successful recovery that silently duplicates an external action is still a failed test.
Why it matters in an agent harness
Agentic workflows cross boundaries that ordinary function tests often avoid. They call remote tools, write durable state, delegate work, wait for approval, and continue from earlier context. Each transition creates a gap between intent, execution, acknowledgement, and recorded state. Fault injection testing makes those gaps observable before production failures find them.
The first engineering outcome is state integrity. If a worker stops after producing a result but before recording completion, the harness must distinguish unfinished work from an acknowledged result. Treating both states as “retry” can duplicate side effects. Treating both as “done” can lose work. The recovery decision should come from durable evidence, not from an optimistic status label.
The second outcome is bounded recovery. A retry policy is not sufficient when an action may already have occurred. The harness may need an idempotency key, a reconciliation read, a compensating action, or an operator-visible unknown-outcome state. Injection tests show whether those controls are connected to the actual execution path rather than merely present in configuration or documentation.
The third outcome is failure containment. A failed subagent, unavailable tool, or malformed response should not silently weaken permissions, bypass an approval gate, or cause unrelated branches to restart. Tests should verify the blast radius: which state can change, which credentials remain usable, which work is cancelled, and which downstream steps are prevented from running.
Finally, fault injection improves observability. A harness should record the selected action, the fault boundary, the last durable checkpoint, any attempted side effect, and the recovery decision. Without that chain, operators can see that a run failed but cannot determine whether resuming it is safe. Recovery that cannot be explained is difficult to trust and harder to improve.
Fault Injection Testing vs recovery testing
The distinction changes how cases are designed. Recovery testing usually begins with a failed or unavailable system and asks whether service can be restored. Fault injection testing chooses the failure boundary and timing so the harness can be evaluated under a specific partial-execution condition.
| Concern | Fault injection testing | Recovery testing |
|---|---|---|
| Starting point | A running workflow with a deliberately introduced fault | A system already placed in a failed or degraded condition |
| Primary question | Does this boundary preserve invariants when execution is interrupted here? | Can the system return to an acceptable operating state? |
| Typical evidence | Journal entries, checkpoints, side-effect records, retry decisions | Restored availability, reconstructed state, completed recovery procedure |
| Main design value | Exposes unsafe transition boundaries and ambiguous outcomes | Validates the broader restoration path |
Both are necessary for durable agent operation. Recovery tests can pass while hiding a duplicated action or an incorrectly committed step. Fault injection narrows the test until that transition can be judged directly.
The Rifty take
We optimize for explicit recovery semantics at every boundary that can mutate durable or external state. We accept additional state and test complexity when it makes an interrupted run legible and safely resumable. If the harness cannot prove whether an effect occurred, it should preserve that uncertainty rather than convert it into a confident retry or success.
Implementation checks
- Define the invariant, expected recovery action, and observable evidence before injecting the fault.
- Test immediately before and after state commits, tool calls, approvals, handoffs, and external writes.
- Verify that retries cannot duplicate non-idempotent effects.
- Represent unknown outcomes explicitly when acknowledgement and execution can diverge.
- Confirm that checkpoints contain enough information to resume without replaying unrelated work.
- Check that compensation is scoped to the exact committed effect, not the entire workflow.
- Assert that failed branches cannot bypass permission or approval boundaries during recovery.
- Preserve the fault location, attempted action, recovery decision, and final state in the execution journal.
- Run adjacent-boundary cases because moving an interruption by one step can change the correct recovery decision.
- Treat a completed run with missing evidence, duplicated effects, or silently discarded work as a failed test.
Frequently asked questions
Where should fault injection points be placed in an agent workflow?
Place injection points around boundaries where authority, durable state, or external effects change. High-value locations include tool calls, state commits, approval decisions, agent handoffs, checkpoint writes, and acknowledgements. Test both sides of each boundary because interruption before execution requires a different response from interruption after an unconfirmed effect.
How is fault injection testing different from simply retrying failed runs?
Fault injection testing evaluates whether retrying is safe; it does not assume that retry is the correct response. A failed call may have completed its external effect without returning acknowledgement. The harness must use idempotency, reconciliation, compensation, or an explicit unknown-outcome state before deciding how execution should continue.
What should a fault injection test assert?
Assert durable invariants rather than only final completion. Check committed state, attempted and confirmed side effects, retry counts, permission boundaries, journal records, recovery decisions, and downstream cancellation. The test should also prove that an operator can distinguish safe resumption from cases requiring reconciliation or compensation.
Can fault injection testing cover nondeterministic agent behavior?
Yes, but the assertions should target harness guarantees rather than exact model text. Hold the injected boundary and control policy constant, then verify allowed state transitions, permissions, side effects, and recovery outcomes. Variable reasoning is acceptable when every resulting trajectory remains inside the same tested control and safety envelope.