How it works
A deterministic function maps one input to exactly one output, every time. A non-deterministic process does not: the same input can yield different results across runs. In agentic systems, non-determinism enters through several doors at once.
- Sampling. A language model produces a probability distribution over next tokens, then draws from it. Temperature, top-p, and seed handling decide how much that draw varies. Even at temperature zero, floating-point accumulation, batching, and hardware differences can shift the argmax.
- Hidden state. Retrieved context, tool results, timestamps, and prior turns change what the model actually sees. Two runs with the "same" prompt often carry different context.
- Tool ordering and concurrency. When an agent decides which tool to call and in what order, small variations in one step cascade into different trajectories.
- Environmental drift. Model updates, changed APIs, and moving data mean yesterday's output is not guaranteed today.
The practical result: an agent's behavior is a distribution, not a value. You do not get one answer you can pin down; you get a spread you have to characterize and bound.
Why it matters in an agent harness
Deterministic software lets you reason about a single execution path. Non-determinism removes that guarantee, so the harness has to supply reliability that the model cannot. This is the core reason a harness exists.
Concretely, non-determinism forces four design commitments:
- Reproducibility must be engineered, not assumed. Log the prompt, context, model version, sampling parameters, and every tool result. Without that trace you cannot reproduce a failure or explain a decision after the fact.
- Controls sit outside the model. You cannot trust a sampled decision to respect a limit. Permission boundaries, approval gates, and tool budgets have to be enforced deterministically in the harness, around the agent, so a rare bad draw cannot cross a line that matters.
- Reversibility becomes mandatory. If any run can produce an unexpected action, you need checkpoints and rollback so an unwanted trajectory is recoverable rather than final. Reversibility is how you make a non-deterministic actor safe to run.
- Evaluation shifts from pass/fail to distributional. A single passing run proves little. You evaluate over many runs and reason about success rate, variance, and worst case, because the thing you are testing has a spread.
Non-determinism is not a defect to eliminate; the sampling that produces variance is also what produces useful, flexible behavior. The engineering job is to contain it: keep the flexible reasoning inside a boundary of deterministic checks.
Non-determinism vs determinism
The distinction matters because it tells you which parts of a system you are allowed to trust and which parts need a harness around them.
| Property | Deterministic | Non-deterministic |
|---|---|---|
| Same input | Same output, always | Output is a distribution |
| Reproducible | By default | Only with full trace capture |
| Verification | One run suffices | Many runs; reason over the spread |
| Right role | Guardrails, verifiers, side effects | Reasoning, planning, drafting |
The useful architecture is not "pick one." It is to route each responsibility to the layer that fits: let the model reason non-deterministically, and let deterministic code decide what is allowed to happen as a result.
The Rifty take
We treat non-determinism as the load the harness is built to carry, not a flaw to argue away. We optimize for containment and reversibility over raw autonomy: the model is free to explore, and a deterministic boundary decides what becomes real. We accept a modest cost in speed and freedom to keep every consequential action logged, bounded, and undoable. The line we hold is simple — reasoning may be non-deterministic; effects on the world may not be.
Common failure modes
- Trusting a single green run. One success hides a low failure rate. Evaluate over many runs and look at variance and worst case, not the best trajectory you saw.
- Assuming temperature zero means deterministic. It reduces variance but does not guarantee identical output across batches, hardware, or model versions.
- No captured trace. Without the exact prompt, context, model version, and sampling parameters, a bad run is unreproducible and therefore undiagnosable.
- Enforcing limits inside the prompt. "Never do X" in an instruction is a suggestion, not a boundary. Non-deterministic sampling will eventually violate it; enforce it in deterministic harness code.
- Irreversible side effects mid-trajectory. Letting an agent take unrecoverable actions before a checkpoint turns normal output variance into permanent damage.
- Silent environmental drift. A pinned prompt against an unpinned model or API quietly changes behavior over time; version and monitor both.
Frequently asked questions
Does temperature zero make a language model deterministic?
Not fully. Temperature zero selects the highest-probability token and sharply reduces variance, but floating-point accumulation, batching, hardware differences, and model updates can still change the output. Treat it as lower-variance, not reproducible. For real reproducibility, capture the full prompt, context, model version, and sampling parameters.
Why not just eliminate non-determinism from an agent?
Because the same sampling that creates variance also produces flexible, useful reasoning. Removing it would strip the capability you wanted. The engineering goal is containment, not elimination: let the model reason non-deterministically inside a deterministic boundary of checks, budgets, and reversible actions that decide what actually happens.
How do I test a non-deterministic agent?
Test the distribution, not one run. Execute the same task many times and reason over success rate, variance, and worst case rather than a single pass. Pair this with a deterministic verifier that checks each output against fixed criteria, so evaluation does not itself depend on a lucky draw.
Where should determinism live in an agent harness?
Put determinism in the layers that decide consequences: guardrails, permission boundaries, approval gates, verifiers, and side-effect execution. Keep non-determinism in reasoning, planning, and drafting. The boundary is the rule to hold — reasoning may vary, but what is allowed to affect the world must be enforced by deterministic code.
What is the biggest risk non-determinism introduces?
Irreversible actions taken on a rare bad trajectory. If an agent can perform unrecoverable side effects before a checkpoint, normal output variance becomes permanent damage. Mitigate with checkpoints, rollback, tool budgets, and enforced permission boundaries so any single run stays bounded and undoable.