How it works
A stochastic process is a sequence of states where at least one transition draws from a probability distribution instead of following a fixed rule. Start it from the same state twice and you can get two different trajectories, because chance is part of the mechanism, not noise layered on afterward.
In an LLM harness the randomness enters at a few nameable points:
- Sampling. Token selection draws from the model's output distribution. Temperature, top-p, and top-k widen or narrow that draw; greedy decoding collapses it toward a single most-likely path.
- Seeding. A fixed seed can pin the draw so the same distribution yields the same sample. Without one, every call re-rolls.
- Execution order. Concurrent tool calls, batched requests, and low-level numeric reductions add their own small non-determinisms that interact with sampling.
The useful move is to treat randomness as a knob with a known location. You can turn it down (low temperature, fixed seed), fence it (sample once, then verify with a deterministic check), or lean into it (sample many candidates, then select the best). What you cannot safely do is assume randomness is absent because your last few runs happened to agree.
Why it matters in an agent harness
An agent loop is a stochastic process wrapped in control. Every model call is a draw, and draws compound: a slightly different plan on step two sends the whole trajectory somewhere else by step five. That has direct engineering consequences.
- Reproducibility. If you cannot re-run a failure, you cannot debug it. Recording the seed, sampling parameters, model version, and inputs is what turns a stochastic incident into a reproducible one.
- Evaluation. A single passing run tells you almost nothing about a stochastic system. Evals have to sample repeatedly and report a distribution — pass rate, variance, worst case — not a point estimate.
- Failure containment. Because any given step can draw a bad action, the harness should assume the draw will sometimes be wrong and keep the blast radius small: scoped permissions, cheap rollback, and deterministic verification after the stochastic step.
- Legibility. When outputs vary, operators need to see why. Logging the draw and its parameters keeps a varying system observable instead of mysterious.
The design shift is to stop treating variation as a defect to be stamped out and start treating it as a property to be bounded.
Stochastic process vs non-determinism
These get used interchangeably, and the conflation costs you. Non-determinism is the whole family of "same input, different output." Stochastic behavior is one member of that family: the part driven by an actual random draw. Other members — concurrency races, ordering effects, environment drift, hidden state — are non-deterministic without being random.
| Stochastic process | Non-determinism (general) | |
|---|---|---|
| Cause | A genuine random draw | Any source of run-to-run variation |
| Control | Seed, temperature, sampling params | Serialization, pinning, isolation, idempotency |
| Reproduce by | Fixing the seed and parameters | Fixing the seed and the environment/order |
| Mental model | A knob you can turn | A surface you have to map |
Why it changes a decision: if your variation is stochastic, a fixed seed may pin it. If it is ordering or environment drift, the seed does nothing and you are chasing the wrong fix. Name the source before you reach for a control.
The Rifty take
We do not try to make agents deterministic; we make the stochastic step bounded. We optimize for locating randomness precisely — which call, which parameters — so it can be reproduced, measured, and fenced behind a deterministic verifier and cheap rollback. The tradeoff we accept is that a harness runs a little slower and logs a little more, in exchange for variation that stays legible and reversible instead of surprising.
Common failure modes
- Assuming determinism from a small sample. Three matching runs are not proof; the fourth draws differently. Test with repeated sampling.
- Not recording the seed and sampling parameters. An unrecorded draw is an un-reproducible failure.
- Blaming the model for ordering bugs. Concurrency and batch-order effects masquerade as "flaky model" behavior; the seed won't fix them.
- Evaluating on a single pass. A stochastic system needs a pass-rate distribution, not one green checkmark.
- Fencing nothing after the draw. If a bad sample can take an irreversible action, temperature is not your problem — missing verification and rollback are.
- Turning temperature to zero and calling it deterministic. Greedy decoding narrows the draw but does not remove every non-deterministic source underneath it.
Frequently asked questions
Is a stochastic process the same as a non-deterministic one?
No. Stochastic behavior is one specific cause of non-determinism — an actual random draw. Non-determinism is the broader family that also includes concurrency races, execution ordering, and environment drift, none of which involve randomness. Naming which one you face decides whether a fixed seed will help.
Where does randomness enter an LLM agent harness?
Mainly at token sampling, where selection draws from the model's output distribution under temperature, top-p, and top-k. Seeding can pin that draw. Additional variation comes from concurrent tool calls, request batching, and numeric ordering, which are non-deterministic but not strictly random.
Does setting temperature to zero make an agent deterministic?
Not fully. Low or zero temperature narrows sampling toward the most-likely token, which reduces stochastic variation, but it does not remove non-deterministic sources beneath it — concurrency, batch ordering, and numeric reductions can still make runs differ. Pin the seed and the environment, not just temperature.
How should I evaluate a stochastic agent?
Sample repeatedly and report a distribution, not a single run. Track pass rate, variance, and worst-case behavior across many trials, and record the seed, sampling parameters, and model version so any failure is reproducible. One passing trajectory tells you almost nothing about a stochastic system.
How do I keep a stochastic agent safe if any step can draw a bad action?
Assume the draw will sometimes be wrong and bound its consequences. Fence the stochastic step behind a deterministic verifier, scope permissions tightly, and keep rollback cheap, so a bad sample produces a caught, reversible error instead of an irreversible one.