How it works
A model-based check inserts a model into a control point and asks it a bounded question: does this content or action violate a policy that only makes sense when you read the meaning? It runs as a distinct step, not as part of the agent's own reasoning, so its verdict is separable from the thing it judges.
A typical implementation is a short loop:
- Capture the artifact under review — a tool call argument, a draft answer, a plan, a retrieved document.
- Prompt a model or classifier with the policy and the artifact, asking for a structured verdict (pass, block, or escalate) plus a reason.
- Parse the verdict deterministically. The model supplies judgment; your code supplies the branch.
- Act on the verdict: allow, block, rewrite, or route to a human.
The check reads for things rules cannot enumerate: a request that is technically in-scope but adversarial, a summary that quietly invents a number, a response that leaks context it should not, an instruction smuggled in from a retrieved page. Because the judge is itself a stochastic model, its output is a probability of correctness, not a proof — which is exactly why you keep it narrow, give it one policy at a time, and treat its verdict as evidence rather than a verdict of last resort.
Why it matters in an agent harness
Rule-based checks are cheap, fast, and legible, but they only catch violations you can spell out in advance. Agents fail in the space between the rules. A model-based check covers that space: it is your semantic layer over a tool surface that a deterministic filter cannot fully describe.
In harness terms, it buys you three things. First, failure containment — a check on tool arguments can stop a destructive or out-of-scope action before it commits, tightening blast radius without hard-coding every bad case. Second, observability of intent — the check's reasons become logged evidence about why the harness blocked something, which is far more useful during triage than a silent regex match. Third, graduated control — a model-based check pairs naturally with an approval gate: high-confidence passes flow through, ambiguous cases escalate to a human, and only clear violations hard-block.
The cost is real. You are adding latency, spend, and a second stochastic component to a path that was previously deterministic. That means a model-based check belongs where semantic nuance actually decides the outcome — prompt-injection screening, claim verification, safety and scope enforcement — and not where a string comparison would do. Put deterministic checks first and cheapest; reserve the model for what only meaning can resolve.
Model-based check vs rule-based check
The distinction changes where you spend latency and how you reason about failure.
| Dimension | Rule-based check | Model-based check |
|---|---|---|
| Mechanism | Regex, schema, allowlist, assertion | LLM or classifier reading meaning |
| Catches | Enumerable, literal violations | Nuanced, contextual, adversarial ones |
| Determinism | Deterministic, reproducible | Stochastic; verdict is a probability |
| Latency / cost | Near-zero | Meaningful per call |
| Failure mode | Misses novel cases | False positives/negatives, can be gamed |
| Auditability | Trivial | Reason text helps, but not a proof |
The practical rule: rules for what you can name, models for what you can only recognize. Layer them — deterministic gate first, model check second — so the expensive judgment only runs on inputs that survived the cheap one.
The Rifty take
We treat a model-based check as evidence in a control loop, never as the loop's final authority. We optimize for a legible verdict — a structured pass/block/escalate plus a logged reason — over a clever one, because a check you cannot audit is a check you cannot trust when it fails. We accept the latency and spend only at control points where meaning genuinely decides the outcome, and we always seat a deterministic check underneath, so a single stochastic judge is never the last thing standing between an agent and an irreversible action.
Common failure modes
- Treating the verdict as ground truth. The judge is a model; it has false positives and negatives. Keep an escalation path and a deterministic backstop for irreversible actions.
- Judge and agent share a failure. If the same model class or prompt weakness affects both, a jailbreak that fools the agent may fool the check. Vary the model or narrow the check's job.
- Injectable judges. A check that reads untrusted content can itself be prompt-injected. Frame the artifact as data to inspect, not instructions to follow.
- Overloaded policies. One prompt enforcing ten rules degrades on all of them. Give each check a single, sharp question.
- Unparsed free text. If your code branches on prose instead of a structured field, the verdict is unreliable. Force a machine-readable output and parse it deterministically.
- Silent cost creep. A model check on a hot path multiplies latency and spend. Gate it behind cheaper checks and measure how often it actually fires.
Frequently asked questions
When should I use a model-based check instead of a rule?
Use a model-based check when the violation depends on meaning you cannot enumerate — intent, tone, scope, injected instructions, or fabricated claims. Use rules for anything literal: schemas, allowlists, formats. Layer them, running the cheap deterministic check first and reserving the model for what only semantic judgment can resolve.
Can a model-based check be trusted to block irreversible actions alone?
No. A model-based check is stochastic; its verdict is a probability, not a proof, so it produces false positives and negatives. For irreversible actions, seat a deterministic guardrail underneath it and route ambiguous verdicts to a human approval gate rather than letting a single model judgment be the last line of defense.
How is a model-based check different from LLM-as-judge?
They overlap in mechanism but differ in purpose. LLM-as-judge scores quality, usually offline for evaluation. A model-based check is an inline runtime guardrail that gates an action or output — pass, block, or escalate — on the live path. One measures; the other enforces containment as the agent runs.
Can a model-based check itself be attacked?
Yes. If the check reads untrusted content — a retrieved page or user input — that content can carry a prompt injection aimed at the judge, not just the agent. Frame the artifact as data to inspect rather than instructions to obey, and consider using a different model class than the agent so both do not share one weakness.
How do I keep model-based checks accurate over time?
Give each check one sharp policy, force a structured pass/block/escalate output, and parse it deterministically. Then treat the check as a component under evaluation: build a labeled set of edge cases, run regression evals when you change the prompt or model, and track false-positive and false-negative rates as first-class metrics.