How it works
A trust boundary classifies every input by authority before that input can influence execution. Instructions from the operator, harness policy, and approved configuration may control behavior. Retrieved documents, web pages, tool responses, uploaded files, and messages from other agents remain data, even when they contain text that looks like an instruction.
The boundary is enforced through a small control loop:
- Identify the source and provenance of each input.
- Assign the input an explicit trust class and permitted use.
- Keep untrusted content separate from policy and control messages during context assembly.
- Validate proposed actions against permissions, schemas, and run policy outside the model.
- Record which source influenced each consequential decision.
The model can still read untrusted material and reason about it. The restriction is that the material cannot grant itself authority. A retrieved page saying “ignore previous instructions” is evidence about the page, not a policy update. A tool response containing a shell command is output to inspect, not permission to execute it.
This boundary must survive transformations. Summarization, memory writes, agent handoffs, and context compaction can erase provenance unless the harness carries trust metadata forward. Once trusted and untrusted material has been flattened into an indistinguishable string, later controls have little reliable basis for separating them.
Why it matters in an agent harness
Agents turn text into decisions, and decisions into side effects. That makes text provenance an execution concern rather than a formatting detail. Without a trust boundary, any source placed in the context can compete with operator instructions using the same natural-language channel. The model may follow the wrong source even when the surrounding application correctly authenticated the user and restricted network access.
A sound boundary improves failure containment in several ways. It prevents retrieved content from redefining the task, blocks tool output from expanding its own permissions, and stops one agent from transferring more authority than it received. It also makes approval meaningful: the harness can present a proposed action derived from untrusted data without treating that proposal as authorization.
Trust classification should affect what the system permits, not merely what a prompt says. A warning such as “the following text is untrusted” is useful context, but it is not enforcement. Consequential operations still need deterministic checks around credentials, destinations, arguments, write scopes, and approval requirements. The model may propose an action; the harness decides whether that action is admissible.
Observability depends on the same separation. A useful trace should show where an instruction originated, how its trust class changed, and which policy allowed it to affect execution. This makes prompt injection, memory poisoning, and confused-deputy failures diagnosable. It also supports replay: operators can determine whether a bad outcome came from model judgment, corrupted context, excessive permissions, or a broken boundary.
The cost is additional structure. Inputs need provenance, context assembly becomes typed rather than purely textual, and handoffs must preserve authority metadata. That overhead is justified wherever an agent reads external material or can create durable side effects.
Trust Boundary vs Authorization Boundary
The two boundaries reinforce each other, but they answer different questions. Treating them as interchangeable leaves gaps in both context handling and execution control.
| Boundary | Primary question | Typical enforcement |
|---|---|---|
| Trust boundary | Which inputs may direct the agent, and which are only data? | Provenance labels, context separation, instruction precedence, memory admission rules |
| Authorization boundary | Which actions may this identity or run perform? | Scoped credentials, permission checks, approval gates, destination and argument validation |
An input can be trusted without authorizing a particular action. An operator instruction may be authentic but still request a write outside the run’s permitted scope. Conversely, an agent may possess a credential that technically allows a write while the request producing that write came from untrusted content. Safe execution requires both tests to pass independently.
The Rifty take
We treat trust as explicit metadata that must survive the full execution path, not as a tone the model is expected to infer. We accept more context plumbing in exchange for being able to explain why an input influenced an action. When provenance is missing at a consequential boundary, the system should narrow authority or fail closed.
Common failure modes
- Concatenating policy, retrieved content, and tool output into one undifferentiated prompt.
- Treating prompt wording as the only defense against hostile or misleading instructions.
- Losing source and trust metadata during summarization, compaction, memory writes, or handoffs.
- Allowing a tool response to trigger another tool merely because it contains actionable text.
- Letting an agent delegate authority it was never granted.
- Assuming authenticated input is automatically safe or authorized for every operation.
- Logging the final action without recording which inputs and policies caused it.
- Defaulting unknown provenance to trusted instead of constraining or rejecting it.
Frequently asked questions
Is labeling content as untrusted enough to enforce a trust boundary?
No. A label gives the model useful context, but enforcement belongs in the harness. The system should independently validate proposed actions, preserve input provenance, restrict available credentials, and apply approval policy. Otherwise, an untrusted instruction can still succeed whenever the model chooses to follow it.
Where should a trust boundary be placed in an agent system?
Place it wherever information crosses from a source with one authority level into a component that can make decisions or cause effects. Common crossings include retrieval, tool responses, uploaded files, agent handoffs, memory admission, and external messages. Each crossing should preserve provenance and narrow authority explicitly.
How does a trust boundary reduce prompt-injection risk?
It prevents injected text from being interpreted as authoritative merely because it appears in the model’s context. The harness marks external content as data, isolates it from control instructions, and checks resulting actions independently. This contains injection even when the model partially accepts the hostile instruction.
Can trusted instructions bypass permission checks?
No. Trust determines whether an input may direct the agent; permissions determine whether the requested action is allowed. A valid operator instruction can still exceed the run’s scope, target the wrong resource, or require approval. The harness should evaluate trust and authorization as separate conditions.
What should happen when an input’s provenance is unknown?
Treat unknown provenance as untrusted for consequential decisions. The system can still inspect, summarize, or compare the material, but it should not let that material change policy, grant permissions, enter durable memory unchecked, or trigger side effects. Recovery should require provenance restoration or explicit reclassification.