How it works
Memory poisoning occurs when an agent retains information that should have been rejected, qualified, scoped, or allowed to expire. The bad entry can originate in user input, retrieved content, a tool result, another agent, or the model’s own unsupported summary. The defining mechanism is persistence: the information crosses a memory-write boundary and becomes available to later runs.
A typical poisoning path has four stages:
- The system observes an untrusted or incorrect statement.
- A memory writer stores the statement, often after compressing away its source and uncertainty.
- A later task retrieves the entry because it appears semantically relevant.
- The agent treats the retrieved entry as established context and changes a decision, plan, or tool call.
The stored content need not contain an explicit instruction. A false preference, obsolete constraint, invented fact, or misattributed decision can be enough. The harm depends on how often the memory is retrieved and what authority the harness gives it.
Memory controls therefore need to govern both writing and reading. A safe write path records provenance, scope, confidence, creation time, and the reason for retention. A safe read path filters entries by task, identity, freshness, and trust before adding them to model context. Deletion and correction must also propagate to derived summaries rather than changing only the original record.
Why it matters in an agent harness
Memory changes a local model error into durable system state. Without memory, an incorrect claim may disappear when the context window or run ends. With cross-session memory, the same claim can influence unrelated work, survive restarts, and be repeated by multiple agents. Persistence expands both the duration and the blast radius of the original failure.
The harness owns this risk. The model may propose something worth remembering, but the surrounding system decides whether the proposal is stored, how it is labeled, and when it can be retrieved. Treating every model-generated summary as authoritative state collapses that distinction.
A controllable harness separates at least three kinds of retained information: observed evidence, operator-approved decisions, and model-produced interpretations. They may all be searchable, but they should not carry the same authority. A retrieved interpretation should not silently override a current instruction or become evidence merely because it has appeared in several previous contexts.
Reversibility also depends on memory design. Operators need to locate the entry that influenced an action, identify its source, invalidate it, and determine which derived records or actions may also be affected. An append-only audit trail helps explain the sequence, while a current-state view tells the agent what remains valid. Neither is sufficient alone.
Evaluation should test trajectories rather than isolated answers. Seed a plausible but incorrect memory, run tasks that may retrieve it, and observe whether the harness rejects, qualifies, or acts on it. Useful checks include cross-user isolation, expiry behavior, conflicting evidence, correction propagation, and whether sensitive actions require fresh verification instead of remembered assertions.
Permissions limit the consequence. Memory may suggest an action, but it should not grant authority to perform it. Tool permissions, approval gates, and deterministic checks remain separate controls when a poisoned entry reaches the action path.
Memory Poisoning vs Prompt Injection
The distinction changes where controls belong. Prompt injection targets interpretation during a model call. Memory poisoning targets retained state and may continue after the triggering content is gone. One event can be both: an injected instruction may persuade an agent to store a durable rule or false fact.
| Dimension | Memory poisoning | Prompt injection |
|---|---|---|
| Primary target | Retained memory or derived state | The current model context and instruction hierarchy |
| Duration | Potentially cross-run or cross-session | Usually tied to contexts containing the injected content |
| Main control point | Memory writes, retrieval, provenance, expiry, and correction | Input isolation, instruction handling, tool policy, and output checks |
| Recovery | Invalidate entries and trace derived effects | Remove the input, rebuild context, and review resulting actions |
Blocking prompt injection does not guarantee clean memory. Incorrect tool output, stale operator decisions, and unsupported model summaries can poison memory without adversarial text. Conversely, strict memory controls do not stop an injected prompt from causing immediate harm within the current run.
The Rifty take
We treat durable memory as a privileged state transition, not a transcript convenience. We optimize for attributable, correctable memory even when that adds friction to writing and retrieval. When provenance or scope is unclear, the harness should lower the entry’s authority rather than letting repetition turn it into fact.
Common failure modes
- Automatic promotion: Every conversation summary or model inference is written into long-term memory without a retention policy.
- Provenance loss: Compression preserves a claim but drops who supplied it, when it was observed, and whether it was verified.
- Scope collapse: A preference or constraint belonging to one user, project, task, or environment is retrieved in another.
- Authority inflation: Retrieved text is treated like an operator decision merely because it came from the memory subsystem.
- Correction without propagation: The original record is fixed, but embeddings, summaries, caches, or downstream memories still contain the poisoned value.
- No negative evidence: The system stores supporting observations but cannot represent contradiction, revocation, or uncertainty.
- Unbounded freshness: Time-sensitive entries remain eligible for retrieval without expiry or revalidation.
- Weak attribution: Traces show that memory was retrieved but not which entry changed the plan or action.
- Memory-derived permission: A retained claim is allowed to expand credentials, approval status, or tool scope.
- Happy-path evaluation: Tests measure useful recall but never seed hostile, stale, conflicting, or cross-tenant entries.
Frequently asked questions
How can a harness prevent memory poisoning?
Put policy on both memory writes and retrieval. Record provenance, scope, confidence, and freshness; separate observations from approved decisions and model interpretations; require stronger verification for consequential use; and provide invalidation that reaches summaries, indexes, caches, and other derived representations.
Is memory poisoning always an intentional attack?
No. It can result from adversarial input, but ordinary system failures also cause it. Stale facts, incorrect tool results, model-generated summaries, identity mix-ups, and scope mistakes can all create poisoned memory when the harness retains them and later presents them with excessive authority.
Should agents be allowed to write their own long-term memories?
They can propose memories, but unrestricted durable writes create unnecessary risk. A harness should validate the proposed entry, label its origin and authority, apply retention rules, and reserve operator approval for high-impact decisions. The stricter the later use, the stricter the write path should be.
How do you test for memory poisoning?
Seed memories that are false, stale, conflicting, malicious, or assigned to the wrong scope. Then run realistic trajectories and inspect retrieval, reasoning, tool use, and correction behavior. The test passes only when the harness contains the bad state without hiding that it was encountered.
What should happen after poisoned memory is discovered?
Invalidate the entry, identify its source, and trace every derived summary, cache, decision, and side effect that may depend on it. Correcting one row is insufficient when copies remain retrievable. Recovery should preserve an audit record while preventing the invalid content from influencing new work.