How it works
RAG splits a single "answer" into two steps: retrieve, then generate. Before the model produces anything, a retriever pulls the passages most likely to matter for the current task, and those passages are placed into the context window as grounding.
The loop usually runs like this:
- Index. Documents are chunked, embedded into vectors, and stored, often alongside a keyword or metadata index.
- Query. The user's request (or an agent-rewritten version of it) is turned into a query.
- Retrieve. The retriever returns the top candidates by semantic similarity, keyword match, or a hybrid of both, then optionally re-ranks them.
- Assemble. Selected passages are formatted into the prompt with their identifiers or source markers.
- Generate. The model answers from the supplied passages and, ideally, cites which chunk each claim came from.
The important property is that knowledge lives outside the weights. You change what the agent knows by changing the index, not by retraining. That makes the knowledge base auditable, updatable, and revocable. The quality ceiling of a RAG system is set less by the model than by retrieval: if the right passage never enters the context, no amount of generation skill recovers it.
Why it matters in an agent harness
In a harness, RAG is less about "chatting with your docs" and more about controlling what an agent is allowed to reason over. Three properties make it a control primitive:
- Legibility. When answers carry chunk IDs, you can trace a claim back to a source. That turns an opaque generation into something you can review, spot-check, and dispute. A harness that logs retrieved context makes agent behavior explainable after the fact.
- Freshness and revocability. Weights are frozen; an index is not. When a fact changes or a document must be pulled, you update the store and the next call reflects it. There is no retrain, and no stale claim baked into the model.
- Permission scoping. Retrieval is a natural place to enforce least privilege. The agent sees only the passages its identity is allowed to see, so access control moves into the retriever instead of relying on the model to "decline" politely.
RAG also narrows a failure mode you actually care about: fabrication on lookup-able facts. It does not make an agent honest, but it removes the excuse of not having the data. What remains is a bounded, inspectable question: did retrieval surface the right passage, and did the model use it?
RAG vs fine-tuning
Both inject knowledge, but they put it in different places, and that changes how you operate the system.
| Dimension | RAG | Fine-tuning |
|---|---|---|
| Where knowledge lives | External index | Model weights |
| Update cost | Re-index (cheap, fast) | Retrain (slow, costly) |
| Revoke a fact | Delete from store | Hard; requires retrain |
| Attribution | Per-chunk citations possible | None inherent |
| Best for | Volatile, private, factual data | Style, format, tacit behavior |
The rule of thumb: fine-tune to change how the model behaves; retrieve to change what it knows. Facts that move, or that you may need to pull, belong in retrieval where they stay auditable and reversible.
The Rifty take
We treat retrieval as a controlled boundary, not a knowledge dump. We optimize for precision over recall in the harness: a small set of correct, attributable passages beats a large context stuffed with plausible noise, because noise is where injected instructions and confident wrong answers hide. We accept the tradeoff of occasionally missing a passage in exchange for context we can trust, log, and defend. Retrieved text is untrusted evidence, never a command.
Common failure modes
- Retrieval miss. The answer exists but never enters context; the model fills the gap by guessing. Log retrieval hits and measure them separately from generation.
- Context poisoning. A retrieved passage carries embedded instructions the model follows. Treat retrieved content as data, isolate it, and never let it escalate the agent's permissions.
- Stale index. The store lags the source of truth, so the agent cites a fact that is no longer true. Version your index and track re-index freshness.
- Over-stuffing. Dumping many chunks dilutes the signal and raises cost without raising accuracy. Cap retrieved passages and re-rank for precision.
- Silent no-attribution. The model paraphrases beyond its sources without flagging it. Require citations and check that claims map to retrieved chunks.
- Unscoped access. The retriever returns documents the caller should not see. Enforce permissions in the retriever, not in the prompt.
Frequently asked questions
Does RAG stop an agent from hallucinating?
No. RAG removes the excuse of missing data on lookup-able facts, but the model can still ignore, misread, or over-extend retrieved passages. It narrows fabrication to a bounded, inspectable question: did retrieval surface the right source, and did the answer actually use it? Require citations and verify them.
When should I choose RAG over fine-tuning?
Use RAG when knowledge is volatile, private, or must stay revocable and attributable, since you update an index instead of retraining. Use fine-tuning to change how a model behaves, such as style or output format. The two compose: fine-tune behavior, retrieve facts. Facts that move belong in retrieval.
What is the main security risk RAG introduces?
Context poisoning and prompt injection. Retrieved passages can carry embedded instructions the model then follows, or return documents the caller should not see. Treat retrieved content as untrusted evidence, never as commands, isolate it in the prompt, and enforce access control inside the retriever rather than trusting the model to decline.
Why is retrieval quality more important than the model?
Because RAG's ceiling is set by what enters the context. If the correct passage is never retrieved, no amount of generation skill recovers it, so the answer is a guess. Measure retrieval hit rate separately from generation quality; most failures traced through a harness turn out to be retrieval misses, not model errors.
How much context should I retrieve per query?
Favor precision over volume. A small set of correct, attributable passages beats a large context stuffed with plausible noise, which raises cost and hides injected instructions and confident wrong answers. Cap the number of chunks, re-rank for relevance, and treat over-stuffing as a failure mode rather than a safety margin.