How it works
Agentic RAG turns retrieval from a fixed preprocessing step into a bounded control loop. The agent does not receive one query result and immediately generate an answer. It inspects the task, identifies missing information, chooses an available retrieval path, and decides whether the returned material is sufficient.
A typical loop looks like this:
- Decompose the task into answerable information needs.
- Select a retrieval tool, index, filter, or source class for the current need.
- Form a query and retrieve candidate material.
- inspect relevance, authority, conflicts, and remaining gaps.
- Refine the query, change tools, or retrieve supporting material when needed.
- Stop when the evidence threshold, iteration budget, or terminal condition is reached.
- Assemble only the accepted evidence into generation context.
The model may choose the next action, but the harness should own the boundaries. It defines available tools, source permissions, query and token budgets, required attribution, stopping rules, and the state recorded after each step. That separation matters because retrieval quality and execution control are different problems. An agent can reason well about what to search next while still looping indefinitely, crossing a permission boundary, or treating a weak result as authoritative. Agentic RAG becomes an engineering pattern only when those behaviors are constrained and observable.
Why it matters in an agent harness
Single-pass retrieval works when the information need is predictable and one query reliably returns enough context. Production tasks are often less tidy. A request may contain several subquestions, use ambiguous language, depend on different repositories, or require the agent to reconcile conflicting material. Iterative retrieval lets the system respond to what it finds instead of pretending the first result set was complete.
That flexibility creates a larger control surface. Each additional retrieval decision can change cost, latency, context composition, and the set of data exposed to the model. A robust harness therefore treats every retrieval action as an attributable tool call. The trace should show the query, selected source, filters, returned identifiers, acceptance decision, and reason for continuing or stopping. Without that record, an operator can see a bad answer but cannot tell whether the failure came from planning, retrieval, evidence selection, context assembly, or generation.
Permissions also need to apply per source and per call. An agent allowed to search public documentation should not silently inherit access to private notes because both happen to share a retrieval interface. Source selection is an authorization decision as well as a relevance decision. Credentials, indexes, and metadata filters should remain scoped to the run's declared purpose.
Failure containment is equally important. Empty results, contradictory sources, unavailable retrievers, and exhausted budgets are valid states. They should produce explicit outcomes rather than trigger hidden source changes or an unsupported answer. When retrieval state is checkpointed, an interrupted run can resume without repeating expensive calls or losing the evidence decisions already made. Evaluation can then test the trajectory, not just the final prose: whether the agent chose appropriate sources, refined queries usefully, respected limits, and stopped for the right reason.
Agentic RAG vs retrieval-augmented generation
The distinction changes where control and evaluation must live.
| Design axis | Conventional RAG | Agentic RAG |
|---|---|---|
| Retrieval path | Predetermined pipeline or single query | Chosen and revised during execution |
| Number of passes | Usually fixed | Variable within an explicit budget |
| Main control point | Retrieval configuration | Retrieval configuration plus agent loop |
| Primary failure | Poor initial retrieval | Poor planning, looping, tool misuse, or weak retrieval |
| Evaluation target | Retrieved context and final answer | Context, answer, and retrieval trajectory |
Agentic RAG is not automatically better retrieval. It trades predictability and simplicity for adaptive search. If one known query against one controlled corpus consistently supplies the required context, a fixed RAG pipeline is easier to operate and evaluate. The agentic form earns its complexity when the system must discover what to retrieve, choose among meaningfully different sources, or revise its search based on evidence.
The Rifty take
We optimize for adaptive retrieval inside a deterministic operating envelope. The model may decide what information to seek next, but the harness owns permission scope, evidence requirements, budgets, attribution, and stop conditions. We accept that these limits may end a run with an explicit evidence gap because that outcome is safer and more legible than an unconstrained loop producing confident prose from weak material.
Implementation checks
- Define which retrieval tools and source classes each run may access.
- Record every query, filter, result identifier, and source transition in the execution trace.
- Set independent limits for retrieval calls, elapsed time, context size, and repeated queries.
- Require a visible outcome for empty results, conflicting evidence, provider failure, and budget exhaustion.
- Keep retrieved text untrusted; do not let embedded instructions alter permissions or loop policy.
- Separate evidence acceptance from answer generation so weak material can be rejected before context assembly.
- Make the stop condition inspectable and test both premature stopping and runaway retrieval.
- Evaluate retrieval trajectories against representative tasks, not only the fluency of final answers.
Frequently asked questions
When should I use Agentic RAG instead of a fixed RAG pipeline?
Use Agentic RAG when the agent must discover multiple information needs, choose among distinct sources, or revise searches after inspecting evidence. Prefer fixed RAG when a predictable query against a controlled corpus reliably supplies sufficient context, because it is simpler to constrain, observe, and evaluate.
What should stop an Agentic RAG retrieval loop?
The harness should stop retrieval when a defined evidence threshold is met, the agent reaches an iteration or cost limit, further queries repeat prior work, or an operational failure prevents reliable retrieval. Each terminal state should be recorded explicitly so completion cannot hide exhaustion, weak evidence, or provider failure.
How do I evaluate an Agentic RAG system?
Evaluate both the final answer and the retrieval trajectory. Check whether the agent decomposed the task correctly, chose permitted and relevant sources, refined queries productively, rejected weak evidence, preserved attribution, respected budgets, and stopped appropriately. Answer quality alone can conceal wasteful or unsafe retrieval behavior.
How does prompt injection affect Agentic RAG?
Retrieved material is untrusted input and may contain instructions aimed at changing the agent's behavior. The harness should prevent retrieved text from expanding permissions, selecting unauthorized tools, modifying stop rules, or overriding the task. Treat source content as evidence to inspect, never as authority over the control loop.