Agentic RAG¶
Agentic RAG extends standard Retrieval-Augmented Generation by replacing the fixed one-shot retrieval pipeline with a dynamic loop driven by an AI agent. Where standard RAG retrieves once and generates once — a librarian who fetches one book and hands it over — Agentic RAG uses an agent as a researcher who decides what to retrieve, from where, whether the result is good enough, and whether to try again before generating a final answer.
The practical consequence is that Agentic RAG can handle question types that standard RAG cannot: multi-hop questions that require chaining retrievals, ambiguous queries that need rewriting before retrieval succeeds, questions whose answers live in multiple data sources (a vector store AND a SQL database AND a live web search), and cases where the first retrieval returns irrelevant chunks that would otherwise produce a hallucinated answer.
Why Standard RAG Falls Short¶
Standard RAG works as a straight pipeline:
- Embed the user question.
- Search a vector database for top-k chunks.
- Pass those chunks plus the question to the LLM.
- Return the generated answer.
There are no decisions, no retries, and no ability to route to different sources. This breaks on four common problems:
Multi-hop questions. "What was the Q4 revenue of the company that acquired Acme in 2023?" requires two sequential retrievals — first finding who acquired Acme, then finding that company's financials. A single retrieval over the original question fails because no document contains both facts in proximity.
Ambiguous queries. "What about the new policy?" — the retrieval system has no way to know which policy, and will fetch whatever matches the words best, producing an unreliable answer.
Multiple data sources. If the answer to a question requires a SQL database for structured numbers, a vector store for document context, and a web search for current events, standard RAG (which hits one source) cannot serve it.
Poor retrieval quality. Standard RAG cannot tell when the returned chunks are not actually relevant. It passes whatever it found to the LLM regardless. Agentic RAG evaluates retrieved content before deciding whether to generate or to retry with a better query.
The Agentic RAG Loop¶
The agent is in control at every step:
User Question
|
v
[AI Agent — the LLM as brain]
| plans: do I need to retrieve? Which tool? What query?
v
[Retrieval Tool]
| returns chunks
v
[AI Agent — evaluates: is this good enough?]
|
+-- No --> retrieve again with a refined query
+-- Yes --> generate the final answer
The agent maintains a scratchpad — a running log of every action taken and every result seen in the current loop. This memory lets the agent avoid repeating failed retrieval attempts and build on partial results. Importantly, the LLM does not call tools directly: it outputs the name of the tool and its arguments as text; the runtime around the LLM executes the actual tool call and feeds the result back.
Three Building Blocks¶
1. The Agent (the brain). An LLM that reads the question, plans the next step, recommends which tool to use, and evaluates whether the retrieved result is sufficient to answer. It does not answer directly from its weights; it orchestrates the retrieval process.
2. The Tools (the hands). The set of retrieval and search capabilities the agent can invoke: - Vector search: semantic similarity over document corpora - Keyword search: exact term matching - SQL query: structured data in relational databases - Web search: fresh, real-time information - Knowledge graph query: relationship traversal between entities
The agent picks the right tool for the question, just as a researcher chooses the right reference type.
3. The Loop (the workflow). The runtime that ties agent and tools together: sends the agent's recommended tool call, returns the result, and repeats until the agent signals completion. The runtime also maintains the scratchpad across iterations.
Common Patterns¶
ReAct-style RAG. The agent follows a "Think → Act → Observe" cycle: it reasons about what to do, takes a retrieval action, observes the result, and repeats. This is the most general pattern and the basis of most practical Agentic RAG systems.
Self-RAG. After retrieval, the agent grades the returned chunks on two dimensions: relevance (are these chunks about the question?) and support (do they actually back up a potential answer?). Chunks that fail the grading are skipped or down-weighted before the final generation step. Self-RAG is focused on output quality control.
Corrective RAG (CRAG). The agent specifies a fallback retrieval strategy when the primary source fails. If a vector search returns irrelevant chunks, the agent switches to a web search rather than failing. CRAG is focused on robustness to source failures.
Standard RAG vs. Agentic RAG¶
| Aspect | Standard RAG | Agentic RAG |
|---|---|---|
| Flow | Fixed pipeline | Dynamic agent-driven loop |
| Retrievals per question | One | Many, as needed |
| Tools available | Typically one (vector search) | Multiple (vector, SQL, web, graph) |
| Query rewriting | No | Yes |
| Quality check on retrieved chunks | No | Yes |
| Multi-hop questions | Cannot handle | Handles |
| Latency | Low | Higher (multiple LLM + tool calls) |
| Cost | Lower | Higher (more tokens per request) |
| Build complexity | Simple pipeline | Agent, tools, loop, stopping rules |
When to Use Agentic RAG¶
Use Agentic RAG when: - Questions require information from more than one retrieval step (multi-hop). - Data lives in heterogeneous sources that cannot be unified into a single vector index. - Questions are ambiguous and benefit from query rewriting or clarification. - Retrieval quality is more important than latency — research, legal, medical, and compliance use cases. - The system needs to decide dynamically whether retrieval is even necessary.
Use Standard RAG when: - Questions are direct and single-step. - All relevant data is in one homogeneous index. - Latency and cost are primary constraints.
Relationship to GraphRAG¶
GraphRAG (covered in graphrag) is a specific architectural variant that augments RAG with an LLM-constructed knowledge graph to improve community-level and relationship-aware retrieval. Agentic RAG is an orthogonal concept: it describes the control flow (an agent driving retrieval), whereas GraphRAG describes the data structure (a graph augmenting a vector index). A system can be both: an agent that decides when to query the knowledge graph vs. the vector index vs. a web search is simultaneously Agentic and Graph-augmented RAG.
Limitations¶
Latency. Each loop iteration involves one or more LLM calls plus tool execution. Complex queries may require 4–8 iterations before the agent is satisfied, making total latency significantly higher than a single-shot retrieval.
Cost. More LLM calls plus longer context windows (the scratchpad grows with each iteration) mean higher API costs per query.
Debugging complexity. The agent's decisions are not deterministic. The same question can take different retrieval paths on different runs, making systematic debugging harder than tracing a fixed pipeline.
Unbounded loops. Without stopping rules, a poorly prompted agent can loop indefinitely. Common stopping conditions: the LLM signals it has enough information, a confidence threshold is reached, or a maximum iteration count is enforced.
More moving parts. Agentic RAG requires designing tool schemas, evaluation prompts, stopping rules, and scratchpad format in addition to the retrieval infrastructure.
Changelog¶
2026-05-18 — Page created from https://outcomeschool.com/blog/agentic-rag (Type C/D blog, confidence 72, moderate-confidence). Single source — additional sources from peer-reviewed literature or vendor documentation would strengthen confidence. Related to existing graphrag.md page.