# Agentic RAG: architecture, and when it actually pays off > An opinionated architecture explainer on agentic RAG: retrieval-as-a-tool, query planning, self-correction loops, the latency and cost tax, and when naive RAG is still the right answer. **HTML version:** https://www.paiteq.com/blog/agentic-rag/ **Published:** 2026-06-13T03:54:19.132Z **Author:** Navin Sharma, Founder · AI Engineering Lead **Reading time:** ~13 min --- Search for *agentic rag* and you get a wall of vendor explainers that all describe the same four behaviors (the system rewrites your query, picks which source to search, checks whether the results are any good, and loops if they aren't), and then every one of them resolves to "...and that's why our platform does this." The most useful sentence in the whole search result isn't on any vendor page. It's in a Reddit thread titled "Agentic RAG is mostly hype," and it reads: most people don't have a RAG problem, they have a garbage-in, garbage-out problem. That's the tension this piece is about. Agentic RAG is a genuinely different architecture, and on the right workload it's worth the cost. On the wrong one it's an expensive way to get the same answer slower. We build retrieval systems, we've added the agentic layer, and we've ripped it back out when it turned out to be solving a problem the team didn't actually have. Here's how the architecture works, what each loop buys you, what it costs, and where the line sits. ## Agentic RAG in one paragraph, and the question it's actually answering Agentic RAG is retrieval-augmented generation where the model decides whether to retrieve, what to retrieve, and whether what it got back is good enough, instead of retrieving once, unconditionally, before it answers. Naive RAG is a fixed pipeline: embed the query, search the vector store, stuff the top chunks into the prompt, generate. Agentic RAG wraps that pipeline in a reasoning loop and turns retrieval into a tool the model can choose to call, skip, repeat, or reroute. That single change is the whole idea. Everything else (query rewriting, source routing, self-grading, multi-hop research) falls out of giving the model judgment about its own retrieval. The question it's really answering isn't "how do I make RAG smarter." It's "my single-pass retrieval is hitting a ceiling on certain queries, can the model dig itself out?" If you don't have that ceiling yet, you don't have an agentic RAG problem. You have a retrieval-quality problem, and there are cheaper places to spend your latency budget than an LLM loop. ## Naive RAG vs agentic RAG: what actually changes The clearest way to see the difference is the control flow. Naive RAG is a straight line with no branches: the same five steps run for every query, whether the query is "what's our refund policy" or "compare the indemnity clauses across our last three vendor contracts." Agentic RAG vs RAG is a difference of control structure, not of components, since both use an embedding model and a vector store. One runs the pipeline once; the other runs it under a controller that can branch and repeat. That "fails silently" line in the left column is the real argument for agentic RAG. A naive pipeline retrieves whatever the embedding search returns and answers from it, even when the retrieved chunks are irrelevant. It has no mechanism to notice. The agentic version can at least ask the question (is this enough to answer?) before it commits. The cost of asking is more LLM calls, and that cost is the entire trade-off. ## Retrieval as a tool: the one architectural move that makes RAG agentic Here's the move, concretely. In naive RAG, retrieval is wired into the request handler, so it always runs. In agentic RAG, retrieval is registered as a tool, the same way you'd register a calculator or a SQL function, and the model gets to decide whether calling it helps. This is the retrieval-as-a-tool pattern, and it's the canonical implementation LangGraph and LlamaIndex both build around. Below is the smallest version that shows the shift: the retriever is a tool the model can bind to and invoke, not a step the framework forces. Read the docstring; it's doing real work. It tells the model when to retrieve ("only when the question needs facts you don't already have") and when to rewrite first. That's the agentic part. The model isn't following a pipeline; it's making a decision and can make it again. Once retrieval is a tool, query planning, source routing, and the retry loop are all just the model choosing how to use that tool. You didn't add four features. You added one decision point and got the rest as consequences. ## The agentic RAG architecture, end to end Put the pieces together and the agentic rag architecture is a loop with a guard, not a line. A query comes in, the controller plans (rewrite, decide which source), retrieval runs as a tool call, a grader judges sufficiency, and the controller either generates the answer or loops back to retrieve again, bounded by a maximum-iteration count so it can't run forever. The fallback edge matters as much as the happy path: when the iteration budget is spent, the system should answer from what it has rather than spin. We treat that fallback as a hard requirement in every loop we ship, because it's the difference between a slow answer and no answer at all. ## Query planning: how an agent decides what to retrieve Query planning is the first thing the controller does, and it's where a lot of the recall gain actually comes from. A user types "what changed in the contract," which is a terrible embedding query, too vague to match anything specific. The planner rewrites it into something a vector store can act on, decides which source to hit (a contracts index, not the general docs), and on a complex question decomposes it into sub-queries that get retrieved separately. Routing across heterogeneous sources is where multi-agent setups earn their keep, and we cover how those controllers coordinate in our guide to [multi-agent orchestration patterns](/blog/multi-agent-orchestration-patterns/), because once you have more than one retrieval source, picking the right one per query is the same routing problem. Two patterns dominate the planning step. Query rewriting (sometimes via HyDE, where the model drafts a hypothetical answer and embeds that instead of the raw question) fixes the vague-input problem. Routing fixes the wrong-source problem: a financial question goes to the SQL tool, a policy question goes to the vector store. Both are cheap wins that don't need a full agentic loop; you can bolt query rewriting onto naive RAG and get most of the benefit for one extra LLM call. That's worth remembering when you're deciding how far down the agentic road to go. ## Self-correction loops: the part everyone shows and nobody bounds Every vendor diagram shows the self-correction loop (retrieve, grade, retry if the grade is low) and every vendor diagram stops there, as if the loop just converges. In production it doesn't always. The grader is itself an LLM call, and an LLM is perfectly capable of grading a wrong answer as correct (the self-graded hallucination) or grading a correct answer as insufficient and looping anyway. Without a hard guard, a self-correction loop on an unanswerable query will retry until it hits a timeout or a token limit, which is the worst possible failure: slow and expensive and still wrong. > [!NOTE] (rich block: callout) ## Single-agent, multi-agent, and graph: the three agentic RAG patterns The 2025 arXiv survey on agentic RAG (Singh et al., arXiv:2501.09136, the widely-cited reference taxonomy) sorts the field into three patterns, and the taxonomy is genuinely useful because each one has a different cost curve and a different failure mode. Pick the simplest pattern that covers your query distribution, because complexity you don't need is just latency and debugging surface you pay for. ## What it actually buys you: recall and accuracy, measured The honest answer is: it depends entirely on the query, and you can measure it before you commit. The gain concentrates on multi-hop questions, the ones where the answer requires chaining two or more retrieved facts. On those, a single-pass retrieval often can't surface both facts at once, and the iterative loop is what closes the gap. On single-hop, FAQ-style questions where the answer lives in one chunk, the agentic loop adds latency and token cost for a marginal accuracy delta. Run RAGAS or a similar eval over your real query distribution and you'll see which bucket dominates. On one multi-hop-heavy internal corpus we measured in 2026-Q1, the agentic loop reached 81% accuracy on RAGAS answer-correctness, up from a naive baseline near 70%; on a single-hop FAQ set the same loop barely moved the number and just burned tokens. Those figures are field-typical from our own runs, not a client deliverable, and your corpus is the only one whose numbers matter. That eval costs a handful of dollars in API spend on a mid-tier model over a few-thousand-document corpus, which means there's no excuse for shipping an agentic loop you never measured against the naive baseline (2026 pricing). ## The tax: latency, token cost, and the LLM-call budget Here's the part the AI Overview reduces to a bullet point that says "slower and more expensive." Put numbers on it. A naive RAG query is one retrieval plus one generation, a single LLM round-trip. An agentic query is a planning call, one or more retrieval-decision calls, a grading call, and a generation call. That's commonly three to eight LLM round-trips where naive RAG made one. Each round-trip is latency you can't parallelize away (the grade depends on the retrieve, the next retrieve depends on the grade) and tokens you pay for. On a frontier model the per-query cost difference is real, and your p95 latency stops being a number and becomes a distribution with a long tail from the queries that loop. Two levers cut the tax without abandoning the architecture. First, route the cheap calls to a cheap model. The grader and the planner don't need a frontier model; Claude Haiku 4.5 or GPT-5 mini grades sufficiency fine, and you reserve the frontier model for the final generation. Second, set a tight iteration cap and a latency budget so the long tail can't run away. If you can't bound the loop and can't fall back to single-pass retrieval when the budget's blown, you've built something that fails slow and expensive. Make it fail fast and cheap instead. ## Agentic RAG frameworks: LangGraph, LlamaIndex, CrewAI, AutoGen, Haystack There's no single best framework; there's a best fit for how much control you want over the loop versus how much you want the framework to handle. Here's our honest read on the five agentic rag frameworks we see most, including where each one is the wrong choice. DSPy and Semantic Kernel are worth a mention too: DSPy if you want to optimize the prompts in the loop programmatically, Semantic Kernel if you're in a .NET shop. ## When to use agentic RAG, and when naive RAG is the right answer The cleanest way to decide when to use agentic rag is to not decide globally; decide per query class. A lot of production systems route: FAQ-shaped queries take the cheap single-pass path, and only the queries that look multi-hop or ambiguous get escalated to the agentic loop. You get the recall gain where it matters and pay the tax only on the queries that earn it. Here's that router in two stacks, where a cheap classifier picks the path before any retrieval happens. Use agentic RAG when your query distribution is genuinely multi-hop, your sources are heterogeneous and need routing, or your naive pipeline is provably hitting a recall ceiling you've measured. Use naive RAG, and stop there, when queries are well-formed, the answer lives in one place, and latency is in a user-facing path where a 2-5× p95 hit is unacceptable. Most chatbots over a single clean corpus are the second case. Adding agents there is the textbook example of building a control system to solve a problem a reranker would've handled. ## Fix retrieval before you add agents: the hype the Reddit thread got right The Reddit thread that ranks fourth for this query, the one titled "Agentic RAG is mostly hype," is right about the thing that matters most, and it's worth taking seriously rather than dismissing as a hot take. The claim is that most people don't have a RAG problem; they have a garbage-in, garbage-out problem, and agentic RAG just adds fancy plumbing to a clogged pipe. In practice that's exactly what we see. An agentic loop wrapped around bad chunking and a weak embedding model spends more tokens arriving at the same wrong answer, slower. The loop can't retrieve a good chunk that was never indexed well in the first place. This is the same discipline as any infrastructure decision: measure the cheap fix before you adopt the expensive one. If you're not sure whether your team's retrieval foundation is solid enough to build on, an honest [AI readiness assessment](/blog/ai-readiness-assessment/) of the data and eval setup will tell you more than another framework will, because agentic RAG can't compensate for a corpus that was never indexed well. ## FAQ: agentic RAG questions, answered straight --- ## About Paiteq Enterprise AI engineering — production agents, RAG, LLM apps, automation, generative AI. Eval-first, senior-led, fixed-scope engagements. - **Site index for agents:** https://www.paiteq.com/llms.txt - **Full content for agents:** https://www.paiteq.com/llms-full.txt - **Book a call:** https://www.paiteq.com/contact/