What Retrieval-Augmented Generation Actually Does

What RAG is

Most conversations about AI focus on what the model says. The engineering problem that determines whether the answer is grounded or fabricated is what the model reads before it says anything. That's where RAG lives.

Retrieval-Augmented Generation is a pattern that fetches relevant documents at query time and feeds them to a language model so its response is grounded in specific source material. The model doesn't learn anything new. It reads before it answers, the way you'd pull up a KB article before responding to a customer's question. RAG exists because language models are fluent, confident, and perfectly willing to fabricate. Giving them source material to cite is the primary production mechanism for keeping answers tethered to reality.

• RAG defined: A pattern that retrieves relevant documents at query time and feeds them to a language model, grounding the model's response in specific source material rather than its training data alone.

How the loop works

Four steps, running in sequence every time a user asks a question.

Chunk. Source documents get split into pieces. A typical chunk runs 128 to 512 tokens, roughly two to three paragraphs of normal prose. The chunk is the unit of retrieval. Everything downstream operates on chunks, not whole documents. Keep that in mind. It matters.

Embed. Each chunk gets converted into a vector: a list of numbers that represents the chunk's meaning in a form that math can operate on. A modern embedding model produces roughly 1,500 to 3,000 numbers per chunk. The numbers encode relationships between concepts, which is why "federal compliance requirements" and "government regulatory obligations" end up with similar vectors despite sharing zero words. These vectors go into a vector database. That's your index.

Retrieve. The user's question gets embedded into the same vector space. The system finds the chunks whose vectors are closest to the query vector. In a basic setup, you grab the top 3 to 10 closest chunks. In production, you might pull the top 50, then rerank against the original question using a cross-encoder model and keep the best 5. Reranking is one of the most effective ways to improve retrieval quality. The tradeoff: a simple vector search takes milliseconds; a cross-encoder reranker can add seconds and cents per query.

Generate. The retrieved chunks get inserted into the prompt alongside the user's question. The language model reads them and writes a response. From the model's perspective, this is just reading and answering. It doesn't know a retrieval pipeline assembled the reading material. End-to-end latency for the full loop runs 1 to 3 seconds in production.

That's the loop. Chunk, embed, retrieve, generate. Every RAG system runs some version of it, from a startup chatbot to the Pentagon's GenAI.mil platform, which explicitly uses RAG to ground Gemini's responses for 3 million DoD personnel.

• The RAG loop: Chunk documents into pieces, embed each piece as a numerical vector, retrieve the vectors closest to the user's query, feed those chunks to the language model, generate a grounded response.

Where naive RAG breaks

The loop looks clean on a whiteboard. In production, three failure modes eat it alive.

Chunking severs meaning. A fixed-size chunker doesn't know where ideas begin and end. It splits on token count, not on semantic boundaries. A policy document that says "Contractors must complete security training within 30 days of onboarding" can get split so that one chunk says contractors need training and the next chunk says something about 30 days and CISO approval. Neither chunk contains the complete requirement. Research confirms that optimal chunk size is query-dependent, and "getting it right" means different things for different queries against the same corpus.

Similarity search returns the wrong "close enough." Vector similarity measures conceptual proximity, which turns out to be a weaker guarantee than actual relevance. A query about "access revocation procedures" might retrieve chunks about "access provisioning procedures" because the vectors are neighbors in embedding space. They share the concept of access management. They answer completely different questions. This is the embedding similarity miss, and it's particularly vicious because the system has no signal that it happened. The retrieved chunks look relevant to the model. The model generates a confident, well-structured, wrong answer. Nobody in the pipeline raises a hand.

Single-shot retrieval has no error recovery. Naive RAG retrieves once and generates. If the retrieval step missed the right chunks, there's no second pass, no mechanism to say "that doesn't look right, let me search again with different terms." One practitioner blog estimates a ~40% miss rate in production, though no methodology is published for that number and it should be treated as directional, not precise. Even if the real figure is half that, it names a real problem.

Anthropic's research on contextual retrieval, which prepends explanatory context to each chunk before embedding, reduced retrieval failure rates by 35 to 49% depending on configuration. That improvement tells you the baseline failure rate was high enough to justify the engineering investment.

• Naive RAG failures: Fixed-size chunking severs meaning at arbitrary boundaries, embedding similarity returns conceptually adjacent but wrong results, and single-shot retrieval has no mechanism to detect or recover from a miss.

Agentic retrieval and the retry loop

Agentic retrieval adds one capability that naive RAG lacks: it can look at what it found and decide to search again.

An orchestrating agent examines initial results, evaluates whether they actually answer the question, and can reformulate the query, search a different index, or request additional context before the language model ever sees the chunks. Mechanically: an analyst asks about access revocation procedures for departing contractors. The first retrieval pass returns chunks about access provisioning, onboarding workflows, contractor badge issuance. The agent evaluates: these are about granting access, and the question was about removing it. It strips the broad "access management" framing, reformulates the query around "revocation" and "offboarding" specifically, and retries. The second pass returns the deprovisioning policy, the 24-hour revocation SLA, the ISSO notification requirement. Those chunks go to the model.

The cost is latency and compute. Each retry is another embedding, another search, another evaluation pass. The benefit is something resembling self-correction, at least for the retrieval step. The generation step still has no ground truth to check against. That's a separate problem entirely.

• Agentic retrieval: An orchestrating agent evaluates retrieval results and can reformulate queries, search additional indexes, or retry, adding error recovery that naive single-shot RAG lacks.

The filesystem-access shift

A newer pattern is starting to surface in architecture conversations, and it abandons the vector store entirely. Tools like Claude Code don't use vector stores at all. The agent reads source files directly from the filesystem at runtime.

An arXiv analysis of Claude Code describes the mechanism precisely:

Every piece of context the agent sees is a plain-text file a human can read, edit, and version-control. No opaque retrieval layer deciding what's relevant. The design principle is explicit: stored context should be inspectable and editable by the user. That's a deliberate trade against the flexibility of embedding-based lookup.

Anthropic's own documentation draws the boundary: for knowledge bases under roughly 500 pages, they recommend loading material directly into the context window and skipping retrieval entirely. Above that, you need RAG or something like it.

	Vector-Store RAG	Direct File Access
Scale	Millions of documents	Bounded by context window
Inspectability	Low. Retrieval is opaque.	High. Files are readable.
Infrastructure	Indexing pipeline + vector DB	Files on disk
Staleness	Requires re-embedding on updates	No index to go stale
Authorization	Must be built explicitly	OS-level file permissions

That last row matters for identity. It also gets complicated fast: OS-level file permissions work for a developer on a local machine, which is Claude Code's primary context. In a multi-user federal environment where hundreds of analysts need differentiated access to the same document corpus, "OS-level permissions" is not an authorization architecture. The authorization story for filesystem-access patterns at enterprise scale is genuinely unsettled. Nobody has a clean answer yet.

• Filesystem-access pattern: Agents read source files directly at runtime instead of querying pre-indexed vector stores, trading scale for inspectability and eliminating the embedding pipeline. The authorization model for this pattern in enterprise environments remains an open question.

The permission logic that lived in SharePoint or Confluence doesn't travel with the chunk into the embedding index. If a user with limited access asks a question, naive RAG will retrieve and surface documents they were never supposed to see, wrapped in a confident answer that feels completely legitimate. AWS's security guidance states it plainly: LLMs should be considered untrusted entities because they do not implement authorization as part of a response. The production-safe architecture requires authorization filtering before the similarity search runs. The OWASP 2025 Top 10 for LLM Applications added Vector and Embedding Weaknesses as a new category specifically because of this failure mode.

When you'll need this

You're in a meeting with a federal CAIO or CISO. They're describing a knowledge management system that lets analysts query policy documents, internal memos, and case files using natural language. They call it "AI-powered search" or "intelligent document retrieval." What they're describing is RAG.

Three questions that matter in that room:

"How does the system handle documents with different access levels?" This is the question that stalls deals. If CUI and unclassified documents are embedded in the same vector store without authorization-aware retrieval, the system will surface controlled information to unauthorized users. Federal agencies may need paragraph-level classification rather than document-level, and no standard authorization framework handles that natively today. This is a genuine open problem. Saying so is more credible than pretending it's solved.

"Who owns the RAG index?" GSA's draft AI clause explicitly asserts government ownership of RAG indexes, prompt libraries, and integration scripts built under contract. If your buyer is building a RAG pipeline on agency data, the index itself is a data sovereignty question. Most sellers haven't internalized this yet.

"What happens when the source documents change?" In a vector-store RAG system, updated documents need to be re-chunked and re-embedded. Until they are, the system retrieves stale content and generates answers based on outdated policy. The filesystem-access pattern sidesteps this but only works at smaller scale. Ask the buyer what their re-indexing cadence is. If they don't have one, that's your opening.

You don't need to build RAG systems. But recognizing these as the right questions, and knowing that the buyer asking them is technically serious, is the difference between following the conversation and losing it.

• Practical context: In public sector conversations, RAG surfaces as "AI-powered search" over policy documents and case files. The deal-critical questions are about access control at the retrieval layer, data sovereignty over the index, and staleness when source documents change.

Next in this series: how fine-tuning differs from RAG, and why the choice between them is an architectural decision with real tradeoffs.