Naive RAG
What it is: A single-pass system that embeds documents into vectors, retrieves the most similar chunks to a user's query, and feeds those chunks to an LLM to generate an answer.
What it does: It solves two problems a plain API call to GPT-4 or Claude cannot. First, knowledge cutoff: the model wasn't trained on your agency's internal policy documents, so it can't answer questions about them. Second, grounding: you want the model answering from your specific source material, not from whatever the internet looked like during training. The mechanism is a four-stage loop:
- Documents get split into chunks.
- Chunks get converted into numerical vectors (embeddings) and stored in a vector database.
- At query time, the system converts the user's question into a vector and finds the stored chunks whose vectors are closest (cosine similarity).
- Those chunks get stuffed into the prompt alongside the question. The model generates from that context.
The foundational architecture comes from Lewis et al. at Facebook AI Research (2020). Every major cloud provider and LLM framework ships some version of this loop. It's the default output of every "build a chatbot in 30 minutes" tutorial published since 2023.
What makes it distinct: The system has no mechanism to detect that it retrieved the wrong chunks. It just generates confidently from whatever it found. Fixed-size chunking splits documents at arbitrary token boundaries regardless of content. One embedding model and cosine similarity are the only retrieval signal. One shot at getting the right context, no fallback, no quality check. The spec calls this "retrieval-augmented generation," which is a generous framing for what's actually happening in the naive version: retrieval-and-hope.
Production-Grade RAG
What it is: A multi-stage retrieval pipeline that combines vector search with keyword search, reranks results, enforces document-level access controls, and includes evaluation mechanisms to detect when retrieval fails.
What it does: It solves the same two problems (knowledge cutoff, grounding) and survives conditions that break the naive version: large corpora, users who don't phrase questions the way the system expects, documents with overlapping content, and the requirement that users only see content they're authorized to see. The key architectural addition is hybrid search: running both a vector (dense) retrieval and a keyword (BM25/sparse) retrieval against the same query, then fusing the results. A cross-encoder reranker re-scores the merged candidates before they enter the LLM's context window. Authorization checks filter out documents the requesting user shouldn't access. Evaluation loops measure whether retrieved chunks actually answered the question.
No single vendor owns this pattern. It's an architectural consensus that emerged from production failures throughout 2024-2025. Weaviate, Elastic, and Qdrant ship hybrid search natively. Reranking models come from Cohere, open-source cross-encoders, and cloud-provider APIs. The authorization layer is where identity vendors enter the architecture.
What makes it distinct: It has mechanisms to catch what the naive version silently gets wrong. Hybrid search covers the complementary failure modes of vectors and keywords. Reranking applies deeper relevance judgment. Access controls prevent the retrieval layer from becoming an unauthorized data surface. All of it is plumbing. The quality of that plumbing determines whether the system is a liability or an asset.
Five Buyer Scenarios Where This Matters
I'm using scenario mapping here because the value for your next call lives in recognizing what the buyer is describing and knowing what to ask next. Each scenario surfaces a specific point where naive and production-grade RAG diverge. The through-line: RAG works for stable corpora and controlled queries. Every scenario below is a way that stability breaks.
"We embedded our policy docs and it works great."
Naive RAG's genuine sweet spot: a small, stable corpus with well-formed queries. It does work for this. The question worth asking is what happens at scale. When the corpus grows past a few hundred documents, when users ask about exact regulation numbers or policy identifiers, pure vector search starts to miss. Embeddings represent meaning, not characters. "Automobile" and "car" match beautifully. But policy number FCD-2025-0147 and FCD-2024-0147 look nearly identical to an embedding model. That one-digit difference is the difference between current guidance and last year's.
Hybrid search addresses this by running keyword retrieval in parallel, catching exact matches that vectors miss. Practitioner benchmarks consistently show hybrid retrieval delivering meaningful recall improvements over dense vectors alone. The consensus here is settled: hybrid search has decisively beaten pure-vector approaches for production workloads. The fusion method most teams reach for is Reciprocal Rank Fusion (RRF), which combines results by rank position rather than raw score, sidestepping the problem of normalizing scores across two very different retrieval methods.
"Our chatbot hallucinates on internal documents."
Almost certainly a retrieval failure. The generation models are good enough. Retrieval is where these systems break. Fixed-size chunking, where you split every 512 tokens regardless of content boundaries, routinely cuts sentences in half. The embedding model faithfully represents this mangled fragment. The vector search faithfully retrieves it. The LLM faithfully generates nonsense from it. Everyone did their job. The output is still wrong.
Production-grade systems address this with semantic chunking (splitting on content boundaries), hybrid retrieval, and reranking (a cross-encoder model re-scores candidates using deeper semantic judgment before they reach the LLM). Worth internalizing: a reranker can only reorder what was already retrieved. If the retriever missed a relevant document entirely, reranking can't conjure it. Hybrid search is what gives the reranker something worth working with.
"Everyone on the team uses the same knowledge base."
A shared vector index with no per-user access controls means every authenticated user can semantically query every document in the index. OWASP classified this as LLM08:2025: Vector and Embedding Weaknesses. The specific risk in multi-tenant environments: context leakage between users or queries. Semantic similarity can surface document content even when no explicit document is "returned" to the user. Someone asking a tangentially related question may receive a synthesized answer that originated in a document they have no permissions to see.
Production-grade RAG enforces document-level authorization at retrieval time, filtering candidates against the requesting user's permissions before chunks enter the context window. The naive version doesn't have this layer. It doesn't even have a place to put it.
A shared vector index with no per-user filtering is analogous to granting all authenticated users read access to every file in a file share. Your IDAM intuition applies: the authorization boundary needs to exist at the resource level, not just at the application perimeter. If a buyer says "everyone uses the same knowledge base," that's your signal to ask about document-level authorization at retrieval time.
"We're evaluating vector databases."
The buyer is making an infrastructure decision that matters less than they think. Practitioner consensus: the vector database choice accounts for roughly 5–10% of a RAG system's quality. Chunking strategy, embedding model selection, and retrieval pipeline design matter far more. The choice does matter for hybrid search support, though. Weaviate and Qdrant ship hybrid search natively. Pinecone added it but doesn't expose tuning parameters. pgvector requires you to build it yourself.
For public sector accounts: Elastic Cloud Hosted achieved FedRAMP High authorization on AWS GovCloud in April 2026, explicitly supporting RAG use cases. That's a concrete procurement data point worth having in your pocket.
"We need to keep the embeddings up to date."
One of four maintenance concerns, and the one buyers mention first. The others: chunking strategy drift (new document types that don't fit the original chunking logic), retrieval quality degradation (query patterns shifting as users learn the system), and access control drift (permissions changing in the source system while the vector index retains stale authorization metadata). Applied AI's practitioner guide reports naive RAG success rates of 10–40% in enterprise environments. The range is wide because these failure modes compound.
Production-grade RAG treats re-indexing as a continuous operational concern. This is where the 2023-pattern limitation bites hardest: RAG works for stable corpora precisely because stability means you can re-index infrequently. Fast-moving content, where documents change weekly or daily, turns re-indexing into an operational burden that many teams underestimate at prototype stage.
RAG inherits a version of the SCIM sync problem you already understand. When group membership changes or document permissions are revoked in the source system, the vector index doesn't automatically update. The lag window creates a period where a revoked user can still retrieve content through semantic queries. The question to surface: "When permissions change in your source system, how quickly does the vector index reflect that change?"
Okta FGA is documented as the enterprise pattern for retrieval-time authorization in RAG pipelines. The RAG system queries FGA during retrieval to determine whether the requesting user is permitted to access each candidate document fragment. Unauthorized content is excluded before it enters the model's context window. The authorization check happens per-chunk, at query time, not at authentication.
How to Say This in the Field
| Don't say | Do say | Why it matters |
|---|---|---|
| "RAG is just search plus AI." | "RAG is a retrieval pipeline that feeds your documents into the model's context at query time. The retrieval quality determines the answer quality." | Positions you as someone who understands the mechanism. |
| "You need better embeddings." | "How are you handling queries that involve exact identifiers, like regulation numbers or policy codes? Pure vector search tends to miss those." | Names a specific, testable failure mode the buyer can verify. |
| "Your chatbot hallucinates because the model is bad." | "That's usually a retrieval problem. The model generates from whatever context it receives." | Reframes from model selection to pipeline architecture, where you have more to say. |
| "You should use hybrid search." | "Are you running keyword retrieval alongside your vector search? Most production systems fuse both because the combination catches exact matches that embeddings miss." | Gives the buyer a concrete architectural question to evaluate their own pipeline. |
| "What vector database are you using?" | "Which vector databases are you evaluating, and do they support hybrid search natively? That matters more than raw vector performance." | Steers the infrastructure conversation toward the feature that affects retrieval quality. |
| "Is your RAG system secure?" | "When a user queries your RAG pipeline, are the retrieved chunks filtered by that user's authorization? Or does everyone hit the same index?" | Surfaces the access control gap without sounding like an audit. |
| "You need Okta for your RAG pipeline." | "Document-level authorization at the retrieval layer is where identity infrastructure connects to your AI architecture. That's a solvable problem." | Opens the door without pushing product. The buyer asks the next question. |
| "RAG is outdated." | "RAG works well for stable document corpora. It gets harder with fast-moving content that requires frequent re-indexing." | Shows you know the pattern's limits without dismissing a pattern the buyer already invested in. |
| "We can help with your AI strategy." | "Your retrieval layer has an authorization surface. Are you filtering retrieved documents by user permissions at query time?" | Connects your expertise to a specific architectural concern. |
| "That's an OWASP risk." | "OWASP LLM08 specifically covers vector and embedding weaknesses. Your security team will want to know how the pipeline handles document-level access." | Gives the buyer a named reference their security team will recognize. |
| "How's your AI project going?" | "Are you past the prototype stage with your RAG pipeline? The failure modes in production are different from what you see in testing." | Signals you know the demo-to-deployment gap without implying their project is broken. |
Federal Context
There is no dedicated federal guidance document governing RAG pipeline architecture. NIST's AI 100-2e2025 classifies RAG-specific attacks in its adversarial ML taxonomy, and FISMA/FedRAMP requirements apply to the underlying infrastructure, but agencies are deploying RAG without a RAG-specific framework. NIST's CAISI initiative explicitly excluded non-agentic RAG from its AI agent security scope. The applicable standards are general: NIST AI RMF, FISMA, FedRAMP, and OWASP LLM Top 10.
If a buyer asks what framework governs their RAG deployment, the honest answer is that it's governed by the same security and privacy requirements as any other system handling the same data classification. OMB M-25-21 preserves safeguards for privacy, civil rights, and security that would implicate retrieval pipelines exposing unauthorized content. M-25-22 restricts using non-public agency data to train commercial AI absent explicit agency consent, which is directly relevant when agency documents are embedded in a hosted vector database.
The absence of RAG-specific guidance is itself a useful thing to know. It means the buyer's security team is making interpretive decisions under existing authorities. Those decisions are where conversations happen.
Things to follow up on...
- PoisonedRAG research at USENIX: A 2025 study demonstrated that injecting just five malicious texts into a RAG knowledge base achieved a 90% success rate in manipulating LLM outputs, framing knowledge corruption as an optimization problem rather than a brute-force attack.
- NIST adversarial ML taxonomy: NIST AI 100-2e2025 formally classifies RAG-specific attack vectors including indirect prompt injection through retrieved content and training-data reconstruction as adversarial machine learning threats.
- Agentic RAG is next: A 2025 survey identifies three defining properties of agentic RAG — autonomous strategy selection, iterative multi-round retrieval, and interleaved tool use — that move retrieval from a static pipeline into a dynamic agent loop.
- AWS retrieval-time authorization pattern: AWS's security blog documents a concrete architecture where the application exchanges an IdP identity token with IAM Identity Center to retrieve temporary credentials scoped to the requesting user's document permissions before any vector search executes.