The model doesn't know what's true. It knows what comes next.
That distinction is the entire lesson. Everything else follows from it.
The Mechanism
Large language models are trained to predict the next token in a sequence — a token being roughly a word fragment — given everything that came before it. The training objective is to minimize prediction error across an enormous corpus of text. The result is a system that has learned, with considerable precision, what human-generated text looks like. It knows that certain words follow certain other words, that certain sentence structures appear in certain contexts, that certain facts tend to appear near certain claims.
Whether any of those facts are true was never part of the training objective. The model optimizes for plausible continuation, and plausible continuation is a statistical property of text, not a property of the world.
Hallucination is not a bug. It is not an edge case that will be patched in the next release. It is a structural consequence of the training objective. A model that produces fluent, coherent, confidently-stated text is doing exactly what it was designed to do. The problem is that fluency and accuracy are orthogonal. There is no internal mechanism to distinguish between a sentence generated from accurate training data and one generated by interpolating between two plausible-sounding patterns that happen to produce a false conclusion.
The confidence is the tell. When a model lacks sufficient signal — when it would hedge, in human terms — it doesn't. It generates the most statistically plausible continuation of the prompt, which is often a confident, well-structured, completely fabricated answer. Uncertainty in the training distribution produces fluency, not hedging.
What This Looks Like in Practice
A federal CAIO asks an AI assistant to summarize the relevant FedRAMP authorization requirements for a new SaaS procurement. The model returns a clean, well-organized response: authorization boundary definitions, required control families, timelines for the authorization process. The prose is authoritative. The structure is correct. Some of the specific requirements cited are accurate. Some are confident interpolations of regulatory language that doesn't exist in the actual FedRAMP documentation.
The CAIO cannot tell which is which without checking the source. Neither can the seller in the room, unless they already know the answer.
Agencies are running pilots, watching models produce polished outputs, and then discovering — sometimes before deployment, sometimes after — that specific factual claims don't survive contact with the primary source. The buyer skepticism you're encountering is the appropriate response to a structural property they've already observed, not irrational caution about new technology.
Okta Concept Mapping
Hallucination most closely resembles a claims-based identity system where the IdP has no external anchor. The model issues assertions about the world the way an IdP issues attribute claims — with apparent authority, in a recognized format. Unlike an OIDC token whose signature is verifiable against a published key, the model's assertions carry no cryptographic or authoritative backing. Grounding adds the anchor: a retrieval system that ties the model's output to a checkable source, so the claim can be verified against something outside the model itself.
Where the analogy breaks: in IDAM, a bad claim is an adversarial condition — a compromised IdP, a forged token, an attacker manipulating attributes. In an LLM, confident wrong claims are the default output mode, not an attack. The incident response playbook for a compromised IdP is the wrong mental model for a hallucinating model. This is structural mitigation territory, not security operations.
Grounding as Mitigation
Grounding is the family of techniques that constrain model outputs to verifiable sources. The three you'll encounter most often:
Retrieval-augmented generation (RAG) pulls relevant documents from a controlled corpus at inference time and provides them to the model as context. The model generates its response based on that retrieved content rather than relying solely on training data. The output can be checked against the source documents.
Citations require the model to attribute specific claims to specific passages in a source. This doesn't prevent the model from misquoting or misrepresenting the source — it still can — but it creates a verification path. A human reviewer can check the citation.
Tool-use with verifiable outputs connects the model to external systems — databases, APIs, calculators — whose outputs are authoritative by construction. The model calls the tool; the tool returns a fact; the model incorporates it. The answer to "what is the current FedRAMP authorization status of this vendor" comes from the FedRAMP marketplace API, not from the model's training data.
None of these techniques change the underlying mechanism. Next-token prediction is still what's happening. What grounding does is constrain the generation space to content that can be checked, so the workflow has a verification path even if the model's output is wrong.
The Trust Posture
The practical takeaway isn't a blanket prohibition on using AI for anything consequential. That's not a useful position and it's not where the market is going. The posture is more specific: never delegate a consequential decision to an AI system without a verification path built into the workflow.
Drafting, summarizing, surfacing options, generating first-pass analysis — these are appropriate uses where the human reviewing the output is the verification path. Autonomous action on factual claims, regulatory interpretations, or policy determinations — these require grounding to a checkable source before the output leaves the AI layer.
When a buyer asks "how do we know it won't just make things up," the honest answer is: you build the workflow so that consequential outputs are grounded and verifiable before they drive a decision. An architecture requirement, not an apology.
The model is very good at what it does. What it does is not the same as knowing what's true.