The word "prompt" is doing too much work in enterprise AI conversations. When a CAIO says "we need to govern our prompts," they might mean they want to control what employees type into an AI tool. Or they might mean they want to control the instructions that shape the model's behavior before any employee touches it. These are different problems with different solutions, and conflating them is how AI governance initiatives end up governing the wrong thing.
The three turns in every model conversation explain why.
The Three Turns
Every interaction with a large language model follows the same basic structure. Before the model generates a response, it receives a sequence of text organized into distinct roles.
The system prompt comes first. It's a set of instructions written by whoever built or deployed the AI application — a product team, an agency IT shop, a vendor. The system prompt tells the model what it is, what it should and shouldn't do, what persona to adopt, and what constraints to observe. The user never sees it. In a federal agency's benefits assistance chatbot, the system prompt might specify that the model should only answer questions about the agency's programs, respond in plain language, and decline to provide legal advice. That instruction set is written once, at deployment, and applies to every conversation that follows.
The user prompt is what the person actually types. It arrives after the system prompt in the same text sequence, marked with a different role label. The user controls this turn entirely — which is why "governing prompts" in the sense of controlling user input is genuinely hard. You can filter, you can add guardrails, but you can't prevent a determined user from trying to push the model in directions the system prompt didn't anticipate.
The assistant turn is the model's response. It's generated based on everything that came before it — system prompt, user prompt, and any prior turns in the conversation. The model doesn't maintain a separate memory or state between exchanges; it reads the full accumulated text each time it generates output. The assistant turn is the response, but it's also part of the input for the next exchange.
Why Wording Is an Engineering Problem
The exact wording of a prompt produces measurably different outputs. Not stylistically different — substantively different. Researchers at the Allen Institute for AI found that rephrasing a factual question without changing its meaning shifted model accuracy by as much as 20 percentage points across standard benchmarks. The model pattern-matches against its training; different phrasings activate different learned associations, regardless of whether those phrasings mean the same thing to a human reader.
Prompt engineering is an engineering discipline. It has a feedback loop: write a prompt, measure the outputs across a range of inputs, identify failure modes, revise. Teams building enterprise AI features run systematic evaluations — sometimes hundreds of test cases — to validate that a system prompt produces consistent, appropriate behavior before it ships. The variance they're managing is real, and the stakes in a public sector deployment (wrong benefits information, inappropriate legal guidance, hallucinated policy details) make that variance consequential.
Okta Concept Mapping: System Prompts as Authorization Policy
The system prompt most resembles an authorization policy in an IDAM architecture — a set of rules that governs what the system will and won't do, applied before the user's request is processed. Both are set by administrators; both define the scope of permitted behavior; both are invisible to the end user during normal operation.
The break is specific: an authorization policy is enforced by infrastructure. The policy enforcement point doesn't negotiate. A system prompt is enforced by the model's learned tendencies, which means a sufficiently crafted user prompt can sometimes override it. The model follows the pattern an adversarial input activated — that's prompt injection in a sentence. There's no PEP in the loop. The security boundary is probabilistic, not deterministic, and that's a fundamentally different trust model than the one your buyers think they're buying.
What This Looks Like in a Procurement Conversation
When a federal agency is evaluating an AI product, the system prompt is often the thing nobody asks about — and the thing that matters most. A vendor can demo a model that seems perfectly calibrated to the agency's needs. What the demo doesn't show is what the system prompt says, who can change it, how changes are logged, and what happens when a user's input conflicts with the system prompt's instructions.
Ask to see the system prompt. Ask who owns it. The model is infrastructure. The system prompt is policy. And policy governance is a conversation agencies already know how to have.
When a CAIO asks how the agency will govern its AI tools, the answer starts there — before the user types a single word.