When you type a question into ChatGPT, you're not the first person to write something the model will read. There's already text in the context window before your message arrives. You just can't see it.
That gap — between what you typed and what the model actually received — is where enterprise AI deployments live. Understanding it is the difference between thinking "the model said something weird" and knowing exactly why it said it, who's responsible, and what lever to pull.
What a Prompt Actually Is
A prompt is text. That's it. The model receives text, processes it, and produces text in return. There's no separate channel for instructions, no privileged input that gets parsed differently at the architectural level. Everything the model knows about what you want, who you are, and what it's allowed to do arrives as a sequence of tokens in a single context window.
A token is roughly three to four characters of text, not a word, not a sentence, just a chunk the model uses as its unit of processing. A context window is the total amount of text the model can hold in a single call. Modern models handle anywhere from 32,000 to 200,000 tokens depending on the deployment, enough to hold a short novel, or a very long policy document, or a conversation that's been running for an hour. (These numbers move fast; treat them as order-of-magnitude anchors, not specs to memorize.)
Within that context window, the model receives a structured sequence of turns. Three of them. Each has a defined role. Each has a defined owner.
- Prompt: Text submitted to a language model that instructs or queries it. Everything the model receives is a prompt in the broad sense — but in practice, the word usually refers specifically to the user's input.
The Three Turns
System Prompt
The system prompt is written before the conversation starts. It's set by whoever built or deployed the application — not by the user, not by the model provider. The user never sees it.
Its job is to establish context, persona, and constraints. A system prompt might tell the model what role it's playing ("You are a procurement assistant for a federal civilian agency"), what it's allowed to discuss ("Answer questions about FAR Part 15 and agency acquisition policy"), what it should refuse ("Do not provide legal advice"), and how it should handle edge cases ("If a question falls outside federal acquisition, say: 'That's outside my scope.'").
In a typical enterprise deployment, the system prompt might run 500 to 2,000 tokens. In a 128,000-token context window, that's a small fraction, but it's the fraction that shapes everything else. Engineers write it. The organization deploying the application owns it. Users interact with its effects without knowing its contents.
User Prompt
This is what the user types. Submitted at runtime. Visible to the user, obviously, because they wrote it. In a well-designed enterprise deployment, the user prompt is the only part of the conversation the user controls directly.
The user prompt arrives after the system prompt in the context window. The model reads both together. It doesn't process them sequentially in the way a human might read a document top to bottom and then answer a question. It attends to the full context simultaneously. But position and labeling matter: the model has been trained to treat system prompt content differently from user prompt content, and it generally does.
Assistant Turn
The assistant turn is the model's response. In a single-turn exchange, this is the end of the conversation. In a multi-turn conversation, previous assistant turns become part of the context for subsequent calls — the model's earlier responses shape its later ones. That's a topic for a later article. For now: the assistant turn is output, and in enterprise deployments, it's the thing the user sees.
- Three turns: System prompt (operator-written, pre-set, user-invisible), user prompt (user-written, runtime, visible), assistant turn (model-generated, the output).
From ChatGPT to Enterprise Feature
An agency's IT shop is deploying an AI-assisted Q&A tool for contracting officers. The tool is supposed to help with FAR questions, reduce the load on the acquisition shop, and surface relevant policy without requiring officers to dig through 2,000 pages of regulation.
The contracting officer types: "Can I sole-source this contract?"
What the model actually receives is something like this:
[System] You are a procurement assistant for a federal civilian agency. You answer questions about FAR Part 15 and agency-specific acquisition policies. You do not provide legal advice. If a question falls outside federal acquisition, respond: "That's outside my scope — please contact your agency's general counsel." Always cite the relevant FAR section when possible.
[User] Can I sole-source this contract?
The contracting officer wrote one sentence. The model read a paragraph and a sentence. The agency's IT shop wrote the paragraph. The model provider wrote neither.
When the CAIO asks "what guardrails do we have on this model?" — the answer is: the system prompt. That's the organizational policy layer. It's where the agency's constraints live. It's what the IT shop is responsible for maintaining, testing, and updating when policy changes.
The feature is the system prompt. The consumer experience hides it. The enterprise deployment owns it.
And when a procurement question comes in that the system prompt didn't anticipate — say, a contracting officer asks about a topic that's adjacent to acquisition but clearly legal advice — the model's behavior depends entirely on how well the system prompt was written. Not on the model provider's guardrails. Not on the user's intent. On the text the IT shop put in that box.
- Enterprise deployment: The system prompt is the organizational policy layer — operator-owned, engineer-written, invisible to users. The user prompt is the request. The assistant turn is the output the user sees.
IDAM Concept Mapping
The analogy that works: The system prompt functions like a policy layer in an access management deployment. It's operator-owned, user-invisible, and it shapes what the system can do with a given request before the request is processed. The user submits a prompt; the system prompt determines the frame within which that prompt is handled. If you've ever written an Okta policy that says "users in this group can only access these applications during these hours," you understand the structural role the system prompt plays.
Where it breaks:
In IDAM, policy is enforced by a separate enforcement point. The policy decision point evaluates the request against the policy and returns a decision. The policy engine and the request processor are architecturally distinct. A policy that says "deny access to /admin" is a hard boundary — no request gets through, regardless of how the request is phrased.
The system prompt has no separate enforcement point. It's interpreted by the same model that interprets the user prompt. Both live in the same context window, as adjacent text. The model doesn't have a policy engine that evaluates the user prompt against the system prompt and returns a decision. It has a context window where everything coexists, and it produces output based on all of it together.
This means the system prompt establishes a soft boundary, not a hard one. The model will generally comply with system prompt instructions — it's been trained to treat them as authoritative — but a sufficiently crafted user prompt can sometimes override or circumvent them. This is why prompt injection is a real attack surface, and why it has no clean analog in traditional IDAM. There's no equivalent of injecting a malicious SAML assertion into a policy decision point and having the PDP comply because the assertion was phrased persuasively.
When a CAIO asks "how do we know the guardrails hold?" the honest answer is that you test them, iteratively, against adversarial inputs. That's not the answer IDAM people are used to giving. It's the right answer here.
Prompt Engineering: What the Term Actually Means
Prompt engineering is the practice of writing system prompts (and user prompts) that reliably produce the model behavior you intend. It's a real discipline.
The reason it's a discipline rather than just good writing: the model doesn't interpret meaning the way a human reads a policy document. It processes token sequences. "Do not discuss competitor products" and "only discuss our products" are not logically equivalent to the model, even though a human reader would treat them as nearly synonymous. The first instruction leaves open a space where the model might discuss competitors in a neutral or comparative way; the second forecloses it more completely. Small differences in phrasing produce measurably different outputs.
Prompt engineering involves specificity of instruction, ordering of constraints, explicit handling of edge cases, and systematic testing against inputs the system prompt's author didn't anticipate. It's closer to writing precise legal language than to programming, and less deterministic than either. A well-engineered system prompt doesn't guarantee behavior; it makes the desired behavior reliably more likely.
You don't need to be able to do this. You need to be able to recognize when a buyer says "we just need to set up the prompts" that they're describing a non-trivial engineering task, and to ask the right follow-up: Who owns that, how do you test it, and what happens when it breaks?
- Prompt engineering: The discipline of writing model instructions that reliably produce intended behavior. Real, non-trivial, and closer to policy drafting than programming, with less determinism than either.
What This Means in a Buyer Conversation
When a CAIO or CIO talks about "configuring the AI" or "setting guardrails on the model," they're almost always talking about the system prompt, whether they know it or not. The system prompt is the lever the operator has. It's also the lever the operator is responsible for.
The questions worth asking in that conversation:
- Who in your organization owns the system prompt? (If the answer is "the vendor," follow up on what that means for your agency's accountability.)
- How do you version-control it? (A system prompt is a policy document. It should be treated like one.)
- How do you test it against adversarial inputs? (This is the question most agencies haven't gotten to yet.)
- What's your process when the underlying model is updated? (Model updates can change how a system prompt behaves without anyone touching the system prompt itself.)
None of these questions require you to be an AI engineer. They require you to understand that the system prompt is organizational policy, that it's soft-enforced rather than hard-enforced, and that the gap between those two things is where the interesting problems live.
The model doesn't know your agency's intent. It knows your agency's text. Write it carefully.