Enterprise AI governance is the set of controls an organization puts between its employees and the AI services its employees are already using. That's it. The architecture is not complicated in concept. In practice, most organizations don't have one — they have the beginning of one, assembled from whatever they bolted on after the last incident.
This chapter maps the architecture. Six layers: employee, SSO, gateway, policy engine, provider, observability. Each one exists for a reason. Understanding the reason is more useful than understanding the layer, because the reason tells you why the conversation is happening in your account right now, and what the CIO is actually trying to solve.
How Organizations Get Here
The typical sequence goes like this.
A team needs access to an AI service. Someone puts a corporate credit card on file, generates an API key, and shares it in a Slack channel. This takes about four minutes. Six months later, fourteen teams are billing to the same key, nobody knows which team is responsible for the $40,000 overage, and the key has been committed to a public GitHub repository twice. The security team finds out about the GitHub incident from a third-party scanner, not from internal monitoring.
That's the FinOps incident. It produces the first layer of governance: some form of spend attribution and key management.
Then HR deploys a resume screener. It works well. Someone notices, buried in the vendor's data processing agreement, that the model provider retains input data for thirty days for model improvement. HR has been sending candidate PII — names, addresses, employment history — to a model that's been keeping it. Legal gets involved. The screener gets turned off while someone figures out what the data residency requirements actually are.
That's the data governance incident. It produces the next layer: some form of policy control over which data can go where.
Then an auditor asks which employees have access to which AI services, and nobody has a clean answer. The access is happening through individual accounts, browser extensions, API keys in developer environments, and a handful of sanctioned SaaS tools that were approved without anyone thinking about the AI features embedded in them.
That's the visibility incident. It produces the layer that should have come first: identity-anchored access, so you can answer the auditor's question.
The pattern repeats. Each incident reveals a gap. Each gap produces a layer. By the time a CIO is ready to talk about "AI governance architecture," they're usually describing the scar tissue from three or four of these events, not a system they designed.
The Six Layers
The architecture that emerges from this pattern has a recognizable shape. Here's the block diagram, with just enough on each layer to orient before the downstream lessons fill in the detail.
Employee. The principal. Every AI interaction traces back to a human identity, or should. The employee layer is where access decisions originate and where accountability lands. When governance breaks down, it usually breaks down here first: the access was never tied to an identity the organization controlled.
SSO. The enforcement point for identity. If an AI service isn't in the SSO catalog, the organization can't enforce access policy, can't deprovision cleanly, and can't answer the auditor's question. SSO integration is also where per-user attribution becomes possible, which matters a great deal when you're trying to figure out who spent $40,000.
Gateway. The traffic layer. An AI gateway sits between the employee and the provider, inspecting and routing requests. This is where data loss prevention logic lives, where prompt injection detection happens, and where the organization gets its first real view of what's actually being sent to external models. Most organizations add this layer after the data governance incident, not before.
Policy engine. The decision layer. A gateway can inspect traffic; a policy engine decides what to do with it. Which users can access which models? Which data classifications can leave the network? What happens when a request matches a sensitive pattern? The policy engine is where governance intent becomes operational control. Most organizations underestimate this layer, because writing the policy turns out to be harder than deploying the infrastructure.
Provider. The AI service itself — the model, the API, the contractual relationship. The provider layer is where data residency commitments, retention policies, and model behavior live. The organization doesn't control this layer, but it negotiates with it, and the terms of that negotiation matter more than most buyers realize when they're clicking "accept" on a standard agreement.
Observability. The audit layer. What happened, when, by whom, to which model, with what data. Observability is the layer that makes every other layer accountable. It also gets added last, because it's easy to defer until someone asks a question you can't answer.
IDAM Bridge — In identity, organizations build out their IAM stack reactively too: Active Directory first, SSO when the helpdesk tickets pile up, MFA after a breach, PAM when the auditor asks about privileged accounts. The AI governance stack is assembling the same way. It diverges here: in IAM, every layer shares a common identity substrate — each layer knows what a user is. In AI governance, the layers often don't agree on what the principal is. The employee is clear. The AI agent making an API call on the employee's behalf is not. That gap is where the interesting governance problems live, and it's why your IAM intuition will carry you most of the way through these conversations, but not all of it.
What This Chapter Covers
The six lessons that follow take each layer in turn. Not as an architecture review — as a set of conversations you're likely to have in accounts where AI governance is becoming a procurement requirement.
Lesson 1 covers the shadow AI problem: why unsanctioned AI use is harder to scope than unsanctioned SaaS, and what the discovery conversation looks like. Lesson 2 covers AI gateways: what they actually do, where they fit in the stack, and the questions worth asking before a buyer commits to one. Lesson 3 covers SSO integration for AI services: the mechanics of per-user attribution and why it matters beyond FinOps. Lesson 4 covers spend governance: token budgets, chargeback models, and the conversation that follows the credit-card incident. Lesson 5 covers data residency: what providers actually commit to, what zero data retention means in practice, and where the gaps are. Lesson 6 covers observability: what a useful AI audit trail looks like and why "we have logs" is not the same answer.
None of these lessons require the full architecture to be in place before the conversation is useful. Most accounts you're walking into have two or three layers and a plan for the rest. The goal is to understand what's missing, why it's missing, and what incident will eventually make it a priority.