What "Data Governance" Means Here
In the general enterprise sense, data governance is the set of policies, classifications, and controls that determine how data is handled across its lifecycle. You know this. With hosted AI inference, the enforcement architecture is different.
When a prompt leaves your perimeter and hits a provider's inference endpoint, it enters infrastructure you don't operate. The prompt is processed, a response is generated, and depending on the provider's configuration and your agreement terms, some record of that exchange may be logged, retained, and potentially used to improve the underlying model. Your data classification policy applies at the point of egress. After egress, in an environment where your policy has no technical reach, is where the governance problem lives.
The enforcement gap between the perimeter you control and the infrastructure you're trusting — that's the specific problem this article is about.
How the Levers Work
Enterprises have three operational levers for managing this gap.
Zero Data Retention (ZDR). Enterprise agreements from major providers — OpenAI, Anthropic, Azure OpenAI Service, AWS Bedrock, Google Vertex AI — typically include ZDR commitments at the enterprise tier. The commitment: the provider does not log prompt and completion content for training, and does not retain it beyond the time required to serve the response. Azure OpenAI's enterprise terms, for instance, specify that prompt data is not used to train or improve Microsoft or OpenAI models, and that content is not retained after the API call completes.
The nuance legal teams often miss: ZDR is a retention commitment, not a no-logging commitment. Providers maintain abuse detection and safety systems that may process prompt content in transit. The ZDR contract says the content isn't stored; it doesn't say the content isn't touched. That distinction matters when a CISO asks whether the provider could reconstruct a conversation under a government request.
Regional Endpoints. Most major providers offer regional inference endpoints — US, EU, specific sovereign cloud configurations for regulated sectors. The EU endpoint routes your traffic to data centers within the EU and, under the relevant DPA terms, commits to EU data residency. This is what satisfies the GDPR-adjacent question: "Is our EU user data leaving the EU?"
Worth asking before the legal team asks it: is the model itself EU-resident, or just the inference infrastructure? Some providers run inference in-region while model weights and training infrastructure remain US-based. Contractually, the data residency commitment covers the inference data. Technically, the model that processes it was built elsewhere. Whether that matters for your compliance posture is a legal question, not a technical one — but you need to be able to articulate the distinction.
Prompt-Boundary DLP. Gateway-layer DLP — whether through a dedicated AI security gateway or an extended enterprise DLP policy — inspects prompt content before it leaves the perimeter and can block, redact, or alert on content that matches defined patterns. PII, confidential project names, internal financial data: the gateway catches it before the ZDR question becomes relevant.
The limitation is pattern-matching at the boundary. DLP catches what it's configured to catch. It doesn't catch novel sensitive content, context-dependent sensitivity, or the aggregation problem — where no single prompt is sensitive but a sequence of prompts reveals something that is.
The Matrix. What legal teams end up maintaining is a providers-by-regions-by-retention-policies matrix: which providers are approved for which data classifications in which regions, under which agreement terms. This artifact doesn't exist in most organizations at the start of an AI deployment. It exists by the time the second incident question gets asked. Building it proactively, before the deployment review, is the difference between a smooth legal sign-off and a three-week hold.
Okta Concept Mapping
The closest IDAM analogue is data residency in federated identity — specifically, the question of where tokens are validated and where user attributes flow in a cross-domain federation. In that model, you can verify residency by inspecting the trust architecture: which IdP issued the assertion, which endpoints are in the metadata, where the SAML response was signed. The analogy holds for the concept of data residency as a governance property. It breaks on enforcement. In a federation you operate or audit, residency is a technical fact you can verify. With hosted AI inference, residency is a contractual commitment backed by the provider's architecture — which you can read about in their security documentation but cannot independently inspect. You are trusting the contract, not operating the control.
The Conversation You'll Have
A CISO asks: "If we get a subpoena for our AI-generated content, what's our exposure?" The honest answer has two parts.
First: under a ZDR agreement, the provider has committed not to retain prompt content, so there may be nothing to subpoena at the provider level. But the subpoena goes to the provider's legal team, not yours. Their response process, their legal hold procedures, their interpretation of what "retained" means — those are governed by their policies, not your contract. You don't control the response.
Second: your own logs are a different matter. If your gateway or observability layer is logging prompt content — for audit, for debugging, for attribution — that's in your perimeter and fully subpoenable. The ZDR commitment covers the provider's retention. It says nothing about yours.
Per-user attribution is where identity infrastructure enters the picture. If your AI gateway integrates with SSO and logs which authenticated user sent which prompt, you have a complete audit trail. That's operationally valuable. It's also a liability surface your legal team needs to know about before they sign off on the deployment.
For AI inference, the governance question is whether you understand where the technical controls end and the contractual commitments begin — and whether you can explain that distinction to the person across the table who's about to ask.
Most of the confident answers circulating in this space are contractual answers dressed up as technical ones. The buyer who's done this before will know the difference.