Buyers customizing AI models are choosing between three approaches: prompt engineering, retrieval-augmented generation (RAG), and fine-tuning. You'll hear all three in the same meeting, often mashed together under "we need to train our own model." The sentence that earns you the next five minutes of the conversation:
"Are you trying to change what the model knows, or how it behaves? Because those are different problems with different price tags."
Prompt Engineering
What it is: Writing instructions that shape a model's output without changing the model itself.
What it does: You provide context, constraints, examples, and a task description in the input. System prompts set persistent behavior: "respond only in bullet points," "you are a federal procurement analyst," "cite your sources." Few-shot examples show the model what good output looks like by including two or three completed examples in the prompt. The model doesn't learn anything between calls. It follows directions from scratch every time, like an extremely capable contractor who never remembers yesterday's standup.
Who's behind it / where it comes from: Every model provider supports it. OpenAI, Anthropic, Google, Meta's Llama, Mistral. If you can call the API, you can prompt-engineer. It's the default interface.
What makes it distinct: Zero infrastructure cost beyond the API call. No training data, no pipeline, no GPUs. You're renting the model's attention one conversation at a time. The tradeoff: your instructions consume tokens (the units models charge by), and a long system prompt on every call adds up at volume. But for most use cases, this is where you start and where you stay.
Prompt engineering works like configuring an authentication policy — you're setting rules that shape behavior at runtime, same IdP underneath. The policy changes what the system does. Where the analogy breaks: a misconfigured auth policy fails closed. A bad prompt fails open, generating something plausible but wrong. In a buyer conversation about risk tolerance, that's a meaningful difference.
Retrieval-Augmented Generation (RAG)
What it is: A pattern where the system retrieves relevant documents from an external source before the model generates a response.
What it does: Instead of hoping the model "knows" your agency's policy manual, you store the manual in a searchable database (typically a vector store, which indexes text by meaning rather than keywords). When a user asks a question, the system searches the database, pulls the most relevant chunks, and feeds them to the model alongside the question. The model generates an answer grounded in your actual documents. The knowledge lives outside the model. The model is the reasoning engine. The document store is the filing cabinet.
Who's behind it / where it comes from: RAG isn't a product. It's an architecture pattern, first described by Facebook AI Research in 2020. Every major cloud provider offers managed components: Azure AI Search, Amazon Kendra, Google Vertex AI Search. Open-source frameworks like LangChain and LlamaIndex wire the pieces together. The model provider doesn't matter. RAG works with any model that accepts context in its input.
What makes it distinct: The knowledge is always current because it lives in documents you control, updated on your schedule, independent of model training cycles. When the policy changes, you update the document store. No retraining. The tradeoff: retrieval quality is the ceiling. If the search returns the wrong chunks, the model generates a confident answer from the wrong source material. RAG doesn't fail by saying "I don't know." It fails by sounding right.
RAG works like an IdP querying a directory store at authentication time — the IdP doesn't memorize every user's attributes, it queries LDAP or a SCIM-provisioned directory at the moment it needs to make a decision. RAG does the same thing with documents. Where this breaks: a directory lookup returns structured data with a defined schema. RAG retrieval returns unstructured text ranked by semantic similarity. The confidence model is fundamentally different, and that difference matters when a buyer asks about accuracy guarantees.
Fine-Tuning
What it is: A training process that adjusts a model's internal weights using your data, changing its default behavior.
What it does: You feed the model hundreds or thousands of example input-output pairs. The training process nudges the model's weights so it produces outputs matching your examples' style, format, tone, or task-specific patterns. A fine-tuned model for medical notes defaults to clinical language. A fine-tuned model for legal briefs structures arguments without being told to. The behavior becomes the model's new baseline.
Who's behind it / where it comes from: OpenAI offers fine-tuning for GPT-4.1 variants. Google Vertex AI supports Gemini model tuning. Anthropic does not offer fine-tuning for current-generation Claude models. Open-weight models like Meta's Llama and Mistral can be fine-tuned on any compatible infrastructure. That last point matters: if your buyer's preferred model is Claude, fine-tuning isn't on the menu.
What makes it distinct: It changes the model's default behavior permanently (or until the next fine-tune). No system prompt needed to enforce the style. The tradeoff: it requires training data, compute time, and ongoing maintenance. And critically, fine-tuning changes behavior and style. Facts are a separate problem, and conflating the two derails more buyer conversations than any other misconception.
When Each Approach Pays Off
I'm organizing this by buyer scenario rather than a flat feature table because the choice between these three approaches depends entirely on which problem you're solving. Picking the wrong one wastes money and creates a specific, predictable failure. Each scenario below names the right tool, its sweet spot, and the failure mode that tells you it's time to move.
"We need the model to follow our agency's response format"
Right tool: Prompt engineering. Specify the format in the system prompt. Include two examples. Done. Cost: the tokens in your prompt, multiplied by every API call.
Sweet spot: Any behavioral constraint you can describe in plain language. Format, tone, length, structure, audience.
Failure mode: System prompts have a complexity ceiling. As instructions accumulate, they start competing with each other. A prompt that says "be concise" and also "always include three supporting examples with full citations" will produce inconsistent results as the model arbitrates between contradictory goals. At high volume, slight variations in user input cause the model to drift from the desired style. When you notice the model ignoring instructions it used to follow, or when your system prompt has grown past a page and a half of dense text, you've hit the rung's limit. That's when you look at fine-tuning to bake the behavior in, or RAG if the real problem is that you're stuffing reference material into the prompt.
"We need the model to answer questions using our internal policy documents"
Right tool: RAG. Store the documents in a vector database. Retrieve relevant sections at query time. The model reasons over your actual documents.
Sweet spot: Any use case where the model needs access to information that changes, that's specific to your organization, or that didn't exist when the model was trained.
Failure mode if you escalate: This is the misconception worth understanding cold. A buyer fine-tunes on policy documents expecting the model to "learn" them. Research from EMNLP 2024 (Gekhman et al.) found that models absorb new factual information introduced during fine-tuning significantly slower than behavioral patterns. Worse: as those new facts are eventually learned, they linearly increase the model's tendency to hallucinate on other topics. The mechanism matters. Fine-tuning teaches the model to produce answers that sound like your facts, whether or not those answers are accurate. A 2026 preprint (Kaplan et al., not yet peer-reviewed) extends this finding and frames it as factual forgetting from continual learning dynamics.
Six months after the fine-tune, the policies have changed but the model's weights haven't. You now have a model that confidently cites outdated policy. RAG avoids this because the knowledge lives in the document store, not the weights.
Failure mode if you don't escalate enough: You can paste documents into the context window, and for small document sets this works. But context windows have limits, and stuffing 200 pages of policy into every API call is expensive and degrades the model's precision on any individual passage. RAG scales where prompt-stuffing doesn't.
"We need the model to match our agency's voice and classification style, thousands of times a day"
Right tool: Fine-tuning. You have a well-defined behavioral pattern (tone, structure, classification taxonomy) that needs to be the default across high-volume production use. Fine-tuning bakes the behavior in, eliminating the per-call cost of a long system prompt.
Sweet spot: High-volume, consistent behavioral patterns where the style is stable and well-documented through existing examples.
This is where PEFT and LoRA enter the conversation. Parameter-Efficient Fine-Tuning (PEFT) is a family of techniques that fine-tune only a small fraction of the model's weights instead of all of them. LoRA (Low-Rank Adaptation) is the most widely used PEFT method. Instead of remodeling the entire house, you're swapping out the fixtures. LoRA typically modifies less than 1% of the model's parameters. Training costs drop by roughly an order of magnitude. Training runs finish in hours instead of days. And you can store multiple LoRA adapters (one per use case) on top of the same base model. For a buyer running multiple classification tasks, that means one base model with several lightweight behavioral overlays, rather than several fully trained models.
The frontier-model crossover point: When does fine-tuning a smaller open-weight model (say, 7B–13B parameters) beat calling a frontier model API with good prompts? The honest answer: it depends on volume, and published cost analyses disagree on where the line falls. The spread is wide. Comparisons that include only GPU rental costs put the crossover as low as a few million tokens per month. Comparisons that include the loaded cost of an inference engineer to keep the stack running push it much higher. One practitioner analysis puts the threshold around 8,000 interactions per day before self-hosted infrastructure costs less than managed APIs; others put it significantly higher when engineering overhead is fully loaded. (Source note: these thresholds come from practitioner infrastructure blogs like devtk.ai and similar cost-analysis sites. They're useful as directional signals but embed different assumptions about model size, GPU pricing, and staffing. No single number is authoritative here.)
The hidden variable is the engineer. A self-hosted inference stack needs someone to maintain it. That person's fully loaded cost often exceeds the API spend they're replacing, especially below enterprise scale. A team spending $8K/month on API calls cannot justify a $25K/month engineer to bring it in-house.
For most federal agencies exploring AI today, the API is cheaper. The exceptions are agencies with existing GPU infrastructure and dedicated ML engineering staff. This threshold is genuinely unsettled and shifts as API pricing drops (roughly 80% industry-wide from 2025 to 2026), so treat any specific number as directional, not definitive.
Failure mode if you don't escalate: At thousands of calls per day, a long system prompt on every request becomes a real cost line. And as noted above, prompt-based behavior is brittle at scale. Fine-tuning makes the behavior more consistent and cheaper per call.
"We need to train our own model"
Right approach: Decompose the requirement. When a federal buyer says this, they're typically bundling three distinct concerns that have different solutions.
Concern 1 — Data residency: Data can't leave agency infrastructure. That's an infrastructure question. Where the model runs solves it. A commercial model deployed on-premises or in a FedRAMP-authorized environment meets the residency requirement without custom training.
Concern 2 — Compliance constraints: FedRAMP, ITAR, and CMMC requirements limit which cloud-hosted models are actually available. As of early 2026, government cloud regions lag commercial availability: Azure Government offers GPT-4.1 and o3-mini, AWS GovCloud Bedrock offers Claude Sonnet 4.5 and Llama 3, and Google Cloud doesn't offer Vertex AI generative models under ITAR-scoped Assured Workloads. This model lag sometimes makes open-weight model deployment the practical choice. But deploying an open-weight model is still a long way from training one from scratch. (Source note: FedRAMP model availability from sweetspot.so, a vendor-positioned procurement tool. Useful for the availability matrix; verify against provider documentation for specific authorization scopes.)
Concern 3 — Domain-specific behavior: The agency needs the model to act like it understands their domain. This is the only concern that might justify fine-tuning. And even then, the escalation ladder applies: prompt engineering first, RAG if you need domain knowledge, fine-tuning only if the behavioral pattern needs to be baked in at high volume.
Failure mode: Treating all three concerns as one problem leads to the most expensive possible solution. No federal policy document reviewed, including NIST AI RMF 1.0 and OMB M-25-21/M-25-22, prescribes fine-tuning as a security or compliance requirement. "We need to train our own model" is a derived preference. No policy mandates it.
Your IDAM intuition helps here. You already know that where a system runs and what a system does are separate architectural decisions. An Okta tenant in GovCloud meets FedRAMP requirements because of where it's deployed and how it's operated, and Okta didn't rebuild the product for government. The same logic applies to AI models. Separating the deployment question from the customization question in a buyer conversation is the single most useful thing you can do.
How to Say This in the Field
| Don't say | Do say | Why it matters |
|---|---|---|
| "You don't need to fine-tune." | "What specific behavior are you trying to change? That tells us which approach fits." | Opens discovery instead of closing a door. |
| "Fine-tuning teaches the model your data." | "Fine-tuning changes how the model behaves — tone, format, style. For knowledge, you want RAG." | Corrects the most common misconception cleanly. |
| "RAG is better than fine-tuning." | "RAG handles knowledge. Fine-tuning handles behavior. They solve different problems." | Prevents a false binary that collapses in a technical conversation. |
| "Just use prompt engineering." | "Start with prompt engineering. If that hits a ceiling, you escalate. Most teams don't need to." | Frames the ladder without dismissing the buyer's ambition. |
| "Training your own model is too expensive." | "Self-hosting starts to make economic sense at high volume, but the engineer to run it often costs more than the API bill. Where's your volume?" | Turns a cost discussion into a discovery question. |
| "You can fine-tune any model." | "Fine-tuning availability depends on the provider. Claude doesn't offer it for current models. OpenAI and open-weight models do." | Prevents planning around a capability that doesn't exist. |
| "LoRA is a fine-tuning technique." | "LoRA lets you fine-tune less than 1% of the model's weights. Same behavioral result, fraction of the cost." | Makes PEFT tangible without a tutorial. |
| "You need a sovereign AI strategy." | "Let's separate where the model runs from how it behaves. Those have different solutions." | Decomposes the sovereign AI bundle into actionable questions. |
| "Fine-tuning is like training." | "Fine-tuning adjusts an existing model. Training from scratch builds a new one. Almost nobody needs to train from scratch." | Prevents scope creep into a multi-million-dollar project. |
| "The model will learn your policies." | "The model will learn to sound like your policies. For accurate policy answers, put the documents in a retrieval system." | Draws the line between style and knowledge precisely. |
The escalation ladder holds all of this together. Prompt engineering first. RAG when you need external knowledge. Fine-tuning when you need behavioral consistency at scale. Each step up costs more, takes longer, and requires more infrastructure. Most buyer problems resolve on the first or second step. Your job is figuring out which rung the buyer actually needs.
Things to follow up on...
- LoRA's hidden tradeoffs: A peer-reviewed paper found that LoRA and full fine-tuning produce structurally different internal solutions, with LoRA introducing "intruder dimensions" that can affect out-of-distribution behavior, which matters if a vendor claims LoRA is equivalent to full fine-tuning.
- GovCloud model lag: The gap between commercially available frontier models and what's actually authorized in government cloud regions is a moving target that shapes procurement decisions, and it shifts every time a provider completes a new FedRAMP authorization.
- Context engineering's rise: Anthropic Engineering now frames prompt engineering as a subset of a broader discipline called context engineering, which manages everything the model sees at inference time, including retrieved documents, tool results, and compressed conversation history.
- Fine-tuning hallucination research: A 2026 preprint by Kaplan et al. proposes that selectively freezing specific model parameters during fine-tuning can suppress factual hallucinations while preserving task learning, which could change the risk calculus if the findings survive peer review.