Fine-tuning and prompting are both methods for shaping how a language model behaves. You encounter them in accounts whenever a buyer's engineering team is evaluating whether to use a foundation model as-is or modify it — and the question almost always surfaces as "should we train our own model?" The language that buys you credibility in that conversation: fine-tuning adjusts a model's default response patterns without changing its factual knowledge base, and prompting accomplishes similar behavioral shaping without the infrastructure cost. Those two sentences are the entire argument. Everything below is the mechanism behind them.
Prompting
What it is: Runtime instruction delivered to a model through its context window, shaping response behavior without modifying model weights.
What it does: Directs the model's output by establishing persona, constraints, format requirements, and task framing at inference time. Every token of instruction consumes context window capacity and incurs inference cost, but nothing persists between sessions unless the system is designed to carry it forward. The model itself is unchanged. The input is different; that's all.
Where it comes from: Prompting as a formal practice emerged from OpenAI's GPT-3 release in 2020, which demonstrated that model behavior was highly sensitive to input framing. The field formalized rapidly around 2022–2023, producing documented techniques including chain-of-thought prompting (Wei et al., 2022), few-shot examples, and structured output constraints. Every major model provider — OpenAI, Anthropic, Google DeepMind — publishes prompting guidance as primary documentation. The technique requires no ML infrastructure and no training data.
What makes it distinct: Prompting is the only approach that requires nothing except the model you already have. A prompt update deploys in seconds. It can be versioned, tested, and rolled back like any other configuration artifact. The tradeoff is that instructions must be re-delivered at every inference call, and a sufficiently long system prompt on a high-volume application accumulates real cost.
Fine-Tuning
What it is: A training process that adjusts a pre-trained model's weights using a curated dataset, producing a modified model with different default behaviors.
What it does: Shifts the model's baseline tendencies — its default tone, format, vocabulary, and response patterns — by exposing it to examples of the desired behavior during a secondary training run. It does not add new factual knowledge in a reliable way. Facts introduced during fine-tuning are subject to the same hallucination risks as facts from pre-training, and they cannot be updated without another training run. The model learns style and pattern. It does not learn facts the way a database stores records.
Where it comes from: Fine-tuning is a standard transfer learning method that predates large language models. For LLMs specifically, the practice was formalized by OpenAI's InstructGPT work (2022), which used reinforcement learning from human feedback to fine-tune GPT-3 for instruction-following. Today, fine-tuning infrastructure is available through OpenAI's API, Google Vertex AI, AWS Bedrock, and open-source tooling for locally-hosted models. The output is a model artifact — a new set of weights — that can be deployed independently.
What makes it distinct: Fine-tuning is the only approach that modifies the model itself. The result can be served at lower latency than a larger frontier model, cached more efficiently, and deployed in environments where sending data to a third-party API is not permitted. The tradeoff is that producing the artifact requires labeled training data, compute budget, and a pipeline to manage model versions. PEFT methods (see below) reduce the compute cost substantially; they do not eliminate the data and pipeline requirements.
Comparison
Chosen structure: trait-led analysis. The central argument in this piece is categorical, not situational — fine-tuning is being systematically misunderstood as a knowledge-injection method, and that misunderstanding drives expensive decisions. A scenario-mapping structure would imply the choice is always contextual. It isn't. The traits below establish what each method actually does before addressing when to use which.
Knowledge vs. behavior
Fine-tuning shapes behavior. It does not reliably inject knowledge.
When a buyer says "we need to fine-tune our model on our policy documents," they are describing a knowledge problem. They want the model to know things it doesn't currently know — agency-specific procedures, current regulations, internal terminology. Fine-tuning will not solve this. A model fine-tuned on policy documents will learn to sound like those documents. It will not reliably know the facts in them, and it will not update when the documents change.
The method that solves the knowledge problem is retrieval — covered in Lesson 2. Fine-tuning solves a different problem: the model's default output format is wrong, its tone is off, it uses vocabulary the agency doesn't use, or it needs to follow a specific response structure consistently without being told every time. Those are legitimate problems. They are just not the problems most buyers think they're solving.
Cost and infrastructure
Prompting has no upfront cost and no infrastructure requirement beyond the model you're already calling. The ongoing cost is inference: every token in your system prompt is billed at inference rates, every call. For a 2,000-token system prompt running at high volume, this adds up.
Fine-tuning has significant upfront cost — labeled training data, compute for the training run, storage for the resulting model artifact, and a pipeline to manage versions. PEFT methods reduce the compute cost. LoRA (Low-Rank Adaptation), introduced by Hu et al. in 2021 and now the most widely adopted PEFT approach, injects small trainable matrices called adapters into the model's attention layers rather than retraining all weights. A LoRA fine-tune of a 7B-parameter model can run on a single GPU in hours rather than days on a cluster. The cost reduction is substantial.
What LoRA does not change: what fine-tuning fundamentally does. A LoRA-tuned model has the same knowledge limitations as a fully fine-tuned model. The adapters shape behavior more cheaply. They do not add a knowledge layer. Vendors who pitch LoRA as making fine-tuning "accessible" are correct about the cost curve and silent about the capability boundary.
Latency
A fine-tuned smaller model — a 7B or 13B parameter model with LoRA adapters — will generate tokens faster and at lower per-token cost than a frontier model. For high-volume, latency-sensitive applications: classification pipelines, format normalization, routine extraction at scale — this is a genuine operational advantage.
The latency advantage disappears as a decision criterion when the task requires reasoning depth the smaller model doesn't have. A fine-tuned Mistral 7B responding in 200ms with a wrong answer is not faster than a GPT-4o response in 800ms with a correct one. Speed is only an advantage when quality is held constant.
When frontier-model prompting outperforms a tuned smaller model
A well-prompted frontier model outperforms a fine-tuned smaller model when the task requires multi-step reasoning across domains, when the input is novel relative to the training distribution, or when the cost of a wrong answer is high enough that quality dominates cost as a decision criterion.
For public sector accounts specifically: the tasks that matter most — policy analysis, anomaly detection in access logs, complex query interpretation, legal or regulatory reasoning — are exactly the tasks where frontier models maintain their advantage. The reasoning gap between a 7B-parameter fine-tuned model and GPT-4o or Claude 3.5 Sonnet is not a gap that fine-tuning closes. Fine-tuning adjusts behavior; reasoning capacity comes from architecture and training scale. A model that couldn't work through a multi-step policy question before fine-tuning won't do it after.
The tasks where a fine-tuned smaller model wins — high-volume classification, consistent format output, domain-specific terminology normalization — are also the tasks where the cost savings are most meaningful. The buyer needs to know which problem they're actually solving before they choose the method.
Updateability
Prompting updates instantly. Change the system prompt, change the behavior. No retraining, no new artifact, no deployment pipeline.
Fine-tuning updates require a new training run. If the agency's policies change — and in federal civilian agencies, they change constantly — the fine-tuned model's trained behavior is now misaligned with current policy. Correcting it means another training run, another labeled dataset, another deployment. This is not a theoretical concern. It is the most common reason fine-tuning projects stall after the initial deployment.
Field Language Guide
| Don't say | Do say | Why it matters |
|---|---|---|
| "We can train the model on your documents" | "Fine-tuning shapes how the model responds, not what it knows — for knowledge, we use retrieval" | Prevents the buyer from expecting fine-tuning to solve a knowledge problem it can't solve |
| "Fine-tuning teaches the model new information" | "Fine-tuning adjusts the model's default style and format — the knowledge has to come from somewhere else" | Corrects the most common executive misconception before it drives a bad architecture decision |
| "You need to train your own model" | "What behavior are you trying to change — and is that a style problem or a knowledge problem?" | Forces the buyer to specify the actual problem before committing to an infrastructure approach |
| "LoRA makes fine-tuning easy" | "LoRA reduces the compute cost of fine-tuning — it doesn't change what fine-tuning does" | Prevents vendors from using efficiency claims to obscure capability limitations |
| "A fine-tuned model will always be better for your domain" | "A fine-tuned smaller model is faster and cheaper for high-volume routine tasks — for complex reasoning, a frontier model with a good prompt usually wins" | Gives the buyer a decision criterion instead of a preference |
| "Prompting is just telling the model what to do" | "Prompting is runtime configuration — it shapes every response without modifying the model, and it can be updated instantly" | Elevates prompting from a workaround to a legitimate architectural choice |
| "You'll need ML infrastructure to do this" | "Fine-tuning requires labeled training data, a training pipeline, and model version management — prompting requires none of those" | Makes the infrastructure cost concrete before the buyer commits |
| "We can fine-tune it to stay current" | "Fine-tuning is static — every policy change requires a new training run; prompting and retrieval update without retraining" | Surfaces the maintenance cost that fine-tuning projects routinely underestimate |
| "PEFT is basically the same as fine-tuning" | "PEFT methods like LoRA make fine-tuning cheaper — the result is still a fine-tuned model with the same knowledge limitations" | Prevents efficiency framing from obscuring what the method actually produces |
| "The model will learn your agency's terminology" | "The model will learn to use your terminology consistently — it won't learn the facts behind it" | Keeps the buyer's expectations calibrated to what fine-tuning actually delivers |
Callout: Okta Concept Mapping
IDAM analog: Default attribute mappings in an identity provider.
When you configure an application in Okta and define which claims get asserted by default — department, role, clearance level — you're shaping how the IdP presents identity to that application. The mapping changes the presentation; it doesn't change what the user actually has access to. Fine-tuning works the same way: it shapes how the model presents its outputs by default, without changing the underlying capability.
Where it holds: Both default claim mappings and fine-tuning shape default behavior without modifying underlying capabilities. Both require deliberate configuration effort upfront. Both produce consistent, predictable output in the cases they were designed for.
Where it breaks: Claim mappings are deterministic. Fine-tuning shifts probabilistic tendencies. A fine-tuned model can still be prompted out of its trained behavior — a sufficiently crafted input can override fine-tuned style preferences in ways that a claim mapping cannot be overridden at runtime. In a buyer conversation, this matters: fine-tuning is not a security control, and it is not a behavioral guarantee. If the buyer is thinking about fine-tuning as a way to prevent the model from doing something, that's a different problem requiring a different solution.