A language model doesn't compute a single correct answer to your prompt. It computes a probability distribution across thousands of possible next tokens, then samples from that distribution to pick one. Then it does it again for the next token. And the next. The output you receive is one path through a forest of weighted possibilities — not the path, just a path. Run the same prompt again and the model walks a different path, because sampling is stochastic by design.
By design. The variability is the mechanism that makes language models useful for open-ended tasks. It's also the mechanism that makes them difficult to evaluate, audit, and deploy in contexts where reproducibility is a requirement. Three parameters control how much variability you get: temperature, top-p, and seed.
Temperature
After the model computes its probability distribution over candidate next tokens, temperature scales that distribution before sampling happens. Think of it as a dial that controls how peaked or flat the distribution is.
At high temperature (say, 1.5 or 2.0), the distribution flattens. Tokens that were unlikely become meaningfully more likely. The model takes more surprising turns. At low temperature (0.2, 0.1), the distribution sharpens. The highest-probability token dominates. The model becomes more focused, more predictable, and more repetitive. At temperature 0, you get greedy decoding: the model always picks the single most probable token. Same prompt, same model, same output, every time.
High temperature produces creative, varied outputs. Low temperature produces consistent, conservative ones. Neither is correct in the abstract; the right setting depends entirely on what the output is for.
Top-P
Top-p (also called nucleus sampling) works differently but toward a related end. Instead of scaling the full distribution, it restricts the candidate pool before sampling. At top-p = 0.9, the model identifies the smallest set of tokens whose cumulative probability reaches 90%, then samples only from that set. Everything outside the nucleus is off the table.
When the model is confident — when one or two tokens account for most of the probability mass — the nucleus is small and the output is focused. When the model is uncertain — when probability is spread across many candidates — the nucleus expands and the output has more room to vary. Top-p adapts to the model's confidence state in a way that a fixed temperature setting doesn't.
Most production deployments use temperature and top-p together, as complementary controls. The specific values that work well vary by model and use case; treat any published defaults as starting points, not targets.
Seed
Seed is the reproducibility control. It initializes the random number generator that drives the sampling process. Same seed, same temperature, same top-p, same prompt, same model version: same output.
The caveat matters: same model version. Model providers update their models, and an update can change the underlying probability distributions enough to break seed reproducibility even when everything else is held constant. Seed guarantees reproducibility within a stable deployment, not across model updates. If you're using seed for evaluation or audit purposes, you need to pin the model version alongside it. Most major providers expose version pinning in their APIs; treat it as a paired requirement with seed, not an optional detail.
The Production Tradeoff
This is where configuration becomes a real decision.
A federal agency evaluating an AI system for an ATO needs to demonstrate that the system behaves consistently and that evaluation results are reproducible. If the same test prompt returns different answers on different runs, the evaluator can't score it reliably, can't document the behavior for an auditor, and can't defend the procurement. The technical answer to this problem is low temperature plus a fixed seed plus a pinned model version. That combination turns a probabilistic system into something that behaves deterministically enough to evaluate, document, and defend.
The same logic applies to any workflow where the output feeds a downstream process: classification tasks, structured data extraction, policy summarization. Variability in those contexts isn't creative latitude — it's noise that propagates into decisions.
Conversely, a drafting assistant, a brainstorming tool, a system that generates options for human review — these benefit from temperature. An output that's always the same isn't useful when the point is to surface alternatives.
The question your accounts will actually face is which workflows need reproducibility controls, and whether the deployment configuration enforces them. That's a procurement and architecture conversation, and it belongs earlier in the evaluation process than most buyers realize.
Okta Concept Mapping
The closest IDAM analogue to seed is a nonce — a value that makes a transaction unique and traceable. The analogy breaks immediately, though: a nonce is designed to prevent replay and maximize unpredictability for security. A seed does the opposite, making outputs replayable and predictable. In IDAM, controlled randomness is a security property you maximize. In LLM inference, it's a quality property you tune. Your instinct that "randomness = risk" is exactly right in auth contexts and exactly wrong here. The mechanism looks similar; the design intent is inverted.
The underlying point, once the mechanism is visible: language models are probabilistic engines with tunable variance. Temperature and top-p set how much variance you allow. Seed collapses that variance to zero for a given configuration. Government procurement and evaluation workflows need the variance collapsed. Most other use cases benefit from leaving some of it in.
Know which workflow you're in before you configure the system. The controls exist precisely so you don't have to accept the default.