LLM temperature is a number, usually between 0 and 2, that controls how much randomness a model injects when picking its next word. Low temperature makes output predictable and repetitive. High temperature makes it varied and sometimes incoherent. Almost nobody sets it correctly, and that single number explains more about why your chatbot sounds "boring" or "unhinged" than any prompt trick you've tried.
Every time a language model generates a token, it doesn't just pick the single best word. It computes a probability distribution over every possible next token in its vocabulary, often tens of thousands of options. Temperature reshapes that distribution before the model samples from it. That's the entire mechanism. No magic, no hidden reasoning layer. Just math applied to a list of probabilities.
Under the hood, a model produces raw scores called logits for every possible next token. Those logits get converted into probabilities using a function called softmax. Temperature divides the logits by itself before that conversion happens. Divide by a small number like 0.2, and the differences between logits get exaggerated, so the top candidate becomes overwhelmingly likely. Divide by a large number like 1.5, and the differences flatten out, giving weaker candidates a real shot at being picked.
This is why temperature 0.1 feels safe and repetitive. The model keeps grabbing the single most probable word, over and over, because the distribution has been squeezed into near-certainty. It's also why temperature 1.8 can produce sentences that drift into nonsense. Words that would normally have a 0.01% chance of appearing suddenly get picked at a rate that breaks coherence.
None of this changes what the model "knows." Temperature doesn't add facts or remove them. It only changes which facts, phrasings, or word choices get surfaced from the same underlying probability landscape. That distinction matters, because people often blame temperature for hallucinations when the real cause is a weak or ambiguous prompt.
Setting temperature to 0 tells the model to always pick the highest-probability token, a method called greedy decoding. In theory that should make output identical every single time for the same input. In practice it often doesn't, and this trips up a lot of engineers building anything that depends on reproducibility.
Two things break the illusion of determinism. First, floating-point math on GPUs isn't perfectly consistent across hardware batches, so tiny numerical differences can tip a near-tie between two tokens. Second, many production APIs run requests through load-balanced clusters and apply internal optimizations that introduce small variance even at temperature 0. If you need true reproducibility, ask the provider about a seed parameter, and understand that even seeds only guarantee consistency within the same model version and infrastructure.
The practical takeaway: treat temperature 0 as "mostly deterministic, rarely identical," not as a hard guarantee. If your application logs exact outputs for auditing or testing, build in tolerance for minor variation rather than assuming byte-for-byte repeatability.
Temperature is not the only sampling control, and confusing it with the others leads to bad configuration. Top-k restricts the model to choosing only among the k most probable tokens, discarding everything else before sampling. Top-p, also called nucleus sampling, instead keeps the smallest set of tokens whose cumulative probability crosses a threshold like 0.9, so the pool size shifts dynamically depending on how confident the model is.
| Parameter | What it controls | Typical range | Effect at high setting |
|---|---|---|---|
| Temperature | Sharpness of the probability curve | 0 to 2 | More varied, sometimes incoherent output |
| Top-k | Fixed number of candidate tokens | 1 to 100 | More lexical variety, less relevance filtering |
| Top-p | Cumulative probability cutoff | 0.1 to 1.0 | Wider candidate pool when model is uncertain |
Most APIs let you set all three at once, which is exactly how people end up with confusing results. If temperature is high but top-p is also high, you get maximum randomness stacked on maximum randomness, and output degrades fast. A cleaner approach is to adjust one lever at a time, usually temperature, and leave the others at sensible defaults like top-p 0.9 unless you have a specific reason to touch them.
Code generation, data extraction, and anything requiring a single correct answer should run at low temperature, typically 0 to 0.3. You want the model converging on its best guess, not exploring alternatives. A SQL query or a JSON parser has one right shape. Randomness here only adds bugs.
Creative writing, brainstorming, and marketing copy benefit from temperature in the 0.7 to 1.0 range. You want the model to consider less obvious word choices and sentence structures, because the "most probable" continuation is often the most clichéd one. Ask for ten taglines at temperature 0.2 and you'll get ten near-identical variations. Ask at 0.9 and you actually get options worth comparing.
Anything above 1.2 is rarely useful in production. It's a setting for research, stress-testing, or generating deliberately weird text. If your output at 1.4 reads like a fever dream, that's the setting working as designed, not a model failure.
The most common error is treating temperature as a single global "creativity dial" for an entire product. A support chatbot answering billing questions should run near 0. The same product's marketing-copy generator feature should run near 0.8. Shipping one temperature setting across every use case in an app guarantees that some feature feels wrong.
The second mistake is blaming temperature for factual errors. A model hallucinating a nonexistent API method or a fake statistic is a retrieval and grounding problem, not a sampling problem. Dropping temperature to 0 won't fix a model that never had the right information to begin with. It'll just make the model confidently wrong in a more consistent way, which is arguably worse because the error becomes harder to catch through repeated sampling.
Third, people forget that low temperature increases repetition risk over long outputs. Greedy decoding can get stuck in loops, repeating a phrase or clause because the highest-probability token keeps pointing back to itself. If you see a model looping on long generations, raising temperature slightly, or adding a repetition penalty if the API supports one, usually fixes it faster than any prompt rewrite.
Run the same prompt five times at your chosen temperature before shipping anything. If the outputs are near-identical at 0.7, you probably don't need that much randomness and can drop it for more consistency. If outputs at 0.3 are wildly different from each other, something else is unstable, possibly the prompt itself is ambiguous enough that even low-randomness sampling can't nail down a single answer.
Treat temperature like any other production parameter: version it, log it, and test it against real inputs rather than a handful of demo prompts. A setting that looks great in a quick test can behave differently once real users start sending messy, unpredictable queries. The number itself is simple. Getting it right for your actual use case takes the same discipline as tuning any other system parameter.
Temperature controls how sharply a language model favors its highest-probability next word over weaker alternatives. Low temperature makes the model pick the most likely word almost every time, producing predictable output. High temperature flattens those probabilities, letting less likely words get chosen, which produces more varied but sometimes less coherent text.
No, not always. Temperature 0 uses greedy decoding, which should pick the same top token every time for the same input, but floating-point rounding on GPUs and infrastructure-level variance can still produce slightly different outputs across requests. It's close to deterministic in practice, but not a strict guarantee unless the provider offers a fixed seed and identical hardware conditions.
Use a low temperature, generally between 0 and 0.3, for code generation and any task with a single correct answer. Higher temperatures introduce unnecessary variation into syntax and logic where consistency matters far more than creativity.
No, temperature and hallucination are separate problems. Hallucinations usually come from missing or poorly grounded information, not from how the model samples among probable words. Lowering temperature can make a hallucinating model more consistently wrong rather than fixing the underlying accuracy issue.
Generally no, because stacking two randomness controls at high settings compounds unpredictability and makes output quality hard to diagnose. It's cleaner to hold top-p at a sensible default like 0.9 and adjust temperature alone until you get the balance of consistency and variety your task needs.