Temperature & Sampling
Gen AI · Customizing Model Behavior
Theory
Temperature is a setting that controls how randomly a model picks the next token from its predicted probabilities. Internally, each candidate token's score is divided by the temperature before the scores are turned into probabilities. A low temperature sharpens the distribution, so the top token wins almost every time and answers become focused and repeatable. A high temperature flattens it, so less likely tokens get picked more often and answers become varied, creative and more error-prone.
Top-p (nucleus sampling) is a related setting: the model only samples from the smallest group of tokens whose combined probability reaches p, such as 0.9, cutting off the long tail of unlikely tokens. Use low temperature for extraction, classification, SQL and code, and higher values for brainstorming. Adjust one setting at a time.
Why we use it
One model serves very different jobs: extracting an invoice date needs the same answer every time, while brainstorming names needs variety. Temperature and top-p let you pick that behaviour per request.
When to use it
Set temperature near 0 for facts, data extraction, code and anything you parse. Raise it moderately for creative writing or generating several options. Even at 0, outputs may not be perfectly identical across runs, so do not rely on exact repeats.
How it works
- Model scores every candidate next token
- Divide each score by the temperature
- Convert scores to probabilities (softmax)
- Is top-p set below 1? Yes: Keep only the top tokens that reach p combined. No: Keep all candidate tokens.
- Randomly sample one token from what remains
- Append the token and repeat for the next one
Example
Prompt: "Give me a name for a coffee shop."
temperature=0: "The Daily Grind" (same answer almost every time)
temperature=1.3: "Bean There, Brewed That" / "Roast & Wander" / "Steam Dream Cafe" (varies each run)
For extracting a date from an invoice, use temperature=0. For a creative tagline, try temperature=1.0-1.3.Key points
- Low temperature (near 0) is focused and repeatable: good for facts, SQL and code
- High temperature adds variety and creativity, but also error risk
- Top-p (nucleus sampling) limits sampling to the most likely tokens
- Change temperature or top-p, not both at once, when tuning
- Temperature 0 reduces but may not remove run-to-run variation