Temperature & Sampling

Gen AI · Customizing Model Behavior

Theory

Temperature is a setting that controls how randomly a model picks the next token from its predicted probabilities. Internally, each candidate token's score is divided by the temperature before the scores are turned into probabilities. A low temperature sharpens the distribution, so the top token wins almost every time and answers become focused and repeatable. A high temperature flattens it, so less likely tokens get picked more often and answers become varied, creative and more error-prone.

Top-p (nucleus sampling) is a related setting: the model only samples from the smallest group of tokens whose combined probability reaches p, such as 0.9, cutting off the long tail of unlikely tokens. Use low temperature for extraction, classification, SQL and code, and higher values for brainstorming. Adjust one setting at a time.

Why we use it

One model serves very different jobs: extracting an invoice date needs the same answer every time, while brainstorming names needs variety. Temperature and top-p let you pick that behaviour per request.

When to use it

Set temperature near 0 for facts, data extraction, code and anything you parse. Raise it moderately for creative writing or generating several options. Even at 0, outputs may not be perfectly identical across runs, so do not rely on exact repeats.

How it works

  1. Model scores every candidate next token
  2. Divide each score by the temperature
  3. Convert scores to probabilities (softmax)
  4. Is top-p set below 1? Yes: Keep only the top tokens that reach p combined. No: Keep all candidate tokens.
  5. Randomly sample one token from what remains
  6. Append the token and repeat for the next one

Example

Prompt: "Give me a name for a coffee shop."

temperature=0:   "The Daily Grind" (same answer almost every time)
temperature=1.3: "Bean There, Brewed That" / "Roast & Wander" / "Steam Dream Cafe" (varies each run)

For extracting a date from an invoice, use temperature=0. For a creative tagline, try temperature=1.0-1.3.

Key points

Gen AI topics

Foundations of Generative AI

Prompt Engineering

Retrieval & Embeddings

Customizing Model Behavior

AI Agents & Tool Use

Multimodal & Applied AI

Ethics, Cost & Careers

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Generative AI for the Data Stack

LLMs, prompt engineering, RAG, embeddings, agents and the tools shaping modern AI products — explained practically. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Section {{ sec.num }}

{{ sec.label }}

Gen AI  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

💡

{{ current.tldr }}

Explained

{{ para }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
See It In Action
{{ current.example }}
Key Takeaways — click to check off
{{ pt.mark }} {{ pt.text }}

Share this topic