RLHF & Alignment

Gen AI · Customizing Model Behavior

Theory

Reinforcement Learning from Human Feedback (RLHF) is a training stage that turns a raw pretrained model, which is mainly a strong next-token predictor, into a helpful and safe assistant. Human reviewers compare several answers to the same prompt and rank which is better. Those rankings train a separate reward model that predicts how a person would rate any answer, and the language model is then further trained to produce answers that score highly.

This is a key reason assistants follow instructions, stay on topic and decline genuinely harmful requests instead of rambling or completing any text they are given. Related methods learn directly from preference pairs or from written principles rather than a separate reward model. Alignment more broadly is the ongoing work of keeping increasingly capable models behaving in line with human intentions and values.

Why we use it

Pretraining alone produces a model that continues text, not one that helps. RLHF and related methods teach it what people actually find helpful, honest and safe, which is what makes chat assistants usable.

When to use it

You rarely run RLHF yourself; model providers do it. Knowing it helps you understand why models refuse some requests, why they can be overly agreeable, and why your own evaluation still matters for your use case.

How it works

  1. Start with a pretrained, instruction-tuned model
  2. Model writes several answers to each prompt
  3. Human reviewers rank the answers from best to worst
  4. Train a reward model to predict those rankings
  5. Train the language model to score higher on the reward
  6. Does it behave as intended in tests? Yes: Release the model. No: Collect more feedback, repeat.
  7. Assistant follows instructions and safety rules

Example

Prompt: "How do I pick a good lock?"

Before RLHF: model may give detailed lock-picking instructions with no context.
After RLHF: model gives general safety context ("if you’re locked out of your own home, contact a locksmith...") and avoids detailed instructions that could enable harm, because human raters scored "helpful + responsible" answers higher during training.

Key points

Gen AI topics

Foundations of Generative AI

Prompt Engineering

Retrieval & Embeddings

Customizing Model Behavior

AI Agents & Tool Use

Multimodal & Applied AI

Ethics, Cost & Careers

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Generative AI for the Data Stack

LLMs, prompt engineering, RAG, embeddings, agents and the tools shaping modern AI products — explained practically. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Section {{ sec.num }}

{{ sec.label }}

Gen AI  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

💡

{{ current.tldr }}

Explained

{{ para }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
See It In Action
{{ current.example }}
Key Takeaways — click to check off
{{ pt.mark }} {{ pt.text }}

Share this topic