RLHF & Alignment
Gen AI · Customizing Model Behavior
Theory
Reinforcement Learning from Human Feedback (RLHF) is a training stage that turns a raw pretrained model, which is mainly a strong next-token predictor, into a helpful and safe assistant. Human reviewers compare several answers to the same prompt and rank which is better. Those rankings train a separate reward model that predicts how a person would rate any answer, and the language model is then further trained to produce answers that score highly.
This is a key reason assistants follow instructions, stay on topic and decline genuinely harmful requests instead of rambling or completing any text they are given. Related methods learn directly from preference pairs or from written principles rather than a separate reward model. Alignment more broadly is the ongoing work of keeping increasingly capable models behaving in line with human intentions and values.
Why we use it
Pretraining alone produces a model that continues text, not one that helps. RLHF and related methods teach it what people actually find helpful, honest and safe, which is what makes chat assistants usable.
When to use it
You rarely run RLHF yourself; model providers do it. Knowing it helps you understand why models refuse some requests, why they can be overly agreeable, and why your own evaluation still matters for your use case.
How it works
- Start with a pretrained, instruction-tuned model
- Model writes several answers to each prompt
- Human reviewers rank the answers from best to worst
- Train a reward model to predict those rankings
- Train the language model to score higher on the reward
- Does it behave as intended in tests? Yes: Release the model. No: Collect more feedback, repeat.
- Assistant follows instructions and safety rules
Example
Prompt: "How do I pick a good lock?"
Before RLHF: model may give detailed lock-picking instructions with no context.
After RLHF: model gives general safety context ("if you’re locked out of your own home, contact a locksmith...") and avoids detailed instructions that could enable harm, because human raters scored "helpful + responsible" answers higher during training.Key points
- Turns a raw next-token predictor into a helpful assistant
- Human raters compare outputs; a reward model learns their preferences
- Related methods learn from preference pairs or written principles directly
- Can make models over-cautious or overly agreeable if the signal is off
- Alignment is an active research area as models become more capable