Prompting vs RAG vs Fine-Tuning
Gen AI · Customizing Model Behavior
Theory
When a model does not behave the way you need, there are three main levers, and they fix different problems. Prompting (clear instructions, examples, output format) changes behaviour for one request and costs nothing to try. RAG gives the model facts it does not have by retrieving your documents at question time, which fixes missing or changing knowledge. Fine-tuning changes the model's default behaviour through extra training, which fixes consistent style, format or a narrow task at high volume.
A useful rule: if the model lacks information, use RAG; if it has the information but behaves wrongly, improve the prompt first and fine-tune only if that is not enough. Try them in order of cost: prompt, then RAG, then fine-tuning, measuring each step on the same test set. Many production systems use all three together.
Why we use it
Teams often jump to fine-tuning when a better prompt or retrieval would solve the problem faster and cheaper. Knowing what each lever fixes saves weeks of effort and keeps knowledge easy to update.
When to use it
Use this decision whenever an LLM feature falls short. Diagnose the failure first: wrong facts or missing private data points to RAG; wrong format, tone or task behaviour points to prompting, then fine-tuning.
How it works
- The model's answers are not good enough
- Improve the prompt: instructions, examples, format
- Is the problem missing or changing knowledge? Yes: Add RAG over your documents. No: Keep the improved prompt.
- Is behaviour still inconsistent at scale? Yes: Fine-tune on curated examples. No: Ship the current setup.
- Measure every change on the same test set
Example
Problem 1: Bot does not know this month's refund policy
-> knowledge gap -> RAG over the policy documents
Problem 2: Bot knows the answer but replies in long paragraphs
-> behaviour -> add format rules + 2 examples to the prompt
Problem 3: 50,000 tickets/day must be tagged in a strict house format,
prompt works but is long and costly
-> fine-tune a smaller model on 2,000 labelled ticketsKey points
- Missing knowledge points to RAG; wrong behaviour points to prompting, then fine-tuning
- Try levers in order of cost: prompt, then RAG, then fine-tune
- Fine-tuning is a poor way to add facts that change often
- Measure every change on the same test set
- Production systems often combine all three