RAG (Retrieval-Augmented Generation)
Gen AI · Retrieval & Embeddings
Theory
Retrieval-Augmented Generation (RAG) combines search with generation. Before answering, the system retrieves the most relevant pieces of your own data (documents, tickets, manuals, wiki pages), inserts them into the prompt as context, and only then asks the model to answer. The model can now use specific, current or private information it was never trained on, such as last week's policy change.
The usual pipeline has two halves. Offline: split documents into chunks, embed each chunk and store the vectors. At question time: embed the question, find the top-matching chunks, and send them with the question and an instruction like "answer only from this context and cite the source." Updating knowledge is as simple as re-indexing changed documents, with no model retraining. Answer quality depends heavily on retrieval: if the right chunk is not found, the model cannot use it.
Why we use it
Models do not know your private or recent data and may guess. RAG grounds answers in real retrieved text, which reduces (but does not eliminate) hallucination and lets you show sources.
When to use it
Use RAG when answers must come from your own or frequently changing documents, such as policies, product docs or support history. If the problem is style or format rather than missing knowledge, better prompting or fine-tuning fits better.
How it works
- User asks a question
- Embed the question with the same model as the documents
- Search the vector store for the top-k chunks
- Were relevant chunks found? Yes: Build prompt: question + chunks + rules. No: Say the docs lack the answer.
- Model generates an answer from the given context
- Return the answer with source references
Example
User question: "What is our refund window for annual plans?"
1. Embed the question, search company docs vector store
2. Retrieve top match: "Annual plans: 30-day refund window..."
3. Build prompt:
"Using ONLY this context: [retrieved text]\nAnswer: What is our refund window for annual plans?"
4. Model answers: "Annual plans have a 30-day refund window."Key points
- Reduces hallucination by grounding answers in retrieved text
- Typical pipeline: chunk documents, embed, store, retrieve, generate
- Update knowledge by re-indexing documents, no retraining needed
- Retrieval quality limits answer quality: test what gets retrieved
- Ask the model to cite sources and to say when the context lacks the answer