Chunking Strategies
Gen AI · Retrieval & Embeddings
Theory
Chunking is the RAG step where long documents are split into smaller pieces before they are embedded and stored. Retrieval works on whole chunks, so chunk size strongly affects answer quality. Chunks that are too small lose context: a sentence like "It costs ₹499" is useless without knowing what "it" is. Chunks that are too large blend many topics into one embedding, dilute relevance and waste context-window tokens.
A common starting point is a few hundred tokens per chunk with a small overlap of a sentence or two, so an idea on a boundary is not cut in half. Splitting along the document's own structure (headings, sections, paragraphs, table rows) usually beats fixed-length cuts. Adding the document title or section heading to each chunk as metadata also helps retrieval and citations.
Why we use it
Good chunks make the right passage easy to find and complete enough to answer from. Bad chunking is one of the most common reasons a RAG app retrieves the wrong text or misses the answer entirely.
When to use it
Decide on chunking whenever you index documents for RAG. Use structure-aware splitting for manuals, policies and wikis; fixed-size chunks with overlap for unstructured text; and re-test sizes with real questions.
How it works
- Load a source document
- Does it have clear headings? Yes: Split by heading or section. No: Split into fixed-size windows.
- Is any piece much larger than the target size? Yes: Split it again with a small overlap. No: Keep it as one chunk.
- Attach title, section and source as metadata
- Embed and store each chunk
- Test retrieval with real user questions
Example
Bad: 1 chunk = entire 40-page manual (embedding blurs every topic together)
Bad: 1 chunk = single sentence (loses context: "It costs $49." — what does?)
Good: chunk by section heading, ~300 tokens each, 1-sentence overlap:
Chunk 1: "## Pricing\nOur Pro plan costs $49/month..."
Chunk 2: "...costs $49/month. It includes unlimited exports and priority support."Key points
- Too small loses context; too large dilutes relevance and wastes tokens
- Overlapping chunks help avoid cutting an idea in half at a boundary
- Chunk by structure (headings, paragraphs) when possible, not just fixed length
- Store title and section as metadata for better retrieval and citations
- Re-test chunk sizes with real questions; there is no universal best size