Chunking Strategies

Gen AI · Retrieval & Embeddings

Theory

Chunking is the RAG step where long documents are split into smaller pieces before they are embedded and stored. Retrieval works on whole chunks, so chunk size strongly affects answer quality. Chunks that are too small lose context: a sentence like "It costs ₹499" is useless without knowing what "it" is. Chunks that are too large blend many topics into one embedding, dilute relevance and waste context-window tokens.

A common starting point is a few hundred tokens per chunk with a small overlap of a sentence or two, so an idea on a boundary is not cut in half. Splitting along the document's own structure (headings, sections, paragraphs, table rows) usually beats fixed-length cuts. Adding the document title or section heading to each chunk as metadata also helps retrieval and citations.

Why we use it

Good chunks make the right passage easy to find and complete enough to answer from. Bad chunking is one of the most common reasons a RAG app retrieves the wrong text or misses the answer entirely.

When to use it

Decide on chunking whenever you index documents for RAG. Use structure-aware splitting for manuals, policies and wikis; fixed-size chunks with overlap for unstructured text; and re-test sizes with real questions.

How it works

  1. Load a source document
  2. Does it have clear headings? Yes: Split by heading or section. No: Split into fixed-size windows.
  3. Is any piece much larger than the target size? Yes: Split it again with a small overlap. No: Keep it as one chunk.
  4. Attach title, section and source as metadata
  5. Embed and store each chunk
  6. Test retrieval with real user questions

Example

Bad: 1 chunk = entire 40-page manual (embedding blurs every topic together)
Bad: 1 chunk = single sentence (loses context: "It costs $49." — what does?)

Good: chunk by section heading, ~300 tokens each, 1-sentence overlap:
Chunk 1: "## Pricing\nOur Pro plan costs $49/month..."
Chunk 2: "...costs $49/month. It includes unlimited exports and priority support."

Key points

Gen AI topics

Foundations of Generative AI

Prompt Engineering

Retrieval & Embeddings

Customizing Model Behavior

AI Agents & Tool Use

Multimodal & Applied AI

Ethics, Cost & Careers

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Generative AI for the Data Stack

LLMs, prompt engineering, RAG, embeddings, agents and the tools shaping modern AI products — explained practically. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Section {{ sec.num }}

{{ sec.label }}

Gen AI  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

💡

{{ current.tldr }}

Explained

{{ para }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
See It In Action
{{ current.example }}
Key Takeaways — click to check off
{{ pt.mark }} {{ pt.text }}

Share this topic