Chunking for RAG: how to split documents the right way
What chunking is, how to choose chunk size and overlap for RAG, structure-aware splitting, parent-child retrieval, and how to measure it with recall@k.
Retrieval can only return what a chunk contains. If a boundary lands in the middle of the answer, the system either never finds the passage or hands the model half a sentence and lets it guess the rest. Chunking is the earliest and cheapest place where RAG quality is decided. New to the topic? Start with our plain-English RAG guide.
What is chunking?
Chunking is splitting a long document into smaller pieces, each one a self-contained unit for three jobs: embedding, retrieval, and citation. A good chunk is narrow enough that its vector describes one idea, and wide enough that the idea stands without the neighbouring text.
Size is a trade-off, not a setting with a correct value. Small chunks give sharp vectors but starve the model of context. Large chunks keep context but blur the vector across several topics and flood the prompt.
Why chunking decides RAG quality
Four failures, all caused by the boundary rather than the model.
- Too small loses context. The line "applies within 14 days" does not say 14 days from when, or for what. Its vector is close to meaningless.
- Too large dilutes the signal. One 768-dimension vector covering three topics in 1,500 tokens matches any single-topic question only partially, so the score drops and the right passage falls out of the top-k. Five such chunks also mean 7,500 tokens pushed into the prompt.
- A cut splits the answer. The question needs two consecutive sentences; each lives in a different chunk, and neither is enough alone.
- Losing the heading loses the meaning. "Step 3: send the confirmation email" only makes sense when you know whether it belongs to returns or to hiring.
Three ways to split
Fixed size. Count tokens and cut every N tokens, with overlap. Fast, easy to code, works on any format — but cuts mid-sentence and ignores structure entirely.
Structure-aware. Cut at headings, paragraph ends, list items, table boundaries. Chunks keep natural units of meaning, at the cost of depending on parser quality and uneven section lengths.
Semantic. Embed each sentence, measure similarity between neighbours, cut where the topic shifts. Follows the topic more closely, but costs one embedding pass over the whole corpus and is hard to debug when it goes wrong.
| Approach | Pros | Cons | Use when |
|---|---|---|---|
| Fixed size | Fast, simple, any format | Cuts mid-sentence, ignores structure | First draft, unstructured text |
| Structure-aware | Keeps units of meaning, easy to cite | Parser-dependent, uneven lengths | Markdown, HTML, headed documents |
| Semantic | Follows topics, little mixed content | Expensive, hard to explain | Long text without clear headings |
Figure 1. The same document, three kinds of boundary. Fixed-size splitting breaks a sentence; the other two keep units of meaning intact.
Choosing size and overlap
300–500 tokens is a reasonable starting range when your model caps input at 512; go up to 800 only when the model accepts long input (8k or more). Below 300 the context is usually too thin; above 800 the vector starts to blur and prompt cost climbs fast.
Overlap of 10–15% keeps a sentence cut at the end of one chunk intact at the start of the next. Do not overdo it: 50% overlap means every idea is embedded twice, the top-k fills with near-duplicates, and the prompt pays for repetition.
Do not measure Vietnamese by characters. At the same character count Vietnamese needs more tokens than English, because of tone marks and the way subword tokenizers split syllables. Count with the tokenizer of the model you actually use, on your own documents.
Then check the model's hard limit: many older embedders (MiniLM, E5) truncate at 512 tokens, while current models such as text-embedding-3 (8,191) and BGE-M3 (8,192) accept far longer input. Verify the max input of the model you actually use; if it is 512, an 800-token chunk gets silently cut at embedding or reranking time, and what is dropped is usually the tail — where the important condition tends to sit.
Splitting along document structure
For Markdown, cut at heading lines and store the heading path in metadata (Returns policy > Time limit > Promotional items). When the chunk reaches the prompt, that prefix restores context for a few tokens.
For HTML, use heading tags and text blocks rather than stripping tags and cutting by characters. For PDFs, extract by layout: cutting blindly on extracted text often merges two columns into one line.
Three practical rules: keep tables whole, and repeat the header row in every chunk if a table must be split; never separate a list from its intro ("Documents required:" belongs with the list); never exceed the embedder's token limit.
Parent-child (small-to-big)
This splits two conflicting needs apart: precise matching wants small units, answering wants large context.
Index small child chunks — a paragraph, a section — so the vector stays close to the content, and have each child point to its parent, the full section. When a query matches a child, the system sends the parent to the LLM.
The result: matching as sharp as 150-token chunks, context as complete as 1,000-token chunks, and a prompt holding one section instead of five disconnected fragments.
Figure 2. Two levels: the index holds child chunks, the answer holds the parent. The query matches below, the LLM reads above.
Minimal code
The snippet below splits on headings first, then packs paragraphs inside each section until the token budget is full, with overlap carried from the previous chunk:
import re
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
def ntok(text: str) -> int:
return len(enc.encode(text))
def split_sections(md: str) -> list[str]:
# a heading stays with the text that follows it
parts = re.split(r"(?m)^(?=#{1,3} )", md)
return [p.strip() for p in parts if p.strip()]
def pack(section: str, budget: int = 500, overlap: int = 60) -> list[str]:
paras = [p.strip() for p in re.split(r"\n\s*\n", section) if p.strip()]
chunks, buf = [], ""
for p in paras:
if buf and ntok(buf + "\n\n" + p) > budget:
chunks.append(buf)
tail = enc.decode(enc.encode(buf)[-overlap:])
buf = tail + "\n\n" + p
else:
buf = f"{buf}\n\n{p}" if buf else p
if buf:
chunks.append(buf)
return chunks
chunks = [c for s in split_sections(md) for c in pack(s)]
Two things to tune on real data: budget must stay under the embedder limit (500 tokens is safe for a 512-token model), and overlap should be 10–15% of budget. A section holding one paragraph longer than budget still produces an oversized chunk, so add a sentence-level split for that case.
Measuring chunk quality
Build a golden set of 20–50 questions, each tied to the passage that contains the answer, then measure:
- recall@k before and after a change. The most direct signal: better chunking puts the right passage in the top-k more often.
- Whether the answer fits inside one chunk. If two chunks are needed to assemble it, the boundary is wrong.
- Chunk size distribution. A few 20-token chunks beside a few 3,000-token chunks means a broken parser.
- Orphan chunks. No heading, no context, or punctuation only — count them and check whether they occupy the top-k.
The most convincing check is an A/B of two configurations on the same golden set: 400 versus 700 tokens, with and without the heading prefix. Read the 20 worst results by hand to see whether the failure sits at the boundary or later in the pipeline. If the correct passage is already in the top-100 but ranked low, chunking is no longer the problem — that is when you add a cross-encoder to reorder the results.
Common mistakes
- Cutting mid-sentence because you counted characters instead of cutting at punctuation.
- A chunk larger than the model limit, silently truncated at embedding time.
- Too much overlap, so the same content takes three slots in the top-k and in the prompt.
- No heading in metadata, leaving chunks without context when they stand alone.
- Splitting a table between rows without repeating the header, so numbers lose their meaning.
- One chunk for the whole page, turning each vector into an average of every topic on it.
FAQ
What chunk size is best?
There is no universal number. Start at 400–500 tokens with 10% overlap, measure recall@k, then try 300 and 700 on the same golden set. Short FAQ entries suit small chunks; policies full of conditions suit larger ones.
Do I need overlap?
Yes, but a little. 10–15% is enough to rescue a sentence cut at a boundary. If you split on structure and every chunk already holds a complete section, you can drop overlap close to zero without losing anything.
Tokens or characters?
Tokens. Characters do not map to what the model actually charges you for, and for Vietnamese the token-per-character ratio drifts even further. Count with the tokenizer of the model you use.
Conclusion
Chunking is the cheapest lever in RAG: cut at real boundaries, keep the heading, size by real tokens, and measure with recall@k on a golden set. Do those four things before changing models or adding a reranker. If you would rather have it built and measured for you, our engineering team at Cross Tech Edu handles the whole pipeline, and our roundup of free AI tools for small business covers the ready-made options if you only need something that works today.