Chunking strategies for RAG: fixed, recursive, and semantic

Chunking is the partition of a document's text into retrieval units. Each unit gets embedded into one vector, indexed, and returned whole when it matches a query. The chunk is therefore three things simultaneously, and the tension between them is the entire design problem:

1. The unit of embedding. One vector must summarise the whole chunk. Longer chunk, more topics compressed into a fixed number of dimensions, blurrier vector.
2. The unit of retrieval. Either the chunk matches and comes back entirely, or it does not and none of it comes back. There is no partial retrieval.
3. The unit of context. The chunk is what the model actually reads. If the sentence that answers the question is present but the noun it refers to was left in the previous chunk, the model has an ambiguous pronoun and no way to resolve it.

Requirement 1 pushes chunks smaller. Requirement 3 pushes them larger. Requirement 2 says you cannot compromise by retrieving half a chunk. Every chunking strategy is a way of managing that three-way tension, and every strategy is a trade rather than a solution.

There is no official optimal chunk size. NVIDIA's published RAG material describes ingestion, chunking, and overlap as pipeline stages without prescribing a universal number, and no exam-governing source states one. [NVIDIA-DOC] Chunk-size figures circulating in tutorials and blog posts are [FIELD] at best — someone's result on someone's corpus with someone's queries and embedding model, reported without those variables. Treat any specific number in this lesson as an illustration of arithmetic, never as a recommendation you can carry to your own corpus without measuring.

L1 — Why you cannot embed a whole document

An embedding model compresses a span of text into a fixed-length vector — a few hundred to a few thousand numbers, however long the input. That compression is lossy, and the loss grows with length. Embed a single paragraph about warranty periods and the vector points squarely at "warranty periods". Embed a 40-page chapter covering warranty, shipping, returns, and liability and the vector points at the average of four topics, which is a place in embedding space where nothing in particular lives. A query about warranty will not match it strongly, because the vector is not about warranty; it is about a blend.

This is the whole reason chunking exists, and it explains why "the model has a huge context window, so I can skip chunking" is wrong. The constraint is not the model's reading capacity. The constraint is the embedding's representational capacity. Those are different components, and one of them got much better over the last few years while the other did not change nearly as much.

L2 — The three strategies mechanically

Fixed-size chunking. Walk the text and cut every N units — tokens or characters — with an optional overlap of M units carried from the end of one chunk into the start of the next. Mechanically trivial, completely predictable, and structurally blind: it will cut mid-sentence, mid-word, mid-table, and mid-number.

Fixed-size, N = 500 tokens, overlap = 50 tokens

chunk 1  tokens    0 –  500
chunk 2  tokens  450 –  950     (450–500 repeated)
chunk 3  tokens  900 – 1400     (900–950 repeated)
...

For a 100,000-token document:
  chunks ≈ 100,000 / (500 − 50) = 100,000 / 450 ≈ 223 chunks
  stored tokens ≈ 223 × 500 = 111,500
  storage overhead from overlap ≈ 11.5%
Illustrative arithmetic on a constructed document.

That last number is the cost of overlap, and it is worth internalising the formula rather than the figure: overhead ≈ overlap / (chunk size − overlap). At 500/50 it is about 11%. At 500/250 — a 50% overlap, which people do reach for — it is 100%: you store and embed your corpus twice, and every passage exists in two chunks that are near-duplicates of each other, which is a deduplication problem you have manufactured on purpose. 06-04 deals with the consequences.

Recursive chunking. Given an ordered list of separators — typically ["\n\n", "\n", ". ", " "] — split the text on the first separator. If a resulting piece is still over the size limit, split that piece on the next separator down, and recurse. If a piece is under the limit, keep it. Pieces are then merged greedily up to the size limit so you do not emit a chunk per sentence.

The effect is that cuts land on paragraph boundaries wherever paragraphs are small enough, on sentence boundaries where they are not, and only fall back to arbitrary cuts in pathological cases such as a single unbroken 5,000-token block. This is why recursive chunking is the sensible default for prose: it costs nothing extra, it needs no model, and it respects structure whenever structure is available.

Semantic chunking. Split the text into sentences. Embed each sentence. Walk the sequence comparing each sentence's embedding to the running context — typically by cosine similarity, the operation from 01-04 — and cut where similarity drops below a threshold, on the theory that a sharp drop marks a topic boundary. Merge the resulting spans up to a size ceiling.

Semantic chunking, sketch of the signal

sentence     cosine similarity to previous
s1  →  s2                 0.88
s2  →  s3                 0.85
s3  →  s4                 0.31   ← below threshold: cut here
s4  →  s5                 0.90
s5  →  s6                 0.87
Illustrative values, constructed to show the shape of the signal, not measured.

Semantic chunking costs an embedding call per sentence at ingestion time, adds a threshold you have to tune, and can behave erratically on text without clear topical structure — a specification table has no topic boundaries in this sense, and a tightly argued page of prose may have none either. It is the strategy with the highest ceiling and the highest overhead, and it should be the last thing you try rather than the first.

L3 — What a chunk boundary does to the vector

At the deepest tier, all three strategies are attempts to answer one question: what does this vector claim to be about? An embedding is a claim about a span's meaning, and a chunk boundary determines what span makes the claim.

Two failure geometries matter. The first is dilution: too much unrelated content in one chunk pulls the vector towards a centroid of several topics, so it matches nothing strongly. The chunk is present, indexed, and never retrieved for the query it should answer, because a shorter and less relevant chunk out-scored it. The second is decontextualisation: too little content, so the chunk is a sharp vector pointing at a fragment that cannot be understood alone. "This limit does not apply to enterprise agreements" is a maximally crisp embedding of a sentence that is useless without the previous paragraph naming the limit.

Overlap is the standard mitigation for decontextualisation at boundaries, and it is a partial one. It handles the case where the answer straddles a cut. It does not handle the case where a chunk's meaning depends on a heading twelve paragraphs earlier — that is a metadata and context-enrichment problem, and it is what 06-03 is for.

This is the confusable pair the module was built to resolve, and it deserves its own table.

The relationship between them is real but indirect, and it runs in exactly one direction: the context window constrains how many chunks you can retrieve, not how large each one should be. If you have 8,000 tokens of usable context budget after instructions and chat history, and your chunks are 400 tokens, you can afford roughly 20 chunks; if they are 2,000 tokens, roughly 4. That is a top-k decision, which is 07-09's territory, and the budgeting arithmetic behind it is 04-06's.

Read the failure-mode row again, because it is the exam-relevant asymmetry. Getting the context window wrong throws an error or truncates visibly. Getting chunk size wrong produces a system that works and quietly answers a class of questions badly.

A fourth family deserves naming even though it is not one of the three the exam is likeliest to list: structure-aware or document-aware chunking, which uses the document's own hierarchy as the split rule — one chunk per Markdown section, per HTML h2 block, per contract clause, per table row. When the document has real structure and the parser from 06-01 preserved it, this frequently beats all three of the above, because the author already did the semantic segmentation and you are simply not throwing it away. It is also the strategy most dependent on parsing quality, which is why the module teaches parsing first.

A constructed comparison. Every number below is arithmetic on an invented document, chosen to make the trade-offs visible — not measured retrieval results. Constructed scenario.

Take an internal HR policy document: 60 pages, roughly 30,000 tokens, structured as 47 numbered sections with headings, most sections two to five paragraphs, plus 9 tables of entitlements by employee grade.

Run A — fixed-size, 512 tokens, 64-token overlap.

chunks ≈ 30,000 / (512 − 64) = 30,000 / 448 ≈ 67 chunks
stored tokens ≈ 67 × 512 ≈ 34,300      overhead ≈ 14%
section headings landing inside a chunk body rather than at its start: most of them
tables cut across a chunk boundary: 6 of 9 (a 512-token chunk cannot hold a large table)
Constructed arithmetic.

The predictable symptom: ask "how many days of study leave does a grade 4 employee get?" and the retrieved chunk contains the second half of the entitlements table with the header row in the previous chunk. The answer is present in the corpus and unusable in the retrieved context.

Run B — recursive, 512-token target, 64-token overlap, separators ["\n\n", "\n", ". ", " "].

chunks ≈ 71 (slightly more, because pieces end early at paragraph boundaries)
chunks ending mid-sentence: ~0
tables cut across a boundary: still 6 of 9 — a paragraph separator does not protect a
  table, because the table is one long block with no blank lines inside it
Constructed arithmetic.

Better prose behaviour, identical table behaviour. This is the most important observation in the worked example: recursive chunking fixes sentence integrity and does nothing for tables. Tables need either structure-aware handling at parse time or a larger size limit specifically for table blocks. Recursive chunking is not a substitute for the structure extraction from 06-01.

Run C — structure-aware, one chunk per numbered section, 1,024-token cap with recursive fallback for oversize sections.

sections                                   47
sections under the cap → one chunk each    41
sections over the cap → recursively split   6  → 15 chunks
total chunks                               56
tables cut across a boundary                1  (one multi-page table exceeded the cap)
every chunk begins with its own section heading: yes, by construction
Constructed arithmetic.

Fewer chunks, each aligned to a unit a human author already decided was coherent, each carrying its heading. The cost is that it only works because the document had numbered sections and the parser kept them. Give the same code a scanned fax and it degrades to recursive chunking on a soup of text — which is the honest reason to always have the recursive path as a fallback.

What the example does not tell you is which run retrieves best, because that is measurable only against real queries. That measurement is the by-hand retrieval test from 03-04, and the point of chunking is that it is the earliest pipeline knob you can evaluate with it: chunk three ways, run twenty real questions through each, count how often the answering passage appears in the top 5. Twenty questions and an afternoon will tell you more about your corpus than every chunk-size blog post combined.

Chunking sits inside objective 1.4, "Curate and embed content datasets for RAGs" — chunking is the step between curation and embedding, and it is named as part of the ingestion path in NVIDIA's own RAG description alongside overlap and reranking. [OFFICIAL] on the objective; [NVIDIA-DOC] on the pipeline description. Objective 1.3, "Build LLM use cases such as retrieval-augmented generation (RAG), chatbots, and summarizers," is served because you cannot build the use case without making this decision. Objective 1.6, "Familiarity with the capabilities of Python natural language packages (spaCy, NumPy, vector databases, etc.)," is why sentence segmentation matters: splitting text into sentences reliably is a classical-NLP capability, and spaCy is the named package that does it. [OFFICIAL]

Question phrasings to expect:

• "What is the main trade-off when increasing chunk size in a RAG system?" — the keyed answer names more context per chunk against less precise retrieval, or equivalently a blurrier embedding.
• "Why is overlap used between chunks?" — so that content spanning a boundary appears intact in at least one chunk.
• "A model with a 128k context window is used in a RAG pipeline. What chunk size should be used?" — the correct reasoning is that the context window does not determine chunk size. Watch for an option that says "128k" or "as large as the context window allows".
• "Which chunking strategy splits on paragraph, then sentence, then word boundaries?" — recursive.
• "Which chunking strategy requires computing embeddings during ingestion?" — semantic.
• "A RAG system retrieves chunks that begin mid-sentence and answers are frequently incomplete. What change is most likely to help?" — move from fixed-size to a boundary-respecting strategy, and/or add overlap.
• "After changing the chunk size, what else must be done?" — re-embed and re-index the whole corpus. Chunk size is not a query-time parameter.

Distractor families:

That last row deserves emphasis because it is another silent failure in the family established in 06-01. Embedding models have a maximum input length of their own, and it is typically far smaller than an LLM's context window. Hand an embedding model more tokens than it accepts and the usual behaviour is truncation, not an error — so the last portion of every oversized chunk contributes nothing to its vector while remaining in the text the model eventually reads. The chunk retrieves as if it were shorter than it is. Checking your chunk size against your embedding model's input limit is a one-line assertion that removes a whole class of confusion, and it is a genuine constraint from the embedding model, unlike the context window, which is not.

There is no correct answer independent of your corpus, your queries, and your embedding model, and any source that gives you one without those three has told you about their corpus rather than yours. What can be said honestly is the direction of the trade and how to find your own answer.

The trade-off, stated cleanly:

The procedure that actually answers the question for your case:

1. Freeze a set of at least 20 real questions and, for each, note by hand which passage in the corpus answers it — the discipline from 01-08 and 03-04.
2. Chunk the corpus three ways: a small setting, a medium setting, and a large setting, roughly a factor of two or three apart. Use recursive splitting for all three so you are varying one thing.
3. Embed and index each version separately. Do not mix them in one index.
4. For each question, retrieve top 5 from each index and record whether the answering passage is present and intact.
5. Pick the winner. Re-run this whenever the corpus composition or the embedding model changes.

Two cheap heuristics worth stating as heuristics rather than rules. First, a chunk should usually be able to answer a question by itself — if you read the chunk cold and cannot tell what it is about, it is too small or missing its heading. Second, a chunk should usually be about one thing — if you can write two unrelated summary sentences for it, it is too large. Both are judgement calls, both are [FIELD]-grade rules of thumb, and both are more useful than a number, because they transfer between corpora and a number does not.

Sometimes, and less than people assume. Overlap exists to solve one specific problem: an answer that straddles a chunk boundary, so neither chunk contains it whole. Overlap guarantees the straddling span exists intact in at least one chunk, provided the overlap is longer than the span.

Its costs are concrete. Storage and embedding compute scale by roughly overlap / (size − overlap). Duplicated text produces chunks that are near-duplicates of one another, and when a query matches the overlapped region, several near-identical chunks can occupy your top-k slots — spending prompt budget on the same sentences repeated. That is a real degradation, and it is why 06-04 treats near-duplicate chunks as a retrieval problem and not only a storage one.

The honest position: use a modest overlap by default with a boundary-respecting strategy, and treat larger overlaps as something to justify with measurement. If you are chunking on real structure — one chunk per section, per clause, per table — overlap is often unnecessary, because the boundaries were chosen where meaning genuinely breaks rather than where a counter hit 512.

Not as a first move. Recursive chunking is nearly free, deterministic, has no model dependency, and captures most of the available benefit for any document with paragraph structure. Semantic chunking adds an embedding pass over every sentence at ingestion, a similarity threshold you must tune, and a dependency that makes your chunk boundaries a function of your embedding model — so upgrading the model changes your corpus partition and forces a full re-index. See 12-12 for why index migration is a project rather than a config change.

Semantic chunking earns its cost in a narrow band: long unstructured narrative where topic shifts are real but unmarked — interview transcripts, meeting recordings, long-form reporting, OCR output whose paragraph breaks were lost during parsing. In those cases the structural signal recursive chunking depends on genuinely is not there, and inferring boundaries from meaning is the only option left.

The decision rule, compactly:

• Chunk size is granularity; the context window is a ceiling. The window tells you how many chunks fit in a prompt. It never tells you how big a chunk should be.
• The real constraint on chunk size is the embedding model, not the LLM. One vector represents the whole chunk, and that compression degrades with length — plus embedding models have their own hard input limit that truncates silently.
• Fixed-size is blind, recursive respects structure, semantic infers it. Name all three, and know that semantic is the only one that costs an embedding pass at ingestion.
• Recursive chunking is the sensible default for prose: free, deterministic, structure-respecting, no model dependency.
• Chunking cannot repair parsing. A table whose header was flattened in 06-01 stays broken however you cut the text.
• Overlap is a boundary-straddling fix with a real cost, roughly overlap / (size − overlap) in extra storage and embedding compute, plus near-duplicate chunks competing in your top-k.
• Every chunking change requires re-chunking and re-embedding the corpus. It is an ingestion-time decision, never a query-time parameter.
• There is no official optimal chunk size. Two chunks, two heuristics: a chunk should answer a question alone, and a chunk should be about one thing. Then measure on 20 real queries.

Chunking leaves you with a corpus of self-contained-ish text spans, and the "-ish" is the problem the next lesson attacks. A chunk that reads "the limit rises to 90 days for employees in this category" is a perfectly good chunk of a document and a bad standalone answer, because the section heading naming the category, the document it came from, its effective date, and its authority all sat outside the span you cut. Some of that context should be folded into the text that gets embedded, so it influences retrieval; some should be attached alongside as fields that are stored, filtered on, and returned to the reader as a citation but never embedded at all. Those are two different decisions with two different consequences, and mixing them up produces either unsearchable chunks or unciteable answers.

Next: 06-03 Metadata in RAG: what to embed versus what to return — the split between the text that becomes a vector and the fields that travel with it.