Chunking, and why fixed-size is usually wrong

What it is

Chunking splits documents into units that get embedded, indexed and retrieved. The chunk is the atom of a retrieval system: it is what the embedding represents, what BM25 scores, what the reranker orders, and what the generator reads.

Fixed-size chunking cuts every N tokens with some overlap. It is the default in every tutorial and it is wrong for most corpora, for one reason: it cuts where the counter says, not where the meaning ends. A chunk boundary through the middle of a procedure produces two chunks, neither of which answers the question, and neither of which scores well enough to be retrieved.

Commonly confused with the context window problem. Chunking is not primarily about fitting things in the window; frontier windows are large enough to hold most documents whole. It is about retrieval granularity: what unit is small enough to be discriminating and large enough to be self-contained.

The problem it solves

Two competing pressures, and every chunking decision is a point between them.

Too large and the embedding is a blurry average of several topics, so it matches nothing precisely. A 4,000-token chunk covering four subjects has an embedding near the centroid of all four and is beaten in retrieval by a focused chunk on any one of them. You also pay for the whole chunk in context when you needed a paragraph.

Too small and the chunk loses the context that makes it interpretable. "This must be filed within 30 days" is useless without knowing what "this" is. Retrieval finds it; the generator cannot use it.

The chunking strategy is the answer to "what is the smallest self-contained unit in this corpus", and the answer depends on the corpus, which is why a universal default is suspect.

Mechanics

The strategies, in ascending order of how much they know about the document

1. Fixed-size with overlap. Cut every N tokens, overlap by M.

def fixed_size(text, size=512, overlap=64):
    """The default. Works when documents have no structure, which is rare."""
    toks = tokenize(text)
    return [toks[i:i + size] for i in range(0, len(toks), size - overlap)]

Overlap is a hedge against boundary cuts: with 64 tokens of overlap, a fact spanning a boundary appears whole in one of the two chunks. It costs index size and introduces near-duplicates that fill your top-$k$ with the same content.

2. Recursive character splitting. Try to split on the largest natural separator that keeps chunks under the size limit: paragraphs, then sentences, then words. Strictly better than fixed-size for the same cost, and it should be the floor.

3. Structure-aware splitting. Use the document's own markup. Split markdown at headings, HTML at sections, code at function boundaries, and never split a table or a code block.

def structure_aware(markdown, max_tokens=512):
    """Respect the author's structure and carry the heading path into each
    chunk, so an orphaned paragraph still knows what it is about."""
    chunks = []
    for section in split_on_headings(markdown):        # H1/H2/H3 boundaries
        header_path = " > ".join(section.heading_path) # "Leave > Parental > Eligibility"
        body = section.body
        if count_tokens(body) <= max_tokens:
            chunks.append(f"{header_path}\n\n{body}")
        else:
            for para_group in pack_paragraphs(body, max_tokens - count_tokens(header_path)):
                chunks.append(f"{header_path}\n\n{para_group}")  # path repeated
    return chunks

The heading path is the highest-return single line in that function. It is nearly free and it fixes a large fraction of "chunk found but not interpretable".

4. Semantic chunking. Embed sentences, and cut where consecutive-sentence similarity drops below a threshold, on the theory that a topic shift shows up as an embedding discontinuity. Appealing, expensive (an embedding call per sentence at index time), and in published comparisons the gains over good structure-aware chunking are inconsistent. Worth trying, not worth assuming.

5. Parent-document retrieval. Embed and retrieve small chunks; return their larger parent to the generator.

Index:   small chunks (150 tokens) -> precise embeddings, good discrimination
Return:  the parent section (800 tokens) -> full context for generation

This directly resolves the size tension: small for matching, large for comprehension. It is the highest return-to-complexity move in this whole list and it is under-used.

6. Late chunking. Embed the whole document with a long-context embedding model, then pool the token embeddings per chunk. Each chunk's vector is computed with the entire document in attention, so it carries document context without any text duplication. Elegant, requires a long-context embedding model, and is the newer idea here.

7. Contextual retrieval. Prepend a generated one-or-two-sentence description of where the chunk sits in the document, before embedding.

Original chunk:
  "Employees must submit the form within 30 days."

Contextualised:
  "From the Parental Leave Policy (2024), section 4, Eligibility, which covers
   how employees apply for statutory parental leave.
   Employees must submit the form within 30 days."

This costs one LLM call per chunk at index time, which is real money on a large corpus, and prompt caching over the shared document makes it much cheaper than it first appears. Anthropic reported that contextual embeddings reduced top-20 retrieval failure rate by about 35 percent, and about 49 percent combined with contextual BM25, rising to about 67 percent with reranking. Those are the most useful published numbers in this area.

Choosing by corpus

That last row is the point most teams miss: chunking is a per-document-type decision, and a single global strategy over a heterogeneous corpus is guaranteed to be wrong for some of it.

Sizing

The honest answer is "measure", and the useful starting points:

• 256 to 512 tokens for question-answering over documents. Small enough to discriminate, large enough to be self-contained with a heading path attached.
• 10 to 20 percent overlap if you are not using structure-aware splitting; near zero if you are, because natural boundaries already contain the meaning.
• Embedding models have a maximum sequence length, and content beyond it is silently truncated, which is a quiet way to lose the second half of every large chunk. Check the limit.

A worked example

An HR policy corpus. 4,000 documents, markdown with headings. Evaluation set of 180 questions with labelled source documents.

The numbers are the shape of a typical result rather than a published benchmark, and I would say so. The two structural findings they illustrate are reproducible and are the substance of the answer.

First: the heading path is the best ratio in the table. Six points of recall for prepending "Parental Leave Policy > Eligibility" to each chunk. It works because retrieval failures in structured corpora are dominated by orphaned fragments: the right paragraph exists but has no words in it that connect to the question.

Second: look at what is still failing. Of the 16 remaining failures at 0.91:

• 7 spanned a document boundary: the answer required two policies read together. Chunking cannot fix this; query decomposition or a graph can.
• 5 were tables where the retrieved row group had lost its header despite the rule, because the table was split across a page boundary in the source PDF.
• 4 were genuine vocabulary gaps.

None of those are chunking parameters. That is the general lesson: once you are structure-aware with a heading path, further chunking tuning has sharply diminishing returns, and the remaining failures live in ingestion quality, query understanding, or the corpus itself. Teams that keep sweeping chunk sizes after this point are optimising the part that already works.

Production evidence

Anthropic's contextual retrieval write-up is the best public measurement in this area: contextual embeddings cut top-20 retrieval failure rate by roughly 35 percent, contextual embeddings plus contextual BM25 by roughly 49 percent, and adding reranking took the total to roughly 67 percent. It also documents the prompt-caching trick that makes per-chunk contextualisation affordable at corpus scale.

Günther et al., "Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models" (Jina AI, 2024) introduced late chunking and showed gains over naive chunking without duplicating text into each chunk.

LangChain and LlamaIndex both ship recursive character splitting as the recommended default over fixed-size, and both provide structure-aware splitters for markdown, HTML and code, which is a reasonable proxy for community consensus about the floor.

Parent-document retrieval appears in both frameworks as a first-class retriever (sometimes called small-to-big or auto-merging retrieval), which is evidence that the match-small-return-large pattern is standard rather than exotic.

The debate

The credible alternative is not chunking at all: put whole documents in a long-context model and skip retrieval granularity entirely. For a small corpus this is simpler and avoids every failure mode above.

It stops working on cost, because you pay for every token on every request and only prompt caching mitigates it; on the position effect, where relevant content buried in a long context is recalled worse; and on access control, which long context handles by not handling it. It also does not scale past a few hundred thousand tokens of corpus.

The other alternative is semantic chunking, which sounds principled and in practice delivers inconsistent gains over good structure-aware chunking at meaningfully higher index cost. I would try it and I would not assume it.

My position: structure-aware chunking with the heading path prepended, as the floor for any corpus with structure. Add parent-document retrieval, because it resolves the small-versus-large tension directly and costs almost nothing. Add contextual prefixes when you have measured that the corpus needs them and the index cost is acceptable, which the published numbers suggest is often. And dispatch on document type rather than applying one strategy globally, because a table, a function and a policy section have different natural units.

Chunking work is the wrong focus when the failures are elsewhere, and after structure-aware plus heading path they usually are: ingestion quality (a PDF parsed into soup), query understanding, or a corpus that genuinely does not contain the answer. Sweeping chunk sizes past that point is the most common form of retrieval busywork.

Follow-up Q&A

"Why is fixed-size chunking usually wrong?" Because it cuts where the token counter says rather than where the meaning ends, so a procedure or a definition gets split and neither half is retrievable or usable. It also ignores structure the author already provided: headings, sections, function boundaries. Recursive character splitting is strictly better at the same cost and should be the floor; structure-aware splitting is better still where the corpus has structure.

"What is the highest-return change you can make to chunking?" Prepending the heading path to each chunk. It is one line, it costs nothing at query time, and it fixes the dominant failure in structured corpora, which is an orphaned fragment that contains the answer but no words connecting it to the question. Second is parent-document retrieval: embed small for precise matching, return the parent section for comprehension, which resolves the size tension directly.

"How do you choose chunk size?" Measure on a golden set rather than guessing. Starting point is 256 to 512 tokens for document question-answering, with 10 to 20 percent overlap if you are not structure-aware and close to zero if you are. Then check the embedding model's maximum sequence length, because content beyond it is silently truncated and you can lose the second half of every large chunk without any error. Then stop tuning, because past structure-aware plus heading path the returns collapse.

"What is contextual retrieval and is it worth the index cost?" Prepending a generated description of where each chunk sits in its document before embedding, so the chunk carries context it would otherwise lack. Anthropic reported roughly a 35 percent reduction in top-20 retrieval failure rate from contextual embeddings alone, and about 49 percent combined with contextual BM25. The cost is one LLM call per chunk at index time, which prompt caching over the shared document reduces substantially. Worth it for a high-value, relatively stable corpus; questionable for one that churns daily, where you re-pay the cost constantly.

"Your recall is 0.91 and the remaining failures are not chunk-related. Now what?" Categorise them, because the fix is elsewhere. In the corpus I worked through: cross-document questions needing two policies read together, which is query decomposition or a knowledge graph, not chunking; table rows that lost their header during PDF parsing, which is an ingestion fix; and vocabulary gaps, which is a synonym list built from query logs. The general point is that chunking has a ceiling, and past it the remaining work is in ingestion quality and query understanding.

Common misconceptions

The most common is that chunking exists to fit content in the context window. It exists to set retrieval granularity, and modern windows are large enough that the fitting problem is mostly gone.

The second is that more overlap is safer. It inflates the index, creates near-duplicates that consume your top-$k$ with the same content, and is largely unnecessary once you split on natural boundaries.

The third is that one chunking strategy fits a corpus. A mixed corpus of policies, tables and code needs three strategies dispatched on document type, and applying one globally guarantees it is wrong for some of the content.

Interview delivery note

Say this: "Fixed-size is the default and it's usually wrong, because it cuts where the token counter says rather than where the meaning ends, so a procedure gets split and neither half is retrievable. I'd use structure-aware splitting on the document's own headings, and prepend the heading path to every chunk, which is one line and fixes the dominant failure in structured corpora: an orphaned paragraph that contains the answer but nothing connecting it to the question."

Then the two upgrades with their costs: "Parent-document retrieval resolves the size tension directly, embed small for precision and return the parent section for comprehension. And contextual prefixes, where you generate a one-line description of where the chunk sits before embedding, which Anthropic measured at roughly a 35 percent reduction in retrieval failure rate, about 49 percent with contextual BM25 alongside. That costs an LLM call per chunk at index time, which prompt caching makes affordable."

The depth signal is knowing when to stop: "past structure-aware with a heading path, further chunk-size tuning has sharply diminishing returns, and the remaining failures are usually ingestion quality or cross-document questions. Sweeping chunk sizes after that point is optimising the part that already works."

Further reading

• Anthropic, "Introducing Contextual Retrieval" (2024), for the measured failure-rate reductions and the prompt-caching trick that makes it affordable.
• Günther et al., "Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models" (Jina AI, 2024).
• LlamaIndex documentation on auto-merging and small-to-big retrieval, for the parent-document pattern.
• LangChain's text splitter documentation, for the recursive and structure-aware splitters and the reasoning behind recommending recursive over fixed-size.