Retrieval and chunking: choosing what goes in

TL;DR. Compression shrinks what you decided to send; selection decides it. This chapter is the missing input-side lever: split a corpus into chunks, index them, retrieve only what the query needs, and stop at a token budget. The lab measures the whole trade on this book's own chapters with a from-scratch TF-IDF retriever: recall climbs from 5/12 correct at 400-character chunks to 10/12 at 1,600, then falls as chunks grow (dilution), and a greedy budget fill hits 12/12 with about 2,000 tokens of context, after which every doubled budget buys nothing. The knee, not the window size, is the setting. The same numbers explain the production stack (chunking strategies, embedding retrieval, reranking, hybrid search) and the Claude Code angle: the agent's grep-then-read loop is a retrieval pipeline, and CLAUDE.md, skills, and MCP resources are all selection decisions you are already making.

Contents

• Selection is the fourth compression lever
• Chunking: the unit of retrieval
• The lab: chunk size and budget, measured
• Reading the results
• The production stack above the toy
• Few-shot examples are retrieval too
• Claude Code as a retrieval pipeline
• Further reading
• Takeaways

Chapter 3 compressed a context you had already assembled; Chapter 5 selected code by structure. This chapter generalizes the selection problem to any corpus: documentation, tickets, transcripts, a knowledge base. It is the lever the industry calls RAG (retrieval-augmented generation), and it belongs in the compression family because its output is the same: fewer, better tokens in the window. The difference is where it acts. Compression asks "which of these tokens can go?"; selection asks "which of these documents should never arrive?", and the second question is worth ten of the first, because a document never retrieved costs zero on every turn forever.

Selection is the fourth compression lever

The economics come straight from Chapter 2 and Chapter 17. Whatever you put in the window is re-sent on every later turn of the session, so over-retrieval is not a one-time tax; it compounds. Retrieving 8,000 tokens where 2,000 would answer does not cost 6,000 tokens, it costs 6,000 times the number of turns the session still has to run (discounted by the cache's 0.1x, which helps and does not absolve). That is why the tuning variable in this chapter is a token budget, not a top-k: k is a count of chunks, but the bill and the window are denominated in tokens, and the right k varies with chunk size while the right budget does not.

Chunking: the unit of retrieval

Retrieval can only return what indexing separated, so the chunking decision quietly bounds everything downstream. The strategies, in ascending sophistication:

Two rules dominate the strategy choice. First, a chunk should be self-contained: retrieval returns it without its neighbors, so a chunk that starts "as shown above, this approach..." is dead weight (structural splitting and overlap both exist to fix this). Second, the chunk is what you bill: whatever you index is what a hit drags into the window, which is why the lab measures token cost per retrieval next to recall rather than recall alone.

The lab: chunk size and budget, measured

The corpus is this book's own chapters (one labeled document each); the queries are twelve questions whose answering chapter is known; the retriever is a from-scratch TF-IDF cosine scorer, the bag-of-words cousin of Chapter 7's embedding similarity, chosen so the lab runs with the standard library. Retrieval is judged correct when the top chunk comes from the answering chapter.

"""Retrieval and chunking, measured on this book's own text. Real data.

The selection problem: a corpus is too big for the window, so you split it
into chunks, index them, and retrieve only the ones relevant to the query.
Every design choice (chunk size, how many to retrieve, budget) trades recall
against tokens, and this lab measures that trade on a real corpus: the
chapters of this book.

Setup:
  - CORPUS: every chapter .md in src/, each one a labeled document.
  - QUERIES: 12 questions, each answerable by exactly one known chapter
    (the label). Retrieval is CORRECT if the top-scoring chunk comes from
    that chapter.
  - RETRIEVER: from-scratch TF-IDF cosine similarity (Chapter 7 built the
    embedding version of this; TF-IDF keeps the lab dependency-free and the
    lesson identical: score chunks against the query, take the best).

Two experiments:
  1. CHUNK SIZE sweep: same retriever, same queries, chunks of 400 to
     12,800 characters. Small chunks are precise but fragmentary; huge
     chunks dilute the match AND bill you for everything around the answer.
  2. BUDGET-BASED SELECTION: instead of a fixed top-k, fill a token budget
     greedily by score. Report answer-bearing recall per budget, which is
     the number a context engineer actually tunes.

Standard library + this book's src/ only.
"""

import math
import re
from collections import Counter
from pathlib import Path

SRC = Path(__file__).resolve().parent.parent / "src"

# 12 queries, each labeled with the chapter that contains its answer.
QUERIES = [
    ("what is byte pair encoding and how are tokens merged", "02"),
    ("how do I remove low-information tokens from a long prompt", "03"),
    ("how do I make the model write fewer output tokens", "04"),
    ("select code by call graph instead of whole files", "05"),
    ("what do keys and values store in attention and why cache them", "06"),
    ("return a stored answer for a similar question", "07"),
    ("how does vLLM page KV memory across requests", "08"),
    ("how does an agent store and retrieve facts across sessions", "09"),
    ("summarize old turns when the window fills up", "11"),
    ("what are the four fields of the usage block", "23"),
    ("why did the timestamp in the system prompt break the cache", "24"),
    ("where does Claude Code write session transcripts on disk", "25"),
]

WORD = re.compile(r"[a-z]{3,}")


def tokens_of(text):
    return WORD.findall(text.lower())


def chunk(text, size):
    """Fixed-size character chunks, split on paragraph boundaries where
    possible. Real systems add overlap and structure-awareness; the sweep's
    shape survives those refinements."""
    paras = text.split("\n\n")
    chunks, cur = [], ""
    for p in paras:
        if len(cur) + len(p) > size and cur:
            chunks.append(cur)
            cur = ""
        cur += p + "\n\n"
    if cur.strip():
        chunks.append(cur)
    return chunks


def build_index(chunk_size):
    """(chapter_label, chunk_text, term_counts) for every chunk, plus IDF."""
    entries = []
    df = Counter()
    for f in sorted(SRC.glob("[0-9][0-9]-*.md")):
        label = f.name[:2]
        for c in chunk(f.read_text(), chunk_size):
            tf = Counter(tokens_of(c))
            entries.append((label, c, tf))
            df.update(set(tf))
    n = len(entries)
    idf = {t: math.log(n / d) for t, d in df.items()}
    return entries, idf


def score(query_tf, tf, idf):
    """TF-IDF cosine between query and chunk."""
    dot = sum(q * tf.get(t, 0) * idf.get(t, 0) ** 2 for t, q in query_tf.items())
    nq = math.sqrt(sum((q * idf.get(t, 0)) ** 2 for t, q in query_tf.items()))
    nc = math.sqrt(sum((c * idf.get(t, 0)) ** 2 for t, c in tf.items()))
    return dot / (nq * nc) if nq and nc else 0.0


def ranked(entries, idf, query):
    qtf = Counter(tokens_of(query))
    return sorted(entries, key=lambda e: -score(qtf, e[2], idf))


def experiment_chunk_size():
    print("=== 1. Chunk size vs retrieval quality vs tokens billed ===")
    print(f"{'chunk chars':>12}{'chunks':>8}{'top-1 correct':>15}"
          f"{'~tok/chunk':>12}{'~tok for top-3':>15}")
    for size in (400, 800, 1600, 3200, 6400, 12800):
        entries, idf = build_index(size)
        hits = sum(ranked(entries, idf, q)[0][0] == lbl for q, lbl in QUERIES)
        avg = sum(len(c) for _, c, _ in entries) / len(entries) / 4
        print(f"{size:>12,}{len(entries):>8,}{hits:>11}/12"
              f"{avg:>12,.0f}{3 * avg:>15,.0f}")
    print("""
Read the two ends against the middle. Tiny chunks fragment the answer
(the matching paragraph often lacks the surrounding definition), huge
chunks dilute the term match and triple your bill per retrieval. The
plateau in the middle is why practitioners land near 1-3k characters:
recall stops improving while the token cost keeps climbing.
""")


def experiment_budget():
    print("=== 2. Fill a token budget by score (what you actually tune) ===")
    entries, idf = build_index(1600)
    print(f"{'budget (tok)':>13}{'answer chapter in context':>28}{'avg chunks':>12}")
    for budget in (500, 1_000, 2_000, 4_000, 8_000):
        hits = 0
        total_chunks = 0
        for q, lbl in QUERIES:
            got, spent = [], 0
            for label, c, _ in ranked(entries, idf, q):
                t = len(c) // 4
                if spent + t > budget:
                    break
                got.append(label)
                spent += t
            total_chunks += len(got)
            hits += lbl in got
        print(f"{budget:>13,}{hits:>24}/12{total_chunks / len(QUERIES):>12.1f}")
    print("""
The budget view is the honest one: recall saturates while tokens keep
doubling. Past the knee, every extra retrieved chunk is a token you pay
to re-send on every later turn of the session, for information the task
did not need. Set the budget at the knee, not at the window size.
""")


if __name__ == "__main__":
    experiment_chunk_size()
    experiment_budget()

Running it (the corpus is this book's own chapters, so the exact chunk counts grow as the book does; the shape of the curves is what to read):

=== 1. Chunk size vs retrieval quality vs tokens billed ===
 chunk chars  chunks  top-1 correct  ~tok/chunk ~tok for top-3
         400   1,641          5/12         114            343
         800   1,142          7/12         164            493
       1,600     579         10/12         324            973
       3,200     284         10/12         661          1,983
       6,400     139          9/12       1,350          4,051
      12,800      78          9/12       2,407          7,220

Read the two ends against the middle. Tiny chunks fragment the answer
(the matching paragraph often lacks the surrounding definition), huge
chunks dilute the term match and triple your bill per retrieval. The
plateau in the middle is why practitioners land near 1-3k characters:
recall stops improving while the token cost keeps climbing.

=== 2. Fill a token budget by score (what you actually tune) ===
 budget (tok)   answer chapter in context  avg chunks
          500                      10/12         1.0
        1,000                      11/12         2.6
        2,000                      12/12         5.8
        4,000                      12/12        12.2
        8,000                      12/12        24.2

The budget view is the honest one: recall saturates while tokens keep
doubling. Past the knee, every extra retrieved chunk is a token you pay
to re-send on every later turn of the session, for information the task
did not need. Set the budget at the knee, not at the window size.

Reading the results

• Both ends of the chunk sweep lose, for different reasons. At 400 characters, recall is 5/12: the matching fragment exists but competes with over 1,600 siblings and often misses the query's other terms (fragmentation). At 12,800, recall is 9/12 and each retrieval bills about 2,400 tokens: the chunk contains the answer plus twenty paragraphs of dilution that drag its score down and your bill up. The 10/12 plateau at 1,600 to 3,200 characters is the shape every production team rediscovers.
• The budget experiment is the tuning you should copy. With 1,600-character chunks, a 2,000-token greedy fill gets the answer into context for all twelve queries; 8,000 tokens gets... the same twelve, at four times the compounding cost. This is Chapter 22's lesson arriving from the input side: the knee of the recall curve is the budget, and everything past it is volume the cache must serve and the model must wade through (Chapter 33 shows the wading is not even free in quality terms).
• Measure with labeled queries, always. Twelve questions with known answers turned every opinion in this chapter into a number in fourteen seconds of compute. Before you tune a real pipeline, build the same thing at whatever scale you can afford: a list of (question, document-that-answers-it) pairs. It is the eval that makes chunk size, k, budget, and reranker decisions boring instead of ideological.

The production stack above the toy

The from-scratch scorer maps onto the real stack layer by layer:

• Embedding retrieval replaces TF-IDF with dense vectors (Chapter 7 built one; Chapter 9 uses it for memory). It wins on paraphrase ("cut my bill" matching "cost reduction") and loses on exact identifiers, which is why hybrid search (dense + BM25 keyword, scores fused) is the production default; the from-scratch BM25 is one formula away from this lab's TF-IDF.
• Reranking runs a slower, better model over the top 30-100 candidates and reorders them before the budget fill. It is the cheapest quality upgrade in the stack because it never touches the index, and it pairs with a deliberately generous first-stage k.
• The vector index at scale is this book's other series: HNSW and IVF-PQ are the approximate-nearest-neighbor structures under every vector database (their from-scratch treatments are the hnsw and ivf-pq books on this site).
• The projects: LlamaIndex and LangChain's text splitters implement every chunking row in the table above; Chroma, Qdrant, Weaviate, pgvector, and Milvus are the index; Cohere and open cross-encoder models are the rerankers. The landscape rule applies unchanged: name the lever first (splitter, index, reranker, budget policy), then pick the tool, and keep the labeled-query eval, because it transfers across all of them.

Few-shot examples are retrieval too

The other thing routinely over-stuffed into prompts is examples. A static block of ten few-shot examples is a selection decision made once, badly, for every future query; the retrieval frame fixes it the same way it fixed documents. Index your example library, retrieve the 2 or 3 most similar to the current input, and spend the freed budget on nothing. Measured pipelines repeatedly find a handful of relevant examples beats a wall of generic ones on both quality and tokens, and the same knee logic applies: past a few examples, accuracy saturates while the per-call bill (and the cache-unfriendly churn of a changing prefix, Chapter 24) keeps growing. If the examples rarely change, they belong before the volatile content, cached; if they are retrieved per query, they are conversation content and should be tiny.

Claude Code as a retrieval pipeline

Claude Code does not ship a vector database, and it is still the most instructive retrieval system in this book, because its selection loop is visible in every transcript:

• Grep-then-read is hybrid search. The agent's Grep is the keyword stage, its choice of which hits to Read is the rerank, and the limit/offset parameters on Read are the budget fill. When Chapter 29's engineered prompt named the file and bounded the reads, it was hand-running this chapter's pipeline with a budget of one chunk.
• CLAUDE.md is the static few-shot block, and its 500-token budget (Chapter 20) is the knee argument applied to instructions.
• Skills are parent-child chunking: the index (name + description) sits in context; the body is fetched only on a hit. MCP's deferred tool loading (Chapter 17) is the same design for tool schemas.
• Adding a real vector store is one claude mcp add away when the corpus outgrows grep (a wiki, a ticket archive, a design-doc trove): the memory servers of Chapter 26 and any vector-DB MCP server slot in as tools, and the budget discipline of this chapter is what keeps their results from bloating the window. Measure the addition like any component: the differential /context and per-session audit from Chapter 30.

Don't be confused. Retrieval and memory (Chapter 9) share machinery (embeddings, similarity, top-k) and differ in what they index. Retrieval indexes a corpus that exists outside the conversation (docs, code, tickets); memory indexes what the conversation itself produced (facts, preferences, decisions). The failure modes differ too: retrieval fails by fetching the wrong passage; memory fails by staleness and contradiction, which is why Chapter 9 spends its pages on invalidation and this chapter spends them on chunking.

Further reading

• Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (arxiv.org): the paper that named the pattern.
• Robertson and Zaragoza, "The Probabilistic Relevance Framework: BM25 and Beyond": the keyword half of hybrid search, one step up from this lab's TF-IDF.
• The splitter documentation of LlamaIndex and LangChain, as catalogs of chunking strategies; the hnsw and ivf-pq books on this site for the index internals.
• Chapter 33: why over-retrieval hurts quality, not just cost.

Takeaways

• Selection precedes compression: a document never retrieved costs zero on every turn. Tune in tokens (a budget), not chunks (a k).
• Measured on this book's corpus: recall 5/12 at 400-char chunks, 10/12 at 1,600 to 3,200, falling past that as dilution sets in; a 2,000-token budget fill reached 12/12 and further doubling bought nothing. Set the budget at the knee.
• Chunks must be self-contained and are what you bill; structural splitting, overlap, and parent-child indexing exist to reconcile search precision with answer completeness.
• The production stack is layers on the same toy: hybrid (dense + BM25) retrieval, a reranker over a generous first stage, a vector index (HNSW/IVF-PQ) at scale, and always a labeled-query eval, which is the only part that transfers across every tool choice.
• Few-shot examples are a retrieval problem with the same knee; retrieve a few relevant ones or cache a small static set, never both worlds' costs.
• Claude Code already runs this pipeline as grep-then-read with bounded reads; skills and deferred tools are parent-child chunking for capabilities, and a vector store joins as an MCP tool priced like any component.

👉 Selection done well fills the window with exactly what the task needs. The next question is how not to pay for those same well-chosen tokens on every later call: the caching family. Continue to KV-cache and prefix caching. (Two deeper questions about what happens once context is in the window, whether the model uses it and whether you can trust it, wait in Context evals and Hostile context.)