The tokenizer: where tokens come from

TL;DR. Fifty chapters of this book count tokens; this one builds the machine that makes them. Byte-pair encoding is a loop you can write in an afternoon: merge the most frequent adjacent pair, repeat, and the merge list is the vocabulary. The lab trains 1,200 merges on this book's own chapters and watches token, cache, and context become single tokens, then turns the tokenizer on practical questions: the same 20 records cost 100% as pretty JSON, 81% with one-character keys, 44% as CSV, 37% as TSV; digits shatter (a 17-character number becomes 14 tokens); languages the training corpus lacked cost 2x or more (the documented tokenizer-unfairness effect); and base64 lands at about one token per character, the most expensive way to put bytes in a window. Claude's tokenizer is not public, so the one exact instrument is the API's free count_tokens endpoint; everything else here is the mechanism that explains what it returns.

Contents

Chapter 2 priced the token and every later chapter counted them, mostly through the chars/4 estimate. What none of them said is what a token is, and the omission has a cost: format choices, number-heavy data, non-English text, and encoded blobs all move your bill in ways you cannot predict without knowing how text becomes tokens.

The algorithm, and where it came from

Byte-pair encoding entered NLP through machine translation (Sennrich, Haddow, and Birch, "Neural Machine Translation of Rare Words with Subword Units", 2015), as an answer to the out-of-vocabulary problem: instead of a fixed word list that fails on any new word, learn subword units from data. Training is one greedy loop:

  1. Start with characters (or bytes) as the symbols.
  2. Count every adjacent symbol pair across the corpus.
  3. Merge the most frequent pair into one new symbol, everywhere.
  4. Repeat for N merges. The ordered merge list is the tokenizer.

Encoding replays the same merges greedily on new text. Frequent strings (common words, common code idioms, common JSON keys) get merged all the way to single tokens; rare strings stay fragmented into pieces. That single sentence is the origin of every cost effect in this chapter.

GPT-2 made the alphabet bytes instead of characters, so any UTF-8 string tokenizes with no out-of-vocabulary case, and set the shape modern stacks still use. The other lineage is WordPiece (BERT) and the Unigram model in SentencePiece (Kudo, 2018), which prunes a large candidate vocabulary probabilistically instead of growing one greedily; different training, same economics. Production vocabularies grew from GPT-2's ~50k entries to the ~100k and ~200k classes of current OpenAI tokenizers and the 128k of Llama 3, because a bigger vocabulary means more strings compress to one token, which is cheaper inference for the provider and shorter effective prompts for you.

Don't be confused. The tokenizer is not part of the model's intelligence; it is a lossless compression codec agreed between you and the model, frozen before training. The model never sees your characters, only token ids, which is why token boundaries have visible fingerprints in behavior (arithmetic on digit chunks, spelling questions, rhyming) and why "the same" prompt can cost different token counts on different vendors: different codecs, same text.

The lab: a tokenizer trained on this book

"""BPE from scratch, trained on this book, then pointed at your data formats.

Every chapter of this book counts tokens; this lab builds the thing that makes
them. Byte-pair encoding (Sennrich et al., 2016), the algorithm under GPT-2's
tokenizer and its descendants, is two loops:

  TRAIN   count adjacent symbol pairs across a corpus, merge the most frequent
          pair into a new symbol, repeat N times; the merge list IS the vocab
  ENCODE  split text into words, then greedily apply the learned merges in
          training order until no merge applies

The lab trains 1,200 merges on this book's own chapters, checks the chars-per-
token ratio this book's /4 heuristic rests on, then measures what the SAME 20
records cost in seven serialization formats, what digits do, what happens to
languages the corpus did not contain, and what base64 costs. A 1,200-merge
vocabulary is tiny next to a production 100k-200k one, so treat magnitudes as
directional and the mechanisms as exact.

Standard library only. Run:  python3 bpe_lab.py
"""

import base64
import json
import re
from collections import Counter
from pathlib import Path

# ---------------------------------------------------------------------------
# TRAIN: merge-by-merge, on this book's own text.
# ---------------------------------------------------------------------------
SRC = Path(__file__).resolve().parent.parent / "src"
train_files = sorted(SRC.glob("[0-3]*.md"))[:30]
corpus = "".join(f.read_text(errors="replace") for f in train_files)[:250_000]

def to_words(text):
    """Whitespace-prefix marker, SentencePiece-style: ' the' != 'the'."""
    return ["▁" + w for w in text.split()]

word_freq = Counter(to_words(corpus))
words = [[list(w), f] for w, f in word_freq.items()]

pair_counts = Counter()
pair_where = {}
for idx, (syms, freq) in enumerate(words):
    for pair in zip(syms, syms[1:]):
        pair_counts[pair] += freq
        pair_where.setdefault(pair, set()).add(idx)

def merge_word(syms, pair, joined):
    out, i = [], 0
    while i < len(syms):
        if i < len(syms) - 1 and (syms[i], syms[i + 1]) == pair:
            out.append(joined)
            i += 2
        else:
            out.append(syms[i])
            i += 1
    return out

N_MERGES = 1200
merges = []
for _ in range(N_MERGES):
    if not pair_counts:
        break
    best = max(pair_counts, key=pair_counts.get)
    joined = best[0] + best[1]
    merges.append(best)
    for idx in list(pair_where.get(best, ())):
        syms, freq = words[idx]
        for pair in zip(syms, syms[1:]):          # retire old pairs
            pair_counts[pair] -= freq
            if pair_counts[pair] <= 0:
                del pair_counts[pair]
            pair_where[pair].discard(idx)
        syms = merge_word(syms, best, joined)
        words[idx][0] = syms
        for pair in zip(syms, syms[1:]):          # register new pairs
            pair_counts[pair] += freq
            pair_where.setdefault(pair, set()).add(idx)

rank = {pair: i for i, pair in enumerate(merges)}

def encode_word(word, cache={}):
    if word in cache:
        return cache[word]
    syms = list(word)
    while len(syms) > 1:
        pairs = list(zip(syms, syms[1:]))
        best = min(pairs, key=lambda p: rank.get(p, 1 << 30))
        if best not in rank:
            break
        syms = merge_word(syms, best, best[0] + best[1])
    cache[word] = syms
    return syms

def tokens(text):
    return sum(len(encode_word(w)) for w in to_words(text))

print(f"=== Trained {len(merges)} merges on {len(corpus):,} chars of this book ===")
print("first 12 merges learned:", " ".join(a + "+" + b for a, b in merges[:12]).replace("▁", "_"))
domain = [m[0] + m[1] for m in merges if (m[0] + m[1]).lstrip("▁") in
          ("token", "tokens", "cache", "context", "prompt", "model", "memory")]
print("domain words that became single tokens:",
      " ".join(sorted(set(domain))).replace("▁", "_"))

held_out = (SRC / "40-static-facts-ledger.md").read_text()
print(f"\nheld-out chapter 40: {len(held_out):,} chars -> {tokens(held_out):,} tokens "
      f"= {len(held_out) / tokens(held_out):.2f} chars/token")
print("(the book's chars/4 heuristic, grounded; production vocabs land near 4)")

# ---------------------------------------------------------------------------
# ENCODE: the same 20 records, seven formats.
# ---------------------------------------------------------------------------
RECORDS = [dict(name=f"tool-{i:02d}", org=f"org-{i % 7}", stars=1000 + 137 * i,
                license="Apache-2.0" if i % 3 else "MIT") for i in range(20)]

def yamlish(rs):
    return "".join(f"- name: {r['name']}\n  org: {r['org']}\n"
                   f"  stars: {r['stars']}\n  license: {r['license']}\n" for r in rs)

def csvish(rs, sep=","):
    head = sep.join(RECORDS[0])
    return head + "\n" + "\n".join(sep.join(str(v) for v in r.values()) for r in rs)

def mdtable(rs):
    return ("| name | org | stars | license |\n|---|---|---|---|\n"
            + "\n".join(f"| {r['name']} | {r['org']} | {r['stars']} | {r['license']} |"
                        for r in rs))

short = [dict(zip("nosl", r.values())) for r in RECORDS]
formats = [
    ("JSON, indent=2",       json.dumps(RECORDS, indent=2)),
    ("JSON, minified",       json.dumps(RECORDS, separators=(",", ":"))),
    ("JSON, 1-char keys",    json.dumps(short, separators=(",", ":"))),
    ("YAML",                 yamlish(RECORDS)),
    ("CSV",                  csvish(RECORDS)),
    ("TSV",                  csvish(RECORDS, "\t")),
    ("Markdown table",       mdtable(RECORDS)),
]
print("\n=== The same 20 records, seven formats ===")
print(f"{'format':<18}{'chars':>7}{'tokens':>8}{'vs JSON indent=2':>18}")
print("-" * 51)
base_t = tokens(formats[0][1])
for name, text in formats:
    t = tokens(text)
    print(f"{name:<18}{len(text):>7,}{t:>8,}{t / base_t:>17.0%}")
print("-" * 51)

# ---------------------------------------------------------------------------
# Where tokenization surprises live: digits, other languages, base64.
# ---------------------------------------------------------------------------
print("\n=== Surprises ===")
num = "1234567890.250128"
print(f"digits: '{num}' -> {encode_word(chr(0x2581) + num)!r}".replace("▁", "_"))

sentences = [
    ("English (in-domain)",  "the cache invalidates the prefix"),
    ("German",               "der Cache invalidiert das Praefix"),
    ("Finnish",              "valimuisti mitatoi etuliitteen"),
]
for name, s in sentences:
    print(f"{name:<22} {tokens(s):>3} tokens for {len(s)} chars")

blob = base64.b64encode(bytes(range(256)) * 4).decode()
print(f"base64 of 1 KB of bytes: {len(blob):,} chars -> {tokens(blob):,} tokens "
      f"({len(blob) / tokens(blob):.2f} chars/token)")

print("""
Lessons: frequent strings become single tokens, so the format that repeats
long keys (pretty JSON) pays for them once per record while CSV/TSV pay for
the header once; digits fragment; text unlike the training corpus costs
multiples (the tokenizer-unfairness effect across languages); and base64
defeats merging almost entirely, the most expensive way to put bytes in a
window. Real tokenizers soften the magnitudes with 100k+ vocabularies and
byte fallback, but every direction here survives, and the only exact counter
for Claude is the API's count_tokens endpoint.""")

Verified output:

=== Trained 1200 merges on 250,000 chars of this book ===
first 12 merges learned: _+t h+e _+a r+e _t+he i+n o+n _+s e+r _+c e+n _+i
domain words that became single tokens: cache context token tokens _cache _context _memory _model _prompt _token _tokens

held-out chapter 40: 16,863 chars -> 5,859 tokens = 2.88 chars/token
(the book's chars/4 heuristic, grounded; production vocabs land near 4)

=== The same 20 records, seven formats ===
format              chars  tokens  vs JSON indent=2
---------------------------------------------------
JSON, indent=2      1,933     946             100%
JSON, minified      1,332     886              94%
JSON, 1-char keys   1,032     766              81%
YAML                1,251     624              66%
CSV                   573     417              44%
TSV                   573     353              37%
Markdown table        801     471              50%
---------------------------------------------------

=== Surprises ===
digits: '1234567890.250128' -> ['_1', '2', '3', '4', '56', '7', '8', '9', '0.', '25', '0', '1', '2', '8']
English (in-domain)      8 tokens for 32 chars
German                  16 tokens for 33 chars
Finnish                 18 tokens for 30 chars
base64 of 1 KB of bytes: 1,368 chars -> 1,328 tokens (1.03 chars/token)

Lessons: frequent strings become single tokens, so the format that repeats
long keys (pretty JSON) pays for them once per record while CSV/TSV pay for
the header once; digits fragment; text unlike the training corpus costs
multiples (the tokenizer-unfairness effect across languages); and base64
defeats merging almost entirely, the most expensive way to put bytes in a
window. Real tokenizers soften the magnitudes with 100k+ vocabularies and
byte fallback, but every direction here survives, and the only exact counter
for Claude is the API's count_tokens endpoint.

Reading the results

  • The first merges are the language's skeleton. _t, he, _the, in: BPE rediscovers English frequency order in its first dozen steps, unsupervised, exactly as it did in 2015.
  • The corpus becomes the vocabulary. Trained on this book, _token, _cache, _context, _prompt, and _memory all become single tokens within 1,200 merges. The production version of this effect: code-heavy training corpora are why def, return, };, and four spaces of indentation are cheap in every modern tokenizer, and why prose about Kubernetes costs less than prose about your company's internal product names.
  • The chars/4 heuristic is a vocabulary-size statement. Our 1,456-symbol vocabulary reaches 2.88 chars/token on held-out text; production vocabularies two orders of magnitude larger do better (Llama 3's report gives 3.94 characters per token for English, and Anthropic's own glossary says a Claude token is "approximately 3.5 English characters"). Same curve, further along, and note the book's round /4 slightly undercounts tokens against Anthropic's 3.5 figure. When precision matters, stop estimating and count (last section).

The format bill

The middle table is the one to keep. The same 20 records, identical information, spans 2.7x between the most and least expensive serialization, and the ranking follows directly from the algorithm:

  • Pretty JSON pays per record for what CSV pays for once. Every record repeats "name":, "org":, "stars":, "license": plus quotes, braces, and indentation; CSV and TSV state the keys once in a header. Repeated keys do get merged into cheap tokens, which is why minifying saves less than the char count suggests (94% of pretty for 69% of the characters), but cheap is not free when multiplied by every record of every tool result of every turn.
  • Shorter keys help less than fewer keys. One-character keys save 13 points here; moving to a header-based format saves 50. If a tool result is tabular, make it a table: Chapter 3's tool-output compressors and Chapter 35's RTK are doing exactly this transformation, and now you can see why it works at the token level rather than just the character level.
  • This applies to what you emit, too. Structured outputs (Chapter 37) and tool schemas are billed as output and input respectively; a schema that returns arrays of rows instead of arrays of objects is the same 2x, on the 5x-priced side (Chapter 47).

The honest caveat: magnitudes shift with the tokenizer (a 200k vocabulary merges more of JSON's syntax away than our toy does), so treat the ranking as durable and re-measure the gaps with count_tokens before making a decision that depends on them.

The surprises: digits, languages, base64

Digits fragment. Our toy splits a 17-character number into 14 tokens, and production tokenizers fragment by rule: the current OpenAI encodings and Llama 3 cap digit runs at three characters in their pre-tokenization regex (\p{N}{1,3}), and the original LLaMA went further, splitting "all numbers into individual digits" by design (it helps arithmetic). So long ids, timestamps, and high-precision floats are consistently more token-dense than the prose around them. The engineering consequence: a column of ids or metrics can cost more than the sentence describing it; round floats, shorten ids, and prefer names over numbers when either would do.

Languages are not priced equally. Our English-trained toy charges German 2x and Finnish 2.25x per sentence of equal meaning. The production effect is documented as tokenizer unfairness (Petrov et al., NeurIPS 2023): the same content translated across languages can differ in tokenization length "up to 15 times," with everything that follows for cost, latency, and effective window size. If your users write in Thai, Hindi, or Finnish, your per-conversation budget is not what your English tests measured.

Base64 defeats the codec. One token per character, because uniform random-looking strings contain no frequent pairs to merge. A kilobyte of bytes becomes 1,368 characters becomes ~1,300 tokens: the same kilobyte as English prose would have been ~250 tokens described, or better, referenced by path and read by a tool. Never inline encoded blobs; this is also Chapter 34's argument for the Files API over base64 attachments, now with the mechanism visible.

Counting Claude's tokens exactly

Anthropic has not published a tokenizer for Claude 3 and later models (the legacy TypeScript tokenizer repo says plainly that its algorithm "is no longer accurate" for them), so nothing in this chapter, in tiktoken, or in any third-party "Claude token counter" counts current Claude tokens exactly. Two official numbers exist and are worth memorizing: a Claude token is roughly 3.5 English characters (the glossary figure), and models from Opus 4.7 onward, including Fable 5 and Sonnet 5, use a newer tokenizer that produces about 30% more tokens for the same text than earlier models, so cross-model cost comparisons must recount, not assume. The exact instrument is the API's token counting endpoint: POST /v1/messages/count_tokens takes the same shape as a Messages call (system, messages, tools) and returns the input token count without running the model, free to use within its own rate limits. Follow-along, output illustrative:

# Follow-along: requires the anthropic SDK and an API key.
import anthropic

client = anthropic.Anthropic()
count = client.messages.count_tokens(
    model="claude-opus-4-8",
    system="You are a terse assistant.",
    messages=[{"role": "user", "content": open("records.json").read()}],
)
print(count.input_tokens)   # the exact bill for this prompt, before paying it

That endpoint is how you turn this chapter's rankings into decisions: render the same data both ways, count both, ship the cheaper one. Inside Claude Code, the coarse equivalents are /context (what occupies the window) and the transcript's usage blocks (Chapter 23), which report the same accounting after the fact.

Remember. The tokenizer is a frequency mirror: what the training corpus said often is cheap, and everything else is expensive in proportion to its strangeness. You cannot change the codec, but you choose what to feed it: formats that repeat less, numbers that say no more than needed, references instead of blobs. Those choices compound through every turn (Chapter 2) and every cache write (Chapter 6).

Further reading

  • Sennrich, Haddow, Birch, "Neural Machine Translation of Rare Words with Subword Units" (2015): the BPE paper; short and readable.
  • Kudo, "Subword Regularization" (2018) and the SentencePiece toolkit: the Unigram alternative most non-OpenAI stacks use.
  • Petrov, La Malfa, Torr, Bibi, "Language Model Tokenizers Introduce Unfairness Between Languages" (2023): the cross-language cost measurements.
  • openai/tiktoken: the reference fast BPE implementation; reading its _educational module is the production version of this chapter's lab.
  • Anthropic token counting docs (platform.claude.com/docs/en/build-with-claude/token-counting): the exact-count endpoint used above.

Takeaways

  • BPE is merge-the-most-frequent-pair, repeated; the merge list is the vocabulary, and encoding replays it. Frequent strings become single tokens; everything else fragments.
  • Trained on this book, the algorithm makes token, cache, and context single tokens and reaches 2.88 chars/token with a toy vocabulary; production vocabularies reach ~3.5 to 4 characters per token on English (Anthropic's official figure is ~3.5), which is all the chars/4 heuristic ever was.
  • Format is a 2.7x lever on identical data: pretty JSON 100%, minified 94%, one-char keys 81%, YAML 66%, markdown table 50%, CSV 44%, TSV 37%. Fewer repeated keys beat shorter keys.
  • Digits fragment, unfamiliar languages cost multiples (tokenizer unfairness), and base64 is ~1 token/char: round your numbers, budget per language, never inline blobs.
  • No public tokenizer exists for current Claude models, and Opus 4.7+/Fable 5/Sonnet 5 use a newer one that yields ~30% more tokens than earlier models; count_tokens is free and exact. Estimate with chars/4, decide with the endpoint, recount when you switch models.

👉 Tokens are made; next, they are processed, and the two phases of that processing have almost nothing in common. One is a parallel matrix multiply, the other a memory-bound crawl, and the gap between them is why your bill prices output five times above input. Continue to Prefill and decode.