The token economy

TL;DR. A token is a learned sub-word piece, not a word, and both your bill and your context window are counted in tokens. Output tokens cost about 5x input tokens on the Anthropic models, so when you decide what to optimize, weight a saved output token as five saved input tokens. Estimate with words * 1.3 for a gut check, but count with the provider's own count_tokens and the exact model id before any decision touches a budget.

Contents

What a token is
How byte pair encoding builds the vocabulary
Why code, JSON, and non-English cost more
Output is the expensive half
Counting tokens for real
Pricing a context before you send it
Further reading
Takeaways

Every technique in this book is justified by a number: tokens saved, dollars saved, milliseconds saved. So before the techniques, we need to be able to count. This chapter covers what a token is, why output tokens cost several times more than input tokens (which quietly decides what is worth optimizing), and how to estimate a context's size and price before you send it. It is the measuring tape for everything in Chapter 1: once you can put a price on a context, "this feels bloated" turns into a number you can act on.

What a token is

Models do not read characters or words. They read tokens: chunks of text drawn from a fixed vocabulary, usually sub-word pieces. A token is just an integer id that the model looks up in an embedding table, so the first thing any model does with your text is chop it into these ids. The chopping is done by a separate piece of software called the tokenizer, which ships with the model and is frozen the day the model is trained. Two facts follow from that, and both matter for the rest of this book.

First, the vocabulary is finite, typically on the order of 100,000 to 200,000 entries. Every possible input has to be expressed using only those entries, so anything not already in the vocabulary gets built up from smaller pieces that are. Common words are a single token (context), rarer or longer words split into several (contextual becomes context + ual), and unusual strings fall apart into many small pieces, down to individual characters or even individual bytes in the worst case. There is always a fallback: every single byte has its own token, so no input is ever un-encodable, only expensive.

Second, the split is learned from data, not derived from a rule like "split on whitespace." The model never sees the word contextual; it sees the id for context followed by the id for ual, and it learns during training that those two ids tend to appear together. So the unit the model actually reasons over is the token, and the unit you are billed for is the token. Your intuitions about "length" are about words and characters, which is exactly why they mislead you about cost.

This matters for context engineering because your bill and your window are measured in tokens, not words or characters, and the ratio between them is not fixed. English prose runs about 1.3 tokens per word; code, JSON, and non-English text run higher because they fragment more (we will see why in a moment). That variability is the whole reason you cannot eyeball a budget reliably, and the whole reason every provider ships a count_tokens endpoint.

How byte pair encoding builds the vocabulary

The algorithm that learns the split is byte pair encoding (BPE). It is worth understanding in detail, because once you see how the vocabulary is built you can predict which of your own strings will be cheap and which will be expensive, without running anything.

BPE has a training phase (run once, by the model maker, to build the vocabulary) and an encoding phase (run every time you send a request, to chop your text into ids). The demo at the end of this chapter implements both from scratch in a few dozen lines.

Training. Start with a large corpus of text and a starting vocabulary of just the individual characters (or bytes). Now repeat one step over and over:

Count every adjacent pair of current tokens across the whole corpus.
Find the single most frequent pair.
Merge it into one new token, add that token to the vocabulary, and record the merge in an ordered list (the first merge is rank 0, the next is rank 1, and so on).

Each pass adds exactly one token to the vocabulary, so after $N$ passes you have $N$ merges plus the original characters. Because you always merge the most frequent pair available, the merges that happen early are the ones that pay off most: pairs that occur constantly in your corpus get fused first. If your corpus is English text, the pair t + h is merged almost immediately because th is everywhere; later merges fuse th + e into the, and so on up to whole common words. Rare sequences never become frequent enough to win a round, so they never get a merge and stay shattered into small pieces.

The output of training is two things: the vocabulary (the set of tokens) and the ordered merge list (which pair becomes which token, and in what order). The order is the important part, which the next phase explains.

Encoding. To tokenize a new word you do not re-run the counting. You replay the merge list in rank order. Start with the word as a sequence of single characters, then walk the merge list from rank 0 downward: whenever the current pair-to-merge appears in your sequence, apply it. Lower-ranked (earlier, more frequent) merges always win, so encoding is deterministic and fast. A word the tokenizer "knows well" collapses to one or two tokens because there is a chain of merges that reaches it; a word it has never effectively seen bottoms out as a pile of character tokens because no merge in the list applies.

This is the mechanism behind the rule of thumb. A word that was common in the tokenizer's training corpus has a merge path and costs one token. A rare identifier such as a UUID, a hash, or a made-up variable name has no merge path and fragments into many. The tokenizer is not being clever or dumb about your specific text; it is mechanically replaying a fixed list of merges that was frozen long before it ever saw your input.

The demo makes this concrete. After learning just twelve merges on a tiny corpus, the word context has been fused into a single token, while token itself, which never appeared in that corpus, stays shattered into five character tokens. Read that output closely when you reach it: it is the entire BPE idea in four lines.

Why code, JSON, and non-English cost more

The "1.3 tokens per English word" figure is an average over ordinary prose, which is exactly what tokenizers are mostly trained on. Three kinds of content reliably break it, and knowing why lets you predict your bill instead of being surprised by it.

Code. Identifiers like getUserById or kvCacheBlockSize are camel-cased or underscore-joined compounds that rarely appear verbatim in a training corpus, so they fragment. Punctuation-heavy syntax ({, }, =>, ::, indentation) adds tokens that prose does not. A line of code can easily run 2 or more tokens per "word."
JSON. Every key is repeated on every object, and the structural characters ({, }, [, ], ", :, ,) are all tokens. A record with ten fields pays for those ten key strings plus a fixed tax of braces and quotes, every single time it appears. This is why dumping a large JSON blob into a prompt is one of the most expensive things you can do per unit of actual information, and why Chapter 3 spends time on trimming structural redundancy.
Non-English text. Tokenizers are trained predominantly on English, so English gets the most merges and the best compression. Other languages, especially those in non-Latin scripts, fall back to short pieces or raw bytes more often. The same sentence can cost noticeably more tokens in one language than in another, purely as an artifact of what the tokenizer was trained on.

The practical consequence: the moment your context contains code, structured data, or non-English text, stop trusting the 1.3 multiplier and count for real. The purpose of understanding all this is not trivia. It is so you can do three concrete jobs. Budgeting: put a defensible dollar figure on a feature before you ship it. Choosing what to optimize: know whether your spend is dominated by a fat JSON payload, a long system prompt, or chatty output, so you attack the right one. Capacity planning: size a context window and a rate-limit budget against real token counts rather than a guess that is off by 30 percent on exactly the inputs you care about.

Output is the expensive half

Here is the fact that reshapes what you optimize: you are billed for both the tokens you send (input) and the tokens the model writes (output), but output is priced several times higher than input. On the Anthropic models, output is exactly 5x the input rate. On Opus 4.8 input is $5 per million tokens and output is $25; on Sonnet 4.6 it is $3 and $15; on Haiku 4.5 it is $1 and $5. The ratio holds across the lineup.

Why is output dearer? Input tokens are processed in one parallel forward pass (the model reads the whole prompt at once), but output tokens are produced one at a time, each requiring a fresh forward pass that attends back over everything generated so far. Output is sequential and compute-heavy in a way input is not, and the price reflects that.

The strategic consequence has to be stated precisely, because it is easy to get half-right. Per token, an output token is worth five input tokens, so when you are deciding where to spend optimization effort, weight a saved output token as five saved input tokens. But the total bill depends on the input:output ratio you actually run. A long-prompt, short-answer workload (say 8,000 input tokens and 400 output tokens per call) is input-heavy in total even though each output token is pricier, simply because there is 20x more input. So the rule is not "output dominates the bill"; the rule is "weight each saved output token as five, then multiply by how many of each you actually have." Get this right and you stop optimizing the wrong half.

Remember. Output is about 5x input per token on the Anthropic models. When you choose what to trim, count a saved output token as five saved input tokens, then multiply by the real counts. And whenever a number touches a budget, count it with the provider's own count_tokens for the exact model you will call, never an estimate or a foreign tokenizer.

The demo below shows three things from that one fact. First, the 5x asymmetry across models. Second, a worked workload: token for token, cutting output is worth 5x cutting input, even though in a long-prompt/short-answer workload the input still dominates the total bill (the asymmetry is per token, not per call). Third, a from-scratch BPE tokenizer so "sub-word merging" is concrete rather than a phrase, plus the two practical estimates you can apply in your head.

"""The token economy: why output is the expensive half, and how to estimate cost.

Two ideas, both measurable:
  1. You are billed per token, and OUTPUT tokens cost several times more than INPUT
     tokens. So trimming what the model WRITES often saves more than trimming what
     you SEND.
  2. You can estimate a prompt's token count before you send it. Real tokenizers use
     sub-word merging (BPE); we show a tiny BPE to demystify it, plus the practical
     rule of thumb. (For exact counts, call the provider's count_tokens endpoint;
     never use another vendor's tokenizer, it will be wrong.)

Standard library only. Run:  python3 token_economy.py
"""

from collections import Counter

# Published Anthropic prices, US dollars per MILLION tokens (input, output).
# These anchor the asymmetry; verify current numbers before quoting them anywhere.
PRICES = {
    "claude-opus-4-8":   (5.0, 25.0),   # 1M context
    "claude-sonnet-4-6": (3.0, 15.0),   # 1M context
    "claude-haiku-4-5":  (1.0,  5.0),   # 200K context
}


def cost(model, in_tok, out_tok):
    pin, pout = PRICES[model]
    return (in_tok / 1e6) * pin + (out_tok / 1e6) * pout


print("=== 1. Output is the expensive half ===")
print("Per-token, output costs 5x input on every model here:\n")
for m, (pin, pout) in PRICES.items():
    print(f"  {m:18s} input ${pin:>5.2f}/Mtok   output ${pout:>5.2f}/Mtok   "
          f"output is {pout/pin:.0f}x")
print()

# A support agent: a big stable prompt in, a short answer out, called a lot.
in_tok, out_tok, calls = 8000, 400, 100_000
m = "claude-opus-4-8"
base = cost(m, in_tok, out_tok) * calls
print(f"Workload: {calls:,} calls, {in_tok} input + {out_tok} output tokens each "
      f"({m}).")
print(f"  total cost: ${base:,.0f}\n")
# Per-token, trimming OUTPUT is worth 5x trimming the same count of INPUT.
save_out = (cost(m, in_tok, out_tok) - cost(m, in_tok, out_tok - 100)) * calls
save_in = (cost(m, in_tok, out_tok) - cost(m, in_tok - 100, out_tok)) * calls
print(f"  cut 100 OUTPUT tokens/call:  saves ${save_out:,.0f}")
print(f"  cut 100 INPUT  tokens/call:  saves ${save_in:,.0f}")
print(f"  -> token for token, output is worth {save_out/save_in:.0f}x as much to cut.")
print("  (Caching makes input cheaper still, ~0.1x on a hit, widening the gap;")
print("   see chapter 6. Whether INPUT or OUTPUT dominates your bill depends on the")
print("   ratio: a long prompt with a short answer is input-heavy in TOTAL, even")
print("   though each output token is pricier.)\n")


def bpe_encode(word, merges):
    """Encode one word with a learned merge list (the heart of GPT/Claude-style
    tokenizers). Start from characters, then repeatedly glue the highest-priority
    adjacent pair until no learned merge applies."""
    toks = list(word)
    while True:
        pairs = [(toks[i], toks[i + 1]) for i in range(len(toks) - 1)]
        ranked = [(merges[p], i) for i, p in enumerate(pairs) if p in merges]
        if not ranked:
            return toks
        _, i = min(ranked)  # apply the earliest-learned (highest priority) merge
        toks[i:i + 2] = ["".join(toks[i:i + 2])]


def learn_merges(corpus, n):
    """Learn n merges greedily: repeatedly fuse the most frequent adjacent pair."""
    seqs = [list(w) for w in corpus.split()]
    merges = {}
    for rank in range(n):
        pairs = Counter()
        for s in seqs:
            for i in range(len(s) - 1):
                pairs[(s[i], s[i + 1])] += 1
        if not pairs:
            break
        best = pairs.most_common(1)[0][0]
        merges[best] = rank
        for s in seqs:  # apply the new merge everywhere
            i = 0
            while i < len(s) - 1:
                if (s[i], s[i + 1]) == best:
                    s[i:i + 2] = ["".join(best)]
                else:
                    i += 1
    return merges


print("=== 2. A tiny tokenizer (BPE), from scratch ===")
corpus = "context contexts contextual contexted engineering engineer engineered " * 50
merges = learn_merges(corpus, n=12)
for w in ["context", "contextual", "engineering", "token"]:
    toks = bpe_encode(w, merges)
    print(f"  {w:12s} -> {toks}   ({len(toks)} token(s))")
print()

print("=== 3. The practical estimate ===")
prompt = ("Summarize the deployment failure and propose a fix. "
          "Keep it under three sentences. ") * 4
words = len(prompt.split())
chars = len(prompt)
print(f"  prompt: {words} words, {chars} chars")
print(f"  ~words * 1.3 = {round(words * 1.3)} tokens")
print(f"  ~chars / 4   = {round(chars / 4)} tokens")
print("  Both are estimates. For billing-grade counts, call the model's own")
print("  count_tokens endpoint with the SAME model id you will send to.")

Running it:

=== 1. Output is the expensive half ===
Per-token, output costs 5x input on every model here:

  claude-opus-4-8    input $ 5.00/Mtok   output $25.00/Mtok   output is 5x
  claude-sonnet-4-6  input $ 3.00/Mtok   output $15.00/Mtok   output is 5x
  claude-haiku-4-5   input $ 1.00/Mtok   output $ 5.00/Mtok   output is 5x

Workload: 100,000 calls, 8000 input + 400 output tokens each (claude-opus-4-8).
  total cost: $5,000

  cut 100 OUTPUT tokens/call:  saves $250
  cut 100 INPUT  tokens/call:  saves $50
  -> token for token, output is worth 5x as much to cut.
  (Caching makes input cheaper still, ~0.1x on a hit, widening the gap;
   see chapter 6. Whether INPUT or OUTPUT dominates your bill depends on the
   ratio: a long prompt with a short answer is input-heavy in TOTAL, even
   though each output token is pricier.)

=== 2. A tiny tokenizer (BPE), from scratch ===
  context      -> ['context']   (1 token(s))
  contextual   -> ['context', 'u', 'a', 'l']   (4 token(s))
  engineering  -> ['enginee', 'r', 'i', 'ng']   (4 token(s))
  token        -> ['t', 'o', 'k', 'e', 'n']   (5 token(s))

=== 3. The practical estimate ===
  prompt: 52 words, 332 chars
  ~words * 1.3 = 68 tokens
  ~chars / 4   = 83 tokens
  Both are estimates. For billing-grade counts, call the model's own
  count_tokens endpoint with the SAME model id you will send to.

Read the BPE output closely, because it shows the algorithm working. After learning just twelve merges on a small corpus, context has been fused into a single token, while token itself, which never appeared in the training corpus, stays shattered into five character tokens. That is exactly why a word common in your domain is cheap and a rare identifier is expensive: the tokenizer has a merge for the former and not the latter. Notice also engineering, which appeared in the corpus but not enough times to merge fully, so it lands between the two extremes (enginee plus three character tokens). Real production tokenizers run this same process to the tune of tens of thousands of merges, so they fuse far more, but the principle on display in those four lines is identical to the one inside Opus 4.8.

The cost asymmetry is the strategic takeaway. It is why Chapter 4 is dedicated entirely to making the model write less, and why a one-line "be concise" instruction can have a bigger return than compressing a long document. It does not mean input is free: in the workload above the input still costs more in total because there is 20x more of it. The rule is per token. When you are choosing what to optimize, weight a saved output token as five saved input tokens, then multiply by how many of each you actually have. And there is a fourth lever the formula does not show on its face: repeated input can be cached so that a cache hit costs roughly a tenth of the base input rate, which widens the output-versus-input gap further still. Chapter 6 is about exactly that, and it is why a long, stable preamble is often cheaper to keep than to compress.

Counting tokens for real

Estimates (words * 1.3, chars / 4) are fine for a sanity check. For anything that touches a budget or a billing decision, count exactly, with the same model id you will send to, because different model families tokenize differently. Anthropic exposes this as count_tokens, which takes the same messages you would send and returns the input token count without running the model. The call is free and does no generation, so you can run it in a tight loop while you tune a prompt. It also counts the way the real request will be billed: it includes the system prompt, any tool definitions, and the message structure, not just the raw text, so the number you get is the number you pay for. The following is follow-along (the build machine has no API key), but it is the exact call:

# Illustrative: requires the anthropic SDK and an API key.
import anthropic
client = anthropic.Anthropic()

n = client.messages.count_tokens(
    model="claude-opus-4-8",
    system=SYSTEM_PROMPT,
    messages=[{"role": "user", "content": LONG_DOCUMENT}],
).input_tokens
print(n, "input tokens")  # exact, free, no generation

There is a lesson here that costs people real money: never use another vendor's tokenizer to count tokens for Claude. A common shortcut is to reach for a familiar library, often a GPT tokenizer such as tiktoken, because it is already installed and runs offline. It will return a number, and the number will be wrong. Each model family has its own vocabulary and its own merge list, so a foreign tokenizer typically lands 15 to 20 percent off on prose and further off on code. Worse, the model maker can change its own tokenizer between model generations, so even a count that was right last year can drift. That drift is precisely the kind of error that does not show up in a quick test and does show up at the end of the month on the bill. The fix is the same in every case: count with the provider's own count_tokens for the exact model id you will call.

Don't be confused. Do not reach for another vendor's tokenizer library (for example a GPT tokenizer) to count tokens for Claude, or vice versa. Each model family has its own vocabulary and merges, so a foreign tokenizer can be off by 15 to 20 percent, more on code. "Approximately right" is fine for a gut check and wrong for a budget. Use the provider's own count_tokens for the model you will actually call.

Pricing a context before you send it

Put the two halves together and you can price a call before making it. The recipe is short:

Count the input with count_tokens and the exact model id (this is the one number you can know precisely up front).
Estimate the output from how long the answer tends to run, measured on a few real calls rather than guessed (output is the number you cannot know in advance, so estimate it from data, not optimism).
Multiply by the rates and add: $\text{cost} = \dfrac{\text{in_tok}}{10^6},r_\text{in} + \dfrac{\text{out_tok}}{10^6},r_\text{out}$, where $r_\text{in}$ and $r_\text{out}$ are the per-million-token rates for the model.
Multiply by call volume to get a daily or monthly figure, which is the number a budget conversation actually needs.

The cost(model, in_tok, out_tok) function in the demo is steps 3 and 4 in code. Doing this up front is what turns context engineering from a vibe ("this feels bloated") into a decision ("this preamble costs $2,000 a day re-sent uncached; caching it drops that to $200"). It is also how you capacity-plan: the same per-call cost, divided into a rate-limit or a budget ceiling, tells you how many calls per minute you can actually run. Every later chapter ends up cashing out as a change to one of the three numbers in that formula: fewer input tokens (Chapter 3), fewer output tokens (Chapter 4), or a cheaper rate on repeated input (Chapter 6).

Takeaways

Tokens are learned sub-word pieces, not words. Count is roughly 1.3 tokens per English word and higher for code or JSON; the ratio is not fixed, so estimates drift.
BPE builds the vocabulary by merging frequent adjacent pairs; a domain-common word is one cheap token, a rare identifier is many expensive ones.
Output tokens cost about 5x input tokens. Token for token, trimming output is worth 5x trimming input, but whether input or output dominates your total bill depends on the ratio you actually run.
Estimate with words * 1.3 for a gut check; count with the provider's count_tokens and the exact model id for anything that touches a budget. Never use a foreign tokenizer.
Pricing a context up front (count in, estimate out, multiply by rates) turns "feels bloated" into a number you can act on.

👉 Now that we can measure a context and its price, we can start shrinking it. The next chapter compresses the input: removing the tokens a long prompt does not need while keeping the ones it does.

Context Engineering in Depth