Tokenisation, BPE, and the multilingual token-count penalty

What it is

A language model does not see text. It sees a sequence of integers, each an index into a fixed vocabulary of perhaps 32,000 to 256,000 entries. Tokenisation is the mapping from text to those integers, and the tokeniser is a separate artifact from the model, trained on its own corpus, frozen before the model trains, and unchangeable afterwards.

Byte Pair Encoding (BPE) is how nearly every current tokeniser is built. Start with individual bytes and repeatedly merge the most frequent adjacent pair, recording each merge, until the vocabulary reaches the target size. Common sequences become single tokens; rare ones stay fragmented.

"tokenisation"  ->  ["token", "isation"]           2 tokens
"tokenización"  ->  ["token", "izaci", "ón"]       3 tokens
"токенизация"   ->  ["то", "кен", "иза", "ция"]    4+ tokens
"トークン化"      ->  ["ト", "ー", "ク", "ン", "化"]    5 tokens

That progression is the entire practical content of this page. The same word in different languages costs a different number of tokens, because the tokeniser's merges were learned from a corpus that was mostly English. That difference propagates into price, latency, and how much context a user actually gets.

What it is confused with: tokens are not words and not characters. The commonly-cited "1 token ≈ 4 characters" or "≈ 0.75 words" holds for English prose and is wrong for code, wrong for other languages, and wrong for structured text. Estimating a bill or a context budget with it will be wrong by a factor of two or more on non-English input.

The problem it solves

Why not just use characters, or just use words?

Character-level vocabularies are tiny (a few hundred entries) and produce very long sequences. Since attention is quadratic in sequence length and the KV cache is linear in it (see the transformer walked through with tensor shapes), a 5x longer sequence is a large cost multiplier for the same content.

Word-level vocabularies produce short sequences and cannot handle anything outside the vocabulary. Every typo, every product name, every new word is <UNK>, and the model learns nothing about it. Morphologically rich languages (Finnish, Turkish, Hungarian) have effectively unbounded word forms, so a word vocabulary covers them badly at any size.

Subword tokenisation is the compromise: frequent whole words stay whole, rare words decompose into pieces, and nothing is ever out-of-vocabulary because the fallback is bytes. It gives short sequences for common text and graceful handling of anything else.

The problem it creates, which is the substance here: the merge table is learned from a specific corpus, so it encodes that corpus's distribution. Text unlike the training corpus tokenises inefficiently, and "inefficiently" means more tokens, which means more money, more latency and less usable context.

Mechanics

Training BPE

def train_bpe(corpus, target_vocab_size):
    # Start from bytes: 256 entries, and nothing is ever out-of-vocabulary.
    vocab = {bytes([i]): i for i in range(256)}
    merges = []

    words = [list(w.encode('utf-8')) for w in corpus]

    while len(vocab) < target_vocab_size:
        # Count every adjacent pair across the corpus.
        pairs = Counter()
        for word in words:
            for a, b in zip(word, word[1:]):
                pairs[(a, b)] += 1

        if not pairs:
            break
        best = max(pairs, key=pairs.get)        # most frequent pair
        merges.append(best)                      # ORDER MATTERS at encode time
        vocab[best] = len(vocab)
        words = [merge_pair(w, best) for w in words]

    return vocab, merges

Encoding applies the merges in the order they were learned, which is why the merge list is part of the tokeniser and not just the vocabulary. Two tokenisers with identical vocabularies and different merge orders produce different token sequences.

Byte-level BPE (GPT-2 onward) starts from the 256 byte values rather than from Unicode characters, which guarantees any input encodes without an unknown token. The cost is that a character outside ASCII occupies 2 to 4 bytes and therefore starts as 2 to 4 separate tokens before any merges apply. That is the root of the multilingual penalty.

The multilingual penalty, measured

The same sentence, tokenised by cl100k_base (GPT-4's tokeniser, 100k vocabulary):

English:    "The quick brown fox jumps over the lazy dog"
            9 words, 43 chars  ->  9 tokens      (4.8 chars/token)

Spanish:    "El rápido zorro marrón salta sobre el perro perezoso"
            9 words, 52 chars  ->  15 tokens     (3.5 chars/token)   1.7x

German:     "Der schnelle braune Fuchs springt über den faulen Hund"
            9 words, 54 chars  ->  16 tokens     (3.4 chars/token)   1.8x

Russian:    "Быстрая коричневая лиса прыгает через ленивую собаку"
            7 words, 51 chars  ->  29 tokens     (1.8 chars/token)   3.2x

Japanese:   "素早い茶色のキツネが怠け者の犬を飛び越えます"
            21 chars           ->  32 tokens     (0.66 chars/token)  3.6x

Thai:       "สุนัขจิ้งจอกสีน้ำตาลกระโดดข้ามสุนัขขี้เกียจ"
            42 chars           ->  61 tokens     (0.69 chars/token)  6.8x

Thai costs roughly 7x more tokens than English for equivalent content. That is not a quality issue; it is a direct multiplier on price, on latency, and on how much of the context window the user gets.

The consequences, stated plainly:

A 128k context window holds:
  English:   ~96,000 words
  Spanish:   ~56,000 words
  Russian:   ~30,000 words
  Thai:      ~14,000 words

The same product gives a Thai user one seventh of the working memory it gives an English user, at the same price per token. For a document-analysis product that is a material difference in capability, and it is invisible unless someone measures it.

Code tokenises badly too

def calculate_total(items):
    return sum(item.price for item in items)

Whitespace: leading indentation is often its own token or several
Identifiers: "calculate_total" -> ["calculate", "_", "total"]
Punctuation: each of ( ) . : is typically its own token

68 characters -> 24 tokens  (2.8 chars/token, vs 4.8 for English prose)

GPT-4's cl100k_base added multi-space tokens specifically for this, so runs of 2, 4, 8 and 16 spaces are single tokens. That change alone cut Python token counts by roughly 10 to 15 percent against GPT-3's tokeniser. It is a good illustration of the tokeniser being tuned for an expected workload.

Why tokenisation causes specific model failures

Arithmetic. Numbers tokenise inconsistently:

cl100k_base:
  "2024"     -> ["202", "4"]         2 tokens, split mid-number
  "1234"     -> ["123", "4"]
  "12345"    -> ["123", "45"]
  "999"      -> ["999"]              1 token

The model sees 2024 as two arbitrary pieces, so digit-position arithmetic has to be learned across an inconsistent segmentation. Llama 3 tokenises every digit separately, precisely to make arithmetic learnable, at the cost of more tokens for numeric text. That is a deliberate trade and a good example of tokeniser design affecting capability.

Character-level tasks. "How many r's in strawberry?" is hard because the model sees ["str", "aw", "berry"] and never sees individual letters. It is not a reasoning failure; the information is not in the input representation. Reversing a string, counting characters, and simple ciphers all fail for the same reason.

Trailing whitespace. "Hello" and "Hello " tokenise differently, and in most tokenisers a leading space is part of the following token (" world" is one token, "world" is another). A prompt ending in a trailing space puts the model in a state where the natural next token would have started with a space, and the results degrade noticeably. Never end a prompt with a trailing space is a real rule with a mechanical cause.

Glitch tokens. Tokens that appear in the tokeniser's training corpus and almost never in the model's, so their embeddings are effectively untrained. SolidGoldMagikarp is the famous case: a Reddit username frequent enough in the tokeniser corpus to earn a merge, absent from the model's training data, producing bizarre behaviour when prompted. It is a consequence of the tokeniser and the model being trained on different data.

Counting tokens correctly

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
n = len(enc.encode(text))

# Anthropic: the count_tokens endpoint, since the tokeniser is not public.
# Llama / Mistral: transformers AutoTokenizer.

Estimating from character count is where budgets go wrong. A team sizing a multilingual product on "4 characters per token" will underestimate Japanese and Thai usage by a factor of five or more, and will discover it on the bill.

A worked example: a support product that priced out one market

A customer-support summarisation product. Ingests a ticket thread, produces a summary and a suggested reply. Launched in English, then expanded to Spanish, German, Japanese and Thai.

Pricing was set from English usage:

Average ticket thread:     1,840 tokens
Average output:              310 tokens
Cost per ticket:           $0.0142
Price per ticket:          $0.05        (3.5x margin)

Three months after the multilingual launch:

market      tickets/mo   avg input tokens   cost/ticket   margin
English       1.2M            1,840          $0.0142       3.5x
Spanish       410k            3,180          $0.0244       2.0x
German        280k            3,410          $0.0261       1.9x
Japanese      190k            6,720          $0.0509       0.98x   <- at cost
Thai           95k           12,140          $0.0918       0.54x   <- LOSING money

Thai tickets cost 1.8x the price charged for them. The product was losing about $4,000 a month on Thai alone, and nobody had noticed because unit economics were tracked in aggregate and Thai was 4 percent of volume.

The second problem, which was worse. Their context limit was 8,000 tokens, and threads longer than that were truncated:

market      % of threads truncated
English            2.1%
Spanish            8.4%
German             9.7%
Japanese          31.2%
Thai              58.9%       <- most Thai threads lost content

Fifty-nine percent of Thai support threads were being silently truncated, so the summaries were being generated from partial conversations. Summary quality complaints from the Thai market had been logged as a model quality issue and were a tokenisation issue.

What they did.

# 1. Measure, per market, in the actual tokeniser. Not estimated.
def thread_cost(thread: str, model: str) -> dict:
    enc = tiktoken.encoding_for_model(model)
    n_in = len(enc.encode(thread))
    return {"input_tokens": n_in,
            "est_cost": n_in * INPUT_PRICE + AVG_OUT * OUTPUT_PRICE,
            "truncated": n_in > CONTEXT_LIMIT}

# 2. Per-market pricing, set from measured token ratios rather than one global price.
   English $0.05, Spanish/German $0.07, Japanese $0.12, Thai $0.19

That was the commercially honest fix and it was also uncomfortable, so they paired it with reducing the cost:

# 3. Compress the input before it reaches the model.
   - Strip quoted reply chains (the same text repeated per message)
   - Drop signatures and legal footers
   - Deduplicate repeated boilerplate

market      before      after compression    reduction
English     1,840          1,190               35%
Japanese    6,720          4,010               40%
Thai       12,140          6,880               43%

Compression helped the expensive markets more, because quoted chains and boilerplate were repeated content, and repeated content is exactly what BPE tokenises badly in non-English text (the same phrase costs 3x as many tokens each time it appears).

# 4. Chunk-and-summarise for threads still over the limit, rather than truncating.

Final:

market      cost/ticket   price    margin    truncated
English       $0.0092     $0.05     5.4x       0.3%
Spanish       $0.0154     $0.07     4.5x       1.1%
German        $0.0166     $0.07     4.2x       1.4%
Japanese      $0.0304     $0.12     3.9x       2.8%
Thai          $0.0521     $0.19     3.6x       4.2%

Every market profitable, truncation under 5 percent everywhere, and the Thai quality complaints stopped.

The lesson worth carrying: token count is a per-language property and pricing, context budgets and truncation thresholds all inherit it. A single global number for any of the three is wrong in a way that scales with how different your users' languages are from English. The team had built the product correctly and had one implicit assumption, "a ticket is about 2,000 tokens," which was true only for the market they launched in.

Production evidence

OpenAI publishes tiktoken with the exact encodings per model (cl100k_base for GPT-4 and GPT-3.5-turbo, o200k_base for GPT-4o), and the vocabulary size increase from 100k to 200k in o200k_base was motivated substantially by non-English efficiency. Their own documentation notes improved token counts for non-English languages as a headline benefit.

Llama 3 increased its vocabulary from 32,000 (Llama 2) to 128,256 and Meta's model card attributes meaningful efficiency gains to it, particularly for non-English text and code. Llama 3 also tokenises digits individually, a deliberate choice to make arithmetic learnable.

The "All Languages Are Not Created (Tokenized) Equal" analysis and related work measured tokenisation ratios across dozens of languages against GPT tokenisers, finding multipliers of 2x to 15x relative to English depending on script. Languages using non-Latin scripts and morphologically rich languages fare worst, and the effect is consistent across tokenisers trained on English-dominant corpora.

Anthropic does not publish its tokeniser and provides a count_tokens API endpoint instead, which is worth knowing because it means client-side estimation is not available and token budgeting requires an API call.

SentencePiece (Kudo and Richardson, Google) is the other major implementation, used by T5, Llama 2 and many multilingual models. Its distinguishing property is operating directly on raw text including whitespace, so it is language-agnostic and does not require a pre-tokenisation step, which matters for languages without spaces between words.

The SolidGoldMagikarp investigation (Rumbelow and Watkins, 2023) documented glitch tokens systematically, showing they cluster near the centroid of the embedding space because they were never trained, and demonstrating reproducible anomalous behaviour across models sharing a tokeniser.

The debate

Should vocabulary be larger? Larger vocabularies mean fewer tokens per text, so cheaper inference and more content per context window. They also mean a larger embedding matrix and output layer: at d = 4096, going from 32k to 128k vocabulary adds about 786 M parameters (embedding plus output head), which for an 8B model is a tenth of the model spent on the vocabulary. The trend is clearly toward larger (32k to 128k to 200k), which suggests the sequence-length saving outweighs the parameter cost at current scales, and the balance shifts with how multilingual the target usage is.

Should you train a domain-specific tokeniser? For a specialised domain (medical codes, chemical formulae, a single non-English language, a proprietary log format) a custom tokeniser can cut sequence length substantially. The blocking constraint is that you cannot change a tokeniser without retraining the model, so this is only available if you are training from scratch or doing very substantial continued pretraining. For almost everyone the answer is no, and the practical lever is compressing the text rather than changing the tokeniser.

Is the multilingual penalty a fairness problem? Yes, and it is worth being direct about it: users writing in Thai or Japanese pay several times more for the same content and get a fraction of the context window, purely because of a corpus choice made when the tokeniser was trained. My position: measure it per market and price accordingly or absorb it deliberately, but do not leave it implicit. The worked example is what implicit looks like: an unprofitable market and a quality complaint that was really a truncation problem.

Do tokeniser-caused failures matter in practice? The character-counting failures are mostly curiosities. The ones that matter commercially are cost and context budgeting, because they are systematic and scale with volume. The arithmetic issues matter if your product does numeric work, and the honest mitigation is not to rely on the model for arithmetic at all: give it a calculator tool.

Will tokenisation go away? Byte-level and tokeniser-free architectures exist (MegaByte, and Meta's Byte Latent Transformer) and the motivation is exactly the problems here: no vocabulary bias, no glitch tokens, no multilingual penalty. They are not yet competitive at scale for general use. Worth knowing as a direction; not worth planning around.

Follow-up Q&A

"How does BPE work?"

Start with a base vocabulary of the 256 byte values, count every adjacent pair in the training corpus, merge the most frequent pair into a new token, and repeat until the vocabulary reaches the target size. The merge list is ordered and encoding replays it in order, so the merges are part of the tokeniser, not just the vocabulary. Starting from bytes rather than characters guarantees nothing is ever out-of-vocabulary, at the cost that a non-ASCII character begins as 2 to 4 separate byte tokens.

"Why does Japanese cost more than English?"

The merge table was learned from a corpus that was overwhelmingly English, so English character sequences earned merges and became single tokens while others did not. Combined with byte-level encoding, where a Japanese character is 3 UTF-8 bytes, the result is roughly 3 to 4 tokens per character before merges help. Measured on the same sentence, Japanese is about 3.6x English and Thai about 6.8x. That multiplies price, latency and how much of the context window a user gets.

"How would you budget context for a multilingual product?"

Per language, measured with the actual tokeniser rather than a characters-per-token rule. The rule that "1 token ≈ 4 characters" is an English-prose approximation and underestimates Japanese and Thai by a factor of five. Concretely: measure the token distribution per market, set the truncation threshold and the price from that distribution, and alert on truncation rate per market rather than in aggregate, because a market that is 4 percent of volume can be 59 percent truncated and invisible in the average.

"Why can't models count letters in a word?"

They never see letters. "strawberry" is tokenised as something like ["str", "aw", "berry"], and the model's input is three integers. Counting r's requires character-level information that is not present in the representation. It is not a reasoning failure, it is an input representation limitation, and the same cause explains failures at string reversal and simple ciphers.

"Why does a trailing space in a prompt hurt?"

In most BPE tokenisers a leading space is part of the following token: " world" is one token distinct from "world". A prompt ending with a trailing space means the model must now produce a token that does not start with a space, which is an unusual state relative to its training distribution, and output quality degrades. It is a small thing with a mechanical cause and it is worth knowing because prompts assembled by string concatenation acquire trailing spaces easily.

"Can you change the tokeniser after training?"

No, not without retraining. The embedding matrix maps token IDs to vectors and every weight in the model was learned against that mapping. A new tokeniser produces different IDs for the same text, so the embeddings are meaningless. This is why tokeniser choices (vocabulary size, digit handling, whitespace handling) are made before pretraining and are permanent for the model's life.

Common misconceptions

"A token is about 4 characters." For English prose. It is about 2.8 for code, 1.8 for Russian, and 0.7 for Japanese and Thai. Budgeting or pricing with the English figure on multilingual input will be wrong by several times.

"Tokens are words." They are frequent subword pieces. Common words are single tokens, rare words fragment, and a leading space is usually part of the token.

"Bigger vocabulary is strictly better." It shortens sequences and enlarges the embedding and output layers. At d = 4096, 32k to 128k vocabulary is about 786 M extra parameters, which is a tenth of an 8B model.

"Model quality is why it fails at counting letters." The characters are not in the input. No amount of model capability recovers information the tokenisation discarded.

"The tokeniser and the model are trained together." The tokeniser is trained first, on its own corpus, and frozen. When the two corpora differ you get glitch tokens: entries with essentially untrained embeddings, like SolidGoldMagikarp.

Interview delivery note

Say this verbatim: "Token count is a per-language property, not a global constant. The same sentence is roughly 3.6x more tokens in Japanese and 6.8x in Thai than in English, because the merge table was learned from an English-dominant corpus. That multiplies cost, latency, and how much of the context window the user actually gets, so pricing and truncation thresholds have to be per market." A specific measured claim with its cause and its three consequences.

The senior-versus-staff separator is the truncation consequence rather than only the cost one. A senior engineer knows non-English text costs more tokens. A staff engineer notices that a fixed 8,000-token limit truncates 2 percent of English threads and 59 percent of Thai ones, that this presents as a model quality complaint from one market, and that the aggregate metric hides it entirely because that market is 4 percent of volume. Connecting a tokenisation property to a support ticket is the reasoning being tested.

The second signal is knowing that the tokeniser is frozen before the model trains and cannot be changed afterwards, so the practical lever is compressing the input text rather than the tokenisation. In the worked example, stripping quoted reply chains helped the expensive languages more, because repeated content costs proportionally more when each repetition tokenises badly.

The Staff & Team Lead Interview Reference