Source: llm · llm.md · updated 2026-07-25 · 🔒 secret gist

Synced verbatim from gist.github.com/bl9.

How an LLM Actually Works

A mental model for someone who already knows transformers on paper but can't reconcile "matrix multiplications over vectors" with "it understood what I meant."

Contents

Part I — The object

• 0. The one thing to unlearn first
• 1. Concrete anchor: a real model's dimensions
• 2. Tokenization
• 3. The residual stream
• 4. Attention, dissected properly
• 5. The MLP — where knowledge is stored
• 6. Superposition
• 7. Circuits — worked examples
• 8. Depth = composition, and where intent lives
• 9. From vector to token: the output side

Part II — Putting it together

• 10. Full worked trace
• 11. Why prediction produces intent modeling
• 12. In-context learning
• 13. Chain of thought = serial compute

Part III — How it got that way

• 14. Training: SGD as a compiler
  • 14.1 The reframe
  • 14.2 What one SGD step actually does
  • 14.3 Why one update improves a million unrelated cases
  • 14.4 Why it cannot memorize
  • 14.5 The numbers
  • 14.6 The loss curve has structure
  • 14.7 Where modern gains come from
  • 14.8 The post-training stages
  • 14.9 The disanalogy: no reject state

Part IV — Applied

• 15. Worked scenario: a landing page
• 16. Worked scenario: a compiler pass
• 17. Failure modes
• 18. What is honestly still unknown

Part V — Reference

• 19. Every concept, defined and modeled
  • 19.1 The substrate
  • 19.2 Architecture
  • 19.3 Training
  • 19.4 Inference
  • 19.5 Interpretability
  • 19.6 Practical and operational
  • 19.7 Behavior and failure
• Appendix A: the compressed mental model
• Appendix B: the analogy that actually holds
• Further reading

0. The one thing to unlearn first

If your intuition for "text → vector" comes from embedding models (sentence-transformers, bi-encoders, HNSW indexes), that intuition is actively blocking you.

An embedding is a photograph. A transformer forward pass is a film — and the frames modify each other.

That is the whole reason one can only do retrieval while the other can write a compiler pass.

1. Concrete anchor: a real model's dimensions

Vague talk about "dimensions" is why this feels like magic. Here are actual numbers, using a Llama-3-70B-class model:

d_model (residual stream width)   8,192
layers                               80
attention heads per layer            64
head dimension                      128
MLP hidden width                 28,672
vocabulary                      128,256
context window                  128,000 tokens
total parameters             70,000,000,000

Derived facts worth internalizing:

• Attention head computations per token: 80 × 64 = 5,120. Not one similarity lookup — five thousand of them, each with its own learned notion of what to look for and what to bring back.
• Parameter split: roughly 1/3 attention, 2/3 MLP. Most of the model is memory, not routing.
• FLOPs per generated token: ~2 × 70B = 140 billion. For a 500-token answer, ~70 trillion floating point operations. It is not a lookup. It is a lot of computation.
• 8,192 numbers per position. In fp16 that's 16 KB. That 16 KB is the model's entire working belief about that word in that context, at that depth.

2. Tokenization — the layer everyone skips, and shouldn't

Text is split into subword tokens by BPE. Rough rule: 1 token ≈ 3.5–4 characters of English.

"the tests are failing"     → ["the", " tests", " are", " failing"]        4 tokens
"unfortunately"             → ["unfort", "unately"]                        2 tokens
"strawberry"                → ["str", "aw", "berry"]                       3 tokens
"getUserByID"               → ["get", "User", "By", "ID"]                  4 tokens
"128256"                    → ["128", "256"]                               2 tokens
"色即是空"                   → ["色", "即", "是", "空"] or worse            4+ tokens

Two consequences that explain real behavior:

1. The model cannot see letters. "strawberry" arrives as three opaque chunks. Asking it to count the r's is asking someone to count letters in a word they only ever heard spoken. This single fact explains a whole genre of "how is it so dumb about this" failures.
2. Non-English and code are tokenized less efficiently. The same sentence in Thai or Tamil may cost 3–5× the tokens of English. Since compute is per-token, this is also a fairness and cost issue in multilingual retrieval systems — the tokenizer is a quiet source of bias before any model math happens.

Each token ID indexes a row of the embedding matrix: 128,256 × 8,192. That row is the starting value of the residual stream at that position. It encodes essentially nothing contextual yet — just "this token, generically."

Position is added separately (modern models use RoPE, which rotates query and key vectors by an angle proportional to position, so attention naturally sees relative distance rather than absolute index).

3. The residual stream — the single most important object

Picture a spreadsheet:

• Columns = token positions in your prompt
• Rows = layers, 80 of them, bottom to top
• Each cell = an 8,192-dimensional vector

                 "the"     "tests"    "are"    "failing"
             ┌──────────┬──────────┬──────────┬──────────┐
Layer 80     │ v80,1    │ v80,2    │ v80,3    │ v80,4    │ ← this one becomes the prediction
             ├──────────┼──────────┼──────────┼──────────┤
    ...      │   ...    │   ...    │   ...    │   ...    │
             ├──────────┼──────────┼──────────┼──────────┤
Layer 2      │ v2,1     │ v2,2     │ v2,3     │ v2,4     │
             ├──────────┼──────────┼──────────┼──────────┤
Layer 1      │ v1,1     │ v1,2     │ v1,3     │ v1,4     │
             ├──────────┼──────────┼──────────┼──────────┤
Embedding    │ e(the)   │ e(tests) │ e(are)   │ e(fail…) │
             └──────────┴──────────┴──────────┴──────────┘

Critically, the update rule is additive:

x ← x + attention_output(x)
x ← x + mlp_output(x)

Not x ← f(x). The stream is a shared workspace that every component reads from and writes into. Nothing is overwritten; contributions accumulate.

This is why the mental model of "a bus" or "a whiteboard" is better than "a pipeline." Layer 3 can write a fact that nothing touches until layer 61 reads it. Components communicate across depth by leaving things in the stream.

Two directions of information flow:

• Vertical (within a column): MLPs. Enrich this position's representation with knowledge.
• Horizontal (between columns): Attention. This is the only mechanism by which positions see each other. Remove attention and you have 4 independent MLPs that have never heard of one another.

4. Attention, dissected properly

The textbook formula is softmax(QKᵀ/√d)V. The useful decomposition is different — split it into two independent learned circuits:

The QK circuit — "where do I look?"

Each position produces a query ("what am I looking for right now?") and a key ("what am I, advertised to others?"). Dot product, scaled, softmaxed over all earlier positions → attention weights.

This is exactly the retrieval you know from vector search, with three differences that change everything:

1. The corpus is your own prompt, not a static index.
2. The query is recomputed from the current stream state at every layer — so a position's question at layer 60 is informed by what it learned at layers 1–59.
3. It runs 5,120 times per token.

The OV circuit — "what do I bring back?"

Separately, each position produces a value vector, and the head has an output matrix. What gets copied back into the stream is attention_weights × values × W_O.

The decoupling is the point. What matched and what gets copied are different learned functions. A head can attend to the word "France" but write a country-code direction rather than the France representation. Retrieval systems don't do this; they return the document they matched.

Multi-head = parallel specialists

64 heads per layer, each with a 128-dim subspace. They specialize sharply:

That last row is worth pausing on. Some heads exist to say no. The network implements something like negation and inhibition using attention.

Causal masking

Position i can only attend to positions ≤ i. This is what makes training efficient — one forward pass over a 4,000-token document yields 4,000 training signals at once, each a legitimate "predict the next token" problem. It's also why the whole thing generates left to right.

KV cache — why the first token is slow and the rest are fast

• Prefill: your whole prompt is processed in one parallel pass. All keys and values for all positions are computed and cached. This is compute-bound and is the delay before the first token appears.
• Decode: each new token attends against the cache. Only one new column is computed per step. This is memory-bandwidth-bound.

Cache size scales linearly with context length. At 128K context this is tens of GB. This is the real reason long context is expensive — not the attention math, the memory.

5. The MLP — where knowledge is stored

Two-thirds of the parameters live here, and the structure is a key-value memory (Geva et al., 2021):

h = GELU(W_in · x)        # 8,192 → 28,672 : 28,672 pattern detectors
out = W_out · h           # 28,672 → 8,192 : each fires, writes its associated vector back

• Each of the 28,672 rows of W_in is a key: a direction in stream-space it detects.
• Each corresponding column of W_out is a value: what to add to the stream when that key fires.

So: "if the stream currently contains [Paris-ness] + [capital-of-relation-active], add [France-direction]."

Multiply by 80 layers: 2.3 million key-value slots, chained so later ones read the results of earlier ones. That's the fact store — and because it's composed across depth, it does multi-hop lookups, not just single retrievals.

6. Superposition — the answer to "how can dimensions do this"

The intuitive objection: 8,192 dimensions can't hold millions of concepts.

The intuition is wrong, and here's the geometry.

In 8,192 dimensions you can only fit 8,192 perfectly orthogonal vectors. But if you relax to "almost orthogonal" — say, any pair within 85–95° — you can fit an exponential number. This is the Johnson–Lindenstrauss lemma. Concretely, in d dimensions you can pack roughly exp(εd) near-orthogonal directions.

At d = 8,192 that number is astronomically larger than the millions of concepts a model needs.

The tradeoff: features interfere slightly. Every feature bleeds a little noise into every other. This is tolerable if only a handful of features are active at any moment — which is exactly the case, since any given token in any given context is only a few things at once. This is sparse coding, and the network learns it because it's the optimal use of limited width.

This has been verified empirically, not just theorized. Anthropic's sparse autoencoder work on Claude 3 Sonnet extracted tens of millions of interpretable directions from the residual stream: a Golden Gate Bridge feature, a "code with a security vulnerability" feature, a "sycophantic praise" feature, an "inner conflict" feature. Amplify one and the model's behavior changes accordingly and predictably.

So the correct reformulation:

A residual stream vector is not a point in an 8,192-concept space. It is a sparse sum of a few dozen active features drawn from a dictionary of ~10⁷ learned directions.

That reframing is what makes "8,192 numbers" stop feeling too small.

7. Circuits — worked examples of the machinery

These aren't hypotheses. They've been reverse-engineered and causally verified by ablation.

7a. Induction heads — the origin of in-context learning

Setup. A two-head circuit spanning two layers:

1. A previous-token head in layer L: at each position, copies the identity of the token before it into the stream. So position i now carries "I am preceded by X."
2. An induction head in layer L+1: its query is "find positions preceded by the token I currently am." It matches, then copies that position's own token to the output.

Net effect: given [A][B] … [A], predict [B].

Worked example — why the model can track your made-up variable names:

const zqBuffer = new ArrayBuffer(1024);
...200 lines...
const view = new Uint8Array(zq▮

zqBuffer appears nowhere in training data. There is no stored fact. Yet the completion is Buffer, near-certainly.

Trace:

• Position of the final zq emits a query: "where else have I seen zq?"
• Duplicate-token / previous-token machinery has tagged the earlier occurrence.
• The induction head attends to the token after the earlier zq — which is Buffer.
• OV circuit copies Buffer into the output stream.
• Unembedding turns it into a very high logit.

Why this matters far beyond variable names: induction heads generalize from exact-match copying to fuzzy, abstract pattern completion. The same circuit family is what lets few-shot prompting work. When you write:

sea otter -> loutre de mer
peppermint -> menthe poivrée
plush girafe ->

induction-style heads recognize the X -> Y structure, identify the transformation as translation, and apply it. No weights changed. The "learning" happened entirely as data movement inside one forward pass.

There is a striking empirical detail: during pretraining, induction heads form abruptly at a specific point, and the loss curve visibly bends at exactly that moment. In-context learning is not a smooth emergent haze — you can watch the mechanism switch on.

7b. The IOI circuit — indirect object identification

Prompt: "When Mary and John went to the store, John gave a drink to ___"

Correct answer: Mary. Both names appear; the model must pick the one that isn't the repeated subject. The full circuit was reverse-engineered in GPT-2 small (Wang et al., 2022) — 26 heads in three groups:

So the algorithm is literally: find all names → detect which one is duplicated → inhibit it → copy the survivor.

This is a genuine algorithm, discovered by gradient descent, implemented in attention patterns, verified by ablating individual heads and watching the answer flip to John. Nobody designed it. Nobody wrote "if duplicate then suppress." It was the lowest-loss way to model English discourse.

When you ask "how does it know what I meant" — this is the shape of the answer. Thousands of circuits like this, composed.

8. Depth = composition, and where intent physically lives

A rough map of what happens as you climb the 80 layers:

The middle band is the interesting one, and there's direct evidence for it.

Task vectors / function vectors

Take a few-shot prompt for some task (say, "translate English to French"). Read the residual stream at a mid layer at the final position. Average across many examples. You get a single vector.

Now take a completely unrelated prompt, with no examples and no instructions, and add that vector into the stream at the same layer.

The model starts translating to French.

Let that land. There is a specific vector, in a specific place, that is the concept "the task right now is: translate to French," represented independently of any wording that expressed it. You can extract it, store it, transplant it.

This is the mechanistic answer to "how does it know what I meant." Your intent stops being words and becomes a vector somewhere in the middle of the network. Different phrasings of the same request converge to nearby vectors — which is exactly why "make this faster," "this is too slow," and "optimize perf here" all get you the same kind of help.

Logit lens — reading the model's mind mid-computation

You can apply the final unembedding matrix to intermediate layers to see what the model would predict if it stopped there. Typical trajectory for a factual question:

Layer 10 : generic function words ("the", "a", "of")
Layer 30 : right category, wrong specifics ("city", "place", "region")
Layer 55 : right answer starting to appear, low confidence
Layer 70 : right answer dominant
Layer 80 : sharpened, high confidence

The prediction is built up, progressively refined. It is not looked up.

9. From vector to token: the output side

At the last position, after layer 80:

1. Unembed: logits = W_U · x → 128,256 raw scores.
2. Softmax with temperature: p = softmax(logits / T). T=0 → always the argmax (deterministic). T=1 → the raw learned distribution. T>1 → flatter, more random.
3. Truncate: top-k (keep the k best) or top-p / nucleus (keep the smallest set summing to p, typically 0.9–0.95). This cuts the long tail of garbage tokens.
4. Sample one token.
5. Append it to the input and run the entire thing again.

That last step is the loop, and it's why the model can't un-say something. Each token is committed and becomes input. There's no backtracking, no edit buffer.

Dissecting an actual distribution. For the prompt "The tests are failing after I added the cache":

token          logit    prob
" layer"        18.2    0.41
" ."            17.6    0.23
" to"           16.9    0.11
" middleware"   16.1    0.05
" and"          15.8    0.04
" invalidation" 15.4    0.03
...128,250 more tokens, together ~0.13

Two things to notice:

• The model is never certain. Even "obvious" continuations carry substantial mass elsewhere. Confidence is a distribution, not a boolean.
• The whole answer is present in embryo. The distribution already encodes "this is a technical debugging context, in English, mid-sentence, about caching." Every subsequent token narrows further.

10. Full worked trace

Prompt: "the tests are failing after I added the cache layer"

Note what is not in that sentence: no question mark, no request, no "please help," no specification of what output you want. Yet you'd get back a ranked list of likely causes with debugging steps. Let's trace how.

Prefill — layers 1 through 80, at the final position

Where did the "request" come from?

Nowhere in the text. It was inferred, and here's the mechanism, not the hand-wave:

During pretraining, the model saw millions of instances of this exact discourse pattern — a Stack Overflow post, a Slack thread, a GitHub issue, an IRC log — where someone states a symptom and the very next thing in the document is a diagnostic response. To minimize prediction loss on those documents, the model had to learn: "symptom stated by developer" → "diagnosis follows."

Post-training then sharpened which of the many possible continuations it produces (a helpful diagnosis rather than, say, a snarky reply or another user's unrelated question), but the underlying inference was already there in the base model.

Your intent was reconstructed from statistical regularities in how humans structure conversations. Not from a rule. Not from an intent classifier. From compression.

Decode — dissecting the actual answer

Suppose the response begins:

"A few likely culprits, roughly in order of how often they bite:

1. Cache state leaking between tests — the cache isn't cleared in teardown, so test B sees what test A wrote."

Dissect where each part came from:

Nothing here was retrieved. There is no stored answer to "tests failing after cache layer." Every token was computed. The reason it's right is that the machinery for reasoning about cache-and-test interactions is genuinely encoded in the weights, in the form of features and circuits that generalize.

11. Why next-token prediction produces intent modeling

This is the philosophical crux, and it's actually rigorous.

The setup. Minimize −log P(next token | all previous tokens) over ~15 trillion tokens of human-written text.

The critical observation: that text was not generated randomly. It was generated by humans with goals, knowledge, moods, expertise levels, and plans. The text is a downstream shadow of those hidden variables.

So to predict it well, you must model the hidden variables. There is no shortcut:

• To predict the next line of a Stack Overflow answer, you must model what the asker needs.
• To predict the next token of a legal brief, you must model the argument being constructed.
• To predict the closing brace of a C function, you must model the scope structure.
• To predict the punchline, you must model what the audience finds surprising.
• To predict the next move in a chess transcript, you must model the position.

And memorization is unavailable. 70B parameters versus 15T training tokens is roughly 200 tokens compressed per parameter. A lookup table is not on the table. The only representation that fits is general machinery.

This gives the chain:

accurate prediction → requires compression → requires generalization →
requires modeling the generating process → and the generating process is
a human with an intent

Intent modeling is not a feature that was added. It is the lowest-loss solution to the objective, and gradient descent found it because it was cheapest.

The same argument explains capabilities that look unrelated to text: arithmetic, code execution traces, spatial reasoning, theory of mind. Each is a compression win on some slice of the corpus.

12. In-context learning — learning without learning

Weights are frozen at inference. Nothing is stored. Yet:

Prompt:
  gxlfj -> 5
  hqm   -> 3
  aabbccdd -> 8
  zzz ->

Answer: 3. The task ("count characters") was never named, never trained on with these strings, and no gradient was computed.

What actually happens: induction-style heads recognize the input -> output structure, mid-layer circuits form a task representation from the demonstrations, and later layers apply it to the new input. The forward pass implements a learning algorithm.

There's evidence that in some settings this is literally gradient descent implemented inside the attention layers — a linear-regression task in-context produces internal updates that approximate one or more gradient steps on that task. The transformer learned to run an optimizer as a subroutine, because doing so lowers prediction loss on documents containing patterns.

Practical corollary: your prompt isn't a query. It's a program that reconfigures the model's effective computation. That's why prompt structure matters so much — you're not phrasing a search, you're building a temporary specialist.

13. Chain of thought = renting more serial compute

Fixed architecture means fixed serial depth: 80 layers. Some problems require more sequential steps than that, no matter how wide the model is.

The escape hatch is the output itself. Every emitted token is fed back as input, so:

serial computation available = 80 layers × number of tokens generated

Writing intermediate steps turns the context window into external working memory and buys unbounded serial depth.

Direct:  "What is 47 × 83?"  →  "3,901"        ← one 80-layer pass. Often wrong.

CoT:     "47 × 83
          = 47 × 80 + 47 × 3
          = 3,760 + 141
          = 3,901"                              ← ~40 tokens = ~3,200 layers of serial work

Each line is computed, then read back as ground truth for the next line. This is why "think step by step" works, and why reasoning-tuned models generate long internal traces. They're not being verbose — they're allocating compute.

Important caveat: the written trace is not guaranteed to be the actual causal path. Models can reach an answer internally and then generate a plausible-looking justification. Chain of thought is a compute mechanism and a post-hoc narrative, and telling those apart is an open research problem.

14. Training: SGD as a compiler

14.1 The reframe: the training loop is the compiler

If you know compilers, you already have the right structure in your head — it just sits one level up from where you're looking.

gcc pipeline
  C source  ──►  gcc  ──────────►  binary  ──►  CPU
  human-written  passes by hand     ~5 MB       can reject

LLM pipeline
  15T tokens ──►  SGD  ──────────►  weights ──►  forward pass
  human-written  passes by search   ~140 GB     never rejects

Same shape. Two differences carry all the weight:

1. Nobody wrote the passes. In gcc, a human wrote constant folding, a human wrote loop unrolling, and each is provably correct. In an LLM, a stochastic search over a 70-billion-dimensional program space found the passes. The 80 layers genuinely contain bracket matching, scope tracking, type propagation, dead-branch suppression — but they were discovered, not authored, and nobody has fully read them back.
2. The runtime has no reject state. More on this in §14.9; it's the single most clarifying fact about why the thing "accepts everything."

The closest thing in your world is superoptimization — searching the space of instruction sequences for one better than a human would write. SGD is superoptimization with a different objective, a vastly larger search space, and no proof of anything.

14.2 What one SGD step actually does

Forget "training" as a vague process. A single step is small and entirely mechanical. Say the batch contains this snippet scraped from GitHub:

def process(items):
    result = []
    for item in items:
        result.append(transform(item))
    return ▮

Step 1 — forward pass. The model produces a distribution over all 128,256 tokens at that position. Suppose it assigns p(" result") = 0.31.

Step 2 — loss. Cross-entropy is just the negative log of the probability assigned to the actual next token:

loss = −ln(0.31) = 1.17 nats

If it had been confident:  −ln(0.95) = 0.05
If it had been clueless:   −ln(1/128256) = 11.76

The loss is literally "how surprised were you." Nothing more.

Step 3 — backward pass. Backpropagation computes ∂loss/∂w for every one of the 70 billion weights — one number per weight saying "nudge me this direction to have been less surprised." This is the chain rule applied to a computation graph, executed in reverse.

Step 4 — update.

w ← w − learning_rate × ĝ

With a learning rate around 1e-4 (decaying on a cosine schedule), each weight moves by roughly one part in ten thousand. In practice the optimizer is AdamW, which keeps running averages of the gradient and its variance so each weight gets its own effective step size.

That's the entire step. In isolation it is nearly worthless — the model became microscopically less surprised by one snippet. Everything comes from doing this ~10⁶ times over ~10¹³ tokens.

14.3 Why one update improves a million unrelated cases

This is the crux, and it has no compiler analogue.

To predict result correctly, the network had to run a real inference chain:

• result was bound at the top of this scope
• it's an accumulator — mutated in a loop, never otherwise read
• Python functions conventionally return their accumulator
• we are at return, in this function's scope, not the enclosing one

The gradient does not reward "the string result." It rewards whichever circuits performed that inference: the variable-binding tracker, the scope-boundary detector, the accumulator-pattern recognizer.

And those circuits are shared. The identical variable-binding machinery fires for:

• Rust let bindings
• SQL aliases in a CTE
• pronoun resolution in English prose
• "the aforementioned party" in a contract
• a defined term in a mathematical proof

So one Python snippet measurably improves the model's handling of legal documents. Not by analogy — literally, the same weights, updated once, better everywhere.

This is the generalization mechanism, and it explains the otherwise baffling fact that 15 trillion tokens of miscellaneous internet text produces something that can write a compiler pass. There's a well-documented empirical consequence: adding code to the training mixture improves reasoning on non-code benchmarks, because code is unusually dense in explicit logical structure and, crucially, was checked by a compiler before being committed.

14.4 Why it cannot memorize, even if it wanted to

15,000,000,000,000  training tokens
    70,000,000,000  parameters
──────────────────────────────────
              ~200  tokens compressed per parameter

At 2 bytes per parameter in fp16, that's roughly 0.01 bytes of storage per training token. Lossless memorization is off the table by three orders of magnitude.

SGD is therefore under brutal compression pressure, and the resolution is the one you'd reach yourself if you were hand-writing a codec: store the rules, not the instances.

• A generative rule for "how Python scoping works" costs a few thousand parameters and covers billions of tokens.
• Memorizing those same tokens costs billions of parameters and generalizes to nothing.

Gradient descent finds the first solution because the second doesn't fit. Generalization is not an achievement the model unlocks. It is the only thing that fits in the space available.

This is also the cleanest answer to "how does it know almost everything." It doesn't store everything. It stores the generative structure of everything, and re-derives specifics at inference time — which is exactly why it's fluent on common things and confidently wrong on rare ones.

14.5 The numbers, for calibration

That last row deserves a pause. There is no drilling, no repetition, no curriculum of the same material. The model learns Python scoping from seeing millions of different correct examples one time each — which is precisely the setup that forces it to learn the rule instead of the instances. Repetition would encourage memorization; the single pass forbids it.

The 6 × params × tokens figure decomposes as 2 for the forward pass and 4 for the backward pass (backward costs roughly twice forward). Handy for estimating any training run.

14.6 The loss curve has structure

Training loss is not a smooth glide. Capabilities arrive as phase transitions:

loss
 │╲
 │ ╲___                    learning token frequencies, basic syntax
 │     ╲
 │      ╲__               ← visible bend: induction heads form
 │         ╲___             in-context learning switches ON here
 │             ╲__        ← arithmetic circuits consolidate
 │                ╲____
 └──────────────────────► tokens seen

The induction-head bend is the famous one. Before it, few-shot prompting essentially doesn't work. After it, it does. You can watch a mechanism switch on by looking at a loss curve — the capability is not a smooth emergent haze, it's a circuit forming.

This is the same shape as grokking: a network can sit at memorization-level performance for a long time and then abruptly generalize, when the general circuit finally beats the memorized lookup on the loss. You are watching a search find a better program and swap it in.

14.7 Where the modern gains actually come from

Architecture is roughly frozen. A 2019 transformer with modern normalization and RoPE is close to a 2026 one. Two things moved instead:

1. Data curation — the real secret sauce, and the least published part.

• Aggressive deduplication (near-duplicate web pages are actively harmful — they induce memorization)
• Quality classifiers trained to recognize "textbook-like" writing
• Mixture ratios: how much code vs. web vs. books vs. math vs. multilingual, tuned empirically
• Annealing: saving the highest-quality data for the last few percent of training, where the learning rate is tiny and the effect is disproportionate. The model's final impressions are its most durable ones.
• Synthetic data: using a strong model to generate clean training material for the next one

2. RLVR — reinforcement learning from verifiable rewards. This is the stage that made code good, and it's the one a compiler person will find most familiar:

sample N candidate solutions
  ↓
run the test suite / check the proof / execute the program
  ↓
reward the trajectories that passed
  ↓
gradient-ascend toward those

Unlike RLHF ("did a human prefer this?"), the reward here is ground truth. You can optimize against it hard without immediately Goodharting, because a test suite has no taste to exploit. This is why RLVR adds genuine capability while RLHF mostly adds manners.

It also explains the capability profile you observe in practice: models are disproportionately strong in domains with cheap automated verification (code, math, formal logic) and weaker where correctness is a matter of judgment.

14.8 The post-training stages in detail

The mental model for the middle three: the base model can simulate an enormous range of authors. Post-training doesn't teach it anything new; it selects which author it becomes when addressed as an assistant, and makes that selection reliable.

The cost asymmetry is stark — pretraining is 98%+ of the compute, post-training is a rounding error. Yet post-training is what makes the artifact usable. It is a thin, cheap layer of steering over an enormously expensive engine.

The known downside: RLHF optimizes a proxy for human approval, and approval is not truth. This is the mechanistic origin of sycophancy — reversing a correct position under pushback is what the reward signal rewarded.

14.9 The disanalogy that resolves "it accepts everything"

You're conflating two properties. Separate them and most of the mystery dissolves.

Property 1: "It accepts anything." This is architectural, and not impressive.

Look at where a compiler can fail versus where a transformer can:

There is no code path in a transformer for "I don't understand this input." No grammar, no parser, no error state, no partial-failure mode. Every stage is a total function over its input domain. Feed it line noise and it will confidently predict the next token of line noise.

That's not intelligence — it's the absence of a rejection mechanism. And it's the direct architectural cause of hallucination: a system that cannot fail to parse also cannot fail to answer. When the model doesn't know something, there is no representational slot for "unknown" and no branch to take. The output head produces a peaked distribution regardless, because that's the only thing it can do.

Everything that looks like the model saying "I'm not sure" is learned behavior from post-training, generated as ordinary tokens — not a structural check. It's a string, not a status code.

Property 2: "It knows almost everything." This is capacity plus compression — and it's less true than it feels.

It knows the high-frequency structure of nearly everything, and degrades smoothly into the tail:

frequency in training data
       high ──────────────────────────────► low
       │                                     │
   reliable                            plausible
   and correct                         and wrong
       │                                     │
  "how does a         "LLVM             "the signature of
   dominator           dominator          getAnalysisUsage
   tree work"          tree API"          in LLVM 17.0.3"

And the confidence is flat across that entire axis, because nothing in the architecture measures how well-supported a fact was. The useful heuristic that falls out: trust its structure, verify its specifics. Algorithms and architecture generalize from circuits. API signatures, version numbers, and citations are memorized facts, and memorized facts decay with training frequency and go stale with time.

15. Worked scenario: "build me a landing page for a dog-walking service"

Why does this work, when the exact page was never in training data?

The abstractions that exist in the weights:

• landing-page schema: hero, value proposition, social proof, pricing, CTA, footer
• local-service-business schema: service area, trust signals, booking mechanism
• HTML/CSS syntax circuits: matched tags, valid property names, correct nesting
• Design conventions: contrast, hierarchy, whitespace, mobile-first

What the forward pass does:

1. L25–45: composes landing-page ∩ local-service ∩ pet-industry into a joint plan. Pet industry contributes warm colors, photography-forward, informal-but-trustworthy tone.
2. L45–65: instantiates concrete sections in order.
3. Decode: emits HTML. Here the syntactic circuits do heavy lifting — an open <div> creates a strong attention signal that must eventually be discharged by </div>. This is bracket-matching implemented as attention, and it's why generated code is structurally valid even when semantically wrong.
4. Each token conditions the rest. Having written a hero section with a specific headline, the CTA later in the page will echo its phrasing — because that's what coherent documents do, and coherence is what was optimized.

Where it breaks and why: it will confidently produce a plausible-looking page that may use a CSS property that doesn't exist, or a framework API from a version that changed. It's generating from a distribution over plausible code, and plausible ≠ correct. It has no compiler.

16. Worked scenario: a compiler pass

Prompt: "Write an LLVM pass that eliminates redundant loads."

Knowledge assembled from the MLPs: SSA form, dominance relations, alias analysis, LoadInst and StoreInst APIs, the runOnFunction signature, memory dependence.

Reasoning assembled from circuits: the actual algorithm — walk the dominator tree, track available loads, kill on any aliasing store, replace dominated redundant loads with the earlier value.

Why it can do this: the LLVM codebase, its documentation, dozens of textbooks, and thousands of blog posts are in the training data. But more importantly, the abstractions generalize: "dominance," "available expressions," "kill sets" are structures the model represents, not strings it memorized. Ask for the same pass targeting a different IR and it adapts, because the structure transfers.

Where it breaks: it will get the algorithm right and the API version wrong. Reasoning circuits generalize; specific API signatures are memorized facts, and memorized facts decay with training-data frequency and go stale with time. The intuition to carry: trust its structure, verify its specifics.

17. Failure modes, and what each one reveals about the architecture

Failures are the best evidence for the mechanism, because they map onto architectural constraints exactly.

Notice that none of these are random. Every one falls out of a structural property. That's a good sign your mental model is correct — a correct model predicts the failures.

18. What is honestly still unknown

Intellectual honesty matters here, because a lot of writing on this topic overclaims.

Established: the architecture; superposition; specific circuits (induction, IOI, and dozens more); feature extraction via sparse autoencoders; task vectors; scaling laws.

Not established:

• No end-to-end account of any complex behavior. Nobody can trace why a specific 500-token compiler answer came out correct. We have footholds, not a map.
• Why in-context learning is as good as it is. Mechanisms are known in pieces; the full story isn't.
• Whether chain of thought is faithful to the actual computation.
• What the mid-layer "planning" representations really are — evidence exists that models plan several tokens ahead (e.g. picking a rhyme word before writing the line that leads to it), but the general structure is unclear.
• Whether anything like understanding is present in a sense beyond functional. This is partly empirical and partly a question about what the word means, and pretending it's settled in either direction is unjustified.

19. Every concept, defined and modeled

Each entry: what it literally is, then the mental model that makes it make sense. Where a compiler analogy helps, it's there.

19.1 The substrate: numbers and shapes

19.2 Architecture

19.3 Training

19.4 Inference

19.5 Interpretability

19.6 Practical and operational

19.7 Behavior and failure

Appendix A: the compressed mental model

1. Your text becomes tokens; tokens become vectors.
2. Those vectors are a shared workspace, one column per position, and every layer adds to rather than replaces them.
3. Attention moves information sideways — learned retrieval, recomputed 5,120 times per token, where "what to match" and "what to copy" are separate learned functions.
4. MLPs add knowledge vertically — ~2.3 million learned key-value slots.
5. Each vector holds a sparse combination of millions of learned features, packed via near-orthogonality. Width was never the constraint.
6. Depth is composition. By mid-network, your intent exists as an extractable vector, independent of your wording.
7. The final vector projects to a distribution over 128K tokens; one is sampled; the loop runs again.
8. It knows what you meant because predicting text written by intentional agents requires modeling intent, and compression made that the cheapest available strategy.
9. Generating tokens buys serial compute; that's what chain of thought is for.
10. It fails exactly where the architecture says it should. That's the strongest evidence the model above is right.

Appendix B: the analogy that actually holds

Not a search engine. Not a database. Not a person.

A very large, very fast, learned interpreter — running a program written by gradient descent, on data supplied by your prompt. The weights are the program. Your prompt is the input and a runtime reconfiguration of which parts of the program execute. The output is what that program computes.

The reason it feels like understanding is that the program it learned is, in substantial part, a model of the people who wrote its training data — including their intentions.

Further reading (primary sources)

• A Mathematical Framework for Transformer Circuits — Elhage et al., 2021
• In-context Learning and Induction Heads — Olsson et al., 2022
• Toy Models of Superposition — Elhage et al., 2022
• Interpretability in the Wild (the IOI circuit) — Wang et al., 2022
• Transformer Feed-Forward Layers Are Key-Value Memories — Geva et al., 2021
• Scaling Monosemanticity — Anthropic, 2024
• Function Vectors in Large Language Models — Todd et al., 2023
• Training Compute-Optimal Language Models (Chinchilla) — Hoffmann et al., 2022