How the window got long: positions, RoPE, and the stretch tricks

TL;DR. A million-token window is not a bigger buffer; it is a geometry trick applied three times. Modern models encode position by rotating query and key vectors (RoPE), one rotation speed per dimension pair, so relative position falls out of the dot product. The catch: slow-turning dimensions only sweep a partial arc during training, so positions past the trained length feed attention phases it has never seen. The lab builds RoPE, verifies the relative-position property to 1e-14, shows that at training length 512 exactly half the dials never complete a revolution, and scores the three fixes at a 4x stretch: raw extrapolation leaves 13 of 32 dials deep out-of-distribution, position interpolation zeroes that but blurs every dial 4x, and NTK-style base rescaling keeps the fast dials sharp while mostly fixing the slow ones, the imperfection YaRN's per-frequency ramp then closes. Real long-context models stack this with staged training (Llama 3: six stages, 8K to 128K, ~800B tokens) and distributed attention (Ring Attention), and the punchline for this book is Chapter 33's: stretched positions plus finite long-data is why effective length trails advertised length.

Contents

Position is a problem you can rotate away
The lab: dials, arcs, and three stretches
Reading the lab
From tricks to shipping models
What this means for your window
Further reading
Takeaways

Chapter 14 covered how attention got cheap at length; this chapter covers how it got valid at length, which is a different problem. Attention itself is permutation-blind: without positional information, "the cache invalidates the prefix" and "the prefix invalidates the cache" are the same bag of tokens. Something must encode order, and the choice of that something is what decides how far a window can stretch.

Position is a problem you can rotate away

The first transformers added a position vector to each token embedding (sinusoidal or learned). That couples position to content in the residual stream, and learned versions simply have no entry for position 2049 if trained to 2048. Two 2021-era ideas replaced it:

RoPE (Su et al., 2021, "RoFormer"): encode position m by rotating each query/key dimension pair i by angle $m \theta_i$, with $\theta_i = \text{base}^{-2i/d}$ (base 10,000). Because rotation matrices compose, the attention score between positions m and n depends only on $m - n$: exact relative position, no vectors added to the content stream, and a spectrum of speeds: pair 0 turns a full radian per token (a fast dial that distinguishes neighbors), the last pair turns ~0.0001 radian per token (a slow dial that encodes coarse, document-scale distance). Nearly every current open model (Llama, Qwen, DeepSeek, Mistral) ships RoPE.
ALiBi (Press, Smith, Lewis, ICLR 2022): skip embeddings entirely and subtract a distance-proportional penalty from attention scores. It extrapolates gracefully by construction (their 1024-trained model matched a 2048-trained baseline at 2048) but ties every head to a fixed recency bias; the field mostly chose RoPE's expressiveness and then spent two years fixing its extrapolation instead.

Don't be confused. RoPE has no maximum position; the math runs forever. What breaks at long range is not the formula but the training distribution: each dimension pair only ever experienced angles up to $\theta_i \times L_{train}$, and a network is only reliable on inputs like those it saw. "Context window: 200k" is a statement about training and validation, not about the encoding, which is exactly why it can be stretched after the fact by remapping positions back into the trained range.

The lab: dials, arcs, and three stretches

"""RoPE from scratch: why long windows need tricks, and what the tricks do.

Rotary Position Embedding (Su et al., 2021) encodes position by ROTATING each
query/key vector: dimension pair i turns by angle m * theta_i at position m,
with theta_i = base^(-2i/d). Because rotations compose, the attention dot
product between positions m and n depends only on m - n: relative position
for free, no position embeddings added to the residual stream.

The catch appears when you run PAST the trained length. Each dimension pair
is a dial that turns at its own speed; fast dials complete many revolutions
within training and have shown the model every phase, but slow dials only
ever swept a partial arc. Positions beyond the trained length push the slow
dials into phases the model has NEVER seen: out-of-distribution inputs to
every attention head. The extension tricks are different ways to avoid that:

  interpolation (PI)   squeeze new positions into the trained arc (divide all
                       angles by the extension factor); nothing is OOD, but
                       neighboring positions crowd 4x closer on every dial
  NTK-style rescale    raise the base so slow dials are interpolated while
                       fast dials keep their speed; local resolution survives
                       (YaRN refines this per-frequency and adds a softmax
                       temperature)

This lab builds RoPE, verifies the relative-position property numerically,
counts the OOD dials at a 4x extension, and compares the fixes. NumPy only.

Run:  python3 rope_lab.py
"""

import numpy as np

D, BASE, L_TRAIN, EXT = 64, 10_000.0, 512, 4     # head dim, base, lengths
L_NEW = L_TRAIN * EXT
PAIRS = D // 2
i = np.arange(PAIRS)

def thetas(base):
    return base ** (-2.0 * i / D)

def rotate(x, m, th):
    """Rotate vector x (dim D) to position m under frequencies th."""
    ang = m * th
    c, s = np.cos(ang), np.sin(ang)
    x1, x2 = x[0::2], x[1::2]
    out = np.empty_like(x)
    out[0::2] = x1 * c - x2 * s
    out[1::2] = x1 * s + x2 * c
    return out

# ---------------------------------------------------------------------------
# 1. The relative-position property, verified numerically.
# ---------------------------------------------------------------------------
rng = np.random.default_rng(0)
q, k = rng.standard_normal(D), rng.standard_normal(D)
th = thetas(BASE)
worst = max(abs(rotate(q, 100, th) @ rotate(k, 40, th)
                - rotate(q, 100 + s, th) @ rotate(k, 40 + s, th))
            for s in (1, 17, 400, 5000))
print("=== Relative position, verified ===")
print(f"score(100,40) vs score(100+s,40+s), worst |diff| over shifts: {worst:.2e}")
print("(the dot product depends only on m-n: that is RoPE's whole contract)\n")

# ---------------------------------------------------------------------------
# 2. The dials, and who finished a revolution during training.
# ---------------------------------------------------------------------------
arc = th * (L_TRAIN - 1)                       # angle each pair swept in training
full = int((arc >= 2 * np.pi).sum())
print("=== The dials at training length 512 ===")
print(f"fastest pair: {th[0]:.3f} rad/step ({arc[0] / (2 * np.pi):.0f} revolutions in training)")
print(f"slowest pair: {th[-1]:.6f} rad/step ({np.degrees(arc[-1]):.1f} degrees total: a sliver)")
print(f"pairs that completed a full revolution: {full}/{PAIRS}")
print(f"pairs that saw only a partial arc:      {PAIRS - full}/{PAIRS}  <- the extension problem\n")

# ---------------------------------------------------------------------------
# 3. Three ways to run at 4x, scored.
# ---------------------------------------------------------------------------
def ood_pairs(th_used, scale, pos):
    """Pairs whose phase at `pos` was never seen in training, and how far
    past the trained arc they sit (severity, in radians)."""
    seen_arc = thetas(BASE) * (L_TRAIN - 1)          # what training exposed
    phase = (th_used * scale * pos) % (2 * np.pi)
    flag = (seen_arc < 2 * np.pi) & (phase > seen_arc + 1e-9)
    excess = float((phase - seen_arc)[flag].mean()) if flag.any() else 0.0
    return int(flag.sum()), excess

ntk_base = BASE * EXT ** (D / (D - 2))               # the NTK-aware rescale
schemes = [
    ("extrapolate (no fix)", thetas(BASE),     1.0),
    ("interpolate (PI)",     thetas(BASE),     (L_TRAIN - 1) / (L_NEW - 1)),
    ("NTK-aware base",       thetas(ntk_base), 1.0),
]
print(f"=== Running at {L_NEW} (4x), per scheme ===")
print(f"{'scheme':<22}{'OOD pairs':>10}{'mean excess':>12}{'adjacent-step angle':>21}")
print("-" * 65)
for name, th_used, scale in schemes:
    ood, excess = ood_pairs(th_used, scale, L_NEW - 1)
    adj = th_used[0] * scale                          # fastest dial's step
    print(f"{name:<22}{f'{ood}/{PAIRS}':>10}{excess:>8.2f} rad{adj:>18.3f} rad")
print("-" * 65)
print("""extrapolation leaves a third of the dials deep in phases the model
never trained on (the mean excess is the whole story: radians of unexplored
dial). PI zeroes the OOD column by squeezing every position into the trained
arc, at the price of slowing EVERY dial 4x, including the fast ones that
tell neighboring tokens apart. The NTK rescale keeps the fastest dial at its
full 1.000 rad and pushes the slow dials back toward the trained arc; the
middle dials still peek slightly past it, which is exactly the imperfection
YaRN's per-frequency ramp (plus a softmax temperature) was built to close.
Every '1M-token window' you rent is some refinement of this move followed by
long-context training, never 'the same dials, run further'.""")

Verified output:

=== Relative position, verified ===
score(100,40) vs score(100+s,40+s), worst |diff| over shifts: 2.13e-14
(the dot product depends only on m-n: that is RoPE's whole contract)

=== The dials at training length 512 ===
fastest pair: 1.000 rad/step (81 revolutions in training)
slowest pair: 0.000133 rad/step (3.9 degrees total: a sliver)
pairs that completed a full revolution: 16/32
pairs that saw only a partial arc:      16/32  <- the extension problem

=== Running at 2048 (4x), per scheme ===
scheme                 OOD pairs mean excess  adjacent-step angle
-----------------------------------------------------------------
extrapolate (no fix)       13/32    1.27 rad             1.000 rad
interpolate (PI)            0/32    0.00 rad             0.250 rad
NTK-aware base             14/32    0.48 rad             1.000 rad
-----------------------------------------------------------------
extrapolation leaves a third of the dials deep in phases the model
never trained on (the mean excess is the whole story: radians of unexplored
dial). PI zeroes the OOD column by squeezing every position into the trained
arc, at the price of slowing EVERY dial 4x, including the fast ones that
tell neighboring tokens apart. The NTK rescale keeps the fastest dial at its
full 1.000 rad and pushes the slow dials back toward the trained arc; the
middle dials still peek slightly past it, which is exactly the imperfection
YaRN's per-frequency ramp (plus a softmax temperature) was built to close.
Every '1M-token window' you rent is some refinement of this move followed by
long-context training, never 'the same dials, run further'.

Reading the lab

Half the head is a partially explored dial. At training length 512, 16 of 32 pairs never complete a revolution; the slowest swept 3.9 degrees of its circle. Those pairs are where long-range position lives, and they are precisely the ones that go out-of-distribution when you run long. This is the quantitative form of "RoPE does not extrapolate."
The three fixes are one trade-off. Raw extrapolation preserves local resolution (1.000 rad between neighbors) but feeds 13 dials phases 1.27 radians past anything trained. Position interpolation (Chen et al., 2023, with kaiokendev's SuperHOT blog as the community prior art) divides all positions by the stretch factor: OOD goes to zero, LLaMA reached 32k with under a thousand fine-tuning steps, but every dial slows 4x, including the fast ones that tell "the cache" from "cache the." NTK-aware rescaling (bloc97's r/LocalLLaMA post, 2023: raise the base instead of scaling positions) is the frequency-aware compromise: the fastest dial keeps its full 1.000 rad while slow dials are interpolated. The lab's mean-excess column shows its known flaw, middle dials still drifting 0.48 rad past the trained arc.
YaRN is the lab's table, turned into a recipe. YaRN (Peng, Quesnelle, Fan, Shippole, ICLR 2024) makes the compromise explicit per frequency: leave high-frequency dims untouched, linearly interpolate low-frequency dims, blend the band between, and scale the softmax temperature ($\sqrt{1/t} = 0.1 \ln s + 1$). It reported 10x fewer tokens and 2.5x fewer steps than earlier methods to reach the same extensions, and it shipped: DeepSeek-V3's config declares rope_scaling: {"type": "yarn", "factor": 40} over a 4,096 original window, which is exactly $4{,}096 \times 40 = 163{,}840$ positions. LongRoPE (Microsoft, 2024) pushes the same idea to its limit, searching per-dimension rescale factors evolutionarily and reaching a 2M-token window.

From tricks to shipping models

A frontier long-context model is the position trick plus two more ingredients, and the Llama 3 report is unusually explicit about both:

Staged length training. Llama 3 405B was pre-trained at 8K and then extended "in six stages, starting from the original 8K context window and ending in the final 128K context window," spending "approximately 800B training tokens" on the long-context stage alone. Length is grown gradually so the model adapts each stretch before the next; you cannot buy a long window with remapping alone, because remapped positions still need data that actually exercises them.
Distributed attention. A 128K-token attention matrix does not fit one device. Ring Attention (Liu, Zaharia, Abbeel, ICLR 2024) computes attention blockwise across devices, passing KV blocks around a ring while overlapping communication with compute, so sequence length scales with device count "without resorting to approximations"; Llama 3 productionizes the idea as context parallelism (the sequence split into 2xCP chunks per rank for load balancing). This is also the compute story behind every 1M-window API price premium: long attention is genuinely more expensive to serve.

And the honest third ingredient: evaluation, because stretched does not mean solved. RULER (Hsieh et al., COLM 2024) tested 17 long-context models on 13 tasks and found that of models claiming 32K+ windows, "only half of them can maintain satisfactory performance at the length of 32K." The gap between advertised and effective length that Chapter 33 measured behaviorally now has its mechanism: remapped dials, finite long-range training data, and attention spread thin across positions the model met late in training and rarely.

What this means for your window

Four consequences for practice, each one an earlier chapter's advice with the physics attached:

The window is a budget, not a room. Positions near the stretch limit are the least exercised in training, so quality is not uniform across the window; effective length is measured, never quoted (Chapter 33). Filling a 1M window because it exists is spending your task's attention on the model's worst-trained regime.
Position of content matters and will keep mattering. The lost-in-the-middle curve is not a bug awaiting a patch; it is downstream of how position is encoded and trained. Claude Code putting rules at the front and reminders at the tail (Chapter 28) is engineering around geometry.
Long context and retrieval are complements, not rivals. The stretch tricks make big windows possible; they do not make every token in them equally usable. Retrieval (Chapter 31) and compaction (Chapter 11) decide what deserves the well-trained part of the window.
Expect the numbers to move. Bases, factors, and stages are per-model recipes; windows will keep growing and the quality curves will keep shifting. The lab's dials are the durable part: any RoPE-family stretch is some allocation of OOD-avoidance versus local resolution, and you can now read a rope_scaling config block and know which trade it chose.

Remember. "1M tokens" means: positions were remapped into the arcs the model knows, the model was then trained long in stages, and attention was sharded across machines to serve it. All three cost real money, which is why long context carries price premiums, and none of the three makes token 900,000 as well-handled as token 900. Long windows are real; uniform windows are not.

Takeaways

RoPE encodes position as per-dimension rotation; relative position falls out of the dot product (verified to 1e-14 in the lab), and each dimension pair is a dial with its own speed, fast dials for neighbors, slow dials for document-scale distance.
The extension problem is distributional: at training length 512, half the dials never complete a revolution, so longer runs feed attention unseen phases. Extrapolation breaks (13/32 dials, 1.27 rad deep), interpolation blurs (every dial 4x slower), NTK rescaling splits the difference, and YaRN tunes it per frequency; DeepSeek-V3 ships exactly that recipe at factor 40.
Shipping long windows adds staged length training (Llama 3: six stages to 128K, ~800B tokens) and distributed attention (Ring Attention / context parallelism), which is where the long-context price premium comes from.
Effective length trails advertised length (RULER: only half of 32K+ claimants hold up at 32K), because stretched positions are the model's least-trained regime; measure your own effective length and spend the good part of the window on the work.

👉 That closes the model-internals arc: tokens made, tokens processed, and the window that holds them stretched to a million. From here the reference part waits: the ecosystem map, the tool tour, and the benches that put numbers on whatever this book has not already measured. Continue to The open-source landscape.

Context Engineering in Depth