RoPE and ALiBi explained, not named

What it is

Attention is permutation-equivariant: shuffle the input tokens and the outputs shuffle identically. Nothing in softmax(QK^T)V knows that token 3 comes before token 7. So position must be injected, and how you inject it determines whether the model can handle sequences longer than it was trained on.

Three generations of answer:

Method	Mechanism	Extrapolates?
Learned absolute (GPT-2, BERT)	A trainable vector per position, added to embeddings	No. Position 4097 has no vector at all
RoPE (Llama, Mistral, Qwen, GPT-NeoX)	Rotate Q and K by an angle proportional to position	Poorly on its own; extends well with scaling
ALiBi (BLOOM, MPT)	Add a linear distance penalty to attention scores	Yes, natively

RoPE is what nearly every current model uses, and understanding it is the difference between "we extended the context to 128k" being a config change you can reason about and a magic incantation.

What this is confused with: "context length" is not one number. A model has a trained context length (what it saw during training) and an effective one (where quality holds up). A model advertised at 128k that was trained at 8k and extended by interpolation has a real quality curve that degrades well before 128k, and the "needle in a haystack" benchmarks that report otherwise are measuring retrieval of a single fact, not reasoning over the whole window.

The problem it solves

Absolute learned embeddings fail in two specific ways.

They cannot extrapolate at all. A model trained with 2,048 position vectors has literally nothing to use at position 2,049. Generation past the trained length produces garbage, not degraded output.

And they encode the wrong thing. What attention needs is usually relative position: "this token is 5 back" is more useful and more transferable than "this token is at index 1,847." A model that learns absolute positions must learn the same relative relationship separately at every offset.

The requirement, stated properly: the attention score between positions m and n should depend on m - n, not on m and n individually. That single property is what RoPE achieves exactly and what ALiBi achieves by construction.

Mechanics

RoPE: rotate, do not add

Absolute embeddings add a position vector to the token embedding. RoPE rotates the query and key vectors by an angle proportional to their position, in 2D pairs of dimensions.

Take the query vector for position m, split into d/2 pairs. Rotate pair i by angle m * θ_i:

$$\theta_i = 10000^{-2i/d}$$

def rope(x, positions, d):
    # x: [B, H, S, d_h], positions: [S]
    # Each 2D pair (2i, 2i+1) rotates by position * theta_i.
    theta = 10000.0 ** (-torch.arange(0, d, 2).float() / d)   # [d/2]
    angles = positions[:, None] * theta[None, :]              # [S, d/2]
    cos, sin = angles.cos(), angles.sin()

    x1, x2 = x[..., 0::2], x[..., 1::2]        # even and odd dims
    return torch.stack([
        x1 * cos - x2 * sin,                    # standard 2D rotation
        x1 * sin + x2 * cos,
    ], dim=-1).flatten(-2)

Why rotation gives relative position, in one line of algebra. A rotation by angle α is multiplication by e^{iα} in the complex plane. So for query at m and key at n:

$$\langle R_m q, R_n k \rangle = \text{Re}\left[(q e^{im\theta})\overline{(k e^{in\theta})}\right] = \text{Re}\left[q \bar{k} e^{i(m-n)\theta}\right]$$

The dot product depends only on m - n. The absolute positions cancel exactly. That is not an approximation or an empirical finding; it is a property of rotations, and it is why RoPE works so cleanly.

The frequency spread matters. With θ_i = 10000^{-2i/d}:

i = 0    (first pair):    theta = 1.0        period ~6 tokens      (fine detail)
i = 32                    theta = 0.0316     period ~199 tokens
i = 63   (last pair):     theta = 0.0001     period ~62,832 tokens (coarse)

High-frequency pairs encode local position, low-frequency pairs encode global position. That decomposition is exactly what the context-extension methods exploit.

Extending RoPE beyond the trained length

The problem: a model trained at 4k has never seen the rotation angles that positions past 4k produce. Position 8,192 in the highest-frequency dimension has wrapped around many times into angles the model has seen, but in combinations it has not.

Position Interpolation (PI). Squeeze positions into the trained range by dividing:

# Trained at 4096, want 32768. Scale factor 8.
positions = positions / 8.0        # position 32768 -> 4096

Every position now falls inside what the model saw. Simple, works with brief fine-tuning, and it compresses high-frequency detail: tokens 1 and 2 now differ by 0.125 of a position unit where the model learned them as 1 apart, so fine-grained local ordering degrades.

NTK-aware scaling. The insight is that PI's uniform squeeze is wrong: high-frequency dimensions (local position) should be squeezed less and low-frequency dimensions (global position) more. Achieve it by changing the base rather than the positions:

# Instead of scaling positions, scale the base.
base = 10000 * (scale ** (d / (d - 2)))     # for scale = 8, base ~ 10000 * 8.4
theta = base ** (-torch.arange(0, d, 2).float() / d)

This preserves local resolution and stretches the long-range dimensions, and it often works with no fine-tuning at all, which is why it spread fast in the open-model community.

YaRN refines this further: interpolate only the dimensions whose wavelength exceeds the trained context, leave short-wavelength ones untouched, and apply a temperature correction to the attention scores. It reaches longer extensions with less fine-tuning data than PI, and it is what several long-context open models use.

Trained 4k -> extended 32k:
  PI:         works, needs fine-tuning, local detail degrades
  NTK-aware:  often works with no fine-tuning
  YaRN:       best quality per unit of fine-tuning data

ALiBi: no embeddings, just a penalty

ALiBi adds no position information to the vectors at all. It adds a linear bias to the attention scores based on distance:

# scores: [B, H, S, S]
distance = query_pos[:, None] - key_pos[None, :]      # [S, S], >= 0 causally
scores = scores - m_h * distance                       # per-head slope m_h

The slopes are a fixed geometric sequence, one per head:

H = 8 heads:  m = 1/2, 1/4, 1/8, 1/16, 1/32, 1/64, 1/128, 1/256

Heads with a steep slope attend locally (distant tokens get a large negative bias and are effectively masked out); heads with a shallow slope attend globally. The model gets a built-in range of receptive fields, and it never has to learn what position means.

The extrapolation property falls out for free: the bias formula is defined for any distance, so a model trained at 2k runs at 8k with no modification, no interpolation, and no fine-tuning. Its quality degrades gradually rather than collapsing.

The cost is that ALiBi has a strong recency prior baked in. Distant tokens are penalised monotonically, so a fact 6,000 tokens back is systematically disadvantaged relative to one 60 tokens back, regardless of relevance. For long-context retrieval tasks that is the wrong inductive bias, and it is the main reason the field consolidated on RoPE plus extension methods rather than on ALiBi.

A worked example: extending a 4k model to 32k, three ways

A team fine-tuning an open 7B model (RoPE, trained at 4,096) for document analysis needing 32k context. They evaluated three approaches on the same held-out task: answer questions about a contract, with the relevant clause placed at controlled depths.

Baseline: no modification, just feed 32k.

context used    accuracy
  4,096           84.2%
  8,192           31.7%
 16,384            6.1%
 32,768            2.8%     (essentially random)

Past the trained length, quality collapses rather than degrades. This is what "cannot extrapolate" means concretely: not worse output, but incoherent output.

Position Interpolation, scale 8, with 1B tokens of fine-tuning at 32k.

context used    accuracy
  4,096           79.1%     <- WORSE than baseline at short context
  8,192           77.4%
 16,384           74.8%
 32,768           71.2%

Long context works. Short context regressed by 5.1 points, which is the compression of local detail showing up: at scale 8, adjacent tokens are 0.125 position units apart where the model learned them as 1 apart, and fine local ordering suffers. That short-context regression is the cost people do not measure, because they evaluate the new capability and not the old one.

NTK-aware scaling, no fine-tuning at all.

context used    accuracy
  4,096           83.8%     <- essentially unchanged
  8,192           76.2%
 16,384           68.4%
 32,768           58.1%

Short context preserved, long context worse than PI-with-fine-tuning. This is the zero-cost option and it is a genuinely useful data point: most of the capability is available without training anything.

YaRN, with 200M tokens of fine-tuning (a fifth of PI's).

context used    accuracy
  4,096           83.4%
  8,192           81.9%
 16,384           79.6%
 32,768           76.8%

Best on both ends, at a fifth of the fine-tuning budget.

The finding that decided the project, and it came from a test nobody had planned: they varied where in the 32k window the relevant clause sat.

YaRN model, 32k context, accuracy by clause position:
  first 10% of the window       81.2%
  middle 40-60%                 61.4%     <- "lost in the middle"
  last 10%                      84.7%

A 20-point gap between the middle and the edges. The model attends well to the beginning and end of its context and much less well to the middle, which is a documented effect independent of the extension method. The practical consequence for the product: they stopped relying on the model to find the relevant clause in 32k of context and put a retrieval step in front of it, feeding 4k of retrieved passages instead.

                              accuracy    p50 latency    cost/query
32k full document (YaRN)        76.8%        8.4s          $0.094
4k retrieved passages           89.1%        1.1s          $0.011

Better, 8x faster and 9x cheaper. The long-context capability was real and using it was still the wrong design.

That is the honest lesson of the whole exercise: extension methods work, and "the model supports 128k" is not the same as "putting 128k in the prompt is a good idea." See chunking strategies and hybrid retrieval for the alternative.

Production evidence

RoPE (Su et al., 2021, RoFormer) is used by Llama (all versions), Mistral, Qwen, GPT-NeoX, PaLM, and most current open models. Its adoption is close to universal, which is unusual for an architectural component and reflects that the relative-position property falls out exactly rather than approximately.

Position Interpolation (Chen et al., Meta, 2023) extended Llama to 32k with 1,000 fine-tuning steps, and the paper explicitly documents the short-context regression, which is the honest reporting that made the technique trustworthy.

NTK-aware scaling originated in a Reddit post by "bloc97" in mid-2023 and was adopted into production stacks within weeks, which is a notable case of a community contribution becoming standard practice ahead of any paper. YaRN (Peng et al., 2023) formalised and improved it.

ALiBi (Press et al., 2021) was used by BLOOM and MPT. Its extrapolation property was demonstrated convincingly and the field still moved to RoPE, largely because the recency bias hurts long-context retrieval, which turned out to be the application people cared about.

"Lost in the Middle" (Liu et al., 2023) documented the U-shaped attention curve across several models and context lengths: accuracy is high when the relevant information is at the start or end of the context and drops substantially in the middle. It is the single most practically important paper on long context, because it says the capability is not uniform across the window.

Llama 3.1 extended to 128k using a staged approach with continued pretraining at increasing lengths, which is the highest-effort and highest-quality path: not interpolation alone, but actual training data at length.

The debate

RoPE plus extension, or ALiBi? The field chose RoPE, and it is the right choice for retrieval-style long context. ALiBi's native extrapolation is genuinely elegant and its recency bias is the wrong prior when the relevant information may be anywhere in the window. My position: RoPE with YaRN or NTK scaling for anything where position in the context should not determine importance, and ALiBi remains defensible for streaming or conversational workloads where recency genuinely is the right prior.

Is long context a substitute for retrieval? This is the live product question and the answer from the worked example is no, for three reasons. Cost scales linearly with context, so 32k costs 8x what 4k does per query. Latency scales with prefill, so it is noticeably slower. And "lost in the middle" means quality is not uniform, so a fact in the middle of a large context is substantially less likely to be used than the same fact retrieved into a short one. Long context is a capability that makes retrieval systems more forgiving (bigger chunks, less precise retrieval), not a replacement for them.

How much should you trust an advertised context length? Not much on its own. Ask how it was achieved: continued pretraining at length is the strongest, YaRN or NTK with fine-tuning is good, and pure interpolation without fine-tuning is the weakest. Needle-in-a-haystack benchmarks measure single-fact retrieval and overstate real capability, because they do not test reasoning that requires combining information from multiple positions in the window. Evaluate on your own task with the relevant information placed at varying depths, which takes an afternoon and is the only measurement that matters.

Should you extend a model's context yourself? Only if you cannot buy the capability. NTK-aware scaling is a configuration change with no training and gets you a useful fraction of the way, so it is worth trying first. Fine-tuning with YaRN is a real project requiring long-context training data, which is scarce and expensive to construct. For most teams the ordering is: use a model that ships with the context you need, then retrieval, then extension as a last resort.

Follow-up Q&A

"Why does attention need position information at all?"

softmax(QK^T)V is permutation-equivariant: permute the input tokens and the outputs permute identically. Nothing in the operation distinguishes "the cat sat" from "sat the cat." Position must be injected somewhere, and where you inject it determines whether the model can handle lengths it never saw. Adding a learned vector per position (GPT-2) means position 4,097 has no vector at all.

"How does RoPE encode relative position?"

It rotates the query and key vectors by an angle proportional to their absolute positions. Because a rotation is multiplication by e^{iθ} in the complex plane, the dot product of a query rotated by m and a key rotated by n depends only on e^{i(m-n)θ}: the absolute positions cancel exactly. So you apply absolute rotations and the attention score sees only the relative offset, which is exactly the property you want and it holds algebraically rather than approximately.

"How do you extend a RoPE model's context?"

Three methods in increasing sophistication. Position Interpolation divides positions by a scale factor so everything falls in the trained range: simple, needs fine-tuning, and it compresses local detail so short-context quality regresses. NTK-aware scaling changes the RoPE base instead, squeezing low-frequency (global) dimensions more than high-frequency (local) ones, which often works with no fine-tuning. YaRN interpolates only the dimensions whose wavelength exceeds the trained context and adds a temperature correction, giving the best quality per unit of fine-tuning data.

"What is ALiBi and why did the field not adopt it?"

It adds no positional information to the vectors, only a linear penalty to attention scores proportional to distance, with a different slope per head so some heads attend locally and some globally. It extrapolates natively, because the penalty is defined for any distance. The field went to RoPE because ALiBi's monotonic distance penalty is a strong recency prior, and for long-context retrieval the relevant information may be anywhere, so systematically discounting distant tokens is the wrong inductive bias.

"A model advertises 128k context. What do you actually check?"

How it was achieved: continued pretraining at length is strongest, YaRN or NTK with fine-tuning is good, pure interpolation is weakest. Then evaluate on your own task with the relevant information placed at varying depths, because of "lost in the middle": models attend well to the start and end of the window and much less well to the middle, and the gap can be 20 points. Needle-in-a-haystack benchmarks test single-fact retrieval and overstate capability, since they do not test combining information across positions.

"When is long context better than retrieval?"

When the whole document genuinely must be reasoned over jointly and cannot be decomposed: a contract where clauses interact, a codebase change spanning many files. Otherwise retrieval wins on all three axes: cost is linear in context so 32k is 8x the price of 4k, latency scales with prefill, and quality is better because the retrieved passages sit at the edges of a short window rather than in the middle of a long one. In one measured case retrieval was 12 points more accurate, 8x faster and 9x cheaper than feeding the full document.

Common misconceptions

"RoPE adds position embeddings." It rotates Q and K. Nothing is added, and the rotation is applied inside each attention layer rather than once at the input, which is why it composes with the depth of the model.

"A model can just be run at longer context." Past the trained length, quality collapses rather than degrades: in one measurement, 84 percent accuracy at 4k became 2.8 percent at 32k, which is random. Extension requires an explicit method.

"Extending context is free." Position Interpolation regressed short-context accuracy by 5 points in the worked example, because compressing positions degrades fine local ordering. Measure the old capability, not just the new one.

"128k context means the model uses all 128k equally." "Lost in the middle" is a robust, cross-model effect: accuracy is high at the start and end of the window and substantially lower in the middle. Position within the context affects whether information is used.

"ALiBi is strictly worse." It extrapolates natively with no modification, which RoPE cannot do. Its recency bias is wrong for retrieval and right for streaming and conversation. It lost on the application the field prioritised, not on the merits generally.

Interview delivery note

Say this verbatim: "RoPE rotates queries and keys by an angle proportional to position, and because rotation is multiplication by e^{iθ}, the dot product depends only on the difference of the positions. Absolute rotations, relative result, exactly rather than approximately. That is why extension methods work by rescaling the rotation rather than retraining position embeddings that do not exist." The mechanism and why it enables the extension story, in two sentences.

The senior-versus-staff separator is "lost in the middle" and its product consequence. A senior engineer explains RoPE and names YaRN. A staff engineer adds that extending context does not make the window uniform, that accuracy at the middle of a 32k window can be 20 points below the edges, and therefore that a retrieval step in front of a long-context model is usually better than using the long context, with the numbers: more accurate, 8x faster, 9x cheaper. Knowing that the capability is real and using it is still the wrong design is the judgment being tested.

The second signal is measuring the short-context regression when extending. Almost everyone evaluates the new capability; the 5-point drop at 4k after Position Interpolation is the cost that goes unmeasured.

The Staff & Team Lead Interview Reference