References and further reading

How to use this page: you do not need any of it to have understood the book. Treat it as a shelf to reach for when a specific question comes up. It collects the projects named throughout, the papers behind the ideas, a glossary of the terms the chapters built up, and the runnable code so you can find a demo again.

A note on links: the field moves quickly and URLs rot, so most entries name the thing and, where I am confident of the address, give a plain GitHub org or root domain. Everything else is a name you can type into a search engine. Nothing here is fabricated; where I was unsure of an exact address I left the name and let you search.

Projects, by lever

  • Prompt and context compression (Ch 3): LLMLingua and LLMLingua-2 (github.com/microsoft/LLMLingua); RTK (github.com/rtk-ai/rtk), Headroom (github.com/chopratejas/headroom), lean-ctx (github.com/yvgude/lean-ctx).
  • Output token reduction (Ch 4): the provider's own controls (Anthropic effort, structured outputs, stop sequences); output shapers such as caveman and Headroom's shaper.
  • Code and structure-aware context (Ch 5): CodeCompressor, lean-ctx, tree-sitter (tree-sitter.github.io), Aider's repo map.
  • Retrieval and chunking (Ch 31): LlamaIndex (github.com/run-llama/llama_index) and LangChain text splitters; vector indexes Chroma (github.com/chroma-core/chroma), Qdrant, Weaviate, Milvus, pgvector; rerankers (Cohere, open cross-encoders); the hnsw and ivf-pq books on this site for the index internals.
  • KV-cache and prefix caching (Ch 6): provider-native prompt caching (Anthropic and other vendors); CacheAligner-style tooling.
  • Semantic and response caching (Ch 7): GPTCache (github.com/zilliztech/GPTCache), Redis LangCache (redis.io).
  • KV-cache serving optimization (Ch 8): vLLM (github.com/vllm-project/vllm, PagedAttention), SGLang (github.com/sgl-project/sglang, RadixAttention).
  • Agent memory and persistence (Ch 9): Mem0 (github.com/mem0ai/mem0), Letta (github.com/letta-ai/letta, formerly MemGPT), Zep (github.com/getzep/zep).
  • Temporal knowledge graphs (Ch 10): Graphiti (github.com/getzep/graphiti), Zep.
  • Context-window compaction (Ch 11): Letta tiered memory; the provider's own compaction and context-editing features (Anthropic).
  • Failure and procedural learning (Ch 12): LangMem (github.com/langchain-ai/langmem), headroom learn.
  • Context orchestration (Ch 13): LangGraph (github.com/langchain-ai/langgraph), lean-ctx.
  • Long-context attention efficiency (Ch 14): DeepSeek (github.com/deepseek-ai, sparse attention DSA and Multi-head Latent Attention), MiniMax (github.com/MiniMax-AI, lightning/linear attention).
  • Context evals (Ch 33): RULER (github.com/NVIDIA/RULER), Greg Kamradt's needle-in-a-haystack harness, LongBench, Chroma's context-rot protocol (research.trychroma.com), promptfoo for running grids in CI.
  • Hostile context / prompt injection (Ch 32): OWASP Top 10 for LLM Applications (owasp.org); the model's own operator channel and Claude Code permission model as the primary defenses.
  • Measurement, benchmarking, and metering (Ch 25, Ch 26): ccusage (github.com/ryoppippi/ccusage, transcript-based cost reports), Repomix (github.com/yamadashy/repomix, repository packing with token counts), Langfuse (github.com/langfuse/langfuse, traces and token dashboards), promptfoo (github.com/promptfoo/promptfoo, prompt benchmarking with cost tracking), LiteLLM (github.com/BerriAI/litellm, gateway with budgets and spend tracking), and Claude Code's built-in OpenTelemetry exporter (code.claude.com/docs/en/monitoring-usage).
  • Benchmarked on the bench (Ch 27): RTK and Headroom (measured on this repo's real traffic), files-to-prompt (github.com/simonw/files-to-prompt), gitingest (github.com/coderamp-labs/gitingest), code2prompt (github.com/mufeedvh/code2prompt), Serena (github.com/oraios/serena, LSP-backed MCP for symbol-level code context), LMCache (github.com/LMCache/LMCache, cross-node KV sharing for vLLM), DSPy (github.com/stanfordnlp/dspy, metric-driven prompt compilation), and the observability alternates Helicone (github.com/Helicone/helicone), Arize Phoenix (github.com/Arize-ai/phoenix), and OpenLLMetry (github.com/traceloop/openllmetry).
  • Decoder controls and constrained decoding (Ch 37): the provider's own output controls (Anthropic structured outputs, strict tools, stop sequences; note that temperature/top_p/top_k return a 400 on the current frontier models); grammar-constrained decoders Outlines (github.com/dottxt-ai/outlines), llguidance (github.com/guidance-ai/llguidance), and XGrammar (github.com/mlc-ai/xgrammar).
  • Language-server code tools (Ch 38): Serena (github.com/oraios/serena, LSP-backed MCP for symbol-level retrieval and editing, built on its solidlsp fork of multilspy), multilspy (github.com/microsoft/multilspy), and the Language Server Protocol spec (microsoft.github.io/language-server-protocol).
  • Monitor-Guided Decoding (Ch 39): monitors4codegen (github.com/microsoft/monitors4codegen, the paper's monitors plus the PragmaticCode / DotPrompts datasets) and multilspy underneath it.
  • Memory systems in depth (Ch 41): Letta (github.com/letta-ai/letta, memory blocks, sleep-time agents, the .af Agent File format), mem0 and OpenMemory MCP (github.com/mem0ai/mem0), Graphiti (github.com/getzep/graphiti), cognee (github.com/topoteretes/cognee), Memori (github.com/GibsonAI/memori), LangMem (github.com/langchain-ai/langmem), the MCP reference memory server (github.com/modelcontextprotocol/servers, package @modelcontextprotocol/server-memory); Anthropic's memory tool and context editing (platform.claude.com/docs).
  • Memory and long-horizon evals (Ch 42): LongMemEval (github.com/xiaowu0162/LongMemEval), LoCoMo (github.com/snap-research/locomo), MemoryAgentBench (github.com/HUST-AI-HYZ/MemoryAgentBench), promptfoo's Anthropic and Claude Agent SDK providers (promptfoo.dev), and Claude Code headless mode (code.claude.com/docs/en/headless) as the harness runner.
  • The caching lineage (Ch 43): vLLM (vllm-project/vllm, PagedAttention and automatic prefix caching), SGLang (sgl-project/sglang, RadixAttention), FlashAttention (github.com/Dao-AILab/flash-attention), Prompt Cache (github.com/yale-sys/prompt-cache), LMCache (github.com/LMCache/LMCache, with CacheGen and CacheBlend underneath), Mooncake (github.com/kvcache-ai/Mooncake), StreamingLLM (github.com/mit-han-lab/streaming-llm), KIVI (github.com/jy-yuan/KIVI).
  • Model internals (Ch 46 to Ch 48): tiktoken (github.com/openai/tiktoken, including its _educational module), SentencePiece (github.com/google/sentencepiece), Anthropic's count_tokens endpoint (docs), Medusa (github.com/FasterDecoding/Medusa), EAGLE (github.com/SafeAILab/EAGLE), the YaRN reference code (github.com/jquesnelle/yarn), LongRoPE (github.com/microsoft/LongRoPE).
  • Claude Code extension surfaces (Ch 49): the skills, hooks, sub-agents, MCP (with tool search), and plugins documentation at code.claude.com/docs; the Agent Skills standard (agentskills.io).

Papers behind the ideas

Named so you can find the current version on arxiv.org or the project page; the exact identifiers change as papers revise, so search the title.

  • LLMLingua and LLMLingua-2 (Microsoft Research): prompt compression by perplexity (v1) and by a learned token classifier distilled from a strong model (v2).
  • MemGPT (now Letta): an operating-system metaphor for LLM memory, with core, recall, and archival tiers the model pages between.
  • PagedAttention ("Efficient Memory Management for Large Language Model Serving"): the paper behind vLLM, treating the KV cache like OS virtual memory.
  • RadixAttention (SGLang): automatic KV prefix sharing across requests via a radix tree.
  • DeepSeek-V2 / V3 technical reports: Multi-head Latent Attention (MLA) for KV compression, and the sparse attention used in later versions.
  • MiniMax-01 technical report: lightning (linear) attention for long sequences.
  • The Curious Case of Neural Text Degeneration (Holtzman et al.): introduced nucleus (top-p) sampling and the case against pure greedy or pure sampling (Chapter 37).
  • Monitor-Guided Decoding of Code LMs with Static Analysis of Repository Context (Agrawal, Kanade, Goyal, Lahiri, Rajamani, NeurIPS 2023; arXiv also titles it "Guiding Language Models of Code with Global Context using Monitors"): masks a code model's logits with the valid members a language server reports, lifting compilation rate across model sizes (Chapter 39).
  • Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (Chhikara, Khant, Aryan, Singh, Yadav, 2025): the two-phase extract-then-reconcile pipeline (ADD/UPDATE/DELETE/NOOP) and the LoCoMo numbers the benchmark dispute is about (Chapter 41).
  • Sleep-time Compute: Beyond Inference Scaling at Test-time (Lin, Snell, Wang, Packer, Wooders, Stoica, Gonzalez, 2025): memory maintenance and pre-thinking moved off the hot path (Chapter 41).
  • LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (Wu, Wang, Yu, Zhang, Chang, Yu, ICLR 2025): five memory abilities, 500 questions, and the 30% accuracy drop (Chapter 42).
  • Evaluating Very Long-Term Conversational Memory of LLM Agents (Maharana, Lee, Tulyakov, Bansal, Barbieri, Fang, ACL 2024): the LoCoMo dataset and its question typology (Chapter 42).
  • Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents (Chen, 2026): the ConstraintRot benchmark; policy violations go from 0% in full context to 30% average after compaction (Chapter 42).
  • Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., NeurIPS 2023): model-graded evaluation validated against humans, with position, verbosity, and self-enhancement biases named (Chapter 42).
  • FlashAttention 1, 2, and 3 (Dao et al. 2022; Dao 2023; Shah et al. 2024): IO-aware exact attention, the kernel under every long cached prefix (Chapter 43).
  • SGLang: Efficient Execution of Structured Language Model Programs (Zheng, Yin, Xie, et al., NeurIPS 2024): RadixAttention, the fleet-wide KV radix tree with LRU eviction (Chapter 43).
  • Prompt Cache: Modular Attention Reuse for Low-Latency Inference (Gim et al., MLSys 2024): reusable prompt modules cached position-independently (Chapter 43).
  • CacheGen (Liu et al., SIGCOMM 2024) and CacheBlend (Yao et al., EuroSys 2025 best paper): the KV cache as compressible, shippable, fusable data; productized as LMCache (Chapter 43).
  • Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (Qin et al., FAST 2025 best paper): datacenter-scale KV pooling behind the Kimi assistant (Chapter 43).
  • StreamingLLM / H2O / SnapKV / KIVI (Xiao et al., ICLR 2024; Zhang et al., NeurIPS 2023; Li et al., NeurIPS 2024; Liu et al., ICML 2024): the lossy branch: attention sinks, heavy-hitter eviction, pre-generation compression, 2-bit KV quantization (Chapter 43).
  • MQA and GQA (Shazeer 2019; Ainslie et al., EMNLP 2023): fewer KV heads, the architectural end of the cache lineage (Chapter 43).
  • Neural Machine Translation of Rare Words with Subword Units (Sennrich, Haddow, Birch, ACL 2016): BPE for open vocabularies, the algorithm under every modern tokenizer (Chapter 46); with Kudo's Unigram/SentencePiece line (2018) as the main alternative.
  • Language Model Tokenizers Introduce Unfairness Between Languages (Petrov, La Malfa, Torr, Bibi, NeurIPS 2023): up to 15x tokenization-length differences across languages (Chapter 46).
  • Scaling Laws for Neural Language Models (Kaplan et al., 2020): source of the 2-FLOPs-per-parameter-per-token forward-pass accounting (Chapter 47).
  • Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., OSDI 2022): continuous batching (Chapter 47).
  • Fast Inference from Transformers via Speculative Decoding (Leviathan, Kalman, Matias, ICML 2023) and Accelerating Large Language Model Decoding with Speculative Sampling (Chen et al., 2023): the twin speculative-decoding papers; Medusa (2024) and EAGLE (2024-2025) are the refinements (Chapter 47).
  • RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., 2021): RoPE (Chapter 48); with ALiBi (Press et al., ICLR 2022) as the main alternative.
  • Extending Context Window of Large Language Models via Positional Interpolation (Chen et al., 2023) and YaRN: Efficient Context Window Extension of Large Language Models (Peng, Quesnelle, Fan, Shippole, ICLR 2024): the stretch line, from linear scaling to per-frequency interpolation; LongRoPE (Ding et al., 2024) is the search-based extreme (Chapter 48).
  • Ring Attention with Blockwise Transformers for Near-Infinite Context (Liu, Zaharia, Abbeel, ICLR 2024) and The Llama 3 Herd of Models (2024): serving and training the stretched window: context parallelism, six-stage length extension (Chapter 48).
  • The provider documentation for prompt caching, token counting, compaction, and context editing is the authority for the exact parameters; consult the claude-api reference for the model you call.

Claude Code and the CLI context tools

Most levers in this book show up in a coding agent (Claude Code) and a small set of CLI tools that sit underneath it. These are the practical entry points; see A professional workflow for them combined end to end.

  • Claude Code (code.claude.com/docs): the agent itself. The context-relevant commands are /cost (token and cache usage), /context (what is filling the window), /compact (summarize the history, with an optional focus instruction), /init (write a CLAUDE.md from the repo), and claude mcp add (connect an external tool over the Model Context Protocol). A small, stable CLAUDE.md serves double duty as a cached prefix and as procedural memory.
  • RTK (github.com/rtk-ai/rtk): a CLI proxy that compresses noisy command output (git, find, tests) before it reaches the agent. rtk init -g installs a shell hook; rtk gain and rtk cc-economics report the savings. Measured on the build box at 71% on git log and 88% on find (Chapter 3).
  • Headroom (github.com/chopratejas/headroom, pip install headroom-ai): a context compression layer (library, proxy, and MCP server) with compress, a CacheAligner for prefix caching, a SemanticCache, a Memory store, and headroom learn for procedural rules.
  • lean-ctx (github.com/yvgude/lean-ctx): a local binary that runs as an MCP server, giving the agent leaner context tools and a CLI-output compressor (lean-ctx serve).
  • caveman (github.com/JuliusBrussee/caveman): an output-side skill that forces a terse style to cut output tokens, and claude-token-efficient (github.com/drona23/claude-token-efficient), a drop-in terse-output CLAUDE.md. Both are weighed honestly in Field notes, which validates the savings claims with measurements.

Books and longer reads

For the systems thinking under these techniques, and the breadth around them:

  • Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly): the best single book on caching, storage, indexes, and eviction. Not LLM-specific, but the mental models are exactly the ones context engineering borrows for caching (Chapter 6, Chapter 8) and memory (Chapter 9).
  • Jay Alammar and Maarten Grootendorst, Hands-On Large Language Models (O'Reilly): a practical, visual tour of tokenization, embeddings, and how models read context.
  • Jay Alammar, "The Illustrated Transformer" (jalammar.github.io): the clearest free explanation of attention, the mechanism behind the KV cache and long-context efficiency (Chapter 6, Chapter 14).
  • Anthropic, "Building effective agents" and the prompt-engineering and context guidance on anthropic.com and platform.claude.com: the provider's own account of assembling and managing context, tools, and memory.
  • Lilian Weng, "LLM Powered Autonomous Agents" (lilianweng.github.io): a widely cited survey of agent memory, planning, and tool use that frames where these levers fit.

Glossary

Terms the chapters built up, in one place.

  • Context. The full token sequence sent to the model on one call: system prompt, tools, retrieved documents, history, and the user's message. See Chapter 1.
  • Token. A sub-word unit from the model's vocabulary; the thing you are billed and windowed in. See Chapter 2.
  • Context window. The maximum number of tokens a model can read at once.
  • BPE (byte pair encoding). The algorithm that builds the tokenizer by merging frequent adjacent pairs. See Chapter 2.
  • Compression ratio. Original tokens divided by compressed tokens. See Chapter 3.
  • Effort / structured output / stop sequence. Provider controls that make the model write fewer output tokens. See Chapter 4.
  • AST (abstract syntax tree). A program's structure as a tree; the basis for selecting code by call graph. See Chapter 5.
  • Query, key, value. The three projections inside attention. See Chapter 6.
  • KV cache. Stored keys and values for past tokens so they are not recomputed each step. See Chapter 6.
  • Prefix caching / prompt caching. Reusing the cached work and charge for a stable prompt prefix across calls. See Chapter 6.
  • Semantic cache. Returning a stored answer for an approximately-similar query. See Chapter 7.
  • Embedding / cosine similarity. A vector for a piece of text, and the angle-based measure of how close two vectors are. See Chapter 7 and Chapter 9.
  • PagedAttention / RadixAttention. Engine techniques: KV memory in fixed-size pages, and KV prefix sharing across requests. See Chapter 8.
  • Extraction / retrieval / invalidation. The three memory operations: pull facts out, fetch the relevant ones, replace stale ones. See Chapter 9.
  • Bi-temporal / validity interval. Tracking when a fact was true (event time) and when the system learned it (ingestion time). See Chapter 10.
  • Compaction. Summarizing old context when the window fills, as opposed to deleting it. See Chapter 11.
  • Procedural memory. Learned how-to rules that change the agent's behavior, stored in its instructions, as opposed to facts. See Chapter 12.
  • Orchestration. Deciding which context, tools, and state to assemble per turn. See Chapter 13.
  • Sparse / linear attention, MLA. Architecture-level ways to cut attention compute or KV memory so long windows are feasible. See Chapter 14.
  • Retrieval / RAG, chunking, reranking. Selecting which of a corpus reaches the window: split into chunks, index, retrieve by similarity, rerank, fill a token budget. See Chapter 31.
  • Context rot / effective context length. The measured decline of a fact's usefulness with context length, middle position, and distractor similarity; the usable window is smaller than the advertised one. See Chapter 33.
  • Prompt injection (direct / indirect), exfiltration, context poisoning. Hostile instructions smuggled through the data channel; the window as an attack surface. See Chapter 32.
  • Image token estimate. (width * height) / 750 after downscaling to a per-model long-edge cap; pixels are billed, not bytes, and PDFs bill per page as image plus text. See Chapter 34.
  • Usage block. The per-response accounting object (input_tokens, cache_creation_input_tokens, cache_read_input_tokens, output_tokens); the full prompt is the sum of the three input-side fields. See Chapter 23.
  • Cache breakpoint / TTL / minimum. The cache_control marker (at most 4 per request), the 5-minute or 1-hour lifetime refreshed by use, and the per-model smallest cacheable prefix. See Chapter 24.
  • Transcript (JSONL). Claude Code's per-session event log under ~/.claude/projects/, carrying one usage block per assistant turn. See Chapter 25.
  • Injection channel. Claude Code's append-only path for dynamic state: harness content rendered as <system-reminder> blocks in user turns (recorded as typed attachment records), keeping the cached prefix byte-stable. See Chapter 28.

This book's code

Every demo is in the code/ folder of this book's repository, runnable with python3 and NumPy, no API key. By chapter:

  • context_assemble.py, token_economy.py: foundations.
  • compress.py, output_shape.py, code_context.py: compression.
  • kv_cache.py, semantic_cache.py, kv_serving.py: caching.
  • agent_memory.py, temporal_kg.py, compaction.py, procedural_learning.py: memory.
  • orchestrate.py, attention_efficiency.py: architecture.
  • usage_anatomy.py, cache_shape_lab.py, usage_ledger.py: the measurement lab (usage_ledger.py reads your own Claude Code transcripts, standard library only).
  • tool_bench.py: the tool benchmark (needs rtk on PATH and headroom-ai installed; measures them against your repository's real command traffic).
  • injection_census.py: the injection-channel census and prefix-continuity proof, run against your own transcripts (standard library only).
  • session_audit.py: the per-session, per-turn before/after instrument for the guided lab (point it at any session transcript; standard library only).
  • retrieval_lab.py: chunk-size and token-budget sweep of a TF-IDF retriever over this book's own chapters (standard library + src/).
  • injection_lab.py: the prompt-injection mechanics, a heuristic scanner with precision/recall, and provenance-framing overhead (standard library only).
  • multimodal_tokens.py: image and PDF token economics from dimensions (standard library only).
  • logits_control.py: the decoder's dials (softmax, temperature, top-k/p, min-p, logit bias) with a seeded 4000-draw diversity experiment (NumPy only).
  • symbol_server.py: a from-scratch symbol server (get_symbols_overview / find_symbol / find_referencing_symbols) showing the 85% token cut of a symbol read (standard library only).
  • mgd_lab.py: Monitor-Guided Decoding as a prefix automaton masking the logits, taking a toy model from 34% to 100% type-correct (NumPy only).
  • static_facts_cost.py: the reference-file cost model (four loading designs, usage and growth sweeps, break-even; standard library only).
  • memgpt_core.py: the MemGPT loop from scratch (self-editing core, archival, eviction under memory pressure, fresh-window quiz; standard library only).
  • memory_eval_harness.py: plant / disturb / probe / score over five memory conditions, with the compaction by-type fidelity table (standard library only).
  • radix_cache.py: a fleet-wide prefix cache over one simulated working day, four policies plus the radix-tree printout (standard library only).
  • cache_autopsy.py: the per-call cache lifecycle of one real session, from your own transcripts: extends, invalidations, TTL lapses, and the no-cache counterfactual (standard library only).
  • bpe_lab.py: BPE trained on this book's own chapters, then the seven-format cost bench, digits, languages, and base64 (standard library only).
  • prefill_decode.py: the prefill/decode roofline on one model and one GPU, the 77x machine-time gap, and a Monte Carlo of speculative decoding against the closed form (NumPy only).
  • rope_lab.py: RoPE built and verified, the partial-arc dials, and the three stretch schemes scored at 4x (NumPy only).
  • surfaces_audit.py: the live audit of this machine's skills, agents, hooks, MCP config, and CLAUDE.md layers, priced resident versus lazy (standard library only).
  • optimization_lab.py: the capstone cost model.

Read them, change a number, and watch the trade-off move. That is the fastest way to make the ideas your own.

One last thing

Context engineering is bookkeeping with stakes. The model is only ever as good as the tokens you put in front of it and only ever as cheap as the tokens you avoid re-paying for. You now have the levers and the map. When a system is slow, expensive, or forgetful, you can name which of the four pressures it is failing and reach for the matching tool. That is the whole job.