The language server as context engine: how Serena works

TL;DR. Chapter 5 selected code by walking a call graph you built from the AST; Chapter 37 ended by promising a constraint computed live from the codebase. Both need the same engine: a language server, the daemon your IDE already runs to power go-to-definition and find-references. Serena (MIT, from Oraios AI) is the open-source toolkit that hands that engine to a coding agent as a set of MCP tools, so the model can ask for one method's body instead of reading a 2000-line file, ask for a symbol's real callers instead of grepping, and replace a symbol's body instead of re-emitting the whole file. It is built on solidlsp, its synchronous fork of Microsoft's multilspy, which wraps a real language server per language (a Python server, typescript-language-server, gopls, rust-analyzer, and dozens more) behind one Python API. This chapter builds a symbol server from scratch on Python's ast (cutting one edit from 118 tokens to 18, an 85% reduction), then maps every piece onto Serena's real tools, its .serena/memories/ markdown store, its onboarding pass, and the claude-code context whose prompt literally forbids the agent from reading files to explore. The request_completions call this chapter introduces is the exact query the next chapter wires into the sampling loop.

Contents

From files to symbols
What a language server actually does
multilspy and solidlsp: the uniform API
Serena's tools, and what each one asks the server
The demo
The memory system
Contexts and modes
Using the real tool: commands and before/after proof
Further reading
Takeaways

Chapter 5 made the case that you compress a codebase by selecting whole units along its structure rather than trimming lines, and it built a call graph from Python's ast to do the selecting. That chapter computed the structure itself, by hand, for one language. This chapter is about the tool that already computes it, for every language, continuously, and far more accurately than an AST walk: the language server. A coding agent that can talk to a language server does not have to parse the repository to know what a symbol is or where it is used. It can ask. Serena is the toolkit that lets it ask, and its design is the cleanest available answer to the question this whole book keeps circling: how do you keep the window lean on a codebase far too large to send?

From files to symbols

The lazy unit of code context is the file. "Show me deploy.py" pulls in 600 lines to reason about one 30-line function, and every one of those lines is a token you pay for on this turn and, because it sits in the transcript, on every later turn of the session. Chapter 5 already named the fix in the abstract: select by structure, send only what the task touches. The symbol is that structure made concrete. A symbol is a named, bounded piece of a program: a function, a class, a method, a field. It has an exact location (a file, a start line and column, an end line and column) and it has relationships (what it calls, what calls it, what it inherits). If a tool can hand the model a symbol by name, return just its body, and list just its real callers, then the file stops being the unit and the token bill drops with it.

The reason this is not simply "grep for the function name and read those lines" is the same reason Chapter 5 parsed a tree instead of matching text. A regex cannot tell a definition from a call, a real reference from the same word in a comment or a string, or one withdraw from a different class's withdraw. To get symbols right, you need something that understands the language's grammar and has resolved its types. That something exists, it runs on your machine right now if you use an IDE, and it speaks a documented protocol.

What a language server actually does

When you use go-to-definition in VS Code, the editor is not doing the analysis. A separate long-running process, the language server, is. The Language Server Protocol (LSP), introduced by Microsoft, standardized the conversation between an editor and that process so any editor can talk to any language's server over the same JSON-RPC messages. The server starts up, you hand it the project root, it indexes the whole project (parses every file, resolves imports, builds the type and reference graphs), and from then on it answers questions about the code as facts, not guesses.

A handful of LSP requests carry almost everything a coding agent needs, and it is worth knowing them by name because Serena's tools are thin wrappers over exactly these:

textDocument/documentSymbol returns the symbol tree of one file: every class, method, and function, each with its kind and, crucially, two ranges. The range is the symbol's full extent (the whole method body); the selectionRange is just the name. This is the map of a file.
workspace/symbol searches that index across the whole project by name, so you can find a symbol without knowing which file it lives in.
textDocument/references returns every real use site of a symbol: the true call sites from the server's cross-reference index, with comments and string look-alikes excluded. This is the grep that is actually correct.
textDocument/definition jumps from a use site to where the symbol is defined; textDocument/hover returns its type signature and doc comment.
textDocument/completion returns the identifiers legal at a given cursor position: given account., the members that the type of account actually has. Hold onto this one. It is the query the next chapter masks the logits with.

The key property is that these answers are repository-aware and type-resolved. The server already knows that account is an Account because it followed the imports and the assignments, so textDocument/references on Account.withdraw finds the real callers across every file and ignores an unrelated withdraw on some other class. An AST walk in one file cannot do that; the language server does it for the whole project, and keeps the index warm.

multilspy and solidlsp: the uniform API

There is a catch that has kept language servers out of most tools: every language has a different server (a Python server, typescript-language-server, gopls, rust-analyzer, Eclipse JDT for Java, clangd for C++), each is a separate binary with its own launch quirks, and each speaks LSP with its own dialect of initialization parameters. Wiring one up by hand is a project; wiring up twelve is a career.

multilspy, a Python library from Microsoft Research, exists to erase that. It launches the right language server as a subprocess, manages the JSON-RPC over stdio, carries hand-tuned initialization parameters for each server, and exposes one uniform Python API so the same calls work across languages. Its request methods map one-to-one onto the LSP requests above, with zero-indexed (line, column) positions:

request_document_symbols(file)         -> textDocument/documentSymbol
request_workspace_symbol(query)        -> workspace/symbol
request_references(file, line, col)    -> textDocument/references
request_definition(file, line, col)    -> textDocument/definition
request_hover(file, line, col)         -> textDocument/hover
request_completions(file, line, col)   -> textDocument/completion

You create a server, enter start_server() (which spawns the subprocess and runs the initialize handshake), and call the request methods. multilspy was built as the static-analysis layer for the Monitor-Guided Decoding research of the next chapter, which is why request_completions is a first-class citizen: that project needed the set of valid members at a cursor, computed live.

Serena is built on a fork of multilspy called solidlsp (it lives in Serena's own repository under src/solidlsp/). The fork does two things: it makes the LSP calls synchronous (simpler to drive from an agent's tool loop), and it adds the symbolic logic Serena needs on top, chiefly the part that takes a symbol's LSP range and slices exactly those lines out of the file so a tool can return one method's source and nothing else. That slice is the whole token argument, and the demo below builds it.

Serena's tools, and what each one asks the server

Serena exposes the language server to the model as a set of MCP tools (Chapter 28 covered the tool channel; Chapter 26 toured MCP servers). The names and exact arguments have evolved across releases, but the conceptual core is stable, and it is small:

get_symbols_overview(file) returns a file's top-level symbols with their kinds, no bodies. Serena's own instruction to the model is that this "should be the first tool to call when you want to understand a new file". It is one textDocument/documentSymbol call, rendered as a table of contents. You read structure, not text.
find_symbol(name_path, include_body=False) is the workhorse. It takes a name path into the symbol tree: "withdraw" matches any symbol with that name, "Account/withdraw" matches that method inside that class, a leading slash makes it absolute from the file root, and a [1] suffix disambiguates overloads. With include_body=False you get just the location and signature; with include_body=True Serena slices the symbol's LSP range and returns only that symbol's source, never the surrounding file. Scoped lookups use documentSymbol; global ones use workspace/symbol.
find_referencing_symbols(name_path, file) returns every symbol that references the target, with the referencing snippets, grouped by file. It is textDocument/references, so it is the true call sites, the change's blast radius, not a name-grep.
replace_symbol_body, insert_after_symbol, insert_before_symbol are the edit side. replace_symbol_body writes a new body into the symbol's exact range, so the model emits only the new body, not the whole rewritten file. This is the output-token half of the savings from Chapter 4: re-emitting a 2000-line file to change one method is 2000 lines of expensive output, while replace_symbol_body is a few dozen. insert_before_symbol on the first symbol is how you add imports; insert_after_symbol on the last is how you append a new definition.

There are also file and search tools (read_file, list_dir, search_for_pattern for regex, a text/regex replace) for the cases where symbol granularity does not apply, shell execution, and the memory tools we come to next. But the symbolic tools are the point: they are what a language server buys you that cat and grep cannot.

The demo

The script below builds a symbol server from scratch, standing in Python's ast for the language server exactly the way Chapter 5 did (real Serena uses solidlsp so it works across languages; the operations are identical). It implements the three core reads, get_symbols_overview, find_symbol by name path, and find_referencing_symbols, on a small module whose task is "fix the overdraft check in Account.withdraw". Then it measures the payoff.

"""A symbol server from scratch: what Serena does to save context tokens.

Serena gives a coding agent symbol-level access to a codebase through a
language server: instead of reading a whole file to edit one method, the agent
asks for exactly that symbol's body. This lab builds the same three operations
Serena's core exposes, using Python's standard-library `ast` as a stand-in for
the Language Server Protocol (real Serena uses solidlsp/multilspy so it works
across languages; the operations are identical):

  get_symbols_overview   -> a file's top-level symbols, bodies omitted
  find_symbol(name_path) -> ONE symbol's exact source, e.g. "Account/withdraw"
  find_referencing       -> which symbols call a given symbol

Then it measures the payoff: editing one method by symbol costs a fraction of
the tokens that reading the whole file would, which is the entire reason a
language-server tool beats "cat the file into the prompt".

Standard library only (ast, the same word*1.3 token estimate as Chapter 2).
"""

import ast

# A small module, held as a string so the lab is self-contained. The target
# task: "fix the overdraft check in Account.withdraw". A whole-file read pays
# for Ledger, Report, and the helpers too; a symbol read pays for one method.
MODULE = '''\
import math


def audit(entries):
    total = sum(e["amount"] for e in entries)
    return total


class Account:
    def __init__(self, owner, balance=0):
        self.owner = owner
        self.balance = balance

    def deposit(self, amount):
        self.balance += amount
        return self.balance

    def withdraw(self, amount):
        if amount > self.balance:
            raise ValueError("overdraft")
        self.balance -= amount
        return self.balance


class Ledger:
    def __init__(self):
        self.accounts = {}

    def open(self, owner):
        acct = Account(owner)
        self.accounts[owner] = acct
        return acct

    def transfer(self, src, dst, amount):
        self.accounts[src].withdraw(amount)
        self.accounts[dst].deposit(amount)


def monthly_report(ledger):
    rows = [(o, a.balance) for o, a in ledger.accounts.items()]
    return audit([{"amount": b} for _, b in rows])
'''

LINES = MODULE.splitlines()


def est_tokens(text):
    """Same rough estimate as Chapter 2: words * 1.3."""
    return round(len(text.split()) * 1.3)


def source_of(node):
    """Exact source text of an AST node, via its line span. This is the
    language server's job: map a symbol to the byte/line range that defines it
    so a tool can return just that slice."""
    return "\n".join(LINES[node.lineno - 1:node.end_lineno])


def build_symbols(tree):
    """Walk the module once and record every symbol with a Serena-style name
    path: top-level 'audit', 'Account', and nested 'Account/withdraw'. Returns
    an ordered dict of name_path -> node."""
    symbols = {}
    for node in tree.body:
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
            symbols[node.name] = node
        elif isinstance(node, ast.ClassDef):
            symbols[node.name] = node
            for child in node.body:
                if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)):
                    symbols[f"{node.name}/{child.name}"] = child
    return symbols


def kind(node):
    if isinstance(node, ast.ClassDef):
        return "class"
    return "function"


def get_symbols_overview(symbols):
    """Serena's get_symbols_overview: the file's map, bodies omitted. This is
    what the agent reads FIRST to decide which symbol it actually needs."""
    print("=== get_symbols_overview  (the map, no bodies) ===")
    for name, node in symbols.items():
        sig = f"{kind(node):8} {name}"
        span = f"L{node.lineno}-{node.end_lineno}"
        print(f"  {sig:34} {span}")
    overview_text = "\n".join(symbols)
    print(f"  overview cost: ~{est_tokens(overview_text)} tokens "
          f"(vs ~{est_tokens(MODULE)} to read the whole file)\n")


def find_symbol(symbols, name_path):
    """Serena's find_symbol: return ONE symbol's exact source by name path.
    The token win lives here, this is the slice the agent edits."""
    node = symbols[name_path]
    body = source_of(node)
    print(f"=== find_symbol('{name_path}')  (just this symbol) ===")
    print("\n".join("  " + ln for ln in body.splitlines()))
    print(f"  symbol cost: ~{est_tokens(body)} tokens\n")
    return body


def find_referencing_symbols(symbols, target_leaf):
    """Serena's find_referencing_symbols: which symbols call target_leaf.
    Built here by walking each symbol's Call nodes for an attribute access of
    the given name (real Serena asks the language server for exact references,
    which also resolves types and ignores same-named unrelated methods)."""
    print(f"=== find_referencing_symbols('{target_leaf}') ===")
    hits = []
    for name, node in symbols.items():
        if "/" not in name and isinstance(node, ast.ClassDef):
            continue  # class shells are covered by their methods
        for call in ast.walk(node):
            if isinstance(call, ast.Call):
                fn = call.func
                if isinstance(fn, ast.Attribute) and fn.attr == target_leaf:
                    hits.append((name, call.lineno))
    for name, line in hits:
        print(f"  {name:22} references .{target_leaf}()  at L{line}")
    print(f"  ({len(hits)} referencing site(s); the change's blast radius)\n")
    return hits


def measure(symbols):
    print("=== The payoff: symbol read vs whole-file read ===")
    whole = est_tokens(MODULE)
    target = source_of(symbols["Account/withdraw"])
    just = est_tokens(target)
    print(f"  whole file            : ~{whole} tokens")
    print(f"  Account/withdraw only : ~{just} tokens")
    print(f"  reduction             : {1 - just / whole:.0%} fewer tokens to "
          f"edit one method")
    print("""
  The agent read a ~10-line overview to locate the symbol, pulled the one
  method it needed, and checked the three-line blast radius before editing.
  It never paid for Ledger, monthly_report, or the imports. On a 2000-line
  file the ratio is far steeper: this is why a language-server tool beats
  pasting the file, and why Serena stores what it learns in .serena/memories
  so the next session skips the exploration entirely.""")


if __name__ == "__main__":
    tree = ast.parse(MODULE)
    symbols = build_symbols(tree)
    get_symbols_overview(symbols)
    find_symbol(symbols, "Account/withdraw")
    find_referencing_symbols(symbols, "withdraw")
    measure(symbols)

Running it:

=== get_symbols_overview  (the map, no bodies) ===
  function audit                     L4-6
  class    Account                   L9-22
  function Account/__init__          L10-12
  function Account/deposit           L14-16
  function Account/withdraw          L18-22
  class    Ledger                    L25-36
  function Ledger/__init__           L26-27
  function Ledger/open               L29-32
  function Ledger/transfer           L34-36
  function monthly_report            L39-41
  overview cost: ~13 tokens (vs ~118 to read the whole file)

=== find_symbol('Account/withdraw')  (just this symbol) ===
      def withdraw(self, amount):
          if amount > self.balance:
              raise ValueError("overdraft")
          self.balance -= amount
          return self.balance
  symbol cost: ~18 tokens

=== find_referencing_symbols('withdraw') ===
  Ledger/transfer        references .withdraw()  at L35
  (1 referencing site(s); the change's blast radius)

=== The payoff: symbol read vs whole-file read ===
  whole file            : ~118 tokens
  Account/withdraw only : ~18 tokens
  reduction             : 85% fewer tokens to edit one method

Follow the agent's path, because it is the whole method. It reads the overview first, a 13-token table of contents, and locates Account/withdraw without loading a single body. It pulls that one symbol, 18 tokens, the exact slice a textDocument/documentSymbol range would give it. It checks the blast radius, one caller, Ledger/transfer, so it knows what a signature change would break before it makes one. And it never pays for Ledger, monthly_report, audit, or the imports. Editing one method cost 18 tokens where reading the file cost 118, an 85% cut, and that ratio is the floor: on a real 2000-line, 30,000-token module the same one-method edit is still a few dozen tokens, so the reduction climbs past 99%. That is the number that makes a language-server tool worth the setup, and it is why Serena's whole reason to exist is to keep the file out of the window.

Remember. The three reads are a discipline, not just three tools: get_symbols_overview to see the map, find_symbol to pull the one body you need, find_referencing_symbols to see what a change touches. An agent that follows that discipline pays for symbols; an agent that reaches for read_file pays for files. The gap is small on a toy and enormous on a repository.

The memory system

Symbol-level retrieval keeps a single task lean. Serena's second idea keeps successive tasks lean, and it is a direct instance of the agent-memory lever from Chapter 9 and the Claude Code memory layers from Chapter 18. When Serena learns something durable about a project (how to run the tests, what the module layout is, a convention), it writes it to a plain markdown file under .serena/memories/ in the project. These files are meant to be committed with the code and read by a human, not just the model, and they persist across every session.

The mechanics are deliberately simple. write_memory(name, content) creates .serena/memories/<name>.md; read_memory(name) reads one back; list_memories() lists the names; delete_memory(name) removes one. Names can use slashes to organize into subfolders, and memories cross-reference each other with a mem:NAME convention. The cost argument is in when the model reads them: it is shown the list of memory names on startup, which is cheap, and it pulls the body of only the one relevant to the task, inferring relevance from the name. It does not load the whole store every session; it loads the index and fetches on demand, which is exactly the retrieval discipline of Chapter 31 applied to the agent's own notes.

The store is seeded by onboarding. The first time Serena activates a project it checks whether any memories exist, and if none do, it runs an onboarding pass: it explores the project and writes an initial set of memories, typically a core overview, a tech_stack, a suggested_commands (how to build, test, and lint, which the shell tool is told to consult before running anything), a conventions, and a task_completion (what "done" means). After that first pass the exploration is amortized: a new session reads the memories instead of re-crawling the codebase to rediscover the same facts. That is the persistence half of the token economy. The first conversation pays to learn the project once; every later conversation reads the summary for a few hundred tokens instead of re-deriving it for thousands.

Don't be confused. Serena's .serena/memories/ and Claude Code's own memory (Chapter 18) are the same idea (durable facts on disk, re-loaded next session) at two layers. Claude Code's CLAUDE.md and auto-memory are the harness's memory; Serena's memories are the tool's memory, scoped to the code it navigates and written in the vocabulary of symbols and commands. When both are present they stack: the harness remembers how you like to work, the tool remembers how the codebase is shaped.

Contexts and modes

Two small configuration layers decide which of Serena's tools are live, and they matter because they are how Serena avoids stepping on its host. A context says who the client is and is fixed for a session: desktop-app, agent, ide, and claude-code are the main ones. A mode says how to operate and can stack and change mid-session: interactive (ask for clarification), editing (all tools, tuned for edits), planning (read-only, all mutating tools removed), one-shot (run autonomously). Running Serena inside Claude Code in planning mode, for instance, gives the agent every navigation tool and no way to write, which is a precise way to get an analysis without risking an edit.

The claude-code context is the one worth dwelling on, because it encodes the entire thesis of this chapter. Claude Code already has its own Read, Edit, Bash, and Grep, so Serena's claude-code context excludes its own copies of those and contributes only the symbolic tools and memory. And its system prompt does something blunt: it tells the model that using the plain file Read for code discovery is forbidden, and that it must use get_symbols_overview and find_symbol instead, on the stated grounds that the symbolic tools are far more token-efficient than reading files. Serena is not asking the model to prefer symbols; it is removing the file-reading habit and replacing it with the language-server one.

Using the real tool: commands and before/after proof

You install Serena and register it with Claude Code as an MCP server. The commands below are the current shape (this is follow-along, since Serena is not installed on this box; the from-scratch demo above is the measured part):

# install
uv tool install -p 3.13 serena-agent

# register with Claude Code, using the claude-code context, project = cwd
claude mcp add --scope user serena -- \
  serena start-mcp-server --context claude-code --project-from-cwd

Once registered, the agent gains the symbolic tools. The before/after is the same shape the demo measured, now live. Before (no Serena), a request to "fix the overdraft check in Account.withdraw" tends to Read the whole file:

> Read account.py                      # 600 lines, ~4,000 tokens into the window
> Edit account.py  (whole-file diff)   # re-emits the surrounding lines

After (Serena present, claude-code context), the same request navigates by symbol:

> get_symbols_overview account.py      # the map, ~a few hundred tokens
> find_symbol "Account/withdraw" include_body=true    # one method, ~40 lines
> find_referencing_symbols "withdraw"  # one caller, the blast radius
> replace_symbol_body "Account/withdraw" <new body>   # emits only the new body

You can watch the difference in the same gauges Chapter 21 read: /context shows a few small symbols loaded instead of a whole file, and the input token count on each turn stays flat instead of climbing with file size. Serena also ships a web dashboard (by default at http://localhost:24282/dashboard/index.html) that logs every tool call and every underlying LSP round-trip, so you can see the documentSymbol and references requests fire and confirm the model is navigating rather than dumping. The number to trust is still the on-box one: the demo cut a one-method edit from 118 tokens to 18. Serena applies that same slice to every read a live agent makes, on files where the file-versus-symbol ratio is not 6x but hundreds.

Takeaways

The unit of code context should be the symbol, not the file. A language server already computes symbols (definitions, references, types) for the whole project and answers over LSP, far more accurately than a regex or a single-file AST walk.
Serena hands that server to a coding agent as MCP tools, built on solidlsp, its synchronous fork of Microsoft's multilspy, which wraps one real language server per language behind a uniform Python API (request_document_symbols, request_references, request_completions, and the rest).
The three core reads are a discipline: get_symbols_overview (the map), find_symbol with a name path (one body), find_referencing_symbols (the blast radius). The demo cut a one-method edit from 118 tokens to 18, an 85% reduction that grows past 99% on a real file. replace_symbol_body extends the win to the output side by emitting only the changed body.
Serena's .serena/memories/ markdown store, seeded by an onboarding pass, is the agent-memory lever: learn the project once, read the summary cheaply forever after. The model sees the memory list and fetches only the relevant body on demand.
The claude-code context proves the thesis: it removes Serena's own file-reading tools and instructs the agent that reading files to explore is forbidden, forcing the language-server tools because they are far more token-efficient.

👉 Serena uses the language server to retrieve the right symbols into the context. The final chapter takes the same request_completions query and pushes it one level deeper, into the sampling loop, so the model cannot even emit a method that does not exist. Continue to Monitor-Guided Decoding.

Context Engineering in Depth