The memory eval harness: before, after, and across

TL;DR. Memory claims are cheap and memory evals are cheaper, so build the eval. Every memory benchmark reduces to one loop: plant known facts, disturb them (a session boundary, a compaction, a model swap, a new tool), probe with questions whose answers you know, and score recall, staleness, and abstention. A from-scratch lab runs that loop over five conditions and catches, with numbers, what each disturbance does: a window cap erases session 1 (recall 1/5), compaction silently drops the detail and procedure facts (3/5), full replay recalls well but serves a superseded fact (4/5 with 1 stale answer), a store with update-in-place scores clean (5/5). The rest of the chapter is the same loop as five field recipes on claude -p: before/after one change, cross-session recall, compaction fidelity, model swaps, and grading, so every claim from Chapter 40 and Chapter 41 becomes checkable on your own sessions.

Contents

• Why your own eval
• The anatomy of a memory eval
• The lab: the harness from scratch
• Recipe 1: before and after one change
• Recipe 2: cross-session recall
• Recipe 3: compaction fidelity
• Recipe 4: the model swap
• Recipe 5: grading beyond grep
• Further reading
• Takeaways

Chapter 27 built the A/B protocol for token tools: same task, tool off then on, judged on the usage fields. Memory needs a different judge, because the failure is not cost but knowledge: did the fact come back, was it the current version, and did the system know when it had nothing? This chapter builds that judge.

Why your own eval

Three facts from the published record make the case better than any argument:

• LongMemEval (Wu et al., ICLR 2025), the closest thing to a standard, reports that commercial chat assistants show around a 30% accuracy drop on remembering information across sustained interactions. Memory features ship broken more often than their marketing suggests.
• The vendors cannot agree on each other's scores. Chapter 41 told the story: mem0's paper scored Zep at 66% on LoCoMo; Zep re-ran it and published 75% with a misconfiguration analysis; mem0 published counter-corrections. Both are competent teams measuring the same benchmark. The gap is configuration, and your configuration is the one neither of them tested.
• The benchmark itself may not test memory. Zep's sharpest observation was that LoCoMo conversations average roughly 16k to 26k tokens, small enough to fit whole in a modern window, so a full-context baseline (no memory system at all) beat mem0's own pipeline in mem0's own paper. A benchmark that fits in context measures reading, not remembering.

So published numbers rank papers, not your setup. The harness below costs an afternoon, runs on your own facts and sessions, and answers the only question that matters: does the thing you configured last week actually remember, still, today?

The anatomy of a memory eval

Every published benchmark is the same four-step loop wearing different data:

1. Plant. Put known facts into sessions: typed (identity, preference, decision, detail, procedure), timestamped, some deliberately superseded later (the fact changes), some deliberately absent (nothing planted, to test abstention).
2. Disturb. Apply exactly one thing you want to measure: a session boundary, a window cap, a compaction, a memory system toggled on, a different model.
3. Probe. Ask questions whose ground truth you wrote down in step 1. Include probes for the superseded facts (the correct answer is the new version) and the absent ones (the correct answer is "I don't know").
4. Score. Recall (current fact returned), staleness (superseded version returned; the most damaging failure, because it looks like success), abstention (declined when nothing was planted), plus tokens carried to get the answer.

The published suites are this loop at scale, and their categories are a ready-made checklist for your probe set:

Note what recurs: knowledge updates and abstention appear in every modern suite, because they are where systems actually fail. A probe set of only "what did I say my name was?" questions will pass systems that are badly broken on "what do I prefer now?"

Don't be confused. Resuming a session is not memory. claude --resume replays the transcript, so facts "survive" trivially, until the transcript is compacted or the window caps out; that tests replay. A memory eval proper probes a fresh session, where the only carriers are the things you are evaluating: auto memory, CLAUDE.md, an MCP store. Recipe 2 runs both on purpose, because the pair tells you which layer is doing the work.

The lab: the harness from scratch

The lab implements the loop end to end, small enough to read whole. The reader is a deterministic word-overlap matcher (ties to the earliest candidate) so the numbers are exactly reproducible; the recipes swap in a real model but keep the probes and scoring unchanged.

"""A memory eval harness from scratch: plant, disturb, probe, score.

Every memory benchmark (LongMemEval, LoCoMo, MemoryAgentBench) is the same
four-step loop, and this lab builds it small enough to read:

  PLANT    put known facts into sessions, with types and timestamps
  DISTURB  apply the thing you are testing: a session boundary, a window
           cap, a compaction, a memory store
  PROBE    ask questions whose answers you know, including questions with
           NO planted answer (abstention probes) and questions whose answer
           CHANGED (staleness probes)
  SCORE    recall, staleness, abstention, and tokens carried, per condition

The reader is deliberately mechanical (best word-overlap match, ties go to
the earliest candidate) so every number is reproducible; the point is the
harness, not the reader. Chapter 42 swaps the reader for a real model via
`claude -p` and keeps everything else. Deterministic, standard library only.

Run:  python3 memory_eval_harness.py
"""

def toks(s):
    return len(s) // 4

def words(s):
    return set("".join(c if c.isalnum() else " " for c in s.lower()).split())

# ---------------------------------------------------------------------------
# PLANT: facts arrive across two sessions; session 3 is the probe session.
# salience 2 = the kind of thing a summarizer keeps; 1 = supporting detail.
# ---------------------------------------------------------------------------
FACTS = [
    dict(s=1, typ="identity",   sal=2, text="the user's handle is s0x and the timezone is America/Toronto"),
    dict(s=1, typ="decision",   sal=2, text="the deploy target we chose is Cloudflare Pages"),
    dict(s=1, typ="preference", sal=2, text="the user prefers pip for Python installs"),
    dict(s=1, typ="detail",     sal=1, text="the CI budget is 15 minutes per run"),
    dict(s=1, typ="procedure",  sal=1, text="release procedure: build all books, check links, then push to main"),
    dict(s=2, typ="detail",     sal=1, text="the 2026-03 incident was caused by a bad symlink in build.sh"),
    dict(s=2, typ="preference", sal=2, text="the user prefers uv for Python installs",
         supersedes="the user prefers pip for Python installs"),
    dict(s=2, typ="decision",   sal=2, text="we chose mdBook over Sphinx for the books"),
]

PROBES = [
    ("what is the user's handle?",                        "s0x",        None),
    ("what is our deploy target?",                        "cloudflare", None),
    ("which installer does the user prefer for Python?",  "uv",         "pip"),
    ("what is the CI budget per run?",                    "15",         None),
    ("what is the release procedure for the books?",      "build all",  None),
    ("which database did we choose?",                     None,         None),  # never planted
]

# ---------------------------------------------------------------------------
# DISTURB: each condition builds the fact pool session 3 actually sees.
# ---------------------------------------------------------------------------
def cond_fresh():
    """A fresh window, nothing carried over. The floor."""
    return []

def cond_replay_full():
    """Replay the whole transcript: every planted fact, old versions included."""
    return [f["text"] for f in FACTS]

def cond_replay_capped():
    """Replay under a window cap that evicted session 1 (oldest-first)."""
    return [f["text"] for f in FACTS if f["s"] >= 2]

def cond_compacted():
    """A summarizer kept what it judged salient, latest version only."""
    superseded = {f["supersedes"] for f in FACTS if f.get("supersedes")}
    return [f["text"] for f in FACTS if f["sal"] >= 2 and f["text"] not in superseded]

def cond_store():
    """A chapter-9 store: everything kept, contradictions updated in place."""
    superseded = {f["supersedes"] for f in FACTS if f.get("supersedes")}
    return [f["text"] for f in FACTS if f["text"] not in superseded]

CONDITIONS = [
    ("fresh window",    cond_fresh),
    ("replay, full",    cond_replay_full),
    ("replay, capped",  cond_replay_capped),
    ("compacted",       cond_compacted),
    ("memory store",    cond_store),
]

# ---------------------------------------------------------------------------
# PROBE + SCORE.
# ---------------------------------------------------------------------------
def read(pool, question, min_overlap=2):
    """The mechanical reader: best word-overlap candidate, ties to earliest,
    abstains below the overlap threshold."""
    q = words(question)
    best, best_score = None, 0
    for fact in pool:
        score = len(words(fact) & q)
        if score > best_score:
            best, best_score = fact, score
    return best if best_score >= min_overlap else None

def evaluate(pool):
    recall = stale = abstain_ok = 0
    answerable = sum(1 for _, needle, _ in PROBES if needle)
    for question, needle, stale_needle in PROBES:
        answer = read(pool, question)
        low = answer.lower() if answer else ""
        if needle is None:
            abstain_ok += answer is None
        elif answer is None:
            pass
        elif stale_needle and stale_needle in low and needle not in low:
            stale += 1
        elif needle in low:
            recall += 1
    return recall, answerable, stale, abstain_ok, toks("\n".join(pool))

print("=== The scorecard: five conditions, one probe set ===")
print(f"{'condition':<16}{'recall':>8}{'stale':>7}{'abstain':>9}{'tokens':>8}")
print("-" * 48)
for name, build in CONDITIONS:
    r, n, st, ab, tk = evaluate(build())
    print(f"{name:<16}{f'{r}/{n}':>8}{st:>7}{f'{ab}/1':>9}{tk:>8}")
print("-" * 48)

# ---------------------------------------------------------------------------
# The compaction fidelity table: WHICH facts die is more useful than HOW MANY.
# ---------------------------------------------------------------------------
print("\n=== What the compaction kept, by fact type ===")
kept = set(cond_compacted())
by_type = {}
superseded = {f["supersedes"] for f in FACTS if f.get("supersedes")}
for f in FACTS:
    if f["text"] in superseded:
        continue                     # replaced facts are not 'lost', they are history
    alive, total = by_type.get(f["typ"], (0, 0))
    by_type[f["typ"]] = (alive + (f["text"] in kept), total + 1)
for typ, (alive, total) in by_type.items():
    marker = "kept" if alive == total else "LOST"
    print(f"  {typ:<11} {alive}/{total}  {marker}")

print("""
Reading the tables:
- 'replay, full' recalls everything but answers the preference probe with
  the SUPERSEDED fact: both versions sit in the window and the reader has
  no reason to prefer the newer one. High recall can hide staleness.
- 'replay, capped' is distance decay: everything planted in session 1 is
  gone, and the eval catches it as recall, not as a vague feeling.
- 'compacted' keeps every salient fact and silently drops the detail and
  procedure types; the by-type table is the compaction contract made
  visible, and it is what a /compact focus instruction exists to change.
- 'memory store' scores clean on recall AND staleness because update-in-
  place removed the old version before the probe ever ran.
Swap the mechanical reader for a real model (claude -p) and these same
probes, scores, and tables become your production memory eval.""")

The verified output:

=== The scorecard: five conditions, one probe set ===
condition         recall  stale  abstain  tokens
------------------------------------------------
fresh window         0/5      0      1/1       0
replay, full         4/5      1      1/1      98
replay, capped       1/5      0      1/1      35
compacted            3/5      0      1/1      47
memory store         5/5      0      1/1      88
------------------------------------------------

=== What the compaction kept, by fact type ===
  identity    1/1  kept
  decision    2/2  kept
  detail      0/2  LOST
  procedure   0/1  LOST
  preference  1/1  kept

Reading the tables:
- 'replay, full' recalls everything but answers the preference probe with
  the SUPERSEDED fact: both versions sit in the window and the reader has
  no reason to prefer the newer one. High recall can hide staleness.
- 'replay, capped' is distance decay: everything planted in session 1 is
  gone, and the eval catches it as recall, not as a vague feeling.
- 'compacted' keeps every salient fact and silently drops the detail and
  procedure types; the by-type table is the compaction contract made
  visible, and it is what a /compact focus instruction exists to change.
- 'memory store' scores clean on recall AND staleness because update-in-
  place removed the old version before the probe ever ran.
Swap the mechanical reader for a real model (claude -p) and these same
probes, scores, and tables become your production memory eval.

Four lessons, each one a real system's failure mode in miniature. Full replay's stale answer is Chapter 41's add-only debate made concrete: when both versions of a fact are present, something has to prefer the newer one, and "hope the model picks right" is not a mechanism. The capped condition is every long project's week two. The compaction row previews Recipe 3. And the store's clean sweep is why Chapter 9 insisted on invalidate-in-place.

Recipe 1: before and after one change

The base recipe, for any single change: a rewritten CLAUDE.md, a new memory MCP server, a moved reference file (Chapter 40's migration), a compaction setting. It is Chapter 27's protocol with knowledge scoring bolted on.

Write the probe set once, as data. A tab-separated file is enough:

what is our deploy target?	Cloudflare
which installer do I prefer for Python?	uv
what is the CI budget per run?	15 min
which database did we choose?	ABSTAIN

Then a loop over claude -p, headless print mode. Follow-along (output shape illustrative; the flags are current as of July 2026):

# eval.sh: run the probe file against the current setup, score by needle.
while IFS=$'\t' read -r q needle; do
  a=$(claude -p "$q" --output-format json | jq -r .result)
  if [ "$needle" = "ABSTAIN" ]; then
    case "$a" in *"don't know"*|*"no record"*) echo "PASS abstain: $q";;
                 *) echo "FAIL abstain: $q -> $a";; esac
  else
    case "$a" in *"$needle"*) echo "PASS recall:  $q";;
                 *) echo "FAIL recall:  $q -> $a";; esac
  fi
done < probes.tsv

Protocol: run it before the change and after, in fresh sessions each time, changing one variable, and keep the probe file under version control so next month's numbers are comparable. --output-format json also returns total_cost_usd and the session id, so the same loop doubles as the cost meter; and note --bare exists for the opposite experiment, running probes without CLAUDE.md, hooks, and MCP, which is your floor condition (the lab's "fresh window" row) made real.

Recipe 2: cross-session recall

The disturbance is the session boundary itself. Two runs, distinguished by what carries:

# Session A: plant, in a session with a known id.
sid=$(uuidgen)
claude -p --session-id "$sid" \
  "For the record: we deploy on Cloudflare Pages, the CI budget is 15 minutes, \
   and I now prefer uv over pip (I used to prefer pip)."

# Probe 1, replay: the transcript itself carries the facts.
claude -p --resume "$sid" "Quiz, one line each: deploy target? CI budget? installer?"

# Probe 2, memory: a FRESH session. Only durable layers can answer now.
claude -p "Quiz, one line each: deploy target? CI budget? installer?"

Score both with Recipe 1's loop. The pair separates the layers: probe 1 passing is expected (replay); probe 2 passing means a durable layer (auto memory, CLAUDE.md, an MCP store) actually captured the facts, and probe 2's staleness answer on the installer question tells you whether that layer reconciles updates or accumulates them. You can also inspect the middle directly: after session A, look at what auto memory wrote (~/.claude/projects/<project>/memory/, per Chapter 18) and whether your MCP store's list_memories shows one installer fact or two. Repeat probe 2 a week and a month later; distance decay is a curve, not a bit, and the lab's capped-replay row is what it looks like when it arrives.

Recipe 3: compaction fidelity

The disturbance is summarization, and it deserves its own recipe because the loss is silent and typed. The published data is blunt: a 2026 study (ConstraintRot) planted policy constraints in long agent sessions and measured violations at 0% while the policy sat in full context, an average of 30% after compaction, and up to 59% for some model families; when the constraint survived the summary the violation rate stayed at 0%, and when it was dropped it hit 38%. What the summarizer keeps is not a detail of UX; it is the behavior contract.

The interactive recipe, in a working Claude Code session:

1. Work normally until real context has accumulated, then plant your typed probe facts in conversation (not in CLAUDE.md: the point is to test the summary, and the project-root CLAUDE.md, unscoped rules, and auto memory are re-read from disk after compaction, so they are not at risk; nested CLAUDE.md files and path-scoped rules, note, are lost until a matching file is read again).
2. /compact, first with no instruction.
3. Probe all facts, tally by type, exactly the lab's by-type table.
4. Repeat with a focus instruction: /compact keep the deploy decisions, the CI budget, and the release procedure. The by-type table before and after the focus instruction is the measured value of that instruction, and the lab predicts the shape: salient types survive either way; detail and procedure types are what the focus clause rescues.

The same experiment runs at API level for your own agents: Anthropic's server-side compaction (beta header compact-2026-01-12) summarizes past a trigger (default 150k input tokens) and accepts an instructions field that replaces the default summarizer prompt, which is the focus instruction as a first-class parameter, plus pause_after_compaction so a harness can probe right at the boundary. Context editing's clear_tool_uses strategy (Chapter 41) is the blunter cousin; probe it the same way, with facts planted inside tool results versus conversation text, and watch the difference.

Recipe 4: the model swap

The disturbance is the model. Anthropic's own migration guidance is exactly this chapter's discipline: hold your eval set constant, re-run it on the new model, and re-baseline cost and latency on your own workloads rather than trusting release notes. The harness makes that a one-line change:

for m in claude-opus-4-8 claude-sonnet-5; do
  echo "== $m =="
  claude -p --model "$m" "$q" --output-format json | jq -r '.result, .total_cost_usd'
done

Two memory-specific effects to watch that generic migration checklists miss. First, retrieval behavior: models differ in how eagerly they call memory tools (search_memory before answering versus answering from priors), so a swap can silently change your memory system's hit rate even though the store is identical; probe 2 of Recipe 2, run per model, is the detector. Second, window behavior: a model with different effective-context characteristics (Chapter 33) changes where the capped-replay cliff sits, so re-run the distance-decay curve as well as the single-session probes.

Recipe 5: grading beyond grep

Needle-matching scores facts; it cannot score "did the summary preserve the intent of the release procedure." The upgrade path, in order of machinery:

1. Normalized exact match for facts with canonical forms (the lab's needles, lowercased).
2. Model-graded rubric for everything else. Anthropic's eval guidance is direct about this: structure questions for automated grading, prefer volume of questions over hand-graded perfection, and use an ordinal scale graded by a model, with one caution repeated across their docs: grade with a different model than the one that generated the answer. The canonical citation for why is Zheng et al. (NeurIPS 2023), which validated LLM judges at over 80% agreement with humans and named the standing biases: position bias, verbosity bias, and self-enhancement bias. For memory probes the rubric is mercifully simple: "Does the answer state ? Yes or no. If it states an older, superseded value, answer STALE."
3. A harness product when the probe set outgrows shell loops. promptfoo (Chapter 26) speaks Anthropic natively (anthropic:messages:<model> providers; llm-rubric, factuality, and similar assertions grade via your ANTHROPIC_API_KEY), and its Claude Agent SDK provider (anthropic:claude-agent-sdk) runs probes through the real agent loop, working directory, tools, --max-turns and all, which is the only honest way to eval agentic memory, since the thing under test is partly the agent's decision to call the memory tool at all.

Remember. The harness is only trustworthy while it is boring: the probe file versioned and stable, one disturbance per run, fresh sessions unless replay is the thing being tested, staleness and abstention probes always present, and the judge a different model than the answerer. Every exciting memory-eval number you will ever read violated at least one of those.

Further reading

• Wu et al., "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory" (ICLR 2025, arXiv 2410.10813): the five abilities, and the 30% drop; its question typology is the best template for a personal probe set.
• Maharana et al., "Evaluating Very Long-Term Conversational Memory of LLM Agents" (ACL 2024, arXiv 2402.17753): LoCoMo; read together with Zep's and mem0's dueling re-evaluations as a case study in why configuration is the result.
• Chen, "Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents" (2026, arXiv 2606.22528): the ConstraintRot numbers behind Recipe 3, and the "constraint pinning" mitigation.
• Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023, arXiv 2306.05685): the judge's license to operate, and its bias disclosures.
• Anthropic, "Define success criteria and build evaluations" (platform.claude.com/docs/en/test-and-evaluate/develop-tests) and the compaction and headless mode docs (code.claude.com/docs/en/headless): the grading patterns and every flag the recipes used.

Takeaways

• One loop underneath every memory benchmark: plant typed facts (including superseded and absent ones), disturb one thing, probe against ground truth, score recall, staleness, abstention, and tokens carried.
• The lab catches each failure as a number: window caps erase by distance (1/5), compaction drops by type (3/5, details and procedures first), full replay recalls but serves stale facts (4/5 with 1 stale), update-in-place scores clean (5/5).
• Resume tests replay; fresh sessions test memory. Run both and the difference names the layer doing the work.
• Compaction is a behavior contract, not housekeeping: 0% policy violations before, 30% average after, 59% worst-case; probe by fact type and buy back the losses with focus instructions (or the API's instructions field).
• Published memory numbers rank papers, and the vendors publicly dispute each other's configurations; hold your own probe set constant across changes, sessions, and models, and grade with a model that did not write the answers.

👉 That closes the memory arc: what your facts cost to carry, how the real systems carry them, and how to prove any of it. The next part takes the same discipline underground: the caching research lineage (the people, papers, and code behind every price multiplier), an autopsy of a real 272-call session's cache lifecycle, and the practice card that compresses this whole book into rules with proofs attached. Continue to The caching lineage.