Long-term memory pipelines: consolidation, staleness, and proof
The state store holds a run's events; the vector index makes memories searchable; the knowledge chapter separated authored knowledge from earned memory. This chapter closes the loop: how the agent's own experience becomes durable memory, why that memory rots if left alone, and the harness that turns "our memory system feels good" into a number. It is where the Part 3 preview of the five-stage pipeline gets its hard back half built, because capture and storage were the easy stages and consolidation, staleness, and evaluation are where memory systems actually fail.
Consolidation: transcripts are not memory
Raw transcripts are a terrible memory. They are long, repetitive, full of dead ends, and mostly irrelevant to any future task. The capture stage hands you all of it (the event ledger already has every turn); the consolidation stage is what turns that firehose into memory worth keeping, and it is a distillation job, usually a model call: read a finished run's transcript, extract the handful of durable facts worth carrying forward, discard the rest.
The design decisions live here, not in the storage:
- What to extract. Stable, reusable facts ("the payments-db pool is sized in deploy config, not code"), not run-specific ephemera ("checked the metrics at 02:14"). Tuned loose, consolidation floods the store with trivia that poisons retrieval; tuned tight, it forgets what mattered. This is a precision-recall dial with no vendor default that fits your domain, which is why the eval harness below is not optional.
- When to run it. Consolidation is an offline job, not a hot-path step: it runs after a run completes (or on a schedule over recent runs), which is a natural fit for the batch tier and its half price, since nothing waits on it.
- How to write it. Every consolidated memory is also an event, carrying its provenance (below) into the same durable, replayable ledger as everything else.
Provenance and staleness: the hard half
Two properties separate a memory system from a rumor mill, and both are absent from the naive "embed the fact, store the vector" design.
Provenance is the memory's evidence: which run produced it, from which turn, citing which tool result. A memory without provenance is unfalsifiable, you cannot check it, correct it, or explain it, and an agent acting on an unfalsifiable memory is guessing with extra steps. The fix is cheap because event sourcing already records the source: consolidation carries the source event ids onto the memory, and the verification mesh can then check a memory the same way it checks any claim.
Staleness is the deeper problem, and the Part 3 preview named it: facts age. "The pool size is 10" was true after v841 and false after the rollback. A memory store that only ever adds accumulates contradictions, and, as the lab is about to show, retrieval by similarity has no idea which of two contradictory memories is current, because recency is not a direction in the vector space. The policies that fix it: timestamps on every memory, supersede links when a new fact contradicts an old one (retire the old, do not just add the new), and overwrite-on- contradiction during consolidation (detect that a fresh fact conflicts with a stored one and replace rather than accumulate). Forgetting, it turns out, is a feature you must build.
Lab 7.3: does the memory work?
Everything above is asserted; memory quality has to be measured, and the harness that does it is the plant-disturb-probe-score loop: plant known facts, disturb the store with the topically-adjacent noise every real store accumulates, probe with queries derived from the facts, and score whether retrieval still finds them. Reusing the Chapter 33 vector store:
python3 memory_eval.py
plant 5 facts, probe each, sweep distractor count
distractors recall@1 recall@3
0 100% 100%
20 100% 100%
100 80% 100%
500 80% 80%
recall decays as the store fills with same-domain noise: a distractor that shares a probe word can outrank the true fact.
--- the staleness failure: ranking cannot retire a fact ---
planted a fact, then a superseding correction of it.
probe: 'when is the deploy freeze window'
0.894 The deploy freeze window is Fridays after 3pm.
0.661 Update: the deploy freeze window is now all day Friday.
both are retrieved and similarity does not encode recency, so the agent cannot tell which is current.
Two failures, both quantified, both the point:
- Memory rots as it grows. At an empty store, retrieval is perfect; add same-domain noise and recall decays, 100% to 80% at recall@1 by 100 distractors, and recall@3 slipping by 500. The mechanism is honest even for a toy embedding: a distractor sharing a probe word can outrank the true fact, and more distractors mean more chances. Real stores with model embeddings decay more gracefully but they do decay, and the number that matters, retrieval quality, is not a launch-day constant but a metric that drifts down as the store fills. If you are not measuring it on a schedule, you are trusting a quantity you have watched degrade.
- Ranking cannot retire a fact. The staleness demo is the sharper result: the stale fact (0.894) outranks its own correction (0.661), because the stale wording happens to match the probe more closely and similarity knows nothing of time. An agent trusting top-1 would act on the retired fact. No embedding upgrade fixes this; it is a data-model problem, solved by the supersede and timestamp policies above, not by a better vector.
This harness is the memory analog of Part 1's replay and Part 6's verification mesh: a way to make a fuzzy quality concrete and regression-testable. Wired into Part 9's eval gates, plant-disturb-probe-score runs in CI against your real store and real embeddings, so a consolidation change that quietly tanks recall fails a test instead of a user.
The memory plane, assembled
Part 7 in one arc: exact-keyed run state on a single conditional-write table (Chapter 32); similarity search built from scratch and its store chosen by workload (Chapter 33); retrieval reframed as an iterating tool rather than a one-shot pipeline (Chapter 34); and now earned memory with consolidation, provenance, staleness policy, and an eval harness that proves the whole thing works. The state-and-memory plane of the reference architecture now exists in runnable form, and every piece of it feeds the capstones.
Don't be confused: forgetting vs losing. Losing memory is a bug: a crash that drops the event ledger, data you needed and cannot recover. Forgetting is a feature: deliberately retiring stale facts, pruning trivia consolidation should never have kept, decaying memories no longer consulted. A memory system judged only on what it retains is half-designed; the other half is what it correctly discards, and the eval harness is how you tell disciplined forgetting from accidental amnesia.
Full source
"""Lab 7.3: does the memory actually work? Plant, disturb, probe, score.
Memory quality is not a feeling; it is a measurement. This harness
plants known facts, disturbs the store with same-domain distractor
memories (the topically-adjacent noise every real store accumulates),
probes with queries derived from the planted facts, and scores whether
retrieval still surfaces the right one. It sweeps the distractor count
to show recall decaying as the store fills, then demonstrates the
staleness failure: a superseded fact that ranking alone cannot retire.
Reuses the Lab 7.2 vector store. Standard library only. Deterministic.
"""
from __future__ import annotations
import random
from vector_mem import VectorStore
# planted (fact, probe) pairs: the probe shares vocabulary with its fact
PLANTED = [
("The primary datacenter for the payments service is in Frankfurt.",
"which datacenter hosts the payments service"),
("The backup retention window for customer data is ninety days.",
"how long is the backup retention window for customer data"),
("The load test target for checkout is three thousand requests per second.",
"what is the load test target for checkout"),
("Release approvals require two reviewers from the security team.",
"how many reviewers does a release need from the security team"),
("Incident postmortems are due within five business days.",
"when are incident postmortems due"),
]
# same-domain vocabulary: distractors are plausible near-misses, not gibberish
NOUNS = ["datacenter", "payments", "checkout", "backup", "retention",
"release", "reviewers", "security", "incident", "postmortems",
"data", "service", "deploy", "latency", "pipeline", "customer",
"window", "target", "team", "approvals"]
VERBS = ["requires", "runs", "hosts", "targets", "reviews", "schedules",
"retains", "monitors", "rotates", "audits"]
def distractors(n: int) -> list[str]:
rng = random.Random(1)
out = []
for _ in range(n):
a, b = rng.choice(NOUNS), rng.choice(NOUNS)
out.append(f"The {a} {rng.choice(VERBS)} the {b} "
f"{rng.choice(['weekly', 'nightly', 'per region', 'on call'])}.")
return out
def score(n_distract: int) -> tuple[float, float]:
facts = [f for f, _ in PLANTED]
store = VectorStore(facts + distractors(n_distract))
hit1 = hit3 = 0
for fact, probe in PLANTED:
ranked = [m for _, m in store.search(probe, k=3)]
if ranked and ranked[0] == fact:
hit1 += 1
if fact in ranked:
hit3 += 1
return hit1 / len(PLANTED), hit3 / len(PLANTED)
if __name__ == "__main__":
print(f"plant {len(PLANTED)} facts, probe each, sweep distractor count\n")
print(f"{'distractors':>12}{'recall@1':>11}{'recall@3':>11}")
for n in (0, 20, 100, 500):
r1, r3 = score(n)
print(f"{n:>12}{r1:>10.0%}{r3:>11.0%}")
print("\nrecall decays as the store fills with same-domain noise: a "
"distractor that shares a probe word can outrank the true fact.")
print("this is the number that must be monitored, not assumed; a "
"memory store silently rots as it grows.")
print("\n--- the staleness failure: ranking cannot retire a fact ---")
stale = "The deploy freeze window is Fridays after 3pm."
fresh = "Update: the deploy freeze window is now all day Friday."
store = VectorStore([stale, fresh])
print("planted a fact, then a superseding correction of it.")
print("probe: 'when is the deploy freeze window'")
for sc, mem in store.search("when is the deploy freeze window", k=2):
print(f" {sc:.3f} {mem}")
print("both are retrieved and similarity does not encode recency, so "
"the agent cannot tell which is current.")
print("the fix is not a better embedding; it is a staleness policy: "
"timestamps, supersede links, or overwrite-on-contradiction.")
👉 Next, per the map: trust (Part 8) and operations (Part 9) turn this machinery into a platform you can defend and run, then the protocols and capstones that finish the book.