Vector memory: similarity search, built and chosen

The state store answers exact-key questions: give me run 42, give me unit 7's lease. Memory asks a different kind of question, "what do we know about database timeouts," where the useful answer is memories that are similar to the query without containing its exact words. That is similarity search, its engine is a vector index, and this chapter builds one from scratch so the mechanism is demystified, then gives the honest decision matrix for which managed vector store to actually run.

How similarity search works

The idea is two moves. First, embed: turn each piece of text into a vector, a list of numbers, positioned so that texts about similar things land near each other in the space. Second, rank: embed the query the same way and return the stored vectors closest to it, where "close" is usually cosine similarity, the cosine of the angle between two vectors, which is 1 for identical direction and 0 for unrelated. Embed once per memory, embed each query, sort by cosine, take the top few. That is the entire mechanism, and the from-scratch lab is barely thirty lines.

The one piece the lab cannot honestly fake is the embedding itself. Real systems use a model to embed, which is what lets "database timeout" retrieve a memory about "connection pool exhaustion" despite zero shared words: the model maps meaning, not spelling. The lab uses a deliberately labeled toy, a hashed bag-of-words, which matches on shared vocabulary rather than shared meaning. Everything else, the cosine ranking, the top-k, the store interface, is exactly what a production vector store does; only the embedding function would be swapped. Keeping the toy honest (it is lexical, not semantic) is what lets the lab teach the mechanics without pretending to intelligence it does not have.

Lab 7.2: vectors next to grep

The lab pits the vector store against a grep tool (exact substring match) over a small memory corpus, precisely to mark where each wins:

python3 vector_mem.py

--- vector search: a fuzzy, multi-concept query ---
query: 'why did checkout time out because of the database pool at night'
no single memory contains that phrase; vectors rank by overlap:
  0.320  Rollback of v841 restored the connection pool and cleared the timeouts.
  0.204  Checkout p99 latency is normally around 210 ms.
  0.192  Incident INC-2091: pool exhaustion caused checkout timeouts overnight.
grep for the full phrase: 0 hits (exact-substring search finds nothing)

--- grep wins: exact set membership, no threshold ---
task: 'every memory that mentions v841' (a specific deploy id)
grep: exactly 2 memories, complete and precise:
  The payments-db connection pool was resized from 50 to 10 in deploy v841.
  Rollback of v841 restored the connection pool and cleared the timeouts.
vector top-3 for 'v841' must return 3 whether or not 3 are relevant:
  0.277  [relevant] Rollback of v841 restored the connection poo...
  0.267  [relevant] The payments-db connection pool was resized ...
  0.000  [NOISE (below any threshold)] The staging environment deploys are frozen o...

The two halves are the chapter's real lesson, and it is not "vectors are better." Vectors win at fuzzy recall: a seven-concept query that no single memory contains verbatim still gets a sensible ranked answer, with the rollback and the incident surfacing near the top, while grep for the literal phrase returns nothing. Grep wins at exact set membership: "every memory mentioning v841" has a precise, complete answer (exactly two), and the vector store can only approximate it, forced to return three whether or not three are relevant, its tail a 0.000 noise entry you would need a similarity threshold to exclude. Ranked-and-fuzzy versus exact-and-complete are different tools, and the closing line is the design: an agent wants both. Grep (or a keyword index) for ids, codes, and names; vector search for themes. A memory system that offers only one has amputated half of retrieval, and the frequent, embarrassing failure is a vector-only store that cannot reliably answer "find the ticket numbered EXACTLY INC-2091."

Choosing the store to run

The mechanism is universal; the operational engine is a real decision, and this is where S3 Vectors, OpenSearch, pgvector, and an in-process index genuinely differ. The axes that decide it:

Three heuristics compress the table. Start smallest that fits: an in-process index over a few thousand memories, rebuilt from the event store on boot, beats standing up a cluster you do not yet need. Let the query shape choose the middle: if you need heavy metadata filtering and hybrid keyword-plus-vector ranking at low latency, that is OpenSearch's job and S3 Vectors will frustrate you; if you need cheap storage of a huge, occasionally- queried memory and can tolerate ~100ms, that is S3 Vectors' job and an OpenSearch cluster is a standing bill you did not need. Follow the pipeline: the managed memory services and Bedrock Knowledge Bases increasingly use S3 Vectors underneath, so "which store" is sometimes already answered by "which pipeline," and the decision moves up a level. (Prices and exact latencies move; re-check them, and re-run the cost arithmetic against your own vector count and query rate before committing.)

Don't be confused: an embedding vs a memory. The embedding is a lossy address, a point in space that says roughly what a memory is about; the memory is the text itself. You retrieve by the embedding but you use the text. Two consequences follow: never store only the vector (you cannot read a vector back to the agent), and never trust the vector's neighborhood as truth (it is a fuzzy address, and the next-but-one chapter's eval harness exists precisely because that fuzziness degrades as the store grows).

Full source

"""Lab 7.2: vector memory from scratch, next to a grep tool.

A vector store, stripped to its mechanics: embed every memory once,
embed the query, rank by cosine similarity, return the top few. The
embedding here is a deliberately honest toy: a hashed bag-of-words, so
it matches on shared vocabulary, NOT on meaning. Real systems swap in a
model embedding, which extends the same cosine mechanic from
"same words" to "same idea"; everything else in this file is unchanged.

The demo contrasts vector search with a grep tool (exact substring) to
show where each wins: vectors rank fuzzy, multi-concept queries that no
single memory contains verbatim; grep nails exact identifiers that
vectors dilute among common words.

Standard library only. Deterministic: same run, same rankings.
"""

from __future__ import annotations

import hashlib
import math
import re

DIM = 256


def tokens(text: str) -> list[str]:
    return re.findall(r"[a-z0-9]+", text.lower())


def embed(text: str) -> list[float]:
    """Hashed bag-of-words into DIM dims, L2-normalized. A toy stand-in
    for a model embedding: shared words -> similar vectors."""
    v = [0.0] * DIM
    for tok in tokens(text):
        h = int(hashlib.sha1(tok.encode()).hexdigest(), 16)
        v[h % DIM] += 1.0
    norm = math.sqrt(sum(x * x for x in v)) or 1.0
    return [x / norm for x in v]


def cosine(a: list[float], b: list[float]) -> float:
    return sum(x * y for x, y in zip(a, b))


class VectorStore:
    def __init__(self, memories: list[str]):
        self.memories = memories
        self.vectors = [embed(m) for m in memories]

    def search(self, query: str, k: int = 3) -> list[tuple[float, str]]:
        q = embed(query)
        scored = sorted(((cosine(q, v), m)
                         for v, m in zip(self.vectors, self.memories)),
                        reverse=True)
        return scored[:k]


def grep(memories: list[str], phrase: str) -> list[str]:
    return [m for m in memories if phrase.lower() in m.lower()]


MEMORIES = [
    "The payments-db connection pool was resized from 50 to 10 in deploy v841.",
    "Checkout p99 latency is normally around 210 ms.",
    "The staging environment deploys are frozen on Fridays after 3pm.",
    "Incident INC-2091: pool exhaustion caused checkout timeouts overnight.",
    "The receipt email service uses template engine v3 since March.",
    "Database migrations run through the Flyway pipeline, never by hand.",
    "The on-call rotation for payments is owned by the Ledger team.",
    "Rollback of v841 restored the connection pool and cleared the timeouts.",
]


if __name__ == "__main__":
    store = VectorStore(MEMORIES)

    print("--- vector search: a fuzzy, multi-concept query ---")
    q1 = "why did checkout time out because of the database pool at night"
    print(f"query: {q1!r}")
    print("no single memory contains that phrase; vectors rank by overlap:")
    for score, mem in store.search(q1, k=3):
        print(f"  {score:.3f}  {mem}")
    print(f"grep for the full phrase: {len(grep(MEMORIES, q1))} hits "
          "(exact-substring search finds nothing)")

    print("\n--- grep wins: exact set membership, no threshold ---")
    print("task: 'every memory that mentions v841' (a specific deploy id)")
    hits = grep(MEMORIES, "v841")
    print(f"grep: exactly {len(hits)} memories, complete and precise:")
    for mem in hits:
        print(f"  {mem}")
    print("vector top-3 for 'v841' must return 3 whether or not 3 are "
          "relevant:")
    for score, mem in store.search("v841", k=3):
        tag = "relevant" if "v841" in mem else "NOISE (below any threshold)"
        print(f"  {score:.3f}  [{tag}] {mem[:44]}...")
    print("the ranked list needs a similarity cutoff to recover grep's "
          "clean set; pick it too low and the tail is noise, too high and "
          "you drop real hits.")

    print("\nthe lesson: agents want BOTH as tools. Grep for exact set "
          "membership (ids, codes, names: precise, complete, no cutoff);")
    print("vector search for 'what do we know about X' where X is a "
          "theme, and rank-with-a-threshold is exactly what you want.")

👉 Next: knowledge for agents, where retrieval stops being a chat-style one-shot lookup and becomes something an agent can iterate, reformulate, and verify, which changes the design more than the store choice does.