Semantic and response caching

TL;DR. When a new question means the same thing as one you have already answered, return the stored answer and skip the model entirely: no prefill, no decode, no output tokens. You decide "means the same thing" by turning each question into a vector with an embedding, measuring the angle between vectors with cosine similarity, and comparing that score to a threshold. The threshold is a precision/recall dial: too low and you serve one question's answer to a different question (a false hit), too high and you needlessly call the model on real paraphrases (a false miss). The saving scales with how repetitive your traffic is.

Contents

Chapter 6 cached the stable front of the prompt so the model did not re-read it, but the model still ran on every call. This chapter goes one step further: when a new question is close enough to one you have already answered, return the stored answer and skip the model entirely. No prefill, no decode, no output tokens. The whole call collapses into a lookup.

That is a much bigger saving than prefix caching, and it comes with a risk prefix caching does not have. Prefix caching reuses input only on an exact byte match, so a reused prefix is always correct by construction. Semantic caching reuses the answer on an approximate match, so it can be wrong: serve a cached answer to a question that only looked similar, and the user gets a confidently wrong reply. The whole craft is in setting the match bar high enough to avoid that.

The idea: answer once, serve many

Support and FAQ traffic is full of repeats. "How do I reset my password?", "How can I reset my password?", and "I forgot my password, how do I reset it?" are three spellings of one question with one answer. A naive system calls the model three times and pays three times. A semantic cache calls the model once, stores the answer, and serves the other two from memory.

The purpose is to remove calls you should not be making at all. The benefit is that on a near-duplicate you pay a millisecond-scale lookup instead of a full generation, which on the claude-opus-4-8 rates from Chapter 2 is where the dollars and the seconds both go. The use cases are any workload with a high duplicate rate: FAQ and support bots (the same handful of questions, asked a thousand ways), repeated analytics ("what were sales last quarter?" asked fifty times across a team), and any service with heavy near-duplicate traffic. The places it does not help are open-ended creative work and per-user-unique requests, where almost nothing repeats and every lookup is a guaranteed miss that still costs you the embedding step.

To do that, the cache needs to decide whether a new question means the same thing as one it has seen. It cannot compare the raw text, because the words differ: "reset my password" and "I forgot my password, how do I reset it?" share almost no characters in common order, yet they are the same request. String equality, edit distance, and keyword overlap all break on paraphrase. The cache needs a way to measure meaning, not spelling. That is what an embedding gives you.

What an embedding really is

An embedding is a function from text to a fixed-length list of numbers (a vector), built so that texts with similar meaning produce vectors that point in similar directions. The "fixed-length" part matters: every question, long or short, becomes the same number of coordinates (say 256, or 768 in a real model), so you can compare any two questions with the same arithmetic. The "similar meaning, similar direction" part is the whole point: the embedder lays out a space where nearness encodes relatedness, and "is this a paraphrase?" becomes "are these two points close?"

A vector representation of text is just that point. Picture each coordinate as one axis of meaning. In a trained embedder no single axis is human-readable, but the geometry is real: the vector for "refund" sits near the vector for "money back" and far from the vector for "office hours", because the network was trained to place them that way. You never look at the individual numbers; you only ever ask how two vectors relate, and the next section makes that question precise.

The mechanics of comparing by vector do not depend on how good the embedder is, so we can build a crude one from scratch and the caching logic around it is identical to the real thing. Our toy embedder is a hashing bag-of-words: it splits the question into words, drops filler words ("how", "do", "the"), hashes each remaining content word to a slot in a fixed-size vector, adds to that slot, and scales the whole vector to unit length. Two questions that share content words end up adding to the same slots, so they point the same way.

That is a weak embedder, and it is worth being honest about why, because the weakness is exactly what a real embedder fixes. A hashing bag-of-words knows only which words appear, not what they mean. It has three blind spots:

  • Synonyms look unrelated. "refund" and "money back" share no words, so they land in different slots and score near zero, even though they mean the same thing. Our toy patches this slightly with character trigrams (so "refund" and "refunds" overlap), but it cannot connect two entirely different words for one concept.
  • Word order is ignored. "dog bites man" and "man bites dog" produce the identical vector, because a bag of words throws order away. For short FAQ questions this rarely bites, but it is a real limit.
  • Shared filler inflates similarity. Two unrelated questions that both start "How do I..." would look similar if we kept those words, which is why the toy drops stopwords by hand. A real embedder learns to down-weight them.

A trained sentence embedder has none of these blind spots, because it was not hand-built; it was learned. A network (Sentence-BERT and its successors are the standard example) is trained on millions of sentence pairs labeled "same meaning" or "different meaning", and it adjusts its weights until same-meaning pairs land close together and different-meaning pairs land far apart. The result is a function that puts "How do I get a refund?" near "What is your refund policy?" even though they share one word, and keeps "cancel my subscription" far from "cancel my flight" even though they share a word. That is the difference between counting words and modeling meaning, and it is why every production cache uses a trained embedder. The toy is here so you can see the geometry without a GPU; swap in a real embedder and the numbers get better while the machinery stays identical.

Cosine similarity: measuring meaning by angle

Given two unit vectors, how "similar" are they? Use cosine similarity, the cosine of the angle between them:

$$\text{cos}(a, b) = \frac{a \cdot b}{\lVert a \rVert , \lVert b \rVert}$$

Here $a \cdot b$ is the dot product (multiply the two vectors element by element and sum the results), and $\lVert a \rVert$ is the length (norm) of $a$. The ratio runs from $-1$ (opposite meaning) through $0$ (unrelated) to $1$ (identical direction). When both vectors are already unit length, the denominator is $1 \times 1 = 1$, so cosine similarity is just the dot product. That is why the embedder normalizes: it makes the comparison a single cheap multiply-and-add.

Why the angle and not the straight-line distance? Because the angle measures direction, and direction is where the embedder put the meaning. Two questions about the same topic point the same way regardless of how long each one is; a short "reset my password" and a verbose "I forgot my password, could you tell me how to reset it?" should match, and they do, because normalizing to unit length strips out length and leaves only direction. Cosine similarity asks the one question you care about, "do these point the same way?", and ignores the one you do not, "how long is each vector?". This is also why it is the standard score for text search and for every semantic cache: it is cheap (one dot product), bounded (always in $[-1, 1]$), and length-invariant.

Remember. The embedding turns "do these two questions mean the same thing?" into "do these two vectors point the same way?", and cosine similarity answers that with a single dot product. Everything else in this chapter (the threshold, the false hit, the eviction policy) is bookkeeping around that one comparison. Get the embedder right and the comparison is trustworthy; get it wrong and no threshold can save you.

The threshold: a precision/recall dial

A threshold is the cutoff that turns a similarity score into a yes/no decision. Pick a number, say $0.8$. For a new question, embed it, score it against every cached question, and take the best match.

  • If the best score is at or above the threshold, that is a hit: return the stored answer and skip the model.
  • If the best score is below the threshold, that is a miss: call the model, store the new (question, answer) pair, and return the fresh answer.

The cleanest way to think about the threshold is as a precision/recall dial, the same trade-off that runs through all of retrieval. In this setting:

  • Precision is the fraction of hits that are correct, the cache returned the right answer. A false hit (wrong answer served) hurts precision.
  • Recall is the fraction of true paraphrases that the cache actually catches. A false miss (a real paraphrase sent to the model anyway) hurts recall.

Raise the threshold and the cache becomes strict: precision goes up (every hit is a genuine match) but recall goes down (you miss real paraphrases that scored just under the bar). Lower it and the cache becomes greedy: recall goes up (you catch more paraphrases) but precision falls (you start serving cached answers to questions that only share a few words). There is no single setting that maximizes both; the dial moves them in opposite directions, and you choose where to sit based on what a mistake costs.

For a semantic cache the two mistakes are not symmetric, and that asymmetry decides the dial. A false miss costs one extra model call: real money, but bounded and recoverable, and the user still gets a correct answer. A false hit costs trust: the user gets a confidently wrong answer with no signal that anything went wrong. Because a wrong answer is far more expensive than a redundant call, you set the dial toward precision, which means a high threshold. The lesson, stated plainly: you cannot push the hit rate to 100% without eventually serving someone the wrong answer. Pick the threshold for your risk tolerance, not for the prettiest hit-rate number. Most production caches land around $0.8$ to $0.85$ on a good embedder and tune from there against real traffic.

That trade-off is the heart of the chapter, and the demo makes it concrete with exact numbers.

Don't be confused. Prefix caching (Chapter 6) and semantic caching are both "caching", but they reuse different things under different match rules. Prefix caching reuses the input tokens of a stable prompt prefix, keyed on an exact byte match, and the model still runs to produce the answer: a reused prefix is always correct because the bytes are identical. Semantic caching reuses the whole answer, keyed on an approximate similarity match, and the model does not run at all. The payoff is larger (you skip generation, the expensive half from Chapter 2) and so is the failure mode: an approximate match can be wrong, handing one question's answer to a different question. Prefix caching can never serve a wrong answer; semantic caching can, and guarding against that is the threshold's job.

The cache, from scratch

The demo builds the whole thing in NumPy and the standard library: the hashing embedder, cosine similarity, the threshold decision, and a stub model that stands in for a real call. It runs a workload of ten questions, several of which are paraphrases of earlier ones, and reports the hit rate plus the latency and dollars saved. Then it lowers the threshold and demonstrates a false hit: a different question wrongly served a cached answer.

"""Semantic (response) caching from scratch: return a STORED ANSWER for a
SIMILAR question, skipping inference entirely.

Chapter 6 caches the prompt PREFIX (the stable input tokens) and still runs the
model. This is different: we cache the whole ANSWER, and on a near-duplicate
question we skip the model call completely. That is a much bigger saving and a
new risk (a FALSE HIT: a different question served the wrong cached answer).

Four pieces, all in numpy + stdlib:
  1. A tiny embedding: hash each word into a fixed-dim vector, sum, L2-normalize.
     No external embedding model. (Real caches use a learned sentence embedder;
     the mechanics of compare-by-vector are identical.)
  2. Cosine similarity between a new question's vector and each cached vector.
  3. A THRESHOLD: best match >= threshold -> HIT (return stored answer, skip LLM);
     else MISS (call the stub LLM, store (question, answer), return it).
  4. The threshold trade-off: lower it and watch a different question get a wrong
     cached answer.

Standard library + numpy only. Run:  python3 semantic_cache.py
"""

import hashlib

import numpy as np


def _hash(s):
    """A deterministic hash of a string to a non-negative int. Python's built-in
    hash() is randomized per process (so output would change run to run); md5
    keeps this demo reproducible. Any fixed hash works; the values are arbitrary."""
    return int.from_bytes(hashlib.md5(s.encode()).digest()[:8], "big")

# --- Cost / latency model (one place to change the assumptions) ---------------
# A cache lookup is cheap and local; an LLM call is slow and billed per token.
LOOKUP_MS = 10        # embed + cosine search, in milliseconds
LLM_MS = 1500         # one model call, in milliseconds
OUT_TOKENS = 500      # tokens the model writes per answer
PRICE_PER_MTOK = 25.0 # output price, US dollars per million tokens (claude-opus-4-8)
COST_PER_CALL = (OUT_TOKENS / 1e6) * PRICE_PER_MTOK  # dollars per LLM call

DIM = 256  # embedding dimension; bigger = fewer hash collisions

# Words that carry no topic ('how', 'do', 'the', ...). A real embedder learns to
# down-weight these; we just drop them so the vector reflects the CONTENT words.
STOPWORDS = set(
    "how do i can you what is are your please tell about me a an the to it of "
    "my your for".split()
)


# --- 1. A tiny embedding: hashing bag-of-words --------------------------------
def embed(text):
    """Turn text into a unit vector so that similar questions land near each other.

    'Embedding' just means: a function from text to a fixed-length vector of
    numbers, built so that texts with similar meaning point in similar directions.
    A real embedder is a trained network; this one is a cheap stand-in with two
    honest tricks so plain word-overlap is not fooled by filler:

      * drop STOPWORDS, so 'how do i' does not dominate the vector;
      * for each remaining word, hash the WORD and also its character trigrams
        (3-letter windows). Trigrams make 'refund' and 'refunds' look alike and
        give content words more weight than a single slot would.

    We hash each feature to an index in [0, DIM) and add to that slot, building a
    bag-of-features count vector, then L2-normalize it (divide by its length) so
    every vector sits on the unit sphere. Normalizing is what lets cosine
    similarity below be a plain dot product.
    """
    v = np.zeros(DIM, dtype=np.float64)
    for raw in text.lower().split():
        word = raw.strip("?.,!;:'\"")
        if not word or word in STOPWORDS:
            continue
        v[_hash("word:" + word) % DIM] += 1.0           # the whole word
        for i in range(len(word) - 2):                  # its character trigrams
            v[_hash("tri:" + word[i:i + 3]) % DIM] += 0.5
    norm = np.linalg.norm(v)
    return v / norm if norm > 0 else v


# --- 2. Cosine similarity -----------------------------------------------------
def cosine(a, b):
    """Cosine similarity: the cosine of the angle between two vectors.

      cos(a, b) = (a . b) / (||a|| ||b||)

    The dot product a . b sums the element-wise products; ||a|| is a's length.
    The result runs from -1 (opposite) through 0 (unrelated) to 1 (identical
    direction). Because embed() already returns unit-length vectors, ||a|| and
    ||b|| are both 1, so this reduces to the dot product a . b.
    """
    return float(np.dot(a, b))


# --- 3. The stub LLM (what we are trying to avoid calling) --------------------
LLM_CALLS = 0  # count real model calls so we can prove the saving

def stub_llm(question):
    """Stand-in for a real model call: slow and expensive in the cost model.
    Returns a canned 'answer' so the demo is deterministic and offline."""
    global LLM_CALLS
    LLM_CALLS += 1
    return f"[answer to: {question!r}]"


# --- The cache ----------------------------------------------------------------
class SemanticCache:
    def __init__(self, threshold):
        self.threshold = threshold
        self.vectors = []   # cached question embeddings
        self.questions = [] # the original question text (for inspection)
        self.answers = []   # the stored answer for each cached question

    def lookup(self, question):
        """Return (answer, matched_question, score) on a hit, or None on a miss.
        A hit means: some cached question's embedding is at least `threshold`
        similar to this one. We return the closest match above the line."""
        if not self.vectors:
            return None
        q = embed(question)
        scores = [cosine(q, v) for v in self.vectors]
        best = int(np.argmax(scores))
        if scores[best] >= self.threshold:
            return self.answers[best], self.questions[best], scores[best]
        return None

    def store(self, question, answer):
        self.vectors.append(embed(question))
        self.questions.append(question)
        self.answers.append(answer)

    def ask(self, question):
        """The full path: try the cache, else call the model and cache the result.
        Returns (answer, was_hit, latency_ms, dollars_spent)."""
        hit = self.lookup(question)
        if hit is not None:
            answer, _, _ = hit
            return answer, True, LOOKUP_MS, 0.0
        answer = stub_llm(question)
        self.store(question, answer)
        # A miss pays the lookup AND the model call (we looked first, found nothing).
        return answer, False, LOOKUP_MS + LLM_MS, COST_PER_CALL


# --- A workload with deliberate near-duplicate paraphrases --------------------
# Three distinct intents, each asked several ways. A good cache should serve the
# paraphrases from the answer it computed the first time.
WORKLOAD = [
    "How do I reset my password?",            # intent A, first time -> MISS
    "What is your refund policy?",            # intent B, first time -> MISS
    "How can I reset my password?",           # A paraphrase -> should HIT
    "I forgot my password, how do I reset it?",  # A paraphrase -> should HIT
    "How do I get a refund?",                 # B paraphrase -> should HIT
    "Where is your office located?",          # intent C, first time -> MISS
    "How do I change my password?",           # A-ish paraphrase -> should HIT
    "Tell me about your refund policy please",   # B paraphrase -> should HIT
    "What are your office hours?",            # C-ish, different intent -> MISS
    "reset my password",                      # A paraphrase -> should HIT
]


def run(cache, label):
    print(f"=== {label} (threshold = {cache.threshold}) ===")
    global LLM_CALLS
    LLM_CALLS = 0
    hits = 0
    total_ms = 0.0
    total_dollars = 0.0
    for q in WORKLOAD:
        _, was_hit, ms, dollars = cache.ask(q)
        hits += was_hit
        total_ms += ms
        total_dollars += dollars
        tag = "HIT " if was_hit else "MISS"
        print(f"  {tag}  {q}")
    n = len(WORKLOAD)
    print(f"\n  hit rate:   {hits}/{n} = {hits / n:.0%}")
    print(f"  LLM calls:  {LLM_CALLS} (one per MISS)")
    return total_ms, total_dollars, hits


# --- BEFORE vs AFTER: no cache vs cache ---------------------------------------
n = len(WORKLOAD)
# Baseline: every question hits the model.
no_cache_ms = n * (LLM_MS)
no_cache_dollars = n * COST_PER_CALL

print("Cost model: cache lookup ~{}ms, LLM call ~{}ms and {} output tokens "
      "at ${}/Mtok = ${:.5f}/call.".format(
          LOOKUP_MS, LLM_MS, OUT_TOKENS, PRICE_PER_MTOK, COST_PER_CALL))
print(f"Workload: {n} questions, with paraphrases of earlier ones mixed in.\n")

print(f"--- BEFORE (no cache): every question calls the model ---")
print(f"  {n} LLM calls")
print(f"  latency: {no_cache_ms:,.0f} ms")
print(f"  cost:    ${no_cache_dollars:.5f}\n")

cache = SemanticCache(threshold=0.8)
cache_ms, cache_dollars, hits = run(cache, "AFTER (semantic cache)")

print(f"\n--- SAVED by the cache ---")
print(f"  latency: {no_cache_ms - cache_ms:,.0f} ms "
      f"({1 - cache_ms / no_cache_ms:.0%} faster)")
print(f"  cost:    ${no_cache_dollars - cache_dollars:.5f} "
      f"({1 - cache_dollars / no_cache_dollars:.0%} cheaper)")
print()


# --- 4. The threshold trade-off: a FALSE HIT ----------------------------------
# Lower the bar and the cache gets greedy: it starts serving cached answers to
# questions that only LOOK similar. Here two DIFFERENT intents share a content
# word ('cancel') and a loose threshold collapses them into one answer.
print("=== The threshold trade-off: a FALSE HIT ===")
first = "How do I cancel my subscription?"
second = "How do I cancel my flight?"   # different intent, shares 'cancel'
score = cosine(embed(first), embed(second))
print(f"  cached question:  {first}")
print(f"  new question:     {second}")
print(f"  similarity:       {score:.2f}\n")

# At the safe threshold the cache correctly MISSES and would ask the model.
safe = SemanticCache(threshold=0.8)
safe.store(first, stub_llm(first))
print(f"  threshold 0.8 -> "
      f"{'HIT' if safe.lookup(second) else 'MISS'}  (correct: these are different "
      f"questions, ask the model)")

# Drop the threshold below the score and the same pair becomes a wrong HIT.
greedy = SemanticCache(threshold=0.4)
greedy.store(first, stub_llm(first))
hit = greedy.lookup(second)
answer = hit[0] if hit else None
print(f"  threshold 0.4 -> {'HIT' if hit else 'MISS'}  served: {answer}")
print("  ^ WRONG. A loose threshold served the SUBSCRIPTION answer to a question")
print("    about a FLIGHT. This is the core risk of semantic caching: too low a")
print("    threshold trades correctness for hit rate. You cannot push hit rate to")
print("    100% without eventually serving someone the wrong answer.")

Running it:

Cost model: cache lookup ~10ms, LLM call ~1500ms and 500 output tokens at $25.0/Mtok = $0.01250/call.
Workload: 10 questions, with paraphrases of earlier ones mixed in.

--- BEFORE (no cache): every question calls the model ---
  10 LLM calls
  latency: 15,000 ms
  cost:    $0.12500

=== AFTER (semantic cache) (threshold = 0.8) ===
  MISS  How do I reset my password?
  MISS  What is your refund policy?
  HIT   How can I reset my password?
  HIT   I forgot my password, how do I reset it?
  MISS  How do I get a refund?
  MISS  Where is your office located?
  MISS  How do I change my password?
  HIT   Tell me about your refund policy please
  MISS  What are your office hours?
  HIT   reset my password

  hit rate:   4/10 = 40%
  LLM calls:  6 (one per MISS)

--- SAVED by the cache ---
  latency: 5,900 ms (39% faster)
  cost:    $0.05000 (40% cheaper)

=== The threshold trade-off: a FALSE HIT ===
  cached question:  How do I cancel my subscription?
  new question:     How do I cancel my flight?
  similarity:       0.43

  threshold 0.8 -> MISS  (correct: these are different questions, ask the model)
  threshold 0.4 -> HIT  served: [answer to: 'How do I cancel my subscription?']
  ^ WRONG. A loose threshold served the SUBSCRIPTION answer to a question
    about a FLIGHT. This is the core risk of semantic caching: too low a
    threshold trades correctness for hit rate. You cannot push hit rate to
    100% without eventually serving someone the wrong answer.

Read the AFTER block question by question. The first time each of the three topics appears (password, refund, office) it MISSes and calls the model: there is nothing to match against yet. After that, the paraphrases start hitting. "How can I reset my password?" scores high enough against the stored "How do I reset my password?" to clear the bar, so it returns the stored answer for free. So do "I forgot my password, how do I reset it?", "Tell me about your refund policy please", and the terse "reset my password".

Notice the two paraphrases that miss. "How do I get a refund?" is clearly about refunds, and a human would serve it the refund answer, but it shares almost no content words with the stored "What is your refund policy?" ("get" and "refund" versus "refund" and "policy"), so its score lands below $0.8$ and the cache plays it safe by asking the model. "How do I change my password?" misses too, and arguably it should: changing a password is a different operation from resetting one. These misses are the cost of a strict threshold. A higher hit rate is available, but only by lowering the bar, and the false-hit demo shows where that road ends.

What the saving is

The BEFORE/AFTER totals come straight from the cost model at the top of the file: a cache lookup is about 10ms, a model call is about 1500ms and writes about 500 output tokens at $25 per million tokens (the claude-opus-4-8 output rate from Chapter 2). On this ten-question workload, four hits cut six model calls down from ten, which is 40% fewer dollars and 39% less latency. The two numbers track each other because, in this simple model, every hit saves one full model call. Whether a real workload saves 5% or 60% depends entirely on its duplicate rate: the fraction of traffic that is a near-repeat of something already answered. FAQ bots and repeated analytics questions can be very high; open-ended creative requests can be near zero.

The false hit, and why the threshold is everything

The last block is the warning. Take two genuinely different questions that happen to share a content word: "How do I cancel my subscription?" and "How do I cancel my flight?". Both contain "cancel", so their vectors are not orthogonal; they score $0.43$. At the safe threshold of $0.8$ that is a MISS, which is correct: these are different questions, so the cache asks the model. Drop the threshold to $0.4$ and the same pair becomes a HIT, and now the cache hands the subscription-cancellation answer to someone asking about a flight. That is a false hit, and it is the one failure mode prefix caching cannot produce.

This is the trade-off in one picture. The threshold is a dial between two kinds of mistake. Set it too high and you get false misses: real paraphrases that should have hit but ask the model anyway, costing money you did not need to spend (the refund and change-password misses above). Set it too low and you get false hits: different questions served the wrong answer, costing trust. You cannot push the hit rate to 100% without eventually serving someone the wrong answer, so the right setting is conservative: better to pay for an extra model call than to confidently answer the wrong question. Most production caches land around $0.8$ to $0.85$ on a good embedder and tune from there against real traffic.

A real embedder changes the numbers but not the shape of this trade-off. A trained sentence embedder would score "How do I get a refund?" against "What is your refund policy?" much higher than our toy hashing embedder does, so it would catch that paraphrase at a safe threshold and lift the hit rate. It would also keep "cancel my subscription" and "cancel my flight" far apart. A better embedder buys you both a higher hit rate and fewer false hits at the same time, which is exactly why production systems use one. The threshold dial is still there; the embedder just makes its safe range wider.

Where the cache lives, eviction, and keys

Three operational questions decide whether a semantic cache helps or hurts in a real system, and none of them is about the embedder. They are about where the cache sits, when an entry is allowed to go stale, and what a single cache is allowed to contain.

Where it lives: client side, in front of the model. A semantic cache is part of your application, sitting between your code and the model API. Your request arrives, you embed it, you search your own stored vectors, and only on a miss do you call the model. The provider never sees the cache and is not involved in the decision; the embedder, the vector store, and the threshold are all yours. This is the opposite of prompt caching from Chapter 6, which is a server-side feature inside the provider's infrastructure. The two are independent and stack cleanly: the semantic cache removes calls you should not make, and prompt caching makes the calls you do make cheaper. This client-side placement is also why a semantic cache works with any provider and why it is the natural home for application-specific logic like scoping, which we get to below.

Eviction and TTL: stop serving stale answers. A cache that only ever grows has two problems. It runs out of memory, and worse, it keeps serving answers that have gone out of date. If your refund policy changed last week, an entry stored a month ago is now wrong, and a high threshold will not save you, because the question genuinely matches; only the answer is stale. Two standard controls handle this. A TTL (time-to-live) stamps each entry with an expiry, say 24 hours, after which a lookup treats it as absent and re-asks the model; you set the TTL to how long an answer stays true for your domain (minutes for live inventory, days for a stable FAQ). Eviction bounds the total size: when the store is full, drop the entry least likely to be useful, usually the least-recently-used (LRU) one, so hot questions stay cached and cold ones age out. Without TTL a semantic cache slowly turns into a source of confidently outdated answers; with it, staleness is bounded by a number you chose on purpose.

Keys and scope: do not pool answers that should not mix. The from-scratch demo has one global cache, which is fine when every user should get the same answer to "what is your refund policy?". It is a bug the moment answers are user-specific or tenant-specific. "What is my account balance?" embeds almost identically for two different users, so a global cache would serve one user's balance to another, a false hit that no threshold can prevent because the questions really are near-identical; only the answers differ. The fix is to scope the cache by a key: namespace entries by user ID, tenant, locale, or model version, and only ever search within the matching namespace. Scoping is a second line of defense alongside the threshold: the threshold guards against different questions colliding, and the key guards against identical questions that must not share an answer. A wrong answer is the failure mode of this whole technique, and these three controls (TTL, eviction, scope) are how you bound it beyond the threshold alone.

How this looks in production

You do not build the embedder, the vector store, and the similarity search by hand for a real system. Two representative tools:

GPTCache is an open-source semantic cache. Its architecture is the same pipeline as the demo, split into named stages: an adapter that wraps the LLM call, a pre-processor that normalizes the incoming request, an embedding generator that turns the query into a vector, a cache manager that stores vectors and answers, a similarity evaluator that scores a new query against the stored ones (cosine similarity with a threshold around $0.8$, the same decision the demo makes), and a post-processor that returns the chosen answer. It plugs into orchestration frameworks like LangChain and LlamaIndex, so an existing app can route calls through the cache without rewriting them.

Redis LangCache is a managed version of the same idea: embedding, storage, and similarity search behind a single API call, so you do not run the vector store yourself. The managed search adds a small amount of latency (on the order of a handful of milliseconds) to find a match, which is the cache-lookup cost in our model, and on a hit it saves the full model call, which is the slow, expensive part. The economics are the BEFORE/AFTER of the demo: trade a cheap lookup for an expensive generation, on the fraction of traffic that repeats.

Where these pay off is exactly where the duplicate rate is high: FAQ and support bots, repeated analytics questions ("what were sales last quarter?" asked fifty ways), and any service with heavy near-duplicate traffic. Where they do not pay off is open-ended or per-user-unique work, where almost nothing repeats and every lookup is a guaranteed miss that still costs you the embedding step. The decision to add a semantic cache is, at bottom, a bet on how repetitive your traffic is.

Using the real tool: commands and before/after proof

The from-scratch demo above is the engine. In an app you wire in a packaged version of it instead of hand-rolling the embedder and vector search. Here is what that looks like with the two tools from the last section, and how you prove the win.

GPTCache: wrap the LLM call

GPTCache is an open-source semantic cache that sits in front of your model call. Install it:

pip install gptcache

The library is not installed on this box and the snippets below need a model API key, so treat them as follow-along: the calls are the real, documented API, but the output shown is illustrative, not run here. This is the semantic-cache setup straight from the GPTCache README. It builds the same pipeline as our demo (an embedder, a vector store, a distance-based similarity check) and points GPTCache's OpenAI adapter at it:

from gptcache import cache
from gptcache.adapter import openai
from gptcache.embedding import Onnx
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation

onnx = Onnx()                                    # a real (small) sentence embedder
data_manager = get_data_manager(                 # where vectors and answers live
    CacheBase("sqlite"),
    VectorBase("faiss", dimension=onnx.dimension),
)
cache.init(
    embedding_func=onnx.to_embeddings,           # text -> vector (our toy embedder's job)
    data_manager=data_manager,                   # store + search (our cache dict's job)
    similarity_evaluation=SearchDistanceEvaluation(),  # the threshold decision
)
cache.set_openai_key()

SearchDistanceEvaluation is the threshold knob from the demo, now expressed as a distance (low distance means similar) rather than a similarity (high means similar). After cache.init, you call the model through openai from gptcache.adapter instead of the normal client. The adapter checks the cache first and only calls the model on a miss. You do not change your prompt; you change which client you call through. GPTCache also plugs into LangChain: you set it as the global LLM cache with LangChain's set_llm_cache(...), and every model call in the app routes through the cache without rewriting the call sites.

Redis LangCache / redisvl: a before/after you can time

Redis LangCache is the managed service; redisvl (the Redis Vector Library) is the open client that exposes the same SemanticCache you would run yourself against a Redis instance. It gives you a clean check/store pair, which makes the before/after easy to time. Install it:

pip install redisvl

Again follow-along (no Redis, no model key, no redisvl on this box; output is illustrative). The recipe: time two similar questions. The first is a cold MISS that pays for a full model call; the second is a HIT served from the cache with no model call.

import time
from redisvl.extensions.cache.llm import SemanticCache

cache = SemanticCache(
    name="llmcache",
    redis_url="redis://localhost:6379",
    distance_threshold=0.1,   # COSINE distance in [0, 2]; LOWER is stricter (0 = identical)
)

def answer(question):
    hit = cache.check(prompt=question)        # vector search against stored questions
    if hit:
        return hit[0]["response"], "HIT"      # served from cache, model NOT called
    reply = call_the_model(question)          # your real claude-opus-4-8 call goes here
    cache.store(prompt=question, response=reply)
    return reply, "MISS"

for q in ["How do I reset my password?",      # cold: nothing stored yet -> MISS
          "How can I reset my password?"]:     # paraphrase of the first -> HIT
    t0 = time.perf_counter()
    reply, status = answer(q)
    dt = (time.perf_counter() - t0) * 1000
    print(f"{status:4}  {dt:8.1f} ms  {q}")

Illustrative output (expected shape, not measured here):

MISS    1503.7 ms  How do I reset my password?
HIT       14.2 ms  How can I reset my password?

The MISS pays the full model call (about 1500 ms and a few hundred output tokens). The HIT is just the embed-and-search lookup (about 15 ms) and calls no model, so it costs no output tokens. That is roughly a 100x latency drop on the repeat and the entire avoided generation cost, which on the claude-opus-4-8 rates from Chapter 2 (output billed at 5x input) is where the dollars go.

The knob is distance_threshold. Note the inversion from our demo: redisvl uses cosine distance (lower is stricter, $0$ is identical), while the from-scratch demo used cosine similarity (higher is stricter, $1$ is identical). They are two faces of the same dial. Set it too loose (a high distance threshold here) and you get the false hit the demo proved on "cancel my subscription" versus "cancel my flight": a different question served the wrong stored answer. The on-box semantic_cache.py demo is the real, verified proof of that failure mode and of the BEFORE/AFTER arithmetic; the snippets here are the same logic packaged behind a tool.

In Claude Code / at the app layer

Semantic caching lives in your application, in front of the model call, and it does not care which provider you use: the embedder, the vector store, and the threshold are all yours. That makes it independent of, and complementary to, Anthropic's prompt caching from Chapter 6. Prompt caching is a server-side feature that reuses the input prefix tokens when the model does run; semantic caching is a client-side layer that skips the model entirely on a near-duplicate. Use both: prompt caching makes the calls you do make cheaper, and the semantic cache removes the calls you should not be making at all. The same client-side cache is one of the building blocks behind agent memory in Chapter 9, where "have I answered something like this before?" becomes "have I seen this state before?"

Further reading

  • GPTCache (the open-source semantic cache used above), code and docs at GitHub zilliztech/GPTCache, and the accompanying paper "GPTCache: An Open-Source Semantic Cache for LLM Applications" on arxiv.org. The clearest end-to-end reference for the adapter / embedder / cache-manager / similarity-evaluator pipeline.
  • Redis LangCache and redisvl (the managed service and the open Redis Vector Library), documented at redis.io. Read the SemanticCache reference for the distance_threshold knob, TTL, and scoping in a real vector store.
  • Sentence-BERT ("Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", Reimers and Gurevych) on arxiv.org, with models and how-to guides at sbert.net. This is the reference for why a trained sentence embedder beats a hashing bag-of-words, and how to run one.
  • Cosine similarity and approximate nearest-neighbor search: any vector-search primer covers the geometry and the index structures that make searching millions of stored vectors fast. The earlier books in this series, hnsw and ivf-pq, build those indexes from scratch.

Takeaways

  • Semantic caching returns a stored answer for an approximately similar question and skips the model entirely, saving the expensive output half of the call, not just the input prefix that Chapter 6 reuses.
  • The decision is three pieces: an embedding (text to vector), cosine similarity (the angle between two vectors, a plain dot product on unit vectors), and a threshold (the cutoff that turns a score into hit or miss).
  • The threshold is a dial between false misses (paraphrases that needlessly call the model) and false hits (different questions served the wrong answer). False hits are the failure mode prefix caching cannot have, so tune conservatively, around $0.8$ on a good embedder.
  • The dollar and latency saving scales with your duplicate rate: high for FAQ/support bots and repeated analytics, near zero for open-ended or per-user-unique work.
  • A better embedder widens the safe threshold range, raising the hit rate and lowering false hits at once. Tools like GPTCache and Redis LangCache package the embedding, storage, and similarity search so you wire in the cache instead of building it.

👉 We have skipped the model on a cache hit. The next chapter goes back inside the model for the calls you cannot skip: how the KV-cache is served at scale, so the generation you do pay for is as fast and cheap as the hardware allows.