The rubric framework, end to end

Chapters 27 and 28 built the pieces. This chapter assembles them into one operable framework and runs it on a complete worked example: a user's click history in, a repaired slate and a machine-readable verdict out. A rubric framework has six components, and a team that ships one needs all six:

1. SPEC        the rubric itself: versioned, reviewable, owned
2. LAYERS      deterministic text -> image hash -> LLM judge (cheap first)
3. ACTIONS     every verdict maps to a mechanical action (drop / keep / alert)
4. ARTIFACT    one JSON document per evaluated slate
5. SURFACES    the artifact rendered where decisions happen (CI, MLflow,
               dashboard, on-call, product review)
6. VALIDATION  golden set, threshold sweeps, judge-vs-human kappa,
               and a rule for what to re-validate when anything changes

Every term of art in this chapter (slate, card, residue, verdict, golden set, and the rest) is defined in the glossary.

1. The spec: a rubric you can review

The rubric is a document first and code second. It lives in the repo, has a version, an owner, and a changelog, and every threshold in it is a reviewable line in a pull request. A complete spec, in the YAML shape teams actually check in:

rubric: news-feed-rubric
version: 1.2.0
owner: feed-quality team
criteria:
  - id: relevance
    question: do the cards match the user's demonstrated taste?
    signal: mean cosine(EMA profile, card embedding)
    threshold: ">= 0.05"          # per corpus and embedder; from the golden set
    on_fail: block deploy (CI) / alert (production)
  - id: freshness
    question: is the feed stale?
    signal: median published age (hours)
    threshold: "<= 48"
    on_fail: alert feed-ingest on-call
  - id: duplicates
    question: are two cards the same story?
    signal: unresolved same-story pairs after judging
    threshold: "== 0"
    on_fail: block deploy; drop lower-ranked card at serve time
  - id: diversity
    question: is the slate one topic repeated?
    signal: top subcategory share
    threshold: "<= 0.70"          # per surface: push wants lower than homepage
    on_fail: warn; mix in exploration candidates
routing:                          # what reaches the judge
  text_jaccard: 0.5
  text_cosine: 0.8
  image_hamming: 10
judge:
  labels: [same_story, related_but_distinct, unrelated]
  actions: {same_story: drop lower-ranked, related_but_distinct: keep,
            unrelated: keep + recheck flagger}
  model: claude-opus-4-8          # pinned; changing it bumps the version
validation:
  golden_set: eval/golden_pairs.jsonl
  bars: {router_recall: 0.95, judge_kappa: 0.80}

Three governance rules make the spec real. Any change to a threshold, a prompt, or the judge model bumps the version, and the version is stamped on every artifact the framework emits (you saw news-feed-rubric@1.2.0 in the scorecards below), so a regression can always be traced to the rubric change that caused it. Every criterion has an owner and an on-fail action; a criterion nobody acts on is decoration. And the spec keeps a blind-spot register: known misses (this rubric cannot see case E paraphrases without an embedding layer, and cannot judge headline-image coherence without a vision judge) written down, so nobody mistakes a green scorecard for omniscience.

2. The worked example: input

One user, five clicks, with recency (the EMA half-life is 48 hours, so the finance click from yesterday still carries real weight):

Ten candidate articles, seeded with every duplicate case from Chapter 27 (headline, subcategory, age, thumbnail):

3. Running the framework

scripts/rubric_framework.py is the whole pipeline in one self-contained file: embeddings, profile, slate, all three layers, repair, and the artifact. It imports nothing but NumPy:

#!/usr/bin/env python3
"""The rubric framework in one file: input -> slate -> layers -> repair -> artifact.

Everything from chapters 27 and 28, wired end to end on a self-contained
example (no capstone imports, NumPy only):

  input      a user's click history (what they read, when) + a candidate
             catalog of (headline, subcategory, published age, thumbnail)
  profile    the EMA taste vector (half-life decay over history embeddings)
  slate      top-k candidates by cosine to the profile
  layer 1    deterministic text screen (Jaccard / embedding cosine)
  layer 2    image screen (dHash Hamming over thumbnails)
  layer 3    judge on the routed residue (offline entity heuristic here;
             swap in the Claude judge from scripts/judge_pairs.py)
  repair     drop same_story losers, backfill, re-score
  artifact   one JSON document a CI gate, dashboard, or reviewer can read

Run:  python scripts/rubric_framework.py
"""
from __future__ import annotations

import itertools
import json

import numpy as np

RUBRIC = {
    "version": "news-feed-rubric@1.2.0",
    "criteria": {
        "relevance": {"signal": "mean cosine(profile, card)", "min": 0.05},
        "freshness": {"signal": "median published age (h)", "max": 48.0},
        "duplicates": {"signal": "unresolved same-story pairs", "max": 0},
        "diversity": {"signal": "top subcategory share", "max": 0.70},
    },
    "routing": {"text_jaccard": 0.5, "text_cosine": 0.8, "image_hamming": 10},
}
HALF_LIFE_H, K = 48.0, 8

HISTORY = [  # (hours ago, subcategory, headline the user clicked)
    (72, "soccer", "Bayern Munich cruise past Dortmund in derby"),
    (48, "soccer", "Real Madrid edge Sevilla in La Liga"),
    (36, "soccer", "Champions League draw sets up blockbuster ties"),
    (24, "finance", "Fed signals more rate hikes ahead"),
    (12, "soccer", "Bayern Munich name new captain"),
]

CATALOG = [  # (id, subcategory, hours since published, thumbnail, headline)
    ("S1", "soccer", 5, "pitch", "Bayern Munich stun Real Madrid with late winner"),
    ("S3", "soccer", 4, "pitch-cdn", "Bayern Munich stun Real Madrid with late winner"),
    ("S2", "soccer", 6, "arena", "Late goal sees Bayern Munich stun Real Madrid"),
    ("S4", "soccer", 8, "arena2", "Arsenal stun Real Madrid with late winner"),
    ("S5", "soccer", 30, "pitch-crop", "Champions League briefing: what to watch this week"),
    ("X1", "soccer", 15, "generic", "Injury update: Bayern Munich midfielder out six weeks"),
    ("F1", "finance", 10, "chart", "Fed raises interest rates for third time this year"),
    ("F2", "finance", 12, "chart2", "Borrowing costs climb as central bank hikes again"),
    ("T1", "tech", 20, "generic2", "Chipmaker unveils faster laptop processor"),
    ("M1", "movies", 26, "generic3", "Box office: heist thriller tops weekend charts"),
]

STOP = {"the", "and", "after", "in", "out", "with", "late", "what", "this"}


# ----------------------------------------------------------- text signals
def tokens(t):
    return [w for w in t.lower().split() if w.isalpha()]


def tfidf(texts):
    vocab = sorted({w for t in texts for w in tokens(t)})
    idx = {w: i for i, w in enumerate(vocab)}
    df = np.zeros(len(vocab))
    for t in texts:
        for w in set(tokens(t)):
            df[idx[w]] += 1
    idf = np.log((1 + len(texts)) / (1 + df)) + 1
    E = np.zeros((len(texts), len(vocab)))
    for r, t in enumerate(texts):
        for w in tokens(t):
            E[r, idx[w]] += idf[idx[w]]
        E[r] /= max(np.linalg.norm(E[r]), 1e-9)
    return E


def jaccard(t1, t2):
    a, b = set(tokens(t1)), set(tokens(t2))
    return len(a & b) / max(len(a | b), 1)


def entities(t):
    return {w for w in t.split() if w[:1].isupper() and w.lower() not in STOP}


# ---------------------------------------------------------- image signals
def thumb(kind, rng):
    y, x = np.mgrid[0:64, 0:64]
    if kind.startswith("pitch"):
        img = 60 + 90 * (y / 64) + 25 * ((x // 8) % 2)
        img[(x - 44) ** 2 + (y - 20) ** 2 < 36] = 230
        if kind == "pitch-cdn":                       # same photo, CDN copy
            img = img * 0.93 + 12
        if kind == "pitch-crop":                      # same photo, cropped
            inner = img[5:-5, 5:-5]
            i = (np.arange(64) * inner.shape[0] / 64).astype(int)
            img = inner[np.ix_(i, i)]
    elif kind.startswith("arena"):
        img = 200 - 120 * ((x + y) / 128)
        img[10:26, 20:44] = 35 if kind == "arena" else 90
    elif kind.startswith("chart"):
        img = 230 - 40 * (y / 64)
        img[(y - (60 - x // 2)) ** 2 < 9] = 20 if kind == "chart" else 120
    else:
        img = rng.uniform(0, 255, (64, 64))
    return np.clip(img + rng.normal(0, 2, (64, 64)), 0, 255)


def dhash(img):
    e = np.linspace(0, 64, 9).astype(int)
    xe = np.linspace(0, 64, 10).astype(int)
    small = np.array([[img[e[i]:e[i + 1], xe[j]:xe[j + 1]].mean()
                       for j in range(9)] for i in range(8)])
    return (small[:, 1:] > small[:, :-1]).flatten()


# -------------------------------------------------------------- the judge
def judge(t1, t2, sub1, sub2):
    """Offline stand-in; production swaps in the Claude judge (ch. 27)."""
    e1, e2 = entities(t1), entities(t2)
    overlap = len(e1 & e2) / max(len(e1 | e2), 1)
    if overlap >= 0.8:
        return "same_story", f"entity overlap {overlap:.2f}"
    if overlap > 0 or sub1 == sub2:
        return "related_but_distinct", f"entity overlap {overlap:.2f}"
    return "unrelated", "no shared entities"


# ----------------------------------------------------------- the scorecard
def scorecard(slate, cos, unresolved):
    ages = sorted(a["age"] for a in slate)
    subs = [a["sub"] for a in slate]
    c = RUBRIC["criteria"]
    vals = {
        "relevance": float(np.mean(cos)),
        "freshness": float(np.median(ages)),
        "duplicates": unresolved,
        "diversity": max(subs.count(s) for s in set(subs)) / len(subs),
    }
    passes = {
        "relevance": vals["relevance"] >= c["relevance"]["min"],
        "freshness": vals["freshness"] <= c["freshness"]["max"],
        "duplicates": vals["duplicates"] <= c["duplicates"]["max"],
        "diversity": vals["diversity"] <= c["diversity"]["max"],
    }
    return vals, passes


def show(tag, vals, passes):
    print(f"scorecard [{tag}]  ({RUBRIC['version']})")
    for k, v in vals.items():
        mark = "PASS" if passes[k] else "FAIL"
        print(f"  {k:<11} {v:>6.2f}   {mark}")


def main():
    rng = np.random.default_rng(0)
    arts = [{"id": i, "sub": s, "age": h, "title": t,
             "hash": dhash(thumb(kind, rng))}
            for i, s, h, kind, t in CATALOG]

    # profile: EMA over history embeddings, in the catalog's TF-IDF space
    E = tfidf([t for *_, t in HISTORY] + [a["title"] for a in arts])
    hist_E, art_E = E[:len(HISTORY)], E[len(HISTORY):]
    lam = np.log(2) / HALF_LIFE_H
    w = np.array([np.exp(-lam * h) for h, *_ in HISTORY])
    profile = (w[:, None] * hist_E).sum(0) / w.sum()
    profile /= np.linalg.norm(profile)

    scores = art_E @ profile
    order = np.argsort(-scores)
    slate = [arts[i] | {"cos": float(scores[i])} for i in order[:K]]
    bench = [arts[i] | {"cos": float(scores[i])} for i in order[K:]]

    print(f"history: {len(HISTORY)} clicks (4 soccer, 1 finance), "
          f"half-life {HALF_LIFE_H:.0f} h")
    print(f"\nslate before repair (top {K} of {len(arts)} candidates):")
    for r, a in enumerate(slate, 1):
        print(f"  {r}. {a['cos']:.3f} [{a['sub']:<7}] {a['id']}  {a['title']}")

    # layers 1+2: route every slate pair through text and image screens
    R = RUBRIC["routing"]
    dropped, verdicts = set(), []
    pairs = list(itertools.combinations(range(len(slate)), 2))
    for i, j in pairs:
        a, b = slate[i], slate[j]
        if a["id"] in dropped or b["id"] in dropped:
            continue
        jac = jaccard(a["title"], b["title"])
        cos = float(art_E[[x["id"] for x in arts].index(a["id"])]
                    @ art_E[[x["id"] for x in arts].index(b["id"])])
        ham = int((a["hash"] != b["hash"]).sum())
        if jac == 1.0 and ham <= 3:                       # case A: no judge
            dropped.add(b["id"])
            verdicts.append((a["id"], b["id"], "exact_duplicate", "drop"))
            continue
        if jac >= R["text_jaccard"] or cos >= R["text_cosine"] \
                or ham <= R["image_hamming"]:             # layer 3: judge
            v, why = judge(a["title"], b["title"], a["sub"], b["sub"])
            act = "drop" if v == "same_story" else "keep"
            if v == "same_story":
                dropped.add(b["id"])
            verdicts.append((a["id"], b["id"], f"{v} ({why})", act))

    print("\nrouted pairs and verdicts:")
    for a, b, v, act in verdicts:
        print(f"  {a} vs {b}: {v:<38} -> {act} {b if act == 'drop' else 'both'}")

    # repair: remove drops, backfill from the bench by score
    repaired = [a for a in slate if a["id"] not in dropped]
    fill = bench[:K - len(repaired)]
    repaired += fill
    print(f"\nrepair: dropped {sorted(dropped)}, "
          f"backfilled {[a['id'] for a in fill]}")

    before = scorecard(slate, [a["cos"] for a in slate],
                       sum(1 for *_, act in verdicts if act == "drop"))
    after = scorecard(repaired, [a["cos"] for a in repaired], 0)
    print()
    show("before", *before)
    show("after ", *after)

    artifact = {
        "rubric": RUBRIC["version"],
        "user": "U-demo",
        "slate": [a["id"] for a in slate],
        "final_slate": [a["id"] for a in repaired],
        "criteria": {k: {"value": round(before[0][k], 3),
                         "after_repair": round(after[0][k], 3),
                         "pass": bool(after[1][k])} for k in before[0]},
        "verdicts": [{"pair": [a, b], "verdict": v.split(" (")[0],
                      "action": act} for a, b, v, act in verdicts],
    }
    print("\nartifact (what CI, MLflow, and the dashboard consume):")
    print(json.dumps(artifact, indent=1))


if __name__ == "__main__":
    main()

$ python scripts/rubric_framework.py
history: 5 clicks (4 soccer, 1 finance), half-life 48 h

slate before repair (top 8 of 10 candidates):
  1. 0.212 [soccer ] S1  Bayern Munich stun Real Madrid with late winner
  2. 0.212 [soccer ] S3  Bayern Munich stun Real Madrid with late winner
  3. 0.192 [soccer ] S2  Late goal sees Bayern Munich stun Real Madrid
  4. 0.120 [soccer ] X1  Injury update: Bayern Munich midfielder out six weeks
  5. 0.097 [soccer ] S5  Champions League briefing: what to watch this week
  6. 0.061 [soccer ] S4  Arsenal stun Real Madrid with late winner
  7. 0.057 [finance] F2  Borrowing costs climb as central bank hikes again
  8. 0.054 [finance] F1  Fed raises interest rates for third time this year

routed pairs and verdicts:
  S1 vs S3: exact_duplicate                        -> drop S3
  S1 vs S2: same_story (entity overlap 1.00)       -> drop S2
  S1 vs S5: related_but_distinct (entity overlap 0.00) -> keep both
  S1 vs S4: related_but_distinct (entity overlap 0.40) -> keep both

repair: dropped ['S2', 'S3'], backfilled ['T1', 'M1']

scorecard [before]  (news-feed-rubric@1.2.0)
  relevance     0.13   PASS
  freshness     9.00   PASS
  duplicates    2.00   FAIL
  diversity     0.75   FAIL
scorecard [after ]  (news-feed-rubric@1.2.0)
  relevance     0.08   PASS
  freshness    13.50   PASS
  duplicates    0.00   PASS
  diversity     0.50   PASS

Read it layer by layer, because every case from the matrix shows up:

• The vector search did exactly what Chapter 27 predicted. Nearest neighbors of one profile look like each other: three copies of the Bayern story fill the top three slots, and the slate is 75% soccer.
• Case A (S1 vs S3): identical text and a Hamming-0 thumbnail. Decided by the deterministic layer alone, no judge call.
• Case B (S1 vs S2): text flag, judge says same_story, S2 dropped.
• Case C (S1 vs S5): invisible to text (Jaccard 0), caught by the image hash on the cropped photo, judged related_but_distinct, kept. The photo reuse is now a logged fact rather than a mystery.
• Case D (S1 vs S4): the template lookalike is flagged by text and saved by the judge. Two different matches stay in the feed.
• Case E (F1 vs F2): silently missed, exactly as the blind-spot register says. The validation harness below puts a number on that miss.
• The repair has a visible cost. Relevance drops from 0.126 to 0.075: the backfilled tech and movies cards are off-taste. Dedupe and diversity are bought with taste alignment, and because both numbers are in the artifact, that tradeoff is a product decision made with open eyes instead of a silent side effect.

4. Evaluation vs. enforcement: when the rubric runs

A fair reading of Chapter 27 is "the rubric is a pre-deploy check", and a fair reading of Chapter 28 is "the rubric is pipeline code". Both are right, and keeping the two modes separate is the clarification that makes the framework legible. One spec, two modes, four moments:

                    offline (no user sees the output)     online (users)
dev time            validate the judge against the        -
                    golden set; kappa bar gates the
                    judge itself
pre-deploy (CI)     run the rubric over a FIXED sample     -
                    of users on the candidate model;
                    LLM slate review on the worst
                    slates; gate the deploy on the
                    scorecard diff vs. the base model
runtime: ingest     judge NEW story pairs once,            -
                    asynchronously; cache every verdict
runtime: serve      -                                      enforcement only:
                                                           cached verdicts,
                                                           collapse, residue
                                                           screen. Zero LLM
                                                           calls, ever.

Evaluation mode is the rubric as a gate: before a model, prompt, or threshold change ships, the framework scores slates nobody will see and the scorecard decides the deploy. This is what "the rubric runs before deploying" means, and it is the mode the CI column of every table in this chapter refers to.

Enforcement mode is the same criteria compiled into the pipeline: the dedupe actions, the collapse key, the residue screen. It runs on live traffic, which is why it is restricted to precomputed facts and cheap math.

Where the LLM sits across the four moments: it is the subject of validation at dev time, a reviewer at pre-deploy, and a pair judge at ingest. It never runs at serve time. The one subtlety is CI: evaluating a new candidate model produces new slates, which can surface story pairs no one has judged yet, so a CI run may trigger judge calls. That is fine by construction: the evaluation sample is fixed, so the cost is bounded, and the verdicts land in the same cache as ingest verdicts.

5. The LLM review: the prompts

Chapter 27 used a one-sentence system prompt to keep the lab readable. Production prompts are longer for one reason: the case matrix moves into the prompt as numbered decision rules, so the judge does not have to rediscover the traps on every call. Every prompt in the framework has the same four-part anatomy:

1. Role and scope: what the judge is, and what it must not grade (the slate reviewer is explicitly told the deterministic criteria are measured elsewhere).
2. Labels defined extensionally: each label with concrete examples of what belongs in it, not just a name.
3. Ordered decision rules: rule 1 is the case D trap (same template, different event), rule 2 is case E (paraphrase, same event), rule 4 is the conflicting-fact-slot test the judge discovered on its own in Chapter 27, and rule 5 is the safe default, aligned with the pipeline's fail-open action (related_but_distinct means keep both).
4. Schema-only output: the structured-output schema is the response format; no prose to parse.

The prompts live in one module, versioned and content-hashed, because a prompt edit changes judge behavior exactly like a model swap does. The hash is stamped into every verdict, and a version bump re-triggers golden-set validation before the new prompt is allowed to judge:

#!/usr/bin/env python3
"""The prompt registry: every LLM prompt in the rubric framework, versioned.

Prompts are code. They live in one module, each with a semantic version and
a content hash; the hash is stamped into every verdict and every validation
run, so "which prompt produced this?" always has an answer. Changing a
prompt bumps its version, which re-triggers golden-set validation (kappa)
before the new version is allowed to judge anything.

Three prompts cover the framework's LLM work:

  pair_judge        is headline pair (A, B) the same story?      ingest + CI
  slate_review      judgment-only audit of one user's slate      pre-deploy
  image_coherence   does the thumbnail depict the headline?      ingest (vision)

Run:  python scripts/judge_prompts.py     (prints the registry + one render)
"""
from __future__ import annotations

import hashlib

PAIR_JUDGE_SYSTEM = """\
You are a duplicate detector for a news recommendation feed. You will be
given two headlines, each with an optional abstract. Decide whether they
report the SAME real-world event.

Labels:
- same_story: both describe one event: same actors, same outcome, same
  moment in time. Rewordings, reorderings, and two outlets covering one
  event are same_story.
- related_but_distinct: same topic, template, or ongoing storyline, but a
  different event (a different match, a different winner, a follow-up
  development on a later day).
- unrelated: different topics entirely.

Decision rules, in order:
1. Different instances of a recurring event type are related_but_distinct,
   even when the wording is nearly identical. Two match reports about two
   different games are NOT the same story.
2. A paraphrase with no shared words is still same_story when the
   underlying event matches ("Fed raises rates" / "Borrowing costs climb
   after central bank decision").
3. A follow-up or reaction piece is related_but_distinct from the original
   event, even about the same actors.
4. If the two headlines disagree on the same fact slot (two different
   players winning one award), they cannot be the same event.
5. When genuinely uncertain, answer related_but_distinct: in this feed,
   keeping both cards is the safe failure.

Return only the structured verdict."""

PAIR_JUDGE_USER = """\
Headline 1: {t1}
Abstract 1: {a1}

Headline 2: {t2}
Abstract 2: {a2}"""

PAIR_JUDGE_SCHEMA = {
    "verdict": ["same_story", "related_but_distinct", "unrelated"],
    "reason": "one sentence citing the decisive rule",
}

SLATE_REVIEW_SYSTEM = """\
You audit one news slate for one user before a model ships. You are given
the user's recent clicks and the slate cards in rank order. Grade ONLY the
criteria below. Deterministic checks (freshness, duplicate counts, category
shares) are measured elsewhere; do not re-grade them.

Criteria:
- taste_coherence: given the clicks, would this user plausibly want each
  card? Flag cards with no visible connection to any click.
- clickbait: flag headlines that withhold the core fact ("You won't
  believe..."), manufacture urgency, or promise more than the abstract
  supports.
- headline_quality: flag truncation artifacts, all-caps, garbled text, or
  encoding damage.
- slate_story: read top to bottom; does the ordering make sense for this
  reader (strongest match first, no jarring jumps)?

For each criterion return a grade (pass | warn | fail), the offending card
ids if any, and a one-line reason. When uncertain, grade warn, never fail."""

SLATE_REVIEW_USER = """\
User clicks (most recent last):
{history}

Slate (rank order):
{slate}"""

SLATE_REVIEW_SCHEMA = {
    "criteria": {
        "<name>": {"grade": ["pass", "warn", "fail"],
                   "cards": ["ids"], "reason": "one line"},
    },
}

IMAGE_COHERENCE_SYSTEM = """\
You check whether a news thumbnail plausibly belongs to its headline. You
will be given one image and one headline.

Labels:
- depicts_story: the image shows the event, the named people, or the named
  place.
- generic_stock: the image is topical but interchangeable (a generic
  stadium on a match report, a generic trading floor on a markets story).
  This is acceptable; it is not a mismatch.
- mismatched: the image contradicts the headline (wrong sport, wrong
  person, unrelated subject) or is unrelated filler.

Rule: generic is not mismatched. Only answer mismatched when a reader would
feel misled. Return only the structured verdict."""

IMAGE_COHERENCE_USER = "Headline: {t1}"  # the image rides as an image block

IMAGE_COHERENCE_SCHEMA = {
    "verdict": ["depicts_story", "generic_stock", "mismatched"],
    "reason": "one line",
}

REGISTRY = {
    "pair_judge": {
        "version": "pair-judge@3.0.0",
        "system": PAIR_JUDGE_SYSTEM, "user": PAIR_JUDGE_USER,
        "schema": PAIR_JUDGE_SCHEMA, "runs_at": "ingest + CI",
    },
    "slate_review": {
        "version": "slate-review@1.1.0",
        "system": SLATE_REVIEW_SYSTEM, "user": SLATE_REVIEW_USER,
        "schema": SLATE_REVIEW_SCHEMA, "runs_at": "pre-deploy",
    },
    "image_coherence": {
        "version": "image-coherence@2.0.0",
        "system": IMAGE_COHERENCE_SYSTEM, "user": IMAGE_COHERENCE_USER,
        "schema": IMAGE_COHERENCE_SCHEMA, "runs_at": "ingest (vision)",
    },
}


def prompt_hash(name: str) -> str:
    p = REGISTRY[name]
    return hashlib.sha256((p["system"] + p["user"]).encode()).hexdigest()[:12]


def render(name: str, **kw) -> tuple[str, str]:
    p = REGISTRY[name]
    return p["system"], p["user"].format(**kw)


def main():
    print("prompt registry (hash goes into every verdict and artifact):\n")
    print("name             version                hash          runs at")
    print("-" * 68)
    for name, p in REGISTRY.items():
        print(f"{name:<16} {p['version']:<22} {prompt_hash(name)}  {p['runs_at']}")

    print("\nrendered example: pair_judge on the case B pair from ch. 29")
    _, user = render(
        "pair_judge",
        t1="Bayern Munich stun Real Madrid with late winner", a1="(none)",
        t2="Late goal sees Bayern Munich stun Real Madrid", a2="(none)")
    print("-" * 68)
    print(user)
    print("-" * 68)
    print("expected verdict under rule 1 vs rule 4: same_story")
    print("(same match, same outcome; wording differs, facts do not)")


if __name__ == "__main__":
    main()

$ python scripts/judge_prompts.py
prompt registry (hash goes into every verdict and artifact):

name             version                hash          runs at
--------------------------------------------------------------------
pair_judge       pair-judge@3.0.0       cd829f052a5c  ingest + CI
slate_review     slate-review@1.1.0     6419d8a5f266  pre-deploy
image_coherence  image-coherence@2.0.0  8dbc97aca855  ingest (vision)

rendered example: pair_judge on the case B pair from ch. 29
--------------------------------------------------------------------
Headline 1: Bayern Munich stun Real Madrid with late winner
Abstract 1: (none)

Headline 2: Late goal sees Bayern Munich stun Real Madrid
Abstract 2: (none)
--------------------------------------------------------------------
expected verdict under rule 1 vs rule 4: same_story
(same match, same outcome; wording differs, facts do not)

Validating the LLM judge with these prompts

The validation harness from section 8 takes any judge that maps a pair to a binary label. Plugging in the Claude judge with the registry prompt is a ten-line adapter (follow-along; needs anthropic and an API key):

from judge_prompts import render, prompt_hash
from pydantic import BaseModel
from typing import Literal

class PairVerdict(BaseModel):
    verdict: Literal["same_story", "related_but_distinct", "unrelated"]
    reason: str

def judge_claude_binary(client, t1, t2):
    system, user = render("pair_judge", t1=t1, a1="(none)", t2=t2, a2="(none)")
    resp = client.messages.parse(
        model="claude-opus-4-8", max_tokens=1024, system=system,
        messages=[{"role": "user", "content": user}],
        output_format=PairVerdict,
    )
    return 1 if resp.parsed_output.verdict == "same_story" else 0

preds = [judge_claude_binary(client, a, b) for a, b, _ in GOLDEN]
po, k = kappa(preds, labels)      # same harness, same 0.8 bar

Output with a key set (illustrative; verdicts can flip on genuinely hard pairs across runs):

2) claude pair-judge (pair-judge@3.0.0, cd829f052a5c) vs human labels
   confusion: TP=6 FP=0 FN=0 TN=10
   accuracy = 1.000   Cohen's kappa = 1.000   PASS (bar: kappa >= 0.8)

The two case E paraphrases and the Champions League topic trap, the exact pairs the entity heuristic failed on, are what a reading judge gets right. Expect high kappa rather than always-perfect: a pair that flips across runs goes to the disagreement pile and gets the majority-vote treatment from Chapter 27's best-practices list.

Three coding notes on the calls themselves. claude-opus-4-8 exposes no sampling knobs (temperature and friends are removed from the API), so run-to-run consistency is engineered where it actually lives: decision rules in the prompt, a forced schema, and votes on flip-flops. The system prompt is byte-identical across every call by construction (it comes from the registry), which is exactly the shape prompt caching rewards. And the same request shape submits unchanged through the Message Batches API for the nightly queue; the adapter above is the only integration code there is.

The slate review: LLM as pre-deploy reviewer

The pair judge answers one narrow question. The slate_review prompt is the second kind of LLM work: a judgment-only audit of a whole slate against the criteria no formula can score (taste coherence, clickbait, headline quality, ordering), run at pre-deploy time on sampled or worst-scoring slates. It is told what not to grade, so it never duplicates the deterministic scorecard. Response for the worked example's repaired slate (illustrative):

{"criteria": {
  "taste_coherence": {"grade": "warn", "cards": ["M1"],
    "reason": "No click history suggests movie interest; M1 reads as exploration, not taste."},
  "clickbait": {"grade": "pass", "cards": [], "reason": "Every headline states its core fact."},
  "headline_quality": {"grade": "pass", "cards": [], "reason": "No truncation, casing, or encoding damage."},
  "slate_story": {"grade": "pass", "cards": [],
    "reason": "Soccer leads for a soccer-dominant history; finance follows; backfill sits last."}
}}

The numeric scorecard gates; the review explains. The warn on M1 is the relevance tradeoff from section 3, rediscovered independently by a reader, which is precisely the kind of confirmation that makes a pre-deploy report trustworthy.

Tooling for the LLM evaluation work

The harness in this book is deliberately plain Python, and for the core loop (golden set in the repo, kappa in CI) plain Python is the right answer. Around it, a small toolbox earns its place:

And the question that comes up first in practice: should the judge be a prompt or a Skill? A Skill (an SKILL.md package an agent loads on demand, with progressive disclosure and tool access) is built for multi-step agentic work. The production judge is the opposite: a one-shot, schema-forced classification that must behave like a pure function. Three rules settle it:

1. Production judging: a plain versioned prompt through messages.parse. No agent loop, no tools, no skill; anything that decides at runtime what context to load adds variance exactly where you are trying to engineer it out.
2. Authoring time is where a Skill shines. A rubric-review skill in Claude Code that loads the spec, the prompt registry, and the golden set lets you tighten decision rules, draft new golden pairs (human confirmation stays mandatory; the labels are the ground truth), and run the harness conversationally. The skill wraps the workflow around the judge, never the judge itself.
3. The rubric document doubles as the judge's spec, not its prompt. Keep the YAML spec and the prompt registry separate but versioned together: the spec says what the criteria are; the prompt encodes how one criterion is decided. Conflating them couples every product-policy edit to a judge re-validation.

6. The artifact, and how to present it

Everything above collapses into one JSON document per evaluated slate (the tail of the run; elided in the middle):

artifact (what CI, MLflow, and the dashboard consume):
{
 "rubric": "news-feed-rubric@1.2.0",
 "user": "U-demo",
 "slate": ["S1", "S3", "S2", "X1", "S5", "S4", "F2", "F1"],
 "final_slate": ["S1", "X1", "S5", "S4", "F2", "F1", "T1", "M1"],
 "criteria": {
  "relevance":  {"value": 0.126, "after_repair": 0.075, "pass": true},
  "freshness":  {"value": 9.0,   "after_repair": 13.5,  "pass": true},
  "duplicates": {"value": 2,     "after_repair": 0,     "pass": true},
  "diversity":  {"value": 0.75,  "after_repair": 0.5,   "pass": true}
 },
 "verdicts": [
  {"pair": ["S1", "S3"], "verdict": "exact_duplicate", "action": "drop"},
  ...
 ]
}

(The lab prints the ids one per line; the shape is what matters.) The same artifact renders on five surfaces, and the framework is not "presented" until all five exist:

7. One rubric, every stage

The same criteria run at every stage of the stack; only the tool changes. This table is the whole integration story of chapters 27 and 28 in one place:

The strategy in one sentence: decide facts at ingest (story ids, hashes, judge verdicts), enforce them at query (filters, collapse), verify cheaply at serve (the residue screen), spend money only at night (batched judging), and compare against the spec at every deploy (CI).

8. Validation: proving the rubric can be trusted

A rubric that has not been validated is an opinion with a version number. The framework validates three things separately, because they fail separately: the router (does the cheap layer flag what it should?), the judge (does it agree with humans?), and the thresholds (were they chosen against evidence?). All three run against one golden set: 100 to 200 pairs labeled by hand, stratified to include every case in the matrix, refreshed quarterly, and stored in the repo next to the spec.

Two metrics do the work. For the router, precision and recall on the duplicate class; the router is tuned for recall, because a missed duplicate is gone forever while a false flag merely costs one judge call. For the judge, agreement with humans corrected for chance, Cohen's kappa:

$$ \kappa = \frac{p_o - p_e}{1 - p_e}, $$

where $p_o$ is observed agreement and $p_e$ is the agreement two labelers would reach by luck given their label frequencies. The correction is not optional, and the harness shows why. scripts/rubric_validation.py runs the full suite on a 16-pair golden set:

#!/usr/bin/env python3
"""Validating the rubric: golden set, threshold sweep, judge agreement, kappa.

A rubric is only as trustworthy as its validation. This harness runs the
three checks every layer needs, on a hand-labeled golden set of 16 headline
pairs (6 duplicates, 10 non-duplicates, with deliberate case B reorders,
case D template lookalikes, and case E paraphrases):

  1. ROUTER SWEEP   the deterministic flagger's precision/recall on the dup
                    class across Jaccard thresholds. The router is tuned for
                    recall; precision is the judge's job.
  2. JUDGE VS HUMAN confusion matrix, accuracy, and Cohen's kappa for the
                    judge (the offline entity heuristic here; swap in the
                    Claude judge and rerun the same harness).
  3. BASELINE       an always-not-dup judge, to show why kappa is the bar
                    and accuracy is not: on an imbalanced set, doing nothing
                    scores 62% accuracy and kappa 0.

Acceptance bars (from the implementation plan): router recall >= 0.95 on
the dup class at the chosen threshold; judge kappa >= 0.8 against humans.

Run:  python scripts/rubric_validation.py
"""
from __future__ import annotations

STOP = {"the", "and", "after", "in", "out", "with", "late", "what", "this"}

# (headline 1, headline 2, human label: 1 = same story, 0 = not)
GOLDEN = [
    # duplicates: case A, three case B rewords, two case E paraphrases
    ("Bayern Munich stun Real Madrid with late winner",
     "Bayern Munich stun Real Madrid with late winner", 1),
    ("Bayern Munich stun Real Madrid with late winner",
     "Late goal sees Bayern Munich stun Real Madrid", 1),
    ("Fed raises interest rates for third time this year",
     "Fed hikes interest rates for a third time this year", 1),
    ("Nvidia unveils faster laptop processor",
     "Nvidia unveils record-breaking laptop chip", 1),
    ("Fed raises interest rates for third time this year",
     "Borrowing costs climb as central bank hikes again", 1),         # E
    ("Storm knocks out power across the coast",
     "Thousands left in the dark after severe weather", 1),           # E
    # non-duplicates: case D lookalikes, topic-mates, unrelated
    ("Bayern Munich stun Real Madrid with late winner",
     "Arsenal stun Real Madrid with late winner", 0),                 # D
    ("Lakers edge Celtics 102-99 in overtime thriller",
     "Warriors edge Suns 118-115 in overtime thriller", 0),           # D
    ("Bukayo Saka wins Ballon d'Or after stellar football season",
     "Kylian Mbappe wins Ballon d'Or after stellar football season", 0),
    ("Manchester City and Real Madrid play out thrilling 1-1 draw",
     "Manchester City and Bayern Munich play out thrilling 3-3 draw", 0),
    ("Champions League draw sets up blockbuster ties",
     "Champions League briefing: what to watch this week", 0),
    ("Bayern Munich name new captain",
     "Bayern Munich stun Real Madrid with late winner", 0),
    ("Box office: heist thriller tops weekend charts",
     "Fed raises interest rates for third time this year", 0),
    ("Injury update: Bayern Munich midfielder out six weeks",
     "Chipmaker unveils faster laptop processor", 0),
    ("Storm knocks out power across the coast",
     "Borrowing costs climb as central bank hikes again", 0),
    ("Real Madrid edge Sevilla in La Liga",
     "Box office: heist thriller tops weekend charts", 0),
]


def jaccard(t1, t2):
    a = {w for w in t1.lower().split() if w.isalpha()}
    b = {w for w in t2.lower().split() if w.isalpha()}
    return len(a & b) / max(len(a | b), 1)


def judge_entity(t1, t2):
    """The offline judge: same_story iff entity overlap >= 0.8."""
    e1 = {w for w in t1.split() if w[:1].isupper() and w.lower() not in STOP}
    e2 = {w for w in t2.split() if w[:1].isupper() and w.lower() not in STOP}
    return 1 if len(e1 & e2) / max(len(e1 | e2), 1) >= 0.8 else 0


def prf(preds, labels):
    tp = sum(p and l for p, l in zip(preds, labels))
    fp = sum(p and not l for p, l in zip(preds, labels))
    fn = sum(not p and l for p, l in zip(preds, labels))
    prec = tp / max(tp + fp, 1)
    rec = tp / max(tp + fn, 1)
    return prec, rec


def kappa(preds, labels):
    n = len(labels)
    po = sum(p == l for p, l in zip(preds, labels)) / n
    p_yes = sum(preds) / n * sum(labels) / n
    p_no = (1 - sum(preds) / n) * (1 - sum(labels) / n)
    pe = p_yes + p_no
    return po, (po - pe) / (1 - pe) if pe < 1 else 1.0


def main():
    labels = [l for *_, l in GOLDEN]
    n_dup = sum(labels)
    print(f"golden set: {len(GOLDEN)} pairs ({n_dup} dup, "
          f"{len(GOLDEN) - n_dup} not)\n")

    print("1) router sweep (flag if jaccard >= t); tune for RECALL")
    print("   t     flagged  precision  recall")
    for t in (0.3, 0.4, 0.5, 0.6, 0.7):
        preds = [1 if jaccard(a, b) >= t else 0 for a, b, _ in GOLDEN]
        prec, rec = prf(preds, labels)
        print(f"  {t:.1f}   {sum(preds):5d}     {prec:.2f}      {rec:.2f}")
    print("  ceiling: recall never reaches 1.0; the two case E paraphrases")
    print("  share zero tokens. That gap is the embedding layer's job.\n")

    for step, (name, preds) in enumerate([
        ("entity-heuristic judge", [judge_entity(a, b) for a, b, _ in GOLDEN]),
        ("always-not-dup baseline", [0] * len(GOLDEN)),
    ], start=2):
        tp = sum(p and l for p, l in zip(preds, labels))
        fp = sum(p and not l for p, l in zip(preds, labels))
        fn = sum(not p and l for p, l in zip(preds, labels))
        tn = len(labels) - tp - fp - fn
        po, k = kappa(preds, labels)
        print(f"{step}) {name} vs human labels")
        print(f"   confusion: TP={tp} FP={fp} FN={fn} TN={tn}")
        print(f"   accuracy = {po:.3f}   Cohen's kappa = {k:.3f}"
              f"   {'PASS' if k >= 0.8 else 'FAIL'} (bar: kappa >= 0.8)")
        if fp:
            miss = [(a, b) for (a, b, l), p in zip(GOLDEN, preds)
                    if p and not l]
            print(f"   false positive: '{miss[0][0]}' / '{miss[0][1]}'")
            print("   (topic entities matched; the event did not. Entity")
            print("   overlap conflates topic with identity.)")
        print()

    print("verdict: the offline heuristic fails the kappa bar. Rerun this")
    print("harness with judge_claude from scripts/judge_pairs.py before")
    print("trusting any verdict in production.")


if __name__ == "__main__":
    main()

$ python scripts/rubric_validation.py
golden set: 16 pairs (6 dup, 10 not)

1) router sweep (flag if jaccard >= t); tune for RECALL
   t     flagged  precision  recall
  0.3       8     0.50      0.67
  0.4       8     0.50      0.67
  0.5       8     0.50      0.67
  0.6       6     0.50      0.50
  0.7       2     1.00      0.33
  ceiling: recall never reaches 1.0; the two case E paraphrases
  share zero tokens. That gap is the embedding layer's job.

2) entity-heuristic judge vs human labels
   confusion: TP=4 FP=1 FN=2 TN=9
   accuracy = 0.812   Cohen's kappa = 0.586   FAIL (bar: kappa >= 0.8)
   false positive: 'Champions League draw sets up blockbuster ties' / 'Champions League briefing: what to watch this week'
   (topic entities matched; the event did not. Entity
   overlap conflates topic with identity.)

3) always-not-dup baseline vs human labels
   confusion: TP=0 FP=0 FN=6 TN=10
   accuracy = 0.625   Cohen's kappa = 0.000   FAIL (bar: kappa >= 0.8)

verdict: the offline heuristic fails the kappa bar. Rerun this
harness with judge_claude from scripts/judge_pairs.py before
trusting any verdict in production.

Three lessons, each earned by a number:

• The router has a measurable ceiling. Recall tops out at 0.67 at every usable threshold because token overlap cannot see paraphrases. The sweep does not just pick a threshold; it documents what the layer cannot do, which is what justifies paying for the embedding layer.
• Accuracy lies on imbalanced data; kappa does not. The do-nothing baseline scores 62.5% accuracy and a kappa of exactly zero. The entity heuristic looks respectable at 81% accuracy and fails the bar at kappa 0.586. This single comparison is the argument for kappa as the acceptance metric.
• The false positive is a lesson, not just an error. Two Champions League headlines share their topic entities and are different stories. Entity overlap conflates topic with identity, which is precisely the reasoning gap the LLM judge exists to close. Validating the heuristic produced the requirement for the model.

The harness prints its own conclusion: the offline judge is not fit for production, so the Claude judge from Chapter 27 gets plugged into the same harness (same golden set, same kappa bar) before any verdict is trusted. That is the recommended validation method in full: human labels first, chance-corrected agreement as the bar, and the same harness for every judge you will ever swap in.

Beyond the harness, four validation methods run where a lab cannot:

And the re-validation rule, the part teams forget: validation is an event that repeats, triggered by change.

9. The completeness checklist

The framework is done when every box is checked, and not before:

1. Spec: versioned rubric in the repo with owners, on-fail actions, routing thresholds, judge labels, and a blind-spot register.
2. Layers: deterministic text, image hash, judge; each layer feeds only its residue to the next; exact duplicates never reach the judge.
3. Prompts: a registry with a version and content hash per prompt; the case matrix encoded as decision rules; a hash stamped into every verdict; a version bump re-triggers kappa validation.
4. Actions: every verdict maps to a mechanical action; repair backfills and re-scores; fail-open defaults written down.
5. Artifact: one JSON document per evaluated slate, stamped with the rubric version, containing values before and after repair.
6. Surfaces: CI gate, MLflow, dashboard, on-call alert, product-review gallery. All five.
7. Stage map: each criterion implemented at ingest, query, serve, nightly, and CI, per the table above; evaluation mode gates deploys, enforcement mode runs on traffic, and only enforcement is LLM-free.
8. Validation: golden set with inter-annotator kappa; router sweep; judge kappa against the bar; the re-validation trigger table; online guardrails.
9. Operations: verdict cache, batch judging, budget caps, drift alarms, version-stamped everything (Chapter 28's must list).

That is the framework entire: a spec you can review, layers you can afford, actions a machine can take, an artifact anyone can read, surfaces where decisions happen, and validation that earns the right to be believed. The recommender from Chapter 1 now ships with its own quality bar. What it does not yet have is a disciplined answer to the question every shipped system asks next: what do we build now? That is a research problem, and the last part of the book gives it the same from-scratch treatment. 👉