Rubric evaluation: grading the feed itself

The metrics from Chapter 3 and the leaderboard from Chapter 13 answer one question: did the held-out click appear near the top? That question has a blind spot. Our capstone recommender builds an EMA profile (the time-decayed average of clicked-article embeddings from Chapter 5 is an exponentially weighted moving average, EMA for short) and retrieves that vector's nearest neighbors. Nearest neighbors of one point look like each other. So the model can score well on recall@10 and still ship a feed of ten near-identical soccer cards: same story from three outlets, three headlines cut from one template, thumbnails that are the same syndicated photo.

A rubric grades the slate itself: a fixed list of named criteria, each with a measurement and an explicit pass threshold. This chapter builds one for the capstone feed in three layers: deterministic checks (cheap, run on every slate), an image layer (the thumbnails carry signal the text misses), and an LLM judge for the calls that need reading comprehension.

Don't be confused: metric vs. rubric. recall@k needs a held-out label (the future click) and grades the model on average. A rubric needs no labels and grades one artifact (a single user's slate) against written criteria: relevant? fresh? duplicate-free? diverse? Metrics tell you which model to ship; rubrics tell you whether what it just produced is fit to show a user.

The rubric

One criterion per row, each with the cheapest tool that can measure it:

The design rule behind the table: deterministic first, LLM for the residue. A 10-card slate has $\binom{10}{2} = 45$ pairs. Screening the pairs with token overlap costs microseconds; sending all 45 to a model costs real money and seconds of latency. The cheap layer decides the easy 90% and routes only the ambiguous remainder to the judge.

The duplicate cases (headline x image)

A card is a (headline, image) pair, and the two channels fail independently. Writing the cases out first tells you which tool catches which:

Case D is the trap this chapter keeps returning to. "Lakers edge Celtics 102-99 in overtime thriller" and "Warriors edge Suns 118-115 in overtime thriller" share most of their words and are two different games. Token overlap flags them as duplicates; dropping one would delete a real story. Case E is the mirror-image trap: "Fed raises rates" and "Borrowing costs climb again" share zero tokens and are the same story. Every deterministic text signal has one of these two failure modes, which is exactly the gap the LLM judge fills.

Layer 1: the deterministic scorecard

scripts/rubric_eval.py builds the capstone recommender, produces a top-10 slate for one user, and grades four criteria with explicit thresholds:

#!/usr/bin/env python3
"""Deterministic rubric evaluation for one recommendation slate.

Builds the capstone recommender, produces a top-k slate for one user, and
grades the slate against a rubric of four criteria:

  relevance   mean cosine(user profile, card embedding)
  freshness   median story age, using each article's most recent click as a
              proxy for its publish time
  dup_titles  pairs of cards whose headlines look like the same story
              (token Jaccard OR embedding cosine above a threshold)
  diversity   distinct subcategories and the share of the biggest one

Each criterion prints PASS / WARN / FAIL against an explicit threshold, so a
slate change that quietly breaks the feed fails loudly in CI.

Run from code/capstone:
    python scripts/rubric_eval.py [user_id]
"""
from __future__ import annotations

import itertools
import os
import sys

import numpy as np

sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from newsreco.config import Config
from newsreco.data import load_all
from newsreco.recommender import NewsRecommender

# Thresholds are policy, not math: set them from a handful of slates you have
# looked at (and revisit them when the catalog changes).
REL_MIN = 0.15        # mean profile-cosine below this = off-taste slate
FRESH_MAX_H = 96.0    # median story age above this = stale slate
DUP_JACCARD = 0.5     # headline token overlap that counts as "same story?"
DUP_COSINE = 0.80     # or embedding cosine this high
DIVERSITY_TOP_SHARE = 0.80  # one subcategory filling more than this = bubble


def title_tokens(title: str) -> set:
    return {w for w in title.lower().split() if w.isalpha()}


def jaccard(a: set, b: set) -> float:
    return len(a & b) / max(len(a | b), 1)


def evaluate_slate(rec, data, user, k=10):
    slate = rec.recommend(user, k=k)
    profile = rec.profile(user)
    arts = [data.articles[nid] for nid in slate]
    rows = [rec.id_to_row[nid] for nid in slate]

    # -- relevance: cosine of every card to the taste vector -----------------
    cos = rec.emb[rows] @ profile
    relevance = float(cos.mean())

    # -- freshness: age of each story's most recent click --------------------
    last_click = {}
    for _, nid, t in data.interactions:
        last_click[nid] = max(last_click.get(nid, 0.0), t)
    ages_h = [(rec.now - last_click[nid]) / 3600.0 for nid in slate if nid in last_click]
    freshness = float(np.median(ages_h)) if ages_h else float("nan")

    # -- near-duplicate headlines: every pair, two signals --------------------
    dups = []
    for i, j in itertools.combinations(range(len(slate)), 2):
        jac = jaccard(title_tokens(arts[i].title), title_tokens(arts[j].title))
        sim = float(rec.emb[rows[i]] @ rec.emb[rows[j]])
        if jac >= DUP_JACCARD or sim >= DUP_COSINE:
            dups.append((i, j, jac, sim))

    # -- diversity: subcategory spread ----------------------------------------
    subcats = [a.subcategory for a in arts]
    counts = {s: subcats.count(s) for s in set(subcats)}
    top_share = max(counts.values()) / len(subcats)

    return slate, arts, {
        "relevance": relevance, "freshness_h": freshness,
        "dups": dups, "n_subcats": len(counts), "top_share": top_share,
    }


def verdict(ok, warn=False):
    return "PASS" if ok else ("WARN" if warn else "FAIL")


def main():
    user = sys.argv[1] if len(sys.argv) > 1 else "U106"
    cfg = Config()
    data = load_all(cfg)
    rec = NewsRecommender(cfg.embedder, cfg.half_life_hours).fit(data)
    slate, arts, r = evaluate_slate(rec, data, user, k=10)

    print(f"slate for user {user} (k={len(slate)}):")
    for i, a in enumerate(arts, 1):
        print(f"  {i:2d}. [{a.subcategory}] {a.title}")

    rows_out = [
        ("relevance", f"mean cosine = {r['relevance']:.3f}",
         verdict(r["relevance"] >= REL_MIN), f"min {REL_MIN}"),
        ("freshness", f"median story age = {r['freshness_h']:.1f} h",
         verdict(r["freshness_h"] <= FRESH_MAX_H), f"max {FRESH_MAX_H:.0f} h"),
        ("dup-titles", f"{len(r['dups'])} pair(s) flagged",
         verdict(not r["dups"], warn=True), f"jac>={DUP_JACCARD} or cos>={DUP_COSINE}"),
        ("diversity", f"{r['n_subcats']} subcats, top share = {r['top_share']:.2f}",
         verdict(r["top_share"] <= DIVERSITY_TOP_SHARE), f"top share max {DIVERSITY_TOP_SHARE}"),
    ]
    print("\nrubric scorecard")
    print("-" * 66)
    for name, desc, verd, rule in rows_out:
        print(f"{name:<11} {desc:<33} {verd:<5} ({rule})")
    print("-" * 66)

    if r["dups"]:
        print("flagged pairs (send these to the LLM judge, not the whole slate):")
        for i, j, jac, sim in r["dups"]:
            print(f"  ({i + 1},{j + 1}) jaccard={jac:.2f} cosine={sim:.2f}")
            print(f"      - {arts[i].title}")
            print(f"      - {arts[j].title}")


if __name__ == "__main__":
    main()

Running it on the soccer-history user from Chapter 18:

$ python scripts/rubric_eval.py U106
slate for user U106 (k=10):
   1. [soccer] Bayern Munich and Chelsea play out thrilling 2-2 draw
   2. [soccer] Barcelona and Inter Milan play out thrilling 4-4 draw
   3. [soccer] Bukayo Saka wins Ballon d'Or after stellar football season
   4. [soccer] Manchester City and Paris Saint-Germain play out thrilling 4-4 draw
   5. [soccer] Barcelona and Paris Saint-Germain play out thrilling 4-4 draw
   6. [soccer] Kylian Mbappe wins Ballon d'Or after stellar football season
   7. [soccer] Paris Saint-Germain and Inter Milan play out thrilling 3-3 draw
   8. [soccer] Erling Haaland wins Ballon d'Or after stellar football season
   9. [soccer] Manchester City and Real Madrid play out thrilling 1-1 draw
  10. [soccer] Manchester City and Bayern Munich play out thrilling 3-3 draw

rubric scorecard
------------------------------------------------------------------
relevance   mean cosine = 0.575               PASS  (min 0.15)
freshness   median story age = 57.2 h         PASS  (max 96 h)
dup-titles  13 pair(s) flagged                WARN  (jac>=0.5 or cos>=0.8)
diversity   1 subcats, top share = 1.00       FAIL  (top share max 0.8)
------------------------------------------------------------------
flagged pairs (send these to the LLM judge, not the whole slate):
  (1,5) jaccard=0.50 cosine=0.72
      - Bayern Munich and Chelsea play out thrilling 2-2 draw
      - Barcelona and Paris Saint-Germain play out thrilling 4-4 draw
  (1,10) jaccard=0.70 cosine=0.60
      - Bayern Munich and Chelsea play out thrilling 2-2 draw
      - Manchester City and Bayern Munich play out thrilling 3-3 draw
  (2,5) jaccard=0.67 cosine=0.79
      - Barcelona and Inter Milan play out thrilling 4-4 draw
      - Barcelona and Paris Saint-Germain play out thrilling 4-4 draw
  ...
  (9,10) jaccard=0.64 cosine=0.71
      - Manchester City and Real Madrid play out thrilling 1-1 draw
      - Manchester City and Bayern Munich play out thrilling 3-3 draw

Three things in this scorecard would never show up on a recall leaderboard:

• diversity FAIL. The whole slate is one subcategory. This is the over-specialization weakness from Chapter 5 surfacing in production form: the EMA profile sits in the middle of the "soccer" region, and its nearest neighbors are all soccer. A leaderboard rewards this; a user tires of it.
• 13 flagged pairs, and every one is Case D. The sample corpus writes match reports from one template, so token Jaccard fires constantly on different games. The deterministic layer cannot tell; it can only route. Note the last line of the script: the judge gets the 13 flagged pairs, not all 45.
• Thresholds are policy, not math. 0.15 relevance and 96 h freshness were chosen by looking at a handful of slates, and belong in code review like any other constant. The point is that a change that quietly degrades the feed (an embedder swap, a half-life tweak) now fails loudly in CI.

Layer 2: images (dHash from scratch)

Headlines miss Case C entirely: two outlets can run different headlines over the same syndicated photo, and the same outlet can reuse a stock image on a new story. The standard cheap tool is a perceptual hash. We build dHash (difference hash) in a few lines of NumPy: shrink the image to a 9x8 grid of block means $s$, then keep one bit per horizontal neighbor pair,

$$ b_{ij} = \big[, s_{i,j+1} > s_{i,j} ,\big], \qquad i \in [0,8), ; j \in [0,8), $$

giving a 64-bit fingerprint of the image's gradient structure. Brightness shifts, recompression, and light crops barely move it; a different photo flips roughly half the bits. Similarity is the Hamming distance (number of differing bits).

scripts/thumb_dedup.py generates synthetic 64x64 "thumbnails" (so the lab runs with zero image dependencies) and plays out cases A through D against a reference card:

#!/usr/bin/env python3
"""Near-duplicate detection for news cards: headline text + thumbnail image.

A news card is (headline, image). Two cards can collide four ways:

  A  same headline, same image        -> exact duplicate
  B  similar headline, new image      -> same story from a second outlet
  C  new headline, same image         -> syndicated / stock photo
  D  similar headline, new image,     -> template lookalike: DIFFERENT story
     but a different event               (deterministic text check misfires)

Text side: token Jaccard on the headlines. Image side: dHash (difference
hash), built here from scratch with NumPy: shrink the image to 9x8, compare
each pixel to its right neighbor, and keep the 72 resulting bits. Images that
are "the same photo" (recompressed, brightened, lightly cropped) land within
a few bits of each other; unrelated photos differ in dozens of bits.

Run:  python scripts/thumb_dedup.py
"""
from __future__ import annotations

import numpy as np

SIZE = 64  # synthetic thumbnails are 64x64 grayscale, values 0..255


# --------------------------------------------------------------- thumbnails
def photo_pitch(rng):
    """A 'stadium at night' photo: dark gradient + bright ball + pitch stripes."""
    y, x = np.mgrid[0:SIZE, 0:SIZE]
    img = 60 + 90 * (y / SIZE)                              # vertical gradient
    img += 25 * ((x // 8) % 2)                              # pitch stripes
    ball = (x - 44) ** 2 + (y - 20) ** 2 < 36               # the ball
    img[ball] = 230
    return np.clip(img + rng.normal(0, 2, img.shape), 0, 255)


def photo_arena(rng):
    """An unrelated 'indoor arena' photo: diagonal gradient + dark scoreboard."""
    y, x = np.mgrid[0:SIZE, 0:SIZE]
    img = 200 - 120 * ((x + y) / (2 * SIZE))                # diagonal gradient
    img[10:26, 20:44] = 35                                  # scoreboard block
    return np.clip(img + rng.normal(0, 2, img.shape), 0, 255)


def rerender(img, rng):
    """Same photo after a CDN round-trip: brightness shift + compression noise."""
    return np.clip(img * 0.93 + 12 + rng.normal(0, 3, img.shape), 0, 255)


def crop(img, px=5):
    """Same photo cropped by `px` on every side, scaled back up (nearest)."""
    inner = img[px:-px, px:-px]
    idx = (np.arange(SIZE) * inner.shape[0] / SIZE).astype(int)
    return inner[np.ix_(idx, idx)]


# ------------------------------------------------------------------- dHash
def resize_mean(img, h, w):
    """Shrink by averaging blocks (a tiny stand-in for a real resize)."""
    ye = np.linspace(0, img.shape[0], h + 1).astype(int)
    xe = np.linspace(0, img.shape[1], w + 1).astype(int)
    return np.array([[img[ye[i]:ye[i + 1], xe[j]:xe[j + 1]].mean()
                      for j in range(w)] for i in range(h)])


def dhash(img, hash_size=8):
    small = resize_mean(img, hash_size, hash_size + 1)      # 8 rows, 9 cols
    return (small[:, 1:] > small[:, :-1]).flatten()         # 64 bits


def hamming(h1, h2):
    return int((h1 != h2).sum())


# ------------------------------------------------------------- text signal
def jaccard(t1, t2):
    a = {w for w in t1.lower().split() if w.isalpha()}
    b = {w for w in t2.lower().split() if w.isalpha()}
    return len(a & b) / max(len(a | b), 1)


def route(jac, ham, t_jac=0.5, t_ham=10):
    title_same, image_same = jac >= t_jac, ham <= t_ham
    if title_same and image_same:
        return "exact duplicate -> drop one"
    if title_same:
        return "same story, two outlets? -> LLM judge"
    if image_same:
        return "syndicated photo, new angle? -> LLM judge"
    return "distinct -> keep both"


def main():
    rng = np.random.default_rng(0)
    pitch = photo_pitch(rng)
    base = ("Lakers edge Celtics 102-99 in overtime thriller", pitch)
    cases = [
        ("A", "Lakers edge Celtics 102-99 in overtime thriller", rerender(pitch, rng)),
        ("B", "Lakers beat Celtics in 102-99 overtime thriller", photo_arena(rng)),
        ("C", "NBA playoff race tightens after week of upsets", crop(pitch)),
        ("D", "Warriors edge Suns 118-115 in overtime thriller", photo_arena(rng)),
    ]

    h_base = dhash(base[1])
    print(f"reference card: '{base[0]}'\n")
    print("case  jaccard  hamming  routing")
    print("-" * 72)
    for tag, title, img in cases:
        jac = jaccard(base[0], title)
        ham = hamming(h_base, dhash(img))
        print(f"  {tag}    {jac:5.2f}    {ham:4d}    {route(jac, ham)}")
        print(f"        '{title}'")
    print("-" * 72)
    print("thresholds: title_same if jaccard >= 0.5, image_same if hamming <= 10")


if __name__ == "__main__":
    main()

$ python scripts/thumb_dedup.py
reference card: 'Lakers edge Celtics 102-99 in overtime thriller'

case  jaccard  hamming  routing
------------------------------------------------------------------------
  A     1.00       0    exact duplicate -> drop one
        'Lakers edge Celtics 102-99 in overtime thriller'
  B     0.71      36    same story, two outlets? -> LLM judge
        'Lakers beat Celtics in 102-99 overtime thriller'
  C     0.00       7    syndicated photo, new angle? -> LLM judge
        'NBA playoff race tightens after week of upsets'
  D     0.50      37    same story, two outlets? -> LLM judge
        'Warriors edge Suns 118-115 in overtime thriller'
------------------------------------------------------------------------
thresholds: title_same if jaccard >= 0.5, image_same if hamming <= 10

Read the routing column. Case A is decided outright (both channels agree: drop one card, no model call). Case C is invisible to text (Jaccard 0.00) and caught by the image hash at 7 bits. And cases B and D land in the same bucket with the same signals: similar headline, different image. One is a true duplicate story, the other is a different game. No threshold on these two numbers can separate them; something has to read the headlines.

Don't be confused: dHash is a perceptual hash, not a cryptographic one. MD5/SHA answer "are these files byte-identical?" and flip completely on a one-pixel change. A perceptual hash answers "do these look alike?" and moves a few bits under recompression. For real thumbnails swap this toy in for imagehash.phash (or dhash) over Pillow; the routing logic stays identical.

Layer 3: the LLM judge

The judge answers exactly one narrow question per call, and its output is forced into a three-label schema so the pipeline can act on it mechanically:

scripts/judge_pairs.py re-runs the deterministic pass and judges the flagged pairs. Like the RAG assistant, generation is pluggable: with an API key it asks Claude; without one it falls back to a transparent entity heuristic (capitalized tokens are entities; same entities, same story), so the script always runs:

#!/usr/bin/env python3
"""LLM judge for the headline pairs the deterministic rubric flagged.

The deterministic pass (scripts/rubric_eval.py) flags every pair of slate
cards whose headlines overlap heavily. Overlap alone cannot tell "the same
story written twice" from "the same TEMPLATE about two different events"
(different match, different player). That call needs reading comprehension,
so each flagged pair goes to a judge with a three-label rubric:

  same_story             the two headlines describe one event -> drop one card
  related_but_distinct   same topic, different event          -> keep both
  unrelated              the flagger misfired                  -> keep both

Generation is pluggable, like the RAG assistant:
  * with ANTHROPIC_API_KEY set and the `anthropic` package installed, each
    pair is judged by Claude, forced into the schema via structured output;
  * otherwise a transparent entity heuristic answers (capitalized tokens =
    entities; same entities = same story), so the script always runs.

Run from code/capstone:
    python scripts/judge_pairs.py [user_id]
"""
from __future__ import annotations

import os
import sys

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from rubric_eval import evaluate_slate  # noqa: E402
from newsreco.config import Config  # noqa: E402
from newsreco.data import load_all  # noqa: E402
from newsreco.recommender import NewsRecommender  # noqa: E402

LABELS = ("same_story", "related_but_distinct", "unrelated")
ACTION = {"same_story": "drop lower-ranked card",
          "related_but_distinct": "keep both",
          "unrelated": "keep both (recheck flagger)"}

JUDGE_SYSTEM = (
    "You judge pairs of news headlines for a recommendation feed. Decide if "
    "they report the SAME event, DIFFERENT events on the same topic, or are "
    "unrelated. Two match reports about different games are NOT the same "
    "story, even if the wording matches."
)

STOP = {"the", "and", "after", "in", "out"}


def entities(title: str) -> set:
    return {w for w in title.split() if w[:1].isupper() and w.lower() not in STOP}


def judge_offline(t1: str, t2: str):
    """Fallback judge: same entities = same story."""
    e1, e2 = entities(t1), entities(t2)
    overlap = len(e1 & e2) / max(len(e1 | e2), 1)
    if overlap >= 0.8:
        return "same_story", f"entity overlap {overlap:.2f}"
    if overlap > 0.0:
        return "related_but_distinct", f"entity overlap {overlap:.2f}"
    return "unrelated", "no shared entities"


def judge_claude(client, model, t1: str, t2: str):
    """Real judge: Claude, forced into the rubric schema."""
    from pydantic import BaseModel
    from typing import Literal

    class PairVerdict(BaseModel):
        verdict: Literal["same_story", "related_but_distinct", "unrelated"]
        reason: str

    resp = client.messages.parse(
        model=model, max_tokens=1024, system=JUDGE_SYSTEM,
        messages=[{"role": "user",
                   "content": f"Headline 1: {t1}\nHeadline 2: {t2}"}],
        output_format=PairVerdict,
    )
    v = resp.parsed_output
    return v.verdict, v.reason


def main():
    user = sys.argv[1] if len(sys.argv) > 1 else "U106"
    cfg = Config()
    data = load_all(cfg)
    rec = NewsRecommender(cfg.embedder, cfg.half_life_hours).fit(data)
    _, arts, r = evaluate_slate(rec, data, user, k=10)

    client = None
    if cfg.anthropic_api_key:
        try:
            import anthropic
            client = anthropic.Anthropic(api_key=cfg.anthropic_api_key)
        except ImportError:
            pass
    mode = "claude" if client else "offline entity heuristic"
    print(f"judging {len(r['dups'])} flagged pair(s)  [mode: {mode}]\n")

    for i, j, jac, _ in r["dups"]:
        t1, t2 = arts[i].title, arts[j].title
        if client:
            verdict, reason = judge_claude(client, cfg.llm_model, t1, t2)
        else:
            verdict, reason = judge_offline(t1, t2)
        print(f"({i + 1},{j + 1}) jac={jac:.2f} -> {verdict:<22} {ACTION[verdict]}")
        print(f"      '{t1}'  vs  '{t2}'  ({reason})")


if __name__ == "__main__":
    main()

Offline mode, on the 13 pairs from layer 1:

$ python scripts/judge_pairs.py U106
judging 13 flagged pair(s)  [mode: offline entity heuristic]

(1,5) jac=0.50 -> unrelated              keep both (recheck flagger)
      'Bayern Munich and Chelsea play out thrilling 2-2 draw'  vs  'Barcelona and Paris Saint-Germain play out thrilling 4-4 draw'  (no shared entities)
(1,10) jac=0.70 -> related_but_distinct   keep both
      'Bayern Munich and Chelsea play out thrilling 2-2 draw'  vs  'Manchester City and Bayern Munich play out thrilling 3-3 draw'  (entity overlap 0.40)
  ...
(6,8) jac=0.60 -> related_but_distinct   keep both
      'Kylian Mbappe wins Ballon d'Or after stellar football season'  vs  'Erling Haaland wins Ballon d'Or after stellar football season'  (entity overlap 0.20)
(9,10) jac=0.64 -> related_but_distinct   keep both
      'Manchester City and Real Madrid play out thrilling 1-1 draw'  vs  'Manchester City and Bayern Munich play out thrilling 3-3 draw'  (entity overlap 0.33)

All 13 flags survive as "keep both": the deterministic layer produced zero true duplicates and 13 template lookalikes, and the judge (even the heuristic one) corrected every one. That asymmetry is the normal shape of this pipeline. The cheap layer is tuned for recall (never miss a possible dup), the judge supplies the precision.

Turning on Claude

With ANTHROPIC_API_KEY set (and pip install anthropic pydantic), the same script sends each pair to claude-opus-4-8. The load-bearing detail is structured output: client.messages.parse() with a Pydantic model whose verdict field is a Literal of the three labels. The API guarantees the response matches the schema, so there is no output parsing, no "the model answered in prose" failure mode, and resp.parsed_output.verdict is safe to branch on:

class PairVerdict(BaseModel):
    verdict: Literal["same_story", "related_but_distinct", "unrelated"]
    reason: str

resp = client.messages.parse(
    model="claude-opus-4-8", max_tokens=1024, system=JUDGE_SYSTEM,
    messages=[{"role": "user", "content": f"Headline 1: {t1}\nHeadline 2: {t2}"}],
    output_format=PairVerdict,
)

Output with a key set (illustrative; wording varies run to run):

judging 13 flagged pair(s)  [mode: claude]

(1,5) jac=0.50 -> related_but_distinct   keep both
      'Bayern Munich and Chelsea play out thrilling 2-2 draw'  vs  'Barcelona and Paris Saint-Germain play out thrilling 4-4 draw'  (Two different fixtures with different clubs and different scores.)
  ...
(3,6) jac=0.60 -> related_but_distinct   keep both
      'Bukayo Saka wins Ballon d'Or after stellar football season'  vs  'Kylian Mbappe wins Ballon d'Or after stellar football season'  (Same award, but each headline names a different winner, so at most one can be the real event.)
  ...

Notice the (3,6) reason: the judge did more than compare entities, it spotted that two players cannot both win the same award, which is a data-quality smell no string metric would ever raise. This is the concrete value an LLM adds over the heuristic: verdicts come with reasons you can log and audit.

The same pattern extends to the criteria the deterministic table marked "needs an LLM". For headline-image coherence, send the thumbnail as an image content block next to the headline and ask for a Literal["depicts_story", "generic_stock", "mismatched"]. For clickbait, a Literal["informative", "teaser", "clickbait"] with a one-line reason. One criterion, one call, one forced schema, every time.

Judge best practices

Rules that keep an LLM judge honest and affordable, each earned the hard way by someone:

1. One criterion per call. "Rate this slate 1 to 10" produces numbers that drift with mood and prompt phrasing. "Are these two headlines the same event?" produces verdicts you can count, compare, and regress.
2. Force the schema. Always structured output with a Literal label set, never free text you parse with a regex. Add a reason field: it costs a few tokens and turns every disagreement into a debuggable log line.
3. Anchor the labels in the prompt. Our system prompt includes the Case D trap explicitly ("two match reports about different games are NOT the same story"). The rubric lives in the prompt, not in the model's imagination.
4. Judge the residue, not the universe. Deterministic screen first, LLM on flagged pairs only. Here that is 13 calls instead of 45 per slate; on a real feed the screen typically removes well over 90% of pairs.
5. Cache verdicts. Key on the sorted pair of article ids. Catalogs repeat across users and days; the same pair should never be judged twice.
6. Neutralize order bias. Judges measurably favor one position in pairwise comparisons. Canonicalize (sort the two headlines) or judge both orders and keep agreements; treat disagreements as related_but_distinct (the safe default here is keeping both cards).
7. Vote on the borderline. For pairs the judge flips on across runs, sample three verdicts and take the majority. Spend this only on disagreements, not on every pair.
8. Calibrate against a golden set. Label 100 to 200 pairs by hand once (include deliberate B, D, and E cases). Report the judge's agreement with humans (percent agreement, or Cohen's kappa to correct for chance) before trusting it, and re-run the golden set whenever the prompt or model version changes. An uncalibrated judge is just a slower random number generator.
9. Keep the judge out of the training loop. Grading slates is safe. Feeding judge labels back as ranker training data invites Goodhart's law: the ranker learns to please the judge, and the judge's blind spots become the product's blind spots. If you must, audit a sample by hand first.
10. Pin versions and log everything. Judge model id, prompt hash, and verdict go into the run log next to the slate. When the feed regresses, you want to know whether the recommender changed or the judge did.

Wiring it into the capstone

• CI gate. Run rubric_eval.py over a fixed sample of users in the training pipeline and log the scorecard numbers to MLflow next to recall@k. A model that raises recall but doubles flagged pairs is now a visible tradeoff instead of a silent one.
• Serving guard. The same_story verdicts become one more filter in candidates, right after the exact-title dedupe that is already there. With the verdict cache, this adds no model calls on the hot path.
• Drift monitor. Nightly, sample 100 users, run the rubric, and chart the four scorecard numbers over time. The diversity FAIL above is the filter bubble of Chapter 5; watching top-share drift upward is how you catch it before users do.

That completes the toolkit: metrics to pick the model, a leaderboard to rank the candidates, and now a rubric (deterministic checks, an image layer, and a calibrated LLM judge) to grade what the model actually puts in front of people. What remains is taking the prototype to production hardware: profiles in a database, articles in a search index, and duplicates excluded before the query even returns. 👉