The verification mesh: buying precision with tokens

A fleet can now run, share a quota, survive crashes, and stay inside its budget. None of that makes its findings true. Generation is cheap and confident; agent output at fleet scale arrives with a real false-positive rate, and a report that is half noise is worse than no report, because someone has to read it. The verification mesh is the structural answer: a second, adversarial layer of agents whose only job is to kill claims, arranged so that precision becomes a number you measure and pay for rather than a hope. This chapter designs the mesh and prices it in a lab.

The adversarial stance

The design's heart is an asymmetry of instructions. Finders are prompted to find (coverage-biased, permitted to be wrong, because a downstream filter exists). Verifiers are prompted to refute: here is a claim and its evidence, try to demolish it, and default to rejection when uncertain. The same model that generates a plausible finding will, in a fresh context with reversed incentives, cheerfully tear it apart, and that asymmetry is the whole machine. Chapter 22 listed independence as one of the four things a second agent buys; the mesh is where it gets cashed.

Independence has operational content, not just vibes:

Fresh context per verdict. A verifier sees the claim and its evidence trail (by event id, from the ledger), never the finder's reasoning, which would anchor it.
Blind panels. Panelists must not see each other's votes; showing them converts three opinions into one opinion with two echoes. In practice this means parallel calls, not a conversation.
Dedup before the panel. Four finders rediscover the same bug; judging it four times wastes three panels. Deduplication (by claim key, or embedding similarity for fuzzier claims) sits between finding and verifying, and it is plain code, not an agent.

Lab 6.3: the mesh, measured

Synthetic corpus, sixty files, twenty-five planted bugs; four noisy finders (60% each per real bug, 12% per file false-alarm rate); refuter panels that keep real findings 90% of the time and spurious ones 15%. The rates are stand-ins; the arithmetic they feed is the production design, and every number is seeded:

python3 verify_mesh.py

corpus: 60 files, 25 planted bugs | 4 noisy finders

--- raw findings (deduplicated) ---
48 findings: 24 real, 24 spurious | precision 50%, recall 96% | 36,000 tokens

--- refuter panel sweep (majority vote) ---
 votes  kept  real  spur  precision  recall  verify tokens
     1    26    22     4       85%     88%       120,000
     3    24    23     1       96%     92%       360,000
     5    24    24     0      100%     96%       600,000

--- the 3-vote panel's trade, itemized ---
killed 24 findings: 23 false alarms removed, 1 real finding(s) lost
price: 360,000 verify tokens = 15,652 tokens per false alarm removed
every kept finding now carries 3 independent votes as provenance; the report's precision is a measured number, not a hope

What the numbers teach, row by row:

The raw layer is exactly as advertised: coverage, not truth. Four cheap finder passes caught 24 of 25 bugs (96% recall) and produced a report that is literally a coin flip (50% precision). Shipping that report burns the reader's trust in one meeting.
Even one verifier transforms it. A single vote took precision to 85% for a third of the eventual panel cost. The first verify token is the best-spent token in the whole system.
The panel sweep is a menu with prices. Three votes: 96% precision, one surviving false alarm, one real finding lost. Five votes: a perfect corpus, at two-thirds more verify spend. Note the sweep's quiet surprise: recall rose with panel size (88%, 92%, 96%), because majority-of-five forgives an individual refuter's wrongful rejection more often than majority-of-one. Bigger panels are gentler on truth as well as harsher on noise.
Verification dominated the budget, and that is correct. Finding cost 36k tokens; the 3-vote mesh cost 360k. Ten times more spent on checking than on finding is a normal, healthy ratio in high-stakes fleets, and teams that find it shocking are usually the ones shipping the 50%-precision report.

The chapter's operating rule falls out: fix the precision your report's consumer requires, then buy the smallest panel that clears it, using exactly this sweep run against your own corpus. And keep running it: plant known-true and known-false decoys in a few percent of production units, and the mesh's live precision becomes a monitored metric (Part 9's dashboards) instead of a launch-day measurement that quietly rots.

Where the mesh sits in the machine

The mesh is a topology (judge panels from Chapter 22) run by the same scheduler as everything else: verdict requests are units, panels are parallel dispatches, votes settle against the ledger, and the model portfolio decides who judges. A frequent, effective composition: a small model pre-filters obvious noise cheaply, the frontier panel judges what survives, and the arithmetic from this lab decides both thresholds. Capstone A runs precisely that stack over repository findings; Capstone D runs it over research claims with citation evidence.

Don't be confused: verification vs evaluation. Verification is in the run: it gates individual artifacts before they reach the report, and its output is keep or kill, per claim, at run time. Evaluation (Chapter 20, Part 9) is about the system: graders scoring trajectories offline to decide whether the fleet itself got better or worse this week. The mesh uses eval-like machinery (model judges, rubrics) for an in-line job. Confuse them and you either ship ungated artifacts because "we have evals," or block production on offline tooling never built for the hot path.

Full source

"""Lab 6.3: the verification mesh. Buying precision with tokens.

A synthetic audit: a corpus of files with planted bugs, noisy finder
agents that catch most real bugs but also cry wolf, and an adversarial
refuter panel that votes on every finding. The lab measures what the
panel buys (precision), what it costs (tokens, and a little recall),
and how panel size moves the trade.

The finders and refuters are seeded stochastic stand-ins with realistic
hit and false-alarm rates; the mesh arithmetic they feed is exactly the
production design. Standard library only. Deterministic.
"""

from __future__ import annotations

import random

FILES = 60
REAL_BUGS = 25                  # planted in files 0..24, one each
FINDERS = 4
FIND_HIT = 0.60                 # each finder's chance to catch a real bug
FIND_FALSE = 0.12               # per finder per file: spurious finding
REFUTER_TRUE_KEEP = 0.90        # refuter votes 'real' on a real finding
REFUTER_FALSE_KEEP = 0.15       # refuter votes 'real' on a spurious one
FINDER_TOKENS = 9_000           # tokens per finder per corpus shard
REFUTER_TOKENS = 2_500          # tokens per refuter vote
SEED = 0


def find_all() -> set[tuple[int, bool]]:
    """Union of all finders' findings, deduplicated.

    A finding is (file, is_real). Dedup matters: four finders rediscover
    the same real bug, but the panel should judge it once.
    """
    findings: set[tuple[int, bool]] = set()
    for finder in range(FINDERS):
        for f in range(FILES):
            rng = random.Random(f"{SEED}:find:{finder}:{f}")
            if f < REAL_BUGS and rng.random() < FIND_HIT:
                findings.add((f, True))
            if rng.random() < FIND_FALSE:
                findings.add((f, False))
    return findings


def panel_keeps(finding: tuple[int, bool], votes: int) -> bool:
    """Majority of an independent refuter panel votes to keep."""
    f, is_real = finding
    keep_p = REFUTER_TRUE_KEEP if is_real else REFUTER_FALSE_KEEP
    kept = sum(
        random.Random(f"{SEED}:refute:{f}:{is_real}:{v}").random() < keep_p
        for v in range(votes))
    return kept > votes // 2


def score(findings: set[tuple[int, bool]]) -> tuple[int, int, float, float]:
    real = sum(1 for _, is_real in findings if is_real)
    spurious = len(findings) - real
    precision = real / len(findings) if findings else 0.0
    recall = real / REAL_BUGS
    return real, spurious, precision, recall


if __name__ == "__main__":
    print(f"corpus: {FILES} files, {REAL_BUGS} planted bugs | "
          f"{FINDERS} noisy finders")

    raw = find_all()
    real, spur, p, r = score(raw)
    find_cost = FINDERS * FINDER_TOKENS
    print(f"\n--- raw findings (deduplicated) ---")
    print(f"{len(raw)} findings: {real} real, {spur} spurious | "
          f"precision {p:.0%}, recall {r:.0%} | {find_cost:,} tokens")

    print(f"\n--- refuter panel sweep (majority vote) ---")
    print(f"{'votes':>6}{'kept':>6}{'real':>6}{'spur':>6}"
          f"{'precision':>11}{'recall':>8}{'verify tokens':>15}")
    for votes in (1, 3, 5):
        kept = {f for f in raw if panel_keeps(f, votes)}
        k_real, k_spur, kp, kr = score(kept)
        cost = len(raw) * votes * REFUTER_TOKENS
        print(f"{votes:>6}{len(kept):>6}{k_real:>6}{k_spur:>6}"
              f"{kp:>10.0%}{kr:>8.0%}{cost:>14,}")

    kept3 = {f for f in raw if panel_keeps(f, 3)}
    killed = raw - kept3
    fp_removed = sum(1 for _, is_real in killed if not is_real)
    real_lost = sum(1 for _, is_real in killed if is_real)
    cost3 = len(raw) * 3 * REFUTER_TOKENS
    print(f"\n--- the 3-vote panel's trade, itemized ---")
    print(f"killed {len(killed)} findings: {fp_removed} false alarms "
          f"removed, {real_lost} real finding(s) lost")
    print(f"price: {cost3:,} verify tokens = "
          f"{cost3 // max(fp_removed, 1):,} tokens per false alarm removed")
    print(f"every kept finding now carries {3} independent votes as "
          f"provenance; the report's precision is a measured number, "
          f"not a hope")

👉 Next: communication, the part's closing question: when agents need to talk to each other at all, and the three patterns (results-only, blackboard, A2A) in escalating order of regret.

Agentic Systems on AWS

The verification mesh: buying precision with tokens

The adversarial stance

Lab 6.3: the mesh, measured

Where the mesh sits in the machine

Full source