Research mechanics: parallel agents, graphs, and what to try next

Chapter 33 walked one research cycle end to end and produced documents: a research report, a brief, a PRD. This chapter is about the machinery underneath that cycle, and it answers three questions that come up the moment you do this seriously. Should the work be split across parallel agents? Where should the accumulating knowledge live so it stays queryable? And when the sprint budget runs low, how do you decide which thread to push and which to drop?

The first question is about orchestration, the second about data structures, the third about decision theory. Each section ends in a runnable lab, because "a graph would help here" is a claim, and claims get tested.

Are subagents and parallel work helpful?

Sometimes, and the boundary is sharp. Anthropic published the internals of its claude.ai Research feature: an orchestrator agent decomposes the question and spawns worker subagents that search in parallel, each acting as an "intelligent filter" over its own slice of sources. On their internal research eval, the multi-agent system (an Opus lead with Sonnet workers) outperformed a single agent by 90.2%. The same write-up reports the bill: multi-agent runs burn roughly 15x the tokens of a normal chat.

That pair of numbers is the whole tradeoff. Parallelism wins when the work decomposes; it wastes money when it doesn't. For our feed research the split looks like this:

Three parallel patterns cover most research work:

1. Fan-out sweep. One subagent per source type or per question. Each returns claims with URLs, not prose. The orchestrator merges and deduplicates.
2. Adversarial verification. For any claim your decision will rest on ("PinnerSage showed averaging hurts"), spawn two or three agents prompted to disprove it against the primary source. A claim that survives hostile readers is worth building on; most citation errors die here.
3. Completeness critic. After the sweep, one agent whose only job is "what is missing: a search angle not run, a claim unverified, a source unread?" Its findings become the next round.

BMAD splits its own work along exactly this boundary. The document pipeline is sequential by design (Mary, then John, then Winston, each reading the previous document) because it is a chain of judgment calls. But inside the research step, Deep Recon's Run mode is a parallel fan-out: breadth-first topology puts independent sub-questions on parallel assistants, depth-first attacks one question from several angles, and the effort presets (2, 3, or 6 assistants) are a parallelism dial. At the implementation end, bmad-dev-auto refuses to run without subagent support, and the bmad-loop module (added in v6.10) exists to dispatch several stories to concurrent subagent sessions unattended. Parallelize sweeps, serialize judgment: the same default holds whatever tool you use.

Don't be confused: parallel agents vs. the party-mode chat. BMAD's party mode puts several personas in one conversation taking turns; it is still one context window and one token stream, useful for debate, not for throughput. Parallel subagents run in separate context windows at the same time; they buy wall-clock speed and independent perspectives, at 10x-plus token cost.

Storing what you learn: from flat notes to a graph

A one-week sweep produces 20 to 50 sources and a few dozen extracted claims. Flat notes hold up fine at ten sources. Past that, three questions start requiring a full reread of the notes file:

• Coverage: which open question is thinly evidenced?
• Contradiction: do any two claims disagree, and which sources back each side?
• Priority: of the sources still unread, which one first?

All three are join-shaped: they connect claims to questions to sources. That is what a typed graph is for. research_graph.py stores a small sweep (the corpus is synthetic; the machinery is the point) as three node types and four edge types, then runs the three queries:

"""A research knowledge graph: coverage, contradictions, and what to read next.

Notes from a literature sweep usually land in a flat file, and a flat file
cannot answer the three questions that decide what you do tomorrow morning:
which open question is under-covered, which two claims disagree, and which
unread source is worth opening next. Stored as a typed graph, all three
become short traversals.

The corpus below is synthetic (stand-ins for a real reading list); the
graph machinery is the point. Stdlib + NumPy only.
"""

import numpy as np

# ---------------------------------------------------------------- the graph
# Three node types. Every edge is (src, relation, dst).

QUESTIONS = {
    "Q1": "do image embeddings on cards lift engagement?",
    "Q2": "which EMA half-life fits news interest drift?",
    "Q3": "does MMR diversity re-ranking cost short-term clicks?",
}

SOURCES = {  # id: (title, read?)  -- unread sources have no claims yet
    "S1": ("clip-overview", True),
    "S2": ("visual-news-rec", True),
    "S3": ("text-is-enough", True),
    "S4": ("interest-drift-study", False),
    "S5": ("mmr-tradeoffs", True),
    "S6": ("diversity-retention", False),
    "S7": ("half-life-survey", False),
    "S8": ("two-tower-methods", False),
}

CLAIMS = {  # id: (text, question, polarity)  +1 for, -1 against, 0 background
    "C1": ("image embeddings lifted CTR in an online test", "Q1", +1),
    "C2": ("no lift over a strong text-embedding baseline", "Q1", -1),
    "C3": ("images and text can share one embedding space", "Q1", 0),
    "C4": ("MMR re-ranking cost 2% CTR in the short term", "Q3", +1),
}

MAKES = [("S2", "C1"), ("S3", "C2"), ("S1", "C3"), ("S5", "C4")]

RELEVANT = [  # tagged at collection time (from the abstract), before reading
    ("S4", "Q2"), ("S7", "Q2"), ("S6", "Q3"), ("S8", "Q1")]

CITES = [("S2", "S1"), ("S3", "S1"), ("S3", "S2"), ("S5", "S2"),
         ("S6", "S5"), ("S7", "S4"), ("S6", "S2"), ("S2", "S8"), ("S3", "S8")]

# ------------------------------------------------- query 1: coverage per question
def coverage():
    """Sources whose extracted claims bear on each question (read evidence)."""
    per_q = {q: set() for q in QUESTIONS}
    for src, claim in MAKES:
        per_q[CLAIMS[claim][1]].add(src)
    return {q: sorted(srcs) for q, srcs in per_q.items()}

# --------------------------------------------- query 2: contradiction detection
def contradictions():
    """Two claims on the same question with opposite (nonzero) polarity."""
    out = []
    ids = sorted(CLAIMS)
    for i, a in enumerate(ids):
        for b in ids[i + 1:]:
            _, qa, pa = CLAIMS[a]
            _, qb, pb = CLAIMS[b]
            if qa == qb and pa * pb < 0:
                out.append((a, b, qa))
    return out

# ------------------------------------- query 3: source authority via PageRank
def pagerank(damping=0.85, iters=50):
    ids = sorted(SOURCES)
    idx = {s: i for i, s in enumerate(ids)}
    n = len(ids)
    M = np.zeros((n, n))
    for citer, cited in CITES:           # authority flows citer -> cited
        M[idx[cited], idx[citer]] = 1.0
    col = M.sum(axis=0)
    M[:, col > 0] /= col[col > 0]        # column-stochastic where possible
    r = np.full(n, 1.0 / n)
    for _ in range(iters):
        r = (1 - damping) / n + damping * (M @ r + r[col == 0].sum() / n)
    return {s: r[idx[s]] for s in ids}

# ------------------------------------------------------------- the report
if __name__ == "__main__":
    n_edges = len(MAKES) + len(CITES) + len(RELEVANT) + len(CLAIMS)
    print(f"graph: {len(SOURCES)} sources, {len(CLAIMS)} claims, "
          f"{len(QUESTIONS)} questions, {n_edges} edges\n")

    cov = coverage()
    print("coverage (read sources with extracted claims, per question)")
    for q, srcs in sorted(cov.items(), key=lambda kv: -len(kv[1])):
        tag = "  <- under-covered" if len(srcs) < 2 else ""
        print(f"  {q} {QUESTIONS[q]:<52} {len(srcs)} source(s){tag}")

    print("\ncontradictions (same question, opposite polarity)")
    for a, b, q in contradictions():
        for cid in (a, b):
            text, _, pol = CLAIMS[cid]
            src = next(s for s, c in MAKES if c == cid)
            print(f"  {cid} ({'+' if pol > 0 else '-'}) '{text}'  [{SOURCES[src][0]}]")
        print(f"     -> {q} is contested: schedule our own experiment")

    pr = pagerank()
    print("\nsource authority (PageRank over the citation edges)")
    for s, score in sorted(pr.items(), key=lambda kv: -kv[1]):
        title, read = SOURCES[s]
        print(f"  {score:.3f}  {title:<22} {'read' if read else 'UNREAD'}")

    # read-next = unread sources, authority x (2x bonus if tagged relevant
    # to an under-covered question)
    gaps = {q for q, srcs in cov.items() if len(srcs) < 2}
    touches = dict(RELEVANT)
    ranked = sorted(((pr[s] * (2.0 if touches.get(s) in gaps else 1.0), s)
                     for s, (_, read) in SOURCES.items() if not read),
                    reverse=True)
    print("\nread next (unread, authority x coverage-gap bonus)")
    for i, (score, s) in enumerate(ranked, 1):
        gap = (f"  (relevant to {touches[s]}, a coverage gap)"
               if touches.get(s) in gaps else "")
        print(f"  {i}. {SOURCES[s][0]:<22} score {score:.3f}{gap}")

$ python3 research_graph.py
graph: 8 sources, 4 claims, 3 questions, 21 edges

coverage (read sources with extracted claims, per question)
  Q1 do image embeddings on cards lift engagement?        3 source(s)
  Q3 does MMR diversity re-ranking cost short-term clicks? 1 source(s)  <- under-covered
  Q2 which EMA half-life fits news interest drift?        0 source(s)  <- under-covered

contradictions (same question, opposite polarity)
  C1 (+) 'image embeddings lifted CTR in an online test'  [visual-news-rec]
  C2 (-) 'no lift over a strong text-embedding baseline'  [text-is-enough]
     -> Q1 is contested: schedule our own experiment

source authority (PageRank over the citation edges)
  0.205  visual-news-rec        read
  0.177  clip-overview          read
  0.177  two-tower-methods      UNREAD
  0.130  interest-drift-study   UNREAD
  0.100  mmr-tradeoffs          read
  0.070  text-is-enough         read
  0.070  diversity-retention    UNREAD
  0.070  half-life-survey       UNREAD

read next (unread, authority x coverage-gap bonus)
  1. interest-drift-study   score 0.260  (relevant to Q2, a coverage gap)
  2. two-tower-methods      score 0.177
  3. half-life-survey       score 0.140  (relevant to Q2, a coverage gap)
  4. diversity-retention    score 0.140  (relevant to Q3, a coverage gap)

Read the output bottom to top. The reading order is not the authority order: two-tower-methods outranks two of the picks on PageRank, but the gap bonus pushes interest-drift-study first, because Q2 has zero read evidence and tomorrow's PRD needs a half-life recommendation. The contradiction report is the other payoff: C1 and C2 disagree about the exact question our PRD must answer, and the graph both detects it and names the resolution (run our own ablation; Chapter 31's walkthrough turned exactly this into an experiment story). A flat notes file contains the same facts and answers none of these questions without a reread.

The polarity field earns its keep too: claim C3 ("images and text can share one embedding space") is background, polarity 0, so the contradiction detector correctly ignores it. Not every fact takes a side.

The industrial version: GraphRAG and citation graphs

Our 40-line graph scales to a few hundred nodes. Two families of tools pick up from there:

• Microsoft GraphRAG (arXiv:2404.16130; microsoft/graphrag, v3.x as of mid-2026) runs the same idea over an entire private corpus: an LLM extracts entities, relationships, and claims from text chunks, the graph is clustered hierarchically with the Leiden algorithm, and each community gets a pre-written summary. "Global" queries ("what are the main themes across everything we collected?") are answered from community summaries; "local" queries fan out from one entity to its neighbors; DRIFT search mixes the two. The paper's pitch is precisely our coverage query at corpus scale: questions that no single chunk answers. The docs warn, fairly, that indexing a large corpus through an LLM is expensive.
• Citation graphs already exist for the academic slice of your sweep; you query rather than build. The Semantic Scholar API exposes a graph of about 214M papers and 2.5B citation edges (free tier, plus SPECTER2 paper embeddings); OpenAlex indexes over 320M works and since 2026 wants an API key ($1/day of free usage); Connected Papers renders a similarity neighborhood around a seed paper using co-citation and bibliographic coupling, which works even for papers too new to have been cited. Our PageRank-with-gap-bonus is a toy of the same shape: authority from the link structure, priority from your questions.

Don't be confused: knowledge graph vs. vector index. The capstone's OpenSearch index answers similarity-shaped questions: "what is near this embedding?" A graph answers join-shaped questions: "which claims about Q2 come from sources that cite each other?" Similarity has no notion of contradicts or covers; edges do. GraphRAG uses both: vectors to find entry points, edges and communities to reason across them.

When is the graph overkill? When the sweep fits in your head. A three-day, ten-source question deserves a markdown table, not a schema. The graph starts paying at roughly the point where you catch yourself rereading notes to remember whether something was settled, and it becomes mandatory the day two claims quietly disagree and you ship the wrong one.

Deciding what to investigate next: the decision-theory view

The user asked the sharpest question last: does research benefit from framing as a Markov decision process? Honest answer first: no published canon treats literature research as an MDP, and you should distrust anyone who claims to run value iteration over their reading list. The framing is an analogy. But it is an analogy built on real, load-bearing formalisms (POMDPs make information-gathering actions part of an optimal policy; bandits and value-of-information are deployed daily), and the analogy computes: it turns "should I read more or just run the experiment?" from a mood into a number. Three labs, three sizes of the idea.

Research as an MDP: the escalation ladder, derived

An MDP is four things: states, actions, transition probabilities, and rewards. Model one research thread that way:

• State: your belief that the hypothesis is true, $P(\text{helps}) \in \{0.1, 0.3, 0.5, 0.7, 0.9\}$ ("image embeddings help our feed").
• Actions: desk research (cheap, noisy signal), an offline experiment (pricier, strong signal), or decide now (ship or drop, then stop).
• Transitions: a signal updates the belief by Bayes' rule, $$ P(\text{helps} \mid +) = \frac{p \cdot r}{p \cdot r + (1-p)(1-r)}, $$ where $r$ is the probability the signal points the right way.
• Rewards: shipping a real improvement +100, shipping a dud -60, dropping a real improvement -40 (opportunity cost), and each action's cost.

Value iteration (the same loop as any RL textbook, 20 lines here) then computes the best action in every belief state:

"""Research as a Markov decision process: value iteration derives the plan.

States are belief buckets for one hypothesis ("image embeddings help our
feed"): P(helps) in {0.1, 0.3, 0.5, 0.7, 0.9}. Actions are the moves a
researcher actually has, each with a cost and a signal quality, plus
"decide-now" (ship if EV(ship) > EV(drop), then stop).

Payoffs at decision time: shipping a real improvement +100, shipping a dud
-60, dropping a real improvement -40 (opportunity cost), dropping a dud 0.
Signals update the belief by Bayes' rule, snapped to the nearest bucket.
Value iteration then gives the optimal action in every belief state -- and
we solve it twice, because the answer depends on the price list.

Stdlib + NumPy only.
"""

import numpy as np

BELIEFS = np.array([0.1, 0.3, 0.5, 0.7, 0.9])
SHIP_WIN, SHIP_LOSE, DROP_WIN, DROP_LOSE = 100.0, -60.0, -40.0, 0.0
GAMMA = 0.99

CONFIGS = {  # action: (cost, P(signal is correct))
    "fresh problem: plenty to read": {
        "desk-research": (2.0, 0.65), "offline-exp": (8.0, 0.90)},
    "reading exhausted: desk barely informs": {
        "desk-research": (2.0, 0.55), "offline-exp": (8.0, 0.90)},
}


def decide_value(p):
    ev_ship = p * SHIP_WIN + (1 - p) * SHIP_LOSE
    ev_drop = p * DROP_WIN + (1 - p) * DROP_LOSE
    return max(ev_ship, ev_drop), ("ship" if ev_ship >= ev_drop else "drop")


def snap(p):
    return int(np.argmin(np.abs(BELIEFS - p)))


def posterior(p, r, positive):
    like_h = r if positive else (1 - r)          # P(signal | helps)
    like_n = (1 - r) if positive else r          # P(signal | not)
    return p * like_h / (p * like_h + (1 - p) * like_n)


def q_value(p, cost, r, V):
    p_pos = p * r + (1 - p) * (1 - r)
    return -cost + GAMMA * (p_pos * V[snap(posterior(p, r, True))]
                            + (1 - p_pos) * V[snap(posterior(p, r, False))])


def solve(actions):
    V = np.zeros(len(BELIEFS))
    for _ in range(500):
        newV = np.array([
            max(decide_value(p)[0],
                *(q_value(p, c, r, V) for c, r in actions.values()))
            for p in BELIEFS])
        if np.allclose(newV, V, atol=1e-10):
            break
        V = newV
    return V


def policy(V, actions):
    out = []
    for p in BELIEFS:
        dv, verdict = decide_value(p)
        best_name, best_q = f"decide-now ({verdict})", dv
        for name, (cost, r) in actions.items():
            q = q_value(p, cost, r, V)
            if q > best_q:
                best_name, best_q = name, q
        out.append((p, best_name, best_q))
    return out


if __name__ == "__main__":
    for label, actions in CONFIGS.items():
        V = solve(actions)
        print(f"config: {label}")
        print(f"  {'P(helps)':>8}  {'best action':<20} {'value':>7}")
        for p, name, q in policy(V, actions):
            print(f"  {p:>8.1f}  {name:<20} {q:>7.1f}")
        print()

    # one seeded rollout from the middle, following the first policy
    actions = CONFIGS["fresh problem: plenty to read"]
    V = solve(actions)
    pol = {p: name for p, name, _ in policy(V, actions)}
    rng = np.random.default_rng(3)
    helps = True                     # nature's hidden coin for this rollout
    i, spent, path = snap(0.5), 0.0, ["p=0.5"]
    while True:
        act = pol[BELIEFS[i]]
        if act.startswith("decide-now"):
            path.append(decide_value(BELIEFS[i])[1] + "!")
            break
        cost, r = actions[act]
        spent += cost
        positive = rng.random() < (r if helps else 1 - r)
        i = snap(posterior(BELIEFS[i], r, positive))
        path.append(f"{act}({'+' if positive else '-'}) -> p={BELIEFS[i]:.1f}")
    print(f"rollout (hypothesis secretly TRUE): {'  '.join(path)}")
    print(f"research spent: {spent:.0f}")

$ python3 research_mdp.py
config: fresh problem: plenty to read
  P(helps)  best action            value
       0.1  decide-now (drop)       -4.0
       0.3  desk-research            9.5
       0.5  desk-research           31.6
       0.7  desk-research           58.3
       0.9  decide-now (ship)       84.0

config: reading exhausted: desk barely informs
  P(helps)  best action            value
       0.1  decide-now (drop)       -4.0
       0.3  offline-exp              6.9
       0.5  offline-exp             31.6
       0.7  decide-now (ship)       52.0
       0.9  decide-now (ship)       84.0

rollout (hypothesis secretly TRUE): p=0.5  desk-research(+) -> p=0.7  desk-research(+) -> p=0.9  ship!
research spent: 4

The solver derives, rather than assumes, the folk wisdom:

• Decide at the extremes. At 0.1 and 0.9 no instrument is worth its cost; more research cannot change the decision, so it has no value. This is the formal version of "stop researching when the answer would not change what you do."
• Cheap instrument first, while it still informs. In the first config, desk research dominates everywhere in the middle: at cost 2 and 65% reliability, skimming beats experimenting.
• The ladder is an economic fact, not a virtue. Degrade desk research to 55% reliability (you have read everything useful; the next blog post tells you nothing) and the policy flips to the offline experiment, and the ship threshold drops to 0.7: when information is expensive, acting on decent odds beats buying more certainty. Re-derive the plan when the prices change; that is the entire benefit of writing them down.

The realistic caveat: your true state is not one number, signals are not independent coin flips, and belief buckets are a crutch. The formal name for "acting under a belief you cannot observe directly" is a POMDP (partially observable MDP), the setting where gathering information is itself an optimal action; that literature is real and deep (robotics uses it for search-and-track), and this lab is its five-state cartoon. Use the cartoon for what it is good at: making the decision structure explicit.

A portfolio of threads: bandits

The MDP above manages one hypothesis. A sprint usually carries several, and the allocation question ("which thread gets tomorrow?") is a multi-armed bandit: each thread is an arm, an hour of work is a pull, and a decision-grade finding is a win. The dilemma is explore vs. exploit: keep pulling the thread that has produced findings, or probe the neglected ones in case they are better?

Thompson sampling (1933, still the standard) solves it with one elegant move: keep a Beta posterior per arm, sample a plausible hit-rate from each posterior, work on the arm whose sample is highest. Arms with little data have wide posteriors and occasionally sample high (exploration happens automatically); arms with proven records sample high consistently (exploitation). research_bandit.py replays a 40-hour sprint 200 times, because one noisy sprint proves nothing either way:

"""Allocating research hours: round-robin vs. Thompson sampling, plus EVPI.

Part 1 treats four research threads as bandit arms. Pulling an arm = one
focused hour on that thread; a "success" = the hour produces a
decision-grade finding (a number, a reproduced result, a ruled-out option).
The true hit-rates are hidden from the scheduler, exactly as in real life.
One 40-hour sprint is noisy, so we replay the sprint over 200 seeds and
report averages; no single lucky run decides the verdict.

Part 2 prices a single experiment with the oldest tool in decision theory:
the expected value of perfect information (EVPI).

Stdlib + NumPy only. Seeded, so the output is reproducible.
"""

import numpy as np

THREADS = {  # hidden truth: P(one hour yields a decision-grade finding)
    "image-embeddings": 0.35,
    "half-life-sweep": 0.15,
    "dedup-gray-zone": 0.10,
    "two-tower-swap": 0.05,
}
BUDGET = 40   # research hours per sprint
SPRINTS = 200


def run(policy, seed):
    rng = np.random.default_rng(seed)
    truth = np.array(list(THREADS.values()))
    wins = np.zeros(len(truth))     # findings per thread
    pulls = np.zeros(len(truth))    # hours per thread
    for t in range(BUDGET):
        if policy == "round-robin":
            arm = t % len(truth)
        else:  # thompson: sample a plausible hit-rate per arm, take the max
            samples = rng.beta(wins + 1, pulls - wins + 1)
            arm = int(np.argmax(samples))
        hit = rng.random() < truth[arm]
        pulls[arm] += 1
        wins[arm] += hit
    return pulls, wins


if __name__ == "__main__":
    print(f"part 1: {BUDGET}h sprints over {len(THREADS)} threads, "
          f"averaged across {SPRINTS} replays\n")
    for policy in ("round-robin", "thompson"):
        P = np.zeros(len(THREADS))
        W = np.zeros(len(THREADS))
        for seed in range(SPRINTS):
            pulls, wins = run(policy, seed)
            P += pulls
            W += wins
        P /= SPRINTS
        W /= SPRINTS
        print(f"{policy}: {W.sum():.1f} findings per sprint (mean)")
        for name, rate, p, w in zip(THREADS, THREADS.values(), P, W):
            bar = "#" * round(p)
            print(f"  {name:<18} true {rate:.2f}  {p:>4.1f}h "
                  f"{bar:<22} {w:.1f} finding(s)")
        print()

    # ---------------------------------------------------- part 2: EVPI
    # Decision: ship image embeddings into the profile, or not?
    # If they help (prior belief p), shipping is worth +80 (k$/yr of
    # engagement value); if they don't, shipping costs -20 (serving +
    # complexity). Not shipping is always 0.
    p, up, down = 0.6, 80.0, -20.0
    ev_ship = p * up + (1 - p) * down
    best_now = max(ev_ship, 0.0)
    # A perfect oracle: ship only when it truly helps.
    ev_oracle = p * up + (1 - p) * 0.0
    evpi = ev_oracle - best_now
    print("part 2: what is the experiment worth? (EVPI)")
    print(f"  prior P(helps) = {p:.1f};  ship now EV = {ev_ship:+.1f};  "
          f"skip EV = +0.0")
    print(f"  best decision without information: ship  (EV {best_now:+.1f})")
    print(f"  with a perfect experiment first:   EV {ev_oracle:+.1f}")
    print(f"  -> EVPI = {evpi:+.1f}  (spend up to this on the experiment; "
          f"a week of offline\n     ablation costs ~2, a full A/B costs ~25 "
          f"-- run the ablation, skip the A/B)")

$ python3 research_bandit.py
part 1: 40h sprints over 4 threads, averaged across 200 replays

round-robin: 6.5 findings per sprint (mean)
  image-embeddings   true 0.35  10.0h ##########             3.5 finding(s)
  half-life-sweep    true 0.15  10.0h ##########             1.6 finding(s)
  dedup-gray-zone    true 0.10  10.0h ##########             1.0 finding(s)
  two-tower-swap     true 0.05  10.0h ##########             0.5 finding(s)

thompson: 8.2 findings per sprint (mean)
  image-embeddings   true 0.35  17.3h #################      5.8 finding(s)
  half-life-sweep    true 0.15   8.8h #########              1.3 finding(s)
  dedup-gray-zone    true 0.10   7.4h #######                0.7 finding(s)
  two-tower-swap     true 0.05   6.5h #######                0.3 finding(s)

part 2: what is the experiment worth? (EVPI)
  prior P(helps) = 0.6;  ship now EV = +40.0;  skip EV = +0.0
  best decision without information: ship  (EV +40.0)
  with a perfect experiment first:   EV +48.0
  -> EVPI = +8.0  (spend up to this on the experiment; a week of offline
     ablation costs ~2, a full A/B costs ~25 -- run the ablation, skip the A/B)

Round-robin spends a quarter of the sprint on the 5% thread out of fairness; Thompson notices within a few pulls that image-embeddings produces and shifts hours there, for 26% more findings from the same budget. It never fully abandons the weak arms (6.5 hours still went to two-tower-swap on average): that residual exploration is the posterior's uncertainty doing its job, insurance against a slow-starting arm that is secretly good.

Two details make this more than a metaphor for a recommender team. First, bandits are not exotic here: this exact algorithm family is production recsys machinery. LinUCB ran Yahoo's front-page news module in 2010, and Netflix picks per-title artwork with contextual bandits; your product already exploits and explores. Pointing the same mathematics at your own research calendar is the least surprising reuse imaginable. Second, the same "true hit-rates are hidden" honesty applies to you: nobody hands you the 0.35. You act on posteriors, and the discipline is simply to update them (count findings per thread per week) instead of allocating by enthusiasm.

Part 2 is the oldest tool in this chapter, Howard's 1966 expected value of perfect information. Before running any experiment, price it:

$$ \text{EVPI} = \mathbb{E}[\text{best decision with the answer}] - \mathbb{E}[\text{best decision now}]. $$

Here the prior already says ship (EV +40), a perfect experiment lifts the expected outcome only to +48, so information is worth at most 8. The offline ablation (cost ~2) clears that bar easily; the full A/B (cost ~25) is worth negative 17 even if it were perfect. The pattern generalizes: the closer your prior is to the decision boundary, the more an experiment is worth; far from the boundary, testing is theater. Teams run expensive tests on foregone conclusions constantly; one EVPI line in the research plan kills those.

Don't be confused: the bandit lab allocates your hours across research threads; the A/B tests and interleaving of Chapter 3 and the capstone allocate user traffic across models. Same explore/exploit mathematics, two different scarce resources. Netflix uses interleaving to prune candidate rankers in days before spending months of A/B traffic on the survivors: that is a bandit-shaped research pipeline operating on users.

Which tool when

Broader topics worth a look

Each of these is one step beyond this chapter, with a canonical entry point (all in References):

• Active learning (Settles' survey): the same "query what is most informative" idea applied to labeling; directly useful when building the golden sets of Chapter 29.
• Bayesian optimization (Snoek et al.; Frazier's tutorial): VoI industrialized for expensive experiments; the knowledge-gradient acquisition function literally maximizes expected value of information per measurement. The natural next step when the half-life sweep of Chapter 31 gets expensive.
• Interleaving (Chapelle et al.; Netflix): rank-sensitive online evaluation that reaches significance with far less traffic than A/B; the production analog of "cheap instrument first."
• Trustworthy online experiments (Kohavi, Tang & Xu): the book on not fooling yourself once research graduates to A/B tests.
• Tree search over reasoning (Tree of Thoughts; LATS): agents that explore and backtrack over intermediate steps, MDP thinking applied inside a single model's deliberation.
• Claim verification (SciFact): datasets and models for SUPPORTS/REFUTES over scientific claims; the adversarial-verification pattern, benchmarked.

Where this leaves the research model

The arc of this part mirrors the arc of the whole book. The recommender needed a spec before code (Chapter 30), a method to produce it (Chapter 31), a research skill at the head of that method (Chapter 32), evidence riding down the chain (Chapter 33), and mechanics that keep the evidence honest (this chapter): parallel agents where work decomposes, a graph where claims accumulate, and a little decision theory where the budget runs out. The same three moves that built the feed, pointed one level up, at the process that decides what to build next.

Two questions remain before this part closes. This chapter and the last treated hosted research engines as interchangeable gatherers; the next one stops being polite about that, comparing them on artifacts, automation, benchmarks, and above all on what each does with the questions you type, and where those questions end up. And then one final chapter leaves the codebase entirely, for the research domains no engineer sees coming. 👉