Benchmarking BMAD: measuring every dimension

You cannot improve what you do not measure, and you cannot trust a method, or any of the tools from the previous page, on its marketing. This page is how to benchmark BMAD across every dimension a subject-matter expert cares about: token utilization, cache and context efficiency, cost, speed, accuracy, response quality, rework, human effort, and the one metric that actually decides whether the method pays for itself. It builds on the measurement scorecard from page 7 and the levers from page 14, and it hosts the guide's one runnable lab.

On this page

• The honest starting point
• The dimensions, and how to measure each
• Two kinds of metric, and why the distinction matters
• The instruments: what to actually run
• The protocol: A/B done right
• The harness
• Reading the results: BMAD’s actual shape
• Running the benchmark with Claude Code
• Hands-on: capture one real trial and score it
• Vanity metrics and Goodhart’s law
• Pitfalls of benchmarking

The honest starting point

Two facts have to sit at the front, because ignoring either produces numbers that lie.

There is no rigorous, independent public BMAD benchmark. The claims you will find (BMAD's own, and every memory or code tool from the last page) are anecdotal or vendor-authored. Distrust any single number, including your own first run.

Benchmarking an agentic method is genuinely hard, for three reasons that never go away:

• Non-determinism. The same prompt yields different outputs run to run, so one trial is a data point, not a verdict.
• One-shot tasks. You cannot re-run a real project; the second time, the team and the agent both know the answer.
• Confounds. Model version, task difficulty, and the human in the loop all move the result independently of the method.

So a benchmark is an A/B comparison on matched tasks, with repeated trials and honest error bars, not a single figure. Everything below is built to survive those three problems.

The dimensions, and how to measure each

Two kinds of metric, and why the distinction matters

Deterministic metrics you can trust straight away, tokens, cost, wall-clock, cache-hit, git-derived rework, come from usage blocks, timestamps, and git. Read them directly; no judgment involved.

Judged metrics are noisy and need a fixed rubric, accuracy against acceptance criteria, and response quality. Grade them with a stable rubric and a blind judge (a claude-opus-4-8 structured-output call that scores the artifact without being told which condition produced it), the same LLM-judge pattern the recon skill uses for verification. If the grader can see the condition, the score is contaminated.

The instruments: what to actually run

• Tokens and cost: count_tokens before a workflow, the usage block after, or the transcript ledger for a whole session. Multiply by a dated price sheet.
• Cache: compare cache_read to fresh input in the usage block; that ratio is your context-reuse efficiency.
• Research: recon_kit tally counts claims by status straight from the ledger, so research coverage is a number, not a vibe (page 11).
• Rework: git log --stat, the baseline_revision..final_revision bracket a dev-auto run records, revert count, and line churn. This is the most honest quality proxy you have, because redo is expensive and visible.
• Speed: artifact and commit timestamps; CI duration.
• Accuracy: the qa-generate-e2e-tests suite pass rate and the acceptance- criteria checklist from the story.
• Quality: the blind rubric judge over the readiness and code-review artifacts.

If your host is Claude Code, all of these become one repeatable command; the section Running the benchmark with Claude Code below turns each instrument into an exact recipe.

The protocol: A/B done right

1. Pick matched tasks representative of your real work: a feature, an epic, a bugfix. Same tasks for both conditions.
2. Two conditions. BMAD (the track you would actually use) versus a baseline: ad-hoc "vibe" coding, or BMAD track X versus track Y when you are tuning ceremony.
3. Hold constant everything you can: model, effort level, a fresh context per workflow, pinned tool and module versions.
4. Repeat. Run k trials per cell and randomize the order, so learning and time-of-day do not masquerade as the method.
5. Blind the grader to the condition.
6. Report deltas with spread and a matched-pair sign test, never a point estimate.
7. Mind the baseline problem. The honest baseline is "the same team without the method," which you almost never get cleanly. The usable proxies are track A versus track B, or method-on versus method-off for a sprint.

The harness

A from-scratch, stdlib-only harness that takes matched trials and reports every dimension, plus cost-per-correct-outcome and a matched-pair verdict. Drop your own measured trials into TRIALS and it does the arithmetic. The fixture shipped with it is synthetic and illustrative (clearly labeled so), because the value here is the method, not invented scores.

"""
benchmark_harness.py  —  a from-scratch A/B scorecard for a spec-driven method.

Pure stdlib. It ingests a set of matched *trials* (the same task run under two
conditions, "bmad" and "adhoc") and reports every dimension a subject-matter
expert cares about: token utilization, cache/context efficiency, cost, speed,
accuracy, response quality, rework, and human effort. It also computes the metric
that actually decides the trade, cost per *correct* outcome, and flags whether a
difference is a real signal or just noise by checking whether the two conditions'
observed ranges overlap.

IMPORTANT: the TRIALS below are a small, SYNTHETIC teaching fixture, not measured
BMAD results. There is no rigorous public BMAD benchmark; the point of this file
is the *method* and the harness, so you can drop your own measured trials in and
get an honest comparison. Treat every number in the fixture as illustrative.
"""

from statistics import mean

# --- price sheet (Claude Opus 4.8, USD per token; dated, edit for your model) ---
PRICE = {
    "in":     5.00 / 1_000_000,   # base input
    "out":   25.00 / 1_000_000,   # output
    "cread":  0.50 / 1_000_000,   # cache read  (~0.1x input)
    "cwrite": 6.25 / 1_000_000,   # cache write (~1.25x input, 5-minute TTL)
}

# --- the fixture: matched tasks, two conditions, repeated trials -----------------
# tokens are absolute counts; wall is wall-clock minutes; tests is (passed, total);
# rework is redo commits; quality is a 0-5 rubric; approvals is human gate touches.
TRIALS = [
    # task, cond, in, out, cread, cwrite, wall, (passed,total), rework, quality, approvals
    ("auth-feature",  "bmad",  180_000,  95_000, 1_200_000, 220_000, 210, (24, 24), 1, 4.6, 6),
    ("auth-feature",  "bmad",  175_000,  90_000, 1_150_000, 210_000, 195, (23, 24), 2, 4.4, 6),
    ("auth-feature",  "adhoc", 140_000, 120_000,    90_000,  40_000, 150, (18, 24), 7, 3.2, 1),
    ("auth-feature",  "adhoc", 150_000, 135_000,    80_000,  45_000, 165, (17, 24), 9, 3.0, 1),

    ("billing-epic",  "bmad",  320_000, 160_000, 2_400_000, 380_000, 430, (41, 42), 3, 4.5, 11),
    ("billing-epic",  "bmad",  310_000, 155_000, 2_300_000, 360_000, 410, (42, 42), 2, 4.7, 11),
    ("billing-epic",  "adhoc", 260_000, 240_000,   160_000,  70_000, 360, (28, 42), 18, 2.8, 2),
    ("billing-epic",  "adhoc", 275_000, 255_000,   150_000,  75_000, 380, (31, 42), 15, 3.0, 2),

    ("typo-fix",      "bmad",   14_000,   3_000,    60_000,  12_000,  12, (2, 2), 0, 4.8, 1),
    ("typo-fix",      "bmad",   13_500,   2_800,    58_000,  11_500,  11, (2, 2), 0, 4.8, 1),
    ("typo-fix",      "adhoc",   6_000,   1_500,     2_000,   1_000,   4, (2, 2), 0, 4.7, 0),
    ("typo-fix",      "adhoc",   6_500,   1_400,     1_800,     900,   5, (2, 2), 0, 4.6, 0),
]

FIELDS = ["task", "cond", "in", "out", "cread", "cwrite", "wall", "tests",
          "rework", "quality", "approvals"]


def row(t):
    return dict(zip(FIELDS, t))


def cost(r):
    return (r["in"] * PRICE["in"] + r["out"] * PRICE["out"]
            + r["cread"] * PRICE["cread"] + r["cwrite"] * PRICE["cwrite"])


def total_tokens(r):
    return r["in"] + r["out"] + r["cread"] + r["cwrite"]


def cache_hit(r):
    # share of *input* that was served from cache: context-reuse efficiency
    served = r["cread"]
    fresh = r["in"] + r["cwrite"]
    return served / (served + fresh) if (served + fresh) else 0.0


def accuracy(r):
    p, tot = r["tests"]
    return p / tot if tot else 0.0


# metric -> (extractor, "higher"|"lower" is better, formatter)
METRICS = [
    ("tokens/task (k)",     lambda r: total_tokens(r) / 1000, "info",  "{:.0f}"),
    ("cache hit ratio",     cache_hit,                        "higher", "{:.0%}"),
    ("cost/task ($)",       cost,                             "lower",  "{:.2f}"),
    ("wall-clock (min)",    lambda r: r["wall"],              "lower",  "{:.0f}"),
    ("accuracy (tests)",    accuracy,                         "higher", "{:.0%}"),
    ("quality (0-5)",       lambda r: r["quality"],           "higher", "{:.2f}"),
    ("rework (commits)",    lambda r: r["rework"],            "lower",  "{:.1f}"),
    ("human approvals",     lambda r: r["approvals"],         "lower",  "{:.1f}"),
]


def cond_mean(rows, cond, fn):
    vals = [fn(r) for r in rows if r["cond"] == cond]
    return mean(vals) if vals else 0.0


def task_list(rows):
    seen = []
    for r in rows:
        if r["task"] not in seen:
            seen.append(r["task"])
    return seen


def winner(a_mean, b_mean, direction):
    if direction == "info" or a_mean == b_mean:
        return "-"
    a_better = (a_mean > b_mean) if direction == "higher" else (a_mean < b_mean)
    return "bmad" if a_better else "adhoc"


def consistency(rows, fn, direction, champ):
    # matched-pair sign test: on how many tasks does the pooled winner also win?
    # tasks differ in scale, so pooling ranges is misleading; agreement is not.
    agree = total = 0
    for task in task_list(rows):
        b = [fn(r) for r in rows if r["task"] == task and r["cond"] == "bmad"]
        a = [fn(r) for r in rows if r["task"] == task and r["cond"] == "adhoc"]
        if not b or not a:
            continue
        total += 1
        bm, am = mean(b), mean(a)
        if bm == am:
            continue
        task_champ = "bmad" if ((bm > am) == (direction == "higher")) else "adhoc"
        if task_champ == champ:
            agree += 1
    return agree, total


def scorecard(rows):
    print("Dimension            bmad        adhoc       winner   agree")
    print("-" * 60)
    for name, fn, direction, fmt in METRICS:
        a = cond_mean(rows, "bmad", fn)
        b = cond_mean(rows, "adhoc", fn)
        w = winner(a, b, direction)
        if direction == "info" or w == "-":
            tag = "-"
        else:
            ag, tot = consistency(rows, fn, direction, w)
            tag = "{}/{}".format(ag, tot)
        print("{:<20} {:>10} {:>11}   {:<8} {}".format(
            name, fmt.format(a), fmt.format(b), w, tag))


def cost_per_correct(rows):
    print("\nCost per *correct* outcome, by task (the metric that decides it)")
    print("-" * 62)
    print("Task            bmad $/correct   adhoc $/correct   pays back?")
    print("-" * 62)
    tasks = []
    for r in rows:
        if r["task"] not in tasks:
            tasks.append(r["task"])
    for task in tasks:
        out = {}
        for cond in ("bmad", "adhoc"):
            trs = [r for r in rows if r["task"] == task and r["cond"] == cond]
            total_cost = sum(cost(r) for r in trs)
            correct = sum(accuracy(r) for r in trs)  # expected tasks-worth passing
            out[cond] = total_cost / correct if correct else float("inf")
        pays = "yes" if out["bmad"] < out["adhoc"] else "no"
        print("{:<15} {:>13.2f}   {:>15.2f}   {}".format(
            task, out["bmad"], out["adhoc"], pays))


if __name__ == "__main__":
    rows = [row(t) for t in TRIALS]
    print("A/B benchmark: spec-driven (bmad) vs ad-hoc (adhoc)")
    print("Fixture: {} trials over {} tasks (SYNTHETIC, illustrative)\n".format(
        len(rows), len({r["task"] for r in rows})))
    scorecard(rows)
    cost_per_correct(rows)

Running it prints the scorecard and the decider table:

A/B benchmark: spec-driven (bmad) vs ad-hoc (adhoc)
Fixture: 12 trials over 3 tasks (SYNTHETIC, illustrative)

Dimension            bmad        adhoc       winner   agree
------------------------------------------------------------
tokens/task (k)            1647         384   -        -
cache hit ratio             74%         28%   bmad     3/3
cost/task ($)              4.79        4.12   adhoc    3/3
wall-clock (min)            211         177   adhoc    3/3
accuracy (tests)            99%         81%   bmad     2/3
quality (0-5)              4.63        3.55   bmad     3/3
rework (commits)            1.3         8.2   bmad     2/3
human approvals             6.0         1.0   adhoc    3/3

Cost per *correct* outcome, by task (the metric that decides it)
--------------------------------------------------------------
Task            bmad $/correct   adhoc $/correct   pays back?
--------------------------------------------------------------
auth-feature             5.24              5.79   yes
billing-epic             9.11             11.47   yes
typo-fix                 0.24              0.07   no

The agree column is a matched-pair sign test: on how many of the tasks the pooled winner also wins. It replaces a naive range check, because pooling a two-line fix with a multi-epic feature inflates the spread and would hide real differences. The 2/3 on accuracy and rework is the trivial task tying (both perfect, zero redo), which is the honest result, not a defect.

Reading the results: BMAD's actual shape

Read the (synthetic) output as the shape to expect and then verify on your own work:

• BMAD spends more raw tokens (about 4x here) but most are cheap cache reads (74% hit versus 28%), because it re-reads a stable project-context.md and the artifact chain each workflow. So the 4x token count is only a modest cost bump.
• BMAD wins where quality lives: rubric quality (3/3), accuracy (2/3), and rework (2/3, tied only on the trivial task). Fewer redo commits is the clearest signal, because redo is expensive and hard to fake.
• Ad-hoc wins wall-clock and human approvals, both 3/3. Ceremony has a real human-latency cost, the gates, and pretending otherwise is dishonest.
• The decider is cost per correct outcome. BMAD pays back on the feature and the epic and does not on the typo-fix, where ad-hoc is several times cheaper. That single row is the entire argument for Quick Flow: do not run the full method on a one-liner.

The trade in one sentence: BMAD buys lower rework and higher first-pass correctness with higher up-front spend and more gate latency; it wins when the task is large enough for rework to dominate, and loses when it is not. Your benchmark job is to find, for your work, where that crossover sits.

Running the benchmark with Claude Code

The harness above needs real trials, and if BMAD runs inside Claude Code, that is where you generate them. Claude Code's headless mode turns one A/B cell into one command whose JSON output already carries the cost and token numbers, so a shell loop produces the rows you feed to benchmark_harness.py. The flags below are current as of Claude Code v2.1.x; treat them as dated and check claude --help, since the CLI moves.

Instrument a single run. claude -p (print/headless) runs non-interactively; --output-format json returns a structured result instead of prose.

time claude -p "implement STORY-42 per docs/stories/STORY-42.md" \
  --model claude-opus-4-8 --effort xhigh --bare \
  --allowedTools "Read,Edit,Bash" --permission-mode acceptEdits \
  --max-budget-usd 5 --output-format json > run.json

Two flags matter for honest measurement here. --effort is the thinking dial from page 12 (low/medium/high/xhigh/ max), so you benchmark a specific effort, not a default. --bare strips hooks, skills, MCP servers, auto-memory, and CLAUDE.md from the run, which removes confounds you did not mean to measure; keep it on and hold it constant across cells (turn it off only when the thing you are benchmarking is one of those surfaces). --max-budget-usd caps a runaway trial so one outlier cannot skew the batch.

The JSON result carries the deterministic metrics directly (field names as documented; parse with jq):

jq '{cost: .total_cost_usd, usage: .usage_by_model}' run.json

The shape you get back (illustrative, one entry per model queried):

{
  "result": "...response text...",
  "session_id": "b3f1...",
  "total_cost_usd": 4.79,
  "usage_by_model": {
    "claude-opus-4-8": {
      "input_tokens": 180000,
      "output_tokens": 95000,
      "cache_read_input_tokens": 1200000,
      "cache_creation_input_tokens": 220000,
      "cost": 4.79
    }
  }
}

Those four token fields map one-to-one onto the harness row (in, out, cread, cwrite), and total_cost_usd cross-checks the harness's own price-sheet math. One caveat: total_cost_usd is computed at list rates and ignores any discount, so for billing truth use the Console usage page; for benchmarking, list rates are the right apples-to-apples number as long as both conditions use them.

Speed. The published JSON does not guarantee a wall-clock field, so time the process yourself (time, or a date delta around the call). In an interactive session, /usage reports both API and wall duration plus per-model tokens, and /context shows what is filling the window.

A matched A/B loop. Run the same task under two conditions, k trials each, and append one JSONL row per run:

for cond in bmad adhoc; do
  for i in 1 2 3; do
    start=$(date +%s)
    claude -p "$(cat prompt_$cond.md)" \
      --model claude-opus-4-8 --effort xhigh --bare \
      --output-format json > /tmp/run.json
    wall=$(( $(date +%s) - start ))
    jq -c --arg cond "$cond" --argjson wall "$wall" \
      '{cond:$cond, wall:$wall, cost:.total_cost_usd, u:.usage_by_model}' \
      /tmp/run.json >> trials.jsonl
  done
done

Feed trials.jsonl to the harness by replacing its synthetic TRIALS fixture with a few-line loader that reads each row and maps the fields; everything downstream (the scorecard, the matched-pair verdict, cost-per-correct) is unchanged.

The judged metrics need two more steps per run. After a trial finishes:

• Accuracy: run the story's tests and capture pass/total, e.g. pytest -q | tail -1, and write it into the row.

• Quality, judged blind: score the change with a separate headless call, a different model so the judge is independent, fed the diff without the condition label, and constrained to structured output:

  git diff main...HEAD > change.diff
  claude -p "Score this diff 0-5 on correctness, clarity, and test coverage. \
    Return JSON matching the schema." \
    --model claude-sonnet-5 --bare \
    --json-schema rubric.schema.json --output-format json < change.diff \
    | jq '.structured_output'
  

  --json-schema validates the judge's output and returns it under structured_output; a different --model and a label-free prompt are what keep the score honest (the blind rubric judge from earlier, made concrete).

Two dimensions Claude Code will not hand you for free. Rework comes from git, not the run (git log, revert count, churn over baseline..final). And human approvals vanish in headless mode, because --permission-mode acceptEdits (or bypassPermissions) removes the human, so a fully headless A/B measures the autonomous path and reports zero gate latency. To benchmark the gated workflow's human cost, run that arm interactively and count the approvals, or model the gate latency separately; do not let "headless was faster" quietly drop the dimension that headless deleted.

Reproducibility knobs. Use a fresh session per trial (the default) so runs do not contaminate each other, matching the fresh-context discipline from page 14; --session-id <uuid> pins an id when you need to find a transcript later, and --no-session-persistence skips writing one. For per-turn token granularity beyond the summary JSON, enable OpenTelemetry (CLAUDE_CODE_ENABLE_TELEMETRY=1 with an OTLP endpoint) and read the claude_code.token.usage metric. Do not parse the ~/.claude/projects/… transcript JSONL for numbers; that format is internal and shifts between releases, which is exactly the kind of silent breakage a benchmark must avoid.

Hands-on: capture one real trial and score it

The harness is real; its fixture is synthetic. Here is the loop that swaps one synthetic cell for numbers you measured, end to end, on a single story. Nothing in this section is verified on this box (BMAD and Claude Code are not installed); run it in your own repo.

Pick one story that has a test suite. Write two prompts for the same task: prompt_bmad.md (the spec-driven track) and prompt_adhoc.md (a bare "just build it"). Run each arm once, timed, headless:

time claude -p "$(cat prompt_bmad.md)" \
  --model claude-opus-4-8 --bare --output-format json > run.json

time prints the wall-clock to stderr; keep that number. Pull the deterministic metrics out of the result:

jq '{cost:.total_cost_usd, usage:.usage_by_model}' run.json

Illustrative (run it in your own repo; not machine-verified here).

{
  "cost": 4.79,
  "usage": {
    "claude-opus-4-8": {
      "input_tokens": 180000,
      "output_tokens": 95000,
      "cache_read_input_tokens": 1200000,
      "cache_creation_input_tokens": 220000
    }
  }
}

Those four token fields are the harness's in, out, cread, cwrite, and total_cost_usd cross-checks its price-sheet math. Now the metric headless will not hand you: run the story's tests and record pass/total.

pytest -q | tail -1   # "47 passed" -> tests 47/47

Append one JSONL row per trial, then repeat both commands for prompt_adhoc.md:

jq -c '{cond:"bmad", cost:.total_cost_usd, u:.usage_by_model, tests:"47/47", wall:198}' \
  run.json >> trials.jsonl

Point the few-line loader from the section above at trials.jsonl (it replaces the synthetic TRIALS), then print the scorecard:

python3 benchmark_harness.py

You get the same scorecard and cost-per-correct-outcome table shown earlier, now over your two real rows. One trial per arm is a data point, not a verdict; loop each arm three times before you trust the agree column. The point is that the numbers came off your own machine.

Vanity metrics and Goodhart's law

• Tokens-saved is a vanity metric if rework rises. Measure cost per correct outcome, not cost. A run that is 30% cheaper and redone twice is a loss.
• Do not optimize the metric; optimize the outcome. The moment a number becomes a target it stops measuring what it measured (Goodhart). Cross-check any single metric against a second one that would move the opposite way if you were gaming it.
• Speed without accuracy is fast wrong answers. Never report latency alone.

Pitfalls of benchmarking

• Single-run conclusions. Non-determinism means one trial proves nothing; repeat and report spread.
• Unmatched tasks. Different difficulty across conditions is the most common way a benchmark lies to you.
• Grader leakage. A judge that knows the condition scores the label, not the work.
• Cross-version comparisons. A model upgrade between conditions is a confound; pin the model.
• Means without spread. A point estimate hides whether the difference is real.
• Trusting vendor numbers, including BMAD's and every tool's. Regenerate them yourself with this harness before you quote them.

Benchmarking is what turns "the method feels better" into a number you can defend and a crossover you can act on. The next page is the operations cookbook, the exact recipes for the work whose cost and quality you now know how to measure. 👉