Cost engineering: tokens as an SLO

Every chapter of this book has touched cost: caching in Part 2, fleet arithmetic in Part 6, the token ledger in Part 9's opening. This chapter assembles them into a discipline. Agent platforms have a cost profile unlike ordinary software, spend scales with usage through token consumption, not with a fixed server bill, so a single bad prompt change can multiply the invoice with no code getting slower and no alarm firing. Treating cost as an engineering property with budgets, gates, and dashboards, rather than a monthly surprise, is what keeps an agent platform financially operable, and it is the last piece of gate 3.

The cost model, in one equation

Where does the money go? For an agent platform, almost entirely into tokens, and the shape is worth stating plainly:

run cost ≈  Σ over turns [ fresh_input × in_price
                         + cached_input × in_price × 0.1
                         + output × out_price ]

Three levers fall out, and every cost intervention in the book is one of them:

Fewer tokens per turn. Shorter contexts, pruned transcripts, context management so old tool results do not ride along forever.
More of the input cached. The caching discipline: freeze the prefix, grow append-only, and the bulk of input bills at a tenth. This is usually the largest single lever, which is why the cache-hit ratio is a dashboard gauge.
Cheaper tokens. The model portfolio (small models for small jobs) and the batch tier (half price for offline work). Right-model and right-tier per role.

Notice these are the same levers the whole book developed; cost engineering is not new machinery, it is the discipline of measuring these levers and defending them.

Budgets: the enforced kind

The scheduler already made budgets real at admission: a run is admitted only if its remaining budget covers the projected cost, and it lands as run.exhausted rather than as a surprise invoice. Cost engineering adds the tiers around that single mechanism:

Per-run budgets bound one job, the admission check from Part 6.
Per-tenant budgets bound a customer, the multi-tenant cap that stops one tenant spending the shared pool.
Per-fleet budgets bound a batch: the overnight audit gets N tokens, and it stops at N with partial results, not at the invoice.

The unifying principle is that a budget is admission control, not accounting. Accounting tells you what you spent after the fact; admission control refuses to spend past the cap in the first place. The ledger enforcing the budget lives in the scheduler, so no prompt and no model behavior can overspend it, which is the only design that holds when the thing spending the money is a model you do not fully control.

Lab 9.2: the cost regression gate

Budgets stop a run overspending; they do nothing about a change that makes every future run cost more. That is a regression, and Part 9's eval discipline handles regressions with a gate. The cost gate treats tokens-per-task exactly as the eval suite treats quality: a committed baseline, a candidate run, a pass/fail on drift:

python3 cost_gate.py

=== candidate: prompt-cache tuning ===
    total tokens/task 50,800 -> 41,656 (-18%)
    GATE PASS (exit 0) -- saved 18%

=== candidate: new verbose system prompt ===
    total tokens/task 50,800 -> 53,300 (+5%)
    GATE FAIL (exit 1):
      - task 'classify-ticket' up +42% (limit +25%)

in CI, candidate 2's non-zero exit blocks the merge before the cost ships. its aggregate (+5%) sits UNDER the 10% limit and would have passed; the per-task rule is what caught the +42% on 'classify-ticket' (fixed prompt bloat hurts small tasks most).

Two lessons, one obvious and one not. The obvious one: a cost change can now fail a merge the same way a broken test does. The caching improvement passes and reports its 18% saving; the verbose-system-prompt regression exits non-zero and blocks. Cost stopped being a thing you discover on the invoice and became a thing CI catches before it ships, which is the whole point.

The non-obvious lesson is in why the per-task rule exists alongside the aggregate one. The regression's aggregate was +5%, comfortably under the 10% limit, so an aggregate-only gate would have waved it through. But one task, classify-ticket, jumped 42%, because a fixed system-prompt bloat is a large fraction of a small task and a tiny fraction of a big one, so averages hide it and per-task checks expose it. This is the same tails-versus-averages lesson as the warm-pool p99 and the pool-exhaustion 429 storm: the aggregate is where regressions go to hide, and the per-item view is where you find them. A cost gate that checks only the total is a cost gate with a blind spot exactly the shape of your highest-volume small task.

Showback: attributing the spend

The last piece is answering "who spent this," because a platform whose cost is one undifferentiated line item cannot be optimized or governed. The token ledger, tagged per run, per tenant, per team, and per change, turns the monthly total into an attributed breakdown, showback: this tenant cost this much, this feature cost that much, this prompt change moved the needle this way. Showback is what makes cost a shared engineering concern rather than a surprise the finance team escalates: the team that owns the expensive agent sees its own number, the tenant nearing its budget gets a warning not a shutdown, and a change that moved the cost curve is attributable to the commit that moved it. Attribution is cheap (it is the ledger you already emit, keyed by tags) and its absence is expensive (cost you cannot attribute is cost you cannot manage).

Don't be confused: cost vs latency. They usually move together (a longer transcript costs more tokens and takes longer) but they are optimized against different limits and can diverge. The batch tier is the sharp example: it cuts cost in half while making latency worse by hours, a great trade for an overnight audit and a terrible one for an interactive agent. Caching, by contrast, improves both. Know which you are optimizing: an interactive product defends latency and pays for it; an offline fleet defends cost and waits. A dashboard that tracks only one will cheerfully let you wreck the other.

Full source

"""Lab 9.2: the cost regression gate. A CI test for tokens-per-task.

A prompt, tool, or model change can quietly multiply what every run
costs. This gate treats cost like tests treat correctness: a committed
baseline of tokens-per-task, a run of the golden task set on the
candidate, and a pass/fail on drift, per task and in aggregate. In CI
it exits non-zero to block a merge.

The demo runs the gate against two candidates: a caching improvement
(passes, and reports the saving) and a system-prompt bloat regression
(fails, and names the offending tasks). Standard library only.
Deterministic.
"""

from __future__ import annotations

# committed baseline: tokens per golden task, from the last accepted run
BASELINE = {
    "extract-invoice":   12_000,
    "triage-alert":       8_500,
    "summarize-thread":   6_200,
    "classify-ticket":    3_100,
    "audit-shard":       21_000,
}

AGG_THRESHOLD = 0.10      # fail if total tokens rise > 10%
TASK_THRESHOLD = 0.25     # or if any single task rises > 25%


def gate(candidate: dict[str, int]) -> tuple[bool, list[str]]:
    reasons = []
    base_total = sum(BASELINE.values())
    cand_total = sum(candidate.values())
    agg = (cand_total - base_total) / base_total
    if agg > AGG_THRESHOLD:
        reasons.append(f"aggregate tokens/task up {agg:+.0%} "
                       f"(limit +{AGG_THRESHOLD:.0%})")
    for task, base in BASELINE.items():
        delta = (candidate[task] - base) / base
        if delta > TASK_THRESHOLD:
            reasons.append(f"task '{task}' up {delta:+.0%} "
                           f"(limit +{TASK_THRESHOLD:.0%})")
    return (len(reasons) == 0), reasons


def report(name: str, candidate: dict[str, int]) -> bool:
    passed, reasons = gate(candidate)
    base_total, cand_total = sum(BASELINE.values()), sum(candidate.values())
    agg = (cand_total - base_total) / base_total
    print(f"=== candidate: {name} ===")
    print(f"    total tokens/task {base_total:,} -> {cand_total:,} "
          f"({agg:+.0%})")
    if passed:
        print(f"    GATE PASS (exit 0)"
              + (f" -- saved {-agg:.0%}" if agg < 0 else ""))
    else:
        print(f"    GATE FAIL (exit 1):")
        for r in reasons:
            print(f"      - {r}")
    return passed


if __name__ == "__main__":
    # candidate 1: a prompt-cache improvement shaves fresh input everywhere
    improved = {k: int(v * 0.82) for k, v in BASELINE.items()}
    # candidate 2: a bloated system prompt inflates every task, worst on
    # the short ones (fixed overhead hurts small tasks most)
    regressed = {
        "extract-invoice":   12_300,
        "triage-alert":       8_700,
        "summarize-thread":   6_400,
        "classify-ticket":    4_400,   # +42%: fixed bloat, small task
        "audit-shard":       21_500,
    }

    ok1 = report("prompt-cache tuning", improved)
    print()
    ok2 = report("new verbose system prompt", regressed)

    cand_agg = (sum(regressed.values()) - sum(BASELINE.values())) \
        / sum(BASELINE.values())
    print(f"\nin CI, candidate 2's non-zero exit blocks the merge before "
          f"the cost ships. its aggregate ({cand_agg:+.0%}) sits UNDER the "
          f"10% limit and would have passed; the per-task rule is what "
          f"caught the +42% on 'classify-ticket' (fixed prompt bloat hurts "
          f"small tasks most).")
    raise SystemExit(0 if (ok1 and ok2) else 1)

👉 Next: on-call for agents, the chapter that assumes all of this is in place and something still goes wrong at 3 a.m., what pages, what the runbook says, and why "the model got weird" is a banned root cause.