Design cost attribution for LLM spend across 40 teams
"The AI bill is $400,000 a month and nobody can say what it buys. Design the system that fixes that."
Step 1: clarify, and separate three different asks (4 minutes)
"Cost attribution" is three requests wearing one name, and they need different systems. Establishing which is being asked for is the first job.
SHOWBACK Tell each team what they spent. Informational.
Cheap to build, and it changes behaviour on its own
more than people expect.
CHARGEBACK Actually move the money to their budget. Requires
accuracy good enough to survive a finance dispute,
which is a much higher bar than showback.
UNIT ECONOMICS Cost per business outcome: per resolved ticket,
per generated report, per active user. This is the
one leadership actually wants and almost nobody
builds, because it requires joining spend to
business events.
Assume all three are eventually wanted, and that the order is showback, then unit economics, then chargeback. That ordering is a position worth defending: showback is cheap and changes behaviour, unit economics answers the question being asked, and chargeback is the most work and the least additional insight.
The clarifications:
Spend ~$400k/month. Model APIs, plus GPU instances for
self-hosted models, plus vector database, plus the
embedding pipeline. It is NOT all model API spend,
and teams usually only instrument that part.
Teams 40, in a 4-level org hierarchy
Granularity Team -> project -> feature -> environment. Team-level
alone is not actionable: "team X spent $40k" produces
a shrug; "feature Y inside team X is $38k of it"
produces a change.
Accuracy Showback: within a few percent. Chargeback: must
reconcile to the provider invoice.
Latency Daily is enough for reporting. Budget ENFORCEMENT
needs near-real-time, which is a different system.
The question that reframes the project: "Is the goal to allocate the cost or to reduce it? Because if it is to reduce it, attribution is the first 20 percent of the work and the optimisation levers are the other 80, and I would build attribution only as far as it identifies where the levers are."
Step 2: capacity math (3 minutes)
Event volume
~2,000 model calls/sec at peak across 40 teams
= ~50M usage records/day, ~18B/year
Each record ~400 bytes structured -> 20 GB/day raw
-> This is an analytics pipeline, not a database table.
Aggregation
Dimensions: team x project x feature x env x model x provider
x cache_tier x day
Cardinality: 40 x ~5 x ~8 x 3 x ~6 x ~3 x ~4 x 365
= ~150M rows/year at daily grain. Very manageable.
-> Pre-aggregate to daily and hourly; keep raw records for
30 days for drill-down, then drop.
Reconciliation
The provider invoice is monthly and arrives days later. Our
computed total must match within a tolerance, and the gap
must be explained rather than absorbed.
Typical sources of gap: requests that errored after tokens
were consumed, retries counted once or twice, price changes
mid-month, and calls made outside the gateway.
*** The last one is the big one and it is an organisational
problem, not a technical one. ***
Non-API spend
GPU instances for self-hosted models: allocated by GPU-hours,
which requires per-request GPU time, which requires the
inference server to report it. Often forgotten entirely, and
at a serious deployment it is a third of the bill.
The number that matters is not the spend, it is the fraction of it that flows through an instrumented path. If 70 percent of calls go through the gateway and 30 percent are direct, no attribution system is accurate, and the fix is organisational.
Step 3: architecture
40 teams' applications
│ every call carries attribution context
▼
┌─────────────────────┐
│ LLM GATEWAY │ the ONLY sanctioned path.
│ (see: llm-gateway) │ emits one usage record per call
└──────────┬──────────┘
▼
┌─────────────────────┐ ┌──────────────────────┐
│ KAFKA usage stream │ │ NON-API SPEND │
└──────────┬──────────┘ │ GPU hours, vector DB, │
│ │ embedding jobs │
│ └──────────┬───────────┘
▼ ▼
┌──────────────────────────────────────────────────────┐
│ COST PIPELINE │
│ price table (VERSIONED) x usage -> cost │
│ allocate shared costs by a declared rule │
│ join to business events -> unit economics │
└──────────┬───────────────────────────────────────────┘
▼
┌──────────────────┐ ┌──────────────┐ ┌───────────┐
│ daily/hourly │ │ RECONCILE vs │ │ BUDGET │
│ aggregates │ │ provider │ │ ENFORCEMENT│
│ (warehouse) │ │ invoice │ │ (real-time)│
└──────────────────┘ └──────────────┘ └───────────┘
The single most important design property: attribution context is captured at the call site and carried through, never inferred afterwards. Inferring which feature made a call from IP address, API key or timing is guesswork that is wrong in exactly the cases you care about.
Step 4: the usage record and the price table
@dataclass(frozen=True)
class UsageRecord:
ts: datetime
request_id: str
# Attribution: captured at the call site, propagated through
# the SDK, validated at the gateway. A call with no attribution
# is rejected in staging and flagged in production.
team: str
project: str
feature: str
environment: str # prod / staging / dev
user_id_hash: str | None # for per-user unit economics
# Consumption
provider: str
model: str
input_tokens: int
cached_input_tokens: int # priced differently. MUST be separate.
output_tokens: int
reasoning_tokens: int # where the provider reports it
# Cost, computed HERE from a versioned price table.
price_version: str
cost_usd: Decimal
# Attribution for unit economics
business_event_id: str | None # ticket id, report id, session id
Two decisions to defend:
Cost is computed at record time from a versioned price table, not at query time from current prices. If a provider changes prices on the 15th, historical records keep their original cost, every past report stays stable, and reconciliation against the invoice remains possible. Computing at query time means every historical number silently changes when a price does, which destroys trust in the reports and makes month-over-month comparison meaningless.
Cached and uncached input tokens are separate fields. They are priced very differently, and merging them makes prompt caching, which is usually the single largest available saving, invisible in the reporting. A team cannot optimise a lever they cannot see.
-- The price table. Versioned and effective-dated, because prices
-- change and history must not.
CREATE TABLE model_prices (
price_version TEXT,
provider TEXT,
model TEXT,
effective_from TIMESTAMPTZ,
effective_to TIMESTAMPTZ,
input_per_mtok NUMERIC(12,6),
cached_input_per_mtok NUMERIC(12,6),
cache_write_per_mtok NUMERIC(12,6),
output_per_mtok NUMERIC(12,6),
PRIMARY KEY (provider, model, effective_from)
);
Step 5: shared costs, which is where the arguments happen
Not all spend maps cleanly to one team, and how you allocate the rest is a policy decision that must be declared before anyone disputes it.
Directly attributable (~75%)
Model API calls with attribution context. Easy.
Shared infrastructure (~20%)
Vector database, embedding pipeline, gateway compute,
self-hosted GPU fleet.
Allocation options:
- by usage (queries, vectors stored, GPU-seconds): fairest,
requires per-team instrumentation of each shared service
- by headcount: trivially simple, and it makes a small team
with heavy usage look cheap, which distorts behaviour
- equal split: simplest, and it removes any incentive to
reduce usage, so avoid it
*** By usage, and instrument the shared services. Any other
rule creates an incentive to shift cost onto the shared pool. ***
Platform overhead (~5%)
The platform team's own evaluation runs, load tests, the
gateway's own operation.
-> Keep this in a platform cost centre rather than allocating
it. Allocating overhead produces disputes worth more in
engineering time than the amount allocated.
The general principle worth stating: an allocation rule creates an incentive, and the rule should create the incentive you want. Allocating the vector database by headcount means a team can add 50 million vectors at no visible cost to themselves, so they will.
Step 6: unit economics, the part that answers the actual question
Cost per team is an accounting fact. Cost per outcome is a business fact, and it is what leadership is asking for even when they say "attribution".
-- Join spend to business events. This requires the business
-- event id to be carried on the usage record, which requires
-- the application to pass it, which is the integration work
-- that makes the whole thing valuable.
SELECT
u.team, u.feature,
count(DISTINCT u.business_event_id) AS outcomes,
sum(u.cost_usd) AS total_cost,
sum(u.cost_usd) / nullif(count(DISTINCT u.business_event_id),0)
AS cost_per_outcome,
sum(u.input_tokens + u.output_tokens)
/ nullif(count(DISTINCT u.business_event_id),0)
AS tokens_per_outcome,
sum(u.cached_input_tokens)::numeric
/ nullif(sum(u.input_tokens),0) AS cache_hit_rate
FROM usage u
WHERE u.day >= current_date - 30 AND u.environment = 'prod'
GROUP BY 1, 2
ORDER BY total_cost DESC;
What that query makes visible, and why each column earns its place:
cost_per_outcome The number to compare against the value of the
outcome. "$0.42 per resolved ticket" against a
$12 human handling cost is a business case;
"$40k a month" is a line item.
tokens_per_outcome The efficiency measure. If it is rising while
outcomes are flat, something regressed: a
prompt grew, retrieval is returning more, or
an agent is looping.
cache_hit_rate The single most actionable number in the table.
A feature with a 5% hit rate and a large stable
system prompt has an obvious, cheap fix.
And the comparison that lands with leadership:
Feature Cost/month Outcomes $/outcome Value/outcome
------------------------------------------------------------------------
Support ticket triage $84,000 420,000 $0.20 $6.00
Doc search assistant $61,000 95,000 $0.64 $0.30 <--
Code review assistant $52,000 31,000 $1.68 $9.00
Marketing copy gen $38,000 4,200 $9.05 $2.00 <--
Two features are costing more than they return, and that is invisible in a per-team report. Producing that table is the point of the system, and it is the argument for carrying business event ids on usage records even though it is the hardest part of the integration.
Step 7: from attribution to reduction
Attribution that does not lead to action is a reporting project. The pipeline should surface the levers, in the order they pay:
1. PROMPT CACHING Usually the largest and cheapest win.
Signal: cache_hit_rate < 20% on a feature with a large stable
prefix. Fix is message ordering: stable system prompt and
examples first, variable content last.
2. MODEL RIGHT-SIZING Second largest.
Signal: a classification or extraction task on the largest
model. Fix requires an eval to prove the smaller model holds
quality, which is the eval pipeline design.
3. CONTEXT SIZE Often invisible.
Signal: tokens_per_outcome rising while outcomes are flat.
Fix: retrieval returning 20 chunks where 5 suffice, or an
agent accumulating context it no longer needs.
4. RETRY AND LOOP WASTE
Signal: cost per outcome with a long tail. A small number of
runs consuming 50x the median usually means an agent loop or
a retry storm.
5. ENVIRONMENT WASTE
Signal: dev and staging spend as a fraction of the total.
Above ~10% usually means an unbounded test harness or a
load test nobody turned off.
Ranking the levers is more valuable than the attribution itself, and a report that says "your cache hit rate is 4 percent and here is the message ordering that would fix it" gets acted on where a spend table does not.
Step 8: failure modes
Calls made outside the gateway
-> The attribution is silently incomplete and reconciliation
shows an unexplained gap. This is the top failure and it is
organisational: the fix is provider API keys issued ONLY to
the gateway, so a direct call is not possible rather than
discouraged.
Missing or wrong attribution tags
-> Reject unattributed calls in dev and staging, flag and
default-bucket them in production. Never fail a production
request over a missing tag; do make the unattributed bucket
visible and embarrassing.
Reconciliation gap vs the invoice
-> Expect 1-3%. Sources: errored requests that still consumed
tokens, retries, mid-month price changes, rounding. Track the
gap as a metric and investigate above a threshold rather than
absorbing it, because a growing gap means an uninstrumented
path.
Price table drift
-> A provider changes prices and nobody updates the table, so
every cost is wrong in the same direction. Reconciliation
catches it, which is the reason reconciliation is a
first-class component rather than a monthly chore.
Team gaming the attribution
-> Tagging expensive work as "platform" or "shared". Make the
allocation rules explicit and the reports visible across
teams; cross-team visibility is the enforcement mechanism.
Budget enforcement causing an outage
-> A hard cap in production takes down a customer feature at
month end. Soft degradation and loud alerting in production,
hard caps in dev and staging only. See: llm-gateway.
"Provider keys issued only to the gateway" is the most important line in this design, and it is an organisational control rather than a technical one. Every other accuracy problem is a rounding error compared with spend that never appears.
Step 9: what changes at ten times the scale
At $4 million a month and 150 teams:
Attribution becomes a FinOps function, with a named owner, a monthly review with finance, and forecasting rather than only reporting. The engineering system is the same; the organisational apparatus around it is new.
Forecasting matters more than reporting. "We will exceed budget in 11 days at the current rate" is more useful than last month's breakdown, and it requires trend modelling per team plus known upcoming launches.
Commitment management appears. At that spend, provider commitments and reserved capacity are on the table, which means the system must forecast confidently enough to support a contractual commitment, and under-consumption of a commitment is itself a cost.
Self-hosted inference becomes a real option for the highest-volume, lowest-complexity workloads, and evaluating that requires the cost-per-outcome numbers to compare against GPU amortisation, which is exactly what this system produces.
Production evidence
The FinOps Foundation's framework (inform, optimise, operate) and the FOCUS specification for cloud cost data are the reference for the showback/chargeback distinction and for why allocation rules must be declared in advance.
Anthropic's prompt caching pricing, where cache reads cost a fraction of fresh input tokens, is the reason cached and uncached tokens must be separate fields: merging them makes the largest available saving invisible.
Cloud provider cost allocation tags (AWS, GCP, Azure) work on exactly this model of tags applied at resource creation and propagated, and the well-documented failure mode is untagged resources, which is the direct analogue of unattributed calls.
LiteLLM, Portkey, Helicone and Langfuse all ship per-key and per-tag spend tracking as a headline feature, which is convergent evidence for the usage-record schema here.
OpenTelemetry's GenAI semantic conventions define token-count and model attributes on spans, which is the right basis for the usage record rather than a bespoke schema.
The debate
The case for full chargeback: costs that land on a team's budget change behaviour immediately and permanently. Showback informs; chargeback creates ownership.
The case for showback only: chargeback requires accuracy that survives a finance dispute, it generates arguments about allocation rules that consume more engineering time than they save, and most of the behaviour change comes from visibility alone.
The case for skipping attribution and optimising centrally: a platform team can find and fix the top ten cost drivers faster than 40 teams each learning to read a dashboard. Faster in the short run, and it does not scale and creates no ownership.
My position: showback with unit economics, chargeback only if finance requires it, and the pipeline's primary output is a ranked list of levers rather than a spend table.
The property I would insist on is that cost is computed at record time from a versioned price table. It looks like a detail and it is the difference between reports that are stable and reports where every historical number silently changes when a provider adjusts pricing. Once that has happened once, nobody trusts the numbers again.
The organisational control I would insist on is provider keys issued only to the gateway, so a direct call is impossible rather than discouraged. Every technical accuracy concern is a rounding error next to spend that never appears in the system at all, and this is the only reliable fix.
And the reframe I would push: cost per outcome, not cost per team. "$0.20 per resolved support ticket against a $6 human handling cost" is a business case; "$84,000 a month" is a line item that invites a blunt cut. Building the join from usage records to business events is the hardest part of the integration and it is what converts a reporting project into a decision-making tool, because it is what reveals the features costing more than they return.
Where I would push back on the request: if the goal is to reduce the bill rather than to allocate it, attribution is the first fifth of the work. I would build attribution only as far as it identifies the levers, then spend the remaining effort on prompt caching, model right-sizing and context reduction, which is where the money actually is.
Follow-up Q&A
"Showback or chargeback?" Showback first, and possibly only. Chargeback needs accuracy that survives a finance dispute and it generates arguments about allocation rules that consume more engineering time than they save. Most of the behaviour change comes from visibility alone, especially when the reports are cross-team visible, because nobody wants to be the most expensive feature on a shared dashboard. I would build chargeback only if finance requires it for budgeting.
"Why compute cost at ingest rather than at query time?" Because prices change. If cost is computed from current prices at query time, every historical report silently changes when a provider adjusts pricing, month-over-month comparison becomes meaningless, and reconciling against an invoice from three months ago is impossible. Computing at record time from a versioned, effective-dated price table means history is stable. It looks like a detail and it is the thing that determines whether anyone trusts the numbers.
"What is the biggest source of inaccuracy?" Calls that never reach the gateway. If 30 percent of spend is direct provider calls, no amount of pipeline engineering makes the attribution correct, and reconciliation just shows an unexplained gap. The fix is organisational: provider API keys issued only to the gateway, so a direct call is impossible rather than discouraged. Everything else, retries, errored requests that consumed tokens, mid-month price changes, is a one to three percent reconciliation gap that you track and explain.
"Why separate cached from uncached input tokens?" Because they are priced very differently and merging them makes prompt caching invisible in reporting, which is usually the single largest available saving. A team with a five percent cache hit rate on a feature with a twenty-thousand-token stable system prompt has an obvious and cheap fix, and they cannot see it if the report shows one input-token number.
"How do you allocate shared infrastructure?" By usage, and instrument the shared services to make that possible. The reason is that an allocation rule creates an incentive: allocating the vector database by headcount means a team can add fifty million vectors at no visible cost to themselves, so they will. Equal split is worse, because it removes any incentive to reduce usage at all. Platform overhead I would keep in a platform cost centre rather than allocate, because those disputes cost more engineering time than the amount in question.
"What does leadership actually want?" Cost per outcome, even when they ask for attribution. "Eighty-four thousand a month for ticket triage" invites a blunt cut. "Twenty cents per resolved ticket against a six-dollar human handling cost" is a business case, and the same table shows two features costing more than they return, which is invisible in a per-team report. Getting there requires carrying a business event id on the usage record, which is the hardest part of the integration and the thing that makes the system worth building.
"Where does the money actually go, in practice?" Usually prompt structure rather than model choice. Cache hit rate is the first thing I would rank features by, because the fix is message ordering and costs nothing. Then model right-sizing, which needs an eval to prove a smaller model holds quality. Then context size, where the signal is tokens per outcome rising while outcomes are flat, meaning retrieval is returning twenty chunks where five would do or an agent is accumulating context. Then retry and loop waste, visible as a long tail where a few runs cost fifty times the median.
"Should budgets be enforced by this system?" Enforcement is a different system with different latency requirements: reporting is fine daily, enforcement needs near-real-time counters at the gateway. And I would be careful about hard caps in production, because a team hitting its cap on the 28th takes down a customer feature, which turns a cost-control system into an availability incident. Hard caps in dev and staging, soft degradation and loud alerting in production.
"How do you know the numbers are right?" Reconciliation against the provider invoice, as a first-class monitored component rather than a monthly chore. Expect a one to three percent gap from errored requests, retries and rounding. Track it as a metric and investigate above a threshold, because a growing gap is the earliest signal that an uninstrumented path has appeared or that the price table is stale.
Common misconceptions
"Attribution means tagging." Tagging is necessary and it does not help if a third of spend bypasses the instrumented path. The control is key issuance, not tag discipline.
"Compute cost at query time from current prices." Then every historical report changes when a price does, and nobody trusts the numbers.
"Cost per team is the deliverable." Cost per outcome is what answers the question being asked, and it is what reveals features that cost more than they return.
"Cached and uncached tokens can be one field." Merging them hides the largest available saving.
"Allocation rules are an accounting detail." Each rule creates an incentive. Headcount allocation of a shared vector database guarantees it will be over-used.
Interview delivery note
Split the request into three, because the ask is ambiguous and the answer depends: "Cost attribution is three different requests. Showback tells teams what they spent. Chargeback moves the money, which needs accuracy that survives a finance dispute. Unit economics is cost per business outcome, and that's the one leadership actually wants even when they ask for the other two. I'd do showback and unit economics, and chargeback only if finance requires it."
Then name the failure that makes everything else moot: "And before any pipeline design, the top accuracy problem isn't technical. If thirty percent of calls bypass the gateway, no attribution system is correct. So provider keys are issued only to the gateway, which makes a direct call impossible rather than discouraged. That's an organisational control and it matters more than everything downstream of it."
Give the two schema decisions with their reasons: "Cost is computed at record time from a versioned price table, not at query time, because otherwise every historical report changes when a provider adjusts pricing and nobody trusts the numbers again. And cached input tokens are a separate field from uncached, because they're priced very differently and merging them hides prompt caching, which is usually the largest saving available."
Land the reframe, because it is what makes this a staff answer: "and the deliverable is cost per outcome, not cost per team. 'Eighty-four thousand a month' invites a blunt cut. 'Twenty cents per resolved ticket against a six-dollar human handling cost' is a business case, and the same table shows the two features that cost more than they return, which a per-team report can't."
Close by scoping honestly: "And I'd ask whether the goal is to allocate the cost or reduce it, because if it's to reduce it, attribution is the first fifth of the work. I'd build it only as far as it ranks the levers, then spend the rest on caching, right-sizing and context reduction, which is where the money is."
Further reading
- The FinOps Foundation framework and the FOCUS specification, for showback versus chargeback and for declared allocation rules.
- Anthropic's prompt caching documentation, for why cached and uncached tokens must be separately priced and reported.
- AWS and GCP cost allocation tag documentation, for the tag-at-creation model and its untagged-resource failure mode.
- OpenTelemetry GenAI semantic conventions, for the usage-record attributes.
- Langfuse and Helicone documentation, as reference implementations of per-feature LLM spend tracking.