LLM cost engineering

"Cut LLM spend 60 percent without hurting quality. What's the order of moves?"

What it is

The systematic reduction of cost per unit of delivered value in an LLM product. The unit matters: cost per successful task, not cost per token, because a cheaper model that fails and gets retried is more expensive than the model it replaced.

The basic identity:

$$\text{cost} = (T_{in} \times p_{in}) + (T_{out} \times p_{out})$$

with two multipliers that dominate everything else in practice: the cache hit rate on input tokens, and the model each request is routed to. Output tokens are typically priced several times higher than input, which sounds like it makes output the target, and usually does not, because input volume is normally an order of magnitude larger.

Commonly confused with picking a cheaper model. Model choice is one lever and rarely the first one, because it is the lever most likely to cost quality and the easiest to reach for without measurement.

The problem it solves

LLM spend has a shape that surprises finance: it is variable per request, it scales with usage rather than with infrastructure, and it can grow by an order of magnitude from a prompt change nobody reviewed. A team that adds three few-shot examples to a prompt has just multiplied the input cost of every request, permanently, with no deploy-time signal.

The second problem is attribution. "We spent $180,000 on tokens last quarter" is not actionable. "Feature X costs $0.11 per invocation, is invoked 400,000 times a month, and 70 percent of those invocations are the same twelve questions" is.

Mechanics

Move 0: measure, before anything else

You cannot cut what you cannot attribute, and every trace should carry enough to do it:

{
  "request_id": "...", "team": "support-platform", "feature": "ticket-summary",
  "tenant_id": "acme", "model": "<pinned version>",
  "input_tokens": 4210, "cached_input_tokens": 3800, "output_tokens": 180,
  "cost_usd": 0.0138, "latency_ms": 2140,
  "outcome": "accepted"
}

outcome is the field teams omit and the one that makes the whole exercise honest, because it lets you compute cost per successful task rather than cost per call. cached_input_tokens is the second: without it you cannot tell whether caching is working, and caching silently not working is the most common cost bug.

The first report to build is a Pareto by feature. In every deployment I have seen, a small number of features account for most of the spend, and one of them is doing something nobody intended.

The levers, ranked by return per unit of effort

1. Prompt caching. Largest single lever for anything with a stable prefix. A cache read costs roughly a tenth of an uncached input token; a write costs 1.25x (short TTL) or 2x (long TTL), so break-even is two requests on the short TTL and three on the long.

The engineering is ordering, not code: stable content first, volatile content last, because caching is a prefix match and any byte change invalidates everything after it. The silent killers are a timestamp in the system prompt, a per-request UUID, unsorted JSON serialisation, and a tool list built per user. All four look harmless in review.

Verify with the cache-read token count in the response. If it is zero across repeated requests with what should be an identical prefix, something is invalidating and no amount of configuration fixes it.

2. Model routing with escalation. Send every request to a small model first; escalate to a large one only when the small one signals low confidence or a validator rejects the output.

def answer(query, context):
    """Route cheap-first. The escalation predicate is the whole design:
    it must be cheaper than the expensive call and correlated with quality."""
    draft = small_model(query, context)

    if validator_rejects(draft):          # schema violation, missing citation,
        return large_model(query, context) # refusal, or a self-reported low
                                           # confidence score
    return draft

The economics, for a workload where the small model is roughly a fifth the price and handles 70 percent of traffic:

Baseline (all large):        1.00 x volume x large_price
Routed:                      0.70 x 0.2  +  0.30 x (0.2 + 1.0)
                           = 0.14 + 0.36 = 0.50   -> ~50% saving

Note the escalated 30 percent pays both calls. That is why the escalation rate matters more than the price ratio: at 60 percent escalation the routing saves almost nothing and adds latency to most requests. Measure the rate before committing to the architecture.

3. Context trimming. The cheapest tokens are the ones you do not send. Two places it hides:

• Retrieval k. Teams retrieve 20 chunks because the window allows it. Add a reranker and drop to 5, and you cut input tokens roughly 75 percent while improving quality, because you removed distractors. This is the rare change that is a win on both axes.
• Tool output. A tool returning 40,000 tokens of log with a truncation to 2,000 plus a fetch-more handle is often the single largest saving in an agent, and it is a few lines of code.

4. Semantic caching. Cache by embedding similarity rather than exact match, so "how do I reset my password" hits the entry for "password reset steps".

This is the lever with a correctness risk and it should be presented that way. Two semantically similar queries can require different answers ("cancel my subscription" versus "cancel my subscription without a refund"), and a similarity threshold that is too loose returns confidently wrong cached answers. Use it for narrow, high-volume, low-variance intents; never for anything personalised, account-specific, or time-sensitive; and always with a TTL.

5. Distillation. Use production traffic from the large model to fine-tune a small one for your specific task. The largest possible saving and the largest investment: you need traffic volume, a labelling or filtering pipeline, training infrastructure, and an evaluation suite good enough to prove the small model has not regressed. Worth it for a high-volume, narrow, stable task; not worth it for a general assistant whose behaviour changes monthly.

6. Batch APIs. Asynchronous processing at a substantial discount (commonly around 50 percent), for anything not latency-sensitive: overnight enrichment, backfills, offline evaluation, bulk classification. Free money for the right workload and inapplicable to interactive ones.

7. Output length. Priced highest per token, so worth an explicit instruction and a max_tokens ceiling. The reason it is seventh rather than first is that output volume is usually much smaller than input volume, so a 30 percent reduction in a small number is a small number. Do it, and do not expect it to be the answer.

Guardrails, so the saving does not leak back

• Per-tenant and per-feature token quotas, enforced at the gateway.
• A circuit breaker on spend rate, not just on error rate. A prompt-injection loop or a retry storm can burn a month's budget in an afternoon.
• Cost per request as a CI gate, so a prompt change that triples input tokens fails the build rather than showing up on the invoice.
• Alert on cost-per-request drift, which catches the slow leak that no single change is responsible for.

A worked example: the 60 percent cut

A support assistant. $180,000 per quarter. 2.1 million requests. $0.086 per request.

Attribution first (week 1). The Pareto is stark:

The moves, in order, with measured effect:

The two observations worth making in the room. First, the largest single saving was fixing a bug: a timestamp in the system prompt meant prompt caching had never worked, and nobody knew because nothing measured cache hit rate. Second, the $34,000 from reducing $k$ came with a quality improvement, because fewer distractors meant better grounding. The framing "cut cost without hurting quality" assumes a tradeoff that the first two moves do not have.

What was considered and rejected, which is the part that makes it a real answer: semantic caching on answer drafting, because support answers are account-specific and a near-miss returns another customer's context, which is a data-exposure incident rather than a quality regression. And distillation, because at 400,000 requests a quarter the training and evaluation investment does not pay back inside a year.

Production evidence

Prompt caching is offered by every major provider with published pricing multipliers (reads at roughly a tenth of input price, writes at a premium), and the prefix-match semantics are documented, including the minimum cacheable prefix and the response fields that report cache hits. That documentation is the primary source for the break-even arithmetic above.

Batch APIs with an approximately 50 percent discount are standard across providers, which is a strong signal about the value of latency insensitivity.

Model routing is productised: gateway products and open-source routers exist specifically to implement cheap-first-with-escalation, and the research literature on LLM routing (for example RouteLLM, Ong et al. 2024) reports substantial cost reductions at near-parity quality on general assistant workloads.

Distillation is well-established as a technique (Hinton et al., 2015, for the original framing), and its modern form for LLMs is training a small model on the large model's outputs for a specific task.

The debate

The credible counter-position: cost engineering is premature for most teams. Engineering time is more expensive than tokens until spend is material, and a team that spends a quarter building a routing layer to save $40,000 has spent more than it saved. The correct first answer for a small deployment is often "do nothing except measure, and revisit at ten times the volume".

The counter-counter: the two highest-return moves, prompt caching and reducing retrieval $k$, cost days rather than quarters, and both are things you should do anyway for latency and quality. So the "premature" objection applies to routing, semantic caching and distillation, not to the whole exercise.

My position: measure first with per-feature attribution and an outcome field, because the Pareto always surprises. Then take the two cheap structural wins, caching and context trimming, which usually improve quality as well. Then route by task rather than by model preference. Treat semantic caching as a correctness risk requiring a narrow use case, and treat distillation as a real project with a payback calculation, not a tactic.

Cost engineering is the wrong priority when the product has not found its shape and the prompts change weekly, because you will optimise something you are about to delete; and when spend is genuinely small relative to team cost, where the honest answer to a VP asking about token spend is "it is 2 percent of this team's cost and I am not going to optimise it yet".

Follow-up Q&A

"Cut LLM spend 60 percent without hurting quality. What's the order?" Measure first, per feature, with an outcome field so the metric is cost per successful task rather than cost per call. Then prompt caching, which is the largest single lever and usually broken for a silly reason like a timestamp in the prefix. Then context trimming, especially retrieval $k$, which typically improves quality because you removed distractors. Then model routing, cheap first with escalation. Then batch APIs for anything not latency-sensitive. Semantic caching and distillation last, because one carries a correctness risk and the other is a project.

"How do you decide the routing threshold?" From the escalation rate, not the price ratio, because escalated requests pay for both calls. If the small model is a fifth the price and escalates 30 percent of the time, you save about half; at 60 percent escalation you save almost nothing and have added latency to most requests. So measure the escalation rate on real traffic before committing, and choose the escalation predicate carefully: it must be cheaper than the expensive call and actually correlated with quality. A schema validator or a citation check is a good predicate; a self-reported confidence score is a weak one.

"What's the risk with semantic caching?" Two semantically similar queries can require different answers, so a loose similarity threshold returns confidently wrong cached content. "Cancel my subscription" and "cancel my subscription without losing my data" embed closely and need different answers. Worse, in a multi-tenant product a near-miss can return another customer's context, which is a data-exposure incident rather than a quality bug. Use it only for narrow, high-volume, non-personalised intents, with a TTL, and measure the false-hit rate explicitly on a labelled set.

"How do you attribute cost across 40 teams?" Tag every request at the gateway with team, feature and tenant, and store cost per request rather than tokens, so the number does not need re-deriving when prices change. Then publish a per-team dashboard and a monthly report, and set per-team quotas enforced at the gateway so attribution has teeth. The failure mode to avoid is attributing only at the model level, which tells you that you spent a lot on one model and nothing about who or why.

"Your cost per request drifted up 40 percent with no deploy. What happened?" Most likely the cache stopped hitting: someone added a dynamic value to the prefix, or the tool list started varying, or a provider changed the minimum cacheable prefix. Check the cached-input-token field first, since that is one query. Second candidate is a change in traffic mix rather than in the system: a new customer with much longer documents shifts the average without anything regressing. Third is a retry loop, which shows as a request-count increase rather than a per-request one.

Common misconceptions

The most common is that cheaper models are the first move. They are the lever most likely to cost quality and the easiest to reach for without measurement, and they are usually third or fourth in return.

The second is that output tokens dominate because they are priced higher per token. Input volume is normally an order of magnitude larger, so input times a lower price usually exceeds output times a higher one. Check your own ratio before optimising.

The third is that caching is a flag. It is a prefix property, and one timestamp in the system prompt disables it entirely and silently. The only way to know is the cache-hit field in the response.

Interview delivery note

Say this: "Measure first, per feature, with an outcome field so I'm optimising cost per successful task rather than cost per call. The Pareto always surprises. Then prompt caching, which is the largest lever and is usually broken because something dynamic is in the prefix. Then context trimming, especially retrieval k, which normally improves quality because you removed distractors. Then routing: cheap model first, escalate on a validator failure. Then batch APIs for anything not latency-sensitive. Semantic caching and distillation last, because one is a correctness risk and the other is a project."

The depth signal is the routing arithmetic: "the escalation rate matters more than the price ratio, because escalated requests pay for both calls. At 30 percent escalation with a fifth-price model you save about half; at 60 percent you save almost nothing." And the strongest close is honesty about the premise: "'without hurting quality' assumes a tradeoff, and the first two moves usually improve quality, because caching changes nothing semantically and trimming context removes distractors."

Further reading

• Provider prompt-caching documentation, for the read and write price multipliers, the prefix-match semantics, and the cache-hit response fields.
• Ong et al., "RouteLLM: Learning to Route LLMs with Preference Data" (2024), for routing as a measured technique rather than a heuristic.
• Hinton, Vinyals and Dean, "Distilling the Knowledge in a Neural Network" (2015), for the original framing.
• Provider batch API documentation, for the discount and the latency contract.