Cloud cost architecture: the levers, in order of size

What it is

The set of changes that reduce a cloud bill, ordered by how much they typically move it, plus the observation that the order is stable across companies and almost nobody works it in order.

THE LEVERS, roughly by size of effect

1. DELETE THINGS NOBODY USES          10-30%
2. RIGHT-SIZE OVER-PROVISIONED
   COMPUTE                            10-25%
3. COMMITMENT DISCOUNTS
   (reserved, savings plans)          20-40% on committed
4. STORAGE TIERING AND LIFECYCLE      10-30% of storage
5. ARCHITECTURAL: caching, batching,
   and removing a component            variable, often the
                                       largest single win
6. SPOT / PREEMPTIBLE FOR
   INTERRUPTIBLE WORK                 up to 70-90% on that
                                       workload
7. EGRESS AND CROSS-AZ TRAFFIC        often the surprise
8. INSTANCE FAMILY AND ARCHITECTURE
   (Graviton and equivalents)         10-40% on compute

Commonly confused with a FinOps reporting exercise. Attribution tells you where the money goes and changes nothing; the levers are what change it, and a cost programme that produces dashboards and no architectural change is the standard failure.

Also commonly confused with a one-time project. Cloud cost is a rate, not a balance, so a one-time 30 percent reduction is reversed by two quarters of growth unless something structural changed.

The problem it solves

The bill grows faster than traffic, and nobody can say why.

THE SHAPE OF THE PROBLEM

  traffic     +40% year on year
  bill        +85% year on year

The gap is the thing to explain, and it is almost never one
cause. It is typically:
  environments nobody turned off
  instances sized for a launch that never came
  storage with no lifecycle policy
  a chatty service pair that moved across an AZ boundary
  a data pipeline that reprocesses everything nightly
    because incremental was never built

And the reason the gap persists: nobody owns it. Engineering owns latency, product owns features, finance owns the total, and the per-service cost that would let anyone act sits between them.

Mechanics

Lever 1: delete things nobody uses

Consistently the largest and cheapest lever, and consistently done last because it is unglamorous.

WHAT TO LOOK FOR
  non-production environments running 24/7 that are used
    9 hours a weekday
    -> 168 hours to 45 is a 73% reduction on those
  unattached storage volumes and old snapshots
  load balancers with no healthy targets
  idle instances from a migration that finished
  logs retained for years with a 30-day access pattern
  dev databases nobody has connected to in 90 days
  duplicate monitoring agents from two overlapping tools

THE MECHANISM THAT MAKES IT STICK
  Not a cleanup sprint. A scheduled scaledown for
  non-production (off at 19:00, on at 08:00, weekends off),
  and a tag policy where an untagged resource older than
  N days is a candidate for deletion with a notification
  first.

The scheduled scaledown is worth arguing for specifically, because it is a one-time change that keeps saving, whereas a cleanup sprint reverses within two quarters.

Lever 2: right-sizing, and the metric that matters

THE COMMON STATE
  Instances chosen by copying the last service, or sized
  for a peak that was estimated rather than measured.

  Typical finding: 20-40% average CPU utilisation and
  memory at 30%, on instances that could be one size down.

THE METRIC TO USE
  Not average CPU. p95 CPU and p95 memory over a full
  weekly cycle, because the point is whether the smaller
  size survives the peak.

THE DISCIPLINE
  Right-size DOWN one step, watch for a week, repeat.
  A single large jump is how you cause an incident and lose
  the mandate for the whole programme.

And the caution that keeps this honest: right-sizing has a floor set by the latency SLO, not by utilisation. A service at 40 percent CPU may be correctly sized if it needs headroom for burst, and queueing theory says waiting explodes as utilisation approaches one. Sizing to 80 percent average CPU is not efficiency, it is a latency incident scheduled for the next traffic spike.

Lever 3: commitments, and how to size them

The discount is real (typically 20-40% for one to three
year commitments) and the risk is committing to capacity
you stop needing.

THE SIZING RULE
  Commit to your FLOOR, not your average.
  Look at the last 12 months of usage and commit to roughly
  the minimum sustained level, so the commitment is covered
  even in a trough.

  Then cover the middle with a shorter or more flexible
  commitment, and the peak with on-demand.

  A common shape: 60-70% committed, 20-30% flexible,
  10-20% on-demand.

WHAT MAKES IT SAFE
  Convertible or flexible commitment types, which allow
  changing instance family, cost slightly less discount and
  remove most of the regret risk.
  And commit AFTER right-sizing, never before, or you have
  committed to instances you were about to shrink.

"Right-size before you commit" is the ordering error that costs the most, because a three-year commitment to over-provisioned instances locks in the waste.

Lever 5: the architectural levers, which are the largest single wins

Levers 1 through 4 are hygiene and they are bounded. The architectural ones are unbounded and they are where the outsized results are.

CACHING
  A 40 ms query served from a 0.5 ms cache at 90% hit rate
  removes 90% of the database load, which frequently
  removes a database tier.

BATCHING
  Per-request work batched into per-second work. One API
  call per event at 2,000 events/sec against one call per
  100 events is a 100x reduction in request charges.

REMOVING A COMPONENT
  The largest single win available, and the one nobody
  looks for. A service whose entire job could be a library
  call. A queue between two services that always run
  together. A cache in front of something already fast.

DATA PROCESSING SHAPE
  A nightly job that reprocesses everything because
  incremental was never built. Changing it to incremental
  is often a 90%+ reduction on that pipeline.

STORAGE FORMAT
  Uncompressed JSON to compressed Parquet is typically
  5-10x on both storage and scan cost for analytical data.
  Columnar means a query touching 3 of 40 columns reads
  3/40 of the bytes.

The example worth carrying: an LLM workload's cost is dominated by prompt structure rather than model choice. Putting the stable system prompt and examples first so the provider's cache applies can remove most of the input cost, and it is message ordering rather than an architecture change. See LLM cost engineering.

Lever 7: egress and cross-AZ, the surprise

THE THING PEOPLE DO NOT MODEL

  Cross-AZ traffic is charged in both directions on most
  clouds. Two chatty services in different AZs pay for
  every message, twice.

  A 200 MB/s service-to-service link across AZs, at typical
  per-GB rates, is a five-figure monthly line item that
  appears under "data transfer" and is attributed to
  nobody.

  Internet egress is more expensive still, which is why a
  CDN in front of anything user-facing pays for itself on
  the transfer alone before considering latency.

WHAT TO DO
  Keep chatty pairs AZ-local, with cross-AZ replicas for
  availability rather than for traffic.
  Use VPC endpoints for cloud service traffic so it does
  not traverse the internet gateway.
  Compress everything crossing a boundary. At a 500 µs
  round trip, compression is free in latency terms and it
  is bandwidth you are billed for.
  Put a CDN in front of anything served to users.

"Data transfer" being the third-largest line item and attributed to nobody is a common finding, and it is invisible in per-service cost reporting because it is charged to the account rather than to a workload.

The order to work them

1. Delete unused              cheap, large, immediate
2. Right-size                 cheap, large, needs a week
                              of watching per step
3. Storage lifecycle          cheap, large on storage
4. Fix the egress surprise    usually one or two chatty
                              pairs
5. Commitments                AFTER right-sizing
6. Architectural              expensive, largest, and it
                              is the only category that
                              changes the growth RATE
7. Spot for interruptible     high value, needs the
                              workload to tolerate
                              interruption
8. Instance architecture      often a rebuild-and-test,
                              10-40% on compute

Steps 1 to 5 reduce the current bill. Step 6 changes the slope, and that distinction is the one to make to leadership, because a cost programme that only does 1 to 5 delivers a step change that growth erases.

A worked example: a bill growing at twice the traffic rate

CONTEXT
  $180k/month, growing 85% year on year against 40% traffic
  growth. Leadership asked for a 30% reduction.

THE BREAKDOWN (two days of work, and the first useful step)
  compute                      $71k   39%
  managed databases            $38k   21%
  data transfer                $26k   14%   <- unattributed
  object storage               $22k   12%
  managed services (other)     $19k   11%
  observability                 $4k    2%

WHAT THE LEVERS FOUND

  1. DELETE. Three non-production environments running
     24/7, two idle load balancers, 4 TB of snapshots older
     than a year, and a duplicate APM agent on every host
     from a migration that finished 14 months earlier.
     -> $19k/month. Two engineer-weeks.

  2. RIGHT-SIZE. p95 CPU across the fleet was 22%. Stepped
     down one size at a time over six weeks.
     -> $14k/month.

  4. EGRESS. The 14% data transfer line was almost entirely
     two services: the API and the feature store, which
     were in different AZs and exchanged 180 MB/s.
     Co-locating them (with cross-AZ replicas kept for
     availability) removed most of it.
     -> $17k/month, from a change that took three days.
     *** This was the single largest per-effort win and
         nobody had looked, because "data transfer" was
         attributed to no team. ***

  5. COMMITMENTS. Applied AFTER right-sizing, at 65% of the
     new floor, on flexible plans.
     -> $16k/month.

  6. ARCHITECTURAL. The nightly analytics job reprocessed
     the full dataset because incremental had never been
     built. Six engineer-weeks to make it incremental.
     -> $11k/month, AND it stopped growing with data
        volume, which is the part that matters.

  TOTAL: $77k/month, 43% against a 30% target.

THE FINDING WORTH REPORTING
  Five of the six were hygiene, and together they were a
  step change that 40% traffic growth would erase in about
  eighteen months. Only the sixth changed the growth rate,
  and it was the only one that required real engineering.
  So the recommendation to leadership was: take the 43%,
  and fund two more architectural items, because otherwise
  we are having this conversation again next year.

Production evidence

The FinOps Foundation framework (inform, optimise, operate) is the standard organising model, and its central claim, that cost optimisation is a continuous practice rather than a project, is the basis for the rate-versus-balance framing.

Cloud provider pricing documentation on cross-AZ and internet egress charges is the source for the data-transfer point, and the fact that cross-AZ is billed in both directions on major providers is the specific detail teams miss.

AWS's Graviton, and equivalent ARM offerings elsewhere, publish price-performance improvements in the 20 to 40 percent range for suitable workloads, which is the basis for the instance-architecture lever, and the caveat is that it needs a rebuild and a performance test rather than a config change.

Spot and preemptible instance documentation across providers gives the 70 to 90 percent discount range and the interruption semantics that determine which workloads can use it.

Columnar format benchmarks (Parquet versus row-oriented formats, and the ClickHouse and DuckDB literature) support the 5 to 10x storage and scan reduction for analytical workloads.

Queueing theory, and the Universal Scalability Law, is the basis for the right-sizing floor: waiting time scales as one over one minus utilisation, so sizing to high average utilisation trades a cost saving for a latency cliff.

The debate

The case for working the hygiene levers first: they are cheap, fast, low-risk and they build the mandate. A team that delivers 20 percent in a month gets funded for the architectural work; one that proposes a six-month re-architecture first does not.

The case for going straight to architecture: hygiene is a step change that growth erases, and the only durable improvement is changing what the system does. Spending two quarters on right-sizing delays the work that matters.

The case for buying a cost tool: attribution is genuinely hard and the tools do it well.

The case against the tool: attribution changes nothing on its own, and a cost programme that produces dashboards and no architectural change is the standard failure mode.

My position: work the hygiene levers in order to fund the credibility, then spend it on the architectural ones, and be explicit with leadership about which category each item is in.

The distinction I would make in every cost conversation is that hygiene reduces the bill and architecture changes the slope. In the worked example five of six items were hygiene, delivering a 43 percent reduction that 40 percent traffic growth erases in about eighteen months. Only the incremental-processing change altered the growth rate. Reporting a percentage without that distinction sets up the same conversation next year.

The single highest-return item in my experience is the one nobody looks at: egress and cross-AZ traffic. In the example it was 14 percent of the bill, almost entirely two chatty services in different availability zones, and three days of work removed most of it. It stays invisible because "data transfer" is charged to the account rather than to a team, so per-service cost reporting shows nobody responsible for it.

The ordering error that costs the most is committing before right-sizing, because a three-year commitment to over-provisioned instances locks in the waste for three years. Right-size, watch for a few weeks, then commit to the new floor.

And the caution that keeps right-sizing from causing an incident: the floor is set by the latency SLO, not by utilisation. Queueing time scales as one over one minus utilisation, so a service at 40 percent CPU may be correctly sized for burst headroom, and sizing everything to 80 percent average is a latency incident scheduled for the next spike. Step down one size at a time and watch for a week, because a single large jump is how you cause an incident and lose the mandate for the whole programme.

Where I would push back on a request for a cost target: ask whether the goal is the bill or the unit economics. A bill that grows 40 percent while traffic grows 60 percent is a success, and a flat bill on flat traffic with rising cost per transaction is a failure. Cost per unit of business outcome is the number that survives growth.

Follow-up Q&A

"Where do you start on a cloud bill?" With the breakdown, then the levers in order of effort-to-effect: delete what nobody uses, right-size, storage lifecycle, fix the egress surprise, apply commitments, then architecture. Deleting unused resources is consistently the largest and cheapest lever and consistently done last because it is unglamorous, and the version that sticks is a scheduled scaledown for non-production rather than a cleanup sprint, because a sprint reverses within two quarters.

"What's the lever people miss?" Egress and cross-AZ traffic. Cross-AZ is billed in both directions on major clouds, so two chatty services in different availability zones pay for every message twice. In one case that was fourteen percent of the bill, almost entirely two services exchanging 180 megabytes a second across an AZ boundary, and co-locating them took three days. It stays invisible because "data transfer" is charged to the account rather than to a team, so it appears in no per-service report and nobody owns it.

"How do you size a commitment?" To your floor, not your average: look at twelve months of usage and commit to roughly the minimum sustained level, so it is covered even in a trough. Then flexible commitments for the middle and on-demand for the peak, typically something like 65 percent committed. And the ordering matters more than the sizing: right-size first, then commit, because a three-year commitment to over-provisioned instances locks in the waste for three years.

"What's the risk in right-sizing?" Causing a latency incident and losing the mandate for the whole programme. The floor is set by the SLO rather than by utilisation, because queueing time scales as one over one minus utilisation, so a service at 40 percent CPU may be correctly sized for burst headroom and sizing everything to 80 percent average is an incident scheduled for the next spike. So: p95 CPU and memory over a full weekly cycle rather than averages, and step down one size at a time watching for a week.

"Which lever is actually the largest?" The architectural ones, and they are unbounded where the others are not. Caching that removes a database tier, batching that turns per-request work into per-second work, changing a nightly full reprocess to incremental, or removing a component entirely. In one case making an analytics job incremental was six engineer-weeks for $11k a month, and more importantly it stopped that cost growing with data volume.

"What's the distinction you'd make to leadership?" Hygiene reduces the bill; architecture changes the slope. In the case I worked, five of six items were hygiene and delivered 43 percent against a 30 percent target, and 40 percent traffic growth erases that in about eighteen months. Only the incremental-processing change altered the growth rate. Reporting a percentage without that distinction sets up the identical conversation next year, so the recommendation was to take the 43 percent and fund two more architectural items.

"Is a FinOps tool worth it?" For attribution, yes, because it is genuinely hard and the tools do it well. But attribution changes nothing on its own, and the standard failure is a cost programme that produces excellent dashboards and no architectural change. I would treat the tool as the input to the lever list rather than as the deliverable, and I would judge the programme on cost per unit of business outcome rather than on the reporting.

"What if leadership asks for a flat bill?" I would ask whether the goal is the bill or the unit economics, because those diverge. A bill growing 40 percent while traffic grows 60 percent is a success; a flat bill on flat traffic with rising cost per transaction is a failure. Cost per resolved ticket or per active user is the number that survives growth, and it is also the number that tells you whether a feature is worth what it costs.

"What about spot instances and ARM?" Both real and both conditional. Spot is 70 to 90 percent off for workloads that tolerate interruption, which means batch, CI, stateless workers with fast rescheduling, and not a stateful primary. ARM instances are typically 20 to 40 percent better on price-performance for suitable workloads, and it is a rebuild-and-performance-test rather than a config change, so it belongs later in the list than its headline number suggests.

Common misconceptions

"Cost optimisation is a project." It is a rate, not a balance. A one-time 30 percent reduction is erased by two quarters of growth unless something structural changed.

"Attribution is the work." Attribution tells you where the money goes and changes nothing. The levers change it, and a dashboard-only programme is the standard failure.

"Right-size to high utilisation." Queueing time explodes as utilisation approaches one. The floor is the latency SLO, not the utilisation number.

"Commit early for the discount." Commit after right-sizing, or you lock in the over-provisioning for the length of the term.

"Data transfer is a small line item." Cross-AZ is billed both directions and is frequently the third-largest category, attributed to nobody.

Interview delivery note

Give the levers in order and name the one people skip: "There's a stable order to these: delete what nobody uses, right-size, storage lifecycle, fix egress, commitments, then architecture. The first is consistently the largest and cheapest and consistently done last because it's unglamorous, and the version that sticks is a scheduled scaledown for non-production rather than a cleanup sprint."

Volunteer the surprise, because it is the highest per-effort win: "The one nobody looks at is cross-AZ traffic, which is billed in both directions. In one case it was fourteen percent of the bill, almost entirely two services exchanging a hundred and eighty megabytes a second across an AZ boundary, and co-locating them took three days. It's invisible because 'data transfer' is charged to the account rather than a team, so no per-service report shows it."

Name the ordering error: "And right-size before you commit, never the other way round, because a three-year commitment to over-provisioned instances locks in the waste for three years."

Give the caution that shows you have done it: "Right-sizing has a floor set by the SLO rather than by utilisation. Queueing time scales as one over one minus utilisation, so a service at forty percent CPU may be correctly sized for burst. I'd step down one size at a time and watch for a week, because a single large jump causes an incident and you lose the mandate for the whole programme."

Close with the framing for leadership: "and I'd be explicit that hygiene reduces the bill and architecture changes the slope. In the case I worked, five of six items were hygiene, delivering forty-three percent, which forty percent traffic growth erases in about eighteen months. Only the incremental-processing change altered the rate. Reporting a percentage without that distinction sets up the same conversation next year."

The Staff & Team Lead Interview Reference