Back-of-envelope capacity math: a worked search-service sizing

What it is

The practice of deriving a system's shape from arithmetic in a few minutes, before drawing any boxes. It is not estimation for its own sake: the numbers decide the architecture, and a design produced without them is a design produced from familiarity.

The discipline has three parts:

1. STATE ASSUMPTIONS EXPLICITLY, so they can be corrected.
   "Assume 50 million monthly actives" is a number the
   interviewer can revise; "assume high traffic" is not.

2. WORK IN ROUND NUMBERS. 100,000 seconds per day, not 86,400.
   The answer needs to be right to within a factor of two, and
   precision beyond that is wasted time and invites arithmetic
   errors.

3. DERIVE THE ARCHITECTURE FROM THE RESULT, not before it.
   The sentence that matters is "X is Y, therefore Z", and a
   candidate who computes numbers and then designs something
   unrelated to them has done the arithmetic as decoration.

Commonly confused with capacity planning, which is a production activity with real measurements. This is design-time sizing: the goal is to establish which constraint binds, not to size a purchase order.

The problem it solves

Without the arithmetic, every design decision is made from familiarity rather than from constraint, and the failure mode is specific: a system designed for the wrong bottleneck.

Design instinct               What the arithmetic says
-----------------------------------------------------------------
"We need a distributed        1 million jobs/day is 12/sec and
 database"                    10 GB. That is one Postgres.

"Vector search needs a        2 million vectors x 256 fp16 is
 dedicated cluster"           1 GB. It fits in every process.

"We should cache this"        The cache round trip is 500 µs and
                              the query is 800 µs. Not worth it.

"Just add more shards"        The index is 40 GB. Twelve shards
                              means 3 GB each and the query cost
                              is dominated by scatter-gather.

Each of those is a real design conversation settled in thirty seconds by arithmetic.

Mechanics: the numbers to have

TIME
  Seconds per day             ~100,000  (86,400, rounded)
  Seconds per month           ~2,500,000
  Seconds per year            ~30,000,000

SIZES
  ASCII character                 1 byte
  UUID                           16 bytes (36 as a string)
  Timestamp                       8 bytes
  A typical JSON record       200 bytes to 2 KB
  A web page                  1 to 3 MB
  A photo                     200 KB to 5 MB
  An embedding (768 fp32)      3 KB   (768 int8: 768 bytes)

THROUGHPUT (single node, order of magnitude)
  Postgres simple reads        10,000 to 50,000/sec
  Postgres writes               5,000 to 20,000/sec
  Redis                       100,000/sec (pipelined: 1M/sec)
  Kafka per broker            100 MB/sec sustained
  HTTP service (simple)        5,000 to 20,000 req/sec/core-ish
  NVMe                        500,000 IOPS, 3 GB/sec

RATIOS
  Peak to average              2x to 5x   (10x for events)
  Read to write                10:1 to 1000:1 for user-facing
  Compression                  3:1 to 10:1 for text

The two formulas that turn traffic into fleet size:

LITTLE'S LAW          L = λ × W
                      concurrency = arrival rate × service time
                      This sizes thread pools, connection pools
                      and instance counts.

UTILISATION           At utilisation ρ, queueing time is
                      W_q = W_service × ρ / (1 - ρ)
                      At ρ = 0.8, waiting is 4x service time.
                      At ρ = 0.9, it is 9x.
                      *** Never size for above 70% utilisation. ***

See Little's Law and queueing for the derivation.

The worked sizing: a product search service

The problem statement: search over a 50-million-item catalogue for a marketplace with 20 million monthly active users. p99 under 200 ms.

Step 1: traffic

ASSUMPTIONS (state them, invite correction)
  20M monthly active users
  Each active user searches ~5 times per active day
  Users are active ~10 days per month
  -> 20M x 5 x 10 = 1 billion searches/month

AVERAGE QPS
  1e9 / 2.5e6 seconds = 400 QPS

PEAK
  Marketplace traffic is diurnal with a 3x peak, and a seasonal
  peak (Black Friday) at 4x the normal peak.
  -> Design for 400 x 3 = 1,200 QPS normal peak
  -> Headroom for 400 x 12 = 4,800 QPS seasonal

The first architectural consequence: 1,200 QPS is not a large number. This tells us immediately that the design is not about throughput, and a candidate who starts sharding for QPS at this point has not done the arithmetic.

Step 2: storage

CATALOGUE
  50M items x ~2 KB of source data = 100 GB

INVERTED INDEX
  Rule of thumb: a Lucene index is 0.3x to 1x the source text
  depending on stored fields and doc values.
  Assume 0.5x on the indexed text (say 60% of the source):
  50M x 2 KB x 0.6 x 0.5 = 30 GB

VECTORS (for hybrid retrieval)
  50M x 768 dims x 4 bytes (fp32)         = 154 GB
  50M x 768 dims x 1 byte  (int8)         =  38 GB
  + HNSW graph at M=32: 50M x ~280 bytes  =  14 GB
  -> 52 GB quantised, 168 GB unquantised

TOTAL SERVING FOOTPRINT
  30 GB lexical + 52 GB vector = 82 GB

The second architectural consequence, and the one that decides the design: 82 GB fits comfortably on a single 128 GB instance. This is not a distributed search problem. It is a replicated-single-node problem, which is a completely different and much simpler system.

And the counterfactual that shows the arithmetic mattered: at fp32 the vectors alone are 154 GB and the total is 184 GB, which does not fit and forces sharding. The quantisation decision is what determines whether this is a sharded system, and that is worth saying out loud because it reverses the usual order: normally you shard and then optimise, and here the optimisation removes the need to shard.

Step 3: fleet size, from Little's Law

SERVICE TIME per search (budget, from the funnel)
  lexical retrieval        15 ms
  vector retrieval         25 ms   (parallel with lexical)
  fusion + filter           3 ms
  rerank (top 50)          35 ms
  serialise                 2 ms
  -> ~65 ms of work, with retrieval overlapped

CONCURRENCY at peak
  L = λ × W = 1,200 QPS × 0.065 s = 78 concurrent requests

THREADS AND UTILISATION
  Never size for above 70%: 78 / 0.7 = 112 concurrent slots.

PER NODE
  A 16-core node with the index in memory handles roughly
  16 / 0.065 = 246 requests/sec at 100% CPU,
  so ~170/sec at 70% utilisation.

FLEET
  1,200 / 170 = 7 nodes for the normal peak.
  Round to 9 for AZ balance (3 per AZ across 3 AZs).
  For the seasonal 4,800 QPS peak: 28 nodes, so autoscale
  3x-4x or over-provision seasonally.

The third consequence: nine nodes, each holding the entire 82 GB index. Every node can answer every query with no scatter-gather, which removes the tail-at-scale problem entirely. That is a substantial simplification and it fell out of the storage arithmetic rather than from a preference.

Step 4: the write path

CATALOGUE CHANGES
  Price and stock: assume 4% of items change per day
    50M x 0.04 = 2M changes/day = 23/sec average, 100/sec peak
  Text: assume 0.2% per day
    50M x 0.002 = 100k/day = ~1/sec

EMBEDDING COST
  Only text changes need re-embedding: 100k/day.
  At 2,000 embeddings/sec on one GPU: 50 seconds of GPU per day.
  *** Negligible. ***

  If we re-embedded on EVERY change including price:
  2M/day at 2,000/sec = 1,000 seconds/day, still not huge,
  but it would churn the HNSW graph continuously and force
  constant rebuilds.
  -> Separate the paths: price/stock to an attribute store
     applied at ranking time; text through the embedding
     pipeline.

INDEX REBUILD
  HNSW build for 50M vectors at M=32, efConstruction=200:
  roughly 4-8 hours on 32 cores.
  -> Weekly offline rebuild, alias swap. Not an online operation.

The fourth consequence: the freshness split is forced by the rebuild cost, not chosen for elegance. Two million daily price changes cannot go through a pipeline whose rebuild takes six hours.

Step 5: check the assumptions that would change the answer

This step is what separates a good sizing from a recited one. Which assumption, if wrong by a factor of three, changes the architecture?

ASSUMPTION                  If 3x wrong, does the design change?
------------------------------------------------------------------
20M MAU                     No. 3,600 QPS still fits 25 nodes,
                            still no sharding.
50M items                   YES. 150M items x int8 = 156 GB,
                            which does not fit one node. This is
                            the assumption that matters.
2 KB per item               YES, for the same reason via index size.
65 ms service time          No. 195 ms would break the p99 budget
                            long before it broke the fleet size,
                            so it is a latency problem not a
                            capacity one.
4% daily price change       No. Even 12% is 6M/day = 70/sec.

"The catalogue size is the assumption that decides whether this is sharded" is the sentence to say, because it tells the interviewer exactly what to probe and demonstrates you know which number is load-bearing.

The summary that falls out

ARCHITECTURE, derived rather than chosen:

  9 nodes, 3 per AZ, each holding the full 82 GB index
  int8-quantised vectors (this is what avoids sharding)
  No scatter-gather, so no tail-at-scale problem
  Autoscale to ~28 for seasonal peaks
  Two write paths: attributes (seconds) and text (hourly)
  Weekly offline HNSW rebuild with an alias swap
  A read-through cache for head queries, since the top 1% of
    queries are typically 30-50% of traffic

Production evidence

Jeff Dean's "Numbers Everyone Should Know" and his Stanford lecture on building large-scale systems established this practice as an explicit interview and design skill at Google, and the framing that the numbers should determine the design is his.

Little's Law (Little, 1961) is the theorem behind the concurrency arithmetic, and its generality (it holds for any stable queueing system regardless of arrival or service distribution) is why it can be applied without knowing the distributions.

The Universal Scalability Law (Gunther) supplies the utilisation caution: contention and coherence terms mean that throughput does not scale linearly with nodes, so sizing at 70 percent utilisation rather than 90 is not conservatism but an acknowledgement of the queueing term.

Elasticsearch's shard-sizing guidance (target tens of gigabytes per shard rather than a fixed count) is the vendor version of the "does it fit on one node" question, and its existence reflects how often over-sharding is the actual problem.

The quantisation figures come from the standard HNSW memory formula (dimensions times bytes per component, plus roughly eight times M for the graph), which is documented in hnswlib and reproduced in every vector database's capacity guidance.

The debate

The case for doing the arithmetic first: it prevents designing for the wrong bottleneck, which is the most common and most expensive design error. A distributed system built for a workload that fits on one machine carries permanent complexity for no benefit.

The case against over-indexing on it: the numbers are assumptions, real workloads are skewed in ways averages hide, and a design justified by arithmetic that turns out to be wrong is still wrong. Measurement beats estimation whenever measurement is available.

My position: do the arithmetic, state the assumptions, and name which one is load-bearing.

The part I would insist on is the last one. Any sizing rests on assumptions and only one or two of them change the architecture, so identifying those and saying so is more valuable than the numbers themselves. In the worked example, tripling the user count changes nothing and tripling the catalogue size forces sharding, and knowing that tells you exactly what to verify before committing.

The specific discipline I would apply is checking whether it fits on one node before designing a distributed system. The 82 GB figure is the whole design: it means nine replicas rather than a sharded cluster, no scatter-gather, and no tail-at-scale problem. And notice the direction of causation, which is unusual: the int8 quantisation decision is what keeps it under the threshold, so an optimisation removed the need for a distributed architecture rather than the architecture coming first and the optimisation later.

Where I would push back on the exercise: an average is a poor summary of a skewed workload, and search traffic is extremely skewed. Four hundred QPS average hides that the top 1 percent of queries are typically 30 to 50 percent of traffic, which means a cache is worth far more than the average suggests, and that a p99 sized from average service time will be wrong. The arithmetic gives you the shape; the distribution gives you the tail, and I would say which one I am answering.

Follow-up Q&A

"How do you approach a sizing question?" State assumptions explicitly so they can be corrected, work in round numbers because the answer only needs to be right within a factor of two, and then derive the architecture from the result rather than the other way round. The sentence that matters is "X is Y, therefore Z". A candidate who computes numbers and then designs something unrelated to them has treated the arithmetic as decoration.

"What decided the architecture in that example?" The storage figure. Eighty-two gigabytes fits on one node, so this is nine replicas each holding the full index, not a sharded cluster, which means no scatter-gather and therefore no tail-at-scale problem. And what is interesting is the direction: at fp32 the vectors alone are 154 gigabytes and it does not fit, so the int8 quantisation decision is what removed the need for sharding. An optimisation determined the architecture rather than following from it.

"Which assumption matters most?" The catalogue size, and I would say so unprompted. If users triple, 3,600 QPS still fits about 25 nodes and nothing structural changes. If the catalogue triples to 150 million items, the index is 156 gigabytes and it no longer fits one node, so it becomes a sharded system with all the scatter-gather latency that implies. Knowing which number is load-bearing tells the interviewer exactly what to probe and tells me what to verify before committing.

"How do you get from QPS to fleet size?" Little's Law: concurrency equals arrival rate times service time. Twelve hundred QPS at 65 milliseconds is 78 concurrent requests. Then divide by the target utilisation, and I would never size above 70 percent, because queueing time is service time times rho over one minus rho, so at 80 percent utilisation you are waiting four times the service time and at 90 percent, nine times. That gives 112 slots, and at roughly 170 requests per second per 16-core node it is seven nodes, rounded to nine for three-AZ balance.

"Why round 86,400 to 100,000?" Because the answer needs to be right within a factor of two and the rounding introduces 16 percent error, which is far smaller than the uncertainty in the assumptions themselves. Meanwhile it makes the arithmetic doable in your head without mistakes, and an arithmetic slip in an interview costs more credibility than a 16 percent imprecision. The numbers to have are 100,000 seconds a day and 2.5 million a month.

"Where does the average mislead you?" Search traffic is extremely skewed, so 400 QPS average hides that the top one percent of queries are typically 30 to 50 percent of volume. That means a cache is worth much more than an average-based calculation suggests, and a p99 estimated from average service time will be wrong because the tail is driven by the slow minority. The arithmetic gives you the shape and the distribution gives you the tail, and I would be explicit about which one I am answering.

"When would you not do this?" When measurement is available. This is design-time sizing to establish which constraint binds, not capacity planning. If the system exists, its actual p99, its actual cache hit rate and its actual query distribution beat any estimate, and the right move is to measure rather than to argue from assumptions.

"What is the most common error you see?" Designing for the wrong bottleneck. A million jobs a day is twelve per second and ten gigabytes, which is one Postgres instance, and teams reach for a distributed database. Two million vectors at 256 fp16 features is one gigabyte, which fits in every serving process, and teams put it in a remote store and then discover the lookup rate is unservable. In both cases thirty seconds of arithmetic settles it and the design conversation changes completely.

Common misconceptions

"Precision matters." The answer needs to be right within a factor of two. Precision beyond that costs time and invites arithmetic errors.

"Size for peak." Size for peak at a target utilisation well below saturation, because queueing time explodes as utilisation approaches one.

"More nodes is safer." More nodes means more scatter-gather and a worse tail. Check whether it fits on one node first.

"The average tells you the shape." For skewed workloads it does not. The average sizes the fleet and the distribution sizes the tail.

"Do the arithmetic, then design." Do the arithmetic and let it determine the design. If the numbers and the architecture are unrelated, the arithmetic was decoration.

Interview delivery note

Announce the method before doing it, because the structure is part of what is being scored: "I'll state assumptions explicitly so you can correct them, work in round numbers because this only needs to be right within a factor of two, and then let the numbers pick the architecture."

Then make the derivation audible: "Twenty million monthly actives at five searches on ten active days is a billion a month, which over two and a half million seconds is 400 QPS average and about 1,200 at peak. That's not a large number, which already tells me this isn't a throughput problem."

Land the decisive number and say what it decides: "Fifty million items gives a thirty gigabyte lexical index, and int8 vectors plus the HNSW graph is fifty-two, so eighty-two gigabytes total. That fits on one node, so this is nine replicas each holding everything, not a sharded cluster, and there's no scatter-gather and no tail-at-scale problem. And notice that at fp32 the vectors alone are a hundred and fifty-four gigabytes and it doesn't fit, so the quantisation decision is what removed the need to shard."

Then the move that most candidates skip: "The assumption that's load-bearing is the catalogue size. Triple the users and nothing structural changes; triple the catalogue and it no longer fits one node and this becomes a sharded system. So that's the number I'd want to be right about."

And show calibration on the method itself: "though the average hides a lot here. Search traffic is heavily skewed, the top one percent of queries is usually thirty to fifty percent of volume, so a cache is worth much more than this calculation implies and the p99 won't come from the average service time."

The Staff & Team Lead Interview Reference