The latency numbers, and the derived facts

What it is

A table of the time each layer of the memory and network hierarchy takes, updated for current hardware, plus the conclusions that follow from it. Jeff Dean's "Latency Numbers Every Programmer Should Know" is the original; the numbers below reflect roughly 2020s hardware.

Operation                                    Time        Relative
--------------------------------------------------------------------
L1 cache reference                           0.5 ns          1x
Branch mispredict                              3 ns          6x
L2 cache reference                             4 ns          8x
Mutex lock/unlock (uncontended)               17 ns         34x
Main memory reference                        100 ns        200x
Compress 1 KB with a fast codec            2,000 ns          4k x
Read 1 MB sequentially from memory         3,000 ns          6k x
Send 1 KB over a 10 Gbps network             500 ns          1k x
Round trip within the same datacenter    500,000 ns          1M x
Read 1 MB sequentially from NVMe SSD      50,000 ns        100k x
NVMe random read (4 KB)                   20,000 ns         40k x
Disk seek (spinning)                   3,000,000 ns          6M x
Read 1 MB sequentially from spinning disk 5,000,000 ns       10M x
Round trip CA -> Netherlands -> CA     150,000,000 ns        300M x

Commonly confused with a memorisation exercise. The table is not the point; the ratios are, and an interviewer asking for it is testing whether you can turn them into a design argument rather than whether you have the numbers memorised.

Also commonly confused with something static. The relative ordering has been stable for decades and the absolute values have not: NVMe made random SSD reads roughly five times faster than the SATA figures in older versions of this table, which changes several conclusions below.

The problem it solves

Every architecture decision is a placement decision: where does this data live, and how many boundaries does a request cross to reach it. Without the ratios, that decision is made by taste. With them, it is arithmetic.

The specific arguments the table settles:

"Should we cache this?"          -> compare the compute cost against
                                    the 500 µs round trip to Redis
"Should we denormalise?"         -> compare one query against N
                                    round trips
"Is this microservice split
 worth it?"                      -> each split adds 500 µs, minimum,
                                    on the happy path
"Should we go multi-region
 active-active?"                 -> 150 ms cross-continent is not
                                    something you optimise away

Mechanics: the derived facts

These are what to have instantly available, because each one settles a design argument by itself.

1. Memory is 200 times slower than L1, and disk is 200 times slower than memory

L1              0.5 ns
Memory        100 ns      200x slower than L1
NVMe random  20,000 ns    200x slower than memory

Two clean orders of magnitude at each step. The consequence: cache-friendly data layout matters as much as algorithmic complexity for in-memory work, and an algorithm with better asymptotic complexity but worse locality routinely loses. An array scan can beat a linked list traversal at sizes where big-O says otherwise, because the array is prefetched and the list is a chain of cache misses.

2. A datacenter round trip costs 5,000 main-memory reads

500 µs / 100 ns = 5,000

This is the number that governs microservice granularity. Splitting a service adds one round trip to the happy path, and that round trip costs what five thousand memory accesses would. If the split saves less computation than that, it is a net loss on latency and it is being justified by organisational reasons rather than performance ones, which is fine as long as it is said out loud.

The corollary that matters more: N+1 query patterns are catastrophic at any N. Fifty sequential round trips is 25 ms of pure waiting, during which the CPU does nothing. That is the DataLoader argument in one line.

3. Sequential is 3 to 60 times faster than random, on every storage medium

Memory:   1 MB sequential      3 µs
NVMe:     1 MB sequential     50 µs   vs 4 KB random 20 µs
                                      (= 5,120 µs per MB random)
Spinning: 1 MB sequential      5 ms   vs a seek 3 ms

On NVMe, sequential is roughly 100 times faster per byte than random. This is why log-structured storage (LSM trees, write-ahead logs, Kafka) wins: it converts random writes into sequential appends. It is also why a full table scan can beat an index lookup when the selectivity is poor: the scan is sequential and the index is a chain of random reads, and query planners model exactly this trade.

4. Compression is usually free, and often negative-cost

Compress 1 KB:                    2 µs
Send 1 KB over 10 Gbps:         0.5 µs
Datacenter round trip:          500 µs

At first glance compression costs four times the transmission. But the round trip dominates both by three orders of magnitude, so for anything crossing a network, compression is free in latency terms and saves bandwidth and egress cost. Across regions, where the round trip is 150 ms, it is not close.

The exception worth naming: for very small payloads over a local socket, compression can genuinely cost more than it saves, which is why gRPC does not compress by default below a size threshold.

5. Cross-region is a physics problem, not an engineering one

CA to Netherlands, round trip: ~150 ms
Great-circle distance: ~8,900 km
Speed of light in fibre: ~200,000 km/s (2/3 of c)
Theoretical minimum round trip: 2 x 8,900 / 200,000 = 89 ms

We are within a factor of 1.7 of the speed of light.

There is no optimisation left. Cross-region latency is a placement decision: either the data is near the user or the request waits. That single fact is why multi-region active-active forces a choice between coordinating (and paying the round trip) and not coordinating (and resolving conflicts).

6. A mutex is cheap; contention is not

Uncontended mutex lock/unlock:    17 ns
Contended (with a context switch): 1-10 µs, so 60-600x

The lock is not the cost; the waiting is. Which is why lock-free structures win under contention and lose under none, and why the practical advice is to reduce the contended window rather than to eliminate locks.

A worked example: using the numbers to settle an argument

The proposal: split the user-profile lookup out of the search service into its own microservice.

CURRENT (in-process)
  Profile lookup from a local in-memory cache:  ~200 ns
  Cache miss (5%) -> Redis:                     ~500 µs
  Effective per request: 0.95 x 0.0002 + 0.05 x 500 = 25 µs

PROPOSED (separate service)
  Every request: a datacenter round trip:       ~500 µs
  Plus the profile service's own lookup:        ~25 µs
  Effective per request: ~525 µs

  *** 21x worse on the profile lookup, and it adds 500 µs to
      EVERY request rather than 5% of them. ***

The conclusion to state: at 20,000 requests per second, this adds 10 seconds of aggregate waiting per second of wall clock, which is 10 cores' worth of blocked threads. If the request budget is 50 ms, it consumes 1 percent of it, which may be acceptable. The decision is whether the organisational benefit is worth a hard 500 µs floor, and framing it that way is more useful than arguing about microservices in the abstract.

And the number that changes the answer: if the profile service can be co-located and called over a Unix socket, or if the data can be pushed to the search service and cached locally with change-data-capture invalidation, the round trip disappears. The best answer to a latency problem is usually to remove the boundary rather than to speed up the call.

A second worked example: does caching help?

Query: a 7-table join, 40 ms at the database.
Cache: Redis, same AZ, 500 µs round trip.

Speedup per hit: 40 ms / 0.5 ms = 80x

Hit rate needed for the cache to be worth its complexity?
  At 50% hit rate:  0.5 x 0.5 + 0.5 x 40 = 20.25 ms  (2x better)
  At 90% hit rate:  0.9 x 0.5 + 0.1 x 40 =  4.45 ms  (9x better)
  At 99% hit rate:  0.99 x 0.5 + 0.01 x 40 = 0.90 ms (44x better)

The shape to notice: the benefit is dominated by the miss rate, not the hit rate. Going from 90 to 99 percent hit rate is a 5x improvement; going from 50 to 90 is only 4.5x. The last few percent of hit rate are worth more than the first fifty, which is a counter-intuitive result that falls straight out of the arithmetic and is worth being able to state.

Production evidence

Jeff Dean's "Latency Numbers Every Programmer Should Know" (Google, various talks from 2009 onward) is the original table, and its purpose was explicitly to make back-of-envelope design possible rather than to be memorised.

Colin Scott's interactive "Latency Numbers Every Programmer Should Know" visualisation tracks the numbers by year and shows which have changed: network and disk have improved substantially, memory latency almost not at all, which is why the memory wall keeps widening.

The speed of light in fibre is roughly two thirds of $c$ due to the refractive index of glass, which is why the theoretical minimum for a transatlantic round trip is around 89 ms and why measured latencies of 150 ms represent a factor of 1.7 rather than an optimisation opportunity.

NVMe versus SATA SSD figures differ by roughly a factor of five on random reads, which is why tables published before about 2015 substantially understate SSD performance and lead to wrong conclusions about when to cache.

Brendan Gregg's "Systems Performance" provides the measured methodology behind numbers of this kind and is the reference for actually verifying them on your own hardware, which is the right response to any specific claim.

The debate

The case for memorising the table: it lets you do capacity and placement arithmetic in the room, without a laptop, and interviewers at large companies expect it. More practically, it makes bad designs visible immediately: a proposal with fifty sequential service calls is 25 ms of pure waiting and you can say so in the meeting rather than discovering it in a load test.

The case against: the absolute numbers change, they vary by an order of magnitude across cloud instance types, and a candidate reciting 2009 figures for SSD random reads will reach wrong conclusions. Measuring your own system is always better than quoting a table.

My position: memorise the ratios, not the absolute values, and know which numbers have moved. The ratios have been stable for decades: L1 to memory is roughly 200x, memory to random NVMe is roughly 200x, and a datacenter round trip is roughly 5,000 memory accesses. Those three carry almost all the design value and they will still be approximately right in five years.

The absolute values I would treat as needing verification, particularly storage, because NVMe changed random-read performance by a factor of five and any conclusion drawn from pre-2015 SSD figures about when to cache is likely wrong.

The derived facts I would actually deploy in a design discussion are three. A datacenter round trip costs 5,000 memory accesses, which settles microservice granularity and makes N+1 patterns obviously fatal. Sequential beats random by about 100x per byte on NVMe, which explains why every high-throughput storage system is log-structured. And cross-region is within a factor of two of the speed of light, which means it is a placement decision rather than a performance problem, and no amount of engineering removes it.

Where I would push back on the question itself: if an interviewer asks for the table as a recitation, the useful answer is to give three ratios and then immediately use them. The table is a tool and demonstrating the tool being used is worth more than demonstrating that you own it.

Follow-up Q&A

"What are the latency numbers?" I would give the ratios rather than a list, because the ratios are what survive hardware changes. L1 is about half a nanosecond, main memory about 100, so memory is roughly 200 times slower than L1. Random NVMe is about 20 microseconds, so another 200 times slower than memory. A same-datacenter round trip is about 500 microseconds, which is 5,000 memory accesses. And a cross-continental round trip is about 150 milliseconds, which is within a factor of two of the speed of light in fibre.

"What follows from a datacenter round trip being 500 microseconds?" Two things. Splitting a service adds that to the happy path, permanently, and it costs what 5,000 memory accesses would, so if the split saves less computation than that it is a net latency loss and is being justified organisationally rather than technically. And N+1 patterns are catastrophic at any N: fifty sequential calls is 25 milliseconds of pure waiting with the CPU idle, which is why batching or a DataLoader is not an optimisation but a correctness-of-design issue.

"Why is sequential access so much faster than random?" Because of prefetching and because of how storage devices actually work. On NVMe, one megabyte sequential is about 50 microseconds, while a megabyte of 4-kilobyte random reads is about 5,000, so roughly 100 times per byte. That is the reason every high-throughput storage system is log-structured: LSM trees, write-ahead logs and Kafka all convert random writes into sequential appends. It is also why a full scan can beat an index lookup at poor selectivity, and query planners model exactly that.

"Is compression worth it?" Almost always, for anything crossing a network. Compressing a kilobyte is about 2 microseconds and sending it over 10 gigabit is about half a microsecond, so compression looks four times more expensive. But the round trip is 500 microseconds, three orders of magnitude larger than both, so compression is free in latency terms and saves bandwidth and egress cost. Across regions at 150 milliseconds it is not close. The exception is very small payloads over a local socket, which is why gRPC has a size threshold.

"Can you optimise cross-region latency?" Essentially no, and that is the useful answer. California to the Netherlands is about 8,900 kilometres, light in fibre travels at about two thirds of $c$, so the theoretical minimum round trip is 89 milliseconds and real measurements are around 150. That is a factor of 1.7 from physics, so there is no engineering left. It makes cross-region a placement decision: either the data is near the user or the request waits, which is exactly why active-active designs have to choose between coordinating and resolving conflicts.

"How do you use these in a design discussion?" As arithmetic that settles arguments. If someone proposes splitting a lookup into its own service, I can say it adds a hard 500 microsecond floor to every request rather than the 25 microseconds it costs in-process, so 21 times worse, and at 20,000 requests per second that is 10 cores' worth of blocked threads. Then the conversation becomes whether the organisational benefit is worth that, which is the real question, rather than an abstract argument about microservices.

"Which of these numbers have actually changed?" Storage, dramatically. NVMe random reads are roughly five times faster than the SATA SSD figures in tables published before about 2015, so any conclusion about when caching pays that was drawn from older numbers is probably wrong. Network has improved substantially too. Main memory latency has barely moved in twenty years, which is why the gap between CPU and memory keeps widening and why cache-friendly data layout keeps getting more important relative to algorithmic complexity.

"Where does the caching arithmetic get counter-intuitive?" The benefit is dominated by the miss rate rather than the hit rate. With a 40 millisecond query and a 500 microsecond cache, going from 50 to 90 percent hit rate is about 4.5 times better, and going from 90 to 99 percent is another 5 times. So the last few percent of hit rate are worth more than the first fifty, which is why cache-key design and stampede protection matter more than people expect once you are already at a decent hit rate.

Common misconceptions

"These numbers are current." Storage figures in older versions of the table predate NVMe and understate random reads by roughly five times. Verify anything storage-related.

"Memorising the table is the skill." Using three ratios to settle a design argument is the skill. The table is the tool.

"An SSD makes random access cheap." It makes it much cheaper. Sequential is still about 100 times faster per byte on NVMe, which is why log-structured designs still win.

"Cross-region latency can be engineered away." It is within a factor of two of the speed of light. It is a placement decision.

"Locks are slow." An uncontended mutex is 17 nanoseconds. Contention with a context switch is 60 to 600 times that. Reduce the contended window, not the number of locks.

Interview delivery note

Give ratios rather than a recitation, and then immediately use one: "The three I actually use are: memory is about 200 times slower than L1, random NVMe is about 200 times slower than memory, and a datacenter round trip is about 500 microseconds, which is 5,000 memory accesses. That last one is the one that settles arguments."

Then demonstrate it on whatever is being discussed: "So if we split that lookup into its own service, we're adding a hard 500 microsecond floor to every request where it currently costs 25 microseconds in-process. At twenty thousand requests a second that's ten cores' worth of blocked threads. Which might be fine, but the trade is organisational benefit against a permanent latency floor, and that's the conversation rather than microservices in the abstract."

The number that lands hardest, because it closes off a whole category of discussion: "cross-region is 150 milliseconds round trip and the speed of light in fibre puts the floor at 89. We're within a factor of 1.7 of physics, so there's no optimisation available. It's a placement decision: either the data is near the user or the request waits."

And show calibration about the table itself: "though I'd verify anything storage-related rather than quoting it, because NVMe changed random reads by about five times and a lot of circulating versions of this table predate it."

The Staff & Team Lead Interview Reference