Why you cannot average percentiles

What it is

A percentile is a quantile of a distribution, and quantiles do not combine linearly. Averaging the p99 values reported by ten instances does not give you the fleet's p99, and the error is not small.

Instance A: p99 = 100 ms, serving 1,000 requests
Instance B: p99 = 100 ms, serving     10 requests

mean(p99) = 100 ms

The fleet's true p99 is dominated by A's distribution, because A
has 99% of the requests, and B contributes almost nothing. Here
they happen to agree, and that is luck rather than arithmetic.

The failure is much larger when the distributions differ:

Instance A: 1,000 requests, all at 10 ms.        p99 = 10 ms
Instance B:    10 requests, all at 5,000 ms.     p99 = 5,000 ms

mean(p99) = 2,505 ms

TRUE fleet p99: sort all 1,010 requests, take the 999th.
  1,000 requests at 10 ms, then 10 at 5,000 ms.
  The 999th value is 10 ms.
  *** True p99 = 10 ms. The average said 2,505 ms. ***
  Off by 250x, and in the direction that makes you chase a
  problem that does not exist.

Commonly confused with a precision issue. It is not approximation error; it is a category error. A percentile is a property of a distribution, and you cannot recover a distribution's quantile from other distributions' quantiles without the underlying data.

Also commonly confused with the related but distinct fact that percentiles cannot be averaged over time either. The mean of 24 hourly p99 values is not the day's p99, for the same reason.

The problem it solves

Every monitoring system makes this easy to do wrong, and the resulting numbers are used for SLOs, capacity decisions and incident triage.

Prometheus:
  avg(http_request_duration_p99)         WRONG, silently
  avg by (service) (rate(...))           fine, rates DO average

Datadog / New Relic / CloudWatch:
  A p99 metric aggregated across hosts is often a mean of
  per-host p99s unless you configure otherwise.

CloudWatch specifically:
  Percentile statistics on a metric with multiple dimensions
  are computed per dimension, and the dashboard's aggregation
  across them is not a true percentile.

The consequences are practical rather than academic:

Under-reporting  You believe the p99 is 80 ms; users experience
                 400 ms. The SLO says you are fine and support
                 tickets say you are not.
Over-reporting   You chase a phantom regression caused by one
                 low-traffic instance, as in the example above.
Wrong triage     You cannot tell whether one instance is slow or
                 all of them are, which is the first question in
                 a latency incident.

Mechanics

Why the arithmetic fails

A percentile is defined by ORDER over the full sample.
p99 = the value below which 99% of observations fall.

To compute it for a union of sets, you need the union's
ORDERING, which requires either the raw values or a
structure that preserves enough distributional information.

Per-instance p99 values discard exactly that: each is a
single number summarising a distribution, and the summary
is not composable.

Nor are percentiles averageable over time, which is the same fact in a different direction:

Hour 1: p99 = 100 ms over 1M requests
Hour 2: p99 = 900 ms over 1K requests (a deploy, low traffic)

mean = 500 ms.
True daily p99: the slow hour is 0.1% of requests, so it
barely moves the 99th percentile of the day. True p99 ≈ 105 ms.

What to do instead: histograms

The correct approach is to aggregate the distribution, not the summary.

Each instance exports a HISTOGRAM: bucket counts, not a
percentile.

  le=0.005  count=  120
  le=0.010  count= 3,400
  le=0.025  count= 8,900
  le=0.050  count= 9,700
  le=0.100  count= 9,950
  le=0.250  count= 9,990
  le=+Inf   count=10,000

Bucket counts ARE additive. Sum them across instances, then
compute the percentile from the summed histogram.

# CORRECT. Sum the bucket counters across instances first,
# then compute the quantile from the aggregate distribution.
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

# WRONG, and it is the query people write.
avg(http_request_duration_seconds_p99)

The sum by (le) is the entire correction, and it works because the histogram's bucket counters are counts, and counts add.

The cost of histograms, and the accuracy trade

Histograms are not free and their accuracy is bounded by bucket layout.

Prometheus classic histograms:
  One time series PER BUCKET, per label combination.
  A 12-bucket histogram with 5 label values on 3 dimensions
  is 12 x 5^3 = 1,500 series for one metric.
  -> This is the cardinality cost model, and it is why
     histogram bucket count is a real decision.

Accuracy:
  histogram_quantile INTERPOLATES LINEARLY within the bucket
  containing the target quantile. If p99 falls in the bucket
  [0.1, 0.25], the answer is somewhere in that range and the
  interpolation assumes a uniform distribution inside it,
  which is usually false.
  *** Your p99 is only as accurate as your bucket boundaries
      near the p99. ***

Which produces the practical rule: put bucket boundaries where you care.

DEFAULT (Prometheus): .005 .01 .025 .05 .1 .25 .5 1 2.5 5 10
  Good for a 100 ms service; useless for a 3 ms one, where
  everything lands in the first bucket and the p99 is
  interpolated across [0, 0.005].

TUNED for a service with a 300 ms SLO:
  .05 .1 .15 .2 .25 .3 .35 .4 .5 .75 1 2.5
  Dense around the SLO threshold, sparse elsewhere.

Native histograms (Prometheus 2.40+, experimental then stable) change this: exponential buckets with a configurable resolution, stored as one series rather than one per bucket, giving much better accuracy at a fraction of the cardinality. Naming them is a good signal that you have followed the tooling.

The alternative: t-digest and DDSketch

Sketch data structures that ARE mergeable and give bounded
RELATIVE error at the tails, which is where percentiles matter.

t-digest      Relative error is smallest at the extremes,
              which is exactly where you want it for p99 and
              p999. Used by Elasticsearch's percentile
              aggregation and by many APM vendors.

DDSketch      Guarantees relative error at ANY quantile, and
              is fully mergeable. Used by Datadog.

The trade against histograms: sketches give better tail accuracy without needing bucket boundaries tuned per service, at the cost of a more complex data structure and a non-standard wire format. For a fleet with heterogeneous latency profiles, sketches avoid the per-service bucket-tuning problem entirely, which is a real operational advantage.

"Our p99 latency is 200 ms" for a request that fans out to 5
backends, each with a p99 of 200 ms.

The request waits for the SLOWEST backend, so its latency is
the MAXIMUM of 5 samples, not one sample.

P(all 5 under their p99) = 0.99^5 = 0.951
-> 4.9% of requests have at least one backend above its p99.
-> The request's p99 corresponds to roughly each backend's
   p99.8, not its p99.

This is the tail-at-scale result and it compounds with the averaging error: a fleet whose reported p99 is already wrong is then used to reason about a fan-out that makes the true tail worse still. See the tail at scale.

What you can and cannot aggregate

CAN AGGREGATE ACROSS INSTANCES
  counters              sum
  rates                 sum (they are counters per second)
  histogram buckets     sum
  means                 weighted by count
  min / max             min / max

CANNOT
  percentiles           no
  medians               no (a median IS a percentile)
  standard deviations   not without sums of squares
  unique counts         no (need HyperLogLog or the raw set)

"Means can be aggregated if weighted by count" is worth stating, because it is the one exception people assume applies to percentiles too. A mean is a sum divided by a count, and both of those are additive; a percentile is an order statistic, and order is not.

A worked example: an incident that was not

ALERT: "API p99 latency 2.4 s, threshold 500 ms."
Dashboard: avg(api_latency_p99) across 40 pods.

INVESTIGATION
  Per-pod p99, sorted:
    pod-31    41,200 ms      <-- 
    pod-07       210 ms
    pod-12       195 ms
    ... 37 more pods, all 180-260 ms

  avg of 40 values with one at 41,200:
    (41,200 + 39 x ~210) / 40 = 1,235 ms
  The dashboard's higher number came from a shorter window
  and more extreme skew.

  TRUE fleet p99, from summed histogram buckets: 268 ms.

WHAT WAS ACTUALLY HAPPENING
  pod-31 had been serving 3 requests per minute since a
  readiness probe flap, and those 3 requests were long-poll
  connections. Its p99 was computed over 3 samples and was
  meaningless.
  The fleet was healthy. The alert was arithmetic.

THE FIXES
  1. Alert on histogram_quantile over SUMMED buckets, never
     on an average of per-instance percentiles.
  2. Suppress percentile computation below a minimum sample
     count. A p99 over 3 samples is not a p99.
  3. Alert SEPARATELY on per-pod outliers, because "one pod
     is slow" is a real and different signal that the fleet
     aggregate correctly hides.

Point 3 is the subtlety worth stating: the correct fleet percentile deliberately hides a single bad instance, which is right for an SLO and wrong for triage. You need both signals, computed differently, and conflating them is what produced the incident.

Production evidence

Prometheus's histogram_quantile documentation states explicitly that you must aggregate the bucket counters with sum by (le) before computing the quantile, and warns about interpolation error within buckets.

Prometheus native histograms (2.40 onward) were introduced specifically to address the cardinality cost of classic bucketed histograms and their fixed-boundary accuracy problem, which is direct evidence that both were real operational limits.

Dunning and Ertl's t-digest paper documents the mergeable-sketch approach with relative error smallest at the extremes, and Elasticsearch's percentile aggregation uses it, with documentation noting the accuracy characteristics.

Datadog's DDSketch paper (Masson, Rim and Lee, VLDB 2019) gives a mergeable sketch with a relative-error guarantee at any quantile, and their write-ups on why averaging percentiles is wrong are among the clearest public explanations.

Gil Tene's "How NOT to Measure Latency" talk is the canonical treatment of percentile misuse, including coordinated omission, which is a related and equally serious measurement error where a load generator stops sending during a stall and therefore fails to record the worst latencies at all.

AWS CloudWatch's documentation on percentile statistics notes that percentiles are computed per-metric and that cross-dimension aggregation does not produce a true percentile, which is the vendor stating the problem.

The debate

The case for histograms everywhere: correct aggregation, standard tooling, and bucket counters are cheap counters. The interpolation error is bounded and manageable if the buckets are placed sensibly.

The case for sketches (t-digest, DDSketch): better tail accuracy with a relative-error guarantee, no per-service bucket tuning, and full mergeability. For a heterogeneous fleet where every service has a different latency profile, tuning bucket boundaries per service is work that sketches eliminate.

The case for just using averages: they aggregate correctly, they are cheap, and for many purposes the mean is a fine signal. It is also true that the mean hides exactly the tail behaviour that percentiles exist to expose, so it is not a substitute.

My position: histograms with buckets tuned around the SLO threshold, aggregated with sum by (le), plus a separate per-instance outlier alert.

The correction itself is one clause, sum by (le) before histogram_quantile, and getting it wrong is silent, so it belongs in a review checklist rather than in individual judgement. The default query people write is the wrong one.

The bucket placement is the part that gets neglected. Your p99 is only as accurate as the bucket boundaries near your p99, because histogram_quantile interpolates linearly inside the containing bucket and assumes a uniform distribution there, which is almost never true. Prometheus's default buckets are tuned for a roughly 100 millisecond service and are actively misleading for a 3 millisecond one, where everything lands in the first bucket. So: dense buckets around the SLO threshold, sparse elsewhere.

The design decision I would defend hardest is two alerts rather than one. The correctly aggregated fleet percentile deliberately hides a single bad instance, which is right for an SLO and useless for triage, because "is one instance slow or are all of them" is the first question in a latency incident. Conflating them is what produced the phantom incident in the example: an average-of-percentiles alert that was really an outlier detector with wrong arithmetic.

And a guard that is cheap and rarely present: suppress percentile computation below a minimum sample count. A p99 over three samples is not a p99, and a pod serving three requests a minute after a probe flap will produce arbitrary numbers that then pollute any aggregate that touches them.

Where I would push further than the question asks: coordinated omission is the larger measurement error and almost nobody checks for it. If a load generator waits for a response before sending the next request, then during a stall it sends nothing, so the worst latencies are never recorded at all and the reported p99 is optimistic by orders of magnitude. That is worth raising whenever someone quotes a benchmark percentile.

Follow-up Q&A

"Why can't you average percentiles?" Because a percentile is an order statistic over a distribution, and you cannot recover the union's order statistic from the components' order statistics. It is a category error rather than an approximation error. The concrete version: a thousand requests at 10 milliseconds and ten requests at 5,000 gives per-instance p99s of 10 and 5,000, whose average is 2,505, while the true p99 over all 1,010 requests is 10, since the slow ten are only one percent. Off by 250 times, in the direction that makes you chase a problem that does not exist.

"What do you do instead?" Aggregate the distribution, not the summary. Each instance exports histogram bucket counts, which are counters and therefore additive, so you sum the buckets across instances and compute the quantile from the aggregate. In Prometheus that is histogram_quantile(0.99, sum by (le) (rate(..._bucket[5m]))), and the sum by (le) is the entire correction. The query people actually write is avg of a p99 gauge, which is silently wrong.

"How accurate is that?" Only as accurate as your bucket boundaries near the percentile you care about, because histogram_quantile interpolates linearly within the bucket containing the target and assumes a uniform distribution inside it, which is almost never true. Prometheus's defaults are tuned for a roughly 100 millisecond service; for a 3 millisecond service everything lands in the first bucket and the p99 is interpolated across zero to five milliseconds, which is meaningless. So place buckets densely around the SLO threshold and sparsely elsewhere.

"What's the cost of histograms?" Cardinality. Classic Prometheus histograms are one time series per bucket per label combination, so a twelve-bucket histogram with three label dimensions of five values each is 1,500 series for one metric. That is why bucket count is a real decision rather than a formality. Native histograms address it with exponential buckets stored as a single series, which gives better accuracy at a fraction of the cardinality.

"What about t-digest and DDSketch?" Mergeable sketches with bounded relative error at the tails, which is where percentiles matter. t-digest has smallest error at the extremes and is what Elasticsearch's percentile aggregation uses; DDSketch guarantees relative error at any quantile and is what Datadog uses. Their advantage over histograms is that they need no per-service bucket tuning, which for a heterogeneous fleet is real operational work eliminated.

"Can you average percentiles over time?" No, same reason. The mean of 24 hourly p99s is not the day's p99. If one hour has a p99 of 900 milliseconds over a thousand requests and the others are 100 over a million each, the slow hour is a tenth of a percent of the day's traffic and barely moves the daily 99th percentile, while the average of the hourly values reports roughly 500.

"What can you aggregate?" Counters, rates and histogram buckets, all by summing. Means, if weighted by count, because a mean is a sum over a count and both are additive. Min and max. What you cannot aggregate is percentiles, medians (which are percentiles), standard deviations without the sums of squares, and unique counts without a sketch. The weighted-mean exception is worth knowing because it is the one people assume extends to percentiles.

"Your p99 alert fired but the fleet was healthy. What happened?" Almost certainly an average of per-instance percentiles, where one pod serving three requests a minute after a readiness probe flap computed a p99 over three long-poll connections and dragged the average up by orders of magnitude. Three fixes: alert on the summed-histogram quantile, suppress percentile computation below a minimum sample count because a p99 over three samples is not a p99, and add a separate per-instance outlier alert, because the correct fleet percentile deliberately hides one bad instance and that is a real signal you still want.

"Anything else people get wrong about percentiles?" Coordinated omission, and it is larger. If a load generator waits for a response before sending the next request, then during a stall it sends nothing, so the worst latencies are never recorded and the reported p99 is optimistic by orders of magnitude. Gil Tene's talk is the canonical treatment. It is worth raising whenever someone quotes a benchmark percentile, because most load tools have the problem by default.

Common misconceptions

"Averaging percentiles is approximately right." It is a category error and the result can be off by orders of magnitude in either direction.

"A p99 is a p99." It depends on the bucket boundaries, the aggregation window, the sample count and whether the measurement suffered coordinated omission.

"The monitoring tool handles it." Most dashboards make the wrong aggregation the default, and it fails silently.

"Means have the same problem." Means aggregate correctly when weighted by count. That is why they are the one exception.

"A correct fleet percentile is enough." It hides a single bad instance by construction, which is right for an SLO and wrong for triage. You need both signals.

Interview delivery note

Lead with a number, because the magnitude of the error is what makes the point: "You can't, and the error isn't small. A thousand requests at ten milliseconds and ten at five seconds gives per-instance p99s of ten and five thousand, so the average is 2,505. The true p99 over all of them is ten, because the slow ten are one percent of the sample. Off by 250 times, and in the direction that makes you chase a problem that doesn't exist."

Give the reason as a category distinction: "A percentile is an order statistic over a distribution, so you'd need the union's ordering to compute it, and per-instance percentiles have thrown exactly that away. It's not an approximation error, it's the wrong operation."

Then the fix and the clause that is the fix: "So aggregate the distribution rather than the summary. Export histogram buckets, which are counters and therefore additive, sum them across instances, and compute the quantile from the aggregate. In Prometheus the whole correction is sum by (le) before histogram_quantile, and the query people actually write is an avg of a p99 gauge."

Volunteer the accuracy caveat, because it shows you have tuned one: "and your p99 is only as accurate as the bucket boundaries near it, because histogram_quantile interpolates linearly inside the containing bucket. Prometheus's defaults are tuned for a hundred-millisecond service; on a three-millisecond service everything lands in the first bucket and the p99 is interpolated across zero to five, which is meaningless."

Close with the design point, which is where the judgement is: "and I'd run two alerts, not one. The correct fleet percentile hides a single bad instance by construction, which is right for an SLO and useless for triage, because 'is one host slow or all of them' is the first question in a latency incident. Conflating them is exactly what produces phantom alerts."

The Staff & Team Lead Interview Reference