Cardinality: the observability cost model

"A label explosion took down Prometheus. What happened, and what's the policy fix?"

What it is

In a dimensional metrics system, a time series is identified by a metric name plus its complete set of label key-value pairs. http_requests_total{method="GET", status="200", route="/orders"} and http_requests_total{method="GET", status="500", route="/orders"} are two distinct series with independent storage, independent indexes and independent memory.

Cardinality is the number of distinct series, and it is the multiplicative product of the distinct values of every label:

$$\text{series} = \prod_{i} |\text{values}(\text{label}_i)|$$

Add a label with 1,000 possible values to a metric that had 200 series, and you now have 200,000. Add a label whose values are unbounded (user ID, request ID, customer email, full URL path with IDs in it) and you have an unbounded number of series, which is a memory leak with a dashboard on top.

Cardinality is commonly confused with volume. Ingesting a million samples per second into ten series is cheap; ingesting a thousand samples per second into a million series is not. The cost model is series count, not sample rate. Getting that backwards is why teams add a label to "get better visibility" and take the metrics system down.

The problem it solves

Nothing. Cardinality is not a feature, it is the bill. The reason to understand it is that dimensional metrics are otherwise so pleasant that the bill arrives as a surprise, and it arrives as an outage of the system you use to detect outages.

Two properties make it particularly nasty. The failure is delayed: a label added on Monday causes an out-of-memory kill on Thursday, when enough distinct values have accumulated. And the failure is self-concealing: when the metrics backend dies, so do the metrics you would use to diagnose it, and every alert based on those metrics either fires spuriously or stops firing altogether.

Mechanics

Why series are expensive

Prometheus keeps every active series in memory: the label set, an index entry per label pair, and an open chunk of recent samples. Community measurements consistently land in the range of a few kilobytes of resident memory per active series, and that number is the one to hold.

At roughly 3 KB per series:

Active series	Approximate memory
100,000	0.3 GB
1,000,000	3 GB
5,000,000	15 GB
20,000,000	60 GB

The memory is not the whole cost. The inverted index that maps label pairs to series grows too, so query planning slows down. A query like sum(rate(http_requests_total[5m])) by (route) must resolve the matcher to a set of series and then merge them, so its cost scales with the number of matching series regardless of how few samples each holds.

And there is a second-order effect that turns a degradation into an outage: write-ahead log replay on restart. A Prometheus with 15 million series takes a long time to come back after a crash, during which it is scraping nothing. So the OOM kill is followed by an extended blind window, and if the pod is memory-constrained it may OOM again during replay, producing a crash loop that never recovers without intervention.

Churn: the cardinality you cannot see

Active series drive memory. Total series over the retention window drive disk and index size. The gap between them is churn.

A label whose value changes frequently (a pod name in a deployment that rolls hourly, a Kubernetes pod IP, a build SHA) creates a new series each time. At any instant the active count looks fine. Over a week the index has accumulated every pod that ever existed.

# Active series right now
prometheus_tsdb_head_series

# Series created per second: the churn rate. This is the leading indicator.
rate(prometheus_tsdb_head_series_created_total[10m])

# Top offenders by metric name (Prometheus 2.14+ exposes this directly)
topk(10, count by (__name__)({__name__=~".+"}))

That last query is the one to run first in an incident. It answers "which metric exploded" in about two seconds, and the answer is almost always a metric someone added last week.

The four classic offenders

Identifiers as labels. user_id, request_id, session_id, order_id, trace_id. Unbounded by definition. This is the number one cause.
Unparameterised URL paths. path="/orders/8f2a91" creates one series per order. The fix is route templating: route="/orders/{id}". Frameworks that auto-instrument HTTP handlers usually do this correctly; hand-rolled middleware usually does not.
Error messages or free text. error="connection refused to 10.0.3.44:5432" embeds an address. Use a bounded error_class instead.
Cross products. Five labels with 10, 20, 50, 8 and 100 values is $10 \times 20 \times 50 \times 8 \times 100 = 8{,}000{,}000$ potential series from a single metric. Each label looked reasonable in isolation. Multiply before you ship.

The three-layer fix

Policy alone does not work, because policy is a document and people ship code. Enforcement alone does not work, because it tells you no without telling you why. You need all three.

Layer 1: a rule with a rationale. "No unbounded label values in metrics." The rationale that makes it stick: metrics are for aggregates, traces and logs are for individuals. If you want to know which user was affected, that is a trace lookup or a log query, not a metric dimension. Say this once and most of the problem disappears, because the request for a user_id label is nearly always a request for the wrong telemetry type.

Layer 2: enforcement in the collection path.

# Prometheus scrape config: drop the offending label before ingestion.
metric_relabel_configs:
  # Remove a known-bad label entirely.
  - regex: 'user_id|session_id|request_id'
    action: labeldrop
  # Or template a path that slipped through unparameterised.
  - source_labels: [path]
    regex: '/orders/[0-9a-f]+'
    target_label: path
    replacement: '/orders/{id}'
  # Hard ceilings: refuse a target that misbehaves rather than absorbing it.
sample_limit: 20000          # per scrape
label_limit: 30              # labels per series
label_value_length_limit: 200

sample_limit is the important one and it is underused. A target that suddenly emits 200,000 series gets its scrape rejected and an alert, instead of taking down the server. Failing one target loudly beats degrading the whole system silently, and that is the design principle to articulate.

The same enforcement in an OpenTelemetry Collector, which is where it belongs if you have one, because it is central and language-agnostic:

processors:
  attributes/scrub:
    actions:
      - key: user.id
        action: delete
      - key: http.route
        action: extract
        pattern: ^(?P<route>/[a-z]+)/.*$

The Collector is the right enforcement point for the same reason a shared linter beats a style guide: one place to change, applies to every service, and does not depend on every team remembering.

Layer 3: monitoring the monitoring.

# Alert well before the ceiling. The threshold is your capacity, not a constant.
- alert: PrometheusCardinalityHigh
  expr: prometheus_tsdb_head_series > 4000000
  for: 30m

# Churn is the leading indicator: it moves days before active series do.
- alert: PrometheusSeriesChurnHigh
  expr: rate(prometheus_tsdb_head_series_created_total[1h]) > 200
  for: 1h

Where the high-cardinality data should go

The rule "no unbounded labels" only holds if there is somewhere else to put the data. Three places, and you should be able to say which is which:

Traces, with exemplars linking a histogram bucket to a specific trace. This is the modern workflow: the metric tells you p99 got worse, the exemplar takes you to an actual slow request. It gives you the per-request detail without the per-request series.
Structured logs, sampled, with a trace ID for correlation.
Wide events: one richly annotated event per request, queried arbitrarily. This is the Honeycomb-style argument that dimensional metrics are the wrong primitive because they force you to decide your dimensions before the incident. It is a real position and worth having an opinion about.

A worked example: the incident

A payments team adds a label to their existing request counter so they can see which merchant is generating errors:

# Before: 4 methods x 6 status codes x 12 routes = 288 series
requests.labels(method=m, status=s, route=r).inc()

# After: x merchant_id
requests.labels(method=m, status=s, route=r, merchant_id=mid).inc()

There are 14,000 active merchants. Potential cardinality is $288 \times 14{,}000 = 4$ million series from one metric. In practice each merchant only exercises a few route and status combinations, so the realised count lands nearer 400,000, which is still a 1,400-fold increase.

Day 1. Deployed at 14:00. Series count climbs from 1.1 million to 1.4 million. Prometheus memory goes from 4 GB to 5 GB. Nobody notices.

Day 3. Long-tail merchants trickle in, plus the natural churn of pod restarts creating new series for each. 2.9 million series, 9 GB. Dashboard queries that were 200 ms are now 3 seconds. An engineer mentions Grafana feels slow.

Day 4, 09:20. A traffic peak pushes it over the 12 GB memory limit. OOMKill. The pod restarts and begins WAL replay of 3 million series, which takes several minutes, during which nothing is being scraped. Alerting rules cannot evaluate, so every alert enters an unknown state.

09:31. Replay completes. Prometheus starts scraping, immediately reallocates the same series, and OOMs again. Crash loop.

09:40. On-call is now debugging a production system with no metrics at all, and the SLO burn-rate alerts have been silent for twenty minutes, which means the team cannot tell whether anything else is also broken.

The response. Add a metric_relabel_config dropping merchant_id, raise the memory limit temporarily so the pod can complete replay, restart, confirm series count falls, then remove the temporary limit. Total incident: 55 minutes of no observability.

The follow-up that matters. Not "be careful with labels". Three concrete changes: sample_limit: 20000 on every scrape config, so a single misbehaving target is rejected rather than absorbed; a cardinality alert at 60 percent of capacity with a churn alert alongside it; and a documented answer for the original question, which was legitimate. Per-merchant error rates belong in a trace-derived metric with a bounded top-N, or in the logs, or in a separate purpose-built store, not as a Prometheus label.

That last point is what makes the postmortem useful. The engineer wanted something reasonable. If the answer is only "don't", they will ask again next quarter.

Production evidence

Prometheus's own documentation states the rule directly in its instrumentation best practices: keep cardinality low, and do not use labels for values with unbounded ranges such as user IDs or email addresses. sample_limit, label_limit and metric_relabel_configs exist in the scrape configuration specifically as guard rails, which is the maintainers' acknowledgement that policy is insufficient.

Grafana built Mimir and Cortex with per-tenant series limits as a first-class feature, because in a multi-tenant metrics platform one tenant's label explosion is everyone's outage. The existence of tenant-level cardinality quotas in every hosted metrics product is strong evidence about how routine this failure is.

Datadog bills on custom metrics, where a custom metric is defined as a unique combination of metric name and tag values, which makes cardinality literally the pricing model. Their documentation on high-cardinality tags is written as cost guidance rather than reliability guidance, and both framings are correct.

Honeycomb built their product on the opposite premise: store wide events with arbitrary high-cardinality fields and query them, rather than pre-aggregating into series. Charity Majors has written extensively on why high cardinality is exactly what you need during an incident and why the metrics model makes you choose your dimensions before you know what went wrong. It is the strongest counter-argument and worth citing by name.

The debate

The alternative to policing cardinality is choosing a system that does not charge for it. Wide-event stores, columnar log platforms and trace backends can absorb per-request identifiers because they store events rather than series, and they let you slice by any field after the fact.

The case for that model is genuinely strong: the dimension you need during an incident is the one you did not think to add, and dimensional metrics force the decision in advance. The case against is cost at high volume (you are storing every event rather than counters), query latency for long time ranges, and the fact that alerting on a pre-aggregated counter is cheap and reliable in a way that querying an event store is not.

My position: metrics for the things you alert on, traces and wide events for the things you investigate with. Keep metric cardinality deliberately low and bounded, enforce it in the collection pipeline rather than by policy, and link the two with exemplars so a bad p99 is one click from an actual slow request. The teams that get into trouble are the ones trying to make one system do both jobs.

Cardinality control is the wrong focus when the real problem is that the team has no traces, so metrics are the only tool available and every question becomes a label request. In that situation, adding trace collection removes the pressure entirely, and telling someone "no" without giving them somewhere to go is how you get the label added anyway with a # TODO: remove comment.

Follow-up Q&A

"A label explosion took down Prometheus. What happened and what's the policy fix?" Someone added a label whose values are unbounded, usually an ID, so series count is now the product of every label's cardinality and it grows without limit. Each active series costs a few kilobytes of resident memory plus index, so the server OOMs; then WAL replay of millions of series delays recovery, and during replay nothing is scraped and no alerts evaluate. The fix is three layers: a rule that metrics are for aggregates and identifiers belong in traces or logs; enforcement in the collection path with metric_relabel_configs and a sample_limit so a bad target fails loudly instead of degrading everything; and alerting on prometheus_tsdb_head_series and the churn rate at a fraction of capacity.

"How do you find the offending metric quickly?" topk(10, count by (__name__)({__name__=~".+"})) gives you series count per metric name in one query. Then count(count by (label_name)(metric_name)) for each label on the suspect metric identifies which dimension is unbounded. If Prometheus is too degraded to answer queries, the TSDB has a tsdb-status page with the top series by metric and by label, and promtool tsdb analyze works against the data directory offline.

"Someone needs per-user error rates. What do you tell them?" That the metric is the wrong instrument, and then where to go instead. Per-user detail lives in traces or logs, correlated by trace ID, and exemplars link the aggregate metric to a concrete example. If they genuinely need alerting on a per-tenant basis, options are a bounded top-N (track the 50 largest tenants explicitly, aggregate the rest into an other bucket), a separate purpose-built store with tenant quotas, or a trace-derived metric with sampling. The answer that fails is "no", because they will find a way and you will find out during the next incident.

"What is churn and why does it matter more than the active count?" Churn is the rate at which new series are created. Active series drive memory, but total series over the retention window drive index and disk, and a label that changes value frequently (pod name, container ID, build SHA) generates new series continuously while the active count stays flat. So a system can look healthy on the memory graph and be accumulating an index that makes every query slower, until a restart turns a slow WAL replay into an outage. rate(prometheus_tsdb_head_series_created_total[10m]) is the leading indicator and it moves days before memory does.

"Does the same problem exist in logs and traces?" Not in the same form, because neither pre-aggregates by dimension. Logs cost by volume and index shape; traces cost by sampled span count. Both have their own version of the failure, though: an unbounded number of distinct field names (as opposed to values) does hurt a columnar log store, because each field becomes a column. High-cardinality values are fine in logs and traces; high-cardinality schemas are not.

Common misconceptions

The most damaging is that cost scales with the number of data points. It scales with the number of series. A metric scraped once a minute with a million series costs far more than one scraped every second with ten.

The second is that a label with a bounded set of values is automatically safe. Bounded is necessary and not sufficient, because cardinality multiplies: five labels of 10 to 100 values each, all individually reasonable, produce millions of series in combination.

The third is that raising the memory limit is a fix. It buys time and delays the failure to a worse moment, when the series count is higher and WAL replay takes longer. Drop the label.

Interview delivery note

Say this: "Series count is the product of every label's distinct values, and each active series costs a few kilobytes of memory plus index, so one unbounded label like a user ID turns a few hundred series into millions. Prometheus OOMs, and then WAL replay of millions of series means you're blind for minutes after the restart, with alerting rules unable to evaluate. The policy is that metrics are for aggregates and identifiers belong in traces or logs, linked by exemplars. But policy isn't enough: I'd enforce it in the collection path with metric relabelling to drop known-bad labels, and a sample_limit so a single bad target gets rejected loudly instead of degrading the whole server."

The depth signal is the recovery failure, not the OOM. Everyone knows high cardinality is bad. Describing the WAL replay blind window and the crash loop shows you have run the system, and adding "and I'd give the engineer who asked for the label somewhere else to put the data" shows you understand why the rule keeps getting broken.

The Staff & Team Lead Interview Reference