Design a metrics and observability pipeline

"Design the metrics pipeline for 5,000 services. Ten million active time series, thirteen-month retention, and queries that return in under a second."

Step 1: clarify (4 minutes)

Metrics, logs and traces are three different systems. Say so, because a candidate who designs one pipeline for all three has not understood the workload difference.

Metrics  Numeric, regular, aggregatable, tiny per sample.
         Optimised for: many series, long retention, fast range queries.
Logs     Text, irregular, high volume per event, searched not aggregated.
         Optimised for: full-text search, short retention, high write volume.
Traces   Structured spans, sampled, joined by trace id.
         Optimised for: point lookup by id, low retention, high cardinality.

Assume metrics only, and note that the design deliberately excludes the other two.

Push or pull? This is the first real architectural fork and it has a defensible answer either way. Assume pull for long-lived services, push for short-lived and batch work, which is what almost every mature setup converges on.

What is the cardinality budget? 10 million active series is the stated number, and the question behind it is what happens when someone adds user_id as a label. Assume enforcement is required, because without it this system's failure mode is not gradual.

What is the query pattern? Assume: dashboards (many small range queries, repeated every 30 seconds), alerts (the same query evaluated continuously), and ad hoc investigation (rare, wide, expensive). Those three have very different cache and capacity implications.

What is the retention shape? Thirteen months is stated, and the implicit question is at what resolution. Assume 15-second raw for 15 days, 5-minute rollups for 90 days, 1-hour for 13 months. Retaining 15-second data for thirteen months is 2,600 times more data for information nobody queries at that age.

Step 2: capacity math (5 minutes)

Series and samples
  10M active series, scraped every 15 s
  = 10M / 15 = 667,000 samples/sec ingest

Raw storage, uncompressed
  667k samples/sec x 16 bytes (timestamp + float64)
  = 10.7 MB/sec = 920 GB/day.  Not viable.

With Gorilla compression (delta-of-delta timestamps,
XOR float encoding) real-world is ~1.3 to 2 bytes/sample:
  667k x 1.7 bytes = 1.13 MB/sec = ~98 GB/day
  15 days raw = ~1.5 TB.  Now viable.

That 10x reduction is the single most important fact about
metrics storage, and it is why a general-purpose database is
the wrong tool.

Rollups
  5-minute rollups: 10M series x (1 sample / 300 s) x 90 days
    = 10M x 288/day x 90 = 259 billion samples
    Store min/max/sum/count (4 values) -> ~1.4 TB
  1-hour rollups: 10M x 24 x 395 days x ~7 bytes = ~660 GB

Total: ~1.5 TB raw + 1.4 TB + 0.7 TB = ~3.6 TB.
On object storage, that is negligible cost. On local SSD it is not.
-> This drives the tiered-storage decision in step 4.

Index memory
  The inverted index (label -> series) is the memory constraint,
  not the samples. Roughly 1-3 KB per active series in memory
  for labels and postings:
  10M x 2 KB = 20 GB per replica holding the full index.
  -> This is why horizontal sharding is about the INDEX, not the data.

Query load
  2,000 dashboard panels x 1 query / 30 s     = 67 QPS
  4,000 alert rules evaluated every 30 s      = 133 QPS
  Ad hoc                                      = ~5 QPS but 100x heavier
  Total ~200 QPS, of which the ad hoc queries dominate resource use.

Two numbers drive the whole design: compression takes 16 bytes per sample to under 2, and the in-memory index is 20 GB. The first makes the storage affordable; the second is why you shard.

Step 3: architecture

  5,000 services
    │  /metrics endpoint (long-lived)     │ push (batch, lambda, cron)
    ▼                                      ▼
  ┌──────────────────┐              ┌──────────────┐
  │ SCRAPERS         │              │ PUSH GATEWAY  │
  │ (sharded by      │              │ (short-lived  │
  │  target hash)    │              │  jobs only)   │
  └────────┬─────────┘              └──────┬───────┘
           └──────────────┬────────────────┘
                          ▼
              ┌───────────────────────┐
              │  RELABEL / ENFORCE     │  drop high-cardinality labels,
              │  cardinality limits    │  enforce per-team series budgets
              └───────────┬───────────┘
                          ▼
              ┌───────────────────────┐
              │  INGEST (sharded by    │  hash(series labels) so a series
              │  series hash)          │  always lands on one shard
              └───────────┬───────────┘
                          ▼
        ┌─────────────────┴──────────────────┐
        ▼                                    ▼
  ┌────────────┐                     ┌───────────────┐
  │ HOT (local │  2 h in memory,     │ OBJECT STORAGE │  blocks uploaded
  │ SSD, 15 d) │  then 2 h blocks    │  (13 months)   │  every 2 h
  └─────┬──────┘                     └───────┬───────┘
        │                                    │
        └──────────────┬─────────────────────┘
                       ▼
              ┌───────────────────────┐
              │  QUERY FEDERATION      │  fan out, dedupe replicas,
              │  + result cache        │  merge, cache by (query, step)
              └───────────┬───────────┘
                          ▼
              ┌───────────────────────┐
              │  DASHBOARDS + ALERTS   │
              └───────────────────────┘

Shard ingest by series hash, not by service. Sharding by service creates a hot shard for the largest service and means a series can move shards when a service is renamed. Hashing the full label set means a given series always lands on the same shard, so the index for it is local and queries for it are targeted.

Step 4: storage, and why it is purpose-built

The compression that makes it possible

Timestamps: delta-of-delta.
  Scrapes are regular, so consecutive deltas are nearly identical.
    t:      1700000000, 1700000015, 1700000030, 1700000045
    delta:              15,         15,         15
    delta²:                          0,          0     <- 1 bit each

Values: XOR against the previous value.
  Metric values change slowly, so consecutive float64s share most
  of their bits. XOR yields many leading and trailing zeros, which
  are encoded as a count.
    1024.0 XOR 1024.5 -> a handful of meaningful bits

Combined: ~1.37 bytes/sample in Facebook's Gorilla paper.

This is why a relational database is the wrong tool, and it is worth stating rather than asserting that you "use a TSDB". Postgres storing 667,000 rows per second of (series_id, timestamp, value) is 16 bytes plus row overhead plus index maintenance, so roughly 10x the storage and orders of magnitude more write amplification, and its B-tree index is optimised for point lookups rather than for range scans over a series.

The tiering

Tier          Where            Retention   Resolution   Query latency
------------------------------------------------------------------
In-memory     Ingester heap    2 h         15 s         ~10 ms
Local SSD     Ingester disk    15 d        15 s         ~50 ms
Object store  S3/GCS blocks    13 months   15s/5m/1h    ~500 ms to 5 s

Blocks are uploaded to object storage every two hours, then compacted (merging small blocks, deduplicating replicas, and computing downsampled versions). Compaction is where rollups are produced, so downsampling is a property of the storage layer rather than a separate pipeline.

Why 13 months specifically: year-over-year comparison. "Is this Black Friday worse than last Black Friday" requires slightly more than twelve months, and that single query pattern is why the number is 13 and not 12.

Step 5: cardinality, the failure mode that matters

This is the deep dive, because it is how these systems actually die.

A well-behaved metric:
  http_requests_total{service, method, status, endpoint}
  50 services x 5 methods x 8 statuses x 20 endpoints = 40,000 series

The same metric after one pull request:
  http_requests_total{service, method, status, endpoint, user_id}
  40,000 x 2,000,000 users = 80 BILLION series

Cardinality is multiplicative across labels, and one label with unbounded values ends the system. Not degrades: ends. The ingester's in-memory index grows until it OOMs, and because it is sharded by series hash, the bad series are spread across every shard, so every shard dies at once.

The defences, in order of when they act:

1. AT INSTRUMENTATION: lint rules in CI that reject label values
   drawn from unbounded sources (ids, emails, paths with ids in them,
   raw URLs). Cheapest place to stop it.

2. AT SCRAPE: relabel_config drops known-bad labels before ingest.
   Fast, and it is the emergency lever during an incident.

3. AT INGEST: per-tenant and per-metric series limits, enforced with
   a hard reject and a clear error. The rejection must name the
   metric and the offending label, or nobody can act on it.

4. CONTINUOUSLY: a cardinality report per team, per metric, with
   week-over-week growth. Ranked by series count. Sent to owners.

# The scrape-time emergency lever. Worth having pre-written, because
# during an incident nobody wants to learn relabel_config syntax.
metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'http_requests_total'
    target_label: user_id
    replacement: ''            # blank it out
  - regex: '(user_id|session_id|request_id|trace_id)'
    action: labeldrop          # never allow these as metric labels

The per-tenant limit is the load-bearing control, and the argument for it is blast radius: without it, one team's mistake takes down observability for all 5,000 services, at exactly the moment everyone needs it most. A hard limit means one team loses metrics and everyone else is fine.

And the exemplar mechanism is the right answer for the thing people want user_id for. They want to get from a latency spike to a specific slow request. Exemplars attach a trace id to a histogram bucket sample, so a dashboard can jump from "p99 spiked" to a specific trace, without the id becoming a label. That is the correct answer to the request rather than a refusal.

Step 6: querying

# What a dashboard actually asks
histogram_quantile(0.99,
  sum by (le, service) (
    rate(http_request_duration_seconds_bucket{env="prod"}[5m])
  )
)

To answer that, the query engine must select every series matching env="prod" for that metric across every le bucket and every service, which at 5,000 services and 12 buckets is tens of thousands of series, then compute a rate over each, then aggregate.

The optimisations that matter, in order:

1. RESULT CACHE, split by time.
   Dashboards re-query the same range every 30 s. Cache per time
   step, so a query for the last 6 hours reuses 5h55m of cached
   result and computes only the newest step.
   This alone typically removes 80-90% of dashboard query cost.

2. RECORDING RULES for expensive, frequently-used expressions.
   Precompute at ingest time; the dashboard reads a single series.
   The above query becomes:
     service:http_request_duration:p99_5m
   Cost at query time: one series read instead of tens of thousands.

3. QUERY SHARDING for wide ad hoc queries.
   Split by series hash across queriers, merge results.
   Turns one 30-second query into ten 3-second parallel ones.

4. LIMITS: max series touched, max samples scanned, max duration.
   An unbounded ad hoc query must not be able to take down the
   system that everyone else is using to debug their incident.

Recording rules are the highest-leverage optimisation, and the discipline is: every query in an alert rule or a frequently-viewed dashboard should be a recording rule. Alerts in particular, because they run continuously and a slow alert query is a slow alert, which is a direct reliability problem.

Step 7: failure modes

Ingester crashes
  -> In-memory data (up to 2 h) is lost unless there is a WAL.
     There must be a WAL, replayed on restart. The replay is the
     slow part of a restart (see: Prometheus WAL replay), and it
     scales with active series, which is another cardinality cost.

Object storage unavailable
  -> Recent data still queryable from ingesters. Historical queries
     fail. Degrade the dashboard rather than the alert path, and
     make sure ALERTS only depend on recent data, which is a design
     constraint on alert rules, not a runtime decision.

Scrape target down
  -> `up == 0`, which is itself a metric and the basis of the most
     important alert in the system. Absence of data must alert;
     a metric that stops arriving looks identical to a healthy
     zero if you only alert on thresholds.

One team blows the cardinality budget
  -> Hard reject for that tenant with a named error. Everyone else
     is unaffected. This is the whole reason for per-tenant limits.

Query overload from one ad hoc user
  -> Per-query limits on series touched and samples scanned, plus a
     separate query pool for ad hoc traffic so it cannot starve
     alert evaluation. Alerts and dashboards must not share a pool
     with exploration.

Clock skew between targets
  -> Samples timestamped by the scraper, not the target, so skew
     does not corrupt the series. A target reporting its own
     timestamps must be trusted deliberately.

The alerting-path independence constraint is worth stating as a design rule: alert queries must be answerable from recent, local data only. An alert that depends on object storage cannot fire during an object-storage incident, which is exactly when you need it.

Step 8: what changes at ten times the scale

At 100 million active series:

The index stops fitting in memory. 100 million series at 2 KB is 200 GB per full index, so the index itself must shard, and queries become federated across index shards with a merge step. This is where systems move from "Prometheus with remote write" to Cortex, Mimir or Thanos with proper multi-tenancy.

Per-tenant isolation becomes mandatory rather than advisable. At this size a shared ingester fleet means one tenant's cardinality spike affects others regardless of limits, because it consumes shared CPU and memory. Dedicated shards per large tenant, with the small tenants pooled.

Downsampling becomes the primary storage decision. Thirteen months of raw at 100 million series is petabyte scale, so the rollup schedule and what is retained at each resolution is a cost decision with real money attached rather than a default.

Streaming aggregation moves upstream. Instead of storing every series and aggregating at query time, compute the common aggregations at ingest, store those, and retain raw series only for a short window. This trades query flexibility for cost and it is the right trade at this size.

Production evidence

Pelkonen et al., "Gorilla: A Fast, Scalable, In-Memory Time Series Database" (VLDB 2015) is the source of the delta-of-delta plus XOR compression scheme, and the reported average of 1.37 bytes per sample is the number that makes this class of system viable. Every modern TSDB implements a variant.

Prometheus's TSDB design (Fabian Reinartz's write-ups and the tsdb package documentation) covers the two-hour block model, the WAL, and the inverted index, and its documented advice against high-cardinality labels is the primary source for the cardinality section.

Thanos, Cortex and Grafana Mimir are the horizontally-scalable implementations, and their shared architecture (ingesters, object-storage blocks, a compactor producing downsampled versions, a query frontend with result caching and query sharding) is convergent evidence for the design here. Mimir's published benchmarks describe operating at the 1-billion-series scale.

OpenMetrics and the exemplars specification define the trace-id-on-a-sample mechanism that answers the "I need user_id on my metric" request correctly.

Google's SRE Book, chapter 6 ("Monitoring Distributed Systems"), is the source for the four golden signals and for the argument that alert queries should be simple and fast, which is the basis for the recording-rule discipline and the alert-path independence rule.

The debate

The case for pull: the scraper controls the sample rate, so a misbehaving service cannot flood the pipeline. up is a free health signal. Service discovery makes targets explicit, so you know what should exist and can alert on its absence, which is the failure push cannot detect.

The case for push: short-lived jobs (batch, serverless, CI) may not exist long enough to be scraped. Network topology (NAT, firewalls, edge devices) sometimes makes pull impossible. And push scales the ingest tier independently of the target count.

The case for buying rather than building: Datadog, Grafana Cloud and Chronosphere exist, and at 10 million series the licence cost is real but so is the platform team you would otherwise hire. The build-versus-buy crossover is genuinely close at this size.

My position: pull for long-lived services, push for short-lived jobs, and enforce cardinality limits per tenant from day one. The hybrid is what mature setups converge on because the two failure modes are different: pull cannot see a job that lived for eight seconds, and push cannot tell you a service that should exist has vanished.

The decision I hold most firmly is per-tenant cardinality limits, enforced with a hard reject. It is unpopular, because a team's metrics get dropped and they are annoyed. The alternative is that one team adding user_id to a label takes down observability for all 5,000 services during the incident that mistake caused. A hard limit converts a company-wide outage into one team's inconvenience, and that trade is not close.

I would also insist on alert queries being answerable from recent local data only. An alert that queries object storage cannot fire during an object-storage incident, and that is precisely when it matters. This is a constraint on how alert rules are written, so it needs to be enforced in review or in CI rather than hoped for.

Where I would push back on the requirement: thirteen months at full resolution is almost never wanted. The query that motivates thirteen months is year-over-year comparison, and that is answered fine at hourly resolution. Storing 15-second data for a year is 2,600 times the volume for a question nobody asks at that granularity, and the rollup schedule should be presented as a decision rather than a default.

Follow-up Q&A

"Why not just use Postgres?" Compression, mainly. Gorilla-style delta-of-delta timestamps plus XOR float encoding gets you from 16 bytes per sample to under 2, which is a 10x storage difference and a much larger difference in write amplification. Postgres storing 667,000 rows a second with B-tree index maintenance is a completely different cost profile, and its index is optimised for point lookups rather than range scans over a single series. The access pattern and the compression together are why this is a purpose-built category.

"How does a metrics system actually die?" Cardinality, and it dies suddenly rather than gradually. Cardinality is multiplicative across labels, so a single label with unbounded values, user_id being the classic, takes a 40,000-series metric to 80 billion. The ingesters' in-memory index grows until they OOM, and because ingest is sharded by series hash the bad series are spread evenly, so every shard dies at once. That is why per-tenant limits with a hard reject are the load-bearing control: they turn a company-wide outage into one team's problem.

"A team says they need user_id on their metric. What do you say?" I ask what question they are trying to answer, and it is almost always "the p99 spiked, which request was it?". The correct mechanism for that is exemplars: a trace id attached to a histogram bucket sample, so the dashboard can jump from the spike to a specific trace without the id becoming a label. That answers the actual need rather than refusing the request, which matters because a flat refusal gets routed around.

"What's the highest-leverage query optimisation?" Result caching split by time step, because dashboards re-run the same query every thirty seconds and a six-hour range reuses five hours and fifty-five minutes of cached result. That typically removes most of the dashboard cost on its own. After that, recording rules for anything in an alert or a frequently-viewed dashboard, which turns a query touching tens of thousands of series into a single series read. Alerts especially, because a slow alert query is a slow alert.

"Why 13 months and not 12?" Year-over-year comparison. "Is this Black Friday worse than last Black Friday" needs slightly more than twelve months of history. That one query pattern is the entire justification for the number, and it also tells you the resolution needed: hourly is fine for that question, so retaining 15-second data for a year is 2,600 times the volume for something nobody asks at that granularity.

"Push or pull?" Both, for different targets. Pull for long-lived services, because the scraper controls the rate so a misbehaving service cannot flood the pipeline, and because up == 0 gives you a free health signal and lets you alert on a service that should exist and does not. Push for short-lived jobs, batch and serverless, which may not live long enough to be scraped, and for network topologies where pull is impossible. The two failure modes are different, which is why mature setups run both.

"What must not break during an object storage outage?" Alerting. Which means alert queries have to be answerable from recent local data only, and that is a constraint on how alert rules are written rather than something the runtime can decide. An alert that ranges over thirty days cannot fire when historical storage is down, which is exactly when you want it. I would enforce it in rule review or in CI.

"How do you detect a service that stopped reporting?" Absence of data has to alert, and it is the failure that threshold alerts miss entirely, because a metric that stops arriving looks identical to a healthy zero. With pull you get it free from up == 0, because service discovery told you the target should exist. With push you need a separate liveness expectation, which is one of the strongest arguments for pull on long-lived services.

"Would you build this or buy it?" At 10 million series it is genuinely close, and I would want the numbers rather than a preference. Building means Mimir or Thanos plus roughly one to two engineers of ongoing platform work; buying means a licence cost that scales with series count and a vendor dependency in your incident path. I would lean toward buying unless there is a specific reason (data residency, an unusual retention requirement, cost at a scale where the licence exceeds the team), because observability platform work is rarely where a company's differentiation lies.

Common misconceptions

"Metrics, logs and traces are one pipeline." Three workloads with different write patterns, retention and query shapes. Designing one system for all three produces something bad at all three.

"Cardinality degrades performance." It kills the system, and it does so suddenly, because the in-memory index OOMs and sharding by series hash means every shard fails together.

"Store everything at full resolution." Thirteen months of 15-second data is 2,600 times the volume for questions nobody asks at that granularity. Rollups are the design, not a nice-to-have.

"Push is simpler." It removes the ability to detect that something which should exist has vanished, which is a failure mode threshold alerts cannot catch.

"Alerts and dashboards can share a query pool." An ad hoc investigation must not be able to starve alert evaluation, especially during the incident that prompted it.

Interview delivery note

Separate the three signal types in the first thirty seconds, because it establishes that you know why this is its own system: "First, metrics, logs and traces are three different systems with different write patterns, retention and query shapes. I'll design metrics, and I'd resist one pipeline for all three, because you end up bad at all of them."

Then do the compression arithmetic, because it justifies the purpose-built store: "667 thousand samples a second at sixteen bytes is 920 gigabytes a day, which isn't viable. With Gorilla-style delta-of-delta timestamps and XOR float encoding it's about 1.7 bytes a sample, so 98 gigabytes a day. That 10x is why this isn't a Postgres problem, and it's also why the in-memory index rather than the samples is the memory constraint."

Volunteer cardinality as the failure mode, because it is what the question is really about: "The way these systems die is cardinality, and it's sudden rather than gradual. It's multiplicative across labels, so one pull request adding user_id takes a forty-thousand-series metric to eighty billion. The ingesters OOM, and because ingest is sharded by series hash the bad series spread evenly, so every shard dies at once."

Then the control, with its justification: "So per-tenant hard limits, and I'd defend that even though teams hate having metrics rejected. Without it, one team's mistake takes down observability for five thousand services during the incident that mistake caused. A hard limit makes that one team's inconvenience."

The line that shows you have run one of these: "and alert queries have to be answerable from recent local data only. An alert that ranges over thirty days can't fire during an object storage incident, which is exactly when you want it. That's a constraint on how rules are written, so it needs enforcing in review rather than hoping."

The Staff & Team Lead Interview Reference