The observability pipeline: Collector, sampling, exemplars, wide events, profiling

What it is

Modern observability is not "metrics, logs, and traces" as three separate systems. It is a pipeline that collects signals, processes them (sampling, redaction, enrichment), and routes them to backends, with the signals correlated so an investigation moves between them.

   apps (SDKs)          COLLECTOR                    backends
       │            ┌──────────────┐
  metrics ─────────▶│ receive      │──▶ metrics store (Prometheus, Mimir)
  traces  ─────────▶│ process:     │──▶ trace store (Tempo, Jaeger)
  logs    ─────────▶│  - sample    │──▶ log store (Loki, ELK)
  profiles ────────▶│  - redact    │──▶ profile store (Pyroscope)
                    │  - enrich    │
                    │  - batch     │
                    └──────────────┘

The OpenTelemetry Collector is the pipeline's centre: receivers ingest, processors transform, exporters route, and the whole thing is vendor-neutral, so the backend is a configuration choice rather than an instrumentation rewrite.

What this is confused with: "observability 1.0", three pillars queried separately. The argument of "observability 2.0" is that the useful primitive is not three signal types but one wide event per unit of work, carrying every dimension, from which metrics, traces and logs are all derived. The distinction is not academic: it determines whether you can ask a question you did not instrument for in advance, which is the entire promise of observability over monitoring.

The problem it solves

Monitoring answers questions you predicted. Observability answers questions you did not.

Monitoring:  "alert when p99 latency > 300 ms"        (predicted)
Observability: "why is p99 latency high, and is it
                concentrated in one customer, one
                region, one API version, one feature
                flag cohort, on the write path?"       (not predicted)

The pipeline exists because raw signals are too expensive to keep at full fidelity and too uncorrelated to investigate:

A service at 50,000 req/s, one trace per request:
  full traces:       50,000 traces/s, ~150 GB/day per service
  storage cost:      dominant, and most traces are of successful,
                     unremarkable requests

You cannot keep every trace, and the ones you throw away must not be the interesting ones, which is the sampling problem. And a metric spike with no way to jump to an example trace is a dead end, which is the correlation problem.

Mechanics

The Collector pipeline

receivers:
  otlp:
    protocols: {grpc: {endpoint: 0.0.0.0:4317}, http: {}}

processors:
  memory_limiter:                    # BACKPRESSURE: drop rather than OOM
    check_interval: 1s
    limit_percentage: 80
  batch:                             # amortise export overhead
    timeout: 5s
    send_batch_size: 8192
  attributes/redact:                 # strip PII BEFORE it leaves your network
    actions:
    - {key: user.email, action: delete}
    - {key: http.request.header.authorization, action: delete}
  tail_sampling:                     # see below
    policies:
    - {name: errors, type: status_code, status_code: {status_codes: [ERROR]}}
    - {name: slow, type: latency, latency: {threshold_ms: 500}}
    - {name: sample-rest, type: probabilistic, probabilistic: {sampling_percentage: 1}}

exporters:
  otlphttp/traces: {endpoint: https://tempo:4318}
  prometheusremotewrite: {endpoint: https://mimir/api/v1/push}

service:
  pipelines:
    traces:  {receivers: [otlp], processors: [memory_limiter, tail_sampling, batch], exporters: [otlphttp/traces]}
    metrics: {receivers: [otlp], processors: [memory_limiter, batch], exporters: [prometheusremotewrite]}

memory_limiter first is not optional: without it the Collector under a traffic spike buffers until it OOMs, taking down the telemetry pipeline exactly when you need it. It applies backpressure by refusing data, which is the correct failure mode for an observability system.

attributes/redact before the exporter is where PII leaves your control. A trace attribute carrying an email or a token is a data-exposure incident once it reaches a third-party backend, and redaction at the Collector is the single enforcement point.

Head sampling versus tail sampling

The distinction that decides whether you keep the interesting traces:

HEAD sampling:  decide at the START of the trace, before you know anything.
                "keep 1% of all traces."
                -> cheap, stateless, and it throws away errors and slow
                   requests at the same 1% rate as everything else.

TAIL sampling:  decide at the END, having seen the whole trace.
                "keep 100% of errors, 100% of slow traces, 1% of the rest."
                -> keeps what matters, and requires BUFFERING every trace
                   until it completes, which is memory and a stateful Collector.

Head sampling's flaw is that it is blind: it decides before it knows whether the request errored or was slow, so a 1 percent head sample keeps 1 percent of your errors, which is useless for debugging a rare failure.

Tail sampling keeps the interesting traces and costs a stateful buffer:

The tail-sampling trade:
  buffer window:   ~10-30 s (a trace must complete within it)
  memory:          proportional to in-flight traces x trace size
  the catch:       all spans of a trace must reach the SAME Collector
                   instance, so you need a load-balancing exporter that
                   routes by trace ID.

The trace-ID routing requirement is the operational cost of tail sampling and the reason it is deployed as a two-tier Collector (a first tier routing by trace ID to a second tier that buffers and samples). Head sampling is stateless and blind; tail sampling is stateful and smart, and for anything where errors are rare, tail is worth the complexity.

Exemplars: the metric-to-trace bridge

An exemplar attaches a trace ID to a metric data point, so a spike on a latency histogram links directly to an example trace of a slow request.

histogram bucket [500ms, 1s]:  count 42, exemplar: trace_id=abc...

# The p99 is high. Click the exemplar on the graph -> the trace that
# was in that bucket -> the span that was slow -> the log lines for it.
histogram_quantile(0.99, sum(rate(http_duration_bucket[5m])) by (le))

Exemplars are what make the metric-first debugging workflow work: you alert on a metric, see the spike, and jump to a representative trace without knowing in advance which request to look at. Without exemplars, a metric spike is a signal with no example, and you are back to grepping. It is a small feature with a large effect on time-to-diagnosis.

Structured logs and trace correlation

{ "timestamp": "2026-08-04T09:12:44Z", "level": "error",
  "msg": "payment declined", "trace_id": "abc123", "span_id": "def456",
  "customer_id": "4471", "amount_cents": 4250, "decline_code": "insufficient_funds" }

The trace_id in every log line is the correlation key. A log without it is an island; a log with it is a node in a trace, so an error log links to the trace that produced it and the metric that counted it. Structured logs (key-value, not a formatted string) are the prerequisite, because you cannot correlate or query a string.

Retention tiers, because full-fidelity logs are expensive:

Hot (7 days):     everything, indexed, fast query
Warm (30 days):   sampled, or indexed on fewer fields
Cold (1 year):    object storage, queryable slowly, for compliance

Wide events and observability 2.0

The observability 2.0 argument: stop emitting three signal types and emit one wide event per unit of work.

Instead of:
  a counter increment, a log line, and a span, each with a subset of context,

Emit ONE wide event per request:
  { trace_id, span_id, duration_ms, http_method, http_route, http_status,
    customer_id, customer_tier, region, api_version, feature_flags: [...],
    db_queries: 4, db_time_ms: 12, cache_hits: 2, cache_misses: 1,
    upstream_calls: [...], error: null, ... 50+ dimensions }

The point is high cardinality on purpose. A wide event carries every dimension you might want to slice by, so "is the latency concentrated in one customer on one API version behind one feature flag" is a query, not a re-instrumentation. Metrics, traces and logs are then derived views of the same events rather than separately-instrumented signals.

The cost is cardinality, which is the metrics cost model (see cardinality): a metric with customer_id as a label explodes into a series per customer. Wide events push that cardinality into a columnar event store (Honeycomb, ClickHouse) rather than a time-series database, because the time-series model cannot hold it. That is the architectural commitment observability 2.0 asks for, and it is why it is a store choice, not a config flag.

The honest position: observability 2.0 is genuinely better for debugging novel problems and it requires a different storage engine. For a team on Prometheus and Grafana, adopting it is a migration, not a setting, and the argument is strongest for services whose failures are unpredictable and whose questions are not known in advance.

Continuous profiling: the fourth signal

Metrics, traces and logs tell you that a service is slow and where in the call graph. Profiling tells you which line of code, continuously, in production.

A service's p99 rose. The trace shows the time is in one span.
The profile shows that span is 60% in JSON serialisation, on one type.

Continuous profiling (Pyroscope, Parca, Polar Signals) runs at roughly 1 percent overhead, sampling stacks continuously, so a regression is a diff between two time windows' flame graphs rather than a reproduction. It is the same wall-clock-versus-CPU distinction as the JVM profiling page: CPU profiles for compute, and wall-clock for latency.

Its place in the pipeline is the last-mile drill-down: metric spike (exemplar) → trace (slow span) → profile (slow line). Each signal narrows the search by an order of magnitude, and profiling is the one that ends at code.

A worked example: a debugging workflow that took 3 hours and then 90 seconds

A payments platform. About 200 services, Prometheus plus Jaeger, head sampling at 1 percent, no exemplars, unstructured logs.

The incident, before the changes:

alert: checkout p99 latency > 800 ms
investigation:
  1. see the metric spike (5 min)
  2. no exemplar, so no example trace. Which request?
  3. grep logs for slow checkouts (unstructured, no trace_id): 40 min
  4. find a few slow request IDs; search Jaeger for them
  5. head sampling kept 1% -> the slow requests were mostly not sampled
  6. eventually find one slow trace, see the time is in a downstream call
  7. that service's logs, again unstructured: another 40 min
total: ~3 hours to identify a slow downstream dependency

Head sampling had discarded 99 percent of the slow traces, so the specific requests that were slow were mostly not captured, and the unstructured logs could not be correlated to the few that were.

The changes:

1. Tail sampling, keeping errors and slow traces.

tail_sampling:
  policies:
  - {name: errors, type: status_code, status_code: {status_codes: [ERROR]}}
  - {name: slow, type: latency, latency: {threshold_ms: 500}}
  - {name: baseline, type: probabilistic, probabilistic: {sampling_percentage: 1}}

traces kept:            1% of all -> 100% of errors + 100% of slow + 1% baseline
storage:                +8% (slow and error traces are a small fraction)
slow-trace capture:     ~1% -> 100%

Storage went up 8 percent and slow-trace capture went from 1 percent to 100 percent, because slow and error traces are a small fraction of total volume. This is tail sampling's whole argument: you keep the interesting traces at full fidelity for a small storage increase.

2. Exemplars on the latency histograms.

alert -> click the exemplar on the p99 graph -> the exact slow trace,
no grepping.

3. Structured logs with trace_id.

every log line: JSON, with trace_id and span_id.
-> from a trace span, one click to its logs; from a log, one click to
   its trace.

The same incident, after:

alert: checkout p99 > 800 ms
  1. click the exemplar on the graph -> the slow trace (10 s)
  2. the trace shows the time in the fraud-check span (5 s)
  3. click the span's logs -> "fraud model cold, reloading" (10 s)
  4. the fraud service had just deployed and was cold
total: ~90 seconds

Three hours to ninety seconds, because each signal linked to the next: metric spike → exemplar → trace → span → correlated logs. The individual improvements were modest; the correlation between them was the transformation.

4. Later: a wide-events pilot on the checkout service, because a recurring question was "is this latency concentrated in one customer segment."

Prometheus with customer_id as a label:  cardinality explosion, refused.
Wide events in ClickHouse:               the query is
    SELECT quantile(0.99)(duration_ms) FROM events
    WHERE service = 'checkout' GROUP BY customer_tier, api_version
    -> answered in one query, sliceable by ANY of 50 dimensions

The question that Prometheus could not hold as a metric was a routine query over wide events, which is the observability 2.0 argument in one example: cardinality that breaks a time-series database is the normal case for an event store.

5. Continuous profiling on the three highest-traffic services.

a p99 regression after a deploy:
  before: reproduce locally, profile, guess. Hours to a day.
  after:  diff the flame graph between the two deploys. The regression
          was a regex compiled per request. ~10 minutes.

Final:

                              before      after
mean time to identify a
  slow component              ~3 hours    ~90 s
slow-trace capture rate       ~1%         100%
trace storage                 baseline    +8%
"which customer segment"      not askable a query (wide events pilot)
p99 regression diagnosis      hours-day   ~10 min (profiling)

The transferable finding: observability value is in the correlation, not the signals. The team had metrics, traces and logs before, and each was an island. Exemplars, trace IDs in logs, and tail sampling to keep the interesting traces are what turned three separate systems into one investigation, and that was worth more than any individual backend.

Production evidence

The OpenTelemetry Collector is the CNCF-standard pipeline, and its receiver-processor- exporter model is what makes the backend a configuration choice. Its adoption as the vendor- neutral standard is why "instrument once, route anywhere" is now realistic.

Tail sampling with trace-ID-aware load balancing is documented in the Collector's tailsamplingprocessor and loadbalancingexporter, and the two-tier deployment is the reference pattern for it. The requirement that all spans of a trace reach the same instance is the operational cost the docs are explicit about.

Exemplars are part of OpenMetrics and Prometheus (native histograms carry them), and Grafana's exemplar support is what makes the click-through-to-trace workflow real. The feature is small and its effect on time-to-diagnosis is the reason it exists.

Charity Majors and Honeycomb's "observability 2.0" framing (one wide event per request, derive everything from it, store in a columnar event database) is the reference argument, and Honeycomb, ClickHouse-based stacks, and Grafana's exploration of the model are the production expressions. The cardinality argument is the crux and it is why the store differs from a TSDB.

Continuous profiling (Google's Google-Wide Profiling paper, then Pyroscope, Parca and Polar Signals) established profiling as an always-on production signal at roughly 1 percent overhead, and the flame-graph-diff workflow for regressions is its defining use.

Grafana's LGTM stack (Loki, Grafana, Tempo, Mimir) and the exemplar-and-trace-ID correlation between them is the clearest production instance of the "correlation over signals" argument, because it is built around jumping between the three.

The debate

Head or tail sampling? Tail, for anything where errors and slow requests are rare and are what you debug, because head sampling keeps them at the same low rate as everything else. The cost is a stateful, trace-ID-routing Collector tier, and head sampling remains right where volume is so high that even buffering is infeasible or where all traces are equally interesting (rare). The default should be tail with error-and-latency policies.

Is observability 2.0 worth adopting? For a team whose failures are novel and whose questions are not predictable in advance, the wide-event model is genuinely better, and it requires a columnar event store rather than a TSDB. It is a migration, not a config change, so the honest answer is to pilot it on the service where "slice by an arbitrary dimension" is a recurring need, rather than rearchitecting everything. The cardinality argument is the deciding one: if your important questions have high-cardinality dimensions, the TSDB cannot hold them.

Are three pillars obsolete? Not obsolete, but the framing is limiting. The three signals are still what you store; the shift is treating them as derived views of events rather than separately-instrumented systems, and instrumenting the wide event once. The practical middle ground most teams occupy is three correlated signals (exemplars, trace IDs in logs), which is most of the value of 2.0 without the store migration.

Is continuous profiling worth the overhead? At roughly 1 percent, yes, for high-traffic services, because it turns a p99-regression investigation from a reproduction into a flame-graph diff. The constraint is the same as any profiling in a container: it needs perf_events access, which is a node-level decision. For low-traffic services the value is smaller and on-demand profiling suffices.

What is the single highest-value change for a team with three uncorrelated signals? Trace IDs in every log line and exemplars on the key metrics, because they turn separate systems into one investigation for very little effort. In the worked example that was the three-hours-to- ninety-seconds change, and it required no new backend.

Follow-up Q&A

"Head sampling or tail sampling?"

Tail, when errors and slow requests are rare and are what you debug, because head sampling decides before it knows whether a request errored, so a 1 percent head sample keeps 1 percent of your errors, which is useless for a rare failure. Tail decides after seeing the whole trace, so you keep 100 percent of errors and slow traces and 1 percent of the rest. The cost is a stateful Collector that buffers until traces complete and routes all spans of a trace to the same instance by trace ID, which is a two-tier deployment.

"What is an exemplar and why does it matter?"

A trace ID attached to a metric data point, so a spike on a latency histogram links to an example trace of a slow request. It matters because it enables the metric-first debugging workflow: alert on the metric, see the spike, click through to a representative trace without knowing in advance which request to look at. Without it, a metric spike is a signal with no example and you are back to grepping logs for slow request IDs, which in one case was 40 minutes of a three-hour investigation.

"What is the observability 2.0 argument?"

Stop instrumenting three signal types and emit one wide event per unit of work, carrying every dimension you might slice by (customer, tier, region, API version, feature flags, query counts), and derive metrics, traces and logs from it. The point is high cardinality on purpose, so "is the latency concentrated in one customer on one API version" is a query rather than a re-instrumentation. The cost is that this cardinality breaks a time-series database, so it needs a columnar event store, which makes it a migration rather than a config change.

"How do metrics, traces, logs and profiles fit together?"

As a drill-down where each narrows the search by an order of magnitude. A metric spike (with an exemplar) links to a trace; the trace shows which span is slow; the span's logs (correlated by trace ID) show the error; and a profile shows which line of code. In one case that chain took a three-hour investigation to ninety seconds, because the value was the correlation between the signals rather than any one of them.

"Is continuous profiling worth it?"

For high-traffic services, yes, at roughly 1 percent overhead. It turns a p99-regression investigation from a local reproduction into a diff between two time windows' flame graphs: in one case a regression was a regex compiled per request, found in ten minutes by diffing flame graphs across two deploys rather than hours of reproduction. It needs perf_events access, which is a node-level decision, and for low-traffic services on-demand profiling is enough.

What is Scuba, and why is it cited so often in this area? Scuba is Facebook's in-memory, schema-flexible datastore for real-time ad hoc analysis, described in a 2013 VLDB paper. Rows are arbitrary sets of key-value pairs with no fixed schema, they are held in memory across a fleet and aged out, and queries are aggregations over a time range that return in about a second by sampling and fanning out across every node holding relevant data. The design choice that makes it interesting is that it deliberately gives up completeness for speed: results are approximate, computed over whatever subset answered within the deadline, and the response reports the sampling rate so you can reason about it.

It is cited constantly because it is the clearest published statement of a position this chapter keeps arriving at: for debugging, a fast approximate answer over wide, high cardinality data beats a slow exact one over pre-aggregated metrics. A metrics system with fixed label sets cannot answer "which combination of app version, country and device is driving this error spike" unless someone predicted that question when defining the metric. Scuba answers it because it stores the events, not the aggregates. That lineage runs directly into the modern wide-event and high-cardinality observability tools, and naming the paper is a better citation than naming any vendor.

Common misconceptions

"1 percent sampling is fine." Head sampling at 1 percent keeps 1 percent of your errors and slow requests, which are exactly what you debug. Tail sampling keeps those at 100 percent for a small storage increase.

"Observability is metrics, logs, and traces." That framing treats them as separate systems. The value is their correlation, and observability 2.0 treats them as derived views of one wide event.

"More dimensions means a bigger metrics bill." In a time-series database, yes, because each label combination is a series. Wide events push that cardinality into a columnar store designed for it, which is the architectural point.

"Exemplars are a minor feature." They are the bridge from a metric spike to an example trace, which is the difference between a diagnosable spike and a dead end. Small feature, large effect.

"Profiling is for development." Continuous profiling runs in production at about 1 percent overhead and turns regression diagnosis into a flame-graph diff.

Interview delivery note

Say this verbatim: "The value of observability is in the correlation, not the signals. In one case an investigation went from three hours to ninety seconds, and the change was exemplars on the metrics, trace IDs in the logs, and tail sampling to keep the slow traces, so a metric spike linked to a trace linked to the exact log lines. Each signal narrowed the search by an order of magnitude." The drill-down chain and a concrete before-and-after.

The senior-versus-staff separator is head-versus-tail sampling with the blindness argument. A senior engineer knows sampling reduces cost. A staff engineer explains that head sampling decides before it knows whether a request errored, so it keeps errors at the same low rate as everything else and is useless for rare failures, that tail sampling keeps errors and slow traces at full fidelity for a small storage increase, and that the cost is a stateful trace-ID-routing Collector tier. Knowing why head sampling fails is the depth signal.

The second signal is the observability 2.0 cardinality argument stated as a storage decision. "High-cardinality dimensions break a time-series database, so wide events need a columnar event store, which makes 2.0 a migration rather than a config flag" shows you understand why the framing is a real architectural choice rather than a fashion.

The Staff & Team Lead Interview Reference