SRE and observability

Reliability questions have a contractual half and an engineering half, and candidates usually prepare only one. The engineering half is SLIs, SLOs, error budgets and burn-rate alerting. The contractual half is what an SLA actually promises, why you always set it looser than your SLO, and why your dependencies' SLAs bound your own.

The most practically useful thing in this chapter is cardinality. A label with unbounded values will melt a metrics backend, and being able to explain the failure and the policy fix is the single most common real observability question asked at this level.

What this chapter covers

• [done] Burn-rate alerting
• [done] SLI selection and SLO targets covers measuring closest to the user, good events over valid, deriving targets from tolerance, and the minutes-per-month table
• [done] OpenTelemetry: API, SDK, Collector, OTLP, and semantic conventions
• [done] Load testing: open vs closed, coordinated omission
• [done] Cardinality: the observability cost model, and the policy fix
• [done] The observability pipeline covers the annotated Collector config, head vs tail sampling, exemplars and the modern debugging workflow, structured logs with trace correlation and retention tiers, wide events and the observability 2.0 argument, and continuous profiling as the fourth signal
• [done] Incident management covers roles, severity triggers, mitigate before diagnose, blameless postmortems with repeat-incident rate, and per-tenant SLAs with the observability requirement they create
• [done] Chaos-testing an untested assumption
• [done] SLA, SLO and SLI, the contractual layer
• [done] Composite SLOs and dependency math
• [done] The error budget policy as a signed contract
• [done] Why you cannot average percentiles

Source: §28, §36.