USE and RED, and the tooling for each
What it is
Two checklists for deciding what to measure, aimed at different things:
USE (Brendan Gregg) is for resources: CPUs, disks, network interfaces, memory, controllers, buses. For every resource, measure three things:
Utilisation: the fraction of time the resource was busy
Saturation: the degree of QUEUED work it could not service
Errors: error events
RED (Tom Wilkie) is for services: anything that handles requests. For every service, measure three things:
Rate: requests per second
Errors: failed requests per second
Duration: the distribution of request latency
The distinction that makes them complementary: USE is bottom-up and RED is top-down. RED tells you a service is unhealthy and cannot tell you why; USE tells you a resource is saturated and cannot tell you which user-facing thing that breaks. A system with only RED knows it is broken; a system with only USE knows something is busy.
What they are confused with: the Four Golden Signals (Google SRE: latency, traffic, errors, saturation), which is RED plus saturation. The addition matters, and the honest framing is that RED plus USE covers the same ground with a clearer division of labour: RED on the service, USE on the resources it depends on.
Saturation is the underused term in both. Utilisation at 100 percent tells you a resource is busy; saturation tells you how much work is waiting, and it is the metric that predicts latency. A disk at 100 percent utilisation with a queue depth of 1 is fine; the same disk with a queue depth of 40 is not, and utilisation cannot distinguish them.
The problem it solves
"What should we monitor?" produces either nothing or everything. Teams instrument what is easy (CPU, memory, disk space) and discover during an incident that none of it answers the question. Or they instrument everything and have 40,000 metrics and no dashboard anyone trusts.
The checklists convert a design question into an enumeration:
USE: list your resources. For each, do you have utilisation,
saturation and errors?
RED: list your services. For each, do you have rate, errors
and duration?
The gaps this exposes are consistently the same ones, and they are the reason the checklists earn their place:
Commonly measured: CPU utilisation, memory usage, disk space, request count
Commonly MISSING:
- saturation for ANY resource (run queue, disk queue depth, connection
pool waits, thread pool queue depth)
- error rate as a RATE rather than a count
- latency as a DISTRIBUTION rather than a mean
- the resources that are not CPU or memory: file descriptors, connection
pools, thread pools, network buffers
Connection pool saturation is the single most commonly missing metric in a service, and it is where a large fraction of latency incidents live: the pool is exhausted, requests queue for a connection, and every dashboard shows a healthy CPU.
Mechanics
USE, applied
Resource Utilisation Saturation Errors
──────────────────────────────────────────────────────────────────────────────────
CPU %busy per core run queue length, machine
PSI cpu pressure check
Memory used / total swapping, PSI memory, ECC errors
pgmajfault rate
Disk %time busy (iostat) avgqu-sz, PSI io SMART, I/O
errors
Network interface bytes/s vs capacity drops, overruns, errors,
tx queue depth CRC
Connection pool in-use / max threads WAITING for a timeouts
connection
Thread pool active / max queue depth rejected
tasks
File descriptors open / ulimit n/a EMFILE
Conntrack table count / max n/a table full
The saturation column is the one that is usually empty, and it is the one that predicts failure. Two examples of the difference:
Disk at 100% utilisation, avgqu-sz 0.9: saturated by ONE request at a time.
Fine. It is just always busy.
Disk at 100% utilisation, avgqu-sz 38: 38 requests waiting. Latency is
38x the service time.
Connection pool 20/20 in use, 0 waiting: fully utilised, no queue. Fine.
Connection pool 20/20 in use, 84 waiting: every request pays the wait.
This is the incident.
PSI (Pressure Stall Information) is the modern saturation metric and it is directly comparable across resources:
$ cat /proc/pressure/io
some avg10=42.18 avg60=38.02 avg300=21.44 total=...
full avg10=18.02 avg60=14.11 avg300=8.90 total=...
some is the fraction of time at least one task was stalled on this resource. A some
of 42 percent on I/O means nearly half the time something was waiting for disk, which is a
statement about lost time rather than about busyness. See
memory and the OOM killer for the memory case.
The USE tooling, per resource
# CPU: utilisation and saturation together
$ vmstat 1
r b swpd free ... us sy id wa st
8 2 0 210344 62 8 4 26 0
^ ^^ ^^
run queue (saturation) busy iowait
# Disk: the important columns are the queue and the wait, not %util
$ iostat -xz 1
Device r/s w/s rkB/s wkB/s aqu-sz r_await w_await %util
nvme0n1 412 1840 6592 29440 38.4 12.10 41.20 100.0
^^^^ ^^^^^ ^^^^^ ^^^^^
SATURATION latency utilisation
# Network: drops and overruns are the errors, not just throughput
$ ip -s link show eth0
RX: bytes packets errors dropped overrun mcast
... 0 4821 0 0
^^^^ receive queue overflow
# Everything, pressure-based
$ cat /proc/pressure/{cpu,io,memory}
%util on an SSD is close to meaningless because SSDs service requests in parallel: 100
percent means "at least one request was in flight," not "at capacity." aqu-sz and
await are the numbers on modern storage.
RED, applied
# Rate
sum(rate(http_requests_total{service="api"}[5m]))
# Errors, as a RATIO (the SLO-relevant form)
sum(rate(http_requests_total{service="api",code=~"5.."}[5m]))
/ sum(rate(http_requests_total{service="api"}[5m]))
# Duration: a HISTOGRAM, so percentiles are computable and aggregatable
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket{service="api"}[5m])) by (le))
Three implementation details determine whether RED is useful:
Histograms, not summaries. A summary computes quantiles per instance, and quantiles do
not average: you cannot combine per-instance p99s into a fleet p99. A histogram's buckets
are additive, so sum(rate(..._bucket)) across instances gives a correct fleet-wide
percentile. This is the same point as the
percentiles page and it is the most common
instrumentation error.
Errors as a ratio, not a count. A count of 200 errors per second means nothing without the rate; 200 of 200,000 is 0.1 percent and 200 of 400 is 50 percent. SLOs are ratios, and an error count on a dashboard invites the wrong conclusion during a traffic drop.
Bucket boundaries chosen for your SLO. The default Prometheus buckets are
.005 .01 .025 .05 .1 .25 .5 1 2.5 5 10, and if your SLO is 300 ms the nearest boundaries
are 250 ms and 500 ms, so histogram_quantile interpolates within a bucket that spans your
target. Add a boundary at your SLO threshold or the number you report is an interpolation
across the decision point.
The four-signal version, and why saturation appears in both
RED omits saturation deliberately, on the argument that a service's saturation shows up as duration. That is true in the limit and unhelpful in advance: by the time saturation appears as latency, the queue is already deep. The Golden Signals add it back, and in practice the useful form for a service is the saturation of its own bounded resources:
Service-level saturation, the things worth measuring:
- connection pool: threads waiting for a connection
- thread pool: queue depth and rejected task count
- inbound queue: depth and oldest-item age
- in-flight requests vs any concurrency limit
In-flight requests against a concurrency limit is the best single saturation metric for a service, and it is the same signal recommended for autoscaling on the autoscaling page, which is not a coincidence: the metric that tells you to add capacity is the metric that tells you you are saturated.
Composing them: the drill-down
Alert: RED says the api service's p99 duration breached the SLO.
│
▼
RED per dependency: which downstream call's duration rose?
│ (the same three metrics, per outbound call)
▼
USE on that dependency's resources: is a resource saturated?
│
▼
database: aqu-sz 38, PSI io some=42% -> disk saturation
or
connection pool: 84 threads waiting -> pool saturation
or
nothing saturated -> the dependency is itself
waiting on something
RED narrows to a component and USE explains it. A monitoring system with only one of them stops at the first or second step.
A worked example: 40,000 metrics and no answers
A platform team with 340 services and a mature-looking observability stack.
Baseline:
metrics collected: ~41,000 series per service
dashboards: ~180
alerts: ~2,400
mean time to identify the
component causing an incident: 47 minutes
alerts that fired and were
actioned: 9% <- 91% ignored or auto-resolved
on-call sentiment: "the dashboards do not help"
Ninety-one percent of alerts ignored is the signal that the monitoring is not instrumenting decisions.
The audit, using the two checklists as a gap analysis.
RED coverage, 340 services:
rate: 312 services (92%)
errors as a RATIO: 88 services (26%) <- mostly counts
duration as a HISTOGRAM: 61 services (18%) <- mostly summaries or means
duration with an SLO-aligned
bucket boundary: 14 services (4%)
USE coverage, per node:
CPU utilisation: yes
CPU saturation (run queue/PSI): NO
memory utilisation: yes (and using the WRONG metric: usage, not working set)
memory saturation (PSI/pgmajfault): NO
disk utilisation (%util): yes
disk saturation (aqu-sz): NO
network errors/drops: NO
connection pool saturation: NO on 338 of 340 services
thread pool queue depth: NO on all
Saturation was missing for every resource, and connection pool waits were missing on 338 of 340 services. Every one of the 41,000 series was a utilisation or a count.
Step 1: RED, properly, on every service.
# A shared library, so it is not 340 separate decisions.
REQUEST_DURATION = Histogram(
"http_request_duration_seconds",
"Request duration",
["service", "method", "route", "code"],
buckets=[.005,.01,.025,.05,.1,.2,.3,.5,.75,1,2,5,10],
# ^^^ the SLO boundary, added explicitly
)
services with correct RED: 61 -> 340
metric series per service: ~41,000 -> ~2,800 (-93%)
The series count fell by 93 percent because the audit also removed metrics nobody queried: 41,000 series per service was largely per-endpoint gauges that had accumulated.
Step 2: saturation, which was the gap that mattered.
// Connection pool saturation: the metric that was missing everywhere.
Gauge.builder("db_pool_pending_threads", pool, HikariPool::getThreadsAwaitingConnection)
.register(registry);
Gauge.builder("db_pool_active", pool, HikariPool::getActiveConnections)
.register(registry);
# Node-level PSI, via node_exporter's pressure collector.
- --collector.pressure
# The alert that did not exist:
alert: ConnectionPoolSaturated
expr: db_pool_pending_threads > 0
for: 2m
annotations:
summary: "Threads are waiting for a database connection. Latency is queueing."
Within three weeks that alert fired for four services, all of which had latency complaints attributed to "the database being slow":
service pool size p99 latency pending threads at p99
payments-api 10 890 ms 41
search-api 10 1,240 ms 68
notify-worker 5 2,100 ms 112
reporting 10 410 ms 8
Every one had a default pool size of 10 against concurrency of 40 to 100. The database was not slow; requests were queueing for a connection, and no metric in 41,000 series showed it.
after right-sizing pools (and adding PgBouncer, see the storage chapter):
payments-api p99: 890 ms -> 74 ms
search-api p99: 1,240 ms -> 118 ms
notify-worker p99: 2,100 ms -> 190 ms
Step 3: the alert reduction, driven by the same framework.
alert audit against the checklists:
alerts on a RESOURCE utilisation with no user impact: 1,412 -> deleted
alerts on a count rather than a rate: 384 -> converted
alerts on a mean rather than a percentile: 291 -> converted
alerts with no runbook: 822 -> deleted or written
alerts that map to a RED or USE signal with user impact: ~180 -> kept
From 2,400 alerts to 218. The deletion criterion was the checklists: an alert that does not correspond to a RED signal on a service or a USE saturation on a resource that a service depends on is measuring something nobody acts on.
before after
alerts 2,400 218
alerts actioned 9% 71%
pages per week 41 6
Step 4: the drill-down dashboard, one per service, in the same shape every time:
Row 1: RED for this service (rate, error ratio, duration percentiles)
Row 2: RED for each outbound call (the same three, per dependency)
Row 3: USE for this service's own resources
connection pool: active/max, PENDING
thread pool: active/max, QUEUE DEPTH, rejected
in-flight requests vs limit
Row 4: USE for the node (CPU, memory, disk, network, all with saturation)
mean time to identify the component: 47 min -> 6 min
A uniform dashboard shape mattered more than any individual panel, because on-call engineers stopped having to learn each service's bespoke layout during an incident.
Final:
before after
metric series per service ~41,000 ~2,800
services with correct RED 61/340 340/340
services with pool saturation 2/340 340/340
alerts 2,400 218
alerts actioned 9% 71%
pages per week 41 6
MTTI (component) 47 min 6 min
p99 latency (4 services with
pool exhaustion) 890-2,100ms 74-190ms
The largest single win was a metric that did not exist: threads waiting for a database connection. Four services had been diagnosed as "the database is slow" for months, and the database was fine.
The transferable practice: use the checklists as a gap analysis rather than as a design. The team already had 41,000 metrics per service. Enumerating resources and asking "do I have saturation for this" found the missing 1 percent that mattered, and enumerating services and asking "is duration a histogram with an SLO-aligned bucket" found that 96 percent of them could not compute a correct fleet percentile.
Production evidence
Brendan Gregg introduced the USE method and publishes a checklist mapping each resource to the specific Linux tools and counters for utilisation, saturation and errors. The checklist form is the point: it is designed to be worked through rather than read.
Tom Wilkie introduced RED at Weaveworks as a service-oriented complement, explicitly building on USE and on Google's Four Golden Signals, and the framing that RED is the same three questions for every service is what makes uniform dashboards possible.
Google's SRE book defines the Four Golden Signals (latency, traffic, errors, saturation) and is explicit that saturation is the leading indicator: it predicts the latency that has not happened yet.
Prometheus histograms versus summaries is documented directly, including that summary quantiles cannot be aggregated. Native histograms (an experimental Prometheus feature) reduce the bucket-choice problem by storing an exponential bucket schema, which addresses the SLO-boundary issue structurally.
PSI was contributed by Facebook and is exposed by node_exporter's pressure collector.
Its adoption as a saturation metric across cgroups and system-wide is the current direction,
because it is directly comparable across resources in a way that per-resource queue depths are
not.
The RED-per-dependency pattern (measuring rate, errors and duration for each outbound call, not just inbound) is what service meshes provide automatically, and it is the reason mesh telemetry is often adopted for observability rather than for traffic management.
The debate
USE or RED? Both, on different things. RED on every service, USE on every resource, because they answer different questions and a system with one of them stops halfway through an incident. The failure mode of USE-only is knowing a disk is busy and not knowing which user-facing thing is broken; of RED-only, knowing a service is slow and having no next step.
Is saturation worth the effort? It is the highest-value gap and the one consistently missing. Utilisation at 100 percent does not distinguish a disk servicing one request at a time from one with 38 queued, and those have a 38x latency difference. Connection pool pending-thread count is the specific metric I would add first to any service, because pool exhaustion is extremely common and invisible in every other signal.
Should you use the Four Golden Signals instead? They are RED plus saturation, and the substance is the same. The reason to prefer RED plus USE is the division of labour: RED is per service and USE is per resource, which makes both enumerable and makes the drill-down from one to the other explicit. The Golden Signals blend service and resource concerns into one list.
Histograms or summaries? Histograms, without qualification, for anything you will aggregate. Summary quantiles are computed per instance and cannot be combined, so a fleet p99 from summaries is not a p99 of anything. The cost is cardinality (a series per bucket) and choosing bucket boundaries, and native histograms address both.
How many metrics is too many? The number is not the metric; the question is whether each one is queried. In the worked example 41,000 series per service dropped to 2,800 by deleting what nothing queried, and the useful additions were about a dozen saturation gauges. Cardinality is a cost you pay continuously and value you receive only when someone looks, so an unqueried metric is pure cost.
Do these frameworks apply to LLM systems and async workloads? Yes, with the mapping adjusted. For an async consumer, RED's "duration" is end-to-end processing time and the saturation signal is queue depth or consumer lag. For an LLM service, rate and errors are unchanged and duration should be split into time-to-first-token and total, because they have different causes. The checklists are about enumerating, and the enumeration works for any component that has resources and serves requests.
Follow-up Q&A
"What are USE and RED and when do you use each?"
USE is per resource: utilisation, saturation and errors for every CPU, disk, network interface, pool and queue. RED is per service: rate, errors and duration for anything handling requests. RED is top-down and tells you a service is unhealthy without saying why; USE is bottom-up and tells you a resource is saturated without saying what it breaks. The drill-down is RED on the service, RED on each dependency to narrow, then USE on that dependency's resources to explain.
"What is saturation and why does it matter more than utilisation?"
Saturation is the amount of work queued that a resource could not service. Utilisation at 100
percent tells you a resource is busy; it cannot distinguish a disk servicing one request at a
time from one with 38 queued, and that is a 38x latency difference. Saturation is the leading
indicator, because it predicts the latency that has not happened yet. PSI is the modern form
and it is comparable across resources: some avg10=42 on I/O means nearly half the recent
time something was stalled waiting for disk.
"What is the most commonly missing metric?"
Connection pool saturation: the count of threads waiting for a connection. Pool exhaustion is extremely common, it produces latency that looks exactly like a slow dependency, and it is invisible in CPU, memory and even in the pool's own active-connection gauge, which reads a healthy 10 of 10. In one audit it was missing on 338 of 340 services, and adding it found four services whose "slow database" was a default pool size of 10 against concurrency of 40 to 100.
"Histogram or summary for latency?"
Histogram, for anything you aggregate. Summary quantiles are computed per instance and
quantiles do not average, so you cannot combine per-instance p99s into a fleet p99: the number
you get is not a percentile of anything. Histogram buckets are additive, so
histogram_quantile over summed bucket rates is correct. And add a bucket boundary at your
SLO threshold, or the reported number is an interpolation across the exact point you are
making decisions at.
"You have 40,000 metrics and incidents still take 45 minutes to diagnose. What do you do?"
Use the checklists as a gap analysis rather than adding more. Enumerate services and check whether each has a rate, an error ratio and a duration histogram; enumerate resources and check whether each has a saturation metric. In one case that found saturation missing for every resource and error ratios missing for 74 percent of services, while the 40,000 series were nearly all utilisation gauges and counts. The additions were about a dozen metrics and the deletions were 93 percent of the series.
"How would you decide which alerts to keep?"
Against the same checklists: an alert should correspond to a RED signal on a service with user impact, or a USE saturation on a resource a service depends on. Alerts on resource utilisation with no user impact, on counts rather than rates, on means rather than percentiles, or with no runbook, are not instrumenting a decision. In one audit that took 2,400 alerts to 218 and the actioned rate from 9 percent to 71.
Common misconceptions
"USE and RED are alternatives." They cover different things. RED tells you a service is broken; USE tells you why. A system with one stops halfway through the drill-down.
"Utilisation is saturation." Utilisation is the fraction of time busy; saturation is the queued work. A disk at 100 percent with a queue depth of 1 is fine and the same disk with a queue depth of 38 is not, and only one metric distinguishes them.
"%util tells you a disk is at capacity." On SSDs it means at least one request was in
flight, because they service requests in parallel. aqu-sz and await are the meaningful
numbers on modern storage.
"Summaries and histograms are interchangeable." Summary quantiles are per instance and cannot be aggregated, so a fleet-wide percentile from summaries is not a percentile.
"More metrics is better observability." Cardinality is a continuous cost and an unqueried metric is pure cost. One team dropped 93 percent of their series and improved time-to-diagnosis sevenfold, because the missing 1 percent was saturation.
Interview delivery note
Say this verbatim: "USE is per resource and RED is per service, and the drill-down is RED on the service, RED on each dependency to narrow, then USE on that dependency's resources to explain. The gap I find every time is saturation: everyone measures utilisation, and a disk at 100 percent with a queue depth of 1 and the same disk with a queue depth of 38 look identical in utilisation and differ by 38x in latency." The division of labour, the drill-down, and the specific gap.
The senior-versus-staff separator is connection pool saturation. A senior engineer applies both frameworks correctly at the node and service level. A staff engineer knows that the service's own bounded resources are where the incidents are, that threads-waiting-for-a- connection is invisible in CPU, memory and even the pool's active-connection gauge (which reads a healthy 10 of 10 during exhaustion), and that four services diagnosed as "the database is slow" for months were queueing on a default pool size of 10.
The second signal is using the checklists as a gap analysis on an existing stack rather than as a greenfield design. Most teams have too many metrics, not too few, and the useful exercise is enumerating resources and asking "do I have saturation for this," which finds a dozen missing gauges among forty thousand existing series.
Further reading
- Brendan Gregg's USE method page, including the per-resource checklist mapping to Linux tools and counters.
- Tom Wilkie's RED method talks and posts, for the service-oriented framing and uniform dashboards.
- Google SRE book, "Monitoring Distributed Systems," for the Four Golden Signals and the argument that saturation is the leading indicator.
- Prometheus documentation on histograms versus summaries, and the native histograms proposal for the bucket-boundary problem.