CPU limits and CFS throttling

What it is

A Kubernetes CPU limit is enforced by the Linux Completely Fair Scheduler's bandwidth controller. The kernel gives the container's cgroup a quota of CPU time per fixed period (100 ms by default). Once the cgroup's threads have consumed the quota within a period, every thread in the cgroup is descheduled until the next period begins. That stall is CFS throttling.

The name misleads people. Throttling does not mean "runs at reduced speed". It means "runs at full speed until the budget is gone, then stops completely for the remainder of the period". A container limited to 1 CPU with 8 runnable threads burns 100 ms of quota in 12.5 ms of wall clock, then sits frozen for 87.5 ms. The average utilisation looks like 1 core, which is what you asked for, and the latency distribution has an 87 ms cliff in it that you did not.

CPU is a compressible resource: exceeding the limit slows you down. Memory is incompressible: exceeding the limit gets you OOMKilled. That asymmetry is why the advice for the two is different, and why "always set limits" is bad advice when applied uniformly.

The problem it solves, and the problem it creates

Limits exist to bound blast radius. Without them a runaway loop in one pod can starve every other pod on the node, and in a multi-tenant cluster a tenant can consume capacity they did not pay for. Limits also determine QoS class: a pod whose limits equal its requests for every resource is Guaranteed, which puts it last in the eviction order and makes it eligible for exclusive CPU pinning via the static CPU manager policy.

The problem they create is that the quota is enforced against wall-clock periods rather than against contention. A container that is well under its limit on average is throttled whenever its instantaneous parallelism exceeds quota / period, even on a node that is 20 percent idle. You are being throttled against a budget, not against a neighbour.

Mechanics

The cgroup interface

On cgroups v2, a container's limit is one file:

# Inside a pod with resources.limits.cpu: "1"
$ cat /sys/fs/cgroup/cpu.max
100000 100000        # quota_us period_us  -> 100ms of CPU per 100ms wall

# limits.cpu: "500m"
$ cat /sys/fs/cgroup/cpu.max
50000 100000         # 50ms per 100ms

# no limit set
$ cat /sys/fs/cgroup/cpu.max
max 100000

# Requests become the CPU weight (shares), used only under contention.
$ cat /sys/fs/cgroup/cpu.weight
39                   # derived from requests.cpu

The evidence that you are being throttled is in cpu.stat:

$ cat /sys/fs/cgroup/cpu.stat
usage_usec 412300000
nr_periods 300000        # periods elapsed
nr_throttled 41200       # periods in which we hit the quota  <- 13.7%
throttled_usec 2871000000  # total time frozen: 2871 seconds

In Prometheus, the same thing:

# Fraction of periods in which the container was throttled.
rate(container_cpu_cfs_throttled_periods_total{pod=~"search-.*"}[5m])
  / rate(container_cpu_cfs_periods_total{pod=~"search-.*"}[5m])

# Seconds of freeze per second of wall clock. Above ~0.02 is worth investigating
# for a latency-sensitive service; above 0.1 you have found your p99.
rate(container_cpu_cfs_throttled_seconds_total[5m])

Why parallelism, not utilisation, causes it

The condition for throttling in a period is

$$\sum_{\text{threads}} \text{cpu-time consumed} > \text{quota}$$

With $P$ runnable threads all doing work, the quota is consumed after $\text{quota} / P$ of wall clock. So:

Limit	Runnable threads	Quota exhausted after	Frozen for
1 CPU	1	100 ms (never throttled)	0
1 CPU	4	25 ms	75 ms
1 CPU	16	6.25 ms	93.75 ms
2 CPU	16	12.5 ms	87.5 ms

The runtime picks the thread count, and by default it picks it from the number of CPUs it can see, which is the node's core count, not the cgroup quota. A Go binary sets GOMAXPROCS to runtime.NumCPU(). A JVM before container support, or with it disabled, sizes GC threads, the common ForkJoinPool and the JIT compiler threads the same way. On a 64-core node with a 1 CPU limit, that is 64 threads sharing 100 ms of quota, and it is the most common origin of this problem: the container is not busy, it is merely parallel.

The historical kernel bug

Between roughly Linux 4.18 and 5.4 there was a genuine kernel defect in the bandwidth controller: per-CPU quota slices expired in a way that could throttle a cgroup well below its configured quota, producing throttling on applications using a small fraction of their limit. It was fixed in 5.4 (and backported by distributions) by removing slice expiration. If you are debugging this on an old kernel, check the version before you tune anything, because the symptom is identical and the fix is different.

A worked example

A JVM search service. Node has 32 cores. The pod is configured requests.cpu: 1, limits.cpu: 2. Steady-state CPU usage is 0.7 cores, so on every dashboard the container looks comfortable at 35 percent of its limit. Reported symptom: p50 is 40 ms, p99 is 340 ms, and the p99 is spiky rather than correlated with load.

Investigation:

$ kubectl exec search-7d9 -- cat /sys/fs/cgroup/cpu.stat
nr_periods 180000
nr_throttled 21600      # 12% of periods throttled
throttled_usec 1490000000   # 1490s frozen over 5 hours => ~8% of wall clock

$ kubectl exec search-7d9 -- java -XX:+PrintFlagsFinal -version | grep -E 'ActiveProcessorCount|ParallelGCThreads'
     int ActiveProcessorCount    = -1
    uint ParallelGCThreads       = 23      # sized from 32 host cores

The JVM sized its parallel GC to 23 threads because UseContainerSupport derives processor count from the quota only when a limit is set as an integer multiple of a CPU in the way the JVM expects, and in this deployment the container saw the host's 32 cores. A young-generation collection therefore tries to run 23 threads against a 200 ms quota, consumes it in under 10 ms, and the entire process, including the request threads, freezes until the next period. The 87 to 190 ms freeze lands directly in p99.

Three fixes, applied in order:

Match runtime parallelism to the quota. -XX:ActiveProcessorCount=2, or for Go, import go.uber.org/automaxprocs which reads cpu.max and sets GOMAXPROCS accordingly. p99 drops to about 120 ms immediately because GC no longer detonates the quota.
Raise the limit to give headroom for bursts. limits.cpu: 4 against a 0.7 core steady state. Throttled periods fall to under 1 percent. p99 to about 70 ms.
Reconsider whether the limit earns its keep at all. On a dedicated node pool for this workload, removing the CPU limit entirely (keeping the request at 1) eliminates throttling by construction while requests still guarantee the service 1 core under contention. p99 to about 55 ms, which matches the p50 plus normal variance.

Each step is cheaper than the last in engineering effort and more controversial in policy, which is exactly the shape of the discussion to have out loud.

Production evidence

Kubernetes' own documentation states that CPU is a compressible resource and that exceeding a CPU limit results in throttling rather than termination, and the cpu.max mechanism is documented in the kernel's cgroup v2 admin guide.

The practice of setting CPU requests without limits for latency-sensitive workloads has been publicly argued for by Kubernetes maintainers and adopted by a number of large operators; Zalando's engineering team and Buffer both published detailed write-ups of throttling incidents that were resolved by removing CPU limits, and Zalando documented the policy in their cluster configuration guidance. Datadog and Grafana both ship dashboards with container_cpu_cfs_throttled_periods_total as a first-class panel, which tells you how routinely this is encountered.

The automaxprocs library came out of Uber, built specifically because Go services in containers were being throttled by a GOMAXPROCS derived from host core count. On the JVM side, container awareness (UseContainerSupport) has been on by default since JDK 10, and ActiveProcessorCount exists as the explicit override precisely because the automatic derivation does not always produce what you want.

The debate

The case for always setting CPU limits: predictability and fairness. Without limits, a pod's performance depends on its neighbours, so a load test on an empty node tells you nothing about production. Capacity planning becomes guesswork. Multi-tenant clusters need them as a hard requirement, since "trust every team not to burst" is not a security model. And Guaranteed QoS, which requires limits to equal requests, is what gets you exclusive CPU pinning and the best eviction protection.

The case against, for latency-sensitive services: requests already provide the guarantee that matters. Under contention, cpu.weight derived from requests gives you your proportional share. When the node is not contended, a limit prevents you from using idle capacity for no benefit to anyone, and charges you a tail-latency penalty for the privilege. Throttling is invisible on utilisation dashboards and shows up only in p99, which is the worst possible combination of harmful and hard to find.

My position: always set CPU requests, accurately, from measured usage. Always set memory limits, because memory is incompressible and the alternative is a node-level OOM that takes down innocent pods. For CPU limits: set them generously (two to four times the request) on shared clusters, and consider omitting them on dedicated node pools for latency-sensitive services where you control every workload on the node. Regardless of the decision, make the container runtime's thread count follow the quota, because that single change removes most of the throttling most teams experience, and it is uncontroversial.

Removing CPU limits is the wrong answer in a multi-tenant cluster, on nodes running batch alongside serving, when your compliance regime requires enforceable resource boundaries, or when you need Guaranteed QoS for CPU pinning of a latency-critical workload. Say those exceptions unprompted; the interviewer is usually checking whether you understand that this is a policy tradeoff and not a universal trick.

Follow-up Q&A

"Why might removing CPU limits improve latency?" Because the limit is enforced by freezing every thread in the cgroup for the remainder of a 100 ms period once the quota is consumed. A bursty, parallel workload with a modest average consumes its quota early in the period and eats a freeze of up to nearly 100 ms, which lands in tail latency. Removing the limit removes the freeze; requests still guarantee a proportional share under contention.

"How do you prove throttling is your problem rather than a coincidence?" container_cpu_cfs_throttled_periods_total divided by container_cpu_cfs_periods_total gives the fraction of periods throttled, and throttled_seconds gives the magnitude. Correlate the throttled-seconds series against p99 latency: if they move together, you have your answer. Then verify the mechanism by checking the runtime's thread count against the quota, because the throttling is usually a symptom of oversized parallelism rather than of genuine CPU need.

"Does raising the CPU limit always help?" It reduces the frequency of throttling but not the mechanism. If the parallelism is wildly mismatched to the quota, say 64 GC threads against 4 CPUs, you still exhaust the quota early and still freeze. Fixing the thread count is the change that removes the failure mode; raising the limit only makes it rarer.

"What is the equivalent problem for memory?" There is no throttling: the kernel's OOM killer terminates the process and the kubelet reports OOMKilled. The JVM analogue of this whole discussion is sizing -Xmx to the container limit, which guarantees an OOMKill because the JVM's total footprint is heap plus metaspace plus thread stacks plus code cache plus direct buffers. Use -XX:MaxRAMPercentage around 65 to 75 percent rather than an absolute -Xmx equal to the limit.

"A pod is healthy but slow. Walk your diagnosis in order." Throttling metrics first, because they are one query and catch a large fraction of cases. Then GC or runtime pause logs. Then per-dependency latency histograms to see whether the slowness is downstream. Then node-level pressure: is a neighbour saturating a shared resource, is the node's CPU steal time non-zero, is disk I/O saturated. Then DNS, specifically whether ndots: 5 is causing five lookups per external resolution. Then probe configuration, because an aggressive liveness probe on a throttled container causes restarts that look like a different problem entirely.

Common misconceptions

The most damaging is that a container using 35 percent of its CPU limit cannot be throttled. Averages hide the mechanism entirely; throttling is a function of instantaneous parallelism against a 100 ms budget, and a container can average 35 percent while being frozen for 10 percent of wall clock.

The second is that throttling degrades gracefully, that the container just runs proportionally slower. It does not; it stops. The distribution is bimodal, not shifted, which is why the effect appears in p99 and is invisible in the mean.

The third is that setting the limit equal to the request is universally good practice because it yields Guaranteed QoS. Guaranteed QoS is genuinely valuable for eviction protection and CPU pinning, and for a bursty service it also guarantees you will be throttled during every burst. Know which property you are buying.

Interview delivery note

Say this: "CFS enforces the limit by freezing the whole cgroup once it has used its quota within a 100 millisecond period, so throttling is about instantaneous parallelism, not average utilisation. A JVM or Go runtime that sizes its thread pool from the node's core count instead of the cgroup quota will burn a 1 CPU quota in a few milliseconds and then stall for the rest of the period, which lands straight in p99. I check container_cpu_cfs_throttled_periods_total first, then make the runtime's parallelism follow the quota."

The depth signal is the ratio, not the concept. Saying "a 1 CPU limit with 16 runnable threads exhausts the quota in 6 milliseconds and freezes for 94" shows you have looked at cpu.stat on a real incident. Following it with the policy tradeoff, including the cases where removing limits is wrong, shows you have had to defend the decision to a platform team.

The Staff & Team Lead Interview Reference