Drills 19 to 23: compute, Kubernetes and the kernel

Five drills, ninety seconds each, out loud. These are the ones where a candidate either has operated a system or has read about operating one, and the difference shows in whether the answer contains a sequence and a number.

The shape that works for all five: name the thing you would check first and why it is first, then the ordering, then the number that makes it concrete. Candidates who list possible causes without ordering them sound like they are reciting; candidates who say "I'd check X first because it rules out half the space" sound like they have done it.

Drill 19. A pod is healthy but slow. Diagnose in order.

Healthy but slow means the health check passes and the work is late, so I'd start by separating "it isn't running" from "it's running slowly", because those have completely different causes.

First check is CFS throttling, container_cpu_cfs_throttled_seconds_total, because it's the one that's invisible in the metric everyone looks at. Average CPU utilisation can sit at 30 percent while the container is being descheduled inside every 100-millisecond period, and nothing on a normal dashboard shows that.

Second is off-CPU time. If it's not throttled, the thread is waiting on something: a connection pool, a lock, a disk, or a dependency. offcputime gives me a flame graph with the blocking stack, which distinguishes all four in one artifact.

Third is the dependency. Compare what my trace says the call took against what the dependency says it took. If my span says 340 milliseconds and the database says 4, that gap is client-side, and it's almost always pool starvation.

Fourth is memory: not OOM, because that would kill it, but GC pressure or page cache eviction causing the working set to come off disk.

What I wouldn't do is restart it, because that destroys the evidence and it comes back in twenty minutes.

Depth signal: naming CFS throttling first with the reason it is first, and the caller versus callee latency gap as the specific signature of pool starvation.

Full treatment: A pod is healthy but slow.

Drill 20. Why might removing CPU limits improve latency?

Because a CPU limit isn't a limit on average utilisation, it's a quota per 100-millisecond period. If the container has a limit of one core and a burst of work needs two cores for 50 milliseconds, it exhausts the quota in the first 50 and gets descheduled for the remaining 50, even though the machine has idle cores sitting there.

So average utilisation can read 30 percent while p99 latency is dominated by throttling, and that's the disconnect: people look at utilisation, see headroom, and conclude CPU isn't the problem.

The metric that shows it is container_cpu_cfs_throttled_seconds_total, and I'd want that on the dashboard before anything else. In the cases I've seen, removing the limit improved p99 by around 40 percent while average utilisation barely moved.

The thing I'd be careful about is that removing limits removes isolation, so a noisy neighbour can now starve everyone on the node. My position is: keep requests, which drive scheduling and are what actually reserve capacity, and remove limits for latency-sensitive services while keeping them for batch work. And there was a period where the kernel had a throttling bug that made this much worse than it should have been, fixed around 4.18, so on an old kernel this is worse than the design implies.

Depth signal: the period-quota mechanism rather than "limits cause throttling", and the requests-versus-limits distinction in the recommendation.

Full treatment: CPU limits and CFS throttling.

Drill 21. Serverless or containers for this workload? Walk your math.

I'd compute both rather than argue about it. Functions are priced per request plus GB-seconds; containers are vCPU-hours and GB-hours, and I'd size the container from Little's Law: concurrency is throughput times latency.

So at 50 requests a second and 200 milliseconds, that's 10 in flight, about two vCPU with headroom, roughly $70 a month. The same traffic on functions is about 130 million requests, which is $26 of request charge plus $216 of compute, so about $240. Containers win by three and a half times.

Now drop it to 2 requests a second. The container still costs $70 because it's always on; the functions cost about $10. Functions win by seven times, and nothing about the code changed.

So the crossover is roughly 30 to 40 percent average utilisation, and committed-use discounts push it down toward 20. Below that you're buying idle time; above it you're paying a per-request premium on capacity you're already using.

But I'd check constraints before cost, because they override it. Connection management first: a function per invocation can't hold a pool, so hundreds of concurrent functions exhaust the database, and that's the most common way this fails, at scale rather than on the bill. Then the 15-minute execution limit, then cold-start tolerance in a user-facing path.

And I'd expect the answer to be a mixture. Synchronous API on containers, event handlers and scheduled jobs on functions.

Depth signal: computing both sides with Little's Law, stating the crossover as a utilisation percentage, and checking constraints before cost.

Full treatment: Serverless or containers.

Drill 22. Where does fsync fit in a durability guarantee?

It's the durability boundary. A write returns when the data is in the kernel page cache, which is RAM, so it survives a process crash and nothing else. fsync pushes those pages to the device and flushes the device's own cache, so it survives power loss.

In a database the commit path is: append the commit record to the write-ahead log, fsync the log, then acknowledge the client. Everything before that fsync can be lost; everything after is promised. And the log exists precisely so that one sequential flush covers a transaction that touched many random pages.

The cost is five to ten milliseconds on a spinning disk, one to two on a consumer SSD, and under a hundred microseconds on an enterprise NVMe with a capacitor-backed cache. Group commit amortises it, so several transactions share one flush.

The part worth knowing is what happens when it fails. Since the 2018 PostgreSQL fsyncgate work we know that on Linux a writeback failure can be reported to one caller and the dirty pages then marked clean, so a retried fsync returns success against data that's gone. Which is why PostgreSQL 12 onwards panics on fsync failure rather than retrying: deliberately crashing and replaying the log is the safe response.

And fsync only covers one machine. Machine loss needs replication, which is why Raft's commit latency is a local flush plus a quorum round trip, and why Kafka deliberately doesn't fsync per message and relies on acks=all instead.

Depth signal: fsyncgate and the panic-rather-than-retry consequence, plus widening from one machine to replication at the end.

Full treatment: Where fsync fits in durability.

Drill 23. How would you use eBPF to debug intermittent latency?

The case where I reach for it is when the caller and the callee disagree. The trace says the database call took 340 milliseconds and the database says 4. That gap is client-side and no application instrumentation can see it, because both ends are instrumented at the application layer.

I'd work four questions in order. Off-CPU first, with offcputime for a flame graph and runqlat for scheduler delay, because most intermittent latency is time not spent on CPU and a sampling profiler is blind to it. Then block I/O with biolatency and biosnoop, which attributes I/O to a PID so I can see a noisy neighbour. Then tcpretrans, because a retransmit costs a retransmission timeout and produces a distinctly bimodal histogram rather than a smear. Then funclatency on a uprobe for application internals, with no redeploy.

But intermittent means I can't reproduce it on demand, so the real answer is continuous eBPF profiling running permanently at about one percent overhead, so the data already exists when the spike happens. Investigating after the fact with ad hoc tools is the fallback, not the plan.

And I'd be honest about the limits: it needs CAP_BPF, a reasonably modern kernel for portable tooling, and a host you control, so it's off the table on Lambda or Fargate.

Depth signal: the caller-callee disagreement as the trigger, off-CPU before everything else with the reason, and continuous profiling as the actual answer to "intermittent".

Full treatment: eBPF for intermittent latency.

How to practise these

These five reward a different practice than the leadership drills. Say the ordering out loud and then justify the first item. The failure mode here is not vagueness, it is a correct but unordered list: "it could be CPU, memory, network, the database, GC" is technically complete and tells the interviewer nothing about how you would actually work.

The three tests to apply to your own answer:

1. Did you name what you would check FIRST, and why? "CFS throttling, because it's invisible in average utilisation" is an answer. "I'd look at metrics" is not.
2. Did you include one number? 40 percent p99 improvement, 7.8 milliseconds per token, the 30 to 40 percent utilisation crossover. A number turns a description into a memory.
3. Did you name a limit? Removing CPU limits removes isolation. eBPF needs a host you control. Functions can't hold a connection pool. Naming the constraint is what separates a recommendation from a slogan.

And the pattern that runs through all five: the useful answer is usually about a metric nobody looks at. Throttled seconds rather than CPU utilisation. Off-CPU time rather than a CPU profile. Connection pool wait rather than query time. That is the shape of the depth signal in this whole chapter, and it is worth saying explicitly when you give the answer.