Using eBPF to debug intermittent latency

"You have intermittent p99 latency spikes with no correlated logs. How would you use eBPF to find the cause?"

What it is

eBPF is a virtual machine inside the Linux kernel that runs small, verified programs attached to kernel and userspace events. You write a program, the kernel's verifier proves it terminates and cannot read arbitrary memory, it is JIT-compiled to native code, and it runs at the attach point with overhead measured in tens of nanoseconds per event.

The practical consequence: you can instrument a running production system, at kernel level, without changing the application, restarting anything, or installing a kernel module. That combination is what makes it different from everything that came before. printf debugging requires a code change and a deploy. strace uses ptrace and can slow a process by an order of magnitude. Kernel modules can panic the box. An eBPF program that fails verification simply does not load.

Commonly confused with a tracing tool. eBPF is the mechanism; bpftrace, BCC, Cilium, Falco, Pixie and Parca are tools built on it. When someone says "we use eBPF", ask which of those they mean.

Also commonly confused with sampling profilers. A profiler tells you where CPU time goes. Most intermittent latency is time not spent on CPU, which is exactly the blind spot eBPF fills.

The problem it solves

Distributed tracing tells you which span was slow. Application metrics tell you that p99 moved. Neither tells you why, because both are instrumented at the application layer and the cause is usually below it: the process was off-CPU waiting for a run queue slot, a page fault, a lock, a disk, or a TCP retransmit.

The specific gap:

Span "db.query" took 340 ms   <- distributed tracing tells you this
Database says the query took 4 ms  <- the database's own metrics say this
                                       Where did 336 ms go?

Candidate answers, none visible from either side: the connection waited in a pool; the thread was descheduled and sat on the run queue; a TCP retransmit cost an RTO; the process hit a major page fault; a sync from another container stalled the block device; a garbage collection pause landed in the middle.

Every one of those is directly observable with eBPF and invisible to application instrumentation. That is the argument for the tool, and it is the framing to lead with in an interview.

Mechanics

Attach points, and what each is for

Prefer tracepoints over kprobes when one exists, because kprobes attach to internal function names that change between kernel releases and a tool built on them silently stops working after an upgrade.

The four questions, and the tool for each

Intermittent latency has a small number of causes and there is a specific investigation for each. This ordering is the answer to the drill.

1. Was the thread off-CPU, and waiting for what?

This is the first question because it is the most common answer and the hardest to see any other way.

# Scheduler latency: time spent runnable but not running.
# If this is high, you are CPU-starved or throttled, not slow.
sudo /usr/share/bcc/tools/runqlat -m 10 1

     msecs        : count     distribution
         0 -> 1   : 84523    |****************************************|
         2 -> 3   : 1204     |                                        |
         4 -> 7   : 89       |                                        |
         8 -> 15  : 12       |                                        |
        16 -> 31  : 4        |                                        |

A tail out to 31 ms in run-queue latency means the process was ready to run and the scheduler had nothing to give it. In a container that usually means CFS throttling, which connects directly to CPU limits and throttling.

# Off-CPU analysis: where the thread blocked, with a stack.
sudo /usr/share/bcc/tools/offcputime -p $(pgrep -n java) -f 30 > out.stacks
# Feed to flamegraph.pl for an off-CPU flame graph.

Off-CPU flame graphs are the single highest-value artifact in this whole area, because they answer "what was it waiting on" with a stack trace rather than a guess.

2. Was it the disk?

# Block I/O latency distribution
sudo /usr/share/bcc/tools/biolatency -m 10 1

# Which process issued the slow I/O, with latency per operation
sudo /usr/share/bcc/tools/biosnoop | awk '$NF > 50'

biosnoop attributes each I/O to a PID, which matters in a shared environment where the stall is caused by a neighbouring container's log rotation rather than by you.

3. Was it the network?

# TCP retransmits, with the connection. Each one costs an RTO,
# typically 200 ms minimum, which shows up as a clean bimodal latency
# distribution rather than a smear.
sudo /usr/share/bcc/tools/tcpretrans

# Connection establishment latency, to separate "slow to connect"
# from "slow to respond"
sudo /usr/share/bcc/tools/tcpconnlat

Retransmits produce a very characteristic signature: a latency histogram with a normal body and a distinct second cluster around 200 ms or 1 s. If you see that shape, check retransmits first.

4. Was it inside the application?

# Latency of a specific userspace function, no code change,
# no restart. This is uprobes, and it is the capability that has
# no equivalent in any other tool.
sudo /usr/share/bcc/tools/funclatency -u \
     '/opt/app/lib/libhandler.so:process_request' -m

bpftrace: the ad hoc layer

BCC tools are prewritten. bpftrace is the language you use when the question is specific to your system.

# Histogram of read() syscall latency for one process
sudo bpftrace -e '
  tracepoint:syscalls:sys_enter_read /pid == 12345/ {
    @start[tid] = nsecs;
  }
  tracepoint:syscalls:sys_exit_read /@start[tid]/ {
    @us = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
  }'

# Every process that spent more than 10 ms off-CPU, with the reason,
# during a 60-second window. This is the "what is stalling" one-liner.
sudo bpftrace -e '
  kprobe:finish_task_switch {
    $prev = (struct task_struct *)arg0;
    @off[$prev->comm] = hist(nsecs - @ts[$prev->pid]);
  }'

# Correlate: which files are being opened during the spike window?
sudo bpftrace -e '
  tracepoint:syscalls:sys_enter_openat {
    printf("%-6d %-16s %s\n", pid, comm, str(args->filename));
  }'

The value of bpftrace in an interview answer is that it demonstrates you can form a hypothesis and test it in one line rather than reaching for a dashboard someone else built.

Continuous profiling: the version you run all the time

The tools above are for an active investigation. The mature posture is a continuous profiler running permanently at low overhead, so that when a spike happens you already have the data.

Parca, Pyroscope and Polar Signals all use eBPF perf-event sampling to profile every process on a host with roughly 1 percent overhead, with no application instrumentation and no language-specific agent. That last property matters in a polyglot estate: one agent profiles the JVM services, the Go services and the Python services identically.

The argument for it: intermittent means you cannot reproduce it on demand, so the only reliable strategy is to already be recording. That is the sentence that turns this from a debugging answer into an operations answer.

The costs, honestly

• Kernel version. Most of this needs 4.9 or later; CO-RE and BTF, which is what makes tools portable across kernels without recompiling, needs 5.2 or later realistically. On an old enterprise kernel you may be limited.
• Privileges. CAP_BPF and CAP_PERFMON, or root. In Kubernetes that means a privileged DaemonSet, which is a security conversation with a real answer (the agent is a known component, it is signed, it is scoped) but it is a conversation.
• Overhead is low but not zero. A kprobe on a very hot path, read() on a busy server, can cost real CPU. Filter in the kernel, not in userspace, and prefer histograms to per-event output.
• Managed platforms may not allow it. Fargate, Lambda and most serverless platforms do not give you the host. This is a real limitation and worth naming because it interacts with the serverless vs containers decision.

A worked example

A search API has a clean p50 of 40 ms and a p99 that spikes to 900 ms for a few minutes, several times a day, with no correlated deploy, no error rate change and nothing in the application logs. Distributed tracing shows the time inside a span labelled opensearch.query, but OpenSearch's own took field reports 8 ms.

336 ms unaccounted for, on the client side of the call. Walk the four questions.

# 1. Off-CPU: is the thread waiting rather than working?
sudo offcputime -p $(pgrep -n java) -f 60 > off.stacks
# Flame graph shows 71% of off-CPU time in
#   futex_wait -> ... -> HikariPool.getConnection

That is already the answer's shape: the thread is blocked acquiring a connection, not waiting for the database. But confirm the rest, because a single signal is a hypothesis.

# 2. Run-queue latency, to rule out CPU starvation
sudo runqlat -m 10 6
#   p99 = 3 ms. Not the cause, but not nothing.

# 3. Retransmits, to rule out the network
sudo tcpretrans
#   2 in 60 seconds, both to a different service. Not the cause.

# 4. Block I/O
sudo biolatency -m 10 3
#   p99 = 2 ms. Not the cause.

So: connection pool starvation. Now the second question, which is the one that actually gets fixed. Why does the pool run dry only sometimes?

# Latency of the query path in the application, per call,
# during the spike window. uprobes, no redeploy.
sudo funclatency -u '/opt/app/app.so:executeSearch' -m 60

# And: how many connections are in flight?
sudo bpftrace -e '
  uprobe:/opt/app/app.so:HikariPool_getConnection { @waiting = count(); }
  uretprobe:/opt/app/app.so:HikariPool_getConnection { @waiting = count(); }
  interval:s:1 { print(@waiting); clear(@waiting); }'

The picture that emerges: a scheduled reindex job runs every four hours, issues large scroll queries through the same pool, holds 18 of the 20 connections for 30 to 60 seconds each, and the interactive traffic queues behind it. Little's Law does the rest: with 2 usable connections and a 8 ms service time, the interactive path can sustain 250 requests per second, and it is receiving 400.

Fixes, in the order I would do them: separate pool for batch work, which is a config change and removes the coupling entirely; then a query timeout on the scroll so a stuck job cannot hold a connection indefinitely; then the pool sizing exercise properly, because 20 was inherited rather than derived.

The point of the example: neither tracing nor metrics could have found this, because both instrument the application and the application's own view was "the database call was slow". eBPF found it in about fifteen minutes without deploying anything, and the crucial artifact was the off-CPU flame graph rather than any dashboard.

Production evidence

Netflix has published extensively on eBPF-based production performance analysis; Brendan Gregg's work there produced both the BCC tool collection and the off-CPU analysis methodology, and his BPF Performance Tools is the canonical reference.

Cilium replaces kube-proxy's iptables rules with eBPF programs at the tc and XDP layers, which removes the O(n) iptables chain traversal that degrades as service count grows. It is the largest production eBPF deployment by node count and is a CNCF graduated project.

Meta runs eBPF at scale for load balancing (Katran, an XDP-based L4 load balancer) and has contributed a large part of the upstream BPF subsystem.

Google uses eBPF in GKE Dataplane V2 (built on Cilium) and contributed BPF LSM for runtime security enforcement.

Falco and Tetragon use eBPF for runtime security: syscall-level detection of container escape attempts and unexpected process execution, with the kernel-level visibility that makes evasion harder than at the userspace layer.

Parca, Pyroscope and Polar Signals run continuous whole-fleet profiling on eBPF perf events at roughly 1 percent overhead, language-agnostically.

The debate

The case for eBPF as a standard part of the toolkit: it answers questions nothing else can, at production-acceptable overhead, without touching the application. For intermittent problems specifically, it is often the only tool that works, because you cannot add instrumentation to a problem you cannot reproduce.

The case against reaching for it first: it is a specialist skill, the tooling assumes kernel familiarity, and most latency problems are not kernel problems. A team that reaches for bpftrace before checking whether their thread pool is sized correctly is doing sophisticated work on the wrong layer. Distributed tracing, application metrics and a look at the pool configuration solve the large majority of cases faster.

My position: instrument the application first, and keep continuous eBPF profiling running permanently so it is available when the application layer runs out. The ordering is: metrics tell you something is wrong, tracing tells you which component, and eBPF tells you why when the component's own view disagrees with the caller's view. That last case, where the client says 340 ms and the server says 4 ms, is the specific signature that should make you reach for it, and I would say so explicitly rather than presenting eBPF as a general-purpose first move.

The one place I would insist on it up front is shared-infrastructure noisy-neighbour problems, because application instrumentation cannot see the neighbour by construction.

Follow-up Q&A

"How would you use eBPF to debug intermittent latency?" I would work four questions in order. Was the thread off-CPU, using offcputime to get an off-CPU flame graph, and runqlat to check for scheduler starvation or CFS throttling. Was it the disk, using biolatency and biosnoop, which attributes I/O to a PID so I can see a noisy neighbour. Was it the network, using tcpretrans, since a retransmit costs an RTO and produces a distinctly bimodal latency histogram. And was it inside the application, using funclatency on a uprobe, which gets me per-function latency with no redeploy. The reason to start with off-CPU is that most intermittent latency is time not spent on CPU, which is exactly the blind spot of a sampling profiler.

"What makes eBPF different from strace or a kernel module?" strace uses ptrace and stops the process twice per syscall, so it can slow a busy process by an order of magnitude, which makes it unusable in production. A kernel module can panic the box. An eBPF program is verified before it loads: the verifier proves it terminates and cannot read arbitrary memory, and then it is JIT-compiled, so the per-event cost is tens of nanoseconds. That combination, safe plus fast plus no application change, is the whole value proposition.

"What's the overhead, honestly?" Low but not free. A histogram-producing probe on a moderately hot path is well under 1 percent. A per-event printing probe on read() for a busy server is not, because you pay the perf buffer and the userspace consumer. The discipline is to aggregate in the kernel rather than shipping every event to userspace, and to filter in the predicate rather than in awk. Continuous profilers run at roughly 1 percent because they sample rather than trace.

"What can't it do?" It cannot see inside a managed platform where you do not control the host, so Lambda and Fargate are out. It needs CAP_BPF or root, which in Kubernetes means a privileged DaemonSet and a security review. It needs a reasonably modern kernel, realistically 5.2 or later for portable CO-RE tooling. And it does not understand your business logic: it can tell you a thread blocked on a futex for 200 milliseconds, and connecting that to "the batch job is stealing connections" is still your job.

"How do you find the cause when the spike is intermittent and you can't reproduce it?" You cannot start an investigation after the fact, so the answer is to already be recording: a continuous eBPF profiler across the fleet, which gives you off-CPU and on-CPU profiles for the spike window retrospectively. Failing that, a triggered capture: a script watching the p99 metric that starts offcputime and biosnoop for 60 seconds when the threshold trips. Both are better than sitting at a terminal hoping it happens while you watch.

"Your trace says 340 ms and the database says 4 ms. Where do you look first?" That gap is client-side by definition, so I would go straight to off-CPU analysis on the calling process. In practice it is one of four things: connection pool starvation, which is the most common; scheduler delay from CPU limits; a TCP retransmit, which shows a 200 ms or 1 s cluster in the histogram; or a garbage collection pause landing inside the call. The off-CPU flame graph distinguishes all four in one artifact, because each has a different blocking stack.

Common misconceptions

"eBPF is a tracing tool." It is a kernel execution environment. Tracing is one application; networking (Cilium, Katran), security (Falco, Tetragon) and profiling (Parca) are others.

"A CPU profiler will find it." Most intermittent latency is off-CPU time, which a sampling CPU profiler does not see at all. This is the single most useful correction in the topic.

"eBPF programs can crash the kernel." The verifier rejects unbounded loops and unchecked memory access before load. The realistic failure mode is that your program does not load, or that a hot probe costs more CPU than you expected.

"You need to recompile per kernel." That was true before CO-RE and BTF. Modern tooling compiles once and relocates against the running kernel's type information.

Interview delivery note

Lead with the gap it fills, not with the technology: "The case where I reach for eBPF is when the caller and the callee disagree. The trace says the database call took 340 milliseconds and the database says 4. That gap is client-side and no application instrumentation can see it, because both ends are instrumented at the application layer."

Then the ordering, which is the actual answer: "I'd work four questions. Off-CPU first, with offcputime for a flame graph and runqlat for scheduler delay, because most intermittent latency is time not spent on CPU and a sampling profiler is blind to it. Then block I/O with biolatency and biosnoop, which attributes to a PID so I can see a noisy neighbour. Then tcpretrans, because a retransmit costs an RTO and gives you a bimodal histogram rather than a smear. Then funclatency on a uprobe for application internals, with no redeploy."

The staff-level move is the operational framing: "but intermittent means I can't reproduce it on demand, so the real answer is continuous eBPF profiling running permanently at about 1 percent overhead, so the data already exists when the spike happens. Investigating after the fact with ad hoc tools is the fallback, not the plan."

And be honest about the limits, because it makes the rest credible: "it needs CAP_BPF, a 5.2-ish kernel for portable tooling, and a host you control, so it's off the table on Lambda or Fargate."

Further reading

• Brendan Gregg, BPF Performance Tools (2019), the canonical reference, and his off-CPU analysis and flame graph write-ups.
• The bpftrace reference guide and the BCC tools directory, both of which double as a catalogue of what is observable.
• Cilium documentation and the eBPF.io "What is eBPF" guide, for the networking and architecture side.
• The Linux kernel BPF documentation, particularly the verifier and CO-RE/BTF sections, for why the safety guarantee holds.