I/O models: blocking, epoll, io_uring, zero-copy

What it is

Four generations of answer to one question: how does a process wait for I/O without wasting the CPU?

Model	Waiting costs	Syscalls per operation	Scales to
Blocking + thread per connection	A thread	1 (`read` blocks)	Thousands
`select`/`poll`	A scan of every fd	1 poll + 1 per ready fd	Hundreds
`epoll`	Nothing (kernel notifies)	1 wait + 1 per ready fd	Hundreds of thousands
`io_uring`	Nothing	Amortised to near zero	Hundreds of thousands, less CPU

The progression is about what the kernel has to do per operation, and each step removes one class of work:

blocking:   the kernel parks a thread. Cost = a thread (~1MB stack, scheduler entry).
select:     the kernel scans ALL fds you passed, every call. O(n) per call.
epoll:      the kernel maintains a ready list. O(ready), not O(watched).
io_uring:   the kernel takes work from a shared ring. No syscall per operation.

Zero-copy is orthogonal to all four: it is about how many times the data itself is copied between kernel and user space, not about how you wait.

What this is confused with: async and non-blocking are not the same thing. epoll is readiness notification: the kernel tells you a socket is readable and you then call read, which does the work in your thread. io_uring is genuine asynchronous completion: you submit a read and the kernel performs it and tells you it is done. The distinction matters because readiness models do not work for regular files (a file is always "ready"), which is why epoll never solved disk I/O and io_uring does.

The problem it solves

The C10K problem, and then the C10M problem. A thread per connection at 10,000 connections is 10,000 threads, roughly 10 GB of stacks at the default 1 MB, and a scheduler run queue that spends its time context-switching rather than working.

Thread-per-connection, 10,000 idle connections:
  memory:              ~8-10 GB of stacks (default 8MB virtual, ~1MB touched)
  context switches:    high, and mostly for threads with nothing to do
  scheduler overhead:  O(runnable threads)

epoll, 10,000 idle connections:
  memory:              ~1 MB of epoll structures + a handful of threads
  context switches:    only when data arrives

The second problem, which is current rather than historical, is syscall cost. Spectre and Meltdown mitigations (KPTI, retpolines) raised syscall overhead substantially, and a busy epoll server makes two to three syscalls per request:

Approximate syscall cost, modern kernel with mitigations:
  pre-2018:     ~60-100 ns
  post-KPTI:    ~200-500 ns

An epoll server at 500k requests/sec x 3 syscalls x 300ns = 0.45 CPU-seconds/sec
                                                            of pure syscall overhead

That is the gap io_uring closes. It is not primarily about a better waiting mechanism; epoll waits fine. It is about submitting many operations with one syscall or none.

The third problem is copies. A conventional file-to-socket transfer copies the data four times and switches context four times, which for a 1 GB file transfer is real CPU spent moving bytes that never change.

Mechanics

Blocking, and why thread-per-connection came back

while (1) {
    int fd = accept(listen_fd, NULL, NULL);
    pthread_create(&t, NULL, handle, (void*)(long)fd);   /* a thread per connection */
}

Simple, and the code reads top to bottom, which is its enormous advantage. It fails at scale on memory and scheduler overhead.

It came back, twice. Go's goroutines are ~2 KB stacks multiplexed onto OS threads by a runtime that uses epoll underneath, and Java 21's virtual threads do the same thing on the JVM. You write blocking code and the runtime does the event loop, which is why the readability advantage no longer costs scalability. See virtual threads vs reactive.

epoll

int ep = epoll_create1(0);
struct epoll_event ev = { .events = EPOLLIN | EPOLLET, .data.fd = conn_fd };
epoll_ctl(ep, EPOLL_CTL_ADD, conn_fd, &ev);

struct epoll_event events[MAX];
for (;;) {
    int n = epoll_wait(ep, events, MAX, -1);      /* returns only READY fds */
    for (int i = 0; i < n; i++) {
        /* still one read() per ready fd, in THIS thread */
        ssize_t r = read(events[i].data.fd, buf, sizeof buf);
    }
}

The key property: epoll_wait returns in O(ready), not O(watched). select scanned every descriptor you passed on every call, so 10,000 watched descriptors with 3 ready cost a 10,000-element scan. epoll maintains the ready list in the kernel as events arrive.

Level-triggered versus edge-triggered is the operational subtlety:

Level-triggered (default): epoll_wait keeps reporting readable while data remains.
                           Forgiving. Read some, get told again.
Edge-triggered (EPOLLET):  reported once per transition to readable.
                           You MUST read until EAGAIN or you will hang forever.

The edge-triggered hang is a classic bug: read 4 KB from a socket holding 16 KB, return to epoll_wait, and it never reports that fd again because no new data arrived. The connection stalls with data sitting in the receive buffer. Edge-triggered is faster (fewer wakeups) and requires draining to EAGAIN every time.

What epoll cannot do: regular files. A file descriptor for a regular file is always "ready," so epoll reports it immediately and the subsequent read blocks on disk anyway. That is why every epoll-based server historically used a thread pool for file I/O, and why io_uring mattered.

io_uring

Two shared ring buffers between userspace and the kernel:

        userspace                       kernel
   ┌──────────────────┐           ┌──────────────────┐
   │ Submission Queue │──────────▶│  reads entries   │
   │  (you write)     │           │  performs I/O    │
   └──────────────────┘           └────────┬─────────┘
   ┌──────────────────┐                    │
   │ Completion Queue │◀───────────────────┘
   │  (you read)      │
   └──────────────────┘

struct io_uring ring;
io_uring_queue_init(256, &ring, 0);

/* Submit a read. No syscall yet. */
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_read(sqe, fd, buf, len, offset);
io_uring_sqe_set_data(sqe, conn);

/* ONE syscall submits everything queued, and waits for completions. */
io_uring_submit_and_wait(&ring, 1);

struct io_uring_cqe *cqe;
unsigned head;
io_uring_for_each_cqe(&ring, head, cqe) {
    handle(io_uring_cqe_get_data(cqe), cqe->res);
}
io_uring_cq_advance(&ring, count);

Three properties that matter:

Batching. Queue 100 operations, submit with one syscall. At 300 ns per syscall that is 30 microseconds saved per 100 operations, and at high request rates it is percentage points of total CPU.

True asynchrony, including for files. io_uring performs the operation rather than telling you it is possible, so regular file I/O is genuinely async for the first time on Linux. That is the capability epoll never had.

SQPOLL: zero syscalls in steady state.

struct io_uring_params p = { .flags = IORING_SETUP_SQPOLL, .sq_thread_idle = 2000 };
io_uring_queue_init_params(256, &ring, &p);
/* A kernel thread polls the submission queue. Userspace writes an entry and
   the kernel picks it up. NO syscall at all while the poller is awake. */

Trading a kernel thread's CPU for zero syscalls is worth it at high request rates and wasteful at low ones, which is the trade to state when asked.

The measured shape, from published benchmarks and consistent with what teams report:

Small random reads, 4 KB, NVMe, one core:
  blocking + thread pool:   ~180k IOPS
  epoll + thread pool:      ~210k IOPS
  io_uring:                 ~480k IOPS
  io_uring + SQPOLL:        ~700k IOPS

Network echo server, 64-byte messages:
  epoll:                    ~1.1M req/s, 78% CPU in kernel
  io_uring:                 ~1.6M req/s, 51% CPU in kernel

And the caveat that decides adoption in many places: io_uring has had a substantial security history. Google disabled it in ChromeOS and Android and restricted it on production servers after a run of exploitable bugs; several container runtimes block it in their default seccomp profiles. A workload that needs io_uring may need a seccomp exception, which is a security conversation rather than a performance one.

Zero-copy

The conventional path copies four times:

read(file_fd, buf, len);  write(sock_fd, buf, len);

disk -> kernel page cache      (DMA)
     -> user buffer            (CPU copy)   <- copy 1
     -> kernel socket buffer   (CPU copy)   <- copy 2
     -> NIC                    (DMA)
4 context switches, 2 CPU copies

sendfile(sock_fd, file_fd, &offset, len);

disk -> kernel page cache      (DMA)
     -> socket buffer          (CPU copy, or scatter-gather DMA on modern NICs)
     -> NIC                    (DMA)
2 context switches, 0-1 CPU copies

sendfile never touches user space, which is why it is the mechanism behind static file serving in nginx and behind Kafka's consumer read path.

// Kafka's fetch path, effectively:
FileChannel.transferTo(position, count, socketChannel);   // -> sendfile(2)

Kafka's throughput rests on this. A consumer fetch is a range of a segment file sent directly from page cache to the socket, with no deserialisation and no user-space copy, which is why a broker can saturate a NIC with modest CPU. It is also why encryption breaks it: TLS requires the data in user space to encrypt it, so enabling TLS on Kafka costs the zero-copy path and measurably increases broker CPU. KTLS (kernel TLS) restores it by performing encryption in the kernel or on the NIC.

Other members of the family:

splice()      move data between two fds via a pipe, no user space
vmsplice()    map user pages into a pipe
MSG_ZEROCOPY  zero-copy send() for sockets, with completion notification
SO_ZEROCOPY   the socket option enabling it

MSG_ZEROCOPY has a threshold: below roughly 10 KB the bookkeeping costs more than the copy, so it helps for large sends and hurts for small ones.

A worked example: a media proxy at 40 percent CPU in `memcpy`

A video segment proxy. Fetches HLS segments from origin, caches on local NVMe, serves to clients. Written in Rust with Tokio (epoll under the hood), serving about 14 Gb/s per node.

Baseline:

throughput per node:         14 Gb/s
CPU utilisation:             84% of 32 cores
  in userspace:              31%
  in kernel:                 53%
p99 latency (cached segment): 41ms
nodes:                       28

Fifty-three percent of CPU in the kernel for a proxy is the signal. A proxy does very little computation; if the kernel is the majority of the time, it is syscalls and copies.

$ perf top -p $(pgrep proxy)
  18.4%  [kernel]  copy_user_enhanced_fast_string     # <- CPU copies
  11.2%  [kernel]  entry_SYSCALL_64                   # <- syscall entry
   8.1%  [kernel]  __sys_recvfrom
   6.9%  [kernel]  tcp_sendmsg
   4.2%  [kernel]  ext4_file_read_iter

18.4 percent in copy_user_enhanced_fast_string is the user-space copy, and 11.2 percent in syscall entry is the mitigation-inflated syscall overhead.

Fix 1: sendfile for cached segments.

The serving path was read() the segment file into a buffer, then write() to the socket.

#![allow(unused)]
fn main() {
// Before: read into a Vec, then write.
let mut buf = vec![0u8; len];
file.read_exact(&mut buf).await?;
socket.write_all(&buf).await?;

// After: sendfile, via tokio's blocking pool for the syscall.
tokio::task::spawn_blocking(move || {
    nix::sys::sendfile::sendfile(sock_fd, file_fd, Some(&mut offset), len)
}).await??;
}

                        before      after
CPU (kernel)            53%         38%
copy_user in perf       18.4%       2.1%
throughput per node     14 Gb/s     19 Gb/s
p99 (cached)            41ms        28ms

Thirty-six percent more throughput from removing one copy. The remaining 2.1 percent of copies were the origin-fetch path, which cannot use sendfile because the data is coming from a socket rather than a file.

Fix 2: io_uring for the disk path.

Cache misses fetched from origin and wrote to NVMe, and the write path was on Tokio's blocking thread pool because epoll cannot do file I/O.

blocking pool threads:       512 (default sized from core count x 
                             a large multiplier, then raised twice)
context switches/sec:        ~840,000

#![allow(unused)]
fn main() {
// tokio-uring for the file path specifically.
let file = tokio_uring::fs::File::create(&path).await?;
file.write_all_at(buf, offset).await?;
}

                        before      after
blocking pool threads   512         16 (kept only for sendfile)
context switches/sec    840k        190k
CPU (kernel)            38%         29%
throughput per node     19 Gb/s     24 Gb/s
p99 (cache miss)        180ms       112ms

A 77 percent reduction in context switches, because file writes were no longer bouncing between a thread pool and the event loop.

The problem they hit: their base image's seccomp profile blocked io_uring_setup.

runtime error: io_uring_setup: Operation not permitted (os error 1)

# A custom seccomp profile allowing the three io_uring syscalls, applied
# ONLY to this workload, on a dedicated node pool.
{
  "defaultAction": "SCMP_ACT_ERRNO",
  "syscalls": [
    {"names": ["io_uring_setup","io_uring_enter","io_uring_register"],
     "action": "SCMP_ACT_ALLOW"},
    ...
  ]
}

That required a security review, and it was granted on the basis that the workload handles no untrusted input beyond HTTP range requests and runs on an isolated node pool. On a multi-tenant cluster it would have been refused, and that is the honest constraint on io_uring adoption.

Fix 3: SQPOLL, evaluated and rejected.

                        io_uring    io_uring + SQPOLL
throughput per node     24 Gb/s     26 Gb/s
CPU utilisation         71%         89%       <- a poller core per ring, always busy
cost per Gb/s           baseline    +14%

Two more Gb/s for 18 points of CPU was not worth it at their utilisation. SQPOLL dedicates a kernel thread that spins, so it is a win when you are syscall-bound at very high rates and a waste otherwise.

Final:

                        before      after
throughput per node     14 Gb/s     24 Gb/s     (+71%)
CPU utilisation         84%         71%
  kernel                53%         29%
context switches/sec    840k        190k
p99 (cached)            41ms        24ms
p99 (cache miss)        180ms       112ms
nodes                   28          17          (-39%)

Thirty-nine percent fewer nodes, from sendfile on the serve path and io_uring on the disk path. No change to the application's logic.

The transferable diagnostic is the kernel-time fraction. A proxy spending 53 percent of CPU in the kernel is doing syscalls and copies rather than work, and perf top names which in one command. copy_user_enhanced_fast_string high in a profile means you are copying data you did not need to copy, and that is a sendfile or splice opportunity almost every time.

Production evidence

nginx uses sendfile for static content and has since its early versions; the sendfile on; directive is one of the first things in any nginx tuning guide, and the mechanism is why it serves static files with so little CPU.

Kafka's throughput rests on sendfile, via FileChannel.transferTo. The design documents are explicit that the consumer read path goes from page cache to socket with no user-space copy, and that this is why brokers can saturate network interfaces. It is also documented that TLS defeats it, which is why enabling encryption measurably raises broker CPU.

io_uring was introduced by Jens Axboe in kernel 5.1 and has been the most significant Linux I/O change in a decade. It is used by ScyllaDB (via Seastar), by recent QEMU storage paths, by Netflix's video pipeline work, and increasingly in databases.

And its security history is equally documented. Google reported that a large share of their kernel exploit submissions in one period targeted io_uring, and subsequently disabled it in ChromeOS and Android and restricted it on production servers. Docker's and containerd's default seccomp profiles block it. Any answer about io_uring that does not mention this is incomplete, because it is the reason many organisations cannot use it.

Go's runtime and Java's virtual threads both implement the thread-per-connection programming model on top of epoll, which is the strongest evidence that the readability of blocking code was worth recovering. Go has experimented with io_uring backends and has not adopted one by default, partly for portability and partly for the security surface.

The debate

Should you use io_uring? Only if you are measurably syscall-bound or need genuinely async file I/O, and only if you can accept the security posture. The performance case is real (roughly 2x on small random reads, 45 percent on a network echo server in published benchmarks) and the security history is also real, and container runtimes block it by default. For most services, epoll via a runtime like Go, Tokio or Netty is fast enough and the syscall overhead is not the bottleneck. Measure kernel time first: if you are not spending double-digit percentages in entry_SYSCALL_64, io_uring is solving a problem you do not have.

Is epoll obsolete? No, and it will not be for a long time. It is portable across every Linux kernel in production, it is what every mature runtime uses, it is not blocked by seccomp profiles, and for a workload doing meaningful work per request the syscall overhead is a small fraction. io_uring wins at very high operation rates with small operations, which is a specific shape.

Thread-per-connection or an event loop? With Go's goroutines or Java's virtual threads, write blocking code: the runtime multiplexes onto an event loop and you keep readable stack traces and working debuggers. The event-loop programming model is now an implementation detail you should not have to write by hand, and choosing a reactive style for scalability that virtual threads provide is the mistake described on the virtual threads page.

Is zero-copy always worth it? For large file-to-socket transfers, unambiguously. MSG_ZEROCOPY for sockets has a threshold around 10 KB below which the completion bookkeeping costs more than the copy. And zero-copy is incompatible with anything that needs to see the data: compression, encryption, transformation. TLS versus sendfile is the trade people meet most often, and KTLS is the resolution where it is available.

What should you actually optimise first? Not the I/O model. The ordering that pays is: reduce the number of operations (batching, larger reads, connection reuse), then remove copies (sendfile where the data passes through unchanged), then reduce syscalls (io_uring). Most services are doing far more small operations than they need to, and fixing that is cheaper and safer than changing the I/O model.

Follow-up Q&A

"Walk me from select to io_uring."

select and poll scan every descriptor you pass on every call, so they are O(watched). epoll keeps a ready list in the kernel, so epoll_wait returns in O(ready), which is what made hundreds of thousands of connections feasible. But epoll is readiness notification: it tells you a socket is readable and you still call read yourself, and it cannot help with regular files because a file is always ready. io_uring is completion-based: you submit operations to a shared ring and the kernel performs them, so you get batching, genuinely async file I/O, and with SQPOLL no syscalls at all in steady state.

"What is the difference between readiness and completion?"

Readiness models (select, poll, epoll) tell you an operation would not block, and you then perform it in your thread. Completion models (io_uring, Windows IOCP) perform the operation for you and tell you it finished. The practical consequence is that readiness cannot work for regular files, since a file descriptor is always ready and the subsequent read blocks on disk anyway. That is why every epoll server used a thread pool for file I/O until io_uring.

"What is the edge-triggered epoll bug?"

With EPOLLET, an fd is reported once per transition to readable. If you read 4 KB from a socket holding 16 KB and return to epoll_wait, it never reports that fd again, because no new data arrived, and the connection stalls with data sitting in the receive buffer. Edge mode requires reading until EAGAIN every time. Level-triggered keeps reporting while data remains, which is forgiving and costs extra wakeups.

"What does sendfile save?"

Two context switches and one or two CPU copies. The conventional path is disk to page cache by DMA, page cache to a user buffer by CPU copy, user buffer to socket buffer by CPU copy, socket buffer to NIC by DMA. sendfile goes disk to page cache to socket buffer to NIC, never entering user space. It is why nginx serves static files cheaply and why Kafka's consumer path can saturate a NIC, and it is also why enabling TLS on Kafka raises broker CPU measurably: encryption requires the data in user space, unless KTLS is available.

"Why is io_uring controversial?"

A substantial security history. Google reported a large share of kernel exploit submissions in one period targeting it, and disabled it in ChromeOS and Android and restricted it on production servers. Docker and containerd block the three io_uring syscalls in their default seccomp profiles. So adopting it is a security conversation as much as a performance one: in one case it needed a custom seccomp profile and a dedicated node pool, and on a multi-tenant cluster it would have been refused.

"How do you know your I/O model is the bottleneck?"

Kernel time as a fraction of CPU, then perf top. A proxy spending 53 percent of CPU in the kernel is doing syscalls and copies rather than work. copy_user_enhanced_fast_string high in the profile means data is being copied that need not be, which is a sendfile or splice opportunity. entry_SYSCALL_64 high means syscall overhead, which is where batching or io_uring helps. If neither is prominent, the I/O model is not your problem.

Common misconceptions

"Non-blocking means asynchronous." epoll is readiness notification and the read still happens in your thread. Completion-based asynchrony (io_uring) is a different model, and the difference is exactly why epoll never solved file I/O.

"epoll handles all I/O." Not regular files: a file fd is always ready, so epoll reports it immediately and the read blocks on disk anyway. Every epoll server used a thread pool for files until io_uring.

"Thread-per-connection does not scale." It did not with OS threads. With goroutines or virtual threads at a few kilobytes each, multiplexed onto an event loop by the runtime, it scales fine and you keep readable code and working stack traces.

"io_uring is strictly better." It is faster at high operation rates and it carries a security history that has led major vendors to disable it and container runtimes to block it by default. That constraint decides adoption more often than the performance does.

"Zero-copy is always faster." MSG_ZEROCOPY costs more than a copy below roughly 10 KB, and zero-copy is incompatible with anything that needs to read the data: compression, encryption, transformation.

Interview delivery note

Say this verbatim: "The progression is about what the kernel does per operation: select scans everything you watch, epoll keeps a ready list so it is O(ready), and io_uring takes work from a shared ring so there is no syscall per operation. The distinction that matters is readiness versus completion, which is why epoll never solved file I/O and io_uring does." The organising principle plus the specific consequence.

The senior-versus-staff separator is naming the io_uring security posture unprompted. A senior engineer describes the ring architecture and the performance gain. A staff engineer adds that Google disabled it in ChromeOS and Android after a run of exploitable bugs, that Docker and containerd block its syscalls in the default seccomp profile, and that adopting it therefore needs a security exception and is refused outright on multi-tenant clusters. Knowing why you cannot use the fast thing is more useful than knowing it is fast.

The second signal is the diagnostic ordering: kernel-time fraction, then perf top, then decide. Saying "if copy_user_enhanced_fast_string is high you have a copy to remove, and if entry_SYSCALL_64 is high you have syscalls to batch, and if neither is prominent the I/O model is not your problem" shows you would measure before rewriting.

The Staff & Team Lead Interview Reference