Never block the event loop: the number-one WebFlux production bug

What it is

An event loop is a thread that runs a loop: take the next ready I/O event, run its handler, repeat. Netty (and therefore Spring WebFlux, and therefore most reactive JVM services) runs a small fixed pool of them, by default one per CPU core. Every connection is assigned to one event loop for its lifetime, and every callback for that connection runs on that thread.

Blocking the event loop means calling anything that parks that thread: a JDBC query, a RestTemplate call, Thread.sleep, a synchronous file read, .block() on a reactive type, a synchronized block that contends, or a CompletableFuture.get().

The reason this is the number-one WebFlux production bug is arithmetic. A servlet container has 200 threads, so blocking one costs 0.5 percent of capacity. A WebFlux service on an 8-core box has 8 event loops, so blocking one costs 12.5 percent, and it does not cost 12.5 percent of one request, it stalls every connection assigned to that loop. With 10,000 connections spread evenly, blocking one event loop for 200 milliseconds freezes 1,250 connections for 200 milliseconds.

What this is confused with: it is not a performance problem, it is a concurrency ceiling. The symptom is not "each request is a bit slower." The symptom is that total throughput becomes event_loops / service_time and stays there no matter how much traffic you send or how many cores you add.

The problem it solves

Understanding the failure is the point, so here is what it looks like from the outside, because the signature is distinctive and is what you will be shown in an interview:

concurrency    throughput      p99 latency     CPU
     10           44 rps          230ms         14%
     50           45 rps        1,120ms         15%
    200           44 rps        4,600ms         15%
   1000           43 rps       23,000ms         14%

Flat throughput, linearly growing latency, and low CPU. Every one of those three is diagnostic:

Flat throughput means a fixed number of servers, not a resource limit. Adding load adds queue, not work.
Linear latency growth with concurrency is the queue in front of those servers.
Low CPU rules out computation being the bottleneck. The threads are not busy, they are parked.

Little's Law names the number of servers directly:

$$L = \lambda W \quad\Rightarrow\quad \text{busy servers} = 44 \times 0.23 = 10.1$$

Ten busy servers. If the box has 8 cores and the service has 8 event loops (plus a couple of requests in other stages), you have found it without a profiler. That calculation takes ten seconds and is the single most useful diagnostic on this page.

Mechanics

What "blocked" means to Netty

An event loop's job is:

// Conceptually, what an event loop thread does forever:
while (!shutdown) {
    List<Event> ready = selector.select();        // which channels have I/O ready
    for (Event e : ready) {
        e.handler().run();                        // YOUR CODE RUNS HERE
    }
    runScheduledTasks();
}

Your handler runs inside that loop. While it runs, the loop is not selecting, so no other channel assigned to this loop gets serviced. A 200-millisecond JDBC call means 200 milliseconds during which every other connection on that loop is invisible: their data sits in socket buffers, their timeouts tick, their clients wait.

There is a second-order effect worth knowing: readTimeout and idle-state handlers also run on the event loop, so a blocked loop cannot fire the timeouts that would have cleaned up its own stuck connections.

The blocking calls, ranked by how often they appear

Call	Why it appears	Fix
JDBC / JPA / Hibernate	The team migrated the web layer and not the data layer	R2DBC, or `boundedElastic`, or do not use WebFlux
`RestTemplate`, Apache HttpClient	Copied from an existing service	`WebClient`
`.block()` on a Mono/Flux	"Just to get the value here"	Restructure with `flatMap`
`Thread.sleep`	Retry backoff written by hand	`Mono.delay`, `retryWhen`
`synchronized` on a contended lock	Shared cache or counter	Lock-free structure, or `single()` scheduler
File I/O (`Files.readAllBytes`)	Config or template loading per request	Load once at startup, or `boundedElastic`
`CompletableFuture.get()`	Bridging an async API badly	`Mono.fromFuture`
Logging to a synchronous appender	Nobody thinks of logging as I/O	Async appender with a bounded queue

The last one deserves emphasis because it is genuinely surprising. A synchronous file or console appender does a blocking write per log statement. Under load, with a slow disk or a container writing to a captured stdout that some agent is reading slowly, that write can take milliseconds. Logback's AsyncAppender or Log4j2's async logger fixes it, and the queue must be bounded with a discard policy or you have moved the problem to memory.

`.block()` is the one that looks innocent

// This compiles, passes unit tests, and destroys the service under load.
@GetMapping("/order/{id}")
public Mono<OrderView> get(@PathVariable String id) {
    Order order = orderService.find(id).block();          // blocks the event loop
    Customer c = customerService.find(order.customerId()).block();   // again
    return Mono.just(new OrderView(order, c));
}

Reactor 3.2+ actually throws here (block() on a non-blocking thread raises IllegalStateException), which is a genuine improvement. The pattern survives in disguised forms it cannot detect: .toFuture().get(), .toIterable(), .blockFirst() inside a map, or a library that blocks internally.

The correct version composes:

@GetMapping("/order/{id}")
public Mono<OrderView> get(@PathVariable String id) {
    return orderService.find(id)
        .flatMap(order -> customerService.find(order.customerId())
            .map(c -> new OrderView(order, c)));
}

And when the two calls are independent, they should run concurrently, which is something the blocking version could not do at all:

return Mono.zip(orderService.find(id), configService.current())
           .map(t -> new OrderView(t.getT1(), t.getT2()));

The containment: `boundedElastic`, sized properly

When the blocking call cannot be removed (a vendor SDK, a legacy SOAP client, JDBC against a database with no R2DBC driver):

private final Scheduler jdbcScheduler = Schedulers.newBoundedElastic(
    80,                 // sized from Little's Law, see below
    400,                // queue before RejectedExecutionException
    "jdbc",             // named, so thread dumps are readable
    60, true);

public Mono<Order> find(String id) {
    return Mono.fromCallable(() -> jdbcTemplate.queryForObject(...))
               .subscribeOn(jdbcScheduler);
}

Size it from the target, not from cores: threads = target_rps * service_time_seconds. For 400 rps at 180 ms, that is 72, so 80 with headroom. See publishOn vs subscribeOn for the full treatment of scheduler selection.

Use a dedicated scheduler per blocking integration, not the shared Schedulers.boundedElastic(). A shared pool means a slow dependency starves every other blocking integration in the process, which is the bulkhead argument from resilience patterns applied to threads.

Detection: BlockHound

<dependency>
    <groupId>io.projectreactor.tools</groupId>
    <artifactId>blockhound</artifactId>
    <scope>test</scope>
</dependency>

@BeforeAll
static void setUp() {
    BlockHound.install(builder -> builder
        // Known-safe exceptions, added deliberately and with a comment each.
        .allowBlockingCallsInside("org.springframework.boot.SpringApplication", "run")
    );
}

BlockHound instruments known-blocking JDK methods and throws BlockingOperationError when one is called from a thread marked non-blocking. Two things make it work in practice: run it in integration tests that exercise real handlers, because a unit test that calls a service method directly runs on the test thread and detects nothing; and treat every allowBlockingCallsInside as a debt entry with a comment, because the allow-list is where this quietly stops working.

A worked example: 12 percent CPU and a full-scale outage

A catalogue service. WebFlux, 6 pods, 4 cores each, so 24 event loops in total. It served product detail pages, calling three downstream services with WebClient (all non-blocking) and reading a feature-flag configuration.

The feature-flag lookup:

private FlagSet flags() {
    return flagClient.getFlags();       // synchronous HTTP, SDK provided by the vendor
}

@GetMapping("/product/{id}")
public Mono<ProductView> get(@PathVariable String id) {
    FlagSet f = flags();                            // BLOCKING, on the event loop
    return Mono.zip(pricing.get(id), inventory.get(id), reviews.get(id))
               .map(t -> render(t, f));
}

The vendor SDK cached flags for 60 seconds, so 99.98 percent of calls were an in-memory map lookup taking microseconds. In steady state the block was invisible, and the service ran at 3,400 rps across the fleet with a p99 of 61 ms for eight months.

Then the flag vendor had an incident. Their API started taking 8 to 30 seconds to respond instead of 40 milliseconds.

14:22  flag vendor p99: 40ms -> 12s
14:22  cache entries begin expiring; refresh calls now take 12s ON THE EVENT LOOP
14:23  each pod: 4 event loops, each blocked ~12s per cache refresh
14:23  throughput: 3,400 rps -> 61 rps    (a 98% drop)
14:24  health checks fail: /actuator/health is served BY AN EVENT LOOP
14:25  Kubernetes kills all 6 pods as unhealthy
14:25  replacements start with a COLD flag cache: every request refreshes
14:26  new pods blocked immediately, fail readiness, killed
14:26  crash loop. Service fully down.
14:51  vendor recovers. Service recovers on its own.

Twenty-nine minutes of total outage, caused by a dependency that was not on the critical path and that the design treated as a local cache lookup.

Three properties combined to make it total rather than partial:

One blocked call took out every connection on its loop. Four loops per pod, so the pod's entire concurrency was four.
The health check ran on an event loop too. A blocked loop cannot answer a liveness probe, so the orchestrator concluded the pod was dead and killed a pod that was merely stuck.
Restarting made it worse. The vendor's cache was in-process, so every restart guaranteed a cold cache and an immediate blocking call, which is why it became a crash loop instead of a degradation.

That third point is the one worth carrying: an in-process cache turns a restart from a remedy into an amplifier.

The fix:

// 1. Never on the event loop. Dedicated pool, and it is now visible in thread dumps.
private final Scheduler flagScheduler =
    Schedulers.newBoundedElastic(20, 50, "flags", 60, true);

// 2. Refresh in the background on a timer, never on the request path.
private final AtomicReference<FlagSet> cached = new AtomicReference<>(FlagSet.defaults());

@PostConstruct
void startRefresh() {
    Flux.interval(Duration.ZERO, Duration.ofSeconds(30))
        .flatMap(t -> Mono.fromCallable(flagClient::getFlags)
            .subscribeOn(flagScheduler)
            .timeout(Duration.ofSeconds(2))          // 3. hard timeout
            .doOnError(e -> flagErrors.increment())
            .onErrorResume(e -> Mono.empty()))       // 4. keep the last good value
        .subscribe(cached::set);
}

@GetMapping("/product/{id}")
public Mono<ProductView> get(@PathVariable String id) {
    FlagSet f = cached.get();                        // in-memory read, always
    return Mono.zip(pricing.get(id), inventory.get(id), reviews.get(id))
               .map(t -> render(t, f));
}

Plus a health endpoint on a separate port with its own event loop group, so a stalled request path cannot fail a liveness probe, and BlockHound in the integration test suite.

The redesign's key move is not the scheduler, it is removing the dependency from the request path entirely. A flag lookup is a read of local state; the refresh is a background concern. Once framed that way, the vendor being down means serving slightly stale flags, which is exactly the correct degradation.

Verified by injecting the failure in staging (a proxy adding 15 seconds to the flag API):

                              original      after
throughput during outage      61 rps        3,380 rps  (99.4% of normal)
p99 during outage             24,000ms      63ms
pods killed by liveness       6/6           0/6
flag staleness during outage  n/a           up to 29 min (last good value)
recovery                      crash loop    automatic, no restart

The service now degrades to "flags are stale" instead of "service is down," and the flag staleness is bounded only by the outage length, which the team accepted explicitly after checking that no flag governed anything safety-critical.

Production evidence

BlockHound exists as a dedicated project under the Reactor organisation, built by Sergei Egorov, for exactly this bug class. A purpose-built JVM agent for a single category of mistake is strong evidence about how frequently it occurs and how hard it is to find by reading code.

Reactor made .block() throw on non-blocking threads in 3.2 (IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-http-nio-N). Framework authors adding a runtime guard against a specific method call is the same signal.

Spring's WebFlux documentation states it directly: "if you use a blocking API in a WebFlux application, you must run it on a separate thread," and Spring Boot's spring.threads.virtual.enabled for MVC exists partly because the Spring team concluded that most applications with blocking data access are better served by MVC with virtual threads than by WebFlux.

Vert.x, which has the same architecture, ships a blocked-thread checker on by default: BlockedThreadChecker logs a warning with a stack trace when an event loop is occupied for more than 2 seconds. That an alternative framework independently concluded runtime detection was necessary reinforces the point.

Node.js has the identical failure mode with a single event loop, and the community guidance ("don't block the event loop") is the direct analogue. The --max-old-space-size and worker-threads advice maps closely onto boundedElastic. Anyone who has debugged a synchronous fs.readFileSync in an Express handler has debugged this bug in another language.

The debate

Should a service with blocking dependencies use WebFlux at all? My position is blunt: usually not. If most of your I/O is JDBC, WebFlux gives you the full complexity of reactive programming plus a thread per in-flight blocking call, which is the worst of both. Spring MVC with virtual threads (Java 21) gives you the scalability without the programming model change, and the debugging story is incomparably better: real stack traces, working debuggers, ThreadLocal that works. WebFlux earns its place when I/O is non-blocking end to end, when streaming large responses, or when you need composition operators for fan-out. See virtual threads vs reactive.

Is boundedElastic sufficient as a policy? It contains the damage and it does not remove it. The thread is still blocked; you have relocated the cost to a pool sized for it. The residual risks are real: pool exhaustion under an unexpectedly slow dependency, and the fact that someone will eventually add a blocking call without the scheduler. That second risk is why BlockHound in CI matters more than any code review policy.

Should the health endpoint share the event loop group? No, and this is under-appreciated. A liveness probe served by a blocked event loop fails, the orchestrator kills a pod that was stuck rather than dead, and if the cause is process-local (a cold cache, as in the example) the replacement fails identically. Run health checks on a separate port with a separate event loop group, which Spring Boot supports via management.server.port. It costs a port and it converts a self-amplifying crash loop into a stable degradation.

How much should you trust BlockHound? It catches JDK-level blocking (sockets, files, locks, Thread.sleep) reliably. It does not catch a busy-wait loop, a long-running computation, a JNI call, or a blocking operation implemented in native code by a vendor SDK. A 500-millisecond CPU-bound computation on an event loop is just as damaging as a 500-millisecond blocking I/O call and BlockHound will not say a word, so CPU-heavy work also belongs on parallel().

Follow-up Q&A

"How do you know an event loop is blocked, from metrics alone?"

The signature is flat throughput, linearly growing latency with concurrency, and low CPU. Then apply Little's Law: busy servers equals throughput times service time. If that comes out at roughly your event-loop count, you have your answer. Confirm with a thread dump under load, where you will see reactor-http-nio-N threads parked in a socket read or a JDBC call rather than in epollWait. The low-CPU part is what distinguishes this from an actual capacity limit and it is the piece people skip.

"Why is blocking one thread so much worse in WebFlux than in Spring MVC?"

Count and assignment. MVC has around 200 container threads and each handles one request, so blocking one costs 0.5 percent of capacity and affects one request. WebFlux has one event loop per core, and each loop owns many connections for their lifetime, so blocking one costs 12.5 percent on an 8-core box and stalls every connection assigned to it, not just the one being processed. The multiplier is both the smaller denominator and the connection-to-loop assignment.

"You cannot avoid the blocking call. What is the full mitigation?"

Five things, and I would do all five. A dedicated boundedElastic scheduler, named and sized from Little's Law, so it is a bulkhead rather than a shared resource. A hard timeout on the call, because an unbounded blocking call will exhaust any pool. A circuit breaker, so a sustained failure stops consuming threads at all. Health checks on a separate port and event loop group, so a stalled request path does not get the pod killed. And BlockHound in the integration test suite, so the next person to add a blocking call finds out in CI.

"Is a synchronous logging appender really a problem?"

Yes, and it is the one nobody looks for. A synchronous file or console appender performs a blocking write per statement. Normally sub-millisecond; under a slow disk, a full disk, or a container whose stdout is being consumed slowly by a log agent, it can take tens of milliseconds, on the event loop, on every request. The fix is an async appender with a bounded queue and an explicit discard policy, because an unbounded one converts a latency problem into a memory problem.

"Reactor throws on .block() now. Is the problem solved?"

It closes the most obvious form. It does not catch .toFuture().get(), .toIterable(), JDBC, Thread.sleep, a contended synchronized block, file I/O, or a vendor SDK that blocks internally. The guard is useful and narrow, and treating it as comprehensive is how the flag-SDK example happened: nothing in that code said block() anywhere.

"Explain the crash loop in that outage."

Three compounding properties. The blocking call stalled the event loop, so throughput collapsed. The health endpoint was served by the same event loops, so a stalled pod failed its liveness probe and was killed. And the flag cache was in-process, so every replacement pod started cold and made a blocking call immediately, failing readiness before it could serve anything. Restarting, normally a remedy, guaranteed the failure. Any in-process cache in front of a slow dependency has this property, and it is worth checking for whenever you see a restart make things worse.

Common misconceptions

"Blocking makes the request slow." It makes every request on that event loop slow. The blast radius is the loop's connection assignment, not the one request, which is why the throughput curve goes flat instead of degrading gracefully.

"More cores fixes it." More cores means more event loops, so the ceiling moves from 8 to 16 concurrent requests. If the service needs 500 concurrent requests, that is not a fix, it is a doubling of a number that is three orders of magnitude too small.

"It only matters under high load." It matters most under downstream slowness. The flag example ran fine for eight months at 3,400 rps because the blocking call took microseconds. A dependency getting slower, not your traffic getting higher, is the trigger.

"BlockHound in tests means we are covered." Only for JDK-level blocking, and only on paths the tests exercise on real event loop threads. A unit test calling a service method directly runs on the JUnit thread and proves nothing. Integration tests through the actual handler are what count.

"CPU-bound work on the event loop is fine, it is not blocking." A 500-millisecond computation occupies the loop for 500 milliseconds, exactly like a 500-millisecond socket read. BlockHound will not flag it. CPU-heavy work belongs on Schedulers.parallel().

Interview delivery note

Say this verbatim: "WebFlux has one event loop per core, and each loop owns many connections for their lifetime, so blocking one for 200 milliseconds stalls every connection assigned to it. The tell is flat throughput with linearly growing latency and low CPU, and Little's Law gives you the event-loop count in one line." Diagnosis plus mechanism plus the arithmetic, which is a complete answer in three clauses.

The senior-versus-staff separator is the health check on the event loop. A senior engineer explains why blocking is bad and reaches for boundedElastic. A staff engineer notices that a stalled event loop cannot answer a liveness probe, so the orchestrator kills a pod that was stuck rather than dead, and if the cause is an in-process cold cache the replacement fails identically and you have a crash loop. Recognising that a restart can amplify rather than remedy is systems thinking, not framework knowledge.

The second signal is naming the non-obvious blocking sources: a synchronous log appender, a contended synchronized block, a vendor SDK that blocks internally, and CPU-bound work that BlockHound will never flag. Anyone can say "don't use JDBC."

The Staff & Team Lead Interview Reference