Drills 10 to 15: streaming, reactive and APIs

Six questions, ninety seconds each, out loud. The answers below are written in spoken form: this is roughly what you would say in a room, not what you would write in a design doc. Read the question, answer it aloud, then compare.

The written form and the spoken form are different artifacts. A written answer can be dense and can assume the reader will re-read a sentence. A spoken answer needs a structure the listener can follow in real time, which means leading with the conclusion, keeping to three points, and stopping.

Drill 10. Explain exactly-once in Kafka, and what it does not cover

Exactly-once in Kafka is really three things stacked. The idempotent producer gives each producer a PID and a per-partition sequence number, so the broker can recognise a retry and discard it. Transactions make a set of writes plus the consumer offset commit atomic, so a consume-transform-produce loop either does all of it or none of it. And read_committed consumers only read up to the last stable offset, so they never see records from a transaction that hasn't committed.

What it doesn't cover is anything outside Kafka. The moment I write to a database, call an HTTP service, or send an email, the transaction can't span it, so I need an idempotency key at that boundary anyway. And since I need that anyway, I design for it first and treat the Kafka transaction as an optimisation that removes duplicate work inside the pipeline.

The operational thing I'd flag is that an open transaction blocks read_committed consumers on that partition. So a stuck processor shows up as climbing consumer lag with a completely flat error rate, which is a confusing incident until you've seen it once.

Depth signal: the last-stable-offset stall. Everyone can describe the producer and the transaction; very few volunteer the lag-with-no-errors failure mode.

Full treatment: Kafka exactly-once, end to end.

Drill 11. flatMap vs concatMap: behaviour, and when each is wrong

flatMap subscribes to the inner publishers eagerly and emits results in completion order, so it's concurrent and unordered. concatMap subscribes to one inner at a time, so it's sequential and preserves source order. There's a third, flatMapSequential, which subscribes eagerly like flatMap but buffers so the output stays in source order.

concatMap is wrong when you need throughput: a thousand calls at fifty milliseconds each is fifty seconds, sequentially. flatMap is wrong when order matters. If the inners have side effects that must happen in sequence, like applying state mutations, concatMap is the only correct choice, and note that flatMapSequential doesn't help there because it reorders the output while still executing concurrently.

The thing I'd actually flag in review is that flatMap's default concurrency is 256. That's a shared buffer-size constant, not a considered limit, so against a downstream sized for twenty concurrent calls it's a self-inflicted load test. I always pass an explicit concurrency, and I derive it from Little's Law: target throughput times call latency gives you the number in flight, and that should match the connection pool.

Depth signal: deriving the concurrency argument from Little's Law and the downstream pool, rather than just naming the operators.

Full treatment: flatMap vs concatMap.

Drill 12. Do virtual threads make WebFlux obsolete?

They remove the main reason most teams adopted it, which was scaling I/O-bound concurrency without a huge thread pool. Virtual threads give you that with blocking-style code, so you keep readable stack traces, working debuggers, thread-locals and correct profiler attribution.

What they don't replace is what reactive uniquely provides: demand signalling across a network boundary. request(n) lets a slow consumer tell a remote producer to slow down, and there's no equivalent with blocking code. So for real streaming with backpressure, reactive is still the right model.

My position for a new service on JDK 21 or later is virtual threads with structured concurrency, and reactive only for genuine streaming. And I wouldn't rewrite a working reactive service for this. The benefit is developer experience, which is worth something but not a quarter.

The thing I'd watch in a migration is that the thread pool was also acting as an accidental rate limiter. Once threads are free, the connection pool becomes the ceiling, so I'd put an explicit semaphore in front of each downstream.

Depth signal: the thread pool as an accidental rate limiter, and mentioning that the synchronized pinning advice changed in JDK 24.

Full treatment: Virtual threads vs reactive.

Drill 13. How do you fix N+1 in GraphQL, and why doesn't caching solve it?

DataLoader. It's created per request, collects the keys requested during an execution tick, calls one batch function with all of them, and hands results back to the individual promises. Fifty resolvers each asking for a customer become one query.

Caching doesn't solve it for three reasons. The problem is fifty lookups for fifty different keys, which is a batching problem, and a cache only helps with repeats of the same key. GraphQL is a POST with the query in the body, so HTTP caching doesn't apply without persisted queries. And client-shaped queries are unique by construction, so a response cache is cold for anything new.

Two implementation details I'd raise. The batch function has to return results in key order with nulls for misses, because the database returns rows in its own order and omits missing ones, and positional mismatch silently attaches the wrong record to the wrong parent. And the loader must never be a module-level singleton, because its memoisation would persist across requests and across users, which turns a performance optimisation into a cross-user data leak.

Depth signal: the per-request scoping as a security property, not a performance one.

Full treatment: GraphQL N+1 and DataLoader.

Drill 14. Why does an L4 load balancer break gRPC?

Because gRPC multiplexes many RPCs over one long-lived HTTP/2 connection, and an L4 balancer makes its decision once, when the connection is established. So every RPC from that client goes to whichever backend won the lottery, forever, and newly scaled pods receive nothing because no new connections are being made.

Four fixes. Client-side round-robin over a headless service, so the client resolves all the backend addresses and balances per RPC. Note the default policy is pick_first, so you have to set round_robin explicitly. An L7 proxy or service mesh that balances per stream. xDS lookaside balancing at scale. Or, as a cheap mitigation for clients you don't control, MaxConnectionAge on the server so connections recycle every ten minutes and rebalance.

The part that makes this confusing in production is the autoscaler. Unbalanced traffic means average CPU across the deployment looks low, so the HPA scales down, which concentrates load further. It looks like a capacity problem and it's a routing problem.

And it's not really a gRPC issue. It's HTTP/2 connection reuse, so REST over HTTP/2 has it too.

Depth signal: the HPA feedback loop, and correcting the framing to HTTP/2 rather than gRPC.

Full treatment: Why an L4 load balancer breaks gRPC.

They tell the system when it's safe to close a window. In stream processing you care about event time (when something actually happened) rather than processing time, because events arrive late and out of order from mobile clients, retries and partitioned sources. So if I'm computing a one-minute window, I need to know when I've probably seen everything for that minute.

A watermark is an assertion flowing through the stream that says "I don't expect any more events with a timestamp earlier than T". When the watermark passes the end of a window, the window fires. It's a heuristic, not a guarantee: you're trading completeness against latency, and the watermark strategy is where you set that dial.

Then there are two escape hatches for what arrives after. Allowed lateness keeps the window state around a bit longer and re-fires on late events. Side outputs route anything later than that to a separate stream so you can log it, correct downstream, or at least count it, because the count of dropped late events is a metric you want, otherwise data goes missing silently.

The operational failure I'd name is a stalled watermark from an idle partition. If one Kafka partition stops producing, its watermark stops advancing, and because the operator takes the minimum across inputs, the whole job's watermark freezes and no windows fire at all. The fix is an idleness timeout on the source.

Depth signal: the idle-partition watermark stall, and framing watermarks as a completeness-versus-latency dial rather than a correctness mechanism.

How to practise these

Not by reading them. The failure mode of a drill bank is that recognition feels like recall.

The loop: read only the question. Set a timer for ninety seconds. Answer out loud, standing up, as if to a person. Then read the written answer and note the one thing you missed. Move on. Do not re-answer immediately, because you will just recite what you have read.

Record yourself once a week and watch it at 1.5x. It is uncomfortable and it is the fastest way to find your filler words, your rambling, and the exact moment you lost the thread. Fix one thing per session.

The structural pattern in every answer above: lead with the direct answer in one sentence, give two or three supporting points, then one thing that shows depth (a failure mode, an operational consequence, a correction of the question's premise), then stop. The stopping is the hardest part and the most valuable. An answer that lands in seventy seconds and ends cleanly reads as more confident than the same content delivered in three minutes.

When you genuinely do not know, use the sequence: state the boundary plainly, reason from adjacent knowledge, name how you would find out. "I haven't run Flink in production; here's what I'd expect to transfer from Kafka Streams, and here's what I'd test first." Never bluff, and never stop at "I don't know" without the reasoning attempt.