Streaming, reactive and APIs

Three separate bodies of knowledge that share one interview question: how do two systems talk to each other without lying about what happened. Kafka and Flink answer it with a replayable log and watermarks. Reactive streams answer it with demand signalling. gRPC and GraphQL answer it with a schema and a deadline.

The source document marked this whole area as outline-only, and it is the area where a candidate with a Java background can most cheaply build a large advantage, because the questions are specific and the wrong answers are well-known.

What this chapter covers

• [done] ISR, min.insync.replicas, acks and unclean leader election
• [done] Kafka exactly-once, end to end
• [done] Log compaction vs retention, and compacted topics as state
• [done] Consumer group rebalancing: eager, cooperative sticky, static membership
• [done] Consumer lag as the health metric, and lag-based autoscaling
• [done] Schema Registry compatibility modes as a design decision
• [done] flatMap vs concatMap in Project Reactor
• [done] Why an L4 load balancer breaks gRPC
• [done] Deadline propagation and cancellation across a call tree
• [done] Apollo Federation v2 and Netflix DGS
• [done] GraphQL caching, persisted queries, depth and complexity limits
• [done] Protobuf wire format and compatibility rules
• [todo] ISR, min.insync.replicas, acks=all and unclean leader election
• [todo] Log compaction vs retention; compacted topics as state
• [todo] Consumer group rebalancing: eager, cooperative sticky, static membership
• [todo] Consumer lag as the health metric; KEDA lag-based autoscaling
• [todo] Schema Registry compatibility modes as a design decision
• [done] Watermarks in Flink
• [done] Flink state backends, checkpointing and savepoints
• [done] Kafka Streams vs Flink vs Spark Structured Streaming
• [done] CDC with Debezium, and the dual-write problem
• [done] Dead letter queues, poison pills and retry tiers
• [todo] Flink state backends, checkpointing and savepoints
• [todo] Kafka Streams vs Flink vs Spark Structured Streaming
• [todo] CDC with Debezium and the dual-write problem
• [todo] Dead letter queues, poison pills, retry topics with backoff tiers
• [todo] Reactive Streams: Publisher, Subscriber, request(n), backpressure strategies
• [todo] publishOn vs subscribeOn, and the schedulers
• [todo] Never block the event loop: the number-one WebFlux production bug
• [done] Reactive Streams: Publisher, Subscriber, request(n), backpressure
• [done] publishOn vs subscribeOn, and the schedulers
• [done] Never block the event loop
• [done] Virtual threads vs reactive
• [done] GraphQL N+1 and DataLoader
• [todo] Apollo Federation v2 and Netflix DGS
• [todo] GraphQL caching, persisted queries, depth and complexity limits
• [todo] Protobuf wire format and compatibility rules
• [done] Sagas vs two-phase commit
• [done] The API decision matrix
• [todo] Sagas vs two-phase commit, expanded

Source: §17, §46 of the source.