Kafka Streams vs Flink vs Spark Structured Streaming

What it is

Three ways to run continuous computation over an unbounded stream, which look interchangeable in a feature matrix and are not interchangeable in a system.

Kafka Streams is a library. You add a JAR to a Java application, write a topology, and run the application however you already run applications: a container, a JVM, a Kubernetes Deployment. There is no cluster, no scheduler, no separate thing to operate. Parallelism comes from running more instances of your application, and they coordinate through Kafka's consumer group protocol.

Apache Flink is a distributed runtime. You submit a job to a cluster consisting of a JobManager and TaskManagers, and that cluster schedules, distributes, checkpoints and recovers your job. It is a piece of infrastructure you operate, separate from your application.

Spark Structured Streaming is a streaming API over a batch engine. It expresses a stream as an unbounded table and executes it as a series of small batch jobs (micro-batches) on Spark's existing engine, with an optional continuous processing mode that has never reached parity.

The distinction that decides most real choices is the first one: library versus cluster. Everything else (latency, state size, semantics) can usually be worked around; "do we now operate a Flink cluster" cannot.

What they are confused with: Kafka Streams is not a lightweight Flink. It is a different shape of thing. Flink can read from twenty sources and write to thirty sinks; Kafka Streams reads from Kafka and writes to Kafka, period. That constraint is not a limitation to be worked around, it is the assumption the entire design rests on, and it is why Kafka Streams needs no cluster: Kafka is the cluster, providing the partitioning, the coordination and the durable state.

The problem it solves

The shared problem: you have an unbounded stream and need to compute something over it continuously, with state that survives failures, at a parallelism greater than one machine.

Without a framework you would build: partition assignment and rebalancing, offset management, a state store with checkpointing, watermark tracking for event time, window triggering and eviction, exactly-once coordination with the sink, and rescaling that redistributes state. Every one of those is subtle and every one of those has been gotten wrong by teams who thought "it's just a consumer with a hash map."

The framework choice is really a choice about where the operational burden sits and what the ceiling is. All three will compute a windowed aggregate correctly. They differ in what happens when the state is 4 TB, when you need a second source, when the team on call is three people, and when the latency requirement is 200 milliseconds.

Mechanics

The comparison that matters

Kafka Streams: the topology is your application

StreamsBuilder builder = new StreamsBuilder();

KStream<String, Order> orders = builder.stream("orders",
    Consumed.with(Serdes.String(), orderSerde));

orders
    .filter((k, o) -> o.getAmount() > 0)
    .groupBy((k, o) -> o.getMerchantId(), Grouped.with(Serdes.String(), orderSerde))
    .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(1)))
    .aggregate(
        MerchantTotals::new,
        (key, order, agg) -> agg.add(order),
        Materialized.<String, MerchantTotals, WindowStore<Bytes, byte[]>>as("merchant-5m")
            .withValueSerde(totalsSerde))
    .toStream()
    .to("merchant-totals", Produced.with(windowedSerde, totalsSerde));

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

That is a complete deployable. merchant-5m becomes a local RocksDB store plus a compacted changelog topic <app-id>-merchant-5m-changelog (see log compaction), which is how state survives an instance dying: a new instance replays the changelog to rebuild the store.

Parallelism is bounded by partition count, exactly as for a consumer group, because Kafka Streams is a consumer group with a state store attached. A topology reading a 12-partition topic runs at most 12 tasks, and running 20 application instances leaves 8 idle. This is the same ceiling described in consumer rebalancing and it is the most common surprise.

Standby replicas (num.standby.replicas) keep a warm copy of each store on another instance, which turns a failover from "replay the whole changelog" (minutes for a large store) into "catch up the tail" (seconds). For any Kafka Streams application with meaningful state, set it to 1. The cost is a second copy of state and the changelog read traffic to maintain it.

Flink: a job submitted to a runtime

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(60_000);
env.setStateBackend(new EmbeddedRocksDBStateBackend(true));

env.fromSource(kafkaSource, WatermarkStrategy
        .<Order>forBoundedOutOfOrderness(Duration.ofSeconds(10))
        .withTimestampAssigner((o, ts) -> o.getEventTime()),
      "orders")
   .keyBy(Order::getMerchantId)
   .window(TumblingEventTimeWindows.of(Time.minutes(5)))
   .allowedLateness(Time.minutes(1))
   .aggregate(new MerchantAggregator())
   .uid("merchant-5m-aggregate")
   .sinkTo(jdbcSink);                      // NOT Kafka. This is the point.

env.execute("merchant-aggregates");

The last line before execute is the whole argument for Flink in one token: the sink is a database. Kafka Streams cannot do that as a first-class operation. You would write to a Kafka topic and run a Connect sink, which is a second system with its own delivery semantics and its own lag.

Flink's other structural advantage is that state is not bounded by partition count. Parallelism is bounded by maxParallelism (key groups), which you set at job creation and can make 4096. A Flink job reading a 12-partition topic can still run its aggregation at parallelism 200, because the shuffle after keyBy redistributes across key groups rather than partitions. For a job where the expensive work is downstream of the source, that is decisive.

See Flink state and checkpoints for the checkpointing machinery, which is the other half of what you get for operating a cluster.

Spark Structured Streaming: micro-batch as a feature

orders = (spark.readStream
    .format("kafka")
    .option("subscribe", "orders")
    .load()
    .select(from_json(col("value").cast("string"), order_schema).alias("o"))
    .select("o.*"))

totals = (orders
    .withWatermark("event_time", "1 minute")
    .groupBy(window(col("event_time"), "5 minutes"), col("merchant_id"))
    .agg(sum("amount").alias("total")))

(totals.writeStream
    .outputMode("update")
    .trigger(processingTime="30 seconds")
    .option("checkpointLocation", "s3://.../checkpoints/merchant-totals")
    .foreachBatch(upsert_to_warehouse)      # each micro-batch is a DataFrame
    .start())

foreachBatch is the tell for where Spark fits. Each micro-batch is a real DataFrame, so you can do a batch upsert into a warehouse, a Delta Lake merge, or a join against a large static table using Spark's full batch machinery. That is not an approximation of streaming, it is a genuinely different and often better model when the destination is analytical.

The micro-batch model's cost is latency. The trigger interval is a floor, and the practical floor is a few hundred milliseconds because each batch has scheduling overhead. Continuous processing mode exists, offers at-least-once only, and supports a narrow subset of operations, so treating Spark as sub-second is not something to plan around.

A worked example: three teams, three correct answers

A retail company, three streaming needs that arrived within a year of each other. They were tempted to standardise on one framework and did not, and the reasoning is the useful part.

Team A: fraud scoring on the payment path. Score every transaction against a per-card rolling feature set, block above a threshold. Requirement: p99 under 50 ms added latency; state is roughly 40 million cards at about 400 bytes each, about 16 GB. Source is Kafka, destination is Kafka.

Chose Kafka Streams. Reasoning: the latency requirement rules out Spark immediately. Between Kafka Streams and Flink, the deciding factor was that the fraud service already existed as a Spring Boot application with an on-call rotation, and Kafka Streams meant adding a topology to it rather than introducing a Flink cluster plus the platform work to run one. State fits comfortably in RocksDB across the 24 partitions. Measured p99 added latency: 11 ms.

Team B: real-time inventory across 3,000 stores. Join a Kafka stream of stock movements against a Postgres table of product metadata, aggregate into per-store availability, write to both Elasticsearch (for the storefront) and Postgres (for internal tools). Late events arrive up to 20 minutes late because store systems batch their uploads over unreliable links.

Chose Flink. Reasoning: two sinks, neither of them Kafka, plus a source that is not Kafka. Doing this in Kafka Streams means writing to Kafka topics and running two Connect sinks, which is three systems and three sets of lag to monitor instead of one. The 20-minute lateness needed real event-time handling with a long allowed lateness, and Flink's watermark and side-output machinery handles it directly. This job justified standing up the Flink cluster, and once it existed the marginal cost of the next Flink job was much lower, which is a real and often-decisive dynamic.

Team C: hourly revenue attribution into the warehouse. Join click events against order events over a 24-hour window, apply an attribution model, upsert into Delta Lake for the BI layer. Freshness requirement: within 15 minutes.

Chose Spark Structured Streaming. Reasoning: the destination is a lakehouse, the team already ran Spark for their batch attribution job, and the streaming version could share about 70 percent of its code with the batch one, including the attribution model itself. foreachBatch doing a Delta merge is one line and is the correct primitive; doing an equivalent upsert from Flink or Kafka Streams means building it. The 15-minute freshness requirement makes the micro-batch latency floor irrelevant.

What standardising would have cost, priced out at the time:

Everything on Flink:
  Team A's fraud job: works, but adds a cluster dependency to the payment
  path and a second on-call surface for a team of four. Estimated 2-3 weeks
  of platform work plus permanent operational load.
  Team C's job: works, but reimplements the Delta merge and loses code
  sharing with the batch job. Estimated 6+ weeks and a permanent fork.

Everything on Kafka Streams:
  Team B's job: needs 2 Connect sinks and a JDBC source connector. Three
  systems' worth of lag and failure modes instead of one.
  Team C's job: not viable. No warehouse sink, no batch code sharing.

Everything on Spark:
  Team A's job: not viable. 50 ms p99 against a micro-batch floor.

The point is not that standardisation is wrong. It is that the frameworks are differentiated on axes that map to real requirements, and standardising means one team gets the wrong tool. The right question is whether the operational saving of one framework exceeds the cost imposed on the team that gets the poor fit, and here it clearly did not.

The one thing they did standardise was the schema layer: one Schema Registry, one compatibility mode, one set of event definitions across all three (see Schema Registry compatibility). That is the standardisation that pays, because it is about the data contract rather than the compute engine, and it is portable across all three.

Production evidence

Kafka Streams powers ksqlDB, which is Confluent's SQL layer built entirely on the Streams library, and that is the strongest evidence for its production maturity: a commercial product with its own semantics implemented on top of it. Confluent's documentation for Streams is explicit that it is Kafka-to-Kafka by design.

Alibaba runs Flink at Singles' Day scale with multi-terabyte state, and their Blink fork was merged back into Flink upstream. Netflix uses Flink for its Keystone pipeline and real-time personalisation. Uber built AthenaX on Flink and runs thousands of streaming SQL jobs. The pattern in all three: a dedicated platform team operating Flink as shared infrastructure, which is precisely the cost that makes it wrong for a single application team with no platform organisation.

Databricks built Structured Streaming and Delta Live Tables around it, and the lakehouse pattern (stream into Delta, query with SQL) is where Spark Streaming's adoption is concentrated. Their published guidance recommends trigger intervals in the tens of seconds to minutes, which is a fair statement of where the model fits.

Pinterest, LinkedIn and Yelp have all published on Kafka Streams for application-level stream processing, and the common thread is the deployment argument: a Streams application deploys like any other service, which means no new deployment story, no new on-call surface and no new cluster.

Flink's Table API and SQL have matured substantially since 1.14, and Flink SQL is now a credible replacement for ksqlDB in many cases, which shifts the comparison: five years ago "I want SQL over streams" pointed at ksqlDB or Spark, and today Flink SQL is at least as strong.

The debate

The single most important variable is whether you already run a cluster. If you have a Flink platform with a team behind it, use Flink for nearly everything, because the marginal job is cheap and the ceiling is highest. If you do not, and the job is Kafka-to-Kafka, Kafka Streams is almost always the right answer, and the argument that "Flink is more powerful" is true and irrelevant, because you are comparing a library you add to an existing service against infrastructure you must build, staff and carry.

Kafka Streams versus Flink, when both are available. Three things push toward Flink: a source or sink that is not Kafka, state or parallelism beyond what partition count allows, and complex event-time semantics with long lateness windows or side outputs. Three push toward Kafka Streams: the application already exists and this is a feature of it, the team is small, and the topology is genuinely Kafka-to-Kafka. My default: Kafka Streams for application-embedded stream processing, Flink for anything that is a data pipeline in its own right. The phrase "is this a feature of a service or is it a pipeline" separates them well in practice.

Spark's honest position. Spark Structured Streaming is the right choice when the destination is analytical, when you want one API across batch and streaming with real code sharing, and when latency in the tens of seconds is fine. It is the wrong choice for anything on a request path. The common failure is a team that already runs Spark choosing it for a low-latency use case because it is what they know, then spending months fighting the micro-batch floor. Familiarity is a legitimate input to this decision and it does not override a hard latency requirement.

Where I would push back on all three: if the computation is stateless, or the state is a small per-key value with no windowing, you may not need any of them. A plain consumer with an idempotent write is less code, has no rebalancing subtleties beyond the ones you already have, and does not add a framework's failure modes. Reaching for a stream processor for filter and map is over-engineering, and the tell is a topology with no aggregation, no join and no window in it.

Follow-up Q&A

"Kafka Streams or Flink, and what decides it?"

Whether the job touches anything other than Kafka, and whether you already operate a Flink cluster. If the topology is Kafka in, Kafka out, and you do not have a Flink platform, Kafka Streams: it deploys as part of an existing application, there is no cluster, and the state and coordination story is Kafka's, which you already run. The moment you need a non-Kafka sink, or parallelism beyond the source's partition count, or state larger than the partitions can carry, Flink. And if a platform team already runs Flink, the calculus flips because the marginal cost of another job is small.

"Why can't Spark hit low latency?"

The execution model is micro-batch: each trigger schedules a batch job, and job scheduling has fixed overhead. Continuous processing mode was added to address this and offers at-least-once only with a limited operator set, so it is not a general answer. The practical floor is a few hundred milliseconds and the realistic operating range is seconds. For an analytics pipeline that is fine; for anything on a request path it is disqualifying.

"How does state work in each?"

Kafka Streams: local RocksDB plus a compacted changelog topic per store, so recovery means replaying the changelog, and standby replicas keep a warm copy to make failover fast. Flink: heap or RocksDB locally, with periodic checkpoints to blob storage and incremental upload when on RocksDB, so recovery means downloading the state. Spark: a state store checkpointed to HDFS or S3 per micro-batch, with RocksDB available as a backend since Spark 3.2. The architectural difference: Kafka Streams' durable state lives in Kafka and Flink's lives in object storage, which means Kafka Streams adds load to your Kafka cluster proportional to state churn, and that is a real capacity consideration people miss.

"You have a Kafka Streams app and need to write to Postgres. Now what?"

Three options and I would rank them. Write to a Kafka topic and use a Connect JDBC sink: keeps the Streams model intact, adds a second system to operate and monitor, and is what I would do if this is the only non-Kafka destination. Do the write inside a process() node: possible, and it breaks the exactly-once story since the external write is not in the Kafka transaction, so it requires idempotent upserts and careful failure handling. Move to Flink: correct if this is the first of several non-Kafka sinks, because that is the pattern Flink is built for and the second and third sink cost nothing extra. The decision hinges on whether this is an exception or the start of a trend.

"Which has the best exactly-once story?"

They differ in scope rather than strength. Kafka Streams gives exactly-once Kafka-to-Kafka using Kafka transactions, which is genuinely end to end within that boundary and is the cleanest of the three because both ends are the same system. Flink gives exactly-once state via checkpointing plus end-to-end with sinks implementing two-phase commit, which covers far more destinations but requires the sink to cooperate and delays downstream visibility by up to a checkpoint interval. Spark gives exactly-once with idempotent sinks and a write-ahead log of offsets, which in practice means the sink must support idempotent upsert, and Delta Lake does.

"When would you use none of them?"

Stateless processing, or per-key state with no windowing or joining. A plain consumer with an idempotent write handles filter, map, enrich-by-lookup and write-to-a-database perfectly well, with less code and no framework failure modes. I would also skip them when the volume is genuinely low: a stream processor for 200 events per second is a lot of machinery for something a single-threaded consumer handles with room to spare.

Common misconceptions

"Kafka Streams is Flink for small jobs." Different shape, not different size. Kafka Streams applications routinely handle very high volume; what they cannot do is read from or write to anything but Kafka, and that is a design assumption rather than a capacity limit.

"Flink is always better, just harder." Flink is better on the axes it optimises for: heterogeneous sources and sinks, state and parallelism decoupled from partition count, sophisticated event-time semantics. It is worse on the axis that matters most to a small team, which is that it is a cluster you operate. That cost is not "harder," it is a permanent staffing commitment.

"Spark Structured Streaming is not real streaming." It is real streaming with a micro-batch execution model and correct event-time and watermark semantics. The latency floor is real; the correctness is not in question. Dismissing it costs you the one framework with genuine batch and stream code sharing.

"Kafka Streams has no cluster, so it has no coordination problems." It has exactly the coordination problems of a consumer group, including rebalancing, and adds state migration on top. A rebalance in a Kafka Streams app moves state stores between instances, which is slower than moving partitions alone. Standby replicas exist precisely because of this.

"Pick one and standardise." Reasonable as a default and wrong when a team's requirement sits outside the chosen tool's range. Standardise the schema layer and the data contracts, which are portable; be pragmatic about the engine, which is not.

Interview delivery note

Say this verbatim: "The first question is not latency or state size, it is whether this job touches anything other than Kafka, and whether we already operate a cluster. Kafka Streams is a library you add to an existing service; Flink is infrastructure you staff. That difference decides more real cases than any feature." It reframes the question from a feature comparison to an operational one, which is the staff-level framing.

The senior-versus-staff separator is naming the partition-count ceiling on Kafka Streams. A senior engineer compares latency, state backends and exactly-once. A staff engineer points out that Kafka Streams' parallelism is bounded by the source topic's partition count because it is a consumer group, so a job whose expensive work is downstream of the source cannot scale past it, while Flink's shuffle after keyBy decouples the two. That is a structural constraint that shows up as a wall you cannot configure your way past.

The second signal is refusing to standardise reflexively while identifying what should be standardised: the schema layer and the data contracts, because those are portable across engines and are where the real coupling between teams lives.

Further reading

• Kafka Streams documentation, "Streams Architecture," for the tasks-equal-partitions model and standby replicas.
• Flink documentation, "Flink Architecture" and the DataStream API guide, for the JobManager/TaskManager model and the keyBy shuffle.
• Spark documentation, "Structured Streaming Programming Guide," particularly the sections on triggers, foreachBatch and continuous processing limitations.
• Tyler Akidau et al., "The Dataflow Model" (VLDB 2015), for the event-time and windowing semantics all three implement in some form.