Consumer group rebalancing: eager, cooperative sticky, static membership

What it is

A Kafka consumer group is a set of consumer instances that share the work of consuming a topic's partitions, with the rule that each partition is assigned to exactly one consumer in the group at a time. That exclusivity is what gives you ordered processing per partition and prevents duplicate work.

Rebalancing is the protocol that recomputes the assignment when group membership or topic metadata changes: an instance joins, an instance leaves or dies, or the partition count grows.

The thing to internalise, because everything else follows from it: under the original (eager) protocol, a rebalance stops the entire group. Every consumer revokes every partition it holds, all of them rejoin, an assignment is computed, and only then does anyone resume. This is called the stop-the-world rebalance, and the name is accurate. A group of 60 consumers where one instance restarts pauses all 60.

What it is confused with: partition reassignment, which is a broker-side operation moving partition replicas between brokers, and has nothing to do with consumer groups. If someone says "rebalancing" in a Kafka context, ask which one they mean, because one is a consumer coordination protocol and the other is a data movement operation.

The problem it solves

You need N consumer instances to divide M partitions among themselves, with no central scheduler, tolerating instance failure, in a way all instances agree on. The agreement part is the hard part: if two consumers both believe they own partition 7, you get duplicate processing and, if they both commit offsets, offset corruption.

Kafka's answer is a group coordinator (a broker chosen by hashing the group ID) that manages membership, plus a group leader (one of the consumers, chosen arbitrarily by the coordinator) that computes the actual assignment. Putting the assignment logic in a client rather than the broker was a deliberate choice: it lets applications supply custom assignors without a broker upgrade, which is how Kafka Streams implements its own topology-aware assignment.

The cost of that design is the protocol itself, and the protocol's cost is rebalance downtime. On a group processing 200,000 messages per second, a 45-second rebalance is 9 million messages of accumulated lag, and if rebalances happen every few minutes because of a misconfiguration, the group never catches up. Rebalance storms are one of the most common Kafka production incidents, and they usually have a mundane cause: a slow processing loop tripping max.poll.interval.ms.

Mechanics

The two-phase protocol

Every rebalance is a JoinGroup phase followed by a SyncGroup phase.

1. Trigger: member joins, leaves, dies (session timeout), or metadata changes.
   Coordinator increments the generation ID.

2. JoinGroup: every member sends JoinGroup with its subscription and its
   supported assignors. The coordinator holds all requests until every known
   member has joined OR rebalance.timeout.ms expires.
   The coordinator picks one member as LEADER and returns the full member
   list to it. Everyone else gets an empty response.

3. Assignment: the leader runs the assignor locally, producing a
   member -> partitions map.

4. SyncGroup: the leader sends the assignment to the coordinator. Everyone
   else sends an empty SyncGroup and waits. The coordinator distributes each
   member its slice.

5. Members begin fetching. Generation ID is attached to every offset commit,
   so a commit from a stale generation is rejected with
   ILLEGAL_GENERATION.

Step 2 is where the latency lives: the coordinator waits for the slowest member, up to rebalance.timeout.ms (which equals max.poll.interval.ms, default 5 minutes). A member that is busy in a long poll() processing loop cannot send JoinGroup until it finishes, so one slow consumer sets the rebalance duration for the whole group.

Eager: revoke everything

RangeAssignor (the historical default) and RoundRobinAssignor are eager. Before sending JoinGroup, each consumer calls onPartitionsRevoked for all its partitions and stops fetching. The group is idle from that moment until SyncGroup completes.

RangeAssignor has an additional defect worth knowing: it assigns per topic, laying out each topic's partitions in order and dividing by consumer count. With multiple topics, the same early consumers get the remainder from every topic, so consumer 0 is systematically overloaded. RoundRobinAssignor distributes across all topic-partitions together and does not have this skew.

Neither is sticky, so a rebalance can move every partition to a different consumer even when only one member changed, which for a stateful consumer means discarding and rebuilding local state.

StickyAssignor and CooperativeStickyAssignor

Sticky means: produce a balanced assignment while preserving as many existing assignments as possible. With 10 consumers and 100 partitions, adding an 11th should move about 9 partitions, not 100.

Cooperative (KIP-429, Kafka 2.4) changes the protocol itself so partitions that are not moving are never revoked. It uses two rebalances instead of one:

Rebalance 1: everyone joins reporting their CURRENT assignment.
             The leader computes the target assignment.
             Members whose partitions are being taken away revoke ONLY those.
             Members keep and KEEP PROCESSING everything else.

Rebalance 2: triggered immediately. The now-free partitions are assigned to
             their new owners.

Two rounds sounds worse and is dramatically better in practice, because during both rounds every consumer keeps processing every partition it is not losing. On a group where one instance out of 20 restarts, roughly 95 percent of partitions never pause at all. CooperativeStickyAssignor is the default in Kafka 3.0+ for new consumers and there is essentially no reason to use eager assignors for new applications.

Upgrading to cooperative requires two rolling restarts, and getting this wrong breaks the group:

// Rolling restart 1: both protocols supported. The group stays eager
// because the leader picks the assignor common to all members.
props.put(PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
          "org.apache.kafka.clients.consumer.CooperativeStickyAssignor," +
          "org.apache.kafka.clients.consumer.RangeAssignor");

// Rolling restart 2: only after EVERY member is on restart-1 config.
props.put(PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
          "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");

Skipping to the second config directly, while old members are still in the group, means members disagree about whether partitions were revoked, and you get duplicate ownership.

Static membership: not rebalancing at all

Static membership (KIP-345, Kafka 2.3) attacks a different problem: a rolling restart or a pod reschedule should not trigger a rebalance at all, because the instance is coming back with the same identity in a few seconds.

Give each consumer a stable group.instance.id. The coordinator then remembers the assignment for that ID. When the member disappears, the coordinator does not trigger a rebalance; it waits for session.timeout.ms and, if the same group.instance.id rejoins, hands back the identical assignment with no rebalance.

// Stable per-instance, survives restarts. In Kubernetes, the StatefulSet
// ordinal is the natural source.
props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG,
          System.getenv("POD_NAME"));          // e.g. "orders-consumer-3"
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "120000");  // 2 min

The trade is explicit and important: a genuinely dead static member's partitions stay unassigned for the full session.timeout.ms. You are trading recovery time for restart stability. With session.timeout.ms=120000, a hard node failure means two minutes of no progress on that member's partitions. That is the right trade for frequent deploys and stateful consumers, and the wrong one for a group where fast failover matters more than deploy smoothness.

Static membership and cooperative rebalancing compose: use both.

The timeouts, and which one you are actually hitting

Three separate timeouts, routinely confused:

Config	Default	Measures	Sent by
`heartbeat.interval.ms`	3s	How often the background thread pings the coordinator	Background heartbeat thread
`session.timeout.ms`	45s	How long without a heartbeat before the member is declared dead	Coordinator's clock
`max.poll.interval.ms`	300s	Max time between successive `poll()` calls	Consumer's own clock

Since KIP-62 (Kafka 0.10.1), heartbeats run on a background thread, so a consumer stuck processing a batch keeps heartbeating and stays "alive" from the coordinator's view. What catches it is max.poll.interval.ms: if the application does not call poll() again within that window, the consumer itself proactively leaves the group.

This means the two failures look completely different in logs:

Session timeout exceeded: the process is gone, GC-paused for tens of seconds, or network-partitioned. Coordinator-initiated.
max.poll.interval exceeded: the process is alive and busy. You will see Member ... sending LeaveGroup request and then a CommitFailedException on the next commit. Self-initiated.

The second is the common production incident, and it produces a distinctive death spiral: processing is slow, the consumer is evicted, the group rebalances, the partitions move to another consumer that is also slow, it gets evicted, and the group rebalances forever while lag grows without bound.

A worked example: a rebalance storm and its arithmetic

An order-processing group: 24 consumer pods, 96 partitions, each message requiring a synchronous call to a pricing service. Configuration was defaults: max.poll.records=500, max.poll.interval.ms=300000.

Normal pricing latency was 8 ms, so a batch of 500 took about 4 seconds, comfortably inside the 5-minute limit. Then the pricing service degraded to a p99 of 900 ms during an incident of its own.

The arithmetic that broke it. A batch of 500 records where a meaningful fraction hit the slow path:

500 records x ~600ms average  =  300 seconds  =  max.poll.interval.ms exactly

Consumers began dropping out of the group. Each departure triggered an eager rebalance (RangeAssignor), stopping all 24 consumers. Measured rebalance duration was 38 seconds, dominated by waiting for members still stuck in their processing loops.

14:02  pricing p99 -> 900ms
14:06  consumer-7 exceeds max.poll.interval, leaves group
14:06  REBALANCE (38s). All 24 consumers idle.
14:07  processing resumes; the backlog is now larger, so batches are full 500
14:09  consumer-3 and consumer-11 exceed, leave
14:09  REBALANCE (41s)
...
14:40  group in rebalance ~60% of wall-clock time. Lag: 4.1M messages.

The group was spending more time rebalancing than consuming, and lag grew even though the pricing service had partially recovered by 14:20.

The immediate fix, applied at 14:44:

max.poll.records=50               # 500 -> 50: worst case 30s per batch
max.poll.interval.ms=120000       # 300s -> 120s: fail fast, and 4x headroom

Reducing max.poll.records is the primary lever and it is counterintuitive: throughput per poll goes down, total throughput goes up, because the group stops rebalancing. Lag drained in 26 minutes.

The durable fixes, applied over the following week:

partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
group.instance.id=${POD_NAME}     # static membership
session.timeout.ms=60000

plus a bulkhead and timeout on the pricing call (see resilience patterns) so a slow dependency degrades one call rather than the whole batch.

Measured after:

                              before     after
rebalance duration (p50)       38s        1.2s      (cooperative: most partitions never pause)
rebalances per deploy          24         0         (static membership)
rebalances per week            190        3
worst-case lag during a
  dependency incident          4.1M       210k
throughput (steady state)      41k/s      44k/s

Steady-state throughput went up slightly despite max.poll.records dropping 10x, which is the clearest evidence that the batch size was never the bottleneck.

The number worth carrying: rebalances per deploy went from 24 to 0. A 24-pod rolling restart previously caused 24 stop-the-world rebalances, one per pod, each pausing the entire group. That was a self-inflicted cost paid on every single deploy, and static membership removes it entirely.

Production evidence

KIP-429 (incremental cooperative rebalancing) was driven by Confluent and motivated explicitly by Kafka Connect and Kafka Streams, where stop-the-world rebalancing was the dominant availability problem: a Connect cluster rebalancing all connectors because one worker restarted. Connect got cooperative rebalancing first (KIP-415), then consumers.

KIP-345 (static membership) came from Pinterest, whose motivation in the KIP is exactly the deploy problem: rolling restarts of large consumer fleets in Kubernetes causing repeated full rebalances. The design note that the coordinator simply does not react to a static member's departure until session timeout is the whole idea.

Kafka Streams supplies its own assignor (StreamsPartitionAssignor) rather than using a stock one, because it must co-locate partitions of joined topics on the same instance and place standby replicas. This is why the pluggable client-side assignor design exists, and it is the strongest argument for it.

KIP-848, the next-generation consumer rebalance protocol, moves assignment computation from the client leader to the broker-side coordinator and makes reconciliation fully incremental, removing the JoinGroup barrier that makes one slow member gate the whole group. It became generally available in Kafka 4.0. The motivation stated in the KIP is that the client-side leader design makes rebalance duration a function of the slowest client, which is exactly the failure in the worked example above.

Uber has published on running large consumer groups and describes both the max.poll.interval.ms failure mode and the practice of keeping consumer groups small (splitting a large group into several) specifically to limit rebalance blast radius.

The debate

Cooperative versus eager. There is no real debate for new applications: cooperative is strictly better and is the default from Kafka 3.0. The only argument for eager is that a migration requires two coordinated rolling restarts, and on a group nobody wants to touch, that is a real cost. My position: do the migration, and do it before you need it, because you will want it during an incident and that is the worst time to be executing a two-phase config rollout.

Static membership: is longer failure detection acceptable? This is a genuine trade. With session.timeout.ms=120000, a hard failure leaves partitions unassigned for up to two minutes. Against that: deploys cause zero rebalances, and for most teams deploys are several orders of magnitude more frequent than hard node failures. The variables that decide it: how often you deploy, whether your consumers hold expensive local state, and what two minutes of lag on a subset of partitions costs you. My default is static membership with session.timeout.ms around 60 seconds, which is long enough to cover a pod reschedule and short enough that a genuine failure is not catastrophic. Set it much higher only for stateful consumers where state restore is expensive.

Should you have fewer, larger consumer groups or more, smaller ones? Smaller. Rebalance cost scales with group size and blast radius is bounded by the group, so one group of 200 consumers is operationally worse than four groups of 50 doing different jobs. The counter-argument is that splitting means each group re-reads the topic, multiplying broker fetch load, which is a real cost on a high-volume topic. The balance point is roughly: split by processing concern (things that fail independently and deploy independently), not to hit a number.

How many partitions? Partition count sets the maximum consumer parallelism, because a partition has exactly one consumer in a group. Extra consumers beyond the partition count sit completely idle, which surprises teams who scale up during an incident and see no improvement. But partitions are not free: more partitions means more open file handles, more replication fetch traffic, longer leader-election time on broker failure, and longer rebalances. A common default is to size for 2x peak consumer count so you have headroom to scale out without repartitioning, and repartitioning a topic that requires key-ordering is not something you want to discover you need. See also log compaction, where partition count is frozen permanently.

Follow-up Q&A

"Your consumer group rebalances constantly. Walk me through the diagnosis."

First, distinguish the two causes from the logs. max.poll.interval.ms exceeded produces a self-initiated LeaveGroup and a subsequent CommitFailedException; session timeout produces coordinator-initiated eviction with no LeaveGroup. The first means processing is too slow, and the fix is max.poll.records down (the fast lever), then finding what is slow, usually a synchronous dependency without a timeout. The second means the process is unhealthy: check GC pause logs, CPU throttling in the container (a CPU limit causing throttling looks exactly like a network problem), and network. Third possibility, and I would check it early because it is embarrassing to miss: consumers with different subscriptions or different assignors in the same group, which causes a rebalance on every member join, forever. That happens when a deploy is half-rolled with a config change.

"Why does one slow consumer stall the whole group?"

Because the coordinator holds all JoinGroup requests until every known member joins or rebalance.timeout.ms expires. A member busy inside poll() cannot send JoinGroup until it returns, so the barrier is set by the slowest member. This is inherent in the client-leader design and is precisely what KIP-848 fixes by moving assignment to the broker and making reconciliation incremental with no barrier.

"Cooperative rebalancing does two rebalances. Isn't that worse?"

Two rebalances, but partitions that are not moving are never revoked, so consumers keep processing throughout both. The right comparison is not "one rebalance versus two," it is "100 percent of partitions paused for 38 seconds versus 5 percent of partitions paused for a couple of seconds." On a 20-node group where one node restarts, cooperative moves roughly 1/20 of partitions and the other 19/20 never stop.

"What exactly does static membership avoid?"

The rebalance on departure. Normally a member leaving triggers an immediate rebalance. A static member's departure is ignored until session.timeout.ms, and if the same group.instance.id returns first, it gets its previous assignment back with no rebalance at all. So a rolling restart where each pod is back within the session timeout causes zero rebalances. Note it does not avoid the rebalance when the group genuinely changes size, and it does not help if your orchestrator gives pods new identities on restart, which is why a StatefulSet ordinal or an explicit env var is required rather than a hostname or a UUID.

"You have 96 partitions and 120 consumers. What happens?"

Twenty-four consumers get nothing and idle. Partition count is the hard ceiling on group parallelism. If you need more parallelism the options are: add partitions (one-way for ordering-sensitive topics, and impossible for compacted ones), or decouple fetching from processing by handing records to a worker pool inside each consumer, which recovers parallelism but costs you in-order processing per partition and makes offset commits much harder to get right. I would reach for the first, and consider the second only when ordering genuinely does not matter.

"How do you commit offsets safely across a rebalance?"

Commit in onPartitionsRevoked before releasing the partition, and be aware that under cooperative rebalancing only the partitions actually being moved appear in that callback, so the code is simpler and cheaper. Any commit carrying a stale generation ID is rejected with ILLEGAL_GENERATION or surfaces as CommitFailedException, which is the protocol protecting you from a zombie consumer committing after it has lost the partition. The bigger design point is to make processing idempotent, because rebalances will reprocess a batch, and no offset commit discipline eliminates that entirely.

Common misconceptions

"Heartbeats prove the consumer is healthy." Since KIP-62 heartbeats come from a background thread, so a consumer stuck in a 10-minute processing loop happily heartbeats the whole time. Liveness for processing is max.poll.interval.ms, and that is the timeout that actually fires in production incidents.

"Rebalancing is fast." Eager rebalancing on a large group is tens of seconds, because the coordinator waits for the slowest member's JoinGroup. Measure it (kafka.consumer:type=consumer-coordinator-metrics,name=rebalance-latency-avg) rather than assuming.

"More consumers means more throughput." Only up to the partition count. Beyond that, consumers idle. This is the most common wrong reaction to consumer lag.

"Sticky and cooperative are the same thing." Sticky is a property of the assignment (minimise movement). Cooperative is a property of the protocol (do not revoke what is not moving). StickyAssignor is sticky but eager: it computes a minimal-movement assignment and still revokes everything first. CooperativeStickyAssignor is both.

"Static membership eliminates rebalances." It eliminates the ones caused by a member leaving and returning with the same ID. Scaling the group, changing subscriptions, and adding partitions still rebalance, as they must.

Interview delivery note

Say this verbatim: "The timeout that fires in production is max.poll.interval.ms, not session.timeout.ms, because heartbeats run on a background thread. So a busy consumer looks alive to the coordinator right up until it evicts itself, and the fix is almost always max.poll.records down rather than the timeout up." That is the diagnosis most teams take an incident to learn, and saying it in one sentence signals you have been on the wrong end of it.

The senior-versus-staff separator is counting the rebalances a deploy causes. A senior engineer explains eager versus cooperative correctly. A staff engineer observes that a 24-pod rolling restart under eager rebalancing means 24 stop-the-world rebalances, that this cost is paid on every deploy regardless of whether anything is wrong, and that static membership takes it to zero. Framing it as a recurring tax rather than an incident is the shift in perspective.

If asked what you would configure on a new group, commit to the full set: CooperativeStickyAssignor, group.instance.id from the pod ordinal, session.timeout.ms around 60 seconds, max.poll.records sized so a worst-case batch finishes in well under a third of max.poll.interval.ms, and idempotent processing because rebalances will reprocess. Then name the trade you accepted: up to 60 seconds of unassigned partitions on a genuine node failure.

The Staff & Team Lead Interview Reference