Consumer lag as the health metric, and lag-based autoscaling

What it is

Consumer lag is, per partition, the log end offset minus the consumer group's committed offset: how many messages have been produced that this group has not yet processed.

$$\text{lag}_p = \text{LEO}_p - \text{committed offset}_p$$

It is the single most useful health metric for a streaming consumer, and the reason is that it is the only metric that directly measures whether you are keeping up. CPU, memory, throughput and error rate all describe how the consumer is behaving; lag describes whether that behaviour is sufficient. A consumer at 30 percent CPU processing 40,000 messages per second is healthy or catastrophically behind depending entirely on the production rate, and only lag tells you which.

The thing it is confused with, and the confusion causes bad autoscaling: lag in messages is not lag in time. A lag of 50,000 messages is four seconds on a partition producing 12,000 per second and eight hours on one producing 1.7 per second. Alerting on message-count lag means your threshold is wrong for every partition whose rate differs from the one you tuned against, and rates differ by orders of magnitude across topics and across the day.

What you usually want is time lag: how old is the next message this consumer will process. Kafka does not expose that directly, which is why it gets computed downstream or approximated.

The problem it solves

Three questions that no other metric answers.

"Is the consumer keeping up?" Throughput cannot answer this, because a consumer processing 40,000 per second while 45,000 arrive is at maximum throughput and falling behind at 5,000 per second. Only the comparison matters, and lag is the comparison.

"How stale is my downstream data?" For a materialised view, a search index, or a cache fed from a topic, lag is a direct measure of staleness, and staleness is usually the actual user-facing SLO. "Search results reflect inventory changes within 30 seconds" is a lag SLO wearing product clothing.

"How many consumers do I need right now?" Traffic on a streaming system is bursty in ways request/response systems are not, because upstream batch jobs, retry storms and backfills all dump work in at once. CPU-based autoscaling responds to these badly: a consumer blocked on a slow downstream call has low CPU while lag grows, so CPU-based scaling scales down exactly when you need to scale up. This is not a hypothetical; it is the standard failure of putting an HPA on a Kafka consumer without thinking.

Mechanics

Getting the number

kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --describe --group orders-processor

GROUP            TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID
orders-processor orders  0          8842019         8842104         85   consumer-3-a1b2
orders-processor orders  1          8839472         8901883         62411 consumer-7-c3d4
orders-processor orders  2          8841203         8841290         87   consumer-1-e5f6

Partition 1 is the story here. Aggregate lag would be about 62,600, which averaged across 12 partitions looks like 5,200 per partition and unremarkable. Lag is almost always concentrated, and the aggregate hides it. The causes of a single hot partition are key skew (one tenant, one popular product), an unbalanced assignment, or one consumer instance that is unhealthy while its peers are fine.

Programmatically, the two halves come from different places:

// Committed offsets: from the group coordinator.
Map<TopicPartition, OffsetAndMetadata> committed =
    admin.listConsumerGroupOffsets(groupId).partitionsToOffsetAndMetadata().get();

// Log end offsets: from the partition leaders.
Map<TopicPartition, ListOffsetsResultInfo> ends =
    admin.listOffsets(committed.keySet().stream()
        .collect(toMap(tp -> tp, tp -> OffsetSpec.latest()))).all().get();

committed.forEach((tp, off) ->
    lag.put(tp, ends.get(tp).offset() - off.offset()));

Note this measures committed lag, not in-flight lag. Records fetched and being processed but not yet committed count as lag. That is usually what you want (they are not done), but it means lag never reads zero on a busy consumer, and it means a consumer that commits every 5 seconds shows sawtooth lag with an amplitude of 5 seconds' worth of messages.

The consumer also exposes records-lag-max from its own metrics, which is cheaper to collect but only covers partitions that consumer owns and reads uncommitted position rather than committed offset. For alerting, the admin-client view is the right one, which is what Burrow and kafka-lag-exporter provide.

Time lag, and why you want it

Time lag is the timestamp of the log end offset's record minus the timestamp of the record at the committed offset. Computing it requires reading two records, which is why exporters approximate instead. The standard approximation:

$$\text{time lag} \approx \frac{\text{message lag}}{\text{consumption rate}}$$

This is the projected drain time and it is directly interpretable: "at the current rate, we are 47 seconds behind." It is also what you should alert on, and what you should put in an SLO, because it is stable across partitions with different rates and it does not change meaning when traffic doubles.

The refinement worth making: use the consumption rate, not the production rate, because you are asking how long this consumer needs. If consumption has stopped entirely the formula divides by zero, which is correct in spirit (infinite drain time) and needs a guard in code.

Lag-based autoscaling with KEDA

KEDA (Kubernetes Event-Driven Autoscaling) is the standard mechanism. It runs a scaler that polls lag and drives a normal HPA.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: orders-processor
spec:
  scaleTargetRef:
    name: orders-processor
  minReplicaCount: 3
  maxReplicaCount: 12          # == partition count. More would idle.
  pollingInterval: 15
  cooldownPeriod: 300          # 5 min before scaling to zero-ish; prevents flapping
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 30      # react fast to a backlog
          policies:
            - type: Percent
              value: 100                       # allowed to double
              periodSeconds: 60
        scaleDown:
          stabilizationWindowSeconds: 600     # leave slowly: scaling down rebalances
          policies:
            - type: Pods
              value: 1
              periodSeconds: 180               # one pod every 3 minutes
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: broker:9092
        consumerGroup: orders-processor
        topic: orders
        lagThreshold: "1000"                   # target lag PER REPLICA
        activationLagThreshold: "100"

Three things in that config carry the design and are worth stating explicitly.

maxReplicaCount equals the partition count. A consumer group cannot have more active consumers than partitions; the extras idle. An HPA that can scale past the partition count will do so under load, and the additional pods consume quota, trigger a rebalance on the way in, and do nothing. This is the most common KEDA misconfiguration on Kafka.

Asymmetric scale-up and scale-down. Scaling up is cheap and urgent. Scaling down triggers a rebalance, which pauses processing, which increases lag, which can trigger a scale-up: a flapping loop that spends its life rebalancing. The 600-second stabilisation window and one-pod-per-three-minutes policy make scale-down deliberately reluctant. Combined with cooperative rebalancing and static membership, the cost of each scaling event drops substantially, and you should do that work before turning on autoscaling rather than after.

lagThreshold is per replica, not total. KEDA computes desired replicas as ceil(totalLag / lagThreshold). With lagThreshold: 1000 and a total lag of 6,500 it asks for 7 replicas. Choosing the number means asking: how much lag per consumer is acceptable given per-message processing time? If a consumer handles 500 messages per second, a 1,000-message threshold targets roughly 2 seconds of lag per replica. Derive it from the time target rather than guessing.

The scaling ceiling nobody accounts for

Autoscaling a consumer group has a hard limit at the partition count, and a softer one below it. Suppose lag is concentrated on 2 of 12 partitions because of key skew. Scaling to 12 consumers gives each one partition, and the two hot partitions still have one consumer each, processing at single-consumer speed. Autoscaling does not help a skewed partition at all, and this is the case where teams scale to the ceiling, see no improvement, and conclude Kafka is slow.

The fixes are upstream of the consumer: repartition on a better key, add a salt to hot keys and handle the fan-in downstream, or move to a work-queue model where ordering is not required. None of them is an autoscaling setting, which is why diagnosing skew before reaching for the HPA matters.

A worked example: an autoscaler that scaled the wrong way

A notifications consumer: 24 partitions, 6 pods, each message sending an HTTP push to a vendor. The team had an HPA on CPU with a 70 percent target.

At 19:00 the vendor's API latency went from 40 ms to 1,100 ms.

19:00  vendor p99: 40ms -> 1100ms
19:02  per-pod throughput: 2,400/s -> 190/s
19:02  pod CPU: 62% -> 11%     (threads blocked on I/O, not computing)
19:04  HPA sees CPU 11% vs target 70%, scales 6 -> 2 pods
19:04  REBALANCE. Processing pauses ~35s.
19:06  lag: 180k and climbing at ~7k/s
19:20  lag: 6.1M. On-call paged by a downstream staleness alert, not by the consumer.

The autoscaler removed two thirds of the capacity at the exact moment throughput per pod had dropped by 92 percent, because blocked I/O threads consume no CPU. This is the structural failure of CPU-based autoscaling for I/O-bound consumers, and it is not a tuning problem; the signal is measuring the wrong thing.

The redesign.

First, the target. Product wanted notifications delivered within 60 seconds at p99. Steady-state production was about 14,000 messages per second, and measured per-pod throughput with a healthy vendor was 2,400 per second.

Required replicas at steady state = 14,000 / 2,400 = 5.8  -> 6
Lag budget for 60s of delay      = 60 x 14,000 = 840,000 messages total
Per-replica lag threshold        = 840,000 / 24 (max replicas) = 35,000

That threshold was deliberately reduced to 12,000, targeting roughly 20 seconds of drain rather than 60, leaving headroom for the scale-up itself to take effect (pod start plus rebalance is 30 to 40 seconds, during which lag keeps growing).

minReplicaCount: 4
maxReplicaCount: 24              # == partitions
triggers:
  - type: kafka
    metadata:
      consumerGroup: notifications
      topic: notifications
      lagThreshold: "12000"

Second, the consumer itself: a bounded worker pool inside each pod so a slow vendor call blocks one worker rather than the poll loop, with a hard 3-second timeout and a circuit breaker, plus max.poll.records dropped from 500 to 100.

Third, cooperative rebalancing and static membership, so scaling events cost about a second instead of 35.

The same incident, replayed six weeks later when the vendor degraded again:

                              CPU-based (before)    lag-based (after)
scaling direction at t+2min   6 -> 2 pods           6 -> 19 pods
peak lag                      6.1M                  310k
time to drain after recovery  74 min                4 min
p99 notification delay        4h 20m                 51s
rebalance time per event      35s                    1.1s

Peak lag dropped by a factor of 20 and the product SLO held. Note the scale-up went to 19 rather than 24, because lag divided by threshold landed there, and note it did not need to go to 24 because the circuit breaker was shedding the worst of the slow calls.

The number worth keeping: the autoscaler must react faster than lag accumulates, and the reaction is not instant. Pod start plus rebalance was 30 to 40 seconds here, during which production continued at 14,000 per second, adding roughly 500,000 messages of lag before the new capacity did anything. That is why the threshold was set at a third of the budget rather than at the budget.

Production evidence

KEDA is a CNCF graduated project and its Kafka scaler is among the most-used, with the lagThreshold semantics described above and an explicit allowIdleConsumers flag that defaults to false, specifically to stop people scaling past the partition count.

LinkedIn's Burrow was built because threshold alerting on lag produced too many false positives at their scale. Burrow's approach is to evaluate lag as a trend over a sliding window of offset commits, classifying a partition as OK, WARN or ERR based on whether the consumer is making progress and whether lag is growing, rather than comparing to a fixed number. This is the right insight: growing lag matters, large lag does not necessarily, because a consumer at 2 million lag draining steadily after a backfill is healthy, and one at 5,000 lag that has not committed in three minutes is not.

Confluent Cloud, Datadog and Grafana's Kafka integrations all surface both lag and the estimated time lag, and Confluent's guidance recommends alerting on the time-based figure for the reasons above.

Uber's uReplicator and their Kafka operations writing describe lag as the primary SLI for stream processing, with per-partition lag rather than aggregate as the alerting unit, precisely because of the concentration problem.

Kafka Streams exposes lag differently: KafkaStreams.allLocalStorePartitionLags() gives you the lag of each state store's restoration, which is the metric that matters during a rebalance, since a task cannot serve interactive queries until its store has caught up.

The debate

Lag-based versus CPU-based autoscaling. For a Kafka consumer, lag-based wins and the worked example is why: CPU is a proxy that inverts under the most common failure mode, an I/O-bound consumer blocked on a slow dependency. The counter-argument is that lag-based scaling reacts to upstream behaviour, so a producer burst scales your consumers even when they are perfectly healthy, and if the downstream dependency is the bottleneck, adding consumers makes it worse by increasing concurrency against an already-struggling service. That is a real failure and the mitigation is a circuit breaker plus a bounded concurrency limit per pod, so scaling adds queue drain capacity without multiplying pressure on the dependency.

My position: scale on lag, bound the total concurrency against downstream dependencies independently, and cap replicas at the partition count. Those three together are the design; any one alone has a clear failure mode.

Should you alert on lag at all, or on the SLO it implies? Alert on the SLO. "Lag above 100,000" pages someone during a backfill that is draining fine. "Search index staleness above 60 seconds, measured as projected drain time, sustained for 5 minutes" is actionable and does not fire spuriously. The Burrow model of evaluating progress rather than magnitude is the more sophisticated version of the same idea.

Absolute lag or lag derivative? Both, for different purposes. The derivative (is lag growing?) is the health signal, because a consumer whose lag is growing will eventually breach any threshold, and catching it early is the whole point. The absolute value drives the autoscaler, because that is what determines how much capacity you need. Alerting on the derivative and scaling on the magnitude is the right split, and conflating them gives you either a noisy alert or a sluggish autoscaler.

When is lag the wrong metric? When the consumer's work is not proportional to message count. A consumer where one message triggers a 40-minute batch job has a lag of 1 that means 40 minutes of work, and a lag of 3 that means two hours. For those, lag in messages is nearly meaningless and you should measure the actual queue of work in whatever units it comes in. Same for consumers with wildly variable per-message cost: a lag of 1,000 tiny messages and 1,000 huge ones are different situations and the metric cannot tell them apart.

Follow-up Q&A

"Aggregate lag is 500,000. Is that bad?"

Unanswerable as stated, and I would ask three things. What is the production rate, because 500,000 at 100,000 per second is five seconds and fine, while at 100 per second it is 83 minutes. Is it growing or shrinking, because a draining backlog after a deploy is healthy and a growing one at any magnitude is not. How is it distributed across partitions, because 500,000 spread evenly over 24 partitions is a capacity question and 500,000 on one partition is a skew question, and those have entirely different fixes. The metric I actually want is projected drain time per partition.

"Why not just use CPU for autoscaling?"

Because an I/O-bound consumer blocked on a slow dependency has low CPU while lag grows, so CPU-based scaling scales down exactly when you need to scale up. I have watched this remove two thirds of capacity during a vendor slowdown. CPU is a reasonable secondary signal for a genuinely compute-bound consumer, but even then lag is the metric that maps to the user-facing SLO, and CPU is only a proxy for it.

"You scaled to the partition count and lag is still growing. Now what?"

You are out of consumer parallelism, so the options are all structural. Increase partitions, which increases the ceiling but requires care with key ordering and is impossible on a compacted topic. Make each consumer faster: batch the downstream calls, remove a synchronous dependency, or process asynchronously within the consumer with a worker pool, accepting the loss of per-partition ordering. Or shed load: if the messages have differing value, route low-value ones to a separate topic with its own group and let that one lag. I would look at per-message processing time first, because a consumer doing 190 messages per second is usually waiting on something rather than computing, and removing that wait is cheaper than any repartitioning project.

"How do you avoid autoscaling flapping?"

Asymmetric behaviour: fast scale-up, slow scale-down, with a long stabilisation window on the way down (I use 600 seconds) and a policy limiting removal to one pod every few minutes. The reason scale-down must be reluctant is that it triggers a rebalance, which pauses processing, which raises lag, which can trigger a scale-up. Cooperative rebalancing and static membership reduce the cost of each event enough that the loop is much less likely to start, which is why I would fix rebalancing before enabling autoscaling.

"Lag is zero but users say data is stale. Explain."

Several possibilities, in the order I would check. The producer is behind, so the data never reached Kafka; check producer-side buffer metrics and the timestamp of the latest record versus wall clock. The consumer is committing without processing, which happens with auto-commit and asynchronous handoff to a worker pool: offsets advance when the record is handed off, not when the work completes. A downstream stage is the slow one, and the consumer writes to a database or search index that is itself behind. Or lag is being measured on the wrong group, for example a monitoring group that reads the topic and does nothing. The second one is the interesting bug because lag looks perfect while nothing is being done.

Common misconceptions

"Lag zero is the goal." Committed lag never reaches zero on a healthy busy consumer, because records in flight and the commit interval both contribute. A sawtooth between zero and one commit interval's worth of messages is what healthy looks like. Chasing zero leads to committing per record, which costs throughput substantially.

"Aggregate lag tells you the story." It hides concentration, and concentration is the normal case. Alert per partition or on the maximum, not on the sum.

"More consumers always reduce lag." Only up to the partition count, and not at all if the lag is concentrated on one hot partition. Scaling a skewed workload is the classic wasted response.

"Lag is a Kafka problem." Lag is almost always a consumer problem or a downstream problem. Brokers rarely limit consumption. The usual root cause is a synchronous call in the processing loop.

"Auto-commit makes lag accurate." Auto-commit advances offsets on a timer for records that poll() returned, regardless of whether your code finished with them. With any asynchronous handoff, auto-commit makes lag optimistic: it reports work as done that is still queued in your process, and a crash loses it. Manual commit after processing is what makes lag mean what you think it means.

Interview delivery note

Say this verbatim: "I would not autoscale a Kafka consumer on CPU, because an I/O-bound consumer blocked on a slow dependency has low CPU while lag grows, so the autoscaler scales down exactly when you need it to scale up. Lag is the only metric that measures whether you are keeping up." It is a concrete, defensible position with a stated failure mode, which is what the question is actually testing.

The senior-versus-staff separator is converting lag into time and deriving the threshold from an SLO. A senior engineer explains lag and points at KEDA. A staff engineer says "the product target is 60 seconds of delay, production is 14,000 per second, so the total lag budget is 840,000; I set the threshold to a third of that because pod start plus rebalance is 30 to 40 seconds and lag keeps growing during the reaction." That arithmetic is the difference between configuring an autoscaler and designing one.

The second signal is capping maxReplicaCount at the partition count unprompted, and noting that autoscaling cannot fix a skewed partition. Both show you have watched an autoscaler run into a ceiling and understood why.

The Staff & Team Lead Interview Reference