Dead letter queues, poison pills and retry topics with backoff tiers

What it is

A poison pill is a message that will fail every time it is processed, no matter how many times you retry: malformed JSON, a schema the consumer cannot deserialise, a reference to an entity that was deleted, a value that violates an invariant. Its defining property is that retrying is useless, and its defining danger is that a naive consumer will retry it forever, blocking the partition behind it.

A dead letter queue (DLQ) is where such a message goes so the consumer can move on: a separate topic holding messages that could not be processed, along with enough context to diagnose and reprocess them.

Retry topics with backoff tiers are the mechanism in between. Rather than retrying in place (which blocks the partition) or dead-lettering immediately (which gives up on transient failures), the message is republished to a topic whose consumer deliberately waits before processing, with several such topics at increasing delays.

The distinction that organises the whole design: transient failures and permanent failures need opposite treatment. A downstream service returning 503 is transient and will succeed on retry; a message with a null required field is permanent and will never succeed. Treating a permanent failure as transient burns retries and blocks the partition. Treating a transient failure as permanent dead-letters messages that would have worked, and a downstream outage then fills your DLQ with tens of thousands of perfectly good messages.

What this is confused with: a DLQ is not an error log. An error log is for humans to read; a DLQ is a queue of work that has not been done and, in most systems, still needs to be. If nobody ever reprocesses from your DLQ, you have built an expensive way to delete messages.

The problem it solves

Head-of-line blocking. Kafka delivers a partition in order, and a consumer that cannot process message N cannot skip to N+1 without abandoning N. A naive retry-in-place loop on a poison pill stops that partition permanently. With 12 partitions, one poison pill takes out 8 percent of your throughput, and the lag on that partition grows without bound while the other 11 look healthy, so aggregate lag dashboards understate the problem badly.

The retry storm that amplifies an outage. A downstream service degrades. Every consumer retries immediately, several times, so the failing service now receives several times its normal request rate at exactly the moment it is struggling. Immediate retries convert a partial degradation into a full outage. This is the same dynamic described in resilience patterns, and it is why backoff is not a nicety.

Silent loss through a swallowed exception. The most common "solution" in code review:

try {
    process(record);
} catch (Exception e) {
    log.error("failed to process", e);      // and the message is gone forever
}

The consumer keeps up, lag is zero, dashboards are green, and messages are being discarded. This is worse than head-of-line blocking, because blocking is loud and this is silent. Any consumer whose catch block does not either retry, dead-letter, or deliberately drop with a metric is losing data.

Mechanics

Classify the failure first

The design decision that everything else follows from is a function that answers "is this worth retrying."

enum Disposition { RETRY, DEAD_LETTER, DROP }

Disposition classify(Exception e) {
    // Permanent: the message itself is wrong. No amount of retrying helps.
    if (e instanceof DeserializationException) return DEAD_LETTER;
    if (e instanceof SchemaValidationException) return DEAD_LETTER;
    if (e instanceof IllegalArgumentException)  return DEAD_LETTER;
    if (e instanceof EntityNotFoundException)   return DEAD_LETTER;

    // Transient: the world is temporarily wrong. Retry will likely succeed.
    if (e instanceof TimeoutException)          return RETRY;
    if (e instanceof ConnectException)          return RETRY;
    if (e instanceof HttpServerErrorException)  return RETRY;   // 5xx
    if (e instanceof OptimisticLockException)   return RETRY;

    // 4xx from a downstream is permanent: the request is bad, not the server.
    if (e instanceof HttpClientErrorException)  return DEAD_LETTER;

    // Unknown: retry a bounded number of times, then dead-letter.
    return RETRY;
}

The HttpClientErrorException line is the one worth arguing about in a design review. A 400 from a downstream service means your request is malformed, and retrying an identical malformed request produces an identical 400 forever. The exceptions are 408 (request timeout) and 429 (too many requests), which are 4xx and genuinely transient, so a real implementation special-cases them.

Non-blocking retry with backoff tiers

The pattern, popularised by Uber's writing on it:

                  ┌──────────────┐
    orders ──────▶│   consumer   │──── success ──▶ done
                  └──────┬───────┘
                         │ transient failure
                         ▼
                  orders.retry.5s ────▶ consumer waits 5s, retries
                         │ still failing
                         ▼
                  orders.retry.1m ────▶ waits 1m, retries
                         │
                         ▼
                  orders.retry.10m ───▶ waits 10m, retries
                         │
                         ▼
                    orders.DLQ ───────▶ human inspection, manual replay

Each retry topic has its own consumer group, so the main topic is never blocked. A message that fails goes to a retry topic and the main consumer immediately commits and moves to the next record.

The delay mechanism is the interesting part. You cannot simply sleep per record, because that blocks the retry topic's own partition. The standard implementation uses pause() and resume() on the consumer:

@KafkaListener(topics = "orders.retry.1m", groupId = "orders-retry-1m")
public void onRetry(ConsumerRecord<String, Order> record, Acknowledgment ack,
                    Consumer<?, ?> consumer) {
    long publishedAt = headerLong(record, "x-retry-published-at");
    long readyAt = publishedAt + Duration.ofMinutes(1).toMillis();
    long waitMs = readyAt - System.currentTimeMillis();

    if (waitMs > 0) {
        // Pause the partition and seek back. The broker is not polled for this
        // partition until resume(), so we hold no lock and block nothing else.
        consumer.pause(Set.of(new TopicPartition(record.topic(), record.partition())));
        consumer.seek(new TopicPartition(record.topic(), record.partition()), record.offset());
        scheduler.schedule(() -> consumer.resume(
            Set.of(new TopicPartition(record.topic(), record.partition()))),
            waitMs, MILLISECONDS);
        return;                                  // do NOT ack
    }

    try {
        process(record.value());
        ack.acknowledge();
    } catch (Exception e) {
        forward(record, nextTier(record), e);    // 1m -> 10m, or 10m -> DLQ
        ack.acknowledge();
    }
}

Because messages arrive in a retry topic in publish order and all wait the same duration, the head of the partition is always the one that becomes ready first, so pausing on the head is correct and does not starve anything behind it. That property is what makes fixed-delay tiers work and is why the tiers have fixed delays rather than per-message exponential backoff: with variable delays, a message needing 10 minutes at the head blocks one needing 5 seconds behind it.

Spring Kafka provides this with @RetryableTopic, which generates the topics and the listeners:

@RetryableTopic(
    attempts = "4",
    backoff = @Backoff(delay = 5_000, multiplier = 12.0),   // 5s, 1m, 12m
    dltStrategy = DltStrategy.FAIL_ON_ERROR,
    autoCreateTopics = "true",
    exclude = {DeserializationException.class,               // straight to DLT
               IllegalArgumentException.class})
@KafkaListener(topics = "orders")
public void onOrder(Order order) { process(order); }

@DltHandler
public void onDlt(Order order,
                  @Header(KafkaHeaders.ORIGINAL_TOPIC) String topic,
                  @Header(KafkaHeaders.EXCEPTION_MESSAGE) String reason) {
    dlqMetrics.increment(topic, reason);
    log.error("dead-lettered from {}: {}", topic, reason);
}

exclude is the classification function in declarative form: those exception types skip the retry tiers entirely and go straight to the dead-letter topic, which is the correct handling for permanent failures.

Deserialisation failures need special handling

A poison pill that fails deserialisation cannot be handled by your listener at all, because the listener never runs. The failure happens in the consumer's deserialiser before your code sees anything, and the default behaviour is an exception that kills the poll loop, restarts, reads the same record, and fails again: an infinite loop that consumes CPU and blocks the partition.

The fix is a delegating deserialiser that captures the failure and hands your listener a marker:

props.put(VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);
props.put(ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS, KafkaAvroDeserializer.class);

ErrorHandlingDeserializer catches the exception, passes a null value plus a header carrying the failure, and lets the error handler dead-letter it normally. Without this, a single malformed message on a topic is an unrecoverable partition stall, and the only manual fix is seeking the consumer group past the offset, which means someone runs kafka-consumer-groups --reset-offsets in production under pressure.

What must be on the dead letter record

A DLQ message without context is a message nobody can act on. Minimum:

ProducerRecord<byte[], byte[]> dlqRecord = new ProducerRecord<>(dlqTopic,
    null, record.key(), record.value());

Headers h = dlqRecord.headers();
h.add("x-original-topic",      record.topic().getBytes());
h.add("x-original-partition",  Ints.toByteArray(record.partition()));
h.add("x-original-offset",     Longs.toByteArray(record.offset()));
h.add("x-original-timestamp",  Longs.toByteArray(record.timestamp()));
h.add("x-exception-class",     e.getClass().getName().getBytes());
h.add("x-exception-message",   String.valueOf(e.getMessage()).getBytes());
h.add("x-stacktrace",          truncate(stackTrace(e), 4096).getBytes());
h.add("x-retry-count",         Ints.toByteArray(retryCount));
h.add("x-consumer-group",      groupId.getBytes());
h.add("x-app-version",         appVersion.getBytes());       // which build failed
h.add("x-dead-lettered-at",    Longs.toByteArray(now));

The original offset is the one people forget and the one that matters most for triage: it lets you go back to the source topic and see what was around the failing message, which is how you discover that the whole batch was corrupt rather than one record. x-app-version matters because a bug fixed in a later build means the DLQ contents from the earlier build are now replayable.

Replay is a feature, not an afterthought

// Replay tool: read the DLQ, republish to the original topic, with a guard.
public void replay(String dlqTopic, Predicate<ConsumerRecord<?, ?>> filter, int max) {
    int replayed = 0;
    for (ConsumerRecord<byte[], byte[]> r : poll(dlqTopic)) {
        if (!filter.test(r)) continue;
        if (replayed++ >= max) break;                       // bounded, always

        String original = header(r, "x-original-topic");
        ProducerRecord<byte[], byte[]> out =
            new ProducerRecord<>(original, r.key(), r.value());
        out.headers().add("x-replayed-from-dlq", TRUE_BYTES);   // consumers can tell
        out.headers().add("x-replay-batch-id", batchId.getBytes());
        producer.send(out);
    }
}

Two details that are load-bearing. The bound (max) prevents a well-intentioned replay of 400,000 messages from overwhelming a service that has just recovered. The x-replayed-from-dlq header lets consumers distinguish a replay from live traffic, which matters for metrics (a replay spike is not a traffic spike) and for any logic that should not fire twice, like sending a notification.

A worked example: a DLQ nobody could use

A payments reconciliation consumer processing about 2 million settlement records a day. It had a DLQ from day one, and after fourteen months the DLQ held 847,000 messages that had never been examined.

The team had done the visible part correctly (a DLQ existed, nothing was silently swallowed) and none of the rest.

What an audit of the 847,000 found:

cause (reconstructed by sampling 500)          share      estimated count
─────────────────────────────────────────────────────────────────────────
downstream 503 during 4 known outages          61%        ~517,000
schema change: added required field, old       22%        ~186,000
  consumers rejected 3 days of traffic
genuine bad data (null merchant_id)             9%         ~76,000
unknown: no exception info on the record        8%         ~68,000

Eighty-three percent of the DLQ was replayable. Half a million messages went to the DLQ because a downstream service was down for a total of about 40 minutes across four incidents, and the consumer had attempts=3 with no delay, so all three attempts happened within about 200 milliseconds and all three hit the same outage.

The 8 percent "unknown" was the more damaging finding: the DLQ record carried only the payload, so nobody could tell what had failed or why, and those 68,000 messages were unrecoverable as a practical matter.

The redesign:

@RetryableTopic(
    attempts = "4",
    backoff = @Backoff(delay = 10_000, multiplier = 30.0),  // 10s, 5m, 2.5h
    exclude = {DeserializationException.class, ValidationException.class})

The tier spacing was chosen from incident data rather than intuition: their four outages had lasted 4, 11, 19 and 6 minutes, so a tier at 5 minutes catches roughly half and a tier at 2.5 hours catches essentially everything they had ever seen. The old configuration's three attempts spanned 200 milliseconds, which cannot survive any outage measured in minutes. That mismatch between retry span and actual outage duration is the single most common DLQ design error.

Plus: full context headers, a circuit breaker so a sustained downstream failure stops attempting rather than filling retry topics, a bounded replay tool, and two alerts.

The alerts are the part that changed behaviour, more than any config:

- alert: DLQMessagesArriving
  expr: rate(dlq_messages_total[5m]) > 0
  for: 10m
  annotations:
    summary: "Messages are being dead-lettered. Something is broken now."

- alert: DLQBacklogNotDraining
  expr: dlq_depth > 0 and changes(dlq_replayed_total[7d]) == 0
  for: 7d
  annotations:
    summary: "DLQ has messages and nothing has been replayed in a week."

The second alert is the unusual one and it is the one that prevents the failure mode described here: a DLQ that accumulates because nobody owns draining it. A DLQ with a permanent backlog is unprocessed work, not archived errors, and without an alert that fact stays invisible indefinitely.

Twelve months later:

                              before          after
DLQ arrivals per month        ~60,000         ~340
DLQ standing depth            847,000         0-200 (drained weekly)
messages lost to a 15-min
  downstream outage           ~22,000         0 (retried at the 5m tier)
mean time to notice a
  systematic failure          14 months       10 minutes
replay tooling                none            bounded CLI + runbook

The 22,000-to-0 line is the retry tiers doing their job. The 14-months-to-10-minutes line is the alert, and it required no code at all.

Production evidence

Uber's engineering post "Building Reliable Reprocessing and Dead Letter Queues with Apache Kafka" is the canonical reference for the tiered retry topic pattern, and their motivation is exactly the head-of-line blocking problem: in-place retry on a partition blocks every subsequent message, so retries must move to a different topic.

Spring Kafka's @RetryableTopic (from 2.7) implements this pattern directly, with DltStrategy, exception classification via include/exclude, and automatic topic creation. Its existence as a first-class framework feature, rather than a recipe, is a signal the pattern has settled.

Kafka Connect has built-in DLQ support via errors.deadletterqueue.topic.name, with errors.deadletterqueue.context.headers.enable=true adding the original topic, partition, offset and exception as headers. That header set is a good model for what your own DLQ records should carry, since Connect's designers had the same triage problem.

AWS SQS has DLQs as a native feature with a maxReceiveCount redrive policy, and a "redrive to source" operation added in 2021 specifically because customers had DLQs they could not easily replay from. The addition of that feature five years after DLQs themselves is a nice illustration of the point that replay tooling is the part everyone skips.

RabbitMQ implements dead-lettering via exchanges (x-dead-letter-exchange), and the common delayed-retry recipe there uses a queue with a TTL that dead-letters back to the main exchange on expiry, which is the same tiered-delay idea built from different primitives.

The debate

Retry in place or retry via topics? In-place retry is simpler and preserves ordering, and it blocks the partition for the duration. That is acceptable when retries are fast (a few hundred milliseconds total) and ordering genuinely matters. Topic-based retry does not block and does not preserve ordering, since a retried message is reprocessed after messages that came later.

My position: in-place for a short bounded retry (three attempts inside a second), topic-based for anything longer. The trap is configuring long in-place retries, because a 30-second exponential backoff in the consumer thread also blows through max.poll.interval.ms and triggers a rebalance, converting a slow message into a group-wide stall (see consumer rebalancing).

Does topic-based retry break ordering, and does that matter? It breaks it, and whether it matters depends on the operation. For an idempotent upsert of a current state, reordering is harmless as long as you carry a version and reject stale writes. For an append-only ledger or a state machine with ordered transitions, it is not harmless, and the honest answer for those is that you probably should block: stop the partition, alert, and fix it, because processing message N+1 while N is unresolved produces a wrong result that the DLQ will not tell you about.

How many tiers, and what delays? Derive them from your actual outage durations, not from a geometric sequence that looks tidy. Pull the last year of downstream incidents, take the median and p90 durations, and put tiers around them. In the worked example, outages of 4 to 19 minutes made a 5-minute tier and a 2.5-hour tier sensible, and a tidy 1s/10s/100s ladder would have caught none of them. Three tiers is almost always enough, and more than four is a sign you are trying to use retries to solve availability, which is what a circuit breaker is for.

Should a DLQ have a retention policy? Yes, and it should be long: 30 days minimum. The temptation is infinite retention "so nothing is lost," which produces the 847,000-message situation where retention is doing the job that ownership should. My position: 30-day retention plus an alert when the DLQ has not been drained in a week. The alert is what makes finite retention safe, and the finite retention is what forces someone to own it.

Is a DLQ always right? No. For a message stream where a failure means the message is genuinely worthless (a metrics sample, a cache invalidation that the next write will supersede), drop it with a counter and skip the DLQ. Building a DLQ you will never replay from is machinery pretending to be diligence. The test: if a message lands in this DLQ, will anyone do anything about it? If the honest answer is no, increment a metric and move on.

Follow-up Q&A

"A poison pill is blocking a partition in production right now. What do you do?"

Immediate mitigation is to get past the offset: either the consumer's error handler dead-letters it (if one is configured) or you reset the group's offset past it with kafka-consumer-groups --reset-offsets --to-offset N+1 --topic t:p --execute, which requires stopping the consumer group first. Capture the message before you skip it, by consuming that single offset with a raw byte deserialiser, because once you have moved past it you will want to know what it was. Then the real fix: an ErrorHandlingDeserializer so a deserialisation failure dead-letters instead of stalling, because if this happened once it will happen again and the manual procedure is not something you want to run at 3am.

"How do you decide retry versus dead-letter?"

Classify the exception. Transient (timeouts, connection failures, 5xx, optimistic lock conflicts, 429) means the world is temporarily wrong and a retry will plausibly succeed. Permanent (deserialisation failures, validation errors, 4xx other than 408 and 429, references to entities that no longer exist) means the message is wrong and no number of retries changes it. Unknown exceptions get a bounded retry then dead-letter, because the cost of retrying something permanent a few times is small and the cost of dead-lettering something transient is a full DLQ during an outage.

"Your DLQ has 100,000 messages after a downstream outage. Now what?"

First, do not replay them all at once into a service that just recovered, because that is a self-inflicted second outage. Sample 50 to confirm they share the outage's signature rather than hiding several distinct causes, since a DLQ during an incident usually contains at least two problems. Then replay in bounded batches with a rate limit, tagged with a replay header so metrics and any notify-the-customer logic can distinguish them. Monitor the failure rate during the replay and stop if it is high, because that means the underlying issue is not actually fixed. And the real lesson from 100,000 messages in the DLQ is that the retry tiers were shorter than the outage, so that is the config change to make afterwards.

"Doesn't topic-based retry break ordering?"

Yes, and you have to decide whether that is acceptable per topic. If processing is an idempotent upsert with a version check, reordering is harmless. If it is an ordered state machine or an append-only ledger, it is not, and for those the correct design is to block the partition and alert rather than to reorder silently. The failure I want to avoid is a system that reorders without anyone having decided that reordering is acceptable.

"How do you stop a DLQ from becoming a graveyard?"

Two alerts and an owner. Alert on the arrival rate, because a nonzero arrival rate means something is broken right now. Alert on backlog age, specifically a DLQ with messages and no replay activity in seven days, because that is the failure where a DLQ silently becomes an archive. Then name an owner in the runbook. The second alert is the unusual one and it is the one that works, because it turns "we should look at that" into a page.

"What about the DLQ's own failures?"

If publishing to the DLQ fails, you have a decision to make and it should be explicit. The options are: block the consumer (safest, and it will page someone), retry the DLQ publish with its own bounded backoff, or write to local disk as a last resort. What you must not do is catch the DLQ-publish exception and log it, because that is the silent-loss pattern one level up. I would block, on the reasoning that a DLQ I cannot write to means I have no way to avoid losing data, and stopping is preferable.

Common misconceptions

"A DLQ prevents data loss." It relocates data. If nobody replays from it, the messages are lost with extra steps and a storage bill. The DLQ is only as good as the replay process attached to it.

"Retry three times and give up is a reasonable default." Three immediate retries span a few hundred milliseconds, which cannot survive any outage measured in minutes, and downstream outages are measured in minutes. Three attempts with no delay is functionally one attempt. Either space them out or accept that transient failures will fill your DLQ.

"Exponential backoff in the consumer thread is fine." It blocks the partition for the whole backoff, and if the total exceeds max.poll.interval.ms the consumer is evicted and the entire group rebalances. A "helpful" backoff of 30 seconds turns a single slow message into a group-wide stall.

"Deserialisation failures go to the DLQ like anything else." Only with an ErrorHandlingDeserializer or equivalent. By default the failure happens before your code runs, the poll loop dies, the consumer restarts, reads the same record, and loops forever. It is the one poison pill your error handler never sees.

"The DLQ needs infinite retention so nothing is lost." Infinite retention is what lets a DLQ accumulate for fourteen months without anyone noticing. Finite retention plus an alert on staleness forces ownership, which is the actual goal.

Interview delivery note

Say this verbatim: "Retry in place blocks the partition, so anything longer than about a second goes to a retry topic with its own consumer group and its own delay tier. And I size the tiers from our actual downstream outage durations, because three immediate retries span 200 milliseconds and every real outage lasts minutes." The second sentence is the one that lands, because it is a specific, checkable mistake that nearly every team has made.

The senior-versus-staff separator is the alert on DLQ staleness. A senior engineer designs the retry tiers, the classification and the headers correctly. A staff engineer adds that a DLQ with a permanent backlog is unprocessed work rather than archived errors, and alerts on "has messages and nothing replayed in seven days," because the organisational failure (nobody owns draining it) is more likely than the technical one. Pairing that with a named owner in the runbook shows you have seen a DLQ with 800,000 messages in it.

The second signal is ErrorHandlingDeserializer. Knowing that a deserialisation failure never reaches your error handler, and that the default behaviour is an infinite restart loop on the same offset, is the kind of specific that only comes from having hit it.

The Staff & Team Lead Interview Reference