Retries and dead-letter topics

Some records cannot be processed right now (the database is restarting), and some can never be processed (the payload is malformed, the referenced order does not exist). A consumer that treats both the same either stalls a partition forever on the second kind or drops the first kind on the floor. Retries handle the first; a dead-letter topic handles the second; the design question is where the retrying happens and what a dead letter means once it is there. This lesson is blocking retries, retry topics with backoff, the poison pill, and the operational side of a DLT.

#Blocking retry

The simplest retry is in the consumer: catch, wait, try again, on the same thread, holding the partition.

@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template);         // → <topic>.DLT after retries
    var handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 2L));   // 1 s apart, 2 retries
    handler.addNotRetryableExceptions(DeserializationException.class, IllegalArgumentException.class);
    return handler;
}

Blocking retry preserves ordering: nothing behind the failing record is processed until it succeeds or is given up on. That is the right behaviour when order matters and outages are short. Its cost is exactly that: the partition is stalled, lag climbs on it, and if the total retry time approaches max.poll.interval.ms the consumer is evicted from the group and the record is redelivered to another member, which retries it all over again. Keep blocking retries short (seconds, not minutes) and few.

#Non-blocking retry: retry topics

For long backoffs without stalling the partition, the failed record is published to a retry topic and the consumer moves on. A second listener on the retry topic waits out the backoff and tries again; a further failure goes to the next retry topic with a longer delay; the last failure goes to the dead-letter topic.

@RetryableTopic(
    attempts = "4",
    backoff = @Backoff(delay = 1_000, multiplier = 3.0, maxDelay = 60_000),   // 1 s, 3 s, 9 s
    exclude = { DeserializationException.class, ValidationException.class }, // straight to the DLT
    dltTopicSuffix = "-dlt"
)
@KafkaListener(topics = "payments", groupId = "ledger")
public void onPayment(Payment p) {
    ledger.apply(p);          // throws on a transient failure → payments-retry-1000, -3000, -9000, then payments-dlt
}
 
@DltHandler
public void onDead(Payment p, @Header(KafkaHeaders.EXCEPTION_MESSAGE) String why) {
    alerts.raise("payment dead-lettered", p.id(), why);
}

Spring Kafka creates the topics (payments-retry-1000, payments-retry-3000, payments-retry-9000, payments-dlt) and the listeners, and each retry listener pauses its partition until the record's due time. The main partition never stalls; a failing record costs the group nothing but the retry topic's own throughput.

What it costs is ordering. A record in a retry topic is processed after records that arrived behind it on the main topic. For a payment ledger where each payment is independent, that is fine; for a per-key sequence ("order created" then "order shipped") a retried "created" can land after "shipped". The consumer must be able to handle that, usually by treating the record as an upsert keyed on the entity's version, or the pipeline must use blocking retries and accept the stall. There is no configuration that gives non-blocking retries and strict ordering; it is a design choice, and it should be written down.

#Backoff

Immediate retries against a struggling dependency are a load test on a system that is already failing. Exponential backoff (1 s, 3 s, 9 s, 27 s) spaces attempts out; a cap keeps the last one reasonable; jitter (a random fraction added to each delay) stops every consumer that failed at the same moment from retrying at the same moment. Spring's @Backoff(random = true) adds it. The total budget should be longer than the outages you expect to ride through and shorter than the time the business can tolerate a stuck record: a few minutes for most transactional pipelines, hours for batch-like ones.

Retries should be attempted for transient failures only: timeouts, connection refused, 503, a database deadlock, a lock timeout. A 400, a validation error, a NullPointerException in the handler, a foreign key that does not exist: retrying these produces the same failure a minute later, three times, then a dead letter that arrived late. Classify exceptions, and send the permanent ones straight to the DLT.

#The poison pill

A record that cannot even be deserialised is the special case: the failure happens inside poll(), before any handler runs, so a handler's try/catch never sees it, and the consumer throws, restarts, polls the same offset, throws again. The partition is stuck and the consumer is in a crash loop. Spring's ErrorHandlingDeserializer wraps the real deserialiser, catches the failure, and delivers a record with a null value and the exception in a header, so the error handler can dead-letter it and the consumer moves past:

spring.kafka.consumer:
  key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
  value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
  properties:
    spring.deserializer.key.delegate.class: org.apache.kafka.common.serialization.StringDeserializer
    spring.deserializer.value.delegate.class: io.confluent.kafka.serializers.KafkaAvroDeserializer

Every consumer gets this. The poison pill is not rare; it is one bad producer deploy away.

#What a dead letter is

The DLT is not a bin. It is a queue of records that need a person, each with the evidence attached: the DeadLetterPublishingRecoverer adds headers for the original topic, partition and offset, the exception class and message, the stack trace, and the timestamp. The operational rules:

• Alert on it. Depth above zero for more than a few minutes pages someone. A DLT nobody watches is a silent data loss with extra steps.
• Diagnose from the headers, which say where the record came from and why it died, without touching the record's payload in a log.
• Fix the cause, which is either a bug in the consumer (deploy the fix) or bad data (correct it at the source, or decide it is discardable).
• Replay. A small tool that reads the DLT and re-publishes each record to its original topic, keeping the original key so it lands on the same partition. Spring Kafka's retry-topic machinery can be pointed at the DLT for this; a fifty-line CLI does it too. Replay after the fix is deployed, not before, or the records return to the DLT.
• Retain it long. The DLT's retention should be weeks, not the default seven days; the person who fixes the bug may need last month's dead letters.

A record that is replayed is, from the consumer's point of view, a duplicate delivery, which is one more reason the consumer is idempotent (the Offsets lesson).

#Under the hood: what the container does with an exception

The listener container's loop is poll(), then for each record invokeListener, then commit according to the ack mode. When the listener throws, the container hands the exception, the remaining records of the poll, and the consumer to the CommonErrorHandler. DefaultErrorHandler is a seek-based handler: for a retriable failure it does not call the listener again in a loop; it seeks the consumer back to the failed record's offset on that partition, records the attempt count and next due time for that (topic, partition, offset) in a FailedRecordTracker, and returns. The next poll() fetches the same record again; the tracker sees it is not yet due and the container pauses the partition (consumer.pause()) and keeps polling the others so the heartbeat and poll interval stay healthy, then resumes it when the backoff has elapsed. That is why blocking retries do not trip max.poll.interval.ms on their own the way a Thread.sleep in the listener would, and why the other partitions owned by that consumer keep flowing while one is stalled. When attempts are exhausted the handler calls the recoverer, which for DeadLetterPublishingRecoverer produces to the DLT with the original headers plus kafka_dlt-* diagnostics, and then the container commits past the record.

@RetryableTopic builds a different machine. For each retry level it registers a separate listener container on <topic>-retry-<delay> (or -retry-0, -retry-1 with the fixed-delay strategy) plus one on the DLT. A failure on the main topic publishes the record to the first retry topic with headers carrying the attempt number, the original topic and the due timestamp; the retry container reads it, compares the due time with the record's timestamp and, if it is early, pause()s the partition and later resumes it, sleeping in between with the consumer still polling. Each retry topic is one consumer group per listener per level, which is the "topics multiply" cost, and the pause-per-partition design means a retry topic with few partitions and many delayed records processes them strictly in due order per partition, not in parallel.

Deserialisation lives before all of this. poll() calls the deserialiser inside the fetcher; an exception there propagates out of poll() itself, the container's loop catches it, logs, and polls again at the same position, which is the crash loop. ErrorHandlingDeserializer catches inside the deserialiser, returns null for the value, and stores the exception serialised in a header (springDeserializerExceptionValue); the container detects the header, wraps it as a DeserializationException, and routes it to the error handler as if the listener had thrown, which is why the "not retryable" classification and the DLT path work for a poison pill exactly as for a handler failure.

#Walkthrough: the retry that made the outage worse

A fulfilment consumer called a warehouse API per record. The API slowed to 8 s per call during an incident, then failed outright.

@Bean DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> t) {
    return new DefaultErrorHandler(new DeadLetterPublishingRecoverer(t), new FixedBackOff(0L, 9L));  // 10 tries, no wait
}

1. Each record failed after an 8 s timeout and was retried nine more times immediately: 80 s per record, ten calls per record against a system that was already drowning, from every consumer in the group at once. The warehouse team saw request volume triple during their incident.
2. The container's seek-and-retry kept poll() alive, so no consumer was evicted; but each partition advanced one record per 80 s. Lag climbed for an hour. When the API recovered, the group had 40,000 records behind it and each still took its full retry sequence if the first call happened to fail.
3. After ten attempts each record went to the DLT with kafka_dlt-exception-message: Read timed out. Nobody was alerted on the DLT; 3,100 orders sat there over a weekend, each one a customer waiting for a shipment.
4. The rewrite: ExponentialBackOffWithMaxRetries(5) starting at 1 s with multiplier 3 and a 60 s cap, plus jitter; addNotRetryableExceptions for validation and 4xx responses; a RestClient timeout of 2 s so a slow call failed fast; a circuit breaker around the API call so that after N failures the handler failed immediately without calling out, letting the backoff do its job without load. And a DLT depth alert, paging after five minutes above zero.
5. The replay tool was written that weekend: read the DLT, strip the kafka_dlt-* headers, re-publish with the original key to the original topic. Idempotent handlers made the replay safe for the records that had partially succeeded.

A retry policy is a load policy. Immediate retries against a failing dependency are the load that keeps it failing, and a DLT with no alert is where the consequences go to be forgotten.

#Try it yourself

#Misconceptions

• "Blocking retry sleeps in the listener." DefaultErrorHandler seeks back and pauses the partition while polling continues; the poll interval is not consumed and other partitions flow.
• "Retry topics are a configuration detail." Each level is a topic and a consumer group per listener; they cost partitions, retention and monitoring, and they change ordering.
• "The DLT is where bad records go to die." It is a work queue for a person; without an alert and a replay path it is silent data loss.
• "A deserialisation error is just another exception." It happens inside poll(), before the handler; only ErrorHandlingDeserializer (on key and value) turns it into a handleable failure.
• "More retries mean more resilience." Against a failing dependency they are more load. Backoff, jitter, fail-fast timeouts and a circuit breaker are what resilience looks like.

#Going deeper

• Spring for Apache Kafka reference, "Handling Exceptions" (DefaultErrorHandler, DeadLetterPublishingRecoverer, ErrorHandlingDeserializer) and "Non-Blocking Retries".
• DefaultErrorHandler, FailedRecordTracker and SeekUtils in the spring-kafka source, for the seek-and-pause loop.
• Uber Engineering, "Building Reliable Reprocessing and Dead Letter Queues with Apache Kafka", the origin of the retry-topic pattern.
• Resilience4j documentation, circuit breaker and retry modules, for the client-side half.
• Marc Brooker, "Exponential Backoff And Jitter" (AWS Architecture Blog).