Producers and acknowledgements

producer.send(record) returns before the record has gone anywhere. It is appended to a buffer, batched with its neighbours, compressed, sent when a batch fills or a timer fires, acknowledged by a broker according to a setting most people never look at, and retried by a background thread that reports failure through a callback the calling code did not register. A producer configured carelessly loses messages silently and in order; configured well it loses none and duplicates none. This lesson is the settings that decide which, and the story of the message that was "sent" and never arrived.

#acks: what "sent" means

acks is the producer's definition of success:

acks=1 looks like a sensible middle and is the setting behind most "Kafka lost my message" reports: a write is acknowledged, the leader broker fails a second later, a follower without the record becomes leader, and the record is gone with no error anywhere. acks=all with min.insync.replicas=2 on a three-replica topic is the durable configuration, and its latency cost is one more network hop inside the cluster, which is small next to everything else in a request.

#The idempotent producer

A retry can duplicate: the broker appends the record, the acknowledgement is lost on the way back, the producer retries, the broker appends it again. enable.idempotence=true gives each producer a session id and each record a sequence number per partition, and the broker discards a record whose sequence it has already seen. Retries then cannot duplicate and cannot reorder, and the producer can keep up to five requests in flight per connection without breaking order.

It has been the default since Kafka 3.0, and turning it on implies acks=all, retries=Integer.MAX_VALUE and max.in.flight.requests.per.connection<=5. If a configuration explicitly sets acks=1 while idempotence is on, the producer refuses to start, which is the right outcome. Idempotence is per producer session and per partition; it does not deduplicate a record the application sends twice, and it does not survive a producer restart. Application-level deduplication is the consumer's job, in the Offsets lesson.

#Batching and linger

The producer does not send records one at a time. It groups records for the same partition into a batch, up to batch.size bytes (16 KB by default), and sends the batch when it is full or when linger.ms has passed since the first record joined it. linger.ms is 0 by default, so a lightly loaded producer sends tiny batches immediately. Setting it to 5–20 ms lets batches fill and multiplies throughput for a latency cost of that many milliseconds, which most pipelines never notice.

compression.type (lz4 or zstd; snappy and gzip exist) compresses whole batches, so it pays off with batching and costs almost nothing in CPU. Brokers store the compressed batch as is, and consumers decompress. For JSON or Avro payloads this is a several-fold reduction in network and disk.

Two buffers to know. buffer.memory (32 MB) is the total the producer will hold unsent; when it is full, send() blocks for up to max.block.ms (60 s) and then throws. A producer that suddenly blocks for a minute is a producer whose broker fell behind. And max.request.size / the broker's message.max.bytes (1 MB) cap a single record; a bigger one fails immediately with RecordTooLargeException, which is not retriable and must be handled by the application.

#Serialisation

The producer takes bytes. key.serializer and value.serializer turn objects into them: StringSerializer and ByteArraySerializer for the simple cases, KafkaAvroSerializer or a Protobuf serializer with a schema registry for anything with a shape that will evolve. JSON via Jackson is common and fine for a start, and the Schema evolution lesson explains what it costs once two teams share a topic. Whatever the format, the key's serialisation must be stable: the partition is computed from the serialised bytes, so a key serialiser that changes changes the ordering scope.

#Errors, callbacks and the message that never arrived

// 1. Fire and forget. Exceptions go nowhere. Never in application code.
producer.send(record);
 
// 2. Synchronous. Correct, and it serialises every send behind a network round trip.
producer.send(record).get();
 
// 3. Asynchronous with a callback. The usual right answer.
producer.send(record, (metadata, exception) -> {
    if (exception == null) {
        log.debug("orders p{} @{}", metadata.partition(), metadata.offset());
    } else if (exception instanceof RetriableException) {
        // the producer already retried until delivery.timeout.ms; this is final
        metrics.increment("kafka.send.failed");
        outbox.markFailed(record.key(), exception);
    } else {
        // RecordTooLarge, serialisation, authorisation: retrying will not help
        alert.raise("unsendable record", record.key(), exception);
    }
});

The message that was "sent" and never arrived is almost always form 1, combined with one of: the application exited before the buffer was flushed (producer.close() or flush() was never called, and the JVM ended with records in memory); acks=1 and a leader failover; a non-retriable error nobody looked at; or retries that ran out after delivery.timeout.ms (two minutes) with the callback missing. Every producer needs a callback that at least counts failures, and every service needs producer.close() on shutdown, which Spring's KafkaTemplate does for you.

Retriable errors (NotEnoughReplicasException when the ISR shrank below the minimum, LeaderNotAvailableException during an election, timeouts) are retried by the producer itself within delivery.timeout.ms; the callback fires only when that budget is spent. Non-retriable ones (RecordTooLargeException, SerializationException, AuthorizationException) fire immediately. The distinction tells you whether the fix is capacity or code.

#When a send must not be lost at all

A callback that logs a failure has still lost the record. When the record represents something that happened in a database transaction (an order was placed) and must reach Kafka, write it to an outbox table in the same transaction, and have a separate process publish from the outbox and mark rows sent. The database commit is the source of truth and Kafka is eventually consistent with it. The Microservices course's outbox lesson builds this; here it is enough to know that "send inside the transaction" and "send after the transaction" both lose messages in one failure mode each, and the outbox loses none.

#Under the hood: the accumulator, the sender thread, and the sequence number

send() is two hops inside the client. The calling thread serialises the key and value, asks the partitioner for a partition, and appends the record to the RecordAccumulator: a map from partition to a deque of batches, where the last batch is open if it has room. That append is the entire synchronous cost of send(), unless buffer.memory is exhausted, in which case it blocks for up to max.block.ms. A single background Sender thread then loops: it finds batches that are ready (full, or older than linger.ms, or whose partition has a batch already in flight that just completed), groups them by the broker that leads each partition, and writes one ProduceRequest per broker over that broker's connection. Up to max.in.flight.requests.per.connection (5) requests can be outstanding per broker; the response carries the base offset per partition or an error, and the Sender completes each record's future and callback on its own thread, which is why a callback that blocks stalls every send to every partition.

The idempotent producer is a small piece of state in that path. On start, initProducerId obtains a producer id (PID) and epoch from a broker. Every batch carries the PID and a sequence number per partition that increments per record; the broker keeps the last five sequences per PID per partition and rejects a batch whose sequence it has seen (DuplicateSequenceException, silently treated as success) or that skips ahead (OutOfOrderSequenceException, fatal, meaning something was lost in between). With retries and five in-flight requests, batch 3 can be retried after batch 4 succeeded, and the broker's sequence check is what keeps the log in order. The state is per producer session: a restart gets a new PID and the broker cannot recognise a resend from the old one, which is the boundary of the guarantee.

delivery.timeout.ms (120 s) is the total budget from send() returning to the callback: it covers time waiting in the accumulator, request.timeout.ms (30 s) per attempt, and retry.backoff.ms between attempts, and it must be at least linger.ms + request.timeout.ms or the producer refuses to start. Within it the Sender retries retriable errors indefinitely (retries is effectively unlimited under idempotence); when it expires the batch fails with a TimeoutException and the callback finally runs. A record can therefore take two minutes to fail, and a service that treats send() returning as "sent" has no idea.

#Walkthrough: the messages that vanished on every deploy

An audit service logged every admin action to Kafka. Once a week, an auditor found gaps of a few seconds, always at deploy times.

@Service
class AuditPublisher {
    private final KafkaProducer<String, AuditEvent> producer = new KafkaProducer<>(props);
    void publish(AuditEvent e) { producer.send(new ProducerRecord<>("audit", e.actor(), e)); }   // no callback
}
// application shutdown: SIGTERM → Spring context closes → JVM exits

1. send() appended to the accumulator and returned. With linger.ms=20 and a quiet service, records sat in open batches for up to 20 ms, and under load a few batches were in flight at any moment.
2. On deploy, Kubernetes sent SIGTERM; the Spring context closed; the KafkaProducer was a plain field, not a bean, so nothing called close(). The JVM exited with the accumulator holding whatever had not yet been sent and the Sender mid-request. Those records were gone. No exception: the callback that would have reported the failure was never registered, and there was no one to run it anyway.
3. acks=1 compounded it on the broker side: one deploy coincided with a leader change, and a batch the old leader had acknowledged was truncated when the follower took over. Two independent losses with the same symptom.
4. The fix was four lines. producer.close(Duration.ofSeconds(10)) in a @PreDestroy (or make the producer a bean and let Spring's KafkaTemplate do it), which flushes the accumulator and waits for in-flight acks. acks=all with enable.idempotence=true. A callback that counts failures into a metric with an alert. And delivery.timeout.ms lowered to 30 s so a failure surfaces while the incident is still happening.
5. The auditor's check was turned into a test: a Testcontainers broker, 1,000 sends, close(), and an assertion that the topic holds 1,000 records; it failed before the fix and passed after.

send() hands a record to a thread that outlives the call and not the process. What is in the buffer when the process exits was never sent, and nothing will say so.

#Try it yourself

#Misconceptions

• "send() sends." It appends to a buffer; a background thread sends later, and only close() or flush() guarantees the buffer is drained before exit.
• "The callback runs on my thread." It runs on the Sender thread; blocking in it stalls every partition's delivery.
• "Idempotence deduplicates my messages." It deduplicates the library's retries within one producer session. Application resends and restarts produce distinct records.
• "Retries are bounded by retries." Under idempotence they are bounded by delivery.timeout.ms; a failure can take two minutes to surface.
• "Compression fixes RecordTooLargeException." The size check runs on the uncompressed record before batching. Raise the limits together or use a claim check.

#Going deeper

• Kafka documentation, producer configs: delivery.timeout.ms, max.in.flight.requests.per.connection, enable.idempotence, and the "Producer" section of the design page.
• KIP-98 (exactly-once semantics: idempotent producer and transactions) and KIP-91 (delivery timeout).
• KafkaProducer, RecordAccumulator and Sender in the Apache Kafka client source, clients/src/main/java/org/apache/kafka/clients/producer/internals.
• Confluent, "Kafka Producer Internals: Preparing Event Data" and the producer tuning guide (linger.ms, batch.size, compression).
• Kafka: The Definitive Guide, chapter 3.