Topics, partitions and ordering

Kafka is easier to understand than its reputation suggests once you hold on to one idea: a topic is an append-only log, split into partitions, and a partition is the only thing Kafka orders. Every guarantee it makes, every limit it has, and every design decision you will take about it (the key, the partition count, the replication factor) follows from that. This lesson is the log, what a partition is and is not, how a key decides where a record lands, and how replication keeps the log alive when a broker dies.

#The log

A partition is a file on a broker's disk to which records are appended and from which they are read by position. That position is the offset: a monotonically increasing integer, assigned when the record is appended, never reused. Consumers do not delete anything; they remember an offset. Two consumers reading the same partition at different speeds read the same records in the same order, and a consumer that crashes resumes from the offset it last recorded. This is why Kafka can be both a queue and a replayable history, and why it is fast: sequential writes to disk, sequential reads, and the operating system's page cache doing most of the serving.

Records are retained by policy, not by consumption. retention.ms (seven days by default) or retention.bytes decides when old segments of the log are deleted, whether anyone read them or not. A consumer that is down for eight days on a seven-day topic has lost data, and nothing warns it except its own offset being older than the earliest one available, at which point auto.offset.reset decides whether it starts from the beginning or the end.

#Partitions

A topic with one partition is one log on one broker, read by at most one consumer in a group. A topic with twelve is twelve logs, spread across brokers, read by up to twelve consumers in parallel. Partitions are the unit of parallelism, of placement (which broker holds what) and of ordering:

Records in one partition are delivered in the order they were appended. Records in different partitions have no ordering relationship at all.

That second sentence is the one to design around. "All events for order 42, in order" is achievable; "all events, in order" is a one-partition topic, and one partition is one consumer's throughput.

The partition count is chosen at creation and can be raised later but never lowered, and raising it changes which partition a key maps to, which breaks per-key ordering for the transition. Pick it from the target throughput divided by what one consumer can process, with headroom, and from how many consumers you will ever want in the group. Too many partitions cost broker memory, file handles and longer rebalances; a few thousand per broker is a practical ceiling.

#Keys and ordering

// With a key: hash(key) mod partitions, the same key always to the same partition
producer.send(new ProducerRecord<>("orders", order.getId(), payload));
 
// Without a key: the sticky partitioner fills a batch on one partition, then moves on
producer.send(new ProducerRecord<>("audit-log", null, payload));

The default partitioner hashes the key (murmur2) and takes it modulo the partition count, so every record with the same key lands in the same partition and is therefore ordered relative to every other record with that key. The key is the ordering scope: the order id, the account id, the device id, whatever entity's events must be seen in sequence. Choose it for ordering, and check it for distribution: a key with a few very hot values (a country field where one country is 60% of traffic) puts 60% of the load on one partition and one consumer, however many partitions exist.

A null key goes wherever the partitioner likes, and since Kafka 2.4 that is "sticky": one partition until the current batch is full, then another. Good for throughput, no ordering. A custom partitioner is possible and rarely worth it.

#Replication and the ISR

Each partition has a leader on one broker and followers on others, the count set by the replication factor (three is the production norm). Producers write to the leader; followers fetch from it and append to their own copy. The followers that are caught up form the in-sync replica set, the ISR. A record is committed when every replica in the ISR has it, and only committed records are visible to consumers.

When a leader dies, the controller picks a new leader from the ISR, and no committed record is lost, because every ISR member had it. Two settings decide how strong that promise is:

• min.insync.replicas (topic or broker): the smallest ISR that will accept a write with acks=all. With replication factor 3 and this set to 2, a write succeeds only when the leader and at least one follower have it, and the topic keeps accepting writes with one broker down. Set to 1, "committed" means "on the leader", and a leader crash loses data.
• unclean.leader.election.enable: whether a follower that fell out of the ISR may become leader when no ISR member is available. false (the default) chooses unavailability over data loss. Leave it.

The Producers lesson pairs these with acks. The trio to remember is replication factor 3, min.insync.replicas 2, acks=all: one broker can fail with no loss and no downtime.

Kafka 4.0 removed ZooKeeper entirely; cluster metadata and controller election run inside Kafka on the KRaft protocol. Nothing above changes, but a cluster you inherit may still be on the old arrangement, and the migration is a real project.

#Retention and compaction

cleanup.policy=delete is the default: whole segments expire by age or size. cleanup.policy=compact is different in kind. A compacted topic keeps the latest record for each key forever, deleting older records with the same key in the background. It turns a log into a table: the current state of every entity, replayable from the beginning to rebuild a cache or a database. A record with a null value is a tombstone and deletes the key, after delete.retention.ms so that consumers in flight see it. Compaction is what Kafka's own __consumer_offsets topic uses, what Kafka Streams uses for its changelogs, and what a "current customer profile" topic should use. It compacts eventually, not immediately, so a consumer reading from the start may still see a few superseded records and must be written to take the last.

#Under the hood: segments, the page cache, and the high watermark

A partition on disk is a directory of segments: a .log file of records, an .index mapping relative offsets to byte positions (sparse, one entry per log.index.interval.bytes), and a .timeindex mapping timestamps to offsets. The active segment is the one being appended to; when it reaches segment.bytes (1 GB) or segment.ms (seven days), it is closed and a new one opened. Retention and compaction operate on closed segments only, which is why a low-traffic topic keeps data past retention.ms (its active segment never rolls) and why a compacted topic still shows stale values for a while. A read at offset N is a binary search in the index for the nearest entry, then a short scan forward in the log; the broker hands the bytes to the socket with sendfile, never copying them into JVM heap, and because writes and reads are sequential the OS page cache serves most of it. That is the reason Kafka's throughput is bounded by disk and network bandwidth rather than CPU, and why a broker with a small heap and a large amount of free RAM is the right shape.

Records are not written one by one. The producer ships a record batch per partition: a header with the base offset, a producer id and sequence for idempotence, a compression codec, and the records as varint-delta-encoded entries; the broker assigns offsets to the batch and appends it as one unit, compressed as received. Consumers fetch batches and decompress locally. Since the 0.11 format the batch also carries a transaction marker bit, which is what read_committed consumers use to skip aborted work.

Committed has a precise meaning. Every replica tracks its log end offset (LEO), the next offset it would append. The leader's high watermark is the minimum LEO across the ISR, and a consumer's fetch is answered only up to it, so a record is invisible until every in-sync replica has it. A follower stays in the ISR while it is within replica.lag.time.max.ms (30 s) of the leader; fall behind and the controller shrinks the ISR, which is what lets the high watermark advance without the slow replica, and what min.insync.replicas guards against going too far. On leader failover the new leader's high watermark becomes the truth and any follower with records beyond it truncates them, which is why only committed records survive and why acks=1 writes, which were past the watermark on the old leader alone, vanish. Followers fetch with the same protocol consumers use, and since Kafka 2.4 a consumer may read from the nearest follower (replica.selector.class, client.rack) to save cross-zone bandwidth, still bounded by the high watermark.

#Walkthrough: the hot partition that looked like a slow consumer

A notifications pipeline had twelve partitions, twelve consumers, and lag that climbed on one partition every evening while the other eleven sat at zero.

producer.send(new ProducerRecord<>("notifications",
        event.tenantId(),                 // the key
        event));

1. The key was the tenant id, chosen so each tenant's notifications stayed ordered. One tenant, a marketplace, generated 55% of all events. murmur2(tenantId) mod 12 put all of them on partition 7.
2. The consumer on partition 7 processed at the same rate as the others; it simply received seven times the records. Scaling the group to 24 consumers changed nothing: twelve of them were idle and partition 7 still had one reader. Doubling the partition count to 24 changed nothing either, since the hot key still hashed to exactly one of them.
3. The team read the broker metric kafka.log:type=Log,name=Size per partition and the consumer's records-lag per partition, and the imbalance was obvious in a way the group-level lag graph had hidden.
4. The fix was to move the ordering scope down a level. Did notifications for a tenant need a total order, or only per recipient? Per recipient. The key became tenantId + ":" + userId, which spread the marketplace across all partitions while keeping each user's notifications in sequence. Where a tenant-wide order was genuinely needed (a "tenant suspended" event), a separate low-volume topic keyed by tenant carried it.
5. The lesson recorded for the next topic: choose the key from the smallest entity whose events must be ordered, then check the key's distribution with a histogram before creating the topic, and treat per-partition lag, not group lag, as the alert.

Parallelism is per partition and a key is a single partition. A hot key is a single consumer's throughput, and no amount of scaling around it helps.

#Try it yourself

#Misconceptions

• "A partition is a queue." It is a log that nothing consumes from; readers keep positions and data leaves only by retention or compaction.
• "A write acknowledged by the leader is durable." Only a write past the high watermark, on every ISR member, survives a failover. acks=1 writes sit above the watermark and can be truncated.
• "More partitions fix a hot key." A key maps to one partition regardless of the count. Only a finer key spreads it.
• "Retention deletes records older than the limit." It deletes whole closed segments; the active segment and low-traffic topics keep data far longer.
• "Compacted topics hold exactly one record per key." Eventually, in closed segments; readers must take the last value per key and expect stale ones in between.

#Going deeper

• Kafka documentation, "Design" section: persistence, the log, replication and the ISR, and "Message Format" for record batches.
• KIP-101 (leader epochs and truncation), the change that made the high-watermark rules safe across failovers.
• Jay Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction".
• Kafka documentation, topic configuration: segment.ms, min.insync.replicas, cleanup.policy, delete.retention.ms.
• Gwen Shapira et al., Kafka: The Definitive Guide (2nd ed.), chapters 6 (internals) and 7 (reliable data delivery).