Advanced

Apache Kafka

A log, and everything that follows from it.

Topics, partitions and the ordering they give, producers and acknowledgements, consumer groups, rebalancing, offsets and commits, retries and dead-letter topics, schema evolution, exactly-once semantics, and Kafka Streams.

8 lessons written2 modules~2h reading

After this course you can

Choose a partition key and explain the ordering and throughput consequences
Configure a producer for durability and a consumer for at-least-once without duplicates causing harm
Handle poison messages with retries and a dead-letter topic
Explain what exactly-once means in Kafka and what it does not cover

Curriculum

8 lessons · outlined lessons show their plan

The log

4/4 written · ~57 min

Topics, partitions and orderingThe log abstraction, partitions as the unit of parallelism, keys and ordering guarantees, and replication and ISR.15 min Producers and acknowledgementsacks, idempotent producers, batching and linger, and the message that was "sent" and never arrived.14 min Consumer groups and rebalancingGroup coordination, partition assignment, the rebalance storm, cooperative rebalancing, and why one consumer per partition is the ceiling.14 min Offsets and delivery guaranteesCommit strategies, at-most-once and at-least-once, the idempotent consumer, and the message processed twice after a crash.14 min

In production

4/4 written · ~54 min

Retries and dead-letter topicsBlocking versus non-blocking retries, retry topics with backoff, poison pills, and what to do with a dead letter.13 min Schema evolutionAvro, Protobuf and JSON Schema with a registry, compatibility modes, and the consumer that broke on a renamed field.13 min Kafka with SpringKafkaTemplate, @KafkaListener, concurrency, error handlers, and the tests that use an embedded broker.12 min Kafka StreamsStream and table, stateful operations, windows, joins, and exactly-once processing for the pipelines that transform events.16 min