A notification platform that survives a replay
Events in, emails and push out. The hard part is that Kafka will deliver some of them twice, and a customer who gets the same invoice email four times will say so publicly.
The business problem
Other services publish OrderPlaced, PaymentFailed, ShipmentDispatched. Your platform turns those into notifications across channels, with per-user preferences and a retry policy.
At-least-once delivery is the default and it is not a bug. Build for it.
What you will have at the end
- A consumer that is safe to replay
- A dead-letter topic with enough context to act on
- Retries with backoff that do not amplify an outage
- A schema you can evolve without stopping the producers
Milestones
Each one ends in something you can observe. Without that a milestone is a heading, and you have no way to know you finished.
Consume and commit deliberately
Manual acknowledgement. Auto-commit is what turns a crash into either lost work or repeated work.
done whenKilling the consumer mid-batch redelivers the batch instead of skipping it.
Idempotent handling
A processed_events table with a unique constraint on the event id, written in the same transaction as the effect.
done whenThe same event delivered twice sends one notification.
Retry and DLQ
Exponential backoff with jitter, a bounded attempt count, then a dead-letter topic carrying the original event and the last error.
done whenA permanently bad event lands in the DLQ instead of blocking the partition forever.
Preferences and fan-out
One event, several channels, per-user opt-outs applied before the send rather than after.
done whenA user who disabled email gets push only, and the email path is never called.
Schema evolution
Add a field to the producer without redeploying the consumer. Then remove one, and find out why that is the hard direction.
done whenAn old consumer still handles a new event, and you can explain which changes are safe.
Events
OrderPlaced, PaymentFailed, ShipmentDispatched in; notification.sent and notification.dlq out. Key by user id if ordering per user matters — and know that this is also how you create a hot partition.