Code10x
Queues & eventsIntermediate

Notifications, where the password reset must not wait behind the newsletter

Sending a message is easy. Sending the right one, once, on the right channel, while a marketing campaign of ten million is in the same pipes, is the design.

The brief

Other services ask this system to notify a user — an order shipped, a login code, a weekly digest — and it delivers by push, email or SMS through external providers.

Users control which notifications they receive on which channel.

Requirements

Functional

  • Accept a notification request for a user, with a type and data
  • Choose channels from the type and the user's preferences, and render a template
  • Deliver through external providers, with retries on failure
  • Record what was sent, for support and for 'did the user get it?'

Non-functional

  • A one-time login code must arrive within seconds, whatever else is queued
  • Duplicates are tolerable for a shipping update and unacceptable for a message that looks like a charge
  • An unsubscribe must take effect before the next send, not the next day
  • A provider outage must degrade one channel, not the whole system

Back-of-envelope

Assume

  • 20 million users, 5 transactional notifications per user per week
  • One marketing campaign a day to 10 million users
  • Login codes: 2 million a day, peaking at 5× the average rate
  • The SMS provider accepts 200 messages a second on this account

Therefore

  • Transactional: 1e8 a week ≈ 14 million a day ≈ 165 a second on average. Small.
  • A campaign: 10 million messages. Sent as fast as possible, it is thousands a second for about an hour — a single campaign is more traffic than a week of everything else in its first hour.
  • Login codes: 2e6 / 86,400 ≈ 23 a second, ≈ 115 a second at peak. That is the traffic with the deadline, and it is a tiny fraction of the total.
  • At 200 SMS a second, a campaign of 1 million SMS takes 5,000 seconds — about 83 minutes — of the entire SMS allowance. If login codes share that allowance, every code sent during those 83 minutes waits.

The last line is the design. Nothing about total volume is hard; what is hard is that the smallest, most urgent traffic shares a rate-limited provider with the largest, least urgent.

The interface

POST /notifications { userId, type, data, idempotencyKey } → 202 { notificationId }202, because delivery is asynchronous and callers must not wait on an email provider. The idempotency key comes from the caller's own event — 'order 1041 shipped' — so a caller that retries does not notify twice.
PUT /users/{id}/preferences { type, channel, enabled }Read on every send, not cached for hours, because an unsubscribe that takes effect tomorrow is a complaint today — and in some jurisdictions a legal problem.
GET /users/{id}/notifications?since=… → [{ type, channel, status, sentAt }]For support. 'Did they get the code?' is the most common question this system is asked, and the answer must come from a record rather than from searching provider dashboards.

What is stored

notificationsnotification_id · user_id · type · idempotency_key UNIQUE · created_at
The unique key is where duplicate requests stop. It is checked on insert, so two identical requests arriving at the same moment cannot both pass.
deliveriesdelivery_id · notification_id · channel · status · attempts · provider_message_id · INDEX (user_id, created_at)
One row per channel attempt, because a notification can succeed by push and fail by email. provider_message_id is what joins a provider's delivery-report webhook back to this row.
preferencesuser_id · type · channel · enabled
Small and read constantly, so it is cached — with the cache invalidated on write rather than expired on a timer, for the unsubscribe reason above.

The design

IntakeValidates the request, enforces the idempotency key, resolves channels from preferences, and publishes one message per channel onto a queue for its priority.
Priority queuesSeparate queues — critical (login codes, security alerts), transactional (orders), bulk (campaigns, digests) — each with its own consumers. A campaign can fill the bulk queue to the brim and the critical consumers never see it.
Channel workersOne pool per channel, rendering templates and calling providers. Each respects the provider's rate limit and reserves a share of it for the critical queue.
Retry and dead letterFailed deliveries go to delay queues with backoff; after the last attempt, to a dead-letter queue that someone looks at. A login code is not retried for an hour — after a minute it is useless, and it should fail over to another channel instead.

The decisions

Each of these could go the other way. The choice, the reason, and what it costs — a design that lists only what it chose teaches the choice; one that lists what it gave up teaches the judgement.

One queue with a priority field, or separate queues per priority?Chose — Separate queues and separate consumersBecause — A priority field in one queue still makes the urgent message wait behind whatever a consumer has already fetched, and a stuck bulk batch blocks the consumer for everyone. Separate queues make isolation a property of the topology.Capacity is split: critical consumers sit idle most of the day while the bulk queue has a backlog. That idle capacity is the price of the login code arriving in seconds.
Exactly-once delivery?Chose — At-least-once with deduplication before the provider callBecause — Exactly-once to an external SMS gateway does not exist; the provider can accept a message and time out before telling you. The achievable goal is to make duplicates rare and to suppress them where they are noticed.A small number of duplicates will reach users, typically after a provider timeout. For message types where a duplicate looks alarming, the template must be written so a second copy reads as a copy.
Retry a failed login code on the same channel?Chose — Fail over to another channel after a short deadlineBecause — The value of a login code decays in about a minute. Backoff retries that succeed after five minutes have delivered nothing useful.A user may receive the code twice, by SMS and by email, when the first channel was merely slow — and the second channel must be one the user has actually verified.

What breaks first

In order. Each names what you would actually observe, and each fix carries its cost.

The provider's rate limitSymptom — Queue depth climbs and provider 429 responses appear while workers are mostly idle — adding workers makes the 429s worse, not the queue shorter.Fix — A token bucket per provider account in front of the workers, with a reserved share for critical traffic; a second provider for overflow.Campaigns take longer to send, and a second provider means a second set of templates, credentials, delivery reports and bills.
Template rendering for a campaignSymptom — Worker CPU pinned during campaigns while provider calls are well under their limit — the workers are busy rendering, not sending.Fix — Render once per template and locale, and substitute per-user fields cheaply; render ahead of the send window for scheduled campaigns.Personalisation that goes beyond simple substitution loses the benefit, and pre-rendering needs somewhere to store ten million rendered bodies briefly.
Preference readsSymptom — Database read load spikes at the start of every campaign, one preference lookup per recipient.Fix — Resolve preferences in bulk when the campaign is expanded into recipients, not per message.A user who unsubscribes after expansion but before their message is sent still receives it, unless the worker re-checks — which puts part of the load back.

When something fails

The SMS provider is downSMS deliveries back up in their queue and push and email continue unaffected, because the channels do not share workers. Login codes fail over to email for users who have it; users with only SMS cannot log in, and that is a product decision to make before the outage.
A provider accepts a message and the call times outThe system cannot know whether it was sent. Retrying risks a duplicate; not retrying risks a lost message. The provider's delivery-report webhook, joined by provider_message_id, is what settles it afterwards.
A bug in a campaign template renders every message wrongWithout a kill switch, ten million wrong messages go out at the provider's rate. A per-campaign pause that stops workers taking bulk messages for it within seconds is the only thing that limits the damage to the first few thousand.

Scaling it

Each step is triggered by a number, not a feeling — and carries what it costs.

10× transactional volumeMove — More consumers on the transactional queue and partitioned delivery records; intake is stateless and scales horizontally.Delivery history spans partitions, so support's per-user view must query by user id, which must be the partition key.
Campaigns to 100 million usersMove — Spread sends over a window rather than as fast as possible, and negotiate higher provider limits.A 'send now' campaign arrives over hours, which marketing must plan around rather than discover.
Users in many time zonesMove — Schedule bulk sends per user's local time and honour quiet hours.A campaign becomes 24 smaller campaigns across a day, and 'when did it go out?' no longer has one answer.

What gets probed

The design is the easy half. These are where the conversation goes, and each has a defensible answer above.

  • A campaign to ten million users starts at 9:00. A user requests a login code at 9:05. Trace the code's path and say how long it waits.
  • A user unsubscribes and receives the next campaign anyway. Which component let that happen?
  • The provider times out on a send. Do you retry? What decides it, per message type?
  • Where does exactly-once break down, and what do you promise instead?