Code10x
Queues & eventsSenior

Chat, where order and 'delivered' have to mean something

Millions of long-lived connections, messages that must appear in the same order for everyone in a conversation, and a phone that goes offline in a tunnel mid-send.

The brief

Users send messages in one-to-one and group conversations, see them arrive in real time, and see messages they missed when they come back online.

Users have several devices, and every device must show the same conversation.

Requirements

Functional

  • Send and receive messages in real time in one-to-one and group conversations
  • Deliver messages sent while a device was offline when it reconnects
  • Show the same order of messages to every participant and every device
  • Show sent, delivered and read states

Non-functional

  • A message the sender saw as sent must never be lost
  • A message must not appear twice, even when the sender's phone retried it
  • Real-time delivery under a second for online recipients
  • Losing a connection server must not lose messages, only connections

Back-of-envelope

Assume

  • 50 million daily users, 20 million connected at peak
  • 40 messages sent per user per day, average 200 bytes
  • Groups capped at 500 members
  • A connection server holds 100,000 idle WebSocket connections

Therefore

  • Messages: 5e7 × 40 = 2e9 a day ≈ 23,000 a second on average, perhaps 70,000 a second at peak.
  • Storage: 2e9 × 200 bytes = 400 GB a day of message bodies, before replication and indexes — about 150 TB a year. Retention is a cost decision, not a default.
  • Connections: 2e7 / 1e5 = 200 connection servers at peak, before headroom. The fleet is sized by connections held, not by messages sent.
  • A message to a 500-member group is 500 deliveries. One busy group sending a message a second generates as much delivery work as 500 one-to-one conversations.

Assume 100,000 connections per server is optimistic for a server that also does TLS and heartbeats; halve it and the fleet doubles. Measure the real number before believing the third line.

The interface

WebSocket: send { conversationId, clientMessageId, body } → ack { clientMessageId, messageId, seq }clientMessageId is generated on the phone before the first attempt, so a retry after a lost ack carries the same id and the server returns the original seq instead of storing a second message.
WebSocket: server push { conversationId, messageId, seq, senderId, body }seq is assigned by the server, per conversation. Clients order by seq, never by timestamp — two phones' clocks disagree, and a conversation ordered by sender time reorders itself as messages arrive.
GET /conversations/{id}/messages?afterSeq=N&limit=100Reconnection sync. The device remembers the highest seq it has, and asks for everything after it. Gaps in seq are how a device knows it missed something, even while connected.

What is stored

messagespartition: conversation_id · clustering: seq · message_id · sender_id · client_message_id · body · sent_at
Every read is 'messages in this conversation after seq N', which is a range scan within one partition. A wide-column store fits that access pattern exactly; a query across conversations is never needed on the hot path.
conversation countersconversation_id · next_seq
Assigning seq needs a per-conversation atomic increment. It serialises writes within one conversation — which is fine, because a single conversation's write rate is human-sized.
connection registryuser_id → [{ deviceId, connectionServer }] with a TTL refreshed by heartbeat
Delivery needs to know which server holds each device's socket. The TTL means a crashed server's entries disappear on their own rather than routing messages into a dead socket forever.

The design

Connection serversHold WebSockets and nothing else of importance: they authenticate, forward sends inward and push deliveries outward. Stateless apart from the sockets, so losing one loses connections, not data.
The message serviceDeduplicates by clientMessageId, assigns seq, persists the message, then acks the sender. The ack is sent only after the write is durable — that ordering is what 'sent' means.
Fan-outConsumes persisted messages from a log partitioned by conversation id, looks up each participant's devices in the registry, and pushes through their connection servers. Offline devices are skipped; they will sync.
Push notificationsFor devices with no live connection, a mobile push that says a message exists. The push is a wake-up, not the delivery; the message arrives by sync when the app opens.

The decisions

Each of these could go the other way. The choice, the reason, and what it costs — a design that lists only what it chose teaches the choice; one that lists what it gave up teaches the judgement.

Order by timestamp or by a server-assigned sequence?Chose — A sequence per conversation, assigned by the serverBecause — Clocks on phones are wrong by seconds to minutes, and a server fleet's clocks drift too. A per-conversation counter gives a total order that every device agrees on and makes gaps detectable.Seq assignment is a single point of serialisation per conversation, and a message is not ordered until the server has seen it — so an offline user's queued message takes its place when it arrives, not when it was typed.
Fan out on write to every recipient's inbox, or store once per conversation?Chose — Store once per conversation; fan out only the real-time pushBecause — Copying every group message into 500 inboxes multiplies storage by group size. Stored once, a device syncs by reading its conversations after its last seq.A device opening the app must check every conversation it belongs to for new messages, so a 'what's new' index per user is still needed — a lighter form of fan-out, but not none.
What does 'delivered' mean?Chose — The recipient's device acknowledged receiptBecause — 'Pushed to the connection server' is not delivered: the socket may have died a moment earlier. Only the device can say it has the message.Every delivery produces an ack message back through the system, roughly doubling message traffic, and group 'delivered' state is per member — 500 states for one message in a large group, which is why many products stop showing it for groups.

What breaks first

In order. Each names what you would actually observe, and each fix carries its cost.

Reconnection stormsSymptom — A connection server restarts and 100,000 clients reconnect within seconds; authentication and sync endpoints spike, and the neighbouring servers that absorb them start dropping connections too.Fix — Clients reconnect with exponential backoff and random jitter; servers drain connections gradually on planned restarts.Users on the affected server see 'connecting…' for longer after an incident, by design.
Very large groupsSymptom — Fan-out lag for one partition climbs while others are idle, because one busy group's messages all hash to the same partition and each needs hundreds of pushes.Fix — Handle large groups on a separate path with more parallel fan-out workers, and cap real-time delivery state for them.Two code paths with different guarantees, and a group that crosses the size threshold changes behaviour.
The connection registrySymptom — Registry write load tracks connections times heartbeat frequency, not messages, and grows when heartbeats are made more frequent to detect dead sockets faster.Fix — Longer TTLs with heartbeats batched per server rather than per connection.Stale entries live longer, so more deliveries are attempted to devices that have already gone and must fall back to sync.

When something fails

The sender's phone loses signal after sending and before the ackThe phone retries on reconnect with the same clientMessageId. If the first attempt was stored, the server returns the original seq and nothing is duplicated; if it was not, this is the first write. Either way the sender sees one message.
A connection server crashesIts sockets close, its registry entries expire by TTL, and its clients reconnect elsewhere and sync from their last seq. Messages pushed to it in its last moments were already persisted, so they arrive by sync — late, not lost.
The fan-out consumer falls behindMessages are stored and acked but real-time delivery lags; recipients see them late, or on their next sync. Consumer lag on the fan-out log is the metric that shows it before users do.

Scaling it

Each step is triggered by a number, not a feeling — and carries what it costs.

10× usersMove — More connection servers, message partitions by conversation id, and more fan-out consumers — each tier scales on its own axis.Rebalancing the fan-out log's partitions pauses delivery for the affected conversations while consumers reassign.
Users in several regionsMove — Connection servers in each region, with a conversation's messages owned by one home region.A conversation between two continents pays cross-region latency for every message, and a home-region outage makes those conversations unavailable, not just slow.
Media messagesMove — Upload media to object storage directly and send only a reference through the chat path.A message can now arrive before its media is available, so the client needs a pending state, and deleting a message must also delete the object.

What gets probed

The design is the easy half. These are where the conversation goes, and each has a defensible answer above.

  • A phone sends a message, the ack is lost, and the phone retries. Show why the recipient sees it once.
  • Two people in a group send a message at the same moment. Why does every device show them in the same order?
  • A connection server dies. What is lost, what is late, and what is untouched?
  • What exactly does the 'delivered' tick promise, and who is allowed to set it?