Queues and asynchronous design

A queue lets one part of a system say "this needs doing" without waiting for it to be done. That single change — separating the request from the work — is what lets a checkout finish while the confirmation email is still being written, lets a spike of traffic be absorbed rather than rejected, and lets a failing dependency slow one feature down instead of taking every feature down with it.

It is also a change in what "done" means, and most of the design work is about that.

#The checkout that waited for everything

A checkout does five things: save the order, reserve stock, send a confirmation email, record analytics, and notify the warehouse. Written synchronously, the customer waits for all five. With each step simulated by its typical duration:

synchronous, all healthy:   placed in  213 ms
asynchronous, all healthy:  placed in   24 ms

The asynchronous version does only what the customer must wait for — save the order and reserve the stock — and records an event saying the order was placed. The email, analytics and warehouse notification happen after, driven by that event. The numbers are simply the sums of the simulated steps, so the ratio belongs to this example; the structure is the point.

Now the email provider has an outage, and times out after two seconds:

synchronous, email down:    FAILED after 2023 ms (email provider timed out after 2 s)
asynchronous, email down:   placed in   23 ms
events waiting for the email consumer: 2

The synchronous checkout failed a sale because an email could not be sent. The asynchronous one placed the order in the same 23 ms and left two events waiting; the emails go out when the provider recovers.

Look again at the failed synchronous run, because it is worse than it appears. The order was saved and the stock was reserved before the email step threw. The customer was told the checkout failed, so they try again — and now there are two orders and twice the reserved stock. A synchronous chain of side effects that fails halfway leaves exactly this kind of mess, and the idempotency lesson in the distributed systems course is what cleans it up.

#What belongs on the synchronous path

The dividing question for each step: does the caller need the result to continue?

Everything in the second half is allowed to be late and must not be lost. That combination is exactly what a durable queue offers.

#The event must not be lost either

Moving work to a queue introduces a new failure: the order is saved, and the process crashes before the event is published. Or the event is published and the database transaction rolls back. Writing to a database and a broker are two separate operations, and no transaction spans both.

The standard answer is the transactional outbox: write the event as a row in an outbox table in the same database transaction as the order, and have a separate relay publish outbox rows to the broker. Either the order and its event both commit, or neither does. The relay may publish an event twice after a crash, so consumers must be idempotent — which, because every broker delivers at least once anyway, they had to be regardless.

#What a queue guarantees, and what it does not

"Exactly once" appears in broker documentation with careful conditions attached — typically within the broker's own read-process-write loop. A consumer that charges a card or sends an email is outside those conditions and must deduplicate itself.

#Backpressure and the backlog

A queue absorbs the difference between how fast work arrives and how fast it is done. That is its value during a spike and its danger during a sustained overload: the backlog grows, the age of the oldest message grows with it, and "late" turns into "too late to matter". The backpressure lesson measures an unbounded queue doing exactly that.

For a design, three decisions:

• What is the acceptable age of a message for each consumer? Email within minutes; a fraud check within seconds; analytics within hours.
• How does the system scale consumers, and what limits them — partitions, database connections, a downstream rate limit?
• What happens past the limit — shed the lowest-value work, pause producers, or alert and let it catch up?

#Failures: retry, then dead-letter

A consumer that fails on a message has three options, and a real system uses all three:

1. Retry, with backoff, for failures that are likely transient: a timeout, a 503.
2. Dead-letter the message after a limited number of attempts, so one poison message does not block everything behind it. A dead-letter queue is not a bin; it needs an alert, an owner and a way to replay.
3. Skip with a record, for messages that can never succeed — invalid data from an old producer — so they are visible without being retried forever.

The Kafka course's lesson on retries and dead-letter topics covers blocking and non-blocking retries in detail.

#When the workflow is really a state machine

As more steps move to asynchronous processing, a single business operation becomes a chain of events: order placed, payment captured, stock allocated, shipped. Implemented as consumers that each trigger the next, the flow exists nowhere in the code, and "where is order 1041 stuck?" is answered by reading five services' logs.

The better model is an explicit state machine, stored with the entity:

PLACED ──payment captured──▶ PAID ──stock allocated──▶ ALLOCATED ──shipped──▶ SHIPPED
   │                           │
   └──payment failed──▶ CANCELLED ◀──allocation failed (refund issued)

Each event handler performs one transition, as a conditional update that names the state it expects:

UPDATE orders SET status = 'PAID'
WHERE id = ? AND status = 'PLACED'

That one WHERE clause does three jobs. A duplicate event affects zero rows, so the transition is idempotent. An event that arrives out of order — "stock allocated" before "payment captured" — affects zero rows instead of skipping a state. And the order's status column is the answer to "where is it stuck", in one query.