Code10x
Queues & eventsSenior

Food delivery, where three parties must agree and none of them is your software

An order is a promise made to a customer, a restaurant and a courier, each of whom can back out, go silent or be late. The system's job is to keep the three views consistent enough that nobody is cooking food nobody will collect.

The brief

Customers browse restaurants near them, place an order, pay, and watch it move from accepted to cooked to picked up to delivered.

Restaurants accept or reject orders and say when food is ready. Couriers are offered deliveries and accept them. The customer's estimate must be honest and the order must never be charged for and not delivered without a refund.

Requirements

Functional

  • List restaurants that deliver to the customer's location with their current menus and availability
  • Place an order and pay; the restaurant accepts or rejects within a short window
  • Assign a courier close to the restaurant near the time the food is ready
  • Show live status and an estimated arrival time to the customer
  • Cancel with the right money movement at every stage

Non-functional

  • A restaurant must never see an order the customer was not charged for, and a customer must never be charged for an order the restaurant did not accept — the consistency rule
  • Menu browsing is the bulk of the traffic and must be fast and cacheable
  • The system must keep working when the payment provider or the maps provider is slow
  • Every state change is auditable: disputes are about who did what when

Back-of-envelope

Assume

  • 2 million orders a day across all cities, peaking at 4× the average in the two dinner hours
  • 40 browsing sessions per order, each loading 10 restaurant cards and 2 full menus
  • 100,000 restaurants, each menu about 50 KB as JSON; menus change a few times a day
  • 200,000 couriers online at the dinner peak, reporting position every 5 seconds
  • An order's lifetime is about 45 minutes with 10 state changes

Therefore

  • Orders: 2M a day is 23 a second averaged; at the 4× dinner peak about 95 a second, each starting a saga of 10 steps. The write path is small. Every one of those writes is money or a promise, so it is the path where correctness costs are paid.
  • Browsing: 80 million sessions × 12 payloads = 960 million reads a day, 11,000 a second averaged and 44,000 at peak. That is 500 times the order rate and it is all cacheable by (restaurant, menu version); the read path is a cache problem, not a database one.
  • Menu storage: 100,000 × 50 KB = 5 GB. It fits in a cache entirely; the database is the source of truth and is read on a cache miss and on edits.
  • Courier positions: 200,000 / 5 = 40,000 writes a second, superseded in 5 seconds — the same shape as the ride-matching brief, and the same answer: an in-memory geospatial index, not rows.
  • State changes: 2M × 10 = 20 million events a day, 230 a second averaged, about 1,000 at peak. Each is an outbox row and a Kafka message; the topic is small and the consumers are many (customer app, courier app, restaurant tablet, analytics, the ETA model).

If the peak is 6× instead of 4× — a rainy Friday — browsing reads go to 66,000 a second and orders to 140 a second. The cache absorbs the first; the second is where the restaurant's acceptance window and the courier supply, not your servers, become the bottleneck.

The interface

GET /restaurants?lat&lng&cursor → 200 { restaurants: [ { id, name, etaMinutes, open, menuVersion } ], nextCursor }Served from a cache keyed by a geohash cell of the customer's location and a short TTL. etaMinutes is a rough estimate from distance and the restaurant's recent preparation times, not a promise; the promise is made at order time.
POST /orders { restaurantId, items, address, paymentMethodId, Idempotency-Key } → 201 { orderId, status: PENDING_PAYMENT }Validates the items against the current menu version (a stale price is rejected with 409 and the new menu), records the order, and starts the saga. The idempotency key is the customer's double-tap protection, from the idempotency brief.
POST /orders/{id}/accept | /reject { reason } (restaurant) → 200 | 409 already decided | 410 acceptance window closedThe restaurant's decision is a conditional state transition. 410 is the timeout path: an order not accepted within the window is cancelled and refunded without the restaurant's involvement.
POST /orders/{id}/cancel (customer) → 200 { refund: FULL | PARTIAL | NONE }The refund depends on the state: full before acceptance, partial once cooking has started (the restaurant is compensated for food made), none once the courier has collected. The state machine is the refund policy.
GET /orders/{id}/events → server-sent stream of { state, at, etaMinutes }The customer's live view. A stream rather than polling because 2 million customers refreshing every 10 seconds is 3,300 requests a second for data that changes ten times in 45 minutes.

What is stored

ordersorder_id · customer_id · restaurant_id · courier_id · state · amount_paise · payment_ref · menu_version · placed_at · version
The saga's state, with a version column so every transition is a conditional update (PENDING_ACCEPT → ACCEPTED only if still PENDING_ACCEPT), which is what makes a restaurant tapping accept twice, or accept racing the timeout, resolve to exactly one outcome.
order_events (append-only)order_id · seq · event (PLACED, PAID, ACCEPTED, COOKING, READY, COURIER_ASSIGNED, COLLECTED, DELIVERED, CANCELLED, REFUNDED) · actor · at · details
The audit trail and the source of the customer's timeline. Written in the same transaction as the state change, and it is also the outbox: the relay publishes each row and marks it. Disputes are answered from this table, never from the current state.
menusrestaurant_id · version · items JSON · published_at
Versioned, immutable per version: the cache key includes the version, an order records the version it was priced against, and a restaurant's edit publishes a new version rather than mutating the one 40 customers are looking at.
courier positions (in memory, per city)geospatial index: courier_id → (lat, lng) · status (AVAILABLE | OFFERED | DELIVERING) · updated_at
Same as ride matching: the value expires in seconds, the only query is spatial, and assignment is a conditional status update that prevents one courier being offered two deliveries.

The design

Catalogue and menu cacheRestaurants by location cell and menus by (restaurant, version), in a distributed cache with a TTL for the location lists and effectively infinite for a menu version. Handles the 44,000 reads a second; the database sees edits and misses.
Order orchestratorOwns the saga: reserve payment → ask the restaurant → on accept, capture payment and tell the kitchen → on READY, request a courier → track to DELIVERED. Every step is a state transition with a timeout and a compensation. Persists its state in the orders row, so a restart resumes any order mid-saga.
Restaurant service and tabletReceives new orders over a persistent connection to the restaurant's device, records accept/reject/ready, and tracks preparation times per restaurant, which feed the ETA.
Courier dispatchGiven a READY (or about-to-be-ready) order, finds available couriers near the restaurant, offers sequentially with a short timeout, and assigns with a conditional update — the ride-matching brief's matcher, with the pickup point fixed at the restaurant.
ETA serviceCombines the restaurant's recent preparation times, courier availability and travel time from the maps provider into an estimate, recalculated on each state change and published on the event stream. Degrades to distance-based estimates when the maps provider is slow.
Payments adapterAuthorise at order time, capture on acceptance, void on rejection or timeout, refund on cancellation — four idempotent calls to the provider, each keyed by order id and step, behind a circuit breaker.

The decisions

Each of these could go the other way. The choice, the reason, and what it costs — a design that lists only what it chose teaches the choice; one that lists what it gave up teaches the judgement.

Charge the customer before or after the restaurant accepts?Chose — Authorise before, capture afterBecause — Authorising first means a rejected card never reaches the restaurant, and capturing only on acceptance means a rejected order never charges the customer — both halves of the consistency rule, with the provider's authorise/capture split doing the work.An authorisation holds the customer's money for the acceptance window plus a void, which some banks show as a pending charge for days; and two provider calls per order instead of one, each of which can fail independently and needs its own compensation.
Orchestrate the order, or let the services react to each other's events?Chose — One orchestrator that owns the order's saga, with the other services as participantsBecause — Ten steps, five participants, timeouts at three of them and compensations at four: choreography would spread that state machine across five codebases and nobody could answer 'why is order 42 stuck'. The orchestrator persists its position in the orders row and is the one place the sequence exists.The orchestrator knows about every participant and every transition, and it is the component that grows business rules it should not own. Kept as a sequencer, with the rules (refund policy, acceptance window) as data it reads.
Assign the courier when the order is accepted, or when the food is nearly ready?Chose — When it is nearly ready, predicted from the restaurant's preparation timesBecause — Assigning at acceptance sends a courier to wait 20 minutes outside a kitchen — idle supply during the peak, and a courier who learns to decline that restaurant. Assigning at READY leaves hot food waiting for a courier who is 10 minutes away.The prediction is sometimes wrong in both directions: a courier arrives before the food or the food waits. The ETA the customer sees carries the same uncertainty, and the honest answer is a range, not a minute.
Menus as mutable rows or as immutable versions?Chose — Immutable versions, with the order recording which one it was priced atBecause — A price change while a customer is checking out is the most common dispute. With versions, the order is either priced at a version that existed (valid) or at a version the restaurant has replaced (409, show the new menu); there is no third case.Storage for every version (small — menus are kilobytes) and a cache that holds old versions until they expire, so a restaurant that edits ten times an hour churns the cache and the location list, which is cached separately with a short TTL, may name a version a moment before it changes.

What breaks first

In order. Each names what you would actually observe, and each fix carries its cost.

The restaurant's acceptance window at peakSymptom — Orders sit in PENDING_ACCEPT; the orchestrator's timeout counter climbs; cancellations and voids rise while the API is healthy. The kitchen is the bottleneck, and no server change helps.Fix — Throttle at the source: mark a restaurant BUSY in the catalogue when its pending count or preparation time crosses a threshold, so customers see longer ETAs or a closed sign before ordering.Lost orders for that restaurant by design, and a threshold that is a business decision per restaurant, not a constant.
The maps provider inside the ETASymptom — ETA recalculation latency rises with every state change; the event stream lags; the provider's error rate climbs and the circuit breaker opens.Fix — Degrade: distance-based ETA with a per-city speed model when the breaker is open, and cache provider results per (restaurant, cell) for a few minutes.Less accurate estimates during the outage, shown as a wider range; and a cache that can serve a route that no longer reflects a road closure.
The event topic's consumersSymptom — The customer app's stream is seconds behind the courier's; consumer lag on the analytics group climbs at the peak and never catches up.Fix — Partition by order id so one order's events stay ordered and consumers scale by partition; separate the analytics consumer group so it cannot slow the user-facing ones.Cross-order aggregates (how many orders are cooking at this restaurant) must be computed by a consumer that sees all partitions, which is a projection with its own lag.

When something fails

The payment provider times out on capture, after the restaurant acceptedThe orchestrator retries the capture with the same idempotency key on a backoff; the provider either completes the earlier attempt or runs it once. The kitchen is already cooking, so the order proceeds; if capture ultimately fails, the order completes and the customer is invoiced afterwards — the compensation for a captured-nothing is a debt record, not a cancelled meal.
The restaurant's tablet goes offline after acceptingThe order is ACCEPTED and cooking as far as anyone knows; READY never arrives. The orchestrator's READY timeout fires at 2× the restaurant's usual preparation time, dispatches a courier anyway, and flags the order for support. The courier confirms at the door whether food exists.
The orchestrator restarts with 5,000 orders mid-sagaEach order's state and version are in its row; on restart the orchestrator scans for orders in non-terminal states, re-arms their timeouts from the persisted timestamps, and continues. No order is repeated, because every step is a conditional transition on the version.
A courier accepts and then disappears with the foodCOLLECTED without DELIVERED past a timeout: the order is marked FAILED_DELIVERY, the customer is refunded in full, the restaurant is paid, and the courier's account is flagged. The money movement is the compensation; the system cannot recover the food.

Scaling it

Each step is triggered by a number, not a feeling — and carries what it costs.

10× citiesMove — Nothing in the order path changes: cities are independent partitions for couriers and restaurants, and the orders table shards by city or by order id with no cross-city queries.Support tooling that searches across cities needs a global index or a fan-out, and the ETA model needs per-city speed data before a new city launches.
Group orders and scheduled ordersMove — Scheduled: the saga starts at a future time with the authorisation deferred to then. Group: several customers' carts merge into one order with one payment split across authorisations.Scheduled orders make the catalogue's 'open now' answer wrong for them and need a menu version pinned hours ahead; group orders multiply the payment compensations by the number of payers.
Marketplace pressure: more orders than couriers at peakMove — Surge: raise delivery fees and courier pay by city cell when the available/pending ratio drops, and cap orders per restaurant by kitchen capacity.A pricing system with fairness and regulatory consequences, and a cap that turns away revenue on purpose; both are product decisions the system merely enforces.

What gets probed

The design is the easy half. These are where the conversation goes, and each has a defensible answer above.

  • The restaurant accepts and the capture fails. Who has the money, who has the food, and what does the system do?
  • A customer taps 'place order' twice on a slow connection. Show every layer that stops the second one.
  • Draw the state machine, and for each state say what a cancellation refunds and why.
  • The orchestrator is restarted at the dinner peak. What happens to the orders that were mid-saga?
  • Menu browsing is 500 times the order rate. Which single number in your design decides whether the database survives the peak?