Scale & cachingIntermediate
A feed, and the fan-out decision underneath it
Read-time or write-time is the only question here, and the honest answer is that a real system does both and switches between them per account.
The brief
Users follow other users. A user's feed shows recent posts from everyone they follow, newest first.
Some accounts have a handful of followers. A few have millions.
Requirements
Functional
- Post something
- Read your feed, paginated
- Follow and unfollow
Non-functional
- Opening the app is a feed read, so feed reads vastly outnumber everything else
- A post appearing a few seconds late is acceptable; a feed that fails to load is not
- The follower distribution is not a bell curve, and any design that assumes it is will fall over on the tail
Back-of-envelope
Assume
- 10 million daily active users
- Each opens the feed 10 times a day, and posts twice
- The average account has 200 followers, but the largest have 10 million
Therefore
- Feed reads: 1e7 × 10 / 86,400 ≈ 1,160 per second.
- Posts: 1e7 × 2 / 86,400 ≈ 230 per second.
- Fan-out on write, average case: 230 × 200 ≈ 46,000 feed inserts per second. Large, steady, and entirely manageable with a queue.
- Fan-out on write, worst case: ONE post by a 10-million-follower account is 10 million inserts. At 46,000/s of steady capacity that single post is roughly three and a half minutes of the entire system's write budget.
The average told you the system is fine. The tail told you the design. This is why the follower distribution is an assumption in its own right rather than a single average — writing down only the mean would have hidden the only interesting number here.
The interface
GET /feed?cursor=<opaque>&limit=20A cursor, never an offset. Posts arrive while a user reads, so page 2 by offset shows them items they already saw — the bug is invisible in testing because nobody posts during a test.
POST /posts { body } → 201 { id }Returns as soon as the post is durable, not when fan-out finishes. Waiting for ten million inserts before returning a 201 is how one celebrity makes the compose box look broken.
PUT / DELETE /follows/{userId}Cheap to call and expensive to honour: a follow changes which of two read paths that user's feed comes from. The API hides that, and the design must not.
What is stored
posts
A time-sortable id means merging feeds is merging already-sorted lists. With a random id you would carry a timestamp alongside every entry and sort at read time, on the hottest path in the system.
id (time-sortable) · authorId · body · createdAtA time-sortable id means merging feeds is merging already-sorted lists. With a random id you would carry a timestamp alongside every entry and sort at read time, on the hottest path in the system.
feeds
The read is a range scan from the head. Capped because nobody scrolls to post 5,000, and storing it is paying rent on something no one reads.
userId → capped list of post ids, newest firstThe read is a range scan from the head. Capped because nobody scrolls to post 5,000, and storing it is paying rent on something no one reads.
follows, indexed BOTH ways
Push needs followers-of-an-author; pull needs followees-of-a-reader. The hybrid uses both, so both indexes exist, and that duplication is a real storage cost rather than an oversight.
followerId → followees, and followeeId → followersPush needs followers-of-an-author; pull needs followees-of-a-reader. The hybrid uses both, so both indexes exist, and that duplication is a real storage cost rather than an oversight.
The design
Fan-out on write (push)On posting, append the post id to each follower's precomputed feed list. Reads become one range scan. Writes become expensive in proportion to follower count.
Fan-out on read (pull)On reading, gather recent posts from everyone the user follows and merge. Writes are trivial. Reads pay for the merge, every time.
The hybridPush for ordinary accounts, pull for the handful with enormous followings, merged at read time. This is the design, and the threshold between the two is a number you choose and monitor.
The feed storePer-user list of post ids, capped. Nobody scrolls to post 5,000; storing it is paying to keep something no one reads.
The decisions
Each of these could go the other way. The choice, the reason, and what it costs — a design that lists only what it chose teaches the choice; one that lists what it gave up teaches the judgement.
Push, pull, or both?Chose — Both, split by follower countBecause — Pure push cannot survive a celebrity post; pure pull makes every feed read a fan-in across hundreds of accounts. The split isolates the expensive case to the few accounts that cause it.Two code paths and a merge, plus a threshold that is a tuning knob nobody can set correctly in advance. It also means a user following one celebrity and 200 friends reads from both paths on every load.
Store post ids or whole posts in the feed?Chose — Ids, hydrated on readBecause — A post edited or deleted after fan-out would otherwise need rewriting in millions of lists. Ids make the feed a pointer list and the post the single source of truth.Every feed read becomes a second lookup for the bodies. That lookup is a batch fetch of items that are heavily cached, which is the cheap kind of extra round trip.
Strict chronological order, or ranked?Chose — Chronological, until asked otherwiseBecause — Ranking is a different system — features, a model, training data, and a feedback loop — bolted onto this one. Designing it in from the start buries the feed problem under a machine-learning problem.Chronological feeds are worse for engagement at scale, which is usually why ranking arrives. Say that out loud rather than pretending the choice is purely technical.
What breaks first
In order. Each names what you would actually observe, and each fix carries its cost.
One post by a ten-million-follower accountSymptom — Three and a half minutes of the entire fan-out budget, spent on one post. Everyone else's post queues behind it, so the whole product feels broken because one account is popular.Fix — The hybrid handles it by not fanning out at all above a threshold; below that, large fan-outs go to their own queue so ordinary posts are never behind them.Two queues, and a threshold nobody can set correctly in advance. Set it too high and ordinary posts still starve behind a merely-large one.
The feed store, in write volumeSymptom — 46,000 inserts a second against 230 posts a second — the feed store absorbs two hundred times the writes of the post store. It is the first thing that needs sharding, and it is not where people look.Fix — Shard feeds by userId; each feed is written and read as one key, so the shard key is obvious and never needs a cross-shard query.Fan-out now writes to every shard for a popular author, so one post touches the whole cluster instead of one machine.
A user who follows several thousand accountsSymptom — On the pull path the cost grows with how many you FOLLOW, not with how many follow you — the opposite axis to the celebrity problem, and it hits a different, quieter set of users.Fix — Cap the merge at the most recently active followees, and keep those users mostly on the push path.Their feed is no longer complete, and nothing in the product tells them so.
When something fails
A fan-out worker dies halfway through a postSome followers have the post and some never will, and nothing in the system is aware. Fan-out has to resume from a durable cursor and be idempotent per (postId, followerId), or a partial delivery is permanent and invisible.
The feed store is unavailableFall back to the pull path for everyone. It is slow and it is expensive, but the app opens — which is the requirement that said a late post is survivable and a feed that will not load is not.
A user unfollows while a post is being fanned outThe post lands in a feed it no longer belongs in. Filter at read time and treat the feed list as a cache rather than the truth; trying to chase the write is a race you cannot win.
Scaling it
Each step is triggered by a number, not a feeling — and carries what it costs.
10× daily active usersMove — Shard feeds by userId and cache the first page per user, which is most of what anyone reads.That cache has to be invalidated or appended to on every new post from anyone they follow, which puts write traffic back onto the read path you just optimised.
More accounts crossing the celebrity thresholdMove — Make the threshold a percentile of the live follower distribution rather than a constant.An account can cross mid-session, so a reader may hit both paths for the same author and see the post twice unless the merge de-duplicates. That is a bug you inherit from your own elasticity.
Users in several regionsMove — Regional feed stores, written by regional fan-out workers.A cross-region follow makes one post a cross-region write, and two readers in different regions can see the same two posts in different orders. Decide whether that is acceptable before it is discovered.
What gets probed
The design is the easy half. These are where the conversation goes, and each has a defensible answer above.
- An account with 10 million followers posts. Walk through the next five minutes.
- A user follows someone new. Does their back catalogue appear in the feed, and what did that cost?
- Where is the threshold between push and pull, and how would you know it is wrong?
- A post is deleted after fan-out. What does a follower's feed show?