Scale & cachingIntermediate
File upload and storage, where the bytes never touch your servers
Large files through an application server is the design most systems start with and the first one they replace. The better design sends bytes straight to storage and keeps only the record.
The brief
Users upload files — documents, images, videos up to several gigabytes — and share them with other users.
Uploads happen on unreliable mobile connections, and some files must be scanned before anyone can download them.
Requirements
Functional
- Upload files up to 5 GB, resuming after a dropped connection
- Download files, including by users the owner shared them with
- Scan uploaded files and block downloads of anything unsafe
- Delete files, and actually remove the bytes
Non-functional
- An upload interrupted at 90% must not start again from zero
- Nobody downloads a file they are not allowed to, including by guessing a URL
- Storage cost grows with use, so rarely read files should cost less
Back-of-envelope
Assume
- 10 million users, 1 million uploads a day
- Average file 20 MB; 1% of uploads are videos around 1 GB
- Each file is downloaded 5 times in its first week, and rarely afterwards
- Files are kept until the owner deletes them
Therefore
- Ingest: 1e6 × 20 MB = 20 TB a day ≈ 230 MB/s on average. Proxied through application servers, that is a fleet sized by bandwidth doing no application work.
- Storage: ≈7.3 PB a year of new data before replication. Most of it is not read after the first week — which is what makes tiering the biggest cost lever.
- Downloads: 5 × 20 TB = 100 TB a week of egress for new files. Egress is often priced higher than storage, so where downloads are served from matters more than where files are kept.
- A 1 GB upload at 5 Mbit/s takes about 27 minutes. The chance of a mobile connection surviving 27 minutes uninterrupted is low, which is why resumable upload is a requirement rather than a feature.
The average file size hides the distribution. The 1% of uploads that are videos are about half of all bytes, so a change in how videos are handled moves every number above.
The interface
POST /files { name, size, contentType } → 201 { fileId, uploadId, partSize, partUrls: [presigned PUT …] }The application creates a multipart upload in object storage and returns short-lived signed URLs, one per part. The client uploads parts directly to storage; the application never sees the bytes.
POST /files/{id}/complete { parts: [{ number, etag }] } → 202 { status: SCANNING }Completing assembles the parts in storage and starts the scan. 202, because the file exists but is not yet downloadable, and the client must show that state.
GET /files/{id}/download → 302 Location: short-lived signed URLAuthorization is checked here, in the application, every time. The redirect target expires in minutes, so a URL pasted into a chat stops working instead of becoming a permanent public link.
What is stored
files
status is what download checks; the storage key is random, never derived from the file name or owner, so knowing one file's key reveals nothing about another's.
file_id · owner_id · name · size · content_type · storage_key · status (UPLOADING | SCANNING | READY | BLOCKED | DELETED) · created_atstatus is what download checks; the storage key is random, never derived from the file name or owner, so knowing one file's key reveals nothing about another's.
shares
Serves the one question download asks — may this user read this file — as a primary-key lookup.
file_id · grantee_id · permission · PRIMARY KEY (file_id, grantee_id)Serves the one question download asks — may this user read this file — as a primary-key lookup.
object storage
Bytes live only here. Lifecycle rules move objects to cheaper storage classes after a period without reads, and abort multipart uploads that were never completed.
bucket/storage_key → bytes, with lifecycle rules by ageBytes live only here. Lifecycle rules move objects to cheaper storage classes after a period without reads, and abort multipart uploads that were never completed.
The design
The upload APICreates the file record and the multipart upload, hands out signed part URLs, and completes the upload. It handles kilobytes of metadata per file, however large the file.
Object storageReceives parts directly from clients, stores them durably, and assembles them on completion. It is the part of the system built to handle bandwidth.
The scannerTriggered by the completion event, reads the object, and moves the file to READY or BLOCKED. Until then, download refuses it.
CDNServes downloads close to users using signed URLs, so popular files in their first week do not all pay the full egress path from the storage region.
The decisions
Each of these could go the other way. The choice, the reason, and what it costs — a design that lists only what it chose teaches the choice; one that lists what it gave up teaches the judgement.
Upload through the application, or directly to storage?Chose — Directly to storage with presigned URLsBecause — Proxying 230 MB/s through application servers ties up threads and bandwidth for the duration of each upload, and a server restart kills every upload in progress on it.The application loses sight of the bytes: it cannot validate content during upload, and it must trust the client's declared size and type until the scanner has looked. Every check moves to after completion.
Deduplicate identical files across users by content hash?Chose — Not across usersBecause — Cross-user deduplication saves storage but leaks information: if uploading a file completes instantly, the uploader has learned that someone else already has that exact file.Identical popular files are stored many times. Deduplication within one user's own files keeps most of the saving without the leak.
Delete immediately, or soft-delete first?Chose — Soft-delete, then remove bytes after a short grace periodBecause — Users delete by mistake, and an undo window avoids restore-from-backup requests. A scheduled job removes the object after the window.A deleted file still exists for the grace period, which must be stated honestly — and for personal data, the removal job becomes something you must be able to prove ran.
What breaks first
In order. Each names what you would actually observe, and each fix carries its cost.
Small-file overheadSymptom — Request counts and storage request charges dominate the bill for workloads of many tiny files, while bandwidth is low.Fix — Single-request upload below a size threshold instead of multipart; batch metadata writes.Two upload paths in the client, with the threshold becoming another thing to test on flaky networks.
Scanning large filesSymptom — The SCANNING backlog grows after large uploads while the upload path is healthy; users see files stuck in 'processing' for minutes.Fix — Scan in parallel workers sized by bytes queued rather than files queued, with a size cap beyond which files are scanned in chunks.More scanner capacity, idle most of the day, and chunked scanning can miss threats that span chunk boundaries.
Egress for viral filesSymptom — One file shared publicly produces most of a day's egress; the bill spikes with no corresponding rise in users.Fix — CDN caching with a longer TTL for public files, and per-file download rate limits.Revoking access to a cached public file now requires a CDN purge, and a rate limit will occasionally stop legitimate demand.
When something fails
The connection drops at part 45 of 50The client asks which parts storage already has and uploads only the missing ones, using fresh signed URLs if the originals expired. The user loses at most one part's worth of progress.
An upload is started and never completedThe parts sit in storage, invisible and billed. The lifecycle rule that aborts incomplete multipart uploads after a few days is what stops this becoming a permanent, silent cost.
The scanner is downFiles remain in SCANNING and are not downloadable. Uploads keep working. That is the safe direction to fail: letting unscanned files through 'temporarily' is the exact outcome the scanner exists to prevent.
Scaling it
Each step is triggered by a number, not a feeling — and carries what it costs.
10× uploadsMove — Nothing in the upload path: object storage absorbs the bytes and the metadata API scales horizontally.None in architecture — but storage and egress costs grow ten times, and they were already the largest line.
Years of retained filesMove — Lifecycle tiering to infrequent-access and archive storage classes.Archived files take minutes to hours to retrieve, so the product must show 'restoring' for old files instead of an instant download.
Users in several regionsMove — Buckets per region, with a file stored in its owner's region.Sharing across regions pays cross-region transfer, and data-residency rules may forbid replicating some files at all.
What gets probed
The design is the easy half. These are where the conversation goes, and each has a defensible answer above.
- A 4 GB upload fails at 90%. What does the client do, and how much is sent again?
- Someone copies a download link into a public forum. What happens an hour later?
- Why is the storage key random instead of owner/filename?
- Where does the cost of this system actually go, and which single change reduces it most?