Object storage, CDNs and search

Three components that show up in most designs and rarely get thought about, which is how each becomes the thing that breaks. They share a shape: each takes a job away from your application server, and each introduces a copy of something.

#Object storage: never through your service

The wrong design is the obvious one — the browser uploads to your API, your API writes to storage. It costs you the bandwidth twice, ties up a request thread for the length of a large upload, and puts a file-sized object through your heap.

The right design is a pre-signed URL:

1. The client asks your API where to put a file.
2. Your API checks permission, generates a pre-signed URL valid for a few minutes, and returns it.
3. The client uploads straight to S3. Your service is not in the path.
4. S3 fires an event, or the client calls back, and your service records the metadata.

Your API stays fast and small; the storage does what it is good at. The same works in reverse for downloads — a pre-signed GET means you authorise the download without proxying it.

Three details that decide whether it works in practice:

• The metadata row is the system of record. The file is in S3, the row in Postgres says it exists, who owns it and its state. A file with no row is invisible; a row with no file is a broken link. Write the row on the callback, and reconcile.
• Uploads fail halfway. Multipart upload for anything large, and a lifecycle rule that deletes incomplete uploads after a day — otherwise you pay for them forever.
• Storage classes are the whole cost story. Standard, infrequent access, and archival differ by an order of magnitude. A lifecycle policy moving objects after 30 or 90 days is usually the single largest saving available in a storage bill.

#A CDN, and the thing it is actually for

A CDN caches your content at hundreds of locations near users. The obvious benefit is latency; the one that matters more is that the request never reaches you at all.

For a static asset served a million times, the CDN serves it a million times and your origin serves it once. That is not a percentage improvement — it is the difference between needing an autoscaling group and not.

What decides whether it works is cache-control and cache keys, which is the HTTP caching lesson applied at a different layer:

• Cache-Control: public, max-age=31536000 plus a content hash in the filename. app.a1b2c3.js never changes, so cache it for a year; a new build produces a new name. This is why every front-end build tool hashes filenames, and it removes invalidation from your life.
• Never cache a personalised response as public. The security lesson's incident shape: a CDN stores one user's response under a URL and serves it to everyone.
• The cache key matters. If your response varies by a header — language, currency — the CDN must be told to vary on it, or users get each other's variant.

Invalidation is slow and rate-limited on every CDN. Designing so that you never need it — immutable URLs — is the difference between a deploy and an incident.

#Search, and why it is never the system of record

Text search is a different problem from data retrieval, and a relational LIKE '%term%' cannot use an index, so it scans. That is fine at ten thousand rows and not at ten million.

An inverted index answers a different question: which documents contain this term, ranked by relevance. Tokenising, stemming, stop words, fuzzy matching and scoring are what you are buying, and none is expressible in SQL.

The architecture is always the same, and the direction is the important part:

Postgres (the truth)  ──indexed into──▶  Elasticsearch (derived, rebuildable)

One direction. Search is never written to directly and never holds anything that is not in the store of record, because it can be rebuilt from scratch and must be — after a bad mapping change, a lost cluster, or a bug in the indexer.

Which leaves the design decision that actually matters: how the index is fed.

The middle row is the usual answer, and it is the idempotency lesson again: the indexer must tolerate the same event twice, because at-least-once delivery is what you get.

#What they have in common

All three take a job off your service and leave you with a copy of something. That copy is the whole cost:

• The CDN has a copy of your asset, and it is stale until it expires.
• Search has a copy of your data, and it is behind by the indexing lag.
• Object storage has the file, and your database has a row claiming it exists.

So the questions to ask of each are the same two: how stale can this get, and who notices? And: what happens when the copy and the truth disagree? A design that has answers for those is a design that survives; one that assumes they agree will produce a bug report that reads "the search says it exists but the page 404s".