Code10x
nginxhard5-8 years

A service's rate limiter reads `X-Forwarded-For` from every incoming request and blocks abusive IPs by that value. An attacker gets around it easily. What's the actual mistake, and what should the rate limiter read instead?

X-Forwarded-For is a request header, which means the client sending the original request controls its initial value completely — an attacker can simply set X-Forwarded-For: 1.2.3.4 themselves and the service, reading the header naively, believes whatever the attacker claimed. The header only becomes trustworthy at the point a proxy the service actually controls appends the real connecting client's address to it, which means the only part of the header's value that can be trusted is the entry the trusted proxy itself added — the last one, not the first. Trusting the first entry (which is exactly what the attacker controls) defeats the whole point of the check; trusting the last entry — the one nginx appended — is what a rate limiter actually needs to read, alongside X-Real-IP, which nginx sets to the single real connecting address and cannot be spoofed by the client because nginx overwrites it regardless of what the client sent.

The lesson behind it →
PreviousA team wants to send exactly 5% of production traffic to a canary release and watch its error rate before widening. Someone proposes adding a second DNS record for the service's hostname pointing at the canary, expecting roughly 5% of lookups to resolve there. Why doesn't this hold anywhere near 5%, and what would actually work?Next During a traffic spike, users report a mix of errors: some requests get an instant failure, others hang for exactly 60 seconds before failing. The on-call engineer sees both 502s and 504s in the nginx access log at the same time. Walk through what's different about these two failure modes and where you'd look for each.
More on nginx
nginx handles thousands of concurrent connections with a handful of worker processes — typically one per CPU core. Mechanically, how does one worker do that without one OS thread per connection, and why does a thread-per-connection server run out of resources before nginx does at the same connection count?During a traffic spike, users report a mix of errors: some requests get an instant failure, others hang for exactly 60 seconds before failing. The on-call engineer sees both 502s and 504s in the nginx access log at the same time. Walk through what's different about these two failure modes and where you'd look for each.