Secrets and API security

A secret is a value that gives whoever holds it your privileges. The engineering problem is not choosing a good one — it is that a secret has to reach a running process without being written down anywhere it can be read, copied, or kept.

Most of this lesson is about the places it gets written down anyway.

#The one that survives being fixed

Somebody commits a token, notices, and removes it in the next commit:

### the working tree is clean:
###   api.token=${API_TOKEN}
### git log --oneline:
###   7b29a3d Move the token to an environment variable
###   49c49b7 Add config

The repository looks correct. The commit message says the right thing. And then, from that same repository, with one command:

$ git log -p -S 'PLACEHOLDER-NOT-A-SECRET'
###   49c49b7 Add config
###   +api.token=PLACEHOLDER-NOT-A-SECRET-123

Removing a secret from a file does not remove it from history. It is in every clone, every fork, every CI cache, and every backup, and git log -S finds it in seconds — which is exactly what the bots scanning public repositories do, in under a minute, automatically.

So the rule is short and non-negotiable:

Prevention is cheap and worth having before you need it: .gitignore the file, a pre-commit secret scanner (gitleaks, trufflehog), and the same scan in CI so a bypassed hook is still caught. This project runs a secret scan over the diff on every edit and again at the end of each turn, which is the shape to copy.

#Environment variables, and what they are actually good for

The standard answer is "put it in an environment variable", and it is a genuine improvement over a file in the repository — the value is not in version control, and it differs per environment without a code change.

Their limits are worth knowing precisely, because "we use environment variables" is often where a team stops:

• They are visible to the process and its children. Anything you shell out to inherits them.
• They leak into diagnostics. A crash reporter, a debug endpoint, an /actuator/env, a stack dump, a docker inspect — several of these print the environment by default.
• They do not rotate. Changing one means restarting the process.
• They are often in a file anyway — a .env, a Compose file, a Kubernetes manifest — and that file ends up in a repository surprisingly often.

This project's own arrangement is a reasonable middle: secrets live in /etc/code10x.env, owned by root with mode 0600, referenced by the systemd unit's EnvironmentFile=. Not in the repository, not in the unit file, readable only by root, and there is exactly one definition of each value.

#A secret manager is what the next step looks like

Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault — all the same idea: the application asks for the secret at run time, authenticating as itself, instead of being handed it at start time.

What that buys, in order of how much it matters:

• Rotation without a deploy. The manager changes the value; the application picks it up on its next fetch or on a refresh.
• An audit trail. Who read which secret, when. You cannot ask that of an environment variable.
• Short-lived credentials. The best version: the manager issues a database password valid for an hour, and a leaked one expires on its own.
• Access control per secret, rather than per machine.

The cost is real and should be named: another dependency on the start-up path, and a bootstrap problem — the application needs a credential to authenticate to the secret manager. Cloud platforms solve that with workload identity, where the instance's own role is the credential and nothing is stored at all.

#Rotation is a property of the system, not an event

The question that decides whether you actually have rotation: can you change this secret right now, without downtime? If the answer is "we would need a maintenance window", you do not have rotation, and the next forced rotation — after an incident, at the worst moment — will be an outage on top of an incident.

What makes it possible is accepting two valid secrets at once during the change:

1. Add the new secret alongside the old; both are accepted.
2. Move every consumer to the new one.
3. Remove the old.

That is expand → migrate → contract again — the same shape as the schema migrations lesson and the API versioning lesson. It is the general answer to changing something that other things depend on.

And rotate on a schedule, not only after an incident. A rotation you have performed fifty times is routine; one you have never performed is a plan.

#TLS, and the "internal traffic" assumption

TLS everywhere, including between your own services. "It is internal" is an assumption about the network being trusted, and it is the assumption every lateral-movement attack is built on: one compromised pod on the same network reads everything.

• Terminate TLS at the edge — nginx, a load balancer — and use TLS inside as well. A service mesh will do the second for you.
• Verify certificates. Disabling verification to make a self-signed certificate work in development, and leaving that in a shared configuration, is a recurring way to turn TLS into theatre.
• mTLS is the step beyond: both sides present certificates, so the server knows which service is calling and not merely that somebody is. That gives you service identity without a shared token, and its real cost is certificate lifecycle — issuing, distributing and rotating them, which is why it usually arrives with a mesh rather than by hand.

#Abuse protection is part of security

An API with perfect authentication and no limits is still exploitable, just not by breaking in:

• Rate limits per principal, with Retry-After, as the rate-limiting lesson covers. A login endpoint without one is a brute-force target and a denial of service against your own hashing.
• Bound every payload. A maximum request body size, a maximum page size, a maximum array length. Without them a single request can exhaust memory.
• Cap anything unbounded a client can ask for — a date range that scans a year of data, a regular expression a client supplies, a nesting depth in JSON.
• Timeouts on every outbound call. A dependency that hangs becomes your outage, and a thread pool full of waiting requests is indistinguishable from being down.