Flaky tests

A flaky test is one that passes and fails on the same code. It is worse than having no test at all, and the reason is not the time it wastes — it is what it does to everybody's reading of a red build.

A suite with no test for something tells you nothing. A suite with a flaky test teaches people that red does not mean broken, and once that is learned it applies to every other test in the suite, including the ones that are telling the truth.

#One from this project, last week

The deploy stopped here:

✗ fix-the-race-condition — THE STARTER PASSES EVERY CASE. The fixture does not test what the problem asks for.
  deploy failed before the switch — removing /srv/code10x/releases/20260912-204250

That guard checks that a practice problem's broken starter code actually fails its own tests — otherwise the problem is not testing what it claims. The starter's bug is a data race on count++.

It had passed five deploys that day. Run locally immediately afterwards, it passed six times out of six on the same image.

Which is the exact shape of the problem: the failure carried no information about the change being deployed, and the temptation — run it again, it will probably pass — is precisely how a team learns to ignore red.

#Diagnose, do not retry

Re-running is not a diagnosis, and "it passed the second time" is not a result. What the investigation established:

1. The change could not be the cause. The deploy carried four markdown lessons. Nothing Java.
2. It was not reproducible locally — 6 of 6. So the difference was the environment, not the code.
3. A mechanism that explains both. A lost update needs a thread preempted between the read and the write of count++. On a laptop with real parallelism that happens constantly. On a small VM the pool's threads do not truly overlap, and a JIT-compiled loop over a non-volatile field can be hoisted to a single read-add-write — at which point the broken code produces the correct total.

Only with a mechanism is a fix more than a guess. The general shape:

• What is different between the runs that pass and the runs that fail? Machine, core count, time of day, order, load.
• Can you make it fail on demand? Run it a hundred times, on one core, under load. A flake you can reproduce is a bug you can fix.
• Is the test wrong, or the code? Sometimes the flake is the only honest thing in the suite, and there is a real race in production.

#The usual causes

Nearly every flaky test is one of these:

• Time. LocalDate.now() in a test that fails on the last day of the month, at midnight, or in another timezone. Inject a Clock; never read the real one in a test.
• Order dependence. Test A leaves a row behind, test B assumes an empty table. It passes until the runner reorders or parallelises. Run your suite in random order deliberately — JUnit 5 has @TestMethodOrder(Random.class) — and find these before CI does.
• Shared state. A static field, a cache, a singleton with memory. Two tests pass alone and fail together.
• Concurrency. Both directions: a race that does not manifest (the one above), and a race that does. The second is a real bug.
• Thread.sleep. The most common flake there is. A sleep is a bet on how long something takes; CI is slower than your laptop and the bet loses. Await a condition — Awaitility, a latch, a poll with a timeout — never a duration.
• The network. Any test that reaches something real is one outage away from red. That is what Testcontainers and WireMock exist to prevent.
• Unordered results. select ... with no order by returns rows in whatever order it likes, and a test asserting on the first row passes until the plan changes. This is the pagination lesson's warning arriving as a test failure.

#Quarantine is a decision, not a drawer

When one cannot be fixed today, the options are:

1. Fix it. Always first.
2. Quarantine it — tagged, excluded from the blocking run, still executed and reported. With an owner and a date.
3. Delete it. A legitimate choice. A test nobody trusts and nobody will fix is costing attention and giving nothing.

The failure mode is @Disabled with no comment, which is deletion pretending to be quarantine — the test is gone and everyone believes it is covered.

#Retrying, and doing it honestly

Automatic retries are the standard response and usually the wrong one, because they hide the number above. A suite with retries on looks green while getting steadily worse.

There is one case where retrying is correct, and the distinction is worth stating precisely: when the property being asserted is genuinely probabilistic.

The race fixture is that case. It asserts this broken code fails its tests, and a data race may legitimately not manifest in a given run. So the guard now runs the starter up to three times before declaring the fixture broken.

What makes that honest rather than a cover-up:

• The assertion did not weaken. A fixture that genuinely fails to discriminate passes all three attempts and still fails the check — verified by pointing a starter at its own reference solution, which reported THE STARTER PASSES EVERY CASE, in 3 attempt(s).
• The retries are only spent on the path that was about to fail, so a green run costs nothing.
• It is written down — in the guard, in the fixture, and in the project's testing document, with the reason, so the next person does not remove it as clutter.

A retry you can explain in those terms is engineering. A retry added because the build was annoying is how a suite stops meaning anything.