Code10x
Linux signalsmedium3-5 years

A systemd unit's `ExecStart` runs `/bin/sh -c "java -jar app.jar"`. `systemctl stop` is issued, and the JVM keeps running past `TimeoutStopSec` until systemd kills it with `SIGKILL`. What actually happened to the `SIGTERM`, and why does `exec java -jar app.jar` fix it?

sh -c "java -jar app.jar" makes sh the process systemd is tracking as the service's main PID; sh in turn forks java as a separate child process to actually run the command. SIGTERM from systemctl stop is delivered to the PID systemd tracks — sh — not automatically to every process it happened to spawn. sh's default handling of SIGTERM is to terminate itself, and that does not necessarily forward the signal to the still-running java child, so the JVM's own SIGTERM handler — the one that triggers Spring Boot's graceful shutdown and any registered shutdown hooks — never fires, because the JVM process itself was never sent a SIGTERM at all. It keeps running, orphaned, until systemd's cgroup accounting (which tracks the whole service's process tree, not just the one PID it directly launched) escalates to SIGKILL after TimeoutStopSec. exec java -jar app.jar fixes it by replacing the shell process image with the JVM's in place — same PID, no separate child — so the signal systemd sends the tracked PID lands on the JVM directly.

The lesson behind it →
PreviousA rolling deploy ships a migration that renames `users.email` to `users.email_address` in the same release as the code that reads the new name. Midway through the roll, requests start failing. Why, and what's the actual fix — not "don't rename columns," but the mechanism that makes a rename safe under a rolling deploy?Next nginx handles thousands of concurrent connections with a handful of worker processes — typically one per CPU core. Mechanically, how does one worker do that without one OS thread per connection, and why does a thread-per-connection server run out of resources before nginx does at the same connection count?
More on Linux signals
A production incident: a Spring Boot service's shutdown hook hangs during a deploy, blocked forever on a socket read to a downstream service that never responds. `TimeoutStopSec=15` is configured. Walk through exactly what happens, mechanically, from `systemctl stop` to the process actually being gone — and explain why the JVM's shutdown-hook mechanism doesn't run your cleanup code directly inside the OS signal handler in the first place.