Code10x
Distributed tracingmedium3-5 years

A checkout request's trace looks complete and healthy in the tracing UI — but an asynchronous confirmation email, submitted to a thread pool from inside that same request, shows up as its own separate, single-span trace instead of a child span. Explain exactly why, using the propagation mechanism, not just 'context was lost.'

A trace's 'current span' is stored per-thread — OpenTelemetry's Java SDK keeps it in a ThreadLocal-backed Context, the same primitive MDC is built on — so handing work to an executor hands over a reference to data, not the thread-local slot that data currently lives in. The new pool thread's Context.current() starts completely empty, exactly the way a fresh pooled thread's MDC map starts empty, unless something explicitly captures the context at submission time and re-attaches it on the worker thread before any span-starting code runs there. Because nothing did that here, the email's span has no parent to attach to — from the tracer's point of view this is indistinguishable from a request that arrived with no traceparent header at all, so it correctly starts a brand-new, unrelated trace rather than failing loudly.

The lesson behind it →
PreviousA teammate adds `.tag("userId", userId)` to an existing HTTP request timer "so we can see per-user latency." What happens to the metrics platform, and why does this one line matter more than it looks like it should?Next A platform team sets head sampling at 1% to control tracing storage cost. During an incident with a 0.1% failure rate, an engineer complains that almost none of the failed requests have a trace. Is that a bug, a misconfiguration, or expected behavior of head sampling — and what would tail sampling have done differently, at what cost?
More on Distributed tracing
A platform team sets head sampling at 1% to control tracing storage cost. During an incident with a 0.1% failure rate, an engineer complains that almost none of the failed requests have a trace. Is that a bug, a misconfiguration, or expected behavior of head sampling — and what would tail sampling have done differently, at what cost?