Written October 2026, looking back at Systems8 min read
Keep the failing traces
Adding OpenTelemetry to Watson serving at IBM, and why I biased sampling toward errors and slow requests instead of sampling uniformly.
From IBM · May 2024 – Aug 2025
Part of my work at IBM was adding OpenTelemetry tracing to the Watson serving path. The instrumentation itself was the straightforward part. The interesting question turned out to be which traces to keep.
Why you can't keep everything
A trace records one request's journey through the system: every service it touched, how long each step took, where it failed. They're incredibly useful when something goes wrong. They're also expensive to export, store, and index, so at any meaningful traffic level you can't keep all of them. You sample.
The default approach is head sampling. When a request starts, you roll a die, and if it comes up, you record the whole trace. It's simple, cheap, and consistent across services, because the decision is made once at the start and propagated.
The problem is that the decision happens before you know anything about the request.
Head sampling throws away the interesting ones
Think about what you actually open a trace viewer for. Almost never a healthy request that came back in the usual time. You open it when something failed, or when a request took ten times longer than it should have.
Those requests are rare. That's what makes them interesting, and it's also why uniform sampling at a low rate mostly misses them. You end up with a large pile of healthy traces that all look the same, and when someone asks why a particular call failed, the honest answer is that the trace was probably sampled out.
It's a uniquely frustrating feeling: you go looking for the trace of a slow request and find plenty of fast traces from the same minute, none of the one you need.
The arithmetic is unforgiving, because a head sampler is blind to outcome: every request, failed or not, is kept with the same probability p. So the chance that the one failing request someone asks about was kept is exactly p. At a 1% head rate, 99 out of 100 "why did this call fail?" questions start with no trace. Even across a burst of related failures, the chance of keeping at least one is 1 - (1 - p)^N: with p = 1% you need about 70 failures before that's even a coin flip (0.99^69 ≈ 0.50). A rare, one-off failure is the case you can least afford to miss, and it's the case head sampling is worst at.
- Head sampling, p = 1%
- Head sampling, p = 5%
- Keep-all-errors at the tail
Show data
| Occurrences of the same failure | Head sampling, p = 1% | Head sampling, p = 5% | Keep-all-errors at the tail |
|---|---|---|---|
| 1 | 1.0% | 5.0% | 100% |
| 5 | 4.9% | 23% | 100% |
| 10 | 9.6% | 40% | 100% |
| 25 | 22% | 72% | 100% |
| 50 | 40% | 92% | 100% |
| 100 | 63% | 99% | 100% |
Raising p until the failures survive doesn't work either: storage scales with p across all traffic, while the thing you want is a small, specific slice of it.
Error-biased sampling
So the policy I pushed for is simple to say:
- keep every trace that ends in an error
- keep every trace that's slow, above a latency threshold
- sample the healthy, fast ones at a low rate, enough to know what normal looks like
The catch is that you can only know whether a request failed or was slow at the end. That means the decision has to move from the head to the tail. Spans get buffered until the trace is complete, or close enough, and then a policy decides whether the whole thing is kept.
In OpenTelemetry terms, that's tail-based sampling in the collector rather than a sampler in the SDK. The services still emit spans; the collector holds them briefly, groups them by trace, and applies the rules. The collector's tail sampling processor maps onto the three rules almost one to one: a status_code policy for errors, a latency policy for slow traces (it measures from the trace's earliest span start to its latest span end), and a probabilistic policy for the healthy baseline. A trace is kept if any policy says keep.
The reason this costs about the same as uniform sampling is that errors and slow requests are a small share of traffic. As an estimate, with illustrative inputs rather than Watson's real mix: if 0.5% of requests fail, 1% are slow, and healthy traffic is sampled at 1%, the kept fraction is
kept = errors + slow + p x healthy
= 0.005 + 0.01 + 0.01 x 0.985
≈ 2.5% of all traces
A head sampler with the same 2.5% budget keeps 2.5% of the failures. The tail policy keeps all of them, and spends the same budget doing it.
- Share of error traces kept
- Share of all traces kept
Show data
| policy | Share of error traces kept | Share of all traces kept |
|---|---|---|
| Head, uniform 2.5% | 2.5% | 2.5% |
| Tail, error-biased | 100% | 2.5% |
What it doesn't save is the cost upstream of the decision. With head sampling, the decision is propagated in the trace context, so unsampled requests never export a span at all. With tail sampling every service exports every span to the collector, and only storage and indexing shrink. So the trade is network and collector CPU in exchange for keeping the traces that matter. I think that's the right trade, but it's a real one, and it's why the collector tier needs sizing rather than being an afterthought.
Things I'd watch out for
Tail sampling isn't free, and a few things surprised me.
Memory and timing. The collector has to hold spans until it decides. If a trace is very long, or some spans arrive late, you have to choose how long to wait. Too short and you decide on partial traces; too long and memory grows. There's no perfect setting, only one you've tested against your actual traffic.
The sizing is Little's law: traces held in memory is roughly new traces per second times the decision wait. The processor's defaults are a 30-second wait and a 50,000-trace buffer, and when the buffer is full the oldest trace is evicted to make room. As an estimate with an illustrative rate: at 1,000 new traces per second, a 30-second wait holds about 30,000 traces, inside the default. At 2,000 per second it wants 60,000, and the buffer silently starts evicting the oldest traces before they're decided. So the buffer size isn't a tuning detail: it's set by peak traffic, and the failure mode at 2x traffic is losing traces quietly rather than loudly. Late spans have a similar trap: once a trace's data has been released, a late span triggers a fresh decision of its own unless the processor's decision cache remembers which trace IDs were kept.
All spans for a trace need to land in one place. If you run several collectors, spans from the same trace must be routed to the same one, or each sees a fragment and makes its own decision. This is the kind of thing that works fine in a single-instance test and breaks at scale. The standard fix is two layers: a first layer of collectors that does nothing but route spans by trace ID (the load-balancing exporter), and a second layer that does the sampling. The catch at 10x is resizing: when the second layer scales out, the trace-ID hash ring changes, and traces in flight at that moment can be split across two instances. Consistent hashing limits that to the traces whose keys moved, and only the ones that straddle the change, so it's a burst of fragmented decisions per scaling event rather than an ongoing leak. Acceptable, as long as you know it happens and don't autoscale that layer on a hair trigger.
Cost during incidents. Keeping every error trace sounds great until a backend goes down and suddenly everything is an error. Your trace volume spikes at the exact moment the system is under stress. With the illustrative mix above, normal volume is about 2.5% of traces; if every request fails, it's 100%, a 40x jump in what the trace backend has to ingest, at the moment you most need it to stay up. A cap or a rate limit on the "keep errors" rule is worth having, even if it feels like it contradicts the whole idea. One subtlety: because policies combine with "keep if any says keep", a standalone rate-limiting policy doesn't cap the error rule, it just adds another way to be kept. The cap has to wrap the error rule itself, for example a composite policy that gives errors a fixed share of a spans-per-second budget. During an incident you only need a representative sample of a failure that's happening thousands of times; it's the one-off failure on a normal day that needs every trace.
What counts as slow. The latency rule takes one threshold per policy, and a serving path has endpoints with very different normal latencies. One global threshold set for the slowest endpoint never fires on the fast ones, and one set for the fast endpoints keeps every request to the slow ones, which quietly turns "keep the slow ones" into "keep everything from that endpoint". The threshold belongs near each endpoint's own tail (somewhere around its p99), which means one policy per class of endpoint rather than one number for the whole system.
Cardinality. It's tempting to attach every useful-looking attribute to spans. Some of those, like raw IDs or free-form strings, explode the number of distinct values your backend has to index. I learned to be deliberate about which attributes are for searching and which are just context.
What I took from it
The broader lesson is that sampling is a product decision, not just a cost knob. The question isn't "what fraction can we afford" but "which requests will someone need to look at later". Once you frame it that way, biasing toward failures is obvious.
The test I'd apply to any sampling policy: list the last few incidents and ask whether the traces you needed would have survived it. That's a much better test than any dashboard. It's also the same instinct as the Go client: design for the bad day, because the good days take care of themselves.