7 — Distributed Tracing Backend
A trace is never produced in one place. Its spans arrive from N different services, at different
times, out of order, each one only knowing its own trace_id and its parent span_id — not the
shape of the tree it belongs to. Everything a tracing backend does starts from that one fact: it has
to assemble a tree from parts that show up piecemeal, before it can be queried, rendered, or (per
the Collector chapter) even sampled.
Assembly: waiting for a trace to be “done”
Spans for the same trace_id land at the backend independently and asynchronously. Before a trace
can be considered complete, the backend has to buffer incoming spans under that trace_id and wait
out a completion window — long enough that a slow downstream span reasonably has time to arrive, but
not so long that memory holds open traces indefinitely. This is the same completeness problem
tail sampling
solves at the collector layer, one stage earlier: a gateway collector has to see every span before
deciding whether to keep the trace at all, and a storage backend has to see every span before it can
present the trace as one coherent tree rather than a handful of orphaned fragments.
Two storage models, one trade-off
The indexed model (Jaeger’s original architecture, in the Dapper/Zipkin lineage) writes every
span into a search-optimized index — by trace_id, service, operation name, tags — backed by a
store like Cassandra or Elasticsearch. This buys rich ad-hoc query: “find traces where
service=checkout and http.status_code=500 and duration>1s,” with no prior knowledge of which
trace you’re looking for. The cost is the index itself — built and stored for every span, at full
span volume, whether or not that span is ever searched by tag.
The radical simplification (Tempo) drops the secondary index entirely: spans are
grouped by trace_id and written straight to object storage, with no way to query by tag at all.
You can only retrieve a trace if you already know its trace_id — from an exemplar, a correlated
log line, or a link from another trace. This is dramatically cheaper to store, but it’s a direct bet
on 3 — Cross-Signal Correlation actually being wired up everywhere: it only works if something else
(a metric’s exemplar, a log’s trace_id field) always hands you the identifier, because the backend
itself has given up the ability to help you find one by content.
| Model | Query by tag, no known trace_id | Storage cost | What it’s betting on |
|---|---|---|---|
| Indexed (Jaeger-style) | Yes — that’s the point | High — full span index | You’ll sometimes need to search, not just look up |
| Object storage (Tempo) | No — trace_id lookup only | Low — no secondary index | Correlation always hands you a trace_id first |
Neither model is wrong in isolation; each is correct for a different assumption about how traces actually get found in practice. A platform betting on the object-storage model without correlation actually working everywhere has quietly removed its own ability to find a trace when correlation fails — which is precisely when someone would need to search for one.
What the backend hands back is a query-time concern
Once a trace is assembled, Trace Shape covers what that call tree should look like in the rendered waterfall so a slow branch is visible; this chapter has been about how the backend gets from a pile of independently-arriving spans to that tree in the first place — the storage-side half of the same problem.
What sampling upstream already decided for you
By the time a span reaches this backend, the collector pipeline and whatever head-vs-tail sampling policy it enforced have already decided which traces exist to be stored at all. A tracing backend’s storage model and its upstream sampling policy are not independent decisions — an indexed backend that expects to answer “show me every slow checkout trace” needs a sampling policy biased toward keeping the traces that question needs, not a uniform random sample that keeps 1% of everything indiscriminately.
Tooling: Tempo is the object-storage-only model above; Jaeger is the classic indexed model, now (as of Jaeger v2) rebuilt on the OpenTelemetry Collector for ingestion rather than bespoke code, even though its storage model remains the indexed one.
Why this matters for an Observability Architect
Choosing a tracing backend is choosing which question you’re allowed to ask without already knowing the answer. An object-storage backend is the cheaper, and usually correct, choice for a platform where correlation is genuinely solid everywhere — but adopting one is also a bet that nobody will need “search by tag with no starting trace_id” often enough to matter. That’s a real product decision about how engineers are expected to start an investigation, not just a storage-cost optimization.
Metadata
| Dimension | Detail |
|---|---|
| Author | Amit Singh |
| Scope | observability |
Local graph
Linked from 8 notes
5 — Continuous Profiling
What makes always-on, sampling-based profiling cheap enough to run in production continuously, why it earns that cost mainly for hot or expensive services, and how a profile correlates back to the one trace that was running during the sample.
What is Jaeger
CNCF-graduated distributed tracing system built at Uber in 2015, Dapper-lineage like Zipkin before it — and, since Jaeger v2, rebuilt on top of the OpenTelemetry Collector rather than bespoke ingestion code.
What is Tempo
Grafana Labs' distributed tracing backend — the radical simplification vs. Jaeger's classic architecture: no dedicated index, just object storage and a trace-ID lookup, queried with TraceQL and linked from metrics via exemplars.
3 — Cross-Signal Correlation
Metrics, logs, and traces are only more useful together than apart if something ties a specific instance of each of them back to the same event. The shared identifier that makes that jump possible — and what breaks when a hop in the call chain doesn't carry it.
9 — Fan-Out Metrics and Trace Shape
The Prometheus metrics that instrument a fan-out pattern (width, shard latency, aggregation, partial results, cancelled workers, hedged requests), the OTel trace waterfall shape that reveals the tail shard, and when requests fan out to multiple shards vs route to a single one.
Grafana Cloud
A book-shaped table of contents for Grafana Cloud: platform foundations through telemetry collection, Mimir/Loki/Tempo/Pyroscope, visualization, application observability, reliability tooling, developer experience, governance, and enterprise reference architectures — cross-linking existing notes instead of duplicating them.
Observability Engineering
A book-shaped table of contents for observability engineering: foundations through architecture, metrics, logging, tracing, profiling, OpenTelemetry, instrumentation, Kubernetes/cloud, data platforms, visualization, alerting, SRE integration, cost, security, platform engineering, AI-driven operations, and MAANG interview preparation — cross-linking existing prometheus/grafana-cloud/kubernetes/sre/platform-engineering notes instead of duplicating them.
Chapter 4 — Distributed Tracing Backend
Trace assembly from spans, tail-based vs. head-based sampling.
Related notes
8 — Log Aggregation
Schema-on-write vs. schema-on-read as competing bets about when to pay indexing cost, and the two different deduplication problems a log pipeline actually has to solve.
7 — Metrics Storage (TSDB)
Chunk/block encoding, the write-ahead log, compaction and the write amplification it trades for query speed, and why a cardinality spike is a storage-engine problem, not just a cost line item.
2 — The Signals
Metrics, logs, traces, profiles, and events — what each is built to capture, what it costs, and which question it actually answers vs. which one people mistakenly ask it.
1 — Dashboard Design
The three-question test for a vanity panel, why the same underlying data needs a different dashboard for different audiences, and the top-down layout that mirrors how an investigation actually drills down.