Notes / Observability / Reference

What is Tempo

Grafana Labs' distributed tracing backend — the radical simplification vs. Jaeger's classic architecture: no dedicated index, just object storage and a trace-ID lookup, queried with TraceQL and linked from metrics via exemplars.

Updated July 12, 2026 · §202607121601-8 ·

Tempo is Grafana Labs’ distributed tracing backend, built around one deliberate simplification: unlike What is Jaeger‘s classic architecture (which requires Cassandra or Elasticsearch to index span tags for search), Tempo maintains no secondary index at all. The only lookup it needs is trace ID → storage block, which means the only infrastructure dependency is object storage — no index cluster to size, operate, or pay for.


Ingest: multi-protocol by design

OTLP  ──┐
Jaeger ─┼──▶  Tempo distributor  ──▶  Ingester  ──▶  Object storage (blocks)
Zipkin ─┘                                                    │

                                                         Compactor

Tempo accepts OTLP, Jaeger, and Zipkin wire protocols natively at the same distributor — a service or vendor agent that “sends to Jaeger” doesn’t need a separate Jaeger deployment; pointing its Jaeger-protocol exporter at Tempo’s Jaeger receiver is a drop-in replacement.

Finding a trace without an index

Access pathHow it works
Trace ID lookupDirect block lookup — the one operation Tempo was built to make cheap
Exemplars from MimirA Prometheus/Mimir histogram bucket carries a sampled trace ID linking straight to the request that produced that latency value
Log-to-trace correlationA structured log line in What is Loki carrying trace_id links directly into Tempo
TraceQL searchStructured query over span attributes/duration/status, added later to close the gap with tag-indexed search
{ span.http.status_code = 500 && duration > 800ms }

TraceQL search still has to scan recent blocks rather than hit a pre-built tag index, which is the real tradeoff behind Tempo’s cost model: cheap storage and ingestion, in exchange for search being scoped to recent time windows rather than instant across all history the way an ES-backed Jaeger index would be.

Exemplars: the concrete metrics-to-trace bridge

Mimir histogram: http_request_duration_seconds_bucket{le="1.0"}

                       │  exemplar: trace_id="a1b2c3...", value=0.94

                  Tempo trace a1b2c3...  (the actual request that produced that sample)

This is the mechanism What is Cardinality (in observability) points to as the alternative to keeping high-churn fields (request IDs, user IDs) as metric labels: instead of a label multiplying series count, a sampled exemplar carries just enough of a pointer to jump from an aggregate metric straight to one concrete trace — no series growth, full request-level detail on demand.

Tempo vs Jaeger

ConcernTempoJaeger
Storage requirementObject storage onlyCassandra / Elasticsearch (indexed) or Badger (local)
Operational overheadLow — no index cluster to runHigher — index cluster sizing, retention, upgrades
Tag/attribute searchTraceQL, scoped to recent blocksFull tag index, historically stronger ad hoc search
Ingest protocolsOTLP, Jaeger, Zipkin — all nativeOTLP-native since Jaeger v2 (rebuilt on OTel Collector)

Why it matters here: Tempo is the trace-storage tier in the ShipSolid Grafana Cloud stack, and exemplars are the standing answer to “how do I get request-level detail without adding a label” — whenever a proposed metric label is actually about identifying one specific request, the exemplar path into Tempo is the correct redirect, not a What is Cardinality (in observability) exception.

Local graph

Full graph →

Linked from 17 notes

7 — Distributed Tracing Backend

How spans that arrive out of order, from different services, get assembled into one trace — and the two competing storage models (indexed search vs. object storage plus a trace-ID lookup) that trade query flexibility for cost.

What is Jaeger

CNCF-graduated distributed tracing system built at Uber in 2015, Dapper-lineage like Zipkin before it — and, since Jaeger v2, rebuilt on top of the OpenTelemetry Collector rather than bespoke ingestion code.

Grafana Cloud Deployment

How Alloy exports traces, metrics, and logs to Grafana Cloud, including credential architecture, Azure Key Vault setup, and troubleshooting.

7. Component Map (What Exists in the Wild)

OSS and managed-SaaS options for every layer of the telemetry ingestion pipeline, mapped against ShipSolid's own production experience.

Telemetry Gateways: Protocol-Specific Ingestion Points

The ingestion frontier is a fleet of protocol-specific gateways — OTLP, Prometheus remote-write, Syslog, Kafka, and legacy tracing/metrics protocols — each terminating a different producer's wire format before a shared auth/rate-limit/tenant-routing layer.

Agentic AI: Projects & Engineering Mastery

A book-shaped table of contents for Agentic AI: Projects & Engineering Mastery: hands-on practitioner builds, Principal/Staff-level technical leadership, and the lookup appendices and vendor/framework reference notes for the whole series. Book 6 of the AI Systems Engineering series.

Grafana Cloud

A book-shaped table of contents for Grafana Cloud: platform foundations through telemetry collection, Mimir/Loki/Tempo/Pyroscope, visualization, application observability, reliability tooling, developer experience, governance, and enterprise reference architectures — cross-linking existing notes instead of duplicating them.

2 — The Signals

Metrics, logs, traces, profiles, and events — what each is built to capture, what it costs, and which question it actually answers vs. which one people mistakenly ask it.

7 — Multi-Tenancy

Two separate guarantees hiding under one name — data isolation and performance fairness — and the tenant identification, quota enforcement, and selective backpressure that make both hold under shared infrastructure.

1 — OpenTelemetry SDKs & Semantic Conventions

OpenTelemetry is a specification and an API/SDK, not a backend — the pieces that make it up, and the semantic-convention vocabulary that lets two unrelated teams' telemetry be queried the same way.

9 — OTel Collector Pipeline Design

Receivers, processors, and exporters chained into a pipeline; why a platform runs more than one; and the agent/gateway topology that tail sampling specifically forces on that design.

Observability Engineering

A book-shaped table of contents for observability engineering: foundations through architecture, metrics, logging, tracing, profiling, OpenTelemetry, instrumentation, Kubernetes/cloud, data platforms, visualization, alerting, SRE integration, cost, security, platform engineering, AI-driven operations, and MAANG interview preparation — cross-linking existing prometheus/grafana-cloud/kubernetes/sre/platform-engineering notes instead of duplicating them.