Chapter 3 — Monitoring at Scale
Part 08 of the System Design curriculum. Full treatment: Storage & Query in the Observability book, and the Prometheus book for Prometheus-specific depth.
A metrics time-series database optimizes for one access pattern almost no other datastore does: an enormous, constant write rate (one sample per series per scrape interval, forever) against queries that are almost always a range-scan over time for a known set of series, never a point lookup by an arbitrary key. Every design choice below follows from that one asymmetry.
Write path: head block, WAL, and compaction
Samples land first in an in-memory head block — one append-only chunk per series, compressed with delta/XOR-style encoding — mirrored to a write-ahead log (WAL) on disk so a crash doesn’t lose data that only ever existed in memory. Once the head block hits a size or time threshold, it flushes to disk as an immutable block. Freshly flushed blocks are small and numerous, so compaction merges them into fewer, larger ones — better compression, faster queries — at the cost of write amplification: the same sample gets physically rewritten every time a block containing it is compacted into a larger one.
Cardinality is a storage-engine problem, not just a bill
Every new label combination is a brand-new series, which means a brand-new chunk stream in the head block. A sudden cardinality spike doesn’t just bloat storage over time — it inflates the live, in-memory head block right now, which can slow ingestion and compaction for every other series sharing that tenant or instance. A cardinality incident is a memory-pressure incident on the ingestion path before it’s ever a cost-line-item.
From one node to a fleet: Mimir, Cortex, Thanos
A single Prometheus instance’s TSDB is local — durable and fast, but not horizontally scalable and not multi-tenant. Scaling that model out means sharding both storage and query across many nodes: samples are sharded by tenant and series across ingesters, and a query spanning that sharding has to scatter out to every relevant node and gather the results back. Mimir, Cortex, and Thanos are three different projects solving that same scale-out problem around the same Prometheus block format and query language (PromQL), differing mainly in storage backend and operational model. Full treatment: Metrics Storage (TSDB).
What this means for a system design interview
“We’ll use Prometheus” doesn’t survive a 10x-scale follow-up. The interview-worthy answer names the specific bottleneck a single-node TSDB hits first (ingestion memory pressure from cardinality, not disk space), and states the scatter-gather mechanics of the horizontally-scaled alternative rather than just naming Mimir/Cortex/Thanos as if they were interchangeable drop-in replacements.
Where to go deeper
- Metrics Storage (TSDB)
- Metrics Storage (TSDB) — applied case study (Part 15, stub)
- Prometheus book
Metadata
| Author | Amit Singh |
| Scope | system-design |
Local graph
Linked from 2 notes
4. Observability of the Pipeline Itself
What to instrument at every layer of the telemetry ingestion pipeline, the pipeline's own SLOs, distributed tracing of the pipeline itself, and the synthetic canary that catches stalls no component metric surfaces.
System Design
Principal/Staff-level system design reference collection for MAANG interview preparation — observability pipelines, distributed systems, reliability engineering, and beyond.
Related notes
Chapter 4 — Alerting Systems
Multi-window burn-rate alerts, recording rules, routing, and deduplication as the difference between an actionable page and noise.
Chapter 1 — Observability Architecture
Metrics, logs, traces, and profiles as the four correlated signal types every observability platform is built around.
Chapter 2 — Telemetry Pipelines
OpenTelemetry, OTLP, collectors, sampling, and aggregation as the pipeline that gets a signal from emission to storage without becoming the outage itself.
Chapter 4 — Distributed Tracing Backend
Trace assembly from spans, tail-based vs. head-based sampling.