2 — Long-Term Storage
The gap this chapter frames
A single Prometheus server owns its own TSDB on local disk, and nothing else. There is no replication, no clustering, and no built-in mechanism for keeping data beyond the local retention window. That is a deliberate design choice, not an oversight — Prometheus trades away distributed storage complexity for a simple, reliable, single-binary operational model. But it means two questions that come up the moment Prometheus goes into production have no native answer:
- How do I keep metrics for a year, not two weeks? Local disk and local retention don’t scale to that horizon, and nobody wants to run one Prometheus server sized for a year of full-resolution data.
- How do I make “the monitoring system” itself highly available without ending up with two independent servers that each hold half the truth?
Prometheus’s own answer to both is the same escape hatch: remote_write. Rather than solving
long-term storage and HA inside the server, Prometheus ships every sample out over remote_write to
a receiver that implements roughly the same query surface (PromQL, and often the same HTTP API) at a
horizontally-scaled, multi-tenant layer. That receiver is where retention, replication,
downsampling, and multi-tenancy actually get solved. Thanos, Cortex, Grafana Mimir, and
VictoriaMetrics are the well-known implementations of that receiver pattern — different projects,
different storage engines, but the same shape: take the remote-write firehose from many Prometheus
servers, and give back a single, durable, queryable, long-horizon view.
Why this chapter stays short
None of the source material behind this book goes deep on any of these backends — there’s no walkthrough of Thanos’s sidecar-and-store-gateway split, no Cortex ingester/distributor architecture, no Mimir microservice map, no VictoriaMetrics storage internals. Rather than write a thin, second-hand comparison that duplicates content that already exists in more depth elsewhere in this wiki, this chapter is intentionally a short frame plus pointers. The real depth lives in the linked notes below — treat this page as the map, not the territory.
Where to go next
- What is Mimir — a dedicated note on Grafana Mimir specifically: its microservice write path and read path, and why it’s the system actually serving every PromQL query and remote-write in a Grafana Cloud metrics stack. Go here for the “Grafana Mimir” answer.
- 3.4 Scaling Each Layer and 3.6 Multi-Tenancy — comparative scaling and multi-tenancy context across Cortex, Thanos, and VictoriaMetrics-style architectures, from the telemetry ingestion pipeline system design.
- Metrics Storage (TSDB) — TSDB internals (chunk/block encoding, WAL, compaction), and the single-node vs. horizontally-scaled framing that this chapter only gestures at.
Metadata
| Author | Amit Singh |
| Scope | prometheus |
Local graph
Linked from 7 notes
What is Cortex (cortexproject)
CNCF Incubating, horizontally-scalable multi-tenant long-term storage for Prometheus — the project Grafana Mimir forked from in 2022, still maintained as the vendor-neutral, community-governed alternative once Grafana Labs redirected engineering effort to Mimir.
What is Mimir
Grafana Labs' horizontally-scalable, multi-tenant long-term storage for Prometheus metrics — the 2022 successor to Cortex, and the actual system serving every PromQL query and remote-write in a Grafana Cloud metrics stack.
1 — Prometheus Components
The functional pieces inside a Prometheus server — scrape manager, TSDB, rule engine, query engine — and the real commands used to install and run one on a VM, under systemd, or in Docker.
5 — Prometheus Configuration Reference
A field-by-field reference for prometheus.yml — global settings, scrape_configs options, and worked examples pulled from real multi-job configurations.
Chapter 3 — Monitoring at Scale
Prometheus, Mimir, Cortex, and Thanos as the horizontally-scaled answer to a single Prometheus instance running out of room.
3.7 Data Tiering and Compaction (Mimir/Thanos)
Data tiering and compaction in Mimir/Thanos — the ingester-to-object-store journey, compaction levels, vertical compaction/dedup, compaction storms, and the config knobs that control them.
Prometheus
A book-shaped table of contents for Prometheus: monitoring foundations through architecture, data model, instrumentation, service discovery, PromQL, alerting, production operation, PCA certification, and MAANG interview prep — cross-linking existing notes instead of duplicating them.
Related notes
4 — High Availability
Running Prometheus HA pairs, the duplicate-sample problem that creates, deduplication strategies, and failover/DR posture for a monitoring system that is itself a dependency.
3 — Performance Tuning
Tuning Prometheus's own resource footprint and query performance — memory, CPU, WAL replay time, compaction cadence, scrape interval, and retention as levers, with cardinality as the dominant cost driver.
1 — Scaling Prometheus
Federation, functional and horizontal sharding, HA server pairs, and remote_write as Prometheus's own answers to 'this one server can't hold it all anymore.'
3 — Prometheus in the Observability Ecosystem
Where Prometheus sits in the CNCF landscape — its pull-based cloud-native origins, its companion projects, and where this book does (and doesn't yet) connect it to the wider stack.