Appears in: Telemetry Ingestion Pipeline — §3, Deep Dives — this is §3.7.
3.7 Data Tiering and Compaction (Mimir/Thanos)
This section is commonly missing from candidate answers. The architecture diagram shows Mimir as a black box, but the ingester → blocks → object store journey matters for durability, query latency, and cost.
flowchart TD
PROC["Processor"] -->|write| ING["Mimir Ingester\nhot tier — in-memory block\nlast 2h"]
ING -->|"flush every 2h"| OBJ[("Object Store\ncold tier\n/tenant_id/blocks/")]
OBJ -.->|"async reads"| COMP["Compactor\nmerge overlapping ranges\ndeduplicate RF=3 replicas"]
COMP -->|"compacted blocks"| OBJ
QR["Querier"] -->|"recent 2h from memory"| ING
QR --> SG["Store-gateway\nblock index from object store"]
SG -->|"block reads"| OBJ
ING & SG --> MERGE["Merge + sort results"]
style PROC fill:#4a9eff,color:#fff
style QR fill:#4a9eff,color:#fff
Compaction levels:
| Level | Time range | Produced from | Purpose |
|---|---|---|---|
| L1 | 2h | Ingester flush | Raw blocks; many small files |
| L2 | 12h | Compact L1 × 6 | Fewer files; faster range queries |
| L3 | 24h | Compact L2 × 2 | |
| L4 | 7d | Compact L3 × 7 | Long-range scan efficiency |
Vertical compaction (deduplication): With RF=3 ingesters, each series is written to three ingesters simultaneously. After flush, the compactor deduplicates overlapping blocks by label fingerprint + timestamp. Without vertical compaction, storage cost is 3×.
Compaction storms: When many tenants flush large volumes simultaneously, the compactor queue backs up. Symptoms: query latency spikes as the store-gateway must scan un-compacted L1 blocks. Mitigation: rate-limit ingester flushes per tenant, or run compactors per-tenant shard.
Key config knobs to know:
max-block-duration: longer = fewer files per query; shorter = faster flush cycle (2h is the Mimir default)- Compaction concurrency: limits CPU spike on the compactor
- Store-gateway lazy loading: don’t load all block indices into memory at startup (critical at millions of blocks)
Local graph
Linked from 8 notes
What is Mimir
Grafana Labs' horizontally-scalable, multi-tenant long-term storage for Prometheus metrics — the 2022 successor to Cortex, and the actual system serving every PromQL query and remote-write in a Grafana Cloud metrics stack.
2 — Long-Term Storage
Why single-node Prometheus has no built-in long-term-storage or HA story, and how remote_write receivers like Thanos, Cortex, Mimir, and VictoriaMetrics fill that gap at a horizontally-scaled, multi-tenant layer.
3.4 Scaling Each Layer
Scaling unit and trigger for every layer of the telemetry ingestion pipeline, from the ingestion gateway through to storage.
3.5 Failure Modes and Mitigations
Failure modes and mitigations across the telemetry ingestion pipeline — gateway crashes, broker failure, processor crashes, span explosion, cardinality rejection, storage saturation, and clock skew.
8. Quick-Reference Cheat Sheet
One-line answers for every load-bearing design decision in the telemetry ingestion pipeline — the last thing to review before an interview.
Q6: Compactor Queue Backing Up During a Multi-Tenant Flush
Full principal-level solution: diagnose a compactor backlog causing query latency spikes during a large multi-tenant flush, and mitigate it without pausing ingestion.
Q9: Cut Ingestion Infrastructure Cost 40% Without Violating SLOs
Full principal-level solution: a FinOps-driven cost-reduction pass on a telemetry ingestion pipeline — where to look first, what levers exist at each layer, and what you trade away.
Chapter 1 — Telemetry Ingestion Pipeline
Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.
Related notes
Chapter 1 — Telemetry Ingestion Pipeline
Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.
Q1: 500M Samples/Sec, Zero Drop on Rolling Deploy
Full principal-level solution: design a telemetry ingestion pipeline for 500M metric samples/sec from 100K services globally with a zero-drop guarantee during rolling deployment of the ingestion tier.
Q10: Self-Service Tenant Onboarding With Zero Platform-Team Involvement
Full principal-level solution: design a self-service tenant onboarding API for a telemetry pipeline that protects shared infrastructure from a misbehaving new tenant on day one.
Q8: Counters Resetting to Zero After an OTel SDK Upgrade
Full principal-level solution: diagnose and fix a tenant's dashboards showing counters reset to zero every few minutes after an OTel SDK upgrade, without requiring instrumentation changes.