Appears in: Telemetry Ingestion Pipeline — §3, Deep Dives — this is §3.4.
3.4 Scaling Each Layer
| Layer | Scaling unit | Scaling trigger | Notes |
|---|---|---|---|
| Ingestion gateway | Pod replicas (HPA) | CPU > 70% OR active connections > 10K | Stateless; L7 load balancer distributes |
| Kafka brokers | Partition count | Consumer lag growing + broker CPU > 80% | Increase partitions; re-assign partition leaders |
| Metric processor | Consumer group pods | Consumer lag > target (e.g., 60s worth) | Pods = partition count for maximum parallelism |
| Trace assembler | Stateful shard pods | Memory > 70% OR assembly latency rising | Resharding is costly; over-provision or use consistent hashing |
| Log processor | Consumer group pods | Consumer lag | Stateless; easy to scale |
| Storage (Mimir) | Ingester pods | Series per ingester > target | Cortex/Mimir uses ring-based consistent hashing for ingesters |
Local graph
Linked from 7 notes
3 — Prometheus in the Observability Ecosystem
Where Prometheus sits in the CNCF landscape — its pull-based cloud-native origins, its companion projects, and where this book does (and doesn't yet) connect it to the wider stack.
2 — Long-Term Storage
Why single-node Prometheus has no built-in long-term-storage or HA story, and how remote_write receivers like Thanos, Cortex, Mimir, and VictoriaMetrics fill that gap at a horizontally-scaled, multi-tenant layer.
3.7 Data Tiering and Compaction (Mimir/Thanos)
Data tiering and compaction in Mimir/Thanos — the ingester-to-object-store journey, compaction levels, vertical compaction/dedup, compaction storms, and the config knobs that control them.
Q3: Trace Sampling Loses Spans During Incident Peaks — Redesign
Full principal-level solution: the trace pipeline drops spans exactly when incidents spike trace volume — diagnose the failure mode and redesign the tail-sampling pipeline to survive it.
Q6: Compactor Queue Backing Up During a Multi-Tenant Flush
Full principal-level solution: diagnose a compactor backlog causing query latency spikes during a large multi-tenant flush, and mitigate it without pausing ingestion.
Prometheus
A book-shaped table of contents for Prometheus: monitoring foundations through architecture, data model, instrumentation, service discovery, PromQL, alerting, production operation, PCA certification, and MAANG interview prep — cross-linking existing notes instead of duplicating them.
Chapter 5 — Telemetry Ingestion Pipeline
Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.
Related notes
Chapter 5 — Telemetry Ingestion Pipeline
Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.
Q1: 500M Samples/Sec, Zero Drop on Rolling Deploy
Full principal-level solution: design a telemetry ingestion pipeline for 500M metric samples/sec from 100K services globally with a zero-drop guarantee during rolling deployment of the ingestion tier.
Q10: Self-Service Tenant Onboarding With Zero Platform-Team Involvement
Full principal-level solution: design a self-service tenant onboarding API for a telemetry pipeline that protects shared infrastructure from a misbehaving new tenant on day one.
Q8: Counters Resetting to Zero After an OTel SDK Upgrade
Full principal-level solution: diagnose and fix a tenant's dashboards showing counters reset to zero every few minutes after an OTel SDK upgrade, without requiring instrumentation changes.