Appears in: Telemetry Ingestion Pipeline — §3, Deep Dives — this is §3.5.
3.5 Failure Modes and Mitigations
| Failure | Impact | Mitigation |
|---|---|---|
| Gateway pod crash | In-flight requests lost (small window) | Agents retry with backoff; gRPC deadline ensures fast failure; stateless so restart fast |
| Kafka broker failure | Partition leaders re-elected (10–30s) | RF=3, unclean.leader.election=false; producers retry; consumer lag grows briefly |
| Processor crash mid-batch | Messages re-consumed from Kafka offset | At-least-once; downstream TSDB deduplicates |
| Trace assembler OOM (span explosion) | Incomplete traces flushed or dropped | Hard memory limit + backpressure to Kafka; alert on orphan span rate |
| TSDB write rejection (cardinality) | Metric data loss for offending tenant | Cardinality check upstream (processor) to reject early; platform alert to tenant |
| Storage tier full | Write rejection cascade | Backpressure to Kafka → gateway 429 → agent backoff; auto-scale storage (object store) |
| Clock skew between agents | Out-of-order samples rejected by TSDB | Mimir/Prometheus accept samples up to 1h out of order with out-of-order ingestion enabled |
Local graph
Related notes
Chapter 1 — Telemetry Ingestion Pipeline
Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.
Q1: 500M Samples/Sec, Zero Drop on Rolling Deploy
Full principal-level solution: design a telemetry ingestion pipeline for 500M metric samples/sec from 100K services globally with a zero-drop guarantee during rolling deployment of the ingestion tier.
Q10: Self-Service Tenant Onboarding With Zero Platform-Team Involvement
Full principal-level solution: design a self-service tenant onboarding API for a telemetry pipeline that protects shared infrastructure from a misbehaving new tenant on day one.
Q8: Counters Resetting to Zero After an OTel SDK Upgrade
Full principal-level solution: diagnose and fix a tenant's dashboards showing counters reset to zero every few minutes after an OTel SDK upgrade, without requiring instrumentation changes.