Appears in: Telemetry Ingestion Pipeline — §3, Deep Dives — this is §3.8.
3.8 Global Deployment Topology
At Netflix/Google scale, a single-region ingestion cluster is both a single point of failure and introduces cross-region write latency for globally distributed agents.
flowchart TD
subgraph US["US-EAST"]
A1["Agents"] --> GW1["Regional Ingestion\nGateway"]
GW1 --> KF1["Regional Kafka\nCluster"]
KF1 --> PR1["Regional Processor\nFleet"]
PR1 --> MI1["Regional Mimir\nWrite Path"]
end
subgraph EU["EU-WEST"]
A2["Agents"] --> GW2["Regional Ingestion\nGateway"]
GW2 --> KF2["Regional Kafka\nCluster"]
KF2 --> PR2["Regional Processor\nFleet"]
PR2 --> MI2["Regional Mimir\nWrite Path"]
end
MI1 & MI2 -->|"async replication"| GQ["Global Query Tier\nRuler · Querier\nreads from all regional stores + merges"]
Trade-off: regional isolation vs global query:
| Approach | Write latency | Query complexity | Failure blast radius |
|---|---|---|---|
| Regional writes + federated query | Local (< 10ms) | Fan-out query across regions | Single region; others unaffected |
| Global single cluster | Cross-region (50–150ms) | Simple | Global — one cluster failure affects all |
| Regional writes + async cross-region replication | Local | Simple (global store) | Replication lag window |
Answer: Regional writes + async replication for global query. Write latency stays local. A region outage means that region’s tenants lose telemetry for the outage duration — acceptable if the alternative is global blast radius. This matches Mimir’s multi-cluster deployment model and how Grafana Cloud is actually architected.
Agent failover: If a region’s gateway is unreachable, the agent must reroute. Options:
- DNS-based failover (low TTL, agent retries to secondary endpoint)
- Anycast routing (agent always sends to the same IP; routing layer redirects to healthy region)
- Agent-side fallback list (explicitly configured secondary endpoint in agent config)
Local graph
Linked from 5 notes
6. Interview Anchor Points (What to Say Out Loud)
The sentences that signal principal-level thinking for the telemetry ingestion pipeline design — ready to say unprompted in an interview.
8. Quick-Reference Cheat Sheet
One-line answers for every load-bearing design decision in the telemetry ingestion pipeline — the last thing to review before an interview.
Q7: A Region's Gateway Goes Dark — Blast Radius Walkthrough and Redesign
Full principal-level solution: walk through the consequences of a 10-minute regional ingestion gateway outage, then redesign the topology to shrink the blast radius.
Chapter 1 — Telemetry Ingestion Pipeline
Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.
Tenant Identification and Routing at the Ingestion Frontier
How the gateway decides whose data this is — cert/API-key-derived tenant identity, propagation as a Kafka header and X-Scope-OrgID, and the shared-topic-with-filter vs per-tenant-topic vs shuffle-sharded-pool routing trade-off.
Related notes
Chapter 1 — Telemetry Ingestion Pipeline
Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.
Q1: 500M Samples/Sec, Zero Drop on Rolling Deploy
Full principal-level solution: design a telemetry ingestion pipeline for 500M metric samples/sec from 100K services globally with a zero-drop guarantee during rolling deployment of the ingestion tier.
Q10: Self-Service Tenant Onboarding With Zero Platform-Team Involvement
Full principal-level solution: design a self-service tenant onboarding API for a telemetry pipeline that protects shared infrastructure from a misbehaving new tenant on day one.
Q8: Counters Resetting to Zero After an OTel SDK Upgrade
Full principal-level solution: diagnose and fix a tenant's dashboards showing counters reset to zero every few minutes after an OTel SDK upgrade, without requiring instrumentation changes.