Appears in: Telemetry Ingestion Pipeline — this is §4 of the full design, split into its own file so the root stays a table of contents.
4. Observability of the Pipeline Itself
This is your unfair advantage. State this unprompted.
What to instrument
Gateway:
telemetry_gateway_requests_total{protocol, signal_type, tenant, status_code}— request rate by outcometelemetry_gateway_request_duration_seconds— p50/p95/p99 per protocoltelemetry_gateway_active_connections— fan-in healthtelemetry_gateway_backpressure_429_total— upstream congestion signal
Kafka:
- Consumer group lag per topic+partition (the primary health signal)
- Producer send latency
- Broker disk utilization
- Under-replicated partitions (leading indicator of broker issues)
Processors:
telemetry_processor_spans_dropped_total{reason}— sampling decisionstelemetry_processor_cardinality_limit_exceeded_total{tenant}— tenant abusetelemetry_processor_processing_latency_seconds— pipeline throughputtelemetry_processor_kafka_consumer_lag— restate lag at the processor level for easy alerting
Storage:
- Mimir/Loki write latency and error rate
- Active series per tenant
- Chunk compression ratio (degradation means label churn)
- WAL replay time (recovery latency signal)
SLOs for the pipeline
| SLO | Objective | Measurement |
|---|---|---|
| Ingestion success rate | 99.9% | 1 - (errors / total_requests) at gateway |
| End-to-end latency (metric) | P99 < 60s | Time from agent send to Mimir queryable (timestamp delta) |
| End-to-end latency (trace) | P99 < 5m | Time from first span received to trace queryable in Tempo |
| Tail sampling decision latency | P99 < 30s | Time from root span arrival to sampling decision |
| Cardinality budget breach rate | < 0.1% | Tenants hitting cardinality limits per day |
Distributed tracing of the pipeline itself
Instrument the pipeline with OTel traces that follow a telemetry batch through each layer. A trace that starts at the gateway and ends at the storage write gives you end-to-end visibility. At MAANG scale, sample these at 1% (head-based is fine here — you’re tracing the pipeline, not the business traces).
Synthetic canary — end-to-end SLO verification
The canary is the most important operational signal: it catches pipeline stalls that no individual component metric will surface (e.g., a processor consuming from Kafka but writing to a dead Mimir ingester). Run it as a sidecar or scheduled job in each region:
# Canary pseudocode — one cycle every 60 seconds
canary_id = str(uuid4())
t0 = time.monotonic()
# 1. Emit a canary metric with a unique trace label
push_otlp_metric(
name="telemetry_canary_probe",
value=1,
labels={"canary_id": canary_id, "region": REGION},
)
# 2. Poll query endpoint until the metric appears or SLO window expires
while time.monotonic() - t0 < SLO_WINDOW_SECONDS:
result = query_mimir(f'telemetry_canary_probe{{canary_id="{canary_id}"}}')
if result:
latency = time.monotonic() - t0
push_metric("telemetry_canary_e2e_latency_seconds", latency)
break
sleep(2)
else:
fire_alert("TelemetryPipelineSLOBreach", region=REGION)
The canary exercises: gateway → Kafka produce → consumer lag → processor → Mimir write → Mimir query path. It does not cover the agent-to-gateway path — test that separately with a synthetic agent.
Alert: telemetry_canary_e2e_latency_seconds > 60 for 2 consecutive windows →
page on-call. At MAANG scale, run one canary per region per signal type
(metrics / logs / traces).
Local graph
Linked from 9 notes
8 — Self-Observability
The bootstrapping problem — a platform can't fully trust itself to tell you it's failing — and the two mechanisms that get around it: an independent out-of-band health path, and a synthetic canary that catches silent stalls no internal metric surfaces.
3.2 Layer 2: Durable Buffer (Kafka)
Layer 2 of the telemetry ingestion pipeline: the Kafka durable buffer — topic design, partitioning strategy, hot-spots, retention, retry/delivery semantics, producer config, consumer lag, and schema evolution.
3.8 Global Deployment Topology
Global deployment topology for the telemetry ingestion pipeline — regional writes vs. a global cluster, async replication to a global query tier, and agent failover.
6. Interview Anchor Points (What to Say Out Loud)
The sentences that signal principal-level thinking for the telemetry ingestion pipeline design — ready to say unprompted in an interview.
Q4: A Metric's Journey From Pod to Dashboard — Every Failure Point
Full principal-level solution: trace a single metric data point from a Kubernetes pod to a queryable dashboard, identifying every failure point along the way and how each is detected.
Q5: Adding Continuous Profiling to an Existing MELT Pipeline
Full principal-level solution: extend an existing metrics + logs + traces pipeline with continuous profiling as a fourth signal, without a full redesign.
Q7: A Region's Gateway Goes Dark — Blast Radius Walkthrough and Redesign
Full principal-level solution: walk through the consequences of a 10-minute regional ingestion gateway outage, then redesign the topology to shrink the blast radius.
Chapter 1 — Telemetry Ingestion Pipeline
Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.
Schema Validation and Rejection at the Ingestion Frontier
What the gateway actually checks before accepting a payload — structural validation vs semantic cardinality checks, why rejection has to happen before the buffer, OTLP PartialSuccess as an alternative to whole-batch rejection, and the forward-compatibility trap of validating too strictly.
Related notes
Chapter 1 — Telemetry Ingestion Pipeline
Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.
Q1: 500M Samples/Sec, Zero Drop on Rolling Deploy
Full principal-level solution: design a telemetry ingestion pipeline for 500M metric samples/sec from 100K services globally with a zero-drop guarantee during rolling deployment of the ingestion tier.
Q10: Self-Service Tenant Onboarding With Zero Platform-Team Involvement
Full principal-level solution: design a self-service tenant onboarding API for a telemetry pipeline that protects shared infrastructure from a misbehaving new tenant on day one.
Q8: Counters Resetting to Zero After an OTel SDK Upgrade
Full principal-level solution: diagnose and fix a tenant's dashboards showing counters reset to zero every few minutes after an OTel SDK upgrade, without requiring instrumentation changes.