Appears in: Telemetry Ingestion Pipeline — this is §1 of the full design, split into its own file so the root stays a table of contents.
1. Clarify Requirements First
The first 5 minutes of the interview is yours to steer. Ask these; the answers change the entire architecture.
Signal types
- Metrics only, or metrics + logs + traces (unified MELT pipeline)?
- Are traces sampled at head or tail?
This determines whether spans need to be held in memory for assembly.
- Any profiling signals (continuous profiling, eBPF)?
Scale envelope
| Dimension | Small (startup) | Mid (Netflix-class) | Large (hyper-scale) |
|---|---|---|---|
| Agent count | ~1K | ~100K | ~10M+ |
| Metric series (active) | ~10M | ~1B | ~100B+ |
| Ingest rate | ~1M pts/sec | ~500M pts/sec | ~50B pts/sec |
| Trace spans/sec | ~100K | ~10M | ~1B |
| Log lines/sec | ~1M | ~500M | ~50B |
At MAANG interviews, assume Netflix/Google scale unless told otherwise.
Consistency and durability
- Can we drop data during a rolling restart? (Usually: yes for metrics, no for billing traces.)
- What is the maximum acceptable ingestion lag before data lands queryable? (SLO: typically < 60s for metrics, < 5 min for traces.)
- Retry policy: at-least-once or exactly-once? Exactly-once is expensive — push back unless there is a billing requirement.
Multi-tenancy
- Single tenant (internal platform) or multi-tenant SaaS?
- Per-tenant quota enforcement? Isolation at the storage layer or earlier?
Protocol
- Must support OTLP? Prometheus remote-write? Both? Datadog agent wire format?
- Is the interviewer implying a greenfield (pick the best protocol) or a brownfield (must accept legacy agents)?
Local graph
Related notes
Chapter 1 — Telemetry Ingestion Pipeline
Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.
Q1: 500M Samples/Sec, Zero Drop on Rolling Deploy
Full principal-level solution: design a telemetry ingestion pipeline for 500M metric samples/sec from 100K services globally with a zero-drop guarantee during rolling deployment of the ingestion tier.
Q10: Self-Service Tenant Onboarding With Zero Platform-Team Involvement
Full principal-level solution: design a self-service tenant onboarding API for a telemetry pipeline that protects shared infrastructure from a misbehaving new tenant on day one.
Q8: Counters Resetting to Zero After an OTel SDK Upgrade
Full principal-level solution: diagnose and fix a tenant's dashboards showing counters reset to zero every few minutes after an OTel SDK upgrade, without requiring instrumentation changes.