Notes / System Design / 15 Complete Case Studies / 01 Telemetry Ingestion Pipeline

9. Practice Interview Questions

Twelve full-length practice prompts for the telemetry ingestion pipeline design, each linked to its own worked, principal-level answer.

· §202607161608 ·
system-designobservabilitytelemetrymaang-preppractice-questions
Chapter Navigation
On This Page

Appears in: Telemetry Ingestion Pipeline — this is §9 of the full design, split into its own file so the root stays a table of contents.

9. Practice Interview Questions

  1. Design a telemetry ingestion pipeline that can ingest 500M metric samples/sec from 100K services globally. The system must never drop data during a rolling deployment of the ingestion tier.

  2. A tenant is sending 50M unique label combinations per minute and causing TSDB compaction storms. How does your pipeline detect and mitigate this without affecting other tenants?

  3. Your trace sampling pipeline is losing spans during incident peaks — exactly when you need traces most. How do you redesign it?

  4. Walk me through how a single metric data point travels from a Kubernetes pod to being queryable in a dashboard. Identify every failure point and how you’d detect it.

  5. How would you add a new signal type (continuous profiling) to an existing metrics + logs + traces pipeline without a full redesign?

  6. The compactor queue is backing up and query latency is spiking during a large multi-tenant flush. Diagnose the failure mode and describe how you’d mitigate it without pausing ingestion.

  7. One region’s ingestion gateway just went dark for 10 minutes. Walk through what happens to agents, to buffered data, and to dashboards during that window — then explain what you’d change in the design to shrink the blast radius.

  8. A tenant upgraded their OTel SDK and now every counter in their dashboard resets to zero every few minutes. Diagnose the root cause and fix it without asking the tenant to change their instrumentation.

  9. You’re asked to cut ingestion infrastructure cost by 40% without violating any SLOs. Where do you look first, and what are you willing to trade away?

  10. Design self-service tenant onboarding: a new tenant should be able to start sending telemetry via an API call with zero platform-team involvement, while the platform still protects itself from a misbehaving or malicious new tenant on day one.

  11. Assume a compromised agent is sending malformed and adversarial payloads — oversized batches, spoofed tenant IDs, garbage label values. Redesign the ingestion frontier for this threat model.

  12. You must support exactly-once ingestion for one tenant because their metric samples drive billing, while every other tenant stays at-least-once. Where in the pipeline does that requirement have to be enforced, and what does it cost you?

Local graph

Full graph →

Related notes

Chapter 1 — Telemetry Ingestion Pipeline

Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.

Q1: 500M Samples/Sec, Zero Drop on Rolling Deploy

Full principal-level solution: design a telemetry ingestion pipeline for 500M metric samples/sec from 100K services globally with a zero-drop guarantee during rolling deployment of the ingestion tier.

Q10: Self-Service Tenant Onboarding With Zero Platform-Team Involvement

Full principal-level solution: design a self-service tenant onboarding API for a telemetry pipeline that protects shared infrastructure from a misbehaving new tenant on day one.

Q8: Counters Resetting to Zero After an OTel SDK Upgrade

Full principal-level solution: diagnose and fix a tenant's dashboards showing counters reset to zero every few minutes after an OTel SDK upgrade, without requiring instrumentation changes.
← All notes
System Design notes Edit on GitHub
Next → Authentication at the Ingestion Frontier: mTLS, Bearer Tokens, API Keys