Chapter 1 — System Design: Telemetry Ingestion Pipeline
Interview level: Principal / Staff Engineer (L6/L7 bar) Your angle: You have lived experience with Alloy → Mimir/Loki/Tempo at global scale. Use it to make every trade-off concrete, not theoretical.
How to Use This Doc
Practice the five-step format from the study guide for each section:
- Clarify requirements
- High-level design
- Deep dive
- Observability of the system itself
- Trade-offs at 10x scale
Concept Map
The load-bearing ideas, condensed into one picture. If you can redraw this from memory, you can carry the interview.
%%{init: {'theme': 'base', 'themeVariables': {
'primaryColor': '#0648d6',
'primaryTextColor': '#f5f5f5',
'primaryBorderColor': '#14213d',
'lineColor': '#94a3b8'
}}}%%
flowchart TD
subgraph L1["Layer 1 · Ingestion Frontier"]
direction TB
l1m["⚙ \nOTLP + Prometheus RW gateway\nauth / rate-limit / schema check\n\n→ 429 backpressure to agent WAL"]
l1d["⚖ \nPush vs pull\nOTLP vs remote-write"]
l1w["👁 \nTenant ID extracted here — isolation starts"]
end
subgraph L2["Layer 2 · Buffer (Kafka)"]
direction TB
l2m["⚙ \nPartition by trace_id / hash\nat-least-once + downstream dedup"]
l2d["⚖ \nKafka buffer vs direct write"]
l2w["👁 \nConsumer lag = #1 signal"]
end
subgraph L3["Layer 3 · Processing"]
direction TB
l3m["⚙ \nHyperLogLog cardinality \n tail-sample assembler\n\ndelta→cumulative \n k8s enrich at agent"]
l3d["⚖ \nHead-based vs tail-based sampling"]
l3w["👁 \nCardinality breach = tenant abuse signal"]
end
subgraph ST["Storage · Mimir"]
direction TB
stm["⚙ \nIngester hot 2h → object store cold\ncompactor dedups RF3"]
std["⚖ \nRegional writes + async replication\n(never one global cluster)"]
stw["👁 \nSynthetic canary e2e every 60s"]
end
L1 --> L2 --> L3 --> ST
classDef mech fill:#7db3ec22,stroke:#7db3ec,stroke-width:1.5px
classDef decision fill:#ffab7322,stroke:#ffab73,stroke-width:1.5px
classDef watch fill:#f0669722,stroke:#f06697,stroke-width:1.5px
classDef layer fill:transparent,stroke:#14213d,stroke-width:2px
class l1m,l2m,l3m,stm mech
class l1d,l2d,l3d,std decision
class l1w,l2w,l3w,stw watch
class L1,L2,L3,ST layer
Mnemonic — 4 stations, 3 questions each:
⚙ how it works · ⚖ the debate to raise · 👁 the signal to watch.
One exception: multi-tenancy isn’t a single station — it’s enforced at every one
(tenant ID at the gateway, cardinality budget at the processor,
X-Scope-OrgIDat storage).Never rely on a single enforcement point.
1. Clarify Requirements First
Full section moved to 1. Clarify Requirements First — the first-5-minutes clarifying questions — signal types, scale envelope, consistency/durability, multi-tenancy, and protocol — whose answers change the entire architecture.
2. High-Level Architecture
Full section moved to 2. High-Level Architecture — the producers → gateway → Kafka → processors → storage diagram, gateway links, and the push-over-pull key insight to state early.
3. Deep Dives
3.1 Layer 1: Ingestion Frontier
Full section moved to 3.1 Layer 1: Ingestion Frontier — responsibilities (each with its own dedicated companion note: protocol termination, TLS offload, authentication, tenant identification and routing, rate limiting, schema validation), the Layer 1 concept diagram, the fan-in problem at 100K+ agents, protocol negotiation, batching, and backpressure.
3.2 Layer 2: Durable Buffer (Kafka)
Full section moved to 3.2 Layer 2: Durable Buffer (Kafka) — topic design, partitioning strategy and hot-spots, retention, retry policies and delivery semantics, producer configuration, consumer lag as the scaling trigger, and schema evolution.
3.3 Layer 3: Processing / Enrichment
Full section moved to 3.3 Layer 3: Processing / Enrichment — the metric processor and cardinality enforcement, tail-based sampling and the span assembler, the log processor, metric temporality (delta vs. cumulative), and Kubernetes metadata enrichment.
3.4 Scaling Each Layer
Full section moved to 3.4 Scaling Each Layer — the scaling unit and trigger for the gateway, Kafka, each processor type, and storage.
3.5 Failure Modes and Mitigations
Full section moved to 3.5 Failure Modes and Mitigations — what breaks at each layer, its impact, and the mitigation, from a gateway pod crash through to clock skew between agents.
3.6 Multi-Tenancy
Full section moved to 3.6 Multi-Tenancy — the isolation layers from network inbound to storage, and the quota enforcement points at the gateway, processor, and storage.
3.7 Data Tiering and Compaction (Mimir/Thanos)
Full section moved to 3.7 Data Tiering and Compaction — the ingester-to-object-store journey, compaction levels, vertical compaction/dedup, and compaction storms.
3.8 Global Deployment Topology
Full section moved to 3.8 Global Deployment Topology — regional writes vs. a global cluster, async replication to a global query tier, and agent failover.
4. Observability of the Pipeline Itself
Full section moved to 4. Observability of the Pipeline Itself — what to instrument at each layer, the pipeline’s own SLOs, distributed tracing of the pipeline itself, and the synthetic canary that catches stalls no component metric surfaces.
5. Trade-offs at 10x Scale
Full section moved to 5. Trade-offs at 10x Scale — Kafka vs. direct write, trace-assembly sharding, schema-on-read vs. write, sampling strategy, protocol choice, and push vs. pull.
6. Interview Anchor Points (What to Say Out Loud)
Full section moved to 6. Interview Anchor Points (What to Say Out Loud) — the sentences that signal principal-level thinking, ready to say unprompted.
7. Component Map (What Exists in the Wild)
Full section moved to 7. Component Map (What Exists in the Wild) — OSS vs. managed options per layer, and where ShipSolid’s own production experience maps onto each.
8. Quick-Reference Cheat Sheet
Full section moved to 8. Quick-Reference Cheat Sheet — the one-line answer for every load-bearing design decision in this doc.
9. Practice Interview Questions
Full section moved to 9. Practice Interview Questions — Twelve full-length practice prompts, each linked to its own worked answer.
Local graph
Linked from 44 notes
Protocol Inventory
Every protocol referenced across the telemetry ingestion pipeline design, plus a general L7-termination reference table for the broader 'design an API gateway / load balancer' interview question.
HTTP/2 vs HTTP/1.1
Why the ingestion gateway prefers HTTP/2 (multiplexed gRPC) over HTTP/1.1 — connection reuse, binary framing, and header compression at 100K+ agent fan-in.
gRPC
What gRPC actually is underneath the shorthand this design uses it for — call shapes, status-code backpressure, deadline propagation, and the connection-level load-balancing gotcha at 100K+ agent fan-in.
TLS Offload
Terminating TLS at the ingestion frontier instead of in every backend pod — why it's a Layer 1 responsibility, what it costs in defense-in-depth, and how mTLS re-encryption closes the gap.
3 — Push-Based vs Pull-Based Ingestion
Core mental model for telemetry and data ingestion patterns — when to push, when to pull, and how to reason about the trade-offs at principal/staff interview bar.
Chapter 2 — Telemetry Pipelines
OpenTelemetry, OTLP, collectors, sampling, and aggregation as the pipeline that gets a signal from emission to storage without becoming the outage itself.
OSI Layer Model (L1-L7)
What L1 through L7 actually mean, why 'L7 gateway' and 'L4 load balancer' are load-bearing terms in system design interviews, and why this numbering is unrelated to the pipeline's own Layer 1/2/3 architecture labels.
1. Clarify Requirements First
The first-5-minutes clarifying questions for the telemetry ingestion pipeline design — signal types, scale envelope, consistency/durability, multi-tenancy, and protocol — whose answers change the entire architecture.
2. High-Level Architecture
The producers → ingestion gateway → Kafka → processors → storage diagram for the telemetry ingestion pipeline, plus the push-over-pull key insight to state early in the interview.
3.1 Layer 1: Ingestion Frontier
Layer 1 of the telemetry ingestion pipeline: the ingestion frontier — responsibilities, fan-in at 100K+ agents, protocol negotiation, batching, backpressure, and rate limiting.
3.2 Layer 2: Durable Buffer (Kafka)
Layer 2 of the telemetry ingestion pipeline: the Kafka durable buffer — topic design, partitioning strategy, hot-spots, retention, retry/delivery semantics, producer config, consumer lag, and schema evolution.
3.3 Layer 3: Processing / Enrichment
Layer 3 of the telemetry ingestion pipeline: processing and enrichment — the metric processor, cardinality enforcement, tail-based sampling, the log processor, metric temporality, and Kubernetes metadata enrichment.
Related notes
Q1: 500M Samples/Sec, Zero Drop on Rolling Deploy
Full principal-level solution: design a telemetry ingestion pipeline for 500M metric samples/sec from 100K services globally with a zero-drop guarantee during rolling deployment of the ingestion tier.
Q10: Self-Service Tenant Onboarding With Zero Platform-Team Involvement
Full principal-level solution: design a self-service tenant onboarding API for a telemetry pipeline that protects shared infrastructure from a misbehaving new tenant on day one.
Q8: Counters Resetting to Zero After an OTel SDK Upgrade
Full principal-level solution: diagnose and fix a tenant's dashboards showing counters reset to zero every few minutes after an OTel SDK upgrade, without requiring instrumentation changes.
Q9: Cut Ingestion Infrastructure Cost 40% Without Violating SLOs
Full principal-level solution: a FinOps-driven cost-reduction pass on a telemetry ingestion pipeline — where to look first, what levers exist at each layer, and what you trade away.