# Networks
All Networks notes →Protocol Inventory
Every protocol referenced across the telemetry ingestion pipeline design, plus a general L7-termination reference table for the broader 'design an API gateway / load balancer' interview question.
HTTP/2 vs HTTP/1.1
Why the ingestion gateway prefers HTTP/2 (multiplexed gRPC) over HTTP/1.1 — connection reuse, binary framing, and header compression at 100K+ agent fan-in.
gRPC
What gRPC actually is underneath the shorthand this design uses it for — call shapes, status-code backpressure, deadline propagation, and the connection-level load-balancing gotcha at 100K+ agent fan-in.
TLS Offload
Terminating TLS at the ingestion frontier instead of in every backend pod — why it's a Layer 1 responsibility, what it costs in defense-in-depth, and how mTLS re-encryption closes the gap.
# System Design
All System Design notes →Chapter 1 — Telemetry Ingestion Pipeline
Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.
1. Clarify Requirements First
The first-5-minutes clarifying questions for the telemetry ingestion pipeline design — signal types, scale envelope, consistency/durability, multi-tenancy, and protocol — whose answers change the entire architecture.
2. High-Level Architecture
The producers → ingestion gateway → Kafka → processors → storage diagram for the telemetry ingestion pipeline, plus the push-over-pull key insight to state early in the interview.
3.1 Layer 1: Ingestion Frontier
Layer 1 of the telemetry ingestion pipeline: the ingestion frontier — responsibilities, fan-in at 100K+ agents, protocol negotiation, batching, backpressure, and rate limiting.
3.2 Layer 2: Durable Buffer (Kafka)
Layer 2 of the telemetry ingestion pipeline: the Kafka durable buffer — topic design, partitioning strategy, hot-spots, retention, retry/delivery semantics, producer config, consumer lag, and schema evolution.
3.3 Layer 3: Processing / Enrichment
Layer 3 of the telemetry ingestion pipeline: processing and enrichment — the metric processor, cardinality enforcement, tail-based sampling, the log processor, metric temporality, and Kubernetes metadata enrichment.
3.4 Scaling Each Layer
Scaling unit and trigger for every layer of the telemetry ingestion pipeline, from the ingestion gateway through to storage.
3.5 Failure Modes and Mitigations
Failure modes and mitigations across the telemetry ingestion pipeline — gateway crashes, broker failure, processor crashes, span explosion, cardinality rejection, storage saturation, and clock skew.
3.6 Multi-Tenancy
Multi-tenancy isolation layers and quota enforcement points across the telemetry ingestion pipeline, from network inbound to the storage write path.
3.7 Data Tiering and Compaction (Mimir/Thanos)
Data tiering and compaction in Mimir/Thanos — the ingester-to-object-store journey, compaction levels, vertical compaction/dedup, compaction storms, and the config knobs that control them.
3.8 Global Deployment Topology
Global deployment topology for the telemetry ingestion pipeline — regional writes vs. a global cluster, async replication to a global query tier, and agent failover.
4. Observability of the Pipeline Itself
What to instrument at every layer of the telemetry ingestion pipeline, the pipeline's own SLOs, distributed tracing of the pipeline itself, and the synthetic canary that catches stalls no component metric surfaces.
5. Trade-offs at 10x Scale
The 'what would you do differently at 10x' trade-off questions for the telemetry ingestion pipeline: Kafka vs. direct write, trace-assembly sharding, schema-on-read vs. write, sampling strategy, protocol choice, and push vs. pull.
6. Interview Anchor Points (What to Say Out Loud)
The sentences that signal principal-level thinking for the telemetry ingestion pipeline design — ready to say unprompted in an interview.
7. Component Map (What Exists in the Wild)
OSS and managed-SaaS options for every layer of the telemetry ingestion pipeline, mapped against ShipSolid's own production experience.
8. Quick-Reference Cheat Sheet
One-line answers for every load-bearing design decision in the telemetry ingestion pipeline — the last thing to review before an interview.
9. Practice Interview Questions
Twelve full-length practice prompts for the telemetry ingestion pipeline design, each linked to its own worked, principal-level answer.
Authentication at the Ingestion Frontier: mTLS, Bearer Tokens, API Keys
How the gateway proves an agent's credential is valid — mTLS handshake validation, JWT bearer token signature checks, and API key lookups — with the revocation-speed vs operational-complexity trade-off between them, and credential rotation at 10M-agent scale.
Head vs. Tail Sampling for Distributed Traces
The sampling decision point determines whether a trace pipeline needs to buffer spans in memory — head-based decides at trace start, tail-based decides after the trace completes.
Protocol Termination at the Ingestion Frontier
What actually happens where the wire protocol ends — TCP/TLS handoff, HTTP/2 frame demux, gRPC message decode, protobuf deserialization — and why L4 vs L7 termination and connection-lifecycle tuning are the load-bearing decisions here, not the crypto itself.
Q1: 500M Samples/Sec, Zero Drop on Rolling Deploy
Full principal-level solution: design a telemetry ingestion pipeline for 500M metric samples/sec from 100K services globally with a zero-drop guarantee during rolling deployment of the ingestion tier.
Q10: Self-Service Tenant Onboarding With Zero Platform-Team Involvement
Full principal-level solution: design a self-service tenant onboarding API for a telemetry pipeline that protects shared infrastructure from a misbehaving new tenant on day one.
Q11: Redesigning the Ingestion Frontier for a Compromised-Agent Threat Model
Full principal-level solution: redesign the telemetry ingestion frontier assuming a compromised agent sends malformed and adversarial payloads — oversized batches, spoofed tenant IDs, garbage label values.
Q12: Exactly-Once for One Billing Tenant While Others Stay At-Least-Once
Full principal-level solution: support exactly-once ingestion for a single billing-critical tenant in a shared pipeline that is at-least-once everywhere else, and account for what it costs.
Q2: Cardinality Storm — Detect and Mitigate Without Affecting Other Tenants
Full principal-level solution: a tenant sends 50M unique label combinations/minute causing TSDB compaction storms — design detection and mitigation that isolates the blast radius to that tenant.
Q3: Trace Sampling Loses Spans During Incident Peaks — Redesign
Full principal-level solution: the trace pipeline drops spans exactly when incidents spike trace volume — diagnose the failure mode and redesign the tail-sampling pipeline to survive it.
Q4: A Metric's Journey From Pod to Dashboard — Every Failure Point
Full principal-level solution: trace a single metric data point from a Kubernetes pod to a queryable dashboard, identifying every failure point along the way and how each is detected.
Q5: Adding Continuous Profiling to an Existing MELT Pipeline
Full principal-level solution: extend an existing metrics + logs + traces pipeline with continuous profiling as a fourth signal, without a full redesign.
Q6: Compactor Queue Backing Up During a Multi-Tenant Flush
Full principal-level solution: diagnose a compactor backlog causing query latency spikes during a large multi-tenant flush, and mitigate it without pausing ingestion.
Q7: A Region's Gateway Goes Dark — Blast Radius Walkthrough and Redesign
Full principal-level solution: walk through the consequences of a 10-minute regional ingestion gateway outage, then redesign the topology to shrink the blast radius.
Q8: Counters Resetting to Zero After an OTel SDK Upgrade
Full principal-level solution: diagnose and fix a tenant's dashboards showing counters reset to zero every few minutes after an OTel SDK upgrade, without requiring instrumentation changes.
Q9: Cut Ingestion Infrastructure Cost 40% Without Violating SLOs
Full principal-level solution: a FinOps-driven cost-reduction pass on a telemetry ingestion pipeline — where to look first, what levers exist at each layer, and what you trade away.
Rate Limiting Architecture: Token Bucket, Gossip, and Envoy Global Limits
Three ways to enforce rate limits across a replicated gateway fleet — centralized Redis token bucket, decentralized gossip-based estimation, and Envoy's sidecar-plus-global-service pattern — with the precision/SPOF/latency trade-offs between them.
Retry Policies and the Delivery Semantics They Produce
Every retry decision is made along three axes — trigger, backoff, budget — before delivery semantics even enter the picture. At-least-once, at-most-once, and exactly-once are the accumulated side effect of those decisions at every hop, not a separate design choice.
Schema Validation and Rejection at the Ingestion Frontier
What the gateway actually checks before accepting a payload — structural validation vs semantic cardinality checks, why rejection has to happen before the buffer, OTLP PartialSuccess as an alternative to whole-batch rejection, and the forward-compatibility trap of validating too strictly.
Telemetry Gateways: Protocol-Specific Ingestion Points
The ingestion frontier is a fleet of protocol-specific gateways — OTLP, Prometheus remote-write, Syslog, Kafka, and legacy tracing/metrics protocols — each terminating a different producer's wire format before a shared auth/rate-limit/tenant-routing layer.
Tenant Identification and Routing at the Ingestion Frontier
How the gateway decides whose data this is — cert/API-key-derived tenant identity, propagation as a Kafka header and X-Scope-OrgID, and the shared-topic-with-filter vs per-tenant-topic vs shuffle-sharded-pool routing trade-off.