# Observability
All Observability notes →3 — Aggregation Composability — Why You Can't Average Percentiles
Some statistics merge correctly across shards, replicas, and time windows — sum, count, max. Percentiles do not. The distinction that decides whether a fleet-wide dashboard is trustworthy or quietly wrong.
3 — Cross-Signal Correlation
Metrics, logs, and traces are only more useful together than apart if something ties a specific instance of each of them back to the same event. The shared identifier that makes that jump possible — and what breaks when a hop in the call chain doesn't carry it.
2 — Tail Latency
Why the p99/p999 request latency matters more than the average in distributed systems — a single slow dependency in a fan-out can dominate the response time even when most calls are fast.
8 — Query Sharding
Splitting a single logical query into N independently-executable sub-queries that run in parallel and merge into one result — how Grafana Mimir and Loki answer high-cardinality queries within tight SLOs without adding more data shards.
9 — Fan-Out Metrics and Trace Shape
The Prometheus metrics that instrument a fan-out pattern (width, shard latency, aggregation, partial results, cancelled workers, hedged requests), the OTel trace waterfall shape that reveals the tail shard, and when requests fan out to multiple shards vs route to a single one.
8 — Deadline Propagation
How a client deadline must be inherited by every downstream goroutine or service call so that cancelled work stops consuming resources rather than running to completion unobserved.
2 — Shards vs Workers
Clarifies the distinction between shards (persistent data partitions) and workers (execution units): workers fan out to query shards, each concept serves a different dimension of scale.
5 — Partial Results vs Fail-Fast
Three policies for partial failures in fan-out calls: fail-fast (abort if any dependency fails), best-effort (return what succeeded and surface what's missing), and minimum quorum (K-of-N replicas). Default to fail-fast unless correctness explicitly permits partial data.
3 — Push-Based vs Pull-Based Ingestion
Core mental model for telemetry and data ingestion patterns — when to push, when to pull, and how to reason about the trade-offs at principal/staff interview bar.
1 — Observability System Design Questions
Covers the recurring system-design prompt shape — 'design a metrics/logging/tracing platform at scale' — and the tradeoffs interviewers probe for.
2 — Troubleshooting Case Studies
Covers worked troubleshooting scenarios (e.g. a collector agent pinned at 100% CPU) as a rehearsal for live debugging interview questions.
3 — Telemetry Design Exercises
Covers exercises in designing the telemetry (metrics/logs/traces/labels) for a given service from scratch, a common interview format.
4 — Incident Walkthroughs
Covers narrating a real incident timeline and RCA in interview-answer form, structured for a behavioral or systems-thinking question.
5 — Production Debugging
Covers the live-debugging interview format — given a symptom, which signal do you check first and why.
6 — Capacity Planning
Covers estimating ingest rate, series count, and storage growth for a hypothetical platform, a common quantitative interview question.
7 — Scaling to Millions of Metrics
Covers the specific architectural changes (sharding, downsampling, federation) required as series count crosses common scale thresholds.
8 — Whiteboard Architecture Problems
Covers open-ended whiteboard prompts on observability platform architecture and the tradeoff-driven answer structure interviewers expect.
9 — Maang Interview Questions
Covers a curated question bank spanning system design, troubleshooting, and behavioral formats specific to MAANG-level observability/SRE interviews.
Observability Engineering
A book-shaped table of contents for observability engineering: foundations through architecture, metrics, logging, tracing, profiling, OpenTelemetry, instrumentation, Kubernetes/cloud, data platforms, visualization, alerting, SRE integration, cost, security, platform engineering, AI-driven operations, and MAANG interview preparation — cross-linking existing prometheus/grafana-cloud/kubernetes/sre/platform-engineering notes instead of duplicating them.
# Patterns
All Patterns notes →Observability KPIs for the Fan-out / Fan-in Pattern
Observability KPIs for the Fan-out / Fan-in Pattern
Q1 Answer — Search Fan-Out Design
Worked answer to Fan-Out/Fan-In Practice Q1: partitioning, deadline propagation, partial-result policy, and instrumentation priority for a 200-shard search API at 150ms P99.
Q2 Answer — Hedging Trade-off
Worked answer to Fan-Out/Fan-In Practice Q2: the load-vs-latency math of hedged requests, when to enable them, and what to instrument first to justify the decision.
Q3 Answer — Context Cancellation Leak
Worked answer to Fan-Out/Fan-In Practice Q3: diagnosing a ghost-request leak where client-visible errors look healthy but infra cost and downstream CPU are elevated.
Q4 Answer — Aggregator Bottleneck
Worked answer to Fan-Out/Fan-In Practice Q4: min-heap merge strategy for a 500-shard top-K aggregation, its complexity, and how to keep aggregator latency from contaminating per-shard dashboards.
Q5 Answer — Sizing the Fan-Out Width
Worked answer to Fan-Out/Fan-In Practice Q5: the questions to ask and safeguards to add before accepting a design that fans out to all 8,000 tenant shards in prod.
Q6 Answer — Retry Storm
Worked answer to Fan-Out/Fan-In Practice Q6: how uncapped per-worker retries turn a transient blip into a full outage, and the retry policy that prevents it.
Q7 Answer — Backpressure and Load Shedding
Worked answer to Fan-Out/Fan-In Practice Q7: why an aggregate latency average hides a single overloaded shard, and where the fix belongs — dispatcher, worker, or shard.
Q8 Answer — Hierarchical Fan-Out
Worked answer to Fan-Out/Fan-In Practice Q8: budgeting a deadline across two nested fan-out levels and preventing partial failure from silently compounding across hops.
Q9 Answer — Validating Hedging and Deadline Propagation
Worked answer to Fan-Out/Fan-In Practice Q9: fault-injection, load testing, and canary comparison for validating a deadline-propagation and hedging rewrite before it reaches production.
01 — Monolith — Modular and Majestic
A single deployable unit with well-defined internal module boundaries. Underrated at MAANG interviews — the right answer when decomposition cost exceeds the benefit.
02 — Strangler Fig
Incrementally replace a legacy system by routing new functionality to a new implementation while the old system continues to run. Named after the fig tree that grows around and eventually replaces its host.
04 — Fan-Out / Fan-In
Decompose a request into parallel sub-tasks (fan-out), execute concurrently, then merge results (fan-in). The foundational pattern for latency-bound aggregation.
05 — Backpressure
Signal from a slow consumer to a fast producer to slow down. Prevents unbounded queue growth, OOM, and cascading overload. The foundational flow-control pattern.
07 — Circuit Breaker
A state machine (Closed → Open → Half-Open) that stops calls to a failing dependency before they cascade. The foundational resilience pattern for distributed systems.
08 — Retry with Exponential Backoff and Jitter
Retry transient failures with exponentially increasing wait times and randomised jitter to prevent thundering-herd recovery storms. The foundational pattern for resilient RPC.
09 — Bulkhead
Partition resources (thread pools, connection pools, semaphores) so that a failure or overload in one partition cannot exhaust resources for others. Limits blast radius.
10 — Hedged Requests
A tail-latency optimization that issues the same idempotent request to multiple replicas and uses whichever responds first, trading extra compute for dramatically lower P99/P999.
12 — CQRS — Command Query Responsibility Segregation
Separate the write model (commands) from the read model (queries). Unlocks independent scaling, optimised projections, and eventual-consistency trade-offs at scale.
13 — Event Sourcing
Store state as an immutable, append-only sequence of domain events. Current state is derived by replaying the log. Gives audit trail, temporal queries, and projection flexibility for free.
14 — Transactional Outbox
Atomically write to the database and publish a message by using a single local transaction. The outbox table is polled or tailed by CDC to publish reliably. Solves the dual-write problem.
15 — Saga
Manage distributed transactions across multiple services using a sequence of local transactions with compensating actions on failure. The microservices answer to 2PC.
01 — Sidecar
Co-locate a helper container with the application container to handle cross-cutting concerns — TLS, observability, auth, retries — without modifying application code.
01 — What Is a Pattern?
The history and vocabulary of pattern thinking — Christopher Alexander's pattern language, the Gang of Four's catalog, anti-patterns, and how a pattern differs from a one-off design decision.
02 — Pattern Selection & Trade-offs
How to choose between competing patterns under real forces: naming the context, weighing trade-offs and consequences, composing multiple patterns together, and recognizing when a pattern has outlived its fit.
03 — SOLID Revisited — Principal-Level Framing
SRP, OCP, LSP, ISP, and DIP reframed at architecture scale — stable dependencies and stable abstractions as the load-bearing idea, not the five-bullet mnemonic.
01 — Creational Patterns
Controlling object instantiation at the composition-root level: Singleton, Factory Method, Abstract Factory, Builder, and Prototype — when each earns its complexity and how they compose in a real dependency graph.
02 — Structural Patterns
Composing objects and classes into larger structures without duplicating behavior: Adapter, Bridge, Composite, Decorator, Facade, Flyweight, and Proxy.
03 — Behavioral Patterns
Distributing responsibility and communication between objects: Strategy, Observer, Command, Chain of Responsibility, Mediator, Memento, Interpreter, Iterator, State, Template Method, and Visitor.
01 — Layering Patterns
Layered Architecture, Hexagonal (Ports & Adapters), Onion, Clean Architecture, and Screaming Architecture — how each draws the boundary between domain logic and infrastructure, and what that boundary costs.
02 — Domain Modeling Patterns
DDD's tactical toolkit: Rich Domain Model vs. Anemic Model, Aggregate, Entity, Value Object, Repository, Factory, and Specification — the vocabulary a domain model needs to stay consistent under concurrent writes.
03 — Transaction Patterns
Unit of Work, Identity Map, Lazy Loading, and Optimistic vs. Pessimistic Locking — the patterns that keep an object graph and its persisted state from drifting apart.
04 — Integration Patterns
DTO, Gateway, Data Mapper, Service Layer, Table Data Gateway, and Active Record — the patterns that mediate between a domain model and everything outside it (databases, external services, the wire).
01 — Dependency Management Patterns
Constructor, Method, and Property Injection, Service Locator, and IoC Containers — how a dependency graph gets assembled and why constructor injection is the default the others are exceptions to.
02 — Extensibility Patterns
Plugin Architecture, Module Pattern, Strategy Registration, Reflection, and Dynamic Loading — how a system stays open to new behavior without recompiling its core.
03 — Service Decomposition
Decomposing by business capability, domain, or bounded context, and building self-contained systems — the remaining decomposition axes beyond the Monolith-vs-Strangler-Fig choice already covered in this book.
06 — Service Communication
Request-Response, Async Messaging, Event Streaming, Pub/Sub, and RPC/gRPC — the communication styles a service boundary can choose between, beyond the Fan-Out/Fan-In and Backpressure patterns already covered in this book.
11 — Reliability Patterns (Microservice Building Blocks)
Timeout, Rate Limiter, Fallback, and Adaptive Concurrency — the remaining resilience building blocks beyond Circuit Breaker, Retry, Bulkhead, and Hedged Requests, which already have their own chapters in this book.
16 — Data Patterns (Microservice)
Database-per-Service, Shared Database, and Materialized View — the remaining data-ownership patterns beyond CQRS, Event Sourcing, Outbox, and Saga, which already have their own chapters in this book.
01 — Consensus Patterns
Raft, Paxos, Leader Election, and Quorum — how a distributed system agrees on a single value or leader despite node failures and network partitions.
02 — Coordination Patterns
Distributed Lock, Lease, Heartbeat, Membership, and Gossip — the primitives nodes use to coordinate without a single point of failure.
03 — Replication Patterns
Leader-Follower, Leaderless, Multi-Leader, and Read Replica replication — how copies of the same data stay available and how they diverge under partition.
04 — Consistency Patterns
Strong, Eventual, Causal, Read-Your-Writes, and Monotonic Reads consistency models — what guarantee a client actually gets, and what it costs in latency and availability.
01 — Message Broker Patterns
Queue, Topic, Fan-out, Dead Letter Queue, and Delayed Queue — the delivery topologies a message broker offers and when each one fits.
02 — Event Patterns
Event Notification vs. Event-Carried State Transfer, and Choreography vs. Orchestration — how services stay decoupled through events, building on the Event Sourcing and Saga patterns already covered in this book.
03 — Streaming Patterns
Windowing, Watermarks, and Exactly-Once vs. At-Least-Once vs. At-Most-Once delivery semantics — the vocabulary for reasoning about correctness in a stream processing pipeline.
01 — REST Patterns
Resource modeling, pagination, filtering, HATEOAS, and versioning — the design decisions that separate a well-behaved REST API from an RPC call wearing HTTP verbs.
02 — RPC Patterns
gRPC and Protobuf, and the unary, streaming, and bidirectional-streaming call shapes — when RPC's tighter contract and lower overhead beat REST's looser one.
03 — API Gateway Patterns
Gateway, Backend for Frontend, Aggregation, and Federation — where cross-cutting API concerns (auth, rate limiting, fan-out) belong relative to the services behind them.
01 — Database Design Patterns
Normalization, denormalization, sharding, partitioning, and archiving — the structural decisions that determine how a database scales and how expensive its queries stay.
02 — Caching Patterns
Cache-Aside, Read-Through, Write-Through, Write-Back, and Refresh-Ahead — the five ways an application and its cache can disagree about who owns writing to the source of truth.
03 — Search Patterns
Inverted Index, Secondary Index, Bloom Filter, and Skip List — the data structures that make search and existence-checks fast at scale.
02 — Kubernetes Patterns
Ambassador, Adapter, Init Container, and Operator — the remaining multi-container and control-plane patterns beyond Sidecar, which already has its own chapter in this book.
03 — Cloud Infrastructure Patterns
Immutable Infrastructure, Auto Scaling, and the Blue-Green, Canary, and Rolling Update deployment strategies — how infrastructure changes roll out without a full-stop cutover.
04 — Multi-Region Patterns
Active-Passive, Active-Active, Geo-Replication, and Traffic Steering — the patterns that keep a system available when an entire region fails.
01 — Monitoring Patterns
RED, USE, the Four Golden Signals, and Saturation — the metric frameworks that decide what to measure on a service before an incident forces the question.
02 — Logging Patterns
Structured Logging, Correlation IDs, Log Sampling, and Log Aggregation — how logs stay searchable and affordable at scale instead of becoming a second, worse metrics system.
03 — Tracing Patterns
Distributed Tracing, Context Propagation, and Tail vs. Head Sampling — how a single request's path across services becomes reconstructable instead of a pile of disconnected spans.
04 — Alerting Patterns
Multi-window Burn Rate alerts, SLO Alerts, Composite Alerts, and Noise Reduction — the alerting design that pages on user-facing pain instead of every internal wobble.
01 — Resilience Patterns
Graceful Degradation, Load Shedding, Fail Fast, and Self-Healing — the system-level resilience postures that sit above any single pattern like Circuit Breaker or Bulkhead.
02 — Availability Patterns
High Availability, Disaster Recovery, Backup/Restore, and Chaos Engineering — the practices that turn an availability target into something actually tested, not just assumed.
03 — Scalability Patterns
Horizontal vs. Vertical Scaling, Elasticity, and Partitioning — the levers for handling more load, and why horizontal scaling is usually the one worth designing for first.
01 — Authentication Patterns
OAuth, OpenID Connect, API Keys, and Mutual TLS — proving identity between a client and a service, and between services themselves.
02 — Authorization Patterns
RBAC, ABAC, ReBAC, and Capability-Based Security — the models for deciding what an authenticated identity is allowed to do.
03 — Secure Communication Patterns
Zero Trust, Service Mesh, and Secrets Management — how a system stops trusting the network itself and starts verifying every call.
01 — Threading Patterns
Thread Pool, Producer-Consumer, and Reader-Writer — the foundational patterns for sharing work and data across threads without corrupting either.
02 — Lock-Free Programming
Compare-and-Swap, Atomic Variables, and Wait-Free vs. Lock-Free Queues — trading locks for retry loops to avoid blocking, priority inversion, and deadlock.
03 — Async Patterns
Futures, Promises, Reactive Streams, and the Actor Model — the abstractions for composing asynchronous work without callback-driven spaghetti.
01 — LLM System Patterns
RAG, Multi-Agent, Tool Calling, Planner-Executor, Reflection, and Memory — the pattern vocabulary for composing LLM calls into a system, at the design layer above any single framework.
02 — AI Infrastructure Patterns
Model Routing, Prompt Chaining, Guardrails, Human-in-the-Loop, and Evaluation Pipelines — the operational scaffolding that makes an LLM system reliable enough to run in production.
01 — Team Topologies
Stream-Aligned, Platform, Enabling, and Complicated-Subsystem teams — the four fundamental team types and the interaction modes between them.
02 — Conway's Law
Inverse Conway Maneuver, Team APIs, and Cognitive Load — why a system's architecture mirrors its org chart, and how to design the org chart on purpose instead of by accident.
03 — Engineering Leadership Patterns
RFC Process, ADRs, Design Reviews, Technical Governance, and Architecture Council — the decision-making scaffolding that lets an organization make architecture calls without every one becoming a meeting.
01 — Decision-Making Patterns
Buy vs. Build, Build vs. Platform, Sync vs. Async, SQL vs. NoSQL, and Monolith vs. Microservices — recurring architecture forks and the questions that actually resolve them.
02 — Trade-off Analysis Frameworks
CAP, PACELC, Latency vs. Throughput, Cost vs. Reliability, and Simplicity vs. Flexibility — the frameworks for making a trade-off explicit instead of leaving it implicit in the design.
01 — Combining Patterns
Pattern layering, pattern synergy, and pattern conflicts — how patterns interact once more than one is applied to the same system, and where two reasonable patterns pull against each other.
02 — Anti-Patterns
Distributed Monolith, Shared Database, God Object, Big Ball of Mud, Chatty Services, Golden Hammer, and Vendor Lock-in — the failure modes that look like patterns but are really a pattern applied without its context.
03 — Pattern Case Studies
How Amazon, Google, Netflix, Uber, Stripe, LinkedIn, and Cloudflare have combined these patterns in production — to be researched and written company by company, not as a single pass.
Patterns
A book-shaped table of contents for reusable engineering patterns spanning object-oriented design, enterprise architecture, distributed systems, messaging, APIs, cloud infrastructure, observability, security, concurrency, AI/agentic systems, and organizational design — grounded in production experience at scale.
# Agentic Ai Engineering
All Agentic Ai Engineering notes →# Agentic Ai Projects And Mastery
All Agentic Ai Projects And Mastery notes →# Ai Architecture And System Design
All Ai Architecture And System Design notes →# Ai Foundations
All Ai Foundations notes →# Aptitude
All Aptitude notes →# Building Agentic Systems
All Building Agentic Systems notes →# Ci Cd
All Ci Cd notes →# Data Engineering
All Data Engineering notes →# Data Structures Algorithms
All Data Structures Algorithms notes →# Dbms
All Dbms notes →# Grafana Cloud
All Grafana Cloud notes →# Infrastructure Platform Engineering
All Infrastructure Platform Engineering notes →# Internal Developer Platforms
All Internal Developer Platforms notes →# Kubernetes Platform Engineering
All Kubernetes Platform Engineering notes →# Kubernetes
All Kubernetes notes →# Low Level Design
All Low Level Design notes →# Networks
All Networks notes →OSI Layer Model (L1-L7)
What L1 through L7 actually mean, why 'L7 gateway' and 'L4 load balancer' are load-bearing terms in system design interviews, and why this numbering is unrelated to the pipeline's own Layer 1/2/3 architecture labels.
Protocol Inventory
Every protocol referenced across the telemetry ingestion pipeline design, plus a general L7-termination reference table for the broader 'design an API gateway / load balancer' interview question.
HTTP/2 vs HTTP/1.1
Why the ingestion gateway prefers HTTP/2 (multiplexed gRPC) over HTTP/1.1 — connection reuse, binary framing, and header compression at 100K+ agent fan-in.
gRPC
What gRPC actually is underneath the shorthand this design uses it for — call shapes, status-code backpressure, deadline propagation, and the connection-level load-balancing gotcha at 100K+ agent fan-in.
TLS Offload
Terminating TLS at the ingestion frontier instead of in every backend pod — why it's a Layer 1 responsibility, what it costs in defense-in-depth, and how mTLS re-encryption closes the gap.
Computer Networks
A book-shaped table of contents for computer networking, from first principles to production systems: Ethernet through IP, TCP/UDP/QUIC, DNS, the HTTP ecosystem, security, cloud/Kubernetes networking, performance engineering, observability/debugging, and distributed-systems networking — cross-linking existing kubernetes/sre/system-design/tech notes instead of duplicating them.
# Object Oriented Programming
All Object Oriented Programming notes →# Operating System
All Operating System notes →# Philosophy
All Philosophy notes →# Platform Engineering Fundamentals
All Platform Engineering Fundamentals notes →# Production Agent Systems
All Production Agent Systems notes →# Productivity
All Productivity notes →# Projects
All Projects notes →# Prometheus
All Prometheus notes →# Sre
All Sre notes →# System Design
All System Design notes →Chapter 4 — Capacity Planning System
Growth modeling, headroom analysis, cost vs. reliability simulation.
Chapter 1 — Telemetry Ingestion Pipeline
Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.
1. Clarify Requirements First
The first-5-minutes clarifying questions for the telemetry ingestion pipeline design — signal types, scale envelope, consistency/durability, multi-tenancy, and protocol — whose answers change the entire architecture.
2. High-Level Architecture
The producers → ingestion gateway → Kafka → processors → storage diagram for the telemetry ingestion pipeline, plus the push-over-pull key insight to state early in the interview.
3.1 Layer 1: Ingestion Frontier
Layer 1 of the telemetry ingestion pipeline: the ingestion frontier — responsibilities, fan-in at 100K+ agents, protocol negotiation, batching, backpressure, and rate limiting.
3.2 Layer 2: Durable Buffer (Kafka)
Layer 2 of the telemetry ingestion pipeline: the Kafka durable buffer — topic design, partitioning strategy, hot-spots, retention, retry/delivery semantics, producer config, consumer lag, and schema evolution.
3.3 Layer 3: Processing / Enrichment
Layer 3 of the telemetry ingestion pipeline: processing and enrichment — the metric processor, cardinality enforcement, tail-based sampling, the log processor, metric temporality, and Kubernetes metadata enrichment.
3.4 Scaling Each Layer
Scaling unit and trigger for every layer of the telemetry ingestion pipeline, from the ingestion gateway through to storage.
3.5 Failure Modes and Mitigations
Failure modes and mitigations across the telemetry ingestion pipeline — gateway crashes, broker failure, processor crashes, span explosion, cardinality rejection, storage saturation, and clock skew.
3.6 Multi-Tenancy
Multi-tenancy isolation layers and quota enforcement points across the telemetry ingestion pipeline, from network inbound to the storage write path.
3.7 Data Tiering and Compaction (Mimir/Thanos)
Data tiering and compaction in Mimir/Thanos — the ingester-to-object-store journey, compaction levels, vertical compaction/dedup, compaction storms, and the config knobs that control them.
3.8 Global Deployment Topology
Global deployment topology for the telemetry ingestion pipeline — regional writes vs. a global cluster, async replication to a global query tier, and agent failover.
4. Observability of the Pipeline Itself
What to instrument at every layer of the telemetry ingestion pipeline, the pipeline's own SLOs, distributed tracing of the pipeline itself, and the synthetic canary that catches stalls no component metric surfaces.
5. Trade-offs at 10x Scale
The 'what would you do differently at 10x' trade-off questions for the telemetry ingestion pipeline: Kafka vs. direct write, trace-assembly sharding, schema-on-read vs. write, sampling strategy, protocol choice, and push vs. pull.
6. Interview Anchor Points (What to Say Out Loud)
The sentences that signal principal-level thinking for the telemetry ingestion pipeline design — ready to say unprompted in an interview.
7. Component Map (What Exists in the Wild)
OSS and managed-SaaS options for every layer of the telemetry ingestion pipeline, mapped against ShipSolid's own production experience.
8. Quick-Reference Cheat Sheet
One-line answers for every load-bearing design decision in the telemetry ingestion pipeline — the last thing to review before an interview.
9. Practice Interview Questions
Twelve full-length practice prompts for the telemetry ingestion pipeline design, each linked to its own worked, principal-level answer.
Authentication at the Ingestion Frontier: mTLS, Bearer Tokens, API Keys
How the gateway proves an agent's credential is valid — mTLS handshake validation, JWT bearer token signature checks, and API key lookups — with the revocation-speed vs operational-complexity trade-off between them, and credential rotation at 10M-agent scale.
Head vs. Tail Sampling for Distributed Traces
The sampling decision point determines whether a trace pipeline needs to buffer spans in memory — head-based decides at trace start, tail-based decides after the trace completes.
Protocol Termination at the Ingestion Frontier
What actually happens where the wire protocol ends — TCP/TLS handoff, HTTP/2 frame demux, gRPC message decode, protobuf deserialization — and why L4 vs L7 termination and connection-lifecycle tuning are the load-bearing decisions here, not the crypto itself.
Q1: 500M Samples/Sec, Zero Drop on Rolling Deploy
Full principal-level solution: design a telemetry ingestion pipeline for 500M metric samples/sec from 100K services globally with a zero-drop guarantee during rolling deployment of the ingestion tier.
Q10: Self-Service Tenant Onboarding With Zero Platform-Team Involvement
Full principal-level solution: design a self-service tenant onboarding API for a telemetry pipeline that protects shared infrastructure from a misbehaving new tenant on day one.
Q11: Redesigning the Ingestion Frontier for a Compromised-Agent Threat Model
Full principal-level solution: redesign the telemetry ingestion frontier assuming a compromised agent sends malformed and adversarial payloads — oversized batches, spoofed tenant IDs, garbage label values.
Q12: Exactly-Once for One Billing Tenant While Others Stay At-Least-Once
Full principal-level solution: support exactly-once ingestion for a single billing-critical tenant in a shared pipeline that is at-least-once everywhere else, and account for what it costs.
Q2: Cardinality Storm — Detect and Mitigate Without Affecting Other Tenants
Full principal-level solution: a tenant sends 50M unique label combinations/minute causing TSDB compaction storms — design detection and mitigation that isolates the blast radius to that tenant.
Q3: Trace Sampling Loses Spans During Incident Peaks — Redesign
Full principal-level solution: the trace pipeline drops spans exactly when incidents spike trace volume — diagnose the failure mode and redesign the tail-sampling pipeline to survive it.
Q4: A Metric's Journey From Pod to Dashboard — Every Failure Point
Full principal-level solution: trace a single metric data point from a Kubernetes pod to a queryable dashboard, identifying every failure point along the way and how each is detected.
Q5: Adding Continuous Profiling to an Existing MELT Pipeline
Full principal-level solution: extend an existing metrics + logs + traces pipeline with continuous profiling as a fourth signal, without a full redesign.
Q6: Compactor Queue Backing Up During a Multi-Tenant Flush
Full principal-level solution: diagnose a compactor backlog causing query latency spikes during a large multi-tenant flush, and mitigate it without pausing ingestion.
Q7: A Region's Gateway Goes Dark — Blast Radius Walkthrough and Redesign
Full principal-level solution: walk through the consequences of a 10-minute regional ingestion gateway outage, then redesign the topology to shrink the blast radius.
Q8: Counters Resetting to Zero After an OTel SDK Upgrade
Full principal-level solution: diagnose and fix a tenant's dashboards showing counters reset to zero every few minutes after an OTel SDK upgrade, without requiring instrumentation changes.
Q9: Cut Ingestion Infrastructure Cost 40% Without Violating SLOs
Full principal-level solution: a FinOps-driven cost-reduction pass on a telemetry ingestion pipeline — where to look first, what levers exist at each layer, and what you trade away.
Rate Limiting Architecture: Token Bucket, Gossip, and Envoy Global Limits
Three ways to enforce rate limits across a replicated gateway fleet — centralized Redis token bucket, decentralized gossip-based estimation, and Envoy's sidecar-plus-global-service pattern — with the precision/SPOF/latency trade-offs between them.
Retry Policies and the Delivery Semantics They Produce
Every retry decision is made along three axes — trigger, backoff, budget — before delivery semantics even enter the picture. At-least-once, at-most-once, and exactly-once are the accumulated side effect of those decisions at every hop, not a separate design choice.
Schema Validation and Rejection at the Ingestion Frontier
What the gateway actually checks before accepting a payload — structural validation vs semantic cardinality checks, why rejection has to happen before the buffer, OTLP PartialSuccess as an alternative to whole-batch rejection, and the forward-compatibility trap of validating too strictly.
Telemetry Gateways: Protocol-Specific Ingestion Points
The ingestion frontier is a fleet of protocol-specific gateways — OTLP, Prometheus remote-write, Syslog, Kafka, and legacy tracing/metrics protocols — each terminating a different producer's wire format before a shared auth/rate-limit/tenant-routing layer.
Tenant Identification and Routing at the Ingestion Frontier
How the gateway decides whose data this is — cert/API-key-derived tenant identity, propagation as a Kafka header and X-Scope-OrgID, and the shared-topic-with-filter vs per-tenant-topic vs shuffle-sharded-pool routing trade-off.
Chapter 2 — Metrics Storage (TSDB)
Write amplification, chunk encoding, compaction, cardinality explosion.
Chapter 3 — Log Aggregation System
Structured vs. unstructured, schema-on-read vs. schema-on-write, deduplication.
Chapter 4 — Distributed Tracing Backend
Trace assembly from spans, tail-based vs. head-based sampling.
Chapter 5 — OpenTelemetry Collector Pipeline
Multi-pipeline routing, processor chaining, exporter fan-out.
Chapter 6 — Multi-tenant Observability Platform
Tenant isolation, quota enforcement, cost attribution.
Chapter 7 — SLO / Error Budget Tracking System
Burn rate calculation, multi-window alerting, budget ledger.
Chapter 8 — Distributed Message Queue (Kafka-like)
Partitioning, consumer groups, at-least-once vs. exactly-once.
Chapter 9 — Distributed Key-Value Store (DynamoDB-like)
Consistent hashing, replication, read/write quorum.
Chapter 10 — Stream Processing System (Flink-like)
Watermarks, windowing, stateful operators, exactly-once.
Chapter 11 — Rate Limiter (Distributed)
Token bucket, leaky bucket, sliding window, Redis-backed global limiter.
Chapter 12 — Consensus & Leader Election
Raft/Paxos, split-brain prevention, fencing tokens.
Chapter 13 — Runbook Automation / AIOps Engine
LLM-powered diagnosis, trigger-action mappings, safety guardrails.
Chapter 14 — Observability Data Lake
Cold/warm/hot tiers, Parquet storage, query federation (Thanos/Cortex/Mimir).
Chapter 15 — Cost Optimization Pipeline
Adaptive sampling, metric drop rules, cardinality-aware ingestion.
Chapter 16 — Incident Management Platform
Alert correlation, incident lifecycle, escalation, runbook automation.
Chapter 17 — Distributed Search Engine (Elasticsearch-like)
Inverted indexes, sharding, near-real-time indexing.
System Design
Principal/Staff-level system design reference collection for MAANG interview preparation — observability pipelines, distributed systems, reliability engineering, and beyond.