# Observability
All Observability notes →3 — Aggregation Composability — Why You Can't Average Percentiles
Some statistics merge correctly across shards, replicas, and time windows — sum, count, max. Percentiles do not. The distinction that decides whether a fleet-wide dashboard is trustworthy or quietly wrong.
3 — Cross-Signal Correlation
Metrics, logs, and traces are only more useful together than apart if something ties a specific instance of each of them back to the same event. The shared identifier that makes that jump possible — and what breaks when a hop in the call chain doesn't carry it.
2 — Tail Latency
Why the p99/p999 request latency matters more than the average in distributed systems — a single slow dependency in a fan-out can dominate the response time even when most calls are fast.
8 — Query Sharding
Splitting a single logical query into N independently-executable sub-queries that run in parallel and merge into one result — how Grafana Mimir and Loki answer high-cardinality queries within tight SLOs without adding more data shards.
9 — Fan-Out Metrics and Trace Shape
The Prometheus metrics that instrument a fan-out pattern (width, shard latency, aggregation, partial results, cancelled workers, hedged requests), the OTel trace waterfall shape that reveals the tail shard, and when requests fan out to multiple shards vs route to a single one.
8 — Deadline Propagation
How a client deadline must be inherited by every downstream goroutine or service call so that cancelled work stops consuming resources rather than running to completion unobserved.
2 — Shards vs Workers
Clarifies the distinction between shards (persistent data partitions) and workers (execution units): workers fan out to query shards, each concept serves a different dimension of scale.
5 — Partial Results vs Fail-Fast
Three policies for partial failures in fan-out calls: fail-fast (abort if any dependency fails), best-effort (return what succeeded and surface what's missing), and minimum quorum (K-of-N replicas). Default to fail-fast unless correctness explicitly permits partial data.
# Patterns
All Patterns notes →Observability KPIs for the Fan-out / Fan-in Pattern
Observability KPIs for the Fan-out / Fan-in Pattern
Q1 Answer — Search Fan-Out Design
Worked answer to Fan-Out/Fan-In Practice Q1: partitioning, deadline propagation, partial-result policy, and instrumentation priority for a 200-shard search API at 150ms P99.
Q2 Answer — Hedging Trade-off
Worked answer to Fan-Out/Fan-In Practice Q2: the load-vs-latency math of hedged requests, when to enable them, and what to instrument first to justify the decision.
Q3 Answer — Context Cancellation Leak
Worked answer to Fan-Out/Fan-In Practice Q3: diagnosing a ghost-request leak where client-visible errors look healthy but infra cost and downstream CPU are elevated.
Q4 Answer — Aggregator Bottleneck
Worked answer to Fan-Out/Fan-In Practice Q4: min-heap merge strategy for a 500-shard top-K aggregation, its complexity, and how to keep aggregator latency from contaminating per-shard dashboards.
Q5 Answer — Sizing the Fan-Out Width
Worked answer to Fan-Out/Fan-In Practice Q5: the questions to ask and safeguards to add before accepting a design that fans out to all 8,000 tenant shards in prod.
Q6 Answer — Retry Storm
Worked answer to Fan-Out/Fan-In Practice Q6: how uncapped per-worker retries turn a transient blip into a full outage, and the retry policy that prevents it.
Q7 Answer — Backpressure and Load Shedding
Worked answer to Fan-Out/Fan-In Practice Q7: why an aggregate latency average hides a single overloaded shard, and where the fix belongs — dispatcher, worker, or shard.
Q8 Answer — Hierarchical Fan-Out
Worked answer to Fan-Out/Fan-In Practice Q8: budgeting a deadline across two nested fan-out levels and preventing partial failure from silently compounding across hops.
Q9 Answer — Validating Hedging and Deadline Propagation
Worked answer to Fan-Out/Fan-In Practice Q9: fault-injection, load testing, and canary comparison for validating a deadline-propagation and hedging rewrite before it reaches production.
01 — Monolith — Modular and Majestic
A single deployable unit with well-defined internal module boundaries. Underrated at MAANG interviews — the right answer when decomposition cost exceeds the benefit.
02 — Strangler Fig
Incrementally replace a legacy system by routing new functionality to a new implementation while the old system continues to run. Named after the fig tree that grows around and eventually replaces its host.
04 — Fan-Out / Fan-In
Decompose a request into parallel sub-tasks (fan-out), execute concurrently, then merge results (fan-in). The foundational pattern for latency-bound aggregation.
05 — Backpressure
Signal from a slow consumer to a fast producer to slow down. Prevents unbounded queue growth, OOM, and cascading overload. The foundational flow-control pattern.
07 — Circuit Breaker
A state machine (Closed → Open → Half-Open) that stops calls to a failing dependency before they cascade. The foundational resilience pattern for distributed systems.
08 — Retry with Exponential Backoff and Jitter
Retry transient failures with exponentially increasing wait times and randomised jitter to prevent thundering-herd recovery storms. The foundational pattern for resilient RPC.
09 — Bulkhead
Partition resources (thread pools, connection pools, semaphores) so that a failure or overload in one partition cannot exhaust resources for others. Limits blast radius.
10 — Hedged Requests
A tail-latency optimization that issues the same idempotent request to multiple replicas and uses whichever responds first, trading extra compute for dramatically lower P99/P999.
12 — CQRS — Command Query Responsibility Segregation
Separate the write model (commands) from the read model (queries). Unlocks independent scaling, optimised projections, and eventual-consistency trade-offs at scale.
13 — Event Sourcing
Store state as an immutable, append-only sequence of domain events. Current state is derived by replaying the log. Gives audit trail, temporal queries, and projection flexibility for free.
14 — Transactional Outbox
Atomically write to the database and publish a message by using a single local transaction. The outbox table is polled or tailed by CDC to publish reliably. Solves the dual-write problem.
15 — Saga
Manage distributed transactions across multiple services using a sequence of local transactions with compensating actions on failure. The microservices answer to 2PC.
01 — Consensus Patterns
Raft, Paxos, Leader Election, and Quorum — how a distributed system agrees on a single value or leader despite node failures and network partitions.
02 — Coordination Patterns
Distributed Lock, Lease, Heartbeat, Membership, and Gossip — the primitives nodes use to coordinate without a single point of failure.
03 — Replication Patterns
Leader-Follower, Leaderless, Multi-Leader, and Read Replica replication — how copies of the same data stay available and how they diverge under partition.
04 — Consistency Patterns
Strong, Eventual, Causal, Read-Your-Writes, and Monotonic Reads consistency models — what guarantee a client actually gets, and what it costs in latency and availability.
Patterns
A book-shaped table of contents for reusable engineering patterns spanning object-oriented design, enterprise architecture, distributed systems, messaging, APIs, cloud infrastructure, observability, security, concurrency, AI/agentic systems, and organizational design — grounded in production experience at scale.
# Operating System
All Operating System notes →1 — OS in Cloud Computing
Covers virtual machines, hypervisors, containers, microVMs, and resource isolation.
2 — Operating Systems for Kubernetes
Covers cgroups, namespaces, OverlayFS, the PID namespace, the network namespace, and the mount namespace.
3 — Operating Systems for Observability
Covers process metrics, CPU metrics, memory metrics, I/O metrics, context switches, syscalls, and eBPF observability.
# Sre
All Sre notes →1 — CAP Theorem
Why every distributed system is already choosing between consistency and availability during a partition, whether or not the team ever wrote that choice down.
10 — Distributed Caching
Cache invalidation, consistency, and the thundering-herd failure mode that turns a cache miss into a cascading origin outage.
11 — Service Discovery
How a service finds a healthy instance of its dependency at runtime, and what happens to that discovery layer's own reliability under churn.
12 — API Gateways
Centralizing auth, rate limiting, and routing at the edge — and the single point of blast radius that centralization creates in exchange.
13 — Message Brokers
At-least-once vs. exactly-once delivery, backpressure, and why the broker's own durability guarantees are usually the real SLA you're depending on.
14 — Event-Driven Architectures
The reliability trade-offs of decoupling services through events — replay, ordering, and the debugging cost of a causal chain with no single call stack.
2 — Consensus Algorithms
The problem every consensus algorithm is solving — getting a set of unreliable nodes to agree on one value — and why it's harder than it sounds.
3 — Raft
Leader election, log replication, and the safety guarantees Raft trades for being easier to reason about than Paxos.
4 — Paxos
The original consensus protocol, why it's notoriously hard to implement correctly, and where it still shows up under the hood of production systems.
5 — Distributed Transactions
Two-phase commit, saga patterns, and why 'just wrap it in a transaction' stops being an option the moment a write crosses a service boundary.
6 — Eventual Consistency
What you're actually promising a caller when a system is 'eventually consistent,' and the read-your-own-writes gaps that turn into support tickets.
7 — Leader Election
How a cluster picks a single coordinator without a coordinator, and the split-brain failure mode that shows up when the election protocol itself degrades.
8 — Distributed Locks
Why a distributed lock is a liveness and safety trade-off, not a free primitive, and the fencing tokens that keep a stale lock holder from corrupting state.
9 — Time Synchronization
Clock skew, NTP, and why 'just use timestamps to order events' quietly breaks in any system spanning more than one machine.
# System Design
All System Design notes →Chapter 1 — Distributed System Fundamentals
Why distributed computing is fundamentally about partial failure and unbounded message delay, not just "more than one machine."
Chapter 2 — Consistency Models
The spectrum from linearizability through sequential, session, and eventual consistency, and which guarantee each one actually buys you.
Chapter 3 — CAP Theorem & PACELC
Why CAP only describes behavior during a partition, and why PACELC's latency-vs-consistency trade-off matters far more often in practice.
Chapter 4 — Consensus Algorithms
How Paxos and Raft achieve agreement despite failures, and where leader election, quorums, and split-brain prevention show up in real systems.
Chapter 5 — Distributed Transactions
Two-phase and three-phase commit, the Saga pattern, outbox/inbox, and idempotency as the toolkit for correctness across service boundaries.
Chapter 6 — Data Replication
Leader-follower, multi-leader, and leaderless replication topologies, and the replication-lag trade-offs each one accepts.
Chapter 7 — Partitioning & Sharding
Hashing strategies, consistent hashing, rebalancing, and how hot partitions emerge even with a theoretically even hash function.
Chapter 8 — Distributed Message Queue (Kafka-like)
Partitioning, consumer groups, at-least-once vs. exactly-once.
Chapter 9 — Distributed Key-Value Store (DynamoDB-like)
Consistent hashing, replication, read/write quorum.
Chapter 10 — Stream Processing System (Flink-like)
Watermarks, windowing, stateful operators, exactly-once.
Chapter 11 — Rate Limiter (Distributed)
Token bucket, leaky bucket, sliding window, Redis-backed global limiter.
Chapter 12 — Consensus & Leader Election
Raft/Paxos, split-brain prevention, fencing tokens.