# Observability
All Observability notes →3 — Aggregation Composability — Why You Can't Average Percentiles
Some statistics merge correctly across shards, replicas, and time windows — sum, count, max. Percentiles do not. The distinction that decides whether a fleet-wide dashboard is trustworthy or quietly wrong.
3 — Cross-Signal Correlation
Metrics, logs, and traces are only more useful together than apart if something ties a specific instance of each of them back to the same event. The shared identifier that makes that jump possible — and what breaks when a hop in the call chain doesn't carry it.
2 — Tail Latency
Why the p99/p999 request latency matters more than the average in distributed systems — a single slow dependency in a fan-out can dominate the response time even when most calls are fast.
8 — Query Sharding
Splitting a single logical query into N independently-executable sub-queries that run in parallel and merge into one result — how Grafana Mimir and Loki answer high-cardinality queries within tight SLOs without adding more data shards.
9 — Fan-Out Metrics and Trace Shape
The Prometheus metrics that instrument a fan-out pattern (width, shard latency, aggregation, partial results, cancelled workers, hedged requests), the OTel trace waterfall shape that reveals the tail shard, and when requests fan out to multiple shards vs route to a single one.
8 — Deadline Propagation
How a client deadline must be inherited by every downstream goroutine or service call so that cancelled work stops consuming resources rather than running to completion unobserved.
2 — Shards vs Workers
Clarifies the distinction between shards (persistent data partitions) and workers (execution units): workers fan out to query shards, each concept serves a different dimension of scale.
5 — Partial Results vs Fail-Fast
Three policies for partial failures in fan-out calls: fail-fast (abort if any dependency fails), best-effort (return what succeeded and surface what's missing), and minimum quorum (K-of-N replicas). Default to fail-fast unless correctness explicitly permits partial data.
3 — Push-Based vs Pull-Based Ingestion
Core mental model for telemetry and data ingestion patterns — when to push, when to pull, and how to reason about the trade-offs at principal/staff interview bar.