Observability Engineering

If this were a book, this page is the table of contents. Each Part below is a chapter; each chapter links out to the concepts, designs, and platform notes that already exist elsewhere in this wiki instead of duplicating them. Unwritten chapters are listed as Planned rows, not empty files.

Parts

00 — Foundations of Observability

The mental models that separate observability from monitoring: control theory, the signal types available, the lifecycle a signal travels through, and the maturity stages a team or platform progresses through.

• 1 — What Observability Actually Means
• 2 — The Signals
• 3 — Telemetry Lifecycle — (stub)
• 4 — Observability Maturity Model — (stub)

01 — Observability Architecture

The platform-level architectural decisions that shape everything downstream — how data flows in, where control decisions live, and how the platform stays available and fair across tenants.

• 1 — Designing An Observability Platform — (stub)
• 2 — Data Plane vs Control Plane — (stub)
• 3 — Push-Based vs Pull-Based Ingestion
• 4 — Agent Based vs Agentless Collection — (stub)
• 5 — Edge Aggregation — (stub)
• 6 — Centralized vs Federated Observability — (stub)
• 7 — Multi-Tenancy
• 8 — Self-Observability

02 — Metrics Engineering

The time-series data model in depth — types, histograms, cardinality, storage engines, and query performance. See also the dedicated Prometheus book for full PromQL and production-operations depth.

• 1 — Time Series Fundamentals — (stub)
• 2 — Metric Types — (stub)
• 3 — Aggregation Composability — Why You Can’t Average Percentiles
• 4 — Cardinality Management — (stub)
• 5 — Label & Attribute Schema Design
• 6 — Recording Rules — (stub)
• 7 — Metrics Storage (TSDB)
• 8 — Query Sharding

03 — Logging Engineering

Structured logging, schema design, correlation, and the pipeline/cost/search concerns specific to log data at scale.

• 1 — Structured Logging — (stub)
• 2 — Log Schemas — (stub)
• 3 — Cross-Signal Correlation
• 4 — Log Pipelines — (stub)
• 5 — Log Sampling — (stub)
• 6 — Log Retention — (stub)
• 7 — Cost Optimization — (stub)
• 8 — Log Aggregation

04 — Distributed Tracing

Why tracing exists, how a trace is modeled and propagated, sampling-strategy tradeoffs, and how stored traces support service-graph and critical-path analysis. See also Tempo and Jaeger for backend-specific detail.

• 1 — Why Tracing Exists — (stub)
• 2 — Trace Context — (stub)
• 3 — Span Modeling — (stub)
• 4 — Context Propagation — (stub)
• 5 — Trace Sampling — (stub)
• 6 — Tail Sampling — (stub)
• 7 — Distributed Tracing Backend
• 8 — Service Graphs — (stub)
• 9 — Fan-Out Metrics and Trace Shape

05 — Continuous Profiling

Sampling-based CPU, memory, heap, and concurrency profiling, and what changes when profiling runs continuously in production rather than on demand.

• 1 — CPU Profiling — (stub)
• 2 — Memory Profiling — (stub)
• 3 — Heap Analysis — (stub)
• 4 — Goroutines and Threads — (stub)
• 5 — Continuous Profiling

06 — OpenTelemetry

OpenTelemetry’s architecture end to end — protocol, SDK internals, instrumentation strategy, semantic conventions, and Collector pipeline design and scaling.

• 1 — OpenTelemetry SDKs & Semantic Conventions
• 2 — OTLP Protocol — (stub)
• 3 — SDK Internals — (stub)
• 4 — Auto vs. Manual Instrumentation
• 5 — Manual Instrumentation — (stub)
• 6 — Semantic Conventions — (stub)
• 7 — Resources — (stub)
• 8 — Deadline Propagation
• 9 — OTel Collector Pipeline Design
• 10 — Collector Pipelines — (stub)
• 11 — Processors — (stub)
• 12 — Exporters — (stub)
• 13 — Connectors — (stub)
• 14 — Scaling Collectors — (stub)

07 — Instrumentation Patterns

A catalog of instrumentation approaches across common workload shapes — APIs, microservices, messaging, data stores, Kubernetes, serverless, and batch/background work.

• 1 — Instrumenting Web APIs — (stub)
• 2 — Microservices — (stub)
• 3 — Messaging Systems — (stub)
• 4 — Databases — (stub)
• 5 — Caches — (stub)
• 6 — Kubernetes Workloads — (stub)
• 7 — Serverless — (stub)
• 8 — Batch Jobs — (stub)
• 9 — Background Workers — (stub)

08 — Kubernetes Observability

Kubernetes-native telemetry sources from control plane to container runtime, plus service-mesh and eBPF-based collection. See also the Kubernetes book’s Observability Part for kubectl-level operational detail.

• 1 — Kubernetes Metrics — (stub)
• 2 — Control Plane Monitoring — (stub)
• 3 — Node Monitoring — (stub)
• 4 — Pod Monitoring — (stub)
• 5 — Cluster Events — (stub)
• 6 — Container Runtime — (stub)
• 7 — Service Mesh Observability — (stub)
• 8 — eBPF Based Observability — (stub)

09 — Cloud Observability

Native telemetry surfaces across AWS, Azure, and Google Cloud, and the added complexity of hybrid and multi-cloud environments.

• 1 — AWS — (stub)
• 2 — Azure — (stub)
• 3 — Google Cloud — (stub)
• 4 — Hybrid Cloud — (stub)
• 5 — Multi Cloud — (stub)

10 — Observability Data Platforms

The storage backends underneath the signals covered elsewhere in this book — Prometheus/Mimir for metrics, Loki for logs, Tempo for traces, Pyroscope for profiles, and the Elasticsearch/ClickHouse/OpenSearch family for general-purpose event storage. See also grafana-cloud/README.md for Grafana Cloud-specific operational depth on Mimir/Loki/Tempo/Pyroscope.

• 1 — Prometheus — (stub)
• 2 — Shards vs Workers
• 3 — Loki — (stub)
• 4 — Tempo — (stub)
• 5 — Pyroscope — (stub)
• 6 — Elasticsearch — (stub)
• 7 — Clickhouse — (stub)
• 8 — Opensearch — (stub)

11 — Visualization

Dashboard design principles and the golden-signals/RED/USE framing that keeps a dashboard answering real questions, across executive, engineering, and business-facing audiences.

• 1 — Dashboard Design
• 2 — Tail Latency
• 3 — RED Method — (stub)
• 4 — USE Method — (stub)
• 5 — Executive Dashboards — (stub)
• 6 — Engineering Dashboards — (stub)
• 7 — Business Observability — (stub)

12 — Alert Engineering

Alert design philosophy — symptom-based alerting, SLO-derived thresholds, deduplication and routing — and the on-call practice built on top of it.

• 1 — Alerting & Alert Routing
• 2 — Symptoms vs Causes — (stub)
• 3 — Slo Based Alerts — (stub)
• 4 — Multi Window Burn Rate Alerts — (stub)
• 5 — Alert Deduplication — (stub)
• 6 — Routing — (stub)
• 7 — Alert Fatigue — (stub)
• 8 — On Call Engineering — (stub)

13 — Reliability & SRE Integration

SLIs, SLOs, and error budgets as the reliability contract observability serves, plus the incident detection, response, postmortem, and chaos-engineering practices built on that contract. See also sre/README.md for the broader SRE discipline this integrates with.

• 1 — SLIs — (stub)
• 2 — SLOs & Error Budgets
• 3 — Error Budgets — (stub)
• 4 — Incident Detection — (stub)
• 5 — Partial Results vs Fail-Fast
• 6 — Postmortems — (stub)
• 7 — Chaos Engineering — (stub)

14 — Cost Engineering

The cost drivers behind an observability platform’s bill, and the sampling, downsampling, retention, compression, and tiering levers used to control it.

• 1 — Cost Drivers — (stub)
• 2 — Telemetry Sampling — (stub)
• 3 — Downsampling — (stub)
• 4 — Retention Policies — (stub)
• 5 — Compression — (stub)
• 6 — Tiered Storage — (stub)
• 7 — FinOps for Observability — (stub)

15 — Security & Governance

Access control, tenancy isolation, and the privacy/compliance/secret-management obligations that apply specifically to telemetry data.

• 1 — RBAC — (stub)
• 2 — Multi Tenancy — (stub)
• 3 — Data Privacy — (stub)
• 4 — PII Redaction — (stub)
• 5 — Security & Compliance
• 6 — Audit Logging — (stub)
• 7 — Secret Management — (stub)

16 — Observability Platform Engineering

Building observability as a self-service, paved-road capability on an internal developer platform — pipelines, observability-as-code, GitOps, and platform APIs.

• 1 — Building a Platform Team
• 2 — Driving Adoption
• 3 — Telemetry Pipelines — (stub)
• 4 — Observability-Driven Development
• 5 — GitOps — (stub)
• 6 — Terraform — (stub)
• 7 — Platform APIs — (stub)
• 8 — Multi Region Design — (stub)

17 — AI & Intelligent Observability

AIOps, automated root-cause analysis, anomaly and event correlation, and the emerging LLM-assisted and autonomous-remediation layer — with an explicit read/write safety boundary.

• 1 — AIOps / Agentic RCA
• 2 — Root Cause Analysis — (stub)
• 3 — Anomaly Detection — (stub)
• 4 — Event Correlation — (stub)
• 5 — Predictive Alerting — (stub)
• 6 — LLM Assisted Troubleshooting — (stub)
• 7 — Autonomous Remediation — (stub)

18 — Interview Preparation

A structured question bank for MAANG-level observability and SRE interviews — system design, troubleshooting, and quantitative capacity-planning formats.

• 1 — Observability System Design Questions — (stub)
• 2 — Troubleshooting Case Studies — (stub)
• 3 — Telemetry Design Exercises — (stub)
• 4 — Incident Walkthroughs — (stub)
• 5 — Production Debugging — (stub)
• 6 — Capacity Planning — (stub)
• 7 — Scaling to Millions of Metrics — (stub)
• 8 — Whiteboard Architecture Problems — (stub)
• 9 — Maang Interview Questions — (stub)

19 — Real-World Case Studies

How major engineering organizations have approached observability at scale, drawn from their public engineering writing, and the transferable lessons across them.

• 1 — Uber — (stub)
• 2 — Google — (stub)
• 3 — Meta — (stub)
• 4 — Netflix — (stub)
• 5 — Amazon — (stub)
• 6 — Microsoft — (stub)
• 7 — Cloud Native CNCF Projects — (stub)
• 8 — Case Study: Reactive → Resilient → Autonomous

20 — Appendices

Quick-reference material — semantic conventions, query-language cheat sheets, and cost/readiness checklists — for use alongside the chapters above.

• 1 — OpenTelemetry Semantic Conventions — (stub)
• 2 — Promql Cheat Sheet — (stub)
• 3 — Logql Cheat Sheet — (stub)
• 4 — Traceql Cheat Sheet — (stub)
• 5 — OTLP Reference — (stub)
• 6 — Kubernetes Telemetry Reference — (stub)
• 7 — Observability Design Patterns — (stub)
• 8 — Common Anti Patterns — (stub)
• 9 — Telemetry Cost Estimation — (stub)
• 10 — Production Readiness Checklist — (stub)

Metadata

Author	Amit Singh
Scope	observability