Observability Engineering

A book-shaped table of contents for observability engineering: foundations through architecture, metrics, logging, tracing, profiling, OpenTelemetry, instrumentation, Kubernetes/cloud, data platforms, visualization, alerting, SRE integration, cost, security, platform engineering, AI-driven operations, and MAANG interview preparation — cross-linking existing prometheus/grafana-cloud/kubernetes/sre/platform-engineering notes instead of duplicating them.

2. High-Level Architecture

The producers → ingestion gateway → Kafka → processors → storage diagram for the telemetry ingestion pipeline, plus the push-over-pull key insight to state early in the interview.

3.1 Layer 1: Ingestion Frontier

Layer 1 of the telemetry ingestion pipeline: the ingestion frontier — responsibilities, fan-in at 100K+ agents, protocol negotiation, batching, backpressure, and rate limiting.

Protocol Termination at the Ingestion Frontier

What actually happens where the wire protocol ends — TCP/TLS handoff, HTTP/2 frame demux, gRPC message decode, protobuf deserialization — and why L4 vs L7 termination and connection-lifecycle tuning are the load-bearing decisions here, not the crypto itself.

Telemetry Gateways: Protocol-Specific Ingestion Points

The ingestion frontier is a fleet of protocol-specific gateways — OTLP, Prometheus remote-write, Syslog, Kafka, and legacy tracing/metrics protocols — each terminating a different producer's wire format before a shared auth/rate-limit/tenant-routing layer.

4 — Auto vs. Manual Instrumentation

Four ways a span gets created — hand-written, framework-level auto-instrumentation, eBPF, and service-mesh sidecar capture — and the trade-off between code changes and business context each one makes.

5 — Label & Attribute Schema Design

Cardinality budget, naming conventions, and the high-churn label traps that turn a cheap metric into a production incident — the design discipline for the labels semantic conventions don't already cover for you.

9 — OTel Collector Pipeline Design

Receivers, processors, and exporters chained into a pipeline; why a platform runs more than one; and the agent/gateway topology that tail sampling specifically forces on that design.

1 — OpenTelemetry SDKs & Semantic Conventions

OpenTelemetry is a specification and an API/SDK, not a backend — the pieces that make it up, and the semantic-convention vocabulary that lets two unrelated teams' telemetry be queried the same way.