# Agentic Ai Projects And Mastery
All Agentic Ai Projects And Mastery notes →Managing, Monitoring, and Securing Azure AI Services
Operational reference for running Azure AI Services in production — activity logs vs. diagnostic logs vs. metrics, cost model and budgets, key rotation and Key Vault, Entra ID auth, network isolation (service vs. private endpoints), Responsible AI principles, CI/CD and IaC for AI resources, and container deployment.
What is Azure SRE Agent (Microsoft)
Microsoft's AI agent embedded in Azure for autonomous incident response — acknowledges alerts from PagerDuty/ServiceNow/Azure Monitor, investigates via Azure Monitor/App Insights/Kusto, executes configurable Incident Response Plans with tunable autonomy, and learns across incidents via Session Insights.
What is HolmesGPT
Robusta.dev's open-source SRE agent (CNCF Sandbox) for investigating production incidents across Kubernetes, VMs, cloud services, and databases — an agentic tool-calling loop over 70+ toolsets, not a chatbot or RAG system, with a proactive 'operator mode' that monitors and opens fix PRs without a human trigger.
# Observability
All Observability notes →What is Cortex (cortexproject)
CNCF Incubating, horizontally-scalable multi-tenant long-term storage for Prometheus — the project Grafana Mimir forked from in 2022, still maintained as the vendor-neutral, community-governed alternative once Grafana Labs redirected engineering effort to Mimir.
1 — What Observability Actually Means
Observability vs. monitoring, the three-pillars critique, and why observability is a property of how a system was instrumented — not a tool you bought or a dashboard you built.
2 — The Signals
Metrics, logs, traces, profiles, and events — what each is built to capture, what it costs, and which question it actually answers vs. which one people mistakenly ask it.
7 — Multi-Tenancy
Two separate guarantees hiding under one name — data isolation and performance fairness — and the tenant identification, quota enforcement, and selective backpressure that make both hold under shared infrastructure.
8 — Self-Observability
The bootstrapping problem — a platform can't fully trust itself to tell you it's failing — and the two mechanisms that get around it: an independent out-of-band health path, and a synthetic canary that catches silent stalls no internal metric surfaces.
5 — Label & Attribute Schema Design
Cardinality budget, naming conventions, and the high-churn label traps that turn a cheap metric into a production incident — the design discipline for the labels semantic conventions don't already cover for you.
7 — Metrics Storage (TSDB)
Chunk/block encoding, the write-ahead log, compaction and the write amplification it trades for query speed, and why a cardinality spike is a storage-engine problem, not just a cost line item.
8 — Log Aggregation
Schema-on-write vs. schema-on-read as competing bets about when to pay indexing cost, and the two different deduplication problems a log pipeline actually has to solve.
7 — Distributed Tracing Backend
How spans that arrive out of order, from different services, get assembled into one trace — and the two competing storage models (indexed search vs. object storage plus a trace-ID lookup) that trade query flexibility for cost.
5 — Continuous Profiling
What makes always-on, sampling-based profiling cheap enough to run in production continuously, why it earns that cost mainly for hot or expensive services, and how a profile correlates back to the one trace that was running during the sample.
1 — OpenTelemetry SDKs & Semantic Conventions
OpenTelemetry is a specification and an API/SDK, not a backend — the pieces that make it up, and the semantic-convention vocabulary that lets two unrelated teams' telemetry be queried the same way.
4 — Auto vs. Manual Instrumentation
Four ways a span gets created — hand-written, framework-level auto-instrumentation, eBPF, and service-mesh sidecar capture — and the trade-off between code changes and business context each one makes.
9 — OTel Collector Pipeline Design
Receivers, processors, and exporters chained into a pipeline; why a platform runs more than one; and the agent/gateway topology that tail sampling specifically forces on that design.
1 — Dashboard Design
The three-question test for a vanity panel, why the same underlying data needs a different dashboard for different audiences, and the top-down layout that mirrors how an investigation actually drills down.
1 — Alerting & Alert Routing
Symptom-based vs. cause-based alerting, the noise-reduction problem (dedup, grouping, correlation), and routing/escalation — the design discipline standing between a real page and a bad one.
2 — SLOs & Error Budgets
SLI/SLO/SLA, the error budget as a spendable resource rather than a compliance number, burn rate as the mechanism connecting the two, and why multi-window multi-burn-rate alerting exists at all.
5 — Security & Compliance
Why PII ends up in telemetry by accident rather than by design, the pipeline-layer scrubbing that catches what application discipline misses, tenant-scoped access control, and why the query audit log is itself security-relevant telemetry.
1 — Building a Platform Team
A platform team's product is other teams' ability to self-serve reliable telemetry — team topology, the paved road that makes everything earlier in this book the default instead of a manual step, and the ticket-queue failure mode to watch for.
2 — Driving Adoption
A paved road nobody travels on didn't help anyone — onboarding time as the leading indicator, self-service as the mechanism that actually moves it, and why migrating an existing service is a harder adoption problem than a greenfield one.
4 — Observability-Driven Development
The TDD analogy taken seriously: SLOs and instrumentation defined at design time as acceptance criteria, not retrofitted after an incident — and why this only sticks as a launch gate, not a guideline.
1 — AIOps / Agentic RCA
What's actually new versus a static runbook — an investigation loop, not a fixed trigger-action mapping — why it depends on everything earlier in this book already being solid, and the read-vs-write safety line most real deployments draw.
8 — Case Study: Reactive → Resilient → Autonomous
An illustrative three-act arc — built on the ShipSolid platform maturity model, not any single real deployment — showing why the disciplined middle act is what actually earns the reliability, and why the autonomous act doesn't work without it.
3 — Aggregation Composability — Why You Can't Average Percentiles
Some statistics merge correctly across shards, replicas, and time windows — sum, count, max. Percentiles do not. The distinction that decides whether a fleet-wide dashboard is trustworthy or quietly wrong.
3 — Cross-Signal Correlation
Metrics, logs, and traces are only more useful together than apart if something ties a specific instance of each of them back to the same event. The shared identifier that makes that jump possible — and what breaks when a hop in the call chain doesn't carry it.
What is eBPF
Extended Berkeley Packet Filter — sandboxed, verified bytecode run inside the Linux kernel without a module or a restart. The foundation under Cilium, Grafana Beyla, and Pyroscope: zero-instrumentation traces, metrics, and continuous profiling.
What is Fluent Bit
CNCF-graduated, C-written log/metrics/trace forwarder — Fluentd's lightweight sibling, the de facto node-level log-collection DaemonSet in most Kubernetes clusters, and Grafana Alloy's main incumbent competitor for that slot.
What is Jaeger
CNCF-graduated distributed tracing system built at Uber in 2015, Dapper-lineage like Zipkin before it — and, since Jaeger v2, rebuilt on top of the OpenTelemetry Collector rather than bespoke ingestion code.
What is Loki
Grafana Labs' log aggregation system — 'like Prometheus, but for logs': index only labels, store compressed chunks in object storage, query with LogQL. Shares its distributor/ingester/compactor architecture with Mimir and Tempo.
What is Mimir
Grafana Labs' horizontally-scalable, multi-tenant long-term storage for Prometheus metrics — the 2022 successor to Cortex, and the actual system serving every PromQL query and remote-write in a Grafana Cloud metrics stack.
What is Prometheus
CNCF's second graduated project (2018) — the pull-based metrics monitoring system and query language (PromQL) that defined the exposition format nearly every metrics tool now speaks, and the API that Grafana Mimir scales out horizontally.
What is StatsD
Etsy's 2011 UDP-based metrics protocol and daemon — the simplest possible fire-and-forget instrumentation format, superseded as a client API by OTel/Prometheus but still alive everywhere as a compatibility ingestion shim.
What is Tempo
Grafana Labs' distributed tracing backend — the radical simplification vs. Jaeger's classic architecture: no dedicated index, just object storage and a trace-ID lookup, queried with TraceQL and linked from metrics via exemplars.
What is Cardinality (in observability)
The number of unique time series (or unique log/trace label combinations) a metric produces — the single biggest driver of ingest cost and query latency in Prometheus-family backends (Mimir, Cortex, Thanos), and the reason unbounded labels are a production incident waiting to happen.
What is Telegraf
InfluxData's plugin-driven metrics/events/logs collection agent — 300+ input/output plugins, written in Go, single static binary — the collector layer in the InfluxDB (TICK-stack-descendant) ecosystem, comparable in role to Grafana Alloy.
2 — Tail Latency
Why the p99/p999 request latency matters more than the average in distributed systems — a single slow dependency in a fan-out can dominate the response time even when most calls are fast.
8 — Query Sharding
Splitting a single logical query into N independently-executable sub-queries that run in parallel and merge into one result — how Grafana Mimir and Loki answer high-cardinality queries within tight SLOs without adding more data shards.
9 — Fan-Out Metrics and Trace Shape
The Prometheus metrics that instrument a fan-out pattern (width, shard latency, aggregation, partial results, cancelled workers, hedged requests), the OTel trace waterfall shape that reveals the tail shard, and when requests fan out to multiple shards vs route to a single one.
3 — Telemetry Lifecycle
Traces a signal path from generation through collection, transport, storage, query, visualization, alerting, and retention.
4 — Observability Maturity Model
Maps the crawl/walk/run/autonomous stages of observability maturity to concrete platform and process capabilities.
1 — Designing An Observability Platform
Frames a platform-level design exercise: ingestion scale, tenancy, storage tiering, and query latency as first-class requirements.
2 — Data Plane vs Control Plane
Separates the high-throughput telemetry data path from the low-throughput configuration/policy path, and why conflating them causes outages.
3 — Push-Based vs Pull-Based Ingestion
Core mental model for telemetry and data ingestion patterns — when to push, when to pull, and how to reason about the trade-offs at principal/staff interview bar.
4 — Agent Based vs Agentless Collection
Weighs sidecar/DaemonSet agents against agentless eBPF or vendor-API scraping for coverage, overhead, and maintenance cost.
5 — Edge Aggregation
Covers pre-aggregating and filtering telemetry at the collection edge to cut cardinality and egress cost before it reaches the backend.
6 — Centralized vs Federated Observability
Contrasts a single global observability backend against per-region or per-BU federated backends with cross-federation query.
1 — Time Series Fundamentals
Covers the time-series data model — series identity, sample resolution, and the write/query tradeoffs baked into that model.
2 — Metric Types
Distinguishes counter, gauge, histogram, and summary semantics and the aggregation rules each type permits or forbids.
4 — Cardinality Management
Covers estimating and bounding active series count before a label change ships, and the incident patterns an unbounded label causes.
6 — Recording Rules
Covers pre-computing expensive PromQL expressions into new series to keep dashboard and alert queries fast at scale.
1 — Structured Logging
Covers moving from freeform text logs to structured key-value records that a query engine can filter and aggregate on.
2 — Log Schemas
Covers designing a consistent field schema across services so logs from different teams remain queryable together.
4 — Log Pipelines
Covers the collection-to-storage pipeline for logs — parsing, enrichment, and routing before they land in a backend.
5 — Log Sampling
Covers reducing log volume by sampling non-error traffic while preserving full fidelity on errors and slow requests.
6 — Log Retention
Covers setting retention windows per log tier and the tradeoff between debugging lookback and storage cost.
7 — Cost Optimization
Covers the log-specific levers — sampling, field-level filtering, and tiered storage — for controlling ingest and storage spend.
1 — Why Tracing Exists
Covers the request-fan-out problem that metrics and logs can't solve alone, and why tracing became necessary in microservice architectures.
2 — Trace Context
Covers the trace ID / span ID / trace flags that identify a request and its position in a trace, and how they are carried across process boundaries.
3 — Span Modeling
Covers what a span should represent — operation boundaries, parent/child relationships, and span attributes vs. events.
4 — Context Propagation
Covers how trace context survives async boundaries, message queues, and batch jobs — and the common places it silently breaks.
5 — Trace Sampling
Covers head-based sampling decisions made at trace start, and their tradeoff against tail-based sampling on completeness vs. cost.
6 — Tail Sampling
Covers sampling decisions made after a trace completes, keeping error and slow traces at the cost of buffering full traces at the collector.
8 — Service Graphs
Covers deriving a live service dependency graph from trace data, and using it for blast-radius and dependency-health analysis.
1 — CPU Profiling
Covers sampling-based CPU profiling — what a flame graph represents and how to read one to find a hot function.
2 — Memory Profiling
Covers allocation profiling and how it differs from CPU profiling in what it samples and what questions it answers.
3 — Heap Analysis
Covers heap snapshot analysis for finding retained-object leaks that GC alone will not surface.
4 — Goroutines and Threads
Covers profiling concurrency primitives — goroutine/thread counts and blocking profiles — to find contention and leaks.
10 — Collector Pipelines
Covers composing multiple named pipelines in one Collector for signal-specific or team-specific routing.
11 — Processors
Covers batching, filtering, attribute-mutation, and tail-sampling processors and the order sensitivity of a processor chain.
12 — Exporters
Covers configuring multiple concurrent exporters and the retry/queueing behavior that protects against backend outages.
13 — Connectors
Covers connectors that derive one signal type from another inside the Collector, e.g. generating span metrics from trace data.
14 — Scaling Collectors
Covers horizontally scaling Collector fleets — load balancing, trace-ID-hash routing for tail sampling, and per-tier resource sizing.
2 — OTLP Protocol
Covers the OTLP wire protocol — its protobuf schema and gRPC/HTTP transport — as the common export format across signals.
3 — SDK Internals
Covers how an OTel SDK turns instrumentation calls into batched, exported telemetry — processors, exporters, and the pipeline between them.
5 — Manual Instrumentation
Covers hand-written spans, metrics, and log correlation for business-specific telemetry auto-instrumentation cannot infer.
6 — Semantic Conventions
Covers OTel's shared attribute vocabulary and why consistent naming is what makes two teams' telemetry queryable together.
7 — Resources
Covers Resource attributes — the identity of the process/host/service emitting telemetry — as distinct from per-signal attributes.
1 — Instrumenting Web APIs
Covers span and metric conventions for HTTP/gRPC API instrumentation — route templating, status code buckets, and latency histograms.
2 — Microservices
Covers instrumenting service-to-service calls consistently enough that a fleet-wide service graph and RED dashboard fall out for free.
3 — Messaging Systems
Covers instrumenting producer/consumer boundaries in queues and streams, where trace context propagation is easiest to get wrong.
4 — Databases
Covers instrumenting query spans and connection-pool metrics without leaking query parameter values as high-cardinality attributes.
5 — Caches
Covers hit/miss/eviction metrics and cache-specific span attributes that distinguish a cache problem from a backing-store problem.
6 — Kubernetes Workloads
Covers instrumenting pods and controllers so workload telemetry correlates cleanly with cluster-level Kubernetes signals.
7 — Serverless
Covers instrumenting cold-start latency and short-lived execution contexts where traditional agent-based collection does not fit.
8 — Batch Jobs
Covers instrumenting long-running, non-request-driven jobs where RED-method dashboards do not directly apply.
9 — Background Workers
Covers instrumenting queue-consumer worker pools — backlog depth, processing latency, and retry/dead-letter visibility.
1 — Kubernetes Metrics
Covers the cAdvisor/kubelet/kube-state-metrics metric surfaces and which one answers which question about a cluster.
2 — Control Plane Monitoring
Covers monitoring the API server, etcd, scheduler, and controller-manager — the control plane's own health as a distinct concern from workload health.
3 — Node Monitoring
Covers node-level resource pressure signals and how they surface as pod evictions and scheduling failures.
4 — Pod Monitoring
Covers pod lifecycle, restart, and readiness/liveness signal correlation with application-level telemetry.
5 — Cluster Events
Covers the Kubernetes Events API as a signal type distinct from metrics and logs, and its short default retention.
6 — Container Runtime
Covers container-runtime-level signals (CRI metrics, OOM kills) that sit below the kubelet's own reporting.
7 — Service Mesh Observability
Covers the telemetry a sidecar mesh generates for free — mTLS, retries, and per-hop latency — versus what still needs app-level instrumentation.
8 — eBPF Based Observability
Covers kernel-level eBPF telemetry collection as a zero-instrumentation alternative for network and syscall-level visibility.
1 — AWS
Covers CloudWatch's metric/log/trace surfaces and where AWS-native telemetry needs augmenting with OTel for cross-account visibility.
2 — Azure
Covers Azure Monitor and Application Insights as the native telemetry surface, and their integration points with an OTel-based pipeline.
3 — Google Cloud
Covers Google Cloud's operations suite (Cloud Monitoring/Logging/Trace) and its native OTLP ingestion path.
4 — Hybrid Cloud
Covers unifying telemetry across on-prem and cloud environments where network topology and identity differ per environment.
5 — Multi Cloud
Covers the added complexity of a telemetry pipeline that must normalize signals from more than one cloud provider's native tooling.
1 — Prometheus
Covers Prometheus as the reference pull-based metrics engine — see the dedicated Prometheus book for full depth; this chapter covers only its role in the broader platform.
3 — Loki
Covers Loki's index-light, label-indexed log storage model and how it differs from full-text log indexing.
4 — Tempo
Covers Tempo’s object-storage-backed, trace-ID-lookup model for cost-efficient distributed trace storage.
5 — Pyroscope
Covers Pyroscope as a continuous-profiling backend and its data model for flame-graph-over-time queries.
6 — Elasticsearch
Covers Elasticsearch as a full-text-indexed log and event store, and its cost/flexibility tradeoff against label-indexed alternatives.
7 — Clickhouse
Covers ClickHouse as a columnar OLAP engine increasingly used as a unified backend for logs, traces, and wide events.
8 — Opensearch
Covers OpenSearch as the open-source Elasticsearch fork and its divergence points relevant to an observability backend choice.
3 — RED Method
Covers Rate/Errors/Duration as the request-driven-service adaptation of the golden signals.
4 — USE Method
Covers Utilization/Saturation/Errors as the resource-driven adaptation of the golden signals, for infrastructure rather than services.
5 — Executive Dashboards
Covers designing business-outcome dashboards for an audience that does not want a raw p99 latency panel.
6 — Engineering Dashboards
Covers designing debugging-oriented dashboards for the on-call engineer, optimized for time-to-first-signal during an incident.
7 — Business Observability
Covers connecting telemetry to business KPIs — conversion, revenue, order completion — so reliability work has a business narrative.
2 — Symptoms vs Causes
Covers distinguishing 'users are affected' alerts from 'a specific subsystem misbehaved' alerts, and why only the former should page.
3 — Slo Based Alerts
Covers deriving alert thresholds from an SLO's error budget rather than from arbitrary static thresholds.
4 — Multi Window Burn Rate Alerts
Covers the multi-window, multi-burn-rate alerting technique that balances fast detection against alert noise.
5 — Alert Deduplication
Covers grouping and suppressing duplicate alerts from the same root cause so on-call sees one page, not fifty.
6 — Routing
Covers alert routing rules — team ownership, severity, and escalation paths — as configuration distinct from the alert condition itself.
7 — Alert Fatigue
Covers diagnosing and reversing an alert-fatigue trend before it causes a real page to get ignored.
8 — On Call Engineering
Covers structuring on-call rotations, handoffs, and runbook discipline as an engineering practice, not just a schedule.
1 — SLIs
Covers choosing a Service Level Indicator that actually reflects user-perceived reliability, not just what's easiest to measure.
3 — Error Budgets
Covers treating the error budget as a spendable risk resource that governs release velocity, not a compliance scorecard.
4 — Incident Detection
Covers the telemetry-to-detection path — how observability signals trigger the moment an incident is declared.
6 — Postmortems
Covers writing a blameless postmortem that traces the incident timeline back to instrumentation and observability gaps, not just the code fix.
7 — Chaos Engineering
Covers using deliberate fault injection to validate that observability signals actually fire the way an incident response plan assumes.
1 — Cost Drivers
Covers the ingest-volume, cardinality, and retention-window levers that actually drive observability platform cost.
2 — Telemetry Sampling
Covers sampling as a cost lever across all three signal types, and the fidelity it trades away.
3 — Downsampling
Covers reducing metric resolution over time as data ages, and the query-accuracy tradeoff that comes with it.
4 — Retention Policies
Covers setting differentiated retention per signal type and per tier, driven by actual debugging-lookback needs rather than defaults.
5 — Compression
Covers the compression techniques (chunk encoding, columnar compression) that let TSDBs and log stores shrink storage cost per sample.
6 — Tiered Storage
Covers hot/warm/cold storage tiering — recent data on fast disks, older data in object storage — and its query-latency tradeoff.
7 — FinOps for Observability
Covers attributing observability spend back to the teams generating the telemetry, and using that attribution to drive down cost at the source.
1 — RBAC
Covers role-based access control for telemetry — who can query which tenant or team’s data, and at what granularity.
2 — Multi Tenancy
Covers the isolation guarantees a shared observability platform must enforce so one tenant can never read another’s telemetry.
3 — Data Privacy
Covers the privacy obligations that apply to telemetry data specifically, distinct from the privacy obligations on the underlying application data.
4 — PII Redaction
Covers how PII ends up in telemetry by accident (log lines, span attributes, user IDs) and where in the pipeline to catch it.
6 — Audit Logging
Covers the query audit log — who ran what query against what data — as security telemetry about the platform itself.
7 — Secret Management
Covers keeping API keys, tokens, and credentials out of telemetry payloads and out of collector/exporter configuration in plaintext.
3 — Telemetry Pipelines
Covers building the reusable pipeline infrastructure (Collector fleets, routing config) that self-service onboarding depends on.
5 — GitOps
Covers deploying observability-as-code configuration through the same GitOps reconciliation loop as application deployments.
6 — Terraform
Covers managing observability backend resources (data sources, alert rules, access policies) as Terraform-managed infrastructure.
7 — Platform APIs
Covers designing the API surface a platform team exposes so other teams can provision telemetry resources programmatically.
8 — Multi Region Design
Covers designing an observability platform's own multi-region topology so it does not share a single point of failure with the workloads it observes.
2 — Root Cause Analysis
Covers automated root-cause analysis as an investigation loop over existing telemetry, not a fixed trigger-action mapping.
3 — Anomaly Detection
Covers statistical and ML-based anomaly detection on time series, and its false-positive tradeoff against static thresholds.
4 — Event Correlation
Covers correlating alerts, deploys, and changes across systems to collapse a flood of related signals into one incident.
5 — Predictive Alerting
Covers forecasting-based alerting that pages before a threshold breach, and the calibration risk that comes with prediction.
6 — LLM Assisted Troubleshooting
Covers using an LLM over existing telemetry for incident triage, and the hard boundary between read-only investigation and write-capable remediation.
7 — Autonomous Remediation
Covers safely scoping autonomous remediation actions, and why the read/write safety line matters more here than anywhere else in the stack.
1 — Observability System Design Questions
Covers the recurring system-design prompt shape — 'design a metrics/logging/tracing platform at scale' — and the tradeoffs interviewers probe for.
2 — Troubleshooting Case Studies
Covers worked troubleshooting scenarios (e.g. a collector agent pinned at 100% CPU) as a rehearsal for live debugging interview questions.
3 — Telemetry Design Exercises
Covers exercises in designing the telemetry (metrics/logs/traces/labels) for a given service from scratch, a common interview format.
4 — Incident Walkthroughs
Covers narrating a real incident timeline and RCA in interview-answer form, structured for a behavioral or systems-thinking question.
5 — Production Debugging
Covers the live-debugging interview format — given a symptom, which signal do you check first and why.
6 — Capacity Planning
Covers estimating ingest rate, series count, and storage growth for a hypothetical platform, a common quantitative interview question.
7 — Scaling to Millions of Metrics
Covers the specific architectural changes (sharding, downsampling, federation) required as series count crosses common scale thresholds.
8 — Whiteboard Architecture Problems
Covers open-ended whiteboard prompts on observability platform architecture and the tradeoff-driven answer structure interviewers expect.
9 — Maang Interview Questions
Covers a curated question bank spanning system design, troubleshooting, and behavioral formats specific to MAANG-level observability/SRE interviews.
1 — Uber
Covers Uber's published observability architecture and scaling decisions as a case study, cited from public engineering sources.
2 — Google
Covers Google's observability practices (Monarch, Dapper) and their influence on the broader industry's approach, cited from public sources.
3 — Meta
Covers Meta's internal observability and tracing infrastructure as described in public engineering writing.
4 — Netflix
Covers Netflix's observability and chaos engineering practices as described in public engineering writing.
5 — Amazon
Covers Amazon's operational excellence and observability practices as described in public engineering writing.
6 — Microsoft
Covers Microsoft's observability practices across Azure and first-party services as described in public engineering writing.
7 — Cloud Native CNCF Projects
Covers the CNCF observability landscape (OTel, Prometheus, and related projects) as a case study in open-source-driven standardization.
1 — OpenTelemetry Semantic Conventions
A quick-reference index of OTel semantic convention attribute names by signal and domain.
10 — Production Readiness Checklist
A reference checklist for verifying a service has adequate observability coverage before a production launch.
2 — Promql Cheat Sheet
A quick-reference index of common PromQL functions and query patterns.
3 — Logql Cheat Sheet
A quick-reference index of common LogQL query patterns for Loki.
4 — Traceql Cheat Sheet
A quick-reference index of common TraceQL query patterns for Tempo.
5 — OTLP Reference
A quick-reference index of the OTLP protocol's message types and transport options.
6 — Kubernetes Telemetry Reference
A quick-reference index of Kubernetes-native telemetry sources and what each one exposes.
7 — Observability Design Patterns
A quick-reference index of recurring observability design patterns introduced throughout this book.
8 — Common Anti Patterns
A quick-reference index of common observability anti-patterns and the failure mode each one causes.
9 — Telemetry Cost Estimation
A worked reference for estimating telemetry ingest volume and cost from service count, request rate, and label cardinality.
Observability Engineering
A book-shaped table of contents for observability engineering: foundations through architecture, metrics, logging, tracing, profiling, OpenTelemetry, instrumentation, Kubernetes/cloud, data platforms, visualization, alerting, SRE integration, cost, security, platform engineering, AI-driven operations, and MAANG interview preparation — cross-linking existing prometheus/grafana-cloud/kubernetes/sre/platform-engineering notes instead of duplicating them.
# Projects
All Projects notes →Replication Guides: Instrumenting Your Own Project
Step-by-step, copy-paste guides for replicating SignalForge's OpenTelemetry instrumentation pattern in a new .NET/Python/Angular/RabbitMQ/K8s project.
Guide: Collector & Pipeline Setup
Step-by-step: stand up a Grafana Alloy + grafana/k8s-monitoring Helm chart pipeline that receives OTLP traces/metrics/logs from your services and exports to Grafana Cloud or a self-hosted backend.
Guide: .NET Instrumentation
Step-by-step: instrument an ASP.NET Core / gRPC .NET 8 service with OpenTelemetry — SDK wiring, custom spans and metrics, and RabbitMQ producer-side async trace propagation via the outbox pattern.
Guide: Frontend RUM Instrumentation
Step-by-step: instrument an Angular frontend with Grafana Faro for browser RUM — SDK setup, runtime config injection, source-map upload, and browser-to-backend trace linkage.
Guide: Python Instrumentation
Step-by-step: instrument a Python FastAPI service with OpenTelemetry — SDK wiring, custom metrics, and RabbitMQ consumer-side async trace propagation via manual context extraction and SpanLink.
Log-to-Trace Correlation
How SignalForge correlates logs to traces via node-level tailing and Loki structured metadata, across both local and cloud monitoring modes.
Exemplars
How exemplars link histogram metric observations to sampled traces end-to-end, from SDK emission through Prometheus/Mimir to Grafana.
OTel Signal Contracts
The OpenTelemetry signal contracts—spans, metrics, and log fields—for every SignalForge service and the frontend RUM app.
Observability Pipeline
How the Grafana Alloy collector pipeline differs between SignalForge's local (hand-authored River) and cloud (Helm chart) monitoring modes.
Tail-Based Sampling
How SignalForge's tail-based sampling policies decide which traces to keep, and why span metrics are generated before sampling runs.
SLOs & burn-rate alerts
SignalForge's published SLOs, how their SLIs are computed from span metrics, and how multi-window burn-rate alerts are structured.
SignalForge Instrumentation Reference
Reference explaining every OpenTelemetry instrumentation decision in the signal-forge lab — what's configured, why, and what correct behavior looks like.
Observability Architecture: Questions to Ask
A chronological sequence of 216 questions an architect asks when designing a production-grade observability platform — from business context through multi-tenancy, SLOs, onboarding, and validation.
# System Design
All System Design notes →Chapter 1 — Observability Architecture
Metrics, logs, traces, and profiles as the four correlated signal types every observability platform is built around.
Chapter 2 — Telemetry Pipelines
OpenTelemetry, OTLP, collectors, sampling, and aggregation as the pipeline that gets a signal from emission to storage without becoming the outage itself.
Chapter 3 — Monitoring at Scale
Prometheus, Mimir, Cortex, and Thanos as the horizontally-scaled answer to a single Prometheus instance running out of room.
Chapter 4 — Alerting Systems
Multi-window burn-rate alerts, recording rules, routing, and deduplication as the difference between an actionable page and noise.
Chapter 1 — Telemetry Ingestion Pipeline
Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.
1. Clarify Requirements First
The first-5-minutes clarifying questions for the telemetry ingestion pipeline design — signal types, scale envelope, consistency/durability, multi-tenancy, and protocol — whose answers change the entire architecture.
2. High-Level Architecture
The producers → ingestion gateway → Kafka → processors → storage diagram for the telemetry ingestion pipeline, plus the push-over-pull key insight to state early in the interview.
3.1 Layer 1: Ingestion Frontier
Layer 1 of the telemetry ingestion pipeline: the ingestion frontier — responsibilities, fan-in at 100K+ agents, protocol negotiation, batching, backpressure, and rate limiting.
3.2 Layer 2: Durable Buffer (Kafka)
Layer 2 of the telemetry ingestion pipeline: the Kafka durable buffer — topic design, partitioning strategy, hot-spots, retention, retry/delivery semantics, producer config, consumer lag, and schema evolution.
3.3 Layer 3: Processing / Enrichment
Layer 3 of the telemetry ingestion pipeline: processing and enrichment — the metric processor, cardinality enforcement, tail-based sampling, the log processor, metric temporality, and Kubernetes metadata enrichment.
3.4 Scaling Each Layer
Scaling unit and trigger for every layer of the telemetry ingestion pipeline, from the ingestion gateway through to storage.
3.5 Failure Modes and Mitigations
Failure modes and mitigations across the telemetry ingestion pipeline — gateway crashes, broker failure, processor crashes, span explosion, cardinality rejection, storage saturation, and clock skew.
3.6 Multi-Tenancy
Multi-tenancy isolation layers and quota enforcement points across the telemetry ingestion pipeline, from network inbound to the storage write path.
3.7 Data Tiering and Compaction (Mimir/Thanos)
Data tiering and compaction in Mimir/Thanos — the ingester-to-object-store journey, compaction levels, vertical compaction/dedup, compaction storms, and the config knobs that control them.
3.8 Global Deployment Topology
Global deployment topology for the telemetry ingestion pipeline — regional writes vs. a global cluster, async replication to a global query tier, and agent failover.
4. Observability of the Pipeline Itself
What to instrument at every layer of the telemetry ingestion pipeline, the pipeline's own SLOs, distributed tracing of the pipeline itself, and the synthetic canary that catches stalls no component metric surfaces.
5. Trade-offs at 10x Scale
The 'what would you do differently at 10x' trade-off questions for the telemetry ingestion pipeline: Kafka vs. direct write, trace-assembly sharding, schema-on-read vs. write, sampling strategy, protocol choice, and push vs. pull.
6. Interview Anchor Points (What to Say Out Loud)
The sentences that signal principal-level thinking for the telemetry ingestion pipeline design — ready to say unprompted in an interview.
7. Component Map (What Exists in the Wild)
OSS and managed-SaaS options for every layer of the telemetry ingestion pipeline, mapped against ShipSolid's own production experience.
8. Quick-Reference Cheat Sheet
One-line answers for every load-bearing design decision in the telemetry ingestion pipeline — the last thing to review before an interview.
9. Practice Interview Questions
Twelve full-length practice prompts for the telemetry ingestion pipeline design, each linked to its own worked, principal-level answer.
Authentication at the Ingestion Frontier: mTLS, Bearer Tokens, API Keys
How the gateway proves an agent's credential is valid — mTLS handshake validation, JWT bearer token signature checks, and API key lookups — with the revocation-speed vs operational-complexity trade-off between them, and credential rotation at 10M-agent scale.
Head vs. Tail Sampling for Distributed Traces
The sampling decision point determines whether a trace pipeline needs to buffer spans in memory — head-based decides at trace start, tail-based decides after the trace completes.
Protocol Termination at the Ingestion Frontier
What actually happens where the wire protocol ends — TCP/TLS handoff, HTTP/2 frame demux, gRPC message decode, protobuf deserialization — and why L4 vs L7 termination and connection-lifecycle tuning are the load-bearing decisions here, not the crypto itself.
Q1: 500M Samples/Sec, Zero Drop on Rolling Deploy
Full principal-level solution: design a telemetry ingestion pipeline for 500M metric samples/sec from 100K services globally with a zero-drop guarantee during rolling deployment of the ingestion tier.
Q10: Self-Service Tenant Onboarding With Zero Platform-Team Involvement
Full principal-level solution: design a self-service tenant onboarding API for a telemetry pipeline that protects shared infrastructure from a misbehaving new tenant on day one.
Q11: Redesigning the Ingestion Frontier for a Compromised-Agent Threat Model
Full principal-level solution: redesign the telemetry ingestion frontier assuming a compromised agent sends malformed and adversarial payloads — oversized batches, spoofed tenant IDs, garbage label values.
Q12: Exactly-Once for One Billing Tenant While Others Stay At-Least-Once
Full principal-level solution: support exactly-once ingestion for a single billing-critical tenant in a shared pipeline that is at-least-once everywhere else, and account for what it costs.
Q2: Cardinality Storm — Detect and Mitigate Without Affecting Other Tenants
Full principal-level solution: a tenant sends 50M unique label combinations/minute causing TSDB compaction storms — design detection and mitigation that isolates the blast radius to that tenant.
Q3: Trace Sampling Loses Spans During Incident Peaks — Redesign
Full principal-level solution: the trace pipeline drops spans exactly when incidents spike trace volume — diagnose the failure mode and redesign the tail-sampling pipeline to survive it.
Q4: A Metric's Journey From Pod to Dashboard — Every Failure Point
Full principal-level solution: trace a single metric data point from a Kubernetes pod to a queryable dashboard, identifying every failure point along the way and how each is detected.
Q5: Adding Continuous Profiling to an Existing MELT Pipeline
Full principal-level solution: extend an existing metrics + logs + traces pipeline with continuous profiling as a fourth signal, without a full redesign.
Q6: Compactor Queue Backing Up During a Multi-Tenant Flush
Full principal-level solution: diagnose a compactor backlog causing query latency spikes during a large multi-tenant flush, and mitigate it without pausing ingestion.
Q7: A Region's Gateway Goes Dark — Blast Radius Walkthrough and Redesign
Full principal-level solution: walk through the consequences of a 10-minute regional ingestion gateway outage, then redesign the topology to shrink the blast radius.
Q8: Counters Resetting to Zero After an OTel SDK Upgrade
Full principal-level solution: diagnose and fix a tenant's dashboards showing counters reset to zero every few minutes after an OTel SDK upgrade, without requiring instrumentation changes.
Q9: Cut Ingestion Infrastructure Cost 40% Without Violating SLOs
Full principal-level solution: a FinOps-driven cost-reduction pass on a telemetry ingestion pipeline — where to look first, what levers exist at each layer, and what you trade away.
Rate Limiting Architecture: Token Bucket, Gossip, and Envoy Global Limits
Three ways to enforce rate limits across a replicated gateway fleet — centralized Redis token bucket, decentralized gossip-based estimation, and Envoy's sidecar-plus-global-service pattern — with the precision/SPOF/latency trade-offs between them.
Retry Policies and the Delivery Semantics They Produce
Every retry decision is made along three axes — trigger, backoff, budget — before delivery semantics even enter the picture. At-least-once, at-most-once, and exactly-once are the accumulated side effect of those decisions at every hop, not a separate design choice.
Schema Validation and Rejection at the Ingestion Frontier
What the gateway actually checks before accepting a payload — structural validation vs semantic cardinality checks, why rejection has to happen before the buffer, OTLP PartialSuccess as an alternative to whole-batch rejection, and the forward-compatibility trap of validating too strictly.
Telemetry Gateways: Protocol-Specific Ingestion Points
The ingestion frontier is a fleet of protocol-specific gateways — OTLP, Prometheus remote-write, Syslog, Kafka, and legacy tracing/metrics protocols — each terminating a different producer's wire format before a shared auth/rate-limit/tenant-routing layer.
Tenant Identification and Routing at the Ingestion Frontier
How the gateway decides whose data this is — cert/API-key-derived tenant identity, propagation as a Kafka header and X-Scope-OrgID, and the shared-topic-with-filter vs per-tenant-topic vs shuffle-sharded-pool routing trade-off.
Chapter 2 — Metrics Storage (TSDB)
Write amplification, chunk encoding, compaction, cardinality explosion.
Chapter 3 — Log Aggregation System
Structured vs. unstructured, schema-on-read vs. schema-on-write, deduplication.
Chapter 4 — Distributed Tracing Backend
Trace assembly from spans, tail-based vs. head-based sampling.
Chapter 5 — OpenTelemetry Collector Pipeline
Multi-pipeline routing, processor chaining, exporter fan-out.
Chapter 6 — Multi-tenant Observability Platform
Tenant isolation, quota enforcement, cost attribution.
Chapter 7 — SLO / Error Budget Tracking System
Burn rate calculation, multi-window alerting, budget ledger.
System Design
Principal/Staff-level system design reference collection for MAANG interview preparation — observability pipelines, distributed systems, reliability engineering, and beyond.
# Grafana Cloud
All Grafana Cloud notes →What is gcx
Grafana's official CLI for managing Grafana Cloud/Enterprise/OSS resources, optimized for agentic usage — dashboards, alerts, SLOs, metrics/logs/traces/profiles queries, and every major Cloud product, plus a bundled Agent Skills set for Claude Code and other .agents-compatible harnesses.
What is Grafana MCP
Grafana's official open-source MCP server (mcp-grafana) — gives AI agents tool access to query metrics/logs/traces, manage dashboards and alert rules, and work with Incident and Sift; also available as a hosted OAuth 2.1 remote server with 50+ tools.
What is Grafana Skills
Grafana's take on reusable agent skills — captures a team's troubleshooting expertise as standardized, shareable procedures that can trigger MCP actions, plus gcx: a Grafana Cloud CLI shipping 22 bundled skills for Claude Code and other .agents-compatible harnesses.
# Networks
All Networks notes →What is Envoy
CNCF-graduated L7 proxy built at Lyft — the de facto data plane for service mesh (Istio, Linkerd's predecessor lineage) — now extending into AI traffic via Envoy AI Gateway, which reached v1.0 with a native MCP Gateway in 2026.
Protocol Inventory
Every protocol referenced across the telemetry ingestion pipeline design, plus a general L7-termination reference table for the broader 'design an API gateway / load balancer' interview question.
HTTP/2 vs HTTP/1.1
Why the ingestion gateway prefers HTTP/2 (multiplexed gRPC) over HTTP/1.1 — connection reuse, binary framing, and header compression at 100K+ agent fan-in.
gRPC
What gRPC actually is underneath the shorthand this design uses it for — call shapes, status-code backpressure, deadline propagation, and the connection-level load-balancing gotcha at 100K+ agent fan-in.
TLS Offload
Terminating TLS at the ingestion frontier instead of in every backend pod — why it's a Layer 1 responsibility, what it costs in defense-in-depth, and how mTLS re-encryption closes the gap.
# Patterns
All Patterns notes →01 — Sidecar
Co-locate a helper container with the application container to handle cross-cutting concerns — TLS, observability, auth, retries — without modifying application code.
01 — Monitoring Patterns
RED, USE, the Four Golden Signals, and Saturation — the metric frameworks that decide what to measure on a service before an incident forces the question.
02 — Logging Patterns
Structured Logging, Correlation IDs, Log Sampling, and Log Aggregation — how logs stay searchable and affordable at scale instead of becoming a second, worse metrics system.
03 — Tracing Patterns
Distributed Tracing, Context Propagation, and Tail vs. Head Sampling — how a single request's path across services becomes reconstructable instead of a pile of disconnected spans.
04 — Alerting Patterns
Multi-window Burn Rate alerts, SLO Alerts, Composite Alerts, and Noise Reduction — the alerting design that pages on user-facing pain instead of every internal wobble.
# Data Engineering
All Data Engineering notes →1 — Monitoring Pipelines
Monitoring data pipelines with metrics, logs, and traces, and defining pipeline health through SLIs and SLOs.
2 — Alerting
Alerting on the failure modes specific to data pipelines — freshness, completeness, volume anomalies, latency, and outright failures.
3 — Data Reliability
Data reliability engineering — data contracts, lineage as a debugging tool, incident management, and root cause analysis for pipeline failures.
# Infrastructure Platform Engineering
All Infrastructure Platform Engineering notes →1 — Infrastructure Monitoring
Covers monitoring the infrastructure platform itself — compute, networking, storage, and managed cloud services.
2 — Logging Infrastructure
Covers logging for infrastructure platform components and provisioning operations.
3 — Infrastructure Tracing
Covers tracing infrastructure provisioning and orchestration workflows to diagnose latency and failure points.
4 — Capacity Planning
Covers capacity planning for the infrastructure platform — forecasting demand and provisioning headroom.
5 — Infrastructure SLOs
Covers defining SLOs for the infrastructure platform itself — provisioning latency, availability, and success rate targets.
# Kubernetes Platform Engineering
All Kubernetes Platform Engineering notes →1 — Observability Architecture
Covers Metrics, Logs, Traces, and Profiles.
2 — Platform Monitoring
Covers Cluster Monitoring, Node Monitoring, Control Plane Monitoring, and Workload Monitoring.
3 — Logging Platforms
Covers Centralized Logging, Log Pipelines, Multi-Tenant Logging, and Retention.
4 — Platform Alerting
Covers SLO-Based Alerting, Alert Routing, Runbooks, and Incident Response.
5 — Platform Dashboards
Covers Platform KPIs, Capacity, Reliability, and Developer Metrics.
# Kubernetes
All Kubernetes notes →1 — Logging
Why Kubernetes has no built-in log aggregation by design — stdout/stderr capture by the kubelet is node-local and ephemeral, so durability is a platform-team responsibility, not a cluster feature.
2 — Metrics
Why metrics-server only ever powers kubectl top and the HPA — it holds no history by design, which is exactly the gap Prometheus was built to fill in every real cluster.
3 — Tracing
Why distributed tracing across a cluster is a service-mesh and instrumentation problem, not a kubelet one — Kubernetes has no native concept of a request, so context has to survive every sidecar hop on its own.
4 — Events
Why Kubernetes Events default to a 1-hour TTL in etcd — they're built as a live debugging signal for right-now, not an audit trail, and vanish before most incident retros even start.
5 — kubectl Debug
Why kubectl debug's ephemeral containers can attach to a running pod's process namespace without restarting it — the only clean way to get a shell into a distroless container that ships none of its own.
6 — Troubleshooting Production Clusters
Why most production cluster incidents trace back to control-plane pressure or misconfigured resource requests rather than application bugs — the debugging path starts at the scheduler and kubelet, not the pod logs.