1 — What Observability Actually Means
“We have Grafana, so we’re observable” is the single most common misuse of the word. Observability isn’t a tool, a dashboard, or a signal type — it’s a property of whether a system’s existing telemetry can answer a question nobody thought to ask in advance. Everything else in this chapter is a consequence of that one distinction.
The control-theory definition
Observability is a term borrowed from control theory: a system is observable if its internal state can be inferred entirely from its external outputs. Applied to software, that becomes: can you determine why a system is behaving a certain way, using only the telemetry it already produces — without shipping new code, adding a log line, and waiting for a redeploy to find out.
That last clause is the practical test. If answering a new question about production requires a code change first, the system wasn’t observable for that question — it was, at best, monitored for the questions someone already anticipated.
Monitoring answers known unknowns; observability covers unknown unknowns
Monitoring is watching for failure modes you already know about: a dashboard for CPU utilization, an alert on error rate, a check on disk space. It’s built by someone who sat down and enumerated, in advance, the ways the system might fail. This works well for exactly the failure modes on that list — a known unknown: you don’t know when disk will fill up, but you knew disk-filling-up was worth watching for.
Observability is what you need for an unknown unknown — the failure mode nobody wrote a check for, because nobody anticipated it, precisely because it’s novel. You can’t pre-build a dashboard for a question you don’t know you’ll need to ask. What you can do instead is capture rich enough, high-cardinality enough telemetry up front that any question — including ones invented during the incident itself — can be answered by slicing and correlating data you already have.
Monitoring: failure mode → predefined check → alert
Observability: (any behavior) → rich telemetry → arbitrary question, asked after the fact
This is why “we have dashboards for everything that’s ever gone wrong before” is a monitoring achievement, not an observability one — it says nothing about the failure mode that hasn’t happened yet.
The three-pillars critique
Metrics, logs, and traces are commonly sold as “the three pillars of observability” — implying that owning all three tools makes a system observable. This framing gets the causality backwards. Having three separate telemetry types, each queried in its own tool, describes what you collect, not whether you can actually answer something with it. Three pillars that don’t share a correlation key are three separate monitoring tools bolted together, not one observable system — see 3 — Cross-Signal Correlation for the mechanism (a shared trace ID and exemplars) that’s actually doing the work of turning three signal types into one investigable system.
The Signals covers what each of the five signal types (the “three pillars” plus profiles and events) is and isn’t good for in isolation. This chapter is the argument for why none of that matters on its own — the pillars are necessary, not sufficient. A team with beautiful per-pillar dashboards and no way to pivot from a metric spike to the one trace and log line that explains it has bought the tools without the property the tools were supposed to deliver.
Observability is a property of the instrumentation, not the tooling
Two teams can run the identical observability stack — same Grafana, same Prometheus, same Tempo — and land on opposite sides of “observable.” The difference is upstream of the tooling, in what gets captured at emission time:
- High-cardinality, high-dimensionality context. Answering an unanticipated question usually means slicing by a dimension nobody thought to pre-aggregate — which build version, which customer tier, which specific pod. If that context was discarded before storage (or never captured at all) because it looked like a cardinality risk, no query written afterward can recover it. See What is Cardinality (in observability) and 5 — Label & Attribute Schema Design for where that trade-off actually gets made, one label at a time, long before anyone asks the question that needed it.
- Correlation designed in, not queried in. 3 — Cross-Signal Correlation makes the same point from the wire-format side: a shared identifier has to be stamped at emission time across every signal, or there’s no way to reconstruct it retroactively once an incident is already underway.
- Wide, structured events over narrow pre-aggregated ones. A single wide event per request — one structured record carrying dozens of fields (route, tenant, cache outcome, retry count, feature flags in effect) — preserves the ability to group by any combination of them later. A metric pre-decides its grouping dimensions at instrumentation time; if the dimension the incident needs wasn’t one of them, the metric can’t retroactively grow it.
None of this is purchasable after the fact. A vendor migration doesn’t make a system observable if the underlying services still only emit four pre-aggregated counters and an unindexed text log.
The practical test
Before calling anything “observable,” ask: the last time production did something nobody predicted, could an engineer explain why using only telemetry that already existed — or did someone have to add a log line, ship it, and wait? The former is observability doing its job. The latter is monitoring with an observability-shaped marketing budget.
For a much longer, Socratic version of this same question — working through it across business context, multi-tenancy, SLOs, and platform validation — Observability Architecture: Questions to Ask is a 200-plus-question sequence built for exactly that kind of drilling.
Why this matters for an Observability Architect
Every instrumentation decision in this book is really a bet about which unknown unknowns are worth paying for in advance. 5 — Label & Attribute Schema Design is about not discarding a dimension you’ll need later; 4 — Auto vs. Manual Instrumentation is about not leaving whole services uninstrumented until someone happens to notice; 3 — Cross-Signal Correlation is about not losing the thread between signals once an investigation starts. None of those chapters matter if the underlying belief is “we bought the three pillars, we’re covered” — that belief is exactly the gap this chapter exists to close before it costs an incident to discover.
Metadata
| Dimension | Detail |
|---|---|
| Author | Amit Singh |
| Scope | observability |
Local graph
Linked from 6 notes
4 — Observability-Driven Development
The TDD analogy taken seriously: SLOs and instrumentation defined at design time as acceptance criteria, not retrofitted after an incident — and why this only sticks as a launch gate, not a guideline.
Chapter 1 — Observability Architecture
Metrics, logs, traces, and profiles as the four correlated signal types every observability platform is built around.
2 — The Signals
Metrics, logs, traces, profiles, and events — what each is built to capture, what it costs, and which question it actually answers vs. which one people mistakenly ask it.
3 — Cross-Signal Correlation
Metrics, logs, and traces are only more useful together than apart if something ties a specific instance of each of them back to the same event. The shared identifier that makes that jump possible — and what breaks when a hop in the call chain doesn't carry it.
Observability Engineering
A book-shaped table of contents for observability engineering: foundations through architecture, metrics, logging, tracing, profiling, OpenTelemetry, instrumentation, Kubernetes/cloud, data platforms, visualization, alerting, SRE integration, cost, security, platform engineering, AI-driven operations, and MAANG interview preparation — cross-linking existing prometheus/grafana-cloud/kubernetes/sre/platform-engineering notes instead of duplicating them.
Prometheus
A book-shaped table of contents for Prometheus: monitoring foundations through architecture, data model, instrumentation, service discovery, PromQL, alerting, production operation, PCA certification, and MAANG interview prep — cross-linking existing notes instead of duplicating them.
Related notes
2 — The Signals
Metrics, logs, traces, profiles, and events — what each is built to capture, what it costs, and which question it actually answers vs. which one people mistakenly ask it.
3 — Telemetry Lifecycle
Traces a signal path from generation through collection, transport, storage, query, visualization, alerting, and retention.
4 — Observability Maturity Model
Maps the crawl/walk/run/autonomous stages of observability maturity to concrete platform and process capabilities.
1 — Dashboard Design
The three-question test for a vanity panel, why the same underlying data needs a different dashboard for different audiences, and the top-down layout that mirrors how an investigation actually drills down.