3 — Prometheus in the Observability Ecosystem
This book spends most of its pages inside Prometheus itself — scrape configs, PromQL, alerting rules. Before going there, it’s worth placing Prometheus on the map relative to everything around it, so later chapters don’t read as if Prometheus operates in isolation.
Cloud-Native Origins
Prometheus was originally built at SoundCloud and joined the Cloud Native Computing Foundation in 2016 (the second project to do so, after Kubernetes). It’s written in Go, and its design choices — a pull-based scrape model, a purpose-built local TSDB, service discovery as a first-class concept — reflect the operational reality of container-orchestrated infrastructure: instances come and go, and something needs to keep asking “what’s currently running, and is it healthy” rather than waiting to be told.
The architecture, at its simplest, is a straight line:
Targets → Service Discovery (DNS, Kubernetes, AWS, Consul, custom…) → Prometheus (TSDB) → Grafana Web UI
Targets are anything exposing metrics on an HTTP endpoint — natively instrumented applications, or services fronted by an adapter (an “exporter”) that translates existing metrics into Prometheus’s exposition format. Service discovery is the source of truth for which targets currently exist; Prometheus polls that list, scrapes it, stores the result, and answers PromQL queries against it. Full history, the CNCF governance angle, and how this connects to Grafana Cloud as a managed destination are covered in tech/prometheus.md — this chapter isn’t re-deriving that.
Companion Projects in the Same Stack
Prometheus solves metrics. It deliberately does not solve logs, traces, or dashboards — those are separate CNCF projects designed to interoperate with it, not built into it:
- Grafana — the visualization and dashboarding layer most commonly paired with Prometheus as a data source; it’s also where PromQL results actually get looked at day to day.
- Loki — the logs counterpart, using a similar label-based indexing philosophy so log streams and metric series can be correlated by the same labels.
- Tempo — the traces counterpart, for the request-level detail that metrics and logs can’t provide on their own.
- Alertmanager — not a separate observability signal but Prometheus’s own alert-routing companion: Prometheus evaluates alerting rules and pushes firing alerts to Alertmanager, which deduplicates, groups, and routes them to email, Slack, PagerDuty, and similar destinations.
None of these get a deep dive in this book — they’re named here because a Prometheus deployment in practice is rarely deployed alone, and this chapter would be dishonest if it implied otherwise.
Scaling Beyond a Single Prometheus Server
A single Prometheus server has real limits — local disk retention, no built-in horizontal scale-out, no native multi-tenancy. Projects like Mimir, Cortex, Thanos, and VictoriaMetrics exist specifically to take the Prometheus data model and TSDB format and make it horizontally scalable, long-term, and multi-tenant. Mimir’s positioning is covered in tech/mimir.md. The comparative architecture question — how each of these layers actually scales ingestion, storage, and query fan-out, and how multi-tenancy gets implemented at that scale — is covered in Scaling Each Layer and Multi-Tenancy. This chapter is only flagging that the question exists, not answering it.
OpenTelemetry Relationship
This book does not yet cover how Prometheus relates to OpenTelemetry — that’s an honest gap, not an oversight to paper over.
Metadata
| Author | Amit Singh |
| Scope | prometheus |
Local graph
Linked from 2 notes
What is Prometheus
CNCF's second graduated project (2018) — the pull-based metrics monitoring system and query language (PromQL) that defined the exposition format nearly every metrics tool now speaks, and the API that Grafana Mimir scales out horizontally.
Prometheus
A book-shaped table of contents for Prometheus: monitoring foundations through architecture, data model, instrumentation, service discovery, PromQL, alerting, production operation, PCA certification, and MAANG interview prep — cross-linking existing notes instead of duplicating them.
Related notes
2 — Exporters
What a Prometheus exporter is, installing Node Exporter as a systemd service, and monitoring the container runtime itself via Docker Engine metrics and cAdvisor.
1 — Discovery Mechanisms
How Prometheus finds scrape targets — static configs, file-based service discovery with watched JSON files, and DNS service discovery via SRV/A records — plus validating and reloading configuration safely.
5 — Advanced PromQL
The complete recording-rule syntax reference (rule files, worked example, level:metric_name:operations naming), the offset and @ modifiers, and subqueries.
3 — Aggregation Operators
The PromQL aggregation operator table, the by clause, the without clause, and worked collapsing examples across single and multiple labels.