Prometheus is a metrics monitoring system built at SoundCloud in 2012, modeled on Google’s internal Borgmon, and donated to the CNCF in 2016 — the second project accepted after Kubernetes itself, and the first to graduate alongside it in 2018. Its exposition format and query model (PromQL) are now the de facto standard: OTel metrics, cloud-provider exporters, and most vendor agents either speak Prometheus format natively or convert to/from it at the boundary.
Core model: pull, not push
Prometheus server
│
│ scrape (HTTP GET) on interval, via service discovery
▼
target:9090/metrics ──▶ text exposition format
│
http_requests_total{method="GET",status="200"} 84213
http_request_duration_seconds_bucket{le="0.1"} 71200
node_memory_available_bytes 3.4e9
Prometheus reaches out to targets rather than waiting for them to push — service discovery (k8s SD,
file SD, Consul, EC2, …) tells it what to scrape, and a missed scrape is itself an observable
signal (up == 0), which a push model doesn’t get for free.
Four metric types
| Type | Client-side semantics |
|---|---|
| Counter | Monotonically increasing — use rate()/increase() in PromQL, never read raw |
| Gauge | Point-in-time value that can go up or down |
| Histogram | Pre-defined buckets, server-side percentile math via histogram_quantile() |
| Summary | Client-side quantiles — cheaper to query, can’t be aggregated across instances |
Histograms are almost always preferred over summaries in a multi-instance deployment specifically because they aggregate correctly across replicas — a summary’s client-computed p99 across 20 pods isn’t mathematically a real p99 of anything.
What Prometheus alone doesn’t do
| Limitation | Why it matters |
|---|---|
| Local, single-node TSDB | No built-in HA or horizontal scale-out — one server, one disk, one blast radius |
| No long-term retention by design | Local storage is meant to be a buffer, not a durable multi-year store |
| No native multi-tenancy | One Prometheus = one tenant; multi-team isolation needs external tooling |
This is precisely the gap What is Mimir fills: Mimir implements the Prometheus remote-write receive
API and the PromQL query API at horizontal, multi-tenant scale, so a Prometheus server (or Alloy’s
scrape component, which increasingly replaces standalone Prometheus for this role) becomes the local
scraper that ships data onward via remote_write, while Mimir is the system of record.
Alloy/Prometheus (scrape + remote_write) ──▶ Mimir (store + serve PromQL) ──▶ Grafana
Alertmanager
Alerting is deliberately a separate component: Prometheus evaluates alerting rules and fires alerts, Alertmanager deduplicates, groups, silences, and routes them to receivers (PagerDuty, Slack, webhook). Keeping rule evaluation and routing decoupled is why the same Alertmanager deployment can sit downstream of many Prometheus/Mimir rulers without duplicating routing logic per source.
Why it matters here: Prometheus’s exposition format and query semantics are the contract every
ShipSolid service’s /metrics endpoint and every Grafana dashboard PromQL query is written against
— Mimir is the horizontally-scaled implementation of that same API, not a different query language
to learn. Every label added to a scrape target is subject to the same What is Cardinality (in observability) budget as if
it were being written straight into a single-node Prometheus, just distributed across Mimir’s
ingesters instead of one disk.
Local graph
Linked from 14 notes
KPIs, SLIs, SLOs & SLAs
Defines the **metrics hierarchy** used to align technical observability signals with business
Metric Label Standards for Cost Attribution
**Goal:** Attribute Grafana Cloud metrics ingestion cost to business unit, product, and environment.
What is Cortex (cortexproject)
CNCF Incubating, horizontally-scalable multi-tenant long-term storage for Prometheus — the project Grafana Mimir forked from in 2022, still maintained as the vendor-neutral, community-governed alternative once Grafana Labs redirected engineering effort to Mimir.
What is Mimir
Grafana Labs' horizontally-scalable, multi-tenant long-term storage for Prometheus metrics — the 2022 successor to Cortex, and the actual system serving every PromQL query and remote-write in a Grafana Cloud metrics stack.
What is StatsD
Etsy's 2011 UDP-based metrics protocol and daemon — the simplest possible fire-and-forget instrumentation format, superseded as a client API by OTel/Prometheus but still alive everywhere as a compatibility ingestion shim.
Helm Monitoring Stack
How the grafana/k8s-monitoring Helm chart deploys Alloy roles for collecting and exporting telemetry, and why it replaced the legacy Makefile flow.
Local Deployment
Step-by-step guide to deploying the full lab locally on k3d via deploy-local.sh, including cluster setup and mode switching.
3 — Prometheus in the Observability Ecosystem
Where Prometheus sits in the CNCF landscape — its pull-based cloud-native origins, its companion projects, and where this book does (and doesn't yet) connect it to the wider stack.
7. Component Map (What Exists in the Wild)
OSS and managed-SaaS options for every layer of the telemetry ingestion pipeline, mapped against ShipSolid's own production experience.
Agentic AI: Projects & Engineering Mastery
A book-shaped table of contents for Agentic AI: Projects & Engineering Mastery: hands-on practitioner builds, Principal/Staff-level technical leadership, and the lookup appendices and vendor/framework reference notes for the whole series. Book 6 of the AI Systems Engineering series.
2 — The Signals
Metrics, logs, traces, profiles, and events — what each is built to capture, what it costs, and which question it actually answers vs. which one people mistakenly ask it.
7 — Metrics Storage (TSDB)
Chunk/block encoding, the write-ahead log, compaction and the write amplification it trades for query speed, and why a cardinality spike is a storage-engine problem, not just a cost line item.
Related notes
What is Cortex (cortexproject)
CNCF Incubating, horizontally-scalable multi-tenant long-term storage for Prometheus — the project Grafana Mimir forked from in 2022, still maintained as the vendor-neutral, community-governed alternative once Grafana Labs redirected engineering effort to Mimir.
What is Fluent Bit
CNCF-graduated, C-written log/metrics/trace forwarder — Fluentd's lightweight sibling, the de facto node-level log-collection DaemonSet in most Kubernetes clusters, and Grafana Alloy's main incumbent competitor for that slot.
What is Jaeger
CNCF-graduated distributed tracing system built at Uber in 2015, Dapper-lineage like Zipkin before it — and, since Jaeger v2, rebuilt on top of the OpenTelemetry Collector rather than bespoke ingestion code.
What is Mimir
Grafana Labs' horizontally-scalable, multi-tenant long-term storage for Prometheus metrics — the 2022 successor to Cortex, and the actual system serving every PromQL query and remote-write in a Grafana Cloud metrics stack.