Notes / Observability / Reference

What is Prometheus

CNCF's second graduated project (2018) — the pull-based metrics monitoring system and query language (PromQL) that defined the exposition format nearly every metrics tool now speaks, and the API that Grafana Mimir scales out horizontally.

Updated July 12, 2026 · §202607121601-6 ·

Prometheus is a metrics monitoring system built at SoundCloud in 2012, modeled on Google’s internal Borgmon, and donated to the CNCF in 2016 — the second project accepted after Kubernetes itself, and the first to graduate alongside it in 2018. Its exposition format and query model (PromQL) are now the de facto standard: OTel metrics, cloud-provider exporters, and most vendor agents either speak Prometheus format natively or convert to/from it at the boundary.


Core model: pull, not push

Prometheus server

     │  scrape (HTTP GET) on interval, via service discovery

target:9090/metrics  ──▶  text exposition format

http_requests_total{method="GET",status="200"} 84213
http_request_duration_seconds_bucket{le="0.1"}  71200
node_memory_available_bytes                     3.4e9

Prometheus reaches out to targets rather than waiting for them to push — service discovery (k8s SD, file SD, Consul, EC2, …) tells it what to scrape, and a missed scrape is itself an observable signal (up == 0), which a push model doesn’t get for free.

Four metric types

TypeClient-side semantics
CounterMonotonically increasing — use rate()/increase() in PromQL, never read raw
GaugePoint-in-time value that can go up or down
HistogramPre-defined buckets, server-side percentile math via histogram_quantile()
SummaryClient-side quantiles — cheaper to query, can’t be aggregated across instances

Histograms are almost always preferred over summaries in a multi-instance deployment specifically because they aggregate correctly across replicas — a summary’s client-computed p99 across 20 pods isn’t mathematically a real p99 of anything.

What Prometheus alone doesn’t do

LimitationWhy it matters
Local, single-node TSDBNo built-in HA or horizontal scale-out — one server, one disk, one blast radius
No long-term retention by designLocal storage is meant to be a buffer, not a durable multi-year store
No native multi-tenancyOne Prometheus = one tenant; multi-team isolation needs external tooling

This is precisely the gap What is Mimir fills: Mimir implements the Prometheus remote-write receive API and the PromQL query API at horizontal, multi-tenant scale, so a Prometheus server (or Alloy’s scrape component, which increasingly replaces standalone Prometheus for this role) becomes the local scraper that ships data onward via remote_write, while Mimir is the system of record.

Alloy/Prometheus (scrape + remote_write)  ──▶  Mimir (store + serve PromQL)  ──▶  Grafana

Alertmanager

Alerting is deliberately a separate component: Prometheus evaluates alerting rules and fires alerts, Alertmanager deduplicates, groups, silences, and routes them to receivers (PagerDuty, Slack, webhook). Keeping rule evaluation and routing decoupled is why the same Alertmanager deployment can sit downstream of many Prometheus/Mimir rulers without duplicating routing logic per source.

Why it matters here: Prometheus’s exposition format and query semantics are the contract every ShipSolid service’s /metrics endpoint and every Grafana dashboard PromQL query is written against — Mimir is the horizontally-scaled implementation of that same API, not a different query language to learn. Every label added to a scrape target is subject to the same What is Cardinality (in observability) budget as if it were being written straight into a single-node Prometheus, just distributed across Mimir’s ingesters instead of one disk.

Local graph

Full graph →

Linked from 14 notes

KPIs, SLIs, SLOs & SLAs

Defines the **metrics hierarchy** used to align technical observability signals with business

Metric Label Standards for Cost Attribution

**Goal:** Attribute Grafana Cloud metrics ingestion cost to business unit, product, and environment.

What is Cortex (cortexproject)

CNCF Incubating, horizontally-scalable multi-tenant long-term storage for Prometheus — the project Grafana Mimir forked from in 2022, still maintained as the vendor-neutral, community-governed alternative once Grafana Labs redirected engineering effort to Mimir.

What is Mimir

Grafana Labs' horizontally-scalable, multi-tenant long-term storage for Prometheus metrics — the 2022 successor to Cortex, and the actual system serving every PromQL query and remote-write in a Grafana Cloud metrics stack.

What is StatsD

Etsy's 2011 UDP-based metrics protocol and daemon — the simplest possible fire-and-forget instrumentation format, superseded as a client API by OTel/Prometheus but still alive everywhere as a compatibility ingestion shim.

Helm Monitoring Stack

How the grafana/k8s-monitoring Helm chart deploys Alloy roles for collecting and exporting telemetry, and why it replaced the legacy Makefile flow.

Local Deployment

Step-by-step guide to deploying the full lab locally on k3d via deploy-local.sh, including cluster setup and mode switching.

3 — Prometheus in the Observability Ecosystem

Where Prometheus sits in the CNCF landscape — its pull-based cloud-native origins, its companion projects, and where this book does (and doesn't yet) connect it to the wider stack.

7. Component Map (What Exists in the Wild)

OSS and managed-SaaS options for every layer of the telemetry ingestion pipeline, mapped against ShipSolid's own production experience.

Agentic AI: Projects & Engineering Mastery

A book-shaped table of contents for Agentic AI: Projects & Engineering Mastery: hands-on practitioner builds, Principal/Staff-level technical leadership, and the lookup appendices and vendor/framework reference notes for the whole series. Book 6 of the AI Systems Engineering series.

2 — The Signals

Metrics, logs, traces, profiles, and events — what each is built to capture, what it costs, and which question it actually answers vs. which one people mistakenly ask it.

7 — Metrics Storage (TSDB)

Chunk/block encoding, the write-ahead log, compaction and the write amplification it trades for query speed, and why a cardinality spike is a storage-engine problem, not just a cost line item.