Cardinality is the number of unique label-value combinations a metric produces — each unique combination is one time series. It’s the central cost and performance variable in any Prometheus- family backend (Prometheus, Mimir, Cortex, Thanos): every series has to be stored, indexed, and held in memory for querying, so cardinality growth is what actually breaks these systems, not raw sample volume.
Why one label can multiply cardinality
A metric’s cardinality is the product of its label value counts, not the sum:
http_requests_total{method, status, endpoint, pod}
method: 4 values (GET, POST, PUT, DELETE)
status: 6 values (200, 201, 400, 401, 404, 500)
endpoint: 50 values (routes in the service)
pod: 20 values (replica count, churns on every deploy)
Total series = 4 × 6 × 50 × 20 = 24,000 active series
for ONE metric name
Add one high-churn label — pod in the example above is already a warning sign, because pod names
change on every rollout, meaning old series go stale and new ones spin up constantly (series
churn, a second cost dimension beyond flat cardinality). Add something genuinely unbounded — a
request_id, a raw user_id, a full URL with query params — and that same metric goes from 24,000
series to unbounded, unpredictable growth.
Where the cost actually lands
| System | What cardinality drives |
|---|---|
| Prometheus/Mimir | Active series held in memory (ingester working set) — the primary OOM cause |
| Query engine | Every PromQL query touching that metric scans all matching series |
| Grafana Cloud bill | Priced on active series + samples/sec — cardinality is the ingest cost lever |
| Loki (structured logs) | Same problem, one layer up: high-cardinality labels on log streams (not the log lines themselves) fragment streams and blow up the index |
| Tempo (traces) | Span attributes with unbounded values don’t multiply series the same way, but still bloat storage and hurt trace-search performance |
The label-keep vs. label-drop decision
The standing rule: default to dropping or hashing a label before keeping it, and treat any label sourced from a high-churn field (request IDs, user IDs, raw timestamps, full paths with path params) as an automatic stop.
New label proposed
│
▼
Is the value set bounded and known ahead of time?
│ │
YES NO
│ │
▼ ▼
Keep as a label Push it elsewhere:
(method, status, • Log line (Loki) — searchable, not a series dimension
region, env) • Trace attribute (Tempo) — per-span, not per-series
• Exemplar — sampled link from a metric sample to a trace
This is exactly the job of a cardinality budget: before a new metric or label ships into an Alloy/OTel-collector/Prometheus config, estimate active series count, samples/sec, and monthly ingest cost impact — never ship on a “looks fine” guess. See the Cardinality Budget Calculator skill for the concrete estimation workflow.
Mitigation techniques, in order of preference
| Technique | What it does |
|---|---|
| Don’t add the label | Cheapest fix — ask whether the dimension is needed at query time at all |
| Drop at collection | Alloy/OTel processor drops the label before it ever reaches the write path |
| Hash / bucket | Collapse a wide value set into a small number of buckets (e.g. status class 2xx/4xx/5xx instead of raw status code) |
| Recording rules | Pre-aggregate high-cardinality raw series into a smaller derived series for the dashboards/alerts that actually get queried |
| Route to logs/traces instead | Move the high-churn dimension out of the metrics data plane entirely — see What is Telegraf for an example of aggregation happening at the collector layer, same principle Alloy applies via label-drop |
| Per-tenant limits | Backend-enforced ceiling (Mimir tenant limits) as the last line of defense, not the primary control |
Quick mental check before adding any label
Ask: if this service scales from 20 pods to 200, or onboards 10x the users, does this label’s value count grow with it? If yes, it’s not a label — it’s an exemplar, a log field, or a trace attribute.
Why it matters here: this is the gating question behind every Alloy/OTel/Prometheus config change in the observability pillar — the standing rule is that cardinality impact is surfaced in the same response as the config itself, never silently.
Local graph
Linked from 32 notes
5 — Label & Attribute Schema Design
Cardinality budget, naming conventions, and the high-churn label traps that turn a cheap metric into a production incident — the design discipline for the labels semantic conventions don't already cover for you.
7 — Metrics Storage (TSDB)
Chunk/block encoding, the write-ahead log, compaction and the write amplification it trades for query speed, and why a cardinality spike is a storage-engine problem, not just a cost line item.
3 — Deep Dive Discussions
Interview-framed answers to the 'why' questions candidates get asked about Prometheus — why pull, why not SQL, why labels — honestly scoped to what this book actually has source material for.
What is ArgoCD
CNCF-graduated declarative GitOps continuous delivery tool for Kubernetes — pull-based reconciliation from Git via an Application CRD, the App-of-Apps pattern for fleet management, and Argo Rollouts for canary/blue-green progressive delivery.
1 — What Observability Actually Means
Observability vs. monitoring, the three-pillars critique, and why observability is a property of how a system was instrumented — not a tool you bought or a dashboard you built.
3 — Push-Based vs Pull-Based Ingestion
Core mental model for telemetry and data ingestion patterns — when to push, when to pull, and how to reason about the trade-offs at principal/staff interview bar.
What is Fluent Bit
CNCF-graduated, C-written log/metrics/trace forwarder — Fluentd's lightweight sibling, the de facto node-level log-collection DaemonSet in most Kubernetes clusters, and Grafana Alloy's main incumbent competitor for that slot.
What is Loki
Grafana Labs' log aggregation system — 'like Prometheus, but for logs': index only labels, store compressed chunks in object storage, query with LogQL. Shares its distributor/ingester/compactor architecture with Mimir and Tempo.
What is Mimir
Grafana Labs' horizontally-scalable, multi-tenant long-term storage for Prometheus metrics — the 2022 successor to Cortex, and the actual system serving every PromQL query and remote-write in a Grafana Cloud metrics stack.
What is Prometheus
CNCF's second graduated project (2018) — the pull-based metrics monitoring system and query language (PromQL) that defined the exposition format nearly every metrics tool now speaks, and the API that Grafana Mimir scales out horizontally.
What is StatsD
Etsy's 2011 UDP-based metrics protocol and daemon — the simplest possible fire-and-forget instrumentation format, superseded as a client API by OTel/Prometheus but still alive everywhere as a compatibility ingestion shim.
What is Telegraf
InfluxData's plugin-driven metrics/events/logs collection agent — 300+ input/output plugins, written in Go, single static binary — the collector layer in the InfluxDB (TICK-stack-descendant) ecosystem, comparable in role to Grafana Alloy.
Related notes
What is Cortex (cortexproject)
CNCF Incubating, horizontally-scalable multi-tenant long-term storage for Prometheus — the project Grafana Mimir forked from in 2022, still maintained as the vendor-neutral, community-governed alternative once Grafana Labs redirected engineering effort to Mimir.
What is Mimir
Grafana Labs' horizontally-scalable, multi-tenant long-term storage for Prometheus metrics — the 2022 successor to Cortex, and the actual system serving every PromQL query and remote-write in a Grafana Cloud metrics stack.
What is Prometheus
CNCF's second graduated project (2018) — the pull-based metrics monitoring system and query language (PromQL) that defined the exposition format nearly every metrics tool now speaks, and the API that Grafana Mimir scales out horizontally.
What is StatsD
Etsy's 2011 UDP-based metrics protocol and daemon — the simplest possible fire-and-forget instrumentation format, superseded as a client API by OTel/Prometheus but still alive everywhere as a compatibility ingestion shim.