Unbounded Cardinality, Zero Alerts, and 14-Second Dashboard Loads: The Structural Limits of the Grafana Azure Monitor Plugin
The Grafana Azure Monitor datasource plugin connects directly to Azure Monitor at query time. On a 12-panel dashboard, that means 12 sequential ARM API calls on every open — 14 seconds before the first chart renders. It cannot feed Mimir alerting rules. It cannot attach environment labels without custom queries per panel. And promoting Azure tags as Prometheus labels explodes cardinality to thousands of series per resource group. This post documents the five structural gaps and the specific design decisions in a push-based exporter that close each one.
The Self-Silencing Anti-Pattern: Why Your Observability Stack Goes Blind When You Need It Most
The system effectively silences its own diagnostics during the moment of maximum operational need.
Why Cardinality Kills Observability Platforms (and How to Stop It)
Cardinality is the silent killer of Prometheus-based observability platforms. Here's how it happens, how to detect it early, and the label schema discipline that keeps ingestion costs sane at scale.
Push vs Pull Was Never the Point — Rethinking Metrics for the OTLP Era
The push-versus-pull debate is settled for application metrics: OTLP push through a collector. The migration that actually matters is what comes with it — resource-attribute promotion instead of relabel configs, exemplars linking metrics to traces, and a deliberate delta-vs-cumulative temporality choice. Lift-and-shift your Prometheus scrape config into OTLP and you keep the mechanics while losing the point.
Monitoring Answers the Questions You Wrote Down. Observability Answers the Ones You Didn't.
A PromQL query grouped by deployment_environment silently returned nothing — no error, no failed scrape, no alert, every dashboard green. Nothing was broken in the way monitoring understands broken; the label had just stopped being promoted. This is the practical line between monitoring (a fixed set of questions frozen at design time) and observability (asking new questions of data you already collected), and what it costs to confuse the two.