How a 'Consistency Fix' in Program.cs Silently Broke Label Promotion Across Mimir and Loki
Renaming an OTel resource attribute from deployment.environment to deployment_environment in application code looked like the fix for inconsistent labels across Grafana Cloud's three signals. It's backwards: Mimir and Loki both promote and convert dotted attribute names on ingestion, so a pre-converted attribute silently fails to match and drops out of both label sets. The real fix belongs in the Alloy pipeline, scoped to traces only.
How Alloy's Default max_shards Turned a Mimir Blip Into a Production Monitoring Blackout
When Grafana Mimir slowed down, Alloy's default retry configuration — 200 parallel shard workers — consumed the majority of CPU on a shared VM, starving the co-located business process and forcing an engineer to kill the observability agent during an active incident. Here's the exact three-layer fix and why shared-VM deployments require explicit resource budgets at every level.
Bridging OT and IT Observability: You Meet at the Historian, Not the PLC
Manufacturing observability tempts you to instrument the plant floor directly — point a collector at the PLCs, scrape everything. That path runs into a safety boundary, an air gap, and a cardinality wall: an unscoped Windows process collector was already emitting 3,900 series per plant mid-rollout. The integration point that actually works is the process historian, and the fragile link is a service account nobody tested.
On-Prem Observability Breaks Every Assumption Your Cloud Collector Made
The Alloy config that works flawlessly as an AKS DaemonSet becomes a liability on a plant-floor VM. No elastic compute means a retry storm starves the workload it shares a host with. No managed identity means static token rotation. Egress restrictions mean the Grafana Cloud endpoint isn't reachable the way you assume. On-prem isn't cloud with worse latency — it's a different set of constraints.
One Un-Instrumented Hop Breaks the Whole Trace
Distributed tracing gives you the illusion of end-to-end visibility right up until a request crosses a message queue, a legacy proxy, or a cron-triggered batch job that doesn't forward the trace context. The span chain snaps, Tempo stores two unrelated fragments, and the 3am question — where did the time go — has no answer. Context propagation is the whole game.
Hybrid Observability Scales on the Label Schema, Not the Collector Count
The hard part of hybrid and multi-cloud observability isn't running collectors in every environment — it's making one query resolve identically whether the data came from AKS, an on-prem plant VM, or SAP RISE. That needs a single enforced label schema, set at the collector layer, with no segment optional. Without it, cross-environment queries become per-source OR branches and the cost dashboard becomes an unattributable blob.
Push vs Pull Was Never the Point — Rethinking Metrics for the OTLP Era
The push-versus-pull debate is settled for application metrics: OTLP push through a collector. The migration that actually matters is what comes with it — resource-attribute promotion instead of relabel configs, exemplars linking metrics to traces, and a deliberate delta-vs-cumulative temporality choice. Lift-and-shift your Prometheus scrape config into OTLP and you keep the mechanics while losing the point.
Learn the Observability Pipeline, Not the Tools — a Map That Survives a Vendor Swap
Beginners drown memorizing Prometheus vs Loki vs Tempo vs Jaeger vs Alloy vs Fluent Bit. The durable model is a six-stage pipeline — instrument, collect, process, store, query, alert — where every tool is a swappable implementation of one stage. When we evaluated four whole observability stacks for the platform I lead, the pipeline shape was identical across all four; only the slots changed.
Monitoring Answers the Questions You Wrote Down. Observability Answers the Ones You Didn't.
A PromQL query grouped by deployment_environment silently returned nothing — no error, no failed scrape, no alert, every dashboard green. Nothing was broken in the way monitoring understands broken; the label had just stopped being promoted. This is the practical line between monitoring (a fixed set of questions frozen at design time) and observability (asking new questions of data you already collected), and what it costs to confuse the two.