How a 'Consistency Fix' in Program.cs Silently Broke Label Promotion Across Mimir and Loki
Renaming an OTel resource attribute from deployment.environment to deployment_environment in application code looked like the fix for inconsistent labels across Grafana Cloud's three signals. It's backwards: Mimir and Loki both promote and convert dotted attribute names on ingestion, so a pre-converted attribute silently fails to match and drops out of both label sets. The real fix belongs in the Alloy pipeline, scoped to traces only.
One Un-Instrumented Hop Breaks the Whole Trace
Distributed tracing gives you the illusion of end-to-end visibility right up until a request crosses a message queue, a legacy proxy, or a cron-triggered batch job that doesn't forward the trace context. The span chain snaps, Tempo stores two unrelated fragments, and the 3am question — where did the time go — has no answer. Context propagation is the whole game.
The Three Pillars of Observability Are a Storage Detail, Not a Strategy
Metrics, logs, and traces name three storage engines with different index models and cost curves — not three things to instrument separately. Teams that organize around the pillars build three disconnected silos and still can't say why one request was slow. The unit that matters is the correlated event: one trace_id and an exemplar that stitches a histogram bucket to the trace to the logs.