Blog
Writing from the trenches
Practitioner-level posts on observability, SRE, platform engineering, and the occasional deep-dive into distributed systems. No fluff.
The Productivity Trap Nobody Talks About: When "Productive" Work Becomes Procrastination
Reinstalling Linux, rebuilding dotfiles, reorganizing repos — it all feels like progress, until you notice the thing you actually needed to learn never got touched. A framework for telling optimization work apart from avoidance.
Build Day: Agent Harness & Memory — Key Learnings on Agent Harnesses, Context Engineering, and Memory Systems
Reflections from Build Club & Mem0, Pune (June 2026)
Unbounded Cardinality, Zero Alerts, and 14-Second Dashboard Loads: The Structural Limits of the Grafana Azure Monitor Plugin
The Grafana Azure Monitor datasource plugin connects directly to Azure Monitor at query time. On a 12-panel dashboard, that means 12 sequential ARM API calls on every open — 14 seconds before the first chart renders. It cannot feed Mimir alerting rules. It cannot attach environment labels without custom queries per panel. And promoting Azure tags as Prometheus labels explodes cardinality to thousands of series per resource group. This post documents the five structural gaps and the specific design decisions in a push-based exporter that close each one.
Agent Memory as Infrastructure: The Context Injection Pattern That Replaces Fine-Tuning
Built four production AI specialists off a single LLM using plain markdown files and a 20-line fetch call — no fine-tuning, no frameworks. The real insight is that LLM behavioral specialization is a configuration problem, not a training problem, and applying IaC discipline to memory files gives you diffability, rollback, and CI-gated behavioral regression testing that fine-tuning can never provide. At MAANG scale, this extends to prompt caching economics, governance workflows, distributed session state, and multi-model routing.
Prioritization Frameworks for Engineering: Make Decisions That Actually Move the Needle
In on-call engineering cultures, urgency and importance feel identical—until you're weeks into work no one needed. This post combines the Complexity/Effort Matrix with the Eisenhower Decision Matrix into a five-tier execution system, with a ready-made 4×4 template for your Monday backlog triage.
How Alloy's Default max_shards Turned a Mimir Blip Into a Production Monitoring Blackout
When Grafana Mimir slowed down, Alloy's default retry configuration — 200 parallel shard workers — consumed the majority of CPU on a shared VM, starving the co-located business process and forcing an engineer to kill the observability agent during an active incident. Here's the exact three-layer fix and why shared-VM deployments require explicit resource budgets at every level.
The Self-Silencing Anti-Pattern: Why Your Observability Stack Goes Blind When You Need It Most
The system effectively silences its own diagnostics during the moment of maximum operational need.
The Terraform Module That Has No Provider: How a Pure YAML Registry Drives a Multi-Tenant Grafana Platform
We onboard products to two Grafana Cloud stacks — dashboards, alert rules, RBAC, LBAC, contact points, OnCall — by editing one YAML file. No Terraform file changes. The key architectural decision was a provider-free Terraform module that reads the registry and derives everything else. This post covers the pattern, why it works, and where it breaks.
The Grafana Terraform Provider Silently Drops LBAC Rules — and the Three-Layer Fix
When we codified Grafana Cloud multi-tenancy as Terraform, label-based access control rules applied cleanly in the plan and silently did nothing at runtime. The provider ignores LBAC config on Terraform-provisioned datasources. This post covers what actually works: a three-layer architecture splitting access control across provider aliases, manually-created datasource UIDs, and Alloy write-time label enforcement.
Building an SRE Agent: From Playbook to Autonomous Incident Response
What happens when you put an LLM in the incident response loop? We built an SRE Agent on AKS that runs playbooks, queries Grafana, and generates post-mortems. Here's what worked, what didn't, and what we learned about trusting AI in production.
Why Cardinality Kills Observability Platforms (and How to Stop It)
Cardinality is the silent killer of Prometheus-based observability platforms. Here's how it happens, how to detect it early, and the label schema discipline that keeps ingestion costs sane at scale.