Agent Memory as Infrastructure: The Context Injection Pattern That Replaces Fine-Tuning
Built four production AI specialists off a single LLM using plain markdown files and a 20-line fetch call — no fine-tuning, no frameworks. The real insight is that LLM behavioral specialization is a configuration problem, not a training problem, and applying IaC discipline to memory files gives you diffability, rollback, and CI-gated behavioral regression testing that fine-tuning can never provide. At MAANG scale, this extends to prompt caching economics, governance workflows, distributed session state, and multi-model routing.
Building an SRE Agent: From Playbook to Autonomous Incident Response
What happens when you put an LLM in the incident response loop? We built an SRE Agent on AKS that runs playbooks, queries Grafana, and generates post-mortems. Here's what worked, what didn't, and what we learned about trusting AI in production.
How a 'Consistency Fix' in Program.cs Silently Broke Label Promotion Across Mimir and Loki
Renaming an OTel resource attribute from deployment.environment to deployment_environment in application code looked like the fix for inconsistent labels across Grafana Cloud's three signals. It's backwards: Mimir and Loki both promote and convert dotted attribute names on ingestion, so a pre-converted attribute silently fails to match and drops out of both label sets. The real fix belongs in the Alloy pipeline, scoped to traces only.
How Alloy's Default max_shards Turned a Mimir Blip Into a Production Monitoring Blackout
When Grafana Mimir slowed down, Alloy's default retry configuration — 200 parallel shard workers — consumed the majority of CPU on a shared VM, starving the co-located business process and forcing an engineer to kill the observability agent during an active incident. Here's the exact three-layer fix and why shared-VM deployments require explicit resource budgets at every level.
You Don't Get Root on SAP RISE: Observability Inside a Managed Black Box
RISE with SAP hands you a mission-critical ERP landscape and takes away the thing every observability playbook assumes: OS access to put an agent on the host. The workable model is to monitor the contract, not the internals — synthetic business transactions, the interfaces SAP does expose, and the integration layers you still own. Plan to drop Alloy on the app servers and the project stalls at 'we can't get in.'
Bridging OT and IT Observability: You Meet at the Historian, Not the PLC
Manufacturing observability tempts you to instrument the plant floor directly — point a collector at the PLCs, scrape everything. That path runs into a safety boundary, an air gap, and a cardinality wall: an unscoped Windows process collector was already emitting 3,900 series per plant mid-rollout. The integration point that actually works is the process historian, and the fragile link is a service account nobody tested.
The Grafana Terraform Provider Silently Drops LBAC Rules — and the Three-Layer Fix
When we codified Grafana Cloud multi-tenancy as Terraform, label-based access control rules applied cleanly in the plan and silently did nothing at runtime. The provider ignores LBAC config on Terraform-provisioned datasources. This post covers what actually works: a three-layer architecture splitting access control across provider aliases, manually-created datasource UIDs, and Alloy write-time label enforcement.
The Terraform Module That Has No Provider: How a Pure YAML Registry Drives a Multi-Tenant Grafana Platform
We onboard products to two Grafana Cloud stacks — dashboards, alert rules, RBAC, LBAC, contact points, OnCall — by editing one YAML file. No Terraform file changes. The key architectural decision was a provider-free Terraform module that reads the registry and derives everything else. This post covers the pattern, why it works, and where it breaks.
Hybrid Observability Scales on the Label Schema, Not the Collector Count
The hard part of hybrid and multi-cloud observability isn't running collectors in every environment — it's making one query resolve identically whether the data came from AKS, an on-prem plant VM, or SAP RISE. That needs a single enforced label schema, set at the collector layer, with no segment optional. Without it, cross-environment queries become per-source OR branches and the cost dashboard becomes an unattributable blob.