1 — AIOps / Agentic RCA
A static runbook has always been “if condition X, run script Y” — a fixed, human-authored mapping, brittle the moment a failure doesn’t match the exact condition someone anticipated. What’s actually new about agentic RCA isn’t “AI is now involved in ops” — it’s replacing that fixed mapping with an investigation loop: an agent that can query metrics, pull the trace an exemplar points at, read the log line that trace’s span produced, form a hypothesis, and query again to test it — the same drill-down The Signals describes a human doing, run by something that can do it in seconds and never gets tired of checking one more dashboard.
This depends on everything earlier in this book already being true
An agent doing this well isn’t doing anything a human couldn’t do with the same access — it’s doing it faster, and only as well as the telemetry underneath it actually supports:
- It needs 3 — Cross-Signal Correlation genuinely wired up, the same way a human does, to pivot from a metric spike to the one trace and log line that explain it — an agent facing three disconnected signal types is stuck exactly where an untrained human is.
- It needs consistent semantic conventions and label discipline across services, so a tool call that works against one service’s telemetry generalizes to the next one instead of needing bespoke prompting per service.
- It’s automating the same top-down, symptom-to-cause drill path Dashboard Design lays a human dashboard out for — a badly-instrumented system gives the agent the same blind spots it gives a human, just delivered with more confident-sounding wrong answers.
Agentic RCA is best understood as automating a well-instrumented human investigation, not as a substitute for one. It doesn’t route around bad instrumentation — it inherits every blind spot bad instrumentation already has.
Trigger-action mappings, redefined
The trigger-action idea survives, but the “action” changes shape. A static runbook’s action is a fixed script; an agentic system’s action is “invoke an agent, with a bounded toolset, inside explicit guardrails” — the actual remediation, if any, is delegated to a scoped investigation loop rather than hard-coded in advance. This is more flexible than a static mapping and correspondingly harder to reason about in advance, which is exactly why the guardrails matter more, not less.
Safety guardrails: read access first, write access is a different risk class entirely
Two separate lines matter:
- Investigation vs. remediation. An agent scoped to read telemetry and propose a hypothesis is a fundamentally lower-risk category than one authorized to act — restart a pod, roll back a deploy, scale a service. Most production deployments today are investigation-only: the agent surfaces a diagnosis for a human to approve, rather than acting autonomously.
- Blast-radius discipline for anything that does act. A remediation-capable agent should get the same discipline any automation gets before it’s trusted with production: a dry-run mode, a staged or canary rollout of its own fix, and a human approval gate before anything irreversible. An agent that can act autonomously without those guardrails isn’t a faster on-call engineer — it’s a new, less-tested failure mode with write access.
HolmesGPT is a concrete real-world example of the investigation-only end of this
spectrum: an open-source, tool-calling agent over 70+ toolsets, with a proactive “operator mode”
that opens fix PRs for a human to review rather than applying fixes itself. The closest lived
version of this work outside this site’s own content is the SRE agent scaffold in the main
learning-lab monorepo (h-aiops/) — mentioned here in prose since it lives outside this site’s
content collection, not as something to link to.
Where to go for the hands-on build
This chapter is the conceptual frame; the AI Systems Engineering series’ own progression is where the build-it-yourself depth is scoped to live — from a first tool-using agent through a dedicated Automated Root-Cause Analysis chapter and log/trace investigation tools in Agentic AI: Projects & Engineering Mastery, to a multi-agent Supervisor Architectures chapter in Building & Evaluating Agents and the reliability concerns in Prompt Injection for agentic systems specifically in Production Agent Systems. That series owns the “how do you actually build one” ground this chapter deliberately doesn’t retread.
Why this matters for an Observability Architect
The question worth asking about any agentic RCA proposal isn’t “can the model do this” — for a well-instrumented system, often yes. It’s “what’s the blast radius if it’s confidently wrong,” and “where exactly is the read/write line drawn.” An investigation agent that’s wrong wastes an on-call engineer’s time double-checking a bad hypothesis; a remediation agent that’s wrong, without a dry-run and an approval gate in between, can turn a contained incident into a self-inflicted one.
Metadata
| Dimension | Detail |
|---|---|
| Author | Amit Singh |
| Scope | observability |
Local graph
Linked from 8 notes
What is Azure SRE Agent (Microsoft)
Microsoft's AI agent embedded in Azure for autonomous incident response — acknowledges alerts from PagerDuty/ServiceNow/Azure Monitor, investigates via Azure Monitor/App Insights/Kusto, executes configurable Incident Response Plans with tunable autonomy, and learns across incidents via Session Insights.
What is HolmesGPT
Robusta.dev's open-source SRE agent (CNCF Sandbox) for investigating production incidents across Kubernetes, VMs, cloud services, and databases — an agentic tool-calling loop over 70+ toolsets, not a chatbot or RAG system, with a proactive 'operator mode' that monitors and opens fix PRs without a human trigger.
8 — Case Study: Reactive → Resilient → Autonomous
An illustrative three-act arc — built on the ShipSolid platform maturity model, not any single real deployment — showing why the disciplined middle act is what actually earns the reliability, and why the autonomous act doesn't work without it.
8. AI Agent Use Cases
Surveys production-proven agent use cases — customer support triage, code review, incident investigation, and research synthesis — with the common architectural shape each one shares underneath the domain-specific framing.
3 — Cross-Signal Correlation
Metrics, logs, and traces are only more useful together than apart if something ties a specific instance of each of them back to the same event. The shared identifier that makes that jump possible — and what breaks when a hop in the call chain doesn't carry it.
Observability Engineering
A book-shaped table of contents for observability engineering: foundations through architecture, metrics, logging, tracing, profiling, OpenTelemetry, instrumentation, Kubernetes/cloud, data platforms, visualization, alerting, SRE integration, cost, security, platform engineering, AI-driven operations, and MAANG interview preparation — cross-linking existing prometheus/grafana-cloud/kubernetes/sre/platform-engineering notes instead of duplicating them.
Production Agent Systems
A book-shaped table of contents for Production Agent Systems: the runtime substrate, observability, reliability/security/governance, performance/cost engineering, and platform engineering underneath every agent in production. Book 4 of the AI Systems Engineering series.
System Design
Principal/Staff-level system design reference collection for MAANG interview preparation — observability pipelines, distributed systems, reliability engineering, and beyond.
Related notes
5 — Continuous Profiling
What makes always-on, sampling-based profiling cheap enough to run in production continuously, why it earns that cost mainly for hot or expensive services, and how a profile correlates back to the one trace that was running during the sample.
4 — Observability-Driven Development
The TDD analogy taken seriously: SLOs and instrumentation defined at design time as acceptance criteria, not retrofitted after an incident — and why this only sticks as a launch gate, not a guideline.
2 — Root Cause Analysis
Covers automated root-cause analysis as an investigation loop over existing telemetry, not a fixed trigger-action mapping.
3 — Anomaly Detection
Covers statistical and ML-based anomaly detection on time series, and its false-positive tradeoff against static thresholds.