Notes / tag / incident-management

#incident-management

13 notes

1 — Incident Response Lifecycle

Detect, triage, mitigate, resolve, review — the phases every incident moves through, and why skipping the review phase is how the same incident happens twice.

sre incident-management book

10 — Blameless Postmortems

Why a blameless structure is what makes Five Whys produce an honest systemic answer instead of a defensive, cover-yourself one.

sre incident-management book

11 — Communication During Incidents

Status page updates, stakeholder comms, and internal channel discipline — the incident-adjacent work that determines how the outage is remembered as much as the fix does.

sre incident-management book

12 — Chaos Engineering

Deliberately injecting failure in a controlled experiment to validate the failure modes a design only claims to handle, before production finds them for you.

sre incident-management book

13 — Game Days

Scheduled, team-wide incident simulations that build on-call muscle memory and test the runbooks nobody's had to actually use yet.

sre incident-management book

2 — Severity Classification

The objective criteria that decide how big a response an incident gets, so severity is a judgment call made once, consistently, not renegotiated mid-incident.

sre incident-management book

3 — Incident Command System

The role structure — commander, comms lead, ops lead — that keeps a live incident from collapsing onto one overloaded engineer trying to do everything at once.

sre incident-management book

4 — On-call Engineering

Designing the rotation and escalation policy itself as an engineering problem, not just a schedule — the discipline distinct from the day-to-day on-call handbook.

sre incident-management book

5 — Escalation Policies

What happens when the first responder doesn't acknowledge in time, and the escalation chain that has to be correct precisely when everyone is least likely to check it.

sre incident-management book

6 — Runbooks

Step-by-step operational procedures written before the incident, so 3am execution doesn't depend on anyone's memory of how a system behaves under stress.

sre incident-management book

7 — Playbooks

Decision trees for ambiguous or novel incidents, one level up from a runbook's fixed steps — how to reason through a failure mode nobody's documented yet.

sre incident-management book

8 — Root Cause Analysis

Moving past 'a bad deploy caused it' to the systemic condition that let a bad deploy reach production undetected in the first place.

sre incident-management book

9 — Five Whys

The simplest RCA technique that works — repeatedly asking why until you hit a structural cause — and where it breaks down on genuinely multi-causal incidents.

sre incident-management book