Notes / tag / alerting

#alerting

13 notes across 2 topics

1 — Dashboard Design

The three-question test for a vanity panel, why the same underlying data needs a different dashboard for different audiences, and the top-down layout that mirrors how an investigation actually drills down.

observability slo alerting incident-response book
Jul 17, 2026

1 — Alerting & Alert Routing

Symptom-based vs. cause-based alerting, the noise-reduction problem (dedup, grouping, correlation), and routing/escalation — the design discipline standing between a real page and a bad one.

observability slo alerting incident-response book
Jul 17, 2026

2 — SLOs & Error Budgets

SLI/SLO/SLA, the error budget as a spendable resource rather than a compliance number, burn rate as the mechanism connecting the two, and why multi-window multi-burn-rate alerting exists at all.

observability slo alerting incident-response book
Jul 17, 2026

2 — Symptoms vs Causes

Covers distinguishing 'users are affected' alerts from 'a specific subsystem misbehaved' alerts, and why only the former should page.

observability alerting book

3 — Slo Based Alerts

Covers deriving alert thresholds from an SLO's error budget rather than from arbitrary static thresholds.

observability alerting book

4 — Multi Window Burn Rate Alerts

Covers the multi-window, multi-burn-rate alerting technique that balances fast detection against alert noise.

observability alerting book

5 — Alert Deduplication

Covers grouping and suppressing duplicate alerts from the same root cause so on-call sees one page, not fifty.

observability alerting book

6 — Routing

Covers alert routing rules — team ownership, severity, and escalation paths — as configuration distinct from the alert condition itself.

observability alerting book

7 — Alert Fatigue

Covers diagnosing and reversing an alert-fatigue trend before it causes a real page to get ignored.

observability alerting book

8 — On Call Engineering

Covers structuring on-call rotations, handoffs, and runbook discipline as an engineering practice, not just a schedule.

observability alerting book