It's 3 a.m., the email with severity 1 arrives, and the person on call hasn't looked at the Kladde in three weeks. What do they do now? The answer must not live in their memory but in a runbook: a short guide for exactly this alert that says what it means, how to check in two minutes whether it is real, which queries help, which actions are safe and when to get help.
Since lesson 3.4.5, the Kladde requires every alert to name its query. The burn-rate alerts from 3.15.2 also name a runbook that doesn't exist yet. This lesson writes the runbooks, one for every alert, and turns the step that would most often be typed by hand, and most error-prone, into a script.
What belongs in a runbook
A runbook is not a manual. It is read under pressure, often at night, and must lead to the first action within five minutes. That is why every runbook of the Kladde has the same sections in the same order:
| Section | Question | Example |
|---|---|---|
| Meaning | What is broken, for whom, how urgent? | "Customers can't book; the budget is burning 14 times as fast as planned." |
| First check | Is the alert real? Two minutes at most | a query or a call of the page |
| Queries | What do I narrow it down with? | files under ops/queries/, never queries in the text |
| Safe actions | What may I do without asking, in what order? | roll back, turn off a switch, restart a job |
| Do not | What looks obvious and does damage? | "Don't empty the queue." |
| Escalation | When do I get whom? | "SEV 1 not mitigated after 30 minutes: call the deputy" |
Safe here means three things: The action can be undone, it changes and deletes no data, and it grants no permissions. Rolling back to a previous revision is safe; restoring the database to an earlier point in time is not. What isn't safe goes in the runbook as an escalation, not as an action.
Runbooks live as Markdown under runbooks/ in the repository, next to the code they concern. That means what applies to code applies to them: Changes
go through a pull request with a review, and whoever changes an alert sees in the same pull request whether the runbook still fits.