In lesson 3.4.6 you cleared up an incident that you had built in yourself: you knew which switch it was, and pretended you didn’t. In this lesson, someone else knows. Someone from your circle (the examiners from here on) builds an incident into the Kladde, you get the alert, and you find out what happened, with the queries, alerts and the operations manual that you built in this module. At the end there are the cause, the queries that prove it, and a postmortem.
The drill is part of the finale of K3. It isn’t a trap: what matters is how you proceed, not whether you’re done in ten minutes. A cleanly documented path that ends in a reasoned assumption shows more than a lucky hit without evidence.
How the drill runs
| When | What happens |
|---|---|
| Beforehand | With a runbook, you give the examiners access to rg-kladde-test, and only there. The Kladde in test runs at the state of main, all alerts are resolved. |
| Some time on an agreed day | The examiners build in an incident. You don’t learn when or what. |
| Start | You get the first alert, or the examiners get in touch with a symptom, the way a user would report it: “Since noon today there have been no new tenders.” |
| 90 minutes | You investigate. You may do everything you could do in operations: portal, queries, Azure CLI, runbooks, manual, rolling back, a new delivery. You don’t ask the examiners about the cause. |
| Afterward | You write the postmortem and hand it in. The examiners revert their change, in case you haven’t done so already, and you revoke their access. |
What the examiners build in isn’t stated in this lesson. It can be a setting, a message, a file, a load; it can be noticed quickly or only after hours. It’s always something that can happen in real operations, and it can be found with what the Kladde has in telemetry and queries.