Up to here, the Kladde has learned to tell its story: logs, traces, metrics, queries, alerts. In this lesson you use all of it for the moment it was built for: something is wrong, and you have to find out what. There is a procedure that is the same for every incident, and a document that ends every incident, the postmortem.

You practice on a prepared incident: a switch makes one of the Kladde's dependencies slow. Best if a second person sets it without telling you which one; if you learn alone, set it yourself and then proceed as if you didn't know.

The procedure

Anyone searching under pressure likes to jump to the first hunch and then look for evidence for it. Fixed steps protect against that:

  1. Pin down the symptom. What do users see? Since when? Whom does it affect, everyone or some? A sentence like “Since this morning, no new tenders arrive” is a good start, “the Kladde is acting up” is not.
  2. Find the time. When exactly did it begin? A curve over the last few hours shows the kink.
  3. Narrow it down. Which requests, which dependency, which revision? From the overview to a single example.
  4. Ask what has changed. Almost every incident follows a change: a new revision, a changed setting, more data, a service the Kladde depends on. Lay the time from step 2 next to the list of changes.
  5. Limit the damage. Before you understand the cause, make sure it does no more harm: roll back to the previous revision (lesson 3.7.6), turn a switch off, temporarily disable a feature.
  6. Prove the cause and fix it. With queries that show it was so, not just with a story that fits.
  7. Write the postmortem. What happened, and what do we change so that it doesn't happen again or is noticed sooner?

Take notes while you search: the time, what you checked, what came out. These notes later become the timeline of the postmortem. All times in UTC, because that is how they appear in the tables.