Since module 3.4, the Kladde has had alerts: “error rate above 5% in 5 minutes,” “two of three locations can’t reach /health/live.” Those are good
alerts, but there’s a question they don’t answer: How reliable should the Kladde actually be? Is an error rate of 1% all day long acceptable?
And is it bad if the tender import fails for one night?
This module gives an answer to that which you can measure, and builds on it the craft of operations known as Site Reliability Engineering (SRE): targets that come from what users need, a budget for errors, an orderly process for incidents, postmortems, runbooks and experiments that cause outages on purpose. The final project (3.10) demands exactly that: alerts, a recovery, an operations manual and an incident drill with a postmortem.
| Lesson | What it produces | h |
|---|---|---|
| 3.15.1 SLIs and SLOs | docs/slo.md, four queries under ops/queries/, a workbook |
4 |
| 3.15.2 Error budgets | burn-rate alerts in Bicep, a rule for the budget that’s used up | 3 |
| 3.15.3 On-call and the course of an incident | the incident process in docs/operations.md, a tabletop exercise |
3 |
| 3.15.4 Blameless postmortems | the template from 3.4.6, extended, and a review | 3 |
| 3.15.5 Runbooks | runbooks/ in the repository, an executable runbook with a test |
3 |
| 3.15.6 Chaos engineering with Azure Chaos Studio | an experiment with a hypothesis and a result | 3 |
In this first lesson, you define what “reliable” means for the Kladde: two measures each for two flows, one target each, the queries for them in the repository, and a workbook that shows the state.
From the component to the user flow
The alerts from 3.4.5 look at components: the web app, the database, the availability test. Users don’t see components. A visitor wants to book an appointment; whether a database, a cache or three containers do the work makes no difference to her. In the morning, the office wants to see the new tenders; when and how the import ran overnight interests it only if they’re missing.
Microsoft’s Well-Architected Framework calls such a process a flow: a sequence of steps that achieves something for a user or for the business. There are user flows that a person starts and system flows that run without one. Recommendation RE:04 says: reliability targets belong to the important flows, not to individual resources. The Kladde has these flows:
| Flow | Kind | Who notices an outage? | How important |
|---|---|---|---|
Book an appointment (public page /appointment) |
user flow | the office’s customers, immediately | high: the office loses appointments and trust |
| Import tenders (job and worker) | system flow | the office, the next morning | high: tender deadlines are short |
| Maintain contacts (after sign-in) | user flow | the office, immediately | medium: it can wait an hour |
| Confirmation mail of a booking (outbox) | system flow | the customer, later | medium: the booking is already made |
The first two get targets in this lesson. For the others, what the alerts from 3.4.5 already see is enough for now. A rule of thumb in the Kladde: at most three measures per flow, better two. Every additional one dilutes attention, and in 3.15.2 an alert hangs on each.