So far, the Kladde has objectives, budgets, alerts, a procedure for incidents, postmortems and runbooks. Whether all of that holds shows itself at the first real incident, and then it is too late to improve things. Chaos engineering reverses the order: You cause an incident on purpose, at a time you choose, in an environment where nobody is harmed, and check whether the Kladde and the team behave the way you expect.
This is not breaking things out of curiosity. A chaos experiment is an experiment in the scientific sense: with a hypothesis that can turn out to be false, with a fixed scope and with rules for when it is aborted. Microsoft offers Azure Chaos Studio for this. This lesson shows what it can do for the Kladde and what it can't, and carries out a first experiment in the test subscription.
Planning an experiment
Every experiment of the Kladde begins with a page under docs/chaos/, before anything happens in Azure:
| Part | Question | Example |
|---|---|---|
| Steady state | How can you tell that the Kladde is running normally? | the SLIs from 3.15.1: booking above 99.5%, processing above 99% |
| Hypothesis | What do you expect when the disruption comes? As verifiable sentences | "The import fails with an exit code other than 0; the booking notices nothing." |
| Disruption | What exactly is injected, for how long? | tender-import accepts no message for ten minutes |
| Scope | What is affected, what explicitly isn't? | only the queue tender-import in test |
| Abort criteria | When does it end immediately? | a burn-rate alert of the booking; more than 15 minutes; dead letters in another entity |
| Observation | Which queries, which alerts, who is watching? | import-runs.kql, slo-booking-burn.kql, the alert emails |
| Way back | How does everything return to the initial state, even if the tool fails? | az servicebus queue update --status Active |
The hypothesis is the core. It must be able to fail. "The Kladde stays stable" can't fail; "the next run after the end catches up on all reports, without a duplicate" can. An experiment that refutes a hypothesis is a success: It found an error before a customer did.