In lesson 3.6.5 you rehearsed the recovery for the first time: from your machine, in dev, with a firewall rule for your own address
and the new sqlcmd. The runbook scripts/restore-drill.ps1 restored the database under a test name, counted rows and deleted the copy
again. This lesson rehearses the same thing where it counts: in the new subscription, against prod, from the pipeline, with a protocol
that a program checks. After that the drill runs by itself every month in test.
The issue "Recovery rehearsed" from 3.10.1 has four criteria: the database of prod is restored under a test name and counted, a deleted
file from tenders-raw is brought back, the duration is under the RTO of four hours or is explained, and the copy is deleted afterward.
What a drill measures
RTO means: how long may it take until the Kladde runs again? But the clock doesn't start at the command az sql db restore. It starts
when the damage happens. Break the time down into phases and write next to each one where its number comes from:
| Phase | What happens | Where the number comes from |
|---|---|---|
| Detect | from the damage to the email to "Kladde-Operations" | measured in the alarm drill, docs/alarm-probe.md (3.10.6) |
| Decide | determine the point in time (ops/queries/sql-dropped-tables.kql), choose the way: replace or copy back (3.6.5) |
not measurable without a real incident; an assumption, marked as such; the incident drill in 3.10.11 provides a reference point |
| Restore | Azure creates a new database from the backup | measured in this drill |
| Switch over | replace (rename, restart) or copy rows back | an assumption with the way from 3.6.5; described in the operations manual (3.10.10) |
| Check | count rows, the Kladde answers | measured in this drill |
This lesson measures the phases that can be measured without real damage. It doesn't rehearse the switch-over: a restored database can never be a free one again (3.6.5), and a Kladde that runs on a different database after the drill than its platform describes would itself be an incident. A number you haven't measured appears in the protocol explicitly as an assumption. That is more honest than a total time that pretends everything was measured.
The drill also says something about the RPO, the permitted data loss: it counts the rows in the original now and in the copy at the chosen point in time. The difference is exactly what a restore to this point in time would lose.