At the end of the month, the head of engineering at Nordlicht Reisen GmbH, a fictional travel agency, asks: "How much faster have we become?" The team "Portal" answers with a feeling. That is not enough. A metric is only worth something if you know what it measures, where the data comes from and what it does not tell you. In this lesson you design the measurement for the booking portal: which metrics, on which dashboard, from which queries.
These are the two skills of the exam: designing dashboards (lead time, cycle time, time to recovery) and metrics and queries for planning, development, testing, security, delivery and operations.
Prior knowledge: Processes, work items and traceability are in lesson 1.1, the connection to GitHub in lesson 1.2. K3, module 3.4, teaches telemetry, KQL and alerts for the application itself; monitoring of the delivery chain comes in lesson 6.1. This lesson measures the work flow and the delivery, not the application in operation.
Note: All statements come from the documentation of Microsoft Learn and GitHub; the pages are listed under "Further reading". The commands, queries and menu paths of this lesson were not run; they come from the reference of Microsoft and GitHub.
What to measure: the terms
Azure DevOps offers two time measures as widgets. The page "Configure lead time and cycle time widgets" defines them:
| Metric | Definition | What it is good for |
|---|---|---|
| Lead time | the elapsed time from the creation of a work item until it reaches the state Completed | estimating delivery times and tracking agreements on performance |
| Cycle time | the elapsed time from the first entry into the state In Progress until Completed | finding bottlenecks in the process and measuring the efficiency of the team |
Remember the difference: lead time starts when the request arises, cycle time when the team starts. The time in between is waiting time in the backlog. There is a relationship between the two that Microsoft names in the guidance on the cumulative flow diagram: more work in progress (WIP) leads to longer cycle and lead times, less to shorter ones. That is why the page recommends WIP limits on the board.
The mapping of the states ("In Progress", "Completed") to the columns of your board works through state categories (page "About workflow states in backlogs and boards"). Whoever changes the states of a custom process also changes what these widgets measure.
The metrics of delivery performance (DORA)
In its recommendations for DevOps, Microsoft names the four metrics of DevOps Research and Assessment (DORA) for the performance of software delivery:
| Metric | Meaning (according to Microsoft) |
|---|---|
| Lead Time for Change | the time a change (bug fix, feature or anything else) takes from the idea to deployment in production |
| Deployment Frequency | how often per day code changes go into production |
| Mean Time to Restore (time to recovery) | the time it takes to restore the service after an incident in production |
| Change Fail Percentage | the share of changes to production that lead to a failure |
On time to recovery, the page "What is DevOps?" says: the "mean time to recovery" shows how long it takes to recover from a failure or an attack; teams should measure and improve it.
Important for the exam and for your design: The Microsoft pages read for this lesson name no ready-made widget for time to recovery. The widgets for lead time and cycle time measure work items. Whoever wants to measure recovery time has to derive it from data they define themselves. This is a design of the course, not a feature set according to Microsoft: create incidents as work items of a custom type (or as bugs with a tag), record the start of the incident when you create the item, close the item when the service is running again, and use the lead time widgets with a filter on this type. Then the lead time of these items is an approximation of the time to recovery. The limit: it is only as accurate as the discipline in creating and closing.