Nordlicht Reisen GmbH, a fictional travel agency, has accumulated many pipelines over the years. Some run every night although nothing has changed, nobody understands an old release pipeline with approvals anymore, and the storage holding old runs keeps growing. You are to plan the maintenance of the pipelines: measure where things stall, reduce cost and run time, decide what is retained and for how long, and migrate the classic pipelines to YAML. Those are the skills of this lesson: monitor pipeline health, optimize cost, time, performance, reliability and concurrency, define retention and migrate classic pipelines to YAML.
Prior knowledge: The four health numbers (failure rate, time to verdict, second attempts, wait time), the script pipeline-health.ps1, cache and parallel jobs in GitHub, and retention in GitHub and Azure Pipelines in outline are in K3, lesson 3.7.7. Lesson 4.2 compares runners, agents and parallel jobs, lesson 4.7 environments with checks, lesson 6.1 alerts for pipelines. This lesson starts where K3 ends: with many pipelines, an organization and the legacy burden.
Not run against Azure DevOps; from Microsoft's reference: All commands, YAML files and menu paths of this lesson come from Microsoft's documentation and were not run in this course. GitHub's page on retention and metrics was not read again for this lesson; for GitHub, K3, lesson 3.7.7, applies.
Monitoring health
The reports in Azure Pipelines
Every pipeline has an Analytics tab. It summarizes success rate and duration; a click on the card opens the full report. The source of the reports is the pipeline's runs, collected over a period. Three reports are available:
| Report | What it shows | What you use it for |
|---|---|---|
| Pipeline pass rate | success rate and its trend; summary (default period 14 days), Failure trend (failures per day, separated by stage if there are several stages), Top failing tasks & their failed runs | find the task that makes the most runs fail, and fix that one first |
| Pipeline duration | how long the pipeline usually takes to succeed; trend; the tasks with the longest duration | find the task with the largest share of the run time |
| Test failures | the most frequently failing tests with details | find tests that turn runs red |
You can filter by date range (default: the last 14 days) and by branch. The branch filter matters: A pipeline that is often red for pull request branches but stable on main has a different problem from one that is red on main.
From the number to the measure
A number is useful only when a decision hangs on it. A table that connects health and maintenance (the mapping is ours, the reports are Microsoft's):
| Finding | Cause probably lies with | Measure |
|---|---|---|
| one task heads the list of failures | the task itself or its environment (agent, tool, network) | fix the task; if it fails only on one agent, check the agent (lesson 4.5) |
| the duration has been rising slowly for weeks | growing scope (tests, dependencies, history) | cache, shallow fetch, splitting (lesson 4.3) |
| the duration jumps suddenly | a change to the pipeline, image or agent | compare the run before and after the jump |
| the same tests fail alternately | flaky tests | repair instead of restarting (K3, lesson 3.8.4) |
the success rate is low on branches, high on main |
pull requests without local checks | protection rules and local checks (lesson 2.2) |
Alerts when a pipeline turns red are covered in lesson 6.1; here it is about looking regularly: once a week at the report of the most important pipelines.