The booking portal of Nordlicht Reisen GmbH, a fictional travel agency, is changed often, and every change should only continue if certain conditions are met: the tests green, the coverage of the changed lines sufficient, the response time under a limit, no alerts after the rollout. Nobody should tick boxes by hand for that. You are to design the gates, set the test strategy behind them and decide where tests, coverage and load tests sit in the chain. Those are the skills of this lesson: quality and release gates (also for security and governance), test strategy (local, unit, integration, load), tests and test agents in a pipeline and code coverage.
Prior knowledge: How you make test results and line coverage visible in a pipeline and why the exit code matters is in K3, lesson 3.7.3; load tests with Azure Load Testing, failure criteria and the cost in virtual user hours are in K3, lesson 3.8.6; approvals, environments and rolling back are in K3, lesson 3.7.6. You know branch policies and protection rules from lesson 2.2, and the parallelism of jobs from lesson 4.3. This lesson doesn't repeat them, but arranges them into a strategy.
Not run against Azure, GitHub or Azure DevOps; from the reference of Microsoft and GitHub: All commands, YAML files and menu paths of this lesson come from the documentation or are derived from it; they were not run in this course.
Quality and release gates
What a gate is
A gate is a rule that decides, without a person, whether a change may continue to the next step. An approval, by contrast, is a decision by a person (lesson 4.7). The two belong together, but gates are the part that turns out the same for every change without anyone lifting a finger. Among other use cases for gates, Microsoft names:
- Quality check: Query pipeline metrics such as the pass rate of tests or the code coverage and only deploy if they are within the given range.
- Security check: Examine artifacts, sign code, check policies; a gate can start the scan and wait for its end, or only check whether it has finished.
- Incident management: Only deploy if no bugs of the highest severity are open.
- Change management: Wait for a procedure in a system such as ServiceNow.
- State of the infrastructure: After the rollout, check whether the resources are healthy and comply with the rules.
The thought behind it is important: Most health values fluctuate between healthy and unhealthy. That is why gates are evaluated repeatedly until all of them pass in the same round, or the time runs out and the deployment is rejected.
Where gates sit in the chain
A chain has several places where a gate makes sense. The order is also an order from cheap to expensive:
| Place | Example of a gate | Azure DevOps | GitHub |
|---|---|---|---|
| before the merge | tests green, coverage of the changed lines, analyses | branch policy with build validation, status check | required status checks in a ruleset (lesson 2.2) |
| in the build | tests run, minimum number of tests, coverage is published | step fails, failOnMinTestsNotRun |
step fails, exit code |
| before the stage | branch protected, template required, artifact meets policy, check service says yes | checks on the environment or service connection | protection rules of the environment, custom protection rules |
| after the rollout | no alerts, load test passed, health signals | check "Query Azure Monitor alerts", call of a function or a REST service | follow-up job with a check, rollback on failure |
Important for the exam: The gates of the classic release pipelines and the checks of the YAML pipelines are two versions of the same idea. Microsoft describes gates as part of release pipelines (pre- and postconditions of a stage, with a delay before evaluation, a time between re-evaluations and a timeout; the smallest values you can set are a six-minute timeout and a five-minute interval) and checks as part of resources such as environments. The signals are the same: alerts in Azure Monitor, queries of work items, calls of an Azure Function or a REST service. The task AzureMonitor@1 states explicitly that it runs in a release pipeline job without an agent and succeeds if no alert rule is fired at the time of the query.
Security and governance as gates
"Governance" here means: Whoever writes a pipeline shouldn't be able to bypass the organization's rules. For that, Azure Pipelines has checks that are not in the YAML file and therefore can't be changed by its authors (lesson 4.7 shows all of them):
- Branch control: The sources of the run must come from allowed branches, with protection switched on if you wish.
- Required template: A pipeline fails if it doesn't extend the required template. This is the lever to force security steps into every pipeline (lesson 4.4).
- Evaluate artifact: The artifact is checked against your own policies; Microsoft names as a restriction that this currently applies only to container images.
- Invoke REST API / Azure Function: This is where you plug in services that Microsoft doesn't ship. To the question of how to wait for the end of a security scan, Microsoft answers: with an external scan service and a REST check that queries its interface. The check itself scans nothing.
With GitHub, the governance sits in the settings of the repository and the organization: rulesets for branches (lesson 2.2), protection rules of the environment (lesson 4.7) and, as a public preview according to GitHub, custom protection rules as a GitHub App that answers the webhook deployment_protection_rule.
Two design rules from Microsoft's guide to safe deployments: Quality signals must be watched daily, otherwise a team writes tests only to turn a yellow sign green. And before the deployment, checks such as code review, security scans and compliance checks belong to the "predeployment checks"; you don't rely on one kind of test, because different tests find different classes of errors.