The booking portal of Nordlicht Reisen GmbH, a fictional travel agency, is growing. What was a build with tests has become a chain: build, test on two operating systems, deliver to the development environment, deliver to production, and then a report that always appears, even when something went wrong. You are to write the pipeline so that the order is right, what may run in parallel really runs in parallel, and a red stage does not lead to a delivery. That is the skill of this lesson: developing pipelines in YAML, with the order of jobs, parallelism and multi-stage pipelines.

Prior knowledge: You know the structure with trigger, pool, stages, jobs and steps from K3, lesson 3.7.2, that of a workflow from K3, lesson 3.7.1, and when a run starts from lesson 4.1. This lesson is about what happens between the stages and jobs.

Not run against Azure or GitHub; from the reference of Microsoft and GitHub: All YAML files in this lesson come from the documentation or are derived from it; they were not run in this course.

Azure Pipelines: stages, jobs and steps

Microsoft describes three levels. A step is the smallest unit (script, bash, pwsh, task). A job is a sequence of steps that runs as a unit on one agent, and the smallest unit that can be scheduled. A stage is a logical boundary at which you can pause the pipeline and carry out checks, and it contains jobs; a stage may have up to 256 jobs. Every pipeline has at least one stage, even if you do not write any.

Stages and their order

Stages run one after another by default, in the order of the file. With dependsOn you build a graph. At least one stage must have no dependency.

YAML
stages:
  - stage: Test

  - stage: DeployUS1
    dependsOn: Test

  - stage: DeployUS2
    dependsOn: Test

  - stage: DeployEurope
    dependsOn:
      - DeployUS1
      - DeployUS2

DeployUS1 and DeployUS2 run in parallel after Test (fanning out, fan-out), and DeployEurope waits for both (merging, fan-in). With dependsOn: [], a stage detaches itself from the one before it and runs in parallel with it. A pool at stage level applies to all jobs of the stage, unless a job names its own.

Two keywords are useful for delivery: trigger: manual on a stage, so that it does not start automatically but is started by hand, and isSkippable: false, so that nobody can skip a stage, for example the malware check that a template inserts into every pipeline. And a limit that reviews like to overlook: YAML pipelines have no queuing policies. Every run is independent of the others; two commits start two runs that go through the same sequence of stages without regard for each other. Microsoft recommends controlling the order, when it matters, with manual approvals (lesson 4.7).

Jobs in a stage

Several jobs of a stage run in parallel by default, as long as there is no dependsOn. But: Every agent runs only one job at a time. For real parallelism you need several agents and enough parallel jobs in the organization (lesson 4.2). If you write three jobs without a dependency and have one parallel job, you will see them run one after another.

YAML
jobs:
  - job: InitialA
    steps:
      - script: echo hello from initial A
  - job: InitialB
    steps:
      - script: echo hello from initial B
  - job: Subsequent
    dependsOn:
      - InitialA
      - InitialB
    steps:
      - script: echo hello from subsequent

The complete structure of a job shows what you control here: dependsOn, condition, strategy (with parallel or matrix; both together is an error, and maxParallel applies only with matrix), continueOnError, pool, workspace, container, timeoutInMinutes, cancelTimeoutInMinutes, variables, steps and services. Added to that are the deployment job (deployment: instead of job:), which lessons 4.7 and 4.8 cover, and the server job (pool: server), which needs no agent and knows only a few tasks such as Delay, InvokeRESTAPI and ManualValidation.

Time limits: timeoutInMinutes limits the run time of a job. Without a value, 60 minutes apply; 0 means the largest possible limit, which is unlimited on self-hosted agents, 360 minutes on hosted agents in public projects and 60 minutes on hosted agents in private projects without additionally purchased capacity. A value above the limit of the hosted agent has no effect. The time before the start (waiting for an agent) does not count.

Working directory: On self-hosted agents, the working directory is kept between runs (except for the folder for artifacts and test results); with workspace: clean: outputs | resources | all you clean up. On hosted agents, every job is fresh. Whether the next job runs on the same agent is certain only if the pool has exactly one suitable agent.