Nordlicht Reisen GmbH, a fictional travel agency, now has six repositories, and almost the same pipeline sits in all of them: install the SDK, restore, build, test, publish the result. Every improvement has to be made six times, and two repositories have already drifted apart. You are to design one central building block that everyone uses, that can be versioned, and from which nobody can leave out the company's basic rules. That is the skill of this lesson: designing reusable pipeline elements, namely YAML templates, task groups, variables and variable groups, and their counterparts on GitHub.

Prior knowledge: A simple template with typed parameters for steps and the variable group with Key Vault are shown in K3, lesson 3.7.2. Here the lesson starts with what goes beyond that: extends and inserting jobs and stages, templates from other repositories, the precedence rules of variables, task groups and the reusable workflows of GitHub. The security side of extends (required templates, approvals) is covered in more depth by module D5.

Not run against Azure or GitHub; from the reference of Microsoft and GitHub: All YAML files and commands in this lesson come from the documentation or are derived from it; they were not run in this course.

Templates in Azure Pipelines

Microsoft distinguishes two kinds:

  • Includes templates insert content, like an include statement in a programming language. The content of the template then sits in the pipeline or template that includes it.
  • Extends templates define logic and structure that a pipeline must follow. This serves to enforce security, compliance and company standards.

Inserting: steps, jobs, stages, variables

The same template syntax - template: <path> stands at the place where the content is to appear. A template for steps begins with steps:, for jobs with jobs:, for stages with stages: and for variables with variables:.

YAML
# File: templates/insert-npm-steps.yml
steps:
  - script: npm install
  - script: npm test
YAML
# File: azure-pipelines.yml
jobs:
  - job: Linux
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      - template: templates/insert-npm-steps.yml
  - job: Windows
    pool:
      vmImage: 'windows-latest'
    steps:
      - script: echo This script runs before the template's steps, only on Windows.
      - template: templates/insert-npm-steps.yml
      - script: echo This step runs after the template's steps.

More steps before and after the template are allowed. For jobs, a rule from Microsoft applies: if you insert several jobs with the same template, leave out the name in the template's job (- job: without a name) or make it a parameter, so that the names do not collide. A template for variables may only define variables; for steps and logic, use extends. Variable templates also take parameters:

YAML
# File: templates/package-release-with-params.yml
parameters:
  - name: DIRECTORY
    type: string
    default: "."

variables:
  - name: RELEASE_COMMAND
    value: echo release from ${{ parameters.DIRECTORY }}

Microsoft advises using parameters instead of variables when types and security matter (for example for arguments of shell tasks).

Extending: enforcing a fixed shape

With extends, the pipeline calls a template that determines the framework, and the pipeline fills in only the gaps provided. In Microsoft's example, buildSteps is a parameter of type stepList; the template inserts its own steps before and after and rejects steps of a certain type:

YAML
# File: start-extends-template.yml
parameters:
  - name: buildSteps
    type: stepList
    default: []
stages:
  - stage: secure_buildstage
    pool:
      vmImage: windows-latest
    jobs:
      - job: secure_buildjob
        steps:
          - script: echo This happens before code
            displayName: 'Base: Pre-build'
          - ${{ each step in parameters.buildSteps }}:
              - ${{ each pair in step }}:
                  ${{ if ne(pair.value, 'CmdLine@2') }}:
                    ${{ pair.key }}: ${{ pair.value }}
                  ${{ if eq(pair.value, 'CmdLine@2') }}:
                    '${{ pair.value }}': error
          - script: echo This happens after code
            displayName: 'Base: Signing'
YAML
# File: azure-pipelines.yml
trigger:
  - main

extends:
  template: start-extends-template.yml
  parameters:
    buildSteps:
      - bash: echo Test #Passes
        displayName: succeed
      - task: CmdLine@2
        inputs:
          script: echo "Script Test"

The step with CmdLine@2 is rejected with a YAML syntax error ("Unexpected value 'CmdLine@2'"); the step with bash passes. Whoever enforces the shape this way adds a required template approval so that nobody runs the pipeline without extends: module D5 covers this in more depth; Microsoft describes it on the page about security with templates (security/templates).

Parameters and data types

Parameters have a name and a data type; default values may only be literals. Microsoft lists the types: string, stringList (only outside templates), number, boolean, object, step, stepList, job, jobList, deployment, deploymentList, stage and stageList. With values: you restrict the choice:

YAML
parameters:
  - name: myNumber
    type: number
    default: 2
    values:
      - 1
      - 2
      - 4
  - name: mySteplist
    type: stepList
    default:
      - script: echo step one

A trap that Microsoft describes explicitly: a parameter without a type is treated as a string, and non-empty strings are true in a Boolean context. eq(true, parameters['myparam']) is therefore true even when the parameter contains the word false. So specify the type boolean, or compare strings explicitly.

With ${{ if … }} you include elements conditionally, with ${{ each … }} you iterate over lists and objects. templateContext attaches additional properties to jobList, deploymentList and stageList, which the template reads while iterating.

Templates from other repositories

Central building blocks live in a repository of their own. The pipeline names it under resources and refers to the name with @:

YAML
resources:
  repositories:
    - repository: templates
      type: github
      name: Contoso/BuildTemplates
      ref: refs/tags/v1.0

jobs:
  - template: common.yml@templates
    parameters:
      vmImage: 'windows-latest'

According to Microsoft, a prerequisite on GitHub is a GitHub service connection. The repository is resolved once at the start; after that only the content of the templates is used, and the fully expanded pipeline runs as if it stood entirely in the source repository. For that reason the pipeline cannot use scripts from the template repository, only its templates.

Pinning to a version: Without ref, refs/heads/main applies. A ref is a branch (refs/heads/<name>) or a tag (refs/tags/<name>); if you want a specific commit, first put a tag on it or give the 40-character SHA. With @self, a template refers to the repository of the original pipeline.

Limits: Microsoft sets limits so that templates do not grow out of control: at most 100 included YAML files (directly or indirectly), at most 100 levels of nesting and at most 20 megabytes of memory when reading. Templates must be in the file system when the run starts; you cannot include any from an artifact. And, from lesson 4.1: triggers belong in the main file, not in a template; a pipeline completion trigger is not supported in templates.