Complex scenarios: hybrid, VM templates, self-hosted runners and agents in operation

Nordlicht Reisen GmbH, a fictional travel agency, runs a payment database in its own data center; it cannot be reached from the internet, and database changes are rolled out to two servers in the data center. The application itself is built and tested in Azure. The build machines need a tool that is missing on the hosted machines, and the team no longer wants to set the machines up by hand. You are to design how it all works together: a hybrid pipeline, a self-built VM image and the operation of the agents and runners. These are the skills of this lesson: complex pipeline scenarios, namely hybrid pipelines, VM templates and self-hosted runners and agents.

Prior knowledge: Lesson 4.2 compared the kinds of agents and runners, their costs and limits, lesson 4.7 covers environments and checks in YAML, and lesson 4.8 the deployment strategies. Why foreign code does not belong on a self-hosted machine is in K3, lesson 3.7.7. This lesson shows how you put the means together and run them.

Not run against Azure or GitHub; from the reference of Microsoft and GitHub: All commands and YAML files of this lesson come from the documentation and were not run in this course.

Hybrid pipelines

Hybrid here means: one part of the work runs in the cloud, another part in a network that the cloud cannot reach. The question is always the same: Who sets up the connection, and from where?

The principle: the agent fetches

According to Microsoft, the agent always communicates with Azure Pipelines itself: it polls the job queue with an HTTP long poll, downloads a job with a short-lived token limited to that job, runs it and reports logs and result back. All messages go over HTTP or HTTPS, and the payload is encrypted with an asymmetric key pair of the agent that is exchanged at registration. This pull model allows it to run behind a firewall without inbound connections. On GitHub the same holds: the runner sets up outbound HTTPS connections over port 443 (lesson 4.2).

For a deployment the agent also needs "line of sight" to the targets. Microsoft says: if on-premises environments have no connection to the hosted pool, which is the normal case because of firewalls, you must set up self-hosted agents on machines in the company network that can reach the target and the internet. Hosted agents can not be bound to the company network through ExpressRoute or VPN.

Three ways to the target in the company network

1. A self-hosted agent or runner in the company network. It runs the deployment job; building continues in the hosted pool, and the artifact travels through Azure Pipelines (tasks PublishPipelineArtifact@1 and DownloadPipelineArtifact@2) to the agent.
2. A pool in your own network. Scale Set agents sit in your VNet; for Managed DevOps Pools you delegate a subnet of your virtual network to the service (lesson 4.2). On GitHub, larger runners with Azure Private Networking are the counterpart, and ARC for Kubernetes.
3. Environment VMs. An agent runs on the target machine and is registered as a resource of an environment. The deployment job runs right there.

The third way in Azure Pipelines: under Pipelines, Environments you create an environment, choose Virtual machines as the resource and copy the registration script (it contains a PAT of the signed-in user, which expires three hours after it is generated and serves only to install the agent). You run the script on every target machine, Linux or Windows; the machines appear under Resources of the environment. You select machines with tags:

jobs:
  - deployment: VMDeploy
    displayName: Deploy to VM
    environment:
      name: VMenv
      resourceType: virtualMachine
      tags: windows,prod
    strategy:
      runOnce:
        deploy:
          steps:
            - script: echo "Hello world"

Several tags mean: the job runs only on machines that carry all of them. A deployment job has the lifecycle preDeploy, deploy, routeTraffic, postRouteTraffic and then on: success or on: failure; with environment VMs, preDeploy runs on the build agent, everything else on the registered VMs. The machines must therefore have the tools that the steps need (for example the Azure CLI). runOnce runs the steps once per VM, without parallelism; the rolling strategy works with groups of up to five target machines, depending on maxParallel, which also determines the minimum number of available machines (lessons 4.8 and 4.9). To remove a VM from the environment, you run ./config.sh remove (Linux) or ./config.cmd remove from an administrator PowerShell (Windows) on the machine.

Deployment groups exist only for classic release pipelines; for YAML you use environments with VM resources.

Both worlds, one pattern