Avoiding downtime, the hotfix path, resilience of the deployment

The booking portal of Nordlicht Reisen GmbH, a fictional travel agency, is also in use in the morning, when travelers book. A restart with every release is not acceptable then. And when, on a Friday afternoon, a defect shows up that prevents bookings, the team needs a way that is faster than the usual one without leaving the rules of the release. Those are the skills of this lesson: minimizing downtime (load balancing, rolling update, slots and swap), a hotfix path and a resilience strategy for the deployment.

Prior knowledge: Rolling back revisions in Azure Container Apps is taught in K3, lesson 3.7.6; the strategies as a design, blue-green and weighted traffic are in lesson 4.8, checks and bypassing them in an emergency in lesson 4.7, branching models in lesson 2.1. This lesson takes that as known and asks: How does the application stay reachable during the release, what does the emergency path look like, and what belongs in the strategy so that a failed release stays small?

Not run against Azure, GitHub or Azure DevOps; from the reference of Microsoft and GitHub: All commands, YAML files and menu paths of this lesson come from the documentation or are derived from it; they were not run in this course.

Minimizing downtime

Microsoft sets the goal high: Teams should approach the deployment of new versions with the attitude of no downtime at all. Modern infrastructure and pipelines made that possible for almost every team. A second attitude follows from that: If nothing goes down, it doesn't matter when you release, and it is even an advantage to release during working hours, ideally early in the day and early in the week, because everyone is there and the blast radius of a defect stays controllable. There are three means for this.

Load balancing

A load balancer distributes requests across instances. For deployments, its health probe counts: If it fails for an instance, the load balancer sends no new connections there anymore. The instance can therefore be taken out of rotation, updated and put back. What Microsoft says about it:

• The probe decides which instances get new connections. Existing TCP connections to a failed instance keep running on a Standard Load Balancer until the application ends them, an idle limit kicks in or the VM is shut down. An update therefore interrupts them if the machine restarts.
• The probe port should reflect the state of the instance and the service. The application can control its answer itself: With a failing probe it relieves itself or prepares for maintenance, and the connections run out (draining).
• The administrative status (admin state) overrides the probe for an instance: Down takes it out of rotation regardless of the probe, Up keeps it in. That is meant for maintenance, that is, for deployments that you control yourself.
• The interval and threshold of the probe can be set.

A probe that is too strict is a risk, not a help: If the application answers too slowly at startup, the load balancer takes it out again immediately. As common causes of failed probes, Microsoft names an application that doesn't listen on the probe port, a network security group that blocks the probe's source address (168.63.129.16), and an application that answers more slowly than the probe's timeout.

Rolling update

In a rolling update, instances are replaced in groups, so that part of the instances always answers. Two ways in our toolset:

Deployment job with rolling (lesson 4.8): The batch is set with maxParallel (number or percent), preDeploy, deploy, routeTraffic and postRouteTraffic run per batch, followed by on: success or on: failure. According to Microsoft, the strategy applies only to resources of the type virtual machine; according to the description, the application stays reachable on the other machines while the batch is updated, and a health check after each batch can stop the release.

Virtual Machine Scale Sets have an upgrade policy with three modes:

Microsoft calls Rolling the safest way to bring instances to the latest model without sacrificing availability. Condition: The scale set needs a health signal, namely with uniform orchestration a probe of the load balancer or the Application Health extension, with flexible orchestration the extension. Some changes require a restart, a reimage or a redeployment: password or SSH key, image references, size changes and adding availability zones. That is the limit: Even Rolling doesn't prevent the restart of the individual instance, it only prevents all of them from restarting at the same time.

Slots and swap

A deployment slot is a live app with its own hostname in the same App Service plan. The prerequisite is the tier Standard, Premium or Isolated; the number of slots depends on the tier (five with Standard). As advantages, Microsoft names:

• You verify changes in the slot before you swap them into production.
• You warm up all instances of the slot before the swap; that removes downtime, and the redirection of traffic drops no requests.
• After the swap, the slot holds the previous production version; if something turns up, you swap back immediately.
• Slots cost nothing extra (the plan counts).

What happens during the swap (excerpt of the steps): App Service first applies the settings of the target (for example of the production slot) to all instances of the source: slot-specific settings and connection strings, settings of continuous deployment and of authentication. That restarts all instances of the source. It waits until all instances have finished the restart; if one fails, the swap reverts all changes to the source and aborts. Then the warm-up runs: Without further specification, App Service requests the root of the application (/), and an instance counts as warm as soon as it returns any HTTP response. Only when all instances of the source are warm do the two slots swap their routing rules. The whole preparation happens in the source; the target stays online. That is why production should always be the target.

What is swapped and what is not: Among other things, framework settings, app settings, connection strings, mounted storage, handler mappings, public certificates and WebJobs content are swapped. Among other things, the following are not swapped: protocol settings (HTTPS only, TLS version), publishing endpoints, custom domains, non-public certificates, scale settings, IP restrictions, Always On, diagnostic logs, CORS, managed identities and the integration into a virtual network. You can pin an app setting to the slot (Deployment slot setting) so that it doesn't move with the content; for values like the address of each environment's database, that is the normal case.

Controlling the warm-up: If the root isn't enough, specify a page of your own. Either in the Web.config with applicationInitialization, or with app settings: WEBSITE_SWAP_WARMUP_PING_PATH (path that starts with a slash, default /), WEBSITE_SWAP_WARMUP_PING_STATUSES (comma-separated status codes, for example 200,202; if the page returns another code, warm-up and swap stop; by default any code is valid) and WEBSITE_WARMUP_PATH (requested at every restart of the site, not only at the swap). The swap waits 90 seconds per request and tries up to five times; if all fail, it aborts.

Swap with preview (swap with preview, multi-phase): App Service runs the first step (settings of the target onto the source, restart) and stops. Now you verify the source with the settings of production. Then you complete the swap or cancel; when you cancel, App Service resets the settings of the source. That doesn't work if the site's authentication is switched on in one of the slots.

az webapp deployment slot swap --resource-group <gruppe> --name <app> --slot staging --target-slot production --action preview
az webapp deployment slot swap --resource-group <gruppe> --name <app> --slot staging --target-slot production --action swap
az webapp deployment slot swap --resource-group <gruppe> --name <app> --slot staging --target-slot production --action reset

The first line starts the preview, the second completes the swap, the third cancels. Without --action, the command swaps in one step. Swapping back means: swap the same two slots again immediately. Auto swap swaps by itself after every push into the slot as soon as it is warm; it is not available for web apps on Linux and Web App for Containers, and Microsoft advises trying it first on a non-production target. Important for the design: The old production instances are recycled in the last step; whatever runs there (long operations) aborts. Write the application to be fault tolerant.

In the pipeline

Azure Pipelines has tasks for both. Deploy into the slot, then swap:

- task: AzureWebApp@1
  inputs:
    azureSubscription: <Dienstverbindung>
    appType: webApp
    appName: <app>
    deployToSlotOrASE: true
    resourceGroupName: <gruppe>
    slotName: staging
    package: $(Pipeline.Workspace)/drop/*.zip

- task: AzureAppServiceManage@0
  inputs:
    azureSubscription: <Dienstverbindung>
    WebAppName: <app>
    ResourceGroupName: <gruppe>
    SourceSlot: staging
    SwapWithProduction: true

AzureAppServiceManage@0 also knows the actions for the swap with preview (Start Swap With Preview, Complete Swap, Cancel Swap) and for deleting a slot. In GitHub Actions you call the same az commands in run steps, after signing in with OpenID Connect (K3, lesson 3.7.5); the action azure/webapps-deploy has an input of its own for slots, whose name you look up in its documentation. In both worlds you put the swap in the stage that carries the approval of production (lesson 4.7), and the way back as a run of its own that can be started by hand.