Proxies and load balancers

Between the browser and the Kladde in Azure there are at least one and often two programs that accept requests and pass them on: the Container Apps ingress and, as soon as you need more than one region or a firewall for web applications, Front Door. They spread load, end TLS, check which instance is healthy and cut off requests that take too long. Each of them changes what the Kladde knows about a request. In this lesson you put two instances behind a reverse proxy of your own, stop one and watch the proxy take it out. And you learn why a header like X-Forwarded-For sometimes tells the truth and sometimes doesn’t.

Forward proxy and reverse proxy

A proxy accepts a request and passes it on by itself. There are two kinds, depending on whose side it stands on:

• A forward proxy stands next to the client. A company network, for example, sends all requests from its computers to the internet through a proxy, to filter or log them. The servers out there only see the proxy. .NET respects such a proxy through the environment variables HTTPS_PROXY and NO_PROXY; you’ll meet one in the build agents of companies.
• A reverse proxy stands next to the server. It accepts requests from the internet and spreads them over one or more instances behind it. The client only sees the proxy. That is the Kladde’s case: no browser ever talks to the container directly.

A reverse proxy that spreads requests over several identical instances is also called a load balancer.

Layer 4 and layer 7

Load balancers differ in how deeply they look into a request:

• Layer 4 (transport): The balancer only sees addresses and ports and passes on whole TCP or UDP connections. It knows nothing about HTTP, so it can’t balance by path or host, but it can pass on any protocol, SQL and SSH included.
• Layer 7 (application): The balancer speaks HTTP. It ends TLS, reads host, path and headers, balances accordingly, adds headers, can cache responses and can turn requests away.

For the Kladde that means: the Container Apps ingress is always there, you already used it in K2. Front Door joins in module 3.3, in outline and only as an option, because it has a base fee (price calculator). You don’t need Load Balancer or Application Gateway for the Kladde; they belong to AZ-104 and K4.

What a load balancer decides

Where to? The simplest methods are round robin (one after the other) and random. Better ones look at the load: the fewest open requests, or pick two at random and take the less busy one (power of two choices, YARP’s default method).

Who is healthy? A balancer that sends requests to a crashed instance turns a half outage into a full one. That’s why it checks:

• actively: At fixed intervals it calls a path such as /health on each instance. Whoever fails to answer, or fails to answer with 2xx, several times in a row is taken out until it answers again.
• passively: It watches the real requests. If errors pile up on one instance, it takes that instance out for a while.

Time passes between the failure and the next active check, and during that time requests land on the dead instance. How much time, you’ll see in the example in a moment.

Does a user stay with the same instance? That’s called session affinity (sticky sessions): The balancer sets a cookie and sends all further requests from the same browser to the same instance. The Kladde needs this as soon as it has more than one replica, because its interactive Blazor islands hold a long-lived connection to one particular instance. Container Apps offers affinity (in single revision mode). Better is what gets by without it: state in the database instead of in the memory of one instance.

How long does it wait? Every proxy has timeouts:

The rule that follows: inside shorter than outside. If a request to the Kladde takes 90 seconds, Front Door answers the browser with an error after 60 seconds while the Kladde keeps working. The user clicks again, and the same work runs twice. Long work therefore doesn’t belong in a request but in a background service that accepts it and reports back later (module 3.5).

TLS at the proxy

A proxy at layer 7 has to read HTTP, so it has to end TLS: It holds the certificate from lesson 3.0.4 and the private key. Behind it, it talks to the instance unencrypted or with new TLS. With Container Apps, the ingress speaks HTTP to the Kladde on port 8080. That’s why the Kladde doesn’t know on its own that the browser used HTTPS. If it asks Request.Scheme, it gets http, builds wrong addresses for redirects and sign-in callbacks with it, and in the worst case redirects every request to HTTPS that was already HTTPS, in a circle.

X-Forwarded-*: what the instance learns about the request

So that the instance still knows what the browser did, the proxy writes it into headers:

The standard for this is actually called Forwarded (RFC 7239), but the X- variants are more widespread. ASP.NET Core reads them with the Forwarded Headers middleware, which then sets RemoteIpAddress and Request.Scheme as if the client had asked directly.

Now the most important sentence of this lesson: A header is only as trustworthy as whoever wrote it. Any client can write X-Forwarded-For: 203.0.113.66 into its request. What happens then depends on the proxy:

• YARP replaces such a header with the client’s real address by default.
• The Container Apps ingress appends the real address. The Kladde gets 203.0.113.66, <real address>, and only the right-hand entry comes from the platform. Everything to the left of it was written by the client.
• No proxy at all, because the instance is also reachable directly: Then the header arrives exactly as the client wrote it.

Three rules follow, which lesson 3.3.4 puts into practice for the Kladde:

1. The instance only believes X-Forwarded-* if the connection comes from a known proxy (KnownProxies, KnownIPNetworks).
2. From X-Forwarded-For it takes only as many entries from the right as it has proxies of its own (ForwardLimit, default 1).
3. The instance is reachable only through the proxy. Otherwise anyone gets around the first rule by coming directly.

This is what the middleware looks like in Program.cs; the address is a placeholder for your proxy’s:

// Program.cs, at the top: using System.Net; using Microsoft.AspNetCore.HttpOverrides;
builder.Services.Configure<ForwardedHeadersOptions>(options =>
{
    options.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
    options.KnownProxies.Add(IPAddress.Parse("10.20.0.10"));
});

var app = builder.Build();
app.UseForwardedHeaders(); // before every other middleware

A shortcut you’ll find in guides: the environment variable ASPNETCORE_FORWARDEDHEADERS_ENABLED=true. It switches the middleware on and empties the lists of known proxies, so it believes everyone. That’s convenient and exactly as unsafe as it sounds; in 3.3.4 you’ll see when it is still defensible and what else is needed then.

YARP: a reverse proxy in .NET

YARP (Yet Another Reverse Proxy) is a library from Microsoft for building a reverse proxy inside an ASP.NET Core application: routes, clusters of destinations, load balancing, health checks, rewriting requests and responses. You don’t need it for the Kladde, because Container Apps brings its own ingress. But with YARP you can see on your own computer what every proxy does, and in K4 it becomes the gateway in front of a service that has been split out.

This program starts two small instances and, in front of them, a proxy with an active health check every two seconds. Then it asks four times, stops one instance, asks again right away and a third time after six seconds. #:package fetches YARP from nuget.org, #:property PublishAot=false switches off the checks for Native AOT that are otherwise on for a single file.

// proxy.cs: two instances behind a YARP reverse proxy with an active health check; one instance stops.
#:sdk Microsoft.NET.Sdk.Web
#:package Yarp.ReverseProxy@2.3.0
#:property PublishAot=false

using Yarp.ReverseProxy;
using Yarp.ReverseProxy.Configuration;

await using var a = await StartInstanceAsync("instance-a", 5101);
await using var b = await StartInstanceAsync("instance-b", 5102);

var builder = WebApplication.CreateBuilder(args);
builder.Logging.ClearProviders();
builder.Services.AddReverseProxy().LoadFromMemory(
    [new RouteConfig { RouteId = "kladde", ClusterId = "kladde", Match = new RouteMatch { Path = "{**rest}" } }],
    [
        new ClusterConfig
        {
            ClusterId = "kladde",
            LoadBalancingPolicy = "RoundRobin",
            HealthCheck = new HealthCheckConfig
            {
                Active = new ActiveHealthCheckConfig
                {
                    Enabled = true,
                    Interval = TimeSpan.FromSeconds(2),
                    Timeout = TimeSpan.FromSeconds(1),
                    Policy = "ConsecutiveFailures",
                    Path = "/health",
                },
            },
            Destinations = new Dictionary<string, DestinationConfig>
            {
                ["a"] = new() { Address = "http://127.0.0.1:5101" },
                ["b"] = new() { Address = "http://127.0.0.1:5102" },
            },
        },
    ]);
await using var proxy = builder.Build();
proxy.MapReverseProxy();
proxy.Urls.Add("http://127.0.0.1:5100");
await proxy.StartAsync();

using var http = new HttpClient { BaseAddress = new Uri("http://127.0.0.1:5100") };
Console.WriteLine("Both instances are running:");
await AskAsync(http, 4);
Console.WriteLine($"What an instance learns about the request: {await http.GetStringAsync("/who")}");

await b.StopAsync();
Console.WriteLine("instance-b stopped, right afterward:");
await AskAsync(http, 4);

await Task.Delay(TimeSpan.FromSeconds(6));
Console.WriteLine($"After the next health checks ({Health(proxy, "a")} / {Health(proxy, "b")}):");
await AskAsync(http, 4);

static async Task AskAsync(HttpClient http, int times)
{
    for (var i = 1; i <= times; i++)
    {
        using var response = await http.GetAsync("/");
        var text = response.IsSuccessStatusCode ? await response.Content.ReadAsStringAsync() : $"{(int)response.StatusCode} {response.ReasonPhrase}";
        Console.WriteLine($"  {i}  {text}");
    }
}

static string Health(WebApplication proxy, string destination)
{
    var state = proxy.Services.GetRequiredService<IProxyStateLookup>();
    return state.TryGetCluster("kladde", out var cluster) && cluster.Destinations.TryGetValue(destination, out var d)
        ? $"{destination}: {d.Health.Active}"
        : $"{destination}: ?";
}

static async Task<WebApplication> StartInstanceAsync(string name, int port)
{
    var builder = WebApplication.CreateBuilder();
    builder.Logging.ClearProviders();
    var app = builder.Build();
    app.Urls.Add($"http://127.0.0.1:{port}");
    app.MapGet("/", () => name);
    app.MapGet("/health", () => "ok");
    app.MapGet("/who", (HttpContext context) =>
        $"{name}: sender {context.Connection.RemoteIpAddress}, X-Forwarded-For {context.Request.Headers["X-Forwarded-For"]}, " +
        $"X-Forwarded-Proto {context.Request.Headers["X-Forwarded-Proto"]}, X-Forwarded-Host {context.Request.Headers["X-Forwarded-Host"]}");
    await app.StartAsync();
    return app;
}

dotnet run proxy.cs

Both instances are running:
  1  instance-b
  2  instance-a
  3  instance-b
  4  instance-a
What an instance learns about the request: instance-b: sender 127.0.0.1, X-Forwarded-For 127.0.0.1, X-Forwarded-Proto http, X-Forwarded-Host 127.0.0.1:5100
instance-b stopped, right afterward:
  1  instance-a
  2  502 Bad Gateway
  3  instance-a
  4  502 Bad Gateway
After the next health checks (a: Healthy / b: Unhealthy):
  1  instance-a
  2  instance-a
  3  instance-a
  4  instance-a

What you see:

• Round robin: The requests take turns.
• What the instance learns: The connection comes from the proxy (sender), and the headers say who asked, over which protocol and under which host. Here everything is 127.0.0.1, because everything runs on your computer.
• The gap: Right after the stop, the proxy still sends every second request to the dead instance and answers with 502 Bad Gateway. It doesn’t know yet.
• The check kicks in: After two failed checks in a row (the default of the ConsecutiveFailures policy), b counts as Unhealthy, and all requests go to a.

The gap is as long as the interval between checks times their number. Shorter intervals shrink it and cost load; passive checks and a retry on the next instance almost close it, the latter only for requests that are safe to send twice (GET, not booking an appointment). This very trade-off comes back in module 3.9.

// kladde-proxy.cs: a reverse proxy in front of two instances of the Kladde; its routes and checks come from appsettings.json.
#:sdk Microsoft.NET.Sdk.Web
#:package Yarp.ReverseProxy@2.3.0
#:property PublishAot=false

using Yarp.ReverseProxy.Model;
using Yarp.ReverseProxy.Transforms;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddReverseProxy()
    .LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"))
    .AddTransforms(context => context.AddResponseTransform(transform =>
    {
        // Name the instance that answered, so a test can see where each request went.
        var destination = transform.HttpContext.Features.Get<IReverseProxyFeature>()?.ProxiedDestination;
        transform.HttpContext.Response.Headers["X-Kladde-Target"] = destination?.DestinationId ?? "-";
        return ValueTask.CompletedTask;
    }));

var app = builder.Build();
app.MapReverseProxy();
app.Run("http://127.0.0.1:5100");

{
  "Logging": {
    "LogLevel": {
      "Default": "Warning",
      "Yarp.ReverseProxy.Health": "Information"
    }
  },
  "ReverseProxy": {
    "Routes": {
      "kladde": {
        "ClusterId": "kladde",
        "Match": { "Path": "{**rest}" }
      }
    },
    "Clusters": {
      "kladde": {
        "LoadBalancingPolicy": "RoundRobin",
        "HealthCheck": {
          "Active": {
            "Enabled": true,
            "Interval": "00:00:05",
            "Timeout": "00:00:02",
            "Policy": "ConsecutiveFailures",
            "Path": "/health"
          }
        },
        "Destinations": {
          "a": { "Address": "http://127.0.0.1:5101" },
          "b": { "Address": "http://127.0.0.1:5102" }
        }
      }
    }
  }
}

1..6 | ForEach-Object {
    $r = Invoke-WebRequest -Uri http://127.0.0.1:5100/health -SkipHttpErrorCheck
    '{0}  {1}  {2}' -f $_, $r.StatusCode, ($r.Headers['X-Kladde-Target'] -join '')
}

1  200  b
2  200  a
3  200  b
4  200  a
5  200  b
6  200  a

1  502  b
2  200  a
3  502  b
4  200  a
5  502  b
6  200  a

1  200  a
2  200  a
3  200  a
4  200  a
5  200  a
6  200  a

info: Yarp.ReverseProxy.Health.DestinationHealthUpdater[50]
      Active health state of destination `a` on cluster `kladde` is set to 'Healthy'.
warn: Yarp.ReverseProxy.Health.DestinationHealthUpdater[49]
      Active health state of destination `b` on cluster `kladde` is set to 'Unhealthy'.

// echo.cs: answers with what it learns about the request: the connection's sender and the X-Forwarded headers.
#:sdk Microsoft.NET.Sdk.Web
#:property PublishAot=false

var app = WebApplication.Create(args);
app.MapGet("/health", () => "Healthy");
app.MapGet("/{**rest}", (HttpContext context) =>
    $"Sender {context.Connection.RemoteIpAddress}, X-Forwarded-For '{context.Request.Headers["X-Forwarded-For"]}', " +
    $"X-Forwarded-Proto '{context.Request.Headers["X-Forwarded-Proto"]}'\n");
app.Run();

dotnet run echo/echo.cs -- --urls http://127.0.0.1:5101
dotnet run echo/echo.cs -- --urls http://127.0.0.1:5102
curl.exe -s -H 'X-Forwarded-For: 203.0.113.66' http://127.0.0.1:5100/contacts
curl.exe -s -H 'X-Forwarded-For: 203.0.113.66' http://127.0.0.1:5101/contacts

Sender 127.0.0.1, X-Forwarded-For '127.0.0.1', X-Forwarded-Proto 'http'
Sender 127.0.0.1, X-Forwarded-For '203.0.113.66', X-Forwarded-Proto ''

The Kladde after this lesson

The Kladde doesn’t change. But you now know why behind a proxy it has to evaluate headers, why two instances need a shared key ring and why long work doesn’t belong in a request. All three points come back in K3: in lessons 3.3.4, 3.3.2 and 3.9.1.

What you can do now

• You tell forward and reverse proxies apart and load balancers at layer 4 and 7, and you place the services of Azure.
• You explain active and passive health checks and the gap between failure and detection.
• You know when session affinity is needed and why it’s better to get by without it.
• You know the timeouts on the way to the Kladde and the rule “inside shorter than outside”.
• You know what TLS termination at the proxy means for Request.Scheme and how X-Forwarded-Proto helps.
• You believe X-Forwarded-For only from your own proxy and know why the instance may only be reachable through it.
• You’ve set up a proxy with a health check using YARP.

Read more

• Introduction to Azure Load Balancer: a Microsoft Learn module on the layer 4 load balancer.
• Load-balancing options: when to use Load Balancer, Application Gateway, Front Door or Traffic Manager.
• Ingress in Azure Container Apps: timeout, headers and affinity of the ingress.
• Configure ASP.NET Core to work with proxy servers and load balancers: the Forwarded Headers middleware in detail.
• YARP overview and YARP destination health checks.