TLS and the chain of trust

Every request to the Kladde in Azure goes over HTTPS. The padlock in the browser is so much a given that you forget how much is behind it: a key exchange, a chain of signatures up to an authority that your operating system trusts, and a check whether the server is really called what you called it. In this lesson you look at that on real servers, and then you build it yourself: a small certificate authority of your own, a server signed by it, and a client that rejects it until you tell it whom to believe.

A word on terms: What a server presents in TLS is an X.509 certificate, a signed document with its public key and the names it may speak for. The organizations that issue such certificates are called certificate authorities (CA).

What TLS protects

TLS (Transport Layer Security) sits between TCP and HTTP and protects three things:

• Confidentiality: Anyone who reads the packets along the way, on café Wi-Fi, at the provider, in a data center, sees only encrypted data. A sign-in cookie of the Kladde or a contact stays secret.
• Integrity: Nobody can change anything along the way without it being noticed. One changed byte, and the connection breaks off.
• Authenticity of the server: The client knows it is really talking to kladde.example and not to someone who has put themselves in between. The certificate does that, and that is the hardest part.

What TLS does not protect: what the server does with the data, what ends up in logs and who is signed in at the other end. And the name you call: it stands readable in the first packet when the connection is set up (more on that in a moment).

The TLS 1.3 handshake

After the TCP handshake from lesson 3.0.2 comes the TLS handshake. In TLS 1.3 it needs only one round trip:

Client                                                   Server
  | -- ClientHello: TLS versions, ciphers, key share,           |
  |    SNI "learn.microsoft.com" -------------------------->    |
  | <-- ServerHello: chosen cipher, key share                   |
  |     encrypted from here on:                                 |
  |     certificate and chain, proof of the private key,        |
  |     Finished ------------------------------------------     |
  | Client checks chain and name                                |
  | -- Finished ------------------------------------------->    |
  |              encrypted HTTP requests                        |

The most important terms in it:

• Key exchange: Both sides each send a public share (today mostly with the method X25519) and calculate the same secret session key from them, without it ever going over the wire. A new one is created for every connection. Whoever later steals the server’s private key still can’t decrypt old, recorded connections.
• Proof of the private key: Anyone can show the certificate, it is public. That is why the server also signs the handshake so far with its private key. Only someone who has the private key can do that.
• SNI (Server Name Indication): The client writes into its first packet which name it wants to reach. That way one address can serve many names and show the matching certificate for each. Front Door and the ingress of Container Apps live off this.

Older versions such as TLS 1.0 and 1.1 are considered insecure, and Azure services no longer accept them. TLS 1.2 is still common, TLS 1.3 the standard.

The chain of trust

How does your machine know that the certificate is genuine? It trusts a small list of root certificate authorities (roots) that comes with the operating system. A root never signs a server directly, though, but an intermediate CA, and that signs the server. This creates a chain:

Root          DigiCert Global Root G3              sits in your operating system's store
  signs
Intermediate  Microsoft TLS ECC Root G2            sent along by the server
  signs
Intermediate  Microsoft TLS G2 ECC CA OCSP 06      sent along by the server
  signs
Server        learn.microsoft.com                  presented by the server

The client checks every step: Is every signature genuine? Is every link still valid, that is, not expired and not valid only in the future? May every link except the last sign on its own? Does the chain end at a root it trusts? And finally: Is the name I called in the certificate? The names are in the field Subject Alternative Name (SAN). Modern clients look only there, not in the name in the Subject field, even if often the same thing is in it.

The server doesn’t send the root along. It sends its certificate and the intermediates; the client must already have the root. If the server is missing an intermediate, some clients fail and others don’t, depending on whether they already know it. That is a classic error that occurs “only for some users”.

What a server presents: with .NET

This program sets up a TLS connection and shows what the server presents. In .NET, SslStream takes care of that; the callback RemoteCertificateValidationCallback gets the chain that .NET built and the result of the check.

// tlscheck.cs: connects to a server with TLS and shows what it presented: protocol, chain, names, days left.
// dotnet run tlscheck.cs -- <host>
using System.Net.Security;
using System.Net.Sockets;
using System.Security.Cryptography.X509Certificates;

var host = args.Length > 0 ? args[0] : "learn.microsoft.com";
var errors = SslPolicyErrors.None;
var chainLines = new List<string>();
var names = new List<string>();

using var tcp = new TcpClient();
await tcp.ConnectAsync(host, 443);
using var tls = new SslStream(tcp.GetStream(), false, (_, _, chain, sslErrors) =>
{
    errors = sslErrors;
    foreach (var (element, level) in chain!.ChainElements.Select((e, i) => (e, i)))
    {
        var c = element.Certificate;
        var days = (c.NotAfter - DateTime.Now).Days;
        chainLines.Add($"  {level}  {c.GetNameInfo(X509NameType.SimpleName, false),-32} by {c.GetNameInfo(X509NameType.SimpleName, true),-32} {days} days left");
    }

    var leaf = chain.ChainElements[0].Certificate;
    names.AddRange(leaf.Extensions.OfType<X509SubjectAlternativeNameExtension>().SelectMany(e => e.EnumerateDnsNames()));
    return sslErrors == SslPolicyErrors.None;
});
await tls.AuthenticateAsClientAsync(host);

Console.WriteLine($"{host}: {tls.SslProtocol}, {tls.NegotiatedCipherSuite}");
Console.WriteLine($"Check: {errors}");
Console.WriteLine("Chain (0 = server, the root last):");
chainLines.ForEach(Console.WriteLine);
Console.WriteLine($"Names in the certificate: {string.Join(", ", names.Take(6))}{(names.Count > 6 ? $" and {names.Count - 6} more" : "")}");

dotnet run tlscheck.cs -- learn.microsoft.com

learn.microsoft.com: Tls13, TLS_AES_256_GCM_SHA384
Check: None
Chain (0 = server, the root last):
  0  learn.microsoft.com              by Microsoft TLS G2 ECC CA OCSP 06  150 days left
  1  Microsoft TLS G2 ECC CA OCSP 06  by Microsoft TLS ECC Root G2        937 days left
  2  Microsoft TLS ECC Root G2        by DigiCert Global Root G3          954 days left
  3  DigiCert Global Root G3          by DigiCert Global Root G3          4085 days left
Names in the certificate: learn.microsoft.com, www.learn.microsoft.com

Your numbers, and perhaps the links of the chain too, will be different; certificates are renewed regularly. What you see:

• Tls13 and the cipher TLS_AES_256_GCM_SHA384: encryption is AES-256 in GCM mode, which at the same time ensures integrity.
• Check: None means: no errors. Chain fine, name matches.
• The root in line 3 signed itself. That is the case with every root; it becomes trustworthy only by being in your system’s list.
• The server’s certificate expires in about five months, the root in more than ten years. Server certificates keep getting shorter: the maximum lifetime that the browsers and certificate authorities set together is falling in steps from 398 to 47 days. Nobody renews them by hand reliably anymore, and that is intentional.

A warning about this callback: on the web you find examples that set ServerCertificateCustomValidationCallback = (_, _, _, _) => true, “because otherwise it doesn’t work”. That turns off the whole check. Anyone who puts themselves into the connection is then accepted. In code that runs anywhere other than on your machine, this has no place. When a check fails, find out why, as in the rest of this lesson.

What a server presents: with openssl

In the Codespace and in any Linux container with the package openssl, you ask the same without a program of your own. The < /dev/null makes sure that s_client leaves again right after the handshake instead of waiting for input; -servername sets the SNI.

openssl s_client -connect learn.microsoft.com:443 -servername learn.microsoft.com -brief < /dev/null

CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Peer certificate: C = US, ST = WA, L = Redmond, O = Microsoft Corporation, CN = learn.microsoft.com
Hash used: SHA256
Signature type: ECDSA
Verification: OK
Server Temp Key: X25519, 253 bits
DONE

The chain, as the server sends it (s: is the holder, i: the issuer):

openssl s_client -connect learn.microsoft.com:443 -servername learn.microsoft.com -showcerts < /dev/null 2>/dev/null | grep -E '^ *[0-9] s:|^ *i:'

 0 s:C = US, ST = WA, L = Redmond, O = Microsoft Corporation, CN = learn.microsoft.com
   i:C = US, O = Microsoft Corporation, CN = Microsoft TLS G2 ECC CA OCSP 06
 1 s:C = US, O = Microsoft Corporation, CN = Microsoft TLS G2 ECC CA OCSP 06
   i:C = US, O = Microsoft Corporation, CN = Microsoft TLS ECC Root G2
 2 s:C = US, O = Microsoft Corporation, CN = Microsoft TLS ECC Root G2
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G3

Three links, no root: exactly as described above. The client adds the root DigiCert Global Root G3 from its own store.

You read the certificate itself with openssl x509. -checkend checks whether it expires within the given number of seconds, here in 30 days, and reports that in the exit code. You can build monitoring on that:

openssl s_client -connect learn.microsoft.com:443 -servername learn.microsoft.com < /dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -ext subjectAltName
openssl s_client -connect learn.microsoft.com:443 -servername learn.microsoft.com < /dev/null 2>/dev/null \
  | openssl x509 -noout -checkend 2592000; echo "Exit code: $?"

subject=C = US, ST = WA, L = Redmond, O = Microsoft Corporation, CN = learn.microsoft.com
issuer=C = US, O = Microsoft Corporation, CN = Microsoft TLS G2 ECC CA OCSP 06
X509v3 Subject Alternative Name: 
    DNS:learn.microsoft.com, DNS:www.learn.microsoft.com
Certificate will not expire
Exit code: 0

-dates shows the start and end of the validity period as a date. In the browser you see the same through the icon at the left of the address bar; in the developer tools (F12), the “Security” tab shows protocol, cipher and chain.

Revocation

What if a private key is stolen? Then the certificate has to become invalid before it expires. For that there are revocation lists (CRL), which an authority publishes regularly, and a query at the moment of the connection (OCSP). Both work only moderately in practice: clients have to get the answer in time, and if they don’t get it, many allow the connection anyway. The most effective protection against stolen keys is therefore short lifetimes and a renewal that happens by itself. If the private key of one of your servers ever ends up in a repository, a log or a chat, that means: have it reissued at once, have the old one revoked, find out who saw it (lesson 3.3.7).

ACME and managed endpoints

Short lifetimes work only with automation. The ACME protocol governs how a program orders a certificate from an authority and proves that it owns the domain: by serving a file at http://<domain>/.well-known/acme-challenge/ or by setting a TXT record in DNS (lesson 3.0.3). Let’s Encrypt has used this to make free, automatically renewed certificates the normal case.

In Azure you mostly don’t have to worry about it at all:

• Azure Container Apps issues a managed certificate for a custom domain for free and renews it itself. The prerequisite is the DNS records from lesson 3.0.3: an A or CNAME record and the TXT proof asuid.
• Azure Front Door does the same for the domains that run through Front Door, with the TXT proof _dnsauth.
• Whoever needs material of their own, say because a partner demands a particular authority, puts it in Key Vault and lets Container Apps or Front Door fetch it from there. Lesson 3.3.7 builds that.

In both cases TLS ends at the edge: at the ingress of Container Apps or at Front Door. Behind it, the Kladde speaks unencrypted HTTP on port 8080 inside the platform. Why that is so, and what the Kladde then needs to know about the original request, comes in lesson 3.0.5.

If your DNS zone has a CAA record (lesson 3.0.3), the authority that the managed service uses must be allowed in it. Otherwise issuance fails, and the message rarely names the reason clearly.

The development certificate from K2

In K2 you ran dotnet dev-certs https --trust. Now you can place what that does: it creates a certificate for localhost and a few other local names that signs itself, so root and server in one, and enters it into your user’s list of trusted roots. From then on browsers and HttpClient on your machine accept https://localhost:… without a warning. On every other machine this certificate is worthless, and that is as it should be.

dotnet dev-certs https --check --trust

shows whether there is a valid one and whether it is trusted. In the Codespace you mostly don’t need it: a forwarded port can be reached there through an address under app.github.dev, and GitHub provides its TLS. The development certificate never belongs in the repository, and no private key at all: files such as *.pfx and *.key are in .gitignore.

A practice CA of your own

Now you build the chain yourself. The program creates a small CA of its own, the “Kladde Practice CA”, has it sign a certificate for localhost, starts a TLS server on 127.0.0.1 with it and connects three times: once with a client that trusts only the system’s roots, once with one that trusts exactly the practice CA, and once with one that calls a different name. Everything happens in memory. Nothing is entered into your operating system’s store, and that is also the right way for tests.

// ownca.cs: an own practice authority signs a server's TLS identity for localhost; a client refuses it until it trusts the authority.
using System.Net;
using System.Net.Security;
using System.Net.Sockets;
using System.Security.Authentication;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;

// 1. The practice authority: a key pair and a self-signed certificate that may sign others (CA = true).
using var authorityKey = ECDsa.Create(ECCurve.NamedCurves.nistP256);
var authorityRequest = new CertificateRequest("CN=Kladde Practice CA", authorityKey, HashAlgorithmName.SHA256);
authorityRequest.CertificateExtensions.Add(new X509BasicConstraintsExtension(true, false, 0, true));
authorityRequest.CertificateExtensions.Add(new X509KeyUsageExtension(X509KeyUsageFlags.KeyCertSign | X509KeyUsageFlags.CrlSign, true));
authorityRequest.CertificateExtensions.Add(new X509SubjectKeyIdentifierExtension(authorityRequest.PublicKey, false));
using var authority = authorityRequest.CreateSelfSigned(DateTimeOffset.UtcNow.AddMinutes(-5), DateTimeOffset.UtcNow.AddDays(30));

// 2. The server's identity: its own key pair, the names it may answer for, signed by the authority.
using var serverKey = ECDsa.Create(ECCurve.NamedCurves.nistP256);
var serverRequest = new CertificateRequest("CN=localhost", serverKey, HashAlgorithmName.SHA256);
var names = new SubjectAlternativeNameBuilder();
names.AddDnsName("localhost");
names.AddIpAddress(IPAddress.Loopback);
serverRequest.CertificateExtensions.Add(names.Build());
serverRequest.CertificateExtensions.Add(new X509BasicConstraintsExtension(false, false, 0, false));
serverRequest.CertificateExtensions.Add(new X509KeyUsageExtension(X509KeyUsageFlags.DigitalSignature, true));
serverRequest.CertificateExtensions.Add(new X509EnhancedKeyUsageExtension([new Oid("1.3.6.1.5.5.7.3.1")], false));
serverRequest.CertificateExtensions.Add(X509AuthorityKeyIdentifierExtension.CreateFromCertificate(authority, true, false));
using var issued = serverRequest.Create(authority, DateTimeOffset.UtcNow.AddMinutes(-5), DateTimeOffset.UtcNow.AddDays(7), RandomNumberGenerator.GetBytes(16));
using var withKey = issued.CopyWithPrivateKey(serverKey);
// Windows' TLS stack wants the private key in a key store, not only in memory: a round trip through PKCS#12 puts it there.
using var server = X509CertificateLoader.LoadPkcs12(withKey.Export(X509ContentType.Pkcs12), null);
using var authorityPublic = X509CertificateLoader.LoadCertificate(authority.RawData);

var san = server.Extensions.OfType<X509SubjectAlternativeNameExtension>().Single();
Console.WriteLine($"CA:     {authority.Subject}");
Console.WriteLine($"Server: {server.Subject}, issued by {server.Issuer}");
Console.WriteLine($"Names:  {string.Join(", ", san.EnumerateDnsNames().Concat(san.EnumerateIPAddresses().Select(a => a.ToString())))}");

// 3. A TLS server on a free port of 127.0.0.1 that presents its identity and the authority's.
var listener = new TcpListener(IPAddress.Loopback, 0);
listener.Start();
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
var context = SslStreamCertificateContext.Create(server, new X509Certificate2Collection(authorityPublic), offline: true);
_ = Task.Run(async () =>
{
    while (true)
    {
        var client = await listener.AcceptTcpClientAsync();
        _ = ServeAsync(client, context);
    }
});

// 4. Three clients.
Console.WriteLine($"{"1) trusts only the system:",-38} {await ConnectAsync(port, "localhost", null)}");
Console.WriteLine($"{"2) trusts the practice CA:",-38} {await ConnectAsync(port, "localhost", authorityPublic)}");
Console.WriteLine($"{"3) calls kladde.local, not localhost:",-38} {await ConnectAsync(port, "kladde.local", authorityPublic)}");
listener.Stop();

static async Task<string> ConnectAsync(int port, string targetHost, X509Certificate2? trusted)
{
    using var tcp = new TcpClient();
    await tcp.ConnectAsync(IPAddress.Loopback, port);
    using var tls = new SslStream(tcp.GetStream());
    var options = new SslClientAuthenticationOptions { TargetHost = targetHost };
    if (trusted is not null)
    {
        // Trust exactly this authority, and no other. It publishes no revocation list, so nothing to check there.
        options.CertificateChainPolicy = new X509ChainPolicy
        {
            TrustMode = X509ChainTrustMode.CustomRootTrust,
            RevocationMode = X509RevocationMode.NoCheck,
        };
        options.CertificateChainPolicy.CustomTrustStore.Add(trusted);
    }

    var seen = SslPolicyErrors.None;
    var status = string.Empty;
    options.RemoteCertificateValidationCallback = (_, _, chain, errors) =>
    {
        seen = errors;
        status = string.Join(", ", chain!.ChainStatus.Select(s => s.Status).Distinct());
        return errors == SslPolicyErrors.None;
    };

    try
    {
        await tls.AuthenticateAsClientAsync(options);
        return $"accepted, {tls.SslProtocol}";
    }
    catch (AuthenticationException)
    {
        return $"rejected: {seen}{(status.Length > 0 ? $" ({status})" : "")}";
    }
}

static async Task ServeAsync(TcpClient client, SslStreamCertificateContext context)
{
    using (client)
    using (var tls = new SslStream(client.GetStream()))
    {
        try
        {
            await tls.AuthenticateAsServerAsync(new SslServerAuthenticationOptions { ServerCertificateContext = context });
        }
        catch (Exception e) when (e is AuthenticationException or IOException)
        {
            // The client refused the identity: nothing more to say to it.
        }
    }
}

dotnet run ownca.cs

CA:     CN=Kladde Practice CA
Server: CN=localhost, issued by CN=Kladde Practice CA
Names:  localhost, 127.0.0.1
1) trusts only the system:             rejected: RemoteCertificateChainErrors (UntrustedRoot)
2) trusts the practice CA:             accepted, Tls13
3) calls kladde.local, not localhost:  rejected: RemoteCertificateNameMismatch

On Linux, line 1 says PartialChain instead of UntrustedRoot; the meaning is the same: the chain doesn’t end at a root that this client trusts. This is how you read the program:

• The CA is just a key pair and a certificate it signed itself, with the note “may sign others” (X509BasicConstraintsExtension with true). A root is technically nothing more. What distinguishes a real root from yours is solely that your operating system trusts it.
• The server has a key pair of its own. The CA signs only its public key, the names (localhost and 127.0.0.1 in the SAN) and the purpose “server authentication” (the identifier 1.3.6.1.5.5.7.3.1). The CA never sees the server’s private key.
• Client 1 builds the chain and finds at the end a root that is not in its list. Rejected.
• Client 2 trusts exactly this one CA with CustomRootTrust and no other. Accepted. That is the way to use a CA of your own in tests without changing the machine.
• Client 3 trusts the CA too, but calls a name that is not in the certificate. Rejected, and rightly so: whoever has a valid certificate for localhost may not appear as kladde.local with it.

The detour through Export and LoadPkcs12 is a peculiarity of Windows: its TLS stack takes a private key only from a key store, not only from memory. On Linux the detour does no harm.

using var issued = serverRequest.Create(authority, DateTimeOffset.UtcNow.AddDays(-8), DateTimeOffset.UtcNow.AddDays(-1), RandomNumberGenerator.GetBytes(16));

using var authority = authorityRequest.CreateSelfSigned(DateTimeOffset.UtcNow.AddDays(-30), DateTimeOffset.UtcNow.AddDays(30));

1) trusts only the system:             rejected: RemoteCertificateChainErrors (UntrustedRoot, NotTimeValid)
2) trusts the practice CA:             rejected: RemoteCertificateChainErrors (NotTimeValid)
3) calls kladde.local, not localhost:  rejected: RemoteCertificateNameMismatch, RemoteCertificateChainErrors (NotTimeValid)

names.AddDnsName("kladde.local");

var port = args.Length > 1 ? int.Parse(args[1]) : 443;
await tcp.ConnectAsync(host, port);

dotnet run tlscheck.cs -- localhost 7245

localhost: Tls13, TLS_AES_256_GCM_SHA384
Check: None
Chain (0 = server, the root last):
  0  localhost                        by localhost                         280 days left
Names in the certificate: localhost, *.dev.localhost, *.dev.internal, host.docker.internal, host.containers.internal

The Kladde after this lesson

The Kladde doesn’t change. Just check that .gitignore excludes private keys (*.pfx, *.key), and add the lines if they are missing.

What you can do now

• You explain what TLS protects and what it doesn’t, and know the steps of the TLS 1.3 handshake with SNI and key exchange.
• You read a chain of trust with .NET, openssl and in the browser and know what the client checks at each link.
• You know why lifetimes are getting shorter and renewal has to run automatically, and what Container Apps and Front Door take off your hands there.
• You place the development certificate of dotnet dev-certs.
• You have built a CA of your own and know how a client trusts it without changing the machine.
• You recognize the errors UntrustedRoot or PartialChain, NotTimeValid and RemoteCertificateNameMismatch and know what to do.

Further reading

• TLS/SSL best practices in .NET: how .NET chooses TLS versions and why you shouldn’t pin them.
• X509ChainPolicy Class: TrustMode, CustomTrustStore and RevocationMode.
• Custom domain names and free managed certificates in Azure Container Apps: managed certificates and the DNS records they need.
• Configure a custom domain on Azure Front Door: the check with _dnsauth and the automatic renewal.
• Enforce HTTPS in ASP.NET Core: the development certificate, HTTPS redirection and HSTS.