How the web talks: HTTP

From this module on, the Kladde moves into the browser. Before you build the first page, you learn the language that browsers and servers use to talk to each other: HTTP. Every page, every image, every submitted form and, later, every call to the Kladde API is an HTTP request with an HTTP response. If you can read those two, you’ll find most errors on the web in minutes instead of hours.

The Kladde itself doesn’t change in this lesson. You work with a small practice server, send it requests, and learn to predict what it answers.

Your practice folder for this module

The practice servers in this module are file-based apps, like the ones you know from K0. Create a folder of their own for them next to the Kladde’s repository, not inside it. Otherwise the Kladde’s rules apply to the exercises (its Directory.Build.props, for example), and the Kladde’s Git history picks up files that don’t belong to it.

cd ~
mkdir k2-practice
cd k2-practice

If your Kladde is in ~/kladde, the exercises are now in ~/k2-practice. In Codespaces, create the folder under /workspaces, also next to the repository.

Request and response

HTTP is a conversation with fixed rules. The client (a browser, curl, later an HttpClient in C#) sends a request. The server sends back exactly one response. Then the conversation is over; the next request knows nothing about the previous one. We say HTTP is stateless. Whatever a server has to know across several requests, such as who is signed in, is sent again with every request, for example in a cookie (module 2.6).

A request in HTTP/1.1 is text and looks like this:

GET /contacts HTTP/1.1
Host: localhost:5080
User-Agent: curl/8.21.0
Accept: */*

• The first line is the request line: the method (GET), the target (/contacts, the path with an optional query) and the version of the protocol.
• Below it are the headers, one per line as Name: Value. Host is mandatory, because many websites can sit behind one address on the same machine.
• An empty line ends the headers.
• After that, a body can follow, such as the data of a form. A GET has none.

The response is built the same way:

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked

["Mara Feld","Ilse Kranich"]

• The status line gives the version, the status code (200) and a text for it (OK). Programs read only the number.
• Then come the response’s headers, an empty line and the body.

Newer versions, HTTP/2 and HTTP/3, pack the same parts in binary and faster, but they mean exactly the same thing: method, target, headers, status code and body exist in all versions. So everything in this lesson applies to all three.

The practice server

In k2-practice, create the file server.cs. You don’t have to understand every line yet; you’ll get to know ASP.NET Core properly in module 2.3. Still, read through the code once: every line with app.Map… answers one method on one path.

// server.cs: a small practice server for the lesson on HTTP
#:sdk Microsoft.NET.Sdk.Web
#:property PublishAot=false

var builder = WebApplication.CreateBuilder();
builder.Logging.SetMinimumLevel(LogLevel.Warning);
var app = builder.Build();

Dictionary<int, string> contacts = new() { [1] = "Mara Feld", [2] = "Ilse Kranich" };
int nextId = 3;

app.MapGet("/", () => Results.Content(
    "<!doctype html><html lang=\"en\"><title>Kladde</title><h1>Hello from server.cs</h1></html>",
    "text/html; charset=utf-8"));

app.MapGet("/contacts", () => contacts.Values);

app.MapGet("/contacts/{id:int}", (int id) =>
    contacts.TryGetValue(id, out string? name)
        ? Results.Ok(new { id, name })
        : Results.NotFound());

app.MapPost("/contacts", (NewContact input) =>
{
    if (string.IsNullOrWhiteSpace(input.Name))
    {
        return Results.Text("A contact needs a name.", "text/plain; charset=utf-8", statusCode: 400);
    }

    int id = nextId++;
    contacts[id] = input.Name.Trim();
    return Results.Created($"/contacts/{id}", new { id, name = contacts[id] });
});

app.MapGet("/old", () => Results.Redirect("/contacts", permanent: true));

app.MapGet("/logo.svg", (HttpResponse response) =>
{
    response.Headers.CacheControl = "public, max-age=86400";
    return Results.Content(
        "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"16\" height=\"16\"><rect width=\"16\" height=\"16\" fill=\"teal\"/></svg>",
        "image/svg+xml");
});

app.MapGet("/time", (HttpResponse response) =>
{
    response.Headers.CacheControl = "no-store";
    return DateTimeOffset.Now.ToString("HH:mm:ss");
});

Console.WriteLine("The practice server is running on http://localhost:5080. Quit with Ctrl+C.");
app.Run("http://localhost:5080");

record NewContact(string? Name);

Two lines at the start are new:

• #:sdk Microsoft.NET.Sdk.Web turns the file-based app into a web app. That makes WebApplication and everything else a web server needs available without further packages.
• #:property PublishAot=false switches off a default of file-based apps: they’re set up to be published as Native AOT later, and for that the conversion to JSON is switched off unless you prepare it. Without this line, the server answers every request with 500, and the terminal shows a NotSupportedException with JsonTypeInfo metadata … was not provided. For practice servers you don’t need Native AOT.

Start the server in a terminal of its own:

dotnet run server.cs

The practice server is running on http://localhost:5080. Quit with Ctrl+C.

The first start takes a few seconds, because dotnet has to build the app first. Leave this terminal open; you send the requests from a second one. As a test, open http://localhost:5080 in the browser: it says “Hello from server.cs”. In Codespaces, the codespace forwards the port and offers to open the address in the browser.

The parts of an address

An address on the web is called a URL (Uniform Resource Locator). .NET can take its parts apart with the Uri class. Create address.cs:

// address.cs
var address = new Uri("https://kladde.example:8443/contacts/search?q=feld&page=2#results");

Console.WriteLine($"Scheme:   {address.Scheme}");
Console.WriteLine($"Host:     {address.Host}");
Console.WriteLine($"Port:     {address.Port}");
Console.WriteLine($"Path:     {address.AbsolutePath}");
Console.WriteLine($"Query:    {address.Query}");
Console.WriteLine($"Fragment: {address.Fragment}");

var withoutPort = new Uri("https://kladde.example/contacts");
Console.WriteLine($"Port when none is given: {withoutPort.Port}");

var baseAddress = new Uri("https://kladde.example/contacts/list");
Console.WriteLine(new Uri(baseAddress, "details"));
Console.WriteLine(new Uri(baseAddress, "/appointments"));
Console.WriteLine(new Uri(baseAddress, "../appointments"));

string company = "Feld & Partner";
string encoded = Uri.EscapeDataString(company);
Console.WriteLine(encoded);
Console.WriteLine(Uri.UnescapeDataString(encoded));
Console.WriteLine(Uri.EscapeDataString("Café"));

dotnet run address.cs

Scheme:   https
Host:     kladde.example
Port:     8443
Path:     /contacts/search
Query:    ?q=feld&page=2
Fragment: #results
Port when none is given: 443
https://kladde.example/contacts/details
https://kladde.example/appointments
https://kladde.example/appointments
Feld%20%26%20Partner
Feld & Partner
Caf%C3%A9

What the parts mean:

• Scheme (https): how the conversation is held. http is unencrypted, https is encrypted.
• Host (kladde.example): which machine. The ending .example is reserved for examples, just like example.com; such addresses belong to nobody. In exercises and test data of the Kladde, you use only such names.
• Port (8443): which service on the machine. If it’s missing, 80 applies for http and 443 for https. Your practice server runs on 5080.
• Path (/contacts/search): which resource on the server.
• Query (?q=feld&page=2): pairs of name and value, separated by &. Forms with GET send their fields this way (lesson 2.1.3).
• Fragment (#results): a spot in the page. The browser never sends the fragment to the server; it just jumps there on its own.

The three lines with baseAddress show relative addresses: details replaces the last part of the path, /appointments starts at the host, ../appointments goes up one level. That’s also how the browser resolves links and the Location header, which often contain only a path.

Characters such as spaces, &, ?, # or accented letters have a meaning of their own in an address or aren’t allowed. They get percent-encoded: %20 is a space, %26 is an &, and an é becomes the two bytes of its UTF-8 encoding, %C3%A9. If you wrote “Feld & Partner” unencoded as ?company=Feld & Partner, the server would see two pairs: company with the value “Feld” and a second pair named “Partner” without a value, because the & separates pairs. Uri.EscapeDataString encodes a single value correctly; never put addresses together from unencoded input.

Methods: safe and idempotent

The method says what the request wants. The most important ones and their two properties, as the HTTP standard (RFC 9110) defines them:

• Safe means the request changes nothing on the server. A browser, a search engine or a preview service may send a safe request at any time, even if nobody clicks. That’s why a link (always GET) must never delete anything.
• Idempotent means the same request twice has the same effect on the server as once. DELETE /contacts/2 twice deletes the contact once; the second response is 404, but the state is the same. POST /contacts twice, on the other hand, creates two contacts.

Why this matters: networks lose responses. If no response comes back, the client doesn’t know whether the request arrived. It can repeat an idempotent request without risk, but not a non-idempotent one. That’s why the browser asks before it reloads a submitted form, and why the Kladde follows this rule: Only what is idempotent gets retried. In module 2.8 you’ll see how a well-meant retry sends an email twice.

Whether a method is really safe and idempotent depends on the server sticking to it. The standard describes what clients may expect. A GET /contacts/2/delete that deletes breaks that, and the server bears the consequences.

Status codes

The first digit of the status code names the class:

• 1xx Information: the request is still in progress. You rarely run into these.
• 2xx Success.
• 3xx Redirection: the answer is somewhere else.
• 4xx Client error: the request itself is wrong, incomplete or not allowed. Sending the same request again doesn’t help.
• 5xx Server error: the request was fine, the server couldn’t fulfill it. It may work later.

The codes you’ll meet in the Kladde:

A rule of thumb for your own API: 4xx means “you have to change something”, 5xx means “we have to change something”. An error in the user’s data is never a 500.

Headers you need to know

Headers carry everything that isn’t the body. You’ll run into these all the time:

• Content-Type says what the body is: text/html; charset=utf-8, application/json; charset=utf-8, text/css, image/svg+xml. The browser uses it to decide whether to show something as a page, apply it as a stylesheet or save it as a file. In a request, it says in which format the body is sent. charset=utf-8 sets the encoding; if it’s missing, the browser guesses, and “Café” turns into “Café”.
• Content-Length gives the length of the body in bytes. If Transfer-Encoding: chunked is there instead, the body comes in pieces whose total length the server didn’t know beforehand.
• Location gives the address of the new resource for 201 and the target of the redirect for 3xx.
• Allow names the allowed methods for 405.
• Cache-Control says whether and for how long a response may be cached:
  • no-store: don’t store it at all, for example for personal data or the time of day.
  • max-age=86400: reuse it for a day (86,400 seconds) without asking again.
  • public: caches along the way may keep it too. private: only the browser of this person.
  • no-cache: store it, yes, but ask the server before every use. If the response has an ETag (an identifier of the version), the browser asks with If-None-Match, and the server answers 304 if nothing has changed. You’ll see that in the browser in lesson 2.1.5.
• Accept and Accept-Language in the request say which formats and languages the client prefers.
• Set-Cookie in the response creates a cookie, Cookie in every following request sends it back. More on that in module 2.6.

Header names are case-insensitive. HTTP/2 writes them in lowercase; content-type and Content-Type are the same header.

Requests with curl

curl is a program for sending requests from the command line. It’s already installed on Windows, macOS, Linux and in Codespaces. In PowerShell 7 you call it as curl, like in bash; only in the old Windows PowerShell 5.1 do you have to write curl.exe, because curl is an alias for Invoke-WebRequest there.

The most important switches:

With the practice server running, in the second terminal:

curl -i http://localhost:5080/contacts

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: <weekday, date and time in GMT>
Server: Kestrel
Transfer-Encoding: chunked

["Mara Feld","Ilse Kranich"]

Date carries the time of the response, Server the name of ASP.NET Core’s web server, Kestrel. In the following outputs, these two lines are left out; on your screen they’re there every time.

With -v you also see what curl sent. The lines with > are the request, the ones with < the response, the ones with * notes from curl itself:

curl -v http://localhost:5080/contacts

* Host localhost:5080 was resolved.
...
> GET /contacts HTTP/1.1
> Host: localhost:5080
> User-Agent: curl/8.21.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Transfer-Encoding: chunked
<
["Mara Feld","Ilse Kranich"]

The version of curl in User-Agent may be a different one on your machine.

Creating a contact, with JSON as the body:

curl -i -X POST -H "Content-Type: application/json" -d '{"name":"Tom Wieland"}' http://localhost:5080/contacts

HTTP/1.1 201 Created
Content-Type: application/json; charset=utf-8
Location: /contacts/3
Transfer-Encoding: chunked

{"id":3,"name":"Tom Wieland"}

The same line works in bash and in PowerShell 7. And a contact without a name:

curl -i -X POST -H "Content-Type: application/json" -d '{"name":" "}' http://localhost:5080/contacts

HTTP/1.1 400 Bad Request
Content-Length: 23
Content-Type: text/plain; charset=utf-8

A contact needs a name.

The sentence has 23 characters, all plain ASCII, so 23 bytes. If it contained an é, it would be one more, because UTF-8 needs two bytes for it.

Also try curl -I http://localhost:5080/contacts. The practice server answers 405 Method Not Allowed and Allow: GET, POST, because MapGet answers only GET. Many servers answer HEAD on their own like a GET without a body; this one doesn’t. That’s a good example of how you can predict what a server does by reading its code.

A pitfall: the practice server’s data lives only in memory. Every POST creates a contact that stays until you stop the server. After a restart there are two contacts again, and the next one gets the number 3 again. The outputs in this lesson assume that you run the examples in order.

Requests with PowerShell

PowerShell has two commands of its own. Invoke-WebRequest returns the whole response as an object, Invoke-RestMethod returns the body right away, already translated into objects for JSON:

$response = Invoke-WebRequest http://localhost:5080/contacts
$response.StatusCode
$response.Headers['Content-Type']
$response.Content

200
application/json; charset=utf-8
["Mara Feld","Ilse Kranich","Tom Wieland"]

Invoke-RestMethod -Method Post -Uri http://localhost:5080/contacts -ContentType 'application/json' -Body '{"name":"Jana Ost"}'

id name
-- ----
 4 Jana Ost

You should know two quirks:

• For a status code from 400 up, Invoke-WebRequest throws an error instead of returning the response. With -SkipHttpErrorCheck you get the response and check the code yourself:

  PowerShell

  (Invoke-WebRequest http://localhost:5080/contacts/7 -SkipHttpErrorCheck).StatusCode
  

  404
  

• Invoke-WebRequest follows redirects on its own. With -MaximumRedirection 0 it doesn’t, but it still reports the redirect it didn’t follow as an error, even with -SkipHttpErrorCheck. With -ErrorAction SilentlyContinue the message stays away, and the response is there:

  PowerShell

  $response = Invoke-WebRequest http://localhost:5080/old -MaximumRedirection 0 -SkipHttpErrorCheck -ErrorAction SilentlyContinue
  $response.StatusCode
  $response.Headers.Location
  

  301
  /contacts
  

Redirects and who follows them

A browser follows a redirect immediately and shows the new address. curl does that only with -L:

curl -i -L http://localhost:5080/old

HTTP/1.1 301 Moved Permanently
Content-Length: 0
Location: /contacts

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked

["Mara Feld","Ilse Kranich","Tom Wieland","Jana Ost"]

.NET’s HttpClient also follows redirects unless you tell it not to. Create client.cs and run it while the server is running:

// client.cs: the same address, two clients
using var follows = new HttpClient();
using var response1 = await follows.GetAsync("http://localhost:5080/old");
Console.WriteLine($"Follows redirects: {(int)response1.StatusCode}, landed on {response1.RequestMessage?.RequestUri}");

using var handler = new HttpClientHandler { AllowAutoRedirect = false };
using var doesNotFollow = new HttpClient(handler);
using var response2 = await doesNotFollow.GetAsync("http://localhost:5080/old");
Console.WriteLine($"Follows no redirects: {(int)response2.StatusCode}, Location: {response2.Headers.Location}");
Console.WriteLine($"IsSuccessStatusCode: {response1.IsSuccessStatusCode} and {response2.IsSuccessStatusCode}");

dotnet run client.cs

Follows redirects: 200, landed on http://localhost:5080/contacts
Follows no redirects: 301, Location: /contacts
IsSuccessStatusCode: True and False

For a person in a browser, following is right. For a program that calls an API, it’s dangerous: if the server redirects to a sign-in page, the client fetches that page, gets 200 and takes it for success, although nothing was saved. That’s why the Kladde has this rule from module 2.3 on: Machine clients don’t follow redirects, and a 3xx response counts there as an error. In module 2.7 you’ll write a test that turns exactly this case red.

HTTPS in a nutshell

https is HTTP in an encrypted channel, TLS (Transport Layer Security). Before the first request goes out, browser and server negotiate keys. After that, TLS does three things:

• Confidentiality: nobody along the way, such as on the Wi-Fi in a café, can read along, not even the path, the query, the headers or cookies. What stays visible is only which host is being talked to.
• Integrity: nobody can change anything along the way without it being noticed.
• Authenticity: the server proves with a digital certificate that it really belongs to this host name. The certificate was signed by a party your browser trusts. If it doesn’t match, the browser warns in big red letters. Never just click that warning away.

What TLS does not do: it doesn’t protect you from a server that is malicious itself, and not from errors in your application. A form over https can still accept any input an attacker sends.

The practice server takes only http, because it runs exclusively on your own machine under localhost, where nothing leaves the network. Everything that goes to the internet goes over https. In module 2.3 you’ll set up HTTPS for development too, with the command dotnet dev-certs https --trust.

curl -i http://localhost:5080/contacts/2
curl -i http://localhost:5080/contacts/abc
curl -i http://localhost:5080/old
curl -i -X PUT http://localhost:5080/contacts
curl -i -X POST -H "Content-Type: application/json" -d '{"name":"Jana Ost"}' http://localhost:5080/contacts
curl -i -X POST -d 'name=Jana Ost' http://localhost:5080/contacts

1  GET /contacts/2               200 OK, Content-Type: application/json; charset=utf-8
                                 {"id":2,"name":"Ilse Kranich"}
2  GET /contacts/abc             404 Not Found, no body
3  GET /old                      301 Moved Permanently, Location: /contacts, no body
4  PUT /contacts                 405 Method Not Allowed, Allow: GET, POST, no body
5  POST /contacts (JSON)         201 Created, Location: /contacts/3
                                 {"id":3,"name":"Jana Ost"}
6  POST /contacts (form)         415 Unsupported Media Type, no body

1  401   not signed in
2  201   created, with Location
3  301   moved for good
4  409   contradicts the state
5  204   success without a body
6  400   the request is faulty
7  500   error in the server
8  403   signed in, but not permitted
9  429   too many requests
10 303   get the result with GET

# check-server.ps1: checks the status codes of the practice server
$base = 'http://localhost:5080'
$expected = @(
    @{ Method = 'GET';    Path = '/';              Status = 200 }
    @{ Method = 'GET';    Path = '/contacts';      Status = 200 }
    @{ Method = 'GET';    Path = '/contacts/7';    Status = 404 }
    @{ Method = 'GET';    Path = '/old';           Status = 301 }
    @{ Method = 'DELETE'; Path = '/contacts';      Status = 405 }
    @{ Method = 'GET';    Path = '/does-not-exist'; Status = 404 }
)

$mismatches = 0
foreach ($case in $expected) {
    $response = Invoke-WebRequest -Uri ($base + $case.Path) -Method $case.Method `
        -MaximumRedirection 0 -SkipHttpErrorCheck -ErrorAction SilentlyContinue
    $actual = [int]$response.StatusCode
    if ($actual -eq $case.Status) {
        Write-Host ("ok      {0,-6} {1,-16} {2}" -f $case.Method, $case.Path, $actual)
    }
    else {
        Write-Host ("WRONG   {0,-6} {1,-16} {2}, expected {3}" -f $case.Method, $case.Path, $actual, $case.Status)
        $mismatches++
    }
}

if ($mismatches -gt 0) {
    Write-Host "$mismatches mismatch(es)."
    exit 1
}
Write-Host 'All status codes as expected.'
exit 0

pwsh -NoProfile -File ./check-server.ps1
$LASTEXITCODE

ok      GET    /                200
ok      GET    /contacts        200
ok      GET    /contacts/7      404
ok      GET    /old             301
ok      DELETE /contacts        405
ok      GET    /does-not-exist  404
All status codes as expected.
0

// excerpt from server.cs
app.MapDelete("/contacts/{id:int}", (int id) =>
    contacts.Remove(id) ? Results.NoContent() : Results.NotFound());

curl -i -X DELETE http://localhost:5080/contacts/2
curl -i -X DELETE http://localhost:5080/contacts/2
curl http://localhost:5080/contacts

HTTP/1.1 204 No Content

HTTP/1.1 404 Not Found
Content-Length: 0

["Mara Feld"]

What you can do now

• You read an HTTP request (method, target, headers, body) and a response (status code, headers, body).
• You take an address apart into scheme, host, port, path, query and fragment, and encode values with Uri.EscapeDataString.
• You know which methods are safe and which are idempotent, and you retry only what is idempotent.
• You match situations to status codes and tell 4xx from 5xx.
• You read Content-Type, Location, Allow and Cache-Control.
• You send requests with curl and PowerShell and decide on purpose whether a redirect is followed.

Read more

• Make HTTP requests with the HttpClient class: the methods from this lesson with .NET’s HttpClient, including HEAD and OPTIONS.
• File-based apps: all directives such as #:sdk and #:property.
• Inspect network activity: looking at requests and responses in the browser, the topic of lesson 2.1.5.
• HTTP on MDN: detailed reference pages on every method, every status code and every header.
• RFC 9110, HTTP Semantics: the standard itself, for the finer points of “safe” and “idempotent”.