Linux and the shell in a container

K3 begins with this module. In K2 you built the Kladde, packed it into an image and delivered it to Azure once. Before K3 turns to subscriptions, Bicep and pipelines, you look one level deeper, at what every cloud application stands on. A container is a Linux process. A request is a TCP connection to a port. An address is a DNS name. What lies between the browser and the Kladde is TLS, proxies and firewalls. If you understand that, you find errors in minutes instead of days.

For this module you don’t need an Azure subscription yet. Everything runs in the Codespace, in Docker Desktop (or Podman) and against public addresses. The counterparts in Azure are named along the way and built from module 3.1 on. That leaves the rule “budget first, then the first resource” from 3.1.2 untouched.

Where the Kladde stands

At the end of K2, the Kladde is a web application with contacts, public appointment booking, tenders from the TED interface, sign-in with Entra ID and two languages. src/Kladde.Web has a multi-stage Dockerfile of its own (K2 2.9.7): it is built in the SDK image and run in the chiseled image aspnet:10.0-noble-chiseled-extra, as user app on port 8080. With the compose.yaml from K2 2.9.9 you start the Kladde along with SQL Server, without Aspire. A background service imports the tenders every night (K2 2.8.2).

The house rules from K1 and K2 still apply: names in code are English, texts for the users are English too, warnings are errors, the exit code counts, secrets never belong in code, logs or chats. This module adds one clarification: We write our scripts in PowerShell. You use bash where there is nothing else, namely inside a Linux container. PowerShell runs on Linux too (pwsh); it is already installed in the .NET SDK image.

Why Linux

Almost everything that surrounds your application in the cloud runs on Linux:

• Containers: A Linux container is an ordinary process on a Linux kernel, just shielded. It sees a file system of its own, process numbers of its own and a network of its own. On Windows and macOS, Docker Desktop starts a small Linux machine in the background for this.
• Azure Container Apps: Your Kladde runs there in a Linux container. The service ends it with a Linux signal (more on that in a moment).
• Build agents: The hosted runners of GitHub Actions and Azure Pipelines mostly run Ubuntu. What goes wrong in the build, you see in a Linux log.
• Codespaces: Your Codespace is a Linux container with a bash in the terminal.

You don’t have to become a Linux administrator. But you do have to find your way around a Linux shell, read permissions, find a process and search a log. That is what this lesson is about.

A shell in a container

Start an Ubuntu container with a bash:

docker run --rm -it --name exercise ubuntu:24.04 bash

The parts of the command:

• --rm deletes the container as soon as you leave it. Practice containers clean up after themselves this way.
• -it connects your terminal to the container (-i for input, -t for a terminal).
• --name exercise gives it a name you can use to reach it from a second terminal.
• ubuntu:24.04 is the image, bash the program that starts in it.

You see a prompt like root@3f2a9c1d7b6e:/#. That means: you are the user root, the machine is named after the ID of the container, and you are standing in the folder /. The # at the end indicates root; a normal user sees $. With exit or Ctrl+D you leave the shell, and the container is gone.

In the Codespace you don’t need Docker for this: the terminal there is already a bash in a Linux container. There you are not root, though, but a normal user (usually codespace).

The file system

Linux has no drive letters. Everything hangs under one root /. The folders you need most often:

You know the commands for moving around in a similar form from PowerShell, which even has aliases with the same names:

pwd                      # where am I?
cd /etc                  # change folder
ls -la                   # show everything, hidden files too (name starts with .)
cat /etc/os-release      # print a file
mkdir -p /tmp/a/b        # create a folder, intermediate folders too
cp /etc/hosts /tmp/a/    # copy
mv /tmp/a/hosts /tmp/a/b/hosts-copy
rm -r /tmp/a             # delete, with -r including contents; there is no recycle bin

cat /etc/os-release shows, among other things, in the container:

PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"

The last digit of the version may be higher on your machine. Case matters: /etc/Hosts does not exist.

Users and permissions

Every process runs as a user, and every file belongs to a user and a group. id shows who you are:

id

uid=0(root) gid=0(root) groups=0(root)

root has the number 0 and may do anything. That is exactly why your application should not run as root: whoever finds a hole in the Kladde would otherwise have all rights in the container. Since .NET 8, the .NET images therefore ship with a user app with the number 1654, and the chiseled images are already set to it. Because ports below 1024 need special rights on Linux, ASP.NET Core in these images listens on port 8080 instead of 80. That is the reason for the 8080 you have met everywhere since K2.

What is allowed on a file is shown by ls -l:

drwxr-x--- 2 app  app  4096 Nov  9 10:12 data
-rw-r--r-- 1 root root  221 Nov  9 10:12 hosts

The first character says what it is: d a folder, - a file. Then come three groups of three characters each:

For a folder, x means you may change into it, and w means you may create and delete files in it. After that come the number of links, owner, group, size, modification time and name.

You change permissions with chmod and the owner with chown. chmod has two notations. The numbers are shorter: reading counts 4, writing 2, executing 1, and you add them up per group.

chmod 750 /data            # owner everything, group read and enter, others nothing
chmod u=rwx,g=rx,o= /data  # the same in letters: u owner, g group, o others
chown app:app /data        # owner app, group app (only root may do this)
stat -c '%A %a %U:%G %n' /data

drwxr-x--- 750 app:app /data

stat with this format is nicer for checking than ls -l, because it shows exactly the fields you want to know, without a date.

groupadd --gid 1654 app
useradd --uid 1654 --gid app --no-create-home app
useradd --uid 2000 --no-create-home guest

groupadd --gid 1654 app
useradd --uid 1654 --gid app --no-create-home app
useradd --uid 2000 --no-create-home guest
mkdir /data
chown app:app /data
chmod 750 /data
stat -c '%A %U:%G' /data
runuser -u app -- touch /data/a.txt; echo "app: $?"
runuser -u guest -- touch /data/b.txt; echo "guest writes: $?"
runuser -u guest -- ls /data; echo "guest reads: $?"

drwxr-x--- app:app
app: 0
touch: cannot touch '/data/b.txt': Permission denied
guest writes: 1
ls: cannot open directory '/data': Permission denied
guest reads: 2

Packages in the image

The image ubuntu:24.04 is deliberately small. A lot of what you’re used to is missing:

ps

bash: ps: command not found

You install packages with apt-get. First update fetches the list of available packages, then install installs the ones you want:

apt-get update
apt-get install -y --no-install-recommends procps curl less ca-certificates

-y answers the confirmation prompt with yes, --no-install-recommends leaves out recommended extra packages. curl needs ca-certificates to be able to verify HTTPS peers (more on that in lesson 3.0.4). In a Dockerfile these steps go in one instruction, so that the package list does not stay behind as a layer of its own in the image:

RUN apt-get update \
 && apt-get install -y --no-install-recommends curl ca-certificates \
 && rm -rf /var/lib/apt/lists/*

In the Kladde’s runtime image there is neither apt-get nor a shell. That is deliberate: a chiseled image contains only what .NET needs to run. What isn’t there can’t have a security hole, and an attacker who finds a hole in the Kladde has no shell there to continue with. How you can look inside anyway comes further down.

Processes and signals

Every process has a number, the PID. In a container, the first process that starts is PID 1. In your practice container that is the bash, in the Kladde’s container it is dotnet. In the practice container, start a process in the background (the & at the end) and look at the processes:

sleep 600 &
ps -o pid,user,args

    PID USER     COMMAND
      1 root     bash
     23 root     sleep 600
     24 root     ps -o pid,user,args

Your numbers may be different. You end a process by sending it a signal. The most important ones:

Without an argument, kill sends SIGTERM. After a process has ended, wait returns its exit code. If it was ended by a signal, that is 128 plus the number of the signal:

sleep 600 &
kill -TERM $!   # $! is the PID of the process last started in the background
wait $!; echo "Exit code: $?"
sleep 600 &
kill -KILL $!
wait $!; echo "Exit code: $?"

Exit code: 143
Exit code: 137

In between, the shell itself reports lines such as [1]+ Terminated sleep 600 or Killed; they are not part of the program’s output. These two numbers are worth remembering. 137 in a container log almost always means: the process was ended hard, either because it didn’t react to SIGTERM in time or because the container wanted more memory than it was allowed.

This is how container platforms end a container:

1. They send PID 1 a SIGTERM.
2. They wait out a grace period: docker stop 10 seconds, Azure Container Apps 30 seconds.
3. If the process is still alive then, SIGKILL comes.

For the SIGTERM to reach your application, it has to be PID 1 itself. That is why the Kladde’s Dockerfile has the exec form ENTRYPOINT ["dotnet", "Kladde.Web.dll"] with square brackets. The shell form ENTRYPOINT dotnet Kladde.Web.dll would put a shell in front as PID 1, and it doesn’t pass the signal on; in the chiseled image that shell wouldn’t even exist.

What .NET does with SIGTERM

ASP.NET Core and every other .NET host understand SIGTERM on their own. The host stops accepting new requests, gives running requests time, tells all background services through their CancellationToken that it’s over, and waits for that for HostOptions.ShutdownTimeout at most (default: 30 seconds). This small program shows it.

You create this module’s practice programs in a folder of their own outside the Kladde repository, for example with mkdir ~/k3-exercises && cd ~/k3-exercises. That way the Kladde’s build rules don’t apply there, and the exercises don’t end up in a pull request by accident. In the Codespace, create a file shutdown.cs there:

// shutdown.cs
#:sdk Microsoft.NET.Sdk.Web

var builder = WebApplication.CreateBuilder(args);
builder.Logging.AddFilter("Microsoft.AspNetCore", LogLevel.Warning);
builder.Services.AddHostedService<NightlyImport>();
builder.Services.Configure<HostOptions>(options => options.ShutdownTimeout = TimeSpan.FromSeconds(25));

var app = builder.Build();
app.Lifetime.ApplicationStopping.Register(() => app.Logger.LogInformation("Stopping: no new requests, open ones may finish."));
app.Lifetime.ApplicationStopped.Register(() => app.Logger.LogInformation("Stopped cleanly."));
app.MapGet("/", () => $"Process {Environment.ProcessId} is running.");
app.Run();

sealed class NightlyImport(ILogger<NightlyImport> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        logger.LogInformation("Import is waiting for its next run.");
        try
        {
            await Task.Delay(Timeout.Infinite, stoppingToken);
        }
        catch (OperationCanceledException)
        {
            logger.LogInformation("Import stopped before the process ends.");
        }
    }
}

The #:sdk line turns the single file into a web application. The grace period here is deliberately 25 seconds, below the 30 seconds of Azure Container Apps: the host should be done before the platform reaches for SIGKILL. Start the program in the first terminal:

dotnet run shutdown.cs -- --urls http://localhost:5080

In the second terminal, ask the application for its PID and send it a SIGTERM. Don’t ask ps about dotnet: dotnet run starts your program as a child process of its own, and kill should hit exactly that one.

curl http://localhost:5080/
kill -TERM 4711   # use the number from your answer instead of 4711

Process 4711 is running.

In the first terminal you then see:

info: NightlyImport[0]
      Import is waiting for its next run.
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://localhost:5080
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /home/codespace/k3-exercises
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
info: shutdown[0]
      Stopping: no new requests, open ones may finish.
info: NightlyImport[0]
      Import stopped before the process ends.
info: shutdown[0]
      Stopped cleanly.

PID and path will be different on your machine. The last four messages are the clean farewell: the host understood the signal, the background service saw its token and left before the process did. The exit code is 0.

With kill -KILL instead of kill -TERM, these four lines are missing entirely, and the exit code is 137. Whatever the import was just writing stays half written. That is why idempotency and the outbox from K2 2.8 matter so much: a process can end hard at any time, and the next instance still has to carry on correctly.

apt-get update && apt-get install -y --no-install-recommends procps
sleep 600 &
sleep 600 &
ps -o pid,args
kill -TERM %1; wait %1; echo "first: $?"
kill -KILL %2; wait %2; echo "second: $?"

    PID COMMAND
      1 bash
    310 sleep 600
    311 sleep 600
    312 ps -o pid,args
first: 143
second: 137

Environment variables

Every process has environment variables, pairs of name and value that it inherits from the process that starts it. Containers are configured almost only through them. In the .NET images a few are already set, among them APP_UID=1654, ASPNETCORE_HTTP_PORTS=8080 and DOTNET_RUNNING_IN_CONTAINER=true.

printenv HOME                 # read a variable
echo "$HOME"                  # the same inside a text
export GREETING=hello         # set it and pass it on to child processes
bash -c 'echo "Child sees: $GREETING"'
unset GREETING

/root
/root
Child sees: hello

ASP.NET Core reads environment variables as configuration. A double underscore stands for the colon of the configuration keys: ConnectionStrings__Kladde becomes ConnectionStrings:Kladde. That is how the compose.yaml from K2 told the Kladde its database.

A rule that will come up often in K3: Never print a container’s whole environment, so no bare env, no printenv without a name and no docker inspect with all variables into a ticket, a chat or a log. Connection strings and keys are often in there. If you want to know which variables are set, have only the names shown:

env | cut -d= -f1 | sort

Pipes and redirections

Linux tools are small and each does one thing. They become powerful when you connect them:

Every program has two outputs: the normal one (stdout) and the one for errors (stderr). If you write only >, you won’t find the error messages in the file, but still on the screen.

And the exit code? $? holds the one of the last command. For a pipe, that is the last command of the chain, not the first:

grep 'not-there' kladde.log | sort; echo "Exit code: $?"
set -o pipefail
grep 'not-there' kladde.log | sort; echo "Exit code: $?"

Exit code: 0
Exit code: 1

grep found nothing and ended with 1, sort sorted an empty input properly and ended with 0. Without pipefail, the whole chain looks successful. That is the same trap as in PowerShell when you pipe a command into Select-Object -First and then read $LASTEXITCODE: a script that decides on the exit code has to know whose exit code it is reading. bash scripts in a container therefore begin with set -euo pipefail.

grep, less and curl

For the examples in this section, create a file kladde.log with this content (in the Codespace: code kladde.log, paste the text, save). Each line is a request: time, method, path, status, duration.

08:14:02 GET /contacts 200 41ms
08:14:09 GET /appointments/book 200 63ms
08:14:11 POST /appointments/book 302 120ms
08:15:40 GET /tenders 200 1432ms
08:15:52 GET /contacts/17 404 12ms
08:16:03 GET /tenders 500 2051ms
08:16:30 GET /health 200 3ms
08:17:12 GET /tenders 500 2207ms
08:17:45 POST /appointments/book 500 88ms
08:18:01 GET /contacts 200 38ms
08:18:20 GET /health 200 2ms
08:19:05 GET /tenders 200 1187ms

grep finds lines that contain a pattern:

grep ' 500 ' kladde.log        # all lines with status 500
grep -c ' 500 ' kladde.log     # just count
grep -v ' 200 ' kladde.log     # all lines WITHOUT 200
grep -i 'POST' kladde.log      # ignoring case

08:16:03 GET /tenders 500 2051ms
08:17:12 GET /tenders 500 2207ms
08:17:45 POST /appointments/book 500 88ms
3
08:14:11 POST /appointments/book 302 120ms
08:15:52 GET /contacts/17 404 12ms
08:16:03 GET /tenders 500 2051ms
08:17:12 GET /tenders 500 2207ms
08:17:45 POST /appointments/book 500 88ms
08:14:11 POST /appointments/book 302 120ms
08:17:45 POST /appointments/book 500 88ms

The spaces in ' 500 ' matter: without them, grep would also find a duration of 1500 ms. With -E, grep understands extended regular expressions, such as ' 5[0-9]{2} ' for every status from 500 to 599.

With cut, sort and uniq this becomes a small analysis. cut -d' ' -f3 takes the third field, separated by spaces, the path. uniq -c counts identical lines that are directly one below the other; that is why you sort first:

cut -d' ' -f3 kladde.log | sort | uniq -c | sort -rn

      4 /tenders
      3 /appointments/book
      2 /health
      2 /contacts
      1 /contacts/17

less shows a long file page by page: less kladde.log. With the space bar you page, with /500 and Enter you search, n jumps to the next match, G to the end, q quits. tail -n 20 file shows the last 20 lines, tail -f file stays on it and shows new lines as soon as they come. For containers there is the same as docker logs --tail 20 -f kladde-web.

curl is the browser of the command line:

curl -i http://localhost:5080/                                   # answer with headers
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:5080/  # just the status
curl -fsS http://localhost:5080/health                           # errors as exit code

-s turns off the progress display, -S shows errors anyway, -o /dev/null throws the content away, -w prints values of the response. Most important for scripts is -f: without -f, curl ends with exit code 0 even on a 500 response, because the connection did work, after all. With -f it ends with 22 on any status from 400 up. If curl gets no connection at all, the exit code is 7. In PowerShell the counterpart is Invoke-WebRequest, which throws an error on its own on a status from 400 up.

grep -E ' 5[0-9]{2} ' kladde.log | wc -l
grep ' 500 ' kladde.log | cut -d' ' -f3 | sort | uniq -c | sort -rn | head -n 1
grep -E ' [0-9]{4,}ms$' kladde.log

3
      2 /tenders
08:15:40 GET /tenders 200 1432ms
08:16:03 GET /tenders 500 2051ms
08:17:12 GET /tenders 500 2207ms
08:19:05 GET /tenders 200 1187ms

grep -cE ' 5[0-9]{2} ' kladde.log; echo "Exit code: $?"
grep -cE ' 9[0-9]{2} ' kladde.log; echo "Exit code: $?"

3
Exit code: 0
0
Exit code: 1

Looking into a chiseled image

When the Kladde runs in its container, you can’t simply attach to it:

docker exec -it kladde-web bash

OCI runtime exec failed: exec failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown

There is no bash, not even sh. You don’t need a shell in the image anyway. Start a debug container next to it instead, one that shares the processes and the network of the Kladde container:

docker run --rm -it --pid container:kladde-web --network container:kladde-web ubuntu:24.04 bash

In it you install what you need (procps, curl), and then with ps -o pid,user,args you see the Kladde’s dotnet process as PID 1 with the user 1654, and curl -i http://localhost:8080/health asks the Kladde from the inside, the way the platform does. The tools exist only in the debug container and disappear with it. The Kladde’s image stays small.

Azure Container Apps offers the same as a debug console: az containerapp debug opens a container of its own with tools such as ps, wget, openssl, dig and traceroute next to your app. az containerapp exec, on the other hand, requires a shell in the image and fails with the Kladde just like docker exec. You use both from module 3.2 on, when the Kladde runs in Azure again.

# scripts/check-shutdown.ps1
# Stops the Kladde's container with SIGTERM and checks that it shut down cleanly: exit code 0 and the shutdown lines in its log.
[CmdletBinding(SupportsShouldProcess)]
param(
    [string] $Container = 'kladde-web',
    [int] $GraceSeconds = 30
)

if ($PSCmdlet.ShouldProcess($Container, "docker stop -t $GraceSeconds")) {
    docker stop -t $GraceSeconds $Container | Out-Null
    if ($LASTEXITCODE -ne 0) { throw "docker stop $Container failed." }
}

$exitCode = docker inspect $Container --format '{{.State.ExitCode}}'
if ($LASTEXITCODE -ne 0) { throw "docker inspect $Container failed. Does the container exist?" }
$log = (docker logs $Container 2>&1) -join "`n"

$checks = [ordered]@{
    'Exit code 0'                    = $exitCode -eq '0'
    'Host understood SIGTERM'        = $log.Contains('Application is shutting down...')
    'Import honored its token'       = $log.Contains('Import stopped')
}
foreach ($check in $checks.GetEnumerator()) {
    '{0,-32} {1}' -f $check.Key, ($check.Value ? 'ok' : 'MISSING')
}
if ($checks.Values -contains $false) { exit 1 }

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    using var timer = new PeriodicTimer(TimeSpan.FromHours(1), timeProvider);
    try
    {
        while (await timer.WaitForNextTickAsync(stoppingToken))
        {
            if (IsImportHour(timeProvider.GetLocalNow()))
            {
                await using var scope = scopeFactory.CreateAsyncScope();
                var import = scope.ServiceProvider.GetRequiredService<TenderImport>();
                await import.RunAsync(stoppingToken);
            }
        }
    }
    catch (OperationCanceledException)
    {
        logger.LogInformation("Import stopped because the host is shutting down.");
    }
}

builder.Services.Configure<HostOptions>(options => options.ShutdownTimeout = TimeSpan.FromSeconds(25));

services:
  web:
    stop_grace_period: 30s

Exit code 0                      ok
Host understood SIGTERM          ok
Import honored its token         ok

## Shutting down

- Platforms end the Kladde with SIGTERM, and after a grace period with SIGKILL:
  docker stop 10 s (compose: stop_grace_period 30s), Azure Container Apps 30 s.
- The Kladde waits 25 s at most (HostOptions.ShutdownTimeout), so below every grace period.
- The nightly import honors its CancellationToken and reports "Import stopped".
- To check: scripts/check-shutdown.ps1 (exit code 0, both log lines).
- Exit code 137 in the platform's log means: SIGKILL. Then check the import first.

The Kladde after this lesson

• The nightly import passes its CancellationToken on everywhere and reports when it stops because of the shutdown.
• Kladde.Web waits at most 25 seconds when shutting down, compose.yaml gives the container 30.
• scripts/check-shutdown.ps1 checks the clean shutdown and lives in the repository.
• docs/operations.md has a section “Shutting down”.

Commit that as a pull request, like every step of the Kladde.

What you can do now

• You move around a Linux shell, read permissions and set them with chmod and chown as tightly as necessary.
• You know why the .NET images run as user app with the number 1654 on port 8080.
• You install packages in an image and know why the Kladde’s runtime image has no shell.
• You find processes, end them with SIGTERM and tell from 143 and 137 how a process ended.
• You know the grace periods of Docker, Azure Container Apps and the .NET host and tune them to each other.
• You evaluate a log with grep, cut, sort and uniq and know whose exit code a pipe returns.
• You look into a container without a shell using a debug container.

Further reading

• Introduction to Bash: a Microsoft Learn module with the basic commands of the shell and exercises of its own.
• .NET container images: the kinds of images, chiseled and distroless.
• .NET Generic Host: Host shutdown: which signals the host understands and what it does when shutting down.
• Application lifecycle management in Azure Container Apps: SIGTERM and the 30 seconds until SIGKILL.
• Connect to a container debug console in Azure Container Apps: the debug console for images without a shell.