What a language model does

The AI track starts with this lesson. You bring the Kladde from K2: a web application with contacts, appointment booking, the tenders from the TED interface, sign-in with Entra ID and two languages. Over the course of the track it gets an assessment of tenders, a support chat, an agent for offers and an MCP server. Before you build the first of these features, this module takes a close look at the tool itself: what a language model actually does, what it costs, which model suits which job, and where the limits are that the law and common decency draw.

The whole track runs locally first. Today you set up Foundry Local, Microsoft’s runtime for models on your own computer. You don’t need an Azure subscription for that, and there is no cost per request. Where a later lesson shows the cloud, that part is optional and runs in your own subscription.

A language model continues text

A language model doesn’t answer questions, at least not the way a database answers a query. It does something simpler: it gets a text and estimates how that text most likely continues. Then it appends a piece and estimates again, with the lengthened text. That’s how an answer comes about, piece by piece.

The pieces are called tokens. A token is often a whole short word, but often just part of a word, a punctuation mark, or a space together with the beginning of a word. The next lesson shows exactly how a model splits text into tokens. For today it’s enough to know that the model works in tokens, not in letters and not in words.

At every step, the model gives every token it knows a number that says how well it fits at this spot. These numbers are called logits. They become a probability distribution, and the next token is drawn from that. That a model often writes correct things is because correct sentences were frequent in its training data. It has no knowledge that it looks things up in.

From word to probability, in miniature

The following program replays this one step, with four words instead of a hundred thousand tokens. The scores are made up; the calculation is the same as in the big model. For the AI track, create a folder ki-track next to your Kladde, and in it the file next-word.cs:

// next-word.cs – how a language model picks the next word, in miniature.
using System.Globalization;

// After "The appointment with Ms. Brandt is on" the "model" allows these words.
// The numbers are its scores (logits): the higher, the better the fit.
string[] words = ["Monday", "Tuesday", "Friday", "Moon"];
double[] logits = [2.0, 1.5, 1.0, -1.0];

foreach (double temperature in new[] { 0.2, 1.0, 2.0 })
{
    double[] p = Probabilities(logits, temperature);
    string line = string.Join("  ", words.Select((w, i) => $"{w} {Math.Round(p[i] * 100)}%"));
    Console.WriteLine($"Temperature {temperature.ToString("0.0", CultureInfo.InvariantCulture)}: {line}");
}

Console.WriteLine();
var random = new Random(7); // fixed seed: every run draws the same words
foreach (double temperature in new[] { 0.0, 1.0 })
{
    var drawn = Enumerable.Range(0, 8).Select(_ => words[Pick(logits, temperature, random)]);
    Console.WriteLine($"Eight draws at {temperature.ToString("0.0", CultureInfo.InvariantCulture)}: {string.Join(", ", drawn)}");
}

// Softmax with temperature: scores become probabilities that add up to 1.
static double[] Probabilities(double[] logits, double temperature)
{
    double[] weights = logits.Select(l => Math.Exp(l / temperature)).ToArray();
    double sum = weights.Sum();
    return weights.Select(w => w / sum).ToArray();
}

// Temperature 0 means: always the most probable word. Otherwise we draw according to the probabilities.
static int Pick(double[] logits, double temperature, Random random)
{
    if (temperature == 0)
    {
        return Array.IndexOf(logits, logits.Max());
    }

    double[] p = Probabilities(logits, temperature);
    double r = random.NextDouble();
    for (int i = 0; i < p.Length; i++)
    {
        r -= p[i];
        if (r < 0)
        {
            return i;
        }
    }

    return p.Length - 1;
}

dotnet run next-word.cs

Temperature 0.2: Monday 92%  Tuesday 8%  Friday 1%  Moon 0%
Temperature 1.0: Monday 49%  Tuesday 30%  Friday 18%  Moon 2%
Temperature 2.0: Monday 38%  Tuesday 30%  Friday 23%  Moon 9%

Eight draws at 0.0: Monday, Monday, Monday, Monday, Monday, Monday, Monday, Monday
Eight draws at 1.0: Monday, Friday, Tuesday, Monday, Monday, Tuesday, Monday, Friday

The percentages are rounded, so the ones in a line don’t always add up to exactly 100. The program writes the temperature with a period, the way you write it in a request to a model.

Temperature and randomness

Temperature divides the scores before they become probabilities. That has a simple effect:

• A low temperature amplifies the differences. At 0.2, “Monday” gets 92 percent and the rest almost nothing. The answers become uniform and predictable.
• Temperature 1 leaves the distribution as the model calculated it.
• A high temperature flattens the differences. At 2, even “Moon” has 9 percent. The answers become more varied, and nonsense becomes more likely.
• Temperature 0 is a special case: you can’t divide by 0. What’s meant is to always take the token with the highest score. That’s why Pick handles the case on its own.

That explains why the same question gets different answers: at any temperature above 0, a token is drawn, and not once but at every token. A different token in third place leads to a different sentence, and that to a different answer. The program above draws the same words in every run only because new Random(7) has a fixed seed. Some models offer a parameter seed for this. Still, don’t rely on a model writing exactly the same thing word for word at the same temperature and the same seed: in its documentation for Azure OpenAI, Microsoft explicitly says that seed only improves repeatability and doesn’t guarantee it.

Besides temperature there is top_p: the model then draws only among the most probable tokens that together reach that share. At top_p 0.9, the least probable tokens that together make up just 10 percent drop out. Change either the temperature or top_p, not both at once, or you won’t know what had an effect.

Not every model accepts these two settings. According to Microsoft, reasoning models in the cloud, including the gpt-5.4 family you meet in T1.3, support neither temperature nor top_p; there you set how much the model thinks before it answers instead. Local models such as qwen2.5-0.5b accept the temperature, which is why you try it there in this lesson.

What does that mean for the Kladde? An assessment of tenders should say the same thing today as tomorrow, so it runs at a low temperature where the model accepts one. A text suggestion for a cover letter may be more varied. And because neither a low temperature nor a different model gives a guarantee, the code checks every result before it’s saved. You learn that in T2.

The context window

A model sees only what’s in the request: the instruction, the previous messages, the data you send along. This text plus the answer must fit into the context window, a fixed number of tokens per model. For gpt-5.4-mini, for example, Microsoft gives 400,000 tokens, of which at most 272,000 are for the input and 128,000 for the output. Small local models have much smaller windows; the number is in the details of the model.

Three consequences of this come up throughout the track:

• A model remembers nothing. Between two requests it has no memory. A chat feels like a conversation only because the application sends the whole history along again with every question.
• Long conversations get expensive. Because the history goes along every time, you pay for it again with every question, in the cloud as tokens.
• What doesn’t fit is missing. If the request is too long, the service rejects it, or the application has to shorten it beforehand. A model that has never seen part of a tender can’t assess it, and it doesn’t notice.

Confident and wrong

A model writes what sounds probable. Most of the time that’s also correct, but not always, and it sounds equally sure in both cases. Invented but fluently delivered statements are often called hallucinations. Typical cases:

• What can’t exist: an invented law, a prize that was never awarded, a book by a made-up person. The model knows the shape of such statements and fills it in.
• What happened after training: every model has a cutoff of its training data. Ask about something later and it guesses.
• Exact numbers and quotations: case numbers, section numbers, phone numbers, page numbers. They look plausible and are often wrong.
• Tasks that cut across tokens: counting letters or writing words backward. The model sees no letters, only tokens.

For the Kladde this becomes a rule of the house rules that you’ll see again in every module: What must never happen is in the code, not in the prompt. A number that’s in no source is filtered out by the code. An email is sent by the code only after a human has approved it. The model may make suggestions; your program decides and checks.

Setting up Foundry Local

Foundry Local has two parts you can use separately:

• the command-line tool foundry for browsing the catalog, downloading and trying things out. It exists for Windows and for macOS on Apple silicon; both versions are currently a preview.
• the SDK Microsoft.AI.Foundry.Local, a NuGet package for your programs. It runs on Windows, macOS and Linux, doesn’t need the tool, and downloads models itself. On Linux, including in a codespace, you work with the SDK only.

Plan on at least 8 GB of memory and a few gigabytes of free disk space. More memory is better, because the model sits entirely in memory while it answers. You don’t need a graphics card; Foundry Local picks a version for the CPU, the graphics card or an AI accelerator to suit your hardware.

On Windows, install the tool with winget:

winget install Microsoft.FoundryLocal

On macOS with Homebrew:

brew tap microsoft/foundrylocal
brew install foundrylocal

Afterward, close the terminal and open it again so that foundry is found. Check the installation and look at the catalog:

foundry --version
foundry model list

On the first foundry model list, Foundry Local downloads the execution providers for your hardware, which takes a moment. After that you see the models that can run on your computer. Three terms matter:

• The alias is the short name of a model, such as qwen2.5-0.5b or phi-4-mini. With the alias, Foundry Local itself picks the version that suits your hardware.
• The model ID names exactly one of these versions, with device and version number. foundry model list --verbose shows it together with the license, and foundry model list --variants shows all versions of an alias. The topmost version of an alias is the one Foundry Local takes.
• The Task column says what a model is good for. If it says tools, the model can call tools; you need that in T7.

You filter with --type, for example foundry model list --type chat for chat models and --type embedding for the models you need for search in T4.

Try a small model directly in the terminal. The course takes qwen2.5-0.5b as its first model: it’s small, loads fast, runs on almost any computer, and Microsoft’s examples for Foundry Local use it as well.

foundry chat qwen2.5-0.5b

On the first start, Foundry Local downloads the model. Then you type a question and press Enter. You end the conversation with /exit. A model with half a billion parameters is small; it writes fluently but is often wrong. For this lesson, that’s exactly what’s useful.

You’ll need two more commands often: foundry cache list shows which models are on disk, and foundry cache remove <alias> frees the space again.

The file-based app ask.cs

For the exercises you get a finished program. It uses the SDK, loads the model, starts Foundry Local’s built-in web service at a fixed address and sends the same question ten times: five times with temperature 0, five times with temperature 1. It talks to the web service through the OpenAI package, that is, through the interface that the models in the cloud offer as well. From T2 on, the Kladde builds on exactly that.

Create ask.cs in the ki-track folder:

// ask.cs – asks the same question five times with temperature 0 and five times with temperature 1.
#:package Microsoft.AI.Foundry.Local@2.3.1
#:package OpenAI@2.15.0

using System.ClientModel;
using Microsoft.AI.Foundry.Local;
using Microsoft.Extensions.Logging.Abstractions;
using OpenAI;
using OpenAI.Chat;

const string Alias = "qwen2.5-0.5b";
const string Url = "http://127.0.0.1:52495";
string question = args.Length > 0 ? string.Join(' ', args) : "What is the capital of Australia?";

var config = new Configuration
{
    AppName = "kladde",
    Web = new Configuration.WebService { Urls = Url },
};
await FoundryLocalManager.CreateAsync(config, NullLogger.Instance);
FoundryLocalManager manager = FoundryLocalManager.Instance;
try
{
    // On Windows the SDK fetches the execution providers for your hardware here; elsewhere the call returns at once.
    await manager.DownloadAndRegisterEpsAsync();
    ICatalog catalog = await manager.GetCatalogAsync();
    IModel model = await catalog.GetModelAsync(Alias)
        ?? throw new InvalidOperationException($"The model '{Alias}' is not in the catalog.");

    // Downloads the model only the first time; after that it comes from the cache.
    await model.DownloadAsync(progress => Console.Write($"\rDownload {Alias}: {progress,5:F1}%"));
    Console.WriteLine();
    await model.LoadAsync();
    await manager.StartWebServiceAsync();

    // From here on it's the same code as against a model in the cloud: the OpenAI package, just a different address.
    var client = new OpenAIClient(new ApiKeyCredential("local"), new OpenAIClientOptions { Endpoint = new Uri(Url + "/v1") });
    ChatClient chat = client.GetChatClient(model.Id);

    Console.WriteLine($"Model: {model.Id}");
    Console.WriteLine($"Question: {question}");
    foreach (float temperature in new[] { 0f, 1f })
    {
        var options = new ChatCompletionOptions { Temperature = temperature, MaxOutputTokenCount = 120 };
        Console.WriteLine();
        Console.WriteLine($"Temperature {temperature}");
        for (int i = 1; i <= 5; i++)
        {
            ChatCompletion answer = await chat.CompleteChatAsync([new UserChatMessage(question)], options);
            string text = answer.Content.Count > 0 ? answer.Content[0].Text : "(no answer)";
            Console.WriteLine($"{i}: {text.ReplaceLineEndings(" ").Trim()}");
        }
    }

    await manager.StopWebServiceAsync();
    await model.UnloadAsync();
}
finally
{
    manager.Dispose();
}

What happens in it:

• The two #:package lines fetch the packages from nuget.org, with a fixed version. Every lesson in this track names the versions its examples were built with.
• FoundryLocalManager.CreateAsync starts the runtime in your own process. NullLogger.Instance means: we’re not interested in the SDK’s log lines here. GetModelAsync(Alias) looks the model up in the catalog and picks the version for your hardware.
• StartWebServiceAsync opens an address on your computer only (127.0.0.1). Other computers on the network can’t reach it.
• The “key” local is a placeholder. The OpenAI package insists on one, and Foundry Local doesn’t check it. Real keys never go into code, not later in the cloud either: there the Kladde signs in without a key (T10).
• MaxOutputTokenCount = 120 limits the length of each answer. Without a limit, a small model can keep writing for a long time.

Start the program with a question. Everything after -- goes to the program:

dotnet run ask.cs -- "What is the capital of Australia?"

The first start takes a few minutes: the SDK downloads the model. After that it comes from the cache, and the download line stays empty. The output has this form; the model ID and the answers themselves depend on your hardware and on chance:

Download qwen2.5-0.5b: 100.0%
Model: <the model ID of the version for your computer>
Question: What is the capital of Australia?

Temperature 0
1: …
2: …
3: …
4: …
5: …

Temperature 1
1: …
2: …
3: …
4: …
5: …

If the program reports that the address is in use, something else is already running there. Then change the number in Url, for example to 52496.

Don’t send real people, companies or customer data to a model, not even a local one. That’s part of the Kladde’s house rules. For the exercises, questions about general knowledge and invented cases are enough.

string[] words = ["Monday", "Tuesday", "Friday", "Weekend", "Moon"];
double[] logits = [2.0, 1.5, 1.0, 0.5, -1.0];

foreach (double temperature in new[] { 0.5, 1.0, 5.0 })

Temperature 0.5: Monday 64%  Tuesday 24%  Friday 9%  Weekend 3%  Moon 0%
Temperature 1.0: Monday 44%  Tuesday 27%  Friday 16%  Weekend 10%  Moon 2%
Temperature 5.0: Monday 25%  Tuesday 23%  Friday 20%  Weekend 18%  Moon 14%

Eight draws at 0.0: Monday, Monday, Monday, Monday, Monday, Monday, Monday, Monday
Eight draws at 1.0: Monday, Friday, Tuesday, Monday, Monday, Tuesday, Monday, Weekend

The Kladde: record the local model

From today the Kladde has a document about its models. It grows over the next lessons: in T1.3 the choice for each feature is added, in T10 the comparison with a classic model. In your Kladde repository, create a branch and in it docs/model-choice.md:

# Model choice

## Local model for development

- Runtime: Foundry Local, SDK Microsoft.AI.Foundry.Local 2.3.1
- Alias: qwen2.5-0.5b
- Model ID on my computer: <output of ask.cs, line "Model:">
- License: <from foundry model list --verbose>
- Computer: <operating system, memory, graphics card if any>
- Set up on: <date>

## Observations

- Temperature 0: <what exercise 1 showed>
- Sounding sure and wrong: <the question from exercise 2 and the answer>

The files next-word.cs and ask.cs stay in your exercise folder ki-track, not in the Kladde. In T2.1 the Kladde gets a project of its own for its AI features. Submit the document as a pull request, like every step, the way you’ve been used to since K1.

What you can do now

• You explain that a language model continues text token by token and draws from a probability distribution as it does.
• You know what temperature does, why temperature 0 is a special case, and why the same question gets different answers.
• You know that the context window limits input and output together, and that a model retains nothing between two requests.
• You count on a model saying false things with confidence, and you decide that rules go in the code.
• You’ve set up Foundry Local, know alias, model ID and the Task column, and have a local model answer from C#.

Read more

• How generative AI and LLMs work: Microsoft’s overview for .NET developers, with tokens, embeddings and temperature.
• Get started with Foundry Local: the quickstart with the SDK, for Linux and macOS too.
• Foundry Local CLI reference: all foundry commands, with the filters for foundry model list.
• Integrate inference SDKs with Foundry Local: the web service and the OpenAI package, as in ask.cs.