Middleware: measuring, caching, logging

The Kladde now asks a model in three places: in the conversation on the console, in the island in the browser, and in the assessment of tenders. With a local model, that costs only time. As soon as a model in the cloud answers, every call costs money, and sooner or later someone wants to know: Which feature uses how much? How long do users wait? What went wrong last night?

Not every place should collect the answers to these itself. That’s what middleware is for: layers that wrap around the IChatClient and see every call that passes through them. In this lesson you build the Kladde’s pipeline. It counts tokens and costs per AI feature, writes telemetry for the Aspire dashboard, logs without prompts ending up in logs, and caches answers where that helps. With that, the first house rule applies to AI features: No model call without a meter. These packages are new:

Layers around the client

You know the principle from ASP.NET Core (K2.3): A request runs through a chain of middleware, and each one can do something before and after. With Microsoft.Extensions.AI, it’s the same. A ChatClientBuilder takes the actual client and wraps one layer after another around it with Use…. At the end, Build() returns an IChatClient that looks like any other from the outside.

Which layer lies outside is shown by this example without a model. Two layers write down when they come up before and after the call:

// order.cs
#:package Microsoft.Extensions.AI@10.11.0

using System.Runtime.CompilerServices;
using Microsoft.Extensions.AI;

List<string> trail = [];
IChatClient client = new ChatClientBuilder(new Model(trail))
    .Use(async (messages, options, next, cancellationToken) =>
    {
        trail.Add("A before");
        await next(messages, options, cancellationToken);
        trail.Add("A after");
    })
    .Use(async (messages, options, next, cancellationToken) =>
    {
        trail.Add("B before");
        await next(messages, options, cancellationToken);
        trail.Add("B after");
    })
    .Build();

await client.GetResponseAsync("Hello?");
Console.WriteLine(string.Join(" → ", trail));

// Stands in for the model and notes when it is asked.
sealed class Model(List<string> trail) : IChatClient
{
    public Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default)
    {
        trail.Add("Model");
        return Task.FromResult(new ChatResponse(new ChatMessage(ChatRole.Assistant, "Sure.")));
    }

    public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        ChatResponse response = await GetResponseAsync(messages, options, cancellationToken);
        foreach (ChatResponseUpdate update in response.ToChatResponseUpdates())
        {
            yield return update;
        }
    }

    public object? GetService(Type serviceType, object? serviceKey = null) => null;

    public void Dispose()
    {
    }
}

A before → B before → Model → B after → A after

The first layer you attach lies on the outside. It sees every call first and the answer last. From that follows what arrives in which layer: A layer outside the cache sees every call, including the ones the cache answers. A layer inside sees only the ones that really go to the model. With this in mind, you’ll choose the Kladde’s order in a moment.

This form of Use with next is the shortest. It’s enough when a layer only does something before and after. If you want to read the answer, such as its usage, you write a class of your own.