Streaming responses

A local model on an ordinary computer writes an answer of ten sentences in a few seconds, and on a laptop without a graphics card sometimes in half a minute. With GetResponseAsync from the last lesson, you see nothing during that time. For an assessment in the background, that doesn’t matter. But anyone sitting in front of an input field takes a silent program for crashed after a few seconds.

Language models generate their answer token by token anyway (T1.1). They can pass on every piece as soon as it’s there. That’s called streaming, and in this lesson you build it into the Kladde: on the console, in tests without a model, and in the web interface. The packages are the same as in lesson 1.

The answer in pieces

GetStreamingResponseAsync takes the same messages and options as GetResponseAsync. Instead of a finished answer, it delivers an IAsyncEnumerable<ChatResponseUpdate>, a sequence whose elements arrive one after another. You read it with await foreach: The loop looks like a foreach, but waits for each element with await, just as you waited for a single task in K1.9. Every ChatResponseUpdate is a piece of the answer:

• Text is the text of this piece, often just a word or part of one.
• Contents holds all the contents of the piece. Usually it’s a TextContent. With many providers, the last piece of an answer carries a UsageContent with the token usage and no text.
• FinishReason appears only in one of the last pieces, often one without text; in all the others it’s empty.

So that you can try this without a model, the following example has a small class in the model’s place. It fulfills IChatClient, answers word by word, and waits 150 milliseconds before every word. The example asks the same question twice, the second time with a time limit that runs out before the end of the answer:

// stream-without-model.cs
#:package Microsoft.Extensions.AI@10.11.0

using System.Diagnostics;
using System.Runtime.CompilerServices;
using Microsoft.Extensions.AI;

IChatClient client = new SlowModel("A tender publicly announces a contract to bidders.");

foreach (TimeSpan limit in new[] { TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(700) })
{
    using var cancel = new CancellationTokenSource(limit);
    long start = Stopwatch.GetTimestamp();
    TimeSpan? first = null;
    List<ChatResponseUpdate> updates = [];
    try
    {
        await foreach (ChatResponseUpdate update in client.GetStreamingResponseAsync("What is a tender?", cancellationToken: cancel.Token))
        {
            first ??= Stopwatch.GetElapsedTime(start);
            updates.Add(update);
            Console.Write(update.Text);
        }

        Console.WriteLine();
        Console.WriteLine($"First piece after {first?.TotalMilliseconds:0} ms, done after {Stopwatch.GetElapsedTime(start).TotalMilliseconds:0} ms");
    }
    catch (OperationCanceledException)
    {
        Console.WriteLine(" …");
        Console.WriteLine($"Canceled after {Stopwatch.GetElapsedTime(start).TotalMilliseconds:0} ms");
    }

    ChatResponse whole = updates.ToChatResponse();
    Console.WriteLine($"Pieces: {updates.Count}, together: \"{whole.Text.TrimEnd()}\"");
    Console.WriteLine();
}

// Stands in for a model: answers word by word, 150 ms per word.
sealed class SlowModel(string answer) : IChatClient
{
    public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        foreach (string word in answer.Split(' '))
        {
            await Task.Delay(150, cancellationToken);
            yield return new ChatResponseUpdate(ChatRole.Assistant, word + " ");
        }
    }

    public Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default) =>
        GetStreamingResponseAsync(messages, options, cancellationToken).ToChatResponseAsync(cancellationToken);

    public object? GetService(Type serviceType, object? serviceKey = null) => null;

    public void Dispose()
    {
    }
}

dotnet run stream-without-model.cs

On your machine, the milliseconds will differ by a few:

A tender publicly announces a contract to bidders.
First piece after 164 ms, done after 1278 ms
Pieces: 8, together: "A tender publicly announces a contract to bidders."

A tender publicly announces  …
Canceled after 716 ms
Pieces: 4, together: "A tender publicly announces"

Read the result closely:

• The first word is there after a little over a tenth of a second, the whole answer after more than a second. Whoever is watching experiences the short time.
• In the second pass, the time limit runs out after 700 milliseconds. Task.Delay notices that from the token and throws an OperationCanceledException, which the await foreach passes on to your catch. What came until then is there: four pieces.
• ToChatResponse puts the pieces back together into a whole answer, with text, role and usage. For a sequence you haven’t read yet, there’s ToChatResponseAsync; the SlowModel class uses it to also make an answer without a stream out of the stream.
• The class is also an example of how to write an IAsyncEnumerable yourself: async, yield return and the attribute [EnumeratorCancellation], so that the token from GetStreamingResponseAsync also arrives in the body.