With T11.2 and T11.3, every AI feature is finished on its own. This lesson works through issue 5, operations. It checks what no feature can prove on its own: that every feature leaves traces, that the cap from T10.3 applies to all of them, that every feature has a test set, and that all of this isn't green only on your machine. At the end stands a sentence from the README: "Clone, pwsh ./check.ps1, exit code 0." A new team member can run it without the cloud, without a key and without asking you.

The criteria of issue 5 and their tests:

Criterion Test or tool
Every model call leaves a span with tokens and without content outside development. EveryModelCallLeavesASpanWithItsFeatureAndTokensButNoContent
Once the month's cap is reached, no call goes to a model anymore; in the new month again. AtTheCapNoVisitorQuestionReachesTheModelUntilTheNextMonth
Every call costs in the month what the meter counts. TheMonthHoldsEveryCostTheMeterCounted
Every AI feature has a test set in the build. EveryFeatureHasATestSetOrAReasonButNotBoth, EveryTestSetNamedHereExistsAndIsJson
A fresh clone passes check.ps1 with exit code 0. check.ps1, tools/frischer-klon.ps1

The third criterion wasn't in the issue word for word. It's the condition for the second to mean anything: A cap that costs slip past holds nothing back.

Traces as acceptance

In T10.2 you looked at traces in the Aspire dashboard and checked with tests that the chat span knows its feature. The acceptance asks the same thing once more through the container, that is, with the pipeline the web app builds, and with the settings it runs with outside development. AcceptanceHost from T11.2 doesn't set Ai:CaptureContent, just like appsettings.json.

A span is caught by an ActivityListener from System.Diagnostics. It has a property that quickly makes such a test flaky: It listens to every source of that name in the whole process. xUnit runs test classes in parallel, and the acceptance tests from T11.2 create their own spans from the source Kladde.Ai at the same time. The first version of this test collected them all in a List<Activity> and ran green on its own. In the run of all tests, it failed like this:

failed Kladde.Ai.Tests.Acceptance.OperationsAcceptanceTests.EveryModelCallLeavesASpanWithItsFeatureAndTokensButNoContent (576ms)
  Xunit.MicrosoftTestingPlatform.XunitException: System.InvalidOperationException : Collection was modified; enumeration operation may not execute.

Two errors at once: The list was written to by several threads, and it contained spans of other tests. The version below fixes both. It collects in a ConcurrentQueue, starts a trace of its own and takes only the spans of this trace:

C#
// tests/Kladde.Ai.Tests/Acceptance/OperationsAcceptanceTests.cs
using System.Collections.Concurrent;
using System.Diagnostics;
using Kladde.Ai.Budget;
using Kladde.Ai.Metering;
using Kladde.Ai.Support;
using Kladde.Ai.Tenders;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Time.Testing;

namespace Kladde.Ai.Tests.Acceptance;

/// <summary>Issue 5, "Betrieb": traces, the monthly cap and the month's sum, through the Kladde's registrations.</summary>
[Trait("Abnahme", "Betrieb")]
public sealed class OperationsAcceptanceTests
{
    private const string Good = """{"suitability":"Suitable","reasons":["Wi-Fi for a school fits exactly"],"risks":[],"score":8}""";

    private const string TestSourceName = "Kladde.Ai.Tests.Acceptance";

    private static readonly ActivitySource TestSource = new(TestSourceName);

    private static readonly UsageDetails Usage = new() { InputTokenCount = 1900, OutputTokenCount = 350 };

    private static readonly TenderNotice SchoolNetwork = new(
        "T-0001", "Wi-Fi for Mühlbach Elementary School", "Municipality of Beispielstadt", "Beispielstadt", new DateOnly(2025, 3, 31),
        "Survey and setup of a Wi-Fi network in twelve classrooms.");

    private static CancellationToken Ct => TestContext.Current.CancellationToken;

    [Fact]
    public async Task EveryModelCallLeavesASpanWithItsFeatureAndTokensButNoContent()
    {
        // A listener hears every source of that name in the whole process, also the spans of tests that run at the same time.
        // The test therefore starts a trace of its own and keeps only the spans inside it.
        ConcurrentQueue<Activity> stopped = new();
        using var listener = new ActivityListener
        {
            ShouldListenTo = source => source.Name is AiServiceCollectionExtensions.TelemetrySource or TestSourceName,
            Sample = (ref ActivityCreationOptions<ActivityContext> _) => ActivitySamplingResult.AllDataAndRecorded,
            ActivityStopped = stopped.Enqueue,
        };
        ActivitySource.AddActivityListener(listener);
        var model = new FakeChatClient().Answer(Good, usage: Usage);
        using ServiceProvider provider = AcceptanceHost.Build(model);

        ActivityTraceId trace;
        using (Activity test = TestSource.StartActivity("acceptance")!)
        {
            trace = test.TraceId;
            await provider.GetRequiredService<TenderAssessor>().AssessAsync(SchoolNetwork, Ct);
        }

        Activity span = Assert.Single(stopped, s => s.TraceId == trace && s.Source.Name == AiServiceCollectionExtensions.TelemetrySource);
        Assert.Equal("chat", span.GetTagItem("gen_ai.operation.name"));
        Assert.Equal(AiFeatures.TenderAssessment, span.GetTagItem(AiFeatures.PropertyName));
        Assert.Equal(1900, span.GetTagItem("gen_ai.usage.input_tokens"));
        Assert.Equal(350, span.GetTagItem("gen_ai.usage.output_tokens"));
        Assert.DoesNotContain(span.TagObjects, tag => tag.Key.EndsWith(".messages", StringComparison.Ordinal) || tag.Key == "gen_ai.system_instructions");
    }

    [Fact]
    public async Task AtTheCapNoVisitorQuestionReachesTheModelUntilTheNextMonth()
    {
        var model = new FakeChatClient().Answer(SupportChatAcceptanceTests.Draft("termine-absagen#als-kunde"));
        using ServiceProvider provider = AcceptanceHost.Build(model, new FakeHelpSearch(SupportChatAcceptanceTests.Cancel));
        var clock = (FakeTimeProvider)provider.GetRequiredService<TimeProvider>();
        AiBudget budget = provider.GetRequiredService<AiBudget>();

        // The cap is 1 USD a month (AcceptanceHost); earlier calls of this month have used it up.
        await provider.GetRequiredService<IAiSpendStore>().AddAsync(BudgetRules.MonthOf(clock.GetUtcNow()), AiFeatures.TenderAssessment, 1m, Ct);

        SupportAnswer atTheCap = await AskAsync(provider);
        Assert.Null(atTheCap.Text);
        Assert.Empty(model.Requests);
        Assert.False(await budget.AllowsAsync(AiWork.Background, Ct));
        Assert.True(await budget.AllowsAsync(AiWork.Staff, Ct));

        clock.SetUtcNow(BudgetRules.StartOfNextMonth(clock.GetUtcNow()));
        SupportAnswer nextMonth = await AskAsync(provider);
        Assert.Equal(SupportOutcome.Answered, nextMonth.Outcome);
        Assert.Single(model.Requests);
    }

    [Fact]
    public async Task TheMonthHoldsEveryCostTheMeterCounted()
    {
        var model = new FakeChatClient()
            .Answer(Good, usage: Usage)
            .Answer(SupportChatAcceptanceTests.Draft("termine-absagen#als-kunde"), usage: new UsageDetails { InputTokenCount = 2600, OutputTokenCount = 250 });
        using ServiceProvider provider = AcceptanceHost.Build(model, new FakeHelpSearch(SupportChatAcceptanceTests.Cancel));

        await provider.GetRequiredService<TenderAssessor>().AssessAsync(SchoolNetwork, Ct);
        await AskAsync(provider);

        // 1,900 × 2 + 350 × 8 = 6,600 and 2,600 × 2 + 250 × 8 = 7,200 millionths of a dollar.
        decimal counted = provider.GetRequiredService<AiUsageMeter>().Snapshot().Values.Sum(usage => usage.CostUsd);
        Assert.Equal(0.0138m, counted);
        Assert.Equal(counted, (await provider.GetRequiredService<AiBudget>().StateAsync(Ct)).SpentUsd);
    }

    private static async Task<SupportAnswer> AskAsync(ServiceProvider provider)
    {
        using IServiceScope scope = provider.CreateScope();
        return await scope.ServiceProvider.GetRequiredService<SupportChat>().AskAsync("How do I cancel an appointment?", "en", Ct);
    }
}

On the first test:

  • The trace of its own. TestSource.StartActivity creates an activity, and because Activity.Current travels along with the await, the chat span of the assessment becomes its child and carries the same TraceId. Spans of other tests have different ones. Assert.Single with a condition takes only the one match and reports loudly if there are none or two.
  • The source's name as a constant. ShouldListenTo compares names. A comparison with the object TestSource failed in one attempt: The static field came into being only while the listener was registering, and the source didn't get it. StartActivity then returned null. With the name, the order doesn't matter.
  • Which contents there are. With Ai:CaptureContent set to true, UseOpenTelemetry writes the messages into the attributes gen_ai.input.messages and gen_ai.output.messages, and instructions from ChatOptions.Instructions, as the agent uses them, into gen_ai.system_instructions. The test demands that none of them is present. The tokens, on the other hand, always stand in gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, as numbers.

On the second: The test fills the month directly in the cap's store up to the cap, as earlier calls would have done. After that, the chat asks no model, background work isn't allowed, a person from the office is (T10.3). Then the test sets the clock to the start of the next month, and the same question reaches the model. Text is null at the cap: How your SupportAnswer otherwise names the reason depends on your version from T10.3; without text, the island shows the form.

On the third: Two features, two calls, and the meter's sum must be the month's sum to the cent. If the cap is missing from the pipeline, say because MeteringChatClient doesn't get it, the month stays at 0 and this test turns red, together with EveryCallCountsUnderTheFeatureAndIntoTheMonth from T11.2.