Since T2.3 the Kladde has been assessing tenders, and since T3.4 the prompt for it exists in three versions. Which one is better? And what happens when you switch to a different model tomorrow? Your unit tests from T2 don’t help with that question. They run against a fake client with prepared answers and show that the code around the model is right: the answer is read, a broken answer is repeated once, after that the result is “not assessed”. Whether the model recognizes a tender for building cleaning as unsuitable, they don’t show. That takes a test set.

In this module you build an evaluation for the tender assessment, step by step:

  • T9.1 (this lesson): the test set and the fixed checks that need no model.
  • T9.2: measuring quality with Microsoft.Extensions.AI.Evaluation, with an evaluator of your own and a judge model.
  • T9.3: reports and the response cache, so that a second run no longer needs a model.
  • T9.4: how far you can trust a language model as a judge.
  • T9.5: the evaluation in the build, so that a worse prompt version turns the pull request red.

After that the house rule of the Kladde holds: every AI feature has a test set with an evaluation in the build, and the exit code says whether it passes.

What a test set is

A test set is a fixed list of inputs with the result you expect. For the assessment these are invented tenders, each with the suitability and the score that an experienced person in the office would give. You send this list through the real AI feature, with the real model, and compare.

That sounds like a unit test, but it differs in two points:

  • The result varies. The same model can answer the same question differently (T1.1). A single red case therefore says little. What carries weight is the rate over many cases: 18 of 20 right.
  • What’s expected comes from the field, not from the code. What “suitable” means is decided by the office. The test set records that decision, and the prompt has to follow it, not the other way round.

From this follows a rule that is easily broken: write down what you expect before you ask the model. If you first read the model’s answer and then enter it as expected, all you check is whether the model agrees with itself.