After T9.2 the Kladde measures the assessment over the whole test set. Two things still get in the way. Every run asks the model and the judge anew: that takes time, and because models vary, every run looks a little different even if you changed nothing. And after the run, everything is gone except a table. Whether it was better yesterday, you can’t see.

The package Microsoft.Extensions.AI.Evaluation.Reporting solves both. It stores every result on disk, keeps the answers of model and judge in a response cache and, with the tool dotnet aieval, generates a report over several runs. In this lesson you build it in and prove that a second run with an unchanged prompt no longer needs a model.

Scenario, run, iteration

The library arranges results on three levels:

  • A scenario is a case that you evaluate, for the Kladde a case of the test set. Its name is also the name of a folder. Keep it short and without characters that are forbidden in file names. Dots in the name divide the report into a tree view: Kladde.TenderAssessment.T01 becomes the levels “Kladde”, “TenderAssessment” and “T01”.
  • A run (execution) is one pass over all scenarios. Its name holds the results of one pass together. Give it one. Without a name, every run is called Default, and every run overwrites the previous one.
  • An iteration is a repetition of the same scenario in the same run; the default is 1. With three iterations you see how strongly the model varies on a case. Every iteration has its own place in the cache.

A ReportingConfiguration holds this together. It knows the evaluators, the judge, the place for results and cache and the name of the run. For a folder on disk, DiskBasedReportingConfiguration.Create creates it. From the configuration you get a ScenarioRun for every case. It evaluates, and when it is disposed, it writes its result to the store.