The group-travel team of Nordlicht Reisen GmbH has a test set of 60 cases, 10 of them malicious (M-12), and F-06 demands three things: the test set in the build, human review in Foundry, and samples from production with feedback from the planner. This lesson settles the first two and asks what is actually evaluated in a team. An overall grade for the quote draft does not say whether the memory delivered another customer, the index found the wrong clause or a tool received wrong arguments. The samples and the improvement loop follow in lesson 3.2.
Prior knowledge. "AI-103, lesson 2.6" shows Foundry's built-in evaluators (agent, quality, safety), the run against an agent version, the running sample and the search for failure patterns. "AI track, lesson T9.1" builds the test set, "T9.2" measures with Microsoft.Extensions.AI.Evaluation, "T9.4" measures the judge model against people (agreement, biases), "T9.5" puts the quality gate into the build. None of that is repeated. What is new is the view of a team: several agents, several sources, one run.
New in this lesson. (1) Every case of the test set additionally carries the list erwarteteWerkzeuge (names as in lesson 1.1). (2) The planner's review sheet has four questions (below). (3) Before publication, two people review a sample of 20 of the 60 cases (assumption of the exercise). The test set itself and F-06 stay unchanged.
The requirement
All numbers are assumptions of the exercise.
| No. | Requirement | Type | Measure | Weight | Source |
|---|---|---|---|---|---|
| 3.1-01 | For every kind of change (prompt, tool description, index, memory) there is a check plan saying which checks run before release. | F | plan per kind of change | Must | F-06 |
| 3.1-02 | Two people review 20 of 60 cases before a publication. | Q | 20 cases, 2 reviewers | Should | F-06 |
| 3.1-03 | No draft and no memory hit contains data of another customer. | R | 0 hits in the test set | Must | M-07 |
| 3.1-04 | The expected tools of a case are called. | Q | 90% of the cases | Should | M-04 |
| 3.1-05 | A failed run can be traced to one agent. | F | partial result per agent | Must | M-03 |
Cost frame. Cost drivers: tokens of the judge model per case and evaluator, time of the reviewers, Application Insights per gigabyte. Prices: pricing pages of Foundry and Azure Monitor. All data is made up (M-12).