The travel assistant has a test set of 40 cases and evaluation in the build (R-10). Now there is the agent reiseassistent (version 3) in the Foundry Agent Service, and the questions are: Which version may go live? Does it get worse in operation? And when cases fail: what is the cause?
Prior knowledge. "AI track, lesson T9.1" builds the test set, "T9.2" measures with Microsoft.Extensions.AI.Evaluation, "T9.4" examines the judge model (biases, agreement with people) and gives an overview of Foundry's evaluation, "T9.5" puts the quality gate into the build, "T10.2" shows traces, "T10.4" drift and the switch of a model. Here: the Foundry side, that is built-in evaluators, runs in the project, observation in operation and the search for failure patterns.
The requirement
All numbers are assumptions of the exercise.
| No. | Requirement | Type | Measure | Weight | Source |
|---|---|---|---|---|---|
| 2.6-01 | The test set (40 cases, R-10) runs against every agent version before it goes live. | Q | Task Adherence: 90% of the cases pass | Must | Quality |
| 2.6-02 | No case of the test set violates the safety evaluators. | R | 0 failed safety cases | Must | Legal |
| 2.6-03 | Refund and legal questions never pass through a model: they are rules in code (R-02). | R | Unit tests, not model evaluation | Must | Legal |
| 2.6-04 | In operation, a sample of the real answers is evaluated continuously. | Q | Sample per hour: 20 runs | Should | Operations |
| 2.6-05 | Failed cases can be found by pattern, not only one by one. | Q | Pattern named within one working day | Should | Development |
Cost frame. Tokens per million per evaluated case and evaluator (judge model), tokens of the cluster analysis, Application Insights per gigabyte. Prices: pricing pages of Foundry and Azure Monitor. All evaluation data is made up (R-12).