The group-travel team of Nordlicht Reisen GmbH runs with seven agents, three MCP servers, a queue and a Container App (M-03, M-04, M-10). After weeks a run can become slower, more expensive or worse without anything failing: a new prompt, a different deployment, a supplier with a new response. The lesson asks how you notice before the trip planner does, and what to do then. It builds on the trace convention from lesson 1.4.
Prior knowledge. Traces and Application Insights: "AI track, lesson T10.2"; drift and model change: "T10.4"; measurement plan, alert on 429: "AI-103, lesson 1.3"; observing Foundry and Application Insights: "AI-103, lesson 2.8"; Application Insights: "K3, lesson 3.4.2", dead letter: "K3, lesson 3.5.5". The convention from "lesson 1.4", the annotations from "3.1" and the times from "3.3" apply and are not repeated.
New in this lesson. (1) Every agent span additionally carries agentVersion (from the configuration, F-07). (2) The regression rule (assumptions): at least 30 runs of the new version, 95th percentile at most 1.25 times the old one, error rate at most 5 percentage points more. (3) Two alert levels: warning (trip planner and operations) and outage (operations immediately). (4) Prompts and responses are never in traces in production (T10.2).
The requirement
All numbers are assumptions of the task.
| No. | Requirement | Type | Measure | Weight | Source |
|---|---|---|---|---|---|
| 3.4-01 | Every run has exactly one trace; every span carries mandantId, laufId, agent, agentVersion. |
F | 100% of own spans | Must | M-11, 1.4 |
| 3.4-02 | An outage of the team (error rate of the runs, growing queue) is reported within 15 minutes. | Q | 15 minutes | Must | Operations |
| 3.4-03 | A new agent version with a regression is recognized in the canary before it gets all runs. | Q | Rule met, before 100% | Must | F-07 |
| 3.4-04 | Drift is measured monthly by quality, tokens and feedback. | Q | 1 report per month | Should | T10.4 |
| 3.4-05 | Every alert has a remedy and an owner. | F | Table complete | Must | Operations |
| 3.4-06 | No contents of prompts or responses in traces of production; mandantId only as a key. |
R | 0 contents | Must | M-07, T10.2 |
Cost frame. Cost drivers: Application Insights per gigabyte (traces, metrics, logs), queries and alert rules, retention. Prices: the pricing page of Azure Monitor.