The group-travel team is running, and the planner approves or changes drafts (M-02). Every change is a hint, but it appears in no test set. Lesson 3.1 showed how a team is evaluated; this lesson asks how evaluations become an improvement without the team getting worse on the side. There are four building blocks: the judge model, which makes evaluations cheap; synthetic data, which fills gaps in the test set; optimizers, which deliver suggestions for prompts; and the planner's feedback, which brings real errors.
Prior knowledge. "AI track, lesson T9.4" measures the judge model against people, "T10.4" shows drift, the switch by configuration and the shadow run, "AI-103, lesson 2.6" the running sample (rule with an upper limit per hour) and "AI-103, lesson 2.7" the rule to change behavior only after a measurement. "Lesson 3.1" supplies check plan and review sheet. None of that is repeated; what is new is the loop that connects everything, and its limits.
New in this lesson. (1) The test set M-12 (60 cases) stays unchanged; new is an extension set for synthetic cases and reviewed cases from production. It never counts for acceptance, only for the loop. (2) The change gate (assumptions): gain of at least 0.03 of the score, at most 10% more tokens, no violated hard rule, at most 3 rounds per occasion. (3) The source text of the prompts stays in the files from lesson 2.1; a suggestion is adopted there, not in the service.
The requirement
All numbers are assumptions of the exercise.
| No. | Requirement | Type | Measure | Weight | Source |
|---|---|---|---|---|---|
| 3.2-01 | A change is adopted only on a measurable gain. | R | gain of at least 0.03 | Must | AI-103 2.7 |
| 3.2-02 | No hard rule is violated by an improvement (legal, A3, tenant separation, the 10 malicious cases). | R | 0 violated cases | Must | M-05, M-07, M-12 |
| 3.2-03 | A loop has a round limit and a token limit. | R | 3 rounds, +10% tokens | Must | R-09 |
| 3.2-04 | The planner's feedback is reviewed within one week. | Q | 100% of the thumbs down | Should | Operations |
| 3.2-05 | No case of the extension set contains data of a real customer or a mandantId. |
R | 0 hits | Must | M-07, M-12 |
Cost frame. Cost drivers: tokens for agents, judge and optimizer in every round (calls grow with candidates, rows and evaluators), time of the planner. Prices: pricing page of Foundry.