M-12 gives the group-travel team a test set of 60 cases, ten of them malicious. It is written by hand, and that means: it finds what its authors could imagine. The AI Red Teaming Agent of Foundry generates attacks automatically and in large numbers and thereby promises to find risks before an agent version reaches customers ("shift left" in Learn). The lesson asks: What can it do for our team, what can it not do according to the documentation, and how do you fit it into the release process?
Prior knowledge. Prompt injection and attack sets: "AI track, lesson T6.1" and "T6.2". Hard rules in code: "AI track, lesson T6.4". Evaluation in the build: "AI track, lesson T9.5". Risk and safety evaluators, region and limits of the evaluation: "AI-103, lesson 1.5". Reading Python: "AI-103, lesson 0.3". The design of the guardrails follows in lesson 4.4, their tests in 4.5, the pipeline in 4.7.
New in this lesson. The scans run in a test environment (in Learn: "purple environment", a non-production environment with production-like resources): resource group rg-ai500-rotteam, Foundry resource with project gruppenreisen-rotteam. This is not a fifth environment of the case and does not change M-09. The agent versions from gruppenreisen are copied there; the region is open (decision 2). Every scan carries the name of the agent and its version.
The requirement
All numbers are assumptions of the task.
| No. | Requirement | Type | Measure | Weight | Source |
|---|---|---|---|---|---|
| 4.3-01 | Every new agent version is scanned before acceptance. | R | 7 agents, 100 % of new versions | Must | F-05 |
| 4.3-02 | Every successful attack is read by a person. | R | 100 % | Must | Quality |
| 4.3-03 | Every confirmed finding becomes a test case and, where needed, a rule. | R | 0 findings without a test case | Must | M-12 |
| 4.3-04 | Scans use only made-up data, nothing from operation. | R | 0 real data | Must | M-12, R-12 |
| 4.3-05 | Scans run in the EU. | R | region in the EU | Must | M-08 |
| 4.3-06 | After release a scheduled scan runs. | Q | monthly | Should | F-06 |
Cost drivers: model calls of the attacks and of the evaluation per run, the test environment (Foundry resource, deployments). Prices: the price pages.