The assessment of tenders is running. It has a test set, an evaluation in the build, traces and a cap. Even so, it can be worse tomorrow than today without anyone changing a line: the tenders change, a model is retired and replaced, or a new, cheaper model beckons. In this lesson, you learn to notice such changes and to switch model or prompt in a way that goes through only if the quality stays and the costs drop, and that can be undone with a single value.

Everything in this lesson runs without the cloud: the drift check and the approval rule as pure code with tests, the shadow run with two local models. There are no new packages.

What changes without anyone changing anything

A test set measures a feature on one day with one model. After that, three things can shift:

  • The data. In the fall, different tenders arrive than in the spring, a new procurement platform words its texts differently, or the office looks for new kinds of contracts. The test set from T9.1 doesn't know these cases, and the pass rate in the build stays green while reality looks different.
  • The model. When a model version is retired, Azure moves deployments with OnceCurrentVersionExpired (T10.1) to the next version. Locally, after an update, foundry model download may fetch a new build under the same alias. In both cases, a different model answers the same prompt.
  • The expectation. The office learns what it wants from the assessment. What was right three months ago as "partially suitable" it now sees as "unsuitable".

The same thing helps against all three: measure continuously, on real assessments, and before every switch, check on the test set, in the shadow.