Until now the evaluation starts when you think of it. In this lesson it becomes part of the build: every pull request shows whether the assessment of the tenders is still as good as before, and a worse prompt version turns it red. For this the build costs nothing and needs no model. It uses the response cache from T9.3, and you set the thresholds in code.

With that, the house rule of the Kladde holds in full: every AI feature has a test set with an evaluation in the build, and the exit code says whether it passes.

Two stages

The evaluation consists of two kinds of checks, and both run on every pull request:

  • Fixed checks without a model: the checks of the test set and of the answers from T9.1, the tests of your evaluator from T9.2 and the trial run with the trial clients. They take seconds and check that the measurement itself is right.
  • The model assessment: all 20 cases through the assessment, with evaluators and thresholds. In the build, every answer comes from the cache that you put into the repository. No model, no costs, and every run with the same answers gives the same result.

What happens when you change the prompt? Then the key in the cache no longer fits, and without a model there is no answer. The build turns red and says so. For every change to prompt, model or test set, a run with a model is therefore needed before the pull request can turn green. There are three ways for this run:

  1. Locally before the pull request, with Foundry Local. This is the way of this course. The run writes new entries into the cache, you commit them, and the build assesses with exactly these answers. In the pull request everyone sees from the new files in the cache that a run with a model took place.
  2. On a runner of your own with Foundry Local. GitHub can run jobs on a computer that you register yourself. For a public repository GitHub advises against it: a pull request from a fork could run arbitrary code there.
  3. With a model in the cloud, in your own subscription with a small budget and sign-in without a key through a federated identity (K3, T10.1). Every run then costs tokens.

Which way proves itself in everyday work shows in practice. For the Kladde the first is enough.