Quality goals

In the last lesson you wrote down what your Kladde is today. Before you rebuild it, you need a second list: what it should be good at. Not which functions, you know those, but how reliable, how fast, how secure and how affordable. This list is called quality goals. It is the yardstick against which every decision in the following modules is measured. Without it, no decision can be justified, only asserted.

At the end of the lesson, docs/quality-goals.md is in the Kladde: quality goals with numbers, an order with reasons, and a first goal that a test in the build checks. One of the goals is “several offices in one Kladde”. It won't be fully met until the final project, but it is fixed from today on.

Functions say what, quality goals say how well

“Customers book an appointment” is a function. Whether the booking answers in half a second or in eight seconds, whether it keeps working when the TED interface is down, and whether a customer sees other people's appointments while doing it, the function doesn't say. Yet exactly these questions determine the architecture:

• Two systems with the same functions can be built completely differently, because one has to be reachable every second and the other may rest at night.
• A function can almost always be added later. A quality often can't: Whoever built a system without tenant isolation doesn't add it in one pull request.

That's why the quality goals come before the big decisions, and why they should be discussed with the subject-matter experts, not just within the team. How long booking may be down is decided by the office owner, not by the developer. The developer tells her what each answer costs.

The five pillars and two additions

The Azure Well-Architected Framework sorts the qualities of a cloud system into five pillars. They are a good checklist so that no quality is forgotten while you think:

The pillars look at the running system. For a system that is developed further for years, two qualities are missing that live in the code. The standard for software quality, ISO/IEC 25010, lists both under maintainability:

• Changeability: Can you make a change in one place without ten others having to follow? A new field on the contact should need one pull request in one part of the system, not in all of them.
• Testability: Can you check a rule without a database, network and browser? Does the whole test suite run fast enough that you start it before every push?

The pillars have a property that is more important than their list: They pull in different directions. That is what the section further down is about.

A goal you can check: the quality scenario

“The Kladde should be fast” is not a quality goal. Nobody can say whether it is met, and so no decision can be justified with it. Microsoft's guide on performance targets is clear: Avoid words like “fast” or “slow” and set numbers.

A proven form for this is the quality scenario. It describes a situation and the expected reaction of the system in five parts:

As a sentence: “When a signed-in employee searches for a contact in normal operation (up to 20 concurrent requests, 5,000 contacts), the web application shows the first page of hits in 95% of searches in at most 300 ms on the server, measured over 24 hours.”

The environment is the part that is easiest to forget, and one of the most important. The same system should behave differently in normal operation than during a deploy, under load or when a dependency fails. A scenario for reliability therefore almost always has an environment like “the TED interface hasn't answered for two hours” or “a replica is terminated in the middle of a request”.

Three rules make a scenario usable:

• A number somebody measures. If nobody can collect the number, it is decoration. Write down who or what measures it.
• A number somebody is accountable for. The 300 ms come from a conversation with the subject-matter expert (“If it takes more than half a second, I type a second time”), not from the developer's gut.
• A number that is achievable. A goal that is missed from the start is soon ignored by everyone. Better an honest goal that you tighten later.

How a scenario is checked

For every scenario, the question has to be answered: How do we notice that it is violated? Four kinds of checks cover almost everything:

The test in the build is the strongest check, because it can't be forgotten. A test that checks a quality of the system instead of a function is often called a fitness function in the literature. You'll build one in a moment.

Measuring with percentiles

Response times are distributed unevenly: Most requests are fast, a few are very slow, for example when the cache is cold, the database is just waking up or a replica is restarting. The mean hides exactly these slow requests. Microsoft's guide therefore recommends percentiles instead of means. The 95th percentile (p95) is the time that 95% of all requests don't exceed.

A small program shows the difference using 40 measured response times of the contact search. Put it in a practice folder outside the Kladde as p95.cs:

// p95.cs: mean, median and 95th percentile from measured response times of the contact search
double[] durationsMs =
[
    82, 95, 101, 77, 120, 88, 93, 110, 450, 99,
    85, 132, 91, 104, 780, 97, 89, 115, 72, 108,
    96, 1400, 87, 125, 100, 94, 610, 83, 118, 92,
    106, 79, 950, 98, 112, 86, 103, 520, 90, 117,
];

var sorted = durationsMs.Order().ToArray();
Console.WriteLine($"Measurements:     {sorted.Length}");
Console.WriteLine($"Mean:             {durationsMs.Average():F0} ms");
Console.WriteLine($"Median:           {Percentile(sorted, 50):F0} ms");
Console.WriteLine($"95th percentile:  {Percentile(sorted, 95):F0} ms");
Console.WriteLine($"Maximum:          {sorted[^1]:F0} ms");

// Nearest-rank method: the smallest measured value at which at least p percent of all measurements are equal or smaller.
static double Percentile(double[] sorted, int p)
{
    var rank = (int)Math.Ceiling(p / 100.0 * sorted.Length);
    return sorted[Math.Max(rank, 1) - 1];
}

dotnet run p95.cs

Measurements:     40
Mean:             201 ms
Median:           99 ms
95th percentile:  780 ms
Maximum:          1400 ms

With the mean of 201 ms, the goal “at most 300 ms” would be met. The 95th percentile shows the truth: At least every twentieth search takes 780 ms or longer, and it is exactly these searches that the employee notices. The median of 99 ms says how it feels most of the time; the p95 says how often it annoys.

In operation, you don't calculate this yourself. Application Insights stores every request with its duration, and KQL has the function percentile. A query for the scenario above, as you've been filing them under ops/queries/ since K3:

// ops/queries/contact-search-p95.kql: contact search, 95th percentile of the last 24 hours
AppRequests
| where TimeGenerated > ago(24h)
| where AppRoleName endswith "web"
| where Name has "contacts"
| summarize Requests = sum(ItemCount), P95 = percentile(DurationMs, 95) by Name

Remember the rule from K3: First check with AppRequests | summarize count() by AppRoleName whether your filter finds anything at all. A query that finds nothing reports no problem, and that looks exactly like a goal that is met. What the contact search's requests are called in your system, you see with summarize count() by Name.

A goal as a test: the fitness function

Since K2, the house rules have said: The business rules live in Kladde.Core, and EF Core is an adapter that the core doesn't know. That is a quality goal of changeability: If the core knows nothing about database, web and cloud, it can be changed and tested without them. So far only the review checks that. A test checks it with every pull request.

In tests/Kladde.Core.Tests, create the file QualityGoalTests.cs:

using Kladde.Core;

namespace Kladde.Core.Tests;

// Quality goal Q6 in docs/quality-goals.md: The business rules depend on no framework for database, web or cloud.
public sealed class QualityGoalTests
{
    private static readonly string[] ForbiddenPrefixes =
    [
        "Microsoft.EntityFrameworkCore",
        "Microsoft.AspNetCore",
        "Microsoft.Data.SqlClient",
        "Azure.",
    ];

    [Fact]
    public void TheCoreUsesNoFrameworkForDatabaseWebOrCloud()
    {
        var used = typeof(Contact).Assembly.GetReferencedAssemblies().Select(assembly => assembly.Name ?? "");

        var violations = used.Where(name => ForbiddenPrefixes.Any(prefix => name.StartsWith(prefix, StringComparison.Ordinal)));

        Assert.Empty(violations);
    }
}

typeof(Contact).Assembly is the assembly of Kladde.Core; any type from it would do. GetReferencedAssemblies returns the assemblies whose types the core actually uses. If the core is clean, the test is green. --filter-class from K3 starts only this one class so that the output stays short (here and below only its end):

dotnet test --project tests/Kladde.Core.Tests --filter-class "*QualityGoalTests"

Test run summary: Passed!
  total: 1
  failed: 0
  succeeded: 1
  skipped: 0
  duration: 2s 218ms

Now the counter-test, so that you know the test finds something. Add the package Microsoft.EntityFrameworkCore.Abstractions to the core for the experiment, and put an attribute from it on the contact, as happens in many projects:

using Microsoft.EntityFrameworkCore;

namespace Kladde.Core;

[Index(nameof(Email))]
public sealed record Contact(int Id, string Name, string? Email);

Your Contact has more properties; for the experiment, one that carries an attribute is enough. The test turns red and names the assembly that doesn't belong in the core (shortened by paths and call stack):

failed Kladde.Core.Tests.QualityGoalTests.TheCoreUsesNoFrameworkForDatabaseWebOrCloud (77ms)
  Assert.Empty() Failure: Collection was not empty
  Collection: ["Microsoft.EntityFrameworkCore.Abstractions"]

Test run summary: Failed!
  total: 1
  failed: 1
  succeeded: 0
  skipped: 0
  duration: 1s 809ms
Test run completed with non-successful exit code: 2 (see: https://aka.ms/testingplatform/exitcodes)

Afterwards, take the package and the attribute out again; the test is green again.

You should know one limit of this test: The compiler records only assemblies whose types the code really uses. A package that is in the project but isn't used anywhere isn't found by the test. That is intended here, because the goal is that the core doesn't use any of it. tools/deps.cs from the last lesson shows you the unused reference. In module 4.3, this one test grows into an architecture test that checks the allowed directions between all modules.

Not everything is possible at once

Whoever wants every quality at its maximum gets a system that is never finished and can't be paid for. The pillars pull in different directions, and Microsoft describes, for each pillar, its trade-offs with the others. Some that affect the Kladde:

That's why quality goals need an order. It decides what applies when two goals contradict each other. Two questions help with ordering:

1. How bad is it if the goal is missed, and can it be made good again? A slow search annoys people. An office that sees another office's contacts loses trust, and that can't be taken back.
2. How hard is the goal to retrofit later? Performance can often be improved afterwards, with an index or a cache. Tenant isolation, changeability and testability, hardly.

For every place in the order, write down what you give up for it. An order without this sentence is a wish list. With it, it is an agreement that later decisions can refer to: “For now we won't use a private endpoint for the database. Q5 says we'll only build it once the threat model demands it, and that's how we agreed.”

A goal for later, from today

The Kladde has so far belonged to a single office. In K4's final project, it should serve several offices that never see each other. Why is that already in the quality goals now, months before it's built?

Because it is exactly the kind of quality that is hard to retrofit. Tenant isolation affects every key, every query, every store and every message. If it's built in at the end, everything has to be touched again. If it's fixed as a goal today, then from module 4.3 on every new key carries a tenant ID, even though there is only one until the final project. K4's house rules say: The tenant ID comes from the token, never from the request.

The scenario for this is a security goal, and it gets a number like any other: zero foreign records, across all read and write places.

# Quality goals of the Kladde

As of: start of K4. The order is at the end. Every number has a check; what isn't checked yet has an issue.

## Q1 Several offices that never see each other (Security)

- Stimulus: A signed-in person from office A calls a page, an endpoint, a search or an export, even with an
  identifier that belongs to office B.
- Affected part: all read and write places.
- Environment: normal operation, at least two offices with data.
- Response: The system shows and changes only data of office A; a foreign identifier is treated as if it didn't exist (404).
- Measure: 0 foreign records in the isolation test suite; every read and write place is included in it.
- Check: test in the build, still open (issue #41, built in the final project). Until then, every new key carries a tenant ID.

## Q2 Response time of the working pages (Performance)

- Stimulus: A signed-in person searches for a contact or opens the weekly overview of appointments.
- Affected part: Kladde.Web.
- Environment: normal operation, up to 20 concurrent requests, 5,000 contacts, 2,000 appointments.
- Response: The page is shown.
- Measure: p95 at most 300 ms on the server over 24 hours.
- Check: measurement, ops/queries/contact-search-p95.kql; alert at p95 above 300 ms within an hour.

## Q3 Booking despite an outage of the TED interface (Reliability)

- Stimulus: The TED interface doesn't answer or answers with errors.
- Affected part: TED fetch, worker, public booking.
- Environment: outage of up to 48 hours.
- Response: Booking keeps working without restriction. After the outage, the fetch catches up on everything.
- Measure: 0 failed bookings because of TED; within 24 hours after the end of the outage, all notices are there, none twice.
- Check: integration test TenderFetchCatchesUpAfterOutage against an imitated service; alert on dead-letter messages.

## Q4 Delivering and rolling back (Operations)

- Stimulus: A pull request is merged into main and approved for prod.
- Affected part: pipeline, all programs.
- Environment: normal operation, while customers are booking.
- Response: The new version goes live without manual steps, without a running booking being lost. On an error, the previous
  revision becomes active again.
- Measure: at most 30 minutes from approval to live; rollback in at most 10 minutes.
- Check: measurement of the run time in the pipeline; drill of the rollback once per module, log in docs/operations.md.

## Q5 Cost per office (Cost)

- Stimulus: A month comes to an end.
- Affected part: all resources of the Kladde.
- Environment: normal operation with today's number of offices.
- Response: The cost per office is known.
- Measure: cost per office and month at most the target value in docs/cost.md; it falls when an office is added.
- Check: measurement in Cost Management, monthly; budget with alert (lesson 4.1.5).

## Q6 The core knows no framework (Changeability)

- Stimulus: A developer changes a business rule or switches the database.
- Affected part: Kladde.Core.
- Environment: development.
- Response: The business rules change only in the core; a switch of the database hits only Kladde.Data.
- Measure: Kladde.Core uses 0 assemblies of EF Core, ASP.NET Core, SqlClient or Azure.
- Check: test in the build, QualityGoalTests.TheCoreUsesNoFrameworkForDatabaseWebOrCloud.

## Q7 Fast tests without a network (Testability)

- Stimulus: A developer starts the unit tests before a push.
- Affected part: all test projects except integration and end-to-end.
- Environment: a typical development computer without a network connection.
- Response: All tests run through.
- Measure: at most 60 seconds, exit code 0, no test needs a network, database or browser.
- Check: measurement of the step's run time in the pipeline; still open: a run without a network in the build (issue #42).

## Order

1. Q1 Several offices that never see each other. A data leak between offices can't be taken back and ends trust.
   What we give up for it: Every query and every key becomes a little more awkward, and some shortcuts (a query across all
   offices) don't exist.
2. Q6 The core knows no framework. K4 rebuilds the Kladde; without a clean core, every rebuild gets expensive. What we give up for it:
   We write mappings between model and database ourselves instead of putting attributes on the classes.
3. Q7 Fast tests without a network. They secure every rebuild. What we give up for it: The rules are cut so that they run without a
   database, even if a SQL query would sometimes be shorter.
4. Q3 Booking despite an outage. Booking is what customers see. What we give up for it: The import is a path of its own with
   its own monitoring instead of a simple method in the web process.
5. Q4 Delivering and rolling back. What we give up for it: Every change to the database and messages has to be backward compatible, because old
   and new version briefly run at the same time.
6. Q5 Cost per office. A small office doesn't pay for high availability it doesn't need. What we give up for it: no
   zone redundancy, no private endpoints, as long as the threat model doesn't demand them.
7. Q2 Response time. The Kladde has few concurrent users; performance can be improved later with an index and a cache. What we
   give up for it: We optimize only when the measurement misses the goal.

The Kladde after this lesson

Two pull requests, each green on its own:

1. docs/quality-goals.md with the scenarios and the order. Let the subject-matter expert read the numbers; where she disagrees, her number applies, and you write down what that costs.
2. QualityGoalTests in tests/Kladde.Core.Tests and the query ops/queries/contact-search-p95.kql.

For every scenario with the note “still open”, create an issue and enter its number.

What you can do now

• You tell the five pillars of the Well-Architected Framework apart and add changeability and testability to them.
• You write quality goals as scenarios with stimulus, affected part, environment, response and a number in the measure.
• For each goal you decide whether a test, a measurement, a drill or a review checks it.
• You judge response times with the median and the 95th percentile instead of the mean.
• You bring a quality goal into the build as a fitness function and verify the test with a counter-test.
• You order goals by damage and retrofittability and write down what each place costs.

Further reading

• Azure Well-Architected Framework pillars: the five pillars with their principles and, per pillar, the trade-offs with the others.
• Architecture strategies for defining performance targets: performance targets with numbers and percentiles.
• Architecture strategies for defining reliability targets: targets for availability and recovery, the terms SLO, RTO and RPO.