A flaky test is one that is sometimes green and sometimes red without anything having changed in the code. It is worse than a red one: a red test tells the truth. A flaky one teaches the team to restart red runs until they are green, and eventually a real error drowns in the noise. You already know some of the most common causes: tests that share data (lesson 3.8.2), and ports that turn out not to be free (the next lesson). This lesson is about the other two: time and concurrency.

Field report: two timers that come due at the same time

From practice, without names: a service collected messages and sent them off 250 milliseconds after the first one was collected. Its test kicked off the service, waited 750 milliseconds and then checked whether the message was out. Three times as long as necessary; that seemed safe. On the developers' machines the test was always green. In the pipeline it was red in four of 55 runs. More load on the processor did not make the error visible: 24 runs on two cores with 64 busy threads were all green.

The cause was not a slow processor but a standing process. When the whole process halts, for example for a full garbage collection or because the agent's virtual machine pauses briefly, no timers run. If the halt lasts longer than 750 milliseconds, afterward both timers are due, the service's and the test's, and they run in any order, often at the same time on different threads. If the test's continuation comes first, it checks before the service has sent.

For the Kladde the same case looks like this: the confirmations of an appointment booking are collected briefly so that several quick bookings result in one round of mail.

C#
// src/Kladde.Core/ConfirmationBatcher.cs
namespace Kladde.Core;

public interface IMailSender
{
    void Send(IReadOnlyList<string> addresses);
}

// Collects confirmations and sends them off 250 ms after the first one was collected.
public sealed class ConfirmationBatcher(IMailSender mail, TimeProvider time)
{
    public static readonly TimeSpan Pause = TimeSpan.FromMilliseconds(250);

    private readonly Lock _lock = new();
    private readonly List<string> _pending = [];
    private ITimer? _timer;

    public void Add(string address)
    {
        lock (_lock)
        {
            _pending.Add(address);
            _timer ??= time.CreateTimer(_ => Flush(), null, Pause, Timeout.InfiniteTimeSpan);
        }
    }

    private void Flush()
    {
        string[] batch;
        lock (_lock)
        {
            batch = [.. _pending];
            _pending.Clear();
            _timer?.Dispose();
            _timer = null;
        }

        mail.Send(batch);
    }
}

The class is already built correctly: it receives its clock as a TimeProvider and builds its timer with time.CreateTimer. The error is made only by the test, which passes TimeProvider.System and waits on the real clock:

C#
// not like this: the test waits on the real clock and hopes
[Fact]
public async Task ABookingIsConfirmedSoonAfterwards()
{
    var mail = new RecordingMailSender();
    var batcher = new ConfirmationBatcher(mail, TimeProvider.System);

    batcher.Add("mara.feld@example.com");
    await Task.Delay(TimeSpan.FromMilliseconds(750), TestContext.Current.CancellationToken);

    Assert.Single(mail.Batches);
}