Kladde.Verlauf needs an interface that writes and reads appointments. Before lesson 2.1 writes the first query, the client comes first: the class CosmosClient, which carries all of the SDK's connections, retries and caches. Almost all errors in operation that are "Cosmos DB's fault" are the fault of its setup.

Prior knowledge. Managed identity and Key Vault are in K3, roles and key-free accounts in lesson 3.1 and (the disableLocalAuth switch) in lesson 1.1. The account of the lesson has F-01: cosmos-kladde, database kladde, main region germanywestcentral, no key.

What was run. All programs of this lesson were compiled with Microsoft.Azure.Cosmos 3.64.0 (latest stable version on the day of writing), Newtonsoft.Json 13.0.4, Azure.Identity 1.21.0 and Microsoft.Extensions.DependencyInjection and ran without an account: The client is created but never contacted. Not run against Azure is everything that needs a response from the service (the sign-in, the regions, a real 429).

One client, one account

Learn calls it the first rule: one CosmosClient per account for the lifetime of the application. At startup it fetches metadata (databases, containers, partitions, addresses) and keeps connections open; a new client per request repeats that every time. Add the limit from lesson 1.1: Management calls share 240 RU/s per account and cannot be raised. For the same reason you take the names of database and container from the configuration and call CreateIfNotExists only at setup.

All tenants (offices) of an account run through the same client, because a tenant is a value of the partition key. You need one client per account only if each tenant has its own account; Learn then advises limiting the number of clients per machine. All calls are asynchronous; blocking with .Result or .Wait() leads to thread pool starvation, according to Learn.

Gateway or Direct

  • Direct (the default in the .NET SDK) talks to the replicas directly over TCP. According to Learn that is the faster choice, but it needs ports in the range 10000 to 20000 (0 to 65535 with private endpoints); if they are closed, the client reports 503.
  • Gateway talks HTTPS over port 443 to an intermediate node, with one more hop. It is the choice behind strict firewalls or with few permitted sockets, and it is the mode of the emulator (K3).

The run confirms the default: A freshly created CosmosClientOptions reports ConnectionMode: Direct, RequestTimeout: 00:00:06, and ConsistencyLevel and MaxRetryAttemptsOnRateLimitedRequests as not set (null). null means here: The SDK default applies, and according to Learn that is 9 retries, so up to 10 attempts per request.

Regions

Run the application in the same Azure region as the account. With several regions in the account, you set ApplicationRegion (one region) or ApplicationPreferredRegions (a list in order). According to Learn, on a failover writes go to the current write region and reads to the first reachable region of the list. F-01 names one main region; the second region in the example below (North Europe) is an assumption of the exercise. The names are the constants of the class Regions (Regions.GermanyWestCentral gave the text Germany West Central in the run). Open: which second region Kladde.Verlauf really gets is decided by lesson 3.2.

Retries and 429

Whoever consumes more than is provisioned gets 429 with the wait time in the header x-ms-retry-after-ms. The SDK waits that time and tries again; Learn names 9 retries and a maximum of 30 seconds of total waiting. Only after that does your code see the error. Two consequences:

  • 429 in the portal does not mean an error in the application: The SDK has usually caught it. Learn considers 1 to 5 percent of 429 at acceptable latency a healthy sign that the throughput is being used to the full (with evenly distributed partitions).
  • The wait time is read from the error object: CosmosException.RetryAfter. Whoever wants to carry on after the SDK has given up waits that time.

You do not retry what does not change: A 413 (item larger than 2 MB, F-08) stays 413. This helper code retries only 429 and was checked against a stand-in:

C#
public static async Task<T> WithRetryAfterAsync<T>(Func<Task<T>> call, int maxAttempts, TimeSpan fallback, Func<TimeSpan, Task> delay)
{
    for (var attempt = 1; ; attempt++)
    {
        try { return await call(); }
        catch (CosmosException ex) when (ex.StatusCode == HttpStatusCode.TooManyRequests && attempt < maxAttempts)
        {
            await delay(ex.RetryAfter ?? fallback);
        }
    }
}

The wait function is passed in so that the test does not really wait. A call that fails twice with 429 and succeeds the third time, and one that fails with 413, gave:

result ok after 3 attempts, waited 250 ms, 250 ms
413 after 1 attempt(s)

The 250 ms are the fallback: A self-built CosmosException has no header with RetryAfter. With a real 429, the service's time is there (that is not run against Azure).

Time limits: According to Learn, you get a total duration per call only with RequestTimeout and a CancellationToken together.

Signing in with Entra

No key in code or in settings (F-01). Instead of the key, the client takes a TokenCredential; DefaultAzureCredential tries several sources in turn, in Azure the managed identity (K3). The identity needs a role on the data plane of the account (built in: Cosmos DB Built-in Data Reader and Data Contributor; for creating and assigning, Learn names rights on sqlRoleDefinitions and sqlRoleAssignments). Locally, development runs against the emulator, which checks no sign-in, with the connection from the user secrets (K3, lesson 3.6.4).

Registration with dependency injection

C#
public static IServiceCollection AddKladdeCosmos(this IServiceCollection services, CosmosSettings settings)
{
    services.AddSingleton(provider =>
    {
        var options = new CosmosClientOptions
        {
            ConnectionMode = ConnectionMode.Direct,
            ApplicationPreferredRegions = settings.PreferredRegions,
            EnableContentResponseOnWrite = false,
            UseSystemTextJsonSerializerWithOptions = new JsonSerializerOptions(JsonSerializerDefaults.Web),
            MaxRetryAttemptsOnRateLimitedRequests = 9,
            MaxRetryWaitTimeOnRateLimitedRequests = TimeSpan.FromSeconds(30),
        };
        return new CosmosClient(settings.Endpoint, provider.GetRequiredService<TokenCredential>(), options);
    });
    return services;
}

What each line wants: EnableContentResponseOnWrite = false saves memory and network with many writes, because the service does not send the created item back (the RU header and diagnostics remain; according to Learn the response of a trigger is also dropped). The last two lines set the default values explicitly; you set them only to change them. UseSystemTextJsonSerializerWithOptions has your types written with System.Text.Json; with the web settings Id becomes id, which Cosmos DB requires.

Newtonsoft.Json remains mandatory. The SDK uses it internally for its own types, even if you set System.Text.Json, and fails the build if you do not reference it yourself. Learn recommends version 13.0.4 (at least 13.0.4 from SDK 3.54.0) and warns against 10.x:

XML
<PackageReference Include="Microsoft.Azure.Cosmos" Version="3.64.0" />
<PackageReference Include="Newtonsoft.Json" Version="13.0.4" />

The test without an account gets the client twice from the container and asks for the settings. It gave:

same instance twice:  True
connection mode:      Direct
preferred regions:    Germany West Central, North Europe
429 retries / wait:   9 / 00:00:30
content on write:     False

Diagnostics

Every response and every CosmosException carries Diagnostics: all attempts, regions and wait times of this one call. Learn warns against parsing the text, because its structure changes with every version. Log it as a whole, and only for slow calls:

C#
if (diagnostics.GetClientElapsedTime() > threshold)
{
    write($"{operation}: {diagnostics}");
}

With a stand-in for CosmosDiagnostics (a class that overrides GetClientElapsedTime and ToString), one call of 8 ms and one of 450 ms were checked against the threshold of 100 ms; the line for the fast one stayed out:

log lines: 1: slow query: {"elapsedMs":450}

Exercise 1

Part A, without an account. Create client.cs with AddKladdeCosmos, Resilience and SlowCallLog (packages in the #:package lines at the top of the program).

  1. Do not leave the regions in the code but have them come from the configuration: read CosmosSettings from IConfiguration, with Endpoint and PreferredRegions as settings.
  2. Add a test that checks that WithRetryAfterAsync gives up after maxAttempts and passes the 429 on.
  3. Why is a new CosmosClient(...) in a method that runs per request dangerous? Name two consequences with a number.

Part B, with a test account (not run against Azure): Replace Endpoint with that of your account, give yourself the Data Contributor role and read an item. Cleaning up: remove the role assignment.

How to check it: Part A ends with exit code 0; the second test shows 429 after the last retry.

Solution 1
  1. The Cosmos section in appsettings.json holds Endpoint (no key!) and PreferredRegions as a list; services.Configure<CosmosSettings>(configuration.GetSection("Cosmos")) and the registration reads IOptions<CosmosSettings>. Regions and endpoint differ per environment, which is why they do not belong in the code.
  2. The test lets call always fail with 429 and checks that the exception comes out of WithRetryAfterAsync after maxAttempts calls and that attempts == maxAttempts holds. The frame is in the code above: catch (CosmosException ex) when (… && attempt < maxAttempts) lets the last attempt through.
  3. For example: Every new client fetches the metadata again, and management calls share 240 RU/s per account; in addition, every client opens its own connections (on average four per physical partition, according to Learn), which fills ports and memory.

Part B stays unchecked.

What you can do now

  • You set up one CosmosClient per account as a singleton and register it with dependency injection.
  • You choose Direct or Gateway and set preferred regions.
  • You distinguish what the SDK does by itself on 429 and what your code must do, and you do not retry 413.
  • You sign in with Entra, set System.Text.Json and keep Newtonsoft.Json as a mandatory reference.
  • You log diagnostics selectively and test the client code without an account.

Further reading