Since K2, the Kladde has been fetching new tenders every night from the EU's TED interface. A BackgroundService in Kladde.Web calls
TED, translates the response and writes the notices to the database with an upsert. Since module 3.6, the fetch also stores every
raw notice as JSON in the blob container tenders-raw. That works as long as everything goes well. In this module you rebuild
the import so that it also works correctly when something goes wrong: a fetch that only fetches, a queue in between and a
worker that processes at its own pace.
This first lesson doesn't write any code against Azure yet. It clarifies the terms the next six lessons work with, and it ends with a sketch of the new import in your repository.
What's wrong with the nightly import today
Walk through a night in your head in which not everything goes smoothly:
- The import runs in the web process. If
Kladde.Webscales to three replicas in Container Apps, the background service runs three times. The upsert from K2 prevents duplicate rows, but TED gets three fetches and the database does triple the work. - A restart interrupts it. If the pipeline rolls out a new revision at 2 a.m., the old process ends in the middle of the import. What was already saved stays; what was still missing only arrives the next night.
- A slow database slows everything down. For every notice, the fetch waits until SQL has answered. If the database is briefly overloaded, the fetch keeps the connection to TED open and may run into a timeout.
- One error affects everyone. A single notice with unexpected content ends the run with an exception, and the rest are not processed.
All four problems have the same cause: fetching and processing are tightly coupled. One waits for the other, and they share a process. A message separates the two.