So far, every help page is a single record. That works as long as pages are short. A real help has long pages with many sections, and the question “How long does the call-off link work?” concerns only one of them. In this lesson, you cut the pages at their headings, give every section a note of where it comes from, and build the reading so that a second run without a change costs not a single token.
The tool for it is Microsoft.Extensions.DataIngestion. It is a preview: Names and behavior may still change before the first stable version. The lesson pins the version everything ran with and shows in two places how to pin behavior with a test instead of relying on it.
| Package | Version |
|---|---|
Microsoft.Extensions.DataIngestion, Microsoft.Extensions.DataIngestion.Markdig |
10.11.0-preview.1.26513.7 |
Microsoft.ML.Tokenizers, Microsoft.ML.Tokenizers.Data.O200kBase |
2.0.1 |
Microsoft.Extensions.Hosting.Abstractions |
10.0.14 |
CommunityToolkit.VectorData.InMemory |
1.0.1 |
xunit.v3.mtp-v2 |
4.0.2 |
Why cut
An embedding sums up a whole text in one vector. For a sentence, that is a clear direction. For a page with five topics, it is a mixture of all five, and a question about one of them is only moderately close to the mixture. Smaller pieces have sharper vectors and are therefore found better. Three practical reasons come on top:
- The chat model gets only what it needs. In the next module, the pieces found go to the model as text. Three matching sections cost fewer tokens than three whole pages and distract the model less (T5.1 divides up the context window).
- Sources become more precise. “See Moving and calling off appointments, section Calling off as a customer” helps more than just the page title.
- Models have a limit. Every embedding model accepts only texts up to a certain number of tokens.
But too small is wrong as well. A section of half a sentence loses its context: “It works until 24 hours before the appointment” alone doesn’t say what “it” is. The right size depends on the texts. It is measured in tokens, not in characters, because the model calculates in tokens (T1.2). For help pages, a few hundred tokens per section are a good start, and T5.4 measures whether it fits.