A person takes in an image in a second; a computer needs models for it. Nordlicht Reisen GmbH has four kinds of material that is not pure text: photos of hotel rooms that travelers upload, invoices and booking confirmations as PDF, audio recordings of the hotline and perhaps one day videos. This lesson clarifies three things: how machines understand images (computer vision), how models generate images and how text, images, audio and video become structured information. The lesson runs none of it against Azure. The details come from Microsoft Learn.

Understanding images

For tasks on images, Learn names two ways. The first is models with image input (multimodal models) in Foundry: you put an image together with a question into the prompt and get text back (lesson 2.8). The second is Azure Vision (on Learn "Azure Vision in Foundry Tools") with three product groups:

Service What it does Status according to Learn
Face detects, compares and analyzes faces in images its own service
Image Analysis reads features from images: description (caption), tags, objects, people listed as "legacy", version 4.0 deprecated
OCR (Optical Character Recognition) reads printed and handwritten text from images and documents listed as "legacy"

What Image Analysis 4.0 returns is known to the exam as terms: caption (one sentence for the whole image), dense captions (descriptions of individual objects with a bounding box), tags (keywords for objects, scenery, actions), object detection (objects with the coordinates of the bounding box), people detection and reading text. According to Learn, the image analysis service is deprecated and will be retired; Learn recommends moving to alternatives. As replacements it names: for text recognition Azure Document Intelligence, for faces the Face service, for image description and custom evaluations GPT models in Foundry and for a managed solution Azure Content Understanding. The retirement date is on Learn and does not belong in this lesson. Whoever builds something new starts with a model with image input (lesson 2.8).

Nordlicht: A room photo such as zimmerfoto.jpg is to be checked. An image analysis service would return, for example, a caption, tags and detected objects. A model with image input answers free questions such as "Is a balcony visible?" (the answer here is only the example, not measured). Photos of travelers can show faces. That touches privacy and security (lesson 1.1).

Generating images

An image generation model creates a new image from a text description. Among the models sold by Azure, Learn lists a number of them, for example gpt-image-1; others come from other providers. For the description ("prompt"), Learn names a limit of 4,000 characters for these models. In addition, existing images can be edited: a mask determines which area is changed, and the rest stays untouched.

The limits and safeguards are relevant to the exam:

  • Built-in protection: Azure checks the input and output of all image generation models with content filters and abuse detection. If the service detects a harmful prompt, it returns no image.
  • Photorealistic images of minors are blocked by default.
  • No immunity: According to Learn, image models can produce harmful or unexpected content despite technical measures, for example violence, sexual content, depictions of public figures or protected brands. No generative model is immune to targeted attacks. The content filter does not replace review by people (AI track, T6.3).
  • Transparency: Learn recommends labeling AI-generated images as such when publishing them and observing the applicable terms of use and copyright.

Nordlicht: For a travel catalog, a generated atmospheric image could do, with a label. A generated "photo" of a real hotel room, by contrast, would be deception. That is the case's rule and follows from transparency.

Extracting information

Information extraction turns unstructured content into structured data, such as fields with values that a database accepts. The technique depends on the input:

Input Techniques Result
Text entities and key phrases (lesson 1.5); a language model can pull fields from text lists, fields
Image OCR for text in the image; fields according to a schema, such as condition or count text, fields
Document (PDF, scan) OCR, layout detection (paragraphs, tables, selection marks) and field extraction Markdown or fields with value
Audio transcription, speaker diarization, fields such as summary and sentiment transcript, fields
Video transcription of spoken language, detection of important visual elements, splitting into scenes descriptions, fields

Azure Content Understanding (in Foundry Tools) is the tool that processes documents, images, audio and video with the help of generative AI into an output format that you define. An analyzer determines how processing is done and contains the schema of the fields. The fields come about in three ways: Extract (take values as they appear in the document), Classify (choose from predefined categories) and Generate (form freely from the data, such as a summary). Each value comes with confidence scores (0 to 1) and the place of origin (grounding), so that people can check quickly. The output is Markdown or JSON. Content Understanding is available in the Foundry portal, and lesson 2.9 builds with it.

Nordlicht:

  • rechnung-seeblick.pdf: hotel name, invoice date and amount as fields. A low confidence score sends the invoice to a person for review.
  • anruf-hotline.wav: Lesson 1.4 assigned the call to speech. According to Learn, Content Understanding also offers a prebuilt analyzer for calls (transcript with roles, summary, sentiment). That is an alternative; the case's decision stands.
  • For an image that is mainly about text, Learn advises a document schema instead of an image analyzer.

Exercise 1

Choose a technique or tool and justify briefly:

  1. The name on a photo of the hotel sign should be read.
  2. For uploaded photos, a one-sentence description should be created each, and the service is being built new.
  3. Date and total from scanned invoices should go into a table.
  4. A brochure needs a cover image with a beach and palm trees, and the department has no photo.
  5. Speaker roles, summary and sentiment should come out of hotline recordings.

Solution 1
  1. OCR, that is, reading text from images.
  2. A model with image input in Foundry. The older image analysis service is deprecated.
  3. Field extraction for documents (for example with Content Understanding), with confidence scores for review.
  4. An image generation model. The result is labeled as AI-generated.
  5. An audio analyzer of Content Understanding. It returns a transcript with roles and fields such as summary and sentiment.

Further reading