Nordlicht Reisen's hotline takes calls as audio recordings; the file anruf-hotline.wav stands for one of them. This lesson shows two ways to bring speech into an application: a model that understands speech and speaks itself, and the service Azure Speech in Foundry Tools (Azure Speech for short) for speech recognition and speech synthesis. Everything here is not run against Azure; this is how Microsoft Learn shows it. Answers and outputs are not printed.

Path 1: an audio model

Learn describes models that bring audio into the familiar chat interface (/chat/completions). For example gpt-audio and gpt-audio-mini: they accept text, audio or both and answer with text, audio or both. For "spoken requests to a model", this is the path of the exam. A second family, gpt-realtime, is meant according to Learn for conversations with low latency.

Learn names limits: audio files of at most 20 MB, output formats such as wav and mp3, a fixed list of voices. The name of the API version contains the word preview; whether the audio model itself is a preview, the page does not state clearly: open. Equally open: whether it is available in your region.

The steps in the portal, as Learn shows them for gpt-audio-mini:

  1. In the Foundry portal, open the project (at Nordlicht, nordlicht-ki).
  2. Open Models + endpoints, choose Deploy model and Deploy base model, find the model and deploy it. That is lesson 2.1 with a different model.
  3. Under Playgrounds, open the Chat playground (not the Audio playground: Learn says it does not support this model) and choose the deployment.
  4. There you can record audio, attach an audio file or enter text, and you hear the answer.

For the code, Learn has tabs for JavaScript, Python, TypeScript and REST on this topic, none for C#. The principle of the request is the same in all of them: The audio file is read, packed as Base64 text and sent as a content part of type input_audio with the format (such as wav) next to a text part, plus the desired kinds of output (modalities) and the voice and format of the answer. The request example in Learn's REST tab is Python code; for the exam, you read it there (note in lesson 2.3). There is no unverified C# code here.

Path 2: Azure Speech

Azure Speech is a service of its own for converting speech to text and text to speech. Learn shows both for C# with the NuGet package Microsoft.CognitiveServices.Speech. You need an AI Services resource for Speech, a key and an address. For production, Learn recommends Entra ID with managed identities instead of a key in the code.

Speech recognition

Excerpt from Learn's quickstart (the lines before it read key and address from environment variables):

C#
var speechConfig = SpeechConfig.FromEndpoint(new Uri(endpoint), speechKey);
speechConfig.SpeechRecognitionLanguage = "en-US";

using var audioConfig = AudioConfig.FromDefaultMicrophoneInput();
using var speechRecognizer = new SpeechRecognizer(speechConfig, audioConfig);
var speechRecognitionResult = await speechRecognizer.RecognizeOnceAsync();
  • SpeechConfig carries the address, the key and the language of the recording. Without a language, en-US is the default according to Learn. For German, you enter the code of the German language. Which codes are valid is on Learn's "Language support" page, which we have not read: open.
  • AudioConfig says where the audio comes from. For a file, Learn names AudioConfig.FromWavFileInput("YourAudioFile.wav") instead of the microphone; at Nordlicht, anruf-hotline.wav goes there.
  • RecognizeOnceAsync recognizes utterances of up to 30 seconds or until the first silence, according to Learn. For longer recordings, Learn points to continuous recognition. For recording files, Learn names as alternatives fast transcription (supports, among other things, language detection and speaker diarization) and batch transcription.

Speech synthesis

C#
speechConfig.SpeechSynthesisVoiceName = "en-US-Ava:DragonHDLatestNeural";

using var audioConfig = AudioConfig.FromWavFileOutput("output.wav");
using var speechSynthesizer = new SpeechSynthesizer(speechConfig, audioConfig);
await speechSynthesizer.SpeakTextAsync("Hello, world!");

Learn explains the name format of the voices: {locale}-{VoiceName}:{VoiceType}, in the example en-US, Ava and DragonHDLatestNeural. All Neural voices are multilingual according to Learn; if a voice does not speak the language of the text, the service outputs no audio. FromWavFileOutput saves to a file instead of the speaker.

The case: a call

What fits what at Nordlicht? The helper transcribes a hotline call and puts the text in the file; a person from customer service reads it. That is speech recognition, so Azure Speech. The text can then go into the text analysis of lesson 2.6. A summary and sentiment straight from the call come from lesson 2.9 with Content Understanding. The pilot does not use speech synthesis for answers to customers; whether and when is decided by a person. No model decides about refunds.

Recordings: an honest remark

Calls are recordings of people. Learn writes about speech recognition: audio in which a person speaks is personal information; you must comply with the laws on privacy, communications and wiretapping of your jurisdiction, process only audio within users' reasonable expectations, and have all the necessary consents. Learn advises a legal review and says itself that it is not legal advice. Whether Nordlicht may record and evaluate calls, which announcement is needed at the start of a call and how long the files stay is for Nordlicht's legal counsel to settle. This lesson does not decide that.

Try it (optional)

A project, a model deployment and a Speech call cost money per use. Try only with examples you speak yourself, never with real calls. Afterwards, delete the resource group so that nothing keeps running.

Exercise 1

Assign a path to each task: audio model, Azure Speech (speech recognition), Azure Speech (speech synthesis) or a path from another lesson.

  1. An adviser asks the helper a question about a booking confirmation out loud and hears the answer.
  2. anruf-hotline.wav should be in the customer file as text.
  3. The booking confirmation should be available in the portal as an audio file to listen to.
  4. A summary and sentiment should be produced automatically for a call.

Solution 1
  1. Audio model: spoken request, spoken answer. For conversations with low latency, Learn names gpt-realtime.
  2. Azure Speech, speech recognition: AudioConfig.FromWavFileInput with the file, and the person reads the result.
  3. Azure Speech, speech synthesis: FromWavFileOutput saves the spoken text as a file.
  4. Lesson 2.9: Content Understanding has an analyzer for audio. Alternatively: transcribe first (Azure Speech), then text analysis (lesson 2.6). Both are defensible; a test with real recordings decides.

Further reading