Integrate GPT 4o without TTS/STT

[clemlesne]

OpenAI GPT 4o model supports both in and out of text, image and audio. Understanding is finer than usual STT > model > TTS approach because the model has direct access to user behavior, emotions, etc.

Is there a way to use Communication Services and receive the raw audio flow, bypassing the STT step?

[Qwatro55]

I'm also interested in this question.

[agentverket]

What about response time?
What about costs?
Can you stream data?

[clemlesne]

I know I know :) OpenAI APIs are not yet available:

• https://community.openai.com/t/what-will-the-gpt-4o-audio-api-look-like/754242
• https://community.openai.com/t/gpt-4o-chat-completion-with-audio-response/752149/6

Plus, Communication Services APIs are not yet available to use with raw audio stream.

If you have ideas, don't hesitate!

[JunJD]

m

[clemlesne]

Audio streaming is now available with Communication Services!

https://learn.microsoft.com/en-us/azure/communication-services/how-tos/call-automation/audio-streaming-quickstart?pivots=programming-language-python