There are some obvious signs that can quickly distinguish between casual and advanced AI users. One, for example, is the use of voice AI in everyday tasks. While most users are still working hard on their keyboard to get the perfect speed, someone with experience in using AI now just talks to it. A well-placed question in a conversation saves time, effort, and often yields better results than a stand-alone text. Despite these benefits, Voice AI is largely limited to the elite. OpenAI now plans to change that with three real-time voice models in the API.
Three new audio models: GPT-Realtime-2, GPT-Realtime-Translate, and GPT-Realtime-Whisper, are designed to help developers build voice applications that can listen, think, translate, transcribe, and take action while a conversation is taking place. OpenAI describes them as “a new generation of real-time speech models” that can work as people speak.
Here, we will examine 3 models in detail and understand why they can change the use of AI as we know it. But before we get started, here’s what you need to know about real-time voice models.
What Are Real-Time Voice Models?
Real-time voice models are AI models that can understand and respond to speech while the conversation is taking place.
Typically, voice AI works in steps. First, it records your voice. It then converts speech to text. Then another model reads the text and prepares an answer. Then another system converts that response into speech. This works, but it can feel slow and unnatural. Real-time voice models bridge that gap.
They are built to listen, understand, and react almost instantly. So instead of waiting for a full sentence or a full audio file to finish, AI can process speech as it comes in. This makes the conversation feel natural, especially when users pause, interrupt, change direction, or ask follow-up questions.
In simple terms, real-time voice models make AI conversations feel like you’re talking to a real assistant. And that’s exactly the experience OpenAI is targeting with its new launch.
New Voice Models for OpenAI
OpenAI introduced three new audio models to the API: GPT-Realtime-2, GPT-Realtime-Translate, and GPT-Realtime-Whisper. Together, they are designed for applications where AI needs to work when a person is speaking. That means AI can capture a conversation, understand context, translate speech, record live audio, and use tools during a conversation. OpenAI says these models are designed to help developers create voice emotions that sound natural and “can act in real time.”
Again, this is important because voice AI goes beyond simple commands. A helpful voice agent shouldn’t just hear words and respond. It must understand what the person is looking for, remember context, handle corrections, use tools, and react naturally. OpenAI says the goal is to move real-time audio from simple “call and answer” systems to voice interfaces that can actually work as the conversation progresses.
Each of OpenAI’s 3 voice models solves a specific part of that desire.
GPT-Realtime-2
GPT-Realtime-2 is the main dialog voice model. Designed for voice agents who need to speak naturally, understand context, handle distractions, and take action during a live chat.
For example, a customer support agent built into GPT-Realtime-2 can understand a user’s problem, ask follow-up questions, check order details using the tool, and respond while the call is in progress.
GPT-Realtime-Translate
As the name suggests, GPT-Realtime-Translate is designed to translate live speech. It can take a speech in one language and translate it into another language while the person is speaking. The demo shared by OpenAI shows the model in action, and I dare say it looks like a versatile tool for translation needs in live chats or addresses.
You can understand how this can be useful for global meetings, travel applications, multilingual customer support, educational forums, and live events where people need to translate quickly.
GPT-Realtime-Whisper
GPT-Realtime-Whisper is designed for live recording. It converts speech to text in real time instead of waiting for the entire audio file to finish. Which means you will see the words written in front of you as soon as you speak them.
This can help with live captions, meeting transcripts, call notes, classroom recordings, interviews, and any application where spoken words need to be quickly translated into text.
OpenAI Voice Models: Key Features
From their capabilities listed above, we can imagine how useful these 3 OpenAI voice types can be. However, there are many additional features that improve this application.
1. Voice Agents Can Take Action
GPT-Realtime-2 is designed for voice agents that do more than respond. It can run an application, call tools, handle fixes, and carry on a conversation while work is in progress. OpenAI says this takes voice AI to systems that “can actually do the work.”
2. Better Handling of Disruptions and Corrections
Real conversations are not clean. People stop, change their minds, interrupt, or correct themselves. GPT-Realtime-2 is designed to handle these sessions better, so the conversation doesn’t stop every time the user changes direction. OpenAI says it has a “robust recovery policy” for such cases.
3. Long Context for Complex Functions
OpenAI increased the context window from 32K to 128K for GPT-Realtime-2. In simple words, the model can remember and process more information during long conversations. This is useful for complex voice operations such as support calls, travel planning, health care conversations, or work assistants.
4. Live Translation in All Languages
GPT-Realtime-Translate can translate speech from 70+ input languages to 13 output languages while compatible with the speaker. This makes it useful for multilingual customer support, global meetings, live events, education, and developer forums.
5. Live Transcripts While People Are Talking
GPT-Realtime-Whisper can convert speech to text while the person is speaking. This can enable live captions, meeting notes, call transcripts, class notes, and quick follow-up workflows.
6. Greater Control of Voice and Communication
Developers can control how the voice agent sounds and how much thinking effort it uses. For example, a model can sound calm during a support problem, empathetic when a user is frustrated, or overjoyed while confirming a job. Developers can also choose levels of reasoning from minimum to x-high, depending on the task.
OpenAI speech models: Implementations
Based on the above capabilities, OpenAI’s 3 new voice models are sure to serve as a great advantage in the following tasks:
1. Customer Support Agents
A company can create voice agents that answer customer calls, understand the issue, ask follow-up questions, check order or account information, and complete basic actions during the call.
2. Live Meeting Translation
International teams can use GPT-Realtime-Translate to translate conversations while people are speaking. This can make world meetings easier without having to wait for manual translation later.
3. Live Captions and Transcripts
GPT-Realtime-Whisper can be used to create live captions for calls, webinars, classes, discussions, and events. It can also turn the conversation into searchable text.
4. Travel and Reservation Assistants
A travel app can use real-time voice models to help users search for flights, compare hotels, change reservations, or ask travel questions through natural voice chat.
5. Health Care Call Assistants
Healthcare providers can use voice agents to help with appointment scheduling, patient intake, follow-up calls, or basic information gathering. The final medical decision should still rest with doctors and professional staff.
6. Voice Assistants at work
Companies can create internal voice assistants that help employees find files, summarize meetings, create task lists, update records, or extract information from internal systems.
Pricing and Availability
All three models: GPT-Realtime-2, GPT-Realtime-Translate, and GPT-Realtime-Whisper, are available through OpenAI’s Realtime API. Developers can also test them in the OpenAI Playground before building them into apps.
- GPT-Realtime-2: $32 per 1M audio input tokens, $0.40 per 1M cached input tokens, and $64 per 1M audio output tokens.
- GPT-Realtime-Translate: $0.034 per minute.
- GPT-Realtime-Whisper: $0.017 per minute.
The conclusion
OpenAI’s real-time voice models clearly show where voice AI is headed next.
It’s no longer just about asking a question and getting a spoken answer. With the new GPT voice models, developers can now build more action-oriented voice applications. All this, within the context of a seamless conversation.
Practically, think of this as a quick support call. The meeting is held in many languages. A class that receives live scripts. The most interactive travel app. A work assistant that moves from text chat to natural speech.
Of course, this does not mean that all voice agents will be perfect. Developers will still need strong monitoring protocols, clear user disclosures, privacy controls, and human reviews in sensitive areas such as healthcare, finance, and legal support.
But the direction is clear. From artificial speech communication to real-time interactive assistance, and OpenAI wants to be its leader.
Sign in to continue reading and enjoy content curated by experts.
