- Published on
An Audio-Native Dialog Model That Fuses Turn-Taking, Speech Recognition, Function Calling, And Response
- Authors
- Name
- DeepHorus
PolyAI has released Dialog-RSN-1, an audio-native dialog model that processes caller voice directly instead of reading a transcript. It unifies turn-taking, speech recognition, function calling, and response generation into one system, and is already managing live production calls. This approach gives AI agents a fuller understanding of tone, hesitation, and uncertainty.
The design departs from both dominant architectures. Cascaded systems pass only the ASR's top guess to the LLM, losing crucial audio nuance, while speech-to-speech models like GPT Realtime and Gemini Live keep audio but fix the voice in the model, limiting control and often reserving a GPU for the entire call. Dialog-RSN-1 listens to raw audio on input, while a separate, promptable text-to-speech engine handles output, preserving voice flexibility.
PolyAI reports sub-300ms response times on A100 GPUs, an 11% containment gain at a restaurant group, and a 37% latency drop at an insurer. The model is request-based, probed only when needed, and runs English-only at launch. It is available exclusively through PolyAI's platformâno open weights or public APIâtargeting enterprises with high call volumes.
Turn-taking is encoded in the model's first output token: EMPTY, ONGOING, or COMPLETE. PolyAI built it by post-training open-weight multimodal models with supervised and reinforcement fine-tuning, testing Gemma, GPT-OSS, Qwen, and Mistral. Optimizations include prefilling attention caches, append-only prompts, per-caller GPU routing, and a finetuned speculative drafter accepting 3.9 tokens on average.
On its internal Dialog-Eval benchmark, Dialog-RSN-1 scored highest among real-time-capable models, while cascades' audio-aware ceiling was near 77 and GPT Realtime 2.1 matched cascades. Transcription also improved: gpt-4o-transcribe's word error rate fell from 7.8% to 6.9% with added context, and Dialog-RSN-1 did even better. PolyAI plans to open-source Dialog-Eval.
The model fits PolyAI's enterprise focus: 100+ customers and 2,000+ live deployments, supported by its $86M Series D in December 2025. Applications include booking, billing, authentication, routing, order management, and troubleshooting across restaurants, insurance, finance, healthcare, hotels, retail, telecom, travel, and utilities. Existing customers can enable it now; new customers can request early access. PolyAI will publish a technical report and paper, while Raven 3.5 remains recommended for non-English use cases.
