- Published on
An Open Full-Duplex Speech-to-Speech Model with ~450 ms Turn-Taking and Live Tool Calling
- Authors
- Name
- DeepHorus
NVIDIA has released NemotronLabs VoiceChat 11B, an open 11B end-to-end speech-to-speech model for real-time full-duplex conversation. It replaces cascaded ASR, LLM, and TTS with a single unified network, cutting smooth turn-taking latency to 448 ms on Full-Duplex-Bench 1.0.
The model listens while speaking, enabling user barge-in with a take-over rate of 1.00 at 480 ms. It is the first open full-duplex model to support live tool calling via a separate output channel. Operators can define on-hold lines to avoid dead air while APIs run.
Built as a hybrid Mamba/Transformer, it combines a Fast Conformer encoder, Nemotron Nano v2 LLM, and NVIDIA TTS decoder. Outputs include agent audio, agent text, and user transcription. Training used roughly 550,000 hours of audio from real and synthetic corpora.
On Full-Duplex-Bench 1.0, it posts 0.82 TOR at 448 ms and 1.00 user interruption at 480 ms. Its tool-calling average is 56.1% on BFCL-v3, with 82.5% tool selection on Full-Duplex-Bench v3. NVIDIA says it ranks #2 among open full-duplex models on VoiceBench.
Weights and container are public, but NVIDIA labels the checkpoint "ready for research purposes only." Documented failure modes include a two-minute audio context ceiling, non-recoverable gibberish after several turns, and runaway self-talk. Thus deployment is limited to pilots, not production.
Running it requires one GPU with at least 80 GB VRAM, like an A100, H100, RTX 6000 Pro, or B200 on x86_64 Linux. No hosted API exists, so only GPU-equipped teams can evaluate it. Future versions may fix context and stability limits, but today it gives developers a concrete research platform for duplex voice agents with tool calling.
