- Published on
A Unified Audio-Text LLM That Preserves the Text Intelligence of Its Backbone
- Authors
- Name
- DeepHorus
NVIDIA has released Audex (Nemotron-Labs-Audex-30B-A3B), a unified audio-text large language model capable of understanding and generating both audio and speech while preserving the text intelligence of its backbone. The model, along with a smaller 2B parameter variant, is available under a noncommercial license. This release addresses a common issue in multimodal models where adding audio or vision output often degrades text benchmarks, a phenomenon termed 'text tax.'
Audex is a single Mixture-of-Experts (MoE) Transformer decoder with 30 billion total parameters and 3 billion activated per token. Its backbone is the text-only Nemotron-Cascade-2-30B-A3B, a hybrid Mamba-Transformer with 52 layers, 128 routable experts, and 6 activated experts. The design is deliberately simple: audio inputs are encoded and projected into the text embedding space, and text tokens and quantized audio tokens are treated uniformly during generation. This approach eliminates the need for thinker-talker splits or stacked model cascades.
The architecture relies on three components: an audio encoder (AF-Whisper based on Whisper Large-v3), two-layer MLP adapters for audio feature mapping, and an extended vocabulary with 205,312 tokens (up from 131,072). Two codecs handle output: X-Codec2 for speech at 50 tokens per second using single-layer finite scalar quantization, and X-Codec for non-speech sound at 200 tokens per second using four flattened residual vector quantization layers. The model supports context lengths up to 1 million tokens and runs on standard LLM stacks like Megatron-LM and vLLM.
Training requires no audio pretraining; it starts from the text-only supervised fine-tuning (SFT) checkpoint. A multi-stage curriculum adds capabilities in order: text SFT, audio warmup, audio generation, then audio understanding. During audio warmup, text token embeddings stay frozen to preserve text quality. After SFT, text-only Cascade Reinforcement Learning (Cascade RL) and multi-domain on-policy distillation (MOPD) are applied, improving text scores while showing marginal or no regression on audio tasks. The data mix includes 157.4 billion audio tokens and 320.5 billion text tokens, covering tasks like ASR, AST, TTS, text-to-audio, and audio understanding.
Benchmark results show Audex tracks its backbone closely on text: scoring 86.4 on MMLU-Redux against the backbone’s 86.3, and leading on IMO AnswerBench (81.1 vs. 79.3). Small drops appear on MMLU-Pro and GPQA-Diamond. On speech recognition, Audex achieves a 6.82 average word error rate on the OpenASR leaderboard, outperforming comparable open models. The model also tops the text-only Qwen3.5-35B-A3B on several reasoning and alignment benchmarks.
The implications are significant for open-source multimodal AI. Audex is one of the few open models that generate general audio beyond speech, avoiding the typical trade-off between modality addition and text performance. NVIDIA’s approach of multi-stage training and text-only RL offers a replicable method for other researchers. The release may accelerate development in voice assistants, audio content generation, and accessibility tools.
The research team emphasizes that Audex’s simple design runs on standard LLM infrastructure, lowering the barrier for adoption. While currently under a noncommercial license, the model provides a strong foundation for further development. Future work may extend context lengths, improve audio understanding, or explore commercial applications. As multimodal AI evolves, models like Audex could redefine how machines process and generate multiple modalities without sacrificing core language capabilities.
