- Published on
A 975B-Parameter Open-Weights Multimodal MoE With 41B Active Parameters And Controllable Thinking Effort
- Authors
- Name
- DeepHorus
Thinking Machines Lab has released Inkling, a 975-billion-parameter open-weights multimodal Mixture-of-Experts (MoE) model with 41 billion active parameters. The lab pitches this model as a foundation for customization, emphasizing its fine-tunability on the Tinker platform. Inkling supports a 1-million-token context window and was pretrained on 45 trillion tokens spanning text, images, audio, and video. It accepts multimodal inputs but outputs only UTF-8 text. The release also previews Inkling-Small, a 276-billion-parameter MoE with 12 billion active parameters that matches or exceeds its larger sibling on many benchmarks; its weights will be released after testing.
The architecture behind Inkling is a 66-layer decoder-only transformer with a sparse MoE feed-forward backbone. Each MoE layer contains 256 routed experts plus two shared experts, with six routed experts activating per token and both shared experts always active. A sigmoid-based router employs an auxiliary-loss-free load-balancing bias. The attention mechanism interleaves sliding-window and global layers at a 5:1 ratio with 8 KV heads, using relative positional embeddings instead of RoPE for better extrapolation. Short convolutions are applied after key/value projections and on residual branch outputs, while multimodality is encoder-free: audio is processed as dMel spectrograms, and images are divided into 40×40 pixel patches via a four-layer hMLP.
Training utilized Muon for large matrix weights and Adam for other parameters on NVIDIA GB300 NVL72 systems. Post-training began with supervised fine-tuning on synthetic data, including data generated by Kimi K2.5, but most compute went into asynchronous reinforcement learning (RL) with over 30 million rollouts. This RL process produced the model's main control surface: controllable thinking effort. By varying the system message and per-token cost during RL, the model learned to allocate different token budgets per rollout. Effort can be set on a scale from 0.2 to 0.99, and in Transformers, it is exposed as a reasoning_effort argument with named levels.
Performance evaluations were conducted at effort=0.99 and temperature 1.0. On key benchmarks, Inkling leads the open-weights group on FORTRESS Adversarial with 78.0%. It achieves 73.5% on MMMU Pro, 91.4% on VoiceBench, and scores 1257 on Design Arena’s Agentic Web Dev leaderboard based on blinded human evaluation. On Terminal Bench 2.1, it trails GLM 5.2 by 18.9 points. Efficiency data shows Inkling uses only one-third as many tokens as Nemotron 3 Ultra for equal performance on Terminal Bench 2.1, making cost and latency tunable per call rather than fixed per model.
The open-weights nature of Inkling carries significant implications for the AI community. By releasing weights and enabling fine-tuning, Thinking Machines Lab positions the model as a versatile base for customized applications. The controllable thinking effort allows users to balance performance and computational cost dynamically, potentially democratizing access to large-scale reasoning capabilities. The lab also targeted trustworthiness directly during training, though specific metrics were not detailed.
Reactions from the AI community are still emerging, but the model's architecture and efficiency improvements have drawn attention. The use of relative positional embeddings and encoder-free multimodality represents a departure from common practices like RoPE and separate encoders. The competitive performance against established open-weights models such as GLM and Nemotron suggests Inkling could become a strong contender in customizable AI solutions.
Looking ahead, Thinking Machines Lab plans to release Inkling-Small weights once testing concludes, and models are already available for deployment. Running Inkling requires substantial hardware: BF16 checkpoints need at least 2 TB aggregated VRAM (e.g., 8x NVIDIA B300 or 16x H200), while NVFP4 checkpoints reduce memory to 600 GB. Runtimes include SGLang, vLLM, TokenSpeed, Unsloth, and Hugging Face Transformers. As the lab continues to develop open-weight models, Inkling represents a significant step toward more accessible, adaptable AI systems that can be tailored to specific needs.
