- Published on
An Open Hybrid Mamba-Transformer MoE Foundation Model For German And English
- Authors
- Name
- DeepHorus
A German research consortium has released Soofi S 30B-A3B, an open hybrid Mamba-Transformer Mixture-of-Experts (MoE) foundation model designed for German and English. The model, developed under the coordination of KI Bundesverband and funded by the German Federal Ministry for Economic Affairs and Energy, was trained entirely on Deutsche Telekom’s Industrial AI Cloud in Munich. Preview weights are available on Hugging Face, and the pretraining report highlights that Soofi S records the highest English and German aggregate scores among fully open base models tested.
Soofi S totals approximately 31.6 billion parameters but activates only 3.2 billion per token, making it efficient for deployment. As a base model, it lacks instruction tuning or alignment, serving as a raw foundation for further fine-tuning. The consortium includes Fraunhofer IAIS, DFKI, TU Darmstadt, ellamind, and Merantix Momentum, reflecting a collaborative national effort to advance AI sovereignty.
The architecture combines 52 layers: 23 Mamba-2 sequence-mixing layers, 23 granular MoE layers with 128 routed experts (activating 6 per token plus 2 shared experts), and 6 Grouped-Query Attention layers. This design limits the KV cache to only those 6 layers, enhancing serving efficiency. The model adopts the Nemotron 3 Nano reference design without modifications to isolate the impact of the data recipe.
Training consumed approximately 26.68 trillion tokens across three phases following a Warmup-Stable-Decay schedule. Phase 1 used 20 trillion tokens at a 1e-3 plateau; Phase 2 used 6.58 trillion high-quality annealing data decaying to 1e-5; Phase 3 used 0.1 trillion tokens at 1,048,576-token sequence length, extending context to 1 million tokens. German language proportion deliberately increased from 7.2% in Phase 1 to 15.32% in Phase 2, sourced from datasets like HPLT, German Commons, and Genios (193 million articles). Training ran from March 24 to May 13, 2026, on up to 512 NVIDIA B200 GPUs, consuming ~253,000 GPU-hours.
In evaluations against 16 open base models, Soofi S outperformed its architecture-identical reference by 1.8 points on English aggregate, 4.2 on German, and 6.7 on held-out English. Against larger models, it scored 70.1 English (comparable to Gemma 3 27B and Ministral 3 14B) and led in German with 79.1. However, Qwen3.5 35B-A3B held the highest overall means.
The release underscores Germany’s push for AI sovereignty, providing a competitive open alternative for bilingual applications. The modular design and efficient architecture may influence future European AI projects. As a base model, it invites innovation in fine-tuning and alignment, potentially narrowing gaps with proprietary systems in European languages. Researchers can access the gated preview and custom modeling code on Hugging Face to reproduce results or build upon the model.
