- Published on
A 2.8 Trillion Parameter Open MoE Model With Kimi Delta Attention and 1M Context
- Authors
- Name
- DeepHorus
Moonshot AI has released Kimi K3, a 2.8-trillion-parameter open Mixture-of-Experts (MoE) model with native vision and a 1-million-token context window. The company calls it the worldâs first open 3T-class model, marking a milestone in the open-source AI landscape. This release builds on Moonshotâs track record of pushing the boundaries of open-model sizes.
Kimi K3 employs two novel architectural updates: Kimi Delta Attention (KDA) and Attention Residuals (AttnRes). KDA is a hybrid linear attention mechanism that enables up to 6.3x faster decoding in million-token contexts, while AttnRes selectively retrieves representations across depth, delivering roughly 25% higher training efficiency at under 2% additional cost. The model activates only 16 of its 896 experts, using Stable LatentMoE with techniques like Quantile Balancing and Per-Head Muon to optimize routing and training stability.
According to Moonshot, K3 achieves approximately 2.5x better overall scaling efficiency than its predecessor, Kimi K2. In performance evaluations, K3 leads on benchmarks such as Program Bench, SWE Marathon, BrowseComp, Automation Bench, and OmniDocBench. However, it still trails proprietary models like Claude Fable 5 and GPT 5.6 Sol on tasks like FrontierSWE and DeepSWE. Moonshot is transparent that overall performance lags behind the most powerful closed-source systems.
The release carries significant implications for the AI community. K3 is available on Kimi.com, Kimi Work, Kimi Code, and via API using the OpenAI SDK. Pricing is flat with no tiering by context length: $0.30 per million tokens for cache-hit input, $3.00 for cache-miss input, and $15 for output. The model applies quantization-aware training from the SFT stage, using MXFP4 weights and MXFP8 activations for broad hardware compatibility. Moonshot also contributed an implementation to vLLM to address prefix caching challenges.
Industry reactions highlight K3âs role as a competitive open alternative to proprietary giants. Moonshotâs direct admission of K3âs limitations relative to top closed models underscores the ongoing gap but also the rapid progress in open-source AI. The modelâs ability to handle long-horizon coding and knowledge work positions it as a powerful tool for researchers and developers seeking transparency and customization.
Looking ahead, Kimi K3 sets a new benchmark for open large language models, potentially driving further innovation in sparse MoE architectures and efficient attention mechanisms. Moonshot AIâs continued emphasis on open release encourages broader adoption and collaborative improvement. As the field evolves, K3 provides a foundation for future breakthroughs balancing performance and accessibility.
