- Published on
Tencent's Hunyuan Hybrid 4 Preview Lightweight Version Open-Sourced for Joint Inference on Heterogeneous Devices
- Authors
- Name
- DeepHorus
On September 1st, Tencent's Hunyuan Team officially released a lightweight version of the Hy4 preview large language model, successfully compressing its model weights from nearly 1.5TB to approximately 214GB. This major advancement enables this advanced model, which has 770 billion total parameters and 49 billion activated parameters, to run under a much wider range of hardware conditions, significantly lowering the deployment barrier. The released lightweight version is optimized based on the Hy4 preview model open-sourced by Tencent on August 28th, which originally supported a context length of up to 1 million tokens.
This compression was primarily achieved through two core quantization techniques. The first is the Sherry sparse ternary quantization algorithm, which compresses the weights of the routing expert layers in the model to an average of 1.25 bits per weight. The second is the MIX-STQ1_0 mixed-precision strategy, which uses calibration data to analyze layer by layer and intelligently assigns quantization bit widths to different layers of the model. Layers more sensitive to precision adopted the IQ2_XXS scheme (2.06 bits), while less sensitive layers used a more aggressive STQ1_0 scheme (1.31 bits).
The combination of these two techniques achieved an excellent balance. Test data shows that the lightweight version model performed stably across multiple mainstream benchmarks. On long-context understanding tasks, its performance was nearly identical to the original BF16 precision model. Multi-turn long-context retrieval tasks also basically remained at the same level. Mathematical reasoning ability showed only a slight decline, with the MCP Atlas score dropping marginally from 83.7 to 83.2, and the SWE-Bench multi score decreasing from 82.9 to 81.3, keeping overall performance loss within an acceptable range.
Notably, the lightweight version model also validated the feasibility of joint inference across heterogeneous devices. Tencent's Hunyuan collaborated with prima.cpp to conduct tests on a laptop equipped with a single 4090 graphics card and a server equipped with four A4000 graphics cards. These two devices collectively provided 80GB of video memory and 64GB of system memory and belonged to different local area networks. Through prima.cpp's heterogeneous scheduling technology, the model's MoE layers were intelligently split and allocated, ultimately allowing the 214GB lightweight version model to achieve an inference speed of 1.02 tokens per secondâapproximately 6 times faster than offline loading on the laptop alone.
This lightweight version model has now been open-sourced and released on Hugging Face and GitHub platforms. Developers can access the resources through the specified quantized GGUF files and code repositories. The officials stated that the model already supports mainstream inference frameworks such as llama.cpp, vLLM, and SGLang, greatly facilitating community deployment and secondary development.
The successful release of the Hy4 preview lightweight version marks an important step forward in the efficient compression and edge deployment of large language models. It not only demonstrates Tencent's leading strength in model quantization technology but also provides a practical solution for running ultra-large models on resource-constrained local devices or hybrid hardware environments, promising to drive exploration in a wider range of application scenarios.
