- Published on
An Open-source RL System from ByteDance Seed and Tsinghua AIR
- Authors
- Name
- DeepHorus
ByteDance Seed and Tsinghua AIR have unveiled DAPO, a fully open-source reinforcement learning system for large language models. The release includes the algorithm, code infrastructure, and dataset. DAPO achieved more than 50 points on the AIME 2024 benchmark using the Qwen2.5-32B base model.
The system is built around the Decoupled Clip and Dynamic sAmpling Policy Optimization algorithm, or DAPO. It outperforms the previous state-of-the-art DeepSeek-R1-Zero-Qwen-32B while requiring about 50 percent fewer training steps. The project is based on the open-source verl framework.
The team also released model weights for DAPO-Qwen-32B, trained from Qwen2.5-32B. They published wandb training records, including a full run that surpassed 50 percent on AIME 2024. An earlier version, without token-level policy gradient loss and dynamic sampling, reached 44 percent on the same benchmark.
The release addresses a growing need for reproducible reinforcement learning in the AI community. Many advanced RL systems remain proprietary, limiting research and adoption. By open-sourcing the entire pipeline, the team aims to let broader researchers benefit from scalable RL.
The team highlighted training dynamics that contribute to DAPO's stability. Response length grows steadily, enabling more exploration and complex reasoning. Reward scores increase consistently, while entropy follows a controlled rise after an initial decrease, balancing exploration and exploitation.
For developers, the project provides setup instructions using conda and Python 3.10. Inference code is available through vLLM and Hugging Face Transformers, with example math problems included. The team encourages questions and discussion through GitHub issues.
Looking ahead, DAPO's open-source design could accelerate progress in LLM reasoning and reinforcement learning. By sharing infrastructure and results, the project lowers barriers for reproducing state-of-the-art performance. This may inspire further innovations in training stability and efficiency.
