- Published on
An Apache-2.0 Lean 4 Code Agent Model Solving 587 of 672 PutnamBench Problems
- Authors
- Name
- DeepHorus
Mistral AI has released Leanstral 1.5, an open-source code agent model for the Lean 4 proof assistant, capable of solving 587 out of 672 problems from the PutnamBench benchmark. The model, released under the Apache 2.0 license, is available via a free API endpoint named leanstral-1-5. This release marks a significant step in automated theorem proving and proof engineering, making advanced verification tools accessible to the broader research community.
Leanstral 1.5 is built on the Mistral Small 4 family with a mixture-of-experts (MoE) architecture. The model uses 128 experts, activating four per token, resulting in 119 billion total parameters but only 6.5 billion active per inference. It supports a context length of 256,000 tokens and accepts multimodal input (text and images), though output is text-only. This design balances computational efficiency with high capacity, enabling the model to handle complex mathematical and logical tasks.
The training process involved three stages: mid-training, supervised fine-tuning, and reinforcement learning with a technique called CISPO. Two distinct reinforcement learning environments shaped the model's agentic behavior. In the multiturn environment, the model receives a theorem statement and iteratively submits proofs, refining based on Lean compiler feedback until success or exhaustion of attempts. In the code agent environment, it operates within a raw filesystem, editing files, running bash commands, and interacting with the Lean language server to obtain real-time goal and error information. This allows Leanstral to complete partial proofs, build auxiliary lemmas, and manage long tasks through context compaction.
Benchmark results demonstrate Leanstral 1.5's prowess. It achieves 100% on the miniF2F validation and test sets. On the PutnamBench, it solves 587 of 672 problems, outperforming Seed-Prover 1.5 high by seven problems while costing approximately $4 per problem compared to Seed-Prover's estimated $300 or more. The model sets state-of-the-art scores on the FATE-H (87%) and FATE-X (34%) algebra benchmarks. On FLTEval, derived from real pull requests to the Fermat's Last Theorem repository, Leanstral achieves pass@1 of 28.9% and pass@8 of 43.2%, surpassing Opus 4.6's 39.6% at one-seventh the cost.
Test-time scaling is a defining feature of Leanstral 1.5. Increasing the token budget per attempt dramatically improves performance on PutnamBench: from 44 solved problems at 50,000 tokens to 244 at 200,000, 493 at 1 million, and 587 at 4 million. This scaling behavior highlights the model's ability to leverage additional compute for more complex reasoning tasks, making it adaptable to varying resource constraints.
Beyond benchmarks, Leanstral 1.5 demonstrates practical utility in two case studies. First, it proved O(log n) time complexity for a real AVL tree implementation using structural induction and monadic time tracking, processing over 2.7 million tokens across 22 compactions. Second, it found real bugs in 57 open-source Rust repositories by translating Rust code to Lean via the Aeneas tool and inferring corrective properties. The model flagged 47 violated properties, showcasing its potential for code verification in software engineering.
The release has garnered attention for its open-source nature and cost efficiency. Mistral AI positions Leanstral 1.5 as a tool that democratizes formal verification, previously limited to expensive proprietary systems. By outperforming larger models at a fraction of the cost, it could accelerate adoption in mathematics and software reliability. Looking ahead, Leanstral 1.5 paves the way for more accessible automated theorem proving, with the free API endpoint enabling researchers and developers to experiment with proof engineering and code verification tasks.
