- Published on
Benchmarking AI models on private, real-world, enterprise codebases
- Authors
- Name
- DeepHorus
Specific Labs has released Real-SWE, a new benchmark that tests frontier AI models on private, real-world enterprise codebases. Announced in September 2026, it uses tasks licensed from actual production systems, requiring agents to navigate proprietary code unavailable on the public internet. The goal is to see whether coding agents can handle the work of a software engineer in a real business.
Unlike synthetic or expert-generated benchmarks, Real-SWE draws on verbatim tasks that company engineers actually face. Each task carries business consequences, such as fixing invoice billing, calculating taxes, or migrating customers across services. Agents must also learn company-specific conventions and work within existing code, adding complexity that public benchmarks lack.
Initial results show no model is close to full autonomy. Fable 5.1, running on Claude Code, leads with a 38.8 percent resolution rate. GPT-6 Astra via Codex CLI follows at 33.8 percent, and Gemini 3.8 Flash with Gemini CLI reaches 31.2 percent. GLM 5.3 with Claude Code scores 28.8 percent, Grok 4.6 and Muse Spark 1.3 tie at 23.8 percent, while Kimi K3 and GPT-5.6 Sol trail at 18.8 and 16.2 percent.
The benchmark evaluates model-and-harness combinations, not models in isolation, reflecting how enterprise engineers work. Resolution rates are averaged over eight independent runs per task, with 95 percent confidence intervals reported. This approach shows that the choice of coding tool matters as much as the underlying model.
Industry observers say the results highlight the gap between AI coding assistants and dependable enterprise engineering. Private codebases introduce hidden dependencies, business rules, and external services that remain difficult for AI to reason about. The low top score suggests human oversight will stay essential for complex production changes.
Real-SWE is likely to become a reference point for evaluating coding agents in enterprise settings. As models and harnesses improve, future versions may track progress on these harder tasks. For now, the benchmark shows that fully autonomous software engineering still faces significant obstacles.
