· Updated
Traditional coding benchmarks have become saturated, presenting a misleading picture of AI coding ability — developers spend only 16% of their time writing code, per IDC. The remaining 84% is deployment, monitoring, and debugging: exactly what APEX-SWE measures. Models that top 75% on SWE-bench Verified drop to under 42% here.
If you're deciding how much to trust AI agents for production engineering — not just code writing — APEX-SWE gives you the number that matters. The eval harness and a 50-task dev set are open-source on GitHub and Hugging Face for anyone testing models against real work.
Traditional coding benchmarks do not reflect how software is actually built and maintained. That's why we built a new benchmark, APEX-SWE, in partnership with @cognition. It measures whether AI models can perform complex, real-world software engineering work to ship systems https://t.co/5nJfKQBxfA
More like this
CognitionApr 15
OpenAIMar 15