Updated version of the SWE-Bench Pro V2 benchmark for AI agents released

In September 2026, the updated SWE-Bench Pro V2 benchmark was released, designed to evaluate the capabilities of AI agents in real-world software engineering. The best result on the new version of the dataset reached 99.4% of successfully solved tasks.

The platform is designed to test models within the context of existing engineering projects from GitHub repositories. The agent is given access to the codebase, dependencies, and a problem description, after which it must independently analyze the project, make changes, and generate a working patch without breaking the rest of the code.

During the update process, researchers faced the need to manually refine the test environment. To ensure the reliability of the results, it was necessary to fix some of the exam tasks, eliminate situations involving data leakage from the web, and adjust the operation of automated tests.

The emergence of more accurate metrics allows developers to more reliably assess the autonomy of AI systems when solving complex software development tasks and integrating code into existing work environments.