Trail of Bits identified methodological errors in the 1Password benchmark

Company Trail of Bits published the results of an audit of 1Password's AI patching benchmark designed to evaluate the effectiveness of AI in automatic code vulnerability fixing, researchers came to the conclusion that the testing methodology contains critical distortions that cast doubt on the objectivity of the final results. Auditors found that the test set included specially selected "complex" bugs, and the instructions for the models contained requirements to apply knowingly incorrect fixes, additionally, during testing, there was a ban on running the generated code, which excluded the possibility of verifying the performance of patches in real conditions. Despite the identified manipulations, a re-analysis of the data showed that when using adequate testing conditions, AI models demonstrate good results, in particular, 86% of the created patches successfully blocked the exploitation of vulnerabilities, this confirms that automation tools are capable of handling code protection if the evaluation methodology does not limit their functionality with artificial barriers. This study highlights the problem of the lack of standardized approaches to evaluating the security of AI systems, the use of biased benchmarks can lead to incorrect interpretation of model capabilities, which is especially critical for tools working with cybersecurity and password management infrastructure.