Researchers evaluated the accuracy of models in solving physics problems

A research group has published the results of testing advanced AI models, including GPT-5.6-Sol, on specialized physical benchmarks. The analysis showed that the real capabilities of the systems are significantly higher than the indicators recorded in the official reports.

During the work, experts identified systemic errors in the wording of questions and reference solutions of the CritPt dataset. After adjusting the test tasks, the pass@4 indicator for the tested models reached 94.4%. Previously, the underestimated results were due to incorrect problem conditions that prevented the neural networks from correctly interpreting physical laws.

The authors of the study emphasize that current methods for evaluating "frontier" models often do not take into account the quality of the benchmarks themselves. Correcting the testing methodology makes it possible to more accurately measure the ability of models to logical inference in narrowly specialized scientific disciplines, such as theoretical physics.

The results of the work are available in the preprint on arXiv. The discussion of the evaluation methodology and the errors found in the datasets is being held on the Hacker News platform. The obtained data indicate the need to revise the testing standards for models claiming to solve expert-level tasks.