Allen Institute introduced BenchMIRT for analyzing benchmark quality

Researchers from the Allen Institute for AI introduced the BenchMIRT methodology designed to assess what modern large language model benchmarks actually measure. The authors analyzed the structure of popular tests and identified critical dependencies between the task format and final performance metrics. In the course of the study, experts applied statistical analysis to 15 popular benchmarks, including MMLU and GSM8K. It turned out that up to 40% of the variability in model results is explained not by their reasoning abilities, but by the features of response formatting and sensitivity to prompt patterns. The toolkit allows for a quantitative assessment of data "pollution" and the impact of random factors on the final score. The developers of BenchMIRT propose using a new approach to decompose metrics. Instead of evaluating the overall score, the system breaks down the metrics into the ability to extract knowledge, logical inference, and robustness to changes in wording. This makes it possible to identify cases where the model simply "memorized" answers from the training sample, rather than learning to solve problems. The results of the work show that current evaluation methods often overestimate the capabilities of models by 15–25% in logic tasks. Researchers emphasize the need to transition from static question sets to dynamic tests that change the task conditions in real-time, eliminating the possibility of simple matching with a database. This approach changes the standards for verifying AI systems, emphasizing the transparency of the evaluation process. The use of the proposed methodology will allow developers to more accurately determine the real progress of models, separating marketing indicators from the actual ability of systems to generalize knowledge and perform complex instructions.