Researchers analyzed the evolution of benchmarks for evaluating LLM

A group of scientists published a large-scale study dedicated to the transformation of approaches to testing large language models. The authors analyzed 14,767 scientific papers published on the arXiv platform from January 2022 to August 2026 to identify how the requirements for evaluating AI performance are changing.

The analysis showed a steady shift towards tasks related to active interaction and performing professional functions. Researchers note that the use of LLMs as tools for automatic evaluation of results is becoming a standard for both agent and non-agent systems. At the same time, the application of models for generating test materials themselves does not demonstrate similar stable growth in the latest cohorts of research.

The work systematizes changes in target domains, testing conditions, and scoring mechanisms. The authors emphasize that modern benchmarks are increasingly including elements that require models to perform complex sequential actions, rather than just answering questions. This reflects the industry's transition from evaluating static knowledge to checking the ability of models to function in real working environments.

The results of the study question the objectivity of current evaluation methods. Scientists warn that the active involvement of neural networks in creating and testing tests can lead to "looping," where benchmarks begin to reproduce the preferences and blind spots of the models themselves involved in their development. The work is distributed under an arXiv license and provides a basis for revising the standards for validating AI systems.