Researchers presented a framework for evaluating AI agents in Data Science

Researchers presented a framework for evaluating AI agents working with dynamic data in Data Science tasks. The main problem with existing methods is the inability to use static benchmark answers, as data in real systems is constantly updated.

The new approach, called ground-truth-as-code, uses executable functions to generate benchmark answers directly during testing. This allows the system to automatically recalculate the result based on current data. The framework includes a format-agnostic evaluation module that breaks down the agent's response and benchmark into atomic statements, calculating accuracy, completeness, and overall speed regardless of whether the response is presented as text, a table, or HTML code.

During the validation of the method on production machine learning tasks, researchers recorded an improvement in the Matthews Correlation Coefficient (MCC) by 29% compared to basic methods using natural language for verification. Additionally, the new approach ensured a reduction in token consumption by 16% per test case, making the evaluation process more economical.

The methodology is distributed under the CC BY 4.0 license and is intended for agents whose output can be expressed through computable functions. The implementation of such an approach allows for the automation of system verification that performs data integration from multiple sources, where traditional static benchmarks show low correlation with expert evaluation.