Surge AI has introduced the DAYJOB benchmark for evaluating tax AI agents

The company Surge AI has released a specialized benchmark, DAYJOB: Tax, designed to test the capabilities of AI agents in solving professional tax-related tasks. The dataset includes 50 complex cases covering eight different areas of tax law and reporting.

During initial testing of models on these tasks, a low success rate was recorded. The leader among the tested systems was the Claude Sonnet 5.5 model, which achieved 10.8% correct answers. Other models showed even lower results, highlighting the difficulty of automating highly specialized financial and legal processes.

The benchmark is focused on verifying the ability of agents to work with real-world professional requirements, where the cost of error is critically high. The evaluation methodology includes checking the accuracy of calculations, the interpretation of tax legislation, and compliance with formal documentation requirements.

For AI system developers, this tool will serve as an indicator of how ready current models are for autonomous work in the field of tax consulting. The test results show a significant gap between the general capabilities of LLMs and the accuracy requirements necessary to fully replace a specialist in this field.