Researchers identified a fivefold disparity in cost when using AI agents

Researchers from UC Berkeley and Arena conducted a comparative analysis of seven different models working as part of popular coding tools. During testing, Claude Code, Codex CLI, and Pi were evaluated to determine the relationship between the choice of software environment and the final costs of task execution. Results showed that with identical success rates in completing tasks on two key benchmarks, the cost of using the same model can differ by up to 5 times. The difference is due to the characteristics of the harness implementation - the infrastructure layer that manages context, prompts, and model interaction with the file system. During the experiment, it was found that inefficient organization of requests and redundant use of tokens in certain tools significantly increase financial expenses without improving code quality. This raises questions about the practice of evaluating AI agents solely by the name of the model used, ignoring the architecture of their integration. This study highlights the need for standardization of efficiency metrics for AI tool developers. Now, when choosing a solution for programming automation, users should consider not only the capabilities of the LLM itself but also the optimization of the specific agent, as overhead costs for context management can become the main cost factor for system operation.