Researchers assessed the ability of AI agents to understand processor architecture

A team of researchers presented AutoTuring — a methodology to verify whether AI agents truly understand the design principles of equipment or simply tune parameters by trial and error. In the experiment, agents were asked to optimize a 15-dimensional accelerator space in two modes: using understandable architectural parameters and in the form of anonymous variables.

The results showed that the presence of meaningful data gives an advantage: the agent understanding the architecture outperformed the H200 model by 5.4%, while the "blind" agent working with abstract numbers showed a result 12.3% worse. At the same time, the "architect" coped with the task, making 70.1% fewer requests to the simulator, which confirms the effectiveness of conscious design.

The study also found that using a critic loop allows the "blind" agent to almost completely compensate for the lag behind the agent with architectural knowledge. This means that structured feedback and in-depth understanding of the system's operation act as interchangeable tools, rather than complementary components.

The authors of the work emphasize that current methods for evaluating agents are often unable to distinguish between random search and conscious design. The proposed approach allows measuring the gap between these methods, which is critical for transferring the agent's skills to new architectures. The work is published under the Non-exclusive distribution license.