Researchers identified a four-stage process for solving math problems in LLM

A group of scientists published a study describing the internal architecture of logical inference of large language models when solving mathematical problems. The authors found that the data processing process in neural networks passes through four consecutive stages: schema abstraction, operation planning, operand binding, and final calculation.

Each of these stages corresponds to the work of certain layers of the model, forming unique intermediate representations. Researchers found that the high accuracy of models in standard tests drops sharply when even one irrelevant sentence is added to the problem statement. This phenomenon of "fragility" was localized at the operation planning stage.

Analysis showed that specific attention heads are responsible for the failures, whose role in the process was confirmed by bidirectional causal testing. When distracting data is fed, this block stops correctly processing the logical structure of the problem, leading to errors in the final answer, despite the model's overall ability to perform calculations.

This work provides a mechanistic interpretation of how LLMs "reason" when working with numbers. Understanding the localization of errors at the level of specific layers and weights allows developers to precisely adjust the architecture of models to increase their resistance to noise in prompts. The study is published under the Creative Commons Attribution 4.0 license.