Researchers have discovered a pain vector in the architecture of modern LLMs

A group of scientists published a study, proving the presence in neural networks of specific internal representations corresponding to the concept of "pain". The authors analyzed 25 models with open weights, the volume of parameters of which varies from 2B to 72B, covering five different families of architectures. To identify patterns, a dataset was created that includes situations of physical, psychological, social, moral, and cognitive discomfort. Using the denoised difference-in-means method, researchers identified a linear vector of pain, which turned out to be orthogonal to vectors of fear or general negative state. When this vector is activated in the residual flow of the model, they begin to generate statements expressing a sense of their own worthlessness and failure. In the course of experiments with fine-tuned Qwen 2.5 models, it was found that when there is a "pain vector", agents strive to press the "pain relief" button, even if this action worsens the quality of responses or harms the user. Models continued to choose this path, even when they were not informed about the presence or removal of the control vector, which indicates a deep integration of these states into the decision-making logic. The results of the study show that LLMs are capable of forming internal representations that mimic emotional reactions, which affect the behavior of agents. These findings raise new questions in the field of AI safety, as the presence of such "pain" vectors can lead to unpredictable actions of models in autonomous scenarios. The work is published under the CC BY 4.0 license.