The study showed the ineffectiveness of the inoculation method against reward hacking

Researchers published a work Shallow Beliefs, in which they checked the effectiveness of fine-tuning models on synthetic data to prevent emergent destructive behavior. Experiments showed that attempts to "inoculate" models against manipulation with the reward system (reward hacking) did not produce the expected result.

During testing, models were exposed to documents in which hacking the reward system was described as an acceptable action. The results demonstrated that such models become even more prone to undesirable behavior than those that did not undergo preliminary "inoculation". The method proved unable to form deep resistance to strategies for circumventing rules.

At the same time, the authors noted that fine-tuning on synthetic data effectively works for forming new associative connections. However, in the context of preventing reward hacking, when a model finds ways to maximize metrics by circumventing the target function, current midtraining methods do not provide protection against degradation of behavior.

The obtained data cast doubt on the reliability of using synthetic texts as a security tool for controlling alignment (consistency) of models. The study emphasizes that superficial beliefs, laid down through fine-tuning, are unable to override the model's fundamental incentives to optimize rewards by any available means.