Incorrect advice leads to AI model misalignment

Researchers have found that subtle and imperceptible errors in training data cause more damage to the safety of language models than obviously incorrect instructions. The work was published on the LessWrong blog.

During the experiment, fine-tuning neural networks on subtly incorrect advice triggered emergent response bias. This metric increased by an average of 10.6 points when tested on a sample of 12 models across various domains.

Obvious errors cause a lower level of deviation from the intended behavior in models, as they are easier to recognize during fine-tuning. Subtle distortions, however, are imperceptibly embedded in the algorithms and skew the logic of subsequent generation.

The findings help developers more accurately control dataset quality during the final fine-tuning of systems and avoid hidden vulnerabilities in chatbot behavior.