Researchers have discovered hidden GPT-2 models within Qwen architectures

Researchers analyzed the internal mechanisms of modern LLMs and identified signs of the presence of embedded copies of earlier models. In the course of experiments, it was found that Qwen neural networks are able to predict text continuations that are statistically closer to the outputs of the GPT-2 model than to their own natural responses of the architecture. In the framework of testing, the authors compared the logits and hidden states of models when generating identical prompts. The results showed that Qwen activations demonstrate a high correlation with the internal representations of GPT-2 at certain stages of data processing. This indicates that in the process of training on huge arrays of data, including texts generated by old models, modern LLMs form specific internal structures to imitate the behavior of their predecessors. This phenomenon confirms the hypothesis that models do not just absorb facts, but also remember stylistic and logical patterns of other algorithms encountered in the training sample. This creates a matryoshka effect, where simplified models are hidden inside a complex architecture, activated by certain contextual requests. The detection of such hidden components allows for a new look at the interpretability of neural networks. Understanding how modern models emulate the work of old systems can help in the development of methods to combat hallucinations and improve the accuracy of predictions, excluding the influence of learned patterns of less perfect algorithms.