Stanford University finds that scientific papers on LLMs become obsolete quickly

Researchers from Stanford University have analyzed how quickly academic papers on large language models become obsolete. They tested 50 frequently cited claims about LLMs against 20 current commercial and open-source models.

The results showed that 44 percent of the tested findings have already lost their relevance, and another 34 percent have significantly weakened. The average lifecycle of a scientific claim in the field of generative artificial intelligence is just 15.7 months.

Hypotheses and claims about what models are incapable of doing become obsolete the fastest. The constant and rapid updating of architectures and training methods means that the theoretical base is beginning to lag behind practical implementation.

The data obtained demonstrate to practitioners and researchers the need for constant revision of benchmarks and safety constraints. Relying on established academic papers from previous years when designing new systems is becoming ineffective.