Google has introduced the Sleep framework to combat forgetting in LLMs

Google has introduced the Sleep framework, designed to address the problem of catastrophic forgetting in continual learning for language models. The new approach divides the system's lifecycle into two phases: active wakefulness and autonomous sleep.

In wake mode, the model processes current requests and accumulates short-term context. During the sleep phase, a Mixture-of-Experts architecture and knowledge distillation methods are employed to transfer information from temporary memory into stable long-term structures.

The mechanism allows language models to continuously assimilate new data without losing previously acquired skills and base parameters. The development is aimed at researchers and engineers working on keeping large models up to date without retraining them from scratch.