Baidu has introduced the ERNIE 5.5 Flash Base model

Baidu has released a new language model, ERNIE 5.5 Flash Base. The architecture of the new model is a sparse MoE (Mixture-of-Experts) model with a total of 280 billion parameters, while the number of active parameters per token is 14 billion.

The model was pre-trained on a dataset of 33.5 trillion tokens. One of the key features of the release is support for a context window of up to 1 million tokens. According to the developers, these parameters ensure high efficiency when performing fine-tuning and inference tasks across various use cases.

Technical details and documentation for the model are available on the official ERNIE 5.5 Flash Base page. The release is aimed at developers who need a balance between the performance of a large model and the speed achieved through a sparse architecture.

Using 14 billion active parameters significantly reduces computational costs while maintaining the broad capabilities characteristic of models with hundreds of billions of weights. This solution simplifies the integration of advanced language features into high-load systems that require real-time processing of long contexts.