The openPangu-2.0-Pro language model with 505 billion parameters is presented

Developers presented openPangu-2.0-Pro — a new major language model built on the Mixture-of-Experts (MoE) architecture. The system contains 505 billion parameters and was trained using computing power based on Ascend NPU processors. The model supports a contextual window of 512 thousand tokens. The technical implementation includes MLA (Multi-Head Latent Attention) mechanisms and a 4-stream mHC architecture. To optimize the training process, developers applied the Muon algorithm, which allowed them to increase the stability and speed of weight convergence when working with such a volume of data. According to internal test results, the novelty shows superiority over non-thinking versions of models in specialized benchmarks. In particular, an increase in accuracy was recorded in HMMT and LiveCodeBench tasks, which confirms the effectiveness of the chosen architecture for solving complex logical and programming tasks. The release of the model is aimed at expanding the capabilities of working with long contexts and high-load calculations. Source data and weights are available for review on the Hugging Face platform, where the model has already received its first community ratings for performance in specialized usage scenarios.