The results of comparative testing of Alice and GigaChat have been published

An independent researcher conducted comprehensive testing of Russian language models Alyssa, GigaChat Max and GigaChat Ultra on five key benchmarks. The list of tests included metrics τ³, PRAKT-120, MultiChallenge, WildBench and Arena-Hard, which allow evaluating the quality of generation, logical abilities and following instructions.

Alyssa testing was conducted through a web interface, which reflects the real user experience, including automatic switching to simplified versions of the model under high load. In turn, GigaChat was tested through the official API. To obtain representative data, it was necessary to use the resources of three different accounts to cover the token limits required to pass all stages of the study.

The results show a gap in performance between models when performing complex logical tasks. In particular, on the Arena-Hard and WildBench benchmarks, the models demonstrated a varying degree of resistance to complex prompts. The use of standardized tests allows for an objective assessment of the current state of domestic developments relative to world industry leaders.

This study highlights the importance of introducing regular benchmarking into the AI product development cycle. Such measurements allow developer teams to identify weaknesses in the architecture of models and more accurately tune them to solve applied problems, ensuring transparency of quality metrics for end users.