Researchers presented the Brood War benchmark for evaluating AI in StarCraft

Developers have published Brood War Bench — a specialized set of tests for evaluating the skills of language models in managing game units in the strategy StarCraft: Brood War. During testing, the ability of agents to make tactical decisions and coordinate army actions in real-time was checked. The winner of the tests was the Codex Astra model, which demonstrated a 100% win rate within the given scenarios. The overall benchmark result was 49 points, which took 6 hours of machine time to complete. The testing showed that current models still experience serious difficulties with microcontrol and strategic planning. The results showed that most participants work at the level of beginner players. For example, the Grok model was unable to implement the basic mechanics of army deployment, and Claude Fable, despite its claimed ambitions in management, encountered critical errors when executing commands. A detailed analysis of each model's performance is available in the discussion on Hacker News. The creation of such environments is necessary for evaluating the cognitive abilities of models in dynamic conditions, where not only text generation is required, but also consistent execution of actions. Unlike static tests, Brood War Bench requires AI to take into account the constantly changing game situation and manage resources within a tight time limit.