MaxxBench reveals performance drop in language models

Researchers have introduced a new tool, MaxxBench, to test the ability of language models to adapt to new requirements rather than simply memorizing tests. The benchmark modifies standard tasks from the DeepSWE set to evaluate real-world programming and problem-solving skills.

Testing revealed that modifying the conditions leads to a significant decline in results for popular systems. For example, the performance of the Gemini 3.8 Flash model drops to 53.4% on the new tasks compared to 73.8% on the original DeepSWE benchmark.

A total of 89 tasks were modified for the evaluation to eliminate the mechanical reproduction of memorized answers. This approach allows for a more accurate assessment of how well systems handle modified conditions in real-world development.