Researchers presented the TranSGrid benchmark for evaluating system generalization

A group of researchers presented TranSGrid — a new test set designed to evaluate the ability of neural networks to systematic generalization through the combination of deductive, inductive, and abductive types of reasoning. The authors of the work claim that existing benchmarks oversimplify the task, relying on linear composition of actions and explicit goals, which does not allow for a full check of the cognitive skills of models.

In the course of experiments, seven different Transformers architectures were tested on 4800 tasks from the TranSGrid set. The results showed a significant gap in performance: if on standard test samples the largest model shows an accuracy of 79.6%, then in TranSGrid conditions this figure drops to 55.3%. On the most complex subsets of tasks, the model's efficiency decreases to 15.8%.

The study proves that current evaluation methods, focusing exclusively on productivity, are insufficient for measuring real systematic generalization. When the authors artificially simplified the tasks in TranSGrid, making the composition of actions linear or setting goals explicitly, the models' success rates returned to their usual high values. This confirms that it is the inductive and abductive requirements that are the critical barriers for modern systems.

The work highlights the need to transition to more complex testing methods that require neural networks to simultaneously apply all three forms of logical inference. The obtained data indicate that current architectures still have difficulty solving fundamentally new tasks that require combining known elements in non-standard conditions.