Researchers proposed a method of neural network pruning through the Fisher metric

A group of researchers presented a new mathematical approach to optimization of neural architectures, based on the use of a differentially-geometric distance in the model parameter space. The method determines the optimal weights for removal by calculating the geodesic distance to the hypersurface where the parameter value is zero, using the Fisher metric.

In contrast to traditional magnitude pruning, the proposed algorithm takes into account the actual impact of removing a parameter on the model's performance. The researchers tested the method on fully connected networks and Vision Transformers, using the MNIST and CIFAR-10 datasets. Experiments were conducted across the entire range of parameter removal from 0% to 100% using five different random initializations.

The results showed that the new approach outperforms standard pruning methods in terms of accuracy and Matthews correlation coefficient in all tested architectures. The authors also developed a hierarchy of geodesic distance approximations, which allows for the creation of intermediate compression schemes that preserve high efficiency at reduced computational costs.

This work provides a theoretical foundation for neural network pruning processes, which previously often relied on heuristic methods. The mathematical model allows for more accurate prediction of performance degradation when removing specific neural connections, which is critical for deploying models on devices with limited resources. The study is published under the Creative Commons BY 4.0 license.