Researchers applied linear programming to correct LLM weights

Researchers have introduced a method called Astra, which uses linear programming for targeted modification of neural network weights. The technology allows for correcting logical errors in models without resorting to full fine-tuning or changing the architecture.

During the experiment, the authors applied the approach to the Qwen3.5-0.8B model, which exhibited cyclical errors when solving mathematical problems. By changing the direction of the residual stream, they managed to increase the model's accuracy on the MATH-500 benchmark from 7.9% to 48.1%. In doing so, only 0.016% of the total number of weights were modified.

The method demonstrates the possibility of "surgical" intervention in the operation of pre-trained models. Instead of adjusting the entire weight matrix, the algorithm finds the minimum necessary change that eliminates a specific behavioral defect. This allows for correcting errors in reasoning logic while preserving the model's general knowledge and parameters.

For practitioners, this means the emergence of a new way to debug neural networks that requires significantly fewer computational resources than classic fine-tuning. The technique can be useful for fixing specific bugs in compact models, where it is critical to maintain original performance with minimal interference in the weight structure.