Hugging Face introduced a method of fine-tuning models through GRPO

The team at Hugging Face has published the results of applying the GRPO algorithm to optimize structured output for compact language models. In the experiment, researchers fine-tuned a model with 350 million parameters using just 100 optimization steps to improve adherence to specified data formats.

The GRPO (Group Relative Policy Optimization) method allows the model to generate multiple response options for a single query, evaluate their quality, and adjust weights based on the relative advantage of the best results. In this case, the model was trained to strictly follow the JSON format, which is critical for integrating AI into software interfaces. The use of the TRL (Transformer Reinforcement Learning) library made it possible to reduce computational costs during fine-tuning.

The results showed that even with a small number of iterations, the model demonstrates a significant increase in syntax compliance accuracy. The application of LoRA adapters in combination with GRPO made it possible to avoid degradation of the model's basic skills while maintaining high generation speed. The use of 100 steps was sufficient for the model to start stably producing valid data structures without the need for lengthy SFT training.

This approach opens the way to efficient adaptation of small models to specialized tasks that require a predictable output format. The technology allows developers to reduce resource requirements when creating specialized agents that work with APIs and databases, where a single character error makes the response unsuitable for further processing.