Researchers have proven the impact of SFT on the diversity of LLM responses

Researchers published a work Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs in which they analyzed the reasons for the loss of diversity of responses in large language models. The authors found that the tendency of models to mode collapse or over-dispersion directly depends on the quality and volume of data for fine-tuning. In the course of the experiment scientists used three types of models: small transformers on synthetic languages, four LLMs fine-tuned on survey results, and the same models tuned on the CodeNet dataset. To evaluate the results the metric of the probability of coincidence of responses to identical prompts was applied. It turned out that with a sufficient amount of data for SFT supervised fine-tuning the diversity of the model's responses tends to the indicators of the target distribution. Theoretical analysis showed that the absolute gap in diversity is limited by the square root of the KL divergence Kulback-Leibler divergence between the model and the target distribution. This means that a model approximated to the optimal state by cross-entropy cannot demonstrate arbitrarily distorted diversity of responses. The authors' mathematical model proves that SFT has no built-in bias towards uniformity or chaos. Generation errors occur due to the limited sample size during training. Increasing the volume of target data allows for effective adjustment of the model's calibration bringing its behavior closer to human or given reference responses. The results of the study provide a new tool for diagnosing models. Now developers can use the proposed formula to assess how much the current fine-tuning process contributes to the degradation of diversity and determine the required amount of data to achieve stable results.