Researchers presented a method for training models taking into account individual preferences

Scientists have introduced a new approach to training models based on feedback that takes into account the heterogeneity of human preferences instead of using a universal utility function. The study, published in a paper on arXiv, proposes an architecture for evaluating individualized utility functions that depend on the context of decision-making. As part of the work, a large dataset was collected, including more than 575,000 paired aesthetic ratings. The experiment involved 2,398 people who evaluated car wheel designs. The use of individualized models allowed for a significant improvement over the performance of standard universal utility models, including basic solutions based on foundation models. The authors have shown that the discrepancies in ratings between users are not "noise" in the data, but a reflection of significant heterogeneity of preferences. The proposed multi-stage architecture allows the model to explicitly consider whose preferences it represents, which increases the accuracy of predictions in subjective domains. The results of the study emphasize the need to collect annotator attributes for more accurate training of reward models. This approach enables the creation of systems that more accurately reflect the diversity of human choice, which is critical for the development of personalized AI tools in the future. The research materials are available under the Creative Commons BY-NC-ND 4.0 license. This work changes the approach to processing RLHF data, shifting the focus from averaged values to modeling individual user profiles.