Researchers presented the CommentBench benchmark for evaluating AI comments

Researchers have published CommentBench — a specialized test set for evaluating the ability of language models to imitate human comments on AI safety publications. The project's goal was to determine how accurately modern systems can reproduce the argumentation and style of an expert community.

During the testing of various architectures, the best result was an accuracy of 8.3%, which was demonstrated by the Fable 5 model. The other tested models showed significantly lower indicators, which indicates the complexity of the task of generating meaningful and contextually correct responses in specialized discussions.

The CommentBench methodology involves comparing generated text with real user comments on the LessWrong platform. Researchers note that the performance of models remains stable when changing generation settings, however, the overall level of correspondence to the human style remains extremely low even for advanced systems.

The research results highlight the gap between the models' ability to generate general text and their ability to conduct in-depth discussions on safety issues. This benchmark allows for quantitative measurement of progress in the field of understanding context and argumentation, which is critical for developing systems capable of participating in expert discussions.