Developers introduced the KaLM-Jev service for evaluating probabilities

The team behind KaLM-Jev has released a local service designed for automated evaluation of language model responses. The tool specializes in choice, score, and judgment tasks, using specialized KaLM-Reranker family models for analysis.

The system architecture involves running each model in a separate isolated process. Unlike standard chat interfaces, the service does not generate text responses, but returns structured probabilities. This approach allows the system to be integrated into automated testing and quality evaluation pipelines without the need for natural language parsing.

The project is aimed at developers who require high accuracy when comparing multiple response options. The use of KaLM-Reranker enables the system to effectively rank results based on mathematical probability metrics, rather than subjective text descriptions. The solution's source code is published on GitHub under a free license.

The implementation of such tools allows for the standardization of the LLM evaluation process, eliminating the influence of "hallucinations" or stylistic preferences of the judge model. The use of structured data instead of text reports accelerates the processing of large data sets when benchmarking new versions of neural networks.