Researchers presented a benchmark for recommending SNOMED CT concepts

A group of researchers published a benchmark for recommending SNOMED CT concepts, based on the task of entity linking in clinical texts. The work uses data from the MIMIC-IV-Note dataset, which includes 272 discharge summaries, to automate the standardization of medical terminology.

The dataset contains 75,491 annotations, divided into 204 training and 68 test notes. During the experiment, target mentions were masked from the context, after which the system ranked available concepts. The authors compared several approaches, including TF-IDF, latent semantic analysis, and hybrid methods using database search.

The best result was shown by the sparse TF-IDF method, which achieved a Recall@1 of 14.81% and Recall@10 of 33.43%. At the same time, the MRR metric was 0.2114, and nDCG@10 was 0.2297. The use of retrieval-augmented search did not lead to an improvement in the baseline metrics, showing Recall@10 at 31.99%.

The study revealed a critical dependence of accuracy on the frequency of concept mentions in the training sample. For terms that occur more than 10 times, the Recall@10 reaches 43.90%, while for rare concepts (1-2 mentions) it drops to 7.74%. About 9.66% of test pairs contain concepts that were absent during training.

The obtained data highlights the importance of local lexical context for working with medical ontologies in conditions of limited resources. The results of the work are available under the CC BY-SA 4.0 license and serve as a reproducible standard for future developments in the field of biomedical encoders.