A synthetic dataset QVAC Genesis III is presented for training STEM models

Researchers have published QVAC Genesis III — a large-scale synthetic dataset designed for efficient training of language models in the field of STEM. The dataset contains 191.43 billion tokens, covering 19 different domains with varying levels of complexity and educational styles. The dataset was created using a dual-generation strategy based on knowledge distillation from a "teacher" to a "student". The system analyzes errors of compact models, converting them into corrective explanations, and successful responses are supplemented with contrastive reasoning for all options. To evaluate the quality of the data, the authors implemented the LLM-as-a-parser protocol, which tracks the accuracy and validity of responses in a free format. During testing on models with 1.7 billion parameters trained from scratch, QVAC Genesis III demonstrated an advantage over the open corpus Cosmopedia-v2, with a performance increase of up to +28.57% on ARC-E and +21.35% on ARC-C, while the valid answer rate reached 99.45%. The research materials and data are distributed under the Creative Commons Attribution 4.0 license. The development is aimed at optimizing the training of compact models for operation on peripheral devices, where the token budget and computational resources are strictly limited.