Hugging Face introduced a methodology for optimizing speech recognition benchmarks

The team at Hugging Face has published the results of a study on optimizing the evaluation process of automatic speech recognition (ASR) systems. The experts analyzed the impact of various decoding parameters and computational strategies on the final accuracy and audio processing speed metrics.

During testing, Whisper family models were used, for which changes in the WER (Word Error Rate) metric were recorded when varying the beam size and using different search algorithms. The researchers found that optimizing decoding parameters can reduce inference time by 25-40% without losing transcription quality on standard datasets such as LibriSpeech.

Technical analysis showed that using Flash Attention and specific kernels for CUDA provides a throughput increase of 1.8 times when working with long audio files. Applying quantization to INT8 allowed reducing video memory consumption by 50%, while maintaining recognition accuracy within 0.5% of the original FP16 values.

The methodology includes using specialized tools for profiling delays at each stage of the processing pipeline. The authors emphasize that choosing the optimal batch size and caching strategy for model states is a critical factor for scaling ASR systems in production.

This study provides developers with ready-to-use configurations for fine-tuning speech recognition models to specific hardware constraints. The obtained data allows for more efficient use of computational resources when deploying models on GPU infrastructure.