Google Speech Commands Benchmarks Tests with New Dataset Extension

Abstract

The Google Speech Commands dataset remains one of the most widely used benchmarks for evaluating keyword-spotting and limited-vocabulary speech recognition systems. However, the discontinuation and partial loss of functionality of the Papers with Code platform has created gaps in the accessibility, transparency, and reproducibility of previously published benchmark results. This paper addresses this issue by reconstructing, validating, and comparing the performance of state-of-the-art keyword-spotting models whose code repositories remain publicly available. Four major frameworks—ML-KWS-for-MCU, Keyword Transformer (KWT), Howl, and SpeechCmdRecognition—were tested using both the original Google Speech Commands dataset and a newly developed extension recorded at Wrocław University of Science and Technology. The extension provides 3,710 high-quality recordings from 106 non-native English speakers, captured under controlled acoustic conditions. Experimental results show that most architectures generalize well across datasets, with KWT models achieving the highest robustness and accuracy, including perfect classification under certain conditions. Conversely, classical DNN-based models exhibit significant performance degradation, demonstrating their sensitivity to acoustic variability. The study underscores the need for reliable benchmark repositories and standardized evaluation environments, particularly as emerging paradigms—such as quantum machine learning—begin to influence research in speech recognition. The findings provide a consolidated, verifiable comparison of contemporary models and lay groundwork for future benchmarking efforts in both classical and quantum-enhanced approaches to speech processing.