A Novel Deep Multi-Module Fusion Network for Imagined Speech EEG Decoding

Abstract

To enhance the decoding performance of speech imagery EEG (SI-EEG) signals, a novel deep multi-module fusion network for SI-EEG decoding is proposed and evaluated on an individual SI-EEG dataset. This method introduces a spatio-temporal-frequency convolution (STFCV) module to extract rich and fine-grained multi-domain features from raw SI-EEG signals. A multi-head self-attention (MHSA) mechanism is further incorporated to emphasize critical EEG features within the SI-EEG signals. Additionally, a convolution-based sliding window data augmentation strategy is employed to enhance data diversity. A temporal convolutional network (TCN) is also integrated to effectively model temporal dependencies, thereby boosting the decoding capability for SI-EEG. Experimental results on the publicly available BCI2020 competition dataset (five-class phrase task) demonstrate that the proposed method achieves an overall average decoding accuracy of 62.93% and a Kappa coefficient of 0.537 at the individual level, outperforming the best existing approach by 3.86%. This framework considerably improves the decoding accuracy of SI-EEG, and demonstrates strong potential for practical use in speech imagery-based brain-computer interfaces.