Cardiovascular disease is one of the main causes of death worldwide. Arrhythmias are an important group of cardiovascular diseases. The standard 12-lead electrocardiogram signals are an important tool for diagnosing arrhythmias. Although 12-lead electrocardiogram signals provide more comprehensive arrhythmia information than single-lead electrocardiogram signals, it is difficult to effectively fuse information between different leads. In addition, most of the current researches working on automatic diagnosis of cardiac arrhythmias are based on modeling and analysis of single-mode features extracted from one-dimensional electrocardiogram sequences, ignoring the frequency domain features of electrocardiogram signals. Therefore, developing an automatic arrhythmia detection algorithm based on 12-lead electrocardiogram with high accuracy and strong generalization ability is still challenging. In this paper, a multimodal feature fusion model based on the mechanism is developed. This model utilizes a dual channel deep neural network to extract different dimensional features from one-dimensional and two-dimensional electrocardiogram time–frequency maps, and combines attention mechanism to effectively fuse the important features of 12-lead, thereby obtaining richer arrhythmia information and ultimately achieving accurate classification of nine types of arrhythmia signals. This study used electrocardiogram signals from a mixed dataset to train, validate, and evaluate the model, with an average ofF1score and average accuracy reached 0.85 and 0.97, respectively. Experimental results show that our algorithm has stable and reliable performance, so it is expected to have good practical application potential.