Spectral Cross-Domain Neural Network with Soft-adaptive Threshold
  Spectral Enhancement
Electrocardiography (ECG) signals can be considered as multi-variable
time-series. The state-of-the-art ECG data classification approaches, based on
either feature engineering or deep learning techniques, treat separately
spectral and time domains in machine learning systems. No spectral-time domain
communication mechanism inside the classifier model can be found in current
approaches, leading to difficulties in identifying complex ECG forms. In this
paper, we proposed a novel deep learning model named Spectral Cross-domain
neural network (SCDNN) with a new block called Soft-adaptive threshold spectral
enhancement (SATSE), to simultaneously reveal the key information embedded in
spectral and time domains inside the neural network. More precisely, the
domain-cross information is captured by a general Convolutional neural network
(CNN) backbone, and different information sources are merged by a self-adaptive
mechanism to mine the connection between time and spectral domains. In SATSE,
the knowledge from time and spectral domains is extracted via the Fast Fourier
Transformation (FFT) with soft trainable thresholds in modified Sigmoid
functions. The proposed SCDNN is tested with several classification tasks
implemented on the public ECG databases \textit{PTB-XL} and \textit{MIT-BIH}.
SCDNN outperforms the state-of-the-art approaches with a low computational cost
regarding a variety of metrics in all classification tasks on both databases,
by finding appropriate domains from the infinite spectral mapping. The
convergence of the trainable thresholds in the spectral domain is also
numerically investigated in this paper. The robust performance of SCDNN
provides a new perspective to exploit knowledge across deep learning models
from time and spectral domains. The repository can be found:
https://github.com/DL-WG/SCDNN-TS
Che Liu, Sibo Cheng, Weiping Ding, Rossella Arcucci
# Spectral Cross-Domain Neural Network with Soft-adaptive Threshold Spectral Enhancement

###### Abstract

Electrocardiography (ECG) signals can be considered as multi-variable time-series. The state-of-the-art ECG data classification approaches, based on either feature engineering or deep learning techniques, treat separately spectral and time domains in machine learning systems. No spectral-time domain communication mechanism inside the classifier model can be found in current approaches, leading to difficulties in identifying complex ECG forms. In this paper, we proposed a novel deep learning model named Spectral Cross-domain neural network (SCDNN) with a new block called Soft-adaptive threshold spectral enhancement (SATSE), to simultaneously reveal the key information embedded in spectral and time domains inside the neural network. More precisely, the domain-cross information is captured by a general Convolutional neural network (CNN) backbone, and different information sources are merged by a self-adaptive mechanism to mine the connection between time and spectral domains. In SATSE, the knowledge from time and spectral domains is extracted via the Fast Fourier Transformation (FFT) with soft trainable thresholds in modified Sigmoid functions. The proposed SCDNN is tested with several classification tasks implemented on the public ECG databases _PTB-XL_ and _MIT-BJH_. SCDNN outperforms the state-of-the-art approaches with a low computational cost regarding a variety of metrics in all classification tasks on both databases, by finding appropriate domains from the infinite spectral mapping. The convergence of the trainable thresholds in the spectral domain is also numerically investigated in this paper. The robust performance of SCDNN provides a new perspective to exploit knowledge across deep learning models from time and spectral domains. The repository can be found: [https://github.com/DL-WG/SCDNN-TS](https://github.com/DL-WG/SCDNN-TS)

 Deep Learning; Spectral Domain Neural Network; ECG Signal; Medical Time-Series; Cross-domain Learning

## I Introduction

Time Series (TS) has been widely studied in research fields such as physics, finance, medical, and nature language processing [1]. Due to the time dependency [2], classifying TS is different from traditional classification tasks involving image or sequence classifications without time dependencies [3]. The performance of traditional classification tasks often relies on the number of samples with proper labels [4]. In fact, well-labelled TS is often out of reach in real applications as stated in [2]. Moreover, the issue of imbalanced datasets commonly exists in medical TS data [4]. As pointed out by [3], another critical challenge of Time Series Classification (TSC) involves the model generalization. It is found that different models are often required for distinct classification tasks even with the same input TS data [5].

The state-of-the-art TSC approaches are either based on handcrafted Feature Engineering (FE) [2, 6, 7, 8, 9] or Deep Learning (DL) methods [10, 11, 12, 13, 14]. The former can be divided into three main categories, namely statistical feature extractions [15], entropy-based methods [16] and frequency-based methods [17]. After preprocessing [18], traditional machine learning classifiers such as Gradient Boosting Decision Tree (GBDT), Support Vector Machine (SVM) and Random Forest (RF) are employed to accomplish the classification tasks using handcrafted features as model input. However, the performance of FE-based methods is extremely sensitive to the quality of feature extractions on a case-to-case basis [19]. Furthermore, distinct FE methods are often required for different TSC tasks even for the same dataset [20]. Thus, there is an insurmountable obstacle to extending specific FE methods to general classification tasks.

Such challenges can be found in medical TSC as stated in the work of [21, 22]. Electrocardiography (ECG) is a specific type of medical TS that describes the heartbeats of 12 different leads to detect various aspects of heart health [23]. In clinical applications, ECG data are widely used to diagnose cardiovascular diseases, such as myocardial infarction(MI), hypertrophy(HYP), and Conduction Disturbance(CD) [24]. A standard ECG TS consists of three waves, known as P-wave, QRS-complex and T-wave respectively [25]. As recognized in numerous researches [26, 27, 28], various cardiovascular diseases will impact these three waves, resulting in unrecognizable signals (also called repolarization in medical science). Thus identifying these abnormal signals is crucial for clinical diagnosis. In fact, successful ECG classification methods not only improve the accuracy of cardiology diagnosis, but also enable the possibility of monitoring the state of human health state using wearable devices.

Classical FE-based TSC approaches extract the features relying on different waveforms of 12-lead ECG, mostly focusing on the P-QRS-T wave [25] and RR-interval [29]. FE is carried out to extract statistical, energy-based and frequency-based features which are used to classify ECG TS via traditional Machine Learning (ML) classifiers [6, 7,
[8, 9]. In FE based methods, the quality of the features and thus the classification performance is sensitive to the algorithms selected for feature computation [30]. Therefore, specific FE algorithms often need to be designed for different classification tasks.

In recent years, much research attention has been given to applying DL approaches for ECG classification, and more generally, for TSC tasks. For example, Convolutional Neural Networks (CNNs) have been widely employed to classify 12-lead ECG signals [11, 12, 13, 14], improving the accuracy of disease diagnosis in comparison to traditional approaches. The works of [31, 32] have used Long Short-term Memory (LSTM) [33], a variant of Recurrent Neural Network (RNN), to classify ECG pattern with imbalanced data. A wavelet layer before LSTM has been added in [34] to merge Spectral Domain (SD) knowledge into the Neural Network (NN). In their work, the spectral operation has been performed only on the input ECG TS, instead of processing deep knowledge inside the NN.

Recent works of [35, 36] have used multi-scale Deep Convolutional Neural Network (DCNN) with ensemble learning to detect heart arrhythmia from 12-lead ECG. MLFB-Net [37] with CNN concatenated bidirectional GRU [38] and attention mechanism [39] has also been implemented for ECG classification where each lead ECG is treated individually. In addition, the work of [40] has tried the Transformer structure with an attention mechanism to capture latent and deep knowledge from 12-lead ECG simultaneously. However, their works only consider the ECG signals in time domain, and neglect the valuable information embedded in the spectral domain. The knowledge representation on the SD of images has been introduced in the work of [41]. Following this idea, [42, 43, 44, 45] have further explored the SD representation of image data on CNNs. Nevertheless, none of the aforementioned studies has established the links between the Time Domain (TD) and the SD inside the NN. In other words, the spectral information is either processed outside the neural network [41, 42] or inside the NN but without connections to the TD. The very recent works of [46, 47] have extended SD knowledge to an infinite mapping space to approximate the proper space with Fourier Neural Operator (FNO) for image prediction tasks. In [48], FNO has been utilized to mix multi-scale features in the SD to reach a more general representation of images. However, in these works, the dimension of the SD is fixed and only low-frequency domain information is kept in the neural network. Hence, it is extremely challenging to obtain the proper SD due to the pre-fixed threshold, and the high-frequency domain is ignored. The latter may lead to potential information loss due to the hard thresholds [49].

As discussed, although much effort has been dedicated to time series classification involved spectral knowledge, the noted algorithms suffer from the following limitations and challenges:

1. Current approaches either process the spectral information outside the deep learning model or treat time and spectral domains separately in the neural network. No communication between time and spectral domains inside the neural network has been performed;
2. When filtering information in the spectral domain, pre-selected number of spectral modes are employed, leading to potential information loss.

To overcome these bottlenecks, we propose a novel neural network structure, named Spectral Cross-domain neural network (SCDNN), that interacts the TD and the SD inside the NNs. In addition, we have developed a new soft self-adaptive threshold determination mechanism that is deployed in the Soft-adaptive Threshold Spectral Enhancement (SATSE) block of Spectral Cross-domain Neural Network (SCDNN). More precisely, this new mechanism enables SCDNN to find the optimal number of spectral modes instead of using pre-fixed thresholds as implemented in existing approaches. Furthermore, the information loss of spectral filtering can be decreased thanks to the controllable soft threshold. To verify our model's performance and versatility, four diverse ECG classification tasks are deployed. These tasks consist of cardiovascular disease and ECG Form classifications on the public database \(PTB-XL\) and \(MIT-BIH\) with different sub-datasets. The former, published recently in 2020, contains the most detailed ECG lables while the latter is considered as one of the largest openaccess ECG datasets. SCDNN achieves substantial enhancement compared to existing TS classification models, in relation to almost all comparison metrics evaluated. In particular, numerical results demonstrate that SCDNN contributes to tackle the bottleneck of identifying ECG patterns with complex waveforms (without necessarily pathological symptoms) on the \(PTB-XL\) database.

In summary, the principle contributions of our work are listed below:

1. We designed a novel neural network structure, named SCDNN, to establish the communication of spectral and time domains inside NNs. More precisely, Fast Fourier Transformation (FFT) and Inverse Fast Fourier Transformation (IFFT) are added on the ResNet backbone after each Res block to entract the information in the spectral domain.
2. In the SATSE block of SCDNN, trainable soft thresholds in the spectral domain enable the proposed model to find optimal spectral modes and reduce the information loss compared to using pre-fixed thresholds.
3. We have tested SCDNN in four different ECG classification tasks performed on the public ECG database \(PTB-XL\) and \(MIT-BIH\). SCDNN outperforms the state-of-the-art works on all classification tasks with imbalanced data. The method proposed in this study is data-agnostic, and can be applied/extended to other time-series classification/prediction problems.

The rest of this paper is organized as follows. Section II describes the proposed SCDNN with a detailed explanation of information processing in the spectral domain. The numerical results of the proposed network, compared to state-of-the-art approaches, on the \(PTB-XL\) and \(MIT-BIH\) databases are
presented in Section \(3\) and \(4\), respectively. We end the paper with a conclusion in Section \(5\) where we have also mentioned potential future works.

## II SCDNN: methodology

In this section, we introduce the workflow of the proposed SCDNN with a special focus on the SATSE block. The proposed model is composed of ResNet18 [50] backbone, four SATSE blocks, one adaptive average pooling layer [51], one adaptive max pooling layer [52] and one fully-connected layer as displayed in Fig 1. The backbone includes \(4\) Res blocks (also known as residual blocks [50]). The structure of each block is depicted in Fig 2 where all Res blocks used in this study have the same structures with a different number of channels, following the standard ResNet structure [50]. Each Res block is combined with one SATSE block to form a Convolutional Fourier (ConvF) component as illustrated in Fig 2. The SATSE block enables the NN to learn from the spectral domain.

In each ConvF block, the output of SATSE will be added to the output of the Res block to avoid the information loss from spectral learning [53] as illustrated in Fig 1. Two pooling layers are applied individually on the output of the last ConvF block for dimension reduction. Two pooled features are concatenated with channel dimension, then flattled to a \(1D\) vector before passing to the fully-connected layer for classification.

### _Res Block_

Since untrainable issues have been reported for very deep neural networks [54], ResNet was proposed by [50] where skip connections between various convolutional layer were adopted to decrease the issue of information vanishing in deep layers of neural networks. A light version of ResNet, namely ResNet18 [55] is chosen in this work as the backbone. Fig 2 depicts the layout of each Res block, consisting of four \(1D\) convolutional layers with a common kernel size and stride as defined in [50]. The convolutional layers are employed to extract features across all channels. To enhance the gradient stabilization, a \(BatchNorm\) layer [56] is deployed after each convolutional layer followed by a \(ReLU\)[57] activation function to avoid the gradient vanishing in deep neural networks [58].

### _Satse_

In this section, we explain in details the soft adaptive threshold and the spectral learning in SD. The pipeline of SATSE is illustrated in Fig 3. First, FFT is deployed in SATSE for spectral knowledge converting. The model is thus capable of capturing deep features in TD and SD simultaneously. To choose the most proper spectral domain, we make use of trainable thresholds in each SATSE block to select the spectral modes. The soft threshold mechanism is capable of avoiding information loss in modes selection since the threshold value is obtained through network back propagation. Additionally, as shown in Fig 3, a trainable weight matrix is added to establish connections among different spectral modes. Therefore, those spectral modes in different SATSE blocks are determined mutually. For the \(i^{th}(i=1,...,4)\) SATSE block in SCDNN, the output of the Res block is denoted as \(\{f_{i,k}^{(j)}\}\) where \(\{j,k\}\in\{0,...,L_{i}-1\}\times\{0,...,C_{i}-1\}\). Here \(C_{i}\) and \(L_{i}\) are the number of channels and the signal length respectively. Let \(\mathcal{F}\) and \(\mathcal{F}^{-1}\) denote the FFT and IFFT on discrete signals, the computation of SD features \(f_{i,k}^{S,(j)}\) of each SATSE block

Fig. 1: Workflow of SCDNN where \(\lambda_{L}\) and \(\lambda_{H}\) denote the coefficients corresponding to low-frequency and high-frequency information

Fig. 2: Res Block Architecture
and each channel is performed via discrete Fourier transform,

\[f_{i,k}^{S,(j)} =\sum_{n=0}^{L_{i}-1}e^{-\frac{i}{2\pi}nj}f_{i,k}^{(n)}. \tag{1}\] \[\mathcal{F}(f_{i,k}) =[f_{i,k}^{S,(j)}]_{j=0,..,L_{i}-1} \tag{2}\]

For the sake of clarity, \(\hat{i}\) denotes the imaginary unit in \(e^{-i\frac{2\pi}{1}}\) while the index \(i\) (e.g., \(f_{i,k}\)) is related to the order of the SATSE block. To compute the soft trainable thresholds, modified sigmoid functions in the high- and low- frequency domain are defined respectively as :

\[\sigma^{L}(x) =1-\frac{1}{1+e^{\gamma_{i}(-x+\varphi_{i}\cdot L_{i})}} \tag{3}\] \[\sigma^{H}(x) =\frac{1}{1+e^{\gamma_{i}(-x+\varphi_{i}\cdot L_{i})}} \tag{4}\]

where \(\varphi_{i}\) denotes the trainable threshold ratio and \(\gamma\) controls the slope of the modified sigmoid function. The dual sigmoid functions \(\sigma^{L}\) and \(\sigma^{H}\) enable the SATSE block to capture knowledge in both low- and high-frequency domains. The use of soft thresholds in Eq (4) enables the NN back-propagation compared to hard threshold functions, i.e.,

\[\sigma^{L}_{\text{hard}}(x) =\mathbb{1}_{x>\varphi_{i}L_{i}} \tag{5}\] \[\sigma^{H}_{\text{hard}}(x) =\mathbb{1}_{x\leq\varphi_{i}L_{i}}. \tag{6}\]

We then obtain filtered high- and low-frequency elements in the spectral domain via

\[\widetilde{f_{i,k}^{S^{H}}}=\sigma^{H}(k)f_{i,k}^{S},\quad\ \widetilde{f_{i,k}^{S^{L}}}=\sigma^{L}(k)f_{i,k}^{S}. \tag{7}\]

By definition in Eq (3) and (4), when the values of \(\gamma\) (i.e., the slope of the sigmoid functions) are important, \(\sigma^{L}(x)\approx\sigma^{L}_{\text{hard}}(x)\) and \(\sigma^{H}(x)\approx\sigma^{H}_{\text{hard}}(x)\). Therefore, symmetric results can be obtained when \(\varphi=\lambda\) and \(\varphi=1-\lambda,\forall\lambda\in[0,1]\) by simply reversing the role of \(\sigma^{L}\) and \(\sigma^{H}\). Thus the initial value of \(\varphi\) is set to be smaller than 0.5 in the training. To enable the communication of cross-domain and cross-block information, trainable weight matrices \(W_{i,k}\in\mathbb{C}^{4,C_{i}}\) for different SATSE blocks and different channels are employed in the inverse Fourier transformation,

\[\widetilde{f_{i,k}^{S^{H}}} =W_{i,k}\odot\widetilde{f_{i,k}^{S^{H}}}\,,\quad\ \widetilde{f_{i,k}^{S^{L}}}=W_{i,k}\odot\widetilde{f_{i,k}^{S^{L}}} \tag{8}\] \[f_{i,k}^{H^{\prime}} =\mathcal{F}^{-1}(\widetilde{f_{i,k}^{S^{H}}})=\frac{1}{L_{i}} \sum_{n=0}^{L-1}e^{i\frac{1}{2\pi}L_{i}n}\widetilde{f_{i,n}^{S^{H}}}\] (9) \[f_{i,k}^{L^{\prime}} =\mathcal{F}^{-1}(\widetilde{f_{i,k}^{S^{L}}})=\frac{1}{L_{i}} \sum_{n=0}^{L}e^{i\frac{2\pi}{L_{i}}kn}\widetilde{f_{i,n}^{S^{L}}}, \tag{10}\]

where \(\{f_{i,k}^{H^{\prime}}\}\) and \(\{f_{i,k}^{L^{\prime}}\}\) (\(i=0..3,k=0...L_{i}-1\)) represent the inverse Fourier sequences in time domain. The element-wise multiplication is denoted as \(\odot\) in Eq 8, enabling cross spectral domain communications. Finally, the low- and high-frequency domain knowledge is converted to time domain through IFFT separately, following Eq. (9) and (10). We then obtain the output of the SATSE block \(O_{i}^{\text{SATSE}}\) by adding the original time domain component \(\{f_{i,k}\}\) thanks to two real trainable parameters \(\lambda_{L}\) and \(\lambda_{H}\),

\[O_{i,k}^{SATSE} =f_{i,k}+\lambda_{L}f_{i,k}^{L^{\prime}}+\lambda_{H}f_{i,k}^{H^{ \prime}} \tag{11}\] \[O_{i}^{SATSE} =[O_{i,k}^{SATSE}]_{k\in\{0,...,C_{i}-1\}}.\]

Instead of a pre-defined threshold (e.g., [46]), trainable soft thresholds \(\varphi_{i}\) in the SATSE block avoid the information loss due to prior assumptions in SCDNN.

The training process of SCDNN is summarized in Algorithm 1.

```
1:Initialize model parameters
2:Set Epochs number \(E\), learning rate \(\eta\), batch size \(m\), number of sequential Res blocks \(r\)
3:
4:for\(k=0\) to \(E\)do
5: Load ECG signals and labels with batch size
6:for\(i=0\) to \(r\)do
7: Forward propagation to ResBlock\({}_{i}\)
8: Convert the output from ResBlock\({}_{i}\) to SD via \(FFT\) (cf., Eq (1))
9: Feed SD feature toward \(SATSE_{i}\) block
10: Filtering SD features individually (cf., Eq (7))
11: Share mutually information on SD (cf., Eq (8))
12: Convert features from SD to TD via \(IFFT\) (cf., Eq (9) and (10))
13: Obtain the output of SATSE (cf., Eq (11))
14: Adaptive average and max pooling with Concatenation
15: Flatten pooled features
16: Apply softmax to smooth the output probability
17: Compute the model loss
18: optimize model parameters with loss and \(\eta\) via Adams optimizer
19:endfor
20:endfor
```

**Algorithm 1** Training of SCDNN

## III Experiments on PTB-XL database

In this section, three different classification tasks using the same ECG dataset [59] are tested to verify the model performance. Moreover, the comparison against the state-of-the-art approaches and the ablation study are also explained.

### _Database Description_

The novel network SCDNN is tested on the public ECG dataset \(PTB-XL\)[59], which includes 21837 12-lead ECG signals from 18885 patients. All data are downsampled under 100 Hz for a signal duration of 10 seconds. Four types of labels can be found in this dataset, namely Form, Rhythm, Super-class diseases and Sub-class diseases. For evaluating the model robustness, three classification tasks are assigned, corresponding to Form, Super-class and Sub-class labels respectively. Classification of Rhythm labels is not performed due to the extremely imbalanced classes [59]. In each of the three classification tasks, all ECG signals are single-labelled.
The definition and the classification task for each of the three types of labels are listed here:

* **Super-class**: classify the raw ECG signals to \(4\) types of diseases, namely CD, HVP, STTC and MI, and a Norm class (no disease).
* **Sub-class**: classify the raw ECG signals to \(22\) specific diseases (each belongs to one of the Super-classes) and a Norm class.
* **Form**: classify the raw ECG signals to \(19\) different forms, which are not explicitly related to disease detection. These forms include non-diagnostic T abnormalities(NDT), ventricular premature complex(PVC) [59] etc.

The benchmark of these three tasks using state-of-the-art TS classification methods on well-separated training, validation and test datasets is provided in the work of [60]. In this paper, we make use of the same settings to implement SCDNN.

The numbers of ECG signals in training, test and validation datasets are displayed in Table I. The classification of Super-class and Sub-class labels relies on the same data separation. As depicted in Fig 4, significant label imbalance problems can be observed in all three tasks mainly due to the large number of samples with the Norm (i.e., no disease) label.

### _Implementation_

#### Iii-B1 Model Input

The training set is used to train the model, and the validation set is employed to validate the model after each epoch. For all three classification tasks, the model inputs consist of 12-lead ECG signals without extra preprocessing while the outputs include the probability of each label. The class with the highest probability is selected as the final prediction after a _softmax_ layer.

#### Iii-B2 Training Parameters Setting

In all three tasks, the number of epochs, batch size, learning rate and weight decay rate are fixed as 100, \(256\), \(1e-3\) and \(2e-5\), respectively. To fairly compare with the benchmark [60], \(CrossEntropyLoss\)[61] and Adams [62] are selected as the loss function and the optimizer for the training. In SATSE blocks, the initial values of \(\varphi_{i}\) and \(\gamma_{i}\) are set as \(0.4\) and \(0.5\) in each block while \(\lambda_{L}\) and \(\lambda_{H}\) are initialized as \(0\). These values will be updated in network back-propagation.

### _Evaluation and Comparison_

In this paper, we adopt \(Precision,Recall,F1\ score\) and _Accuracy_ as metrics to compare our model performance regarding the state-of-the-art approaches [35]. These metrics are commonly used in evaluating the performance of classification problems especially for imbalanced data [63]. Since all three tasks are multi-class classifications, the metrics must be averaged by proper scheme [64]. \(Macro\) averaging method is used here because of the label imbalance as suggested by [65]. Area under Curve (AUC) is also widely employed for evaluating classification performance [66, 67]. However, since \(AUC\) is not sensitive to imbalanced data [60], this metric is not appropriate to quantify the test results in this study. To compare the performance of SCDNN, eight existing DL models (as listed in the first columns of Table II, III and IV) used in the benchmark paper [60] are adopted for the three classification tasks. In addition, the standard ResNet18 is tested on three tasks for evaluating the enhancement of SCDNN. All models are trained with the same parameters mentioned in Section III-B2, and their performance is evaluated by the same test dataset.

The numerical results of SCDNN against the state-of-the-art approaches for each of the three tasks are displayed in Table II, III and IV, respectively. In all three tasks, SCDNN reaches

\begin{table}
\begin{tabular}{l c c c c c} \hline Task & Total & Training & Test & Validation & \(n^{o}\) class \\ \hline Super-class & 21430 & 17111 & 2163 & 2156 & 5 \\ Sub-class & 21430 & 17111 & 2163 & 2156 & 23 \\ Form & 8988 & 7201 & 882 & 905 & 19 \\ \hline \end{tabular}
\end{table} TABLE I: Number of samples of different classification tasks in _PTB-XL_ 2020

Fig. 3: SATSE Architecture
the highest score of \(Macro\ Recall,Macro\ Precision,F1\) and _Accuracy_, which ensures the robustness of the proposed approach.

What can be apparently observed in Table III is that SCDNN surpasses all baseline works significantly on all three metrics in Super-class classification, demonstrating the significant advantage of SCDNN on slightly imbalanced classification tasks. In particular, thanks to the extra spectral information captured by SATSE, SCDNN overperforms other Residual based models, namely _resnet1d_wang_, _inception1d_ and _xresnet1d101_.

The Sub-class classification is extremely difficult due to the highly imbalanced labels. As illustrated by the last column of Table III, SCDNN still achieves the best _Accuracy_ score of \(71\%\) regarding other methods. On the other hand, as shown by Fig 4, for some sub-classes there is no sufficient data to learn from the training set. Thus, the scores of the other three metrics (i.e., _MacroPrecision, MacroRecall_ and _MacroF1_) drop significantly compared to the Super-class classification (Table II). Nevertheless, SCDNN still achieves the best result on \(MacroRecall,MacroPrecision\) and \(MacroF1\), meaning that this model reaches the best performance in terms of identifying illed individuals (i.e., true positives) with the lowest rate of mis-specification (false positives) regarding other baseline works.

We also perform the Form classification for ECG signals as suggested in the benchmark paper [60]. This task consists of classifying ECG TS regarding different pre-defined forms, such as abnormal QRS (ABQRS), ventricular premature complex (PVC) or low amplitude T-waves (LOWT) [59]. As shown in Table IV, SCDNN manages to improve significantly the state-of-the-art results in terms of all four metrics with an advantage of \(25\%,17\%,19\%\) and \(9\%\) respectively. In fact, the information embedded in the deep frequency domain can be well captured by SCDNN, which enhances, in particular, the recognition of complex waveforms. On the other hand, the listed existing approaches do not take into account the deep frequency domain. As a consequence, the trained classification models can be potentially mislead by complex ECG signals consisted of many unrecognized waveforms. As pointed out by [73], some subclinical cardiac diseases can lead to chaotic waveforms other than P-QRS-T. These complex waveforms, difficult to identify in the TS, can be potentially better recognized using deep SD features. The substantial improvement on _Macro Precision_ and _Macro Recall_ clearly demonstrates the impact of frequency-domain features in reducing overfittings.

In summary, numerical results demonstrate that SCDNN

\begin{table}
\begin{tabular}{l l l l l} \hline Methods & Precision(M) & Recall(M) & F1(M) & _Accuracy_ \\ \hline resnet1d\_wang [68] & 0.70 & 0.62 & 0.63 & 0.72 \\ inception1d [69] & 0.68 & 0.60 & 0.60 & 0.70 \\ xresnet1d101 [60] & 0.70 & 0.62 & 0.62 & 0.72 \\ LSTM\_bidir [70] & 0.68 & 0.61 & 0.62 & 0.71 \\ fcn\_wang [68] & 0.71 & 0.62 & 0.63 & 0.72 \\ LSTM [33] & 0.70 & 0.62 & 0.63 & 0.72 \\ Wavelet+NN [71] & 0.58 & 0.51 & 0.52 & 0.63 \\ naive bayes [72] & 0.08 & 0.20 & 0.12 & 0.42 \\ \hline Resnet18 [50] & 0.67 & 0.62 & 0.64 & 0.72 \\
**Proposed SCDNN** & **0.72** & **0.67** & **0.69** & **0.75** \\ \hline \end{tabular}
\end{table} TABLE II: Super-class results on _PTB-XL_ database

Fig. 4: Label distribution in three classification tasks

\begin{table}
\begin{tabular}{l l l l l} \hline Methods & Precision(M) & Recall(M) & F1(M) & _Accuracy_ \\ \hline resnet1d\_wang & 0.36 & 0.33 & 0.3 & 0.68 \\ inception1d & 0.37 & 0.34 & 0.34 & 0.68 \\ xresnet1d101 & 0.38 & 0.36 & 0.35 & 0.68 \\ LSTM\_bidir & 0.37 & 0.33 & 0.33 & 0.67 \\ fcn\_wang & 0.45 & 0.32 & 0.33 & 0.68 \\ LSTM & 0.40 & 0.34 & 0.33 & 0.68 \\ Wavelet+NN & 0.25 & 0.21 & 0.21 & 0.57 \\ naive bayes & 0.02 & 0.05 & 0.03 & 0.42 \\ \hline Resnet18 & 0.35 & 0.28 & 0.29 & 0.68 \\
**Proposed SCDNN** & **0.51** & **0.40** & **0.43** & **0.71** \\ \hline \end{tabular}
\end{table} TABLE III: Sub-class results on _PTB-XL_
overperforms the benchmark approaches in terms of both disease identification and form recognition for ECG signals.

To further inspect the impact of the novel SATSE block in SCDNN, the three classification tasks are also performed using standard \(ResNet18\) for an ablation study. Compared to the original \(ResNet18\), the advantage of SCDNN can be observed on all metrics in Table II to IV, especially for Sub-class and Form classifications which are considered more difficult..

A variety of initial values of \(\varphi_{i}(i=1,..,4)\) have also been tested for the Super-class classification task to ensure the model robustness. The initial values are set to \(\varphi_{i}^{(0)}\in[0.1,0.2,0.3,0.4,0.5]\) and the total number of Epochs is set to 100. For the three classification tasks, the obtained results on the four metrics are shown in Fig. 5. The solid blue lines represent the averaged score of SCDNN regarding different initial values of \(\varphi_{i}^{(0)}\) while the transparent zones refer to the standard deviation. The red curves illustrate the average of all benchmark approaches (except naive bayes) as displayed in Table II to IV where the red transparent zones are the associated standard deviations. In addition, the best score among existing approaches (of each metric) and the results of ResNet18 (for ablation experiment) are drawn by purple and green dashed lines, respectively. From Fig. 5, one can clearly observe the significant advantage of SCDNN compared to the state-of-the-art approaches, which is consistent with our analysis of Table II to IV. Further more, the proposed SCDNN shows a strong robustness by overpassing the highest score of existing approaches (purple dashed line) with almost all different initial values tested. The standard deviation of SCDNN regarding the values of \(\varphi_{i}^{(0)}\) is also relatively small. We have also displayed these metrics of \(fcn_{wang}\), \(xresnet101\) and SCDNN for each of the classes in all three classification tasks as shown in Fig. 6. To further inspect the algorithm performance in relation to the size of each classes, we set the x-axis in a decreasing order regarding the number of samples in each class in the test dataset. Overall, a consistent advantage of the proposed SCDNN, compared to the other two approaches can be observed. In particular, \(fcn_{wang}\) and \(resnet101\) reach a high precision with classes of small numbers of samples in Super-class (Fig 6 (a)) and Form (Fig 6 (b)) tasks. However, the associated \(Recall\) and \(F1\) score drop substantially for the same classes, implying considerable overfittings. On the other hand, the three scores of SCDNN remain consistent in all three tasks, showing the good generalisation of SCDNN.

The evolution of model parameters against training Epochs is illustrated in Fig 7. Regardless the initial values, the convergence can be observed for \(\varphi_{1},\varphi_{2}\) and \(\varphi_{3}\) in Fig 7(a-c). On the other hand, \(\varphi_{4}\) converges numerically to 0 or 1 regarding different initial values as shown in Fig 7 (d). In fact, as explained in Section II-B, \(\varphi_{i}\to 0\) and \(\varphi_{i}\to 1\), this will lead to the same numerical outputs for the neural network by simply reversing the role of \(\sigma^{L}\) and \(\sigma^{H}\). We also display the evolution of \(\lambda^{H}\) and \(\lambda^{L}\) accordingly in Fig 7(e,f). Regardless the initial values of \(\varphi^{i}\), the evolution of \(\lambda^{H}\) and \(\lambda^{L}\) remains numerically stable. These results underline the robustness of the proposed SCDNN with a large range of initial parameters.

### _Computational Efficiency_

For fair comparison, we also illustrate the computational efficiency of different CNN-based approaches in Fig 8 regarding Giga floating point operations per second (GFlops) and number of model parameters (per million), evaluated on the _PTB-XL_ database. GFlops stands for the number of operations in GPU, thus representing the complexity of the neural networks. Both the GFLOPs (blue columns) and the number of parameters (red columns) show that our proposed SCDNN has a relatively low computational cost, especially when compared with other Resnet based approaches such as _xresnet1d101_ and _resnet1d_wang_. In particular, regarding the performance of _xresnet1d101_ on all three classification tasks (e.g., \(F_{1}\) as shown in Fig 8), the strength of time-spectral domain communication in SCDNN can not be compensated by employing a deeper neural network. Among all studied

Fig. 5: The score of SCDNN on the four metrics against the benchmark approaches. The transparent zones refer to the standard deviation regrading different initial thresholds for SCDNN and different approaches for the Mean Benchmark.
models, _fcn_wang_ is the most computationally efficient thanks to its simplified structure. The low computational cost highlights the possibility of extending SCDNN to high-dimensional problems, for instance, images or videos.

To summarise, SCDNN offers considerable improvement in comparison to existing approaches on various tasks, especially for imbalanced time-series datasets, with a low computational cost. Furthermore, the ablation experiment highlights the contribution of the proposed SATSE block with soft trainable thresholds in the spectral domain.

## IV Experiment on MIT-BIH database

To further ensure the validity of SCDNN, we have also evaluated its performance on the MIT-BIH database [74], which includes 48 recordings of two-lead ECG signals [75]. Here we use the preprocessed data1 where these ECG signals are split into 109444 heartbeat samples, classified in five groups, namely, normal beat (N), unclassifiable beat (Q), supraventricular premature beat (S), premature ventricular contraction (V) and fusion of ventricular beat (F) [74]. The number of samples regarding different labels is shown in Fig 9, splits into a training and a test dataset of 87553 and 21891 samples, respectively. Similar to the _PTB-XL_ database, an imbalanced issue of ECG labels can be noticed here due to the dominance of the Normal label.

Footnote 1: [https://www.kaggle.com/datasets/mondejar/mithh-database](https://www.kaggle.com/datasets/mondejar/mithh-database)

We display the performance of SCDNN against existing approaches (those have been performed on MIT-BIH database) in terms of _Average precision_, _Average recall_, _Average F1_ and _Accuracy_ in Table V. The _Average recall_ and the _Average precision_ are also specified as sensitivity and specificity in some references (e.g., [76]). As pointed out by [77], it is worth mentioning that different preprocessing strategies have been chosen for these methods as shown in the second column of Table V. All methods presented in Table V achieve a high _Accuracy_ score topped by SCDNN and CNN-LSTM [78]. An advantage of SCDNN can also be found in the other three metrics compared to the works of [76] and [79]. The outstanding performance of SCDNN on MIT-BIH database further demonstrate its robustness in TS classification tasks.

## V Discussion

On both _PTX-BL_ and _MIT-BH_ databases, SCDNN exhibits a significantly better performance compared to benchmark approaches, for ECG classifications. More importantly, our exploratory work brings new insight into the communication between time and spectral domains inside the neural network through trainable coefficients. In summary, the strength of such model can be highlighted by following observations

1. **better prediction with low computational cost:** SCDNN outperforms all existing approaches on both databases without adding a considerable number of layers or parameters. This fact demonstrates that the spectral domain information is difficult to be recognised by simply constructing deeper neural networks.
2. **ablation experiment:** The proposed SCDNN outperforms considerably the model backbone (i.e., _Resnet18_), which demonstrates the impact of spectral domain processing in SCDNN, namely SATSE blocks.
3. **robustness regarding initial values:** All different metrics of classification performance varies little against the initial parameter settings in SATSEs, and these trainable parameters converge almost to the same values during the training.

As a pioneer work, we didn't consider the attention mechanism, which is now commonly employed in deep learning models. In fact, implement extra attentions on the Fourier modes in the spectral domain can potentially improving the information filtering. Along this line, only FFT and IFFT are investigated in this work. Other spectral-time transformation operators can also be integrated in SCDNN.

## VI Conclusion and future work

ECG time series classification is crucial in cardiovascular disease diagnosis, while current methods struggle to detect

Fig. 6: The (Macro) _Precision_, _Recall_ and \(F_{1}\) for all three tasks on _PTB-XL_ in each different class. The x-axis represents the number of samples in each class.
correct patterns of ECG signals [83, 84]. In this paper, we proposed a novel neural network named Spectral Cross-Domain Neural Network to combine the information embedded in time and spectral domains. For this purpose, the self-adaptive threshold spectral enhancement (SATSE) block

\begin{table}
\begin{tabular}{l l l l l l} \hline Methods & Preprocessing & Average & Average Recs & Average Recs & Average & _Accuracy_ \\  & & Precision & call & F1 & \\ \hline  & Daubechies wavelet Z-score & 0.91 & 0.97 & / & 0.94 \\
9-layer CNN [79] & & Balanced sampling & / & / & / & 0.93 \\
11-layer CNN [14] & & Balanced sampling & / & / & / & 0.97 \\ Modified U-net [80] & & Z-score & / & / & / & 0.97 \\ CNN-LSTM [78] & & Standardization (0-1) & / & / & / & **0.99** \\ Multi-Perspective & & Symbolic Baseline Corrected & / & / & / & 0.96 \\ CNN [81] & & & & & & \\ CNN-LSTM with multiple input layers [82] & & approximation (SBCX) & / & / & / & 0.94 \\ Resnet33-CBAM [76] & & Balanced sampling & **0.99** & 0.97 & 0.97 & 0.98 \\ \hline
**Proposed SCDNN** & Raw ECG & **0.99** & **0.99** & **0.99** & **0.99** \\ \hline \end{tabular}
\end{table} TABLE V: Classification results on the testset of MIT-BIH database

Fig. 8: GFLOPs and number of parameters of different approaches against the \(F_{1}\) score where the exact values are shown in Table II-IV

Fig. 7: Evolution of \(\varphi_{i},i=1,..,4,\lambda^{H}\) and \(\lambda^{L}\) against training Epochs with different initial values of \(\varphi_{i}\)

Fig. 9: Label distribution in three classification tasks
is developed in this work where trainable thresholds are employed to filter the information in the high- and the low-frequency domains, respectively. The new model is tested on two different 12-lead ECG signal database with total four different classification tasks. Compared to the state-of-the-art approaches, our model shows a significant advantage of different metric scores on all the classification tasks considered. These results highlight the potential of SD knowledge for reducing information loss between Convolutional layers. Furthermore, the trainable thresholds enable the block to choose the proper frequency modes in SD. The contribution of SATSE has been also numerically demonstrated by performing ablation experiments on SCDNN.

The current version of SCDNN treat all channels in time and spectral domains equivalently. The potential of cross-domain attention is worth investigating in the upcoming work. Furthermore, various methods for SD features exploitation could be attempted, such as Fractional Fourier transform [85] and Wavelet transform [86]. We are also considering further examination of the convergent values of the soft thresholds, for instance, regarding the position/depth of the current layer in the neural network.

The model proposed in this paper is general for time-series data. Applications of SCDNN and SATSE in other research fields, including computer vision, natural language processing, speech signals and physics systems, will be further investigated.

## Acronyms

\begin{tabular}{l l}
**NN** & Neural Network \\
**AUC** & Area under Curve \\
**ConvF** & Convolutional Fourier \\
**ML** & Machine Learning \\
**TS** & Time Series \\
**TSC** & Time Series Classification \\
**FFT** & Fast Fourier Transformation \\
**SD** & Spectral Domain \\
**FNO** & Fourier Neural Operator \\
**TD** & Time Domain \\
**SCDNN** & Spectral Cross-domain Neural Network \\
**RNN** & Recurrent Neural Network \\
**IFFT** & Inverse Fast Fourier Transformation \\
**CNN** & Convolutional Neural Network \\
**LSTM** & Long Short-term Memory \\
**DL** & Deep Learning \\
**SATSE** & Soft-adaptive Threshold Spectral Enhancement \\
**ECG** & Electrocardiography \\
**FE** & Feature Engineering \\
**GBDT** & Gradient Boosting Decision Tree \\
**SVM** & Support Vector Machine \\
**RF** & Random Forest \\
**DCNN** & Deep Convolutional Neural Network \\ \end{tabular}
