RRCNN$^{+}$: An Enhanced Residual Recursive Convolutional Neural Network
  for Non-stationary Signal Decomposition
Time-frequency analysis is an important and challenging task in many
applications. Fourier and wavelet analysis are two classic methods that have
achieved remarkable success in many fields. They also exhibit limitations when
applied to nonlinear and non-stationary signals. To address this challenge, a
series of nonlinear and adaptive methods, pioneered by the empirical mode
decomposition method have been proposed. Their aim is to decompose a
non-stationary signal into quasi-stationary components which reveal better
features in the time-frequency analysis. Recently, inspired by deep learning,
we proposed a novel method called residual recursive convolutional neural
network (RRCNN). Not only RRCNN can achieve more stable decomposition than
existing methods while batch processing large-scale signals with low
computational cost, but also deep learning provides a unique perspective for
non-stationary signal decomposition. In this study, we aim to further improve
RRCNN with the help of several nimble techniques from deep learning and
optimization to ameliorate the method and overcome some of the limitations of
this technique.
Feng Zhou, Antonio Cicone, Haomin Zhou
RRCNN\({}^{+}\): An Enhanced Residual Recursive Convolutional Neural Network for Non-stationary Signal Decomposition

###### Abstract

Time-frequency analysis is an important and challenging task in many applications. Fourier and wavelet analysis are two classic methods that have achieved remarkable success in many fields. They also exhibit limitations when applied to nonlinear and non-stationary signals. To address this challenge, a series of nonlinear and adaptive methods, pioneered by the empirical mode decomposition method have been proposed. Their aim is to decompose a non-stationary signal into quasi-stationary components which reveal better features in the time-frequency analysis. Recently, inspired by deep learning, we proposed a novel method called residual recursive convolutional neural network (RRCNN). Not only RRCNN can achieve more stable decomposition than existing methods while batch processing large-scale signals with low computational cost, but also deep learning provides a unique perspective for non-stationary signal decomposition. In this study, we aim to further improve RRCNN with the help of several nimble techniques from deep learning and optimization to ameliorate the method and overcome some of the limitations of this technique.

 Empirical mode decomposition; Non-stationary signal decomposition; Deep learning; Attention mechanism.

## I Introduction

Time-frequency analysis has gone through over 200 years of development, and its beginning can be traced back to the Fourier transform [1]. To process signals more effectively, wavelet transform, which provides focusing capability, has been studied since the late 1980s [2]. On one hand, Fourier and wavelet methods have remarkable success in a wide range of applications. On the other hand, they are linear transforms, which become a limitation when handling non-stationary signals. In 1998, Huang and collaborators proposed the empirical mode decomposition (EMD) [3], which is a nonlinear procedure for decomposing a signal into multiple quasi-stationary components so that they can be better analyzed. EMD has gained a lot of popularity and found numerous applications in different disciplines. However, its mathematical foundation is still lacking. Many alternative algorithms with improved performance have emerged in recent years. They can be classified into two categories: methods based on iterations and methods based on optimization.

Among the iteration-based methods, we mention the one based on moving average computation [4, 5], partial differential equation (PDE) solution [6, 7], and recursive filter application [8, 9]. In this category, Lin et al. proposed the iterative filtering (IF) method to calculate the local average. The idea is to apply filters to replace the mean of the upper and lower envelopes computation in the sifting process of EMD [8]. Cicone et al. conducted in-depth research on IF and extended it to high-dimensional [10] and non-stationary signals [9]. The optimization-based methods include compressed sensing [11], variational optimization [12, 13, 14, 15], and a few other techniques [16, 17, 18].In the variational optimization-based methods, Dragomiretskiy et al. proposed in [12] variational mode decomposition (VMD) with the goal of decomposing a signal into a few modal functions with a specific sparsity. Osher et al. in [14] further develop the idea contained in VMD, by developing the geometric mode decomposition which is based on VMD itself.

After an in-depth comparison of existing methods, we find some common features. 1) The local average of a signal is critical. Many methods aim to find it in a reasonable way. 2) It is unrealistic to find a single local average method handling all signals. 3) Existing methods usually require parameters. Their results are generally sensitive to the selection of parameters. These observations inspire us to consider the problem from a new perspective. Namely, finding the local average customized as the "pattern" of a signal, which is a typical task routinely accomplished by modern deep learning methods.

In [19] we proposed a deep-learning-based method, called residual recursive convolutional neural network (RRCNN) for non-stationary signal decomposition. Experiments show that RRCNN not only is more stable and effective than existing methods on artificially synthesized signals but also can replicate the decomposition of existing methods on real-life signals with a significant reduction in boundary effects. More importantly, once the RRCNN model is trained, it yields decompositions in real time that are unparalleled by any of the existing methods. However, since RRCNN was the first deep learning algorithm designed for this purpose, it did not realize the full potential of deep learning in non-stationary signal decomposition. For example, its weights are always
the same in predicting different signals. In this sense, it is not fully adaptive to different classes of signals. In addition, the decomposition produced via RRCNN may contain high-frequency oscillations with small amplitude in some components. These small artifacts can degrade the subsequent time-frequency analysis of the signal.

In this work, we employ several simple techniques from deep learning, including the multi-scale convolution [20, 21], attention [22], residue [23], and the total-variation-based denoising (TVD) [24, 25], to further improve RRCNN. The proposed model is called RRCNN\({}^{+}\). The main contributions are summarized here: 1) The new improved module composed of the multi-scale convolution, attention, and residue techniques allows RRCNN\({}^{+}\) to extract heterogeneous features, and have a stronger adaptability. 2) TVD allows RRCNN\({}^{+}\) to remove the small amplitude high-frequency oscillations that are often observed in RRCNN. The resulting components appear to have more physical meaning than the one produced with the standard RRCNN algorithm in some applications.

The rest of this paper is organized as follows. We review RRCNN in Section II. Then, we introduce RRCNN\({}^{+}\) and illustrate how it works in Section III. Experiments are discussed in Section IV. Section V provides the conclusion.

## II RRCNN

Given a non-stationary signal \(X\in\mathbb{R}^{N}\), RRCNN can be described as the following optimization problem,

\[\begin{cases}\min_{\{\mathcal{W}_{m}\}_{m=1}^{M}}\|\hat{\mathbf{Y}}-\mathbf{Y} \|_{F}^{2}+\eta\,QTV(\hat{\mathbf{Y}})\\ \text{s.t., }\hat{Y}_{m}=F(X_{m-1},\mathcal{W}_{m}),X_{m}=X_{m-1}-\hat{Y}_{m}, \end{cases} \tag{1}\]

where \(m=1,2,\ldots,M\), \(M\) denotes the number of the expected components (each is called an intrinsic mode function (IMF) [3]), \(\hat{\mathbf{Y}}=\{\hat{Y}_{m}|\hat{Y}_{m}\in\mathbb{R}^{N}\}_{m=1}^{M}\) and \(\mathbf{Y}=\{Y_{m}|Y_{m}\in\mathbb{R}^{N}\}_{m=1}^{M}\) represent the predicted and true IMFs respectively, \(\|\cdot\|_{F}\) is the Frobenius norm, \(\eta\) denotes the penalty parameter, \(QTV(\hat{\mathbf{Y}})=\sum_{t=1}^{N-1}\sum_{m=1}^{M}(\hat{Y}_{(t+1),m}-\hat{ Y}_{t,m})^{2}\) is added to ensure the smoothness of each \(\hat{Y}_{m}\), \(\mathcal{W}_{m}\) denotes the set of parameters involved in finding the \(m\)-th IMF. To facilitate the understanding of the calculation process of \(F(X_{m-1},\mathcal{W}_{m})\), \(X_{m-1}\) and \(\mathcal{W}_{m}\) are denoted here as \(X\) and \(\mathcal{W}\), \(F(X,\mathcal{W})\) is obtained by \(f\left(X^{(S-1)},\mathcal{W}^{(S-1)}\right)\), where \(S\) represents the number of recursion, \(\mathcal{W}^{(S-1)}\) is the parameter set composed of the undetermined weights in the \((S-1)\)-th recursion, \(f(\cdot,\cdot)\) and \(X^{(i+1)}\) are calculated as: \(f\left(X^{(i)},\mathcal{W}^{(i)}\right)=\sigma\left(X^{(i)}*W_{1}^{(i)}\right) \ast\tilde{W}_{2}^{(i)}\), \(X^{(i+1)}=X^{(i)}-f(X^{(i)},\mathcal{W}^{(i)})\), where \(i=0,1,\ldots,S-2\), \(X^{(0)}=X\), \(\mathcal{W}^{(i)}=\{W_{1}^{(i)},W_{2}^{(i)}|W_{1}^{(i)}\in\mathbb{R}^{K_{1}},W _{2}^{(i)}\in\mathbb{R}^{K_{2}}\}\) (\(K_{1}\) and \(K_{2}\) represent the convolutional filter lengths), \(\tilde{W}_{2}^{(i)}=\text{softmax}\left(W_{2}^{(i)}\right)\), \(\sigma(\cdot)\) is the activation function, and \(\ast\) denotes the 1-D convolution operation.

Although it is empirically observed that RRCNN is superior to the existing methods, there is room for further improvements. First of all, \(\{\mathcal{W}_{m}\}_{m=1}^{M}\) do not depend on the specific signal. That is to say, once RRCNN is trained, it uses the same \(\{\mathcal{W}_{m}\}_{m=1}^{M}\) to process signals of different classes in the prediction phase. To some extent, this is not adaptive, while adaptivity is a desirable trait in non-stationary signal decomposition. Secondly, adding \(QTV\) to the objective function may bring two issues. (i) A bad choice in \(\eta\) can confuse RRCNN in the learning process. So the selection of \(\eta\) becomes critical and sensitive. (ii) For non-stationary signal decomposition, the smoothness has a specific physical meaning, it aims to avoid the high-frequency, low-amplitude oscillations in generating IMF. Adding the QTV directly to the objective function is difficult to avoid these oscillations.

## III RRCNN\({}^{+}\)

To improve RRCNN, we incorporate some techniques including a mechanism composed of multi-scale convolution [20, 21], attention [22] and residue [23], and total-variation-denoising (TVD) [26] into model (1). More precisely, the proposed RRCNN\({}^{+}\) method is expressed as

\[\begin{cases}\min_{\{\mathcal{W}_{m}\}_{m=1}^{M}}\|\hat{\mathbf{Y}}-\mathbf{Y} \|_{F}^{2}\\ \text{s.t., }\hat{Y}_{m}=\tilde{F}(X_{m-1},\mathcal{W}_{m}),\hat{Y}_{m}\leftarrow \text{TVD}\left(\hat{Y}_{m}\right)\\ \quad\quad\quad\quad X_{m}=X_{m-1}-\hat{Y}_{m},\end{cases} \tag{2}\]

where \(\text{TVD}\left(\hat{Y}_{m}\right)\) is used to smooth \(\hat{Y}_{m}\), \(\tilde{F}(X_{m-1},\mathcal{W}_{m})\) denotes the structure of \(F(X_{m-1},\mathcal{W}_{m})\) improved by the multi-scale convolution, attention, and residue techniques. The other symbols are the same as in (1). \(\tilde{F}(\cdot,\cdot)\) is defined as \(\tilde{F}(X,\mathcal{W})=\tilde{f}\left(X^{(S-1)},\mathcal{W}^{(S-1)}\right)\), \(\tilde{f}(\cdot,\cdot)\) and \(X^{(i+1)}\) are calculated as follows,

\[\begin{cases}X_{k}=\sigma\left(X^{(i)}*W_{1k}^{(i)}\right),\ X_{att}=\text{ Attention}\left(X^{(i)}\right),\\ \tilde{f}\left(X^{(i)},\mathcal{W}^{(i)}\right)=\sigma\left(\text{Concat}[(X^{(i)}, \{X_{k}\},X_{att})*W_{2}^{(i)}\right)*\tilde{W}_{3}^{(i)},\\ X^{(i+1)}=X^{(i)}-\tilde{f}\left(X^{(i)},\mathcal{W}^{(i)}\right),\end{cases}\]

where \(k=1,2,3\), \(\mathcal{W}^{(i)}\) composed of convolution kernel weights at different scales, i.e., \(\{W_{1k}^{(i)}\in\mathbb{R}^{[K_{1}/2^{k-1}]}\}_{k=1}^{3}\), \(W_{2}^{(i)}\in\mathbb{R}^{K_{2}}\) and \(W_{3}^{(i)}\in\mathbb{R}^{K_{3}}\) (\(K_{1}\), \(K_{2}\) and \(K_{3}\) represent the filter lengths); \(X_{att}=\text{Attention}\left(X^{(i)}\right)\) denotes the attention layer of \(X^{(i)}\) proposed in [22]; \(\tilde{W}_{3}^{(i)}=\text{softmax}\left(W_{3}^{(i)}\right)\), and the other symbols are the same as in RRCNN.

Given a component \(\hat{Y}\in\mathbb{R}^{N}\), the output of \(\text{TVD}(\hat{Y})\) is generated by solving the following optimization problem: \(\arg\min_{Y\in\mathbb{R}^{N}}\{\frac{1}{2}\|\hat{Y}-Y\|_{2}^{2}+\lambda\text{ TV}(Y)\}\), where \(\lambda\) denotes a penalty parameter, \(\text{TV}(Y)=\|\mathbf{D}Y\|_{1}\), and \(\mathbf{D}\) is the 1-order difference matrix. The solution of \(\text{TVD}(\hat{Y})\) is given in Algorithm 1 of the supplementary material.

The overall architecture of RRCNN\({}^{+}\) is shown in Fig. 1. The pseudocode of RRCNN\({}^{+}\) is reported in Algorithm 2 of the supplementary material. Compared to RRCNN, the improvements of RRCNN\({}^{+}\) are reflected in two aspects. First, three convolutions with different kernel scales and an attention layer are carried out, their outputs and shared input are concatenated as the input of the Conv. 2 layer. Second, TVD is added in front of each IMF. The improvements are called the multi-scale convolutional attention and TVD modules, respectively, where the former enhances the adaptability and the diversity of the extracted features, while the latter obtains smooth components that are more meaningful from a physical perspective.
## IV Experiments

We first evaluate the TVD and multi-scale convolutional attention modules that are adopted in RRCNN\({}^{+}\). For the convenience of notation, we denote the models improved by TVD and multi-scale convolutional attention as RRCNN_TVD and RRCNN_ATT, respectively. Then, RRCNN\({}^{+}\) is also compared with the state-of-the-art methods, including EMD [3], EEMD [27], VMD [12], EWT [28], FDM [29], IF [30], INCMD [31], SYNSQ_CWT and SYNSQ_STFT [32]. Details of the experimental data, settings, and evaluation metrics are described in the supplementary material.

### _Are TVD and multi-scale convolutional attention effective?_

To justify that TVD and multi-scale convolutional attention modules are effective, we first compare RRCNN_TVD, RRCNN_ATT and RRCNN\({}^{+}\) with RRCNN on both training and validation datasets of Dataset_1 and Dataset_2. The results, measured by MAE (mean absolute error), RMSE (root mean squared error), MAPE (mean absolute percentage error) and TV (total variation), are listed in Table I. We obtain the following findings: (i) For the vast majority of cases of both training and validation sets of Dataset_1 and Dataset_2, TVD and multi-scale convolutional attention effectively improve the performance over RRCNN. (ii) Combining TVD and multi-scale convolutional attention, i.e., RRCNN\({}^{+}\), improves performance over RRCNN in all cases. In addition, we examine the smoothness by comparing the TV norm. The TV values for the components generated by RRCNN\({}^{+}\) are essentially the closest to those of the true components.

To test the generalization capability of both modules, two signals not contained in both datasets are constructed. The first one is \(x_{1}(t)=\cos(6.4\pi t)+\cos(5\pi t)\), where \(\cos(6.4\pi t)\) and \(\cos(5\pi t)\) are denoted as the components \(c_{1}\) and \(c_{2}\) of \(x_{1}\), respectively. The frequencies of \(c_{1}\) and \(c_{2}\) are very close, which makes \(x_{1}\) can be used to evaluate the deep-learning-based models trained on Dataset_1. The second signal is \(x_{2}(t)=\cos\left(8\pi t+2t^{2}+\cos(t)\right)+\cos(5\pi t)+\varepsilon(t)\), where \(\cos\left(8\pi t+2t^{2}+\cos(t)\right)\) and \(\cos(5\pi t)\) are called the components \(c_{1}\) and \(c_{2}\) of \(x_{2}\), respectively, \(\varepsilon(t)\) is additive Gaussian noise with SNR = \(15dB\). \(x_{2}\) contains stronger noise than that added to the signals in Dataset_2, and it is used to test against the trained deep-learning-based models on Dataset_2.

Results of \(x_{1}\), \(x_{2}\) are given in Table II and Fig. 2. In Table II, we observe that the results obtained by RRCNN_TVD, RRCNN_ATT and RRCNN\({}^{+}\) are always better, measured by error metrics, than that of RRCNN, which indicates the effectiveness of introducing TVD and multi-scale convolutional attention. From the TV norm comparison, we see that RRCNN_TVD, RRCNN_ATT and RRCNN\({}^{+}\) improve the results obtained by RRCNN, except in one case on \(c_{1}\) of \(x_{2}\), where the TV norm of RRCNN is the closest to that of true component. By looking closely at Fig. 2, we found that RRCNN_TVD, RRCNN_ATT and RRCNN\({}^{+}\) greatly improve the performance of RRCNN at the peaks and troughs.

### _Can RRCNN\({}^{+}\) outperform the state-of-the-art models?_

RRCNN\({}^{+}\) is also compared with existing methods, including EMD, EEMD, VMD, EWT, FDM, IF, INCMD, SYNSQ_CWT, SYNSQ_STFT, RRCNN, RRCNN_TVD and RRCNN_ATT. Again, we take \(x_{1}\) and \(x_{2}\) that are constructed in Section IV-A as the test signals.

The results obtained by different methods are shown in Table VI of the supplementary material. Here we only list the results of the true and the obtained components by the top three methods in Table VI. We find that: (i) Since the frequencies of the constituent components of \(x_{1}\) are very close, and \(x_{2}\) is disturbed by high level noise, some existing methods, such as EWT, FDM and VMD for \(x_{1}\), and EEMD and SYNSQ_STFT for \(x_{2}\) have a hard time distinguishing the two components. Other methods, like EWT, INCMD and SYNSQ_CWT only

Fig. 1: Graphic illustration of the network structure of RRCNN\({}^{+}\).
show relatively accurate components on \(x_{2}\). (ii) Deep-learning-based methods are generally impressive on both \(x_{1}\) and \(x_{2}\).

The components and the corresponding time-frequency distributions by the top three models are depicted in Fig. 4 in the supplementary material. Here, we only depict time-frequency distributions of true and the components of RRCNN\({}^{+}\) in Fig. 3. Although the time-frequency information of non-stationary signals is not considered in RRCNN\({}^{+}\), its result is still relatively reasonable. We expect that a more accurate time-frequency distribution will be yielded when time-frequency information is incorporated into the model.

## V Conclusion

We recalled RRCNN and introduced RRCNN\({}^{+}\), deep-learning-based methods for non-stationary signal decomposition. We demonstrated that by introducing the multi-scale convolutional attention and TVD, RRCNN\({}^{+}\) improves the performance of RRCNN, and overcomes some of its limitations.

Yet, RRCNN\({}^{+}\) has some limitations too. For example, its network does not take into account the time-frequency information of the signal, which leads to the lack of time-frequency physical meaning in the results. RRCNN\({}^{+}\) is a supervised learning model that needs to assign a label to each training data. It is not easy to extend this approach to real signals because labels for real signals are difficult to obtain. We plan to further address these issues in future work.

Fig. 3: Time-frequency distribution by IMFogram [33] for the true (top) and resulting components (bottom) of \(x_{1}\) (left) and \(x_{2}\) (right) by RRCNN\({}^{+}\).

Fig. 2: Results of \(x_{1}\), \(x_{2}\) by deep-learning-based models trained on Dataset_1, Dataset_2, respectively. Left: \(x_{1}\); Right: \(x_{2}\).