Two-stage Neural Network for ICASSP 2023 Speech Signal Improvement
  Challenge
In ICASSP 2023 speech signal improvement challenge, we developed a dual-stage
neural model which improves speech signal quality induced by different
distortions in a stage-wise divide-and-conquer fashion. Specifically, in the
first stage, the speech improvement network focuses on recovering the missing
components of the spectrum, while in the second stage, our model aims to
further suppress noise, reverberation, and artifacts introduced by the
first-stage model. Achieving 0.446 in the final score and 0.517 in the P.835
score, our system ranks 4th in the non-real-time track.
Mingshuai Liu, Shubo Lv, Zihan Zhang, Runduo Han, Xiang Hao, Xianjun Xia, Li Chen, Yijian Xiao, Lei Xie
# Two-Stage Neural Network for ICASSP 2023 Speech Signal Improvement Challenge

###### Abstract

In ICASSP 2023 speech signal improvement challenge, we developed a dual-stage neural model which improves speech signal quality induced by different distortions in a stage-wise divide-and-conquer fashion. Specifically, in the first stage, the speech improvement network focuses on recovering the missing components of the spectrum, while in the second stage, our model aims to further suppress noise, reverberation, and artifacts introduced by the first-stage model. Achieving 0.446 in the final score and 0.517 in the P.835 score, our system ranks 4th in the non-real-time track.

Mingshuai Liu\({}^{1}\), Shubo Lv\({}^{1}\), Zihan Zhang\({}^{1}\), Runduo Han\({}^{1}\), Xiang Hao\({}^{1}\), Xianjun Xia\({}^{2}\), Li Chen\({}^{2}\), Yijian Xiao\({}^{2}\), Lei Xie\({}^{1*}\)\({}^{1}\)Audio, Speech and Language Processing Group (ASLP@NPU), School of Software,

Northwestern Polytechnical University, Xi'an, China

\({}^{2}\)ByteDance, China speech distortion, speech enhancement

## 1 Introduction

During audio communication, speech signals may be degraded by multiple distortions, including coloration, discontinuity, loudness, noisiness, and reverberation. Existing methods have an impressive performance in noise suppression, but there is still an open problem on how to repair several distortions simultaneously. To stimulate research in this direction, ICASSP 2023 has specifically held the first Speech Signal Improvement Challenge1, as a flagship event of Signal Processing Grand Challenge.

Footnote 1: [https://www.microsoft.com/en-us/research/academic-program/speech-signal-improvement-challenge-icassp-2023/](https://www.microsoft.com/en-us/research/academic-program/speech-signal-improvement-challenge-icassp-2023/) * Corresponding author.

Inspired by the idea of decoupling difficult tasks into multiple easier sub-tasks [1], we propose a neural speech signal improvement approach with a training procedure that involves two stages. In the first stage, we adopt DCCRN [2] as a repair tool and substitute its oracle convolution structure with a more powerful gated convolution with the aim to mainly repair the missing components in the spectrum. Thus with simulated paired data of perfect and impaired speech, GatedDCCRN learns to improve quality problems caused by coloration, discontinuity, and loudness. For the loss function, besides SI-SNR loss and power-law compressed loss [3], we also integrate adversarial loss further to improve speech naturalness. In the second stage, a variant of S-DCCRN [4] is cascaded to the first stage GatedDCCRN model to particularly remove noise, and reverberation, and suppress possible artifacts introduced by the previous stage. Specifically, S-DCCRN is a powerful denoising model working on super wide-band and full-band signals, consisting of two small-footprint DCCRNs - one operates on sub-band signal and one works on full-band signal, benefiting from both local and global frequency information. With this denoising model, we further update its bottleneck layers from LSTM to STCM [5] for better temporal modeling. The proposed system has achieved 0.446 in the final evaluation score and 0.517 in P.835 score, leading our submission to the 4th place in the non-real-time track.

## 2 Approach

As illustrated in Fig.1, the training procedure of our system consists of two stages. The details of the network architecture, training data, and loss function used in these two stages are introduced below.

### Stage 1: Repairing Net

Considering its impressive ability in signal mapping, we employ DCCRN [2] as the backbone of our first-stage speech repair network. DCCRN is a U-Net structured complex network working on complex-valued spectrum, where the encoder and decoder are both composed of layered convolution, and LSTM is served as the bottleneck for temporal modeling. Inspired by the superior performance of gated convolution in image inpainting [6], we update the complex convolution with gated complex convolution, as shown in Fig.1, resulting in GateDCCRN. For the model training loss, in addition to SI-SNR (\(\mathcal{L}_{\text{SI-SNR}}\)) and power-law compressed loss (\(\mathcal{L}_{\text{PLC}}\)), we particularly integrate adversarial loss (\(\mathcal{L}_{\text{Adv}}\)) by adding the Multi-Period Discriminator [7] and Multi-Scale Discriminator [7] into the model optimization process to further improve the speech naturalness. Thus the final loss function is \(\mathcal{L}_{\text{Aug1}}=\mathcal{L}_{\text{SI-SNR}}+10\cdot\mathcal{L}_{ \text{PLC}}+15\cdot\mathcal{L}_{\text{Adv}}\).

### Stage 2: Denoising Net

In the second stage, the pre-trained GateDCCRN and an S-DCCRN [4] are chained as the denoising structure. Specifically, as shown in Fig.1 stage 2, two lightweight DCCRN sub-modules consequently work on sub-band and full-band signals, with the design to model fine details of different frequency bands with further inter-band smoothing. Different from the oracle S-DCCRN, we substitute LSTM with squeezed temporal convolution module (STCM) [5] in the bottleneck layer of the two DCCRNs, which aims to further strengthen the temporal modeling ability. With this update, the new model is named S-DCCSN. During training, we add noise and reverberation to the data simulated in the first stage to train the second stage model, which makes the model further achieve the ability to suppress noise, reverberation, and artifacts introduced by GateDCCRN. Note that the parameters of the pre-trained GateDCCRN are frozen in this training stage. We adopt SI-SNR loss (\(\mathcal{L}_{\text{SI-SNR}}\)), power-law compressed loss (\(\mathcal{L}_{\text{PLC}}\)), and mean square error loss (\(\mathcal{L}_{\text{Mtg}}\)) to optimize the model parameters, and the final loss becomes \(\mathcal{L}_{\text{stage2}}=\mathcal{L}_{\text{SI-SNR}}+\mathcal{L}_{\text{PLC }}+\mathcal{L}_{\text{Mtg}}\).

## 3 Experiments

### Datasets

The training set is created using the DNS4 dataset. Specifically, the DNS4 dataset includes a total of 750 hours of clean speech and 181 hours of noise. In addition, 50,000 RIR clips are simulated by HYB method. The RT60 of RIRs ranges from 0.2s to 1.2s. The room
size ranges from \(5\times 3\times 3m^{3}\) to \(8\times 5\times 4m^{3}\). Totally, there are 110,248 RIR clips, combined from the simulated RIR set and the provided RIR set by the challenge. We perform dynamic simulations of speech distortions during the training stage. In the first stage, we generate training data degraded by coloration, discontinuity, and loudness, accounting for 60\(\%\), 25\(\%\), and 15\(\%\) respectively. For coloration, we follow the description in [8] to design a low pass filter. We convolve it with full-band speech and perform resampling on the filtered result to generate low-band speech. Besides producing low-bandwidth distortions, we also restrict signal amplitudes within a range \([-\eta,+\eta]\) (\(\eta\in(0,1)\)). When we produce coloration distortion, the low-bandwidth speech and clipping speech account for 60\(\%\) and 40\(\%\) respectively. Specifically, full band speech is down-sampled to 4K, 8K, 16K, and 24K with the same probability. For discontinuity, the value of speech samples is set to zero randomly with a window size of 20ms. And the probability of setting the value of the sample point to zero is 10\(\%\). For loudness, we multiply the signal amplitudes by a scale within a range [0.1, 0.5]. In the second training stage, noise is further added to the first-stage training data with an SNR range of [0, 20]dB, and reverberation is further added to 50\(\%\) of the first-stage training data. Dynamic mixing is still used during training. We denote the first-stage and second-stage training data as S1 and S2 respectively, where S2 includes simulations for all signal distortions.

### Experiment Setup

The window length and frame shift are 20ms and 10ms respectively, resulting in 10ms algorithmic latency and 10ms buffering latency. The STFT length is 1024. The number of channels for DCCRN / GateDCCRN is {16, 32, 64, 128, 256, 256}, and the convolution kernel size and stride are set to (5,2) and (2,1) respectively. There are two LSTM layers with 256 nodes followed by a 2048 \(\times\) 256 fully connected layer between the encoder and decoder. The number of channels for the sub-band module and the full-band module of S-DCCRN / S-DCCSN is {64, 64, 64, 64, 128, 128}, and the convolution kernel size and stride are set to (5,2) and (2,1) respectively. For the CED and CFD of S-DCCRN / S-DCCSN, the number of channels is 32 and the depth of DenseBlock is 5. In S-DCCSN, The hidden channels of STCM adopted by the sub-band module and full-band module are 64. While in S-DCCRN, there are two LSTM layers with 256 nodes instead. And a fully connected layer with 512 nodes is adopted after the last LSTM layer. Models are optimized by Adam with the initial learning rate of 0.001, halved if the validation loss of two consecutive epochs no longer decreases.

### Results

We conduct ablation experiments to validate each proposed module. In Table 1, we train DCCRN and GateDCCRN with and without (w/o) discriminator using the first-stage training data (S1). Then we froze the parameters of the pre-trained GateDCCRN and cascade this model with S-DCCRN and S-DCCSN respectively. And the cascaded models are trained using the second-stage training data (S2). In addition, we also train a GateDCCRN and an S-DCCSN with the second-stage training data only (S2). DNSMOS results in Table 1 show that gated convolution and adversarial training using a discriminator can effectively improve the signal improvement ability of DCCRN. Moreover, S-DCCSN surpasses S-DCCRN in all evaluation metrics with a small increase in model size, which proves the better temporal modeling ability of STCM. Finally, the single-stage models (GateDCCRN and S-DCCSN) trained using S2 reflecting all distortions are inferior to the multi-stage model (GateDCCRN+S-DCCSN). Table 2 shows the subjective results of our submitted two-stage model (GateDCCRN+S-DCCSN) on the official test set. We can see that speech quality is clearly improved for different types of distortions except for discontinuity. The bad cases may attribute to our imperfect data simulation on discontinuity which desires further investigation. The number of parameters of the submitted two-stage system is 10M. The RTF is 1.478 tested on Intel(R) Xeon(R) CPU E5-2678 v3 2.4GHz using a single thread.
