Phase Repair for Time-Domain Convolutional Neural Networks in Music
  Super-Resolution
Audio Super-Resolution (SR) is an important topic as low-resolution
recordings are ubiquitous in daily life. In this paper, we focus on the music
SR task, which is challenging due to the wide frequency response and dynamic
range of music. Many models are designed in time domain to jointly process
magnitude and phase of audio signals. However, prior works show that approaches
using Time-Domain Convolutional Neural Network (TD-CNN) tend to produce
annoying artifacts in their waveform outputs, and the cause of the artifacts is
yet to be identified. To the best of our knowledge, this work is the first to
demonstrate the artifacts in TD-CNNs are caused by the phase distortion via a
subjective experiment. We further propose Time-Domain Phase Repair (TD-PR),
which uses a neural vocoder pre-trained on the wide-band data to repair the
phase components in the waveform outputs of TD-CNNs. Although the vocoder and
TD-CNNs are independently trained, the proposed TD-PR obtained better mean
opinion score, significantly improving the perceptual quality of TD-CNN
baselines. Since the proposed TD-PR only repairs the phase components of the
waveforms, the improved perceptual quality in turn indicates that phase
distortion has been the cause of the annoying artifacts of TD-CNNs. Moreover, a
single pretrained vocoder can be directly applied to arbitrary TD-CNNs without
additional adaptation. Therefore, we apply TD-PR to three TD-CNNs that have
different architecture and parameter amount. Consistent improvements are
observed when TD-PR is applied to all three TD-CNN baselines. Audio samples are
available on the demo page.
Yenan Zhang, Guilly Kolkman, Hiroshi Watanabe
# Phase Repair for Time-Domain Convolutional Neural Networks in Music Super-Resolution

###### Abstract

Audio Super-Resolution (SR) is an important topic in the field of audio processing. Many models are designed in time domain due to the advantage of waveform processing, such as being able to avoid the phase problem. However, in prior works it is shown that Time-Domain Convolutional Neural Network (TD-CNN) approaches tend to produce annoying artifacts in their output. In order to confirm the source of the artifact, we conduct an AB listening test and found phase to be the cause. We further propose Time-Domain Phase Repair (TD-PR) to improve TD-CNNs' performance by repairing the phase of the TD-CNNs' output. In this paper, we focus on the music SR task, which is challenging due to the wide frequency response and dynamic range of music. Our proposed method can handle various narrow-bandwidth from 2.5kHz to 4kHz with a target bandwidth of 8kHz. We conduct both objective and subjective evaluation to assess the proposed method. The objective evaluation result indicates the proposed method achieves the SR task effectively. Moreover, the proposed TD-PR obtains the much higher mean opinion scores than all TD-CNN baselines, which indicates that the proposed TD-PR significantly improves perceptual quality. Samples are available on the demo page1.

Footnote 1: [https://mannmaruko.github.io/demopage/dpr.html](https://mannmaruko.github.io/demopage/dpr.html)

## I Introduction

Audio Super-Resolution (SR), also known as bandwidth extension and bandwidth expansion, aims to predict the High-Resolution (HR) components from the Low-Resolution (LR) input audio. LR audio is common in our daily life (_e. g._, historical recordings, unprofessional-made modern recordings). The real-world LR recordings have a variety of bandwidth or even ambiguous bandwidth. Therefore, addressing audio SR in real world is challenging. Deep Neural Networks (DNNs) have become the mainstream on audio SR tasks compared with conventional methods [1, 2, 3, 4], but only a few works focus on the music SR [2]. In this paper, we focus on the music SR with solo piano music. We perform SR on solo instrument music instead of the orchestra, since the task will be simpler on single source data. Among many music instruments, we choose solo piano music as the representative, since piano has the most broad frequency range.

Various works have delved deeply into DNN-based approaches for audio SR. Since Frequency-Domain Convolutional Neural Networks (FD-CNN) can only process the magnitude, they generally require additional signal processing to estimate the corresponding phase information, such as Griffin-Lim algorithms [2] or a neural vocoder [4]. Compared with FD methods, Time-Domain Convolutional Neural Networks (TD-CNNs) that directly learn wave-to-wave mapping, are considered being able to avoid the phase problem on audio SR tasks [2]. However, TD-CNNs (_e. g._, AudioUNet [1]) tend to produce annoying artifacts in their results. To alleviate the artifact, Lim _et. al._ proposed a time-frequency model [5] based on AudioUNet. Wang _et. al._ made efforts on object function that employing the FD loss [6] during the TD-CNN's training. The data augmentation strategy was proposed [7] to improve the robustness of TD-CNNs. However, none of the above CNN methods in time domain succeeds in removing the artifact according to the provided audio samples. This observation encourages us to explore other aspects that may have caused the artifact. In terms of up-sampling ratio, many works perform the SR on the fixed ratio (_e. g._, 2\(\times\)) [1, 2], which would be a limitation when apply these models to real world problems.

We investigate the existing issue of artifacts in the following ways. First, we train our models to handle LR music with various bandwidth, which is applicable to real world problems. Second, We conduct an AB listening test which, to the best of our knowledge, is the first to demonstrate the artifact in TD-CNNs is caused by the phase distortion via a subjective test. Last but not least, we propose the Time-Domain Phase Repair (TD-PR) method to improve the performance of TD-CNNs for the music SR. TD-PR can be directly used for general TD-CNN models without parameter adjustment. The subjective evaluation shows the TD-PR significantly improves the perceptual quality of three TD-CNN baselines, which validates its effectiveness and generalization ability.

## II Related Work

Various approaches in audio SR have been developed and some of them work in frequency domain. Li _et. al._ propose the FD approach for speech SR, which consists of 2 steps [8]. The first step is mapping the magnitude component from narrow-bandwidth to wide-bandwidth by DNN. The second step is to estimate the corresponding phase by signal processing. Following this work, Hu _et. al._ introduce Generative Adversarial Network (GAN) into both steps and got the better performance [2]. However, training two GAN-based models is difficult due to the instability of GAN training. Furthermore, this SR system works on fixed up-sampling ratio, which limits it's application for real world problems. Liu _et. al._ use a GAN-based neural vocoder for the second step without using GAN in the first step, which still performs speech SR successfully with the