A Multiscale Autoencoder (MSAE) Framework for End-to-End Neural Network
  Speech Enhancement
Neural network approaches to single-channel speech enhancement have received
much recent attention. In particular, mask-based architectures have achieved
significant performance improvements over conventional methods. This paper
proposes a multiscale autoencoder (MSAE) for mask-based end-to-end neural
network speech enhancement. The MSAE performs spectral decomposition of an
input waveform within separate band-limited branches, each operating with a
different rate and scale, to extract a sequence of multiscale embeddings. The
proposed framework features intuitive parameterization of the autoencoder,
including a flexible spectral band design based on the Constant-Q transform.
Additionally, the MSAE is constructed entirely of differentiable operators,
allowing it to be implemented within an end-to-end neural network, and be
discriminatively trained. The MSAE draws motivation both from recent multiscale
network topologies and from traditional multiresolution transforms in speech
processing. Experimental results show the MSAE to provide clear performance
benefits relative to conventional single-branch autoencoders. Additionally, the
proposed framework is shown to outperform a variety of state-of-the-art
enhancement systems, both in terms of objective speech quality metrics and
automatic speech recognition accuracy.
Bengt J. Borgstrom, Michael S. Brandstein
# A Multiscale Autoencoder (MSAE) Framework for End-to-End Neural Network Speech Enhancement

###### Abstract

Neural network approaches to single-channel speech enhancement have received much recent attention. In particular, mask-based architectures have achieved significant performance improvements over conventional methods. This paper proposes a multiscale autoencoder (MSAE) for mask-based end-to-end neural network speech enhancement. The MSAE performs spectral decomposition of an input waveform within separate band-limited branches, each operating with a different rate and scale, to extract a sequence of multiscale embeddings. The proposed framework features intuitive parameterization of the autoencoder, including a flexible spectral band design based on the Constant-Q transform. Additionally, the MSAE is constructed entirely of differentiable operators, allowing it to be implemented within an end-to-end neural network, and be discriminatively trained. The MSAE draws motivation both from recent multiscale network topologies and from traditional multiresolution transforms in speech processing. Experimental results show the MSAE to provide clear performance benefits relative to conventional single-branch autoencoders. Additionally, the proposed framework is shown to outperform a variety of state-of-the-art enhancement systems, both in terms of objective speech quality metrics and automatic speech recognition accuracy.

 Speech Enhancement, End-to-End Neural Networks, Multiscale Representations, Multiresolution Transforms

## I Introduction

When captured in realistic acoustic environments, speech signals typically suffer from distortions such as additive noise and reverberation. For human listeners, this can lead to reduced intelligibility [1] and increased listener fatigue [2]. It can also result in performance degradation for automated speech applications such as speech and speaker recognition [3, 4, 5, 6, 7]. Speech enhancement can be employed to improve the perceptual quality of captured signals, and to curb these negative effects. For many decades, speech enhancement relied on statistical model-based techniques [8, 9, 10, 11]. Recently, however, deep neural networks (DNNs) have achieved impressive performance improvements due to their high modelling capacity [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36].

Neural network speech enhancement systems can be categorized into _generative_ and _mask-based_ approaches. Generative systems are regression-based and synthesize the enhanced waveform using a set of learned filters, generally operating without much constraint on the output [13, 14, 32, 33, 34, 35, 36]. While they can freely synthesize output waveforms and potentially reconstruct speech signal components that have been attenuated due to channel effects, they can also result in distorted speech signals or unnatural residual noise. Mask-based approaches, on the other hand, are constrained to apply a multiplicative mask in a time-frequency space in order to suppress interfering signal components [12, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]. While this constraint can help avoid distorted speech, it also limits the ability of the system to reconstruct any signals components not present in the original waveform.

This paper focuses on mask-based approaches to neural network speech enhancement. In such systems, an encoder first transforms the input signal into an embedding space. A multiplicative mask is then applied to suppress signal components from interfering sources. Finally, a decoder synthesizes the output waveform. A crucial component of mask-based end-to-end enhancement networks is the autoencoder which maps input signals into an embedding space which is effective at separating speech and noise components. Some prior studies have relied on mappings such as the Short-Time Fourier Transform (STFT) or Discrete Cosine Transform (DCT) [20, 22, 30, 37, 38, 39], while others have explored trainable encoders and decoders [29, 28, 12].

The human cochlea is well known to implement a complex set of auditory filters that reflect a broad trade-off between frequency precision and temporal resolution [40], extracting a multiscale representation of speech signals. The resulting filterbank conveys a non-uniform time-frequency tiling with a general emphasis on frequency resolution at the lower end of the auditory range and finer temporal resolution at the upper end. For decades speech scientists have successfully exploited the ear's frequency characteristics to great effect. The input features to the vast majority of speech-derived analysis methods include some form of frequency scale manipulation (e.g. mel and bark scale warping) that emulates the cochlea's spectral properties. Similarly, but to a lesser degree, the ear's temporal characteristics have been mimicked via a multiplicity of analysis window durations that allow for the detection of transient events without sacrificing the required frequency resolution. The narrow-band and wide-band spectrogram [41] represent two extremes of time-frequency trade-off, while methods such as the multi-resolution short-time Fourier Transform [42], Constant-Q transform [43], and wavelet filtering
[44] implement filtering schemes that reflect the cochlea's temporal-spectral properties to some degree.

In the field of deep learning, several recent studies have explored neural network topologies with multiscale representations [45, 46, 47, 48]. Such approaches typically aggregate representations across network layers in order to simultaneously model information at different scales. Multiscale network architectures have been shown highly successful for tasks such as image classification and segmentation [45, 46, 47]. Additionally, [48] and [49] showed the effectiveness of multiscale networks in modeling speech for the task of speaker verification.

This paper proposes the multiscale autoencoder (MSAE) for mask-based end-to-end neural network speech enhancement, which is motivated both by recent multiscale network topologies and traditional multiresolution transforms in speech processing. The framework provides the encoder and decoder mappings required by mask-based approaches, and is not specific to the architecture used for mask estimation. The MSAE performs spectral decomposition of an input waveform within separate band-limited branches, each operating with a different rate and scale, to extract a sequence of embeddings. The proposed framework features intuitive parameterization of the autoencoder, including a flexible spectral band design based on the Constant-Q transform. Additionally, the MSAE is constructed entirely of differentiable operators, allowing it to be implemented within an end-to-end neural network, and discriminatively trained for the task of speech enhancement.

To assess the performance of the proposed MSAE framework, it is combined with an example mask estimator based on the \(U\)-Net architecture [50] to form the MSAE-UNet end-to-end enhancement system. The paper provides experimental results comparing the performance of MSAE-UNet to several state-of-the-art methods in terms of objective speech quality metrics, and shows the proposed system to significantly outperform them. Additionally, the paper presents experimental results for automatic speech recognition using enhancement as a pre-processing step, and again clearly shows the benefit of the proposed framework.

This paper is organized as follows. Section II provides an overview of mask-based end-to-end speech enhancement systems. The MSAE framework is presented in Section III. Section IV then introduces an example mask estimation architecture, that when integrated in the proposed autoencoder framework forms the end-to-end MSAE-UNet. Experimental results are presented in VI and Section VII provides conclusions.

## II Mask-Based End-to-end Enhancement

Let \(\mathbf{s}\in\mathbb{R}^{D}\) be a clean speech waveform, where

\[\mathbf{s}=\left[s\left(1\right),\ldots,s\left(D\right)\right]^{T}. \tag{1}\]

Let an observed speech waveform, \(\mathbf{x}\in\mathbb{R}^{D}\), be similarly defined. Here, the observed signal is captured in a real-world environment, and may suffer from distortions such as additive noise and reverberation. In this paper, speech enhancement is defined as the task of inferring the clean waveform, \(\mathbf{s}\), from the observed version, \(\mathbf{x}\), yielding the estimate \(\hat{\mathbf{s}}\in\mathbb{R}^{D}\).

This paper focuses on a mask-based approach to end-to-end neural network speech enhancement. Specifically, we use the \(b\)-Net framework from [12], illustrated in Figure 1, and named for its likeness to the lower-case letter. The general network architecture is delineated by its application of a multiplicative mask in a general, possibly learned, embedding space. Similar networks have been explored in the context of source separation [28, 51] as well as signal enhancement [12]. The \(b\)-Net structure can be interpreted as a generalization of statistical model-based speech enhancement methods [8, 9, 10, 11]. With these earlier systems the short-time magnitude spectrogram is extracted from the noisy input waveform, manipulated via a multiplicative mask, and the output waveform is then generated from the enhanced spectrogram through overlap-and-add synthesis using the original noisy phase signal. With \(b\)-Net, the Fourier analysis and overlap-and-add synthesis are replaced by generalized encoder and decoder mappings.

Within the \(b\)-Net framework, as can be observed in Figure 1, an encoder network extracts a set of embeddings from the input according to the generalized mapping

\[\mathbf{Z}=\mathcal{E}\left(\mathbf{x}\right), \tag{2}\]

where \(\mathbf{Z}\) is a tensor of size \(T\times K\times C\). Here, the first axis corresponds to time, and \(T\) denotes the number of frames extracted from the input waveform. The second axis corresponds to frequency, and \(K\) denotes the number of bins used in the frequency representation. Finally, \(C\) represents the number of channels in the feature map \(\mathbf{Z}\). Enhancement is performed via multiplicative masking in the embedding space defined by \(\mathcal{E}\), so that interfering signal components are appropriately attenuated. A mask estimation network extracts this multiplicative mask according to

\[\mathbf{M}=\mathcal{M}\left(\mathbf{Z}\right), \tag{3}\]

where \(\mathbf{M}\) is a tensor of the same size as \(\mathbf{Z}\), and contains values in the range \([0,1]\). Enhanced versions of the embeddings are obtained via element-wise multiplication according to

\[\hat{\mathbf{Z}}=\mathbf{M}\otimes\mathbf{Z}, \tag{4}\]

Fig. 1: Overview of the \(b\)-Net Architecture [12]. Note that with the masking mechanism disabled, the \(b\)-Net results in an autoencoder.
where \(\otimes\) denotes the Kronecker product. Finally, a decoder network extracts the output waveform from the enhanced embeddings according to the generalized mapping

\[\hat{\mathbf{s}}=\mathcal{D}\left(\hat{\mathbf{Z}}\right). \tag{5}\]

Note that the overall operation of the \(b\)-Net framework can be expressed concisely as

\[\hat{\mathbf{s}}=\mathcal{D}\left(\mathcal{M}\left(\mathcal{E}\left(\mathbf{x} \right)\right)\otimes\mathcal{E}\left(\mathbf{x}\right)\right). \tag{6}\]

A key feature of the \(b\)-Net architecture is the ability to disable the masking mechanism. This results in an autoencoder path expressed as

\[\hat{\mathbf{x}}=\mathcal{D}\left(\mathcal{E}\left(\mathbf{x}\right)\right), \tag{7}\]

where \(\hat{\mathbf{x}}\) is the reconstructed version of the input \(\mathbf{x}\). Typically the autoencoder path is able to produce an accurate reconstruction of the input, leading to the approximation

\[\mathbf{x}\approx\mathcal{D}\left(\mathcal{E}\left(\mathbf{x}\right)\right). \tag{8}\]

Note that in general, end-to-end architectures do not contain an analogous autoencoder path, such as in Fully Convolutional Networks (FCNs) [21, 24, 26]. The existence of this path allows the user to dynamically control the level of noise suppression via a minimum gain level by adapting (6) so that

\[\hat{\mathbf{s}}=\mathcal{D}\left(\max\left\{G_{min},\mathcal{M}\left( \mathcal{E}\left(\mathbf{x}\right)\right)\right\}\otimes\mathcal{E}\left( \mathbf{x}\right)\right), \tag{9}\]

where \(G_{min}\in[0,1]\) can be tuned during inference [12]. In this way, the user can efficiently navigate the inherent trade-off between noise suppression and speech quality. In [12], it was shown that dynamic control of the level of noise suppression can provide benefits in enhancement performance, both for objective speech quality metrics and for subjective listening tests.

The focus of this paper is the design of the autoencoder path defined in (7). Several studies have used the Short-Time Fourier Transform (STFT) to define encoder and decoder mappings [30, 37, 20, 38], while other work has relied on the Discrete Cosine Transform (DCT) [22, 39]. Finally, trainable \(1\)-dimensional Convolutional Neural Network (CNN) layers have showed promise for learning an embedding space for effective masking-based speech enhancement [12, 28, 29]. In the following section, the MSAE is developed to provide a multirate and multiscale embedding space for mask-based speech enhancement.

## III The Multiscale Autoencoder

This section presents the Multiscale Autoencoder. First, definitions are provided for the analysis operator, \(\mathcal{A}\), and synthesis operator, \(\mathcal{S}\), which represent the basic building blocks of the MSAE. Next, the encoder network is described, which performs spectral decomposition of an input waveform within separate band-limited branches, each operating with a different rate and scale. Finally, the decoder network is described, which synthesizes the output waveform from the encoder output. Note that the encoder and decoder networks can be constructed entirely using standard neural network operations, allowing discriminative training of autoencoder parameters.

### _The Analysis Operator_

The analysis operator, \(\mathcal{A}\), performs a band-limited spectral decomposition of the input waveform \(\mathbf{x}\), outputting a tensor \(\mathbf{Y}\). It is parameterized by the tuple \((N,\omega_{1},\omega_{2})\), were \(N\) is the duration of the analysis window in samples, and \([\omega_{1},\omega_{2}]\) is the range of normalized frequencies included in the analysis.

The analysis operator relies on an invertible transform, either fixed or learned, to perform spectral analysis. The DFT represents one possible instantiation of the analysis operator and is used in this study, but other transforms, e.g. the DCT, can also be used. Let \(\mathbf{F}=[\mathbf{f}_{1},\ldots,\mathbf{f}_{N}]\) be the \(N\)-dimensional DFT transform. Define \(\mathbf{W}\) to contain the basis vectors of \(\mathbf{F}\) which correspond to the intended spectral range \([\omega_{1},\omega_{2}]\), i.e.

\[\mathbf{W}=\left[\mathbf{f}_{\left\lceil N\omega_{1}/2\right\rceil},\ldots, \mathbf{f}_{\left\lceil N\omega_{2}/2\right\rceil}\right]\in\mathbb{R}^{N \times K_{W}}, \tag{10}\]

where \(K_{W}=\left\lfloor N\omega_{2}/2\right\rfloor-\left\lceil N\omega_{1}/2 \right\rceil+1\). To perform spectral analysis, the \(\mathcal{A}\) operator applies \(4\) parallel \(1\)-dimensional CNN layers, using the kernels \(\mathfrak{Re}\left(\mathbf{W}\right)\), \(\mathfrak{Re}\left(-\mathbf{W}\right)\), \(\mathfrak{Im}\left(\mathbf{W}\right)\), and \(\mathfrak{Im}\left(-\mathbf{W}\right)\), respectively. Each layer uses a stride of \(N/2\) and Rectified Linear Unit (ReLU) activation functions, resulting in parallel tensors of shape \(\left\lfloor 2D/N\right\rfloor\times K_{W}\). The outputs of the CNN layers are then concatenated to form the tensor \(\mathbf{Y}\) with shape \(\left\lfloor 2D/N\right\rfloor\times K_{W}\times 4\). Since \(\mathbf{x}\) is a real-valued signal, its spectrum is conjugate-symmetric. For the sake of computational efficiency, the kernel \(\mathbf{W}\) in (10) therefore only contains DFT basis functions in the normalized frequency range \([0,1]\), since the remainder of the spectrum can be reconstructed from these basis functions.

Note that the motivation for using \(4\) CNN layers is to enable representation of complex numbers as a vector of non-negative values. However, for the sake of efficiency, the \(4\) CNN layers can be implemented as \(2\) CNN layers with kernels \(\mathfrak{Re}\left(\mathbf{W}\right)\) and \(\mathfrak{Im}\left(\mathbf{W}\right)\). The outputs can then be expanded to \(4\) parallel layers by separately considering non-negative and negative values of each, prior to the ReLU activations.

For improved spectral properties, windowing functions can be leveraged when constructing the kernel in (10). That is, a window \(\mathbf{h}\in\mathbb{R}^{N}\) can be applied to the DFT basis functions as \(\mathbf{f}_{k}\leftarrow\mathbf{h}\otimes\mathbf{f}_{k}\). Note that the chosen windowing function should provide unity overlap-and-add when used in conjunction with the synthesis operator [52]. In this study, the square-root of the Hanning window is used.

### _The Synthesis Operator_

The synthesis operator, \(\mathcal{S}\), generates a band-limited waveform \(\mathbf{x}\) from the tensor \(\mathbf{Y}\). It applies a \(1\)-dimensional Transpose CNN layer to each of the \(4\) slices in \(\mathbf{Y}\), using the corresponding kernels defined in Section III-A. A stride of \(N/2\) is used for each layer, and the outputs are summed to form \(\mathbf{x}\). Note that when identically parameterized, the analysis operator inverts the synthesis operator, i.e.

\[\mathbf{Y}=\mathcal{A}\left(\mathcal{S}\left(\mathbf{Y};N,\omega_{1},\omega_{2} \right);N,\omega_{1},\omega_{2}\right). \tag{11}\]

Note that \(\mathbf{W}\) in (10) only contains DFT basis functions within the normalized frequency range \([0,1]\), omitting the other half of the spectrum. Since \(\mathbf{x}\) is a real-valued signal, its spectrum is conjugate-symmetric, and the spectrum can be
reconstructed by simply scaling certain DFT basis functions during synthesis. Specifically, when constructing \(\mathbf{W}\) in the synthesis operator, all basis functions \(\mathbf{f}_{k}\) are scaled by \(2\), except for those corresponding to the DC and Nyquist frequencies.

### _The Encoder Network_

Having defined the analysis and synthesis operators, the encoder and decoder networks can be presented. The encoder performs spectral decomposition of an input waveform \(\mathbf{x}\) within separate band-limited branches, each operating with a different rate and scale, and outputs a tensor, \(\mathbf{Z}\), providing the mapping in (2). The network relies on the analysis operator \(\mathcal{A}\) for spectral processing. The encoder network is parameterized by the number of branches, \(B\), and the duration of the base analysis window in samples, \(N_{o}\). Additionally, the encoder network is parameterized by the normalized frequency ranges which split the input spectrum into \(B\) roughly orthogonal bands, requiring the \(B+1\) band edges \(\omega_{b}\) for \(0\leq b\leq B\).

Figure 2 provides an overview of the encoder network. The input waveform \(\mathbf{x}\) is first processed by \(B\) parallel analysis operators, resulting in spectral decomposition tensors \(\mathbf{Y}_{b}\), each operating at a different frame rate, and each of shape

\[\left\lfloor 2^{-(B-b-1)}D/N_{o}\right\rfloor \tag{12}\] \[\times\left(\left\lfloor 2^{B-b-1}N_{o}\omega_{b}\right\rfloor- \left\lceil 2^{B-b-1}N_{o}\omega_{b-1}\right\rceil+1\right)\times 4.\]

Next, tensors \(\mathbf{Y}_{b}\) are upsampled by rate \(2^{B-b}\) in time, via frame repetition, resulting in tensors \(\mathbf{Z}_{b}\) all operating at the same base frame rate. Finally, tensors \(\mathbf{Z}_{b}\) are concatenated along the \(2^{nd}\) axis, resulting in the output tensor \(\mathbf{Z}\), containing a multi-scale spectral decomposition of input \(\mathbf{x}\). The shape of \(\mathbf{Z}\) is \(\left\lfloor 2D/N_{o}\right\rfloor\times K_{T}\times 4\), where

\[K_{T}=B+\sum_{b=1}^{B}\left(\left\lfloor 2^{B-b-1}N_{o}\omega_{b} \right\rfloor-\left\lceil 2^{B-b-1}N_{o}\omega_{b-1}\right\rceil\right). \tag{13}\]

### _The Decoder Network_

The decoder network, illustrated in Figure 3, synthesizes the output waveform \(\hat{\mathbf{x}}\) from the encoder output \(\mathbf{Z}\), according to (6). First, \(\mathbf{Z}\) is split into branch-specific tensors \(\mathbf{Z}_{b}\). Next, Max-Pooling in time is applied to each, with pool size \(2^{B-b}\) and stride \(2^{B-b}\), yielding tensors \(\mathbf{Y}_{b}\). Finally, the synthesis operator \(\mathcal{S}\) is utilized to generate the reconstructed waveform according to

\[\hat{\mathbf{x}}=\sum_{b=1}^{B}\mathcal{S}\left(\mathbf{Y}_{b};2^{B-b}N_{o}, \omega_{b-1},\omega_{b}\right). \tag{14}\]

Note that besides the trivial case of \(B=1\), the MSAE does not provide perfect reconstruction, which is due to adjacent DFT basis functions at branch edges having different dimensions and therefore not being strictly orthogonal. However, distortion due to reconstruction error was deemed perceptually negligible for MSAE configurations used during experimentation, leading to the approximation in (8).

### _The Design of Spectral Bands_

The spectral bands of the MSAE can be designed in various ways. For example, the spectrum can be split into \(B\) bands of uniform width. In this paper, however, we use a generalized Constant-Q design [43], where the quality factor \(Q\) is defined as the ratio of band center frequency to band width,

\[Q=\frac{\omega_{b}+\omega_{b-1}}{2\left(\omega_{b}-\omega_{b-1}\right)}, \tag{15}\]

and is constant across bands. Rearranging (15) leads to the recursive expression

\[\omega_{b-1}=\rho^{-1}\omega_{b}, \tag{16}\]

where

\[\rho=\frac{2Q+1}{2Q-1}, \tag{17}\]

for \(Q>0.5\). If the uppermost band edge is set to the Nyquist frequency, i.e. \(\omega_{B}=1\), then (16) leads to the band edge design

\[\omega_{b}=\left\{\begin{array}{ll}\rho^{-B+b}&\text{if}\ \ 0<b\leq B\\ 0&\text{if}\ \ b=0\end{array}\right.. \tag{18}\]

Note that if \(Q=1.5\), then \(\rho=2\), representing a dyadic decomposition.

Using the Constant-Q band design, MSAE configurations can be fully parameterized by the tuple (\(B\),\(Q\),\(N_{o}\)). However, in the remainder of the paper signals will be assumed to have a sampling frequency of \(F_{s}=16\)kHz, and configurations will instead be specified by the tuple (\(B\),\(Q\),\(T_{o}\)), where \(T_{o}=N_{o}/F_{s}\), which allows for more intuition when comparing MSAE configurations. Additionally, for MSAE systems with \(B=1\), the band design is irrelevant, and the \(-\) symbol will be used to specify quality factor.

The embedding space defined by the MSAE encoder differs from conventional time-frequency representations with respect to two main characteristics. First, the multiscale processing of the MSAE provides the ability to capture improved spectral resolution in lower frequency bands, while simultaneously achieving greater temporal resolution in higher frequency regions. Second, since the longer analysis windows used to model low frequency regions provide higher spectral resolution, the time-frequency representation defined by the MSAE naturally includes a warped frequency axis with more emphasis applied to lower frequency bands. A magnitude spectral representation of the tensor \(\mathbf{Z}\) can be derived by calculating the root-mean-square (RMS) along the \(3^{rd}\) axis, yielding a \(T\times K\) matrix. Figure 4 provides example magnitude spectral representations of \(\mathbf{Z}\) for various MSAE configurations, for a clean speech signal from the VCTK Noisy and Reverberant set [53] with the transcription "Please call Stella." The top panel shows the \((1,-,2.5\)ms) configuration, which corresponds to conventional wideband processing with an analysis window duration of \(2.5\)ms. The second panel shows the \((1,-,40\)ms) configuration, which corresponds to conventional narrowband processing with an analysis window duration of \(40\)ms. Finally, the bottom panel provides the \((5,2.0,2.5\)ms) configuration, which corresponds to five bands spanning analysis window durations between \(2.5\)ms and \(40\)ms.
Fig. 4: Magnitude spectral representations of \(\mathbf{Z}\) for various MSAE configurations, for a clean speech signal from the VCTK Noisy and Reverberant set [53] with the transcription ”Please call Stella”. The top panel shows the \((1,-,2.5\text{ms})\) configuration, which corresponds to conventional wideband processing with an analysis window duration of \(2.5\text{ms}\). The second panel shows the \((1,-,40\text{ms})\) configuration, which corresponds to conventional narrowband processing with an analysis window duration of \(40\text{ms}\). Finally, the bottom panel provides the \((5,2.0,2.5\text{ms})\) configuration, which operates with five bands spanning analysis window durations between \(2.5\text{ms}\) and \(40\text{ms}\).

Fig. 3: The Decoder Network of the MSAE: The synthesis operator \(\mathcal{S}\) is defined in Section III-B, the \(\downarrow K\) operator denotes Max-Pooling in time with pool size \(K\) and stride \(K\), and stacked blocks denote tensor concatenation along the \(2^{nd}\) axis.

Fig. 2: The Encoder Network of the MSAE: The analysis operator \(\mathcal{A}\) is defined in Section III-A, the \(\uparrow K\) operator denotes nearest-neighbor upsampling in time by rate \(K\), and stacked blocks denote tensor concatenation along the \(2^{nd}\) axis.
As can be observed in Figure 4, the wideband time-frequency representation of the top panel captures fine temporal patterns such as the plosives \(/k/\) and \(/t/\) at \(0.75\)s and \(1.5\)s, respectively. Conversely, the narrowband representation of the second panel achieves better resolution of harmonics and formant frequencies, e.g. the \(/a/\) vowel at \(2.0\)s. The multiscale representation in the bottom panel, however, achieves both wideband representation in higher frequency regions, and narrowband representation in lower frequency regions. Additionally, it can be observed that the multiscale embedding space in the bottom panel results in a warped frequency scale. This frequency warping can also be observed in Figure 5, which illustrates the frequency responses of the \(1\)-dimensional CNN filters utilized by the \((5,2.0,2.5\)ms) MSAE. In Figure 5, it can be observed that a greater number of CNN filters is allocated to lower frequency ranges. Additionally, due to the longer analysis windows applied in the lower frequency bands, the corresponding filters show a narrower passband.

### _The MSAE as a Trainable Network_

The MSAE encoder and decoder networks are composed of differentiable operations, and can be constructed entirely using standard neural network blocks. This allows the kernels \(\mathbf{W}\) to be trained jointly with the mask estimation network for the task of speech enhancement. Note that the kernels can be initialized according to (10), and allowed to adapt during the overall network training process. Additionally, the tensors \(\mathbf{W}\) can be overcomplete, i.e. \(K_{W}>\lfloor N\omega_{2}/2\rfloor-\lceil N\omega_{1}/2\rceil+1\), providing greater modeling capacity in the spectral decomposition performed by the encoder network. In this case, an overcompleteness factor, \(\kappa\), is defined such that

\[K_{W}=\lfloor\kappa\left(\lfloor N\omega_{2}/2\rfloor-\lceil N\omega_{1}/2 \rceil+1\right)\rfloor. \tag{19}\]

In this study, for \(\kappa>1\), the tensor \(\mathbf{W}\) was initialized with DFT basis vectors with interpolated center frequencies.

## IV Mask Estimation

The role of the mask estimator within the \(b\)-Net framework is to map \(\mathbf{Z}\) to a tensor \(\mathbf{M}\) via (3), which is used as a multiplicative mask to perform speech enhancement in the embedding space defined by the encoder. The proposed MSAE framework is not specific to the architecture of the mask estimation network, and is instead compatible with the generalized mapping given in (3). However, in order to assess the performance of the MSAE framework, this study leverages an example mask estimation network based on the \(U\)-Net architecture from [50], which is described in this section. During experimentation, when the example \(U\)-Net mask estimation architecture is combined with the proposed MSAE framework, the resulting end-to-end enhancement system is referred to as the MSAE-UNet.

An overview of the mask estimation network used in this study is provided in Figure 6. As a basic component, the _CNN Block_ is defined as the following series of operations: a \(2\)-dimensional CNN layer, Batch Normalization [54], and non-linear activation functions which are ReLUs unless otherwise stated. This block is parameterized by the dimensions of the CNN filters, the number of input channels, and the number of output channels. Configurations of _CNN Blocks_ are expressed as tuples of these parameters, e.g. \((3\times 3,32,64)\).

As can be observed in Figure 6, the mask estimation network first applies Frequency-wise Normalization. This includes the element-wise \(\log\left(\mathbf{Z}+1\right)\) operator, followed by mean- and variance-normalization across time frames and channels, resulting in zero-mean and unit-variance feature map distributions per frequency bin. Frequency-wise Normalization is designed to promote invariance to domain shifts in the input signals, and is based on the Cepstral Extraction operator in [12]. A _CNN Block_ is then applied to expand the feature map to \(16\) channels prior to processing by the \(U\)-Net architecture,

The mask estimation network next utilizes a \(U\)-Net structure. As can be observed in Figure 6, a contraction path first

Fig. 5: Frequency responses of the \(1\)-dimensional CNN filters utilized by the MSAE with \((5,2.0,2.5\)ms) configuration.

Fig. 6: Overview of the Mask Estimation Network: \(\mathbf{Z}\) is the set of embeddings extracted from the input waveform via (2), and \(\mathbf{M}\) is the multiplicative mask used for enhancement in (6).
applies a series of _Contraction Blocks_, defined in Figure 7, in order to increase the spectral and temporal context captured in the feature map. Max-Pooling using a \(2\times 2\) window size and a \(2\times 2\) stride is first applied, halving the dimensions of the feature map. A _CNN Block_ is then applied to double the number of channels, followed by two more _CNN Blocks_. The _Contraction Blocks_ also provide a second output, which is a skip connection from the input. The contracting path includes \(4\) levels, resulting in an output feature map with \(256\) channels.

The mask estimation network also includes an expansion path which applies a series of _Expansion Blocks_, defined in Figure 8, in order to reconstruct the original resolution of the feature map. Within each _Expansion Block_, a _CNN Block_ is first applied to halve the number of channels, followed by nearest-neighbor upsampling with a \(2\times 2\) window. The feature map is then concatenated with the skip connection from the corresponding level in the contraction path, doubling the number of channels. A _CNN Block_ is then applied to halve the number of channels in the feature map, followed by two additional _CNN Blocks_. After the expansion path, a _CNN Block_ with Sigmoid activation functions is applied to generate a mask tensor \(\mathbf{M}\) which is of the same shape as \(\mathbf{Z}\) and contains output values in the range \([0,1]\) which are appropriate for multiplicative masking.

As Figure 6 illustrates, the modeling capacity of the mask estimation network can be improved with the inclusion of base-level processing between the U-Net's contraction and expansion paths. The example architecture here utilizes a base-level network consisting of \(5\) Squeeze-and-Excitation Residual Network (SE-ResNet) blocks [55]. These blocks, detailed in Figure 9, combine Squeeze-and-Excitation (SE) channel calibration with a Residual Network (ResNet) topology [56], and have been shown to be effective in a variety of tasks.

## V The MSAE-UNet Training Process

This section presents the training process of the MSAE-UNet. Specifically, the derivation of target signals is discussed. Additionally, the generation of parallel training data is outlined. Finally, the training loss is described.

### _Designing Target Signals_

Typically, end-to-end systems are trained with parallel data in which known clean speech is corrupted with additive noise; the system learns the inverse mapping from \(\mathbf{x}\) to \(\mathbf{s}\) through minimization of some distance measure \(d\left(\mathbf{s},\hat{\mathbf{s}}\right)\). However, in many realistic environments, speech signals are captured in

Fig. 8: Overview of the Expansion Block: For a general Input shape of \(T_{i}\times K_{i}\times C_{i}\), the Skip Connection and Output have shape \(2T_{i}\times 2K_{i}\times C_{i}/2\).

Fig. 9: Overview of the Squeeze-and-Excitation Residual Network (SE-ResNet) Block: The Input and Output have the same shape, and \(C_{i}\) denotes the number of channels in the Input feature map.
the presence of reverberation in addition to additive noise and other distortions. While recent end-to-end speech enhancement systems have been shown successful at suppressing additive noise [16, 17, 18, 21, 26, 31], few studies have successfully addressed suppression of reverberation with an end-to-end system [12, 19]. Training an end-to-end enhancement system to learn the mapping from \(\mathbf{x}\) to \(\mathbf{s}\) in the presence of reverberation may be difficult due to the phase distortion caused by convolutional noise.

In this study we utilize the target signal design proposed in [12], which enables the enhancement system to learn joint suppression of additive and convolutional distortion. Let \(\mathbf{v}\) be the reverberated-only version of \(\mathbf{s}\), which excludes any further distortion present in \(\mathbf{x}\). The STFT of \(\mathbf{s}\) is given by \(\mathbf{S}_{m,l}\) where \(m\) and \(l\) are the frequency channel and frame indices. Let \(\mathbf{V}_{m,l}\) be defined similarly. An enhanced version of \(\mathbf{V}\) can be obtained using an oracle Wiener Filter,

\[\mathbf{S}_{m,l}^{*}=\max\left\{\min\left\{\frac{\left|\mathbf{S}_{m,l}\right| ^{2}}{\left|\mathbf{V}_{m,l}\right|^{2}},1\right\},0\right\}\mathbf{V}_{m,l}. \tag{20}\]

The corresponding waveform, \(\mathbf{s}^{*}\), can be synthesized via the inverse STFT. The signal \(\mathbf{s}^{*}\) then represents a version of the reverberant signal \(\mathbf{v}\) with the majority of late reflections suppressed, but with the phase distortion introduced by early reflections still present. This allows an end-to-end system to be trained to perform joint suppression of noise and reverberation by learning a mapping from \(\mathbf{x}\) to \(\mathbf{s}^{*}\) through the minimization of some distance measure \(d\left(\mathbf{s}^{*},\hat{\mathbf{s}}\right)\).

### _Generating Training Data_

Training an end-to-end system requires parallel data consisting of the original (clean) signal, \(\mathbf{s}\), and its corrupted, observed counterpart, \(\mathbf{x}\). Utilizing the target signals discussed in Section V-A also requires the reverberated-only version, \(\mathbf{v}\). In practice, field recorded data of this nature is relatively limited and not of sufficient quantity or scope to be useful for (at least initial) training purposes. As a result, considerable care has been taken to synthetically generate realistic parallel data representing a broad range of operational conditions.

Our in-house training material was dynamically generated from an extensive corpus of speech sources, reverberant channels, and noise conditions. The clean speech consists of 456 hours of audio collected from multiple publicly available corpora (e.g. the Linguistic Data Consortium (LDC), LibriSpeech [57], TIMIT [58]). Currently 2458 unique talkers are utilized, representing a spectrum of languages and ages. Clean speech signals, \(\mathbf{s}\), are randomly selected from this set. The generator then applies a room impulse response to produce \(\mathbf{v}\), and has the capacity to incorporate varying room impulse responses (currently 1708) derived from a number of sources (including [59, 60, 61]) to recreate a generalized ensemble of enclosure sizes and types. Additive noise is then added to produce the observed signal \(\mathbf{x}\). The noise conditions are delineated as background (i.e. ambient) and non-stationary. The 250+ hours of noise samples (collected primarily from the MUSAN [62], Voice-Home [60], Noisex [63], and AudiSet [58]), represent 218 distinct background and 88756 non-stationary noise conditions. In an effort to further expand the scope of observed material, the room impulse responses and additive noise signals are perturbed by random rate-resampling and spectral-shaping. Additionally, the resulting mixed signal is subjected to multiple potential modifications associated with the signal acquisition process, such as clipping, quantization, and the presence of interceding communications channel.

### _Training Loss_

Training an end-to-end speech enhancement system requires a distance measure which operates on time-domain samples. Initial studies on end-to-end enhancement systems optimized network parameters to minimize the mean square-error (MSE) between the output waveform, \(\hat{\mathbf{s}}\), and the clean waveform, \(\mathbf{s}\),

\[d_{MSE}\left(\mathbf{s},\hat{\mathbf{s}}\right)=\frac{1}{D}\sum_{n=1}^{D}\! \left(s\left(n\right)-\hat{s}\left(n\right)\right)^{2}. \tag{21}\]

However, the MSE does not take into account properties of human perception of speech, and may not result in an enhanced signal which optimizes perceptual quality. Recent studies have proposed loss functions which address these issues [15, 21, 25, 31, 16]. In this study, however, we leverage the perceptually-motivated MSE (pMSE) distance proposed in [12].

The MSE loss from (21) can be generalized to include the effects of both signal pre-emphasis [52] and \(\mu\)-law amplitude companding [64], leading to

\[d_{pMSE}\left(\mathbf{s},\hat{\mathbf{s}}\right)= \tag{22}\] \[\frac{1}{D}\sum_{n=1}^{D}\!\left(f_{\mu}\left(s\left(n\right)- \beta s\left(n-1\right)\right)\right.\] \[\left.-f_{\mu}\left(\hat{s}\left(n\right)-\beta\hat{s}\left(n-1 \right)\right)\right)^{2}\!\!\!,\]

where \(\beta\) is the pre-emphasis constant, and the \(\mu\)-law companding function is given by

\[f_{\mu}\!\left(s\left(n\right)\right)=\text{sign}\!\left(s\left(n\right)\right) \!\frac{\log\left(1+\mu\left|s\left(n\right)\right|\right)}{\log\left(1+\mu \right)}. \tag{23}\]

Equation (22) offers a generalized distance measure which can be tuned to account for various properties of human perception via the parameters \(\beta\) and \(\mu\). Note that for settings \(\beta=0.0\) and \(\mu\to 0.0\), the proposed measure is equivalent to the standard MSE in (21). More details regarding the motivation for the pMSE cost can be found in [12].

Note that several studies have proposed the use of spectral-based cost functions which don't take into account the phase signal [65, 31, 66, 32]. During our experimentation, we found that while the inclusion of such spectral-based components in the overall training loss provided some advantage with the objective speech quality measures, they also lead to the inclusion of 'buzzy' artifacts in the resulting speech, particularly in the higher frequencies. For this reason the time-domain pMSE loss, which offers both a degree of perceptual modelling as well as a sensitivity to speech phase distortion, was utilized exclusively in this study.
In [12], a dual-path loss was proposed for mask-based end-to-end enhancement systems in order to ensure that the autencoder path provides high quality reconstruction when encoder and decoder filters are trained. The dual-path loss is given by \(d_{pMSE}\left({{\bf{s}},{\tilde{\bf{s}}}}\right)+d_{pMSE}\left({{\bf{x}},{\tilde{ \bf{x}}}}\right)\), where the additional second term enforces the approximation from (8). In this study, the dual-path loss was utilized whenever MSAE kernels were learnable, as discussed in Section III-F.

The construction of training data can have significant impact on the behavior of the resulting neural network enhancement systems. For example, the ratio of active speech to inactive speech within the data can change the system's emphasis on speech quality versus noise suppression. To better control this tradeoff during training, a prior probability of active speech can be induced by weighting individual training waveforms. It is straightforward to determine whether a clean reference sample contains a significant amount of active speech. For a given training batch, let \(M_{1}\) and \(M_{0}\) denote the total number of samples that are determined to correspond to active speech and inactive speech, respectively. To introduce an effective prior, \(\pi\in\left[0,1\right]\), active speech and inactive speech segments can be weighted by \(\pi\left({{M_{0}}+{M_{1}}}\right)/{M_{1}}\) and \(\left({1-\pi}\right)\left({{M_{0}}+{M_{1}}}\right)/{M_{0}}\), respectively. In this way, using a large \(\pi\) focuses network training on speech quality, whereas a small value puts more emphasis on noise suppression.

## VI Experimental Results

This section presents experimental results for speech enhancement using MSAE-Unet, both in terms of objective speech quality metrics and automatic speech recognition performance. During experimentation, a variety of MSAE configurations are analyzed. For all systems, input waveforms are split into windows of dimension \(D=20480\), corresponding to \(1.28\) sec, with \(50\%\) overlap, before processing. Output waveforms are then reconstructed using overlap-and-add. All windows are scaled to unit variance prior to enhancement, and gains are reintroduced after network processing. Networks are trained using target signals generated according to Section V-A. Parallel training data is generated as discussed in Section V-B. Networks are optimized by minimizing the pMSE loss described in Section V-C, using an active speech prior of \(\pi=0.75\). In all testing, the minimum gain, \(G_{min}\), is set to \(-50\)dB, which is observed to provide a good trade-off between noise suppression and speech quality during informal listening.

### _Objective Speech Quality_

Speech enhancement experiments are performed on the Voice Cloning Toolkit (VCTK) Noisy and Reverberant Corpus [53], a parallel clean-corrupted set comprised of \(824\) speech files with synthetically added reverberation and background noise. To assess performance, a variety of objective speech quality metrics are used. Specifically, results are reported in terms of Perceptual Evaluation of Speech Quality (PESQ) [67] and Short-time Objective Intelligibility (STOI) [68]. Results are also reported in terms of the Composite Signal (CSIG), Composite Background (CBAK), and Composite Overall (COVL) quality scores from [69], which are weighted combinations of various signal quality measures where the weightings are trained to correlate with subjective listening tests. Finally, the relative improvement in COVL compared to the unprocessed signal is provided to offer more clarity. In all tables, bold entries denote the best result for each metric.

#### Vi-A1 The Effect of Analysis Window Duration

The first set of experiments aim at exploring the effect of analysis window duration in neural network speech enhancement systems. Table I provides speech quality measures on the VCTK Noisy and Reverberant Corpus for MSAE-UNet with the single-branch MSAE configuration, \((1,-,T_{o})\), and varying \(T_{o}\). Note that short window durations, e.g. \(T_{o}=5\)ms, correspond to conventional wideband analysis in the autoencoder, whereas long windows, e.g. \(T_{o}=40\)ms, correspond to conventional narrowband analysis. From Table I, it can be observed that analysis window duration has a significant impact on speech quality metrics, with the best performance achieved for intermediate durations. With respect to COVL, an analysis window duration of \(T_{o}=10\)ms maximizes the quality of enhanced speech. These results show the sensitivity of neural network speech enhancement to the design of analysis windows, and motivates the MSAE framework which can represent separate frequency bands with appropriate window durations.

#### Vi-A2 The Multi-Branch MSAE Framework

The next set of experiments aim at assessing the benefits of the proposed MSAE framework within mask-based enhancement systems. An important parameter of the MSAE is the number of branches, \(B\). Table II provides speech quality measures on the VCTK Noisy and Reverberant Corpus for MSAE-UNet with the multi-branch MSAE configuration, \((B,1.5,2.5\)ms), and varying number of branches. All systems use a default dyadic spectral band design, corresponding to the quality factor \(Q=1.5\). It can be observed that speech quality improves significantly for MSAE configurations with multiple branches, when compared to the single-branch system, with the best performance achieved for \(B=5\). Specifically, the \(B=5\) MSAE provides an additional \(16\%\) relative improvement in COVL compared to the single-branch autoencoder. These results clearly show the benefit of the proposed multiscale autoencoder, relative to conventional systems.

#### Vi-A3 Spectral Band Design within the MSAE

The MSAE is also parameterized by the Constant-Q band design. The next experiments explore the role of the quality factor, \(Q\), in enhancement performance. Table III provides speech quality measures on the VCTK Noisy and Reverberant Corpus for the MSAE configuration \((5,Q,2.5\)ms), for different values of \(Q\). It can be observed that relative to a dyadic band design, speech quality can be improved by tuning the quality factor to \(Q=2.0\), which corresponds to narrower bands and a higher degree to frequency warping than that illustrated in Figure 5. Specifically, the improved band design provides an additional \(2\%\) relative improvement in COVL.

#### Vi-A4 Trainable MSAE Filters

The final set of experiments explore the use of trainable encoder and decoder filters within the MSAE framework. As discussed in Section III-F, the MSAE encoder and decoder networks are composed of differentiable operations, and can be constructed entirely using standard neural network blocks, allowing the kernels \(\mathbf{W}\) to be trained
jointly with the mask estimation network. Table IV compares speech enhancement performance of the MSAE-UNet with the \((5,2.0,2.5\)ms) MSAE configuration, with fixed and learned kernels. In the latter case, the overcompleteness factor, \(\kappa\), is specified. In the table, it can be observed that allowing MSAE filters to adapt during training provides performance improvements across speech quality metrics. Specifically, the use of learned filters and an overcompleteness factor \(\kappa=1.5\) provides an additional \(6\%\) relative improvement in COVL. Note that in the remainder of this section, only results for this MSAE-UNet configuration will be reported.

#### Iv-B5 An Ablation Study

In order to give a clear summary of the previous experiments, Table V provides an ablation study of neural network speech enhancement using the MSAE framework. The table first includes objective speech quality metrics for the original degraded signals in the VCTK Noisy and Reverberant Corpus. The next row provides results for a baseline mask-based enhancement system which uses a single-branch autoencoder with a \(10\)ms analysis window, and is trained to learn the mapping from noisy signals \(\mathbf{x}\) to clean signals \(\mathbf{s}\), using the standard MSE loss. Each successive row then provides results when cumulatively adding an additional feature to the enhancement system.

In Table V, the poor results of the baseline system may be due to the use of clean target signals. The presence of potentially severe reverberation in the training data may introduce a high degree of phase distortion, making it difficult for an end-to-end system to learn the mapping from the noisy signal \(\mathbf{x}\) to the clean signal \(\mathbf{s}\). In the third row, Wiener-filtered target signals described in Section V-A are instead used during training, leading to a sigificant improvement in enhancement performance. In the fourth row, the conventional MSE loss is replaced by the perceptual MSE during training, as discussed in Section V-C, leading to further performance improvements.

In Table V, the fifth row provides results for a multi-branch MSAE. Specifically, the \((5,1.5,2.5\)ms) MSAE configuration with fixed \(\mathbf{W}\) filters is used, providing further performance improvements. Next, the Constant-Q spectral band design within the MSAE was tuned from a dyadic decomposition to a quality factor of \(Q=2.0\), yielding the results in the sixth row of the table. Finally, the seventh row provides results for the use of adapted MSAE kernels, providing a slight performance improvement in certain objective metrics.

#### Iv-B6 Comparison to State-of-the-Art Enhancement Systems

Finally, the MSAE-Unet was compared to state-of-the-art neural network speech enhancement systems, and a variety of baselines was chosen to assess the performance of the proposed system. First, the MetricGAN+ [34] system uses a Bidirectional Long Short Term Memory (BLSTM) architecture [13], trained within a Generative Adversarial Network (GAN)
[70] framework using a perceptually-motivated loss. Next, the Mimic system [23] uses a neural network architecture, and is trained jointly to both minimize the MSE loss and to improve senone classification of the enhanced speech. Finally, the DE-MUCS system [32] uses a U-Net architecture, and is trained with a composite time-domain and spectral-domain loss. Note that the MetricGAN+ and Mimic models are provided by the SpeechBrain Toolkit [71], and were trained with data from the VoiceBank [72] and Deep Noise Suppression (DNS) [73] corpora. The DEMUCS model was trained on the VCTK Noisy [14] and DNS corpora. Table VI provides a performance comparison of MSAE-UNet with the baseline systems on the VCTK Noisy and Reverberant Corpus. Additionally, it includes the number of trainable parameters in each model studied. As can be observed, the proposed system achieves significant performance improvements across speech quality metrics. Specifically, the MSAE-UNet provides additional relative improvements in COVL of \(12\%\)-\(24\%\) compared to the baseline systems. This is especially noteworthy considering the small size of MSAE-UNet relative to the Mimic and DEMUCS models.

### _Automatic Speech Recognition_

Enhancement can not only improve the perceptual quality of distorted speech, alleviating listener fatigue, but can also improve performance of applications such as automated speech recognition (ASR). This section provides experimentation using enhancement as pre-processing for speech recognition. In order to provide general results, two ASR models are studied, namely Deep Speech [74] and the _Large_ variant of Whisper [75], both representing state-of-the-art end-to-end neural network approaches. Experiments are performed on the VCTK Noisy and Reverberant Corpus and the Voices Obscured in Complex Environmental Settings (VOiCES) Corpus [61]. The latter is a large set of far-field audio collected by replaying speech and noise distractor signals within various reverberant rooms. The corpus captures a wide range of acoustic environments by varying room dimensions, noise types, microphone types, and microphone placements. Throughout experimentation, results are reported as Word Error Rate (WER), calculated using the NIST Scoring Toolkit (SCTK) [76].

Table VII provides ASR results for the Deep Speech and Whisper models, when using enhancement as a pre-processing step. It can be observed that with the Deep Speech model, the baseline enhancement systems generally result in performance degradation relative to the original signals. This degradation is especially large for the VOiCES Corpus. The MSAE-UNet system, however, provides ASR improvements for both benchmarks. For the Whisper model, the baseline enhancement systems again lead to performance degradation in many cases. The MSAE-UNet system, however, provides clear performance improvements for the VCTK benchmark, and minor degradation for the VOiCES Corpus.

## VII Conclusion

This paper proposed the multiscale autoencoder (MSAE) for mask-based end-to-end neural network speech enhancement. This framework provides the encoder and decoder mappings for such networks, and is not specific to the mask estimation architecture. The MSAE performs spectral decomposition of an input waveform within separate band-limited branches, each operating with a different rate and scale, to extract a sequence of multiscale embeddings. The proposed framework features intuitive parameterization of the autoencoder, including a flexible spectral band design based on the Constant-Q transform. Additionally, the MSAE is constructed entirely of differentiable operators, allowing it to be implemented within an end-to-end neural network, and discriminatively trained for the task of speech enhancement. To assess the performance of the proposed MSAE framework, it was integrated with an example mask estimator based on the \(U\)-Net architecture. The
resulting end-to-end enhancement system was shown to outperform several state-of-the-art speech enhancement methods both in terms of objective speech quality metrics and automatic speech recognition accuracy.
