plaintop table

# Listen, Look and Deliberate: Visual context-aware speech recognition using
pre-trained text-video representations

###### Abstract

In this study, we try to address the problem of leveraging visual signals to
improve Automatic Speech Recognition (ASR), also known as visual context-aware
ASR (VC-ASR). We explore novel VC-ASR approaches to leverage video and text
representations extracted by a self-supervised pre-trained text-video
embedding model. Firstly, we propose a multi-stream attention architecture to
leverage signals from both audio and video modalities. This architecture
consists of separate encoders for the two modalities and a single decoder that
attends over them. We show that this architecture is better than fusing
modalities at the signal level. Additionally, we also explore leveraging the
visual information in a second pass model, which has also been referred to as
a ‘deliberation model’. The deliberation model accepts audio representations
and text hypotheses from the first pass ASR and combines them with a visual
stream for an improved visual context-aware recognition. The proposed
deliberation scheme can work on top of any well trained ASR and also enabled
us to leverage the pre-trained text model to ground the hypotheses with the
visual features. Our experiments on HOW2 dataset show that multi-stream and
deliberation architectures are very effective at the VC-ASR task. We evaluate
the proposed models for two scenarios; clean audio stream and distorted audio
in which we mask out some specific words in the audio. The deliberation model
outperforms the multi-stream model and achieves a relative WER improvement of
6% and 8.7% for the clean and masked data, respectively, compared to an audio-
only model. The deliberation model also improves recovering the masked words
by 59% relative.

Index Terms—  Multi-modal learning, visual context aware speech recognition,
robust speech recognition, deliberation model

##  1 Introduction

Humans are inherently able to process and combine multi-modal information to
understand language. For example, when we watch a narrated instructional
video, we leverage all kinds of visual cues to resolves ambiguities we might
have in understanding what has been said. Consequently, the counterpart
problem of how to leverage visual information to improve ASR and natural
language processing models has gained much attention in recent years. Previous
works have demonstrated improvement by exploiting visual information for ASR
[1, 2, 3, 4, 5, 6], translation [7], question-answering [8], dialog systems
[9] , and text summarization [10].

It should be noted that Visual Context-aware Automatic Speech Recognition (VC-
ASR) is different from the task of Audio-Visual Speech Recognition (AV-ASR)
[11, 12]. AV-ASR can only be applied to videos in which the speaker’s face
information is available. However, it is not straightforward to apply such a
model to open-domain videos. On the other hand, in VC-ASR, we leverage
general-level visual information such as actions, objects, and places. This
makes it suitable for open-domain videos but also much more challenging since
the correlation between visual and speech signals in VC-ASR task is usually
much smaller than AV-ASR.

For ASR, [13] was one of the early works that investigated leveraging visual
context to improve the modeling performance. They leveraged the objects and
places features by early fusion, where they appended the visual features to
the audio features, or normalized the audio features. This latter technique
has also been referred to as visual adaptive training (VAT) in [13].
Leveraging visual information for language modeling has been investigated in
[3], where they used the visual features as the beginning token of each
sentence and demonstrated that the visual features have the potential to
improve the ASR model at the language modeling level in addition to acoustic
modeling. In [5], Ozan et al. attempted to exploit the visual features for a
sequence-to-sequence (S2S) model using VAT or image captioning-inspired
approaches that leverage the visual features to initialize the encoder and/or
the decoder. However, in the aforementioned studies, the contribution of
visual features is not clear. For example, Ozan et al. [5] observed that the
trained multi-modal model performs the same if the visual information is
discarded during the inference. In our study, to validate whether the
improvement is resulting from additional visual information or due to some
regularization effects, inspired by [14], we will also evaluate the VC-ASR
models with random misaligned videos during inference time.

Nearly all the previous VC-ASR studies have extracted the visual features
using models trained to classify pre-defined visual categories such as
objects, places, and actions. However, the visual variability can not be
covered by such a limited number of classes, and consequently, the features we
get from these models are only representing the visual information partially.
To address this problem, multi-modal joint pre-training [15, 16, 17, 18] is
shown to be a potential solution. Miech et al. [15] have pre-trained a text-
video model to map the video and text to an embedding space where the
similarity between text and video is high when they are semantically similar.
In this study, we leverage this pre-trained model to extract rich visual
information. Furthermore, as opposed to previous studies that average pool all
the features of a video into a fixed vector, we extract a sequence of visual
features as an input stream for the VC-ASR models.

Given two sequences of the audio and visual features, we propose to use a
multi-stream attention-based S2S model to leverage these two inputs. In this
model, which is an extension of an encoder-decoder based S2S model [19, 20],
two attention heads are used to attend on audio and visual features to form a
combined context vector. In addition to this model, we also propose to
leverage the visual context to post-process the N-best hypotheses generated by
an audio-only model. For this purpose, we use a deliberation model
architecture [21]. Previous studies have demonstrated the efficacy of the
deliberation model for speech recognition [22, 23], machine translation [21],
and speech-to-text translation [24]. For ASR, Hu et al. [22], have shown
improvement using the deliberation model on top of a streaming recurrent
neural network transducer (RNN-T). In their study, the deliberation model had
access to the future and past context and was able to refine the hypotheses
produced by the first pass streaming model. In our study, we propose to
leverage the visual context in a deliberation model trained on top of a pre-
trained (audio-only) S2S model. The motivation behind this approach is that
the deliberation can leverage the visual information to resolve the
ambiguities in the N-best hypotheses and refine them to generate higher
quality outputs. An important advantage of the deliberation model is that it
has access to the N-best hypotheses generated by the first decoder. Given the
hypotheses, we can leverage the pre-trained text-video model to extract the
visual context for each word in the hypotheses. Feeding the visual context
along with the word embeddings to the deliberation model would help the model
to resolve the ambiguities easier. This approach will be referred to as
Visually Grounded Hypotheses (VG-Hypotheses).

Since the speech modality is inherently a noisy medium, one practical scenario
for which the visual information can be beneficial is where the speech
modality is distorted. For VC-ASR, Tejas et al. [1] have investigated the
utility of visual context where specific words are masked out in the audio
modality. They concluded that the visual information is helpful to recover the
masked words while the audio-only model is not able to. Inspired by this
study, we also investigate how our proposed VC-ASR approaches perform when we
replace some words in the audio modality with white noise.

In this study, we conduct the experiments on HOW2 data [25] which is collected
from instructional videos on YouTube. We first examine the efficacy of the
proposed multi-stream model to leverage the visual features for an improved
VC-ASR model. The experiments demonstrate that this model performs 3.4%
relatively better than an audio-only model for clean data and 6.4% for the
masked data. The deliberation model trained on top of an audio-only model
performs comparable with the multi-stream model, however, our analyses show
that the deliberation model is superior in handling uncorrelated visual
context. Furthermore, our results show that grounding hypotheses with the
visual context (VG-Hypotheses) brings additional benefits in both the clean
and masking scenarios. We hypothesized that VG-Hypotheses performs better when
the N-best hypotheses from the first decoder are of greater quality.
Therefore, we also train a deliberation 2nd pass model on top of a multi-
stream audio-video 1st pass since it has better WER than the audio-only model.
The resulting model achieves the best performance for the clean and masked
data, with 6.0% and 8.7% relative WER improvement for clean and masked data,
respectively, and 59% relative improvement in recovering the masked words
compared to an audio-only model.

##  2 Visual features

To extract the visual features, we leverage a jointly pre-trained video-text
embedding model [15]. This model is trained to map 32 frames of video and its
transcription to an embedding space where the similarity between these
embeddings is high when they are similar semantically and is low otherwise.
The detail of this model can be found in [15]. Even though this model has been
trained to map the entire sentence to an embedding, in our analysis we
realized that the representation of each word has a high correlation with some
parts of the video in which an object/action/place associated with that word
appears, as illustrated in Fig.1. This observation inspired us to use the pre-
trained video model to map each video to a sequence of visual embeddings. The
VC-ASR models in this study are designed to attend to this sequence of visual
features to get the corresponding visual context for each word/token.

To match the training conditions of the pre-trained text-video model, for each
video of $T\times H\times W\times 3$, first, we resize the frames resulting in
a $T\times 200\times 250\times 3$ tensor. Then, the feature extraction is
performed on a window of 32 frames with a sliding rate of 16 frames. The
output of the pre-trained video model would be a $\frac{T}{16}\times 1024$
tensor.

Fig. 1: A demonstration of the similarity matrix computed by multiplying word
embedding (generated using the pre-trained text model) and visual features
(generated using the pre-trained video model) of an utterance.

##  3 METHODOLOGY

In this section, we introduce our VC-ASR architectures. First, we describe our
multi-stream S2S model and illustrate the detail of how this model processes
and fuses the two input modalities. Then, we introduce the deliberation model
and how we employ this model for the VC-ASR problem. These approaches are also
illustrated in Fig. 2.

(a) Multi-stream model

(b) Deliberation model

(c) Visually Grounded Hypotheses

Fig. 2: The proposed visual context-aware speech recognition models. For all
the models, the visual features are extracted using the pre-trained video
model. The visual context features are then leveraged by one of three
architecture. (a) Multi-stream model: This model extends an audio-only S2S
model by adding a separate video encoder and an attention model to get the
visual context. (b) Deliberation model: In this model, the visual features are
leveraged in a deliberation model on top of a trained (audio-only) S2S model.
(c) Visually Grounded Hypotheses: This approach extends the deliberation
model. For each hypothesis, the predicted words are used to attend on the
visual stream to get the visual contexts. The resulting visual context vectors
are then appended to the word embeddings before being passed to the hypothesis
encoder.

###  3.1 Multi-stream model

In this approach, the model has two independent encoders to process the audio
and video input streams. Then, two attention heads are used to attend on audio
and visual encoded streams. At every step of outputting a token, the output of
the previous decoder is used as a query to get the audio and video contexts.
Then these context vectors along with the decoder output are passed through a
fusion network (denoted as Context-Fusion) that fuses these vectors into a
combined context vector used to predict the next token. The S2S model outputs
token probabilities which are matched to the ground truth tokens to train the
model.

We have investigated two architectures for the Context-Fusion; concatenation
and gating mechanism.

Concatenation (Cat) – In this architecture, given
$\textbf{V}^{C},\;\textbf{A}^{C},\;\\\ and\;\textbf{Dec}^{O}$ as the visual
context, audio context, and decoder output, respectively, the concatenation
approach is performed as follows,

$Out_{i}=(\textbf{W}^{cat}_{1}Cat(\textbf{V}^{C}_{i},\textbf{A}^{C}_{i})+\textbf{b}^{cat}_{1})+\textbf{Dec}^{O}_{i},$
(1)

where $\textbf{W}^{cat}_{1}\in\mathbb{R}^{2d\times d}$ and
$\textbf{b}^{cat}_{1}\in\mathbb{R}^{d\times d}$ are parameters of a linear
layer; and $Cat$ stands for concatenating the context vectors on their last
dimension that results in a combined context vector of size $2d$.

Gating mechanism (Gate) – As we see in Fig. 1, for many words (e.g., stop
words) of the utterances, there is no corresponding information in the video
frames. Given the weak and noisy correlation between the transcriptions and
the visual modality, for many of the decoding steps, the visual context does
not provide any discriminative information for the decoder. On the other hand,
for scenarios where the audio modality is distorted, the audio context would
not be reliable too. Therefore, we propose to apply a gating mechanism to the
audio and visual context to enable the model to ignore the noisy information
from these modalities:

$\textbf{G\\_V}^{C}_{i}=\textbf{V}^{C}_{i}\odot\sigma(\textbf{W}^{gate}_{1}Cat(\textbf{V}^{C}_{i},\textbf{A}^{C}_{i},\textbf{Dec}^{O}_{i})+\textbf{b}^{gate}_{1}),\\\
\textbf{G\\_A}^{C}_{i}=\textbf{A}^{C}_{i}\odot\sigma(\textbf{W}^{gate}_{2}Cat(\textbf{V}^{C}_{i},\textbf{A}^{C}_{i},\textbf{Dec}^{O}_{i})+\textbf{b}^{gate}_{2}),\\\
Out_{i}=\textbf{G\\_V}^{C}_{i}+\textbf{G\\_A}^{C}_{i}+\textbf{Dec}^{O}_{i},$
(2)

where $\textbf{W}^{gate}_{1},\textbf{W}^{gate}_{2}\in\mathbb{R}^{3d\times d}$,
$\textbf{b}^{gate}_{1},\textbf{b}^{gate}_{2}\in\mathbb{R}^{d\times d}$ are
learnable parameters of linear layers, $\sigma$ is an element-wise sigmoid
activation and $\odot$ is the element-wise multiplication.

###  3.2  Deliberation model

As shown in Fig. 2, the proposed visual context-aware deliberation model
consists of two steps. In the first step, the audio encoder and the first
decoder process the audio modality to predict the target tokens. We first
train these two components by minimizing the cross-entropy loss between the
outputs of the first decoder and the ground truth tokens. Then, we freeze
these components for the next step when we train the deliberation components.

The encoded audio and the N-best hypotheses from the first decoder are used as
input for the deliberation model. In addition to these two input streams, we
also leverage the visual context at this step. The video frames are mapped to
visual features using the pre-trained video model, then encoded by a separate
video encoder. To process the hypotheses, following [22], for each utterance,
given the N-best hypotheses as $H_{1},H_{2},...,H_{N}$, each hypothesis is
processed separately by the hypothesis encoder, then the encoded hypotheses
are concatenated in time to form an input stream for the second decoder.
Similar to the multi-stream model, three attention heads are used to attend on
audio, visual, and hypotheses encoded streams. We again examine the two
Context-Fusion architectures, however, in this case, we have three context
vectors including visual context, audio context, and hypothesis context. The
resulting context vector from the Context-Fusion is used to predict the target
tokens. In the second step of the training, we train the deliberation
components to minimize the cross-entropy between the second decoder’s outputs
and the ground truth tokens.

The proposed deliberation model has two important advantages compared to the
multi-stream model introduced in section 3.1. First, this model introduces a
flexible way to leverage the visual information on top of an existing audio-
only model that can be trained with a large amount of data. The amount of
audio-visual-text paired data might be much smaller than audio-text paired
data. In such a scenario, given a smaller amount of multi-modal data, we can
leverage a pre-trained audio-only ASR model to produce the N-best hypotheses
and the encoded audio representations, as input to a deliberation model. Then,
we train a visual context-aware deliberation model to learn to leverage the
visual context to refine the N-best hypotheses and generate higher quality
outputs.

Visually Grounded Hypotheses – The second advantage of the deliberation model
is that we also have a text modality input in form of the N-best hypotheses
from the first pass audio-only model. It gives us an opportunity to exploit
the correlation between visual and text representations as shown in Fig 1. For
each hypothesis, which is a sequence of words, $w_{1},w_{2},...,w_{n}$, we
leverage the pre-trained text model to map each word to a vector. Then, for
word $w_{i}$ the visual context $\textbf{c}_{i}$ is computed as a weighted sum
of the visual features $\textbf{v}_{1},\textbf{v}_{2},...,\textbf{v}_{m}$,

$\textbf{c}_{i}=\sum_{j=1}^{m}\alpha_{ij}\textbf{v}_{j},$ (3)

where $\alpha_{ij}$ for each visual vector $v_{j}$ is computed by

$\alpha_{ij}=\frac{\textbf{S}_{ij}}{\sum_{j=1}^{m}\textbf{S}_{ij}},$ (4)

where

$\textbf{S}_{ij}=embedding(w_{i})\cdot\textbf{v}_{j}+2,$ (5)

where $embedding$ stands for the pre-trained text model representation. To
make $\textbf{S}_{ij}>0.0$ for all cases, we added 2 to the result of the dot
product in equation 5. It should be noted that the dot product can effectively
compute the similarity between the word embeddings and the visual features
because the pre-trained text and video models are jointly optimized to
maximize the dot product between text and visual representations of
semantically similar pairs [15].

The resulting visual contexts are appended to the word embeddings before being
passed to the hypothesis encoder. Appending these visual contexts can help the
deliberation model to resolve the ambiguities in the N-best hypothesis. The
mechanism to augment the N-best hypotheses with visual features is illustrated
in Fig 2.

We hypothesize that since audio is the dominant modality for the VC-ASR
problem, and also since the visual information is noisy and has a weak
correlation with the target tokens, the multi-stream model, introduced in
section 3.1, might not be trained well to get the optimum visual context from
the input visual stream. We expect that the proposed deliberation model that
uses the pre-trained text model to directly get the visual context can
leverage the visual information better and consequently perform better.

##  4 Experiments and results

###  4.1 Experimental Setup

Dataset & features – All experiments are conducted on the HOW2 dataset [25]
which is a collection of instructional videos from YouTube. This dataset
provides around 185k (298 hours) audio-visual utterances for training, 2022
utterances (3 hours) for development, and 2305 (4 hours) for evaluation. The
official dataset comes with the audio features and action-based visual
features for the three splits. To follow the same experimental setting used in
the baseline paper [5], we also leverage the provided audio and visual
features. For audio features, 40-dimensional filterbanks are extracted from
time windows of 25ms with a 10ms frame shift. The resulting features are
concatenated with 3-dimensional pitch features to form the final audio
features. For the action-level visual features, a model trained to classify
400 different actions is leveraged [26]. The action-based visual features are
extracted from every 16 frames, then averaged pool into a 2048-dimension per
each video. It should be noted that each video is split into smaller segments
to create the utterances. However, the same video-level feature vector
represents the visual features for all utterances of a video. These visual
features are used for the baseline experiments. However, for the proposed
models, we use the pre-trained text-video model described in section 2 to
extract a sequence of visual features per each video. In our experiments, we
observed that the VC-ASR models trained with video-level visual features were
always superior to ones trained with utterance-level visual-feature.
Therefore, for all experiments with the proposed models, we only report
results trained with the former.

The ground truth transcripts are preprocessed by removing the punctuations and
mapping numbers to their words. Then, a SentencePiece tokenizer [27] model
with 5000 vocabulary size is trained to map each transcript to a sequence of
tokens.

Audio-only model – The audio-only baseline is an attention-based sequence-to-
sequence model [19, 20]. The (audio) encoder consists of 6 encoding layers.
Each encoding layer is composed of a bidirectional gated recurrent units (GRU)
[28] with 512 hidden units for each direction followed by a layer
normalization. The decoder consists of two uni-directional GRU layers with 512
cells in each layer. A location-aware attention mechanism is used between the
encoder and the decoder.

The VC-ASR baseline model – In previous studies, the best VC-ASR performance
for HOW2 data was achieved by applying VAT with the action-level visual
features [5]. To apply VAT, we first pre-train the audio-only model to full
convergence. Starting from this pre-trained model, we add a linear layer to
project the visual features to the audio feature space, then we add the
projected vectors to all audio features. The whole model, including the
projection layer, is tuned until convergence. This model is referred to as
feature-level fusion in the result tables.

Multi-stream model – This model extends the audio-only model by adding a video
encoder that has 3 encoding layers. A single-head location-aware attention
mechanism is also used for the video modality. For this model, we don’t need
to pre-train the audio-only part and the entire model is trained from scratch.

Deliberation model – The model for the first step has the same architecture as
the audio-only model. The first decoder generates 10-best hypotheses which are
padded with <EOS> token to the same length. The hypotheses’ tokens are mapped
to a 512-dimensional vector using the frozen embedding layer of the first
decoder, then encoded by the hypotheses encoder which has 2 encoding layers.
The visual features are also passed through a visual encoder with three
encoding layers. The deliberation decoder consists of two uni-directional GRU
layers with 512 cells in each layer. Attention on each input stream (i.e.,
audio, video, and first-pass hypotheses) is performed using separate and
independent single-head location-aware attention mechanisms [20].

All the models in this study are trained to minimize the cross-entropy loss
between the model outputs and one-hot encoding of the ground-truth tokens.
Label-smoothing regularization [29] with the probability of 0.2 is also
applied. To train the models we use Adam optimizer [30] with initial learning
rate $4.0\times 10^{-4}$. The learning rate gets halved after every 50k
iterations. The final model is chosen by monitoring the model’s performance on
the development set. In the inference time, the decoding is performed with
beam width 10 without using any external language model. The performance of
the models is examined by measuring the word error rate (WER).

Masking Words – We augment the How2 dataset by masking out specific words in
the audio input in order to compare the ability of these models to recover the
masked words. For each utterance, we leverage the pre-trained video and text
models to compute the similarity between words and the visual features (Fig. 1
shows an example of this similarity matrix). Then, words with similarity
$>3.5$ are considered as candidates to be masked out from the audio stream.
For the training data, for each utterance, we randomly sample from the
candidates but limiting the number of masked words to be less than 10% of the
total number of words in the utterance. For the development and test sets,
since we want to fix the masked words, we sort the candidate words based on
their similarity with the visual features and choose the top-10% words. For
each chosen word, we set the masked region in audio features with white noise.

###  4.2 Results

In this section, we compare the performance of the proposed VC-ASR models with
various audio-only and audio-visual baselines. The results for the clean audio
and masked words settings are reported in Table 1-4. The performance of the
models with random misaligned videos is also reported in the tables. For the
masking scenario, we also report recovery rate (RR) of the masked words which
is computed as

$RR=\frac{\\#correctly\;recovered\;masked\;words}{\\#masked\;words\;in\;the\;test\;set\;}\times
100.$ (6)

Baseline models – As shown in Table 1, our audio-only model is performing
better than both the audio-only baseline and the visual context-aware baseline
that was reported in [5]. Since we use the same data and feature extraction as
the baseline, the improvement is mainly attributed to our S2S model’s
architecture. We consider our improved audio-only model as the new baseline
for the VC-ASR experiments. We do not achieve any improvement by applying
feature fusion (VAT) with action-level features on top of our audio-only
model. We hypothesize that applying VAT with action-level features is serving
as a regularizer, as is also shown in [5], and since our audio-only model is
already performing well, this type of regularization can not improve the model
performance.

Table 1:  WERs of the baselines and the multi-stream model for the clean audio scenario. The results are reported in 3 sections as follows. First, the results from the previous studies. Second, our audio-only and the early fusion approach using action-level features. Third, the multi-stream model with just the visual features extracted from the pre-trained text-video model.  ID |  Model |  Visual Features |  Test WER(%) |  Misalgnd Videos WER(%)
---|---|---|---|---
|  Audio-only [5] |  – |  19.1 |  –
|  Feature-Fusion |  Action- |  |
|  (VAT) [5] |  based |  18.0 |  –
A1 |  Our Audio-only |  – |  17.77 |  –
|  Our Feature- |  Action- |  |
|  Fusion (VAT) |  based |  17.95 |  17.93
AV1 |  Multi-Stream |  Text- |  |
|  |  Video |  17.16 |  18.64
Table 2:  WERs and Recovery rates (RR) of audio-only and multi-stream models for the masked words scenario.  ID |  Model |  Context Fusion |  Test WER (%) |  RR(%) |  Misalgnd Videos WER(%)
---|---|---|---|---|---
A2_M |  Audio-only |  – |  22.64 |  22.29 |  –
AV2_M |  Multi-Stream |  Gate |  21.17 |  33.75 |  29.3
|  Multi-Stream |  Cat |  22.4 |  28.0 |  24.56

[capbesideposition=right,top,capbesidewidth=12cm]figure[]

Ref: you need a lot of paint on your brush
Audio-only: you need a lot of pin on your project
VC-ASR: you need a lot of paint on your brush

[capbesideposition=right,top,capbesidewidth=12cm]figure[]

Ref: there is not a whole lot to them they are not that robust they are mostly
feathers
Audio-only: there’s not a whole lot of you them they’re not that room bust
they are mostly forward
VC-ASR: there’s not a whole lot you them they’re not that robust they’re
mostly feathers

[capbesideposition=right,top,capbesidewidth=12cm]figure[]

Ref: the key on this one and something to keep in mind on this position is
this
Audio-only: the key on this one and something to keep in mind on this machine
is this
VC-ASR:
Hyp1: the key on this one and something to keep in mind on this jump is this
Hyp2: the key on this one and something to keep in mind on this position is
this  Fig. 3: Examples of how the audio-only and the multi-stream VC-ASR
models perform for recovering the masked word. The blue words are the masked
words that their corresponding region in the audio is set to white noise, the
green words indicate the correctly recovered words, and the red words indicate
the incorrect recovered words.

Multi-stream models – As is shown in Table 1, for the clean audio data,
leveraging the visual information in the multi-stream architecture improves
the performance compared to the audio-only model (WER is improved from 17.77%
$\rightarrow$ 17.16%). To validate that the improvement is resulting from
leveraging the visual information and not because of any regularization
effects, the performance of the multi-stream model with misaligned videos is
also reported. The results demonstrate that by feeding random misaligned
videos the performance of the multi-stream model drops significantly and even
becomes worse than the audio-only model. This observation confirms the
effectiveness of the multi-stream model in leveraging the visual context. It
should be noted that an ideal VC-ASR model should leverage correlated visual
context and disregard uncorrelated ones without hurting the performance.
However, the multi-stream model did not show this behavior.

For the masked data, as is shown in Table 2, the experiments demonstrate the
superiority of the multi-stream model compared to the audio-only model. The
multi-stream model achieves 6.4% relative improvements in WER and 51.4%
relative improvement in the recovery rate. Here, the gating mechanism performs
considerably better than the concatenation for fusing the context vectors from
different modalities. We think this is because the gating mechanism can
flexibly change the contribution of the audio and visual streams and
compensate for the missing information in the audio stream better than the
concatenation layer.

Table 3:  WERs of the deliberation models for the clean audio scenario. ID |  1st Pass |  Visual Stream |  VG-Hypthss |  Context Fusion |  Test WER(%) |  Misalgnd Videos WER(%)
---|---|---|---|---|---|---
DB |  A1 |  ✗ |  ✗ |  Cat |  17.58 |  –
D1 |  A1 |  ✓ |  ✗ |  Cat |  17.26 |  17.7
D2 |  A1 |  ✓ |  ✓ |  Cat |  17.07 |  17.87
D3 |  A1 |  ✓ |  ✗ |  Gate |  17.33 |  17.83
D4 |  A1 |  ✓ |  ✓ |  Gate |  17.23 |  17.91
D5 |  AV1 |  ✓ |  ✗ |  Cat |  16.98 |  18.65
D6 |  AV1 |  ✓ |  ✓ |  Cat |  16.7 |  18.68
Table 4:  WERs and Recovery rates (RR) of the deliberation models for the masked words scenario.  1st Pass |  Visual Stream |  VG-Hypthss |  Context Fusion |  Test WER (%) |  RR(%) |  Misalgnd Videos WER(%)
---|---|---|---|---|---|---
A2_M |  ✓ |  ✗ |  Cat |  22.33 |  25 |  25.4
A2_M |  ✓ |  ✗ |  Gate |  21.06 |  34 |  23.19
A2_M |  ✓ |  ✓ |  Gate |  20.94 |  33.5 |  23.12
AV2_M |  ✓ |  ✓ |  Gate |  20.65 |  35.5 |  24.11

In Fig. 3 we consider a couple of examples of the test data where the multi-
stream model performs better than the audio-only model. In the first two
examples, the audio-only model is relying on the decoder’s implicit language
model to predict the masked word but it is unable to successfully recover
them. However, the multi-stream model correctly recovers the masked words by
leveraging the visual context. In the third example, the multi-stream model is
not able to recover the masked in the first hypothesis of the beam search,
however, correctly recovers it in the second hypothesis. An interesting
observation in this example, which is also observed in many other utterances,
is that the model substituted the target word “position” with the word “jump”
which also has a high correlation with the visual context. It looks like the
multi-stream model substitutes the masked words with words that are correlated
with the visual context and have a high probability in the implicit language
model. We validate this assumption by examining the quality of the N-best
hypotheses which is measured by computing the best WER among the hypotheses
(oracle WER). The oracle WER of the audio-only and multi-stream models are
18.9% and 16.05%, respectively. This means that the multi-stream model can
recover many of the masked words in the N-best hypotheses. We will leverage
this advantage in deliberation models.

Deliberation models – As shown in Table 3, for the clean data, the
deliberation model outperforms the audio-only model and the multi-stream model
(D2 vs A1 & AV1). However, to validate the contribution of the visual stream,
we also examine a deliberation model without the visual stream (DB). The
results confirm that the improvement is mainly attributed to leveraging the
visual context rather than the deliberation process itself (D2 vs DB). For the
masked data (Table 4), the deliberation model performs slightly better than
the multi-stream model (21.17 $\rightarrow$ 21.06). For Context-Fusion, the
gating mechanism performs better for masked data and the concatenation
performs slightly better for the clean data.

The results show that the deliberation model can disregard uncorrelated visual
context better than the multi-stream model. For the misaligned videos, the
deliberation model performs almost the same as the audio-only model. One
possible explanation is that since the deliberation model processes the
hypotheses generated by the audio-only model, it learns to attend more on the
hypotheses as a reliable source, and leverage the visual context to resolve
the ambiguities and correct the errors that exist in the hypotheses.
Therefore, if the visual context does not bring any additional information
this model would be able to perform close to the audio-only model.

Visually Grounded Hypotheses – The experiments confirm that the VG-Hypotheses
approach brings additional benefits in both the clean and masking scenarios.
However, since VG-Hypotheses helps to resolve ambiguities among the
candidates, this approach performs better if the ground truth words are
predicted in the N-best list. Therefore, a model with lower oracle WER would
be a better candidate for the first pass of the deliberation model. The multi-
stream models are performing significantly better than the audio-only models
in terms of oracle WER. For the clean data, the oracle WER for the audio-only
model and the multi-stream models are 14.37% and 12.48%, respectively, and for
the masked data, the results are 18.9% and 16.05%, respectively. Therefore, we
use the best multi-stream models, AV1 and AV2-M for clean and masked data,
respectively, as the first pass. The results of these deliberation models are
reported in the last rows of Table 3 & 4.

The experiments support our hypothesis and we achieve the best performance for
the clean and masked data using the deliberation with VG-Hypotheses on top of
multi-stream models. For the clean data, we achieve almost 6% relative
improvement, and for the masked data, we achieve 8.7% relative improvement
compared to the audio-only models. The recovery rate for masked words is also
improved by 59% relatively compared to the audio-only model.

##  5 Conclusion

In this paper, we explored different approaches to leverage a pre-trained
text-video representation for an improved visual context-aware ASR model. We
proposed multi-stream and deliberation models to leverage the visual features
extracted by the pre-trained video model. For the deliberation model, since we
also have the text modality as input in form of N-best hypotheses from the
first pass, we leveraged the pre-trained text model to attend on the visual
features directly. Experimental evaluation demonstrates the efficacy of the
multi-stream and the deliberation models. In addition to a clean audio stream
scenario, we also examined how the VC-ASR models perform when the audio stream
is distorted. For this purpose, we augment the data by masking out specific
words in the audio stream. The results show that the deliberation model
performs better than the multi-stream model and achieves a relative WER
improvement of 6% and 8.7% for the clean and masked data, respectively,
compared to an audio-only model. The recovery rate of the masked words is also
improved by 59% relatively compared to the audio-only model.

## References

  * [1] Tejas Srinivasan, Ramon Sanabria, and Florian Metze,  “Looking enhances listening: Recovering missing speech using images,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6304–6308.
  * [2] Shruti Palaskar, Ramon Sanabria, and Florian Metze,  “End-to-end multimodal speech recognition,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5774–5778.
  * [3] Abhinav Gupta, Yajie Miao, Leonardo Neves, and Florian Metze,  “Visual features for context-aware speech recognition,”  in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, pp. 5020–5024.
  * [4] Tejas Srinivasan, Ramon Sanabria, and Florian Metze,  “Analyzing utility of visual context in multimodal speech recognition under noisy conditions,”  arXiv preprint arXiv:1907.00477, 2019.
  * [5] Ozan Caglayan, Ramon Sanabria, Shruti Palaskar, Loic Barraul, and Florian Metze,  “Multimodal grounding for sequence-to-sequence speech recognition,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 8648–8652.
  * [6] Yasufumi Moriya and Gareth JF Jones,  “Lstm language model adaptation with images and titles for multimedia automatic speech recognition,”  in 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018, pp. 219–226.
  * [7] Ozan Caglayan, Pranava Swaroop Madhyastha, Lucia Specia, and Loïc Barrault,  “Probing the need for visual context in multimodal machine translation,”  in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4159–4170.
  * [8] Chiori Hori, Huda Alamri, Jue Wang, Gordon Wichern, Takaaki Hori, Anoop Cherian, Tim K Marks, Vincent Cartillier, Raphael Gontijo Lopes, Abhishek Das, et al.,  “End-to-end audio visual scene-aware dialog using multimodal attention-based video features,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 2352–2356.
  * [9] Shachi H Kumar, Eda Okur, Saurav Sahay, Jonathan Huang, and Lama Nachman,  “Investigating topics, audio representations and attention for multimodal scene-aware dialog,”  Computer Speech & Language, p. 101102, 2020.
  * [10] Jindrich Libovickỳ, Shruti Palaskar, Spandana Gella, and Florian Metze,  “Multimodal abstractive summarization of open-domain videos,”  in Proceedings of the Workshop on Visually Grounded Interaction and Language (ViGIL). NIPS, 2018.
  * [11] Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman,  “Deep audio-visual speech recognition,”  IEEE transactions on pattern analysis and machine intelligence, 2018\.
  * [12] Yannis M Assael, Brendan Shillingford, Shimon Whiteson, and Nando De Freitas,  “Lipnet: End-to-end sentence-level lipreading,”  arXiv preprint arXiv:1611.01599, 2016.
  * [13] Yajie Miao and Florian Metze,  “Open-domain audio-visual speech recognition: A deep learning approach.,”  in Interspeech, 2016, pp. 3414–3418.
  * [14] Desmond Elliott,  “Adversarial evaluation of multimodal machine translation,”  in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 2974–2978.
  * [15] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman,  “End-to-end learning of visual representations from uncurated instructional videos,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9879–9889.
  * [16] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu,  “Hero: Hierarchical encoder for video+ language omni-representation pre-training,”  arXiv preprint arXiv:2005.00200, 2020.
  * [17] David Harwath, Adria Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, and James Glass,  “Jointly discovering visual objects and spoken words from raw sensory input,”  in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 649–665.
  * [18] Herman Kamper, Gregory Shakhnarovich, and Karen Livescu,  “Semantic speech retrieval with a visually grounded model of untranscribed speech,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 2514–2517.
  * [19] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals,  “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,”  in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016, pp. 4960–4964.
  * [20] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio,  “Neural machine translation by jointly learning to align and translate,”  in 3rd International Conference on Learning Representations, ICLR 2015, 2015.
  * [21] Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu,  “Deliberation networks: Sequence generation beyond one-pass decoding,”  in Advances in Neural Information Processing Systems, 2017, pp. 1784–1794.
  * [22] Ke Hu, Tara N Sainath, Ruoming Pang, and Rohit Prabhavalkar,  “Deliberation model based two-pass end-to-end speech recognition,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7799–7803.
  * [23] Tara N Sainath, Ruoming Pang, David Rybach, Yanzhang He, Rohit Prabhavalkar, Wei Li, Mirkó Visontai, Qiao Liang, Trevor Strohman, Yonghui Wu, et al.,  “Two-pass end-to-end speech recognition,”  Proc. Interspeech 2019, pp. 2773–2777, 2019.
  * [24] Tzu-Wei Sung, Jun-You Liu, Hung-yi Lee, and Lin-shan Lee,  “Towards end-to-end speech-to-text translation with two-pass decoding,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 7175–7179.
  * [25] Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loïc Barrault, Lucia Specia, and Florian Metze,  “How2: A large-scale dataset for multimodal language understanding,”  in NeurIPS, 2018.
  * [26] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh,  “Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?,”  in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2018, pp. 6546–6555.
  * [27] Taku Kudo and John Richardson,  “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,”  in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2018, pp. 66–71.
  * [28] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio,  “Empirical evaluation of gated recurrent neural networks on sequence modeling,”  in NIPS 2014 Workshop on Deep Learning, December 2014, 2014.
  * [29] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna,  “Rethinking the inception architecture for computer vision,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826.
  * [30] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  in 3rd International Conference on Learning Representations, ICLR 2015, 2015.


# CycleGAN-based Non-parallel Speech Enhancement with an Adaptive Attention-
in-attention Mechanism

Guochen Yu12, Yutian Wang1, Chengshi Zheng2, Hui Wang1, and Qin Zhang1 1 State
Key Laboratory of Media Convergence and Communication, Communication
University of China, Beijing, China
E-mail: {yuguochen, wangyutian, hwang<EMAIL_ADDRESS>2 Key Laboratory
of Noise and Vibration Research, Institute of Acoustics, Chinese Academy of
Sciences, Beijing, China
E-mail<EMAIL_ADDRESS>
###### Abstract

Non-parallel training is a difficult but essential task for DNN-based speech
enhancement methods, for the lack of adequate noisy and paired clean speech
corpus in many real scenarios. In this paper, we propose a novel adaptive
attention-in-attention CycleGAN (AIA-CycleGAN) for non-parallel speech
enhancement. In previous CycleGAN-based non-parallel speech enhancement
methods, the limited mapping ability of the generator may cause performance
degradation and insufficient feature learning. To alleviate this degradation,
we propose an integration of adaptive time-frequency attention (ATFA) and
adaptive hierarchical attention (AHA) to form an attention-in-attention (AIA)
module for more flexible feature learning during the mapping procedure. More
specifically, ATFA can capture the long-range temporal-spectral contextual
information for more effective feature representations, while AHA can flexibly
aggregate different AFTA’s intermediate output feature maps by adaptive
attention weights depending on the global context. Numerous experimental
results demonstrate that the proposed approach achieves consistently more
superior performance over previous GAN-based and CycleGAN-based methods in
non-parallel training. Moreover, experiments in parallel training verify that
the proposed AIA-CycleGAN also outperforms most advanced GAN-based and Non-GAN
based speech enhancement approaches, especially in maintaining speech
integrity and reducing speech distortion.

##  1 Introduction

Speech enhancement (SE) aims to recover clean speech components from the
noise-corrupted mixture, so as to improve speech quality and intelligibility.
It has become a fundamental technique in many communication applications such
as the front-ends for automatic speech recognition (ASR) systems and hearing
assistant devices [1]. Due to the unprecedented development of deep neural
networks (DNNs), many DNN-based SE approaches have demonstrated better
performance over traditional signal-processing-based approaches [2]. These
DNN-based approaches can be divided into two categories, namely masking-based
approaches [3, 4, 5] and mapping-based approaches [6, 7, 8]. Recently,
Generative Adversarial Networks (GANs) have shown their promising performance
in the SE area for its powerful capability of mapping the target output
distribution from the original input distribution [9, 10, 11, 12], in which a
generator ($G$) tries to conduct the enhancement process and a discriminator
($D$) tries to distinguish between real inputs and fake outputs generated by
this generator.

Figure 1: Training procedure of the proposed method. Forward noisy-clean-noisy
cycle and backward clean-noisy-clean cycle are illustrated in the left and
right parts, respectively.

In the standard formulation of supervised speech enhancement methods, the
mapping functions are trained to minimize the loss between the output features
of the enhanced speech and the features of the corresponding clean speech.
Therefore, they always need a large number of paired clean-noisy samples to
conduct supervised training and improve the generalization of the whole
network. However, there exist many practical scenarios in which it is
difficult or impossible to obtain parallel recordings of clean-noisy pairs,
and sometimes we can only acquire clean data that mismatches the source noisy
data. To resolve this problem, CycleGAN was adopted for both standard parallel
and non-parallel training in the SE area [14, 15, 16], which was originally
proposed for unpaired image-to-image translation [17] and began to thrive in
speech applications in recent years [18]. Nonetheless, these previous non-
parallel CycleGAN-based SE methods with unpaired data can hardly achieve
competitive performance when compared with the standard parallel training,
because of the limited mapping ability of the generator.

In this paper, a novel CycleGAN-based system is proposed with an adaptive
attention-in-attention mechanism to cope with non-parallel speech enhancement.
Specifically, two generators (dubbed $G_{X\rightarrow Y}$ and $F_{Y\rightarrow
X}$) and two discriminators (dubbed $D_{X}$ and $D_{Y}$) are jointly trained
with relativistic adversarial losses, cycle-consistency losses and an identity
mapping loss. To improve the mapping ability of the generators, we propose a
novel attention mechanism dubbed attention-in-attention (AIA) in generators
for more powerful feature fusion and feature correlation learning. This AIA
consists of adaptive time-frequency attention (ATFA) and adaptive hierarchical
attention (AHA). Specifically, ATFA aims at capturing the long-range temporal-
spectral contextual dependency in parallel, while AHA aims to flexibly
aggregate all the output feature maps of ATFA together by the hierarchical
attention weights depending on the global context. For discriminators, multi-
scale discriminators are adopted to force the generator to pay more attention
to finer details. Besides, considering the effectiveness of the power
compression in the dereverberation and denoising task [19, 20], the magnitude
of the spectrum is compressed as the input features to better attenuate the
background noise.

The remainder of the paper is organized as follows. In Section 2, the proposed
framework is described in detail. The experimental setup is presented in
Section 3, while experimental results are provided and discussed in Section 4.
Finally, conclusions are drawn in Section 5.

##  2 Proposed Scheme

###  2.1 Problem Formulation

In the speech enhancement task, when taking the short-time Fourier transform
(STFT) to the mixture signal, in the time-frequency (T-F) domain, we can have:

${X_{t,f}}=Y_{t,f}+Z_{t,f},$ (1)

where $X_{t,f}=\left|X_{t,f}\right|e^{j\theta_{X_{t,f}}}\in\mathbb{C}$,
$Y_{t,f}=\left|Y_{t,f}\right|e^{j\theta_{Y_{t,f}}}\in\mathbb{C}$ and
$Z_{t,f}=\left|Z_{t,f}\right|e^{j\theta_{Z_{t,f}}}\in\mathbb{C}$ denote the
T-F representations of the mixture, clean speech and noise components in the
time index of $t$ and frequency index of $f$, respectively. Most recently,
using the power-compressed spectra as input features dramatically improves
speech quality in the dereverberation and denoising task [19, 20], so we
conduct the power compression on the spectral magnitude before feeding into
the mapping network. The compression coefficient is set to 0.5, which is
reported an optimal choice in [20]. Therefore, the enhanced spectral magnitude
can be expressed as:

$\begin{gathered}\left|\tilde{X}_{t,f}\right|^{\eta}=G_{X\rightarrow
Y}(\left|{X}_{t,f}\right|^{\eta};\phi_{G}),\end{gathered}$ (2)

where $\eta=0.5$; $G_{X\rightarrow Y}$ denotes the mapping function of the
generator $G$ and $\phi_{G}$ denotes its parameter set.

Figure 2:  The framework of the proposed generators. IN, GLU, and PRelu
indicate instance normalization, gated linear unit, and parametric Relu
activation, respectively.

###  2.2 Network architecture

In our CycleGAN-based SE system, a forward noisy-to-clean generator $G$ is
first employed to enhance the noisy features to the clean ones, while an
inverse clean-to-noisy generator $F$ is applied to convert the enhanced
features back to the original domain. As illustrated in Fig. 1, a forward-
inverse noisy-clean-noisy cycle and an inverse-forward clean-noisy-clean cycle
jointly constrain $G$ and $F$ to conduct non-parallel mapping. Discriminators
$D_{X}$ and $D_{Y}$ are trained to classify the target speech features as real
and the generated speech features as fake. As shown in Fig. 2, the generator
is composed of three components, including three downsampling layers, an
adaptive attention-in-attention (AIA) module and three homologous upsampling
layers. Each downsampling/upsampling layer block is composed of a 2D
convolution/deconvolution layer, followed by instance normalization (IN),
Parametric Relu activation function (PRelu) and gated liner units (GLUs) [21].
The proposed AIA consists of six ATFA modules and an AHA module, where ATFA is
proposed to capture the long-range dependencies along temporal-spectral
dimensions with low computational cost and AHA is introduced to aggregate
different intermediate features to capture the long-term hierarchical
contextual information by the adaptive weights depending on the global
context.

The discriminator is composed of six 2D convolutions, each of which is
followed by spectral normalization (SN) and PRelu, so as to compress the
feature maps into a high-level representation. SN can stabilize the training
process of the discriminator and avoid vanishing or exploding gradients [22].
Note that we set the configuration of the utilized discriminators the same as
our previous study [23]. Inspired by recent studies on image enhancement [24],
we propose to apply a multi-scale discriminator that uses the intermediate
layer of the discriminator with a smaller receptive field, which can force the
generator to produce speech features with global consistency and finer
details.

Figure 3: The diagram of adaptive time-frequency attention (ATFA) modules.
$\bigodot$ and $\bigotimes$ denote the matrix multiplication and element-wise
multiplication, respectively.

###  2.3 Adaptive Time-Frequency Attention

Attention mechanism [25] has been widely used in speech processing tasks for
its capability of leveraging the contextual information in the feature maps.
Following the terminology in [26], we compute the attention function on the
output feature maps $F_{in}\in\mathbb{R}^{B\times T\times
F^{{}^{\prime}}\times C^{\prime}}$ of the downsampling layers. Here, $B$
denotes the batch size of input features, $T$ denotes the number of frames,
$F^{{}^{\prime}}$ denotes the number of frequency bins and $C^{\prime}$
denotes the number of channels in each feature map. To alleviate the heavy
computational complexity of conventional self-attention, we introduce an
adaptive time-frequency attention (ATFA) mechanism as a lightweight solution
to capture the long-range correlations exhibited in T-F spectrogram as in
[27]. As illustrated in Fig. 3, the proposed ATFA consists of two branches: an
adaptive time attention branch (ATAB) and an adaptive frequency attention
branch (AFAB). The two branches cooperate to capture the global dependencies
along temporal and spectral dimensions in parallel. By factorizing the
original attention into time-dimension and frequency-dimension, we can reduce
the large attention weight matrix to two much smaller ones, i.e., $(T\times
T)$ and $(F^{{}^{\prime}}\times F^{{}^{\prime}})$. Along the time path, we
reshape the input feature features into $C^{t}F^{\prime}$ vectors with
dimension $1\times T$ to calculate temporal self-attention, which can be
calculated as:

$\begin{gathered}Q^{t}=Conv^{t}_{Q}(F_{in}),K^{t}=Conv^{t}_{K}(F_{in}),V^{t}=Conv^{t}_{V}(F_{in}),\\\
Q^{t}_{Res},K^{t}_{Res},V^{t}_{Res}=Reshape^{t}(Q^{t},K^{t},V^{t}),\\\
\sigma^{t}=softmax((Q^{t}_{Res})\cdot(K^{t}_{Res})^{T}),\\\
Out_{ATAB}=Reshape^{t^{\prime}}(\sigma^{t}\cdot V^{t}_{Res}),\\\
\end{gathered}$ (3)

where $Q^{t}_{Res},K^{t}_{Res},V^{t}_{Res}\in R^{B\times T\times(C^{t}\times
F^{\prime})}$, $C^{t}=\left\\{{\frac{C}{8},\frac{C}{8},C}\right\\}$,
$Out_{ATAB}\in\mathbb{R}^{B\times T\times F^{\prime}\times C}$, and $C=64$.
Analogously, we reshape the input into $C^{f}T$ vectors with dimension
$1\times F^{\prime}$ to calculate the adaptive attention $Out_{AFAB}$ along
the frequency axis in parallel. Finally, The output features of these two
branches and the original features are then combined together by two adaptive
weights to generate the final output of ATFA module, which can be formulated
as:

$\begin{gathered}Out_{ATFA}=F_{in}+\alpha Out_{ATAB}+\beta
Out_{AFAB}\end{gathered}$ (4)

where $F_{in}$, $Out_{ATAB}$ and $Out_{AFAB}$ represent the original input
feature map given by the last downsampling layer, the output of ATAB and the
output of AFAB, respectively. Here, $\alpha$ and $\beta$ are initialized as 0
and gradually lean to assign a larger weight. In summary, each branch has the
following steps: (1) Reshape the input features; (2) Extract the long-range
contextual dependencies along time and frequency axes, respectively; (3)
Perform feature fusion along different dimensions with adaptive weights.

Figure 4: The diagram of the adaptive hierarchical attention (AHA) module.
$ⓢ$, $\bigodot$ and $\bigotimes$ denote the softmax function, matrix
multiplication and element-wise multiplication, respectively.

###  2.4 Adaptive Hierarchical Attention

As shown in Fig. 4, we introduce an adaptive hierarchical attention (AHA)
module to integrate different hierarchical feature maps given a set of ATFA
modules’ outputs
$F=\left\\{F_{n}\right\\}^{N}_{n=1},F_{n}\in\mathbb{R}^{T\times
F^{\prime}\times C}$, where $N$ is the number of ATFA blocks and set to be 6.
Specifically, we first employ an average pooling layer $Pool_{Avg}$ and a
$1\times 1$ convolutional layer $W_{n}$ to squeeze the output feature map of
each ATFA modules into a global representation:
$P_{n}^{h}=Pool_{Avg}(F_{n})\ast W_{n}\in R^{1\times 1\times 1}$, and then we
concatenate all the outputs as $P^{h}\in\mathbb{R}^{1\times 1\times N\times
1}$, which is then fed into the Softmax function to obtain the hierarchical
attention map $W^{h}\in\mathbb{R}^{1\times 1\times N\times 1}$. After that we
cascade all inputs $F=\left\\{F_{n}\right\\}^{N}_{n=1}$ to obtain a global
feature map $F^{h}\in\mathbb{R}^{T\times F^{\prime}\times C\times N}$.
Subsequently, we incorporate the global contextual information by performing a
matrix multiplication between $F^{h}\in\mathbb{R}^{T\times F^{\prime}\times
C\times N}$ and the hierarchical attention weights $W_{N}^{h}$, which can be
defined as:

$\begin{gathered}G_{N}=\sum_{i=1}^{N}W^{h}_{i}F_{i}^{h}\end{gathered}$ (5)

where $G_{N}\in\mathbb{R}^{T\times F^{\prime}\times C}$ denotes the global
contextual feature maps. Finally, we perform an element-wise sum operation
between the output feature map $F_{N}$ the last ATFA module and the global
contextual feature map $G_{N}$ to obtain the final
$Out_{AHA}\in\mathbb{R}^{T\times F^{\prime}\times C}$:

$\begin{gathered}Out_{AHA}=F_{N}+\gamma\sum_{i=1}^{N}W^{h}_{i}F_{i}^{h}\\\
=F_{N}+\gamma\sum_{i=1}^{N}\frac{\exp(Pool_{Avg}(F_{i})\ast
W_{i})}{\sum_{n=1}^{N}\exp(Pool_{Avg}(F_{n})\ast W_{n})}F_{i}.\end{gathered}$
(6)

Note that $\gamma$ is a learnable scalar coefficient and initialized as 0.
This adaptive learning weight gradually learns to assign larger weight to
merge global contextual information effectively. In a nutshell, $Out_{AHA}$ is
a weighted sum of all ATFA modules’ outputs, thus helping to fuse the global
context of all intermediate feature maps with different weights and
progressively guide the enhancement procedure.

###  2.5 Loss function

To ensure the effective mapping in non-parallel training, we use the following
losses, namely relativistic adversarial losses, cycle-consistency losses, and
an identity mapping loss, to jointly optimize the proposed model.

Relativistic adversarial loss: For the noisy-to-clean mapping procedure, we
employ the relativistic average least-square (RaLS) adversarial loss [28] to
make the enhanced compressed spectral magnitude $G_{X\rightarrow
Y}(\left|{X}_{t,f}\right|^{0.5})$ appear to the target clean ones
$\left|{Y}_{t,f}\right|^{0.5}$, which can be expressed as below:

$\begin{gathered}\mathcal{L}_{RA}(D_{Y})=\mathbb{E}_{y}\left[(D_{Y}(y)-\mathbb{E}_{x}D_{Y}(G_{X\rightarrow
Y}(x))-1)^{2}\right]+\\\ \mathbb{E}_{x}\left[(D_{Y}(G_{X\rightarrow
Y}(x))-\mathbb{E}_{y}(D_{Y}(y))+1)^{2}\right]\end{gathered}$ (7)
$\begin{gathered}\mathcal{L}_{RA}(G_{X\rightarrow
Y})=\mathbb{E}_{x}\left[(D_{Y}(G_{X\rightarrow
Y}(x)))-\mathbb{E}_{y}D_{Y}(y)-1)^{2}\right]\\\
+\mathbb{E}_{y}\left[(D_{Y}(y)-\mathbb{E}_{x}(D_{Y}(G_{X\rightarrow
Y}(x)))+1)^{2}\right]\end{gathered}$ (8)

where $x$ and $y$ are the compressed magnitudes of noisy and clean spectrum
(i.e. $\left|{X}_{t,f}\right|^{0.5}$ and $\left|{Y}_{t,f}\right|^{0.5}$),
respectively. Here, the generator $G_{X\rightarrow Y}$ tries to synthesize the
enhanced spectral magnitude that can deceive the discriminator $D_{Y}$ by
minimizing $\mathcal{L}_{RA}(G_{X\rightarrow Y})$, whereas the discriminator
$D_{Y}$ attempts to distinguish the generated spectral magnitude from the
clean one $y$ by minimizing $\mathcal{L}_{RA}(D_{Y})$. Analogously, the
inverse clean-to-noisy generator $F_{Y\rightarrow X}$ and its corresponding
discriminator $D_{X}$ are optimized using $\mathcal{L}_{RA}(D_{X})$ and
$\mathcal{L}_{RA}(F_{Y\rightarrow X})$.

Cycle-consistency loss: Without parallel supervision, generators may map
source feature space to any random permutation of the target space within only
adversarial losses. To constrain the non-parallel mapping, a cycle-consistency
loss is utilized to bring the output back to original input data. The cycle-
consistency loss $\mathcal{L}_{cycle}(G_{X\rightarrow Y},F_{Y\rightarrow X})$
can help two generators $G_{X\rightarrow Y}$ and $F_{Y\rightarrow X}$ to
identity the pseudo pair $(x,y)$ without paired data as follows:

$\begin{gathered}\mathcal{L}_{cycle}(G_{X\rightarrow Y},F_{Y\rightarrow
X})=\mathbb{E}_{x}\left[\left\|F_{Y\rightarrow X}(G_{X\rightarrow
Y}(x))-x\right\|_{1}\right]\\\ +\mathbb{E}_{y}\left[\left\|G_{X\rightarrow
Y}(F_{Y\rightarrow X}(y))-y\right\|_{1}\right]\end{gathered}$ (9)

where $\left\|\cdot\right\|_{1}$ indicates the $L1$ Norm.

Identity-mapping loss: Since the generator should not modify the compositions,
such as linguistic information of the target speech feature when it is fed
into the generator as the input [16], too much, we regularize generators $G$
and $F$ to be as close as possible to the identity mapping by minimizing an
identity-mapping loss [17], which can be given by:

$\begin{gathered}\mathcal{L}_{id}(G_{X\rightarrow Y},F_{Y\rightarrow
X})=\mathbb{E}_{x}\left[\left\|F_{Y\rightarrow X}(x)-x\right\|_{1}\right]+\\\
\mathbb{E}_{y}\left[\left\|G_{X\rightarrow
Y}(y)-y\right\|_{1}\right]\end{gathered}$ (10)

where the magnitudes of the target spectrum (i.e., $y$ and $x$) are provided
as the inputs of the generators (i.e., $G_{X\rightarrow Y}$ and
$F_{Y\rightarrow X}$), respectively. In summary, the overall loss can be
summarized as follows:

$\begin{gathered}\mathcal{L}_{Full}=\mathcal{L}_{RA}(G_{X\rightarrow
Y},D_{Y})+\mathcal{L}_{RA}(F_{Y\rightarrow X},D_{X})\\\
+\lambda_{cycle}\mathcal{L}_{cycle}(G_{X\rightarrow Y},F_{Y\rightarrow
X})+\lambda_{id}\mathcal{L}_{id}(G_{X\rightarrow Y},F_{Y\rightarrow
X})\end{gathered}$ (11)

where $\lambda_{cycle}$ and $\lambda_{id}$ are tunable hyper-parameters, which
are set to be 5 and 10, respectively.

##  3 Experiments

###  3.1 Datasets

The dataset used in this work is publicly available as proposed in [29], which
is a selection of the Voice Bank corpus [30] with 28 speakers for training and
another 2 unseen speakers for the test. The training set consists of 11572
mono audio samples, while the test set contains 2 speakers’ (one male and one
female) 824 utterances. For the training set, audio samples are mixed together
with one of the 10 noise types, i.e., two artificial (babble and speech
shaped) and eight real noise from the DEMAND database [31], at four SNRs of 0,
5, 10 and 15 dB. The test utterances are created with 5 unseen test-noise
types (all from the DEMAND database) at SNRs of 2.5, 7.5, 12.5 and 17.5 dB.
The original raw waveforms are downsampled from 48kHz to 16kHz beforehand.

Table 1: Ablation study for Normal and Compressed magnitude under non-parallel
training.

Models | PESQ | SSNR | STOI | DNSMOS
---|---|---|---|---
Unprocessed | 1.97 | 1.68 | 0.921 | 3.02
Normal magnitude
CycleGAN (baseline) | 2.47 | 5.69 | 0.924 | 3.36
CycleGAN+ATAB | 2.53 | 6.28 | 0.929 | 3.40
CycleGAN+AFAB | 2.54 | 6.42 | 0.928 | 3.39
CycleGAN+ATFA | 2.59 | 6.65 | 0.932 | 3.41
AIA-CycleGAN | 2.61 | 6.69 | 0.931 | 3.43
Compressed magnitude (parameter $\eta=0.5$)
CycleGAN (baseline) | 2.56 | 6.21 | 0.926 | 3.38
CycleGAN+ATAB | 2.59 | 6.67 | 0.928 | 3.41
CycleGAN+AFAB | 2.61 | 6.61 | 0.931 | 3.43
CycleGAN+ATFA | 2.64 | 6.94 | 0.930 | 3.45
AIA-CycleGAN | 2.67 | 7.23 | 0.932 | 3.47

###  3.2 Implementation Setup

The Hanning window of length 32ms is utilized, with 75% overlap between
adjacent frames. The 512-point STFT is utilized and the 257-dimension
compressed spectral magnitude is used as the input feature. For the non-
parallel training strategy, we randomly crop a fixed-length segment (i.e., 108
frames) from a randomly selected noisy audio file as the input, while the
target is a randomly selected clean audio file that is different from the
input audio. That is to say, the input totally mismatches the target speech
features. We adopt the Adam optimizer [32] with the momentum term
$\beta_{1}=0.9$, $\beta_{2}=0.999$ and train the networks with an initial
learning rate of 0.0001 for discriminators and 0.0002 for generators,
respectively. The same learning rates are maintained for the first 50 epochs,
while they linearly decay in the remaining iterations. We set the batch size
to 4 and use $\mathcal{L}_{id}$ only for the first 20 epochs.

##  4 Results and Analysis

We use the following objective metrics to evaluate speech enhancement
performance: the perceptual evaluation of speech quality (PESQ) [33], short-
time objective intelligibility (STOI) [34], segmental signal-to-noise ratio
(SSNR), the mean opinion score (MOS) prediction of the speech signal
distortion (CSIG) [35], the MOS prediction of the intrusiveness of background
noise (CBAK) and the MOS prediction of the overall effect (COVL) [35]. In
addition, we evaluate the subjective quality by DNSMOS [36], which is a robust
non-intrusive perceptual speech quality metric designed to evaluate noise
suppressors. Higher values of all metrics indicate better performance.

###  4.1 Ablation study

We first investigate the effectiveness of the proposed attention modules and
the power compression. As shown in Table 1, we set the CycleGAN without
proposed attention modules as the baseline, which is also trained with the
relativistic average least-square loss. From the results, we can have the
following observations. Firstly, compared with non-compressed methods,
CycleGAN-based approaches fed with the compressed spectral magnitude achieve
better performance, indicating that the power compression facilitates more
accurate spectrum recovery. For example, compressed-magnitude CycleGAN
achieves average 0.09 PESQ and 0.52dB SSNR improvements over the normal-
magnitude CycleGAN. The possible rationale is that, when the compression
operation is applied, the gap between the magnitude of the speech and noise
components is narrowed, that is to say, the residual noise components in the
weak energy regions (i.e., middle and high-frequency regions) are given more
priority during the enhancement produce [19]. Secondly, by adding the proposed
attention branches, CycleGAN+ATAB improves the average PESQ and STOI by 0.03
and 0.002 over the compressed-magnitude baseline, while CycleGAN + AFAB
improves the average PESQ and STOI by 0.05 and 0.004. This indicates that the
ATAB and AFAB can effectively guide the feature learning procedure of the
generators. By integrating ATAB and ATFB as the ATFA modules, we also observe
considerable improvements on PESQ, SSNR and DNSMOS scores. Finally, by
integrating the AHA module and ATFA modules as an AIA module, we can see that
the proposed AIA-CycleGAN significantly outperforms other comparisons, which
verifies the effectiveness of the proposed AIA module in improving the speech
quality.

###  4.2 Comparison under non-parallel and parallel training

To investigate the effectiveness of our proposed method with both the parallel
and non-parallel training strategy, we compare our proposed method with the
reference methods including conventional GANs and CycleGANs. Here, ”GAN-
normal” is fed with the normal magnitude as the input features while ”GAN-
compressed” is fed with the compressed magnitude. From Table 2, we can observe
the following two phenomena. Firstly, GAN-based methods yield similar
performance with CycleGAN in the parallel training, whereas CycleGAN
outperforms GAN-based methods by a large margin in the non-parallel training.
This indicates the cycle consistency constraint can prevent the mapping
ability of the generators from sharply degrading under unpaired data. Besides,
by employing the AIA module, the proposed approach under unpaired data
achieves similar and competitive performance compared with parallel training,
and significantly outperforms all the baselines. For example, AIA-CycleGAN
surpasses GAN-compressed by a large margin in PESQ, CSIG, CBAK, COVL and
DNSMOS using non-parallel training, which is 0.64, 0.38, 0.46, 0.54 and 0.79,
respectively.

Fig. 5 shows the spectrograms of the noisy/clean utterances and the utterances
enhanced by GAN-compressed, CycleGAN and AIA-CycleGAN in the non-parallel
training. This figure demonstrates that CycleGANs dramatically surpass the
conventional GAN-based method. Moreover, we observe that AIA-CycleGAN can
effectively surpass the original CycleGAN in terms of noise suppression. For
example, as shown in the red sign area and green sign area of Fig. 5 (c) and
(d), AIA-CycleGAN shows a more powerful capability of suppressing residual
noise components.

Table 2: Experimental results among different models under non-parallel and
parallel training. bold indicates the best results for different training
conditions.

Methods | PESQ | STOI | CSIG | CBAK | COVL | DNSMOS
---|---|---|---|---|---|---
Noisy | 1.97 | 0.921 | 3.35 | 2.44 | 2.63 | 3.02
Non-parallel training
GAN-Normal | 2.01 | 0.914 | 3.48 | 2.74 | 2.67 | 2.68
GAN-Compressed | 2.03 | 0.916 | 3.54 | 2.78 | 2.72 | 2.72
CycleGAN | 2.56 | 0.927 | 3.78 | 3.14 | 3.16 | 3.38
AIA-CycleGAN | 2.67 | 0.932 | 3.86 | 3.20 | 3.21 | 3.47
Parallel training
GAN-Normal | 2.56 | 0.931 | 3.72 | 3.23 | 3.16 | 3.33
GAN-Compressed | 2.60 | 0.934 | 3.78 | 3.25 | 3.18 | 3.35
CycleGAN | 2.62 | 0.932 | 3.87 | 3.16 | 3.24 | 3.41
AIA-CycleGAN | 2.74 | 0.936 | 3.96 | 3.25 | 3.29 | 3.49

Figure 5: Visualization of the noisy/clean spectrogram and enhanced
spectrogram using different methods.

###  4.3 Comparison with other competitive GAN-based and Non-GAN based
approaches

Our proposed model is also compared with several other competitive GAN-based
and Non-GAN based baselines under standard paired data. As seen from Table 3,
AIA-CycleGAN outperforms these advanced GAN-based methods in terms of PESQ,
CSIG and COVL by considerable improvements, while providing similar STOI and
CBAK with CP-GAN [12]. For example, AIA-CycleGAN provides average 0.58 PESQ,
0.011 STOI, 0.48 CSIG, 0.31 CBAK and 0.49 COVL improvements than SEGAN [9],
which is the first GAN-based SE approach in the time domain. The significant
improvements in PESQ, CSIG and COVL indicate that our proposed method better
maintains speech integrity while reducing speech distortion. When compared
with other Non-GAN based methods, our AIA-CycleGAN also provides competitive
performance especially in terms of PESQ and CSIG scores. Note that we
reimplement GCRN [8] and DCCRN [5] in Voice bank + DEMAND dataset, and
directly use the reported scores of other methods in their original papers.

Table 3: Experimental results among different models under parallel training.

Methods | PESQ | STOI | CSIG | CBAK | COVL
---|---|---|---|---|---
Noisy | 1.97 | 0.921 | 3.35 | 2.44 | 2.63
GAN-based methods
SEGAN [9] | 2.16 | 0.925 | 3.48 | 2.94 | 2.80
MMSEGAN [13] | 2.53 | 0.930 | 3.80 | 3.12 | 3.14
RSGAN [10] | 2.51 | 0.937 | 3.78 | 3.23 | 3.16
RaSGAN [10] | 2.57 | 0.937 | 3.83 | 3.28 | 3.20
CP-GAN [12] | 2.64 | 0.940 | 3.93 | 3.29 | 3.28
Non-GAN based methods
Wave-U-net [37] | 2.64 |  – | 3.56 | 3.08 | 3.09
DFL-SE [38] |  – |  – | 3.86 | 3.33 | 3.22
CRN-MSE [39] | 2.61 | 0.938 | 3.78 | 3.11 | 3.24
GCRN [8] | 2.51 | 0.940 | 3.71 | 3.24 | 3.09
DCCRN [5] | 2.68 | 0.939 | 3.88 | 3.18 | 3.27
AIA-CycleGAN | 2.74 | 0.936 | 3.96 | 3.25 | 3.29

##  5 Conclusions

In this paper, we propose a novel adaptive attention-in-attention CycleGAN
(AIA-CycleGAN) to solve the difficulty in non-parallel speech enhancement
task. Specifically, we use relativistic adversarial losses, cycle-consistency
losses and an identity loss to jointly constrain the forward noisy-clean-noisy
cycle and backward clean-noisy-clean cycle. To effectively improve the feature
correlation learning in the generators, we integrate adaptive time-frequency
attention and adaptive hierarchical attention to form an attention-in-
attention module to capture local and global long-range dependencies. By
employing ATFA, generators can capture the long-range temporal and frequency
contextual information to distinguish different types of information for more
effective feature representations. By employing AHA, generators can capture
the long-range hierarchical contextual information to flexibly aggregate
different global feature maps by learnable weights. Experimental results
demonstrate that the proposed approach provides consistently better speech
enhancement performance than the previous GAN-based and CycleGAN-based
baselines under both parallel and non-parallel training.

## Acknowledgment

This work was supported in part by the National Natural Science Foundation of
China under Grant 61631016 and Grant 61501410, and in part by the Fundamental
Research Funds for the Central Universities under Grant 3132018XNG1805.

## References

  * [1] P. C. Loizou,  Speech enhancement: theory and practice,  CRC press, 2013.
  * [2] D. L. Wang and J. Chen,  “Supervised speech separation based on deep learning: An overview,”  IEEE/ACM Trans. Audio. Speech, Lang. Process., vol. 26, no. 10, pp. 1702–1726, 2018.
  * [3] Y. Wang, A. Narayanan, and D. L. Wang,  “On training targets for supervised speech separation,”  IEEE/ACM Trans. Audio. Speech, Lang. Process., vol. 22, no. 12, pp. 1849–1858, 2014.
  * [4] C. Hummersone, T. Stokes, and T. Brookes,  “On the ideal ratio mask as the goal of computational auditory scene analysis,”  in Blind source separation, pp. 349–368. Springer, 2014.
  * [5] Y. Hu, Y. Liu, S. Lv, M. Xing, and L. Xie,  “Dccrn: Deep complex convolution recurrent network for phase-aware speech enhancement,”  arXiv preprint arXiv:2008.00264, 2020.
  * [6] X. Lu, Y. Tsao, S. Matsuda, and C. Hori,  “Speech enhancement based on deep denoising autoencoder.,”  in Proc. of Interspeech, 2013, vol. 2013, pp. 436–440.
  * [7] Y. Xu, J. Du, L. R. Dai, and C. H. Lee,  “A regression approach to speech enhancement based on deep neural networks,”  IEEE/ACM Trans. Audio. Speech, Lang. Process., vol. 23, no. 1, pp. 7–19, 2014.
  * [8] K. Tan and D. L. Wang,  “Learning complex spectral mapping with gated convolutional recurrent networks for monaural speech enhancement,”  IEEE/ACM Trans. Audio. Speech, Lang. Process., vol. 28, pp. 380–390, 2019.
  * [9] S. Pascual, A. Bonafonte, and J. Serra,  “Segan: Speech enhancement generative adversarial network,”  Proc. of Interspeech , pp. 3642–3646, 2017.
  * [10] D. Baby and S. Verhulst,  “Sergan: Speech enhancement using relativistic generative adversarial networks with gradient penalty,”  in Proc. of ICASSP. IEEE, 2019, pp. 106–110.
  * [11] S. W. Fu, C. F. Liao, Y. Tsao, and S. D. Lin,  “Metricgan: Generative adversarial networks based black-box metric scores optimization for speech enhancement,”  in Proc. IMCL. PMLR, 2019, pp. 2031–2041.
  * [12] G. Liu, K. Gong, X. Liang, and Z. Chen,  “Cp-gan: Context pyramid generative adversarial network for speech enhancement,”  in Proc. of ICASSP. IEEE, 2020, pp. 6624–6628.
  * [13] M. H. Soni, N. Shah, and H. A. Patil,  “Time-frequency masking-based speech enhancement using generative adversarial network,”  in Proc. of ICASSP. IEEE, 2018, pp. 5039–5043.
  * [14] Y. Xiang and C. Bao,  “A parallel-data-free speech enhancement method using multi-objective learning cycle-consistent generative adversarial network,”  IEEE/ACM Trans. Audio. Speech, Lang. Process., vol. 28, pp. 1826–1838, 2020.
  * [15] G. Yu, Y. Wang, H. Wang, Q. Zhang, and C. Zheng,  “A two-stage complex network using cycle-consistent generative adversarial networks for speech enhancement,”  arXiv preprint arXiv:2109.02011, 2021.
  * [16] Z. Meng, J. Li, Y. Gong, et al.,  “Cycle-consistent speech enhancement,”  arXiv preprint arXiv:1809.02253, 2018.
  * [17] J. Y. Zhu, T. Park, P. Isola, and A. A. Efros,  “Unpaired image-to-image translation using cycle-consistent adversarial networks,”  in Proc. of ICCV, 2017, pp. 2223–2232.
  * [18] T. Kaneko and H. Kameoka,  “CycleGAN-vc: Non-parallel voice conversion using cycle-consistent adversarial networks,”  in 2018 26th European Signal Processing Conference (EUSIPCO). IEEE, 2018, pp. 2100–2104.
  * [19] A. Li, C. Zheng, R. Peng, and X. Li,  “On the importance of power compression and phase estimation in monaural speech dereverberation,”  JASA Express Letters, vol. 1, no. 1, pp. 014802, 2021.
  * [20] A. Li, W. Liu, X. Luo, G. Yu, C. Zheng, and X. Li,  “A simultaneous denoising and desreverberation framework with target decoupling,”  arXiv preprint arXiv:2106.12743, 2021.
  * [21] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier,  “Language modeling with gated convolutional networks,”  in Proc. IMCL. PMLR, 2017, pp. 933–941.
  * [22] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida,  “Spectral normalization for generative adversarial networks,”  in Proc. ICLR, 2018.
  * [23] Y. Wang, G. Yu, J. Wang, H. Wang, and Q. Zhang,  “Improved relativistic cycle-consistent gan with dilated residual network and multi-attention for speech enhancement,”  IEEE Access, vol. 8, pp. 183272–183285, 2020.
  * [24] Z. Ni, W. Yang, S. Wang, L. Ma, and S. Kwong,  “Towards unsupervised deep image enhancement with generative adversarial network,”  IEEE Transactions on Image Processing, vol. 29, pp. 9140–9151, 2020\.
  * [25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin,  “Attention is all you need,”  arXiv preprint arXiv:1706.03762, 2017.
  * [26] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena,  “Self-attention generative adversarial networks,”  in Proc. IMCL. PMLR, 2019, pp. 7354–7363.
  * [27] C. Tang, C. Luo, Z. Zhao, W. Xie, and W. Zeng,  “Joint time-frequency and time domain learning for speech enhancement,”  in Proc. of AAAI, 2020, pp. 3816–3822.
  * [28] A. Jolicoeur-Martineau,  “The relativistic discriminator: a key element missing from standard gan,”  arXiv preprint arXiv:1807.00734, 2018.
  * [29] C. Valentini-Botinhao, X. Wang, S. Takaki, and J. Yamagishi,  “Investigating rnn-based speech enhancement methods for noise-robust text-to-speech.,”  in Proc. SSW, 2016, pp. 146–152.
  * [30] C. Veaux, J. Yamagishi, and S. King,  “The voice bank corpus: Design, collection and data analysis of a large regional accent speech database,”  in Proc. O-COCOSDA/CASLRE. IEEE, 2013, pp. 1–4.
  * [31] J. Thiemann, N. Ito, and E. Vincent,  “The diverse environments multi-channel acoustic noise database: A database of multichannel environmental noise recordings,”  Acoustical Society of America Journal, vol. 133, no. 5, pp. 3591, 2013.
  * [32] D. Kingma and J. Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.
  * [33] A. Rix, J. Beerends, M. Hollier, and A. Hekstra,  “Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,”  in Proc. of ICASSP. IEEE, 2001, vol. 2, pp. 749–752.
  * [34] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen,  “A short-time objective intelligibility measure for time-frequency weighted noisy speech,”  in Proc. of ICASSP. IEEE, 2010, pp. 4214–4217.
  * [35] Y. Hu and P. C. Loizou,  “Evaluation of objective quality measures for speech enhancement,”  IEEE/ACM Trans. Audio. Speech, Lang. Process., vol. 16, no. 1, pp. 229–238, 2007.
  * [36] C. K. Reddy, V. Gopal, and R. Cutler,  “Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,”  arXiv preprint arXiv:2010.15258, 2020.
  * [37] D. Stoller, S. Ewert, and S. Dixon,  “Wave-u-net: A multi-scale neural network for end-to-end audio source separation,”  arXiv preprint arXiv:1806.03185, 2018.
  * [38] F. Germain, Q. Chen, and V. Koltun,  “Speech denoising with deep feature losses,”  in Proc. of Interspeech, 2019, pp. 2723–2727.
  * [39] K. Tan and D. L. Wang,  “A convolutional recurrent neural network for real-time speech enhancement.,”  in Proc. of Interspeech, 2018, pp. 3229–3233.