# Audiovisual Singing Voice Separation

Bochen Li, Yuxuan Wang, and Zhiyao Duan Bochen Li and Zhiyao Duan are with the
Department of Electrical and Computer Engineering, University of Rochester,
NY, USA. Yuxuan Wang is with the ByteDance Inc. E-mails<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Separating a song into vocal and accompaniment components is an active
research topic, and recent years witnessed an increased performance from
supervised training using deep learning techniques. We propose to apply the
visual information corresponding to the singers’ vocal activities to further
improve the quality of the separated vocal signals. The video frontend model
takes the input of mouth movement and fuses it into the feature embeddings of
an audio-based separation framework. To facilitate the network to learn
audiovisual correlation of singing activities, we add extra vocal signals
irrelevant to the mouth movement to the audio mixture during training. We
create two audiovisual singing performance datasets for training and
evaluation, respectively, one curated from audition recordings on the
Internet, and the other recorded in house. The proposed method outperforms
audio-based methods in terms of separation quality on most test recordings.
This advantage is especially pronounced when there are backing vocals in the
accompaniment, which poses a great challenge for audio-only methods.

###### Index Terms:

Source separation, audiovisual analysis, singing performance.

##  I Introduction

Vocal performance is an important art form of music. The task of singing voice
separation is to isolate vocals from the audio mixture, which contains other
instrumental sounds that help to define the harmony, rhythm, and genre.
Singing voice separation is often the first step towards many application-
oriented vocal processing tasks including pitch correction, voice
beautification, and style transfer, as implemented in some mobile Apps such as
WeSing and Smule. It is also often a pre-processing step for other research
tasks such as singer identification [1], lyrics alignment [2], and tone
analysis [3].

There are various scenarios when video recordings are available for singing
performances, such as operas, music videos (MV), and self-recorded singing
activities. In pop music, creative visual performances give artists a
substantial competitive advantage. Moreover, due to the rapid growth of
Internet bandwidth and smartphone users, videos of singing activities are
becoming popular in a number of video sharing platforms such as TikTok and
Instagram.

Visual information, e.g., lip movement, has been incorporated and shown its
benefits in speech signal processing, such as audiovisual speech separation
[4], enhancement [5], and recognition [6]. Visual information has also been
incorporated in music analysis [7], such as source association [8, 9, 10],
source separation [11], multi-pitch analysis [12], playing technique analysis
[13], cross-modal retrieval [14] and generation [15, 16]. For singing
performances, however, little work has been done. It is reasonable to think
that visual information would also help to analyze singing activities, and in
particular, separate singing voices from background music. This is based on
the fact that mouth movements and facial expressions of the singer are often
correlated with the singing voice signal fluctuations. The advantages of
audiovisual analysis over audio-only analysis can be best demonstrated in
songs with backing vocals in the accompaniment and songs with multiple singers
singing simultaneously. However, to what extent does the incorporation of
visual information help singing voice separation is still a question.
Different from speech signals, singing voices (except for rap music) generally
change slower [17], showing less frequent matching with mouth movements [18].
Furthermore, some musically important fluctuations of the singing voice such
as pitch modulations show little, if any, correlation with mouth movements
[19].

Therefore, it is our intention to answer the following research question in
this paper: Can visual information about the singer improve singing voice
separation, and if yes, how much? It is noted that while traditional singing
voice separation tasks (e.g., SiSEC111A community-based signal separation
evaluation campaign. https://sisec18.unmix.app/#/ or MIREX222Music Information
Retrieval Evaluation eXchange. https://www.music-ir.org/mirex/wiki/MIREX_HOME)
define all vocal components in a song as the singing voice, in this work we
define it as separating the solo singing voice from the accompaniments, where
the accompaniments may contain backing vocals. We argue that our definition is
more musically meaningful as it separates solo, typically presenting the main
melody, from accompaniment, typically presenting harmony. Separating the solo
voice enables many applications such as pitch refinement and voice
beautification for the soloist without affecting the backing vocal sources.
The solo singing voice separation problem is somewhat similar to speech
enhancement with babble noise. However, music accompaniment is typically much
louder and richer in timbre than background noise in speech enhancement
settings. In addition, music accompaniment, especially backing vocal, shows
very strong correlations with the solo vocal signal. They make the problem at
hand very challenging.

To answer the above-mentioned research question, we design an audiovisual
neural network model to separate the solo singing voice from the
accompaniments that may contain backing vocals. This network model takes both
the audio mixture signal and the mouth region of the singing video as input.
The audio processing sub-network is designed based on the MMDenseLSTM [20],
the champion of SiSEC2018 (the latest one of SiSEC). The visual processing
sub-network uses convolutional and LSTM layers to encode mouth movements of
the singer. The audio and visual encodings are fused before they are used to
reconstruct the solo singing magnitude spectrogram. The training target of the
proposed audiovisual network is to minimize the Mean-Square-Error (MSE) loss
of the magnitude spectrogram reconstruction of the solo singing voice. To
facilitate the network to learn audiovisual correlation of singing activities,
we add extra vocal signals irrelevant to the solo singer to the audio mixture
during training. To investigate the benefits of visual information, we compare
the proposed audiovisual model with several state-of-the-art audio-based
singing separation methods and an audiovisual speech enhancement method. We
further vary the architecture and input of the visual processing sub-network
to compare their performances.

One challenge we encounter in this work is the lack of audiovisual datasets of
singing. For training, this can be addressed by randomly mixing solo singing
videos downloaded from the Internet with irrelevant accompaniment music. We
download a cappella audition vocal performance videos and randomly mix their
audio with accompaniment audio tracks from the MUSDB18 dataset (officially
provided by SiSEC2018) to generate mixtures. We name this the _Audition-
RandMix_ dataset, and partition it into training, validation and test subsets.
For evaluation on real songs, however, we need audiovisual recordings of
singing with its relevant accompaniment music in separate tracks. To our best
knowledge, no such dataset exists. Therefore, we record a new audiovisual
dataset named _URSing_ , where singers are recruited to sing along with
prepared accompaniment tracks.

We conduct experiments on both the Audition-RandMix test set and the URSing
dataset. Results on both sets show that the proposed audiovisual method
outperforms baseline methods in most test conditions, no matter if the
accompaniment tracks contain the backing vocals or not. We further conduct
subjective evaluations on a cappella video performances in the wild to prove
the advantages of our proposed method.

The contributions of this paper include:

  * •

The first work to incorporate visual information to the state-of-the-art music
source separation framework to address the singing voice separation problem,

  * •

A proposal of solo voice separation where backing vocal components, if exist,
are regarded as accompaniment tracks, which better fits many application
scenarios, and

  * •

The first audiovisual singing performance dataset, URSing, free for
download333http://www.ece.rochester.edu/projects/air/projects/URSing.html.

##  II Related Work

###  II-A Singing Voice Separation

Early methods for singing voice separation include non-negative matrix
factorization [21], adaptive Bayesian modeling [22], robust principal
component analysis [23], and auto-correlation [24]. Recently, deep learning
based methods are proposed to model convolutional [25] or recurrent structures
[26] of magnitude spectral representations of music signals. Some works also
learn to reconstruct spectral phases in addition to magnitudes [27, 28], while
others directly work on time-domain waveforms with an end-to-end training
strategy [29, 30]. A direct comparison of recently proposed methods is
available at the SiSEC2018 post. The best performing methods in SiSEC2018 use
a DenseNet structure with a recurrent structure to process magnitude
spectrograms [31, 20], where the feature reuse strategy inside each dense
block greatly reduces the model size. Later some open-sourced methods/tools
have been proposed with comparable results, such as Open-Unmix [32] and
Spleeter [33]. In this paper, we build upon the DenseNet to propose an
audiovisual model.

###  II-B Audiovisual Source Separation

Most audiovisual separation works are proposed for speech signals. For speech
separation, one challenge is the permutation problem where the separated
components need to be assigned to the correct talkers. [34] specifically
address the problem by applying the visual information as a post-processing
step to adjust the separation mask. Later the same group proposes to fuse the
visual information to an audio-based deep clustering framework to propose an
audiovisual deep clustering model for speech separation [4]. Another work is
described in [35], where the input is the mixture spectrogram and the face
embeddings of all the appeared speakers in the audio sample. The training
target is the complex mask that can be applied to the original spectrogram to
recover the complex spectrogram of each speaker. It is noted that speech
separation algorithms typically assume a noiseless or less noisy environment
in which speech signals are mixed. In addition, speech signals to be separated
are typically assumed to be from different speakers. Both assumptions are not
true in solo singing separation, as the background music is often quite strong
and the backing vocal often comes from the same singer as the soloist.

Speech enhancement aims at separating speech signal from background noise. It
is more relevant to singing voice separation from background music considering
the foreground-background relations of sources. [36] address the speech
enhancement problem using a two-stream structure that takes both noisy speech
and frames of the cropped mouth regions as inputs to compute their features.
These features are then concatenated by a fusion network which also outputs
corresponding clean speech and reconstructed mouth regions. Another
audiovisual speech enhancement work proposed in [5] uses 1D convolutional
layers to reconstruct the magnitude spectrogram of the clean speech and uses
it to further estimate its phase spectrogram. The input of the visual branch
is the feature embeddings on the lip region that are pre-trained on lip
reading tasks.

Less work has been proposed for audiovisual music separation. [37] apply non-
negative matrix factorization (NMF) to separate string ensembles, where the
bowing motions are used to derive additional constraints on the activation of
audio dictionary elements. This method, however, is only evaluated on randomly
assembled video scenes of string instruments where distinct bowing motions of
each player are clearly captured. [38] propose to learn static audiovisual
correspondence with cross-modal source localization; The correlation between
each pixel in a given video frame and the sound component can be constructed.
Later the same group proposes to learn dynamic audiovisual correspondence [11]
which captures correlations between motions and sound fluctuations. Another
followup work is to recognize visual gestures for sound source separation
[39]. This line of research achieves promising results in audiovisual music
separation, but have not addressed singing voice separation.

##  III Method

###  III-A Network Architecture

Figure 1: The proposed model structure. Dashed arrows denote the concatenation
operation. Downsample/upsample are applied to both time and frequency
dimensions in the outer layers (marked by *), while they are only applied to
the frequency dimension in the inner layers.

The proposed system builds upon a state-of-the-art audio separation model
named MMDenseLSTM [20] with a video front-end model. The MMDenseLSTM model
consists of convolutional layers stacked into dense blocks, which alternates
downsample/upsample layers to form a multi-scale structure. It first embeds an
input magnitude spectrogram into an encoded feature space and decodes it to
recover the separated magnitude spectrogram. Skip connections are added as
concatenations on the corresponding layers with the same feature map size.
This “encoder-decoder” structure with skip connections is widely applied in
several music separation models [40, 30, 11, 41]. The video front-end model
extracts visual features from mouth movements, which are fused with the
encoded audio feature. The network structure is illustrated in Figure 1. We
explain each part of the model in detail as follows.

####  III-A1 Audio Separation Model

Following MMDenseLSTM, our audio separation model consists of:

  * •

Dense Block. It applies 2D convolutional layers444A convolutional layer
includes BatchNormalization+ReLU+Conv2D throughout the paper. and the output
feature maps of all layers are concatenated with each other along the channel
dimension. This structure reuses the feature maps from previous layers and
greatly reduces the model size.

  * •

Compression layer. It is a convolutional layer with 1$\times$1 kernels. We use
a compression ratio of 0.2, which means that the number of feature maps
(channels) is reduced by 80% after each compression layer. We apply a
compression layer right after each dense block, which improves the model
compactness.

  * •

Downsample-Upsample. These layers are used to resize the feature maps without
changing the the number of channels. Downsample layers are average pooling
with 2$\times$2 kernels after the first compression layer, and 1$\times$2
kernels in the following layers. In other words, downsampling is performed
along both the time and freuqncy dimensions in the first layer, but only to
the frequency dimension in other layers. Symmetrically, upsample layers apply
transposed convolutional layers with 2$\times$2 kernels and strides at the
last upsample layer but 1$\times$2 for the other layers. Different from [20]
where downsample/upsample always addresses both time and frequency dimensions
in multiple scales, our proposed strategy downsamples/upsamples the time
dimension only once, making the audio stream have the same frame rate as the
video stream. The encoded audio spectrogram feature is denoted as
$\mathbf{S}_{A}\in\mathbb{R}^{M\times T\times F}$, with the channel ($M$),
downsampled time ($T$), and frequency ($F$) dimensions.

  * •

Multi-Band. Following [31], we also equally divide the spectrogram into a low-
frequency band and a high-frequency band and apply the above-mentioned
encoder-decoder structure on each sub-band. The dense blocks of low-frequency
band have a higher channel number. Detailed parameters can be referred to
[31].

####  III-A2 Video Front-End Model

We propose to apply a separate input branch to parse the input video stream
and fuse it with the encoded audio features. The video stream is a sequence of
mouth region images in consecutive video frames. Raw RGB values are normalized
to zero mean and unit variance. We use 2D convolutional layers and LSTM layers
to extract the visual features from the input RGB frames. Each frame is
processed independently sharing the same CNN parameters before being fed into
the following LSTM layer. Detailed network architecture is Conv2D@16 (channel
number is 16), Conv2D@16, Conv2D@32, Conv2D@32, FC@256, LSTM@128, and FC@$N$,
where $N$ is the dimension of the encoded feature vector for each video frame.
The input video stream with $T$ frames results in a feature map
$\mathbf{S}_{V}\in\mathbb{R}^{N\times T}$. There is no pooling operation along
the time dimension thus the temporal information is preserved.

####  III-A3 Audiovisual Fusion

The extracted visual feature map from the video branch is fused with the
encoded audio spectrogram feature map $\mathbf{S}_{A}$. To do so, the visual
feature map $\mathbf{S}_{V}$ is inflated along the third dimension and then
concatenated with the audio feature to obtain the audiovisual feature
$\mathbf{S}_{AV}\in\mathbb{R}^{L\times T\times F}$, where $L=M+N$ is the
concatenated channel dimension. Note that the temporal information from both
the audio and video branches is correlated during this fusion; This is
different from some works where audiovisual fusion is performed on feature
maps that aggregate information along time.

In addition to minor structural changes, we also drop the LSTM structure of
the original MMDenseLSTM model [20] when we design the audio branch of our
proposed model. This follows the observation that the addition of the LSTM
structure does not achieve substantial improvement in SiSEC2018 yet the number
of parameters would be increased significantly for audiovisual fusion.

###  III-B Training

We train the model to predict the magnitude spectrogram of the source signal
and use the original mixture’s phase to recover the time-domain waveform. Many
spectral-domain source separation methods, especially those for speech
signals, use a spectrogram mask as the training target; This mask is then
multiplied element-wise with the mixture signal’s magnitude spectrogram to
recover the source magnitude spectrogram. For music separation, some recent
works train networks to directly output the source magnitude spectrogram [26,
20] using a Mean-Squared-Error (MSE) loss. We follow the same way and take the
source magnitude spectrogram as the training target. However, we have a mask
layer that regularizes the feature maps into the range of [0, 1] using a
Sigmoid function and multiplies the mask layer with the input spectrogram. We
find that this is necessary for fusing the visual input into the audio
feature, as the audiovisual method even degrades the separation performance in
some experimental settings when the mask layer is not in place. We have a
comparative experiment in Section V-E1.

The model input is the magnitude spectrogram of the original audio mixture
which contains both the solo vocal and background music, and the mouth region
of the video frames corresponding to the vocals. The output is the magnitude
spectrogram of the source audio. Note that the magnitude spectrogram has been
converted to logarithm scale followed by normalization along each frequency
axis, which better weighs the contribution of high frequency bins. Compared to
the audio mixture input, the visual input provides much less information about
the source signals, therefore, the training loss may not be propagated back
sufficiently into the visual branch, making the audiovisual network difficult
to train. One way to address this is to explicitly learn audiovisual matching,
either through pre-training [34] or early audiovisual fusion [4]. Another way
might be to add visual reconstruction as another training target, leading to a
chimera-like network structure [36].

In this work, we address this problem by adding some extra vocal components to
the original mixture, which are not related to the mouth movements and thus
are not included in the target vocal spectrogram. This is similar to adding an
additional speaker in the training data in the case of audio-visual speech
separation [35], which forces the model to learn audiovisual correlations
after the fusion and only separate the vocal components that are related to
the visual input. Note that in the training samples all of the vocal and
accompaniment components are randomly mixed, so neither the extra vocal
components or the solo vocal components have harmonic relations with the
accompaniment tracks. In the experiments, we show that the strategy of
training with randomly generated vocal-accompaniment pairs performs decently
on real songs.

##  IV Dataset

Since there is no publicly available audiovisual singing voice dataset
containing isolated vocal tracks, we collect our own data for training and
evaluating the proposed method.

###  IV-A A Cappella Audition Vocals (AAV)

We curated 491 YouTube videos of solo singing performances by querying the
YouTube search API with the keyword “Academic Acappella Audition”. We only
selected video excerpts where the singer faces the camera and sings without
accompaniment. The total length of these excerpts is about 8 hours. As it is
difficult to find relevant and appropriate accompaniment tracks, in our
experiments we simply randomly chose instrumental accompaniment tracks (from
the “accompaniments” track in the MUSDB18 dataset) and mixed them with the
solo singing excerpts to create singing-accompaniment mixtures.

The randomly mixed samples are used for training, validation, and evaluation.
Before the mixing process, vocals in AAV are divided into
training/validation/evaluation sets roughly as 8:1:1 (50 tracks for
evaluation). Accompaniment tracks from MUSDB18 (which contains a wide range of
music genres and instrument types) are also divided into the three sets
following the official way (also 50 tracks for evaluation). Then mixing is
applied on each split independently to form the training/validation/evaluation
sets. Volume of each track is normalized using the root-mean-square (RMS)
value. For training and validation sets, each track is split into short
samples (around 2.5 seconds) for random mixing, resulting in a massive amount
of mixed samples. We do not balance the volume of each individual sample so
the mixing may have different SNRs. For evaluation, mixing is performed on a
random bijection between the 50 vocals and 50 accompaniments. For each mixing,
we pick a 30-second excerpt (with both vocal and accompaniments present) for
evaluation, following the same strategy as the MUSDB18 dataset. This set is
referred to as “Audition-RandMix” in the following experiments. In addition,
we randomly add extra vocals from another vocal track in the test set (with
the same RMS value as the target vocal) into these mixings for evaluations,
referred as “Audition-RandMix (v+)”, in order to explore the model performance
in more challenging cases. Note that all the testing samples in this condition
are not musically meaningful and cannot represent real songs.

###  IV-B URSing

To evaluate the proposed method in more realistic singing performances, we
create the University of Rochester Multi-Modal Singing Performance Dataset
(URSing). In this paper, we only use the URSing dataset for evaluation. A
brief description of the creation process is described below.

Figure 2: A sample photo and floor plan of the sound booth for the recording
process of the URSing dataset.

####  IV-B1 Singer Recruiting

Singers are students at the University of Rochester. Audition is performed to
filter out unqualified singers who could not sing in tune. Each participant
receives $5 for recording each song, and is allowed to record up to 5 songs.

####  IV-B2 Piece Selection

To ensure high recording efficiency, the singers pick their own songs and
their favorite accompaniment tracks to sing along. We do not put constraints
on song genres, but filter out songs of which the accompaniment tracks are of
low sound quality.

####  IV-B3 Recording

To ensure synchronization, the singers listen to the accompaniment track
through earphones while recording their singing voice. Their voices are
recorded using an AT2020 condenser microphone hosted by Logic Pro X, and their
videos are recorded using iPhone 11. The recording is conducted in a semi-
anechoic sound booth. A sample photo and the floor plan of the sound booth are
shown in Figure 2.

####  IV-B4 Post-processing

For each solo vocal recording we use the following plug-ins to simulate the
typical audio production procedure in commercial recordings: a) static noise
reduction (Klevgrand Brusfri and Waves X-noise), b) pitch refinement
(Melodyne), c) sound compression (Fabfilter Pro-C 2), and d) reverberation
(Fabfilter Pro-R). We also adjust the vocal volume to balance it with the
accompaniment track. Beyond this, we do not perform any other editing on the
audio recording (e.g., time warping or rhythmic refinement) to preserve the
synchronization with the visual performance. To synchronize the audio
recording captured by the AT2020 microphone with the video recording captured
by the smartphone, we use the audio recording captured by the built-in
microphone of the smartphone as the bridge, through cross correlation.

####  IV-B5 Annotation

Since the mouth movements are mostly relevant to the singing performance, we
provide the annotations of the mouth regions in the dataset. This is performed
using the Dlib library [42], an automatic tool for facial landmark detection,
followed by manual check. The mouth region is represented as a square bounding
box with the side length equal to 1.2 times of the maximum horizontal distance
for all mouth landmarks.

Figure 3: Examples of video frames of the URSing dataset and cropped mouth
region pictures as the input to the video branch of the proposed method.

This results in 65 songs, totaling 4 hours of audiovisual recordings of
singing performance. For each song, we provide the audio recording of the solo
singing voice (in WAV, 44.1 KHz, 16 bits, mono) and the corresponding
accompaniment audio track (same format, mono or stereo). We also provide the
video recording of the soloist (in MP4, 1080P portrait, 29.97 FPS), where the
soundtrack is the mixture (direct sum) of the solo vocal and the accompaniment
tracks. Note that when we prepare the accompaniment tracks, we do not avoid
the tracks containing backing vocals, as they are the challenging and useful
cases to study in this paper. Example video frames and cropped mouth region
pictures are provided in Figure 3.

We also choose a set of 30-sec excerpts where both solo vocal and
accompaniment tracks are prominent to form a benchmark evaluation set.
Specifically, for each of the 65 songs, we choose one 30-sec excerpt without
backing vocal and one with back vocal, if such excerpts are available. We
provide this information in the metadata. This results in 54 excerpts with
accompaniment tracks that only contain instrumental components (referred as
“URSing” in the following experiments) and 26 excerpts with accompaniment
tracks that also contain backing vocals (referred as “URSing (v+)”. The
latter, presumably, are more challenging for solo vocal separation and more
useful for showing advantages of audiovisual methods. In this paper, since we
do not use any songs from URSing for training, we only use these 30-sec
excerpts for evaluation.

##  V Experiments

###  V-A Implementation Details

For audiovisual singing videos, audio is downsampled to 32 KHz. We use a frame
length of 1024 and a hop size of 640 (20 ms) for spectrogram calculation.
Video data is converted to 25 FPS (equivalent to 40 ms frame hop size), and
the frame size of mouth regions is interpolated into 64 $\times$ 64\. Each
data sample is 2.56 seconds long, containing 128 audio frames and 64 video
frames. The input/output audio spectrogram has the shape of
2$\times$128$\times$513 (channels $\times$ frames $\times$ frequency bins),
and each input video stream has the shape of 64$\times$64$\times$64 (frames
$\times$ width $\times$ height).

During training, for half of the training/validation samples, we add extra
vocal components that are not related to the mouth movements to encourage the
model to learn audiovisual correlations. We use batch size of 8 for training
on a TITAN X GPU with 11.9 GB graphic memory. It takes about 40 hours to train
for 50 epochs. We adopt early stopping when the validation loss does not
decrease for 10 consecutive epochs.

For evaluations, we calculate the signal-to-distortion ratio (SDR) between the
separated vocal waveforms and the ground-truth ones using the BSS Eval toolbox
V4, same as the evaluation measure applied in SiSEC2018. Specifically, for
each 30-sec evaluation excerpts, we calculate the median SDR over all 1-sec
audio segments.

###  V-B Baselines

We first use the original mixture recording (referred as “MIX” in the
experiments) as the separated vocal for evaluation on our dataset. This sets
lower bounds of separation results without any separation techniques. Then we
apply two oracle filtering techniques that utilize ground-truth source
signals: The ideal binary mask (IBM) assigns each time-frequency bin to the
predominant source. The ideal ratio mask (IRM) distributes the power of each
time-frequency bin into different sources according to the power ratio of the
ground-truth sources. The IBM and IRM set upper bounds for time-frequency
masking-based source separation methods.

We then compare our proposed method with several audio-based music separation
methods as baselines.

  * •

RX7. A commercial software developed by iZotope555https://www.izotope.com. We
apply batch processing of the “music rebalance” function with the preset
“isolate vocals” on “medium” level. Training data for the model inside this
software is unknown to us.

  * •

UMX [32]. An open-sourced separation tool known as “Open-unmix” . The model
employs the BLSTM structure and is trained on the MUSDB18 dataset.

  * •

Spleeter [33]. An open-sourced separation tool with a CNN+Unet model trained
on their in-house dataset of 24,097 songs. It achieved best separation results
among all open-source tools on the evaluation set of the MUSDB18 dataset up to
date.

  * •

Spleeter-train. Same model as “Spleeter” but trained on our Audition-RandMix
dataset using the same conditions as those for our proposed audiovisual method
as a direct comparison.

  * •

MMDenseLSTM [20]. The method that achieved the best results in SiSEC2018, even
without training on extra data. We implemented this method from scratch. Our
implementation has been validated by achieving similar results to the reported
ones on MUSDB18, following SiSEC2018’s official train/test split. We then
trained this model on our Audition-RandMix dataset as a direct comparison.

We also implement an audiovisual speech enhancement method named AVDCNN
proposed in [36]. This method applies 2D CNNs to take noisy speech and the
mouth region visual recording as inputs, fuses encoded audio and visual
features to output the enhanced speech signal as well as reconstructed video
frames of mouth movements. After the fusion layers, we used LSTM instead of
fully-connected layers as used in [36], which shows higher performance in our
experiment scenarios.

We choose audiovisual speech enhancement instead of audiovisual speech
separation as the baseline, because we believe that speech enhancement is more
relevant to singing voice separation from background music in terms of
foreground-background relations of sources, as explained in Section II-B. In
addition, audiovisual speech separation usually assumes the availability of
all talkers, while in our setting, only the video of the solo singing voice is
used.

We present the model sizes of baseline models that are open-source or
implemented by us in Table 1, together with that of the proposed model.

Method | UMX | Spleeter | MMDenseLSTM | AVDCNN | Proposed
---|---|---|---|---|---
Parameter |  |  |  |  |
($\times 10^{6}$) | 8.5 | 19.7 | 1.22 | 11.3 | 2.05
TABLE I: Comparison of model size of different methods.

###  V-C Objective Evaluation on Synthetic Mixtures

Figure 4: The SDR (dB) comparison on separated solo vocals with different
methods on different evaluation sets. (“v+” denotes for songs where
accompaniments contain vocal components.)

We evaluate the comparison methods on the four test sets described in Section
IV: Audition-RandMix, Audition-RandMix (v+), URSing, and URSing (v+). “v+”
means that the accompaniments contain vocal components. Note that all these
songs are synthetic mixtures, e.g., Audition-RandMix is random mixed samples
and URSing is recorded in controlled environment. Boxplots of SDR results are
shown in Figures 4, where each data point in the boxplots is the median SDR of
the separated vocal of all 1-sec segments of a 30-sec excerpt. The horizontal
line inside each box indicates the median value across all excerpts. Several
interesting observations can be made from the results.

####  V-C1 Benefits of Visual Information

The proposed method outperforms all audio-based separation baselines in most
of the evaluation sets. This shows the advantage of incorporating visual
information about the singer’s mouth movement for solo singing voice
separation. Among the audio-based baseline methods, MMDenseLSTM is much
stronger than RX7, because MMDenseLSTM is our own implementation and is
trained on our dataset while RX7 is not. However, Spleeter slightly
outperforms our proposed system on the URSing set. We believe that this is
because Spleeter is trained on a much larger in-house dataset that contains
24,097 songs totalling 79 hours. This is verified by the fact that, Spleeter-
train, the same model as Spleeter but trained on our dataset as a fair
comparison, does not outperform MMDenseLSTM nor the proposed method. We
suggest that this is because our proposed model (and MMDenseLSTM) has a much
smaller model size than Spleeter, making it less prone to overfitting given a
small training set.

Comparing songs with backing vocals (Audition-RandMix (v+) and URSing (v+)) to
songs without backing vocals (Audition-RandMix and URSing), we can see that
the outperformance of the proposed method is better pronounced on songs with
backing vocals. Wilconxon signed-rank tests show that the improvement of the
proposed method over MMDenseLSTM on Audition-RandMix (v+) and URSing (v+) are
both significant, with $p$ values of $6.2\times 10^{-3}$ and $4.3\times
10^{-2}$, respectively. We argue that this is because audio-only methods tend
to assign all the vocal components to the separated singing voice, while the
proposed audiovisual method learns to only separate the vocal signals that are
correlated to the solo singer’s mouth movements.

The reason that the improvement is more pronounced on Audition-RandMix (v+)
than on URSing (v+), we argue, are twofold: 1) backing vocals in URSing (v+)
are not as strong as the intentionally added backing vocals in Audition-
RandMix (v+), and 2) backing vocals in URSing (v+) often overlap with solo
vocals and share the same lyrics, showing high correlations with the mouth
movements of the solo singer, while the added backing vocals in Audition-
RandMix (v+) are irrelevant to the solo vocal.

Figure 5 shows one 10-sec sample as an extreme case to compare the
spectrograms of audio-based MMDenseLSTM method and the proposed audiovisual
method when backing vocal components are strong (e.g., the middle part of the
sample). We also show the mouth movement in several frames throughout this
excerpt. It can be seen that MMDenseLSTM recognizes the backing vocal
components in the middle frames as the solo vocal, while the audiovisual
method suppresses those components significantly.

Figure 5: One 10-sec example comparing audio-based separation (MMDenseLSTM)
with audiovisual separation (proposed) on a song excerpt with strong backing
vocals. The four spectrograms from top to bottom are original mixture, ground-
truth vocal, audio-based vocal separation, and audiovisual vocal separation.
This sample result has 10-sec long, and one mouth frame of each second is
attached.)

On songs without backing vocals, the outperformance of the proposed method can
still be observed. Subjective listening by the authors suggests that the
visual information helps to reduce high-frequency percussive sounds from the
solo vocal, as the former do not correlate with mouth movements well.

####  V-C2 Superiority of Proposed Audiovisual Architecture

The proposed method outperforms the audiovisual speech enhancement baseline
significantly in all evaluation sets. Note that the baseline is trained and
evaluated on the same dataset as the proposed method. This shows the
superiority of the proposed network architecture on the solo singing voice
separation task. In particular, we argue two main reasons. First, the proposed
model utilizes the commonly used U-net structure with skip connections, which
generally achieves good results in music separation [40, 30, 31]. Second, in
our audiovisual fusion scheme we preserve the temporal correspondence, which
prevents a substantial increase of the number of trainable parameters in the
fusion layer. This is important when the DenseNet-based audio sub-network has
a small model size. The variations of different video sub-networks, however,
does not make much difference on the separation performance, as we analyzed in
Section V-E1.

####  V-C3 Limitations and Room for Improvement

Compared with reported SDR values in SiSEC2018, the SDR values in Figure 4 are
much higher. For example, MMDenseLSTM reaches over 10 dB on URSing but only
less than 7 dB in SiSEC2018 (method “TAK1” in [43]). We argue that the songs
used in SiSEC2018 (i.e., the MUSDB18 dataset) are professionally recorded,
mastered and mixed vocals. They often contain complex components such as
polyphonic vocals, background humming, and strong reverberation. They are
mastered and mixed by professional music producers to intentionally make them
better fused into the background music. In contrast, the ground-truth vocals
in our datasets are solo vocals recorded in controlled environments with
limited vocal effects added. It is reasonable to believe that the benefits of
visual information can be further demonstrated on more professionally produced
songs. In addition, the performance difference between the Audition-RandMix
test sets and the URSing test sets seems to be small for all methods,
including the oracle results. This shows that randomly mixed songs, although
lacking harmonic and rhythmic coherence, are not easier to separate than the
more realistically mixed songs, suggesting that it may be reasonable to use
randomly mixed songs to train the methods [44]. However, whether this is still
true for professionally produced songs is still a question.

On the other hand, there is still some gap between the proposed method and the
oracle results on the SDR metric in our evaluation sets. It is likely that
this gap will be even bigger on professionally produced songs. This suggests
that much work can be done to improve the separation performance. For example,
time-domain separation for the audio branch may further improve the
performance [45].

###  V-D Subjective Evaluation on Professional A Cappella Songs

Figure 6: One sample frame of an a cappella song for subjective evaluation.

Figure 7: Statistics of the 26 subjects’ musical background related to the
subjective evaluation.

Figure 8: The subjective ratings of the separation quality in response to the
three questions. Each error bar shows mean $\pm$ standard deviation.

In this section, We further evaluate the benefits of visual information
incorporated in our proposed method on real a cappella songs in the wild. We
collect 35 audiovisual a cappella recordings from YouTube. These collections
represent the extreme cases where all the accompaniment components are vocals
(except for several cases where additional percussive instruments are also
present), to study how much the proposed audiovisual method is advantageous
while the audio-based method is very likely to fail. Here we use the
MMDenseLSTM baseline as the audio-based method for comparison, which yields
the best separation results among audio-based baselines. Most of these songs
are chorus performance with a solo singer accompanied by harmonic vocals
and/or vocal beatbox, while some are performance with multiple solo singers.
We only keep the videos where the solo singer’s mouth is visible and clear,
without video shot transition for at least 10 seconds. A sample frame of one
song is shown in Figure 6 with the mouth region of the targeted solo singer
highlighted.

As we do not have access to the source tracks, we cannot evaluate the
separation performance using common objective evaluation metrics. Instead, we
conduct a subjective evaluation over 51 people. Some subjects are students or
faculty from the University of Rochester, others are subscribers from the
International Society for Music Information Retrieval (ISMIR) community.
Statistics of the subjects’ music background is shown in Figure 7. Each survey
asks a subject to rate 7 of the 35 songs, and each subject may take more than
one surveys. For each song, the subjects first watch a 10-sec excerpt of the
original performance and then watch the same video twice with the solo singing
voice separated by two different singing voice separation methods in a random
order to rate the separation quality. Due to the variations across these
songs, the original recording serves as a reference for a consistent scoring
scheme. For each video we also highlight the mouth region of the target solo
singer (see Figure 6) to help subjects focus on the corresponding solo voice.
The specific evaluation questions are:

  * •

Question 1: What do you think about the overall separation quality for the
targeted singer?

  * •

Question 2: What do you think about the separation quality in terms of
removing backing vocal accompaniments in the separated solo voice?

  * •

Question 3: What do you think about the separation quality in terms of not
introducing artifacts into the separated solo voice?

The subjects need to answer each question using a scale from 1 to 5, where “1”
represents Very bad and “5” represents Very good. The three questions are
related to the common definitions of the three objective source separation
evaluation metrics, SDR, SIR, and SAR, respectively.

The results of the subjective evaluations are presented in Figure 8. According
to the collected responses for Question 1, the proposed audiovisual method is
rated significant higher than the baseline audio-based method (Wilconxon
signed-rank test shows a $p$ value of $3.5\times 10^{-31}$); The average
rating is raised from 3.1 to 3.9. For Question 2, the difference is even more
significant, as the average rating is increased from 2.6 to 3.8 (with a $p$
value of $3.1\times 10^{-45}$), showing that the proposed method is especially
beneficial for removing accompaniments from the mixture. Regarding the
artifacts introduced into the separated solo vocals in Question 3, both
methods achieve a rating between “neutral” and “good”, and the difference is
not statistically significant (with a $p$ value of $0.46$).

###  V-E Ablation Studies

To further study how the visual information helps with the separation
performance, we design several complementary experiments as ablation studies.
We first modify the network structure by replacing the video front-end model
with other existing widely-applied visual feature extraction framework to
explore the key factor of the audiovisual separation framework and the
robustness. Then we feed the visual branch with non-informative or even
misleading inputs to observe how the separation quality degrades.

####  V-E1 Different Video Front-End Models

To investigate the effects of the video front-end on the separation
performance, we replace the proposed Conv2D+LSTM video front-end with several
other widely-used visual feature extraction frameworks:

  * •

No-mask. This experiment has the same video branch, but without a mask layer
after the audiovisual fusion.

  * •

Conv3D. The Conv3D model takes all the video frames from each sample as a
feature map and a 4-th dimension is added as the channel dimension set as 64.
We then apply 2 Conv2D layers (with the channel dimension 128 and 256) on each
frame to share the channel dimension with Conv3D. Followed by pool operation
and fully-connected layers, we obtain the video feature with the same
dimension as $\mathbf{V}_{\texttt{Conv3D}}\in\mathbb{R}^{N\times T}$. Note
that in this structure, the temporal information is only parsed at the very
first Conv3D structure, since no recurrent network is applied.

  * •

Dense+LSTM. Different from the proposed model, we replace the Conv2D layers
with a dense block from the DenseNet structure. Each dense block has 2 layers
with growth rate of 12. Then a Conv2D layer with 1$\times$1 kernels is applied
to compress the channel number to 32, resulting in the same feature dimension
as the proposed CNN+LSTM model before feeding into the FC@256.

  * •

Lip-reading. This variation uses a pre-trained model proposed in [6] on the
lip reading task on the LRW dataset [46]. The original model structure
consists of Conv3D, ResNet-34, and GRU. We only use the pre-trained model to
extract the visual feature to integrate into our proposed audiovisual source
separation model.

Figure 9: The SDR (dB) comparison on the separated solo vocal from the
audiovisual method using different video front-end models.

A comparison of different video front-end models is shown in Figure 9. It can
be seen that the proposed (Conv2D+LSTM) model achieves the highest SDR values
for most cases, but some video front-end models do not make much difference.
Applying a mask layer is critical, as otherwise audiovisual method even
degrades from the audio-based method. Note that for audio-based baseline
method (MMDenseLSTM), we have also experimented models with a mask layer or
not, but it does not make difference on the separation results. The Conv3D
framework slightly degrades the performance, but still outperforms the audio-
based baseline method (MMDenseLSTM). One reason for this performance drop may
be that in this framework, there is no recurrent structure, and the temporal
evolution of visual information is only processed by the Conv3D structure. As
the Conv3D structure takes the raw input of mouth frames, it may be sensitive
to mouth position changes due to landmark detection errors. The model pre-
trained on lip reading ranks the worst among the audiovisual models. This is
because the lip reading model was trained on the LRW dataset where for each
sample containing several words, only one word around the center frames is
annotated as the training target. This makes the model only attend to the
middle frames of a video excerpt, leading to limited guidance for the singing
voice separation and even degradation from audio-based methods.

####  V-E2 Non-Informative Visual Input

Figure 10: The SDR (dB) comparison on the separated solo vocal of the proposed
audiovisual method with non-informative visual inputs.

To further investigate how the incorporation of visual information affects the
separation performance, in this section, we substitute the visual input (i.e.,
mouth region of the solo singer) with some irrelevant content.

  * •

Constant. We feed the visual branch with constant zero values all the time.

  * •

White-noise. We feed the visual branch with white noise that is normalized to
the same range as the videos of mouth regions.

  * •

White-noise*. The white noise is directly fused with the audio embedding,
replacing the whole visual branch.

  * •

Mismatch. The input of the visual branch is the mouth region video of an
irrelevant singer to provide misleading information about the singing
activity.

Figure 10 shows the separation results on different experimental settings. The
model performance always degrades from the audio-based baseline MMDenseLSTM
when feeding with irrelevant or misleading information. This suggests that a
non-informative visual input is harmful for separation. The performance
degradation by feeding white noise or a mismatched singer is more noticeable
than a constant input. This may be because the model is more likely to overfit
irrelevant visual fluctuations in the training data, while for a constant
visual input the model is more likely to ignore it. Nonetheless, in all of
these circumstances, the separation performance still achieves a median SDR
over 5dB for most cases. This suggests that the audio branch is dominant in
the model inference. Comparing with the “No-mask” results in Figure 9, this
also confirms our claim in Section V-E1 that the mask layer helps to improve
the model robustness, even when the visual input is less informative.

##  VI Conclusion

In this paper, we proposed an audiovisual approach to address the solo singing
voice separation problem by analyzing both the auditory signal and mouth
movement of the solo singer in the visual signal. To evaluate our proposed
method, we created the URSing dataset, the first publicly available dataset of
audiovisual singing performances recorded in isolation for singing voice
separation research. We also curated a solo singing voice dataset from YouTube
for training. Both objective evaluations on artificially mixed singing music
and subjective evaluation on professionally produced a cappella songs showed
that the proposed method significantly outperforms state-of-the-art audio-
based methods. The advantages of the proposed method is especially pronounced
when the accompaniment track contains backing vocals, which have been
difficult to separate from solo vocals by audio-based methods.

## Acknowledgment

We thank all of the singers who participated in our dataset recording process,
and Haiqin Yin for post-processing the audio recordings. We also acknowledge
the funding agency.

## References

  * [1] A. Berenzweig, D. P. Ellis, and S. Lawrence, “Using voice segments to improve artist classification of music,” in _Proceedings of the AES 22nd International Conference: Virtual Synthetic and Entertainment Audio_ , 2002.
  * [2] H. Fujihara, M. Goto, J. Ogata, K. Komatani, T. Ogata, and H. G. Okuno, “Automatic synchronization between lyrics and music cd recordings based on viterbi alignment of segregated vocal signals,” in _Proceedings of the IEEE International Symposium on Multimedia (ISM)_ , 2006, pp. 257–264.
  * [3] H. Fujihara and M. Goto, “A music information retrieval system based on singing voice timbre.” in _Proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2007, pp. 467–470.
  * [4] R. Lu, Z. Duan, and C. Zhang, “Audio–visual deep clustering for speech separation,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 27, no. 11, pp. 1697–1712, 2019.
  * [5] T. Afouras, J. S. Chung, and A. Zisserman, “The conversation: Deep audio-visual speech enhancement,” in _Proceedings of the International Conference on Spoken Language Processing (Interspeech)_ , 2018.
  * [6] S. Petridis, T. Stafylakis, P. Ma, F. Cai, G. Tzimiropoulos, and M. Pantic, “End-to-end audiovisual speech recognition,” in _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 6548–6552.
  * [7] Z. Duan, S. Essid, C. Liem, G. Richard, and G. Sharma, “Audiovisual analysis of music performances: Overview of an emerging field,” _IEEE Signal Processing Magazine_ , vol. 36, no. 1, pp. 63–73, 2019, dOI: https://doi.org/10.1109/MSP.2018.2875511.
  * [8] B. Li, K. Dinesh, C. Xu, G. Sharma, and Z. Duan, “Online audio-visual source association for chamber music performances,” _Transactions of the International Society for Music Information Retrieval_ , vol. 2, no. 1, 2019.
  * [9] B. Li, K. Dinesh, Z. Duan, and G. Sharma, “See and listen: score-informed association of sound tracks to players in chamber music performance videos,” in _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2017, pp. 2906–2910, dOI: https://doi.org/10.1109/ICASSP.2017.7952688.
  * [10] B. Li, C. Xu, and Z. Duan, “Audiovisual source association for string ensembles through multi-modal vibrato analysis,” in _Proceedings of the Sound and Music Computing (SMC) Conference_ , 2017, pp. 159–166.
  * [11] H. Zhao, C. Gan, W.-C. Ma, and A. Torralba, “The sound of motions,” in _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2019, pp. 1735–1744.
  * [12] K. Dinesh, B. Li, X. Liu, Z. Duan, and G. Sharma, “Visually informed multi-pitch analysis of string ensembles,” in _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , 2017, pp. 3021–3025, dOI: https://doi.org/10.1109/ICASSP.2017.7952711.
  * [13] B. Li, K. Dinesh, G. Sharma, and Z. Duan, “Video-based vibrato detection and analysis for polyphonic string music,” in _Proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2017, pp. 123–130.
  * [14] B. Li and A. Kumar, “Query by video: Cross-modal music retrieval.” in _Proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2019, pp. 604–611.
  * [15] L. Chen, S. Srivastava, Z. Duan, and C. Xu, “Deep cross-modal audio-visual generation,” in _Proceedings of the ACM Thematic Workshops of Multimedia_ , 2017, pp. 349–357.
  * [16] B. Li, A. Maezawa, and Z. Duan, “Skeleton plays piano: online generation of pianist body movements from MIDI performance,” in _Proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2018.
  * [17] A. Mesaros and T. Virtanen, “Recognition of phonemes and words in singing,” in _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2010, pp. 2146–2149.
  * [18] A. Cadalbert, T. Landis, M. Regard, and R. E. Graves, “Singing with and without words: Hemispheric asymmetries in motor control,” _Journal of Clinical and Experimental Neuropsychology_ , vol. 16, no. 5, pp. 664–670, 1994\.
  * [19] L. Connell, Z. G. Cai, and J. Holler, “Do you see what i’m singing? visuospatial movement biases pitch perception,” _Brain and cognition_ , vol. 81, no. 1, pp. 124–130, 2013.
  * [20] N. Takahashi, N. Goswami, and Y. Mitsufuji, “Mmdenselstm: An efficient combination of convolutional and recurrent neural networks for audio source separation,” in _Proceedings of the International Workshop on Acoustic Signal Enhancement (IWAENC)_. IEEE, 2018, pp. 106–110.
  * [21] S. Vembu and S. Baumann, “Separation of vocals from polyphonic audio recordings.” in _Proceedings of the International Society for Music Information Retrieval (ISMIR)_. Citeseer, 2005, pp. 337–344.
  * [22] A. Ozerov, P. Philippe, F. Bimbot, and R. Gribonval, “Adaptation of bayesian models for single-channel source separation and its application to voice/music separation in popular songs,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 15, no. 5, pp. 1564–1578, 2007.
  * [23] P.-S. Huang, S. D. Chen, P. Smaragdis, and M. Hasegawa-Johnson, “Singing-voice separation from monaural recordings using robust principal component analysis,” in _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2012, pp. 57–60.
  * [24] Z. Rafii and B. Pardo, “A simple music/voice separation method based on the extraction of the repeating musical structure,” in _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2011, pp. 221–224.
  * [25] P. Chandna, M. Miron, J. Janer, and E. Gómez, “Monoaural audio source separation using deep convolutional neural networks,” in _Proceedings of the International Conference on Latent Variable Analysis and Signal Separation_. Springer, 2017, pp. 258–266.
  * [26] S. Uhlich, M. Porcu, F. Giron, M. Enenkl, T. Kemp, N. Takahashi, and Y. Mitsufuji, “Improving music source separation based on deep neural networks through data augmentation and network blending,” in _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2017, pp. 261–265.
  * [27] N. Takahashi, P. Agrawal, N. Goswami, and Y. Mitsufuji, “Phasenet: Discretized phase modeling with deep neural networks for audio source separation.” in _Proceedings of the International Conference on Spoken Language Processing (Interspeech)_ , 2018, pp. 2713–2717.
  * [28] W. Choi, M. Kim, J. Chung, and D. L. S. Jung, “Investigating deep neural transformations for spectrogram-based musical source separation,” _arXiv preprint arXiv:1912.02591_ , 2019.
  * [29] F. Lluis, J. Pons, and X. Serra, “End-to-end music source separation: is it possible in the waveform domain?” in _Proceedings of the International Conference on Spoken Language Processing (Interspeech)_ , 2019.
  * [30] D. Stoller, S. Ewert, and S. Dixon, “Wave-u-net: A multi-scale neural network for end-to-end audio source separation,” in _Proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2018, pp. 334–340.
  * [31] N. Takahashi and Y. Mitsufuji, “Multi-scale multi-band densenets for audio source separation,” in _Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_ , 2017, pp. 21–25.
  * [32] F.-R. Stöter, S. Uhlich, A. Liutkus, and Y. Mitsufuji, “Open-unmix - a reference implementation for music source separation,” _Journal of Open Source Software_ , 2019. [Online]. Available: https://doi.org/10.21105/joss.01667
  * [33] R. Hennequin, A. Khlif, F. Voituret, and M. Moussallam, “Spleeter: A fast and state-of-the-art music source separation tool with pre-trained models,” 2019\.
  * [34] R. Lu, Z. Duan, and C. Zhang, “Listen and look: audio–visual matching assisted speech source separation,” _IEEE Signal Processing Letters_ , vol. 25, no. 9, pp. 1315–1319, 2018.
  * [35] A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein, “Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation,” _ACM Transactions on Graphics (TOG)_ , vol. 37, no. 4, 2018, dOI: https://doi.org/10.1145/3197517.3201357.
  * [36] J.-C. Hou, S.-S. Wang, Y.-H. Lai, Y. Tsao, H.-W. Chang, and H.-M. Wang, “Audio-visual speech enhancement using multimodal deep convolutional neural network,” _IEEE Transactions on Emerging Topics in Computational Intelligence_ , 2018.
  * [37] S. Parekh, S. Essid, A. Ozerov, N. Duong, P. Perez, and G. Richard, “Motion informed audio source separation,” in _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2017, pp. 6–10, dOI: https://doi.org/10.1109/ICASSP.2017.7951787.
  * [38] H. Zhao, C. Gan, A. Rouditchenko, C. Vondrick, J. McDermott, and A. Torralba, “The sound of pixels,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , vol. 1, 2018, pp. 587–604, dOI: https://doi.org/10.1007/978-3-030-01246-5_35.
  * [39] C. Gan, D. Huang, H. Zhao, J. B. Tenenbaum, and A. Torralba, “Music gesture for visual sound separation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 10 478–10 487.
  * [40] A. Jansson, E. Humphrey, N. Montecchio, R. Bittner, A. Kumar, and T. Weyde, “Singing voice separation with deep u-net convolutional networks,” in _Proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2017.
  * [41] J.-Y. Liu and Y.-H. Yang, “Denoising auto-encoder with recurrent skip connections and residual regression for music source separation,” in _Proceedings of the IEEE International Conference on Machine Learning and Applications (ICMLA)_ , 2018, pp. 773–778.
  * [42] D. E. King, “Dlib–ml: A machine learning toolkit,” _Journal of Machine Learning Research_ , vol. 10, no. Jul., pp. 1755–1758, 2009.
  * [43] F.-R. Stöter, A. Liutkus, and N. Ito, “The 2018 signal separation evaluation campaign,” in _International Conference on Latent Variable Analysis and Signal Separation_. Springer, 2018, pp. 293–305.
  * [44] Y. Luo, Z. Chen, J. R. Hershey, J. Le Roux, and N. Mesgarani, “Deep clustering and conventional networks for music separation: Stronger together,” in _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , 2017, pp. 61–65.
  * [45] Y. Luo and N. Mesgarani, “Tasnet: time-domain audio separation network for real-time, single-channel speech separation,” in _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP))_ , 2018, pp. 696–700.
  * [46] J. S. Chung and A. Zisserman, “Lip reading in the wild,” in _Proceedings of the Asian Conference on Computer Vision_. Springer, 2016, pp. 87–103.