# Self-Supervised Learning for Personalized Speech Enhancement

Aswin Sivaraman  Minje Kim

###### Abstract

Speech enhancement systems can show improved performance by adapting the model
towards a single test-time speaker. In this personalization context, the test-
time user might only provide a small amount of noise-free speech data, likely
insufficient for traditional fully-supervised learning. One way to overcome
the lack of personal data is to transfer the model parameters from a speaker-
agnostic model to initialize the personalized model, and then to finetune the
model using the small amount of personal speech data. This baseline marginally
adapts over the scarce clean speech data. Alternatively, we propose self-
supervised methods that are designed specifically to learn personalized and
discriminative features from abundant in-the-wild noisy, but still personal
speech recordings. Our experiment shows that the proposed self-supervised
learning methods initialize personalized speech enhancement models better than
the baseline fully-supervised methods, yielding superior speech enhancement
performance. The proposed methods also result in a more robust feature set
under the real-world conditions: compressed model sizes and fewness of the
labeled data.

speech enhancement, model specialization, self-supervised learning,
contrastive learning, few-shot learning

##  1 Introduction

Speech enhancement (SE) algorithms are a vital component of many everyday
technologies such as hearing prostheses, mobile telephony, and automatic
speech recognition (Maas et al., 2012). In contrast to the more general task
of audio source separation, speech enhancement models operate on the
assumption that there is only one source of interest—a speech signal—and that
the remaining sounds are interfering background noises or unwanted filtering
effects which must be suppressed. Although algorithms designed for multiple
microphones tend to perform better, some applications limit the number of
microphones to one, making a single-channel speech enhancement challenging. In
this work we focus on the single-channel algorithms that are designed to
suppress additive interference sources.

Single-channel speech denoising methods are typically formulated as a
parametric mapping function learned in a fully supervised way; once it is
trained it is expected to generalized well to an unseen test speaker’s speech
characteristics as well as the test-time acoustic environment. To this end, a
large dataset of clean speech signals spoken by many speakers is artificially
mixed with many non-stationary noises; an optimization process estimates the
model parameters, so that the model maps the mixture signals to the estimated
clean speech source. Today, speech enhancement researchers employ deep neural
network (DNN) models, which are known to generalize well to unseen test data
if their architecture is large enough to learn a complex mapping function from
the big training dataset (Wang & Chen, 2018). We consider these models as
generalists: in order for a generalist model to account for the peculiarities
of a specific test-time speaker, one must increase both the diversity of its
training speech and noise corpora and the model capacity.

In this work, we target at a popular use case of SE which the generalist
models have not addressed efficiently: an SE system specializes in a
particular person and his or her acoustic environment as a specialist. Given
the widespread use of speech-based interfaces on embedded systems, such as an
on-device virtual assistants, many use cases fall in the category of personal
use. We posit that a personalized speech enhancement (PSE) model can focus on
the smaller problem defined by that test-time user and environment, making the
learning task easier than a generalist model on the original general-purpose
speech enhancement problem. For example, it has been shown that specialization
of single-channel SE models (e.g. based on a particular speaker gender or
noise type) can yield improvements in signal quality (Kolbæk et al., 2017).
The improved enhancement quality also leads to a model compression effect: a
smaller model can achieve similar performance as shown in (Sivaraman & Kim,
2020). Given that PSE is for personal devices, efficient inference is
critical. While these models have shown merits of specialization in general,
they did not discuss the impact of personalization to target at the test-time
speaker, which is our goal in this paper.

Another important issue in the PSE tasks is that the solution may require
private data from the users. In a straightforward transfer learning scenario,
one can achieve personalization first by pretraining a generalist model using
the ordinary large speech corpora and sound databases, and then by finetuning
it via an additional speaker- and environment- specific dataset. However, this
kind of additional data could expose the user’s privacy to the third party,
which the user may not want. In this paper, therefore, we employ a few-shot
learning (FSL) approach to the PSE task which is to handle the case that there
is only a limited amount of labeled training data available (Wang et al.,
2020a). For example, the user might only allow their device to record a few
seconds of clean speech data in order to aid personalization. Even if the PSE
model is efficient and provides optimal improvement, reliance on private user
data is a critical issue. Hence, while it is obvious that the transfer
learning framework will benefit from more speech signals of the test user, our
goal is to achieve the best possible personalization even if the clean
utterances are limited to only a few seconds.

In this work, as an alternative to straightforward transfer learning, we
propose a universal self-supervised learning (SSL) methodology for PSE. We
show that the SSL approach can better initialize the PSE model than a speaker-
agnostic SE task, especially when the clean speech for finetuning is not
enough. Based on the assumption that the noisy speech data of the target user
is more available and less private than the clean ones, we propose SSL methods
that fully utilize large amounts of noisy speech data which can be seen as
unlabeled data in the usual supervised learning setup. To this end, we propose
two SSL algorithms, pseudo speech enhancement (PseudoSE) and contrastive
mixtures (CM), that are designed to extract speaker-specific discriminative
features. Both approaches re-purpose the noisy speech data as the reference
speech source and mix additional noise sources to them to simulate an
artificial mixture. The fact that the clean test speech is not necessary
limits the performance of this model as an SE model, which is why we call it
“pseudo” SE and it requires a downstream finetuning step. However, the fact
that it learns from the very test speaker’s utterances, although they are
still noisy, the features learned via this SSL method can be still more
representative than the generic features. The CM method regularizes PseudoSE
further via a contrastive learning mechanism that works on the pairs of input
signals. For its positive pairs, a pair starts from the same noisy speech
utterance as the pseudo-source, but it is contaminated by two different noise
sources. Hence, after performing PseudoSE on each of the pair, we regularize
the reconstructed reference sources by maximizing their agreement. Conversely,
as for the negative pairs, we prepare two different reference sources, whose
PseudoSE results should disagree from each other. Our goal is to show that
this additional contrastive learning objective can improve the performance of
PseudoSE, which is already a better pretraining option than a speaker-agnostic
generalist model.

Our main contributions are summarized as follows:

  * •

We formulate the personalized speech enhancement problem as a few-shot
learning task to address the privacy-preservation issues.

  * •

We propose a self-supervised pseudo speech enhancement pretraining procedure;
it minimizes the amount of speaker-specific clean utterances used during
finetuning by leveraging speaker-specific noisy utterances, which we posit are
relatively more abundant.

  * •

We propose a self-supervised contrastive mixtures pretraining procedure; this
method may be viewed as a regularized version of pseudo speech enhancement, as
it re-contextualizes the abundant noisy speech through positive and negative
pairs. This method expands upon the contrastive learning formulation “SimCLR”
which was proposed for self-supervised image classification (Chen et al.,
2020).

  * •

We show that PSE can achieve model compression without a loss of enhancement
performance.

##  2 Personalized Speech Enhancement

Due to the FSL constraint that the the available clean speech data from the
test speaker may be scarce, on the order of seconds, initializing the
enhancement model with random weights will make it quickly over-fit to the
scarce data. In this section, we overview three potential methods for
initializing (i.e. pretraining) a personalized speech enhancement model. Table
1 summarizes the notations and datasets used.

Table 1: Glossary of datasets used and their respective notations. Details on dataset usage are covered throughout Sections 2 and 3.3. Corpus |  Subset |  Notation |  Description
---|---|---|---
Librispeech |  train-clean-100 |  $\mathbb{G}$ |  Speaker-agnostic noise-free utterances, used for pretraining.
|  $\mathbb{S}_{\text{te}}$ |  Speaker-specific noise-free utterances, unavailable to models.
|  $\mathbb{S}_{\text{ft}}$ |  Small subset of $\mathbb{S}_{\text{te}}$ representing clean speech provided by the specific speaker, used for model fine-tuning.
MUSAN |  free-sound |  $\mathbb{N}_{\text{tr}}$ |  Large set of seen noises, used for pretraining and fine-tuning.
|  sound-bible |  $\mathbb{N}_{\text{te}}$ |  Small set of unseen noises, used for evaluation.
DEMAND |  ch01 |  $\mathbb{N}_{\text{pm}}$ |  Noises which contaminate $\mathbb{S}_{\text{te}}$; the self-supervised methods treat the noisy version $\tilde{\mathbb{S}}_{\text{te}}$ as reference signals.
|  |  |

###  2.1 Multi-Speaker SE Initialization

We first train a speaker-agnostic model; this requires a large corpus of many
anonymous speakers $\mathbb{G}$ as well as a large corpus of various non-
stationary noises $\mathbb{N}_{\text{tr}}$. A set of artificial mixture
signals $\bm{x}$ can be made by selecting random utterances
$\bm{s}\in\mathbb{G}$ and noises $\bm{n}\in\mathbb{N}_{\text{tr}}$ and summing
the signals, i.e. $\bm{x}=\bm{s}+\bm{n}$ (with a different mixing ratio which
we omit for brevity). The universal model learns a parametric mapping function
$g(\cdot)$ such that $g(\bm{x};{\mathcal{W}}_{g})=\bm{y}\approx\bm{s}$, where
the estimate $\bm{y}$ approximate the training target $\bm{s}$ and
${\mathcal{W}}_{g}$ stands for the model parameters. After this pretraining
process initializes the model parameters ${\mathcal{W}}_{g}$, finetuning
updates them further using another kind of input signals $\bm{x}$ which is the
mixture of the small amount of user-specific clean speech data
$\mathbb{S}_{\text{ft}}$ with the training noises $\mathbb{N}_{\text{tr}}$.
This finetuning process is still a fully-supervised setup—the model has access
to the corresponding clean speech sources $\bm{s}$ which compose each mixture
$\bm{x}$. During both pretraining and finetuning, the model’s objective is to
minimize the discrepancy ${\mathcal{E}}$ between the denoised estimate signals
$\bm{y}$ and the ground-truth clean source signals $\bm{s}$. The baseline
procedure for training a PSE model can be summarized as follows:

Mixture: $\displaystyle\bm{x}=\bm{s}+\bm{n};\quad\bm{s}\in\mathbb{G},\hskip
5.69054pt\bm{n}\in\mathbb{N}_{\text{tr}}$ (1) Pretraining:
$\displaystyle\operatornamewithlimits{\arg\min}_{{\mathcal{W}}_{g}}{\mathcal{E}}(\bm{y}=g(\bm{x};{\mathcal{W}}_{g})||\bm{s})$
(2) $\displaystyle\begin{split}\text{Finetuning:
}&\operatornamewithlimits{\arg\min}_{{\mathcal{W}}_{g}}{\mathcal{E}}(\bm{y}=g(\bm{x};{\mathcal{W}}_{g})||\bm{s});\\\
&\qquad\qquad\bm{x}\in\mathbb{S}_{\text{ft}}\times\mathbb{N}_{\text{tr}},\hskip
5.69054pt\bm{s}\in\mathbb{S}_{\text{ft}}\end{split}$ (3)

The success of this multi-speaker initialization scheme, which adapts a
generalist model into a specialist model, relies on the speaker-specific clean
utterance set $\mathbb{S}_{\text{ft}}$ being large enough to represent the
test speaker. We assume that this is not the case due to the aforementioned
goal of maximizing the test speaker’s privacy. In an FSL context, the naïve
adaptation of speaker-agnostic parameters may lead to two failure modes:
either under- or over-fitting.

###  2.2 Single-Speaker Pseudo SE Initialization

We postulate that a personal smart device might have access to many noisy
recordings of the test-time speaker $\mathbb{\tilde{S}}_{\text{te}}$, likely
much more than the clean set—i.e.
$|\mathbb{\tilde{S}}_{\text{te}}|\gg|\mathbb{S}_{\text{ft}}|$. Similarly to
the other applications, where unlabeled data are abundant, exploiting them is
our primary goal in doing SSL.

The noisy utterances $\tilde{\bm{s}}\in\mathbb{\tilde{S}}_{\text{te}}$ must
have been already contaminated by some noise signal
$\bm{m}\in\mathbb{N}_{\text{pm}}$ which we call premix noise, i.e.
$\tilde{\bm{s}}=\bm{s}+\bm{m}$. Based on our assumptions, neither
$\mathbb{S}_{\text{te}}$ or $\mathbb{N}_{\text{pm}}$ is known to the denoising
system. However, it may be reasonable to assume that this premixture noise
source’s loudness varies over time, depending on the user’s test-time acoustic
condition. Hence, if the premixture signal-to-noise ratio (SNR) is reasonably
high, the noisy speech utterances $\mathbb{\tilde{S}}_{\text{te}}$ can work as
a pseudo speech source.

In this regard, we pretrain a denoising model using the large noisy set
$\mathbb{\tilde{S}}_{\text{te}}$ the target reference then finetune it using
the small clean set $\mathbb{S}_{\text{ft}}$. We refer to the pretraining part
as pseudo speech enhancement (PseudoSE) because it uses $\tilde{\bm{s}}$ as
the training target in place of the ground-truth clean speech. PseudoSE is to
offer a SSL alternative to the ordinary pretraining using the generalist.

Since PseudoSE treats the premixture noisy data
$\mathbb{\tilde{S}}_{\text{te}}$ as the target, we prepare mixtures
$\tilde{\bm{x}}$ through noise injection (Yin et al., 2015; Liang et al.,
2018) by combining $\tilde{\bm{s}}$ with additional deformation signals
$\bm{n}\in\mathbb{N}_{\text{tr}}$, i.e.
$\tilde{\bm{x}}=\tilde{\bm{s}}+\bm{n}$. The model learns a PseudoSE mapping
function $f$ such that
$f(\tilde{\bm{x}})=\tilde{\bm{y}}\approx\tilde{\bm{s}}$. Note that $f$ can
only recover the noisy utterance $\tilde{\bm{s}}$ as if it was a clean source,
hence $f\neq g$. Ultimately, $f$ is finetuned into $g$ by using the limited
clean speech data $\mathbb{S}_{\text{ft}}$. The whole transfer learning
process is summarized as follows:

Premixture:
$\displaystyle\tilde{\bm{s}}=\bm{s}+\bm{m};\quad\bm{s}\in\mathbb{S}_{\text{te}},\hskip
5.69054pt\bm{m}\in\mathbb{N}_{\text{pm}}$ (4) Noise Injection:
$\displaystyle\tilde{\bm{x}}=\tilde{\bm{s}}+\bm{n};\quad\bm{n}\in\mathbb{N}_{\text{tr}}$
(5) Pretraining:
$\displaystyle\operatornamewithlimits{\arg\min}_{{\mathcal{W}}_{f}}{\mathcal{E}}(\tilde{\bm{y}}=f(\tilde{\bm{x}};{\mathcal{W}}_{f})||\tilde{\bm{s}})$
(6) Initialization: $\displaystyle{\mathcal{W}}_{g}={\mathcal{W}}_{f}$ (7)
$\displaystyle\begin{split}\text{Finetuning:
}&\operatornamewithlimits{\arg\min}_{{\mathcal{W}}_{g}}{\mathcal{E}}(\bm{y}=g(\bm{x};{\mathcal{W}}_{g})||\bm{s});\\\
&\qquad\qquad\bm{x}\in\mathbb{S}_{\text{ft}}\times\mathbb{N}_{\text{tr}},\hskip
5.69054pt\bm{s}\in\mathbb{S}_{\text{ft}}\end{split}$ (8)

The PseudoSE method serves as a pretraining mechanism that makes the best use
of the potentially abundant noisy speech of the same test speaker. Although
the method does not exactly solve the desired speaker-specific SE problem, we
conjecture that the features learned in this way can still represent the
peculiarity of the test speaker than the baseline multi-speaker initialization
scheme. Previous work has shown that mixture signals can work as prediction
targets in other source separation context (Wisdom et al., 2020); we extend
this insight directly towards the FSL scenario of personalized speech
enhancement.

###  2.3 Single-Speaker Contrastive Mixtures Initialization

(a) $\bm{s}^{\oplus}$ (b) $\bm{m}^{\oplus}$

(c) $\tilde{\bm{s}}^{\oplus}\\!=\\!\bm{s}^{\oplus}\\!+\\!\bm{m}^{\oplus}$ (d)
$\bm{n}_{1}^{\oplus}$

(e) $\bm{n}_{2}^{\oplus}$ (f)
$\tilde{\bm{x}}_{1}^{\oplus}\\!=\\!\tilde{\bm{s}}^{\oplus}\\!+\\!\bm{n}_{1}^{\oplus}$

(g)
$\tilde{\bm{x}}_{2}^{\oplus}\\!=\\!\tilde{\bm{s}}^{\oplus}\\!+\\!\bm{n}_{2}^{\oplus}$

Figure 1: Example audio spectrograms for all of the components that create a
positive input pair ($\tilde{\bm{x}}_{1}^{\oplus}$,
$\tilde{\bm{x}}_{2}^{\oplus}$).

(a) $\bm{s}_{1}^{\ominus}$ (b) $\bm{s}_{2}^{\ominus}$

(c)
$\tilde{\bm{s}}^{\ominus}_{1}\\!=\\!\bm{s}_{1}^{\ominus}\\!+\\!\bm{m}_{1}^{\ominus}$
(d)
$\tilde{\bm{s}}_{2}^{\ominus}\\!=\\!\bm{s}_{2}^{\ominus}\\!+\\!\bm{m}_{2}^{\ominus}$

(e) $\bm{n}^{\ominus}$ (f)
$\tilde{\bm{x}}_{1}^{\ominus}\\!=\\!\tilde{\bm{s}}_{1}^{\ominus}\\!+\\!\bm{n}^{\ominus}$

(g)
$\tilde{\bm{x}}_{2}^{\ominus}\\!=\\!\tilde{\bm{s}}_{2}^{\ominus}\\!+\\!\bm{n}^{\ominus}$

Figure 2: Example audio spectrograms for all of the components that create a
negative input pair ($\tilde{\bm{x}}_{1}^{\ominus}$,
$\tilde{\bm{x}}_{2}^{\ominus}$).

(a)  (b)

Figure 3: The proposed framework for contrastive mixtures. Solid lines
indicate signal path while dashed lines show loss terms.

We hypothesize that the quality of the pretraining procedure greatly impacts
how the denoising model will personalize when the clean speech for finetuning
is not enough. Even if the premixed noisy speech set
$\mathbb{\tilde{S}}_{\text{te}}$ and the deformation noise set
$\mathbb{N}_{\text{tr}}$ are large, the quality of the features learned
through PseudoSE are bounded by how noisy $\mathbb{\tilde{S}}_{\text{te}}$
really is. Our proposed contrastive mixtures (CM) pretraining procedure
addresses this by employing a pairwise contrastive learning mechanism.

In the CM framework, the denoising model $f(\cdot)$ pretrains over pairs of
mixtures ($\tilde{\bm{x}}_{1}$, $\tilde{\bm{x}}_{2}$) and outputs pseudo-
cleaned estimates ($\tilde{\bm{y}}_{1}$, $\tilde{\bm{y}}_{2}$). We create two
kinds of mixture pairs, positive and negative, which are illustrated in Figure
3 with example spectrograms shown in Figures 2 and 2.

In a positive pair, both examples ($\tilde{\bm{x}}_{1}^{\oplus}$,
$\tilde{\bm{x}}_{2}^{\oplus}$) share the same premixture source
$\tilde{\bm{s}}^{\oplus}$, but are differently deformed; that is, the mixing
process makes the input pair slightly disimilar. Therefore, in addition
maximizing the similarities between estimates and source
($\tilde{\bm{y}}_{1}^{\oplus}$ to $\tilde{\bm{s}}^{\oplus}$ and
$\tilde{\bm{y}}_{2}^{\oplus}$ to $\tilde{\bm{s}}^{\oplus}$), the model
$f(\cdot)$ must also satisfy the contrastive objective based on the fact that
$\tilde{\bm{y}}_{1}^{\oplus}$ and $\tilde{\bm{y}}_{2}^{\oplus}$ stemmed from
the same pseudo-source. We express these objectives as a positive pair loss
function $\mathcal{L}_{p}$ in the following form:

$\mathcal{L}_{p}={\mathcal{E}}(\tilde{\bm{s}}^{\oplus}||\tilde{\bm{y}}_{1}^{\oplus})+{\mathcal{E}}(\tilde{\bm{s}}^{\oplus}||\tilde{\bm{y}}_{2}^{\oplus})+\lambda_{p}\big{[}{\mathcal{E}}(\tilde{\bm{y}}_{1}^{\oplus}||\tilde{\bm{y}}_{2}^{\oplus})\big{]},$
(9)

where $\lambda_{p}$ scales the contribution of the contrastive loss term.

In a negative pair, each mixture is made from a different noisy source
($\tilde{\bm{s}}_{1}^{\ominus}\neq\tilde{\bm{s}}_{2}^{\ominus}$), but with a
shared deformation, i.e.,
$\tilde{\bm{x}}_{1}^{\ominus}=\tilde{\bm{s}}_{1}^{\ominus}+{\bm{n}}^{\ominus}$
and
$\tilde{\bm{x}}_{2}^{\ominus}=\tilde{\bm{s}}_{2}^{\ominus}+{\bm{n}}^{\ominus}$;
in other words, the negative pair mixing process makes the inputs more similar
to one another. Accordingly, in addition to the source-wise denoising
objectives, the dissimilarity between the estimates
$\tilde{\bm{y}}_{1}^{\ominus}$ and $\tilde{\bm{y}}_{2}^{\ominus}$ must be
enforced. We express these objectives as a negative pair loss function
$\mathcal{L}_{n}$ in the following form:

$\begin{split}\mathcal{L}_{n}&={\mathcal{E}}(\tilde{\bm{s}}_{1}^{\ominus}||\tilde{\bm{y}}_{1}^{\ominus})+{\mathcal{E}}(\tilde{\bm{s}}_{2}^{\ominus}||\tilde{\bm{y}}_{2}^{\ominus})\\\
&+\lambda_{n}\big{[}{\mathcal{E}}(\tilde{\bm{s}}_{1}^{\ominus}||\tilde{\bm{s}}_{2}^{\ominus})-{\mathcal{E}}(\tilde{\bm{y}}_{1}^{\ominus}||\tilde{\bm{y}}_{2}^{\ominus})\big{]}^{2},\end{split}$
(10)

where $\lambda_{n}$ controls the contribution of the contrastive loss term.
Note that this term is defined as a squared error between the difference of
the target pseudo-sources and that of the reconstruction pair. The squared
error associates the difference between $\tilde{\bm{y}}_{1}^{\ominus}$ and
$\tilde{\bm{y}}_{2}^{\ominus}$ with that of the original difference between
the pseudo-sources $\tilde{\bm{s}}_{1}^{\ominus}$ and
$\tilde{\bm{s}}_{2}^{\ominus}$, rather than enforcing an unbounded difference.

Both $\mathcal{L}_{p}$ and $\mathcal{L}_{n}$ consist of two terms: the source-
to-estimate errors and the estimate-to-estimate errors, where the former works
as the main SE loss while the latter regularizes through the proposed
contrastive mixtures process. The model ultimately minimizes the sum of these
two losses, i.e.
$\mathcal{L}_{\text{CM}}=\sum_{i=1}^{T}\mathcal{L}_{p}(t)+\sum_{i=1}^{T}\mathcal{L}_{n}(t)$
where $T$ is the number of positive or negative pairs within the batch and
$\mathcal{L}_{p}(t)$ and $\mathcal{L}_{n}(t)$ denote the loss for the $t$-th
pair. If the regularizing contrastive terms are omitted, i.e. $\lambda_{p}=0$
and $\lambda_{n}=0$, it can be shown that $\mathcal{L}_{\text{CM}}$ reduces to
Eq. (6). We empirically found that scaling the estimate-to-estimate errors
with $\lambda_{p}=0.05$ and $\lambda_{n}=0.0001$ minimized training loss.

The proposed CM approach differs from the SimCLR model in multiple regards:
(a) it uses a more advanced data augmentation process to mimic the noisy
speech mixture generation process, i.e. sophisticated non-stationary noise
injection; (b) the introduction of the negative pairs reflects the source
separation concept underlying our SE problem and yields a more discriminative
feature than a positive pair only; and, (c) our interpretation of having the
contrastive loss as a regularizer enriches the total loss function allowing
the traditional SE loss to prevent trivial solution: very similar
$\tilde{\bm{y}}_{1}^{\ominus}$ and $\tilde{\bm{y}}_{2}^{\ominus}$ that do not
recover the pseudo-sources.

##  3 Experiment Setup

###  3.1 Model Architectures

We evaluate the four initialization schemes using two types of neural networks
that are with smaller architectures to prove our efficiency-related claim. The
first model consists of a two-layer gated recurrent unit network (GRU) (Cho et
al., 2014) followed by a dense layer. We set the number of GRU hidden units to
be either $64$, $128$, and $256$, which exponentially varies the total number
of model parameters. The GRU-based model is trained to perform speech
denoising through time-frequency ratio mask estimation (Narayanan & Wang,
2013). The second model is the recently proposed ConvTasNet (CTN) speech
separation system with the smallest suggested configuration, i.e. $N=128$
filters (Luo & Mesgarani, 2019). The number of total parameters in each of
these four architectures is listed in Table 2.

The GRU-based models process batches of $128$ mixtures, while the ConvTasNet
model batches contain $8$ mixtures. Because the CM pretraining is done
pairwise, batches are built using $64$ and $4$ pairs respectively. A batch of
mixtures $\bm{x}$ is made using $1\text{\,}\sec$ long, randomly offset clips
from the appropriate speech and noise sets. Inputs to the ConvTasNet model
remain in the time-domain; however, inputs to the GRU-based models are
converted to the time-frequency domain using the short-time Fourier transform
(STFT) with a Hann window of $1024$ samples and $75\text{\,}\%$ overlap. The
estimated ratio mask is applied onto the input mixture spectrogram to obtain a
denoised estimate spectrogram.

All parameter updates are made using the Adam optimizer (Kingma & Ba, 2015)
with various learning rates specific to each system configuration in
comparison. These learning rates, in addition to the CM regularizer weights
$\lambda_{p}$ and $\lambda_{n}$, were identified using Tune, which optimized
for validation loss (Liaw et al., 2018).

###  3.2 Metrics

In a study surveying various speech enhancement loss functions, it was found
that functions which are based on negative scale-invariant signal-to-
distortion ratio (SI-SDR) proved to be optimal for neural network training
(Roux et al., 2018; Kolbæk et al., 2020). SI-SDR augments conventional SDR
with the inclusion of a scaling factor $\alpha$ which maintains orthogonality
of the residual vector between some target signal $\bm{v}$ and an estimate
signal $\hat{\bm{v}}$. It is defined as:

$\operatorname{SI-
SDR}(\bm{v}||\hat{\bm{v}})=10\log_{10}\left[\frac{\sum_{t}(\alpha
v_{t})^{2}}{\sum_{t}(\alpha v_{t}-\hat{v}_{t})^{2}}\right],$ (11)

where $\alpha=\frac{\hat{\bm{v}}^{\top}\bm{v}}{\bm{v}^{\top}\bm{v}}$. We use
this metric to evaluate denoising performance during validation and testing.
However, our models use the scale-dependent variant for the training loss to
account for cases of improper output scaling. It is defined similar to Eq.
(11) but omits the $\alpha$ value in the denominator. Because SD-SDR is a
similarity metric between two signals, it must be negated to be used as a loss
function, i.e. ${\mathcal{E}}=-\operatorname{SD-SDR}(\bm{v}||\hat{\bm{v}})$.

###  3.3 Data Preparation

Our experiments combine audio data from three public corpora: Librispeech
(Panayotov et al., 2015), MUSAN (Snyder et al., 2015), and DEMAND (Thiemann et
al., 2013).

We denote two partitions for MUSAN: $\mathbb{N}_{\text{tr}}$ as training
noises from the free-sound subset, and $\mathbb{N}_{\text{te}}$ for evaluation
noises from the sound-bible subset. We use the ch01 subset from DEMAND
exclusively when adding premixture noise which we denote as
$\mathbb{N}_{\text{pm}}$.

Of the $251$ speakers in the LibriSpeech train-clean-100 subset, we designate
$20$ to be test-time users, i.e. each test speaker $k$ has their own utterance
set $\mathbb{S}^{(k)}$ (we will omit speaker index for brevity). The remaining
speakers make up speaker-agnostic set $\mathbb{G}$. Each speaker-specific set
contains $|\mathbb{S}_{\text{ft}}|<$180\text{\,}\sec$$, from which we sample a
few seconds of clean speech utterances for finetuning, and
$|\mathbb{S}_{\text{te}}|<$1320\text{\,}\sec$$, the larger subset representing
the entirety of the test speaker’s speech, made inaccessible to the models.
Instead, we use its noisy version to construct the speaker-specific premixture
signals, i.e.
$\tilde{\mathbb{S}}_{\text{te}}=\mathbb{S}_{\text{te}}\times\mathbb{N}_{\text{pm}}$
(Eq. (4)).

All competing models are finetuned using artificial mixtures
$\bm{x}\in\mathbb{S}_{\text{ft}}\times\mathbb{N}_{\text{tr}}$, and their
denoising performance is evaluated on unseen mixtures
$\bm{x}\in\mathbb{S}_{\text{te}}\times\mathbb{N}_{\text{te}}$. The mixing SNR
is randomly chosen between $-5\text{\,}\mathrm{dB}$ and
$5\text{\,}\mathrm{dB}$ per sample in a training batch (Eq. (1)). We vary the
size of the clean speech set $\mathbb{S}_{\text{ft}}$ to be either $0$, $3$,
$5$, $10$, $30$, or $60\text{\,}\sec$ in total duration; this controls the
fewness of our posed few-shot learning scenario.

For pretraining, the baseline model is trained using mixtures made from the
speaker-agnostic set, i.e. $\bm{x}\in\mathbb{G}\times\mathbb{N}_{\text{tr}}$.
Through multi-speaker initialization, the model processes $\bm{x}$ and
generates clean speech estimates $\hat{\bm{s}}$ (Eq. (2)). Our proposed
pretraining schemes PseudoSE and CM procedures differ in that they rely on the
speaker-specific noisy utterance set $\tilde{\mathbb{S}}_{\text{te}}$. Signals
for noise injection are sampled from $\mathbb{N}_{\text{tr}}$ (Eq. (5)). As
conjectured in Section 2.3, how degraded $\mathbb{\tilde{S}}_{\text{tr}}$ is
will impact the model’s ability to learn speaker-specific features. We
investigate this by trying $5$ and $10\text{\,}\mathrm{dB}$ premixture SNRs
for $\tilde{\mathbb{S}}_{\text{te}}$.

##  4 Results & Discussion

Table 2: Mean SI-SDR improvement (in dB) with standard deviation in
parentheses. Averages are computed over twenty test speakers from the
LibriSpeech corpus. Columns compare the amount of clean speech from the test
speaker used to finetune the model ($|\mathbb{S}_{\text{ft}}|$). Rows compare
the three discussed PSE model initialization methods. Premixture SNR applies
to the two proposed methods which involve training on the large set of noisy
speech from the test speaker ($\tilde{\mathbb{S}}_{\text{tr}}$). The four
tested architectures are arranged by model size, i.e. in order of number of
parameters. Note that only the Multi-Speaker models with $0\text{\,}\sec$ of
finetuning are non-personalized.

Initialization Method |  Premixture SNR (dB) |  $\operatorname{GRU:64\times 2}\hskip 5.69054pt$ (169k params.)
---|---|---
|  |  $0\text{\,}\sec$ |  $3\text{\,}\sec$ |  $5\text{\,}\sec$ |  $10\text{\,}\sec$ |  $30\text{\,}\sec$ |  $60\text{\,}\sec$
Multi-Speaker |  n/a |  8.32 (0.710) |  8.60 (0.813) |  8.68 (0.809) |  8.98 (0.781) |  9.36 (0.699) |  9.58 (0.737)
PseudoSE |  5 |  7.63 (0.660) |  8.70 (0.785) |  8.82 (0.764) |  9.02 (0.810) |  9.25 (0.806) |  9.37 (0.789)
|  10 |  8.51 (0.632) |  9.15 (0.730) |  9.24 (0.744) |  9.37 (0.749) |  9.50 (0.725) |  9.59 (0.694)
Contrastive Mixtures |  5 |  7.95 (0.612) |  8.81 (0.760) |  8.96 (0.712) |  9.16 (0.770) |  9.36 (0.802) |  9.50 (0.760)
|  10 |  8.78 (0.637) |  9.20 (0.721) |  9.30 (0.730) |  9.46 (0.760) |  9.62 (0.791) |  9.70 (0.729)
|  |  |  |  |  |  |

Initialization Method |  Premixture SNR (dB) |  $\operatorname{GRU:128\times 2}\hskip 5.69054pt$ (412k params.)
---|---|---
|  |  $0\text{\,}\sec$ |  $3\text{\,}\sec$ |  $5\text{\,}\sec$ |  $10\text{\,}\sec$ |  $30\text{\,}\sec$ |  $60\text{\,}\sec$
Multi-Speaker |  n/a |  8.75 (0.616) |  9.44 (0.723) |  9.52 (0.738) |  9.80 (0.727) |  10.13 (0.690) |  10.14 (0.698)
PseudoSE |  5 |  8.09 (0.599) |  9.44 (0.685) |  9.56 (0.656) |  9.71 (0.669) |  9.93 (0.650) |  9.96 (0.684)
|  10 |  9.27 (0.649) |  10.01 (0.687) |  10.04 (0.670) |  10.13 (0.697) |  10.25 (0.689) |  10.28 (0.694)
Contrastive Mixtures |  5 |  8.46 (0.623) |  9.59 (0.718) |  9.72 (0.708) |  9.91 (0.725) |  10.14 (0.724) |  10.23 (0.723)
|  10 |  9.51 (0.581) |  10.06 (0.701) |  10.14 (0.683) |  10.24 (0.720) |  10.38 (0.725) |  10.46 (0.715)
|  |  |  |  |  |  |

Initialization Method |  Premixture SNR (dB) |  $\operatorname{GRU:256\times 2}\hskip 5.69054pt$ (1.1m params.)
---|---|---
|  |  $0\text{\,}\sec$ |  $3\text{\,}\sec$ |  $5\text{\,}\sec$ |  $10\text{\,}\sec$ |  $30\text{\,}\sec$ |  $60\text{\,}\sec$
Multi-Speaker |  n/a |  9.75 (0.687) |  10.02 (0.758) |  10.09 (0.754) |  10.36 (0.791) |  10.80 (0.784) |  10.91 (0.773)
PseudoSE |  5 |  8.60 (0.604) |  10.18 (0.744) |  10.30 (0.718) |  10.45 (0.728) |  10.71 (0.719) |  10.77 (0.713)
|  10 |  9.76 (0.495) |  10.77 (0.657) |  10.81 (0.635) |  10.92 (0.645) |  11.08 (0.643) |  11.13 (0.647)
Contrastive Mixtures |  5 |  8.91 (0.621) |  8.93 (0.703) |  10.49 (0.710) |  10.68 (0.730) |  10.90 (0.736) |  11.08 (0.732)
|  10 |  10.14 (0.503) |  10.96 (0.579) |  11.02 (0.592) |  11.13 (0.609) |  11.30 (0.617) |  11.41 (0.607)
|  |  |  |  |  |  |

Initialization Method |  Premixture SNR (dB) |  $\operatorname{ConvTasNet}\hskip 5.69054pt$ (1.4m params.)
---|---|---
|  |  $0\text{\,}\sec$ |  $3\text{\,}\sec$ |  $5\text{\,}\sec$ |  $10\text{\,}\sec$ |  $30\text{\,}\sec$ |  $60\text{\,}\sec$
Multi-Speaker |  n/a |  10.08 (0.667) |  10.70 (0.779) |  10.87 (0.730) |  11.15 (0.758) |  11.52 (0.734) |  11.70 (0.744)
PseudoSE |  5 |  8.30 (0.531) |  11.98 (0.597) |  12.12 (0.621) |  12.32 (0.657) |  12.62 (0.666) |  12.76 (0.661)
|  10 |  10.84 (0.497) |  12.60 (0.722) |  12.70 (0.742) |  12.83 (0.768) |  13.03 (0.792) |  13.16 (0.777)
Contrastive Mixtures |  5 |  8.34 (0.621) |  11.91 (0.715) |  12.05 (0.749) |  12.25 (0.788) |  12.53 (0.797) |  12.67 (0.787)
|  10 |  10.80 (0.614) |  12.49 (0.663) |  12.59 (0.682) |  12.74 (0.698) |  12.95 (0.715) |  13.10 (0.695)
|  |  |  |  |  |  |

Figure 4: SI-SDR improvement averaged over $100$ unseen mixtures from each
test speaker, plotted with respect to how much finetuning each personalized
model undergoes. The inner x-axis ticks correspond to different sizes of the
clean speech set $\mathbb{S}_{\text{ft}}$: $0$, $3$, $5$, $10$, $30$, and
$60\text{\,}\sec$ respectively.

In this section, we detail the key takeaways from the results of our
experiments. Table 2 compares averaged SI-SDR improvement across all test-time
speakers, segmented in terms of model architecture and initialization method.
Figure 4 shows the impact of finetuning on clean speech with regards to the
different initialization schemes, with subplot columns for individual test-
time speaker and rows for each model architecture.

First, from Table 2 we can see that the finetuned models using
$|\mathbb{S}_{\text{ft}}\leq 3\text{sec}$ consistently improves the
performance over their corresponding pretrained model (0 sec). It is a general
trend found in all competing models: the more the clean speech is the better
the finetuning performance. It aligns with our conjecture that PSE can be
achieved via a transfer learning process and its performance is positively
correlated with the amount of personal clean speech.

We see that across all architectures, a PSE model pretrained using either
proposed self-supervised method universally outperforms models which are
pretrained using the baseline fully-supervised method, contingent on the
unlabeled dataset $\tilde{\mathbb{S}}_{\text{te}}$ having an premixture SNR of
at least $10\text{\,}\mathrm{dB}$. We also found that the proposed methods
quickly achieve high performance with only a small amount of clean speech,
earlier than the baseline. For example, for the GRU 256$\times$2 architecture,
CM improves the pretrained model from 10.14 to 10.96 dB (8.1% increase) while
the baseline improves from 9.75 to 10.02 (2.8%) when both are using only 3
seconds of clean speech for finetuning. Or, as for ConvTasNet, the PseudoSE’s
improvement is by 16.2% while the baseline’s only 6.2%. However, the
robustness to the few-shot learning scenario is maximized when the models are
larger as also reported in SimCLR (Chen et al., 2020).

We note that CM consistently outperforms PseudoSE in all the GRU-based models.
The gap between the two models explain the merit of the additional contrastive
loss that works as a regularizer. This did not hold true with ConvTasNet,
which saw PseudoSE slightly outperforming CM. Our future work may investigate
whether alternate choices for $\lambda_{p}$ and $\lambda_{n}$ are needed for
ConvTasNet.

Models pretrained using $5\text{\,}\mathrm{dB}$ premixtures usually
underperforms the 10 dB mixtures, meaning premixture signals’ quality matters
for a successful SSL feature training. However, the proposed models pretrained
on the $5\text{\,}\mathrm{dB}$ premixtures remain competitive with the multi-
speaker-based models.

As discussed in Section 1, a model with a small number of parameters achieving
comparable performance to a model made up of many more parameters suggests
that the smaller model is more advantageous on devices with limited resources.
There is an inherent tradeoff between model compression and our overarching
FSL objective, i.e., we want to minimize the use of clean speech, while the
more of it results in a better performance. But overall, the usage of a self-
supervised initialization method allows smaller PSE models to exceed the
performance of larger baseline models. For example, a CM-pretrained
$\operatorname{GRU:128\times 2}$ model using $10\text{\,}\mathrm{dB}$
premixtures requires only $5\text{\,}\sec$ of clean speech from the target
user before it can outperform a baseline-pretrained
$\operatorname{ConvTasNet}$ model—an almost $70\text{\,}\mathrm{\char
37\relax}$ reduction in model parameters is made possible using self-
supervised pretraining and a modest amount of clean speech from the test-time
speaker.

With the individual speaker’s PSE results shown in Figure 4, we can
immediately observe how the baseline model—which during pretraining is
speaker-agnostic—sometimes struggles to adapt to certain test speakers; two
notable cases of this are speaker #83 and #200. The proposed SSL
initialization methods remedy these case successfuly. There are few anomalous
cases where the baseline model does adapt competitively or even superlatively
compared to the SSL methods when the model is the smallest (GRU 64 $\times$
2), but the average trend shows the self-supervised models universally
personalizing. Matching our initial hypothesis, the generalist model does not
comprehensively generalize to this $20$ speaker test set, showing that the
peculiarities of certain speakers do require an efficient specialist model.
This trend is more obvious when the clean set is limited, suggesting that the
SSL methods are a more robust solution in the FSL context.

##  5 Related Work

In this section, we address prior works which motivated our investigation of
the personalized speech enhancement task.

#### Model Adaptation

Prior works have shown that adaptive speech denoising autoencoders (Kim &
Smaragdis, 2015) and modularized SE systems (Kim, 2017) can increase
performance through test-time adaptation at the cost of increased model
complexity. Ensemble networks have been used to compress the test-time
adaptated SE models using a no-shot non-personalized context (Sivaraman & Kim,
2020). There is also prior research on speaker-aware SE systems in the single-
channel (Wang et al., 2018) and multi-channel (Žmolíková et al., 2017;
Delcroix et al., 2018) formulations. In all these works, a generalist SE model
is conditioned using a speaker-identifying feature vector. As discussed in
Section 1, the single-speaker enhancement task is a subset of the broader
speaker-agnostic enhancement task; therefore, conditioning a generalist model
does not exploit the benefit of reducing model capacity.

#### Pretext Tasks

This paper formulate the PSE task as a unique opportunity to apply self-
supervised learning (Schmidhuber, 1990) which has gained significant traction
in image and video processing research communities (Wang & Gupta, 2015; Veit
et al., 2017). Self-supervised models are pretrained on an auxiliary “pretext
task”, learning meaningful features out of the unlabeled data which are then
reused for the downstream task. For visual object classification, various
pretext tasks have shown promising performances, e.g., colorization and
rotation (Dosovitskiy et al., 2015; Gidaris et al., 2018; Zhang et al., 2015),
by predicting relative positions of local patches (Doersch et al., 2015;
Noroozi & Favaro, 2016), and by sharing network weights between supervised and
self-supervised models (Zhai et al., 2019; Sun et al., 2019). Stochastic
masking has also been shown to add robustness to the learned self-supervised
features; this was proven using denoising autoencoders (Vincent et al., 2008)
and bidirectional encoder representations from transformer (BERT) (Devlin et
al., 2019). Among the many studied pretext tasks, contrastive learning has
gained recent popularity as it directly competes with features learned through
fully-supervised means (van den Oord et al., 2018; He et al., 2020; Chen et
al., 2020). Our CM pretraining extends the insights from these works.

#### Data Augmentation

Data augmentation is frequently seen in audio signal processing research. With
musical instrument source separation, one can generate vast amounts of
incoherent unrealistic mixtures from random musical stems to augment training
data (Manilow et al., 2019). Towards speech separation, a model may train in
an unsupervised permutation-invariant way by indefinitely mixing mixtures and
separating them into an arbitrary number of sources minimizing signal-to-noise
ratio (SNR) loss (Wisdom et al., 2020). A recent work applies self-supervised
learning directly towards speech enhancement—the authors train an autoencoder
on unlabeled noisy data, coupling its parameters with another autoencoder
pretrained on clean data (Wang et al., 2020b). In all prior works, the exact
amount of data augmentation is left unquantified—more specifically, minimizing
the required amount of speaker-specific clean speech data is not addressed. In
contrast to these works, for personalized speech enhancement, we constraint
our finetuning procedure to a small fixed amount of clean speech data from the
test-time user.

##  6 Conclusion

In this work, we presented two self-supervised learning approaches towards
personalized speech enhancement, highlighting their ability to quickly adapt
using only a few seconds of test-user clean speech data. Compared with the
naïve transfer learning approach, as well with an intermediate self-supervised
alternative, the contrastive mixtures process succeeds in the few-shot problem
thanks to added regularization. Our results suggests that speaker-
discriminative features can be found even in noisy recordings, promoting the
use of robust privacy-preserving speech processing applications.

##  7 Acknowledgement

This material is based upon work supported by the National Science Foundation
under Grant No. 2046963.

## References

  * Chen et al. (2020) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G.  A Simple Framework for Contrastive Learning of Visual Representations.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2020.
  * Cho et al. (2014) Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_ , 2014.
  * Delcroix et al. (2018) Delcroix, M., Žmolíková, K., Kinoshita, K., Ogawa, A., and Nakatani, T.  Single channel target speaker extraction and recognition with speaker beam.  In _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , 2018.
  * Devlin et al. (2019) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT)_ , pp. 4171–4186, 2019\.
  * Doersch et al. (2015) Doersch, C., Gupta, A., and Efros, A. A.  Unsupervised Visual Representation Learning by Context Prediction.  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2015.
  * Dosovitskiy et al. (2015) Dosovitskiy, A., Fischer, P., Springenberg, J. T., Riedmiller, M., and Brox, T.  Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks.  _arXiv preprint arXiv:1406.6909_ , 2015.
  * Gidaris et al. (2018) Gidaris, S., Singh, P., and Komodakis, N.  Unsupervised Representation Learning by Predicting Image Rotations.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2018.
  * He et al. (2020) He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R.  Momentum Contrast for Unsupervised Visual Representation Learning.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Kim (2017) Kim, M.  Collaborative deep learning for speech enhancement: A run-time model selection method using autoencoders.  In _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , 2017.
  * Kim & Smaragdis (2015) Kim, M. and Smaragdis, P.  Adaptive denoising autoencoders: A fine-tuning scheme to learn from test mixtures.  In _Proceedings of the International Conference on Latent Variable Analysis and Signal Separation (LVA/ICA)_ , August 2015.
  * Kingma & Ba (2015) Kingma, D. and Ba, J.  Adam: A method for stochastic optimization.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2015.
  * Kolbæk et al. (2017) Kolbæk, M., Tan, Z. H., and Jensen, J.  Speech intelligibility potential of general and specialized deep neural network based speech enhancement systems.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 25(1):153–167, Jan 2017.
  * Kolbæk et al. (2020) Kolbæk, M., Tan, Z.-H., Jensen, S. H., and Jensen, J.  On Loss Functions for Supervised Monaural Time-Domain Speech Enhancement.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:825–838, 2020.
  * Liang et al. (2018) Liang, D., Huang, Z., and Lipton, Z. C.  Learning noise-invariant representations for robust speech recognition.  In _2018 IEEE Spoken Language Technology Workshop (SLT)_ , pp. 56–63. IEEE, 2018.
  * Liaw et al. (2018) Liaw, R., Liang, E., Nishihara, R., Moritz, P., Gonzalez, J. E., and Stoica, I.  Tune: A Research Platform for Distributed Model Selection and Training.  _arXiv preprint arXiv:1807.05118_ , 2018.
  * Luo & Mesgarani (2019) Luo, Y. and Mesgarani, N.  Conv-TasNet: Surpassing ideal time–frequency magnitude masking for speech separation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 27(8):1256–1266, 2019.
  * Maas et al. (2012) Maas, A. L., Le, Q. V., O’Neil, T. M., Vinyals, O., Nguyen, P., and Ng, A. Y.  Recurrent neural networks for noise reduction in robust ASR.  In _Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech)_ , pp. 22–25, 2012.
  * Manilow et al. (2019) Manilow, E., Wichern, G., Seetharaman, P., and Roux, J. L.  Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantity.  In _Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_ , pp. 45–49, 2019.
  * Narayanan & Wang (2013) Narayanan, A. and Wang, D. L.  Ideal ratio mask estimation using deep neural networks for robust speech recognition.  In _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , pp. 7092–7096, May 2013\.
  * Noroozi & Favaro (2016) Noroozi, M. and Favaro, P.  Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2016.
  * Panayotov et al. (2015) Panayotov, V., Chen, G., Povey, D., and Khudanpur, S.  Librispeech: An ASR corpus based on public domain audio books.  In _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , pp. 5206–5210. IEEE, 2015\.
  * Roux et al. (2018) Roux, J. L., Wisdom, S., Erdogan, H., and Hershey, J. R.  SDR - half-baked or well done?  _arXiv preprint arXiv:1811.02508_ , 2018.
  * Schmidhuber (1990) Schmidhuber, J.  Making the world differentiable: On using self-supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non-stationary environments.  1990\.
  * Sivaraman & Kim (2020) Sivaraman, A. and Kim, M.  Sparse Mixture of Local Experts for Efficient Speech Enhancement.  In _Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech)_ , 2020.
  * Snyder et al. (2015) Snyder, D., Chen, G., and Povey, D.  MUSAN: A Music, Speech, and Noise Corpus.  _arXiv preprint arXiv:1510.08484_ , 2015.
  * Sun et al. (2019) Sun, Y., Tzeng, E., Darrell, T., and Efros, A. A.  Unsupervised Domain Adaptation through Self-Supervision.  _arXiv preprint arXiv:1909.11825_ , 2019.
  * Thiemann et al. (2013) Thiemann, J., Ito, N., and Vincent, E.  The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings.  _Journal of the Acoustical Society of America_ , 133(5):3591–3591, 2013.
  * van den Oord et al. (2018) van den Oord, A., Li, Y., and Vinyals, O.  Representation Learning with Contrastive Predictive Coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Veit et al. (2017) Veit, A., Alldrin, N., Chechik, G., Krasin, I., Gupta, A., and Belongie, S.  Learning From Noisy Large-Scale Datasets With Minimal Supervision.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 839–847, 2017.
  * Vincent et al. (2008) Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A.  Extracting and composing robust features with denoising autoencoders.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , pp. 1096–1103, 2008.
  * Wang & Chen (2018) Wang, D. L. and Chen, J.  Supervised speech separation based on deep learning: An overview.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 26(10):1702–1726, 2018.
  * Wang et al. (2018) Wang, Q., Muckenhirn, H., Wilson, K., Sridhar, P., Wu, Z., Hershey, J., Saurous, R. A., Weiss, R. J., Jia, Y., and Moreno, I. L.  Voicefilter: Targeted voice separation by speaker-conditioned spectrogram masking.  _arXiv preprint arXiv:1810.04826_ , 2018.
  * Wang & Gupta (2015) Wang, X. and Gupta, A.  Unsupervised Learning of Visual Representations using Videos.  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , pp. 2794–2802, 2015.
  * Wang et al. (2020a) Wang, Y., Yao, Q., Kwok, J. T., and Ni, L. M.  Generalizing from a Few Examples: A Survey on Few-shot Learning.  _ACM Computing Surveys_ , 53(3), 2020a.  ISSN 0360-0300.  doi: 10.1145/3386252.  URL https://doi.org/10.1145/3386252.
  * Wang et al. (2020b) Wang, Y.-C., Venkataramani, S., and Smaragdis, P.  Self-supervised Learning for Speech Enhancement.  _arXiv preprint arXiv:2006.10388_ , 2020b.
  * Wisdom et al. (2020) Wisdom, S., Tzinis, E., Erdogan, H., Weiss, R. J., Wilson, K., and Hershey, J. R.  Unsupervised Sound Separation Using Mixtures of Mixtures.  _arXiv preprint arXiv:2006.12701_ , 2020.
  * Yin et al. (2015) Yin, S., Liu, C., Zhang, Z., Lin, Y., Wang, D., Tejedor, J., Zheng, T. F., and Li, Y.  Noisy training for deep neural networks in speech recognition.  _EURASIP Journal on Audio, Speech, and Music Processing_ , 2015(1):2, 2015.
  * Zhai et al. (2019) Zhai, X., A. Oliver, A. K., and Beyer, L.  S4L: Self-Supervised Semi-Supervised Learning.  _arXiv preprint arXiv:1905.03670_ , 2019.
  * Zhang et al. (2015) Zhang, R., Isola, P., and Efros, A. A.  Colorful Image Colorization.  _arXiv preprint arXiv:1603.08511_ , 2015.
  * Žmolíková et al. (2017) Žmolíková, K., Delcroix, M., Kinoshita, K., Higuchi, T., Ogawa, A., and Nakatani, T.  Speaker-aware neural network based beamformer for speaker extraction in speech mixtures.  In _Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech)_ , 2017.