# CCC-wav2vec 2.0: Clustering aided Cross Contrastive Self-supervised learning
of speech representations

###### Abstract

While Self-Supervised Learning has helped reap the benefit of the scale from
the available unlabeled data, the learning paradigms are continously being
bettered. We present a new pre-training strategy named ccc-wav2vec 2.0, which
uses clustering and an augmentation based cross-contrastive loss as its self-
supervised objective. Through the clustering module we scale down the
influence of those negative examples that are highly similar to the positive.
The Cross-Contrastive loss is computed between the encoder output of the
original sample and the quantizer output of its augmentation, and vice-versa,
bringing robustness to the pre-training strategy. ccc-wav2vec 2.0 achieves
upto 15.6% and 12.7% relative WER improvement over the baseline wav2vec 2.0 on
the test-clean and test-other sets respectively of LibriSpeech, without the
use of any language model. The proposed method also achieves upto 14.9%
relative WER improvement over the baseline wav2vec 2.0, when fine-tuned on
Switchboard data.

Index Terms—  self-supervised learning, automatic speech recognition, domain
adaptation, telephone speech

##  1 Introduction

The use of SSL to learn high-level representations from unlabeled data has
received much attention in the last few years in the domains of Computer
Vision (CV) [1], Natural Language Processing (NLP), [2] and Spoken Language
Processing (SLP) [3]. Though much progress has been made in CV and NLP, self-
supervised learning for SLP has been relatively understudied. In a majority of
prior work, SSL for SLP solves a variant of Masked Acoustic Modeling (MAM),
either through instance discrimination using contrastive learning [3, 4], or
masked prediction [5, 6]. There is, however, much potential to improve the
existing self-supervised tasks for better representation learning. In this
paper, we introduce two improvements over the standard wav2vec 2.0, which help
learn better and more robust speech representations through SSL.

Though SSL has been seen to benefit from scale [7], the performance of SSL for
speech with limited unlabeled data needs further attention. Considering that
even unlabeled data in most languages is limited in real-world scenarios, SSL
algorithms that can learn useful representations even in low-resource regimes
are the need of the hour [7]. Data augmentation has proven to be an effective
strategy for supervised learning setups [8, 9] when the amount of labeled data
is limited. Very recently, self-supervised learning in speech has also shown
to benefit from data augmentation [10, 11]. [12] showed that data augmentation
benefits Contrastive Predictive Coding [13] when a limited amount of unlabeled
data is available. [11] shows how introducing specific augmentations makes
their speech recognition model more robust to far-field, multi-talker noisy
environments. Contrastive learning to learn representations that maximize the
agreement between differently augmented views of the same data is a
methodology predominant in CV and has achieved state-of-the-art results in
many applications [1, 14]. Inspired by this, we add an auxiliary task to the
standard wav2vec 2.0 Contrastive Learning task, wherein we contrast the
anchors with negatives generated from an augmented sample and vice-versa. This
makes the model more robust to augmentations, which in turn helps learn better
representations.

Surprisingly, the choice of negative samples in a Contrastive Learning for SLP
setup has drawn much less attention in the literature. Very often, given an
“anchor” point $x$, “negative samples” $x^{-}_{i}$s are randomly sampled from
the training data, independent of how informative they may be for the learned
representation. Though very recently CV has seen growing attention to this
line of research [15, 16, 17], to the best of our knowledge, there is no
existing work in speech despite many state-of-the-art systems solving a
contrastive learning task for self-supervised speech representation learning
[3, 4].

We look at Masked Acoustic Modelling (MAM) from the lens of language modeling
and hypothesize that, similar to the Contrastive Learning setups in NLP [18],
it is important to sample negatives that are semantically different. The need
amplifies in speech representation learning, where the negative sampling
strategy becomes all the more important due to the quasi-stationary nature of
speech which makes several consecutive speech frames correspond to the same
phone or sound. Moreover, Contrastive Learning models that use instance
discrimination as a pre-training task tend to fall into over clustering [19]
during training. Thus, we hypothesize that negative examples mapped very close
to the anchor in terms of their similarity might represent the same phone or
class. Considering such negative examples would contradict the representation
learning task, which should primarily focus on discriminating between sounds
or phones.

To arrive at more informative negatives for the contrastive loss, we propose
to cluster our potential negative examples and diminish the effect of those
negatives in the loss computation that fall into the same cluster as the
positive. Simply put, this process identifies the weak non-informative
negatives from our population and reduces their impact on the loss
computation.

We also demonstrate the robustness of the proposed approach through tasks such
as Domain Adaptation and zero-shot decoding on the Switchboard [20] and Wall
Street Journal (WSJ) [21] datasets, respectively. To summarize, our primary
contributions are as follows:

  * •

We introduce an augmentation of the original sample and use its
representations to add an auxiliary Cross-Contrastive loss to the existing
contrastive loss in wav2vec 2.0.

  * •

We demonstrate the usefulness of a clustering module to segregate the negative
examples and thereby control the effect of the weak non-informative negative
examples in the contrastive learning task.

  * •

Combining the above two modules leads to the development of ccc-wav2vec 2.0, a
robust pre-training approach that consistently outperforms wav2vec 2.0 in
tasks such as ASR, Domain Adaptation, and zero-shot decoding.

Our code and models are publicly available on
GitHub111https://github.com/Speech-Lab-IITM/CCC-wav2vec-2.0 222Correspondence
to Vasista Sai Lodagala<EMAIL_ADDRESS>
##  2 Related Work

SSL for speech representation learning has been prevalent in the form of MAM.
Most of the MAM approaches introduced in literature aim to either predict the
class of the masked entity using a classification objective as in [5, 6], or
reconstruct the original frame as in [22, 23], or enforce similarity between
the prediction of the network for the masked frame and a quantized
representation of the original masked frame by solving a Contrastive Learning
task as in [3]. On the other hand, some of them propose to solve two of these
tasks simultaneously [24, 4, 25, 26].

###  2.1 Negative Sampling

Contrastive Learning has been observed to dominate self-supervised speech
representation learning methodologies in various forms [3, 4], constantly
achieving new State of the Art (SOTA) results on a variety of SLP tasks [27].
However, it is hard to find works that discuss efficient negative mining for
Contrastive Learning in SLP. Alternatively, this line of research has seen
great success in CV [17, 15, 16, 28]. Also, this idea has been successfully
applied in metric learning, where most of the works [29, 30] observe that it
is helpful to use negative examples that are difficult to be discriminated
from the current embedding. [13] was one of the first works to analyze the
effect of efficient negative sampling, wherein they observed a drop in
performance when negatives were not mined from the same speaker. To the best
of our knowledge, this is the first work to design the Contrastive Learning
task for speech SSL to explicitly control the choice of negatives.

###  2.2 Data Augmentation

The usefulness of data augmentations for robust SLP has been explored
extensively, primarily on a supervised learning setup [8, 9]. A wide range of
SLP tasks like ASR [31, 32], Speaker Identification [33] etc., have been seen
to benefit from data augmentations. Aspects of low-resource learning [31, 32],
far-field and noisy environment recognition [34, 35] have primarily seen to
benefit the most with data augmentation. Very recently, the benefits of data
augmentation have been explored in SSL-based speech representation learning
[10, 11], where the former focuses on low-resource and the latter on improving
ASR in far-field and noisy environments. In a recent work based on wav2vec
2.0, [11] proposes a Multi-Variant Consistency based objective wherein
multiple augmented versions of the same audio sample are created. The original
audio sample is discarded, and a contrastive loss between the multiple
augmented versions is computed. Our proposed approach differs from this work
in the following ways: 1) We retain the original audio sample and use the
cross-contrastive loss with the augmentation as an auxiliary loss in addition
to the original wav2vec 2.0 objective. 2) In our loss computation, the effect
of the various negative examples is controlled by the clustering module,
leading to an informed contrast with the “anchor” in the contrastive loss.

##  3 Methodology

In the following subsections, we elaborate on the Cross-Contrastive setup and
the Clustering module, both of which are key components of the proposed ccc-
wav2vec 2.0. Finally, we present how these two components are integrated to
form ccc-wav2vec 2.0.

Fig. 1: Illustration of the ccc-wav2vec 2.0 framework

###  3.1 Cross-Contrastive Learning

Deciphering speech and sound in noisy environments is not a challenging task
for humans. However, the same cannot be expected of SSL models, unless they
have been trained for it. In order to bring robustness to the pre-training
approach, we tap into the augmentations of speech samples.

Given an audio sample $X$, we apply an augmentation over it using the
torchaudio-augmentations library [36], to get $X^{{}^{\prime}}$ as the
augmented sample. We pass both $X$ and $X^{{}^{\prime}}$ through the wav2vec
2.0 model to get the quantized representations $Q_{t}$, $Q_{t}^{{}^{\prime}}$
and the context representations $C$, $C^{{}^{\prime}}$. Unless specified
otherwise, the quantized representations are computed only for unmasked latent
representations corresponding to the masked regions. The subscript $t$,
indicates the masked time steps. An important point to note is that the
original and the augmented sample share the same set of masked indices (time-
steps) to facilitate the computation of the cross-contrastive loss. In the
case of wav2vec 2.0, the standard contrastive loss is defined over each masked
time step $t$ as follows:

$L_{c}=-log\frac{exp(sim(c_{t},q_{t})/\kappa)}{\sum_{\tilde{q}\sim
Q_{t}}exp(sim(c_{t},\tilde{q})/\kappa)}$ (1)

where, $c_{t}\in C$, $q_{t}\in Q_{t}$ and
$sim(\mathbf{m},\mathbf{n})=\mathbf{m}^{T}\mathbf{n}/\|\mathbf{m}\|\|\mathbf{n}\|$
is used to compute the cosine similarity between context and quantized
representations. $\kappa$ represents the temperature parameter. It is to be
noted that we do not use all of the frames in $Q_{t}$ as negative examples.
The symbol $\sim$ in the denominator represents the specific number of
examples sampled from $Q_{t}$, to function as distractors or negative
examples.

Given the representations from the augmentations, we define the following loss
terms:

$L_{cross}=-log\frac{exp(sim(c_{t},q_{t}^{{}^{\prime}})/\kappa)}{\sum_{\tilde{q}^{{}^{\prime}}\sim
Q_{t}^{{}^{\prime}}}exp(sim(c_{t},\tilde{q}^{{}^{\prime}})/\kappa)}$ (2)

and

$L_{cross^{{}^{\prime}}}=-log\frac{exp(sim(c_{t}^{{}^{\prime}},q_{t})/\kappa)}{\sum_{\tilde{q}\sim
Q_{t}}exp(sim(c_{t}^{{}^{\prime}},\tilde{q})/\kappa)}$ (3)

where, $c_{t}^{{}^{\prime}}\in C^{{}^{\prime}}$, $q_{t}^{{}^{\prime}}\in
Q_{t}^{{}^{\prime}}$. Essentially, the equations 2 and 3 compute the
contrastive loss between the context representations of the original sample
and the quantized representations of its augmentation, and vice-versa over the
masked time steps.

Finally, we define the overall cross-contrastive loss $L_{cc}$ as follows:

$L_{cc}=\alpha L_{c}+\beta L_{cross}+\gamma L_{cross^{{}^{\prime}}}$ (4)

where, $\alpha$, $\beta$ and $\gamma$ are the scalar hyper-parameters that
would determine the importance of each of the individual loss terms.

Evidently, the performance of models pre-trained with the $L_{cc}$ loss also
depends on the choice of augmentations. The different choices of augmentations
are described in Section 5. It is to be noted that, where ever we mention any
variant of contrastive loss, the diversity loss in [3] is also added to the
final loss computation.

###  3.2 Clustering Module

As mentioned earlier, the function of the clustering module is to identify the
set of weakly informative negatives and diminish their effect in the
computation of the contrastive loss.

####  3.2.1 Need for Clustering

As the negative examples are drawn from the quantized representations $Q_{t}$,
we perform clustering over the entire set of frames in $Q_{t}$. Given that the
frames in $Q_{t}$, are discrete representations computed via product
quantization, the need for clustering over $Q_{t}$ may not be clear
immediately. As discussed earlier, the primary motivation behind clustering
over quantized representations is that language, spoken or written, is
composed of discrete tokens. Negative examples drawn randomly from the
population might represent the same class of tokens. In speech, this need is
amplified due to its quasi-stationary nature, where several consecutive speech
frames could represent the same phone or sound. We mitigate this through
negative example elimination and try to get better and more informative
negatives by using a “closeness” measure (in our case, cosine similarity) with
the help of clustering.

Another important reason for clustering over quantized representations is
that, by default, the product quantization stage of wav2vec 2.0 uses two
codebooks, each of size 320. So, the total number of discrete representations
possible with this configuration are $320\times 320=102400$. Given such a huge
universe of possible discrete representations (coming from product
quantization), for any given audio sample $X$, it would be uncommon to find
quantized representations $q_{t}$ in $Q_{t}$, having the same discrete
representation for similar frames. So, to identify such definite similarities,
we use a clustering module based on cosine distance. One clear example of such
an occurrence is in the case of padded frames. Since the padding is added to
samples to create uniformly sized mini-batches during training, the quantized
representations of such padded frames can easily be identified and grouped by
a clustering module

####  3.2.2 Cluster Factor (CF)

We perform k-means clustering [37] on $Q_{t}$ with cosine distance as the
metric. The maximum number of iterations is limited to 100. To ensure that the
computation speed is not compromised with the introduction of the clustering
module, we use the fast-pytorch-kmeans library to perform the clustering on
GPUs. The only hyper-parameter to be fixed now is the number of clusters. In
our view, this is a hyper-parameter that cannot be fixed easily. This is
because speech samples are usually of varied durations. So, padding is
introduced in each mini-batch of speech samples to keep the same number of
frames per audio. However, the number of frames per audio varies drastically
across mini-batches. On the LibriSpeech dataset, we have observed that this
number varies between 60 to 380. Given this variance in the number of frames
per audio, it is difficult to fix the number of clusters. Doing so could lead
to sub-optimal clustering results.

To overcome this hurdle, we introduce a new parameter called cluster factor
$(CF)$. Because the clustering is performed separately in each mini-batch, we
can fix the number of clusters based on the number of frames per audio in the
mini-batch. Given a mini-batch of audio samples, let the number of frames per
audio be $NF$, after padding. Then, the number of clusters in the clustering
module, for each audio in this mini-batch is given by $ceil(NF/CF)$. It is to
be noted that when $CF=1$, no clustering happens, and there would be no
alteration to the standard wav2vec 2.0 architecture.

####  3.2.3 Scale Factor (SF)

Once the clustering has been performed on $Q_{t}$, we identify the cluster to
which the positive sample $q_{t}$ belongs. Let us denote the cluster to which
$q_{t}$ belongs, by $K$. Our interest then is to control the “influence” of
those negative examples that fall into the same cluster $K$. What is meant by
“influence” here is the effect they have on the computation of contrastive
loss as defined in equation 1. As cosine similarity is the metric used over
the negative examples, controlling the influence would effectively mean
controlling the cosine similarity value. We scale down the cosine similarity
of the negative examples in $K$ with $c_{t}$, by a scaling factor $SF$. Let
the sampled set of negatives be represented by the set $Q^{*}$. That is,
$Q^{*}=\\{\tilde{q}\sim Q_{t}\\}$. Let the samples in $Q^{*}$ be denoted by
$q$. The formula for contrastive loss would then assume the form,

$L_{c}=-log\frac{e^{(sim(c_{t},q_{t})/\kappa)}}{\displaystyle\sum_{q\in
K}e^{(sim(c_{t},q).SF/\kappa)}+\displaystyle\sum_{q\not\in
K}e^{(sim(c_{t},q)/\kappa)}}$ (5)

From the construction of equation 5, it is clear that the influence of the
negatives sharing the same cluster as the positive example is controlled by
the scalar $SF$. When $SF=1$, equation 5 would be the same as the standard
contrastive loss as defined in equation 1. However, when $SF=-\infty$, all
such negative examples falling in the same cluster as the positive are
completely discarded in the loss computation. In our experiments other than
the baseline wav2vec 2.0, we vary $SF$ between $-\infty$ and $0.5$. The
rationale behind choosing values for $SF$ other than $-\infty$ for certain
experiments is as follows: The negative examples in $K$ could sometimes be the
hard-negatives [15], as opposed to the common assumption of them being weak
non-informative negatives. To account for such possibilities, we experiment
with a range of values for $SF$. Models pre-trained using the contrastive loss
in equation 5 are presented in section 5, for different choices of $CF$ and
$SF$.

###  3.3 ccc-wav2vec 2.0

As illustrated in Fig.1, the cross-contrastive setup from section 3.1, and the
clustering module from section 3.2 are brought together in the design of ccc-
wav2vec 2.0. Given the availability of two sets of quantized representations
from the original and augmented samples, we have the choice to cluster these
representations after pooling all of them.

Once the clustering module clusters the quantized representations from both
the samples (original and augmented), it can be seen from Fig.1 that the
scaling factor $SF$ is applied to those negative examples belonging to the
same cluster as the positive. The clusters can be identified from the shade of
color, that the quantized representations bear. For example, $SF$ has been
applied on $q_{5}$ as it belongs to the same cluster as the positive example
$q_{3}$. Similarly, $SF$ has been applied on $q_{1}^{{}^{\prime}}$ as it
belongs to the same cluster as the positive example $q_{3}^{{}^{\prime}}$.
However, in Fig.1, $SF$ has not been applied on $q_{4}^{{}^{\prime}}$ though
it belongs to the same cluster as $q_{3}^{{}^{\prime}}$. This is because,
though $q_{4}^{{}^{\prime}}$ is a potential negative example, it has not been
sampled in the negative sampling process. As it does not feature in the loss
computation, no scaling factor is applied to it.

The loss computation for ccc-wav2vec 2.0 would be the same as that mentioned
in the equations 1,2,3 and 4 of section 3.1, but with the inclusion of the
scaling factor $SF$ in each of those equations, as detailed in section 3.2
using equation 5. Models pre-trained using the ccc-wav2vec 2.0 architecture
are presented in section 5, for different configuration choices.

##  4 Experimental Setup

###  4.1 Pre-training

All the models mentioned in tables 1,2 and 3 are wav2vec 2.0 BASE models pre-
trained on the 360-hour split of the LibriSpeech dataset [38] using the
fairseq toolkit[39]. Owing to the compute resource constraints, the entire 960
hours couldn’t be used for pre-training. The models have been pre-trained for
26200 updates, which corresponds to 100 epochs over LibriSpeech-360h, on 4
A-100 GPUs. The number of warmup updates have been set to 19650 and the
polynomial decay learning rate scheduler has been used. Both the baseline and
the proposed SSL model have the same number of model parameters. Rest of the
parameters follow standard configurations made available through [3, 39].

###  4.2 Fine-tuning

Fine-tuning the pre-trained wav2vec 2.0 models is performed by adding a
randomly initialized output layer on top of the Transformer to predict
characters [3]. CTC loss has been used during the fine-tuning stage.

The pre-trained models from Tables 1, 2 and 3 are fine-tuned for 36400 updates
or 300 epochs on the LibriSpeech-100h split, on 4 A-100 GPUs.

We also fine-tune these pre-trained models for 11000 updates or 610 epochs, on
the 30-hour split of the Switchboard dataset. Fine-tuning the models on
Switchboard data which is telephonic speech (out-of-domain), would showcase
the robustness of ccc-wav2vec 2.0’s pre-training approach.

Table 1: Effect of different augmentations Model |  | dev | test
---|---|---|---
|  | clean | other | clean | other
Baseline wav2vec 2.0 |  | 12.3 | 30.6 | 12.8 | 31.6
Augmentation I |  | 11.1 | 28.5 | 11.5 | 29.6
Augmentation II (*) |  | 11.5 | 28.8 | 12.1 | 30.0
Augmentation II |  | 11.1 | 28.1 | 11.5 | 29.2
Table 2: Effect of different Clustering choices Model |  | dev | test
---|---|---|---
|  | clean | other | clean | other
Baseline wav2vec 2.0 |  | 12.3 | 30.6 | 12.8 | 31.6
CF (8), SF ($-\infty$) |  | 11.3 | 29.7 | 11.9 | 30.3
CF (8), SF (0.1) |  | 12.1 | 30.2 | 12.6 | 31.5
CF (8), SF (0.3) |  | 11.6 | 29.6 | 12.1 | 30.7
CF (8), SF (0.5) |  | 11.3 | 29.6 | 11.9 | 30.5
CF (16), SF ($-\infty$) |  | 11.5 | 29.5 | 11.9 | 30.5
CF (16), SF (0.1) |  | 12.5 | 30.5 | 12.9 | 31.4
CF (16), SF (0.3) |  | 11.4 | 29.2 | 11.7 | 30.3
CF (16), SF (0.5) |  | 12.0 | 29.7 | 12.4 | 31.3
CF (24), SF ($-\infty$) |  | 11.4 | 29.3 | 12.0 | 30.6
CF (24), SF (0.1) |  | 11.8 | 29.7 | 12.3 | 30.7
CF (24), SF (0.3) |  | 11.8 | 29.8 | 12.2 | 31.3
CF (24), SF (0.5) |  | 11.6 | 30.0 | 12.3 | 31.0
Table 3: ccc-wav2vec 2.0 performance (WER) over different test sets Model |  | dev |  | test |  | WSJ |  | Switchboard
---|---|---|---|---|---|---|---|---
|  | clean | other |  | clean | other |  | dev93 | eval93 |  | Dev
Baseline wav2vec 2.0 |  | 12.3 | 30.6 |  | 12.8 | 31.6 |  | 31.2 | 31.7 |  | 34.9
Augmentation II |  | 11.1 | 28.1 |  | 11.6 | 29.2 |  | 29.7 | 29.3 |  | 31.8
CF (16), SF (0.3) |  | 11.4 | 29.2 |  | 11.7 | 30.3 |  | 29.9 | 29.0 |  | 33.2
CCC - CF(8), SF(0.3) |  | 10.8 | 27.7 |  | 11.1 | 28.4 |  | 29.7 | 28.5 |  | 31.4
CCC - CF(8), SF(0.3) - pooled |  | 10.5 | 27.1 |  | 10.9 | 28.2 |  | 29.6 | 28.6 |  | 30.6
CCC - CF(8), SF(0.5) |  | 10.7 | 27.3 |  | 11.1 | 28.3 |  | 29.4 | 28.3 |  | 31.1
CCC - CF(8), SF(0.5) - pooled |  | 10.6 | 27.3 |  | 10.9 | 28.3 |  | 29.3 | 28.4 |  | 30.5
CCC - CF(16), SF(0.3) |  | 10.7 | 26.9 |  | 11.3 | 28.0 |  | 29.1 | 28.8 |  | 29.9
CCC - CF(16), SF(0.3) - pooled |  | 10.4 | 26.7 |  | 10.8 | 27.6 |  | 28.9 | 28.3 |  | 29.7
CCC - CF(16), SF(0.5) |  | 10.5 | 27.1 |  | 10.9 | 28.1 |  | 29.0 | 28.3 |  | 30.4
CCC - CF(16), SF(0.5) - pooled |  | 10.5 | 27.3 |  | 10.9 | 28.0 |  | 29.2 | 28.3 |  | 30.9
Table 4: Performance of ccc-wav2vec 2.0 pre-trained on LibriSpeech-960h over several downstream tasks from SUPERB [27] Model | Score | KS | IC | PR | ASR | ER | QbE | SF-F1 | SF-CER | SID | SV | SD | SE-STOI | SE-PESQ | SS | ST
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ccc-wav2vec2.0 BASE | 1259 | 96.7 | 96.5 | 5.95 | 6.3 | 64.2 | 6.73 | 88.1 | 24.3 | 72.8 | 5.61 | 4.27 | 94.9 | 3.06 | 10.86 | 16.2
wav2vec2.0 BASE | 674 | 96.2 | 92.4 | 5.74 | 6.43 | 63.4 | 2.33 | 88.3 | 24.8 | 75.2 | 6.02 | 6.08 | 93.9 | 2.55 | 9.77 | 14.8

##  5 Results and Analysis

The models from Table 1 have been pre-trained using the loss in equation 4.
For the baseline wav2vec 2.0, the $\alpha$, $\beta$ and $\gamma$ parameters in
equation 4 have been set to $1$, $0$ and $0$ respectively. Augmentation I in
Table 1 refers to randomly cropping 25% of the original audio sample and
replacing the cropped region with zeroes. Augmentation II, on the other hand,
is a combination of 3 different data augmentations. With a probability of
$0.6$, additive noise is added to the audio sample at a random signal-to-noise
ratio (SNR) between 3 and 15dB. Following this, with a probability of $0.7$,
the audio signal is convolved with a random Reverberation Impulse Response
(RIR). Finally, with a probability of $0.8$, background noise has been added
from random noise samples from the noise set of the MUSAN corpus[40], at a
random signal-to-noise ratio (SNR) between 0 and 15dB. The choice of this set
of augmentations for Augmentation II comes from the work of [41], which uses
data augmentations in a supervised setting. The Augmentation I and
Augmentation II models in Table 1, have been pre-trained using the loss
mentioned in equation 4, with $\alpha$, $\beta$ and $\gamma$ set to $1$, $0.5$
and $0.5$ respectively, with the cross-contrastive loss aiding the contrastive
loss. However, the Augmentation II (*) model has $\alpha$, $\beta$ and
$\gamma$ set to $0$, $1$ and $1$ respectively. This experiment has been
carried out to demonstrate that, the cross-contrastive loss aiding the
contrastive loss would lead to better results compared to having only the
cross-contrastive loss.

The models from Table 2 have been pre-trained using the loss in equation 5.
Experiments in Table 2 have been carried out to demonstrate the impact of the
clustering module. Therefore, no augmentations are involved for the
experiments in Table 2. The Cluster Factor $(CF)$ as defined in section 3.2.2,
and the Scale Factor $(SF)$ as defined in section 3.2.3, have been varied to
arrive at the best possible clustering configuration. The different choices
for $CF$ and $SF$ can be found in Table 2. We find that having a cluster
factor of 16, and a scale factor of 0.3, results in the best average
performance among the different choices made for $CF$ and $SF$. The fact that
$SF=0.3$ out-performs $SF=-\infty$, for $CF=16$, goes on to prove the presence
of hard-negatives, among the weakly non-informative negatives.

Finally, the models from Table 3, prefixed with CCC, have been pre-trained
using the loss in the equation 4. However, with the introduction of the
clustering module, each loss term in equation 4, has the scaling factor $SF$
included. For models in Table 3, the suffix “pooled” indicates that the
quantized representations from the original sample and its augmentation are
clustered together. Given the performance of different augmentations in Table
1, we choose Augmentation II, for all our experiments involving pre-training
of ccc-wav2vec 2.0.

The best configuration of ccc-wav2vec 2.0 from Table 3, when fine-tuned on the
LibriSpeech-100h split, achieves up to 15.4%, 12.7% and 15.6%, 12.7% relative
WER improvement compared to wav2vec 2.0, over the dev-clean, dev-other and
test-clean, test-other sets of the LibriSpeech dataset respectively. The same
model achieves up to 7.3% and 10.7% relative WER improvement compared to
wav2vec 2.0 over the dev93 and eval93 sets of the WSJ dataset. This shows the
efficiency of our pre-training approach over unseen data. Also, the robustness
of the pre-training approach is evident from its domain adaptation to the
Switchboard data. When fine-tuned on the 30-hour subset of the Switchboard
data, the model achieves up to 14.9% relative WER improvement compared to
wav2vec 2.0.

SUPERB evaluation: The BASE variant of ccc-wav2vec 2.0 has been pre-trained on
LibriSpeech-960h and was evaluated over the several downstream speech tasks
presented over SUPERB [27]. The model is ranked 6 over the Challenge public
set of the benchmark, outperforming the wav2vec 2.0 BASE model which has been
ranked 12 in the same leaderboard.

##  6 Conclusion and Future Work

In this paper, we introduce ccc-wav2vec 2.0, a novel SSL-based pre-training
approach based on wav2vec 2.0, that improves on the original wav2vec 2.0
discrimination task. Our approach consistently outperforms the wav2vec 2.0
architecture while training the same number of parameters. As a part of future
work, we would like to explore techniques to sample negatives that make the
instance discrimination task harder to solve and also explore new data
augmentation techniques.

## References

  * [1] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton,  “A simple framework for contrastive learning of visual representations,”  in ICML 2020, pp. 1597–1607.
  * [2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina N. Toutanova,  “Bert: Pre-training of deep bidirectional transformers for language understanding,”  2018\.
  * [3] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  NeurIPS 2020, pp. 12449–12460.
  * [4] Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, et al.,  “Unispeech-sat: Universal speech representation learning with speaker aware pre-training,”  in IEEE ICASSP 2022, pp. 6152–6156.
  * [5] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [6] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei,  “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”  IEEE Journal of Selected Topics in Signal Processing, pp. 1–14, 2022.
  * [7] Awni Hannun,  “The history of speech recognition to the year 2030,”  arXiv preprint arXiv:2108.00084, 2021.
  * [8] Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur,  “Audio augmentation for speech recognition,”  in Sixteenth annual conference of the ISCA, 2015.
  * [9] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al.,  “Deep speech 2: End-to-end speech recognition in english and mandarin,”  in ICML 2016, pp. 173–182.
  * [10] Anuroop Sriram, Michael Auli, and Alexei Baevski,  “Wav2vec-aug: Improved self-supervised training with limited data,”  arXiv preprint arXiv:2206.13654, 2022.
  * [11] Changfeng Gao, Gaofeng Cheng, and Pengyuan Zhang,  “Multi-variant consistency based self-supervised learning for robust automatic speech recognition,” 2021.
  * [12] Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazaré, Matthijs Douze, and Emmanuel Dupoux,  “Data augmenting contrastive learning of speech representations in the time domain,”  in IEEE SLT 2021, pp. 215–222.
  * [13] Aaron van den Oord, Yazhe Li, and Oriol Vinyals,  “Representation learning with contrastive predictive coding,”  arXiv preprint arXiv:1807.03748, 2018.
  * [14] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin,  “Unsupervised learning of visual features by contrasting cluster assignments,”  NeurIPS 2020, pp. 9912–9924.
  * [15] Joshua David Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka,  “Contrastive learning with hard negative samples,”  in ICLR 2021.
  * [16] Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka,  “Debiased contrastive learning,”  NeurIPS 2020, pp. 8765–8775.
  * [17] Jinqiang Wang, Tao Zhu, Liming Chen, Huansheng Ning, and Yaping Wan,  “Negative selection by clustering for contrastive learning in human activity recognition,”  arXiv preprint arXiv:2203.12230, 2022.
  * [18] Amrita Bhattacharjee, Mansooreh Karami, and Huan Liu,  “Text transformations in contrastive self-supervised learning: A review,”  arXiv preprint arXiv:2203.12000, 2022.
  * [19] Guangrun Wang, Keze Wang, Guangcong Wang, Philip HS Torr, and Liang Lin,  “Solving inefficiency of self-supervised representation learning,”  in IEEE CVPR 2021, pp. 9505–9515.
  * [20] J.J. Godfrey, E.C. Holliman, and J. McDaniel,  “Switchboard: telephone speech corpus for research and development,”  in ICASSP 1992.
  * [21] Douglas B Paul and Janet Baker,  “The design for the wall street journal-based csr corpus,”  in Speech and Natural Language: Proceedings of a Workshop Held at Harriman, 1992.
  * [22] Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee,  “Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,”  in IEEE ICASSP 2020, pp. 6419–6423.
  * [23] Andy T Liu, Shang-Wen Li, and Hung-yi Lee,  “Tera: Self-supervised learning of transformer encoder representation for speech,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 2351–2366, 2021.
  * [24] Dongwei Jiang, Wubo Li, Miao Cao, Wei Zou, and Xiangang Li,  “Speech simclr: Combining contrastive and reconstruction objective for self-supervised speech representation learning,”  arXiv preprint arXiv:2010.13991, 2020.
  * [25] Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu,  “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,”  in IEEE ASRU 2021, pp. 244–250.
  * [26] Kaizhi Qian, Yang Zhang, Heting Gao, Junrui Ni, Cheng-I Lai, David Cox, Mark Hasegawa-Johnson, and Shiyu Chang,  “Contentvec: An improved self-supervised speech representation by disentangling speakers,”  in ICML 2022, pp. 18003–18017.
  * [27] Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al.,  “Superb: Speech processing universal performance benchmark,”  arXiv preprint arXiv:2105.01051, 2021.
  * [28] Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng,  “Contrastive clustering,”  in AAAI 2021, pp. 8547–8555.
  * [29] Florian Schroff, Dmitry Kalenichenko, and James Philbin,  “Facenet: A unified embedding for face recognition and clustering,”  in IEEE CVPR 2015, pp. 815–823.
  * [30] Yumin Suh, Bohyung Han, Wonsik Kim, and Kyoung Mu Lee,  “Stochastic class-based hard example mining for deep metric learning,”  in IEEE CVPR 2019, pp. 7251–7259.
  * [31] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  arXiv preprint arXiv:1904.08779, 2019.
  * [32] Linghui Meng, Jin Xu, Xu Tan, Jindong Wang, Tao Qin, and Bo Xu,  “Mixspeech: Data augmentation for low-resource automatic speech recognition,”  in IEEE ICASSP 2021, pp. 7008–7012.
  * [33] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck,  “Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,”  arXiv preprint arXiv:2005.07143, 2020.
  * [34] Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L Seltzer, and Sanjeev Khudanpur,  “A study on data augmentation of reverberant speech for robust speech recognition,”  in IEEE ICASSP 2017, pp. 5220–5224.
  * [35] Emiru Tsunoo, Kentaro Shibata, Chaitanya Narisetty, Yosuke Kashiwagi, and Shinji Watanabe,  “Data augmentation methods for end-to-end speech recognition on distant-talk scenarios,”  arXiv preprint arXiv:2106.03419, 2021.
  * [36] Janne Spijkervet,  “Spijkervet/torchaudio-augmentations,” 2021.
  * [37] Xin Jin and Jiawei Han,  K-Means Clustering, pp. 563–564,  Springer US, Boston, MA, 2010.
  * [38] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: an asr corpus based on public domain audio books,”  in IEEE ICASSP 2015, pp. 5206–5210.
  * [39] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli,  “fairseq: A fast, extensible toolkit for sequence modeling,”  in NAACL-HLT 2019: Demonstrations.
  * [40] David Snyder, Guoguo Chen, and Daniel Povey,  “MUSAN: A Music, Speech, and Noise Corpus,” 2015,  arXiv:1510.08484v1.
  * [41] Jagadeesh Balam, Jocelyn Huang, Vitaly Lavrukhin, Slyne Deng, Somshubra Majumdar, and Boris Ginsburg,  “Improving noise robustness of an end-to-end neural model for automatic speech recognition,” 2020.