# CAME: Contrastive Automated Model Evaluation

Ru Peng1, Qiuyang Duan1111Our code is publicly available at:
https://github.com/pengr/Contrastive_AutoEval, Haobo Wang1, Jiachen Ma1,2,
Yanbo Jiang1,
Yongjun Tu3, Xiu Jiang3, Junbo Zhao1 Both authors contributed equally to this
research.Corresponding author. 1Zhejiang University 2DI-Lab, Zhejiang
University 3OPPO
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

The Automated Model Evaluation (AutoEval) framework entertains the possibility
of evaluating a trained machine learning model without resorting to a labeled
testing set. Despite the promise and some decent results, the existing
AutoEval methods heavily rely on computing distribution shifts between the
unlabelled testing set and the training set. We believe this reliance on the
training set becomes another obstacle in shipping this technology to real-
world ML development. In this work, we propose Contrastive Automatic Model
Evaluation (CAME), a novel AutoEval framework that is rid of involving
training set in the loop. The core idea of CAME bases on a theoretical
analysis which bonds the model performance with a contrastive loss. Further,
with extensive empirical validation, we manage to set up a predictable
relationship between the two, simply by deducing on the unlabeled/unseen
testing set. The resulting framework CAME establishes a new SOTA results for
AutoEval by surpassing prior work significantly. 111Our code is publicly
available at: https://github.com/pengr/Contrastive_AutoEval

##  1 Introduction

During the last decade, the technological advancement of artificial
intelligence and machine learning has attained unprecedented achievements,
affecting a variety of domains or verticals. Ubiquitously, off these
milestones, to properly evaluate, assess and benchmark the trained models is
undoubtedly pivotal, particularly when considering the deployment towards the
production in real-world scenarios. To do that, the traditional means often
relies on a pre-split and static testing set for model evaluation, which is
principally left out of the sight during training or validation phase.
However, several recent works has pointed out the drawback of this
standardized scheme due to its requirement of careful sample selection,
randomization due to the sample set split, the OOD gap between the deployment
environment, and (somewhat) expensive label annotation [14, 13, 5], etc. Most
recently, we see that _Automated Model Evaluation_ (AutoEval) has emerged to
tackle these problems [14].

In particular, the vanilla prototype of the Automated Model Evaluation
approaches aim at estimating a provided model’s performance on an unlabeled
testing set. Notably, these approaches first generate a few _meta-sets_ by
adopting pre-selected data augmentations on the training set. In what follows,
one can estimate a certain distance — for instance, the Frechet distance [16]
— averaged between the meta-sets with the testing set. As a result, the prior
work has proactively shown that this averaged distance measurement is related
to the final model performance on the testing set. Indeed, we believe this
setup of AutoEval on the testing set possesses positive prospects because it
manifests a high similarity towards real production — where the testing set is
acquired on the fly in the real-world, leaving no time/space for these samples
to be annotated or persist. A graphical illustration of the AutoEval against
the conventional static testing set evaluation is depicted in Figure 1.

Despite its promise and prospect, we realize that the current paradigm of
AutoEval may still fail in its real-world deployment, under certain
conditions. On one hand, it is widely acknowledged that the prior works are
dedicated to avoiding annotating the testing samples and to amortizing the
vexing randomness through the massive generation of meta-sets offline. On the
other hand, however, these techniques still demand the full presence of the
sample input from the training set, which in many — if not most — of the
occasions probably imply expensive storage and computation cost. Hence, we
argue that this requirement cannot be easily ensured in many scenarios, most
notably on limited-capacity, limited-storage, or low-power platforms such as
edge devises for IOT or autonomous driving. Hereby, we pose the core
motivation of the design of this work: can we establish an AutoEval framework
without keeping the training set in the loop?

To reach this target is not trivial, and it cannot be achieved by
incrementally changing the prior method. This is mostly due to the heavy bond
of the final model performance regressor with the meta-sets induced from the
training data. In this work, we hope to break this paradigm commonly used in
prior work, and propose a novel paradigm — Contrastive Automatic Model
Evaluation, dubbed (CAME). Unlike the previous approaches, CAME aims to
regress to the final model performance that assumes the absence of the
training data. In particular, CAME is very much motivated by the following
series of the theories:

Figure 1: Illustration of workflow differences between AutoEval and
traditional model evaluation.

###### Theorem 1.1

[59] Given a model $f$ with an optimal functional minimizer
$f^{*}=\arg\min\mathcal{L}_{NCE}(f)$, its classification risk can be upper-
and lower-bounded by its contrastive learning risk as

$\displaystyle\mathcal{L}_{NCE}(f^{*})-\mathcal{O}(M^{-1/2})$
$\displaystyle\leq\mathcal{L}_{CE}^{\mu}(f^{*})+\log(M/K)$ (1)
$\displaystyle\leq\mathcal{L}_{NCE}(f^{*})+\mathcal{O}(M^{-1/2})$

where $M$ is the number of negative samples in contrastive learning, $K$ is
the number of classes, $\mathcal{L}_{NCE}$ [50] is the InfoNCE loss for
contrastive learning, and $\mathcal{L}_{CE}^{\mu}$ [59] is the mean CE Loss
used to indicate the downstream classification risk (definitions in section
3).

Theorem 1.1 indicate that under mild assumptions, the contrastive loss
constantly bounded the cross-entropy loss and thus, can reflect the overall
trends of generalization. Moreover, analogous theoretical guarantees of
bounding CE loss through CL risk are also evident in [53] and [2]. Notably, in
the AutoEval problem, with distribution shifts and the absence of ground-truth
labels on the test sets, the cross-entropy loss $\mathcal{L}_{CE}$ is
inaccessible. Fortunately however, $\mathcal{L}_{NCE}$ is self-supervised and
can be inferred purely from testing inputs. Based on the theoretical analysis,
we cast a hypothesis as follows. The contrastive loss — calculated from the
testing set alone — is informative towards predicting the performance of the
provided model.

To this regard, we briefly introduce our framework, CAME. It is prerequisite
composed of two conditions: (i)-the model is trained jointed of a normal task
loss together with a contrastive loss and (ii)-the model performance is not
affected by jointly contrastive learning. Based on the model yielded this way,
we conduct a rigorous empirical study — guided by the theories we pose above —
that we prove the correlation between the contrastive loss on the testing set
with its performance truly exists. The AutoEval established this way enjoys
the following two major merits: (i)-it shreds the need for the training set
during the evaluation phase which further extends the AutoEval technique
towards production in real-world; (ii)-CAME sets a new record of testing
performance estimation by exceeding the prior work by significant margins.

(a) MNIST

(b) CIFAR-10

(c) CIFAR-100

(d) COCO

(e) TinyImageNet

Figure 2: Experimental results about the correlation study. On the above
different datasets, we show that there exists strong linear correlation
between contrastive accuracy (x-axis) and classification accuracy (y-axis).
The symbols $r$ represents Pearson’s correlation coefficient and $\rho$
indicates Spearman’s rank correlation coefficient.

##  2 Related Works

Automated Model Evaluation. [14, 13, 56, 60] build regression models on many
test sets with distribution shifts to predict model’s accuracy on an unlabeled
test set. [23, 20] use confidence based methods, . Our work does not use
distribution shift measurements. Instead, we directly use contrastive accuracy
to regress classification accuracy.

Model Generalization Prediction. This problem is to estimate the
generalization gap and predict the generalization error. From model
perspective, [10, 34, 35] predict generalization error by leveraging model
parameters. From data perspective, [9, 62] predict the generalization gap
under distribution shifts via data representations. Different from these
works, our work aims to directly predict a model’s accuracy on unseen
unlabeled test sets.

Out-of-Distribution Detection. The OOD Detection task [28, 44, 46, 26] is to
detect test samples subject to a distribution different from the training data
distribution. It focuses on the data distribution of training data and testing
data. Different from this, our work focuses on estimating model’s accuracy on
unlabeled OOD test set.

Contrastive Learning. Contrastive Learning (CL) [6, 24, 7, 22, 3, 8] is a
typical self-supervised learning paradigm to learn effective representation of
input samples. Used as a pre-training task, it can significantly enhance the
downstream semantic classification task, which means CL learns information-
rich features for object classification. Inspired by these pioneering efforts,
we choose contrastive learning as the auxiliary learning task, and take the
classical CL framework (SimCLR) to validate the feasibility of CL in this
work.

Unsupervised Domain Adaptation. Unsupervised domain adaptation is also an
active research field, which aims to use labeled source data and unlabeled
target data to learn a model generalizing well from the source domain to the
target domain. In recent years, many researches such as [55, 47, 57, 43, 31]
have proposed different measures and frameworks to promote the development of
this field. In this work, given a model trained on the source dataset, we
focus on obtaining a precise estimation of its accuracy on unlabeled target
sets.

##  3 Contrastive Automated Model Evaluation

In this section, we describe our proposed method, CAME, in detail. The core
idea of CAME is simple. We first formulate a multi-task learning framework by
integrating a normal task loss with a contrastive learning loss objective.
After attaining the model through regular optimization process, on an unseen
and unlabeled testing set, we build a simple and separate neural network to
regress from the contrastive loss to a proximal model performance. A pseudo
code is provided in algorithm 1.

Algorithm 1 Contrastive Automated Model Evaluation

0: Training set $\mathcal{D}_{o}$, seed set $\mathcal{D}_{e}$, unseen test set
$\mathcal{D}_{t}$; shared encoder $f$, projection heads $h$, $g$; contrastive
loss weight $\lambda$; data transformations $t\sim\mathcal{T}_{s}$ // Multi-
task learning

1: for $epoch=1,2,...T$ do

2: sample the mini-batch $\mathcal{D}_{l}$, $\mathcal{D}_{u}$ from
$\mathcal{D}_{o}$

3: for $x\in\mathcal{D}_{l}$ do

4: $z=g(f(x))\in\mathbb{S}^{2N-1}$

5: $Wf(x)=h(f(x))\in\mathbb{R}^{K}$

6: $\mathcal{L}=\mathcal{L}_{CE}+\lambda\mathcal{L}_{NCE}$, $\mathcal{L}_{CE}$
and $\mathcal{L}_{NCE}$ in Sec. 3.3

7: end for

8: end for // Synthesizing Sample Sets

9: for $i=1,2,...,a$ do

10: $\mathcal{D}_{s}^{i}={t}_{i}(\mathcal{D}_{e})$

11:
$Acc_{con}^{i}=\sum_{i=0}^{N-1}\mathbb{I}\left[i=\mathop{\arg\max}\limits_{j\in[0,N-1]}g_{j}(f(D_{s}^{i}))\right]/N$

12:
$Acc_{cla}^{i}=\sum_{i=0}^{N-1}\mathbb{I}\left[y=\mathop{\arg\max}\limits_{i\in\\{1,2,\cdots,K\\}}h_{i}(f(D_{s}^{i}))\right]/N$

13: end for

14: Correlation Coefficients: $r$, $\rho$ statistic from
$\mathcal{D}_{s}=\\{(Acc_{con}^{1},Acc_{cla}^{1}),\ldots,(Acc_{con}^{a},Acc_{cla}^{a})\\}$

15: Linear regressor $R:Acc_{cla}=W^{\top}(Acc_{con})+b$ // Automated Model
Evaluation via Regression

16: In $\mathcal{D}_{t}:\hat{Acc_{cla}}=R(Acc_{con})$; $Acc_{con}$,
$Acc_{cla}$ in Eq. 8

17: Mean Absolute Error: $\varepsilon=|Acc_{cla}-\hat{Acc_{cla}}|$

17: Pearson’s correlation $r$, Spearman’s rank correlation $\rho$ and Mean
Absolute Error $\varepsilon$.

###  3.1 Problem Definition

Consider a image classification task, we aim to estimate the classification
performance of a trained classifier on different unseen and unlabeled test
sets automatically. We denote the training set as
$\mathcal{D}_{o}=\\{(x_{i},y_{i})\\}_{i=1}^{I}$, where
$y_{i}\in\\{1,2,\cdots,K\\}$ denotes the label of the image $x_{i}$. These
unseen and unlabeled test set are denoted as
$D_{t}=\\{\\{x_{j}\\}_{j=1}^{J_{1}},\ldots,\\{x_{j}\\}_{j=1}^{J_{M}}\\}$,
where $\\{x_{j}\\}_{j=1}^{J_{M}}$ represent the $m$-th unseen test set. Based
on the theoretical anaylsis in Theorem 1.1 — “For the contrastive learning
model, its classification risk can be upper and lower bounded by its
contrastive risk in any unseen test data distribution”, which means that it is
feasible to predict classification accuracy with contrastive accuracy.
Motivated by this exhilarating finding, in unseen test set
$D_{t}^{m}=\\{x_{j}\\}_{j=1}^{J_{m}}$, we fit a linear regressors [32]
$R:(f,g,D_{t}^{m})\rightarrow\hat{Acc}$ to predict a classifier’s accuracy
$\hat{Acc_{cla}}$ by its contrastive accuracy:

$\hat{Acc_{cla}}=R(f,g,D_{t}^{m})=R(Acc_{con}),$ (2)

where $f,g$ are the shared encoder and contrastive projection head in Section
3.3.

###  3.2 Correlation Analysis

To further corroborate the feasibility of predicting classification accuracy
from contrastive learning accuracy, we analyze the correlation between the two
sources of accuracies among various data setup in Figure 2. For each data
setup, we train CAME on the training set, then evaluate its accuracy pairs on
each synthetic sample set (mentioned in Section 3.4). Finally, we report the
scatter plot and both Pearson’s correlation coefficient $r$ and Spearman’s
rank correlation coefficient $\rho$. Here, each data point in the scatter plot
corresponds to the accuracy pair of a synthetic sample set. From Figure 2, we
can see that among various data environments, the contrastive learning
accuracy and classification accuracy exhibit strong linear correlation
($r>0.937$ and $\rho>0.898$). This finding prompts us to train a linear
regressor to predict classification accuracy on unlabeled unseen test
distribution, rather than building a non-linear MLP like previous work.

###  3.3 Multi-task Learning

In this section, we begin by introducing how CAME trains the model via pivotal
multi-task learning, to reveal the inherent strong correlation between
classification accuracy and contrastive learning accuracy. In summary, the co-
training paradigm consists of the supervised classification and the self-
supervised contrastive learning. Specifically, given a mini-batch of N labeled
samples $\mathcal{D}_{l}=\left\\{\left(x_{i},y_{i}\right)\right\\}_{i=1}^{N}$
where labels $y_{i}\in\\{1,\ldots,K\\}$ and N unlabeled samples
$\mathcal{D}_{u}=\left\\{x_{i}\right\\}_{i=1}^{N}$. We adopt a network-
unrestricted encoder $f\in\mathcal{F}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{d}$
to extract image representations from input samples. Then, we apply two
projection heads $g:\mathbb{R}^{d}\rightarrow\mathbb{S}^{2N-1}$ and
$h:\mathbb{R}^{d}\rightarrow\mathbb{R}^{K}$ to map the shared representations
into two spaces where contrastive loss and classification loss are applied,
respectively. The illustration of our model is shown in Figure 3.

Contrastive Learning. Taken a training sample $x\in\mathcal{D}_{u}$, we apply
a set of random data augmentation $t\sim\mathcal{T}$ to generate its positive
sample $x^{+}=t(x)$, and treat the other augmented samples
$\left\\{x_{i}^{-}\right\\}_{i=1}^{2(N-1)}$ within a mini-batch as its
negative samples. Then, the shared encoder $f$ can be learned by the InfoNCE
contrastive loss [50] which mapping from the d-dimensonal image
representations to a unit hypersphere:

$\displaystyle\mathcal{L}_{\mathrm{NCE}}(f,g)=\mathop{\mathbb{E}}\limits_{p\left(x,x^{+}\right)}\mathop{\mathbb{E}}\limits_{\left\\{p\left(x_{i}^{-}\right)\right\\}}\left[-\log\frac{\exp\left(z^{\top}z^{+}\right)}{\sum\limits_{i=1}^{2N-1}\exp\left(z^{\top}z_{i}^{-}\right)}\right],$
(3)

where $z=g(f(x))$, $p\left(x\right)$ is the data distribution, and
$p\left(x,x^{+}\right)$ is the joint distribution of positive data pairs.

Classification Learning. Given a labeled data pairs $(x,y)\in\mathcal{D}_{l}$,
we usually use the cross-entropy (CE) loss to train the shared encoder $f$
through classification learning:

$\displaystyle\mathcal{L}_{\mathrm{CE}}(f,h)=\mathbb{E}_{p(x,y)}\left[-\log\frac{\exp\left(f(x)^{\top}w_{y}\right)}{\sum_{i=1}^{K}\exp\left(f(x)^{\top}w_{i}\right)}\right],$
(4)

where $h=\left[w_{1},w_{2},\ldots,w_{K}\right]$ is the classification head.

Hereafter, we train our classifier based on the aforementioned multi-task
learning paradigm, by minimizing the following loss:

$\mathcal{L}=\mathcal{L}_{CE}+\lambda\mathcal{L}_{NCE},$ (5)

where $\lambda$ is a weighing coefficient.

Figure 3: The model architecture of our multi-task learning. Here, we adopt
SimCLR [6] as the contrastive learning framework. Note that we only consider
the original batch when calculating the classification accuracy.

###  3.4 Synthesizing Sample Sets

To fit the linear regressor (Eq. 2) for performance prediction, we need to
collect the contrastive accuracy and the counterpart classification accuracy
in various test environments by the multi-task-learning based model. For this,
we require many test sets which should include: _i)-various distributions,_
_ii)-the same classes as the training set but without image overlap,_ and
_iii)-sufficient samples._ However, one obstacle still is to collect such
testing sets from natural distributions. As a surrogate, we synthesize these
“sample sets”. Overall, we synthesize these sample sets by applying a
combination of transformations $t\sim\mathcal{T}_{s}$ on a seed dataset. Note
that all possibilities of the transformation sequences are not calculated by
permutation and combination, so there are many random states when applying
these transformations. Therefore, we generate image sets of various
distributions. For all the data transformation setup, we inherit the labels
from the seed dataset directly because the transformations do not alter the
profound semantics. Meanwhile, We generate 400 synthetic sample sets to
calculate the accuracy pairs
$\\{(Acc_{con}^{1},Acc_{cla}^{1}),\ldots,(Acc_{con}^{a},Acc_{cla}^{a})\\}$,
which is applied to fit a linear regression model and statistic the
correlation among these accuracy pairs. $a$ is the amount of synthetic sample
sets, the default value is set as 400. Specific setups are elaborated as
follow.

  * •

MNIST setup. We synthesize sample sets by applying various transformations on
MNIST test dataset. MNIST as simple gray-scale images, we first consider
change their black background to color ones. Specifically, for each sample of
MNIST, we randomly select an image from COCO dataset and randomly crop a
patch, which is then used as the background of the hand-written digit sample.
After that, we randomly apply three out of the six transformations on the
background-replaced images: {_autoContrast, rotation, color, brightness,
sharpness, translation_}.

  * •

CIFAR-10 setup. We synthesize sample sets based on CIFAR-10 test set by
adopting three image transformations on it. The three transformations are
randomly selected from the six transformations of MNIST Setup.

  * •

CIFAR-100 setup. We study on CIFAR-100 to explore if a strong correlation
still exists between contrastive learning and classification accuracy in more
semantic classes case. The applied transformations are followed as CIFAR-10
setup. Here, we use the training split of CIFAR-100 dataset to train our model
and the testing split to synthesize the various sample sets.

  * •

COCO setup. Following the practice in [13] and [51], we select 12 common
classes: aeroplane, bike, bird, boat, bottle, bus, car, dog, horse, monitor,
motorbike and person. Resort to the annotation boxes, we crop out the objects
of images belonging to these categories in the COCO training set and
validation set to build training and testing set for the classification task.
Based on the testing set, we use 15 common corruptions from CIFAR-10-C to
generate 60 sample sets, and randomly use three transformations in MNIST setup
to generate the rest of sample sets.

  * •

TinyImageNet setup. The synthetic sample sets of TinyImageNet are synthesized
from its validation split. As in the MNIST setup mentioned above, we randomly
use three kinds of transformations to synthesize sample sets for linear
regression and correlation study.

###  3.5 Automated Model Evaluation via Regression

As the final step, we propose to use contrastive learning accuracy to regress
to the model’s classification accuracy on unseen testing set. Notably, for the
input and output side of training the regressor, we adopt the soft version
attained from contrastive learning and classification tasks. On the input
side, we define it as the probability of an augmented sample being positive-
sample. For the output, we use the confidence value of the sample being
corrected predicted.

Consequently we write down the forms as follow:

$\displaystyle
Acc_{con}=\sum_{i=0}^{N-1}\mathbb{I}\left[i=\mathop{\arg\max}\limits_{j\in[0,N-1]}g_{j}(f(x))\right]/N,$
(6) $\displaystyle
Acc_{cla}=\sum_{i=0}^{N-1}\mathbb{I}\left[y=\mathop{\arg\max}\limits_{i\in\\{1,2,\cdots,K\\}}h_{i}(f(x))\right]/N$
(7) $\displaystyle\varepsilon=|Acc_{cla}-\hat{Acc_{cla}}|,$ (8)

where $\mathbb{I}\left[\cdot\right]$ is an indicator function, $N$ is the
mini-batch size, $i$ is the index of an image in original batch, $y$ is the
ground-truth class label. $\varepsilon$ is the mean absolute error (MAE) for
classification accuracy estimation.

Table 1: Pearson’s correlation ($r$) and Spearman’s rank correlation ($\rho$)
on the different data setup (higher is better). “-” indicates that the results
are not reported in original paper.

Method | MNIST | CIFAR-10 | CIFAR-100 | COCO
---|---|---|---|---
$\mathnormal{r}$ | $\rho$ | $\mathnormal{r}$ | $\rho$ | $\mathnormal{r}$ | $\rho$ | $\mathnormal{r}$ | $\rho$
Frechet [14] | 0.912 | - | - |  | - | - | 0.908 | -
Rotation [13] | - | 0.960 | - | 0.981 | - | 0.950 | - | 0.881
Jigsaw [49] | - | - | - | 0.958 | - | - | - | -
CAME (ours) | 0.948 | 0.954 | 0.953 | 0.957 | 0.963 | 0.960 | 0.898 | 0.962
|  |  |  |  |  |  |  |

##  4 Experiments

###  4.1 Experimental Setup

We conduct extensive experiments on various datasets. Similar to prior work,
the final assessment of CAME is based on a homogeneous but different unseen
test environment. For instance, in hand-written datasets, we train our model
and regressor on MNIST but test them in SVHN [48] and USPS [33]. The differing
distribution of the training and testing set is significant but the tasks are
indeed homogeneous. This protocol effectively validates the generalizability
of AutoEval approaches.

For natural images, we train DenseNet-40-12 [30] on CIFAR-10 [37], then
conducting test on CIFAR-10.1 [52] and CIFAR-10-C [27]. For CIFAR-100 setup
[37], we keep the setttings on CIFAR-10, and test it on CIFAR100-C. For COCO
setup [45], following [13], we choose 12 object classes (aeroplane, bicycle,
bird, boat, bottle, bus, car, dog, horse, tv-monitor, motorcycle, person). To
build the training set, object annotation boxes among these 12 classes are
cropped from the COCO training images containing them. These images are used
to train a ResNet-50 backbone [25]. Similarly, we build unseen testing sets
for classification accuracy from Caltech-256 [21], PASCAL VOC 2007 [18],
ImageNet [12], which carrying the same 12 categories. For TinyImageNet setup
[39], we train ResNet-50 and the testing is conducted on TinyImageNet-C.

Table 2:  Mean absolute error (MAE) results for evaluating the classifier
accuracy on the unseen test sets. The training set of each group is MNIST,
CIFAR-10, CIFAR-100, COCO and TinyImageNet, respectively. In MNIST group, the
merged cells represent the average MAE value among the SVHN and USPS. “-”
indicates that the results are not reported in original paper. The time cost
for different algorithms we count at here33footnotemark: 3.

Method | MNIST | CIFAR-10 | CIFAR-100 | COCO | TinyImageNet
---|---|---|---|---|---
SVHN | USPS | CIFAR-10.1 | CIFAR10-C | CIFAR100-C | Caltech | Pascal | ImageNet | TinyImageNet-C
Pred ($\tau$ = 0.8) [28, 44] | 10.58 | 21.18 | 3.00 | 1.82 | - | 3.25 | 2.45 | 2.66 | 6.21
Pred ($\tau$ = 0.9) [28, 44] | 0.99 | 35.13 | 1.30 | 1.26 | - | 8.31 | 8.43 | 8.00 | 8.54
Entropy ($\tau$ = 0.2) [36] | 3.57 | 32.29 | 1.05 | 1.80 | - | 5.81 | 6.29 | 5.33 | 8.48
Ens. AC [38] | 80.05 | 12.84 | - | 23.7 | - | - | - | - | -
Proxy Risk [9] | 13.20 | 1.21 | - | 5.3 | - | - | - | - | -
Ens. RI [4] | 79.56 | 8.01 | - | 14.9 | - | - | - | - | -
Ens. RM [4] | 3.88 | 0.65 | - | 2.2 | - | - | - | - | -
Frechet [14] | 0.82 | 13.94 | 0.96 | 1.94 | - | 13.63 | 2.26 | 5.64 | 8.13
Frechet + $\mu$ \+ $\sigma$ [14] | 2.06 | 0.03 | 0.83 | 1.83 | - | 4.01 | 1.63 | 2.99 | 7.96
Rotation [13] | 1.78 | 12.42 | 3.74 | 1.99 | - | 1.91 | 2.86 | 3.15 | 8.21
SSDR [56] | 0.76 | - | 0.74 | 1.28 | - | - | - | - | 5.95
AC [28, 17] | 21.17 | 9.88 | 16.50 | 23.61 | - | - | - | 32.44
IM [5] | 18.48 | 6.60 | 12.33 | 13.69 | - | - | - | 19.86
DOC [23] | 20.19 | 7.25 | 13.87 | 14.60 | - | - | - | 25.02
GDE [35] | 24.42 | 4.77 | 6.55 | 9.85 | - | - | - | 5.41
ATC-MC [20] | 5.02 | 3.21 | 4.65 | 5.50 | - | - | - | 5.93
ATC-NE [20] | 3.14 | 2.99 | 4.21 | 4.72 | - | - | - | 5.00
CAME (ours) | 0.52 | 0.24 | 0.49 | 0.84 | 2.34 | 0.80 | 0.85 | 0.81 | 2.50
|  |  |  |  |  |  |  |  |

###  4.2 Main Results

In Table 3, we report the mean absolute error (MAE) results of estimating
classifier accuracy on unseen test sets. From this table, among all data
setup, we conclude that our method reduces the accuracy estimation error by
about 47.2% on average upon prior SOTA method. Further, CAME shows strong
performance regardless of the image domains or the granularities of the
classification categories. Additionally, in Table 1 and 3, we report some more
detailed results, regarding statistical correlation scores with other
intermediate results.

Validity of Multi-task Co-training Setup. To guarantee a fair-minded
assessment of model performance, we must ensure that auxiliary contrastive
learning task satisfies the following criteria: i)-no extra learning
complexity for the main task, ii)-minimal network changes and iii)-does not
degrade classification accuracy. To substantiate the soundness of our co-
training buildup, we report the ground-truth accuracies of the co-trained
classifiers in Table 3. And below, we give an apple-to-apple comparison with
classification-only model – FD [14]: SVHN (23.42 vs. 25.46), USPS (88.64 vs.
64.08), Pascal (85.04 vs. 86.13), Caltech (96.08 vs. 93.40), ImageNet (82.66
vs. 88.83). Drawing from these results we can know that co-training as a
feasible strategy will not degrade the performance of the model to be
evaluated. This signifies our adherence to the principle of fairly evaluating
a model’s performance that deserves testing.

Table 3: Results of ground-truth classification accuracy, predicted
classification accuracy, contrastive learning accuracy and MAE from CAME
(ours) in different data setups.

Dataset | MNIST | CIFAR-10 | CIFAR-100 | COCO | TinyImageNet
---|---|---|---|---|---
SVHN | USPS | CIFAR-10.1 | CIFAR10-C | CIFAR100-C | Caltech | Pascal | ImageNet | TinyImageNet-C
Ground-truth Cla. Acc. | 23.42 | 88.64 | 80.80 | 73.71 | 48.04 | 96.08 | 85.04 | 82.66 | 40.41
Predicted Cla. Acc. | 23.94 | 88.40 | 80.31 | 74.55 | 45.70 | 96.88 | 85.89 | 83.47 | 42.51
Con. Acc. | 14.19 | 42.98 | 88.47 | 80.60 | 98.74 | 98.43 | 91.74 | 90.02 | 82.83
MAE | 0.52 | 0.24 | 0.49 | 0.84 | 2.34 | 0.80 | 0.85 | 0.81 | 2.50
|  |  |  |  |  |  |  |  |

Figure 4: Pearson’s correlation ($r$), Spearman’s rank correlation ($\rho$)
and MAE with different augmentation strength on the RandomResizedCrop
operation under CIFAR-10 setup.

###  4.3 Ablation Studies

####  4.3.1 Contrastive Learning Parameters

As we adopt SimCLR in our framework, we want to know if our overall
performance is consistent to its basic settings (i.e. the linear correlation
coefficient achieves its maximum and the estimation error achieves its minimum
under the best training parameters of SimCLR). According to [59], among the
data augmentations adopted in SimCLR, RandomResizedCrop is the most important
augmentation, and ColorJitter is the second. So we study the impact of these
two kinds of augmentations on CIFAR-10 in our work.

For RandomResizedCrop, to quantify its influence, we use the augmentation
strength defined in [59]. For a RandomResizedCrop operator with scale range
$[a,b]$, its aug-strength can be defined as $r=(1-b)+(1-a)$. In Figure 4, we
show that under different augmentation strengths, the accuracy estimation
error achieves its minimum at the default strength value 0.92.

For ColorJitter, we study its parameters: brightness, contrast, saturation and
hue, where the augmentation strength is corresponding to the parameter value.
Note that all other augmentations in SimCLR are kept default. In Figure 5, for
each of these parameters, we plot the changes of the accuracy estimation error
under the CIFAR-10 setup. Here we have similar observation that the default
parameter values ($Brightness=Contrast=Saturation=0.8,Hue=0.2$) in SimCLR
yield best performance.

(a) Brightness, Contrast, Saturation

(b) Hue

Figure 5: Pearson’s correlation ($r$), Spearman’s rank correlation ($\rho$)
and MAE with different augmentation strength on color jittering operations
under CIFAR-10 setup.

Also, temperature scaling is an important factor during the training process
of SimCLR. We study the temperature parameter $\tau$ on CIFAR-10. As Figure 6
shows, when using default temperature value $\tau=0.07$, we can obtain best
performance for both MAE and correlation coefficient.

Figure 6: Pearson’s correlation ($r$), Spearman’s rank correlation ($\rho$)
and MAE with different temperature scaling parameters under CIFAR-10 setup.

Finally, we investigate how to assign appropriate task weights in the proposed
multi-task training. Here we fix the classification weight to 1.0, and change
different CL task weights: 1.0, 0.1, 0.01, 0.001. In Table 4, we report the
linear correlation score and estimation error. The insight is that our
framework is more likely to perform well when assigning small weight ($<0.01$)
for the CL task.

Table 4: Pearson’s correlation ($r$), Spearman’s rank correlation ($\rho$) and
MAE with different contrastive learning task weight $\lambda$ under CIFAR-10
setup.

CL task weight | 1.0 | 0.1 | 0.01 | 0.001
---|---|---|---|---
$r$ | 0.953 | 0.946 | 0.872 | 0.953
$\rho$ | 0.957 | 0.933 | 0.860 | 0.957
MAE | 0.71 | 2.87 | 0.91 | 0.49
|  |  |  |

In summary, these empirical results demonstrate that when adopting CL
frameworks, keeping default optimal settings is most likely to build strong
linear correlation between the CL accuracy and classification accuracy, as
well as obtain lowest accuracy estimation error on the final unlabeled test
set.

####  4.3.2 Different Training Settings

Different contrastive learning augmentation groups. In this paper, we adopt
SimCLR, to study if other frameworks can fit in well, we change SimCLR to
MoCo-v1 [24], MoCo-v2 [7], and BYOL [22]. From Table 5, we can observe that on
CIFAR-10, the linear correlations are all strong across different CL
frameworks ($r>0.86$).

Table 5: Pearson’s correlation ($r$), Spearman’s rank correlation ($\rho$) and
MAE with different CL data augmentation under CIFAR-10 setup.

CL Data Aug | SimCLR | MoCo-v1 | MoCo-v2 | BYOL
---|---|---|---|---
$r$ | 0.957 | 0.868 | 0.893 | 0.922
$\rho$ | 0.953 | 0.864 | 0.897 | 0.902
MAE | 0.49 | 1.20 | 0.88 | 0.59
|  |  |  |

Different amounts and sizes of synthetic datasets. In this paper, we
synthesize many sample sets to build a regression model for accuracy
estimation, so we study the influence of the amount of synthetic test sets and
the size of each test set. Here, we refer to them as _sample set amount_ and
_sample set size_ respectively. As Figure 7 shows, the linear correlation is
quite robust and the estimation error is also robust when there are enough
test sets.

(a) Sample set amount

(b) Sample set size

Figure 7: Pearson’s correlation ($r$), Spearman’s rank correlation ($\rho$)
and MAE with different meta-set size and sample-set size under CIFAR-10 setup.

Different backbones. In experimental setup, we use DenseNet-40-12 for CIFAR-10
setup in default. Here we change it to other model structures (ResNet-18,
ResNet-34, VGG-11, VGG-19 [54]) to study if the choice of backbone
significantly influences the performance. As Table 6 shows, our framework is
robust against different backbones (strong linear correlation and precise
accuracy estimation).

Table 6: Pearson’s correlation ($r$), Spearman’s rank correlation ($\rho$) and
MAE with different CNN backbones under CIFAR-10 setup.

Backbone | ResNet-18 | ResNet-34 | VGG-11 | VGG-19 | DenseNet-40-12
---|---|---|---|---|---
$r$ | 0.853 | 0.854 | 0.816 | 0.863 | 0.957
$\rho$ | 0.858 | 0.853 | 0.787 | 0.849 | 0.953
MAE | 0.92 | 0.63 | 1.17 | 0.77 | 0.49
|  |  |  |  |

Different random seeds. To check if the experimental results are robust to the
initial random state, we choose different random seeds for training (use 0 as
default seed). As Table 7 shows, the performance of our framework is robust to
randomness.

Table 7: Pearson’s correlation ($r$), Spearman’s rank correlation ($\rho$) and
MAE with different random seeds under CIFAR-10 setup.

Random Seed | 0 | 21 | 42
---|---|---|---
$r$ | 0.957 | 0.892 | 0.869
$\rho$ | 0.953 | 0.894 | 0.845
MAE | 0.49 | 0.65 | 0.51
|  |  |

Figure 8: The MAE of linear regressor on transformed test sets (ImageNet,
Pascal, and Caltech). The transformed datasets are denoted by “-A” and “-B”
with new transformations such Cutout [15], Shear, Equalize and
ColorTemperature [11]. (-) / (+) denotes the estimated accuracy is lower and
higher than the ground-truth accuracy, respectively.

(a) Only Classification

(b) Pre-train + Fine-tune

(c) Multi-task

Figure 9: T-SNE visualization of the classification head features on CIFAR-10
validation set. Different colors correspond to different classes.

##  5 Method Interpretability

Regressor Robustness on transformed test sets. To further check the regressor
robustness in the unobserved testing environment. We rigorously shift three
natural datasets (ImageNet, Pascal and Caltech) with new transformations that
do not overlap with the various transformations constructed when fitting
regression curves. Specifically, we use Cutout [15], Shear, Equalize and
ColorTemperature [11] to generate ImageNet-A$/$B, Pascal-A$/$B, Caltach-A$/$B.
We note the following observations from Figure 8. First, the classifier
accuracies will fluctuate after these test sets are shifted. Second, even in
these unseen test sets cases undergoing new transformations, our method
consistently achieves superior results. We conjecture that this phenomenon
stems from contrastive learning performing well across a broad spectrum of
underlying distributions.

Underlying relationship between contrastive accuracy and classification
accuracy. In common practice, contrastive learning is often used as a pre-
training task, which has been proved effective for learning visual
representations for downstream classification task. In this work, we adopt
SimCLR in multi-task training. Here we compare the two ways in Table 8. We can
observe that both of them can yield strong linear correlation and precise
accuracy estimation, while the multi-task way is better. This finding further
reveals the underlying relationship between contrastive accuracy and
classification accuracy, regardless of the training way.

Table 8: Comparison between the pre-train+fine-tune (Pre+Fine) and multi-task
training on CIFAR-10.

Paradigm | $r$ | $\rho$ | $Acc_{con}$ | $Acc_{cla}$ | $\hat{Acc_{cls}}$ | MAE
---|---|---|---|---|---|---
| Pre+Fine
---
0.900 | 0.928 | 94.87 | 61.20 | 60.20 | 1.00
Multi-task | 0.957 | 0.953 | 88.47 | 80.80 | 80.41 | 0.49
|  |  |  |  |  |

(a) Loss

(b) Accuracy

Figure 10: Learning curve plot of accuracy and loss in training phase.

Multi-task training learns better features for AutoEval. Intuitively, strong
linear correlation bewteen contrastive accuracy and classification accuracy
can be built because they both learn class-wise visual representations. What
kind of image features will be learned in multi-task training? In Figure 9 and
10, we plot some intermediate results during the training process. Notably,
compared to pre-train + fine-tune, multi-task yields better feature clusters,
which well corresponds to the results in Table 8. This further justifies the
usage of contrastive learning in a multi-task paradigm is indeed feasible.

##  6 Conclusion

In this paper, we propose a novel framework CAME for estimating classifier
accuracy on invisible test sets without ground-truth labels. We find that
there is a strong linear correlation between contrastive accuracy and
classification accuracy and give the theoretical analysis to support this
discovery. Thus, our work indicates that it is feasible to estimate classifier
accuracy by self-supervised contrastive accuracy using linear regression.
Extensive experimental results show that our method achieves new SOTA results
for AutoEval by outperforming previous works. In future, we will explore the
feasibility of other self-supervised learning tasks in AutoEval, and extend
CAME to other fields such as natural language processing and graph learning
tasks.

## Acknowledgement

This work is majorly supported by the Fundamental Research Funds for the
Central Universities (No. 226-2022-00028), and in part by the NSFC Grants (No.
62206247). RP, HW and JZ also thank the sponsorship by the CAAI-Huawei Open
Fund.

## Limitation

Our method is grounded on an assumption that approximates the image
distribution of unknown data environments via image transformations applied to
the proposed synthetic sample sets. However, this might encounter certain
intricate real-world cases where not be able to work. For example, there could
be non-negligible samples in testing sets whose classes have never appeared in
the training label space. Consequently, even though we will still predict an
estimated accuracy, it is essentially unavailable. Despite the seemingly
extreme situation, this issue could be well alleviated by employing out-of-
distribution detection techniques [29, 46, 42, 44] to help detect and reject
such samples. Furthermore, our method is not a plug-and-play solution for
model evaluation due to the co-training strategy. Nonetheless, our method
serves as a general technique with the potential for extension across various
fields, supported by the widespread deployment of contrastive learning work.

## References

  * [1] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi.  A theoretical analysis of contrastive unsupervised representation learning.  arXiv preprint arXiv:1902.09229, 2019.
  * [2] Han Bao.  On the surrogate gap between contrastive and supervised losses.  ICML, 2022.
  * [3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  Advances in neural information processing systems, 33:9912–9924, 2020.
  * [4] Jiefeng Chen, Frederick Liu, Besim Avci, Xi Wu, Yingyu Liang, and Somesh Jha.  Detecting errors and estimating accuracy on unlabeled data with self-training ensembles.  Advances in Neural Information Processing Systems, 34:14980–14992, 2021.
  * [5] Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré.  Mandoline: Model evaluation under distribution shift.  In International Conference on Machine Learning, pages 1617–1629. PMLR, 2021.
  * [6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  arXiv preprint arXiv:2002.05709, 2020.
  * [7] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [8] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750–15758, 2021.
  * [9] Ching-Yao Chuang, Antonio Torralba, and Stefanie Jegelka.  Estimating generalization under distribution shifts via domain-invariant representations.  arXiv preprint arXiv:2007.03511, 2020.
  * [10] Ciprian A Corneanu, Sergio Escalera, and Aleix M Martinez.  Computing the testing error without a testing set.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2677–2685, 2020.
  * [11] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.  Autoaugment: Learning augmentation strategies from data.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 113–123, 2019.
  * [12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [13] Weijian Deng, Stephen Gould, and Liang Zheng.  What does rotation prediction tell us about classifier accuracy under varying testing environments?  In International Conference on Machine Learning, pages 2579–2589. PMLR, 2021.
  * [14] Weijian Deng and Liang Zheng.  Are labels always necessary for classifier accuracy evaluation?  In Proc. CVPR, 2021.
  * [15] Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  arXiv preprint arXiv:1708.04552, 2017.
  * [16] DC Dowson and BV666017 Landau.  The fréchet distance between multivariate normal distributions.  Journal of multivariate analysis, 12(3):450–455, 1982.
  * [17] Hady Elsahar and Matthias Gallé.  To annotate or not? predicting performance drop under domain shift.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2163–2173, 2019.
  * [18] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88(2):303–338, 2010.
  * [19] Alex Gain and Hava Siegelmann.  Abstraction mechanisms predict generalization in deep neural networks.  In International Conference on Machine Learning, pages 3357–3366. PMLR, 2020.
  * [20] Saurabh Garg, Sivaraman Balakrishnan, Zachary C Lipton, Behnam Neyshabur, and Hanie Sedghi.  Leveraging unlabeled data to predict out-of-distribution performance.  arXiv preprint arXiv:2201.04234, 2022.
  * [21] Gregory Griffin, Alex Holub, and Pietro Perona.  Caltech-256 object category dataset.  2007\.
  * [22] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in neural information processing systems, 33:21271–21284, 2020.
  * [23] Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt.  Predicting with confidence on unseen distributions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1134–1144, 2021.
  * [24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  arXiv preprint arXiv:1911.05722, 2019.
  * [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [26] Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song.  Scaling out-of-distribution detection for real-world settings.  arXiv preprint arXiv:1911.11132, 2019.
  * [27] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  arXiv preprint arXiv:1903.12261, 2019.
  * [28] Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  arXiv preprint arXiv:1610.02136, 2016.
  * [29] Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira.  Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10951–10960, 2020.
  * [30] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [31] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu.  Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data.  Advances in Neural Information Processing Systems, 34:3635–3649, 2021.
  * [32] Peter J Huber.  Robust statistics, volume 523.  John Wiley & Sons, 2004.
  * [33] Jonathan J. Hull.  A database for handwritten text recognition research.  IEEE Transactions on pattern analysis and machine intelligence, 16(5):550–554, 1994.
  * [34] Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio.  Predicting the generalization gap in deep networks with margin distributions.  arXiv preprint arXiv:1810.00113, 2018.
  * [35] Yiding Jiang, Vaishnavh Nagarajan, Christina Baek, and J Zico Kolter.  Assessing generalization of sgd via disagreement.  arXiv preprint arXiv:2106.13799, 2021.
  * [36] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  Advances in neural information processing systems, 30, 2017.
  * [37] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [38] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  Advances in neural information processing systems, 30, 2017.
  * [39] Ya Le and Xuan Yang.  Tiny imagenet visual recognition challenge.  CS 231N, 7(7):3, 2015.
  * [40] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [41] Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo.  Predicting what you already know helps: Provable self-supervised learning.  Advances in Neural Information Processing Systems, 34:309–323, 2021\.
  * [42] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  Advances in neural information processing systems, 31, 2018.
  * [43] Mao Li, Kaiqi Jiang, and Xinhua Zhang.  Implicit task-driven probability discrepancy measure for unsupervised domain adaptation.  Advances in neural information processing systems, 34:25824–25838, 2021.
  * [44] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant.  Enhancing the reliability of out-of-distribution image detection in neural networks.  arXiv preprint arXiv:1706.02690, 2017.
  * [45] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [46] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li.  Energy-based out-of-distribution detection.  Advances in neural information processing systems, 33:21464–21475, 2020.
  * [47] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan.  Learning transferable features with deep adaptation networks.  In International conference on machine learning, pages 97–105. PMLR, 2015.
  * [48] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * [49] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI, pages 69–84. Springer, 2016.
  * [50] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [51] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko.  Visda: The visual domain adaptation challenge.  arXiv preprint arXiv:1710.06924, 2017.
  * [52] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do cifar-10 classifiers generalize to cifar-10?  arXiv preprint arXiv:1806.00451, 2018.
  * [53] Nikunj Saunshi.  A theoretical analysis of contrastive unsupervised representation learning.  ICML, 2019.
  * [54] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [55] Baochen Sun, Jiashi Feng, and Kate Saenko.  Return of frustratingly easy domain adaptation.  In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
  * [56] Xiaoxiao Sun, Yunzhong Hou, Hongdong Li, and Liang Zheng.  Label-free model evaluation with semi-structured dataset representations.  arXiv preprint arXiv:2112.00694, 2021.
  * [57] Korawat Tanwisuth, Xinjie Fan, Huangjie Zheng, Shujian Zhang, Hao Zhang, Bo Chen, and Mingyuan Zhou.  A prototype-oriented framework for unsupervised domain adaptation.  Advances in Neural Information Processing Systems, 34:17194–17208, 2021.
  * [58] Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin.  Predicting neural network accuracy from weights.  arXiv preprint arXiv:2002.11448, 2020.
  * [59] Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin.  Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap.  arXiv preprint arXiv:2203.13457, 2022.
  * [60] Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt.  Predicting out-of-distribution error with the projection norm.  arXiv preprint arXiv:2202.05834, 2022.
  * [61] Yi Zhang, Arushi Gupta, Nikunj Saunshi, and Sanjeev Arora.  On predicting generalization using gans.  arXiv preprint arXiv:2111.14212, 2021.
  * [62] Julian Zilly, Hannes Zilly, Oliver Richter, Roger Wattenhofer, Andrea Censi, and Emilio Frazzoli.  The frechet distance of training and test distribution predicts the generalization gap.  2019\.

##  Appendix A Details of Dataset Setup

In our work, we consider both natural and synthetic distribution shifts in
empirical evaluation. The details of the dataset settings are shown in Table
9.

Table 9: Details of the datasets considered in our paper, where the validation
set that has not undergone data transformation is used as the seed set.

Train set (source) | Valid set (source) | Unseen test set (target)
---|---|---
MNIST (train) | MNIST (valid) | USPS, SVHN
CIFAR-10 (train) | CIFAR-10 (valid) | CIFAR-10.1, 95 CIFAR10-C (Fog and Motion blur, etc.)
CIFAR-100 (train) | CIFAR-100 (valid) | 95 CIFAR-100-C (Fog and Motion blur, etc.)
COCO 2014 (train) | COCO 2014 (valid) | Caltech256 (test), PASCAL VOC 2007 (test), ImageNet (test)
TinyImageNet (train) | TinyImageNet (valid) | 95 TinyImageNet-C (Fog and Motion blur, etc.)

##  Appendix B Baseline Methods

Prediction Score (Pred) [28] It is defined as the maximum softmax output of
the model. If the prediction score of a sample is greater than a given
threshold $\tau$, it is regarded correct.

Entropy Score (Entropy) [36] It is defined as the normalized entropy of
softmax outputs (normalized by $\log K$, where $K$ is the number of classes.
If the entropy score of a sample is less than a given threshold $\tau$, it is
regarded correct.

Proxy Risk [9] This work propose a set of domain-invariant predictors as a
proxy for the unknown, true target labels. They train the check model and
fine-tune it to maximize the disagreement using a separate target training
dataset sampled from the distribution of the test dataset. If the check model
gives a different prediction, it is regarded as an error prediction of the
given model.

Ensemble Average Confidence (Ens. AC) [38] This method involves training
multiple neural networks with different initializations and architectures, and
then combining their predictions to obtain a probabilistic estimate of the
target variable. The model’s accuracy is estimated by the average confidence
calculated from the model ensemble.

Ensemble Method $\mathcal{T}_{RI}$ (Ens. RI) [4] This method uses the same
training algorithm as for the given model to train a model ensemble from
different random initialization (RI). If the model gives a different
prediction, it is regarded as an error prediction of the given model.

Ensemble Method $\mathcal{T}_{RM}$ (Ens. RM) [4] This method is also based on
model ensemble similar to the Ens RI mentioned above, but designed with the
representation matching (RM) technique for domain adaptation, which can
potentially improve the accuracy of the ensemble on some test inputs related
to the training data.

Frechet [14] This method first synthesizes many test sets. And then, it
computes the frechet Distance (FD) between the training set and each of the
test set. Using the (FD, acc) value pairs, it can build a regression model to
estimate the model’s accuracy on an unlabeled test set.

Frechet + $\mu$ \+ $\sigma$ [14] Similar to frechet mentioned above, but adds
the mean and variance values to the Frechet Distance and train a neural
network regression to estimate the testing performance of unlabeled set.

Semi-Structured Dataset Representations (SSDR) [56] Similar to the frechet
Distance based method mentioned above, it uses a semi-structured dataset
feature to regress the model’s accuracy.

Average Confidence (AC) [28] It uses the average of the model’s confidence
(maximum softmax output) as the model’s accuracy on the test set.

Difference of Confidence (DoC) [23] This method uses the difference of
confidences on source and target distribution to regress the model’s accuracy.

Importance-re-weighting (IM) [5] This method estimates the model’s accuracy on
target data by the importance ratios, by using this, the trained model’s
accuracy can be converted to the accuracy on the unlabeled target test set.

Generalization Disagreement Equality (GDE) [35] This method first trains two
models, which are trained on the same training set but with different
initialization or different data ordering. Then, it simultaneously uses the
two models to predict, if their predictions disagree, it’s considered as an
error prediction.

Average Thresholded Confidence (ATC-MC and ATC-NE) [20] This method proposes
average thresholded confidence, which learns a threshold on a score of model
confidence on validation source data and predicts target domain accuracy as
the fraction of unlabeled target points that receive a score above that
threshold. ATC-MC uses the mean confidence as the score, while ATC-NE uses the
negative entropy.

##  Appendix C Training Details

Overall, we train classification along with SimCLR in a multi-task way. Here
are some detailed training parameters under different setups. MNIST We train
LeNet-5 on MNIST. We choose the Adam optimizer, with learning rate $3e^{-4}$,
and train 700 epochs with batch size 2048.

CIFAR-10 and CIFAR-100 For CIFAR-10 and CIFAR-100, the model architecture we
use is DenseNet-40-12 (40 layers with growth rate 12). In the training phase,
we use the SGD optimizer with momentum 0.9, and train 300 epochs with batch
size 128. The initial learning rate is 0.1, and decay by multiplying 0.1 at
epoch 150 and epoch 225.

COCO We use pre-trained ResNet-50, and train 50 epochs with batch size 128.
For training, we use the SGD optimizer with momentum 0.9. The initial learning
rate is $1e^{-3}$, decayed by multiplying 0.1 at epoch 20 and epoch 30.

TinyImageNet We use pre-trained ResNet-50, and train 50 epochs with batch size
128. For training, we use the SGD optimizer with momentum 0.9. The initial
learning rate is $5e^{-3}$, decayed by multiplying 0.1 at epoch 20 and epoch
30.

##  Appendix D Sample Visualization of Synthetic Sets

In section 3.4, we describe how we synthesize sample sets by applying various
transformations on the original seed set. Here we provide some visualizations
for the generated sample sets (see Figure 11, 12).

(a) original set

(b) sample set 1

(c) sample set 2

Figure 11: MNIST sample

##  Appendix E Additional Theoretical Discussion

Recalling to Theorem 1.1, we provide some more detailed discussions of this
theorem at here, including its basic assumptions and a extended theorems under
weaker conditions [59].

###### Assumption E.1

$\forall x,x^{+}\sim p(x,x^{+})$, the labels are deterministic(one-hot) and
consistent: $p(y|x)=p(y|x^{+})$.

Under the premise of satisfying the natural and minimum assumption — label
consistency, we can extend Theorem 1.1 to the situation of any model:

###### Theorem E.1

For any model $f\in\mathcal{F}$, its downstream classification risk
$\mathcal{L}^{\mu}_{CE}(f)$ can be bounded by the contrastive learning risk
$\mathcal{L}_{NCE}(f)$

$\displaystyle\mathcal{L}_{NCE}(f)-\sqrt{Var(f(x)|y})-\frac{1}{2}\sum_{j=1}^{m}\sqrt{Var(f_{j}(x)|y)}$
(9)
$\displaystyle-\mathcal{O}(M^{-1/2})\leq\mathcal{L}_{CE}^{\mu}(f)+\log(M/K)\leq$
$\displaystyle\mathcal{L}_{NCE}(f)+\sqrt{Var(f(x)|y)}+\mathcal{O}(M^{-1/2})$

where
$\mathcal{L}_{CE}^{\mu}(f)=\mathbb{E}_{p(x,y)}[-\log\frac{\exp({f(x)^{T}\mu_{y}})}{\sum_{i=1}^{K}\exp({f(x)^{T}\mu_{i}})}]$,
$\log(M/K)$ is a constant, $\mathcal{O}(M^{-1/2})$ denotes the order of the
approximation error by using $M$ negative samples, $f_{j}(x)$ denotes the
$j$-th coordinate of $f(x)$, and
$Var(f(x)|y)=\mathbb{E}_{p(y)}[\mathbb{E}_{p(x|y)}||f(x)-\mathbb{E}_{p(x|y)}f(x)||^{2}]$
denotes the conditional variance.

(a) original set

(b) sample set 1

(c) sample set 2

Figure 12: CIFAR-10 sample Figure 13: T-SNE clustering visualization of the
classification features on CIFAR-10.1 test set. Different colors correspond to
different classes.

##  Appendix F Tightness Analysis of Bounded Theorem

(a) $\lambda=1$

(b) $\lambda=0.1$

(c) $\lambda=0.01$

(d) $\lambda=0.001$

Figure 14: Scatter plots of the linear correlation with different contrastive
learning task weights ($\lambda$).

In this section, we aim to delve into the tightness of the bounds defined by
Theorem E.1, specifically examining whether the variance term under domain
shift without fine-tuning on the test set can be neglected. Revisiting the
literature [59], eliminating the troublesome variance term must be based on
the concurrent fulfillment of the _Intra-class Connectivity_ and _Perfect
alignment_ assumptions. The _Intra-class Connectivity_ is defined as follows:
for a given data set $\mathcal{D}$, there exists a appropriate augmentation
set $\mathcal{T}$ in which different intra-class samples can overlap with an
aggressive augmentation from $\mathcal{T}$. The _Perfect alignment_ means that
the classifier has a minimum InfoNCE Loss. To this end, we did an interesting
visualization experiment on the CIFAR-10.1 test set.

In Figure 13, we present an intriguing visualization of the CIFAR-10.1 test
set. Our method concurrently attains high contrastive accuracy (88.47%) and
well intra-class clustering effect in CIFAR-10.1. This implies that we are
able to fulfill the above-mentioned assumptions of negligible variance term
under the shifted test set. These results further guaranteed that our CL
accuracy can being a good indicator of classification accuracy in widely
spread unseen test distributions.

##  Appendix G Linear correlation with different training settings.

In this section, we display the scatter plots of linear correlations under
different training settings, as follows in Figure 14, 15, 16, 17, 18, 19, 20,
21, 22, 23, 24.

(a) $aug=0.3$

(b) $aug=0.6$

(c) $aug=0.92$

(d) $aug=1.2$

(e) $aug=1.5$

(f) $aug=1.95$

Figure 15: Scatter plots of the linear correlation with different
RandomResizedCrop augmentation strengths ($aug$).

(a) $brightness=0$

(b) $brightness=0.3$

(c) $brightness=0.5$

(d) $brightness=0.8$

(e) $brightness=1.0$

Figure 16: Scatter plots of the linear correlation with different color
jittering parameter $brightness$.

(a) $contrast=0$

(b) $contrast=0.3$

(c) $contrast=0.5$

(d) $contrast=0.8$

(e) $contrast=1.0$

Figure 17: Scatter plots of the linear correlation with different color
jittering parameter $contrast$.

(a) $saturation=0$

(b) $saturation=0.3$

(c) $saturation=0.5$

(d) $saturation=0.8$

(e) $saturation=1.0$

Figure 18: Scatter plots of the linear correlation with different color
jittering parameter $saturation$.

(a) $hue=0$

(b) $hue=0.1$

(c) $hue=0.2$

(d) $hue=0.4$

(e) $hue=0.5$

Figure 19: Scatter plots of the linear correlation with different color
jittering parameter $hue$.

(a) DenseNet-40-12

(b) ResNet-18

(c) ResNet-34

(d) VGG-11

(e) VGG-19

Figure 20: Scatter plots of the linear correlation with different backbones.

(a) $seed=0$

(b) $seed=21$

(c) $seed=42$

Figure 21: Scatter plots of the linear correlation with different random
seeds.

(a) $num=200$

(b) $num=400$

(c) $num=500$

(d) $num=600$

(e) $num=800$

(f) $num=1000$

Figure 22: Scatter plots of the linear correlation with different sample set
amounts $num$ (each sample set contains 10000 images).

(a) $size=2000$

(b) $size=4000$

(c) $size=6000$

(d) $size=8000$

(e) $size=10000$

Figure 23: Scatter plots of the linear correlation with different sample set
sizes $size$ (using 500 sample sets).

(a) Multi-task

(b) Pretain + fine-tune

Figure 24: Scatter plots of the linear correlation with training ways.


# PL-EESR: PERCEPTUAL LOSS BASED END-TO-END ROBUST SPEAKER REPRESENTATION
EXTRACTION

###### Abstract

Speech enhancement aims to improve the perceptual quality of the speech signal
by suppression of the background noise. However, excessive suppression may
lead to speech distortion and speaker information loss, which degrades the
performance of speaker embedding extraction. To alleviate this problem, we
propose an end-to-end deep learning framework, dubbed PL-EESR, for robust
speaker representation extraction. This framework is optimized based on the
feedback of the speaker identification task and the high-level perceptual
deviation between the raw speech signal and its noisy version. We conducted
speaker verification tasks in both noisy and clean environment respectively to
evaluate our system. Compared to the baseline, our method shows better
performance in both clean and noisy environments, which means our method can
not only enhance the speaker relative information but also avoid adding
distortions.

Index Terms—  Perceptual Loss, End-to-end training, Speaker representation,
Speech enhancement

##  1 Introduction

Speaker embedding refers to a fixed-length continuous-valued vector extracted
from a variable-length utterance [1]. In speaker verification, the extracted
speaker embedding is forwarded to the backend classifier. It is important that
a speaker embedding charaterizes well the speaker’s individuality. Ideal
speaker embedding contains solely the speaker information so that extracted
speaker embeddings are similar for the same speaker, and very different
between speakers. Traditionally, GMM supervectors [2, 3] and i-vector [4] were
used to extract speaker embedding. With the development of deep learning,
d-vector [5] and x-vector [6] were proposed recently. The basic hypothesis of
all of these extraction methods is that the speaker embedding contains solely
one speaker’s information. However, in the presence of background noise and
interference, it is difficult to record only the voice of the interested
speaker.

Fig. 1: A diagram illustrating noise-robust speaker representation using a
separate training scheme (top panel), and an end-to-end training scheme
(bottom panel).

In this work, we seek to extract consistent speaker embedding for a monaural
utterance under either noisy or clean conditions.

Speech enhancement, which aims to keep the target signal of interest and
filter out additive background noise [7], is a conventional method for
handling noisy utterances. Recently, deep learning based methods have shown
significant improvement over the classical methods. These methods usually
generate a mask imposed element-wise on the original signal to estimate the
underlying clean signal [8, 9, 10, 11, 12, 13]. Usually the model is trained
using mean square loss on noisy samples and targets. This function only
guarantees proximity of the enhanced speech to clean speech and simply
accounts for the averaged errors, which leads to artifacts in the output. It
has been observed that speaker recognition accuracy is often hurt by first
enhancing the speech signal and then performing the recognition [14, 15].

In addition to using classification loss, we can use also task-specific loss,
such as perceptual loss, or also called deep feature loss [16]. Perceptual
loss is based on the difference of high-level feature representations
extracted from a pre-trained auxiliary network. It was first proposed to deal
with the image style transfer and super-resolution task [16]. This idea can be
generalized for example to speaker verification task-specified enhancement
training [17, 18]. The approach used in [17, 18] is similar to our method to
compare the activation of enhanced and referenced signals using the auxiliary
network. However, one drawback of [17, 18] is that the auxiliary network is
different from the speaker representation network although these two models
are trained using the same task. The number of the total parameters in their
system is up to 26.1M, which is larger than a standard x-vector network. This
also means that they adopt a two-stage training strategy and the enhancement
model is independent of the speaker verification model. Another drawback is
that the perceptual loss used in [17, 18] relies heavily on clean speech as a
training target which we need to select manually. In this paper, we propose an
end-to-end joint training network, of which the total number of parameters is
9.8M. And we modified the perceptual loss to make it work for clean utterances
as well.

Recent research suggests an end-to-end scheme that combines the speech
enhancement task with another downstream speech task [19] because it reduces
the distortion. As for joint training of speech enhancement with speaker
verification, similar work is [20], in which the enhancement module is trained
using the loss function of speaker identification task to improve the accuracy
of speaker verification in both clean and noisy conditions. Although their
method shows improvement for speaker verification, intuitively the proposed
VoiceID loss based on softmax only tries to enlarge the inter-class
differences among different speakers. However, a robust speaker representation
network should not only maximize the inter-class distance but also minimize
the intra-class variations of the learned embeddings. For this reason, we
proposed the PL-EESR model, which focused on both inter-class and intra-class
distance efficiently.

The contribution of this paper is twofold. Firstly, we proposed a robust end-
to-end speaker representation network optimized using perceptual loss and
cross entropy loss. Our model contains two parts: a task-specific enhancement
module and a speaker embedding extraction module. During training, firstly
these two modules are pre-trained in proper order: the embedding extraction
module is trained on speaker identification task and fixed first; then the
enhancement module is trained using cross entropy loss and perceptual loss
with the embedding extraction module as an auxiliary network. Secondly, these
two modules are fine-tuned simultaneously to reduce potential mismatch. The
idea is illustrated in Fig. 1 To verify the effectiveness of our proposed
network, we performed a speaker verification task on the development and
evaluation part of Speaker In The Wild(SITW). The SITW is viewed as a high SNR
condition in our work. We also simulated a noisy SITW by corrupting SITW with
background noise. This noisy SITW is set as low SNR condition.

##  2 perceptual loss

###  2.1 Problem description

Let us denote the $X$ to be the log-mel spectrogram of a noisy utterance. The
utterance is corrupted by the additive background noise so that it can be
written as $X=S+N$, where $S$ and $N$ are log-mel spectrogram of the clean
utterance and the noise. The target of speech enhancement in our work is to
estimate a mask $M$ and we can get the estimated clean spectrogram $\hat{S}$
by the element-wise multiplication of the mask and the input utterance:
$\hat{S}=X\otimes M$.

Conventionally, a speech enhancement network is trained using the Euclidean
distance between the estimated clean spectrogram and the ground-truth:
$\mathcal{L}(S,\hat{S})=\rVert S-\hat{S}\rVert_{2}$. However, this method may
cause some speaker related information loss or distortion in the resultant
enhanced spectrogram. If we apply the enhanced spectrogram on the speaker
embedding extraction task directly, it may cause degraded performance,
especially for high SNR conditions. We believe this is because the Euclidean
distance can not capture the perceptual difference between estimated and
ground-truth spectrogram.

Table 1: Dataset and loss function for each stage.
Stage | Clean data | Noisy data | Loss
---|---|---|---
Pre-training 1 | vox_clean_aug | - | CE loss
Pre-training 2; Finetune | vox_clean | vox_noisy |  CE loss; Percep -tual loss
vox_clean |  | vox_noisy_aug
---
(MUSAN)
| vox_clean_aug
---
(RIRs)
| vox_noisy_aug
---
(RIRs)

###  2.2 Perceptual loss

It is believed that a trained deep network shows different activations in
their hidden layers, which helps the model to learn characteristics from the
input feature. Based on this assumption, the perceptual loss, which is the
distance of activations in hidden layers of the trained auxiliary network with
estimated and ground-truth spectrogram as input respectively, is proposed in
[16].

According to the principle of perceptual loss, the auxiliary network decides
what information to be kept. In this paper, our goal is to enhance the speaker
relative information. Therefore, the speaker verification network can be
trained first, then it is fixed as the auxiliary network to extract perceptual
loss for the enhancement module:

$\mathcal{L}(X)=\sum_{i=1}^{j}\rVert\phi_{i}(X)-\phi_{i}(\theta(X))\rVert_{2}$
(1)

where the $\theta$ and the $\phi$ are the enhancement module and auxiliary
speaker embedding extraction module respectively, and $\phi_{i}$ denotes the
$i_{th}$ layer of embedding extraction module. Equation (1) is what [17, 18]
used.

##  3 METHOD

###  3.1 Architecture

We now introduce the end-to-end model that incorporates enhancement process
and speaker representation together. The flow chart is illustrated in Fig. 2
(d).

As shown in Fig. 2 (d), at first, the feature (30-dimensional log-Mel
spectrogram) is extracted from the speech signal in the time domain. It was
used for both the enhancement module and the representation module.

To remove the effect of the corpus channel, as mentioned in [21], we applied
the channel normalization to help the enhancement module converge rapidly. The
mean $\mu$ and standard deviation $\sigma$ of the clean and noisy channel are
computed using the clean and noisy training set respectively. Noisy utterance
is normalized before enhancement, as follow:

$X(k)\leftarrow\frac{X(k)-\mu_{n}(k)}{\sigma_{n}(k)^{2}}$ (2)

An inverse normalization is then performed on the enhanced speech:

$\hat{S}(k)\leftarrow\hat{S}(k)\times\sigma_{c}(k)^{2}+\mu_{c}(k)$ (3)

where $\sigma_{n}$ and $\mu_{n}$ are the statistical parameters of the noisy
training set, and $\sigma_{c}$ and $\mu_{c}$ are corresponding clean version.
The $k$ denotes the index of Mel-filter and $X(k)$ is the feature
corresponding to $k_{th}$ Mel-filter. After the inverse-channel-normalization,
instance normalization is applied before speaker representation extraction:

$\hat{s}_{i}(k)\leftarrow\frac{\hat{s}_{i}(k)-\mu_{i}(k)}{\sigma_{i}(k)^{2}}$
(4)

where $\sigma_{i}$ and $\mu_{i}$ are the statistical parameters of each
utterance $\hat{s}_{i}$.

To exploit the temporal context information effectively, three bidirectional
long short-term memory (BLSTM) layers and one fully-connected layer are
stacked as the enhancement module. The number of features in the hidden state
of BLSTM is 128. The activation of the sigmoid is applied on the output of
fully-connected layer to generate the mask in the range 0 to 1.

Our verification module adopts the x-vector. The first five TDNN layers
extract frame-level information. Then two fully connected layers extract
information in utterance-level. The two phases are connected by a global
average pooling layer. Since we don’t know the effect of activation in each
layer on our task, all of the five TDNN layers and the first fully connected
layer are used to compute perceptual loss in the training stage. The speaker
embedding was extracted from the first fully connected layer during inference.

###  3.2 Training strategy

The perceptual as in (1) was used in [17] to train the speech enhancement
network. The idea is illustrated in Fig. 2(b): the clean utterance is used as
the target and the noisy utterance is trained to show the same activations of
the auxiliary network as the target does. However, the gradient of (1) only
concerns the enhancement for noisy utterance. It may cause some potential
detrimental to the clean utterance. Therefore, we propose the training scheme
as in Fig. 2(c) and (d): the mask is generated and applied on clean utterance
as well, and the enhancement module is optimized using the gradient of both
noisy utterance and clean utterance. The goal of the enhancement module is to
make sure the noisy utterance and the clean utterance have a consistent
representation. So the perceptual loss in our system is modified as:

$\mathcal{L}_{pcptl}(\theta(X),\theta(S))=\sum_{i=1}^{j}\rVert\phi_{i}(\theta(X))-\phi_{i}(\theta(S))\rVert_{2}$
(5)

where the $X$ and $S$ are the noisy and clean utterance respectively. As
mentioned in Section 3.1, $j$ equals $6$ in our case.

Fig. 2: Block diagrams of (a) optimizing speech enhancement module using cross
entropy loss; (b) optimizing speech enhancement module using perceptual loss
on noisy speech; (c)optimizing speech enhancement module using cross entropy
loss and perceptual loss on both noisy and clean speech; (d)optimizing speech
enhancement and speaker extraction modules jointly using cross entropy loss
and perceptual loss on both noisy and clean speech. The “Ch Norm” and “Ch Inv-
Norm” denote channel normalization and channel inverse-normalization, “Inst
Norm” is the instance normalization.

A good speaker representation network should minimize the distance between the
utterances belonging to the same speaker, while maximize the distance of
different speakers in embedding vector space. The cross entropy loss in (6) is
a common loss function in the classification problem to enlarge the distance
of different speakers.

$\mathcal{L}_{ce}(\phi(\theta(S)),y)=-log\frac{exp(\phi(\theta(S))[y])}{\sum_{y^{\prime}}exp(\phi(\theta(S))[y^{\prime}])}$
(6)

In our case, the $\phi(\theta(S))$ is the speaker label estimated by the
speaker representation module following the speech enhancement process, and
$y$ is the ground-truth speaker label. Intuitively, the softmax in (6) enlarge
the intra-class discrimination, but it shows no effect on inter-class
distance. To remedy this limitation, we train our system using cross entropy
loss and perceptual loss jointly:

$\mathcal{L}=\lambda\times\mathcal{L}_{pcptl}+(1-\lambda)\times\mathcal{L}_{ce}$
(7)

where the perceptual loss can be viewed as the criterion for inter-class
distance. The $\lambda$ is a constant for balancing the intra-class and inter-
class distance. We set the value of $\lambda$ as 0.5 in our work. (Code is
available online111Source code: https://github.com/mmmmayi/PL-EESR.)

##  4 Experiments

###  4.1 Datasets

Training set: we combine VoxCeleb1 and VoxCeleb2 [22] as our training set.
Background noise in this dataset is inevitable because it is collected from
YouTube. But we still set samples in this dataset as clean utterances in our
experiment and we name it as vox_clean. It should be noted that speakers who
are in both VoxCele2 and Speakers in the Wild (SITW) [23] are removed from
voxceleb2 because we use the SITW as our test set. The vox_clean is augmented
with noise drawn from MUSAN [24] and is convolved with simulated RIRs [25].
The augmentation process is based on Kaldi SITW/v2 recipe [26]. The augmented
vox_clean is called vox_clean_aug. To simulate speech in noisy conditions, we
corrupt the vox_clean using the noise randomly selected from MUSAN as noisy
utterance. The SNR of each noisy utterance is randomly selected from the range
of 0 to 20 excluding $\\{0,5,10,15,20\\}$. This data set is named vox_noisy.
Each sample in the vox_noisy has a corresponding clean version in the
vox_clean. To be consistent with the vox_clean_aug, the vox_noisy is augmented
in the same procedure and we call it vox_noisy_aug.
Test set: We use two datasets to evaluate the performance of our method in
noisy and clean conditions. The first dataset is SITW. Being similar to
VoxCeleb, the speech in SITW is collected from open-source media channels as
well. We use the development and evaluation part core-core condition of SITW
to evaluate our network in clean condition. We generated the noisy Speakers in
the Wild (NSITW) as our second test set. The noise used to corrupt the SITW is
provided by DNS-challenge [27]. We manually select the noise categories which
are similar to MUSAN (bubble, music and plain noise). Totally 282 categories
are used and the SNR of each noisy utterance in NSITW is randomly selected
from set $\\{0,5,10,15,20\\}$. The NSITW is used to evaluate the performance
of our model in noisy conditions.

###  4.2 Feature

In VoxCeleb1 and VoxCeleb2, multiple scenarios were given for each speaker,
and there are several utterances in each scenario. We concatenated utterances
in the same scenario to one training utterance. We generate 30-D log Mel-
spectrogram as the feature from each training utterance. The sample rate of
all utterances in our work is 16kHz. Our feature is extracted with a Hann
window of 400 frames width and 160 frames hop size. This feature is used for
both speech enhancement module and speaker representation module, but we need
to note that mean and variance normalization (MVN) is performed only for
speaker representation. The training utterance whose length is less than 500
frames and the speaker whose utterance amount is less than 10 are removed.
Therefore, there are 5916 speakers in the training set. We randomly clipped a
consecutive 300-frame segment from each training utterance to optimize our
network during training. We do not use voice activity detection in the
training stage because the silence segment has been removed from VoxCeleb1 and
VoxCeleb2, but energy based voice activity detection is performed in the test
stage.

###  4.3 Training

Our training process includes two stages: pre-training stage and finetune
stage.

For the pre-training stage, firstly, we trained our speaker representation
module (Pre-training 1) with a batch size of 512 and optimizer of stochastic
gradient descent (SGD). The initial learning rate is 0.2 and it decreases by
half when the loss decrease ratio in each epoch is less than 0.01. The early
stopping scheme was applied as soon as the learning rate decreases twice in
succession. As shown in Table 1, the vox_clean_aug is used to pre-train the
speaker representation module. Only the cross entropy loss on the speaker
label is used in this stage. We use the trained speaker representation module
as one of our baseline models to show the effect of speech enhancement as
well.

The trained speaker representation module is fixed as the auxiliary network to
pre-train our speaker enhancement module (Pre-training 2). The enhancement
module is optimized using the perceptual loss and cross entropy loss jointly.
As summarized in Table 1, three pairs of clean and noisy utterance are used in
this stage. For vox_noisy, their clean data is the corresponding sample in
vox_clean. For vox_noisy_aug, two types of augmentation are included just like
vox_clean_aug. When utterances in vox_noisy are augmented with MUSAN, we
expect our enhancement module to remove the effect of augmentation as well. So
the clean data for these utterances are their corresponding utterances in
vox_clean. However, for the utterances in vox_noisy_aug which are convolved
with RIRs, we don’t expect our enhancement work for dereverberation. So their
corresponding clean utterances should come from vox_clean_aug in which samples
augmented with RIRs. The batch size of this stage is 128, which includes 64
noisy utterances and their clean version. We use the Adadelta optimizer and
initial learning rate of 0.3 in this stage. The learning rate decrease and
early stopping scheme are identical with Pre-training 1.

After pre-training, the speech enhancement module and speaker representation
module are finetuned together to reduce potential mismatch. Except the initial
rate is 0.0001 in this stage, both training setting and data set are same as
what we used in Pre-training 2.

###  4.4 Evaluation

To evaluate the performance of our system to extract robust speaker
representation in both clean and noisy environments, we conduct the speaker
verification task using both SITW and NSITW. In the inference stage,
vox_noisy_aug passing through the enhancement module is used to train the PLDA
backend. The Equal Error Rate (EER) and minimum Detection Cost Function
(minDCF) with target prior $p=0.05$ are used to evaluate our system.

###  4.5 Baseline

As we mentioned in Section 4.3, one of our baseline models is our proposed
model without speech enhancement module (PL-EESR w/o enh), which is used to
compare the effect of speech enhancement on our system. Then we use the
standard x-vector as our second baseline model (x-vector). We trained the
model using the Kaldi SITW/v2 recipe. We set the x-vector as the state-of-the-
art model for the speaker verification task. We trained PL-EESR w/o enh using
the setting of Pre-training 1 which is introduced in Section 4.3. This is the
difference between it and x-vector.

##  5 Results

Table 2: Comparison of our PL-EESR with two baseline models on SITW and NSITW. Test set | System | Dev |  | Eval
---|---|---|---|---
|  | EER | DCF |  | EER | DCF
SITW | x-vector | 2.965 | 0.1946 |  | 3.417 | 0.2241
PL-EESR w/o enh | 3.656 | 0.2215 |  | 3.442 | 0.2210
PL-EESR | 2.851 | 0.1805 |  | 2.460 | 0.1690
NSITW | x-vector | 6.816 | 0.3685 |  | 7.190 | 0.4131
PL-EESR w/o enh | 7.470 | 0.3835 |  | 8.066 | 0.4550
PL-EESR | 4.698 | 0.3007 |  | 5.272 | 0.3334
Table 3: Comparison of our end-to-end training scheme with different training
settings. “Joint” and “Separate” denote whether the enhancement module and
speaker representation module are fine-tuned together. “Noisy” and “Clean”
denote whether the network is optimized on the gradient of noisy utterance or
clean utterance.
Test set | System | Training | CE Loss Objective | Perceptual Loss Objective | Dev |  | Eval
---|---|---|---|---|---|---|---
|  |  |  |  | EER | DCF |  | EER | DCF
SITW | (a) | Separate | Noisy | - | 2.734 | 0.1906 |  | 2.788 | 0.1833
(b) | Separate | - | Noisy | 3.812 | 0.2274 |  | 3.827 | 0.2400
(c) | Separate | Noisy; Clean | Noisy; Clean | 2.811 | 0.1919 |  | 2.570 | 0.1804
(d) | Joint | Noisy; Clean | Noisy; Clean | 2.851 | 0.1805 |  | 2.460 | 0.1690
NSITW | (a) | Separate | Noisy | - | 5.930 | 0.3351 |  | 5.931 | 0.3706
(b) | Separate | - | Noisy | 7.624 | 0.3951 |  | 8.124 | 0.4611
(c) | Separate | Noisy; Clean | Noisy; Clean | 5.160 | 0.3161 |  | 5.632 | 0.3539
(d) | Joint | Noisy; Clean | Noisy; Clean | 4.698 | 0.3007 |  | 5.272 | 0.3334

###  5.1 Baseline results

Table 2 shows the comparison of the baselines and our PL-EESR on the
development set and evaluation set of SITW (top) and NSITW (bottom)
respectively. The text in bold is the best performance for each metric in
different conditions. We can find in clean condition, compared without
enhancement processing, our method achieves 28.5% and 23.5% relative
improvements in terms of EER and minDCF respectively in evaluation set, 22.0%
and 18.5% in the development set. In a noisy environment, the relative
improvements in terms of EER and minDCF are 34.6% and 26.7% for the evaluation
set, 37.1% and 21.6% for the development set. Compared to the standard
x-vector, which can be seen as the state of the art model for speaker
representation, the relative improvements are 28.0% and 24.6% in terms of EER
and minDCF for evaluation set, 3.8% and 7.2% for the development set for SITW.
For NSITW, the improvements in evaluation set and development set are 26.7%,
19.3% and 31.1%, 18.4% respectively.

We can conclude from this comparison that our proposed method is beneficial to
speaker representation extraction from both clean and noisy utterances.

###  5.2 Overall comparisons

Table 3 shows the performance of our end-to-end model with different training
settings. Firstly, we trained the speech enhancement module and speaker
verification module separately to verify the effect of the perceptual loss,
which means there is no finetune stage in System (a)-(c) in Table 3 and all of
these three systems are optimized in the stage of Pre-training 2. In the first
experiment, we use only the feedback of the speaker classification task, which
is the cross entropy loss on predicted speaker ID with noisy utterance as
input. The system flow chart is shown in 2(a), and this training scheme is
based on the basic idea in [20]. The results are summarized as System (a) in
Table 3.

System (b) is the original perceptual loss used in [17]. In this experiment,
the clean utterance is set as the target and the module is trained on the
gradient of noisy utterance as shown in 2 (b). This performance is summarized
in System (b) in Table 3.

Then we modified the perceptual loss function. Since the cross entropy loss on
the speaker label can be used to enlarge the inter-class distance, the
perceptual loss in our system focus on decrease the intra-class distance.
Specifically, after computing the perceptual loss, the module is optimized on
both the gradient of noisy utterance and clean utterance. To make sure this
idea works for utterance in clean environments as well, the cross entropy is
computed for both noisy utterance and clean utterance. The flow chart of this
experiment is shown in Fig. 2 (c) and results are summarized in System (c) of
Table 3.

System (d) is used to verify the facility of training jointly to decrease the
mismatch between these two modules. The only difference between System (c) and
System (d) in Table 3 is that System(d) is trained in finetune stage.

Comparing the results of System (a)-(c) in Table 3 with baseline performance
in Table 2, we can find using System (b) harmful for both clean and noisy
environments. Perhaps this degradation is because the vox_clean still involved
some background noise although we set it as the target in training. However,
both System (a) and System (c) outperform the baseline. This shows the
necessity of end-to-end speech enhancement to reduce distortion and
information loss. Finally, the System (d) gains the most improvements in all
of these four systems in most conditions except the EER of devaluation set of
SITW.

##  6 Conclusion

Motivated by the unsatisfactory performance of speech enhancement applied on
speaker embedding extraction task, we proposed the end-to-end training scheme
for robust speaker representation. The model is trained on loss functions that
aim at mapping the noisy and clean utterances to the identical representation
as well as classifying speaker labels. The experiment results show that our
method outperforms baseline models in both clean and noisy environments.

## References

  * [1] K. A. Lee, V. Vestman, and T. Kinnunen,  “ASVtorch toolkit: Speaker verification with deep neural networks,”  SoftwareX, vol. 14, pp. 100697, 2021.
  * [2] W. M. Campbell, D. E. Sturim, and D. A. Reynolds,  “Support vector machines using gmm supervectors for speaker verification,”  IEEE signal processing letters, vol. 13, no. 5, pp. 308–311, 2006\.
  * [3] P. Kenny, P. Ouellet, N. Dehak, V. Gupta, and P. Dumouchel,  “A study of interspeaker variability in speaker verification,”  IEEE Transactions on Audio, Speech, and Language Processing, vol. 16, no. 5, pp. 980–988, 2008.
  * [4] N. Dehak, R. Dehak, P. Kenny, N. Brümmer, P. Ouellet, and P. Dumouchel,  “Support vector machines versus fast scoring in the low-dimensional total variability space for speaker verification,”  in Tenth Annual conference of the international speech communication association, 2009.
  * [5] E. Variani, X. Lei, E. McDermott, I. L. Moreno, and J. Gonzalez-Dominguez,  “Deep neural networks for small footprint text-dependent speaker verification,”  in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2014, pp. 4052–4056.
  * [6] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur,  “X-vectors: Robust dnn embeddings for speaker recognition,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5329–5333.
  * [7] P. C. Loizou,  Speech enhancement: theory and practice,  CRC press, 2007.
  * [8] U. Kjems, J. B. Boldt, M. S. Pedersen, T. Lunner, and D. Wang,  “Role of mask pattern in intelligibility of ideal binary-masked noisy speech,”  The Journal of the Acoustical Society of America, vol. 126, no. 3, pp. 1415–1426, 2009.
  * [9] Y. Wang, A. Narayanan, and D. Wang,  “On training targets for supervised speech separation,”  IEEE/ACM transactions on audio, speech, and language processing, vol. 22, no. 12, pp. 1849–1858, 2014.
  * [10] H. Yu, W. Zhu, and Y. Yang,  “Constrained ratio mask for speech enhancement using dnn,”  Proc. Interspeech 2020, pp. 2427–2431, 2020.
  * [11] D. S. Williamson, Y. Wang, and D. Wang,  “Complex ratio masking for monaural speech separation,”  IEEE/ACM transactions on audio, speech, and language processing, vol. 24, no. 3, pp. 483–492, 2015.
  * [12] A. Pandey and D. Wang,  “Learning complex spectral mapping for speech enhancement with improved cross-corpus generalization,”  Proc. Interspeech 2020, pp. 4511–4515, 2020.
  * [13] T. Liu, R. K. Das, M. Madhavi, S. Shen, and H. Li,  “Speaker-utterance dual attention for speaker and utterance verification,”  arXiv preprint arXiv:2008.08901, 2020.
  * [14] S. O. Sadjadi and J. H. Hansen,  “Assessment of single-channel speech enhancement techniques for speaker identification under mismatched conditions,”  in Eleventh Annual Conference of the International Speech Communication Association, 2010.
  * [15] Y. Shi, Q. Huang, and T. Hain,  “Robust speaker recognition using speech enhancement and attention model,”  arXiv preprint arXiv:2001.05031, 2020.
  * [16] J. Johnson, A. Alahi, and F. Li,  “Perceptual losses for real-time style transfer and super-resolution,”  in European conference on computer vision. Springer, 2016, pp. 694–711.
  * [17] S. Kataria, P. S. Nidadavolu, J. Villalba, N. Chen, P. Garcia-Perera, and N. Dehak,  “Feature enhancement with deep feature losses for speaker verification,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7584–7588.
  * [18] S. Kataria, P. S. Nidadavolu, J. Villalba, and N. Dehak,  “Analysis of deep feature loss based enhancement for speaker verification,”  arXiv preprint arXiv:2002.00139, 2020.
  * [19] N. Hou, C. Xu, J. T. Zhou, E. S. Chng, and H. Li,  “Multi-task learning for end-to-end noise-robust bandwidth extension,”  Proc. Interspeech 2020, pp. 4069–4073, 2020.
  * [20] S. Shon, H. Tang, and J. Glass,  “Voiceid loss: Speech enhancement for speaker verification,”  arXiv preprint arXiv:1904.03601, 2019.
  * [21] A. Pandey and D. Wang,  “On cross-corpus generalization of deep learning based speech enhancement,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2489–2499, 2020.
  * [22] J. S. Chung, A. Nagrani, and A. Zisserman,  “Voxceleb2: Deep speaker recognition,”  arXiv preprint arXiv:1806.05622, 2018.
  * [23] M. McLaren, L. Ferrer, D. Castan, and A. Lawson,  “The speakers in the wild (sitw) speaker recognition database.,”  in Interspeech, 2016, pp. 818–822.
  * [24] D. Snyder, G. Chen, and D. Povey,  “Musan: A music, speech, and noise corpus,”  arXiv preprint arXiv:1510.08484, 2015.
  * [25] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur,  “A study on data augmentation of reverberant speech for robust speech recognition,”  in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, pp. 5220–5224.
  * [26] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, et al.,  “The kaldi speech recognition toolkit,”  in IEEE 2011 workshop on automatic speech recognition and understanding. IEEE Signal Processing Society, 2011, number CONF.
  * [27] C. K. Reddy, H. Dubey, V. Gopal, R. Cutler, S. Braun, H. Gamper, R. Aichner, and S. Srinivasan,  “Icassp 2021 deep noise suppression challenge,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6623–6627.