# Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling

Haw-Shiuan Chang Ruei-Yao Sun11footnotemark: 1 22footnotemark: 2
Amazon
USA
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>&Kathryn Ricci11footnotemark: 1 Andrew McCallum
CICS, UMass, Amherst
140 Governors Dr., Amherst, MA, USA
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
indicates equal contribution The work is done while the authors were at UMass

###### Abstract

Ensembling BERT models often significantly improves accuracy, but at the cost
of significantly more computation and memory footprint. In this work, we
propose Multi-CLS BERT, a novel ensembling method for CLS-based prediction
tasks that is almost as efficient as a single BERT model. Multi-CLS BERT uses
multiple CLS tokens with a parameterization and objective that encourages
their diversity. Thus instead of fine-tuning each BERT model in an ensemble
(and running them all at test time), we need only fine-tune our single Multi-
CLS BERT model (and run the one model at test time, ensembling just the
multiple final CLS embeddings). To test its effectiveness, we build Multi-CLS
BERT on top of a state-of-the-art pretraining method for BERT (Aroca-Ouellette
and Rudzicz, 2020). In experiments on GLUE and SuperGLUE we show that our
Multi-CLS BERT reliably improves both overall accuracy and confidence
estimation. When only 100 training samples are available in GLUE, the Multi-
CLS BERT${}_{\text{Base}}$ model can even outperform the corresponding
BERT${}_{\text{Large}}$ model. We analyze the behavior of our Multi-CLS BERT,
showing that it has many of the same characteristics and behavior as a typical
BERT 5-way ensemble, but with nearly 4-times less computation and memory.

##  1 Introduction

BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al.,
2019) is one of the most widely-used language model (LM) architectures for
natural language understanding (NLU) tasks. We often fine-tune the pretrained
BERT or its variants such as RoBERTa (Liu et al., 2019) so that the LMs learn
to aggregate all the contextualized word embeddings into a single CLS
embedding for a downstream text classification task.

Figure 1: Comparison of Multi-CLS BERT and the classic BERT ensemble. Multi-
CLS BERT only ensembles the multiple CLS embeddings in one BERT encoder rather
than ensemble multiple BERT encoders with different parameter weights.

During fine-tuning, different initializations and different training data
orders significantly affect BERT’s generalization performance, especially with
a small training dataset (Dodge et al., 2020; Zhang et al., 2021a; Mosbach et
al., 2021). One simple and popular solution to the issue is to fine-tune BERT
model multiple times using different random seeds and ensemble their
predictions to improve its accuracy and confidence estimation. Although very
effective, the memory and computational cost of ensembling a large LM is often
prohibitive (Xu et al., 2020; Liang et al., 2022). Naturally, we would like to
ask, “Is it possible to ensemble BERT models at no extra cost?”

To answer the question, we propose Multi-CLS BERT, which enjoys the benefits
of ensembling without sacrificing efficiency. Specifically, we input the
multiple CLS tokens to BERT and encourage the different CLS embeddings to
aggregate the information from different aspects of the input text. As shown
in Figure 1, the proposed Multi-CLS BERT shares all the hidden states of the
input text and only ensembles different ways of aggregating the hidden states.
Since the input text is usually much longer than the number of inputted CLS
embeddings, Multi-CLS BERT is almost as efficient as the original BERT.

Allen-Zhu and Li (2020) discovered that the key of an effective ensembling
model is the diversity of individual models and the models trained using
different random seeds have more diverse predictions compared to simply using
dropout (Srivastava et al., 2014; Gal and Ghahramani, 2016) or averaging the
weights of the models during training (Fort et al., 2019). To ensure the
diversity of CLS embeddings without fine-tuning Multi-CLS BERT using multiple
seeds, we propose several novel diversification techniques. For example, we
insert different linear layers into the transformer encoder for different CLS
tokens. Furthermore, we propose a novel re-parametrization trick to prevent
the linear layers from learning the same weights during fine-tuning.

We test the effectiveness of these techniques by modifying the multi-task
pretraining method proposed by Aroca-Ouellette and Rudzicz (2020), which
combines four self-supervised losses. In our experiments, we demonstrate that
the resulting Multi-CLS BERT can significantly improve the accuracy on GLUE
(Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a), especially when the
training sizes are small. Similar to the BERT ensemble model, we further show
that multiple CLS embeddings significantly reduce the expected calibration
error, which measures the quality of prediction confidence, on the GLUE
benchmark.

###  1.1 Main Contributions

  * •

We propose an efficient ensemble BERT model that does not incur any extra
computational cost other than inserting a few CLS tokens and linear layers
into the BERT encoder. Furthermore, we develop several diversification
techniques for pretraining and fine-tuning the proposed Multi-CLS BERT
model.111We release our code at https://github.com/iesl/multicls/.

  * •

We improve the GLUE performance reported in Aroca-Ouellette and Rudzicz (2020)
using a better and more stable fine-tuning protocol and verify the
effectiveness of its multi-task pretraining methods in GLUE and SuperGLUE with
different training sizes.

  * •

Building on the above state-of-the-art pretraining and fine-tuning for BERT,
our experiments and analyses show that Multi-CLS BERT significantly
outperforms the BERT due to its similarity to a BERT ensemble model. The
comprehensive ablation studies confirm the effectiveness of our
diversification techniques.

##  2 Method

In sections 2.1 and 2.2, we first review its state-of-the-art pretraining
method from Aroca-Ouellette and Rudzicz (2020). In Section 2.3, we modify one
of its losses, quick thoughts (QT), to pretrain our multiple embedding
representation. In Section 2.4, we encourage the CLS embeddings to capture the
fine-grained semantic meaning of the input sequence by adding hard negatives
during the pretraining. To diversify the CLS embeddings, we modify the
transformer encoder in Section 2.5 and propose a new reparametrization method
during the fine-tuning in Section 2.6.

Figure 2: Our MCQT, SO, MLM, and TFIDF loss, which are a modification of
multi-task pretraining proposed in Aroca-Ouellette and Rudzicz (2020). The
multi-CLS quick thought (MCQT) loss maximizes the CLS similarities between a
sequence (sentences 1 and 2) and the next sequence (sentences 3 and 4) while
minimizing the CLS similarities to other random sequences and the sequence
after the next one (sentences 5 and 6). Notice that sentence 4 is inputted
before sentence 3 because the sentence order is swapped for the SO loss.

###  2.1 Multi-task Pretraining

After testing many self-supervised losses, Aroca-Ouellette and Rudzicz (2020)
find that combining the masked language modeling (MLM) loss, TFIDF loss,
sentence ordering (SO) loss (Sun et al., 2020), and quick thoughts (QT) loss
(Logeswaran and Lee, 2018) could lead to the best performance.

The MLM loss is to predict the masked words and the TFIDF loss is to predict
the importance of the words in the document. Each input text sequence consists
of multiple sentences. They swap the sentence orders in some input sentences
and use the CLS embedding to predict whether the order is swapped in the SO
loss. Finally, QT loss is used to encourage the CLS embeddings of the
consecutive sequences to be similar.

To improve the state-of-the-art pretraining method, we modify the multi-task
pretraining method by using multiple CLS embeddings to represent the input
sequence and using non-immediate consecutive sentences as the hard negative.
Our training method is illustrated in Figure 2.

###  2.2 Quick Thoughts Loss

Two similar sentences tend to have the same label in a downstream application,
so pretraining should pull the CLS embeddings of these similar sentences
closer. The QT loss achieves this goal by assuming consecutive text sequences
are similar and encouraging their CLS embeddings to be similar.

Aroca-Ouellette and Rudzicz (2020) propose an efficient way of computing QT
loss in a batch by evenly splitting each batch with size $B$ into two parts.
The first part contains $B/2$ text sequences randomly sampled from the
pretrained corpus, and the second part contains each of the $B/2$ sentences
that are immediately subsequent to those in the first part. Then, for each
sequence in the first part, they use the consecutive sequence in the second
part as the positive example and the other $B/2-1$ sequences as the negative
examples. We can write the QT loss for the sequences containing sentences 1,
2, 3, and 4 as

$L_{QT}(s^{1-2},s^{3-4})=-\log(\frac{\exp(\text{Logit}^{QT}_{s^{1-2},s^{3-4}})}{\sum_{s}\exp(\text{Logit}^{QT}_{s^{1-2},{s}})}),$
(1)

where $s$ is the sentences in the second part of the batch,
$\text{Logit}^{QT}_{s^{1-2},s^{3-4}}=(\frac{{\bm{c}}^{{1-2}}}{||{\bm{c}}^{{1-2}}||})^{T}\frac{{\bm{c}}^{{3-4}}}{||{\bm{c}}^{{3-4}}||}$
is the score for classifying sequence $s^{3-4}$ as the positive example,
$\frac{{\bm{c}}^{{1-2}}}{||{\bm{c}}^{{1-2}}||}$ is the L2-normalized CLS
embedding for sentences 1 and 2. The normalization is intended to stabilize
the pretraining by limiting the gradients’ magnitudes.

###  2.3 Multiple CLS Embeddings

A text sequence could have multiple facets; two sequences could be similar in
some facets but dissimilar in others, especially when the text sequences are
long. The QT loss squeezes all facets of a sequence into a single embedding
and encourages all facets of two consecutive sequences to be similar,
potentially causing information loss.

Some facets might better align with the goal of a downstream application. For
example, the facets that contain more sentiment information would be more
useful for sentiment analysis. To preserve the diverse facet information
during pretraining, we propose multi-CLS quick thoughts loss (MCQT). The loss
integrates two ways of computing the similarity of two sequences. The first
way computes the cosine similarity between the most similar facets, and the
second computes the cosine similarity between the summations of all facets. We
linearly combine the two methods as the logit of the two input sequences:

$\displaystyle\text{Logit}^{MC}_{s^{1-2},s^{3-4}}=\lambda\max_{i,j}(\frac{{\bm{c}}_{i}^{{1-2}}}{||{\bm{c}}_{i}^{{1-2}}||})^{T}\frac{{\bm{c}}_{j}^{{3-4}}}{||{\bm{c}}_{j}^{{3-4}}||}+$
$\displaystyle(1-\lambda)(\frac{\sum_{i}{\bm{c}}_{i}^{{1-2}}}{||\sum_{i}{\bm{c}}_{i}^{{1-2}}||})^{T}\frac{\sum_{j}{\bm{c}}_{j}^{{3-4}}}{||\sum_{j}{\bm{c}}_{j}^{{3-4}}||}.$
(2)

where $\lambda$ is a constant hyperparameters; ${\bm{c}}^{1-2}_{k}$ and
${\bm{c}}^{3-4}_{k}$ are the CLS embeddings of sentences 1-2 and sentences
3-4, respectively.

The first term only considers the most similar facets to allow some facets to
be dissimilar. Furthermore, the term implicitly diversifies CLS embeddings by
considering each CLS embedding independently. In contrast, the second term
encourages the CLS embeddings to work collaboratively, as in a typical
ensemble model, and also let every CLS embedding receive gradients more
evenly. Notice that we sum the CLS embeddings before the normalization so that
the encoder could predict the magnitude of each CLS embedding as its weight in
the summation.

To show that the proposed method can improve the state-of-the-art pretraining
methods, we keep the MLM loss and TFIDF loss unchanged. For the sentence
ordering (SO) loss, we project the $K$ hidden states ${\bm{h}}^{c}_{k}$ into
the embedding ${\bm{h}}^{SO}$ with the hidden state size $D$ for predicting
the sentence order: ${\bm{h}}^{SO}=L^{SO}(\oplus_{k}{\bm{h}}^{c}_{k})$, where
$\oplus_{k}{\bm{h}}^{c}_{k}$ is the concatenation of $K$ hidden states with
size $K\times D$.

###  2.4 Hard Negative

For a large transformer-based LM, distinguishing the next sequence from random
sequences could be easy. The LM can achieve low QT loss by outputting nearly
identical CLS embeddings for the sentences with the same topic while ignoring
the fine-grained semantic information (Papyan et al., 2020). In this case,
multiple CLS embeddings might become underutilized.

The hard negative is a common method of adjusting the difficulties of the
contrastive learning (Baldini Soares et al., 2019; Cohan et al., 2020). Our
way of collecting hard examples is illustrated in the bottom-left block of
Figure 2. To efficiently add the hard negatives in the pretraining, we split
the batch into three parts. For each sequence in the first part, we would use
its immediate next sequence in the second part as the positive example, use
the sequence after the next one in the third part as the hard negative, and
use all the other sequences in the second or the third part as the easy
negatives. We select such sequence after the next one as our hard negatives
because the sequence usually share the same topic with the input sequence but
is more likely to have different fine-grained semantic facets compared to the
immediate next sequence.

After adding the hard negative, the modified QT loss of the three consecutive
sequences becomes

$\displaystyle L_{MCQT}(s^{1-2},s^{3-4},s^{5-6})=$
$\displaystyle-\log\left(\frac{\exp(\text{Logit}^{MC}_{s^{1-2},s^{3-4}})}{\sum\limits_{s\in\\{s^{3-4},...,s^{5-6},...\\}}\exp(\text{Logit}^{MC}_{s^{1-2},{s}})}\right)$
$\displaystyle-\log\left(\frac{\exp(\text{Logit}^{MC}_{s^{5-6},s^{3-4}})}{\sum\limits_{s\in\\{s^{3-4},...,s^{1-2},...\\}}\exp(\text{Logit}^{MC}_{s^{5-6},{s}})}\right),$
(3)

where MCQT refer to multi-CLS quick thoughts, $\\{s^{3-4},...,s^{5-6},...\\}$
are all the sequences in the second and the third part, and
$\\{s^{3-4},...,s^{1-2},...\\}$ are all the sequences in the first and the
second part.

Figure 3: The architecture of Multi-CLS BERT encoder that is built on
BERT${}_{\text{Base}}$ model. The different linear layers are applied to the
hidden states corresponding to different CLS tokens to increase the diversity
of the resulting CLS embeddings.

###  2.5 Architecture-based Diversification

Initially, we simply input multiple special CLS tokens ([C1], …, [CK]) after
the original CLS token, [CLS0], and take the corresponding hidden states as
the CLS embeddings, but we found that the CLS embeddings quickly become almost
identical during the pretraining.

Subsequently, instead of using the same final transformation head $H^{QT}$ for
all CLS hidden states, we use a different linear layer $L_{O,k}$ in the final
head $H_{k}^{MC}$ to transform the hidden state ${\bm{h}}^{c}_{k}$ for the
$k$th CLS. We set the bias term in $L_{O,k}$ to be the constant ${\bm{0}}$
because we want the differences between the CLS to be dynamic and context-
dependent.

Nevertheless, even though we differentiate the resulting CLS embeddings
${\bm{c}}_{k}=H_{k}^{MC}({\bm{h}}^{c}_{k})$, the hidden states
${\bm{h}}^{c}_{k}$ before the transformation usually still collapse into
almost identical embeddings.

To solve the collapsing problem, we insert multiple linear layers $L_{l,k}$
into the transformer encoder. In Figure 3, we illustrate our encoder
architecture built on the BERT${}_{\text{Base}}$ model. After the $4$th
transformer layer, we insert the layers $L_{4,k}$ to transform the hidden
states before inputting them to the $5$th layer. Similarly, we insert
$L_{8,k}$ between the $8$th transformer layer and $9$th transformer layer. For
BERT${}_{\text{Large}}$, we insert $L_{l,k}(.)$ after layer 8 and layer 16.
Notice that although the architecture looks similar to the adapter (Houlsby et
al., 2019) or prefix-tuning (Li and Liang, 2021), our purpose is to diversify
the CLS embeddings rather than freezing parameters to save computational time.

###  2.6 Fine-Tuning

As shown in Figure 3, we input multiple CLS tokens into the BERT encoder
during fine-tuning and pool the corresponding CLS hidden states into the
single CLS embedding for each downstream task fine-tuning in order to avoid
overfitting and increasing computational overhead. As a result, we can use the
same classifier architecture on top of Multi-CLS BERT and BERT, which also
simplifies their comparison.

We discover that simply summing all the CLS hidden states still usually makes
the hidden states and the inserted linear layers (e.g., $L_{O,k}$) almost
identical after fine-tuning. To avoid collapsing, we aggregate the CLS hidden
states by proposing a novel re-parameterization trick:

${\bm{c}}^{MCFT}=\sum_{k}\left(L^{FT}_{O,k}({\bm{h}}^{c}_{k})\right),$ (4)

where
$L^{FT}_{O,k}({\bm{h}}^{c}_{k})=({\mathbf{W}}_{O,k}-\frac{1}{K}\sum_{k^{\prime}}{\mathbf{W}}_{O,k^{\prime}}){\bm{h}}^{c}_{k}$,
and ${\mathbf{W}}_{O,k}$ is the linear weights of $L_{O,k}$. Then, if all the
$L^{FT}_{O,k}$ become identical (i.e., $\forall
k,{\mathbf{W}}_{O,k}=\frac{1}{K}\sum_{k^{\prime}}{\mathbf{W}}_{O,k^{\prime}}$),
$L^{FT}_{O,k}({\bm{h}}^{c}_{k})={\bm{0}}={\bm{c}}^{MCFT}$. However, gradient
descent would not allow the model to constantly output the zero vector, so
$L^{FT}_{O,k}$ remains different during the fine-tuning.

##  3 Experiments

The parameters of neural networks are more restricted as more training samples
are available (MacKay, 1995) and the improvement of deep ensemble models comes
from the diversity of individual models (Fort et al., 2019), so the benefits
of ensembling are usually more obvious when the training set size is smaller.
Therefore, in addition to using the full training dataset, we also test the
settings where the models are trained by 1k samples (Zhang et al., 2021a) or
100 samples from each task in GLUE (Wang et al., 2019b) or SuperGLUE (Wang et
al., 2019a). Another benefit of the 1k- and 100-sampling settings is that the
average scores would be significantly influenced by most datasets rather than
by only a subset of relatively small datasets (Card et al., 2020).

###  3.1 Experiment Setup

To accelerate the pretraining experiments, we initialize the weights using the
pretrained BERT models (Devlin et al., 2019) and continue the pretraining
using different loss functions on Wikipedia 2021 and BookCorpus Zhu et al.
(2015).

All of the methods are based on uncased BERT as in Aroca-Ouellette and Rudzicz
(2020). We compare the following methods:

  * •

Pretrained: The pretrained BERT model released from Devlin et al. (2019).

  * •

MTL: Pretraining using the four losses selected in Aroca-Ouellette and Rudzicz
(2020): MLM, QT, SO, and TFIDF. We remove the continue learning procedure used
in ERNIE (Sun et al., 2020) because we find that simply summing all the losses
leads to better performance (see our ablation study in Section 3.3).

  * •

Ours (K=5, $\mathbf{\lambda}$): The proposed Multi-CLS BERT method using 5 CLS
tokens. We show the results of setting $\lambda=\\{0,0.1,0.5,1\\}$ in
subsection 2.3. We reduce the maximal sentence length by 5 to accommodate the
extra 5 CLS tokens.

  * •

Ours (K=1): We set $K=1$ in our method to verify the effectiveness of using
multiple embeddings. During fine-tuning, the CLS embedding is a linear
transformation of the single facet $\text{CLS}=L_{O,1}({\bm{h}}^{f}_{1})$.

|  |  | GLUE | SuperGLUE
---|---|---|---|---
Configuration $\downarrow$ | Model Name $\downarrow$ | Model Size $\downarrow$ | 100 | 1k | Full | 100* | 1k* | Full
BERT Base | Pretrained | 109.5M | 55.71 | 71.67 | 82.05 | 57.18 | 61.55 | 65.04
|  |  $\pm$ 0.62 |  $\pm$ 0.15 |  $\pm$ 0.08 |  $\pm$ 0.43 |  $\pm$ 0.37 |  $\pm$ 0.36
MTL | 109.5M | 59.29 | 73.26 | 83.30$\dagger$ | 57.50 | 62.94 | 66.33
|  |  $\pm$ 0.27 |  $\pm$ 0.13 |  $\pm$ 0.07 |  $\pm$ 0.41 |  $\pm$ 0.36 |  $\pm$ 0.33
Ours (K=1) | 111.3M | 57.84 | 73.28 | 83.40 | 57.31 | 63.35 | 66.29
|  |  $\pm$ 0.32 |  $\pm$ 0.13 |  $\pm$ 0.07 |  $\pm$ 0.35 |  $\pm$ 0.18 |  $\pm$ 0.18
Ours (K=5, $\lambda=0$) | 118.4M | 61.54 | 74.14 | 83.41 | 58.29 | 63.71 | 66.80
|  |  $\pm$ 0.32 |  $\pm$ 0.12 |  $\pm$ 0.07 |  $\pm$ 0.33 |  $\pm$ 0.26 |  $\pm$ 0.25
Ours (K=5, $\lambda=0.1$) | 118.4M | 61.80 | 74.10 | 83.47 | 58.20 | 63.61 | 66.74
|  |  $\pm$ 0.35 |  $\pm$ 0.13 |  $\pm$ 0.05 |  $\pm$ 0.31 |  $\pm$ 0.27 |  $\pm$ 0.26
| Ours (K=5, $\lambda=0.5$) | 118.4M | 60.49 | 74.02 | 83.47 | 58.41 | 63.78 | 66.80
|  |  |  $\pm$ 0.35 |  $\pm$ 0.12 |  $\pm$ 0.08 |  $\pm$ 0.38 |  $\pm$ 0.25 |  $\pm$ 0.24
| Ours (K=5, $\lambda=1$) | 118.4M | 59.86 | 73.75 | 83.43 | 57.84 | 63.56 | 66.39
|  |  |  $\pm$ 0.34 |  $\pm$ 0.14 |  $\pm$ 0.07 |  $\pm$ 0.40 |  $\pm$ 0.22 |  $\pm$ 0.22
BERT Large | MTL | 335.2M | 61.39 | 75.30 | 84.13 | 59.03 | 65.21 | 69.16
|  |  $\pm$ 0.37 |  $\pm$ 0.27 |  $\pm$ 0.11 |  $\pm$ 0.54 |  $\pm$ 0.38 |  $\pm$ 0.37
Ours (K=1) | 338.3M | 59.19 | 75.35 | 84.59 | 57.35 | 64.67 | 69.24
|  |  $\pm$ 0.43 |  $\pm$ 0.21 |  $\pm$ 0.07 |  $\pm$ 0.42 |  $\pm$ 0.43 |  $\pm$ 0.41
Ours (K=5, $\lambda=0$) | 350.9M | 63.19 | 75.73 | 84.51 | 59.46 | 65.43 | 69.56
|  |  $\pm$ 0.49 |  $\pm$ 0.26 |  $\pm$ 0.05 |  $\pm$ 0.44 |  $\pm$ 0.38 |  $\pm$ 0.31
Ours (K=5, $\lambda=0.1$) | 350.9M | 64.24 | 76.27 | 84.61 | 59.88 | 65.58 | 70.03
|  |  $\pm$ 0.40 |  $\pm$ 0.12 |  $\pm$ 0.08 |  $\pm$ 0.43 |  $\pm$ 0.26 |  $\pm$ 0.25
Ours (K=5, $\lambda=0.5$) | 350.9M | 63.02 | 75.95 | 84.49 | 59.42 | 65.84 | 69.79
|  |  |  $\pm$ 0.42 |  $\pm$ 0.10 |  $\pm$ 0.08 |  $\pm$ 0.34 |  $\pm$ 0.25 |  $\pm$ 0.25
| Ours (K=5, $\lambda=1$) | 350.9M | 62.07 | 75.85 | 84.61 | 58.74 | 65.00 | 69.04
|  |  |  $\pm$ 0.45 |  $\pm$ 0.17 |  $\pm$ 0.07 |  $\pm$ 0.50 |  $\pm$ 0.29 |  $\pm$ 0.27

Table 1: The macro average scores on the development set. All numbers are
percentages. The standard errors are shown as the confidence intervals. We
make the best scores of the model built on BERT${}_{\text{Base}}$ boldface and
similar for the models built on BERT${}_{\text{Large}}$. $\dagger$The number
is much higher than $81.4$, the GLUE score reported by Aroca-Ouellette and
Rudzicz (2020) because we continue training from the pretrained BERT and we
use better fine-tuning hyperparameters. *The scores do not contain ReCoRD in
SuperGLUE.333In SuperGLUE 100 and 1k, we exclude the ReCoRD dataset because
the performance of all models is much worse than the most frequent class
baseline.

The GLUE and SuperGLUE scores are significantly influenced by the pretraining
random seeds (Sellam et al., 2021) and fine-tuning random seeds (Dodge et al.,
2020; Zhang et al., 2021a; Mosbach et al., 2021). To stably evaluate the
performance of different pretraining methods, we pretrain models using four
random seeds and fine-tune each pretrained model using four random seeds, and
report the average performance on the development set across all 16 random
seeds. To further stabilize the fine-tuning process and reach better
performance, we follow the fine-tuning suggestions from Zhang et al. (2021a)
and Mosbach et al. (2021), including training longer, limiting the gradient
norm, and using Adam Kingma and Ba (2015) with bias term and warmup.

###  3.2 Main Results

Our results are presented in footnote 3. We can see that Ours (K=5) is
consistently better than other baselines and that the improvement is larger in
datasets with fewer training samples. For example, in GLUE 100, it achieves
$61.80$ on average using BERT${}_{\text{Base}}$ with 118.4M parameters, which
outperforms MTL using BERT${}_{\text{Large}}$ with 335.2M parameters
($61.39$). Please see Appendix E for the scores of individual tasks. MTL
significantly improves the scores of original BERT model (Pretrained),
confirming the effectness of the QT, SO, and TFIDF losses. Compared to MTL,
Ours (K=1) is slightly better in GLUE 1k and GLUE Full, but worse in GLUE 100.

We observe that $\lambda=0.1$ usually performs well, which justifies the
inclusion of both the highest logit and average logit in subsection 2.3. The
$\lambda=0$ model has significantly worse performance only in
BERT${}_{\text{Large}}$ model. This suggests that the benefits of Multi-CLS
BERT depend on our pretraining method and maximizing the highest logit
stabilizes the pretraining of a larger model.

###  3.3 Ablation Study

In our ablation studies, we would like to test the effectiveness of the design
choices in our baseline MTL and our best model, Ours (K=5,
$\mathbf{\lambda=0.1}$). The model variants we test include:

  * •

MLM only: Removing the QT, SO, and TFIDF losses in MTL. That is, we simply
continue training Pretrained using only the MLM loss.

  * •

CMTL+: The best pretrained method reported in Aroca-Ouellette and Rudzicz
(2020). It uses the continual learning method (Sun et al., 2020) to weight
each loss in MTL.

  * •

MLM+SO+TFIDF: MTL without the QT loss.

  * •

No Inserted Layers: Removing the $L_{l,k}(.)$ in the transformer encoder from
our method.

  * •

No Hard Negative: Removing the hard negatives described in Section 2.4 from
our method.

  * •

Sum Aggregation: Simply summing the facets (i.e., using $L_{O,k}$ to replace
$L^{FT}_{O,k}$ in Equation 4). This baseline removes the proposed
reparametrization trick to test its effectiveness.

  * •

Default: Ours (K=k, $\mathbf{\lambda=0.1}$), where $k=\\{1,3,5,10\\}$.

  * •

SWA: Stochastic weight averaging (Ruppert, 1988; Izmailov et al., 2018)
averages the weights along the optimization trajectory.

  * •

Ensemble on Dropouts: Running the forward pass of Ours (K=1) with dropout
using 5 different seeds and averaging their prediction probabilities for each
class in each task.

  * •

Ensemble on FT Seeds: Fine-tuning Ours (K=1) or Ours (K=5,
$\mathbf{\lambda=0.1}$) using 5 different seeds and averaging their prediction
probabilities.

|  |  | GLUE | SuperGLUE*
---|---|---|---|---
Model $\downarrow$ | Model Description $\downarrow$ | K $\downarrow$ | 100 | 1k | 100 | 1k
Baselines (BERT Base) | Pretrained | 1 | 56.85 | 71.68 | 57.90 | 62.14
MLM only | 1 | 55.38 | 70.74 | 57.39 | 61.77
CMTL+ | 1 | 58.65 | 72.57 | 56.88 | 62.63
MLM + SO + TFIDF | 1 | 60.35 | 72.65 | 57.88 | 62.60
MTL | 1 | 59.53 | 73.12 | 57.51 | 62.95
Ours (BERT Base) | No Inserted | 1 | 58.06 | 73.18 | 57.97 | 63.34
Layers | 5 | 60.12 | 73.35 | 56.46 | 62.00
No Hard | 1 | 58.44 | 73.30 | 57.19 | 63.33
Negative | 5 | 61.77 | 74.18 | 58.89 | 63.86
Sum Aggregation | 5 | 58.87 | 73.94 | 57.41 | 63.82
Default | 1 | 57.76 | 73.30 | 57.53 | 63.22
3 | 61.09 | 73.95 | 57.85 | 63.31
5 | 62.62 | 74.49 | 58.82 | 63.86
10 | 60.99 | 73.59 | 58.25 | 62.82
SWA | 1 | 57.31 | 72.91 | - | -
Ensemble on Dropouts | 1 | 58.45 | 72.86 | - | -
| Ensemble on FT Seeds | 1 | 60.07 | 75.20 | - | -
| 5 | 63.34 | 75.35 | - | -
Ours (BERT Large) | No Hard | 1 | 60.36 | 75.69 | 58.47 | 65.04
Negative | 5 | 63.23 | 75.77 | 60.33 | 65.75
Default | 1 | 60.01 | 76.03 | 57.38 | 65.10
5 | 64.33 | 76.38 | 59.99 | 65.51

Table 2: The macro average scores on the development set for our ablation
study. We highlight the best performance after excluding the ensemble
baselines, which require much more computation. The scores are different in
footnote 3 because we use two pretraining random seeds instead of four in the
ablation study. SWA refers to Stochastic weight averaging (Izmailov et al.,
2018). *SuperGLUE score does not contain ReCoRD.

Our results are presented in Table 2. We can see that continuing training
using MLM only loss degrades the performance, which indicates that our
improvement does not come from training BERT longer. Removing QT loss results
in mixed results. The better performance of MTL compared to CMTL+ suggests
that the continual training technique used in Aroca-Ouellette and Rudzicz
(2020) is harmful with our evaluation settings.

Removing the inserted layers (No Inserted Layers) or removing the re-
parametrization trick (Sum Aggregation) makes the performance of Ours (K=5,
$\mathbf{\lambda=0.1}$) close to the Ours (K=1) baseline. This result
highlight the importance of diversity of CLS embeddings. The performance of
Ours (K=3) and Ours (K=10) is usually better than Ours (K=1), but are worse
than Ours (K=5). In both BERT${}_{\text{Base}}$ and BERT${}_{\text{Large}}$
models, removing hard negatives degrades the GLUE scores but slightly
increases the SuperGLUE scores.

In GLUE 100 and 1k, we do not get good results by using other efficient
ensembling methods such as SWA and Ensemble on Dropouts. This suggests that
the gradient descent trajectory and different dropout maps might not produce
prediction diversity sufficient for an effective BERT ensemble model (Fort et
al., 2019).

On the other hand, ensembling the models that are fine-tuned using different
random seeds indeed boosts the performance at the expense of high
computational costs. The ensembled Multi-CLS BERT (Ensemble on FT Seeds K=5)
still outperforms the ensembled K=1 baseline, but ensembling makes their
performance differences smaller. These results imply that the improvements of
Multi-CLS BERT overlap with the improvements of a BERT ensemble model.

| Inference | GLUE* (ECE)
---|---|---
| Time (s) | 100 | 1k
Ours (K=1) | 0.2918 | 25.22 | 19.32
|  $\pm$ 0.0002 |  $\pm$ 1.99 |  $\pm$ 1.64
Ours (K=5, $\lambda=0.1$) | 0.3119 | 15.46 | 17.01
|  $\pm$ 0.0004 |  $\pm$ 1.79 |  $\pm$ 1.64
Ensemble of Ours (K=1) | 1.4590 | 13.85 | 10.80
|  $\pm$ 0.0012 |  $\pm$ 0.97 |  $\pm$ 0.88

Table 3: The comparison of inference time and expected calibration error
(ECE). The confidence intervals are standard errors. *Only includes the
classification tasks (i.e., excludes STS-b).

| GLUE* 100 | GLUE* 1k
---|---|---
Multi-CLS vs ENS | 32.57 | 41.35
Dropout vs ENS | 37.17 | 45.53
Least vs ENS | 39.57 | 48.85
ENS vs ENS | 38.67 | 50.14

Table 4: The overlapping ratio of the top 20% most uncertain examples using
different uncertainty estimation methods. ENS is ensemble of Ours (K=5,
$\lambda=0.1$) with different fine-tuning seeds. *Only includes the
classification tasks (i.e., excludes STS-b).

###  3.4 Ensembling Analysis

We compare the inference time and expected calibration error (ECE) (Naeini et
al., 2015) of using multiple CLS embeddings, using a single CLS embedding, and
ensembling BERT models with different fine-tuning seeds in Table 3. A lower
ECE means a better class probability estimation. For example, if a model
outputs class 1 with 0.9 probability for 100 samples, ECE = 0 means that 90
samples among them are indeed class 1.

Table 3 shows that Ours (K=5) is much faster than the BERT ensemble and almost
as efficient as Ours (K=1), because a BERT ensemble needs to run for multiple
forward passes and we reduce the maximal sentence length by 5 in Ours (K=5).
Additionally, the ECE of Ours (K=5) is lower than Ours (K=1) but not as low as
the ECE from ensembling BERT models with different fine-tuning seeds. That is,
without significantly increasing inference time, ensembling multiple CLS
embeddings improves the output confidence, even though not as much as
ensembling BERT models.

Next, we analyze the correlation of uncertainty estimation from different
methods in Table 4. When ensembling BERT models with different dropout maps
(Dropout) or different fine-tuning seeds (ENS), we can estimate the prediction
uncertainty by the variance of the prediction probability from each individual
BERT model. We can also use one minus prediction probability as the
uncertainty (Least). In Multi-CLS, we measure the disagreement among the CLS
embeddings as the uncertainty444See Section B.3 for details and would like to
see how many top-20% most uncertain samples from the disagreement of CLS
embeddings are also the top-20% most uncertain samples for a BERT ensemble
model.

Table 4 reports the ratio of the number of the overlapping uncertain samples
from two estimation methods to the number of 20% samples in the development
set. We can see that the ratio from Multi-CLS BERT and the BERT ensemble model
(Multi-CLS vs ENS) is close to the ratios from other uncertainty estimations
and the BERT ensemble model (Dropout vs ENS, Least vs ENS, and ENS vs ENS).
This shows that different CLS embeddings can classify the uncertain samples
differently, as is the case for the different BERT models in a BERT ensemble
model. In Appendix C, we visualize the CLS embeddings of some uncertain
samples to show how different CLS embeddings solve a task in different ways.

##  4 Related Work

Due to its effectiveness, ensembling BERT in a better or more efficient way
has recently attracted researchers’ attention. Nevertheless, the existing
approaches often need to rely on distillation (Xu et al., 2020; Matsubara et
al., 2022; Zuo et al., 2022) or still require significant extra computational
cost during training and testing Kobayashi et al. (2022); Liang et al. (2022).

Some recent vision models can also achieve ensembling almost without extra
computational cost by sharing the weights (Wen et al., 2020), partitioning the
model into subnetworks (Havasi et al., 2021; Zhang et al., 2021b), or
partitioning the embeddings (Lavoie et al., 2022). However, it is unknown if
the approaches are applicable to the pretraining and fine-tuning of language
models.

Similar to Multi-CLS BERT, mixture of softmax (MoS) (Yang et al., 2018) also
uses multiple embeddings to improve the pretraining loss. Recently, Narang et
al. (2021); Tay et al. (2022) have found that MoS is one of the few
modifications that can improve on the original BERT architecture on the NLU
benchmarks. Nevertheless, Narang et al. (2021) also point out that MoS
requires significant extra training cost to compute the multiplication between
each hidden state and all the word embeddings in the vocabulary.

Chang et al. (2021) propose represent the sentence using multiple embeddings
and demonstrate its improvement over the single embedding baseline on
unsupervised sentence similarity tasks. Similar to our subsection 2.3, their
non-negative sparse coding loss also encourages multiple sentence embeddings
to collaborate during pretraining. Nevertheless, our loss is more
computationally efficient and is designed to improve downstream supervised
tasks rather than similarity tasks.

Some approaches also represent a text sequence using multiple embeddings, such
as contextualized word embeddings (Khattab and Zaharia, 2020; Luan et al.,
2021) for information retrieval applications, sentence embeddings (Liu and
Lapata, 2019; Iter et al., 2020; Mysore et al., 2022; Sul and Choi, 2023), or
entity pair embeddings (Xue et al., 2022). However, the goal of this approach
is to improve the representation of a relatively long text sequence and it is
unknown if its benefits could be extended to the GLUE tasks that require fine-
tuning and often involve only one or two sentences.

##  5 Conclusion

In this work, we propose representing the input text using $K$ CLS embeddings
rather than using the single CLS embedding in BERT. Compared to BERT, Multi-
CLS BERT significantly increases the GLUE and SuperGLUE scores and reduces the
expected calibration error in GLUE, while its only added cost is to reduce the
maximal text length by $K$ and add a little extra time for computing the
inserted linear transformations. Therefore, we recommend the wide use of
multiple CLS embeddings for the almost free performance gain.

To solve the collapsing problem of CLS embeddings, we modify the pretraining
loss, BERT architecture, and fine-tuning loss. The ablation study shows that
all of these modifications contribute to the performance improvement of Multi-
CLS BERT. In our analyses for investigating the source of the improvement, we
find that  a) ensembling the original BERT leads to greater improvement than
ensembling the Multi-CLS BERT and  b) the disagreement of different CLS
embeddings highly correlates with the disagreement of the BERT models from
different fine-tuning seeds.  Both findings support our perspective that
Multi-CLS BERT is an efficient ensembling method.

##  6 Acknowledgement

We thank Jay Yoon Lee and the anonymous reviewers for their constructive
feedback. This work was supported in part by the Center for Data Science and
the Center for Intelligent Information Retrieval, in part by the Chan
Zuckerberg Initiative under the project Scientific Knowledge Base
Construction, in part by the IBM Research AI through the AI Horizons Network,
in part using high performance computing equipment obtained under a grant from
the Collaborative R&D Fund managed by the Massachusetts Technology
Collaborative, and in part by the National Science Foundation (NSF) grant
numbers IIS-1922090 and IIS-1763618. Any opinions, findings, conclusions, or
recommendations expressed in this material are those of the authors and do not
necessarily reflect those of the sponsor.

##  7 Limitations

Our methods are evaluated using BERT as many previous recent work such as
Aroca-Ouellette and Rudzicz (2020); Dodge et al. (2020); Sellam et al. (2021);
Gu et al. (2021); Qin et al. (2021); Wang et al. (2021); Xu et al. (2022);
Zhou and Srikumar (2022); Hou et al. (2022); Wang et al. (2022a); Liu et al.
(2022); Zhao et al. (2022); Zhou et al. (2022); Zheng et al. (2022); Fu et al.
(2022). Our limited computational resources do not allow us to conduct similar
experiments on RoBERTa (Liu et al., 2019) because pretraining RoBERTa requires
much powerful GPUs and a much larger CPU memory to store the corpora. For the
similar reason, we are unable to test our methods on larger language models.
We are also not able to conduct a more comprehensive search for the
pretraining and fine-tuning hyperparameters. We haven’t tested if the multiple
embedding representation could also improve other language model architectures
such as XLNet (Yang et al., 2019), or other fine-tuning methods such as prompt
(Radford et al., 2019; Li and Liang, 2021), or adapter (Houlsby et al., 2019;
Wang et al., 2022b).

Our conclusion mainly draws from the overall scores of GLUE or SuperGLUE
benchmarks, which only include English datasets and might contain some dataset
selection bias (Dehghani et al., 2021).

Although much more efficient, Multi-CLS BERT is still worse than the classic
BERT ensemble model in terms of expected calibration error and accuracy when
more training data are available (e.g., in GLUE 1k). We also do not know if
Multi-CLS BERT could provide efficient and high-quality uncertainty estimation
for other applications such as active learning Pop and Fulop (2018).

##  8 Ethical and Broader Impact

Multi-CLS BERT can provide better confidence estimation compared to BERT while
better efficiency compared to the classic BERT ensemble. This work might
inspire prospective efficient ensembling approaches that produce more robust
predictions (Clark et al., 2019b) with lower the energy consumption.

On the other hand, the readers of the paper might not notice the limitations
of the study (e.g., the confidence estimation of Multi-CLS BERT is still
sometimes far behind the classic BERT ensemble model) and mistakenly believe
that Multi-CLS BERT has all the benefits of the classic BERT ensemble model.

## References

  * Allen-Zhu and Li (2020) Zeyuan Allen-Zhu and Yuanzhi Li. 2020.  Towards understanding ensemble, knowledge distillation and self-distillation in deep learning.  _ArXiv preprint_ , abs/2012.09816.
  * Aroca-Ouellette and Rudzicz (2020) Stéphane Aroca-Ouellette and Frank Rudzicz. 2020.  On Losses for Modern Language Models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4970–4981, Online. Association for Computational Linguistics.
  * Baldini Soares et al. (2019) Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019\.  Matching the blanks: Distributional similarity for relation learning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2895–2905, Florence, Italy. Association for Computational Linguistics.
  * Bentivogli et al. (2009) Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009.  The fifth pascal recognizing textual entailment challenge.  In _TAC_.
  * Card et al. (2020) Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020.  With little power comes great responsibility.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9263–9274, Online. Association for Computational Linguistics.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14.
  * Chang et al. (2021) Haw-Shiuan Chang, Amol Agrawal, and Andrew McCallum. 2021.  Extending multi-sense word embedding to phrases and sentences for unsupervised semantic applications.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 6956–6965.
  * Clark et al. (2019a) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019a.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2924–2936.
  * Clark et al. (2019b) Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019b.  Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4069–4082, Hong Kong, China. Association for Computational Linguistics.
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020.  SPECTER: Document-level representation learning using citation-informed transformers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2270–2282, Online. Association for Computational Linguistics.
  * De Marneffe et al. (2019) Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019.  The commitmentbank: Investigating projection in naturally occurring discourse.  In _proceedings of Sinn und Bedeutung_ , volume 23, pages 107–124.
  * Dehghani et al. (2021) Mostafa Dehghani, Yi Tay, Alexey A Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. 2021.  The benchmark lottery.  _ArXiv preprint_ , abs/2107.07002.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dodge et al. (2020) Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.  _ArXiv preprint_ , abs/2002.06305.
  * Dolan and Brockett (2005) Bill Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Third International Workshop on Paraphrasing (IWP2005)_.
  * Fort et al. (2019) Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. 2019.  Deep ensembles: A loss landscape perspective.  _ArXiv preprint_ , abs/1912.02757.
  * Fu et al. (2022) Zhiyi Fu, Wangchunshu Zhou, Jingjing Xu, Hao Zhou, and Lei Li. 2022.  Contextual representation learning beyond masked language modeling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2701–2714.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016_ , volume 48 of _JMLR Workshop and Conference Proceedings_ , pages 1050–1059. JMLR.org.
  * Gu et al. (2021) Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han. 2021.  On the transformer growth for progressive bert training.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5174–5180.
  * Havasi et al. (2021) Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew Mingbo Dai, and Dustin Tran. 2021.  Training independent subnetworks for robust prediction.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Hou et al. (2022) Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, and Denny Zhou. 2022.  Token dropping for efficient bert pretraining.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3774–3784.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR.
  * Iter et al. (2020) Dan Iter, Kelvin Guu, Larry Lansing, and Dan Jurafsky. 2020.  Pretraining with contrastive sentence objectives improves discourse performance of language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4859–4870, Online. Association for Computational Linguistics.
  * Izmailov et al. (2018) Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. 2018.  Averaging weights leads to wider optima and better generalization.  In _Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018_ , pages 876–885. AUAI Press.
  * Khashabi et al. (2018) Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018.  Looking beyond the surface: A challenge set for reading comprehension over multiple sentences.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 252–262.
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  Colbert: Efficient and effective passage search via contextualized late interaction over BERT.  In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_ , pages 39–48. ACM.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kobayashi et al. (2022) Sosuke Kobayashi, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2022.  Diverse lottery tickets boost ensemble from a single pretrained model.  In _Challenges & Perspectives in Creating Large Language Models_.
  * Lavoie et al. (2022) Samuel Lavoie, Christos Tsirigotis, Max Schwarzer, Kenji Kawaguchi, Ankit Vani, and Aaron Courville. 2022.  Simplicial embeddings in self-supervised learning and downstream classification.  _ArXiv preprint_ , abs/2204.00616.
  * Levesque et al. (2012) Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012.  The winograd schema challenge.  In _Thirteenth international conference on the principles of knowledge representation and reasoning_.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Liang et al. (2022) Chen Liang, Pengcheng He, Yelong Shen, Weizhu Chen, and Tuo Zhao. 2022.  Camero: Consistency regularized ensemble of perturbed language models with weight sharing.  _ArXiv preprint_ , abs/2204.06625.
  * Liu et al. (2022) Qin Liu, Rui Zheng, Bao Rong, Jingyi Liu, Zhihua Liu, Zhanzhan Cheng, Liang Qiao, Tao Gui, Qi Zhang, and Xuan-Jing Huang. 2022.  Flooding-x: Improving bert’s resistance to adversarial attacks via loss-restricted fine-tuning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5634–5644.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text summarization with pretrained encoders.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3730–3740.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv preprint_ , abs/1907.11692.
  * Logeswaran and Lee (2018) Lajanugen Logeswaran and Honglak Lee. 2018.  An efficient framework for learning sentence representations.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Luan et al. (2021) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021.  Sparse, dense, and attentional representations for text retrieval.  _Transactions of the Association for Computational Linguistics_ , 9:329–345.
  * MacKay (1995) David JC MacKay. 1995.  Probable networks and plausible predictions-a review of practical bayesian methods for supervised neural networks.  _Network: computation in neural systems_ , 6(3):469.
  * Matsubara et al. (2022) Yoshitomo Matsubara, Luca Soldaini, Eric Lind, and Alessandro Moschitti. 2022.  Ensemble transformer for efficient and accurate ranking tasks: an application to question answering systems.  _ArXiv preprint_ , abs/2201.05767.
  * Mosbach et al. (2021) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021.  On the stability of fine-tuning BERT: misconceptions, explanations, and strong baselines.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Mysore et al. (2022) Sheshera Mysore, Arman Cohan, and Tom Hope. 2022.  Multi-vector models with textual guidance for fine-grained scientific document similarity.  In _NAACL_.
  * Naeini et al. (2015) Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. 2015.  Obtaining well calibrated probabilities using bayesian binning.  In _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA_ , pages 2901–2907. AAAI Press.
  * Narang et al. (2021) Sharan Narang, Hyung Won Chung, Yi Tay, Liam Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, and Colin Raffel. 2021.  Do transformer modifications transfer across implementations and applications? In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5758–5773, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Papyan et al. (2020) Vardan Papyan, XY Han, and David L Donoho. 2020.  Prevalence of neural collapse during the terminal phase of deep learning training.  _Proceedings of the National Academy of Sciences_ , 117(40):24652–24663.
  * Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019.  Wic: the word-in-context dataset for evaluating context-sensitive meaning representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1267–1273.
  * Pop and Fulop (2018) Remus Pop and Patric Fulop. 2018.  Deep ensemble bayesian active learning: Addressing the mode collapse issue in monte carlo dropout via ensembles.  _ArXiv preprint_ , abs/1811.03897.
  * Qin et al. (2021) Haotong Qin, Yifu Ding, Mingyuan Zhang, YAN Qinghua, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xianglong Liu. 2021.  Bibert: Accurate fully binarized bert.  In _International Conference on Learning Representations_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392.
  * Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _AAAI spring symposium: logical formalizations of commonsense reasoning_ , pages 90–95.
  * Ruppert (1988) David Ruppert. 1988.  Efficient estimations from a slowly convergent robbins-monro process.  Technical report, Cornell University Operations Research and Industrial Engineering.
  * Sellam et al. (2021) Thibault Sellam, Steve Yadlowsky, Jason Wei, Naomi Saphra, Alexander D’Amour, Tal Linzen, Jasmijn Bastings, Iulia Turc, Jacob Eisenstein, Dipanjan Das, Ian Tenney, and Ellie Pavlick. 2021.  The multiberts: BERT reproductions for robustness analysis.  _ArXiv preprint_ , abs/2106.16163.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1631–1642.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958.
  * Sul and Choi (2023) Jeewoo Sul and Yong Suk Choi. 2023.  Balancing lexical and semantic quality in abstractive summarization.  _arXiv preprint arXiv:2305.09898_.
  * Sun et al. (2020) Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020.  ERNIE 2.0: A continual pre-training framework for language understanding.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 8968–8975. AAAI Press.
  * Tay et al. (2022) Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. 2022.  Scaling laws vs model architectures: How does inductive bias influence scaling?  _arXiv preprint arXiv:2207.10551_.
  * Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 3261–3275.
  * Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Wang et al. (2021) Benyou Wang, Yuxin Ren, Lifeng Shang, Xin Jiang, and Qun Liu. 2021.  Exploring extreme parameter compression for pre-trained language models.  In _International Conference on Learning Representations_.
  * Wang et al. (2022a) Jue Wang, Ke Chen, Gang Chen, Lidan Shou, and Julian McAuley. 2022a.  Skipbert: Efficient inference with shallow layer skipping.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7287–7301.
  * Wang et al. (2022b) Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. 2022b.  Adamix: Mixture-of-adapter for parameter-efficient tuning of large language models.  _ArXiv preprint_ , abs/2205.12410.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Wen et al. (2020) Yeming Wen, Dustin Tran, and Jimmy Ba. 2020.  Batchensemble: an alternative approach to efficient ensemble and lifelong learning.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122.
  * Xu et al. (2022) Runxin Xu, Fuli Luo, Chengyu Wang, Baobao Chang, Jun Huang, Songfang Huang, and Fei Huang. 2022.  From dense to sparse: Contrastive pruning for better pre-trained language model compression.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 11547–11555.
  * Xu et al. (2020) Yige Xu, Xipeng Qiu, Ligao Zhou, and Xuanjing Huang. 2020.  Improving bert fine-tuning via self-ensemble and self-distillation.  _ArXiv preprint_ , abs/2002.10345.
  * Xue et al. (2022) Fuzhao Xue, Aixin Sun, Hao Zhang, Jinjie Ni, and Eng-Siong Chng. 2022.  An embarrassingly simple model for dialogue relation extraction.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6707–6711. IEEE.
  * Yang et al. (2018) Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018.  Breaking the softmax bottleneck: A high-rank RNN language model.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 5754–5764.
  * Zhang et al. (2018) Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018.  ReCoRD: Bridging the gap between human and machine commonsense reading comprehension.  _arXiv preprint arXiv:1810.12885_.
  * Zhang et al. (2021a) Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi. 2021a.  Revisiting few-sample BERT fine-tuning.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Zhang et al. (2021b) Zhilu Zhang, Vianne R Gao, and Mert R Sabuncu. 2021b.  Ex uno plures: Splitting one model into an ensemble of subnetworks.  _ArXiv preprint_ , abs/2106.04767.
  * Zhao et al. (2022) Jing Zhao, Yifan Wang, Junwei Bao, Youzheng Wu, and Xiaodong He. 2022.  Fine-and coarse-granularity hybrid self-attention for efficient bert.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4811–4820.
  * Zheng et al. (2022) Rui Zheng, Bao Rong, Yuhao Zhou, Di Liang, Sirui Wang, Wei Wu, Tao Gui, Qi Zhang, and Xuan-Jing Huang. 2022.  Robust lottery tickets for pre-trained language models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2211–2224.
  * Zhou et al. (2022) Wangchunshu Zhou, Canwen Xu, and Julian McAuley. 2022.  Bert learns to teach: Knowledge distillation with meta learning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7037–7049.
  * Zhou and Srikumar (2022) Yichu Zhou and Vivek Srikumar. 2022.  A closer look at how fine-tuning changes bert.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1046–1061.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015_ , pages 19–27. IEEE Computer Society.
  * Zuo et al. (2022) Simiao Zuo, Qingru Zhang, Chen Liang, Pengcheng He, Tuo Zhao, and Weizhu Chen. 2022\.  Moebert: from bert to mixture-of-experts via importance-guided adaptation.  _ArXiv preprint_ , abs/2204.07675.

##  Appendix A Appendix Overview

In the appendix, we first describe the details of our methods and evaluation
protocol in Appendix B. Then, we visualize the disagreement of CLS embeddings
of some samples in Appendix C and provide a diversity metric during
pretraining in Appendix D. Finally, we compare the performance of individual
tasks in Appendix E.

##  Appendix B Experiment Details

We first describe the architecture details and pretraining details of our
methods and baselines in Section B.1. Then, we list the hyperparameter setup
in the fine-tuning in Section B.2. Finally, we explain the details of the
ensemble baselines and their related analyses in Section B.3.

###  B.1 Our Models and Baselines

The models built on BERT${}_{\text{Base}}$ are pretrained using two billion
tokens and each batch contains 30 sequences. The models built on
BERT${}_{\text{Large}}$ are pretrained using one billion tokens and each batch
contains 48 sequences. The learning rate is $2\cdot 10^{-5}$ and the warmup
ratio is $0.001$ for the pretraining stage.

We implement Multi-CLS BERT by modifying the code of Aroca-Ouellette and
Rudzicz (2020)555https://github.com/StephAO/olfmlm. We use [unused0] –
[unused(K-1)] tokens in the original BERT tokenizer as our input CLS tokens
[C1] – [CK]. We still keep the original CLS tokens to increase the
comparability with the MTL baseline.

We use NVIDIA GeForce RTX 2080, 1080, and TITAN X, M40 GPUs for the
BERT${}_{\text{Base}}$ experiments and use GeForce RTX 8000 and Tesla M40 for
the BERT${}_{\text{Large}}$ experiments. In footnote 3, the model size
excludes the top classifier parameters used in each task.

We test CMTL+ using the default hyperparameters of Aroca-Ouellette and Rudzicz
(2020) and we do not try different hyperparameters or different schedules of
pretraining losses. No Inserted Layers only removes the $L_{l,k}(.)$ while
still using different $H_{k}^{MC}$ on top during pretraining. SWA averages the
weights of every model checkpoint that is evaluated using the validation
dataset.

###  B.2 Fine-tuning

We start from the default evaluation hyperparameters used in Aroca-Ouellette
and Rudzicz (2020) and modify the settings based on the suggestions from Zhang
et al. (2021a) and Mosbach et al. (2021). We find that the best
hyperparameters depend on the training size. For example, batch size 16 works
well in GLUE Full but is much worse than batch size 4 in GLUE 100.
Furthermore, the performance of the default hyperparameters on some tasks is
suboptimal or unstable even after averaging the performance from 16 trials.
Therefore, we coarsely tune the hyperparameters to maximize and stabilize the
performance of the Ours (K=1) baseline under the memory and computational time
constraints in our GPUs. The preliminary results suggest that the
hyperparameters also maximize the performance of MTL.

Next, we list fine-tuning hyperparameters for all the tasks666We use different
values for some hyperparameters in ReCoRD. See the details below.. Our fine-
tuning stops after 20 epochs, 60k batches, or consecutive 10k batches without
a validation improvement (whichever comes first). We use the first 5k
validation samples to select the best fine-tuned model checkpoints for the
evaluation. The maximal gradient norm is 1. The maximal length for sentences
and CLS tokens is 128 for GLUE and 256 for SuperGLUE.

For each task, we select the best learning rate from $c\cdot 10^{-5}$ and
$c=1,2,3,4,5,7$. When running large datasets in GLUE Full and SuperGLUE Full
(MNLI, QQP, QNLI, SST-2, BoolQ, MultiRC, and WiC) using
BERT${}_{\text{Large}}$, we use learning rates $c=2,4,6,8,10,14$ to accelerate
the training. The batch sizes for GLUE 100, 1k, Full are 4, 8, 16,
respectively. The batch size for SuperGLUE is 4 except that the
BERT${}_{\text{Large}}$ models use 8 in SuperGLUE 1k and Full. For
BERT${}_{\text{Base}}$, the warmup ratio is 0.1. For BERT${}_{\text{Large}}$,
the warmup ratio is 0.2 and the weight decay is $10^{-6}$.

For each fine-tuning random seed, we randomly select a different subset in the
settings where only 100 or 1k training samples are available. For the datasets
with less than 500 training samples in SuperGLUE and SuperGLUE 1k (i.e., CB
and COPA), we repeat the experiments 32 times to further stabilize the scores.
For the pre-trained BERT baseline, we use 16 fine-tuning random seeds. To
reduce the computational cost, we use two pretraining random seeds and four
fine-tuning random seeds in our ablation study in Table 2.

Compared to other tasks, ReCoRD needs to be trained much longer than other
tasks in SuperGLUE, so we only use one fine-tuning seed for each of the four
pretrained models with different seeds. Our fine-tuning stops after 600k
batches (BERT${}_{\text{Base}}$) / 300k batches (BERT${}_{\text{Large}}$) or
consecutive 160k batches without a validation improvement (whichever comes
first).

To stabilize the performance of each model on ReCoRD, we use the first 40k
validation samples to select the best fine-tuned model checkpoints. We set
batch size as 8 and learning rate as $1\cdot 10^{-5}$ for
BERT${}_{\text{Base}}$. For BERT${}_{\text{Large}}$, we set batch size as 32
and learning rate as $2\cdot 10^{-5}$.

###  B.3 Ensemble Models

Ensemble on FT Seeds (K=1) in Table 2 is the same as Ensemble of Ours (K=1) in
Table 3. Ensemble on FT Seeds (K=5) in Table 2 is the same as ENS in Table 4.
Ensemble on Dropouts in Table 2 is the same as Dropout in Table 4. All results
are the average of four models that use four different pretrained models and
the best learning rate among $c\cdot 10^{-5}$ ($c=1,2,3,4,5,7$) in the fine-
tuning stage.

In Table 3, we compute the expected calibration error (ECE) (Naeini et al.,
2015) by

$\sum_{j=1}^{10}\frac{|B_{j}|}{N}|\text{acc}(j)-\text{conf}(j)|,$ (5)

where $\text{acc}(j)$ is the model accuracy in the $j$th bin $B_{j}$, $N$ is
the number of validation samples, and
$\text{conf}(j)=\frac{1}{|B_{j}|}\sum_{x\in B_{j}}\text{max}_{y}P(y|x)$ is the
average of the highest prediction probability $P(y|x)$ in the $j$th bin. We
put the samples into 10 equal-size bins according to their highest prediction
probability $\text{max}_{y}P(y|x)$.

In Table 3, we use Tesla M40 to measure the inference time of the models built
on BERT${}_{\text{Base}}$. We set batch size 16 and run 1000 batches to get
the average inference time of one batch in every GLUE task. We repeat the
experiments five times and report their average and standard error. For the
ensemble model, we assume the time of averaging multiple prediction
probabilities is negligible and directly multiply the inference time of Ours
(K=1) by 5.

In Table 4, we would like to see if CLS embeddings disagree with each other as
other ensemble baselines did. In Multi-CLS, we compute the uncertainty of each
sample $x$ as the average variance of prediction probability of each CLS
embedding $\text{mean}_{l}\left(\text{var}_{k}P(y=l|x,k)\right)$ and estimate
the prediction probability of the $k$th CLS embedding by

$P(y=l|x,k)=\frac{\text{exp}\left({\bm{q}}_{l,k}^{T}L^{FT}_{O,k}({\bm{h}}^{c}_{k}(x,y_{gt}))\right)}{\sum_{i}\text{exp}\left({\bm{q}}_{i,k}^{T}L^{FT}_{O,k}({\bm{h}}^{c}_{k}(x,y_{gt}))\right)},$
(6)

where $L^{FT}_{O,k}({\bm{h}}^{c}_{k}(x,y_{gt}))$ is the CLS embedding of the
input $x$ after fine-tuning, and
${\bm{q}}_{i,k}=\frac{1}{N_{i}}\sum_{y_{gt}=i}L^{FT}_{O,k}({\bm{h}}^{c}_{k}(x,y_{gt}))$
is the $i$th class embedding for the $k$th CLS embedding, which is computed by
averaging the $k$th CLS embeddings of the input $x$ with the $i$th class
label.

In Table 4, the two ensemble models for ENS vs ENS use the same set of 5 fine-
tuning seeds and the two Ours (K=5, $\mathbf{\lambda=0.1}$) pretrained with
different random seeds. Both uncertainty estimation models for Multi-CLS vs
ENS, Dropout vs ENS, and Least vs ENS are based on the same pretrained Ours
(K=5, $\mathbf{\lambda=0.1}$) model.

##  Appendix C Visualization of CLS embeddings

Table 5–16 compare the CLS embeddings of Ours (K=1) and Ours (K=5,
$\mathbf{\lambda=0.1}$) after fine-tuning to illustrate how different CLS
embeddings capture distinct aspects of an input sentence in solving a task.
For each task, we select one sample (a sentence or a sentence pair) from the
validation set whose CLS embeddings disagree with each other.

For each selected sample, we visualize its nearest-neighboring sentences in
the validation set with respect to each CLS embedding. The nearest neighbors
for the $k$th CLS embedding are determined by the cosine similarity between
the respective $k$th CLS embedding of the input sentence and other sentences.
Beside each sentence or sentence pair, we show their ground truth label and
the model’s prediction.

In Ours (K=5, $\mathbf{\lambda=0.1}$), two representative sentences are
selected from the top-three nearest neighbors for each CLS, and each CLS is
manually annotated with terms that summarize those aspects that are shared by
the neighbors and relate to the query sentence. For comparison, accompanying
tables show the top-ten nearest neighbors for Ours (K=1).

In almost all the classification tasks, we observe that CLS 3 and CLS 5 vote
for the same class (i.e., their embeddings are close to the neighbors with the
same class prediction). On the other hand, CLS 1 and CLS 4 vote for another
class in these examples where the CLS embeddings disagree. The observation
suggests that the similarity of CLS embeddings after the pretraining stage
correlates with their similarity after the fine-tuning.

Task: CoLA
---
Prediction |  Label | Sentence | Summary
Query:
un- acceptable |  un- acceptable | I lent the book partway to Tony.
CLS-space Neighbors (K=5):
CLS 1
acceptable |  acceptable |  I gave it to Pete to take to the fair. |  _Gave to_
acceptable |  un- acceptable |  Sue gave to Bill a book. |
CLS 2
un- acceptable |  un- acceptable |  We wanted to invite someone, but we couldn’t decide who to. |  Incorrect or extra word
un- acceptable |  un- acceptable |  Jessica crammed boxes at the truck. |
CLS 3
un- acceptable |  un- acceptable |  We wanted to invite someone, but we couldn’t decide who to. |  Incorrect or extra word
un- acceptable |  un- acceptable |  I hit that you knew the answer. |  First person
CLS 4
acceptable |  acceptable |  The paper was written up by John. |  Writing
acceptable |  acceptable |  John owns the book. |
CLS 5
un- acceptable |  un- acceptable |  Chris was handed Sandy a note by Pat. |  Extra word Writing Giving
un- acceptable |  un- acceptable |  What Mary did Bill was give a book. |
Table 5: Visualization of Ours (K=5, $\mathbf{\lambda=0.1}$) using a sample in
CoLA. The neighbors from CLS 2, 3, and 5 are unacceptable sentences that often
contain extra words, as in the query. The neighbors from CLS 1 and 2 are
semantically related to the query.  Task: CoLA
---
Prediction |  Label | Sentence
Query:
un- acceptable |  un- acceptable |  I lent the book partway to Tony.
CLS-space Neighbors (K=1):
un- acceptable |  un- acceptable |  I presented John with it dead.
un- acceptable |  acceptable |  Nora sent the book.
un- acceptable |  un- acceptable |  There seemed to be intelligent.
un- acceptable |  un- acceptable |  The book what inspired them was very long.
un- acceptable |  un- acceptable |  The book was by John written.
un- acceptable |  acceptable |  I met the man who grows peaches.
un- acceptable |  acceptable |  We persuaded Mary to leave and Sue to stay.
un- acceptable |  un- acceptable |  I hit that you knew the answer.
un- acceptable |  un- acceptable |  We think that Leslie likes ourselves.
un- acceptable |  acceptable |  This flyer and that flyer differ.
Table 6:  Visualization of Ours (K=1) using the sample in CoLA.  Task: SST-2
---
Prediction |  Label | Sentence | Summary
Query:
negative |  negative |  An occasionally funny, but overall limp, fish-out-of-water story. |
CLS-space Neighbors (K=5):
CLS 1
negative |  positive |  Based on a devilishly witty script by Heather McGowan and Niels Mueller, the film gets great laughs, but never at the expense of its characters |  _Fun_ or _funny_ Commas _Script_
negative |  positive |  McConaughey’s fun to watch, the dragons are okay, not much fire in the script. |
|
CLS 2
negative |  negative |  Visually rather stunning, but ultimately a handsome-looking bore, the true creativity would have been to hide Treasure Planet entirely and completely reimagine it. |  _Stunning_ or _thrilling_ but negative Sci-fi
negative |  negative |  If looking for a thrilling sci-fi cinematic ride, don’t settle for this imposter. |
|
CLS 3
positive |  positive |  Funny but perilously slight. |  Positive overall but qualified
positive |  positive |  A movie that successfully crushes a best selling novel into a timeframe that mandates that you avoid the Godzilla sized soda. |
CLS 4
negative |  negative |  Passable entertainment, but it’s the kind of motion picture that won’t make much of a splash when it’s released, and will not be remembered long afterwards. |  Positive statement, _but_ negative Liquid
negative |  negative |  It showcases Carvey’s talent for voices, but not nearly enough and not without taxing every drop of one’s patience to get to the good stuff. |
|
|
CLS 5
positive |  positive |  The terrific and bewilderingly underrated Campbell Scott gives a star performance that is nothing short of mesmerizing. |  _Mesmeriz-_ _ing_ or _intense_ and positive
positive |  positive |  … an otherwise intense, twist-and-turn thriller that certainly shouldn’t hurt talented young Gaghan’s resume. |
|
|
Table 7:  Visualization of Ours (K=5, $\mathbf{\lambda=0.1}$) using a sample
in SST-2. The neighbors from CLS 2 and 4 share the same "postive, but
negative" template as in the query. Like the query, CLS 1, 3, and 5 capture
the positive aspects. Some CLSs also capture the semantic aspects of the query
such as _script_ , _sci-fi_ , or _liquid_.  Task: SST-2
---
Prediction |  Label | Sentence
Query:
positive |  negative |  An occasionally funny, but overall limp, fish-out-of-water story.
CLS-space Neighbors (K=1):
positive |  positive |  In a way, the film feels like a breath of fresh air, but only to those that allow it in.
positive |  positive |  A painfully funny ode to bad behavior.
positive |  positive |  Two hours fly by – opera’s a pleasure when you don’t have to endure intermissions – and even a novice to the form comes away exhilarated.
positive |  positive |  Huston nails both the glad-handing and the choking sense of hollow despair.
positive |  positive |  The movie’s relatively simple plot and uncomplicated morality play well with the affable cast.
positive |  positive |  So much facile technique, such cute ideas, so little movie.
positive |  positive |  A psychological thriller with a genuinely spooky premise and an above-average cast, actor Bill Paxton’s directing debut is a creepy slice of gothic rural Americana.
positive |  positive |  The primitive force of this film seems to bubble up from the vast collective memory of the combatants.
positive |  positive |  The continued good chemistry between Carmen and Juni is what keeps this slightly disappointing sequel going, with enough amusing banter – blessedly curse-free – to keep both kids and parents entertained.
positive |  positive |  This flick is about as cool and crowd-pleasing as a documentary can get.
Table 8:  Visualization of Ours (K=1) using the sample in SST-2.  Task: MRPC
---
Prediction |  Label | Sentence Pair | Summary
Query:
equivalent |  equivalent |  S1: A man arrested for allegedly threatening to shoot and kill a city councilman from Queens was ordered held on $100,000 bail during an early morning court appearance Saturday.
S2: The Queens man arrested for allegedly threatening to shoot City Councilman Hiram Monserrate was held on $100,000 bail Saturday, a spokesman for the Queens district attorney said. |
CLS-space Neighbors (K=5):
CLS 1
equivalent |  equivalent |  S1: Myanmar’s pro-democracy leader Aung San Suu Kyi will return home late Friday but will remain in detention after recovering from surgery at a Yangon hospital, her personal physician said.
S2: Myanmar’s pro-democracy leader Aung San Suu Kyi will be kept under house arrest following her release from a hospital where she underwent surgery, her personal physician said Friday. |  Comments Politics Justice
not equivalent |  not equivalent |  S1: Bob Richter, a spokesman for House Speaker Tom Craddick, had no comment about the ruling.
S2: Bob Richter, spokesman for Craddick, R-Midland, said the speaker had not seen the ruling and could not comment. |
CLS 2
equivalent |  equivalent |  S1: They were being held Sunday in the Camden County Jail on $100,000 bail.
S2: They remained in Camden County Jail on Sunday on $100,000 bail. |  Thousands Crime or threat
equivalent |  equivalent |  S1: "More than 70,000 men and women from bases in Southern California were deployed in Iraq.
S2: In all, more than 70,000 troops based in Southern California were deployed to Iraq. |
CLS 3
equivalent |  equivalent |  S1: Robert Walsh, 40, remained in critical but stable condition Friday at Staten Island University Hospital’s north campus.
S2: Walsh, also 40, was in critical but stable condition at Staten Island University Hospital last night. |  Time
equivalent |  equivalent |  S1: Blair’s Foreign Secretary Jack Straw was to take his place on Monday to give a statement to parliament on the European Union.
S2: Blair’s office said his Foreign Secretary Jack Straw would take his place on Monday to give a statement to parliament on the EU meeting the prime minister attended last week. |
CLS 4
not equivalent |  not equivalent |  S1: Franklin County Judge-Executive Teresa Barton said a firefighter was struck by lightning and was taken to the Frankfort Regional Medical Center.
S2: A county firefighter, was struck by lightning and was in stable condition at Frankfort Regional Medical Center. |  Comments Medical or justice
equivalent |  equivalent |  S1: Myanmar’s pro-democracy leader Aung San Suu Kyi will return home late Friday but will remain in detention after recovering from surgery at a Yangon hospital, her personal physician said.
S2: Myanmar’s pro-democracy leader Aung San Suu Kyi will be kept under house arrest following her release from a hospital where she underwent surgery, her personal physician said Friday. |
CLS 5
equivalent |  equivalent |  S1: Unable to find a home for him, a judge told mental health authorities they needed to find supervised housing and treatment for DeVries somewhere in California.
S2: The judge had told the state Department of Mental Health to find supervised housing and treatment for DeVries somewhere in California. |  Court’s ruling
equivalent |  equivalent |  S1: A former employee of a local power company pleaded guilty Wednesday to setting off a bomb that knocked out a power substation during the Winter Olympics last year.
S2: A former Utah Power meter reader pleaded guilty Wednesday to bombing a power substation during the 2002 Winter Olympics. |
Table 9: Visualization of Ours (K=5, $\mathbf{\lambda=0.1}$) using a sample in
MRPC. The neighbors from CLS 2, 3, and 5 focus on different aspects of the
query. The neighbors from CLS 1 and 4 are someone’s comments as in the query
and might not be equivalent. Several CLSs are also related to justice.  Task:
MRPC
---
Prediction |  Label | Sentence Pair
Query:
equivalent |  equivalent |  S1: A man arrested for allegedly threatening to shoot and kill a city councilman from Queens was ordered held on $100,000 bail during an early morning court appearance Saturday.
S2: The Queens man arrested for allegedly threatening to shoot City Councilman
Hiram Monserrate was held on $100,000 bail Saturday, a spokesman for the
Queens district attorney said.
CLS-space Neighbors (K=1):
equivalent |  equivalent |  S1: The Justice Department Aug. 19 gave pre-clearance for the Oct. 7 date for the election to recall Gov. Gray Davis, saying it would not affect minority voting rights.
S2: The Justice Department on Aug. 19 sanctioned the Oct. 7 date for recall
election, saying it would not affect voting rights.
equivalent |  equivalent |  S1: The worm attacks Windows computers via a hole in the operating system, an issue Microsoft on July 16 had warned about.
S2: The worm attacks Windows computers via a hole in the operating system,
which Microsoft warned of 16 July.
equivalent |  equivalent |  S1: O’Brien was charged with leaving the scene of a fatal accident, a felony.
S2: Bishop Thomas O’Brien, 67, was booked on a charge of leaving the scene of
a fatal accident.
equivalent |  equivalent |  S1: "There is no conscious policy of the United States, I can assure you of this, to move the dollar at all," he said.
S2: He also said there is no conscious policy by the United States to move the
value of the dollar.
equivalent |  equivalent |  S1: The AFL-CIO is waiting until October to decide if it will endorse a candidate.
S2: The AFL-CIO announced Wednesday that it will decide in October whether to
endorse a candidate before the primaries.
equivalent |  equivalent |  S1: Speaking for the first time yesterday, Brigitte’s maternal aunt said his family was unaware he had was in prison or that he had remarried.
S2: Brigitte’s maternal aunt said his family was unaware he had been sent to
prison, or that he had remarried in Sydney.
equivalent |  not equivalent |  S1: Rosenthal is hereby sentenced to custody of the Federal Bureau of prisons for one day with credit for time served," Breyer said to tumultuous cheers in the courtroom.
S2: "Rosenthal is hereby sentenced to custody of the Federal Bureau of Prisons
for one day with credit for time served."
equivalent |  equivalent |  S1: Police say CIBA was involved in the importation of qat, a narcotic substance legal in Britain but banned in the United States.
S2: Mr McKinlay said that CIBA was involved in the importation of qat, a
narcotic substance legal in Britain but banned in the US.
equivalent |  equivalent |  S1: Judge Craig Doran said it wasn’t his role to determine if Hovan was "an evil man" but maintained that "he has committed an evil act."
S2: Judge Craig Doran said he couldn’t determine if Hovan was "an evil man"
but said he "has committed an evil act."
equivalent |  equivalent |  S1: But MTA officials appropriated the money to the 2003 and 2004 budgets without notifying riders or even the MTA board members considering the 50-cent hike, Hevesi found.
S2: MTA officials appropriated the surplus money to later years’ budgets
without notifying riders or the MTA board members when the 50-cent hike was
being considered, he said.
Table 10:  Visualization of Ours (K=1) using the sample in MRPC.  Task: MNLI
---
Prediction |  Label | Sentence Pair | Summary
Query:
contradic- tion |  contradic- tion |  S1: There is very little left of old Ocho the scant remains of Ocho Rios Fort are probably the oldest and now lie in an industrial area, almost forgotten as the tide of progress has swept over the town.
S2: There is nothing left of the Ocho Rios Fort. |
CLS-space Neighbors (K=5):
CLS 1
neutral |  contradic- tion |  S1: After the purge of foreigners, only a few stayed on, strictly confined to Dejima Island in Nagasaki Bay.
S2: A few foreigners were left free after the purge on foreigners. |  Size or quantity
neutral |  neutral |  S1: ’Publicity.’ Lincoln removed his great hat, making a small show of dusting it off.
S2: Lincoln took his hat off. |
CLS 2
neutral |  neutral |  S1: There is no tradition of clothes criticism that includes serious analysis, or even of costume criticism among theater, ballet, and opera critics, who do have an august writerly heritage.
S2: Clothes criticism is not serious. |  Historical places or heritage Negation
neutral |  neutral |  S1: All of the islands are now officially and proudly part of France, not colonies as they were for some three centuries.
S2: The islands are part of France now instead of just colonies. |
CLS 3
contradic- tion |  neutral |  S1: And yet, we still lack a set of global accounting and reporting standards that reflects the globalization of economies, enterprises, and markets.
S2: The globalization of economies is not reflected in global accounting standards. |  Industry Region Negation
contradic- tion |  contradic- tion |  S1: The technology used to capture and evaluate information in response to the RFP permits LSC to compile and assess key information about the delivery system at the program, state, regional, and national level.
S2: There is no way for the LSC to compile information about delivery systems. |
CLS 4
neutral |  neutral |  S1: Scotland became little more than an English county. S2: Scotland was hardly better than an English county. |  Historical places Minimiza- tion or negation
neutral |  neutral |  S1: Just as in ancient times, without the River Nile, Egypt could not exist. S2: Without the Nile River, Egypt could not exist. |
|
|
CLS 5
contradic- tion |  neutral |  S1: Beside the fortress lies an 18th-century caravanserai, or inn, which has been converted into a hotel, and now hosts regular folklore evenings of Turkish dance and music.
S2: The 18th century caravanserai is now a hotel. |  Buildings or _properties_ Contrast
contradic- tion |  contradic- tion |  S1: Diamonds are graded from D to X, with only D, E, and F considered good, D being colorless or river white, J slightly tinted, Q light yellow, and S to X yellow.
S2: There is no difference between diamonds, all having the same properties. |
Table 11: Visualization of Ours (K=5, $\mathbf{\lambda=0.1}$) using a sample
in MNLI. Only one sentence in the neighbors of CLS 3 contains negation. Only
the premise in the neighbors from CLS 5 makes a comparison. Both CLSs vote for
the contradiction class. Several CLSs are related to buildings or historical
places.  Task: MNLI
---
Prediction |  Label | Sentence Pair
Query:
contradic- tion |  contradic- tion |  S1: There is very little left of old Ocho the scant remains of Ocho Rios Fort are probably the oldest and now lie in an industrial area, almost forgotten as the tide of progress has swept over the town.
S2: There is nothing left of the Ocho Rios Fort.
CLS-space Neighbors (K=1):
contradic- tion |  contradic- tion |  S1: It was utterly mad.
S2: It was perfectly normal.
contradic- tion |  neutral |  S1: Fixing current levels of damage would be impossible.
S2: Fixing the damage could never be done.
contradic- tion |  contradic- tion |  S1: It was still night.
S2: The sun was blazing in the sky, darkness nowhere to be seen.
contradic- tion |  contradic- tion |  S1: That’s their signal
S2: That isn’t their signal.
contradic- tion |  contradic- tion |  S1: It is extremely dangerous to Every trip to the store becomes a temptation.
S2: Even with every trip to the store, it never becomes a temptation.
contradic- tion |  contradic- tion |  S1: The Revolutionaries couldn’t be dissuaded from destroying most of the cathedral’s statues, although 67 were saved (many of the originals are now housed in the Musée de l’Oeuvre Notre-Dame next door).
S2: All of the cathedrals statues were saved by the Revolutionaries.
contradic- tion |  contradic- tion |  S1: It was deserved.
S2: It was not deserved at all
contradic- tion |  entailment |  S1: And far, far away- lying still on the tracks- was the back of the train.
S2: The train wasn’t moving but then it started up.
contradic- tion |  contradic- tion |  S1: Even if you’re the kind of traveler who likes to improvise and be adventurous, don’t turn your nose up at the tourist offices.
S2: There’s nothing worth seeing in the tourist offices.
contradic- tion |  contradic- tion |  S1: Cybernetics had always been Derry’s passion.
S2: Derry knew nothing of cybernetics.
Table 12:  Visualization of Ours (K=1) using the sample in MNLI.  Task: QNLI
---
Prediction |  Label | Sentence Pair | Summary
Query:
entailment |  entailment |  S1: What factors negatively impacted Jacksonville following the war?
S2: Warfare and the long occupation left the city disrupted after the war. |
CLS-space Neighbors (K=5):
CLS 1
entailment |  entailment |  S1: When was the Russian policy "indigenization" defunded?
S2: Never formally revoked, it stopped being implemented after 1932. |  Time
entailment |  entailment |  S1: How did Luther describe his learning at the university?
S2: He was made to wake at four every morning for what has been described as "a day of rote learning and often wearying spiritual exercises." |
CLS 2
entailment |  not entailment |  S1: How did the 2001 IPCC report compare to reality for 2001-2006?
S2: The study compared IPCC 2001 projections on temperature and sea level change with observations. |  Change
entailment |  entailment |  S1: Who led the most rapid expansion of the Mongol empire?
S2: Under Genghis’s successor Ogedei Khan the speed of expansion reached its peak. |
CLS 3
not entailment |  not entailment |  S1: During which period did Jacksonville become a popular destination for the rich?
S2: This highlighted the visibility of the state as a worthy place for tourism. |  _Jacksonville_ Duration of time
not entailment |  not entailment |  S1: What brought the downfall of Jacksonville filmmaking?
S2: Over the course of the decade, more than 30 silent film studios were established, earning Jacksonville the title of "winter film capital of the world". |
CLS 4
entailment |  entailment |  S1: How did the new king react to the Huguenots?
S2: Louis XIV gained the throne in 1643 and acted increasingly aggressively to force the Huguenots to convert. |  Change
entailment |  not entailment |  S1: What did Luther begin to experience in 1536?
S2: In December 1544, he began to feel the effects of angina. |
CLS 5
not entailment |  not entailment |  S1: What brought the downfall of Jacksonville filmmaking?
S2: Over the course of the decade, more than 30 silent film studios were established, earning Jacksonville the title of "winter film capital of the world". |  Negative event
not entailment |  not entailment |  S1: What cycle AC current system did Tesla propose?
S2: He found the time there frustrating because of conflicts between him and the other Westinghouse engineers over how best to implement AC power. |
Table 13: Visualization of Ours (K=5, $\mathbf{\lambda=0.1}$) using a sample
in QNLI. The neighbors from CLS 1, 2, and 4 are about time or changes. The
neighbors from CLS 3 and 5 are about _Jacksonville_ or negative events. Task:
QNLI
---
Prediction |  Label | Sentence Pair
Query:
entailment |  entailment |  S1: What factors negatively impacted Jacksonville following the war?
S2: Warfare and the long occupation left the city disrupted after the war.
CLS-space Neighbors (K=1):
entailment |  entailment |  S1: How many Africans were brought into the United States during the slave trade?
S2: Participation in the African slave trade and the subsequent treatment of
its 12 to 15 million Africans is viewed by some to be a more modern extension
of America’s "internal colonialism".
entailment |  entailment |  S1: Which country used to rule California?
S2: Though there is no official definition for the northern boundary of
southern California, such a division has existed from the time when Mexico
ruled California, and political disputes raged between the Californios of
Monterey in the upper part and Los Angeles in the lower part of Alta
California.
entailment |  entailment |  S1: In what area of this British colony were Huguenot land grants?
S2: In 1700 several hundred French Huguenots migrated from England to the
colony of Virginia, where the English Crown had promised them land grants in
Lower Norfolk County.
entailment |  entailment |  S1: Who was responsible for the new building projects in Jacksonville?
S2: Mayor W. Haydon Burns’ Jacksonville Story resulted in the construction of
a new city hall, civic auditorium, public library and other projects that
created a dynamic sense of civic pride.
entailment |  entailment |  S1: What did Tesla first receive after starting his company?
S2: The company installed electrical arc light based illumination systems
designed by Tesla and also had designs for dynamo electric machine
commutators, the first patents issued to Tesla in the US.
entailment |  entailment |  S1: In what year did the university first see a drop in applications?
S2: In the early 1950s, student applications declined as a result of
increasing crime and poverty in the Hyde Park neighborhood.
entailment |  entailment |  S1: What was Fresno’s population in 2010?
S2: The 2010 United States Census reported that Fresno had a population of
494,665.
entailment |  entailment |  S1: What was the percentage of Black or African-Americans living in the city?
S2: The racial makeup of the city was 50.2% White, 8.4% Black or African
American, 1.6% Native American, 11.2% Asian (about a third of which is Hmong),
0.1% Pacific Islander, 23.4% from other races, and 5.2% from two or more
races.
entailment |  entailment |  S1: Where did Marin build first fort?
S2: He first constructed Fort Presque Isle (near present-day Erie,
Pennsylvania) on Lake Erie’s south shore.
entailment |  not entailment |  S1: How old was Tesla when he became a US citizen?
S2: In the same year, he patented the Tesla coil.
Table 14: Visualization of Ours (K=1) using the sample in QNLI.  Task: STS-B
---
Prediction |  Label | Sentence Pair | Summary
Query:
2.009 |  2.000 |  S1: Volkswagen skids into red in wake of pollution scandal
S2: Volkswagen’s "gesture of goodwill" to diesel owners |
CLS-space Neighbors (K=5):
CLS 1
2.633 |  3.800 |  S1: Rosberg emulates father with Monaco win S2: FORMULA 1: Rosberg stays modest despite Monaco win |  Motor vehicles Racing
2.754 |  3.000 |  S1: A motorcross driver going by during a race
S2: A race car driver performs in the race of his life. |
CLS 2
1.710 |  1.400 |  S1: A golden dog is running through the snow.
S2: A pack of sled dogs pulling a sled through a town. |  Colors Action
1.917 |  1.400 |  S1: The black and white dog is running on the grass.
S2: A black and white dog swims in blue water. |
CLS 3
2.952 |  2.600 |  S1: Obama endorses same-sex marriage
S2: Obama’s delicate dance on same-sex marriage |  Politics and economics
2.071 |  1.000 |  S1: Spanish jobless rate soars past 25 per cent
S2: US jobless rate seen rising, offering Obama no relief |
CLS 4
0.337 |  0.000 |  S1: Presumably the decision of drivers to slow down in response to work zone signage is influenced by many factors.
S2: This short talk deals with issues of "cheating slightly" :Dan Ariely: Our buggy moral code . |  Motor vehicles Morality
0.946 |  0.600 |  S1: Saudi gas truck blast kills at least 22
S2: Nigeria church blast kills at least 12 |
CLS 5
3.820 |  2.800 |  S1: Stocks dipped lower Tuesday as investors opted to cash in profits from Monday’s big rally despite a trio of reports suggesting modest improvement in the economy.
S2: Wall Street moved tentatively higher Tuesday as investors weighed a trio of reports showing modest economic improvement against an urge to cash in profits from Monday’s big rally. |  Politics and economics
2.952 |  2.600 |  S1: Obama endorses same-sex marriage
S2: Obama’s delicate dance on same-sex marriage |
Table 15: Visualization of Ours (K=5, $\mathbf{\lambda=0.1}$) using a sample
in STS-B. The neighbors from CLS 1 and 4 are about motor vehicles. The
neighbors from CLS 3 and 5 are about politics and economics.  Task: STS-B
---
Prediction |  Label | Sentence Pair
Query:
2.009 |  2.000 |  S1: Volkswagen skids into red in wake of pollution scandal
S2: Volkswagen’s "gesture of goodwill" to diesel owners
CLS-space Neighbors (K=1):
2.133 |  2.600 |  S1: Large silver locomotive engine in a shed.
S2: The silver train is parked in a station.
2.537 |  3.200 |  S1: An AeroMexico jet taxing along a runway.
S2: a silver AreoMexico Jet Liner sitting on the tarmac.
2.189 |  2.800 |  S1: Two women holding checkered flags near an orange car.
S2: Two ladies in skimpy clothes pose next to an old fashioned car.
1.963 |  2.400 |  S1: Two dogs in the snow
S2: Two dogs play in the grass.
2.041 |  1.600 |  S1: Three dogs are playing in the white snow.
S2: Two dogs are playing in the grass.
2.402 |  3.200 |  S1: Once you open it up to toxins, the answer is clearly no, boiling is not enough.
S2: Boiling eliminates only a certain class of contaminants that can make you
ill.
1.990 |  1.400 |  S1: A golden dog is running through the snow.
S2: A pack of sled dogs pulling a sled through a town.
2.230 |  2.200 |  S1: Man sitting on a bench drink from a mug surrounded by rugs.
S2: A man is sitting on one of two red benches and staring into a kiosk.
2.426 |  2.000 |  S1: If you can get over the "ick factor," you have an easily-applied source of organic nitrogen fertilizer close at hand.
S2: The NPK numbers on the fertilizer represents the percent, by weight, of
Nitrogen, P2O5 and K2O, respectively.
2.248 |  3.400 |  S1: Try switching to rats; weanling rats if you need something smaller.
S2: As mentioned in previous answers, rats and gerbils can be offered instead
of mice or in a rotation with mice.
Table 16: Visualization of Ours (K=1) using the sample in STS-B.

##  Appendix D Diversity Measurement between two CLS embeddings

We find that cosine similarities between the CLS embeddings are not a good
measurement of their diversity. For different CLS $k$, if their hidden states
${\bm{h}}^{c}_{k}$ are identical but their output linear layers $L_{O,k}$ have
different biases, the cosine similarity between CLS embeddings could be small
but their diversity is also small.

Motivated by the visualization in Appendix C, we found that the diversity
between two CLS embeddings ($k_{1}$ and $k_{2}$) could be estimated by their
similarity differences to their neighbors. If two CLS embeddings collapse,
their dot products to their neighbors should perfectly correlated with each
other and their resulting nearest neighbors would be the same. Thus, we
estimate the diversity between CLS embeddings during pretraining by

$\displaystyle\text{Corr}\left([({\bm{c}}_{k_{1},i}^{1-2})^{T}{\bm{c}}_{k_{1},j}^{3-4}]_{i,j},[({\bm{c}}_{k_{2},i}^{1-2})^{T}{\bm{c}}_{k_{2},j}^{3-4}]_{i,j}\right),$
(7)

where ${\bm{c}}_{k_{1},i}^{1-2}$ is the $k_{1}$th CLS embedding of the $i$th
sample for sentence 1 and 2 in the batch, ${\bm{c}}_{k_{1},j}^{3-4}$ is the
$k_{1}$th CLS embedding of the $j$th sample for sentence 3 and 4 in the batch,
$[({\bm{c}}_{k_{1},i}^{1-2})^{T}{\bm{c}}_{k_{1},j}^{3-4}]_{i,j}$ is a sequence
containing all the pairwise dot products of $k_{1}$th CLS embeddings in the
batch, and Corr is the pearson correlation coefficient. Lower correlation
means more diverse.

We use the metric to test our diversification methods and detect the
collapsing during pretraining. Without using the diversity tricks we developed
(e.g., inserting linear layers into the transformer encoder), this metric
would be greater than $0.99$ and the improvement would be greatly reduce in
downstream applications (see our ablation study in Table 2). In contrast, our
final best model reaches around $0.9$–$0.95$ in this metric. We found that if
we use the fine-tuning re-parameterization trick during pretraining, we can
have a lower correlation value (i.e., more diverse CLS embeddings), but the
performance on GLUE is much worse. This indicates that there is an ideal
diversity level for the consecutive sentence detection task during
pretraining.

##  Appendix E Performance of Individual Tasks

The GLUE tasks include CoLA (Warstadt et al., 2019), SST-2 (Socher et al.,
2013), MRPC (Dolan and Brockett, 2005) ,
QQP777https://www.quora.com/profile/Ricky-Riche-2/First-Quora-Dataset-Release-
Question-Pairs, STS-B (Cer et al., 2017), MNLI (Williams et al., 2018), QNLI
(Rajpurkar et al., 2016), RTE (Bentivogli et al., 2009), and WNLI (Levesque et
al., 2012). The SuperGLUE tasks include BoolQ (Clark et al., 2019a), CB (De
Marneffe et al., 2019), COPA (Roemmele et al., 2011), MultiRC (Khashabi et
al., 2018), ReCoRD (Zhang et al., 2018), RTE, WiC (Pilehvar and Camacho-
Collados, 2019), and WSC (Levesque et al., 2012).

We report the individual task results of GLUE 100 and 1k in Table 17, the
results of GLUE Full in Table 18, the results of SuperGLUE 100 and 1k in Table
19, the results of SuperGLUE Full in Table 20, the results of the top 20%
uncertain sample overlapping ratio in Table 22, and the results of ECE in
Table 21. In Table 18, we also compare the GLUE score of our MTL baseline with
the scores reported in Aroca-Ouellette and Rudzicz (2020).

In GLUE 100 and SuperGLUE 100, multiple embeddings are almost always better.
In GLUE 1k and Full, the improvement is smaller, so the baselines perform
better in some individual tasks. We also observe that different downstream
tasks might prefer different lambda.

In Table 21, we compute the p value using Chernoff bound:

$P(X>(1+\delta)\mu)<\left(\frac{e^{\delta}}{(1+\delta)^{(1+\delta)}}\right)^{\mu},$
(8)

where $\mu=(0.2)^{2}4N$, $N$ is the number of samples in the validation set,
$\delta=\frac{\sum_{i=1}^{4}S_{i}}{\mu}-1$, and $S_{i}$ is the observed size
of overlapping at the $i$th trial.

GLUE 100 (BERT Base)
---
| CoLA | SST | MRPC | STS-B | QQP | MNLI | QNLI | RTE | Avg.
| MCC | Acc | F1 | Spearman | F1 | Acc | Acc | Acc | -
Pretrained | 18.62 | 75.41 | 80.44 | 62.16 | 59.09 | 38.51 | 59.99 | 54.56 | 55.71
|  $\pm$ 1.96 |  $\pm$ 1.95 |  $\pm$ 0.57 |  $\pm$ 3.68 |  $\pm$ 0.94 |  $\pm$ 0.69 |  $\pm$ 1.33 |  $\pm$ 0.79 |  $\pm$ 0.62
MTL | 9.90 | 70.67 | 81.64 | 78.88 | 59.74 | 43.50 | 73.54 | 57.49 | 59.29
|  $\pm$ 1.18 |  $\pm$ 0.63 |  $\pm$ 0.19 |  $\pm$ 0.57 |  $\pm$ 0.73 |  $\pm$ 0.87 |  $\pm$ 0.56 |  $\pm$ 1.15 |  $\pm$ 0.27
Ours (K=1) | 11.24 | 70.24 | 80.97 | 78.10 | 58.94 | 40.99 | 68.69 | 55.18 | 57.84
|  $\pm$ 1.09 |  $\pm$ 1.30 |  $\pm$ 0.33 |  $\pm$ 0.61 |  $\pm$ 0.73 |  $\pm$ 0.56 |  $\pm$ 0.99 |  $\pm$ 1.19 |  $\pm$ 0.32
Ours (K=5, $\lambda=0$) | 17.44 | 74.31 | 81.98 | 79.53 | 61.98 | 44.47 | 75.94 | 58.44 | 61.54
|  $\pm$ 1.36 |  $\pm$ 1.19 |  $\pm$ 0.22 |  $\pm$ 0.70 |  $\pm$ 0.54 |  $\pm$ 0.67 |  $\pm$ 0.48 |  $\pm$ 1.38 |  $\pm$ 0.32
Ours (K=5, $\lambda=0.1$) | 17.61 | 75.49 | 81.68 | 79.25 | 61.70 | 46.09 | 75.12 | 59.17 | 61.80
|  $\pm$ 1.75 |  $\pm$ 0.96 |  $\pm$ 0.14 |  $\pm$ 0.66 |  $\pm$ 0.63 |  $\pm$ 0.84 |  $\pm$ 0.56 |  $\pm$ 1.48 |  $\pm$ 0.35
Ours (K=5, $\lambda=0.5$) | 13.52 | 74.24 | 81.60 | 79.49 | 61.78 | 44.46 | 74.14 | 57.25 | 60.49
|  $\pm$ 1.94 |  $\pm$ 0.95 |  $\pm$ 0.15 |  $\pm$ 0.51 |  $\pm$ 0.49 |  $\pm$ 0.62 |  $\pm$ 0.66 |  $\pm$ 1.34 |  $\pm$ 0.35
Ours (K=5, $\lambda=1$) | 10.59 | 74.56 | 81.28 | 77.93 | 60.80 | 43.42 | 74.83 | 57.17 | 59.86
|  $\pm$ 1.68 |  $\pm$ 0.89 |  $\pm$ 0.16 |  $\pm$ 0.67 |  $\pm$ 0.82 |  $\pm$ 0.95 |  $\pm$ 0.70 |  $\pm$ 1.20 |  $\pm$ 0.34
GLUE 100 (BERT Large)
| CoLA | SST | MRPC | STS-B | QQP | MNLI | QNLI | RTE | Avg.
MTL | 15.64 | 79.61 | 81.48 | 74.92 | 61.41 | 43.61 | 77.28 | 58.66 | 61.39
|  $\pm$ 1.60 |  $\pm$ 1.63 |  $\pm$ 0.19 |  $\pm$ 0.90 |  $\pm$ 0.47 |  $\pm$ 0.86 |  $\pm$ 0.53 |  $\pm$ 1.24 |  $\pm$ 0.37
Ours (K=1) | 16.87 | 73.21 | 81.35 | 77.48 | 57.58 | 41.51 | 70.79 | 56.20 | 59.19
|  $\pm$ 1.99 |  $\pm$ 1.57 |  $\pm$ 0.18 |  $\pm$ 0.63 |  $\pm$ 0.75 |  $\pm$ 0.87 |  $\pm$ 1.90 |  $\pm$ 0.57 |  $\pm$ 0.43
Ours (K=5, $\lambda=0$) | 22.41 | 80.54 | 82.01 | 76.06 | 61.60 | 46.35 | 77.73 | 60.27 | 63.19
|  $\pm$ 1.97 |  $\pm$ 2.11 |  $\pm$ 0.23 |  $\pm$ 1.97 |  $\pm$ 0.96 |  $\pm$ 0.86 |  $\pm$ 0.64 |  $\pm$ 1.23 |  $\pm$ 0.49
Ours (K=5, $\lambda=0.1$) | 22.02 | 82.67 | 81.81 | 78.44 | 63.49 | 46.94 | 77.58 | 63.51 | 64.24
|  $\pm$ 2.79 |  $\pm$ 0.72 |  $\pm$ 0.22 |  $\pm$ 0.63 |  $\pm$ 0.66 |  $\pm$ 0.74 |  $\pm$ 0.65 |  $\pm$ 0.70 |  $\pm$ 0.40
Ours (K=5, $\lambda=0.5$) | 18.59 | 80.47 | 82.02 | 77.16 | 61.18 | 47.04 | 77.43 | 61.91 | 63.02
|  $\pm$ 2.54 |  $\pm$ 1.33 |  $\pm$ 0.12 |  $\pm$ 0.40 |  $\pm$ 0.69 |  $\pm$ 0.77 |  $\pm$ 0.56 |  $\pm$ 1.27 |  $\pm$ 0.42
Ours (K=5, $\lambda=1$) | 15.76 | 79.98 | 81.83 | 76.73 | 62.27 | 45.27 | 77.99 | 59.24 | 62.07
|  $\pm$ 2.65 |  $\pm$ 1.22 |  $\pm$ 0.18 |  $\pm$ 1.19 |  $\pm$ 0.74 |  $\pm$ 1.09 |  $\pm$ 0.49 |  $\pm$ 1.11 |  $\pm$ 0.45
GLUE 1k (BERT Base)
| CoLA | SST | MRPC | STS-B | QQP | MNLI | QNLI | RTE | Avg.
Pretrained | 42.71 | 87.08 | 86.98 | 85.93 | 70.01 | 58.05 | 77.97 | 64.34 | 71.67
|  $\pm$ 0.54 |  $\pm$ 0.18 |  $\pm$ 0.20 |  $\pm$ 0.16 |  $\pm$ 0.22 |  $\pm$ 0.62 |  $\pm$ 0.33 |  $\pm$ 0.65 |  $\pm$ 0.15
MTL | 41.57 | 86.82 | 87.44 | 87.18 | 71.92 | 62.01 | 82.39 | 66.54 | 73.26
|  $\pm$ 0.68 |  $\pm$ 0.16 |  $\pm$ 0.24 |  $\pm$ 0.15 |  $\pm$ 0.20 |  $\pm$ 0.26 |  $\pm$ 0.26 |  $\pm$ 0.53 |  $\pm$ 0.13
Ours (K=1) | 39.69 | 86.76 | 87.49 | 87.53 | 71.56 | 62.07 | 83.69 | 66.79 | 73.28
|  $\pm$ 0.63 |  $\pm$ 0.21 |  $\pm$ 0.31 |  $\pm$ 0.10 |  $\pm$ 0.28 |  $\pm$ 0.19 |  $\pm$ 0.23 |  $\pm$ 0.58 |  $\pm$ 0.13
Ours (K=5, $\lambda=0$) | 42.24 | 86.98 | 87.69 | 87.91 | 73.09 | 63.08 | 83.94 | 68.00 | 74.14
|  $\pm$ 0.59 |  $\pm$ 0.20 |  $\pm$ 0.34 |  $\pm$ 0.13 |  $\pm$ 0.15 |  $\pm$ 0.25 |  $\pm$ 0.18 |  $\pm$ 0.51 |  $\pm$ 0.12
Ours (K=5, $\lambda=0.1$) | 42.60 | 86.90 | 87.76 | 88.05 | 72.81 | 62.63 | 83.68 | 68.10 | 74.10
|  $\pm$ 0.52 |  $\pm$ 0.24 |  $\pm$ 0.33 |  $\pm$ 0.14 |  $\pm$ 0.21 |  $\pm$ 0.58 |  $\pm$ 0.13 |  $\pm$ 0.49 |  $\pm$ 0.13
Ours (K=5, $\lambda=0.5$) | 42.75 | 86.78 | 87.55 | 87.88 | 72.56 | 62.71 | 83.66 | 68.00 | 74.02
|  $\pm$ 0.49 |  $\pm$ 0.19 |  $\pm$ 0.31 |  $\pm$ 0.11 |  $\pm$ 0.21 |  $\pm$ 0.39 |  $\pm$ 0.17 |  $\pm$ 0.51 |  $\pm$ 0.12
Ours (K=5, $\lambda=1$) | 40.08 | 87.36 | 87.54 | 87.74 | 72.83 | 62.79 | 83.53 | 67.98 | 73.75
|  $\pm$ 0.90 |  $\pm$ 0.13 |  $\pm$ 0.23 |  $\pm$ 0.10 |  $\pm$ 0.16 |  $\pm$ 0.43 |  $\pm$ 0.14 |  $\pm$ 0.30 |  $\pm$ 0.14
GLUE 1k (BERT Large)
| CoLA | SST | MRPC | STS-B | QQP | MNLI | QNLI | RTE | Avg.
MTL | 49.10 | 89.84 | 87.53 | 87.85 | 73.04 | 62.70 | 84.74 | 67.52 | 75.30
|  $\pm$ 0.76 |  $\pm$ 0.18 |  $\pm$ 0.24 |  $\pm$ 0.10 |  $\pm$ 0.18 |  $\pm$ 1.85 |  $\pm$ 0.17 |  $\pm$ 0.75 |  $\pm$ 0.27
Ours (K=1) | 46.89 | 89.54 | 88.41 | 87.61 | 72.58 | 64.51 | 85.20 | 67.61 | 75.35
|  $\pm$ 0.90 |  $\pm$ 0.21 |  $\pm$ 0.25 |  $\pm$ 0.16 |  $\pm$ 0.22 |  $\pm$ 0.35 |  $\pm$ 0.16 |  $\pm$ 1.24 |  $\pm$ 0.21
Ours (K=5, $\lambda=0$) | 49.76 | 89.93 | 87.38 | 87.91 | 72.65 | 63.50 | 85.00 | 69.66 | 75.73
|  $\pm$ 0.63 |  $\pm$ 0.14 |  $\pm$ 0.38 |  $\pm$ 0.11 |  $\pm$ 0.26 |  $\pm$ 1.83 |  $\pm$ 0.23 |  $\pm$ 0.41 |  $\pm$ 0.26
Ours (K=5, $\lambda=0.1$) | 49.80 | 89.94 | 87.27 | 88.31 | 73.84 | 65.34 | 85.17 | 70.83 | 76.27
|  $\pm$ 0.69 |  $\pm$ 0.14 |  $\pm$ 0.27 |  $\pm$ 0.08 |  $\pm$ 0.19 |  $\pm$ 0.32 |  $\pm$ 0.11 |  $\pm$ 0.38 |  $\pm$ 0.12
Ours (K=5, $\lambda=0.5$) | 48.66 | 89.71 | 87.21 | 88.20 | 73.62 | 65.14 | 85.18 | 70.02 | 75.95
|  $\pm$ 0.43 |  $\pm$ 0.11 |  $\pm$ 0.36 |  $\pm$ 0.09 |  $\pm$ 0.16 |  $\pm$ 0.28 |  $\pm$ 0.15 |  $\pm$ 0.33 |  $\pm$ 0.10
Ours (K=5, $\lambda=1$) | 48.43 | 89.90 | 87.02 | 87.86 | 73.22 | 64.64 | 85.07 | 70.64 | 75.85
|  $\pm$ 0.83 |  $\pm$ 0.17 |  $\pm$ 0.39 |  $\pm$ 0.08 |  $\pm$ 0.18 |  $\pm$ 0.83 |  $\pm$ 0.15 |  $\pm$ 0.40 |  $\pm$ 0.17

Table 17: The scores on the GLUE development set. We compare different methods
using BERT${}_{\text{Base}}$ and BERT${}_{\text{Large}}$ in GLUE 100 and 1k.

GLUE Full (BERT Base)
---
| CoLA | SST | MRPC | STS-B | QQP | MNLI | QNLI | RTE | Avg.
| 8.5k | 67k | 3.5k | 5.7k | 363k | 392k | 108k | 2.5k | -
| MCC | Acc | F1 | Spearman | F1 | Acc | Acc | Acc | -
MTL$\dagger$ | 49.4 | 91.2 | 89.1 | 88.3 | 89.0 | 82.0 | 90.5 | 70.8 | 81.4
Pretrained | 59.09 | 92.71 | 89.82 | 88.13 | 87.29 | 84.33 | 91.11 | 64.42 | 82.05
|  $\pm$ 0.37 |  $\pm$ 0.07 |  $\pm$ 0.18 |  $\pm$ 0.06 |  $\pm$ 0.09 |  $\pm$ 0.07 |  $\pm$ 0.09 |  $\pm$ 0.42 |  $\pm$ 0.08
MTL | 59.36 | 92.44 | 90.18 | 89.86 | 88.01 | 84.44 | 91.61 | 70.81 | 83.30
|  $\pm$ 0.28 |  $\pm$ 0.06 |  $\pm$ 0.14 |  $\pm$ 0.04 |  $\pm$ 0.04 |  $\pm$ 0.05 |  $\pm$ 0.04 |  $\pm$ 0.46 |  $\pm$ 0.07
Ours (K=1) | 58.64 | 92.83 | 90.83 | 89.99 | 87.96 | 84.66 | 91.60 | 70.81 | 83.40
|  $\pm$ 0.40 |  $\pm$ 0.06 |  $\pm$ 0.12 |  $\pm$ 0.05 |  $\pm$ 0.06 |  $\pm$ 0.07 |  $\pm$ 0.05 |  $\pm$ 0.32 |  $\pm$ 0.07
Ours (K=5, $\lambda=0$) | 58.38 | 92.53 | 90.84 | 89.94 | 87.91 | 84.48 | 91.59 | 71.74 | 83.41
|  $\pm$ 0.34 |  $\pm$ 0.09 |  $\pm$ 0.14 |  $\pm$ 0.04 |  $\pm$ 0.04 |  $\pm$ 0.06 |  $\pm$ 0.06 |  $\pm$ 0.38 |  $\pm$ 0.07
Ours (K=5, $\lambda=0.1$) | 58.67 | 92.78 | 90.67 | 90.01 | 87.95 | 84.56 | 91.59 | 71.76 | 83.47
|  $\pm$ 0.27 |  $\pm$ 0.08 |  $\pm$ 0.19 |  $\pm$ 0.03 |  $\pm$ 0.09 |  $\pm$ 0.06 |  $\pm$ 0.05 |  $\pm$ 0.20 |  $\pm$ 0.05
Ours (K=5, $\lambda=0.5$) | 59.01 | 92.70 | 90.96 | 89.99 | 87.86 | 84.62 | 91.66 | 71.14 | 83.47
|  $\pm$ 0.22 |  $\pm$ 0.08 |  $\pm$ 0.17 |  $\pm$ 0.04 |  $\pm$ 0.07 |  $\pm$ 0.08 |  $\pm$ 0.07 |  $\pm$ 0.51 |  $\pm$ 0.08
Ours (K=5, $\lambda=1$) | 58.66 | 92.69 | 90.64 | 89.96 | 87.88 | 84.55 | 91.58 | 71.76 | 83.43
|  $\pm$ 0.28 |  $\pm$ 0.08 |  $\pm$ 0.20 |  $\pm$ 0.02 |  $\pm$ 0.10 |  $\pm$ 0.07 |  $\pm$ 0.06 |  $\pm$ 0.35 |  $\pm$ 0.07
GLUE Fulll (BERT Large)
| CoLA | SST | MRPC | STS-B | QQP | MNLI | QNLI | RTE | Avg.
MTL | 62.42 | 93.94 | 90.93 | 90.10 | 86.26 | 84.53 | 92.45 | 72.49 | 84.13
|  $\pm$ 0.26 |  $\pm$ 0.12 |  $\pm$ 0.22 |  $\pm$ 0.06 |  $\pm$ 0.11 |  $\pm$ 0.19 |  $\pm$ 0.06 |  $\pm$ 0.75 |  $\pm$ 0.11
Ours (K=1) | 62.62 | 93.82 | 91.26 | 89.89 | 86.36 | 85.09 | 92.56 | 75.17 | 84.59
|  $\pm$ 0.32 |  $\pm$ 0.11 |  $\pm$ 0.10 |  $\pm$ 0.06 |  $\pm$ 0.07 |  $\pm$ 0.03 |  $\pm$ 0.06 |  $\pm$ 0.44 |  $\pm$ 0.07
Ours (K=5, $\lambda=0$) | 62.81 | 93.93 | 90.69 | 90.04 | 86.36 | 84.84 | 92.53 | 74.96 | 84.51
|  $\pm$ 0.19 |  $\pm$ 0.10 |  $\pm$ 0.15 |  $\pm$ 0.06 |  $\pm$ 0.08 |  $\pm$ 0.11 |  $\pm$ 0.05 |  $\pm$ 0.29 |  $\pm$ 0.05
Ours (K=5, $\lambda=0.1$) | 62.63 | 93.86 | 91.03 | 90.25 | 86.42 | 84.96 | 92.59 | 75.16 | 84.61
|  $\pm$ 0.36 |  $\pm$ 0.08 |  $\pm$ 0.15 |  $\pm$ 0.05 |  $\pm$ 0.06 |  $\pm$ 0.09 |  $\pm$ 0.05 |  $\pm$ 0.45 |  $\pm$ 0.08
Ours (K=5, $\lambda=0.5$) | 62.26 | 94.03 | 90.92 | 90.11 | 86.39 | 84.84 | 92.56 | 74.87 | 84.49
|  $\pm$ 0.34 |  $\pm$ 0.05 |  $\pm$ 0.11 |  $\pm$ 0.05 |  $\pm$ 0.07 |  $\pm$ 0.12 |  $\pm$ 0.07 |  $\pm$ 0.49 |  $\pm$ 0.08
Ours (K=5, $\lambda=1$) | 63.30 | 93.97 | 90.83 | 90.11 | 86.33 | 84.99 | 92.43 | 74.98 | 84.61
|  $\pm$ 0.23 |  $\pm$ 0.09 |  $\pm$ 0.18 |  $\pm$ 0.05 |  $\pm$ 0.13 |  $\pm$ 0.10 |  $\pm$ 0.06 |  $\pm$ 0.46 |  $\pm$ 0.07

Table 18: The scores on the GLUE development set. We compare different methods
using BERT${}_{\text{Base}}$ and BERT${}_{\text{Large}}$ in GLUE Full.
$\dagger$The number copied from Aroca-Ouellette and Rudzicz (2020).

SuperGLUE 100 (BERT Base)
---
| BoolQ | CB | COPA | MultiRC | RTE | WiC | WSC | Avg.
| Acc | Acc | F1 | Acc | F1 | EM | Acc | Acc | Acc | -
Pretrained | 61.21 | 77.68 | 74.53 | 59.63 | 53.81 | 1.27 | 54.41 | 55.78 | 60.16 | 57.18
|  $\pm$ 0.30 |  $\pm$ 1.02 |  $\pm$ 2.42 |  $\pm$ 1.05 |  $\pm$ 1.23 |  $\pm$ 0.21 |  $\pm$ 0.58 |  $\pm$ 0.54 |  $\pm$ 1.06 |  $\pm$ 0.43
MTL | 61.97 | 77.23 | 72.73 | 59.69 | 52.74 | 1.41 | 56.13 | 56.03 | 61.67 | 57.50
|  $\pm$ 0.11 |  $\pm$ 1.27 |  $\pm$ 2.24 |  $\pm$ 1.09 |  $\pm$ 1.16 |  $\pm$ 0.24 |  $\pm$ 1.10 |  $\pm$ 0.39 |  $\pm$ 0.62 |  $\pm$ 0.41
Ours (K=1) | 61.53 | 76.34 | 70.84 | 58.81 | 54.53 | 1.56 | 57.27 | 56.03 | 61.66 | 57.31
|  $\pm$ 0.20 |  $\pm$ 1.27 |  $\pm$ 1.97 |  $\pm$ 0.68 |  $\pm$ 0.83 |  $\pm$ 0.18 |  $\pm$ 0.94 |  $\pm$ 0.45 |  $\pm$ 0.53 |  $\pm$ 0.35
Ours (K=5, $\lambda=0$) | 62.01 | 79.14 | 72.13 | 60.44 | 55.34 | 3.09 | 58.17 | 56.99 | 61.91 | 58.29
|  $\pm$ 0.15 |  $\pm$ 0.73 |  $\pm$ 1.82 |  $\pm$ 0.57 |  $\pm$ 0.47 |  $\pm$ 0.41 |  $\pm$ 1.09 |  $\pm$ 0.42 |  $\pm$ 0.55 |  $\pm$ 0.33
Ours (K=5, $\lambda=0.1$) | 62.04 | 78.79 | 72.67 | 60.63 | 54.31 | 3.24 | 58.15 | 56.74 | 61.24 | 58.20
|  $\pm$ 0.16 |  $\pm$ 1.06 |  $\pm$ 1.34 |  $\pm$ 0.65 |  $\pm$ 0.74 |  $\pm$ 0.39 |  $\pm$ 1.35 |  $\pm$ 0.38 |  $\pm$ 0.57 |  $\pm$ 0.31
Ours (K=5, $\lambda=0.5$) | 62.09 | 78.03 | 71.98 | 61.19 | 55.72 | 3.33 | 57.60 | 57.54 | 61.78 | 58.41
|  $\pm$ 0.14 |  $\pm$ 0.95 |  $\pm$ 1.83 |  $\pm$ 1.13 |  $\pm$ 0.76 |  $\pm$ 0.40 |  $\pm$ 1.14 |  $\pm$ 0.60 |  $\pm$ 0.61 |  $\pm$ 0.38
Ours (K=5, $\lambda=1$) | 61.94 | 77.80 | 69.21 | 59.94 | 55.96 | 3.97 | 57.76 | 56.29 | 62.57 | 57.84
|  $\pm$ 0.25 |  $\pm$ 0.77 |  $\pm$ 2.39 |  $\pm$ 0.59 |  $\pm$ 0.45 |  $\pm$ 0.32 |  $\pm$ 1.17 |  $\pm$ 0.42 |  $\pm$ 0.32 |  $\pm$ 0.40
SuperGLUE 100 (BERT Large)
| BoolQ | CB | COPA | MultiRC | RTE | WiC | WSC | Avg.
MTL | 62.03 | 78.14 | 74.21 | 64.31 | 55.76 | 1.32 | 58.24 | 56.42 | 62.28 | 59.03
|  $\pm$ 0.13 |  $\pm$ 1.80 |  $\pm$ 3.23 |  $\pm$ 1.04 |  $\pm$ 1.40 |  $\pm$ 0.29 |  $\pm$ 1.34 |  $\pm$ 0.35 |  $\pm$ 0.63 |  $\pm$ 0.54
Ours (K=1) | 60.49 | 77.36 | 71.39 | 61.63 | 52.96 | 1.18 | 57.04 | 55.79 | 61.43 | 57.35
|  $\pm$ 0.38 |  $\pm$ 0.92 |  $\pm$ 2.38 |  $\pm$ 1.13 |  $\pm$ 1.04 |  $\pm$ 0.18 |  $\pm$ 0.74 |  $\pm$ 0.42 |  $\pm$ 0.74 |  $\pm$ 0.42
Ours (K=5, $\lambda=0$) | 62.08 | 78.90 | 75.26 | 64.63 | 51.08 | 3.46 | 61.07 | 57.38 | 61.37 | 59.46
|  $\pm$ 0.07 |  $\pm$ 1.45 |  $\pm$ 2.40 |  $\pm$ 1.13 |  $\pm$ 1.23 |  $\pm$ 0.37 |  $\pm$ 0.58 |  $\pm$ 0.69 |  $\pm$ 1.07 |  $\pm$ 0.44
Ours (K=5, $\lambda=0.1$) | 62.18 | 80.36 | 77.08 | 64.19 | 51.48 | 3.61 | 62.43 | 57.37 | 61.13 | 59.88
|  $\pm$ 0.01 |  $\pm$ 1.46 |  $\pm$ 2.46 |  $\pm$ 0.93 |  $\pm$ 1.47 |  $\pm$ 0.35 |  $\pm$ 0.47 |  $\pm$ 0.77 |  $\pm$ 0.71 |  $\pm$ 0.43
Ours (K=5, $\lambda=0.5$) | 62.19 | 80.69 | 77.28 | 63.25 | 52.87 | 2.99 | 60.07 | 56.80 | 61.24 | 59.42
|  $\pm$ 0.01 |  $\pm$ 0.95 |  $\pm$ 1.79 |  $\pm$ 0.89 |  $\pm$ 1.00 |  $\pm$ 0.35 |  $\pm$ 0.77 |  $\pm$ 0.57 |  $\pm$ 0.76 |  $\pm$ 0.34
Ours (K=5, $\lambda=1$) | 62.06 | 79.14 | 73.22 | 62.69 | 50.01 | 3.14 | 60.81 | 57.24 | 61.44 | 58.74
|  $\pm$ 0.08 |  $\pm$ 1.47 |  $\pm$ 3.05 |  $\pm$ 0.88 |  $\pm$ 1.54 |  $\pm$ 0.41 |  $\pm$ 1.00 |  $\pm$ 0.49 |  $\pm$ 0.62 |  $\pm$ 0.50
SuperGLUE 1k (BERT Base)
| BoolQ | CB | COPA | MultiRC | RTE | WiC | WSC | Avg.
Pretrained | 62.89 | 87.00 | 85.63 | 60.94 | 55.37 | 5.19 | 59.39 | 60.40 | 64.54 | 61.55
|  $\pm$ 0.27 |  $\pm$ 0.80 |  $\pm$ 1.51 |  $\pm$ 0.53 |  $\pm$ 0.81 |  $\pm$ 0.72 |  $\pm$ 0.56 |  $\pm$ 0.44 |  $\pm$ 0.34 |  $\pm$ 0.37
MTL | 63.38 | 85.49 | 82.85 | 60.91 | 56.52 | 7.44 | 65.96 | 63.61 | 65.61 | 62.94
|  $\pm$ 0.39 |  $\pm$ 0.79 |  $\pm$ 1.37 |  $\pm$ 0.52 |  $\pm$ 0.69 |  $\pm$ 0.64 |  $\pm$ 1.06 |  $\pm$ 0.21 |  $\pm$ 0.32 |  $\pm$ 0.36
Ours (K=1) | 63.87 | 86.83 | 84.28 | 60.63 | 58.68 | 7.86 | 66.34 | 65.00 | 64.09 | 63.35
|  $\pm$ 0.45 |  $\pm$ 0.47 |  $\pm$ 0.65 |  $\pm$ 0.39 |  $\pm$ 0.19 |  $\pm$ 0.14 |  $\pm$ 0.38 |  $\pm$ 0.29 |  $\pm$ 0.35 |  $\pm$ 0.18
Ours (K=5, $\lambda=0$) | 63.27 | 86.49 | 82.18 | 62.88 | 59.03 | 7.76 | 67.60 | 65.18 | 65.03 | 63.71
|  $\pm$ 0.41 |  $\pm$ 0.46 |  $\pm$ 0.90 |  $\pm$ 0.64 |  $\pm$ 0.72 |  $\pm$ 0.45 |  $\pm$ 0.48 |  $\pm$ 0.34 |  $\pm$ 0.37 |  $\pm$ 0.26
Ours (K=5, $\lambda=0.1$) | 63.20 | 86.38 | 82.63 | 62.53 | 59.16 | 8.36 | 67.11 | 65.11 | 64.45 | 63.61
|  $\pm$ 0.39 |  $\pm$ 0.57 |  $\pm$ 0.98 |  $\pm$ 0.61 |  $\pm$ 0.34 |  $\pm$ 0.23 |  $\pm$ 0.58 |  $\pm$ 0.27 |  $\pm$ 0.38 |  $\pm$ 0.27
Ours (K=5, $\lambda=0.5$) | 63.25 | 86.83 | 84.14 | 61.97 | 59.57 | 8.10 | 66.71 | 65.38 | 64.78 | 63.78
|  $\pm$ 0.38 |  $\pm$ 0.57 |  $\pm$ 0.91 |  $\pm$ 0.51 |  $\pm$ 0.20 |  $\pm$ 0.41 |  $\pm$ 0.49 |  $\pm$ 0.42 |  $\pm$ 0.42 |  $\pm$ 0.25
Ours (K=5, $\lambda=1$) | 63.12 | 86.88 | 83.97 | 61.66 | 58.57 | 7.70 | 66.83 | 65.15 | 64.56 | 63.56
|  $\pm$ 0.42 |  $\pm$ 0.51 |  $\pm$ 0.78 |  $\pm$ 0.52 |  $\pm$ 0.46 |  $\pm$ 0.22 |  $\pm$ 0.41 |  $\pm$ 0.39 |  $\pm$ 0.36 |  $\pm$ 0.22
SuperGLUE 1k (BERT Large)
| BoolQ | CB | COPA | MultiRC | RTE | WiC | WSC | Avg.
MTL | 63.86 | 88.67 | 87.83 | 67.22 | 56.56 | 7.52 | 68.68 | 66.16 | 64.67 | 65.21
|  $\pm$ 0.42 |  $\pm$ 0.87 |  $\pm$ 1.46 |  $\pm$ 0.73 |  $\pm$ 0.64 |  $\pm$ 0.53 |  $\pm$ 0.83 |  $\pm$ 0.29 |  $\pm$ 0.29 |  $\pm$ 0.38
Ours (K=1) | 63.22 | 87.11 | 85.15 | 66.09 | 59.81 | 6.86 | 67.89 | 65.45 | 65.28 | 64.67
|  $\pm$ 0.35 |  $\pm$ 0.85 |  $\pm$ 1.74 |  $\pm$ 0.63 |  $\pm$ 0.35 |  $\pm$ 0.87 |  $\pm$ 0.77 |  $\pm$ 0.23 |  $\pm$ 0.49 |  $\pm$ 0.43
Ours (K=5, $\lambda=0$) | 63.73 | 87.61 | 86.19 | 70.12 | 54.34 | 7.39 | 69.09 | 66.84 | 64.85 | 65.43
|  $\pm$ 0.47 |  $\pm$ 0.75 |  $\pm$ 1.11 |  $\pm$ 0.66 |  $\pm$ 3.53 |  $\pm$ 0.79 |  $\pm$ 0.61 |  $\pm$ 0.33 |  $\pm$ 0.33 |  $\pm$ 0.38
Ours (K=5, $\lambda=0.1$) | 64.73 | 87.51 | 87.14 | 68.09 | 58.56 | 8.96 | 68.85 | 66.57 | 64.31 | 65.59
|  $\pm$ 0.52 |  $\pm$ 0.60 |  $\pm$ 0.82 |  $\pm$ 0.65 |  $\pm$ 0.34 |  $\pm$ 0.20 |  $\pm$ 0.53 |  $\pm$ 0.43 |  $\pm$ 0.46 |  $\pm$ 0.25
Ours (K=5, $\lambda=0.5$) | 63.55 | 87.83 | 87.45 | 68.88 | 58.66 | 8.86 | 69.78 | 66.64 | 64.44 | 65.84
|  $\pm$ 0.46 |  $\pm$ 0.52 |  $\pm$ 0.74 |  $\pm$ 0.78 |  $\pm$ 0.37 |  $\pm$ 0.21 |  $\pm$ 0.45 |  $\pm$ 0.36 |  $\pm$ 0.36 |  $\pm$ 0.25
Ours (K=5, $\lambda=1$) | 63.83 | 86.72 | 84.79 | 67.75 | 56.87 | 8.14 | 68.33 | 66.93 | 64.84 | 65.00
|  $\pm$ 0.38 |  $\pm$ 0.68 |  $\pm$ 1.06 |  $\pm$ 0.59 |  $\pm$ 0.76 |  $\pm$ 0.61 |  $\pm$ 0.44 |  $\pm$ 0.30 |  $\pm$ 0.38 |  $\pm$ 0.29

Table 19: The scores on the development set of the tasks in SuperGLUE except
for ReCoRD. We compare different methods using BERT${}_{\text{Base}}$ and
BERT${}_{\text{Large}}$ in SuperGLUE 100 and 1k.

SuperGLUE Full (BERT Base)
---
| BoolQ | CB | COPA | MultiRC | RTE | WiC | WSC | ReCoRD | Avg.
| 9.4k | 250 | 400 | 5.1k | 2.5k | 6k | 554 | 101k | -
| Acc | Acc | F1 | Acc | F1 | EM | Acc | Acc | Acc | F1 | EM | -
Pretrained | 74.01 | 87.00 | 85.63 | 60.94 | 65.93 | 16.72 | 65.76 | 66.85 | 64.33 | 58.78 | 58.10 | 65.04
|  $\pm$ 0.34 |  $\pm$ 0.80 |  $\pm$ 1.51 |  $\pm$ 0.53 |  $\pm$ 0.13 |  $\pm$ 0.17 |  $\pm$ 0.44 |  $\pm$ 0.29 |  $\pm$ 0.40 |  $\pm$ 0.62 |  $\pm$ 0.62 |  $\pm$ 0.36
MTL | 77.46 | 85.49 | 82.85 | 60.91 | 65.45 | 16.03 | 72.09 | 69.77 | 65.56 | 59.10 | 58.38 | 66.33
|  $\pm$ 0.24 |  $\pm$ 0.79 |  $\pm$ 1.37 |  $\pm$ 0.52 |  $\pm$ 0.13 |  $\pm$ 0.15 |  $\pm$ 0.59 |  $\pm$ 0.25 |  $\pm$ 0.32 |  $\pm$ 0.39 |  $\pm$ 0.40 |  $\pm$ 0.33
Ours (K=1) | 77.46 | 86.83 | 84.28 | 60.63 | 65.34 | 15.89 | 72.19 | 70.76 | 64.55 | 57.68 | 56.98 | 66.29
|  $\pm$ 0.13 |  $\pm$ 0.47 |  $\pm$ 0.65 |  $\pm$ 0.39 |  $\pm$ 0.19 |  $\pm$ 0.18 |  $\pm$ 0.55 |  $\pm$ 0.16 |  $\pm$ 0.32 |  $\pm$ 0.97 |  $\pm$ 0.96 |  $\pm$ 0.18
Ours (K=5, $\lambda=0$) | 77.57 | 86.49 | 82.18 | 62.87 | 65.79 | 16.03 | 72.77 | 70.68 | 65.14 | 60.20 | 59.48 | 66.80
|  $\pm$ 0.31 |  $\pm$ 0.46 |  $\pm$ 0.90 |  $\pm$ 0.64 |  $\pm$ 0.11 |  $\pm$ 0.20 |  $\pm$ 0.44 |  $\pm$ 0.21 |  $\pm$ 0.26 |  $\pm$ 0.57 |  $\pm$ 0.56 |  $\pm$ 0.25
Ours (K=5, $\lambda=0.1$) | 77.29 | 86.38 | 82.63 | 62.53 | 65.66 | 16.17 | 72.24 | 70.58 | 65.31 | 60.27 | 59.55 | 66.74
|  $\pm$ 0.16 |  $\pm$ 0.57 |  $\pm$ 0.98 |  $\pm$ 0.61 |  $\pm$ 0.13 |  $\pm$ 0.24 |  $\pm$ 0.59 |  $\pm$ 0.18 |  $\pm$ 0.28 |  $\pm$ 0.48 |  $\pm$ 0.48 |  $\pm$ 0.26
Ours (K=5, $\lambda=0.5$) | 76.84 | 86.83 | 84.14 | 61.97 | 65.58 | 15.84 | 72.11 | 70.88 | 65.81 | 59.85 | 59.10 | 66.80
|  $\pm$ 0.27 |  $\pm$ 0.57 |  $\pm$ 0.91 |  $\pm$ 0.51 |  $\pm$ 0.13 |  $\pm$ 0.22 |  $\pm$ 0.39 |  $\pm$ 0.20 |  $\pm$ 0.40 |  $\pm$ 0.48 |  $\pm$ 0.48 |  $\pm$ 0.24
Ours (K=5, $\lambda=1$) | 76.69 | 86.88 | 83.97 | 61.66 | 65.33 | 16.23 | 71.54 | 70.43 | 65.14 | 58.62 | 57.88 | 66.39
|  $\pm$ 0.27 |  $\pm$ 0.51 |  $\pm$ 0.78 |  $\pm$ 0.52 |  $\pm$ 0.20 |  $\pm$ 0.18 |  $\pm$ 0.68 |  $\pm$ 0.32 |  $\pm$ 0.30 |  $\pm$ 0.97 |  $\pm$ 0.95 |  $\pm$ 0.22
SuperGLUE Full (BERT Large)
MTL | 77.78 | 88.67 | 87.83 | 67.22 | 65.93 | 16.68 | 71.97 | 71.08 | 64.37 | 69.60 | 68.85 | 69.16
|  $\pm$ 0.35 |  $\pm$ 0.87 |  $\pm$ 1.46 |  $\pm$ 0.73 |  $\pm$ 0.19 |  $\pm$ 0.29 |  $\pm$ 1.08 |  $\pm$ 0.22 |  $\pm$ 0.24 |  $\pm$ 0.60 |  $\pm$ 0.61 |  $\pm$ 0.37
Ours (K=1) | 78.04 | 87.11 | 85.15 | 66.09 | 65.96 | 16.49 | 75.54 | 70.62 | 65.02 | 70.12 | 69.43 | 69.24
|  $\pm$ 0.40 |  $\pm$ 0.85 |  $\pm$ 1.74 |  $\pm$ 0.63 |  $\pm$ 0.15 |  $\pm$ 0.28 |  $\pm$ 0.43 |  $\pm$ 0.14 |  $\pm$ 0.36 |  $\pm$ 0.10 |  $\pm$ 0.10 |  $\pm$ 0.41
Ours (K=5, $\lambda=0$) | 78.21 | 87.61 | 86.19 | 70.12 | 64.91 | 15.62 | 73.34 | 71.73 | 65.26 | 69.28 | 68.58 | 69.56
|  $\pm$ 0.25 |  $\pm$ 0.75 |  $\pm$ 1.11 |  $\pm$ 0.66 |  $\pm$ 0.71 |  $\pm$ 1.18 |  $\pm$ 0.58 |  $\pm$ 0.20 |  $\pm$ 0.38 |  $\pm$ 0.45 |  $\pm$ 0.47 |  $\pm$ 0.31
Ours (K=5, $\lambda=0.1$) | 78.70 | 87.51 | 87.14 | 68.09 | 65.83 | 17.67 | 75.36 | 71.41 | 65.44 | 69.88 | 69.25 | 69.98
|  $\pm$ 0.20 |  $\pm$ 0.60 |  $\pm$ 0.82 |  $\pm$ 0.65 |  $\pm$ 0.13 |  $\pm$ 0.25 |  $\pm$ 0.32 |  $\pm$ 0.24 |  $\pm$ 0.52 |  $\pm$ 0.35 |  $\pm$ 0.40 |  $\pm$ 0.24
Ours (K=5, $\lambda=0.5$) | 78.54 | 87.83 | 87.45 | 68.88 | 66.06 | 16.66 | 74.83 | 71.49 | 64.83 | 68.87 | 68.15 | 69.79
|  $\pm$ 0.39 |  $\pm$ 0.52 |  $\pm$ 0.74 |  $\pm$ 0.78 |  $\pm$ 0.16 |  $\pm$ 0.33 |  $\pm$ 0.62 |  $\pm$ 0.20 |  $\pm$ 0.36 |  $\pm$ 0.52 |  $\pm$ 0.51 |  $\pm$ 0.25
Ours (K=5, $\lambda=1$) | 77.49 | 86.72 | 84.79 | 67.75 | 65.77 | 17.09 | 74.46 | 70.97 | 64.79 | 68.45 | 67.73 | 69.04
|  $\pm$ 0.29 |  $\pm$ 0.68 |  $\pm$ 1.06 |  $\pm$ 0.59 |  $\pm$ 0.27 |  $\pm$ 0.41 |  $\pm$ 0.47 |  $\pm$ 0.21 |  $\pm$ 0.40 |  $\pm$ 0.37 |  $\pm$ 0.39 |  $\pm$ 0.27

Table 20: The scores on the SuperGLUE development set. We compare different
methods using BERT${}_{\text{Base}}$ and BERT${}_{\text{Large}}$ in SuperGLUE
Full.

GLUE 100 ECE (BERT Base)
---
| CoLA | SST | MRPC | QQP | MNLI | QNLI | RTE | Avg.
Ours (K=1) | 27.15 | 19.96 | 10.90 | 32.18 | 32.70 | 23.37 | 30.26 | 25.22
Ours (K=5, $\lambda=0.1$) | 24.21 | 14.40 | 20.06 | 16.01 | 17.02 | 6.53 | 9.96 | 15.46
GLUE 1k ECE (BERT Base)
| CoLA | SST | MRPC | QQP | MNLI | QNLI | RTE | Avg.
Ours (K=1) | 21.72 | 10.19 | 15.21 | 13.93 | 35.21 | 15.58 | 23.43 | 19.32
Ours (K=5, $\lambda=0.1$) | 20.50 | 7.52 | 16.11 | 14.36 | 32.67 | 15.00 | 12.88 | 17.01
GLUE Full ECE (BERT Base)
| CoLA | SST | MRPC | QQP | MNLI | QNLI | RTE | Avg.
Ours (K=1) | 14.90 | 3.07 | 10.45 | 5.28 | 4.67 | 2.38 | 22.20 | 8.99
Ours (K=5, $\lambda=0.1$) | 15.06 | 4.23 | 5.97 | 4.85 | 4.43 | 3.52 | 23.35 | 8.77

Table 21: The comparison of expected calibration error (ECE) in the
classification tasks of GLUE.

GLUE 100 (BERT Base)
---
| CoLA | SST | MRPC | QQP | MNLI | QNLI | RTE | Avg.
CLS vs ENS | 32.93 | 47.13 | 56.79 | 25.70 | 21.70 | 21.47$\dagger$ | 22.27$\dagger$ | 32.57
Dropout vs ENS | 33.65 | 47.56 | 54.63 | 30.88 | 24.76 | 38.28 | 30.45$\dagger$ | 37.17
Least vs ENS | 38.70 | 48.28 | 61.11 | 26.84 | 24.66 | 42.40 | 35.00 | 39.57
ENS vs ENS | 35.34 | 42.53 | 59.88 | 31.87 | 26.58 | 43.13 | 31.36$\dagger$ | 38.67
GLUE 1k (BERT Base)
| CoLA | SST | MRPC | QQP | MNLI | QNLI | RTE | Avg.
CLS vs ENS | 53.25 | 46.98 | 46.91 | 34.76 | 32.63 | 49.45 | 25.45$\dagger$ | 41.35
Dropout vs ENS | 45.55 | 54.31 | 46.30 | 49.91 | 37.76 | 53.50 | 31.36$\dagger$ | 45.53
Least vs ENS | 59.01 | 60.92 | 51.54 | 48.24 | 37.68 | 54.53 | 30.00$\dagger$ | 48.85
ENS vs ENS | 57.09 | 59.48 | 50.31 | 50.62 | 41.00 | 56.14 | 36.36 | 50.14

Table 22: The overlapping ratio of the top 20% most uncertain examples using
different uncertainty estimation methods. All the scores are significantly
larger than the random ratio 0.2 with p value < $10^{-4}$ except for the
values beside $\dagger$.