# On Separate Normalization in Self-supervised Transformers

Xiaohui Chen
Department of Computer Science
Tufts University
Medford, MA 02155
<EMAIL_ADDRESS>
&Yinkai Wang
Department of Computer Science
Tufts University
Medford, MA 02155
<EMAIL_ADDRESS>
Yuanqi Du
Department of Computer Science
Cornell University
Ithaca, NY 14850
<EMAIL_ADDRESS>
&Soha Hassoun
Department of Computer Science
Tufts University
Medford, MA 02155
<EMAIL_ADDRESS>
&Li-Ping Liu
Department of Computer Science
Tufts University
Medford, MA 02155
<EMAIL_ADDRESS>

###### Abstract

Self-supervised training methods for transformers have demonstrated remarkable
performance across various domains. Previous transformer-based models, such as
masked autoencoders (MAE), typically utilize a single normalization layer for
both the $\mathrm{[CLS]}$ symbol and the tokens. We propose in this paper a
simple modification that employs separate normalization layers for the tokens
and the [CLS] symbol to better capture their distinct characteristics and
enhance downstream task performance. Our method aims to alleviate the
potential negative effects of using the same normalization statistics for both
token types, which may not be optimally aligned with their individual roles.
We empirically show that by utilizing a separate normalization layer, the
$\mathrm{[CLS]}$ embeddings can better encode the global contextual
information and are distributed more uniformly in its anisotropic space. When
replacing the conventional normalization layer with the two separate layers,
we observe an average 2.7% performance improvement over the image, natural
language, and graph domains.

##  1 Introduction

Transformer models [Vaswani et al., 2017] have revolutionized natural language
processing (NLP) [Devlin et al., 2018, Liu et al., 2019] and and have
demonstrated remarkable success across a wide range of tasks. The significance
of transformer models lies in their ability to model context and capture
complex linguistic patterns without being constrained by the sequential nature
of data. Moreover, transformers have been applied beyond NLP, finding success
in areas such as Computer Vision (CV) [Han et al., 2022], speech recognition
[Karmakar et al., 2021], and recommendation systems [Sun et al., 2019, Gu et
al., 2020, Wu et al., 2020]. Their flexible architecture and ability to
capture dependencies have made them adaptable to diverse data modalities and
problem domains.

Transformer architectures are studied extensively from various perspectives,
including attention mechanisms, positional encoding [Devlin et al., 2018], and
normalization techniques. Specifically, layer normalization [Ba et al., 2016]
and batch normalization [Ioffe and Szegedy, 2015] are employed to enhance
stability and hasten convergence during training. The literature on
transformers also explores parameter initialization [Xu et al., 2019a],
optimization algorithms [Huang et al., 2020], regularization techniques
[Steiner et al., 2021, Zhou et al., 2020], and architectural modifications
[Han et al., 2021]. This collective research has advanced the understanding of
transformer architectures and their applications in NLP, CV, and other machine
learning domains.

The study of normalization in transformer architectures is motivated by
several factors [Xiong et al., 2020, Shen et al., 2020, Nguyen and Salazar,
2019]. For example, Xiong et al. [2020] reveals the significance of the warm-
up stage in learning rate adjustment and the positioning of layer
normalization for achieving stable training and faster convergence. Shen et
al. [2020] investigates the disadvantage of using batch normalization in
transformers and proposes power normalization. While most previous works focus
on how the normalization layer can be modified to stabilize the training
process, it is less understood how the normalization affects the encoding
abilities of the input instance.

Figure 1: Comparison of the shared normalization (ShareNorm, left) and the
proposed separate normalization (SepNorm, right) configurations for token
normalization. In the ShareNorm setup, both the $\mathrm{[CLS]}$ symbol and
other tokens are normalized using a single-layer normalization. In the SepNorm
setup, separate normalization is employed, e.g., the $\mathrm{[CLS]}$ symbol
is normalized through batch normalization, while the other tokens are
normalized via layer normalization.

In self-supervised transformers, the $\mathrm{[CLS]}$ symbol is frequently
used as a global representation for various downstream tasks [Devlin et al.,
2018, He et al., 2022]. Often, the normalization applied to the
$\mathrm{[CLS]}$ symbol is shared with the rest of the tokens in the sequence.
Given that the $\mathrm{[CLS]}$ symbol plays a special role in representation
learning, a natural question is whether we should treat it separately in the
normalization operation. Driven by this question, our research first
scrutinizes the behavior of Shared Normalization (ShareNorm) in transformers,
particularly exploring the properties of the $\mathrm{[CLS]}$ embedding and
its influence on downstream task performance. Subsequently, we propose a
replacement for ShareNorm with Separate Normalization (SepNorm), which employs
distinct normalization layers for the $\mathrm{[CLS]}$ symbol and the token
features, as depicted in Figure 1. Through extensive analysis, we demonstrate
that $\mathrm{[CLS]}$ embeddings learned using ShareNorm have the issue of
dimensional collapse, which cannot be rectified even by enforcing uniformity
[Wang and Isola, 2020]. However, the straightforward substitution of SepNorm
in place of ShareNorm can substantially mitigate this issue. We empirically
validate the effectiveness of SepNorm on the image, text, and graph domains,
demonstrating the universal applicability of the proposed SepNorm.

##  2 Background

###  2.1 Pretraining Transformers with the $\mathrm{[CLS]}$ symbol

Unsupervised pretraining of a transformer-based model [Vaswani et al., 2017]
is widely investigated in many domains, including NLP, computer vision (CV),
and graphs.

#### Pretraining BERT for NLP.

In NLP, BERT [Devlin et al., 2018] is first proposed to pretrain a
transformer-based language model by performing the following two tasks –
masked language modeling and next sentence prediction. During pretraining,
BERT takes a pair of sentences $(\mathbf{x},\mathbf{y})$, which are
represented as a special sequence

$\displaystyle\mathbf{s}=\big{(}\mathrm{[CLS]},\mathbf{x},\mathrm{[SEP]},\mathbf{y}\big{)}.$
(1)

Here $\mathrm{[SEP]}$ is a special token that separates the two sentences. A
fraction (e.g., 15%) of the tokens in $\mathbf{x}$ and $\mathbf{y}$ are
randomly replaced by a special symbol $\mathrm{[MASK]}$. The first task in
BERT is to predict the original tokens replaced by $\mathrm{[MASK]}$ with
cross-entropy loss. The second task is to predict whether $\mathbf{y}$ is the
next sentence following $\mathbf{x}$, and the decision is made by classifying
the final embedding of the $\mathrm{[CLS]}$ symbol. After pretraining, the
representation in the $\mathrm{[CLS]}$ is usually used for sentence-level
downstream tasks such as sentiment analysis [Medhat et al., 2014].

#### Pretraining MAE for CV.

The Vision Transformer (ViT) [Dosovitskiy et al., 2020] applies the
transformers to computer vision tasks. In ViT, an image is usually voxelized
into $16\times 16$ patches, which are then flattened into a sequence of 256
tokens and fed into the ViT. Using the ViT architecture, He et al. [2022]
proposes a self-supervised training scheme called Masked Autoencoder (MAE).
The input to the MAE is an “image sequence” with 75% of the tokens masked.
When fed into the encoder, the masked sequence also has a $\mathrm{[CLS]}$
symbol attached to the beginning. The encoder then computes the
representations for the unmasked image tokens as well as the $\mathrm{[CLS]}$
symbol. The input to the decoder consists of the representations of encoded
and masked tokens. The decoder then learns to reconstruct the images by
minimizing the mean square error (MSE). Note that only the encoder will be
used for downstream tasks after pretraining. The $\mathrm{[CLS]}$ symbol is
treated as the class token for linear probing and fine-tuning in the
downstream tasks.

#### Pretraining Graphormer for molecule discovery.

Graphormer [Ying et al., 2021] proposes to modify several parts of the
transformer layer for graph representation learning tasks. The learning task
is to predict the property of a graph rather than an individual node or edge.
Specifically, Graphormer introduces a new symbol $\mathrm{[VNode]}$ as a node
connecting to all original graph nodes. Then the vector learned for
$\mathrm{[VNode]}$ represents the global information of the entire graph. The
mechanism of $\mathrm{[VNode]}$ bears resemblance to the $\mathrm{[CLS]}$
symbol in BERT and MAE.

Since the $\mathrm{[CLS]}$ symbol is not a natural data token, we argue that
it should be treated differently as it is now established that it summarizes
information from all other tokens and is widely used by downstream tasks.

###  2.2 Normalization Layers in Transformers

Given that transformers are initially proposed for NLP tasks, layer
normalization (LN), proposed by Ba et al. [2016], is typically the
normalization method of choice [Xiong et al., 2020]. LN normalizes across the
feature dimension and is independent of the sequence length and the batch
size. For any features $\mathbf{h}\in\mathbb{R}^{d}$, the LN is computed as:

$\displaystyle\mathrm{LN}(\mathbf{h})=\mathbold{\gamma}\odot\frac{\mathbf{h}-\mu}{\sigma}+\mathbold{\beta},\quad\mu=\frac{1}{d}\sum_{i=1}^{d}h_{i},\quad\sigma=\sqrt{\frac{1}{d}\sum^{d}_{i=1}\big{(}h_{i}-\mu\big{)}^{2}}.$
(2)

Here $h_{i}$ is the $i$-th dimension of $\mathbf{h}$, $\odot$ represents
element-wise multiplication, and
$\mathbold{\gamma},\mathbold{\beta}\in\mathbb{R}^{d}$ are scale and bias
parameters, respectively. In a transformer, all tokens, including special
tokens, such as $\mathrm{[CLS]}$ and $\mathrm{[SEP]}$, are all treated equally
and share the same LNs.

Batch Normalization (BN) [Ioffe and Szegedy, 2015] works by normalizing the
input data to have zero mean and unit variance along the batch dimension,
followed by an affine transformation to scale the result using gamma and beta
parameters. BN normalizes a given vector $\mathbf{h}$ as:

$\displaystyle\mathrm{BN}(\mathbf{h})=\mathbold{\gamma}\odot\frac{\mathbf{h}-\mathbold{\mu}_{B}}{\mathbold{\sigma}_{B}}+\mathbold{\beta}.$
(3)

Here $\mathbold{\mu}_{B},\mathbold{\sigma}_{B}^{2}\in\mathbb{R}^{d}$ are the
running statistics (mean and variance) maintained by the BN. The running means
and variance are updated during training after each batch. They are usually
calculated as an exponential moving average of the batch mean and variance. BN
is widely adopted in CV but leads to significant performance degradation when
naively used in NLP.

###  2.3 Uniformity of the Learned Representations

The dimensional collapse in self-supervised representation learning is a
common phenomenon where the embedding vectors only span a lower-dimensional
subspace [Jing et al., 2021]. This means that the model fails to capture the
full complexity of the input data and instead collapses to a simpler
representation. Contrastive methods [Oord et al., 2018, Chen et al., 2020b]
have been one of the standard approaches to address this problem.
Specifically, based on Gaussian potential, Wang and Isola [2020] propose the
uniformity metric (loss) to quantify the degree of dimensional collapse. Given
a set of representation vectors $\\{\mathbf{h}_{1},\ldots,\mathbf{h}_{N}\\}$
from a dataset of size $N$, the uniformity metric,
$\mathcal{L}_{\mathcal{U}}$, is computed as follows:

$\displaystyle\mathcal{L}_{\mathcal{U}}=\log\frac{1}{N(N-1)/2}\sum_{\begin{subarray}{c}n=1,\\\
m=n+1\end{subarray}}^{N,N}\mathrm{exp}^{-2\big{\|}\frac{\mathbf{h}_{n}}{\|\mathbf{h}_{n}\|}-\frac{\mathbf{h}_{m}}{\|\mathbf{h}_{m}\|}\big{\|}^{2}}.$
(4)

If the distribution of the representation is perfectly uniform, then the
numerical value of $\mathcal{L}_{\mathcal{U}}$ will converge to -4 as the
dimension of $\mathbf{h}$ increases to infinity [Wang and Isola, 2020].

In self-supervised transformers, the uniformity of the representation is also
taken into consideration by some works. For example, Gao et al. [2021]
finetunes the BERT pretrained model using the InfoNCE loss [Oord et al.,
2018], and Zhang et al. [2022] jointly train the MAE loss along with
uniformity loss.

##  3 Approach

###  3.1 Separate Normalization Layer

We present SepNorm, a normalization scheme that incorporates two types of
normalization layers, with one for the $\mathrm{[CLS]}$ symbol and one for the
other tokens. SepNorm allows different normalization layers for different
token types. In this work, we focus on exploring combinations of BN and LN.

For instance, if we assign BN to the $\mathrm{[CLS]}$ symbol and LN to the
tokens, the learnable parameters are structured as
$g_{1}=\\{\mathbold{\gamma}_{1},\mathbold{\beta}_{1},\mathbold{\mu},\mathbold{\sigma}\\}$
and $g_{2}=\\{\mathbold{\gamma}_{2},\mathbold{\beta}_{2}\\}$. Let
$\mathbf{H}\in\mathbb{R}^{L\times d}$ represent the feature sequence, where
$L$ denotes the sequence length. The normalization process is as follows:

$\displaystyle\mathbf{H}^{\prime}=\big{(}\mathrm{BN}(\mathbf{H}_{0};g_{1}),\mathrm{LN}(\mathbf{H}_{1};g_{2}),\ldots,\mathrm{LN}(\mathbf{H}_{L};g_{2})\big{)},$
(5)

where $\mathbf{H}^{\prime}$ denotes the normalized features. By convention, we
assume the features in the first position correspond to the $\mathrm{[CLS]}$
symbol. This normalization method is designed to allow the $\mathrm{[CLS]}$
features to be encoded distinctly from other tokens, a crucial aspect when
considering the original transformer. In the original transformer, the update
directions of the LN parameters $\\{\mathbold{\gamma},\mathbold{\beta}\\}$ are
primarily driven by the training signal of the tokens:

$\displaystyle\frac{\delta\mathcal{L}}{\delta\gamma_{i}}=\sum_{l=1}^{L}\frac{\delta\mathcal{L}}{\delta\tilde{\mathbf{H}}_{l,i}}\tilde{\mathbf{H}}_{l,i},\quad\frac{\delta\mathcal{L}}{\delta\beta_{i}}=\sum_{l=1}^{L}\frac{\delta\mathcal{L}}{\delta\tilde{\mathbf{H}}_{l,i}},$
(6)
$\displaystyle\text{where}~{}\tilde{\mathbf{H}}_{l,i}=\frac{\mathbf{H}_{l,i}-\mu_{l}}{\sigma_{l}},\mu_{l}=\frac{1}{d}\sum_{i=1}^{d}\mathbf{H}_{l,i},\sigma_{l}=\sqrt{\frac{1}{d}\sum^{d}_{i=1}\big{(}\mathbf{H}_{l,i}-\mu_{l}\big{)}^{2}}.$
(7)

The backpropagated gradient of $\mathcal{L}$ on the $i$-th dimension
parameters in LN is influenced by both token types. Given the potentially
diverse characteristics (i.e., mean and scale) of feature distributions, it
might be challenging to accommodate both token types simultaneously. Moreover,
the requirement of mapping two types of token features into an identical
sphere obliges them to diverge under the constraint
$\sum_{i=1}^{d}{(\mathbf{h}^{\prime}_{i}-\beta_{i})^{2}}/{\gamma_{i}^{2}}=1$.
Since the numbers of seen $\mathrm{[CLS]}$ features are relatively fewer than
the token features, the volume of the hyperspace accommodating the
$\mathrm{[CLS]}$ features is consequently compressed. Figure 2(a, b)
demonstrates this phenomenon in the scenario where both token types utilize a
ShareNorm normalization layer and how using SepNorm mitigates this effect. By
projecting the $\mathrm{[CLS]}$ and token embeddings into two non-shared
spaces, the $\mathrm{[CLS]}$ feature exhibits better representational power
and thus can better encode the global information (see Figrue 2(c,d,e)).

|  |  |
---|---|---
(a) Shared Normalization (ShareNorm) |  | (b) Separate Normalization (SepNorm)
|  |
(c) Pretraining loss |  (d) $\mathrm{[CLS]}$ embeddings |  (e) Token embeddings
Figure 2: The effect of SepNorm on feature distributions and losses. (a) & (b)
The $\mathrm{[CLS]}$ (blue) and token (red) feature distributions of the first
50 dimensions. The $\mathrm{[CLS]}$ features of ShareNorm are more
concentrated around the mean and the mean deviates more from the zero centers,
while in SepNorm, the $\mathrm{[CLS]}$ distribution is more centered and
flattened. (c) Reconstruction loss of the MAE pertaining – MAE with SepNorm
achieves lower MSE loss compared to ShareNorm, demonstrating a better ability
to encode global contextual information. (d) & (e) Comparison of the singular
values of learned ($\mathrm{[CLS]}$ and token) features with ShareNorm and
different configurations of SepNorm. We perform Singular Value Decomposition
on embeddings from different Normalizations. The singular values of the
embeddings learned from SepNorm decay more slowly than those of ShareNorm,
indicating a better representational power.

###  3.2 On the Uniformity of the $\mathrm{[CLS]}$ Embeddings

We further inspect the uniformity of the $\mathrm{[CLS]}$ and token
embeddings. A good uniformity of the feature distribution indicates the
embedding preserves maximal information. Contrastive methods usually employ
negative instances to encourage uniformity. In this section, we investigate
how, under both non-contrastive and contrastive training methods, ShareNorm
and SepNorm can preserve the uniformity of the embeddings.

To this end, we choose MAE and U-MAE [Zhang et al., 2022] as the major
frameworks to perform the case study, where the former is a non-contrastive
method and the latter is a contrastive method. We modify their encoders to
replace the typical normalization layer with our proposed SepNorm layer. The
specification of our SepNorm method involves the incorporation of BN and LN.
We try all four combinations and measure their impact on the uniformity of the
$\mathrm{[CLS]}$ and token embeddings.

#### SepNorm better preserves uniformity in MAE.

Table 3(a) shows the numerical value of the uniformity on the STL10 and
Aircraft datasets [Coates et al., 2011, Maji et al., 2013]. In essence, our
results indicate that compared to ShareNorm, SepNorm significantly enhances
$\mathrm{[CLS]}$ uniformity. Interestingly, the uniformity of token embeddings
remains comparable. We also empirically verify that better uniformity on the
$\mathrm{[CLS]}$ embeddings results in better performance on the downstream
task (Figure 3(b)). Another observation made is that the uniformity of the
$\mathrm{[CLS]}$ is markedly improved when utilizing BN as compared to using
LN. A potential reason is that BN provides an implicit negative term like in
contrastive methods. Though we empirically demonstrate the use of BN on
$\mathrm{[CLS]}$ can better encourage uniformity, it is not necessary when
self-supervised transformers are combined with contrastive loss [Oord et al.,
2018].

#### ShareNorm imposes negative effects on uniformity in U-MAE.

We further pretrain U-MAE with ShareNorm being the normalization layer. The
U-MAE explicitly adds a uniformity loss on the $\mathrm{[CLS]}$ embeddings,
with coefficient $\lambda$ balancing the two objectives:

$\displaystyle\mathcal{L}_{\mathrm{U\mbox{-}MAE}}=\mathcal{L}_{\mathrm{MAE}}+\lambda\mathcal{L}_{\mathcal{U}}.$
(8)

Here $\mathcal{L}_{\mathcal{U}}$ is computed within one batch, and we only
compute the uniformity of the $\mathrm{[CLS]}$ embeddings. Based on the
analysis of the pretrained U-MAE, we have the following finding:

|  | STL10 | Aircraft
---|---|---
NormLayer | $\mathrm{[CLS]}$ | token | $\mathrm{[CLS]}$ | token
LN | -2.5911 | -3.4998 | -0.9097 | -3.3514
BN | -2.9531 | -3.6785 | -1.5321 | -3.3659
LN + LN | -1.1045 | -3.3685 | -1.6874 | -3.3633
LN + BN | -1.3153 | -3.3735 | -1.9634 | -3.5056
BN + LN | -3.4426 | -3.3797 | -2.7705 | -3.3068
BN + BN | -3.4763 | -3.4758 | -2.9788 | -3.7349
|
---
(a) Uniformity of different normalizations. | (b) Uniformity vs. accuracy
Figure 3: Uniformity Analysis. (a) Under SepNorm, the uniformity of the
$\mathrm{[CLS]}$ embeddings are better preserved on the STL10 and Aircraft
datasets. (b) Uniformity is positively related to the downstream task
performance – lower uniformity results in higher classification accuracy on
the STL10 dataset.

The uniformity of the $\mathrm{[CLS]}$ embeddings is critically held down by
ShareNorm even with an explicit contrastive loss.

Figure 4: $\lambda$ vs. uniformity. A higher $\lambda$ indicates a stronger
emphasis on the uniformity of $\mathrm{[CLS]}$ embeddings.

This is because, after LN, all features will distribute in the sphere of a
radius $\sqrt{d}$, assuming the coefficients $\gamma=\mathbf{1}$ and
$\beta=\mathbf{0}$. To distinguish the $\mathrm{[CLS]}$ embeddings from the
token embeddings, the uniformity metric on the $\mathrm{[CLS]}$ embeddings is
minimized within a subarea of the surface.

We empirically verify this phenomenon using the STL10 dataset. Figure 4 shows
the uniformity metrics obtained using different $\lambda$’s. As it is
indicated, when using ShareNorm, the uniformity of the $\mathrm{[CLS]}$
embeddings is no better than -3.088, even with explicit uniformity loss. On
the contrary, our proposed SepNorm can easily achieve a better uniformity
score. The contrastive approach further strengthens the advantage of using the
SepNorm as it allows a better uniformity of the embeddings.

##  4 Related Works

#### Self-supervised transformers.

The field of self-supervised learning in Transformer architectures has been a
vibrant area of research, with significant advancements made in both
contrastive and non-contrastive learning methods. These methods have been
instrumental in improving the stability and performance of Transformers,
contributing to their success in various applications. In this section, we
review some of the key works in this area, focusing on their methodologies,
findings, and the impact they have had on the field.

In the area of self-supervised learning, non-contrastive methods have emerged
as a powerful approach for training Transformer models. These methods, which
do not rely on negative samples for learning, have demonstrated remarkable
success in various tasks. BERT [Devlin et al., 2018] and RoBERTa [Liu et al.,
2019] were proposed in NLP domain. Additionally, There are some works focus on
the specific task, such as speech recognition [Wang et al., 2020], image
generation [Chen et al., 2020a], and heterogeneous graph generation [Hu et
al., 2020b].

Contrastive methods, a cornerstone in self-supervised learning, have been
instrumental in advancing Transformer architectures. These methods, which
learn from positive and negative samples, have been explored in various
contexts. For instance, Gao et al. [2021] and Zhang et al. [2022] have made
significant strides in natural language processing tasks, while [Chen et al.,
2021] have provided valuable insights into the pre-training of Transformers.
Meanwhile, the potential of contrastive methods in vision Transformers has
been demonstrated by works such as those by Caron et al. [2021] and Radford et
al. [2021]. These collective efforts underscore the versatility and efficacy
of contrastive methods in self-supervised Transformers.

#### Normalization layers in transformers.

Normalization layers, including layer normalization and batch normalization,
are integral to Transformer architectures, ensuring stable training and
accelerated convergence. In their study, Xiong et al. [2020] delve into the
role of layer normalization in the Transformer architecture, providing
insights into its contribution to the stability and performance of
Transformers. Similarly, Xu et al. [2019b] explores the intricacies of layer
normalization, offering potential enhancements to its effectiveness. On the
other hand, Shen et al. [2020] introduces a novel approach to batch
normalization in Transformers, termed "Powernorm", which addresses the
limitations of traditional batch normalization. Lastly, Nguyen and Salazar
[2019] focus on improving the normalization process in the self-attention
mechanism of Transformers, proposing methods to optimize the normalization of
self-attention. These collective efforts underscore the critical role of
normalization layers in the optimal operation of Transformer models.

##  5 Experiments

We examine the effectiveness of the proposed SepNorm component in three
domains: CV, NLP, and graphs. Then, we investigate how the ShareNorm and
SepNorm affect the uniformity of the $\mathrm{[CLS]}$ embeddings through an
ablation study.

###  5.1 Computer Vision

#### Datasets.

We investigate the model performance on the STL10 dataset [Coates et al.,
2011], FGVC Aircraft [Maji et al., 2013], the Street View House Numbers (SVHN)
dataset [Netzer et al., 2011], and Oxford 102 Flowers [Nilsback and Zisserman,
2008]. All datasets are used for classification tasks. We follow the
train/test split provided in the papers introducing the datasets. We report
top-1 and top-5 accuracy for all datasets.

#### Vision transformers (ViT) and MAE.

We choose Vision Transformer (ViT) [Dosovitskiy et al., 2020] as our feature
extractor for all datasets. To pretrain the ViT, we adopt the MAE training
scheme [He et al., 2022]. We follow MAE and use a 75% masking ratio on input
image. During the downstream tasks, we use the embeddings of the
$\mathrm{[CLS]}$ symbol to predict the class labels.

#### Experiment setup.

We follow the setup in He et al. [2022] to pretrain and evaluate the ViT. For
pertaining, we train the ViT for 4000 epochs. For linear probing, we freeze
the encoder’s weight and train the last layer on the specific datasets for
2000 epochs. We use a batch size of 512 for pretraining and a batch size of
128 for linear probing.

#### Experiment results.

The results presented in Table 1 demonstrate the performance of our model
compared to the baseline model, MAE, on image classification tasks. Our model
consistently outperforms the baseline across multiple datasets, indicating its
effectiveness in image recognition.

| STL10 | Aircraft | SVHN | Flower
---|---|---|---|---
| ACC@1 | ACC@5 | ACC@1 | ACC@5 | ACC@1 | ACC@5 | ACC@1 | ACC@5
MAE | 92.01 | 99.5 | 52.54 | 84.16 | 88.97 | 99.13 | 27.63 | 53.73
\+ SepNorm | 93.84 | 99.7 | 59.02 | 86.65 | 89.18 | 99.21 | 32.51 | 60.92
Table 1: Comparison of linear probing performance of ShareNorm and SepNorm
across 4 image classification datasets, for
$\text{ViT}_{\text{{\color[rgb]{0.3,0.3,0.3}base}}}$ pretrained with MAE.

In the STL-10 dataset, our approach utilizing the dual normalization layer
achieves improved performance compared to the baseline. Our model achieves a
top-1 accuracy of 93.84% and a top-5 accuracy of 99.7%, surpassing the
baseline’s accuracy of 92.01% and 99.5%, respectively. Similar improvements
are observed in the Aircraft, SVHN, and Flower datasets, where our model
consistently outperforms the baseline in both top-1 and top-5 accuracy
metrics. These results demonstrate the effectiveness of our dual normalization
layer in enhancing image classification performance, enabling better feature
representation and class discrimination.

###  5.2 Natural Language Processing

#### Datasets.

We extensively evaluated our approach using the STS dataset, which comprises
seven semantic textual similarity (STS) tasks. These tasks, including STS
2012-2016 [Agirre et al., 2012, 2013, 2014, 2015, 2016], STS Benchmark [Cer et
al., 2017], and SICK-Relatedness [Marelli et al., 2014]. We also evaluate our
method on multiple transfer tasks, including MR [Pang and Lee, 2005], CR [Hu
and Liu, 2004], SUBJ [Pang and Lee, 2004], MPQA [Wiebe et al., 2005], SST-2
[Socher et al., 2013], TREC Voorhees and Tice [2000], and MRPC [Dolan and
Brockett, 2005]. Following the evaluation settings of SimCSE [Gao et al.,
2021], we excluded an additional regressor and used Spearman’s correlation
coefficient as the evaluation metric.

#### BERT and RoBERTa.

We utilize pretrained checkpoints of BERT (uncased) [Devlin et al., 2018] and
RoBERTa (cased) [Liu et al., 2019] models. The use of these initialized
checkpoints is commonly used in related works [Gao et al., 2021], ensuring
comparability and facilitating the adoption of our approach in the broader
research community. Further, this initialization strategy saves significant
training time and computational resources by leveraging the pre-trained
representations, allowing us to focus on fine-tuning the models for specific
task requirements.

#### Experiment setup.

We follow the experiment setup in Gao et al. [2021] to further finetune the
BERT and RoBERTa models to contrast the $\mathrm{[CLS]}$ embeddings on a
subset of sentences from English Wikipedia. We evaluate the models using
established STS tasks and employ standard evaluation metrics such as
Spearman’s correlation.

|  | STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R | Avg.
---|---|---|---|---|---|---|---|---|---
| Unsupervised Training
$\text{BERT}_{\text{{\color[rgb]{0.3,0.3,0.3}base}}}$ | ShareNorm | 65.28 | 78.82 | 69.65 | 79.02 | 77.21 | 76.4 | 71.74 | 74.04
| SepNorm | 67.01 | 82.16 | 72.48 | 81.38 | 79.11 | 77.56 | 71.36 | 75.87
$\text{RoBERTa}_{\text{{\color[rgb]{0.3,0.3,0.3}base}}}$ | ShareNorm | 68.25 | 81.24 | 72.78 | 81.38 | 80.31 | 79.83 | 68.16 | 76.00
| SepNorm | 66.63 | 82.40 | 74.47 | 82.39 | 80.44 | 81.14 | 69.44 | 76.70
| Supervised Training
$\text{BERT}_{\text{{\color[rgb]{0.3,0.3,0.3}base}}}$ | ShareNorm | 77.72 | 81.07 | 78.97 | 85.15 | 82.00 | 82.36 | 79.74 | 81.00
| SepNorm | 75.32 | 84.41 | 79.94 | 84.91 | 80.87 | 83.63 | 79.61 | 81.23
$\text{RoBERTa}_{\text{{\color[rgb]{0.3,0.3,0.3}base}}}$ | ShareNorm | 77.38 | 80.87 | 78.72 | 84.02 | 82.56 | 83.08 | 78.25 | 80.70
| SepNorm | 75.80 | 84.94 | 80.33 | 85.51 | 82.11 | 84.88 | 79.72 | 81.90
|  | MR | CR | SUBJ | MPQA | SST2 | TREC | MRPC | Avg.
| Transfer Learning
$\text{BERT}_{\text{{\color[rgb]{0.3,0.3,0.3}base}}}$ | ShareNorm | 82.78 | 88.79 | 94.69 | 89.86 | 87.94 | 84.44 | 75.99 | 86.36
| SepNorm | 82.82 | 89.08 | 94.30 | 89.70 | 87.97 | 83.88 | 75.21 | 86.14
$\text{RoBERTa}_{\text{{\color[rgb]{0.3,0.3,0.3}base}}}$ | ShareNorm | 84.45 | 91.50 | 93.94 | 89.45 | 90.96 | 86.80 | 76.13 | 87.61
| SepNorm | 85.11 | 91.56 | 94.30 | 89.43 | 91.66 | 90.96 | 75.58 | 88.37
Table 2: Sentence embedding performance on STS tasks and transfer tasks. We
compare the performance of ShareNorm and SepNorm for
$\text{BERT}_{\text{{\color[rgb]{0.3,0.3,0.3}base}}}$ and
$\text{RoBERTa}_{\text{{\color[rgb]{0.3,0.3,0.3}base}}}$ model, respectively.

#### Experiment results.

The experimental results presented in Table 2 highlight the performance of our
model compared to the SimCSE baseline on various NLP tasks. With the SepNorm
layer, $\text{BERT}_{\text{{\color[rgb]{0.3,0.3,0.3}base}}}$ and
$\text{RoBERTa}_{\text{{\color[rgb]{0.3,0.3,0.3}base}}}$ achieve overall
higher average accuracy compared to ShareNorm’s average accuracy. Only in the
transfer learning tasks, SepNorm works slightly worse than ShareNorm in
$\text{BERT}_{\text{{\color[rgb]{0.3,0.3,0.3}base}}}$, but the difference is
marginal.

###  5.3 Molecule Discovery

#### Datasets.

We conducted experiments using the ZINC dataset [Irwin and Shoichet, 2005],
which contains approximately 250,000 molecular graphs for property prediction
tasks. Additionally, we used a subset of the ZINC dataset, consisting of
12,000 molecular graphs, as recommended by the benchmarking methodology
outlined in Benchmarking Graph Neural Networks [Dwivedi et al., 2020]. This
subset allowed us to ensure comparability with other studies. Despite being
smaller, the subset retained sufficient diversity and complexity for effective
evaluation. Furthermore, we utilized the MolHIV dataset from the OGB [Hu et
al., 2020a] collection, which is widely used for training and evaluating
graph-based models in molecular property prediction tasks.

#### Graphormer.

We utilize Graphormer [Ying et al., 2021] as the transformer backbone for
molecular property prediction tasks. To adapt the transformer architecture to
graph-structured data, it incorporates centrality, spatial, and edge
encodings, along with a special node called [VNode]. Centrality encoding
captures node importance, spatial encoding considers spatial relations and
edge encoding incorporates edge features. [VNode] facilitates graph-level
information aggregation. The special node [VNode] serves the same purpose as
the $\mathrm{[CLS]}$ token in traditional Transformer models. [VNode] is
prepended to the sequence of node embeddings and acts as a global summary of
the graph, capturing essential information for downstream classification
tasks. The state of [VNode] at the output of the Graphormer encoder represents
the overall graph representation for downstream tasks.

#### Experiment setup.

We strictly follow Graphormer [Ying et al., 2021] in terms of the model
architecture, hyperparameters, and training strategies. We replaced the
ShareNorm in Graphormer with SepNorm to investigate the effectiveness of the
proposed component. We evaluate the pretrained model on a broad class of
graph-level prediction tasks. We report the mean absolute error for the ZINC
and ZINC (subset) datasets and the area under the curve (AUC) for the MolHIV
dataset.

Dataset | ZINC | ZINC (subset) | MolHIV
---|---|---|---
Metrics | Mean absolute error$\downarrow$ | AUC$\uparrow$
Graphormer | 0.069 | 0.164 | 73.36%
\+ SepNorm | 0.052 | 0.144 | 75.64%
Table 3: Comparison of graph property prediction tasks of ShareNorm and
SepNorm across 3 datasets.

#### Experiment results.

Table 3 highlights the performance of our model compared to the Graphormer
baseline with ShareNorm on graph-level prediction tasks. For the ZINC
datasets, SepNorm achieves a significantly lower mean absolute error compared
to ShareNorm. In terms of the MolHIV dataset, SepNorm also achieves a higher
AUC of 75.64%, surpassing ShareNorm’s AUC of 73.36%. This suggests that, with
SepNorm, the embeddings of the $\mathrm{[CLS]}$ symbol can better summarize
the properties of the entire graph, thus achieving superior performance on
downstream tasks.

###  5.4 Ablation Study

We further investigate how SepNorm benefits learned embeddings in contrastive
methods. In this experiment, we first show that SepNorm encourages better
uniformity for both the $\mathrm{[CLS]}$ and token embeddings. We then show
that the uniformity of the token embeddings also benefits the downstream
tasks, even though they only utilize the $\mathrm{[CLS]}$ embeddings.

#### Experiment setup.

We pretrain MAE on the STL10 dataset via four different losses:

  * •

MAE loss $\mathcal{L}_{\mathrm{MAE}}$ without any $\mathcal{L}_{\mathcal{U}}$
on $\mathrm{[CLS]}$ and token embeddings.

  * •

MAE loss $\mathcal{L}_{\mathrm{MAE}}$ with $\mathcal{L}_{\mathcal{U}}$ on the
$\mathrm{[CLS]}$ embeddings. We treat all $\mathrm{[CLS]}$ embeddings within
the same batch (except itself) as negative instances.

  * •

MAE loss $\mathcal{L}_{\mathrm{MAE}}$ with $\mathcal{L}_{\mathcal{U}}$ on the
token embeddings. We treat all token embeddings within the same batch or same
images (except itself) as negative instances.

  * •

MAE loss $\mathcal{L}_{\mathrm{MAE}}$ with $\mathcal{L}_{\mathcal{U}}$ on both
$\mathrm{[CLS]}$ and token embeddings.

We choose $\lambda=\\{0,0.01,0.1,1\\}$. Note that the second loss with
$\lambda=0.1$ corresponds to the U-MAE [Zhang et al., 2022]. We also replace
the normalization layer of the ViT in MAE with one of the following: {LN, BN,
BN+LN, BN+BN}. The combination of different losses, different $\lambda$’s, and
different normalization layers yields 40 specifications of the experiments.

Table 4 reports the downstream performance (accuracy) on STL10 across 40
different settings. We summarize our observations: (1) In the non-contrastive
method MAE, with proper configuration, the performance of SepNorm is superior
to as of the ShareNorm. (2) In contrastive methods ($\lambda\neq 0$),
SepNorms’ advantages are further highlighted. For example, when $\lambda=1$,
the performance of SepNorm (BN+LN) is improved by 1.6% compared to the non-
contrastive method. The performance gain in SepNorm (BN+BN) is less obvious as
the double BNs already impose implicit uniformity loss on both
$\mathrm{[CLS]}$ and token embeddings. In contrast to SepNorm, the performance
of ShareNorm is less satisfactory when using contrastive methods. We believe
it is very challenging to encourage the two types of embeddings to be
uniformly distributed in the same sphere and keep them separable at the same
time. (3) The uniformity of the token embeddings is also vital for learning a
good $\mathrm{[CLS]}$ representation, as evident by SepNorm (BN+LN) gaining
accuracy with increasing $\lambda$ on the token embeddings. We hypothesize
that by enforcing uniformity, the token embeddings are forced to contain less
information about others, which encourages the $\mathrm{[CLS]}$ embedding to
encode as much information as possible.

Normalization layer | $\lambda=0$ | Negative pairs | $\lambda=0.01$ | $\lambda=0.1$ | $\lambda=1$ | Best
---|---|---|---|---|---|---
SepNorm (BN+BN) | 93.84 | token | 93.65 | 94.15 | 93.94 | 94.15
$\mathrm{[CLS]}$ | 93.73 | 93.85 | 93.93 | 93.93
$\mathrm{[CLS]}$ \+ token | 93.40 | 94.25 | 94.28 | 94.28
SepNorm (BN+LN) | 92.80 | token | 92.98 | 93.60 | 94.30 | 94.30
$\mathrm{[CLS]}$ | 92.98 | 93.48 | 93.36 | 93.48
$\mathrm{[CLS]}$ \+ token | 92.74 | 93.18 | 94.40 | 94.40
ShareNorm (BN) | 92.84 | token | 92.48 | 93.38 | 92.78 | 93.38
$\mathrm{[CLS]}$ | 93.10 | 93.33 | 92.93 | 93.33
$\mathrm{[CLS]}$ \+ token | 93.41 | 93.46 | 92.99 | 93.46
ShareNorm (LN) | 92.01 | token | 92.61 | 92.74 | 92.14 | 92.74
$\mathrm{[CLS]}$ | 92.28 | 92.75 | 92.36 | 92.75
$\mathrm{[CLS]}$ \+ token | 92.74 | 92.38 | 92.74 | 92.74
Table 4: Ablation study of the effect of $\mathrm{[CLS]}$ and token uniformity
on the downstream tasks, where we varied $\lambda$. We report downstream task
accuracy for the STL10 dataset.

##  6 Conclusion

In this work, we introduced SepNorm, an approach that employs separate
normalization layers for the $\mathrm{[CLS]}$ symbol and the tokens, to
replace the conventional single normalization layer (ShareNorm) approach in
transformers. Applied across various domains, SepNorm consistently outperforms
models equipped with ShareNorm, even under identical parameters and
configurations. Our experiments highlight the importance of applying separate
normalization to the $\mathrm{[CLS]}$ symbol and tokens. Furthermore, we
discovered that SepNorm enhances the transformers’ ability to encode inputs.
This finding stresses the potential of SepNorm as a valuable tool for
improving transformer performance.

## References

  * Agirre et al. [2012] E. Agirre, D. Cer, M. Diab, and A. Gonzalez-Agirre.  Semeval-2012 task 6: A pilot on semantic textual similarity.  In _* SEM 2012: The First Joint Conference on Lexical and Computational Semantics–Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_ , pages 385–393, 2012.
  * Agirre et al. [2013] E. Agirre, D. Cer, M. Diab, A. Gonzalez-Agirre, and W. Guo.  * sem 2013 shared task: Semantic textual similarity.  In _Second joint conference on lexical and computational semantics (* SEM), volume 1: proceedings of the Main conference and the shared task: semantic textual similarity_ , pages 32–43, 2013.
  * Agirre et al. [2014] E. Agirre, C. Banea, C. Cardie, D. Cer, M. Diab, A. Gonzalez-Agirre, W. Guo, R. Mihalcea, G. Rigau, and J. Wiebe.  Semeval-2014 task 10: Multilingual semantic textual similarity.  In _Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014)_ , pages 81–91, 2014.
  * Agirre et al. [2015] E. Agirre, C. Banea, C. Cardie, D. Cer, M. Diab, A. Gonzalez-Agirre, W. Guo, I. Lopez-Gazpio, M. Maritxalar, R. Mihalcea, et al.  Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability.  In _Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015)_ , pages 252–263, 2015.
  * Agirre et al. [2016] E. Agirre, C. Banea, D. Cer, M. Diab, A. Gonzalez Agirre, R. Mihalcea, G. Rigau Claramunt, and J. Wiebe.  Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation.  In _SemEval-2016. 10th International Workshop on Semantic Evaluation; 2016 Jun 16-17; San Diego, CA. Stroudsburg (PA): ACL; 2016. p. 497-511._ ACL (Association for Computational Linguistics), 2016.
  * Ba et al. [2016] J. L. Ba, J. R. Kiros, and G. E. Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Caron et al. [2021] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9650–9660, 2021.
  * Cer et al. [2017] D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia.  Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation.  _arXiv preprint arXiv:1708.00055_ , 2017.
  * Chen et al. [2020a] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever.  Generative pretraining from pixels.  In _International conference on machine learning_ , pages 1691–1703. PMLR, 2020a.
  * Chen et al. [2020b] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020b.
  * Chen et al. [2021] X. Chen, S. Xie, and K. He.  An empirical study of training self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9640–9649, 2021.
  * Coates et al. [2011] A. Coates, A. Ng, and H. Lee.  An analysis of single-layer networks in unsupervised feature learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pages 215–223. JMLR Workshop and Conference Proceedings, 2011.
  * Devlin et al. [2018] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dolan and Brockett [2005] B. Dolan and C. Brockett.  Automatically constructing a corpus of sentential paraphrases.  In _Third International Workshop on Paraphrasing (IWP2005)_ , 2005\.
  * Dosovitskiy et al. [2020] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Dwivedi et al. [2020] V. P. Dwivedi, C. K. Joshi, T. Laurent, Y. Bengio, and X. Bresson.  Benchmarking graph neural networks.  2020\.
  * Gao et al. [2021] T. Gao, X. Yao, and D. Chen.  Simcse: Simple contrastive learning of sentence embeddings.  _arXiv preprint arXiv:2104.08821_ , 2021.
  * Gu et al. [2020] Y. Gu, Z. Ding, S. Wang, L. Zou, Y. Liu, and D. Yin.  Deep multifaceted transformers for multi-objective ranking in large-scale e-commerce recommender systems.  In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, pages 2493–2500, 2020.
  * Han et al. [2021] K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y. Wang.  Transformer in transformer.  _Advances in Neural Information Processing Systems_ , 34:15908–15919, 2021.
  * Han et al. [2022] K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu, et al.  A survey on vision transformer.  _IEEE transactions on pattern analysis and machine intelligence_ , 45(1):87–110, 2022.
  * He et al. [2022] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16000–16009, 2022.
  * Hu and Liu [2004] M. Hu and B. Liu.  Mining and summarizing customer reviews.  In _Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 168–177, 2004.
  * Hu et al. [2020a] W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec.  Open graph benchmark: Datasets for machine learning on graphs.  _Advances in neural information processing systems_ , 33:22118–22133, 2020a.
  * Hu et al. [2020b] Z. Hu, Y. Dong, K. Wang, and Y. Sun.  Heterogeneous graph transformer.  In _Proceedings of the web conference 2020_ , pages 2704–2710, 2020b.
  * Huang et al. [2020] X. S. Huang, F. Perez, J. Ba, and M. Volkovs.  Improving transformer optimization through better initialization.  In _International Conference on Machine Learning_ , pages 4475–4483. PMLR, 2020.
  * Ioffe and Szegedy [2015] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pages 448–456. pmlr, 2015.
  * Irwin and Shoichet [2005] J. J. Irwin and B. K. Shoichet.  Zinc- a free database of commercially available compounds for virtual screening.  _Journal of chemical information and modeling_ , 45(1):177–182, 2005.
  * Jing et al. [2021] L. Jing, P. Vincent, Y. LeCun, and Y. Tian.  Understanding dimensional collapse in contrastive self-supervised learning.  _arXiv preprint arXiv:2110.09348_ , 2021.
  * Karmakar et al. [2021] P. Karmakar, S. W. Teng, and G. Lu.  Thank you for attention: a survey on attention-based artificial neural networks for automatic speech recognition.  _arXiv preprint arXiv:2102.07259_ , 2021.
  * Liu et al. [2019] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Maji et al. [2013] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi.  Fine-grained visual classification of aircraft.  _arXiv preprint arXiv:1306.5151_ , 2013.
  * Marelli et al. [2014] M. Marelli, S. Menini, M. Baroni, L. Bentivogli, R. Bernardi, R. Zamparelli, et al.  A sick cure for the evaluation of compositional distributional semantic models.  In _Lrec_ , pages 216–223. Reykjavik, 2014.
  * Medhat et al. [2014] W. Medhat, A. Hassan, and H. Korashy.  Sentiment analysis algorithms and applications: A survey.  _Ain Shams engineering journal_ , 5(4):1093–1113, 2014.
  * Netzer et al. [2011] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * Nguyen and Salazar [2019] T. Q. Nguyen and J. Salazar.  Transformers without tears: Improving the normalization of self-attention.  _arXiv preprint arXiv:1910.05895_ , 2019.
  * Nilsback and Zisserman [2008] M.-E. Nilsback and A. Zisserman.  Automated flower classification over a large number of classes.  In _2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing_, pages 722–729. IEEE, 2008.
  * Oord et al. [2018] A. v. d. Oord, Y. Li, and O. Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Pang and Lee [2004] B. Pang and L. Lee.  A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts.  _arXiv preprint cs/0409058_ , 2004.
  * Pang and Lee [2005] B. Pang and L. Lee.  Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.  _arXiv preprint cs/0506075_ , 2005.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Shen et al. [2020] S. Shen, Z. Yao, A. Gholami, M. Mahoney, and K. Keutzer.  Powernorm: Rethinking batch normalization in transformers.  In _International Conference on Machine Learning_ , pages 8741–8751. PMLR, 2020.
  * Socher et al. [2013] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1631–1642, 2013.
  * Steiner et al. [2021] A. Steiner, A. Kolesnikov, X. Zhai, R. Wightman, J. Uszkoreit, and L. Beyer.  How to train your vit? data, augmentation, and regularization in vision transformers.  _arXiv preprint arXiv:2106.10270_ , 2021.
  * Sun et al. [2019] F. Sun, J. Liu, J. Wu, C. Pei, X. Lin, W. Ou, and P. Jiang.  Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer.  In _Proceedings of the 28th ACM international conference on information and knowledge management_ , pages 1441–1450, 2019.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Voorhees and Tice [2000] E. M. Voorhees and D. M. Tice.  Building a question answering test collection.  In _Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 200–207, 2000.
  * Wang and Isola [2020] T. Wang and P. Isola.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  In _International Conference on Machine Learning_ , pages 9929–9939. PMLR, 2020.
  * Wang et al. [2020] Y. Wang, A. Mohamed, D. Le, C. Liu, A. Xiao, J. Mahadeokar, H. Huang, A. Tjandra, X. Zhang, F. Zhang, et al.  Transformer-based acoustic modeling for hybrid speech recognition.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6874–6878. IEEE, 2020.
  * Wiebe et al. [2005] J. Wiebe, T. Wilson, and C. Cardie.  Annotating expressions of opinions and emotions in language.  _Language resources and evaluation_ , 39:165–210, 2005\.
  * Wu et al. [2020] L. Wu, S. Li, C.-J. Hsieh, and J. Sharpnack.  Sse-pt: Sequential recommendation via personalized transformer.  In _Proceedings of the 14th ACM Conference on Recommender Systems_ , pages 328–337, 2020.
  * Xiong et al. [2020] R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu.  On layer normalization in the transformer architecture.  In _International Conference on Machine Learning_ , pages 10524–10533. PMLR, 2020.
  * Xu et al. [2019a] H. Xu, Q. Liu, J. van Genabith, D. Xiong, and J. Zhang.  Lipschitz constrained parameter initialization for deep transformers.  _arXiv preprint arXiv:1911.03179_ , 2019a.
  * Xu et al. [2019b] J. Xu, X. Sun, Z. Zhang, G. Zhao, and J. Lin.  Understanding and improving layer normalization.  _Advances in Neural Information Processing Systems_ , 32, 2019b.
  * Ying et al. [2021] C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu.  Do transformers really perform badly for graph representation?  _Advances in Neural Information Processing Systems_ , 34:28877–28888, 2021.
  * Zhang et al. [2022] Q. Zhang, Y. Wang, and Y. Wang.  How mask matters: Towards theoretical understandings of masked autoencoders.  _arXiv preprint arXiv:2210.08344_ , 2022.
  * Zhou et al. [2020] W. Zhou, T. Ge, K. Xu, F. Wei, and M. Zhou.  Scheduled drophead: A regularization method for transformer models.  _arXiv preprint arXiv:2004.13342_ , 2020.