# Bi-ViT: Pushing the Limit of Vision Transformer Quantization

Yanjing Li1, Sheng Xu1, Mingbao Lin2, Xianbin Cao1, Chuanjian Liu4, Xiao Sun5,
Baochang Zhang1,3
1 Beihang University 2 Tencent 3 Zhongguancun Laboratory
4 Huawei Noah’s Ark Lab 5 Shanghai AI Laboratory
<EMAIL_ADDRESS>
###### Abstract

Vision transformers (ViTs) quantization offers a promising prospect to
facilitate deploying large pre-trained networks on resource-limited devices.
Fully-binarized ViTs (Bi-ViT) that pushes the quantization of ViTs to its
limit remain largely unexplored and a very challenging task yet, due to their
unacceptable performance. Through extensive empirical analyses, we identify
the severe drop in ViT binarization is caused by attention distortion in self-
attention, which technically stems from the gradient vanishing and ranking
disorder. To address these issues, we first introduce a learnable scaling
factor to reactivate the vanished gradients and illustrate its effectiveness
through theoretical and experimental analyses. We then propose a ranking-aware
distillation method to rectify the disordered ranking in a teacher-student
framework. Bi-ViT achieves significant improvements over popular DeiT and Swin
backbones in terms of Top-1 accuracy and FLOPs. For example, with DeiT-Tiny
and Swin-Tiny, our method significantly outperforms baselines by 22.1% and
21.4% respectively, while 61.5$\times$ and 56.1$\times$ theoretical
acceleration in terms of FLOPs compared with real-valued counterparts on
ImageNet.

##  1 Introduction

Transformers, which have gained far-flung fame in natural language processing
(NLP) area [8, 28], are also attracting increasing attention in lots of
computer vision (CV) tasks, such as object detection [4], image classification
[9] and many others [13, 31], impelling the widespread research on vision
transformers (ViTs). There has a natural fit for ViTs to achieve better
performance simply by training a larger model on a larger data set. For
example, historical records show better performance of a ViT-H model [9]
accompanying with astonishing 632M parameters and 162G FLOPs. Such a high
model complexity poses a great challenge to deploy models on platforms with
short resource supplies. Therefore, both academia and industry call for an
ultimate compression of these large models, and the past years have witnessed
some promising techniques such as network pruning [38, 5], low-rank
decomposition [7], knowledge distillation [12], and quantization [18, 19].

Figure 1: Performance of real-valued and quantized DeiT [32] with varying bit-
widths. We report results with (a) DeiT-Tiny and (b) DeiT-Small on ImageNet
[17], respectively. Here 8-bit DeiT is quantized with PTQ method [22] and
2/3/4 bit DeiT is trained with QAT method [18]. The binarized ViT is conducted
with the baseline method Bi-Real Net [26].

Network quantization, which represents weights and activations in a low-bit
format, has got great earnestness of many researchers for its reduced memory
access costs and increased compute efficiency as well as performance benefit.
Using the lower-bit quantized data, in particular to the extreme 1-bit case,
requires less data movement, both on-chip and off-chip, and therefore reduces
memory bandwidth and saves significant energy. Existing documentary records
observe 32$\times$ less network size and 58$\times$ speedups beneficial from
xnor and bit-count logics for 1-bit networks [30]. Earlier attempts [25, 22]
apply post-training quantization (PTQ) [1, 40] directly to ViTs without data-
driven fine-tuning, causing sub-optimal performance, in particular to impotent
1-bit ViTs. Therefore, by quantizing while training, quantization-aware
training (QAT) methods are more congenial to 1-bit ViTs. Extensive empirical
studies [24, 20, 36, 28] have well demonstrated the efficacy of QAT methods in
1-bit convolutional neural networks (CNNs) or BERTs, however, the application
to 1-bit ViTs remains not to be fully explored so far.

In this paper, we first build a fully-binarized ViT baseline, a
straightforward solution constructed upon popular binarized QAT method of Bi-
Real Net [26]. Through an empirical study of this baseline, we observe
significant performance drops on the ImageNet dataset [17], as shown in Fig.
1. For instance, extending Bi-Real Net to binarize DeiT-Tiny [32] incurs a
tremendous performance gap of 52.6% in the Top-1 accuracy compared to the
2-bit quantized counterpart. Similar performance drops occur in DeiT-Small as
well. Delving into a deeper analysis, we find that the incompatibility of
existing QAT methods mainly stems from the binarized self-attention module in
ViTs, where a simple application of existing binarization methods [26] leads
to severe attention distortion, as plotted in Fig. 2 (a) and Fig. 2 (b),
especially in the diagonal scores of the map which are supposed to be the most
attentive.

In this paper we dig deeper into this attention distortion problem. Through
empirical analysis, we find that this phenomenon is mainly caused by gradient
vanishing due to the straight-through-estimator (STE) [2] and non-scaled
binarization in self-attention. Meanwhile, a simple distillation utilizing
distillation token in DeiT [32] and KL-divergence in ReActNet [24] is
ineffective in dismissing the ranking disorder, since it neglects the relative
order of the attention map between the binarized ViTs and their real-valued
counterpart. To address the aforementioned issues, a fully-binarized ViT (Bi-
ViT) is developed by reactivating the vanished gradients through a learnable
scaling factor in self-attention and a ranking-aware distillation to further
effectively rectify the disordered ranking of attention (see the overview in
Fig. 3). In addition, we also provide both empirical and theoretical analysis
about how our method can rectify the distorted attention and thus promote the
optimization of Bi-ViT. The contributions of our work are summarized as:

  * •

We identify the bottleneck of a fully-binarized ViT through empirical analyses
and formulate the problem in a theoretical perspective. Based on these, we
introduce learnable head-wise scaling factor into binarized self-attention to
reactivate the vanished gradients.

  * •

We develop a ranking-aware distillation scheme to eliminate attention
distortion. Our distillation method fully utilizes the ranking-aware knowledge
from the real-valued teacher to promote the optimization of Bi-ViT.

  * •

Our Bi-ViT is the first promising way to push the limit of ViT quantization to
the fully-binarized version. Extensive experiments on the ImageNet benchmark
demonstrate that Bi-ViT surpasses both the baseline and prior binarized
methods by a significant margin, achieving a remarkable acceleration rate of
up to 61.5$\times$.

Figure 2: Visualization of the attention map before softmax in the first block
of DeiT-Tiny [32] on ImageNet [17]. From the left to right, is the baseline
method [26], previous binarization method [36], our Bi-ViT and real-valued
counterpart.

##  2 Related Work

Vision Transformer. Unlike traditional CNN-based models, ViTs are capable of
capturing long-range visual relationships through the self-attention
mechanism, and offer a more generalizable paradigm without inductive bias
specific to images. The starting ViT [9] views an image as a sequence of 16
$\times$ 16 patches and uses a unique class token to predict the
classification, yielding promising results. Subsequently, many works, such as
DeiT [32] and PVT [35], have improved upon ViT, making it more efficient and
applicable to downstream tasks. However, these high-performing ViTs have also
accompanied with a significant number of parameters and high computational
overhead, limiting their widespread applications. Thus, designing smaller and
faster ViTs has become a new trend. LeViT [11] makes progress in faster
inference through down-sampling, patch descriptors, and a redesign of the
Attention-MLP block. DynamicViT [29] proposes a dynamic token sparsification
framework to progressively and dynamically prune redundant tokens, achieving a
competitive complexity and accuracy trade-off. Evo-ViT [37] proposes a slow-
fast updating mechanism that ensures information flow and spatial structure,
reducing both the training and inference complexity. While the aforementioned
works focus on efficient model design, this paper aims to boost compression
and acceleration through binarization.

Figure 3: Overview of the proposed Bi-ViT framework. We introduce the
learnable scaling factor in an architecture perspective and a ranking-aware
distillation scheme incorporated in the optimization process. From left to
right, we respectively show the detailed architecture of single block in Bi-
ViT and the distillation framework of Bi-ViT.

Network Binarization. BinaryNet is a technique originally proposed to train
convolutional neural networks (CNNs) with binary weights. BinaryConnect [6] is
the precursor to BinaryNet, where the parameters are binary while the
activations remain in full-precision states. Local binary convolution layers
(LBC) [16] were introduced to binarize the non-linear activations, and XNOR-
Net [30] was introduced to improve convolution efficiency by binarizing the
weights and inputs of convolution kernels. Bi-Real Net [26] explores a new
variant of residual structure to preserve the information of real activations
before the sign function, with a tight approximation to the derivative of the
non-differentiable sign function. Real-to-binary [27] re-scales the feature
maps on the channels according to the input before binarized operations and
adds an SE-Net [15] like gating module. ReActNet [24] replaces the
conventional PReLU and the sign function of the BNNs with RPReLU and RSign
with a learnable threshold, thus improving the performance of BNNs. RBONN [36]
introduces a recurrent bilinear optimization to address the asynchronous
convergence problem for BNNs, which further improves the performance of BNNs.
These techniques improve the efficiency and accuracy of binary neural networks
(BNNs) and allow them to be applied in practical applications. Majorities of
these techniques consider non-scaled binarization in activations, which is
beneficial to conventional CNNs while causing gradient mismatch issue for the
pecularity of self-attention mechanism in ViTs.

##  3 Background

###  3.1 Multi-Head Self-Attention and Binarization

For a multi-head self-attention (MHSA) module, we denote its query, key, and
value set as $\\{{\bf a}_{\\{q,k,v\\}}\in\mathbb{R}^{h\times N\times d}\\}$,
where $h$ denotes head number, $N$ and $d$ represent the patch and channel
numbers of each head. Specifically,
$N=(W_{in}//W_{in}^{P})\times(H_{in}//H_{in}^{P})$ where $W_{in}$ and $H_{in}$
are the width and height of the feature, $W_{in}^{P}$, $H_{in}^{P}$ are the
width and height of patch maps respectively. Then, the attention score ${\bf
A}$ and MHSA module output ${\bf a}_{out}$ are computed as follows [34]:

$\displaystyle{\bf A}=\operatorname{softmax}[({\bf a}_{q}\cdot{\bf
a}_{k}^{\top})/\sqrt{d}],$ (1) $\displaystyle{\bf a}_{out}={\bf A}\cdot{\bf
a}_{v}^{\top},$

where $\operatorname{softmax}(\cdot)$ represents the softmax operation.
Intending to represent query, key, value and attention score, i.e., ${\bf
a}_{q}$, ${\bf a}_{k}$, ${\bf a}_{v}$ and ${\bf A}$, in a 1-bit format, Eq.
(1) changes into:

$\displaystyle{\bf A}=\operatorname{softmax}[({\bf b}_{a_{q}}\cdot{\bf
b}_{a_{k}}^{\top})/\sqrt{d}],$ (2) $\displaystyle{\bf a}_{out}={\bf
b}_{A}\cdot{\bf b}_{a_{v}}^{\top}.$

We follow the common network binarization methods [30] that use the sign
function ${\bf b}_{\cdot}=\operatorname{sign}(\cdot)$ in the binary forward
pass, and STE [2] $\frac{\partial{\bf
b}_{\cdot}}{\partial\cdot}=1_{|\cdot|\leq 1}$ to compute the gradient for sign
function in its backward pass. We omit the non-linear function here for
simplicity. For all the projection and linear layers in binarized ViTs, we
conduct binarization following [28, 26] as ${{\bf a}_{out}}={\bf
b}_{a_{in}}\cdot(\bm{\alpha}_{w}\circ{\bf
b}_{w})^{\top}=\bm{\alpha}_{w}\circ({\bf b}_{a_{in}}\cdot{\bf b}_{w}^{\top})$
where
$\bm{\alpha}_{w}=\\{{\alpha}_{w}^{1},{\alpha}_{w}^{2},...,{\alpha}_{w}^{C_{out}}\\}\in\mathbb{R}^{C_{{out}}}_{+}$
is known as the channel-wise scaling factor vector [30] and $\circ$ represents
channel-wise channel-wise multiplication. The matrix multiplication process,
i.e., ${\bf b}_{a_{in}}\cdot{\bf b}_{w}^{\top}$, can be executed by the
efficient XNOR and Bit-count instructions on edge devices.

Figure 4: Performance of fully-binarized DeiT-Tiny on ImageNet [17] with
different binarized/real-valued settings.

###  3.2 Bottleneck of Fully-Binarized ViTs

The high-performing ViTs are built on premise of transformer’s supreme ability
to model the long-range relationships thanks to the attention mechanism within
the MHSA module. Unfortunately, a binarized version of weights and inputs
significantly weakens the representation ability. In addition, the sign
function and clamp operation also damage the optimization of backpropagation.
To be more evident, we perform quantitative ablative experiments where we
replace weights or activations in each module of the real-valued DeiT-Tiny
[32] with a binarized one and report the resulting Top-1 accuracy drop on the
ImageNet dataset [17] after a total of 50 training epochs. Fig. 4 reports the
results and we go on a deeper analysis below.

Module Degradation. By gradually replacing the multilayer perceptron (MLP) and
MHSA modules with real-valued weights or activations, we have discovered that
maintaining the MLP as “w1a1” (all weights and activations in the MLP are
binarized) still results in satisfactory performance. For instance, keeping
MLP as “w1a1” while keeping MHSA as “w1a32” obtains 26.3% Top-1 accuracy,
which might be acceptable comparing to the 55.2% of real-valued DeiT-Tiny when
taking into consideration 47.3$\times$ acceleration rates. On the contrast,
when maintaining MHSA module as “w1a1”, we observe a significant drop in
performance. To be more specific, even when the MLP was maintained as
“w32a32”, we still observe a significant 50.8% decrease in Top-1 accuracy
(from 55.2% to 4.4%). This result indicates that using binarized weights and
activations in the MHSA module can have a substantial negative impact on the
model’s performance, even when other parts retain in real-valued states.

Operation Degradation. To better understand the impact of fully-binarized
ViT’s performance, we conduct further analyses by examining the operations
within the MHSA module. Specifically, when we maintain the self-attention
activations in Eq. (1) as real-valued (“a32”), we observe only a relatively
small decrease in performance from 48.8% to 37.6%. However, when the self-
attention activations in Eq. (2) are binarized, significant drops in accuracy
occur from 48.8% to 7.6%. This finding highlights the importance of the self-
attention process within the MHSA module and suggests more efforts to mitigate
the negative impact of binarization on the MHSA module.

###  3.3 Gradient Mismatch in Self-Attention

With conclusion from the experimental results in Sec. 3.2 that self-attention
process, i.e., Eq. (2), is the most critical part causing the performance
drops. We attempt to analyze the underlying reasons for this phenomenon from
an optimization perspective. For simplicity, we derive the gradient mismatch
in ${\bf a}_{q}$ as an example, and the analysis can be applicable to explain
${\bf a}_{k}$ as well. We first represent the features before
$\operatorname{softmax}(\cdot)$ in Eq. (2) as:

$\displaystyle{\bf p}$ $\displaystyle=({\bf b}_{a_{q}}\cdot{\bf
b}_{a_{k}}^{\top}.)/\sqrt{d}.$ (3)

The gradient of ${\bf a}_{q}^{h_{i},n,c}$ w.r.t. ${\bf A}$ is formulated as:

$\displaystyle\frac{\partial{\bf A}}{\partial{\bf
a}_{q}^{h_{i},n,c}}=\frac{\partial{\bf A}}{\partial{\bf
p}^{h_{i},n,n^{\prime}}}\cdot\frac{\partial{\bf
p}^{h_{i},n,n^{\prime}}}{\partial{\bf
b}_{{a}_{q}}^{h_{i},n,c}}\cdot\frac{\partial{\bf
b}_{{a}_{q}}^{h_{i},n,c}}{\partial{\bf a}_{q}^{h_{i},n,c}},$ (4)

where $h_{i}\in\mathbb{R}^{h}$, $n$ & $n^{\prime}\in\mathbb{R}^{N}$,
$c\in\mathbb{R}^{d}$ and the gradient of ${\bf a}_{k}$ is likewise. The
explicit form of the first item $\frac{\partial{\bf A}}{\partial{\bf
p}^{h_{i},n,n^{\prime}}}$ in Eq. (4) is:

$\displaystyle\frac{\partial{\bf A}}{\partial{\bf p}_{h_{i},n,n^{\prime}}}$
$\displaystyle=\frac{\partial\operatorname{softmax}({\bf
p}_{h_{i},n,n^{\prime}})}{\partial{\bf p}_{h_{i},n,n^{\prime}}}$ (5)
$\displaystyle={\bf A}_{h_{i},n,n^{\prime}}\otimes(1-{\bf
A}_{h_{i},n,n^{\prime}}),$

where $\otimes$ denotes Hadamard product. And the second item is formulated
as:

$\displaystyle\frac{\partial{\bf p}^{h_{i},n,n^{\prime}}}{\partial{\bf
b}_{{a}_{q}}^{h_{i},n,c}}$ $\displaystyle=\frac{\partial{\bf
b}_{{a}_{q}}^{h_{i},n,c}\cdot{\bf
b}_{{a}_{k}}^{\top{h_{i},c,n^{\prime}}}}{\partial{\bf
b}_{{a}_{q}}^{h_{i},n,c}}$ (6) $\displaystyle={\bf
b}_{{a}_{k}}^{\top{h_{i},c,n^{\prime}}},$

result of which is therefore correlated with ${\bf b}_{{a}_{k}}$. The third
item is solved through STE [2] as:

$\displaystyle\frac{\partial{\bf b}_{{a}_{q}}^{h_{i},n,c}}{\partial{\bf
a}_{q}^{h_{i},n,c}}={\bf 1}_{|{\bf a}_{q}^{h_{i},n,c}|\leq 1}.$ (7)

Combing Eq. (5)$-$Eq. (7), we have the final gradient form in fully-binarized
ViTs as:

$\displaystyle\frac{\partial{\bf A}}{\partial{\bf a}_{q}^{h_{i},n,c}}$
$\displaystyle=\frac{\partial{\bf A}}{\partial{\bf
p}^{h_{i},n,n^{\prime}}}\cdot\frac{\partial{\bf
p}^{h_{i},n,n^{\prime}}}{\partial{\bf
b}_{{a}_{q}}^{h_{i},n,c}}\cdot\frac{\partial{\bf
b}_{{a}_{q}}^{h_{i},n,c}}{\partial{\bf a}_{q}^{h_{i},n,c}}$ (8)
$\displaystyle={{\bf A}_{h_{i},n,n^{\prime}}(1-{\bf
A}_{h_{i},n,n^{\prime}})}\cdot{\bf b}_{{a}_{k}}^{h_{i},c,n^{\prime}}\cdot{\bf
1}_{|{\bf a}_{q}^{h_{i},n,c}|\leq 1}.$

Considering ${\bf b}_{{a}_{q}}^{h_{i},n,:}=[1,\cdots,1]$ and $\cdot{\bf
b}_{{a}_{k}}^{h_{i},n^{\prime},:}=[1,\cdots,1]$ as the extreme condition,
${\bf b}_{{a}_{q}}^{h_{i},n,:}\cdot{\bf
b}_{{a}_{k}}^{\top{h_{i},:,n^{\prime}}}=d$. Therefore, a specific element in
${\bf b}_{a_{q}}\cdot{\bf b}^{\top}_{a_{k}}$ is $\in\\{-d,\cdots,d\\}$. We
plot the curve of a specific element in the first item between $[-64,64]$ in
Fig. 5 (a) as $d=64$ in DeiT-Tiny [32]. We observe $\frac{\partial{\bf
A}}{\partial{\bf p}_{h_{i},n,n^{\prime}}}$ sharply magnified when ${\bf
p}_{h_{i},n,n^{\prime}}$ increases. As shown in Fig. 5 (b), when ${\bf
p}_{h_{i},n,n^{\prime}}$ has a large magnitude, $|{\bf a}_{q}|>1$ and
$\frac{\partial{\bf b}_{{a}_{q}}^{h_{I},n,c}}{\partial{\bf
a}_{q}^{h_{i},n,c}}=0$ . Thus the multiplication of these two items leads to
$\frac{\partial{\bf A}}{\partial{\bf a}_{q}^{h_{i},n,c}}=0$, likewise for
${\bf a}_{k}$. Therefore we formulate the gradient mismatch phenomenon in the
aforementioned theoretical analysis. And such gradient mismatch leads to
distorted gradient in the optimization of ${\bf a}_{q}$ & ${\bf a}_{k}$ and
therefore degrades performance of fully-binarized ViTs.

Figure 5: Illustration of gradient mismatch between Eq. (5) and Eq. (7).

##  4 Our Bi-ViT

In this section, we propose to dismiss the affect of gradient mismatch
mentioned in Sec. 3.3 from perspectives of gradient approximation in Sec. 4.1
and intermediate distillation in Sec. 4.2.

###  4.1 Learnable Head-wise Scaling Factor

As one of the solution to the above mentioned problem, we propose a head-wise
scaling factor binarization scheme for the self-attention process, where the
scaling factors are learned during training to first modify the gradient clip
range in Fig. 5(b). Eq. (2) is changed into:

$\displaystyle\tilde{\bf A}$ $\displaystyle=\operatorname{softmax}(\tilde{\bf
p}),$ (9) $\displaystyle\tilde{\bf p}$
$\displaystyle=(\bm{\alpha}_{q}\otimes\bm{\alpha}_{k})\circ({\bf
b}_{{a}_{q}}\cdot{\bf b}_{{a}_{k}}^{\top})/\sqrt{d}$
$\displaystyle=\bm{\alpha}_{q;k}\circ({\bf b}_{{a}_{q}}\cdot{\bf
b}_{{a}_{k}}^{\top})/\sqrt{d},$

and

$\displaystyle\tilde{\bf a}_{out}$ $\displaystyle=(\bm{\alpha}_{A}\circ{\bf
b}_{A})\cdot(\bm{\alpha}_{v}\circ{\bf b}_{{a}_{v}})^{\top}$ (10)
$\displaystyle=(\bm{\alpha}_{A}\otimes\bm{\alpha}_{v})\circ({\bf
b}_{A}\cdot{\bf b}_{{a}_{v}}^{\top})$
$\displaystyle=\bm{\alpha}_{A;v}\circ({\bf b}_{A}\cdot{\bf
b}_{{a}_{v}}^{\top}),$

where ${\bf b}_{{a}_{\cdot}}=\operatorname{sign}(\frac{{\bf
a}_{\cdot}}{\bm{\alpha}_{\cdot}})$, $\bm{\alpha}_{q}$, $\bm{\alpha}_{k}$,
$\bm{\alpha}_{v}$ and $\bm{\alpha}_{A}$ are the head-wise learnable scaling
factors in binarized MHSA, where
$\bm{\alpha}_{\\{q,k,v,A\\}}=\\{\alpha_{\\{q,k,v,A\\}}^{1},\alpha_{\\{q,k,v,A\\}}^{2},\cdots,\alpha_{\\{q,k,v,A\\}}^{h}\\}\in\mathbb{R}^{h}_{+}$.
The second rows in Eq. (9) & Eq. (10) are established since the scaling
factors are aligned with the head dimension, which is independent with the
matrix multiplication operation. Thus,
$\bm{\alpha}_{q;k}=\\{\alpha_{q;k}^{1},\alpha_{q;k}^{2},\cdots,\alpha_{q;k}^{h}\\}\in\mathbb{R}^{h}_{+}$
and
$\bm{\alpha}_{A;v}=\\{\alpha_{A;v}^{1},\alpha_{A;v}^{2},\cdots,\alpha_{A;v}^{h}\\}\in\mathbb{R}^{h}_{+}$.

Consequently, the gradient $\frac{\partial\tilde{\bf A}}{\partial{\bf
a}_{q}^{:,n,c}}$ in Eq. (8) is further formulated in our Bi-ViT as:

$\displaystyle\frac{\partial\tilde{\bf A}}{\partial{\bf a}_{q}^{h_{i},n,c}}$
$\displaystyle=\underbrace{{\tilde{\bf A}^{h_{i},n,n^{\prime}}(1-\tilde{\bf
A}^{h_{i},n,n^{\prime}})}}_{\frac{\partial\tilde{\bf A}}{\partial{\bf
p}^{h_{i},n,n^{\prime}}}}\cdot\underbrace{\bm{\alpha}_{q;k}^{h_{i}}\circ{\bf
b}_{a_{k}}^{h_{i},c,n^{\prime}}}_{\frac{\partial{\bf
p}^{h_{i},n,n^{\prime}}}{\partial{\bf
b}_{{a}_{q}}^{h_{i},n,c}}}\cdot\underbrace{{\bf 1}_{|{\bf
a}_{q}^{h_{i},n,c}|\leq\bm{\alpha}_{q}}}_{\frac{\partial{\bf
b}_{{a}_{q}}^{h_{i},n,c}}{\partial{\bf a}_{q}^{h_{i},n,c}}}.$ (11)

Since $\operatorname{softmax}(.)$ and $\circ$ are aligned with different
dimensions, the value of Eq. (5) remains unchanged
($\operatorname{softmax}({\bf
p})=\operatorname{softmax}(\bm{\alpha}_{q;k}\circ{\bf p})$). As can be seen,
the threshold of gradient clip in Eq. (7) changes from $1$ into
$\bm{\alpha}_{q}$, which means that we can surpass the occurance of gradient
mismatch by modifying the value of $\bm{\alpha}_{q}$. Note that the scaling
factor ($\bm{\alpha}_{q}$) is to imitate the magnitude of the latent
activations. When $\tilde{\bf p}$ has a large magnitude, i.e., in the circled
part of Fig. 5 (a), $\bm{\alpha}_{q}$ also tends to be larger and ${\bf
a}_{q}^{h_{i},n,c}$ locates in the field that $\frac{\partial{\bf
b}_{{a}_{q}}^{h_{i},n,c}}{\partial{\bf a}_{q}^{h_{i},n,c}}>0$. Thus the
vanishing gradients are reactivated through the introduced learnable scaling
factor.

###  4.2 Ranking-aware Distillation for Bi-ViT

Fig. 2 illustrates a significant difference in the attention map’s relative
order between Bi-RealNet (a) and its real-valued counterpart (c). This
difference could result in a notable decrease in performance. To address this
issue during binarized training, a ranking-aware distillation in a teacher-
student framework is introduced:

$\displaystyle\mathcal{L}_{ranking}=\sum_{l=1}^{L}\|\psi({\bf
A}^{\mathcal{T}})-\psi({\bf A}^{\mathcal{S}})\|_{2},$ (12)

where ${\bf A}^{\mathcal{T}}$ and ${\bf A}^{\mathcal{S}}$ represents the
attention scores from the real-valued teacher and binarized student.
$\psi(\cdot)$ denotes the function for obtaining the ranking, i.e., relative
order of an attention score, which is formulated as:

$\displaystyle\psi({\bf A}^{:,n,:})=\left\\{\begin{aligned} {\bf
A}^{:,n,:}-{\bf A}^{:,n-1,:},&\;\;\;\;\operatorname{if}\;\;0<n\leq N-1\\\ {\bf
A}^{:,0,:}-{\bf
A}^{:,N-1,:},&\;\;\;\;\operatorname{otherwise}.\end{aligned}\right.$ (13)

Detailed relative order computation can be seen in the right part of Fig. 3.
We implement our Bi-ViT under the teacher-student framework [32], thus the
final objective of our method is formulated as:

$\displaystyle\mathcal{L}=\mathcal{L}_{dist}+\lambda\mathcal{L}_{ranking},$
(14)

where $\lambda$ is a hyper-parameter to balance these two loss functions.

##  5 Experiments

In this section, we evaluate the performance of the proposed Bi-ViT model for
image classification task using popular DeiT [32] & Swin [23] backbones and
object detection task using Mask R-CNN [14] & Cascade [3] Mask R-CNN with
Swin-Tiny [23] backbone. To the best of our knowledge, there is no publicly
available source codebase on fully-binarized ViTs at this point, so we
implement the baseline i.e., Bi-Real Net [26] methods by ourselves.

###  5.1 Datasets and Implementation Details

Datasets. The experiments are carried out on the ImageNet ILSVRC12 dataset
[17] for image classification task and COCO dataset [21] for object detection
task. The ImageNet dataset is challenging due to its large scale and greater
diversity. There are 1000 classes and 1.2 million training images, and 50k
validation images in it. In our experiments, we use the classic data
augmentation method described in [32].

The COCO [21] dataset includes images from 80 different categories. All of our
COCO dataset experiments are performed on the object detection track of the
COCO trainval35k training dataset, which consists of 80k images from the COCO
train2014 dataset and 35k images sampled from the COCO val2014 dataset. We
report the average precision (AP) for IoUs$\in$ [0.5:0.05:0.95], designated as
mAP@[.5,.95], using COCO’s standard evaluation metric. For further analyzing
our method, we also report AP50, AP75, APs, APm, and APl.

Experimental settings. In our experiments, we initialize the weights of
binarized model with the corresponding pretrained real-valued model. The
binarized model is trained for 300 epochs with batch-size 512 and the base
learning rate $5e\\!-\\!4$. We do not use warm-up scheme. For all the
experiments, we apply LAMB [39] optimizer with weight decay set as 0,
following DeiT III [33]. Note that we keep the patch embedding (first) layer
and the classification (last) layer as real-valued, following [10].

Backbone. We evaluate our binarized method on two popular vision transformer
networks: DeiT [32] and Swin Transformer [23]. The DeiT-Tiny, DeiT-Small,
DeiT-Base, Swin-Tiny and Swin-Small are adopted as the backbone models, whose
Top-1 accuracy on ImageNet dataset are 72.2%, 79.9%, 81.8%, 81.2%, and 83.2%
respectively. For a fair comparison, we utilize the official implementation of
DeiT and Swin Transformer.

###  5.2 Ablation Study

Hyper-parameter Selection. We $\lambda$ of Eq. (14) in this part, with
experiments conducted on ImageNet [17] dataset. We show the model performance
(Top-1 accuracy) with different setups of hyper-parameter $\lambda$ in Fig. 6,
in which the performances increase first and then decrease with the uplift of
$\lambda$ from left to right. Since $\lambda$ controls the importance of
$\mathcal{L}_{ranking}$, we show that the vanilla baseline ($\lambda=0$)
performs worse than any versions with Ranking-aware Distillation loss
($\lambda>0$), showing the proposed distillation scheme is necessary. With the
varying value of $\lambda$, we find $\lambda=10$ boost the performance of our
Bi-ViT, achieving 28.7%, 40.9% and 50.7% Top-1 accuracy on ImageNet [17] with
DeiT-Tiny, DeiT-Small and Swin-Tiny backbone, respectively.

Figure 6: Effect of hyper-parameter $\lambda$ on ImageNet [17].

Effectiveness of components. We conduct the ablative experiments regarding the
proposed components on DeiT-Tiny network. Firstly, we compose the baseline
network using the binarization method following Bi-Real Net [26]. As shown in
the third row of Tab. 1, the baseline networks only obtains 6.6% Top-1
accuracy, which is far from satisfactory. With the introduction of our first
novelty, i.e., learnable scaling factor (LSF), the baseline network is boosted
by 17.8%, achieving 24.4% Top-1 accuracy. We also observe the other
contribution Ranking-aware Disitllation (RD) singly promotes the baseline
network by 5.9%, which is also significant on ImageNet dataset. By combining
the two main contributions together, we get Bi-ViT, outperforming the vanilla
baseline by 22.1%.

Table 1: Evaluating the components of Bi-ViT based on DeiT-Tiny [32] backbone. “#Bits” denotes the bit-width of weights and activations. We report the Top-1(%) accuracy performances.  Method | #Bits | Top-1(%)
---|---|---
Real-valued | 32-32 | 72.1
Baseline (Bi-Real Net [26]) | 1-1 | 6.6
\+ Learnable Scaling Factor (LSF) | 1-1 | 24.4+17.8
\+ Ranking-aware Distillation (RD) | 1-1 | 12.5+5.9
1-3 \+ LSF + RD (Bi-ViT) | 1-1 | 28.7+22.1
Table 2: Experiments with DeiT [32] and Swin [23] on ImageNet [17]. “#Bits” denotes the bit-width of weights and activations. We report the Top-1(%) and Top-5(%) accuracy performances. The bold denotes the best result with binarized weights and activations. Network | Method | #Bits | Size(MB) | OPs${}_{\rm(10^{8})}$ | Top-1(%) | Top-5(%)
---|---|---|---|---|---|---
| Real-valued | 32-32 | 22.8 | 12.3 | 72.2 | 91.1
|  | 4-4 | 3.0 | 1.6 | 74.3 | 91.7
|  | 3-3 | 2.3 | 0.8 | 71.5 | 91.2
| Q-ViT [18] | 2-2 | 1.7 | 0.4 | 59.0 | 81.8
2-7 | BiBERT [28] |  | 1.0 | 0.2 | 5.9 | 16.0
| RBONN [36] |  | 6.3 | 16.9
| Bi-Real Net [26] |  | 6.6 | 17.1
DeiT-Tiny | Bi-ViT | 1-1 | 28.7+22.1 | 51.7+34.6
| Real-valued | 32-32 | 88.2 | 45.5 | 79.9 | 95.0
|  | 4-4 | 11.4 | 5.8 | 80.9 | 94.9
|  | 3-3 | 8.7 | 3.0 | 79.0 | 94.2
| Q-ViT [18] | 2-2 | 6.0 | 1.5 | 72.1 | 90.3
2-7 | BiBERT [28] |  | 3.4 | 0.8 | 17.4 | 29.7
| RBONN [36] |  | 18.5 | 30.0
| Bi-Real Net [26] |  | 19.2 | 30.3
DeiT-Small | Bi-ViT | 1-1 | 40.9+21.7 | 65.0+34.7
| Real-valued | 32-32 | 346.2 | 174.7 | 81.8 | 95.6
|  | 4-4 | 44.1 | 22.0 | 83.0 | 96.1
|  | 3-3 | 33.4 | 11.1 | 81.0 | 95.1
| Q-ViT [18] | 2-2 | 22.7 | 5.7 | 74.2 | 92.2
2-7 | BiBERT [28] |  | 12.1 | 2.9 | 24.5 | 36.3
| RBONN [36] |  | 26.1 | 38.6
| Bi-Real Net [26] |  | 26.5 | 38.8
DeiT-Base | Bi-ViT | 1-1 | 47.3+20.8 | 72.8+34.0
| Real-valued | 32-32 | 114.2 | 44.9 | 81.2 | 95.5
|  | 4-4 | 14.6 | 5.8 | 82.5 | 97.3
|  | 3-3 | 11.2 | 3.0 | 80.9 | 96.1
| Q-ViT [18] | 2-2 | 10.0 | 1.6 | 74.7 | 92.5
2-7 | BiBERT [28] |  | 4.2 | 0.8 | 34.0 | 46.9
| RBONN [36] |  | 33.8 | 46.7
| Bi-Real Net [26] |  | 34.1 | 46.9
Swin-Tiny | Bi-ViT | 1-1 | 55.5+21.4 | 79.4+32.5
| Real-valued | 32-32 | 199.8 | 87.5 | 83.2 | 96.2
|  | 4-4 | 25.3 | 11.1 | 84.4 | 98.3
|  | 3-3 | 19.2 | 5.6 | 82.7 | 97.5
| Q-ViT [18] | 2-2 | 13.0 | 2.9 | 76.9 | 94.9
2-7 | BiBERT [28] |  | 6.9 | 1.5 | 39.4 | 53.0
| RBONN [36] |  | 39.0 | 52.7
| Bi-Real Net [26] |  | 39.2 | 52.8
Swin-Small | Bi-ViT | 1-1 | 60.7+21.5 | 83.9+31.1

###  5.3 Results on Image Classification

The experimental results are shown in Tab. 2. We compare our method with 1-bit
methods including BiBERT [28], RBONN [36], and Bi-Real Net [26] based on the
same frameworks for the task of image classification with the ImageNet
dataset. We also report the classification performance of the low-bit
training-aware quantization method Q-ViT [18] for further reference. We use
model size and OPs following [26] in comparison to other bit-widthe models for
further reference. We firstly evaluate the proposed method on DeiT models.

Table 3: Experiments with Mask R-CNN [14] and Cascade R-CNN [3] using Swin [23] backbones on COCO [21]. “#Bits” denotes the bit-width of weights and activations. We report the AP (%) with different IoU threshold and AP for objects in various sizes. The bold denotes the best result with binarized weights and activations. Framework | Backbone | Method | # Bits | Size(MB) | AP | AP50 | AP75 | APs | APm | APl
---|---|---|---|---|---|---|---|---|---|---
Mask R-CNN [14] | Swin-Tiny | Real-valued | 32-32 | 191.3 | 43.7 | 66.6 | 47.7 | 28.5 | 47.0 | 57.3
| 4-4 | 94.9 | 43.3 | 66.3 | 47.1 | 28.2 | 46.5 | 57.5
| 3-3 | 91.4 | 40.1 | 63.5 | 43.9 | 25.4 | 42.4 | 54.9
Q-ViT [18] | 2-2 | 88.0 | 30.2 | 53.7 | 33.4 | 15.2 | 32.0 | 45.2
3-11 | BiBERT [28] | 1-1 | 84.5 | 8.9 | 25.0 | 8.6 | 1.7 | 9.0 | 15.9
| RBONN [36] | 9.5 | 25.2 | 8.9 | 1.9 | 9.1 | 16.0
| Bi-Real Net [26] | 9.9 | 25.2 | 9.2 | 2.1 | 9.1 | 16.4
| Bi-ViT | 20.7 | 38.7 | 19.9 | 12.0 | 20.9 | 27.6
Cascade Mask R-CNN [3] | Swin-Tiny | Real-valued | 32-32 | 342.7 | 48.1 | 67.1 | 52.2 | 30.4 | 51.5 | 63.1
| 4-4 | 246.2 | 47.2 | 66.4 | 52.0 | 30.0 | 51.1 | 62.8
| 3-3 | 242.8 | 44.5 | 63.1 | 49.3 | 27.8 | 48.2 | 59.9
Q-ViT [18] | 2-2 | 239.3 | 34.0 | 33.8 | 39.6 | 17.2 | 37.6 | 49.1
3-11 | BiBERT [28] | 1-1 | 235.9 | 16.7 | 32.2 | 18.1 | 11.0 | 18.7 | 25.1
| RBONN [36] | 17.5 | 32.6 | 19.0 | 11.2 | 19.0 | 25.7
| Bi-Real Net [26] | 18.1 | 33.3 | 19.1 | 12.0 | 19.2 | 25.9
| Bi-ViT | 29.0 | 45.0 | 31.1 | 16.8 | 29.4 | 39.8

For DeiT-Tiny backbone, compared with other binary methods, our Bi-ViT
achieves significant performance improvements. For example, our Bi-ViT
surpasses the baseline Bi-Real Net [26] by 22.1% Top-1 accuracy, which is
significant and meaningful for real-world applications. And it is worth noting
that the proposed 1-bit model significantly compresses the DeiT-Tiny by
$61.5\times$ on OPs. The proposed method also boosts the performance of
baseline by 21.7% with the same architecture and bit-width using DeiT-Small
bacobone, a significant improvement on the ImageNet dataset. For larger
DeiT-B, as shown in Tab. 2, the performance of the proposed method outperforms
the Bi-Real Net by 20.8%, a large margin. Also note that the proposed 1-bit
model significantly compresses the DeiT-B by 60.2$\times$ on OPs and
28.6$\times$ on model size.

Also, our method obtains convincing results on Swin-transformers. As shown in
Tab. 2, the performance of the proposed method with Swin-Tiny outperforms the
baseline method by 21.4%, a large margin. For larger Swin-Small, the
performance of the proposed method outperforms the 1-bit baseline by 21.5%.
Also note that our method theoritically accelerates the network by
58.3$\times$, which demonstrates the effectiveness and efficiency of our Bi-
ViT.

###  5.4 Results on Object Detection

We conduct experiments on object detection using the COCO dataset and compare
our Bi-ViT with previous binary neural networks such as BiBERT [28], RBONN
[36], and Bi-Real Net [26]. To provide a basis for comparison, we also
included the performance of 2/3/4-bit Q-ViT in Table 3. Our method outperform
BiBERT, RBONN, and Bi-Real Net in terms of AP@[.5,.95] by 11.8%, 11.2%, and
10.8% when using the Mask R-CNN framework with the Swin-Tiny backbone.

Furthermore, our Bi-ViT model showed superior performance on other APs with
different IoU thresholds and achieved 9.5% lower AP than the 2-bit Q-ViT
model. Our method also yielded a 1-bit transformer-based detector with a
performance of 23.0% AP lower than the real-valued counterpart (43.7% vs.
20.7%) while utilizing 2.3$\times$ less memory. When using the Cascade Mask
R-CNN framework with the Swin-Tiny backbone, our method achieved 29.0%
AP@[.5,.95], outperforming BiBERT, RBONN, and Bi-Real Net by 12.3%, 11.5%, and
10.9% mAP, respectively.

In conclusion, our method demonstrated superior performance in terms of AP
with various IoU thresholds and AP for objects of different sizes on the COCO
dataset, showing its applicability and superiority in various application
settings compared to previous binary neural networks.

##  6 Conclusion

In this paper, we present Bi-ViT, an improved version of fully-binarized ViTs
that offers a high compression ratio and acceptable performance. Initially, we
establish a empirical framework for fully-binarized ViT and analyze the
bottlenecks of the baseline. Our empirical analysis shows that attention
distortion in MHSA is the primary cause of the significant drop in ViT
binarization, which results from gradient vanishing and ranking disorder. To
address these issues, we introduce a learnable scaling factor that reactivates
vanished gradients, which we illustrate through both theoretical and
experimental analysis. Additionally, we propose ranking-aware distillation for
Bi-ViT, which rectifies disordered ranking in a teacher-student framework. Our
work provides a comprehensive analysis and effective solutions for the crucial
issues in ViT full binarization, paving the way for the extreme compression of
ViT.

## References

  * [1] Ron Banner, Yury Nahshan, and Daniel Soudry.  Post training 4-bit quantization of convolutional networks for rapid-deployment.  In Proc. of NeurIPS, pages 7950–7958, 2019.
  * [2] Yoshua Bengio, Nicholas Léonard, and Aaron Courville.  Estimating or propagating gradients through stochastic neurons for conditional computation.  arXiv preprint arXiv:1308.3432, 2013.
  * [3] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In Proc. of CVPR, pages 6154–6162, 2018.
  * [4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Proc. of ECCV, pages 213–229, 2020.
  * [5] Mengzhao Chen, Mingbao Lin, Ke Li, Yunhang Shen, Yongjian Wu, Fei Chao, and Rongrong Ji.  Cf-vit: A general coarse-to-fine method for vision transformer.  In Proc. of AAAI, pages 1–13, 2023.
  * [6] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David.  Binaryconnect: Training deep neural networks with binary weights during propagations.  In Proc. of NeurIPS, pages 3123–3131, 2015.
  * [7] Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando De Freitas.  Predicting parameters in deep learning.  In Proc. of NeurIPS, pages 2148–2156, 2013.
  * [8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In Proc. of ICLR, pages 1–22, 2020.
  * [10] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha.  Learned step size quantization.  arXiv preprint arXiv:1902.08153, 2019.
  * [11] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze.  Levit: a vision transformer in convnet’s clothing for faster inference.  In Proc. of ICCV, pages 12259–12269, 2021.
  * [12] Zhiwei Hao, Jianyuan Guo, Ding Jia, Kai Han, Yehui Tang, Chao Zhang, Han Hu, and Yunhe Wang.  Learning efficient vision transformers via fine-grained manifold distillation.  In Proc. of NeurIPS, pages 1–11, 2021.
  * [13] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proc. of CVPR, pages 16000–16009, 2022.
  * [14] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proc. of ICCV, pages 2961–2969, 2017.
  * [15] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proc. of CVPR, pages 7132–7141, 2018.
  * [16] Felix Juefei-Xu, Vishnu Naresh Boddeti, and Marios Savvides.  Local binary convolutional neural networks.  In Proc. of CVPR, pages 19–28, 2017.
  * [17] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Proc. of NeurIPS, pages 1097–1105, 2012.
  * [18] Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, and Guodong Guo.  Q-vit: Accurate and fully quantized low-bit vision transformer.  In Proc. of NeurIPS, pages 1–12, 2022.
  * [19] Zhexin Li, Tong Yang, Peisong Wang, and Jian Cheng.  Q-vit: Fully differentiable quantization for vision transformer.  In Proc. of WACV, pages 1–10, 2022.
  * [20] Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Yan Wang, Yongjian Wu, Feiyue Huang, and Chia-Wen Lin.  Rotated binary neural network.  In Proc. of NeurIPS, pages 1–9, 2020.
  * [21] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Proc. of ECCV, pages 740–755, 2014.
  * [22] Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou.  Fq-vit: Fully quantized vision transformer without retraining.  In Proc. of IJCAI, pages 1173–1179, 2022.
  * [23] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proc. of ICCV, pages 10012–10022, 2021.
  * [24] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng.  Reactnet: Towards precise binary neural network with generalized activation functions.  In Proc. of ECCV, pages 143–159, 2020.
  * [25] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao.  Post-training quantization for vision transformer.  In Proc. of NeurIPS, 2021.
  * [26] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng.  Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm.  In Proc. of ECCV, pages 722–737, 2018.
  * [27] Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos.  Training binary neural networks with real-to-binary convolutions.  In Proc. of ICLR, pages 1–11, 2020.
  * [28] Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xianglong Liu.  Bibert: Accurate fully binarized bert.  In Proc. of ICLR, pages 1–24, 2022.
  * [29] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh.  Dynamicvit: Efficient vision transformers with dynamic token sparsification.  Proc. of NeurIPS, pages 1–14, 2021.
  * [30] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi.  Xnor-net: Imagenet classification using binary convolutional neural networks.  In Proc. of ECCV, pages 525–542, 2016.
  * [31] Yunjie Tian, Lingxi Xie, Zhaozhi Wang, Longhui Wei, Xiaopeng Zhang, Jianbin Jiao, Yaowei Wang, Qi Tian, and Qixiang Ye.  Integrally pre-trained transformer pyramid networks.  arXiv preprint arXiv:2211.12735, 2022.
  * [32] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In Proc. of ICML, pages 10347–10357, 2021.
  * [33] Hugo Touvron, Matthieu Cord, and Hervé Jégou.  Deit iii: Revenge of the vit.  In Proc. of ECCV, pages 516–533, 2022.
  * [34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proc. of NeurIPS, pages 1–11, 2017.
  * [35] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In Proc. of ICCV, pages 568–578, 2021.
  * [36] Sheng Xu, Yanjing Li, Tiancheng Wang, Teli Ma, Baochang Zhang, Peng Gao, Yu Qiao, Jinhu Lü, and Guodong Guo.  Recurrent bilinear optimization for binary neural networks.  In Proc. of ECCV, pages 19–35, 2022.
  * [37] Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun.  Evo-vit: Slow-fast token evolution for dynamic vision transformer.  In Proc. of AAAI, pages 2964–2972, 2022.
  * [38] Huanrui Yang, Hongxu Yin, Pavlo Molchanov, Hai Li, and Jan Kautz.  Nvit: Vision transformer compression and parameter redistribution.  arXiv preprint arXiv:2110.04869, 2021.
  * [39] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh.  Large batch optimization for deep learning: Training bert in 76 minutes.  Proc. of ICLR, pages 1–37, 2020.
  * [40] Yunshan Zhong, Mingbao Lin, Mengzhao Chen, Ke Li, Yunhang Shen, Fei Chao, Yongjian Wu, and Rongrong Ji.  Fine-grained data distribution alignment for post-training quantization.  In Proc. of ECCV, pages 70–86, 2022.