In neural network binarization, BinaryConnect (BC) and its variants are considered the standard. These methods apply the $\sign$ function in their forward pass and their respective gradients are backpropagated to update the weights. However, the derivative of the $\sign$ function is zero whenever defined, which consequently freezes training. Therefore, implementations of BC (e.g., BNN) usually replace the derivative of $\sign$ in the backward computation with identity or other approximate gradient alternatives. Although such practice works well empirically, it is largely a heuristic or “training trick.” We aim at shedding some light on these training tricks from the optimization perspective. Building from existing theory on ProxConnect (PC, a generalization of BC), we (1) equip PC with different forward-backward quantizers and obtain ProxConnect++ (PC++) that includes existing binarization techniques as special cases; (2) derive a principled way to synthesize forward-backward quantizers with automatic theoretical guarantees; (3) illustrate our theory by proposing an enhanced binarization algorithm BNN++; (4) conduct image classification experiments on CNNs and vision transformers, and empirically verify that BNN++ generally achieves competitive results on binarizing these models.

§ INTRODUCTION

The recent success of numerous applications in machine learning is largely fueled by training big models with billions of parameters, , GPTs in large language models <cit.>, on extremely large datasets.  However, as such models continue to scale up, end-to-end training or even fine-tuning becomes prohibitively expensive, due to the heavy amount of computation,  memory and storage required. Moreover, even after successful training, deploying these models on resource-limited devices or environments that require real-time inference still poses significant challenges.

Inspired by the success of transformers in Natural Language Processing (NLP) tasks, a transformer-based architecture in computer vision, namely the vision transformer, also achieves impressive performance. Vision transformers succeed in applications
such as image classification <cit.>,  object detection <cit.>, and instance segmentation <cit.>. However, due to the size of images, training a vision transformer usually leads to the calculation of a large attention matrix, and accordingly to heavy computations and intensive memory costs.
Moreover, the good performance of vision transformers usually relies on pre-training on huge datasets achievable through over-parameterization <cit.>.

Thus, for real-time or resource-limited devices, it is extremely difficult to apply such models at inference time.

A common way to tackle the above problems is through model compression,
such as pruning <cit.>, reusing attention <cit.>, weight sharing <cit.>, structured factorization <cit.>, and network quantization <cit.>.
Among them, network quantization (i.e., replacing full-precision weights with lower-precision ones) is a popular approach.

In this work we focus on an extreme case of network quantization: binarization, i.e., constraining a subset of the weights to be only binary (, $\pm 1$),

with the benefit of much reduced memory and storage cost, as well as inference time through simpler and faster matrix-vector multiplications, which is one of the main computationally expensive steps in transformers and the recently advanced vision transformers <cit.>.

For neural network binarization, BinaryConnect <cit.> is considered the de facto standard. BC applies the $\sign$ function to binarize the weights in the forward pass, and evaluates the gradient at the binarized weights using the Straight Through Estimator <cit.>[Note that we refer to STE as its original definition by <cit.> for binarizing weights, and other variants of STE (e.g., in BNN) as  approximate gradient.].
This widely adopted training trick has been formally justified from an optimization perspective: <cit.>, among others, identify BC as a nonconvex counterpart of dual averaging, which itself is a special case of the generalized conditional gradient algorithm. <cit.> further propose ProxConnect (PC) as an extension of BC, by allowing arbitrary proximal quantizers (with $\sign$ being a special case) in the forward pass.

However, practical implementations  <cit.> usually apply an approximate gradient of the $\sign$ function on top of STE. For example, <cit.> employ the hard $\tanh$ function as an approximator of $\sign$. Thus, in the backward pass, the derivative of $\sign$ is approximated by the indicator function $\mathbf{1}_{[-1,1]}$, the derivative of hard $\tanh$. Later, <cit.> consider the $\signswish$ function as a more accurate and flexible approximation in the backward pass (but still employs the $\sign$ in the forward pass).

Activation functions directly using figure from BNN+ for now, will replace it soon

Activation functions directly using figure from BNN+ for now, will replace it soon

Despite their excellent performance in practice, approximate gradient approaches cannot be readily understood in the PC framework of <cit.>, which does not equip any quantization in the backward pass.
Thus, the main goal of this work is to further generalize PC and improve our understanding of approximate gradient approaches.
Specifically, we introduce PC++ that comes with a pair of forward-backward proximal quantizers, and we show that most of the existing

approximate gradient approaches are special cases of our proximal quantizers, and hence offering a formal justification of their empirical success from an optimization perspective. Moreover, inspired by our theoretical findings, we propose a novel binarization algorithm BNN++ that  improves BNN+ <cit.> on both theoretical convergence properties and empirical performances. Notably, our work provides direct guidance on designing new forward-backward proximal quantizers in the PC++ family, with immediate theoretical guarantees while enabling streamlined implementation and comparison of a wide family of existing quantization algorithms.

Empirically, we benchmark existing PC++ algorithms (including the new BNN++) on image classification tasks on CNNs and vision transformers. Specifically, we perform weight (and activation) binarization on various datasets and models. Moreover, we explore the fully binarized scenario, where the dot-product accumulators are also quantized to 8-bit integers. In general, we observe that BNN++ is very competitive against existing approaches on most tasks, and achieves 30x reduction in memory and storage with a modest 5-10% accuracy drop compared to full precision training.

We summarize our main contributions in more detail:

* We generalize ProxConnect with forward-backward quantizers and introduce ProxConnect++ (PC++) that includes existing binarization techniques as special cases.

* We derive a principled way to synthesize forward-backward quantizers with theoretical guarantees. Moreover, we design a new BNN++ variant to illustrate our theoretical findings.

* We empirically compare different choices of forward-backward quantizers on image classification benchmarks, and confirm that BNN++ is competitive against existing alternatives.

§ BACKGROUND

In neural network quantization, we aim at minimizing the usual (nonconvex) objective function $\ell(\wv)$ with discrete weights $\wv$:

\begin{align}
\label{eq:prob}
\min_{\wv \in Q} ~ \ell(\wv),
\end{align}

where $Q \subseteq \RR^d$ is a discrete, nonconvex quantization set such as $Q = \{\pm 1\}^d$. The acquired discrete weights $\wv \in Q$ are compared directly with continuous full precision weights, which we denote as $\wv^*$ for clarity.

While our work easily extends to most discrete set $Q$, we focus on $Q = \{\pm 1\}^d$ since this binary setting remains most challenging and leads to the most significant savings.

Existing binarization schemes can be largely divided into the following two categories.

Post-Training Binarization (PTB): we can formulate post-training binarization schemes as the following standard forward and backward pass:

\begin{align}
\wv_{t} = \tP_Q(\wv_{t}^*),
\qquad
\wv_{t+1}^* &= \wv_t^* - \eta_t\widetilde\nabla \ell(\wv_t^*),
\end{align}

where $\tP_Q$ is the projector that binarizes the continuous weights $\wv^*$ deterministically (, the $\sign$ function) or stochastically[We only consider deterministic binarization in this paper.], and $\widetilde\nabla \ell(\wv_t^*)$ denotes a sample (sub)gradient of $\ell$ at $\wv_t^*$.
We point out that PTB is merely a post-processing step, i.e., the binarized weights $\wv_t$ do not affect the update of the continuous weights $\wv_t^*$, which are obtained through normal training. As a result, there is no guarantee that the acquired discrete weights $\wv_t$ is a good solution (either global or local) to <ref>.

Binarization-Aware Training (BAT): we then recall the more difficult binarization-aware training scheme BinaryConnect (BC), first initialized by <cit.>:

\begin{align}
\label{eq:BC}
\wv_{t} = \tP_Q(\wv_{t}^*),
\qquad
\wv_{t+1}^* = \wv_t^* - \eta_t \widetilde\nabla \ell(\wv_t)
\end{align}

where we spot that the gradient is evaluated at the binarized weights $\wv_t$ but used to update the continuous weights $\wv_t^*$. This approach is also known as Straight Through Estimator <cit.>. Note that it is also possible to update the binarized weights instead, effectively performing the proximal gradient algorithm to solve (<ref>), as shown by <cit.>:

\begin{align}
\wv_{t} = \tP_Q(\wv_{t}^*),
\qquad
\wv_{t+1}^* = \wv_t - \eta_t \widetilde\nabla \ell(\wv_t)
\end{align}

This method is known as ProxQuant, and will  serve as a baseline in our experiments.

§.§ ProxConnect

<cit.> proposed ProxConnect (PC) as a broad generalization of BinaryConnect in (<ref>):

\begin{align}
\label{eq:pc}
\wv_t = \tP_{\textsf{r}}^{\mu_{t}} (\wv_t^*), \quad
\wv_{t+1}^* = \wv_t^* - \eta_t \widetilde\nabla \ell(\wv_t),
\end{align}

where $\mu_t := 1 + \sum_{\tau=1}^{t-1} \eta_\tau$, $\eta_t > 0$ is the step size, and
$\tP_{\rsf}^{\mu_{t}}$ is the proximal quantizer:

\begin{align}
\tP_{\rsf}^\mu(\wv) &:= \argmin_{\zv} ~ \tfrac{1}{2\mu}\|\wv- \zv\|_2^2 + \rsf(\zv), \mbox{ and } \\
\Mc_{\rsf}^\mu(\wv) &:= \min_{\zv} ~ \tfrac{1}{2\mu}\|\wv- \zv\|_2^2 + \rsf(\zv).
\end{align}

In particular, when the regularizer $\textsf{r} = \iota_Q$ (the indicator function of $Q$),  $\tP_{\textsf{r}}^{\mu_{t}} = \tP_{Q}$ (for any $\mu_{t}$) and we recover BC in (<ref>). <cit.> showed that the PC update (<ref>) amounts to applying the generalized conditional gradient algorithm to a smoothened dual of the regularized problem:

\begin{align}
\min_{\wv}~ \left [\ell(\wv) + \rsf(\wv) \right] \approx
\min_{\wv^*}~ \ell^*(-\wv^*) + \Mc_{\rsf^*}^\mu(\wv^*),
\end{align}

where $f^*(\wv^*):= \max_{\wv} \langle \wv, \wv^*\rangle - f(\wv)$ is the Fenchel conjugate of $f$.
The theory behind PC thus formally justifies  STE from an optimization perspective. We provide a number of examples of the proximal quantizer $\tP_{\textsf{r}}^{\mu_{t}}$ in <Ref>.

Another natural cousin of PC is the reversed PC (rPC):

\begin{align}
\textstyle
\wv_t = \tP_{\rsf}^{\mu_{t}} (\wv_t^*), \quad
\wv_{t+1}^* = \wv_t - \eta_t \widetilde\nabla \ell(\wv_t^*),
\end{align}

which is able to exploit the rich landscape of the loss by evaluating the gradient at the continuous weights $\wv_t^{*}$. Thus, we also include it as a baseline in our experiments.

We further discuss other related works in <Ref>.

§ METHODOLOGY

Forward and backward pass for ProxConnect and BNN/BNN+.

One popular heuristic to explain BC is through the following reformulation of  problem (<ref>):

\begin{align}
\min_{\wv^*} ~ \ell\bigl(\tP_Q(\wv^*)\bigl).
\end{align}

Applying (stochastic) “gradient” to update the continuous weights we obtain:

\begin{align}
\wv_{t+1}^* = \wv_t^* - \eta_t \cdot \tP_Q'(\wv_t^*) \cdot \widetilde\nabla \ell(\tP_Q(\wv_t^*)).
\end{align}

Unfortunately, the derivative of the projector $\tP_Q$ is 0 everywhere except at the origin, where the derivative actually does not exist. BC <cit.>, see (<ref>),  simply “pretended” that $\tP_Q' = I$. Later works propose to replace the troublesome $\tP_Q'$ by the derivative of functions that approximate $\tP_Q$, e.g., the hard tanh in BNN <cit.> and the $\signswish$ in BNN+ <cit.>. Despite their empirical success, it is not clear what is the underlying optimization problem or if it is possible to also replace the projector inside $\widetilde\nabla \ell$, , allowing the algorithm to evaluate gradients at continuous weights, a clear advantage demonstrated by <cit.> and <cit.>. Moreover, the theory established in PC, through a connection to the generalized conditional gradient algorithm, does not apply to these modifications yet, which is a gap that we aim to fill in this section.

§.§ ProxConnect++

To address the above-mentioned issues,
we propose to study the following regularized problem:

\begin{align}
\label{eq:tran-reg}
\min_{\wv^*}~ \ell(\Tsf(\wv^*)) + \rsf(\wv^*),
\end{align}

as a relaxation of the (equivalent) reformulation of (<ref>):

\begin{align}
\min_{\wv^*} ~ \ell(\tP_Q(\wv^*)) + \iota_Q(\wv^*).
\end{align}

In other words, $\Tsf: \RR^d \to \RR^d$ is some transformation that approximates $\tP_Q$ and the regularizer $\rsf: \RR^d \to \RR$ approximates the indicator function $\iota_Q$. Directly applying ProxConnect in (<ref>) we obtain[We assume throughout that $\Tsf$, and any function whose derivative we use, are locally Lipschitz so that their generalized derivative is always defined, see <cit.>.]:

\begin{align}
\label{eq:PC-ext-tmp}
\wv_t = \tP_{\textsf{r}}^{\mu_{t}} (\wv_t^*), ~~
\wv_{t+1}^* = \wv_t^* - \eta_t \Tsf'(\wv_t) \cdot \widetilde\nabla \ell\big(\Tsf(\wv_t)\big).
\end{align}

Introducing the forward and backward proximal quantizers:

\begin{align}
\label{eq:FB-cons}
\Fsf^{\mu}_{\rsf} := \Tsf \circ \tP_{\rsf}^{\mu}, \quad
\Bsf^{\mu}_{\rsf} := \Tsf' \circ \tP_{\rsf}^{\mu},
\end{align}

we can rewrite the update in (<ref>) simply as:

\begin{align}
\label{eq:prox+}
\wv_{t+1}^* = \wv_t^* - \eta_t \cdot \Bsf^{\mu_{t}}_{\rsf}(\wv_t^*)\cdot \widetilde\nabla \ell\big(\Fsf_{\rsf}^{\mu_{t}} (\wv_t^*) \big).
\end{align}

It is clear that the original ProxConnect corresponds to the special choice

\begin{align}
\Fsf_{\rsf}^\mu = \tP_{\rsf}^\mu, \quad \Bsf_{\rsf}^\mu \equiv I.
\end{align}

Of course, one may now follow the recipe in (<ref>) to design new forward-backward quantizers. We call this general formulation in (<ref>) ProxConnect++ (PC++), which covers a broad family of algorithms.

Forward and backward pass for ProxConnect+ (red arrows indicate the forward pass, and blue arrows indicate the backward pass). Here we show different choices of forward-backward quantizers generalize different binarization methods.

Forward and backward pass for ProxConnect++ algorithms (red/blue arrows indicate the forward/backward pass), where fp denotes full precision, bn denotes binary and back-prop denotes backpropagation.

Conversely, the complete characterization of proximal quantizers in <cit.> allows us also to reverse engineer $\Tsf$ and $\rsf$ from manually designed forward and backward quantizers. As we will see, most existing forward-backward quantizers turn out to be special cases of our proximal quantizers, and thus their empirical success can be justified from an optimization perspective. Indeed, for simplicity, let us restrict all quantizers to univariate ones that apply component-wise. Then, the following result is proven in <Ref>.
A pair of forward-backward quantizers $(\Fsf, \Bsf)$ admits the decomposition in (<ref>) (for some smoothing parameter $\mu$ and regularizer $\rsf$) iff both $\Fsf$ and $\Bsf$ are functions of $\tP(w) := \int_{-\infty}^w \tfrac{1}{\Bsf(\omega)}\rmd \Fsf(\omega)$, which is

(, monotone, compact-valued and with a closed graph).

Importantly, with forward-backward proximal quantizers, the convergence results established by <cit.> for PC directly carries over to PC++ (see <Ref> for details). Let us further illustrate the convenience of <Ref> by some examples.
<cit.> proposed BNN with the choice

\begin{align}
\Fsf = \sign \quad \mbox{and} \quad \Bsf = \one_{[-1,1]},
\end{align}

which satisfies the decomposition in (<ref>). Indeed, let

\begin{align}
\label{eq:BNN-T}
\Tsf(w) &= \min\{1, \max\{-1, w\} \}, \\
\label{eq:BNN-P}
\tP_{\rsf}^\mu(w) &= \begin{cases}
\tfrac{1}{\mu}w+\sign(w)(1-\tfrac{1}{\mu}), & \mbox{ if } |w| > 1 \\
\sign(w), & \mbox{ if } |w| \leq 1
\end{cases}
\end{align}

Since $\Bsf$ is constant over $[-1,1]$, applying <Ref> we deduce that the proximal quantizer $\tP_{\rsf}^\mu$, if exists, must coincide with $\Fsf$ over the support of $\Bsf$. Applying monotonicity of $\tP_{\rsf}^\mu$ we may complete the reverse engineering by making the choice over $|w| > 1$ as indicated above.
We can easily verify the decomposition in (<ref>):

\begin{align}
\Fsf = \sign = \Tsf \circ \tP_{\rsf}^\mu, ~~ \Bsf = \one_{[-1,1]} = \Tsf' \circ \tP_{\rsf}^\mu.
\end{align}

BNN is exactly BinaryConnect applied to the transformed problem in (<ref>),

where the transformation $\Tsf$ is the so-called hard tanh in (<ref>) while the regularizer $\rsf$ is determined (implicitly) by the proximal quantizer $\tP_{\rsf}^\mu$ in (<ref>).

To our best knowledge, this is the first time the (regularized) objective function that BNN aims to optimize has been identified. The convergence properties of BNN hence follow from the general result of <cit.> on ProxConnect, see  <Ref>.

<cit.> adopted the derivative of the sign-Swish (SS) function as a backward quantizer while retaining the sign function as the forward quantizer:

\begin{align}
\Bsf(\wv) &= \nabla \mathrm{SS}(\wv) := \mu[1-\tfrac{\mu\wv}{2} \tanh(\tfrac{\mu\wv}{2})] \tanh'(\tfrac{\mu\wv}{2}),
%\frac{\mu [2-\mu \wv \tanh(\mu\wv/2)}{1+\cosh(\mu\wv)}
~~ \Fsf = \sign,
%\Tsf(\wv) &= 2\sigma(\beta \wv)[1+\beta\wv\{1-\sigma(\beta\wv)\}]-1,
\end{align}

where $\mu$ is a hyperparameter that controls how well SS approximates the sign.
Applying <Ref> we find that the derivative of SS (as backward) coupled with the sign (as forward) do not admit the decomposition in (<ref>), for any regularizer $\rsf$. Thus, we are not able to find the (regularized) objective function (if it exists) underlying BNN+.

We conclude that BNN+ cannot be  justified under the framework of PC++. However, it is possible to design a variant of BNN+ that does belong to the PC++ family and hence enjoys the accompanying theoretical properties:
[BNN++] We propose that a simple fix of BNN+ would be to replace its $\sign$ forward quantizer with the sign-Swish (SS) function:

\begin{align}
\Fsf(\wv) = \mathrm{SS}(\wv) := \tfrac{\mu\wv}{2}  \tanh'(\tfrac{\mu\wv}{2} ) + \tanh(\tfrac{\mu\wv}{2} ),
%\tfrac{\mu\wv}{2} \tanh(\tfrac{\mu\wv}{2}) \tanh'(\tfrac{\mu\wv}{2}),
%2\sigma(\mu \wv)[1+\mu\wv\{1-\sigma(\mu\wv)\}]-1,
\end{align}

which is simply the primitive of $\Bsf$. In this case, the algorithm simply reduces to PC++ applied on (<ref>) with $\rsf = 0$ (and hence essentially stochastic gradient descent).
Of course, we could also compose with a proximal quantizer to arrive at the pair $(\Fsf\circ\tP_{\rsf}^\mu, \Bsf\circ\tP_{\rsf}^\mu)$, which effectively reduces to PC++ applied on the regularized objective in (<ref>) with a nontrivial $\rsf$.
We call this variant BNN++.

We will demonstrate in the next section that BNN++ is more desirable than BNN+ empirically.

More generally, we have the following result on designing new forward-backward quantizers:
If the forward quantizer is continuously differentiable  (with bounded support), then one can simply choose the backward quantizer as the derivative of the forward quantizer.

This follows from <Ref> since $\mathsf{P}(w) \equiv w$ is clearly proximal. Note that the BNN example does not follow from <Ref>. In <Ref>, we provide additional examples of  forward-backward quantizers based on existing methods, and we show that <Ref> consistently improves previous practices.

In summary: (1) ProxConnect++ enables us to design forward-backward quantizers with infinite many choices of $\Tsf$ and $\rsf$, (2) it also allows us to reverse engineer $\Tsf$ and $\rsf$ from existing forward-backward quantizers, which helps us to better understand existing practices, (3) with our theoretical tool, we design a new BNN++ algorithm, which enjoys immediate convergence properties.
<Ref> visualizes ProxConnect++ with a variety of forward-backward quantizers.

§.§ Binarizing Activation Functions

Might need to decide if we want to keep this part.
My current implementation: simply replacing the activation function as a linear layer and binarize it.
The correct implementation: using BCD .

So far, ProxConnect (and its many variants and special cases) are restricted to quantizing the neural network weights. In this section we show how to extend ProxConnect to quantize the activations as well, using a simple penalty function approach that “softens” the activations and treats them similarly as the weights.

Recall that we aim at solving the regularized (and transformed) problem (<ref>). When $\ell$ consists of multiple layers of neural networks, we can write the objective $\ell$ explicitly as:

\begin{align}
\label{eq:formulation}
\ell\bigg(\phi\Big(W_{k-1} \cdot \phi\big(\cdots \phi(W_2 \cdot \phi(W_1 X)) \big) \Big) \bigg),
\end{align}

where $k$ is the number of layers, $X$ is a batch of training samples, $\{W_{\kappa}\}_{\kappa=1}^{k-1}$ are the layer weights to be optimized, $\phi$ corresponds to any element-wise activation function (e.g., ReLU, tanh, sigmoid, and sign function in our case).

Such a formulation is a deeply nested mapping from inputs to outputs.  <cit.>, among many others, argue that solving (<ref>) using backprop algorithms (e.g., SGD) suffers from the vanishing gradient problem (gradients for lower layers are much smaller than those for higher layers), slow convergence and non-parallelizable over layers.
Note that gradient vanishing can be a significant problem for certain quantizers. For instance, the sign function and its approximations have zero gradient almost everywhere.

<cit.> propose another formulation by introducing the activation as a block of auxiliary variables, such that  (<ref>) can be written compactly as a constrained optimization problem:

\begin{align}
\label{eq:constrained}
\min_{\wv} ~\ell(\zv_k) ,
\st \hskip 0.1cm  \zv_\kappa = \phi(W_{\kappa-1}\zv_{\kappa-1}), \kappa=2, \ldots, k
\end{align}

where $\zv_1=X$, and each $\zv_\kappa$ can be seen as the coordinates of $X$ on an intermediate feature space.
It is clear that (<ref>) and (<ref>) are exactly equivalent.

To solve the constrained problem, <cit.> further apply quadratic penalties to approximate (<ref>):

\begin{align}
\label{eq:penalty}
\min_{\wv,\zv} ~ \ell(\zv_k) + \frac{\lambda}{2} \sum_{\kappa=2}^{k}\| \zv_\kappa - \phi(W_{\kappa-1}\zv_{\kappa-1}) \|_F^2,
\end{align}

as under mild conditions when $\mu \rightarrow \infty$ leads to the convergences of the sequence of minimizers to the minimizer of (<ref>).

As a result, with auxiliary variables for activation functions, we can apply the proximal quantizers orginally designed for binarizing weights to binarize activation functions, such that we expand (<ref>) as:

\begin{align} \label{eq:pc_activation}
\textstyle
\wv_t = \tP_{\textsf{r}}^{1/\pi_{t-1}} (\wv_t^*), \quad
\wv_{t+1}^* = \wv_t^* - \eta_t \widetilde\nabla \ell(\wv_t),\\
\zv_t = \tP_{\textsf{r}}^{1/\pi_{t-1}} (\zv_t^*), \quad
\zv_{t+1}^* = \zv_t^* - \eta_t \widetilde\nabla \ell(\zv_t),
\end{align}

where the design rules for binarizing weights directly apply.

2*[-.6ex]Model     2*[-.6ex]Dataset     2*[-.6ex]Setting
Full Precision     2cBC     1cPQ     1crPC   3cPC

GD     STE     SS     LQ     LQ     LQ     SS     Stochastic

6*[-.6ex]ViT-B     2*[-.6ex]CIFAR-10     W only     98.13%     87.97%     88.13%     85.07%     86.21%     90.13%     90.09%     90.22%

W/A     98.13%     82.99%     83.08%     81.55%     80.72%     83.11%     83.25%     83.15%

2*[-.6ex]CIFAR-100     W only     87.13%     76.35%     77.11%    72.09%     73..51%     78.13%     79.21%     78.59%

2*[-.6ex]ImageNet-1K     W only     77.91%     63.23%     63.53%     57.66%     55.31%     66.33%     66.15%    65.98%

6*[-.6ex]DeiT-T     2*[-.6ex]CIFAR-10     W only

2*[-.6ex]CIFAR-100     W only

2*[-.6ex]ImageNet-1K     W only     72.2%     66.6%     66.67%     61.23%     60.36%     67.12%    67.32%    67.11%

6*[-.6ex]DeiT-S     2*[-.6ex]CIFAR-10     W only

2*[-.6ex]CIFAR-100     W only

2*[-.6ex]ImageNet-1K     W only     79.9%     73.16%     73.25%     69.88%     68.75%     73.51%     73.28%     73.11%

6*[-.6ex]DeiT-B     2*[-.6ex]CIFAR-10     W only     98.72%     88.97%     89.03%     85.21%     86.35%     90.53%     90.16%     90.33

2*[-.6ex]CIFAR-100     W only     86.66%     75.39%     76.51%     72.11%     73.39%     78.55%     77.22%     78.31%

2*[-.6ex]ImageNet-1K     W only     81.8%     76.55%     76.63%     72.53%     70.11%     76.61%     76.72%     76.91%

§ EXPERIMENTS

In this section, we perform extensive experiments to benchmark PC++ on CNN backbone models and the recently advanced vision transformer architectures in three settings: (a) binarizing weights only (BW); (b) binarizing weights and activations (BWA), where we simply apply a similar forward-backward proximal quantizer to the activations; and (c) binarizing weights, activations, with 8-bit dot-product accumulators (BWAA) <cit.>.

Variants of ProxConnect++.

Forward Quantizer     Backward Quantizer     Algorithm

identity     identity     FP

$\tP_{Q}$     identity     BC

$\LP{\rho}$     identity     PC

$\tP_{Q}$     $\one_{[-1,1]}$     BNN

$\tP_{Q}$      $\nabla \text{SS}$     BNN+

$\text{SS}$      $\nabla \text{SS}$      BNN++

§.§ Experimental settings

Binarizing weights (BW), binarizing weights and activation (BWA) and binarizing weights, activation, with 8-bit accumulators  (BWAA) on CNN backbones. We consider the fine-tuning (FT) pipeline and the end-to-end (E2E) pipeline. We compare five variants of ProxConnect++ (BC, PC, BNN, BNN+, and BNN++) with FP, PQ, and rPC in terms of test accuracy. For the end-to-end pipeline,  we omit the results for BWAA due to training divergence and report the mean of five runs with different random seeds.

2*[-.6ex]Dataset      2*[-.6ex]Pipeline     2*[-.6ex]Task
2*[-.6ex]FP     2*[-.6ex]PQ     2*[-.6ex]rPC     5cProxConnect++

BC     PC     BNN     BNN+     BNN++

5*[-.6ex]CIFAR-10     3*[-.6ex]FT     BW     92.01%     89.94%     89.98%     90.31%     90.31%     90.35%     90.27%     90.40%

BWA     92.01%     88.79%     83.55%     89.39%     89.95%     90.01%     89.99%     90.22%

BWAA      92.01%     85.39%     81.10%     89.11%     89.21%     89.32%     89.55%    90.01%

2*[-.6ex]E2E      BW     92.01%     81.59%     81.82%     87.51%     88.05%     89.92%     89.39%     90.03%

BWA     92.01%     81.51%     81.60%     86.99%     87.26%     89.15%     89.02%     89.91%

5*[-.6ex]ImageNet-1K     3*[-.6ex]FT     BW     78.87%     66.77%     69.22%     71.35%     71.29%     71.41%     70.22%     72.33%

BWA     78.87%     56.21%     58.19%     65.99%     65.61%     66.02%     65.22%     68.03%

BWAA     78.87%     53.29%     55.28%     58.18%     59.21%     59.77%     59.10%     63.02%

2*[-.6ex]E2E      BW     78.87%     63.23%     66.39%     67.45%     67.51%     67.49%     66.99%     68.11%

BWA     78.87%     61.19%     64.17%     65.42%     65.31%     65.29%     65.98%     66.08%

Datasets: We perform image classification on CIFAR-10/100 datasets <cit.> and ImageNet-1K dataset <cit.>. Additional details on our experimental setting can be found in <Ref>.

Backbone architectures:
(1) CNNs: we evaluate CIFAR-10 classification using ResNet20 <cit.>, and ImageNet-1K with ResNet-50 <cit.>. We consider both fine-tuning and end-to-end training; (2) Vision transformers: we further evaluate our algorithm on two popular vision transformer models: ViT <cit.> and DeiT <cit.>. For ViT, we consider ViT-B model and fine-tuning task across all models[Note that we use pre-trained models provided by <cit.> on the ImageNet-21K/ImageNet-1K for fine-tuning ViT-B model on the ImageNet-1K/CIFAR datasets, respectively.]. For DeiT, we consider DeiT-B, DeiT-S, and DeiT-T, which consist of 12, 6, 3 building blocks and 768, 384 and 192 embedding dimensions, respectively; we consider fine-tuning task on ImageNet-1K pre-trained model for CIFAR datasets and end-to-end training on ImageNet-1K dataset.

For ProxConnect++, we consider the 6 variants in <Ref>. With different choices of the forward quantizer $\Fsf^{\mu}_{\rsf}$ and the backward quantizer $\Bsf^{\mu}_{\rsf}$, we include the full precision (FP) baseline and  5 binarization methods: BinaryConnect (BC) <cit.>, ProxConnect (PC) <cit.>, Binary Neural Network (BNN) <cit.>, the original BNN+ <cit.>, and the modified BNN++ with $\Fsf^{\mu}_{\rsf} = \text{SS}$. Note that we linearly increase $\mu$ in BNN++ to achieve full binarization in the end.
We also compare ProxConnect++ with the ProxQuant and reverseProxConnect baselines.

Hyperparameters: We apply the same training hyperparameters and fine-tune/end-to-end training for 100/300 epochs across all models. For binarization methods:
(1) PQ (ProxQuant): similar to <cit.>, we apply the LinearQuantizer (LQ), see (<ref>) in <Ref>, with initial $\rho_0 = 0.01$ and linearly increase to $\rho_{T} = 10$;
(2) rPC (reverseProxConnect): we use the same LQ for rPC;
(3) ProxConnect++: for PC, we apply the same LQ; for BNN+, we choose $\mu = 5$ (no need to increase $\mu$ as the forward quantizer is $\sign$); for BNN++, we choose $\mu_0 = 5$ and linearly increase to $\mu_T = 30$ to achieve binarization at the final step.

Across all the experiments with random initialization, we report the mean of three runs with different random seeds. Furthermore, we provide the complete results with error bars in <Ref>.

§.§ CNN as backbone

Binarizing weights only (BW) on CNN backbones. We compare five variants of ProxConnect++ (BC, PC, BNN, BNN+, and BNN++) with FP, PQ ,and rPC.

2*[-.6ex]Dataset     2*[-.6ex]Pipeline
2*[-.6ex]FP     2*[-.6ex]PQ     2*[-.6ex]rPC     5cProxConnect++

BC     PC     BNN     BNN+     BNN++

2*[-.6ex]CIFAR-10     Fine-tuning     92.01%     89.94%     89.98%     90.31%     90.31%     90.35%     90.27%     90.40%

End-to-end     92.01%     81.59%     81.82%     87.51%     88.05%     89.92%     87.39%     90.03%

2*[-.6ex]ImageNet-1K     Fine-tuning     78.87%     66.77%     69.22%     71.35%     71.29%     71.41%     70.22%     72.33%

End-to-end     78.87%     63.23%     66.39%     67.45%     67.51%     67.49%     66.99%     68.11%

We first compare PC++ against baseline methods on various tasks employing CNNs:

* Binarizing weights only (BW), where we simply binarize the weights and keep the other components (i.e., activations and accumulations) in full precision.

* Binarizing weights and activations (BWA), while keeping accumulation in full precision. Similar to the weights, we apply the same forward-backward proximal quantizer to binarize activations.

* Binarizing weights, activations, with 8-bit accumulators (BWAA). BWAA is more desirable in certain cases where the network bandwidth is narrow, e.g., in homomorphic encryption. To achieve BWAA, in addition to quantizing the weights and activations,

we follow the implementation of WrapNet <cit.> and quantize the accumulation of each layer with an additional cyclic function. In practice, we find that with 1-bit weights and activations, the lowest bits we can successfully employ to quantize accumulation is 8, while any smaller choice would raise a high overflow rate and cause the network to diverge. Moreover, BWAA highly relies on a good initialization and cannot be successfully trained end-to-end in our evaluation (and hence omitted).

Note that for the fine-tuning pipeline, we initialize the model with their corresponding pre-trained full precision weights. For the end-to-end pipeline, we utilize random initialization.
We report our results in <Ref> and observe:
(1) the PC family outperforms baseline methods (i.e., PQ and rPC), and achieves competitive performance on both small and larger scale datasets; (2) BNN++ performs consistently better and is more desirable among the five variants of PC++, especially on BWA and BWAA tasks. Its advantage over BNN+ further validates our theoretical guidance.

Binarizing weights only: we first consider the relatively easy case, where we only binarize the weights, and keep the other components (i.e., activation and accumulation) in full precision. We report our results in <Ref> and observe: (1) the PC family outperforms baseline methods (i.e., PQ and rPC), and achieves competitive performance on both small and larger scale datasets; (2) BNN++ performs consistently better and is more desirable among the five variations of PC++. Its advantage over BNN+ further validates our theory guidance.

An attempt on full quantization: next we consider more difficult tasks, binarizing both weight and activation (BWA) and full quantization (FQ). FQ is more desirable in certain cases where the network bandwidth is narrow, e.g., in homomorphic encryption. To achieve FQ, in addition to quantizing the input-weights dot product, we follow the implementation of WrapNet <cit.> and quantize the accumulation of each layer with an additional cyclic function. In practice, we find that with 1-bit weights and activation, the most extreme case is to quantize accumulation to 8-bits. We observe any choice smaller than 8 would raise a high overflow rate, which would cause the network to diverge. Notably, FQ highly relies on a good initialization and cannot be successfully trained end-to-end in our evaluation. Thus we perform fine-tuning and report the results in <Ref>. We observe that BNN++ is even more competitive with more difficult tasks and outperforms other baselines by a big margin.

Our results on binarizing vision transformers (binarizing weights only). We compare five variants of ProxConnect++ (BC, PC, BNN, BNN+, and BNN++) with FP, PQ, and rPC. End-to-end training tasks are marked as bold (i.e., ImageNet-1K for DeiT-T/S/B), where the results are the mean of five runs with different random seeds.

2*[-.6ex]Model     2*[-.6ex]Dataset
2*[-.6ex]FP     2*[-.6ex]PQ     2*[-.6ex]rPC     5cProxConnect++

BC     PC     BNN     BNN+     BNN++

3*[-.6ex]ViT-B     CIFAR-10     98.13%     85.06%     86.22%     87.97%     90.12%     89.08%     88.12%     90.24%

CIFAR-100     87.14%     72.07%     73.52%     76.35%     78.13%     77.23%     77.10%     79.22%

ImageNet-1K     77.91%     57.65%     55.33%     63.24%     66.33%     65.31%     63.55%     66.33%

3*[-.6ex]DeiT-T     CIFAR-10     94.86%     82.76%     82.25%     83.10%     85.15%     86.12%     85.91%     86.41%

CIFAR-100     72.37%     54.55%     55.66%     59.65%     60.15%     60.06%     59.77%     60.33%

ImageNet-1K     72.20%     61.23%     60.35%     63.22%     66.15%     65.00%     66.67%     67.34%

3*[-.6ex]DeiT-S     CIFAR-10     95.10%     81.67%     80.23%     84.85%     85.13%     85.09%     85.16%     86.19%

CIFAR-100     73.19%     45.55%     46.66%     60.12%     61.59%     60.55%     60.17%     62.98%

ImageNet-1K     79.91%     69.87%     68.74%     73.16%     73.51%     73.77%    73.23%     73.53%

3*[-.6ex]DeiT-B     CIFAR-10     98.72%     85.22%     86.35%     88.95%     90.53%     90.21%     89.03%     90.67%

CIFAR-100     86.65%     72.11%     73.40%     75.40%     78.55%     76.22%     76.51%     78.30%

ImageNet-1K     81.81%     72.54%     70.11%     76.55%     76.61%     75.60%     76.63%     76.74%

Our results on binarizing vision transformers (binarizing weights only). We compare five variants of ProxConnect++ (BC, PC, BNN, BNN+, and BNN++) with FP, PQ, and rPC. End-to-end training tasks are marked as bold (i.e., ImageNet-1K for DeiT-T/S/B), where the results are the mean of five runs with different random seeds.

2*[-.6ex]Model     2*[-.6ex]Dataset
2*[-.6ex]FP     2*[-.6ex]PQ     2*[-.6ex]rPC     5cProxConnect++

BC     PC     BNN     BNN+     BNN++

3*[-.6ex]ViT-B     CIFAR-10     98.13%     85.07%     86.21%     87.97%     90.13%     89.07%     88.13%     90.22%

CIFAR-100     87.13%     72.09%     73.51%     76.35%     78.13%     77.22%     77.11%     79.21%

ImageNet-1K     77.91%     57.66%     55.31%     63.23%     66.33%     65.33%     63.53%     66.33%

3*[-.6ex]DeiT-T     CIFAR-10     94.85%     82.77%     82.25%     83.09%     85.15%     86.11%     85.92%     86.40%

CIFAR-100     72.37%     54.53%     55.67%     59.66%     60.15%     60.04%     59.78%     60.33%

ImageNet-1K     72.20%     61.23%     60.36%     63.23%     66.15%     64.99%     66.67%     67.32%

3*[-.6ex]DeiT-S     CIFAR-10     95.09%     81.66%     80.23%     84.85%     85.13%     85.09%     85.16%     86.17%

CIFAR-100     73.19%     45.57%     46.66%     60.11%     61.59%     60.55%     60.15%     62.99%

ImageNet-1K     79.90%     69.88%     68.75%     73.16%     73.51%     73.77%    73.25%     73.51%

3*[-.6ex]DeiT-B     CIFAR-10     98.72%     85.21%     86.35%     88.97%     90.53%     90.21%     89.03%     90.66%

CIFAR-100     86.66%     72.11%     73.39%     75.39%     78.55%     76.22%     76.51%     78.29%

ImageNet-1K     81.80%     72.53%     70.11%     76.55%     76.61%     75.59%     76.63%     76.72%

Results on binarizing vision transformers (BW, BWA, and BWAA) on DeiT-T. We compare 5 variants of ProxConnect++ (BC, PC, BNN, BNN+, and BNN++) with FP, PQ, and rPC. End-to-end training tasks are marked as bold (i.e., ImageNet-1K), where we omit the results for BWAA due to training divergence and the reported results are the mean of five runs with different random seeds.

2*[-.6ex]Dataset       2*[-.6ex]Task
2*[-.6ex]FP     2*[-.6ex]PQ     2*[-.6ex]rPC     5cProxConnect++

BC     PC     BNN     BNN+     BNN++

3*[-.6ex]CIFAR-10      BW     94.85%     82.76%     82.25%     83.10%
85.15%     86.12%     85.91%     86.41%

BWA     94.85%     82.56%     82.02%     82.89%
85.01%     85.99%     85.66%     86.12%

BWAA     94.85%     81.34%     80.97%     82.08%     84.31%     84.87%     84.72%     85.31%

3*[-.6ex]CIFAR-100      BW     72.37%     54.55%     55.66%     59.65%     60.15%     60.06%     59.77%     60.33%

BWA     72.37%     53.77%     54.98%     59.21%     59.71%     59.66%     59.12%     59.85%

BWAA     72.37%     52.15%     54.36%     58.15%     59.01%     58.72%     58.15%     59.06%

2*[-.6ex]ImageNet-1K    BW     72.20%     61.23%     60.35%     63.23%     66.15%     65.00%     66.67%     67.34%

BWA     72.20%     60.01%     58.77%     62.13%     65.29%     63.75%     65.29%     65.65%

Comparison between Full Precision (FP) model, BNN++, and Post-training Binarization (PTB) on the fine-tuning task on CIFAR-10.

Ablation study on the effect of the scaling factor, normalization, pre-training, and knowledge distillation. Experiments are performed on CIFAR-10 with ViT-B.

Method     Scaling     Normalization     Pre-train     KD      Accuracy

4*[-.6ex]PC                     0.10%

4*[-.6ex]BNN++                      1.50%

§.§ Vision transformer as backbone

Next, we perform similar experiments on the three tasks on vision transformers.

Implementation on vision transformers: While network binarization is popular for CNNs, its application for vision transformers is still rare[Notably, <cit.> also consider binarizing vision transformers, which we compare our implementation details and experimental results against in <Ref>.  ]. Here we apply four protocols for implementation:

(1) We keep the mean $s_{n}$ of full precision weights $\wv_{n}^*$  for each layer $n$ as a scaling factor (can be thus absorbed into $\Fsf_{\rsf}^{\mu_t}$) for the binary weights $\wv_{n}$. Such an approach keeps the range of $\wv_{n}^*$ during binarization and significantly reduces training difficulty  without additional computation.

(2) For binarized vision transformer models, LayerNorm is important to avoid gradient explosion. Thus, we add one more LayerNorm layer at the end of each attention block.

(3) When fine-tuning a pre-trained model (full precision), the binarized vision transformer usually suffers from a bad initialization. Thus, a few epochs of pre-training on the binarized vision transformer is extremely helpful and can make fine-tuning much more efficient and effective.

(4) We apply the knowledge distillation technique in BiBERT <cit.> to boost the performance. We use full precision pre-trained models as the teacher model.

Main Results:
We report the main results of binarizing vision transformers in <Ref> (BW) and <Ref> (BW, BWA, BWAA), where we compare ProxConnect++ algorithms with the FP, PQ, and rPC baselines on fine-tuning and end-to-end training tasks. We observe that:
(1) ProxConnect++ variants generally outperform PQ and rPC and are able to binarize vision transformers with less than $10\%$ accuracy degradation on the BW task. In particular, for end-to-end training, the best performing ProxConnect++ algorithms achieve $\approx 5\%$ accuracy drop;
(2) Among the five variants, we confirm BNN++ is also generally better overall for vision transformers. This provides evidence that our <Ref> allows practitioners to easily design many and choose the one that performs best empirically;
(3) With a clear underlying optimization objective, BNN++ again outperforms BNN+ across all tasks, which empirically verifies our theoretical findings on vision transformers; (4) In general, we find that weight binarization achieves about 30x reduction in memory footprint, e.g., from 450 MB to 15 MB for ViT-B.

Results of binarizing different components (blocks) of ViT-B architecture on CIFAR-10. Warmer color indicates significant accuracy degradation after binarization.

Results of binarizing different components (blocks) of ViT-B architecture on CIFAR-10. Warmer color indicates significant accuracy degradation after binarization.

Ablation Studies:
We provide further ablation studies to gain more insights and verify our binarization protocols for vision transformers.

Post-training Binarization: in  <Ref>, we verify the difference between PTB (post-training binarization) and BAT (binarization-aware training) on the fine-tuning task on CIFAR-10 across different models. Note that we use BNN++ as a demonstration of BAT. We observe that without optimization during fine-tuning, the PTB approach fails in general, thus confirming the importance of considering BAT for vision transformers.

Effect of binarizing protocols: here we show the effect of the four binarizing protocols mentioned at the beginning, including scaling the binarized weights using the mean of full precision weights (scaling), adding additional LayerNorm layers (normalization), BAT on the full precision pre-trained models (pre-train) and knowledge distillation. We report the results in <Ref> and confirm that each protocol is essential to binarize vision transformers successfully.

Which block should one binarize: lastly, we visualize the sensitivity of each building block to binarization in vision transformers (i.e., ViT-B) on CIFAR-10 in  <Ref>. We observe that binarizing blocks near the head and the tail of the architecture causes a significant accuracy drop.

§ CONCLUSION

In this work we study the popular approximate gradient approach in neural network binarization.
By generalizing ProxConnect and proposing PC++, we provide a principled way to understand forward-backward quantizers and cover most existing binarization techniques as special cases. Furthermore, PC++ enables us to easily design the desired quantizers (e.g., the new BNN++) with automatic theoretical guarantees. We apply PC++ to CNNs and vision transformers and compare its variants in extensive experiments. We confirm empirically that PC++ overall achieves competitive results, whereas BNN++ is generally more desirable.

§ BROADER IMPACTS AND LIMITATIONS

We anticipate our work to further enable training and deploying advanced machine learning models to resource limited devices and environments, and help reducing energy consumption and carbon footprint at large. We do not foresee any direct negative societal impact.
One limitation we hope to address in the future is to build a theoretical framework that will allow practitioners to quickly evaluate different forward-backward quantizers for a variety of applications.

We thank the reviewers and the area chair for thoughtful comments that have improved our final draft.
We thank Arash Ardakani, Ali Mosleh and Marzieh Tahaei for their early participation in this project.
YY gratefully acknowledges NSERC and CIFAR for funding support.

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

notearXiv e-prints
titleGradient L1 Regularization for Quantization Robustness

booktitleInternational Conference on Learning Representations
titleProxQuant: Quantized Neural Networks via Proximal Operators

booktitleAdvances in Neural Information Processing Systems
titlePost training 4-bit quantization of convolutional networks for rapid-deployment

notearXiv preprint arXiv:1308.3432
titleEstimating or propagating gradients through stochastic neurons for conditional computation

booktitleIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops
titleLSQ+: Improving low-bit quantization through learnable offsets and better initialization

notearXiv preprint arXiv:2110.06821
titleLeveraging redundancy in attention with Reuse Transformers

journaltitlearXiv preprint arXiv:2304.01373
titlePythia: A suite for analyzing large language models across training and scaling

journaltitleAdvances in neural information processing systems
titleLanguage models are few-shot learners

booktitleProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
titleZeroq: A novel zero shot quantization framework

notearXiv preprint arXiv:1805.06085
titlePact: Parameterized clipping activation for quantized neural networks

booktitleAdvances in Neural Information Processing Systems
titleBinaryconnect: Training deep neural networks with binary weights during propagations

booktitleNeurIPS Workshop on Energy Efficient Machine Learning and Cognitive Computing
titleRegularized binary network training

booktitleAdvances in Neural Information Processing Systems
titleDemystifying and Generalizing BinaryConnect

booktitleInternational Conference on Learning Representations
titleAn image is worth 16x16 words: Transformers for image recognition at scale

notearXiv preprint arXiv:1902.08153
titleLearned step size quantization

booktitleAdvances in Neural Information Processing Systems
titleIs Integer Arithmetic Enough for Deep Learning Training?

booktitleProceedings of the 32nd International Conference on Machine Learning (ICML)
titleDeep Learning with Limited Numerical Precision

booktitleProceedings of the IEEE conference on computer vision and pattern recognition
titleDeep residual learning for image recognition

notearXiv preprint arXiv:2211.07091
titleBiViT: Extremely Compressed Binary Vision Transformer

journaltitleAdvances in neural information processing systems
titleLatent weights do not exist: Rethinking binarized neural network optimization

notearXiv preprint arXiv:2201.00043
titleMulti-Dimensional Model Compression of Vision Transformer

booktitleAdvances in Neural Information Processing Systems
titleBinarized neural networks

booktitleIEEE Conference on Computer Vision and Pattern Recognition (CVPR)
titleQuantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

notearXiv preprint arXiv:1903.08066
titleTrained quantization thresholds for accurate and efficient fixed-point inference of deep neural networks

booktitleProceedings of the IEEE Conference on Computer Vision and Pattern Recognition
titleLearning to quantize deep networks by optimizing quantization intervals with task loss

booktitleProceedings of the IEEE/CVF conference on computer vision and pattern recognition
titleImproving accuracy of binary neural networks using unbalanced activation distribution

University of Toronto

titleLearning multiple layers of features from tiny images

booktitleAdvances in Neural Information Processing Systems
titleImagenet classification with deep convolutional neural networks

booktitleInternational Conference on Learning Representations
titleBRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction

notearXiv preprint arXiv:2201.07703
titleQ-ViT: Fully Differentiable Quantization for Vision Transformer

journaltitleAdvances in neural information processing systems
titleRotated binary neural network

notearXiv preprint arXiv:2111.13824
titleFQ-ViT: Fully Quantized Vision Transformer without Retraining

booktitleProceedings of the AAAI Conference on Artificial Intelligence
titleSA-BNN: State-aware binary neural network

booktitleProceedings of the IEEE/CVF International Conference on Computer Vision
titleSwin transformer: Hierarchical vision transformer using shifted windows

booktitleInternational conference on machine learning
titleHow do adam and training strategies help bnns optimization

booktitleComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16
titleReactnet: Towards precise binary neural network with generalized activation functions

booktitleProceedings of the European conference on computer vision (ECCV)
titleBi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm

journaltitleAdvances in Neural Information Processing Systems
titlePost-training quantization for vision transformer

booktitleInternational Conference on Learning Representations (ICLR)
titleRelaxed Quantization for Discretized Neural Networks

booktitleInternational Conference on Learning Representations
titleTraining binary neural networks with real-to-binary convolutions

booktitleInternational Conference on Machine Learning
titleUp or down? Adaptive rounding for post-training quantization

booktitleProceedings of the IEEE International Conference on Computer Vision
titleData-free quantization through weight equalization and bias correction

booktitleInternational Conference on Learning Representations ICLR 2021
titleWrapnet: Neural net inference with ultra-low-precision arithmetic

booktitleAdvances in Neural Information Processing Systems
titleIA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers

notearXiv preprint arXiv:2203.06390
titleBibert: Accurate fully binarized bert

booktitleProceedings of the IEEE/CVF conference on computer vision and pattern recognition
titleForward and backward information retention for accurate binary neural networks

booktitleAdvances in Neural Information Processing Systems
titleDynamicvit: Efficient vision transformers with dynamic token sparsification

booktitleEuropean conference on computer vision
titleXnor-net: Imagenet classification using binary convolutional neural networks

journaltitleAdvances in Neural Information Processing Systems
titleCombiner: Full attention transformer with sparse computation cost

titleVariational Analysis

notearXiv preprint arXiv:2106.11297
titleTokenLearner: What Can 8 Learned Tokens Do for Images and Videos?

booktitleInternational Conference on Machine Learning
titleTraining data-efficient image transformers & distillation through attention

booktitleEuropean conference on computer vision
titleAdabin: Improving binary neural networks with adaptive binary sets

booktitleInternational Conference on Machine Learning
titleTowards accurate post-training network quantization via bit-split and stitching

notearXiv preprint arXiv:2201.08050
titleTerViT: An Efficient Ternary Vision Transformer

booktitleProceedings of the IEEE/CVF international conference on computer vision
titleRecu: Reviving the dead weights in binary neural networks

booktitleIEEE/CVF Conference on Computer Vision and Pattern Recognition
titleMiniViT: Compressing Vision Transformers with Weight Multiplexing

booktitleInternational Conference on Learning Representations
titleLinear symmetric quantization of neural networks for low-precision integer hardware

§ MORE ON PROXIMAL QUANTIZERS

<cit.> gave a complete characterization of the proximal quantizer $\tP_{\rsf}$: a (multi-valued) mapping $\tP$ is a proximal quantizer (of some underlying regularizer $\rsf$) iff it is monotone, compact-valued and with a closed graph.

We now give a few examples to illustrate the ubiquity of proximal quantizers, as well as the generality of PC:

* Identity function: apparently, choosing $\tP_{\textsf{r}}^{\mu_{t}}$ as the identity function recovers the full precision training.

* $\tP_{\textsf{r}}^{\mu_{t}} = \tP_{Q}$: as $Q = \{\pm1\}$, this choice recovers exactly BC in (<ref>).

* $\tP_{\textsf{r}}^{\mu_{t}} = \LP{\rho}$: This is the general piecewise linear quantizer designed by
<cit.>. Recall that $Q = \{q_k\}_{k=1}^{2}$, where $q_1 = -1, q_2 = +1$, such that $p_2 =0 $ is the middle point. By introducing two parameters $\rho, \varrho \geq 0$, we can define two shifts:

q_1^- = q_1, q_1^+ = p_2 ∧(q_1 + ρ)

q_2^-= p_2 ∨(q_2 - ρ), q_2^+ = q_2

p_2^- = q_1 ∨(p_2 - ϱ), p_2^+ = q_2 ∧(p_2 + ϱ).

Then, we define $\LP{\rho}$ as the piece-wise linear map (that simply connects the points by straight lines):

\begin{align}
\scriptsize
\label{eq:prox-pl}
\LP{\rho}(w^*) \!=\! \begin{cases}
q_1, & \mbox{if } q_1^- \leq w^* \leq q_1^+ \\
q_1 + (w^*-q_1^+) \tfrac{p_2^- \!-\! q_1}{p_2 - q_1^+}, & \mbox{if } q_1^+ \leq w^* < p_2 \\
p_2^+ + (w^*\!-\!p_2) \tfrac{q_2 - p_2^+}{q_2^- - p_2} & \mbox{if } p_2 < w^* \leq q_2^-\\
q_2, & \mbox{if } q_2^- \leq w^* \leq q_2^+
\end{cases}
\end{align}

For the middle points, $\LP{\rho}(w^*)$ can be regarded as the intermediate state between the identity function and $\tP_{Q}$ such that, where $\LP{\rho}(w^*)$ may take any value within the two limits. Note that $\rho$ controls the discretization vicinity, such that in practice, $\rho$ is linearly increased over time to fulfill binary weights in the end. We visualize examples of $\LP{\rho}(w^*)$ in  <Ref>.

$\rho=0, \varrho = 0.2$.

$\rho=\varrho = 0.2$.

$\rho=0.2, \varrho = 0$.

Different instantiations of the proximal map $\LP{\rho}$ in (<ref>) for $Q = \{-1, 1\}$.

§ RELATED WORKS

Vision Transformer. In computer vision,  vision transformers have become one of the most popular backbone architectures.  <cit.> is the first to modify the transformer model to enable images as input, namely the ViT model. Specifically, <cit.> translates an image to a sequence of flattened image patches as input, and applies a self-attention mechanism to retrieve patch-wise information in the feature representation.
<cit.> further equips ViT with knowledge distillation and proposes DeiT that generalizes well on smaller models and datasets.
<cit.> further proposes Swin as a hierarchical vision transformer that computes representation with shifted windows.

In vision transformers, the main computation overhead is the multi-head attention layer, whose cost is quadratic with the length of the image patches. As a result, such models are in general expensive to train. To reduce the computational cost, different compression techniques have been explored. For instance, <cit.> performs dynamic pruning for less important patches; <cit.> reuses attention scores computed for one layer in multiple building blocks; <cit.> applies multi-dimensional model compression.  In this paper, we focus on an alternative approach, namely network quantization.

Network Quantization. We consider two possible scenarios of network quantization:

(1) Post-training Quantization:
We first discuss the easier post-training quantization methods. Such approaches usually quantize the full-precision pre-trained model and directly apply it for inference.
Post-training quantization  is widely used in CNNs <cit.>.

<cit.> is the first to explore PTQ for vision transformers. It optimizes the quantization intervals and considers ranking information in the loss function. However, it only considers quantization to a 6-bit model without severe performance degradation. For lower-bit quantization, it is essential to leverage training.

(2) Quantization-Aware Training (QAT): different from post-training quantization, quantization-aware training leverage quantization during pre-training or fine-tuning. Thus it can be formulated as an optimization problem for learning the optimal quantized weights  <cit.>.   Compared with PTQ, QAT can obtain less accuracy drop in low-bit quantization compared to the full-precision model.
<cit.> and <cit.> demonstrate that QAT requires a unique design to quantize vision transformers and it is possible to perform quantization to 3 bit without severe performance degradation. <cit.> further performs binarization with softmax-aware binarization and information preservation.

Binarization Techniques:
(1) Here we summarize existing binarization approaches that can be either justified or improved by PC++:
Some existing implementations <cit.> set the forward quantizer as $\text{sign}$ and design the backward quantizer in an ad hoc fashion (based on graphic approximation of the sign function), e.g., <cit.> applies a piece-wise polynomial approximation; <cit.> improves <cit.> with a dynamic polynomial approximation; <cit.> designs a dynamic error decay estimator based on the $\text{tanh}$ function. These methods cannot be justified with PC++, but could be improved by designing new forward quantizers using our <Ref>. We discuss these new variants in <Ref> and compare them with our methods. Other implementations <cit.> applies shift transformation on both forward and backward quantizers, which belongs to the PC++ family.

(2) Architecture design that can be further integrated into our PC++ as future work: <cit.> designs a rectified clamp unit to address "dead weights"; <cit.> applies the absolute mean of weights and activations; <cit.> uses real-to-binary attention matching and data-driven channel re-scaling; <cit.> proposes a shifted activation function.

(3) Optimization refinement, again could be integrated into our framework as future work: <cit.> utilizes state-aware gradient state; <cit.> provides a weight decay scheme; and <cit.> proposes Bop, a new optimizer for BNNs.

§ ADDITIONAL THEORETICAL RESULTS

We first recall the decomposition in (<ref>):

\begin{align}
\tag{19}
\Fsf^{\mu}_{\rsf} := \Tsf \circ \tP_{\rsf}^{\mu}, \quad
\Bsf^{\mu}_{\rsf} := \Tsf' \circ \tP_{\rsf}^{\mu}.
\end{align}

Suppose first that $(\Fsf, \Bsf)$ satisfies the above decomposition. Clearly, both $\Fsf$ and $\Bsf$ are functions of $\tP = \tP_{\rsf}^\mu$. Moreover,

\begin{align}
\frac{\Fsf'(\omega)}{\Bsf(\omega)} = \frac{{\tP_{\rsf}^\mu}'(\omega) \cdot \Tsf'\circ \tP_{\rsf}^\mu}{\Tsf'\circ \tP_{\rsf}^\mu} = {\tP_{\rsf}^\mu}'(\omega)
\end{align}

and thus

\begin{align}
\int_{-\infty}^w \frac{1}{\Bsf(\omega)} \rmd \Fsf(\omega) = \tP_{\rsf}^\mu(w) - \tP_{\rsf}^\mu(-\infty),
\end{align}

which is clearly proximal.

Conversely, let $\tP(w) := \int_{-\infty}^w \frac{1}{\Bsf(\omega)} \rmd \Fsf(\omega)$ be proximal. Taking (generalized) derivative we obtain

\begin{align}
\tP'(\omega) = \frac{\Fsf'(\omega)}{\Bsf(\omega)}.
\end{align}

Since $\Bsf$ is a function of $\tP$, say $\Bsf = \Tsf' \circ \tP$, performing integration we obtain

\begin{align}
\Fsf = \Tsf \circ \tP,
\end{align}

up to some immaterial constant (that can be absorbed into $\Tsf$). Thus, $(\Fsf, \Bsf)$ satisfies the decomposition (<ref>).

The following convergence guarantee for PC++ follows directly from the results in <cit.>:
Fix any $\wv$, the iterates in (<ref>) satisfy:

\begin{align}
\label{eq:pcb}
\sum_{\tau=s}^t \eta_\tau [\langle\wv_\tau - \wv, \widetilde\nabla\ell(\Tsf\wv_\tau)\rangle + \rsf(\wv_{\tau}) - \rsf(\wv) ]\leq %\tfrac{\Lambda_s^t}{2}\|\wv\|_2^2 +
\Delta_{s-1}(\wv) - \Delta_t(\wv) +
\sum_{\tau=s}^t \Delta_{\tau}(\wv_{\tau}),
\end{align}

:= \rsf_{\tau}(\wv) - \rsf_{\tau}(\wv_{\tau+1})  - \inner{\wv-\wv_{\tau+1}}{\wv_{\tau+1}^*}$ is the Bregman divergence induced by the (possibly nonconvex) function $\rsf_\tau(\wv) := \mu_{\tau+1}\cdot \rsf(\wv) + \tfrac{1}{2}\|\wv\|_2^2$. (Recall that $\mu_t := 1 + \sum_{\tau=1}^{t-1} \eta_\tau$.)

The summand on the left-hand side of (<ref>) is related to the duality gap, which is
a natural measure of stationarity for the nonconvex problem (<ref>). Indeed, it reduces to the familiar ones when convexity is present:

For convex $\ell\circ\Tsf$ and any $\wv$, the iterates in (<ref>) satisfy:

\begin{align}
\label{eq:pcc}
\!\!\min_{\tau=s, \ldots, t} ~ \EE [f(\wv_\tau) \!-\! f(\wv)]\leq
\tfrac{1}{\sum_{\tau=s}^t\eta_\tau} \cdot \EE\big[
%\tfrac{\Lambda_s^t}{2}\|\wv\|_2^2 \!+\!
\Delta_{s-1}(\wv) \!-\! \Delta_t(\wv) \!+\!
\sum\nolimits_{\tau=s}^t \Delta_{\tau}(\wv_{\tau}) \big].
\end{align}

If $\rsf$ is also convex, then

\begin{align} \label{eq:pcc2}
\min_{\tau=s, \ldots, t} ~ \EE [f(\wv_\tau) \!-\! f(\wv)]\leq
\tfrac{1}{\sum_{\tau=s}^t\eta_\tau} \cdot \EE\big[
%\tfrac{\Lambda_s^t}{2}\|\wv\|_2^2 \!+\!
\Delta_{s-1}(\wv) \!+\!
\sum\nolimits_{\tau=s}^t \tfrac{\eta_\tau^2}{2} \|\widetilde\nabla\ell(\wv_\tau)\|_2^2 \big],
\end{align}

\begin{align} \label{eq:pcc3}
\EE\big[f(\bar \wv_t) \!-\! f(\wv)\big]\leq
\tfrac{1}{\sum_{\tau=s}^t\eta_\tau} \cdot \EE\big[
%\tfrac{\Lambda_s^t}{2}\|\wv\|_2^2 \!+\!
\Delta_{s-1}(\wv) \!+\!
\sum\nolimits_{\tau=s}^t \tfrac{\eta_\tau^2}{2} \|\widetilde\nabla\ell(\wv_\tau)\|_2^2 \big],
\end{align}

where $\wv_t = \tfrac{\sum_{\tau=s}^t \eta_\tau \wv_\tau}{\sum_{\tau=s}^t \eta_\tau}$, and $f := \ell\circ\Tsf + \rsf$ is the regularized and transformed objective.

The right-hand sides of (<ref>) and (<ref>) diminish iff $\eta_t\to 0$ and $\sum_t\eta_t = \infty$ (assuming boundedness of the stochastic gradient). We note some trade-off in choosing the step size $\eta_\tau$: both the numerator and denominator of the right-hand sides of (<ref>) and (<ref>) are increasing w.r.t. $\eta_\tau$.
The same conclusion can be drawn for (<ref>) and (<ref>), where $\Delta_\tau$ also depends on $\eta_\tau$ (through the accumulated magnitude of $\wv_{\tau+1}^*$).

§ ADDITIONAL EXPERIMENTAL SETTINGS

Hardware and package: All experiments were run on a GPU cluster with  GPUs. The platform we use is PyTorch. Specifically, we apply ViT and DeiT models implemented in Pytorch Image Models (timm) [<https://timm.fast.ai/>].

Pre-trained models: In this work, we applied pre-trained full precision models for fine-tuning tasks. Here we specify the links to the models we used (note that we choose patch size equal to 16 across all models):

* ViT-B (ImageNet-1K): <https://storage.googleapis.com/vit_models/augreg/B_16-i21k-300ep-lr_0.001-aug_medium1-wd_0.1-do_0.0-sd_0.0–imagenet2012-steps_20k-lr_0.01-res_224.npz>;

* ViT-B (ImageNet-21K):

* DeiT-T (ImageNet-1K):

* DeiT-S (ImageNet-1K):

* DeiT-B (ImageNet-1K):

§ COMPARISON WITH BIVIT

We first introduce our target binarization architecture in <cit.>.
Vision transformers share a similar architecture design as the transformers in NLP tasks. They first reshape an image $\xv$ with resolution $(H,W)$ and $C$ channels to a sequence of flattened patches $\xv_p \in \mathbb{R}^{N \times (P^2 C)}$ with resolution $(P,P)$, and then map it to $D$ dimension with a linear projection $\mathbf{E}$:

\begin{align}
\mathbf{a}_0 = [\xv_{\text{class}};\xv_p^1\mathbf{E};...;\xv_p^N\mathbf{E}] + \mathbf{E}_{pos},
\end{align}

where $\mathbf{a}_0$ is the patch embeddings, which is the input to $L$ identical transformer blocks.
A single block of the model consists of a multi-head self-attention (MSA) layer and a fully connected feed-forward (MLP) layer as main elements. Specifically, the MSA layer is a standard $\mathbf{qkv}$ self-attention with $k$ self-attention operations. For the $l$-th block:

\begin{align}
&\!\!\![\mathbf{q}_j,\mathbf{k}_j,\mathbf{v}_j] = \mathbf{a}_{l-1}\mathbf{U}_{qkv}, \mathbf{U}_{qkv} \in \mathbb{R}^{D \times 3D_h} \\
&\!\!\! A_j = \softmax (\mathbf{q}_j\mathbf{k}_j^{\top}/\sqrt{D_h}), A_j \in \mathbb{R}^{N\times N}\\
&\!\!\!\text{MSA}(\mathbf{a}_j) \!=\! [A_1\mathbf{v}_1; ... ; A_k \mathbf{v}_k] \mathbf{U}_{msa}, \mathbf{U}_{msa} \in \mathbb{R}^{kD_h \!\times\! D}
\end{align}

MLP layer is a classification head with two linear layers and GELU nonlinearity.
Combined with LayerNorm (LN) normalization, the $l$-th block can be represented as:

\begin{align}
\mathbf{a}_{l-1/2}  &= \text{MSA}(\text{LN}(\mathbf{a}_{l-1}))+ \mathbf{a}_{l-1} \\
\mathbf{a}_l  &= \text{MLP}(\text{LN}(\mathbf{a}_{l-1/2}))+ \mathbf{a}_{l-1/2}.
\end{align}

§.§ ProxConnect++ on Vision Transformers

Now we are ready to apply ProxConnect++ to vision transformers. We first present the complete algorithm for the universal model architecture in <Ref>.

Training set $\mathcal{D}_{tr}$, a model initialized with full precision weights $\{\wv_{n}^*\}_{n=1}^N$,
the objective $\ell$, initial learning rate $\eta_0$, initial smoothing parameter $\mu_0$, initial forward quantizer $\Fsf_{\rsf}^{\mu_0}$, initial backward quantizer $\Bsf_{\rsf}^{\mu_0}$.

$t =1,...,T$
update $\eta_t, \mu_t, \Fsf_{\rsf}^{\mu_t},\Bsf_{\rsf}^{\mu_t}$

$\wv_{n} \gets \Fsf_{\rsf}^{\mu_t}(\wv_{n}^*)$
*forward pass

calculate intermediate output: $a_{n} \gets a_{n-1}^{\top}\wv_{n} $

calculate loss $\mathcal{L}=\ell(\wv_1, \ldots, \wv_N; \mathcal{D}_{tr})$

$\wv_{n}^* \gets \wv_{n}^* - \eta_t \Bsf_{\rsf}^{\mu_t} \dfrac{\partial a_{n}}{\partial \wv_{n}} $

Binarized weights $\{\wv_{n}\}_{n=1}^{N}$
ProxConnect++ training

When applying ProxConnect++ to vision transformers, we keep the patch embedding in full precision and binarize all the transformer blocks. In particular, binarizing $\mathbf{U}_{qkv}, \mathbf{U}_{msa}$ reduces computation significantly during matrix multiplication.

<cit.> propose BiViT, which considers the same binarization task on vision transformers (specifically, Swin-T and Nest-T). <cit.> follow a different implementation with softmax-aware binarization and information preservation. To fairly compare with this work, we follow the same setting and run PC++ on Swin-T and NesT-T on ImageNet-1K. We observe that BNN++ achieves 71.3% Top-1 accuracy (BiViT:70.8%) and 69.3% Top-1 accuracy (BiViT:68.7%) respectively on Swin-T and NesT-T. Note that BiViT simply applies BNN as the main algorithm and may be further improved with PC++ algorithms.

§ ADDITIONAL FORWARD-BACKWARD QUANTIZERS

Forward and backward pass for 6 additional ProxConnect++ algorithms.

Results for additional ProxConnect++ algorithms on binarizing vision transformers (binarizing weights only), where the results are the mean of five runs with different random seeds.

2*[-.6ex]Model     2*[-.6ex]Dataset
2*[-.6ex]FP     7cProxConnect++

Bi-Real     R-BNN     Poly+     EDE     EDE+      ReAct     BNN++

3*[-.6ex]DeiT-T     CIFAR-10     94.85%     84.11%     84.54%      85.31%     84.99%     85.57%      85.35%     86.41%

CIFAR-100     72.37%     59.01%     59.02%     60.00%     59.32%     60.04%      60.11%     60.33%

ImageNet-1K     72.20%     64.55%     64.59%     64.97%     64.28%     65.01%      65.37%     67.34%

3*[-.6ex]DeiT-S     CIFAR-10     95.09%     85.01%     86.07%     84.99%     85.37%     85.33%     85.91%     86.19%

CIFAR-100     73.19%     59.66%     59.75%     60.14%     60.09%     61.17%     61.09%     62.98%

ImageNet-1K     79.90%     70.51%     70.87%      71.36%     70.66%     72.99%     72.53%     73.53%

In this section, we summarize additional forward-backward quantizers, which can be either improved or justified with our PC++ framework. Specifically, we find that:

* Bi-Real Net <cit.>/R-BNN <cit.>: $\mathsf{F}(\mathbf{w})=\text{sign}$, $\mathsf{B}(\mathbf{w})=\nabla F(\mathbf{w})$, where $F(\mathbf{w})$ is a piewise polynomial function. We simply choose $\mathsf{F} =F(\mathbf{w})$ and arrive at our legitimate variant Poly+. Note that we gradually increase the coefficient of $F(\mathbf{w})$ such that we ensure full binarization at the end of the training phase.

* EDE in IR-Net <cit.>: $\mathsf{F}(\mathbf{w})=\text{sign}$, $\mathsf{B}(\mathbf{w})=\nabla g(\mathbf{w})=kt(1-\text{tanh}^2(t\mathbf{w}))$, where $k$ and $t$ are control variables varying during the training process, such that $g(\mathbf{w})\approx \text{sign}$ at the end of training. Again, we choose $\mathsf{F} =g(\mathbf{w})$ and arrive at our new legitimate variant EDE+.

* ReActNet <cit.> can be well justified and is a special case of PC++.

We visualize these forward-backward quantizers and our new variants in <Ref>. Moreover, we perform experiments on vision transformers to examine the performance of additional quantizers and their modified variants. We report the results in <Ref> and observe that (1)Our new proposed Poly+ and EDE+ always outperform the original algorithms and further confirm that our PC++ framework merits theoretical and empirical justifications; (2)BNN++ still outperforms other algorithms on all tasks.

§ ADDITIONAL RESULTS FOR END-TO-END TRAINING

Finally, we provide the error bars for our main experiments in <Ref> and <Ref> for CNN backbones and vision transformer backbones, respectively.

Error bar for binarizing weights (BW), binarizing weights and activation (BWA) and binarizing weights, activation, with 8-bit accumulators  (BWAA) on CNN backbones. We consider the end-to-end (E2E) pipeline. We compare five variants of ProxConnect++ (BC, PC, BNN, BNN+, and BNN++) with FP, PQ, and rPC. For the end-to-end pipeline, we report the mean of five runs with different random seeds.

2*[-.6ex]Dataset      2*[-.6ex]Task
2*[-.6ex]FP     2*[-.6ex]PQ     2*[-.6ex]rPC     5cProxConnect++

BC     PC     BNN     BNN+     BNN++

4*[-.6ex]CIFAR-10      2*[-.6ex]BW     92.01%     81.59%     81.82%     87.51%     88.05%     89.92%     89.39%     90.03%

$\pm 0.19$     $\pm 0.11$     $\pm 0.16$     $\pm 0.07$     $\pm 0.05$     $\pm 0.11$     $\pm 0.13$     $\pm 0.06$

2*[-.6ex]BWA    92.01%     81.51%     81.60%     86.99%     87.26%     89.15%     89.02%     89.91%

$\pm 0.13$     $\pm 0.16$     $\pm 0.09$     $\pm 0.11$     $\pm 0.23$     $\pm 0.08$     $\pm 0.16$     $\pm 0.09$

4*[-.6ex]ImageNet-1K     2*[-.6ex]BW      78.87%     63.23%     66.39%     67.45%     67.51%     67.49%     66.99%     68.11%

$\pm 0.06$     $\pm 0.11$     $\pm 0.22$     $\pm 0.04$     $\pm 0.09$     $\pm 0.12$     $\pm 0.26$     $\pm 0.02$

2*[-.6ex]BWA     78.87%     61.19%     64.17%     65.42%     65.31%     65.29%     65.98%     66.08%

$\pm 0.18$     $\pm 0.22$     $\pm 0.19$     $\pm 0.22$     $\pm 0.17$     $\pm 0.21$     $\pm 0.15$     $\pm 0.13$

Error bar on binarizing vision transformers (BW and BWA) on ImageNet-1K. We consider the end-to-end (E2E) pipeline. We compare five variants of ProxConnect++ (BC, PC, BNN, BNN+, and BNN++) with FP, PQ, and rPC. The results are the mean of five runs with different random seeds.

2*[-.6ex]Model     2*[-.6ex]Task
2*[-.6ex]FP     2*[-.6ex]PQ     2*[-.6ex]rPC     5cProxConnect++

BC     PC     BNN     BNN+     BNN++

4*[-.6ex]DeiT-T      2*[-.6ex]BW     72.20%     61.23%     60.35%     63.22%     66.15%     65.00%     66.67%     67.34%

$\pm 0.11$     $\pm 0.07$     $\pm 0.19$     $\pm 0.21$     $\pm 0.11$     $\pm 0.15$     $\pm 0.09$     $\pm 0.07$

2*[-.6ex]BWA     72.20%     60.01%     58.77%     62.13%     65.29%     63.75%     65.29%     65.65%

$\pm 0.13$     $\pm 0.12$     $\pm 0.08$     $\pm 0.06$     $\pm 0.19$     $\pm 0.18$     $\pm 0.06$     $\pm 0.03$

2*[-.6ex]DeiT-S     2*[-.6ex]BW     79.91%     69.87%     68.74%     73.16%     73.51%     73.77%    73.23%     73.53%

$\pm 0.21$     $\pm 0.26$     $\pm 0.16$     $\pm 0.19$     $\pm 0.22$     $\pm 0.08$     $\pm 0.11$     $\pm 0.13$

1*[-.6ex]DeiT-B    2*[-.6ex]BW     81.81%     72.54%     70.11%     76.55%     76.61%     75.60%     76.63%     76.74%

$\pm 0.17$     $\pm 0.15$     $\pm 0.23$     $\pm 0.07$     $\pm 0.24$     $\pm 0.17$     $\pm 0.13$     $\pm 0.07$

We anticipate our work to further enable training and deploying advanced machine learning models to resource limited devices and environments, and help reducing energy consumption and carbon footprint at large. We do not foresee any direct negative societal impact.

One limitation we hope to address in the future is to build a theoretical framework that will allow practitioners to quickly evaluate different forward-backward quantizers for a variety of applications.