Resilient Binary Neural Network
Binary neural networks (BNNs) have received ever-increasing popularity for
their great capability of reducing storage burden as well as quickening
inference time. However, there is a severe performance drop compared with
real-valued networks, due to its intrinsic frequent weight oscillation during
training. In this paper, we introduce a Resilient Binary Neural Network (ReBNN)
to mitigate the frequent oscillation for better BNNs' training. We identify
that the weight oscillation mainly stems from the non-parametric scaling
factor. To address this issue, we propose to parameterize the scaling factor
and introduce a weighted reconstruction loss to build an adaptive training
objective. For the first time, we show that the weight oscillation is
controlled by the balanced parameter attached to the reconstruction loss, which
provides a theoretical foundation to parameterize it in back propagation. Based
on this, we learn our ReBNN by calculating the balanced parameter based on its
maximum magnitude, which can effectively mitigate the weight oscillation with a
resilient training process. Extensive experiments are conducted upon various
network models, such as ResNet and Faster-RCNN for computer vision, as well as
BERT for natural language processing. The results demonstrate the overwhelming
performance of our ReBNN over prior arts. For example, our ReBNN achieves 66.9%
Top-1 accuracy with ResNet-18 backbone on the ImageNet dataset, surpassing
existing state-of-the-arts by a significant margin. Our code is open-sourced at
https://github.com/SteveTsui/ReBNN.
Sheng Xu, Yanjing Li, Teli Ma, Mingbao Lin, Hao Dong, Baochang Zhang, Peng Gao, Jinhu Lv
# Resilient Binary Neural Network

###### Abstract

Binary neural networks (BNNs) have received ever-increasing popularity for their great capability of reducing storage burden as well as queueing inference time. However, there is a severe performance drop compared with real-valued networks, due to its intrinsic frequent weight oscillation during training. In this paper, we introduce a Resilient Binary Neural Network (ReBNN) to mitigate the frequent oscillation for better BNNs' training. We identify that the weight oscillation mainly stems from the non-parametric scaling factor. To address this issue, we propose to parameterize the scaling factor and introduce a weighted reconstruction loss to build an adaptive training objective. For the first time, we show that the weight oscillation is controlled by the balanced parameter attached to the reconstruction loss, which provides a theoretical foundation to parameterize it in back propagation. Based on this, we learn our ReBNN by calculating the balanced parameter based on its maximum magnitude, which can effectively mitigate the weight oscillation with a resilient training process. Extensive experiments are conducted upon various network models, such as ResNet and Faster-RCNN for computer vision, as as BERT for natural language processing. The results demonstrate the overwhelming performance of our ReBNN over prior arts. For example, our ReBNN achieves 66.9% Top-1 accuracy with ResNet-18 backbone on the ImageNet dataset, surpassing existing state-of-the-arts by a significant margin. Our code is open-sourced at [https://github.com/SteveTsui/ReBNN](https://github.com/SteveTsui/ReBNN).

1 Beihang University, Beijing, P.R.China \({}^{2}\) Shanghai AI Laboratory, Shanghai, P.R.China

\({}^{3}\) Peking University, Beijing, P.R.China \({}^{4}\) Zhongguancun Laboratory, Beijing, P.R.China

## Introduction

Deep neural networks (DNNs) have dominated the recent advances of artificial intelligence from computer vision (CV) [17, 14] to natural language processing (NLP) [16, 15] and many beyond. In particular, large pre-trained models, _e.g._, ResNet [14] and BERT [13], have continuously broken many records of the state-of-the-art performance. However, the achievement also comes with tremendous demands for memory and computation resources. These demands pose a huge challenge to the computing ability of many devices, especially resource-limited platforms such as mobile phones and electronic gadgets. In light of this, substantial research efforts are being invested in saving memory usage and computational power for an efficient online inference [14, 15, 16, 17]. Among these studies, network quantization is particularly suitable for model deployment on resource-limited platforms for its great reduction in parameter bit-width and practical speedups supported by general hardware devices.

Binarization, an extreme form of quantization, represents weights and activations of CNNs using a single bit, which well decreases the storage requirements by \(32\times\) and computation cost by up to \(58\times\)[15]. Consequently, binarized neural networks (BNNs) are widely deployed on various tasks such as image classification [15, 16, 17, 18] and object detection [16, 17], and have the potential to be deployed directly on next-generation AI chips. However, the performance of BNNs remains largely inequal to the real-valued counterparts, due primarily to the degraded representation capability and trainability.

Conventional BNNs [15, 16] are often sub-optimized, due to their intrinsic frequent weight oscillation during training. We first identify that the weight oscillation mainly stems from the non-parametric scaling factor. Fig. 1(a) shows the epoch-wise oscillation1 of ReActNet, where weight oscillation exists even when the network is convergent. As shown in Fig. 1(b), the conventional ReActNet [16] possesses a channel-wise tri-modal distribution in the 1-bit convolution layers, whose peaks respectively center around the \(\{-1,0,+1\}\). Such distribution leads to a magnified scaling factor \(\alpha\), and thus the quantized weights \(\pm\alpha\) are quite larger than the small weights around \(0\), which might cause the weight oscillation. As illustrated in Fig. 1(c), in BNNs, the real-valued latent tensor is binarized by the sign function and scaled by the scaling factor (the orange dot) in the forward propagation. In the backward propagation, the gradient is computed based on the quantized value \(\pm\alpha\) (indicated by the yellow dotted line). However, the gradient of small latent weights is misleading, when the scaling factor is magnified by the weights around
\(\pm 1\) as ReActNet (Fig. 1(a)). Then the update is conducted on the latent value (the **black** dot), which leads to the oscillation of the latent weight. With extremely limited representation states, such latent weights with small magnitudes frequently oscillate during the non-convex optimization.

To address the aforementioned problem, we aim to introduce a Resilient Binary Neural Network (ReBNN). The intuition of our work is to re-learn the channel-wise scaling factor as well as the latent weights in a unified framework. Accordingly, we propose to parameterize the scaling factor and introduce a weighted reconstruction loss to build an adaptive training objective. We further show that the oscillation is factually controlled by the balanced parameter attached to the reconstruction loss, which provides a theoretical foundation to parameterize it in back propagation. The oscillation only happens when the gradient possesses a magnitude big enough to change the sign of the latent weight. Consequently, we calculate the balanced parameter based on the maximum magnitude of weight gradient during each iteration, leading to resilient gradients and effectively mitigating the weight oscillation. Our main contributions are summarized as follows:

* We propose a new resilient gradient for learning the binary neural networks (ReBNN), which mitigates the frequent oscillation to better train BNNs.
* We parameterize the scaling factor and introduce a weighted reconstruction loss to build an adaptive learning objective. We prove that the occurrence of oscillation is controlled by the balanced parameter attached to the reconstruction loss. Therefore, we utilize resilient weight gradients to learn our ReBNN and effectively mitigate the weight oscillation.
* Extensive experiments demonstrate the superiority of our ReBNN against other prior state-of-the-arts. For example, our ReBNN achieves 66.9% Top-1 accuracy on the ImageNet dataset, surpassing prior ReActNet by 1.0% with no extra parameters. In particular, our ReBNN also achieves state-of-the-art on fully binarized BERT models, demonstrating the generality of our ReBNN.

## Related Work

BinaryNet, based on BinaryConnect, was proposed to train CNNs with binary weights. The activations are triggered at run-time while the parameters are computed during training. Following this line of research, local binary convolution (LBC) layers are introduced in [11] to binarize the non-linearly activations. XNOR-Net [14] is introduced to improve convolution efficiency by binarizing the weights and inputs of the convolution kernels. More recently, Bi-Real Net [15] explores a new variant of residual structure to preserve the information of real activations before the sign function, with a tight approximation to the derivative of the non-differentiable sign function. Real-to-binary [13] re-scales the feature maps on the channels according to the input before binarized operations and adds a SE-Net [15] like gating module. ReActNet [15] replaces the conventional PReLU [12] and the sign function of the BNNs with RPReLU and RSign with a learnable threshold, thus improving the performance of BNNs. RBONN [21] introduces a recurrent bilinear optimization to address the asynchronous convergence problem for BNNs, which further improves the performance of BNNs. However, most of these aforementioned suffer from the weight oscillation mainly stemming from the non-parametric scaling factor.

Unlike prior works, our ReBNN proposes to parameterize the scaling factor and introduces a weighted reconstruction loss to build an adaptive training objective. We further prove the oscillation is controlled by the balanced parameter. Based on the analysis, we introduce a resilient weight gradient to effectively address the oscillation problem.

Figure 1: (a) We show the epoch-wise weight oscillation of ReActNet. (b) We randomly select 2 channels of the first 1-bit layer in ReActNet [15]. Obviously, the distribution is with 3 peaks centering around \(\{-1,0,+1\}\), which magnifies the non-parametric scaling factor (**red** line). (c) We illustrate the weight oscillation caused by such inappropriate scale calculation, where \(\mathbf{w}\) and \(\mathcal{L}\) indicate the latent weight and network loss function (**blue** line), respectively.
## Methodology

### Preliminaries

Given an \(N\)-layer CNN model, we denote its weight set as \(\mathbf{W}=\{\mathbf{w}^{n}\}_{n=1}^{N}\) and input feature map set as \(\mathbf{A}=\{\mathbf{a}^{n}_{in}\}_{n=1}^{N}\). The \(\mathbf{w}^{n}\in\mathbb{R}^{C^{n}_{in}\times C^{n}_{in}\times K^{n}}\times K^{n}\) and \(\mathbf{a}^{n}_{in}\in\mathbb{R}^{C^{n}_{in}\times W^{n}_{in}\times H^{n}_{in}}\) are the convolutional weight and the input feature map in the \(n\)-th layer, where \(C^{n}_{in}\), \(C^{n}_{out}\) and \(K^{n}\) respectively stand for input channel number, output channel number and the kernel size. Also, \(W^{n}_{in}\) and \(H^{n}_{in}\) are the width and height of the feature maps. Then, the convolutional outputs \(\mathbf{a}^{n}_{out}\) can be technically formulated as:

\[\mathbf{a}^{n}_{out}=\mathbf{w}^{n}\otimes\mathbf{a}^{n}_{in}, \tag{1}\]

where \(\otimes\) represents the convolution operation. Herein, we omit the non-linear function for simplicity. Binary neural network intends to represent \(\mathbf{w}^{n}\) and \(\mathbf{a}^{n}\) in a 1-bit format as \(\mathbf{b}^{\mathbf{w}^{n}}\in\{-1,+1\}^{C^{n}_{out}\times C^{n}_{in}\times K^ {n}\times K^{n}}\) and \(\mathbf{b}^{\mathbf{a}^{n}_{in}}\in\{-1,+1\}^{C^{n}_{in}\times W^{n}_{in}\times H ^{n}_{in}}\) such that the float-point convolutional outputs can be approximated by the efficient XNOR and Bit-count instructions as:

\[\mathbf{a}^{n}_{out}\approx\boldsymbol{\alpha}^{n}\circ(\mathbf{b}^{\mathbf{w} ^{n}}\odot\mathbf{b}^{\mathbf{a}^{n}_{in}}), \tag{2}\]

where \(\circ\) represents the channel-wise multiplication, \(\odot\) denotes the XNOR and Bit-count instructions, and \(\boldsymbol{\alpha}^{n}=\{\alpha^{n}_{1},\alpha^{n}_{2},...,\alpha^{n}_{C^{n}_{ out}}\}\in\mathbb{R}^{C^{n}_{out}}_{+}\) is known as the channel-wise scaling factor vector (Rastegari et al., 2016) to mitigate the output gap between Eq. (1) and its approximation of Eq. (2). We denote \(\mathcal{A}=\{\boldsymbol{\alpha}^{n}\}_{n=1}^{N}\). Most existing implementations simply follow earlier studies (Rastegari et al., 2016; Liu et al., 2018) to optimize \(\mathcal{A}\) and latent weights \(\mathbf{W}\) based on a non-parametric bi-level optimization as:

\[\mathbf{W}^{*} =\operatorname*{arg\,min}_{\mathbf{W}}\mathcal{L}(\mathbf{W}; \mathcal{A}^{*}), \tag{3}\] \[\mathrm{s.t.} \boldsymbol{\alpha}^{n*} =\operatorname*{arg\,min}_{\boldsymbol{\alpha}^{n}}\|\mathbf{w}^{ n}-\boldsymbol{\alpha}^{n}\circ\mathbf{b}^{\mathbf{w}^{n}}\|_{2}^{2}, \tag{4}\]

where \(\mathcal{L}(\cdot)\) represents the training loss. Consequently, a closed-form solution of \(\boldsymbol{\alpha}^{n}\) can be derived via the channel-wise absolute mean (CAM) as \(\alpha^{n}_{i}=\frac{\|\mathbf{w}^{n}_{i},\|_{1}}{M^{n}}\) and \(M^{n}=C^{n}_{in}\times K^{n}\times K^{n}\). For ease of representation, we use \(\mathbf{w}^{n}_{i}\) as an alternative of \(\mathbf{w}^{n}_{i,...}\) in what follows. The latent weight \(\mathbf{w}^{n}\) is updated via a standard gradient back-propagation algorithm and its gradient is calculated as:

\[\delta_{\mathbf{w}^{n}_{i}}=\frac{\partial\mathcal{L}}{\partial\mathbf{w}^{n}_ {i}}=\frac{\partial\mathcal{L}}{\partial\hat{\mathbf{w}}^{n}_{i}}\frac{ \partial\hat{\mathbf{w}}^{n}_{i}}{\partial\mathbf{w}^{n}_{i}}=\alpha^{n}_{i} \frac{\partial\mathcal{L}}{\partial\hat{\mathbf{w}}^{n}_{i}}\varoq{1}_{| \mathbf{w}^{n}_{i}|\leq 1}, \tag{5}\]

where \(\varoq{8}\) denotes the Hadmard product and \(\hat{\mathbf{w}}^{n}=\boldsymbol{\alpha}^{n}\circ\mathbf{b}^{\mathbf{w}^{n}}\).

**Discussion**. Eq. (5) shows weight gradient mainly comes from the non-parametric \(\alpha^{n}_{i}\) and the gradient \(\frac{\partial\mathcal{L}}{\partial\hat{\mathbf{w}}^{n}_{i}}\). \(\frac{\partial\mathcal{L}}{\partial\hat{\mathbf{w}}^{n}_{i}}\) is automatically solved in the back propagation and becomes smaller as network convergence, however, \(\alpha^{n}_{i}\) is often magnified by the tri-modal distribution (Liu et al., 2020). Therefore, weight oscillation mainly stems from \(\alpha^{n}_{i}\). Given a single weight \(\mathbf{w}^{n}_{i,j}(1\leq j\leq M^{n})\) centering around zero, the gradient \(\frac{\partial\mathcal{L}}{\partial\hat{\mathbf{w}}^{n}_{i,j}}\) is misleading, due to the significant gap between \(\mathbf{w}^{n}_{i,j}\) and \(\alpha^{n}_{i}\mathbf{b}^{\mathbf{w}^{n}_{i,j}}\). Consequently, the bi-level optimization leads to frequent weight oscillation. To address this issue, we reformulate traditional bi-level optimization using Lagrange multiplier and show that a learnable scaling factor is a natural training stabilizer.

### Resilient Binary Neural Network

We first give the learning objective in this paper as:

\[\operatorname*{arg\,min}_{\mathbf{W},\mathcal{A}}\mathcal{L}(\mathbf{W}, \mathcal{A})+\mathcal{L}_{R}(\mathbf{W},\mathcal{A}), \tag{6}\]

where \(\mathcal{L}_{R}(\mathbf{W},\mathcal{A})\) is a weighted reconstruction loss and defined as:

\[\mathcal{L}_{R}(\mathbf{W},\mathcal{A})=\frac{1}{2}\sum_{n=1}^{N}\sum_{i=1}^{C _{out}}\gamma^{n}_{i}\|\mathbf{w}^{n}_{i}-\alpha^{n}_{i}\mathbf{b}^{\mathbf{w }^{n}_{i}}\|_{2}^{2}, \tag{7}\]

in which \(\gamma^{n}_{i}\) is a balanced parameter. Based on the objective, the weight gradient in Eq. (5) becomes:

\[\begin{split}\delta_{\mathbf{w}^{n}_{i}}&=\frac{ \partial\mathcal{L}}{\partial\mathbf{w}^{n}_{i}}+\gamma^{n}_{i}(\mathbf{w}^{n}_ {i}-\alpha^{n}_{i}\mathbf{b}^{\mathbf{w}^{n}_{i}})\\ &=\alpha^{n}_{i}(\frac{\partial\mathcal{L}}{\partial\hat{\mathbf{ w}}^{n}_{i}}\varoq{1}_{|\mathbf{w}^{n}_{i}|\leq 1}-\gamma^{n}_{i}\mathbf{b}^{\mathbf{w}^{n}_{i}})+ \gamma^{n}_{i}\mathbf{w}^{n}_{i}.\end{split} \tag{8}\]

The \(\mathcal{S}^{n}_{i}(\alpha^{n}_{i},\mathbf{w}^{n}_{i})=\gamma^{n}_{i}(\mathbf{ w}^{n}_{i}-\alpha^{n}_{i}\mathbf{b}^{\mathbf{w}^{n}_{i}})\) is an additional term added in the back-propagation process. We add this item given that a too small \(\alpha^{n}_{i}\) diminishes the gradient \(\delta_{\mathbf{w}^{n}_{i}}\) and causes a constant weight \(\mathbf{w}^{n}_{i}\). In what follows, we state and prove the proposition that \(\delta_{\mathbf{w}^{n}_{i,j}}\) is a resilient gradient for a single weight \(\mathbf{w}^{n}_{i,j}\). We sometimes omit subscript \(i,j\) and superscript \(n\) for an easy representation.

**Proposition 1**.: _The additional term \(\mathcal{S}(\alpha,\mathbf{w})=\gamma(\mathbf{w}-\alpha\mathbf{b}^{\mathbf{w}})\) achieves a resilient training process by suppressing frequent weight oscillation. Its balanced factor \(\gamma\) can be considered as the parameter controlling the occurrence of the weight oscillation._

**Proof:** We prove the proposition by contradiction. For a single weight \(\mathbf{w}\) centering around zero, the straight-through-estimator \(\mathbf{1}_{|\mathbf{w}|\leq 1}=1\). Thus we omit it in the following. Based on Eq. (8), with a learning rate \(\eta\), the weight updating process is formulated as:

\[\begin{split}\mathbf{w}^{t+1}&=\mathbf{w}^{t}-\eta \delta_{\mathbf{w}^{t}}\\ &=\mathbf{w}^{t}-\eta[\alpha^{t}(\frac{\partial\mathcal{L}}{\partial \hat{\mathbf{w}}^{t}}-\gamma\mathbf{b}^{\mathbf{w}^{t}})+\gamma\mathbf{w}^{t}] \\ &=(1-\eta\gamma)\mathbf{w}^{t}-\eta\alpha^{t}(\frac{\partial\mathcal{L}}{ \partial\hat{\mathbf{w}}^{t}}-\gamma\mathbf{b}^{\mathbf{w}^{t}})\\ &=(1-\eta\gamma)\big{[}\mathbf{w}^{t}-\frac{\eta\alpha^{t}}{(1- \eta\gamma)}(\frac{\partial\mathcal{L}}{\partial\hat{\mathbf{w}}^{t}}-\gamma \mathbf{b}^{\mathbf{w}^{t}})\big{]},\end{split} \tag{9}\]

where \(t\) denotes the \(t\)-th training iteration and \(\eta\) represents learning rate. Different weights shares different distances to the quantization level \(\pm 1\), therefore, their gradients should be modified in compliance with their scaling factors and current learning rate. We first assume the initial state \(\mathbf{b}^{\mathbf{w}^{t}}=-1\), and
Similarly, the oscillation probability from the iteration \(t+1\) to \(t+2\) is:

\[P(\mathbf{b}^{\mathbf{w}^{t+1}}\neq\mathbf{b}^{\mathbf{w}^{t+2}})\Big{|}_{ \mathbf{b}^{\mathbf{w}^{t+1}}=1}\leq P(\frac{\partial\mathcal{L}}{\partial\hat{ \mathbf{w}}^{t+1}}\geq\gamma). \tag{11}\]

Thus, the sequential oscillation probability from the iteration \(t\) to \(t+2\) is:

\[\begin{split}& P((\mathbf{b}^{\mathbf{w}^{t+1}}\neq\mathbf{b}^{ \mathbf{w}^{t+2}})\cap(\mathbf{b}^{\mathbf{w}^{t+1}}\neq\mathbf{b}^{\mathbf{w}^ {t+2}}))|_{\mathbf{b}^{\mathbf{w}^{t}}=-1}\\ &\leq P\big{(}(\frac{\partial\mathcal{L}}{\partial\hat{\mathbf{w }}^{t}}\leq-\gamma)\cap(\frac{\partial\mathcal{L}}{\partial\hat{\mathbf{w}}^{t +1}}\geq\gamma)\big{)},\end{split} \tag{12}\]

which denotes that the weight oscillation happens only if the magnitudes of \(\frac{\partial\mathcal{L}}{\partial\hat{\mathbf{w}}^{t}}\) and \(\frac{\partial\mathcal{L}}{\partial\hat{\mathbf{w}}^{t+1}}\) are both larger than \(\gamma\). **As a result, its attached factor \(\gamma\) can be considered as a parameter used to control the occurrence of the weight oscillation**.

However, if the conditions in Eq. (12) are met, with Eq. (9) concluded, the gradient of \(\hat{\mathbf{w}}^{t+1}\) is formulated as:

\[\begin{split}\frac{\partial\mathcal{L}}{\partial\hat{\mathbf{w} }^{t+1}}&=\frac{\partial\mathcal{L}}{\partial\hat{\mathbf{w}}^{t }}-\eta\frac{\partial^{2}\mathcal{L}}{\partial(\hat{\mathbf{w}}^{t})^{2}} \geq\gamma,\\ \eta\frac{\partial^{2}\mathcal{L}}{\partial(\hat{\mathbf{w}}^{t} )^{2}}&\leq\frac{\partial\mathcal{L}}{\partial\hat{\mathbf{w}}^{t }}-\gamma\leq-2\gamma.\end{split} \tag{13}\]

Note that \(\eta\) and \(\gamma\) are two positive variables, thus the second-order gradient \(\frac{\partial^{2}\mathcal{L}}{\partial(\hat{\mathbf{w}}^{t})^{2}}<0\) holds always. Consequently, \(\mathcal{L}(\hat{\mathbf{w}}^{t+1})\) can only be a local maxima, instead of a minima, which raises a contradiction with convergence in the training process. Such a contradiction indicates that the training algorithm will be convergent until no oscillation occurs, due to the additional term \(\mathcal{S}(\alpha,\mathbf{w})\). Therefore, we completes our proof. 

Our proposition and proof reveal that the balanced parameter \(\gamma\) is actually a "threshold". A very small "threshold" fails to mitigate the frequent oscillation effectively while a too large one suppresses the necessary sign inversion and hinders the gradient descent process. To solve this, we devise the learning rule of \(\gamma\) as:

\[\gamma_{i}^{n,t+1}=\frac{1}{M^{n}}\|\mathbf{b}^{\mathbf{w}_{i}^{n,t}}\varoquasw \mathbf{b}^{\mathbf{w}_{i}^{n,t+1}}-\mathbf{1}\|_{0}\cdot\max_{1\leq j\leq M^ {n}}(|\frac{\partial\mathcal{L}}{\partial\hat{\mathbf{w}}_{i,j}^{n,t}}|), \tag{14}\]

where the first item \(\frac{1}{M^{n}}\|\mathbf{b}^{\mathbf{w}_{i}^{n,t}}\varoquasw\mathbf{b}^{ \mathbf{w}_{i}^{n,t+1}}-\mathbf{1}\|_{0}\) denotes the proportion of weights with sign changed. The second item \(\max_{1\leq j\leq M^{n}}(|\frac{\partial\mathcal{L}}{\partial\hat{\mathbf{w}} _{i,j}^{n}}|)\) is derived from Eq. (12), denoting the gradient with the greatest magnitude of the \(t\)-th iteration. In this way, we suppress the frequent weight oscillation by a resilient gradient.

We further optimize the scaling factor as:

\[\delta_{\alpha_{i}^{n}}=\frac{\partial\mathcal{L}}{\partial\alpha_{i}^{n}}+ \frac{\partial\mathcal{L}_{R}}{\partial\alpha_{i}^{n}}. \tag{15}\]

The gradient derived from softmax loss can be easily calculated according to back propagation. Based on Eq. (7), it is easy to derive:

\[\frac{\partial\mathcal{L}_{R}}{\partial\alpha_{i}^{n}}=\gamma_{i}^{n}(\mathbf{ w}_{i}^{n}-\alpha_{i}^{n}\mathbf{b}^{\mathbf{w}_{i}^{n}})\varoquasw\mathbf{b}^{ \mathbf{w}_{i}^{n}}. \tag{16}\]

## Experiments

Our ReBNNs are evaluated first on image classification and object detection tasks for visual recognition. Then, we evaluate ReBNN on the GLUE [22] benchmark with diverse NLP tasks. In this section, we first introduce the implementation details of ReBNN. Then we validate the effectiveness of the balanced parameter in the ablation study. Finally, we compare our method with state-of-the-art BNNs on various tasks to demonstrate the superiority of ReBNNs.

### Datasets and Implementation Details

**Datasets:** For its huge scope and diversity, the ImageNet object classification dataset [23] is more demanding, which has 1000 classes, 1.2 million training photos, and 50k validation images.

The COCO dataset includes images from 80 different categories. All our experiments on COCO dataset are conducted on the COCO 2014 [10] object detection track in the training stage, which contains the combination of 80k images from the COCO train2014 and 35k images sampled from COCO val2014, _i.e._, COCO trainval35k. Then we test our method on the remaining 5k images from the COCO minival. We report the average precision (AP) for IoUs\(\in\)[0.5: 0.05: 0.95], designated as mAP@[.5,.95], using COCO's standard evaluation metric. For further analyzing our method, we also report AP\({}_{50}\), AP\({}_{75}\), AP\({}_{s}\), AP\({}_{m}\), and AP\({}_{l}\).

The GLUE benchmark contains multiple natural language understanding tasks. We follow [22] to evaluate the performance: Matthews correlation for CoLA, Spearman correlation for STS-B, and accuracy for the rest tasks: RTE, MRPC, SST-2, QQP, MNLI-m (matched), and MNLI-mm (mismatched). Also, for machine reading comprehension on SQuAD, we report the EM (exact match) and F1 score.

**Implementation Details:** PyTorch [23] is used to implement ReBNN. We run the experiments on 4 NVIDIA Tesla A100 GPUs with \(80\) GB memory. Following [10], we retain weights in the first layer, shortcut, and last layer in the networks as the real-valued.

For the image classification task, ResNets [11] are employed as the backbone networks to build our ReBNNs. We offer two implementation setups for fair comparison. First, we use **one-stage training** on ResNets, with SGD as the optimization algorithm and a momentum of 0.9, and a weight decay of \(1e-4\) following [21]. \(\eta\) is set to 0.1. The learning rates are optimized by the annealing cosine learning rate schedule. The number of epochs is set as 200. Then, we employ **two-stage training** following [10]. Each stage counts 256 epochs. In this implementation, Adam is selected as the optimizer. And the network is supervised by a real-valued ResNet-34 teacher. The weight decay is set as 0 following [10]. The learning rates \(\eta\) is set as \(1e-3\) and annealed to 0 by linear descent.

For objection detection, we use the Faster-RCNN [10] and SSD [10], which are based on ResNet-18 [10] and VGG-16 [26] backbone, respectively. We fine-tune the detector on the dataset for object detection. For SSD and Faster-RCNN, the batch size is set to 16 and 8, respectively,
with applying SGD optimizer. \(\eta\) is equal to 0.008. We use the same structure and training settings as BiDet (Wang et al., 2020) on the SSD framework. The input resolution is \(1000\times 600\) for Faster-RCNN and \(300\times 300\) for SSD, respectively.

For the natural language processing task, we conduct experiments based on BERT\({}_{\rm BASE}\)(Devlin et al., 2018) (with 12 hidden layers) architecture following BiBERT (Qin et al., 2022), respectively. The detailed training setups are the same as BiBERT. We extend the ReBNN to multi-layer perceptrons (MLPs) and use the Bi-Attention following BiBERT (Qin et al., 2022).

### Ablation study

Since no extra hyper-parameter is introduced, we first evaluate the different calculation of \(\gamma\). Then we show how our ReBNN achieves a resilient training process. In the ablation study, we use the ResNet-18 backbone initialized from the first stage training with W32A1 following (Liu et al., 2020).

**Calculation of \(\gamma\):** We compare the different calculations of \(\gamma\) in this part. As shown in Tab. 1, the performances increase first and then decrease when increasing the value of constant \(\gamma\). Considering that the gradient magnitude varies layer-wise and channel-wise, a subtle \(\gamma\) can hardly be manually set as a global value. We further compare the gradient-based calculation. To avoid extreme values, we set the upper bound as \(2e-4\) and lower bound as \(1e-5\). As shown in the bottom lines, we first use \(\max_{1\leq j\leq M^{n}}(|\frac{\partial\mathcal{L}}{\partial\mathbf{w}_{i,j }^{n}}|)\), the maximum intra-channel gradient of last iteration, which shows a similar performance compared with the constant \(1e-4\). This indicates that only using the maximum intra-channel gradient may also suppress necessary sign flip, thus hindering the training. Inspired by this, we use Eq. (14) to calculate \(\gamma\) and improve the performance by 0.6%, showing that considering the weight oscillation proportion allows the necessary sign flip and leads a more effective training. We also show the training loss curves in Fig. 3(b). As plotted, the curves of \(\mathcal{L}\) almost demonstrate the degrees of training sufficiency. Thus we draw the conclusion that ReBNN with \(\gamma\) calculated by Eq. (14) achieves the lowest training loss as well as an efficient training process.

Note that the loss may not be minimal at each training iteration, but our method is just a reasonable version of gradient descent algorithms by nature, which can be used to solve the optimization problem as general. We empirically prove ReBNN's capability of mitigating the weight oscillation, leading to a better convergence.

**Resilient training process:** We first show the evolution of the latent weight distribution is this section. We plot the distribution of the first channel of the first binary convolution layer per 32 epochs in Fig. 2. As seen, our ReBNN can efficiently redistribute the BNNs towards resilience. Conventional ReActNet (Liu et al., 2020) possesses a tri-model distribution, which is unstable due to the scaling factor with

\begin{table}
\begin{tabular}{c c c} \hline \hline Value of \(\gamma\) & Top-1 & Top-5 \\ \hline \(0\) & 65.8 & 86.3 \\ \(1e-5\) & 66.2 & 86.7 \\ \(1e-4\) & 66.4 & 86.7 \\ \(1e-3\) & 66.3 & 86.8 \\ \(1e-2\) & 65.9 & 86.5 \\ \(\max_{1\leq j\leq M^{n}}(|\frac{\partial\mathcal{L}}{\partial\mathbf{w}_{i,j}^ {n}}|)\) & 66.3 & 86.2 \\
**Eq. (14)** & **66.9** & **87.1** \\ \hline \hline \end{tabular}
\end{table}
Table 1: We compare different calculation method of \(\gamma\), including constant varying from \(0\) to \(1e-2\) and gradient-based calculation.

Figure 3: (a) The epoch-wise weight oscillation ratio of ReActNet (solid), ReCU (dotted) and ReBNN (dashed). (b) Comparing the loss curves of ReActNet and our ReBNN with different calculation of \(\gamma\).

Figure 2: The evolution of latent weight distribution of (a) ReActNet and (b) ReBNN. We select the first channel of the first binary convolution layer to show the evolution. The model is initialized from the first stage training with W32A1 following (Liu et al., 2020). We plot the distribution every 32 epochs.
large magnitudes. In contrast, our ReBNN is constrained by the balanced parameter \(\gamma\) during training, thus leading to a resilient bi-modal distribution with fewer weights centering around zero. We also plot the ratios of sequential weight oscillation of ReBNN and ReActNet for the 1-st, 8-th, and 16-th binary convolution layers of ResNet-18. As shown in Fig. 3(a), the dashed lines gain much lower magnitudes than the solid (ReActNet) and dotted (ReCU [21]) lines with the same color, validating the effectiveness of our ReBNN in suppressing the consecutive weight oscillation. Besides, we can also find that the sequential weight oscillation ratios of ReBNN are gradually decreased to 0 as the training converges.

### Image classification

We first show the experimental results on ImageNet with ResNet-18 and ResNet-34 [14] backbones in Tab. 2. We compare ReBNN with BNN [15], XNOR-Net [17], Bi-Real Net [16], RBNN [18], and ReCU [21] for the one-stage training strategy. For the two-stage training strategy, we compare with ReActNet [16], FDA-BNN [21], and ReCU [21].

We evaluate our ReBNN with one-stage training following [21]. ReBNN outperforms all of the compared binary models in both Top-1 and Top-5 accuracy, as shown in Tab. 2. ReBNN-based ResNet-18 respectively achieves 61.6% and 83.4% in Top-1 and Top-5 accuracy, with 0.6% increases over state-of-the-art RECU. ReBNN further outperforms all compared methods with ResNet-34 backbone, achieving 65.8% Top-1 accuracy. We further evaluate our ReBNN with two-stage training following [16], where ReBNN surpasses ReActNet by 1.0%/0.6% Top-1 accuracy with ResNet-18/34 backbones, respectively.

In this paper, we use memory usage and OPs following [16] in comparison to other tasks for further reference. As shown in Tab. 2, ReBNN theoretically accelerates ResNet-18/34 by 11.17\(\times\) and 19.04\(\times\), which is significant for real-time applications.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline Network & Method & \#Bits & Size\({}_{\rm(MB)}\) & OPs\({}_{\rm(10^{8})}\) & Top-1 & Top-5 \\ \hline \multirow{8}{*}{ResNet-18} & Real-valued & 32-32 & 46.76 & 18.21 & 69.6 & 89.2 \\  & BNN & & & & 22.2 & 67.1 \\  & XNOR-Net & & & & 51.2 & 73.2 \\  & Bi-Real Net & & & & 56.4 & 79.5 \\  & RBNN & 1-1 & 4.15 & 1.63 & 59.6 & 81.6 \\  & ReCU & & & & 61.0 & 82.6 \\  & **ReRNN\({}_{1}\)** & & & **61.6** & **83.4** \\  & ReActNet & & & & 55.9 & 86.2 \\  & FDA-BNN & & & & 65.8 & 86.4 \\  & ReCU & 1-1 & 4.15 & 1.63 & 66.4 & 86.5 \\  & **ReBNN\({}_{2}\)** & & & & **66.9** & **87.1** \\ \hline \multirow{8}{*}{ResNet-34} & Real-valued & 32-32 & 87.19 & 36.74 & 73.3 & 91.3 \\  & Bi-Real Net & & & 62.2 & 83.9 \\ \cline{1-1}  & RBNN & 1-1 & 5.41 & 1.93 & 63.1 & 84.4 \\ \cline{1-1}  & RecU & 1-1 & 5.41 & 1.93 & 65.1 & 85.8 \\ \cline{1-1}  & **ReBNN\({}_{1}\)** & & & **65.8** & **86.2** \\ \cline{1-1}  & ReActNet & 1-1 & 5.41 & 1.93 & 69.3 & 88.6 \\ \cline{1-1}  & **ReBNN\({}_{2}\)** & 1-1 & 5.41 & 1.93 & **69.9** & **88.9** \\ \hline \hline \end{tabular}
\end{table}
Table 2: A performance comparison with SOTAs on ImageNet with different training strategies. #Bits denotes the bit width of weights and activations. We report the Top-1 (%) and Top-5 (%) accuracy performances. ReBNN\({}_{1}\) and ReBNN\({}_{2}\) denote our ReBNN learned with one-stage and two-stage training.

\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline Framework & Backbone & Method & \#Bits & Size\({}_{\rm(MB)}\) & OPs\({}_{\rm(G)}\) & \multicolumn{3}{c}{mAP} & \multirow{2}{*}{\({\rm{\bf{p}}}_{50}\)} & \multirow{2}{*}{\({\rm{\bf{AP}}}_{75}\)} & \multirow{2}{*}{\({\rm{\bf{AP}}}_{s}\)} & \multirow{2}{*}{\({\rm{\bf{AP}}}_{m}\)} & \multirow{2}{*}{\({\rm{\bf{AP}}}_{1}\)} \\  & & Real-valued & 32-32 & 47.48 & 434.39 & 26.0 & 44.8 & 27.2 & 10.0 & 28.9 & 39.7 \\ \multirow{4}{*}{Faster-RCNN} & \multirow{4}{*}{ResNet-18} & Real-valued & 4-4 & 6.73 & 55.90 & 22.9 & 38.6 & 23.7 & 8.0 & 24.9 & 36.3 \\  & & XNOR-Net & & & & 10.4 & 21.6 & 8.8 & 2.7 & 11.8 & 15.9 \\  & & Bi-Real Net & & & & 14.4 & 29.0 & 13.4 & 3.7 & 15.4 & 24.1 \\  & & BiDet & & & 15.7 & 31.0 & 14.4 & 4.9 & 16.7 & 25.4 \\  & & **ReBNN** & & & **19.6** & **37.6** & **20.4** & **7.0** & **20.1** & **33.1** \\ \hline \multirow{8}{*}{SSD} & \multirow{4}{*}{VGG-16} & Real-valued & 32-32 & 105.16 & 31.44 & 23.2 & 41.2 & 23.4 & 5.3 & 23.2 & 39.6 \\  & & DoRe
### Object detection

Because of its size and diversity, the COCO dataset presents a great challenge for object detection. On COCO, the proposed ReBNN is compared against state-of-the-art 1-bit neural networks such as XNOR-Net [14], Bi-Real Net [13], and BiDet [20]. We present the performance of the 4-bit DoReFa-Net [1] for reference.

As shown in Tab. 3, compared with state-of-the-art XNOR-Net, Bi-Real Net, and BiDet, our method improves the mAP@[.5,.95] by 9.2%, 5.2%, and 3.9% using the Faster-RCNN framework with the ResNet-18 backbone. Moreover, on other APs with different IoU thresholds, our ReBNN clearly beats others. Compared to DoReFa-Net, a quantized neural network with 4-bit weights and activations, our ReBNN obtains only 3.3% lower mAP. Our method yields a 1-bit detector with a performance of only 6.1% mAP lower than the best-performing real-valued counterpart (19.6% vs. 26.0%). Similarly, using the SSD300 framework with the VGG-16 backbone, our method achieves 18.1% mAP@[.5,.95], outperforming XNOR-Net, Bi-Real Net, and BiDet by 10.0 %, 6.9%, and 4.9% mAP, respectively. Our ReBNN also achieves highly efficient models by theoretically accelerating Faster-RCNN and SSD by 50.62\(\times\) and 14.76\(\times\).

### Natural language processing

In Tab. 4, we show experiments on the BERT\({}_{\mathrm{BASE}}\) architecture and the GLUE benchmark without data augmentation following BiBERT [20]. Experiments show that outperforms other methods on the development set of GLUE benchmark, including TernaryBERT [1], Bi-O2BERT [1], Q-BERT [1], Q2BERT [1], and BiBERT [20]. Our ReBNN surpasses existing methods on BERT\({}_{\mathrm{BASE}}\) architecture by a clear margin in the average accuracy. For example, our ReBNN surpasses BiBERT by 6.6% accuracy on QNLI dataset, which is significant for the natural language processing task. We observe our ReBNN brings improvements on 7 out of total 8 datasets, thus leading to a 2.6% average accuracy improvement. Our ReBNN also achieves highly efficient models by theoretically accelerating the BERT\({}_{\mathrm{BASE}}\) architecture by 56.25\(\times\).

### Deployment Efficiency

We implement the 1-bit models achieved by our ReBNN on ODROID C4, which has a 2.016 GHz 64-bit quad-core ARM Cortex-A55. With evaluating its real speed in practice, the efficiency of our ReBNN is proved when deployed into real-world mobile devices. We leverage the SIMD instruction SSHL on ARM NEON to make the inference framework BOLT [1] compatible with ReBNN. We compare ReBNN to the real-valued backbones in Tab. 5. We can see that ReBNN's inference speed is substantially faster with the highly efficient BOLT framework. For example, the acceleration rate achieves about 8.64\(\times\) on ResNet-18, which is slightly lower than the theoretical acceleration rate. For ResNet-34 backbone, ReBNN can achieve 9.03\(\times\) acceleration rate with BOLT framework on hardware, which is significant for the computer vision on real-world edge devices.

## Conclusion

In this paper, we analyze the influence of frequent weight oscillation in binary neural networks and proposed a Resilient Binary Neural Network (ReBNN) to provide resilient gradients for latent weights updating. Our method specifically proposes to parameterize the scaling factor and introduces a weighted reconstruction loss to build an adaptive training objective. We further manifest that the balanced parameter can serve as an indicator to reflect the frequency of the weight oscillation during back propagation. Our ReBNN reaches resilience by learning the balanced parameter, leading to a great reduction of weight oscillation. ReBNN shows strong generalization to gain impressive performance on various tasks such as image classification, object detection, and natural language processing tasks, demonstrating the superiority of the proposed method over state-of-the-art BNNs.

\begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline Method & \#Bits & Size\({}_{\mathrm{(MB)}}\) & OPs\({}_{\mathrm{(G)}}\) & MNLI-m/mm & QQP & QNLI & SST-2 & CoLA & STS-B & MRPC & RTE & Avg. \\ \hline Real-valued & 32-32-32 & 418 & 22.5 & 84.9/85.5 & 91.4 & 92.1 & 93.2 & 59.7 & 90.1 & 86.3 & 72.2 & 83.9 \\ - & Q-BERT & 2-8-8 & 43.0 & 6.5 & 76.6/77.0 & - & - & 84.6 & - & - & 68.3 & 52.7 & - \\ Q2BERT & 2-8-8 & 43.0 & 6.5 & 47.2/47.3 & 67.0 & 61.3 & 80.6 & 0 & 4.4 & 68.4 & 52.7 & 47.7 \\ TernaryBERT & 2-2-2 & 28.0 & 1.5 & 40.3/40.0 & 63.1 & 50.0 & 80.7 & 0 & 12.4 & 68.3 & 54.5 & 45.5 \\ BinaryBERT & & & 35.6/35.3 & 66.2 & 51.5 & 53.2 & 0 & 6.1 & 68.3 & 52.7 & 41.0 \\ BiBERT & 1-1-1 & 16.5 & 0.4 & 66.1/67.5 & 84.8 & 72.6 & 88.7 & 25.4 & 33.6 & 72.5 & **57.4** & 63.2 \\
**ReBNN** & & & **69.9/71.3** & **85.2** & **79.2** & **89.3** & **28.8** & **38.7** & **72.6** & 56.9 & **65.8** \\ \hline \end{tabular}
\end{table}
Table 4: Comparison of BERT quantization methods without data augmentation. #Bits denotes the bit width of weights, word embedding, and activations. “Avg.” denotes the average results.

\begin{table}
\begin{tabular}{c c c c c c} \hline Network & Method & W/A & Size\({}_{\mathrm{(MB)}}\) & Memory Saving & Latency\({}_{\mathrm{(ms)}}\) & Acceleration \\ \hline \multirow{2}{*}{ResNet-18} & Real-valued & 32/32 & 46.76 & - & 583.1 & - \\  & ReBNN & 1/1 & 4.15 & 17.26\(\times\) & 67.5 & 8.64\(\times\) \\ \multirow{2}{*}{ResNet-34} & Real-valued & 32/32 & 87.19 & - & 1025.6 & - \\  & ReBNN & 1/1 & 5.41 & 16.12\(\times\) & 113.6 & 9.03\(\times\) \\ \hline \end{tabular}
\end{table}
Table 5: Comparing ReBNN with real-valued models on hardware (single thread).
## Acknowledgment

This work was supported by National Natural Science Foundation of China under Grant 62076016, 62206272, 62141604, 61972016, and 62032016, Beijing Natural Science Foundation L223024.
