NeuroBlend: Towards Low-Power yet Accurate Neural Network-Based
  Inference Engine Blending Binary and Fixed-Point Convolutions
This paper introduces NeuroBlend, a novel neural network architecture
featuring a unique building block known as the Blend module. This module
incorporates binary and fixed-point convolutions in its main and skip paths,
respectively. There is a judicious deployment of batch normalizations on both
main and skip paths inside the Blend module and in between consecutive Blend
modules. Additionally, we present a compiler and hardware architecture designed
to map NeuroBlend models onto FPGA devices, aiming to minimize inference
latency while maintaining high accuracy. Our NeuroBlend-20 (NeuroBlend-18)
model, derived from ResNet-20 (ResNet-18) trained on CIFAR-10 (CIFAR-100),
achieves 88.0\% (73.73\%) classification accuracy, outperforming
state-of-the-art binary neural networks by 0.8\% (1.33\%), with an inference
time of 0.38ms per image, 1.4x faster than previous FPGA implementation for
BNNs. Similarly, our BlendMixer model for CIFAR-10 attains 90.6\%
accuracy(1.59\% less than full precision MLPMixer), with a 3.5x reduction in
model size compared to full precision MLPMixer. Furthermore, leveraging DSP
blocks for 48-bit bitwise logic operations enables low-power FPGA
implementation, yielding a 2.5x reduction in power consumption.
Arash Fayyazi, Mahdi Nazemi, Arya Fayyazi, Massoud Pedram
BendNet: Design and Optimization of a Neural Network-Based Inference Engine Blending Binary and Fixed-Point Convolutions

###### Abstract

This paper presents BlendNet, a neural network architecture employing a novel building block called Blend module, which relies on performing binary and fixed-point convolutions in its main and skip paths, respectively. There is a judicious deployment of batch normalizations on both main and skip paths inside the Blend module and in between consecutive Blend modules. This paper also presents a compiler for mapping various BlendNet models obtained by replacing some blocks/modules in various vision neural network models with BlendNet modules to FPGA devices with the goal of minimizing the end-to-end inference latency while achieving high output accuracy. BlendNet-20, derived from ResNet-20 trained on the CIFAR-10 dataset, achieves 88.0% classification accuracy (0.8% higher than the state-of-the-art binary neural network) while it only takes 0.38ms to process each image (1.4x faster than state-of-the-art). Similarly, our BlendMixer model trained on the CIFAR-10 dataset achieves 90.6% accuracy (1.59% less than full precision MLPMixer) while achieving a 3.5x reduction in the model size. Moreover, The reconfigurability of DSP blocks for performing 48-bit bitwise logic operations is utilized to achieve low-power FPGA implementation. Our measurements show that the proposed implementation yields 2.5x lower power consumption.

## I Introduction

Deep neural networks (DNNs) have surpassed the accuracy of conventional machine learning models in many challenging domains, including computer vision [1] and natural language processing (NLP) [2]. Recently, inspired by the successes in NLP, _transformers_[3] are adopted by the computer vision community. Built with self-attention layers, multi-layer perceptrons (MLPs), and skip connections, transformers make numerous breakthroughs on visual tasks [4]. To reduce the transformer model complexity, MLPMixers [5] which replace the multi-head self-attention module in transformers with a two-layer spatial MLP, are introduced.

Advances in building both general-purpose and custom hardware have been among the key enablers for shifting DNNs and MLPmixers from rather theoretical concepts to practical solutions for a wide range of problems [6]. Unfortunately, many DNN-based inference engines have a high latency cost and use enormous hardware resources which, in turn, prevent their deployment in latency-critical applications, especially on resource-constrained platforms. The high latency and large hardware cost are due to the fact that practical, high-quality deep learning models entail billions of arithmetic operations and millions of parameters, which exert considerable pressure on both the processing and memory subsystems.

Quantization has emerged as a promising model compression method where parameters and/or activations are replaced with low-precision, quantized, fixed-point values. Despite such a transformation, quantized models can match the accuracy of full-precision models utilizing 32-bit floating-point (FP-32) while requiring fewer data transfers and storage space. Early works on binary neural networks (BNNs) [7] and XNOR-net [8] demonstrated the potential advantages of extreme quantization, i.e., binarization. BNNs are 1-bit quantized models where all weights and activations are represented by two values, -1/0 and +1, significantly decreasing the memory footprint. Additionally, to speed up the inference, the multiplication/addition operations are switched out for less complex operations like the XNOR logical operation and bit count [8]. However, this superior performance is achieved at the cost of a significant accuracy drop in deep neural networks.

To address the issue of significant accuracy loss, some prior work [9, 10, 11, 12] propose to modify the well-known architectures (e.g. ResNet) as they show the network architecture can affect BNN performance. Although some of these works can achieve improved accuracy, their proposed models cannot be efficiently deployed on hardware platforms. For instance, ReActNet [11] significantly improves the accuracy by activation shifting and reshaping the MobileNet V1 [13] architecture at the cost of increasing the parameters such that the total number of parameters is about 30 million more than the number of parameters in MobileNet V2 [14].

This paper presents a novel building block called Blend module, which utilizes binary convolution on its main path and fixed-point convolution on its skip path. We show how the Blend module is used to produce BlendNet versions of some well-known vision models, such as ResNet and MLPMixer. The key contributions of this work can be summarized as follows.

* We present BlendNet, a hardware-friendly neural network architecture with binary activations, where all convolutional layers are computed using binary multiply-accumulate (BMAC) operations (except on skip paths that utilize fixed-point convolutions). On a ResNet-20-like architecture designed for the CIFAR-10 dataset, BlendNet outperforms the state-of-the-art binary-based implementation of FracBNN [12] by 0.8% in top-1 accuracy.
* We introduce a powerful and flexible compiler for mapping a given BlendNet inference engine running on any
dataset onto our optimized accelerator design by converting the network model to a computational graph, scheduling the graph's execution, and optimizing its nodes by leveraging intrinsic fusions of the convolution (activation function) and batch normalization layers.
* We present a flexible FPGA-based design that enables the DSP-based realization of BMAC operations. The reconfigurability of DSPs for performing bitwise logic operations is utilized to achieve a low-power implementation.
* We also apply our transformations, integrate our blocks to MLPMixers and improve the naively binarized MLPMixer models by 6%. To the best of our knowledge, this is the first paper that presents a binary model of MLPMixers with negligible accuracy drop.

## II Preliminaries

In this section, we briefly introduce the MLPMixer and then describe the conventional BNN model and architecture used for the hardware implementation. After that, we discuss some recent BNN architectures that have significantly improved accuracy.

### _MLPMixer_

To further reduce the inductive biases introduced by CNNs, MLP-Mixer has recently proposed a more straightforward solution that is fully based on multi-layer perceptrons (MLPs) [5] (see fig. 1). The basic layer in MLP-Mixer consists of two components: the channel-mixing block and the token-mixing block. Each of these mixing blocks has the same units and is shown in Fig. 0(b). In the channel-mixing block, the feature map is projected along the channel dimension for the communications among various channels; while the feature map is projected along the spatial dimension and communications among spatial locations are accomplished concurrently by the token-mixing block.

### _Conventional BNN_

BNNs have several properties that enable a more efficient mapping to FPGAs without affecting network accuracy. For implementing conventional BNN models which are fully binarized (both weights and activations are quantized to 1-bit values), the product of a binary weight and an activation can be replaced with binary XNOR operation.

Furthermore, by assuming that an unset bit represents -1 and a set bit represents +1, there are only two possible values of +1 and -1 for the result of the XNOR operation and, thus, synapse input. Therefore, the summation of a binary dot product can be implemented by a popcount operation that counts the number of set bits instead of accumulation with signed arithmetic. Using popcount approximately reduces the resource usage in terms of LUT and FF to half compared to using signed-accumulate. Furthermore, all BNN layers use batch normalization on convolutional or fully connected layer outputs, then apply the sign function to determine the output activation. For hardware implementation, the same output activation can be computed via thresholding [15]. Lastly, Max-pooling on binary activations can be implemented in hardware using the Boolean OR function.

### _Prior Work on State-of-the-art BNN Architectures_

The state-of-the-art BNN models achieved a high accuracy using blocks similar to ResNet models [16]. Martinez et al. [10] presented a strong baseline for BNN models which is based on the modified ResNet block suitable for 1-bit CNNs. They used double skip connections and PReLU (PReLU was first introduced in [17]) as the activation function. They follow the idea of using real-valued downsample layers proposed in [9] that improves the accuracy significantly. In addition, They tried to decrease the discrepancy among the output of the binary and the corresponding real-valued convolution. They proposed a real-to-binary attention matching that is proper for training 1-bit CNNs. They also presented a method wherein the architectural gap between real and binary networks is step by step bridged via a sequence of teacher-student pairs.

More recently, a more accurate BNN model is proposed called ReActNet [11] to mitigate the precision gap between the binarized model and its counterpart of real-valued. ReActNet took one step further and is based on MobileNetV1 [13] architecture. It reaches a Top1 accuracy of 69.4% in the IMAGENET data set using 4.82G BMACS with a 4.6 MB model size. The key block in ReActNet is a biased PReLU (BPReLU) activation function that shifts and reshapes the feature maps between two convolutional layers. This substantially improves the accuracy of the model. However, this model has as many as 29.3 million parameters (29.3 million bits).

To improve ReActNet [11], The authors in [12] have proposed FracBNN which employs a dual-precision activation scheme to compute features with up to two bits, using an additional sparse binary convolution layer. They have achieved MobileNetV2-level accuracy with competitive model size.

## III The Proposed Building Blocks

The proposed BNN model comprises two types of building blocks, as shown in Fig. 2: one with no operations on the skip path and another with average pooling, convolution, and batch normalization on its skip path. There are a few differences between the presented building blocks compared to prior work.

Fig. 1: Overview of MLPMixer model. (a) Overall MLPMixer and (b) mixing block architectures.
First, both types of building blocks include a batch normalization layer as their final output layer. This batch normalization layer, which does not include any data-driven, trainable channel rescaling/shifting parameters, ensures the output of each block is in a range centered around zero, and is, therefore, amenable to fixed-point or binary quantization. Adding channel rescaling/shifting to this batch normalization layer tends to reduce the classification accuracy by a few percentage points.

Second, a PReLU activation is placed in the main path and before the addition operation, which yields improvements in classification accuracy compared to other activation functions such as ReLU [10].

Last but not least, the building blocks are designed with the hardware implementation cost in mind. For example, as shown in Fig. 4, the final batch normalization layer of a building block can be fused into the next building block, resulting in a thresholding operation in the main path and modified convolutional weights in the skip path (see details in Sec. IV-A). A similar fusing can be performed for batch normalization in the skip path, all leading to reduced end-to-end inference latency. We extend our building blocks to the MLPMixer model and propose a new Mixing block, as shown in Fig. 3.

## IV Proposed Accelerator Design

In this section, we describe the proposed hardware accelerator and the associated compiler to performs optimization tailored to the employed accelerator design.

### _Compiler Optimization_

Our compiler performs optimizations tailored to the employed accelerator design and fuses operations to reduce the hardware cost and generate an intermediate graph. After such optimization and fusions, our compiler compiles the intermediate graph, extracts the required parameters for the accelerator design, and generates a static schedule.

All BNN layers proposed in this paper (cf. Fig 2) begin with a sign function (SG) and end with a batch normalization (BN) operation. First, we move the last BN block of a layer to the next layer and pass it through both feed-forward and skip paths of the next layer (cf. Fig. 3(b). Moreover, the BN block that comes before the SG block can be replaced by a thresholding (TH) function in order to reduce the hardware cost (c.f. Fig. 3(c)). Using such a technique, we can process the input feature map using an unsigned comparison and avoid expensive operations such as multiplication that are required in the BN block. Reference [15] explains how the hardware cost of a regular BN-SG block is reduced from 2 DSPs, 55 FFs, and 40 LUTs for separate BN and SG computations to merely 6 LUTs for the TH block computations using such a technique.

In addition, the BN block that is passed onto the skip path is also fused with the CONV layer in the skip path if one exists. In summary, we will replace a BN-CONV-BN sequence of layers on the skip path with a CONV layer (c.f. Fig. 3(c)). More details of such fusion is provided in the following. Note that the BN block that is passed onto the skip path is passed through the average pooling block in the inference (c.f. Fig. 3(b)). This simplification also reduces the chances of encountering overflow/underflow even if we assign fewer bits than required bits for summation because the weights/biases of the fused BN-CONV-BN layer will help normalize this layer's outputs. In other words, the output of BMAC can be considered 16 bits, and no quantization is required.

Algorithm 1 shows the well-known batch normalization algorithm where the two parameters \(\gamma\) and \(\beta\) are learned during the training process. Note that \(\epsilon\) is a small constant value used to ensure that division-by-zero error is not encountered.

```
Input \(\mathcal{B}=\{y_{0},y_{1},...,y_{m}\}\); \(\gamma\); \(\beta\) Output \(y^{\prime}_{i}=BN_{\gamma,\beta}(y_{i})\)
1:\(\mu_{\text{S}}\leftarrow\frac{1}{\gamma}\sum_{i=1}^{m}y_{i}\)// mini-batch mean
2:\(\sigma_{\text{B}}^{\gamma}\leftarrow\frac{y_{i}}{\gamma}\sum_{i=1}^{m}(y_{i}- \mu_{\text{B}})^{2}\)// mini-batch variance
3:\(\hat{y_{i}}\leftarrow\frac{y_{i}}{\sqrt{\sigma_{\text{B}}^{\gamma}+\epsilon}}\)// normalize
4:\(y^{\prime}_{i}\leftarrow\gamma\hat{y_{i}}+\beta\equiv BN_{\gamma,\beta}(y_{i})\)// scale and shift
```

**Algorithm 1** Batch normalization for activation \(y\) in a mini-batch, \(y^{\prime}\) is the normalized result

When the BN layer is fused into a subsequent convolutional

Fig. 3: The proposed Mixing block in this paper.

Fig. 2: The proposed building blocks. The differences with respect to real-to-binary blocks are highlighted in red.
layer, the fused layer's computations may be written as,

\[y^{\prime}=\mathbf{w}(\gamma\frac{\mathbf{x}-\mathbf{\mu}^{\prime}_{\text{ \emph{B}}}}{\sqrt{\mathbf{\sigma}^{\prime}_{\text{ \emph{B}}}^{2}+\epsilon}}+\mathbf{\beta}^{\prime})+\mathbf{b} \tag{1}\]

Hence, the new fused parameters can be calculated as:

\[\mathbf{w}^{\prime}=\frac{\mathbf{\gamma}^{\prime}\mathbf{w}}{\sqrt{\mathbf{\sigma}^{\prime}_{ \text{\emph{B}}}^{2}+\epsilon}},\quad\mathbf{b}^{\prime}=-\frac{\mathbf{\gamma}^{ \prime}\mathbf{w}\mathbf{\mu}^{\prime}_{\text{\emph{B}}}}{\sqrt{\mathbf{\sigma}^{\prime}_ {\text{\emph{B}}}^{2}+\epsilon}}+\mathbf{w}\mathbf{\beta}^{\prime}+\mathbf{b} \tag{2}\]

Note that \(\mathbf{x}\) in Eqn. 1 is the output of previous layer, so the 4-D \(\mathbf{w}\) must be squeezed into a 2-D vector of size (\(c_{\text{in}}\), \(c_{\text{out}}\)\(\times\)\(w_{\text{k}}\)\(\times\)\(h_{\text{k}}\)). Indeed, if we use zero padding in convolutions, we will have 0 values entering the convolution after the BN. When we fuse the parameters, this must still be the case. So, to correctly fold \(\mathbf{w}\) into the 2-D vector, we have to replace the default 0 values for padding by \(\mathbf{\mu}^{\prime}_{\text{\emph{B}}}-\frac{\mathbf{\beta}^{\prime}\sqrt{\mathbf{\sigma} ^{\prime}_{\text{\emph{B}}}^{2}+\epsilon}}{\mathbf{\gamma}^{\prime}}\), that is, we must apply the BN transformation to the padding. Hence, by multiplying the fused weights \(\mathbf{w}^{\prime}\) to the padding values, we apply the inverse of the BN transformation.

Using algorithm 1 and proceeding with fused parameters as in the previous case, the weights and bias of the resulting fused BN-CONV-BN block may be expressed as:

\[\mathbf{w}^{\prime\prime}=\frac{\mathbf{\gamma}^{\prime\prime}\mathbf{w}^{\prime}}{\sqrt{ \mathbf{\sigma}^{\prime\prime}_{\text{\emph{B}}}^{2}+\epsilon}},\quad\mathbf{b}^{ \prime\prime}=\mathbf{\beta}^{\prime\prime}+\mathbf{\gamma}^{\prime\prime}\frac{\mathbf{ b}^{\prime}-\mathbf{\mu}^{\prime\prime}_{\text{\emph{B}}}}{\sqrt{\mathbf{\sigma}^{\prime \prime}_{\text{\emph{B}}}^{2}+\epsilon}} \tag{3}\]

### _Accelerator Architecture_

We adopted a heterogeneous streaming architecture where each layer uses its own hardware resources for this work.

Our Accelerator consists of the following types of operations:

* \(3\times 3\) BNN convolution
* \(1\times 1\) FPNN convolution
* Average pooling
* Max Pooling
* Linear layer
* Thresholding
* BN-PReLU/BN-PReLU-BN
* Residual connection and summation

We have designed and implemented an efficient accelerator that supports these operations on FPGA devices. We separate our design into three domains: binary, fixed-point and joint domains. The BNN (FPNN) convolutions are placed in binary (fixed-point) domains whereas all other operations are in the joint domain. Our hardware is designed by using Xilinx's High Level Synthesis (HLS) tools. In the following, we will describe the hardware engines and the used HLS design techniques in detail.

#### Iv-B1 BNN and FPNN Block

The BMAC/FPMAC of the proposed accelerator are mapped to DSPs. Each DSP block (i.e., DSP48E2 in Xilinx FPGA) is capable of performing a 18 by 27-bit or a 48-bit bitwise logic operation including AND, OR, NOT, NAND, NOR, XOR, and XNOR. We used this feature for performing 48 XNOR operations simultaneously as one BMAC operation. Note that similar 48 XNOR operations on LUTs results in the usage of 48 LUTs and 49 FFs. So, the input feature map and weights must be packed into groups of size 48. For this purpose, We pack bits along the channel dimension into 48-bit unsigned integers for concurrent access. We will highlight the advantage of mapping logic operations to DSPs in Section V-C.

Our design results in a balanced usage of hardware resources since LUTs are mostly used for other operations while DSPs are used to do BMAC/FPMAC operations. Note that reconfigurability of DSPs in each clock cycle can be used to switch between BMAC and FPMAC operations. This is very helpful in the case of a homogeneous single execution engine design where we only use a fixed set of resources for all layers, although this is not in the scope of the present paper. Note that option can be used in single execution engine architecture. In the streaming architecture, each unit has its own resource. A 2D array of DSPs with the size of 32 * 32 is designed to perform FPMAC operations.

#### Iv-B2 FP/B Joint Blocks

In addition to the accelerators for the convolution operations, which account for the majority of the computations in a vision neural network, we also implement hardware accelerators for other operations that must be performed in the joint domain, including the max (average) pooling and thresholding blocks. For instance, the TH unit compares each output activation from the previous layer with a programmable threshold and outputs +1 (0) if the output activation is greater (smaller) than the corresponding threshold. Since these TH blocks only contain channel-wise parameters, their impact on the total number of model parameters is negligible. Although we can achieve the maximum concurrency by processing all output activations simultaneously, this would require a lot of resources, including both memories (we must increase the number of access lines by partitioning the data to several memory blocks) and TH block. The performance gain may not be worth the cost of additional resources needed. We use the greatest common divisor (GCD) of the height of the systolic array in the FPNN block (e.g., 32) and the width of the array of BMACs in the BNN block (e.g., 48) as the parallelism factor for all blocks in the joint domain.

## V Experimental Results

For evaluation purposes, we targeted a high-end Virtex(r) UltraScale+ FPGA (Xilinx VU9P FPGA, which is available in the cloud as the AWS EC2 F1 instance). This FPGA contains approximately 2.5 million logic elements and approximately 6,800 DSP units.We use Vitis 2020.2 for mapping to the FPGA and set the target frequency to 340 Mhz. We also use the Vivado power report provided by Xilinx to assess the power consumption of each design. Finally, we evaluate our proposed method on a well-known CNN, i.e., ResNet-20 [16] and MLPMixers [5] and a commonly used computer-vision dataset for object recognition i.e., the CIFAR-10 [18] dataset.

### _Experimental setup_

In the case of MLPMixers, the resolution of the input image is 32*32, and the patch size that the experiments are based on is 4*4. So, we have S non-overlapping image patches that are mapped to a hidden dimention \(C\). \(D_{S}\) and \(D_{C}\) are tunable hidden widths in the token-mixing and channel-mixing MLPs,
respectively. The summary of design specifications is shown in Table I.

### _ResNet20_

Table II demonstrates the superiority of BlendNet on ResNet20 compared to other binary-based approaches. We can observe that BlendNet achieves a higher top-1 accuracy by 0.8% compared to the state-of-the-art FracBNN and improves the accuracy by 1.5% to about 9% compared to the other approaches. BlendNet-20 achieves the same model size as FracBNN even with keeping the first and last layer full precision (i.e., 16-bit fixed-point).

### _Hardware cost and performance of BlendNet_

In this section, we evaluate our BlendNet-20 model on the FPGA platform. Compared to the BNN accelerator in FracBNN [12], our design achieves a higher frame rate (3846 FPS compared to 2807 FPS reported in FracBNN [12]) and higher working frequency (342 MHz compared to 250 MHz) while yielding higher accuracy. Note that the FPGA used in this paper is a server-class FPGA while authors in [12] deployed their design on an embedding FPGA with fewer resources. However, the frame rate comparison is fair since they unroll the entire model, similar to what we achieve. The unrolling is feasible because ResNet-20-based models on CIFAR-10 are very compact and can be fitted into an FPGA to eliminate unnecessary transactions between the logic blocks and the DDR memory.

Table III compares the hardware cost of two approaches for implementing the BlendNet-20 on the CIFAR-10 dataset where the DSP-based BlendNet-20 is the approach presented in Section IV-B2. The LUT-based is a naive implementation where logic operations are performed using LUTs. Note that DSP usage in LUT-based implementation is for the first and last layers. Decreasing the LUT usage results in power saving. Our measurements show that the proposed implementation yields 2.5x lower power consumption.

\begin{table}
\begin{tabular}{l c} \hline \hline Approach & Accuracy (\%) \\ \hline DoReFa-Net [19] & 79.3 \\ DSQ [20] & 84.1 \\ ReActNet [11] & 85.8 \\ IR-Net [21] & 86.5 \\ FracBNN [12] & 87.2 \\ \hline
**BlendNet** & **88.0** \\ \hline \hline \end{tabular}
\end{table} TABLE II: Classification accuracy of ResNet20-like models on CIFAR-10.

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline
**Specification** & S/4 & B/4 & 2S/4 \\ \hline
**Sequence length S** & 64 & 64 & 64 \\ \hline
**Hidden size C** & 128 & 192 & 256 \\ \hline
**Patch resolution** & 4*4 & 4*4 & 4*4 \\ \hline
**MLP dimention \(D_{C}\)** & 512 & 768 & 1024 \\ \hline
**MLP dimention \(D_{S}\)** & 64 & 96 & 128 \\ \hline
**Number of layers** & 8 & 12 & 8 \\ \hline \hline \end{tabular}
\end{table} TABLE I: Summary of design specifications for MLPMixers used in this paper. The “S” and “B” (small and base) models scales down follow Tolstikhin et al. [5]. The notation “B/4” means the model of base scale with patches of resolution 4*4.

Fig. 4: (a) The proposed building block. (b) Distribute the BN from the precedent BlendNet block to both main and skip paths. (c) Merge components that result in the optimized BlendNet block. Note that the red and yellow dash boxes show the precedent and current building blocks. The green dash boxes show the potential components for merging.

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|} \hline
**Approach** & DSP482 & LUT & FF & BRAM\_18K & \(f_{target}\) & Power \\ \hline
**DSP-based** & 2240 & 10K & 30K & 912 & 342 & 6.1 \\
**BlendNet-20** & (32\%) & (1\%) & (1\%) & (42\%) & MHz & **w** \\ \hline
**LUT-Based** & 1132 & 500K & 900K & 912 & 342 & 15.3 \\
**BlendNet-20** & (17\%) & (50\%) & (34\%) & (42\%) & MHz & **w** \\ \hline \hline \end{tabular}
\end{table} TABLE III: Comparison between the hardware metrics of different types of BlendNet-20 implementation on CIFAR-10.
### _MLPMixers_

As table IV shows, our model (i.e., BlendMixer) outperforms the naively binarized version of MLPMixer (i.e., BinaryMixer) yet achieves a smaller memory footprint (i.e., model size). Increasing the model size (e.g., model B/4) reduces the accuracy drop due to binarization. Note that FPMAC operations in both BinaryMixer and BlendMixer are due to the patch embedding block and the last FC layer. When it comes to analyzing the performance, BlendMixer can achieve 0.02ms (0.06ms) latency for a small (base) model while MLPMixer can process an image in 0.24ms (1.14ms).

### _Ablation study with normalization unit in mixing blocks_

To further evaluate the influence of the normalization unit on the proposed Mixing blocks, we performed an ablation study with different normalization units, layer normalization as suggested in the paper, batch normalization over channels and batch normalization over patches. As shown in Table V, the Mixing block with batch normalization over channels outperforms other models. Using batch normalization also reduces the memory footprint, requiring storing fewer parameters. Note that the MLPMixer consists of the proposed mixing blocks is called BlendMixer.

### _Computation/memory cost vs accuracy in mixing blocks_

In this section, we assess the trade-off between computation/memory complexity and result accuracy. As demonstrated in Table VI, we can manipulate the type of computation in FC layers of mixing blocks to improve the accuracy at the cost of computation and memory complexity. The BlendMixer(BB/BB-2S/4) is simply the widened version of BlendMixer S/4, and BlendMixer(BB/FPB-2S/4) is the same as the previous architecture with the exception of that the FC layer of the second mixing block being calculated in FP format. The other models are named accordingly. The BlendMixer(BB/FPB-2S/4) achieves a comparable accuracy (less than 1% accuracy drop compared to MLPMixer S/4).

## VI Conclusion

This paper introduces BlendNet, an innovative neural network design that makes use of the Blend module, a new building block that performs binary and fixed-point convolutions in the main and skip paths, respectively. Batch normalizations are deployed intelligently on both the main and skip paths within the Blend module as well as between adjacent Blend modules. BlendNet-20, a descendant of ResNet-20 trained on the CIFAR-10 dataset, achieves 88.0% classification accuracy (0.8 percent better than the state-of-the-art), yet 1.4x higher throughput. BlendMixer outperforms the naively binarized version of MLPMixer yet achieves a smaller memory footprint.
