MobileNets family of computer vision neural networks have fueled tremendous progress in the design and organization of resource-efficient architectures in recent years. New applications with stringent real-time requirements in highly constrained devices require further compression of MobileNets-like already computeefficient networks. Model quantization is a widely used technique to compress and accelerate neural network inference and prior works have quantized MobileNets to 4 − 6 bits albeit with a modest to significant drop in accuracy. While quantization to sub-byte values (i.e. precision ≤ 8 bits) has been valuable, even further quantization of MobileNets to binary or ternary values is necessary to realize significant energy savings and possibly runtime speedups on specialized hardware, such as ASICs and FPGAs. Under the key observation that convolutional filters at each layer of a deep neural network may respond differently to ternary quantization, we propose a novel quantization method that generates per-layer hybrid filter banks consisting of full-precision and ternary weight filters for MobileNets. The layer-wise hybrid filter banks essentially combine the strengths of full-precision and ternary weight filters to derive a compact, energy-efficient architecture for MobileNets. Using this proposed quantization method, we quantized a substantial portion of weight filters of MobileNets to ternary values resulting in 27.98% savings in energy, and a 51.07% reduction in the model size, while achieving comparable accuracy and no degradation in throughput on specialized hardware in comparison to the baseline full-precision MobileNets. Deeper and wider convolutional neural networks (CNNs) has led to outstanding predictive performance in many machine learning tasks, such as image classification ; Krizhevsky et al. (2012) ), object detection ; Ren et al. (2015) ), and semantic segmentation ; Long et al. (2015) ). However, the large model size and corresponding computational inefficiency of these networks often make it infeasible to run many real-time machine learning applications on resource-constrained mobile and embedded hardware, such as smartphones, AR/VR devices etc. To enable this computation and size compression of CNN models, one particularly effective approach has been the use of resource-efficient MobileNets architecture. MobileNets introduces depthwise-separable (DS) convolution as an efficient alternative to the standard 3-D convolution operation.While MobileNets architecture has been transformative, even further compression of MobileNets is valuable in order to make a wider range of applications available on constrained platforms (Gope et al. (2019) ). Model quantization has been a popular technique to facilitate that. Quantizing the weights of MobileNets to binary (-1,1) or ternary (-1,0,1) values in particular has the potential to achieve significant improvement in energy savings and possibly overall throughput especially on custom hardware, such as ASICs and FPGAs while reducing the resultant model size considerably. This is attributed to the replacement of multiplications by additions in binary-and ternary-weight networks. Multipliers occupy considerably more area on chip than adders ), and consume significantly more energy than addition operations (Horowitz (2014) ; Andri et al. (2018) ). A specialized hardware can therefore trade off multiplications against additions and potentially accommodate considerably more adders than multipliers to achieve a high throughput and significant savings in energy for binary-and ternary-weight networks. However, prior approaches to binary and ternary quantization (Rastegari et al. (2016) ; Alemdar et al. (2016) ; ; Tschannen et al. (2018) ) incur significant drop in prediction accuracy for MobileNets. Recent work on StrassenNets (Tschannen et al. (2018) ) presents a more mathematically profound way to approximate matrix multiplication computation (and, in turn, convolutions) using mostly ternary weights and a few full-precision weights. It essentially exploits Strassen's algorithm to approximate a matrix multiplication of a weight matrix with feature maps, where the elements of the product matrix are generated by different combination of few intermediate terms through additions. Computation of each of the intermediate terms requires a multiplication along with combination of different elements of weights and feature maps through additions. The number of intermediate terms (also called hidden layer width) in StrassenNets therefore determines the addition and multiplication budget of a convolutional layer and in turn decides the approximation error of the corresponding convolution operation. While the results in (Tschannen et al. (2018) ) using StrassenNets demonstrates no loss in predictive performance when compared to full-precision models for few networks, the effectiveness of StrassenNets is quite variable, however, depending on the neural network architecture. We observe, for example, that while strassenifying is effective in reducing the model size of DS convolutional layers, this might come with a prohibitive increase in the number of addition operations, reducing the energy efficiency of neural network inference. The exorbitant increase in additions primarily stems from the use of wide hidden layers for closely approximating each convolutional filter in a network layer. While this might be required for some of the convolutional filters in a layer, our observations indicate that all filters may not require wide strassenified hidden layers. As different filters in a network layer tend to capture different features, they may respond differently to ternary quantization, and, in turn, to strassenified convolution with a specific hidden layer units. Some filters can be harder to approximate using ternary bits than others, and have larger impact on the model accuracy loss. Furthermore, given a constrained hidden layer budget for StrassenNets, a group of filters extracting fairly similar features at a layer may respond favorably to ternary quantization, while other filters of the layer extracting significantly different features from those may not. Guided by these insights, we propose a layer-wise hybrid filter banks for the MobileNets architecture capable of giving start-of-the-art accuracy levels, while requiring a fraction of the model size and considerably fewer MAC and multiplication operations per inference. The end-to-end learning of hybrid filter banks makes this possible by keeping precision critical convolutional filters in fullprecision values and strassenifying quantization tolerant filters only to ternary values. The filters that are most sensitive to quantization errors perform traditional convolutions with input feature maps, whereas ternary quantization tolerant filters can perform strassenified convolutions using narrow hidden layers. We apply this proposed quantization scheme to the state-of-the-art MobileNets-V1 architecture. The hybrid filter banks for MobileNets achieves a 46.4% reduction in multiplications, and a 51.07% reduction in model size while incurring modest increase in additions. This translates into a 27.98% savings in energy required per inference while ensuring no degradation in throughput on a DNN hardware accelerator consisting of both MAC and adders when compared to the execution of baseline MobileNets on a MAC-only hardware accelerator. The hybrid filter banks accomplishes this with a very minimal loss in accuracy of 0.51%. To the best of our knowledge, the hybrid filter banks proposed in this work is a first step towards quantizing the already compute-efficient MobileNets architecture to ternary values with a negligible loss in accuracy on a large-scale dataset, such as ImageNet. The remainder of the paper is organized as follows. Section 2 elaborates on the incentives behind the use of per-layer hybrid filter banks for the MobileNets architecture and provides a brief overview of current quantization algorithms along with our observations of applying them to the MobileNets architecture. Failing to find a good balance between accuracy and computation costs shifts our focus towards designing layer-wise hybrid filter banks for MobileNets. Section 3 describes our hybrid filter banks. Section 4 presents results. Section 5 compares hybrid filter banks against prior works and Section 6 concludes the paper. In this work, we propose per-layer hybrid filter banks for MobileNets capable of quantizing its weights to ternary values while exhibiting start-of-the-art accuracy on a large-scale dataset and requiring a fraction of the model size and considerably lower energy per inference pass. We use 16-bit floating-point format to represent the intermediate activations and traditional weight filters of hybrid filter banks in this work. In future, we plan to explore the impact of quantizing them to 8-bit or less. In addition, it will be interesting to see how channel pruning (He et al. (2018) ; Zhuang et al. (2018) ) assists in reducing the computational complexity of strassenified MobileNets. Strasssen's algorithm can multiply 2 × 2 matrices using only 7 multiplications instead of 8 required otherwise by a naïve matrix multiplication algorithm. Figure 3 (a) specifies a set of weight matrices that can perform exact convolution of the 2 × 2 filter bank comprising f j and f k with the feature map using 7 multiplications. Note that the two filters f j and f k do not have any common values. However, owing to the presence of common value of a between f j and f k filters in Figure 3 (b), Strassen's algorithm now can compute the exact product matrix using only 6 multiplications instead of 7 required otherwise in Figure 3 (a). A set of ternary weight matrices implementing an exact convolution in this case is shown in Figure 3(b ) . B RELATION OF PER-LAYER HYBRID FILTER BANKS TO GOOGLENET ARCHITECTURE. The per-layer hybrid filter banks proposed here is inspired by the Inception module from the GoogLeNet architecture (Szegedy et al. (2015) ). In a traditional convolutional network, each layer extracts information from the previous layer in order to transform the input data into a more useful representation. However, salient features of an input volume can have extremely large variation in size. Because of this variation in the size of the required information, choosing the right kernel size
2x savings in model size, 28% energy reduction for MobileNets on ImageNet at no loss in accuracy using hybrid layers composed of conventional full-precision filters and ternary filters Focuses on quantizing the MobileNets architecture to ternary values, lowering the required space and computation in order to make neural networks more energy efficient. The paper proposes layer-wise hybrid filter bank which only quantizes a fraction of convolutional filters to ternary values towards the MobileNets architecture.