Efficient and Effective Methods for Mixed Precision Neural Network
  Quantization for Faster, Energy-efficient Inference
For efficient neural network inference, it is desirable to achieve
state-of-the-art accuracy with the simplest networks requiring the least
computation, memory, and power. Quantizing networks to lower precision is a
powerful technique for simplifying networks. As each layer of a network may
have different sensitivity to quantization, mixed precision quantization
methods selectively tune the precision of individual layers to achieve a
minimum drop in task performance (e.g., accuracy). To estimate the impact of
layer precision choice on task performance, two methods are introduced: i)
Entropy Approximation Guided Layer selection (EAGL) is fast and uses the
entropy of the weight distribution, and ii) Accuracy-aware Layer Precision
Selection (ALPS) is straightforward and relies on single epoch fine-tuning
after layer precision reduction. Using EAGL and ALPS for layer precision
selection, full-precision accuracy is recovered with a mix of 4-bit and 2-bit
layers for ResNet-50, ResNet-101 and BERT-base transformer networks,
demonstrating enhanced performance across the entire accuracy-throughput
frontier. The techniques demonstrate better performance than existing
techniques in several commensurate comparisons. Notably, this is accomplished
with significantly lesser computational time required to reach a solution.
Deepika Bablani, Jeffrey L. Mckinstry, Steven K. Esser, Rathinakumar Appuswamy, Dharmendra S. Modha
# Efficient and Effective Methods for Mixed Precision Neural Network

###### Abstract

For effective and efficient deep neural network inference, it is desirable to achieve state-of-the-art accuracy with the simplest networks requiring the least computation, memory, and power. Quantizing networks to lower precision is a powerful technique for simplifying networks. It is generally desirable to quantize as aggressively as possible without incurring significant accuracy degradation. As each layer of a network may have different sensitivity to quantization, mixed precision quantization methods selectively tune the precision of individual layers of a network to achieve a minimum drop in task performance (e.g., accuracy). To estimate the impact of layer precision choice on task performance two methods are introduced: i) Entropy Approximation Guided Layer selection (EAGL) is fast and uses the entropy of the weight distribution, and ii) Accuracy-aware Layer Precision Selection (ALPS) is straightforward and relies on single epoch fine-tuning after layer precision reduction. Using EAGL and ALPS for layer precision selection, full-precision accuracy is recovered with a mix of 4-bit and 2-bit layers for ResNet-50 and ResNet-101 classification networks, demonstrating improved performance across the entire accuracy-throughput frontier, and equivalent performance for the PSPNet segmentation network in our own commensurate comparison over leading mixed precision layer selection techniques, while requiring orders of magnitude less compute time to reach a solution.

Machine Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Learning, Deep Learning, Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Learning, Deep Learning, Deep Learning, Deep Learning, Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Learning, Deep Learning, Deep Learning, Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Deep Learning, Learning
Mixed precision quantization is challenging as the search space is prohibitively large. For a network with \(L\) layers and \(n\) different precision choices per layer, \(n^{L}\) different network configurations exist, making brute force precision selection for each layer impractical. Various methods are proposed in literature to mitigate this by using metrics that try to estimate the impact of quantization of individual layers on overall task performance (Dong et al., 2019, 2020; Yao et al., 2021; Chen et al., 2021; Liu et al., 2021b), but they often target different criteria to optimize for, e.g. memory footprint or throughput, and fine-tune the resulting network with diverse algorithms. Hence, establishing a principled, direct comparison across quantization metrics based on reported results is not straightforward. The computational cost of many of these methods, combined with their reported inability to recover full precision accuracy at lower precisions (see Table 1 for results on the ResNet-50 benchmark for image classification (Deng et al., 2009)), underscore the need for i) new algorithms to solve this problem, and ii) a unifying framework for commensurate comparison.

Here, we introduce _Accuracy-aware Layer Precision Selection (ALPS)_ and _Entropy Approximation Guided Layer selection (EAGL)_, two efficient and effective new methods for choosing the bit-width configuration for network layers.

The most straightforward way to estimate the contribution of each layer to overall network accuracy, assuming independent contributions, is to lower the precision of each layer one at a time, fine-tune briefly, e.g. for one epoch, and record the accuracy difference as \(G_{l}\), the accuracy gained when keeping layer \(l\) at a higher precision. Intuitively, the network accuracy can be optimized by keeping the layers which show the largest gain in accuracy, \(G_{l}\) at higher precision, and choosing the layers with low \(G_{l}\) for further quantization. This method is referred to as Accuracy-aware Layer Precision Selection (ALPS). Given a fixed dataset, the computational complexity is proportional to the number of layers. For example, the time taken for ResNet-50 is approximately the same as training the network for 50 epochs. ALPS outperforms HAWQv3, a state-of-the-art mixed precision layer selection technique, while being relatively efficient to compute.

EAGL, on the other hand, is built upon the insight that the complexity required by a given layer to achieve good performance can be estimated using the entropy of the empirical distribution of its parameters after training, assuming appropriate regularization. This measure captures the relative frequency of trained layer parameters falling in each quantized bin. If many quantized weights fall in a few of the available bins, the layer is a good candidate for further quantization; whereas if the distribution of quantized weights is more uniform across bins, further compression by quantization can be detrimental to task performance. So, the gain in accuracy \(G_{l}\), of keeping a layer at higher precision is expected to be directly proportional to the entropy of the empirical weight distribution post training. Our formulation relies on empirical estimates for the marginal distributions of layer parameters, and thus, given a trained network, EAGL is a computationally inexpensive technique that does not require access to the training dataset, and can perform as well as the more direct accuracy aware method described above.

Once the layer-wise accuracy gain estimates as described above (using ALPS, EAGL, or another mixed precision technique from the literature), the computational cost of each

\begin{table}
\begin{tabular}{l c c c} \hline \hline \multirow{2}{*}{Method} & Top 1 & \multirow{2}{*}{Compression Ratio} & \multirow{2}{*}{BOPS} \\  & Accuracy & & \\  & Drop & & \\ \hline ALPS & **-0.22** & & \\ (Ours) & **(76.13-76.35)** & & \\ \hline EAGL & **-0.17** & & \\ (Ours) & **(76.13-76.30)** & & \\ \hline HAQ & \begin{tabular}{c} 0.67 \\ (76.15 - 75.48) \\ \end{tabular} & 10.57\(\times\) & 520 \\ \hline \multirow{2}{*}{\begin{tabular}{c} Chen et al. \\ \end{tabular} } & \begin{tabular}{c} 0.85 \\ (76.13-75.28) \\ \end{tabular} & 12.24\(\times\) & - \\ \hline HAWQ-v3 & \begin{tabular}{c} 0.99 \\ (77.72-76.73) \\ \end{tabular} & - & 154 \\ \hline HAWQ-v2 & \begin{tabular}{c} 1.63 \\ (77.39-75.76) \\ \end{tabular} & 12.24\(\times\) & - \\ \hline HAWQ & \begin{tabular}{c} 1.91 \\ (77.39-75.48) \\ \end{tabular} & 12.28\(\times\) & - \\ \hline AutoQ & \begin{tabular}{c} 2.29 \\ (74.80-72.51) \\ \end{tabular} & 10.26\(\times\) & - \\ \hline Sun et al. & \begin{tabular}{c} 0.93 \\ (77.15-76.22) \\ \end{tabular} & 8.00\(\times\) & - \\ \hline Yu et al. & 
\begin{tabular}{c} 1.85 \\ (77.56-75.71) \\ \end{tabular} & 11.03\(\times\) & - \\ \hline \hline \end{tabular}
\end{table}
Table 1: **EAGL and ALPS demonstrate state-of-the-art results for classification (Deng et al., 2009) with ResNet-50 compared to techniques from literature**. Top-1 Accuracy Drop is the accuracy gap between the full precision (FP32) baseline and the mixed precision (MP) network. Negative value indicate that the MP network performs better than the FP32 network. Numbers in bracket indicate FP32 and MP scores, respectively. Compression Ratio is model compression w.r.t FP32 weights. BOPS is Giga-Bit Operations (Yao et al., 2021). HAQ keeps activations at FP32. “-” indicates data not published. FP32 accuracy is different for each method as authors use different checkpoints as the starting point. This highlights a need for a commensurate comparison framework, as an apples-to-apples comparison is challenging.
layer, and a maximum computational budget are determined, the next step is to select the precision of each layer from two or more precision choices in order to maximize the estimated task performance of the resulting network without exceeding the computational budget. In the experiments for this paper, a choice is made between two precisions, 4-bit and 2-bit, for each layer. Both weights and activations in a layer and quantized to the chosen precision. With the assumption that the sum of the layer-wise accuracy contributions reflects the total task accuracy of the network1, this problem of precision selection of the layers of a model is formulated formally as follows:

Footnote 1: See Appendix B for experiments demonstrating the additive nature of layer-wise accuracy contributions

Maximize \(\sum_{l=1}^{L}G_{l}P_{l}\), i.e., the total accuracy gained while keeping \(\sum_{l=1}^{L}C_{l}\leq B\), where \(L\) is the number of layers, \(G_{l}\) is the accuracy gain per layer, \(P_{l}\) is 1 if the precision of layer \(l\) is kept at the higher precision, and 0 otherwise, \(C_{l}\) is the computational cost of layer \(l\) in cycles, and \(B\) is the overall computational budget in cycles.

We solve this optimization problem efficiently by formulating it as a 0-1 Integer Knapsack Problem from combinatorial optimization (Martello and Toth, 1990). Finally, the layer precision choices from the optimization step are used to create a mixed precision network which is fine-tuned until convergence to get the final task performance. This process is depicted in Figure 1.

Using this evaluation framework, it is demonstrated that ALPS and EAGL outperform leading mixed precision layer selection techniques like HAWQ-v3 (Yao et al., 2021) and achieve state-of-the-art accuracy using ResNet-50 and ResNet-101 networks (He et al., 2016) for image classification (Deng et al., 2009) and PSPNet (Zhao et al., 2017) for semantic segmentation (Cordts et al., 2016) using 2-bit and 4-bit layer mixed precision networks.

The main contributions of this paper are (i) two elegant measures for estimating the impact of layer precision on task performance that achieve state-of-the-art results for mixed precision neural networks, and (ii) a framework for fair comparison across varying approaches that decomposes the bit-width configuration selection problem into three steps - accuracy gain estimation, layer-wise precision selection for a given budget using 0-1 Integer Knapsack optimization algorithm, and fine-tuning of resulting mixed precision network.

## 2 Related Work

Methods for optimizing the layer-wise precision of deep neural networks can be divided into several categories. Network Architecture Search methods can naturally incorporate precision as an additional variable but are generally computationally expensive (Wu et al., 2018; Chen et al., 2018; Yu et al., 2020; Sun et al., 2021). HAQ (Wang et al., 2019) and AutoQ (Lou et al., 2019), use reinforcement learning (RL) in the search process. A recent technique (Liu et al., 2021) uses RL in a more efficient manner and employs a separate network trained with RL which learns the distribution of precisions which work well as the network being optimized is trained with random, layer-wise precisions. There are also methods which learn the precision of each layer using stochastic gradient descent (SGD) with a modified loss function (Nikolic et al., 2020; Yang and Jin, 2020). Hessian Aware methods, like HAWQ (Dong et al., 2019) and its successors, HAWQ-v2 (Dong et al., 2020), HAWQ-v3 (Yao et al., 2021), and Chen et al. (Chen et al., 2021) evaluate the curvature of the loss function for each layer to predict, given the quantization error of that layer, how much quantization will increase the loss. HAWQ-v3 achieves state-of-the-art in network compression and throughput. A recent accuracy aware method (Liu et al., 2021) tries to estimate the accuracy gain for each layer using imprinting, which uses one epoch of fine-tuning per layer. Although the authors use insights from few-shot learning to speed up the layer-wise training, a drawback of this method is that it requires network surgery, removing downstream layers and replacing them with a single linear layer, thus evaluating each layer out of context.

## 3 Methods

Mixed precision quantization is based on the intuition that different layers in a network have different sensitivity to quantization and hence some layers are better candidates for more aggressive quantization than others. In particular, we are interested in solving the following problem.

Problem formulationGiven a choice between two precisions, \(b_{1}\) and \(b_{2}\) for each of the layers in a network, find the bit-width configuration for all the layers that achieves the best task performance while remaining within a given computational budget for the network.

The budget can be any target metric evaluating performance on a hardware inference platform, for instance, the memory footprint, latency, power, or a combination of multiple metrics. While the problem is formulated as a binary choice problem above, where the precision of a layer can be one of two precision choices, \(b_{1}\) or \(b_{2}\), this can be extended to the case of more than two precision choices as well. A model is considered for which a layer's computational cost is linear in its bit-width, but the techniques are equally applicable to other cost models such as quadratic.

The evaluation methodology, outlined in the next subsection, provides a unifying framework for commensurate
comparison between various popular mixed precision techniques from literature, as well as the novel layer selection methods introduced in this work. As emphasized earlier, there is a need for a unifying framework for evaluating techniques for mixed precision layer selection, as the lack of an apples-to-apples comparison between different methods proposed in literature and different objectives being optimized for, each with its own merits and limitations, makes it very challenging for a practitioner to select a candidate that is the most suitable for their problem of interest due to the absence of a coherent comparison methodology. It is hoped that the framework for comparison proposed here serves as a benchmark for evaluation of future methods as well.

### Evaluation Methodology

The proposed evaluation framework (Figure 1) is parameterized by a task, a network architecture, a target computational budget, and a fine-tuning recipe. It takes as input a per layer scalar measure of the expected relative task performance improvement when layer \(l\) is quantized at precision \(b_{1}\) instead of a lower precision \(b_{2}\), which is referred to as accuracy gain, \(G_{l}\). As an example, this is the accuracy gained by quantizing a layer in a classification network at 4-bit instead of 2-bit. The higher the gain, the more desirable it is to keep this layer precision at 4-bit instead of 2-bit. Different methods for mixed precision quantization propose different ways of quantifying this cost per layer. This is the key distinction between techniques evaluated in this framework.

Once the layer-wise gain estimates are obtained, the optimization problem described in the introduction is then solved in order to determine the precision setting for each layer. The details of the optimization step are described in the Appendix A. With the choice of precision for each layer from the optimizer, a mixed precision network is created with this configuration, which is then fine-tuned, to get the task performance of the mixed precision network chosen by the technique. The resulting task performance is the final output of the evaluation.

For comparison across different mixed precision techniques, each method provides an accuracy gain, \(G_{l}\) for each layer of a network, and then follows the above standardized evaluation protocol. For all of the experiments in this paper, 4-bit and 2-bit mixed precision choices are used in the problem setup because either previous published works have already demonstrated full precision accuracy for the networks under consideration at 4-bit (McKinstry et al., 2019; Esser et al., 2020), or fine-tuning with the proposed training recipe was sufficient to recover the task performance of a full precision network using a 4-bit network.

In all experiments, by sampling budgets in between the throughput of a 4-bit and a 2-bit network, multiple points along the throughput-accuracy frontiers were sampled for each technique, thereby demonstrating a very compelling test framework to compare different techniques across a wide variety of budgets in an unbiased manner.

The next two subsections describe the two methods introduced here for the problem of mixed precision layer selection described above - specifically the key sub-problem of accurately estimating accuracy gains, \(G_{l}\) for each network layer \(l\) - followed by implementation details specific to the evaluation framework. In the following section, the techniques are compared with state-of-the-art mixed precision techniques from literature and demonstrate superior performance, not only for a fixed budget, but along the entire frontiers.

### Accuracy-aware Layer Precision Selection for quantization (ALPS)

Most published approaches that propose solutions to the mixed precision layer selection problem are indirect proxies for what is really needed - an estimate for the accuracy contribution of each layer _after_ quantization aware training (Yao et al., 2021). In (Liu et al., 2021), the authors do measure the accuracy of a fine-tuned network, but indirectly using a second network. Here, we propose an accuracy aware method which directly measures how well each layer can be fine-tuned. The intuition is that layers that have the highest accuracy gain when quantized to a higher precision \(b_{1}\) (here, 4-bit) instead of a lower precision \(b_{2}\) (here, 2-bit) are good candidates to be kept at \(b_{1}\) in a mixed precision network. This approach is an intuitive, simpler variant of (Liu et al., 2021).

Algorithm 1 describes ALPS. For a given network, the default training parameters used for training the higher precision (here, 4-bit) model are used. Starting from a fully
trained 4-bit model, the precision of each convolution layer is lowered from 4-bit to 2-bit, one at time, and the resulting network is fine-tuned (with all layers at 4-bits and a single layer at 2-bit) for the equivalent of one training epoch (1 epoch for ResNet-50 and ResNet-101 and 250 iterations for PSPNet). The average training set performance (accuracy and loss for Resnet and PSPNet models, respectively) over the training period is then used to estimate the accuracy gained, \(G_{l}\) if the given layer remained at 4-bit. The assumption made here, that total network accuracy is the sum of the layer-wise accuracies, is justified theoretically (Zhou et al., 2018), and empirically in the Appendix B. To the extent that this linearity holds, ALPS avoids the need to try all combinations of layer precision settings. These measures are provided to the optimizer, along with the desired computational cost target, or budget, in order to choose a list of precisions for the network layers which maximizes the overall task performance of the network while satisfying the budget constraint.

### Entropy Approximation Guided Layer selection for quantization (EAGL)

EAGL uses empirical entropy to find a solution for the problem formulation above. The key insight in the development of this metric is that the entropy of the empirical distribution (Eq. (1)) of parameters of a layer in a network represents a measure of the required complexity to achieve the desired performance. With this insight, we develop an entropy-based metric for quantifying the advantage of keeping a layer at a higher precision.

Consider a layer \(l\) in a network at precision \(b\). Post quantization, a parameter in \(l\) can have a value equal to one of \(2^{b}\) distinct values, i.e., the weight can occupy one of \(2^{b}\) bins. When parameters of a layer are spread out evenly in the available bins, and the distribution approaches a uniform distribution, its entropy is high. However, if many of the weights are concentrated in a small subset of bins, the entropy of the distribution of quantized weights is low. Intuitively, a layer with a lower entropy is a better candidate for further quantization as the representation of the transformation learned by the layer parameters can be compressed further more easily. This is illustrated in Fig. 2 using 3 convolutional layers from a trained 4-bit ResNet-101 network.

Let \(\hat{p}^{b}\) be the empirical distribution of the layer's quantized parameters at precision \(b\). EAGL is based on the hypothesis that \(H(\hat{p}^{b})\)2 provides a good measure which can serve as the accuracy gain metric in the mixed precision problem formulation described earlier. Intuitively, \(H(\hat{p}^{b})\) represents the number of bits needed to represent the parameters in that layer according to the empirical distribution, as opposed to the actual allocated bits \(b\). If \(H(\hat{p}^{b})\) is close to the allocated bit-width \(b\), the layer is a bad choice for further quantization; however, if \(H(\hat{p}^{b})\) is much lower than \(b\), the layer is a better candidate for further quantization to a lower precision. This metric can be calculated practically as follows.

Footnote 2: \(H\) is used everywhere to denote the discrete entropy function.

Let the vector \(\mathbf{w}\) represent \(n\) trained parameters for layer \(l\), and denote \(\mathbf{w}=(w_{1},w_{w},\dots,w_{n})\) where \(w_{i}\) are quantized with bit-width \(b\). If \(w_{i}\) takes values from a discrete set \(\mathcal{A}\), then \(|\mathcal{A}|\leq 2^{b}\). For \(c\in\mathcal{A}\), the empirical probability distribution is computed by

\[\hat{p}^{b}_{l}(c)=\frac{1}{n}\sum_{i=1}^{n}\mathbbm{1}_{c}(w_{i}), \tag{1}\]

where

\[\mathbbm{1}_{c}(w_{i}):=\begin{cases}1&\text{if }w_{i}=c\\ 0&\text{if otherwise.}\end{cases} \tag{2}\]

Using \(\hat{p}^{b}_{l}\) from equation 1, we compute its entropy as

\[H(\hat{p}^{b}_{l})=-\sum_{c}\hat{p}^{b}_{l}(c)\;\log\hat{p}^{b}_{l}(c). \tag{3}\]

The value of keeping layer \(l\) at higher precision \(b\) is quantified as the value of this entropy measure \(H(\hat{p}^{b}_{l})\)(Eq. (3)).

Algorithm 2 describes EAGL. EAGL requires only one pre-trained model - one where all layers are quantized at precision \(b\). The empirical distribution of parameters is estimated for each of the layers in the network, and then \(H(\hat{p}^{b}_{l})\) is calculated for each layer. These layer-wise measures are

Figure 2: **Histogram of normalized counts of quantized weights in each bin for 3 layers of a trained 4-bit ResNet-101 network.** EAGL predicts that layers with lower entropy are better candidates for further quantization. For example, between the three layers shown above, EAGL predicts that quantizing the first layer (entropy \(=1.3977\) bits) to 2 bits has lower impact on task accuracy than quantizing the third layer (entropy \(=3.7368\) bits).
provided to the optimizer in the context of our full evaluation framework. EAGL is (i) easy to approximate, and (ii) does not need access to the training dataset to compute, making it faster and more generally applicable to other problem domains than the other metrics compared against.

This work makes use of the Minimal Description Length (MDL) principle (Rissanen, 1978), and its application, in particular, to learning the required precision for representing network parameters (Wallace, 1990; Hinton & van Camp, 1993; Hochreiter & Schmidhuber, 1997; Ullrich et al., 2017; Blier & Ollivier, 2018; Wiedemann et al., 2019). One approach is related to attempts to build MDL principle into the optimization function (Wallace, 1990; Hinton & van Camp, 1993), and derive a learning rule to minimize the number of bits needed to represent the weights (also see (Wiedemann et al., 2019) and references therein). Here, on the other hand, a slightly different approach is taken, and simple insights from information theoretic principles are used to propose a measure that uses the entropy of a layer's learned parameters as the metric to choose between layers to achieve the most profitable trade-off between bits allocated to represent layer parameters and network performance.

A probability distribution on model parameters \(W\) is assumed, and \(\hat{p}_{l}^{b}\) is used as an estimate of the true marginal distribution. This estimation is valid when the layer parameters are assumed to be independent and identically distributed (iid). Even if the iid assumption is not satisfied, it may be assumed that the parameters in each of the layers are identically distributed and that they satisfy a form of the strong-mixing (Pollicott & Yuri, 1998) condition, or equivalently that the parameters that are sufficiently far apart are independent. This practical assumption allows the use of a single instance of parameters learned using SGD and binning to estimate the distribution of these parameters and allows the efficient computation of the entropy terms. It is emphasized that no change to the loss function was made to minimize the measured entropy and that SGD with regularization is relied upon entirely to trade-off between entropy of the empirical distribution and network performance for a given precision.

### Implementation details

A layer's precision is used to determine the representation of its input activations and weights, which thus must match for a given layer. If an activation tensor provides input to multiple layers, all such layers must therefore have the same weight precision as well. Such layers are considered linked, and are treated as a single layer with a computational cost and accuracy gain measure that is the summation of its member values. All other data is represented with, and operations occur at, full precision. A unit of Bit Multiply-Accumulate operations (BMACs) is used for computational cost calculation, which is defined as BMAC = \(b\)*MAC, where \(b\) is the precision of the layer (weights and activations) in bits. Two tasks are evaluated here, image classification (Deng et al., 2009) using ResNet-50 and ResNet-101 (He et al., 2016), and semantic segmentation (Cordts et al., 2016) using PSPNet (Zhao et al., 2017). For both tasks, instead of choosing a fixed budget apriori, a sweep of computational budgets is used as evaluation points. This ensures that the evaluation framework is fair, as some techniques might demonstrate stronger performance for tighter budget constraints while others might be better for less constrained budgets; and hence evaluating only on a single budget can potentially unfairly benefit one technique over another. Eight equally spaced computational budgets are used for ResNet networks and 4 equally spaced computational budgets for PSPNet between the computational cost of a 4-bit and a 2-bit network.

The first and last layers are fixed at 8-bit, following a common practice for quantized networks (Zhou et al., 2016; Zhu et al., 2016), and intermediate layers with less than 128 input features are fixed at 4-bit. Layers with fixed precision do not count towards the computational budget. All models are fine-tuned with weights and activations quantized using LSQ (Esser et al., 2020). The ResNet networks are adapted from the PyTorch Model Zoo, and trained for image classification (Deng et al., 2009) with weight decay of 1e-4, initial learning rate of 0.01 with cosine learning rate decay (Loshchilov & Hutter, 2016), batch size 128, and using knowledge distillation (Hinton et al., 2015). \(160\times 160\) images are used to speed up training, while \(224\times 224\) images are used for testing. The initial quantization step-size for all layers being reduced from 4- to 2-bit is set to \(4s\), where \(s\) is the learned step-size loaded from the 4-bit checkpoint. Models at 4-bit, used as the initial checkpoint to fine-tune mixed precision models, are trained by fine-tuning the full precision checkpoint for 90 epochs. Each mixed precision method was then fine-tuned from the 4-bit trained model for 90 epochs for comparison. PSPNet is adapted from the PyTorch Segmentation Toolbox (Huang et al., 2018) and trained on semantic segmentation (Cordts et al., 2016) with an learning rate of 0.015, weight decay of 5e-5, and batch size 4. Models at 2-bit and 4-bit are trained by fine-tuning a full precision checkpoint for 80,000 and 10,000 iterations respectively, and mixed precision methods are fine-tuned from the 4-bit trained model for 40,000 iterations.
## 4 Results

The performance of ALPS and EAGL is compared against leading mixed precision techniques from the literature in Table 1 using the most commonly used benchmark, ResNet-50 for image classification (Deng et al., 2009). Both techniques have the highest accuracy (no accuracy drop) for the most efficient networks (lowest BOPs). As seen in the table, different techniques are quantized starting from different full precision accuracy checkpoints, trained using different training recipes, and report on different metrics. To mitigate these issues, HAWQ-v3, a leading mixed precision layer selection technique, was re-implemented and results are presented below from apples-to-apples comparison with HAWQ-v3. Other techniques could not be re-implemented due to the significant effort required to reproduce results without released code. Results are presented using three different networks on two large scale vision datasets below across many computational budgets.

### Evaluation using ResNet for image classification

The performance of ALPS and EAGL for layer selection is evaluated on image classification (Deng et al., 2009) using ResNet-50 and ResNet-101, using the methodology described in Section 3.1. The methods are compared with a re-implementation of HAWQ-v3 (Yao et al., 2021)3 and three baselines - a uniform cost baseline, where every layer is given the same value for staying at 4 bits, a method which ranks layers from first to last, and drops the precision of the first \(n\) layers greedily until the resulting network just meets the budget, and a third method which ranks layers from last to first. Instead of choosing a fixed budget apriori, eight different target computational budgets are sampled for evaluation, roughly at 95%, 90%, 85%, 80%, 75%, 70%, 65% and 60% of the computational cost of a 4-bit network. For each budget, networks created using each technique compared are fine-tuned for 90 epochs using 5 seeds.

Footnote 3: See Appendix D for details of the re-implementation of HAWQ-v3 for these experiments.

Results are shown in Fig. 3. ALPS and EAGL outperform the baselines and HAWQ-v3 at all budgets along the entire throughput-accuracy frontier for ResNet-50, demonstrating very strong performance (\(p=0.0079\) for all compute budgets except for EAGL at 60% where \(p=0.0238\), Wilcoxon rank-sum, \(N=5\)). For ResNet-101, EAGL matches or outperforms HAWQ-v3 at all 8 budgets. ALPS does so at 7 out of 8 budgets. ALPS provides solutions significantly better than HAWQ-v3 and EAGL for 4 of the budgets under consideration. For ResNet-50, layer-wise precision selection choices between the five techniques at the 70% budget are compared in the Appendix E. Even with a throughput budget significantly lower than a 4-bit network, with over half of the computation happening at 2-bit instead of 4-bit, EAGL and ALPS match full precision accuracy. For ALPS, one epoch of finetuning for each layer was chosen to minimize computational cost; rerunning the ResNet-50 experiment with two epochs per layer did not improve the result.

The actual computational cost to get the layer-wise estimates for ALPS, EAGL and HAWQ-v3 are listed in Table 2. EAGL's superior performance given the orders of magnitude saving in terms of computational cost makes it the most suitable candidate for layer selection in practice for getting a fast solution across a range of architectures and budgets. Python code used for computation of the EAGL metric is provided in the Appendix F.

### Evaluation using PSPNet on semantic segmentation

Next, ALPS and EAGL are evaluated on the semantic segmentation task (Cordts et al., 2016) using PSPNet. The methods are compared against HAWQ-v3 and a first to last baseline. Four different target throughput budgets are evaluated, roughly at 95%, 85%, 75%, and 65% of the computational cost of a 4-bit PSPNet network. For each budget, each technique is fine-tuned for 40,000 iterations using 5 different seeds. Results are shown in Fig. 4.

The task performance, i.e., mean Intersection over Union (IoU) for networks produced by ALPS and EAGL is not significantly different from HAWQ-v3 (\(p>0.1\) for all compute budgets, Wilcoxon rank-sum test, \(N=5\)), while all methods outperformed the first-to-last baseline (\(p=0.0079\) for all budgets, Wilcoxon rank-sum, \(N=5\)). The computational cost to get the layer-wise estimates for ALPS, EAGL and HAWQ-v3 are listed in Table 2. EAGL and ALPS are able to find good solutions to the mixed precision layer selection problem for this network, while being orders of magnitude faster to compute and easier to implement in practice. The results provide further evidence that both methods are scalable, task agnostic approaches that generalize to other networks and datasets as well.

\begin{table}
\begin{tabular}{l c c} \hline \hline Method & ResNet-50 & PSPNet \\ \hline
**EAGL(ours)** & **3.15 CPU seconds** & **<1 CPU minute** \\ \hline ALPS(Ours) & 166 GPU hours & 67 GPU hours \\ \hline HAWQ-v3 & 2 GPU hours & 1032 GPU hours \\ \hline \hline \end{tabular}
\end{table}
Table 2: **Metric computational cost comparison for ResNet-50 and PSPNet.** The cost compared here includes only the computational resources required for the layer-wise metric estimation for each method, and not the cost for fine-tuning the resulting mixed precision network, nor the cost for training the initial 4-bit checkpoint, as these are the same for all techniques.
## 5 Conclusion and Discussion

Two new metrics for optimizing precision settings in a mixed precision network are introduced. Both techniques outperform leading techniques from the literature while requiring fewer computational resources to calculate and generalize across different tasks and architectures. Further evidence for the quality of these proposed metrics is provided in the Appendix C.

The ALPS metric introduced here is simple to understand intuitively, easy to implement and yet outperforms leading techniques from literature that ignore fine-tuning and use complex proxies which don't scale well in practice for the actual task at hand, both in terms of task performance of the proposed solution and time needed to compute the metric.

The entropy-based EAGL metric is elegant, and applicable even when only the model parameters are known but the training data is inaccessible, making it task agnostic and generalizable to other domains like unsupervised learning. Since it only requires access to a trained checkpoint, EAGL is remarkably fast, and is shown to select more accurate networks than several leading state-of-the-art methods on several datasets and networks in a fair comparison.

Given its strong performance, extremely low computational cost and ease of implementation, EAGL provides effective, practically usable solutions for network compression and faster inference. This makes EAGL the first choice to get good solutions extremely fast. The performance can sometimes be further improved by using ALPS. Although this paper considers only networks consisting of 2- and 4-bit layers, both methods can be used with more than two precision choices by changing the optimizer (e.g. (Chen et al., 2021)). Given the strong results across the entire throughput-accuracy frontier, as the number of hardware platforms supporting mixed precision networks grows, these simple, efficient, and effective approaches should be of great utility to practitioners, directly allowing the deployment of lower power, higher throughput solutions each time they are used.

Figure 4: **ALPS and EAGL meet or exceed the mean IoU of leading techniques on PSPNet across computational budgets.** Mean +/- standard deviation across 5 seeds at each budget.

Figure 3: **ALPS and EAGL perform better than leading mixed precision techniques using ResNet-50 for all computational budgets and meet or exceed the accuracy of ResNet-101 for 7 out of 8 computational budgets on (Deng et al., 2009).** Mean +/- standard deviation across 5 seeds for each technique at each budget. A network with all configurable layers at 4 bits has a computational budget of 100% and a network with all configurable layers at 2 bits has a computational budget of 50% in this plot. The first and last layer are 8-bit and intermediate layers with less than 128 input features are fixed at 4-bit.
## Acknowledgement

This material is based upon work supported by the United States Air Force under Contract No. FA8750-19-C-1518.
