On-Chip Hardware-Aware Quantization for Mixed Precision Neural Networks
Low-bit quantization emerges as one of the most promising compression
approaches for deploying deep neural networks on edge devices. Mixed-precision
quantization leverages a mixture of bit-widths to unleash the accuracy and
efficiency potential of quantized models. However, existing mixed-precision
quantization methods rely on simulations in high-performance devices to achieve
accuracy and efficiency trade-offs in immense search spaces. This leads to a
non-negligible gap between the estimated efficiency metrics and the actual
hardware that makes quantized models far away from the optimal accuracy and
efficiency, and also causes the quantization process to rely on additional
high-performance devices. In this paper, we propose an On-Chip Hardware-Aware
Quantization (OHQ) framework, performing hardware-aware mixed-precision
quantization on deployed edge devices to achieve accurate and efficient
computing. Specifically, for efficiency metrics, we built an On-Chip
Quantization Aware pipeline, which allows the quantization process to perceive
the actual hardware efficiency of the quantization operator and avoid
optimization errors caused by inaccurate simulation. For accuracy metrics, we
propose Mask-Guided Quantization Estimation technology to effectively estimate
the accuracy impact of operators in the on-chip scenario, getting rid of the
dependence of the quantization process on high computing power. By synthesizing
insights from quantized models and hardware through linear optimization, we can
obtain optimized bit-width configurations to achieve outstanding performance on
accuracy and efficiency. We evaluate inference accuracy and acceleration with
quantization for various architectures and compression ratios on hardware. OHQ
achieves 70% and 73% accuracy for ResNet-18 and MobileNetV3, respectively, and
can reduce latency by 15~30% compared to INT8 on real deployment.
Wei Huang, Haotong Qin, Yangdong Liu, Jingzhuo Liang, Yulun Zhang, Ying Li, Xianglong Liu
# OHQ: On-chip Hardware-aware Quantization

###### Abstract

Quantization emerges as one of the most promising approaches for deploying advanced deep models on resource-constrained hardware. Mixed-precision quantization leverages multiple bit-width architectures to un-leash the accuracy and efficiency potential of quantized models. However, existing mixed-precision quantization suffers exhaustive search space that causes immense computational overhead. The quantization process thus relies on separate high-performance devices rather than locally, which also leads to a significant gap between the considered hardware metrics and the real deployment. In this paper, we propose an **On-chip Hardware-aware Quantization** (OHQ) framework that performs hardware-aware mixed-precision quantization without accessing online devices. First, we construct the _On-chip Quantization Awareness_ (OQA) pipeline, enabling perceive the actual efficiency metrics of the quantization operator on the hardware. Second, we propose _Mask-guided Quantization Estimation_ (MQE) technique to efficiently estimate the accuracy metrics of operators under the constraints of on-chip-level computing power. By synthesizing network and hardware insights through linear programming, we obtain optimized bit-width configurations. Notably, the quantization process occurs on-chip entirely without any additional computing devices and data access. We demonstrate accelerated inference after quantization for various architectures and compression ratios, achieving 70% and 73% accuracy for ResNet-18 and MobileNetV3, respectively. OHQ improves latency by 15\(\sim\)30% compared to INT8 on deployment.

1Beihang University, 2ETH Zurich, 3The University of Hong Kong

aaron.weihuang@gmail.com, qinhaotong@buaa.edu.cn, earsaxcs@buaa.edu.cn

liangjz@buaa.edu.cn, yifuding@buaa.edu.cn, liying@buaa.edu.cn, xlliu@buaa.edu.cn

## 1 Introduction

Recently, the deep neural network (DNN) has achieved significant development and shown its great potential in various fields, such as computer vision, and natural language processing. The advancement of DNN can be mainly attributed to the rapid expansion of parameters and the increasing depth of the model. Although over-parameterization has enhanced the performance of DNNs, it has simultaneously posed a serious challenge when deploying real-time inference in resource-constrained edge devices. Edge chips impose stricter constraints on memory footprint, energy consumption, and latency[18]. An intuitive solution to mitigate this issue is quantizing the full-precision 32-bit weights and activations of DNNs to low-bit integers or binaries [12, 13, 14, 15]. The basic quantization involves uniformly compressing all layers of a DNN to a consistent bit-width [16, 17], which overlooks the computational redundancies inherent in different operator layers within the network and subsequently neglects the varying impact on overall performance. To address this limitation, it becomes imperative to adopt variable bit-widths for distinct layers, a concept known as mixed-precision quantization [13, 14, 15]. The mixed-precision quantization considers the operators with different bit widths in search space optimization, which significantly pushes the accuracy and efficiency limits of quantized NNs. However, it also causes significant computation overhead in the quantization process. In a \(L\)-layers DNN, the \(S\) number of bit-width candidates results in a computa

Figure 1: Off-chip _vs._ on-chip quantization. The left shows the traditional off-chip quantization framework involving quantization analysis and deployment steps. The right part is our OHQ framework which is fully integrated on-chip.
tion with O(\(S^{L}\)) complexity. This exemplifies an exponential increase in computational demand for mixed-precision strategies, especially pronounced in deeper architectures.

These facts introduce strong motivation to construct an on-chip mixed-precision quantization framework that harmoniously integrates the quantization algorithm with real hardware. The challenges in constructing this quantization framework stem from two main aspects: (1) Hardware awareness: Since the need to consider the accuracy and speed of quantized models based on their deployed hardware, this framework should on-chip perceive various metrics of operators on real hardware, including but not limited to latency and memory usage. This framework should achieve hardware awareness for the deployed devices. (2) Lightweight process: Given the necessity to thoroughly consider efficiency-oriented hardware metrics, the expanded search space requires an efficient yet resource-lightweight algorithm for on-chip quantization processes. This ensures that the quantization process does not rely on external computational and data resources. Despite the presence of certain hardware awareness and mixed-precision quantization techniques that emerged earlier [4, 13, 14, 15, 16], they still exhibit some significant issues that cannot be overlooked. These issues include the computationally expensive nature of the process and reliance on inaccurately estimated hardware metrics, among others. These factors hinder their progression toward becoming the ideal on-chip quantization methods.

In this work, we propose an On-chip Hardware-aware Quantization (OHQ) framework (see Fig. 1) to overcome the above-mentioned issues. The proposed OHA mainly relies on two novel techniques: the _On-chip Quantization Awareness_ (OQA) pipeline enables perceiving the actual efficiency metrics of the quantization operator on the hardware, which uses synthetic data as input to obtain the latency, memory usage, and power metrics on-chip. Second, we propose _Mask-guided Quantization Estimation_ (MQE) technique to efficiently estimate the accuracy metrics of operators under the constraints of on-chip-level computing power, and then we can search for optimized bit-width configurations simplified as linear programming. Our comprehensive experiments show that OHQ outperforms existing mixed-precision quantization methods in accuracy and efficiency by a substantial margin. We also demonstrate the effectiveness of our OHA on various networks, such as ResNet-18/50 and MobileNetV2/V3, highlighting its versatility across architectures. We summarize our contributions as follows:

* We propose a mixed-precision quantization framework OHQ, which is a totally on-chip design from quantization computation to deployment. By selecting Field Programmable Gate Arrays (FPGAs) as co-processors to aid the on-chip Central Processing Units (CPUs) in quantization computing, we effectively eliminate the necessity for supplementary computing devices.
* A hardware-aware method OQA that operates at the Intellectual Property (IP) core granularity is first proposed. This approach utilizes the chip's clock cycle consumption per computational layer as a metric for hardware, while simultaneously considering the constraints imposed by the available computational power.
* We develop an enhanced and statistically robust sensitivity metric MQE for performing small-batch distilled data inference on edge devices, which support data-free sensitivity feedback and quantization.
* We additionally furnish comprehensive empirical findings for ResNet18/50 [11], MobileNetV2/V3 [12]. These findings delineate the state-of-the-art quantization outcomes achievable in real-world edge scenarios.

## Related Work

### Mixed-Precision Quantization

Quantization maps parameters \(\mathbf{x}\) to fixed-point ranges using the following equation:

\[\mathrm{quantize}(\mathbf{x})=\mathrm{round}\left(\mathbf{x}/S\right)-\mathbf{z}, \tag{1}\]

where \(\mathrm{quantize}\) denotes quantization function, \(S\) and \(\mathbf{z}\) denote the scaling factor and zero-point, respectively. Existing quantization approaches can be divided into quantization-aware training (QAT) and post-training quantization (PTQ). QAT quantizes the network throughout the training process with the original dataset, resulting in accurate quantization while markedly computation overhead [13, 14, 15]. PTQ operates as an offline algorithm, it relies on a few real or synthetic samples to calibrate the quantization functions of weights and activations, thus just utilizing much less computation in the quantization process compared to QAT [13, 14, 15].

For utilizing the accuracy and efficiency potential of the quantized model, mixed-precision quantization emerges as a promising way, which enables accuracy-sensitive layers to retain high precision (i.e., more bit-widths) while others maintain low precision [4, 13, 14, 15, 16]. Nevertheless, a major challenge associated with this strategy lies in identifying the optimized mixed-precision configuration since the search space exhibits an exponential relationship with the number of layers, particularly in resource-limited scenarios. Thus various mixed-precision quantization methods are proposed to improve this. Dong et al. use a second-order Hessian matrix as an accuracy sensitivity metric for each layer [4] and also apply the mean of the eigenvalues of the Hessian to mitigate computational overhead [4]. [15] and [14] propose data-free methods to get rid of the reliance on the data resource on edge. However, it is hard for existing methods to achieve on-chip mixed-precision quantization accurately and efficiently.

### Hardware-Aware Quantization

The accuracy sensitivity of quantized layers can frequently be described as the influence of quantization on the model accuracy. However, the computational expenditure of network operators on physical hardware should also serve as
a constraint for bit width configuration, incorporating aspects such as latency and energy consumption. The heavily-consuming layers should be quantized to low precision, while the light-consuming ones should be retained to high precision. Wang _et al_. sensed the bit-width rationing under different hardware architectures through a reinforcement learning model, which is a black-box strategy model based on hardware perception [22]. However, this exploration approach is computationally complex and difficult to realize offline quantization. Yao _et al_. sensed the computation time of each layer as a bit-width allocation constraint by deploying the quantized network to Graphics Processing Unit (GPU) [23]. But the roughly obtained computation time is affected by software, operating system (OS), and network transmission, and cannot reflect the on-chip computational efficiency accurately.

We select FPGA to implement and evaluate our framework since it serves as a flexible and reliable hardware platform for DNN deployment[15, 24] and validating Application-Specific Integrated Circuit (ASIC) designs through accurate flow verification methodologies[16, 17, 18, 19]. We proposed a fine-grained hardware awareness by clock cycles and energy, which can achieve a more accurate representation of hardware computation consumption at the IP core level. This methodology enables a closer examination of the chip's underlying layer, thereby facilitating a realistic construction of the computational connection between the DNNs and the chips.

## Methodology

In this section, we present our On-Chip Hardware-Aware Quantization (OHQ) framework (see Fig. 2), including the On-Chip Quantization Awareness (OQA) pipeline and Mask-Guided Quantization Estimation (MQE) technique.

### On-Chip Quantization Awareness

#### Generated Synthetic Data

The statistics of the BN layer in the full-precision model, namely the mean standard deviation, correspond to the real dataset in the training process. Consequently, the majority of data-free quantization schemes[16, 1, 17, 18, 19] employ BN statistical losses to capitalize on the information present in the BN layer. The subsequent optimization objectives facilitate the congruence of synthetic data distribution \(x^{d}\) with the BN statistics:

\[\min_{x^{d}}\mathcal{L}_{\text{distill}}=\sum_{i=1}^{L}\parallel\tilde{u}_{i}^ {d}-u_{i}\parallel_{2}^{2}+\parallel\tilde{\sigma}_{i}^{d}-\sigma_{i} \parallel_{2}^{2}, \tag{2}\]

where \(u_{i}\) and \(\sigma_{i}\) are the mean and standard deviation parameters of the pre-trained model's BN layer. \(\tilde{u}_{i}^{d}\) and \(\tilde{\sigma}_{i}^{d}\) represent the mean and standard, respectively, of the feature matrix generated by \(x^{d}\) at the \(i\)-th of the BN layer. Eq. 2 is to minimize the statistical loss \(\mathcal{L}_{distill}\) in each BN layer of the original model, and ultimately to generate synthetic data that matches the input data distribution. Our distilled data is generated via the embedded CPU on the edge platform and subsequently inputted directly into the FPGA, which ascertained the following hardware awareness and MQE technique. This approach aims to bolster the operational efficiency of the proposed framework.

#### Hardware Awareness

While previous study[23] has recommended the use of cloud GPU running time as a hardware-aware constraint, this coarse-grained metric is influenced by the OS[16, 17], and application layers latency and does not provide a comprehensive reflection of the DNN's performance on the physical chip. In order to comprehend the finer-grained hardware constraints, we observed the IP core-level operations on FPGAs and we name it as OQA.

To facilitate NN on FPGAs, we quantize and compile the model following its acquisition, subsequently converting the layers into corresponding arithmetic module implementations, while providing a central controller for regulation. In this process, we employ the following techniques:

1. Img2Col: Converts layer data from discontinuous storage to continuous storage, streamlining transmission and computation.
2. Multiplication and addition tree: Utilizes parallel stacking and cascading multiplication and addition mechanisms to enable simultaneous pointwise multiplication computation across vast amounts of data.
3. Sub-matrix slicing transmission and computation: Divides a large matrix into multiple tiles for transmission to the chip, performing computation and splicing on these tiles to alleviate FPGA resource pressure.

Figure 2: The overview of OHQ framework. This proposed OHQ obtains chip-level sensing parameters and layer-wise differences through a physical deployment (OQA and MQE are respectively described in detail in Fig. 3 and Fig. 4).
4. BRAM large bandwidth fill: The primary storage mechanism employed in on-chip Block Random Access Memory (BRAM), which is divided into rows to facilitate the reading and writing of large bit-width data simultaneously, ensuring computation and access remain logically and physically coherent.

The above methods enable efficient parallel computation of networks, and a reasonable quantitative deployment scheme is formulated with full consideration of the resource and power constraints of hardware.

Our proposed clock-aware approach, predicated on the interaction between the IP core and the BRAM, is illustrated in Fig. 3. During runtime, the IP core on the Programmable Logic (PL) side--which is responsible for four steps including computation of weight and feature map, data transfer, data write-back, and data post-process--automatically collects the number of running clock cycles and stores them in the target BRAM. Then, the FPGA-compatible transmission relays the collected clock cycles to the Processing System (PS) side. This can be denoted as \([c_{1},\ldots,c_{i},\ldots,c_{L}]\), \(c_{i}\) represents the total number of clock cycles in the \(i\)-th layer. Concurrently, the power consumption of the IP core during computation is diminished as \([e_{1},\ldots,e_{i},\ldots,e_{L}]\).

Note that OQA is the first pipeline to propose fine-grained sensing at the chip IP core level. By obtaining the clock information of four calculation steps from the PL respectively, we obtain the exact part of the operation of each layer on-chip.

### Mask-Guided Quantization Estimation

Layerwise Sensitivity on HardwareDNN models comprise \(L\) layers of computational units, which can be represented as \([U_{1},U_{2},...,U_{L}]\), wherein \(U_{i}\) denotes the \(i\)-th layer computational unit. The learnable parameters (e.g., weights) are denoted as \([\theta_{1},\theta_{2},...,\theta_{L}]\), where \(\theta\in\mathcal{R}^{n}\) is float32 type data in full-precision models. Layer-wise sensitivity typically corresponds to the impact of distinct layers within a network on the output results. As previously discussed, sensitivity measurement using the Hessian matrix can be computationally intensive. An alternative approach involves quantizing \(U_{i}\) to 4/8 bits while maintaining full precision for the remaining layers:

\[\theta_{i}^{q}=\mathrm{quantize}(\theta_{i}), \tag{3}\]

\[\begin{cases}\mathcal{M}=F(U_{1}(\theta_{1});\ldots;U_{i}(\theta_{i});\ldots;U _{L}(\theta_{L})),\\ \mathcal{M}_{i}^{q}=F(U_{1}(\theta_{1});\ldots;U_{i}^{q}(\theta_{i}^{q});\ldots ;U_{L}(\theta_{L})),\end{cases} \tag{4}\]

where \(\theta_{i}^{q}\) is the quantized parameters in \(i\)-th layer and the bit width, \(q\) is the selected bit-widths, \(F\) denotes the general function of NN model, \(\mathcal{M}\) represents the full-precision model, \(\mathcal{M}_{i}^{q}\) denotes the quantized model with quantized layer \(U_{i}^{q}\). Then the performance difference is calculated by the following equation:

\[\omega_{i}=\frac{1}{N}\sum_{j=1}^{N}f(\mathcal{M}(x_{j}),\mathcal{M}_{i}^{q}( x_{j})), \tag{5}\]

where \(\omega_{i}\) indicated the sensitivity value of the \(i\)-th layer, \(N\) is the batch-size of distilled data used for inference, \(f(\cdot,\cdot)\) denotes the sensitivity calculation function that compares the output of \(\mathcal{M}\) and \(\mathcal{M}_{i}^{q}\), and \(x\) is the input data.

Nonetheless, this method necessitates the quantization \(\mathbf{L}\) times, resulting in considerable computational overhead for so many quantizations processed. Consequently, this sensitivity calculation presents challenges for on-chip deployment. Therefore, we introduce a masked-guided technique (MQE) displayed in Fig. 4, which is more efficient and suitable for on-chip implementation:

\[\tilde{\theta}_{i}^{q}=g(\alpha,\theta_{i}^{q}), \tag{6}\]

\[\begin{cases}\mathcal{M}^{q}=\ F(U_{1}^{q}(\theta_{1}^{q});\ldots;U_{i}^{q}( \theta_{i}^{q});\ldots;U_{L}^{q}(\theta_{L}^{q})),\\ \tilde{\mathcal{M}}_{i}^{q}=\ F(U_{1}^{q}(\theta_{1}^{q});\ldots;\tilde{U}_{i}^{ q}(\tilde{\theta}_{i}^{q});\ldots;U_{L}^{q}(\theta_{L}^{q})),\end{cases} \tag{7}\]

where \(g(\cdot,\cdot)\) denotes the masking operator, \(\alpha\) is the mask ratio, and \(\tilde{\theta}_{i}^{q}\) is the mask result of \(\theta_{i}^{q}\). To facilitate the on-chip test of FPGA, the parameters must first undergo integer

Figure 4: Illustration of MQE for ResNet18. Specifically, we feed synthesized data into on-chip models. The figure shows the model with the \(5\)-th layer specifically masked out.

Figure 3: The workflow of OQA. (Top) The PL part samples time, power, and other information of four main steps for awareness while computing, which use BRAM to optimize matrix multiplication and data transfer. (Bottom) The PS part controls the whole situation, including accessing data, organizing the network, and instructing IP cores.
bit-wise quantization (in this case, \(q=8\)). Subsequently, the parameters of the target layer are randomly masked by setting them to 0, and the \(\alpha\) is selected as \(0.5\) to map the loss of information loss from \(8\) bits to \(4\) bits. In MQE, only **1** time quantization of the DNN is necessary. Despite the requirement for \(L\) masking operations, their computational consumption is considerably less compared to that of quantization computation. Then, we use the KL divergence:

\[\begin{split} D_{kl}(p\parallel q)&=-\sum_{x}p(x) \log q(x)-\sum_{x}-p(x)\log p(x),\\ &=-\sum_{x}p(x)\log\frac{p(x)}{q(x)},\end{split} \tag{8}\]

where \(p(x)\) and \(q(x)\) denote two probability distributions of the original model and masked model. KL divergence can determine the disparity in output distribution between the original and masked models, thereby effectively measuring the information entropy between the matrices. Then, we update Eq. 5 to get our sensitivity:

\[\omega_{i}=\frac{1}{N}\sum_{j=1}^{N}D_{kl}(\mathcal{M}^{q},\tilde{\mathcal{M} }_{i}^{q}), \tag{9}\]

Our proposed MQE is obtained by inference on edge devices, which better reflects the layer-wise perception in NN in real hardware operation scenarios than simulation experiments on servers and GPUs. Consequently, the masking operations are negligible and can be swiftly executed by the CPU embedded within the edge device.

Mixed-Precision QuantizationWe reveal that the aforementioned correlation is not entirely linear and exhibits fluctuations as displayed in Fig. 6. This observation underscores the capability of our proposed OHQ framework in discerning the genuine performance of both the network and hardware, thereby highlighting the significance of on-chip hardware awareness for quantization. Inspired by this, we would like to keep the layers with high sensitivity at a higher precision, while compressing the layers with high clock and energy consumption to a lower precision. We, therefore, design a quantitative constraint function by combine OQA and MQE:

\[1=\beta+\gamma, \tag{10}\]

\[\Omega_{i}=\beta\hat{\omega}_{i}-\frac{\gamma}{2}(\hat{c}_{i}+\hat{e}_{i}), \tag{11}\]

where \(\beta,\gamma\) are two hyper-parameters used to control the proportion of the sensitivity and hardware resources, in order to fairly consider this hardware awareness, we set \(\beta\) and \(\gamma\) to 0.5 in the subsequent experiments. The hyper-parameters can be manually modified to satisfy the user's personalized needs for the precision and hardware compression rate. \(\hat{\omega}_{i}\), \(\hat{c}_{i}\) and \(\hat{e}_{i}\) are scaled values from \(\omega_{i}\), \(c_{i}\) and \(e_{i}\), which ensures that the different initial awareness value is in same the range \([0,1]\) and \(\Omega_{i}\) is the optimal factor of the \(i\)-th layer. Ultimately, we maximize the sum of \(\Omega_{i}\) in the network through an integer linear programming (ILP) model:

\[\begin{split}\text{Objective}:&\max_{\{b_{i}\}_{i= 1}^{L}}\sum_{i=1}^{L}(b_{i}\Omega_{i}),\\ \text{Constraint}:&\sum_{i=1}^{L}M_{i}^{b_{i}}\leq \text{Model Size Limit},\end{split} \tag{12}\]

\(M_{i}^{b_{i}}\) denotes the parameters size of the \(i\)-th layer under \(b_{i}\) bit-width quantization. And the size of the target compressed model can be set by the user. Since the selectable bit widths are 4 and 8, the target size should be between the uniform 4 bits model and the uniform 8 bits model. It is worth noting that compared to the RL method proposed by Wang et al.[20] and the Hessian matrix computation proposed by Dong et al.[19] the ILP model only takes about 1 second to obtain the optimal bit-width configuration result under the hardware-aware result input condition. This computation is very efficient and can be done on the embedded CPU of the edge platform.

## Experiment

In this subsection, a comprehensive array of experiments is undertaken to empirically validate the performance of the OHQ. The preliminary step encompasses the delineation of the datasets employed and the specific models selected for the experimental evaluations. The ablation experiments compare the decomposition of OQA and MQE. Conclusively, an intricate comparison is executed, juxtaposing the performance metrics of disparate models as facilitated by the OHQ framework. The results of this comparative assessment conspicuously highlight the discernible merits inher

Figure 5: Comparison of on-chip perceptual parameters of MobileNetV3. From left to right, the sensitivity-accuracy, clock-parameter counts, and power consumption-parameter comparison curves at different layers are shown.
ent in our proposed approach, manifesting in superior compression rates, heightened operational performance, and enhanced quantization efficiency.

### Implementation Details

We validate the experiments on ImageNet[4] dataset. The training data (1.5M) encompassing Imagenet was deliberately left unutilized in our study. Instead, only 32 distilled images are generated, specifically dedicated to the computational assessment of perceptual outcomes pertaining to OQA and MQE during the forward inference. Owing to the wide use of residual structures in DNNS, we chose the ResNet18 and ResNet50 networks. In addition, we also conduct experiments on MobileNetV2 and MobileNetV3.

Our experimentation and assessments are exclusively executed on an ECE-EMBD development board, housing components from the ZYNQ chip series. This development board amalgamates a dual-core ARM Cortex-A9 processor endowed with 512MB of DDR3 memory on the PS side. The memory subsystem is defined by the MT41K256M16TW model, boasting a 16-bit bit width. The PL domain encompasses the XC7Z020-CLG400-1 chip, characterized by an assembly of 85K logic resources, and 140 BRAMs with a cumulative capacity of 4.9 Mb. Notably, the on-chip hardware sensing leverages a quantization strategy spanning from 4 bits to 8 bits, judiciously bounding matrix parallel operations to a maximum size of 128 data elements on a single facet. This prudent limitation is designed to preempt an over-allocation of BRAM resources and thus preserve adherence to prescribed quotas. We also validated the performance of the OHQ method for the QAT, where we used a GTX 1080 Ti to perform fine-tuning computations on selected models and present the results in Table 3.

### Ablation Results

#### Hardware-aware Parameters

We present an analytical exposition of statistical insights in Fig. 6 for MobileNetV3. We unveil the sensitivities exhibited across layers(left). Notably, higher sensitivities exert a profound influence on accuracy, underscoring their pivotal role in shaping model performance. We found that the first and the last layers give higher sensitivities in each network, which is due to the fact that the first layer directly deals with original images or feature maps with larger aspects, while the last layer needs to perform the hidden layer classification computation that determines the output data, and is prone to accumulating errors and is sensitive to the weight changes in on-chip low-precision quantitative inference scenarios. The middle demonstrates the change of clocks, and it can be seen that the latency on-chip is not strongly correlated with the parameter number. It is more strongly correlated with the size of the input feature map of each layer, while depthwise convolution has a greater correlation between the computation time and the number of channels compared to ordinary convolution. Power consumption is predominantly shaped by parallel computation scales, indicating the reduced rate of consumption increase with large size considering power and computation efficiency.

#### Optimization Factor Deconstruction

The present investigation undertakes an in-depth analysis of compression and performance dynamics within the framework of diverse model volume constraints. HAWQ-v3 is selected as the vanilla method, which also conducts 4/8 bits mixed-precision quantization. The findings, showcased in Table 4, distinctly highlight the absolute performance advantage achieved through MQE-based sensitivity. While OQA-guided hardware-aware constraints exhibit an appreciable augmentation in compression rates, their linkage primarily centers on the dimensions and channels of the feature map. However, it's noteworthy that the adoption of OQA as a sole optimization factor yields elevated accuracy loss in bit-width configurations. As elucidated by Eq. 11, the amalgamation of MQE and OQA constraints engenders a synergistic effect, yielding an elevated compression rate while ensuring accuracy, in stark comparison to the outcomes derived from individual parameters. This multifaceted analysis underscores the nuanced interplay between compression, performance, and model constraints, facilitating informed decisions regarding optimization strategies and trade-offs in resource-constrained scenarios.

### Comparison Results

The important feature of the OHQ proposed in this paper is the on-chip, and in order to efficiently implement the model, we performed PTQ on ResNet18/50, MobileNetV2, and MobileNetV3. and compared it with previous methods ([11, 12, 13, 14, 15, 16, 17, 18, 19]).

\begin{table}
\begin{tabular}{l c c c c c c c c c c} \multirow{2}{*}{Arch} & \multirow{2}{*}{Ratio} & \multirow{2}{*}{W/A} & \multicolumn{3}{c}{Vanilla} & \multicolumn{3}{c}{MOE} & \multicolumn{3}{c}{OQA} & \multicolumn{3}{c}{MOE + OQA} \\ \cline{4-11}  & & & Size (Mb) & Top-1 (\%) & Size (Mb) & Top-1 (\%) & Size (Mb) & Top-1 (\%) & Size (Mb) & Top-1 (\%) \\ \hline \hline ResNet18 & 0.25 & */* & 7.3 & 70.01 & 6.90 & 70.23 & 6.20 & 69.97 & 6.90 & 70.23 \\  & 0.50 & */* & 7.9 & 70.50 & 8.30 & 71.31 & 7.30 & 70.05 & 7.70 & 71.25 \\  & 0.75 & */* & 9.9 & 71.20 & 9.70 & 71.79 & 8.20 & 70.13 & 8.90 & 71.56 \\  & 0.80 & */* & 10.7 & 71.91 & 10.00 & 72.53 & 10.00 & 71.12 & 10.00 & 72.53 \\ \hline MobileNetV2 & 0.25 & */* & 7.3 & 70.01 & 6.90 & 70.23 & 6.20 & 69.97 & 6.90 & 70.23 \\  & 0.50 & */* & 7.9 & 70.50 & 8.30 & 71.31 & 7.30 & 70.05 & 7.70 & 71.25 \\  & 0.75 & */* & 9.9 & 71.20 & 9.70 & 71.79 & 8.20 & 70.13 & 8.90 & 71.56 \\  & 0.80 & */* & 10.7 & 71.91 & 10.00 & 72.53 & 10.00 & 71.12 & 10.00 & 72.53 \\ \hline \end{tabular}
\end{table}
Table 1: Ablation experiments of different optimization factor calculations on ResNet18 and MobileNetV2. We select HAWQ-v3 as our Vanilla baseline, which is a Hessian-based method. The red data indicates the best performance and cyan data is the second best. W/A is the bit-width of weight and activation. * represents mixed precision.
Table 2 underscores OHQ as the optimal 8-bit quantization strategy across diverse networks. The optimal trade-off between precision and compression ratio in mixed-precision quantization is also shown. In the performance of ResNet18, an accuracy of 70.08% is attained, harmoniously juxtaposed with a compact footprint of 5.8 Mb and latency of 63.5 ms. Meanwhile, the ResNet50 model, compressed to a size of 17.8 Mb with the speed of 147.8 ms, exhibits an accuracy of 77.55%. The highest accuracy and lowest compression Rate and time achieved in MobileNetV2 (71.64%, 1.7 Mb, 70.1 ms) Notably, the MobileNetV3 network attains an accuracy of 73.01% while preserving a remarkably small size of 2.4 Mb and latency 0f 73.4 ms.

We also combine QAT and OHQ due to the ability of the QAT method to improve quantization model accuracy through data retraining. In the ResNet18 network we achieve, with only 6.9Mb of the model, 70.23% accuracy, which is ahead of HAWQ-v3 in terms of compression and performance. Notably, PACT does not quantize the input and output layers, and the activations are retained 23 bits from HAQ and LQ-Nets. For ResNet50, OHQ shows the best results(76.64%, 16.5 Mb, 135.9 Ms). Moreover, in MobileNetV3, the accuracy of OHQ is 0.08% and 0.46% higher than that of HAQ and GZNQ, respectively, and the compression rate of the model is 20% higher than that of GZNQ, which still shows the sota quantization performance. Notably, all the OHQ quantization models have the fastest inference on FPGA.

## Conclusion

In this paper, our proposed OHQ introduces an innovative and effective solution for hardware-aware mixed-precision quantization, offering a substantial stride toward efficient and accurate deployment of DNNs on resource-constrained chips. Firstly, the OQA pipeline furnishes an avenue to comprehend the true efficiency metrics of quantization operators within the hardware ecosystem and yields insights that inform subsequent optimization steps. Secondly, the MQE technique is meticulously designed to efficiently gauge accuracy metrics for operators while adhering to on-chip computational constraints. Synthesizing network and hardware insights through linear programming, we derive optimal bit-width configurations. A remarkable facet of our approach is that the entire quantization process unfolds on-chip.

\begin{table}
\begin{tabular}{l l c c c c c c c} Arch & Method & Int-Only & Uniform & W/A & Data & Size (Mb) & Latency(ms) & Top-1 (\%) \\ \hline \hline \multirow{2}{*}{ResNet18} & Baseline & \(\bigtimes\) & - & 32/32 & 1.2E6 & 44.6 & 39.6\({}^{\star}\) & 73.21 \\ \cline{2-8}  & Min\&Max & \(\bigtimes\) & \(\bigvee\) & 8/8 & 1.2E6 & 11.1 & 78.3 & 71.38 \\ \hline \multirow{8}{*}{ResNet50} & **OHQ (ours)** & \(\bigvee\) & \(\bigvee\) & 8/8 & 32 \(\div\) & 11.1 & 78.3 & 71.52 \\ \cline{2-8}  & ZeroQ & \(\bigvee\) & \(\bigvee\) & */* & 32 \(\div\) & 5.8 & 67.9 & 21.20 \\  & BRECQ & \(\bigvee\) & \(\bigvee\) & */* & 1024 & 5.8 & 67.6 & 69.32 \\ \cline{2-8}  & **OHQ (ours)** & \(\bigvee\) & \(\bigvee\) & */* & 32 \(\div\) & 5.8 & 63.5 & **70.08** \\ \cline{2-8}  & Baseline & \(\bigtimes\) & - & 32/32 & 1.2E6 & 97.8 & 80.2\({}^{\star}\) & 77.72 \\ \cline{2-8}  & Min\&Max & \(\bigtimes\) & \(\bigvee\) & 8/8 & 1.2E6 & 24.5 & 182.6 & 77.70 \\ \cline{2-8}  & **OHQ (ours)** & \(\bigvee\) & \(\bigvee\) & 8/8 & 32 \(\div\) & 24.5 & 182.6 & 77.72 \\ \cline{2-8}  & OCS & \(\bigvee\) & \(\bigvee\) & 6/6 & 1.2E6 & 18.4 & 159.6 & 74.80 \\  & ZeroQ & \(\bigvee\) & \(\bigvee\) & */6 & 32 \(\div\) & 18.3 & 160.3 & 77.43 \\ \cline{2-8}  & **OHQ (ours)** & \(\bigvee\) & \(\bigvee\) & */* & 32 \(\div\) & 17.8 & 147.8 & 77.55 \\ \hline \multirow{2}{*}{MobileNetV2} & Baseline & \(\bigtimes\) & - & 32/32 & 1.2E6 & 13.4 & 11.3\({}^{\star}\) & 73.03 \\ \cline{2-8}  & Min\&Max & \(\bigtimes\) & \(\bigvee\) & 8/8 & 1.2E6 & 3.3 & 100.7 & 70.29 \\ \cline{2-8}  & DFQ & \(\bigvee\) & \(\bigvee\) & 8/8 & 32 \(\div\) & 3.3 & 100.7 & 71.20 \\ \cline{2-8}  & **OHQ (ours)** & \(\bigvee\) & \(\bigvee\) & 8/8 & 32 \(\div\) & 3.3 & 100.7 & 73.00 \\ \cline{2-8}  & FracBits & \(\bigvee\) & \(\bigvee\) & */* & 32 \(\div\) & 1.8 & 85.4 & 69.90 \\ \cline{2-8}  & **OHQ (ours)** & \(\bigvee\) & \(\bigvee\) & */* & 32 \(\div\) & 1.7 & 70.1 & 71.63 \\ \hline \multirow{2}{*}{MobileNetV3} & Baseline & \(\bigtimes\) & - & 32/32 & 1.2E6 & 15.3 & 10.6\({}^{\star}\) & 74.32 \\ \cline{2-8}  & Min\&Max & \(\bigtimes\) & \(\bigvee\) & 8/8 & 32 \(\div\) & 3.9 & 85.0 & 72.98 \\ \cline{2-8}  & **OHQ (ours)** & \(\bigvee\) & \(\bigvee\) & 8/8 & 32 \(\div\) & 3.9 & 85.0 & 74.29 \\ \cline{2-8}  & **OHQ (ours)** & \(\bigvee\) & \(\bigvee\) & */* & 32 \(\div\) & 2.4 & 73.4 & 73.01 \\ \hline \end{tabular}
\end{table}
Table 2: Results of PTQ methods with ResNet18, ResNet50, MobileNetV2, and MobileNetV3. \(\dagger\) indicates using distilled data in the quantization process. \({}^{\star}\) means the latency results are tested on CPU, others are deployed on FPGA (batch=1).

\begin{table}
\begin{tabular}{l l c c c c c} Arch & Method & W/A & Int-Only & \begin{tabular}{c} Size \\ (Mb) \\ \end{tabular} & \begin{tabular}{c} Latency \\ (ms) \\ \end{tabular} & 
\begin{tabular}{c} Top-1 \\ (\%) \\ \end{tabular} \\ \hline \hline \multirow{2}{*}{ResNet18} & PACT\(\sharp\) & 5/5 & \(\bigtimes\) & 7.2 & 70.4 & 69.80 \\  & HAWQ-v3 & */* & \(\bigvee\) & 7.3 & 72.9 & 70.01 \\  & **OHQ (ours)** & */* & \(\bigvee\) & 6.9 & 68.3 & 70.23 \\ \hline \multirow{2}{*}{ResNet50} & HAWQ-v3 & */* & \(\bigvee\) & 18.7 & 172.1 & 75.39 \\  & **OHQ (ours)** & */* & \(\bigvee\) & 16.5 & 135.9 & 76.64 \\ \hline \multirow{2}{*}{MobileNetV2} & GZNQ & 6/6 & \(\bigvee\) & 2.5 & 81.7 & 71.10 \\  & HAQ & */* & \(\bigtimes\) & 1.8 & 75.5 & 71.47 \\ \cline{2-8}  & **OHQ (ours)** & */* & \(\bigvee\) & 1.7 & 70.1 & 72.56 \\ \hline \end{tabular}
\end{table}
Table 3: Results of QAT with ResNet18/50 and MobileNetV2. \(\ddagger\) means not quantizing the first and last layers.