GHN-QAT: Training Graph Hypernetworks to Predict Quantization-Robust
  Parameters of Unseen Limited Precision Neural Networks
Graph Hypernetworks (GHN) can predict the parameters of varying unseen CNN
architectures with surprisingly good accuracy at a fraction of the cost of
iterative optimization. Following these successes, preliminary research has
explored the use of GHNs to predict quantization-robust parameters for 8-bit
and 4-bit quantized CNNs. However, this early work leveraged full-precision
float32 training and only quantized for testing. We explore the impact of
quantization-aware training and/or other quantization-based training strategies
on quantized robustness and performance of GHN predicted parameters for
low-precision CNNs. We show that quantization-aware training can significantly
improve quantized accuracy for GHN predicted parameters of 4-bit quantized CNNs
and even lead to greater-than-random accuracy for 2-bit quantized CNNs. These
promising results open the door for future explorations such as investigating
the use of GHN predicted parameters as initialization for further quantized
training of individual CNNs, further exploration of "extreme bitwidth"
quantization, and mixed precision quantization schemes.
Stone Yun, Alexander Wong
GHN-QAT: Training Graph Hypernetworks to Predict Quantization-Robust Parameters of Unseen Limited Precision Neural Networks

###### Abstract

Graph Hypernetworks (GHN) can predict the parameters of varying unseen CNN architectures with surprisingly good accuracy at a fraction of the cost of iterative optimization. Following these successes, preliminary research has explored the use of GHNs to predict quantization-robust parameters for 8-bit and 4-bit quantized CNNs. However, this early work leveraged full-precision float32 training and only quantized for testing. We explore the impact of quantization-aware training and/or other quantization-based training strategies on quantized robustness and performance of GHN predicted parameters for low-precision CNNs. We show that quantization-aware training can significantly improve quantized accuracy for GHN predicted parameters of 4-bit quantized CNNs and even lead to greater-than-random accuracy for 2-bit quantized CNNs. These promising results open the door for future explorations such as investigating the use of GHN predicted parameters as initialization for further quantized training of individual CNNs, further exploration of "extreme bitwidth" quantization, and mixed precision quantization schemes.

## 1 Introduction

Low-bit neural networks that use limited precision quantization for inference [1; 2] can make state-of-the-art models much more practical. However, perturbations induced by quantization of weights and activations can change DNN behaviour in non-trivial ways. In some cases, state-of-the-art performance can have significant degradation after quantization of weights and activations [3]. So how can we find high-performant, quantization robust CNN parameters?

Recent works by [4] and [5] have shown remarkable performance using Graph Hypernetworks (GHN) to predict all trainable parameters of unseen DNNs in a _single forward pass_. Preliminary research in [6] has explored the use of GHNs to predict quantization-robust parameters for 8-bit and 4-bit quantized CNNs. However, this work trained GHNs to predict parameters for full-precision float32 candidate CNNs and only quantized them for testing. Building on this, we explore quantization-specific training and find that quantization-aware training of GHNs (which we refer to as GHN-QAT) can significantly improve quantized accuracy for GHN predicted parameters of 4-bit quantized CNNs and even achieve greater-than-random accuracy for 2-bit CNNs. More specifically, we simulated quantization (SimQuant) in sampled CNNs such that GHN-QAT adapts to the quantization errors
induced by quantizing GHN-predicted models (see Fig. 1). By finetuning GHNs on a mobile-friendly, **quantized** CNN architecture space, GHNs learn representations specifically for efficient quantized CNNs.

## 2 Experiment

We first investigate SimQuant based quantization training (commonly referred to as quantization-aware training/QAT) on a target design space for limited precision quantization. We evaluate a few low bit-width settings (W4/A4, W4/A8, W2/A2 where W indicates weight bitwidth and A indicates activation bitwidth). Using SimQuant for W2/A2 proved to be unstable and we found that modelling quantization as uniform noise (NoiseQuant) led to much better results. The reported W2/A2 results are from training with NoiseQuant where the sampling distribution is computed based on 2-bit precision. In all cases, GHN-QAT training is precision/bitwidth-specific. Encoding bitwidth into the CNN graph could potentially remove the need for bit-width specific finetuning. We finetuned a CIFAR-10, DeepNets-1M pretrained GHN-2 model obtained from [7] on the ConvNets-250K graph dataset from [6]. We use tensorwise, asymmetric, uniform quantization throughout. Figure 1 shows how GHN-QAT is finetuned to predict efficient, quantization-robust CNNs.

GHN-QAT was finetuned on ConvNets-250K for 100 epochs using CIFAR-10 [8]. We follow a testing procedure similar to [4], [6] and evaluate GHN-QAT by comparing the mean CIFAR-10 test accuracy at the stated target precisions. Table 1 shows the top-1 and top-5 accuracy results on different testing splits. To establish the benefits of QAT, we also include results from [6] where the authors used full-precision float32 training and only quantized CNNs for testing. In [6], W2/A2 degraded to random accuracy. We use weight-tiling and normalization as described in [4] and use \(s^{(max)}=10\) for max shortest path of virtual edges.

## 3 Discussion

As demonstrated, we can easily simulate quantization of CNNs to arbitrary precision. Furthermore, we could even model other scalar quantization methods. Thus, GHN-QAT becomes a powerful tool for quantization-aware design of efficient CNN architectures. The parameters predicted by GHN-QAT are remarkably robust and the QAT finetuning results (see Table 1) show a significant improvement over the full-precision float32 finetuning used in [6]. This shows a clear benefit to adapting GHNs specifically to predict parameters for quantization-aware graphs. Additional possibilities/challenges of leveraging quantization-aware training, such as learned quantization thresholds or reducing QAT

Figure 1: GHN-QAT finetuned on ConvNets-250K. We generate a large number of CNN graphs which are then quantized to target bitwidth for training. Once trained, GHN-QAT can predict robust parameters for unseen CNNs.
oscillations like in [9], should be explored to further improve GHN-QAT, especially for "extreme" low bitwidths. It's possible that such improvements to QAT would make SimQuant more stable for 2-bit quantization.

From GHN-QAT, we can see that introducing quantization into our GHN training allows for greater use of GHNs for quantization-specific neural network parameter prediction. Besides leveraging GHN-QAT for quantized versions of floating point operations, we should be able to encode quantization information such as bit-width and quantization scheme into the graphs. If used as a form of quantized accuracy prediction, GHN-QAT could greatly accelerate the process of searching for accurate, quantized CNNs. Additionally, GHN-QAT could be a useful weight initialization for quantized CNN training. The authors of [10] found noticeable differences in quantized accuracy of CNNs depending on their initialization. Thus, a quantization-aware weight initialization could be more robust than random initialization. If GHN-QAT-predicted parameters can be used as initialization for quantization-aware training rather than first training models to convergence in full float precision before additional QAT, then the training time of quantized models would be significantly reduced. Unfortunately, GHNs do not yet match the accuracy of iterative optimization and further improvements should be explored to bridge this gap. However, the aforementioned benefits could still greatly reduce the costs of designing quantized CNNs.
