On Calibration of Modern Quantized Efficient Neural Networks
We explore calibration properties at various precisions for three
architectures: ShuffleNetv2, GhostNet-VGG, and MobileOne; and two datasets:
CIFAR-100 and PathMNIST. The quality of calibration is observed to track the
quantization quality; it is well-documented that performance worsens with lower
precision, and we observe a similar correlation with poorer calibration. This
becomes especially egregious at 4-bit activation regime. GhostNet-VGG is shown
to be the most robust to overall performance drop at lower precision. We find
that temperature scaling can improve calibration error for quantized networks,
with some caveats. We hope that these preliminary insights can lead to more
opportunities for explainable and reliable EdgeML.
Joey Kuang, Alexander Wong
# On Calibration of Modern Quantized Efficient Neural Networks

###### Abstract

We explore calibration properties at various precisions for three architectures: ShuffleNetv2, GhostNet-VGG, and MobileOne; and two datasets: CIFAR-100 and PathMNIST. The quality of calibration is observed to track the quantization quality; it is well-documented that performance worsens with lower precision, and we observe a similar correlation with poorer calibration. This becomes especially egregious at 4-bit activation regime. GhostNet-VGG is shown to be the most robust to overall performance drop at lower precision. We find that temperature scaling can improve calibration error for quantized networks, with some caveats. We hope that these preliminary insights can lead to more opportunities for explainable and reliable EdgeML.

## 1 Introduction

Enabling critical decision-making machine learning applications on the edge, such as point-of-care testing, increases efficiency and accessibility [1]. Deploying models may require quantization, which calls for consideration to performance drop. Meanwhile, critical decision-making leaves little room for predictive error, thus emphasizing the role of model uncertainty.

The seminal work of Guo _et al_. [2] revealing the calibration dilemma present in modern neural networks brought forth calibration estimators and methods, and studies of intrinsic calibration properties of an architecture [3; 4; 5; 6; 7; 8]. There are few works that intersect the model's quantization behaviour and calibration quality. Studies regarding model quantization have focused on reducing undesirable quantization noise by way of novel quantization methods [9; 10; 11; 12; 13; 14]. Intuitively, we recognize that error in quantized outputs can cause decision swaps and change in confidence. In turn, we suspect possible consequences on calibration properties. One work by Xia _et al_. [15] observes that better calibrated models tend to provide worse post-quantization performance. In this extended abstract, we continue to explore the relationship between quantization and calibration: what do model calibration properties look like across various precision bitwidths and how well does a quantized model receive a post-hoc calibration method?

## 2 Experiments

We conduct our analyses on the CIFAR-100 (32x32) dataset and PathMNIST2D (28x28) [16; 17]. Using biomedical image data provides context for understanding calibration properties due to related high-stakes tasks, such as lesion detection [18]. PathMNIST2D (9 classes, 107k samples total) was sourced from research on predicting survival from colorectal cancer histology slides.
To investigate from a more practical perspective, we select architectures designed for mobile targets and likely to face quantization. Due to the small image sizes involved, we make modifications to the model based on the source paper or other relevant works. For ShuffleNetv2, We follow AugShuffleNet [19] which was modified for CIFAR datasets. While the main GhostNet is built on MobileNetV3 [20], Han _et al_. [21] run toy experiments on CIFAR datasets with a VGG16-like network provided by [22]. With best judgement, we reduce depth, stride, and width of the MobileOne S0 variant to avoid excessive downsampling and overcomplexity [23].

### Training

We train models for 200 epochs with an effective batch size of 32 using SGD with momentum (\(\beta=0.9\)) [24]. We initialize with He initialization [25] and train on cross-entropy loss with a weight decay set to \(10^{-4}\). Note that we specifically avoid other training time regularization methods to avoid confounding with post-hoc calibration [2]. We initialize our learning rate at 0.1 and decay it using a cosine schedule with warm restarts; we follow Scenario 6 of CIFAR-10 settings from [26]. We apply random crop and random horizontal flip.

We follow a modified implementation of the Pytorch post-training quantization library by Xia _et al_. [15] to quantize our weights symmetrically per-channel with a moving average min-max observer; activations are quantized asymmetrically per-tensor with a histogram-based observer. We perform quantization at 3 configurations: W8/A8, W4/A8, W4/A4. Each configuration is quantized at 3 trials with randomly sampled training batches for calibrating ranges, where total calibration size is 1024 [27]. All quantized _inference_ is performed at 8-bit. We choose temperature scaling (TS) as our sole post-hoc calibration method [2] and Expected Calibration Error (ECE) as the primary metric for calibration quality (lower is better) [28].

Figure 1: ECE and top-1 error before (left column) and after (right column) temperature scaling. Model backbones are grouped by color and precision levels are grouped by marker size. Lower bitwidth models are correlated with both poorer accuracy and poorer calibration.
Figure 3: Reliability grid (blue and red) and confidence distribution (striped teal) of CIFAR-100 models (by column) of each precision level (by row) after temperature scaling. TS does well to correct confidence-accuracy gaps of ShuffleNetV2 and GhostNet-VGG, while extremely poorly calibrated pre-TS 4-bit weight MobileOne models cannot recover.

Figure 2: Reliability grid (blue and red) and confidence distribution (striped teal) of CIFAR-100 models (by column) of each precision level (by row) before temperature scaling. Larger red bars indicate greater (worse) confidence-accuracy gap. Although MobileOne FP32 is well-calibrated, more so than GhostNet-VGG FP32 and ShuffleNetV2 FP32, its quantized counterparts are very poorly calibrated.
Figure 4: Reliability grid (blue and red) and confidence distribution (striped teal) of PathMNIST models (by column) of each precision level (by row) before temperature scaling. 4-bit weight model performance and calibration consistently deteriorates for models using inverted residual blocks (ShuffleNetV2 and MobileOne).

Figure 5: Reliability grid (blue and red) and confidence distribution (striped teal) of PathMNIST models (by column) of each precision level (by row) before temperature scaling. While confidence-accuracy gap tends to diminish post-TS, improvement is not as noticeable under PathMNIST settings, especially when the initial gap is incredibly large.
Results and Discussion

Figure 1 shows that calibration properties are consistent between models and their precision variants across datasets. Minderer _et al_. [4] has suggested that architecture is a key determinant of calibration properties. The exact factors are unclear, though they do find that it is not related to model size nor amount of pretraining. We observe that GhostNet-VGG is the most robust to both quantization-induced accuracy decrease and ECE increase, while lower-bit ShuffleNetV2 and MobileOne model performance suffers. Notably, these two backbones consist of inverted residual blocks employing depthwise-separable convolutions, which have been shown to suffer from larger accumulation of quantization errors [29].

We find that TS is at some point limited by how well the model performs in the first place. First, we observe expected TS performance between Figure 2 and Figure 3 for most models, where post-TS models are well-calibrated. However, TS is unable to correct the 4-bit MobileOne models that have a larger initial confidence-accuracy gap. This is further shown by Figure 4, emphasized on PathMNIST: higher-precision models are accurate, well-calibrated (vs. counterpart trained on CIFAR-100), and refined (making more confident decisions) resulting in a tight cluster near the origin in Figure 1. Despite this success, all W4/A4 models perform terribly w.r.t. ECE and Top-1 and large confidence-accuracy gaps remain in post-TS models in Figures 3 and 5. This is consistent with [15]; in addition to well-calibrated floating point models posting worse post-quantization performance, their calibration quality appears to drop as well. Their conclusions were drawn from exploring ResNet models, which points to residual connections as a suspect for both of our findings. Interestingly, while prior work [2, 4] has observed that within a model family (e.g., grouped by size), a model with lower classification error tends to have higher calibration error, we show the exact opposite within a set of model's quantized counterparts. We also find the large discrepancy in quantization performance of ShuffleNetV2 particularly interesting between PathMNIST and CIFAR-100 settings; it may be worth investigating in relation to ECE biases and dataset attributes.

Further analysis includes investigating the correlation between the pre- and post-TS ECE delta and the overall quantization quality to understand possible parallelisms between calibration properties and quantization behaviour. While we provided an initial discussion on certain design patterns contributing to a deployable model's dismay, additional exploration on intrinsic architecture properties relating to quantization _and_ calibration, training policies, quantization methods, and calibrators, are paramount to outline best practices for _trustworthy_ EdgeML solutions.
