Calibrating a Deep Neural Network with Its Predecessors
Confidence calibration - the process to calibrate the output probability
distribution of neural networks - is essential for safety-critical applications
of such networks. Recent works verify the link between mis-calibration and
overfitting. However, early stopping, as a well-known technique to mitigate
overfitting, fails to calibrate networks. In this work, we study the limitions
of early stopping and comprehensively analyze the overfitting problem of a
network considering each individual block. We then propose a novel
regularization method, predecessor combination search (PCS), to improve
calibration by searching a combination of best-fitting block predecessors,
where block predecessors are the corresponding network blocks with weight
parameters from earlier training stages. PCS achieves the state-of-the-art
calibration performance on multiple datasets and architectures. In addition,
PCS improves model robustness under dataset distribution shift.
Linwei Tao, Minjing Dong, Daochang Liu, Changming Sun, Chang Xu
# Calibrating a Deep Neural Network with Its Predecessors

###### Abstract

Confidence calibration - the process to calibrate the output probability distribution of neural networks - is essential for safety-critical applications of such networks. Recent works verify the link between mis-calibration and overfitting. However, early stopping, as a well-known technique to mitigate overfitting, fails to calibrate networks. In this work, we study the limitations of early stopping and comprehensively analyze the overfitting problem of a network considering each individual block. We then propose a novel regularization method, predecessor combination search (PCS), to improve calibration by searching a combination of best-fitting block predecessors, where block predecessors are the corresponding network blocks with weight parameters from earlier training stages. PCS achieves the state-of-the-art calibration performance on multiple datasets and architectures. In addition, PCS improves model robustness under dataset distribution shift.

## 1 Introduction

Deep neural networks (DNNs) have achieved great successes across a variety of domains [3], especially on classification related tasks such as object detection [12, 13, 14, 15] and image classification [16, 17, 18, 19], reaching prediction accuracy far beyond human beings. However, they still suffer from mis-calibrated predictions in the sense that the prediction probability cannot represent the ground-truth probability associated with the class label, either in an over- or under-confidence manner. This may lead to fatal problems when any safety-critical downstream tasks such as autonomous driving [1] and medical diagnosis [1] rely heavily on the prediction probability.

The underlying cause for mis-calibrated predictions is associated with the capacity of modern neural networks that makes them vulnerable to overfitting [19]. Mukhoti _et al._[14] show that overfitting in modern neural networks mostly results from the overconfidence on mis-classified samples and they empirically verify the strong connection between the overfitting issue and calibration performance. Given this observation, some regularization techniques such as weight decay [19], label smoothing [10], and data augmentation [10, 1] are introduced to improve model calibration.

Early stopping [10] is another well-known regularization method, which suspends training once the model performance stops improving on a hold out validation dataset. Mukhoti _et al._[14] conduct a series of empirical experiments and demonstrate that early stopping on training according to multiple criteria fails to yield a well-calibrated model. We mainly attribute this sub-optimal solution to the unitary strategy of conventional early stopping techniques which treat the entire network as a whole. Specifically, an early stopping technique takes a DNN as a black box without investigating the internal components, i.e., the blocks inside the network. However, the increasing depth of modern DNNs makes the optimization more challenging, which could lead to discrepancies of convergence speeds of different blocks in DNNs. Thus, any model calibration via a conventional early-stopping technique could be a sub-optimal solution.

In this paper, instead of taking a DNN as a whole, we consider a block in a DNN as the basic unit and explore the overfitting problem in each block. We empirically observe that blocks in a network overfit at different stages during training. Unlike the conventional early stopping approach [10] that stops the training of the whole network at a certain point to form a network predecessor, we propose to stop the training of each block at its own best-fitting block predecessor to improve model calibration. However, the blocks in DNNs are strongly coupled with each other, and the early stopping of an individual block independently does not ensure an optimal solution. To achieve an effective and adaptive early stopping for each block, we take into consideration all possible block predecessor combinations. Our objective is to discover the predecessor combination (PC) with better calibration performance. We propose a neural architecture search inspired approach, predecessor combination search (PCS) to calibrate the DNNs, which performs a differential search of the optimal block predecessor combination through a relaxation of the search space as well as a predecessors evaluation estimator.
Our contribution can be summarized as follows: **(1)** We study the overfitting problem of individual blocks empirically and show that different blocks reach their own overfitting points at different stages of training. **(2)** We propose a novel differential PCS method to search a better-calibrated model together with a sampling strategy to improve searching efficiency. **(3)** PCS achieves state-of-the-art results for both pre- and post-temperature scaling [14] on a variety of datasets and architectures via a large number of experiments. We show that PCS works well on out-of-distribution (OoD) samples by shifting the dataset from CIFAR-10 to SVHN [1] and CIFAR-10-C [1].

## 2 Related Works

Due to the mis-calibration problem in modern neural networks [14] and the significant importance of calibration, many techniques [20, 1] have been proposed in recent years. The current calibration methods could be divided into three categories. The first category modifies the training loss by replacing the conventionally used cross-entropy loss with a mean square error loss [15] or a focal loss [16] or by adding an auxiliary regularization loss such as the MMCE loss [17] and the AvUC loss [18]. Bohdal _et al._[1] propose a differentiable surrogate for expected calibration error that improves the calibration performance directly. The recent work in [17] introduces a differentiable bin membership function and applies it on bin-based metrics such as expected calibration error (ECE) [14] to make it become a differentiable auxiliary calibration loss.

Another category is the post-hoc calibration approaches that improve the calibration performance by modifying the prediction logits. Platt scaling [22] learns parameters to perform a linear transformation on the original prediction logits. Isotonic regression [1] learns piece-wise functions to transform the original prediction logits. Histogram binning [1] obtains calibrated probability estimates from decision trees and naive Bayesian classifiers. Bayesian binning into quantiles (BBQ) [10] is an extension of histogram binning with Bayesian model averaging. Beta calibration [15] is proposed for binary classification and Kull _et al._[16] generalize the beta calibration method from binary classification to multi-classification with Dirichlet distributions. Wenger _et al._[1] employ a non-parametric representation using a latent Gaussian process. Among these methods, temperature scaling is the most popular post-hoc approach, which tunes the temperature parameter of the softmax function that minimizes the negative log likelihood (NLL) and does not change the prediction results. In this work, we present the calibration performance with both before and after temperature scaling.

All other regularization methods that can calibrate networks form the third category. Label smoothing [13] implicitly calibrates networks by artificially softening targets to prevent a model from overfitting to the "hard label". Mixup [12] and AugMix [1] are two popular data augmentation techniques for calibration. The mix step in data augmentation increases the generality of datasets and reduces the influence of hard samples that can easily cause over-confidence problem. Weight decay, which dominated regularization methods for neural networks in the past, is now less often used by modern neural networks. However, it still plays an important role in improving model calibration [14].

## 3 Problem Formulation

Considering a dataset \(\mathcal{D}=\langle(x_{i},y_{i})\rangle_{i=1}^{N}\) with \(N\) samples from a joint distribution \((\mathcal{X},\mathcal{Y})\), the ground-truth class label is \(y_{i}\in\{1,2,...,\mathcal{K}\}\), where \(\mathcal{K}\) denotes the number of classes. The probability for a class \(y_{i}\) on a given input \(x_{i}\) predicted by network \(F\) with model parameters \(\Theta\) is denoted as \(\hat{p}_{i,y_{i}}=F_{\Theta}(y_{i}|x_{i})\). The predicted label \(\hat{y}_{i}\) and the corresponding confidence \(\hat{p}_{i}\) are defined as

\[\hat{y}_{i} =\operatorname{argmax}_{y_{i}\in\{1,2,...,\mathcal{K}\}}\,\hat{p }_{i,y_{i}},\] \[\hat{p}_{i} =\max_{y_{i}\in\{1,2,...,\mathcal{K}\}}\,\hat{p}_{i,y_{i}}. \tag{1}\]

When the model is _perfectly calibrated_, the prediction confidence \(\hat{p}\) is expected to represent the real probability \(p\) for each sample \(x_{i}\) with class label \(y_{i}\). In other words, the model accuracy \(\mathbb{P}(\hat{y}=y|\hat{p}=p)\) is \(p\), for all \(p\in[0,1]\).

ECE is a widely-accepted metric to measure calibration performance. Formally, the ECE is defined as the expected absolute difference between the model's confidence and its accuracy, which can be formulated as

\[\mathrm{ECE}=\mathbb{E}_{\hat{p}}\big{[}\left|\mathbb{P}(\hat{y}=y|\hat{p})- \hat{p}\right|\big{]}. \tag{2}\]

Due to the finite samples in datasets, Guo _et al._[13] estimate ECE by dividing confidence \(p\in[0,1]\) into \(\mathbb{B}\) equal-width bins. \(B_{i}\) denotes the set of samples with confidences within \(\left(\frac{i-1}{\mathbb{B}},\frac{i}{\mathbb{B}}\right]\). Let \(I_{i}\) and \(C_{i}\) denote the accuracy and average confidence of all samples in bin \(B_{i}\) respectively. Accuracy of bin \(B_{i}\) is computed as \(I_{i}=\frac{1}{|B_{i}|}\sum_{j\in B_{i}}\mathbb{1}\) (\(\hat{y}_{j}=y_{j}\)), where \(\mathbb{1}\) is the indicator function, and \(\hat{y}_{j}\) and \(y_{j}\) are the predicted and ground-truth labels for the \(j^{\mathrm{th}}\) sample. Similarly, the confidence \(C_{i}\) of the \(i^{\mathrm{th}}\) bin is computed as \(C_{i}=\frac{1}{|B_{i}|}\sum_{j\in B_{i}}\hat{p}_{j}\), i.e., \(C_{i}\) is the average confidence of all samples in the bin. Thus, in practice, ECE is formulated as the weighted average of accuracy-confidence difference for the bins:

\[\mathrm{ECE}=\sum_{i=1}^{\mathbb{B}}\frac{|B_{i}|}{N}\left|I_{i}-C_{i}\right|. \tag{3}\]

Along with ECE, the maximum calibration error (MCE) is proposed to minimize the influence of worst-case confidence deviation, which is defined as the maximum difference of bins' accuracy and confidence: \(\mathrm{MCE}=\max_{i\in\mathbb{1},...,\mathbb{B}}\left|I_{i}-C_{i}\right|\).

Guo _et al._[13] and Mukhoti _et al._[1] state that the mis-calibration problem of networks is strongly related to overfitting on training sets. Early stopping, as a well-known regularization technique, mitigates the overfitting problem by
stopping the whole network at the best-fitting epoch. According to previous observations, early stopping should yield a well calibrated model. However, the calibration performance of an early stopped model is often far from satisfactory. Thus, we hypothesize that shallow and deep blocks may suffer from different degrees of overfitting during optimization and a unified regularization criteria to the entire network \(F_{\Theta}\) could lead to a sub-optimal solution for model calibration. Instead, it would be more desirable to resolve the overfitting problems block-wise and explore an adaptive regularization approach to improve the model calibration performance.

### Overfitting in Blocks

Most advanced CNN model architectures are the stack of blocks [10]. Suppose there are \(M\) blocks in the network, the model parameters can be represented as \(\Theta=\{\theta_{i}|i=1,2,\dots,M\}\), where \(\theta_{i}\) denotes the parameter of the \(i^{\text{th}}\) block. \(\Theta^{j}\) and \(\theta^{j}\) are used to identify the network and block parameters at the \(j^{\text{th}}\) training epoch. The degree of fitting of \(F_{\Theta}\) is represented by the trend of validation NLL loss \(\mathcal{L}_{NLL}(F_{\Theta^{j}}(x),y),j=1,2,...,T_{train}\), where \(T_{train}\) is the number of total training epochs over the training process. Conventional early stopping aims to find the weights at a "sweet point"1 epoch to balance between underfitting and overfitting. However, there is no definition of overfitting of individual network blocks. In order to find the "sweet point" epoch of each block, we need to measure the degree of overfitting of an individual block \(f_{i}\). We formally define the _pre-decessors_ of block \(f_{i}\) as \(\{f_{i}^{j}|j\in\{1,2,\dots,T_{train}\}\}\), where \(f_{i}^{j}\) denotes the \(i^{\text{th}}\) block with weight at the \(j^{\text{th}}\) epoch. To achieve the overfitting measurement of \(f_{i}\), we treat the other blocks of the network \(\tilde{F}_{i}=\{f_{i^{-}}|i^{-}\in\{1,2,\dots,M\}\setminus\{i\}\}\) as constant mappings. Then the overfitting degree of block \(f_{i}\) could be represented by the trend of validation NLL loss over the training process \(\mathcal{L}_{NLL}(F(x),y;f_{i}^{j}),j=1,2,...,T_{train}\), with the weights of other network blocks \(\tilde{F}_{i}\) fixed. Based on this idea, an empirical study is conducted to investigate the overfitting behaviors of individual network blocks. A more formal definition of the overfitting measurement of an individual network block is given in the supplementary.

Footnote 1: A sweet point is defined as the balance point between underfitting and overfitting, which normally refers to the lowest point of the validation loss curve.

We present the empirical study in Figure 1, demonstrating the different overfitting behaviors of individual network blocks. A ResNet-50 is first trained on CIFAR-10 for \(T_{train}=350\) epochs. Then each sub-figure shows the evaluation of overfitting behavior of block \(f_{i}\) and each block other than \(f_{i}\) is fixed to a certain predecessor. Note that it is non-trivial to properly select the fixed predecessors for all other blocks \(\tilde{F}_{i}\) and ideally these blocks need to be at their own "sweet points". We adopt a simplified approach and fix \(\tilde{F}_{i}\) at the "sweet point" of the whole model as an approximation, i.e., \(\tilde{F}_{i}=\{f_{i^{-}}^{\rho^{*}}|i^{-}\in\{1,2,\dots,M\}\setminus\{i\}\}\), where \(\rho^{*}\) is obtained either by early stopping the whole model based on the validation loss or classification error. Apart from the validation loss, the validation ECE and classification error are also plotted in Figure 1. The proportionally scaled values of the validation loss and ECE are reported for better visualisation. In the supplementary, another set of evaluation results are reported, where the other blocks are fixed to the weight parameters of the model at the end of training, i.e., \(\rho^{*}=T_{train}\). Through the variation of the validation loss of each block \(f_{i}\) in this experiment as shown in Figure 1, we have the following intriguing observations on overfitting in blocks.

**1. Deeper convolutional blocks tend to have more severe overfitting problems with training.** From the first four columns in Figure 1, we observe that all convolutional blocks with the same block index share a similar overfitting pattern for both settings. However, when we look into each row, a different pattern shows on the convolutional blocks compared with each other. The deeper convolutional block, i.e., \(f_{4}\)

Figure 1: **Empirical evidence on that different blocks have different overfitting behaviors. A ResNet-50 on CIFAR-10 is first trained with 350 epochs with the cross-entropy loss and learning rate scheduling at epochs 150 and 250. ResNet-50 has 5 blocks, i.e., 4 convolutional blocks and 1 final linear layer. Each sub-figure shows the overfitting issue of block \(f_{i}\) through the training epochs with \(\tilde{F}_{i}\) fixed at potential “sweet point” predecessors. Top row and bottom row represent \(\tilde{F}_{i}\) fixed at different predecessors, which are the “sweet point” epoch of the whole model by early stopping on loss (epoch 151) and error (epoch 281) respectively. Columns represent different block \(f_{i}\) under study. All combinations are fine-tuned with one epoch on the training set.**
starts rapid overfitting after the first learning rate scheduler point.

**2. The overfitting problem of the final linear layer is much more complex and depends heavily on other blocks.** When it comes to the final linear block \(f_{5}\) at the last column, the loss presents a totally reverse pattern with different \(\tilde{F}_{i}\). The final block \(f_{5}\) with \(\tilde{F}_{i}\) fixed at predecessors early stopping on loss has a slight overfitting problem from epoch \(50\) to the first learning rate scheduler point at epoch \(150\), while \(f_{5}\) with \(\tilde{F}_{i}\) fixed at predecessors early stopping on classification error continues to overfit from the very first epoch until a few epochs after the first learning rate scheduler point while the error remains at the same level.

**3. Mis-calibration is linked with block overfitting.** When we look at the variation of ECE and validation loss, we observe that the overfitting trend of an individual block is consistent with the variation of ECE, which further extends the link between mis-calibration and overfitting at block-wise level.

According to these observations, it could be very difficult to identify the ideal PC due to the inter-dependency among blocks (the linear blocks in the two rows have totally different behaviours). In other words, simply finding the best-fitting blocks individually and combining them together without taking other blocks into account may not produce a best-fitting model. This motivates us to explore an automatic searching algorithm to find a group of predecessors at "sweet points" that allows \(F^{*}=\{f_{i}^{\rho_{i}^{i}}|i=1,2,\ldots,M\}\) to best-fit to the training set. Here, \(\rho_{i}^{i}\) denotes the optimal block predecessor choice of block \(f_{i}\).

## 4 Methodology

Based on the observation on Figure 1, we propose to explore a PC representation \(\mathcal{P}=\{\rho_{i}|i=1,2,\ldots,M\}\) to tackle the overfitting issue in blocks and improve model calibration performance. The objective can be formulated as

\[\min_{\mathcal{P}}\left(ERR(F)+\lambda\cdot ECE(F)\right) \tag{4}\]

where \(F\) is \(\{f_{i}^{\rho_{i}}|i=1,2,\ldots,M\}\), \(\lambda\) denotes a hyperparameter, \(\rho_{i}\in\{1,2,\ldots,T_{train}\}\) indicates the predecessor selection of block \(f_{i}\), \(ERR\) and \(ECE\) are the classification error and ECE respectively. The direct optimization of Eq. (4) is not feasible via gradient descent due to two issues. First, the discrete representation \(\mathcal{P}\) makes Eq. (4) an optimization problem over a discrete domain. Second, both terms \(ERR\) and \(ECE\) in our objective are not differentiable. Thus, we introduce a PCS framework to tackle the aforementioned issues. Specifically, we first introduce differentiable combination sampling through a continuous relaxation of the PC representation. Proxy classification error and ECE landscape are then introduced to achieve differential optimization of our objective through an estimator for predicting the classification error and ECE.

### Differentiable Combination Sampling

In our PCS framework, the objective is to discover the optimal selection of predecessors from candidate sets for each block. To learn the selection, we first exploit a \(K\)-dimensional trainable parameter \(\alpha_{i}\in\mathbb{R}^{K}\) for this predecessor selection, where \(K\) denotes the number of candidates. And \(\rho_{i}\) can be obtained by the argmax of the selection parameter \(\alpha_{i}\) and further represented in a one-hot encoding format \(\rho_{i}\in\mathbb{R}^{K}\). The PC representation \(\mathcal{P}\) can be written as:

\[\mathcal{P}=\{\rho_{i}|i=1,2,\ldots,M\},\] \[\text{s.t.}\quad\rho_{i}=\mathrm{one}\mathrm{-hot}(\mathrm{argmax }\,\alpha_{i}). \tag{5}\]

To relax the discrete PC representation for gradient-based optimization, we use the Gumbel-Softmax trick [1] to approximate the one-hot distribution and introduce randomness. The \(\rho_{i}\) in Eq. (5) can be relaxed as:

\[\tilde{\rho}_{i}^{k}=\frac{\exp((\alpha_{i}^{k}+\xi_{i}^{k})/\tau)}{\sum_{k^{ \prime}=1}^{K}\exp((\alpha_{i}^{k^{\prime}}+\xi_{i}^{k^{\prime}})/\tau)}, \tag{6}\]

where \(\tau\) is the temperature parameter, \(\xi_{i}^{k}\) is an i.i.d sample from Gumbel(0, 1), \(k\) and \(k^{\prime}\) denote the \(k^{\text{th}}\) and \(k^{\prime\text{th}}\) logit of corresponding \(K\)-dimensional vector respectively. We denote the relaxed PC representation as \(\tilde{\mathcal{P}}=\{\tilde{\rho}_{i}|i=1,2,\ldots,M\}\). With the learning of \(\alpha_{i}\), \(\tilde{\mathcal{P}}\) explores the random combination at the beginning of search and gradually converges to a relatively stable state.

For simplicity, we denote the model with \(\tilde{\mathcal{P}}\) as \(F_{\tilde{\mathcal{P}}}\). The classification error and ECE on the validation set can be denoted as \(ERR(\mathcal{D}_{val},F_{\tilde{\mathcal{P}}})\) and \(ECE(\mathcal{D}_{val},F_{\tilde{\mathcal{P}}})\) respectively. Since the combination is hard-combined, we fine-tune \(F_{\tilde{\mathcal{P}}}\) with one more epoch on \(\mathcal{D}_{train}\), denoted as \(F_{\tilde{\mathcal{P}}}^{*}\). In PCS, the original objective (Eq. (4)) can be reformulated as:

\[\min_{A}\left(ERR(\mathcal{D}_{val},F_{\tilde{\mathcal{P}}}^{*})+\lambda\cdot ECE (\mathcal{D}_{val},F_{\tilde{\mathcal{P}}}^{*})\right), \tag{7}\]

where \(A=\{\alpha_{i}|i=1,2,\ldots,M\}\) is the collection of learnable selection parameters for all blocks.

### Proxy Classification Error and ECE Landscape

For differential optimization of PC representation \(\tilde{\mathcal{P}}\), we use a trainable estimator \(\psi\) to obtain proxies \(E\hat{R}R\), \(E\hat{C}E=\psi(\tilde{\mathcal{P}})\) to approximate the classification error and ECE. Since the input \(\tilde{\mathcal{P}}\) is a sequential data, we utilize a one-layer long short-term memory (LSTM) to build the estimator \(\psi\): \(\mathbb{R}^{M\times K}\rightarrow\mathbb{R}^{d}\) mapping \(\tilde{\mathcal{P}}\) to a \(d\)-dimensional embedding vector and a linear layer: \(\mathbb{R}^{d}\rightarrow\mathbb{R}^{2}\) outputting \(E\hat{R}R\) and \(E\hat{C}E\). The estimator \(\psi\) is trained with a weighted mean squared error loss function:

\[\min_{\psi}L(\psi)= \frac{1}{T_{se}}\sum_{t=1}^{T_{se}}(E\hat{R}R^{(t)}-ERR^{(t)}( \mathcal{D}_{val},F_{\tilde{\mathcal{P}}}^{*}))^{2}\] \[+ \gamma(E\hat{C}E^{(t)}-ECE^{(t)}(\mathcal{D}_{val},F_{\tilde{ \mathcal{P}}}^{*}))^{2}, \tag{8}\]

where \(\gamma\) is a hyperparameter to control the loss ratio of ECE, \(T_{se}\) is the total searching steps and the superscript \((t)\) indicates the evaluation results at the \(t^{\text{th}}\) time step. All pairs
of \(\tilde{\mathcal{P}}\) and its corresponding classification error and ECE are stored in a memory \(\Pi\) to optimize estimator \(\psi\). After each searching step \(t\), memory \(\Pi\) is updated by \(\Pi=\Pi\cup\{(\tilde{\mathcal{P}}^{(t)}:(ECE^{(t)},ERR^{(t)}))\}\). We can then use the optimized estimator \(\psi^{*}\) to reformulate PCS objective Eq. (7):

\[\begin{split}&\min_{A}(E\hat{R}R^{{}^{*}}+\lambda E\hat{C}E^{{}^{*} }),\\ &\text{where }E\hat{R}R^{{}^{*}},E\hat{C}E^{{}^{*}}=\psi^{*}( \tilde{\mathcal{P}}).\end{split} \tag{9}\]

The gradients of \(E\hat{R}R^{{}^{*}}\) and \(E\hat{C}E^{{}^{*}}\) can be used to optimize \(\tilde{\mathcal{P}}\) and thus \(A\):

\[A^{\prime}\gets A-\eta\cdot\nabla_{A}(E\hat{R}R^{{}^{*}}+\lambda E\hat{C}E ^{{}^{*}}), \tag{10}\]

where \(A^{\prime}\) is the new predecessor selection parameter and \(\eta\) is the learning rate. At the next searching time step, the corresponding \(\tilde{\mathcal{P}}^{\prime}\) is based on \(A^{\prime}\), and memory \(\Pi\) is updated to \(\Pi=\Pi\cup\{(\tilde{\mathcal{P}}^{\prime}:(ECE^{{}^{\prime}},ERR^{{}^{\prime} })\}\).

**Remark: Search Procedure** We first train model \(F_{\Theta}\) with \(T_{train}\) epochs and randomly store weight parameters at \(K\) different epochs. After that, we randomly initialize a warm-up population \(\mathbb{H}\) of size \(\mathcal{S}\) and evaluate the classification error \(ERR^{(t)}(\mathcal{D}_{val},F^{*}_{\tilde{\mathcal{P}}_{i}})\) and ECE \(ECE^{(t)}(\mathcal{D}_{val},F^{*}_{\tilde{\mathcal{P}}_{i}})\) of \(\tilde{\mathcal{P}}_{i}\in\mathbb{H}\). The combination-performance pairs are then stored to memory \(\Pi\), which is used to warm up estimator \(\psi\) and equip \(\psi\) with prior knowledge about classification error and ECE before searching. After the initial training of model and warming up of \(\psi\), the searching procedure is conducted with \(T_{se}\) steps. In each step, one PC representation \(\tilde{\mathcal{P}}^{(t)}\) is sampled based on current \(A^{(t)}\). The corresponding \(F_{\tilde{\mathcal{P}}^{(t)}}\) is then fine-tuned with one epoch and evaluated to obtain a validation error and ECE for the training of \(\psi\). \(A^{(t)}\) is then optimized with the proxy classification error \(E\hat{R}R\) and proxy ECE \(E\hat{C}E\). The detailed algorithm is provided in supplementary.

**Remark: Search Space Sampling Strategy** Each selection \(\rho\) is an integer between \(1\) and training epoch \(T_{train}\), and thus the candidate size \(K\) is \(T_{train}\). However, storing weight parameters over all training epochs can be very storage-intensive. For instance, storing all candidates of a ResNet-50 training with 350 epochs can take up to 33GB. Considering that the model parameters of adjacent epochs are similar, especially at late training stages, we propose four sampling strategies to reduce the size of search space and improve storage efficiency, which are (1) _Random Sampling_, randomly sampling \(K\) different epochs ranging from \(1\) to \(T_{train}\); (2) _Uniform Sampling_, uniformly sampling \(K\) epochs ranging from \(1\) to \(T_{train}\); (3) _Laplace Sampling_, due to model parameters changing much faster at earlier epochs, sampling \(K\) epochs ranging from \(1\) to \(T_{train}\) with a Laplace distribution centering at 0; (4) _Piece-Wise Laplace Sampling_, due to model parameters changing much faster at earlier epochs of each learning schedule, sampling \(K\) epochs ranging from \(1\) to \(T_{train}\) with multiple Laplace distributions centering at 0 and other learning schedule epochs (150 and 250 in our case). Since the searching difficulty grows exponentially with the size of \(K\), another benefit of small candidate size is that it improves the searching efficiency.

## 5 Experiments

### Experimental Settings

**Datasets** We conduct experiments on various datasets, including CIFAR-10/100 [11] and TinyImageNet [14] to evaluate the calibration performance. We also include the robustness evaluation on Out-of-Distribution (OoD) datasets, including SVHN [15] and CIFAR-10-C [1]. The details about datasets can be found in supplementary.

**Baselines** To verify the effectiveness of our proposed algorithm, we include different networks for evaluation, including ResNet-50, ResNet-110 [1], Wide-ResNet-26-10 [23] and DenseNet-121 [10], and compare with various approaches, including training with weight decay at \(5\times 10^{-4}\) (we find that weight decay at \(5\times 10^{-4}\) performs the best among multiple values), Brier Loss [1], MMCE loss [21], Label smoothing [23] with a smoothing factor \(\alpha_{LS}=0.05\), focal loss [11] with regularisation parameter \(\gamma_{focal}=3\), and scheduled focal loss FLSD-53 [12] which uses \(\gamma_{focal}=5\) for \(\hat{p}\in[0,0.2)\) and \(\gamma_{focal}=3\) for \(\hat{p}\in[0.2,1)\).

**Other Calibration Metrics** Recent works [11, 10, 12] point out the defects of ECE. To evaluate our method comprehensively, we evaluate our method on three additional calibration metrics, i.e., MCE, Adaptive-ECE [14] and classwise-ECE [13] along with ECE. We also measure PCS with reliability plots. The details about additional calibration metrics and additional experimental results can be found in supplementary.

\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c} \hline \multirow{2}{*}{**Dataset**} & \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Weight Decay**} & \multicolumn{2}{c}{**Brier Loss**} & \multicolumn{2}{c}{**MMCE**} & \multicolumn{2}{c}{**Label Smoothing**} & \multicolumn{2}{c}{**FL-3**} & \multicolumn{2}{c}{**FLSD-53**} & \multicolumn{2}{c}{**PICS-5**} \\  & & & Guo _et al._[11] & Brier _et al._[14] & Kumar _et al._[14] & Szegedy _et al._[14] & Mathieu _et al._[13] & Mathieu _et al._[13] & Ours \\  & & Pre T & Post T & Pre T & Post T & Pre T & Post T & Pre T & Post T & Pre T & Post T & Pre T & Post T & Pre T \\ \hline \multirow{4}{*}{CIFAR-100} & ResNet-50 & 17.52 & 3.42(2).1 & 6.52 & 3.64(1.1) & 15.32 & 2.38(1.8) & 7.81 & 4.09(1.1) & 5.13 & 1.97(1.1) & 4.5 & 2.0(1.1) & **2.0** & **2.0(1.0)** \\  & ResNet-110 & 19.05 & 4.43(2.3) & 7.88 & 4.65(1.1) & 19.44 & 3.86(2.3) & 11.60 & 5.89(1.1) & 8.64 & 3.95(1.1) & 2.56 & 4.1(2.1) & **1.76** & **1.6(1.0)** \\  & Wide-ResNet-260 & 15.35 & 28.28(2.8) & 4.31 & 2.7(1.1) & 13.77 & 4.84 & 4.84(1.0) & 1.23 & 1.25(1.0) & 3.03 & 1.64(1.1) & **1.92** & **1.5(1.5)** \\  & Dense-121 & 20.98 & 4.27(2.3) & 5.17 & 2.28(1.1) & 19.13 & 3.03(2.1) & 12.89 & 7.52(1.2) & 4.15 & 1.25(1.1) & 3.73 & 1.31(1.1) & **2.75** & **1.18(1.0)** \\ \hline \multirow{4}{*}{CIFAR-10} & ResNet-50 & 4.35 & 1.35(2.5) & 1.82 & 1.08(1.1) & 4.56 & 1.19(2.6) & 2.96 & 1.67(0.9) & 1.48 & 1.42(1.1) & 1.55 & 0.95(1.1) & **0.80** & **0.53(1.0)** \\  & ResNet-110 & 4.100 & 10.0(2.8) & 2.56 & 1.28(1.2) & 5.08 & 1.42(2.8) & 2.09 & 2.09 & 1.00(1.0) & 1.55 & 1.02(1.1) & 1.87 & 1.09(1.1) & **0.57** & **0.57(1.0)** \\ \cline{1-1} \cline{2-11}  & Wide-ResNet-26-10 & 3.23 & 6.92(2.2) & 1.25 & 1.25(1.0) & 3.29 & 8.06(2.4) & 2.46 & 1.84(0.9) & 1.69 & 0.97(0.9) & 1.56 & 0.84(0.9) & **0.99** & **0.4(1.2)** \\ \cline{1-1} \cline{2-11}  & DenseNet & ResNet-50 & 12.11 & 4.52 & 1.31(2.4) & 1.53 (1.0) & 5.13(1.0) & 5.16(2.5) & 1.88 & 1.52(0.9) & 1.32 & 1.25(0.9) & 1.22 & 1.2(1.0) & **0.78** & **0.79(1.0)** \\ \hline Tiny-ImageNet & ResNet-50 & 15.52 & 5.48(1.4) & 4.44 &
Training SetupFor training on CIFAR-10/100, we set \(T_{train}=350\). The learning rate is set to \(0.1\) for epoch \(0\) to \(150\), \(0.01\) for \(150\) to \(250\), and \(0.001\) for \(250\) until the end of training. For training on Tiny-ImageNet, we set \(T_{train}=100\). We follow the same training and validation set split setting as Mukhoti _et al._[2020]. The learning rate is set to \(0.1\) for epoch \(0\) to \(40\), \(0.01\) for epoch \(40\) to \(60\), and \(0.001\) for \(60\) until the end of training. The fine-tuning learning rate is set to \(10^{-4}\) for CIFAR-10, \(5\times 10^{-4}\) for CIFAR-100, and \(10^{-3}\) for Tiny-ImageNet. The searching process is performed with \(T_{se}=100\) steps. The population size is \(\mathcal{S}=100\). Experiments are conducted with ResNet-50 on CIFAR-10 if there is no other specification. All networks are optimized using the SGD optimizer with a weight decay at \(5\times 10^{-4}\) and a momentum of 0.9. The training batch size is set to 128. All experiments are conducted on a single Tesla V-100 GPU with all random seeds set to 1. Our code and results of comparison method are based on the public code and the pre-trained weight provided by Mukhoti _et al._[2020]. Other hyperparameters and settings can be found in supplementary.

**Temperature Scaling** Following the setting in the prior work [Mukhoti _et al._, 2020], the temperature parameter \(\tau\) is optimized by grid searching with \(\tau\in[0,0.1,0.2,\dots,10]\) on the validation set and finding the one with the best post-temperature-scaling ECE, which is also applied on the additional calibration metrics.

### Calibration Performance

We report ECE\((\%)\) (computed using 15 bins) along with optimal temperatures in Table 1. PCS achieves the state-of-the-art ECE across all models and datasets and outperforms previous works by large margins, especially pre-temperature-scaling results. More specifically, most PCS pre-temperature-scaling results have already substantially exceeded the post-temperature-scaling results of previous works. The result of ResNet-110 on CIFAR-100 achieves the best calibration performance compared to previous works, with a \(7\%\) decrease in ECE. For comparison approaches, the model trained with focal loss is broadly better-calibrated than other methods. However, it fails in some cases such as the evaluation on Tiny-ImageNet. In addition, the scheduled \(\gamma_{focal}\) trick does not always work better than fixed \(\gamma_{focal}\) and it is hard to ascertain which is the better between FL-3 and FL53. The MMCE auxiliary loss performs worst before temperature scaling. Another notable point of PCS is that multiple results such as those with ResNet-110 on CIFAR-10/100 achieve the innately calibrated model (T=1.0), which means that the PC itself has already yielded a well-calibrated model without temperature scaling.

We also evaluate PCS on other widely-accepted metrics including Adaptive ECE, Classwise-ECE and test set error. PCS also achieves the state-of-the-art calibration results on almost all cases. The test set error on Tiny-ImageNet shows a \(8.52\%\) decrease from \(49.81\%\) to \(41.29\%\), which is mainly because of PCS performing as an early stopping trick. More results are reported in the supplementary.

### Robustness on Out-of-Distribution(OoD) Datasets

A well-calibrated model helps improve the model robustness on OoD datasets [Thulasidasan _et al._, 2019]. However, temperature scaling is known to be fragile under dataset distribution shift [Ovadia _et al._, 2019]. PCS form innately calibrated models and thus perform well on OoD datasets. We utilize AUROC (the higher the better) to evaluate the robustness under dataset shift. Table 2 shows the AUROC (\(\%\)) computed for models trained on CIFAR-10 and tested on the OoD datasets SVHN and CIFAR-10-C. Our method achieves competitive results on almost all cases. The results after temperature scaling tend to drop generally, and approaches yielding better pre-temperature-scaling ECE have better robustness on OoD datasets. Although focal loss works well on calibration, it fails under dataset shift. Our PCS with ResNet-50 on CIFAR-10 achieves a \(4\%\) increase compared to previous methods.

### Searching Results

We visualize the searching results of ResNet-50 on CIFAR-10 in Figure 2. The searching results are obtained based on well-fitting (test set loss \(<\) 2) PCs. The result shows that the last convolutional block (Conv Block 4) tends to choose predecessors in the first half of training (epoch \(50\) to \(180\)) while Conv Block 2 and Conv Block 3 prefer predecessors in the second half (epoch \(150\) to \(350\)). This observation might indicate that the later Conv Blocks tend to overfit earlier than the former ones. The first Conv Block and the final linear block show no particular preference to certain predecessors.

This result is consistent with the evaluation of overfitting of individual blocks in Figure 1. The Conv Block 1 suffers from little overfitting throughout training and thus has no preference to certain predecessors, while the middle blocks (Conv Block 2, 3, 4) prefer predecessors with a low validation loss as shown in Figure 1. We also visualize the searching results of other models and observe the similar pattern. The visualization is reported in supplementary.

\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Weight Deces**} & \multirow{2}{*}{**Brier Loss**} & \multirow{2}{*}{**MMCE**} & \multirow{2}{*}{**Label Smoothing**} & \multirow{2}{*}{**FL-3**} & \multirow{2}{*}{**FIND-53**} & \multirow{2}{*}{**PCS**} \\  & & & Gros _et al._[2017] & Blavier _et al._[1950] & Kumar _et al._[2018] & Szegedy _et al._[2016] & Madhotri _et al._[2020] & Matholou _et al._[2020] & & Ours \\  & & Pre T & Post T & Pre T & Post T & Pre T & Post T & Pre T & Post T & Pre T & Post T & Pre T & Post T \\ \hline \multirow{4}{*}{CIFAR-10SVHN} & ResNe-50 & 94.92 & 94.56 & 94.59 & 94.27 & 81.77 & 64.75 & 78.88 & 78.39 & 88.28 & 88.42 & 92.79 & **81.12** & **97.04** \\  & ResNet-110 & 61.71 & 59.96 & 94.03 & 95.13 & 85.31 & 85.39 & 85.39 & 86.68 & 86.74 & **86.92** & 90.83 & 90.97 & **96.77** & 96.77 \\  & Wide-ResNet-26-10 & 96.82 & 97.62 & 94.51 & 94.51 & 97.35 & 97.95 & 84.63 & 84.66 & 98.19 & 80.58 & **98.29** & **98.20** & 97.55 & 97.84 \\  & DenseNet-121 & 84.48 & 81.57 & 94.65 & 84.66 & 83.58 & 84.87 & 78.79 & 89.84 & 89.42 & 89.59 & 89.59 & **70.27** & **97.2** \\ \hline \multirow{4}{*}{CIFAR-10CEAR-10-C} & ResNe-50 & 86.23 & 86.03 & **90.21** & **90.33** & 89.97 & 90.11 & 72.02 & 90.44 & 89.56 & 89.45 & 89.56 & 97.33 & 89.79 \\  & ResNet-110 & 77.53 & 75.16 & 84.09 & 83.86 & 71.96 & 70.02 & 72.17 & 72.18 & 82.27 & 81.85 & 85.05 & 84.70 & **88.1** & **88.27** \\ \cline{1-1}  & Wide-ResNet-26-10 & 81.06 & 80.68 & 85.03 & 85.05 & 82.17 & 81.72 & 71.10 & 71.16 & 82.17 & 81.86 & 87.05 & 87.30 & **89.62** & **89.95** \\ \cline{1-1}  & DenseNet-121 & 87.61 & 86.41 & 87.38 & 87.38 & 84.9 & 84.88 & 73.67 & 73.8 & 87.12 & 87.53 & 89.47 & **89.47** & **89.52** & **89.52** \\ \hline \hline \end{tabular}
\end{table}
Table 2: **Robustness on Dataset Shift.** AUROC \((\%)\), being the higher the better, is evaluated for different methods with models shifting from CIFAR-10 (in-distribution) to SVHN and CIFAR-10-C as the OoD datasets.
### Ablation Study

Comparison with Other Searching MethodsIn Table 3, our method is compared with different searching methods as well as early stopping methods. When early stopping on the model as a whole, it is hard to ensure a low error and good calibration performance at the same time. Early stopping on loss and ECE shows a large performance drop on classification error. The random search is conducted 5 times to make the results stable and achieve a relatively high performance on classification error but not ECE. We also compare multiple objectives in Eq. (9) with the performance of a single objective optimization on validation NLL loss, i.e.,\(\min_{A}N\hat{L}L\). Searching on ECE or error individually could lead to extremely unbalanced results. PCS achieves a better result on both test set error and ECE. The hyperparameter \(\lambda\) depends on models and datasets, and we use \(\lambda=25\) for ResNet-50 on CIFAR-10. We tune \(\lambda\) with multiple values and find out the suitable \(\lambda\) for each setting. The settings for each model are given in the supplementary.

Weight Sampling StrategyTo compare sampling strategies discussed in the previous section, a scatter plot in Figure 3 shows the searching results. Note that the "Full Sampling" indicates searching on all possible predecessors without sampling, i.e., \(K=T_{train}\). We use the same \(K=50\) for all sampling strategies. From Figure 3, we observe little difference between different sampling strategies, which indicates that we save storage space with little loss of performance. Thus, we use the random sampling strategy throughout the paper due to its simplicity.

Warm-up PopulationA larger population indicates more searching time. To find a balance between searching time and performance, we compare the searching results with different population sizes. We use the number of well-fitting results (test loss under 0.2) to measure the searching performance. All experimental results are averaged over 5 runs. According to Table 4, the larger the population size, the more well-fitting results can be found since estimator \(\psi\) can have better prior knowledge on error and ECE landscape. However, we use a smaller population size as long as the searching provides satisfactory results. Throughout the paper, we use the population size \(\mathcal{S}=100\).

## 6 Conclusion

In this paper, we address a common problem, the miscalibration in modern neural networks. We observe that different blocks in a network have different overfitting patterns. Our proposed predecessor combination search, as a regularization method, is very effective for calibrating models and can also be potentially applied to other tasks such as learning with noisy labels and improving model robustness.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Population Size** & **64** & **100** & **200** & **500** \\ \hline
**GPU Hours** & 3.1 & 3.7 & 5.5 & 10.5 \\
**Well-fitting Results** & 14.8 & 16 & 17.8 & 25.8 \\ \hline \hline \end{tabular}
\end{table}
Table 4: **Warm-up Population**. Well-fitting results indicate the number of searching results that achieve a testing loss under 0.2, being the higher the better. All results are reported as an average of 5 runs with ResNet-50 on CIFAR-10.

Figure 3: **Comparison of Sampling Strategy**. All experiments are conduct 5 runs with ResNet-50 on CIFAR-10. Searching step \(T_{se}\) is set to \(100\) and produces 500 searching results for each sampling strategy. Metrics along the x-axis and y-axis are the lower the better.

\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Methods** & **ERR** & **ECE** & **ECE(T)** \\ \hline
**Early Stopping (Loss)** & 6.58 & 2.32 & 0.56(1.4) \\
**Early Stopping (Error)** & 4.89 & 4.03 & 1.42(2.4) \\
**Early Stopping (ECE)** & 16.92 & 1.71 & 1.09(1.2) \\
**Random Search** & 5.04 & 1.13 & 0.84(1.1) \\
**Search on Loss** & 5.43 & 1.1 & 0.37(1.10) \\
**PCS (\(\lambda\)=10)** & 5.31 & 0.86 & 0.73(1.10) \\
**PCS (\(\lambda\)=25)** & 5.1 & 0.75 & 0.37(1.10) \\
**PCS (\(\lambda\)=50)** & 5.25 & 0.58 & 0.58(1.0) \\
**PCS (\(\lambda\)=100)** & 5.29 & 0.70 & 0.44(1.10) \\ \hline \hline \end{tabular}
\end{table}
Table 3: **Comparison of Searching Methods**. Random search is conducted 5 times with ResNet-50 on CIFAR-10. All searching results are selected by the lowest testing loss.

Figure 2: **Searching Results**. The sub-figures show the predecessor choice frequency of different blocks in ResNet-50. The search results are filtered by cross-entropy loss (being less than 0.2) on the test set.