Calibration of Neural Networks
Neural networks solving real-world problems are often required not only to
make accurate predictions but also to provide a confidence level in the
forecast. The calibration of a model indicates how close the estimated
confidence is to the true probability. This paper presents a survey of
confidence calibration problems in the context of neural networks and provides
an empirical comparison of calibration methods. We analyze problem statement,
calibration definitions, and different approaches to evaluation: visualizations
and scalar measures that estimate whether the model is well-calibrated. We
review modern calibration techniques: based on post-processing or requiring
changes in training. Empirical experiments cover various datasets and models,
comparing calibration methods according to different criteria.
Ruslan Vasilev, Alexander D'yakonov
# Calibration of Neural Networks

###### Abstract

Neural networks solving real-world problems are often required not only to make accurate predictions but also to provide a confidence level in the forecast. The calibration of a model indicates how close the estimated confidence is to the true probability. This paper presents a survey of confidence calibration problems in the context of neural networks and provides an empirical comparison of calibration methods. We analyze problem statement, calibration definitions, and different approaches to evaluation: visualizations and scalar measures that estimate whether the model is well-calibrated. We review modern calibration techniques: based on post-processing or requiring changes in training. Empirical experiments cover various datasets and models, comparing calibration methods according to different criteria.
###### Contents

* 1 Introduction
* 2 Problem Statement
* 3 Calibration Evaluation
	* 3.1 Visualization
	* 3.2 Calibration Metrics
* 4 Calibration Methods
	* 4.1 Post-processing
		* 4.1.1 Histogram Binning
		* 4.1.2 Isotonic Regression
		* 4.1.3 Generalizations of Platt Calibration
	* 4.2 Calibration During Training
		* 4.2.1 Label Smoothing
		* 4.2.2 Focal Loss
* 5 Empirical Experiments
	* 5.1 Experimental Design
	* 5.2 Experiment Results
* 6 Conclusion
* Appendices
* A Classification Accuracy
* B Binning-based Metrics
* C Proper Scoring Rules
Introduction

Deep learning is finding more and more applications in various fields. Neural networks are actively used in clinical practice [1], self-driving cars [2], machine translation [3], and other diverse applications.

Oftentimes, real-world problems require models that produce not only correct prediction but also a reliable measure of confidence in it. _Confidence_ refers to probability estimate that the forecast is correct. For example, if an algorithm predicts a given sample of patients are healthy with confidence \(0.9\), we expect that \(90\%\) of them are really healthy. A model with reliable confidence estimation is called _calibrated_. Along with the interpretation of neural network predictions, confidence calibration is important when probability estimates are fed into subsequent algorithm steps (for example, language models decoding strategies [4]).

Modern neural networks often turn out to be poorly calibrated [5]. However, many other machine learning algorithms also produce biased confidence estimates [6, 7]. Various calibration techniques have been proposed for "classical" machine learning, some of which have been developed in deep learning.

## 2 Problem Statement

Consider a classification problem for objects from set \(\mathcal{X}\) with classes \(\mathcal{Y}=\{1,\ldots,K\}\). Suppose we have trained a _model_ -- an algorithm which produces vector of scores _(confidences)_\(\mathbf{a}(x)=(a_{1}(x),\ldots,a_{K}(x))\), \(\sum_{j=1}^{K}a_{j}(x)=1\), for each \(x\in\mathcal{X}\). Next, the class corresponding to the highest confidence is assigned to the object:

\[\hat{y}(x)\coloneqq\operatorname*{argmax}_{j\in\mathcal{Y}}a_{j},\quad\hat{p} (x)\coloneqq a_{\hat{y}}.\]

We would like to interpret estimator \(\hat{p}\) as the probability that the true label coincides with the predicted one. If the estimate is accurate enough, then the model is called _calibrated_. Formally, the definition of calibration (_perfect calibration_ in [5]) can be written as follows:

\[\mathbb{P}\left(y=\hat{y}\mid\hat{p}=p\right)=p\quad\forall p\in[0,1]\;. \tag{1}\]

There are stronger definitions of model calibration1 than (1). For example, according to [8], the classifier is called calibrated (originally, _well-calibrated_) if

Footnote 1: It should be noted that the term _calibration_ also often refers to _methods_ which make model confidences accurate.

\[\mathbb{P}(y=j\mid a_{j}=p)=p\quad\forall j\in\mathcal{Y},\quad\forall p\in[0,1] \tag{2}\]

that means the assurances given for each class (not just the predicted one) are calibrated.

In the case of real-world data and models, we cannot directly check (1) and (2), so various calibration metrics come in handy as well as visualizations that are reviewed in section 3.
Section 4 describes calibration methods -- techniques that make confidences more reliable. First, one can calibrate confidences afterwards, i.e. find a transformation that maps biased estimates to calibrated. There are different algorithms that find optimal transformations. Second, it is possible to apply special techniques during training, for example, loss function modifications.

Section 5 provides empirical comparison of calibration methods for modern neural network architectures and shows how the choice of loss function affects the calibration.

## 3 Calibration Evaluation

### Visualization

Before defining calibration measures, we simplify the problem to binary classification: \(\mathcal{Y}=\{0,1\}\) -- consider the model generates confidences \(\hat{p}\) that the object belongs to the positive class (\(y=1\)). Binary classification is common in various applications and "classical" machine learning models: logistic regression, support vector machines, gradient boosted trees and others -- the problem of their calibration were studied in [6, 7].

Consider a set of objects (_usually, another validation set_), for each of which the true class is known and the confidence score is produced by the model. Divide the segment of all possible confidence values \([0,1]\) into \(M\) equal-width intervals \(I_{m}\):

\[I_{1}=\left[0,\frac{1}{M}\right),\ I_{2}=\left[\frac{1}{M},\frac{2}{M}\right), \ \ldots,\ I_{M-1}=\left[\frac{M-2}{M},\frac{M-1}{M}\right),\ I_{M}=\left[\frac{M- 1}{M},1\right]\,. \tag{3}\]

Thus, each confidence estimate falls into one of these intervals -- let \(B_{m}\) denote the set of indices of those objects which confidences are in \(I_{m}\). Both \(B_{m}\) and \(I_{m}\) are called _bins_.

For each bin \(B_{m}\), we calculate positive frequency \(A_{m}^{1}\) and average confidence \(C_{m}^{1}\):

Figure 1: Options for visualizing confidence calibration. For clarity, synthetic data was generated. Support vector machine is used as a model (the distances to the separating hyperplane are scaled into \([0,1]\)).
\[A_{m}^{1}=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}1(y_{i}=1),\quad C_{m}^{1}=\frac{1}{|B _{m}|}\sum_{i\in B_{m}}\hat{p}_{i}. \tag{4}\]

Finally, we can draw a plot \((C_{m}^{1},A_{m}^{1})_{m=1}^{M}\) which is called _reliability plot_ Figure 1 (a). Also, the resulting curve is sometimes called calibration curve. The model is considered well-calibrated if its calibration curve is close to the diagonal (_in subsection 3.2 we describe scalar metrics of such closeness_).

Likewise, \((C_{m}^{1},A_{m}^{1})_{m=1}^{M}\) can be depicted using a histogram, which is called _reliability diagram_. In Figure 1 (b) the average confidence is shown in red, and the positive frequency is shown in blue. If the red bar is higher than the blue one, then the algorithm underestimates confidences -- _underconfidence_. The opposite case is called _overconfidence_. For better interpretation, the fraction of objects (bin _weight_) that fell into the bin can also be shown in the graph.

When the number of classes \(n>2\), reliability diagrams are built differently. The most popular approach corresponds to the definition (1). For each bin \(B_{m}\), we estimate "accuracy" \(A_{m}\) and average confidence \(C_{m}\):

\[A_{m}=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}1(y_{i}=\hat{y}_{i}),\quad C_{m}=\frac {1}{|B_{m}|}\sum_{i\in B_{m}}\hat{p}_{i}. \tag{5}\]

The difference between (4) and (5) is that in multiclass case \(\hat{y}_{i}\) and \(\hat{p}_{i}\) correspond to the estimated class and its confidence, while in binary case statistics are calculated only for a positive class.

Note that \(A_{m}\) and \(C_{m}\) estimate the left and the right of (1), respectively. They can be depicted on reliability diagram. For two classes, this approach is illustrated in Figure 1 (c). Bins to the left of 0.5 turn out to be empty because a binary classification algorithm assigns an object to a class that has confidence \(>0.5\).

One can also consider classwise reliability diagrams [9]. To make it, each class should be separately assigned to the positive, while all the others should be treated like the negative, so \(n\) reliability diagrams for the binary case can be built. Although the classwise approach is more informative (2), when the number of classes is large (for example, 1'000 or 22'000 in ImageNet [10]), it is usually impractical due to the interpretation difficulty. Therefore, reliability diagrams considering only the prediction confidence (5) are more common.

### Calibration Metrics

In addition to visualizations, various metrics2 can be used to evaluate the calibration of the model. One of the most common is Expected Calibration Error (ECE) [11]. It estimates the expectation of the absolute difference between confidence and associated accuracy:

Footnote 2: Here, _metric_ refers to measure for the evaluation of algorithms

\[\mathbb{E}_{\hat{p}}\left|\mathbb{P}\left(y=\hat{y}\mid\hat{p}\right)-\hat{p} \right|. \tag{6}\]
(6) can be approximated using the partition of the confidences into bins (\(l\) is the total number of objects in the considered set):

ECE \[=\sum_{m=1}^{M}\frac{|B_{m}|}{n}\left|A_{m}-C_{m}\right|\] (7) \[=\sum_{m=1}^{M}\frac{|B_{m}|}{n}\left|\frac{1}{|B_{m}|}\sum_{i\in B _{m}}\mathbb{1}(\gamma_{i}=\hat{y}_{i})-\frac{1}{|B_{m}|}\sum_{i\in B_{m}}\hat {p}_{i}\right|\] \[=\frac{1}{n}\sum_{m=1}^{M}\left|\sum_{i\in B_{m}}\mathbb{1}(y_{i} =\hat{y}_{i})-\sum_{i\in B_{m}}\hat{p}_{i}\right|\,.\]

Comparing (7) and reliability diagrams for a multiclass problem, we notice that ECE is exactly equal to the weighted average of the gaps between red and blue bars Figure 1.

There are other metrics based on a partition of confidences into bins, although used less often. For example, one can calculate the maximum gap between confidence and accuracy [11]:

\[\text{MCE}=\max_{m}|A_{m}-C_{m}|\,, \tag{8}\]

or take into account the confidence not only for the predicted class but also for all the others (classwise ECE) [9]:

\[\text{cwECE}=\frac{1}{K}\sum_{j=1}^{K}\sum_{m=1}^{M}\frac{|B_{m}^{j}|}{n}|A_{ m}^{j}-C_{m}^{j}|, \tag{9}\]

where \(B_{m}^{j}\), \(A_{m}^{j}\), \(C_{m}^{j}\) are, respectively, \(m\)-th bin, accuracy and confidence, if we consider \(j\)-th class positive, and collect all the others into negative. This metric corresponds to classwise reliability diagrams.

Instead of equal-width bins (3), bins with an equal number of samples can be used -- sometimes confidence diagrams are built in this way. In [12], it was proposed to use equal-frequency bins to count the metrics described above. Further, an equal-width scheme will be considered. Also, in addition to \(l_{1}\)-norm (i.e. averaging modules), we can use \(l_{2}\)[13].

The problem with binning metrics is the dependence on the number of bins. An alternative approach is to use proper scoring rules. We consider Negative Log-Likelihood (NLL):

\[\text{NLL}=-\frac{1}{l}\sum_{i=1}^{n}\log a_{i,y_{i}} \tag{10}\]

where \(y_{i}\) is the true class label, \(a_{i,y_{i}}\) is a confidence of the algorithm in it, \(n\) is a total number of objects.

Another proper scoring rule, which can be used to evaluate model calibration, is Brier Score:

\[\text{BS}=\frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{K}\left(a_{ij}-\mathbb{1}(y_{i} =j)\right)^{2}, \tag{11}\]

where \(K\) is a number of classes.
Calibration Methods

There are two main types of calibration techniques. First, model outputs can be post-processed. Special transformation, _calibration map_, maps biased probability estimates to the calibrated ones. Second, calibration can be incorporated into the model training itself.

### Post-processing

The transformation is usually found on a hold-out set _(calibration set)_\((x_{i},y_{i})_{i=1}^{n}\). It can be the same dataset used for hyperparameter tuning, but not the training set because model outputs distribution on training data is not the same as on unseen data.

#### 4.1.1 Histogram Binning

The method was originally introduced in [14] to calibrate decision trees and naive Bayes classifiers. A calibration map in histogram binning is piecewise constant. Consider the binary case: the set of output confidence values is divided into bins \(B_{1},\ldots,B_{M}\) (_usually equal-width_ (3) _or equal-frequency_), and scores that fall into \(B_{m}\) are replaced with the common \(\theta_{m}\). To find \(\theta_{1},\ldots,\theta_{M}\), the following optimization problem is solved:

\[\sum_{m=1}^{M}\sum_{i\in B_{m}}(\theta_{m}-y_{i})^{2}\ \rightarrow\ \min_{\theta_{1},\ldots,\theta_{M}}. \tag{12}\]

In such a statement the optimal \(\theta_{m}\) equals the fraction of positive objects that fall into \(B_{m}\). The calibration map is illustrated in Figure 2 (a).

The method is generalized to the multiclass case using the strategy _one-vs-rest_: each class is separately declared positive and \(K\) piecewise constant functions are constructed. In the inference stage, a calibrated vector is additionally normalized.

Figure 2: Different calibration maps for binary classification (the same data and model as in Figure 1).
#### 4.1.2 Isotonic Regression

The method was proposed in [8]. For the binary case, the map is piecewise constant again, but both the number of the intervals \(M\) and their boundaries are optimized. The constraint is that the map should be non-decreasing. Thus, the following problem is solved:

\[\sum_{m=1}^{M}\sum_{i\in\tilde{B}_{m}}\left(\theta_{m}-y_{i}\right)^{2}\ \rightarrow\ \min_{ \begin{subarray}{c}M,\\ 0=\alpha_{0}\leqslant\alpha_{1}\leqslant\cdots\leqslant\alpha_{M-1}\leqslant \alpha_{M}=1\end{subarray}} \tag{13}\]

where \(\tilde{B}_{1}=\{i:\alpha_{0}\leqslant\hat{p}_{i}<\alpha_{1}\},\ldots,\tilde{ B}_{m}=\{i:\alpha_{m-1}\leqslant\hat{p}_{i}\leqslant\alpha_{m}\}\). The shape of the function is illustrated in Figure 2.

Isotonic regression is generalized to the multiclass case in the same way as histogram binning.

#### 4.1.3 Generalizations of Platt Calibration

Originally, the method was proposed in [15] for calibration of the support vector machines. As can be seen in the illustrations Figure 1, Figure 2, if we rescale the distances \(r(x)\) from the objects to the separating hyperplane into \([0,1]\) and treat them as confidences in positive class, then the reliability plot will have the form of a sigmoid:

\[\hat{p}(x)=\frac{1}{1+e^{-\left(\alpha\cdot r(x)+\beta\right)}}. \tag{14}\]

Scale parameter \(\alpha\) and location parameter \(\beta\) are optimized on a calibration set using maximum likelihood estimation. In this method, the transformation is continuous and allows different generalizations to the multiclass case.

The last linear layer of a neural network for the object \(x\) outputs the logit vector: \(\mathbf{z}=(z_{1},\ldots,z_{K})\). To estimate class probabilities, the vector is transformed with softmax \(\sigma(\cdot)\):

\[\sigma\left(\mathbf{z}\right)=\frac{1}{\sum_{j=1}^{K}\exp\left(z_{j}\right)} \left(\exp\left(z_{1}\right),\ldots,\exp\left(z_{K}\right)\right),\]

so it is possible to generalize Platt Calibration by introducing scale and location parameters for logits:

\[a(x)=\sigma(\mathbf{W}\cdot\mathbf{z}+\mathbf{b}). \tag{15}\]

Parameters \(\mathbf{W}\) and \(\mathbf{b}\) are also optimized with maximum likelihood estimation on a calibration set, which is equivalent to minimizing NLL (10). Depending on \(\mathbf{W}\) and \(\mathbf{b}\) shapes, different generalization may be defined:

1. Temperature scaling: \[\mathbf{W}=\frac{1}{T}\in\mathbb{R},\ T>0,\ \mathbf{b}=\mathbf{0}.\] Generalization of Platt Calibration with a single scalar parameter. The method is one of the most frequently used. An increase in temperature \(T\) leads to an increase in uncertainty -- an increase in the entropy of the output distribution. Decreasing \(T\), on the contrary,
increases confidence in the predicted class. At the same time, the predicted class remains unchanged.
2. Vector scaling: \[\mathbf{W}=\text{diag}(\mathbf{v})\in\mathbb{R}^{K\times K}\text{ --- diagonal matrix},\mathbf{v}\in\mathbb{R}^{K}.\] In this approach, a different scale factor is optimized for each class (and the bias, if \(\mathbf{b}\neq\mathbf{0}\) is optimized too).
3. Matrix scaling: \[\mathbf{W}\in\mathbb{R}^{K\times K},\mathbf{b}\in\mathbb{R}^{K}.\] Matrix scaling is the most general parametrization in this group of methods and is equivalent to logistic regression in the logit space. However, with a large number of classes, the method has too many parameters, which can lead to overfitting, because a calibration set is usually not large.

Note that to implement any of these methods, it is enough to add a linear layer (of the required dimension) to the frozen neural network.

### Calibration During Training

The performance of neural networks strongly depends on the loss function. Usually, NLL (10) is used. Given an object \(x\), it is equal to the cross-entropy between the true classification one-hot distribution \(\mathbf{y}\) and the predicted distribution:

\[\text{CE}(\mathbf{y},\mathbf{a})=-\sum_{j=1}^{K}y_{j}\log a_{j}. \tag{16}\]

To improve the calibration of the model, one can modify the loss function itself.

#### 4.2.1 Label Smoothing

In this method, the degenerate distribution of the target is replaced by a smoothed one. The smoothing degree can be tuned using the parameter \(\alpha\in[0,1]\):

\[\mathbf{y}=(y_{1},\ldots,y_{K})\mapsto\left((1-\alpha)y_{1}+\frac{\alpha}{K},\ldots,(1-\alpha)y_{K}+\frac{\alpha}{K}\right)=\mathbf{y}^{\prime}. \tag{17}\]

As parameter \(\alpha\) increases, the distribution of \(\mathbf{y}^{\prime}\) becomes more uniform. After this transformation, \(\text{CE}(\mathbf{y}^{\prime},\mathbf{a})\) the cross-entropy between the smoothed classification vector and the predicted distribution is minimized.

Although using of smoothed labels to train a classifier is not a new idea, this approach was proposed for calibration in [16].
#### 4.2.2 Focal Loss

The focal loss was originally introduced to tackle the problem of class imbalance [17]. In terms of confidence calibration, the idea was first used in [18]. For an object belonging to the \(j\)-th class, the focal loss has the following definition:

\[\mathrm{FL}=-(1-a_{j})^{\nu}\cdot\log a_{j},\quad\nu\geqslant 0. \tag{18}\]

Note that the loss function becomes the cross-entropy when \(\nu=0\). Increasing parameter \(\nu\), as we see in Figure 3, decreases the penalty for those objects with already high confidence in the true class. While the cross-entropy is the upper bound of the Kullback-Leibler divergence between the true \(\mathbf{y}\) and the predicted \(\mathbf{a}\) distribution, the focal error has the entropy of the predicted distribution \(H(\mathbf{a})\)[18] subtracted from the estimate:

\[\mathrm{CE}(\mathbf{y},\mathbf{a})\geqslant\mathrm{KL}(\mathbf{y}||\mathbf{ a}),\qquad\mathrm{FL}(\mathbf{y},\mathbf{a})\geqslant\mathrm{KL}(\mathbf{y}|| \mathbf{a})-\nu\cdot\mathrm{H}(\mathbf{a}).\]

Thus, optimizing the focal error additionally increases the entropy of the predicted distribution and helps to tackle overconfidence.

## 5 Empirical Experiments

### Experimental Design

The following datasets are used in the experiments:

* **CIFAR-10**[19]: The dataset consists of 60 000 color images \(32\times 32\). _Training / validation / test_ splits are respectively \(50\,000\) / \(5\,000\) / \(5\,000\).
* **CIFAR-100**[19]: \(60\,000\) color images \(32\times 32\), \(100\) classes. _Training / validation / test_: \(50\,000\) / \(5\,000\) / \(5\,000\).

Figure 3: Focal loss on a single object. \(\hat{p}\) — probability estimation for a true class
* **ImageNet 2012**[10]: Large dataset with color images organized into 1000 classes. _Training / validation / test_: 1.2 m / 25 000 / 25 000.
* **Tiny ImageNet**[10]: 110 000 color images \(64\times 64\), organized into 200 classes, subset of the previous dataset. _Training / validation / test_: 100 000 / 5 000 / 5 000.

Pre-trained neural networks with various architectures from open repositories were used for calculations. In experiments, models and datasets are divided into two main groups:

1. The first group includes neural networks trained on CIFAR-10, CIFAR-100, and ImageNet. The weights for the models are obtained respectively from the repositories [20, 21, 22]. Models from this group are used to compare calibration methods based on post-processing.
2. The second group includes pre-trained neural networks from the repository [23]. The datasets used here are CIFAR-10, CIFAR-100, and Tiny ImageNet. These neural networks were trained in [18] -- focal error and label smoothing were used for the considered models.

The models were trained on the _training_ data (_or its subset_), calibrated on the _validation_ set -- all diagrams and metrics correspond to the _test_ set.

Code used in all experiments is published in [24]: temperature, vector, and matrix scaling are implemented using PyTorch, and other methods and evaluation are implemented with SciPY and scikit-learn. Histogram binning use 20 bins; ECE, cwECE, and MCE are calculated on 15 bins partition; reliability diagrams use 10 bins.

### Experiment Results

Complete tables with measurements are given in appendix, additional reliability diagrams for all the models considered in this work can be found in the accompanying repository [24].

Consider the reliability diagrams for ShuffleNetV2 (CIFAR-100, Figure 4): we see a "typical" state of modern neural networks calibration -- overconfidence. Calibration methods help to correct the situation: in this case, temperature scaling works best for all the metrics. However, histogram binning changes probabilities too aggressively when the number of classes is large, as can be seen from the weights change.

For a small number of classes, on the contrary, histogram binning works best in terms of confidence in prediction (Table 5, Table 6 -- for almost all models on CIFAR-10). Note that here neural networks already solve the classification problem with very high accuracy. Almost all probabilities of the predicted class are close to 1, as, for example, on Figure 5. In terms of MCE (Table 9, Table 10) -- metric, that doesn't consider weights of bins -- histogram binning leads to low calibration.

For matrix scaling, we do not provide reliability diagrams: the method overfits too much when the number of classes is large. As a result, matrix scaling significantly degrades the classification quality (Table 3, Table 4) for all datasets, except, again, the low-class CIFAR-10.
One of the most common methods of neural network calibration is temperature scaling. The method doesn't affect classification predictions, while other calibration options almost always reduce accuracy (Table 3, Table 4).

In terms of NLL, temperature and vector scaling are the best as expected, because this loss was minimized during calibration: Table 11, Table 12. As for Brier Score, the best calibration method in many cases was isotonic regression: Table 13, Table 14.

Focal loss and label smoothing result in better calibrated models than the ones optimizing the cross-entropy -- both in terms of calibration in the predicted class (Table 1) and classwise

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline Dataset & Model & CE & F1.1 & H.2 & F1.3 & 15.05 \\ \hline CIFAR-10 & DenseNet121 & 4.53 & 3.47 & 2.02 & 1.68 & **1.65** \\ CIFAR-10 & ResNet110 & 4.73 & 3.70 & 2.78 & **1.61** & 2.20 \\ CIFAR-10 & ResNet50 & 4.26 & 3.88 & 2.55 & 1.58 & 3.07 \\ CIFAR-10 & Wide-ResNet26-10 & 3.25 & 2.66 & **1.57** & 1.98 & 4.33 \\ CIFAR-100 & DenseNet121 & 20.90 & 14.54 & 8.40 & 4.49 & 13.27 \\ CIFAR-100 & ResNet110 & 19.76 & 15.35 & 12.10 & **9.22** & 11.44 \\ CIFAR-100 & ResNet50 & 18.14 & 13.36 & 8.60 & **4.99** & 8.15 \\ CIFAR-100 & Wide-ResNet26-10 & 16.28 & 9.12 & 4.22 & **2.20** & 5.27 \\ TinyImageNet & ResNet50 & 15.98 & 7.87 & 3.32 & **1.93** & 15.73 \\ \hline \hline \end{tabular}
\end{table}
Table 1: ECE, % – Expected Calibration Error (lower is better), 15 bins without post-processing, columns correspond to different loss functions

Figure 4: CIFAR-100, ShuffleNetV2_x0_5
estimates (Table 2). At the same time, such models can be further calibrated with post-processing. In the original paper [18], temperature scaling was used to calibrate models trained with focal loss. Although this approach does show good results with respect to ECE (Table 6), vector scaling works better for classwise cwECE (Table 8). As for models of the first group (Table 7), vector scaling also basically minimizes cwECE. These results are quite expected since vector scaling finds separate coefficients for each class.

Consider also reliability diagrams for EfficientNet (Figure 6). Among all the models used, the underconfidence is most clearly visible in this neural network: most of the predictions fall not into \([0.9,1]\), but into \([0.8,0.9)\). The reason for this behavior may be training setup: the model was trained with label smoothing (\(\alpha=0.1\)) [22]. All calibration methods resulted in a noticeable increase in the confidence of the answers.

## 6 Conclusion

In this work, the main methods of confidence calibration were compared on different neural network architectures, datasets, and critera.

The applicability of a particular method depends significantly on the amount of data and the selected evaluation criterion. Algorithms, in which separate calibration maps are found for each class, work well only if there is a sufficient amount of data in the calibration set (this can usually be provided when the number of classes is small). Strategies based on linear

Figure 5: CIFAR-10, DenseNet121 (CE)
transformation of logits (for example, temperature scaling) show high quality in problems with a large number of classes but are subject to overfitting when parameterization is excessive (matrix scaling).

Confidence calibration is still open to further research in machine learning: as shown in this work, not only calibration methods but also criteria choice can lead to different results.

Figure 6: ImageNet, EfficientNet_b8