Approaching Test Time Augmentation in the Context of Uncertainty
  Calibration for Deep Neural Networks
With the rise of Deep Neural Networks, machine learning systems are nowadays
ubiquitous in a number of real-world applications, which bears the need for
highly reliable models. This requires a thorough look not only at the accuracy
of such systems, but also at their predictive uncertainty. Hence, we propose a
novel technique (with two different variations, named M-ATTA and V-ATTA) based
on test time augmentation, to improve the uncertainty calibration of deep
models for image classification. By leveraging na adaptive weighting system,
M/V-ATTA improves uncertainty calibration without affecting the model's
accuracy. The performance of these techniques is evaluated by considering
diverse metrics related to uncertainty calibration, demonstrating their
robustness. Empirical results, obtained on CIFAR-10, CIFAR-100, Aerial Image
Dataset, as well as in two different scenarios under distribution-shift,
indicate that the proposed methods outperform several state-of-the-art post-hoc
calibration techniques. Furthermore, the methods proposed also show
improvements in terms of predictive entropy on out-of-distribution samples.
Code for M/V-ATTA available at: https://github.com/pedrormconde/MV-ATTA
Pedro Conde, Tiago Barros, Rui L. Lopes, Cristiano Premebida, Urbano J. Nunes
Approaching Test Time Augmentation in the Context of Uncertainty Calibration for Deep Neural Networks

###### Abstract

With the rise of Deep Neural Networks, machine learning systems are nowadays ubiquitous in a number of real-world applications, which bears the need for highly reliable models. This requires a thorough look not only at the accuracy of such systems, but also to their predictive uncertainty. Hence, we propose a novel technique (with two different variations, named _M-ATTA_ and _V-ATTA_) based on test time augmentation, to improve the uncertainty calibration of deep models for image classification. Unlike other test time augmentation approaches, _MV-ATTA_ improves uncertainty calibration without affecting the model's accuracy, by leveraging an adaptive weighting system. We evaluate the performance of the technique with respect to different metrics of uncertainty calibration. Empirical results, obtained on CIFAR-10, CIFAR-100, as well as on the benchmark Aerial Image Dataset, indicate that the proposed approach outperforms state-of-the-art calibration techniques, while maintaining the baseline classification performance. Code for _MV-ATTA_ available at: [https://github.com/pedromcondo/MV-ATTA](https://github.com/pedromcondo/MV-ATTA)

 Uncertainty Calibration, Reliability, Probabilistic Interpretation, Test Time Augmentation, Deep Neural Networks.

## 1 Introduction

Deep Neural Networks (DNNs) changed the paradigm with regards to the applicability of machine learning (ML) systems to real-world scenarios. Consequently, deep learning (DL) models are now present in critical application domains (_e.g._, autonomous driving, medicine, remote sensing, robotics), where bad decision-making can bear potentially drastic consequences. This requires that DNNs are not only highly accurate, but also highly _reliable_ - decision-makers should be able to "trust" the predictions of these models. This lead us to the problem of _uncertainty calibration_ (also referred as confidence calibration or simply _calibration_): it is required that the confidence output generated by the DL model - that translates as the confidence the model has in the prediction that is making - realistically represents the true likelihood of correctness. For the sake of intuition, a calibrated model would for example, in the long run, correctly classify \(70\%\) of those predictions that have a confidence value of \(0.7\) associated. This accurate quantification of predictive uncertainty results in reliable confidence values associated with each prediction, and therefore, in a more reliable model. As such, it is important to understand how well calibrated are modern DNNs. Further details, including the formalization of the uncertainty calibration problem, will be described in Section 3.

Although increasingly accurate, modern DL architectures have been found to be tendentiously _uncalibrated_[20, 7]. Furthermore, "modern neural networks exhibit a strange phenomenon: probabilistic error and miscalibration worsen even as classification error is reduced" [7]. For this reason, the goal of this work is to improve the uncertainty calibration of DNNs in the task of image classification, by proposing a novel accuracy-consistent weighted test time augmentation method.

Test time augmentation is a general methodology that leverages data augmentation techniques to create multiple samples from the original input at inference (_i.e._, at test time). Therefore, test time augmentation methods can be applied to pre-trained models, since, in this case, the augmentation process is not applied during the training phase. The technique introduced in this work combines the use of test time augmentation with a custom weighting system, guaranteeing that the accuracy of the original DL model is not corrupted, while still being optimized to improve uncertainty calibration. This builds, partially, on the work done in [3], proposing both a reformulated version - _V-ATTA_ (Vector Adaptive Test Time Augmentation) - of the preliminary method presented in [3] and also a generalized version - _M-ATTA_ (Matrix Adaptive Test Time Augmentation) - with a broader and extended empirical evaluation.

_M/V-ATTA_ is evaluated on the benchmark CIFAR-10/CIFAR-100 [11] datasets, as well as on a benchmark satellite imagery dataset, the Aerial Image Dataset (AID) [28]. The results are compared with state-of-the-art _post-hoc_ calibration methods, with respect to the Brier score [2] and the _Expected Calibration Error_ (ECE), for both _common_ and _strong_ uncertainty calibration (see Section 3 for further details).

**Contribution**: We propose a novel calibration technique - with two different variations (_M/V-ATTA_) - based on test time augmentation, that guarantees consistency in the accuracy of deep models in which is applied, while being shown to improve uncertainty calibration-related evaluation metrics, outperforming state-of-the-art _post-hoc_ calibration
methods in most cases. To the best of our knowledge, _M/V-ATTA_ is the first method based on test time augmentation that has been proposed to improve the uncertainty calibration of deep models (besides its predecessor in [3]). Furthermore, the method presented here can be used with pre-trained DNNs, which is advantageous in terms of applicability.

## 2 Related Work

The authors in [7] introduce the topic of uncertainty calibration to the DL community, by evaluating how well calibrated are different modern DNNs, using different datasets (from both computer vision and natural language processing applications). As previously stated, the authors argue that, although more accurate, modern DNNs exhibit problems of miscalibration, often more severe than those found in older - and less accurate - architectures. Several _post-hoc_ calibration techniques - particularly designed to address uncertainty calibration without the need of re-training the original DL models - like _temperature scaling_ (an extension of the _Platt scaling_ algorithm [18, 22]), _histogram binning_[29] and _isotonic regression_[30], are described to address this problem.

Other approaches, like approximate Bayesian models [4, 6] and some regularization techniques [15, 21], have also been used in the context of uncertainty calibration [20]. Nonetheless, these approaches require building new more complex models or modifying and re-training pre-existing ones, contrarily to the previously mentioned _post-hoc_ calibration methods and the proposed _M/V-ATTA_.

For the evaluation of uncertainty calibration, one of the most popular metrics is the ECE [17], where the bin-wise difference between the accuracy and the confidence outputs of a given model is evaluated. Although intuitive, some limitations of this metric have been identified in relevant literature [8, 19, 27], with regard to, for example, its intractability and dependence on the chosen binning scheme. Furthermore, because ECE is not a proper scoring rule [5], there can be found trivial uninformative solutions that obtain an optimal score, like always returning the marginal probability (see example in Supplementary Material, Section 2). A popular alternative to evaluate uncertainty calibration is the Brier score [2]; because it is a proper scoring rule and overcomes some of the identified limitation of the ECE, this metric has been increasingly used by the scientific community [20, 13, 24].

Test time augmentation methods have been gaining some attention in the last few years, for example in biomedical applications [14, 25, 26]. Nonetheless, we see that some potential in this type of technique is still under-researched and, to the best of our knowledge, all relevant literature fails to address its effect on calibration-specific metrics like the Brier score or the ECE (with exception of our preliminary work done in [3]). Contrarily to other recent novel approaches to test time augmentation [10, 12, 23], the method proposed in this work does not focuses on improving the accuracy of DNNs, but instead on improving their uncertainty calibration without altering the original accuracy. In fact, the authors in [23] show that some forms of test time augmentation may produce corrupted predictions - which can ultimately worsen the model's performance - reinforcing the need for a test time augmentation-based methodology that does not alter the predicted class, while still calibrating its confidence value.

## 3 Background

**Notation**: we will use bold notation to denote vectors, like \(\mathbf{p}=\ (p_{1},\ldots,p_{k})\); the \(i\)-th element of some vector \(\mathbf{p}\) will be referred as \(\mathbf{p}_{\{i\}}:=p_{i}\); the \(\downarrow\) symbol, associated with a given metric, informs that a lower value of such metric represents a better performance; the \(\odot\) symbol represents the Hadamard product, \(\sigma:\mathbb{R}^{k}\rightarrow\Delta_{k}\) represents the _softmax_ function. These remarks are valid for all sections of the article; other remarks on notation will be given along the text, when found relevant.

In this section we discuss the problem of uncertainty calibration and present some evaluation metrics commonly used in this context. In relevant literature, the concept of uncertainty calibration related to DL systems in a multi-class scenario is often defined in two different ways. The most common way is the one presented in [7], where the multi-class scenario is considered as an extension of the binary problem in a _one_\(\mathbf{\mathit{vs}}\)_all_ approach, taking in account only the calibration of the highest confidence value for each prediction. Nonetheless, some works consider the more general definition present in [29], that takes into account all the confidence values of the predicted probability vector. Like in [27], we make the respective distinction between a _calibrated_ model and a _strongly calibrated_ model, in the following definitions. Notice that in the case of a binary classifier such definitions are equivalent.

Let us consider a pair of random variables \((X,Y)\), where \(X\) represents an input space (or feature space) and \(Y\) the corresponding set of true labels. Let us now take a model \(f:X\rightarrow\Delta_{k}\), with \(\Delta_{k}=\{(p_{1},\ldots,p_{k})\in[0,1]^{k}:\sum_{i=1}^{k}p_{i}=1\}\) being a probability simplex (this setting corresponds to a classification problem with \(k\) different classes). The model \(f\) is considered **calibrated** if

\[\mathds{P}[Y=\operatorname*{arg\,max}_{i\in\{1,\ldots,k\}}f(X)\mid\max_{i\in \{1,\ldots,k\}}f(X)]=\max_{i\in\{1,\ldots,k\}}f(X). \tag{1}\]

Additionally, the model \(f\) is considered **strongly calibrated** if

\[\mathds{P}[Y=y\mid f(X)_{\{y\}}]=f(X)_{\{y\}}. \tag{2}\]

As stated in [7], achieving perfect calibration is impossible in practical settings. Furthermore, the probability values in the left hand side of both (1) and (2) cannot be computed using finitely many samples, which motivates the need for scoring rules to assess uncertainty calibration.

### _Brier score \(\downarrow\)_

Brier score [2] is a proper scoring rule [5] that computes the squared error between a predicted probability and its true
response, hence its utility to evaluate uncertainty calibration. For a set of \(N\) predictions we define the **Brier score** as

\[\text{BS}=\frac{1}{N}\sum_{j=1}^{N}(p^{j}-o^{j})^{2}, \tag{3}\]

where \(p^{j}\) is the highest confidence value of the prediction \(j\) and \(o^{j}\) equals 1 if the true class corresponds to the prediction and 0 otherwise.

The previous definition is useful to asses calibration (in the more common sense). Nonetheless, [2] also presents a definition for Brier score in multi-class scenario that is suitable to assess strong calibration. For a problem with \(k\) different classes and a set of \(N\) predictions we define the **multi-class Brier score** (mc-Brier score) as

\[\text{mc-BS}=\frac{1}{N}\sum_{j=1}^{N}\sum_{i=1}^{k}(p_{i}^{j}-o_{i}^{j})^{2}, \tag{4}\]

where \(p_{i}^{j}\) is the confidence value in the position \(i\) of the \(j\)-nth prediction probability vector and \(o_{i}^{j}\) equals 1 if the true label equals \(i\) and 0 otherwise.

We refer to [16] and [1] for some thorough insights about the interpretability and decomposition of the Brier score.

### _Expected Calibration Error \(\downarrow\)_

To compute the ECE [17] we start by dividing the interval \([0,1]\) in \(M\) equally spaced intervals. Then a set of bins \(\{B_{1},B_{2},\ldots,B_{M}\}\) is created, by assigning each predicted probability value to the respective interval. The idea behind this measurement is to compute a weighted average of the absolute difference between accuracy and confidence in each bin \(B_{i}\) (\(i=1,\ldots,M\)). We define the **confidence per bin** as

\[conf(B_{i})=\frac{1}{|B_{i}|}\sum_{j\in B_{i}}p^{j}, \tag{5}\]

where \(p^{j}\) is the highest confidence value of the prediction \(j\). The **accuracy per bin** is defined as

\[acc(B_{i})=\frac{1}{|B_{i}|}\sum_{j\in B_{i}}o^{j}, \tag{6}\]

where \(o^{j}\) equals \(1\) if the true class corresponds to the prediction and 0 otherwise. For a total of \(N\) predictions and a binning scheme \(\{B_{1},B_{2},\ldots,B_{M}\}\), the **ECE** is defined as

\[\text{ECE}=\sum_{i=1}^{M}\frac{|B_{i}|}{N}|conf(B_{i})-acc(B_{i})|. \tag{7}\]

## 4 Proposed Methodology

In this section we introduce _M-ATTA_ and _V-ATTA_. Both methods leverage the use of test time augmentation, combined with a custom adaptive weighting system. _V-ATTA_ can be interpreted as restricted version of the more general _M-ATTA_.

### _M-Atta_

Let us start by considering \(m\in\mathbb{N}\) different types of augmentations. Because it is common that some augmentations have random parameters, it can be desirable to apply the same type of augmentation more than once; for this reason, let us consider as \(n_{i}\) (\(i=1,\ldots,m\)) the number of times the \(i\)-nth type of augmentation is applied. As such, we will define for each original input \(I_{0}\), the \(j\)-nth (\(j=1,\ldots,n_{i}\)) augmented input with the \(i\)-nth augmentation type, as \(I_{i,j}\).

We can now take into account the model \(f:X\rightarrow\Delta_{k}\) (where \(X\) is the input space and \(k\) the number of classes) and consider \(g:X\rightarrow\mathbb{R}^{k}\) as such that \(f:=g\circ\sigma\). With this, we now define

\[p^{0}=f(I_{0}),\qquad z_{i,j}=g(I_{i,j}), \tag{8}\]

_i.e._, \(p^{0}\) is the probability vector associated with the original input and \(z_{i,j}\) is the logit associated with the \(j\)-nth augmentation of the \(i\)-nth type. Subsequently, we can define, \(\forall i\in[1,\ldots,m]\),

\[\mathbf{z}^{i}=\frac{\sum_{j=1}^{n_{i}}\mathbf{z}_{i,j}}{n_{i}}\equiv\left(z _{1}^{i},z_{2}^{i},\ldots,z_{k}^{i}\right)\in\mathbb{R}^{k}, \tag{9}\]

and then construct the matrix

\[\mathbf{Z}=\left[\mathbf{z}^{i}\right]_{i=1,\ldots,m}^{\text{T}}=\left[\begin{array} []{cccc}z_{1}^{1}&z_{1}^{2}&\cdots&z_{1}^{m}\\ z_{2}^{1}&z_{2}^{2}&\cdots&z_{2}^{m}\\ \vdots&\vdots&\ddots&\vdots\\ z_{k}^{1}&z_{k}^{2}&\cdots&z_{k}^{m}\end{array}\right]\in\mathbb{R}^{k,m}. \tag{10}\]

Now, for some parameters

\[\omega^{*}\in[0,1],\quad\mathbf{W}=\left[\begin{array}{cccc}\omega_{1}^{1}& \omega_{1}^{2}&\cdots&\omega_{1}^{m}\\ \omega_{2}^{1}&\omega_{2}^{2}&\cdots&\omega_{2}^{m}\\ \vdots&\vdots&\ddots&\vdots\\ \omega_{k}^{1}&\omega_{k}^{2}&\cdots&\omega_{k}^{m}\end{array}\right]\in \mathbb{R}^{k,m}, \tag{11}\]

we finally define, for each prediction, the final prediction probability vector as

\[\mathbf{p}\left(\tilde{\omega}\right)=(1-\tilde{\omega})\mathbf{p}^{0}+ \tilde{\omega}\ \sigma(\mathbf{W}\odot\mathbf{Z})I_{m}, \tag{12}\]

with

\[\tilde{\omega}=\max\Big{\{}\omega\in[0,\omega^{*}]:\operatorname*{arg\,max}_{i \in\{1,\ldots,k\}}\mathbf{p}\left(\tilde{\omega}\right)=\operatorname*{arg\, max}_{i\in\{1,\ldots,k\}}\mathbf{p}^{0}\Big{\}}. \tag{13}\]
We consider \(I_{m}\in\mathbb{R}^{m}\) as an \(m\) dimensional vector where every element equals 1 and remind that \(\sigma:\mathbb{R}^{k}\rightarrow\Delta_{k}\) represents the _softmax_ function. We also note that the learnable parameters \(\mathbf{W}\in\mathbb{R}^{k,m}\) and \(\omega^{*}\in\mathbb{R}\) work, respectively, as an weight matrix and an upper bound for \(\tilde{\omega}\).

The value of \(\tilde{\omega}\) may vary in each prediction, adapting in a way that prevents corruptions in terms of accuracy, according to the definition in (13). Both \(\omega^{*}\in[0,1]\) and \(\mathbf{W}\in\mathbb{R}^{k,m}\) can be optimized with a given validation set. In a practical scenario, the value \(\tilde{\omega}\) is approximated as described in the algorithmic description of _M-ATTA_ in Algorithm 1. In our case \(\epsilon\) in Algorithm 1 is defined as 0.01.

### _V-ATTA_

With _V-ATTA_ we restrict the matrix \(\mathbf{W}\) to a diagonal matrix

\[\mathbf{W}_{d}=\left[\begin{array}{cccc}\omega^{1}&0&\cdots&0\\ 0&\omega^{2}&\cdots&0\\ \vdots&\vdots&\ddots&\vdots\\ 0&0&\cdots&\omega^{m}\end{array}\right]\in\mathbb{R}^{m,m}, \tag{14}\]

and define the new prediction probability vector as

\[\mathbf{p}\left(\tilde{\omega}\right)=(1-\tilde{\omega})\mathbf{p}^{0}+ \tilde{\omega}\;\sigma(\mathbf{W}_{d}\mathbf{Z}^{\mathrm{T}})I_{m}, \tag{15}\]

with \(\tilde{\omega}\) still defined as in (13).

We care to note that, contrarily to _M-ATTA_, _V-ATTA_ has the same number of parameters as the preliminary method presented in [3], while still being able of accomplishing improved results (see Supplementary Material, Section 3).

In this case, the algorithmic description is as represented in Algorithm 2. Once again, \(\epsilon\) is 0.01.

```
Input: Augmented logits matrix (\(\mathbf{Z}\in\mathbb{R}^{k,m}\)),  Original prediction (\(\mathbf{p}^{0}\in\Delta_{k}\)) Output: Calibrated prediction (\(\mathbf{p}\in\Delta_{k}\)) Learnable parameters:\(\mathbf{W}_{d}\in\mathbb{R}^{m,m}\), \(\omega^{*}\in\mathbb{R}\)
1\(\epsilon\gets 0.01\)\(\tilde{\omega}\leftarrow\omega^{*}\)\(c_{0}\leftarrow\arg\max_{i\in\{1,\dots,k\}}\mathbf{p}^{0}\)while\(c\neq c_{0}\;\wedge\;\tilde{\omega}>0\)do
2\(\mathbf{p}\leftarrow(1-\tilde{\omega})\mathbf{p}^{0}+\tilde{\omega}\;\sigma( \mathbf{W}_{d}\mathbf{Z}^{\mathrm{T}})I_{m}\)\(c\leftarrow\arg\max_{i\in\{1,\dots,k\}}\mathbf{p}\)\(\tilde{\omega}\leftarrow\tilde{\omega}-\epsilon\)
```

**Algorithm 2**V-ATTA

## 5 Experiments and Results

Let us start by observing that test time augmentation methods - just like traditional augmentation methods - can be applied with an highly extensive set of augmentations policies, thus possibly resulting in virtually endless different scenarios. Nevertheless, for practical reasons, the experiments here presented were conducted by taking into consideration only four different image transformations, combining them into eight different augmentation policies. The image transformations used were

* _Flip_: a flip around the vertical axis of the image;
* _Crop_: creates a cropped input with size ratio of approximately \(0.8\), extracted from a random position within the original input image;
* _Brightness_: creates, from an original input \(\gamma\), a new input \(\gamma^{\prime}=\gamma+\beta\gamma_{\text{max}}\), where \(\gamma_{\text{max}}\) is the maximum pixel value from \(\gamma\), and \(\beta\) is a random number extracted from a continuous Uniform distribution within the interval \([-0.5,0.5]\);
* _Contrast_: creates, from an original input \(\gamma\), a new input \(\gamma^{\prime\prime}=\gamma(1+\alpha)\), where \(\alpha\) is a random number extracted from a continuous Uniform distribution within the interval \([-0.2,0.2]\).

These four types of augmentations were selected for this work based on the fact that they are common and easily replicable image transformations, thus reinforcing the merits of the proposed methods (_M/V-ATTA_) by showing their applicability without the need of extensively searching a set of complex augmentations.

The composition of the eight different augmentation policies is described in Table I. The rationale behind the selection of such policies is based on some conclusions derived from the results with the preliminary approach presented in [3]. First, all augmentation policies are composed by more than one type of augmentation, based on the fact that this was generally the best approach in [3]. Secondly, the _contrast_ transformation is only present in augmentation policies that are composed by three or more types of augmentations, since there was some evidence in [3] that this transformation had, in general, the lowest positive impact in terms of uncertainty calibration.

In all experiments, a ResNet-50 [9] architecture is used. The achieved classification accuracy values are \(94.21\%\), \(72.78\%\) and \(93.40\%\), for CIFAR-10, CIFAR-100 and AID test sets, respectively. The size of the training, validation and test sets are respectively: 50000, 1000, 9000, for the CIFAR-10/100 datasets, 7000, 1000, 2000, for the AID dataset. We prioritized having the same size in all validation sets, since this is the one used in the optimization process of _M/V-ATTA_.

For each dataset, the experiments made with _M/V-ATTA_ are performed according to the following steps: using each of the eight augmentation policies (see Table I), the learnable parameters are optimized in the validation set, with Negative Log Likelihood (NLL) as loss function, 500 epochs, batch size of 500, Adam optimizer with a learning rate of 0.001 and all weights initialized as 1; the eight obtained versions of M/V-ATTA are compared on the validation

\begin{table}
\begin{tabular}{l||l} \hline \hline Aug. 1: \(F+5Cr\) & Aug. 5: \(F+5Cr+5Ct\) \\ Aug. 2: \(F+5B\) & Aug. 6: \(F+5B+5Ct\) \\ Aug. 3: \(5Cr+5B\) & Aug. 7: \(5Cr+5B+5Ct\) \\ Aug. 4: \(F+5Cr+5B\) & Aug. 8: \(F+5Cr+5B+5Ct\) \\ \hline \hline \end{tabular}
\end{table} TABLE I: Composition of eight different augmentation policies used in this work. \(F\) refers to a _flip_ transformation, \(Cr\) to _crop_, \(B\) to _brightness_ and \(Ct\) to _contrast_. _E.g._, Augmentation Policy 1 (Aug. 1) is composed by one _flip_ transformation and five _crop_ transformations.
set (the same used in the optimization), using the Brier score, where the best performing augmentation policy is selected (see Subsection 5.1); using the previously found augmentation policy, the method is finally evaluated in the test set (see Subsection 5.2).

### _Augmentation policy validation_

In this subsection we compare the eight augmentation policies described in Table I, in terms on how _M-ATTA_ and V-ATTA perform, considering the Brier score. For this purpose, the experiments are done in the respective validation set of each of the three datasets (CIFAR-10, CIFAR-100, AID). The results are presented in Table II. The augmentation policies that achieve the best performance, in each case, are subsequently used in Subsection 5.2 in the experiments done with each respective test set.

We observe, by the analysis of the results in Table II, that while Aug. 8 is the best performing augmentation policy for _M-ATTA_ - independent of which dataset is used - the same does not hold when considering _V-ATTA_, where for each dataset we find a different best performing augmentation policy. Nonetheless, the results obtained with Aug. 8 applied to _V-ATTA_ are still relatively close to best performing scenario, in all three datasets. We also note that Aug. 2 and Aug. 6 - which show the best results for _V-ATTA_ in CIFAR-10 and CIFAR-100, respectively - are the worst performing augmentation policies when considering the AID dataset, suggesting some dataset-dependent behaviour.

Finally, we also observe that _M-ATTA_ performs better than _V-ATTA_ in all cases in the three validation sets.

The analogous results of those in Table II, but for the respective test sets, can be found in the Supplementary Material, Section 3.

### _Main results_

Here we describe and discuss the results summarized in Table III. The results obtained with both _M-ATTA_ and _V-ATTA_ are compared against the performance of _temperature scaling_[7], _isotonic regression_[30], _histogram binning_[29] (for details on these baseline methods see the Supplementary Material, Section 1) and a _vanilla_ approach (referring to the results obtained from the DNN without any type of calibration method). The choice of these baselines takes into consideration that these methods share, with both _M-ATTA_ and _V-ATTA_, two important characteristics: can be applied to pre-trained DNNs, requiring no modification of the network architecture; do not alter the original classification accuracy of the DNN with which they are applied.

All methods are evaluated with the described uncertainty calibration metrics - Brier score (both "classical" and multi-class version) and ECE (with 15 bins) - and also the widely known NLL loss. Although a proper scoring rule (and so, built to evaluate the quality of probabilistic predictions), the NLL loss (traditionally used in the training process of DNNs) cannot be considered a metric of calibration (or strong calibration) given our definition, because

\begin{table}
\begin{tabular}{c|c|c c c c c c c c c} \hline  & & Aug. 1 & Aug. 2 & Aug. 3 & Aug. 4 & Aug. 5 & Aug. 6 & Aug. 7 & Aug. 8 \\ \hline \hline \multirow{2}{*}{CIFAR-10} & M-ATTA & 0.0399 & 0.0371 & 0.0434 & 0.0372 & 0.0376 & 0.0366 & 0.0416 & **0.0362** \\  & V-ATTA & 0.0426 & **0.0402** & 0.0465 & 0.0417 & 0.0419 & 0.0408 & 0.0448 & 0.0408 \\ \hline \hline \multirow{2}{*}{CIFAR-100} & M-ATTA & 0.1149 & 0.1126 & 0.1179 & 0.1081 & 0.1084 & 0.1109 & 0.1134 & **0.1065** \\  & V-ATTA & 0.1277 & 0.1242 & 0.1336 & 0.1246 & 0.1248 & **0.1240** & 0.1311 & 0.1241 \\ \hline \hline \multirow{2}{*}{AID} & M-ATTA & 0.0272 & 0.0453 & 0.0284 & 0.0254 & 0.0253 & 0.0428 & 0.0263 & **0.0247** \\  & V-ATTA & 0.0316 & 0.0529 & 0.0328 & **0.0310** & 0.0314 & 0.0507 & 0.0328 & 0.0315 \\ \hline \end{tabular}
\end{table} TABLE II: Results with respect to the Brier score - in the respective validation sets of the CIFAR-10, CIFAR-100 and AID datasets - comparing the eight different augmentation policies with both our methods (_M-ATTA_ and _V-ATTA_). For each case, the best result is represented in bold.

\begin{table}
\begin{tabular}{c|c c c c|c c c c|c c c} \hline  & \multicolumn{4}{c|}{CIFAR-10} & \multicolumn{4}{c|}{CIFAR-100} & \multicolumn{4}{c}{AID} \\  & Brier & ECE & mc-Brier & NLL & Brier & ECE & mc-Brier & NLL & Brier & ECE & mc-Brier & NLL \\ \hline \hline Vanilla & 0.0438 & 0.0367 & 0.0929 & 0.2356 & 0.1599 & 0.1325 & 0.4055 & 1.2132 & 0.0481 & 0.0306 & 0.1068 & 0.3046 \\ \hline \hline T. Scaling [7] & 0.0390 & 0.0119 & 0.0860 & 0.1909 & 0.1364 & 0.0241 & 0.3766 & 1.0208 & 0.0493 & 0.0300 & 0.1073 & 0.2621 \\ I. Regression [30] & 0.0396 & **0.0069** & 0.0870 & 0.1918 & 0.1352 & 0.0192 & 0.3792 & 1.0471 & 0.0476 & 0.0158 & 0.1074 & 0.2725 \\ H. Binning [29] & 0.0462 & 0.0112 & 0.0967 & 0.2613 & 0.1412 & **0.0135** & 0.3907 & 1.4232 & 0.0467 & **0.0138** & 0.1083 & 0.3376 \\ \hline \hline
**M-ATTA** (ours) & 0.0371 & 0.0090 & 0.0813 & 0.1800 & 0.1350 & 0.0274 & 0.3730 & 1.0042 & **0.0263** & 0.0232 & **0.0632** & **0.1300** \\
**V-ATTA** (ours) & **0.0358** & 0.0130 & **0.0793** & **0.1705** & **0.1270** & 0.0187 & **0.3584** & **0.9565** & 0.0278 & 0.0221 & 0.0645 & 0.1365 \\ \hline \end{tabular}
\end{table} TABLE III: Results with respect to four different performance metrics (Brier score, ECE, multi-class Brier score and NLL) - in the respective test sets of the CIFAR-10, CIFAR-100 and AID datasets - comparing the performance of our methods (_M-ATTA_ and _V-ATTA_) against four different baselines (Vanilla, Temperature Scaling, Isotonic Regression and Histogram Binning). For each case, the best result is represented in bold.
it only considers the quality of the predictions associated with the true class; nevertheless, given that it is a proper scoring rule and a popular metric, it is common to consider the NLL alongside to Brier score and ECE, when evaluating uncertainty calibration methods [13, 20, 24].

We start our analysis of the results in Table III by comparing all the uncertainty calibration methods against the _vanilla_ approach. The first main observation derived in this context, is that our methods (_M/V-ATTA_) are the only that have consistently better performance than the _vanilla_ approach, presenting better results in all evaluation metrics and all three datasets. Contrarily, _temperature scaling_ worsens the performance in the AID dataset (Brier score, mc-Brier score), _isotonic regression_ also worsens the performance in the AID dataset (Brier score) and _histogram binning_ worsens the performance in CIFAR-10 (Brier score, mc-Brier score, NLL), CIFAR-100 (NLL) and AID (mc-Brier score, NLL) datasets.

Focusing now on highlighting the best results for each specific scenario, we observe that in all three datasets, one of our methods (_M-ATTA_ or _V-ATTA_) is the best performing method in terms of Brier score, mc-Brier score and NLL. Focusing in the three aforementioned evaluation metrics: in both CIFAR-10 and CIFAR-100 datasets, _V-ATTA_ is consistently the best performing method, while _M-ATTA_ is consistently the second-best performing method; in the AID dataset, _M-ATTA_ is consistently the best performing method, while _V-ATTA_ is consistently the second-best performing method. On the other hand, when considering ECE, _isotonic regression_ is the best performing method in the CIFAR-10 dataset, while _histogram binning_ is best performing method in the remaining two datasets (these methods are actually biased to ECE evaluation, since they are binning-based methods, what can explain the particularly good behaviour with this metric); still in this context, _M-ATTA_ is the second-best performing method in CIFAR-10, and _V-ATTA_ is the second best performing method in CIFAR-100.

Some additional observations can be derived from Table III: _histogram binning_ seems to be an unreliable uncertainty calibration method, given how it worsens multiple evaluation metrics in different datasets; the AID dataset is found to be an interesting case study, where all three post-hoc calibration baselines struggle to accomplish consistent results; the empirical evidence presented suggests that _M-ATTA_ and _V-ATTA_ are the most consistent and generally best performing methods.

Finally, while _M-ATTA_ is always better - in terms of Brier score - in the validation sets (see Subsection 5.1), this is not observable when using the respective test sets. This behavior can suggest some type of over-fitting is happening in the optimization process of _M-ATTA_. It is not surprising that _M-ATTA_ is more prone to over-fitting phenomena than _V-ATTA_, since it has a larger number of parameters (proportional to the number of classes in the respective dataset).

### _Validation size experiments_

In the previous subsection we observed that, when considering Brier score, mc-Brier score and NLL, _V-ATTA_ is the best performing method in CIFAR-10 and CIFAR-100 (that are datasets with similar characteristics) while _M-ATTA_ is the best performing method in the AID dataset. While this is possibly justified by the different nature of the AID dataset (in comparison to CIFAR-10/100), we explore the possibility that the different behavior is caused by different proportions in terms validation/test sets. As such, this subsection is centered around experimenting with the ratio between the validation and the test set on the CIFAR-10 and CIFAR-100

Fig. 1: Comparing the Brier and mc-Brier scores obtained with _M-ATTA_ and _V-ATTA_, with different validation/test set ratios, evaluated in the respective test sets of both CIFAR-10 and CIFAR-100 datasets.
datasets. With these experiments we also try to reduce the over-fitting phenomena in the _M-ATTA_ method, since we are increasing the size of the validation sets.

The results are shown in Figure 1. In the presented experiments, both _M-ATTA_ and _V-ATTA_ are optimized with different validation/test set size ratios, beginning with 1000 validation set samples (like in Subsection 5.2) and iteratively adding 500 samples (while subtracting them from the test set) until reaching 5000 (this is done in both CIFAR-10 and CIFAR-100 datasets). In the CIFAR-10 dataset, we observe that _M-ATTA_ actually outperforms _V-ATTA_ when the validation set size is around the range between 2500 and 4000 (in this range the validation set size is approximately 1/3 of the test set size, just like in the AID setup for the previous experiments). On the other hand, in the CIFAR-100 experiments, although some improvement in performance is observed with _M-ATTA_, it is not enough to outperform _V-ATTA_. It is somewhat expected that the problem of over-fitting is easily addressed when working with CIFAR-10, since the _M-ATTA_ model used in CIFAR-100 has ten times more parameters than the one used with CIFAR-10.

## 6 Final Remarks

Based on the results presented and discussed in the previous section, we highlight some conclusions derived from this work:

* contrarily to the baseline methods selected for comparison
- they never produce worse results (compared to the _unvilla_ approach) with any of the selected evaluation metrics, and are always the best performing methods in terms of Brier score, mc-Brier score and NLL, independent of the dataset.
* Although generally the best performing, our methods lose to either _isotonic regression_ or _histogram binning_ when evaluated with the ECE. These baseline methods are somewhat biased towards ECE evaluation, since they leverage binning for obtaining better calibrated predictions; we speculate this justifies the inconsistency of these methods with the other evaluation metrics and the specially good performance with the ECE.
* _V-ATTA_ is capable of outperforming _M-ATTA_ with less parameters, in the CIFAR-10 and CIFAR-100 test sets, even though _M-ATTA_ is always the best method in the respective validation sets. We speculate that _M-ATTA_ is suffering of some over-fitting phenomena in these scenarios; in the case of CIFAR-10, we show this can be surpassed by increasing the size of the validation set (when considering Brier and mc-Brier scores).

For future work, we identify some challenges that can be addressed: studying different solutions for the partial over-fitting phenomena found with _M-ATTA_; exploring more complex types of augmentations; adapting our methods and extending their empirical evaluation to scenarios of object detection and/or semantic segmentation.

## Acknowledgments

This work has been supported by the Portuguese Foundation for Science and Technology (FCT), via the project \(GreenBotics\) (PTDC/EEI-ROB/2459/2021), and partially by Critical Software, SA.
