How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods
  for Selective Classification with Deep Neural Networks
This paper addresses the problem of selective classification for deep neural
networks, where a model is allowed to abstain from low-confidence predictions
to avoid potential errors. We focus on so-called post-hoc methods, which
replace the confidence estimator of a given classifier without modifying or
retraining it, thus being practically appealing. Considering neural networks
with softmax outputs, our goal is to identify the best confidence estimator
that can be computed directly from the unnormalized logits. This problem is
motivated by the intriguing observation in recent work that many classifiers
appear to have a "broken" confidence estimator, in the sense that their
selective classification performance is much worse than what could be expected
by their corresponding accuracies. We perform an extensive experimental study
of many existing and proposed confidence estimators applied to 84 pretrained
ImageNet classifiers available from popular repositories. Our results show that
a simple $p$-norm normalization of the logits, followed by taking the maximum
logit as the confidence estimator, can lead to considerable gains in selective
classification performance, completely fixing the pathological behavior
observed in many classifiers. As a consequence, the selective classification
performance of any classifier becomes almost entirely determined by its
corresponding accuracy. Moreover, these results are shown to be consistent
under distribution shift. Our code is available at
https://github.com/lfpc/FixSelectiveClassification.
Luís Felipe P. Cattelan, Danilo Silva
Improving selective classification performance of deep neural networks through post-hoc logit normalization and temperature scaling

###### Abstract

This paper addresses the problem of selective classification for deep neural networks, where a model is allowed to abstain from low-confidence predictions to avoid potential errors. Specifically, we tackle the problem of optimizing the confidence estimator of a fixed classifier, aiming to enhance its misclassification detection performance, i.e., its ability to discriminate between correct and incorrect predictions by assigning higher confidence values to the correct ones. Previous work has found that different classifiers exhibit varying levels of misclassification detection performance, particularly when using the maximum softmax probability (MSP) as a measure of confidence. However, we argue that these findings are mainly due to a sub-optimal confidence estimator being used for each model. To overcome this issue, we propose a simple and efficient post-hoc confidence estimator, named \(p\)-NormSoftmax, which consists of transforming the logits through \(p\)-norm normalization and temperature scaling, followed by taking the MSP, where \(p\) and the temperature are optimized based on a hold-out set. This estimator can be easily applied on top of an already trained model and, in many cases, can significantly improve its selective classification performance. When applied to 84 pretrained Imagenet classifiers, our method yields an average improvement of 16% in the area under the risk-coverage curve (AURC), exceeding 40% for some models. Furthermore, after applying \(p\)-NormSoftmax, we observe that these models exhibit approximately the same level of misclassification detection performance, implying that a model's selective classification performance is almost entirely determined by its accuracy at full coverage.

## 1 Introduction

A reliable model must be able to identify cases where it is likely to make an incorrect prediction and withhold the output to prevent a wrong decision. This ability is essential in many real-world applications, such as in finance, medical diagnosis, and autonomous driving, where the consequences of erroneous decisions can be severe (Zou et al., 2023; Neumann et al., 2018). However, it is well-known that modern deep neural networks, which have increasingly been considered for such applications, often exhibit overconfidence in their predictions (Guo et al., 2017; Goodfellow et al., 2014). This issue has motivated a lot of recent research in the general subject of uncertainty estimation in deep learning (Gawlikowski et al., 2022; Zhang et al., 2023; Abdar et al., 2021).

The task of enhancing a classifier's accuracy by abstaining from low-confidence predictions is known as _selective classification_(Geifman and El-Yaniv, 2017), which is essentially equivalent to the task of misclassification detection (Hendrycks and Gimpel, 2016). In the case of neural networks
with softmax outputs, the natural baseline is to take the maximum softmax probability (MSP) as a confidence estimator (Geifman and El-Yaniv, 2017; Hendrycks and Gimpel, 2016). The vast majority of papers in the area attempt to improve upon this baseline by either modifying the training procedure or designing a specific architecture to provide better uncertainty quantification. While such an approach is potentially optimal, the fact that it requires retraining a model is a significant practical drawback.

An alternative, less explored approach is that of post-hoc learning a confidence estimator, which does not require retraining. Papers that follow this approach typically construct a _meta-model_ that feeds on intermediate features of the base model and is trained to predict whether or not the base model is correct on hold-out samples (Corbiere et al., 2022; Shen et al., 2022). However, depending on the size of such a meta-model, its training may still be computationally demanding. Another approach that may be considered post-hoc is the use of certain ensembles that do not require retraining, such as Monte-Carlo dropout (Gal and Ghahramani, 2016). However, the fact that multiple inference passes need to be performed significantly increases the computational burden at test time.

In this paper, we focus on simple post-hoc methods for confidence estimation that can be computed directly from the network unnormalized _logits_ (pre-softmax output). This approach is practically appealing as it can be directly applied to any pre-trained model. Such post-hoc methods are common in the related (but fundamentally different) task of probability calibration, which aims to provide probability estimates representative of the true likelihood of correctness. The most prominent example is temperature scaling (TS) (Guo et al., 2017), a simple and efficient logit-based method that requires tuning a single parameter and is shown to be remarkably effective for calibration. The usefulness of TS for selective classification has been investigated by (Galil et al., 2023), who observed that, depending on the model, TS may improve or harm selective classification performance. To the best of our knowledge, designing and optimizing logit-based confidence estimators for selective classification has not been attempted before.

Inspired by (Galil et al., 2023), we propose a post-hoc confidence estimator that combines three previous ideas: temperature scaling, logit normalization (Wei et al., 2022) and logit centralization (Jiang et al., 2023), the latter two originally proposed as training techniques. We further extend these ideas by considering a more general \(p\)-norm normalization and by optimizing the temperature (as well as \(p\)) directly to improve selective classification performance. In addition, we propose a simple heuristic to choose the temperature so that only \(p\) needs to be optimized. Our approach, named \(p\)-NormSoftmax, is practically appealing as it can be applied to any existing model without retraining, requires tuning a single parameter, is straightforward to implement, and is very data-efficient. Moreover, it can provide significant gains in selective classification performance for a variety of existing models.

Figure 1: A comparison of RC curves made by three models selected in (Galil et al., 2023), including examples of highest (ViT-L/16-384) and lowest (EfficientNet-V2-XL) AUROC. After the application of our post-hoc method, the apparent pathology in EfficientNet-V2-XL completely disappears, resulting in significantly improved selective classification performance.
Our method apparently solves an intriguing problem reported in (Gail et al., 2023) and illustrated in Fig. 1: some state-of-the-art ImageNet classifiers, despite attaining excellent predictive performance, nevertheless exhibit appallingly poor performance at misclassification detection. After applying our method, this issue completely disappears, suggesting that such pathologies are fixable.

The contributions of this work are:

* We investigate the trade-off between calibration and selective classification metrics for temperature scaling;
* We propose a simple and efficient post-hoc confidence estimator optimized for selective classification;
* An experimental study of the selective classification performance of 84 ImageNet classifiers, showing an average gain of 16% in AURC after the application of our method;
* A comparison of these models showing that, after \(p\)-NormSoftmax, all models exhibit approximately the same level of misclassification detection performance.

## 2 Related Work

Selective prediction is also known as learning with a reject option (see (Zhang et al., 2023; Hendrickx et al., 2021) and references therein), where the rejector is usually a thresholded confidence estimator. Essentially the same problem is studied under the equivalent terms misclassification detection (Hendrycks and Gimpel, 2016), failure prediction (Corbiere et al., 2022; Zhu et al., 2022), and (ordinal) ranking (Moon et al., 2020; Galil et al., 2023). Uncertainty estimation is a more general term that encompasses these tasks (where confidence may be taken as negative uncertainty) as well as other tasks where uncertainty might be useful, such as calibration and out-of-distribution (OOD) detection, among others (Gawlikowski et al., 2022; Abdar et al., 2021). These tasks are generally not aligned: for instance, optimizing for calibration may harm selective classification performance (Ding et al., 2020; Zhu et al., 2022; Galil et al., 2023). Our focus here is on in-distribution selective classification, although we also study robustness to distribution shift. While most approaches consider the base model as part of the learning problem (Geifman and El-Yaniv, 2019; Huang et al., 2020; Liu et al., 2019), we focus on simple post-hoc estimators that can be computed from the logits.1. Note that, from a post-hoc perspective, other tasks can be treated as independent problems.

Footnote 1: Interestingly, Feng et al. (2023) has found that, for some of these approaches, MSP is still the best selective mechanism after the base model is trained.

A popular tool in the uncertainty literature is the use of ensembles (Lakshminarayanan et al., 2017; Gal and Ghahramani, 2016; Teye et al., 2018; Ayhan and Berens, 2018). While constructing a confidence estimator from ensemble component outputs may be considered post-hoc if the ensemble is already trained, recent work has found evidence that ensembles may not be fundamental for uncertainty but simply better predictive models (Abe et al., 2022; Cattelan and Silva, 2022; Xia and Bouganis, 2022). Thus, we do not consider ensembles here.

Applying TS to improve calibration (of the MSP confidence estimator) was proposed in (Guo et al., 2017) based on the negative log-likelihood. Optimizing TS for other metrics has been explored in (Mukhoti et al., 2020; Karandikar et al., 2021; Clarke et al., 2023) for calibration and in (Liang et al., 2023) for OOD detection, but had not been proposed for selective classification. A generalization of TS is adaptive TS (ATS) (A. Balanya et al., 2023), which uses an input-dependent temperature based on logits. Our approach can be seen as a special case of ATS, as logit norms may be seen as an input-dependent temperature; however A. Balanya et al. (2023) investigate a different temperature function than ours and focuses on calibration. Other logit-based confidence estimators proposed for calibration and OOD detection include (Liu et al., 2020; Tomani et al., 2022; Rahimi et al., 2022; Neumann et al., 2018; Gonsior et al., 2022).

Normalizing the logits with the \(L_{2}\) norm before applying the softmax function was used in (Kornblith et al., 2021) and later proposed and studied in (Wei et al., 2022) as a training technique (combined with TS) to improve OOD detection and calibration. A variation where the logits are normalized to unit variance was proposed in (Jiang et al., 2023) to accelerate training.

Benchmarking of models in their performance at selective classification/misclassification detection has been done in (Gail et al., 2023; Ding et al., 2020), however these works mostly consider the
MSP as the confidence estimator. In the context of calibration, Wang et al. (2021) and Ashukha et al. (2020) have argued that models should be compared after simple post-hoc optimizations, since models that appear worse than others can sometimes easily be improved by methods such as TS. Here we advocate and provide further evidence for this approach in the context of selective classification.

## 3 Problem Formulation and Background

### Selective classification

Let \(P\) be an unknown distribution over \(\mathcal{X}\times\mathcal{Y}\), where \(\mathcal{X}\) is the input space and \(\mathcal{Y}=\{1,\ldots,C\}\) is the label space, and \(C\) is the number of classes. A _classifier_ is a prediction function \(h:\mathcal{X}\rightarrow\mathcal{Y}\). The classifier's (true) _risk_ is \(R(h)=E_{P}[\ell(h(x),y)]\), where \(\ell:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}^{+}\) is a given loss function, for instance, the 0/1 loss \(\ell(\hat{y},y)=\mathds{1}[\hat{y}\neq y]\), where \(\mathds{1}[\cdot]\) denotes the indicator function.

A _selective classifier_(Geifman and El-Yaniv, 2017) is a pair \((h,g)\), where \(h\) is a classifier and \(g:\mathcal{X}\rightarrow\mathbb{R}\) is a _confidence estimator_ (also known as _confidence score function_ or _confidence-rate function_), which quantifies the model's confidence on its prediction for a given input. For some fixed threshold \(t\), given an input \(x\), the selective model makes a prediction \(h(x)\) if \(g(x)\geq t\), otherwise it abstains from making a prediction. We say that \(x\) is _selected_ in the former case and _rejected_ in the latter. A selective model's _coverage_\(\phi(h,g)=P[g(x)\geq t]\) is the probability mass of the selected samples in \(\mathcal{X}\), while its _selective risk_\(R(h,g)=E_{P}[\ell(h(x),y)\mid g(x)\geq t]\) is its risk restricted to the selected samples. In particular, a model's risk equals its selective risk at _full coverage_ (i.e., for \(t\) such that \(\phi(h,g)=1\)). These quantities can be evaluated empirically given a given a test dataset \(\{(x_{i},y_{i})\}_{i=1}^{N}\) drawn i.i.d. from \(P\), yielding the _empirical coverage_\(\hat{\phi}(h,g)=(1/N)\sum_{i=1}^{N}\mathds{1}[g(x_{i})\geq t]\) and the _empirical selective risk_

\[\hat{R}(h,g)=\frac{\sum_{i=1}^{N}\ell(h(x_{i}),y_{i})\mathds{1}[g(x_{i})\geq t ]}{\sum_{i=1}^{N}\mathds{1}[g(x_{i})\geq t]}. \tag{1}\]

Note that, by varying \(t\), it is generally possible to trade off coverage for selective risk, i.e., a lower selective risk can usually (but not necessarily always) be achieved if more samples are rejected. This tradeoff is captured by the _risk-coverage (RC) curve_(Geifman and El-Yaniv, 2017), a plot of \(\hat{R}(h,g)\) as a function of \(\hat{\phi}(h,g)\).

While the RC curve provides a full picture of the performance of a selective classifier, it is convenient to have a scalar metric that summarizes this curve. A commonly used metric is the _area under the RC curve_(AURC) (Ding et al., 2020; Geifman et al., 2019). However, when comparing selective models, if two RC curves cross, then each model may have a better selective performance than the other depending on the operating point chosen, which cannot be captured by the AURC. Another interesting metric, which forces the choice of an operating point, is the _selective accuracy constraint_(SAC) (Galil et al., 2023), defined as the minimum coverage required for a model to achieve a specified accuracy.

Misclassification detection (Hendrycks and Gimpel, 2016), which refers to the problem of discriminating between correct and incorrect predictions made by a classifier, is closely related to selective classification. Both tasks rely on ranking predictions according to their confidence estimates, where correct predictions should be ideally separated from incorrect ones. More precisely, if \((x_{1},y_{1}),(x_{2},y_{2})\in\mathcal{X}\times\mathcal{Y}\) are such that \(\ell(h(x_{1}),y_{1})>\ell(h(x_{2}),y_{2})\), then we would like to have \(g(x_{1})<g(x_{2})\), i.e., an optimal \(g\) orders samples in decreasing order of their losses. In the case of the 0/1 loss, a natural metric of ranking performance (Galil et al., 2023) is the area under the ROC curve (AUROC) (Fawcett, 2006) for misclassification detection. Note that this metric is blind to the classifier performance and focuses exclusively on the quality of the confidence estimates, i.e., given a fixed classifier \(h\), different confidence estimators \(g\) can be compared in their ranking performance. Thus, misclassification detection can also be seen as a proxy problem on which to evaluate confidence estimators for selective classification.

### Calibration

Consider a classifier \(h:\mathcal{X}\rightarrow\mathcal{Y}\) and a confidence estimator \(\pi:\mathcal{X}\rightarrow[0,1]\) (which need not be the same function as the confidence estimator \(g\) used for selective classification). We say that \(\pi\) is
perfectly calibrated_(Guo et al., 2017; Gawlikowski et al., 2022) if

\[P[h(x)=y\mid\pi(x)=p]=p,\quad\forall p\in[0,1],\quad(x,y)\sim P. \tag{2}\]

In practice, empirical measures of calibration are used, based on a test dataset \(\{(x_{i},y_{i})\}_{i=1}^{N}\) drawn i.i.d. from \(P\). The most popular one is arguably the _expected calibration error_(ECE) (Naeini et al., 2015), which is computed by grouping predictions into \(M\) equal-sized interval bins \(B_{m}=\{i\in\{1,\ldots,N\}:\pi(x_{i})\in(\frac{m-1}{M},\frac{m}{M}]\}\), \(m=1,\ldots,M\), and then taking a weighted average of the difference between accuracy and confidence in each bin:

\[\text{ECE}=\sum_{m=1}^{M}\frac{|B_{m}|}{N}\left|\text{acc}(B_{m})-\text{conf}( B_{m})\right| \tag{3}\]

where \(\text{acc}(B_{m})=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}\mathds{1}[h(x_{i})=y_{i}]\) and \(\text{conf}(B_{m})=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}\pi(x_{i})\).

### Confidence estimation

From now on we restrict attention to classifiers that can be decomposed as \(h(x)=\arg\max_{k\in\mathcal{Y}}z_{k}\), where \(\mathbf{z}=f(x)\) and \(f:\mathcal{X}\rightarrow\mathbb{R}^{C}\) is a neural network. The network output \(\mathbf{z}\) is referred to as the (vector of) _logits_ or _logit vector_, due to the fact that it is typically applied to a softmax function

\[\sigma:\mathbb{R}^{C}\rightarrow[0,1]^{C},\qquad(\sigma(\mathbf{z}))_{k}= \frac{e^{z_{k}}}{\sum_{j=1}^{C}e^{z_{j}}},\quad k\in\{1,\ldots,C\} \tag{4}\]

to obtain an estimate of the posterior distribution \(P[y|x]\).

The most popular confidence estimator is arguably the _maximum softmax probability_(MSP) (Ding et al., 2020), also known as _maximum class probability_(Corbiere et al., 2022) or _softmax response_(Geifman and El-Yaniv, 2017)

\[g(x)=\text{MSP}(\mathbf{z})\triangleq\max_{k\in\mathcal{Y}}{(\sigma(\mathbf{ z}))_{k}}. \tag{5}\]

Other representative examples are given in (Belghazi and Lopez-Paz, 2021).

### Temperature Scaling

Temperature scaling (TS) (Guo et al., 2017) is a post-processing method that consists in, for a fixed trained classifier, transforming the logits as \(\mathbf{z}^{\prime}=\mathbf{z}/T\), before applying the softmax function. The parameter \(T\), called the temperature, is then optimized over a hold-out dataset \(\{(x_{i},y_{i})\}_{i=1}^{N}\) (not used during training of the classifier). An important property of this method is that it does not change the model's predictions. The conventional way of applying TS, as proposed in (Guo et al., 2017) for calibration and referred to here as _standard_ TS, consists in optimizing \(T\) with respect to the negative log-likelihood (NLL) (Murphy, 2022)

\[\mathcal{L}=-\sum_{i=1}^{N}\log{((\sigma(\mathbf{z}_{i}/T))_{y_{i}})} \tag{6}\]

where \(\mathbf{z}_{i}=f(x_{i})\).

## 4 Post-hoc Confidence Estimation for Selective Classification

### Proposed Method

Refer to the notation of section 3. Our proposed method for post-hoc confidence estimation is given by

\[g(x)=\text{MSP}\left(\beta\frac{\mathbf{z}-\mu(\mathbf{z})}{\|\mathbf{z}-\mu (\mathbf{z})\|_{p}}\right) \tag{7}\]

where \(\beta>0\), \(\mu(\mathbf{z})\triangleq(z_{1}+\cdots+z_{C})/C\), and \(\|\mathbf{z}\|_{p}\triangleq(|z_{1}|^{p}+\cdots+|z_{C}|^{p})^{1/p}\) is the \(p\)-norm of \(\mathbf{z}\). Thus, our method consists of transforming the logits through centralization (\(\mathbf{z}\leftarrow\mathbf{z}-\mu(\mathbf{z})\)),
\(p\)-normalization (\(\mathbf{z}\leftarrow\mathbf{z}/\|\mathbf{z}\|_{p}\)) and temperature scaling (\(\mathbf{z}\leftarrow\beta\mathbf{z}\)), followed by taking the MSP. To ensure that our method can never cause harm, we augment the definition of \(p\)-norm with

\[\|\mathbf{z}\|_{\emptyset}=1 \tag{8}\]

so that the allowed range for \(p\) is \(\mathbb{R}\cup\{\emptyset\}\).

The hyperparameters \(p\) and \(\beta\) are optimized (e.g., via grid search) based on a hold-out set \(\{(x_{i},y_{i})\}_{i=1}^{N}\), using directly the AURC (or the AUROC) as the objective. In practice, the logits \(\mathbf{z}_{i}=f(x_{i})\) of all hold-out samples are pre-computed and stored, so that any metric based on them can be computed very quickly.

We also propose a simple heuristic for choosing \(\beta\) (given \(p\)):

\[\beta=\frac{1}{N}\sum_{i=1}^{N}\,\|\mathbf{z}_{i}\|_{p}. \tag{9}\]

With this choice, we observe that only \(p\) needs to be tuned. In our experiments, we noticed that it suffices to evaluate a few values of \(p\), such as \(p\in\{\emptyset,2,3,4,5,6\}\), to obtain similar results as with the full optimization of \(\beta\) and \(p\).

Our proposed method with the heuristic choice of \(\beta\) is named \(p\)-NormSoftmax, while the method with full optimization of \(\beta\) is denoted \(p\)-NormSoftmax*. Variations with MCP replaced by other confidence estimators that take logits as input are discussed in Appendix A.

The rationale for each component of our method is given in the following subsections.

### Temperature Scaling

As mentioned before, standard TS can harm selective classification performance in some cases (Gail et al., 2023; Zhu et al., 2022); thus, we propose to optimize selective classification metrics directly. Since a single parameter needs to be tuned, this can easily be done via grid search.

Figure 2 shows how the behavior of different metrics as a function of the temperature \(T\) for a ViT-H-4 (Dosovitskiy et al., 2021) model evaluated on ImageNet. In this case, optimizing NLL can lead to better, but not optimal, selective classification performance, measured in terms of AURC and AUROC. Also, it can be seen that optimizing ECE does not necessarily help, illustrating our point that these two problems should be treated independently. Finally, it can be seen that AURC and AUROC exhibit practically identical behavior with temperature, suggesting that they are equally good to be used as the objective.

### \(p\)-Normalization

A natural way to extend TS is to allow for an input-dependent temperature (A. Balanya et al., 2023):

\[\mathbf{z}^{\prime}=\frac{\mathbf{z}}{T(\mathbf{z})}. \tag{10}\]

We propose to use \(T(\mathbf{z})=\|\mathbf{z}-\mu(\mathbf{z})\|_{p}/\beta\), so that high-norm inputs are penalized reducing the confidence of the corresponding predictions. This idea is inspired by (Wei et al., 2022), which uses \(p=2\) in the context of model training.

Wei et al. (2022) argued that, as training progresses, a model will tend to become overconfident on correctly classified training samples by increasing \(\|\mathbf{z}\|_{2}\), a phenomenon that they confirmed
experimentally. Here we remark that their argument holds unchanged for any \(p\), as nothing in their analysis requires \(p=2\). On the other hand, as discussed in Jiang et al. (2023), this argument is more compelling when the logit vector \(\mathbf{z}\) is centralized, since a non-zero mean has no impact on the training loss but affects the \(p\)-norm.

When applying \(p\)-normalization as a post-hoc method, we expect a similar effect: if the model has become too overconfident (through high \(p\)-norm) on input regions that appear as incorrect predictions on the test set, then \(p\)-normalization may reduce this overconfidence, improving selective classification performance. Otherwise, \(p\)-normalization may not help, so we keep the option of \(p=\emptyset\) and recover TS as a special case.

### A Heuristic for Choosing \(\beta\)

Intuitively, we should penalize predictions whose \(p\)-norm is too high. But high compared to what? We propose to use the expected \(p\)-norm of logits as a reference point, choosing

\[\beta=E[\|\mathbf{z}\|_{p}]. \tag{11}\]

This implies that \(T(\mathbf{z})=\|\mathbf{z}\|_{p}/E[\|\mathbf{z}\|_{p}]\) and thus \(\|\mathbf{z}^{\prime}\|_{p}=\beta=E[\|\mathbf{z}\|_{p}]\), so the expected \(p\)-norm of logits does not change after \(p\)-normalization. Moreover, we have \(E[T(\mathbf{z})]=1\). This can be interpreted as trying to change the logits as little as possible, since most of the logits will have their temperature approximately unchanged.

More details and results of the heuristic are presented in Appendix B.

## 5 Experiments

All the experiments2 regarding the proposed method and the subsequent investigations were conducted using the open-source library PyTorch (Paszke et al., 2019) and all of its provided pre-trained classifiers on ImageNet (Deng et al., 2009). Additionally, some models of the Wightman (2019) repository were utilized, particularly the ones highlighted by Galil et al. (2023). The list of the models, together with all the results per model are presented in Appendix E. In total, 84 ImageNet models were used for experiments. The validation set of ImageNet was randomly split into 5000 hold-out images for post-hoc optimization and 45000 for tests and comparisons. Investigations on the stability of this split are presented in Section C.

Footnote 2: Code can be found at [https://github.com/lfpc/pNormSoftmax](https://github.com/lfpc/pNormSoftmax)

Unless specified otherwise, we always use AURC as the objective when optimizing \(p\)-NormSoftmax.

### Comparison of methods

In Figure 3, we shown an example of the RC curves of the proposed methods for a ResNext101-32x8d (Xie et al., 2017). It can be seen that, while standard TS (TS optimizing NLL) outperforms the baseline, optimizing the AURC directly (TS-AURC) achieves better results. Moreover, it can be noted that \(p\)-NormSoftmax leads to even better performance which practically identical to that of \(p\)-Normalization*.

Footnote 3: footnotemark:

Indeed, the conclusion that full optimization of \(\beta\) is unnecessary when using the proposed heuristic (while always optimizing \(p\)) was observed for all considered models. The average AUROC gain of \(\beta\) optimization with respect to the heuristic is 0.0002, while the maximum value across all the analyzed models is 0.0019. These gains are imperceptible in the RC curve and may be considered negligible. The results for all the evaluated models are summarized in Table 1 and presented with more details in Appendix E.

Figure 3: RC curves of the proposed methods, the baseline and standard TS for a ResNext101-32x8d
Similar results are observed for CIFAR-100 (Krizhevsky, 2009) and are presented in Appendix F.

One important aspect of post-hoc methods is its data efficiency (Zhang et al., 2020), i.e., the efficiency of the method in learning with few data. Appendix C presents experiments when a fraction of the hold-out set is used, and lead us to conclude that the proposed \(p\)-NormSoftmax is extremely data efficient, converging to the optimal with few samples (\(<2000\) for ImageNet).

### Comparison of models

Galil et al. (2023) showed that some models are much better than others in selective classification. An example is shown in the left figure of Figure 1. Although the EfficientNet v2 XL (Tan and Le, 2021) has better accuracy than the ViTB/32 SAM (Chen et al., 2022), the latter is better in identifying misclassification and, thus, in most of the RC curve. However, the figure in the right shows that, after \(p\)-NormSoftmax optimization, the ViTs have negligible gain, while the EfficientNet has a huge one, hence becoming better in the RC curve than the ViTB/32 SAM.

In figures 3(a) and 3(b), the AURC and AUROC are presented with respect to the accuracy for each model. It can be seen that, while for the baseline there are models with higher accuracy but worse (higher) AURC than others, this does not happen after the models are optimized with \(p\)-NormSoftmax. The Spearman's correlation between the AURC and the accuracy goes from 0.9169 to 0.9992, indicating that, while not the case for the baseline, the selective classification performance of the optimized models is almost entirely determined by its accuracy at full coverage.

We can also observe that, after the optimization, the AUROC for all models lies within the range \([0.8421,0.8859]\). This small range suggests that all models are at roughly the same level of misclassification detection, although we can still see some dependency on accuracy (better predictive models are slightly better at predicting their own failures).

### Robustness to distribution shift

Up to this point, the presented results have been evaluated utilizing the validation set of ImageNet. Generally, this set is considered to have a data distribution similar to that of the training set. However, a reliable model must also be robust for dataset shifts (Ovadia et al., 2019). For evaluating a model's performance under data shift, we evaluate our methods on ImageNet-C (Hendrycks and Dietterich, 2018), which consists in 15 different corruptions of the ImageNet's validation set. We follow the standard approach for evaluating robustness with this dataset, which is to use it only for inference;

\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{2}{c}{AUROC [\%]} & \multicolumn{2}{c}{AUROC [x100]} \\ \hline Method & Mean & Max & Mean & Max \\ \hline TS-AURC & 12.65 & 44.32 & 1.7 & 9.32 \\ \(p\)-NormSoftmax & 15.9 & 48.46 & 2.61 & 10.60 \\ \(p\)-NormSoftmax* & 16.02 & 48.52 & 2.63 & 10.65 \\ \hline \hline \end{tabular}
\end{table}
Table 1: AUROC and AURC gains for ImageNet. AURC gains are calculated as the reduction of AURC in relation to the baseline. For both, the higher the better.

Figure 4: AUROC and AUROC of all ImageNet models with respect to their accuracy. \(\rho\) is the Spearman’s correlation between the metric and the corresponding accuracy and the color indicates the valued of \(p\) that optimizes each model.
thus, the post-hoc methods are optimized using only the 5000 hold-out images from uncorrupted ImageNet validation dataset.

Generally, classifiers lose accuracy in the presence of data shift. Hence, we use SAC as performance metric, with the target accuracy chosen as the accuracy of the model on ImageNet validation data at full coverage. Table 2 shows these results when \(p\)-NormSoftmax is applied to a ResNet-50 (He et al., 2016). We can see that \(p\)-NormSoftmax enhances the model's performance in selective classification under data shift at all corruption levels.

### When--and why--is \(p\)-NormSoftmax beneficial?

From the evaluated results (see Figure 3(b)), it can be noticed that, while for some models the MSP baseline is a poor confidence estimator and the \(p\)-NormSoftmax method yields exceptional AUROC gains, for some other the baseline is already a seemingly optimal selective mechanism. From these results, one can ask the question: what makes some models have inferior baselines? Experiments regarding the nature of these models were conducted and presented in Appendix D, along with possible explanations. In summary, models generating logits with high average norms tend to be the best ones in misclassification detection, while the ones with low average norms exhibit the largest gains when \(p\)-NormSoftmax is applied.

## 6 Conclusion

We considered the problem of selective classification for deep neural networks. In order to improve the selective mechanism for a given trained model, we proposed \(p\)-NormSoftmax, a post-hoc method for enhancing misclassification detection of neural network classifiers. Our method achieves an improvement in AURC of 16% on average when compared to the baseline for the evaluated classifiers trained on ImageNet, reaching almost 50% for some specific models.

Furthermore, our analysis revealed that, after implementing \(p\)-NormSoftmax, the models exhibited similar levels of misclassification performance. This finding results in a model's selective classification performance being almost completely determined by its accuracy at full coverage, and suggests that the previous observations regarding different performance between models' selective performance are mostly due to the use of sub-optimal confidence estimators. Additionally, \(p\)-NormSoftmax exhibit impressive data efficiency, due to the fact that a single parameter needs to be tuned. Moreover, the method achieves satisfactory gains for selective classification under data shift. It is also worth mentioning that our method is compatible with classifiers constructed directly for improving confidence estimation, including ensembles, specific architectures and models with specific training routines.

Finally, we point out some possible reasons and initial investigations on why and in which circumstances \(p\)-NormSoftmax achieves gains. For future work, we intend to explore more deeply why post-hoc normalization can lead to improved selective mechanisms and to evaluate our method on different tasks.

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & \multicolumn{6}{c}{Corruption level} \\ \cline{3-8}  & Method & 0 & 1 & 2 & 3 & 4 & 5 \\ \hline Accuracy [\%] & - & 80.86 & 68.56 & 60.03 & 51.85 & 39.44 & 27.09 \\ \hline Coverage & Baseline & 100 & 75.97 & 56.79 & 41.43 & 21.65 & 9.09 \\ (SAC) [\%] & TS-AURC & 100 & 77.13 & 60.51 & 45.49 & 27.41 & 13.32 \\  & \(p\)-NormSoftmax & 100 & 78.49 & 62.35 & 47.63 & 29.59 & 15.62 \\  & \(p\)-NormSoftmax* & 100 & 78.52 & 62.39 & 47.76 & 29.67 & 15.66 \\ \hline \hline \end{tabular}
\end{table}
Table 2: \(p\)-NormSoftmax applied to a ResNet-50 under dataset shift. The target accuracy is the one achieved for corruption level 0 (i.e., 80.86%).