induces a ${\bf\Sigma}^{(\alpha)}_{2}$-code in $L$ for $A$. ∎

###  5.3. $\Sigma^{1}_{2}$ sets

While any $\Delta^{1}_{2}$ Borel set with a Borel${}^{({\char
60\relax}\omega_{1})}$code in $L$ already has one in $L_{\tau}$, this is not
the case for $\Sigma^{1}_{2}$ sets.

######  Proposition 5.10.

There exists a $\Sigma^{1}_{2}$ Borel set of rank $\tau$ with a Borel(τ) code
in $L$.

###### Proof.

The proof is similar to Proposition 4.18. We define a $\Pi^{1}_{2}$ set $A$
Borel set of rank $\tau$ with a Borel(τ) code in $L$. It collects codes for
least elements $\alpha$ of $\Pi_{1}$-definable subsets of $\mathrm{WO}$
together with a real witnessing that $\alpha$ is least. In detail, let
$\varphi(n,x)$ be a universal $\Pi_{1}$-formula, i.e., $\varphi(n,x)$ is
$\Pi_{1}$ and $\langle\varphi_{n}(x)\mid n\in\omega\rangle$ enumerates all
$\Pi_{1}$-formulas with one free variable, where
$\varphi_{n}(x)=\varphi(n,x)$. $A$ denotes the set of
$(x,y,n)\in\mathrm{WO}^{2}\times\omega$ such that, $\alpha_{x}<\alpha_{y}$,
$\varphi_{n}(\alpha_{x})$ holds, $L_{\alpha_{y}}$ sees that
$\varphi_{n}(\beta)$ fails for all $\beta<\alpha_{x}$, but no
$\gamma<\alpha_{y}$ sees this. Note that if $\varphi_{n}(x)$ defines a
nonempty subset $B$ of $\mathrm{WO}$, then $\alpha_{x}\leq\min(B)$ for all
$x$, $y$ with $(x,y,n)\in A$. Therefore $\alpha_{x}<\tau$ and
$\alpha_{y}<\sigma_{\alpha_{x}}\leq\tau$ by Lemma 3.3. Thus $A$ is the union
of slices of the form $\mathrm{WO}_{\alpha}\times\mathrm{WO}_{\beta}$ for
unboundedly many $\alpha,\beta<\tau$. Since $\mathrm{WO}_{\leq\gamma}$ is not
$\Pi^{0}_{2\cdot\delta}$ if $\gamma=\omega^{\delta}$ [23],525252See [12, Lemma
1.3]. it follows that the Borel rank of $A$ equals $\tau$. ∎

##  6\. Conclusion and open questions

In Section 4, we constructed various examples where the least rank on a
$\Pi^{1}_{1}$ or $\Sigma^{1}_{2}$ set has a certain minimal countable length
such as $\sigma$, $\sigma_{\sigma}$ etc. Assuming $\omega_{1}^{L}=\omega_{1}$,
one can similarly obtain countable sets with a unique length of ranks. These
phenomena deserve to be explored further: at precisely which countable
ordinals does a new minimal length appear? Assuming
$\omega_{1}^{L}=\omega_{1}$, which ordinals appear as the unique length of
ranks on a set?

Proposition 4.20 shows that there is no provable bound for the lengths of
countable ranks on $\Pi^{1}_{1}$ Borel sets of fixed rank. Is this also true
assuming large cardinals? Is there a result in the converse direction: can one
compute a bound for Borel ranks of $\Pi^{1}_{1}$ sets that admit ranks of a
fixed countable length?

In Section 5, we mentioned our main open question about $\Delta^{1}_{2}$ Borel
sets:

######  Question 6.1.

Does every absolutely $\Delta^{1}_{2}$ Borel set have a Borel${}^{({\char
60\relax}\omega_{1})}$-code in $L$?

Stern proved this assuming $\omega_{1}$ is inaccessible in $L$. The only other
similar result known to us is Louveau’s separation theorem, but his proof
technique fails here. Stern’s result motivates us to ask for a similar
generalisation of Theorem 4.17:

######  Question 6.2.

Does a $\Sigma^{1}_{2}$ Borel set have a Borel${}^{({\char
60\relax}\omega_{1})}$-code in $L_{\tau}$ if and only if it admits a countable
$\Sigma^{1}_{2}$-rank?

For instance, one can ask if this holds at the first and second levels of the
Borel hiearchy. It is not clear if ranks are strong enough to guarantee the
existence of a Borel${}^{({\char 60\relax}\omega_{1})}$-code in $L_{\tau}$.
However, sets that admit ranks of the following form come with a Borel
definition induced by the layers. Call a $\Sigma^{1}_{2}$-rank on a set $A$
_uniform_ if the initial segments are uniformly $\Sigma^{1}_{1}$ and
$\Pi^{1}_{1}$ in the sense that there are $\Sigma^{1}_{1}$ and $\Pi^{1}_{1}$
formulas $\varphi(x,y)$ and $\psi(x,y)$ such that for each
$y\in\mathrm{WO}_{\alpha}$, the statement that $x$ has rank at most $\alpha$
is equivalent to both $\varphi(x,y)$ and $\psi(x,y)$ and moreover, these
formulas define a subset of $A$ for any fixed $y$ with $\mathrm{otp}(y)$ in
the rank’s range. It is easy to see that ranks induced by infinite time
computations satisfy this property.

######  Question 6.3.

Does a $\Sigma^{1}_{2}$ set have a Borel${}^{({\char
60\relax}\omega_{1})}$-code in $L_{\tau}$ if and only if it admits a uniform
countable $\Sigma^{1}_{2}$-rank?

Various other natural questions arise regarding the results in Sections 4 and
5. What are the lengths of $\Pi^{1}_{1}$\- and $\Sigma^{1}_{2}$-ranks on open
and closed sets? Do the suprema of $L$-levels of countable $\Pi^{1}_{2}$ sets
in $L$ and that of Borel ranks of $\Sigma^{1}_{2}$ Borel sets with a
Borel${}^{({\char 60\relax}\omega_{1})}$code in $L$ equal $\tau$? Theorem 4.17
(d) suggests to ask whether every constructible $\Pi^{1}_{2}$ singleton
contained in a countable $\Pi^{1}_{1}$ set, or otherwise how one can
characterise $\Pi^{1}_{2}$ singletons with this property.

Furthermore, we ask for generalisations of the present results to all
projective levels. Firstly, one can ask whether the periodic pattern in Figure
2 repeats itself at the next level:

######  Question 6.4.

Assume $0^{\\#}$ exists. Are the suprema of the values in the third column of
Figure 2 equal?

Assuming the axiom of projective determinacy (PD), one can hope for results
analogous to Theorem 1.1 and those shown in Figure 2 for all projective
levels. At the level of $\Pi^{1}_{3}$ sets, we suggest to combine the proofs
in this paper with techniques from inner model theory and replace $L$ by
$M_{1}$, an inner model with a Woodin cardinal.

######  Question 6.5.

Assume PD. What is the supremum of lengths of countable $\Pi^{1}_{3}$-ranks?

Studying the lengths of projective prewellorders further suggests to study the
related problem of proving basis theorems in order to find representatives for
equivalence classes of prewellorders as in [4, 21].

## References

  * BF [95] Ferenc Beleznay and Matthew Foreman.  The collection of distal flows is not Borel.  American Journal of Mathematics, 117(1):203–239, 1995.
  * BP [69] George Boolos and Hilary Putnam.  Degrees of unsolvability of constructible sets of integers.  The Journal of Symbolic Logic, 33(4):497–513, 1969.
  * CSW [22] Merlin Carl, Philipp Schlicht, and Philip Welch.  Decision times of infinite computations.  Notre Dame Journal of Formal Logic, 63(2):197–212, 2022.
  * Hjo [93] Greg Hjorth.  Thin equivalence relations and effective decompositions.  The Journal of Symbolic Logic, 58(4):1153–1164, 1993.
  * Hjo [96] Greg Hjorth.  $\Pi^{1}_{2}$ Wadge degrees.  Annals of Pure and Applied Logic, 77(1):53–74, 1996.
  * Hjo [10] Greg Hjorth.  Vienna notes on effective descriptive set theory and admissible sets, 2010.  Available at http://www.math.uni-bonn.de/people/logic/events/young-set-theory-2010/Hjorth.pdf.
  * HL [00] Joel David Hamkins and Andy Lewis.  Infinite time Turing machines.  The Journal of Symbolic Logic, 65(2):567–604, 2000.
  * Jec [03] Thomas J. Jech.  Set Theory.  Springer Monographs in Mathematics. Springer, 2003.
  * Kec [12] Alexander Kechris.  Classical descriptive set theory, volume 156.  Springer Science & Business Media, 2012.
  * KL [87] Alexander S. Kechris and Alain Louveau.  Descriptive set theory and the structure of sets of uniqueness, volume 128.  Cambridge University Press, 1987.
  * KL [19] Vladimir Kanovei and Vassily Lyubetsky.  Borel OD sets of reals are OD-Borel in some simple models.  Proceedings of the American Mathematical Society, 147(3):1277–1282, 2019.
  * KMS [89] Alexander S. Kechris, David Marker, and Ramez L. Sami.  $\Pi^{1}_{1}$ Borel sets.  The Journal of Symbolic Logic, 54(3):915–920, 1989.
  * Koe [05] Peter Koepke.  Turing computations on ordinals.  Bulletin of Symbolic Logic, 11(3):377–397, 2005.
  * Kra [82] Evangelos Kranakis.  Reflection and partition properties of admissible ordinals.  Annals of Mathematical Logic, 22(3):213–242, 1982.
  * Lou [80] Alain Louveau.  A separation theorem for $\Sigma^{1}_{1}$ sets.  Transactions of the American Mathematical Society, 260(2):363–378, 1980.
  * Mil [83] Arnold W. Miller.  Mapping a set of reals onto the reals.  The Journal of Symbolic Logic, 48(3):575–584, 1983.
  * Mos [09] Yiannis N. Moschovakis.  Descriptive set theory, second edition, volume 155 of Mathematical Surveys and Monographs.  AMS, 2009.
  * Rat [17] Michael Rathjen.  The higher infinite in proof theory.  In Logic Colloquium, volume 95, 2017.
  * Sam [84] Ramez L. Sami.  On $\Sigma^{1}_{1}$ equivalence relations with Borel classes of bounded rank.  The Journal of Symbolic Logic, 49(4):1273–1283, 1984.
  * [20] Ralf Schindler.  Set Theory.  Universitext. Springer-Verlag, 2014.
  * [21] Philipp Schlicht.  Thin equivalence relations and inner models.  Annals of Pure and Applied Logic, 165(10):1577–1625, 2014.
  * SS [14] Ralf Schindler and John R. Steel.  The core model induction.  2014\.  Unpublished.
  * [23] Jacques Stern.  Evaluation du rang de Borel de certains ensembles.  Unpublished.
  * Ste [84] Jacques Stern.  On Lusin’s restricted continuum problem.  Annals of Mathematics, 120(1):7–37, 1984.
  * Ste [08] John R. Steel.  Scales in $L(\mathbb{R})$.  In A. S. Kechris, B. Löwe, and J. R. Steel, editors, Games, Scales and Suslin Cardinals, The Cabal Seminar, Volume I. Cambridge University Press, 2008.
  * Wes [14] Linda Brown Westrick.  A lightface analysis of the differentiability rank.  The Journal of Symbolic Logic, 79(1):240–265, 2014.
  * Wes [19] Linda Brown Westrick.  Topological completely positive entropy is no simpler in $\mathbb{Z}^{2}$-SFTs.  2019\.  arXiv preprint 1904.11444.
  * Zem [11] Martin Zeman.  Inner models and large cardinals.  de Gruyter, 2011.


Looking at the posterior: on the origin of uncertainty in neural-network classification

H. Linander^1, O. Balabanov^2, H. Yang^1, B. Mehlig^1

$^{1}$ Department of Physics, Gothenburg University, 41296 Gothenburg, Sweden
$^{2}$ Department of Physics, Stockholm University, 10691 Stockholm, Sweden

Bayesian inference can quantify uncertainty in the predictions of neural networks using posterior distributions for model parameters and network output. By looking at these posterior distributions, one can separate the origin of uncertainty into aleatoric and epistemic.  We use the joint distribution of predictive uncertainty and epistemic uncertainty to quantify how this interpretation of uncertainty depends upon model architecture, dataset complexity, and data distributional shifts in image classification tasks. We conclude that the origin of uncertainty is subjective to each neural network and that the quantification of the induced uncertainty from data distributional shifts depends on the complexity of the underlying dataset. Furthermore, we show that the joint distribution of predictive and epistemic uncertainty can be used to identify data domains where the model is most accurate. To arrive at these results, we use two common posterior approximation methods, Monte-Carlo dropout and deep ensembles, for fully-connected, convolutional and attention-based neural networks.

§ INTRODUCTION

Recently there has been an intensive effort towards better understanding uncertainty of neural-network predictions [Gawlikowski et al., 2021, Abdar et al., 2021].
To quantify this uncertainty, and to identify its sources, is of key significance in many applications of machine-learning algorithms using neural networks,
from real-time predictions to active learning
[Gustafsson et al., 2020]. Bayesian inference provides a theoretical
framework to reason about uncertainties by quantifying probability
distributions of the neural-network output given the model architecture and training
data [Neal, 2012]. More precisely, given a neural network with parameters $\theta$ and a training dataset $\mathcal{D} = \left\{(x_{1},y_{1}), (x_{2}, y_{2}),\cdots\right\}$ of pairs $(input, target)$,
Bayesian arguments determine a distribution over the neural-network parameters $p(\theta | \mathcal{D})$ [MacKay, 1992]. This so-called posterior distribution tells us
the probability of different model parameters given the training dataset. Using
this posterior distribution for the parameters, a corresponding posterior distribution of the neural-network predictions can be derived, the posterior predictive distribution $p(y|x, \mathcal{D})$. The Bayesian posterior predictive distribution can be computed conditioned on a particular input, either previously unseen or contained in the training dataset. The posterior predictive distribution can be used to obtain a mean prediction, and more importantly to draw conclusions regarding the uncertainty of the model and the data.

In order to characterize the posterior predictive distribution, it is common to look at derived quantities, such as different measures of entropy, to quantify the uncertainty associated with predictions [Gawlikowski et al., 2021]. A fruitful distinction can be drawn between uncertainty stemming from the model or from the data, captured by classifying uncertainty as either epistemic or
aleatoric [Kiureghian and Ditlevsen, 2009].

The usefulness of this classification of uncertainty can be seen when asking
specific questions such as “Should we trust the predictions of our neural network in this
domain?”. A region with low epistemic uncertainty instills confidence that
the model has seen enough data to make a valid prediction but might still have
a high aleatoric uncertainty due to, for example, the nature of the input. On the other
hand, a region with high epistemic uncertainty is not expected to provide
meaningful predictions. In terms of active learning [Cohn et al., 1996], answers to questions such as “What data should we
annotate to improve our model?” requires us to know of the origin of uncertainty: high epistemic uncertainty corresponds to regions where the model can improve by providing more training data.

Bayesian modeling makes the distinction between these sources clear. The posterior probability $p (\theta | \mathcal{D} \nobracket)$ of the neural-network parameters $\theta$ given the
observations $\mathcal{D} $ contains the epistemic uncertainty, whereas the
likelihood $p (y | \theta, x \nobracket)$ given model parameters
$\theta$ and input data $x$ contains the aleatoric part. The predictive
distribution, i.e. the marginalization over model parameters $\theta$ using the posterior $p(\theta | \mathcal{D})$

\begin{align}
p (y | \mathcal{D}, x \nobracket) & = \int_{\theta} p (y | \theta, x
\nobracket) p (\theta | \mathcal{D} \nobracket) d \theta, \label{eq:bayesian_mean}
\end{align}

contains, through the two factors in the integrand, a mixture of aleatoric and
epistemic uncertainty.
In very simple examples, the posterior can be computed in closed form (see <ref> for an example) or with much computational
effort [Izmailov et al., 2021], but usually we must rely on approximations introducing deviations from the true posterior. A computationally tractable approximation for the posterior predictive distribution is dropout inference [Kendall and Gal, 2017], where the posterior is sampled by applying dropout in multiple inference
passes for the same input, effectively sampling from different parameter
configurations. It can be shown that this yields an approximation to the Bayesian
posterior [Gal and Ghahramani, 2016]. A more computationally demanding, but
conceptually simple, alternative is deep ensembling
[Lakshminarayanan et al., 2017] where a neural network is trained multiple times
starting from different parameter initializations, creating an ensemble distribution to provide an approximation to the true
Recent results show that deep ensembles provide a good approximation to the Bayesian posterior [Izmailov et al., 2021, Gustafsson et al., 2020, Hoffmann and Elster, 2021].

Although uncertainty quantification for neural networks in terms of aleatoric and epistemic uncertainty has been the subject of much work, application and analysis of the different uncertainties continues to be an active field of research. The large set of possible neural-network architectures and data domains present many concrete settings where an understanding of the uncertainty content is important for a wide range of applications. Some of the most important open questions in this context are: How does the separation of uncertainty into aleatoric and epistemic parts depend on the neural-network architecture? For a given model, how does the complexity of the dataset impact the perceived uncertainty for different forms of data distributional shifts? For which models and data domains can uncertainty quantification help us improve our model predictions?

To answer these questions we evaluate uncertainty quantification using the joint distribution of total predictive uncertainty and epistemic uncertainty. Our results show that different neural-network architectures can disagree about the aleatoric and epistemic uncertainty of the same data. In other words, the origin of uncertainty is not objective. Using the joint distribution of total predictive uncertainty and epistemic uncertainty, we quantify how the approximate posteriors of three common neural-network architectures for image classification differ from each other and how they depend on data distributional shifts in the form of impulse noise [Awad, 2019] for MNIST [LeCun et al., 2010] and CIFAR [Krizhevsky et al., 2009]. Furthermore, we show that the joint distribution of total predictive uncertainty and epistemic uncertainty can be used to identify data domains where a model is accurate, and that it is difficult to do so using the predictive or epistemic uncertainty alone. We use two common approximation methods for the posterior predictive distribution, dropout inference and deep ensembles, to evaluate predictive and epistemic uncertainty.

§ CONTRIBUTIONS

* We use the joint distribution of predictive entropy and mutual information of the neural-network parameter posterior to quantify the variability of uncertainty measures over different model architectures, data distributional shifts and posterior approximation methods.

* We show that neural networks with different architectures can disagree about the origin of uncertainty for data distributional shifts in image classification tasks.

* We show that the complexity of the dataset impacts the sensitivity of the uncertainty measures to data distributional shifts.

* We quantify the correlation between model accuracy and the joint distribution of total and epistemic uncertainty.

* We evaluate our results by using two different posterior approximation schemes, Monte-Carlo dropout and deep ensembles, in order to quantify the extent to which our conclusions depend on the posterior approximation.

§ RELATED WORK

Ref. [Depeweg et al., 2018] uses both uncertainty measures separately in connection with active learning, concluding that epistemic uncertainty provides a better selection criterion than the total predictive uncertainty.  In the context of reinforcement learning Ref. [Depeweg et al., 2018] use a combination of aleatoric and epistemic uncertainty to find balanced policies.  The joint distribution, architecture dependence, and dataset dependence is not considered. In Ref. [Ghoshal and Tucker, 2020], the correlation between accuracy and uncertainty is quantified for image classification. The correlation with accuracy is evaluated for total predictive uncertainty and epistemic uncertainty separately, the architecture and dataset dependence is not discussed. The joint distribution of aleatoric and epistemic uncertainty is considered in Ref. [Ghoshal et al., 2021], where the correlation between predictive probabilities and the joint uncertainty distribution is quantified in the context of medical image semantic segmentation. Using a fixed residual U-Net architecture and  datasets for semantic segmentation it was shown that there is a correlation between predictive probabilities and the uncertainty measures, and that accuracy for their semantic segmentation model on the considered datasets shows correlation with epistemic uncertainty. Ref. [Ghoshal et al., 2021] conclude that open questions include the effect of data distributional shifts, model architecture, and data modality on the quality of uncertainty quantification. These are in line with two of our target questions on how the perceived origin of uncertainty depends on model architecture and how dataset complexity impacts uncertainty quantification. Quantifying uncertainty under data distributional shifts was investigated in Ref. [Ovadia et al., 2019] where accuracy, calibration and entropy of the posterior predictive is evaluated for different shifts introduced in Ref. [Hendrycks and Dietterich, 2019]. Here, the epistemic uncertainty is not considered and the model dependence of the uncertainty measures is not analyzed.

§ BACKGROUND

§.§ Bayesian inference

An artificial neural network $f$ with parameters $\theta \in \mathbb{R}$ can be seen as a map from input space $X$ the output space $Y$ [Mehlig, 2021], where we take $Y$ to be the space of distributions over possible outcomes so that the likelihood is given by $p(y| x, \theta) = f(y, x ; \theta)$.
Bayesian inference [Sivia and Skilling, 2006] allows us to reason about
uncertainty in terms of posterior distributions for the parameters of a model.
With a training dataset $\mathcal{D} \in X \times Y$ corresponding to observations $(x_t, y_t)$ with $t\in\{1\cdots N\}$, the posterior distribution for the neural-network parameters $p(\theta | \nobracket
\mathcal{D})$ can be calculated using Bayes theorem in terms of the likelihood as

\begin{equation}
p(\theta | \mathcal{D}) = \frac{p(\mathcal{D} | \theta)p(\theta)}{p(\mathcal{D})},
\end{equation}

where $p(\mathcal{D} | \theta)$ (assuming independent samples in the dataset $\mathcal{D}$) can be expressed as $p(\mathcal{D}|\theta) = \prod_{i=1}^{N} f(y_{i}, x_{i}; \theta)$, $p(\theta)$ the parameter prior and the evidence $p(\mathcal{D})$ is the marginalization over the parameter prior.

A prediction, or more generally a distribution over possible outputs, is computed using the posterior predictive distribution in (<ref>) by
marginalizing over the parameters.
Each parameter configuration is weighted by its posterior probability given the training
data. For large neural-network architectures, the integral over the parameter space $\theta$, which is
typically some subset of $\mathbb{R}^n$ with large $n$, is both analytically and computationally [Izmailov et al., 2021]
infeasible, see <ref> for a simple toy example where the Bayesian posterior is computed in closed form and evaluated numerically.

To implement Bayesian inference for neural networks, the
posterior needs to be approximated, as mentioned in the introduction. There are a number of different methods
available that range from computationally expensive Monte-Carlo simulations
[Neal, 2012] to more efficient dropout approximations
[Kendall and Gal, 2017] and simpler ensembling methods
[Lakshminarayanan et al., 2017]. The most accurate approximation to the true posterior available are the Hamiltonian Monte-Carlo methods. Through intensive computational efforts these have recently been used to compute the posteriors of larger convolutional neural networks such as a 20-layer ResNet [Izmailov et al., 2021]. The HMC computations show that simpler approximation schemes such as ensembling and variational inference can fail to accurately describe the true posterior, but that ensembles often provide more accurate posteriors than more advanced methods.

§.§ Dropout posterior

The posterior predictive distribution can be approximated using dropout [Kendall and Gal, 2017]. In this scheme, the
posterior predictive distribution for a sample $x$ is calculated as

\begin{align}
p(y| x, \mathcal{D}) & = \int_{\theta} f(y, x; \theta) p(\theta | \nobracket \mathcal{D}) \tmop{d\theta} \approx \frac{1}{N} \sum_{i = 0}^N f(y, x; \theta^{(i)}), \label{eq:ppd}
\end{align}

where $\theta^{(i)}$ are $N$ samples of parameters using dropout indexed by $i$, of a neural network trained with dropout [Srivastava et al., 2014] and $L^2$-regularization [Ng, 2004] and
$N$ is the ensemble size.

For a classification problem over a finite number of classes $c\in \{1, \cdots, M\}$, the variable $y$ is discrete, so that (<ref>) becomes

\begin{equation}
p(y=c|x, \mathcal{D}) \approx \frac{1}{N} \sum_{i = 0}^N f_{c}(x; \theta^{(i)}). \label{eq:dropout}
\end{equation}

The dropout approximation is
equivalent to variational inference when the prior distribution over the
weights together with the choice of distribution family $q (\theta)$ satisfies
certain conditions [Kendall and Gal, 2017, Folgoc et al., 2021]. Variational inference is itself an
approximation scheme to the true posterior, so that even if dropout approaches
variational inference for a particular choice of distributional family $q(\theta)$, it may still be a bad approximation to the true posterior.

§.§ Ensemble posterior

A second, frequently employed method to sample from the model space is to train an ensemble
[Lakshminarayanan et al., 2017] of $N$ identical neural networks using different
initial parameter values $\theta_{\tmop{initial}}^{(i)}$ sampled from some prior
distribution. Training these
neural networks to maximize the likelihood of the training data gives a set of
parameters $\theta^{(i)}$, that then provides an approximation to the Bayesian mean
by equation (<ref>), where $\theta^{(i)}$ are now the ensemble member parameters instead of the dropout samplings.
Here it is assumed that the likelihood of the minima attained by $\theta^{(i)}$ are
equally probable. Note that it is not clear a priori whether the minima $\theta^{(i)}$ are
degenerate, but for the regime of neural networks for visual perception
it is typically the case that they are not [Fort and Jastrzebski, 2019].

The difference between the dropout posterior and ensemble posterior lies in the way the parameters are sampled. For dropout, the parameters are sampled by dropout inference whereas for ensembles the different parameter sets are trained independently. The ensemble members are local maximums of the likelihood function whereas a dropout sampling tends to sample around such points [Folgoc et al., 2021].

§.§ Uncertainty quantification

For classification over a discrete set of $M$ classes, the entropy [Shannon, 1948] of the predictive distribution

\begin{equation}
H(p) = -\sum_{c=1}^{M}p( {c} )\log (p( {c} )), \label{eq:entropy}
\end{equation}

provides a measure of the information content and thus its uncertainty.

In terms of the dropout and ensemble approximations of the posterior predictive distribution in (<ref>), this takes the form

\begin{equation}
H\big(y| x, \mathcal{D}\big) = -\sum_{c=1}^{M}\left(\frac{1}{N}\sum_{ i=1 }^N f_{c}(x; \theta^{(i)})\right)\log \left(\frac{1}{N}\sum_{ i=1 }^N f_{c}(x; \theta^{(i)})\right), \label{eq:posterior_entropy_approx}
\end{equation}

where $H(y|x, \mathcal{D}) \equiv H\big(p(y| x, \mathcal{D})\big)$ is the entropy of the posterior predictive distribution, referred to here as total predictive uncertainty, and $H(\theta|\mathcal{D}, (x,y))$ is the entropy of the posterior given a training set containing the union $\mathcal{D} \cup (x, y)$.
Looking at the posterior predictive distribution in (<ref>), we see that the entropy of $p(y|x, \mathcal{D})$ in equation (<ref>), as a measure of uncertainty, contains contributions that are both epistemic and aleatoric.

Epistemic uncertainty stems from uncertainty in model parameters. This is captured by the shape of the posterior distribution of the model parameters. To quantify the epistemic uncertainty associated with a single data sample $x$ we can ask how this shape changes [MacKay, 1992].
One says that the epistemic uncertainty associated with a data sample
$x$ is measured by the expected change in entropy of the model parameter posterior distribution $p(\theta|\mathcal{D})$ when $x$ is added to the observations [MacKay, 1992],

\begin{align}
I(x) & = E_{p(y | x \nobracket, \mathcal{D})} [H
(\theta | \nobracket \mathcal{D}, (x, y)) - H (\theta | \mathcal{D}
\nobracket)], \label{eq:delta_H_theta}
\end{align}

where $H(\theta|\mathcal{D}) = H\big(p(\theta|\mathcal{D})\big)$.
The entropy difference in equation (<ref>) can be related to the more
easily computable entropy of the posterior predictive distribution [Houlsby et al., 2011]:

\begin{align}
I(x) & = H (y | x, \mathcal{D} \nobracket) - E_{p
(\theta | \mathcal{D} \nobracket)} [H (y | \theta, x \nobracket)],
\label{deltaH}
\end{align}

where $H (y | \theta, x \nobracket)$ is the entropy of the likelihood $p(y | \theta, x)$.
See <ref> for an illustration of this relation in a toy model.
The first term in (<ref>) is the entropy of the posterior predictive distribution given the dataset $\mathcal{D}$,
whereas the second term is the expected value of the likelihood entropy over the model posterior distribution. An equivalent way of arriving at equation (<ref>) is compute the
conditional mutual information $I(\theta, y | x, \mathcal{D})$ between the model parameters and target $y$ conditioned on the dataset $\mathcal{D}$ and the new sample $x$ [Smith and Gal, 2018], motivating our simplified notation $I(x)$ for the same quantity as a function of the new data sample $x$.

Using the approximate posteriors from the dropout and ensemble methods, the first term in (<ref>) is given by (<ref>) and the second term is given by

\begin{equation}
E_{p(\theta | \mathcal{D} \nobracket)} [H (y | \theta, x \nobracket)] = \frac{1}{N} \sum_{i=1}^{N}\sum_{c=1}^{M}f_{c}(x; \theta^{(i)})\log \left(f_{c}(x; \theta^{(i)})\right).\label{eq:expected_H}
\end{equation}

Together, equations (<ref>) and (<ref>) provide a concrete way to evaluate epistemic uncertainty as defined by (<ref>) in practice.

The epistemic uncertainty in (<ref>) is large when the posterior predictive entropy is large and the mean likelihood entropy is small. In terms of the ensemble members this corresponds to the situation where each member has a sharp distribution but they disagree about the mean. Large aleatoric uncertainty is ascribed to broad output distributions from the individual members $f(\cdot; \theta^{(i)})$, also implying a large posterior predictive entropy.
Since entropy is positive, both the term $H(y|x, \mathcal{D})$ and the expected entropy of the likelihood in (<ref>) are positive, and so the entropy difference in (<ref>) is bounded by the posterior entropy in (<ref>), resulting in the inequality

\begin{equation}
I(x) \leq H(x). \label{eq:bound}
\end{equation}

This means that a large epistemic uncertainty implies a large posterior predictive entropy. This can be seen by noting that a collection of sharp member distributions that disagree about the mean necessarily adds up to a broad mean distribution. One way to describe a sample with large epistemic uncertainty is that the model might fit the data well, but in many different ways. The parameters with high posterior probability could all result in sharp distributions, whereas the full posterior can be broad [Kirsch et al., 2019].

Since the epistemic uncertainty in (<ref>) measures the change in posterior entropy, a model can have some irrelevant parameters that contribute to a broad posterior distribution, but the change in posterior entropy can still be small for a given sample. If the parameter posterior for some irrelevant parameter is equally broad after we add sample $x$, then we do not want to consider this as a point of high epistemic uncertainty.

In summary, the entropy $H(y|x, \mathcal{D})$ of the posterior predictive distribution $p(y|x, \mathcal{D})$ in (<ref>) and the mutual information in (<ref>) quantify the total predictive uncertainty and epistemic uncertainty respectively.

§ METHODS

§.§ Datasets

Two datasets are used for all numerical experiments, MNIST [LeCun et al., 2010] and a
grayscale version of CIFAR10 [Krizhevsky, 2009]. To be able to compare relative shifts in uncertainty measures between the two datasets we choose to work with a grayscale version of CIFAR denoted CIFAR10G so that the data distributional shift acts in exactly the same way for both MNIST and CIFAR10G.
MNIST is an example of an image classification dataset with
minimal complexity, whereas CIFAR10 provides a more realistic data distribution for image classification.
The grayscale conversion for the RGB data from CIFAR10 is given by the standard BT.601 luminance $Y = 0.2989 r + 0.5870 g + 0.1140 b$.
We apply impulse noise [Awad, 2019], a common corruption present in digital images, to the original data sets (MNIST, CIFAR10G) where the strength of the perturbation is controlled by a noise parameter $\alpha$.
For “salted” noise with parameter $\alpha > 0$ a random sample of pixels of size $\alpha N_{\tmop{pixels}}$ are given the maximum value $1.0$, and for “peppered” noise with $\alpha < 0$ a
corresponding amount of pixels are set to $0$.
We pick  $N_{\alpha}$ distinct values between $\alpha_{\text{min}}=-0.3$ and $\alpha_{\text{max}}=0.3$.

MNIST consists of 60000 grayscale images with resolution $28{\times}28$. The original CIFAR10 dataset consists of 60000 RGB images with resolution $32{\times}32$ that we resample to a single grayscale channel.
See figure <ref> for examples of the different noise levels.

MNIST (left) and CIFAR10G (right) with impulse noise parametrized by
$\alpha$. Darker colors indicate a value closer to $0$. For CIFAR10G the original RGB image is shown in the right-most column. The CIFAR10 classes in the examples are, from top to bottom: airplane, automobile, bird, cat, deer.

§.§ Network architectures

Model     Parameters     MNIST     CIFAR10G

Dense     118k     $99.5\% \pm 0.07\%$    $45.8\% \pm 0.5\%$

CNN     836k     $99.9\% \pm 0.06$     $74.1\% \pm 0.95\%$

Swin     147k     $97.3\% \pm 0.1\%$     $66.2\% \pm 1.3\%$

Summary of the three different neural-network architectures used for comparisons. The accuracy for MNIST and CIFAR10G is over the validation datasets with standard deviation calculated for 10 separate trainings. The dense model uses 3 layers (128, 128, 10) with ReLU activations, CNN is model A in [Springenberg et al., 2014] and the transformer is Swin [Liu et al., 2021].

In order to quantify how the architecture affects the uncertainty estimates, we use three neural-network architectures: fully connected (dense), convolutional (cnn) and attention-based (swin) neural networks.
For the fully connected neural network we use a simple three-layer architecture with two 128-neuron hidden layers using ReLU activations.
The convolutional neural network is identical to model A in [Springenberg et al., 2014], a simple fully convolutional model with 5 layers using max-pooling for spatial down-sampling and ReLU activations.
Finally, we also use Swin-Transformer (swin) [Liu et al., 2021], a popular attention-based model for computer vision.
See Table <ref> for a summary of the model sizes and baseline accuracy on the target datasets.

§ RESULTS

§.§ Model architecture

Figure <ref> shows, for different neural-network architectures (columns) and posterior approximation methods (rows), the joint distribution of the total predictive uncertainty defined by equation (<ref>) and epistemic uncertainty as defined by equation (<ref>) with histograms in the $(H, I)$ plane. The joint distributions in Figure <ref> are evaluated on the test set of CIFAR10G shifted by impulse noise with noise parameter $\alpha=0.09$, corresponding to $9\%$ of pixels set to the maximum grayscale intensity.
The joint distribution allows us to separate the origin of uncertainty: a sample with large value of $I$ on the vertical axis has higher epistemic uncertainty, whereas a sample on the horizontal axis has no epistemic uncertainty. The dashed red diagonal indicates the bound on epistemic uncertainty imposed by equation (<ref>). In particular this means that a sample close to the epistemic bound is dominated by epistemic uncertainty. The mean and standard deviation of the joint distributions in Figure <ref> are summarized in Table <ref> together with prediction accuracies. For reference, the moments and accuracy on the unshifted test set is shown in Table <ref>.

In Figure <ref>, for the fully connected neural network (left column), convolutional neural network (middle column) and Swin model (right column), the joint distribution using ensembling (top) and dropout (bottom) show rough agreement. For CIFAR10G, the posterior of the convolutional neural network shows the most disagreement between the ensemble and dropout posterior, as can be verified numerically in <ref>.

The center column of panels in Figure <ref> shows the joint uncertainty distribution for the convolutional neural network. Here the perceived epistemic uncertainty is larger compared to the other models.
The right-most column of panels in Figure <ref> shows that the attention-based model exhibits similar epistemic uncertainty to the convolutional neural network in the middle column of panels. The attention-based model does however perceive a lower aleatoric uncertainty, although this is not accompanied by a significant increase in accuracy.

For MNIST, the joint distributions of predictive and epistemic uncertainty evaluated on the same distributional shift with $\alpha=0.09$ can be found in Figure <ref>. Table <ref> contains the moments of the joint distributions and the prediction accuracies. In Figure <ref> and <ref> we find that for MNIST , the joint distributions for ensemble (top row) and dropout (bottom row) of the dense (left column) and convolutional (center column) neural networks show better agreement, whereas the Swin model shows a larger discrepancy between the different posterior approximation methods.

§.§ Dataset

To evaluate the dependence of the uncertainty quantification on the training dataset we calculate the induced shift in joint distribution of predictive and epistemic uncertainty when we apply the same data distributional shift to neural networks trained on MNIST and CIFAR10G. In Figure <ref> the observed shift in joint distribution for the convolutional neural network is shown from $\alpha=0$ (top) to $\alpha=0.09$ (bottom) for both MNIST (left) and CIFAR10G (right). The shift in mean predictive and epistemic uncertainty is quantified in <ref>. For the dense and Swin model we observe a significantly smaller shift in both $H$ and $I$ on CIFAR10G compared to MNIST, whereas the convolutional neural network perceives a larger shift on CIFAR10G compared to MNIST.

Joint distribution $(H, I)$, corresponding to predictive uncertainty and epistemic uncertainty, for different neural networks trained on CIFAR10G and evaluated on noised data with noise parameter $\alpha=0.09$. For each model, the joint entropy distribution is shown for dropout (top) and ensemble (bottom). Inset in each frame is the noise level ($\alpha$), accuracy of the model ($acc$) on the noised data together with mean and standard deviation of both axes. The dashed diagonal indicates the bound on epistemic uncertainty from equation (<ref>).

Joint distribution $(H, I)$, corresponding to total and epistemic uncertainty, for different neural networks trained on MNIST and evaluated on noised data with noise parameter $\alpha=0.09$. Inset in each frame is the noise level, accuracy of the model on the noised data together with mean and standard deviation of both axes. The dashed diagonal indicates the bound on epistemic uncertainty from equation (<ref>).

Architecture     Posterior     Acc.     1cH (nat)     1c|I (nat)

Mean and standard deviation for predictive and epistemic uncertainty and accuracy for fully connected (dense), convolutional (cnn) and attention-based (swin) models with posterior approximations using ensembles and dropout evaluated on CIFAR10G without data distributional shift.

Architecture     Posterior     Acc.     1cH (nat)     1c|I (nat)

Mean and standard deviation for predictive and epistemic uncertainty for fully connected (dense), convolutional (cnn) and attention-based (swin) models with posterior approximations using ensembles and dropout for MNIST at fixed data distributional shift $\alpha=0.09$.

Architecture     Posterior     Acc.     1cH (nat)     1c|I (nat)

Mean and standard deviation for predictive and epistemic uncertainty for fully connected (dense), convolutional (cnn) and attention-based (swin) models with posterior approximations using ensembles and dropout evaluated on CIFAR10G at fixed data distributional shift $\alpha=0.09$.

Change in joint distribution $(H, I)$ from $\alpha=0$ (top) to $\alpha=0.09$ (bottom) on CIFAR10G (left) and MNIST (right) for the convolutional model.

Architecture     Dataset     1c$\alpha$     1c$\Delta \bar{H}$     1c|$\Delta \bar{I}$

(nat)     (nat)

Change in mean predictive and epistemic uncertainty, $\Delta \bar{H}$ and $\Delta \bar{I}$, for fully connected (dense), convolutional (cnn) and attention-based (swin) models from $\alpha=0$ to $\alpha=0.09$. Posterior approximation by ensembling.

§.§ Accuracy

Figures <ref> and <ref> show the accuracy of the predictions of three neural networks conditioned on entropy regions and posterior approximations evaluated on the union of all noise levels, including the uncorrupted test set.
We see that lower epistemic and aleatoric uncertainty does not always imply higher accuracy. The top middle panel of figure <ref> shows that for the convolutional neural network, for a fixed moderate predictive uncertainty, the accuracy is not monotonically increasing with epistemic uncertainty. On the other hand, the attention-based Swin architecture in the top right panel shows a monotonically increasing accuracy with decreasing epistemic uncertainty.

The dropout posterior struggles to separate the high and low accuracy regions. Comparing e.g. the two rightmost panels in figure <ref>, the ensemble posterior clearly separates a region of high accuracy whereas the dropout posterior mixes up high- and low accuracy samples. In the same figure the dropout posterior for the convolutional neural network also shows a degraded separation of high accuracy regions. For the simple fully connected neural network there is a more pronounced correlation between the predictive uncertainty and model accuracy.

Accuracy distribution in the $(H, I)$-plane for a union over noise levels $-0.27 \leq \alpha \leq 0.27$ evaluated on CIFAR10G. Prediction accuracy conditioned on the joint distribution of posterior predictive entropy $H$ and epistemic uncertainty $I$ is shown for the ensemble posterior (top) and dropout posterior (bottom).

Accuracy distribution in the $(H, I)$-plane for a union over noise levels $-0.27 \leq \alpha \leq 0.27$ evaluated on MNIST. Prediction accuracy conditioned on the joint distribution of posterior predictive entropy $H$ and epistemic uncertainty $I$ is shown for the ensemble posterior (top) and dropout posterior (bottom).

§ DISCUSSION

§.§ Model architecture

There are three points we want to make regarding how uncertainty quantification depends on model architecture.

First, the origin of uncertainty is not objective. Table <ref> shows that there are significant differences in the perceived origin of uncertainty between the different model architectures for in-domain data. For CIFAR10G, the fully connected neural network perceives a higher degree of aleatoric uncertainty, and low epistemic uncertainty compared to the convolutional and attention-based model. This shows that the origin of uncertainty depends on the model architecture: the fully connected neural network struggles to express the relationship between inputs and classes of CIFAR10G, and thus perceives a higher degree of aleatoric uncertainty. Even though the model has low accuracy, the dropout posterior and ensemble posterior both show a low epistemic uncertainty. Thus we do not expect to be able to salvage the accuracy of the fully connected neural network by increasing the size of the dataset.
For CIFAR10G, Figure <ref> and Table <ref> show that for a moderate data distributional shift, the fully connected neural network perceives a higher degree of aleatoric uncertainty and low epistemic uncertainty compared to the convolutional and attention-based models.

Second, the different model posteriors agree about the origin of uncertainty when the dataset complexity is low. In the case of MNIST, Figure <ref> and Table <ref> show that all neural-network architectures considered here show rough agreement between the joint distributions of predictive and epistemic uncertainty. For the considerably less complex dataset of MNIST, as evident by the higher accuracies in Table <ref>, all three architectures can easily capture the features necessary. Hence, they also agree about the origin uncertainty. The simplicity of the dataset is also reflected in that the different posterior approximations show more agreement compared with CIFAR10G, with the exception of the attention-based model.

Third, the accuracy of posterior approximations depends on architecture. The two posterior approximations, ensemble and dropout, show particularly good agreement regarding the epistemic uncertainty of the fully connected model as shown in Table <ref>. On CIFAR10G, for the convolutional and attention-based neural networks there are larger differences between the approximations, and the posteriors for the convolutional model show the least agreement. Thus, a posterior approximation method that is robust for one architecture, might not perform as well for another.

In summary, besides looking for performant architectures, we should also search for architectures and posterior approximations that provide robust uncertainty quantification for a given data domain.

§.§ Dataset

We discuss two points concerning how the posteriors depend on the dataset.

First, the relative change in perceived uncertainty under a fixed data distributional shift depends on the dataset, and varies with model architecture. By comparing the change in the joint distribution under a fixed data distributional shift in Figure <ref> we observe that a given model architecture perceive the same data distributional shift in different ways, depending on the underlying training dataset. We find an asymmetry in shift sensitivity: only the convolutional model shows a stronger sensitivity of the predictive and epistemic uncertainty when evaluated on CIFAR10G compared to MNIST, as evident by Table <ref> and Figure <ref>.
Furthermore, the fully connected model perceives the impulse noise on CIFAR10G very differently from the impulse noise on MNIST.
The distributional shift for MNIST induces a large change in both epistemic and predictive uncertainty, whereas for CIFAR10C the induced shift in uncertainty is significantly smaller. Since CIFAR10C contains realistic digital images, one might expect impulse noise to be more in-domain than for MNIST. The converse is true for the convolutional model, where the induced shift is larger for CIFAR10G. Thus, the sensitivity to a particular data distributional shift depends on the data domain and on neural-network architecture.
Even though this difference in perceived relative change of uncertainty is present for widely different data, it also has implications for different domains in the same training dataset, something that would be interesting to quantify in more detail.

Secondly, robustness of prediction accuracy under data distributional shifts for a given model depends on the dataset. For CIFAR10G, the higher accuracy of the fully connected neural network on noised data in Table <ref> shows that this architecture is more robust against this particular distributional shift, even though the model is less accurate close to the training domain as seen in Table <ref>. For MNIST, the convolutional model is instead more robust than both the dense and attention-based architectures as seen in Table <ref>.

§.§ Accuracy

In this section we discuss three points regarding the connection between the joint distribution of epistemic and total predictive uncertainty, and prediction accuracy.

First, looking at the joint distribution helps to identify samples where a given model is more accurate. One important goal of uncertainty quantification is to assess when the predictions of a model can be trusted. It is natural to expect that a prediction is more accurate when the total predictive uncertainty is low. We show that the joint distribution of epistemic and total uncertainty can identify accurate predictions where predictive uncertainty or epistemic uncertainty alone cannot. This can be seen in Figures <ref> and <ref>, where the joint distribution clearly resolves where the models are more accurate.  Figures <ref> and <ref> show that the projection of the joint distribution on either the axis of predictive uncertainty or the axis of epistemic uncertainty mixes samples with high and low accuracy. As a consequence, selecting for predictive uncertainty or epistemic uncertainty alone, is not as effective in identifying where a given model is more accurate. This confirms observations made in e.g. Refs. [Ghoshal et al., 2021, Krishnan and Tickoo, 2020], regarding the clustering of incorrect predictions at high uncertainty.

Second, when dropout posteriors differ from ensemble posteriors, they tend to be worse at identifying samples where the model is accurate. In particular, Figure <ref> illustrates that approximate posteriors obtained with untuned dropout rates are significantly worse at separating high and low accuracy regions.

Third, for the attention-based model it is difficult to find a threshold of the projected distribution on either predictive uncertainty or epistemic uncertainty that results in high accuracy. The top-right panels in Figures <ref> and <ref> show a distinct structure for the accuracy conditioned on the joint distribution where the marginalized distribution on either axis mixes samples of high and low accuracy. This has implications for active learning. In the present example, choosing samples based on e.g. posterior predictive uncertainty alone is inefficient, because it results in training on data regions where the model is already accurate.

§ CONCLUSIONS

Posterior predictive entropy and mutual information are used extensively as measures of total predictive uncertainty and epistemic uncertainty to assess the uncertainty and performance of neural networks and their predictions [Abdar et al., 2021].
We introduced the joint distribution of predictive uncertainty and epistemic uncertainty and used it to show the variability of uncertainty measures for different neural-network architectures, data distributional shifts and posterior approximation methods. Our main conclusion is that the origin of uncertainty is not objective. Different neural-network architectures disagree about the epistemic and aleatoric uncertainty of the same data. For a given model, the sensitivity of the uncertainty quantification to a specific type of data distributional shift depends on the underlying training dataset.
Our results imply that the joint distribution contains information about where a model is more accurate, and can be used to improve the separation of high accuracy regions compared to epistemic and predictive uncertainty alone.

An open question is whether it is possible to explain how uncertainty quantification depends on architecture from the mathematical theory of neural networks, and how to use this to build architectures that target robust uncertainty quantification. Do the attention-based models provide a more challenging architecture in general for posterior approximation? Recently there have been efforts towards understanding attention-based models from the Bayesian perspective better [Müller et al., 2021].
In practical application, uncertainty quantification using the Bayesian posterior depends on accurate posterior approximations. Thus it continues to be of utmost importance to find computationally efficient methods to obtain accurate posterior approximations.

Predictive entropy and mutual information are two measures of uncertainty derived from the high-dimensional model posterior and the posterior predictive distribution. If there are other, complementary or more informative, derived quantities that can capture the uncertainty of artificial neural networks better also remains an interesting question.

§ ACKNOWLEDGMENTS

BM and HL were supported in part by Vetenskapsrådet (grant numbers 2017-3865 and 2021-4452). OB was supported by Vetenskapsrådet (grant number 2017-05162).
HL would like to thank Erik Werner for enlightening discussions.

§ TOY MODEL POSTERIOR

The relation between mutual information of the parameter posterior and mutual information of the posterior predictive in equation (<ref>) provides a way of calculating the expected change in entropy of the high-dimensional parameter posterior $p(\theta | \mathcal{D})$ in terms of the typically lower-dimensional posterior predictive distribution $p(y|x,\mathcal{D})$ and the likelihood $p(y|x, \theta)$.
To illuminate this relation, and evaluate the involved quantities in closed-form, we present a detailed verification for a simple toy model. This also serves as an illustration of the computational complexity involved in computing the Bayesian posterior directly.

The toy problem consists of classifying points on the real line into two classes ${c_{1},c_{2}}$ and we use a simple two-parameter linear model

\begin{align}
p\left(\left.c_1\right|x,\theta _1,\theta _2\right)&=
\begin{cases}
1 & x-\theta _1<-\theta _2, \\
\frac{\left(x-\theta _1+\theta _2\right)}{2\theta _2} & \left|x-\theta _1\right|<\theta _2, \\
0 & x-\theta _1>\theta _2, \label{eq:l1}\\
\end{cases} \\
p\left(\left.c_2\right|x,\theta _1,\theta _2\right) &=1.0 - p\left(\left.c_1\right|x,\theta _1,\theta _2\right). \label{eq:l2}
\end{align}

By construction, this model has a strong prior for samples of  class 1 being located to the left of a decision boundary at $\theta_{1}$ and class 2 to the right. See figure <ref> for the resulting probability distributions for a particular choice of the model parameters $\theta_{1}$ and $\theta_{2}$.

Example of the likelihood $p(c|\theta_{1}, \theta_{2}, x)$ in equations <ref> and <ref> over the two classes $c\in\{c_{1},c_{2}\}$ for different values of $x$ with model parameters $\theta_{1}=5$ and $\theta_{2}=1$ corresponding to the decision boundary.

Let $\theta_{1}$ be uniformly distributed on $[-1, 1]$ and $\theta_{2}$ on $[\frac{1}{2}, 2]$. With this prior on the parameters, we can calculate the prior predictive distribution over the two classes by

\begin{equation}
p(\left.c\right|x) = \int_{\theta_{1}}\int_{\theta_{2}}p(c|x, \theta_{1},\theta_{2}) p(\theta_{1},\theta_{2})\mathrm{d}\theta_{1}\mathrm{d}\theta_{2},
\end{equation}

visualized in figure <ref>, where we see that the prior parameter distribution results in a smooth prior predictive distribution.

Prior class distribution given the uniform prior $p(\theta_{1},\theta_{2})$ on the model parameters.

Suppose we observe class $c_{1}$ at $x_{1} = 2$, we can then calculate a posterior distribution for the model parameters

\begin{equation}
p\left(\theta _1,\theta _2|\left\{x_1,c_1\right\}\right)= \frac{p\left(c_1|x_1,\theta _1,\theta _2\right)p\left(\theta _1,\theta _2\right)}{p\left(c_1|x_{1}\right)}
\end{equation}

where we have used Bayes theorem to express the parameter posterior in terms of conditional probabilities that can be calculated explicitly.

With this posterior we calculate the posterior predictive distribution in equation (<ref>) of the introduction, resulting in a slightly shifted distribution for class 1 in figure <ref>, compared to the class prior in figure <ref>.

Posterior predictive distribution given one observation of class $c_{1}$ at $x_{1} = 2$ compared to the prior predictive distribution. The observation of class 1 to the right shifts the posterior in this direction.

Using the toy model we can now explicitly verify the relation between (<ref>) and (<ref>). The expected entropy difference in (<ref>) is given by

\begin{multline}
I(\theta_{1},\theta_{2}|x) = \int_{\theta_{1}}\int_{\theta_{2}}p(\theta_{1},\theta_{2})\log(p(\theta_{1},\theta_{2}))\mathrm{d}\theta_{1}\mathrm{d}\theta_{2}\\ -\sum_{i=1,2}p(c_{i}|x)\int_{\theta_{1}}\int_{\theta_{2}} p\left(\theta_{1},\theta_{2}|\left\{x_1,c_i\right\}\right) \log\left(p\left(\theta_{1},\theta_{2}|\left\{x_1,c_i\right\}\right)\right) \,\mathrm{d}\theta_{1}\mathrm{d}\theta_{2} \label{eq:delta_H_theta_toy}
\end{multline}

and the posterior predictive entropy difference in (<ref>) is given by

\begin{multline}
I(c|x) = \sum_{i=1,2} p(c_{i}|x) \log(p(c_{i}|x))\\ - \int_{\theta_{1}}\int_{\theta_{2}} \left(\sum_{i=1,2} p(c_{i}|x, \theta_{1},\theta_{2}) \log(p(c_{i}|x, \theta_{1},\theta_{2})) \right)p(\theta_{1}, \theta_{2})\,\mathrm{d}\theta_{1}\mathrm{d}\theta_{2}\label{eq:delta_H_c_toy}
\end{multline}

Note that in this case, where we compute the entropy difference when adding a single observation equation (<ref>) only uses the prior distribution.

Numerically evaluating these expressions gives figure <ref> where the two curves are indistinguishable, as expected.

Entropy difference $I(\theta_{1},\theta_{2}|x)$ and $I(c|x)$ for the toy model quantifying the epistemic uncertainty for different $x$ using only the prior.

Figure <ref> shows that the epistemic uncertainty is largest close to the decision boundary of the prior. This can be understood intuitively by the fact that the model and parameter priors are such that adding observations of class 1 far to the left (or class 2 far to the right) does not add new information.

Continuing, we can perform the same calculation but instead add a new observation on top of the first one. Assuming independent observations the posterior now becomes

\begin{equation}
p\left(\theta _1,\theta _2|\left(x_1,c_1\right),\left(x_2, c_2\right)\right)= \frac{p\left(c_1|x_1,\theta _1,\theta _2\right)p\left(c_2|x_2,\theta
_1,\theta _2\right)p\left(\theta _1,\theta _2\right)}{p\left(\left(x_1,c_1\right),\left(x_2, c_2\right)\right)}
\end{equation}

and carefully calculating the entropy differences now instead results in the epistemic uncertainty in figure <ref>. Note first that the two expressions are still in excellent agreement. The observation of class 1 at $x=2$ is in tension with the prior which can be seen by the bi-modal epistemic uncertainty.

The epistemic uncertainty can be compared to the entropy of the posterior predictive distribution in figure <ref> which peaks in the region between the prior decision boundary and the observed class 1 sample.

Entropy difference $I(\theta_{1},\theta_{2}|x, \{x_{1}=2, c=1\})$ and $I(c|x, \{x_{1}=2, c=1\})$ for the toy model quantifying the epistemic uncertainty for different $x$ after a single observation $\{x=2, c=1\}$.

Entropy of the posterior predictive distribution using the prior $H(c|x)$ and after one observation $H(c|x, \{x_{1}=2, c=1\})$ for the toy model quantifying aleatoric and epistemic uncertainty for different $x$.

[Gawlikowski et al., 2021]
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee,
Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung,
Ribana Roscher, Muhammad Shahzad, Wen Yang, Richard Bamler, and Xiao Xiang
A survey of uncertainty in deep neural networks, 2021.

[Abdar et al., 2021]
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu,
Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra
Acharya, Vladimir Makarenkov, and Saeid Nahavandi.
A review of uncertainty quantification in deep learning: Techniques,
applications and challenges.
Information Fusion, 76:0 243–297, 2021.
ISSN 1566-2535.

[Gustafsson et al., 2020]
Fredrik K Gustafsson, Martin Danelljan, and Thomas B Schon.
Evaluating scalable Bayesian deep learning methods for robust
computer vision.
In Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition workshops, pages 318–319, 2020.

[Neal, 2012]
Radford M Neal.
Bayesian learning for neural networks, volume 118.
Springer Science & Business Media, 2012.

[MacKay, 1992]
David J. C. MacKay.
Information-Based Objective Functions for Active Data Selection.
Neural Computation, 40 (4):0 590–604, 07
ISSN 0899-7667.
URL <https://doi.org/10.1162/neco.1992.4.4.590>.

[Kiureghian and Ditlevsen, 2009]
Armen Der Kiureghian and Ove Ditlevsen.
Aleatory or epistemic? does it matter?
Structural Safety, 310 (2):0 105–112, 2009.
ISSN 0167-4730.
Risk Acceptance and Risk Communication.

[Cohn et al., 1996]
David A. Cohn, Zoubin Ghahramani, and Michael I. Jordan.
Active learning with statistical models.
J. Artif. Int. Res., 40 (1):0 129–145, mar
ISSN 1076-9757.

[Izmailov et al., 2021]
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon
What are Bayesian neural network posteriors really like?
In International conference on machine learning, pages
4629–4640. PMLR, 2021.

[Kendall and Gal, 2017]
Alex Kendall and Yarin Gal.
What uncertainties do we need in Bayesian deep learning for
computer vision?
Advances in neural information processing systems, 30, 2017.

[Gal and Ghahramani, 2016]
Yarin Gal and Zoubin Ghahramani.
Dropout as a Bayesian approximation: Representing model uncertainty
in deep learning.
In international conference on machine learning, pages
1050–1059. PMLR, 2016.

[Lakshminarayanan et al., 2017]
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.
Simple and scalable predictive uncertainty estimation using deep
Advances in neural information processing systems, 30, 2017.

[Hoffmann and Elster, 2021]
Lara Hoffmann and Clemens Elster.
Deep ensembles from a bayesian perspective.
arXiv preprint arXiv:2105.13283, 2021.

[Awad, 2019]
Ali Awad.
Denoising images corrupted with impulse, Gaussian, or a mixture of
impulse and Gaussian noise.
Engineering Science and Technology, an International Journal,
220 (3):0 746–753, 2019.
ISSN 2215-0986.

[LeCun et al., 2010]
Yann LeCun, Corinna Cortes, and CJ Burges.
MNIST handwritten digit database.
ATT Labs [Online]. Available:
http://yann.lecun.com/exdb/mnist, 2, 2010.

[Krizhevsky et al., 2009]
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images.

[Depeweg et al., 2018]
Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen
Decomposition of uncertainty in Bayesian deep learning for
efficient and risk-sensitive learning.
In Jennifer Dy and Andreas Krause, editors, Proceedings of the
35th International Conference on Machine Learning, volume 80 of
Proceedings of Machine Learning Research, pages 1184–1193. PMLR,
10–15 Jul 2018.
URL <https://proceedings.mlr.press/v80/depeweg18a.html>.

[Ghoshal and Tucker, 2020]
Biraja Ghoshal and Allan Tucker.
Estimating uncertainty and interpretability in deep learning for
coronavirus (covid-19) detection, 2020.
URL <https://arxiv.org/abs/2003.10769>.

[Ghoshal et al., 2021]
Biraja Ghoshal, Allan Tucker, Bal Sanghera, and Wai Lup Wong.
Estimating uncertainty in deep learning for reporting confidence to
clinicians in medical image segmentation and diseases detection.
Computational Intelligence, 370 (2):0
701–734, 2021.
URL <https://onlinelibrary.wiley.com/doi/abs/10.1111/coin.12411>.

[Ovadia et al., 2019]
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian
Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek.
Can You Trust Your Model's Uncertainty? Evaluating Predictive
Uncertainty under Dataset Shift.
Curran Associates Inc., Red Hook, NY, USA, 2019.

[Hendrycks and Dietterich, 2019]
Dan Hendrycks and Thomas Dietterich.
Benchmarking neural network robustness to common corruptions and
perturbations, 2019.
URL <https://arxiv.org/abs/1903.12261>.

[Mehlig, 2021]
Bernhard Mehlig.
Machine learning with neural networks.
Cambridge University Press, Cambridge, England, October 2021.

[Sivia and Skilling, 2006]
Devinderjit Sivia and John Skilling.
Data analysis: a Bayesian tutorial.
OUP Oxford, 2006.

[Srivastava et al., 2014]
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan
Dropout: A simple way to prevent neural networks from overfitting.
J. Mach. Learn. Res., 150 (1):0 1929–1958,
jan 2014.
ISSN 1532-4435.

[Ng, 2004]
Andrew Y. Ng.
Feature selection, l1 vs. l2 regularization, and rotational
In Proceedings of the Twenty-First International Conference on
Machine Learning, ICML '04, page 78, New York, NY, USA, 2004. Association
for Computing Machinery.
ISBN 1581138385.
URL <https://doi.org/10.1145/1015330.1015435>.

[Folgoc et al., 2021]
Loic Le Folgoc, Vasileios Baltatzis, Sujal Desai, Anand Devaraj, Sam Ellis,
Octavio E. Martinez Manzanera, Arjun Nair, Huaqi Qiu, Julia Schnabel, and Ben
Is MC dropout Bayesian?, 2021.
URL <https://arxiv.org/abs/2110.04286>.

[Fort and Jastrzebski, 2019]
Stanislav Fort and Stanislaw Jastrzebski.
Large scale structure of neural network loss landscapes.
Advances in Neural Information Processing Systems, 32, 2019.

[Shannon, 1948]
C. E. Shannon.
A mathematical theory of communication.
The Bell System Technical Journal, 270 (3):0
379–423, 1948.

[Houlsby et al., 2011]
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel.
Bayesian active learning for classification and preference learning.
arXiv preprint arXiv:1112.5745, 2011.

[Smith and Gal, 2018]
Lewis Smith and Yarin Gal.
Understanding measures of uncertainty for adversarial example
detection, 2018.

[Kirsch et al., 2019]
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal.
Batchbald: Efficient and diverse batch acquisition for deep
Bayesian active learning.
Advances in neural information processing systems,
32:0 7026–7037, 2019.

[Krizhevsky, 2009]
Alex Krizhevsky.
Learning multiple layers of features from tiny images.
Technical report, 2009.

[Springenberg et al., 2014]
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin
Striving for simplicity: The all convolutional net, 2014.
URL <https://arxiv.org/abs/1412.6806>.

[Liu et al., 2021]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and
Baining Guo.
Swin transformer: Hierarchical vision transformer using shifted
In Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV), pages 10012–10022, October 2021.

[Krishnan and Tickoo, 2020]
Ranganath Krishnan and Omesh Tickoo.
Improving model calibration with accuracy versus uncertainty
Advances in Neural Information Processing Systems,
33:0 18237–18248, 2020.

[Müller et al., 2021]
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif
Grabocka, and Frank Hutter.
Transformers Can Do Bayesian Inference.
arXiv e-prints, art. arXiv:2112.10510, December 2021.

[Gawlikowski et al., 2021]
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee,
Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung,
Ribana Roscher, Muhammad Shahzad, Wen Yang, Richard Bamler, and Xiao Xiang
A survey of uncertainty in deep neural networks, 2021.

[Abdar et al., 2021]
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu,
Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra
Acharya, Vladimir Makarenkov, and Saeid Nahavandi.
A review of uncertainty quantification in deep learning: Techniques,
applications and challenges.
Information Fusion, 76:0 243–297, 2021.
ISSN 1566-2535.

[Gustafsson et al., 2020]
Fredrik K Gustafsson, Martin Danelljan, and Thomas B Schon.
Evaluating scalable Bayesian deep learning methods for robust
computer vision.
In Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition workshops, pages 318–319, 2020.

[Neal, 2012]
Radford M Neal.
Bayesian learning for neural networks, volume 118.
Springer Science & Business Media, 2012.

[MacKay, 1992]
David J. C. MacKay.
Information-Based Objective Functions for Active Data Selection.
Neural Computation, 40 (4):0 590–604, 07
ISSN 0899-7667.
URL <https://doi.org/10.1162/neco.1992.4.4.590>.

[Kiureghian and Ditlevsen, 2009]
Armen Der Kiureghian and Ove Ditlevsen.
Aleatory or epistemic? does it matter?
Structural Safety, 310 (2):0 105–112, 2009.
ISSN 0167-4730.
Risk Acceptance and Risk Communication.

[Cohn et al., 1996]
David A. Cohn, Zoubin Ghahramani, and Michael I. Jordan.
Active learning with statistical models.
J. Artif. Int. Res., 40 (1):0 129–145, mar
ISSN 1076-9757.

[Izmailov et al., 2021]
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon
What are Bayesian neural network posteriors really like?
In International conference on machine learning, pages
4629–4640. PMLR, 2021.

[Kendall and Gal, 2017]
Alex Kendall and Yarin Gal.
What uncertainties do we need in Bayesian deep learning for
computer vision?
Advances in neural information processing systems, 30, 2017.

[Gal and Ghahramani, 2016]
Yarin Gal and Zoubin Ghahramani.
Dropout as a Bayesian approximation: Representing model uncertainty
in deep learning.
In international conference on machine learning, pages
1050–1059. PMLR, 2016.

[Lakshminarayanan et al., 2017]
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.
Simple and scalable predictive uncertainty estimation using deep
Advances in neural information processing systems, 30, 2017.

[Hoffmann and Elster, 2021]
Lara Hoffmann and Clemens Elster.
Deep ensembles from a bayesian perspective.
arXiv preprint arXiv:2105.13283, 2021.

[Awad, 2019]
Ali Awad.
Denoising images corrupted with impulse, Gaussian, or a mixture of
impulse and Gaussian noise.
Engineering Science and Technology, an International Journal,
220 (3):0 746–753, 2019.
ISSN 2215-0986.

[LeCun et al., 2010]
Yann LeCun, Corinna Cortes, and CJ Burges.
MNIST handwritten digit database.
ATT Labs [Online]. Available:
http://yann.lecun.com/exdb/mnist, 2, 2010.

[Krizhevsky et al., 2009]
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images.

[Depeweg et al., 2018]
Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen
Decomposition of uncertainty in Bayesian deep learning for
efficient and risk-sensitive learning.
In Jennifer Dy and Andreas Krause, editors, Proceedings of the
35th International Conference on Machine Learning, volume 80 of
Proceedings of Machine Learning Research, pages 1184–1193. PMLR,
10–15 Jul 2018.
URL <https://proceedings.mlr.press/v80/depeweg18a.html>.

[Ghoshal and Tucker, 2020]
Biraja Ghoshal and Allan Tucker.
Estimating uncertainty and interpretability in deep learning for
coronavirus (covid-19) detection, 2020.
URL <https://arxiv.org/abs/2003.10769>.

[Ghoshal et al., 2021]
Biraja Ghoshal, Allan Tucker, Bal Sanghera, and Wai Lup Wong.
Estimating uncertainty in deep learning for reporting confidence to
clinicians in medical image segmentation and diseases detection.
Computational Intelligence, 370 (2):0
701–734, 2021.
URL <https://onlinelibrary.wiley.com/doi/abs/10.1111/coin.12411>.

[Ovadia et al., 2019]
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian
Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek.
Can You Trust Your Model's Uncertainty? Evaluating Predictive
Uncertainty under Dataset Shift.
Curran Associates Inc., Red Hook, NY, USA, 2019.

[Hendrycks and Dietterich, 2019]
Dan Hendrycks and Thomas Dietterich.
Benchmarking neural network robustness to common corruptions and
perturbations, 2019.
URL <https://arxiv.org/abs/1903.12261>.

[Mehlig, 2021]
Bernhard Mehlig.
Machine learning with neural networks.
Cambridge University Press, Cambridge, England, October 2021.

[Sivia and Skilling, 2006]
Devinderjit Sivia and John Skilling.
Data analysis: a Bayesian tutorial.
OUP Oxford, 2006.

[Srivastava et al., 2014]
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan
Dropout: A simple way to prevent neural networks from overfitting.
J. Mach. Learn. Res., 150 (1):0 1929–1958,
jan 2014.
ISSN 1532-4435.

[Ng, 2004]
Andrew Y. Ng.
Feature selection, l1 vs. l2 regularization, and rotational
In Proceedings of the Twenty-First International Conference on
Machine Learning, ICML '04, page 78, New York, NY, USA, 2004. Association
for Computing Machinery.
ISBN 1581138385.
URL <https://doi.org/10.1145/1015330.1015435>.

[Folgoc et al., 2021]
Loic Le Folgoc, Vasileios Baltatzis, Sujal Desai, Anand Devaraj, Sam Ellis,
Octavio E. Martinez Manzanera, Arjun Nair, Huaqi Qiu, Julia Schnabel, and Ben
Is MC dropout Bayesian?, 2021.
URL <https://arxiv.org/abs/2110.04286>.

[Fort and Jastrzebski, 2019]
Stanislav Fort and Stanislaw Jastrzebski.
Large scale structure of neural network loss landscapes.
Advances in Neural Information Processing Systems, 32, 2019.

[Shannon, 1948]
C. E. Shannon.
A mathematical theory of communication.
The Bell System Technical Journal, 270 (3):0
379–423, 1948.

[Houlsby et al., 2011]
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel.
Bayesian active learning for classification and preference learning.
arXiv preprint arXiv:1112.5745, 2011.

[Smith and Gal, 2018]
Lewis Smith and Yarin Gal.
Understanding measures of uncertainty for adversarial example
detection, 2018.

[Kirsch et al., 2019]
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal.
Batchbald: Efficient and diverse batch acquisition for deep
Bayesian active learning.
Advances in neural information processing systems,
32:0 7026–7037, 2019.

[Krizhevsky, 2009]
Alex Krizhevsky.
Learning multiple layers of features from tiny images.
Technical report, 2009.

[Springenberg et al., 2014]
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin
Striving for simplicity: The all convolutional net, 2014.
URL <https://arxiv.org/abs/1412.6806>.

[Liu et al., 2021]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and
Baining Guo.
Swin transformer: Hierarchical vision transformer using shifted
In Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV), pages 10012–10022, October 2021.

[Krishnan and Tickoo, 2020]
Ranganath Krishnan and Omesh Tickoo.
Improving model calibration with accuracy versus uncertainty
Advances in Neural Information Processing Systems,
33:0 18237–18248, 2020.

[Müller et al., 2021]
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif
Grabocka, and Frank Hutter.
Transformers Can Do Bayesian Inference.
arXiv e-prints, art. arXiv:2112.10510, December 2021.