A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors
The distribution of the weights of modern deep neural networks (DNNs) -
crucial for uncertainty quantification and robustness - is an eminently complex
object due to its extremely high dimensionality. This paper proposes one of the
first large-scale explorations of the posterior distribution of deep Bayesian
Neural Networks (BNNs), expanding its study to real-world vision tasks and
architectures. Specifically, we investigate the optimal approach for
approximating the posterior, analyze the connection between posterior quality
and uncertainty quantification, delve into the impact of modes on the
posterior, and explore methods for visualizing the posterior. Moreover, we
uncover weight-space symmetries as a critical aspect for understanding the
posterior. To this extent, we develop an in-depth assessment of the impact of
both permutation and scaling symmetries that tend to obfuscate the Bayesian
posterior. While the first type of transformation is known for duplicating
modes, we explore the relationship between the latter and L2 regularization,
challenging previous misconceptions. Finally, to help the community improve our
understanding of the Bayesian posterior, we will shortly release the first
large-scale checkpoint dataset, including thousands of real-world models and
our codes.
Olivier Laurent, Emanuel Aldea, Gianni Franchi
# A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors

###### Abstract

The distribution of the weights of modern deep neural networks (DNNs) - crucial for uncertainty quantification and robustness - is an emimently complex object due to its extremely high dimensionality. This paper proposes one of the first large-scale explorations of the posterior distribution of deep Bayesian Neural Networks (BNNs), expanding its study to real-world vision tasks and architectures. Specifically, we investigate the optimal approach for approximating the posterior, analyze the connection between posterior quality and uncertainty quantification, delve into the impact of modes on the posterior, and explore methods for visualizing the posterior. Moreover, we uncover weight-space symmetries as a critical aspect for understanding the posterior. To this extent, we develop an in-depth assessment of the impact of both permutation and scaling symmetries that tend to obfuscate the Bayesian posterior. While the first type of transformation is known for duplicating modes, we explore the relationship between the latter and L2 regularization, challenging previous misconceptions. Finally, to help the community improve our understanding of the Bayesian posterior, we will release shortly the first large-scale checkpoint dataset, including thousands of real-world models, along with our codes.

## 1 Introduction

Despite substantial advancements in deep learning, Deep Neural Networks (DNNs) remain black box models. Various studies have sought to explore DNN loss landscapes (Li et al., 2018; Fort and Jastrzebski, 2019; Fort and Scherlis, 2019; Liu et al., 2022) to achieve a deeper understanding of these models. Recent works have, for instance, unveiled the interconnection of the modes obtained with Stochastic Gradient Descent (SGD) via narrow pathways that link pairs of modes, or through tunnels that connect multiple modes simultaneously (Garipov et al., 2018; Draxler et al., 2018). This

Figure 1: **Weight-space symmetries greatly impact the estimated Bayesian posterior**. Permutation symmetries very clearly increase the number of modes of the posterior distribution in the case of a 2-hidden neuron perceptron.
mode connectivity primarily arises from scale and permutation invariances, which imply that numerous weights can represent the same exact function (e.g., Entezari et al. (2022)). Several studies have delved into the relationship between these symmetries and the characteristics of the loss landscape (Entezari et al., 2022; Neyshabur et al., 2015; Brea et al., 2019). Our work investigates the connections between these symmetries and the distribution of DNN weights, a crucial aspect for uncertainty quantification. As shown in Figure 1, it is apparent that these symmetries also exert influence on DNN posterior distributions.

Uncertainty quantification plays a pivotal role in high-stakes industrial applications - such as autonomous driving (Levinson et al., 2011; McAllister et al., 2017; Sun et al., 2019) - where reliable predictions and informed decision-making are paramount. In such critical domains, understanding and effectively managing uncertainties, particularly the model-related epistemic uncertainties (Hora, 1996) arising from incomplete knowledge, is essential. Among the various methods proposed to address these challenges, Bayesian Neural Networks (BNNs) (Tishby et al., 1989) offer a principled and theoretically sound approach. BNNs quantify uncertainty by modeling our beliefs about parameters and outcomes probabilistically (Tishby et al., 1989; Hinton and Van Camp, 1993). However, this perspective faces significant hurdles when applied to deep learning, primarily related to scalability (Izmailov et al., 2021) and the precision of approximations (MacKay, 1995). Due to their very high dimension, BNNs struggle to estimate the posterior distribution, i.e., the probability density of converging towards any particular set of model parameters \(\omega\), given the observed data \(\mathcal{D}\).

Diverging from methods such as the Maximum Likelihood Estimate or Maximum A Posteriori (also in Tishby et al. (1989)), which we typically derive through gradient descent optimization of cross-entropy (with L2 regularization for the latter), BNNs assign a probability to each possible model (or hypothesis) and offer predictions considering the full extent of possible models. In mathematical terms, when we denote the target as \(y\), the input vector as \(\mathbf{x}\), and the weight space as \(\Omega\), we can express this approach through the following intractable formula, often referred to as the marginalization on the parameters of the model (Tishby et al., 1989; Rasmussen et al., 2006):

\[p(y\mid\mathbf{x},\mathcal{D})=\int\limits_{\mathbf{\omega}\in\Omega}p(y\mid\mathbf{x}, \mathbf{\omega})p(\mathbf{\omega}\mid\mathcal{D})d\mathbf{\omega}. \tag{1}\]

The posterior distribution \(p(\mathbf{\omega}|\mathcal{D})\) assumes a central and arguably the most critical role in BNNs. Many successful methods for quantifying uncertainty can be viewed as attempts to approximate this posterior, each with its own trade-offs in terms of accuracy and computational efficiency, as illustrated in previous research (Blundell et al., 2015; Gal and Ghahramani, 2016; Lakshminarayanan et al., 2017). Prior work (Kuncheva and Whitaker, 2003; Fort et al., 2019; Ortega et al., 2022) has established the importance of achieving _diversity_ in the sampled DNNs drawn from the posterior, particularly when dealing with uncertain input data. However, the presence of permutation symmetries and scale symmetries among hidden units in neural networks may lead to an increased number of local minima (Zhao et al., 2023) with no diversity. In the context of BNNs, this phenomenon can result in a proliferation of modes within the posterior distribution.

In this paper, we delve into the impact of weight symmetries on the posterior distribution. While there have been numerous efforts to visualize the loss landscape, we explore the possibility of conducting similar investigations for the posterior distribution. Additionally, we introduce a protocol for assessing the quality of posterior estimation and examine the relationship between posterior estimation and the accuracy of uncertainty quantification. Specifically, our contributions are as follows:

**(1)** We build a new mathematical formalism to highlight the different impacts of the permutation and scaling symmetries on the posterior and on uncertainty estimation in deep neural networks. Notably, we explain the seeming equivalence of the marginals in Figure 1. We also perform the first in-depth exploration of the existence of scaling symmetries and their overlooked effect. **(2)** We evaluate the quality of various methods for estimating the posterior distribution on real-world applications using the Maximum Mean Discrepancy, offering a practical benchmark to assess their performance in capturing uncertainty. **(3)** We release a new dataset that includes the weights of thousands of models across various computer vision tasks and architectures, ranging from MNIST to TinyImageNet. This dataset is intended to facilitate further exploration and collaboration in the field of uncertainty in deep learning. **(4)** Our investigation delves into the proliferation of modes in the context of posterior symmetries and exhibits the capacity of ensembles to converge toward non-functionally equivalent modes. Furthermore, we discuss the influence of symmetries in the training process.
Related work

Epistemic uncertainty, Bayesian inference, and posteriorEpistemic uncertainty (Hora, 1996; Hullermeier & Waegeman, 2021) plays a crucial role in accurately assessing predictive model reliability. However, despite ongoing discussions, estimating this uncertainty remains a challenge. BNNs (Goan & Fookes, 2020) predominantly shape the landscape of methodologies that tackle epistemic uncertainties (Gawlikowski et al., 2023). Given the complexity of dealing with posterior distributions, these approaches have mostly been tailored for enhanced scalability.

For instance, Hernandez-Lobato & Adams (2015) proposed an efficient probabilistic backpropagation, and Blundell et al. (2015) developed BNNs by Backpropagation to learn diagonal Gaussian distributions with the reparametrization trick. Similarly, Laplace methods (Laplace, 1774; MacKay, 1992; Ritter et al., 2018) characterize the posterior distribution of weights, often focusing on the final layer (Ober & Rasmussen, 2019; Watson et al., 2021), again for scalability.

On a different approach, Monte Carlo Dropout, introduced by Gal & Ghahramani (2016) and Kingma et al. (2015), is a framework that models the posterior as a mixture of Dirac distributions when applied on fully-connected layers. Broadening the spectrum, Deep Ensembles (Lakshminarayanan et al., 2017), arguably along with their more efficient alternatives (Lee et al., 2015; Wen et al., 2019; Maddox et al., 2019; Franchi et al., 2020; Bavasai et al., 2021; Laurent et al., 2023), have been interpreted by Wilson & Izmailov (2020) as Monte Carlo estimates of Equation 1.

Markov-chain-based Bayesian posterior estimationNeal et al. (2011) introduced HMC as an accurate method for estimating the posterior of neural networks, but its application to large-scale problems remains challenging due to the very high computational demands. Izmailov et al. (2021) managed to scale full-batch HMC to CIFAR-10 (Krizhevsky, 2009) with ResNet-20 (He et al., 2016) thanks to 512 TPUv3.

In response to these challenges, stochastic approximations of MCMC have gained attention for their ability to provide computationally more feasible solutions. A prominent example is the Langevin dynamics-based approach proposed by Welling & Teh (2011). By introducing noise into the dynamics, Langevin dynamics allows for more practical implementation on large datasets.

In addition to Langevin dynamics, other stochastic gradient-based methods have been introduced to improve the efficiency of MCMC sampling. Chen et al. (2014) presented Stochastic Gradient Hamiltonian Monte Carlo (SGHMC), integrating stochastic gradients into HMC. Likewise, Zhang et al. (2020) proposed C-SGLD and C-SGHMC (Cyclic Stochastic Gradient Langevin Dynamics and Hamiltonian Monte Carlo), introducing controlled noise via cyclic preconditioning.

While stochastic approximation methods offer computational convenience, they come with the trade-off of slowing down the convergence and potentially introducing bias into the resulting inference (Bardenet et al., 2017; Zou & Gu, 2021). As such, the suitability of these approaches depends on the specific application and the level of acceptable bias in the analysis.

Symmetries in neural networksThe seminal work from Hecht-Nielsen (1990) established a foundational understanding by investigating permutation symmetries and setting a lower bound on symmetries in multi-layer perceptrons. Albertini et al. (1993) extended this work and studied flip sign symmetries in networks with odd activation functions. This work was further generalized to a broader range of activation functions by Kurkova & Kainen (1994), who proposed symmetry removal to streamline evolutionary algorithms.

Recent advancements have generalized symmetries to modern neural architectures. Neyshabur et al. (2015) explored the scaling symmetries that arise in architectures containing non-negative homogeneous activation functions, including Nair & Hinton (2010)'s ubiquitous Rectified Linear Unit (ReLU). This perspective extends our understanding of symmetries to ReLU-powered architectures, e.g., AlexNet (Krizhevsky et al., 2012), VGG (Simonyan & Zisserman, 2015), and ResNet networks (He et al., 2016). This paper focuses on scaling and permutation symmetries, but other works, such as Rolnick & Kording (2020); Grigsby et al. (2023), unveil less apparent symmetries.

Closest to our work, Wiese et al. (2023) demonstrated that taking weight-space symmetries into account could reduce the Bayesian posterior support and improve Monte-Carlo Markov Chains posterior estimation.
Symmetries increase the complexity of Bayesian posteriors

We propose to study the most influential weight-space symmetries and their properties related to the Bayesian posterior. Firstly, weight-space symmetries transform the parameters of the neural networks while keeping the networks functionally invariant; in other words,

**Definition 3.1**.: Let \(f_{\mathbf{\omega}}\) be a neural network of parameters \(\mathbf{\omega}\) taking \(n\)-dimensional vectors as inputs. We say that the transformation \(\mathcal{T}\) modifying \(\mathbf{\omega}\) is a weight-space symmetry operator, iff

\[f_{\mathcal{T}(\mathbf{\omega})}=f_{\mathbf{\omega}},\text{ or }\forall\mathbf{x}\in \mathbb{R}^{n},f_{\mathcal{T}(\mathbf{\omega})}(\mathbf{x})=f_{\mathbf{\omega}}(\mathbf{x}). \tag{2}\]

With the notation \(f_{\mathcal{T}(\mathbf{\omega})}(\mathbf{x})\), we apply the symmetry operator \(\mathcal{T}\) on the weights \(\mathbf{\omega}\), resulting in a set of modified weights. In this paper, we show that scaling and permutation symmetries have different impacts on the posterior of neural networks. They can, for instance, complicate Bayesian posteriors, creating artificial functionally equivalent modes.

### A gentle example of artificial symmetry-based posterior modes

To illustrate the considerable impact of symmetries on the Bayesian posterior, we propose a small-scale example in Figure 1. We generate this example by training two-hidden-neuron perceptrons on linearly separable data. Figure 1 presents the estimation of the posterior of the output weights with 10,000 checkpoints. In this figure, the most important mode is duplicated due to the permutation symmetry, which appears to symmetrize the graph. On the other hand, scaling symmetries seem to regroup some points around the modes. We detail this toy experiment in Appendix A. In the following, we develop a new mathematical framework tailored to help understand the impact of these symmetries on the posterior, devise mathematical insights explaining these intuitions, and propose empirical explorations.

### Background and definitions

The full extent of this formalism (including sketches of proofs, other definitions, properties, and propositions) is developed in Appendix C. We deem that the following properties are the minimal information necessary to understand the impact on the Bayesian posterior of the two main types of symmetries: the scaling and permutation symmetries. This part summarizes the most important results for multi-layer perceptrons, but we provide leads for generalizing our results to modern deep neural networks such as convolutional residual networks (He et al., 2016) in Appendix D.4.

### Scaling symmetries

For clarity, we propose the following definitions and properties for two-layer fully connected perceptrons without loss of generality. Please refer to Appendix C and D.4 for extensions. We start with two new operators between vectors and matrices to define scaling symmetries.

**Definition 3.2**.: We denote the line-wise product as \(\bigtriangledown\) and the column-wise product as \(\rhd\). Let \(\mathbf{\omega}\in\mathbb{R}^{n\times m}\), \(\mathbf{\lambda}\in\mathbb{R}^{m}\), and \(\mathbf{\mu}\in\mathbb{R}^{n}\), with \(n,m\geq 1\),

\[\forall i\in\llbracket 1,n\rrbracket,\forall j\in\llbracket 1,m\rrbracket, \ (\mathbf{\lambda}\bigtriangledown\mathbf{\omega})_{i,j}=\lambda_{j}\omega_{i,j}\text{ and }(\mathbf{\mu}\rhd\mathbf{\omega})_{i,j}=\mu_{i}\omega_{i,j}. \tag{3}\]

Given that the rectified linear unit \(r\) is non-negative homogeneous - i.e., for all non-negative \(\lambda\), \(r(\lambda x)=\lambda r(x)\) - we have the following core property for scaling symmetries (Neyshabur et al., 2015), trivially extendable to additive biases:

**Property 3.1**.: _For all \(\mathbf{\theta}\in\mathbb{R}^{\times m}\), \(\mathbf{\omega}\in\mathbb{R}^{m\times n}\), \(\mathbf{\lambda}\in\left(\mathbb{R}_{>0}\right)^{m}\),_

\[\forall\mathbf{x}\in\mathbb{R}^{n},\ (\mathbf{\lambda}^{-1}\bigtriangledown\mathbf{ \theta})\times r(\mathbf{\lambda}\rhd\mathbf{\omega}\mathbf{x})=\mathbf{\theta}\times r(\mathbf{ \omega}\mathbf{x}). \tag{4}\]

When defining \(\mathcal{T}_{s}\) by the transformation Equation 4 - in the case of a two-layer perceptron - the core property directly follows, with the set of parameters \(\Lambda=\{\mathbf{\lambda}\}\):

**Property 3.2** (Scaling symmetries).: _The scaling operation \(\mathcal{T}_{s}\) with a set of non-negative parameters \(\Lambda\) is a symmetry for all neural network \(f_{\mathbf{\omega}}\), i.e._

\[\forall\mathbf{x}\in\mathbb{R}^{n},\ f_{\mathcal{T}_{s}(\mathbf{\omega}, \Lambda)}(\mathbf{x})=f_{\mathbf{\omega}}(\mathbf{x}). \tag{5}\]
### Permutation symmetries

We also propose a simple and intuitive formalism for permutation symmetries, multiplying the weights by permutation matrices. We have the following property for two-layer perceptions, with \(P_{m}\) the set of \(m\)-dimensional permutation matrices:

**Property 3.3**.: _For all \(\mathbf{\theta}\in\mathbb{R}^{\times m}\), \(\mathbf{\omega}\in\mathbb{R}^{m\times n}\), and permutation matrices \(\mathbf{\pi}\in P_{m}\),_

\[\forall\mathbf{x}\in\mathbb{R}^{n},\;\mathbf{\theta}\mathbf{\pi}^{\intercal}\times r\left( \mathbf{\pi}\times\mathbf{\omega}\mathbf{x}\right)=\mathbf{\theta}\times r(\mathbf{\omega}\mathbf{x}). \tag{6}\]

The left term of Equation 6 is the definition of the permutation symmetry operator of parameter \(\Pi=\{\mathbf{\pi}\}\) for a network with two layers. In general, we have the following property:

**Property 3.4** (Permutation symmetries).: _The permutation operation \(\mathcal{T}_{p}\) with a set of parameters \(\Pi\) is a symmetry for all neural networks \(f_{\mathbf{\omega}}\), i.e.,_

\[\forall\mathbf{x}\in\mathbb{R}^{n},\;f_{\mathcal{T}_{p}(\mathbf{\omega},\Pi)}(\mathbf{x}) =f_{\mathbf{\omega}}(\mathbf{x}). \tag{7}\]

### The Bayesian posterior as a mixture of distributions

With this formalism, we can establish the following proposition, clarifying the impact of weight-space symmetries on the Bayesian posterior.

**Proposition 1**.: _Define \(f_{\omega}\) a neural network and \(f_{\tilde{\omega}}\) its corresponding identifiable model - a network transformed for having sorted unit-normed neurons. Let us also denote \(\Pi\) and \(\mathbb{\Lambda}\), the sets of permutation sets and scaling sets, respectively, and \(\tilde{\Omega}\) the random variable of the sorted weights with unit norm. The Bayesian posterior of a neural network \(f_{\omega}\) trained with stochastic gradient descent can be expressed as a continuous mixture of a discrete mixture:_

\[p(\Omega=\mathbf{\omega}\mid\mathcal{D})=\int\limits_{\Lambda\in\Lambda}|\Pi|^{-1 }\underset{\Pi\in\Pi}{\sum}\;p(\tilde{\Omega}=\mathcal{T}_{p}(\mathcal{T}_{s}( \mathbf{\omega},\Lambda),\Pi),\Lambda\mid\mathcal{D})d\Lambda. \tag{8}\]

Proposition 1 provides an expression of the Bayesian posterior that highlights the redundancy of the resulting distribution, explaining the symmetry in Figure 1 (left). Interestingly, a direct corollary of this formula is that _all the layer-wise marginal distributions are identical_. In Equation 8, the permutations play a transparent role, being independent of \(\mathbf{\omega}\) (except for strongly quantized spaces for \(\mathbf{\omega}\), but we leave this particular case for future works). On the other hand, the part played by scaling symmetries is more complex and we discuss their impact in the following section.

### On the effective impact of scaling symmetries

The equiprobability of permutations in Equation 8 leads to a balanced mixture of \(|\Pi|\) permuted terms. We have no such result on scaling symmetries since the standard L2-regularized loss is not invariant to scaling symmetries (contrary to permutations). This absence of invariance obscures the impacts of scaling symmetries, which mostly remains to be addressed, although their "reduction" due to regularization was mentioned in, e.g., Godfrey et al. (2022). To the best of our knowledge, we provide the first analysis of the reality of scaling symmetries and their impact on the Bayesian posterior. With this objective in mind, we propose the following problem.

**Definition 3.3**.: Let \(f_{\omega}\) be a neural network and \(\tilde{\omega}\) its weights without the biases. We define the minimum scaled network mass problem (or _the min-mass problem_) as the minimization of the L2-regularization term of \(f_{\omega}\) (the "mass") under scaling transformations. In other words,

\[m^{*}=\min_{\Lambda\in\Lambda}\;\left|\mathcal{T}_{s}(\tilde{\omega},\Lambda) \right|^{2}. \tag{9}\]

This problem has interesting properties. Notably, we show in Appendix C.7 that the _min-mass_ problem is log-log strictly convex (Boyd & Vandenberghe, 2004; Agrawal et al., 2019):

**Proposition 2**.: _The minimum scaled network mass problem is log-log strictly convex. As such, it is equivalent to a strictly convex problem on \(\mathbb{R}^{|\Lambda|}\) and admits a global minimum attained in a single point denoted \(\Lambda^{*}\)._
It follows from Proposition 2 that, if not already optimal at convergence, there is an infinite number of equivalent networks with training loss lower than the original network. We put the proposition into practice in Figure 2 using trained OptuNets (see Appendix B.2). We measure their mass distribution and compare it to the masses of the optimal networks found with convex optimization. In practice, the effect of scaling symmetries remains present even with weight decay: neural networks always seem subject to scaling symmetries. For completeness, we report the values of the largest elements of each layer, showing that the minimization tends to increase the maxima of the layers with fewer parameters yet does not seem to promote unfeasible values.

In this section, we provided theoretical insights on the contrasted impacts of both scaling and permutation symmetries. In the following, we perform more empirical studies and propose to evaluate the link between posterior quality and experimental performance.

## 4 Comparing the estimations of the posterior by approximate Bayesian methods

In this section, we leverage symmetries to compare popular single-mode methods, namely, Monte Carlo Dropout (Gal & Ghahramani, 2016), Stochastic Weight Averaging Gaussian (SWAG) proposed by Maddox et al. (2019), Bayes by backpropagation BNNs (Blundell et al., 2015), and Laplace methods (Ritter et al., 2018). We also include their multi-modal variations, corresponding to the application of these methods on 10 different independently trained models, as well as SGHMC (Chen et al., 2014) and Deep Ensembles (DE) highlighted by Lakshminarayanan et al. (2017) and proposed earlier by Hansen & Salamon (1990). We compare these methods on 3 image classification tasks with different levels of difficulty, ranging from MNIST (LeCun et al., 1998) with our OptuNet (392 parameters) to CIFAR-100 (Krizhevsky, 2009) and Tiny-ImageNet (Deng et al., 2009) with ResNet-18 (He et al., 2016). To this extent, we propose leveraging maximum mean discrepancy (MMD) to estimate the dissimilarities between the high-dimensional posterior distributions. We estimate the target distribution with 1000 checkpoints and compare it to 100 samples extracted from each of the previously-mentioned techniques. We detail these experiments in Appendix B.

### Evaluating the quality of the estimation of the Bayesian posterior

One approach to assess the similarities between distributions involves estimating the distributions and subsequently quantifying the distance between these estimated distributions (Smola et al., 2007; Sriperumbudur et al., 2010). However, these methods can become impractical when dealing with distributions in extremely high-dimensional spaces, such as the posterior of DNNs. An alternative solution is to embed the probability measures into a Reproducing Kernel Hilbert Space (RKHS) developed by, e.g., Bergmann (1922); Aronszajn (1950); Schwartz (1964). Within this framework, a distance metric MMD (Song, 2008) - defined as the distance between the respective mean elements within the RKHS - is used to quantify the dissimilarity between the distributions. Gretton et al. (2012) proposed to leverage MMD for efficient two-sample tests in high dimensions. For better

Figure 2: **OptuNet trained with weight decay never converges towards the _minimum mass_.** We note that the maxima of the weights of scaled OptuNets at the minimum mass tend to be greater for lighter layers than in the original networks.
efficacy, MMDs are computed on RKHS that allow for comparing all of their moments. In Table 1, we propose to compute the median of the aggregated MMDs (Schrab et al., 2023) with multiple Gaussian and Laplace kernels, with and without symmetries.

For tractability, we report the mean - weighted by the number of parameters of each layer - of the median over the twenty MMD kernels between the layer-wise estimated posterior and the posterior approximated by each method. The estimated posterior includes 1000 independently trained checkpoints, and the posterior approximation of each method includes 100 samples. Please refer to Appendix B for extended details (including the means and maxima of the MMDs).

### Performance metrics and OOD datasets

On top of the MMD quantifying the difference between the posterior estimations, we measure several empirical performance metrics. We evaluate the overall performance of the models using the accuracy and the Brier score (Brier, 1950; Gneiting et al., 2007). Furthermore, we choose the binned Expected Calibration Error (ECE) (Naeini et al., 2015) for top-label calibration and measure the quality of the out-of-distribution (OOD) detection using the area under the precision-recall curve (AUPR) and the false positive rate at \(95\%\) recall (FPR95), as recommended by Hendrycks & Gimpel (2017), as a measure of OOD detection abilities, that are supposed to correlate with the quality of the estimated posterior. Finally, we report the mean diversity of the predictions in each ensemble through the mutual information (MI) (e.g., Ash (1965)), often used to measure epistemic uncertainty (Kendall & Gal, 2017; Ovadia et al., 2019). We use FashionMNIST (Xiao et al.,

\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline  & Method & MMD \(\downarrow\) & NS \(\downarrow\) & Acc \(\uparrow\) & ECE \(\downarrow\) & Brier \(\downarrow\) & AUPR \(\uparrow\) & FPR95 \(\downarrow\) & IDMI \(\downarrow\) & OODMI \(\uparrow\) \\ \hline \multirow{8}{*}{**OOD**} & \multirow{2}{*}{OODMI} & Dropout & 15.0 & 14.3 & 83.3 & 33.4 & 26.1 & 96.4 & 98.6 & 26.1 & 22.2 \\  & & BNN & 18.8 & 17.1 & 78.1 & 74.4 & 30.9 & 67.9 & 93.7 & **0.1** & 0.1 \\  & & SGHMC & 16.7 & 17.7 & 95.1 & 7.6 & **2.8** & 73.7 & 98.4 & 4.3 & 14.5 \\  & & SWAG & 16.0 & 14.6 & 88.3 & 17.7 & 4.9 & 73.4 & 68.6 & 4.0 & 8.7 \\  & & Laplace & 10.6 & 9.5 & 87.9 & 18.1 & 48.8 & 48.2 & 74.6 & 6.2 & 5.9 \\ \cline{2-11}  & & Dropout & 2.1 & 2.1 & 92.1 & 29.2 & 36.8 & **97.2** & 78.2 & 36.6 & 52.5 \\  & & BNN & 2.8 & 2.5 & 86.5 & 17.5 & 24.4 & 96.9 & 27.2 & 21.1 & 52.3 \\  & & SWAG & 1.8 & 1.3 & 95.0 & 13.1 & 17.5 & 88.7 & 24.6 & 27.6 & 62.2 \\  & & Laplace & 1.8 & 0.8 & 94.8 & 12.8 & 15.8 & 95.4 & 32.1 & 21.1 & 52.2 \\ \cline{2-11}  & & DE & **0.0** & **0.9** & **95.8** & 10.7 & 13.5 & 95.7 & **12.8** & 19.3 & **62.6** \\ \hline \multirow{8}{*}{**OOD**} & \multirow{2}{*}{OODMI} & Dropout & 4.5 & 7.5 & 69.3 & 12.1 & 44.3 & 80.1 & 64.0 & 6.0 & 9.6 \\  & & BNN & 9.0 & 10.2 & 57.6 & 21.8 & 62.5 & 81.7 & 62.6 & 0.8 & 2.2 \\  & & SGHMC & 7.5 & 7.9 & 69.3 & 4.3 & 41.5 & 87.5 & 41.4 & **0.0** & 0.1 \\  & & SWAG & 6.7 & 7.2 & 66.7 & 1.7 & 43.8 & 30.7 & 17.1 & 2.7 & 14.7 \\  & & Laplace & 5.7 & 7.0 & 70.7 & 1.4 & 40.0 & 84.5 & 40.6 & 31.3 & 70.6 \\ \cline{2-11}  & & Dropout & 0.7 & 4.5 & 75.5 & 4.8 & 64.2 & 93.5 & 23.7 & 23.8 & 77.0 \\  & & BNN & 6.1 & 5.6 & 66.3 & 1.3 & 45.6 & 91.0 & 30.9 & 44.5 & 110.4 \\  & & SWAG & 5.0 & 5.4 & 68.4 & 2.3 & 42.1 & **97.9** & 17.1 & 7.5 & 33.0 \\  & & Laplace & 0.6 & 4.3 & 75.2 & 7.8 & 35.2 & 95.1 & 20.1 & 50.5 & **128.3** \\  & & DE & **0.0** & **0.5** & **15.9** & 2.3 & **33.3** & 97.1 & **16.1** & 26.5 & 90.1 \\ \hline \multirow{8}{*}{**OOD**} & \multirow{2}{*}{OODMI} & Dropout & 9.5 & 4.9 & 56.4 & 14.2 & 60.3 & 70.8 & 87.6 & 9.5 & 13.5 \\  & & BNN & / & / & / & / & / & / & / & / & / \\  & & SGHMC & 9.8 & 5.3 & 54.4 & 2.3 & 58.9 & 78.1 & 74.4 & **0.1** & 0.2 \\  & & SWAG & 9.1 & 3.9 & 61.7 & 10.0 & 51.8 & 84.6 & 66.7 & 3.3 & 9.0 \\  & & Laplace & 5.5 & 6.1 & 29.7 & 7.4 & 81.3 & 62.3 & 82.3 & 211.7 & 254.8 \\ \cline{2-11}  & & Dropout & 4.3 & 1.8 & 65.3 & 10.5 & 48.2 & 90.9 & 57.7 & 41.9 & 93.2 \\ \cline{2-11}  & & BNN & / & / & / & / & / & / & / & / & / \\ \cline{2-11}  & & SWAG & 6.7 & 5.4 & 64.8 & 2.3 & **46.6** & **98.4** & 53.6 & 20.5 & 55.2 \\ \cline{2-11}  & & Laplace & 0.5 & 3.1 & 34.4 & 13.0 & 79.3 & 64.4 & 77.2 & 227.4 & **267.2** \\ \cline{2-11}  & & DE & **0.0** & **0.0** & **65.4** & 8.5 & 47.3 & 94.2 & **41.6** & 44.7 & 108.5 \\ \hline \hline \end{tabular}
\end{table}
Table 1: **Comparison of popular methods approximating the Bayesian posterior.** All scores are expressed in %, except the MMDs for ResNet-18 networks, expressed in %. Acc stands for accuracy, and **IDMI** and **OODMI** are the in-distribution and out-of-distribution mutual information. NS is the MMD computed after the removal of the symmetries, and DE stands for Deep Ensembles. Multi-mode methods use ten independently trained models.
2017), SVHN (Netzer et al., 2011), and Textures (Cimpoi et al., 2014) as OOD datasets for MNIST, CIFAR-100, and TinyImageNet, respectively.

### Results

In our examination of Table 1 emerges that multi-mode techniques consistently demonstrate superior performance in terms of MMD when compared to their single-mode counterparts. This trend holds true in accuracy and negative log-likelihood. However, an intriguing divergence arises when considering the ECE, where techniques focusing solely on estimating a single mode often exhibit superior performance.

Turning our attention to the assessment of epistemic uncertainty, as quantified by AUPR and FPR95, multi-mode techniques, notably multi-SWAG, and Deep Ensembles, consistently outperform other methods. This underscores the strong connection between posterior estimation and the accuracy of epistemic uncertainty quantification. However, we note that the quality of aleatoric uncertainty quantification does not consistently correlate with that of the posterior distribution estimation.

The final two columns of the table shed light on the diversity of the models sampled from the posterior. The objective here is to minimize in-distribution mutual information (IDMI) while concurrently maximizing out-of-distribution mutual information (OODMI). An analysis shows that mono-mode methods tend to yield lower values for both IDMI and OODMI than multi-mode methods, which tend to exhibit higher IDMI and OODMI values, suggesting greater _diversity_.

## 5 Discussions

We develop further insights on the posterior of Bayesian Neural Networks in relationship with symmetries. Notably, we evaluate the risk of _functional collapse_, i.e., of training very similar networks in Section 5.1 and we discuss the frequency of weights permutations in 5.2. We expand these discussions, add observations on the evaluation of the number of modes of the posterior, and propose visualizations in Appendix D.

### Functional collapse in ensembles: a study of ID and OOD disagreements

Given the potentially very high number of equivalent modes due to permutation symmetries (see Section 3.5), we support broadening the concept of collapse in the parameter-space (e.g., in D'Angelo and Fortuin (2021)) to _functional collapse_ to account for the impact of symmetries on the posterior. Parameter-space collapse is more restrictive and may not be formally involved when ensemble members lack diversity. It is also much harder to characterize as it would require an analysis of the loss landscape, at the very least.

We propose here to quantify functional collapse as a potential ground for the need for more complex repulsive ensembling methods (Masegosa, 2020; Rame and Cord, 2021). To this extent, we randomly select 1000 ResNet-18 (trained to estimate the Bayesian posterior in Section 4) and compute the mean over the test set of their pairwise mutual information, quantifying the divergence between the single models and their average. We make these measures on in-distribution and out-of-distribution data (CIFAR-100 (ID) and SVHN (OOD)). In Figure 3 (left), we see that the in-distribution MI

Figure 3: **Experiments show no hint of a functional collapse between couples of independently trained ResNet-18 on CIFAR-100. Moreover, the in-distribution and out-of-distribution mutual information (IDMI, resp. OODMI) exhibit different variances but do not seem correlated.**
between any two networks has a very low variance. There is, therefore, an extremely low probability of training two similar networks. This may be explained by the high complexity of the network (here, a ResNet-18), and we refer to Appendix D.2 for results on a smaller architecture. In Figure 3 (center), we see that the variance of the out-of-distribution MI is higher. However, we also note in Figure 3 (right) that in contrast to common intuition (and to results for simpler models), we have, in this case, no significant correlation between the in-distribution and the out-of-distribution MI. This highlights that measuring the in-distribution _diversity_ (here with the MI) may be, in practice, a very poor indicator of the OOD detection performance of a model. Moreover, these results indicate that the complexity of the posterior is orders of magnitude higher than what we understand when taking symmetries into account.

### Frequency of weight permutations during training

We propose a new protocol to evaluate if a network tends to permute during training. Given a DNN \(f_{\mathbf{\omega}}\), we can compute, for each step \(s\) of the training, the permutation set \(\Pi_{s}\) sorting its weights (and removing the symmetries). If the DNN tends to permute during the training, this implies a variation in the \(\Pi_{s}\). We suggest to measure the extent of the variations using the Kendall's \(\tau\) correlation coefficient (Kendall, 1938) between successive permutations \(\Pi_{s}\) and \(\Pi_{s+1}\). We plot the variation of the mean over several training instances and the elements of the permutation sets of Kendall's \(\tau\) in Figure 4. We see that on MNIST (left), the variations of the permutation set are scarce yet gathered around points of instability. These instabilities are due to the sorting mechanism based on the maximum values of the weights of the neurons. We have tried other statistics on the values of the weights, but taking the maximum seems the most stable. We see that the weights nearly never permute in the last phase of the training. The results differ for the ResNet-18 (right) since the number of degrees of freedom is much higher. We see a lot of variation during the phases with a high learning rate (reduced after 25 and 50 epochs). However, as for the first case, we do not see any particular sign of permutations in the last part of the training.

## 6 Conclusion

In this study, we have examined Bayesian neural network posteriors, which are pivotal for understanding uncertainty. Our findings suggest that part of the complexity in these posteriors can be attributed to the non-identifiability of modern neural networks viewing the posterior as a mixture of permuted distributions. To explore this further, we have introduced the _min-mass_ problem to investigate the real impact of scaling symmetries. Using real-world applications, we have proposed a method to assess the quality of the posterior distribution and its correlation with model performance, particularly in terms of uncertainty quantification.

While considering symmetries has provided valuable insights, our discussions hint at a deeper complexity going beyond weight-space symmetries. In future work, we plan to continue our exploration of this intriguing area.

Figure 4: **Evolution during training of the mean Kendall’s \(\tau\) correlation between the permutations towards the identifiable model for all successive steps**: The correlation between successive permutations increases when the learning rate decreases.
## 7 Reproducibility statement

We use publicly available datasets, including MNIST, FashionMNIST, CIFAR100, SVHN, ImageNet-200, and Textures, to ensure transparency and accessibility. Please refer to Appendix B.3 for details on these datasets. Our detailed experimental methods are outlined in Appendices A and B, and the proofs for all theoretical results are provided in Appendix C.

To help the replication of our work, we will share the source code for our experiments on GitHub shortly. Notably, we will release a library that tackles the non-identifiability of neural networks.

For our experiments in Section 4, we rely exclusively on open-source libraries, such as the GitHub repository Bayesian-Neural-Networks for SGHMC, BLiTZ (Esposito, 2020) for variational Bayesian neural networks, and Laplace (Daxberger et al., 2021). For the SWAG method, we also use the publicly available code from the original paper (Maddox et al., 2019). Finally, we estimate the maximum mean discrepancies with a torch version of the code from Schrab et al. (2023) and solved our convex optimization problems (see Definition C.6) with cvxpy(Diamond and Boyd, 2016; Agrawal et al., 2018). The statistical experiments, such as Pearson's \(\rho\) and Kendall's \(\tau\), are performed with SciPy (Virtanen et al., 2020).

## 8 Ethics

Our primary goal in this paper is to improve our comprehension of the Bayesian posterior, which we argue is a fundamental element to understand to contribute to the reliability of machine-learning methods.

We note that training a substantial number of checkpoints for estimating the posterior, especially in the case of the thousand models trained on TinyImageNet, was energy intensive (around 3 Nvidia V100 hours per training). To mitigate the environmental impact, we opted for a carbon-efficient cluster.