$\dagger$$\dagger$footnotetext: corresponding author - gianni.franchi@ensta-
paris.fr

# A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors

Olivier Laurent,1,2 Emanuel Aldea,1 & Gianni Franchi2, $\dagger$
SATIE, Université Paris-Saclay,1 U2IS, ENSTA Paris, Institut Polytechnique de
Paris,2

###### Abstract

The distribution of the weights of modern deep neural networks (DNNs) -
crucial for uncertainty quantification and robustness - is an eminently
complex object due to its extremely high dimensionality. This paper proposes
one of the first large-scale explorations of the posterior distribution of
deep Bayesian Neural Networks (BNNs), expanding its study to real-world vision
tasks and architectures. Specifically, we investigate the optimal approach for
approximating the posterior, analyze the connection between posterior quality
and uncertainty quantification, delve into the impact of modes on the
posterior, and explore methods for visualizing the posterior. Moreover, we
uncover weight-space symmetries as a critical aspect for understanding the
posterior. To this extent, we develop an in-depth assessment of the impact of
both permutation and scaling symmetries that tend to obfuscate the Bayesian
posterior. While the first type of transformation is known for duplicating
modes, we explore the relationship between the latter and L2 regularization,
challenging previous misconceptions. Finally, to help the community improve
our understanding of the Bayesian posterior, we will release shortly the first
large-scale checkpoint dataset, including thousands of real-world models,
along with our codes.

Figure 1: Weight-space symmetries greatly impact the estimated Bayesian
posterior. Permutation symmetries very clearly increase the number of modes of
the posterior distribution in the case of a 2-hidden neuron perceptron.

##  1 Introduction

Despite substantial advancements in deep learning, Deep Neural Networks (DNNs)
remain black box models. Various studies have sought to explore DNN loss
landscapes (Li et al., 2018; Fort & Jastrzebski, 2019; Fort & Scherlis, 2019;
Liu et al., 2022) to achieve a deeper understanding of these models. Recent
works have, for instance, unveiled the interconnection of the modes obtained
with Stochastic Gradient Descent (SGD) via narrow pathways that link pairs of
modes, or through tunnels that connect multiple modes simultaneously (Garipov
et al., 2018; Draxler et al., 2018). This mode connectivity primarily arises
from scale and permutation invariances, which imply that numerous weights can
represent the same exact function (e.g., Entezari et al. (2022)). Several
studies have delved into the relationship between these symmetries and the
characteristics of the loss landscape (Entezari et al., 2022; Neyshabur et
al., 2015; Brea et al., 2019). Our work investigates the connections between
these symmetries and the distribution of DNN weights, a crucial aspect for
uncertainty quantification. As shown in Figure 1, it is apparent that these
symmetries also exert influence on DNN posterior distributions.

Uncertainty quantification plays a pivotal role in high-stakes industrial
applications - such as autonomous driving (Levinson et al., 2011; McAllister
et al., 2017; Sun et al., 2019) \- where reliable predictions and informed
decision-making are paramount. In such critical domains, understanding and
effectively managing uncertainties, particularly the model-related epistemic
uncertainties (Hora, 1996) arising from incomplete knowledge, is essential.
Among the various methods proposed to address these challenges, Bayesian
Neural Networks (BNNs) (Tishby et al., 1989) offer a principled and
theoretically sound approach. BNNs quantify uncertainty by modeling our
beliefs about parameters and outcomes probabilistically (Tishby et al., 1989;
Hinton & Van Camp, 1993). However, this perspective faces significant hurdles
when applied to deep learning, primarily related to scalability (Izmailov et
al., 2021) and the precision of approximations (MacKay, 1995). Due to their
very high dimension, BNNs struggle to estimate the posterior distribution,
i.e., the probability density of converging towards any particular set of
model parameters $\omega$, given the observed data $\mathcal{D}$.

Diverging from methods such as the Maximum Likelihood Estimate or Maximum A
Posteriori (also in Tishby et al. (1989)), which we typically derive through
gradient descent optimization of cross-entropy (with L2 regularization for the
latter), BNNs assign a probability to each possible model (or hypothesis) and
offer predictions considering the full extent of possible models. In
mathematical terms, when we denote the target as $y$, the input vector as
${\bm{x}}$, and the weight space as $\Omega$, we can express this approach
through the following intractable formula, often referred to as the
marginalization on the parameters of the model (Tishby et al., 1989; Rasmussen
et al., 2006):

$\displaystyle
p(y\mid{\bm{x}},\mathcal{D})=\int\limits_{{\bm{\omega}}\in\Omega}p(y\mid{\bm{x}},{\bm{\omega}})p({\bm{\omega}}\mid\mathcal{D})d{\bm{\omega}}.$
(1)

The posterior distribution $p({\bm{\omega}}|\mathcal{D})$ assumes a central
and arguably the most critical role in BNNs. Many successful methods for
quantifying uncertainty can be viewed as attempts to approximate this
posterior, each with its own trade-offs in terms of accuracy and computational
efficiency, as illustrated in previous research (Blundell et al., 2015; Gal &
Ghahramani, 2016; Lakshminarayanan et al., 2017). Prior work (Kuncheva &
Whitaker, 2003; Fort et al., 2019; Ortega et al., 2022) has established the
importance of achieving _diversity_ in the sampled DNNs drawn from the
posterior, particularly when dealing with uncertain input data. However, the
presence of permutation symmetries and scale symmetries among hidden units in
neural networks may lead to an increased number of local minima (Zhao et al.,
2023) with no diversity. In the context of BNNs, this phenomenon can result in
a proliferation of modes within the posterior distribution.

In this paper, we delve into the impact of weight symmetries on the posterior
distribution. While there have been numerous efforts to visualize the loss
landscape, we explore the possibility of conducting similar investigations for
the posterior distribution. Additionally, we introduce a protocol for
assessing the quality of posterior estimation and examine the relationship
between posterior estimation and the accuracy of uncertainty quantification.
Specifically, our contributions are as follows:

(1) We build a new mathematical formalism to highlight the different impacts
of the permutation and scaling symmetries on the posterior and on uncertainty
estimation in deep neural networks. Notably, we explain the seeming
equivalence of the marginals in Figure 1. We also perform the first in-depth
exploration of the existence of scaling symmetries and their overlooked
effect. (2) We evaluate the quality of various methods for estimating the
posterior distribution on real-world applications using the Maximum Mean
Discrepancy, offering a practical benchmark to assess their performance in
capturing uncertainty. (3) We release a new dataset that includes the weights
of thousands of models across various computer vision tasks and architectures,
ranging from MNIST to TinyImageNet. This dataset is intended to facilitate
further exploration and collaboration in the field of uncertainty in deep
learning. (4) Our investigation delves into the proliferation of modes in the
context of posterior symmetries and exhibits the capacity of ensembles to
converge toward non-functionally equivalent modes. Furthermore, we discuss the
influence of symmetries in the training process.

##  2 Related work

##### Epistemic uncertainty, Bayesian inference, and posterior

Epistemic uncertainty (Hora, 1996; Hüllermeier & Waegeman, 2021) plays a
crucial role in accurately assessing predictive model reliability. However,
despite ongoing discussions, estimating this uncertainty remains a challenge.
BNNs (Goan & Fookes, 2020) predominantly shape the landscape of methodologies
that tackle epistemic uncertainties (Gawlikowski et al., 2023). Given the
complexity of dealing with posterior distributions, these approaches have
mostly been tailored for enhanced scalability.

For instance, Hernández-Lobato & Adams (2015) proposed an efficient
probabilistic backpropagation, and Blundell et al. (2015) developed BNNs by
Backpropagation to learn diagonal Gaussian distributions with the
reparametrization trick. Similarly, Laplace methods (Laplace, 1774; MacKay,
1992; Ritter et al., 2018) characterize the posterior distribution of weights,
often focusing on the final layer (Ober & Rasmussen, 2019; Watson et al.,
2021), again for scalability.

On a different approach, Monte Carlo Dropout, introduced by Gal & Ghahramani
(2016) and Kingma et al. (2015), is a framework that models the posterior as a
mixture of Dirac distributions when applied on fully-connected layers.
Broadening the spectrum, Deep Ensembles (Lakshminarayanan et al., 2017),
arguably along with their more efficient alternatives (Lee et al., 2015; Wen
et al., 2019; Maddox et al., 2019; Franchi et al., 2020a; b; Havasi et al.,
2021; Laurent et al., 2023), have been interpreted by Wilson & Izmailov (2020)
as Monte Carlo estimates of Equation 1.

##### Markov-chain-based Bayesian posterior estimation

Neal et al. (2011) introduced HMC as an accurate method for estimating the
posterior of neural networks, but its application to large-scale problems
remains challenging due to the very high computational demands. Izmailov et
al. (2021) managed to scale full-batch HMC to CIFAR-10 (Krizhevsky, 2009) with
ResNet-20 (He et al., 2016) thanks to 512 TPUv3.

In response to these challenges, stochastic approximations of MCMC have gained
attention for their ability to provide computationally more feasible
solutions. A prominent example is the Langevin dynamics-based approach
proposed by Welling & Teh (2011). By introducing noise into the dynamics,
Langevin dynamics allows for more practical implementation on large datasets.

In addition to Langevin dynamics, other stochastic gradient-based methods have
been introduced to improve the efficiency of MCMC sampling. Chen et al. (2014)
presented Stochastic Gradient Hamiltonian Monte Carlo (SGHMC), integrating
stochastic gradients into HMC. Likewise, Zhang et al. (2020) proposed C-SGLD
and C-SGHMC (Cyclic Stochastic Gradient Langevin Dynamics and Hamiltonian
Monte Carlo), introducing controlled noise via cyclic preconditioning.

While stochastic approximation methods offer computational convenience, they
come with the trade-off of slowing down the convergence and potentially
introducing bias into the resulting inference (Bardenet et al., 2017; Zou &
Gu, 2021). As such, the suitability of these approaches depends on the
specific application and the level of acceptable bias in the analysis.

##### Symmetries in neural networks

The seminal work from Hecht-Nielsen (1990) established a foundational
understanding by investigating permutation symmetries and setting a lower
bound on symmetries in multi-layer perceptrons. Albertini et al. (1993)
extended this work and studied flip sign symmetries in networks with odd
activation functions. This work was further generalized to a broader range of
activation functions by Kůrková & Kainen (1994), who proposed symmetry removal
to streamline evolutionary algorithms.

Recent advancements have generalized symmetries to modern neural
architectures. Neyshabur et al. (2015) explored the scaling symmetries that
arise in architectures containing non-negative homogeneous activation
functions, including Nair & Hinton (2010)’s ubiquitous Rectified Linear Unit
(ReLU). This perspective extends our understanding of symmetries to ReLU-
powered architectures, e.g., AlexNet (Krizhevsky et al., 2012), VGG (Simonyan
& Zisserman, 2015), and ResNet networks (He et al., 2016). This paper focuses
on scaling and permutation symmetries, but other works, such as Rolnick &
Kording (2020); Grigsby et al. (2023), unveil less apparent symmetries.

Closest to our work, Wiese et al. (2023) demonstrated that taking weight-space
symmetries into account could reduce the Bayesian posterior support and
improve Monte-Carlo Markov Chains posterior estimation.

##  3 Symmetries increase the complexity of Bayesian posteriors

We propose to study the most influential weight-space symmetries and their
properties related to the Bayesian posterior. Firstly, weight-space symmetries
transform the parameters of the neural networks while keeping the networks
functionally invariant; in other words,

######  Definition 3.1.

Let $f_{\bm{\omega}}$ be a neural network of parameters ${\bm{\omega}}$ taking
$n$-dimensional vectors as inputs. We say that the transformation
$\mathcal{T}$ modifying ${\bm{\omega}}$ is a weight-space symmetry operator,
iff

$f_{\mathcal{T}({\bm{\omega}})}=f_{\bm{\omega}},\ \text{or}\ \
\forall{\bm{x}}\in\mathbb{R}^{n},f_{\mathcal{T}({\bm{\omega}})}({\bm{x}})=f_{\bm{\omega}}({\bm{x}}).$
(2)

With the notation $f_{\mathcal{T}({\bm{\omega}})}({\bm{x}})$, we apply the
symmetry operator $\mathcal{T}$ on the weights ${\bm{\omega}}$, resulting in a
set of modified weights. In this paper, we show that scaling and permutation
symmetries have different impacts on the posterior of neural networks. They
can, for instance, complicate Bayesian posteriors, creating artificial
functionally equivalent modes.

###  3.1 A gentle example of artificial symmetry-based posterior modes

To illustrate the considerable impact of symmetries on the Bayesian posterior,
we propose a small-scale example in Figure 1. We generate this example by
training two-hidden-neuron perceptrons on linearly separable data. Figure 1
presents the estimation of the posterior of the output weights with 10,000
checkpoints. In this figure, the most important mode is duplicated due to the
permutation symmetry, which appears to symmetrize the graph. On the other
hand, scaling symmetries seem to regroup some points around the modes. We
detail this toy experiment in Appendix A. In the following, we develop a new
mathematical framework tailored to help understand the impact of these
symmetries on the posterior, devise mathematical insights explaining these
intuitions, and propose empirical explorations.

###  3.2 Background and definitions

The full extent of this formalism (including sketches of proofs, other
definitions, properties, and propositions) is developed in Appendix C. We deem
that the following properties are the minimal information necessary to
understand the impact on the Bayesian posterior of the two main types of
symmetries: the scaling and permutation symmetries. This part summarizes the
most important results for multi-layer perceptrons, but we provide leads for
generalizing our results to modern deep neural networks such as convolutional
residual networks (He et al., 2016) in Appendix D.4.

###  3.3 Scaling symmetries

For clarity, we propose the following definitions and properties for two-layer
fully connected perceptrons without loss of generality. Please refer to
Appendix C and D.4 for extensions. We start with two new operators between
vectors and matrices to define scaling symmetries.

######  Definition 3.2.

We denote the line-wise product as $\bigtriangledown$ and the column-wise
product as $\rhd$. Let ${\bm{\omega}}\in\mathbb{R}^{n\times m}$,
$\bm{\lambda}\in\mathbb{R}^{m}$, and ${\bm{\mu}}\in\mathbb{R}^{n}$, with
$n,m\geq 1$,

$\forall i\in\llbracket 1,n\rrbracket,\forall j\in\llbracket 1,m\rrbracket,\
(\bm{\lambda}\bigtriangledown{\bm{\omega}})_{i,j}=\lambda_{j}\omega_{i,j}\
\text{and}\ ({\bm{\mu}}\rhd{\bm{\omega}})_{i,j}=\mu_{i}\omega_{i,j}.$ (3)

Given that the rectified linear unit $r$ is non-negative homogeneous - i.e.,
for all non-negative $\lambda$, $r(\lambda x)=\lambda r(x)$ \- we have the
following core property for scaling symmetries (Neyshabur et al., 2015),
trivially extendable to additive biases:

######  Property 3.1.

For all ${\bm{\theta}}\in\mathbb{R}^{\cdot\times m}$,
${\bm{\omega}}\in\mathbb{R}^{m\times n}$,
$\bm{\lambda}\in\left(\mathbb{R}_{>0}\right)^{m}$,

$\forall{\bm{x}}\in\mathbb{R}^{n},\
(\bm{\lambda}^{-1}\bigtriangledown{\bm{\theta}})\times
r(\bm{\lambda}\rhd{\bm{\omega}}{\bm{x}})={\bm{\theta}}\times
r({\bm{\omega}}{\bm{x}}).$ (4)

When defining $\mathcal{T}_{s}$ by the transformation Equation 4 \- in the
case of a two-layer perceptron - the core property directly follows, with the
set of parameters $\Lambda=\\{\bm{\lambda}\\}$:

######  Property 3.2 (Scaling symmetries).

The scaling operation $\mathcal{T}_{s}$ with a set of non-negative parameters
$\Lambda$ is a symmetry for all neural network $f_{\bm{\omega}}$, i.e.

$\forall{\bm{x}}\in\mathbb{R}^{n},\
f_{\mathcal{T}_{s}({\bm{\omega}},\Lambda)}({\bm{x}})=f_{\bm{\omega}}({\bm{x}}).$
(5)

###  3.4 Permutation symmetries

We also propose a simple and intuitive formalism for permutation symmetries,
multiplying the weights by permutation matrices. We have the following
property for two-layer perceptions, with $P_{m}$ the set of $m$-dimensional
permutation matrices:

######  Property 3.3.

For all ${\bm{\theta}}\in\mathbb{R}^{\cdot\times m}$,
${\bm{\omega}}\in\mathbb{R}^{m\times n}$, and permutation matrices
$\bm{\pi}\in P_{m}$,

$\forall{\bm{x}}\in\mathbb{R}^{n},\ {\bm{\theta}}\bm{\pi}^{\intercal}\times
r\left(\bm{\pi}\times{\bm{\omega}}{\bm{x}}\right)={\bm{\theta}}\times
r({\bm{\omega}}{\bm{x}}).$ (6)

The left term of Equation 6 is the definition of the permutation symmetry
operator of parameter $\Pi=\\{\bm{\pi}\\}$ for a network with two layers. In
general, we have the following property:

######  Property 3.4 (Permutation symmetries).

The permutation operation $\mathcal{T}_{p}$ with a set of parameters $\Pi$ is
a symmetry for all neural networks $f_{\bm{\omega}}$, i.e.,

$\forall{\bm{x}}\in\mathbb{R}^{n},\
f_{\mathcal{T}_{p}({\bm{\omega}},\Pi)}({\bm{x}})=f_{\bm{\omega}}({\bm{x}}).$
(7)

###  3.5 The Bayesian posterior as a mixture of distributions

With this formalism, we can establish the following proposition, clarifying
the impact of weight-space symmetries on the Bayesian posterior.

######  Proposition 1.

Define $f_{\omega}$ a neural network and $f_{\tilde{\omega}}$ its
corresponding identifiable model - a network transformed for having sorted
unit-normed neurons. Let us also denote $\mathbb{\Pi}$ and $\mathbb{\Lambda}$,
the sets of permutation sets and scaling sets, respectively, and
$\tilde{\Omega}$ the random variable of the sorted weights with unit norm. The
Bayesian posterior of a neural network $f_{\omega}$ trained with stochastic
gradient descent can be expressed as a continuous mixture of a discrete
mixture:

$p(\Omega={\bm{\omega}}\mid\mathcal{D})=\int\limits_{\Lambda\in\mathbb{\Lambda}}|\mathbb{\Pi}|^{-1}\sum\limits_{\Pi\in\mathbb{\Pi}}p(\tilde{\Omega}=\mathcal{T}_{p}(\mathcal{T}_{s}({\bm{\omega}},\Lambda),\Pi),\Lambda\mid\mathcal{D})d\Lambda.$
(8)

Proposition 1 provides an expression of the Bayesian posterior that highlights
the redundancy of the resulting distribution, explaining the symmetry in
Figure 1 (left). Interestingly, a direct corollary of this formula is that
_all the layer-wise marginal distributions are identical_. In Equation 8, the
permutations play a transparent role, being independent of ${\bm{\omega}}$
(except for strongly quantized spaces for ${\bm{\omega}}$, but we leave this
particular case for future works). On the other hand, the part played by
scaling symmetries is more complex and we discuss their impact in the
following section.

###  3.6 On the effective impact of scaling symmetries

The equiprobability of permutations in Equation 8 leads to a balanced mixture
of $|\mathbb{\Pi}|$ permuted terms. We have no such result on scaling
symmetries since the standard L2-regularized loss is not invariant to scaling
symmetries (contrary to permutations). This absence of invariance obscures the
impacts of scaling symmetries, which mostly remains to be addressed, although
their ”reduction” due to regularization was mentioned in, e.g., Godfrey et al.
(2022). To the best of our knowledge, we provide the first analysis of the
reality of scaling symmetries and their impact on the Bayesian posterior. With
this objective in mind, we propose the following problem.

######  Definition 3.3.

Let $f_{\omega}$ be a neural network and $\bar{\omega}$ its weights without
the biases. We define the minimum scaled network mass problem (or _the min-
mass problem_) as the minimization of the L2-regularization term of
$f_{\omega}$ (the ”mass”) under scaling transformations. In other words,

$m^{*}=\min\limits_{\Lambda\in\mathbb{\Lambda}}\
\left|\mathcal{T}_{s}(\bar{\omega},\Lambda)\right|^{2}.$ (9)

This problem has interesting properties. Notably, we show in Appendix C.7 that
the _min-mass_ problem is log-log strictly convex (Boyd & Vandenberghe, 2004;
Agrawal et al., 2019):

######  Proposition 2.

The minimum scaled network mass problem is log-log strictly convex. As such,
it is equivalent to a strictly convex problem on
$\mathbb{R}^{|\mathbb{\Lambda}|}$ and admits a global minimum attained in a
single point denoted $\Lambda^{*}$.

Figure 2: OptuNet trained with weight decay never converges towards the
_minimum mass_. We note that the maxima of the weights of scaled OptuNets at
the minimum mass tend to be greater for lighter layers than in the original
networks.

It follows from Proposition 2 that, if not already optimal at convergence,
there is an infinite number of equivalent networks with training loss lower
than the original network. We put the proposition into practice in Figure 2
using trained OptuNets (see Appendix B.2). We measure their mass distribution
and compare it to the masses of the optimal networks found with convex
optimization. In practice, the effect of scaling symmetries remains present
even with weight decay: neural networks always seem subject to scaling
symmetries. For completeness, we report the values of the largest elements of
each layer, showing that the minimization tends to increase the maxima of the
layers with fewer parameters yet does not seem to promote unfeasible values.

In this section, we provided theoretical insights on the contrasted impacts of
both scaling and permutation symmetries. In the following, we perform more
empirical studies and propose to evaluate the link between posterior quality
and experimental performance.

##  4 Comparing the estimations of the posterior by approximate Bayesian
methods

|  | Method | MMD $\downarrow$ | NS $\downarrow$ | Acc $\uparrow$ | ECE $\downarrow$ | Brier $\downarrow$ | AUPR $\uparrow$ | FPR95 $\downarrow$ |  IDMI $\downarrow$ |  OODMI $\uparrow$
---|---|---|---|---|---|---|---|---|---|---|---
MNIST - OptuNet |  One Mode | Dropout | 15.0 | 14.3 | 83.3 | 33.4 | 26.1 | 96.4 | 98.6 | 26.1 | 22.2
BNN | 18.8 | 17.1 | 78.1 | 7.4 | 30.9 | 67.9 | 93.7 | 0.1 | 0.1
SGHMC | 16.7 | 17.7 | 95.1 | 7.6 | 2.8 | 73.7 | 98.4 | 4.3 | 14.5
SWAG | 16.0 | 14.6 | 88.3 | 17.7 | 4.9 | 73.4 | 68.6 | 4.0 | 8.7
Laplace | 10.6 | 9.5 | 87.9 | 18.1 | 4.8 | 48.2 | 74.6 | 6.2 | 5.9
Multi Mode | Dropout | 2.1 | 2.1 | 92.1 | 29.2 | 36.8 | 97.2 | 78.2 | 36.6 | 52.5
BNN | 2.8 | 2.5 | 86.5 | 17.5 | 24.4 | 96.9 | 27.2 | 21.1 | 52.3
SWAG | 1.8 | 1.3 | 95.0 | 13.1 | 17.5 | 88.7 | 24.6 | 27.6 | 62.2
Laplace | 1.8 | 0.8 | 94.8 | 12.8 | 15.8 | 95.4 | 32.1 | 21.1 | 52.2
DE | 0.0 | 0.0 | 95.3 | 10.7 | 13.5 | 95.7 | 12.8 | 19.3 | 62.6
CIFAR100 - ResNet18 |  One Mode | Dropout | 4.5 | 7.5 | 69.3 | 12.1 | 44.3 | 80.1 | 64.0 | 6.0 | 9.6
BNN | 9.0 | 10.2 | 57.6 | 21.8 | 62.5 | 81.7 | 62.6 | 0.8 | 2.2
SGHMC | 7.5 | 7.9 | 69.3 | 4.3 | 41.5 | 87.5 | 41.4 | 0.0 | 0.1
SWAG | 6.7 | 7.2 | 66.7 | 1.7 | 43.8 | 30.7 | 17.1 | 2.7 | 14.7
Laplace | 5.7 | 7.0 | 70.7 | 1.4 | 40.0 | 84.5 | 40.6 | 31.3 | 70.6
Multi Mode | Dropout | 0.7 | 4.5 | 75.5 | 4.8 | 64.2 | 93.5 | 23.7 | 23.8 | 77.0
BNN | 6.1 | 5.6 | 66.3 | 1.3 | 45.6 | 91.0 | 30.9 | 44.5 | 110.4
SWAG | 5.0 | 5.4 | 68.4 | 2.3 | 42.1 | 97.9 | 17.1 | 7.5 | 33.0
Laplace | 0.6 | 4.3 | 75.2 | 7.8 | 35.2 | 95.1 | 20.1 | 50.5 | 128.3
DE | 0.0 | 0.0 | 75.9 | 2.3 | 33.3 | 97.1 | 16.1 | 26.5 | 90.1
TinyImageNet - ResNet18 |  One Mode | Dropout | 9.5 | 4.9 | 56.4 | 14.2 | 60.3 | 70.8 | 87.6 | 9.5 | 13.5
BNN | / | / | / | / | / | / | / | / | /
SGHMC | 9.8 | 5.3 | 54.4 | 2.3 | 58.9 | 78.1 | 74.4 | 0.1 | 0.2
SWAG | 9.1 | 3.9 | 61.7 | 10.0 | 51.8 | 84.6 | 66.7 | 3.3 | 9.0
Laplace | 5.5 | 6.1 | 29.7 | 7.4 | 81.3 | 62.3 | 82.3 | 211.7 | 254.8
Multi Mode | Dropout | 4.3 | 1.8 | 65.3 | 10.5 | 48.2 | 90.9 | 57.7 | 41.9 | 93.2
BNN | / | / | / | / | / | / | / | / | /
SWAG | 6.7 | 5.4 | 64.8 | 2.3 | 46.6 | 98.4 | 53.6 | 20.5 | 55.2
Laplace | 0.5 | 3.1 | 34.4 | 13.0 | 79.3 | 64.4 | 77.2 | 227.4 | 267.2
DE | 0.0 | 0.0 | 65.4 | 8.5 | 47.3 | 94.2 | 41.6 | 44.7 | 108.5

Table 1: Comparison of popular methods approximating the Bayesian posterior.
All scores are expressed in %, except the MMDs for ResNet-18 networks,
expressed in ‰. Acc stands for accuracy, and IDMI and OODMI are the in-
distribution and out-of-distribution mutual information. NS is the MMD
computed after the removal of the symmetries, and DE stands for Deep
Ensembles. Multi-mode methods use ten independently trained models.

In this section, we leverage symmetries to compare popular single-mode
methods, namely, Monte Carlo Dropout (Gal & Ghahramani, 2016), Stochastic
Weight Averaging Gaussian (SWAG) proposed by Maddox et al. (2019), Bayes by
backpropagation BNNs (Blundell et al., 2015), and Laplace methods (Ritter et
al., 2018). We also include their multi-modal variations, corresponding to the
application of these methods on 10 different independently trained models, as
well as SGHMC (Chen et al., 2014) and Deep Ensembles (DE) highlighted by
Lakshminarayanan et al. (2017) and proposed earlier by Hansen & Salamon
(1990). We compare these methods on 3 image classification tasks with
different levels of difficulty, ranging from MNIST (LeCun et al., 1998) with
our OptuNet (392 parameters) to CIFAR-100 (Krizhevsky, 2009) and Tiny-ImageNet
(Deng et al., 2009) with ResNet-18 (He et al., 2016). To this extent, we
propose leveraging maximum mean discrepancy (MMD) to estimate the
dissimilarities between the high-dimensional posterior distributions. We
estimate the target distribution with 1000 checkpoints and compare it to 100
samples extracted from each of the previously-mentioned techniques. We detail
these experiments in Appendix B.

###  4.1 Evaluating the quality of the estimation of the Bayesian posterior

One approach to assess the similarities between distributions involves
estimating the distributions and subsequently quantifying the distance between
these estimated distributions (Smola et al., 2007; Sriperumbudur et al.,
2010). However, these methods can become impractical when dealing with
distributions in extremely high-dimensional spaces, such as the posterior of
DNNs. An alternative solution is to embed the probability measures into a
Reproducing Kernel Hilbert Space (RKHS) developed by, e.g., Bergmann (1922);
Aronszajn (1950); Schwartz (1964). Within this framework, a distance metric
MMD (Song, 2008) \- defined as the distance between the respective mean
elements within the RKHS - is used to quantify the dissimilarity between the
distributions. Gretton et al. (2012) proposed to leverage MMD for efficient
two-sample tests in high dimensions. For better efficacy, MMDs are computed on
RKHS that allow for comparing all of their moments. In Table 1, we propose to
compute the median of the aggregated MMDs (Schrab et al., 2023) with multiple
Gaussian and Laplace kernels, with and without symmetries.

For tractability, we report the mean - weighted by the number of parameters of
each layer - of the median over the twenty MMD kernels between the layer-wise
estimated posterior and the posterior approximated by each method. The
estimated posterior includes 1000 independently trained checkpoints, and the
posterior approximation of each method includes 100 samples. Please refer to
Appendix B for extended details (including the means and maxima of the MMDs).

###  4.2 Performance metrics and OOD datasets

On top of the MMD quantifying the difference between the posterior
estimations, we measure several empirical performance metrics. We evaluate the
overall performance of the models using the accuracy and the Brier score
(Brier, 1950; Gneiting et al., 2007). Furthermore, we choose the binned
Expected Calibration Error (ECE) (Naeini et al., 2015) for top-label
calibration and measure the quality of the out-of-distribution (OOD) detection
using the area under the precision-recall curve (AUPR) and the false positive
rate at $95\%$ recall (FPR95), as recommended by Hendrycks & Gimpel (2017), as
a measure of OOD detection abilities, that are supposed to correlate with the
quality of the estimated posterior. Finally, we report the mean diversity of
the predictions in each ensemble through the mutual information (MI) (e.g.,
Ash (1965)), often used to measure epistemic uncertainty (Kendall & Gal, 2017;
Ovadia et al., 2019). We use FashionMNIST (Xiao et al., 2017), SVHN (Netzer et
al., 2011), and Textures (Cimpoi et al., 2014) as OOD datasets for MNIST,
CIFAR-100, and TinyImageNet, respectively.

###  4.3 Results

In our examination of Table 1 emerges that multi-mode techniques consistently
demonstrate superior performance in terms of MMD when compared to their
single-mode counterparts. This trend holds true in accuracy and negative log-
likelihood. However, an intriguing divergence arises when considering the ECE,
where techniques focusing solely on estimating a single mode often exhibit
superior performance.

Turning our attention to the assessment of epistemic uncertainty, as
quantified by AUPR and FPR95, multi-mode techniques, notably multi-SWAG, and
Deep Ensembles, consistently outperform other methods. This underscores the
strong connection between posterior estimation and the accuracy of epistemic
uncertainty quantification. However, we note that the quality of aleatoric
uncertainty quantification does not consistently correlate with that of the
posterior distribution estimation.

The final two columns of the table shed light on the diversity of the models
sampled from the posterior. The objective here is to minimize in-distribution
mutual information (IDMI) while concurrently maximizing out-of-distribution
mutual information (OODMI). An analysis shows that mono-mode methods tend to
yield lower values for both IDMI and OODMI than multi-mode methods, which tend
to exhibit higher IDMI and OODMI values, suggesting greater _diversity_.

##  5 Discussions

We develop further insights on the posterior of Bayesian Neural Networks in
relationship with symmetries. Notably, we evaluate the risk of _functional
collapse_ , i.e., of training very similar networks in Section 5.1 and we
discuss the frequency of weights permutations in 5.2. We expand these
discussions, add observations on the evaluation of the number of modes of the
posterior, and propose visualizations in Appendix D.

###  5.1 Functional collapse in ensembles: a study of ID and OOD disagreements

Figure 3: Experiments show no hint of a functional collapse between couples of
independently trained ResNet-18 on CIFAR-100. Moreover, the in-distribution
and out-of-distribution mutual information (IDMI, resp. OODMI) exhibit
different variances but do not seem correlated.

Given the potentially very high number of equivalent modes due to permutation
symmetries (see Section 3.5), we support broadening the concept of collapse in
the parameter-space (e.g., in D’Angelo & Fortuin (2021)) to _functional
collapse_ to account for the impact of symmetries on the posterior. Parameter-
space collapse is more restrictive and may not be formally involved when
ensemble members lack diversity. It is also much harder to characterize as it
would require an analysis of the loss landscape, at the very least.

We propose here to quantify functional collapse as a potential ground for the
need for more complex repulsive ensembling methods (Masegosa, 2020; Rame &
Cord, 2021). To this extent, we randomly select 1000 ResNet-18 (trained to
estimate the Bayesian posterior in Section 4) and compute the mean over the
test set of their pairwise mutual information, quantifying the divergence
between the single models and their average. We make these measures on in-
distribution and out-of-distribution data (CIFAR-100 (ID) and SVHN (OOD)). In
Figure 3 (left), we see that the in-distribution MI between any two networks
has a very low variance. There is, therefore, an extremely low probability of
training two similar networks. This may be explained by the high complexity of
the network (here, a ResNet-18), and we refer to Appendix D.2 for results on a
smaller architecture. In Figure 3 (center), we see that the variance of the
out-of-distribution MI is higher. However, we also note in Figure 3 (right)
that in contrast to common intuition (and to results for simpler models), we
have, in this case, no significant correlation between the in-distribution and
the out-of-distribution MI. This highlights that measuring the in-distribution
_diversity_ (here with the MI) may be, in practice, a very poor indicator of
the OOD detection performance of a model. Moreover, these results indicate
that the complexity of the posterior is orders of magnitude higher than what
we understand when taking symmetries into account.

###  5.2 Frequency of weight permutations during training

Figure 4: Evolution during training of the mean Kendall’s $\tau$ correlation
between the permutations towards the identifiable model for all successive
steps: The correlation between successive permutations increases when the
learning rate decreases.

We propose a new protocol to evaluate if a network tends to permute during
training. Given a DNN $f_{\bm{\omega}}$, we can compute, for each step $s$ of
the training, the permutation set $\Pi_{s}$ sorting its weights (and removing
the symmetries). If the DNN tends to permute during the training, this implies
a variation in the $\Pi_{s}$. We suggest to measure the extent of the
variations using the Kendall’s $\tau$ correlation coefficient (Kendall, 1938)
between successive permutations $\Pi_{s}$ and $\Pi_{s+1}$. We plot the
variation of the mean over several training instances and the elements of the
permutation sets of Kendall’s $\tau$ in Figure 4. We see that on MNIST (left),
the variations of the permutation set are scarce yet gathered around points of
instability. These instabilities are due to the sorting mechanism based on the
maximum values of the weights of the neurons. We have tried other statistics
on the values of the weights, but taking the maximum seems the most stable. We
see that the weights nearly never permute in the last phase of the training.
The results differ for the ResNet-18 (right) since the number of degrees of
freedom is much higher. We see a lot of variation during the phases with a
high learning rate (reduced after 25 and 50 epochs). However, as for the first
case, we do not see any particular sign of permutations in the last part of
the training.

##  6 Conclusion

In this study, we have examined Bayesian neural network posteriors, which are
pivotal for understanding uncertainty. Our findings suggest that part of the
complexity in these posteriors can be attributed to the non-identifiability of
modern neural networks viewing the posterior as a mixture of permuted
distributions. To explore this further, we have introduced the _min-mass_
problem to investigate the real impact of scaling symmetries. Using real-world
applications, we have proposed a method to assess the quality of the posterior
distribution and its correlation with model performance, particularly in terms
of uncertainty quantification.

While considering symmetries has provided valuable insights, our discussions
hint at a deeper complexity going beyond weight-space symmetries. In future
work, we plan to continue our exploration of this intriguing area.

##  7 Reproducibility statement

We use publicly available datasets, including MNIST, FashionMNIST, CIFAR100,
SVHN, ImageNet-200, and Textures, to ensure transparency and accessibility.
Please refer to Appendix B.3 for details on these datasets. Our detailed
experimental methods are outlined in Appendices A and B, and the proofs for
all theoretical results are provided in Appendix C.

To help the replication of our work, we will share the source code for our
experiments on GitHub shortly. Notably, we will release a library that tackles
the non-identifiability of neural networks.

For our experiments in Section 4, we rely exclusively on open-source
libraries, such as the GitHub repository Bayesian-Neural-Networks for SGHMC,
BLiTZ (Esposito, 2020) for variational Bayesian neural networks, and Laplace
(Daxberger et al., 2021). For the SWAG method, we also use the publicly
available code from the original paper (Maddox et al., 2019). Finally, we
estimate the maximum mean discrepancies with a torch version of the code from
Schrab et al. (2023) and solved our convex optimization problems (see
Definition C.6) with cvxpy (Diamond & Boyd, 2016; Agrawal et al., 2018). The
statistical experiments, such as Pearson’s $\rho$ and Kendall’s $\tau$, are
performed with SciPy (Virtanen et al., 2020).

##  8 Ethics

Our primary goal in this paper is to improve our comprehension of the Bayesian
posterior, which we argue is a fundamental element to understand to contribute
to the reliability of machine-learning methods.

We note that training a substantial number of checkpoints for estimating the
posterior, especially in the case of the thousand models trained on
TinyImageNet, was energy intensive (around 3 Nvidia V100 hours per training).
To mitigate the environmental impact, we opted for a carbon-efficient cluster.

## References

  * Agrawal et al. (2018) Akshay Agrawal, Robin Verschueren, Steven Diamond, and Stephen Boyd.  A rewriting system for convex optimization problems.  _Journal of Control and Decision_ , 2018.
  * Agrawal et al. (2019) Akshay Agrawal, Steven Diamond, and Stephen Boyd.  Disciplined geometric programming.  _Optimization Letters_ , 2019.
  * Akiba et al. (2019) Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.  Optuna: A next-generation hyperparameter optimization framework.  In _KDD_ , 2019.
  * Albertini et al. (1993) Francesca Albertini, Eduardo D Sontag, and Vincent Maillot.  Uniqueness of weights for neural networks.  _Artificial Neural Networks for Speech and Vision_ , 1993.
  * Aronszajn (1950) Nachman Aronszajn.  Theory of reproducing kernels.  _Transactions of the American mathematical society_ , 1950.
  * Ash (1965) Robert B Ash.  _Information theory_.  Dover Publication, 1965.
  * Bardenet et al. (2017) Rémi Bardenet, Arnaud Doucet, and Chris Holmes.  On markov chain monte carlo methods for tall data.  _Journal of Machine Learning Research_ , 2017.
  * Bergmann (1922) Stefan Bergmann.  Über die entwicklung der harmonischen funktionen der ebene und des raumes nach orthogonalfunktionen.  _Mathematische Annalen_ , 1922.
  * Bergstra et al. (2011) James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl.  Algorithms for hyper-parameter optimization.  _NeurIPS_ , 2011.
  * Blundell et al. (2015) Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight uncertainty in neural network.  In _ICML_ , 2015.
  * Boyd & Vandenberghe (2004) Stephen P Boyd and Lieven Vandenberghe.  _Convex optimization_.  Cambridge university press, 2004.
  * Brea et al. (2019) Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner.  Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape.  _arXiv preprint arXiv:1907.02911_ , 2019.
  * Brier (1950) Glenn W Brier.  Verification of forecasts expressed in terms of probability.  _Monthly weather review_ , 1950.
  * Brosse et al. (2020) Nicolas Brosse, Carlos Riquelme, Alice Martin, Sylvain Gelly, and Éric Moulines.  On last-layer algorithms for classification: Decoupling representation from uncertainty estimation.  _arXiv preprint arXiv:2001.08049_ , 2020.
  * Carreira-Perpinán & Williams (2003) Miguel A Carreira-Perpinán and Christopher KI Williams.  On the number of modes of a gaussian mixture.  In _ICSSTCV_ , 2003.
  * Chen et al. (2014) Tianqi Chen, Emily Fox, and Carlos Guestrin.  Stochastic gradient hamiltonian monte carlo.  In _ICML_ , 2014.
  * Cimpoi et al. (2014) M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi.  Describing textures in the wild.  In _CVPR_ , 2014.
  * Comaniciu & Meer (2002) Dorin Comaniciu and Peter Meer.  Mean shift: A robust approach toward feature space analysis.  _IEEE Transactions on pattern analysis and machine intelligence_ , 2002.
  * D’Angelo & Fortuin (2021) Francesco D’Angelo and Vincent Fortuin.  Repulsive deep ensembles are bayesian.  In _NeurIPS_ , 2021.
  * Daxberger et al. (2021) Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig.  Laplace redux-effortless bayesian deep learning.  In _NeurIPS_ , 2021.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _CVPR_ , 2009.
  * Diamond & Boyd (2016) Steven Diamond and Stephen Boyd.  CVXPY: A Python-embedded modeling language for convex optimization.  _Journal of Machine Learning Research_ , 2016.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_ , 2021.
  * Draxler et al. (2018) Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht.  Essentially no barriers in neural network energy landscape.  In _ICML_ , 2018.
  * Eisenberger (1964) Isidore Eisenberger.  Genesis of bimodal distributions.  _Technometrics_ , 1964.
  * Entezari et al. (2022) Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur.  The role of permutation invariance in linear mode connectivity of neural networks.  _ICLR_ , 2022.
  * Esposito (2020) Piero Esposito.  Blitz - bayesian layers in torch zoo, 2020.
  * Fort & Jastrzebski (2019) Stanislav Fort and Stanislaw Jastrzebski.  Large scale structure of neural network loss landscapes.  In _NeurIPS_ , 2019.
  * Fort & Scherlis (2019) Stanislav Fort and Adam Scherlis.  The goldilocks zone: Towards better understanding of neural network loss landscapes.  In _AAAI_ , 2019.
  * Fort et al. (2019) Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan.  Deep ensembles: A loss landscape perspective.  _arXiv preprint arXiv:1912.02757_ , 2019.
  * Franchi et al. (2020a) Gianni Franchi, Andrei Bursuc, Emanuel Aldea, Séverine Dubuisson, and Isabelle Bloch.  Encoding the latent posterior of bayesian neural networks for uncertainty quantification.  _arXiv preprint arXiv:2012.02818_ , 2020a.
  * Franchi et al. (2020b) Gianni Franchi, Andrei Bursuc, Emanuel Aldea, Séverine Dubuisson, and Isabelle Bloch.  Tradi: Tracking deep neural network weight distributions.  In _ECCV_ , 2020b.
  * Gal & Ghahramani (2016) Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _ICML_ , 2016.
  * Garipov et al. (2018) Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson.  Loss surfaces, mode connectivity, and fast ensembling of dnns.  In _NeurIPS_ , 2018.
  * Gawlikowski et al. (2023) Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al.  A survey of uncertainty in deep neural networks.  _Artificial Intelligence Review_ , 2023.
  * Gneiting et al. (2007) Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery.  Probabilistic forecasts, calibration and sharpness.  _Journal of the Royal Statistical Society Series B: Statistical Methodology_ , 2007.
  * Goan & Fookes (2020) Ethan Goan and Clinton Fookes.  Bayesian neural networks: An introduction and survey.  _Case Studies in Applied Bayesian Data Science_ , 2020.
  * Godfrey et al. (2022) Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge.  On the symmetries of deep learning models and their internal representations.  _NeurIPS_ , 2022.
  * Gretton et al. (2012) Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola.  A kernel two-sample test.  _Journal of Machine Learning Research_ , 2012.
  * Grigsby et al. (2023) Elisenda Grigsby, Kathryn Lindsey, and David Rolnick.  Hidden symmetries of relu networks.  In _ICML_ , 2023.
  * Hansen & Salamon (1990) Lars Kai Hansen and Peter Salamon.  Neural network ensembles.  _IEEE transactions on pattern analysis and machine intelligence_ , 1990.
  * Havasi et al. (2021) Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew Mingbo Dai, and Dustin Tran.  Training independent subnetworks for robust prediction.  In _ICLR_ , 2021.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Hecht-Nielsen (1990) Robert Hecht-Nielsen.  On the algebraic structure of feedforward network weight spaces.  In _Advanced Neural Computers_. North-Holland, 1990.
  * Hendrycks & Gimpel (2017) Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In _ICLR_ , 2017.
  * Hernández-Lobato & Adams (2015) José Miguel Hernández-Lobato and Ryan Adams.  Probabilistic backpropagation for scalable learning of bayesian neural networks.  In _ICML_ , 2015.
  * Hinton & Van Camp (1993) Geoffrey E Hinton and Drew Van Camp.  Keeping the neural networks simple by minimizing the description length of the weights.  In _COLT_ , 1993.
  * Hora (1996) Stephen C Hora.  Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management.  _Reliability Engineering & System Safety_, 1996.
  * Hüllermeier & Waegeman (2021) Eyke Hüllermeier and Willem Waegeman.  Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.  _Machine Learning_ , 2021.
  * Ioffe & Szegedy (2015) Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _ICML_ , 2015.
  * Izmailov et al. (2021) Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson.  What are bayesian neural network posteriors really like?  In _ICML_ , 2021.
  * Kendall & Gal (2017) Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  In _NeurIPS_ , 2017.
  * Kendall (1938) Maurice G Kendall.  A new measure of rank correlation.  _Biometrika_ , 1938.
  * Kingma & Ba (2015) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _ICLR_ , 2015.
  * Kingma et al. (2015) Durk P Kingma, Tim Salimans, and Max Welling.  Variational dropout and the local reparameterization trick.  In _NeurIPS_ , 2015.
  * Krizhevsky (2009) Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, MIT, 2009.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _NeurIPS_ , 2012.
  * Kuncheva & Whitaker (2003) Ludmila I Kuncheva and Christopher J Whitaker.  Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy.  _Machine learning_ , 2003.
  * Kurle et al. (2021) Richard Kurle, Tim Januschowski, Jan Gasthaus, and Yuyang Bernie Wang.  On symmetries in variational bayesian neural nets.  In _NeurIPSW_ , 2021.
  * Kůrková & Kainen (1994) Věra Kůrková and Paul Kainen.  Functionally equivalent feedforward neural networks.  _Neural Computation_ , 1994.
  * Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In _NeurIPS_ , 2017.
  * Laplace (1774) Pierre-Simon Laplace.  Mémoire sur la probabilité de causes par les évenements.  _Mémoire de l’académie royale des sciences_ , 1774.
  * Laurent et al. (2023) Olivier Laurent, Adrien Lafage, Enzo Tartaglione, Geoffrey Daniel, Jean-Marc Martinez, Andrei Bursuc, and Gianni Franchi.  Packed-ensembles for efficient uncertainty estimation.  In _ICLR_ , 2023.
  * LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 1998.
  * Lee et al. (2015) Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra.  Why m heads are better than one: Training a diverse ensemble of deep networks.  _arXiv preprint arXiv:1511.06314_ , 2015.
  * Levinson et al. (2011) Jesse Levinson, Jake Askeland, Jan Becker, Jennifer Dolson, David Held, Soeren Kammel, J. Zico Kolter, Dirk Langer, Oliver Pink, Vaughan Pratt, Michael Sokolsky, Ganymed Stanek, David Stavens, Alex Teichman, Moritz Werling, and Sebastian Thrun.  Towards fully autonomous driving: Systems and algorithms.  In _IV_ , 2011.
  * Li et al. (2018) Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein.  Visualizing the loss landscape of neural nets.  _NeurIPS_ , 2018.
  * Liu et al. (2022) Chaoyue Liu, Libin Zhu, and Mikhail Belkin.  Loss landscapes and optimization in over-parameterized non-linear systems and neural networks.  _Applied and Computational Harmonic Analysis_ , 2022.
  * MacKay (1992) David JC MacKay.  A practical bayesian framework for backpropagation networks.  _Neural computation_ , 1992.
  * MacKay (1995) David JC MacKay.  Probable networks and plausible predictions-a review of practical bayesian methods for supervised neural networks.  _Network: computation in neural systems_ , 1995.
  * Maddox et al. (2019) Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson.  A simple baseline for bayesian uncertainty in deep learning.  In _NeurIPS_ , 2019.
  * Masegosa (2020) Andres Masegosa.  Learning under model misspecification: Applications to variational and ensemble methods.  In _NeurIPS_ , 2020.
  * McAllister et al. (2017) Rowan McAllister, Yarin Gal, Alex Kendall, Mark Van Der Wilk, Amar Shah, Roberto Cipolla, and Adrian Weller.  Concrete problems for autonomous vehicle safety: Advantages of bayesian deep learning.  In _IJCAI_ , 2017.
  * Metz et al. (2022) Luke Metz, James Harrison, C Daniel Freeman, Amil Merchant, Lucas Beyer, James Bradbury, Naman Agrawal, Ben Poole, Igor Mordatch, Adam Roberts, et al.  Velo: Training versatile learned optimizers by scaling up.  _arXiv preprint arXiv:2211.09760_ , 2022.
  * Naeini et al. (2015) Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In _AAAI_ , 2015.
  * Nair & Hinton (2010) Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In _ICML_ , 2010.
  * Neal et al. (2011) Radford M Neal et al.  Mcmc using hamiltonian dynamics.  _Handbook of markov chain monte carlo_ , 2011.
  * Nesterov (1983) Yurii Evgenevich Nesterov.  A method of solving a convex programming problem with convergence rate $\mathcal{O}\left(\frac{1}{k^{2}}\right)$.  In _Doklady Akademii Nauk_ , 1983.
  * Netzer et al. (2011) Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng.  Reading digits in natural images with unsupervised feature learning.  In _NeurIPS Workshops_ , 2011.
  * Neyshabur et al. (2015) Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro.  Path-sgd: Path-normalized optimization in deep neural networks.  _Advances in neural information processing systems_ , 28, 2015.
  * Ober & Rasmussen (2019) Sebastian W Ober and Carl Edward Rasmussen.  Benchmarking the neural linear model for regression.  In _AABI_ , 2019.
  * Ortega et al. (2022) Luis A Ortega, Rafael Cabañas, and Andres Masegosa.  Diversity and generalization in neural network ensembles.  In _AISTATS_ , 2022.
  * Ovadia et al. (2019) Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  In _NeurIPS_ , 2019.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In _NeurIPS_ , 2019.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 2011.
  * Pourzanjani et al. (2017) Arya A Pourzanjani, Richard M Jiang, and Linda R Petzold.  Improving the identifiability of neural networks for bayesian inference.  In _NeurIPSW_ , 2017.
  * Rame & Cord (2021) Alexandre Rame and Matthieu Cord.  Dice: Diversity in deep ensembles via conditional redundancy adversarial estimation.  In _ICLR_ , 2021.
  * Rasmussen et al. (2006) Carl Edward Rasmussen, Christopher KI Williams, et al.  _Gaussian processes for machine learning_.  Springer, 2006.
  * Ritter et al. (2018) Hippolyt Ritter, Aleksandar Botev, and David Barber.  A scalable laplace approximation for neural networks.  In _ICLR_ , 2018.
  * Rolnick & Kording (2020) David Rolnick and Konrad Kording.  Reverse-engineering deep relu networks.  In _ICML_ , 2020.
  * Schrab et al. (2023) Antonin Schrab, Ilmun Kim, Mélisande Albert, Béatrice Laurent, Benjamin Guedj, and Arthur Gretton.  MMD aggregated two-sample test.  _Journal of Machine Learning Research_ , 2023.
  * Schwartz (1964) Laurent Schwartz.  Sous-espaces hilbertiens d’espaces vectoriels topologiques et noyaux associés (noyaux reproduisants).  _Journal d’analyse mathématique_ , 1964.
  * Simonyan & Zisserman (2015) Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _ICLR_ , 2015.
  * Smola et al. (2007) Alex Smola, Arthur Gretton, Le Song, and Bernhard Schölkopf.  A hilbert space embedding for distributions.  In _ALT_ , 2007.
  * Song (2008) Le Song.  Learning via hilbert space embedding of distributions.  _University of Sydney (2008)_ , 17, 2008.
  * Sriperumbudur et al. (2010) Bharath K Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf, and Gert RG Lanckriet.  Hilbert space embeddings and metrics on probability measures.  _The Journal of Machine Learning Research_ , 11:1517–1561, 2010.
  * Sun et al. (2019) Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse.  Functional variational bayesian neural networks.  In _ICLR_ , 2019.
  * Sutskever et al. (2013) Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton.  On the importance of initialization and momentum in deep learning.  In _ICML_ , 2013.
  * Tishby et al. (1989) Tishby, Levin, and Solla.  Consistent inference of probabilities in layered networks: predictions and generalizations.  In _IJCNN_ , 1989.
  * Virtanen et al. (2020) Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  _Nature Methods_ , 2020.
  * Watson et al. (2021) Joe Watson, Jihao Andreas Lin, Pascal Klink, Joni Pajarinen, and Jan Peters.  Latent derivative bayesian last layer networks.  In _AISTATS_ , 2021.
  * Welling & Teh (2011) Max Welling and Yee W Teh.  Bayesian learning via stochastic gradient langevin dynamics.  In _ICML_ , 2011.
  * Wen et al. (2019) Yeming Wen, Dustin Tran, and Jimmy Ba.  BatchEnsemble: an alternative approach to efficient ensemble and lifelong learning.  In _ICLR_ , 2019.
  * Wiese et al. (2023) Jonas Gregor Wiese, Lisa Wimmer, Theodore Papamarkou, Bernd Bischl, Stephan Günnemann, and David Rügamer.  Towards efficient mcmc sampling in bayesian neural networks by exploiting symmetry.  _arXiv preprint arXiv:2304.02902_ , 2023.
  * Wilson & Izmailov (2020) Andrew G Wilson and Pavel Izmailov.  Bayesian deep learning and a probabilistic perspective of generalization.  In _NeurIPS_ , 2020.
  * Xiao et al. (2017) Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  _arXiv preprint arXiv:1708.07747_ , 2017.
  * Zhang et al. (2020) Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson.  Cyclical stochastic gradient mcmc for bayesian deep learning.  In _ICLR_ , 2020.
  * Zhao et al. (2023) Bo Zhao, Iordan Ganev, Robin Walters, Rose Yu, and Nima Dehmamy.  Symmetries, flat minima, and the conserved quantities of gradient flow.  In _ICLR_ , 2023.
  * Zou & Gu (2021) Difan Zou and Quanquan Gu.  On the convergence of hamiltonian monte carlo with stochastic gradients.  In _ICML_ , 2021.

1

## Table of Contents - Supplementary Material

##  Appendix A Details on the starting example

Figure 5: The training data of the models whose posterior is represented on
the introductory Figure 1. The data is separable to ensure efficient training
of very simple 9-parameter perceptrons.

###  A.1 Training of the two-layer perceptrons

To create the introductory example, presented in Figure 1, we generated the
data corresponding to the two classes from two different normal distributions
of means ${\bm{\mu}}_{A}=(-2,-2)$ and ${\bm{\mu}}_{B}=(2,2)$ and of identity
covariances. We ensured that the samples (200 points each) were fully
separable to make training as simple as possible. We plot the training data in
Figure 5. We then trained 10,000 two-layer perceptrons with two input, two
hidden, and one output neurons with early stopping for 10 epochs with SGD and
a binary cross-entropy loss. For the posterior to exhibit scaling symmetries,
we separated the first and second layers with a ReLU activation function. We
used a batch size of 10 data and a learning rate of 2. Finally, we selected
networks that had a sufficiently low loss on the training set, therefore
removing the few outliers (less than 1%) that had not been trained
successfully.

###  A.2 Removing the symmetries

Figure 1 shows the density of weights of the last layer (excluding the last
bias). The first Figure shows the unaltered projection of the posterior on
these weights, which is then transformed to guarantee a 3-norm per neuron. We
chose 3 as norm for visual purposes since the last layer (whose posterior is
plotted in Figure 1) is not normalized and is subject to the normalization of
the previous layer. Finally, we remove the permutation symmetries by ordering
the weights. Contrary to the formalism developed in the following sections, we
decide here to order the weights starting from the last layer (and not the
first) to convey the message more efficiently. For each network, we check with
random inputs that all symmetry removals do not alter the networks. Notebooks
are available in the supplementary material of the submission and detail the
process for generating the Figure. We provide more detail on the symmetry
removal algorithms in Section C.2.

##  Appendix B Details on the experiments of section 4

###  B.1 Experimental details

In this section, we develop the training recipes of the different models as
well as the parameters used for all the posterior estimation methods. We train
all of our models with PyTorch (Paszke et al., 2019) on V100 clusters. All
networks are trained until the last step to avoid biasing the posterior with
the validation set.

For all the variational Bayesian Neural Networks (vBNN), we use the default
priors from Bayesian layers in torch zoo (Blitz) (Esposito, 2020).

When using ResNet-18 models, we perform last-layer approximations of Laplace
and Dropout. For more information on last-layer approximation, you may refer
to Brosse et al. (2020) for instance.

##### OptuNet - MNIST

We train OptuNet for 60 epochs with batches of size 64 using stochastic
gradient descent (SGD) with a start learning rate of 0.04 and a weight decay
of $2\times 10^{-4}$. We decay the learning rate twice during training, at
epochs 15 and 30, dividing the learning rate by 2.

We train the vBNNs with the same number of epochs, albeit using 3 Monte Carlo
estimates of the Evidence lower bound (ELBO) at each step and using a
Kullback-Leibler divergence (KLD) of $10^{-5}$. We disable weight decay for
the training of the vBNNs.

To compute the Maximum Mean Discrepancies, we use a full Hessian Laplace
optimization. We use a dropout rate of 0.2 on the last layer (both for
training and testing) and perform SWAG using 20 different models (also set as
the maximum number to keep a low-rank matrix), continuing the training with
the classic high-learning rate schedule (starting with a Linear increase) for
twice as long. We collect the models 20 times with 10-epoch intervals between
each of them. We sample the models with a scale of 0.1.

For the dataset, we normalize the data as usual and perform a random crop of
size 28 and padding four during training. We use the normalized test images
for testing.

For more details on the architecture of our OptuNet, please refer to Section
B.2.

##### ResNet-18 - CIFAR-100

We train the ResNet-18 for 75 epochs with batches of size 128 using SGD with
Nesterov (Nesterov, 1983; Sutskever et al., 2013), with a start learning rate
of 0.1, a momentum of 0.9, and a weight decay of $5\times 10^{-4}$. Similarly
to MNIST, we decay the learning rate twice during training, this time at
epochs 25 and 50, and divide the learning rate by 10.

We train the variational BNN with SGD for 150 epochs, with a starting learning
rate of 0.01, which we decay once after 80 epochs. We weigh the KLD with a
coefficient $2\times 10^{-9}$ and perform three ELBO samples at each step.

To compute the MMDs for Laplace methods, we use the last-layer Kronecker
approximation. Indeed, the last layer of ResNet-18 is too large for its full
Hessian to fit in memory. Moreover, Kronecker is not available for the full
network as it is not implemented for batch normalization layers, and the low-
rank version is too long to train. For the dropout models, we perform last-
layer dropout of probability 0.5. As for MNIST, we train the SWAG models twice
the time of the original training recipe with the usual settings (start
learning rate with a linear ramp) and collect 20 models with 10-epoch
intervals. We sample the models with a scale of 0.1.

For CIFAR-100, we perform the classic normalization, as well as a random crop
for a four-pixel padding and a random horizontal split.

##### ResNet-18 - TinyImageNet

We train the ResNet-18 for 200 epochs with a batch size of 128 using SGD with
a start learning rate of 0.2, a weight decay of $10^{-4}$, and a momentum of
0.9. We use a cosine annealing scheduler until the end of the training.

We did not manage to train vBNNs for TinyImageNet. The other methods use
exactly the same hyperparameters as for ResNet-18 on CIFAR-100.

For TinyImageNet, we perform the same preprocessing as for CIFAR-100, except
we keep the original resolution of $64\times 64$ pixels.

###  B.2 Detailed architecture of OptuNet

Figure 6: Detailed architecture of OptuNet. OptuNet includes only 392
parameters yet achieves between 85% and 90% accuracy on MNIST. Only the last
fully connected layer contains biases.

For OptuNet, we aimed to create the smallest number of parameters possible
while keeping decent performance on MNIST. To this extent, we performed an
architecture search with tree-structured Parzen estimators (e.g., (Bergstra et
al., 2011)) in Optuna (Akiba et al., 2019). To reduce the number of
parameters, OptuNet uses grouped convolutions, introduced by Krizhevsky et al.
(2012) (see Laurent et al. (2023) for a formal definition) as its second
convolutional layer.

###  B.3 Details on the Datasets

This part provides some details on the different datasets used throughout the
paper.

####  B.3.1 In distribution datasets

In this work, we used the following datasets for training and testing.

##### MNIST

The MNIST dataset LeCun et al. (1998) comprises 70,000 binary images of
handwritten digits, each of size 28$\times$28 pixels. These images are divided
into a training set with 60,000 samples and a testing set with 10,000 samples.

##### CIFAR-100

CIFAR-100 Krizhevsky (2009) consists of 100 object classes and contains a
training set with 50,000 images and a testing set including 10,000 images.
Each of the images is RGB and of size 32$\times$32 pixels.

##### Tiny-ImageNet

TinyImageNet is a subset of ImageNet-1k Deng et al. (2009). It consists of 200
object classes, each with 500 training and 50 validation images. Additionally,
there are 50 test images per class for evaluating models. The images are RGB
and have a size of 64$\times$64 pixels.

####  B.3.2 Out-of-distribution datasets

The objective of the following out-of-distribution datasets is to evaluate the
quality of the posterior as a means to quantify the epistemic uncertainty. In
this work, we do not consider these datasets as fully representative of
difficult real-world out-of-distribution detection tasks.

##### Fashion-MNIST

The Fashion-MNIST dataset (Xiao et al., 2017) is a set of 28$\times$28 pixel-
grayscale images consisting of 60,000 training and 10,000 test images. We used
the test set as is for the out-of-distribution tasks with MNIST.

##### SVHN

The Street View House Numbers (SVHN) dataset (Netzer et al., 2011) is a large-
scale dataset of 600,000 images of digits obtained from house numbers in
Google Street View. In our work, we kept a fixed set of 10,000 images that we
cropped to squares of 32$\times$32 pixels.

##### Textures

The Describable Textures Dataset (Cimpoi et al., 2014) is a dataset containing
5640 images of textures divided into three subsets. Considering that the
number of images is limited compared to the other testing sets, we use the
concatenation of all the subsets for OOD detection. We resize all the images
to 64$\times$64 pixels to stick to the size of TinyImageNet.

###  B.4 Full MMD Table

|  | Method | MMD | NS MMD
---|---|---|---|---
|  |  | Median | Mean | Max. | Median | Mean | Max.
MNIST - OptuNet |  One Mode | Dropout | 14.9 | 14.6 | 22.9 | 14.3 | 14.3 | 22.5
BNN | 18.8 | 18.3 | 29.5 | 17.1 | 17.0 | 27.4
SGHMC | 16.7 | 16.6 | 27.8 | 17.7 | 18.3 | 32.2
SWAG | 15.9 | 15.8 | 25.9 | 14.6 | 13.8 | 21.6
Laplace | 10.6 | 10.5 | 19.1 | 9.5 | 9.4 | 16.4
Multi Mode | Dropout | 2.1 | 2.3 | 4.1 | 2.1 | 2.0 | 3.5
BNN | 2.8 | 2.9 | 5.8 | 2.5 | 2.4 | 4.3
SWAG | 1.8 | 2.0 | 4.1 | 1.3 | 1.3 | 2.5
Laplace | 1.8 | 2.0 | 4.1 | 0.8 | 0.8 | 1.5
DE | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
CIFAR-100 - ResNet-18 |  One Mode | Dropout | 4.5 | 4.6 | 7.2 | 7.4 | 7.5 | 12.0
BNN | 9.0 | 9.2 | 14.6 | 10.0 | 10.2 | 16.0
SGHMC | 7.5 | 7.3 | 11.0 | 7.9 | 8.0 | 13.0
SWAG | 6.7 | 7.0 | 11.2 | 7.2 | 7.5 | 12.3
Laplace | 5.7 | 6.0 | 9.7 | 7.0 | 7.2 | 11.7
Multi Mode | Dropout | 0.7 | 0.7 | 1.2 | 4.4 | 4.5 | 7.5
BNN | 6.1 | 6.3 | 10.6 | 5.6 | 5.8 | 10.0
SWAG | 5.0 | 5.2 | 8.4 | 5.4 | 5.6 | 9.2
Laplace | 0.6 | 0.6 | 1.0 | 4.3 | 4.4 | 7.2
DE | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
TinyImageNet - ResNet-18 |  One Mode | Dropout | 9.5 | 9.6 | 15.0 | 4.9 | 5.0 | 7.8
BNN | / | / | / | / | / | /
SGHMC | 9.8 | 10.0 | 15.7 | 5.3 | 5.2 | 7.9
SWAG | 9.1 | 9.5 | 15.6 | 5.4 | 5.6 | 9.5
Laplace | 5.5 | 5.8 | 9.6 | 6.1 | 6.3 | 10.1
Multi Mode | Dropout | 4.3 | 4.4 | 7.4 | 1.8 | 1.8 | 3.0
BNN | / | / | / | / | / | /
SWAG | 6.7 | 6.9 | 11.3 | 3.9 | 4.2 | 7.3
Laplace | 0.6 | 0.6 | 9.9 | 3.1 | 3.2 | 5.3
DE | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0

Table 2: Comparison of popular methods approximating the Bayesian posterior.
Mean Maximum Discrepancies for OptuNet are expressed in %. For ResNet-18
networks, MMDs are expressed in ‰. NS MMD is the MMD computed after the
removal of the symmetries. Multi-mode methods use ten independently trained
models. DE stands for Deep Ensembles (Lakshminarayanan et al., 2017).

In Table 2, we report the weighted average over the layers of the medians,
means, and maxima of the Maximum Mean Discrepancies of each layer, computed in
Section 4. We recall that the maximum is used to obtain the best
discriminative power of two-sample tests, for instance, in (Gretton et al.,
2012; Schrab et al., 2023). We report the median in the main table to improve
the representativity of the values. However, we note that the order of the
values does not seem to change, particularly between the means, medians, and
maxima.

We see that, after Deep Ensembles - that have perfect MMD as expected -
Laplace methods perform best on all measures, be it with or without
symmetries, as well as for all types of architectures. However, as shown in
Table 1, they are inferior to SWAG on performance metrics. This hints that the
correlation between the estimated quality of the posterior and the real-world
metrics may not be fully correlated. Please note that for Deep Ensembles, we
used different checkpoints for the second sample than in the sample
corresponding to the estimation of the posterior.

##  Appendix C Detailed formalism, propositions, and proofs

This section details the definitions, properties, and propositions and
provides sketches of proofs.

Throughout the paper, we use an intuitive definition of neural networks that
we propose to formalize in the following definition:

######  Definition C.1 (Neural network).

Let ${\bm{x}}\in\mathbb{R}^{d}$ be an input datum, $r$ the rectified linear
unit, and $s$ an almost everywhere differentiable activation function. We
define the $L$-layer neural network $f_{\theta}$ with
$\theta=\\{W^{[l]},b^{[l]}\\}_{l\in\llbracket 1,L\rrbracket}$ as follows:

$\displaystyle{\bm{a}}^{[0]}={\bm{x}}$ (10) $\displaystyle\forall
l\in\llbracket 1,L\rrbracket,\ {\bm{z}}^{[l]}=W^{[l]}{\bm{a}}^{[l-1]}+b^{[l]}$
(11) $\displaystyle\forall l\in\llbracket 1,L-1\rrbracket,\
{\bm{a}}^{[l]}=r({\bm{z}}^{[l]})$ (12) $\displaystyle
f_{\theta}({\bm{x}})=s({\bm{z}}^{[l]})$ (13)

This definition of neural networks is limited to standard multilayer
perceptions. However, it can be easily extended to a mix of convolutional and
fully connected layers to the cost of a heavier formalism. To do this, we
should replace the relationship between ${\bm{z}}^{[l]}$ and
${\bm{a}}^{[l-1]}$ in Equation 11 by a sum on the kernels of the cross-
correlation operator. It would also be possible to account for residual
connections using the computation graph.

######  Property C.1 (Gradients and backpropagation).

Let us denote the training loss $\mathcal{L}(\theta)$ of the $L$-layer neural
network $f_{\theta}$ and $\theta=\\{W^{[l]},b^{[l]}\\}_{l\in\llbracket
1,L\rrbracket}$. With, for all $l\in\llbracket 1,L\rrbracket$,
${\bm{\delta}}^{[l]}$ as the gradient of $\mathcal{L}(\theta)$ with respect to
${\bm{z}}^{[l]}$, we have,

$\displaystyle{\bm{\delta}}^{[L]}=\nabla_{{\bm{z}}^{[L]}}\mathcal{L}(\theta)$
(14) $\displaystyle\forall l\in\llbracket
1,L-1\rrbracket,{\bm{\delta}}^{[l]}=\nabla_{{\bm{z}}^{[l]}}\mathcal{L}(\theta)=\left(W^{[l+1]^{\intercal}}{\bm{\delta}}^{[l+1]}\right)\circ\nabla_{{\bm{z}}^{[l]}}{\bm{a}}^{[l]}$
(15) $\displaystyle\forall l\in\llbracket
1,L\rrbracket,\nabla_{W^{[l]}}\mathcal{L}(\theta)={\bm{\delta}}^{[l]}{\bm{a}}^{[l-1]^{\intercal}}$
(16) $\displaystyle\forall l\in\llbracket
1,L\rrbracket,\nabla_{b^{[l]}}\mathcal{L}(\theta)={\bm{\delta}}^{[l]}.$ (17)

###  C.1 Extended formalism on symmetries

In this section, we propose several definitions and properties to extend the
formalism quickly described in the main paper.

####  C.1.1 Scaling symmetries

We start with a recall of the definitions of the line-wise and column-wise
products between a vector and a matrix.

######  Definition C.2.

We denote the line-wise product as $\bigtriangledown$ and the column-wise
product as $\rhd$. Let ${\bm{\omega}}\in\mathbb{R}^{n\times m}$,
$\bm{\lambda}\in\mathbb{R}^{m}$, and ${\bm{\mu}}\in\mathbb{R}^{n}$, with
$n,m\geq 1$,

$\forall i\in\llbracket 1,n\rrbracket,j\in\llbracket 1,m\rrbracket,\
(\bm{\lambda}\bigtriangledown{\bm{\omega}})_{i,j}=\lambda_{j}\omega_{i,j}\
\text{and}\ ({\bm{\mu}}\rhd{\bm{\omega}})_{i,j}=\mu_{i}\omega_{i,j}.$ (18)

To handle biases, we extend these definitions to the product of vectors,
considering the right vector as a one-dimensional matrix. It follows that for
all vectors $\bm{\lambda},{\bm{a}}\in\mathbb{R}^{m}$,

$\bm{\lambda}\bigtriangledown{\bm{a}}=\bm{\lambda}\rhd{\bm{a}}=\bm{\lambda}\cdot{\bm{a}}.$
(19)

The line-wise and column-wise products between vectors and matrices _have
priority_ over the classic matrix product ”$\times$” but not over the element-
wise product between vectors ”$\cdot$”. We will include parentheses when
deemed necessary for the clarity of the expressions.

##### Non-negative homogeneity of the rectified linear unit

The rectified linear unit (Nair & Hinton, 2010), used as the standard
activation function in most networks (with the notable exception of vision
transformers (Dosovitskiy et al., 2021) is non-negative homogenous and
therefore authorizes scaling symmetries (Neyshabur et al., 2015).

######  Property C.2.

Denote $r$ the rectified linear unit such that $r(x)=x_{+}$. For all vectors
$\bm{\lambda}\in(\mathbb{R}^{+})^{d}$ and ${\bm{x}}\in\mathbb{R}^{d}$, $r$ is
positive-homogeneous, i.e.

$r(\bm{\lambda}\cdot{\bm{x}})=\bm{\lambda}\cdot r({\bm{x}}).$ (20)

##### Some properties on line-wise and column-wise products

In this paragraph, we propose some simple properties to help achieve an
understanding of these notations.

######  Property C.3.

For all vectors $\bm{\lambda}^{(1)},\bm{\lambda}^{(2)}\in\mathbb{R}^{n}$,
${\bm{\theta}}\in\mathbb{R}^{\cdot\times n}$, and
${\bm{\omega}}\in\mathbb{R}^{n\times\cdot}$

$\displaystyle(\bm{\lambda}^{(1)}\bigtriangledown{\bm{\theta}})\times(\bm{\lambda}^{(2)}\rhd{\bm{\omega}})=(\bm{\lambda}^{(1)}\cdot\bm{\lambda}^{(2)})\bigtriangledown{\bm{\theta}}\times{\bm{\omega}}={\bm{\theta}}\times(\bm{\lambda}^{(1)}\cdot\bm{\lambda}^{(2)})\rhd{\bm{\omega}}$
(21)

###### Proof.

Let us denote
$A=(\bm{\lambda}^{(1)}\bigtriangledown{\bm{\theta}})\times(\bm{\lambda}^{(2)}\rhd{\bm{\omega}})$

$\displaystyle
A_{i,j}=\sum\limits_{k=1}^{n}\lambda_{k}^{(1)}\theta_{i,k}\lambda_{k}^{(2)}\omega_{k,j}=$
$\displaystyle\sum\limits_{k=1}^{n}\theta_{i,k}\lambda^{(2)}_{k}\lambda^{(1)}_{k}\omega_{k,j}=({\bm{\theta}}\times(\bm{\lambda}^{(1)}\cdot\bm{\lambda}^{(2)})\rhd{\bm{\omega}})_{i,j}$
(22) $\displaystyle=$
$\displaystyle\sum\limits_{k=1}^{n}\lambda^{(2)}_{k}\lambda^{(1)}_{k}\theta_{i,k}\omega_{k,j}=((\bm{\lambda}^{(1)}\cdot\bm{\lambda}^{(2)})\bigtriangledown{\bm{\theta}}\times{\bm{\omega}})_{i,j}$
(23)

∎

However, please note that we do not have
$\bm{\lambda}^{(1)}\bigtriangledown(\bm{\lambda}^{(2)}\rhd{\bm{\omega}})=(\bm{\lambda}^{(1)}\cdot\bm{\lambda}^{(2)})\bigtriangledown{\bm{\omega}}$
in general.

######  Property C.4.

For all vectors $\bm{\lambda}^{(1)}\in\left(\mathbb{R}_{\neq 0}\right)^{n}$,
$\bm{\lambda}^{(2)}\in\mathbb{R}^{n}$,
${\bm{\theta}}\in\mathbb{R}^{\cdot\times n}$, and
${\bm{\omega}}\in\mathbb{R}^{n\times\cdot}$

$\displaystyle(\bm{\lambda}^{(1)}\bigtriangledown{\bm{\theta}})\times
r(\bm{\lambda}^{(2)}\rhd{\bm{\omega}})={\bm{\theta}}\times
r\left((\bm{\lambda}^{(1)}\cdot\bm{\lambda}^{(2)})\rhd{\bm{\omega}}\right)$
(24)

###### Proof.

Let us denote $B=(\bm{\lambda}_{1}\bigtriangledown{\bm{\theta}})\times
r(\bm{\lambda}_{2}\rhd{\bm{\omega}})$

$\displaystyle B_{i,j}=$
$\displaystyle\sum\limits_{k=1}^{n}\lambda^{(1)}_{k}\theta_{i,k}r(\lambda^{(2)}_{k}\omega_{k,j})$
(25) $\displaystyle=$
$\displaystyle\sum\limits_{k=1}^{n}\theta_{i,k}r(\lambda^{(1)}_{k}\lambda^{(2)}_{k}\omega_{k,j})$
(26) $\displaystyle=$ $\displaystyle({\bm{\theta}}\times
r((\bm{\lambda}^{(1)}\cdot\bm{\lambda}^{(2)})\rhd{\bm{\omega}}))_{i,j}$ (27)

∎

This property is crucial for scaling symmetries, which are one of its special
cases. Indeed, for all vectors
$\bm{\lambda}\in\left(\mathbb{R}_{>0}\right)^{n}$,

$(\bm{\lambda}^{-1}\bigtriangledown{\bm{\theta}})\times
r(\bm{\lambda}\rhd{\bm{\omega}})={\bm{\theta}}\times r({\bm{\omega}}).$ (28)

Finally, we can gather the previous results to extend Equation 28 to weights
and biases in the following property.

######  Property C.5.

For all vectors $\bm{\lambda}\in\left(\mathbb{R}_{>0}\right)^{n}$,
${\bm{b}}\in\mathbb{R}^{n}$, ${\bm{\theta}}\in\mathbb{R}^{\cdot\times n}$, and
${\bm{\omega}}\in\mathbb{R}^{n\times\cdot}$

$\displaystyle(\bm{\lambda}^{-1}\bigtriangledown{\bm{\theta}})\times
r(\bm{\lambda}\rhd({\bm{b}}+{\bm{\omega}}))={\bm{\theta}}\times
r({\bm{b}}+{\bm{\omega}})$ (29)

###### Proof.

$\displaystyle(\bm{\lambda}^{-1}\bigtriangledown{\bm{\theta}})\times
r(\bm{\lambda}\rhd({\bm{b}}+{\bm{\omega}}))$
$\displaystyle=(\bm{\lambda}^{-1}\bigtriangledown{\bm{\theta}})\times
r(\bm{\lambda}\cdot{\bm{b}}+\bm{\lambda}\rhd{\bm{\omega}})$ (30)
$\displaystyle={\bm{\theta}}\times
r(\bm{\lambda}^{-1}\bm{\lambda}\cdot{\bm{b}}+\bm{\lambda}^{-1}\bm{\lambda}\rhd{\bm{\omega}})$
(31) $\displaystyle={\bm{\theta}}\times r({\bm{b}}+{\bm{\omega}})$ (32)

∎

This property is provided for the simplest case with only two matrices.
However, it can be trivially extended to an arbitrary number of matrices,
chaining the different line-wise and column-wise multiplications. It follows
that we can extend the definition of scaling symmetries to deeper networks.

Moreover, we can also extend the scaling symmetries to mixes of convolutional
and linear networks. To do this, we first extend the definition of the line-
wise and column-wise products to convolutional layers through the following
definition.

######  Definition C.3.

Let the 4-dimensional tensor
${\bm{\omega}}\in\mathbb{R}^{C_{\mathrm{out}}\times C_{\mathrm{in}}\times
k_{v}\times k_{h}}$ be the weight of a convolutional layer of input and output
channels $C_{\mathrm{in}}$ and $C_{\mathrm{out}}$, and kernel $k_{v}\times
k_{h}$. Denote $\bm{\lambda}\in\mathbb{R}^{C_{\mathrm{in}}}$ and
${\bm{\mu}}\in\mathbb{R}^{C_{\mathrm{out}}}$ We extend the operators $\rhd$
and $\bigtriangledown$ to products between vectors and 4-dimensional tensors
and have,

$\displaystyle\forall c_{out}\in\llbracket
1,C_{\mathrm{out}}\rrbracket,c_{in}\in\llbracket 1,$ $\displaystyle
C_{\mathrm{in}}\rrbracket,i\in\llbracket 1,n\rrbracket,j\in\llbracket
1,m\rrbracket,$ (33)
$\displaystyle(\bm{\lambda}\bigtriangledown{\bm{\omega}})_{c_{out},c_{in},i,j}=\lambda_{c_{in}}\omega_{c_{out},c_{in},i,j},$
(34) $\displaystyle\text{and}\
({\bm{\mu}}\rhd{\bm{\omega}})_{c_{out},c_{in},i,j}=\mu_{c_{out}}\omega_{c_{out},c_{in},i,j}.$
(35)

With this definition, we keep the previous properties and enable chaining
transformations on both convolutional and fully connected layers.

##### Equivalence to standard linear algebra

We propose the line-wise ($\bigtriangledown$) and column-wise ($\rhd$)
notations for their intuitiveness. However, when applied to fully connected
layers, they are equivalent to more common notations from linear algebra.
Indeed, sticking to ${\bm{\omega}}\in\mathbb{R}^{n\times m}$,
$\bm{\lambda}\in\mathbb{R}^{m}$, and ${\bm{\mu}}\in\mathbb{R}^{n}$, with
$n,m\geq 1$, we have that

$\bm{\lambda}\bigtriangledown{\bm{\omega}}=\mathrm{diag}(\bm{\lambda})\times{\bm{\omega}}\
\text{and}\
{\bm{\mu}}\rhd{\bm{\omega}}=\omega\times\mathrm{diag}({\bm{\mu}}).$ (36)

Furthermore, we can also write, with $\circ$ the Hadamard (elementwise)
product between matrices,

$\bm{\lambda}\bigtriangledown\left({\bm{\mu}}\rhd{\bm{\omega}}\right)=(\bm{\lambda}^{\intercal}\times{\bm{\mu}})\circ\omega.$
(37)

These equivalences are important for the implementation of the log-log convex
problem to minimize the L2 regularization term on the space of the scaling
symmetries (see Section 3.6).

####  C.1.2 Permutation symmetries

######  Definition C.4.

We define $P_{n}$ as the set of permutations of vectors of size $n$. $P_{n}$
contains $\left|P_{n}\right|=n!$ elements. The elements of $P_{n}$ are doubly-
stochastic binary matrices from $\\{0,1\\}^{n}$. Finally, we have that, for
all $\bm{\pi}_{1}\in P_{n}$,
$\bm{\pi}_{1}\times\bm{\pi}_{1}^{\intercal}=\mathbb{1}_{n}$.

From this definition and property, we can directly derive the following
result:

######  Property C.6.

For all ${\bm{\theta}}\in\mathbb{R}^{\cdot\times m}$,
${\bm{\omega}}\in\mathbb{R}^{m\times n}$, and permutation matrix $\bm{\pi}\in
P_{m}$,

$\forall{\bm{x}}\in\mathbb{R}^{n},\ {\bm{\theta}}\bm{\pi}^{\intercal}\times
r\left(\bm{\pi}\times{\bm{\omega}}{\bm{x}}\right)={\bm{\theta}}\times
r({\bm{\omega}}{\bm{x}}).$ (38)

###### Proof.

Let us take $m,n\in\mathbb{N}$, ${\bm{\omega}}\in\mathbb{R}^{m\times
n},{\bm{b}}\in\mathbb{R}^{m},{\bm{x}}\in\mathbb{R}$, and $\bm{\pi}\in P_{m}$ a
permutation matrix corresponding to the bijective mapping $\sigma$ in
$\llbracket 1,m\rrbracket$. We have that,

$\left[\bm{\pi}({\bm{b}}+{\bm{\omega}})\times{\bm{x}}\right]_{i}={\bm{b}}_{\sigma(i)}+\sum\limits_{k=1}^{n}\omega_{\sigma(i),k}x_{k}.$
(39)

Hence,

$\displaystyle\left[\bm{\pi}^{\intercal}\times
r(\bm{\pi}({\bm{\omega}})\times{\bm{x}})\right]_{i}$
$\displaystyle={\bm{b}}_{\sigma^{-1}(\sigma(i))}+\sum\limits_{k=1}^{n}r\left(\omega_{\sigma^{-1}(\sigma(i)),k}\cdot
x_{k}\right)$ (40)
$\displaystyle=\left[r({\bm{b}}+{\bm{\omega}}\times{\bm{x}})\right]_{i},$ (41)

and it comes directly that,

${\bm{\theta}}\bm{\pi}^{\intercal}\times
r\left(\bm{\pi}\times{\bm{\omega}}{\bm{x}}\right)={\bm{\theta}}\times
r({\bm{\omega}}{\bm{x}}).$ (42)

∎

This property is provided for the simplest case with only two matrices.
However, it can be trivially extended to an arbitrary number of matrices,
chaining the different permutations. As for the scaling symmetries, it follows
that we can extend the definition of permutation symmetries to deeper
networks.

Moreover, as for scaling symmetries in definition C.3, we can extend
permutation operations and symmetries to convolutional layers and their
4-dimensional tensors. To do this, we consider that the permutation matrices
act on the output and input channels and permute the whole kernels.

####  C.1.3 Softmax additive symmetry

For the sake of completeness, we also recall the additive softmax symmetries
in this section. We left this symmetry apart in our analysis as it involves,
at most, one degree of freedom.

######  Definition C.5.

We recall that the softmax function $\sigma$ is defined by the following
equation, given the logits ${\bm{a}}\in\mathbb{R}^{n}$:

$\sigma({\bm{a}})_{i}=\frac{e^{a_{i}}}{\sum\limits_{j=1}^{n}e^{a_{j}}}$ (43)

######  Property C.7.

For all ${\bm{a}},{\bm{b}}\in\mathbb{R}^{n}$, $\lambda\in\mathbb{R}$, and +
the point-wise sum when applied between vectors and scalars,

$\sigma({\bm{a}}+{\bm{b}}+\lambda)=\sigma({\bm{a}}+{\bm{b}})$ (44)

###### Proof.

Let us denote $S=\sigma({\bm{a}}+{\bm{b}}+\lambda)$,

$\displaystyle S_{i}=$
$\displaystyle\frac{e^{a_{i}+b_{i}+\lambda}}{\sum\limits_{j=1}^{n}e^{a_{j}+b_{j}+\lambda}}$
(45) $\displaystyle S_{i}=$
$\displaystyle\frac{e^{\lambda}e^{a_{i}+b_{i}}}{e^{\lambda}\sum\limits_{j=1}^{n}e^{a_{j}+b_{j}}}$
(46) $\displaystyle S_{i}=$ $\displaystyle(\sigma({\bm{a}}+{\bm{b}}))_{i}$
(47)

∎

###  C.2 Removing symmetries a posteriori

##### Removing permutation symmetries

Similar to Pourzanjani et al. (2017), we propose to remove permutation
symmetries by ordering neurons according to the value of their first
corresponding parameter. In dense layers, this first parameter is the weight
of the first input neuron, and in convolutional layers, it corresponds to the
top-left weight of the kernel of the first channel. This solution is more
general than that of Pourzanjani et al. (2017) since neural network layers do
not always have biases. Specifically, practitioners often remove biases from a
layer when it is followed by a batch normalization (Ioffe & Szegedy, 2015) to
minimize the number of computations of the inference and backpropagation
steps.

##### Removing scaling symmetries

In this paper, we propose two different types of algorithms to remove the
scaling symmetries of a neural network _a posteriori_. The first algorithm was
proposed by Neyshabur et al. (2015) and normalizes the norm of the weights of
each neuron in all the layers except the last. This is the algorithm that we
use in Table 1 We also provide an implementation of this algorithm to scale
the standard deviation of the weights to one. Furthermore, we define in
Section 3.6 a new problem that enables the deletion of scaling symmetries. We
show the existence and uniqueness of the _min-mass_ problem. As such, it can
also be used to remove the symmetries of any network. While scalable to
current architectures, the convex optimization is slower than the
normalization of the weights, and its behavior may be slightly more difficult
to grasp. This is why we stick to the former algorithm when many computations
are needed.

##### Removing softmax additive symmetries

To remove this type of symmetry, it is sufficient to scale the sum of the
biases of the last layer to some constant, for instance, to 1.

###  C.3 Permutation-equivariance of the gradient of the loss

Before coming to the equivariance of the training operator, we start by
proving the following lemma.

######  Lemma 1.

Let $f_{\theta}$ be a neural network trained with a loss
$\mathcal{L}(\theta)$, the gradient of the loss $\nabla\mathcal{L}$ is
permutation equivariant, i.e.,

$\forall\Pi\in\mathbb{\Pi},\
\nabla\mathcal{L}(\mathcal{T}_{p}(\theta,\Pi))=\mathcal{T}_{p}(\nabla\mathcal{L}(\theta),\Pi).$
(48)

Please be aware that the notation
$\mathcal{T}_{p}(\nabla\mathcal{L}(\theta),\Pi)$ signifies that we apply a
permutation operation to the gradient of the loss function $\mathcal{L}$ with
respect to the weight $\theta$.

###### Proof.

Let $f_{\theta}$ be a neural network trained with a loss $\mathcal{L}(\theta)$
and $\Pi\in\mathbb{\Pi}$. We prove the case with $\Pi$ containing only
Identity matrices except for the $i$-th matrix, that we denote $\bm{\pi}$.

From property C.1, we have that $\forall l\in\llbracket
1,L\rrbracket,\nabla_{W^{[l]}}\mathcal{L}(\theta)={\bm{\delta}}^{[l]}(\theta){\bm{a}}^{[l-1]^{\intercal}}(\theta)$
with ${\bm{\delta}}^{[l]}(\theta)=\nabla_{{\bm{z}}^{[l]}}\mathcal{L}(\theta)$
and ${\bm{a}}^{[l-1]}(\theta)$ the activation after the layer $l-1$. We add
the notation of dependence of ${\bm{\delta}}^{[l]}$ and ${\bm{a}}^{[l]}$ in
$\theta$ to express that they correspond to the gradient and the activation of
the network of weights $\theta$. We can adapt this result to the layer $i$ and
$\mathcal{T}(\theta,\Pi)$:

$\displaystyle\nabla_{\bm{\pi}W^{[i]}}\mathcal{L}(\mathcal{T}_{p}(\theta,\Pi))$
$\displaystyle={\bm{\delta}}^{[i]}(\mathcal{T}_{p}(\theta,\Pi)){\bm{a}}^{[i-1]^{\intercal}}(\mathcal{T}_{p}(\theta,\Pi))$
(49)
$\displaystyle=\left(\left(W^{[i+1]}\bm{\pi}^{\intercal}\right)^{\intercal}{\bm{\delta}}^{[i+1]}(\mathcal{T}_{p}(\theta,\Pi))\circ\nabla_{\bm{\pi}{\bm{z}}^{[l]}}\bm{\pi}{\bm{a}}^{[l]}(\theta)\right){\bm{a}}^{[i-1]^{\intercal}}(\mathcal{T}_{p}(\theta,\Pi))$
(50)
$\displaystyle=\left(\bm{\pi}W^{[i+1]^{\intercal}}{\bm{\delta}}^{[i+1]}(T(\theta,\Pi))\circ\bm{\pi}\nabla_{{\bm{z}}^{[l]}}{\bm{a}}^{[l]}(\theta)\right){\bm{a}}^{[i-1]^{\intercal}}(\mathcal{T}_{p}(\theta,\Pi))$
(51)
$\displaystyle=\left(\bm{\pi}W^{[i+1]^{\intercal}}{\bm{\delta}}^{[i+1]}(\theta)\circ\bm{\pi}\nabla_{{\bm{z}}^{[l]}}{\bm{a}}^{[l]}(\theta)\right){\bm{a}}^{[i-1]^{\intercal}}(\theta)$
(52)
$\displaystyle=\bm{\pi}{\bm{\delta}}^{[l]}(\theta){\bm{a}}^{[i-1]^{\intercal}}(\theta)$
(53) $\displaystyle=\bm{\pi}\nabla_{W^{[i]}}\mathcal{L}(\theta)$ (54)

This proof can be extended backward to any $\Pi\in\mathbb{\Pi}$ as the action
of $\pi^{\intercal}$ and of $\pi$ cancel out for all the following weights. ∎

We proved Lemma 1 with an MLP, but this result extends to convolutional
layers. To do this, we need to define that the permutation matrices apply on
the channels of the weight tensors.

###  C.4 Permutation-equivariance of stochastic gradient descent

To simplify the notations, we use neural networks and weights indifferently as
arguments for the optimization and symmetry operators $\star$ and
$\mathcal{T}$. For instance, we will simplify $\star(f_{\theta})$ by
$\star(\theta)$, implicitly assuming the architecture of $f_{\theta}$.

######  Proposition 3.

Let $f_{\theta_{0}}$ be a randomly initialized neural network and $\star_{s}$
the stochastic gradient descent operator applying $s$ times the update rule of
learning rate $\gamma_{s}$. For all steps $s\in\mathbb{N}$, $\star_{s}$ is
permutation equivariant. In other words,

$\forall s\in\mathbb{N},\forall\Pi\in\mathbb{\Pi},\
\star_{s}(\mathcal{T}_{p}(\theta,\Pi))=\mathcal{T}_{p}(\star_{s}(\theta),\Pi).$
(55)

We provide the proof below for multi-layer perceptrons, but it extends to
convolutional neural networks.

###### Proof.

Let $\Pi\in\mathbb{\Pi}$ be a permutation set applied on a neural network
$f_{\theta}$ trained with an objective $\mathcal{L}$ of gradient
$\nabla\mathcal{L}$. Let us define $P(s)$ as follows:

$P(s)="\mathcal{T}_{p}\left(\star_{s}\left(\theta_{0}\right),\Pi\right)=\star_{s}\left(\mathcal{T}_{p}\left(\theta_{0},\Pi\right)\right)"$
(56)

We give a proof that $P(s)$ is verified for all $s\in\mathbb{N}$ by induction
on $s$.

_Base case:_ $P(0)$ is trivially true as
$\star_{0}\left(f_{\theta_{0}}\right)=f_{\theta_{0}}$, hence

$\star_{0}\left(\mathcal{T}_{p}\left(\theta_{0},\Pi\right)\right)=\mathcal{T}_{p}\left(\theta_{0},\Pi\right)=\mathcal{T}_{p}\left(\star_{0}\left(\theta_{0}\right),\Pi\right).$
(57)

_Induction step:_ Assume the induction hypothesis that for a particular $k$,
the single case $s=k$ holds, meaning $P(k)$ is true:

$\mathcal{T}_{p}\left(\star_{k}\left(\theta_{0}\right),\Pi\right)=\star_{k}\left(\mathcal{T}_{p}\left(\theta_{0},\Pi\right)\right)$
(58)

Let us start with $\mathcal{T}_{p}(\star_{k+1}(\theta_{0}),\Pi)$. We have that

$\displaystyle\mathcal{T}_{p}(\star_{k+1}(\theta_{0}),\Pi)=$
$\displaystyle\mathcal{T}_{p}(\theta_{k+1},\Pi)$ (59) $\displaystyle=$
$\displaystyle\mathcal{T}_{p}(\theta_{k}-\gamma_{s}\nabla\mathcal{L}(\theta_{k}),\Pi)$
(60) $\displaystyle=$
$\displaystyle\mathcal{T}_{p}(\star_{k}(\theta_{0}),\Pi)-\gamma_{s}\mathcal{T}_{p}(\nabla\mathcal{L}(\star_{k}(\theta_{0})),\Pi)$
(61) $\displaystyle=$
$\displaystyle\star_{k}(\mathcal{T}_{p}(\theta_{0}),\Pi)-\gamma_{s}\nabla\mathcal{L}(\star_{k}(\mathcal{T}_{p}(\theta_{0},\Pi)))$
(62) $\displaystyle=$
$\displaystyle\star_{k+1}(\mathcal{T}_{p}(\theta_{0},\Pi)),$ (63)

Equation 62 uses the permutation-equivariance of both the gradient (see Lemma
1) and the training operator until step $k$. Furthermore, equation 63 is
exactly $P(k+1)$, establishing the induction step.

_Conclusion:_ Since both the base case and the induction step have been
proven, by induction, the statement $P(s)$ holds for every natural number $s$.
∎

We provide the proof of Proposition 3 with SGD for simplicity. However, it
intuitively extends to SGD with momentum since the momentum is also
permutation-equivariant and updated in a permutation-equivariant way.
Furthermore, it also generalizes to (mini-)batch stochastic gradient descent,
provided that the batches contain the same images.

While this also extends to the Adam optimizer (Kingma & Ba, 2015), this is
most likely not true for new meta-learned optimizers such as VeLO (Metz et
al., 2022).

###  C.5 Equiprobability of the permutations

We build upon the results from the previous sections to propose the following
proposition.

######  Proposition 4.

Let $f_{\theta}$ be a neural network with initial weights independently and
layer-wise identically distributed. The probability of converging towards any
symmetrically permuted network $\mathcal{T}_{p}(f_{\theta},\Pi)$ given the
dataset and training hyperparameters does not depend on the permutation set
$\Pi$. In other words,

$p(\Theta_{s}=\mathcal{T}_{p}(\theta_{s},\Pi)|\mathcal{D})=p(\Theta_{s}=\theta_{s}|\mathcal{D}).$
(64)

###### Proof.

For simplicity, we assume an optimization scheme with early stopping and final
weights $\theta_{s}$ at the end of step $s$. Given $\theta_{s}$, we can denote
$\theta_{0\rightharpoonup s}$ the space of initializations with non-zero
probability to converge to $\theta_{s}$ after $s$ steps.

Let there be a permutation set $\Pi$ leaving $f_{\theta_{s}}$ invariant. We
want to prove Equation 64, i.e., that the probability to converge to
$\mathcal{T}_{p}(\Theta_{s}=\theta_{s},\Pi)$ given $\mathcal{D}$ is the same
as for $\theta_{s}$ for any permutation set $\Pi\in\mathbb{\Pi}$.

We can inject information about the potential initialization points, defining
$\theta_{0\rightharpoonup s}=\\{\theta|\star_{s}[\theta]=\theta_{s}\\}$ as the
pre-image of $\theta_{s}$ by the optimization procedure with $s$ steps. It
comes that

$\displaystyle
p(\Theta_{s}=\theta_{s},\mathcal{D})=p(\Theta_{0}\in\theta_{0\rightharpoonup
s},\Theta_{s}=\theta_{s},\mathcal{D}).$ (65)

Indeed, as we use the notation $\Theta_{0}\in\theta_{0\rightharpoonup s}$ for

$p(\Theta_{0}\in\theta_{0\rightharpoonup
s})=\int\limits_{\theta\in\theta_{0\rightharpoonup
s}}p(\Theta_{0}=\theta)d\theta,$ (66)

we can get, with $\Omega$ the set of possible weights,

$\displaystyle\int\limits_{\theta\in\Omega}p(\Theta_{0}=\theta|\Theta_{s}=\theta_{s},\mathcal{D})d\theta=\int\limits_{\theta\in\theta_{0\rightharpoonup
s}}p(\Theta_{0}=\theta|\Theta_{s}=\theta_{s},\mathcal{D})d\theta+\int\limits_{\theta\in\Omega\setminus\theta_{0\rightharpoonup
s}}p(\Theta_{0}=\theta|\Theta_{s}=\theta_{s},\mathcal{D})d\theta$ (67)

The left term is equal to 1, and so is the middle term as, by definition of
$\theta_{0\rightharpoonup s}$, it is impossible to have initialized the
weights in $\Omega\setminus\theta_{0\rightharpoonup s}$ and converge to
$\theta_{s}$ given the dataset $\mathcal{D}$. Since
$p(\Theta_{0}\in\theta_{0\rightharpoonup
s},\Theta_{s}=\theta_{s},\mathcal{D})=p(\Theta_{0}\in\theta_{0\rightharpoonup
s}|\Theta_{s}=\theta_{s},\mathcal{D})p(\Theta_{s}=\theta_{s},\mathcal{D})$, we
have Equation 65.

Furthermore, the definition of conditional probabilities implies that the
probability of converging to $\theta_{s}$ given $\mathcal{D}$ can be rewritten
as

$\displaystyle p(\Theta_{s}=\theta_{s}|\mathcal{D})$
$\displaystyle=\frac{p(\Theta_{0}\in\theta_{0\rightharpoonup
s},\Theta_{s}=\theta_{s},\mathcal{D})\cdot
p(\Theta_{0}\in\theta_{0\rightharpoonup s})}{p(\mathcal{D})\cdot
p(\Theta_{0}\in\theta_{0\rightharpoonup s})}$ (69)
$\displaystyle=\frac{p(\Theta_{0}\in\theta_{0\rightharpoonup
s},\Theta_{s}=\theta_{s},\mathcal{D})\cdot
p(\Theta_{0}\in\theta_{0\rightharpoonup
s})}{p(\Theta_{0}\in\theta_{0\rightharpoonup
s},\mathcal{D})}\quad[\text{independence of}\ \Theta_{0}\ \text{and}\
\mathcal{D}]$ (70)
$\displaystyle=p(\Theta_{s}=\theta_{s}|\Theta_{0}\in\theta_{0\rightharpoonup
s},\mathcal{D})p(\Theta_{0}\in\theta_{0\rightharpoonup s}).$ (71)

We can also apply Equation 71 for a permuted network. Let there be a
permutation set $\Pi$; Equation 71 is also valid for the permuted network
$\mathcal{T}_{p}(\theta_{s},\Pi)$ with its pre-image the set
$\mathcal{T}_{p}(\theta_{0\rightharpoonup s},\Pi)$ (defined as
$\\{\mathcal{T}_{p}(\theta,\Pi)\\}_{\theta\in\theta_{0\rightharpoonup s}}$).
Indeed, Proposition 3 exactly proves that training a network starting from the
permuted weights $\mathcal{T}_{p}(\theta_{0},\Pi)$ converges towards the
permutation of the original weights $\mathcal{T}_{p}(\theta_{s},\Pi)$.

$p(\Theta_{s}=\mathcal{T}_{p}(\theta_{s},\Pi)|\mathcal{D})=p(\Theta_{s}=\mathcal{T}_{p}(\theta_{s},\Pi)|\Theta_{0}\in\mathcal{T}_{p}(\theta_{0\rightharpoonup
s},\Pi),\mathcal{D})p(\Theta_{0}\in\mathcal{T}_{p}(\theta_{0\rightharpoonup
s},\Pi))$ (72)

_Deterministic Training:_ Suppose the optimization algorithm is fully
deterministic, given the initialization state. In practice, the dataloader
seed would be set to some value, and the backpropagation kernels would be
chosen as deterministic. In that case, the convergence to $\theta_{s}$ is
fully determined by the initialization $\Theta_{0}$ and the dataset
$\mathcal{D}$, i.e.,

$p(\Theta_{s}=\theta_{s}|\Theta_{0}\in\theta_{0\rightharpoonup
s},\mathcal{D})=1$ (73)

_Non-deterministic Training:_ In the non-deterministic setting, corresponding
in practice to the order of the inputs in the batches as well as the
stochasticity of backpropagation algorithms. We can marginalize the
(eventually multivariate) sources of stochasticity $\xi\in\Xi$ to regain a
deterministic relationship.

$\displaystyle p(\Theta_{s}=\theta_{s}|\Theta_{0}\in\theta_{0\rightharpoonup
s},\mathcal{D})$
$\displaystyle=\int\limits_{\xi\in\Xi}p(\Theta_{s}=\theta_{s}|\Theta_{0}\in\theta_{0\rightharpoonup
s},\xi,\mathcal{D})d\xi$ (74)
$\displaystyle=\int\limits_{\xi\in\Xi}p(\Theta_{s}=\mathcal{T}_{p}(\theta_{s},\Pi)|\Theta_{0}\in\mathcal{T}_{p}(\theta_{0\rightharpoonup
s},\Pi),\xi,\mathcal{D})d\xi$ (75)
$\displaystyle=p(\Theta_{s}=\mathcal{T}_{p}(\theta_{s},\Pi)|\Theta_{0}\in\mathcal{T}_{p}(\theta_{0\rightharpoonup
s},\Pi),\mathcal{D})$ (76)

In the case of deterministic training, the left term of Equation 72 is equal
to 1 from Equation 73 with the permuted weights
$\mathcal{T}_{p}(\theta_{0},\Pi)$ and $\mathcal{T}_{p}(\theta_{s},\Pi)$.

For the right term, $p(\Theta_{0}\in\mathcal{T}_{p}(\theta_{0\rightharpoonup
s},\Pi))=p(\Theta_{0}\in\theta_{0\rightharpoonup s})$, since the initial
weights are layer-wise identically distributed and the permutations act
separately inside the layers.

Finally, we have

$\displaystyle
p(\Theta_{s}=\mathcal{T}_{p}(\theta_{s},\Pi)|\mathcal{D})=p(\Theta_{s}=\theta_{s}|\mathcal{D}),$
(77)

which is Proposition 4. ∎

###  C.6 Theoretical insights on the Bayesian posterior

We start with the following proposition, independent from Proposition 1 but
that we still deem interesting for the reader.

######  Proposition 5.

Let $f_{\bm{\omega}}$ be a neural network, ${\bm{x}}$ an input vector, and $y$
a scalar. We also denote $\Omega_{>}$ the space of sorted weights, and
$\mathbb{\Pi}$ remains the set of possible permutations for $f_{\bm{\omega}}$.
We have

$p(y\mid{\bm{x}},\mathcal{D})=|\mathbb{\Pi}|\int\limits_{{\bm{\omega}}\in\Omega_{>}}p(y\mid{\bm{x}},{\bm{\omega}})p({\bm{\omega}}\mid\mathcal{D})d{\bm{\omega}}.$
(78)

###### Proof.

Let $f_{\bm{\omega}}$ be a neural network. We recall the marginalization on
the network’s weights:

$p(y\mid{\bm{x}},\mathcal{D})=\int\limits_{{\bm{\omega}}\in\Omega}p(y\mid{\bm{x}},{\bm{\omega}})p({\bm{\omega}}\mid\mathcal{D})d{\bm{\omega}}.$
(79)

We can split the space $\Omega$ and get
$\Omega=\Omega_{\geq}\cup\bigcup\limits_{i\in\llbracket
1,|\mathbb{\Pi}|\rrbracket}\Omega_{>_{i}}$, with $\Omega_{>_{i}}$ the space of
the weights ordered and permuted according to $\Pi$ and $\Omega_{\geq}$ the
space including corner-cases: the networks with at least twice the same
weights in a layer. In our case, $\Omega_{\geq}$ is theoretically of measure
zero. In practice, we note that the reals are discretized on a computer, and
the edge cases may happen with an extremely small probability, decreasing with
the chosen precision. Denote $\Omega_{>}$ the space with descending ordered
weights, we have

$p(y\mid{\bm{x}},\mathcal{D})=\sum\limits_{i=1}^{|\mathbb{\Pi}|}\int\limits_{{\bm{\omega}}\in\Omega_{>}}p(y\mid{\bm{x}},\mathcal{T}_{p}({\bm{\omega}},\Pi_{i}))p(\mathcal{T}_{p}({\bm{\omega}},\Pi_{i})\mid\mathcal{D})d{\bm{\omega}}.$
(80)

By definition of the symmetry operator, we know that it does not affect the
prediction of the transformed network:

$\forall i\in\llbracket 1,|\mathbb{\Pi}|\rrbracket,\
p(y\mid{\bm{x}},\mathcal{T}_{p}({\bm{\omega}},\Pi_{i}))=p(y\mid{\bm{x}},{\bm{\omega}}).$
(81)

Moreover, Proposition 4 provides

$\forall i,j\in\llbracket 1,|\mathbb{\Pi}|\rrbracket,\
p(\mathcal{T}_{p}({\bm{\omega}},\Pi_{i})\mid\mathcal{D})=p({\bm{\omega}}\mid\mathcal{D}).$
(82)

It follows that

$p(y\mid{\bm{x}},\mathcal{D})=|\mathbb{\Pi}|\int\limits_{{\bm{\omega}}\in\Omega_{>}}p(y\mid{\bm{x}},{\bm{\omega}})p({\bm{\omega}}\mid\mathcal{D})d{\bm{\omega}}.$
(83)

∎

Then, we recall Proposition 1 of the main paper:

######  Proposition 6.

Define $f_{\omega}$ a neural network and $\tilde{\omega}$ its corresponding
identifiable model - a network transformed for having sorted unit-normed
neurons. Let us also denote $\mathbb{\Pi}$ and $\mathbb{\Lambda}$,
respectively, the sets of permutation sets and scaling sets, and
$\tilde{\Omega}$ the random variable of the sorted weights with unit norm. The
Bayesian posterior of a neural network $f_{\omega}$ trained with stochastic
gradient descent can be expressed as a continuous mixture of a discrete
mixture:

$p(\Omega={\bm{\omega}}\mid\mathcal{D})=|\mathbb{\Pi}|^{-1}\int\limits_{\Lambda\in\mathbb{\Lambda}}\sum\limits_{\Pi\in\mathbb{\Pi}}p(\tilde{\Omega}=\mathcal{T}_{p}(\mathcal{T}_{s}({\bm{\omega}},\Lambda),\Pi),\Lambda\mid\mathcal{D})d\Lambda.$
(84)

###### Proof.

Let $f_{\omega}$ be a neural network and start with the right term. We can
denote by $\tilde{\Pi}$ and $\tilde{\Lambda}$ the only couple of permutations
and scale such that the weights
$\mathcal{T}_{p}(\mathcal{T}_{s}({\bm{\omega}},\tilde{\Lambda}),\tilde{\Pi})$
are sorted and unit-normed. Given that $\tilde{\Omega}$ is also sorted and
unit-normed, $\tilde{\Pi}$ and $\tilde{\Lambda}$ are the only supports of the
respective sum and integral. Therefore, we have that

$|\mathbb{\Pi}|^{-1}\int\limits_{\Lambda\in\mathbb{\Lambda}}\sum\limits_{\Pi\in\mathbb{\Pi}}p(\tilde{\Omega}=\mathcal{T}_{p}(\mathcal{T}_{s}({\bm{\omega}},\Lambda),\Pi),\Lambda\mid\mathcal{D})d\Lambda=|\mathbb{\Pi}|^{-1}p(\tilde{\Omega}=\mathcal{T}_{p}(\mathcal{T}_{s}({\bm{\omega}},\tilde{\Lambda}),\tilde{\Pi}),\tilde{\Lambda}\mid\mathcal{D}).$
(85)

In parallel, we have that, with $\tilde{{\bm{\omega}}}$ the sorted unit-normed
weights associated to ${\bm{\omega}}$, i.e.,
$\tilde{{\bm{\omega}}}=\mathcal{T}_{p}(\mathcal{T}_{s}({\bm{\omega}},\tilde{\Lambda}),\tilde{\Pi})$,

$\displaystyle p(\Omega={\bm{\omega}}|\mathcal{D})$
$\displaystyle=p(\tilde{\Omega}=\tilde{{\bm{\omega}}},\tilde{\Pi},\tilde{\Lambda}|\mathcal{D})$
(86)
$\displaystyle=p(\tilde{\Omega}=\tilde{{\bm{\omega}}},\tilde{\Lambda}|\mathcal{D})p(\tilde{\Pi})$
(87)
$\displaystyle=|\mathbb{\Pi}|^{-1}p(\tilde{\Omega}=\tilde{{\bm{\omega}}},\tilde{\Lambda}|\mathcal{D}).$
(88)

Indeed, $\Pi$ and $\mathcal{D}$ are independent and $p(\Pi)$ is determined by
the initialization of the neural network that we suppose layer-wise
identically and independently distributed. We obtain the Proposition directly
when replacing $\tilde{{\bm{\omega}}}$ by
$\tilde{{\bm{\omega}}}=\mathcal{T}_{p}(\mathcal{T}_{s}({\bm{\omega}},\tilde{\Lambda}),\tilde{\Pi})$.
∎

###  C.7 The minimum scaled network mass problem

In this section, we develop the formalism leading to the proof of the log-log
strict convexity of the minimum scaled network mass problem (or _min-mass_
problem). We start by recalling its definition.

####  C.7.1 Definitions

######  Definition C.6 (Minimum scaled network mass problem).

Let $f_{\theta}$ a neural network and $\bar{\theta}$ its weights without the
biases. We define the minimum scaled network mass problem or _min-mass_
problem as the minimization of $f_{\theta}$’s L2 regularization term (the
”mass”) under invariant scaling transformations. In other words,

$m^{*}=\min\limits_{\Lambda\in\mathbb{\Lambda}}\
\left|\mathcal{T}(\bar{\theta},\Lambda)\right|^{2}.$ (89)

We also denote the mass of the neural network $f_{\theta}$ as
$m(f_{\theta})=\left|\mathcal{T}(\bar{\theta},\Lambda)\right|^{2}$.

In this work, we are especially interested in the versions of this problem
that are not degenerate. This condition was not given in the main paper for
clarity, but please note that it is by no means restrictive for real-world
cases and could be handled properly for theory with some caution.

######  Definition C.7.

We say that the _min-mass_ problem of the network neural network $f_{\theta}$
is non-degenerate if, taking any $\Lambda\in\mathbb{\Lambda}$, we have that
for all scales $\lambda\in\Lambda$, at least one term in $\lambda_{i}^{-1}$ is
non-zero.

The fact that the problem is non-degenerate is important as it could lead to
the non-desirable divergence of some parameters in the problem.

####  C.7.2 Computing the mass of the network

For a neural network without skip-connections and constituted of linear and
convolutional layers only, we provide first a definition and use it to devise
an important property:

######  Definition C.8.

Let $W$ be the weight of a linear or convolutional layer. We denote $M(W)$,
the matrix representation of the mass of the network. For linear layers, we
have, with $\circ$, the Hadamard product,

$M(W)=W\circ W,$ (90)

and for convolutional layers of kernel size $(h,v)$,

$M(W)=\sum\limits_{i=1}^{h}\sum\limits_{j=1}^{v}\left(W\circ
W\right)[:,:,i,j].$ (91)

######  Property C.8.

Let $f_{\theta}$ be an $L$-layer neural network of weights
$\theta=\\{W^{[l]},b^{[l]}\\}_{l\in\llbracket 1,L\rrbracket}$. The mass of the
neural network scaled by $\Lambda\in\mathbb{\Lambda}$ with
$\Lambda=\\{\bm{\lambda}_{l}\\}_{l\in\llbracket 1,L-1\rrbracket}$ and
$\bm{\lambda}_{0}=\bm{\lambda}_{L}={\bm{1}}$ can be expressed as, with
$\mathrm{sum}(W)$ the sum of all elements of W,

$\displaystyle
m\left(f_{\mathcal{T}_{s}({\theta},\Lambda)}\right)=\sum\limits_{l=1}^{L}\mathrm{sum}\left(\bm{\lambda}_{l}^{2}\bigtriangledown\left(\bm{\lambda}_{l-1}^{-2}\rhd
M\left(W^{[l]}\right)\right)\right)$ (92)

This result can be extended to networks with batch normalization layers and
residual connections. To build the convex problem out of this formula, we use
the connection between the line-wise and column-wise products and linear
algebra, as presented in C.1.1.

####  C.7.3 Log-log convexity

Finally, we have the following core property on _min-mass_ problems:

######  Proposition 7.

Let $f_{\theta}$ be a neural network. The corresponding minimum scaled
network-mass problem is log-log strictly convex, provided that it is non-
degenerate. As such, the problem is equivalent to a strictly convex problem on
$\mathbb{R}^{|\mathbb{\Lambda}|}$ and admits a global minimum attained in a
single point denoted $\Lambda^{*}$.

###### Proof.

Let $f_{\theta}$ be an $L$-layer neural network of weights
$\theta=\\{W^{[l]},b^{[l]}\\}_{l\in\llbracket 1,L\rrbracket}$. We recall the
corresponding _min-mass_ problem:

$P\equiv\min\limits_{\Lambda\in\mathbb{\Lambda}}\
m\left(f_{\mathcal{T}_{s}({\theta},\Lambda)}\right).$ (93)

With Property C.8, we have, for $\Lambda\in\mathbb{\Lambda}$ with
$\Lambda=\\{\bm{\lambda}_{l}\\}_{l\in\llbracket 1,L-1\rrbracket}$ and
$\bm{\lambda}_{0}=\bm{\lambda}_{L}={\bm{1}}$,

$P\equiv\min\limits_{\Lambda\in\mathbb{\Lambda}}\sum\limits_{l=1}^{L}\mathrm{sum}\left(\bm{\lambda}_{1}^{2}\bigtriangledown\left(\bm{\lambda}_{l-1}^{2}\rhd
M\left(W^{[l]}\right)\right)\right).$ (94)

Let us now write ${\bm{u}}_{l}=\log(\bm{\lambda}_{l})$ for all $l\in\llbracket
1,L-1\rrbracket$. This is possible because due to the non-negative homogeneity
of the ReLU activation, the $\bm{\lambda}_{l}$ were already chosen with
elements in $\mathbb{R}_{>0}$. Please note that the need for this change of
variables is why $P$ will be referred to as log-log strictly convex. We have,

$P\equiv\min\limits_{\Lambda\in\mathbb{\Lambda}}\sum\limits_{l=1}^{L}\mathrm{sum}\left(e^{2{\bm{u}}_{l}}\bigtriangledown\left(e^{2{\bm{u}}_{l-1}}\rhd
M\left(W^{[l]}\right)\right)\right)$ (95)

We can convert our operators to products of diagonal matrices and Hadamard
products using the formulae from C.1.1. It comes that,

$\displaystyle P\equiv\min\limits_{\Lambda\in\mathbb{\Lambda}}\Big{[}$
$\displaystyle\mathrm{sum}\left(\mathrm{diag}\left(e^{2{\bm{u}}_{1}}\right)M\left(W^{[1]}\right)\right)$
(96)
$\displaystyle+\sum\limits_{l=2}^{L-1}\mathrm{sum}\left(\left(e^{2{\bm{u}}_{l}^{\intercal}}e^{2{\bm{u}}_{l-1}^{-1}}\right)\circ
M\left(W^{[l]}\right)\right)$ (97)
$\displaystyle+\mathrm{sum}\left(M\left(W^{[L]}\right)\mathrm{diag}\left(e^{-2{\bm{u}}_{L-1}}\right)\right)\Big{]}.$
(98)

In Equation 98, we see that P is equivalent to a sum of convex and strictly
convex functions. Indeed, the product
$e^{2{\bm{u}}_{l}^{\intercal}}e^{2{\bm{u}}_{l-1}^{-1}}$ can be expressed as a
matrix containing elements of the form
$e^{2({\bm{u}}_{l})_{i}-2({\bm{u}}_{l-1})_{j}}$. Therefore, $P$ is equivalent
to a strictly convex problem. It comes that $P$ has, at most, a single
solution. Moreover, since the elements of the ${\bm{u}}_{l}$ are defined on
$\mathbb{R}$ and the inner function is infinite at infinity (as a sum of
positive terms with at least one of them diverging due as the problem is non-
degenerate), we have that $P$ has a solution, which is unique. ∎

This proof can be easily extended to batch normalization layers that include
one-dimensional weights. The extension to residual networks is a bit more
complicated, as one should probably resort to graphs to write the equations
properly.

We then prove the corollary proposed in the main paper.

######  Corollary 1.

Let $f_{\theta}$ be a neural network trained with L2-regularization. Suppose
the mass of $f_{\theta}$ is not optimal in the sense of the minimum scaled
network mass problem. In that case, there is an infinite number of equivalent
networks $\mathcal{T}_{p}(f_{\theta},\Lambda)$ with training loss lower than
that of the original network.

###### Proof.

Let $f_{\theta}$ be a neural network and denote the training loss by
$\mathcal{L}(\theta)=\mathcal{L}_{m}(\theta)+\mathcal{L}_{\mathrm{L2}}(\theta)$
with $\mathcal{L}_{\mathrm{L2}}(\theta)=m(f_{\theta})$ the L2-regularization
term. We have, by definition of the symmetry operator $\mathcal{T}_{s}$,

$\forall\Lambda\in\mathbb{\Lambda},\
\mathcal{L}_{m}(\theta)=\mathcal{L}_{m}(\mathcal{T}_{s}(\theta,\Lambda))$ (99)

Take $\Lambda^{*}$ as the optimal scaling set,
$\mathcal{L}_{\mathrm{L2}}(\theta)$ is continuous everywhere in
$\mathbb{\Lambda}$ as a sum of products of continuous functions and hence
continuous in $\Lambda^{*}\in\mathbb{\Lambda}$. Therefore, we can find an open
ball $B$ around $\Lambda^{*}$, where $\forall\Lambda\in L,\
\mathcal{L}_{m}(\mathcal{T}_{s}(\theta,\Lambda)>\mathcal{L}_{m}(\mathcal{T}_{s}(\theta,\Lambda^{*})$.
Hence the corollary.

∎

##  Appendix D Other discussions and details

###  D.1 Estimating the number of permutation-generated modes

Let us suppose for simplicity that the layer-posterior of the identifiable
model is an $n$-dimensional Dirac distribution of coordinates ${\bm{\theta}}$
(or a distribution on a small bounded domain around ${\bm{\theta}}$). The
number of modes of the resulting posterior will be $(n-k)!$ when we define $k$
as the number of (nearly-)identical terms in ${\bm{\theta}}$. To recall, $n$
is of the order of $10^{6000}$ for a ResNet-18 in good part due to the large
final layers.

This result seems difficult to extend to other distributions. While there
exist some results on the number of modes of a mixture of Gaussians, to the
best of our knowledge, they either treat the case with two elements only
(Eisenberger, 1964) or are conjectures of an upper bound (Carreira-Perpinán &
Williams, 2003).

On an experimental basis, we tried to use the MeanShift algorithm (Comaniciu &
Meer, 2002) to estimate the number of modes. Despite the number of
checkpoints, Mean Shift remained very sensitive to the bandwidth. It was, as
expected, failing to detect modes in high dimensions but also providing
unsatisfactory results in simpler cases, such as for the MLP.

###  D.2 Details on the functional collapse in ensembles

Figure 7: There is a slightly more important functional collapse between
couples of independently trained OptuNet on MNIST. Contrary to ResNet-18 on
CIFAR-100, the in-distribution and out-of-distribution mutual information
(IDMI, resp. OODMI) seem correlated for small values.

##### Functional collapse on MNIST/FashionMNIST with OptuNet

In this paragraph, we study the potential functional collapse of OptuNet on
MNIST and FashionMNIST, depicted in Figure 7. First, we see that the
dispersion of the mutual information both in the training domain (right) and
out of the domain (center) is greater than that of the ResNet-18. Indeed, in
the left plot of Figure 7, there is a non-negligible probability of obtaining
mutual information twice lower than the expected value. We note that the
dispersion of the out-of-distribution values on FashionMNIST is also greater
than for ResNets. Finally, in the case of OptuNet and contrasting with the
results obtained with ResNets, there seems to be some positive correlation
between the in-distribution MI and the OOD MI, at least when considering the
lowest values. We can conclude that there is a high chance of _functional
collapse_ in small networks, which is expected considering that their
posterior is simpler.

##### Functional collapse on TinyImageNet and Textures

Figure 8: The dispersion of the mutual information on TinyImageNet and
Textures with ResNet-18 remains very small. As for ResNet-18 on CIFAR-100, the
in-distribution and out-of-distribution mutual information (IDMI, resp. OODMI)
seem only very little correlated.

Figure 8 shows the mutual information on both the in-distribution (left) and
out-of-distribution (center) datasets and their correlation (right). The plots
are much closer to Figure 3, which was expected as they share the same
architecture. However, we see that the OOD dataset (here, Textures)
potentially impacts the values of the mutual information. The dispersion
(center) is lower than for ResNet-18 on SVHN, which may be more diverse. We
measure the Pearson’s $\rho$ correlation coefficient and obtain a
statistically significant value of $\rho\approx 0.06$. The two values are,
therefore, positively correlated with a very low coefficient.

###  D.3 Visualization

Figure 9: Marginal posteriors of the last weights from a multi-layer
perception: The top rows are the original posterior, the middle rows have been
scaled to a unit norm, and the bottom row has been permuted. We indicate the
p-values of a Kolmogorov-Smirnov two-sample test between the marginals of the
two neurons.

In this section, we propose to visualize the marginals of the posterior of
neural networks, including OptuNets. This visualization proposes the original
marginal distributions and versions without scaling and permutation
symmetries. As explained in the main paper, a corollary of Proposition 1, is
that all layer-wise marginals are identical.

Figure 9 provides the marginal distributions of the weights of a multi-layer
perceptron trained on energy-efficiency. We check with two-sample Kolmogorov-
Smirnov tests that we cannot rule out that the distributions are the same,
indicating the corresponding p-values. Indeed, as long as the permutations are
not removed, we keep the equivalence property of all the marginals.

As stated in the abstract, the posterior lies in a very high dimension. We
tried to apply dimension reduction with principal component analysis with
scikit-learn (Pedregosa et al., 2011), but the explained variance increased
very slowly with the number of elements.

###  D.4 Symmetries in modern layers

In this section, we provide details on how we compute symmetries throughout
the paper.

####  D.4.1 Convolutional layers

In Appendix C, we generalize the row-wise and column-wise products to enable
their use between vectors and four-dimensional tensors. We also detail how to
understand the product between permutation matrices and four-dimensional
tensors.

On a higher level, we see that the number of symmetries of convolutional
layers is smaller compared to the number of parameters in a linear layer.
Indeed, in the formalism we use, the number of permutations and scaling
degrees of freedom (sDoF) of a linear layer correspond to its number of output
neurons. In contrast, the number of permutations of a convolutional layer
equals its number of output channels. The ratio of the number of
permutations/sDoF over the number of parameters between linear and
convolutional layers, therefore, equals the surface of the kernel of the
convolutional layer.

####  D.4.2 Grouped-Convolutional layers

The number of permutations of a grouped convolutional layer (Krizhevsky et
al., 2012) reduces the number of permutations of the previous layer. Indeed,
we have to account for the impossibility of fully reversing the permutation
applied on the output neurons/channels of the preceding layer. This implies
that the permutations of the previous layer need to remain intra-group. Say
that the previous layer has $C_{\mathrm{out}}$ output channels and that the
current layer has $\gamma$ groups. Instead of $C_{\mathrm{out}}!$ possible
permutations, they are reduced to
$\left(\frac{C_{\mathrm{out}}}{\gamma}!\right)^{\gamma}$. On the other hand,
the number of output permutations does not change (provided that the next
layer does not have groups). Likewise, the number of scaling degrees of
freedom remains unchanged as scaling symmetries modify the values of the
weights channel per channel.

####  D.4.3 Batch normalization layers

Counting permutation symmetries on batch normalization layers (Ioffe &
Szegedy, 2015) is quite straightforward. Regarding permutation symmetries,
batch normalization layers are seamless, as one only needs to permute the
weights, bias, running mean, and running variance. As such, they do not
increase nor decrease the number of permutations.

On the other hand, batch normalization layers can increase the number of
degrees of freedom of scaling symmetries. Indeed, we can scale the running
variances and means to the input scaling coefficients and then choose new
output coefficients for the weights and biases. To conclude, they increase the
number of sDoF by their number of features.

####  D.4.4 Residual connections

Residual connections tend to reduce the number of permutation and scaling
symmetries (Kurle et al., 2021), and this is an important aspect in ResNets.
Indeed, residual connections need the output coefficient of the last layers
and the permutation matrices from all inner nodes in the computation graph to
be equal. This potentially reduces both the number of possible permutations
and the number of sDoF. In the case of ResNets, this effect is counterbalanced
by the shortcut convolutional layers that can create permutations and sDoF,
albeit less.