Proving Linear Mode Connectivity of Neural Networks via Optimal
  Transport
The energy landscape of high-dimensional non-convex optimization problems is
crucial to understanding the effectiveness of modern deep neural network
architectures. Recent works have experimentally shown that two different
solutions found after two runs of a stochastic training are often connected by
very simple continuous paths (e.g., linear) modulo a permutation of the
weights. In this paper, we provide a framework theoretically explaining this
empirical observation. Based on convergence rates in Wasserstein distance of
empirical measures, we show that, with high probability, two wide enough
two-layer neural networks trained with stochastic gradient descent are linearly
connected. Additionally, we express upper and lower bounds on the width of each
layer of two deep neural networks with independent neuron weights to be
linearly connected. Finally, we empirically demonstrate the validity of our
approach by showing how the dimension of the support of the weight distribution
of neurons, which dictates Wasserstein convergence rates is correlated with
linear mode connectivity.
Damien Ferbach, Baptiste Goujaud, Gauthier Gidel, Aymeric Dieuleveut
# Proving Linear Mode Connectivity of Neural Networks

###### Abstract

The energy landscape of high-dimensional non-convex optimization problems is crucial to understanding the effectiveness of modern deep neural network architectures. Recent works have experimentally shown that two different solutions found after two runs of a stochastic training are often connected by very simple continuous paths (e.g., linear) modulo a permutation of the weights. In this paper, we provide a framework theoretically explaining this empirical observation. Based on convergence rates in Wasserstein distance of empirical measures, we show that, with high probability, two wide enough two-layer neural networks trained with stochastic gradient descent are linearly connected. Additionally, we express upper and lower bounds on the width of each layer of two deep neural networks with independent neuron weights to be linearly connected. Finally, we empirically demonstrate the validity of our approach by showing how the dimension of the support of the weight distribution of neurons, which dictates Wasserstein convergence rates is correlated with linear mode connectivity.

+
Footnote †: dagger}\) Work done during an internship at Ecole Polytechnique

+
Footnote †: ddagger}\) Canada CIFAR AI Chair

## 1 Introduction and Related Work

Training deep neural networks on complex tasks is a high-dimensional, non-convex optimization problem. While stochastic gradient-based methods (i.e., SGD and its derivatives) have proven highly efficient in finding a local minimum with low test error, the loss landscape of deep neural networks (DNNs) still contains numerous open questions. In particular, Goodfellow et al. (2014) try to find ways to connect two local minima reached by two independent runs of the same stochastic algorithm with different initialization and data orders. This problem has applications in diverse domains such as model averaging (Izmailov et al., 2018; Rame et al., 2022; Wortsman et al., 2022), loss landscape study (Gotmare et al., 2018; Vlaar and Frankle, 2022; Lucas et al., 2021), adversarial robustness (Zhao et al., 2020) or generalization theory (Pittorino et al., 2022; Juneja et al., 2022; Lubana et al., 2023).

An answer to this question is the _mode connectivity phenomenon_. It suggests the existence of a continuous low-loss path connecting all the local minima found by a given optimization procedure. The mode connectivity phenomenon has extensively been studied in the literature (Goodfellow et al., 2014; Keskar et al., 2016; Sagun et al., 2017; Venturi et al., 2019; Neyshabur et al., 2020; Tatro et al., 2020) and _non-linear connecting paths_ have been evidenced for DNNs trained on MNIST and CIFAR10 by Freeman and Bruna (2016); Garipov et al. (2018); Draxler et al. (2018).

(Linear) mode connectivity.Formally, let \(A:=\hat{f}(.,\theta_{A})\) and \(B:=\hat{f}(.,\theta_{B})\) two neural networks sharing a common architecture \(\hat{f}\). They are parametrized by \(\theta_{A}\) and \(\theta_{B}\) after training those networks on a data distribution \(P\) with loss \(\mathcal{L}\), i.e. by minimizing \(\mathcal{E}(\theta):=\mathbb{E}_{(x,y)\sim P}[\mathcal{L}(\hat{f}(x,\theta),y)]\) over \(\theta\). Let \(p\) be a continuous path connecting \(\theta_{A}\) and \(\theta_{B}\), i.e. a continuous function defined on \([0,1]\) with \(p(0)=\theta_{A}\) and \(p(1)=\theta_{B}\). Frankle et al. (2020); Entezari et al. (2021) defined the _error barrier height_ of \(p\) as \(\sup_{t\in[0,1]}\mathcal{E}\left(p(t)\right)-\left((1-t)\mathcal{E}\left( \theta_{A}\right)+t\mathcal{E}\left(\theta_{B}\right)\right)\). The two found solutions \(\theta_{A}\) and \(\theta_{B}\) are said to be _mode connected_ if there is a continuous path with zero error barrier height connecting them. Furthermore, if \(p\) is linear, that is \(p(t)=(1-t)\theta_{A}+t\theta_{B},\ \theta_{A}\) and \(\theta_{B}\) are said to be _linearly mode connected (LMC)_.

Permutation invariance.Recently, Singh and Jaggi (2020), Ainsworth et al. (2022) highlighted the fact that the units in a hidden layer of a given model
can be permuted while preserving the network's functionality. Figure 1 shows how one can permute the hidden layer of a two-layer network to match a different target network without changing the source function. From now on, **we will understand LMC modulo permutation invariance**, i.e. two networks \(A,B\) are said to be linear mode connected whenever there exists a permutation of neurons in each hidden layer of network \(B\) such that the linear path in parameter space between network \(A\) and \(B\) permuted has low loss.

Linear mode connectivity up to permutation.Singh and Jaggi (2020) proposed to use optimal transport (OT) theory to find soft alignment providing a "good match" (in a certain sense) between the neurons of two trained DNNs. Furthermore, the authors propose ways to fusion the aligned networks together in a federated learning context with local-steps. Ainsworth et al. (2022) further experimentally studied linear mode connectivity between two pre-aligned networks. The authors first align network B's weights on the weights of network A before connecting both of them by a linear path in the parameter space. They notably achieved zero-loss barrier for two trained Resnets with SGD on CIFAR10. Moreover, their experiments strongly suggest that the error barrier on a linear path gets smaller for wider networks, with a detrimental effect of big depth.

Prior theoretical explanations.A recent work by Kuditipudi et al. (2019) shows that dropout stable networks (i.e. networks that are functionally stable to the action of randomly setting a fraction of their weights and normalizing the others) exhibit mode connectivity. Shevchenko and Mondelli (2020) use a mean field viewpoint to show that wide two-layer neural networks trained with SGD are dropout stable and hence show (non-linear) mode connectivity for two-layer neural networks in the mean field regime (i.e. one single wide hidden layer). Finally Entezari et al. (2021) show that two-layer neural networks exhibit linear mode connectivity up to permutation at initialization for parameters initialized following uniform independent distribution properly scaled. They highlight that this result could be extended to networks trained in the Neural Tangent Kernel regime where parameters stay close to initialization (Jacot et al., 2018).

Contributions.This paper aims at building theoretical foundations on the phenomenon of linear mode connectivity up to permutation. More precisely, we theoretically prove this phenomenon arises naturally on multi-layer perceptrons (MLPs), which goes beyond two-layer networks on which theoretical works focused so far. We also provide a new efficient way to find the right permutation to apply on the units of a neural network's layer. The paper is organized as follow:

* In Section 3, we focus on two-layer neural networks in the mean field regime. While Shevchenko and Mondelli (2020) proved _non-linear_ mode connectivity in this setting; we go further by proving _linear mode connectivity up to permutation_. Moreover, we provide an upper bound on the minimal width of the hidden layer to guarantee linear mode connectivity.
* In Section 4, we use general OT theory to exhibit tight asymptotics on the minimal width of a multi-layer perceptron (MLP) to ensure LMC.
* In Section 5, we apply our general results to networks with parameters following sub-Gaussian distribution. Our result holds for deep networks, generalizing the result of Entezari et al. (2021) with better bounds. We shed light on the dependence in the dimension of the underlying distributions of the weights in each layer and explain how it connects with previous empirical observations (Ainsworth et al., 2022). Using a model of approximately low dimensional weight distribution as a proxy of sparse feature learning, we yield more realistic bounds on the architectures of DNNs to ensure linear mode connectivity. We therefore, show why LMC is possible after training and how it depends on the complexity of the task. Finally we unify our framework with dropout stability.
* In Section 6, we validate our theoretical framework by showing how the implicit dimension of the weight distribution is correlated with linear mode connectivity for MLPs trained on MNIST with SGD and propose a new weight matching method.

## 2 Preliminaries and notations

**Notations.** Let two multilayer perceptrons (MLP) \(A\) and \(B\) with the same depth \(L+1\) (\(L\) hidden layers), an input dimension \(m_{0}\), intermediate widths

Figure 1: Permuting the neurons in the hidden layer of network \(B\) to align them on network \(A\)
and an output dimension \(m_{L+1}\). Given \(2(L+1)\) weights matrices \(W^{1,\ldots,L+1}_{A,B}\), and a non-linearity \(\sigma\), we define the neural network function of network A by \(\hat{f}_{A}\) (respectively \(\hat{f}_{B}\)): \(\forall x\in\mathbb{R}^{m_{0}}\),

\[\hat{f}_{A}(x):=\hat{f}(x;\theta_{A}):=W^{L+1}_{A}\sigma\left(W^{L}_{A}\ldots \sigma(W^{1}_{A}x)\right) \tag{1}\]

To \(W^{\ell}_{A}\in\mathcal{M}_{m_{\ell},m_{\ell-1}}(\mathbb{R})\) we associate \(\hat{\mu}_{A,\ell}\) the empirical measure of its rows \([W^{\ell}_{A}]_{i:}\in\mathbb{R}^{m_{\ell-1}}\): \(\frac{1}{m_{\ell}}\sum_{i=1}^{m_{\ell}}\delta_{[W^{\ell}_{A}]_{i:}}\) which belongs to the space of probability measures \(\mathcal{P}_{1}(\mathbb{R}^{m_{\ell-1}})\), where \([W^{\ell}_{A}]_{i:}\) is the \(i\)-th row of the matrix and \(\delta\) denotes the Dirac measure. Note that \([W^{\ell}_{A}]_{i:}\) is also the weights vector of the \(i\)-th neuron of the layer \(\ell\) of network \(A\). Given an equi-partition1\(\mathcal{I}^{\ell-1}=\{I^{\ell-1}_{1},...,I^{\ell-1}_{\bar{m}_{\ell-1}}\}\) of \([m_{\ell-1}]\) we denote \(W^{\mathcal{I}^{\ell-1}}_{A}\in\mathcal{M}_{m_{\ell},\bar{m}_{\ell-1}}( \mathbb{R})\) the matrix issued from \(W^{\ell}_{A}\) where we have summed the columns being in the same set of the partition \(\mathcal{I}^{\ell-1}\). In that case \(\hat{\mu}^{\mathcal{I}^{\ell-1}}_{A}\in\mathcal{P}_{1}\left(\mathbb{R}^{\bar{ m}_{\ell}}\right)\) denotes the associate empirical measure of its rows.

Footnote 1: All subsets have the same number of elements

Denote \(\phi^{\ell}_{A}(x)\ :=\ \sigma\left(W^{\ell}_{A}\ldots\sigma(W^{1}_{A}x)\right)\) (respectively \(\phi^{\ell}_{B}\)) the activations of neurons at layer \(\ell\) of network \(A\) on input \(x\). The data \(x\) follows a distribution \(P\) in \(\mathbb{R}^{m_{0}}\).

Given permutations matrices \(\Pi_{\ell}\in\mathcal{S}_{m_{\ell}}\)2\(\ell=1,\ldots,L\) of each hidden layer of network \(B\), the weight matrix at layer \(\ell\) of the permuted network \(B\) is \(\tilde{W}^{\ell}_{B}:=\Pi_{\ell}W^{\ell}_{B}\Pi^{T}_{\ell-1}\) and its new activation vector is \(\tilde{\phi}^{\ell}_{B}(x):=\Pi_{\ell}\phi^{\ell}_{B}(x)\). Finally, \(\forall t\in[0,1]\) we define \(M_{t}\) the convex combination of network \(A\) and \(B\) permuted, with weights matrices \(tW^{\ell}_{A}+(1-t)\tilde{W}^{\ell}_{B}\) and \(\phi^{\ell}_{M_{t}}\) its activations at layer \(\ell\).

Footnote 2: We use interchangeably \(\mathcal{S}_{m}\) to denote the space of permutations of \(\{1,\ldots,m\}\) and the corresponding space of permutations matrices. Given \(\pi\in\mathcal{S}_{m}\) its corresponding permutation matrix \(\Pi\) is defined as \(\Pi_{ij}=1\iff\pi(i)=j\).

**Preliminaries.** We consider networks \(A\) and \(B\) to be independently chosen from the same distribution \(Q\) on parameters. This is coherent with considering two networks initialized independently or trained independently with the same optimization procedure (SS3). We additionally suppose the choice of \(A\) and \(B\) to be independent of the choice of \(x\sim P\), which is valid when evaluating models on test data not seen during training. We denote \(\mathbb{E}_{Q},\mathbb{E}_{P},\mathbb{E}_{P,Q}\) expectations with respect to the choice of the networks, the data, or both.

To show linear mode connectivity of networks \(A\) and \(B\) we will show the existence of permutations \(\Pi_{1},...,\Pi_{L}\) of layers \(1,...,L\) that align the neurons of network \(B\) on the closest neurons weights of network \(A\) at the same layer as shown in Figure 1. In other words, we want to find permutations that minimize for each layer \(\ell\in[L]\) the norm \(\|W^{\ell}_{A}-\Pi_{\ell}W^{\ell}_{B}\Pi^{T}_{\ell-1}\|_{2}\). Recursively on \(\ell\), we solve the following optimization problem:

\[\begin{split}\Pi_{\ell}&=\operatorname*{arg\,min}_{ \Pi\in\mathcal{S}_{m_{\ell}}}\|W^{\ell}_{A}-\Pi W^{\ell}_{B}\Pi^{T}_{\ell-1}\| _{2}^{2}\\ &=\operatorname*{arg\,min}_{\pi\in\mathcal{S}_{m_{\ell}}}\frac{1} {m_{\ell}}\sum_{i=1}^{m_{\ell}}\|[W^{\ell}_{A}]_{i:}-[W^{\ell}_{B}\Pi^{T}_{ \ell-1}]_{\pi_{i}:}\|_{2}^{2}\end{split} \tag{2}\]

For each layer, the problem can be cast as finding a pairing of weights neurons \([W^{\ell}_{A}]_{i:}\) and \([W^{\ell}_{B}\Pi^{T}_{\ell-1}]_{\pi_{i}:}\) to minimize the sum of their Euclidean distances. It is known as the Monge problem is the optimal transport literature Peyre et al. (2019). More precisely Equation (2) can be formulated as finding an optimal transport plan corresponding to the Wasserstein distance between the empirical measures of the rows of \(W^{\ell}_{A}\) and \(W^{\ell}_{B}\Pi^{T}_{\ell-1}\). We provide more details about this connection between Equation (2) and optimal transport in Appendix A.2. In the following, the \(p-\)Wasserstein distance will be denoted \(\mathcal{W}_{p}(\cdot,\cdot)\) and defined with the underlying distance \(\|\cdot\|_{2}\) unless expressed otherwise.

By controlling the cost in Equation (2) at every layer, we show that the permuted s of networks \(A\) and \(B\) are approximately equal. Linearly interpolating both networks will therefore keep activations of all hidden layers unchanged except the last layer which acts as a linear function of the interpolation parameter \(t\in[0,1]\).

## 3 LMC for Two-Layer NNs in the Mean Field Regime

We will first study linear mode connectivity between two two-layer neural networks independently trained with SGD for the same number of steps.

### Background on the Mean Field Regime

We will use some notations from Mei et al. (2019) and consider a two-layer neural network,

\[\hat{f}_{N}(x;\theta)=\frac{1}{N}\sum_{i=1}^{N}\sigma_{*}(x;\theta_{i}) \tag{3}\]

parametrized by \(\theta_{i}=(a_{i},w_{i})\in\mathbb{R}\times\mathbb{R}^{d}\) and where \(\sigma_{*}(x;\theta_{i})=a_{i}\sigma(w_{i}x)\). The parameters evolve as to minimize the following regularized cost \(R_{N}(\theta)=\mathbb{E}_{(x,y)\sim P}[(y-\hat{f}_{N}(x;\theta))^{2}]+\lambda \|\theta\|_{2}^{2}\). Define noisy regularized stochastic gradient descent (or noiseless regularization-free when \(\lambda=0,\tau=0\)) with step size \(s_{k}\), and i.i.d. Gaussian noise \(g^{k}\sim\mathcal{N}(0,I_{d})\):

\[\theta_{i}^{k+1}=(1-2\lambda s_{k})\theta_{i}^{k}\] (SGD) \[+2s_{k}(y_{k}-\hat{f}_{N}(x_{k};\theta^{k}))\nabla_{\theta}\sigma_ {*}(x_{k};\theta_{i}^{k})+\sqrt{\frac{2s_{k}\tau}{4}}g_{i}^{k}\]

It will be useful to consider \(\rho_{N}^{k}:=\frac{1}{N}\sum_{i=1}^{N}\delta_{\theta_{i}^{k}}\) the empirical distribution of the weights after \(k\) SGD steps.
Indeed some recent works (Chizat and Bach, 2018; Mei et al., 2018, 2019) have shown that when setting the width \(N\) to be large and the step size \(s_{k}\) to be small, the empirical distribution of weights during training remains close to an empirical measure drawn from the solution of a partial differential equation (PDE) we explicit in Appendix B.1. Especially, the parameters \(\left\{\theta_{i}^{k},i\in[N]\right\}\) evolve approximately independently.

### Proving LMC in the mean field setting

Define respectively the alignment of a neuron function on the data and the correlation between two neurons:

\[V(\theta_{1}):=av(w):=-\mathbb{E}_{P}[y\sigma_{*}(x;\theta_{1})]\] \[U(\theta_{1},\theta_{2}):=a_{1}a_{2}u(w_{1},w_{2}):=\mathbb{E}_ {P}[\sigma_{*}(x;\theta_{1})\sigma_{*}(x;\theta_{2})]\,.\]

and we for \(\varepsilon>0\) fixed we note the step size

\[s_{k}=\varepsilon\xi(k\varepsilon) \tag{4}\]

where \(\xi\) is a positive scaling function. The underlying training time up to step \(k_{T}\) is defined as \(T:=\sum_{k=1}^{k_{T}}s_{k}\). We now state the standard assumptions to work on the mean field regime (Mei et al., 2019),

**Assumption 1**.: _The function \(t\mapsto\xi(t)\) is bounded Lipschitz. The non-linearity \(\sigma\) is bounded Lipschitz and the data distribution has a bounded support. The functions \(w\mapsto v(w)\) and \((w_{1},w_{2})\mapsto u(w_{1},w_{2})\) are differentiable, with bounded and Lipschitz continuous gradient. The weights at initialization \(\theta_{i}^{0}\) are i.i.d. with distribution \(\rho_{0}\) which has bounded support._

Assumption 1 imposes that the step size is of order \(\mathcal{O}(\varepsilon)\) and its variations are of order \(\mathcal{O}(\varepsilon^{2})\). Constant step size \(\varepsilon\) will work. Bounded non-linearity include arctan and sigmoid but excludes ReLU. While it is a standard assumption in mean field theory ((Mei et al., 2018, 2019)), we mention in SSB that this assumption can be relaxed by the weaker assumption that the non-linearity stays small on some big enough compact set.

The second assumption is technical and only used for studying noisy regularized SGD.

**Assumption 2**.: \(V,U\) _are four times continuously differentiable and \(\nabla_{1}^{k}u(\theta_{1},\theta_{2})\) is uniformly bounded for \(0\leq k\leq 4\)._

The following theorem states that two wide enough two-layer neural networks trained independently with SGD exhibit, with high probability, a linear connection of the prediction modulo permutations for all data.

**Theorem 3.1**.: _Consider two two-layer neural networks as in Equation (3) trained with equation SGD with the same initialization over the weights independently and for the same underlying time \(T\). Suppose Assumptions 1 and 2 to hold. Then \(\forall\delta,\text{err},\exists N_{\min}\) such that if \(N\geq N_{min},\exists\varepsilon_{\max}(N)\) such that if \(\varepsilon\leq\varepsilon_{\max}(N)\) in Equation (4), then with probability at least \(1-\delta\) over the training process, there exists a permutation of the second network's hidden layer such that for almost every \(x\sim P\):_

\[\left|t\tilde{f}_{N}(x;\theta_{A})+(1-t)\hat{f}_{N}(x;\theta_{B})\right.\] \[-\hat{f}_{N}(x;t\theta_{A}+(1-t)\tilde{\theta}_{B})|\leq\text{err},\quad\forall t\in[0,1]\,.\]

**Remark.** Assumption 2 is not used when studying noiseless regularization-free SGD (\(\lambda=0\), \(\tau=0\)).

**Corollary 3.2**.: _Under assumptions of Theorem 3.1, \(\forall\delta,\text{err}>0,\,\exists N^{\prime}_{\min},\,\forall N\geq N^{ \prime}_{\min},\,\exists\varepsilon^{\prime}_{\max}(N),\,\forall\varepsilon \leq\varepsilon^{\prime}_{\max}(N)\) in Equation (4), then with probability at least \(1-\delta\) over the training process, there exists a permutation of the second network's hidden layer such that \(\forall t\in[0,1]\):_

\[\mathbb{E}_{P}\big{[}\big{(}\hat{f}_{N}(x;t\theta_{A}+(1-t)\tilde{ \theta}_{B})-y\big{)}^{2}\big{]}\leq\text{err}\] \[+\mathbb{E}_{P}\big{[}t(\hat{f}_{N}(x;\theta_{A})-y)^{2}+(1-t)( \hat{f}_{N}(x;\theta_{B})-y)^{2}\big{]}\]

**Discussion.** Two wide enough two-layer neural networks wide enough trained with SGD are therefore Linear Mode Connected with an upper bound on the error tolerance we explicit in Appendix B. We have extensively used the independence between weights in the mean field regime to apply OT bounds on convergence rates of empirical measures. To go beyond the two-layer case, we will need to make such an assumption on the distribution of weights. Note that this is true at initialization and after training for two-layer networks. Studying the independence of weights in the multi-layer case is a natural avenue for future work, already studied in Nguyen and Pham (2020).

## 4 General Strategy for proving LMC of multi-layer networks

We now build the foundations to study the case of multi-layer neural networks (see Equation (1)).

We first write one formal property expressing the existence of permutations of neurons of network \(B\) up to layer \(\ell\) such that the activations of network \(A\), network \(B\) permuted and the mean network \(M_{t}\) are close up to layer \(\ell\). This property is trivially satisfied at the input layer. We then show that under two formal assumptions on the weights matrices of networks \(A\) and \(B\), this property still hold at layer \(\ell+1\).
### Formal Property at layer \(\ell\)

Let \(\varepsilon>0,m_{\ell}\geq\tilde{m}_{l}\) and \(m_{\ell+1}\geq\tilde{m}_{\ell+1}\). Assume \(\frac{m_{\ell}}{\tilde{m}_{\ell}},\frac{m_{\ell+1}}{\tilde{m}_{\ell+1}}\in \mathbb{N}\) to simplify technical details but this hypothesis can easily be removed.

**Property 1**.: _There exists two constants \(E_{\ell},E_{\ell}\) such that given weight matrices up to layer \(\ell\), \(W^{1,\dots,\ell}_{A,B},W^{1,\dots,\ell}_{B}\) one can find \(\ell\) permutations \(\Pi_{1},\cdots,\Pi_{\ell}\) of the neurons in the hidden layers \(1\) to \(\ell\) of network \(B\), an equi-partition \(\mathcal{I}^{\ell}=\{I^{\ell}_{1},\dots,I^{\ell}_{\tilde{m}_{\ell}}\}\), and a map \(\phi^{\ell}(x)\in\mathbb{R}^{n}\) such that \(\forall k\in[\tilde{m}_{\ell}]\,,\,\forall i,j\in I^{\ell}_{k},\phi^{\ell}_{i} (x)=\phi^{\ell}_{j}(x)\) such that:_

\[\mathbb{E}_{P,Q}\|\phi^{\ell}(x)\|_{2}^{2}\leq E_{\ell}m_{\ell}\] \[\mathbb{E}_{P,Q}\|\phi^{\ell}_{A}(x)-\phi^{\ell}(x)\|_{2}^{2}\leq E _{\ell}m_{\ell}\] \[\mathbb{E}_{P,Q}\|\tilde{\phi}^{\ell}_{B}(x)-\phi^{\ell}(x)\|_{2} ^{2}\leq E_{\ell}m_{\ell}\] \[\mathbb{E}_{P,Q}\|\phi^{\ell}_{M_{t}}(x)-\phi^{\ell}(x)\|_{2}^{2} \leq E_{\ell}m_{\ell}\,,\quad\forall t\in[0,1],\]

This property not only requires proximity between activations \(\phi^{\ell}_{A}(x),\tilde{\phi}^{\ell}_{B}(x)\) at layer \(\ell\) but requires the existence of a vector \(\tilde{\phi}^{\ell}(x)\) whose coefficients in the same groups of the partition \(\mathcal{I}^{\ell}\) are equal, and therefore lives in a \(\tilde{m}_{\ell}\). It bounds the size of the function space available at layer \(\ell\) and hence allows to use an effective width \(\tilde{m}_{\ell}\) independent of the real width \(m_{\ell}\), which can be much larger. It is crucial in order to show LMC for neural networks of constant width across layers. The introduction of such a map \(\underline{\phi}^{\ell}(x)\) is non trivial and is an important contribution since it allows to extend results of Entezari et al. (2021) beyond two layers.

### Assumptions on the weight distribution

We now make an assumption on the empirical distribution of the weights \(\hat{\mu}_{A,\ell+1}\) at layer \(\ell+1\) of \(W^{\ell+1}_{A}\).

**Assumption 3**.: _There exists an integer \(\tilde{m}_{\ell+1}\) such that for all equi-partition \(\mathcal{I}^{\ell}\) of \([m_{\ell}]\) with \(\tilde{m}_{\ell}\) sub-sets, there exists a random empirical measure \(\hat{\mu}_{\tilde{m}_{\ell+1}}\) independent of \(A\) and \(B\) composed of \(\tilde{m}_{\ell+1}\) vectors in \(\mathbb{R}^{m_{\ell}}\), such that \(\mathbb{E}_{Q}[\mathcal{W}^{2}_{2}(\hat{\mu}^{\mathcal{I}^{\ell}}_{A,\ell+1}, \hat{\mu}^{\mathcal{I}^{\ell}}_{\tilde{m}_{\ell+1}})]\leq C_{1}\)._

This assumption requires that the empirical distribution with \(m_{\ell+1}\) points of the neurons' weights of network \(A\) at layer \(\ell+1\) can be approximated by an empirical measure with a smaller \(\tilde{m}_{\ell+1}\) number of points. Note that it implies proximity in Wasserstein distance between \(\hat{\mu}^{\mathcal{I}^{\ell}}_{A}\) and \(\hat{\mu}^{\mathcal{I}^{\ell}}_{B}\) by a triangular inequality.

We finally assume some central limit behavior when summing the errors made for each neuron of layer \(\ell\).

**Assumption 4**.: _There exists a constant \(C_{2}\) such that \(\forall X\in\mathbb{R}^{m_{l}}\) we have:_

\[\max\left\{\mathbb{E}_{Q}[\|W^{\ell+1}_{A}X\|_{2}^{2}],\mathbb{E}_{Q}[\|W_{ \tilde{m}_{l+1}}X\|_{2}^{2}]\right\}\leq C_{2}\frac{m_{\ell+1}}{m_{\ell}}\|X\|_ {2}^{2},\]

Finally, we consider the following assumption on the non-linearity, verified for example by pointwise ReLU.

**Assumption 5**.: \(\sigma\) _is pointwise, \(1\)-Lipschitz, \(\sigma(0)=0\)._

### Propagating Property 1 to layer \(\ell+1\)

We state now how Property 1 propagates throughout the layers using Assumptions 3 to 5 with new parameters \(E_{\ell+1},E_{\ell+1}\). We give a proof in Appendix A.6.

**Lemma 4.1**.: _Let \(\ell\in\{0,\cdots,L-1\}\) and suppose Property 1 to hold at layer \(\ell\) and Assumptions 3 to 5 to hold, then Property 1 still holds at the next layer with \(\tilde{m}_{\ell+1}\) given in Assumption 3 and_

\[E_{\ell+1} =C_{2}E_{\ell}\] \[E_{\ell+1} =2C_{2}E_{\ell}+2C_{1}\tilde{m}_{\ell}E_{\ell}\]

## 5 LMC for Randomly Initialized NNs with sub-Gaussian Distributions

We will make the following assumption on the empirical distribution of neurons weights \(\hat{\mu}_{A,\ell},\hat{\mu}_{B,\ell}\) of \(W^{\ell}_{A},W^{\ell}_{B}\) at layer \(\ell\).

**Assumption 6** (Independence of neurons weights).: \(\hat{\mu}_{A,\ell},\hat{\mu}_{B,\ell}\) _correspond to two i.i.d drawings of vectors with distribution \(\mu_{\ell}\) i.e., \(\hat{\mu}_{A,\ell},\hat{\mu}_{B,\ell}\) have the law of \(\frac{1}{m_{\ell}}\sum_{i=1}^{m_{\ell}}\delta_{x_{i}}\) where \(x_{i}\sim\mu_{\ell}\) i.i.d._

Assumption 6 is verified for example at initialization but more generally when weights do not depend too much one of each other. This case still holds for wide two-layer neural networks trained with SGD and is at the heart of the proof of Theorem 3.1.

### Showing LMC for multilayer MLPs under Gaussian distribution

We first examine the case \(\mu_{\ell}=\mathcal{N}\left(0,\frac{I_{m_{\ell-1}}}{m_{\ell-1}}\right)\). We moreover assume that the input data distribution has bounded second moment: \(\mathbb{E}_{P}[\|x\|_{2}^{2}]\leq m_{0}\).

Our strategy detailed in Appendix A.7 consists in showing that wide enough such networks will satisfy Assumptions 3 and 4 with well controlled constants \(C_{1},C_{2}\). We can then apply Lemma 4.1 successively \(L\) times to get the following lemma:

**Lemma 5.1**.: _Under normal initialization of the weights, given \(\varepsilon>0\), if \(m_{0}\geq 5\), there exists minimal widths \(\tilde{m}_{1},\dots,\tilde{m}_{L}\) such that if \(m_{1}\geq\tilde{m}_{1},\dots,m_{L}\geq\tilde{m}_{L}\), Property 1 is verified at the last hidden layer \(L\) for \(E_{L}=1,E_{L}=\varepsilon^{2}\). More
over, \(\forall\ell\in[L],\exists T_{\ell}\) which does only depend on \(L,\ell\) such that one can define recursively \(\tilde{m}_{\ell}\) as \(\tilde{m}_{0}=m_{0}\) and_

\[\tilde{m}_{\ell}=\tilde{\mathcal{O}}\left(\frac{T_{\ell}}{\varepsilon}\right)^{ \tilde{m}_{\ell-1}}\]

Discussion.The hypothesis \(m_{0}\geq 5\) is technical and could be relaxed at the price of slightly changing the bound on \(\tilde{m}_{1}\). Lemma 5.1 shows that given two random networks whose widths \(m_{\ell}\) is larger than \(\tilde{m}_{\ell}\), we can permute neurons of the second one such that their activations at layer \(\ell\) are both close to the one of the networks on a linear path in parameter's space.

As \(\varepsilon\) goes to \(0\), the width of the layer \(\ell+1\) must scale at least as \(\left(\frac{1}{\varepsilon}\right)^{\tilde{m}_{\ell-1}}\). This is a fundamental bound due to the convergence rate in Wasserstein distance of empirical measures. It imposes a recursive exponential growth in the width needed with respect to depth. This condition appears excessive as compared to the typical width of neural networks used in practice. We highlight here that Ainsworth et al. (2022) empirically demonstrates that networks at initialization do not exhibit LMC and that the loss barrier is erased only after a sufficient number of SGD steps.

### Showing Linear Mode Connectivity

We make the following assumption on the loss function to show LMC from Lemma 5.1.

**Assumption 7**.: \(\forall y\in\mathbb{R}^{m_{L+1}}\)_, the loss \(\mathcal{L}(\cdot,y)\) is convex and \(1\)-Lipschitz._

We finally prove the following bound on the loss of the mean network \(M_{\text{f}}\) in Appendix A.8:

**Theorem 5.2**.: _Under normal initialization of the weights, for \(m_{1}\geq\tilde{m}_{1},\cdots,m_{L}\geq\tilde{m}_{L}\) as defined in Lemma 5.1, \(m_{0}\geq 5\), and under Assumption 7 we know that \(\forall t\in[0,1]\), with \(Q\)-probability at least \(1-\delta_{Q}\), there exists permutations of hidden layers \(1,\ldots,L\) of network \(B\) that are independent of \(t\), such that:_

\[\mathbb{E}_{P}\left[\mathcal{L}\left(\hat{f}_{M_{t}}(x),y\right) \right]\leq t\mathbb{E}_{P}\left[\mathcal{L}\left(\hat{f}_{A}(x),y\right) \right]+\\ (1-t)\mathbb{E}_{P}\left[\mathcal{L}\left(\hat{f}_{B}(x),y\right) \right]+\frac{4\sqrt{m_{L+1}}}{\delta_{Q}^{2}}\varepsilon\]

Discussion.The minimal width at layer \(\ell\) needed for Theorem 5.2 is recursively \(\tilde{m}_{l}\sim\varepsilon^{-\tilde{m}_{l-1}}\). Applied to randomly initialized two-layer networks, we need a hidden layer's dimension of \(\varepsilon^{-m_{0}}\) as opposed to Entezari et al. (2021) which prove a bound of \(\varepsilon^{-(2m_{0}+4)}\).

### Tightness of the bound dependency with respect to the error tolerance

We discuss here the tightness of the minimal width \(\tilde{m}_{\ell}\) we require in Lemma 5.1 with respect to the error tolerance \(\varepsilon\). The recursive exponential growth of the width in the form \(\tilde{m}_{\ell}\sim\left(\frac{1}{\varepsilon}\right)^{\tilde{m}_{\ell-1}}\) is a consequence of the convergence rate of Wasserstein distance of empirical measures in dimension \(\tilde{m}_{\ell-1}\) at the rate \(\nicefrac{{1}}{{\tilde{m}_{\ell-1}}}\). Theorem 5.3 provides a corresponding lower bound which shows that this recursive exponential growth is tight at the precise rate \(\left(\frac{1}{\varepsilon}\right)^{\tilde{m}_{\ell-1}}\) (just take \(n=\tilde{m}_{\ell-1}\), \(m=\tilde{m}_{\ell}\), \(m=\mu_{\ell}\), \(x=\phi_{L}^{\ell-1}(x),W_{A,B}=W_{A,B}^{\ell}\)). A proof is given in Appendix A.11.

**Theorem 5.3**.: _Let \(n\geq 1,x\sim P\in\mathcal{P}_{1}(\mathbb{R}^{n})\) and \(\mu\in\mathcal{P}(\mathbb{R}^{n})\) such that \(\frac{d\mu_{\ell}}{d\mu_{\ell}}\leq F_{1}\). Suppose \(\Sigma=\mathbb{E}[xx^{T}]\) is full rank \(n\). Let \(m\geq 1\) and \(W_{A},W_{B}\in\mathcal{M}_{m,n}(\mathbb{R})\) whose rows are drawn i.i.d. from \(\mu\). Then, there exists \(F_{0}\) such that_

\[\mathbb{E}_{W_{A},W_{B}}\big{[}\min_{\Pi\in\mathcal{S}_{m}}\mathbb{E}_{P}\|(W _{A}-\Pi W_{B})x\|_{2}^{2}\big{]}\geq F_{0}\left(\frac{1}{m}\right)^{2/n}\]

**Remark 1.** Using an effective width \(\tilde{m}_{\ell-1}\) smaller and independent of the real width \(m_{\ell-1}\) allows to show LMC for networks of constant hidden width \(m_{1}=m_{2}=\ldots=m_{L}\) as soon as they verify \(m_{1}\geq\tilde{m}_{1},\ldots,m_{L}\geq\tilde{m}_{L}\) where \(\tilde{m}_{1},\ldots,\tilde{m}_{L}\) are defined in Lemma 5.1. Without this trick, we need a recursive exponential growth of the real width \(m_{\ell}\sim\left(\frac{1}{\varepsilon}\right)^{m_{\ell-1}}\).

**Remark 2**.: Motivated by the fact that feature learning may concentrate the weight distribution on low dimensional sub-space, we could extend our proofs to the case where the underlying weight distribution has a support with smaller dimension to get recursive bounds no longer at rate \(\tilde{m}_{\ell-1}\) but at a smaller one. Note this is unlikely to happen as we expect the matrix of weight vectors of a given layer to be full rank. Therefore, we study in the next section the case when this matrix is approximately low rank, or equivalently when the weight distribution is concentrated around a low dimensional approximated support.

### Approximately low dimensional supported measures

For the sake of clarity, assume from now on that the layer \(\ell-1\) of network \(A\) has been permuted such that for \(\mathcal{I}^{\ell-1}=\{I_{1}^{\ell-1},\ldots,I_{m_{\ell-1}}^{\ell-1}\}\) (given in Property 1) we have \(I_{1}^{\ell-1}=\{1,\ldots,p_{\ell-1}\}\), \(\ldots\), \(I_{\tilde{m}_{\ell-1}}^{\ell-1}=\{m_{\ell-1}-p_{\ell-1}+1,\ldots,m_{\ell-1}\}\) with \(p_{\ell-1}=\nicefrac{{m_{\ell-1}}}{{\tilde{m}_{\ell-1}}}\). This assumption is mild since we can always consider
a permuted version of network \(A\) without changing the problem.

Motivated by the discussion in Appendix A.9.1 we consider the model where the weights at layer \(\ell\) are initialized i.i.d. multivariate Gaussian \(\mu_{\ell}=\mathcal{N}(0,\Sigma^{\ell-1})\) with

\[\Sigma^{\ell-1}:=\mathrm{Diag}\left(\lambda_{1}^{\ell}I_{p_{\ell-1}},\lambda_{ 2}^{\ell}I_{p_{\ell-1}},\ldots,\lambda_{\tilde{m}_{\ell-1}}^{\ell}I_{p_{\ell-1 }}\right)\]

with \(\frac{1}{m_{\ell-1}}\frac{\tilde{m}_{\ell-1}}{k_{\ell-1}}\geq\lambda_{1}^{ \ell}\geq\lambda_{2}^{\ell}\geq\ldots\geq\lambda_{\tilde{m}_{\ell-1}}^{\ell}\) with \(k_{\ell-1}\leq\tilde{m}_{\ell-1}\) an approximate dimension of the support of the underlying weights distribution. Note that to balance the low dimensionality of the weights distribution, we have replaced the upper-bound on the eigenvalues \(\frac{1}{\tilde{m}_{\ell-1}}\) by the greater value \(\frac{1}{m_{\ell-1}}\frac{\tilde{m}_{\ell-1}}{k_{\ell-1}}\) to avoid vanishing activations when \(\ell\) grows which would have made our result vacuous.

The following assumption states that the weights distribution \(\mu_{\ell}^{\mathcal{I}^{\ell-1}}\) at layer \(\ell\) considered in \(\mathcal{P}_{1}(\mathbb{R}^{\tilde{m}_{\ell-1}})\) (with the operation explicited in Section 2) is approximately of dimension \(k_{\ell-1}=e\tilde{m}_{\ell-1}\). The approximation becomes more correct as \(\eta\to 0\).

**Assumption 8** (Approximately low-dimensionality).: \[\exists\eta,e\in(0,1),\forall\ell\in[L],\,\frac{\sqrt{\sum_{j=k_{\ell-1}}^{ \ell}\lambda_{j}^{\ell}}}{4\sqrt{\sum_{j=1}^{k_{\ell-1}}\lambda_{j}^{\ell}}} \leq\eta,\,\frac{k_{\ell-1}}{\tilde{m}_{\ell-1}}=e\]

**Theorem 5.4**.: _Under Assumptions 7 and 8, given \(\varepsilon>0\), if \(em_{0}\geq 5\) there exists minimal widths \(\tilde{m}_{1},\ldots,\tilde{m}_{L}\) such that if \(\eta^{-k_{0}}\geq m_{1}\geq\tilde{m}_{1},\ldots,\eta^{-k_{L-1}}\geq m_{L}\geq \tilde{m}_{L}\), Property 1 is verified at the last hidden layer \(L\) for \(E_{L}=1,E_{L}=\varepsilon^{2}\). Moreover, \(\forall\ell\in[L],\,\exists\mathcal{I}_{\ell}^{\prime}\) which does only depend on \(L,e,\ell\), such that one can define recursively \(\tilde{m}_{\ell}\) as_

\[\tilde{m}_{\ell}=\tilde{\mathcal{O}}\left(\frac{T_{\ell}^{\prime}}{\varepsilon }\right)^{k_{\ell-1}}=\tilde{\mathcal{O}}\left(\frac{T_{\ell}^{\prime}}{ \varepsilon}\right)^{e\tilde{m}_{\ell-1}}\]

_where \(\tilde{m}_{0}=m_{0}\). Moreover there exists permutations of hidden layers \(1,\ldots,L\) of network \(B\) s.t. \(\forall t\in[0,1]\), with \(Q\)-probability at least \(1-\delta_{Q}\):_

\[\mathbb{E}_{P}\left[\mathcal{L}\left(\hat{f}_{M_{t}}(x),y\right) \right]\leq t\mathbb{E}_{P}\left[\mathcal{L}\left(\hat{f}_{A}(x),y\right)\right]\] \[\qquad+(1-t)\mathbb{E}_{P}\left[\mathcal{L}\left(\hat{f}_{B}(x), y\right)\right]+\frac{4\sqrt{m_{L+1}}}{\sqrt{e\delta_{Q}^{2}}}\varepsilon\]

**Discussion.** We give a proof in Appendix A.10. For \(\eta\) small enough, the distribution of weights is approximately lower dimensional. It yields faster convergence rates until \(m\) becomes exponentially big in \(\eta\). This prevents the previous recursive exponential growth of width with respect to depth, though asymptotically, we recover the same rates as in Theorem 5.2. The smaller \(e\), the lower dimensional are the distributions, and the less the width needs to grow when \(\varepsilon\to 0\). The problem in that model is that the constant \(T_{\ell}^{\prime}\) explodes if \(e\to 0\), which prevents using a model with fixed \(k_{\ell}\) across the layers for the weight distribution. We want to highlight here that the proof can be extended to such a case, but we need to assume that the constant \(C_{2}\) is bounded and not depending on \(e\) across the layers in Lemma 4.1 (recall that with our proof, we had \(C_{2}=\frac{1}{e}\)). This assumption seems coherent because the average activations don't explode across layers in the model. Assuming this, the bound we obtain for \(\tilde{m}_{\ell}\) in Theorem 5.4 is completely independent on \(\tilde{m}_{\ell-1}\), and there is no recursive exponential growth in the width needed across the layers. We give a more explicit discussion in Appendix A.12.

### LMC for sub-Gaussian distributions

Still under the setting of Assumption 6 assume that the underlying distribution \(\mu_{\ell}\) verifies for each layer \(\ell\in[L+1]\): if \(X\sim\mu_{\ell}\) then, \(\forall j\neq k\in[m_{l-1}],X_{j}\amalg X_{k}\). Moreover \(\forall i\in[\tilde{m}_{\ell-1}],\forall j,k\in I_{i}^{\ell-1}\),

\[\mathbb{E}[X_{j}^{2}]=\mathbb{E}[X_{k}^{2}]=\lambda_{i}^{\ell}\]

Finally suppose the variables are sub-Gaussian i.e., \(\exists K>0,\forall i\in[\tilde{m}_{\ell-1}],\forall j\in I_{i}^{\ell-1}\), \(\forall c>0\),

\[\mathbb{P}(|X_{j}|\geq c)\leq 2\exp(-\frac{c^{2}}{K\lambda_{i}^{\ell}})\]

We explain in Appendix A.13 why both Theorem 5.2 (in the case \(\lambda_{1}^{\ell}=\ldots=\lambda_{\tilde{m}_{\ell-1}}^{\ell}=\nicefrac{{1}}{{m _{\ell-1}}}\)) and Theorem 5.4 hold with mild modifications in the constants.

It extends our previous result considerably to LMC for any large enough networks whose weights are i.i.d. and whose underlying distribution has a sub-Gaussian tail (for example uniform distribution).

### Link with dropout stability

In Appendix A.14, we build a first step towards unifying our study with the dropout stability viewpoint (Kuditipudi et al., 2019; Shevchenko and Mondelli, 2020) by showing in a simplified setting how networks become dropout stable in the same asymptotics on the width as the one needed in our Theorem 5.2.

## 6 Experimental validation and new method to find permutations

Our previous study shows the influence of the dimension of the underlying weight distribution on LMC effectiveness. Based on this insight we develop a
new weight matching method at the crossroads between previous naive weight matching (WM) and activation matching (AM) methods [11]. Given \(n\) training points \(x_{i},\,i\in[n]\), denote \(Z_{A}^{\ell}\in\mathcal{M}_{m_{\ell},n}(\mathbb{R})\) (respectively \(Z_{B}^{\ell}\)) the activations \(\phi_{A}^{\ell}(x_{i})\) for the \(n\) data points \(x_{i}\). Further denote \(\Sigma_{A}^{\ell}:=\frac{1}{n}Z_{A}^{\ell}[Z_{A}^{\ell}]^{T}\approx\mathbb{E}_ {P}\left[\phi_{A}^{\ell}(x)[\phi_{A}^{\ell}(x)]^{T}\right]\). We aim at finding for each layer \(\ell\) the optimal permutation \(\Pi\) minimizing the cost (respectively for naive WM, our new WM method and AM):

\[\min_{\Pi\in S_{m_{\ell}}}\left\|W_{A}^{\ell}-\Pi W_{B}^{\ell} \Pi_{\ell-1}^{T}\right\|_{2}^{2}\,,\] (Naive WM) \[\min_{\Pi\in S_{m_{\ell}}}\left\|W_{A}^{\ell}-\Pi W_{B}^{\ell} \Pi_{\ell-1}^{T}\right\|_{2,\Sigma_{A}^{\ell-1}}^{2},\] (WM (ours)) \[\min_{\Pi\in S_{m_{\ell}}}\left\|Z_{A}^{\ell}-\Pi Z_{B}^{\ell} \right\|_{2}^{2}\,,\] (AM)

where \(\|\cdot\|_{2,\Sigma_{A}^{\ell-1}}\) is the norm3 induced by the scalar product \((X,Y)\mapsto\text{tr}(X\Sigma_{A}^{\ell-1}Y^{T})\). We both theoretically support the gain of our method in Theorem C.2 and empirically verify that this method constantly and substantially outperforms naive Weight Matching across different learning rates when training with SGD.

Footnote 3: Semi-norm in full generality (if \(\Sigma_{A}^{\ell-1}\) is not full rank)

We train a three hidden layer MLP of width \(512\) on MNIST with learning rates varying between \(10^{-4}\) and \(10^{-1}\) across \(4\) runs. We plot on Figure 1(b) the approximate dimension of the considered covariance matrix for each matching method: \(W_{A}^{\ell}[W_{A}^{\ell}]^{T}\) for WM (naive), \(W_{A}^{\ell}\Sigma_{A}^{\ell-1}[W_{A}^{\ell}]^{T}\) for WM (ours) and \(\Sigma_{A}^{\ell}\) for AM (see SSC.1). Our code is available at [https://github.com/9aze/OT_LMC/tree/main](https://github.com/9aze/OT_LMC/tree/main). We see the detrimental effect of high approximate dimension on LMC effectiveness, therefore validating our theoretical approach. Note that for a learning rate of \(10^{-1}\) the correlation is less clear but a trend is visible on decreasing dimension for naive WM as it performs better (and increasing dimension for AM and our WM method as it performs comparatively less well). An alternative would be to use a proxy taking the diameter of the distributions into account (and not only the dimension of their support). Finally, experiments on Adam lead to less clear results that we did not report as more experimental investigation is needed. In particular, understanding the impact of the optimizer on the independence of weights during training is crucial, as it is a central assumption in our study.

## 7 Discussion

Optimal transport serves as a good framework to study linear mode connectivity of neural networks. This paper uses convergence rates of empirical measures in Wasserstein distance to upper bound the test error of the linear combination of two networks in weight space modulo permutation symmetries. Our main assumption is the independence of all neuron's weight vectors inside a given layer. This assumption is trivially true at initialization but remains valid for wide two-layers networks trained with SGD. We experimentally demonstrate the correlation between the dimension of the underlying weight distribution with LMC effectiveness and design a new weight matching method that significantly outperforms existing ones. A natural direction for future work is to focus on the behaviour of the weights distribution inside each layer of DNNs and their independence. Moreover, extending our results to only assuming approximate independence of weights is a natural direction as it

Figure 2: Statistics of the average network \(M\) over the linear path between networks \(A\) and \(B\) using respectively weight matching (blue), weight matching using covariance of activations and activations (green), and activation matching (red)
seems a more realistic setting.

## 8 Acknowledgements

The work of B. Goujaud and A. Dieuleveut is partially supported by ANR-19-CHIA-0002-01/chaire SCAI, and HiPlaris. This work was partly funded by the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute).
