Neural network training depends on the structure of the underlying loss landscape, i.e. local minima, saddle points, flat plateaus, and loss barriers. In relation to the structure of the landscape, we study the permutation symmetry of neurons in each layer of a deep neural network, which gives rise not only to multiple equivalent global minima of the loss function but also to critical points in between partner minima. In a network of $d-1$ hidden layers with $n_k$ neurons in layers $k = 1, \ldots, d$, we construct continuous paths between equivalent global minima that lead through a `permutation point' where the input and output weight vectors of two neurons in the same hidden layer $k$ collide and interchange. We show that such permutation points are critical points which lie inside high-dimensional subspaces of equal loss, contributing to the global flatness of the landscape. We also find that a permutation point for the exchange of neurons $i$ and $j$ transits into a flat high-dimensional plateau that enables all $n_k!$ permutations of neurons in a given layer $k$ at the same loss value. Moreover, we introduce higher-order permutation points by exploiting the hierarchical structure in the loss landscapes of neural networks, and find that the number of $K$-th order permutation points is much larger than the (already huge) number of equivalent global minima -- at least by a polynomial factor of order $K$. In two tasks, we demonstrate numerically with our path finding method that continuous paths between partner minima exist: first, in a toy network with a single hidden layer on a function approximation task and, second, in a multilayer network on the MNIST task. Our geometric approach yields a lower bound on the number of critical points generated by weight-space symmetries and provides a simple intuitive link between previous theoretical results and numerical observations. The structure of the loss landscape plays an important role in the optimization of neural network parameters. A large number of numerical (Dauphin et al., 2014; Goodfellow et al., 2014; Li et al., 2018; Sagun et al., 2014; 2016; Ballard et al., 2017; Garipov et al., 2018; Draxler et al., 2018; Baity-Jesi et al., 2018) and theoretical (Choromanska et al., 2015; Rasmussen, 2003; Freeman and Bruna, 2016; Soudry and Carmon, 2016; Nguyen and Hein, 2017) studies have explored the properties of the loss landscape. In particular, in a multilayer network of d − 1 hidden layers with n neurons each, there are (n!) d−1 equivalent configurations corresponding to the permutation of neuron indices in each layer of the network (Goodfellow et al., 2016; Bishop, 1995) . The permutation symmetries give rise to a loss landscape where any given global minimum in the weight space must have (n!) d−1 − 1 completely equivalent partner minima. This property of neural network landscapes is called weight-space symmetry. Several (Saad and Solla, 1995; Amari et al., 2006; Wei et al., 2008) works explored the implications of weight-space symmetry for training dynamics in two-layer networks and found that training dynamics slow down near the singular regions caused by weight-space symmetry. Dauphin et al. (2014) ; Orhan and Pitkow (2017) argue that optimization paths may get close to the singular regions induced by weight-space symmetry and this, in turn, slows down training for deep neural networks. Exploiting weight-space symmetries, we give insights into and partial explanations of three observations on neural network landscapes. Observation 1. Training dynamics are slow near singular regions caused by weight-space symmetry and stochastic gradient descent might travel near these regions throughout training (Saad and Solla, 1995; Wei et al., 2008; Amari et al., 2006; Dauphin et al., 2014; Orhan and Pitkow, 2017) . Observation 2. The Hessian of the loss function has numerous almost-zero eigenvalues throughout training, thus the landscape is flat in many directions Papyan, 2018; Ghorbani et al., 2019) . Related to observation 1 and 2, we prove the existence of numerous connected high-dimensional plateaus extending across the landscape due to weight-space symmetries. Observation 3. The number of saddles can grow exponentially in neural network landscapes (Auer et al., 1996; Dauphin et al., 2014; Choromanska et al., 2015) . Related to observation 3, we prove that there are at least polynomially many more saddles than the global minima due to weight-space symmetries in neural networks, without any further assumptions. In addition, we propose a novel low-loss path finding algorithm to find barriers between partner minima. We start from the known permutation symmetries and consider continuous low-loss paths that connect two equivalent global minima by merging the weight vectors of two neurons in a specific way. At a so-called permutation point, where the distance between the input and output weight vectors of the two neurons vanishes, the indices of the two neurons can be interchanged at no extra cost. After the change, the system returns on the 'mirrored' path back to the original configuration -except for the permutation of one pair of indices. Surprisingly, we find that we can permute all neuron indices in the same layer at the same cost as the loss at a permutation point reached by moving along the path that merges a single pair of neurons. These constant-loss permutations are possible because each permutation point lies in a high-dimensional plateau of critical points. Our theory can be extended to higher-order saddles and provides explicit lower bounds for the number of first-and higher-order permutation points. Numerically, we confirm the existence of first-order permutation saddles. In particular, the specific contributions of our work are: • A simple low-loss path-finding algorithm linking partner global minima via a permutation point, implemented by minimization under a single scalar constraint (distance of weight vectors). • The theoretical characterization of permutation points, for example that these are critical points and several permutation points are connected via paths at equal loss. • A lower bound for the number of first-and higher-order permutation points and their corresponding plateaus. • Numerical demonstrations of the path finding method in multilayer neural networks trained on MNIST. Structure of the landscape. For linear networks, it was shown that all the critical points -except for the global minimum -are saddles in the case of two-layer (Baldi and Hornik, 1989) or multilayer networks (Freeman and Bruna, 2016; Kawaguchi, 2016; Lu and Kawaguchi, 2017) . Interestingly, deep linear networks are reported to exhibit sharp transitions at the edges of extended plateaus (Saxe et al., 2013) , similar to the plateaus observed in deep nonlinear networks (Goodfellow et al., 2014) . For nonlinear multilayer networks, Choromanska et al. (2015) argue that all local minima lie below a certain loss value by drawing connections to the spherical spin-glass model. Improving upon this result, Soudry and Carmon (2016) ; Nguyen and Hein (2017) prove that almost all local minima are global minima for multilayer networks under mild over-parametrization assumptions. Bottom of the landscape. Another line of research studies the bottom of the landscape containing global minima and low-loss barriers between them. Freeman and Bruna (2016) prove the existence of low-loss paths connecting global minima for wide two-layer networks by upper-bounding the loss along the path with a parameter that depends on the number of parameters and data smoothness. Draxler et al. (2018) use Nudged Elastic Band method introduced in Jónsson et al. (1998) to connect independent minima and numerically find that the barrier vanishes consistently for increasing width and depth in DenseNet, ConvNet and ResNet architectures trained on CIFAR datasets. In a simultaneous work, Garipov et al. (2018) confirm that there is no significant barrier by connecting independent minima with polygonal chains. Training dynamics in the landscape. For general loss functions, Lee et al. (2016) show that gradient descent with sufficiently small step-size converges to local minima if all the saddles have at least one negative eigenvalue. For overparametrized neural networks, gradient descent converges to global minima without moving far from initialization (Jacot et al., 2018; Du et al., 2018a; , thus suggesting convex-like behavior around random initialization. For the finite size networks, how training dynamics converge to a minima and in particular how fast they converge remain an open question. For soft-committee machines, it turns out that the initial learning dynamics are slowed down by correlation of hidden neurons (Saad and Solla, 1995; Engel and Van den Broeck, 2001; Inoue et al., 2003) . Amari et al. (2006) ; Wei et al. (2008) show that training dynamics slow down near singular regions due to weight-space symmetry. Dauphin et al. (2014) empirically argue that the large number of saddle points in the landscape makes training slow. Orhan and Pitkow (2017) numerically find that stochastic gradient descent may slow down near plateaus due to weight-space symmetry for deep (30 layers) feedforward networks trained on CIFAR100. In this paper we show that there is an impressively large number of permutation points. Each permutation point is a critical point (either a local minimum or a saddle) with a large number of flat directions, potentially linked to the empirically observed plateaus. In contrast to an earlier study by Fukumizu and Amari (2000) with a scalar output for two-layered networks where a line of critical points around the permutation point was reported, we study a deep network with d − 1 hidden layers and find multi-dimensional equal-loss plateaus. Moreover, we give a novel lower bound on the number of permutation points and construct sample paths between global minima using an algorithm that is different from previously used methods (Garipov et al., 2018; Draxler et al., 2018) , since it exploits the symmetries at the permutation point. We study multilayer neural networks f (x; θ) with input x ∈ R n0 , d layers of n 1 , . . . , n d neurons where g is a nonlinear activation function that operates component-wise on any vector. m of neuron m in layer k as the incoming weights to a neuron m in layer k concatenated with its bias term: ϑ m . (Output weight vector) We define the output weight vector of neuron m in layer k as its outgoing weights from neuron m in layer k to the next layer: Since one can permute the neurons within each layer without changing the network function f (x; θ), any point θ induces a 'permutation set'. Definition 3. (Permutation set) of points θ with f (x; θ) = f (x; θ ), where σ (k) are permutations of the neuron indices {1, . . . , n k } in (hidden) layer k where σ (0) and σ (d) are fixed trivial permutations, since we want to permute neither the indices of the input nor that of the output. We will use the notation θ = σ (k) l⇔m (θ) to indicate a point θ that differs from θ only by swapping neurons l and m in layer k. Note that the cardinality of a permutation set is maximal with |P (θ)| = m are distinct for every l = m and layer k ∈ {1, . . . , d − 1}. In the following, we will assume that, at global minima, all parameter vectors are distinct at every layer k. Definition 4. (Permutation point) Consider a minimum of a multilayer network with (n 1 , . . . , n k − 1, . . . , n d ) neurons per layer. We can map this minimum to a configuration in the landscape of a multilayer network with (n 1 , . . . , n k , . . . , n d ) neurons per layer by duplicating one neuron m ∈ {1, . . . , n k − 1} in layer k as follows: (i) substitute the parameter vector of the new neuron with a copy of the parameter vector ϑ , where : Y × R n d → R is some single-sample loss function. To simplify notation we will usually omit the explicit mentioning of the data in the loss function, i.e. In this section, we will first present a novel method to find a low-loss path between partner minima. Our method ensures that this path passes through a 'permutation point'. We study the properties of permutation points. Furthermore, we will introduce higher-order permutation points and provide a lower-bound on their number. One natural question regarding the geometry of the bottom of the landscape is the following: is it possible to find a continous low-loss path that connects two minima? In this work, we are interested in finding the barriers between partner global minima. In particular, we want to find a continuous low-loss path γ : [0, 1] → Θ connecting two partner minima by first merging two parameter vectors and output weight vectors ('permutation point') and then completing the path using symmetry. We will first introduce some concepts to introduce this low-loss path between partner minima formally. + is a distance function that takes a configuration θ and returns the squared Euclidean distance between the parameter vectors of neuron l and m at layer k: ) under the following constraints for the initial (t = 0) and quarter-way (t = 1 4 ) configurations: γ * (0) = θ, where θ is the parameter configuration at the minimum, and d 1 (see Fig. 1 and pseudocode in Appendix). The quarter-way configuration guarantees that the parameter vectors are identical ϑ m , but puts no constraints on the output weights of neurons m and l in layer k. We can continously move from the quarter-way configuration (t = 1 4 ) to a configuration at t = 1 2 where the outputs weights of the related neurons are equal, without making any changes to the network output or the loss L as follows: we will increase all output weights W k+1 n,l of neuron l and decrease the corresponding output weights W k+1 n,m of neuron m by the same amount continuously so as to keep their sum fixed until W k+1 n,l = W k+1 n,m for every neuron n ∈ {1, . . . , n k+1 } at layer k + 1 (see Appendix Fig. 4 ). Lemma 1. The configuration at t = 1 2 is one of the permutation points. Once we have reached t = 1 2 , we interchange the neuron indices of the 'merged' neurons and continue on the 'mirror' path that results from walking the first half of the path backwards with interchanged neuron indices, until we arrive at the partner minimum at t = 1, i.e. γ( To find such paths algorithmically, we reparametrize ϑ is a positive scalar and e(t) is a unit-length vector. We start with at the global minimum i.e., the initial parameter configuration. Next, we decrease d infinitesimally and perform gradient descent for fixed d on the loss L until convergence. Note that all parameters can change, including ϑ (k) l and e during gradient descent. This procedure is repeated until d = 0 at t = 1 4 . Finally we shift the respective output weights to the same value without changing the network function (see Appendix Fig. 4 ). Since the path connects two partner minima, there must be at least one saddle point on the path γ(t), t ∈ [0, 1], potentially but not necessarily, at the permutation point. Moreover, there is no guarantee that the highest saddle should be located at the permutation point (see Appendix Fig. 5 ). In an earlier work, Fukumizu and Amari (2000) studied a specific set of critical points induced by the hierarchical structure in the neural network landscapes in two-layer neural networks. Let L (H) be the loss function ('landscape') of a two-layer neural network with H neurons in the hidden layer and a single output. They showed that any critical point in the landscape of L (H−1) induces a line of critical points in the landscape of L (H) . We study permutation points in the general setup for the neural networks with multiple outputs and multiple layers. l⇔m are critical points of the original loss function. (ii) Any permutation point lies inside a n k+1 -dimensional equal-loss subspace of critical points. (iii) All other permutations of neuron indices in layer k can be performed by continuous equal-loss transformations starting from permutation points θ (k) l⇔m of neurons l and m, i.e. there is a continuous 1 Note that the parametrization t including the exact t values for the mentioned configurations are arbitrary and only used for conceptualizing the path. The relevant parametrization of the path will be the distance d. 2 Theorem 1 is easily verified by setting the derivatives of all parameters to zero in L (H−1) (critical point) and observing that the derivatives of the corresponding configuration in L (H) will be zero under the mapping stated in the theorem. (1) i of the first layer at the global minimum (blue) and at a permutation point (red) reached after merging the parameter vectors of two neurons. Note that the global minimum (i.e., the starting configuration) is the same for (A), (B) and (C) but the loss at the permutation point can be the same -(A) and (B) -or different -(A) and (C) -depending on the pair of neurons chosen for merging. Numbers indicate neurons. Bottom row. Quadratic loss L as a function of the distance d between the neurons to be merged. The distance was decreased in 200 logarithmically spaced steps l,m (θ * ) to 1/10 4 of the initial value. For each d, full batch gradient descent on the loss L was performed until convergence. Training data was generated by sampling 10 3 two-dimensional input points x µ from a standard normal distribution and computing labels y µ = f (x µ ; θ * ) using a teacher network (shown in blue, equivalent to the configuration at the global minima). The teacher had a single layer of five hidden neurons with rectified-linear activation function g and one linear output layer. Therefore, each of the 2 different permutation points of layer k corresponds to a plateau of n k+1 dimensions. This plateau enables the exchange of all indices in layer k. Note that there can be multiple plateaus on different loss levels that correspond to different local minima of the smaller networks where one of the neurons is dropped at a permutation point. For example in Fig. 2 one can exchange all indices in the hidden layer through the configuration in A and B or through the configuration in C that has another loss level. Note that, amongst all these permutation points embedded in different plateaus, we could for example search for the one with the lowest cost -and this lowest-cost permutation would then also connect all global minima caused by arbitrary permutations of neurons in layer k. Definition 6. (Higher-order permutation point) Consider a minimum of a multilayer network with (n 1 , . . . , n k − K, . . . , n d ) neurons per layer. We can map this minimum to a configuration in the landscape of a multilayer network with (n 1 , . . . , n k , . . . , n d ) neurons per layer by replicating some neurons m j ∈ {1, . . . , n k − K} in layer k to fill out the parameters of new neurons as follows: (i) substitute the parameter vectors of the new neurons with one of the parameter vectors of the initial minimum, (ii) replace the output weight vectors of the new neurons and the output weight of the corresponding parameter vector m j with the initial output weight of the replicated neuron m j normalized so that the mentioned output weight vectors sum up to the original output weight vectors, and (iii) keep all the other parameters the same. This new configuration where the parameter vectors and the output weight vectors of the new neurons and the replicated neuron m j for several j in layer k are the same will be called a K-th order permutation point. This natural generalization on the 1-st order permutation points to higher-orders enables generalizing Proposition 1(i) and (ii) to the K-th order permutation points. with the procedure described in Definition 6. We can then count the number of permutation points that reduce to the same configuration in L (H−K) combinatorially (see Appendix Fig. 6 for the explanation of combinatorial counting). For counting, we consider the cardinality of the permutation set of a permutation point. Since some parameter vectors are replicated, we have to consider permutations of sometimes identical neurons. This enables finding a lower bound on the number of critical points that have higher loss values than the global minima in general. Proposition 3. In a neural network with (n 1 , . . . , n d ) neurons per layer, let T (K, n k ) denote the ratio of the number of K th -order permutations points at layer k to the number of global minima for k = 1, . . . , d − 1 and K ≥ 1. (i) For K = 1, 2, 3 and n k ≥ 2K, we find T (K, n k ) to be: (Proofs: see appendix) Considering all the layers, we note that the number of permutation points of order K is at least times more than the global minima for 2K ≤ min k n k . When one layer has large number of neurons (i.e. n k → ∞) then the ratio T (K, n k ) grows with n K k . Every permutation point lies inside a high-dimensional subspace of equal loss (Proposition 2(ii), see Appendix Fig. 7 for illustration) . Importantly, every permutation point lies inside a distinct but connected subspace. Therefore the count for permutation points holds for the corresponding high-dimensional equal-loss subspaces of critical points. Lemma 3. In a neural network with (n 1 , . . . , n d ) neurons per layer, there are (at least) k=1 n k ! many Kn k+1 -dimensional equal-loss subspaces of critical points at the loss of a K-th order permutation point for 2K ≤ min k n k . We could start at an arbitrary configuration in consider the landscape of L (H−K) and the corresponding equal-loss high-dimensional subspaces in L (H) , where each configuration in the subspace computes the same function as the initial configuration. This procedure again would yield the same number of high-dimensional equal-loss subspaces. Therefore due to weight-space symmetry, neural network landscapes do not only exhibit numerous high-dimensional plateaus of critical points but also numerous high-dimensional plateaus (of usually non-critical points) at various loss values. Using a similar procedure as in the toy example (see Fig. 2 ), we constructed paths between global minima in a fully connected three-layer network with n 1 = n 2 = H and n 3 = 10 neurons (see Fig. 3 ). In order to study global minima we used a student-teacher setting 3 : the teacher network was pre-trained on the MNIST data set using negative log-likelihood loss and its parameters θ * were kept fixed thereafter. We initialized the student with the parameters θ * of the teacher and decreased the distance d between the parameter vectors of two selected neurons m and l in layer k = 2 in 100 logarithmically spaced steps from d (2) m,l (θ * ) to 1/10 4 of the original value. For every value of d, the student was trained on a regression task with a mean-squared error loss L between teacher and student output using full batch gradient descent until convergence. With y µ = f (x µ ; θ * ) ∈ R 10 being the output of the last layer before the softmax operation, we chose L = as the mean squared error loss between teacher and student, where . denotes the mean over patterns and dimensions and µ = 1, . . . , T enumerates the samples of the data set. Apart from a few cases, where the trajectory towards the permutation point passed through a saddle on the way, in most cases the loss increased monotonically until the permutation point. This indicates that the permutation point is a saddle, and not a minimum. As expected from theoretical results (Freeman and Bruna, 2016) and empirically observed by (Draxler et al., 2018) , the barrier height (loss at saddle) decreased with the number H of hidden neurons per layer. A low-loss permutation path in the loss landscape of a multi-layer network using a student-teacher setup trained on MNIST. We merged the parameter vectors of two neurons with high cosine-similarity in the second hidden layer of a three-layer student network. The corresponding teacher network with H = 10, 15, 20 or 25 was trained on MNIST. For each hidden layer size we trained 6 teacher networks with different random seeds and display one curve per hidden layer size and seed. A. In most cases, the mean squared loss L between teacher and student output increases monotonically along our constructed paths from a global minimum until the permutation point. In these cases the latter corresponds to the loss barrier along the path. Note that the barrier height (loss at saddle) decreases with H. B. The MNIST classification accuracy on the training set decreases only marginally when moving to a permutation point. The surprising training performance of neural networks despite their highly non-convex nature has been drawing attention to the structure of the loss landscape. In this paper, we explored how weight-space symmetry induces saddles and plateaus in the neural network loss landscape. We found that special critical points, so-called permutation points, are embedded in high-dimensional flat plateaus. We proved that all permutation points in a given layer are connected with equal-loss paths, suggesting new perspectives on loss landscape topology. We provided a novel lower bound for the number of first-and higher-order permutation points and proposed a low-loss path finding method to connect equivalent minima. The empirical validation of our path finding algorithm in a multilayer network trained on MNIST showed that permutation points could indeed be reached in practice. Additionally, we observed that the loss at the permutation point (barrier) decreased with network size and thus confirmed Freeman and Bruna (2016) (1, 0.5, 0.5) Figure 7: A. Zooming in permutations points of one of the permutation sets (red in Fig. 6) . B. Visualizing how permutation points (at layer k = 1, see A) lie inside equal-loss lines in the weight space of the layer k + 1 = 2. Only two out of three lines are shown for simplicity. We observe that the number of such equal-loss lines (hyperplanes) is equal to the number of permutation points, i.e. each permutation point lies inside one distinct line. Algorithm 1 Finding Low-Loss Paths through Permutation Points Require: parameters θ, indices k, l, m, SCHEDULE of decreasing distances Proof of Lemma 1. Since the parameter vectors and output vectors of the two neurons are identical at t = 1 2 , we can merge the two neurons into a single one (by appropriately rescaling the corresponding output weights) so that the number of neurons n k in layer k is reduced by one. Since our path-finding method minimizes the gradient, the configuration at t = 1 2 is a local minimum of the smaller network, hence a permutation point. Alternatively, we will reformalize the path-finding problem with Lagrange multipliers: minimize l,m (θ) for increasing values of λ, starting with λ = 0 and increasing (potentially) until λ → ∞. The goal here is to decrease the distance down to zero while keeping the loss minimal. For every λ, we will obtain a minimizer θ. This sequence of θ configurations will be an approximate discretization of the ideal path γ * . With the Lagrangian formulation, we can easily show that the half-way configuration is a critical point: it is a minimizer of L(θ) + λd l ) and both are equal to zero at the half way configuration since ϑ l,m (θ) = 0. Consequently, ∇L(θ) = 0. We note that the Lagrangian formulation is an approximation on the ideal path γ * . Proof of Proposition 1. (i) Let us assume we merged the neurons l and m in layer k and scaled the output weight vectors of these neurons with 1 2 . In that case, the output vector in layer k + 1 remains the same. The derivative with respect to an output in the layer k + 1 will be the same, i.e. All the other derivatives remain the same (up to a scalar constant) as the corresponding derivatives before the mapping is performed. Therefore, a critical point in the landscape of the neural network with (n 1 , . . . , n k − 1, . . . , n d ) neurons per layer will map to another critical point in the landscape of the neural network with (n 1 , . . . , n k , . . . , n d ) neurons per layer under the function preserving mapping described in the definition of a permutation point. This proposition is a straightforward extension to the Theorem 1 in (Fukumizu and Amari, 2000) for multiple output and multiple-layer neural networks. (ii) Once the parameter vectors of neurons m and l in layer k are identical, they implement the same function. Any change of an output weight W = c for each n in layer k + 1 defines an n k+1 -dimensional hyperplane of critical points. In particular, at each point in the hyperplane, we have n k+1 directions of the Hessian with zero Eigenvalues if the activation function g is twice-differentiable. (iii) We make the following sequence of continuous transformations that are all possible at fixed loss. First, we decrease the output weights of neuron m to zero while increasing those of l by the same amount, keeping the sum of weights W constant for each j in layer k+1. Second, we change smoothly the input parameter vector of neuron m to match those of an arbitrary other neuron i in the same layer k. Third, we increase the output weights of neuron m while decreasing those of neuron i until all output weights of neuron i are zero, keeping the sum of weights W constant for each j in layer k + 1. Fourth, we reduce the input parameter vector of neuron i to zero. Fifth, we increase the input parameter vector of neuron i to match that of neuron l at the permutation point. Finally, we equally share output weights between neurons i and l so that i has the same weights as previously neuron m at the permutation point. Effectively, this procedure enables us to exchange an arbitrary neuron i with neuron m, but the procedure can be repeated for further permutations. The permutations constructed in the proof of property (ii) start at permutation points where the parameter vectors of neurons l and m merge. Therefore the loss associated with all the permutations constructed in the proof is L(θ (k) l⇔m ), the one of this permutation point. However, we could also begin with two other weight vectors i and j and construct the path leading to another θ (k) i⇔j , that has, in general, a different loss. Proof of Proposition 2. (i) We take a minimum in the landscape of neural network with (n 1 , . . . , n k − K, . . . , n d ) neurons per layer. Using Proposition 1-(i), we map to a critical point in the landscape of neural network with (n 1 , . . . , n k − K + 1, . . . , n d ) neurons per layer where one parameter vector in layer k is duplicated and the corresponding output weights are rescaled with 1 2 . Repeating the same mapping starting at the latter critical point, we map to another critical point in the landscape of neural network with (n 1 , . . . , n k − K + 2, . . . , n d ) neurons per layer. Repeating this mapping for K times, we end up a K-th order permutation point (up to changes in the output weights of the replicated neurons as long as the network function is preserved) and this is a critical point in the landscape of neural network with (n 1 , . . . , n k , . . . , n d ) neurons per layer by induction. (ii) We will denote the parameter vectors of a neural network with (n 1 , . . . , l = n k − K, . . . , n d ) neurons per layer with {ϑ Let's consider an unordered partition of n k = s 1 + s 2 + . . . + s l with s m ≥ 1 for m = 1, . . . , l. Without loss of generality, let's assume that the first s 1 parameter vectors of layer k are the same, then the next s 2 and so on. Equivalently, for all m = 1, . . . , l, we have ϑ The only free variables are the outgoing weights of the replicated neurons-except for the constraint that the summation of these weights is fixed. These constraints correspond to for all neurons i at layer k+1. Overall, there are ( l m=1 s m )n k+1 = n k n k+1 free variables constrained by ln k+1 equations. One equation defines a n k n k+1 − 1 dimensional hyperplane in the n k n k+1 space. Intersecting ln k+1 of these hyperplanes, we end up having a (n k − l)n k+1 = Kn k+1 dimensional equal-loss hyperplane. order permutation points (at layer k) corresponding to this unordered partition. Overall, we have j=1 n j ! many 2-nd order permutation points at layer k. (3) The case K = 3: There are three ways to have n k − 3 distinct vectors out of n k , corresponding to three unordered partitions of n k : (i) n k = 4+1+. . .+1, (ii) n k = 3+2+1+. . .+1, and (iii) n k = 2+2+2+1+. . .+1. (i) n k = 4 + 1 + . . . + 1 For this case, we have n k ! 4! permutations given by permuting the neuron indices of layer k. Therefore, this 3-rd order permutation point induces a permutation set with cardinality |P (θ)| = 1 4! d−1 j=1 n j !. As usual, we should consider other 3-rd order permutation points (at layer k) giving rise to the same network function and corresponding to the same unordered partition. If we had chosen another parameter vector to replicate four times, this 3-rd order permutation point would induce another permutation set. Note that we can choose the parameter vector to replicate out of n k − 3 in j=1 n j ! many 3-rd order permutation points (at layer k) corresponding to this unordered partition. (ii) n k = 3 + 2 + 1 + . . . + 1 For this case, we have inating term for large n k , since every other summand would be a polynomial of at most (K1)-th order. (5) A lower bound for general K: For general K, we have l = n k − K distinct parameter vectors in the small network. There are many ways to partition n k into l positive integers without respecting order. Since we are interested in a lower bound, we only consider the following unordered partition: n k = 2 + . . . + 2 + 1 + . . . + 1, i.e. we have K duplicated parameter vectors and n k − 2K parameter vectors that appear once. For this unordered partition, we have n k −K K ways to choose the duplicated parameter vectors. For each one of these choices, we can permute the neuron indices in n k ! 2 K different ways. Including the permutations in other layers j = k, we end up with j=1 n j ! points in the permutation set. The number is a lower bound of T (K, n k ), because other unordered partitions of n k give rise to other K th -order permutation points at layer k. C.5 PROOF OF LEMMA 2 We can approximate the factorial an integer n using Stirling's formula n! → √ 2πn n e n as n → ∞ We note that this approximation leads to accurate results even for small n. As n → ∞, both n − K → ∞ and n − 2K → ∞ for finite K. Therefore, we can apply Stirling's formula both for n − K and n − 2K: C.6 PROOF OF LEMMA 3 We already know the number of equilavent K-th order permutation points (the ones that reduce to the same configuration in the landscape of the neural network with (n 1 , . . . , n k − K, . . . , n d )) is at least k=1 n k ! (Proposition 3-(ii)). We can easily that see that every permutation point gives rise to a distinct high-dimensional subspace of critical points by observing that their parameter vectors in layer k would be in distinct positions when projected on the parameters of the layer k since we never repeat the same set of parameters in layer k. If two such subspaces were the same, their projection on a lower dimensional space (parameters of the layer k) would be same necessarily. Therefore, the subspaces mentioned are distinct and the number of them is equivalent to the number of related permutation po
Weight-space symmetry in neural network landscapes gives rise to numerous number of saddles and flat high-dimensional subspaces. The paper presented a low-loss method for studying the loss function with respect to parameters in a neural network from the perspective of weight-space symmetry.