# Symmetry Leads to Structured Constraint of Learning

Liu Ziyin
Department of Physics, The University of Tokyo

###### Abstract

Due to common architecture designs, symmetries exist extensively in
contemporary neural networks. In this work, we unveil the importance of the
loss function symmetries in affecting, if not deciding, the learning behavior
of machine learning models. We prove that every mirror symmetry of the loss
function leads to a structured constraint, which becomes a favored solution
when either the weight decay or gradient noise is large. As direct
corollaries, we show that rescaling symmetry leads to sparsity, rotation
symmetry leads to low rankness, and permutation symmetry leads to homogeneous
ensembling. Then, we show that the theoretical framework can explain the loss
of plasticity and various collapse phenomena in neural networks and suggest
how symmetries can be used to design algorithms to enforce hard constraints in
a differentiable way.

##  1 Introduction

Modern neural networks are so large that they often contain an astronomical
number of neurons and connections layered in a highly structured manner. This
design of modern architectures and loss functions means that there are a lot
of redundant parameters in the model and that the loss functions are often
invariant to hidden, nonlinear, and nonperturbative transformations of the
model parameters. We call these invariant transformations the “symmetries” of
the loss function. In physics, symmetries are regarded as fundamental
organizing principles of nature, and systems with symmetries exhibit rich and
hierarchical behaviors (Anderson, 1972). However, the role of symmetries in
affecting the learning of neural networks remains largely a mystery.

Since we will also discuss stochastic aspects of learning, we focus on
studying a generic twice-differentiable non-negative per-sample loss function:

$\ell_{\gamma}=\ell_{0}(\theta,x)+\gamma||\theta||^{2},$ (1)

where $x$ is a minibatch or a single data point of arbitrary dimension and
sampled from a training set. $\theta$ is the model parameter, and $\gamma$ is
the weight decay. $\ell_{0}$ assumes the definition of the model architecture
and is the data-dependent part of the loss. Training with stochastic gradient
descent (SGD), we sample a set of $x$ and compute the gradient of the averaged
per-sample loss over the set. The per-sample loss averaged over the training
set is the empirical risk:
$L_{\gamma}(\theta):=\mathbb{E}_{x}[\ell_{\gamma}]$. Training with gradient
descent (GD), we compute the gradient with respect to $L_{\gamma}$. All the
results we derive for $\ell_{\gamma}$ directly carry over to $L_{\gamma}$.

We first study the effect of three specific types of symmetry one often
encounters in deep learning: (1) rescaling symmetry, (2) rotation symmetry,
and (3) permutation symmetry. See Figure 1 for an illustration. We then
identify a general class of symmetry, the mirror reflection symmetry, that
treats all three types of symmetry in a coherent framework and proves a
general theorem showing that every mirror symmetry leads to a structured
constraint. Section 4 discusses the related works in detail and the
connections of our results to them. All the proofs are given in Appendix B.

Figure 1: When loss function symmetries are present, the model converges to
structurally constrained solutions at a high weight decay or gradient noise.
Left: A vanilla linear regression trained with SGD does not converge to sparse
solutions for any learning rate. When we introduce redundant rescaling
symmetry to every parameter, sparser solutions are favored at higher learning
rates ($\lambda$). Mid: Vanilla $200$ dimensional matrix factorization trained
with SGD prefers lower-rank solutions when the gradient noise is strong due to
the rotation symmetry. The inset shows that the model always stays full-rank
if we remove the rotation symmetry by introducing residual connections. Right:
Correlation of the pre-activation value of neurons in the penultimate layer of
ResNet18. After training, the neurons are grouped into homogeneous blocks when
weight decay is present. The inset shows that such block structures are rare
when there is no weight decay. Also, the patterns are similar for post-
activation values, which further supports the claim that the block structures
are due to the symmetry, not because of linearity. See Appendix A for the
experimental details and more results.

##  2 Consequences of Common Symmetries

While all the theorems in this section can be proved as corollaries of the
general theorem 4, we give independent proofs of them to bring some
concreteness to the general theorem.

###  2.1 Rescaling Symmetry Leads to Sparsity

The simplest type of symmetry in deep learning is the rescaling symmetry (Dinh
et al., 2017; Saxe et al., 2013; Neyshabur et al., 2014; Tibshirani, 2021).
Consider a loss function $\ell_{0}$ for which the following equality holds for
any $x$, arbitrary vectors $u,\ w$ and $\rho\in\mathbb{R}_{/\\{0\\}}$:

$\ell_{0}(u,w,x)=\ell_{0}(\rho u,\rho^{-1}w,x).$ (2)

For the rescaling symmetry and for all the problems we discuss below, it is
also possible for $\ell_{0}$ to contain other parameters $v$ that are
irrelevant to the symmetry: $\ell_{0}=\ell_{0}(u,w,v)$. Since having such $v$
or not does not change our result, we only show $v$ explicitly when necessary.
Also, because the symmetries we consider in this work hold for any $x$, we
also omit writing $x$ unless necessary.

The following theorem states that this symmetry leads to sparsity in the
parameters.

######  Theorem 1.

Let $\ell_{0}(u,w)$ have the rescaling symmetry in Eq. (2). Then, for any $x$,

  1. 1.

if $u=0$ and $w=0$, then $\nabla_{u}\ell_{\gamma}=0$ and
$\nabla_{w}\ell_{\gamma}=0$;

  2. 2.

for any fixed $u$, $w$, there exists $\gamma_{0}$ such that for all
$\gamma>\gamma_{0}$, $\ell_{\gamma}(0,0)<\ell_{\gamma}(u,w)$.

Two parts of the theorem statement convey different insights. Part 1 shows
that the learning dynamics are constrained – namely, GD or SGD will not leave
the condition $(u,w)=(0,0)$ once entered. Part 2 shows that such constrained
solutions can be locally favored for a large regularization. Additionally,
symmetry has strong implications on the structures of the Hessian of the loss
function and global properties of the loss landscapes; we delay its
presentation and discussion after Theorem 4.

This symmetry usually manifests itself in ReLU networks or when part of the
parameters is linearly connected. Previous works have used this property to
either understand the inductive bias of neural networks or design efficient
training algorithms. When the model is a fully connected ReLU network,
Neyshabur et al. (2014) showed that having $L_{2}$ is equivalent to $L_{1}$
constraints of weights. Ziyin & Wang (2023) designed an algorithm to compress
neural networks by transforming a parameter vector $v$ to $u\odot w$, where
$\odot$ is the Hadamard product.

###  2.2 Rotation Symmetry Leads to Low-Rankness

A more involved but common type of symmetry is the rotation symmetry, which
also appears in a few slightly different forms in deep learning. This type of
symmetry appears in matrix factorization problems, where it is a main cause of
the emergence of saddle points (Li et al., 2019). It also appears in Bayesian
deep learning (Tipping & Bishop, 1999; Kingma & Welling, 2013; Lucas et al.,
2019; Wang & Ziyin, 2022), self-supervised learning (Chen et al., 2020; Ziyin
et al., 2023b), and transformers in the form of key-query matrices (Vaswani et
al., 2017; Dong et al., 2021).

Now, we show that rotation symmetry in the landscape leads to low rankness. We
use the word “rotation” in a broad sense, including all orthogonal
transformations. There are two types of rotation symmetry common in deep
learning. In the first kind, we have for any $W$,

$\ell_{0}(W)=\ell_{0}(\Omega W)$ (3)

for any orthogonal matrix $\Omega$ such that $\Omega\Omega^{T}=I$ and $W$ is a
set of weights viewed as a matrix or vector whose left dimension matches the
right dimension of $R$.

######  Theorem 2.

Let $\ell_{0}$ satisfy the rotation symmetry in Eq. (3). Then, for any index
$i$, vector $n$ and $x$,

  1. 1.

if $n^{T}W=0$, then $n^{T}\nabla_{W}\ell_{\gamma}=0$;

  2. 2.

for any fixed $W$, there exists $\gamma_{0}$ such that for all
$\gamma>\gamma_{0}$, $\ell_{\gamma}(W_{/i})<\ell_{\gamma}(W)$;111The notation
$W_{/i}$ denotes the matrix obtained by setting the $i$-th singular value of
$W$ to be zero.

Part 1 of the statement deserves a closer look. $n^{T}W=0$ implies that $W$ is
low-rank and $n$ is a left eigenvector of $W$. That the gradient vanishes in
this direction means that once the weight matrix becomes low-rank, it will
always be low-rank for the rest of the training.

A more common symmetry is a simultaneous rotation symmetry, where $\ell_{0}$
depends on two matrices $U$ and $W$ and satisfies
$\ell_{0}(U,W)=\ell_{0}(UR,R^{T}W)$, for any orthogonal matrix $R$ and any $U$
and $W$. Namely, the loss function is invariant if we simultaneously rotate
two different matrices with the same rotation. In this case, one can show
something similar: $n^{T}W=0$ and $Un=0$ for some fixed direction $n$ is the
favored solution.222The theorem does not imply that $Un=0$ and $n^{T}W=0$ must
happen simultaneously. To see this, consider any loss function that only
depends on the norm of $U$ and $W$.

###  2.3 Permutation Symmetry Leads to Homogeneous Ensembling

The most common type of symmetry in deep learning is permutation symmetry. It
shows up in virtually all architectures in deep learning. A primary and well-
studied example is that in a fully connected network, the training objective
is invariant to any pairwise exchange of two neurons in the same hidden layer.
We refer to this case as the “special permutation symmetry” because it is a
special case of the permutation symmetry we study here. Many recent works are
devoted to understanding the special permutation symmetry (Simsek et al.,
2021; Entezari et al., 2021; Hou et al., 2019). Notably, Entezari et al.
(2021) empirically showed that neural networks under SGD likely converge to
the same type of solution if we take permutation symmetry into consideration.

Here, we study a more general and abstract type of permutation symmetry. The
loss function has a permutation symmetry between parameter subsets
$\theta_{a}$ and $\theta_{a}$ if, for any $\theta_{a}$ and $\theta_{b}$,333As
an example, consider a hidden layer of a network; let $w_{a}$ and $u_{a}$ be
the input and output weights of neuron $a$, and $w_{b}$, $u_{b}$ be the input
and output weights of neuron $b$. We can thus let $\theta_{a}:=(w_{a},u_{a})$
and $\theta_{b}:=(w_{b},u_{b})$.

$\ell_{0}(\theta_{a},\theta_{b})=\ell_{0}(\theta_{b},\theta_{a}).$ (4)

When there are multiple pairs that satisfy this symmetry, one can combine this
pairwise symmetry to generate arbitrary permutations. In this perspective,
permutation symmetries appear far more common than is recognized. For example,
another example is that a convolutional neural network is invariant to a
pairwise exchange of two filters, which is rarely studied. A scalar rescaling
symmetry can also be regarded as a special case of permutation symmetry.

Here, we show that the permutation symmetry tends to make the neurons become
identical copies of each other (namely, encouraging $\theta_{a}$ to be as
close to $\theta_{b}$ as possible).

######  Theorem 3.

Let $\ell_{0}$ satisfy the permutation symmetry in Eq. (4). Then, for any $x$,

  1. 1.

if $\theta_{a}-\theta_{b}=0$, then
$\nabla_{\theta_{a}}\ell_{\gamma}=\nabla_{\theta_{b}}\ell_{\gamma}$;

  2. 2.

for any $\theta_{a}\neq\theta_{b}$, there exists $\gamma_{0}$ such that for
all $\gamma>\gamma_{0}$,
$\ell_{\gamma}((\theta_{a}+\theta_{b})/2,(\theta_{a}+\theta_{b})/2)<\ell_{\gamma}(\theta_{b},\theta_{a})$;

This theorem implies that a permutation symmetry can be seen as a generalized
form of ensembling smaller submodels.444One might suspect the origin is always
favored when a mirror symmetry exists: this is not true. Let us consider a
simple reparametrized linear regression problem:
$L_{\gamma}(w_{1},w_{2})=[(w_{1}+w_{2})x-y]^{2}+\gamma(w_{1}^{2}+w_{2}^{2})$.
A permutation symmetry exists between $w_{1}$ and $w_{2}$. The condition
$\theta_{a}-\theta_{b}=0$ is satisfied for all solutions of the loss whenever
$\gamma>0$, however small $\gamma$ is. Meanwhile, for a finite $\gamma$, no
solution satisfies $\theta_{a}=\theta_{b}=0$. Therefore, achieving such
solutions does not imply we have reached a trivial solution. Special cases of
this result have been proved previously. For a fully connected network,
Fukumizu & Amari (2000) showed that the solutions of subnetworks are also
solutions of the larger network, and Chen et al. (2023) demonstrated that
these subnetwork solutions of fully connected networks can be attractive when
the learning rate is large. Our result is more general because it does not
restrict to the special permutation symmetry induced by fully connected
networks. A novel application is that the networks have block-wise neurons and
activation patterns whenever weight decay is present. See Figure 1.

##  3 Every Mirror Symmetry Leads to a Structured Constraint

A remarkable aspect of Theorems 1, 2 and 3 is that their proofs only require
the symmetry, and no details of the architecture or loss function need to be
specified. This means that these results are more general than the previous
literature, which often specializes in a given architecture (such as a fully
connected network) that happens to have a type of symmetry. The observation
that only knowing the symmetry alone can help us deduce so much about the
behavior of these systems hints at some underlying universal principle.

Let us first define a general type of symmetry called mirror reflection
symmetry.

######  Definition 1.

A per-sample loss function $\ell_{0}(w)$ is said to have the simple mirror
(reflection) symmetry with respect to a unit vector $n$ if, for all $w$,
$\ell_{0}(w)=\ell_{0}((I-2nn^{T})w)$.

Note that the vector $(I-2nn^{T})w$ is the reflection of $w$ with respect to
the plane orthogonal to $n$. Also, the $L_{2}$ regularization term itself
satisfies this symmetry for any $n$ because reflection is norm-preserving. An
important quantity is the average of the two reflected solutions:
$\bar{w}=(I-nn^{T})w$, where $\bar{w}$ is the fixed point of this
transformation and can be called a “symmetric solution.” This mirror symmetry
can be generalized to the case where the loss function is invariant only when
multiple mirror reflections are made.

######  Definition 2.

Let $O$ consist of columns of orthonormal vectors: $O^{T}O=I$, and
$R=I-2OO^{T}$. A loss function $\ell_{0}(w)$ is said to have the $O$-mirror
symmetry if, for all $w$, $\ell_{0}(w)=\ell_{0}(Rw)$.

By construction, $OO^{T}$ and $I-OO^{T}$ are projection matrices, and
$I-2OO^{T}$ is an orthogonal matrix. There are a few equivalent ways to see
this symmetry. First of all, it is equivalent to requiring the loss function
to be invariant only after multiple simple mirror symmetry transformations.
Let $m$ be a unit vector orthogonal to $n$. Reflections to both $m$ and $n$
give

$\displaystyle(I-2mm^{T})(I-2nn^{T})$ $\displaystyle=I-2(nn^{T}+mm^{T}).$ (5)

The matrix $nn^{T}+mm^{T}$ is a projection matrix and, thus, an instantiation
of $OO^{T}$. Secondly, because the composition of orthogonal unit vectors
spans the space of projection matrices, $OO^{T}$ is nothing but a generic
projection matrix $P$. Thus, this symmetry can be equivalently defined with
respect to $P$ such that $\ell_{0}(w)=\ell_{0}((I-2P)w)$. If we let $O$ or $P$
be rank-$1$, the symmetry reduces to the simple mirror symmetry in Definition
1.

We also make a reasonable smoothness assumption, which is only needed for part
4 of the theorem.555It is difficult to imagine what kind of network has a
negatively diverging Hessian eigenvalue. Even if such an example exists, it
goes away if we constrain the parameters in a bounded space.

######  Assumption 1.

The smallest eigenvalue of the Hessian of $\ell_{0}$ is lower-bounded by a
(possibly negative) constant $\lambda_{\rm min}.$

With these definitions, we are ready to prove the following theorem.

######  Theorem 4.

Let $\ell_{0}(w)$ satisfy the $O$-mirror symmetry. Then,

  1. 1.

for any $\gamma$, if $O^{T}w=0$, then $O^{T}\nabla_{w}\ell_{\gamma}=0$;

  2. 2.

if $O^{T}w=0$, a subset of the eigenvector of $\nabla_{w}^{2}\ell_{0}(w)$
spans ${\rm ker}(O^{T})$, and the rest spans ${\rm im}(OO^{T})$;

  3. 3.

if $O^{T}w\neq 0$, there exists $\gamma_{0}$ such that for all
$\gamma>\gamma_{0}$, $\ell_{\gamma}((I-OO^{T})w)<\ell_{\gamma}(w)$;

  4. 4.

there exists $\gamma_{1}$ such that for all $\gamma>\gamma_{1}$, all minima of
$\ell_{\gamma}$ satisfy $O^{T}w=0$.

Parts 1 and 2 are statements regarding the local gradient geometry, regardless
of the weight decay. Parts 3 and 4 are local and global statements regarding
the role of weight decay. It is instructive to show how Theorems 1, 2 and 3
are corollaries of Theorem 4. The simplest application is to the rescaling
symmetry. When the rescaling symmetry exists between two scalars $u$ and $w$,
there are two planes of mirror symmetry: $n_{1}=(1,1)$ and $n_{2}=(1,-1)$.
Here, $n_{1}$ symmetry implies that $u=-w$ is a symmetry solution, and $n_{2}$
symmetry implies that $u=w$ is a symmetry solution. Applying Theorem 4 to
these two mirrors implies that $u=0$ and $w=0$ is a symmetry solution and
obeys Theorem 1. When $u\in\mathbb{R}^{d_{1}}$ and $w\in\mathbb{R}^{d_{2}}$
are vectors of arbitrary dimensions and have the rescaling symmetry, one can
identity the implied mirror symmetry as $O=-I$, and so $I-2P=-I$: the loss
function is symmetric to a simultaneous flip of all the signs of $u$ and $w$.
Applying Theorem 4 to this mirror again allows us to derive Theorem 1.

For permutation symmetry in $\ell_{0}(\theta_{1},\theta_{2})$ with
$\theta_{i}\in\mathbb{R}^{d}$, we can identify the projection as

$P=\frac{1}{2}\begin{bmatrix}I_{d}&-I_{d}\\\ -I_{d}&I_{d}\end{bmatrix}.$ (6)

Let $\theta=(\theta_{1},\theta_{2})$ denote a vector combination of both sets
of the parameters. The permutation symmetry thus implies the mirror symmetry:
$\ell_{0}(\theta)=\ell_{0}((I-2P)\theta)$. The symmetry solution is
$\theta_{1}=\theta_{2}$, and applying the master theorem to this mirror allows
us to obtain Theorem 3.

For rotation symmetry, we note that for any projection matrix $\Pi$, the
matrix $I-2\Pi$ is an orthogonal matrix because
$(I-2\Pi)(I-2\Pi)^{T}=(I-2\Pi)^{2}=I$. Therefore, the rotation symmetry
already implies that for any $\Pi$ and $W$, $\ell_{0}((I-2\Pi)W)=\ell_{0}(W)$.
To apply the theorem, we need to view $W$ as a vector, and the corresponding
reflection matrix is ${\rm diag}(I-2\Pi,...,I-2\Pi)$, a block-wise repetition
of the matrix $I-2\Pi$, where each block corresponds to a column of $W$. By
construction, $P$ is also a projection matrix. Since this holds for an
arbitrary $\Pi$, one can choose $\Pi$ to be the plane that matches the desired
plane in Theorem 2, which can be then proved by invoking Theorem 4. Therefore,
all three main types of symmetry we study are consequences of the general
theorem.

##  4 Applications

###  4.1 Absorbing States and Stationary Conditions

To discuss the implication of symmetries, we introduce the concept of a
“stationary condition.”

######  Definition 3.

For an arbitrary function $f$, $f(\theta)=0$ is a stationary condition of
$L(\theta)$ if $f(\theta_{t})=0$ implies $f(\theta_{t+1})=0$, where
$\theta_{t}$ is the $t$-th step parameter under (stochastic) gradient descent.

A stationary condition can be seen as a special case of an absorbing state,
which is a major theme in the study of Markov processes and is associated with
complex phase-transition-like behaviors (Norris, 1998; Dickman & Vidigal,
2002; Hinrichsen, 2000). Part 1 of Theorem 4 implies the following corollary.

######  Corollary 1.

Every $O$-mirror symmetry implies a linear stationary condition:
$O^{T}\theta=0$.

Alternatively, a stationary condition can be seen as a generalization of a
stationary point because every stationary point in the landscape implies the
existence of a stationary condition – but not vice versa. For example, some
functions of the parameters might reach stationarity before the whole model
reaches stationarity. The existence of such conditions implies that there are
special subspaces in the landscape such that the dynamics of gradient descent
within these subspaces will not leave it. See Appendix Figure 4 for an
illustration of the stationary conditions.

###  4.2 Structure of the Hessian

Part 2 of Theorem 4 has very important implications for the local geometry of
the loss and the dynamics of SGD. Let $H$ denote the Hessian of the loss $L$
or that of the per-sample loss $\ell$. Part 2 states that $H$ close to
symmetry solutions are partitioned by the symmetry condition $I-2P$ to two
subspaces: one part aligns with the images of $P$, and the other part must be
orthogonal to it. Namely, one can transform the Hessian into a two-block form,
$H_{\perp}$ and $H_{\parallel}$, with $O$.666Let $\tilde{O}$ be any orthogonal
matrix whose basis includes all the eigenvectors of $O$. Then, $O^{T}HO$ will
be a two-block matrix. Note that the parameters might also contain other
symmetries, so $H_{\parallel}$ and $H_{\perp}$ may also consist of multiple
sub-blocks. This implies that close to the symmetric solutions, the Hessian of
the loss will take a highly structured form simultaneously for all data points
or batches. See Figure 2.

That the Hessian of neural networks after training takes a similar structure
is supported by empirical works. For example, the illustrative Hessian in
Figure 2 is similar to that computed in (Sagun et al., 2016). That the actual
Hessians after training are well approximated by smaller blocks is supported
by (Wu et al., 2020). Blockwise Hessian matrices can also be related to the
existence of gaps in the Hessian spectrum, which is widely observed (Sagun et
al., 2017; Ghorbani et al., 2019; Wu et al., 2020; Papyan, 2018).

It is instructive to consider the special case where $O=n^{T}$ is rank-$1$.
Part 2 implies that $n$ must be an eigenvector of the Hessian whenever the
model is at a symmetry solution. For illustration, we consider a two-layer
linear network with scalar input and outputs. The loss function can always be
written as $\ell(w,u)=\frac{1}{2}\left(x\sum_{i}^{d}u_{i}w_{i}-y\right)^{2}$.
For each index $i$, $u_{i}w_{i}$ contains the rescaling symmetry and are thus
subject to two symmetries with mirrors $(1,1)$ and $(1,-1)$. Therefore, the
theory predicts that when $u\approx w\approx 0$, the Hessian consists of $d$
$2\times 2$ symmetric matrices with $(1,1)$ and $(1,-1)$ being the
eigenvectors. This can be compared with a direct computation. When $w=u=0$,
the nonvanishing terms of the Hessian are
$\frac{\partial^{2}}{\partial{w_{i}}\partial{u_{i}}}\ell=-xy$:

$H=\begin{bmatrix}0&-xy&&&\\\ -xy&0&&&\\\ &&...&&\\\ &&&0&-xy\\\ &&&-xy&0\\\
\end{bmatrix}.$ (7)

This means that the eigenvectors are indeed $(1,1)$ and $(1,-1)$ with
eigenvalues $xy$ and $-xy$, agreeing with the theory. It is remarkable that we
can identify all the eigenvectors by only examining the symmetry in the model.

Figure 2: When symmetries exist, the stationary conditions correspond to
highly structured Hessians. Left: the symmetry mirror $O$ partitions $H$ into
two blocks: one block parallel to surfaces in $OO^{T}$, and the other
orthogonal to it. When an extra symmetry exists, these two blocks can be
decomposed into additional subblocks. Mid-Right: the loss function around a
symmetric solution has a universal geometry. Here, $s$ is the component of the
parameters along a direction of the $O$-symmetry. The competition between the
signal in the dataset and the regularization strength determines the local
landscape.

###  4.3 Dynamics of Stochastic Gradient Descent

The symmetry in the loss has a lot of consequences for the dynamics of
training with SGD in light of the recent progress in analyzing SGD. Let $O$
denote the mirror and $P=OO^{T}$ the projection matrix. If $OO^{T}w=sn$ where
$n$ is a unit vector, and $s$ is a small quantity, the model is perturbatively
away from the symmetry solution. In this case, one can expand the loss
function to leading orders in $s$:

$\ell(x,w)=\ell(x,w_{0})+\frac{1}{2}w^{T}PH(x)Pw+O(s^{4}),$ (8)

where we have defined the sample Hessian restricted to the projected subspace:
$H(x):=P\nabla_{w}^{2}\ell(x,w_{0})P$, which is a matrix of random variables.
Note that all the odd-order terms in $s$ vanish due to the symmetry in
flipping the sign of $s$. In fact, one can view the training loss
$\ell_{\gamma}$ or $L_{\gamma}$ as a function of $s$, which we denote as
$\tilde{L}(s)$, and this analysis implies that the loss landscape close to
$s=0$ takes a rather universal geometry. See Figure 2.

This allows us to characterize the dynamics of SGD in the symmetry directions:

$Pw_{t+1}=Pw_{t}-{\lambda}HPw_{t},$ (9)

where $\eta$ is the learning rate. Previously, this type of critical point is
shown to exist at interpolation minima of wide networks (Wu et al., 2018). Our
result implies that this type of solution is far more common than previously
understood and exists whenever symmetries are present.

Let us first consider GD. The largest negative eigenvalue of
$\mathbb{E}_{x}[H]$, $\xi^{*}$, thus gives the speed at which SGD escapes the
stationary condition: $Pw_{t}\propto\exp(-\xi^{*}t)$. When weight decay is
present, all the eigenvalues of $H$ will be positively shifted by $\gamma$,
and, therefore, if and only if $\xi^{*}+\gamma>0$, GD will be attracted to
these symmetric solutions. In this sense, $\xi^{*}$ gives a critical weight
decay value at which a symmetry-induced constraint is favored.

For SGD, the dynamics is qualitatively different. Naively, when using SGD, the
model will escape the stationary condition faster due to the noise. However,
this is the opposite of the truth. The existence of the SGD noise due to
minibatch sampling makes these stationary conditions more attractive. The
stability of the type of dynamics in Eq. (9) can be analyzed by studying the
condition for convergence in probability of the solution $Pw=0$ (Ziyin et al.,
2023a). One can show that $Pw$ converges to $0$ in probability if and only if
the Lyapunov exponent of the process $\Lambda$ is negative, which is possible
even if this critical point is a strict saddle.777Convergence to the saddle is
impossible under GD. When does a subspace of $Pw$ converge (or collapse) to
zero? One can derive a satisfactory approximate learning rate by making the
commutation approximation, which assumes that $H(x)$ commutes with
$H(x^{\prime})$ for all $x,\ x^{\prime}$ in the training set. In this case,
each subspace of $H(x)$ has its own Lyapunov exponent and can be analytically
computed. Let $\xi(x)$ denote the eigenvalue of $H(x)$ in this subspace. Then,
this subspace collapses when
$\Lambda=\mathbb{E}_{x}[\log|1-\lambda(\xi(x)+\gamma)|]<0$, which is negative
for a large learning rate (see Appendix B for a formal treatment). The meaning
of this condition becomes clear by expanding to the second order in $\lambda$
to obtain:

$\lambda>\frac{-2\mathbb{E}[\xi+\gamma]}{\mathbb{E}[(\xi+\gamma)^{2}]}.$ (10)

The numerator is the eigenvalue of the empirical loss, and the denominator can
be identified as the minibatch noise effect (Wu et al., 2018), which becomes
larger if the batch size is small or if the dataset is noisy. Therefore, this
phenomenon happens due to the competition between the signal and noise in the
gradient. This example shows that at a large learning rate, the stationary
conditions are favored solutions of SGD, even if they are not favored by GD.
From a Markovian perspective, this critical learning rate is when the Markov
process becomes an absorbing Markov chain.888Alternatively, similar problems
can also be analyzed using a continuous-time approximation and show that when
gradient noise is strong, these points are attractive (Vivien et al., 2022;
Chen et al., 2023). Also, convergence to these symmetry-induced saddles is not
a unique feature of SGD but happens for Adam-type dynamics as well (Ziyin et
al., 2021, 2023a).

Two novel applications of this analysis are to learning a sparse model and a
low-rank model. See Figure 1. We first apply it to a linear regression with
rescaling symmetry. It is known that when both weight decay and rescaling
symmetries are present, the solutions are sparse and identical to lasso (Ziyin
& Wang, 2023). Our result shows that even without weight decay, the solutions
are sparse at a large learning rate. Then, we consider a matrix factorization
problem. Classical results show that the solutions are low-rank when weight
decay is present (Srebro et al., 2004). Our result shows that even if there is
no weight decay, SGD at a large learning rate or gradient noise converges to
these low-rank saddles. The fact that these constrained structures disappear
completely when the symmetry is removed supports our claim that symmetry is
the cause of them.

A strong piece of evidence for the relevance of the theory to real neural
networks is that after training, the Hessian of the loss function is observed
to contain many small negative eigenvalues, which hints at the convergence to
saddle points (Sagun et al., 2016, 2017; Ghorbani et al., 2019; Alain et al.,
2019; Sankar & Balasubramanian, 2017). Another related phenomenon is that of
pathological Fisher information. From a Bayesian perspective, the matrix
$J:=\mathbb{E}_{x}[\nabla_{w}\ell\nabla_{w}^{T}\ell]$ is the Fisher
information of the system (Amari & Nagaoka, 2007). Our result implies that the
Fisher information is singular close to any symmetry solutions. To see this,
note that $O^{T}\nabla_{w}\ell(w,x)=0$ for a symmetry solution and any $x$.
Therefore, $O^{T}J=0$ implies that the Fisher information has a zero
eigenvalue along the directions orthogonal to any mirror symmetry. Many
previous works have demonstrated that the learning of neural networks passes
through regions of singular Fisher information, where the learning dynamics is
prohibitively slow (Wei et al., 2008; Cousseau et al., 2008; Fukumizu, 1996;
Karakida et al., 2019a, b). Therefore, the Fisher information having flat
directions is also strong evidence that the symmetry solutions are reached
after training.

###  4.4 Loss of Plasticity and Neural Collapses

Our theory implies that the commonly observed loss of plasticity problem in
continual and reinforcement learning (Lyle et al., 2023; Abbas et al., 2023;
Dohare et al., 2023) is attributable to symmetries in the model. For a given
task, weight decay or a finite learning rate makes the model converge to
symmetry solutions, which tend to be low-capacity constrained solutions. If we
train on an additional task, the capacity of the model can only decrease
because the symmetry solutions are also stationary conditions, which SGD
cannot escape. Fortunately, our theory suggests at least two ways to fix this
problem: (1) use an alternative parameterization that explicitly removes the
symmetry and/or (2) inject additive noise to the gradient to eliminate the
stationary conditions. There are many ways to achieve (1). An easy way is to
bias every (symmetry-relevant) parameter by a random bias: $w_{i}\to
w_{i}+\beta_{i}$, where $\beta_{i}$ is a small fixed random variable. See
Figure 3 and Appendix A for experimental details.

Figure 3: Loss of plasticity in continual learning in a vanilla linear
regressor (dashed) and linear regressors with rescaling symmetry (solid).
Vanilla regression has no symmetry and does not suffer plasticity loss,
whereas having symmetries leads to the loss of plasticity. One can fix the
problem with one of the two suggested methods, either by removing the symmetry
in the model or removing the absorbing states by injecting noise.

A related phenomenon that symmetry can explain is the collapse of neural
networks. The most common type of collapse is when the learned representation
of a neural network spans a low-rank subspace of the entire available space,
often leading to reduced expressive power and, thus, performance. In Bayesian
deep learning, a posterior collapse happens when the stochastic latent
variables are low-rank (Dai & Wipf, 2019; Alemi et al., 2018; Lucas et al.,
2019; Wang & Ziyin, 2022). This can be attributed to the double rotation
symmetry of the encoder’s last layer weight and the decoder’s first layer
weight. In self-supervised learning, a dimensional collapse happens when the
representation of the last layer is low-rank (Tian, 2022), which has been
found to be explained by the rotation symmetry of the last layer weight that
is often present in common self-supervised learning loss function. This also
explains why many self-supervised learning methods focus on introducing a term
that removes the symmetry (Bardes et al., 2021). The rank collapse that
happens in self-attention may also be relevant. Previous works attributed the
rank collapse to the use of the softmax after the key-query matrices (Dong et
al., 2021). Our theory offers a possible alternative explanation: the rank
collapse happens due to the double rotation symmetry between the key and query
matrices. However, more empirical evidence is needed to decide which
explanation is more likely. In supervised learning, the “neural collapse”
happens when the learned representation of the penultimate learning becomes
low-rank, which happens when weight decay is present (Papyan et al., 2020;
Galanti et al., 2021; Rangamani & Banburski-Fahey, 2022; Rangamani et al.,
2021). Figure 1 shows that such a phenomenon can be attributed to the
permutation symmetry in the fully connected layer. In the past, collapses in
different scenarios are often treated differently. Our result, in contrast,
provides a unified perspective of the collapse phenomenon: collapses are
caused by symmetries in the loss function. Our theory also suggests that these
collapse phenomena have a natural interpretation as “phase transitions” in
theoretical physics. A collapse solution corresponds to a symmetric state
where the “order parameter” $O^{T}w=0$, and a normal solution corresponds to a
nonsymmetric solution where $O^{T}w\neq 0$.

###  4.5 $L_{1}$ Equivalence of Mirror Symmetries

Parts 3 and 4 of Theorem 4 imply that constrained solutions are favored when
weight decay is used. These results can be stated in an alternative way: that
every mirror symmetry plus weight decay has an $L_{1}$ equivalent. To see
this, let the loss function $L_{0}(w)$ be $O$-symmetric, and $P=OO^{T}$. Let
$w$ be an arbitrary weight, which we decompose as $w=w^{\prime}+sPw/||Pw||$,
where we define $s=||Pw||$. Let us define an equivalent loss function
$\tilde{L}_{0}(w^{\prime},Pw/||Pw||,s^{2}):=L_{0}(w)$. By definition, we have
successfully constructed the $L_{1}$ equivalent of the original loss.

$L_{0}(w)+\gamma||w||^{2}=\tilde{L}_{0}(w^{\prime},Pw/||Pw||,s^{2})+\gamma(||w^{\prime}||^{2}+s^{2})=\tilde{L}_{0}(w^{\prime},Pw/||Pw||,|z|)+\gamma(||w^{\prime}||^{2}+|z|),$

where we introduced $|z|=s^{2}$. Therefore, along the symmetry-breaking
direction, the loss function has an equivalent $L_{1}$ form. One can also show
that $\tilde{L}_{0}$ is well defined as an $L_{1}$-constrained loss function.
If $L_{0}$ is differentiable, $\tilde{L}_{0}$ is differentiable except at
$s=0$. Thus, it suffices to show that the right derivative of $\tilde{L}_{0}$
with respect to $z$ exists at $z=0_{+}$. As we have discussed, at $z=0$, the
expansion of $L_{0}$ is second order in $s$. This means that the leading order
term of $\tilde{L}_{0}$ is first order in $z$, and so the $L_{1}$ penalty is
well-defined for this loss function.

Lastly, it is worth commenting that there is a crucial difference between the
original loss with the symmetry and the equivalent $L_{1}$: the $L_{1}$
equivalent is not differentiable, and so one cannot optimize it with GD. In
contrast, the original loss function is generally differentiable and can be
efficiently optimized with standard deep-learning training routines.

###  4.6 An Algorithm for Differentiable Constraint

Sparsity and low-rankness are typical structured constraints that
practitioners often want to incorporate into their models (Tibshirani, 1996;
Meier et al., 2008; Jaderberg et al., 2014). However, the known methods of
achieving these structured constraints tend to be tailored for specific
problems and based on nondifferentiable operations. Our theory shows that
incorporating symmetries is a general and scalable way to introduce such
constraints into deep learning. Consider solving the following constrained
problem: $\min_{\theta}L(\theta)$ s.t. as many elements of $P\theta$ are zero
as possible. Here, $P=OO^{T}$ is a projection matrix. This constraint is
rather general and includes common structured constraints such as parameter
sparsity, group sparsity, and low-rankness.

Our theory implies an algorithm for enforcing such constraints in a
differentiable way: introducing an artificial $O$-symmetry to the loss
function encourages the constraint $O^{T}\theta=0$, which can be achieved by
running GD on the following loss function:

$\min_{w,u,v}L(T(w,u,v))+\alpha(||w||^{2}+||u||^{2}),$ (11)

where $w,\ u,\ v$ have the same dimension as $\theta$ and
$T(w,u,v)=(I-P)v+(Pw)\odot(Pu)$, where $\odot$ denotes the Hadamard product.
We call the algorithm DCS, standing for differentiable constraint by symmetry.
This parameterization introduces the mirror symmetry to which
$O^{T}T(w,u,v)=0$ is a stationary condition. By Theorem 4, a sufficiently
large $\alpha$ ensures that $O^{T}T(w,u,v)=0$ is an energetically favored
solution. Also, note that this parametrization is a “faithful” parametrization
in the sense that it is always true that
$\min_{w,u,v}L(T(w,u,v))=\min_{\theta}L(\theta)$.

##  5 Discussion

In this work, we studied the implications of loss function symmetries on the
gradient-based learning of models. We have shown that every mirror symmetry
leads to a structured constraint of learning. This statement is examined from
two different angles: (1) such solutions are favored when $L_{2}$
regularizations are applied; (2) they are favored when the gradient noise is
strong (which can happen when the learning rate is large, the batch size is
small, or the data is noisy). We showed that the theory can analyze and
understand common structures such as sparsity and low-rankness. We also
discussed a variety of specific problems and phenomena in a unified manner.
Our result is universal in that it only relies on the existence of the
specified symmetries and does not rely on the properties of the loss function,
model architectures, or data distributions. Per se, symmetry and its
associated constraint are both good and bad. On the bad side, it limits the
expressivity of the network and its approximation power. On the good side, it
leads to more condensed models and representations, tends to ignore features
that are noisy and can improve generalization capability thereby.
Understanding symmetry systematically can help us avoid its negative side and
utilize it to our advantage.

## References

  * Abbas et al. (2023) Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado.  Loss of plasticity in continual deep reinforcement learning.  _arXiv preprint arXiv:2303.07507_ , 2023.
  * Alain et al. (2019) Guillaume Alain, Nicolas Le Roux, and Pierre-Antoine Manzagol.  Negative eigenvalues of the hessian in deep neural networks.  _arXiv preprint arXiv:1902.02366_ , 2019.
  * Alemi et al. (2018) Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy.  Fixing a broken elbo.  In _International Conference on Machine Learning_ , pp. 159–168. PMLR, 2018.
  * Amari & Nagaoka (2007) S. Amari and H. Nagaoka.  _Methods of Information Geometry_.  Translations of mathematical monographs. American Mathematical Society, 2007.  ISBN 9780821843024.  URL https://books.google.co.jp/books?id=vc2FWSo7wLUC.
  * Anderson (1972) Philip W Anderson.  More is different: Broken symmetry and the nature of the hierarchical structure of science.  _Science_ , 177(4047):393–396, 1972.
  * Bardes et al. (2021) Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  _arXiv preprint arXiv:2105.04906_ , 2021.
  * Chen et al. (2023) Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli.  Stochastic collapse: How gradient noise attracts sgd dynamics towards simpler subnetworks.  _arXiv preprint arXiv:2306.04251_ , 2023.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pp. 1597–1607. PMLR, 2020.
  * Cousseau et al. (2008) Florent Cousseau, Tomoko Ozeki, and Shun-ichi Amari.  Dynamics of learning in multilayer perceptrons near singularities.  _IEEE Transactions on Neural Networks_ , 19(8):1313–1328, 2008.
  * Dai & Wipf (2019) Bin Dai and David Wipf.  Diagnosing and enhancing vae models.  _arXiv preprint arXiv:1903.05789_ , 2019.
  * Dickman & Vidigal (2002) Ronald Dickman and Ronaldo Vidigal.  Quasi-stationary distributions for stochastic processes with an absorbing state.  _Journal of Physics A: Mathematical and General_ , 35(5):1147, 2002.
  * Dinh et al. (2017) L. Dinh, R. Pascanu, S. Bengio, and Y. Bengio.  Sharp Minima Can Generalize For Deep Nets.  _ArXiv e-prints_ , March 2017.
  * Dohare et al. (2023) Shibhansh Dohare, J Fernando Hernandez-Garcia, Parash Rahman, Richard S Sutton, and A Rupam Mahmood.  Maintaining plasticity in deep continual learning.  _arXiv preprint arXiv:2306.13812_ , 2023.
  * Dong et al. (2021) Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas.  Attention is not all you need: Pure attention loses rank doubly exponentially with depth.  In _International Conference on Machine Learning_ , pp. 2793–2803. PMLR, 2021.
  * Entezari et al. (2021) Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur.  The role of permutation invariance in linear mode connectivity of neural networks.  _arXiv preprint arXiv:2110.06296_ , 2021.
  * Fukumizu (1996) Kenji Fukumizu.  A regularity condition of the information matrix of a multilayer perceptron network.  _Neural networks_ , 9(5):871–879, 1996.
  * Fukumizu & Amari (2000) Kenji Fukumizu and Shun-ichi Amari.  Local minima and plateaus in hierarchical structures of multilayer perceptrons.  _Neural networks_ , 13(3):317–327, 2000.
  * Galanti et al. (2021) Tomer Galanti, András György, and Marcus Hutter.  On the role of neural collapse in transfer learning.  _arXiv preprint arXiv:2112.15121_ , 2021.
  * Ghorbani et al. (2019) Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao.  An investigation into neural net optimization via hessian eigenvalue density.  In _International Conference on Machine Learning_ , pp. 2232–2241. PMLR, 2019.
  * Hinrichsen (2000) Haye Hinrichsen.  Non-equilibrium critical phenomena and phase transitions into absorbing states.  _Advances in physics_ , 49(7):815–958, 2000.
  * Hou et al. (2019) Tianqi Hou, KY Michael Wong, and Haiping Huang.  Minimal model of permutation symmetry in unsupervised learning.  _Journal of Physics A: Mathematical and Theoretical_ , 52(41):414001, 2019.
  * Jaderberg et al. (2014) Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman.  Speeding up convolutional neural networks with low rank expansions.  _arXiv preprint arXiv:1405.3866_ , 2014.
  * Karakida et al. (2019a) Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari.  Pathological spectra of the fisher information metric and its variants in deep neural networks.  _arXiv preprint arXiv:1910.05992_ , 2019a.
  * Karakida et al. (2019b) Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari.  Universal statistics of fisher information in deep neural networks: Mean field approach.  In _The 22nd International Conference on Artificial Intelligence and Statistics_ , pp. 1032–1041. PMLR, 2019b.
  * Kingma & Welling (2013) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Li et al. (2019) Xingguo Li, Junwei Lu, Raman Arora, Jarvis Haupt, Han Liu, Zhaoran Wang, and Tuo Zhao.  Symmetry, saddle points, and global optimization landscape of nonconvex matrix factorization.  _IEEE Transactions on Information Theory_ , 65(6):3489–3514, 2019.
  * Lucas et al. (2019) James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi.  Don’t blame the elbo! a linear vae perspective on posterior collapse, 2019\.
  * Lyle et al. (2023) Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney.  Understanding plasticity in neural networks.  _arXiv preprint arXiv:2303.01486_ , 2023.
  * Meier et al. (2008) Lukas Meier, Sara Van De Geer, and Peter Bühlmann.  The group lasso for logistic regression.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 70(1):53–71, 2008.
  * Neyshabur et al. (2014) Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro.  In search of the real inductive bias: On the role of implicit regularization in deep learning.  _arXiv preprint arXiv:1412.6614_ , 2014.
  * Norris (1998) James R Norris.  _Markov chains_.  Number 2. Cambridge university press, 1998.
  * Papyan (2018) Vardan Papyan.  The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size.  _arXiv preprint arXiv:1811.07062_ , 2018.
  * Papyan et al. (2020) Vardan Papyan, XY Han, and David L Donoho.  Prevalence of neural collapse during the terminal phase of deep learning training.  _Proceedings of the National Academy of Sciences_ , 117(40):24652–24663, 2020.
  * Rangamani & Banburski-Fahey (2022) Akshay Rangamani and Andrzej Banburski-Fahey.  Neural collapse in deep homogeneous classifiers and the role of weight decay.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 4243–4247. IEEE, 2022.
  * Rangamani et al. (2021) Akshay Rangamani, Mengjia Xu, Andrzej Banburski, Qianli Liao, and Tomaso Poggio.  Dynamics and neural collapse in deep classifiers trained with the square loss.  _Center for Brains, Minds and Machines (CBMM) Memo_ , (117), 2021.
  * Sagun et al. (2016) Levent Sagun, Leon Bottou, and Yann LeCun.  Eigenvalues of the hessian in deep learning: Singularity and beyond.  _arXiv preprint arXiv:1611.07476_ , 2016.
  * Sagun et al. (2017) Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou.  Empirical analysis of the hessian of over-parametrized neural networks.  _arXiv preprint arXiv:1706.04454_ , 2017.
  * Sankar & Balasubramanian (2017) Adepu Ravi Sankar and Vineeth N Balasubramanian.  Are saddles good enough for deep learning?  _arXiv preprint arXiv:1706.02052_ , 2017.
  * Saxe et al. (2013) Andrew M Saxe, James L McClelland, and Surya Ganguli.  Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.  _arXiv preprint arXiv:1312.6120_ , 2013.
  * Simsek et al. (2021) Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea.  Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances.  In _International Conference on Machine Learning_ , pp. 9722–9732. PMLR, 2021.
  * Srebro et al. (2004) Nathan Srebro, Jason Rennie, and Tommi Jaakkola.  Maximum-margin matrix factorization.  _Advances in neural information processing systems_ , 17, 2004.
  * Tian (2022) Yuandong Tian.  Deep contrastive learning is provably (almost) principal component analysis.  _arXiv preprint arXiv:2201.12680_ , 2022.
  * Tibshirani (1996) Robert Tibshirani.  Regression shrinkage and selection via the lasso.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 58(1):267–288, 1996.
  * Tibshirani (2021) Ryan J Tibshirani.  Equivalences between sparse models and neural networks.  _Working Notes. URL https://www. stat. cmu. edu/~ ryantibs/papers/sparsitynn. pdf_ , 2021.
  * Tipping & Bishop (1999) Michael E Tipping and Christopher M Bishop.  Probabilistic principal component analysis.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 61(3):611–622, 1999.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Vivien et al. (2022) Loucas Pillaud Vivien, Julien Reygner, and Nicolas Flammarion.  Label noise (stochastic) gradient descent implicitly solves the lasso for quadratic parametrisation.  In _Conference on Learning Theory_ , pp. 2127–2159. PMLR, 2022\.
  * Wang & Ziyin (2022) Zihao Wang and Liu Ziyin.  Posterior collapse of a linear latent variable model.  _Advances in Neural Information Processing Systems_ , 35:37537–37548, 2022.
  * Wei et al. (2008) Haikun Wei, Jun Zhang, Florent Cousseau, Tomoko Ozeki, and Shun-ichi Amari.  Dynamics of learning near singularities in layered networks.  _Neural computation_ , 20(3):813–843, 2008.
  * Wu et al. (2018) Lei Wu, Chao Ma, et al.  How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Wu et al. (2020) Yikai Wu, Xingyu Zhu, Chenwei Wu, Annie Wang, and Rong Ge.  Dissecting hessian: Understanding common structure of hessian in neural networks.  _arXiv preprint arXiv:2010.04261_ , 2020.
  * Ziyin & Wang (2023) Liu Ziyin and Zihao Wang.  spred: Solving L1 Penalty with SGD.  In _International Conference on Machine Learning_ , 2023.
  * Ziyin et al. (2021) Liu Ziyin, Botao Li, James B Simon, and Masahito Ueda.  Sgd can converge to local maxima.  In _International Conference on Learning Representations_ , 2021.
  * Ziyin et al. (2023a) Liu Ziyin, Botao Li, Tomer Galanti, and Masahito Ueda.  The probabilistic stability of stochastic gradient descent, 2023a.
  * Ziyin et al. (2023b) Liu Ziyin, Ekdeep Singh Lubana, Masahito Ueda, and Hidenori Tanaka.  What shapes the loss landscape of self supervised learning?  In _The Eleventh International Conference on Learning Representations_ , 2023b.  URL https://openreview.net/forum?id=3zSn48RUO8M.

##  Appendix A Experimental Concerns

###  A.1 Illustration of Stationary Conditions

See Figure 4.

Figure 4: Stationary conditions in different loss landscapes. Left:
$L=(wu-1)^{2}$. Here, $u=w$ and $u=-w$ are the stationary conditions caused by
the rescaling symmetry. Right: $\theta=(u,w)$ and
$L=-||\theta||^{2}+||\theta||^{4}$. Here, the stationary condition caused by
the rotation symmetry is every straight line crossing the origin. Every
stationary condition delineates a submanifold of the entire landscape. Once
the model is in this submanifold, SGD cannot leave it.

Figure 5: Comparison for the correlation matrix of the neurons in the
penultimate layer at zero weight decay (left) and $0.001$ weight decay
(right). Upper: pre-activation correlation. Lower: post-activation
correlation. After training, the neurons are grouped into homogeneous blocks
when weight decay is present. The inset shows that such block structures are
very rare when there is no weight decay. Also, the patterns are similar for
post-activation values, which further supports the claim that the block
structures are due to the symmetry, not because of linearity.

###  A.2 Experimental Details and Additional Results for Figure 1

Here, we give the experimental details for the experiments in Figure 1.

For the sparsity experiments, we generate online data of batch size $1$ in the
following way. The input $x\in\mathbb{R}^{200}$ is sampled from a diagonal
normal distribution. The label
$y=\frac{1}{200}\sum_{i}x_{i}+\epsilon\in\mathbb{R}$, where $\epsilon$ is a
noise term also sampled from a Gaussian distribution. The training proceeds
with SGD without weight decay or momentum for $10^{5}$ iterations. The vanilla
linear regression (labeled as “w/o rescaling”) is parameterized in the
standard way: $f(x)=w^{T}x$. The regressor with rescaling symmetry is
parameterized as a Hadamard product, as in the spred algorithm (Ziyin & Wang,
2023): $f(x)=(w\odot u)^{T}x$, where $\odot$ denotes the element-wise product.

For the low-rank experiment with matrix factorization, we also generate online
data of batch size $1$ similarly. The input $x\in\mathbb{R}^{200}$ is sampled
from a diagonal normal distribution. The label is $y=\mu
x+(1-\mu)\epsilon\in\mathbb{R}^{200}$, where $\mu$ controls the degree of
noise in the label and can be seen as the effective signal-to-noise ratio in
the data. Here, the noise vector $\epsilon$ have different variances:
$\epsilon_{i}\sim\mathcal{N}(0,2/i)$. The vanilla matrix factorization model
is $f(x)=WUx$, where both $W$ and $U\in\mathbb{R}^{200\times 200}$. The
training proceeds with standard SGD without momentum or weight decay. For the
inset figure, we parameterize the network through residual connections:
$f(x)=(I_{200}+W)(I_{200}+U)x$, thus removing the rotation symmetry.

For the ResNet experiment, we train a standard ResNet18 with roughly 10M
parameters in total on the CIFAR-10 dataset. The SGD algorithm uses a batch
size of $128$ for $100$ epochs with a fixed learning rate of $0.1$ and
momentum of $0.9$, with varying degrees of weight decay. To plot the
activation correlation, we take the penultimate layer neurons of the fully
connected layer with dimension $128$ and compute the correlation matrix over
their activation of $2000$ unseen test points. The neurons are sorted
according to the eigenvector with the largest eigenvalue of the correlation
matrix to reveal its block structure. Importantly, the pre- and post-
activations have a similar correlation structure, showing that the effect is
not due to linearity but the permutation symmetry. See Figure 5 for the
comparison between the pre- and post-activation correlations.

###  A.3 Experimental Detail for Continual Learning

Here, we give the experimental detail for the continual learning experiment in
Figure 3.

For all the experiments in the figure, the training proceeds with Adam without
momentum with a batch size of $16$ for $25000$ steps. Every task consists of a
dataset of $100$ data points drawn from the following distribution. The input
$x\in\mathbb{R}^{100}$ is sampled from a diagonal normal distribution. The
label $y=\frac{1}{100}\sum_{i}x_{i}+\epsilon\in\mathbb{R}$, where $\epsilon$
is a noise term also sampled from a Gaussian distribution. The weights
obtained from training on task $j$ is used as the initialization for task
$j+1$, which consists of another $100$ data points sampled in the same way. We
train for $10$ tasks and record the number of dead neurons in the model. The
dead neurons are defined as the number of parameters that have a vanishing
gradient.

To have strong control over the experimental conditions, we use vanilla linear
regression as a base model, which is shown in the solid curve. Because there
is no symmetry in the model, the vanilla linear regression has a minimal level
of dead neurons, and its number does not increase as the number of tasks
increases.

In contrast, for a linear regression with augmented rescaling symmetry where
we reparameterize every weight of the linear regressor by the Hadamard product
of two independent weights (also see the previous section), the loss of
plasticity problem emerges, and the number of dead neurons increases steadily
as one train on more and more tasks. To show that symmetry is indeed the cause
of the problem, we fix the loss of plasticity problem in this model with the
two suggested methods. First, we inject a very weak Gaussian random noise with
variance $1e-4$ to the gradient every step. Because this removes the absorbing
states, or equivalently the stationary conditions, the number of dead neurons
reduces to the same level as vanilla regression. Alternatively, we bias every
weight parameter by a random and fixed constant: $w_{i}\to w_{t}+\beta_{i}$,
where $\beta_{i}$ is drawn from a Gaussian distribution with variance $1e-4$.
Because this parametrization removes the symmetry in the model, it also fixes
the loss of plasticity problem, as we expect from the theory.

##  Appendix B Theoretical Concerns

###  B.1 A Formal Derivation of Eq. (10)

By Definition 2, the loss function has the $O$-symmetry if for any $x$

$\ell(w,x)=\ell(I-2OO^{T}w,x).$ (12)

As we discussed, this means that for every data point $x$, the per-sample
Hessian $\nabla^{2}_{w}\ell(w,x)$ takes the same block-wise structure outlined
in Fig. 2. For this chapter, the most important consequence of Theorem 4 is
that $O^{T}w=0$ is a symmetry solution of $\ell(w,x)$ for all $x$.

We are interested in the stability of these solutions. Namely, we are
interested in whether the model will be attracted back to the solution if we
are perturbatively away from it. The expansion of the per-sample loss to the
second order gives:

$\ell(w,x)=\ell(w^{(0)},x)+\frac{1}{2}w^{T}P\nabla_{w}^{2}\ell(w^{(0)},x)Pw+O(s^{4}),$
(13)

where $P=OO^{T}$ is a projection matrix, and $w^{(0)}=Pw$ is the component of
$w$ that is orthogonal to the symmetry breaking subspace. Here, we care about
when $Pw$ is attracted towards $0$. The dynamics of $z:=Pw$ is thus a
stochastic linear dynamics:

$z_{t+1}=z_{t}-\lambda\hat{H}(w^{(0)},x)z_{t},$ (14)

where $\hat{H}(w^{(0)},x)=P\nabla_{w}^{2}\ell(w^{(0)},x)P$.

To proceed, we make the following assumption.

######  Assumption 2.

(Stationary background dynamics) The motion of $w_{0}$ is sufficiently slow
that $\hat{H}(w^{(0)},x)=\hat{H}^{*}(x)$ is a constant function in $w^{(0)}$.

This also implies that any eigenvalue of $H$ also only depends on $x$. This
assumption holds when the time scale of relaxation for $w^{(0)}$ is far slower
than that of $Pw$ or when the dynamics is already stationary, namely, close to
convergence.

When Assumption 2 holds, and $O$ is rank-$1$, this dynamics is analytically
solvable. By Theorem 4, if $O=n$ is rank-$1$, $n$ is an eigenvector of
$\hat{H}$ for all $x$. Thus, the dynamics simplifies to a one-dimensional
dynamics, where $h(x)\in\mathbb{R}$ is the corresponding eigenvalue of
$\hat{H}(w_{0},x)$:

$z_{t+1}=z_{t}-\lambda h(x)z_{t}.$ (15)

The sufficient and necessary condition for the stability of this dynamics at
$z=0$ has an analytical solution (Ziyin et al., 2023a), which is Eq. (10).

######  Theorem 5.

(Ziyin et al. (2023a)) Let $w_{t}$ follow Eq. (15). Then, for any data set,

$w_{t}\to_{p}0$ (16)

if and only if999This condition generalizes to the case when the batch size
$S$ is larger than $1$, where $h(x)$ becomes the per-batch Hessian, and the
expectation is taken over all possible batches.

$\mathbb{E}_{x}[\log|1-\lambda h(x)|]<0.$ (17)

###  B.2 Proofs

####  B.2.1 Proof of Theorem 1

###### Proof.

We first show part 1. The rescaling symmetry states that for any $\epsilon\neq
1$ and $w,\ u$,

$\ell_{0}((1+\epsilon)u,w/(1+\epsilon))=\ell_{0}(u,w).$ (18)

For an infinitesimal $\epsilon$, this condition leads to

$\nabla_{w}\ell_{0}\cdot w=\nabla_{u}\ell_{0}\cdot u.$ (19)

Taking the derivative of both sides over $w$, we obtain

$\nabla_{w}\ell_{0}=-\nabla^{2}_{w}\ell_{0}\cdot
w+\nabla_{w}\nabla_{u}\ell_{0}\cdot u.$ (20)

Therefore, the gradient of $\ell_{\gamma}$ is $-\nabla^{2}_{w}\ell_{0}\cdot
w+2\gamma w+\nabla_{w}\nabla_{u}\ell_{0}\cdot u$. When both $w$ and $u$ are
zero, $\nabla_{w}\ell_{\gamma}=0$. Likewise, we can show that
$\nabla_{u}\ell_{\gamma}=0$. This proves part 1.

For part 2, let us denote the quantity $\ell_{\gamma}(0,0)-\ell_{\gamma}(u,w)$
as $\Delta$. Now, note that
$\Delta=\ell_{0}(0,0)-\ell_{0}(u,w)-\gamma(||u||^{2}+||w||^{2})$, and so
setting

$\gamma>\max\left(0,\frac{\ell_{0}(0,0)-\ell_{0}(u,w)}{||u||^{2}+||w||^{2}}\right)$
(21)

fulfills the requirement. Note that because $\ell_{0}$ is differentiable, the
fraction always exists. This proves part 2.

∎

####  B.2.2 Proof of Theorem 2

###### Proof.

We focus on proving part 1. For an arbitrary and fixed index, $i$, of the
singular values of $W$, we consider a continuous transformation of
$W_{0}=W(s)$. Define a diagonal matrix $\tilde{\Sigma}_{jj}=\Sigma_{jj}$ for
all $j\neq i$, and define

$\tilde{\Sigma}_{jj}(s)=\begin{cases}\Sigma_{jj}\text{ if $j\neq i$;}\\\
s\Sigma_{jj}\text{ if $j=i$.}\end{cases}$ (22)

We also define a transformed version of $V$, which depends on an arbitrary
vector $z$:

$\tilde{V}_{kl}(z)=\begin{cases}V_{kl}\text{ if $k\neq i$};\\\ z_{l}\text{ if
$k=i$}.\end{cases}$ (23)

With $\tilde{\Sigma}$ and $\tilde{V}$, we define $\tilde{W}$

$\tilde{W}(s,z)=U\tilde{\Sigma}(s)\tilde{V}.$ (24)

We note two different features of this transformation: (1) $W(0)$ is low-rank,
and (2) for any $s$, $\ell(W(s))=\ell(W(-s))$. To see this, note that there
exists an orthogonal matrix $R$ such that

$RW(s)=W(-s).$ (25)

By the assumed symmetry of the loss function, we have
$\ell(W(s))=\ell(RW(s))=\ell(W(-s))$. Because

$\frac{d}{ds}W_{jk}(s,z)=U_{ji}\Sigma_{ii}\tilde{V}_{ik}(z)=U_{ji}\Sigma_{ii}z_{k},$
(26)

we can take the derivative of $s$ of both sides of the equality
$\ell(W(s))=\ell(W(-s))$ to obtain a low-rank condition on the gradient width
as a matrix:

$\Sigma_{ii}\sum_{jk}\left[\nabla_{W_{jk}}\ell(W(s))+\nabla_{W_{jk}}\ell(W(-s))\right]U_{ji}z_{k}=0.$
(27)

In the limit $s\to 0$, $W(s)=W(-s)$ and so the equality leads to

$2\Sigma_{ii}\sum_{jk}\nabla_{W_{jk}}L(W(0))U_{ji}z_{k}=0.$ (28)

Because this equality must hold for any $z_{k}$, we have that $U_{ji}$ must be
a left eigenvector of $\nabla_{W_{jk}}\ell(W(0))$ with zero eigenvalues.
Substituting into the gradient descent algorithm, we have

$\sum_{j}U_{ji}W_{jk,t+1}=\sum_{j}U_{ji}W_{jk,t}-\lambda\sum_{j}U_{ji}\nabla_{W_{jk}}\ell(W_{t})=0.$
(29)

This proves part 1.

For part 2, we note that the Frobenious norm of a matrix is the sum of its
squared singular values. Therefore, if we hold other singular values unchanged
and shrink one of the singular values to $0$, the $L_{2}$ regularization part
of the loss function will strictly decrease. The rest of part 2 is the same as
the proof of Theorem 1. ∎

####  B.2.3 Proof of Theorem 3

###### Proof.

The symmetry condition is

$\ell_{0}(\theta_{a},\theta_{b})=\ell_{0}(\theta_{b},\theta_{a}).$ (30)

Taking the gradient of both sides with respect to $\theta_{a}$, we obtain

$\nabla_{\theta_{a}}\ell_{0}(\theta_{a},\theta_{b})=\nabla_{\theta_{a}}\ell_{0}(\theta_{b},\theta_{a}).$
(31)

When $\theta_{a}=\theta_{b}$, we can write the above condition as

$\nabla_{\theta_{a}}\ell_{0}(\theta_{a},\theta_{b})=\nabla_{\theta_{b}}\ell_{0}(\theta_{a},\theta_{b}).$
(32)

This proves the first part of the theorem.

We now prove the second part of the theorem. Let us define interpolation
functions $g_{a}$ and $g_{b}$:

$g_{a}(\mu)=(0.5-\mu)\theta_{a}+(0.5+\mu)\theta_{b};$ (33)
$g_{b}(\mu)=(0.5+\mu)\theta_{a}+(0.5-\mu)\theta_{b}.$ (34)

With these definitions, we have $g_{a}(\mu)=g_{b}(-\mu)$. Also, we note that

$g_{a}(0)=g_{b}(0)=0.5\theta_{a}+0.5\theta_{b},$ (35)

which is the solution we want to compare with.

The loss function is given by

$\ell_{\gamma}(\theta_{a},\theta_{b})=\ell_{\gamma}(g_{a}(0.5),g_{b}(0.5)).$
(36)

In contrast, for the homogeneous solution, the loss value is

$\ell_{\gamma}(g_{a}(0),g_{b}(0)).$ (37)

The norms of the two solutions, $\mu=0.5$ and $\mu=0$, can be compared:

$\Delta:=||g_{a}(0)||^{2}+||g_{b}(0)||^{2}-||g_{a}(0.5)||^{2}+||g_{b}(0.5)||^{2}<0,$
(38)

where the inequality follows from the Cauchy-Schwarz inequality and the
assumption that $\theta_{a}\neq\theta_{b}$. Therefore, for any

$\gamma>\frac{\ell_{0}(g_{a}(0.5),g_{b}(0.5))-\ell_{0}(g_{a}(0),g_{b}(0))}{\Delta},$
(39)

$\ell_{\gamma}(g_{a}(0),g_{b}(0))<\ell_{\gamma}(\theta_{a},\theta_{b})$. This
proves the second part of the statement. ∎

####  B.2.4 Proof of Theorem 4

###### Proof.

Part 1. Let $R:=(I-2OO^{T})$. By assumption, we have $O^{T}w=0$. Now, consider
a linearly transformed version of $w$:

$\tilde{w}(s)=w+sn,$ (40)

where $n$ is any unit vector in the image of $OO^{T}$. Note that we have the
following relation:

$R\tilde{w}(s)=(I-2OO^{T})(w+sn)=w-sn=\tilde{w}(-s).$ (41)

Therefore, by definition of the mirror symmetry, we have that for all $s$:

$\ell_{\gamma}(\tilde{w}(s))=\ell_{\gamma}(\tilde{w}(-s)).$ (42)

Dividing both sides by $s$ and taking the limit $s\to 0$, we obtain

$n^{T}\nabla_{w}\ell_{\gamma}(w)=0.$ (43)

Because $n$ is arbitrary, one can select a set of $n$ such that they span the
rows of $O^{T}$, and we obtain that $O^{T}\nabla_{w}\ell_{\gamma}(w)=0$. This
finishes part 1.

Part 2. Let $O^{T}w=0$. By symmetry, we have that for any $s\in\mathbb{R}$ and
$n\in{\rm ker}(O^{T})^{\perp}$:101010We use ${\rm ker}(O^{T})^{\perp}$ to
denote the set of all vectors that is perpendicular to all the vectors in
${\rm ker}(O^{T})$.

$\ell_{0}(w+sn)=\ell_{0}(w-sn).$ (44)

Let $m$ be an arbitrary vector in ${\rm ker}(O)$. Then, we also have that for
any $s^{\prime}\in\mathbb{R}$

$\ell_{0}(w+sn+s^{\prime}m)=\ell_{0}(w-sn+s^{\prime}m).$ (45)

Taking derivative over $s^{\prime}$ for both sides and let $s^{\prime}\to 0$,
we obtain

$m^{T}\nabla\ell_{0}(w+sn)=m^{T}\nabla\ell_{0}(w-sn).$ (46)

Taking derivative over $s$ and let $s\to 0$, we obtain

$2m^{T}\nabla_{w}^{2}\ell_{0}(w)n=0.$ (47)

Since $m$ is an arbitrary vector in ${\rm ker}(O^{T})$ and $n$ is an arbitrary
in ${\rm ker}(O^{T})^{\perp}$, this implies that

$\nabla_{w}^{2}\ell_{0}(w)n\in{\rm ker}(O^{T})^{\perp},$ (48)
$\nabla_{w}^{2}\ell_{0}(w)m\in{\rm ker}(O).$ (49)

Namely, a subset of the eigenvectors of $\nabla_{w}^{2}\ell_{0}(w)$ spans
${\rm ker}(O^{T})^{\perp}$ and the rest spans ${\rm ker}(O)$. This proves part
2.

To prove part 3, we first recognize that if we only look at the $L_{2}$
regularization part of the loss function, an orthogonal solution is always
favored over a non-orthogonal solution. Let $u$ be an arbitrary solution such
that $O^{T}u\neq 0$. We decompose $u$ into an orthogonal part and a non-
orthogonal part:

$u=w+sn.$ (50)

where $O^{T}w=0$ and $OO^{T}n=n$. Since $w$ and $n$ are orthogonal, we have
that

$||u||^{2}-||w||^{2}=s^{2}>0.$ (51)

Therefore, if

$\gamma>\frac{\ell_{0}(w)-\ell_{0}(u)}{s^{2}},$ (52)

we always have $\ell_{\gamma}(u)>\ell_{\gamma}(w)$. However, since we have
$w=(I-OO^{T})u$, this proves part 3.

Part 4. By assumption, the smallest Hessian eigenvalue of $\ell_{0}$ is lower
bounded by $\lambda_{\min}$. Therefore, if $\gamma>\lambda_{\min}$,
$\ell_{\gamma}$ has a positive definite Hessian everywhere, implying that its
gradients are monotone and that the global minimum is unique. Now, suppose
there exists $u=w+c_{0}n$ such that $c_{0}\neq 0$, $O^{T}w=0$, $OO^{T}n=n$,
and

$\nabla\ell_{\gamma}(u)=0.$ (53)

Then,

$n^{T}\nabla\ell_{\gamma}(u)=0=n^{T}\nabla\ell_{\gamma}(w).$ (54)

This implies that the gradient is not monotone, which contradicts the
assumption. Therefore, we have proved part 4. ∎