Learning Linear Groups in Neural Networks
Employing equivariance in neural networks leads to greater parameter
efficiency and improved generalization performance through the encoding of
domain knowledge in the architecture; however, the majority of existing
approaches require an a priori specification of the desired symmetries. We
present a neural network architecture, Linear Group Networks (LGNs), for
learning linear groups acting on the weight space of neural networks. Linear
groups are desirable due to their inherent interpretability, as they can be
represented as finite matrices. LGNs learn groups without any supervision or
knowledge of the hidden symmetries in the data and the groups can be mapped to
well known operations in machine learning. We use LGNs to learn groups on
multiple datasets while considering different downstream tasks; we demonstrate
that the linear group structure depends on both the data distribution and the
considered task.
Emmanouil Theodosis, Karim Helwani, Demba Ba
# Learning Linear Groups in Neural Networks

###### Abstract

Employing equivariance in neural networks leads to greater parameter efficiency and improved generalization performance through the encoding of domain knowledge in the architecture; however, the majority of existing approaches require an a priori specification of the desired symmetries. We present a neural network architecture, Linear Group Networks (LGNs), for learning linear groups acting on the weight space of neural networks. Linear groups are desirable due to their inherent interpretability, as they can be represented as finite matrices. LGNs learn groups without any supervision or knowledge of the hidden symmetries in the data and the groups can be mapped to well known operations in machine learning. We use LGNs to learn groups on multiple datasets while considering different downstream tasks; we demonstrate that the linear group structure depends on _both_ the data distribution and the considered task.

## 1 Introduction

Convolutional neural networks (LeCun et al., 1989, 1998) were one of the first architectures that introduced the concept of equivariance to neural networks. By exploiting translation symmetry, networks can greatly reduce their number of learnable parameters compared to fully connected networks, while at the same time resulting in models that generalize better. While equivariance to translations is a natural choice for image classification or speech recognition, it is not the only one; planar rotations of objects leave the class identity unchanged and variations in pitch do not alter the spoken utterance. Moreover, certain data modalities, such as graphs, do not utilize translational symmetries and instead require other symmetries.

Equivariant convolutional networks (Cohen and Welling, 2016) were proposed as a method to design equivariant neural networks. However, encoding symmetries poses certain challenges: most architectures require an explicit specification of the desired symmetry, lack a general framework and require special treatment for every group, and only allow for _unidirectional_ knowledge flow. Indeed, baseline frameworks do not allow learning the symmetries from the data and instead they need to be specified during construction. At the same time, they are not generalizable as the network architecture needs to change fundamentally in order to account for the different groups, depending on the desired symmetry. Finally, it is possible to embed known symmetries in the model (creating a knowledge path from the designer to the network), but it is not possible to uncover symmetries in the data and extract them. This step is crucial as in many modalities we lack a clear understanding of the present symmetries.

Ideally, the equivariant structure would be learnt from the data, for the specific task. At the center front of this equivariant structure is the operation that characterizes the group: its group action.
However, isomorphisms between groups can inhibit interpretability, as two groups can have the same structure but their actions transform the data in different manners. In that context, recovering a group structure that is isomorphic to a known group, for example \(\mathcal{D}_{4}\) offers little insight without knowing how the elements operate in the data domain. We would like to recover the specific group that is interesting for the data and the task. The _linear group_, the group of invertible transformations, is an extensive group that encapsulates a vast variety of groups as subgroups, including rotations, reflections, translations, and more. Under this group, group actions are clearly understood and can be visualized giving insights on how the matrices interact with the signals from the data domain.

We propose Linear Group Networks (LGNs) that learn elements of the linear group and uses them to construct cyclic groups on the weight space of neural networks. Our main contributions can be summarized as follows:

* We propose a computational framework that learns elements of the _general linear group_\(\mathrm{GL}_{d}(K)\) and use them to construct sets of filters that belong to a (finite) cyclic group whose action is a linear operator. The framework is constructive and requires minimal changes to existing architectures, training procedures, and pipelines.
* We apply the framework on datasets of natural images and recover the group structure of the filter sets. We discover multiple structures of interest, including groups whose actions are _skew-symmetric_, _Toeplitz_, or act on _multiple scales_.
* We analyze the learned actions via ablation studies and draw connections to well-known operations in machine learning. We show that certain sets of groups have actions that have a high correlation with compositions of rotations and median filtering (related to the popular pooling operations in neural networks).

Finally, we emphasize that a main contribution of our work is analyzing the learned group actions for the different filter sets when the architecture is trained on natural images, and drawing interepretable analogues to well-known operations in machine learning. Prior work focused on the ability of their networks to learn carefully curated (and well-studied) known symmetries; instead, we analyze the group structures that organically arise in ubiquitous datasets as an effort to inform our understanding of important symmetries for data modalities and different tasks, rather than provide an architecture purely for symmetry learning.

## 2 Related work

In recent years there has been a resurgence in interest for equivariant neural networks following the concurrent works of Cohen and Welling (2016) and Dieleman et al. (2016), where convolutional frameworks that were equivariant with respect to elementary rotations and reflections were introduced. The work of Kondor and Trivedi (2018) showed that linear equivariant maps are intertwined with group convolutions and inspired a vast array of practical architectures for encoding equivariances, including architectures equivariant to arbitrary rotations (Worrall et al., 2017); steerable representations (Cohen and Welling, 2017); avoiding interpolation artifacts by modeling equivariances on the sphere (Esteves et al., 2020); and extensions to vector fields (Marcos et al., 2017).

A growing body of work has studied the process of learning symmetries: Cohen and Welling (2014); Dehmamy et al. (2021); Moskalev et al. (2022) consider learning Lie groups by utilizing the corresponding Lie algebras; Benton et al. (2020) learn distributions over data augmentations to recover invariances from the data; Romero and Lohit (2022) consider approximate symmetries over exact ones, as exact symmetries might be restrictive for natural data; and Zhou et al. (2021) propose a meta-learning scheme for learning equivariances by reparametrizing the weight matrices. Our work is most closely related to that of Zhou et al. (2021), however our method doesn't rely on a meta-learning framework and learns the filter sets and the corresponding group actions at the same time. Most importantly, we address one of the main limitations of Zhou et al. (2021), where symmetry discovery was not possible in single-task settings. Recently, Sanborn et al. (2023) proposed the use of the bispectrum to learn groups and their orbits. However, their framework acts directly on the _input_ space, whereas ours acts on the _weight_ space.
Preliminaries

In this section we introduce the necessary background for our method, which relies on equivariance, cyclical and linear groups, and unfolded networks. Unfolded architectures have been used in the literature for their parameter efficiency and principled derivation from optimization problems. Moreover, they have been used with great success for compressive sensing (Gregor and LeCun, 2010), state-of-the-art denoising (Tolooshams et al., 2021), downstream tasks and rank-minimization (Rolfe and LeCun, 2013; Jing et al., 2020), and for deep representations (Sulam et al., 2020). We note that our method does not rely on unrolling and is compatible with any network architecture as it acts directly on the weights of each layer. For the bulk of our experiments in Section 5, we opted for unfolded networks because of the residual estimation at every layer: the input is approximately reconstructed at every layer to compute the next representation. As our work considers group actions as the centerpiece of group learning, having filters that act on the data space allows us to learn human-interpretable group actions, which becomes challenging when the group acts on a space of high-dimensional feature maps. At the same time, acting on the data space allows us to maintain moderate model sizes, making our presented networks efficient and scalable.

**Equivariance.** Consider an operator \(f:V\to W\) and a family of actions \(\mathcal{T}\). We call \(f\)_equivariant_ with respect to the family \(\mathcal{T}\) if for \(T\in\mathcal{T}\) and any \(x\in V\) it holds

\[f(T(x))=T^{\prime}(f(x)), \tag{1}\]

for some transformation \(T^{\prime}\in\mathcal{T}^{\prime}\). Note that in general the action \(T\) and transformation \(T^{\prime}\) are not the same; this can be directly seen since \(\mathrm{dom}(T)=V\) and \(\mathrm{dom}(T^{\prime})=W\) (however, even when \(\mathrm{dom}(T)=\mathrm{dom}(T^{\prime})\) the operations need not be the same).

**Cyclic groups.** Consider a non-trivial set \(G\) and an operation \(*\). We call \((G,*)\) a _group_ if \(*\) is associative on \(G\), \(G\) contains an identity element \(e\) with respect to \(*\), and for any \(g\in G\) there exists an inverse element \(g^{-1}\in G\) such that \(g*g^{-1}=e\).

We call the group a cyclic group if every element in the group can be generated via consecutive applications of a basis element \(g\) (i.e., any element of \(G\) can be expressed as \(g^{k}\) for \(k\in\mathbb{Z}\)), and we call \(g\) the _generator_ of \(G\). When the cyclic group is finite and has order \(p\) it can be denoted as

\[G=\{e,g,g^{2},\ldots,g^{p-1}\}, \tag{2}\]

where higher order exponents get mapped to the \(p\) canonical elements, i.e. for \(l\geq p\) it holds \(g^{l}=g^{l\mod p}\).

**Linear groups.** For our purposes, linear groups will refer to subgroups of the _general linear group_\(\mathrm{GL}_{d}(K)\) of \(d\times d\) invertible matrices over the field \(K\). Considering a collection of \(C\) of elements of \(\mathrm{GL}_{d}(K)\), the subgroup generated by \(C\) is a linear group. A special case of interest for our work arises when \(C\) contains only a single element \(c\); then \(C\) becomes a cyclic group generated by \(c\).

**Unrolled sparse autoencoders.** Unrolled networks temporally unroll the steps of optimization algorithms, mapping algorithm iterations to network layers. In that way, the output of the neural network can be interpreted as the output of the optimization algorithm, with theoretical guarantees under certain assumptions. The _Iterative Soft Thresholding Algorithm_ (ISTA), an algorithm for sparse coding, has inspired several architectures (Gregor and LeCun, 2010; Simon and Elad, 2019; Sulam et al., 2020; Tolooshams et al., 2021), due to the desirability of sparse representations for interpretation purposes and also the connection between ReLU and soft thresholding. Within that framework, the representation at layer \(l+1\) is given by

\[\mathbf{z}^{(l+1)}=\mathcal{S}_{\lambda}\left(\mathbf{z}^{(l)}+\alpha\mathbf{W}_{l}^{T}( \mathbf{x}-\mathbf{W}_{l}\mathbf{z}^{(l)})\right), \tag{3}\]

where \(\mathbf{x}\) is the _original_ input, \(\mathbf{z}^{(l)}\) is the representation at the previous layer, \(\mathbf{W}_{l}\) are the weights of layer \(l\), \(\alpha\) is a constant such that \(\frac{1}{\alpha}\geq\sigma_{\max}(\mathbf{W}_{l}^{T}\mathbf{W}_{l})\), and \(\mathcal{S}_{\lambda}\) is the _soft thresholding_ operator defined as

\[\mathcal{S}_{\lambda}(u)=\mathrm{sign}(u)\cdot\mathrm{ReLU}(|u|-\lambda). \tag{4}\]

A one-sided version of (4) arises if we enforce \(u>0\). Then, the soft thresholding operator becomes a shifted version of ReLU, i.e. \(\mathcal{S}_{\lambda}(u)=\mathrm{ReLU}(u-\lambda)\) (and the bias \(\lambda\) can be incorporated to the weights of the network prior the activation). If the weights of all the \(L\) layers are equal, i.e. \(W_{1}=\ldots=W_{L}\), we call the network _tied_. As a final remark, Equation (3) can be rewritten as

\[\mathbf{z}^{(l+1)}=\mathcal{S}_{\lambda}\left((I-\alpha\mathbf{W}_{l}^{T}\mathbf{W}_{l}) \mathbf{z}^{(l)}+\alpha\mathbf{W}_{l}^{T}\mathbf{x}\right)=\mathcal{S}_{\lambda}\left( \mathbf{W}_{z}\mathbf{z}^{(l)}+\mathbf{W}_{x}\mathbf{x}\right),\]
where we let \(\mathbf{W}_{z}=(I-\alpha\mathbf{W}_{l}^{T}\mathbf{W}_{l})\) and \(\mathbf{W}_{x}=\alpha\mathbf{W}_{l}^{T}\), which can be interpreted as a residual network (He et al., 2016), with a residual connection to the input. Convolutional extensions of (3) can be readily derived by replacing the matrix-vector products with correlation/convolution operations.

## 4 Group learning framework

We propose a framework for learning linear groups acting on the filters of neural networks. We consider cyclic linear groups generated by a single element of the generalized linear group; to that end, we will allow \(K\) groups of size \(p\) at every layer \(l\) (all of which are design choices of the model). This implies that at every layer the architecture learns \(K\) filter sets of exactly \(p\) elements each, such that the filter sets are generated via a generating element \(g_{(k,l)}\). Then, the weights of each group are related via the application of the group action

\[[\mathbf{W}_{l}^{k}\quad g_{(k,l)}\mathbf{W}_{l}^{k}\quad\dots g_{(k,l)}^{p-1}\mathbf{W}_{l }^{k}]. \tag{5}\]

Let \(\mathbf{W}_{l}^{k}\in\mathbb{R}^{n\times m}\), i.e., the weights of every group at every layer are real-valued matrices of size \(n\times m\). An initial approach would be to model \(g_{(k,l)}\) as an element of \(\mathrm{GL}_{n}(\mathbb{R})\), however we will see that this has significant limitations.

**Limitations of matrices in the weight space.** Consider using matrices \(\mathbf{A}\in\mathbb{R}^{n\times n}\) to represent the generators \(g_{(k,l)}\) of each group at every layer. Then, assuming a _basis_ filter \(\mathbf{W}_{l}^{k}\in\mathbb{R}^{n\times m}\) for each filter set, generate the rest of the filters by applying the group action on the basis filter of each group

\[[\mathbf{W}_{l}^{k}\quad\mathbf{A}_{(k,l)}\mathbf{W}_{l}^{k}\quad\dots\mathbf{A}_{(k,l)}^{p-1} \mathbf{W}_{l}^{k}].\]

This is a natural model since, if \(\mathrm{rank}(A)=n\) (assuming \(m\geq n\) for now), this linear operator can generate any vector in \(\mathbf{x}\in\mathbb{R}^{n}\), as it is a basis for \(\mathbb{R}^{n}\). However, while such an approach would be able to learn generators that can produce any vector in \(\mathbb{R}^{n}\), it would ignore any spatial relations in the basis filters (or, more generally, any co-dependence or correlation between different columns). Indeed, matrices (and by extension, convolutional neural network filters) cannot be used to represent _any_ linear operator on their own set, as we prove by providing a short counter-example below.

**Proposition 1**.: _There are linear operators on the space of \(n\times m\) matrices that can't be represented via an \(n\times n\) matrix._

Proof.: Assume that every linear operator on \(n\times m\) matrices can be represented by a \(n\times n\) matrix and consider an operator \(f:V\to V\) (with \(V\equiv\mathbb{R}^{n\times m}\)) that swaps the top left with the bottom right element, i.e.

\[\left(f(\mathbf{W})\right)_{ij}=\begin{cases}W_{nm},&\text{ if }(i,j)=(1,1),\\ W_{11},&\text{ if }(i,j)=(n,m),\\ W_{ij},&\text{ otherwise}.\end{cases}\]

This operator is linear since \(f(\alpha\mathbf{X}+\beta\mathbf{Y})=\alpha f(\mathbf{X})+\beta f(\mathbf{Y})\). However, this operator can't be represented by a \(n\times n\) matrix since these matrices, as linear operators, act on \(n\)-dimensional vectors (i.e., treat every column of the input independently). This follows since, assuming \(f\) is parametrized by \(\mathbf{A}\in\mathbb{R}^{n\times n}\), we have

\[\left(f(\mathbf{W})\right)_{ij}=\sum_{k}A_{ik}W_{kj}.\]

In the above equation the operator has a strict dependence on column \(j\) of the input and the computation is independent of all other columns; therefore, swapping the corner most elements columns \(1\) and \(m\) is not feasible using this parametrization.

Note that a simpler example would be an operator such that \(\mathbf{W}\mapsto\mathbf{W}^{T}\), as transposition is a linear map. However, if \(n\neq m\) then the group elements would belong in different linear spaces (\(\mathbb{R}^{n\times m}\) versus \(\mathbb{R}^{m\times n}\)). While this is not prohibitive, it would complicate the notation for the group definition and hence we presented a slightly more involved example. 

**Employing vectorization.** As we showed above, modeling the group actions \(g_{(k,l)}\) as \(n\times m\) matrices leads to restrictive groups where simple linear operators are excluded. Instead, we propose the modeling of the group actions \(g_{(k,l)}\) via vectorization. We first define the vectorization operator along with its inverse.
**Definition 1**.: _Consider a matrix \(\mathbf{A}\in\mathbb{R}^{n\times m}\). Define the vectorization operator, denoted as \(\operatorname{vec}\), as follows_

\[\operatorname{vec}: \mathbb{R}^{n\times m}\to\mathbb{R}^{n\cdot m} \tag{6}\] \[\mathbf{A}\mapsto[A_{11},\dots,A_{n1},A_{12},\dots,A_{n2},\dots,A_{1 m},\dots,A_{nm}]^{T}.\]

_The vectorization operator can also be expressed as a linear sum using the Kronecker product: \(\operatorname{vec}(\mathbf{A})=\sum_{i=1}^{m}\mathbf{e}_{i}\otimes\mathbf{A}\mathbf{e}_{i}\), where \(\mathbf{e}_{i}\in\mathbb{R}^{m}\) denotes the \(i\)-th basis vector of \(\mathbb{R}^{m}\). The inverse map \(\operatorname{vec}^{-1}_{n\times m}\), where the subscript \(n\times m\) will be dropped for conciseness, is also defined via Kronecker products_

\[\operatorname{vec}^{-1}: \mathbb{R}^{n\cdot m}\to\mathbb{R}^{n\times m} \tag{7}\] \[\mathbf{a}\mapsto\left(\operatorname{vec}^{T}(\mathbf{I}_{m})\otimes\bm {I}_{n}\right)(\mathbf{I}_{m}\otimes\mathbf{a}),\]

_where \(\mathbf{I}_{n}\in\mathbb{R}^{n\times n}\) denotes the identity matrix of \(\mathbb{R}^{n}\)._

Using those definitions, we will parametrize each group action of every layer as a linear operator on vectors in \(\mathbb{R}^{n\cdot m}\). Any linear operator on vectors can be uniquely parametrized (up to similarity) via a matrix \(\mathbf{A}\in\mathbb{R}^{n\cdot m\times n\cdot m}\); then, the group action acting on each filter set \(k\) at every layer \(l\), \(g_{(k,l)}\), will be instantiated via the map

\[\phi_{\mathbf{A}}: \mathbb{R}^{n\times m}\to\mathbb{R}^{n\times m} \tag{8}\] \[\mathbf{X}\mapsto\operatorname{vec}^{-1}(\mathbf{A}\operatorname{vec}( \mathbf{X})).\]

Using the formulation of (8), and denoting consecutive compositions with the same function via \(f^{n}=f\circ f^{n-1}\) with \(f\circ f=f^{2}\), we can finally express the weights of each filter set as a function of the group action \(\mathbf{A}_{(k,l)}\in\mathbb{R}^{n\cdot m\times n\cdot m}\)

\[\mathbf{W}_{l_{k}}=[\mathbf{W}_{l}^{k}\quad\phi_{\mathbf{A}_{(k,l)}}(\mathbf{W}_{l}^{k})\quad \dots\phi_{\mathbf{A}_{(k,l)}}^{p-1}(\mathbf{W}_{l}^{k})]. \tag{9}\]

While simple in its inception, the operator of (8) is expressive and can model _all_ linear operators in its domain. Note that \(\operatorname{dom}(\phi_{\mathbf{A}_{(k,l)}})=\mathbb{R}^{n\times m}\) which is different from the domain of \(\mathbf{A}_{(k,l)}\) (\(=\mathbb{R}^{n\cdot m\times n\cdot m}\)) when viewed as an operator (which is trivially linear with respect to its domain).

**Proposition 2** (Linearity).: _Let \(\phi_{\mathbf{A}_{(k,l)}}\) be the map defined in (8); \(\phi_{\mathbf{A}_{(k,l)}}\) is linear. Moreover, any linear map \(\phi:\mathbb{R}^{n\times m}\to\mathbb{R}^{n\times m}\) can be parametrized by \(\phi_{\mathbf{A}_{(k,l)}}\)._

Proof.: \(\phi_{\mathbf{A}_{(k,l)}}\) is a composition of linear maps and therefore is itself linear. Indeed, \(\phi_{\mathbf{A}_{(k,l)}}\) comprises the composition of \(\operatorname{vec}\), matrix multiplication, and \(\operatorname{vec}^{-1}\). Following (6) and (7) in Definition 1, both operators are linear as compositions of linear operations, and therefore \(\phi_{\mathbf{A}_{(k,l)}}\) is itself linear.

For the second part, consider the vector spaces \(\mathbb{R}^{n\times m}\) and \(\mathbb{R}^{n\cdot m}\). The linear map \(\operatorname{vec}\) is an isomorphism from \(\mathbb{R}^{n\times m}\) to \(\mathbb{R}^{n\cdot m}\) since

* \(\operatorname{vec}(\alpha\mathbf{W})=\alpha\operatorname{vec}(\mathbf{W})\), for any \(\alpha\in\mathbb{R},\mathbf{W}\in\mathbb{R}^{n\times m}\), and
* \(\operatorname{vec}(\mathbf{W}+\mathbf{V})=\operatorname{vec}(\mathbf{W})+\operatorname {vec}(\mathbf{V})\), for any \(\mathbf{W},\mathbf{V}\in\mathbb{R}^{n\times m}\),

with the inverse map \(\operatorname{vec}^{-1}\). Since the spaces are isomorphic, parametrizing the a linear map in \(\mathbb{R}^{n\times m}\) reduces to parametrizing a linear map in \(\mathbb{R}^{n\cdot m}\). However, all linear transformations in \(\mathbb{R}^{n\cdot m}\) can be expressed by matrices \(\mathbf{A}\in\mathbb{R}^{n\cdot m\times n\cdot m1}\), which is precisely the parametrization of \(\phi_{\mathbf{A}_{(k,l)}}\). Therefore, any linear map \(\phi\) can be parametrized by \(\phi_{\mathbf{A}_{(k,l)}}\). 

**Architecture.** Composing the contents of this section, we apply our method to an unfolded network. Our building block consists of a _cyclical group layer_, a convolutional layer which utilizes unfolding

\[\mathbf{z}^{(l+1)}=\mathcal{S}_{\lambda}\left(\mathbf{z}^{(l)}+\alpha\mathbf{W}_{l}^{T}*( \mathbf{x}-\mathbf{W}_{l}*\mathbf{z}^{(l)})\right), \tag{10}\]

where \(*\) denotes convolution (correlation) and the weights of each layer \(l\) have \(K\) groups of filter sets such that

\[\mathbf{W}_{l}=[\mathbf{W}_{l_{1}}\quad\mathbf{W}_{l_{2}}\quad\dots\quad\mathbf{W}_{l_{K}}], \tag{11}\]

with \(\mathbf{W}_{l_{k}}\) being defined by (9).
### Invertibility loss

To train the architecture, we consider the loss function best applicable to the downstream task of interest, which would result in the simultaneous, unsupervised learning of the basis filters \(\mathbf{W}_{l}^{k}\) and the group action \(\mathbf{A}_{(k,l)}\) via backpropagation. However, without any regularization, \(\mathbf{A}_{(k,l)}\) are unlikely to be elements of the linear group \(\mathrm{GL}_{n\cdot m}(\mathbb{R})\). The group membership is defining in our work, as otherwise \(\mathbf{A}_{(k,l)}\) do not define a cyclic group. To that end, we introduce an _invertibility loss_. This loss encourages the elements of the cyclic groups to be invertible and thus are elements of \(\mathrm{GL}_{n\cdot m}(\mathbb{R})\):

\[L=\mu\|\mathbf{A}_{(k,l)}\widetilde{\mathbf{A}}_{(k,l)}-\mathbf{I}\|_{F}, \tag{12}\]

where \(\widetilde{\mathbf{A}}_{(k,l)}\) are matrices only used in training to encourage invertibility and \(\mu\) is the a regularization parameter controlling the tradeoff between the performance on the downstream task and the enforcement of the invertibility. More discussions about the invertibility loss, and alternatives, can be found in Appendix B.

## 5 Experiments

We used the architecture introduced in Section 4 in order to learn filter sets governed by different group actions in order to uncover latent symmetries in natural datasets. For our experimental setting, we learned basis filters \(\mathbf{W}_{l}^{k}\in\mathbb{R}^{6\times 6}\). Our architecture consists of \(L=4\) layers, each having \(K=5\) cyclic groups of \(p=4\) elements each. For complete information about hyperparameter values, datasets, training procedures, and the architecture see Appendix C.

### Recovered group structures

We trained an unfolded network using our method introduced in Section 4 on the CIFAR10 dataset for the task of classification and learned the group actions \(\mathbf{A}_{(k,l)}\) and the basis filters \(\mathbf{W}_{l}^{k}\). During our experiments our networks learned multiple interesting group actions; we note three main structures that are of interest and we present them in Figure 1.

**Skew-symmetric structure.** Observing the first column of Figure 1 we notice a skew-symmetric structure. Considering a single row of these matrices, most of them implement a _causal averaging_: the value of each pixel is updated according to a weighted sum of the pixels following it. This is an interesting emerging structure as it corresponds broadly to two very well known operations: _filtering_ (or smoothing) from Computer Vision and _average pooling_ in deep learning.

Figure 1: Emerging group structures when learning group actions for classification on CIFAR10.
**Toeplitz structure.** In the second column, the group actions have a Toeplitz (or approximately circulant) structure. Circulant matrices are of particular interest because of their ties to convolution, indicating a scheme of weight sharing or permutation operation being performed. Moreover, they are diagonalized by the Discrete Fourier Transform, which we explore in Appendix D.

**Multi-scale structure.** Finally, the group actions in the third column have a multi-scale structure. Indeed, examining the recovered matrices at the quadrant level, we observe each quadrant having a dominant sign signature, either positive or negative (for example, in both figures, the lower left quadrant contains mostly positive values). However, "zooming" into specific blocks that describe the relation between rows and columns of the original filter (\(6\times 6\) blocks) we observe significantly different structures, both within quadrants (for example, the last and third to last block in the first "block-row" of the bottom figure) and across quadrants (the first and last blocks in the same "block-row").

#### 5.1.1 Effect of the group actions

To gain better intuition and evaluate our analysis of the recovered structures, we applied the group actions to \(\mathbf{I}_{6}\in\mathbb{R}^{6\times 6}\), the identity matrix of \(\mathbb{R}^{6}\). Due to its simple structure, it allows us to understand the effect of the linear operations defined by \(\mathbf{A}_{(k,l)}\). Note that this is not trivial, as \(\mathbf{A}_{(k,l)}\) acts on the vectorization of \(\mathbf{I}_{6}\). We applied the group actions of the bottom row of Figure 1 on \(\mathbf{I}_{6}\) and we visualize the effects of the group actions in Figure 2.

The effect of the skew-symmetric operator aligns with our intuition from Section 5.1: the earlier pixels have higher intensities compared to the later ones. This is expected, as the average is over more pixels in the upper rows of the relevant group actions. This translates to more elements of the diagonal of \(\mathbf{I}_{6}\) being included in the average, resulting in this gradient-like transformation on the input. For the Toeplitz structure, the diagonal has been slightly rotated towards the lower-left part of the matrix and has been significantly smoothed out. As we will further argue in Section 5.2.1, this behavior is not surprising as the structure of the group actions resembles a composition of known operators. Finally, the multi-scale structure is harder to interpret; however, we note that considering the pixels above the antidiagonal we observe significantly lower values compared to the pixels below the antidiagonal. Examining the upper or lower parts in a different scale, we see different structures (positive values in the upper part and negative values in the lower part), in line with the multi-scale interpretation.

### Group structures in the wild

To further interpret the recovered groups of Section 5.1, we created synthetic experiments where we control the group action that is being performed. To that end, we considered the CIFAR10 dataset and we designed the following experiment

* For every image \(i\) in the dataset, we extract a random patch \(\mathbf{x}_{p_{i}}\) of size \(6\times 6\).
* We apply a transformation on the extracted patch to get \(\mathbf{y}_{p_{i}}\) and consider the tuple \((\mathbf{x}_{p_{i}},\mathbf{y}_{p_{i}})\) as a training point. The transformations we apply are rotations over a fixed angle and average pooling of a fixed radius2. Footnote 2: We used a variation Ross Wightman’s implementation of median filtering for PyTorch.
* We train a single layer linear network with no nonlinearity such that \(\mathbf{\hat{y}}=\mathbf{A}\mathbf{x}\) and train using the MSE loss.

The above experimental setup simulates the application of the group action on the filter groups. By successfully learning \(\mathbf{A}\) in this sterile setting where we control the action between successive elements we can interpret the recovered actions of Section 5.1.

Figure 2: Effect of the group actions of Figure 1 on identity matrices.
The results are presented in Figures 3 and 4 for average pooling and rotating the patches, respectively. We observe that the learned matrices have the "expected" structure: rotation matrices of \(90^{\circ}\) have a familiar grid structure, and the structure of the interpolations aligns with that reported, for example, in (Dehmamy et al., 2021). For average pooling, we again see a diagonal structure that respects the interactions between the pixels. However, we see that both structures differ from the exact observed structure of Figure 0(b), which we discuss below.

#### 5.2.1 Compositions of actions

While average pooling and rotations might not directly translate to the structures of Figure 1, we see that they do have some correlation. This lead us to design another experiment, where we consider a composition of pooling and rotation3. The resulting actions can be seen in Figure 5. We observe that by composing the two operations, the learned group action has the blocks of median filtering at the location grid defined by the rotation action. Reinterpreting the learned actions of Figure 1 under this new knee, we argue that the Toeplitz structure that we uncovered interpolates rotations and simultaneously applies pooling.

Figure 4: Learned group actions when rotating patches using an angle \(\theta\in\{30^{\circ},45^{\circ},60^{\circ},90^{\circ}\}\).

Figure 5: Learned group actions when composing rotations and average pooling using a radius \(r\in\{4,5,6\}\) (with a fixed angle \(\theta=60^{\circ}\)).

Figure 3: Learned group actions when performing average pooling using a radius \(r\in\{3,4,5,6\}\).
### Dependence on data distribution and task

One of our theses was that the element of \(\mathrm{GL}_{n\cdot m}(\mathbb{R})\) that our method learns for each group depends on both the data distribution and the downstream task. We evaluate these hypotheses below.

#### 5.3.1 Group actions on MNIST

To evaluate the dependence of the group actions on the data distribution, we applied our method on a classification task on MNIST, an image dataset consisting of handwritten digits. In Figure 6 we recovered the same structures as we did on CIFAR10, indicating that data coming from the same distribution (in this case, natural images) result in the same group actions.

#### 5.3.2 Reconstruction

Finally, to test our hypothesis that group actions depend on the downstream task, we evaluated our method on the task of reconstructing the input. We present learned actions in Figure 7 for both CIFAR10 and MNIST. We find that the actions for both datasets bear similarity, indicating further the dependence on the data distribution, and the emerging structures are distinctly different: contrary to actions for classification, actions for reconstruction exhibit multiple concentrated blocks.

## 6 Conclusion

We proposed a method, Linear Group Networks (LGNs) for learning linear groups acting on the weights of neural networks. We showed that matrices in the weight space are unable to learn interesting group actions and propose a formulation where the groups act on the lifted space of the vectorized weights, providing theoretical guarantees. In our experiments, we applied our framework on datasets of natural images for different downstream tasks. We discover several interesting groups whose actions are _skew-symmetric_, _Toeplitz_, or _multi-scale_. We drew analogues between these actions and well-known operations in machine learning, such as average pooling and planar rotations. Our work provides a simple, minimal, and extensible framework for symmetry learning by considering linear groups in the weight space. Importantly, we addressed limitations of prior work, namely the inability of existing architectures to learn group structure from single-task data, and our framework is compatible with existing architectures and pipelines with minimal alterations.

Figure 6: Group actions learned on MNIST for classification.

Figure 7: Group actions learned on CIFAR and MNIST when trained for reconstruction.