Joint Group Invariant Functions on Data-Parameter Domain Induce
  Universal Neural Networks
The symmetry and geometry of input data are considered to be encoded in the
internal data representation inside the neural network, but the specific
encoding rule has been less investigated. In this study, we present a
systematic method to induce a generalized neural network and its right inverse
operator, called the ridgelet transform, from a joint group invariant function
on the data-parameter domain. Since the ridgelet transform is an inverse, (1)
it can describe the arrangement of parameters for the network to represent a
target function, which is understood as the encoding rule, and (2) it implies
the universality of the network. Based on the group representation theory, we
present a new simple proof of the universality by using Schur's lemma in a
unified manner covering a wide class of networks, for example, the original
ridgelet transform, formal deep networks, and the dual voice transform. Since
traditional universality theorems were demonstrated based on functional
analysis, this study sheds light on the group theoretic aspect of the
approximation theory, connecting geometric deep learning to abstract harmonic
analysis.
Sho Sonoda, Hideyuki Ishi, Isao Ishikawa, Masahiro Ikeda
# Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networks

###### Abstract

The symmetry and geometry of input data are considered to be encoded in the internal data representation inside the neural network, but the specific encoding rule has been less investigated. In this study, we present a systematic method to induce a generalized neural network and its right inverse operator, called the _ridgelet transform_, from a _joint group invariant function_ on the data-parameter domain. Since the ridgelet transform is an inverse, (1) it can describe the arrangement of parameters for the network to represent a target function, which is understood as the _encoding rule_, and (2) it implies the _universality_ of the network. Based on the group representation theory, we present a new simple proof of the universality by using Schur's lemma in a unified manner covering a wide class of networks, for example, the original ridgelet transform, formal _deep_ networks, and the dual voice transform. Since traditional universality theorems were demonstrated based on functional analysis, this study sheds light on the group theoretic aspect of the approximation theory, connecting geometric deep learning to abstract harmonic analysis.

Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networks

1]Sho Sonoda1

Hideyuki Ishi2

Isao Ishikawa3,1

1]Isaohsonoda@riken.jp

1]Hideyuki Ishi1

1]Hideyuki Ishi@gmu.ac.jp

1]Iasio Ishikawa.isao.zx@ehime-u.ac.jp

1]Masahiro Ikeda1

1]masahiro.ikeda@riken.jp

1]Osaka Central Advanced Mathematical Institute (OCAMI), Osaka Metropolitan University

1]Center for Data Science, Ehime University

[2]Sophia Sanborn, Christian Shewmake, Simone Azeglio, Nina Miolane

## 1 Introduction

The internal data representation of neural networks is expected to reflect the symmetry and geometry of the data domain. In geometric deep learning (Bronstein et al., 2021), several authors have developed novel network architectures that are compatible with the geometric structure of the data (e.g. group equivariant networks). However, these methods typically require handcrafting the network architecture for each specific symmetry and geometry. In this study, we present a systematic method to induce a generalized neural network and its right inverse operator, called the _ridgelet transform_, from a _joint group invariant function_ on the data-parameter domain. Since the ridgelet transform is an inverse, (1) it explicitly describes the arrangement of parameters for the network to represent a target function, and (2) it implies the _universality_ of the network.

**Remark 1**: _Our reviewers have kindly let us know that Cohen et al. (2019), Finzi et al. (2021), and Aslan et al. (2023) have proposed versatile group equivariant network architectures that cover a wide class of groups in a unified manner, and Ravanbakhsh et al. (2017) have investigated the symmetry in the parameters. Since our results are applicable to any network architectures, it would be interesting to find the ridgelet transform for each network._
The proof of a universality theorem contains hints for understanding the internal data processing mechanisms inside neural networks. The year 1989 was the beginning of the universality theorem and a great year, as four different proofs were presented by Cybenko (1989), Hornik et al. (1989), Funahashi (1989), and Carroll and Dickinson (1989). Among them, Cybenko's proof using Hahn-Banach and Hornik et al.'s proof using Stone-Weierstrass are existential proofs, meaning that it is not clear how to assign the parameters. On the other hand, Funahashi's proof reducing to the Fourier transform and Carroll and Dickinson's proof reducing to the Radon transform are constructive proofs, meaning that it is clear how to assign the parameters. The latter constructive methods, which reduce to integral transforms, were refined as the so-called integral representation by Barron (1993) and further culminated as the ridgelet transform discovered by Murata (1996) and Candes (1998).

The ridgelet transform, the main topic of this study, is a pseudo-inverse operator of the integral representation neural network and is a detailed analysis tool that can describe the relationship between data and parameters due to its analytical representation. In the 2000s, thanks to the efforts of Donoho and others, research on ridgelet transforms evolved into geometric multiscale analysis (GMA, see e.g. Donoho, 2002), leading to the development of various x-lets such as curvelets (Candes and Donoho, 2004), contourlet (Do and Vetterli, 2005), shearlet (Labate et al., 2005), bandelet (Pennec and Mallat, 2005), and grouplet (Mallat, 2009b). These lines of studies mainly focused on developing multidimensional wavelet transforms for image processing (i.e., 2D signals) (Starck et al., 2010; Mallat, 2009a) and gradually moved apart from neural networks.

In the 2020s, the concept of integral representations has re-emerged as tools for analyzing deep learning theories, bringing renewed attention to ridgelet transforms. Precisely, they are often referred to by different names such as overparametrization, continuous/infinite width, mean field theory (Nitanda and Suzuki, 2017; Mei et al., 2018; Rotskoff and Vanden-Eijnden, 2018; Chizat and Bach, 2018; Sirignano and Spiliopoulos, 2020), and Langevin dynamics (Suzuki, 2020). Sonoda et al. (2022b, a) have developed ridgelet transforms for various networks, such as group convolutional networks and networks on manifolds, and have shown constructive universality theorems. In these proofs, reducing the network to Fourier transforms was an essential step to find the ridgelet transforms. In this study, we can find the ridgelet transforms even when there is no clear path to reducing them to Fourier transforms, as long as we can find a group invariant function.

The theory of function expansion based on group representations is well investigated in abstract harmonic analysis (Folland, 2015). There are two main streams: one is the generalization of _Fourier transform_, which expands functions on group \(G\) as a sum/integration of multiple irreducible unitary representations (Sugiura, 1990), and the other is the generalization of _wavelet transform_ called the _voice transform_, which expands functions in representation space \(\mathcal{H}\) as a sum/integration of functions generated by a single square-integrable unitary representation (Holschneider, 1998; Berge, 2021). For example, recent studies by Miyato et al. (2022) and Koyama et al. (2023) belong to the Fourier stream, while this study belongs to the wavelet/voice stream. Yet, it is precisely a new integral transform that differs from the conventional voice transform. The generalized ridgelet transform discovered in this study was motivated by the research objective of geometrically analyzing the parameters of neural networks, and we believe it is a missing link for connecting geometric deep learning to abstract harmonic analysis.
## 2 Preliminaries

We showcase the original integral representation and the ridgelet transform, a mathematical model of depth-2 fully-connected network and its right inverse, then list a few facts in the group representation theory.

Notation.For any topological space \(X\), \(C_{\mathcal{C}}(X)\) denotes the Banach space of all compactly supported functions \(f\) on \(X\). \(\mathcal{S}(\mathbb{R}^{d})\) and \(\mathcal{S}^{\prime}(\mathbb{R}^{d})\) denote the classes of rapidly decreasing functions (or Schwartz test functions) and tempered distributions on \(\mathbb{R}^{d}\), respectively.

### Quick Introduction to Integral Representation and Ridgelet Transform

**Definition 2**: _For any measurable function \(\sigma:\mathbb{R}\to\mathbb{C}\) and Borel measure \(\gamma\) on \(\mathbb{R}^{m}\times\mathbb{R}\), put_

\[S_{\sigma}[\gamma](\mathbf{x}):=\int_{\mathbb{R}^{m}\times\mathbb{R}} \gamma(\mathbf{a},b)\sigma(\mathbf{a}\cdot\mathbf{x}-b)\mathrm{d}\mathbf{a}\mathrm{d}b,\quad \mathbf{x}\in\mathbb{R}^{m}. \tag{1}\]

_We call \(S_{\sigma}[\gamma]\) an (integral representation of) neural network, and \(\gamma\) a parameter distribution._

The integration over all the hidden parameters \((\mathbf{a},b)\in\mathbb{R}^{m}\times\mathbb{R}\) means all the neurons \(\{\mathbf{x}\mapsto\sigma(\mathbf{a}\cdot\mathbf{x}-b)\ |\ (\mathbf{a},b)\in\mathbb{R}^{m}\times \mathbb{R}\}\) are summed (or integrated, to be precise) with weight \(\gamma\), hence formally \(S_{\sigma}[\gamma]\) is understood as a continuous neural network with a single hidden layer. We note, however, when \(\gamma\) is a finite sum of point measures such as \(\gamma_{p}=\sum_{i=1}^{p}c_{i}\delta_{(\mathbf{a}_{i},b_{i})}\), then it can also reproduce a finite width network

\[S_{\sigma}[\gamma_{p}](\mathbf{x})=\sum_{i=1}^{p}c_{i}\sigma(\mathbf{a}_ {i}\cdot\mathbf{x}-b_{i}). \tag{2}\]

In other words, the integral representation is a mathmatical model of depth-2 network with _any_ width (ranging from finite to continuous).

**Definition 3**: _For any measurable functions \(\rho:\mathbb{R}\to\mathbb{C}\) and \(f:\mathbb{R}^{m}\to\mathbb{C}\), put_

\[R_{\rho}[f](\mathbf{a},b):=\int_{\mathbb{R}^{m}}f(\mathbf{x})\overline {\rho(\mathbf{a}\cdot\mathbf{x}-b)}\mathrm{d}\mathbf{x},\quad(\mathbf{a},b)\in\mathbb{R}^{m} \times\mathbb{R}. \tag{3}\]

_We call \(R_{\rho}\) a ridgelet transform._

The ridgelet transform is known to be a right-inverse operator to \(S_{\sigma}\). To be precise, the following reconstruction formula holds.

**Theorem 4** (Reconstruction Formula): _Suppose \(\sigma\) and \(\rho\) are a tempered distribution (\(\mathcal{S}^{\prime}\)) and a rapid decreasing function (\(\mathcal{S}\)) respectively. There exists a bilinear form \((\!(\sigma,\rho)\!)\) such that_

\[S_{\sigma}\circ R_{\rho}[f]=(\!(\sigma,\rho)\!)f, \tag{4}\]

_for any square integrable function \(f\in L^{2}(\mathbb{R}^{m})\). Further, the bilinear form is given by_

\[(\!(\sigma,\rho)\!)=\int_{\mathbb{R}}\sigma^{\sharp}(\omega) \overline{\rho^{\sharp}(\omega)}|\omega|^{-m}\mathrm{d}\omega \tag{5}\]

_where \(\sharp\) denotes the 1-dimensional Fourier transform._
See Sonoda et al. (2021, Theorem 6) for the proof. In particular, according to Sonoda et al. (2021, Lemma 9), for any activation function \(\sigma\), there always exists \(\rho\) satisfying \((\!(\sigma,\rho)\!)=1\). Here, \(\sigma\) being a tempered distribution means that typical activation functions are covered such as ReLU, step function, \(\tanh\), gaussian, etc... We can interpret the reconstruction formula as a universality theorem of continuous neural networks, since for any given data generating function \(f\), a network with output weight \(\gamma_{f}=R_{\rho}[f]\) reproduces \(f\) (up to factor \((\!(\sigma,\rho)\!)\)), i.e. \(S[\gamma_{f}]=f\). In other words, the ridgelet transform indicates how the network parameters should be organized so that the network represents an individual function \(f\).

In this study, we showcase a new proof of the reconstruction formula based on the group theoretic arguments, and present a systematic scheme to find the ridgelet transform for a variety of given network architecture based on the symmetry in the data-parameter domain.

### Irreducible Unitary Representation and Schur's Lemma

Let \(G\) be a locally compact group, \(\mathcal{H}\) be a nonzero Hilbert space, and \(U(\mathcal{H})\) be the group of unitary operators on \(\mathcal{H}\). For example, any finite group, discrete group, compact group, and finite-dimensional Lie group are locally compact, while an infinite-dimensional Lie group is not locally compact. A _unitary representation_\(\pi\) of \(G\) on \(\mathcal{H}\) is a group homomorphism that is continuous with respect to the strong operator topology--that is, a map \(\pi:G\to U(\mathcal{H})\) satisfying \(\pi(gh)=\pi(g)\pi(h)\) and \(\pi(g^{-1})=\pi(g)^{-1}=\pi(g)^{*}\), and for any \(\psi\in\mathcal{H}\) map \(G\ni g\mapsto\pi(g)[\psi]\in\mathcal{H}\) is continuous. Suppose \(\mathcal{M}\) is a closed subspace of \(\mathcal{H}\). \(\mathcal{M}\) is called an _invariant_ subspace when \(\pi(g)\mathcal{M}\subset\mathcal{M}\) for all \(g\in G\). Particularly, \(\pi\) is called _irreducible_ when it does not admit any nontrivial invariant subspace \(\mathcal{M}\neq\{0\}\) nor \(\mathcal{H}\).

Let \(C(\pi)\) be the set of all bounded linear operators \(T\) on Hilbert space \(\mathcal{H}\) that commutes with \(\pi\), namely \(C(\pi):=\{T\in B(\mathcal{H})\mid T\pi(g)=\pi(g)T\text{ for all }g\in G\}\).

**Lemma 5** (Schur's lemma): _A unitary representation \(\pi\) of \(G\) is irreducible iff \(C(\pi)\) only contains scalar multiples of the identity, i.e., \(C(\pi)=\{c\operatorname{Id}\mid c\in\mathbb{C}\}\) or \(\{0\}\)._

See Folland (2015, Theorem 3.5(a)) for the proof.

### Calculus on Locally Compact Group

By Haar's theorem, if \(G\) is a locally compact group, then there uniquely exist left and right invariant measures \(\mathrm{d}_{l}g\) and \(\mathrm{d}_{r}g\), satisfying for any \(s\in G\) and \(f\in C_{c}(G)\),

\[\int_{G}f(sg)\mathrm{d}_{l}g=\int_{G}f(g)\mathrm{d}_{l}g,\quad\text{and} \quad\int_{G}f(gs)\mathrm{d}_{r}g=\int_{G}f(g)\mathrm{d}_{r}g.\]

Let \(X\) be a \(G\)-space with transitive left (resp. right) \(G\)-action \(g\cdot x\) (resp. \(x\cdot g\)) for any \((g,x)\in G\times X\). Then, we can further induce the left (resp. right) invariant measure \(\mathrm{d}_{l}x\) (resp. \(\mathrm{d}_{r}x\)) so that for any \(f\in C_{c}(G)\),

\[\int_{X}f(x)\mathrm{d}_{l}x:=\int_{G}f(g\cdot o)\mathrm{d}_{l}g,\quad\text{ resp.}\quad\int_{X}f(x)\mathrm{d}_{r}x:=\int_{G}f(o\cdot g)\mathrm{d}_{r}g,\]

where \(o\in G\) is a fixed point called the origin.
## 3 Main Results

We introduce generalized neural networks and generalized ridgelet transforms induced from joint group invariant functions on data-parameter domain, and present a simple group theoretic proof of the reconstruction formula.

Let \(G\) be a locally compact group equipped with a left invariant measure \(\mathrm{d}g\). Let \(X\) and \(\Xi\) be \(G\)-spaces equipped with \(G\)-invariant measures \(\mathrm{d}x\) and \(\mathrm{d}\xi\), called the data domain and the parameter domain, respectively. Particularly, we call the product space \(X\times\Xi\) the _data-parameter_ domain (like time-frequency domain). By abusing notation, we use the same symbol \(\cdot\) for the \(G\)-actions on \(X\) and \(\Xi\) (e.g., \(g\cdot x\) and \(g\cdot\xi\)).

Let \(\pi\) and \(\widehat{\pi}\) be left-regular actions of \(G\) on \(L^{2}(X)\) and \(L^{2}(\Xi)\), respectively. Namely, for any \(g\in G,f\in L^{2}(X)\) and \(\gamma\in L^{2}(\Xi)\),

\[\pi_{g}[f](x):=f(g^{-1}\cdot x),\quad\text{and}\quad\widehat{\pi}_{g}[\gamma] (\xi):=\gamma(g^{-1}\cdot\xi). \tag{6}\]

**Definition 6** (Joint \(G\)-Invariant Function): _We say a function \(\phi\) on \(X\times\Xi\) is joint \(G\)-invariant when it satisfies for all \(g\in G\) and \((x,\xi)\in X\times\Xi\),_

\[\phi(g\cdot x,g\cdot\xi)=\phi(x,\xi). \tag{7}\]

_By \(\mathcal{A}\), we symbolize the algebra of all joint \(G\)-invariant functions._

Here, \(\mathcal{A}\) is indeed an _algebra_ because if \(\phi\) and \(\psi\) are joint \(G\)-invariant, then so are \(\phi+\psi\) and \(\phi\psi\). Namely, \(\phi,\psi\in\mathcal{A}\implies\phi+\psi,\phi\psi\in\mathcal{A}\). As visualized in Figure 1, a joint \(G\)-invariant function is constant along each \(G\)-orbit \(\{(g\cdot x,g\cdot\xi)\mid g\in G\}\). Hence finding a joint \(G\)-invariant function is not difficult.

**Definition 7** (Generalized Neural Network Induced from Invariant \(\phi\)): _For any joint invariant function \(\phi\in\mathcal{A}\) and Borel measure \(\gamma\) on \(\Xi\), put_

\[\mathtt{NN}[\gamma;\phi](x):=\int_{\Xi}\gamma(\xi)\phi(x,\xi)\mathrm{d}\xi, \quad x\in X. \tag{8}\]

_We call the integral transform \(\mathtt{NN}[\bullet;\phi]\) a \(\phi\)-transform, and each individual image \(\mathtt{NN}[\gamma;\phi]\) a \(\phi\)-network for short._
The \(\phi\)-network is an extension of the original neural network because when \(X=\mathbb{R}^{m},\Xi=\mathbb{R}^{m}\times\mathbb{R}\) and \(\phi(\mathbf{x},(\mathbf{a},b)):=\sigma(\mathbf{a}\cdot\mathbf{x}-b)\) with some activation function \(\sigma:\mathbb{R}\to\mathbb{R}\), it reduces to a fully-connected network \(\int_{\mathbb{R}^{m}\times\mathbb{R}}\gamma(\mathbf{a},b)\sigma(\mathbf{a}\cdot\mathbf{x}-b) \mathrm{d}\mathbf{a}\mathrm{d}b\).

**Definition 8** (Generalized Ridgelet Transform Induced from Invariant \(\phi\)): _For any joint invariant map \(\phi\in\mathcal{A}\) and measurable function \(f\) on \(X\), put_

\[\mathtt{R}[f;\phi](\xi):=\int_{X}f(x)\overline{\phi(x,\xi)} \mathrm{d}x,\quad\xi\in\Xi. \tag{9}\]

_We call the integral transform \(\mathtt{R}[\bullet;\phi]\) a \(\phi\)-ridgelet transform for short._

As long as the integrals are convergent, it is the dual operator of \(\phi\)-transform, since

\[\langle\gamma,\mathtt{R}[f;\phi]\rangle_{L^{2}(\Xi)}=\int_{X\times \Xi}\gamma(\xi)\phi(x,\xi)\overline{f(x)}\mathrm{d}x\mathrm{d}\xi=\langle \mathtt{NN}[\gamma;\phi],f\rangle_{L^{2}(X)}. \tag{10}\]

**Theorem 9**: _Let \(G\) be a locally compact group. For any joint invariant functions \(\phi,\psi\in\mathcal{A}\), suppose that composite \(\mathtt{NN}_{\phi}\circ\mathtt{R}_{\psi}:L^{2}(X)\to L^{2}(X)\) is bounded, and that regular representation \((\pi,L^{2}(X))\) is irreducible. Then, there exists a bilinear form \((\!(\phi,\psi)\!)\in\mathbb{C}\) such that for any function \(f\in L^{2}(X)\),_

\[\mathtt{NN}_{\phi}\circ\mathtt{R}_{\psi}[f]=(\!(\phi,\psi)\!)f. \tag{11}\]

In other words, the \(\psi\)-ridgelet transform \(\mathtt{R}_{\psi}\) is understood as a group theoretic generalization of the original ridgelet transform, as it is a right inverse operator of \(\phi\)-transform \(\mathtt{NN}_{\phi}\).

**Proof** We write \(\mathtt{NN}[\bullet;\phi]\) as \(\mathtt{NN}_{\phi}\) and \(\mathtt{R}[\bullet;\phi]\) as \(\mathtt{R}_{\phi}\) for short. By the left-invariances of \(\mathrm{d}x\) and \(\psi\), for all \(g\in G\), we have

\[\mathtt{R}_{\psi}[\pi_{g}[f]](\xi) =\int_{X}f(g^{-1}\cdot x)\overline{\psi(x,\xi)}\mathrm{d}x =\langle\pi_{g}[f],\psi(\bullet,\xi)\rangle_{L^{2}(X)}\] \[=\int_{X}f(x)\overline{\psi(g\cdot x,\xi)}\mathrm{d}x =\langle f,\pi_{g}^{*}[\psi](\bullet,\xi)\rangle_{L^{2}(X)}\] \[=\int_{X}f(x)\overline{\psi(x,g^{-1}\cdot\xi)}\mathrm{d}x =\langle f,\widehat{\pi}_{g}[\psi](\bullet,\xi)\rangle_{L^{2}(X)}\] \[=\widehat{\pi}_{g}[\mathtt{R}_{\psi}[f]](\xi). \tag{12}\]

Here, \(\pi^{*}\) denotes the dual representation of \(\pi\) with respect to \(L^{2}(X)\)-product. Similarly,

\[\mathtt{NN}_{\phi}[\widehat{\pi}_{g}[\gamma]](x) =\int_{\Xi}\gamma(g^{-1}\cdot\xi)\phi(x,\xi)\mathrm{d}\xi =\langle\widehat{\pi}_{g}[\gamma],\phi(x,\bullet)\rangle_{L^{2}( \Xi)}\] \[=\int_{\Xi}\gamma(\xi)\phi(x,g\cdot\xi)\mathrm{d}\xi =\langle\gamma,\widehat{\pi}_{g}^{*}[\phi](x,\bullet)\rangle_{L^{2} (\Xi)}\] \[=\int_{\Xi}\gamma(\xi)\phi(g^{-1}\cdot x,\xi)\mathrm{d}\xi =\langle\gamma,\pi_{g}[\phi](x,\bullet)\rangle_{L^{2}(\Xi)}\] \[=\pi_{g}[\mathtt{NN}_{\phi}[\gamma]](x). \tag{13}\]

Here, \(\widehat{\pi}^{*}\) denotes the dual representation of \(\widehat{\pi}\) with respect to \(L^{2}(\Xi)\)-product.
As a consequence, \(\mathtt{NN}_{\phi}\circ\mathtt{R}_{\psi}:L^{2}(X)\to L^{2}(X)\) commutes with \(\pi\) as below

\[\mathtt{NN}_{\phi}\circ\mathtt{R}_{\psi}\circ\pi_{g}=\mathtt{NN}_{\phi}\circ \widehat{\pi}_{g}\circ\mathtt{R}_{\psi}=\pi_{g}\circ\mathtt{NN}_{\phi}\circ \mathtt{R}_{\psi} \tag{14}\]

for all \(g\in G\). Hence by Schur's lemma (Lemma 5), there exist a constant \(C_{\phi,\psi}\in\mathbb{C}\) such that \(\mathtt{NN}_{\phi}\circ\mathtt{R}_{\psi}=C_{\phi,\psi}\operatorname{Id}_{L^{2} (X)}\). By the construction of left-hand side, \(C_{\phi,\psi}\) is bilinear in \(\phi\) and \(\psi\). \(\blacksquare\)

## 4 Examples

### Original Ridgelet Transform

This study started from a group theoretic proof of the original reconstruction formula (Theorem 4). The proof is in fact new, thought-provoking and valuable, so we leave it in Appendix A. Below is a sketch of the full proof.

**Example 1**: _Let \(G\) be the affine group \(\operatorname{Aff}(m)=GL(m)\ltimes\mathbb{R}^{m}\), \(X=\mathbb{R}^{m}\) be the data domain with \(G\)-action_

\[g\cdot\boldsymbol{x}:=L\boldsymbol{x}+\boldsymbol{t},\quad g=(L,\boldsymbol{ t})\in G,\ \boldsymbol{x}\in\mathbb{R}^{m}=X\]

_and \(\Xi=\mathbb{R}^{m}\times\mathbb{R}\) be the parameter domain with dual \(G\)-action_

\[g\cdot(\boldsymbol{a},b)=(L^{-\top}\boldsymbol{a},b+\boldsymbol{t}^{\top}L^{- \top}\boldsymbol{a}),\quad g=(L,\boldsymbol{t})\in G,\ (\boldsymbol{a},b)\in \mathbb{R}^{m}\times\mathbb{R}=\Xi. \tag{15}\]

_We can see \(\phi(\boldsymbol{x},(\boldsymbol{a},b)):=\sigma(\boldsymbol{a}\cdot \boldsymbol{x}-b)\) is joint \(G\)-invariant. In fact,_

\[\phi(g\cdot\boldsymbol{x},g\cdot(\boldsymbol{a},b))=\sigma\left(L^{-\top} \boldsymbol{a}\cdot(L\boldsymbol{x}+\boldsymbol{t})-(b+\boldsymbol{t}^{\top}L ^{-\top}\boldsymbol{a})\right)=\sigma(\boldsymbol{a}\cdot\boldsymbol{x}-b)= \phi(\boldsymbol{x},(\boldsymbol{a},b)).\]

_Further, by Lemma 12, the regular representation \(\pi_{g}\) of \(G=\operatorname{Aff}(m)\) is known to be irreducible. Hence we can retain the original neural network and ridgelet transform:_

\[\mathtt{NN}[\gamma](\boldsymbol{x})=\int_{\mathbb{R}^{m}\times\mathbb{R}} \gamma(\boldsymbol{a},b)\sigma(\boldsymbol{a}\cdot\boldsymbol{x}-b) \mathrm{d}\boldsymbol{a}\mathrm{d}b,\quad\text{and}\quad\mathtt{R}[f]( \boldsymbol{a},b)=\int_{\mathbb{R}^{m}}f(\boldsymbol{x})\overline{\rho( \boldsymbol{a}\cdot\boldsymbol{x}-b)}\mathrm{d}\boldsymbol{x},\]

_satisfying \(\mathtt{NN}\circ\mathtt{R}=(\!(\sigma,\rho)\!)\operatorname{Id}_{L^{2}( \mathbb{R}^{m})}\)._

Additionally, a geometric interpretation of dual \(G\)-action (15) is discussed in Appendix B.

### Deep Ridgelet Transform

Sonoda et al. (2023) presented the ridgelet transform for _deep_ neural networks. We noticed their network can also be induced from an invariant function. In other words, from the group representation theory perspective, function approximation with _any depth_ is unified.

**Example 2**: _Let \(G\) be any locally compact group, data domain \(X\) be any \(G\)-space, rewriting its \(G\)-action \(g\cdot x\) as \(g(x)\) so as to formally identify \(g\) with a hidden layer map, and parameter domain \(\Xi\) be the group \(G\) itself with dual \(G\)-action_

\[g\cdot\xi=\xi g^{-1}. \tag{16}\]
_We can see \(\phi(x,\xi):=\psi\circ\xi(x)\) is joint \(G\)-invariant. In fact,_

\[\phi(g\cdot x,g\cdot\xi)=\psi\circ(g\cdot\xi)(g\cdot x)=\psi\circ(\xi\circ g^{-1} )(g(x))=\psi\circ\xi(x)=\phi(x,\xi)\]

_Therefore, assuming that the regular representation \(\pi_{g}=\psi\circ g\) is irreducible on an invariant subspace \(\mathcal{H}\) of \(L^{2}(X)\), we can retain the formal deep network and deep ridgelet transform:_

\[\mathtt{NN}[\gamma](x):=\int_{\Xi}\gamma(\xi)\psi\circ\xi(x)\mathrm{d}\xi, \quad\text{and}\quad\mathtt{R}[f](\xi)=\int_{X}f(x)\overline{\psi\circ\xi(x)} \mathrm{d}x,\]

_satisfying \(\mathtt{NN}\circ\mathtt{R}=(\!(\sigma,\rho)\!)\operatorname{Id}_{\mathcal{H}}\)._

### Voice Transform, or Generalized Wavelet Transform

The voice transform is also known as the _Gilmore-Perelomov coherent states_ and the _generalized wavelet transform_(Perelomov, 1986; Ali et al., 2014). It is well investigated in the research field of _coorbit theory_(Feichtinger and Grochenig, 1988, 1989a,b). We refer to Berge (2021) for a quick review of voice transform and coorbit theory.

**Definition 10**: _Given a unitary representation \((\pi,\mathcal{H})\) of group \(G\) on a Hilbert space \(\mathcal{H}\), the voice transform is defined as_

\[V_{\phi}[f](g):=\langle f,\pi_{g}[\phi]\rangle_{\mathcal{H}},\quad g\in G,\ f, \phi\in\mathcal{H}. \tag{17}\]

This unifies several integral transforms from the perspective of group theory such as short-time Fourier transform (STFT), wavelet transform (Grossmann et al., 1985, 1986; Holschneider, 1998; Laugesen et al., 2002; Gressman et al., 2003), and continuous shearlet transform (Labate et al., 2005; Guo and Labate, 2007; Kutyniok and Labate, 2012).

**Example 3**: _Let \(G\) be any group, data domain \(X\) be any \(G\)-space, and parameter domain \(\Xi\) be the group \(G\) itself with dual \(G\)-action \(g\cdot\xi=g\xi\). We can see \(\theta(x,\xi):=\psi(\xi^{-1}\cdot x)\) is joint \(G\)-invariant. In fact,_

\[\theta(g\cdot x,g\cdot\xi)=\psi((g\cdot\xi)^{-1}\cdot(g\cdot x))=\psi(\xi^{-1 }\cdot x)=\theta(x,\xi).\]

_Therefore, assuming that the regular representation \(\pi_{g}\) is irreducible, we can retain a dual voice transform and voice transform:_

\[\mathtt{NN}[\gamma](x):=\int_{\Xi}\gamma(\xi)\phi(\xi^{-1}\cdot x)\mathrm{d} \xi,\quad\text{and}\quad\mathtt{R}[f](\xi)=\int_{X}f(x)\psi(\xi^{-1}\cdot x) \mathrm{d}x,\]

_satisfying \(\mathtt{NN}\circ\mathtt{R}=(\!(\sigma,\rho)\!)\operatorname{Id}_{L^{2}(X)}\). This is a special case of the voice transform when \(\mathcal{H}=L^{2}(X)\), and \(\pi_{g}[\psi]=\psi(g^{-1}\cdot\bullet)\)._

We note that the voice transform \(V_{\phi}[f](g):=\langle f,\pi_{g}[\phi]\rangle_{\mathcal{H}}\) and the \(\phi\)-ridgelet transform \(\mathtt{R}_{\phi}[f](\xi):=\langle f,\phi(\bullet,\xi)\rangle_{L^{2}(X)}\) have common parts, but are different in general. While the example above and the original wavelet transform \(W_{\psi}[f](b,a):=\int_{\mathbb{R}}f(x)\psi((x-b)/a)\mathrm{d}x/\sqrt{a}\) are simultaneously the voice and ridgelet transforms, a ridgelet transform can be a voice transform only when the representation \((\pi,\mathcal{H})\) is the regular representation on \(L^{2}(X)\), and a voice transform can be a ridgelet transform only when the parameter domain \(\Xi\) is the group \(G\) itself and the feature map \(\phi\) is generated by \(G\)-action on a single function \(\psi\). Hence pursuing parallel results for the coorbit theory would be an interesting future work.
## 5 Discussion

We presented a systematic method to induce a generalized neural network and its ridgelet transform, from a joint group invariant function on the data-parameter domain. Namely, given a joint group invariant function, the marginalization of parameter \(\xi\) (resp. data \(x\)) induces the network (resp. the ridgelet transform). Based on the group theoretic arguments, we demonstrated a simple proof of the reconstruction formula by using Schur's lemma, which implies the universality of the network. Since conventional universality theorems were shown using functional analytic tools, the group theoretic proof is a new contribution to the approximation theory, connecting geometric deep learning to abstract harmonic analysis. Further, since the proposed network covers both shallow and deep networks, the group representation theory can offer a unified perspective on function approximation with _any depth_.

In the past, Sonoda et al. (2022a,b) have developed the ridgelet transforms for neural networks on manifolds and function spaces using the Fourier transforms on manifolds and function spaces, and proposed a systematic scheme to derive a ridgelet transform for neural networks on a given domain based on the Fourier transform on there. Compared to our group theoretic method, the Fourier transform method is indirect and requires additional knowledge (not only on the symmetry on the data domain but also) on the Fourier transform on there. We conjecture that those Fourier-based ridgelet transforms can also be derived in our group-theoretic method.

The authors are extremely grateful to the three anonymous reviewers for their valuable comments and suggestions, which have helped improve the quality of our manuscript. This work was supported by JSPS KAKENHI 20K03657, JST PRESTO JPMJPR2125, JST CREST JPMJCR2015 and JPMJCR1913, and JST ACTX JPMJAX2004.
