# Constructive Universal Approximation Theorems for
Deep Joint-Equivariant Networks by Schur’s Lemma

Sho Sonoda1<EMAIL_ADDRESS>
Yuka Hashimoto2,1<EMAIL_ADDRESS>
Isao Ishikawa3,1<EMAIL_ADDRESS>
Masahiro Ikeda1<EMAIL_ADDRESS>
1Center for Advanced Intelligence Project (AIP), RIKEN a
2NTT Corporation a
3Center for Data Science, Ehime University a
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

(May 21, 2024)

###### Abstract

We present a unified constructive universal approximation theorem covering a
wide range of learning machines including both shallow and deep neural
networks based on the group representation theory. Constructive here means
that the distribution of parameters is given in a closed-form expression
(called the _ridgelet transform_). Contrary to the case of shallow models,
expressive power analysis of deep models has been conducted in a case-by-case
manner. Recently, Sonoda et al. (2023b, a) developed a systematic method to
show a constructive approximation theorem from _scalar-valued joint-group-
invariant_ feature maps, covering a formal deep network. However, each hidden
layer was formalized as an abstract group action, so it was not possible to
cover real deep networks defined by composites of nonlinear activation
function. In this study, we extend the method for _vector-valued joint-group-
equivariant_ feature maps, so to cover such real networks.

##  1 Introduction

An ultimate goal of the deep learning theory is to characterize the internal
data processing procedure inside deep neural networks obtained by deep
learning. We may formulate this problem as a functional equation problem: Let
$\mathcal{F}$ denote a class of data generating functions, and let
$\mathtt{DNN}[\gamma]$ denote a certain deep neural network with parameter
$\gamma$. Given a function $f\in\mathcal{F}$, find an unknown parameter
$\gamma$ so that network $\mathtt{DNN}[\gamma]$ represents function $f$, i.e.

$\displaystyle\mathtt{DNN}[\gamma]=f,$ (1)

which we call a _DNN equation_. An ordinary learning problem by empirical risk
minimization, such as minimizing
$\sum_{i=1}^{n}|\mathtt{DNN}[\gamma](x_{i})-f(x_{i})|^{2}$ with respect to
$\gamma$, is understood as a weak form (or a variational form) of this
equation. Therefore, characterizing the solution space of this equation leads
to understanding the parameters obtained by deep learning. Following previous
studies (Murata, 1996; Candès, 1998; Sonoda et al., 2021a, b, 2022a, 2022b),
we call a solution operator $\mathtt{R}$ that satisfies
$\mathtt{DNN}[\mathtt{R}[f]]=f$ a _ridgelet transform_. Once such a solution
operator $\mathtt{R}$ is found, we can conclude a _universality_ of the DNN in
consideration because the reconstruction formula
$\mathtt{DNN}[\mathtt{R}[f]]=f$ implies for any $f\in\mathcal{F}$ there exists
a DNN that represents $f$. In particular, when $\mathtt{R}[f]$ is found in a
closed-form manner, then it leads to a _constructive_ proof of the
universality since $\mathtt{R}[f]$ could indicate how to assign parameters.

When the network has only one infinitely-wide hidden layer, though it is not
deep but shallow, the characterization problem has been well investigated. For
example, the learning dynamics and the global convergence property (of SGD)
are well studied in the mean field theory (Nitanda and Suzuki, 2017; Rotskoff
and Vanden-Eijnden, 2018; Mei et al., 2018; Chizat and Bach, 2018) and the
Langevin dynamics theory (Suzuki, 2020), and even closed-form solution
operator to a “shallow” NN equation, the original ridgelet transform, has
already been presented (Sonoda et al., 2021a, b, 2022a, 2022b).

On the other hand, when the network has more than one hidden layer, the
problem is far from solved, and it is common to either consider infinitely-
deep mathematical models such as Neural ODEs (Sonoda and Murata, 2017; E,
2017; Li and Hao, 2018; Haber and Ruthotto, 2017; Chen et al., 2018), or
handcraft inner feature maps depending on the problem. For example,
construction methods such as the Telgarsky sawtooth function (or the Yarotsky
scheme) and bit extraction techniques (Cohen et al., 2016; Telgarsky, 2016;
Yarotsky, 2017, 2018; Yarotsky and Zhevnerchuk, 2020; Daubechies et al., 2022;
Cohen et al., 2022; Siegel, 2023; Petrova and Wojtaszczyk, 2023; Grohs et al.,
2023) have been developed to demonstrate the depth separation, super-
convergence, and minmax optimality of deep ReLU networks. Various feature maps
have also been handcrafted in the contexts of geometric deep learning
(Bronstein et al., 2021) and deep narrow networks (Lu et al., 2017; Hanin and
Sellke, 2017; Lin and Jegelka, 2018; Kidger and Lyons, 2020; Park et al.,
2021; Li et al., 2023; Cai, 2023; Kim et al., 2024). Needless to say, there is
no guarantee that these handcrafted feature maps are acquired by deep
learning, so these analyses are considered to be analyses of possible worlds.

Recently, Sonoda et al. (2023b, a) discovered a rich class of ridgelet
transforms for learning machines defined by _scalar-valued joint-group-
invariant_ feature maps, covering both depth-2 fully-connected networks and
the formal deep network (FDN), yielding the first ridgelet transform for deep
models. Their theory is indeed a breakthrough because it could cover both deep
and shallow models simultaneously. However, each hidden layer in the FDN has
to be formalized as an abstract group action, so it was not possible to cover
real deep networks defined by composites of nonlinear activation function. In
this study, we extend their arguments for _vector-valued joint-group-
equivariant_ feature maps (Theorem 3 and Corollary 1), so to cover such real
networks. As an important example, in § 4.2, we obtained the ridgelet
transform for a more realistic DNN, the depth-$n$ fully-connected network with
an arbitrary activation function (not limited to ReLU), without handcrafting
network architecture. In other words, it is a constructive proof of the
$L^{2}(\mathbb{R}^{m};\mathbb{R}^{m})$-universality of the DNNs, and an
explicit characterization of the solution space of the DNN equation for more
realistic setup.

Thanks to Schur’s lemma, a basic and useful result in the representation
theory, the proof of the main theorem is surprisingly simple, yet the scope of
application is wide. The significance of this study lies in revealing the
close relationship between machine learning theory and modern algebra. With
this study as a catalyst, we expect a major upgrade to machine learning theory
from the perspective of modern algebra.

##  2 Preliminaries

We quickly introduce the original integral representation and the ridgelet
transform, a mathematical model of depth-2 fully-connected network and its
right inverse. Then, we list a few facts in the group representation theory.
In particular, _Schur’s lemma_ and the _Haar measure_ play key roles in the
proof of the main results.

##### Notation.

For any topological space $X$, $C_{c}(X)$ denotes the Banach space of all
compactly supported continuous functions on $X$. For any measure space $X$,
$L^{p}(X)$ denotes the Banach space of all $p$-integrable functions on $X$.
$\mathcal{S}(\mathbb{R}^{d})$ and $\mathcal{S}^{\prime}(\mathbb{R}^{d})$
denote the classes of rapidly decreasing functions (or Schwartz test
functions) and tempered distributions on $\mathbb{R}^{d}$, respectively.

###  2.1 Integral Representation and Ridgelet Transform for Depth-2 Fully-
Connected Network

######  Definition 1.

For any measurable functions $\sigma:\mathbb{R}\to\mathbb{C}$ and
$\gamma:\mathbb{R}^{m}\times\mathbb{R}\to\mathbb{C}$, put

$\displaystyle
S_{\sigma}[\gamma](\bm{x}):=\int_{\mathbb{R}^{m}\times\mathbb{R}}\gamma(\bm{a},b)\sigma(\bm{a}\cdot\bm{x}-b)\mathrm{d}\bm{a}\mathrm{d}b,\quad\bm{x}\in\mathbb{R}^{m}.$
(2)

We call $S_{\sigma}[\gamma]$ an (integral representation of) neural network,
and $\gamma$ a parameter distribution.

The integration over all the hidden parameters
$(\bm{a},b)\in\mathbb{R}^{m}\times\mathbb{R}$ means all the neurons
$\\{\bm{x}\mapsto\sigma(\bm{a}\cdot\bm{x}-b)\mid(\bm{a},b)\in\mathbb{R}^{m}\times\mathbb{R}\\}$
are summed (or integrated, to be precise) with weight $\gamma$, hence formally
$S_{\sigma}[\gamma]$ is understood as a continuous neural network with a
single hidden layer. We note, however, when $\gamma$ is a finite sum of point
measures such as $\gamma_{p}=\sum_{i=1}^{p}c_{i}\delta_{(\bm{a}_{i},b_{i})}$
(by appropriately extending the class of $\gamma$ to Borel measures), then it
can also reproduce a finite width network

$S_{\sigma}[\gamma_{p}](\bm{x})=\sum_{i=1}^{p}c_{i}\sigma(\bm{a}_{i}\cdot\bm{x}-b_{i}).$
(3)

In other words, the integral representation is a mathmatical model of depth-2
network with _any_ width (ranging from finite to continuous).

Next, we introduce the ridgelet transform, which is known to be a right-
inverse operator to $S_{\sigma}$.

######  Definition 2.

For any measurable functions $\rho:\mathbb{R}\to\mathbb{C}$ and
$f:\mathbb{R}^{m}\to\mathbb{C}$, put

$\displaystyle
R_{\rho}[f](\bm{a},b):=\int_{\mathbb{R}^{m}}f(\bm{x})\overline{\rho(\bm{a}\cdot\bm{x}-b)}\mathrm{d}\bm{x},\quad(\bm{a},b)\in\mathbb{R}^{m}\times\mathbb{R}.$
(4)

We call $R_{\rho}$ a ridgelet transform.

To be precise, it satisfies the following reconstruction formula.

######  Theorem 1 (Reconstruction Formula).

Suppose $\sigma$ and $\rho$ are a tempered distribution
($\mathcal{S}^{\prime}$) and a rapid decreasing function ($\mathcal{S}$)
respectively. There exists a bilinear form $(\\!(\sigma,\rho)\\!)$ such that

$\displaystyle S_{\sigma}\circ R_{\rho}[f]=(\\!(\sigma,\rho)\\!)f,$ (5)

for any square integrable function $f\in L^{2}(\mathbb{R}^{m})$. Further, the
bilinear form is given by
$(\\!(\sigma,\rho)\\!)=\int_{\mathbb{R}}\sigma^{\sharp}(\omega)\overline{\rho^{\sharp}(\omega)}|\omega|^{-m}\mathrm{d}\omega,$
where $\sharp$ denotes the 1-dimensional Fourier transform.

See Sonoda et al. (2021b, Theorem 6) for the proof. In particular, according
to Sonoda et al. (2021b, Lemma 9), for any activation function $\sigma$, there
always exists $\rho$ satisfying $(\\!(\sigma,\rho)\\!)=1$. Here, $\sigma$
being a tempered distribution means that typical activation functions are
covered such as ReLU, step function, $\tanh$, gaussian, etc… We can interpret
the reconstruction formula as a universality theorem of continuous neural
networks, since for any given data generating function $f$, a network with
output weight $\gamma_{f}=R_{\rho}[f]$ reproduces $f$ (up to factor
$(\\!(\sigma,\rho)\\!)$), i.e. $S[\gamma_{f}]=f$. In other words, the ridgelet
transform indicates how the network parameters should be organized so that the
network represents an individual function $f$.

The original ridgelet transform was discovered by Murata (1996) and Candès
(1998). It is recently extended to a few modern networks by the Fourier slice
method (Sonoda et al., 2024, see e.g.). In this study, we present a systematic
scheme to find the ridgelet transform for a variety of given network
architecture based on the group theoretic arguments.

###  2.2 Irreducible Unitary Representation and Schur’s Lemma

Let $G$ be a locally compact group, $\mathcal{H}$ be a nonzero Hilbert space,
and $\mathcal{U}(\mathcal{H})$ be the group of unitary operators on
$\mathcal{H}$. For example, any finite group, discrete group, compact group,
and finite-dimensional Lie group are locally compact, while an infinite-
dimensional Lie group is not locally compact. A _unitary representation_ $\pi$
of $G$ on $\mathcal{H}$ is a group homomorphism that is continuous with
respect to the strong operator topology—that is, a map
$\pi:G\to\mathcal{U}(\mathcal{H})$ satisfying $\pi(gh)=\pi(g)\pi(h)$ and
$\pi(g^{-1})=\pi(g)^{-1}$, and for any $\psi\in\mathcal{H}$, the map $G\ni
g\mapsto\pi(g)[\psi]\in\mathcal{H}$ is continuous.

Suppose $\mathcal{M}$ is a closed subspace of $\mathcal{H}$. $\mathcal{M}$ is
called an _invariant_ subspace when $\pi(g)\mathcal{M}\subset\mathcal{M}$ for
all $g\in G$. Particularly, $\pi$ is called _irreducible_ when it does not
admit any nontrivial invariant subspace $\mathcal{M}\neq\\{0\\}$ nor
$\mathcal{H}$. The following theorem is a fundamental result of group
representation theory that characterizes the irreducibility.

######  Theorem 2 (Schur’s lemma).

A unitary representation $(\pi,\mathcal{H})$ is irreducible iff any bounded
operator $T$ on $\mathcal{H}$ that commutes with $\pi$ is always a constant
multiple of the identity. In other words, if $\pi(g)T=T\pi(g)$ for all $g\in
G$, then $T=c\operatorname{Id}_{\mathcal{H}}$ for some $c\in\mathbb{C}$.

See Folland (2015, Theorem 3.5(a)) for the proof. We use this as a key step in
the proof of our main theorem.

###  2.3 Calculus on Locally Compact Group

By Haar’s theorem, if $G$ is a locally compact group, then there uniquely
exist left and right invariant measures $\mathrm{d}_{l}g$ and
$\mathrm{d}_{r}g$, satisfying for any $s\in G$ and $f\in C_{c}(G)$,

$\displaystyle\int_{G}f(sg)\mathrm{d}_{l}g=\int_{G}f(g)\mathrm{d}_{l}g,\quad\mbox{and}\quad\int_{G}f(gs)\mathrm{d}_{r}g=\int_{G}f(g)\mathrm{d}_{r}g.$

Let $X$ be a $G$-space with transitive left (resp. right) $G$-action $g\cdot
x$ (resp. $x\cdot g$) for any $(g,x)\in G\times X$. Then, we can further
induce the left (resp. right) invariant measure $\mathrm{d}_{l}x$ (resp.
$\mathrm{d}_{r}x$) so that for any $f\in C_{c}(G)$,

$\displaystyle\int_{X}f(x)\mathrm{d}_{l}x:=\int_{G}f(g\cdot
o)\mathrm{d}_{l}g,\quad\mbox{resp.}\quad\int_{X}f(x)\mathrm{d}_{r}x:=\int_{G}f(o\cdot
g)\mathrm{d}_{r}g,$

where $o\in X$ is a fixed point called the origin.

##  3 Main Results

We introduce the joint-group-equivariant feature map, and present the ridgelet
transforms for learning machines defined by joint-group-equivariant feature
maps, yielding the universality of deep models.

Let $G$ be a locally compact group equipped with a left invariant measure
$\mathrm{d}g$. Let $X$ and $\Xi$ be $G$-spaces equipped with $G$-invariant
measures $\mathrm{d}x$ and $\mathrm{d}\xi$, called the data domain and the
parameter domain, respectively. Particularly, we call the product space
$X\times\Xi$ the _data-parameter_ domain (like time-frequency domain), and
call any map $\phi$ on data-parameter domain $X\times\Xi$ a _feature map_. Let
$\mathcal{H}$ be a separable Hilbert space, let $\mathcal{U}(\mathcal{H})$ be
the space of unitary operators on $\mathcal{H}$, and let
$\upsilon:G\to\mathcal{U}(\mathcal{H})$ be a unitary representation of $G$ on
$\mathcal{H}$. If there is no danger of confusion, we use the same symbol
$\cdot$ for the $G$-actions on $X$, $\mathcal{H}$, and $\Xi$ (e.g., $g\cdot
x$, $g\cdot v$, and $g\cdot\xi$).

In the main theorem, the irreducibility of the following unitary
representation $\pi$ will be a sufficient condition for the universality. Let
$L^{2}(X;\mathcal{H})$ denote the space of $\mathcal{H}$-valued square-
integrable functions on $X$ equipped with the inner product
$\langle\phi,\psi\rangle_{L^{2}(X;\mathcal{H})}:=\int_{X}\langle\phi(x),\psi(x)\rangle_{\mathcal{H}}\mathrm{d}x$.
Put

$\displaystyle\pi_{g}[f](x):=g\cdot f(g^{-1}\cdot x),\quad x\in X,\ f\in
L^{2}(X;\mathcal{H}),\ g\in G.$ (6)

Then, it is a unitary representation of $G$ on $L^{2}(X;\mathcal{H})$. In
fact, $\pi_{g}[\pi_{h}[f]](x)=g\cdot h\cdot f(h^{-1}\cdot g^{-1}\cdot
x)=(gh)\cdot f((gh)^{-1}\cdot x)=\pi_{gh}[f](x)$, and
$\langle\pi_{g}[f_{1}],\pi_{g}[f_{2}]\rangle_{L^{2}(X;\mathcal{H})}=\int_{X}\langle\upsilon_{g}[f_{1}](g^{-1}\cdot
x),\upsilon_{g}[f_{2}](g^{-1}\cdot
x)\rangle_{\mathcal{H}}\mathrm{d}x=\int_{X}\langle
f_{1}(x),\upsilon_{g}^{*}[\upsilon_{g}[f_{2}]](x)\rangle_{\mathcal{H}}\mathrm{d}x=\langle
f_{1},f_{2}\rangle_{L^{2}(X;\mathcal{H})}$.

In addition, let $L^{2}(\Xi)$ denote the space of $\mathbb{C}$-valued square-
integrable functions on $\Xi$, and let $\widehat{\pi}$ be the left-regular
representation of $G$ on $L^{2}(\Xi)$ given by

$\displaystyle\widehat{\pi}_{g}[\gamma](\xi):=\gamma(g^{-1}\cdot\xi),\quad\xi\in\Xi,\
\gamma\in L^{2}(\Xi),\ g\in G.$ (7)

Similarly to $\pi$, $\widehat{\pi}$ is also a unitary representation.

######  Definition 3 (Joint $G$-Equivariant Feature Map).

Let $X,Y$ be data domains, and $\Xi$ be a parameter domain (with $G$-actions).
We say a feature map $\phi:X\times\Xi\to Y$ is _joint- $G$-equivariant_ when

$\displaystyle\phi(g\cdot x,g\cdot\xi)=g\cdot\phi(x,\xi),\quad(x,\xi)\in
X\times\Xi,$ (8)

holds for all $g\in G$. In other words, $\phi$ is a homomorphism (or $G$-map)
of $G$-sets from $X\times\Xi$ to $Y$. So by $\hom_{G}(X\times\Xi,Y)$, we
denote the collection of all joint-$G$-equivariant maps. Additionally, when
$G$-action on $Y$ is trivial, i.e. $\phi(g\cdot x,g\cdot\xi)=\phi(x,\xi)$, we
say it is _joint- $G$-invariant_.

######  Remark 1.

The joint-$G$-equivariance extends an ordinary notion of _$G$ -equivariance_,
i.e. $\phi(g\cdot x,\xi)=g\cdot\phi(x,\xi)$. In fact, $G$-equivariance is a
special case of joint-$G$-equivariance where $G$ acts trivially on parameter
domain, i.e. $g\cdot\xi=\xi$ (see also Figure 1).

Figure 1: An ordinary $G$-equivariant feature map $\phi:X\times\Xi\to Y$ is a
subclass of joint-$G$-equivariant map where the $G$-action on parameter domain
$\Xi$ is _trivial_ , i.e. $g\cdot\xi=\xi$

In order to construct a (non-joint) group-equivariant network, we must
carefully and precisely design the network architecture (see, e.g., a textbook
of geometric deep learning Bronstein et al., 2021). On the other hand, we can
easily and systematically construct joint-$G$-equivariant network from (not at
all equivariant but) _any_ map $f:X\to Y$ according to the following Lemmas 1
and 2.

######  Lemma 1.

Suppose group $G$ acts on sets $X$ and $Y$. Fix an arbitrary map $f:X\to Y$,
and put $\phi(x,g):=g\cdot f(g^{-1}\cdot x)$ for every $x\in X$ and $g\in G$.
Then, $\phi:X\times G\to Y$ is joint-$G$-equivariant.

###### Proof.

Straightforward. For any $g\in G$, $\phi(g\cdot x,g\cdot h)=(gh)\cdot
f((gh)^{-1}\cdot(g\cdot x))=g\cdot\phi(x,h)$. ∎

######  Lemma 2 (Depth-$n$ Feature Map $\phi_{1:n}$).

Given a sequence of $G$-equivariant feature maps
$\phi_{i}:X_{i-1}\times\Xi_{i}\to X_{i}\ (i=1,\ldots,n)$, put
$\phi_{1:n}:X_{0}\times\Xi_{1}\times\cdots\times\Xi_{n}\to X_{n}$ by

$\displaystyle\phi_{1:n}(x,\xi_{1},\ldots,\xi_{n}):=\phi_{n}(\bullet,\xi_{n})\circ\cdots\circ\phi_{1}(x,\xi_{1}).$
(9)

Then, $\phi_{1:n}$ is $G$-equivariant. Following the custom of counting the
number of parameter domains $(\Xi_{i})_{i=1}^{n}$, we say $\phi_{1:n}$ is
depth-$n$.

###### Proof.

In fact,

$\displaystyle\phi_{1:n}(g\cdot x,g\cdot\xi_{1},\ldots,g\cdot\xi_{n})$
$\displaystyle=\phi_{n}(\bullet,g\cdot\xi_{n})\circ\cdots\circ\phi_{2}(\bullet,g\cdot\xi_{2})\circ\phi_{1}(g\cdot
x,g\cdot\xi_{1})$
$\displaystyle=\phi_{n}(\bullet,g\cdot\xi_{n})\circ\cdots\circ\phi_{2}({\color[rgb]{0,0,1}g{\
\cdot\ }}\bullet,g\cdot\xi_{2})\circ\phi_{1}(x,\xi_{1})$
$\displaystyle\phantom{==}\vdots$
$\displaystyle=\phi_{n}({\color[rgb]{0,0,1}g{\ \cdot\
}}\bullet,g\cdot\xi_{n})\circ\cdots\circ\phi_{2}(\bullet,\xi_{2})\circ\phi_{1}(x,\xi_{1})$
$\displaystyle={\color[rgb]{0,0,1}g{\ \cdot\
}}\phi_{n}(\bullet,\xi_{n})\circ\cdots\circ\phi_{2}(\bullet,\xi_{2})\circ\phi_{1}(x,\xi_{1})$
$\displaystyle=g\cdot\phi_{1:n}(x,\xi_{1},\ldots,\xi_{n}).\qed$

######  Definition 4 ($\phi$-Network).

For any vector-valued map $\phi:X\times\Xi\to\mathcal{H}$ and scalar-valued
map $\gamma:\Xi\to\mathbb{C}$, define a vector-valued map $X\to\mathcal{H}$ by

$\displaystyle\mathtt{NN}[\gamma;\phi](x):=\int_{\Xi}\gamma(\xi)\phi(x,\xi)\mathrm{d}\xi,\quad
x\in X,$ (10)

where the integral is understood as the Bocher integral.

We call the integral transform $\mathtt{NN}[\bullet;\phi]$ a $\phi$-transform,
and each individual image $\mathtt{NN}[\gamma;\phi]$ a $\phi$-network for
short. The $\phi$-network extends the original integral representation. In
particular, it inherits the concept of integrating all the possible parameters
$\xi$ and indirectly select which parameters to use by weighting on them,
which _linearize_ parametrization by lifting nonlinear parameters $\xi$ to
linear parameter $\gamma$.

######  Definition 5 ($\psi$-Ridgelet Transform).

For any $\mathcal{H}$-valued feature map $\psi:X\times\Xi\to\mathcal{H}$ and
$\mathcal{H}$-valued Borel measurable function $f$ on $X$, put a scalar-valued
integral transform

$\displaystyle\mathtt{R}[f;\psi](\xi):=\int_{X}\langle
f(x),\psi(x,\xi)\rangle_{\mathcal{H}}\mathrm{d}x,\quad\xi\in\Xi.$ (11)

We call the integral transform $\mathtt{R}[\bullet;\psi]$ a $\psi$-ridgelet
transform for short.

As long as the integrals are convergent, $\phi$-ridgelet transform is the dual
operator of $\phi$-transform, since

$\displaystyle\langle\gamma,\mathtt{R}[f;\phi]\rangle_{L^{2}(\Xi)}=\int_{X\times\Xi}\gamma(\xi)\langle\phi(x,\xi),f(x)\rangle_{\mathcal{H}}\mathrm{d}x\mathrm{d}\xi=\langle\mathtt{NN}[\gamma;\phi],f\rangle_{L^{2}(X;\mathcal{H})}.$
(12)

######  Theorem 3 (Reconstruction Formula).

Assume (1) $\mathcal{H}$-valued feature maps
$\phi,\psi:X\times\Xi\to\mathcal{H}$ are joint-$G$-equivariant, (2) composite
operator $\mathtt{NN}_{\phi}\circ\mathtt{R}_{\psi}:L^{2}(X;\mathcal{H})\to
L^{2}(X;\mathcal{H})$ is bounded (i.e., Lipschitz continuous), and (3) the
unitary representation $\pi$ defined in (6) is irreducible. Then, there exists
a bilinear form $(\\!(\phi,\psi)\\!)\in\mathbb{C}$ (independent of $f$) such
that for any $\mathcal{H}$-valued square-integrable function $f\in
L^{2}(X;\mathcal{H})$,

$\displaystyle\mathtt{NN}_{\phi}\circ\mathtt{R}_{\psi}[f]=(\\!(\phi,\psi)\\!)f.$
(13)

In other words, the $\psi$-ridgelet transform $\mathtt{R}_{\psi}$ is a right
inverse operator of $\phi$-transform $\mathtt{NN}_{\phi}$ as long as
$(\\!(\phi,\psi)\\!)\neq 0,\infty$.

###### Proof.

We write $\mathtt{NN}[\bullet;\phi]$ as $\mathtt{NN}_{\phi}$ and
$\mathtt{R}[\bullet;\phi]$ as $\mathtt{R}_{\phi}$ for short. By using the
unitarity of representation $\upsilon:G\to\mathcal{U}(\mathcal{H})$, left-
invariance of measure $\mathrm{d}x$, and $G$-equivariance of feature map
$\psi$, for all $g\in G$, we have

$\displaystyle\mathtt{R}_{\psi}[\pi_{g}[f]](\xi)$
$\displaystyle=\int_{X}\langle g\cdot f(g^{-1}\cdot
x),\psi(x,\xi)\rangle_{\mathcal{H}}\mathrm{d}x=\int_{X}\langle
f(x),g^{-1}\cdot\psi(g\cdot x,\xi)\rangle_{\mathcal{H}}\mathrm{d}x$
$\displaystyle=\int_{X}\langle
f(x),\psi(x,g^{-1}\cdot\xi)\rangle_{\mathcal{H}}\mathrm{d}x=\widehat{\pi}_{g}[\mathtt{R}_{\psi}[f]](\xi).$
(14)

Similarly,

$\displaystyle\mathtt{NN}_{\phi}[\widehat{\pi}_{g}[\gamma]](x)$
$\displaystyle=\int_{\Xi}\gamma(g^{-1}\cdot\xi)\phi(x,\xi)\mathrm{d}\xi=\int_{\Xi}\gamma(\xi)\phi(x,g\cdot\xi)\mathrm{d}\xi$
$\displaystyle=\int_{\Xi}\gamma(\xi)\ \left(g\cdot\phi(g^{-1}\cdot
x,\xi)\right)\mathrm{d}\xi=\pi_{g}[\mathtt{NN}_{\phi}[\gamma]](x).$ (15)

Here, $\widehat{\pi}^{*}$ denotes the dual representation of $\widehat{\pi}$
with respect to $L^{2}(\Xi)$.

As a consequence,
$\mathtt{NN}_{\phi}\circ\mathtt{R}_{\psi}:L^{2}(X;\mathcal{H})\to
L^{2}(X;\mathcal{H})$ commutes with $\pi$ as below

$\displaystyle\mathtt{NN}_{\phi}\circ\mathtt{R}_{\psi}\circ\pi_{g}=\mathtt{NN}_{\phi}\circ\widehat{\pi}_{g}\circ\mathtt{R}_{\psi}=\pi_{g}\circ\mathtt{NN}_{\phi}\circ\mathtt{R}_{\psi}$
(16)

for all $g\in G$. Hence by Schur’s lemma (Theorem 2), there exist a constant
$C_{\phi,\psi}\in\mathbb{C}$ such that
$\mathtt{NN}_{\phi}\circ\mathtt{R}_{\psi}=C_{\phi,\psi}\operatorname{Id}_{L^{2}(X)}$.
Since $\mathtt{NN}_{\phi}\circ\mathtt{R}_{\psi}$ is bilinear in $\phi$ and
$\psi$, $C_{\phi,\psi}$ is bilinear in $\phi$ and $\psi$. ∎

In particular, because depth-$n$ feature map $\phi_{1:n}$ is $G$-equivariant
(Lemma 2), the following depth-$n$ $\mathcal{H}$-valued deep network
$\mathtt{DNN}[\gamma;\phi_{1:n}]$ is $L^{2}(X;\mathcal{H})$-universal.

Figure 2: Deep $\mathcal{H}$-valued joint-$G$-equivariant network on $G$-space
$X$ is $L^{2}(X;\mathcal{H})$-universal when unitary representation $\pi$ of
$G$ on $L^{2}(X;\mathcal{H})$ is irreducible, and the distribution of
parameters for the network to represent a given map $f:X\to\mathcal{H}$ is
exactly given by the ridgelet transform $\mathtt{R}[f]$

######  Corollary 1 (Deep Ridgelet Transform).

For any maps $\gamma:X\to\mathbb{C}$ and $f\in L^{2}(X;\mathcal{H})$, put

$\displaystyle\mathtt{DNN}[\gamma;\phi_{1:n}](x)$
$\displaystyle:=\int_{\Xi_{1}\times\cdots\times\Xi_{n}}\gamma(\xi_{1},\ldots,\xi_{n})\phi_{n}(\bullet,\xi_{n})\circ\cdots\circ\phi_{1}(x,\xi_{1})\mathrm{d}\bm{\xi},\quad
x\in X,$ (17) $\displaystyle\mathtt{R}[f;\psi_{1:n}](\bm{\xi})$
$\displaystyle:=\int_{\Xi}\langle
f(x),\psi_{n}(\bullet,\xi_{n})\circ\cdots\circ\psi_{1}(x,\xi_{n})\rangle_{\mathcal{H}}\mathrm{d}x,\quad\bm{\xi}\in\Xi_{1}\times\cdots\times\Xi_{n}.$
(18)

Under the assumptions that
$\mathtt{DNN}_{\phi_{1:n}}\circ\mathtt{R}_{\psi_{1:n}}$ is bounded, and that
$\pi$ is irreducible, there exists a bilinear form
$(\\!(\phi_{1:n},\psi_{1:n})\\!)$ satisfying
$\mathtt{DNN}_{\phi_{1:n}}\circ\mathtt{R}_{\psi_{1:n}}=(\\!(\phi_{1:n},\psi_{1:n})\\!)\operatorname{Id}_{L^{2}(X;\mathcal{H})}$.

Again, it extends the original integral representation, and inherits the
_linearization_ trick of nonlinear parameters $\bm{\xi}$ by integrating all
the possible parameters (beyond the difference of layers) and indirectly
select which parameters to use by weighting on them.

##  4 Example: Depth-$n$ Fully-Connected Network with Arbitrary Activation

As a concrete example, we present the ridgelet transform for depth-$n$ fully-
connected network. First, we show the depth-2 case based on a joint-affine-
_invariant_ argument, which was originally demonstrated by Sonoda et al.
(2023b). Then, we show the depth-$n$ case based on a joint-_equivariant_
argument by extending the original arguments.

We use the following known facts.

######  Lemma 3.

The regular representation $\pi$ of the affine group $\operatorname{Aff}(m)$
on $L^{2}(\mathbb{R}^{m})$ (defined below) is irreducible.

See Folland (2015, Theorem 6.42) for the proof.

######  Lemma 4.

Suppose $\sigma$ and $\rho$ are a tempered distribution
($\mathcal{S}^{\prime}$) and a Schwartz test function, respectively. Then,
$S_{\sigma}\circ R_{\rho}:L^{2}(\mathbb{R}^{m})\to L^{2}(\mathbb{R}^{m})$ is
bounded.

See Sonoda et al. (2021b, Lemmas 7 and 8) for the proof.

###  4.1 Depth-2

Set $X:=\mathbb{R}^{m}$ (data domain), $\Xi:=\mathbb{R}^{m}\times\mathbb{R}$
(parameter domain), and $G:=\operatorname{Aff}(m)=GL(m)\ltimes\mathbb{R}^{m}$
be the $m$-dimensional affine group, acting on data domain $X$ by

$\displaystyle g\cdot\bm{x}:=L\bm{x}+\bm{t},\quad g=(L,\bm{t})\in
GL(m)\ltimes\mathbb{R}^{m},\ \bm{x}\in X.$ (19)

Addition to this, let $\pi$ be the regular representation of
$\operatorname{Aff}(m)$ on $L^{2}(X)$, namely

$\displaystyle\pi(g)[f](\bm{x}):=|\det L|^{-1/2}f(L^{-1}(\bm{x}-\bm{t})),\quad
f\in L^{2}(X)\mbox{ and }g=(L,\bm{t})\in GL(m)\ltimes\mathbb{R}^{m}.$ (20)

Further, define a _dual action_ of $\operatorname{Aff}(m)$ on the parameter
domain $\Xi$ as

$\displaystyle
g\cdot(\bm{a},b)=(L^{-\top}\bm{a},b+\bm{t}^{\top}L^{-\top}\bm{a}),\quad
g=(L,\bm{t})\in GL(m)\ltimes\mathbb{R}^{m},\ (\bm{a},b)\in\Xi.$ (21)

Then, we can see the feature map
$\phi(\bm{x},(\bm{a},b)):=\sigma(\bm{a}\cdot\bm{x}-b)$ is joint-$G$-invariant.
In fact,

$\displaystyle\phi(g\cdot\bm{x},g\cdot(\bm{a},b))=\sigma\left(L^{-\top}\bm{a}\cdot(L\bm{x}+\bm{t})-(b+\bm{t}^{\top}L^{-\top}\bm{a})\right)=\sigma(\bm{a}\cdot\bm{x}-b)=\phi(\bm{x},(\bm{a},b)).$

By Lemma 3, the regular representation $\pi$ of $G=\operatorname{Aff}(m)$ is
irreducible. Therefore, by Theorem 3, the depth-$2$ neural network and
corresponding ridgelet transform:

$\displaystyle\mathtt{NN}[\gamma](\bm{x})=\int_{\mathbb{R}^{m}\times\mathbb{R}}\gamma(\bm{a},b)\sigma(\bm{a}\cdot\bm{x}-b)\mathrm{d}\bm{a}\mathrm{d}b,\quad\mbox{and}\quad\mathtt{R}_{2}[f](\bm{a},b)=\int_{\mathbb{R}^{m}}f(\bm{x})\overline{\rho(\bm{a}\cdot\bm{x}-b)}\mathrm{d}\bm{x},$

satisfy the reconstruction formula
$\mathtt{NN}\circ\mathtt{R}_{2}=(\\!(\sigma,\rho)\\!)\operatorname{Id}_{L^{2}(\mathbb{R}^{m})}$.
In Appendix A, we supplemented a detailed proof. In Appendix B, we discussed a
geometric interpretation of dual $G$-action (21).

###  4.2 Depth-$n$

Following Corollary 1, we derive the ridgelet transform for depth-$n$ fully-
connected network by constructing a joint-equivariant network.

Let $O(m)$ be the $m$-dimensional orthogonal group acting on $\mathbb{R}^{m}$
by $Q\bm{v}$ for $Q\in O(m)$ and $\bm{v}\in\mathbb{R}^{m}$, and (re)set
$G:=O(m)\times\operatorname{Aff}(m)$ be the product group, acting on the data
domain $X$ by

$\displaystyle g\cdot\bm{x}:=L\bm{x}+\bm{t},\quad\bm{x}\in X,g=(Q,L,\bm{t})\in
G=O(m)\times\left(GL(m)\ltimes\mathbb{R}^{m}\right).$ (22)

Namely, we set the $O(m)$-action on $X$ is trivial. Define a unitary
representation $\pi$ of $G$ on vector-valued square-integrable functions
$L^{2}(X;X)$ as

$\displaystyle\pi_{g}[\bm{f}](\bm{x}):=Q\bm{f}(L^{-1}(\bm{x}-\bm{t})),\quad\bm{x}\in
X,g=(Q,L,\bm{t})\in G,\bm{f}\in L^{2}(X;X).$ (23)

######  Lemma 5.

The above $\pi:G\to L^{2}(\mathbb{R}^{m};\mathbb{R}^{m})$ is an irreducible
unitary representation.

###### Proof.

Recall that a tensor product of irreducible representations is irreducible.
Since both $O(m)$-action on $\mathbb{R}^{m}$ and
$\operatorname{Aff}(m)$-action on $L^{2}(\mathbb{R}^{m})$ are irreducible, and
$L^{2}(\mathbb{R}^{m};\mathbb{R}^{m})$ is a tensor product
$\mathbb{R}^{m}\otimes L^{2}(\mathbb{R}^{m})$, so the action $\pi$ of product
group $O(m)\times\operatorname{Aff}(m)$ on tensor product
$\mathbb{R}^{m}\otimes
L^{2}(\mathbb{R}^{m})=L^{2}(\mathbb{R}^{m};\mathbb{R}^{m})$ is irreducible. ∎

Following the same arguments in Lemma 1, we first construct a _depth- $2$_
joint-$G$-equivariant network. Take an arbitrary square-integrable (not yet
joint-$G$-equivariant) vector-field $\bm{f}_{0}\in L^{2}(X;X)$. Then, the
following network is joint-$G$-equivariant:

$\displaystyle\mathtt{NN}(\bm{x},\xi)$
$\displaystyle:=\mathtt{NN}[\mathtt{R}_{2}[\pi_{\xi}[\bm{f}_{0}]]](\bm{x})=\int_{\mathbb{R}^{m}\times\mathbb{R}}Q\mathtt{R}_{2}[\bm{f}_{0}](\bm{a},b)\sigma\left(\bm{a}^{\top}L^{-1}(\bm{x}-\bm{t})-b\right)\mathrm{d}\bm{a}\mathrm{d}b,$
(24)

for every $\bm{x}\in X,\xi=(Q,L,\bm{t})\in O(m)\times
GL(m)\ltimes\mathbb{R}^{m}$. Here $\mathtt{R}_{2}$ is the ridgelet transform
for depth-2 case (applied for vector-valued function by element-wise manner).
This is joint-$G$-equivariant because
$\mathtt{NN}(\bm{x},\xi)=\pi_{\xi}[\bm{f}_{0}](\bm{x})$. Henceforth, we
(re)set $\Xi:=G$.

Finally, we construct a _depth- $n$_ joint-$G$-equivariant network by
composing the above depth-$2$ networks as below. Write
$\bm{\xi}:=(\xi_{1},\ldots,\xi_{n})\in\Xi^{n}$ for short. For any measurable
function $\gamma:\Xi^{n}\to\mathbb{C}$ and vector-field
$\bm{f}:\mathbb{R}^{m}\to\mathbb{R}^{m}$, put

$\displaystyle\mathtt{DNN}(\bm{x})$
$\displaystyle:=\int_{\Xi^{n}}\gamma(\bm{\xi})\mathtt{NN}(\bullet,\xi_{n})\circ\cdots\circ\mathtt{NN}(\bm{x},\xi_{1})\mathrm{d}\bm{\xi},\quad\bm{x}\in
X$ (25) $\displaystyle\mathtt{R}_{n}[\bm{f}](\bm{\xi})$
$\displaystyle:=\int_{X}\bm{f}(\bm{x})^{\top}\mathtt{NN}(\bullet,\xi_{n})\circ\cdots\circ\mathtt{NN}(\bm{x},\xi_{1})\mathrm{d}\bm{x},\quad\bm{\xi}\in\Xi^{n}.$
(26)

Then, as a consequence of Corollary 1, there exists a constant
$c\in\mathbb{C}$ satisfying $\mathtt{DNN}\circ\mathtt{R}_{n}[\bm{f}]=c\bm{f}$
for any $\bm{f}\in L^{2}(X;X)$.

##  5 Example: Formal Deep Network

We explain the _formal deep network_ (FDN) introduced by Sonoda et al.
(2023a). Compared to the depth-$n$ fully-connected network introduced in the
previous section, the FDN (introduced in the previous study) is more abstract
because the network architecture is not specified. Yet, we consider this is
still useful for theoretical study of deep networks as it covers a wide range
of groups and data domains (i.e., not limited to the affine group and the
Euclidean space).

###  5.1 Formal Deep Network

Let $G$ be an arbitrary locally compact group equipped with left-invariant
measure $\mathrm{d}g$, let $X$ be a $G$-space equipped with left-invariant
measure $\mathrm{d}x$, and set $\Xi:=G$ with right-invariant measure
$\mathrm{d}\xi$. The key concept is to identify each feature map
$\phi:X\times\Xi\to X$ with a $G$-action $g:X\to X$ with parameter domain
$\Xi$ being identified with group $G$, and the composite of feature maps, say
$g\circ h$, with product $gh$. Since a group is closed under its operation by
definition, the proposed network can represent literally _any depth_ such as a
single hidden layer $g$, double hidden layers $g\circ h$, triple hidden layers
$g\circ h\circ k$, and infinite hidden layers $g\circ h\circ\cdots$. Besides,
to lift the group action on a linear space, the network is formulated as a
regular action of group $G$ on a hidden layer, say $\psi\in L^{2}(X)$.

######  Definition 6 (Formal Deep Network).

For any functions $\psi\in L^{2}(X)$ and $\gamma:\Xi\to\mathbb{C}$, put

$\displaystyle\mathtt{DNN}[\gamma;\psi](x):=\int_{G_{1}\rtimes\cdots\rtimes
G_{n}}\gamma(\xi_{1},\ldots,\xi_{n})\
\psi\circ\xi_{n}\circ\cdots\circ\xi_{1}(x)\mathrm{d}\xi_{1}\cdots\mathrm{d}\xi_{n},\quad
x\in X.$ (27)

Here, $G=G_{1}\rtimes\cdots\rtimes G_{n}$ denotes the semi-direct product of
groups, suggesting that the network gets much complex and expressive as it
gets deeper.

To see the universality, define the dual action of $G$ on the parameter domain
$\Xi=G$ as

$\displaystyle g\cdot\xi:=\xi g^{-1},\quad g\in G,\xi\in\Xi.$ (28)

Then, we can see $\phi(x,\xi):=\psi\circ\xi(x)$ is joint-$G$-_invariant_. In
fact,

$\phi(g\cdot x,g\cdot\xi)=\psi\circ(g\cdot\xi)(g\cdot x)=\psi\circ(\xi\circ
g^{-1})(g(x))=\psi\circ\xi(x)=\phi(x,\xi).$

Therefore, by Theorem 3, assuming that the regular representation
$\pi:G\to\mathcal{U}(L^{2}(X))$ is irreducible, the ridgelet transform is
given by

$\displaystyle\mathtt{R}[f](\xi_{1},\ldots,\xi_{n})=\int_{X}f(x)\overline{\psi\circ\xi_{n}\circ\cdots\circ\xi_{1}(x)}\mathrm{d}x,\quad(\xi_{1},\ldots,\xi_{n})\in
G_{1}\rtimes\cdots\rtimes G_{n}$ (29)

satisfying
$\mathtt{NN}\circ\mathtt{R}=(\\!(\sigma,\rho)\\!)\operatorname{Id}_{L^{2}(X)}$.

###  5.2 Depth Separation

To enjoy the advantage of abstract formulation, we discuss the effect of
depth. For the sake of simplicity, we assume $G$ to be a finite group, which
may be acceptable given that the data domain $X$ in practice is often
discretized (or coarse-grained) into finite sets of representative points, say
$X\approx\overline{X}:=\\{x_{i}\\}_{i=1}^{p}$, and if so the $G$-action is
also reduced to finite representative actions.

Following the concept of the formal deep network, we call group $G$ acting on
$X$ a network. Let us consider depth-1 network $G$ and depth-$n$ network
$G_{1}\rtimes\cdots\rtimes G_{n}$ satisfying $G=G_{1}\rtimes\cdots\rtimes
G_{n}$. The equation indicates that two networks have the same expressive
power, because they can implement the same class of maps $g:X\to X$.

Next, let us define the _width_ of a single layer $G$ as the cardinality
$|G|$. This is reasonable because the set $G$ parametrizes each map $g:X\to
X$. Then, under the assumption that each $G_{i}$ is simple, the depth-$n$
network $G_{1}\rtimes\cdots\rtimes G_{n}$ can express the same class of
depth-1 network exponentially-effectively, because the total widths are
$\sum_{i=1}^{n}|G_{i}|=O(n)$ for depth-$n$ and $\prod_{i=1}^{n}|G_{i}|=\exp
O(n)$ for depth-1. This estimate can be interpreted as the classical thought
that the hierarchical models such as deep networks can represent complex
functions combinatorially more efficient than shallow models.

##  6 Discussion

We have developed a systematic method for deriving a ridgelet transform for a
wide range of learning machines defined by joint-group-equivariant feature
maps, yielding the universal approximation theorems as corollaries. The
previous results by Sonoda et al. (2023b) was limited to scalar-valued joint-
invariant functions, which were insufficient to deal with practical learning
machines defined by composite mappings of vector-valued functions, such as
deep neural networks. For example, they could only deal with abstract
composite structures like formal deep network (Sonoda et al., 2023a). By
extending their argument to vector-valued joint-equivariant functions, we were
able to deal with deep structures. Traditionally, the techniques used in the
expressive power analysis of deep networks were different from those used in
the analysis of shallow networks, as overviewed in the introduction.
Nonetheless, our main theorem cover both deep and shallow networks from the
unified perspective (joint-group-action on the data-parameter domain).
Technically, this unification is due to Schur’s lemma, a basic and useful
result in the representation theory. Thanks to this lemma, the proof of the
main theorem is simple, yet the scope of application is wide. The significance
of this study lies in revealing the close relationship between machine
learning theory and modern algebra. With this study as a catalyst, we expect a
major upgrade to machine learning theory from the perspective of modern
algebra.

###  6.1 Limitations

In the main theorem, we assume the following: (1) joint-equivariance of
feature map $\phi$, (2) boundedness of composite operator
$\mathtt{NN}\circ\mathtt{R}$, (3) irreducibility of unitary representation
$\pi$. In addition, throughout this study, we assume (4) local compactness of
group $G$, and (5) that the network is given by the integral representation.

As discussed in the main text, satisfying (1) is much easier than (non-joint)
equivariance. Also, (2) is often a textbook excersise when the specific
expression is given. (3) is required for Schur’s lemma, and it is often
sufficient to synthesize the known results such as the one for the example of
depth-$n$ fully-connected network. (4) is quite a frequent assumption in the
standard group representation theory, but it excludes infinite-dimensional
groups. When formulated _natively_ , nonparametric learning models including
DNN can be infinite-dimensional groups. However, from the perspective of
learnability, it is nonsense to consider too large a model, and it is common
to assume regularity conditions such as sparsity and low rank in usual
theoretical analysis. So, it is natural to impose additional regularity
conditions for satisfying local compactness. (5) may be rather an advantage
because there are established techniques to show the $cc$-universaity of
finite models by discretizing integral representations. Moreover, there is a
fast discretization scheme called the Barron’s rate based on the quasi-Monte
Carlo method. On the other hand, problems like the minimum width in the field
of deep narrow networks are analyses of finite parameters, and they could be a
different type of parameters. Yet, the current mainstream solutions are the
information theoretic method by Park et al. (2021) and the neural ODE method
by Cai (2023), and both arguments contain the discretization of continuous
models. Therefore, we may expect a high affinity with the integral
representation theory.

This study is the first step in extending the harmonic analysis method, which
was previously applicable only to shallow models, to deep models. The above
limitations will be resolved in our future works.

##  7 Broader Impact

This work studies theoretical aspects of neural networks for expressing square
integrable functions. Since we do not propose a new method nor a new dataset,
we expect that the impact of this work on ethical aspects and future societal
consequences will be small, if any. Our work can help understand the
theoretical benefit and limitations of neural networks in approximating
functions. Our work and the proof technique improve our understanding of the
theoretical aspect of deep neural networks and other learning machines used in
machine learning, and may lead to better use of these techniques with possible
benefits to the society.

#### Acknowledgments

This work was supported by JSPS KAKENHI 20K03657, JST PRESTO JPMJPR2125, JST
CREST JPMJCR2015 and JPMJCR1913, and JST ACTX JPMJAX2004.

## References

  * Bronstein et al. (2021) M. M. Bronstein, J. Bruna, T. Cohen, and P. Veličković.  Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges.  _arXiv preprint: 2104.13478_ , 2021.
  * Cai (2023) Y. Cai.  Achieve the Minimum Width of Neural Networks for Universal Approximation.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Candès (1998) E. J. Candès.  _Ridgelets: theory and applications_.  PhD thesis, Standford University, 1998.
  * Chen et al. (2018) R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud.  Neural Ordinary Differential Equations.  In _Advances in Neural Information Processing Systems_ , volume 31, pages 6572–6583, Palais des Congrès de Montréal, Montréal CANADA, 2018.
  * Chizat and Bach (2018) L. Chizat and F. Bach.  On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport.  In _Advances in Neural Information Processing Systems 32_ , pages 3036–3046, Montreal, BC, 2018.
  * Cohen et al. (2022) A. Cohen, R. DeVore, G. Petrova, and P. Wojtaszczyk.  Optimal Stable Nonlinear Approximation.  _Foundations of Computational Mathematics_ , 22(3):607–648, 2022.
  * Cohen et al. (2016) N. Cohen, O. Sharir, and A. Shashua.  On the Expressive Power of Deep Learning: A Tensor Analysis.  In _29th Annual Conference on Learning Theory_ , volume 49, pages 1–31, 2016.
  * Daubechies et al. (2022) I. Daubechies, R. DeVore, S. Foucart, B. Hanin, and G. Petrova.  Nonlinear Approximation and (Deep) ReLU Networks.  _Constructive Approximation_ , 55(1):127–172, 2022.
  * E (2017) W. E.  A Proposal on Machine Learning via Dynamical Systems.  _Communications in Mathematics and Statistics_ , 5(1):1–11, 2017.
  * Folland (2015) G. B. Folland.  _A Course in Abstract Harmonic Analysis_.  Chapman and Hall/CRC, New York, second edition, 2015.
  * Grohs et al. (2023) P. Grohs, A. Klotz, and F. Voigtlaender.  Phase Transitions in Rate Distortion Theory and Deep Learning.  _Foundations of Computational Mathematics_ , 23(1):329–392, 2023.
  * Haber and Ruthotto (2017) E. Haber and L. Ruthotto.  Stable architectures for deep neural networks.  _Inverse Problems_ , 34(1):1–22, 2017.
  * Hanin and Sellke (2017) B. Hanin and M. Sellke.  Approximating Continuous Functions by ReLU Nets of Minimal Width.  _arXiv preprint: 1710.11278_ , 2017.
  * Kidger and Lyons (2020) P. Kidger and T. Lyons.  Universal Approximation with Deep Narrow Networks.  In _Proceedings of Thirty Third Conference on Learning Theory_ , volume 125 of _Proceedings of Machine Learning Research_ , pages 2306–2327. PMLR, 2020.
  * Kim et al. (2024) N. Kim, C. Min, and S. Park.  Minimum width for universal approximation using ReLU networks on compact domain.  In _The Twelfth International Conference on Learning Representations_ , 2024.
  * Li et al. (2023) L. Li, Y. Duan, G. Ji, and Y. Cai.  Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 19460–19470, 2023.
  * Li and Hao (2018) Q. Li and S. Hao.  An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks.  In _Proceedings of The 35th International Conference on Machine Learning_ , volume 80, pages 2985–2994, Stockholm, 2018. PMLR.
  * Lin and Jegelka (2018) H. Lin and S. Jegelka.  ResNet with one-neuron hidden layers is a Universal Approximator.  In _Advances in Neural Information Processing Systems_ , volume 31, Montreal, BC, 2018.
  * Lu et al. (2017) Z. Lu, H. Pu, F. Wang, Z. Hu, and L. Wang.  The Expressive Power of Neural Networks: A View from the Width.  In _Advances in Neural Information Processing Systems_ , volume 30, 2017.
  * Mei et al. (2018) S. Mei, A. Montanari, and P.-M. Nguyen.  A mean field view of the landscape of two-layer neural networks.  _Proceedings of the National Academy of Sciences_ , 115(33):E7665–E7671, 2018.
  * Murata (1996) N. Murata.  An integral representation of functions using three-layered networks and their approximation bounds.  _Neural Networks_ , 9(6):947–956, 1996.
  * Nitanda and Suzuki (2017) A. Nitanda and T. Suzuki.  Stochastic Particle Gradient Descent for Infinite Ensembles.  _arXiv preprint: 1712.05438_ , 2017.
  * Park et al. (2021) S. Park, C. Yun, J. Lee, and J. Shin.  Minimum Width for Universal Approximation.  In _International Conference on Learning Representations_ , 2021.
  * Petrova and Wojtaszczyk (2023) G. Petrova and P. Wojtaszczyk.  Limitations on approximation by deep and shallow neural networks.  _Journal of Machine Learning Research_ , 24(353):1–38, 2023.
  * Rotskoff and Vanden-Eijnden (2018) G. Rotskoff and E. Vanden-Eijnden.  Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks.  In _Advances in Neural Information Processing Systems 31_ , pages 7146–7155, Montreal, BC, 2018.
  * Siegel (2023) J. W. Siegel.  Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces.  _Journal of Machine Learning Research_ , 24(357):1–52, 2023.
  * Sonoda and Murata (2017) S. Sonoda and N. Murata.  Transportation analysis of denoising autoencoders: a novel method for analyzing deep neural networks.  In _NIPS 2017 Workshop on Optimal Transport & Machine Learning (OTML)_, pages 1–10, Long Beach, 2017.
  * Sonoda et al. (2021a) S. Sonoda, I. Ishikawa, and M. Ikeda.  Ridge Regression with Over-Parametrized Two-Layer Networks Converge to Ridgelet Spectrum.  In _Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (AISTATS) 2021_ , volume 130, pages 2674–2682. PMLR, 2021a.
  * Sonoda et al. (2021b) S. Sonoda, I. Ishikawa, and M. Ikeda.  Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Space.  _arXiv preprint: 2106.04770_ , 2021b.
  * Sonoda et al. (2022a) S. Sonoda, I. Ishikawa, and M. Ikeda.  Universality of Group Convolutional Neural Networks Based on Ridgelet Analysis on Groups.  In _Advances in Neural Information Processing Systems 35_ , pages 38680–38694, New Orleans, Louisiana, USA, 2022a.
  * Sonoda et al. (2022b) S. Sonoda, I. Ishikawa, and M. Ikeda.  Fully-Connected Network on Noncompact Symmetric Space and Ridgelet Transform based on Helgason-Fourier Analysis.  In _Proceedings of the 39th International Conference on Machine Learning_ , volume 162, pages 20405–20422, Baltimore, Maryland, USA, 2022b.
  * Sonoda et al. (2023a) S. Sonoda, Y. Hashimoto, I. Ishikawa, and M. Ikeda.  Deep Ridgelet Transform: Voice with Koopman Operator Proves Universality of Formal Deep Networks.  In _Proceedings of the 2nd NeurIPS Workshop on Symmetry and Geometry in Neural Representations_ , Proceedings of Machine Learning Research. PMLR, 2023a.
  * Sonoda et al. (2023b) S. Sonoda, H. Ishi, I. Ishikawa, and M. Ikeda.  Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networks.  In _Proceedings of the 2nd NeurIPS Workshop on Symmetry and Geometry in Neural Representations_ , Proceedings of Machine Learning Research. PMLR, 2023b.
  * Sonoda et al. (2024) S. Sonoda, I. Ishikawa, and M. Ikeda.  A unified Fourier slice method to derive ridgelet transform for a variety of depth-2 neural networks.  _Journal of Statistical Planning and Inference_ , 233:106184, 2024.
  * Suzuki (2020) T. Suzuki.  Generalization bound of globally optimal non-convex neural network training: Transportation map estimation by infinite dimensional Langevin dynamics.  In _Advances in Neural Information Processing Systems 33_ , pages 19224–19237, 2020.
  * Telgarsky (2016) M. Telgarsky.  Benefits of depth in neural networks.  In _29th Annual Conference on Learning Theory_ , pages 1–23, 2016.
  * Yarotsky (2017) D. Yarotsky.  Error bounds for approximations with deep ReLU networks.  _Neural Networks_ , 94:103–114, 2017.
  * Yarotsky (2018) D. Yarotsky.  Optimal approximation of continuous functions by very deep ReLU networks.  In _Proceedings of the 31st Conference On Learning Theory_ , volume 75 of _Proceedings of Machine Learning Research_ , pages 639–649. PMLR, 2018.
  * Yarotsky and Zhevnerchuk (2020) D. Yarotsky and A. Zhevnerchuk.  The phase diagram of approximation rates for deep neural networks.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 13005–13015, 2020.

##  Appendix A Depth-$2$ Fully-Connected Neural Network and Ridgelet Transform

A non group theoretic proof by reducing to a Fourier expression is given in
Sonoda et al. (2021b, Theorem 6).

###  A.1 Proof

In the following, we identify the group $G$ acting on data domain
$\mathbb{R}^{m}$ with the affine group $\operatorname{Aff}(\mathbb{R}^{m})$,
and introduce the so-called twisted dual group action that leaves a function
$\theta$ invariant. Then, we see the regular action $\pi$ of $G$ on functions
space $L^{2}(\mathbb{R}^{m})$ commutes with composite $S_{\sigma}\circ
R_{\rho}$. Hence, by Schur’s lemma, $S_{\sigma}\circ R_{\rho}$ is a constant
multiple of identity, which concludes the assertion.

###### Proof.

Let $G$ be the affine group
$\operatorname{Aff}(\mathbb{R}^{m})=GL(\mathbb{R}^{m})\ltimes\mathbb{R}^{m}$.
For any $g=(L,\bm{t})\in G$, let

$\displaystyle g\cdot\bm{x}:=L\bm{x}+\bm{t},\quad\bm{x}\in\mathbb{R}^{m}$ (30)

be its action on $\mathbb{R}^{m}$, and let

$\displaystyle\pi(g)[f](\bm{x})$ $\displaystyle:=|\det
L|^{-1/2}f(g^{-1}\cdot\bm{x})$ $\displaystyle=|\det
L|^{-1/2}f(L^{-1}(\bm{x}-\bm{t})),\quad f\in L^{2}(\mathbb{R}^{m})$ (31)

be its left-regular action on $L^{2}(\mathbb{R}^{m})$.

Besides, putting

$\displaystyle\theta((\bm{a},b),\bm{x}):=\bm{a}\cdot\bm{x}-b,\quad(\bm{a},b)\in\mathbb{R}^{m}\times\mathbb{R},\bm{x}\in\mathbb{R}^{m}$
(32)

we define the _twisted dual action_ of $G$ on $\mathbb{R}^{m}\times\mathbb{R}$
as

$\displaystyle
g\cdot(\bm{a},b):=(L^{-\top}\bm{a},b+\bm{a}\cdot(L^{-1}\bm{t})),\quad(\bm{a},b)\in\mathbb{R}^{m}\times\mathbb{R}$
(33)

so that the following invariance hold:

$\displaystyle\theta(g\cdot(\bm{a},b),g\cdot\bm{x})=\theta((\bm{a},b),\bm{x})=\bm{a}\cdot\bm{x}-b.$
(34)

To see this, use matrix expressions with extended variables

$\displaystyle\theta((\bm{a},b),\bm{x})$
$\displaystyle=\begin{pmatrix}\bm{a}^{\top}&b\end{pmatrix}\begin{pmatrix}I_{m}&0\\\
0&-1\end{pmatrix}\begin{pmatrix}\bm{x}\\\
1\end{pmatrix}=:\tilde{\bm{a}}^{\top}\tilde{I}\tilde{\bm{x}},$ (35)
$\displaystyle\widetilde{g\cdot\bm{x}}$
$\displaystyle:=\begin{pmatrix}g\cdot\bm{x}\\\
1\end{pmatrix}=\begin{pmatrix}L&\bm{t}\\\
0&1\end{pmatrix}\begin{pmatrix}\bm{x}\\\
1\end{pmatrix}=:\tilde{L}\tilde{\bm{x}}$ (36)

and calculate

$\displaystyle\tilde{\bm{a}}^{\top}\tilde{I}\tilde{\bm{x}}=(\tilde{\bm{a}}^{\top}\tilde{I}\tilde{L}^{-1}\tilde{I}^{-1})\tilde{I}(\tilde{L}\tilde{\bm{x}})=(\tilde{I}\tilde{L}^{-\top}\tilde{I}\tilde{\bm{a}})^{\top}\tilde{I}(\tilde{L}\tilde{\bm{x}}),$
(37)

which suggests
$\widetilde{g\cdot(\bm{a},b)}:=\tilde{I}\tilde{L}^{-\top}\tilde{I}\tilde{\bm{a}}$,
and we have

$\displaystyle\tilde{I}\tilde{L}^{-\top}\tilde{I}$
$\displaystyle=\begin{pmatrix}I_{m}&0\\\
0&-1\end{pmatrix}\begin{pmatrix}L&\bm{t}\\\
0&1\end{pmatrix}^{-\top}\begin{pmatrix}I_{m}&0\\\ 0&-1\end{pmatrix}$
$\displaystyle=\begin{pmatrix}I_{m}&0\\\
0&-1\end{pmatrix}\begin{pmatrix}L^{-\top}&0\\\
-\bm{t}^{\top}L^{-\top}&1\end{pmatrix}\begin{pmatrix}I_{m}&0\\\
0&-1\end{pmatrix}=\begin{pmatrix}L^{-\top}&0\\\
\bm{t}^{\top}L^{-\top}&1\end{pmatrix}.$

Further, we define its regular-action by

$\displaystyle\widehat{\pi}(g)[\gamma](\bm{a},b)$ $\displaystyle:=|\det
L|^{1/2}\gamma(g^{-1}\cdot(\bm{a},b))$ $\displaystyle=|\det
L|^{1/2}\gamma(L^{\top}\bm{a},b-\bm{a}\cdot\bm{t}),\quad(\bm{a},b)\in\mathbb{R}^{m}\times\mathbb{R}.$
(38)

Then we can see that, for all $g=(L,\bm{t})\in G$,

$\displaystyle R_{\rho}\circ\pi(g)=\widehat{\pi}(g)\circ
R_{\rho},\quad\mbox{and}\quad S_{\sigma}\circ\widehat{\pi}(g)=\pi(g)\circ
S_{\sigma}.$ (39)

In fact, at every $g=(L,\bm{t})\in G$ and
$(\bm{a},b)\in\mathbb{R}^{m}\times\mathbb{R}$,

$\displaystyle R_{\rho}[\pi(g)[f]](\bm{a},b)$ $\displaystyle=|\det
L|^{-1/2}\int_{\mathbb{R}^{m}}f(g^{-1}\cdot\bm{x})\overline{\rho(\theta((\bm{a},b),\bm{x}}))\mathrm{d}\bm{x}$
by putting $\bm{x}=g\cdot\bm{y}=L\bm{y}+\bm{t}$ with $\mathrm{d}\bm{x}=|\det
L|\mathrm{d}\bm{y}$, $\displaystyle=|\det
L|^{1/2}\int_{\mathbb{R}^{m}}f(\bm{y})\overline{\rho(\theta((\bm{a},b),g\cdot\bm{y}})))\mathrm{d}\bm{y}$
$\displaystyle=|\det
L|^{1/2}\int_{\mathbb{R}^{m}}f(\bm{y})\overline{\rho(\theta(g^{-1}\cdot(\bm{a},b),\bm{y}})))\mathrm{d}\bm{y}$
$\displaystyle=\widehat{\pi}(g)[R_{\rho}[f]](\bm{a},b).$ (40)

Similarly, at every $g=(L,\bm{t})\in G$ and $\bm{x}\in\mathbb{R}^{m}$,

$\displaystyle S_{\sigma}[\widehat{\pi}(g)[\gamma]](\bm{x})$
$\displaystyle=|\det
L|^{1/2}\int_{\mathbb{R}^{m}\times\mathbb{R}}\gamma(g^{-1}\cdot(\bm{a},b))\sigma(\theta((\bm{a},b),\bm{x}))\mathrm{d}\bm{a}\mathrm{d}b$
by putting
$(\bm{a},b):=g\cdot(\bm{\xi},\eta)=(L^{-\top}\bm{\xi},\eta+\bm{\xi}\cdot(L^{-1}\bm{t}))$
with $\mathrm{d}\bm{a}\mathrm{d}b=|\det L|\mathrm{d}\bm{\xi}\mathrm{d}\eta$,
$\displaystyle=|\det
L|^{-1/2}\int_{\mathbb{R}^{m}\times\mathbb{R}}\gamma(\bm{\xi},\eta)\sigma(\theta(g\cdot(\bm{\xi},\eta),\bm{x}))\mathrm{d}\bm{\xi}\mathrm{d}\eta$
$\displaystyle=|\det
L|^{-1/2}\int_{\mathbb{R}^{m}\times\mathbb{R}}\gamma(\bm{\xi},\eta)\sigma(\theta((\bm{\xi},\eta),g^{-1}\cdot\bm{x}))\mathrm{d}\bm{\xi}\mathrm{d}\eta$
$\displaystyle=\pi(g)[S_{\sigma}[\gamma]](\bm{x}).$ (41)

Hence $S_{\sigma}\circ R_{\rho}$ commutes with $\pi(g)$ because

$\displaystyle S_{\sigma}\circ
R_{\rho}\circ\pi(g)=S_{\sigma}\circ\widehat{\pi}(g)\circ R_{\rho}=\pi(g)\circ
S_{\sigma}\circ R_{\rho}.$

Since $S_{\sigma}\circ R_{\rho}:L^{2}(\mathbb{R}^{m})\to
L^{2}(\mathbb{R}^{m})$ is bounded (Lemma 4), and $(\pi,L^{2}(\mathbb{R}^{m}))$
is an irreducible unitary representation of $G$ (Lemma 3), Schur’s lemma
(Theorem 2) yields that there exist a constant $C_{\sigma,\rho}\in\mathbb{C}$
such that

$\displaystyle S_{\sigma}\circ R_{\rho}[f]=C_{\sigma,\rho}f$ (42)

for all $f\in L^{2}(\mathbb{R}^{m})$.

Finally, by directly computing the left-hand-side, namely $S_{\sigma}\circ
R_{\rho}[f]$, we can verify that the constant $C_{\sigma,\rho}$ is given by

$\displaystyle
C_{\sigma,\rho}=(\\!(\sigma,\rho)\\!):=(2\pi)^{m-1}\int_{\mathbb{R}}\sigma^{\sharp}(\omega)\overline{\rho^{\sharp}(\omega)}|\omega|^{-m}\mathrm{d}\omega.$
(43)

∎

###  A.2 Proof for (33)

Use matrix expressions with extended variables

$\displaystyle\theta((\bm{a},b),\bm{x})$
$\displaystyle=\begin{pmatrix}\bm{a}^{\top}&b\end{pmatrix}\begin{pmatrix}I_{m}&0\\\
0&-1\end{pmatrix}\begin{pmatrix}\bm{x}\\\
1\end{pmatrix}=:\tilde{\bm{a}}^{\top}\tilde{I}\tilde{\bm{x}},$ (44)
$\displaystyle\widetilde{g\cdot\bm{x}}$
$\displaystyle:=\begin{pmatrix}g\cdot\bm{x}\\\
1\end{pmatrix}=\begin{pmatrix}L&\bm{t}\\\
0&1\end{pmatrix}\begin{pmatrix}\bm{x}\\\
1\end{pmatrix}=:\tilde{L}\tilde{\bm{x}}$ (45)

and calculate

$\displaystyle\tilde{\bm{a}}^{\top}\tilde{I}\tilde{\bm{x}}=(\tilde{\bm{a}}^{\top}\tilde{I}\tilde{L}^{-1}\tilde{I}^{-1})\tilde{I}(\tilde{L}\tilde{\bm{x}})=(\tilde{I}\tilde{L}^{-\top}\tilde{I}\tilde{\bm{a}})^{\top}\tilde{I}(\tilde{L}\tilde{\bm{x}}),$
(46)

which suggests
$\widetilde{g\cdot(\bm{a},b)}:=\tilde{I}\tilde{L}^{-\top}\tilde{I}\tilde{\bm{a}}$,
and we have

$\displaystyle\tilde{I}\tilde{L}^{-\top}\tilde{I}$
$\displaystyle=\begin{pmatrix}I_{m}&0\\\
0&-1\end{pmatrix}\begin{pmatrix}L&\bm{t}\\\
0&1\end{pmatrix}^{-\top}\begin{pmatrix}I_{m}&0\\\ 0&-1\end{pmatrix}$
$\displaystyle=\begin{pmatrix}I_{m}&0\\\
0&-1\end{pmatrix}\begin{pmatrix}L^{-\top}&0\\\
-\bm{t}^{\top}L^{-\top}&1\end{pmatrix}\begin{pmatrix}I_{m}&0\\\
0&-1\end{pmatrix}=\begin{pmatrix}L^{-\top}&0\\\
\bm{t}^{\top}L^{-\top}&1\end{pmatrix}.$

###  A.3 Proof for (39)

In fact, at every $g=(L,\bm{t})\in G$ and
$(\bm{a},b)\in\mathbb{R}^{m}\times\mathbb{R}$,

$\displaystyle R_{\rho}[\pi(g)[f]](\bm{a},b)$ $\displaystyle=|\det
L|^{-1/2}\int_{\mathbb{R}^{m}}f(g^{-1}\cdot\bm{x})\overline{\rho(\theta((\bm{a},b),\bm{x}}))\mathrm{d}\bm{x}$
by putting $\bm{x}=g\cdot\bm{y}=L\bm{y}+\bm{t}$ with $\mathrm{d}\bm{x}=|\det
L|\mathrm{d}\bm{y}$, $\displaystyle=|\det
L|^{1/2}\int_{\mathbb{R}^{m}}f(\bm{y})\overline{\rho(\theta((\bm{a},b),g\cdot\bm{y}})))\mathrm{d}\bm{y}$
$\displaystyle=|\det
L|^{1/2}\int_{\mathbb{R}^{m}}f(\bm{y})\overline{\rho(\theta(g^{-1}\cdot(\bm{a},b),\bm{y}})))\mathrm{d}\bm{y}$
$\displaystyle=\widehat{\pi}(g)[R_{\rho}[f]](\bm{a},b).$ (47)

Similarly, at every $g=(L,\bm{t})\in G$ and $\bm{x}\in\mathbb{R}^{m}$,

$\displaystyle S_{\sigma}[\widehat{\pi}(g)[\gamma]](\bm{x})$
$\displaystyle=|\det
L|^{1/2}\int_{\mathbb{R}^{m}\times\mathbb{R}}\gamma(g^{-1}\cdot(\bm{a},b))\sigma(\theta((\bm{a},b),\bm{x}))\mathrm{d}\bm{a}\mathrm{d}b$
by putting
$(\bm{a},b):=g\cdot(\bm{\xi},\eta)=(L^{-\top}\bm{\xi},\eta+\bm{\xi}\cdot(L^{-1}\bm{t}))$
with $\mathrm{d}\bm{a}\mathrm{d}b=|\det L|\mathrm{d}\bm{\xi}\mathrm{d}\eta$,
$\displaystyle=|\det
L|^{-1/2}\int_{\mathbb{R}^{m}\times\mathbb{R}}\gamma(\bm{\xi},\eta)\sigma(\theta(g\cdot(\bm{\xi},\eta),\bm{x}))\mathrm{d}\bm{\xi}\mathrm{d}\eta$
$\displaystyle=|\det
L|^{-1/2}\int_{\mathbb{R}^{m}\times\mathbb{R}}\gamma(\bm{\xi},\eta)\sigma(\theta((\bm{\xi},\eta),g^{-1}\cdot\bm{x}))\mathrm{d}\bm{\xi}\mathrm{d}\eta$
$\displaystyle=\pi(g)[S_{\sigma}[\gamma]](\bm{x}).$ (48)

##  Appendix B Geometric Interpretation of Dual Action for Original Ridgelet
Transform

We explain a geometric interpretation of the dual action (33) in the previous
section. We note that in general $\theta$ does not require any geometric
interpretation as long as it is joint group invariant on data-parameter
domain.

For each $(\bm{a},b)\in\mathbb{R}^{m}\times\mathbb{R}$, put
$\xi(\bm{a},b):=\\{\bm{x}\in\mathbb{R}^{m}\mid\bm{a}\cdot\bm{x}-b=0\\}$. Then
it is a hyperplane in $\mathbb{R}^{m}$ through point
$\bm{x}_{0}=b\bm{a}/|\bm{a}|^{2}$ with normal vector
$\bm{u}:=\bm{a}/|\bm{a}|$.

Figure 3: The invariant $\phi((\bm{a},b),\bm{x})=\sigma(\bm{a}\cdot\bm{x}-b)$
is the euclidean distance between point $\bm{x}$ and hyperplane
$\xi(\bm{a},b)$ followed by scaling and nonlinearity $\sigma$

For any point $\bm{y}$ in the hyperplane $\xi(\bm{a},b)$, by definition
$\bm{a}\cdot\bm{y}=b$, thus

$\displaystyle\bm{a}\cdot\bm{x}-b$ $\displaystyle=\bm{a}\cdot(\bm{x}-\bm{y}).$
(49) But this means $\bm{a}\cdot\bm{x}-b$ is a scaled distance between point
$\bm{x}$ and hyperplane $\xi(\bm{a},b)$,
$\displaystyle=|\bm{a}|d_{E}(\bm{x},\xi(\bm{a},b)),$ (50) and further a scaled
distance between hyperplanes $\xi(\bm{a},\bm{a}\cdot\bm{x})$ through $\bm{x}$
with normal $\bm{a}/|\bm{a}|$ and $\xi(\bm{a},b)$,
$\displaystyle=|\bm{a}|d_{E}(\xi(\bm{a},\bm{a}\cdot\bm{x}),\xi(\bm{a},b)).$
(51)

Now, we can interpret the invariant
$\theta((\bm{a},b),\bm{x}):=\bm{a}\cdot\bm{x}-b$ in a geometric manner, that
is, it is the distance between point and hyperplane, or between hyperplanes.
We note that we can regard entire $\sigma(\bm{a}\cdot\bm{x}-b)$—the distance
modulated by both scaling and nonlinearity—as the invariant, say $\phi$.

Furthermore, the dual action $g\cdot(\bm{a},b)$ is understood as a parallel
translation of hyperplane $\xi(\bm{a},b)$ to $\xi(g\cdot(\bm{a},b))$ so as to
leave the scaled distance $\theta$ invariant, namely

$\displaystyle
d_{E}(g\cdot\bm{x},g\cdot\xi(\bm{a},b))=d_{E}(\bm{x},\xi(\bm{a},b)).$ (52)

Indeed, for any $g=(L,\bm{t})\in G$,

$\displaystyle g\cdot\xi(\bm{a},b)$
$\displaystyle=\\{g\cdot\bm{x}\mid\bm{a}\cdot\bm{x}-b=0\\}$
$\displaystyle=\\{\bm{y}\mid\bm{a}\cdot(g^{-1}\cdot\bm{y})-b=0\\}$ (by letting
$\bm{y}=g\cdot\bm{x}$)
$\displaystyle=\\{\bm{y}\mid(L^{-\top})\cdot\bm{y}-(b+\bm{a}\cdot(L^{-1}\bm{t}))=0\\}$
$\displaystyle=\xi(g\cdot(\bm{a},b)),$

meaning that the hyperplane with parameter $(\bm{a},b)$ translated by $g$ is
identical to the hyperplane with parameter $g\cdot(\bm{a},b)$.

To summarize, in the case of fully-connected neural network (and its
corresponding ridgelet transform), the invariant is a modulated distance
$\sigma(\bm{a}\cdot\bm{x}-b)$, and the dual action is the parallel translation
of hyperplane so as to keep the distance invariant. Further, from this
geometric perspective, we can rewrite the fully-connected neural network in a
geometric manner as

$\displaystyle
S[\gamma](\bm{x}):=\int_{\mathbb{R}\times\Xi}\gamma(\xi)\sigma(ad_{E}(\bm{x},\xi))\mathrm{d}a\mathrm{d}\xi,$
(53)

where $a\in\mathbb{R}$ denotes signed scale and $\Xi$ denotes the space of all
hyperplanes (not always through the origin). Since each hyperplane is
parametrized by normal vectors $\bm{u}\in\SS^{m-1}$ and distance $p\geq 0$
from the origin, we can induce the product of spherical measure
$\mathrm{d}\bm{u}$ and Lebesgue measure $\mathrm{d}p$ as a measure
$\mathrm{d}\xi$ on the space $\Xi$ of hyperplanes.