Spectral Neural Networks: Approximation Theory and Optimization
  Landscape
There is a large variety of machine learning methodologies that are based on
the extraction of spectral geometric information from data. However, the
implementations of many of these methods often depend on traditional
eigensolvers, which present limitations when applied in practical online big
data scenarios. To address some of these challenges, researchers have proposed
different strategies for training neural networks as alternatives to
traditional eigensolvers, with one such approach known as Spectral Neural
Network (SNN). In this paper, we investigate key theoretical aspects of SNN.
First, we present quantitative insights into the tradeoff between the number of
neurons and the amount of spectral geometric information a neural network
learns. Second, we initiate a theoretical exploration of the optimization
landscape of SNN's objective to shed light on the training dynamics of SNN.
Unlike typical studies of convergence to global solutions of NN training
dynamics, SNN presents an additional complexity due to its non-convex ambient
loss function.
Chenghui Li, Rishi Sonthalia, Nicolas Garcia Trillos
# Spectral neural networks: approximation theory and optimization landscape

###### Abstract.

There is a large variety of machine learning methodologies that are based on the extraction of spectral geometric information from data. However, the implementations of many of these methods often depend on traditional eigensolvers, which present limitations when applied in practical online big data scenarios. To address some of these challenges, researchers have proposed different strategies for training neural networks as alternatives to traditional eigensolvers, with one such approach known as Spectral Neural Network (SNN). In this paper, we investigate key theoretical aspects of SNN. First, we present quantitative insights into the tradeoff between the number of neurons and the amount of spectral geometric information a neural network learns. Second, we initiate a theoretical exploration of the optimization landscape of SNN's objective to shed light on the training dynamics of SNN. Unlike typical studies of convergence to global solutions of NN training dynamics, SNN presents an additional complexity due to its non-convex ambient loss function.

**Acknowledgements:** The authors would like to thank Abiy Tasissa and Yuetian Luo for enlightening discussions on the topics covered in this paper. This material is based upon work supported by the National Science Foundation under Grant Number DMS 1641020 and was started during the summer of 2022 when the authors participated in the AMS-MRC program: _Data Science at the Crossroads of Analysis, Geometry, and Topology._ NGT was supported by the NSF grants DMS-2005797 and DMS-2236447. CL and NGT would like to thank the IPDS at UW-Madison and NSF through TRIPODS grant 2023239 for their support.
Introduction

The study of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of the influence of influence of the
where \(\|x-y\|\) denotes the Euclidean distance between \(x\) and \(y\), \(\varepsilon\) is a proximity parameter, and \(\eta\) is a decreasing, non-negative function. In short, \(\mathbf{G}^{\varepsilon}\) measures the similarity between points according to their proximity. From \(\mathbf{G}^{\varepsilon}\) we define the adjacency matrix \(\mathcal{A}_{\mathbf{n}}\) appearing in Equation 1.1 by

\[\mathcal{A}_{\mathbf{n}}\stackrel{{\mathrm{def}}}{{=}}\mathbf{D} _{\mathbf{G}}^{-\frac{1}{2}}\mathbf{G}\mathbf{D}_{\mathbf{G}}^{-\frac{1}{2}}+a\mathbf{I}, \tag{1.3}\]

where \(\mathbf{D}_{\mathbf{G}}\) is the degree matrix associated to \(\mathbf{G}\) and \(a>1\) is a fixed quantity. Here we distance ourselves slightly from the choice made in the original SNN paper HaoChen et al. (2021), where \(\mathcal{A}_{\mathbf{n}}\) is taken to be \(\mathbf{G}\) itself, and instead consider a normalized version. This is due to the following key properties satisfied by our choice of \(\mathcal{A}_{\mathbf{n}}\) (see also Remark D.1 in Appendix D) that make it more suitable for theoretical analysis.

**Proposition 1**.: _The matrix \(\mathcal{A}_{\mathbf{n}}\) defined in Equation 1.1 satisfies the following properties:_

1. \(\mathcal{A}_{\mathbf{n}}\) _is symmetric positive definite._
2. \(\mathcal{A}_{\mathbf{n}}\) _'s_ \(r\) _top eigenvectors (the ones corresponding to the_ \(r\) _largest eigenvalues) coincide with the eigenvectors of the_ \(r\) _smallest eigenvalues of the symmetric normalized graph Laplacian matrix (see Von Luxburg (2007)):_ (1.4) \[\Delta_{n}\stackrel{{\mathrm{def}}}{{=}}\mathbf{I}-\mathbf{D}_{ \mathbf{G}}^{-1/2}\mathbf{G}\mathbf{D}_{\mathbf{G}}^{-1/2}.\]

The above two properties, proved in Appendix D, are useful when combined with recent results on the regularity of graph Laplacian eigenvectors over proximity graphs Calder et al. (2022) (see Appendix E.1) and some results on the approximation of Lipschitz functions on manifolds using neural networks Chen et al. (2022) (see Appendix E.2). In particular, we answer question Q1, which belongs to the realm of approximation theory, by providing a concrete bound on the number of neurons in a multi-layer ReLU NN that are necessary to approximate the \(r\) smallest eigenvectors of the normalized graph Laplacian matrix \(\Delta_{n}\) (as defined in 1.4) and thus also the \(r\) largest eigenvectors of \(\mathcal{A}_{\mathbf{n}}\); this is the content of Theorem 2.1.

While our answer to question Q1 addresses the existence of a neural network approximating the spectrum of \(\mathcal{A}_{\mathbf{n}}\), it does not provide a _constructive_ way to find one such approximation. We thus address question Q2 and prove that an approximating NN can be constructed by solving the optimization problem 1.1, i.e., by finding a global minimizer of SNN's objective function. A precise statement can be found in Theorem 2.2. To prove this theorem, we rely on our estimates in Theorem 2.1 and on some auxiliary computations involving a global optimizer \(\mathbf{Y}^{*}\) of the "ambient space problem":

\[\min_{\mathbf{Y}\in\mathbb{R}^{n\times r}}\ \ell(\mathbf{Y}). \tag{1.5}\]

For that we also make use of property 1 in Proposition 1, which allows us to guarantee, thanks to the Eckart-Young-Mirsky theorem (see Eckart & Young (1936) ), that solutions \(\mathbf{Y}\) to Equation 1.5 coincide, up to multiplication on the right by a \(r\times r\) orthogonal matrix, with a \(n\times r\) matrix \(\mathbf{Y}^{*}\) whose columns are scaled versions of the top \(r\) normalized eigenvectors of the matrix \(\mathcal{A}_{\mathbf{n}}\); see a detailed description of \(\mathbf{Y}^{*}\) in Appendix D.2.
After discussing our spectral approximation results, we move on to discussing question Q3, which is related to the hardness of optimization problem 1.1. Notice that, while \(\mathbf{Y}_{\theta^{*}}\) is a good approximator for \(\mathcal{A}_{\mathbf{n}}\)'s spectrum according to our theory, it is unclear whether \(\theta^{*}\) can be reached through a standard training scheme. In fact, question Q3, as stated, is a challenging problem. This is not only due to the non-linearities in the neural network, but also because, in contrast to more standard theoretical studies of training dynamics of over-parameterized NNs (e.g., Chizat & Bach (2018), Wojtowytsch (2020)), the spectral contrastive loss function \(\ell\) is non-convex in the "ambient space" variable \(\mathbf{Y}\). Despite this additional difficulty, numerical experiments --see Figure 3 for an illustration-- suggest that first order optimization methods can find global solutions to Equation 1.1, and our goal here is to take a first step in the objective of understanding this behavior mathematically.

Figure 4. (a) and (b) Sum of the norms of the gradients for a two-layer ReLU Neural Network. In (a), the network is initialized near the global optimal solution and in (b) the network is initialized near a saddle point. (c) shows the distance between the current outputs of the neural network and the optimal solution for the case when it was initialized near a saddle point. More details are presented in Appendix B.2.

Figure 3. (B) shows the first eigenvector for the Laplacian of a proximity graph from data points sampled from \(S^{2}\) obtained using an eigensolver. (A) shows the same eigenvector but obtained using SNN. The difference between the two figures is minor, showing that the neural network learns the eigenvector of the graph Laplacian well. See details in Appendix B.1.
To begin, we present some numerical experiments where we consider different initializations for the training of SNN. Here we take 100 data points from MNIST and let \(\mathcal{A}_{n}\) be the \(n\times n\) gram matrix for the data points for simplicity. We remark that while we care about a \(\mathcal{A}_{n}\) with a specific form for our approximation theory results, our analysis of the loss landscape described below holds for an arbitrary positive semi-definite matrix. In Figure 4, we plot the norm of the gradient during training when initialized in two different regions of parameter space. Concretely, in a region of parameters for which \(\mathbf{Y}_{\theta}\) is close to a solution \(\mathbf{Y}^{*}\) to problem 1.5 and a region of parameters for which \(\mathbf{Y}_{\theta}\) is close to a saddle point of the ambient loss \(\ell\). We compare these plots to the ones we produce from the gradient descent dynamics for the ambient problem 1.5, which are shown in Figure 5. We notice a similar qualitative behavior with the training dynamics of the NN, suggesting that the landscape of problem 1.1, if the NN is properly overparameterized, inherits properties of the landscape of \(\ell\).

Motivated by the previous observation, in section 3 we provide a careful landscape analysis of the loss function \(\ell\) introduced in Equation 1.1. We deem this landscape to be "benign", in the sense that it can be fully covered by the union of three regions described as follows: 1) the region of points close to global optimizers of Equation 1.5, where one can prove (Riemannian) strong convexity under a suitable quotient geometry; 2) the region of points close to saddle points, where one can find escape directions; and, finally, 3) the region where the gradient of \(\ell\) is large. Points in these regions are illustrated in Figures 4(b) and 4(c). The relevance of this global landscape characterization is that it implies convergence of most first-order optimization methods, or slight modifications thereof, toward global minimizers of the ambient space problem 1.5. This characterization is suggestive of analogous properties for the NN training problem in an overparameterized regime, but a full theoretical analysis of this is left as an open problem.

In summary, the main contributions of our work are the following:

* We show that we can approximate the eigenvectors of a large adjacency matrix with a NN, provided that the NN has sufficiently many neurons; see Theorem 2.1. Moreover, we show that by solving 1.1 one can _construct_ such approximation provided the parameter space of the NN is rich enough; see Theorem 2.2.

Figure 5. Norms of the gradients for the ambient problem and the distance to the optimal solution. In (a), \(\mathbf{Y}\) is initialized near the global optimal solution, and in (b) \(\mathbf{Y}\) is initialized near a saddle point. c) shows the distance between \(\mathbf{Y}\) and the optimal solution for the case when it was initialized near a saddle point.
* We provide precise error bounds for the approximation of eigenfunctions of a Laplace-Beltrami operator with NNs; see Corollary 1. In this way, we present an example of a setting where we can rigorously quantify the error of approximation of a solution to a PDE on a manifold with NNs.
* Motivated by numerical evidence, we begin an exploration of the optimization landscape of SNN and in particular provide a full description of SNN's associated ambient space optimization landscape. This landscape is shown to be benign; see discussion in Section 3.

### Related work

Spectral clustering and manifold learning.Several works have attempted to establish precise mathematical connections between the spectra of graph Laplacian operators over proximity graphs and the spectrum of weighted Laplace-Beltrami operators over manifolds. Some examples include Tao and Shi (2020), Burago et al. (2014), Garcia Trillos et al. (2020), Lu (2022), Calder and Garcia Trillos (2022), Calder et al. (2022), Dunson et al. (2021), Wormell and Reich (2021). In this paper we use adaptations of the results in Calder et al. (2022) to infer that, with very high probability, the eigenvectors of the normalized graph Laplacian matrix \(\Delta_{n}\) defined in Equation 1.4 are essentially Lipschitz continuous functions. These regularity estimates are one of the crucial tools for proving our Theorem 2.1.

Contrastive Learning.Contrastive learning is a self-supervised learning technique that has gained considerable attention in recent years due to its success in computer vision, natural language processing, and speech recognition Chen et al. (2020), Chen et al. (2020), Chen et al. (2020), He et al. (2020), Theoretical properties of contrastive representation learning were first studied by Arora et al. (2019), Tosh et al. (2021), Lee et al. (2021) where they assumed conditional independence. HaoChen et al. (2021) relaxes the conditional independence assumption by imposing the manifold assumption. With the spectral contrastive loss Equation 1.1 crucially in use, HaoChen et al. (2021) provides an error bound for downstream tasks. In this work, we analyze how the neural network can approximate and optimize the spectral loss function Equation 1.1, which is the pertaining step of HaoChen et al. (2021).

Neural Network Approximations.Given a function \(f\) with certain amount of regularity, many works have studied the tradeoff between width, depth, and total number of neurons needed and the approximation Petersen (2020), Lu et al. (2021). Specifically, Shen et al. (2019) looks at the problem Holder continuous functions on the unit cube, Yarotsky (2018), Shen et al. (2020) for continuous functions on the unit cube, and Petersen (2020), Schmidt-Hieber (2019), HaoChen et al. (2021) consider the case when the function is defined on a manifold. A related area is that of neural network memorization of a finite number of data points Yun et al. (2019). In this paper, we use these results to show that for our specific type of regularity, we can prove similar results.

Neural Networks and Partial Differential Equations.Raissi et al. (2019) introduced Physics Inspired Neural Networks as a method for solving PDEs using neural networks. Specifically, Weinan and Yu (2017), Bhatnagar et al. (2019), Raissi et al. (2019) use neural networks to parameterize the solution as use the PDE as the loss function. Other works such as Guo et al. (2016), Zhu and Zabaras (2018), Adler and Oktem (2017), Bhatnagar et al. (2019) use neural networks to parameterize
the solution operator on a given mesh on the domain. Finally, we have that eigenfunctions of operators on function spaces have a deep connection to PDEs. Recent works such as Kovachki et al. (2021), Li et al. (2020\(a\),_b_) demonstrate how to learn these operators. In this work we show that we can approximate eigenfunctions to a weighted Laplace-Beltrami operator using neural networks.

Shallow Linear Networks and Non-convex Optimization in Linear Algebra Problems.One of the main objects of study is the ambient problem Equation 1.1. This formulation of the problem is related to linear networks. Linear networks are neural networks with identity activation. A variety of prior works have studied many different aspects of shallow linear networks such as the loss landscape and optimization dynamics Baldi & Hornik (1989), Tarmoun et al. (2021_a_), Min et al. (2021), Brechet et al. (2023), and generalization for one layer networks Dobriban & Wager (2018), Hastie et al. (2022), Bartlett et al. (2020), Kausik et al. (2023). Of relevance are also other works in the literature studying optimization problems very closely related to Equation 1.5. For example, in Section 3 in Li & Tang (2017), there is a landscape analysis for problem 1.5 when the matrix \(\mathcal{A}_{\mathbf{n}}\) is assumed to have rank smaller than or equal to \(r\). That setting is typically referred to as overparameterized or exactly parameterized, whereas here our focus is on the underparameterized setting. On the other hand, the case studied in section 3 in Chi et al. (2019) is the simplest case we could consider for our problem and corresponds to \(r=1\). In this simpler case, the non-convexity of the objective is completely due to a sign ambiguity, which makes the analysis more straightforward and the need to introduce quotient geometries less pressing.

## 2. Spectral Approximation with neural networks

Through this section we make the following assumption on the generation process of the data \(\mathcal{X}_{n}\).

**Assumption 2.1**.: _The points \(x_{1},\ldots,x_{n}\) are assumed to be sampled from a distribution supported on an \(m\)-dimensional manifold \(\mathcal{M}\) that is assumed to be smooth, compact, orientable, connected, and without a boundary. We assume that this sampling distribution has a smooth density \(\rho:\mathcal{M}\to\mathbb{R}_{+}\) with respect to \(\mathcal{M}^{\prime}\)s volume form, and assume that \(\rho\) is bounded away from zero and also bounded above by a constant._

### Spectral approximation with multilayer ReLU NNs

**Theorem 2.1** (Spectral approximation of normalized Laplacians with neural networks).: _Let \(r\in\mathbb{N}\) be fixed. Under Assumptions 2.1, there are constants \(c,C\) that depend on \(\mathcal{M},\rho\), and the embedding dimension \(r\), such that, with probability at least_

\[1-C\varepsilon^{-6m}\exp\left(-cn\varepsilon^{m+4}\right)\]

_for every \(\delta\in(0,1)\) there are \(\kappa,L,p,N\) and a ReLU neural network \(f_{\theta}\in\mathcal{F}(r,\kappa,L,p,N)\) (defined in Equation C.2), such that:_

1. \(\sqrt{n}\|\mathbf{Y}_{\theta}-\mathbf{Y}^{*}\|_{\infty,\infty}\leq C(\delta+ \varepsilon^{2})\)_, and thus also_ \(\|\mathbf{Y}_{\theta}-\mathbf{Y}^{*}\|_{\mathrm{F}}\leq C\sqrt{r}(\delta+ \varepsilon^{2})\) _._
2. _The depth of the network,_ \(L\)_, satisfies:_ \(L\leq C\left(\log\frac{1}{\delta}+\log d\right)\)_, and its width,_ \(p\)_, satisfies_ \(p\leq C\left(\delta^{-m}+d\right)\)_._
3. _The number of neurons of the network,_ \(N\)_, satisfies:_ \(N\leq Cr\left(\delta^{-m}\log\frac{1}{\delta}+d\log\frac{1}{\delta}+d\log d\right)\)_, and the range of weights,_ \(\kappa\)_, satisfies_ \(\kappa\leq\frac{C}{n^{1/(2L)}}\)_._
Theorem 2.1 uses regularity properties of graph Laplacian eigenvectors and a NN approximation theory result for functions on manifolds. A summary of important auxiliary results needed to prove Theorem 2.1 is presented in Appendix E and the proof of the theorem itself is presented in Appendix F.

**Remark 2.1**.: _Any improvement of the approximations estimates in Chen et al. (2022) can be immediately applied to improve Theorem 2.1. We have relied on the results in Chen et al. (2022) due to the fact that in their estimates the ambient space dimension \(d\) does not appear in any exponent._

So far we have discussed approximations of the eigenvectors of \(\mathcal{A}_{\mathbf{n}}\) (and thus also of \(\Delta_{n}\)) with neural networks, but more can be said about generalization of these NNs. In particular, the NN in our proof of Theorem 2.1 can be shown to approximate eigenfunctions of the weighted Laplace-Beltrami operator \(\Delta_{\rho}\) defined in Appendix E.1. Precisely, we have the following result.

**Corollary 1**.: _Under the same setting, notation, and assumptions as in Theorem 2.1, the neural network \(f_{\theta}:\mathbb{R}^{d}\to\mathbb{R}^{r}\) can be chosen to satisfy_

\[\left\|\sqrt{\frac{n}{1+a}}f_{\theta}^{i}-f_{i}\right\|_{L^{\infty}(\mathcal{ M})}\leq C(\delta+\varepsilon),\quad\forall i=1,\ldots,r.\]

_In the above, \(f_{\theta}^{1},\ldots,f_{\theta}^{r}\) are the coordinate functions of the vector-valued neural network \(f_{\theta}\), and the functions \(f_{1},\ldots,f_{r}\) are the normalized eigenfunctions of the Laplace-Beltrami operator \(\Delta_{\rho}\) that are associated to \(\Delta_{\rho}\)'s \(r\) smallest eigenvalues._

**Remark 2.2**.: _The \(\varepsilon^{2}\) term that appears in the bound for \(\|\mathbf{Y}_{\theta}-\mathbf{Y}^{*}\|_{\mathrm{F}}\) in Theorem 2.1 cannot be obtained simply from convergence of eigenvectors of \(\Delta_{n}\) toward eigenfunctions of \(\Delta_{\rho}\) in \(L^{\infty}\). It turns out that we need to use a stronger notion of convergence (almost \(C^{0,1}\)) that in particular implies sharper regularity estimates for eigenvectors of \(\Delta_{n}\) (see Corollary 2 in Appendix E.1 and Remark E.2 below it). In turn, the sharper \(\varepsilon^{2}\) term is essential for our proof of Theorem 2.2 below to work; see the discussion starting in Remark E.2._

### Spectral approximation with global minimizers of SNN's objective

After discussing the _existence_ of approximating NNs, we turn our attention to _constructive_ ways to approximate \(\mathbf{Y}^{*}\) using neural networks. We give a precise answer to question Q2.

**Theorem 2.2** (Optimizing SNN approximates eigenvectors up to rotation).: _Let \(r\in\mathbb{N}\) be fixed and suppose that \(\Delta_{\rho}\) is such that \(\Delta_{\rho}\) has a spectral gap between its \(r\) and \(r+1\) smallest eigenvalues, i.e., in the notation in Appendix E.1, assume that \(\sigma_{r}^{\mathcal{M}}<\sigma_{r+1}^{\mathcal{M}}.\) For given \(\kappa,L,p,N\) (to be chosen below), let \(f_{\theta^{*}}\) be such that \(f_{\theta^{*}}\in\arg\min_{f_{\theta}\in\mathcal{F}(r,\kappa,L,p,N)}\| \mathbf{Y}_{\theta}\mathbf{Y}_{\theta}^{\dagger}-\mathcal{A}_{\mathbf{n}}\|_ {\mathrm{F}}^{2}.\)_

_Under Assumptions 2.1, there are constants \(c,C\) that depend on \(\mathcal{M},\rho\), and the embedding dimension \(r\), such that, with probability at least \(1-C\varepsilon^{-6m}\exp\left(-cn\varepsilon^{m+4}\right),\) for every \(\tilde{\delta}\in(0,c)\) (i.e., \(\tilde{\delta}\) sufficiently small) and for \(\kappa=\frac{C}{n^{1/(2L)}}\), \(L=C\left(\log\frac{1}{\delta\varepsilon}+\log d\right)\), \(p=C\left((\tilde{\delta}\varepsilon)^{-m}+d\right)\) and \(N=\infty\), we have_

\[\min_{\mathcal{Q}\in\mathbb{Q}_{r}}\|\mathbf{Y}_{\theta^{*}}-\mathbf{Y}^{*} \boldsymbol{O}\|_{\mathrm{F}}\leq C\varepsilon(\tilde{\delta}+\varepsilon). \tag{2.1}\]
**Remark 2.3**.: _Equation 2.1 says that \(\mathbf{Y}_{\theta^{*}}\) approximates a minimizer of the ambient problem 1.5 and that \(\mathbf{Y}_{\theta^{*}}\) can be recovered but only up to rotation. This is unavoidable, since the loss function \(\ell\) is invariant under multiplication on the right by a \(r\times r\) orthogonal matrix. On the other hand, to set \(N=\infty\) means we do not enforce sparsity constraints in the optimization of the NN parameters. This is convenient in practical settings and this is the reason why we state the theorem in this way. However, we can also set \(N=r\left((\tilde{\delta}\varepsilon)^{-m}\log\frac{1}{\delta\varepsilon}+d \log\frac{1}{\delta\varepsilon}+d\log d\right)\) without affecting the conclusion of the theorem._

## 3. Landscape of SNN's Ambient Optimization Problem

While in prior sections we considered a specific \(\mathcal{A}_{n}\), the analysis in this section only relies on \(\mathcal{A}_{\mathbf{n}}\) being positive definite with an eigengap between its \(r\)-th and \((r+1)\)th top eigenvalues. We analyze the global optimization landscape of the non-convex Problem 1.5 under a suitable Riemannian _quotient geometry_Absil et al. (2009), Boumal (2023). The need for a quotient geometry comes from the fact that if \(\mathbf{Y}\) is a stationary point of 1.5, then \(\mathbf{Y}\boldsymbol{O}\) is also a stationary point for any \(r\times r\) orthogonal matrix \(\boldsymbol{O}\in\mathbb{O}_{r}\). This implies that the loss function \(\ell\) is non-convex in any neighborhood of a stationary point (Li et al. 2019, Proposition 2). Despite the non-convexity of \(\ell\), we show that under this geometry, Equation 1.5 is geodesically convex in a local neighborhood around the optimal solution.

Let \(\overline{\mathcal{N}}_{r+}^{n}\) be the space of \(n\times r\) matrices with full column rank. To define the quotient manifold, we encode the invariance mapping, i.e., \(\mathbf{Y}\to\mathbf{Y}\boldsymbol{O}\), by defining the equivalence classes \([\mathbf{Y}]=\{\mathbf{Y}\boldsymbol{O}:\boldsymbol{O}\in\mathbb{O}_{r}\}\). From Lee (2018), we have \(\mathcal{N}_{r_{+}}^{n}\stackrel{{\text{def}}}{{=}}\overline{ \mathcal{N}}_{r_{+}}^{n}/\mathbb{O}_{r}\) is a quotient manifold of \(\overline{\mathcal{N}}_{r+}^{n}\). See a detailed introduction to Riemannian optimization in Boumal (2023). Since the loss function in 1.5 is invariant along the equivalence classes of \(\overline{\mathcal{N}}_{r_{+}}^{n}\), \(\ell\) induces the following optimization problem on the quotient manifold \(\mathcal{N}_{r_{+}}^{n}\):

\[\min_{[\mathbf{Y}]\in\mathcal{N}_{r_{+}}^{n}}H([\mathbf{Y}])\stackrel{{ \text{def}}}{{=}}\frac{1}{2}\left\|\mathbf{Y}\mathbf{Y}^{\top}- \mathcal{A}_{\mathbf{n}}\right\|_{\text{F}}^{2} \tag{3.1}\]

To analyze the landscape for Equation 3.1, we need expressions for the Riemannian gradient, the Riemannian Hessian, as well as the geodesic distance \(d\) on this quotient manifold. By Lemma 2 from Luo & Garcia Trillos (2022), we have that

\[d\left(\left[\mathbf{Y}_{1}\right],\left[\mathbf{Y}_{2}\right]\right)=\min_{ \mathbf{Q}\in\mathbb{O}_{r}}\left\|\mathbf{Y}_{2}\mathbf{Q}-\mathbf{Y}_{1} \right\|_{\text{F}}\]

and from Lemma 3 from Luo & Garcia Trillos (2022), we have that

\[\overline{\operatorname{grad}H([\mathbf{Y}])} =2\left(\mathbf{Y}\mathbf{Y}^{\top}-\mathcal{A}_{\mathbf{n}} \right)\mathbf{Y},\] \[\overline{\operatorname{Hess}H([\mathbf{Y}])}\left[\theta_{\mathbf{ Y}},\theta_{\mathbf{Y}}\right] =\left\|\mathbf{Y}\theta_{\mathbf{Y}}^{\top}+\theta_{\mathbf{Y}}\mathbf{Y}^{ \top}\right\|_{\text{F}}^{2}+2\left\langle\mathbf{Y}\mathbf{Y}^{\top}- \mathcal{A}_{\mathbf{n}},\theta_{\mathbf{Y}}\theta_{\mathbf{Y}}^{\top} \right\rangle. \tag{3.2}\]

Finally, by the classical theory on low-rank approximation (Eckart-Young-Mirsky theorem Eckart & Young (1936)), \([\mathbf{Y}^{*}]\) is the unique global minimizer of Equation 3.1. Let \(\kappa^{*}=\sigma_{1}\left(\mathbf{Y}^{*}\right)/\sigma_{r}\left(\mathbf{Y}^{*}\right)\) be the condition number of \(\mathbf{Y}^{*}\). Here, \(\sigma_{i}(A)\) is the \(i^{\text{th}}\) largest singular value of \(A\), and \(\|A\|=\sigma_{1}(A)\) is its spectral norm. Our precise assumption on the matrix \(\mathcal{A}_{n}\) for this section is as follows.

**Assumption 3.1** (Eigengap).: \(\sigma_{r+1}(\mathcal{A}_{\mathbf{n}})\) _is strictly smaller than \(\sigma_{r}(\mathcal{A}_{\mathbf{n}})\)._
Let \(\mu,\alpha,\beta,\gamma\geqslant 0\). We then split the landscape of \(H([\mathbf{Y}])\) into the following five regions (not necessarily non-overlapping).

\[\mathcal{R}_{1}\stackrel{{\mathrm{def}}}{{=}}\left\{ \mathbf{Y}\in\mathbb{R}_{*}^{n\times r}\middle|d\left([\mathbf{Y}],[\mathbf{Y}^ {*}]\right)\leqslant\mu\sigma_{r}\left(\mathbf{Y}^{*}\right)/\kappa^{*}\right\},\] \[\mathcal{R}_{2}\stackrel{{\mathrm{def}}}{{=}} \left\{\mathbf{Y}\in\mathbb{R}_{*}^{n\times r}\middle|\begin{array}{l}d \left([\mathbf{Y}],[\mathbf{Y}^{*}]\right)>\mu\sigma_{r}\left(\mathbf{Y}^{*} \right)/\kappa^{*},\|\overline{\mathrm{grad}\,H([\mathbf{Y}])}\|_{\mathrm{F}} \leqslant\alpha\mu\sigma_{r}^{3}\left(\mathbf{Y}^{*}\right)/\left(4\kappa^{*} \right),\\ \|\mathbf{Y}\|\leqslant\beta\left\|\mathbf{Y}^{*}\right\|,\left\|\mathbf{YY}^ {\top}\right\|_{\mathrm{F}}\leqslant\gamma\left\|\mathbf{Y}^{*}{\mathbf{Y}^ {*}}^{\top}\right\|_{\mathrm{F}}\end{array}\right\},\] \[\mathcal{R}_{3}^{\prime}\stackrel{{\mathrm{def}}}{{=}} \left\{\mathbf{Y}\in\mathbb{R}_{*}^{n\times r}\middle|\begin{array}{l}\| \overline{\mathrm{grad}\,H([\mathbf{Y}])}\|_{\mathrm{F}}>\alpha\mu\sigma_{r}^{ 3}\left(\mathbf{Y}^{*}\right)/\left(4\kappa^{*}\right),\|\mathbf{Y}\|\leqslant \beta\left\|\mathbf{Y}^{*}\right\|,\\ \|\mathbf{Y}\mathbf{Y}^{\top}\|_{\mathrm{F}}\leqslant\gamma\left\|\mathbf{Y}^ {*}{\mathbf{Y}^{*}}^{\top}\right\|_{\mathrm{F}}\end{array}\right\},\] \[\mathcal{R}_{3}^{\prime\prime}\stackrel{{\mathrm{def}}}{{=}} \left\{\mathbf{Y}\in\mathbb{R}_{*}^{n\times r}\middle|\|\mathbf{Y}\mathbf{Y}^ {\top}\right\|_{\mathrm{F}}>\gamma\left\|\mathbf{Y}^{*}{\mathbf{Y}^{*}}^{\top }\right\|_{\mathrm{F}}\end{array}\right\}, \tag{3.3}\]

We show that for small values of \(\mu\), the _loss function is geodesically convex_ in \(\mathcal{R}_{1}\). \(\mathcal{R}_{2}\) is then defined as the region outside of \(\mathcal{R}_{1}\) such that the Riemannian gradient is small relative to \(\mu\). Hence this is the region in which we are close to the saddle points. We show that for this region there is _always an escape direction_ (i.e., directions where the Hessian is strictly negative). Finally, \(\mathcal{R}_{3}^{\prime}\), \(\mathcal{R}_{3}^{\prime\prime}\), and \(\mathcal{R}_{3}^{\prime\prime\prime}\) are the remaining regions. We show that the _Riemannian gradient is large_ (relative to \(\mu\)) in these regions. Finally, it is easy to see that \(\mathcal{R}_{1}\bigcup\mathcal{R}_{2}\bigcup\mathcal{R}_{3}^{\prime}\cup \mathcal{R}_{3}^{\prime\prime}\bigcup\mathcal{R}_{3}^{\prime\prime\prime}= \mathbb{R}_{*}^{n\times r}\).

We are now ready to state the first of our main results from this section.

**Theorem 3.1** (Local Geodesic Strong Convexity and Smoothness of Equation 3.1).: _Suppose \(0\leqslant\mu\leqslant\kappa^{*}/3\). Given that Assumption 3.1 holds, for any \(\mathbf{Y}\in\mathcal{R}_{1}\) defined in Equation 3.3._

\[\sigma_{\min}(\overline{\mathrm{Hess}\,H([\mathbf{Y}])})\geqslant\left(2\, (1-\mu/\kappa^{*})^{2}-(14/3)\mu\right)\sigma_{r}\left(\mathcal{A}_{\mathbf{n} }\right)-2\sigma_{r+1}(\mathcal{A}_{\mathbf{n}}),\] \[\sigma_{\max}(\overline{\mathrm{Hess}\,H([\mathbf{Y}])})\leqslant 4 \left(\sigma_{1}\left(\mathbf{Y}^{*}\right)+\mu\sigma_{r}\left(\mathbf{Y}^{*} \right)/\kappa^{*}\right)^{2}+14\mu\sigma_{r}^{2}\left(\mathbf{Y}^{*}\right)/3\]

_In particular, if \(\mu\) is further chosen such that \(\left(2\,(1-\mu/\kappa^{*})^{2}-(14/3)\mu\right)\sigma_{r}\left(\mathcal{A}_{ \mathbf{n}}\right)-2\sigma_{r+1}(\mathcal{A}_{\mathbf{n}})>0\), we have \(H([\mathbf{Y}])\) is geodesically strongly convex and smooth in \(\mathcal{R}_{1}\)._

Theorem 3.1 guarantees that the optimization problem Equation 3.1 is geodesically strongly convex and smooth in a neighborhood of \([\mathbf{Y}^{*}]\). It also shows that if \(\mathbf{Y}\) is close to the global minimizer, then Riemannian gradient descent converges to the global minimizer of the quotient space linearly.

Next, to analyze \(\mathcal{R}_{2}\), we need to understand the other first-order stationary points (FOSP).

**Theorem 3.2** (FOSP of Equation 3.1).: _Let \(\mathbf{Y}^{*}=\overline{\mathbf{U}}\cdot\overline{\mathbf{\Lambda}}\cdot \overline{\mathbf{V}}^{\top}\) and \(\mathbf{Y}=\mathbf{U}\mathbf{D}\mathbf{V}^{\top}\) be the SVDs. Then for any \(S\) subset of \([n]\), we have that \([\overline{\mathbf{U}}_{S}\mathbf{\Lambda}_{S}\mathbf{V}_{S}^{\top}]\) is a Riemannian FOSPs of Equation 3.1. Further, these are the only Riemannian FOSPs._

Theorem 3.2 shows that the linear combinations of eigenvectors can be used to construct Riemannian first-order stationary points (FOSP) of Equation 3.1. This theorem also shows that there are many FOSPs of Equation 3.2. This is quite different from the regime studied in Luo & Garcia Trillos (2022). In general, gradient descent is known to converge to a FOSP. Hence one might expect that if we initialized near one of the saddle points, then we might converge to that saddle point.
However, our next main result of the section shows that even if we initialize near the saddle, there always exist escape directions.

**Theorem 3.3** (Escape Directions).: _Assume that Assumption 3.1 holds. Then for sufficiently small \(\alpha\) and any \(\mathbf{Y}\in\mathcal{R}_{2}\) that is not an FOSP, there exists \(C_{1}(\mathcal{A}_{\mathbf{n}})>0\) and \(\theta_{\mathbf{Y}}\) such that_

\[\overline{\operatorname{Hess}H([\mathbf{Y}])}\left[\theta_{\mathbf{Y}},\theta _{\mathbf{Y}}\right]\leqslant-\,C_{1}(\mathcal{A}_{\mathbf{n}})\left\|\theta_ {\mathbf{Y}}\right\|_{\mathrm{F}}^{2}.\]

In particular, it is possible to exactly quantify the size of \(\alpha\) and the explicitly construct the escape direction \(\theta_{\mathbf{Y}}\). See Theorem H.1 in the appendix for more details.

**Remark 3.1**.: _Theorem 3.3 guarantees that if \(\mathbf{Y}\) is close to some saddle points, then \(\theta_{\mathbf{Y}}\) will make its escape from the saddle point linearly._

Finally, the next result tells that is we are not close to a FOSP, then we have large gradients.

**Theorem 3.4** ((Regions with Large Riemannian Gradient of Equation 1.5).:
1. \(\|\overline{\operatorname{grad}H([\mathbf{Y}])}\|_{\mathrm{F}}>\alpha\mu \sigma_{r}^{3}\left(\mathbf{Y}^{*}\right)/\left(4\kappa^{*}\right),\,\forall \mathbf{Y}\in\mathcal{R}_{3}^{\prime}\)_;_
2. \(\|\overline{\operatorname{grad}H([\mathbf{Y}])}\|_{\mathrm{F}}\;\geqslant\;2 \left(\|\mathbf{Y}\|^{3}-\|\mathbf{Y}\|\,\|\mathbf{Y}^{*}\|^{2}\right)\;>\;2 \left(\beta^{3}-\beta\right)\|\mathbf{Y}^{*}\|^{3}\,,\quad\forall\mathbf{Y}\in \mathcal{R}_{3}^{\prime\prime}\)_;_
3. \(\langle\overline{\operatorname{grad}H([\mathbf{Y}])},\mathbf{Y}\rangle>2(1-1/ \gamma)\left\|\mathbf{Y}\mathbf{Y}^{\top}\right\|_{\mathrm{F}}^{2},\quad \forall\mathbf{Y}\in\mathcal{R}_{3}^{\prime\prime\prime}.\)__

_In particular, if \(\beta>1\) and \(\gamma>1\), we have the Riemannian gradient of \(H([\mathbf{Y}])\) has large magnitude in all regions \(\mathcal{R}_{3}^{\prime},\mathcal{R}_{3}^{\prime\prime}\) and \(\mathcal{R}_{3}^{\prime\prime\prime}\)._

**Remark 3.2**.: _These results can be seen as an under-parameterized generalization to the regression problem of Section 5 in Luo & Garcia Trillos (2022). The proof in Luo & Garcia Trillos (2022) is simpler because in their setting there are no saddle points or local minima that are not global. Conceptually, Tarmoun et al. (2021b) proves that in the setting \(r\geq n\), the gradient flow for Equation 1.5 converges to a global minimum linearly. We complement this result by studying the case \(r<n\)._

**Remark 3.3**.: _To demonstrate strong geodesic convexity, the eigengap assumption is necessary as it prevents multiple global solutions. However, it is possible to relax this assumption and instead deduce a PL condition, which would also imply a linear convergence rate for a first-order method._

**Remark 3.4**.: _In the specific case of \(\mathcal{A}_{\mathbf{n}}\) as in Equation 1.3, and under Assumptions 2.1, Assumption 3.1 should be interpreted as \(\sigma_{r}^{\mathcal{M}}<\sigma_{r+1}^{\mathcal{M}}\), as suggested by Remark E.1. Also, \(\mu\) must be taken to be in the order \(\varepsilon^{2}\). The scale \(\varepsilon^{2}\) is actually a natural scale for this problem, since, as discussed in Remark G.3, the energy gap between saddle points and the global minimizer \([\mathbf{Y}^{*}]\) is \(O(\varepsilon^{2})\)._

## 4. Conclusions

We have explored some theoretical aspects of Spectral Neural Networks (SNN), a framework that substitutes the use of traditional eigensolvers with suitable neural network parameter optimization. Our emphasis has been on approximation theory, specifically identifying the minimum number of neurons of a multilayer NN required to capture spectral geometric properties in data, and investigating the optimization landscape of SNN, even in the face of its non-convex ambient loss function.
For our approximation theory results we have assumed a specific proximity graph structure over data points that are sampled from a distribution over a smooth low-dimensional manifold. A natural future direction worth of study is the generalization of these results to settings where data points, and their similarity graph, are sampled from other generative models, e.g., as in the application to contrastive learning in HaoChen et al. (2021). To carry out this generalization, an important first step is to study the regularity properties of eigenvectors of an adjacency matrix/graph Laplacian generated from other types of probabilistic models.

At a high level, our approximation theory results have sought to bridge the extensive body of research on graph-based learning methods, their ties to PDE theory on manifolds, and the approximation theory for neural networks. While our analysis has focused on eigenvalue problems, such as those involving graph Laplacians or Laplace Beltrami operators, we anticipate that this overarching objective can be extended to develop provably consistent methods for solving a larger class of PDEs on manifolds with neural networks. We believe this represents a significant and promising research avenue.

On the optimization front, we have focused on studying the landscape of the ambient space problem 1.5. This has been done anticipating the use of our estimates in a future analysis of the training dynamics of SNN. We reiterate that the setting of interest here is different from other settings in the literature that study the dynamics of neural network training in an appropriate scaling limit --leading to either a neural tangent kernel (NTK) or to a mean field limit. This difference is mainly due to the fact that the spectral contrastive loss \(\ell\) (see 1.1) of SNN is non-convex, and even local strong convexity around a global minimizer does not hold in a standard sense and instead can only be guaranteed when considered under a suitable quotient geometry.
