Good Lattice Training: Physics-Informed Neural Networks Accelerated by
  Number Theory
Physics-informed neural networks (PINNs) offer a novel and efficient approach
to solving partial differential equations (PDEs). Their success lies in the
physics-informed loss, which trains a neural network to satisfy a given PDE at
specific points and to approximate the solution. However, the solutions to PDEs
are inherently infinite-dimensional, and the distance between the output and
the solution is defined by an integral over the domain. Therefore, the
physics-informed loss only provides a finite approximation, and selecting
appropriate collocation points becomes crucial to suppress the discretization
errors, although this aspect has often been overlooked. In this paper, we
propose a new technique called good lattice training (GLT) for PINNs, inspired
by number theoretic methods for numerical analysis. GLT offers a set of
collocation points that are effective even with a small number of points and
for multi-dimensional spaces. Our experiments demonstrate that GLT requires
2--20 times fewer collocation points (resulting in lower computational cost)
than uniformly random sampling or Latin hypercube sampling, while achieving
competitive performance.
Takashi Matsubara, Takaharu Yaguchi
# Good Lattice Training: Physics-Informed Neural Networks Accelerated by Number Theory

###### Abstract

Physics-informed neural networks (PINNs) offer a novel and efficient approach to solving partial differential equations (PDEs). Their success lies in the physics-informed loss, which trains a neural network to satisfy a given PDE at specific points and to approximate the solution. However, the solutions to PDEs are inherently infinite-dimensional, and the distance between the output and the solution is defined by an integral over the domain. Therefore, the physics-informed loss only provides a finite approximation, and selecting appropriate collocation points becomes crucial to suppress the discretization errors, although this aspect has often been overlooked. In this paper, we propose a new technique called _good lattice training_ (GLT) for PINNs, inspired by number theoretic methods for numerical analysis. GLT offers a set of collocation points that are effective even with a small number of points and for multi-dimensional spaces. Our experiments demonstrate that GLT requires 2-20 times fewer collocation points (resulting in lower computational cost) than uniformly random sampling or Latin hypercube sampling, while achieving competitive performance.

## 1 Introduction

Many real-world phenomena can be modeled as partial differential equations (PDEs), and solving PDEs has been a central topic in the field of computational science. The applications include, but are not limited to, weather forecasting, vehicle design [14], economic analysis [2], and computer vision [23]. A PDE is defined as \(\mathcal{N}[u]=0\), where \(\mathcal{N}\) is a (possibly nonlinear) differential operator, \(u:\Omega\rightarrow\mathbb{R}\) is an unknown function, and \(\Omega\subset\mathbb{R}^{s}\) is the domain. For most PDEs that appear in physical simulations, the well-posedness (i.e., the uniqueness of the solution \(u\) and the continuous dependence on the initial and the boundary conditions) has been well studied, and this is typically guaranteed under certain conditions. Various computational methods, such as finite difference, finite volume, and spectral methods, have been explored for solving PDEs [9; 29; 39]. However, the development of computer architecture has become slower, leading to a growing need for computationally efficient alternatives. A promising approach is physics-informed neural networks (PINNs) [33], which train a neural network so that its output \(\tilde{u}\) satisfies the PDE \(\mathcal{N}[\tilde{u}]=0\). Specifically, given a PDE, PINNs train a neural network to minimize the squared error \(\|\mathcal{N}[\tilde{u}](\mathbf{x}_{i})\|^{2}\) at a finite set of collocation points \(\mathbf{x}_{i}\) so that the output \(\tilde{u}\) satisfies the equation \(\mathcal{N}[\tilde{u}](\mathbf{x}_{i})=0\). This objective, known as the physics-informed loss (or PINNs loss), constitutes the core idea of PINNs [42; 43].

However, the solutions \(u\) to PDEs are inherently infinite-dimensional, and the distance involving the output \(\tilde{u}\) or the solution \(u\) must be defined by an integral over the domain \(\Omega\). In this regard, the physics-informed loss is a finite approximation to the squared 2-norm \(|\mathcal{N}[\tilde{u}]|_{2}^{2}=\int_{\Omega}\|\mathcal{N}[\tilde{u}](\mathbf{x}_ {i})\|^{2}\mathrm{d}\mathbf{x}\) on the function space \(L^{2}(\Omega)\) for \(\mathcal{N}[u]\in L^{2}(\Omega)\), and hence the discretization errors should affect the training efficiency. A smaller number \(N\) of collocation points leads to a less accurate approximation
and inferior performance, while a larger number \(N\) increases the computational cost [3; 37]. Despite the importance of selecting appropriate collocation points, insufficient emphasis has been placed on this aspect. Raissi et al. [33], Zeng et al. [48], and many other studies employed Latin hypercube sampling (LHS) to determine the collocation points. Alternative approaches include uniformly random sampling [15; 21] and uniformly spaced sampling [44; 45]. These methods are exemplified in Fig. 1.

In the field of numerical analysis, the relationship between integral approximation and collocation points has been extensively investigated. Inspired by number theoretic methods for numerical analysis, we propose _good lattice training (GLT)_ for PINNs and their variants. The GLT provides an optimal set of collocation points, as shown in Fig. 1, when the activation functions of the neural networks are smooth enough. Our experiments demonstrate that the proposed GLT requires far fewer collocation points than comparison methods while achieving similar errors, significantly reducing computational cost. The contribution and significance of the proposed GLT are threefold.

Computationally Efficient:The proposed GLT offers an optimal set of collocation points to compute a loss function that can be regarded as a finite approximation to an integral over domain, such as the physics-informed loss, if the activation functions of the neural networks are smooth enough. It requires significantly fewer collocation points to achieve accuracy comparable to other methods, or can achieve lower errors with the same computational budget.

Applicable to PINNs Variants:As the proposed GLT involves changing only the collocation points, it can be applied to various versions of PINNs without modifying the learning algorithm or objective function. In this study, we investigate a specific variant, the competitive PINN (CPINN) [48], and demonstrate that CPINNs using the proposed GLT achieve superior convergence speed with significantly fewer collocation points than CPINNs using LHS.

Theoretically Solid:Number theory provides a theoretical basis for the efficacy of the proposed GLT. This guarantees the general applicability of GLT and enables users to assess its suitability for a given problem setting prior to implementation.

## 2 Related Work

Neural Networks for Data-Driven PhysicsNeural networks have emerged as a powerful tool for processing information from data and have achieved significant success in various fields such as image recognition and natural language processing in recent years [12; 41]. They are also employed for black-box system identification, that is, to learn the differential equations that govern the dynamics of a target system and predict its future behavior [4; 5; 46]. By integrating knowledge from analytical mechanics, neural networks can learn the dynamics that adheres to physical laws, such as the conservation of energy, and even uncover these laws from data [8; 10; 28].

Physics-Informed Neural NetworksNeural networks have also gained attention as computational tools for solving differential equations, particularly PDEs [6; 22]. Recently, Raissi et al. [33] introduced an elegant refinement to this approach and named it PINNs. The key concept behind PINNs is the physics-informed loss, also known as PINNs loss [42; 43]. This loss function evaluates the extent to which the output \(\tilde{u}\) of the neural network satisfies a given PDE \(\mathcal{N}[u]=0\) and its associated initial and boundary conditions \(\mathcal{B}[u]=0\). The physics-informed loss can also be integrated into other models like DeepONet [25; 43]. These advancements have been made possible by recent developments in automatic differentiation algorithms, which allow us to automatically compute the partial derivatives of neural networks' outputs [31].

Figure 1: Examples of 144 sampled collocation points.
PINNs have already been applied to various PDEs [3; 15; 27], and considerable efforts have been dedicated to improving learning algorithms and objective functions [11; 13; 26; 32; 37; 48]. In these studies, the loss functions based on PDE rather than data, including the original physics-informed loss, are defined by evaluating errors at a finite set of collocation points. As shown in [3; 37], there is a trade-off between the number of collocation points (and hence computational cost) and the accuracy of the solution. Thus, selecting collocation points that efficiently cover the entire domain is essential for achieving better results, although this aspect has often been overlooked. An exceptional instance is found in Mao et al. [27], in which more collocation points were sampled around regions where the solution is discontinuous, efficiently capturing the discontinuity. However, such an approach is not always feasible.

## 3 Method

PreliminaryFor simplicity, we consider the cases where the target PDE is defined on an \(s\)-dimensional unit cube \([0,1]^{s}\). Otherwise, the equations should be transformed to such a domain by an appropriate coordinate transformation. Methods for reducing computations on complex regions to those on rectangular regions have been studied in the research field of grid generation for domain decomposition methods. See [18; 40] for example.

PINNs employ the PDE that describes the target physical phenomena as loss functions. Specifically, first, the appropriate collocation points \(\{\mathbf{x}_{j}\mid j=1,\ldots,N\}\) are fixed, and then the sum of the residuals of the PDE at these points

\[\frac{1}{N}\sum_{j=1}^{N}\|\mathcal{N}[\tilde{u}](\mathbf{x}_{j})\|^{2} \tag{1}\]

is minimized. \(\tilde{u}\) is an approximate solution of the equation represented by a neural network. This type of loss function is referred to as a physics-informed loss. Although the sum of such residuals over collocation points is practically minimized, for \(\tilde{u}\) to be the exact solution, this physics-informed loss should be 0 for all \(\mathbf{x}\), not just at collocation points. Therefore, the loss function defined by the following integral must be minimized:

\[\int_{[0,1]^{s}}\|\mathcal{N}[\tilde{u}](\mathbf{x})\|^{2}\mathrm{d}\mathbf{x}. \tag{2}\]

In other words, the practical minimization of (1) essentially minimizes the approximation of (2) with the expectation that (2) will be small enough and hence \(\tilde{u}\) becomes an accurate approximation to the exact solution.

One of the typical methods often used in practice is to set \(\mathbf{x}_{j}\)'s to be uniformly distributed random numbers. PINNs typically target PDEs in multi-dimensional domains, and the loss function is often integrated over a certain multi-dimensional domain. In such multi-dimensional integration, the effect of the curse of dimensionality cannot be ignored. The method of sampling collocation points from the uniform distributions can be interpreted as the Monte Carlo approximation of the value of the integral (2). As is well known, the Monte Carlo method is not affected by the curse of dimensionality and can approximate the integral within an error of \(O(1/\sqrt{N})\)[38]. Therefore, such a choice of collocation points is reasonable for computing the physics-informed loss in multi-dimensional domains. However, if feasible, a method less affected by the curse of dimensionality and more efficient than the Monte Carlo method is desired. In this paper, we propose such a method, which is inspired by number theoretic numerical analysis.

Good Lattice Training: Number Theoretic Method for Computing Physics-Informed LossIn this section, we propose the _good lattice training (GLT)_, in which a number theoretic method is used to accelerate the training of PINNs. The proposed method is inspired by the number theoretic numerical analysis [30; 38; 47], and hence in the following, we use some tools from this theory. First, we define a lattice for computing the loss function.

**Definition 1** ([30; 38; 47]).: _A lattice \(L\) in \(\mathbb{R}^{s}\) is defined as a finite set of points in \(\mathbb{R}^{s}\) that is closed under addition and subtraction._

Given a lattice \(L\), the set of collocation points is defined as \(L^{*}=\{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\}\coloneqq L\cap[0,1]^{s}\). In this paper, we consider an appropriate lattice for computing the physics-informed loss. Considering
that the loss function to be minimized is (2), it is natural to determine the lattice \(L\) (and hence the set of collocation points \(\mathbf{x}_{j}\)'s) so that the difference \(|(2)-(1)|\) of the two loss functions is minimized.

Suppose that \(\varepsilon(\mathbf{x}):=\|\mathcal{N}[\tilde{u}](\mathbf{x})\|^{2}\) is smooth enough and admits the Fourier series expansion:

\[\varepsilon(\mathbf{x})\coloneqq\|\mathcal{N}[\tilde{u}](\mathbf{x})\|^{2}=\sum_{\mathbf{ h}}\hat{\varepsilon}(\mathbf{h})\exp(2\pi\mathrm{i}\mathbf{h}\cdot\mathbf{x}),\]

where \(\mathrm{i}\) denotes the imaginary unit and \(\mathbf{h}=(h_{1},h_{2},\ldots,h_{s})\in\mathbb{Z}^{s}\). Substitution of this Fourier series into \(|(2)-(1)|\) yields

\[\begin{split}\left|\int_{[0,1]^{s}}\|\mathcal{N}[\tilde{u}](\bm {x})\|^{2}\mathrm{d}\mathbf{x}-\frac{1}{N}\sum_{j=1}^{N}\sum_{\mathbf{h}\in\mathbb{Z}^ {s}}\hat{\varepsilon}(\mathbf{h})\exp(2\pi\mathrm{i}\mathbf{h}\cdot\mathbf{x}_{j})\right| \\ =\left|\frac{1}{N}\sum_{j=1}^{N}\sum_{\mathbf{h}\in\mathbb{Z}^{s},\bm {h}\neq 0}\hat{\varepsilon}(\mathbf{h})\exp(2\pi\mathrm{i}\mathbf{h}\cdot\mathbf{x}_{j}) \right|,\end{split} \tag{3}\]

where the equality follows from the fact that the Fourier mode of \(\mathbf{h}=0\) is equal to the integral \(\int_{[0,1]^{s}}\|\mathcal{N}[\tilde{u}](\mathbf{x}_{j})\|^{2}\mathrm{d}\mathbf{x}\). Before optimizing (3), the dual lattice of lattice \(L\) and an insightful lemma are introduced as follows.

**Definition 2** ([30, 38, 47]).: _A dual lattice \(L^{\top}\) of a lattice \(L\) is defined as \(L^{\top}\coloneqq\{\mathbf{h}\in\mathbb{R}^{s}\mid\mathbf{h}\cdot\mathbf{x}\in\mathbb{Z},\ ^{\forall}\mathbf{x}\in L\}\)._

**Lemma 1** ([30, 38, 47]).: _For \(\mathbf{h}\in\mathbb{Z}^{s}\), it holds that_

\[\frac{1}{N}\sum_{j=1}^{N}\exp(2\pi\mathrm{i}\mathbf{h}\cdot\mathbf{x}_{j})=\begin{cases} 1&(\mathbf{h}\in L^{\top})\\ 0&(\mathrm{otherwise}.)\end{cases}\]

Based on this lemma, we restrict the lattice point \(L\) to the form \(\{\mathbf{x}\mid\mathbf{x}=\mathbf{s}+\frac{j}{N}\mathbf{z}\text{ for }\mathbf{s}\in\mathbb{Z}^{s},j\in \mathbb{Z}\}\) with a fixed integer vector \(\mathbf{z}\); the set \(L^{*}\) of collocation points is \(L\cap[0,1]^{s}=\{\text{decimal part of }\frac{j}{N}\mathbf{z}\mid j=0,\ldots,N-1\}\). Then, instead of searching \(\mathbf{x}_{j}\)'s, a vector \(\mathbf{z}\) is searched. This form has the following advantages. By restricting to this form, \(\mathbf{x}_{j}\)'s can be obtained automatically from a given \(\mathbf{z}\), and hence the optimal collocation points \(\mathbf{x}_{j}\)'s do not need to be stored as a table of numbers, making a significant advantage in implementation. Another advantage is theoretical; the optimization problem of the collocation points can be reformulated in a number theoretic way. In fact, for \(L\) as shown above, it is confirmed that \(L^{\top}=\{\mathbf{h}\mid\mathbf{h}\cdot\mathbf{z}\equiv 0\pmod{N}\}\). If \(\mathbf{h}\cdot\mathbf{z}\equiv 0\pmod{N}\) then there exists an \(m\in\mathbb{Z}\) such that \(\mathbf{h}\cdot\mathbf{z}=mN\) and hence \(\frac{j}{N}\mathbf{h}\cdot\mathbf{z}=mj\in\mathbb{Z}\). Conversely, if \(\mathbf{h}\cdot\mathbf{z}\not\equiv 0\pmod{N}\), clearly \(\frac{1}{N}\mathbf{h}\cdot\mathbf{z}\notin\mathbb{Z}\).

Therefore, from the above lemma, (3) is bounded from above by

\[\sum_{\mathbf{h}\in\mathbb{Z}^{s},\mathbf{h}\neq 0,\mathbf{h}\cdot\mathbf{z}\equiv 0\pmod{N}}| \hat{\varepsilon}(\mathbf{h})|, \tag{4}\]

and hence the collocation points \(\mathbf{x}_{j}\)'s should be determined so that (4) becomes small. This problem is a number theoretic problem in the sense that it is a minimization problem of finding an integer vector \(\mathbf{h}\) subject to the condition \(\mathbf{h}\cdot\mathbf{z}\equiv 0\pmod{N}\). This problem has been considered in the field of number theoretic numerical analysis. In particular, optimal solutions have been investigated for integrands in the Korobov spaces, which are spaces of functions that satisfy a certain smoothness condition.

**Definition 3** ([30, 38, 47]).: _The function space that is defined as \(E_{\alpha}=\{f:[0,1]^{s}\to\mathbb{R}\mid\exists c,|\hat{f}(\mathbf{h})|\leq c/( \bar{h}_{1}\bar{h}_{2}\cdots\bar{h}_{s})^{\alpha}\}\) is called the Korobov space, where \(\hat{f}(\mathbf{h})\) is the Fourier coefficients of \(f\) and \(k=\max(1,|k|)\) for \(k\in\mathbb{R}\)._

It is known that if \(\alpha\) is an integer, for a function \(f\) to be in \(E_{\alpha}\), it is sufficient that \(f\) has continuous partial derivatives \(\partial^{q_{1}+q_{2}+\cdots+q_{s}}f/\partial_{1}^{q_{1}}\cdot\partial_{2}^{q_ {2}}\cdots\partial_{s}^{q_{s}},0\leq q_{k}\leq\alpha\ (k=1,\ldots,s)\). For example, if a function \(f(x,y):\mathbb{R}^{2}\to\mathbb{R}\) has continuous \(f_{x},f_{y},f_{xy}\), then \(f\in E_{1}\).

The main result of this paper is the following.
**Theorem 1**.: _Suppose that the activation function of \(\tilde{u}\) and hence \(\tilde{u}\) itself are sufficiently smooth so that there exists an \(\alpha>0\) such that \(\|\mathcal{N}[\tilde{u}]\|^{2}\in E_{\alpha}\). Then, for given integers \(N\geq 2\) and \(s\geq 2\), there exists an integer vector \(\mathbf{z}\in\mathbb{Z}^{s}\) such that \(L^{*}=\{\text{decimal part of }\frac{j}{N}\mathbf{z}\mid j=0,\ldots,N-1\}\) is a "good lattice" in the sense that_

\[\int_{[0,1]^{s}}\|\mathcal{N}[\tilde{u}](\mathbf{x})\|^{2}\mathrm{d}\mathbf{x}\leq \frac{1}{N}\sum_{\mathbf{x}_{j}\in L^{*}}\|\mathcal{N}[\tilde{u}](\mathbf{x}_{j})\|^{2 }+O\left(\frac{(\log N)^{\alpha s}}{N^{\alpha}}\right). \tag{5}\]

For a proof of this theorem, see Appendix A. In this paper, we call the learning method that minimizes

\[\frac{1}{N}\sum_{\mathbf{x}_{j}\in L^{*}}\|\mathcal{N}[\tilde{u}](\mathbf{x}_{j})\|^{2}\]

for a lattice \(L\) that satisfies (5) _the good lattice training (GLT)_.

The proposed GLT, of which the order of the convergence is \(O(\frac{(\log N)^{\alpha s}}{N^{\alpha}})\), can converge much faster to the integrated loss (2) than the uniformly random sampling (i.e., the Monte Carlo method, of which the order of the convergence is \(O(1/\sqrt{N})\)) if the activation function of \(\tilde{u}\) and hence the neural network \(\tilde{u}\) itself are sufficiently smooth so that \(\mathcal{N}[\tilde{u}]\in E_{\alpha}\) for a large \(\alpha\). Furthermore, the error increases only in proportion to \((\log N)^{\alpha s}\) as the dimension increases. Therefore, the proposed GLT remains effective for high-dimensional problems, almost unaffected by the curse of dimensionality. This makes PINNs available for higher dimensional problems.

Although the above theorem shows the existence of a good lattice, we can obtain a good lattice in the following way in practice. When \(s=2\), we can use the Fibonacci sequence \(1,1,2,3,5,8,13,\ldots\), which is defined by \(F_{1}=F_{2}=1,\;F_{k}=F_{k-1}+F_{k-2}\;(k\geq 3)\). It is known that by setting \(N=F_{k}\) and \(\mathbf{z}=(1,F_{k-1})\)[30, 38], we obtain a good lattice. For example, when \(N=13\), \(\mathbf{z}=(1,8)\) gives a good lattice \(L=\{(0,0),(\frac{1}{13},\frac{8}{13}),(\frac{2}{13},\frac{3}{13}),(\frac{3}{13 },\frac{11}{13}),\ldots\}\).

When \(s=2\) and \(N\) is not a Fibonacci number, or when \(s>2\), there is no known method to generate a good lattice using a sequence of numbers. However, it is possible to numerically find the optimal \(\mathbf{z}\) for a given \(N\) with a computational cost of \(O(N^{2})\). The procedure is as follows: First, if \(\|\mathcal{N}[\tilde{u}]\|^{2}\) belongs to Koborov space, then (4) can be estimated as

\[\sum_{\mathbf{h}\in\mathbb{Z}^{s},\mathbf{h}\neq 0,\mathbf{h}\cdot\mathbf{z}\equiv 0\pmod{N}} \frac{c}{(\bar{h}_{1}\bar{h}_{2}\cdots\bar{h}_{s})^{\alpha}}. \tag{6}\]

In particular, essentially this upper bound is achieved when \(\|\mathcal{N}[\tilde{u}]\|^{2}\) becomes the following function:

\[\|\mathcal{N}[\tilde{u}]\|^{2}=\prod_{k=1}^{s}F_{\alpha}(x_{k}),\quad F_{ \alpha}(x_{k})=1+\sum_{h\in\mathbb{Z},h\neq 0}\frac{\exp(2\pi\mathrm{i}hx_{k})}{|h |^{\alpha}},\]

Actually, when \(\alpha\) is even, this function is known to be explicitly written by using the Bernoulli polynomials \(B_{\alpha}(x)\)[38]:

\[F_{\alpha}(x_{k})=1-(-1)^{\frac{\alpha}{2}}\frac{(2\pi)^{\alpha}B_{\alpha}}{ \alpha!}.\]

Since this is a polynomial function, the integral can be found exactly; hence it is possible to find a loss function for this function for each lattice \(L\). Therefore, to find an optimal \(\mathbf{z}\), the loss function with respect to the above function should be minimized. The computational cost for computing the loss function for each \(L\) is \(O(N)\).

Even considering that each component of \(\mathbf{z}\) is in \(\{0,\ldots,N-1\}\), searching for the optimal \(\mathbf{z}\in\mathbb{Z}^{s}\) would require the computational cost \(O(N^{s})\); however, if \(N\) is a prime number or the product of two prime numbers, the existence of a \(\mathbf{z}\) of the form \(\mathbf{z}=(1,l,l^{2}\pmod{N},\ldots,l^{s-1}\pmod{N})\) that gives a good lattice is known [19, 20, 38]. Since \(l\in\{0,1,\ldots,N-1\}\), there are only \(N\) candidates for \(l\), and hence the computational cost for finding the optimal \(\mathbf{z}\) is only \(O(N^{2})\) for each \(N\). Furthermore, as this optimization process can be fully parallelized, the computational time is quite short practically. The values of \(\mathbf{z}\) have been explored in the field of number theoretic numerical analysis, and typical values are available as numerical tables in, e.g., [7, 16].
LimitationsNot limited to the proposed GLT, but many methods to determine collocation points are not directly applicable to non-rectangular or non-flat domain \(\Omega\)[36]. To achieve the best performance, it is recommended to deform the domain \(\Omega\) to fit a flat rectangle. For such a technique, see, e.g., [18; 40]. In addition, because the proposed GLT is based on the Fourier series expansion, it is basically assumed that the loss function is periodic. This problem can be addressed by variable transformations [38]. For example, if an \(s\)-dimensional integral \(\int_{[0,1]^{s}}f(\mathbf{x})\mathrm{d}\mathbf{x}\) is to be computed, the integrand \(f\) can be transformed to a periodic one by using a smooth increasing function \(\phi:[0,1]\rightarrow[0,1]\) with \(\phi^{\prime}(0)=\phi^{\prime}(1)=0\):

\[\int_{[0,1]^{s}}\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!
equations. The state of the NLS equation is complex; we simply treated it as a 2D real vector for training and used its absolute value for evaluation and visualization. Following Raissi et al. [33], we evaluated the performance using the relative error, which is the normalized squared error \(\mathcal{L}(\tilde{u},u;\mathbf{x}_{i})=(\sum_{i=1}^{N_{\mathbf{x}_{i}}}\|\tilde{u}(\mathbf{ x}_{i})-u(\mathbf{x}_{i})\|^{2})^{1/2}/(\sum_{i=1}^{N_{\mathbf{x}_{i}}}\|u(\mathbf{x}_{i})\|^{2 })^{1/2}\) at predefined \(N_{e}\) collocation points \(\mathbf{x}_{i}\). This is also a finite approximation to \(|\tilde{u}-u|_{2}/|u|_{2}\). For Poisson's equation with \(s=2\), we followed the original learning strategy using the L-BFGS-B method preceded by the Adam optimizer [17] to ensure precise convergence. For other datasets, we trained PINNs using the Adam optimizer with cosine decay of a single cycle to zero [24] for 200,000 iterations and sampling a different set of collocation points at each iteration; we found that this strategy improves the final performance. See Appendix B for details.

We determined the collocation points using four different methods: uniformly random sampling, uniformly spaced sampling, LHS, and the proposed GLT. For the GLT, we took the number \(N\) of collocation points and the corresponding integer vector \(\mathbf{z}\) from numerical tables in [7, 16]. We also applied random shifts in (7). To maintain consistency, we used the same value of \(N\) for both uniformly random sampling and LHS. For uniformly spaced sampling, we selected the square number \(N\) (or the biquadratic number when \(s=4\)) that was closest to a number used in the GLT. Subsequently, we created a unit cube of \(N^{1/s}\) points on each side. Additionally, we introduced random shifts to the collocation points, with \(r_{\bullet}\sim U([0,1/N^{1/s}])\), ensuring that the collocation points were uniformly distributed. Then, we transformed the unit cube \([0,1]^{s}\) to the original domain \(\Omega\).

We conducted five trials for each number \(N\) and each method. All experiments were conducted using Python v3.7.16 and tensorflow v1.15.5 [1] on servers with Intel Xeon Platinum 8368.

Results: Accuracy of Physics-Informed LossFirst, we evaluate the accuracy of the physics-informed loss (1) in approximating the integrated loss (2). Regardless of the method, as the number \(N\) of collocation points increases, the physics-informed loss (1) approaches the integrated loss (2). With fewer collocation points, the physics-informed loss varies significantly depending on the selection of collocation points, resulting in a larger standard deviation. We trained the PINNs on the NLS equation using LHS with \(N=610\) collocation points. Then, we evaluated the physics-informed loss of the obtained PINNs for different methods and different numbers of collocation points. For each combination of method and number, we performed 1,000 trials and summarized their standard deviation of the physics-informed loss in Fig. 2.

Except for the GLT, the other methods exhibit a similar trend, showing a linear reduction in standard deviation on the log-log plot. This trend aligns with the theoretical result that the convergence rate of Monte Carlo methods is \(O(1/\sqrt{N})\). On the other hand, the GLT demonstrates an accelerated reduction as the number \(N\) increases. This result implies that by using the GLT, the physics-informed loss (1) approximates the integrated loss (2) more accurately with the same number \(N\) of collocation points, leading to faster training and improved accuracy.

Results: Performance of PINNsFigure 3 shows the average relative error \(\mathcal{L}\) with solid lines and the maximum and minimum errors with shaded areas. These results demonstrate that as \(N\) increases, the relative error \(\mathcal{L}\) decreases and eventually reaches saturation. This is because of numerical errors in the datasets, discretization errors in relative error \(\mathcal{L}\), and the network capacity.

We report the minimum numbers \(N\) of collocation points with which the relative error \(\mathcal{L}\) was saturated in Table 1. Specifically, we consider a relative error \(\mathcal{L}\) below 130 % of the minimum observed one as saturated; the thresholds are denoted by horizontal red lines in Fig. 3. The proposed GLT demonstrated equivalent performance with significantly fewer collocation points. This suggests that the proposed GLT can achieve comparable performance with significantly less computational cost. Next, we equalized the number \(N\) of collocation points (and therefore, the computational cost). In the bottom rows of Table 1, we report the relative error \(\mathcal{L}\) for collocation points with which the relative error \(\mathcal{L}\) of one of the comparison methods saturated, as denoted by vertical green lines in Fig. 3. A smaller loss \(\mathcal{L}\) indicates that the method performed better than others with the same computational cost. The

Figure 2: The number \(N\) of collocation points and the standard deviation of the physics-informed loss.
proposed GLT yielded nearly half or less of the relative error \(\mathcal{L}\) for the NLS, KdV, and AC equations, and the difference is significant for Poisson's equation with \(s=2\). We show the true solutions and the residuals of example results with such \(N\) in Fig. 4. We also investigated longer training with smaller mini-batch sizes in Appendix C.

\begin{table}
\begin{tabular}{l l r r r r} \hline \hline  & & NLS & KdV & AC & \multicolumn{2}{c}{Poisson} \\ \cline{4-6}  & & & & \(s=2\) & \(s=4\) \\ \hline \multirow{3}{*}{\# of points \(N^{\dagger}\)} & \(\bullet\) uniformly random & 6,765 & 10,946 & 4,181 & \(>\)17,711 & 1,019 \\  & \(\bullet\) uniformly spaced & 1,600 & 6,724 & 11,025 & 6,724 & 1,296 \\  & \(\bullet\) LHS & 4,181 & 17,711 & 10,000 & \(>\)17,711 & 562 \\  & \(\bullet\) GLT (proposed) & **610** & **987** & **987** & **1,597** & **307** \\ \hline \multirow{3}{*}{relative error \(\mathcal{L}^{\ddagger}\)} & \(\bullet\) uniformly random & 7.60 & 3.32 & 1.82 & 4.141 & 28.7 \\  & \(\bullet\) uniformly spaced & 4.88 & 3.25 & 1.79 & 1.859 & 29.2 \\  & \(\bullet\) LHS & 6.45 & 2.94 & 1.43 & 6.174 & 24.3 \\  & \(\bullet\) GLT (proposed) & **1.55** & **1.90** & **0.74** & **0.027** & **16.4** \\ \hline \multicolumn{6}{l}{\# of points \(N\) at competitive relative error \(\mathcal{L}\) (under horizontal red line in Fig. 3).} \\ \multicolumn{6}{l}{\(\ddagger\) relative error \(\mathcal{L}\) at competitive \# of points \(N\) (on vertical green line in Fig. 3).} \\ \multicolumn{6}{l}{Shown in the scale of \(10^{-5}\) for Poisson’s equation and \(10^{-3}\) for others.} \\ \end{tabular}
\end{table}
Table 1: Trade-Off between Number \(N\) of Collocation Points and Relative Error \(\mathcal{L}\).

Figure 4: Example results at competitive number \(N\) of collocation points (on vertical green line in Fig. 3). The leftmost panel shows the true solution. The remaining panels show the residuals of PINNs’ results using uniformly random sampling, uniformly spaced sampling, LHS, and GLT, from left to right. The residuals are multiplied by the factors in parentheses.

Figure 3: The number \(N\) of collocation points and the relative error \(\mathcal{L}\).
### Competitive Physics-Informed Neural Networks

Experimental SettingsCompetitive PINNs (CPINNs) are an improved version of PINNs that additionally uses a neural network \(D:\Omega\rightarrow\mathbb{R}\) called a discriminator [48]. Its objective function is \(\frac{1}{N}\sum_{i=1}^{N}D(\mathbf{x}_{i})\mathcal{N}[\vec{u}](\mathbf{x}_{i})\); the discriminator \(D\) is trained to maximize it, whereas the neural network \(\vec{u}\) is trained to minimize it. This setting is regarded as a zero-sum game, and its Nash equilibrium offers the solution to a given PDE. Moreover, this setting enables to use the (adaptive) competitive gradient descent (CGD) algorithm [34; 35], accelerating the convergence. The objective function is also regarded as a finite approximation to the integral \(\int_{\Omega}D(\mathbf{x})\mathcal{N}[\vec{u}](\mathbf{x})\mathrm{d}\mathbf{x}\); therefore, the proposed GLT can be applied to CPINNs.

We modified the code accompanying the manuscript and investigated the NLS and Burgers' equations2. The NLS equation is identical to the one above. See Appendix B for details about Burgers' equation. The number \(N\) of collocation points was 20,000 by default and varied. All experiments were conducted using Python v3.9.16 and Pytorch v1.13.1 [31] on servers with Intel Xeon Platinum 8368 and NVIDIA A100.

Footnote 2: See Supplementary Material at [https://openreview.net/forum?id=z9SIj-IM7tn](https://openreview.net/forum?id=z9SIj-IM7tn) (MIT License)

ResultsWe summarized the results in Fig. 5. For the NLS equation, when the collocation points were determined by LHS, the relative error \(\mathcal{L}\) of CPINNs decreased slowly even with \(N=20,000\), and there was almost no improvement for \(N\leq 2,584\). On the other hand, when the proposed GLT was used, the error decreased rapidly even with \(N=2,584\). In the case of the Burgers' equation, CPINNs using GLT made some progress in learning with \(N=2,584\) collocation points, while CPINNs using LHS required \(N=6,765\) collocation points. These results indicate that the proposed GLT exhibits competitive or superior convergence speed with 2.5 to 8 times fewer collocation points. The original paper demonstrated that CPINNs have a faster convergence rate than vanilla PINNs, but the GLT can further accelerate it.

## 5 Conclusion

This paper highlighted that the physics-informed loss, commonly used in PINNs and their variants, is a finite approximation to the integrated loss. From this perspective, it proposed good lattice training (GLT) to determine collocation points. This method enables a more accurate approximation of the integrated loss with a smaller number of collocation points. Experimental results using PINNs and CPINNs demonstrated that the GLT can achieve competitive or superior performance with much fewer collocation points. These results imply a significant reduction in computational cost and contribute to the large-scale computation of PINNs.

We focused on the physics-informed loss in the domain. If the domain is \(s\geq 3\)-dimensional, the initial and boundary conditions are of \(s\geq 2\) dimensions, and their loss functions are also subject to the proposed GLT. As shown in Figs. 3 and 5, the current problem setting reaches performance saturation with around \(N=1,000\) collocation points. However, Figure 2 demonstrates that the GLT significantly enhances the approximation accuracy even when using further more collocation points. This implies that the GLT is particularly effective in addressing larger-scale problem settings, which will be explored further in future research.

Figure 5: The number of iterations and the relative error \(\mathcal{L}\).