MultiAdam: Parameter-wise Scale-invariant Optimizer for Multiscale
  Training of Physics-informed Neural Networks
Physics-informed Neural Networks (PINNs) have recently achieved remarkable
progress in solving Partial Differential Equations (PDEs) in various fields by
minimizing a weighted sum of PDE loss and boundary loss. However, there are
several critical challenges in the training of PINNs, including the lack of
theoretical frameworks and the imbalance between PDE loss and boundary loss. In
this paper, we present an analysis of second-order non-homogeneous PDEs, which
are classified into three categories and applicable to various common problems.
We also characterize the connections between the training loss and actual
error, guaranteeing convergence under mild conditions. The theoretical analysis
inspires us to further propose MultiAdam, a scale-invariant optimizer that
leverages gradient momentum to parameter-wisely balance the loss terms.
Extensive experiment results on multiple problems from different physical
domains demonstrate that our MultiAdam solver can improve the predictive
accuracy by 1-2 orders of magnitude compared with strong baselines.
Jiachen Yao, Chang Su, Zhongkai Hao, Songming Liu, Hang Su, Jun Zhu
# MultiAdam: Parameter-wise Scale-invariant Optimizer for

###### Abstract

Physics-informed Neural Networks (PINNs) have recently achieved remarkable progress in solving Partial Differential Equations (PDEs) in various fields by minimizing a weighted sum of PDE loss and boundary loss. However, there are several critical challenges in the training of PINNs, including the lack of theoretical frameworks and the imbalance between PDE loss and boundary loss. In this paper, we present an analysis of second-order non-homogeneous PDEs, which are classified into three categories and applicable to various common problems. We also characterize the connections between the training loss and actual error, guaranteeing convergence under mild conditions. The theoretical analysis inspires us to further propose MultiAdam, a scale-invariant optimizer that leverages gradient momentum to parameter-wisely balance the loss terms. Extensive experiment results on multiple problems from different physical domains demonstrate that our MultiAdam solver can improve the predictive accuracy by 1-2 orders of magnitude compared with strong baselines.

Machine Learning, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Nonlinear Optimization, Multi-Adam, Multi-Adam, Nonlinear Optimization, Multi-Adam, Multi-Adam, Nonlinear Optimization, Multi-Adam, Multi-Adam, Nonlinear Optimization, Multi-Adam, Multi-Adam, Nonlinear Optimization, Multi-Adam, Multi-Adam, Nonlinear Optimization, Multi-Adam, Multi-Adam, Nonlinear Optimization, Multi-Adam, Multi-Adam, Nonlinear Optimization, Multi-Adam, Multi-Adam, Nonlinear Optimization, Multi-Adam, Multi-Adam, Nonlinear Optimization, Multi-Adam, Multi-Adam, Nonlinear Optimization, Multi-Adam, Multi-Adam, Nonlinear Optimization, Multi-Adam, Multi-Adam, Nonlinear Optimization, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Nonlinear Optimization, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam, Multi-Adam, Multi-Adam, Multi-Adam, Adam Multi-Adam,
reweighting. Some works (Wight and Zhao, 2020; Elhamod et al., 2022) use manual hyper-parameters to adjust the weights. However, these non-adaptive methods depend on empirical conclusions, which can lead to sub-optimal results. More research works focus on adaptively balancing PINN losses. For example, (Wang et al., 2021) designed a learning rate annealing algorithm using statistics of back-propagated gradients. (Wang et al., 2022) proposed another method to adjust weights from the perspective of Neural Tangent Kernel (NTK). In (Bai et al., 2022), the loss function is modified using the Least Squares Weighted Residual (LSWR) method. Nevertheless, these methodologies primarily concentrate on modifying loss functions, implying that they consider the effect on parameters as a whole. As such, they might overlook the impact of domain scaling on individual parameters of the model.

In this paper, we aim to address the above issues to effectively train PINNs. Specificially, we first present a theoretical error analysis of loss functions for different types of PDEs under mild conditions. This analysis provides a connection between the loss function and the actual performance of the model by bounding the \(L^{\infty}\) error with its PDE loss and boundary loss. This new error boundary not only ensures convergence towards the ground truth under a sufficiently low loss but also serves as an optimization objective as its minimization enables the neural network to approach the true solution more effectively. (Wang et al., 2022)'s work supports that \(L^{\infty}\) loss is a better choice that \(L^{2}\) loss.

Building on the upper-bound error, we propose a scale-invariant optimizer, MultiAdam. MultiAdam leverages the observation that the second momentum of Adam acts as an excellent indicator of the gradient scale. We categorize losses of different scales into separate groups, maintaining the second momentum individually for each group. This momentum is subsequently utilized to re-scale the gradients, aligning them to a nearly identical scale. Extensive experiments demonstrate that the MultiAdam optimizer is robust against unbalanced losses and is effective in various complex PDEs across different domain scales. Moreover, MultiAdam exhibits remarkable stability and a high convergence rate under these conditions.

The rest of the paper is organized as follows. In section 2, we briefly review existing variants of PINNs, especially reweighting techniques. In section 3, we go over the original PINN model and Adam optimizer. Section 4 introduces the effect of domain scaling on PINN losses using an example of 2D Poisson's equation, followed by an introduction to our new optimizer MultiAdam. Then we provide a theoretical analysis on error bounds for PINNs and show the connection between the existing problem and MultiAdam. Section 5 presents numerical experiments and evaluates MultiAdam using a range of representative benchmark examples. Finally, Section 6 encapsulates our findings and contributions.

## 2 Related Work

**Physics-informed Neural Networks** (PINNs) (Raissi et al., 2019) are capable of learning to represent the nonlinear relationship in dynamic systems and providing fast predictions (Karniadakis et al., 2021). However, theoretical analysis for PINNs is typically insufficient. For some special equations such as Kolmogorov equations and Navier-Stokes equations, the total error can be estimated with regard to the training loss and network settings (De Ryck and Mishra, 2022; De Ryck et al., 2022). A more general result is attained on second-order elliptic equations, where the convergence of PINNs is proved (Shin et al., 2020) and the \(L^{\infty}\) error bound is given (Peng et al., 2020) under mild constraints. Yet it still remains unclear for many other PDE problems. Also, analyzing the convergence and accuracy of PINNs is of tremendous challenge, especially for systems with multi-scale characteristics (Li and Feng, 2022). PINNs are commonly optimized by Adam (Kingma and Ba, 2014) and L-BFGS (Liu and Nocedal, 1989). However, they often reach ill situations when the scale and convergence rate of loss terms vary significantly (Hao et al., 2022).

**Reweighting techniques for PINNs** To correct the imbalance, a standard approach is the introduction of weights in the loss functions (McClenny and Braga-Neto, 2020). Currently, several adaptive reweighting methods have been proposed. (Wang et al., 2021) designed a learning rate annealing algorithm using statistics of back-propagated gradients to mitigate the pathology. Neural Tangent Kernel also provides a novel perspective to adaptively adjust the weights (Wang et al., 2022). In (Bai et al., 2022), the loss function is modified using the LSWR method to alleviate the biased training issue.

**Multitask learning methods** The PINN optimization can be regarded as a multitask learning problem since each equation and boundary condition is an individual objective. Therefore, it is also worthwhile to learn from multitask learning (MTL). GradNorm (Chen et al., 2018) and PCGrad (Yu et al., 2020) are two promising approaches along this line. GradNorm tunes gradient magnitudes based on the average gradient norm and the relative training rate of each task, while PCGrad projects the conflicting gradients onto the normal plane.

## 3 Preliminaries

### Physics-Informed Neural Networks

The main objective for Physics-Informed Neural Networks (PINNs) is to solve a physical system using known phys
ical laws and available data. Assume the system can be described by the following PDEs:

\[\begin{split}& f(x;\frac{\partial u}{\partial x_{1}},\cdots,\frac{ \partial u}{\partial x_{d}};\frac{\partial^{2}u}{\partial x_{1}^{2}},\frac{ \partial^{2}u}{\partial x_{1}\partial x_{2}},\cdots;\lambda)=0\\ & B(u,x)=0,\forall x\in\partial\Omega\end{split} \tag{1}\]

where \(f\) is the differential equation, \(u\) is the solution to that equation, \(\Omega\) is the domain and \(\partial\Omega\) is the boundary of it. Moreover, \(\lambda\) is an additional parameter and \(B\) is the boundary condition.

To solve the physical system, PINNs use neural networks to approximate the solution of PDEs. In order to train a neural network meeting all the constraints in Eq. (1), PINNs transform the equations into loss functions defined as follows:

\[\begin{split}& L_{f}(\theta,\lambda;T_{f})=\frac{1}{|T_{f}|}\sum_{x \in T_{f}}\|f(x,\frac{\partial\hat{u}_{\theta}}{\partial x_{1}},\ldots;\frac{ \partial^{2}\hat{u}_{\theta}}{\partial x_{1}^{2}},\cdots;\lambda)\|_{2}^{2} \\ & L_{b}(\theta,\lambda;T_{b})=\frac{1}{|T_{b}|}\sum_{x\in T_{b}}\| B(\hat{u}_{\theta},x)\|_{2}^{2}\end{split} \tag{2}\]

where \(L_{f}\) is the residual loss for the PDE, and \(L_{b}\) is the loss for boundary condition. \(u_{\theta}\) is the prediction by neural network with parameter \(\theta\) and \(T_{f},T_{b}\) are sampling points.

The overall training objective of PINN is then defined as a weighted sum of the two losses:

\[L(\theta,\lambda;T)=w_{f}L_{f}(\theta,\lambda;T_{f})+w_{b}L_{b}(\theta, \lambda;T_{b}) \tag{3}\]

where \(w_{f},w_{b}\) are the non-negative weights for different losses. To effectively train a PINN, we have to optimize the two loss terms at the same time and make every loss as low as possible. Therefore, it is natural to treat it as a multitask learning problem.

### Adam Optimizer

The Adaptive Momentum Estimation (Adam), as proposed by (Kingma & Ba, 2014), is a commonly adopted optimization method for PINNs. It maintains the moving average of the squared gradient, known as the second momentum, to adjust the learning rate for each parameter. The specifics of this algorithm can be seen in Algorithm 1. Despite Adam's robust capability to minimize a single loss function for neural networks, it may struggle with handling multiple optimization objectives. Consequently, the network may fail to converge if the weights in Eq. (3) are not appropriately configured. A detailed discussion on this matter will be provided in Section 4.1.

```
0: learning rate \(\gamma\), betas \(\beta_{1},\beta_{2}\), max epoch \(M\), objective function \(f(\theta)\)
1:for all\(t=1\) to \(M\)do
2:\(g_{t}\leftarrow\nabla_{\theta}f(\theta_{t-1})\)
3:\(m_{t}\leftarrow\beta_{1}m_{t-1}+(1-\beta_{1})g_{t}\)
4:\(v_{t}\leftarrow\beta_{2}v_{t-1}+(1-\beta_{2})g_{t}^{2}\)
5:\(\hat{m}_{t}\gets m_{t}/(1-\beta_{1}^{*})\)
6:\(\hat{v}_{t}\gets v_{t}/(1-\beta_{2}^{*})\)
7:\(\theta_{t}\leftarrow\theta_{t-1}-\gamma\hat{m}_{t,i}/(\sqrt{\hat{v}_{t,i}}+\varepsilon)\)
8:endfor
9:return\(\theta_{t}\)
```

**Algorithm 1** Adam

## 4 Method

We now present our method in detail, starting with an analysis of the imbalance between the terms in the loss objective.

### The effect of domain scaling on loss balancing

We first observe that the PDE loss and boundary loss may be several orders of magnitude away in real cases, leading to a failure to approach the correct solution by the standard Adam optimizer. One of the main reasons for the issue is the improper scaling of the domain. Most PDEs are not scaling invariant, which causes the change in domain to rescale PDE loss. The influence is characterized by the following theorem:

**Theorem 4.1** (Effect of scaling for homogeneous PDEs, Proof in Appendix B.1).: _Suppose \(\Omega\) is the domain of a homogeneous PDE of \(k\) order and \(L^{2}\) loss is used for PINNs. Then, if we narrow the domain by \(t\) times, the boundary loss will stay fixed while the PDE loss will be multiplied by \(t^{2k}\)._

We illustrate this with an example of Poisson's equation in a complex domain. The reference solution is depicted in Figure 1, with the detailed setup available in Appendix A. In this case, we condense the original domain, which spans an \(8\times 8\) square, by a factor of \(8\), resulting in a \(1\times 1\) square. As shown in Figure 2, when training on the \(8\times 8\) domain, the PDE loss and boundary loss do not significantly differ. However, when training on the \(1\times 1\) domain, the PDE loss is nearly \(8^{4}\) times larger than the boundary loss. This substantial discrepancy poses considerable challenges in training PINNs, as demonstrated in Figure 1.

This example further exposes the gap between the loss function that PINN optimizes and its actual performance. In Figure 3, we train PINN on the \(1\times 1\) domain using two different settings--one incorporating manual reweighting of loss while the other not. In the absence of manual reweighting, PINN fails to approach the ground truth. Yet, its loss is lower than that of the reweighted scenario for the first \(10000\) epochs, during which its \(L^{2}\) rela
to the ground truth is significantly higher compared to the reweighted scenario. This suggests that the loss optimized in PINN does not reliably represent the actual performance in this case.

### Error Analysis

Considering the observed inconsistency between total loss and actual performance, we find it crucial to revisit the well-posedness of our objective function, i.e., we question whether optimization based on the loss indeed leads to improved solutions. We offer a theoretical examination of the relationship between loss and error. Given that the majority of PDEs employed across various disciplines do not exceed second order, and that linear ones are relatively prevalent and simpler to analyze, our study primarily concentrates on elliptic, parabolic, and select hyperbolic equations. These represent the majority of second-order linear PDEs (Strauss, 2007). Based on the error bounds, we establish links between the losses in PINNs and the absolute error of the PINN output.

Specifically, we provide error bounds for three types of PDEs separately in the following theorems. Thanks to Theorem 2.1 and Corollary 2.2 in (Peng et al., 2020), we can directly obtain the proof of Error bounds of PINNs on elliptic PDEs as follows:

**Theorem 4.2** (Error bounds of PINNs on elliptic PDEs).: _Suppose \(\Omega\subset\mathbb{R}^{d}\) is a bounded domain, \(\mathcal{L}\) is an elliptic operator and \(\tilde{u}\in C^{0}(\overline{\Omega})\cap C^{2}(\Omega)\) is a solution to the following PDE:_

\[\begin{split}\mathcal{L}[u](x)&=f(x),\ \forall x\in\Omega\\ u(x)&=g(x),\ \forall x\in\partial\Omega\end{split} \tag{4}\]

_If the output \(u_{\theta}\) of the PINN with parameter \(\theta\) satisfies:_

\[\begin{split}& u_{\theta}\in C^{0}(\overline{\Omega})\cap C^{2}( \Omega)\\ &\sup_{x\in\partial\Omega}|u_{\theta}-\tilde{u}|<\delta_{1}\\ &\sup_{x\in\Omega}|\mathcal{L}[u_{\theta}]-f|<\delta_{2},\end{split} \tag{5}\]

_then the absolute error over \(\Omega\) is upper-bounded:_

\[\sup_{x\in\Omega}|u_{\theta}-\tilde{u}|\leq\delta_{1}+C\delta_{2}. \tag{6}\]

_Here, \(C\) is a constant depending only the operator \(\mathcal{L}\) and the domain \(\Omega\). If \(\operatorname{diam}\Omega=d\), then \(C\) is proportional to \(e^{d}-1\) when \(\operatorname{diam}\Omega\) changed._

And we further provide the Error bounds of PINNs on Parabolic PDEs and Hyperbolic PDEs in Theorem 4.3 and 4.4, respectively. The detailed proof is included in the Appendix.

**Theorem 4.3** (Error bounds of PINNs on Parabolic PDEs, proof in Appendix B.2).: _Suppose \(\Omega\subset\mathbb{R}^{d}_{x}\times\mathbb{R}_{t}\) is a bounded domain, \(\mathcal{L}\) is an parabolic operator and \(\tilde{u}\in C^{0}(\overline{\Omega})\cap C^{2}(\Omega)\) is a solution to the PDE in equation 4. If the output \(u_{\theta}\) of the PINN with parameter \(\theta\) satisfies:_

\[\begin{split}& u_{\theta}\in C^{0}(\overline{\Omega})\cap C^{2}( \Omega)\\ &\sup_{x\in\partial\Omega}|u_{\theta}-\tilde{u}|<\delta_{1}\\ &\sup_{x\in\Omega}|\mathcal{L}[u_{\theta}]-f|<\delta_{2},\end{split} \tag{7}\]

_then the absolute error over \(\Omega\) is upper-bounded:_

\[\sup_{x\in\Omega}|u_{\theta}-\tilde{u}|\leq C_{1}(\delta_{1}+C\delta_{2}), \tag{8}\]

_where \(C,C_{1}\) are constants depending only on \(\Omega\) and \(\mathcal{L}\). If \(\operatorname{diam}\Omega=d\), then \(C\) is proportional to \(e^{\alpha d}-1\) when \(\operatorname{diam}\Omega\) changed._

**Theorem 4.4** (Error Bounds for PINNs on Hyperbolic PDEs, proof in Appendix B.3).: _Suppose \(\Omega\subset\mathbb{R}_{x}\times\mathbb{R}^{+}_{t}\) is an admissible domain (defined in Appendix B.3) and \(\mathcal{L}\) is an hyperbolic operator satisfies the requirements in Appendix B.3. If the PINN with parameter \(\theta\) satisfies that:_

\[\begin{split}& u_{\theta}\in C^{1}(\overline{\Omega})\cap C^{2}( \Omega)\\ &\sup_{x\in\partial\Omega}|u_{\theta}-\tilde{u}|<\delta_{1}\\ &\sup_{x\in\Omega}|\mathcal{L}[u_{\theta}]-f|<\delta_{2}\end{split} \tag{9}\]

_Then, we have:_

\[\sup_{x\in\Omega}|u_{\theta}-\tilde{u}|\leq\delta_{1}+C\delta_{2}\]

_where \(C\) is constant depending only on \(\Omega\) and \(\mathcal{L}\). If \(\operatorname{diam}\Omega=d\), then \(C\) is proportional to \(e^{\alpha d}-1\) when \(\operatorname{diam}\Omega\) changed._

We finally provide how to control the absolute error using PINNs' \(L^{2}\) loss as

**Theorem 4.5** (Control Absolute Error using PINNs' \(L^{2}\) Loss, proof in Appendix B.4).: _Suppose the second-order PDE operator \(\mathcal{L}\) and the PINN with parameter \(\theta\) satisfy that:_

\[\sup_{x\in\Omega}|u_{\theta}-\tilde{u}|\leq C_{1}(\sup_{x\in\partial\Omega}|u _{\theta}-\tilde{u}|+C\sup_{x\in\Omega}|\mathcal{L}[u_{\theta}]-f|) \tag{10}\]

_where \(C,C_{1}\) are constants. Then, the error can be bounded by \(L^{2}\) loss of the PINN:_

\[\|u_{\theta}-\tilde{u}\|_{L_{\infty}}\leq C_{2}(\sqrt{L_{b}}+C\sqrt{L_{f}}) \tag{11}\]

_where \(C_{2}\) is constant depend on \(C_{1}\) and selection of sampling points and base functions (used in proof). The detailed definition is in Appendix B.4._
Theorem 4.5 delineates the relationship between the loss of PINNs and the actual error. Although the unweighted sum of losses does not directly reflect the performance of PINNs, the introduction of appropriate weights to the losses can ensure a more accurate correspondence to error. This underlines the necessity of reweighting techniques for PINNs. Broadly, the more precise the estimate of \(C\), the narrower the gap between the optimization objective and the actual error.

The theorem also illuminates the role of domain scaling. For all three types of PDEs, scaling the domain influences the constants \(C\), changing proportionally to \(e^{\alpha d}-1\), where \(d=\operatorname{diam}\Omega\) serves as an indicator of scale. This modification in \(C\) subsequently affects the optimal weight of the two losses. Therefore, it is imperative for the model to account for the scale of the domain to properly adjust the loss weights.

Motivated by this understanding, we propose our MultiAdam optimizer. It maintains the second momentum of gradients for each group of losses, which is then used to adjust the scale of the update, effectively reweighting all loss terms. We found that gradient-based estimation can approximate the factor \(C\), leading to enhanced accuracy.

### Algorithm

Inspired by the analysis above, we introduce MultiAdam, a novel optimizer designed to better estimate the relative importance of losses.

Our motivation stems from two key observations. First, the Adam optimizer maintains estimates of both the first and second momentum, and these momentums tend to be relatively stable. Second, the second momentum effectively reflects the inherent difference between the scale of PDE loss and boundary loss. Utilizing the second momentum as weights allows the PDE loss and boundary loss to be normalized to a comparable scale.

The crux of MultiAdam lies in partitioning the PINN loss into several groups. Specifically, we segregate each PDE loss into a separate group, while all boundary losses are

Figure 1: The left image presents the reference solution for the case. The central image depicts the training result of the baseline PINN on an \(8\times 8\) domain, while the right image showcases the same on a \(1\times 1\) domain. It is evident that the model encounters difficulties in fitting the boundary condition when trained on the \(1\times 1\) domain.

Figure 3: The left figure shows the sum of unweighted loss \(L_{f}+L_{b}\) during training. The right figure shows the \(L^{2}\) relative error between PINN’s prediction and the ground truth. While the loss is lower in the unreweighted case, the prediction is worse off.

Figure 2: The loss curve of PINNs when solving Poisson equation on the \(8\times 8\) domain and the \(1\times 1\) domain using Adam optimizer (manually reweighted on \(1\times 1\) case). \(L_{f},L_{b}\) are defined in equation (2). While the losses are almost the same on \(8\times 8\) case, they differ by several orders of magnitude on the \(1\times 1\) case.
grouped together. We maintain the first and second momentum independently for each group, determining the update for every group in a manner akin to Adam. Lastly, we average the updates for each group and apply this as the final update to the network parameters.

The specific algorithm is outlined in Algorithm 2. We recommend the hyper-parameter settings as \(\gamma=0.001,\beta_{1}=0.99,\beta_{2}=0.99\). The rationale behind these choices can be found in Appendix D.

```
1:learning rate \(\gamma\), betas \(\beta_{1},\beta_{2}\), max epoch \(M\), objective functions \(f_{1}(\theta),f_{2}(\theta),\cdots,f_{n}(\theta)\)
2:for all\(t=1\) to \(M\)do
3:for all\(i=1\) to \(n\)do
4:\(g_{t,i}\leftarrow\nabla_{\theta}f_{i}(\theta_{t-1})\)
5:\(m_{t,i}\leftarrow\beta_{1}m_{t-1,i}+(1-\beta_{1})g_{t,i}\)
6:\(v_{t,i}\leftarrow\beta_{2}v_{t-1,i}+(1-\beta_{2})g_{t,i}^{2}\)
7:\(\hat{m}_{t,i}\gets m_{t,i}/(1-\beta_{1}^{*})\)
8:\(\hat{v}_{t,i}\gets v_{t,i}/(1-\beta_{2}^{*})\)
9:endfor
10:\(\theta_{t}\leftarrow\theta_{t-1}-\frac{\gamma}{n}\sum_{i=1}^{n}\hat{m}_{t,i}/( \sqrt{\hat{v}_{t,i}}+\varepsilon)\)
11:endfor
12:return\(\theta_{t}\)
```

**Algorithm 2** MultiAdam

The reason why we divide every PDE into separate groups is that different PDE has a different intrinsic scaling factor, leading to an imbalance within the same group. Conversely, all Dirichlet boundary losses are grouped together, as they are calculated by measuring the \(L^{2}\) error on sampling points, which remains invariant to the scaling of the domain.

## 5 Experiments

In this section, we deploy our proposed MultiAdam optimizer on various benchmarks to evaluate its convergence and accuracy. Initially, we consider Poisson's equation, a two-dimensional second-order linear PDE. This serves to examine MultiAdam's efficacy in mitigating the imbalance of weights and achieving convergence. We also compare its weight estimation to the theoretically suggested weight, demonstrating its consistency across diverse domain scales. Subsequently, we apply this method to solve the non-linear elliptic-type Helmholtz equation, underscoring the efficiency of MultiAdam. Lastly, we assess the performance of our method against other techniques in solving time-dependent PDEs, such as the Burgers' equation. An ablation study on the selection of hyper-parameters is presented, which is relegated to Appendix D.

We compare our method with a few strong baselines: 1) The Adam optimizer utilized by the original PINNs (Raissi et al., 2019) 2) The learning rate annealing (LRA) algorithm for PINNs (Wang et al., 2021) and 3) The adaptive weighting from the NTK perspective (Wang et al., 2022). Since PINNs involve the interplay of multiple loss terms from PDE and boundary conditions, some multi-task learning methods may be applied to PINNs. Here, we choose two well-known methods, i.e., 4) GradNorm (Chen et al., 2018) and 5) PCGrad (Yu et al., 2020), to compare with.

### Poisson's equation

Poisson's equation is a useful elliptic partial differential equation in theoretical physics for calculating electric or gravitational fields (Wikipedia, 2023), taking the form:

\[\Delta u=f \tag{12}\]

In order to show the scale-invariant ability of MultiAdam, we consider two Poisson's systems, Poisson-8 and Poisson-1, which are actually examples presented in Section 4.1. The Poisson-8 case is as Equation 18 in Appendix A., while the Poisson-1 case just resizes the domain from \([-4,4]^{2}\) to \([-0.5,0.5]^{2}\).

As shown in Table 1, MultiAdam is nearly invariant to the domain scaling and maintains an accurate estimate. For Poisson-8, NTK has the highest precision. However, in the Poisson-1 case, things have changed. Most of the optimizers, other than MultiAdam and NTK, fail to find the solution. MultiAdam performs the best while a significant downgrade (4.17%) is observed on NTK. Overall, MultiAdam can easily handle the domain-scaling effect and keep good performance on both tests while others cannot.

#### 5.1.1 Comparison of weight estimation

To give a deeper understanding on why MultiAdam outperforms other methods when domain is changed, we compare the weights given by different reweighting algorithms with a theoretically suggested weight as summarized in the following theorem.

**Theorem 5.1** (Error bound of Poisson's equation, Proof in Appendix B.5).: _Let \(\Omega\) be the domain described in section 5.1, and \(G:\Omega\times\Omega\rightarrow\mathbb{R}\) be the Green function of Poisson's equation. Denote \(\hat{u}_{\theta}\) as the PINN output and \(\tilde{u}\) the reference

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{Methods} & \multicolumn{2}{c}{Poisson-8} & \multicolumn{2}{c}{Poisson-1} \\  & Absolute & Relative & Absolute & Relative \\ \hline Adam & 7.49E-03 & 2.63\% & 2.98E-01 & 70.78\% \\ LRA & 1.06E-02 & 4.67\% & 6.48E-02 & 16.88\% \\ NTK & **6.58E-03** & **1.94\%** & 2.21E-02 & 6.11\% \\ GradNorm & 8.74E-03 & 2.34\% & 2.94E-01 & 69.10\% \\ PCGrad & N/A & N/A & 3.40E-01 & 77.84\% \\ MultiAdam & 1.10E-02 & 2.94\% & **1.44E-02** & **4.49\%** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Mean absolute error and relative \(L^{2}\) error of different optimization methods on Poisson’s equation. PCGrad runs into NaN due to numerical instability.
solution, then we have:_

\[\|\hat{u}_{\theta}-\tilde{u}\|_{L^{1}}\leq C_{1}\sqrt{L_{f}}+C_{2}\sqrt{L_{b}}, \tag{13}\]

_where \(L_{f},L_{b}\) are losses of PINN and \(C_{1},C_{2}\) are constants by the Green function \(G(x,\xi)\) as follows:_

\[\begin{split} C_{1}&=\int_{\Omega}\sqrt{|\Omega| \int_{\Omega}G^{2}(x,\xi)d\xi}dx\\ C_{2}&=\int_{\Omega}\sqrt{|\partial\Omega|\int_{ \partial\Omega}(\nabla_{\xi}G(x,\xi)\cdot\mathbf{n})^{2}}dx.\end{split} \tag{14}\]

According to the above theorem, the best strategy to minimize \(\|v\|_{L^{1}}\) is to minimize \(\sqrt{C_{1}^{2}L_{f}}+\sqrt{C_{2}^{2}L_{b}}\). This implies the assignment of weight \(C_{1}^{2}\) to the PDE loss and \(C_{2}^{2}\) to the boundary loss.

Then we run MultiAdam for multiple times and record the norm of second momentum for the PDE loss group and boundary loss group separately. Since we use second momentum to rescale the gradients, its norm reflects how we scale the gradient as a whole. Therefore, in the following comparison, the norm of second momentum is used as our estimated weight for different losses.

For comparison purposes, we also incorporate two other reweighting techniques, LRA and NTK. By normalizing the weight on the boundary loss to 1, we can directly compare the normalized weight on the PDE loss and discern how the algorithms balance between different losses. We run the different methods three times, with the results displayed in Figure 4.

We observe that the weight assigned by MultiAdam closely aligns with the theoretical prediction. This implies that MultiAdam accurately discerns the relative importance of different tasks, enabling it to balance the gradients of various groups and approximate the ground truth closely. It's worth noting that the slightly higher PDE weight, compared to the theoretical estimation, is attributed to the difficulty PINNs face in optimizing the PDE loss.

More crucially, MultiAdam successfully mirrors the growth trend of PDE weight under differnt scales. As depicted in Figure 5, MultiAdam exhibits superior estimation in most scales compared to other methods. These results provide a support for MultiAdam's ability to handle problems under different scales.

#### 5.1.2 Gradient pathology

To further investigate the pathology of imbalanced gradients, we study the distribution of the gradients regarding the PDE residual and the boundary loss. The results are shown in Figure 6. We can see that MultiAdam can mitigate the gradient-vanishing problem in PINNs and effectively update parameters. The PDE gradients of the original PINNs are heavily concentrated around zero and barely can parameters be optimized, leading to stagnation. This observation is inline with (Wang et al., 2021)'s work. By contrast, the PDE gradients of MultiAdam PINNs are more spread, thus more parameters can attain useful information, accelerating the overall optimization.

### Helmholtz equation

The Helmholtz equation is a non-linear elliptic differential system representing a time-independent form of the wave

Figure 4: The comparison of normalized weight for PDE loss between MultiAdam, LRA, NTK and theoretical suggestion during training. The domain \(\Omega\) lies in \([-0.5,0.5]^{2}\). The estimation given by MultiAdam is closest to the theoretical suggestion.

Figure 5: The comparison of normalized weight for PDE loss between MultiAdam, LRA, NTK and theoretical suggestion under different domain scales. When domain scale is \(x\), we indicates that the domain \(\Omega\) lies in \([-x/2,x/2]^{2}\)
equation. It appears in various fields of physics, including electromagnetic radiation, seismology, and acoustics (Wikipedia, 2023a). The Helmholtz equation is a good testbed to demonstrate the ability to cope with highly nonlinear problems. Specifically, the equation takes the following form:

\[\begin{split}& u_{xx}+u_{yy}+k^{2}u-f=0,\ \forall x\in\Omega\\ & u(x)=0,\ \forall x\in\partial\Omega\\ &\Omega=[-\frac{b}{2},\frac{b}{2}]^{2},\end{split} \tag{15}\]

where \(k\) is a parameter. The initial-boundary value problem has exact solution \(u(x,y)=\sin(a_{i}x)\sin(a_{2}x)\) when

\[f(x,y)=(k^{2}-a_{1}^{2}\pi^{2}-a_{2}^{2}\pi^{2})\sin(a_{1}\pi x)\sin(a_{2}\pi y) \tag{16}\]

We consider two cases, \((k=1,a_{1}=a_{2}=1,b=1)\) and \((k=1,a_{1}=a_{2}=10,b=0.2)\), denoted as Helmholtz-1 and Helmholtz-0.2 respectively. Figure 9 in Appendix C.3 presents the reference solution.

From the perspective of both absolute error and relative error in Table 2, MultiAdam achieves the highest accuracy among these techniques. It improves the relative \(L^{2}\) error by roughly two orders of magnitude. After resizing the domain, MultiAdam does not suffer while the competitors do, which again demonstrates the robustness of our method against re-scaling.

#### 5.2.1 Rate of convergence

We choose three representative algorithms, namely Adam, MultiAdam and PCGrad, to compare their convergence speeds from the perspective of \(L^{2}\) loss curves. As shown in Figure 7, it is interesting to see that MultiAdam moves slow in the beginning phase (e.g., \(<5000\) epochs), while can quickly converge to better solutions. The reason for this phenomenon is that MultiAdam is estimating the momentum of the PDE and boundary objectives and once it obtains a good estimate, the super-fast convergence rate is observed. In contrast, the other methods converge slowly with much more unstable phenomenons. These results demonstrate the high efficiency and stability of MultiAdam.

### Burgers' equation

The Burgers' equation is a fundamental PDE that describes the evolution of a velocity field in one spatial dimension, represented as follows:

\[\begin{split} u_{t}+uu_{x}-\nu u_{xx}&=0,\ \forall x\in[-1,1],t\in[0,1]\\ u(0,x)&=-\sin(\pi x)\\ u(t,-1)&=u(t,1)=0,\end{split} \tag{17}\]

where \(\nu=\frac{0.01}{\pi}\). It can display parabolic or hyperbolic behaviors depending on the relative importance of the forces present.

Table 3 show the results. We can see that our method has 2.92% lower error than the baseline PINNs, yet NTK reweighting is even lower in this case. Comparing with NTK reweighting, MultiAdam is more stable, as illustrated in Figure 8, where we present the curves of relative \(L^{2}\) error when using Adam, MultiAdam, and NTK methods. We

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{Methods} & \multicolumn{2}{c}{Helmholtz-1} & \multicolumn{2}{c}{Helmholtz-0.2} \\  & Absolute & Relative & Absolute & Relative \\ \hline Adam & 8.50E-02 & 22.46\% & 3.45E-01 & 93.46\% \\ LRA & 4.00E-03 & 1.11\% & 1.65E-01 & 45.87\% \\ NTK & 8.32E-02 & 21.76\% & 5.05E-01 & \(>\)100\% \\ GradNorm & 6.15E-02 & 16.06\% & 3.97E-01 & \(>\)100\% \\ PCGrad & 1.79E-02 & 4.80\% & 8.67E-02 & 22.92\% \\ MultiAdam & **1.56E-03** & **0.43\%** & **3.23E-03** & **0.87\%** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Mean absolute error and relative \(L^{2}\) error of different optimization methods on Helmholtz equation.

Figure 6: The distribution of the back-propagated gradients over different loss groups (PDE, boundary) at epoch 4000.

Figure 7: \(L^{2}\) loss curves in the Helmholtz-1 case trained with Adam, MultiAdam or PCGrad.
can see that for MultiAdam the error stays at a relatively low position since the middle of training, while Adam's error periodically rises up to as large as 30%. The spike phenomenon is not so eminent for NTK reweighting, but it is still remarkably worse than MultiAdam's.

## 6 Conclusion

This study primarily aimed to develop a scale-invariant approach for training Physics-Informed Neural Networks (PINNs). We highlighted the impact of domain scaling on PDE loss terms, which significantly contributes to unbalanced losses, and discussed its negative effect on PINN training. To address this issue, we introduced MultiAdam, a parameter-wise scale-invariant optimizer specifically designed for training PINNs. Our numerical experiments demonstrated that this optimizer is capable of handling a variety of cases across different scales, offering a relatively stable training process. At the same time, we provided a theoretical analysis of the error bounds of PINNs, which characterize the relationship between the PINN loss terms and the actual performance.

## Acknowledgements

This work was supported by the NSF of China Projects (Nos. 62061136001, 61620106010, 62076145, U19B2034, U1811461, U19A2081, 6197222, 62106120, 62076145); a grant from Tsinghua Institute for Guo Qiang; the High Performance Computing Center, Tsinghua University. J.Z was also supported by the New Cornerstone Science Foundation through the XPLORER PRIZE.
