PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks
  for Solving PDEs
While significant progress has been made on Physics-Informed Neural Networks
(PINNs), a comprehensive comparison of these methods across a wide range of
Partial Differential Equations (PDEs) is still lacking. This study introduces
PINNacle, a benchmarking tool designed to fill this gap. PINNacle provides a
diverse dataset, comprising over 20 distinct PDEs from various domains,
including heat conduction, fluid dynamics, biology, and electromagnetics. These
PDEs encapsulate key challenges inherent to real-world problems, such as
complex geometry, multi-scale phenomena, nonlinearity, and high dimensionality.
PINNacle also offers a user-friendly toolbox, incorporating about 10
state-of-the-art PINN methods for systematic evaluation and comparison. We have
conducted extensive experiments with these methods, offering insights into
their strengths and weaknesses. In addition to providing a standardized means
of assessing performance, PINNacle also offers an in-depth analysis to guide
future research, particularly in areas such as domain decomposition methods and
loss reweighting for handling multi-scale problems and complex geometry. To the
best of our knowledge, it is the largest benchmark with a diverse and
comprehensive evaluation that will undoubtedly foster further research in
PINNs.
Zhongkai Hao, Jiachen Yao, Chang Su, Hang Su, Ziao Wang, Fanzhi Lu, Zeyu Xia, Yichi Zhang, Songming Liu, Lu Lu, Jun Zhu
# PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs

###### Abstract

While significant progress has been made on Physics-Informed Neural Networks (PINNs), a comprehensive comparison of these methods across a wide range of Partial Differential Equations (PDEs) is still lacking. This study introduces PINNacle, a benchmarking tool designed to fill this gap. PINNacle provides a diverse dataset, comprising over 20 distinct PDEs from various domains including heat conduction, fluid dynamics, biology, and electromagnetics. These PDEs encapsulate key challenges inherent to real-world problems, such as complex geometry, multi-scale phenomena, nonlinearity, and high dimensionality. PINNacle also offers a user-friendly toolbox, incorporating about 10 state-of-the-art PINN methods for systematic evaluation and comparison. We have conducted extensive experiments with these methods, offering insights into their strengths and weaknesses. In addition to providing a standardized means of assessing performance, PINNacle also offers an in-depth analysis to guide future research, particularly in areas such as domain decomposition methods and loss reweighting for handling multi-scale problems and complex geometry. While PINNacle does not guarantee success in all real-world scenarios, it represents a significant contribution to the field by offering a robust, diverse, and comprehensive benchmark suite that will undoubtedly foster further research and development in PINNs.

## 1 Introduction

Partial Differential Equations (PDEs) are of paramount importance in science and engineering, as they often underpin our understanding of intricate physical systems such as fluid flow, heat transfer, and stress distribution [28]. The computational simulation of PDE systems has been a focal point of research for an extensive period, leading to the development of numerical methods such as finite difference [6], finite element [35], and finite volume methods [11]. Recent advancements have led to the use of deep neural networks to solve forward and inverse problems involving PDEs [34; 47; 10; 42]. Among these, Physics-Informed Neural Networks (PINNs) have emerged as a promising alternative to traditional numerical methods in solving such problems [34; 20]. PINNs leverage the underlying physical laws and available data to effectively handle various scientific and engineering applications. The growing interest in this field has spurred the development of numerous PINN variants, each tailored to overcome specific challenges or to enhance the performance of the original framework.

While PINN methods have achieved remarkable progress, a comprehensive comparison of these methods across diverse types of PDEs is currently lacking. Establishing such a benchmark is
crucial as it could enable researchers to more thoroughly understand existing methods and pinpoint potential challenges. Despite the availability of several studies comparing sampling methods [45] and reweighting methods [2], there has been no concerted effort to develop a comprehensive and rigorous benchmark using realistic and challenging datasets. The sheer variety and inherent complexity of PDEs make it difficult to conduct a comprehensive analysis. Moreover, different mathematical properties and application scenarios further complicate the task, requiring the benchmark to be adaptable and exhaustive.

To resolve these challenges, we propose PINNacle, a comprehensive benchmark for evaluating and understanding the performance of PINNs. As shown in Fig. 1, PINNacle consists of three major components -- a diverse dataset, a toolbox, and evaluation modules. The dataset comprises tasks from over 20 different PDEs from various domains, including heat conduction, fluid dynamics, biology, and electromagnetics. Each task brings its own set of challenges such as complex geometry, multi-scale phenomena, nonlinearity, and high dimensionality, thus providing a rich testing ground for PINNs. The toolbox incorporates more than 10 state-of-the-art (SOTA) PINN methods, enabling a systematic comparison of different strategies including loss reweighting, variational formulation, adaptive activations, and domain decomposition. These methods can be flexibly applied to the tasks in the dataset, offering researchers a convenient way to evaluate the performance of PINNs and gain insights into their suitability for different problems. The evaluation modules provide a standardized means of assessing the performance of different PINN methods across all tasks, ensuring consistency in comparison and facilitating the identification of strengths and weaknesses in various methods.

PINNacle provides a robust, diverse, and comprehensive benchmark suite for PINNs, contributing significantly to the field's understanding and application. Despite its comprehensiveness, we remind users that success within this benchmark does not necessarily guarantee success in the more complex real-world scenarios. Nonetheless, PINNacle represents a major step forward in the evolution of PINNs, fostering more innovative research and development in this exciting field. In a nutshell, our contributions can be summarized as follows:

* We introduce PINNacle, a comprehensive benchmark suite offering a diverse dataset encompassing over 20 challenging PDE problems. These problems encapsulate several critical challenges faced by PINNs, including handling complex geometries, multi-scale phenomena, nonlinearity, and high-dimensional problems.
* We provide a user-friendly toolbox, facilitating the implementation and comparison of diverse PINN methods. We have carefully selected 10 representative variants of PINNs and conducted thorough experiments to evaluate their performance.
* We provide an in-depth analysis to guide future research. We show using loss reweighting and domain decomposition methods could improve the performance on multi-scale and

Figure 1: Architecture of PINNacle. It contains a dataset covering more than 20 PDEs, a toolbox that implements about 10 SOTA methods, and an evaluation module. We implement the toolbox based on DeepXDE. These methods have a wide range of application scenarios like fluid mechanics, electromagnetism, heat conduction, geophysics, and so on.
complex geometry problems. Variational formulation achieves better performance on inverse problems. However, few methods can adequately address nonlinear problems, indicating a future direction for exploration and advancement.

## 2 Related Work

### Benchmarks in scientific machine learning

Scientific benchmarks and datasets substantially differ from those in domains such as computer vision (CV) and natural language processing (NLP) due to their high domain specificity. Their data formats, sizes, and governing principles can dramatically vary across different scientific disciplines. The growing trend of AI applications in scientific research has stimulated the development of various benchmarks and datasets. For instance, [25] presents a benchmark for comparing neural operators, while others have proposed benchmarks to assess methods for learning latent Newtonian mechanics [3; 31]. Furthermore, domain-specific datasets and benchmarks exist in fluid mechanics [15], climate science [32; 5], quantum chemistry [1], and biology [4].

Despite the existence of these domain-specific datasets and benchmarks, physics-informed machine learning has received considerable attention [13; 8] since the advent of Physics-Informed Neural Networks (PINNs) [34]. These methods successfully incorporate physical laws into model training, thereby demonstrating their immense potential across a variety of scientific and engineering domains. Various studies have compared different components within the PINN framework; for instance, [9] and [45] investigate the sampling methods of collocation points in PINNs, and [2] compare reweighting techniques for different loss components. [41] design multiple tasks to compare different methods in scientific machine learning such as PINNs, FNO, and U-Net. Nevertheless, a comprehensive comparison of various PINN approaches remains absent in the literature.

### Softwares and Toolboxes

A plethora of software solutions have been developed for solving PDEs with neural networks. These include SimNet [14], NeuralPDE [33], TorchDiffEq [7], and PyDEns [23]. More recently, DeepXDE [26] has been introduced as a versatile toolbox for implementing PINNs across different backends. However, there remains a void for a toolbox that is both comprehensive and user-friendly for PINNs. Our PINNacle aims to fill this gap, offering a modular structure that facilitates the implementation and comparison of diverse PINN variants. We furnish clear and concise code for researchers to execute benchmarks across all problems and methods.

## 3 PINNacle: A Modularized Library for Benchmarking PINNs

In this section, we first introduce the preliminaries of PINNs. Then we introduce the details of datasets (tasks), PINN methods, the toolbox framework, and the evaluation metrics.

### Preliminaries of Physics-informed Neural Networks

Physics-informed neural networks are neural network-based methods for solving PDEs as well as inverse problems of PDEs which receives much attention recently. Specifically, let's consider a general Partial Differential Equation (PDE) system defined on \(\Omega\), which can be represented as:

\[\mathcal{F}(u(x);x) = 0,\quad x\in\Omega, \tag{1}\] \[\mathcal{B}(u(x);x) = 0,\quad x\in\partial\Omega. \tag{2}\]

where \(\mathcal{F}\) is a differential operator and \(\mathcal{B}\) is the boundary/initial condition. PINN uses a neural network \(u_{\theta}(x)\) with parameters \(\theta\) to approximate \(u(x)\). The objective of PINN is to minimize the following loss function:

\[\mathcal{L}(\theta)=\frac{w_{c}}{N_{c}}\sum_{i=1}^{N_{c}}||\mathcal{F}(u_{ \theta}(x_{c}^{i});x_{c}^{i})||^{2}+\frac{w_{b}}{N_{b}}\sum_{i=1}^{N_{b}}|| \mathcal{B}(u_{\theta}(x_{b}^{i});x_{b}^{i})||^{2}+\frac{w_{d}}{N_{d}}\sum_{i =1}^{N_{d}}||u_{\theta}(x_{d}^{i})-u(x_{d}^{i})||^{2}. \tag{3}\]

where \(w_{c},w_{b},w_{d}\) are weights. The first two terms enforce the PDE constraints on \(\{x_{c}^{i}\}_{1\ldots N_{c}}\) and boundary conditions on \(\{x_{b}^{i}\}_{1\ldots N_{b}}\). The last term is data loss, which is optional when there is data
available. However, PINNs have several inherent drawbacks. First, PINNs optimize a mixture of imbalance loss terms which might hinder its convergence. Second, the high-order derivatives or nonlinear PDEs might lead to unstable optimization. Third, the vanilla MLPs might have difficulty in representing multi-scale or high-dimensional functions. To resolve these challenges, numerous variants of PINNs are proposed. However, a comprehensive comparison of these methods is lacking and thus it is imperative to develop a benchmark.

### Datasets

In order to conduct a thorough comparison of various PINN variants, it is crucial to assemble a collection of PDE problems (datasets) that encapsulate a broad spectrum of challenges. This strategy aims to highlight the strengths and limitations of different methods. Specifically, we select PDEs from multiple domains, taking into account their significance in the sciences and engineering. Notably, the dataset comprises 22 distinct cases and more details can be found in the Appendix B.

* The **Burgers' Equation**, fundamental to fluid mechanics, with both one and two-dimensional problems under consideration.
* The **Poisson's Equation**, widely used in math and physics, and we use four different cases.
* The **Heat Equation**, a time-dependent PDE that describes diffusion or heat conduction, demonstrated in four unique cases.
* The **Navier-Stokes Equation**, describing the motion of viscous fluid substances, showcased in three scenarios: a lid-driven flow (NS2d-C), a geometrically complex backward step flow (NS2d-CG), and a time-dependent problem (NS2d-LT).
* The **Wave Equation**, modeling wave behavior, exhibited in three cases.
* **Chaotic PDEs**, featuring two popular examples: the Gray-Scott (GS) and Kuramoto-Sivashinsky (KS) equations.
* **High Dimensional PDEs**, including the high-dimensional Poisson equation (PNd) and the high-dimensional diffusion or heat equation (HNd).
* **Inverse Problems**, focusing on the reconstruction of the coefficient field from noisy data for the Poisson equation (PINnv) and the diffusion equation (HInv).

It is important to note that we have chosen PDEs encompassing a wide range of mathematical properties. This ensures that the benchmarks do not favor a specific type of PDE. The selected PDE problems introduce several core challenges, which include:

* **Complex Geometry**: Many PDE problems involve complex or irregular geometry, such as heat conduction or wave propagation around obstacles. These complexities pose significant challenges for PINNs in terms of accurate boundary behavior representation.
* **Multi-Scale Phenomena**: Multi-scale phenomena, where the solution varies significantly over different scales, are prevalent in situations such as turbulent fluid flow. Achieving a balanced representation across all scales is a challenge for PINNs in multi-scale scenarios.
* **Nonlinear Behavior**: Many PDEs exhibit nonlinear or even chaotic behavior, where minor variations in initial conditions can lead to substantial divergence in outcomes. The optimization of PINNs becomes intriguing on nonlinear PDEs.
* **High Dimensionality**: High-dimensional PDE problems, frequently encountered in quantum mechanics, present significant challenges for PINNs due to the "curse of dimensionality". This term refers to the increase in computational complexity with the addition of each dimension, accompanied by statistical issues like data sparsity in high-dimensional space.

These challenges have been selected due to their frequent occurrence in numerous real-world applications. As such, a method's performance in addressing these challenges serves as a reliable indicator of its overall practical utility. Table 1 presents a detailed overview of the dataset, the PDEs and the challenges associated with these problems.

### Methods and Toolbox

After conducting an extensive literature review, we present an overview of diverse PINNs approaches for comparison. Then we present the high-level structure of the toolbox of PINNacle.
#### 3.3.1 Methods

These methods are classified into several categories, providing a comprehensive landscape of PINN methods. Roughly they are mainly based on loss functions, architecture, and optimizer [13]. The modifications to loss functions are abundant and diverse, which can be divided into reweighting existing losses and developing novel loss functions like regularization and variational formulation. Variants of architectures include using domain decomposition and adaptive activations. We select variants from these categories that have demonstrated a significant impact in the field.

These methods are tightly linked to the challenges outlined in our datasets. For instance, domain decomposition methods excel in addressing problems with complex geometries and multi-scale phenomena, while loss reweighting strategies deal with the balance for problems with multiple losses. Here, we list the primary categories and representative methods:

* **Loss reweighting/Resampling (2\(\sim\)4)**: PINNs are trained with a mixed loss of PDE residuals, boundary conditions and available data losses. Various methods [43, 44, 2, 27, 37] propose different strategies to balance these weights or resample collocation points, which indirectly adjust the weights [45, 30]. We choose three famous examples, i.e., reweighting using gradient norms (PINN-LRA) [43], using neural tangent kernel (PINN-NTK) [44], and residual-based resampling (RAR)[26, 45].
* **Novel optimizer (5)**: To handle the problem of multi-scale objectives, some new optimizers [24, 46] are proposed. We choose MultiAdam which is resistant to the domain scale changes.
* **Novel loss functions (6\(\sim\)7)**: Some works introduce novel loss functions like variational formulation [39, 22, 21] and regularization terms to improve training. We choose hp-VPINN [21] and gPINN [48, 40] which are representative examples from these two categories.
* **Novel activation architectures (8\(\sim\)10)**: Some works propose various network architectures, such as using CNN and LSTM [49, 12, 36], custom activation functions [17, 18], and domain decomposition [16, 38, 19, 29]. Among adaptive activations, we choose LAAF [17], GAAF [18] which are adaptive activation functions for PINNs and FBPINNs [29] which is a famous domain decomposition method.

#### 3.3.2 Structure of Toolbox

Here we briefly introduce the design logic and ideas of the code base of PINNacle.

\begin{table}
\begin{tabular}{c|c c c c} \hline Dataset & Complex geometry & Multi-scale & Nonlinearity & High dim \\ \hline Burgers1\(\sim\)2 & \(\times\) & \(\times\) & \(\surd\) & \(\times\) \\ Poisson3\(\sim\)6 & \(\times\)/\(\surd\) & \(\times\)/\(\surd\) & \(\times\) & \(\times\) \\ Heat7\(\sim\)10 & \(\times\)/\(\surd\) & \(\times\)/\(\surd\) & \(\times\) & \(\times\) \\ NS1\(\sim\)13 & \(\times\)/\(\surd\) & \(\times\)/\(\surd\) & \(\surd\) & \(\times\) \\ Wave14\(\sim\)16 & \(\times\)/\(\surd\) & \(\times\)/\(\surd\) & \(\times\) & \(\times\) \\ Chaotic17\(\sim\)10 & \(\times\) & \(\surd\) & \(\surd\) & \(\times\) \\ High dim19\(\sim\)20 & \(\times\) & \(\times\) & \(\times\) & \(\surd\) \\ Inverse 21\(\sim\)22 & \(\times\) & \(\times\) & \(\surd\) & \(\times\) \\ \hline \end{tabular}
\end{table}
Table 1: Overview of our datasets along with their challenges. We choose 22 cases in total to evaluate the methods of PINNs. The left picture shows the visualization of cases with these four challenges, i.e., complex geometry, multi-scale, nonlinearity, and high dimension.

\begin{table}
\begin{tabular}{c|c c c c} \hline  & Complex Geometry & Multi-scale & Nonlinearity & High dim \\ \hline Vanilla PINN [34] & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ Reweighting/Resampling2\(\sim\)4[43, 44, 45] & \(\surd\) & \(\surd\) & \(\times\) & \(\times\) \\ Novel Optimizer7[46] & \(\times\) & \(\surd\) & \(\times\) & \(\times\) \\ Novel Loss Functions5\(\sim\)6[21, 48] & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ Novel Architecture8\(\sim\)10[29] & \(\surd\) & \(\surd\) & \(\times\) & \(\times\) \\ \hline \end{tabular}
\end{table}
Table 2: Overview of methods in our PINNacle. \(\surd\) denotes the method is potentially designed to solve or show empirical improvements for problems encountering the challenge and vice versa.
**High-level Training and Custom PDE Development.** Our codebase is based on DeepXDE and provides a series of encapsulated classes and functions to facilitate high-level training and custom PDEs. These utilities allow for a standardized and streamlined approach to the implementation of various PINN variants and PDEs. We could easily build custom PDEs or use our pre-defined PDEs from "BasePDE" class. We wrap some APIs in DeepXDE to enable a more flexible call. We assemble a Model class using a PDE, Net, and Optimizer. Then we compile it using DeepXDE and feed it to the trainer with our custom-defined callbacks. Moreover, to enable easier training and testing across a variety of equation types and dimensions, we have provided many auxiliary functions. These include functions for computing different metrics, visualizing predictions, and recording results.

**Adaptive Multi-GPU Parallel Training Pipeline.** To further enhance the efficiency of systematic evaluations of PINN methods, we have integrated an adaptive multi-GPU parallel training framework. It not only significantly increases computational speed but also allows for the execution of larger and more complex tasks. It addresses the parallelization phase of training on multiple tasks, effectively balancing the computational loads of multiple GPUs. In a nutshell, we provide an example code for training and evaluating PINNs on two Poisson equations using our PINNacle framework.

```
importdeepxdesadde fromtrainerimportTrain fromsrc.pdepimportPDE1,...,PDEn fromsrc.utils.callbacksimportTesterCallback trainer=Trainer('experiment-name',device) forpde_classin[PDE1,...,PDEn]: defget_model(): pde_pde_class() net=dde.nn.FNN([pde.input_dim]+n_layers*[n_hidden]+[pde.output_dim]) opt=torch.optim.Adam(net.parameters(),lr=learning_rate) model=pde.create_model(net) model.compile(opt) returnmodel trainer.add_task( get_model,{'iterations';num_iterations,'callbacks':[TesterCallback()]} trainer.train_all_parallel()
```

### Evaluation

To comprehensively analyze the discrepancy between the PINN solutions and the true solutions, we adopt multiple metrics to evaluate the performance of the PINN variants. Generally, we choose several metrics which are commonly used in literature that applies to all methods and problems. Specifically, we use Mean Square Error (MSE), \(\ell_{2}\) relative error (\(\mathrm{L2RE}\)), and \(\ell_{1}\) relative error (\(\mathrm{L1RE}\)) to measure the quality of the solution. These three metrics are the most widely used ones to measure the quality of the solution. We suppose that \(\mathbf{y}=(y_{i})_{i=1}^{n}\) is the prediction and \(\mathbf{y}^{\prime}=(y_{i}^{\prime})_{i=1}^{n}\) to is ground truth, where \(1\leqslant i\leqslant n\), and \(n\) is the number of testing examples. These three metrics are computed as follows:

\[\mathrm{MSE}=\frac{1}{n}\sum_{i=1}^{n}(y_{i}-y_{i}^{\prime})^{2},\ \mathrm{L2RE}= \sqrt{\frac{\sum_{i=1}^{n}(y_{i}-y_{i}^{\prime})^{2}}{\sum_{i=1}^{n}{y_{i}^{ \prime}}^{2}}},\ \mathrm{L1RE}=\frac{\sum_{i=1}^{n}|y_{i}-y_{i}^{\prime}|}{\sum_{i=1}^{n}|y_{ i}^{\prime}|}. \tag{4}\]

Besides, for time-dependent problems, investigating the quality of the solution along with time is an important metric. Therefore we compute the error varying with time,

\[\mathrm{TL2RE}(t)=\sqrt{\frac{\sum_{i=1}^{n}(y_{i}(t)-y_{i}^{\prime}(t))^{2}}{ \sum_{i=1}^{n}{y_{i}^{\prime}(t)}^{2}}}. \tag{5}\]

We assess the performance of PINNs against the reference from numerical solvers. Experimental results utilizing the \(\ell_{2}\) relative error (\(\mathrm{L2RE}\)) metric are incorporated within the main text, while a more exhaustive set of results, based on the aforementioned metrics, is available in the Appendix E.1.

## 4 Experiments

In this section, we present our experimental results. Unless otherwise stated, we use 0.001 as the learning rate and we train all models with 20,000 epochs. We repeat all experiments for 3 times and record the mean and std.
### Main Experimental Results

We present the main experimental results for each selected baseline on our tasks and display their average \(\ell_{2}\) relative errors in Table 3 (with standard deviation results available in Appendix E.1).

**PINN.** We use PINN-w to denote training PINNs with larger boundary weights. Vanilla PINNs struggle to accurately solve complex physics systems, indicating substantial room for improvement across most tasks in our benchmark. Using an \(\ell_{2}\) relative error (L2RE) of \(10\%\) as a threshold for a successful solution, we find that the two standard PINN configurations only solve 9 out of 22 tasks, most of which involve simpler equations (e.g., \(1.45\%\) on Burgers-1d-C). They encounter significant difficulties when faced with physics systems characterized by complex geometries, multi-scale phenomena, nonlinearity, and longer time spans. This can be attributed to PINNs directly optimizing a weighted average of the PDE residual losses and initial/boundary condition losses using an MLP, which leads to critical issues such as imbalance between multiple optimization objectives, suboptimal convergence and generalization, and limited expressiveness.

**PINN variants.** PINN variants offer different approaches to addressing some of these challenges to varying degrees. Methods involving loss reweighting and resampling have shown improved performance in some cases involving complex geometries and multi-scale phenomena (e.g., \(1.43\%\) on Poisson-2d-CG and \(4.40\%\) on Heat-2d-MS for NTK). This is due to the configuration of loss weights and sampled collocation points, which adaptively places more weight on more challenging domains during the training process. However, these methods still struggle with Wave equations, Navier-Stokes equations, and other cases with higher dimensions or longer time spans. MultiAdam, a representative of novel optimizer methods, solves several simple cases and the chaotic GS equation (\(9.37\%\)), but does not significantly outperform other methods. The new loss term demonstrates significant superiority in solving inverse problems (_e.g.,_\(1.19\%\) on HInv for vPINN), but no clear improvement in fitting error over standard PINN in forward cases. Changes in architecture can enhance expressiveness and flexibility for cases with complex geometries and multi-scale systems. For example, FBPINN achieves the smallest error on the chaotic GS equation (\(7.99\%\)), while LAAF delivers the best fitting result on Heat-2d-CG (\(2.39\%\)).

**Discussion.** For challenges related to complex geometries and multi-scale phenomena, some methods can mitigate these issues by implementing mechanisms like loss reweighting, point resampling, novel

\begin{table}
\begin{tabular}{c c|c c|c c c|c c c|c c} \hline \hline \multirow{2}{*}{L2RE} & \multirow{2}{*}{Name} & \multicolumn{2}{c|}{Vanilla} & \multicolumn{2}{c|}{Loss Reweighting/Sampling} & \multicolumn{2}{c|}{Optimizer} & \multicolumn{2}{c|}{Loss functions} & \multicolumn{2}{c}{Architecture} \\ \cline{3-13}  & & PINN & PINN-w & LRA & NTK & RAR & MultiAdam & gPINN & vPINN & LAAF & GAAF & FBPINN \\ \hline \multirow{2}{*}{Burgers} & 1d-C & 1.45E-2 & 2.63E-2 & 2.61E-2 & 1.84E-2 & 3.32E-2 & 4.85E-2 & 2.16E-1 & 3.47E-1 & **1.43E-2** & 5.20E-2 & 2.32E-1 \\  & 2d-C & 3.24E-1 & 2.70E-1 & **2.60E-1** & 2.75E-1 & 3.45E-1 & 3.33E-1 & 3.27E-1 & 6.38E-1 & 2.77E-1 & 2.95E-1 & – \\ \hline \multirow{4}{*}{Poisson} & 2d-C & 6.94E-1 & 3.49E-2 & 1.17E-1 & **1.32E-2** & 6.99E-1 & 2.63E-2 & 6.87E-1 & 4.91E-1 & -7.68E-1 & 6.04E-1 & 4.49E-2 \\  & 2d-C & 6.36E-1 & 6.08E-2 & 4.34E-2 & **1.43E-2** & 6.48E-1 & 2.76E-1 & 7.92E-1 & 2.86E-1 & 4.80E-1 & 8.71E-1 & 2.90E-2 \\  & 3d-C & 3.04E-0 & 3.66E-0 & 3.67E-0 & 3.12E-0 & 2.96E-0 & 3.64E-0 & 2.35E+0 & 3.08E+0 & 3.07E+0 & 2.86E+0 & 3.30E+0 \\  & 2d-MS & 6.30E-1 & 7.60E-1 & 7.94E-1 & 7.48E-1 & 6.44E-1 & **5.90E-1** & 6.16E-1 & 9.72E-1 & 5.93E-1 & 9.31E-1 & 1.04E+0 \\ \hline Heat & 2d-VC & 1.01E+0 & 2.35E-1 & **2.12E-1** & 2.14E-1 & 9.66E-1 & 4.75E-1 & 2.12E+0 & 9.40E-1 & 6.42E-1 & 8.49E-1 & 9.52E-1 \\  & 2d-MS & 6.21E-2 & 2.42E-1 & 8.79E-2 & **4.40E-2** & 7.49E-2 & 2.18E-1 & 1.13E-1 & 9.30E-1 & 7.40E-2 & 9.85E-1 & 8.20E-2 \\  & 2d-CG & 3.64E-2 & 1.45E-1 & 1.25E-1 & 1.16E-1 & 2.72E-2 & 9.12E-2 & 9.38E-2 & – & **2.39E-2** & 4.61E-1 & 9.16E-2 \\  & 2d-LT & 9.99E-1 & 9.99E-1 & 9.99E-1 & 1.00E+0 & 9.99E-1 & 1.00E+0 & 1.00E+0 & 9.99E-1 & 9.99E-1 & 1.01E+0 \\ \hline NS & 2d-C & 7.40E-2 & 1.45E-1 & NN & 1.98E-1 & 4.69E-1 & 7.27E-1 & 7.70E-2 & 2.91E-1 & **3.60E-2** & 3.79E-2 & 8.45E-2 \\  & 2d-C & 1.19E-1 & 3.26E-1 & 3.32E-1 & 3.33E-1 & 3.34E-1 & 4.31E-1 & 1.54E-1 & 9.94E-1 & **3.42E-1** & **1.74E-1** & 8.27E+0 \\  & 2d-LT & 9.96E-1 & 1.00E+0 & 1.00E+0 & 9.99E-1 & 1.00E+0 & 1.00E+0 & 9.95E-1 & 1.73E+0 & 9.98E-1 & 9.99E-1 & 1.00E+0 \\ \hline Wave & 1d-C & 5.88E-1 & 2.85E-1 & 3.61E-1 & **9.79E-2** & 5.39E-1 & 1.21E-1 & 5.56E-1 & 8.39E-1 & 4.54E-1 & 6.77E-1 & 5.91E-1 \\  & 2d-CG & 1.84E+0 & 1.66E+0 & 1.48E+0 & 2.16E+0 & 1.15E+0 & 1.09E+0 & 8.14E-1 & 7.99E-1 & 8.19E-1 & **7.94E-1** & 1.06E+0 \\  & 2d-MS & 9.75E-1 & 9.66E-1 & 9.54E-1 & 9.68E-1 & 9.73E-1 & 9.33E-1 & 9.72E-1 & 2.10E+0 & 9.71E-1 & 1.08E+0 & 1.52E+0 \\ \hline Chaotic & GS & 3.19E-1 & 1.58E-1 & 9.37E-2 & 2.16E-1 & 9.46E-2 & 9.37E-2 & 2.48E-1 & 1.16E+0 & 9.47E-2 & 9.46E-2 & **7.99E-2** \\  & KS & 1.01E+0 & 9.86E-1 & 9.57E-1 & 9.64E-1 & 1.01E+0 & 9.61E-1 & 9.94E-1 & 9.72E-1 & 1.01E+0 & 1.00E+0 & 1.02E+0 \\ \hline High dim & PNN & 3.04E-3 & 2.58E-4 & **4.58E-4** & 4.64E-3 & 3.95E-3 & 3.98E-3 & 5.05E-3 & – & 4.14E-3 & 7.75E-3 & – \\  & HAV & 3.61E-1 & 4.59E-1 & 3.94E-1 & 3.97E-1 & 3.57E-1 & **3.02E-1** & 3.17E-1 & – & 5.22E-1 & 5.21E-1 & – \\ \hline Inverse & PINP & 9.42E-2 & 1.66E-1 & 1.54E-1 & 1.93E-1 & 9.38E-2 & 1.30E-1 & 8.03E-2 & **1.41E-2** & 1.30E-1 & 2.54E-1 & 8.44E-1 \\  & HInv & 1.57E+0 & 5.26E-2 & 5.09E-2 & 7.52E-2 & 1.52E+0 & 8.04E-2 & 4.84E+0 & **1.19E-2** & 5.59E-1 & 2.12E-1 & 9.27E-1 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Mean L2RE of different PINN variants on our benchmark. We **bold** the best results across all methods. We do not bold any result if errors of all methods are about \(100\%\). “NaN” means the method does not converge and “–” means
optimizers, and better capacity through adaptive activation and domain decomposition. This holds true for the 2D cases of Heat and Poisson equations, which are classic linear equations. However, when systems have higher dimensions (Poisson3d-CG) or longer time spans (Heat2d-LT), all methods fail to solve, highlighting the difficulties associated with complex geometries and multi-scale systems.

In contrast, nonlinear, high-dimensional, and long-time problems are harder to solve. Nearly all methods struggle to solve the 2D Burgers equation, NS equation, and KS equation, which are typical nonlinear differential equations. Alongside the failures over longer time spans, we also observe a general inability to solve the Wave equation, which contains a second-order time derivative and typically displays periodic phenomena over time. While all methods achieve high precision on Poisson-Nd, none are able to solve Heat-Nd. This suggests that systems in very high dimensions continue to pose unsolved challenges. These difficulties represent open problems for future research.

### Hyperparameter Analysis

The performance of PINNs is strongly infected by hyperparameters, with each variant potentially introducing its own unique set. Here we aim to investigate the impact of several shared and method-specific hyperparameters via ablation studies. We consider the batch size and the number of training epochs. The corresponding results are depicted in Figure 2. We focus on a set of problems, i.e., Burgers1d, GS, Heat2d-CG, and Poisson2d-C. Detailed numerical results and additional findings can be found in Appendix E.2.

**Batch Size.** The left figure shows the testing L2RE of PINNs for varying batch sizes. It suggests that larger batch sizes tend to provide superior results, possibly due to improved gradient estimation accuracy. Despite the GS and Poisson2d-C witnessing saturation beyond a batch size of 2048, amplifying the number of collocation points generally enhances results for the remaining problems.

**Training Epochs.** The right figure illustrates the L2RE of PINNs across varying training epochs. Mirroring the trend observed with batch size, we find that an increase in the number of epochs leads to a decrease in overall error. However, the impact of augmenting the number of epochs also has a saturation point. Notably, beyond 20k or 80k epochs, the error does not decrease significantly, suggesting convergence of PINNs around 20\(\sim\) 80 epochs.

**Learning Rates.** The performance of standard PINNs under various learning rates and learning rate schedules is depicted in Figure 3. We observe that the influence of the learning rate on performance is intricate, with optimal learning rates varying across problems. Furthermore, PINN training tends to be unstable. High learning rates, such as \(10^{-2}\), often lead to error spikes, while low learning rates, like \(10^{-5}\), result in sluggish convergence. Our findings suggest that a moderate learning rate, such as \(10^{-3}\) or \(10^{-4}\), or a step decay learning rate schedule, tends to yield more stable performance.

Moreover, each method might introduce its method-specific hyperparameters. Here we choose several representative examples to show their influence, like momentum for PINN-LRA, weights for gPINN, and so on. Due to limited space, we list results in Appendix E.2.

Figure 2: Performance of vanilla PINNs under different batch size (number of collocation points) which is shown in the left figure, and number of training epochs which is shown in the right figure.
## 5 Conclusion and Discussion

In this work, we introduced PINNacle, a comprehensive benchmark offering a user-friendly toolbox that encompasses over 10 PINN methods. We evaluated these methods against more than 20 challenging PDE problems, conducting an extensive set of experiments and an ablation study featuring various hyperparameters. Looking forward, we plan to expand the benchmark by integrating additional state-of-the-art methods and incorporating more practical problem scenarios.

Our rigorous analysis of the experimental results yields several key insights. Firstly, certain methods such as domain decomposition and reweighting are beneficial for addressing problems characterized by complex geometries and multi-scale features. Secondly, the choice of hyperparameters is crucial to the performance of PINNs. Selecting a larger batch size and appropriately weighting losses may significantly reduce the error. Thirdly, we identify high-dimensional and nonlinear problems as a pressing challenge. The overall performance of PINNs is not yet on par with traditional numerical methods. Fourthly, from a theoretical standpoint, the exploration of PINNs' loss landscape during the training of high-dimensional or nonlinear problems remains a largely uncharted area of research. Lastly, from an algorithmic perspective, we propose that integrating the strengths of neural networks with numerical methods may present a promising avenue toward overcoming the challenges identified herein.
