iPINNs: Incremental learning for Physics-informed neural networks
Physics-informed neural networks (PINNs) have recently become a powerful tool
for solving partial differential equations (PDEs). However, finding a set of
neural network parameters that lead to fulfilling a PDE can be challenging and
non-unique due to the complexity of the loss landscape that needs to be
traversed. Although a variety of multi-task learning and transfer learning
approaches have been proposed to overcome these issues, there is no incremental
training procedure for PINNs that can effectively mitigate such training
challenges. We propose incremental PINNs (iPINNs) that can learn multiple tasks
(equations) sequentially without additional parameters for new tasks and
improve performance for every equation in the sequence. Our approach learns
multiple PDEs starting from the simplest one by creating its own subnetwork for
each PDE and allowing each subnetwork to overlap with previously learned
subnetworks. We demonstrate that previous subnetworks are a good initialization
for a new equation if PDEs share similarities. We also show that iPINNs achieve
lower prediction error than regular PINNs for two different scenarios: (1)
learning a family of equations (e.g., 1-D convection PDE); and (2) learning
PDEs resulting from a combination of processes (e.g., 1-D reaction-diffusion
PDE). The ability to learn all problems with a single network together with
learning more complex PDEs with better generalization than regular PINNs will
open new avenues in this field.
Aleksandr Dekhovich, Marcel H. F. Sluiter, David M. J. Tax, Miguel A. Bessa
# iPINNs: Incremental learning for

###### Abstract

Physics-informed neural networks (PINNs) have recently become a powerful tool for solving partial differential equations (PDEs). However, finding a set of neural network parameters that lead to fulfilling a PDE can be challenging and non-unique due to the complexity of the loss landscape that needs to be traversed. Although a variety of multi-task learning and transfer learning approaches have been proposed to overcome these issues, there is no incremental training procedure for PINNs that can effectively mitigate such training challenges. We propose incremental PINNs (iPINNs) that can learn multiple tasks (equations) sequentially without additional parameters for new tasks and improve performance for every equation in the sequence. Our approach learns multiple PDEs starting from the simplest one by creating its own subnetwork for each PDE and allowing each subnetwork to overlap with previously learned subnetworks. We demonstrate that previous subnetworks are a good initialization for a new equation if PDEs share similarities. We also show that iPINNs achieve lower prediction error than regular PINNs for two different scenarios: (1) learning a family of equations (e.g., 1-D convection PDE); and (2) learning PDEs resulting from a combination of processes (e.g., 1-D reaction-diffusion PDE). The ability to learn all problems with a single network together with learning more complex PDEs with better generalization than regular PINNs will open new avenues in this field.

Keywords:Physic-informed neural networks (PINNs) Scientific machine learning (SciML) Incremental learning Sparsity

## 1 Introduction

Deep neural networks (DNNs) play a central role in scientific machine learning (SciML). Recent advances in neural networks find applications in real-life problems in physics [38, 6, 50, 28], medicine [41, 51, 49], finance [22, 66, 17, 58], and engineering [5, 55, 11, 26]. In particular, they are also applied to solve Ordinary Differential
Equations and Partial Differential Equations (ODEs/PDEs) [34, 42, 65, 46]. Consider the following PDE,

\[\mathcal{F}[u(\mathbf{x},t)] =f(\mathbf{x}),\quad\mathbf{x}\in\Omega,\ t\in[t_{0},T], \tag{1}\] \[\mathcal{B}[u(\mathbf{x},t)] =b(\mathbf{x}),\quad\mathbf{x}\in\partial\Omega,\] (2) \[u(\mathbf{x},t_{0}) =h(\mathbf{x}),\quad\mathbf{x}\in\Omega, \tag{3}\]

where \(\mathcal{F}\) is a differential operator, \(\mathcal{B}\) is a boundary condition operator, \(h(\mathbf{x})\) is an initial condition, and \(\Omega\) is a bounded domain.

The first neural network-based approaches incorporated a form of the equation into the loss function with initial and boundary conditions included as hard constraints [31, 32]. However, these works used relatively small neural networks with one or two hidden layers. On the contrary, PINNs [46] encode initial and boundary conditions as soft constraints into the loss function of a DNN. Subsequently, PINNs and their extensions found applications in fluid mechanics [40, 60, 7], inverse problems [13, 37, 61] and finance [46, 2]. Later, the generalized version of PINNs, called XPINNs [24], was proposed by decomposing the domain into multiple subdomains. However, this method uses as many networks as the number of subdomains, increasing the algorithm's complexity. Multi-head PINNs (MH-PINNs) [67] is a multi-task learning approach for PINNs that is employed to learn stochastic processes, synergistic learning of PDEs and uncertainty quantification. MH-PINNs have a shared part of the network and task-specific output heads for prediction. Therefore, it uses additional parameters for every head, increasing the model's size with respect to the number of tasks.

Despite the popularity of DNNs, and PINNs in particular, there are few _incremental_ learning algorithms available in SciML literature. Yet, incremental learning and continual learning algorithms [8, 9, 27] are capable of handling tasks sequentially, instead of altogether as in multi-task learning and other strategies. Moreover, they are still capable of not forgetting how to solve all of the previously learned tasks. If tasks have some similarities with each other, new tasks have the potential of being learned better (i.e., faster or with lower testing error) with the help of previously learned ones. The goal of this work is to propose an incremental learning algorithm for PINNs such that similar symbiotic effects can be obtained.

#### 1.0.1 Background and main challenges.

PINNs formulate the PDE solution problem by including initial and boundary conditions into the loss function of a neural network as soft constraints. Let us denote the output of the network \(\mathcal{N}\) with learnable parameters \(\theta\) as \(\hat{u}(\theta,\mathbf{x},t)=\mathcal{N}(\theta;\mathbf{x},t)\). Then sampling the set of collocation points, i.e. a set of points in the domain, \(\mathcal{CP}=\{(x^{i},t^{i}):x^{i}\in\mathrm{int}\ \Omega,\ t^{i}\in(t_{0},T],\ i=1,2, \ldots N_{\mathcal{F}}\}\), the set of initial points \(\mathcal{IP}=\{(x^{j},t_{0}):x^{j}\in\partial\Omega,\ j=1,2,\ldots,N_{u_{0}}\}\) and the set of boundary points \(\mathcal{BP}=\{(x^{k},t^{k}):x^{k}\in\partial\Omega,\ t^{k}\in(t_{0},T],\ k=1,2, \ldots,N_{b}\}\) one can write the optimization problem and loss function arising from PINNs as follows:
\[\mathcal{L}(\theta) =\mathcal{L}_{\mathcal{F}}(\theta)+\mathcal{L}_{u_{0}}(\theta)+ \mathcal{L}_{b}(\theta)\rightarrow\min_{\theta}, \tag{4}\] \[\mathcal{L}_{\mathcal{F}}(\theta) =\frac{1}{N_{\mathcal{F}}}\sum_{i=1}^{N_{\mathcal{F}}}\big{|} \big{|}\mathcal{F}[\hat{u}(\theta,x^{i},t^{i})]-f(x^{i})\big{|}\big{|}^{2},\quad (x^{i},t^{i})\in\mathcal{CP},\] (5) \[\mathcal{L}_{u_{0}}(\theta) =\frac{1}{N_{u_{0}}}\sum_{j=1}^{N_{u_{0}}}\big{|}\big{|}\hat{u}( \theta,x^{j},t_{0})-h(x^{j})\big{|}\big{|}^{2},\quad(x^{j},t_{0})\in\mathcal{ IP},\] (6) \[\mathcal{L}_{b}(\theta) =\frac{1}{N_{b}}\sum_{k=1}^{N_{b}}\big{|}\big{|}\mathcal{B}[\hat{ u}(\theta,x^{k},t^{k})]-b(x^{k})\big{|}\big{|}^{2},\quad(x^{k},t^{k})\in \mathcal{BP}. \tag{7}\]

However, sometimes PINNs struggle to learn the ODE/PDE dynamics [59, 30, 47, 43] (see Figure 1). Wight & Zhao [62] proposed several techniques to improve the optimization process compared to the original formulation: mini-batch optimization and adaptive sampling of collocation points. Adaptive sampling in time, splits the time interval \([t_{0},T]=\cup_{k=0}^{K}[t_{k-1},t_{k}],\ t_{K}=T\), and solves an equation on the first interval \([t_{0},t_{1}]\), then on \([t_{0},t_{2}]\), and so on up to \([t_{0},T]\). Thus, if a solution can be found on a domain \(\Omega\times[t_{0},t_{k-1}]\), then the network is pretrained well for the extended domain \(\Omega\times[t_{0},t_{k}]\). Krishnapriyan et al. [30] proposed the _seq2seq_ approach that splits the domain into smaller subdomains in time and learns the solution on each of the subdomains with a separate network. Thus, both adaptive sampling in time and _seq2seq_ are based on the idea of splitting the domain into multiple subdomains, on which solutions can be learned easier.

As explained in [47], improving PINN's solutions by considering small subdomains is possible because the loss residuals (\(\mathcal{L}_{\mathcal{F}}\) term) can be trivially minimized in the vicinity of fixed points, despite corresponding to nonphysical system dynamics that do not satisfy the initial conditions. Therefore, the reduction of the domain improves the convergence of the optimization problem (4) and helps to escape nonphysical solutions.

Figure 1: 1-D reaction equation with parameter \(\rho=5\) (see P1.2).
Another strategy is to consider transfer learning. Transfer learning is commonly used in computer vision and natural language processing [3, 57, 48, 23]. It tries to improve the optimization process by starting with better weight initialization. In PINNs, transfer learning is also successfully used to accelerate the loss convergence [18, 45, 10, 63]. For instance, Chen et al. [12] apply transfer learning to learn faster different PDEs creating tasks by changing coefficients or source terms in equations. Analogously, curriculum regularization (similar to curriculum learning [4]) is proposed in [30] to find good initial weights.

#### 1.0.1 Our contribution.

We propose _incremental PINNs_ (iPINNs) and implement this strategy by creating one subnetwork per task such that a complete neural network can learn multiple tasks. Each subnetwork \(\mathcal{N}_{i}\) has its own set of parameters \(\theta_{i}\subset\theta\), and the model is trained sequentially on different tasks. A subnetwork for a new task can overlap with all previous subnetworks, which helps to assimilate the new task. As a result, the network consists of overlapping subnetworks, while the free parameters can be used for future tasks. To illustrate the benefits of the algorithm we consider two problem formulations (Section 3). Firstly, we learn a family of equations (e.g., convection) starting from a simple one and incrementally learning new equations from that family. Secondly, we learn a dynamical system that consists of two processes (e.g., reaction-diffusion) by first learning the individual components of the process. Both scenarios demonstrate that the incremental approach enables an iPINN network to learn for cases where regular PINNs fail. To the best of our knowledge, this is the first example where one network can sequentially learn multiple equations without extending its architecture, with the added benefit that performance is significantly improved.

## 2 Related work

Our methodology is based on creating sparse network representations and, similarly to other PINN research, is sensitive to the choice of activation functions. We briefly highlight key related work herein.

#### 2.0.1 Sparse network representation.

Sparse architectures are often advantageous compared to dense ones [19, 1, 64, 35]. According to the lottery ticket hypothesis (LTH) [16], every randomly initialized network contains a subnetwork that can be trained in isolation to achieve comparable performance as the original network. Based on this observation, the idea of using subnetworks has been adopted in continual learning [39, 53, 54]. In this paradigm, every subnetwork created is associated with a particular task and used only for this task to make a prediction. One of the approaches to find these tasks-related subnetworks is connections' pruning [33, 21, 20, 15, 14] that removes unimportant parameters while exhibiting similar performance.

#### 2.0.2 Choice of the activation function.

There are several studies that investigate how different activation functions affect the performance of neural networks in
classification and regression tasks [56, 25]. It was shown that ReLU[44] activation function which can be powerful in classification tasks, in the case of physics-informed machine learning (PIML) regression, may not be the optimal choice. Meanwhile, hyperbolic tangent (tanh) or sine (sin) perform well for PIML. Sinusoidal representation networks (SIRENs) [52] tackle the problem of modeling the signal with fine details. Special weights initialization scheme combined with sin activation function allows SIREN to learn complex natural signals. Hence, we use sin activation function in our experiments. In Section 6.1, we provide the comparison in results between the discussed activation functions.

## 3 Problem formulation

We focus on two scenarios: (1) incremental PINNs learning, where the network sequentially learns several equations from the same family; and (2) learning a combination of multiple equations that create another physical process. To illustrate these cases, we consider one-dimensional convection, reaction and reaction-diffusion problems with periodic boundary conditions.

### Scenario 1: Equation incremental learning

We consider the problem of learning the sequence of equations that belong to one family:

\[\mathcal{F}_{k}(u(x,t))=0,\quad x\in\Omega,\ t\in[t_{0},T],\ k=1,2,\ldots,K,\] (P1)

where \(\{\mathcal{F}_{k}\}_{k=1}^{K}\) are differential operators from the same family of equations.

\begin{tabular}{c|c}
**1-D convection equation** & **1-D reaction equation** \\ \(\dfrac{\partial u}{\partial t}+\beta_{k}\dfrac{\partial u}{\partial x}=0,\) & (P1.1) \\ \(u(x,0)=h_{1}(x),\) & \(\dfrac{\partial u}{\partial t}-\rho_{k}u(1-u)=0,\) & (P1.2) \\ \(u(0,t)=u(2\pi,t),\) & \(u(x,0)=h_{2}(x),\) \\  & \(u(0,t)=u(2\pi,t),\) \\  & where \\ \(t\in[0,1],\ x\in[0,2\pi],\ \beta_{k}\in\mathcal{B}\subset\mathbb{N}\). & \(t\in[0,1],\ x\in[0,2\pi],\ \rho_{k}\in\mathcal{R}\subset\mathbb{N}\). \\ \end{tabular}

In this case, every task \(k\) is associated with \(\mathcal{D}_{k}=\{(x,t,k):\ x\in[0,2\pi],\ t\in[t_{0},T],\ k\in\mathbb{N}\}\). Following [30], we take \(h_{1}(x)=\sin x\) and \(h_{2}(x)=e^{-\frac{(x-\pi)^{2}}{2(\pi/4)^{2}}}\).

### Scenario 2: Combination of multiple equations

We also consider the case when a dynamic process consists of multiple components. Let us consider the reaction-diffusion equation:
\[\frac{\partial u}{\partial t}-\nu\frac{\partial^{2}u}{\partial x^{2}} -\rho u(1-u) =0,\] (P2) \[u(x,0) =h_{2}(x),\] \[u(0,t) =u(2\pi,t),\]

where \(t\in[0,1],\ x\in[0,2\pi],\ \nu,\rho>0\). This process consists of two parts: reaction term (\(\nu=0\)): \(-\rho u(1-u)\) and diffusion term (\(\rho=0\)): \(-\nu\frac{\partial^{2}u}{\partial x^{2}}\). Therefore, we construct one task as the reaction, another one as the diffusion, and the final one as the reaction-diffusion. We can change the order of the reaction tasks and diffusion tasks to show the robustness of incremental learning. The reaction-diffusion task should be the last one since our goal is first to learn the components of the system and only then the full system.

Considering these two problems, we want to show that better generalization can be achieved by pretraining the network with simpler related problems rather than by dividing the domain into smaller subdomains. In the following section, we show how one network can incrementally learn different equations without catastrophic forgetting.

## 4 Methodology

The proposed method needs to be applicable to both types of problems P1 and P2. However, these problems cannot be solved by one network with the same output head for all \(K\) tasks, since \(\mathcal{F}_{i}(u(x,t))\neq\mathcal{F}_{j}(u(x,t))\) for \(i\neq j\) and \(x\in\Omega,\ t\in[t_{0},T]\). Therefore, we propose iPNNs - an incremental learning algorithm that focuses on learning task-specific subnetworks \(\mathcal{N}_{1},\mathcal{N}_{2},...,\mathcal{N}_{K}\) for each task \(k=1,2,\ldots,K\). To create these subnetworks, we use an iterative pruning algorithm NNrelief [14]. This pruning approach uses input data to estimate the contribution of every connection to the neuron in the pretrained network and

Figure 2: An example of iPNNs with two PDEs: every subnetwork corresponds to only one task (PDE).
delete the least important ones. However, in principle, any connections pruning algorithm or any other approach that is able to find and train sparse network representations is suitable.

iPINN trains task-related subnetworks with pruning, allowing the subnetworks to overlap on some connections. This way the method provides knowledge sharing between the subnetworks. These overlaps are updated with respect to all tasks that are assigned to a particular connection. Let us denote the loss of each task \(\mathcal{D}_{j}\) as \(\mathcal{L}_{j}=\mathcal{L}(\theta_{j};\mathcal{D}_{j})\), where \(\theta_{j}\) is the parameter vector for task \(\mathcal{D}_{j}\), \(1\leq j\leq k\). Then the total loss and its gradient with respect to a parameter \(w\) can be written as:

\[\mathcal{L} =\sum_{j=1}^{k}\mathcal{L}_{j}, \tag{8}\] \[\frac{\partial\mathcal{L}}{\partial w} =\sum_{j=1}^{k}\frac{\partial\mathcal{L}_{j}}{\partial w}=\sum_{j :\ w\in\mathcal{N}_{j}}\frac{\partial\mathcal{L}_{j}}{\partial w}, \tag{9}\]

because if \(w\not\in\mathcal{N}_{j}\), then \(\frac{\partial\mathcal{L}_{j}}{\partial w}=0\). The pseudocode of the algorithm is shown as follows:

```
0: neural network \(\mathcal{N}\), training datasets \(\mathcal{D}_{k}\) (\(k=1,2,\ldots,K\)), training hyperparameters, pruning hyperparameters (\(\textit{num\_iters}\)).
1:for\(k=1,2,\ldots,K\)do
2:\(\mathcal{N}_{k}\leftarrow\mathcal{N}\)\(\triangleright\) set full network as a subnetwork
3: Train \(\mathcal{N}_{1},\mathcal{N}_{2},\ldots,\mathcal{N}_{k}\) on tasks \(\mathcal{D}_{1},\mathcal{D}_{2},\ldots,\mathcal{D}_{k}\) using Eq. 9.
4:for\(it=1,2,\ldots,\textit{num\_iters}\)do\(\triangleright\) repeat pruning
5:\(\mathcal{N}_{k}\leftarrow\textit{Pruning}(\mathcal{N}_{k},\mathcal{D}_{k})\)\(\triangleright\) reduce unimportant connections
6: Retrain subnetworks \(\mathcal{N}_{1},\mathcal{N}_{2},\ldots,\mathcal{N}_{k}\) on tasks \(\mathcal{D}_{1},\mathcal{D}_{2},\ldots,\mathcal{D}_{k}\) using Eq. 9.
7:endfor
8:endfor
```

**Algorithm** PINN incremental learning

The main advantage of the proposed approach is that a neural network learns _all_ tasks (subdomains or equations) that were given during training and not only the last one. This is achieved by constantly replaying old data. In the next section, we experimentally show that pretrained parts of the network help to improve the convergence process.

## 5 Numerical experiments

Our findings illustrate the advantage of the Algorithm over regular PINNs [46]. The Algorithm allows the network to learn multiple equations (P1) from the same family. Furthermore, by starting with simpler tasks, the network can learn more complex ones that cannot be learned separately.
Experiments setup.

Let us start by examining the proposed algorithms on the convection and reaction equations with periodic boundary conditions (P1). Following the setup in [30], we use a four-layer neural network with 50 neurons per layer. We use 1000 randomly selected collocation points on every time interval between 0 and 1 for \(\mathcal{L}_{\mathcal{F}}\). The Adam optimizer [29] is used to train the model.

To evaluate the performance of the algorithms we compare the final error after the last task. In addition, following continual learning literature [36], we compare backward and forward transfer metrics. Let us denote the test set as \(\mathcal{D}^{test}=\{(x^{i},t^{i},l):x^{i}\in[0,2\pi],\ t^{i}\in[0,1],\ l\) is the task-ID\(\}\), the solution of the equation at the point \((x^{i},t^{i},l)\) as \(\mathbf{u}_{l,k}^{i}=u_{l,k}^{i}(x^{i},t^{i})\), and \(\mathbf{\hat{u}}_{l,k}^{i}\) is a prediction of the model at point \((x^{i},t^{i},l)\) after task \(\mathcal{D}_{k}\) is learned. Relative and absolute errors are denoted as \(r_{l,k}\) and \(\varepsilon_{l,k}\), respectively, as they are calculated for task \(l\) after task \(k\) is learned (\(l\leq k\)).

Relative error: \[r_{l,k}=\frac{1}{N}\frac{||\mathbf{u}_{l}-\mathbf{\hat{u}}_{l,k} ||_{2}}{||\mathbf{u}_{l}||_{2}}\times 100\%,\] (10) Absolute error: \[\varepsilon_{l,k}=\frac{1}{N}\sum_{i=1}^{N}\lvert\mathbf{u}_{l}^{i}- \mathbf{\hat{u}}_{l,k}^{i}\rvert,\] (11) Backward Transfer: \[\text{BWT}=\frac{1}{k-1}\sum_{l=1}^{k-1}\varepsilon_{l,k}- \varepsilon_{l,l}\ \ \text{or}\] (12) BWT \[=\frac{1}{k-1}\sum_{l=1}^{k-1}r_{l,k}-r_{l,l}\] (13)

### Results

Table 1 presents the results after all reaction equations are learned varying \(\rho\) from 1 to 5. Figure 3 shows the error history for every equation after incremental steps. The Table summarizes the performance improvement of iPINNs compared to regular PINNs, exhibiting negligible error for all values of \(\rho\), which is especially relevant for cases when \(\rho\) is larger. Moreover, iPINNs provide negative BWT which means that previous subnetworks help to learn the following ones.

Similarly, we observe for the convection equation the same learning behaviour. By learning incrementally the sequence of convection equations, we achieve much lower absolute and relative errors for the equations that are more difficult to learn (\(\beta=30,40\)). In Table 2 we show final errors at the end of training, and Figure 4 shows the absolute error history for each equation.

In Figures 5 and 6, we illustrate the error of iPINNs on convection and reaction equations and the exact solutions for every value of parameter \(\beta\) or \(\rho\) that were considered. Overall, we see that the neural network learns more complicated tasks more accurately if parts of the network are pretrained with easier tasks. At the same time, iPINNs replay the training data for previous PDEs during training for the new one. There are no additional costs to store or
generate input points \((x,t)\) for previous tasks since they can be easily sampled when necessary.

Another illustration of the method is learning the problem P2. We consider the values of \(\rho\) and \(\nu\) for which PINN does not have difficulties with learning each component separately. Results obtained when first learning the reaction

\begin{table}
\begin{tabular}{c c c c} \hline \hline  & regular PINN & iPINN \\ \hline \multirow{2}{*}{\(\rho=1\)} & abs. err & \(1.09\times 10^{-3}\) & \(\mathbf{1.5\times 10^{-4}}\) \\  & rel. err & 0.263\% & \(\mathbf{0.039\%}\) \\ \hline \multirow{2}{*}{\(\rho=2\)} & abs. err & \(1.97\times 10^{-3}\) & \(\mathbf{2.5\times 10^{-4}}\) \\  & rel. err & 0.479\% & \(\mathbf{0.070\%}\) \\ \hline \multirow{2}{*}{\(\rho=3\)} & abs. err & \(6.72\times 10^{-3}\) & \(\mathbf{6.1\times 10^{-4}}\) \\  & rel. err & 2.05\% & \(\mathbf{0.210\%}\) \\ \hline \multirow{2}{*}{\(\rho=4\)} & abs. err & \(1.13\times 10^{-2}\) & \(\mathbf{1.18\times 10^{-3}}\) \\  & rel. err & 3.68\% & \(\mathbf{0.458\%}\) \\ \hline \multirow{2}{*}{\(\rho=5\)} & abs. err & \(5.04\times 10^{-2}\) & \(\mathbf{1.91\times 10^{-3}}\) \\  & rel. err & 12.19\% & \(\mathbf{0.763\%}\) \\ \hline \hline \multirow{2}{*}{BWT} & abs. err & N/A & -\(3.8\times 10^{-4}\) \\  & rel. err & N/A & -0.112\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: Final error and forgetting after all reaction equations are learned.
part (diffusion part) are shown in Table 3 (Table 4). The main finding is that the network can learn almost every equation at least as well as when it is learned independently. In fact, for the reaction equation, the neural network improves significantly the prediction error. Another interesting observation is that the model learns the reaction-diffusion equation with almost the same error, regardless of the order of the tasks.

Figure 5: iPINNs on 1-D reaction equation.

Figure 6: iPINNs on 1-D convection equation.
\begin{table}
\begin{tabular}{l l l c c} \hline \hline parameters & equation & \multicolumn{2}{c}{regular PINN} & iPINN \\ \hline  & diffusion & abs. err & \(\mathbf{1.38\times 10^{-4}}\) & \(8.64\times 10^{-4}\) \\  & & rel. err & \(\mathbf{0.05\%}\) & \(0.28\%\) \\ \(\rho=3,\ \nu=5\) & reaction & abs. err & \(6.72\times 10^{-3}\) & \(\mathbf{2.11\times 10^{-3}}\) \\  & & rel. err & \(2.05\%\) & \(\mathbf{0.68\%}\) \\  & reaction-diffusion & abs. err & \(4.89\times 10^{-3}\) & \(\mathbf{4.07\times 10^{-3}}\) \\  & & rel. err & \(0.80\%\) & \(\mathbf{0.67\%}\) \\ \hline  & diffusion & abs. err & \(4.35\times 10^{-4}\) & \(\mathbf{3.45\times 10^{-4}}\) \\ \(\rho=4,\ \nu=4\) & reaction & abs. err & \(0.16\%\) & \(\mathbf{0.12\%}\) \\  & reaction-diffusion & abs. err & \(1.13\times 10^{-2}\) & \(\mathbf{4.91\times 10^{-3}}\) \\  & reaction-diffusion & rel. err & \(3.68\%\) & \(\mathbf{1.97\%}\) \\  & reaction-diffusion & abs. err & \(4.58\times 10^{-3}\) & \(\mathbf{4.42\times 10^{-3}}\) \\  & & rel. err & \(0.70\%\) & \(\mathbf{0.67\%}\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Final error and forgetting for diffusion \(\rightarrow\) reaction \(\rightarrow\) reaction-diffusion.

\begin{table}
\begin{tabular}{l l l c c} \hline \hline parameters & equation & \multicolumn{2}{c}{regular PINN} & iPINN \\ \hline  & reaction & abs. err & \(6.72\times 10^{-3}\) & \(\mathbf{9.41\times 10^{-4}}\) \\  & & rel. err & \(2.05\%\) & \(\mathbf{0.31\%}\) \\ \(\rho=3,\ \nu=5\) & diffusion & abs. err & \(\mathbf{1.38\times 10^{-4}}\) & \(1.85\times 10^{-4}\) \\  & & rel. err & \(\mathbf{0.05\%}\) & \(0.06\%\) \\  & reaction-diffusion & abs. err & \(4.89\times 10^{-3}\) & \(\mathbf{4.10\times 10^{-3}}\) \\  & reaction-diffusion & rel. err & \(0.80\%\) & \(\mathbf{0.68\%}\) \\ \hline  & reaction & abs. err & \(1.13\times 10^{-2}\) & \(\mathbf{7.88\times 10^{-3}}\) \\ \(\rho=4,\ \nu=4\) & diffusion & abs. err & \(3.68\%\) & \(\mathbf{2.99\%}\) \\  & & abs. err & \(\mathbf{4.35\times 10^{-4}}\) & \(5.84\times 10^{-4}\) \\  & reaction-diffusion & rel. err & \(0.16\%\) & \(0.19\%\) \\  & reaction-diffusion & abs. err & \(4.58\times 10^{-3}\) & \(\mathbf{4.42\times 10^{-3}}\) \\  & reaction-diffusion & rel. err & \(0.69\%\) & \(\mathbf{0.65\%}\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Final error and forgetting for reaction \(\rightarrow\) diffusion \(\rightarrow\) reaction-diffusion.
## 6 Additional study

In this section, we provide additional information about the learning procedure of iPINNs. We highlight some important training details such as the presence of regularization and the choice of activation functions. Also, we explore the subnetworks that our approach produces showing the proportion of parameters allocated to each task.

### Sensitivity to hyperparameters

Here we illustrate the influence of different training hyperparameters on the performance of iPINNs. First, we compare the results with and without regularization parameter (weight decay). In Figure 7, it can be observed that the presence of weight decay worsens the prediction error. However, looking at the result it is clear that iPINNs still work if weight decay is present. We can explain the lack of need for weight decay with the fact that many parameters are assigned to multiple tasks and cannot overfit to a particular one. Each subnetwork is also less parameterized than the original network and therefore does not tend to overfit. Thus, weight decay is not necessary and its presence only worsens the result due to the complication of the optimization procedure.

Furthermore, we compare the performance when using sin and tanh activation functions for two task orderings in Figure 8. We observe that sin works significantly better in both cases. Also, we test ReLU activation but it demonstrates poor performance in both PDEs orderings. If the reaction is learned first, the absolute errors are \(0.4959,0.2369\) and \(0.1493\). If we start with the diffusion equation and then learn reaction and reaction-diffusion PDEs, the errors are \(0.2399,0.2977\) and \(0.3003\).

In addition, we present how different values of pruning parameter \(\alpha\) affect the results. The higher the value of \(\alpha\) is, the less the network is pruned. Therefore, if

Figure 7: Influence of weight decay on the results for reaction (left) and convection (right) equations after all tasks are learned.
\(\alpha=0.95\) the task-specific subnetworks are sparser than with \(\alpha=0.99\) but less sparse if \(\alpha=0.9\). In Figure 9, we observe that for the reaction equation, we can prune less and achieve better performance which can be explained by the fact that PDEs in the reaction family are quite similar. Therefore, we can allow the network to have more overlaps to share knowledge between subnetworks. For the case of learning within the same family of convection PDEs, the value of \(\alpha=0.95\) was revealed to be a better option for constructing a sufficiently expressive task-specific subnetwork and frees space for future tasks. Notwithstanding, the performance is good with any reasonable choice of pruning parameter.

### Subnetworks analysis

In Figure 10, we present the portions of the subnetworks that are occupied by each task. We will illustrate this by considering both orders - when the model learns the reaction equation first (Figure 10a), and when diffusion comes first

Figure 8: Influence of activation function on the results when the reaction learned first (left) and diffusion learned first (right).

Figure 9: iPINNs with different values of pruning parameter \(\alpha\).
(Figure 9(b)). These results are averaged over 3 different runs for each of the orderings. It is noteworthy that the percentage of parameters occupied by all tasks is very similar for both orderings (31.8% and 31.5% respectively of all network parameters). On the other hand, the percentages of used parameters for both cases are 79.5% and 79.3%. This means that the total number of trained parameters for the two incremental procedures is the same for both cases, which shows the robustness of the method. Moreover, the network has about 20% of free connections to learn new tasks.

## 7 Conclusion

In this work, we propose an incremental learning approach for PINNs where every task is presented as a new PDE. Our algorithm is based on task-related subnetworks for every task obtained by iterative pruning. To illustrate our idea, we consider two cases when incremental learning is applicable to a sequence of PDEs. In the first case, we consider the family of convection/reaction PDEs, learning them sequentially. In the second example, we consider the reaction-diffusion equation and learn firstly the components of the process, namely reaction and diffusion, and only then the reaction-diffusion equation. Our main goal is to show the possibility of incremental learning for PINNs without significantly forgetting previous tasks. From our numerical experiments, the proposed algorithm can learn all the given tasks, which is not possible with standard PINNs. Importantly, we also show that future tasks are learned better because they can share connections trained from previous tasks, leading to significantly better performance than if these tasks were learned independently. We demonstrate that this stems from the transfer of knowledge occurring between subnetworks that are associated with each task. Interestingly, the model's performance on previous tasks is also improved by learning the following tasks. In essence, iPINNs demonstrate symbiotic training effects between past and future tasks by learning them with a single network composed of dedicated subnetworks for each task that share relevant neuronal connections.

Figure 10: Percentage of parameters used for every equation with \(\rho=4,~{}\nu=4\).