Understanding and Mitigating Extrapolation Failures in Physics-Informed
  Neural Networks
Physics-informed Neural Networks (PINNs) have recently gained popularity due
to their effective approximation of partial differential equations (PDEs) using
deep neural networks (DNNs). However, their out of domain behavior is not well
understood, with previous work speculating that the presence of high frequency
components in the solution function might be to blame for poor extrapolation
performance. In this paper, we study the extrapolation behavior of PINNs on a
representative set of PDEs of different types, including high-dimensional PDEs.
We find that failure to extrapolate is not caused by high frequencies in the
solution function, but rather by shifts in the support of the Fourier spectrum
over time. We term these spectral shifts and quantify them by introducing a
Weighted Wasserstein-Fourier distance (WWF). We show that the WWF can be used
to predict PINN extrapolation performance, and that in the absence of
significant spectral shifts, PINN predictions stay close to the true solution
even in extrapolation. Finally, we propose a transfer learning-based strategy
to mitigate the effects of larger spectral shifts, which decreases
extrapolation errors by up to 82%.
Lukas Fesser, Luca D'Amico-Wong, Richard Qiu
# Understanding and Mitigating Extrapolation Failures in Physics-Informed Neural Networks

###### Abstract

Physics-informed Neural Networks (PINNs) have recently gained popularity in the scientific community due to their effective approximation of partial differential equations (PDEs) using deep neural networks. However, their application has been generally limited to interpolation scenarios, where predictions rely on inputs within the support of the training set. In real-world applications, extrapolation is often required, but the out of domain behavior of PINNs is understudied. In this paper, we provide a detailed investigation of PINNs' extrapolation behavior and provide evidence against several previously held assumptions: we study the effects of different model choices on extrapolation and find that once the model can achieve zero interpolation error, further increases in architecture size or in the number of points sampled have no effect on extrapolation behavior. We also show that for some PDEs, PINNs perform nearly as well in extrapolation as in interpolation. By analyzing the Fourier spectra of the solution functions, we characterize the PDEs that yield favorable extrapolation behavior, and show that the presence of high frequencies in the solution function is not to blame for poor extrapolation behavior. Finally, we propose a transfer learning-based strategy based on our Fourier results, which decreases extrapolation errors in PINNs by up to \(82\%\).

## 1 Introduction

Understanding the dynamics of complex physical processes is crucial in many applications in science and engineering. Oftentimes, these dynamics are modeled as partial differential equations (PDEs) that depend on time. In the PDE setting, we want to find a solution function \(u(x,t)\) that satisfies a given governing equation of the form

\[f(x,t):=u_{t}+\mathcal{N}(u)=0,x\in\Omega,t\in[0,T] \tag{1}\]

where \(u_{t}:=\frac{\partial u}{\partial t}\) denotes the partial derivative of \(u\) with respect to time, \(\mathcal{N}\) is a - generally nonlinear - differential operator, \(\Omega\subset\mathbb{R}^{d}\), with \(d\in\{1,2,3\}\) is a spatial domain, and \(T\) is the final time for which we're interested in the solution. Moreover, we impose an initial condition \(u(x,0)=u^{0}(x)\), \(\forall x\in\Omega\) on \(u(x,t)\), as well as a set of boundary conditions. Together, these conditions specify the behaviors of the solution on the boundaries of the spatio-temporal domain. Generally, solving these problems is particularly difficult when the differential operator \(\mathcal{N}\) is highly nonlinear.
With recent progress in deep learning, many data-centric approaches based on the universal approximation theorem have been proposed. Among these approaches, physics-informed neural networks (PINN) as introduced in [16] have caught the community's attention because of their simple, but effective way of approximating time-dependent PDEs with relatively simple deep neural networks. PINNs preserve important physical properties described by the governing equations by parameterizing the solution and the governing equation simultaneously with a set of shared network parameters. We will go into more detail on this in the next section. After the great success of the seminal paper [16], many sequels have applied PINNs to solve various PDE applications, e.g. [1, 19, 20, 5].

However, most previous studies using the vanilla PINNs introduced in [16] have demonstrated the performances of their methods in interpolation only. By interpolation, we mean a set of testing points sampled within the same temporal range that the network has been trained. We refer to points sampled beyond the final time of the training domain as extrapolation. In principle, standard PINNs are expected to be able to learn the dynamics in Eq. (1) and, consequently, to approximate \(u(x,t)\) accurately in extrapolation. However, previous work in [7] and [17] has shown that this is not the case: PINNs can deviate significantly from the true solution once they are evaluated in an extrapolation setting, calling into question their capability as a tool for learning the dynamics of physical processes.

The question of whether and when PINNs can extrapolate beyond their training domain is therefore relevant from a foundational standpoint, but it also has immediate consequences for applications. As discussed in [2] and [22], PINNs cannot always be retrained from scratch when faced with a point that is outside their initial training domain, so anticipating whether their predictions remain accurate beyond the training domain is crucial. Several PINN-related methods for improving extrapolation behavior - also called long-time integration - have been proposed. However, these methods either use specialized architectures instead of the standard MLP architecture [11, 21], or recast the problem by sequentially constructing the solution function \(u(x,t)\)[17]. To the best of our knowledge, only [7] propose a method for the vanilla PINN setting introduced in [16]. In addition to this, even a basic characterization of extrapolation behavior for PINNs trained to solve time-dependent PDEs is still absent from the literature, with previous work considering vanilla PINNs incapable of extrapolating beyond the training domain and suspecting the presence of high frequencies in the solution function to be the cause of the problem [17]. In this paper, our contributions are therefore as follows.

* We show that PINNs are capable of almost perfect extrapolation behavior for certain time-dependent PDEs. However, we also provide evidence that even when the learned solution stays close to the true solution in extrapolation, the \(L^{2}\)-error increases exponentially in time, independent of model size, choice of activation function, training time, and number of samples.
* We characterize PDEs for which PINNs can extrapolate well by analyzing the Fourier spectra of the solution functions. We show that unlike with training failures in interpolation, the presence of high frequencies alone is not to blame for the poor extrapolation behavior of PINNs on some PDEs. Rather, standard PINNs seem to generally fail to anticipate shifts in the support of the Fourier spectrum over time. The PDEs for which extrapolation works well exhibit a constant support of the solution function's Fourier spectra over time.
* We show that transfer learning from a wider family of PDEs can reduce extrapolation error, even when the initial training regime does not contain the extrapolation domain. This suggests that placing stronger inductive biases, through transfer learning, may improve extrapolation behavior in vanilla PINNs.

The rest of this paper is structured as follows: in section 2, we formally introduce physics-informed neural networks and define what we mean by interpolation and extrapolation. We also briefly discuss the DPM method proposed by [7] and its performance on several benchmark PDEs. In section 3, we study PINNs' extrapolation behavior for various PDEs, and show that good extrapolation behavior is attainable, even with standard PINNs. In section 4, we go a step further and characterize the PDEs for which good extrapolation accuracy is possible using the Fourier spectra of their solution functions. In section 5, we investigate the viability of transfer learning approaches in improving extrapolation and demonstrate experimental results. Finally, section 6 discusses our results in the context of the existing literature and section 7 concludes.
Background and definitions

### Physics-Informed Neural Networks

As mentioned in the previous section, PINNs parameterize both the solution \(u\) and the governing equation \(f\). Denote the neural network approximating the solution \(u(x,t)\) by \(\tilde{u}(x,t;\theta)\) and let \(\theta\) be the network's weights. Then the governing equation \(f\) is approximated by a neural network \(\tilde{f}(x,t,\tilde{u};\theta):=\tilde{u}_{t}+\mathcal{N}(\tilde{u}(x,t; \theta))\). The partial derivatives here can be obtained via automatic differentiation. We note that \(\tilde{f}(x,t,\tilde{u};\theta)\) shares its network weights with \(\tilde{u}(x,t;\theta)\). The name "physics-informed" neural network comes from the fact that the physical laws we're interested in are enforced by applying an extra, problem-specific, nonlinear activation, which is defined by the PDE in Eq. (1) (i.e., \(\tilde{u}_{t}+\mathcal{N}(\tilde{u})\)).

We learn the shared network weights using a loss function consisting of two terms, which are associated with approximation errors in \(\tilde{u}\) and \(\tilde{f}\), respectively. The original paper [16] considers a loss function consisting of two error terms, i.e. \(L:=\alpha L_{u}+\beta L_{f}\). Here, \(\alpha,\beta\in\mathbb{R}\) are coefficients and \(L_{u}\) and \(L_{f}\) are defined as follows:

\[L_{u} =\frac{1}{N_{u}}\sum_{i=1}^{N_{u}}\left|u(x_{u}^{i},t_{u}^{i})- \tilde{u}(x_{u}^{i},t_{u}^{i};\theta)\right|^{2} \tag{2}\] \[L_{f} =\frac{1}{N_{f}}\sum_{i=1}^{N_{f}}\left|\tilde{f}(x_{f}^{i},t_{f }^{i},\tilde{u};\theta)\right|^{2} \tag{3}\]

\(L_{u}\) enforces the initial and boundary conditions using a set of training data \(\left\{(x_{u}^{i},t_{u}^{i}),u(x_{u}^{i},t_{u}^{i})\right\}_{i=1}^{N_{u}}\). The first element of the tuple is the input to the neural network \(\tilde{u}\) and the second element is the ground truth that the output of \(\tilde{u}\) attempts to match. We can collect this data from the specified initial and boundary conditions since we know them a priori. Meanwhile, \(L_{f}\) minimizes the discrepancy between the governing equation \(f\) and the neural network's approximation \(\tilde{f}\). We evaluate the network at collocation points \(\left\{(x_{f}^{i},t_{f}^{i}),f(x_{f}^{i},t_{f}^{i})\right\}_{i=1}^{N_{f}}\). Note that here, the ground truth \(\left\{f(x_{u}^{i},t_{u}^{i})\right\}_{i=1}^{N_{f}}\) consists of all zeros. We also refer to \(\frac{1}{N_{f}}\sum_{i=1}^{N_{f}}\left|\tilde{f}(x_{f}^{i},t_{f}^{i},\tilde{u };\theta)\right|\) as the mean absolute residual (MAE): its value denotes how far the network is away from satisfying the governing equation.

Note that using this loss, i) no costly evaluations of the solutions \(u(x,t)\) at collocation points are required to gather training data, ii) initial and boundary conditions are enforced using a training dataset that can easily be generated, and iii) the physical law encoded in the governing equation \(f\) in Eq. (1) is enforced by minimizing \(L_{f}\). In the original paper by [16], both loss terms have equal weight, i.e. \(\alpha=\beta=1\), and the combined loss term \(L\) is minimized.

### Interpolation and extrapolation

For the rest of this paper, as mentioned briefly in section 1, we refer to points \((x^{i},t^{i})\) as _interpolation points_ if \(t^{i}\in[0,T_{train}]\), and as _extrapolation points_ if \(t^{i}\in(T_{train},T]\) for \(T>T_{train}\). We are primarily interested in the \(L^{2}\)_error_ of the learned solution, i.e. in \(\|u(x^{i},t^{i})-\tilde{u}(x^{i},t^{i};\theta)\|_{2}\), and in the \(L^{2}\)_relative error_, which is the \(L^{2}\) error divided by the norm of the function value at that point, i.e. \(\|u(x^{i},t^{i})\|_{2}\). When we sample evaluation points from the extrapolation domain, we refer to the \(L^{2}\) (relative) error as the _(relative) extrapolation error_. Similarly, we are interested in the (mean) absolute residual as defined above, i.e. in \(\left|\tilde{f}(x^{i},t^{i},\tilde{u};\theta)\right|\). For points sampled from the extrapolation domain, we refer to this as the _extrapolation residual_.

### Prior work on extrapolation failures in vanilla PINNs

The dynamic pulling method (DPM) proposed by [7] is motivated by the observation that PINNs typically perform poorly in extrapolation. They attribute this poor generalization performance to the noisy domain loss curves during training and they propose the dynamic pulling method (DPM), which modifies the gradient update as a way to smooth the domain loss curves and thereby improve
extrapolation performance. Their method introduces several hyperparameters and they find that following hyperparameter search, their method has better generalization performance compared to vanilla PINNs. We briefly investigate the performance of their method and outline our experiments in the appendix.

## 3 Characterization of extrapolation behavior in PINNs

In this section, we compare the extrapolation errors and residuals which standard PINNs display for the Allen-Cahn equation, the viscous Burgers equation, a diffusion equation, and a diffusion-reaction equation, before varying the model's parameters in the second half of the section. More details on the PDEs under consideration can be found in the appendix.

### Extrapolation behavior for different PDEs

For each of the four PDEs introduced above, we train a 4-layer MLP with 50 neurons per layer and \(tanh\) activation on the interpolation domains specified for 50000 epochs using the adam optimizer. Figures 1 **(a)** and **(b)** show the \(L^{2}\) relative error and the mean absolute residual for \(t\in[0,1]\). We note that for all four PDEs, we achieve zero interpolation error and residual.

We observe that the \(L^{2}\) relative errors for the Burgers' equation and for the Allen-Cahn equation become significantly larger than for the diffusion and diffusion-reaction equations when we move from \(t=0.5\) to \(t=1\). As can be seen in Figure 1**(c)**, the solution learned for the diffusion-reaction equation disagrees only minimally with the true solution, even at \(t=1\), which shows that for this particular PDE, PINNs can extrapolate almost perfectly well. However, we also find that irrespective of the PDE we consider, the \(L^{2}\) relative error and the MAR increase exponentially in \(t\) as soon as we leave the interpolation domain.

Figure 1: **(a)**\(L^{2}\) relative extrapolation error of MLP(4, 50) with \(tanh\) activation, trained on \([0,0.5]\). **(b)** MAR for the same MLP, and **(c)** the solution for the diffusion-reaction equation at \(t=1\) and the function learned by the corresponding MLP.

Figure 2: Mean \(L^{2}\) relative errors over the interpolation (extrapolation) domain of MLP(4, 50) with \(tanh\) activation in **(a)** and with \(\sin\) activation in **(b)** with increasing number of training epochs. **(c)** plots the relative error against the number of samples, in the order (domain, boundary condition, initial condition).
### Varying model parameters

While we observe drastically different extrapolation behaviors depending on the underlying PDE as mentioned above, the extrapolation for a given PDE seems to be more or less independent of model choices, such as number of layers or neurons per layer, activation function, number of samples, or training time. Once the chosen parameters allow the model to achieve zero error in the interpolation domain, adding more layers, neurons, or samples, or alternatively training longer does not seem to have an effect on the extrapolation error and MAR. Figure 2 illustrates this behavior for the Burger's equation by plotting the interpolation and extrapolation errors when training for an increasing number of epochs: **(a)** and **(b)** show this for a \(tanh\) and a \(sin\) activation function, respectively. Figure 2**(c)** shows the effect of increasing the number of samples from the interpolation domain on the extrapolation error. Depicted is the relative \(L^{2}\) error of a 4-layer MLP with 50 neurons each and \(tanh\) activation, trained using the same hyperparameters as in the previous section.

## 4 Fourier-based analysis of extrapolation behavior

Motivated by the results of section 3, one might ask which structural elements of a given PDE determine extrapolation performance. That is, given a defined PDE, can we predict how well PINNs will extrapolate on the equation beyond their training domain?

Recent literature has found that neural networks tend to be biased towards low-complexity solutions due to implicit regularization inherent in their gradient descent learning processes [13; 12]. In particular, deep neural networks have been found to possess an inductive bias towards learning lower frequency functions, a phenomenon termed the _spectral bias_ of neural networks [15; 3].

To better understand how these phenomena relate to the extrapolation behavior of PINNs, we examine the true and predicted solutions of our four PDEs in the Fourier domain. While we find little evidence to support the claim that the presence of high frequency components hurts extrapolation, we find that the dynamics of the Fourier spectra for a given PDE tend to be strongly predictive of extrapolation performance.

Specifically, we decompose the change in the true solution's Fourier spectra over time into two types - changes in the overall amplitude of the components and changes in how these amplitudes are distributed over the support of the spectra. While PINNs are able to handle changes in amplitude quite well, we find that extrapolation is especially poor for those PDEs whose spectra shift over time.

### Examining solutions in the Fourier domain

Towards understanding the interaction of extrapolation behavior with the Fourier spectrum1 of the solution to a given PDE, we plot both the reference solution and the predicted solution in the Fourier domain for all four of our PDEs. We additionally plot the absolute difference between the two Fourier spectra of the reference and predicted solution. Plots for the Burgers' equation are provided in Figure 3 while plots for the other three PDEs are provided in Appendix A.3.

Footnote 1: For a fixed time \(t\), the Fourier spectrum is computed using a discrete Fourier transform on \(256\) equally spaced points (\(201\) for Allen-Cahn) in the spatial domain.

In all cases, the majority of the error in the Fourier domain is concentrated in the lower-frequency regions. While this is partially due to the fact that the low frequency components of the solutions have larger magnitude, it suggests that in extrapolation, PINNs fail even to learn the low frequency parts of the solution. Thus, the presence of high frequencies alone fails to explain the extrapolation failure of PINNs. We provide some additional evidence for this by studying the extrapolation behavior of Multi-scale Fourier feature networks [18] in Appendix A.6. Even though these architectures were designed specifically to make learning higher frequencies easier, we find their extrapolation error to be at least as large or larger than that of standard PINNs.

There are still notable differences between the spectra of the two "good" PDEs and the two "bad" PDEs, however. Namely, both the diffusion and diffusion-reaction equations have Fourier spectra whose support does not shift over time; the amplitudes of each frequency decrease with time, but the distribution of these amplitudes is constant. In contrast, for both the Burgers' and Allen-Cahn
equations, we see decreases in amplitudes accompanied by shifts in the spectra, suggesting that PINNs may struggle to extrapolate well when the true solution's Fourier spectrum shifts over time.

### Decomposing the changes in Fourier spectra

Given the observed differences in the temporal dynamics of the Fourier spectra, we aim to isolate changes in the overall amplitudes from shifts in the Fourier spectra to better understand how each change affects extrapolation behavior.

#### 4.2.1 Changes in amplitude

To isolate the effect of amplitude changes, we train a PINN on the following PDE.

\[\frac{\partial u}{\partial t} =\frac{\partial^{2}u}{\partial x^{2}}+e^{-t}\left(\sum_{j=1}^{K} \frac{(j^{2}-1)}{j}\sin(jx)\right) \tag{4}\] \[u(x,0) =\sum_{j=1}^{K}\frac{\sin(jx)}{j}\qquad u(-\pi,t)=u(\pi,t)=0 \tag{5}\]

for \(x\in[-\pi,\pi]\) and \(t\in[0,1]\). The reference solution is given by \(u(x,t)=e^{-t}\left(\sum_{j=1}^{K}\frac{\sin(jx)}{j}\right)\) As with our other experiments, we use \(t\in[0,0.5]\) as the temporal training domain and consider \(t\in(0.5,1]\) as the extrapolation area. \(K\) here is a hyperparameter that controls the size of the spectrum of the solution. Note that for a fixed \(K\), the support of the Fourier spectrum of the reference solution never changes over time, with only the amplitudes of each component scaled down by an identical constant factor.

For various values of \(K\), we find that our trained PINNs are able to extrapolate well, provided that we use larger architectures for more extreme values of \(K\), as can be seen in Figure 4**(a)**. Together with our observations of the diffusion and diffusion-reaction equation, this provides support for the claim that extrapolation failures seem to be primarily governed by shifts in the underlying spectra

Figure 3: For times \(t=0.25\) (top, interpolation) and \(t=0.99\) (bottom, extrapolation), we plot the reference and predicted solutions in the spatio-temporal (left) and Fourier (middle) domains for the Burgers’ equation. The absolute difference in the Fourier spectra is plotted on the right.
rather than changes in the amplitude of the respective components. Additionally, these observations continue to hold even when we increase the complexity of the reference solution by increasing the support of the Fourier spectra.

For the sake of completeness, we also investigate the effect of the speed of decay of the amplitudes in the Fourier spectra. We train a PINN on the following variation of the Diffusion-Reaction equation.

\[\frac{\partial u}{\partial t}=\frac{\partial^{2}u}{\partial x^{2}}+e^{-Mt} \left(\sum_{j\in\{1,2,3,4,8\}}\frac{(j^{2}-1)}{j}\sin(jx)\right) \tag{6}\]

for \(x\in[-\pi,\pi]\) and \(t\in[0,1]\) with the initial condition \(u(x,0)=\sin(x)+\frac{\sin(2x)}{2}+\frac{\sin(3x)}{3}+\frac{\sin(4x)}{4}+\frac {\sin(8x)}{8}\) and the Dirichlet boundary condition \(u(-\pi,t)=u(\pi,t)=0\). The reference solution is

\[u(x,t)=e^{-Mt}\left(\sin(x)+\frac{\sin(2x)}{2}+\frac{\sin(3x)}{3}+\frac{\sin(4 x)}{4}+\frac{\sin(8x)}{8}\right) \tag{7}\]

with the same interpolation and extrapolation areas as before. Figure 4**(b)** shows the relative interpolation and extrapolation errors against increasing values of \(M\). We find that an increase in the speed of the decay seem to increase the extrapolation error more than an increase in the size of the spectrum. However, since the solutions to the Allen-Cahn equation and to the Burger's equation are not exhibiting exponentially fast changes in their amplitudes, we now focus on shifts in the support of the spectrum instead.

#### 4.2.2 Changes in the support

To quantify the temporal shifts in the support of the Fourier spectrum, we consider the _Wasserstein distance_ between the normalized Fourier spectra at different points in time. Consider two discrete CDFs \(F_{1},F_{2}\) supported on the domain \(\mathcal{X}\). The Wasserstein distance between \(F_{1}\) and \(F_{2}\) is defined as \(W(F_{1},F_{2})=\sum_{x\in\mathcal{X}}|F_{1}(x)-F_{2}(x)|\). Given two discrete Fourier spectra \(f_{1},f_{2}\), the _Wasserstein-Fourier distance_[4]. is defined as \(W\left(\frac{f_{1}}{\|f_{1}\|_{1}},\frac{f_{2}}{\|f_{2}\|_{1}}\right)\). The Wasserstein distance is closely related to optimal transport theory and represents the minimum amount of probability mass needed to convert one distribution into another.

We present plots of the pairwise Wasserstein-Fourier distances for each \(t_{1},t_{2}\in[0,1]\) for each of our four PDEs in appendix A.4. The Wasserstein-Fourier distance of the true solution is zero everywhere for both the diffusion and diffusion-reaction equations, reflecting the constant support of the spectra. In contrast, the pairwise distance matrices for the Burgers' and Allen-Cahn equations exhibit a

Figure 4: Mean \(L^{2}\) relative interpolation and extrapolation errors of MLP(6, 50) with tanh activation, trained on [0, 0.5]. In **(a)**, we plot this against the size of the spectrum i.e. the parameter \(K\) in Equation (4), and in **(b)** we plot this against the speed of the decay of the amplitudes, i.e. the parameter \(M\) in Equation (6).
block-like structure, with times in disjoint blocks exhibiting pronouncedly different distributions in the amplitudes of their respective Fourier spectra. These shifts are not captured by the learned solution, leading to large \(L^{2}\) errors.

## 5 Transfer learning and extrapolation

Finally, we examine whether transfer learning from PINNs trained across a family of similar PDEs can improve generalization performance. Empirically, in other domains, transfer learning across multiple tasks has been effective in improving generalization [6; 10]. Roughly speaking, for PINNs, one might expect transfer learning to cement stronger inductive biases towards learning the true PDE kernel than training from scratch, which will in turn improve generalization performance.

We perform transfer learning following the procedure outlined in [14], where we initially train on a PINN with multiple outputs on a sample from a family of PDEs (e.g. the Burgers' equation with varying values of the viscosity) and transfer to a new unseen PDE in the same family (e.g. the Burgers' equation with a different viscosity) by freezing all but the last layer and training with the loss this new PDE induces. We note that [14] only consider transfer learning for linear PDEs by analytically computing the final PINN layer but we extend their method to nonlinear PDEs by performing gradient descent to learn the final layer instead. We perform transfer learning from a collection of Burgers' equations with varying viscosities (\(\nu/\pi=\{0.01,0.05,0.1\}\)) to a new Burgers equation (\(\nu/\pi=0.075\)). In the first set of experiments, we train on equations in the domain \(t\in[0,0.5]\), and in the second set, we train on equations in the domain \(t\in[0,1]\). For all experiments, we use fully connected PINNs with 5 hidden layers, each 100 hidden units wide, Xavier normal initialization, and \(\tanh\) activation. We train with Adam [8] with default parameters. We use equal weighting of the boundary and domain loss terms. The extrapolation loss reported is sampled uniformly from the extrapolated domain (explicitly \((x,t)\in[-1,1]\times[0.5,1]\)).

We show results from our transfer learning experiments in Figure 5. For each experiment, we perform multiple runs, changing only the random seed. Motivated by [7], we primarily examine the interpolation and extrapolation loss of each run as well as decomposed into domain and boundary terms.

We observe first that for both baseline and transfer learning experiments, the extrapolation loss is dominated by the domain loss; the boundary loss is subdominant. And as one might expect, transfer learning from PDEs on the whole domain (\(t\in[0,1]\)) substantially improves results compared to baseline. However, we find that transfer learning even when the model does not see the extrapolation domain during initial training (e.g. \(t\in[0,0.5]\)) also improves performance over baseline, though less than transfer learning from the full domain. We find the reverse in interpolation. Namely, our baseline model has the lowest interpolation error, followed by half-domain transfer learning, and then full-domain transfer learning, which performs the worst in interpolation.

For a subset of experimental runs, we also examined the \(L^{2}\) error from numerical solutions in extrapolation. Compared to the baseline (no transfer learning), we found an average reduction in extrapolation error of \(82\%\) when transfer learning from the full domain, and of \(51\%\) when transfer learning from half the domain, i.e. with \(t\in[0,0.5]\).

## 6 Discussion

In section 3, we saw that independent of the model's parameters, such as model size, choice of activation functions, training time, and number of samples, the \(L^{2}\) relative error increases exponentially in time once we enter the extrapolation domain. This seems to provide at least some evidence against a double-descent phenomenon for the extrapolation error, which [22] speculated might exist. We also saw, using the diffusion and diffusion-reaction equations as examples, that standard PINNs are able to achieve excellent extrapolation behavior. This provides evidence against the assumption that PINNs may be incapable of yielding accurate solutions for extrapolation problems, made for example in [17].

Motivated by recent literature on the spectral bias of neural networks [3; 15; 18], we examined the reference and true solutions to our four PDEs in the Fourier domain in section 4. We found little evidence to support the claim that the presence of high frequencies hinders extrapolation, as was
speculated in [17]. Instead, we found that extrapolation performance is far better for those PDEs whose Fourier spectra exhibit unchanging support (diffusion, diffusion-reaction) than those PDEs whose spectra shift over time (Burgers', Allen-Cahn). Additionally, we looked at the Wasserstein-Fourier distance as a metric for quantifying the temporal shift in Fourier spectra and link it to extrapolation.

Finally, in section 5, we investigated whether transfer learning from several PDEs in a larger family of PDEs to a specific PDE within the same family can improve generalization performance. We found that compared to training from scratch, extrapolation performance is indeed improved with transfer learning, at the cost of interpolation performance. As one might expect, extrapolation performance is best when including the extrapolation domain in pre-training, though it is still clearly improved when the extrapolation domain is excluded. This may suggest that transfer learning enforces stronger inductive biases from the wider PDE family which in turn improve extrapolation performance.

Our results are limited in that we only examined four example PDEs, which future research might extend to other PDEs. We also leave it as future work to investigate the effects of architectures other than MLPs [16], such as Recurrent Neural Networks [11] and ResNets [7], and of other training paradigms, for example domain decomposition [9].

## 7 Conclusion

In summary, we have examined PINNs' extrapolation behavior with a wide range of tools and pushed back against assumptions previously held in the literature. We first found that for some PDEs, PINNs are capable of near perfect extrapolation. However, we also found that the extrapolation error for a given PINN increases exponentially in time, and that this increase is robust to choice of architecture and other hyperparameters of the model. Following this, we examined the solution space learned

Figure 5: Domain, boundary, and combined mean squared interpolation error (top) and extrapolation error (bottom) between our baseline (PINNs trained from scratch) and transfer learning experiments. The only variation between data points is the random seed. There are 50, 11, and 10 runs of the baseline, transfer with \(t\in[0,1]\), and transfer with \(t\in[0,0.5]\), respectively. Note the vertical scales differ between the interpolation and extrapolation domains.
by PINNs in the Fourier domain, finding that extrapolation performance is improved for PDEs with stable Fourier spectra, and that the presence of high frequencies in the solution function has minimal effect on extrapolation for large enough models. Finally, to shed light on the role of inductive bias in extrapolation, we show that transfer learning can improve extrapolation performance. We believe that our results are of immediate interest for PINN application areas, where standard PINNs are still the most common approach by far, and hope that our work will motivate further investigations into the generalization performance of PINNs.
