Sensitivity analysis using Physics-informed neural networks
The goal of this paper is to provide a simple approach to perform local
sensitivity analysis using Physics-informed neural networks (PINN). The main
idea lies in adding a new term in the loss function that regularizes the
solution in a small neighborhood near the nominal value of the parameter of
interest. The added term represents the derivative of the loss function with
respect to the parameter of interest. The result of this modification is a
solution to the problem along with the derivative of the solution with respect
to the parameter of interest (the sensitivity). We call the new technique
SA-PNN which stands for sensitivity analysis in PINN. The effectiveness of the
technique is shown using four examples: the first one is a simple
one-dimensional advection-diffusion problem to show the methodology, the second
is a two-dimensional Poisson's problem with nine parameters of interest, and
the third and fourth examples are one and two-dimensional transient two-phase
flow in porous media problem.
John M. Hanna, José V. Aguado, Sebastien Comas-Cardona, Ramzi Askri, Domenico Borzacchiello
# Sensitivity analysis using Physics-informed neural networks

###### Abstract

The paper's goal is to provide a simple unified approach to perform sensitivity analysis using Physics-informed neural networks (PINN). The main idea lies in adding a new term in the loss function that regularizes the solution in a small neighborhood near the nominal value of the parameter of interest. The added term represents the derivative of the residual with respect to the parameter of interest. The result of this modification is a solution to the problem along with the derivative of the solution with respect to the parameter of interest (the sensitivity). We call the new technique to perform sensitivity analysis within this context SA-PINN. We show the effectiveness of the technique using 3 examples: the first one is a simple 1D advection-diffusion problem to show the methodology, the second is a 2D Poisson's problem with 9 parameters of interest and the last one is a transient two-phase flow in porous media problem.

keywords: Physics-informed neural networks, sensitivity analysis, two-phase flow in porous media +
Footnote †: journal: Computer Methods in Applied Mechanics and Engineering

## 1 Introduction

Sensitivity analysis is a technique to measure the effect of uncertainties in one or more input parameters on output quantities. Sensitivity can be regarded, quantitatively, as the derivative of output quantities with respect to input parameters that might have some uncertainties. It has great importance in several
engineering applications. Examples of these applications include aerodynamic optimization [1], shape optimization in solid mechanics [2], injection molding [3] and biomedical applications [4; 5]. This is just to name few applications.

Several methods exist to calculate the sensitivities; the simplest one is the finite difference approach. It includes solving the system repeatedly, using any numerical technique, while varying the parameter of interest. Afterwards, the derivatives can be approximated by calculating the finite differences. This method becomes impractical when the number of the parameters of interest increases because the number of times the full system needs to be solved will grow exponentially, thus getting the sensitivities will become computationally intractable [6].

The adjoint method is becoming the state of the art for performing sensitivity analysis specially in CFD applications [7]. The method attempts to solve an adjoint system of equations that is usually derived from the primal system. By solving the adjoint system, one obtains the gradient values with respect to parameters of interest. Despite the success and increasing widespread of the method, some issues still exist. One of these issues is related to the differentiability of the solution with discontinuities appearing in shock wave problems or two-phase flow problems that leads to instabilities [8].

Machine learning based techniques have been growing rapidly to solve problems governed by partial differential equations (PDEs). Physics-informed neural networks (PINN) is one of the fast growing fields that attempts to solve forward and inverse problems governed by PDEs [9]. PINN gained wide interest due to the lack of need to a big data set since the physics governed by PDEs are used to regularize the model. PINN is powerful due to the use of feed forward neural networks, that are universal approximators [10], as the approximation space and the recent advances in automatic differentiation capabilities [11] that facilitated the derivative calculations. The method has been applied to several fields including solid mechanics [12], fluid mechanics [13; 14], additive manufacturing [15], two-phase flow in porous media [16] and many others [17; 18; 19; 20; 21; 22; 23; 24].
In this article, PINN is used as the base framework to develop the new sensitivity analysis technique. The main objectives of the article can be summarized as follows:

* Introducing a new technique to perform sensitivity analysis based on the framework of PINN and we call it SA-PINN.
* Showing the ease and effectiveness of getting the sensitivity with respect to multiple parameters of interest at once.
* Displaying the ability of SA-PINN to get the sensitivities for problems with discontinuities such as moving boundaries or sharp gradients.

The paper is organized in the following manner. In chapter 2, a quick introduction to PINN is given followed by explanation to SA-PINN. Chapter 3 introduces three problems presented in the paper: 1D advection-diffusion problem, 2D Poisson's problem with multiple parameters of interest and a 1D unsteady two-phase flow in porous media problem. Chapter 4 gives the results to the three problems which includes the calculation of some sensitivities of interest. Chapter 5 offers a summary to the method and a conclusion.

## 2 Sensitivity Analysis-PINN (SA-PINN)

We consider a general partial differential equation of the form

\[u_{t}+\mu L(u)=0,\quad\mathbf{x}\in\Omega,\ t\in[0,T] \tag{1}\]

where \(u_{t}\) is the time derivative, \(L\) a general differential operator and \(\mu\) a material parameter. Initial and boundary conditions for the problems are defined as

\[u(0,\mathbf{x}) =u_{0} \tag{2}\] \[u(t,\mathbf{x_{D}}) =u_{D}\] (3) \[B(u(t,\mathbf{x_{N}})) =f(\mathbf{x_{N}}) \tag{4}\]
where \(B\) is a differential operator, \(\mathbf{x_{D}}\) the boundary where Dirichlet boundary condition is enforced and \(\mathbf{x_{N}}\) the boundary where Neumann boundary condition is applied.

### PINN

The first step to solve this problem with PINN is to choose the approximation space. The choice is a feed forward neural network. Through automatic differentiation, a combined loss is formed of the residual of the PDE defined at spatio-temporal points called collocation points and the error in the initial/boundary conditions' enforcement. A solution can be obtained through updating the weights and biases of the neural network by minimizing the loss function using algorithms such as: gradient-descent, Adam, BFGS, etc. The loss function can be written as:

\[Loss=\lambda_{0}\ loss_{0}+\lambda_{D}\ loss_{D}+\lambda_{N}\ loss_{N}+ \lambda_{1}\ loss_{r} \tag{5}\]

where \(\lambda_{i}\) are the weights for each loss term which play an important role in the optimization process and:

\[loss_{0}=\frac{1}{N_{0}}\sum_{i=1}^{N_{0}}r_{0}^{2}(t_{0}^{i},\mathbf{x}_{0} ^{i})=\frac{1}{N_{0}}\sum_{i=1}^{N_{0}}||u(t_{0}^{i},\mathbf{x}_{0}^{i})-u_{0 }^{i}||^{2} \tag{6}\]

\[loss_{D}=\frac{1}{N_{D}}\sum_{i=1}^{N_{D}}r_{D}^{2}(t_{D}^{i},\mathbf{x}_{D} ^{i})=\frac{1}{N_{D}}\sum_{i=1}^{N_{D}}||u(t_{D}^{i},\mathbf{x}_{D}^{i})-u_{D }^{i}||^{2} \tag{7}\]

\[loss_{N}=\frac{1}{N_{N}}\sum_{i=1}^{N_{N}}r_{N}^{2}(t_{N}^{i},\mathbf{x}_{N}^{ i})=\frac{1}{N_{N}}\sum_{i=1}^{N_{N}}||B(u(t_{N}^{i},\mathbf{x}_{N}^{i}))-f_{N}^{i}|| ^{2} \tag{8}\]

\[loss_{r}=\frac{1}{N_{r}}\sum_{i=1}^{N_{r}}||r(t_{r}^{i},\mathbf{x}_{r}^{i})|| ^{2}=\frac{1}{N_{r}}\sum_{i=1}^{N_{r}}||u_{t}+\mu L(u)||_{(t_{r}^{i},\mathbf{ x}_{r}^{i})}^{2} \tag{9}\]

\(loss_{i}\) respectively are the losses representing the initial condition, Dirichlet, Neumann boundary conditions and the PDE residual.
### Sa-Pinn

The main objective of PINN is to find a solution that minimizes the residual of the PDE within the spatiotemporal domain that is represented by the collocation points while respecting the initial and boundary conditions. The result is a solution \(\hat{u}(t,\mathbf{x};\hat{\mu})\) to the PDE at a specific value \(\hat{\mu}\). To perform sensitivity analysis with \(\mu\) as the input parameter of interest, we would like to find not only the solution \(\hat{u}\) but also the derivative of the solution with respect to \(\mu\) (the sensitivity) at a given nominal value \(\hat{\mu}\).

One way to obtain the sensitivity in PINN is to build a parametric model. This is done by changing the structure of the neural network to accommodate for another input which is the parameter of interest \(\mu\). Then, one adds collocation points in the spatiotemporal-parametric space. Then, the residual is minimized in the whole parametric domain while respecting the initial and boundary conditions. Afterwards, the derivative of the solution with respect to \(\mu\) can be easily obtained through automatic differentiation. The main issue of building such parametric models is that the number of collocation points grows exponentially with the number of parameters of interest. The problem can, then, easily become computationally intractable if we have several parameters of interest which is common in most engineering applications.

To overcome this issue, we thought that instead of only minimizing the residual of the PDE, we can also minimize the derivative of the residual with respect to the parameter of interest. First, the structure of the neural network should accommodate for \(\mu\) as an input. Then, the loss function is formed as the sum of the residual and the derivative of the residual with respect to \(\mu\) along with the terms to respect the initial and boundary conditions. This way we make sure that the solution is accurate within a small neighborhood of \(\hat{\mu}\), thus sensitivity can be calculated. We call this technique SA-PINN. The technique can be summarized in points as follows:

* Choose the neural network to have inputs related to space, time and parameter of interest.
* Sample the collocation points only in space and time, however the points will be living in a higher dimension but without adding more points.
* Create the loss function having terms related to PDE residual, the residual derivative with respect to the parameter of interest and the terms related to the initial and boundary conditions.

The modified loss function will then be

\[Loss_{m}=Loss+\lambda_{0\mu}\ loss_{0\mu}+\lambda_{D\mu}\ loss_{D\mu}+\lambda_{N \mu}\ loss_{N\mu}+\lambda_{1\mu}\ loss_{r\mu} \tag{11}\]

where

\[loss_{0\mu}=\frac{1}{N_{0}}\sum_{i=1}^{N_{0}}\biggl{|}\frac{\partial r_{0}(t_ {0}^{i},\mathbf{x}_{0}^{i},\mu_{0}^{i})}{\partial\mu}\biggr{|} \tag{12}\]

\[loss_{D\mu}=\frac{1}{N_{D}}\sum_{i=1}^{N_{D}}\biggl{|}\frac{\partial r_{D}(t_ {D}^{i},\mathbf{x}_{D}^{i},\mu_{D}^{i})}{\partial\mu}\biggr{|} \tag{13}\]

\[loss_{N\mu}=\frac{1}{N_{N}}\sum_{i=1}^{N_{N}}\biggl{|}\frac{\partial r_{N}(t_ {N}^{i},\mathbf{x}_{N}^{i},\mu_{N}^{i})}{\partial\mu}\biggr{|} \tag{14}\]

\[loss_{r\mu}=\frac{1}{N_{r}}\sum_{i=1}^{N_{r}}\biggl{|}\frac{\partial r(t_{r}^{ i},\mathbf{x}_{r}^{i},\mu_{r}^{i})}{\partial\mu}\biggr{|} \tag{15}\]

Figure 1 shows a diagram that summarizes the methodology of SA-PINN. The parts in orange are the added parts from classical PINN. The \(u-\hat{u}\) term represents the mismatch of the solution from the initial and boundary conditions. It must be noted that we sample the collocation points only in space and time, but the points have another coordinate \(\mu\) and all have a nominal value \(\hat{\mu}\).
## 3 Model problems

In this section, we introduce the models of the three examples that are used to show the effectiveness of the technique.

### 1D diffusion-advection equation

The first example is a steady one-dimensional diffusion-advection equation where we would like to study the effect of perturbations in the diffusion term \(\epsilon\) on the solution. The strong form of the problem can be written as follows:

\[\begin{gathered}\epsilon\ y_{xx}-y_{x}+1=0,\quad x\in[0,1],\\ y(0)=1,\ \ y(1)=3\end{gathered} \tag{17}\]

The chosen nominal value for \(\epsilon\) is 0.1. \(y_{xx}\) and \(y_{x}\) are respectively the second and first order derivatives of the solution \(y\). The weights for the different terms in the loss function are set to 1 for the original PINN terms and 0.1 for the added sensitivity terms.

Figure 1: Diagram explaining the methodology of SA-PINN.
### 2D Poisson's problem

The next example is a 2-dimensional Poisson's problem where we have multiple parameters to study their effect on the solution. The domain is shown in figure 2 where there exists 9 subdomains each having different diffusivity value.

The strong form of the problem can be written as:

\[\begin{array}{c}k\ \Delta u=-1,\quad in\ \Omega,\\ u=0,\quad\ on\ \partial\Omega\end{array} \tag{18}\]

where \(\Omega\) is a square with unit sides and \(k\) is the diffusivity. The 9 subdomains have equal areas. The nominal values for the diffusivity is 1; \(k_{1}=k2=...=k_{9}=1\). The main PINN terms weights are set to 1 and 0.1 for the added sensitivity terms.

### 1D two-phase flow in porous media

In this section, we introduce a 1D two-phase flow in porous media problem. The problem is faced in Liquid Transfer Molding composite manufacturing processes, where resin is injected in a mold that has prepositioned fibrous matrix. The problem is shown in figure 3. At \(t=0\), the domain is initially saturated with one fluid (fluid 1). Another fluid (fluid 2) is being injected from the left end at constant pressure \(p_{in}\), while the pressure at the other end is fixed to \(p_{out}\)

Figure 2: 2D Poisson’s problem domain
The momentum equation can be approximated with Darcy's law that can be written in 1D as follow:

\[v=-\frac{k}{\phi\mu}p_{x} \tag{19}\]

where \(v\) is the volume average Darcy's velocity, \(\mu\) the viscosity, and \(p_{x}\) the pressure gradient, and \(\phi\) the porosity. Both fluids are assumed to be incompressible, therefore, the mass conservation equation reduces to

\[v_{x}=0 \tag{20}\]

Pressure boundary conditions can prescribed on the inlet and oulet:

\[p(\mathbf{x_{inlet}},t)=p_{in},\ \ p(\mathbf{x_{outlet}},t)=p_{out} \tag{21}\]

To track the interface between the two fluids, the Volume Of Fluids (VOF) technique is used; a fraction function \(c\) is introduced which takes a value 1 for the resin and 0 for the air. The viscosity \(\mu\) is redefined as

\[\mu=c\mu_{2}+(1-c)\mu_{1} \tag{22}\]

where \(\mu_{2}\) and \(\mu_{1}\) are the two fluids' viscosities. \(c\) evolves with time according to the following advection equation

\[c_{t}+vc_{x}=0 \tag{23}\]

Figure 3: One-dimensional domain (filling problem)
where \(c_{t}\) and \(c_{x}\) are the time and spatial derivative of the fraction function \(c\), respectively.

Initial and boundary conditions are defined to solve the advection of \(c\).

\[c(\mathbf{x},t=0)=c_{0}(\mathbf{x}),\ \ c(\mathbf{x_{inlet}},t)=1 \tag{24}\]

To sum up, the strong form of the problem can be written as:

\[\begin{array}{l}c_{t}+v\ c_{x}=0,\ \ \ x\in[0,l],\ \ t\in[0,T],\\ v=-\frac{k}{\phi\mu}p_{x},\ \ \ \ \ \ \ x\in[0,l],\ \ t\in[0,T],\\ \\ v_{x}=0,\ \ \ \ \ \ \ x\in[0,l],\ \ t\in[0,T],\\ \\
## 4 Results

### 1D diffusion-advection equation

The solution \(u\) using PINN and SA-PINN is shown in figure 4 along with the analytical solution for \(\epsilon=0.1\).

From figure 4, we can see that PINN and SA-PINN acturetly captures the analytical solution to the problem. The derivative of the solution with respect to \(\epsilon\) at to \(\epsilon=0.1\) is shown in figure 5.
The reference finite difference solution in figure 5 is obtained by obtaining different PINN solutions near \(\epsilon=0.1\) and then calculating the derivative. We can see that classical PINN fails to predict the derivative, while, SA-PINN accurately predicts the derivative due to the added regularization term in the loss function. The loss function for different values of \(\epsilon\) is plotted in figure 6 for PINN and SA-PINN.

Figure 5: \(\frac{\partial u}{\partial\epsilon}\) at \(\epsilon=0.1\) using PINN and SA-PINN along with the finite difference solution of the 1D advection-diffusion problem.
As seen in figure 6, SA-PINN has the effect of greatly flattening the loss curve in a neighborhood near the nominal value of \(\epsilon=0.1\). This leads to better solutions that PINN in the neighborhood and accurate derivative calculation at \(\epsilon=0.1\).

### 2D Poisson's problem

The PINN solution of the boundary value problem is shown in figure 7. The solution appears to be accurate and agrees with the analytical solution of the problem.

Figure 6: Loss function for different \(\epsilon\) values using PINN and SA-PINN of the 1D advection-diffusion problem.
The sensitivity terms \(\frac{\partial u}{\partial k_{i}}\) can then be plotted to see the effect of the diffusivity on the solution.

Figure 7: PINN solution of the 2D Poisson’s boundary value problem.
The computational time is plotted versus the number of parameters with respect to which sensitivity terms are added in figure 9.

Figure 8: Different derivatives of the solution with respect to \(k_{i}\) of the 2D Poisson’s problem.
It can be seen from the figure that the computational time grows linearly when increasing the number of parameters the sensitivity is calculated with respect to. This happens because the number of collocation points is the same when adding a new term to the loss function; the added cost is the same when adding new sensitivity terms.

### 1D transient two-phase flow in porous media

First, we plot the front location for three different values of \(k\) by taking the 0.5 level set of the fraction function \(c\) in figure 10. We compare SA-PINN with classical PINN along with the analytical solution.

Figure 9: Computational time vs. no of sensitivity parameters.
We can notice that SA-PINN provides good results for values of \(k\) away from the nominal value \(k=1\). Classical PINN accurately predicts the solution only at the nominal values, however, away from that values, random solutions were obtained which is clear from the two red lines.

In figure 11, we plot the time the flow front reaches \(x=0.5\) vs. \(k\). We compare the solution from SA-PINN with the analytical solution.

Figure 10: Flow front location vs. time for three different values of \(k\) (\(k=1\), \(0.5\) and \(2\)) of the transient two-phase flow in porous media problem.
We can see a good estimation of the filling time at different values of \(k\) using SA-PINN. This result can be useful in applications of injection processes to estimate the filling time as a function of a parameter of interest.

## 5 Conclusion

In the article, we presented a new method to perform sensitivity analysis based on the paradigm of PINN. The method is easy to implement using any of the machine learning libraries as TensorFlow or PyTorch. We show, through the examples, that the technique is easy to use when sensitivity with respect to multiple parameters of interest are studied at the same time. The computation time grows linearly as the parameters increase which is an advantage to the method. We also show through the last example that the method is working for a problem where a discontinuity exists (flow front) and VOF method is used.

Figure 11: Time at which the flow front reaches \(x=0.5\) vs. \(k\) for the transient two-phase flow in porous media problem.
## Acknowledgements

This study was funded under the PERFORM Thesis program of IRT Jules Verne.
