Towards Real-time Training of Physics-informed Neural Networks:
  Applications in Ultrafast Ultrasound Blood Flow Imaging
Physics-informed Neural Network (PINN) is one of the most preeminent solvers
of Navier-Stokes equations, which are widely used as the governing equation of
blood flow. However, current approaches, relying on full Navier-Stokes
equations, are impractical for ultrafast Doppler ultrasound, the
state-of-the-art technique for depiction of complex blood flow dynamics
\emph{in vivo} through acquired thousands of frames (or, timestamps) per
second. In this article, we first propose a novel training framework of PINN
for solving Navier-Stokes equations by discretizing Navier-Stokes equations
into steady state and sequentially solving steady-state Navier-Stokes equations
with transfer learning. The novel training framework is coined as SeqPINN. Upon
the success of SeqPINN, we adopt the idea of averaged constant stochastic
gradient descent (SGD) as initialization and propose a parallel training scheme
for all timestamps. To ensure an initialization that generalizes well, we
borrow the concept of Stochastic Weight Averaging Gaussian to perform
uncertainty estimation as an indicator of generalizability of the
initialization. This algorithm, named SP-PINN, further expedites training of
PINN while achieving comparable accuracy with SeqPINN. Finite-element
simulations and \emph{in vitro} phantoms of single-branch and trifurcate blood
vessels are used to evaluate the performance of SeqPINN and SP-PINN. Results
show that both SeqPINN and SP-PINN are manyfold faster than the original design
of PINN, while respectively achieving Root Mean Square Errors (RMSEs) of 1.01
cm/s and 1.26 cm/s on the straight vessel and 1.91 cm/s and 2.56 cm/s on the
trifurcate blood vessel when recovering blood flow velocities.
Haotian Guan, Jinping Dong, Wei-Ning Lee
Towards Real-time Training of Physics-informed Neural Networks: Applications in Ultrafast Ultrasound Blood Flow Imaging

###### Abstract

Physics-informed Neural Network (PINN) is one of the most preeminent solvers of Navier-Stokes equations, which are widely used as the governing equation of blood flow. However, current approaches, relying on full Navier-Stokes equations, are impractical for ultrafast Doppler ultrasound, the state-of-the-art technique for depiction of complex blood flow dynamics _in vivo_ through acquired thousands of frames (or, timestamps) per second. In this article, we first propose a novel training framework of PINN for solving Navier-Stokes equations by discretizing Navier-Stokes equations into steady state and sequentially solving steady-state Navier-Stokes equations with transfer learning. The novel training framework is coined as SeqPINN. Upon the success of SeqPINN, we adopt the idea of averaged constant stochastic gradient descent (SGD) as initialization and propose a parallel training scheme for all timestamps. To ensure an initialization that generalizes well, we borrow the concept of Stochastic Weight Averaging Gaussian to perform uncertainty estimation as an indicator of generalizability of the initialization. This algorithm, named SP-PINN, further expedites training of PINN while achieving comparable accuracy with SeqPINN. Finite-element simulations and _in vitro_ phantoms of single-branch and trifurcate blood vessels are used to evaluate the performance of SeqPINN and SP-PINN. Results show that both SeqPINN and SP-PINN are manyfold faster than the original design of PINN, while respectively achieving Root Mean Square Errors (RMSEs) of 1.01 cm/s and 1.26 cm/s on the straight vessel and 1.91 cm/s and 2.56 cm/s on the trifurcate blood vessel when recovering blood flow velocities.

 Blood flow, Data-driven scientific computing, Navier-Stokes equations, Physics-informed learning, Ultrafast Doppler Ultrasound.

## I **Introduction**

Depicting hemodynamics in the circulation system is important because it is closely related to the development of cardiovascular diseases, such as myocardial infarction and ischemic stroke due to atherosclerosis. Blood flow velocity and blood pressure are key hemodynamic parameters and directly outline vascular conditions. Ultrafast Doppler ultrasound permits unprecedentedly high full-view acquisition frame rates for high-velocity flow imaging based on the Doppler effect [4]. However, its accompanying low signal-to-noise ratio and Doppler angle dependence lead to velocity estimation bias and variance. Physics-constrained optimization has enabled regularization of color Doppler using planar mass conservation and free-slip boundary conditions [51].

As a governing principle, Navier-Stokes equations are partial differential equations (PDEs) widely used to describe the motion of an incompressible viscous fluid flow [49]. Computational Fluid Dynamics (CFD) modeling is the conventional method for solving Navier-Stokes equations [2]. Given proper boundary conditions, CFD can produce a velocity and a pressure field that obeys Navier-Stokes equations. However, Navier-Stokes equations are highly nonlinear, causing CFD to be computationally expensive. A high level mathematical understanding in the generation of meshing is also required for blood vessels of complex geometries [31]. These inherent difficulties refrain real-time applications of CFD models for personalized assessment of blood flow in clinical settings.

Artificial Intelligence (AI) for science empowered by deep learning has been an emerging research field. Recently, deep learning is popular given the fast advancement in computational resources, such as graphics processing unit (GPU) [1]. With sufficient data, deep learning has become the new paradigm for many scientific discoveries because of its flexibility and capability of extracting features automatically.

Since its emergence, physics-informed neural network (PINN) [40] has made a transformational impact on solving PDEs [47, 12, 48]. Recent research on PINN has demonstrated its excellence in coping with imperfect situations, such as no specified initial or boundary conditions and noisy data [5, 8, 13]. PINN is mesh-free, making it more flexible for arbitrary geometries and more computationally efficient. PINN has also been applied extensively in research works and real world physics systems. Cheng et al combined Resnet block with PINN to solve for fluid dynamics; they showed that Resnet block could improve stability of PINN [15]. Jin et al proposed NSFnets [16] specifically targeting at simulating solutions to Navier-Stokes equations; two alternative formulations of Navier-Stokes equations, namely velocity-pressure (VP) and vorticity-velocity (VV), were considered as part of the loss function. They conducted experiments on simulated laminar flow and turbulent channel flow and analyzed the performance and convergence of NSFnets under different neural network architecture designs and various combinations of weights in loss function.

Despite the success of PINN in many situations as an
alternative of CFD models, the training speed of PINN has not been proven to be faster than CFD [12]. The current PINN training framework exhibits potential to achieve fast training because of its lightweight design of the architecture, a low dimensional data input, and PDE-guided loss functions. These advantages enable fast updates of model parameters but cannot guarantee fast convergence of the model. In fact, the prevalent training framework has several drawbacks. First, training of PINN is associated with domain geometry and initial boundary conditions, so PINN models are usually hard to generalize to different geometries or fluid flow patterns [11]. Retraining PINN is thus imperative for a patient-specific model. However, re-training PINN from scratch is time-consuming due to the use of gradient-based optimization methods, a large number of collocation points, and complexity of PDEs. Gradient-based optimization methods are effective because they move model parameters towards local minima [10]. Nonetheless, owing to the lack of a closed-form solution, it is impossible to estimate the required number of epochs for a model to converge. To develop a well-trained PINN model, it usually takes thousands or even ten thousands of training epochs [16]. Second, Navier-Stokes equations are highly nonlinear and require computation of second derivatives of the fluid velocity field. The loss landscape of PINN can be difficult to optimize [9]. Therefore, fast training of PINN is indispensable but remains challenging.

Abundant efforts have been made to expedite the training of PINN and can be summarized into three categories. The first category aims to accelerate the convergence of PINN. Some studies [18, 20] focused on modifying the loss function. The standard loss function of PINN is comprised of three terms, which impose optimization challenge occasionally. One study [21] proposed to use numerical differentiation coupled with automatic differentiation to enhance the reliability of derivative computation and thus resulted in fast and accurate convergence of PINN. The second category relieves the optimization challenges of PINN by meta-learning to find better initialization for test data (or, tasks). Liu et al. [24] applied the classic Raptile framework from meta-learning to initialize the model parameters of PINN. Seo et al. [26] decomposed physics laws into a spatial derivative and a time derivative module. Then, they proposed a meta-learning approach to solve for the so-called reusable spatial derivative modules. However, the limitation of the meta-learning-based approaches is that the model trained during a meta-training phase must have learned the fundamental rule of finding a solution [22]. Designing appropriate meta-training tasks could be difficult and is more of an art work. Moreover, most methods mentioned previously were built upon the current PINN training framework, which is impractical to incorporate a large dataset as in ultrafast Doppler ultrasound. The third category takes advantages of parallel training by decomposing the computational domain. cPINN [6] offered space parallelization, while XPINN [8] provided both space and time parallelization. The parallelized computation greatly expedites the training of PINN; nonetheless, dividing domains could be tricky when complex geometry or pulsatile velocity is associated with a blood flow profile. A lack of a fast training framework is impeding efficient training of PINN.

In this work, we propose two novel training frameworks that expedite the training of PINN for Navier-Stokes equations and illustrate the effectiveness of the methods with applications in ultrafast Doppler ultrasound (1). Specifically, we first demonstrate that PINN is capable of solving Navier-Stokes equations under the steady-state assumption. Then, we initialize the model with the solution of steady-state Navier-Stokes equations and show that the initialized model together with transfer learning significantly reduces training time for subsequent frames. The number of collocation points used in steady-state Navier-Stokes equations during training is \(N\) times less than the original design of PINN, where \(N\) is the number of timestamps. This largely improves the training efficiency for one epoch. Overall, this work makes the following contributions.

1. We address the need of fast training of PINN. We propose to solve steady-state Navier-Stokes equations and discretize PINN along the time dimension by removing time variable t from input. This significantly alleviates the optimization challenge and computational burden of training. The framework, coined as SeqPINN, is illustrated in the flowchart in Fig. 2. The successful implementation of SeqPINN opens the gate for real-time training of PINN for Navier-Stokes equations.
2. Except the first timestamp, we apply transfer learning on subsequent ones. This promotes fast generalization to any timestamp provided that sparse sampled data are available.
3. With a further assumption of independent timestamps, we propose Sampled-Posterior PINN, coined as SP-PINN. SP-PINN achieves parallel training of PINN after a short initialization of SeqPINN. Fig. 1 illustrates the improvements in performance of SeqPINN and SP-PINN over Vanilla PINN.
4. SeqPINN and SP-PINN are of high practical value since they efficiently provide physics-regularized blood flow estimates from ultrafast Doppler ultrasound in clinical settings. The proposed algorithms show good applicability, as they can be implemented either on CPUs, which are widespread, or on GPUs, which enable even faster training of PINN.

## II **Related Work**

The current acceleration approaches can be divided into three groups--convergence-based approach, meta-learning approach, and domain decomposition approach.

### **Convergence-based Approaches**

The convergence-based approach aims to hasten the learning process by tackling imbalanced gradient during the training of PINN or inaccurate approximation of derivatives so that PINN converges faster. Wang et al. [17] derived the weights of loss terms based on eigenvalues of Neural Tangent Kernel (NTK). Shin et al. [19] proved the convergence theory for data-driven PINNs and derived the Lipschitz Regularized loss to solve linear second-order elliptic and parabolic type PDEs. Xiang [18] defined loss terms using Gaussian probabilistic
models and proposed a noise parameter to update the weights of loss terms. However, the self-adaptive loss function trades training efficiency per epoch with the number of training epochs. The weights of loss terms are calculated at the cost of more complex computation involving gradients. Although the number of total epochs decreases, the training time and complexity per epoch increase. Besides assigning weights to loss terms, some efforts accelerate the convergence of PINN by ensuring more accurate calculation of derivatives. Yu et al. [20] proposed to enhance the gradient of loss terms by enforcing the gradient of loss terms with respect to inputs to be zero, thus leading to more accurate estimation of derivatives. Instead of using Automatic Differentiation (AD), the derivatives can also be more robust by numerical differentiation (ND)-based methods. For example, Chiu et al. [21] proposed a coupled-automatic-numerical differentiation framework for calculating derivatives, improving the convergence speed and accuracy of PINN.

In summary, convergence-based methods aim to alleviate the optimization challenge during training of PINN by reshaping the loss landscape using designed weights for loss terms. Although convergence-based methods can accelerate the training of PINN, they do not simplify the training process. They fail to accelerate the training of PINN when the imbalanced loss is not the most dominant issue.

### **Meta-learning Approaches**

Meta-learning, known as learning to learn, targets at learning the most fundamental rules to learn and understand in a system. In the context of PINN, the goal of meta-learning is to enable better initialization of models for unseen tasks by learning from various tasks. Meta-learning can be classified into three categories: 1) model-based; 2) metric-based; 3) optimization-based. Recently, optimization-based algorithms, such as Model Agnostic Meta-Learning (MAML) [22] and Raptile [23], are promising because of their strong performance and ability to incorporate with any models trained through gradient descent. Liu et al. [24] proposed a new Raptile initialization method for PINN by modifying both the task sampling process and the penalty term of the loss. Another trending approach in meta-learning is to use reusable learning modules [31]. Seo et al. [26] demonstrated the decomposability of the continuity equation into spatial derivative and time derivative modules and adopted the idea of reusable modules. They generated synthetic data and used MAML to meta-initialize the spatial derivatives.

Compared with convergence-based methods, meta-learning based methods develop plug-and-play models that approach the optimal solution of a test case. A good meta-learning model has learned and understood the underlying mechanism of solving PDEs. However, purely meta-learning based approaches rely on the design of meta-training tasks. Test tasks may not benefit from meta-initialized models when the learning process of meta-testing tasks differs a lot from that of meta-training tasks [24].

Convergence-based acceleration and meta-learning acceleration are not optimal choices for PINN acceleration, albeit effective, because they rely on the original training framework of PINN.

### **Domain Decomposition Approaches**

The third approach aiming at expediting the training of PINN is the domain decomposition approach. cPINN [6] and XPINN [8], both of which are successful implementations of the domain decomposition approach, divide the computational domain into non-overlapping subdomains spatially or temporally. The communication between subdomains relies on the interface which is defined as the common boundaries between subdomains. Interface conditions are parts of the governing equations used to stitch subdomains. For different PDEs, interface conditions, such as solution continuity, flux continuity, etc., can be applied. The interface conditions promote propagation of information from one subdomain to its neighboring subdomains. They act as the most important governing equation when no training data are available in a subdomain [7]. In cPINN, the average solution continuity condition is applied to enable spatial decomposition of the computational domain. In XPINN, comparing with cPINN, the more general residual continuity conditions are imposed along with the average solution continuity condition, allowing space-time decomposition of any differential equations. During training, subdomains are initialized independently and then trained in parallel. However, it is not always straightforward when decomposing the computational domain. Unseemly subdomains lead to arduous optimization, and the final solution is available only after solving the most difficult subdomains.

The parallel training scheme supported by domain decomposition is most promising because it guarantees speed-up of PINN, while convergence-based and meta-learning approaches fail to expedite the training of PINN in the worst case scenario. Moreover, for a large dataset, such as ultrafast Doppler ultrasound, it is impractical to train a PINN without modifying the training framework, as in convergence-based and meta-learning approaches. In this work, we resort to decomposing the computational domain and completely renovate the training framework of PINN for Navier-Stokes equations by solving a spatial solution first and then adapting the model temporally.

Fig. 1: Model performance among Vanilla PINN, SeqPINN, and SP-PINN. Lower left corner represents the best model with lowest RMSE and shortest training time.
## III **Preliminaries**

### **Physics-informed Neural Network (PINN) with attention**

The idea of solving PDEs with neural networks can be traced back to 1990s [38]; however, its popularity was endowed by physics-informed deep learning in 2019 [40]. The vanilla PINN was first built to find solutions to nonlinear PDEs in both forward and inverse problems. PINN is outstanding due to its ability to fit experimental data while complying with any underlying laws of physics expressed by PDEs. PINN can be implemented to solve PDEs in a general form of:

\[\begin{split}\mathcal{F}(u(z);\gamma)&=f(z)\quad z \text{ in }\Omega,\\ \mathcal{B}(u(z))&=g(z)\quad z\text{ in }\partial\Omega, \end{split} \tag{1}\]

where \(\gamma\) are the physics-related parameters, \(z\) are the input coordinates (or collocation points), \(u\) represents the desired output, \(f\) and \(g\) are mapping functions, \(\mathcal{F}\) is the nonlinear differential operator, \(\mathcal{B}\) denotes the initial and boundary conditions, and \(\Omega\) is the defined domain. Since the inputs of PINN are spatio-temporal coordinates, PINN utilizes the property that neural networks are universal function approximators [28].

The structural design of PINN is usually very simple: few compositions of fully-connected layers followed by element-wise nonlinear functions to produce the output \(h_{k}\) after each layer as in eq. (2).

\[\mathbf{h}_{k}=\sigma\left(\mathbf{W}_{k-1}^{\top}\mathbf{h}_{k-1}+\mathbf{b} _{k-1}\right), \tag{2}\]

where \(\sigma\) is the activation function, and \(W_{k-1}\) and \(b_{k-1}\) are the weight and bias in the fully-connected layer, respectively. Partial derivatives are calculated using Automatic Differentiation [29], which uses exact expressions with floating-point values, thus avoiding approximation errors [30]. A more complex design of PINN architecture is to implement attention mechanism [50] on top of the original PINN. We find that attention mechanism stabilizes the training of PINN, so we adopt it in the architecture design. To make fair comparisons, vanilla PINN is introduced with attention mechanism. Since the acronym PINN could be misleading, we use _PINN_ to refer to the architecture design and _vanilla PINN_ to specifically refer to the original design of PINN. A flowchart of using steady-state Navier-Stokes equations to obtain full Navier-Stokes equations is demonstrated in Fig. 1.

### **PINN applications on blood flow**

Flow dynamics is mostly governed by Navier-Stokes equations as shown in eq. (3), where \(\mathbf{u}\) is the velocity vector of the fluid, \(p\) is the pressure, \(\rho\) is the density, \(\nu\) is the kinematic viscosity, and \(\nabla\) is the gradient differential operator.

\[\begin{split}\frac{\partial\mathbf{u}}{\partial t}+(\mathbf{u} \cdot\nabla)\mathbf{u}&=-\frac{1}{\rho}\nabla p+\nu\nabla^{2} \mathbf{u}\\ \nabla\cdot\mathbf{u}&=0\end{split} \tag{3}\]

The input to PINN are (x,y,t) collocation points and the outputs are blood velocity and blood pressure. Navier-Stokes equations, comprised of conservation of mass and conservation of momentum for Newtonian Fluids, are embedded as the residual loss denoted as \(\mathcal{L}_{\mathcal{F}}(\theta)\) in the loss function. Initial

Fig. 2: PINN architecture with Navier-Stokes equations as an example.
boundary conditions and sparse velocity samples are enforced by supervised learning using MSE (mean square error) losses, denoted by \(\mathcal{L}_{\mathcal{B}}(\theta)\) and \(\mathcal{L}_{\text{data}}\left(\theta\right)\), respectively. Using \(\theta\) to refer the parameters in neural network, the objective of PINN can be formed as finding the best \(\theta\) that minimizes the total loss:

\[\theta^{*}=\operatorname*{arg\,min}_{\theta}\left(\mathcal{L}_{\mathcal{F}}( \theta)+\mathcal{L}_{\mathcal{B}}(\theta)+\mathcal{L}_{\text{data}}\left( \theta\right)\right) \tag{4}\]

### **Uncertainty Estimation with SWA-Gaussian**

Stochastic Weight Average (SWA) has been shown to improve generalizability of deep learning models without increasing inference time. It uses a single model and subtly performs an average of weights traversed by Stochastic Gradient Descent (SGD) iterates [10]. To encourage exploration of regions where the weights correspond to high-performing networks, a SGD trajectory is obtained using a modified learning rate schedule. SGD often converges to a flat region where the gradient of the total loss is small, and then it tends to oscillate around the local minimum. Thus, averaging SGD iterates makes model parameters move towards the minima. The state-of-art performance of SWA has been shown on various supervised learning and semi-supervised learning tasks [32, 33, 34]. Stochastic Weight Averaging Gaussian (SWAG) [35], using the SWA solution, forms a Gaussian distribution to approximate the posterior distribution over neural network weights. The mean of Gaussian is the SWA solution built using SGD iterates as shown in eq. (5), and the variance also originates from SGD iterates as shown in eq. (6).

\[\bar{\theta}=\frac{1}{T}\sum_{i=1}^{T}\theta_{i} \tag{5}\]

\[\Sigma=\frac{1}{T-1}\sum_{i=1}^{T}\left(\theta_{i}-\bar{\theta}_{i}\right) \left(\theta_{i}-\bar{\theta}_{i}\right)^{\top}=\frac{1}{T-1}DD^{\top} \tag{6}\]

where \(T\) is the total number of SGD iterates and \(\top\) is the matrix transpose operation. Maddox et al. [35] then samples from this Gaussian distribution and performs Bayesian model averaging. SWAG has been widely used when estimating uncertainties in deep learning [37, 39].

## IV **Methods**

### **PINN for Steady-state Navier-Stokes Equations**

The traditional PINN architecture for fluid flow embeds Navier-Stokes equations into the loss function, and solve for flow velocity and pressure fields with sparse samples in the forward problem [12]. In this work, we assume that the blood flow starts at a steady state, and each timestamp follows the steady state with an infinitesimal change in flow velocity and pressure. Theoretically, a steady state is achieved when (i) all data of Navier-Stokes equations are independent of time, and (ii) the Reynolds number is sufficiently small [41]. Time independence can be equivalent to constant flow and constant pressure. To fulfill this requirement, we simulate infinitesimal time steps, making the change in velocity and pressure across time approximately zero. A small Reynolds number can characterize a fully developed flow when flow velocity is small. Mathematically, the loss function of PINN for non-dimensionalized steady-state Navier-Stokes equations can be written as

\[(\mathbf{u}^{\prime}\cdot\nabla)\mathbf{u}^{\prime} =-\nabla p+\frac{1}{\mathrm{Re}}\nabla^{2}\mathbf{u}^{\prime} \tag{7}\] \[\nabla\cdot\mathbf{u}^{\prime} =0 \tag{8}\]

\(\mathbf{u}^{\prime}\) is the non-dimensionalized velocity vector by \(\mathbf{u}^{\prime}=\frac{u}{U}\), where \(U\) is the characteristic velocity. The utilization of steady-state Navier-Stokes equations can be regarded as decomposing the computational domain of Navier-Stokes equations where the spatial and temporal modules are treated separately. The spatial module is shareable across timestamps since the inputs to the model are collocation points and thus identical for all timestamps. For consecutive timestamps, the model is adapted by SGD to a new local minimum led by the infinitesimal change in velocity measurements.

### **SeqPINN**

We first present an algorithm Sequential PINN, coined as SeqPINN, for learning solutions to steady-state Navier-Stokes equations and fast generalization to subsequent timestamps. SeqPINN is initialized by finding a solution for steady-state Navier-Stoke equations. The initial frame is chosen by selecting the timestamp associated with the lowest blood flow velocity in a cardiac cycle based on the temporal profile of the blood flow velocity. This prevents violation of steady-state Navier-stokes equations. In our experiments, the blood flow velocity profile is available as a priori knowledge. When such profile is unknown, starting from an arbitrary timestamp is also feasible. After initialization, the current PINN solution is deemed as a pre-trained model for the next timestamp. Given the boundary conditions and data points sampled at next timestamp, SGD can adapt to a new local minimum with \(m^{*}\) epochs. Section V will show that a pre-defined parameter \(m^{*}\) that optimizes the trade-off between accuracy and training speed exists. In practice, \(m^{*}\) can be determined by Earlystopping. By removing the time dimension from input, we keep the spatial collocation points the same across all timestamps. This significantly reduces computational burden of SGD. The training process of SeqPINN is summarized in Algorithm 1.

### **Data Assimilation**

SeqPINN is essentially a time series framework where we update the model from the previous timestamp and assimilate data to make predictions of flow velocity and pressure at the current timestamp. The sampled locations in the flow velocity field are fixed over time, while the model is being updated by SGD over time. Therefore, the algorithm SeqPINN can be regarded as a data assimilation process. Data assimilation at each timestamp not only promotes the model to find a new local minimum quickly, but also assists the model prediction at future timestamps by updating the model parameters. However, the impact of current sampled measurements decays as the model moves towards the end of the timeline. Here, we only present the most basic version of SeqPINN. If needed,
the weight of the data at particular timestamp can be tuned by modifying the number of epochs trained at a given timestamp. We denote \(s_{i}\) as the number of epochs trained using sampled measurements at current timestamp \(i\) with \(j\) iterates through all timestamps. The weight of each timestamp is calculated as

\[\sigma_{i}=\frac{e^{s_{i}}}{\sum_{j=1}^{t}e^{s_{j}}} \tag{9}\]

### **Computational Complexity**

We demonstrate that SeqPINN is more computationally efficient by reducing the input dimension. Instead of training PINN to solve continuous-time Navier-Stokes equations, solving steady-state Navier-Stokes equations only requires (x,y) pairs as the input at a single timestamp. Assuming a case containing \(n\) pairs of (x,y) and \(N\) timestamps in total, the number of training points in an epoch for original PINN is \(n\cdot N\), while that for SeqPINN is \(n\). Thus, training one epoch of SeqPINN is always \(n\) times more computationally efficient than PINN. Overall, training SeqPINN for 30 epochs over \(N\) timestamps requires running \((30\cdot n)\) data through the neural network, which is equivalent to training PINN, which requires \(30\cdot(n\cdot N)\) points for 30 epochs. Moreover, the size of the model is also smaller when the input dimension is reduced from 3 to 2. This also explains what makes SeqPINN more efficient. SeqPINN also benefits from the use of ultrafast Doppler ultrasound, which operates at thousands of frames per second. Vanilla PINN, feeding collocation points for all timestamps, may not be trained when the number of timestamps is high due to the limitation of GPU memory (VRAM), while SeqPINN can always be trained since it only takes in collocation points for a single timestamp.

### **Sampled-Posterior PINN**

SeqPINN significantly expedites the convergence of PINN while maintaining satisfactory accuracy. However, the sequential training scheme impedes real-time training of PINN in ultrafast Doppler ultrasound since the model at any timestamp is dependent of the model updates at the previous timestamp. A parallel training scheme fits the goal of real-time training to the utmost. In this section, we demonstrate how we adopt the idea of SWAG to ensure trustworthy initialization and achieve parallel training across timestamps based on SeqPINN initialization.

At the initialization stage, instead of promoting a set of PINN parameters optimized for a single timestamp, an initial set of PINN parameters that generalizes to all timestamps fast and accurately is imperative for the parallel training scheme. Similar to what we do in SeqPINN, we initialize the PINN model under the assumption of steady-state Navier-Stokes equations. Then, we view the initialization in SeqPINN \(p(\theta)\) as a prior belief and sparse data at the next \(k\) timestamps \(p(\mathcal{D}\mid\theta)\) as new observations in a set which can be used to update the prior belief. As a result, a training dataset \(\mathcal{D}\) comprised of \(k\) timestamps is formed, and it enables Bayesian approaches to boost network accuracy and achieve better generalizability [52, 53, 32]. The posterior belief \(p(\theta\mid\mathcal{D})\) can be derived from Bayes' rule.

\[p(\theta\mid\mathcal{D})=\frac{p(\mathcal{D}\mid\theta)p(\theta)}{p(\mathcal{ D})} \tag{10}\]

where \(p(\mathcal{D})\) is obtained by marginalizing \(\theta\) from \(p(\mathcal{D},\theta)\), which is the joint probability function of \(\mathcal{D}\) and \(\theta\):

\[p(\mathcal{D})=\int p(\mathcal{D},\theta)\,d\theta \tag{11}\]

However, due to infinite possibilities in neural network training, the marginalization is not tractable. Alternatively, deep ensembles have been shown to be an efficient way for approximating Bayesian posterior distribution [52, 54]. As proved in [53], constant SGD can be used to simulate Markov chain with a stationary distribution as an approximation for Bayesian posterior inference [53]. Thus, by sequentially training the next \(k\) timestamps for \(m^{*}\) epochs with constant SGD, we obtain \(k\) samples from the posterior distribution. Given the temporal causality in physical systems [55], \(m^{*}\) epochs are trained for a single timestamp (data) before the next timestamp is trained, instead of cyclically training \(k\) timestamps for \(m^{*}\) epochs. Finally, by averaging samples from the posterior distribution as in eq. (12), we derive a set of PINN parameters that achieves better generalizability for all timestamps.

\[\theta^{*}=\frac{1}{k}\sum_{i=1}^{k}\theta_{i} \tag{12}\]

One concern is that the initialization frame may find a sharp local minimum, instead of a flat and wide local minimum, on the loss surface; this may cause the solutions of the next
timestamps to have a large variance. The averaged solution may fall into the territory between two local minima and have poor generalizability. To address this, we propose an uncertainty estimation algorithm inspired by SWAG (see Section III-C) to avoid sharp local minima. A Gaussian distribution is formed to approximate the posterior distribution of SeqPINN model parameters. The mean is the averaged SGD solutions, and the variance of the Gaussian distribution is approximated by the solution of \(k\) timestamps. After approximating the posterior distribution of PINN model parameters, uncertainty estimation is performed by Bayesian model averaging. The uncertainty is represented by a standard deviation map (std map) visually, and the uncertainty index is defined as the mean value in a std map. Uncertainty is monitored during training. For all timestamps, we initialize the PINN model with the averaged solution from constant SGD. This method is thus coined as Sampled-Posterior PINN (SP-PINN).

```
1for\(i\gets 1\)to\(N\)do
2if\(i==1\)then
3 initialize the first frame for\(j\gets 1\)to\(n\)do
4\(\theta\)*\(\leftarrow\theta-\eta\cdot\frac{\partial I}{\partial\theta}\)
5 end for
6
7elseif\(i<=k\)then
8 initialize the model with PINN trained from last timestamp for\(j\gets 1\)to\(m^{*}\)do
9\(\theta\),*\(\leftarrow\theta_{j}-\eta\cdot\frac{\partial I}{\partial\theta_{j}}\) \(\widetilde{\theta}\leftarrow\frac{n\theta+\theta^{\prime}}{n+1},\Sigma\leftarrow \frac{1}{n-1}(\theta_{j}-\overline{\theta}_{j})(\theta_{j}-\overline{\theta}_ {j})\)
10 end for
11draw samples from \(\widetilde{\theta}_{j}\sim\mathcal{N}\left(\overline{\theta},\Sigma\right)\) and perform uncertainty estimation
12
13else
14 initialize the model with \(\overline{\theta}\) for\(j\gets 1\)to\(m^{*}\)do
15\(\theta\)*\(\leftarrow\theta-\eta\cdot\frac{\partial I}{\partial\theta}\)
16 end for
17
18 end for
19
20 end for
```

**Algorithm 2** Sampled Posterior Physics-informed Neural Networks.

## V **Experiments**

### **Experimental Setup**

#### V-A1 **Fluid-structure Interaction (FSI) Simulation Data**

We first evaluated our proposed methods on computer simulations of blood flow to demonstrate the feasibility of SeqPINN. It is important to show that SeqPINN can solve Navier-Stokes equations under a pulsatile fluid flow as shown in Fig. 2(c) given its relevance in biomedical applications. Fluid-structure interaction simulations, which better describe blood dynamics in a compliant blood vessel than CFD, were conducted [46] using COMSOL 5.5 software (Comsol Inc. Burlington, MA, USA) and adopted as the ground truth. We designed a single-branch (Fig. 2(a)) and a three-branch (Fig. 2(b)) blood vessel to mimic the common segment of a major artery and branched structure, respectively. In designing COMSOL simulations, surrounding medium such as deformable solid structures and their mechanical properties are first defined to fix the vessel geometry. For the single-branch vessel, the fluid properties were set close to real human blood, whose density was 1060 \(kg/m^{3}\) and dynamic viscosity was 5 \(mPa*s\). The Reynolds number of the blood flow was \(R_{e}=\frac{\rho V_{avg}D}{\mu}=\frac{1060kg/m^{3}*82.8cm/s*5mm}{5mPaS}=877.7\), where \(\rho\) is the fluid density, \(V_{avg}\) is the average fluid velocity, and \(D\) is the entrance diameter. The inlet flow velocity and outlet pressure used in the FSI simulation of the single-branch case are shown in Fig. 2(c) and Fig. 2(d). A 200 \(\times\) 30 grid was generated during COMSOL simulation and 3718 collocation points were used when calculating the equation loss after removing the surrounding medium of the Comsol simulated vessel.

For the three-branch vessel [46], the fluid density and dynamic viscosity were set to 1037 \(kg/m^{3}\) and 4.1 \(mPa*s\), respectively, to match the properties of blood mimicking fluids used in the _in vitro_ phantom experiment. The inlet flow velocity and outlet pressure used in the FSI simulation of the three-branch case are shown in Fig. 2(c) and Fig. 2(d). This scenario aimed to investigate PINN's capability of solving blood flow velocity in a branched structure under the supervision of steady-state Navier-Stokes equations. The Reynolds number of the blood flow was \(R_{e}=\frac{\rho V_{avg}D}{\mu}=\frac{1037kg/m^{3}*18.7cm/s*10mm}{4.1mPa*s}=473\). The simulation was generated automatically using Comsol with previously specified parameters. A larger grid with 500 \(\times\) 300 collocation points was generated due to the complexity of geometry, and 24359 collocation points were used when calculating the equation loss after removing the surrounding medium of the simulated trifurcate vessel. Blood flow was assumed to be an incompressible Newtonian fluid. More details of the fluid-structure simulation design can be read in [45, 46].

#### V-A2 **Evaluation Metrics**

Root Mean Square Error (RMSE) across time is the most common way to evaluate the performance of models, and it is defined as follows:

\[RMSE=\sqrt{\frac{1}{N}\sum_{i=1}^{N}(u_{i}-u_{i})^{2}} \tag{13}\]

where \(u\) is the velocity, and \(i\) represents time index, and \(N\) is the total number of timestamps. Training efficiency is another important metric as the ultimate study goal is to achieve real-time training of PINN. We measure the training efficiency by training time and the ratio between training time and improvement in RMSE. Huge sacrifice in training time for a small increase in RMSE is not desired.
#### Iv-A3 **Training Process**

SeqPINN and SP-PINN were both trained on Nvidia GeForce RTX 3090. The training of SeqPINN and SP-PINN contained two stages: initialization and adaption. During initialization of SeqPINN, the model was trained with 3000 epochs using Adam optimizer. The learning rate was set to 1e-3 for 2000 epochs and 5e-4 for 1000 epochs. The batch size was 1024 in the single-branch vessel and 2048 in the three-branch vessel. The neural network consisted of 8 fully connected layers with 150 neurons in each layer. During adaption of SeqPINN, each timestamp was initialized with a pre-trained model from the previous timestamp and then trained for 30 epochs. The learning rate was set to 5e-4, following the end of training in the initialization stage. Batch size was kept at 1024 for the single-branch case and 2048 for the three-branch case. An identical initialization stage was performed for SP-PINN. Then, all timestamps were initialized with an averaged SGD solution during the adaption stage and thus trained in parallel. In our experiments, since the batch sizes (1024 and 2048) were small, full power of a single GPU could not be employed when a single model was trained. Thus, we utilized Multi-Process Service (MPS) provided by Nvidia to maximize the efficiency of GPU.

#### Iv-A4 **Phantom Experiment**

Phantom data from Dong et al. [46] were used to validate the feasibility of SeqPINN and SP-PINN. Ultrafast Doppler ultrasound data are known to be not noise-free, and accuracy and precision of Doppler estimates for blood flow velocities depend on not only Doppler angles but also ultrafast ultrasound image quality, including the signal-to-noise ratio, contrast, and spatial resolution. Whether PINN could handle ultrafast Doppler ultrasound data was unknown. Thus, the feasibility of using ultrafast Doppler ultrasound data as the input was substantiated using data from phantom experiments.

### **Results**

Tables table I and table II compared model accuracy and efficiency among vanilla PINN, SeqPINN, and SP-PINN in the single-branch and three-branch cases, respectively. Due to the extremely large number of collocation points, it is impractical for vanilla PINN to cover the entire cardiac cycle with a high frame rate of 1000 frames per second, so the comparison was only performed on 100 timestamps. SeqPINN was demonstrated to be much more accurate and efficient than vanilla PINN. Specifically, SeqPINN, achieving RMSE of 1.01 cm/s in the single-branch vessel, outperformed vanilla PINN by 45% in the RMSE value. The training time consumed by SeqPINN was 7.5 times shorter than vanilla PINN. SP-PINN achieved RMSE of 1.26 cm/s, surpassing vanilla PINN by 31%. The training speed of SP-PINN was 11.4 times faster than vanilla PINN. For the three-branch vessel, the RMSE of SeqPINN was 1.91 cm/s, which was 29% lower than that of vanilla PINN. The training speed of SeqPINN was 6.1 times faster than that of vanilla PINN. The performance of SP-PINN was 4% more accurate than vanilla PINN, while the training time with SP-PINN was 10.2 times faster than vanilla PINN. Overall, we suggest SeqPINN in biomedical applications because accuracy outweighs the training speed. In other fields of research where the training speed is crucial, SP-PINN is a preferred option. Interested readers may refer to supplementary videos for fully-recovered blood flow velocity maps in the entire computational domain by SeqPINN and SP-PINN.
of diastole (from 600 ms to 700 ms). The novel training framework of PINN under the assumption of steady-state Navier-Stokes equations was demonstrated to outperform vanilla PINN. It is worth mentioning that the errors of all three models across time had the same trend as the model velocity profile.

### _Discussion_

The number of training epochs needed for subsequent timestamps after initialization, denoted as \(m\), is an important parameter to be tuned in SeqPINN to balance the trade-off between accuracy and efficiency. Increasing \(m\) by one for every timestamp results in \(t\) more training epochs in total. Therefore, we analyzed the trade-off between accuracy (RMSE) and \(m\) empirically and showed that the optimal number of training epochs \(m^{*}\) existed. Fig. 6 illustrates the relationship among accuracy (RMSE), the number of training epochs, and training time on the simulated single-branch (left) and three-branch (right) vessels. On one hand, we noticed that RMSE decreased drastically and almost linearly as we increased the number of training epochs for each timestamp until 30 epochs per timestamp, and then the decrease in RMSE slowed down as we trained each timestamp for more epochs. On the other hand, training time grew linearly as we increased the number of training epochs for each timestamp. A diminishing return on reduction in RMSE was observed for every 10 more training epochs when \(m\) was greater than 30. An effective strategy to overcome the diminishing return is through training SeqPINN cyclically, instead of increasing \(m^{*}\) in a single training cycle. Fig. 7 indicates that training SeqPINN for two cycles with \(m\) equal to 30 in each cycle significantly improved model accuracy from training SeqPINN for one cycle with \(m\) equal to 60. Note that two training strategies have the same computational complexity. The superior performance of training SeqPINN cyclically suggested that models learned across timestamps helped PINN find a global model that alleviated the optimization challenges for all timestamps.

We proposed SeqPINN under the assumption of infinitesimal time-step where ultrafast Doppler ultrasound was a suitable biomedical technology example. In fact, SeqPINN was robust against a large time-step. Fig. 8 shows that RMSE and standard deviation were stable even if we increased the time-step to 50 ms. SeqPINN exhibited a strong transfer learning ability, with slightly degraded performance in accuracy as the step size increased. The number of training epochs \(m^{*}\) was kept at 30. The success of SeqPINN was based on a reasonable initialization for the current timestamp. Clearly, the initialization timestamp could not only be seen as a pre-trained model for the next timestamp, but also a pre-trained model for all timestamps, hereby building a foundation for SP-PINN.

SP-PINN is built upon the implementation of SeqPINN and an Bayesian approach. A well trained solution for the steady-state Navier-Stokes equation followed by a constant SGD training scheme produced a stable posterior Gaussian distribution of PINN, and the mean of the distribution shall have good generalizability across all timestamps. In SP-PINN, generalizability was assessed by an uncertainty map produced by sampling from the posterior distribution and performing Bayesian model averaging. Fig. 9 and Fig. 10 illustrate uncertainty estimation in single-branch and three

Fig. 4: Comparison of lateral (a) and axial (c) velocity maps by conventional ultrafast Doppler ultrasound and lateral (b) and axial (d) velocity maps by PINN-regularized ultrafast Doppler ultrasound.

Fig. 5: Comparison of RMSEs among Vanilla PINN, SeqPINN, and SP-PINN across 100 timestamps in the simulated single-branch (top) and three-branch (below) vessels.
branch simulation cases. Note that in the single-branch vessel case, good initialization and bad initialization were similar visually. It was difficult to distinguish a bad initialization from a good initialization. Therefore, an uncertainty estimation was indispensable. In our experiments, the uncertainty index was monitored once for every 500 training epochs. The size of the training dataset \(k\) leveraged the training speed and training stability. When dealing with a pulsatile velocity field using ultrafast Doppler ultrasound, \(k\) is much smaller than the total number of timestamps \(N\). The lack of the global insight biases any efforts towards an initialization that generalizes well to all timestamps. The Bayesian approach mitigates the bias with an informative prior belief and Bayesian marginalization. Practically, \(k\) was fine-tuned in choices of 5, 10, 15, 20. We found that using the average of 15 subsequent timestamps led to the best accuracy. The idea of averaging constant SGD solutions of consecutive timestamps is a cheap and well-justified way to approximate a Bayesian posterior distribution while ensuring accuracy for subsequent timestamps. Since we used the same initialization for all timestamps, SP-PINN was not affected by the step size in time.

Comparing with SeqPINN, Vanilla PINN was slower to train because the extra dimension in time brings \(N\) times more copies of the collocation points, with \(N\) as the number of timestamps. In contrast, SeqPINN initializes and trains a unique model one at a time for all timestamps, needing

Fig. 8: Comparison of accuracy and time-steps in the simulated single-branch (left) and three-branch (right) vessels. Mean and standard deviation were calculated using RMSEs at each timestamp. Time-steps of 5, 10, 20, 30, 40, and 50 ms were tested.

Fig. 6: Comparison of SeqPINN training time and accuracy as the number of training epochs per frame \(m\) increases in the simulated single-branch (a) and three-branch (b) vessels. The accuracy and training time recorded were for the entire computational domain containing 608 timestamps.

Fig. 7: Comparison of training SeqPINN for one cycle with \(m\)*=30 and \(m\)*=60 and two cycles with \(m\)*=30 in the simulated single-branch (a) and three-branch (b) vessels.

Fig. 9: Uncertainty estimation in the single-branch vessel. (a) a sample from good estimation of the posterior distribution. (b) a sample from bad estimation of the posterior distribution. (c) standard deviation (std) of 30 samples from the approximation of the posterior distribution corresponding to (a). Low std indicates that the initialization is stable. (d) standard deviation (std) of 30 samples from the approximation of the posterior distribution corresponding to (b). High std indicates that the initialization is not stable.
times more space for storage of PINN models. In other words, SeqPINN trades the number of models (storage space) for training speed. Fortunately, individual SeqPINN models were very small in size (about 461 kB), and it was not necessary to store them on GPU. The trade-off between storage space and training speed is of high value in efficient training of PINN.

For the comparison between SeqPINN and SP-PINNN, training speed was no longer a bottleneck; thus, a comparison on the entire velocity profile was done (Fig. 11 and Fig. 12). For the single-branch vessel, SeqPINN and SP-PINN outputted very similar results, while SP-PINN was more oscillating. For the three-branch vessel, SP-PINN had lower accuracy than SeqPINN for almost all timestamps. We speculated that lower model accuracy of SeqPINN for the three-branch vessel led to poor performance of SP-PINN. Due to the implementation of steady-state Navier-Stokes equations, a small batch size cannot fully utilize computing power of GPU. Therefore, SP-PINN will always be faster than SeqPINN using MPS by Nvidia, even under the circumstance when computing resources do not support parallel training due to limited GPUs and the large number of timestamps as in Doppler ultrasound.
## VI **Conclusion**

In this paper, we developed SeqPINN with steady-state Navier-Stokes equations to facilitate the training of PINN, and SP-PINN to conduct parallel training of SeqPINN toward real-time generalization. Fast training of PINN is imperative in many areas, such as ultrafast Doppler ultrasound, which depicts complex blood flow dynamics at thousands of frames per second. We view SeqPINN as the foundation towards real-time training of physics-informed learning for solving Navier-Stokes equations. The novel training framework of SeqPINN significantly reduces the training time, while achieving superior performance compared with current implementation of PINN. SeqPINN is a generic algorithm that can incorporate various techniques mentioned in related work. For example, the design of weighted loss and calculation of derivatives can potentially be built on top of SeqPINN. SP-PINN with uncertainty estimation is a reliable way to initialize a generalizable model that can train SeqPINN in parallel. Meta-learning based approaches can also be adopted to search for a good initialization along the time dimension. The success of SeqPINN is built on the foundation of stead-state Navier-Stokes equations. This paper is envisioned to stimulate more future research on stead-state PDEs, particularly in biomedical applications.
