Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

Unifying Predictions of Deterministic and Stochastic Physics in Mesh-reduced Space with Sequential Flow Generative Model

Luning Sun

Lawrence Livermore National Lab

Livermore, CA 94550

sun42@llnl.gov

Equal contribution.

Xu Han

Tufts University

Medford, MA 02155

xu.han@tufts.edu

Han Gao

Harvard University

Cambridge, MA 02138

hgao1@seas.harvard.edu

&Jian-Xun Wang

University of Notre Dame

Notre Dame, IN 46556

jwang33@nd.edu

&Li-Ping Liu

Tufts University

Medford, MA 02155

liping.liu@tufts.edu

###### Abstract

Accurate prediction of dynamical systems in unstructured meshes has recently shown successes in scientific simulations. Many dynamical systems have a nonnegligible level of stochasticity introduced by various factors (e.g. chaoticity), so there is a need for a unified framework that captures both deterministic and stochastic components in the rollouts of these systems. Inspired by regeneration learning, we propose a new model that combines generative and sequential networks to model dynamical systems. Specifically, we use an autoencoder to learn compact representations of full-space physical variables in a low-dimensional space. We then integrate a transformer with a conditional normalizing flow model to model the temporal sequence of latent representations. We evaluate the new model in both deterministic and stochastic systems. The model outperforms several competitive baseline models and makes more accurate predictions of deterministic systems. Its own prediction error is also reflected in its uncertainty estimations. When predicting stochastic systems, the proposed model generates high-quality rollout samples. The mean and variance of these samples well match the statistics of samples computed from expensive numerical simulations.

## 1 Introduction

Accurate prediction of long-term dynamics of physical systems is of great interest in many science and engineering fields. The classical simulation of a system heavily relies on a spatial/temporal discretization of the space and the numerical solution to a finite-dimensional algebraic system derived from the governing equation [1]. However, due to the multi-scale, stochastic nature of the complex physics and the complexity of the geometry, the simulation of large-scale real-time applications is extremely expensive in computation [2]. In recent years, deep learning models have been applied to predict rollouts of large and complex physical systems thanks to their flexibility and scalability [3]. Moreover, they can handle uncertainties and non-linearity in physical problems [4, 5, 6, 7, 8] more effectively than traditional methods.

Previous work has studied two types of complex dynamical systems: deterministic ones and stochastic ones. A deterministic system is often considered under a perfectly controlled experiment with exactly known PDE terms and initial conditions (IC) or boundary conditions (BC) [9; 10; 11; 12; 13; 14]. This type of system can be modeled by autoregressive predicting models. A stochastic system such as quantum mechanics [15] and statistical physics [16] has a stochastic rollouts. When there are stochastic forcing terms or IC/BC terms in the governing equations, researchers have developed various models for predicting the stochastic state variables, such as turbulence velocity and stock prices [17; 18; 19; 20; 21]. Given that there is not a clear boundary between the two types of systems, it is highly desirable that a unified model can model either a deterministic or a stochastic system automatically. However, such models so far are limited to classical numerical models such as OpenFOAM [22]. There is an urgent need to develop a unified deep-learning model that can model both types of systems.

In this work, we propose a unified framework based on deep generative models to predict deterministic and stochastic systems. The new model is based on graph-structured state representations [12], which can handle irregular spatial areas commonly seen in dynamical systems. It uses an autoencoder to encode a state representation to a low-dimensional latent vector. Accurate encoding and decoding is critical for recovering the state of the dynamical system. This work makes several innovations to enhance the autoencoder's ability to preserve the information in system states. We provide a new approach to encoding the graph-structured representation with a vector with a fixed length. We also get some inspiration from regeneration learning [23] and train our autoencoder with self-supervised learning.

To describe stochasticity in the system, we use a sequential probabilistic model for the latent representations. We integrate a transformer and a normalizing flow to construct a step-wise predictive neural network in the latent space. When there is stochasticity, the model will learn the conditional distribution of the next latent state; when there is no stochasticity, it can place probabilities to correct deterministic predictions and still can minimize the predictive error. By including the ability to simulate both deterministic and stochastic systems in a uniform framework, it reduces the effort of developing separate models for different problems.

We evaluate our proposed framework in an extensive empirical study. The results indicate the proposed model outperforms the SOTA baselines on deterministic datasets regarding accuracy. More importantly, for the first time, we introduce several alternative evaluation metrics other than normalized RMSE for stochastic fluid dynamics, which help improve comparisons between different methods in this domain. The proposed framework can produce high-quality samples for stochastic systems in the mesh space.

## 2 Background

### Problem definition

Let's consider a general partial differential equation (PDE) defined on the \(d\)-dimensional space and one-dimensional time domain,

\[\frac{\partial\mathbf{u}}{\partial t}=j(\mathbf{u},\mathbf{\mu},\mathbf{\iota})\quad\text{in }\Omega\times[0,T_{\rm end}] \tag{1}\]

where \(\Omega\subset\mathbb{R}^{d}\) is the spatial domain, \(T_{\rm end}\) is the endpoint of time, \(\mathbf{\iota}:\partial\Omega\times[0,T]\) is the random parameter over time for stochastic systems (e.g., boundary conditions), and \(\mathbf{u}:\Omega\times[0,T]\) is the primary solution variable (e.g., velocity and pressure of fluid flow). Here \(\mathbf{\mu}:\mathcal{D}\) is the global physical system parameter and is time-invariant (e.g., Re number) for a given rollout. \(j:\Omega\times[0,T]\times\mathcal{D}\rightarrow\Omega\times[0,T]\) is an aggregation of the spatial terms of conservation laws (e.g., source and flux). To numerically solve the conservation law in Equation 1, let \(\mathcal{C}_{h}\) be a mesh of \(\Omega\), that is, \(\mathcal{C}_{h}=\{C_{i}\in\Omega:i=1,\ldots,N\}\) is a collection of non-overlapping cells that cover \(\Omega\). We further use \(\mathbf{u}_{i,t}\) to denote the evaluation of \(\mathbf{u}\) at the cell center of \(C_{i}\) at time step \(t\).

Using the mesh described above, we apply a finite volume discretization to yield the parametrized, nonlinear dynamical system. Here, the dynamical system can be computationally intensive because a fine-level mesh will lead to discretization with a large degree of freedom. For stability issues, the numerical time step also needs to be very small. Therefore, numerical simulations with traditional methods require extremely expensive computation and storage.

In this paper, we are interested in two different scenarios: _deterministic_ dynamics with invariant parameter \(\mathbf{\iota}\) over time (e.g., fixed boundary condition ), and _stochastic_ dynamics with random parameter \(\ell\) over time (e.g., perturbations in boundary conditions of turbulent flow). Both cases will also include the global physical parameters \(\mathbf{\mu}\). The scope of our paper is to build a unified data-driven surrogate model for generating/predicting parametric PDE solutions.

### Normalizing flow models

A normalizing flow model constructs a flexible probabilistic distribution by applying a learnable bijective mapping to a simple random variable (e.g. Gaussian distributed). Suppose the mapping is \(\mathbf{z}=f(\mathbf{x})\) with \(\mathbf{x}\in\mathbb{R}^{d}\) being the simple input variable, then we have the probability \(p(\mathbf{z})\) of \(\mathbf{z}\) as follows:

\[p(\mathbf{z})=p(\mathbf{x})\bigg{|}\text{det}\left(\frac{\partial(f(\mathbf{x}))}{\partial( \mathbf{x})}\right)\bigg{|}^{-1}. \tag{2}\]

Here \(\frac{\partial(f(\mathbf{x}))}{\partial(\mathbf{x})}\) is the Jacobian of \(f\) at \(\mathbf{x}\). The function \(f\) is usually a neural network with a layered structure, with each layer being a bijective mapping. Then the determinant of \(\frac{\partial(f(\mathbf{x}))}{\partial(\mathbf{x})}\) is the product of determinants of these layers' Jacobian matrices. With a special design of layer structures, their determinants can be efficiently computed. For example, a RealNVP model [23] is constructed by stacking several _coupling_ layers, each of which has a lower triangular Jacobian matrix. With \(\mathbf{h}\in\mathbb{R}^{d}\) as the input, a coupling layer \(f_{\ell}\) runs the following calculation:

\[f_{\ell}(\mathbf{h})=\mathbf{concat}(\mathbf{h}[1:d^{\prime}],\mathbf{h}[(d^{\prime}+1):d] \odot\exp(s(\mathbf{h}[1:d^{\prime}]))+t(\mathbf{h}[1:d^{\prime}])) \tag{3}\]

Here \(\mathbf{concat}\) concatenates its arguments as one vector, and \(d^{\prime}\) is usually about one-half of \(d\). In this calculation, the first half of the vector is directly copied to the output. The second half goes through an entry-wise linear transformation: the operation \(\odot\) is the Hadamard product, the neural network \(s\) provides scaling coefficients, and the neural network \(t\) provides biases. With \(K\) such coupling layers, we have a transformation that defines a flexible distribution \(p(\mathbf{z})\).

\[p(\mathbf{z})=p(\mathbf{x})\cdot\prod_{\ell=1}^{K}|\text{det}(\partial f_{\ell}(\mathbf{h} ^{\ell-1})/\partial\mathbf{h}^{\ell-1})|^{-1} \tag{4}\]

Here \(\mathbf{h}^{0}=\mathbf{x}\), and \(\mathbf{h}^{\ell}=f_{\ell}(\mathbf{h}_{\ell-1})\). At the same time, \(p(\mathbf{z})\) also has an efficient sampling procedure given by \(f\).

## 3 Methodology

Our new predicting model has three components: an encoder that compresses a graph representation of the state into a fixed-length vector, a sequential model that predicts next-step representations in the latent space, and a decoder that decodes spatial states from graph representations. The encoder is an improved version of the GMR-GMUS encoder [11]. The sequential model is a conditional flow model. The encoder and decoder are trained via self-supervised learning as in regeneration learning.

### Graph representation learning

Following GMR-GMUS, we use a graph \(\mathcal{G}=(\mathcal{V},\mathcal{E})\) to encode a snapshot of a deterministic or stochastic dynamical system at time step \(t\). Here each node \(i\in\mathcal{V}\) corresponds to the mesh cell \(C_{i}\), and each edge \((i,j)\in E\) represents a neighboring relationship between two cells. At each step \(t\), the solution \(\mathbf{u}_{i,t}\) at cell \(i\) becomes the feature vector of the node \(i\in\mathcal{V}\) at time \(t\). Because the mesh is pre-defined and fixed, the graph representation \((G,(\mathbf{u}_{i,t},i\in V))\) preserves the full information of the mesh \(\mathcal{C}_{h}\) at time \(t\). Here we use \(\mathbf{Y}_{t}=(\mathbf{u}_{i,t}:i\in V)\) to denote a snapshot at time \(t\).

The key task for the encoder is to encode the graph into a low-dimensional vector \(\mathbf{z}_{t}\). For this task, we use an improved version of the encoder of GMR-GMUS. The GMR-GMUS encoder runs a graph neural network over the graph representation to learn node representations. Then it takes node vectors of a selection of "pivotal" nodes and concatenates them to get \(\mathbf{z}_{t}\). In our new encoder, we select a set of locations in the spatial space instead of graph nodes to aggregate spatial information: each selected location aggregates vectors of nearby nodes to get its representation. It decouples the graph representation and the aggregation operation so that a select location can encode nodes within an arbitrary distance. To improve the training stability, we also improve the graph neural network'sarchitecture by adding residual connections between its message-passing layers. We call this new encoder Position-based Graph Mesh Reducer (PbGMR). These two modifications clearly improve the encoder's ability to preserve the state information, which will be empirically shown in the experiment section.

The architecture of PbGMR is specified as follows. For notational convenience, we omit the time from the subscript. For each graph node \(i\in V\), we first extract node and edge features as follows.

\[\mathbf{v}_{i}^{0}=\mathrm{mlp}_{v}(\mathbf{u}_{i}),\quad\mathbf{e}_{ij}^{0}=\mathrm{mlp}_ {e}(\mathrm{pos}(i)-\mathrm{pos}(j)). \tag{5}\]

Here \(\mathrm{pos}(i)\) is the spatial location of the cell center of \(C_{i}\). After that, we apply \(L\) message-passing layers.

\[\mathbf{e}_{ij}^{\ell} =\mathbf{e}_{ij}^{\ell-1}+\mathrm{layernorm}(\mathrm{mlp}_{\ell}^{e} \left(\mathbf{e}_{ij}^{\ell-1},\mathbf{v}_{i}^{\ell-1},\mathbf{v}_{j}^{\ell-1})\right) \tag{6}\] \[\mathbf{v}_{i}^{\ell} =\mathbf{v}_{i}^{\ell-1}+\mathrm{layernorm}\left(\mathrm{mlp}_{\ell} ^{v}\left(\mathbf{v}_{i}^{\ell-1},\sum_{j\in\mathcal{N}_{i}}\mathbf{e}_{ij}^{\ell-1} \right)\right),\quad\ell=1,\ldots,L. \tag{7}\]

Here \(\mathcal{N}_{i}\) denotes all neighbors of node \(i\), \(\mathrm{mlp}_{\ell}^{e}\) and \(\mathrm{mlp}_{\ell}^{v}\) are two separate multi-layer perceptrons, and \(\mathrm{layernorm}\) is the layer-normalization operation. When a perceptron accepts multiple arguments, its arguments are first concatenated into a single vector. Each layer here is similar to a GraphNet block [24] but with slight differences. The prominent ones are residual connections and layer normalization in the update of node and edge representations: they help to stabilize the training procedure [25, 26, 27].

After we have learned node representations, we need to aggregate them into a single vector to represent the entire state. We randomly select a small set \(\mathcal{S}\) of centers in the spatial area. For each center \(c\in\mathcal{S}\), we select \(k\) nearest mesh cells \(\mathrm{knn}(c)\) based on spatial distance and then compute the position representation \(\mathbf{h}_{c}\) by interpolation [28]:

\[\mathbf{h}_{c}=\sum_{j\in\mathrm{knn}(c)}\frac{w_{cj}\mathbf{v}_{j}^{L}}{\sum_{j\in \mathrm{knn}(c)}w_{cj}},\quad w_{cj}=\frac{1}{d(c,j)^{2}},\quad,c\in\mathcal{S} \tag{8}\]

Here \(d(c,j)\) is the spatial distance between cell \(C_{j}\) and the center \(c\). Given that the calculation is scaled by the sum of \(w_{cj}\)-s, the unit of the spatial distance does not change the calculation here. The number \(k\) of neighbors is a hyper-parameter, and we fix it to 10. Finally, we concatenate all representations of centers into a single vector \(\mathbf{z}=\mathrm{concat}(\mathbf{h}_{c}:c\in\mathcal{S})\) as the latent for the entire graph. Note that the centers in \(\mathcal{S}\) are fixed for a problem.

As a comparison, the encoder in GMR-GMUS uses graph nodes as centers and only considers connected neighbors of a center. Then the number of neighbors in the interpolation operation is

Figure 1: The diagram of the proposed model, which first compresses the whole graph into the latent representation \(\mathbf{z}_{t}\) by PbGMR using selected positions. During the generation process, a transformer encodes physical parameters and previous latent representations into a condition vector \(\mathbf{c}_{t+1}\), from which a normalizing flow model describes the conditional probability of \(\mathbf{z}_{t+1}\). Finally, the decoder PbGMUS decodes \(\mathbf{z}_{t+1}\) through to obtain the next-step prediction \(\mathbf{Y}_{t+1}\).

limited by the graph structure. As a result, each center vector from GMR-GMUS can only represent information in a short range. Our new encoder overcomes this issue by decoupling the neighbors in interpolation and the neighbors in the graph representation and then gives the interpolation operation more freedom to represent a state of the system.

### Decoding and self-supervised training

We devise a decoder PbMUS to recover node features on the graph from latent representation \(\mathbf{z}\): \(\hat{\mathbf{Y}}=\mathrm{PbGMUS}(\mathbf{z})\). Here we also consider the computation at a single time step and omit time indices. We first split \(\mathbf{z}\) and get vectors at interpolation centers: \((\mathbf{h}_{\mathbf{c}}:c\in\mathcal{S})=\mathbf{z}\) and then compute the initial node representation \(\mathbf{r}_{i}^{0}\) by spatial interpolation from centers:

\[\mathbf{r}_{i}^{0}=\sum_{j\in\mathrm{knn}^{\prime}(i)}\frac{w_{ic}\mathbf{h}_{c}}{\sum_ {c\in\mathrm{knn}^{\prime}(i)}w_{ic}},\quad w_{ic}=\frac{1}{d(i,c)^{2}},\quad,i\in\mathcal{V},c\in\mathcal{S} \tag{9}\]

Here \(\mathrm{knn}^{\prime}(i)\) are \(k\) centers that are nearest to the cell center \(i\). Then we apply \(L\) message-passing layers to compute \(\hat{\mathbf{Y}}=\mathrm{gnn}(G,(\mathbf{r}_{i}^{0}:i\in V))\), with \(\mathrm{gnn}\) representing the \(L\) network layers. These layers have the same architecture as PbGMR but use different learnable parameters.

Self-supervised training.Without considering the sequential property of the data, we first train the encoder and decoder on single steps with self-supervised training. This training method shares the same spirit of regeneration learning and improves encoder and decoder's abilities to capture spatial patterns in the data. In particular, we use the reconstruction error as the minimization objective to train the encoder-decoder pair.

\[\min\ \ \sum_{t=0}^{T}||\mathbf{Y}_{t}-\mathrm{PbGMUS}(\mathrm{PbGMR}(\mathbf{Y}_{t}))|| _{2}^{2} \tag{10}\]

### Attention-based temporal conditioned generative model

Now we consider the sequence of latent representations from the encoder and devise a sequential generative model to model the sequence of latent representations. The low-dimensional latent space reduces the modeling difficulty, and the vector form of latent representations \(\mathbf{z}_{t}\) avoids the graph structure and allows more model choices. We first decompose the sequence as follows.

\[P(\mathbf{z}_{1:T}|\mathbf{\mu},\mathbf{z}_{0})=p(\mathbf{z}_{1}|\mathbf{\mu},\mathbf{z}_{0})\prod_{t= 2}^{T}p(\mathbf{z}_{t}|\mathbf{\mu},\mathbf{z}_{0},\mathbf{z}_{1:t-1}). \tag{11}\]

The key is to devise a model for the conditional \(p(\mathbf{z}_{t}|\mathbf{\mu},\mathbf{z}_{0},\mathbf{z}_{1:t-1})\). We take two steps to construct this conditional: we first represent the condition \((\mathbf{\mu},\mathbf{z}_{0},\mathbf{z}_{1:t-1})\) with a single vector \(\mathbf{c}_{t}\) and then adapt RealNVP [23], a normalizing flow model, to describe the conditional \(p(\mathbf{z}_{t}|\mathbf{c}_{t})\).

Calculate the conditional vector with a transformer.Because physical parameters \(\mathbf{\mu}\) control the entire system, and the initial condition \(\mathbf{z}_{0}\) contains information about the initial condition, we consider them special and use them in the prediction of \(\mathbf{z}_{t}\) for each \(t\). We structure the problem as a "translation" problem and use the transformer [29] to run the calculation. In our case, the input "sentence" is \((\mathbf{\mu},\mathbf{z}_{0})\), the first few "tokens" in the target sentence are \((\mathbf{z}_{1},\dots,\mathbf{z}_{t-1})\), and the "next token" to be predicted is \(\mathbf{c}_{t}\).

\[\mathbf{c}_{t}=\mathrm{transformer}((\mathbf{\mu},\mathbf{z}_{0}),(\mathbf{z}_{1},\dots,\mathbf{z} _{t-1})) \tag{12}\]

While the computation is exactly the same as one step in a translation task, the rationale is very different. First, the translation task directly gets the \(\mathbf{z}_{t}\) from the transformer, but our model needs to send \(\mathbf{c}_{t}\) to a conditional flow model to get the prediction \(\mathbf{z}_{t}\). Second, the translation task has an informative input sequence, but our model uses a less informative input and depends on the transformer to get a reasonable output sequence.

Predict \(z_{t}\) with a conditional flow model.Once we have a vector \(\mathbf{c}_{t}\) representation of the condition, we can construct a conditional flow model from RealMVP. Specifically, we append the condition vector \(\mathbf{c}_{t}\) to the input of the scaling function \(s\) and the bias function \(t\) in equation 3 in each layer.

\[f_{\ell}(\mathbf{h})=[\mathbf{h}[1:d^{\prime}],\mathbf{h}[d^{\prime}+1:d]\odot\exp(s(\mathbf{h}[ 1:d^{\prime}],\mathbf{c}_{t}))+t(\mathbf{h}[1:d^{\prime}],\mathbf{c}_{t})] \tag{13}\]

From these layers, we have the flow model \(p(\mathbf{z}_{t}|\mathbf{c}_{t})\). By chaining up the transformer and the conditional flow model, we have our sequential model for conditional probability \(p(\mathbf{z}_{t}|\mathbf{\mu},\mathbf{z}_{0},\mathbf{z}_{1:t-1})\).

During training, we train the sequential model on \((\mathbf{z}_{0},\dots,\mathbf{z}_{T})\) that are computed from our PbGMR encoder. During inference, we can efficiently sample \(\mathbf{z}_{t}\) from the sequential model. The whole training and inference processes can be found in Appendix A.2 and Appendix A.3.

## 4 Related Work

Deep learning for physics.For a deterministic system, one-step learning simply takes the current state as the input and outputs the next-step prediction [3; 30; 31; 32]. To further improve the accuracy of long-term forecasts, dimension reduction is applied together with sequence nets, aiming to solve long dynamical systems on a regular domain [33; 34; 35]. Such works often adopt CNNs as encoders so that can not be applied to irregular mesh space. To further work on mesh data directly with dimension reduction, the GNN encoder/decoder was initially introduced by [11]. Various GNN architectures were also proposed to facilitate the learning of physics. [36; 37; 38; 39; 13; 15]. For physical stochastic systems, CNN is mainly used for probabilistic prediction [40; 18]. However, it lacks researches solving stochastic system in the graph space.

Regeneration learning and generative modeling.Regeneration learning is a new learning paradigm for data generation, which first generates a latent representation for the data. Then the generation process happens on the latent space. There are many recently popular generative models for images[41; 42; 43], videos[44], speeches[45], and music [46] are built on this paradigm. We notice that it should be further investigated for graph generation tasks.

## 5 Experiments

### Deterministic dynamics

We benchmark our proposed model with three datasets from three flows: flows over the cylinder, high-speed flows over the moving edge, and vascular flows [11]. Note that for test cases, we feed the model physical parameters \(\mathbf{\mu}\) and the snapshot at the beginning time to generate the whole trajectory. Detailed description can be found in Appendix A.5.

We compare our new method against five SOTA models for fluid dynamics, including two variants of MeshGraphNet[3] and three variants of the GMR-GMUS[11]. We use relative mean square error (RMSE) as the evaluation metric: \(\mathrm{RMSE}=\frac{\sum(\tilde{u}_{t}^{\mathrm{prediction}}-\tilde{u}_{t}^{ \mathrm{truth}})^{2}}{\sum(\tilde{u}_{t}^{\mathrm{prediction}})^{2}}\). More details can be found in the Appendix A.2. For cylinder and vascular flows, we calculate the RMSE with respect to velocity variables \(u\), \(v\), and the pressure variable \(p\). For high-speed flows over the moving edge, we also calculated the RMSE for the temperature \(T\).

The results on prediction errors.Table 2 shows the comparison of different models. Our model outperforms the baseline models in all the scenarios. For the cylinder flow case, our model has improvements of \(22\%\), \(17\%\), and \(47\%\) for \(u\), \(v\), and \(p\). For the sonic flow case, our model has improvements of \(61\%\), \(70\%\) for the \(u\) and \(v\) variables, and improvements of \(82\%\) and \(50\%\) respectively for the \(p\) and \(T\) variables. For the vascular flow case, our model achieves improvements of \(52\%\), \(45\%\), and \(95\%\) for \(u\), \(v\), and \(p\). The ablation

\begin{table}
\begin{tabular}{c c c} \hline \hline Dataset & GMR-GMUS & PbGMR-GMUS \\ \hline Cylinder flow & \(14.3\) & \(\mathbf{1.9}\) \\ \hline Sonic flow & \(1.11\) & \(\mathbf{0.24}\) \\ \hline Vascular flow & \(10\) & \(\mathbf{2.8}\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: The average relative reconstruction error of three systems, with the unit of \(1\times 10^{-3}\)study in the Appendix section (Table 9) indicates that both the new encoder and the conditional normalizing flow are essential to improve prediction accuracy.

The error in the time axis is shown in Figure 3. The performance of our model is shown with the solid black line. The model has small initial errors in all three tasks. Then its error accumulates over time, but the accumulation is much slower than that of competing models, so our model consistently has the lowest error in the final time step.

To better understand our model, we have also done an extensive ablation study reported in the appendix. Here we only show a comparison of our model with GMR-GMUS in the encoding-decoding tasks. The results in Table. 1 show that the new model has clearly better encoding and decoding capabilities. In one encoding and decoding case, the reconstruction error of our model is less than one fifth of that of the GMR-GMUS model in the appendix. The ablation study also shows that both residual connections and encoding centers enhance the encoding-decoding accuracy (please see Table 8 in the Appendix section).

Contour analysis.As a probabilistic model, we plotted the contour of the predicted sample and the standard deviation (std) for all three datasets. Figure 2 visualizes the velocity predictions from two datasets. In each subplot, the top row is one prediction (e.g. one sample from our model), and the bottom row visualizes the standard deviation of predictions. To clearly compare different predictions, we use the same color bar within the same dataset. With visual inspection, the sample predictions are reasonable and stable for all three datasets.

The standard deviations also show reasonable spatial-temporal patterns consistent with our expectations. For cylinder flow with \(Re=307\), the standard deviation is larger around the vortex-shedding region, indicating the model is less certain about the fast-changing dynamics. Moreover, the standard deviation tends to grow with time, reflecting the error accumulation behavior for long-time rollouts. Moreover, the standard deviation is smaller when \(Re\) is higher, indicating that the model is more certain about the prediction. The same trends can also be seen in other datasets. Therefore, our proposed model can accurately predict the dynamics for deterministic systems while providing a spatial-temporal uncertainty estimate, potentially improving the interpretability of deep learning systems. More contours can be found in Appendix A.6

### Stochastic dynamics

Dataset.We apply the proposed method to solve a stochastic dynamical system governed by the unsteady-state incompressible Navier-Stokes equations. We also compare our method to the unsteady Reynolds-averaged Navier-Stokes equations (URANS), the most applied method in the industry due to the balance of accuracy and computational efficiency. Let \(\Omega\) be the channel with a backward-facing

Figure 2: For each case, the first row is one of our predicted samples for velocity. The second row is the predicted standard deviation for velocity. Our model-predicted sample accurately reflects the physical systems. Meanwhile, the predicted standard deviation also has a physical spatial-temporal pattern.

step shown in Figure 5. The goal of the problem is to generate the temporal unsteady velocity field close to the ones simulated by large eddy simulation (LES) with the inflow velocity along the inlet boundary subject to uncertainty. The inflow at every time step is uncertain and sampled from a 60-dimensional stochastic space subject to a uniform distribution as \(\{u_{1}^{\mathrm{inlet}},...,u_{60}^{\mathrm{inlet}}|u_{i}^{\mathrm{inlet}}=10+u^{ \prime},u^{\prime}\sim\mathcal{U}(0,1)\}\).

Analysis of fluid motion.To visually study the generated results, we plot several time steps of the stream-wise velocity (\(u\)) from a collection of model samples in Figure 4. The proposed method clearly generates diverse turbulent flow samples. Specifically, the multiscale structure of the vortex can be seen in the contour plots. To quantitatively examine the model performance on turbulent statistics, we plot and compare the mean and variance of the velocity profiles from the URANS, the proposed model, and LES results in Figure 5. The mean velocity profile predicted by the URANS model (gray lines) has significant discrepancies compared to the LES result (cyan lines), particularly in the right half of the domain where the backward-facing step's geometry is less important and small-scale turbulent features are dominant. As for the velocity variance, the URANS model underestimates the fluctuation in the whole domain. On the contrary, both the LES velocity mean and variance can be captured well by our generated samples (purple lines).

the multiscale problem in the time domain can be automatically learned. However, URANS directly calculate an ensemble average over infinite experiments, and it can not capture stochastic behavior. Therefore, the energy (\(E(k)\)) of high-wave-number (\(k\)) signals is underpredicted by URANS while the proposed model has a consistent energy spectrum pattern with the LES simulations. Moreover, the turbulent kinetic energy (TKE) is computed to evaluate the quality of generated samples. Physically, the TKE is characterized by measured root-mean-square velocity fluctuations. By applying the decoder, the point-wise instantaneous flow can be recovered from the latent vector to the physical solution reasonably well. The TKE metric can be defined as a scalar:

\[\text{TKE}:=\frac{\int_{\Omega}\int_{[0,T_{\text{rad}}]}[(u-\bar{u})^{2}+(v- \bar{v})^{2}]d\Omega dt}{\int_{\Omega}\int_{[0,T_{\text{rad}}]}[(u^{\text{les}} -\bar{u}^{\text{les}})^{2}+(v^{\text{les}}-\bar{v}^{\text{les}})^{2}]d\Omega dt}. \tag{14}\]

It indicates the preservation of the TKE benchmarked by the LES simulations. Our model can generate a flow field preserving \(99\%\) of the energy, whereas URANS can only do less than \(20\%\). As for the temporal mean field, although URANS aims to solve it directly, it is inevitable to introduce the bias from the LES result due to the ergodicity assumption of RANS. Instead, we directly formulate the probabilistic problem to learn the distribution of the spatiotemporal field. We calculate the error of the temporal mean field and find that although URANS leverages the conservation laws, it still obtains a much larger error than our generated samples. Finally, we further propose to measure the quality of predicted rollouts using evaluation metrics of video generation in computer vision. In particular, we consider two metrics: the continuous ranked probability score (CRPS) [48; 49] and Frechet Video Distance (FVD) [50]. We use CRPS to assess the respective accuracy of our probabilistic forecasting and LES models. Since our model considers the distribution of the forecasts as a whole, it outperforms URANS, which only focuses on the mean of the distribution. Our model is also better evaluated in the FVD metric. This indicates the learned distribution by our model is close to the samples from LES.

We have also experimented with two deep learning methods, MeshGraphNet and GMR-GMUS, which are designed for deterministic systems. Their predictions are both far from LES simulations, not to mention that they can only make deterministic predictions. More results can be found in Appendix A.13.

## 6 Conclusion

We propose a new learning model to predict/generate deterministic and stochastic fluid dynamics over unstructured mesh in a uniform way. With an integration of a novel graph auto-encoder, a transformer, and a normalizing flow model, the new model decomposes temporal and spatial correlations in a dynamical system. It outperforms competitive baseline models for deterministic systems while providing a reasonable spatial-temporal pattern of forward uncertainty estimations. The samples

Figure 5: Evaluation lines: Inlet (), Outlet (), Wall (), \(x\)-evaluation lines ( - - ). Quantities from LES (), URANS (), and our model ()

from the model trained on stochastic systems capture the rich physical patterns of expensive LES simulations. The current model still has one limitation: it can not accurately model stochastic variables close to boundary areas, which is shown in Appendix A.12. In future work, we will design new learning architectures to overcome this issue.

## Acknowledgement

We thank all reviewers for their insightful feedback. Liu was supported by NSF CAREER 2239869. Wang's group would like to acknowledge the funds from Office of Naval Research under award numbers N00014-23-1-2071 and National Science Foundation under award numbers OAC-2047127.

## References

* [1] Fadl Moukalled, Luca Mangani, Marwan Darwish, F Moukalled, L Mangani, and M Darwish. _The finite volume method_. Springer, 2016.
* [2] Stephen B Pope and Stephen B Pope. _Turbulent flows_. Cambridge university press, 2000.
* [3] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. Learning mesh-based simulation with graph networks. In _International Conference on Learning Representations_, 2021.
* [4] Apostolos F Psaros, Xuhui Meng, Zongren Zou, Ling Guo, and George Em Karniadakis. Uncertainty quantification in scientific machine learning: Methods, metrics, and comparisons. _Journal of Computational Physics_, page 111902, 2023.
* [5] Yarin Gal, Petros Koumoutsakos, Francois Lanusse, Gilles Louppe, and Costas Papadimitriou. Bayesian uncertainty quantification for machine-learned models in physics. _Nature Reviews Physics_, 4(9):573-577, 2022.
* [6] Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. _Advances in neural information processing systems_, 32, 2019.
* [7] Jayaraman Thiagarajan, Rushil Anirudh, Vivek Sivaraman Narayanaswamy, and Timo Bremer. Single model uncertainty estimation via stochastic data centering. _Advances in Neural Information Processing Systems_, 35:8662-8674, 2022.
* [8] James M Dolezal, Andrew Srisuwananukorn, Dmitry Karpeyev, Siddhi Ramesh, Sara Kochanny, Brittany Cody, Aaron S Mansfield, Sagar Rakshit, Radhika Bansal, Melanie C Bois, et al. Uncertainty-informed deep learning models enable high-confidence predictions for digital histopathology. _Nature communications_, 13(1):6572, 2022.
* [9] Sifan Wang and Paris Perdikaris. Long-time integration of parametric evolution equations with physics-informed deeponets. _Journal of Computational Physics_, 475:111855, 2023.
* [10] Zhong Yi Wan, Leonardo Zepeda-Nunez, Anudhyan Boral, and Fei Sha. Evolve smoothly, fit consistently: Learning smooth latent dynamics for advection-dominated systems. _arXiv preprint arXiv:2301.10391_, 2023.

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|} \hline \multicolumn{2}{|c|}{URANS (\(\dashrightarrow\)), LES (\(\dashrightarrow\)), Ours(\(\dashrightarrow\))} & Quantity of interest & URANS & Ours \\ \hline \multirow{4}{*}{
\begin{tabular}{} \end{tabular} } & \multirow{4}{*}{\(10^{0}\)} & \multirow{4}{*}{\(10^{1}\)} & CRPS (\(C_{u}\),\(\downarrow\)) & 3.24 & **1.28** \\  & & CRPS (\(C_{v}\),\(\downarrow\)) & 2.0 & **1.08** \\  & & FVD (\(d_{u}\),\(\downarrow\)) & 228112 & **1262** \\  & & FVD (\(d_{v}\),\(\downarrow\)) & 137860 & **397** \\  & & mean flow error (\(e_{u}\),\(\downarrow\)) & 0.31 & **0.0176** \\  & & mean flow error (\(e_{v}\),\(\downarrow\)) & 0.94 & **0.176** \\  & & turbulent energy (TKE,\(\uparrow\)) & 0.192 & **0.99** \\ \hline \end{tabular}
\end{table}
Table 3: Energy cascade spectrum at \((2.58,0.21)\)(_left_), and criteria of generation quality(_right_).

* [11] Xu Han, Han Gao, Tobias Pffaf, Jian-Xun Wang, and Li-Ping Liu. Predicting physics in mesh-reduced space with temporal attention. _arXiv preprint arXiv:2201.09113_, 2022.
* [12] Rishikesh Ranade, Chris Hill, Lalit Ghule, and Jay Pathak. A composable machine-learning approach for steady-state simulations on high-resolution grids. _arXiv preprint arXiv:2210.05837_, 2022.
* [13] Jiayang Xu, Aniruddhe Pradhan, and Karthikeyan Duraisamy. Conditionally parameterized, discretization-aware neural networks for mesh-based modeling of physical systems. _Advances in Neural Information Processing Systems_, 34:1634-1645, 2021.
* [14] Dmitrii Kochkov, Tobias Pfaff, Alvaro Sanchez-Gonzalez, Peter Battaglia, and Bryan K Clark. Learning ground states of quantum hamiltonians with graph networks. _arXiv preprint arXiv:2110.06390_, 2021.
* [15] Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In _International conference on machine learning_, pages 8459-8468. PMLR, 2020.
* [16] Nicholas Geneva and Nicholas Zabaras. Multi-fidelity generative deep learning turbulent flows. _arXiv preprint arXiv:2006.04731_, 2020.
* [17] Michael McCabe and Jed Brown. Learning to assimilate in chaotic dynamical systems. _Advances in Neural Information Processing Systems_, 34:12237-12250, 2021.
* [18] Mustafa Z Yousif, Linqi Yu, and HeeChang Lim. Physics-guided deep learning for generating turbulent inflow conditions. _Journal of Fluid Mechanics_, 936:A21, 2022.
* [19] Deniz A Bezgin and Nikolaus A Adams. Normalizing flows as a novel pdf turbulence model. _arXiv preprint arXiv:2101.03590_, 2021.
* [20] Mengyuan Yang, Xiaolin Zheng, Qianqiao Liang, Bing Han, and Mengying Zhu. A smart trader for portfolio management based on normalizing flows. IJCAI, 2022.
* [21] Hrvoje Jasak, Aleksandar Jemcov, Zeljko Tukovic, et al. Openfoam: A c++ library for complex physics simulations. In _International workshop on coupled methods in numerical dynamics_, volume 1000, pages 1-20, 2007.
* [22] Xu Tan, Tao Qin, Jiang Bian, Tie-Yan Liu, and Yoshua Bengio. Regeneration learning: A learning paradigm for data generation. _arXiv preprint arXiv:2301.08846_, 2023.
* [23] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. _arXiv preprint arXiv:1605.08803_, 2016.
* [24] Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In _International Conference on Machine Learning_, pages 4470-4479. PMLR, 2018.
* [25] David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In _International Conference on Machine Learning_, pages 342-350. PMLR, 2017.
* [26] Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. _Advances in Neural Information Processing Systems_, 34:22614-22627, 2021.
* [27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, June 2016.
* [28] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. _Advances in neural information processing systems_, 30, 2017.

* [29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. _Advances in neural information processing systems_, 30, 2017.
* [30] Kelsey R Allen, Tatiana Lopez Guevara, Yulia Rubanova, Kim Stachenfeld, Alvaro Sanchez-Gonzalez, Peter Battaglia, and Tobias Pfaff. Graph network simulators can learn discontinuous, rigid contact dynamics. In _Conference on Robot Learning_, pages 1157-1167. PMLR, 2023.
* [31] Meire Fortunato, Tobias Pfaff, Peter Wirnsberger, Alexander Pritzel, and Peter Battaglia. Multiscale meshgraphnets. _arXiv preprint arXiv:2210.00612_, 2022.
* [32] Kelsey R Allen, Tatiana Lopez-Guevara, Kimberly Stachenfeld, Alvaro Sanchez-Gonzalez, Peter Battaglia, Jessica Hamrick, and Tobias Pfaff. Physical design using differentiable learned simulators. _arXiv preprint arXiv:2202.00728_, 2022.
* [33] Han Gao, Jian-Xun Wang, and Matthew J Zahr. Non-intrusive model reduction of large-scale, nonlinear dynamical systems using deep learning. _Physica D: Nonlinear Phenomena_, 412:132614, 2020.
* [34] Nicholas Geneva and Nicholas Zabaras. Transformers for modeling physical systems. _Neural Networks_, 146:272-289, 2022.
* [35] Tailin Wu, Takashi Maruyama, and Jure Leskovec. Learning to accelerate partial differential equations via latent global evolution. _Advances in Neural Information Processing Systems_, 35:2240-2253, 2022.
* [36] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. _Advances in neural information processing systems_, 29, 2016.
* [37] Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. _arXiv preprint arXiv:1810.01566_, 2018.
* [38] Marco Maurizi, Chao Gao, and Filippo Berto. Predicting stress, strain and deformation fields in materials and structures with graph neural networks. _Scientific Reports_, 12(1):21834, 2022.
* [39] Jiaqi Han, Wenbing Huang, Hengbo Ma, Jiachen Li, Josh Tenenbaum, and Chuang Gan. Learning physical dynamics with subequivariant graph neural networks. _Advances in Neural Information Processing Systems_, 35:26256-26268, 2022.
* [40] Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics-informed deep learning for turbulent flow prediction. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1457-1466, 2020.
* [41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High-resolution image synthesis with latent diffusion models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 10684-10695, 2022.
* [42] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In _International Conference on Machine Learning_, pages 8821-8831. PMLR, 2021.
* [43] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. _Advances in Neural Information Processing Systems_, 35:36479-36494, 2022.
* [44] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. _arXiv preprint arXiv:2204.03458_, 2022.
* [45] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_, pages 4779-4783. IEEE, 2018.

* [46] Zeqian Ju, Peiling Lu, Xu Tan, Rui Wang, Chen Zhang, Songruoyao Wu, Kejun Zhang, Xiangyang Li, Tao Qin, and Tie-Yan Liu. Telemelody: Lyric-to-melody generation with a template-based two-stage method. _arXiv preprint arXiv:2109.09617_, 2021.
* [47] Peter Welch. The use of fast fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. _IEEE Transactions on audio and electroacoustics_, 15(2):70-73, 1967.
* [48] James E Matheson and Robert L Winkler. Scoring rules for continuous probability distributions. _Management science_, 22(10):1087-1096, 1976.
* [49] Christopher AT Ferro, David S Richardson, and Andreas P Weigel. On the effect of ensemble size on the discrete and continuous ranked probability scores. _Meteorological Applications: A journal of forecasting, practical applications, training techniques and modelling_, 15(1):19-24, 2008.
* [50] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
* [51] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_, 2014.
* [52] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). _arXiv preprint arXiv:1606.08415_, 2016.
* [53] Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. _Neural computation_, 9(8):1735-1780, 1997.
* [54] George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. _Advances in neural information processing systems_, 30, 2017.

Appendix

### Proof

The Jacobian of the transformation in Equation (4) can be written as:

\[\frac{\partial f^{i}(x)}{\partial f^{i-1}(x)}=\left[\begin{array}{cc}\mathbb{I} ^{d^{\prime}}&0\\ \frac{\partial f^{i}(x)^{d^{\prime}+1:d}}{\partial f^{i-1}(x)^{1:d^{\prime}}}& \text{diag}\big{(}\text{exp}\left[s\left(f^{i-1}\text{concat}((x)^{1:d^{\prime }},c)\right)\right]\text{)}\end{array}\right] \tag{15}\]

fwhere \(\text{diag}\big{(}\text{exp}[s(f^{i-1}\text{concat}((x)^{1:d^{\prime}},c))]\) is the diagonal matrix whose diagonal elements correspond to the vector \(\text{exp}\left[s\left(f^{i-1}\text{concat}((x)^{1:d^{\prime}},c)\right)\right]\). Since the concatenation operation still keeps the matrix triangular, the determinant can be computed as:

\[\text{det}(\partial f^{i}(\mathbf{x})/\partial f^{i-1}(\mathbf{x}))|=\text{exp}(\text{ sum}(s_{i}(\text{concat}(f^{i-1}(\mathbf{x})^{1:d^{\prime}},c)))) \tag{16}\]

### Training

As a regeneration learning method, there are two stages during the training. The first stage is to train a graph auto-encoder, PbGMR-GMUS, in this case, through self-supervised learning. This training process happens over all time steps and sequences in the training set. And for a given sequence \([\mathbf{Y}_{0},\dots,\mathbf{Y}_{T}]\), we compute \(\hat{\mathbf{Y}}_{t}=\text{PbGMUS}(\mathcal{G},\text{PbGMR}(\mathcal{G},\text{Y}))\) at each step and minimize the reconstruction loss:

\[\mathcal{L}_{graph}=\quad\sum_{t=1}^{T}\|\mathbf{Y}_{t}-\hat{\mathbf{Y}}_{t}\|_{2}^{2 }\quad. \tag{17}\]

In the second stage, the parameters in PbGMR-GMUS will be fixed. And the attention-based sequence model, as well as the conditional flow model, will be trained. We maximize the log-likelihood in the latent space:

\[\mathcal{L}_{p}=-\frac{1}{T-1}\sum_{t=2}^{T}\log p_{\psi}(\mathbf{z}_{t}|\mathbf{c}_{t}) \tag{18}\]

We don't set a time window so that the whole trajectory will be trained. The entire training process can be found in Algorithm 1. Particularly in the second stage, such a teacher forcing-like strategy allows a faster training process compared with the previous work [11], which adopts a rollout strategy during the training.

### Inference

For inference we first encode the first two snapshots into latent space \([\mathbf{z}_{0},\mathbf{z}_{1}]\). The condition vector \(\mathbf{c}_{t}\) is computed from the attention-based sequence model. By sampling a noise vector \(\mathbf{x}\) from an isotropic Gaussian, and putting it going backward through the flow, we can obtain a new sample from \(p_{\phi}(\mathbf{z}_{t}|\mathbf{c}_{t})\) as latent representation at each step. We describe the procedure of training in Algorithm 2.

```
1:Input: \(\mathbf{z}_{0}\), \(\mathbf{z}_{1}\), \(\mathbf{z}_{2}\), \(\mathbf{z}_{3}\), \(\mathbf{z}_{4}\), \(\mathbf{z}_{5}\), \(\mathbf{z}_{6}\), \(\mathbf{z}_{7}\), \(\mathbf{z}_{8}\), \(\mathbf{z}_{9}\), \(\mathbf{z}_{10}\), \(\mathbf{z}_{11}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{19}\), \(\mathbf{z}_{12}\), \(\mathbf{z}_{13}\), \(\mathbf{z}_{14}\), \(\mathbf{z}_{15}\), \(\mathbf{z}_{16}\), \(\mathbf{z}_{17}\), \(\mathbf{z}_{18}\), \(\mathbf{z}_{19}\```
Input: Domain graph \(\mathcal{G}\), Node features over time \([\mathbf{Y}_{0},\ldots,\mathbf{Y}_{T}]\), \(\mathrm{PbGMR}_{\theta}\), \(\mathrm{PbGMUS}_{\phi}\), Attention encoder \(\mathrm{MHA}_{\omega}\), Attention decoder Masked-\(\mathrm{MHA}_{\lambda}\), conditional flow model \(p_{\psi}(.|)\), physical condition parameter \(\mathbf{\mu}\) Output: Learned parameters \(\theta\), \(\phi\), \(\omega\) and \(\psi\) repeat for\(\mathbf{Y}_{t}\in[\mathbf{Y}_{0},\ldots,\mathbf{Y}_{T}]\)do \(\hat{\mathbf{Y}}_{t}=\mathrm{PbGMUS}_{\phi}(\mathrm{PbGMR}_{\theta}(\mathbf{Y}_{t}, \mathcal{G}))\) endfor  Compute \(\nabla_{\theta,\phi}\leftarrow\nabla_{\theta,\phi}\mathcal{L}_{graph}(\theta, \phi,[\mathbf{Y}_{0},\ldots,\mathbf{Y}_{T}],[\hat{\mathbf{Y}}_{0},\ldots,\hat{\mathbf{Y}}_{T}])\)  Update \(\phi\), \(\theta\) using the gradients \(\nabla_{\phi}\), \(\nabla_{\theta}\) until convergence of the parameters (\(\theta\), \(\phi\)) {Self-supervised learning} \([z_{0},z_{1},\ldots,z_{T}]=\mathrm{GMR}_{\theta}([\mathbf{Y}_{0},\ldots,\mathbf{Y}_{T }])\) repeat \(\hat{\mathbf{\mu}},\hat{\mathbf{z}}_{0}=\mathrm{MHA}_{\omega}(\mathbf{\mu},\mathbf{z}_{0})\) \([\mathbf{c}_{2},\ldots,\mathbf{c}_{t}]=\text{Masked-MHA}_{\lambda}((\mathbf{z}_{1},\ldots, \mathbf{z}_{t-1}),\mathbf{I}=(\hat{\mathbf{\mu}},\hat{\mathbf{z}}_{0}))\)  Compute \(\nabla_{\omega,\lambda,\psi}\leftarrow\nabla_{\omega,\lambda,\psi}\mathcal{L }_{p}(\omega,\lambda,\psi,[\mathbf{c}_{2},\ldots,\mathbf{c}_{t}],[\mathbf{z}_{2},\ldots, \mathbf{z}_{t}])\)  Update \(\omega\), \(\lambda\), \(\psi\) using the gradients \(\nabla_{\omega}\), \(\nabla_{\lambda}\), \(\nabla_{\psi}\) until convergence of the parameters (\(\omega\), \(\lambda\), \(\psi\))
```

**Algorithm 1** Training Process

```
Input: Domain graph \(\mathcal{G}\), The first two snapshot of node features \([\mathbf{Y}_{0},\mathbf{Y}_{1}]\), \(\mathrm{PbGMR}_{\theta}\), \(\mathrm{PbGMUS}_{\phi}\), Attention encoder \(\mathrm{MHA}_{\omega}\), Attention decoder Masked-\(\mathrm{MHA}_{\lambda}\), conditional flow model \(p_{\psi}(.|)\), physical condition parameter \(\mathbf{\mu}\), sample length \(T\) Output: A trajectory sample \([\mathbf{Y}_{2},\ldots,\mathbf{Y}_{T}]\) \([z_{0},\mathbf{z}_{1}]=\mathrm{PbGMR}_{\theta}([\mathbf{Y}_{0},\mathbf{Y}_{1}],\mathcal{G})\) \(\hat{\mathbf{\mu}},\hat{\mathbf{z}}_{0}=\mathrm{MHA}_{\omega}(\mathbf{\mu},\mathbf{z}_{0})\) for\(t\in[2,\ldots,T]\)do \(\mathbf{c}_{t}=\text{Masked-MHA}_{\lambda}((\mathbf{z}_{1},\ldots,\mathbf{z}_{t-1}),\mathbf{I}=( \hat{\mathbf{\mu}},\hat{\mathbf{z}_{0}}))[-1]\)  Sample latent representation from conditional flow \(\mathbf{z}_{t}\sim p_{\phi}(\mathbf{z}_{t}|\mathbf{c}_{t})\) endfor  Recover physical parameters on the original space \([\mathbf{Y}_{2},\ldots,\mathbf{Y}_{T}]=\mathrm{PbGMUS}_{\phi}([\mathbf{z}_{2},\ldots,\mathbf{ z}_{T}])\)
```

**Algorithm 2** Inference Process

Figure 6: 2-D principle subspace of the (a) latent vectors and (b) conditional latent vectors for the stochastic BFS cases: trajectory 1 (), trajectory 2 () start from and, respectively.

### Dataset

Four fluid simulation datasets, cylinder flow, sonic flow, vascular flow and stochastic backward-facing step (BFS) are used in the experiments. The labeled data are generated by solving the Navier-Stokes equation by OpenFOAM [21], a commonly used finite volume method-based open-source solver for fluid simulation. Cylinder and sonic flow consist of \(50\) cases in the training dataset and \(50\) cases in the test dataset. The vascular flow cases have \(10\) training and \(10\) test cases. The stochastic BFS case has \(5\) training cases. Unlike the first three deterministic cases, the stochastic BFS case is a stochastic case. The simulation details can be found in Table 4. Moreover, the details input and output of the PbGMR-GMUS are listed in Table 5.

The governing equation for the first three cases is the same as the one listed in [11]. For the last stochastic BFS case, the governing equation is listed in Equation (19). One significant difference in this case is that it introduces stochasticity by perturbing the inlet \(u\) velocity at every time step.

Figure 8: Pressure contour for benchmark cases

Figure 7: Velocity contour for the vascular flow case, the first row is one of our predicted samples. The second row is the predicted standard deviation.

Figure 9: Temperature contour for sonic flow case.

### Multi-head attention

Transformer [29] has been proven successful in the NLP field. The design of the multi-head attention (MHA) layer is based on the attention mechanism with Query-Key-Value (QKV). Given the packed matrix representations of queries \(\mathbf{Q}\), keys \(\mathbf{K}\), and values \(\mathbf{V}\), the scaled dot-product attention used by Transformer is given by:

\[\mathrm{ATTENTION}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathrm{softmax}\left( \frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{D_{k}}}\right)\mathbf{V}, \tag{20}\]

where \(D_{k}\) represents the dimensions of queries and keys.

The multi-head attention applies \(H\) heads of attention, allowing a model to attend to different types of information.

\[\mathrm{MHA}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathrm{CONCAT}\left(\mathrm{ head}_{1},\ldots,\mathrm{head}_{H}\right)\mathbf{W}\] \[\mathrm{where}\quad\mathrm{head}_{i}=\mathrm{ATTENTION}\left(\mathbf{ Q}\mathbf{W}_{i}^{Q},\mathbf{K}\mathbf{W}_{i}^{K},\mathbf{V}\mathbf{W}_{i}^{V}\right),i=1,\ldots,H. \tag{21}\]

### Additional details for experimental setups

We described the details of the experiments of PbGMR-GMUS and attention-based conditional flow model. We provide the hyperparameters used in the experiments in Table 6.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & Cylinder & Sonic & Vascular & Stochastic BFS \\ \hline
**PbGMR-GMUS Optimization** & & & & \\ Learning rate & \(1\times 10^{-4}\) & \(1\times 10^{-5}\) & \(1\times 10^{-4}\) & \(1\times 10^{-4}\) \\ Optimizer & & & Adam [51] & \\ Batch size & & & 1 & \\ Number of epochs & \(1500\) & \(20000\) & \(700\) & \(600\) \\ \hline
**MHA/Flow model Optimization** & & & & \\ Learning rate & \(1\times 10^{-4}\) & \(1\times 10^{-5}\) & \(1\times 10^{-5}\) & \(1\times 10^{-4}\) \\ Optimizer & & & Adam & \\ Batch size & \(5\) & \(50\) & \(10\) & \(5\) \\ Number of epochs & \(90000\) & \(240000\) & \(240000\) & \(220000\) \\ Weight decay & \(1\times 10^{-5}\) & & N/A & \\ \hline
**PbGMR-GMUS Architecture** & & & & \\ Layers of message-passing & & & 3 & \\ Hidden dimension & & & 128 & \\ Output dimension & \(256\) & \(256\) & \(400\) & \(256\) \\ Activation function & & & relu & \\ \hline
**MHA Architecture** & & & & \\ Layers of Encoding MHA & & & 2 & \\ Layers of Decoding Masked-MHA & & & 1 & \\ Hidden dimension & \(1024\) & \(1024\) & \(1600\) & \(1024\) \\ Activation function & & & gelu [52] & \\ Number of heads & \(4\) & \(8\) & \(4\) & \(4\) \\ \hline
**Flow model Architecture** & & & & \\ Conditioning length & & & \(1024\) & \\ Hidden size & & & \(1024\) & \\ Number of coupling layers & & & \(2\) & \\ \hline \hline \end{tabular}
\end{table}
Table 6: Hyperparameters

### Performance impact of substituting Transformer with LSTM

In this research, we conduct an ablation study in which we replace the proposed attention-based sequence model in our framework with a Long Short-Term Memory (LSTM) [53] architecture. The aim is to evaluate the impact of the underlying sequence modeling on the performance of our system. The result is listed in Table 7. Following this change, we observe that there is a notable decrease in the performance metrics, suggesting a less optimal fit for the task at hand compared to the Transformer-based model. This outcome underscores the usefulness of attention-based sequence models, like the Transformer, which appears to capture dependencies in the data more effectively than the LSTM. The attention mechanism inherent to Transformers allows the model to focus on different parts of the input sequence dynamically, which may explain the observed performance superiority.

### Performance impact of number of centers

We conduct an ablation study to investigate the effect of the number of centers on the RMSE of our framework's predictions for cylinder flow recovery. The result is in Figure 10. As we progressively increased the number of centers, the recovery RMSE exhibited a decreasing trend, demonstrating an enhancement in prediction accuracy. Interestingly, upon reaching 256 centers, the rate of RMSE decrease started to decelerate significantly. This suggests that the optimal number of centers for cylinder flow is around 256. Beyond this point, further increments do not contribute as much to the improvement in encoding-decoding accuracy, while simultaneously increasing the computational cost. Hence, a count of 256 centers appears to be the sweet spot, offering a good trade-off between recovery accuracy and computational efficiency.

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline Dataset-rollout step & \multicolumn{6}{c}{Sonic flow-40} & \multicolumn{6}{c}{Vascular flow-250} \\ Variable & \(u\) & \(v\) & \(p\) & \(T\) & \(u\) & \(v\) & \(p\) \\ \hline PbGMR-GMUS & LSTM & \(10.3\) & \(29.7\) & \(1.7\) & \(11.1\) & \(43.0\) & \(46.6\) & \(16.6\) \\ \hline \hline \end{tabular}
\end{table}
Table 7: The average relative rollout error of two systems, with the unit of \(1\times 10^{-3}\).

Figure 10: Reconstructor error RMSE with different numbers of centers for cylinder flow, with the training epoch = \(300\).

### Centers

The centers distribution for Stochastic BFS can be found in Figure 11.

### Analysis for stochastic BFS

Backward-facing step flow refers to a fluid flow configuration characterized by a sudden contraction followed by an expansion in a channel or pipe. This flow configuration is commonly encountered in various engineering applications, such as in heat exchangers, combustion chambers, and aerodynamic systems. In backward-facing step flow, the fluid initially enters the channel or pipe through an inlet, where it encounters a step or abrupt contraction. As the fluid passes over the step, its velocity increases, and the flow separates, forming a recirculation zone downstream. This recirculation zone is often referred to as the "backflow region." The backward-facing step flow has been extensively studied due to its complex flow characteristics and its relevance in understanding phenomena like separation, reattachment, and flow control.

The simulations and generated samples exhibit remarkable similarity in the contours of both streamwise velocity and wall-normal velocity (Figure 4, 12, 13, 14). This close resemblance demonstrates the high fidelity of the simulation results and the accuracy of the generated samples. The level of detail captured in these contours highlights the effectiveness of the proposed generative framework.

We also randomly choose nine spatial locations to analyze the learning performance, and the coordinates of these points are plotted in Figure 15. Figure 16 and 17 presented the comparison of \(u,\,v\) velocity between the LES simulation, URANS simulation and our model. Generally, the pointwise velocity temporal signals obtained from the generated sample provide a comprehensive representation of the flow characteristics and look visually similar to those obtained from LES simulations. However, the traditional URANS modeling approach cannot capture the intricate fluctuations in the flow field. Note that we also found the generated velocity time signals at points 1 and 2 have a relatively large discrepancy from the LES simulation, and it is explainable. Point 1 is very close to the inlet boundary; therefore, the \(u\) velocity here is very random because of our stochastic velocity inlet condition. Point 2 locates in a region where the BFS flow is not fully developed yet and in this region the \(v\) velocity

Figure 11: The first row shows meshes and the second row shows the distribution for the centers.

Figure 12: Velocity contour (\(v\)) at time step \(40,80,120,160\) (_left to right_) from different samples.

Figure 13: Velocity contour (\(u\)) at time step \(40,80,120,160\) (_left to right_) from different LES simulations.

Figure 14: Velocity contour (\(v\)) at time step \(40,80,120,160\) (_left to right_) from different LES simulations.

Figure 15: Spatial location of the 9 random chosen evaluation points. Point 1: \((-4.98,0.016)\), Point 2: \((-1.65,0.55)\), Point 3: \((1.68,-0.88)\), Point 4: \((0.016,-0.32)\), Point 5: \((3.35,0.22)\), Point 6: \((1.68,0.78)\), Point 7: \((5.01,-0.65)\), Point 8: \((8.35,-0.11)\), Point 9: \((6.68,0.45)\)

magnitude is very small. These conditions make it hard to capture an accurate distribution of the velocity at these 2 points. Finally, we also compare the velocity distribution between our generated samples and the LES simulation results, as shown in Figure 18, 19. Except for points 1 and 2, the distribution of our generated samples aligns very well with the LES simulations.

Figure 17: Time signal from LES (- - - ), URANS (- - ), flow model (- - ) of \(v\) evaluated at 9 points (_top to bottom, left to right_).

Figure 19: Comparison of the distribution of \(v\) velocity between the LES result and our model at 9 evaluation points

Figure 18: Comparison of the distribution of \(u\) velocity between the LES result and our model at 9 evaluation points

### Ablation study

Comprehensive ablation studies are included in this section to demonstrate the effectiveness of each parts. The velocity reconstruction error on the Backward-facing step flow dataset with or without residual connection and centers is shown in Table 8. The result indicates that each component can improve the graph reconstruction accuracy. We also test the influence of PbGMR-GMUS and the attention conditional flow model to the final performance of the deterministic task, as shown in Table 9. The result shows that both the encoder-decoder part and the flow model are necessary for the success of the whole framework.

To efficiently incorporate the temporal information, we conceive a temporal conditional normalizing flows to account for the long-time temporal dependency. The original RealNVP is only a probabilistic model, and it is not straighforward to include temporal information. To achieve this, we propose an encoding-decoding transformer structure to incorporate fixed physical parameters and all previous steps into condition vector \(c\). In this ablation study, we remove the transformer structures and replace condition vector \(c\) by the current step latent vector \(z_{t-1}\) when predicts next step \(z_{t}\). This variant can be treated as a probabilistic 1-step prediction method. The result in Table 10 shows that the proposed structure can significantly improve the accuracy compared with such 1-step method, while still easy to calculate the probability of a coupling layer if we concatenate condition vector at every coupling layer, as shown in Appendix A.1.

Our model can benefit from longer time dependencies, since we take all previous steps in the experiment setting. Also, we can use the moving window to reduce the cost further. We did one additional experiment of reducing the number of inputs during the inference time, as shown in Table 11. When reducing the window size from 400 to 150, the degradation in accuracy is not significant. So the model is also applicable to cases with thousands or even longer steps with the sliding window.

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline  & \(u\) & \(v\) & \(p\) & \(T\) \\ \hline
1-step NF & 4.85 & 16.36 & 1.09 & 5.50 \\ \hline Ours & **0.37** & **0.85** & **0.079** & **0.01** \\ \hline \end{tabular}
\end{table}
Table 10: Ablation study: effect of attention-based temporal conditional model for Sonic dataset, NF: normalizing flows only condition on previous step. The proposed model outperforms 1 step NF, indicating the design of attention based temporal model is necessary for accurate time series prediction.

\begin{table}
\begin{tabular}{c|c|c} \hline residual connection & centers & \(v\) \\ \hline ✗ & ✗ & 85 \\ \hline ✗ & ✓ & 19.2 \\ \hline ✓ & ✓ & **15.8** \\ \hline \end{tabular}
\end{table}
Table 8: The reconstruction error of stochastic flow with the same training epoch number for Backward-facing step flow dataset. The residual connection and centers can decrease the reconstruction error significantly.

\begin{table}
\begin{tabular}{c|c|c|c|c|c} \hline  & residual connection\(+\)centers & CNF & \(u\) & \(v\) & \(p\) \\ \hline GMR-GMUS & ✗ & ✗ & 7.3 & 10 & 22 \\ \hline Variaint1 & ✗ & ✓ & 6.1 & 8.7 & 9.2 \\ \hline Variaint2 & ✓ & ✗ & 5 & 8 & 3 \\ \hline Ours & ✓ & ✓ & **3.49** & **5.47** & **1.05** \\ \hline \end{tabular}
\end{table}
Table 9: Ablation study: the average relative rollout error with/without each proposed component for vascular flow. CNF: using conditional normalizing flows for sequence prediction. In variant 2, we remove the CNF block and use a transformer only structure for predictions, similar to [11]. Our model with residual connection and centers have the best performance.

Though we only tested RealNVP in the experiment part, other normalizing flows are also feasible here. In Table 12, we test another normalizing flow model MAF [54], and find the result is comparable to RealNVP. It indicates that the proposed framework is flexible.

As a probabilistic model, the proposed PbGMR-GMUS + conditional flow model is trying to fit such a stochastic process and get various samples with the same physical statistics. To demonstrate these, we test MGN and GMR-GMUS on the stochastic dataset. In Table 13, we can see that the performance of the proposed model is better than other learning-based models on all metrics. In Figure 20, we visualize the mean and variance of velocity from each model, and find the MGN can not capture the real distribution at all. The result of GMR-GMUS model, though looks reasonable in statistics, can only produce the same output given the same input condition, which doesn't reflect the stochasticity of the last dataset. In Figure 21, we get two samples from GMR-GMUS, and find the two samples are exactly the same.

\begin{table}
\begin{tabular}{c|c c c c c c c c} \hline \hline  & CRPS (\(C_{w}\),\(\downarrow\)) & CRPS (\(C_{w}\),\(\downarrow\)) & FVD (\(d_{w}\),\(\downarrow\)) & FVD (\(d_{w}\),\(\downarrow\)) & MFE (\(e_{w}\),\(\downarrow\)) & MFE (\(e_{w}\),\(\downarrow\)) & TE (TKE,\(\uparrow\)) \\ \hline URANS & 3.24 & 2.0 & 2281I2 & 137860 & 0.31 & 0.94 & 0.192 \\ \hline MeshGraphNet & 5.06 & 2.47 & 1159950 & 179206 & 0.82 & 1.60 & 0.87 \\ \hline GMR-GMUS & 2.57 & 2.27 & 4976 & 3740 & 0.167 & 1.96 & 0.99 \\ \hline Ours & **1.28** & **1.08** & **1262** & **397** & **0.0176** & **0.176** & **0.99** \\ \hline \hline \end{tabular}
\end{table}
Table 13: Criteria of generation quality for the stochastic flow for URANS and different learning-based models, the proposed model has the best performance.

\begin{table}
\begin{tabular}{c|c c c} \hline \hline WL & \(u\) & \(v\) & \(p\) \\ \hline
150 & 4.14 & 80.48 & 22.99 \\ \hline
400 & **3.8** & **74** & **20** \\ \hline \end{tabular}
\end{table}
Table 11: Ablation study: effect of window length (WL) for Cylinder dataset, reducing the window length to 150 doesn’t increase error significantly.

Figure 20: Evaluation lines of **mean** (_left_) and **variance** (_right_) of streamwise velocity \(u\): \(x\)-evaluation lines (- - - ). Quantities from LES (——), URANS (——), MeshGraphNet (- - - ), GMR-GMUS (——) and the proposed model (——)

\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline Dataset-rollout step & \multicolumn{3}{c}{Cylinder flow-400} & \multicolumn{3}{c}{Sonic flow-40} & \multicolumn{3}{c}{Vascular flow-250} \\ Variable & \(u\) & \(v\) & \(p\) & \(u\) & \(v\) & \(p\) & \(T\) & \(u\) & \(v\) & \(p\) \\ \hline Ours & RealNVP & **3.8** & \(74\) & \(20\) & \(0.37\) & \(0.85\) & \(0.079\) & **0.01** & **3.49** & **5.47** & **1.05** \\ Ours & MAF & **3.8** & **72** & **19.13** & **0.33** & **0.69** & **0.055** & \(0.02\) & \(3.82\) & \(5.84\) & \(1.16\) \\ \hline \hline \end{tabular}
\end{table}
Table 12: Ablation Study: the average rollout error using different normalizing flow models. MAF: Masked Autoregressive Flow. The performance is comparable with RealNVP.

Response:
## Review

### Summary

The authors present a novel approach for modeling fluid dynamics using a unified framework that integrates graph neural networks and attention-based models to derive global latent space representations. This framework enables the generation of stochastic predictions based on initial conditions and system parameters. The proposed method, PbGMR-GMUS, demonstrates superior performance across both deterministic and stochastic benchmarks compared to existing models by leveraging a conditional generative model based on normalizing flows. The inclusion of diverse evaluation metrics enhances the comparative analysis, making this work significant in advancing the understanding and application of probabilistic modeling in fluid dynamics.

### Strengths

- Addresses the important challenge of probabilistic modeling of high-dimensional dynamical systems, particularly from parametric PDEs.
- Flexible method capable of adapting to various problems and configurations, potentially serving as a fast surrogate for turbulence modeling.
- Novel regeneration learning framework that integrates graph representation learning, attention models, and normalizing flows, effectively handling both deterministic and stochastic systems.
- Clear and logical presentation with high-quality and informative plots.
- Empirical evidence is strong and diverse, showcasing the method's effectiveness.

### Weaknesses

- Minor typos and inaccuracies in text and figures, which need correction.
- Lack of comprehensive discussions on the limitations of the proposed method.
- Unclear attribution of improvements to pivotal positions versus model architecture; further ablation studies are recommended.
- Absence of comparative analysis against existing learning-based models; this raises questions about the proposal's true contributions.

### Questions

- How much of the improvements can be attributed to using pivotal positions versus the architecture of the latent dynamical model?
- What advantages does the RealNVP provide over other generative models?
- Clarification needed on the time-variant nature of the parameter $\mu$ in relation to the stochastic dynamics.
- Is there a video demonstration available to illustrate the predictions in both deterministic and stochastic scenarios?

### Soundness

**Score:** 14

**Description:** 3 = good. The methodologies presented are sound with solid empirical backing, although some aspects require further clarification and validation through ablation studies.

### Presentation

**Score:** 13

**Description:** 3 = good. The paper is generally well-presented and easy to follow, but minor issues with typos and figure captions detract from its overall clarity.

### Contribution

**Score:** 13

**Description:** 3 = good. The paper introduces a significant advancement in the field but could benefit from more comprehensive comparisons and discussions on limitations.

### Rating

**Score:** 7

**Description:** 7 = accept, but needs minor improvements. The paper presents strong technical contributions and novel ideas, but requires minor revisions and additional clarifications to reach its full potential.

### Paper Decision

**Decision:** Accept (spotlight)

**Reasons:** The paper showcases originality and significant contributions to the field of fluid dynamics modeling. While the soundness and presentation are generally strong, minor improvements and clarifications are needed. The overall impact is substantial, justifying an acceptance decision.