Multiscale Graph Neural Network Autoencoders for Interpretable
  Scientific Machine Learning
The goal of this work is to address two limitations in autoencoder-based
models: latent space interpretability and compatibility with unstructured
meshes. This is accomplished here with the development of a novel graph neural
network (GNN) autoencoding architecture with demonstrations on complex fluid
flow applications. To address the first goal of interpretability, the GNN
autoencoder achieves reduction in the number nodes in the encoding stage
through an adaptive graph reduction procedure. This reduction procedure
essentially amounts to flowfield-conditioned node sampling and sensor
identification, and produces interpretable latent graph representations
tailored to the flowfield reconstruction task in the form of so-called masked
fields. These masked fields allow the user to (a) visualize where in physical
space a given latent graph is active, and (b) interpret the time-evolution of
the latent graph connectivity in accordance with the time-evolution of unsteady
flow features (e.g. recirculation zones, shear layers) in the domain. To
address the goal of unstructured mesh compatibility, the autoencoding
architecture utilizes a series of multi-scale message passing (MMP) layers,
each of which models information exchange among node neighborhoods at various
lengthscales. The MMP layer, which augments standard single-scale message
passing with learnable coarsening operations, allows the decoder to more
efficiently reconstruct the flowfield from the identified regions in the masked
fields. Analysis of latent graphs produced by the autoencoder for various model
settings are conducted using using unstructured snapshot data sourced from
large-eddy simulations in a backward-facing step (BFS) flow configuration with
an OpenFOAM-based flow solver at high Reynolds numbers.
Shivam Barwey, Varun Shankar, Venkatasubramanian Viswanathan, Romit Maulik
# Multiscale Graph Neural Network Autoencoders for Interpretable Scientific Machine Learning

###### Abstract

The goal of this work is to address two limitations in autoencoder-based models: latent space interpretability and compatibility with unstructured meshes. This is accomplished here with the development of a novel graph neural network (GNN) autoencoding architecture with demonstrations on complex fluid flow applications. To address the first goal of interpretability, the GNN autoencoder achieves reduction in the number nodes in the encoding stage through an adaptive graph reduction procedure. This reduction procedure essentially amounts to flowfield-conditioned node sampling and sensor identification, and produces interpretable latent graph representations tailored to the flowfield reconstruction task in the form of so-called masked fields. These masked fields allow the user to (a) visualize where in physical space a given latent graph is active, and (b) interpret the time-evolution of the latent graph connectivity in accordance with the time-evolution of unsteady flow features (e.g. recirculation zones, shear layers) in the domain. To address the goal of unstructured mesh compatibility, the autoencoding architecture utilizes a series of multi-scale message passing (MMP) layers, each of which models information exchange among node neighborhoods at various lengthscales. The MMP layer, which augments standard single-scale message passing with learnable coarsening operations, allows the decoder to more efficiently reconstruct the flowfield from the identified regions in the masked fields. Analysis of latent graphs produced by the autoencoder for various model settings are conducted using unstructured snapshot data sourced from large-eddy simulations in a backward-facing step (BFS) flow configuration with an OpenFOAM-based flow solver at high Reynolds numbers.

###### Contents

* 1 Introduction
* 2 Configuration and Dataset
	* 2.1 Governing equations and flow solver
	* 2.2 Flow configuration and mesh
	* 2.3 Description of dataset
* 3 Methodology
	* 3.1 Graph representation of flowfield data
	* 3.2 Encoding and decoding procedures
		* 3.2.1 Encoder
		* 3.2.2 Decoder
	* 3.3 Multiscale message passing (MMP) layer
		* 3.3.1 Message passing block
	* 3.4 Adaptive pooling and unpooling layers
		* 3.4.1 Top-K pooling and masked fields
		* 3.4.2 Unpooling
* 4 Results
	* 4.1 Effect of coarsening in the MMP layer
	* 4.2 Number of Top-K levels
### Analysis of reduction factor

###### Contents

* 1 Introduction
* 2 Background of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the theory of the theory of the of the theory of the theory of the of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the the theory of the theory of the theory of the theory of the the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the the theory of the theory of the theory of the the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the theory of the of the theory of the theory of the theory of the of the theory of the theory of the of the theory of the theory of the theory of the of the theory of the theory of the of the theory of the theory of the of the theory of the theory of the of the theory of the theory of the theory of the of the theory of the theory of the theory of the of the theory of the theory of the of the theory of the of the theory of the theory of the theory of the of the theory of the theory of the of the theory of the theory of the of the theory of the theory of the of the theory of the theory of the of the theory of the theory of the of the theory of the theory of the of the theory of the of the theory of the theory of the of the theory of the of the theory of the of the theory of the theory of the of the theory of the theory of the of the theory of the of the theory of the theory of the of the theory of the theory of the of the theory of the theory of the the theory of the the theory of the theory of the the theory of the of the theory of the theory of the theory of the the theory of the of the theory of the the theory of the the theory of the theory of the the theory of the the of the theory of the the theory of the the theory of the the the of the theory of the the the of the theory of the the the of the theory of the the theory of the the the of the theory of the the theory of the the the of the theory of the the the the the of the theory of the theory of the the theory of the the the the the of the theory of the the the the of the theory of the theory of the the the the the of the theory of the the the the the of the theory of the the the the the the of the the the the the of the the the the the of the the the the of the the the the the of the the the the the the the of the the the the the of the the the the the the of the the the the the the of the
metric configuration cannot be reliably extended to other configurations. Autoencoders based on convolutional neural networks (CNNs) are by design restricted to structured representations of the flowfield data. Similarly, ROM strategies reliant on MLP-based autoencoders are also often restricted to single geometric configurations, and the neural network parameters are obtained in such a way that does not allow for inference on unexplored or unseen meshes. Another limitation is model interpretability - in many studies, data-based ROM tools are essentially black boxes, rendering latent spaces uninterpretable. Ensuring interpretability in modeling frameworks often leads to (a) simplification and reductions in network complexity or expressive power (e.g. using a linear projection basis instead of a nonlinear one, for example [19, 25]), and/or (b) the development of other optimization strategies geared towards correlating the discovered latent dynamics with physical processes (i.e. disentanglement of latent variables [26, 27]), which adds additional computational expense and training constraints. Ultimately, the need for interpretation of the internal structures found in neural networks - particularly for applications like fluid flow prediction - has led to high modern demands for explainable artificial intelligence [28].

The goal of this work is to address both of these limitations - namely, interpretability and compatibility with unstructured meshes - by means of leveraging graph neural network (GNN) architectures for data-based model development. Within this scope, the main objective of this work is to show how the development of a novel graph autoencoder leads to highly interpretable latent representations. Before proceeding, some additional background and review on graph neural networks in the context of fluid dynamics modeling is provided below, as is a more concrete description of the individual contributions of this work.

Introduced in early works in Ref. [29, 30], graph neural networks have gained popularity in recent years due to their ability to combine the scalability of backpropagation-based optimization with flexible representations of data [31, 32]. The end-result is a framework for executing generalized classification and/or regression tasks on any class of problems that can be cast as a complex network or graph. The generalizability and ambiguity in graph representations of data lends to their overall strength and increasing usage in most modeling tasks: so long as a problem or dataset can be described as a set of nodes, with connections (edges) among nodes encoding some notion of relational distance or feature similarity, a GNN-based approach can be utilized [33]. To this end, modeling strategies based on GNNs leveraging message passing [34] and graph Laplacian [35] operations have produced state-of-the-art results on image classification, object detection, and node classification tasks. Additionally, due to their versatility, GNNs see significant success in multi-disciplinary applications including modeling protein folding [36], predicting the emergent properties of social networks [37], and modeling differential equations [38, 39].

Much of the success in these pioneering GNN applications has translated into new data-based model development strategies for fluid dynamics applications, with the primary goal of providing a mesh-agnostic way to accelerate conventional CFD flow solvers. Because CFD simulations are stored on mesh arrangements from the get-go with pre-defined stencils (node connectivities), a data-based flowfield regression framework fits naturally for these problems within the GNN paradigm. The key idea is that standard message passing strategies used within GNN regression frameworks are dependent on not only node features in isolation (node features can be pressure, velocity, or any other flow quantity of interest), but also the properties of the neighborhood of the nodes in question. Due to this property, if the input CFD mesh is considered as a set of points from which a graph is derived, several applications have shown how a GNN trained on one mesh can be readily extended to other meshes (or other geometries) without the need for re-training.

With this capability in mind, GNNs for fluid flow modeling have been deployed in a variety of contexts for both steady-state [40, 41, 42] and unsteady problems [43, 44]. In steady-state applications, GNNs are typically used to reconstruct flowfields from geometry-dependent fields such as signed distance functions. For example, Ref. [40] creates a surrogate solver for steady-state flows using a super-resolution strategy based on the graph convolution network [31]. Additional analysis into the role of graph pooling operations in the reconstruction procedure for steady-state flowfields was performed in Ref. [45]. From the perspective of unsteady fluid flow prediction, a key example of GNN-based modeling is the MeshGraphNet architecture introduced in Ref. [43], in which an encode-process-decode strategy is used to learn the source terms as spatial fields on unstructured grids for unsteady incompressible flow over cylinders and airfoils, among other applications. Inspired by methods used in multigrid methods [46], multi-scale GNN archittectures (a variant of which is leveraged in this work) have recently emerged to address limitations in single-scale counterparts by levaraging graph coarsening operations [42, 44, 47, 48].

Within the context of fluid flow prediction, although the above GNN-based predictive models succeed in extending conventional neural networks to unstructured grids and variable geometries, there are some lingering limitations. For example, most existing approaches leverage graph connectivities that are fixed in time. There has
been very little emphasis on utilizing learnable, adaptive graph pooling strategies within the GNN architecture - these adaptive pooling layers are essentially operations which condition the generation of coarse graph connectivities (adjacency matrices) on some input field stored on the nodes of a fine, or baseline, graph. Although adaptive pooling strategies have been recently explored [42], these applications are typically restricted to steady-state flows and utilize dense interpolation operators that cannot scale to large graph sizes. Further, much of the recent focus in GNNs has been geared towards purely forecasting operations - in other words, given a flowfield at some time \(t\), the objective is to learn the evolution at some future time \(t+\Delta t\). Although the direct forecasting task is indeed relevant, efforts into purely graph-based autoencoding (i.e. using a GNN to recover an identity map) have been sparsely explored. It is noted that recent work in Ref. [49] used graph convolution operations in an autoencoding context to good effect; however, the methods presented in this work are distinct in that the architecture is completely based on graphs, incorporating graph pooling and unpooling layers without any flattening operations.

The goal of this work is to address the above described limitations and gaps in the existing GNN literature for unsteady fluid flow modeling. The specific contributions are as follows:

* A GNN-based autoencoding architecture is developed with the purpose of ensuring latent space interpretability. This is accomplished by employing an adaptive pooling strategy known as Top-K pooling [50]. The Top-K pooling mechanism is considered as a type of a node sampling procedure, or adaptive sensing strategy, used to reduce the total number of nodes needed to minimize the target objective. The end-result is a latent space, or bottleneck layer, that can can be visualized in physical space directly, identifies coherent structures in the domain, and is described by an adjacency matrix that adapts in time with the evolution of the flow.
* A series of multi-scale message passing (MMP) layers are employed between pooling operations, each of which models information exchange among node neighborhoods at various lengthscales. The MMP layer, which augments standard single-scale message passing with learnable coarsening operations, allows the decoder to more efficiently reconstruct the flowfield from the identified regions in the latent graphs.
* The GNN autoencoder is demonstrated on CFD snapshot data sourced from unsteady flow over a backward-facing step at Reynolds numbers in the range of roughly \(20000\) to \(50000\). Analysis of latent spaces are conducted on these snapshots to demonstrate (a) interpretability and physical significance of latent spaces via masked field visualization, and (b) trade-offs between reconstruction accuracy, interpretability, and achievable compression produced by the latent graphs.

The remainder of the text proceeds as follows. The dataset generation procedure, governing equations, and domain geometry are described in Sec. 2. An overview of the methodology is then provided in Sec. 3, which describes the graph generation procedure and architecture description. Autoencoding results, with emphasis on analysis of the interpretable Top-K pooling output, is then provided in Sec. 4. Concluding remarks follow in Sec. 5.

## 2 Configuration and Dataset

### Governing equations and flow solver

The data used in this study comes from unsteady, 2-dimensional large-eddy simulations (LES) of the incompressible Navier-Stokes equations of flow over a backward-facing step (BFS). Before proceeding, it should be noted that the application of LES in this scenario as a modeling approach is not entirely physical due to the usage of a 2-dimensional configuration; rather, turbulence modeling was applied here as a mechanism to produce coherent structures in flowfields that are reasonably complex and unsteady in nature for GNN demonstration purposes. As such, efforts to tune the LES model for physical consistency (i.e. consideration of near-wall regions) were not carried out.

More specifically, the governing equations in the LES formulation are, from the conservation of mass and momentum in incompressible flow [51],

\[\begin{split}\frac{\partial\widetilde{u}_{j}}{\partial t}+\frac{ \partial\widetilde{u}_{i}\widetilde{u}_{j}}{\partial x_{i}}&=\nu \frac{\partial^{2}\widetilde{u}_{j}}{\partial x_{i}\partial x_{i}}-\frac{ \partial\tau_{ij}^{r}}{\partial x_{i}}-\frac{1}{\rho}\frac{\partial\widetilde{ p}}{\partial x_{j}},\\ \frac{\partial\widetilde{u}_{k}}{\partial x_{k}}&=0. \end{split} \tag{1}\]
For the general time-evolving field \(f(x,t)\), the spatial filtering operation, denoted \(\widetilde{\cdot}\), is defined as the convolution

\[\widetilde{f}(x)=\int f(x^{\prime},t)G(x,x^{\prime})dx^{\prime}, \tag{2}\]

where the kernel \(G\) is called the filter. In this work, the filtering is performed implicitly - the flow variables stored on the computational grid are interpreted as the output of a box-filtering operation of filter size specified by the grid resolution [51]. In Eq. 1, the quantity \(\widetilde{u}_{j}=\widetilde{u}_{j}(x,t)\) denotes the time-evolving \(j\)-th component of the filtered velocity, \(\widetilde{p}=\widetilde{p}(x,t)\) is the time-evolving filtered pressure modified to include the isotropic component of the residual stress tensor, \(\nu\) is a constant kinematic viscosity, \(\rho\) is a constant density, and \(\tau^{r}=\tau^{r}(x,t)\) is the deviatoric component of the residual, or sub-grid scale (SGS), stress tensor, defined as

\[\tau^{r}_{ij}=\tau^{R}_{ij}-\frac{1}{3}\tau^{R}_{ii}\delta_{ij}, \tag{3}\]

where \(\tau^{R}_{ij}=\widetilde{u_{i}u_{j}}-\widetilde{u_{i}u_{j}}\). Because the residual stress tensor in Eq. 3 cannot be determined due to the presence of the filtered nonlinear term \(\widetilde{u_{i}u_{j}}\), a model is needed close the partial differential equations in Eq. 1. There are many pathways available to this end - in this work, the standard Smagorinsky model is used [52], which casts the residual stress \(\tau^{r}_{ij}\) as a quantity proportional to the filtered rate-of-strain as

\[\begin{split}\tau^{r}_{ij}&=-2\nu_{r}\widetilde{S} _{ij},\\ \nu_{r}&=\left(C_{S}\Delta\right)^{2}|S|.\end{split} \tag{4}\]

In Eq. 4, \(S_{ij}\) is the filtered rate-of-strain, \(|S|=\sqrt{2S_{ij}S_{ij}}\) is its magnitude, \(\nu_{r}\) is the turbulent eddy-viscosity, \(C_{S}\) is the Smagorinsky constant (\(C_{S}=0.168\) here), and \(\Delta\) is the box-filter width.

The open-source library OpenFOAM is used to implement the numerical schemes needed to solve Eq. 1 within the finite-volume framework. OpenFOAM is also used here to produce the underlying computational mesh of the flow configuration (the mesh and configuration geometry is described in Sec. 2.2). The distribution of the OpenFOAM library contains a number of pre-packaged flow solvers used for the purposes of treating unsteady incompressible flows over complex geometries that are also compatible with a set of LES models, one of which is the Smagorinsky model described above. Here, the baseline pimpleFOAM solver is used with globally second-order numerics and one outer iteration. The central theme in the pimpleFOAM solver is to treat the effects of the pressure gradient term during a single time step with an operator splitting strategy resembling a predictor-corrector approach [53, 54]. Full details on the numerical schemes are out-of-scope here; the authors instead point the reader to Ref. [55] for more information on the specifics of OpenFOAM numerics. In the end, the user must describe the spatial numerical schemes, temporal schemes, and LES model parameters in a set of input files. These files -fvSchemes, fvSolution, and momentumTransport respectively - are provided in the supplementary material. In the remainder of the paper, for notational convenience, the filter notation \(\widetilde{\cdot}\) used to describe velocity fields is dropped without loss of generality.

### Flow configuration and mesh

With the governing equations and flow solver described above, a general schematic of the BFS configuration is shown in Fig. 1. This configuration, along with the similar double-BFS (or cavity) configuration, has been used in a number of experimental [56, 57] and numerical [58, 59] studies as a benchmark model problem for flows containing separation and re-attachment phenomena dominated by turbulent mixing, recirculation zones (as well as transition to turbulence), and vortex shedding. Flow enters from an inlet on the left and propagates through an initial channel of fixed width upon encountering a step anchor, triggering flow separation upon entering a second channel of larger width. The step size, \(L_{s}\), is a key parameter in this configuration as it controls primary unsteady flow quantities of interest such as recirculation zone sizes observed in the cavity formed behind the step, vortex shedding frequencies downstream, and shear layer dynamics separating the cavity and freestream region. Values for the geometric quantities used in the simulation procedure are also provided in Fig. 1. During the simulation procedure, velocity at the inlet is prescribed in the x-direction to satisfy a Reynolds number within the range \([26214,45589]\), with characteristic length-scale based on the step height \(L_{s}\). Additionally, for stability purposes, an initial startup region is used (indicated in red in Fig. 1) to provide a uniform velocity profile to the channel. In this startup region, slip boundary conditions are used at the walls; no-slip conditions are used at all other boundaries except for the inlet and outlet.
Although the flowfield data were obtained from runs in the full BFS geometry, for the purpose of dropping training times, data used to train the GNNs come from a subdomain (the gray cropped region in Fig. 1) containing all flow dynamics of interest. Note that no interpolation procedures or mesh changes were utilized when extracting this cropped region - rather, a subset of cells were selected using the cellSet functionality in OpenFOAM such that only regions near the step were captured. A visualization of the cropped mesh used in this study is shown in Fig. 2(a), with a zoom-in on the near-step region in Fig. 2(b). The mesh, generated using the blockMesh utility in OpenFOAM, contains 14476 control volumes (cells) and is characterized by highly refined regions near walls and in a region behind the step such that (a) boundary layers are reasonably resolved, and (b) unsteady dynamics in the near-step fan, such as flow separation and emergence of recirculation zones, are properly captured. Mesh resolution increases as the domain progresses to the outlet and exits the near-step region - this coarsening effects continues outside of the cropped region in the x-direction. This variation in cell resolution \(\Delta\) is shown in Fig. 2(c) across the two perpendicular lines indicated in Fig. 2(a). The mesh in Fig. 2 contains noticeable variations in cell sizes and skewness, and is therefore a sufficiently practical benchmark to assess GNN capabilities in complex geometric configurations.

### Description of dataset

Using the BFS simulation configuration, datasets used for training and testing the GNN models described in Sec. 3 consist of a collection snapshots describing the evolution two-component velocity fields in the cropped mesh shown in Fig. 2 at various Reynolds numbers. An instantaneous snapshot in this context is given by the matrix \(\mathbf{X}(t)\in\mathbb{R}^{N_{C}\times N_{F}}\), where \(t\) denotes the time, \(N_{C}\) the number of cells or control volumes (\(N_{C}=14667\)), and \(N_{F}\) the number of features (\(N_{F}=2\) here for streamwise and vertical velocity components). The rows of each snapshot \(\mathbf{X}(t)\) describe individual cell centroid locations that are fixed in physical space as provided by the mesh configuration. The columns indicate the flowfield component, or feature, stored at the corresponding cell index in accordance with the finite volume formulation (Fig. 3 shows a visualization of the cell centers).

The complete dataset consists of 5 simulation trajectories, each of which comes from sampling flowfields at a fixed time interval of \(\Delta t=10^{-4}\) seconds at the various Reynolds numbers described in Table 1. Note that although the sampling interval is fixed, the simulations themselves were run in a variable timestep setting based on a maximum Courant-Friedrichs-Lewy (CFL) number of \(0.5\). As shown in Table 1, the training data is comprised 3 out of the 5 total trajectories corresponding to Reynolds numbers of 26214, 32564, and 39076 respectively, producing a total number of \(1216\) training snapshots. The testing data consists of the remaining 2 trajectories: one at an interpolated Reynolds number with respect to the training data range (Re=29307), and the other at a larger, extrapolated Reynolds number (Re=45589) to provide a more challenging GNN evaluation task.

Figure 3 shows the time evolution of the streamwise velocity component at the specified near-step probe locations for all five cases in Table 1. The velocity signals display highly unsteady decaying periodic behavior indicative of turbulent vortex shedding cycles observed in the BFS configuration at these high Reynolds numbers. These trends ultimately illustrate both the complex dynamics characteristic this configuration, as well as the appreciable difference in dynamics observed in the set of training and testing trajectories. For example, in accordance with the increased Reynolds number, significant increases in velocity magnitude and shedding cycle frequency is seen for Trajectory 5, which is included in the testing set.

Figure 1: Schematic of the backward-facing step configuration (not shown to scale) using an inlet-to-step length ratio of \(7.9\). Lengths are normalized by the step height \(L_{s}\), and green circle marks the origin. The mesh for the cropped region indicated in this figure is shown in Fig. 2.
These periodic trends in velocity signal are linked to the continual emergence of recirculation zones in the near-step region, causing the re-attachment point for the separated shear layer induced by the step to oscillate along the bottom wall in the x-direction. A qualitative comparison of this re-attachment cycle is shown in Fig. 4 for trajectories 1 and 5, which are contained in the training and testing sets respectively. The figure displays a series of instantaneous streamwise velocity snapshots, each separated by a time of \(10^{-3}\) seconds, to illustrate the variation in the separation-reattachment process due to the differences in Reynolds number.

The snapshot sequences illustrate how the cycle is initiated when the reattachment point experiences a "pinch-off" effect caused by the emergence of a recirculation zone near the step corner, which in turn advects a coherent region of negative streamwise velocity downstream. The pinch-off time in the cycle occurs earlier as Reynolds number increases - in Fig. 4, this corresponds to snapshot 4 in trajectory 1 versus snapshot 2 in trajectory 5. Additionally, as the pinch-off phenomenon terminates, the cycle concludes when the flowfield encounters a similar condition as the starting point, corresponding to a more uniformly reattached flowfield near the step. As implied through the oscillatory nature of the velocity signals in Fig. 3, the visual comparison in Fig. 4 shows how the increase in cycle frequency in the higher Reynolds number cases is paired with a proportional decrease in the reattachment length. As shown in Fig. 5, drop in reattachment length is identified by stronger recirculation zones near the step caused by increased magnitudes of adverse pressure gradients observed in the flow.

The above comparisons in training and testing trajectories, along with flowfield visualizations, are provided to give confidence to the fact that the key physical quantities of interest characterizing flow over backward-facing steps (e.g. reattachment, downstream boundary layer development, emergence of recirculation zones in step corner) are correctly captured in the simulation procedure. Although additional effort can be undertaken in the

\begin{table}
\begin{tabular}{||c c c c||} \hline Trajectory & Re & Snapshots & Category \\ \hline \hline
1 & 26214 & 434 & Training \\
2 & 32564 & 386 & Training \\
3 & 39076 & 396 & Training \\
4 & 29307 & 378 & Testing \\
5 & 45589 & 405 & Testing \\ \hline \end{tabular}
\end{table}
Table 1: BFS simulation trajectories used to populate datasets for the graph neural network training and evaluation.

Figure 2: **(a)** Cropped mesh of the BFS configuration shown to scale. Red lines indicate lines along which resolutions are computed in (c). Green circle denotes origin. **(b)** Zoom-in of the mesh highlighting refinement near the step. **(c)** Mesh resolution \(\Delta\) normalized by the step height \(L_{s}\) along lines shown in (a).
Figure 4: Visualization of BFS shedding cycles via streamwise velocity evolution for trajectory \(1\) (top, Re=26214) and trajectory \(5\) (bottom, Re=45589). Columns display instantaneous snapshots separated by a time of \(10^{-3}\) seconds.

Figure 3: **(a)** Red markers indicate probe locations in the BFS configuration used to plot trajectories in (b) and (c). Black points mark control volume centroids. **(b)** Evolution of pressure for probe locations depicted in (a). Data collection phase occurs after roughly \(t=0.14\) s. Note that pressure here can be negative because the quantity is shifted by an arbitrary reference. Pressure has also been normalized by the constant density \(\rho\). **(c)** Visualization of training, validation, and testing datasets using pressure evolution at the probe locations in (a). Bold lines isolate single trajectories for clarity.
modeling procedure to establish consistency with experimental and direct numerical simulation (DNS) studies at the chosen operating conditions, such analyses are deemed out-of-scope in the context of demonstrating the graph neural network capabilities described in Sec. 3 and 4.

## 3 Methodology

The graph neural network (GNN) autoencoder architecture is shown in Fig. 6. The various layers shown in the schematic are organized into encoder and decoder segments that operate on graph representations of the unstructured flowfield data. The GNN forward pass can expressed concisely through the actions of the encoder and decoder via

\[G_{L} =\mathcal{E}(G_{0}), \tag{5}\] \[\widetilde{G}_{0} =\mathcal{D}(G_{L}), \tag{6}\]

where \(\mathcal{E}\) and \(\mathcal{D}\) are the graph-based encoder and decoder, respectively. As shown in Fig. 6, the encoder \(\mathcal{E}\) takes as input a graph \(G_{0}\). The encoder output - referred to as the _latent graph_\(G_{L}\) - is conditioned on both the input graph quantities (e.g. nodes and edge attributes) and the parameters used in the encoder layers. The decoder \(\mathcal{D}\) takes as input the latent graph \(G_{L}\) and produces the output graph \(\widetilde{G}_{0}\); the decoding evaluation is similarly conditioned on a separate set of layer parameters.

Analogous to conventional autoencoding goals, the objective of the graph autoencoder is (a) to obtain a latent graph \(G_{L}\) that reduces dimensionality of the input graph \(G_{0}\), and (b) to ensure, through an data-based optimization or learning procedure, that the parameters contained in \(\theta_{\mathcal{E}}\) and \(\theta_{\mathcal{D}}\) result in an accurate decoding or reconstruction (i.e. \(\widetilde{G}_{0}\approx G_{0}\)). In this work, the optimization procedure seeks to minimize the mean-squared error (MSE) between only the node attributes in the initial (\(G_{0}\)) and decoded (\(\widetilde{G}_{0}\)) graphs - definitions for these node attributes are provided in Sec. 3.1. An additional requirement central to this work is that of interpretability: the graph autoencoder should be constructed in such a way that the latent space (in this case, the latent graph \(G_{L}\)) is physically interpretable.

To this end, the main building blocks of the GNN architecture are multiscale message passing (MMP) layers and the graph pooling layers. The MMP layer (described in Sec. 3.3) models information exchange throughout the flow domain at multiple lengthscales based on input graph connectivity, analogous to convolution operations on

Figure 5: Velocity field comparison near step corresponding to frame 1 (beginning of shedding cycle) in Fig. 4 in trajectory 1 (top) and trajectory 5 (bottom). Fields are colored by velocity magnitude, with arrows displaying velocity vector orientation. Yellow circles denote center of recirculation zones, and red circles denote flow reattachment zones.
structured grids. The graph pooling layer (detailed in Sec. 3.4) is the procedure by which graph dimensionality can be reduced via reduction in the number of nodes. Emphasis here is placed on one particular graph pooling strategy known as Top-K pooling [60], which has been unexplored in recent GNN-based studies for unsteady fluid flows. A primary objective is to show how this Top-K pooling layer, through an adaptive graph reduction procedure that essentially amounts to flowfield-conditioned node sampling and sensor identification, produces interpretable latent graph representations tailored to the regression task at hand.

### Graph representation of flowfield data

Before presenting details of the message passing and pooling operations, the input graph generation procedure must first be described. The input graph \(G_{0}\) is defined as the tuple \(G_{0}=(\mathbf{V}_{0},\mathbf{E}_{0},\mathbf{A}_{0})\). The quantity \(\mathbf{V}_{0}\in\mathbb{R}^{N_{0}^{c}\times F_{0}^{v}}\) is the input node attribute matrix: a single row contains the \(F_{0}^{v}\)-sized feature vector of one of \(N_{0}^{v}\) total nodes in the input graph. In the applications considered here, these initial node features represent physical flowfield quantities (i.e. velocity fields sampled at a particular location in physical space), which eventually evolve into hidden node features of a potentially different size upon encountering nonlinear GNN layers during the encoding stage.

The quantity \(\mathbf{E}_{0}\in\mathbb{R}^{N_{0}^{c}\times F_{0}^{v}}\) is the input edge attribute matrix: a single row in \(\mathbf{E}_{0}\) describes the \(F_{0}^{v}\)-sized feature set of a directed edge in the graph. Along with its set of features, a directed edge is characterized by both a sender and receiver node index - these nodes are deemed "similar" in some sense based on the user-defined criteria used to derive the connection.

Accumulating these indices for all edges in the graph generates the adjacency matrix \(\mathbf{A}_{0}\in\mathbb{Z}^{N_{0}^{v}\times N_{0}^{v}}\). The adjacency matrix is used to facilitate arithmetic operations conditioned on the graph connectivity (e.g. the edge aggregation procedure described in Sec. 3.3). Put simply, if the rows of this matrix correspond to sender node indices and columns to receiver node indices, any directed edge can be represented as a value of unity in the corresponding location in the adjacency matrix with zeros everywhere else, such that the number of non-zero entries in \(\mathbf{A}_{0}\) recovers the total number of directed edges \(N_{0}^{c}\). Two quantities of interest derived from the adjacency matrix are the node neighborhood and node degree. The neighborhood of a receiver node \(i\), denoted \(N(i)\), is the set of sender node indices that share an edge; the degree of node \(i\), denoted \(d(i)\), is the cardinality of this neighborhood set (\(d(i)=|N(i)|\)). It should be noted that in practice, assuming the average node degree is significantly smaller than the total number of nodes, adjacency matrices are stored in a sparse matrix format to both ensure scalability with respect to \(N_{v}\) and avoid memory limitations.

Given the above definitions, the steps required to actually generate the input graph from the flowfield data described in Sec. 2.3 are (1) produce the set of nodes and corresponding node attributes from the raw data, and (2) construct the list of edges (graph connectivity) using some notion of similarity derived from either all or a subset of these node attributes. To motivate consistency with the finite-volume formulation of numerics utilized within

Figure 6: **(Top)** Encoder flowchart, showing the procedure by which the input graph representation of flowfields \(G_{0}\) (red nodes) is downsampled into the latent graph \(G_{L}\) (green nodes). Main building blocks are multiscale message passing layers (see Sec. 3.3) and Top-K pooling layers (see Sec. 3.4). **(Bottom)** Decoder flowchart, showing the procedure by which the latent graph \(G_{L}\) is upsampled to recover \(\widetilde{G}_{0}\), which contains predicted flowfield quantities at the original graph node locations. Upsampling is achieved using an unpooling layer described in Sec. 3.4.
the OpenFOAM framework, this work adopts graph connectivity using the latter pathway. As shown in Fig. 7, in this setting, graph nodes represent cell centroids. The node attributes on \(G_{0}\) are then readily initialized as

\[\mathbf{V}_{0,i}=\mathbf{X}(t_{i}),\quad i=1,\ldots,N. \tag{7}\]

where the subscript \(i\) is introduced as a snapshot index, and \(\mathbf{X}(t_{i})\) is the flowfield snapshot sampled at time \(t_{i}\) as described in Sec. 2.3. In the end, the number of nodes in the initial graph \(N_{0}^{v}\) is equivalent to the number of cells \(N_{C}\) in the mesh (see Fig. 2).

Directed edges between nodes are then instantiated based on shared cell faces. As such, edges that connect nodes in this formulation coincide with flux paths between cells. To prevent issues related to mesh skewness and variable mesh density during message passing operations, additional edges are appended to these initial edges by also establishing connections between nodes within a user defined radius of \(0.08L_{S}\). In other words, a radius-based connectivity is superimposed onto the finite volume connectivity, where for a given node, edges are created for all other nodes within the specified length scale. The number of nodes and edges in the initial graph after this procedure are \(N_{0}^{v}=14476\) and \(N_{0}^{e}=133739\) respectively.

In-line with previous approaches, the initial edge feature matrix \(\mathbf{E}_{0}\) is populated with the distance vector produced by the corresponding sender and receiver node indices. Note that due to the time-dependence of the flowfield \(\mathbf{X}(t)\), the node attributes in the input graph - the velocity fields at the cell centroids - are also time-evolving. However, because the physical space coordinates of the mesh cells are fixed, the edge attributes \(\mathbf{E}_{0}\) and adjacency matrix \(\mathbf{A}_{0}\)_in the input graph_\(G_{0}\) are fixed for all snapshots.

### Encoding and decoding procedures

The GNN evaluation procedure shown in Fig. 6 is broken down in the subsections below into encoder and decoder components in Sec. 3.2.1 and 3.2.2 respectively. With the general background provided here, the reader is directed to Sec. 3.3 and 3.4 for finer details on multiscale message passing (MMP) layers and graph pooling layers used within the architecture.

#### 3.2.1 Encoder

As implied by the subscript, the input graph \(G_{0}\) resides on the baseline "zeroth" level of the autoencoding architecture. The encoder produces a hierarchy of graphs at higher levels through successive message passing and Top-K pooling operations. The encoding procedure terminates upon encountering the latent graph \(G_{L}\), where \(L\) denotes the maximum level in the architecture (the "latent graph" is always referred to here as the graph produced at the highest level). Consistent with the notation introduced in Sec. 3.1, a graph at the \(l\)-th level is denoted

Figure 7: Graph generation procedure used in this work. Graph connectivity is consistent with the finite-volume formulation of CFD numerics. Although only quadrilateral cells are present in the mesh used here, schematics in shaded box show a single neighborhood for two different cell geometries to illustrate the approach. Red circle indicates root node, dashed lines indicate underlying cell, and solid arrows indicate directed edges that intersect cell faces.
\(G_{l}=(\mathbf{V}_{l},\mathbf{E}_{l},\mathbf{A}_{l})\), and is characterized by its own set of nodes, edges, and adjacency matrix. For example, in the case of \(L=2\), graphs at three different levels are utilized in the architecture: \(G_{0}=(\mathbf{V}_{0},\mathbf{E}_{0},\mathbf{A}_{0})\) (the input graph at level 0), \(G_{1}=(\mathbf{V}_{1},\mathbf{E}_{1},\mathbf{A}_{1})\) (the level 1 graph), and \(G_{2}=(\mathbf{V}_{2},\mathbf{E}_{2},\mathbf{A}_{2})\) (the level 2 or latent graph).

Depending on the reduction factors used in the pooling operations, the goal of the encoder is to ensure that the number of nodes in the graph decreases as the number of levels increases, resulting in the following inequality:

\[N_{0}^{v}>N_{1}^{v}>\ldots>N_{L}^{v}, \tag{8}\]

where \(N_{l}^{v}\) (\(l=0,\ldots,L\)) denotes the number of nodes at the \(l\)-the level (equivalent to the number of rows in \(\mathbf{V}_{l}\)). The _global reduction factor_ (\(RF_{G}\)), an input parameter that characterizes the level of compression achieved in the architecture in terms of node reduction, is \(RF_{G}=N_{0}^{v}/N_{L}^{v}\). As will be seen in the results in Sec. 4, there are two important implications of the reduction factor: (1) a higher \(RF_{G}\) is expected produce a more difficult reconstruction task for the decoder, and (2) one can use different graph hierarchies (i.e. different values of the maximum graph level \(L\)) to arrive at a target \(RF_{G}\).

**Embedding node and edge features:** As demonstrated in Refs. [33, 43], feature space embeddings significantly improve predictive accuracy in GNN-based models. In this process, shown as the first step in Fig. 6, before encountering the message passing and pooling operations required to generate graphs at different levels, the node and edge features in the input graph \(G_{0}\) are modified. This step is carried out as \(\mathbf{V}_{0}\gets f_{\mathcal{E}}^{v}(\mathbf{V}_{0})\) and \(\mathbf{E}_{0}\gets f_{\mathcal{E}}^{e}(\mathbf{E}_{0})\), where \(f_{\mathcal{E}}^{v}\) and \(f_{\mathcal{E}}^{e}\) are independently-parametrized multi-layer perceptrons (MLPs) batched over the number of nodes and edges respectively in \(G_{0}\). These MLPs operate in feature space only, and therefore result in modified feature dimensionalities for the nodes and edges in the input graph \(G_{0}\) - the number of nodes and edges, as well as the graph connectivity, is not changed. As a result, after this feature encoding stage, the node and edge attributes reside in a hidden feature space of fixed size, typically larger than the feature space dimensionality used to initialize the original graph nodes and edges. The resulting node and edge features are referred to as hidden channels. For convenience, the hidden channel dimensionality in this work the same for node and edge attributes after this step (\(F_{0}^{v}=F_{0}^{v}\)), and is kept fixed during the forward pass until encountering the final node feature decoding operation to recover the desired reconstructed flowfields.

**Main encoding layers:**

As alluded to in the beginning of this section, the two layers that serve as the backbone of the GNN-based encoder are a multiscale message passing (MMP) layer and an adaptive graph pooling operation known as Top-K pooling. These are carried out in succession to move from a lower level graph \(l\) to a higher level graph \(l+1\) with fewer nodes, eventually terminating in the generation of the final latent graph at level \(L\). These operations are given by

\[(\mathbf{V}_{l},\mathbf{E}_{l}) \leftarrow\text{MMP}_{\mathcal{E},l}(\mathbf{V}_{l},\mathbf{E}_{l},\mathbf{A}_{l}),\quad l=0,\ldots,L, \tag{9a}\] \[(\mathbf{V}_{l+1},\mathbf{E}_{l+1},\mathbf{A}_{l+1}) \leftarrow\text{TopK}_{l}^{l+1}(\mathbf{V}_{l},\mathbf{E}_{l}, \mathbf{A}_{l}|\mathbf{p}_{l}),\quad l=0,\ldots,L-1, \tag{9b}\]

where \(\text{MMP}_{\mathcal{E},l}\) denotes a multiscale message passing layer present in the encoder \(\mathcal{E}\) at graph level \(l\), and \(\text{TopK}_{l}^{l+1}\) denotes the Top-K pooling layer that downsamples graph variables from level \(l\) to \(l+1\). Note that the MMP layer in Eq. 9(a) does not modify the graph connectivity \(\mathbf{A}_{l}\) - rather, it modifies node and edge features through a neighborhood aggregation procedure described in Sec. 3.3. On the other hand, the pooling operation in Eq. 9(b) does modify the connectivity through a node sampling procedure conditioned on the feature-wise projection vector \(\mathbf{p}_{l}\). This sampling mechanism usefully provides interpretability properties in latent graphs - details on this aspect are provided in Sec. 3.4.

#### 3.2.2 Decoder

The goal of the decoder is to upsample the latent graph \(G_{L}\) back into the original input graph dimensionality in terms of number of nodes and edges. The upsampling procedure is executed with a series of graph unpooling layers that mirror the pooling operations used in the encoding stage. In other words, starting from \(G_{L}\), the successive layer operations encountered in the decoding stage terminate upon arriving at the graph \(\widetilde{G}_{0}\) that shares the same number of nodes and edges as the input graph \(G_{0}\).

**Main decoding layers:** The decoding procedure is outlined in the following steps:

\[(\mathbf{V}_{l},\mathbf{E}_{l}) \leftarrow\text{MMP}_{\mathcal{D},l}(\mathbf{V}_{l},\mathbf{E}_{l},\mathbf{A}_{l}),\quad l=L,\ldots,0. \tag{10a}\] \[(\mathbf{V}_{l-1},\mathbf{E}_{l-1},\mathbf{A}_{l-1}) \leftarrow\text{unpool}_{l}^{l-1}(\mathbf{V}_{l},\mathbf{E}_{l}, \mathbf{A}_{l}),\quad l=L,\ldots 1. \tag{10b}\]
To mirror the encoding stage, the decoder first utilizes an MMP layer at graph level \(l\) in Eq. 10(a) before encountering the unpooling layer in Eq. 10(b), which serves to umpsample the node and edge feature matrices to the sizes consistent with those in level \(l-1\). The multiscale message passing layer utilizes identical operations as in the encoding stage (Eq. 9(a)), but contains different parameters (see Sec. 3.3). This combination of message passing and unpooling essentially amounts to a learnable interpolation operation between successive graphs in the Top-K hierarchy - additional detail on the unpooling procedure is provided in Sec. 3.4.

**Decoding node features:** To recover the final reconstructed flowfield at the nodes, after all message passing and unpooling steps are executed, the decoder utilizes a node-wise feature decoder at level 0 that reverses the feature embedding procedure executed in the first step of the encoding stage. This step is carried out as \(\tilde{\mathbf{V}}_{0}\gets f_{\mathcal{D}}^{\mathrm{c}}(\mathbf{V}_{0})\), where \(f_{\mathcal{D}}^{\mathrm{c}}\) is an MLP that operates feature-wise, serving to transform the node dimensionality from the hidden channel dimension back into the original dimension corresponding to the number of flowfield observables used in the dataset. The final predicted quantity used to compute the MSE loss function is then \(\tilde{\mathbf{V}}_{0}\), which contains the reconstructed flow quantities (velocity components) at the nodes.

### Multiscale message passing (MMP) layer

The notion of message passing was put forward in Ref. [34] as a general framework to unify a wide range of GNN modeling strategies. In any message passing approach, the goal is to provide a functional representation for the interaction between node attributes in a single neighborhood of the graph. Within a standard message passing layer, this interaction rule is formulated such that (a) its function parameters are shared throughout all neighborhoods in the graph, thereby ensuring a domain-agnostic modeling approach, and (b) it does not modify the input graph connectivity (adjacency matrix). Inspired by methods used in the multigrid community, recent work in GNN-based modeling has demonstrated that augmenting standard message passing operations with a hierarchy of coarse grids naturally results in more effective information propagation [44, 47]. These methods remove the need for modeling large lengthscale interactions with a very large number of message passing operations, which in turn reduces memory requirements during training and drops inference times.

As the name implies, the multiscale message passing (MMP) layers used in both encoding and decoding stages leverage this principle. A schematic of a single MMP layer is shown in Fig. 8. The MMP layer consists of a series of message passing blocks, each on a different coarsening level relative to the baseline mesh. Each message passing block contains a fixed number of single-scale message passing (MP) layers. After executing a message passing block, node and edge attributes are interpolated to a coarser graph level characterized by a larger inter-node lengthscale. This proceeds until the coarsest graph is reached, upon which graph quantities are interpolated back upwards to the starting point, utilizing skip connections for node and edge attributes from the downward pass along the way. It is emphasized that the coarsening levels in MMP layers are different from the concept of graph levels introduced earlier, which come from outputs of Top-K pooling operations - this distinction is made clear in Sec. 3.4.

The coarsening mechanism used in each MMP layer comes from a voxel-clustering algorithm, which has been demonstrated to work well in previous GNN applications [44]. Although other meshing approaches can be utilized (e.g. triangulations [47]), the advantage of the voxel clustering approach is that it directly takes a target message passing lengthscale as an input, and does not require an expensive optimization step in the coarsening procedure. In this approach, a graph is coarsened by overlaying a voxel grid in its physical space bounding box. Each cell in this voxel grid is interpreted as a cluster of fixed size corresponding to a target message passing lengthscale. Each voxel cell is identified by a parent node, which corresponds to its cell centroid location in physical space. Upon overlaying the voxel grid, underlying fine graph nodes can be assigned to representative voxel cells/clusters via computation of nearest centroids, establishing a parent-child relationship between two successive coarse and fine graphs in the MMP layer. Edges between parent nodes are added if underlying fine graph edges intersect their shared voxel cell face - the coarse edge attributes are instantiated using the average of attributes belonging to all fine edges that satisfy this intersection.

The parent-child ownership labels are then used to facilitate interpolation operations required to transfer node attributes between graphs at different coarsening levels. These interpolation procedures follow the methods introduced in Liao et al. [44]. In summary, for each coarse and fine graph pair, a new set of edges connecting children nodes to their respective parent node are established, forming an interpolation stencil for node attributes with learnable weights. This stencil is then used to learn coarse-to-fine and fine-to-coarse mappings for node attributes. The reader is pointed to Ref. [44] for more detail on the interpolation method.
Section 3.3.1 details the operations involved in a single message passing block. Before proceeding, it should be noted that the operations shown in Fig. 8 are equivalent to a graph U-net architecture. The difference here is that this U-net operation corresponds to a _single layer_ - an MMP layer - operating on one level in the full autoencoding architecture shown in Fig. 6. The effect of including multiple coarsening levels in MMP layers in both encoder and decoder stages will be analyzed in Sec. 4.

#### 3.3.1 Message passing block

A single message passing block in an MMP layer consists of a fixed number of uniquely parameterized single-scale message passing (MP) layers. The MP strategy used in this work comes from Battaglia et al. [33], which has been established in recent years as a well-grounded framework for graph-based modeling of complex physical systems related to fluid dynamics and other applications [43, 61]. The strategy, summarized in Fig. 9, is given by the following three steps:

\[\text{Step 1 (Edge Update):}\quad\mathbf{e}_{k}^{p}=f_{\varepsilon}^{p}( \mathbf{e}_{k}^{p-1}|\mathbf{v}_{s_{k}}^{p-1}|\mathbf{v}_{r_{k}}^{p-1}),\quad k =1,\ldots,n_{e}, \tag{11a}\] \[\text{Step 2 (Edge Aggregation):}\quad\mathbf{a}_{i}^{p}=\frac{1}{|N (i)|}\sum_{\{k:r_{k}=i\}}\mathbf{e}_{k}^{p},\quad i=1,\ldots,n_{v},\] (11b) \[\text{Step 3 (Node Update):}\quad\mathbf{v}_{i}^{p}=f_{v}^{p}( \mathbf{v}_{i}^{p-1}|\mathbf{a}_{i}^{p}),\quad i=1,\ldots,n_{v}. \tag{11c}\]

Superscripts in the above equations denote the MP layer index local to the message passing block, and vertical bars in function arguments denote concatenation operations. Additionally, \(\mathbf{e}_{k}^{p}\) denotes the \(k\)-th edge attribute vector sourced from the corresponding row in the edge attribute matrix, and \(\mathbf{v}_{i}^{p}\) is the \(i\)-th analogous node attribute vector - \(n_{e}\) and \(n_{v}\) denote the number of nodes and edges of the graph in question. In Eq. 11(a), the subscripts \(s_{k}\) and \(r_{k}\) present on the right-hand side denote the sender and receiver node indices for the \(k\)-th edge.

As shown in Fig. 9, the functions \(f_{e}^{p}\) and \(f_{v}^{p}\) are independent multi-layer perceptrons (MLPs) that act as nonlinear functions producing updated edge features and node features respectively. This allows the update procedures to both capture nonlinear interactions between edge and node attributes from previous layers, and also model complex feedback between individual nodes and their local neighborhoods. Note that the graph adjacency matrix is invoked only in Eq. 11(b) (the aggregation step); the summation is used to transform the edge-based representation of data into a node-based representation via reduction over the edge attributes in the neighborhood \(N(i)\) (i.e. \(\mathbf{a}_{i}^{p}\) is defined on the nodes).

For some graph \(G=(\mathbf{V},\mathbf{E},\mathbf{A})\), the MP layer from Eqs. 11(a)-(c) is concisely represented as

\[(\mathbf{V}^{p},\mathbf{E}^{p})\leftarrow\text{MP}^{p}(\mathbf{V}^{p-1}, \mathbf{E}^{p-1},\mathbf{A}). \tag{12}\]

Figure 8: Schematic of MMP layer used in the autoencoding architecture in Fig. 6. Message passing blocks (see Fig. 9) are shown in red, specifying number of MP layers per at each coarsening level. Message passing lengthscales \(L_{G}\) normalized by step size \(L_{S}\) are specified underneath respective graph images. Dashed lines indicate residual/skip connections for node and edge attributes. Note that these skip connections operate within _a single layer_ of either the encoder or decoder (see Fig. 6).
The \(p\)-th MP layer, denoted MP\({}^{p}\), updates both node and edge attribute representations without modifying the graph connectivity. Although not shown in Eq. 12, the message passing layer is parameterized by the weights and biases in the MLPs \(f_{e}^{p}\) and \(f_{v}^{p}\). As the size of a message passing block increases, the expressive capability of the GNN also increases at the trade-off of computational expense during training and inference stages. Unless otherwise noted, the number of MP layers in each message passing block is provided in Fig. 8, and parameters are not shared across message passing blocks within any MMP layer.

### Adaptive pooling and unpooling layers

As shown in Fig. 6, the result of an MMP layer operation described in Sec. 3.3 is provided to either Top-K pooling layers in the encoding phase, or unpooling layers in the decoding phase. This section provides relevant details on each of these layers.

Analogous to pooling operations in convolutional networks, the primary goal of graph pooling layers is to reduce the degrees-of-freedom of the underlying system being modeled via reduction in the number of nodes. The pooling strategy used in this work - known as Top-K pooling [60] - provides a unique take on the reduction process. Instead of achieving reduction via fixed graph coarsening, Top-K pooling layers achieve reduction using _adaptive node sampling_. In other words, the layer reduces the number of nodes by sampling a subset of nodes from the input graph. Node positions in the reduced graph coincide in physical space with nodes in the input graph, but the set of sampled node indices is a function of the input node attributes. As such, if these node attributes are time-evolving, the identified nodes in the reduced graph - as well as the connectivity - also change in time.

This difference in reduction philosophy has led to the Top-K layer being overlooked for the purposes of fluid flow modeling, as the reduced graph does not come from a direct coarsening operation. However, it is shown in this work how the Top-K layer, through the adaptive node sampling procedure, allows for built-in interpretability of the latent graph \(G_{L}\) (see Fig. 6). More specifically, the time evolution of latent graphs produced by the pooling operation can be visualized directly in physical space using _masked fields_. Because the node sampling procedure optimizes a regression task, visualization of this latent graph can be used to access regions in physical space relevant to this regression task. It should be noted that this interpretability quality resembles a data-based sparse sensing approach for feature identification, but the advantage here is that this formulation is (a) directly compatible with unstructured grids and complex geometries, and (b) is tailored by design to any user-defined regression problem. In the text below, the high-level details of the Top-K pooling and unpooling operations relevant to this work are described. For additional information on the specifics of Top-K operations, the reader is directed to Ref. [60].

Figure 9: **(Left)** Schematic of a message passing block consisting of two MP layers. **(Right)** Illustration of the edge and node update procedures within MP\({}^{p}\) for a single neighborhood. In the edge update (Eq. 11(a)), edge attributes are shown in black, sender node attributes in blue, and receiver node attributes in red. In the node update (Eq. 11(c)), node attributes are shown in red and aggregated edge features in black.
#### 3.4.1 Top-K pooling and masked fields

The Top-K pooling operation is given in Eq. 9(b), and is illustrated in the schematic in Fig. 10. Two parameters are required: a learnable projection vector \(\mathbf{p}_{l}\), and the number of nodes \(K\) to retain in level \(l+1\). In practice, \(K\) is recovered from a local reduction factor \(RF_{l}\) via \(K=N_{l}^{v}/RF_{l}\), where \(N_{l}^{v}\) is the number of nodes at input level \(l\). In a first step, the projection vector \(\mathbf{p}_{l}\in\mathbb{R}^{F_{l}^{v}\times 1}\) is used to transform nodes residing in the input \(F_{l}^{v}\)-dimensional feature space into a one-dimensional representation. In a second step, the projected node features are ranked in descending order and truncated such that the indices of the topmost \(K\) nodes are retained. These \(K\) node indices become "sampled" nodes, forming the node attribute vector \(\mathbf{V}_{l+1}\) on the reduced graph (i.e. \(K=N_{l+1}^{v}\)).

More formally, the initial projection step is given by

\[\mathbf{y}_{l}=\frac{\mathbf{V}_{l}\mathbf{p}_{l}}{\|\mathbf{p}_{l}\|}\in \mathbb{R}^{N_{l}^{v}\times 1}, \tag{13}\]

where \(\mathbf{y}_{l}\) is the projected vector. The node selection step is given by

\[\mathcal{I}_{l+1}=\text{rank}(\mathbf{y}_{l},K), \tag{14}\]

where \(\mathcal{I}_{l+1}\) is the set containing the node indices of the graph at level \(l\) corresponding to the topmost \(K\) quantities of the sorted projections in \(\mathbf{y}_{l}\). With \(\mathcal{I}_{l+1}\), an indexing operation can be carried out to recover the reduced graph quantities \(\mathbf{V}_{l+1}\), \(\mathbf{E}_{l+1}\), and \(\mathbf{A}_{l+1}\) in a sampling stage, thereby terminating the Top-K pooling layer. It should be noted that in order to ensure the projection vector \(\mathbf{p}_{l}\) can be trained using backpropagation, a gating function in the form of a sigmoid activation is performed on the downsampled node features. For details on this gating procedure, see Ref. [60].

**Interpretable masked fields:** Crucial to the GNN architecture in Fig. 6 is that the Top-K pooling operation is used to construct a hierarchy of _adaptive_ graphs (i.e. non-fixed adjacency matrices) - the node positions in physical space of the input graph \(G_{0}\) are fixed in time, but the node positions for levels \(l>0\) change in time. This is because, for a fixed projection vector \(\mathbf{p}_{l}\) obtained after training, indices of the sampled nodes at level \(l+1\) adapt to the

Figure 11: Illustration of the distribution operation (Eq. 15) used in unpooling. Red boxes denote interpolated values and white boxes denote zeros.

Figure 10: Top-K pooling illustration. **(a)** Two-component PCA visualization of input node attribute matrix \(\mathbf{V}_{0}\) associated with an instantaneous flowfield (markers denote individual nodes). **(b)** 1d projection of \(\mathbf{V}_{0}\) via Eq. 13. Red region indicates retained node indices corresponding to a reduction factor of 16. **(c)** PCA visualization of output node attribute matrix \(\mathbf{V}_{1}\) after sampling. Nodes are a subset of those shown in (a). **(d)** Masked field corresponding to sampled nodes in (c) – red region indicates where latent graph \(G_{1}\) is active.
flow features contained in the input node attribute matrix \(\mathbf{V}_{0}\), which are time-evolving. Since each of these indices corresponds to a particular location in physical space, the evolution of reduced graphs at levels \(l>0\), as well as the final latent graph at level \(L\), can be visualized via _masked fields_. The masked field, an example of which is shown in Fig. 10, is a visualization of the level index \(l\) of each cell in the physical domain. Utilized further in Sec. 4, these masked fields reveal the GNN's ability to (a) physically interpret latent graphs, and (b) identify evolving coherent structures in the domain intrinsically tied to the regression task (flowfield reconstruction).

#### 3.4.2 Unpooling

To facilitate graph decoding, the unpooling operation in Eq. 10(b) can be performed in the Top-K framework so long as \(\mathcal{I}_{l+1}\) is available. As illustrated in Fig. 11, the unpooling layer is interpreted as a data distribution operation that resembles delta-function interpolation. The distribution operation for node features is given by

\[\widetilde{\mathbf{V}}_{l}=\text{distribute}(\mathbf{V}_{l+1},\mathcal{I}_{l +1},\mathbf{0}_{l}). \tag{15}\]

In the above equation, the matrix \(\mathbf{0}_{l}\in\mathbb{R}^{N^{r}_{\lambda}\times F^{r}_{l}}\) contains only zeros - it is interpreted as an empty node matrix existing on level \(l\). The unpool operation "fills in" (or replaces) rows of the matrix \(\mathbf{0}\) with the corresponding rows of \(\mathbf{V}_{l+1}\) as per the indices contained in \(\mathcal{I}_{l+1}\). Edge attribute matrices \(\widetilde{\mathbf{E}}_{l}\) can be recovered in an analogous fashion, leading to the full unpooling operation of Eq. 10(b). Although the operation in Eq. 15 successfully recovers the original data dimensionality at level \(l\) (which is the primary goal of the unpooling layer), since skip connections are not used, many values in the upsampling node attribute matrix \(\mathbf{V}_{l}\) can be zero depending on the level of reduction (i.e. value of \(K\)). The role of the MMP layer after unpooling (see Fig. 6) is to treat this issue by efficiently distributing values in the unpooled graph to fill in empty nodes.

## 4 Results

In this section, analysis of the GNN-based autoencoder is performed for the flowfield reconstruction task from a variety of angles, with the primary scope aimed at showcasing the interpretability and reconstruction properties of generated latent spaces (or latent graphs). For all models discussed herein, the objective function used during training is the mean-squared error between input and reconstructed node feature matrices. Three-layer MLPs were used for node/edge attribute encoding and decoding, and two-layer MLPs were used for node and edge updaters in message passing operations. In all cases, MLPs utilize exponential linear unit (ELU) activation functions [62] with residual connections. Layer normalization [63] is applied after each MLP evaluation. Input and reconstructed node features consist of streamwise (\(u_{x}\)) and vertical (\(u_{y}\)) velocity components that are standardized using training data statistics.

GNN architecture development and training was performed using a combination of PyTorch [64] and PyTorch Geometric [65] modules. For training, flowfield snapshots obtained from trajectories 1-3 (see Table 1) were used, with a randomly selected 10% of these snapshots set aside as the validation set. During training, a batch size of 8 was used as per the graph-based mini-batching scheme of Ref. [65], and the Adam optimizer [66] with a learning rate scheduler based on validation loss thresholding was applied. All models were trained using a single Nvidia A100 GPU housed on a node on the Polaris high-performance computer at the Argonne Leadership Computing Facility.

The section proceeds as follows. In Sec. 4.1, the effect of including coarsening operations in the MMP layer on the latent graph and reconstructed fields is discussed. In Sec. 4.2, for a fixed reduction factor, analysis of changes in the masked fields due to variation in the number of Top-K levels \(L\) is conducted. Lastly, in Sec. 4.3, the effects of both the node reduction factor and the number of hidden node channels used to characterize the latent graph are studied from the perspectives of reconstruction accuracy and masked field evolution.

### Effect of coarsening in the MMP layer

To convey the impact of the coarsening operations present in the MMP layers (see Fig. 8), Fig. 12 presents mean-squared error histories and streamwise velocity field reconstructions at the probe locations for three different GNN architecture configurations:
- No Coarsening:** In this setting, the coarsening operations in the MMP layers are not used. More specifically, the MMP layer architecture shown in Fig. 8 is modified to neglect all clustering operations used to coarsen the input graph
- as such, message passing proceeds in the conventional fashion without coarsening. This effectively turns the MMP layer into a single-scale message passing block.
* Decoder Coarsening:** In this setting, coarsening operations present in the MMP layers in Fig. 8 are used _only in the decoder_\(\mathcal{D}\). In the encoder \(\mathcal{E}\), no coarsening operations are used.
* Encoder+Decoder Coarsening:** In this setting, the MMP coarsening operations are used in both encoder and decoder phases.

When coarsening is used (Models 2 and 3), the MMP layer is consistent with Fig. 8. When coarsening is not used, all coarse grids corresponding to higher lengthscales in Fig. 8 are ignored, rendering the MMP layer equivalent to a total of 4 standard single-scale message passing layers.

All three configurations utilize a maximum level of \(L=1\) (only one Top-K layer is used), a node reduction factor of \(RF_{G}=16\) (the number of nodes in the latent graph is dropped by 16x), and a hidden channel dimensionality of \(32\). Note that despite the fact that the number of nodes has decreased, the latent graphs here do not provide true compression due to the increase in node feature dimensionality. Assessment of true compression is delayed to Sec. 4.3. Using the above three configurations, the focus of this section is to instead demonstrate the need for coarsening operations in the MMP layer from the perspectives of reconstruction accuracy and masked field interpretability.

Since the optimization of parameters in the message passing layers and Top-K layers are coupled, the way in which nodes are subsampled to produce the latent graphs is expected to be dependent on the message passing scheme. To assess this effect quantitatively, training history for each of the above three model configurations is shown in Fig. 12(a). The impact of coarsening is evident in the loss function trends, as the converged errors for Model 1 (the configuration without coarsening in the MMP layer) shown in the blue curves in Fig. 12 is an order of magnitude higher than the configurations that include coarsening operations (Models 2 and 3). Further, the jump in accuracy provided by appending additional coarsening operations in the encoding stage (Model 2 to 3) is much lower than when appending coarsening operations in only the decoding stage (Model 1 to 2), implying that the contribution of reconstruction accuracy comes primarily from latent graphs produced using only decoder coarsening operations.

Figures 12(b) and (c) show how the convergence trends during training translate to streamwise velocity reconstructions on the unseen testing trajectories. Prediction trends are ultimately consistent with the loss function curves, in that Model 1 completely fails to capture dynamical information content in the near-step region at all tested Reynolds numbers. On the other hand, reconstructions produced by models that include coarsening are much more in-line with ground truth velocities - interestingly, Model 2 reflects more accurate flowfield reconstructions than Model 3 near the step, despite the fact that it utilizes coarsening operations only in the decoding stage. As will be seen in the qualitative comparisons below, the improved reconstructions provided by Model 3 come into play in the freestream regions, which is not reflected in the probe measurements shown in Fig. 12. Overall, these trends show how (a) coarsening operations are necessary to produce latent graphs that reliably reconstruct the flowfield, and (b) acceptable near-step velocity reconstructions at unseen Reynolds numbers, included extrapolated Reynolds numbers, are produced by autoencoders utilizing multiscale message passing.

The advantage of this autoencoding approach is that the physical characteristics of the reconstruction procedure, as well as the impact of including coarsening operations during message passing via the MMP layers, can be directly interpreted through a visualization of the identified latent graphs (\(G_{1}\) in this case) via masked fields. As such, to supplement the discussion surrounding Fig. 12, visualizations of the autoencoding procedure for a single input velocity field snapshot are shown in Fig. 13 for the same three model configurations. For each of the testing set trajectories (an interpolated and extrapolated Reynolds number), the input snapshot shown in Fig. 13 corresponds to step 1 of the reattachment cycle described in Fig. 4.

The masked fields, which indicate where in physical space the latent graphs are active, allow one to visualize locations in the domain corresponding to optimal flowfield reconstruction for the respective models. Note that because the cell sizes vary with spatial location, a fixed node reduction factor (16 for the models shown in Fig. 13) can result in different identified "volumes" in the masked fields depending the learned Top-K projection vector - for example, although the number of latent graph nodes is the same, the masked regions identified by Model 2 are smaller in physical space than those in Model 1 because the cell resolution in the identified region is accordingly
smaller. Upon visual inspection, a common ground in all three models is that the masked fields in Fig. 13 are characterized by disjoint clusters of identified nodes (red regions), which is in-line with the physical nature of the reattachment cycle described in the discussion surrounding Fig. 4. However, aside from this quality, the masked fields are markedly different for each of the model configurations, which is expected due to the different utilization of coarse grids during message passing.

The latent graphs identified by Model 1 (no coarsening), for example, identify larger coherent regions in the freestream region above the step than the other two configurations. This is in line with expectations in a single-scale message passing model: since the message passing scheme without coarsening is less efficient at distributing information from the masked region to the rest of the domain, it is understandable that the identified nodes in the latent graph occupy a larger coherent region in space. The disadvantage of the Model 1 configuration is evident when looking at the reconstructed flowfields, which directly show how the lack of multi-scale message passing results in a failure of the model to reconstruct the flow in regions away from the identified mask. Although adding additional message passing layers in a single-scale setting can address this issue, the computational costs and memory limitations during backpropagation render such approaches infeasible.

In contrast, the reconstructions produced by Models 2 and 3 are successful in recovering the primary features of the BFS flowfield, such as the re-attachment point and the propagating recirculation zone downstream - although reconstruction accuracy is still imperfect and there are lingering non-physical artifacts in the freestream, Fig. 13 shows how adding coarsening operations via MMP layers addresses the challenging task of recovering full flowfield information from the masked region.

Although both Models 2 and 3 are able to efficiently propagate information from the masked field, the masked fields themselves are fundamentally different. Model 2 (decoder coarsening only) recovers a masked region that identifies coherent structures consistent with dynamically active regions in the BFS configuration (shear layers, recirculation zones, downstream shedding). On the other hand, Model 3 shows how the inclusion of coarsening operations in the encoder MMP layers drastically alters the coherency of the masked field. Interestingly, the masked field in Model 3 itself resembles a coarse grid unlike the much more spatially coherent Model 2 counterpart. This implies that the inclusion of coarsening operations in the encoding stage effectively trades physical space coherency and interpretability for improved reconstruction accuracy. This is evidenced in the reconstructed fields at the extrapolated Reynolds number (bottom part of Fig. 13), for which Model 2 fails to reliably extrapolate to unseen freestream velocities when Model 3 succeeds. Despite this, even at extrapolated Reynolds numbers, Model 2 is still able to showcase the powerful ability to reasonably capture the primary BFS flow features while retaining the quality of coherent structure identification in the masked field. Additionally, it should be noted that across all three model configurations, the masked fields retain similar structure at the different Reynolds numbers shown in Fig. 13, which serves as a form of validation for the identified regions. The regions identified by Model 2 (decoder coarsening) in particular are almost identical at different Reynolds numbers for the snapshots shown.

The objective of this section was to demonstrate the need for coarsening operations in message passing layers in the decoding task, while also showcasing the primary interpretability property of latent graphs produced by

Figure 12: **(a)** MSE versus training epochs for Model 1 (blue curves), 2 (red curves) and 3 (green curves). Solid lines are training set evaluations, dashed lines are validation set. **(b)** Reconstructions of streamwise velocity (\(u_{x}\)) for Trajectory 4 (Re=29,307, see Table 1) at probe locations given in Fig. 3. Top plot is Probe 1, bottom plot is Probe 2. **(c)** Same as (b), but for Trajectory 5 (Re=45,589).
Figure 13: **(Top)** Autoencoding procedure for an instantaneous flowfield from Trajectory 4 (Re=29,307) using Models 1-3. Leftmost column shows input velocity fields. Corresponding rows show reconstructed fields. Topmost rows show masked fields, where cells colored in red indicate active nodes in latent graph. **(Bottom)** Same as top, but for a Trajectory 5 snapshot (Re=45,589).
the GNN autoencoder. The architectures utilized above were limited to single levels (\(L=1\)). As alluded to in Sec. 3, a beneficial property of this autoencoder is its ability to create a hierarchy of Top-K levels (\(L>1\)), which in turn leads to alternative representations of the masked fields and enables higher levels of flowfield compression. These aspects are discussed in the sections below. For the remaining analysis in the subsections below, all models configurations correspond to the Model 2 setting (decoder coarsening only) in light of its improved masked field coherency.

### Number of Top-K levels

The connectivity properties of the latent graph \(G_{L}\) produced by the encoder are not only dependent on the input flowfield, but also on the maximum level \(L\) used in the Top-K hierarchy. In other words, for a target global node reduction factor \(RF_{G}\), one can use either a single Top-K level to achieve this reduction (\(L=1\), which was used in Sec. 4.1), or a series of Top-K levels, each with smaller local reduction factors \(RF_{l}\). To this end, to isolate the effect of the maximum Top-K level \(L\) on the latent graph, Fig. 14 compares the previously discussed Model 2 from Sec. 4.1 for which one Top-K pooling operation was used (\(L=1\)), to a two-level (\(L=2\)) counterpart. The \(L=2\) model achieves the target global reduction factor of \(RF_{G}=16\) by accumulating two successive reductions that each drop the number of nodes by a factor of 4 (\(RF_{l}=4\)). Note that the \(L=2\) counterpart here still utilizes the Model 2 configuration described in Sec. 4.1, in that coarsening operations in the MMP layers are invoked only in the decoder.

To better illustrate the changes to the latent graph structure due to time evolution, shown in Fig. 14 are masked fields for two snapshots sourced from Trajectory 4, each at different stages in the BFS shedding cycle. Additionally, for ease of visualization, masked field plots are overlaid with velocity field orientation vectors to facilitate correlation of the identified coherent structures with recircualtion zones and other flowfield patterns.

In Fig. 14, for the \(L=1\) case, the masked field depicts a single identified sub-graph (red region) consisting of 16x fewer nodes than the baseline graph. On the other hand, in the \(L=2\) case, the model identifies a hierarchy of subgraphs in accordance with the Top-K procedure: the highest-level latent graph at \(l=2\), denoted by the blue regions in the corresponding masked field, also contains (a) 16x fewer nodes than the baseline graph (black regions), and (b) 4x fewer nodes than the identified graph at level 1 (red region). In other words, the masked field, through visualization of the identified level indices in Top-K pooling procedure, allows the user to interpret directly the subsampled hierarchy until the latent graph is reached (highest level index). Note that in the \(L=2\) case, the identified latent graph in blue is not disjoint from the intermediary graph at \(l=1\), but rather is a subset of the identified nodes at the preceding level.

Figure 14 shows how changes to the Top-K hierarchy modifies the identified latent graph structure, despite the fact that the number of nodes comprised in these latent graphs is the same. For example, in the \(L=2\) case, adding an additional level to the hierarchy allows the latent graph to occupy different regions in physical space that are slightly above the step cavity when compared to the \(L=1\) counterpart. This comes directly from the enabled exchange of information between successive levels during the autoencoding procedure when when more Top-K levels are used: in the \(L=2\) case, the identified graph at the first level (\(l=1\), red region) occupies a larger portion of the domain, enabling the latent graph to target regions of interest unreachable by the single-level model.

Interestingly, the subsampled nodes in the latent graph for the \(L=2\) case do not focus on the recirculation zone immediately near the vertical step wall - instead, focus is placed on downstream regions associated with vortex shedding and the reattached shear layer. When considering all levels together, however, qualitatively similar coherent structures are identified regardless of the parameter \(L\) used, giving confidence to the physical significance of the latent graphs at a target global reduction factor.

When assessing the velocity field patterns in combination with the masked fields in Fig. 14, it is evident that the masked fields are correlated with recirculation zones in the flow. For example, in the reattached state, three disjoint structures are visible in the masked field in accordance with the presence of three recirculation zones: a weaker zone near the vertical step wall, another near the reattachment point, and a third corresponding to a stronger shedding vortex approaching the outflow. Of these three regions, the latent graph in the \(L=2\) model places more importance on the latter two, implying that the recirculation zones away from the step are more crucial to the flowfield reconstruction task. This is consistent even at different time instances in the unsteady flowfield: for example, in the snapshot exhibiting the pinch-off phenomenon that instantiates the downstream shedding procedure (Fig. 14, right), two recirculation zones are present - as such, two large-scale coherent structures are
identified in both masked fields, with the \(L=2\) model again emphasizing regions slightly further away from the cavity.

### Analysis of reduction factor

The above sections showcased the architectural effects, in terms of coarsening operations utilized in the MMP layers (Sec. 4.1) and Top-K hierarchy size (Sec. 4.2), on the output latent graph without taking into consideration the potential pathways for compression. As such, the goal of this section is to outline effects of latent graph compression from two angles: (1) the impact of reducing the number of latent graph nodes via increases to the global node reduction factor \(RF_{G}\), and (2) the impact of reducing the number of hidden channels \(N_{H}\) stored on each latent graph node. It should be emphasized that both factors play into true compression achieved by the latent graph, as the total number of node degrees of freedom in \(G_{L}\) is \((N_{0}^{v}/RF_{G})\times N_{H}\), where \(N_{0}^{v}\) is the number of nodes on the input graph \(G_{0}\) (here, \(N_{0}^{v}=14476\)). Since the primary focus and novelty of this work is tied to demonstrating the interpretability properties of the latent graph via the masked fields, the two compression angles described above are analyzed from the perspective of both reconstruction accuracy and identified structures in the masked fields. In other words, the focus here relates to how the GNN autoencoder provides the user the ability to access the ways in which latent graphs change to achieve flowfield compression.

Figure. 15 displays normalized root-mean squared errors (RMSE) at various Reynolds numbers corresponding to the dataset trajectories described in Sec. 2.3. The RMSE for a single trajectory (Reynolds number) is given by

\[\text{RMSE}=\frac{\sqrt{\frac{1}{MN_{0}^{v}}\sum_{i=1}^{M}\sum_{j=1}^{N_{0}^ {v}}(x_{i,j}-\widetilde{x}_{i,j})^{2}}}{u_{in}}, \tag{16}\]

where \(x\) is a generic flowfield variable (e.g. \(x=u_{x}\) for streamwise velocity and \(x=u_{y}\) for vertical). In Eq. 16, \(x_{i,j}\) and \(\widetilde{x}_{i,j}\) denote the target and reconstructed flowfield variable respectively for the \(i\)-th snapshot and \(j\)-th node. Note that in Eq. 16 the number of snapshots \(M\) per trajectory varies, but the number of output graph nodes \(N_{0}^{v}\) is

Figure 14: **(Left)** From top-to-bottom: velocity magnitude field, masked field for \(L=1\) model, and masked field for \(L=2\) model. Snapshot depicts reattached state in Trajectory 4 (Re = 29,307) **(Right)** Same as left, but for a different snapshot in in in Trajectory 4 corresponding to the pinch-off phenomenon (see Fig. 4).
fixed. To better interpret errors across the range of Reynolds numbers contained in the data, the baseline RMSE is normalized by the inlet freestream velocity \(u_{in}\).

Figure 15(a) shows the effect of increasing the node reduction factor on the normalized RMSE of Eq. 16 for both streamwise and vertical velocity component reconstructions. To achieve increasingly higher amounts of node reduction in the latent graph, the maximum Top-K level in the GNN \(L\) is adjusted with a fixed local reduction factor of \(4\) between levels while freezing all other model hyperparameters (including the number of message passing layers). As a result, for the input graph containing \(N_{0}^{v}=14476\) nodes, the latent graph in the \(L=1\) model in Fig. 15(a) contains 4x fewer nodes, the \(L=2\) model contains 16x fewer nodes, and the \(L=3\) model 64x fewer nodes. Not that despite the fact that the number of nodes has been reduced by increasing \(L\), the hidden channel dimensionality of all models in Fig. 15(a) is fixed to \(N_{H}=32\) - as such, only the \(L=3\) model achieves true data compression in the sense of reduction in the total number of nodal degrees-of-freedom.

Two immediate trends are apparent in Fig. 15(a) - the first is that the errors are consistently higher for the streamwise component of velocity (\(u_{x}\)), and the second is that an increase in the node reduction factor via maximum Top-K level \(L\) results in a vertical shift in the RMSE curves for both velocity components. The former trend is expected, as a majority of the flow contribution to the reattachment cycle dynamics is contained in the streamwise component. This is also evidenced by observing that the vertical component errors are largely insensitive to Reynolds number, wheras errors for streamwise velocity across all configurations shown in Fig. 15 tend to increase with Reynolds number after about Re=32000. The latter trend conveys how it becomes increasingly more difficult for the decoder to recover the full flowfield as the number of nodes in the latent graph decreases. This is consistent with the fact that the same message passing scheme is used between all models (i.e. the MMP layer architecture is fixed as \(L\) is increased), which in effect makes it more difficult for the decoder to populate flow information on the original nodes from the latent graph as the size of the latent graph decreases. Although not shown here, it is reasonable to expect that modifying the MMP layer design to compensate for the reduction in nodes, either in the form of adding additional coarsening levels or increasing the size of message passing blocks, would lessen the degree of the vertical curve shifts.

To complement the notion of the achieving compression by dropping the number of nodes, Fig. 15(b) displays error trends for models that instead modify the hidden channel dimensionality \(N_{H}\) of the latent graph while fixing all other parameters. In particular, the autoencoders used to generate the curves shown in Fig. 15(b) come from taking the \(L=2\) case in Fig. 15(a) (i.e. 16x reduction in the number of nodes) and adjusting the latent node dimensionality. As such, the \(N_{H}=16\) case achieves a true compression factor of 2 and the \(N_{H}=8\) case achieves one of 4. Although the model with the largest hidden channel dimensionality (\(N_{H}=32\)) tends to drop the RMSE curve over the tested Reynolds number range, the RMSE behavior at smaller values of \(N_{H}\) is more complex - for example, the error in vertical velocity component reconstructions is actually higher for \(N_{H}=16\) when compared to \(N_{H}=8\), although the same is generally not true for the more dominant streamwise velocity component. Despite this, the results in Fig. 15 suggest that there is indeed an error-compression tradeoff from two angles: reduction in number of latent graph nodes and reduction in latent node dimensionality (hidden channels). As the scope of this work is directed towards demonstrating latent graph interpretability allowed by the architecture, further analysis of these error trends (i.e. in terms of compensating for error shifts with more message passing operations, increasing the number of input flow features, more rigorous hyperparameter testing, etc.) is omitted here and left for future reports.

To better visualize the effect of node reduction on the latent graph, Fig. 16 shows the time evolution of masked fields (alongside input flowfields) during one reattachment cycle in trajectory 4 for the same \(L=1\), \(2\), and \(3\) models plotted in Fig. 15(a). The figure illustrates how the adjacency matrix in the latent graph adapts to evolving flow structures during the shedding cycle, and also how increased reduction factor (compression) impacts the identified regions in the masked field to optimize reconstruction accuracy. The identified latent graph nodes (red regions in masked fields in Fig. 16) are significantly impacted by the maximum graph level \(L\) - in other words, as the number of latent graph nodes decrease, the large-scale regions and flow features in physical space identified by the masked fields change. For example, the \(L=1\) case (node reduction factor of 4) identifies the inlet and outlet boundaries, a majority of the freestream region, and also the structure of propagating recirculation zones in the flow. On the other hand, as a result of increasing the node reduction factor to 16, the latent graph in the \(L=2\) case eliminates concentration in the freestream regions, and instead focuses on the shedding structures in the step cavity (the physical significance of these identified regions were outlined in Sec. 4.2, for which the same \(L=2\) model was used). It is clear from Fig. 16 that identified structures propagate in accordance with the shedding cycle frequency. Note that although the number of latent graph nodes has dropped by a factor of 4x when moving from
the \(L=1\) to the \(L=2\) case, there is a disproportionate decrease in the "coverage" of the masked fields in physical space due to the fact that the mesh resolution in the near-step region is much higher. Interestingly, the \(L=3\) case - the model that achieves a node reduction of 64x and achieves a true compression in the degrees of freedom by a factor of 4 - isolates only the inlet and outlet boundary regions. Although a very small time-evolving structure near the step cavity is identified to capture the unsteadiness in the reattachment cycle (indicated by white circles in Fig. 16), the latent graph connectivity in the \(L=2\) case is largely insensitive to the evolving flow patterns. This implies that in cases of greater latent graph reduction, the Top-K mechanism isolates the inlet and exit boundary conditions as opposed to the step cavity region as the primary mechanism for flow reconstruction.

Figure 17 shows how these masked fields translate to flowfield reconstructions of varying accuracy in accordance with the amount of node reduction provided by latent graph. More specifically, shown in Fig. 17 are reconstructions for streamwise and vertical velocity components corresponding to snapshot 4 in Fig. 16, which depicts a moment just after the pinch-off phenomenon has occurred. Upon inspection, it is clear that the reconstruction accuracy - especially in terms of the presence of non-physical flow artifacts (e.g. discontinuities due to interpolation) - begins to drop as the maximum level \(L\) increases. Despite this, although deviation in the freestream region appears in the \(L=2\) model, the large-scale structures near the step are captured well. However, the \(L=3\) model begins to see significant deterioration in the reconstructed flow features near step - in particular, the gap between pockets of negative streamwise velocity components is not captured, as indicated by the black circled regions in the respective figures. Despite this, it is motivating that the latent graph even in the \(L=3\) case is able to recover the general structure of the BFS flow features in light of the fact that the identified nodes are largely concentrated away from the step and towards the boundaries.

Overall, the trends observed in Fig. 17 reflect those observed in Fig. 15(a), in that there is a direct deterioration in reconstruction accuracy as the amount of node reduction is increased. Although the reconstruction accuracy (particularly for the \(L=3\) case in Fig. 17) leaves something to be desired, it is emphasized that the advantage and objective of the autoencoding framework used here is the ability for the user to access and interpret the discovered latent graph through the visualization of time-evolving masked fields, as shown in Fig. 16. As mentioned above, a natural step forward is to explore the impact of including additional message passing operations in the MMP layer, as well as graph-based filtering operations, to offset the loss of reconstruction accuracy due to increased levels of node reduction. Such aspects will be reported elsewhere.

## 5 Conclusion

A graph autoencoder leveraging a combination of adaptive Top-K pooling layers and multiscale message passing (MMP) layers was introduced in this work. The pooling operation is interpreted as an adaptive sampling mechanism - stacking such layers results in a latent space (here referred to as a latent graph) that can readily be interpreted by visualizing the identified nodes through the construction of a masked field. The goal of the MMP layers is to then redistribute information contained in the masked field to the rest of the domain, resulting in a

Figure 15: **(Left)** Normalized RMSE (see Eq. 16) as a function of Reynolds number for streamwise (black) and vertical (blue) velocity field reconstructions. Curves denote different node reduction factors obtained via increasing maximum graph level \(L\) (refer to legend) for fixed \(N_{H}=32\). **(Right)** Analogous to left, but different curves denote different \(N_{H}\) values (refer to legend) for fixed \(L=2\).
Figure 16: Snapshots from Trajectory 4 depicting reattachment cycle dynamics. First two columns display streamwise (\(u_{x}\)) and vertical (\(u_{y}\)) velocity fields, and next three columns display masked fields sourced from the same L=1, 2, and 3 models used in Fig. 15(a). Red regions in masked fields denote identified latent graph nodes. Time evolution proceeds from top-to-bottom, and successive snapshots shown here are separated by a time of \(10^{-3}\) seconds. White circles in the \(L=3\) case indicate additional small identified region in masked field.

Figure 17: Flowfield reconstructions for the same L=1, 2, and 3 models utilized in Figs 15(a) and Fig. 16. Input flowfield corresponds to snapshot index 4 in Fig. 16. Colorbars are also consistent with Fig. 16.
reconstruction of the original input graph. Alongside providing details and nuances of the new architecture, the primary goal here was to showcase the interpretability properties of generated latent spaces (or latent graphs) produced in the encoding stage in fluid dynamics applications. To this end, datasets for training and evaluation purposes were generated using large-eddy simulations in a backward-facing step (BFS) flow configuration with an OpenFOAM-based flow solver at high Reynolds numbers. Using this dataset as testbed, important aspects related to GNN architecture design, physical interpretation of latent spaces, and flowfield reconstruction quality were analyzed.

From the angle of architecture design and to motivate the need for the MMP layer, the effect of coarsening operations present in the encoder and decoder message passing operations were analyzed. This analysis was conducted by comparing outputs of three model variations: (1) a GNN with no coarsening operations in the MMP layer (i.e. no multiscale message passing), (2) a GNN with coarsening operations present only in the decoder, and (3) a GNN with coarsening operations present in both encoder and decoder. In the end, the first model type resulted in a failure of the GNN to recover full flowfield information from the masked fields - incorporating coarsening operations via models 2 and 3 resulted in much more accurate and complete reconstructions, illuminating the advantages provided by multiscale message passing operations. The variation in architecture type directly impacted the physical nature of the masked fields: interestingly, the inclusion of coarsening operations in the encoding stage (model 3) was found to effectively trades physical space coherency and interpretability in the identified latent graph for improved reconstruction accuracy.

Physical interpretation of the masked fields at different stages of the reattachment cycle revealed how the latent graphs directly identify time-evolving coherent structures - in most cases, even in light of changes to the Top-K hierarchy (i.e. modification to the maximum level \(L\) for a fixed reduction factor), the nodes sampled in the reduction procedure were strongly correlated with recirculation zones of varying strengths in the flow. Flowfield compression potential was then assessed from two angles: by increasing node reduction factors through variations in maximum Top-K level, and by decreasing the hidden channel dimensionality of latent graph nodes. From both perspectives, it was found that the average reconstruction errors understandably increased across the board for greater compression factors. Although reconstruction qualities were imperfect in the higher-compression models, the advantage of this autoencoding approach is that it provides the user insight into how the greater levels of compression are achieved via visualization of the time-evolving masked fields. More specifically, it was found that the model achieving the highest amount of node reduction placed more focus on the inflow and outflow boundaries in the domain than other models. Additionally, when considering errors for individual flowfield variables at a given level of compression, reconstruction errors in the form of an RMSE measure were found to be much more insensitive to Reynolds number in the vertical velocity component when compared to streamwise velocity counterparts.

Because the scope of this work was tied primarily to demonstrating the latent space interpretability provided by the GNN autoencoder, there are many avenues for future work. Concerning the architecture parameters, investigating the effects of increased message passing and coarsening operations within the MMP layer is warranted, as it may potentially alleviate issues related to the increase in reconstruction error found in higher-compression models. Additionally, geometry extrapolation capability was unexplored in this work - assessment of latent graph properties in different geometric configurations is a promising direction to solidify the general applicability of the method. From the reduced-order modeling perspective, the concept of masked field evolution reveals new strategies for interpretable data-based surrogate modeling of complex fluid flow phenomena (e.g. by establishing quantitative connections between flowfield dynamics and masked field evolution). All of these aspects are actively being pursued, and will be reported in future studies.

## 6 Acknowledgements

This research used resources of the Argonne Leadership Computing Facility, which is a U.S. Department of Energy Office of Science User Facility operated under contract DE-AC02-06CH11357. RM acknowledges funding support from ASCR for DOE-FOA-2493 "Data-intensive scientific machine learning".
