#  Deep convolutional surrogates and degrees of freedom in thermal design

Hadi Keramati<EMAIL_ADDRESS>Feridun Hamdullahpur  Department of
Mechanical and Mechatronics Engineering, University of Waterloo
200 University Avenue, West Waterloo, Ontario, N2L 3G1, Canada

###### Abstract

We present surrogate models for heat transfer and pressure drop prediction of
complex fin geometries generated using composite Bézier curves. Thermal design
process includes iterative high fidelity simulation which is complex,
computationally expensive, and time-consuming. With the advancement in machine
learning algorithms as well as Graphics Processing Units (GPUs), we can
utilize the parallel processing architecture of GPUs rather than solely
relying on CPUs to accelerate the thermo-fluid simulation. In this study,
Convolutional Neural Networks (CNNs) are used to predict results of
Computational Fluid Dynamics (CFD) directly from topologies saved as images.
The case with a single fin as well as multiple morphable fins are studied. A
comparison of Xception network and regular CNN is presented for the case with
a single fin design. Results show that high accuracy in prediction is observed
for single fin design particularly using Xception network. Increasing design
freedom to multiple fins increases the error in prediction. This error,
however, remains within three percent for pressure drop and heat transfer
estimation which is valuable for design purpose.

###### keywords:

Geometric deep learning , Geometry processing , Heat exchanger , Design
freedom , Surrogate modeling

††journal: Elsevier

##  1 Introduction

Engineering design process often includes a series of numerical simulation to
explore multiple topologies with a wide range of applications in countless
industries [1, 2]. Design of heat transfer devices is more complex than
structural or aerodynamic design because of the additional convection-
diffusion equation and its pertinent boundary interaction [3]. High fidelity
simulation used to solve Partial Differential Equations (PDEs) is CPU-
intensive, particularly in presence of complex physics [4]. Since high
fidelity simulation is not important during the concept design process, a high
performance estimator can accelerate the design process to a great extent [5,
6]. Deep Learning (DL) and neural networks provide predictive models for a
wide range of applications. Physics-Informed Neural Network (PINN) and
feedforward neural networks, for example, showed promising results as
surrogate models to predict results of computer simulation [7, 8].
Convolutional Neural Network (CNN) is a class of DL which has gained
popularity in recent years with numerous applications in computer vision,
radiology, and most recently in CFD and design community [9, 10, 11, 12, 13,
14]. CNN can recognize features in variable 2D shapes which makes it an
appropriate model to predict simulation results of different physical shapes
[1, 6]. CNN also benefits from parameter sharing which leads to smaller number
of trainable parameters compared to Fully Connected (FC) Layers [15]. Guo et
al. [16] suggested CNN surrogate model to predict steady state laminar
velocity profile generated using Lattice Boltzmann Method (LBM). They used CNN
operating on Signed Distance Functions (SDFs) sampled on a 2D grid. Several
research studies have been carried out to accelerate time-dependent CFD
simulation [17, 18, 19]. Accelerated CFD models using CNN were introduced for
turbulent flow prediction without the presence of boundaries which results in
faster simulation even with fine grids [20]. These models have wide
applications in fluid flow prediction particularly in weather forecast that
there is no complex boundary condition or complex topology. Google Deepmind
reported a model for velocity and pressure prediction of two dimensional fluid
flow of a cylinder with 1-2 orders of magnitude faster computation than the
finite element solver [21]. Continuous convolutions are used for Lagrangian
fluid simulation without the presence of morphing shapes [22].

U-Net architecture was used as a tool to reconstruct the CFD results of
cylinder flow [23, 24]. Regular CNN was also used for pressure prediction in
flow around a cylinder [25]. CNNs were applied to two dimensional velocity
field estimation of blood flow in artificial lungs using a dataset with the
size of 5000 cases [26]. Generation of high fidelity velocity field from low
fidelity data was studied using GANs [27]. A data-driven model based on FC
layers for drag prediction of rectangular obstacle with different aspect
ratios representing building geometry was reported with as low as 3.17% error
in prediction [28]. Graph Neural Network has also shown promising results as a
fluid dynamics surrogate model in predicting next-step velocity profile for
time-dependent flow [29, 30, 31]. The effort in time-dependent accelerated CFD
is to minimize error accumulation that occurs during timesteps of the
estimation [32, 20]. Error accumulation in next-step prediction particularly
in turbulent flow causes distortion from ground truth values after few
timesteps. Han et al. [32] used pivotal nodes to summarize information of
graphs acquired from mesh representation into a latent vector and predicted
the next-step velocity using FC layers. The information is then decoded
through the pivotal nodes. This method reduced error accumulation and
computational cost associated with transient physics prediction.

In design optimization process, time-averaged properties are favorable [33,
10, 34]. Viquerat et al. [6] used VGG network for drag prediction of the
shapes generated using bezier curve at low Reynolds and steady state
condition. Zhou and Ooka used FC layers to predict CFD results for fixed cubic
geometry for indoor air flow application [35]. They used parameters of the
physics as the input of the neural network and velocity and temperature
tensors as the output. They reported less than 12% error for thermal
distribution prediction. Neural Network in FC form was also used for drag
coefficient prediction of multiple shapes representing cars [36]. Inter-
vehicular distance was used as the input of the model to predict drag
coefficient.

Previous studies focused on prediction of velocity, pressure, and drag
coefficient for constant geometries. In some cases, few geometrical parameters
such as aspect ratio were considered. Primary focus of this paper is heat
transfer and pressure drop prediction from a dataset of shapes generated based
on BREP ( Boundary Representation ). BREP is a method for representing
geometry using limits such as curves controlled by control points. BREP
facilitates boundary condition implementation which is crucial for thermo-
fluid structures where nonlinear PDEs should be solved numerically. BREP also
gained attention among researchers in structural topology optimization [37,
38].

In this paper, modern optimized CNN architectures are used for direct heat
transfer and pressure drop estimation of morphable shapes in contrast to
previous studies performed on constant geometry. This study focuses on heat
transfer and pressure drop prediction directly from images without depending
on mesh representation. Xception model along with a network optimized CNN are
used in this study. A robust model for heat transfer and pressure drop
prediction of different geometries can be used for the next generation design
process. By applying this model, the time required for domain heat transfer
and pressure drop computation for varied morphable topologies can be reduced
from several minutes to few seconds.

##  2 Methodology

Here, a summarized information about the domain is provided. The problem is a
conjugate heat transfer in a two dimensional space
$D=\overline{\Omega_{f}}\cup\overline{\Omega_{s}}\subset\mathbb{R}^{2}$
occupied by multiple solid fins, and an incompressible fluid. Fig. 1 shows the
domain.

###  2.1 High fidelity simulation

A CFD solver is created for the physics shown in Fig. 1. Incompressible
Navier-Stokes equation and convection-diffusion equation are solved using
FEniCS finite element solver. For further details on the numerical method,
please refer to Keramati et al [39]. Parallel computing using 16 cores and 32
threads is used for high fidelity simulation in FEniCS.

Fig. 1: Setting of computational domain

Here, mesh heuristic for multiple shape inside the domain is presented. Five
sets of geometries are considered which are randomly sampled among the shapes
constructed during primary running of the framework. A clipping technique is
used between number of meshes and mesh size on the curve constructed between
points. For the case with single shape, clipping is performed between 15
minimum number of cells and mesh sizes of L/200. However, with the increase in
the number of shapes, mesh size is set to be L/150, and number of meshes
between points remain higher than 12. Minimum number of meshes for multi-shape
framework is decreased compared to the single-agent framework; in case of
single shape, the meshes were constructed between the shape with small meshes
and boundaries of the domain with larger cells. In case of multiple shapes,
meshes are constructed between shapes with small cells on their boundaries
which result in high number of meshes. Fig 2 shows the meshes constructed for
a selected geometry. Implementation of this method results in number of grids
between 22,000 and 30,000 which depends on the size and shape of the
constructed geometry. Figure 3 shows the mesh heuristics based on the minimum
cell numbers between the points. It can be seen that after considering minimum
of 12 cells between the points, amounts of heat transfer remain consistent.

Fig. 2: Body-fitted mesh resolution of a selected geometry in multi-shape
dataset  Fig. 3: Dimensionless heat transfer for five random geometry settings
with different meshing sizes at Re = 10 and Pr = 0.7

###  2.2 CNN models

Several CNN architectures have been proposed in the past few years [40, 41].
These architectures range from regular CNNs stacked together to develop very
deep layers all the way to more complex engineered architecture after google
presented inception network [42]. Most of them are competitive in benchmark
image classification problems [43]. Their performances in real world
predictive tasks are, however, apart from the benchmark datasets [44, 45].
Xception network and a custom network optimized model are deployed in this
study. Xception architecture consists of a modified version of depthwise
separable convolution introduced in Inception model to work in a series of
operation to improve utilization of computing power [46]. For further details
please refer to the original reference [40].

###  2.3 Geometry representation

BREP is a method for representing shapes by defining topological and
geometrical components (e.g. curves, lines and vertices). In this study, we
use curves to provide design freedom for fins. Bézier spline is chosen to form
continuous fin geometries [47]. Mathematical definition of Bézier spline
facilitates the implementation of boundary conditions and conjugate heat
transfer [48]. The continuous derivative on the curve also provides a useful
tool for Neumann boundary condition implementation [39]. Bézier curves with a
set of control points $P_{i}$ with n+1 parameters are defined according to Eq.
1.

$\alpha(u)=\sum_{i=0}^{n}P_{i}B_{i,n}(u);u\in[0,1]$ (1)

Where $B_{i,n}(u)$ is Bernstein polynomials for the i’th function of degree n
which is defined as following:

$B_{i,n}(u)={n\choose i}n^{i}(1-u)^{n-i};i=0,..,n$ (2)

###  2.4 Dataset generation

Geometries are generated using mathematical method explained in section 2.3,
which provide 2D space for Eulerian simulation. Each shape is allowed to
occupy a rectangle with the size of H/4 and L/3. Mesh generation is based on
adaptive method and heuristics. Multiple scenarios for fluid flow are
considered which are controlled by Reynolds and Prandtl number. Since flow is
time-dependent, time-averaged numerical result is considered for a long period
of time over the third part of the computation. Temperature profile for some
of the cases with a single shape at one of the final time instances are shown
in Fig. 4. 4000 shapes using four and five points are generated and saved as
images. The size and resolution of the images are considered to be constant
with a single channel, and 506 $\times$ 506 pixels. CFD results are saved and
labeled accordingly. 70% of the data are used for training and the remainder
for cross-validation and test purpose.

Fig. 4: Temperature profile of some random shapes in the single-shape dataset

34,000 images are generated for multi-shape case using four and five points
for each shape. These images along with their CFD results are used for
training the surrogate model for multiple shapes in the domain. 90% of the
data are used for training the model and only 10% for validation and testing
the model. Temperature profile for a collection of random geometries from the
multishape dataset that is used for training the Xception network is shown in
Fig. 5. Unity-based normalization is used since heat transfer values for five
shapes can go above 1000 watts.

Fig. 5: A collection of random shapes in the dataset which is used for
training the surrogate model of the physics for the case with multiple shapes
inside the domain

###  2.5 Experimental setup

All experiments were conducted on a single AMD Ryzen workstation. The
workstation was equipped with Ubuntu 20.04 LTS, 16-Core Processor 3.40 GHz
CPU, 32.0 GB RAM, and NVIDIA GeForce RTX 3080 Graphics Processing Unit (GPU).
Python 3.8.9, CUDA 10.1, TensorFlow, and Keras are used for deep learning
implementation.

##  3 Results and discussion

###  3.1 Results for a single fin shape

First, we present the results for the case with a single shape.

####  3.1.1 Hyperparameter optimization

Hyperparameter tuning in DL models is often performed using either grid search
or random search [49]. In this study, random search is used for number of
feature extraction layers ($3<$ feature extraction layers $<10$), number of
filters in each layer ($8<$ number of filters $<96$), and number of FC layers
($2<$ FC layers $<5$) . After few experiments with kernel size for the
convolution operators, it is found that changes in accuracy and convergence
between 3 $\times$ 3 and 5 $\times$ 5 kernel size are negligible since the
location of the information is in the center of the image. For efficient use
of computational resources, 3 $\times$ 3 kernel size for convolutional layers
and 2 $\times$ 2 kernel size for maxpooling layers are used. The schematics of
the CNN architecture used for direct heat transfer prediction is shown in Fig
6. It consists of six successive feature extraction layers with two
convolutional layers and one max pooling layer in each of them. Some of the
hyperparameters of the model are shown in Appendix A . Total number of
trainable parameters for the proposed model is 1,160,305. Architecture of the
optimized network is shown in AppendixB. Xception model consists of more than
17 million parameters. Hyperparameter values for Xception model are considered
to be the same as those explained in the original article with the exception
of those shown in Appendix C [40]. Batch size for Xception is considered to be
256 while batch size of 128 leads to the best performance in regular CNN.
Training time for heat transfer prediction using regular CNN and Xception
network on our experimental setup are 45 minutes and 135 minutes,
respectively. The same method and architecture is used for pressure drop
considering the output neuron as pressure drop value. Training for pressure
drop estimation takes almost one hour for regular CNN and an average of 170
minutes for Xception model.

Fig. 6:  CNN architecture for heat transfer prediction

####  3.1.2 Error behavior

Root Mean Squared Error (RMSE), Mean Square Error (MSE), and Mean Absolute
Error (MAE) are used frequently in the literature for loss function definition
to evaluate the quality of a machine learning model during training [32]. In
this study, MSE is used as given in Eq. 3 to train the models [32]. In this
equation, $\hat{{{{y}}}_{i}}$ is the predicted value, and ${{{{y}}}_{i}}$ is
the ground truth value from high fidelity simulation. $N$ is the number of
data points in the batch sample.

$MSE({{{{y}}}_{i}},{{\hat{{y}}}_{i}})=\frac{\Sigma_{i=1}^{N}({{{{y}}}_{i}}-{{\hat{{y}}}_{i}})^{2}}{N}$
(3)

MSE values for each epoch of training and validation of optimized CNN model
and Xception model are shown in Fig. 7, and Fig. 8, respectively. As it can be
seen in these figures, Xception model has lower final MSE value. We provide
more statistical analysis of the models fo single shape and multi-shape case
in the next section.

Fig. 7: MSE value as a function of epoch during the training of optimized CNN
architecture for heat transfer prediction of single shape Fig. 8: MSE value as
a function of epoch during the training of Xception model for heat transfer
prediction of single shape

####  3.1.3 Prediction performance

Here, the performance of the surrogate models are analyzed using $R^{2}$ value
also known as coefficient of determination which is computed by Eq. 4.

$R^{2}=1-\frac{\Sigma_{i}{({{{{y}}}_{i}}-{{\hat{{y}}}_{i}})}^{2}}{\Sigma_{i}{({{{{y}}}_{i}}-\overline{{{{{y}}}}})}^{2}}$
(4)

where $\overline{{y}}$ is the baseline prediction which is considered to be
the mean of ground truth values.

MAE is also used to evaluate the model accuracy according to Eq. 5

$MAE=\frac{\Sigma_{i=1}^{N}({{{{y}}}_{i}}-{{\hat{{y}}}_{i}})}{N}$ (5)

In order to better visualize the estimated values, confidence intervals and
residual plots are also used. Fig. 9 and Fig. 10 show the predicted and ground
truth values for heat transfer in presence of a single fin using optimized CNN
model, and Xception model, respectively. On each plot, 150 data points of the
same shapes in the test dataset are used to avoid clutter. 99 percent
confidence intervals are also plotted in Fig. 9 and Fig. 10 which are barely
visible. This shows high accuracy in prediction. The trained model generalizes
well beyond the training dataset as can be seen from the predicted values from
the test dataset.

Fig. 9: Predicted and ground truth heat transfer in presence of a single fin
using optimized CNN model  Fig. 10: Predicted and ground truth heat transfer
in presence of a single fin using Xception network

Fig. 11 and Fig. 12 show the residual plot for heat transfer estimation
models. Residual is defined as a difference between the predicted and actual
values in the test dataset. Distribution of the residuals and data are also
shown on the right hand side and top of the plot, respectively. One important
observation from residual plot is that shapes with smaller heat transfer have
higher deviation from the actual values. Higher prediction errors are
associated with smaller heat transfer values which are pertinent to smaller
shapes. This might be addressed by choosing the norm based on the shape area
and normalizing the input data based on the shape area computed using Green’s
theorem [50]. However, area normalization was not performed since the accuracy
of the models are within a great range. This normalization can be applied for
higher complexity physics problems.

Fig. 11: Residual plot for heat transfer estimation in presence of a single
fin using optimized CNN model  Fig. 12: Residual plot for heat transfer
estimation in presence of a single fin using Xception network

Table 1 shows the statistical summary for both heat transfer estimation models
used for single shape. As it can be seen from the table, both models provide
high accuracy prediction with Xception network having slightly higher
performance. The coefficient of determination value is close to one and very
hard to achieve in regression models.

Table 1: Statistical results summary of the heat transfer surrogate models for a single morphable shape  Model | $R^{2}$ | MAE
---|---|---
CNN | 0.997 | 0.022
Xception | 0.998 | 0.019

Fig 13 shows the predicted and ground truth values for pressure drop
estimation for a single shape using optimized CNN for 200 data points. Fig. 14
shows the estimated values for Xception network. It can be seen that predicted
heat transfer values are closer to the regression line compared to pressure
drop values for both models. This is because of the data distribution of the
pressure values. Even though the same shapes are used for training the models,
distribution of the CFD results for heat transfer values are more compact than
pressure drop values.

Fig. 13: Predicted and ground truth pressure drop for a single shape using
optimized CNN model (99 percent confidence interval is also plotted)  Fig. 14:
Predicted and ground truth pressure drop for a single shape using Xception
network (99 percent confidence interval is also plotted)

Fig. 15 and Fig. 16 show the residual for pressure drop using CNN and Xception
network, respectively. Residual plot for optimized regular CNN shows very
small bias in predicting pressure drop. The histogram is far from being right-
skewed, but the peak distribution is not exactly at zero line. Small
overestimation can be observed in the prediction which can be accumulated over
the test set and cause larger MAE. Table 2 shows the statistical summary for
the pressure drop prediction of a single shape. As it was mentioned, CNN model
has a larger MAE. From the residual plot and MAE, it can be inferred that
Xception works better for both pressure drop and heat transfer prediction.

Fig. 15: Residual plot for pressure drop estimation of a single fin using optimized CNN model  Fig. 16: Residual plot for pressure drop estimation of a single fin using Xception network  Table 2: Statistical results summary of the pressure drop surrogate models for a single morphable shape  Model | $R^{2}$ | MAE
---|---|---
CNN | 0.999 | 0.0378
Xception | 0.999 | 0.023

Table 3 shows the average time required for the CNN models and CFD solver.
Regular CNN provides faster computation compared to other methods. CFD solver
occasionally fails because of the failure in mesh adaptation during the shape
morphing in presence of sharp edges or acute angles. DL-based computation does
not require mesh generation and provides more robust algorithm. High fidelity
simulation has higher standard deviation in terms of computing time since
variation of the shape causes change in number of meshes which leads to change
in computation time.

Table 3: Computation time for different computing methods for a single fin  Model | Average prediction time
---|---
CNN | 0.42 seconds
Xception | 2.03 seconds
High fidelity simulation | 45 minutes

###  3.2 Multiple fins in domain

In this section, the performance of the surrogate models for multiple shapes
across the domain are shown. Fig. 17 shows the residual for heat transfer test
dataset with the size of 200 images to avoid clutter. Comparing the surrogate
model performance to the ones presented in previous section, it can be
inferred that by increasing the number of shapes and consequently DOF,
accuracy of the machine learning model decreases. Challenges to improve the ML
model for higher accuracy remains open for ML community and competitions. Our
takeaway from the error behavior is that increasing design freedom is
associated with sacrificing accuracy in prediction. Higher DOF in design
requires larger dataset for training the surrogate model. The model shows a
reasonable performance with $R^{2}$ values of 0.968 and MAE of 0.032. Residual
plot shows that overall bias of the model can be considered very little to
none. Normal distribution of the residual with a center at almost zero value
shows the high accuracy of the model. The distribution of the data on top of
the chart in Fig. 17 shows that the center of the distribution is around the
$Q/Q_{b}$ value of 1.3 which is associated with minimum residual as well. This
provides high accuracy around this point. In general, the model overestimates
the larger values of heat transfer and underestimates the smaller values.
However, order of errors are small and can be considered as an accurate model.
Fig 18 shows predicted and ground truth values with 99 percent confidence
interval. As can be seen, the model can be used for heat transfer prediction
with high confidence.

Fig. 17: Residual plot of heat transfer prediction for multiple fin shapes
Fig. 18: Predicted and ground truth heat transfer for multiple shapes (99
percent confidence interval is also plotted)

Xception network is also used for pressure drop prediction of multiple fin
shapes with minimum hyperparameter search and without any specific changes.
Further details are provided in D. Fig 19 shows the residual plot for Xception
network trained for pressure drop prediction. Residual plot for pressure drop
shows almost a normal distribution centered around zero value which indicates
the model is fitted properly. Coefficient of determination is reported to be
0.98. MAE value of 0.033 is reported which is close to the heat transfer
prediction value.

Fig. 19: Residual plot of pressure drop values for multiple shape surrogate
model

Fig. 20 shows estimated values of pressure drop and actual values computed
using high fidelity simulation. The red line shows the regression line with 99
percent confidence interval around the line. Less confidence around the higher
values of pressure drop are seen. However, 99 percent confidence is of great
value for surrogate modeling tasks.

Fig. 20:  Predicted and ground truth values of pressure drop with 99 percent
confidence interval for multiple shape using Xception network

##  4 Conclusion and perspectives

In this study, modern convolutional neural networks were utilized to predict
heat transfer and pressure drop of the fins generated using composite Bezier
curves without expensive CFD computation. An optimized CNN network using
regular convolutions as well as an Xception model were deployed to predict
heat transfer and pressure drop values of a single fin shape. Xception network
showed higher accuracy in prediction compared to regular CNN. Xception network
was used for the domain with multiple fin geometries. The accuracy of the
surrogate models are reported and compared to the case with a single fin
shape. Increase in design freedom of the thermal domain results in higher
error in prediction. Larger dataset is also required to train the model for
the domain with higher degrees of freedom. The errors in prediction remain
within 3% even for multiple fin shapes. These values are promising for
surrogate modeling of morphable geometries. In this paper, time-averaged CFD
results were predicted directly from images of the fin geometries without
using mesh representation. This provides acceleration for heat transfer and
pressure drop computation from several minutes to less than two seconds. The
results presented in this study can be directly used for design optimization
to accelerate the iteration process.

The authors plan to use the presented surrogate models as the computational
engine for multi-agent reinforcement learning framework for design automation
of heat exchangers. This design automation is not possible without using a
surrogate model.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Acknowledgment

The authors thank Mojtaba Barzegari for his constructive feedback on FEniCS
implementation.

## CRediT authorship contribution statement

Hadi Keramati: Conceptualization, Methodology, Software, Validation, Writing
and Editing
Feridun Hamdullahpur: Supervision, Reviewing and Editing

##  Appendix A Hyperparameters for optimized regular CNN

In table 4, we provide hyperparameters of the optimized regular CNN to
facilitate the reproducibility.

Table 4:  Hyperparameter values for optimized regular CNN Hyperparameter | Value
---|---
Learning rate | $1\times 10^{-3}$
Decay | $5\times 10^{-3}$
Number of Convolution layers | 11
Number of MaxPooling layers | 5
Number of FC layers | 2
Activation function hidden layers | ReLU
Activation function output layer | Linear
Optimizer | Adam
Batch size $(N)$ | 128

##  Appendix B Optimized CNN architecture

In this section, we show the architecture of the optimized CNN used for the
case with a single shape. The input image size is downscaled version of the
original image by a factor of two with the size of 253 $\times$ 253 pixels.
Activation functions for convolutional layers and the fully connected layer
are ReLU and the output neuron is using linear function.

Fig. 21: Optimized CNN architecture

##  Appendix C Hyperparameters for Xception network used for single shape

In table 5, some of the hyperparameters of the Xception model used for single
fin shape are provided. Other hyperparameters are used the same as those in
the original article [40].

Table 5: Hyperparameters for Xception network used for single shape Hyperparameter | Value
---|---
Learning rate | $1\times 10^{-3}$
Optimizer | SGD
Batch size $(N)$ | 256

##  Appendix D Hyperparameters for Xception network used for multiple shapes

In table 6, some of the hyperparameters of the Xception model used for
multiple fin shapes are provided. No specific changes were made to the
original architecture [40].

Table 6:  Hyperparameters used for Xception network for multiple shapes Hyperparameter | Value
---|---
Learning rate | $0.0001$
Optimizer | Adam $(\beta_{1}=0.5,\beta_{2}=.997)$
Batch size $(N)$ | 256

## References

  * [1] S. Hoyer, J. Sohl-Dickstein, S. Greydanus, Neural reparameterization improves structural optimization, arXiv preprint arXiv:1909.04240.
  * [2] M. Sheikholeslami, H. Keramati, A. Shafee, Z. Li, O. A. Alawad, I. Tlili, Nanofluid mhd forced convection heat transfer around the elliptic obstacle inside a permeable lid drive 3d enclosure considering lattice boltzmann method, Physica A: Statistical Mechanics and its Applications 523 (2019) 87–104.
  * [3] F. Feppon, G. Allaire, C. Dapogny, P. Jolivet, Body-fitted topology optimization of 2d and 3d fluid-to-fluid heat exchangers, Computer Methods in Applied Mechanics and Engineering 376 (2021) 113638.
  * [4] A. Sanchez-Gonzalez, N. Heess, J. T. Springenberg, J. Merel, M. Riedmiller, R. Hadsell, P. Battaglia, Graph networks as learnable physics engines for inference and control, in: International Conference on Machine Learning, PMLR, 2018, pp. 4470–4479.
  * [5] C. Deng, Y. Wang, C. Qin, Y. Fu, W. Lu, Self-directed online machine learning for topology optimization, Nature Communications 13 (1) (2022) 1–14.
  * [6] J. Viquerat, E. Hachem, A supervised neural network for drag prediction of arbitrary 2d shapes in laminar flows at low reynolds number, Computers & Fluids 210 (2020) 104645.
  * [7] X. Chen, R. Chen, Q. Wan, R. Xu, J. Liu, An improved data-free surrogate model for solving partial differential equations using deep neural networks, Scientific Reports 11 (1) (2021) 1–17.
  * [8] T. Kodippili, S. Lambert, A. Arami, Data-driven prediction of forging outcome: Effect of preform shape on plastic strain in a magnesium alloy forging, Materials Today Communications 31 (2022) 103210.
  * [9] S. Hoyer, J. Sohl-Dickstein, S. Greydanus, Neural reparameterization for optimization of physical designs, uS Patent App. 16/722,587 (Jun. 24 2021).
  * [10] D. Lee, W. Chen, et al., Deep generative models for geometric design under uncertainty, arXiv preprint arXiv:2112.08919.
  * [11] J. Zhuang, D. Kochkov, Y. Bar-Sinai, M. P. Brenner, S. Hoyer, Learned discretizations for passive scalar advection in a two-dimensional turbulent flow, Physical Review Fluids 6 (6) (2021) 064605.
  * [12] K. R. Allen, T. Lopez-Guevara, K. Stachenfeld, A. Sanchez-Gonzalez, P. Battaglia, J. Hamrick, T. Pfaff, Physical design using differentiable learned simulators, arXiv preprint arXiv:2202.00728.
  * [13] R. Yamashita, M. Nishio, R. K. G. Do, K. Togashi, Convolutional neural networks: an overview and application in radiology, Insights into imaging 9 (4) (2018) 611–629.
  * [14] W. Chen, A. Ramamurthy, Deep generative model for efficient 3d airfoil parameterization and generation, in: AIAA Scitech 2021 Forum, 2021, p. 1690.
  * [15] D. R. Rutkowski, A. Roldán-Alzate, K. M. Johnson, Enhancement of cerebrovascular 4d flow mri velocity fields using machine learning and computational fluid dynamics simulation data, Scientific reports 11 (1) (2021) 1–11.
  * [16] X. Guo, W. Li, F. Iorio, Convolutional neural networks for steady flow approximation, in: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016, pp. 481–490.
  * [17] R. Vinuesa, S. L. Brunton, The potential of machine learning to enhance computational fluid dynamics, arXiv preprint arXiv:2110.02085.
  * [18] S. L. Brunton, Applying machine learning to study fluid mechanics, Acta Mechanica Sinica (2022) 1–9.
  * [19] A. Mendible, W. Lowrie, S. L. Brunton, J. N. Kutz, Data-driven modeling of two-dimensional detonation wave fronts, Wave Motion (2022) 102879.
  * [20] D. Kochkov, J. A. Smith, A. Alieva, Q. Wang, M. P. Brenner, S. Hoyer, Machine learning–accelerated computational fluid dynamics, Proceedings of the National Academy of Sciences 118 (21).
  * [21] T. Pfaff, M. Fortunato, A. Sanchez-Gonzalez, P. W. Battaglia, Learning mesh-based simulation with graph networks, arXiv preprint arXiv:2010.03409.
  * [22] B. Ummenhofer, L. Prantl, N. Thuerey, V. Koltun, Lagrangian fluid simulation with continuous convolutions, in: International Conference on Learning Representations, 2019.
  * [23] H. Ma, Y. Zhang, N. Thuerey, X. Hu, O. J. Haidn, Physics-driven learning of the steady navier-stokes equations using deep convolutional neural networks, arXiv preprint arXiv:2106.09301.
  * [24] A. Abucide-Armas, K. Portal-Porras, U. Fernandez-Gamiz, E. Zulueta, A. Teso-Fz-Betoño, A data augmentation-based technique for deep learning applied to cfd simulations, Mathematics 9 (16) (2021) 1843.
  * [25] S. Ye, Z. Zhang, X. Song, Y. Wang, Y. Chen, C. Huang, A flow feature detection method for modeling pressure distribution around a cylinder in non-uniform flows by using a convolutional neural network, Scientific reports 10 (1) (2020) 1–10.
  * [26] C. Birkenmaier, L. Krenkel, Convolutional neural networks for approximation of blood flow in artificial lungs, in: STAB/DGLR Symposium, Springer, 2020, pp. 451–460.
  * [27] M. Pourbagian, A. Ashrafizadeh, Super-resolution of low-fidelity flow solutions via generative adversarial networks, SIMULATION (2021) 00375497211061260.
  * [28] J. Sang, X. Pan, T. Lin, W. Liang, G. Liu, A data-driven artificial neural network model for predicting wind load of buildings using gsm-cfd solver, European Journal of Mechanics-B/Fluids 87 (2021) 24–36.
  * [29] J. Chen, E. Hachem, J. Viquerat, Graph neural networks for laminar flow prediction around random two-dimensional shapes, Physics of Fluids 33 (12) (2021) 123607.
  * [30] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al., Relational inductive biases, deep learning, and graph networks, arXiv preprint arXiv:1806.01261.
  * [31] A. Khan, A. Sohail, U. Zahoora, A. S. Qureshi, A survey of the recent architectures of deep convolutional neural networks, Artificial intelligence review 53 (8) (2020) 5455–5516.
  * [32] X. Han, H. Gao, T. Pffaf, J.-X. Wang, L.-P. Liu, Predicting physics in mesh-reduced space with temporal attention, arXiv preprint arXiv:2201.09113.
  * [33] I. S. Kavvadias, G. K. Karpouzas, E. M. Papoutsis-Kiachagias, D. I. Papadimitriou, K. C. Giannakoglou, Optimal flow control and topology optimization using the continuous adjoint method in unsteady flows, in: Advances in Evolutionary and Deterministic Methods for Design, Optimization and Control in Engineering and Sciences, Springer, 2015, pp. 159–173.
  * [34] G. H. Yoon, Topology optimization method with finite elements based on the k-$\varepsilon$ turbulence model, Computer Methods in Applied Mechanics and Engineering 361 (2020) 112784.
  * [35] Q. Zhou, R. Ooka, Neural network for indoor airflow prediction with cfd database, in: Journal of Physics: Conference Series, Vol. 2069, IOP Publishing, 2021, p. 012154.
  * [36] F. Jaffar, T. Farid, M. Sajid, Y. Ayaz, M. J. Khan, Prediction of drag force on vehicles in a platoon configuration using machine learning, IEEE Access 8 (2020) 201823–201834.
  * [37] W. Zhang, Q. Jiang, W. Feng, S.-K. Youn, X. Guo, Explicit structural topology optimization using boundary element method-based moving morphable void approach, International Journal for Numerical Methods in Engineering 122 (21) (2021) 6155–6179.
  * [38] X. Guo, W. Zhang, W. Zhong, Doing topology optimization explicitly and geometrically—a new moving morphable components based framework, Journal of Applied Mechanics 81 (8).
  * [39] H. Keramati, F. Hamdullahpur, M. Barzegari, Deep reinforcement learning for heat exchanger shape optimization, International Journal of Heat and Mass Transfer 194 (2022) 123112.
  * [40] F. Chollet, Xception: Deep learning with depthwise separable convolutions, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1251–1258.
  * [41] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [42] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1–9.
  * [43] L. Alzubaidi, J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, O. Al-Shamma, J. Santamaría, M. A. Fadhel, M. Al-Amidie, L. Farhan, Review of deep learning: Concepts, cnn architectures, challenges, applications, future directions, Journal of big Data 8 (1) (2021) 1–74.
  * [44] Z. Fan, H. Lin, C. Li, J. Su, S. Bruno, G. Loprencipe, Use of parallel resnet for high-performance pavement crack detection and measurement, Sustainability 14 (3) (2022) 1825.
  * [45] S. L. Brunton, J. N. Kutz, Data-driven science and engineering: Machine learning, dynamical systems, and control, Cambridge University Press, 2019.
  * [46] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna, Rethinking the architecture for computer vision, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826.
  * [47] W. Chen, M. Fuge, J. Chazan, Design Manifolds Capture the Intrinsic Complexity and Dimension of Design Spaces, Journal of Mechanical Design 139.  doi:10.1115/1.4036134.
  * [48] W. Chen, Data-Driven Geometric Design Space Exploration and Design Synthesis, PhD Thesis, University of Maryland, College Park (2019).
  * [49] L. Li, K. Jamieson, A. Rostamizadeh, E. Gonina, M. Hardt, B. Recht, A. Talwalkar, Massively parallel hyperparameter tuning.
  * [50] L. Yang, F. Albregtsen, Fast and exact computation of cartesian geometric moments using discrete green’s theorem, Pattern Recognition 29 (7) (1996) 1061–1073.