# Identify Light-Curve Signals with Deep Learning Based Object Detection
Algorithm. I. Transit Detection

Kaiming Cui Tsung-Dao Lee Institute, Shanghai Jiao Tong University, 800
Dongchuan Road, Shanghai 200240, People’s Republic of China  CAS Key
Laboratory of Optical Astronomy, National Astronomical Observatories, Chinese
Academy of Sciences, Beijing 100101, China  School of Astronomy and Space
Sciences, University of Chinese Academy of Sciences, Beijing 100049, China
Junjie Liu  Megvii Research Insitute, Megvii Technology, Chengdu, Sichuan
610095, China  Fabo Feng Tsung-Dao Lee Institute, Shanghai Jiao Tong
University, 800 Dongchuan Road, Shanghai 200240, People’s Republic of China
Jifeng Liu  CAS Key Laboratory of Optical Astronomy, National Astronomical
Observatories, Chinese Academy of Sciences, Beijing 100101, China  School of
Astronomy and Space Sciences, University of Chinese Academy of Sciences,
Beijing 100049, China  WHU-NAOC Joint Center for Astronomy, Wuhan University,
Wuhan, Hubei 430072, China

###### Abstract

Deep learning techniques have been well explored in the transiting exoplanet
field; however, previous work mainly focuses on classification and inspection.
In this work, we develop a novel detection algorithm based on a well-proven
object detection framework in the computer vision field. Through training the
network on the light curves of the confirmed Kepler exoplanets, our model
yields about 90% precision and recall for identifying transits with signal-to-
noise ratio higher than 6 (set the confidence threshold to 0.6). Giving a
slightly lower confidence threshold, recall can reach higher than 95%. We also
transfer the trained model to the TESS data and obtain similar performance.
The results of our algorithm match the intuition of the human visual
perception and make it useful to find single-transiting candidates. Moreover,
the parameters of the output bounding boxes can also help to find multiplanet
systems. Our network and detection functions are implemented in the Deep-
Transit toolkit, which is an open-source Python package hosted on Github and
PyPI.

Exoplanet detection methods — Transit photometry — Convolutional neural
networks

††software:  Numpy (Harris et al., 2020), Matplotlib (Hunter, 2007), PyTorch
(Paszke et al., 2019), Astropy (Astropy Collaboration et al., 2018, 2013),
Lightkurve (Lightkurve Collaboration et al., 2018), Wōtan (Hippke & Heller,
2019), MegEngine (MegEngine Collaboration, 2014), Jupyter Notebook (Kluyver et
al., 2016), Batman (Kreidberg, 2015)

##  1 Introduction

The increased amount of data in astronomy makes it necessary to apply machine-
learning algorithms for regression, classification, detection, clustering,
forecasting, etc. As a typical branch of machine-learning methods, deep
learning algorithms have been rapidly evolving in recent years, possessing
better performance compared to traditional methods in image classification
(Krizhevsky et al., 2012), visual recognition (Girshick et al., 2014), and
many other areas. Some featured algorithms, such as convolution neural network
(CNN; Lecun et al., 1998), recurrent neural network (RNN; Rumelhart et al.,
1985), long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997), and
generative adversarial networks (GAN; Goodfellow et al., 2014), are also used
in astronomy. For example, Leung & Bovy (2019) develop the astroNN package for
spectra analysis, and Charnock & Moss (2017) and Muthukrishna et al. (2019)
use RNN for transients classification on multiband photometric time series.
Liu et al. (2019) use LSTM to predict solar flares based on some active region
information and flare histories. Schawinski et al. (2017) use GAN for
recovering galaxy morphology.

In the field of signal detection and classification of light curve, 1D CNN,
RNN, and LSTM are commonly used (e.g., Hinners et al., 2018; Feinstein et al.,
2020). As a very important and attractive project, finding transiting
exoplanets from light curves also makes extensive use of deep learning
techniques, with 1D CNN being the primary method. Pearson et al. (2018) first
use 1D CNN to search for exoplanets based on iterative use of a binary
classifier, and Zucker & Giryes (2018) independently develop a similar work
with different data set. Meanwhile, a series of work on exoplanet vetting
based on 1D CNN starting from Shallue & Vanderburg (2018), who work on Kepler
data with local and global views for folded light curves of transit
candidates. Following works introduce external information (e.g., Ansdell et
al., 2018, includes centroid curves and stellar parameters) or applied it to
K2 data set (Dattilo et al., 2019), TESS data set (Yu et al., 2019; Osborn et
al., 2020), Wide Angle Search for Planets program (Schanche et al., 2019), and
the Next Generation Transit Survey (Chaushev et al., 2019). Olmschenk et al.
(2021) develop a systematic pipeline for detecting and vetting transiting
exoplanets from TESS full-frame image (FFI) light curves. They apply a 1D CNN
for outputting the confidence of existing potential transiting exoplanets in a
given FFI light curve.

In this work, we explore a two-dimensional (2D) object detection algorithm to
identify transiting signals. 2D means the detection is performed on the image.
Compared with previously 1D CNN, our approach is straightforward and matches
human visual intuition. We implement our neural network based on Kepler, then
transfer learning to TESS. Section 2 introduces the Kepler training data
preparation (Section 2.1), network architecture (Section 2.2), detailed
training steps (Section 2.3), and test and comparison results (Section 2.4).
In Section 3, we transfer the model trained on Kepler to TESS data. After we
have a trained model, further applications and inspections can be developed
based on our results. Section 4 shows several application examples, and
Section 5 discusses features and limitations of our method. Our training and
detection algorithms are implemented in the Deep-Transit, which is an open-
source Python package. 111https://github.com/ckm3/Deep-Transit

##  2 Implementation and Training

Different from the commonly used 1D CNN and RNN (LSTM) algorithm, we plot the
lightcurve to an image, so that the 1D time series is converted into a 2D
image. This allows the rapidly developing computer vision algorithms to be
applied to light-curve data. In terms of data preparation and network
structure, our approach has two main advantages over previous algorithms.

  1. 1.

The sampling frequency requirement for light curves is relatively lenient. For
1D CNN and RNN (LSTM), the input dimension is a fixed number of data points,
so that different data set require interpolation or binning. In contrast, our
method is less sensitive to the sampling frequency and preserves the original
information of the data.

  2. 2.

Our network has an output combining three scales: large, medium, and small.
Thus it can be adapted to different information densities generated by
different data sets without modifying network structure.

###  2.1 Data Set Preparation

As one of the most successful light-curve products, Kepler light curves are
ideal for our work. Our data sets are selected from the Kepler light curves
for confirmed transiting planets
222https://exoplanetarchive.ipac.caltech.edu/docs/counts_detail.html. The
time, durations, and periods of those transits are collected from the Kepler
Threshold-Crossing Event catalog (Twicken et al., 2016). Compared with
synthetic data sets, real data have more realistic stellar activity signal and
systematics. Before creating our training and validation data, we randomly
select the Kepler light curves for 100 confirmed exoplanets as the test data.

Similar to other transit detection algorithms, we also need to detrend
instrumental and the photometric variation caused by stellar activity. To do
so, we apply the Tukey’s biweight algorithm (Mosteller & Tukey, 1977)
implemented by Wotan 333https://github.com/hippke/wotan, whose performance is
proven in blind search of Kepler and K2 data (Hippke & Heller, 2019). The
parameters we use are the default parameters in Wōtan. Then, we have a sigma
clipping for the detrended light curves. The upper clipping limit is three
standard deviations and the lower clipping limit is 20 standard deviations.

Since our method relies on the visual presentation of the data, the method of
conversion from light-curve data to images is critical. In our work, we first
split the light curve into 30 day segments (less than 30 days are not split)
and make the next window overlap the previous window by 5 days (i.e., a 30 day
sliding window size with a 25 day step). Then, in each 30 day segment, the
light curve is split into 10 day segments, and the next segment overlaps the
previous one by 3 days (i.e., a 10 day sliding window with a 7 day step). We
generate images for each of the 10 day light-curve segments. We choose the
window length based on our experience and data propriety of Kepler and TESS,
and it can be modified for different type of signals. Actually, after
training, a slight modification of the window length has little effect on the
results. For convenience, our image size is chosen to be $416\times 416$,
which is a commonly used size for object detection. Larger size could has
better performance but cost more computation time.

Based on our experience in light-curve identification, we plot data in line
with point markers to enhance the characteristics of the light curve. The line
width is set to 1 pixel, and the data point is the circle with a 2-pixel
radius. These choices are flexible, because after training is complete, the
network has the ability to generalize and can handle slightly different
plotting styles. To keep things as simple as possible, all the lines and
points are black, so the image is a single-channel bitmap with the shape of
(416, 416, 1). The flux range for plotting is the flux range of 30 day
segments, rather than within 10 days. This is because we want our model to
have some perception of a wider range of time and avoid some local false
positives. Also, we extend the lower limit of 30 day segments’ flux by 2% to
avoid the lowest flux of a transit being at the edge of an image.

The format of our bounding box label is
$(x_{\mathrm{middle}},y_{\mathrm{middle}},w,h,\mathrm{S/N})$. The
$x_{\mathrm{middle}}$ and $y_{\mathrm{middle}}$ are the normalized coordinates
of the center of a transit (i.e., convert the coordinates to 0–1, the
coordinate of the left edge is 0, the right edge is 1, the top edge is 0, and
the bottom edge is 1). The width $w$ of our bounding box is set to twice the
normalized duration, and if the width is less than 0.02, then enforce it to
0.02. The top value of the bounding box is 95% of the flux of the 10 day
segment. To ensure that the bounding box can cover the whole transiting
region, we extend the height $h$ down by 4 pixels. We also calculate the
signal-to-noise ratio (S/N) for each transit. The median and standard
deviation are derived from the out-of-transit region of the 10 day segment.
The depth of transit is estimated from the median value of the middle half. In
our training set, only transits with S/N $>$ 3 are generated. Figure 1 is a
schematic diagram of our training data set generation.

Figure 1: Schematic diagram of training data set generation steps. The top
panel shows a part of a detrended light curve. The color shaded sections
indicate 30 day windows, and they have 5 day overlapped areas between adjacent
windows. The dashed red lines are 10 day selection ranges, and they have 3 day
overlapped areas between adjacent ranges. The bottom two panels are the zoom-
ins of two 10 day selection ranges, and the bounding boxes are also plotted
with green rectangular boxes. The coordinates of a bounding box
$(x_{\mathrm{middle}},y_{\mathrm{middle}},w,h)$ and four corners are indicated
in the bottom left panel.

After the data set preparation, we have more than 140,000 images, and each of
them has at least one transit event. Then, we randomly split the data into a
training set and a validation set at a ratio of 85% and 15%. Different from
the previously preserved test set, our training and validation sets are split
on the image level to make the parameters of the transits evenly distributed
in the training and validation sets as much as possible.

###  2.2 Network Architecture

We do not need to invent a new object detection network because there are
already a lot of well-proven algorithms and tools in the computer vision
field. We choose the well-established You Only Look Once, Version 3
architecture (YOLOv3; Redmon & Farhadi, 2018), which has a wide range of
applications (e.g., Choi et al., 2019; Tian et al., 2019; Yurtsever et al.,
2020) and performs better than previous algorithms (e.g., R-CNN; Girshick et
al. 2014, Faster R-CNN; Ren et al. 2017) in terms of small object recognition
and computational speed. Our network architecture is shown in Figure 2. It has
one input head and an output of three scales for large, medium, and small
objects. The final output combines the results of three scales through the Non
Maximum Suppression (NMS) algorithm, which is a standard way for only keeping
best bounding boxes (Rosenfeld & Thurston, 1971). The main procedure of NMS is
to iteratively preserve the boxes with the highest confidence score and then
remove other boxes depending on a threshold.

Figure 2: Network architecture. Our network structure is based on a standard
YOLOv3 architecture with a slightly modified input channel and output
dimension. The dashed line region indicates the Darknet-53 backbone. Each
convolution block is shown as a CONV, it is combined by a 2D convolution
layer, a batch normalization layer, and a Leaky ReLU function (Maas et al.,
2013). The numbers split by dashes of a convolution block show the size and
stride of the convolution layer. Each residual block is abbreviated as RES. As
shown on the right side of the figure, a residual block is combined by two
convolution blocks with a shortcut. The scale block is shown as SCALE, and the
following number is the size of each scale. Similar to the residual block,
each scale block has two convolution blocks but without a shortcut. Combining
the results of the three scales to calculate the NMS, the final output is the
coordinates and confidence level of N-detected bounding boxes.

Original YOLOv3 network is designed for detecting daily common objects, so
they set their prior anchor boxes based on some competition data sets (e.g.,
PASCAL Visual Object Classes Challenge and Microsoft Common Objects in
Context). Anchor boxes are crucial parameters in YOLO, they are a set of
predefined bounding boxes of a certain height and width. However, in our work,
we are aiming at identifying transits, the size of anchors should be
determined by the characteristics of the signals, and since we have only one
class, too many anchor boxes are overkill for our detection. Thus, we measured
the height and width distribution of all the boxes in our data set, and then
we manually selected three anchors to roughly get close to the size of most of
the transiting signals, while considering assigning different sizes of anchors
for different scales. Finally we reset the width and height of anchor boxes to
(0.1, 0.9), (0.05, 0.7), and (0.02, 0.3) for three scales accordingly. The
width is significantly lower than the height, consistent with the basic
pattern of a transit. We try adding more anchors but it does not improve the
performance. We also compare the original YOLOv3 anchors with our modified
version and find that the modified one is better in terms of transit
identification.

Usually, the confidence score of a bounding box is trained to equal the
intersection over union (IOU) between a predicted box and the ground truth.
IOU is defined as a ratio of the areas of intersection and union between the
predicted bounding box and the ground-truth bounding box. IOU describes the
accuracy of a predicted box; the higher the IOU, the more accurate the overlap
with the target. However, the range of the target box is artificially selected
by us, so the high overlap does not have much physical sense. Instead, we
consider a more physical expression of the confidence score to be the S/N. We
can easily calculate the S/N for a single-transit event, and the S/N can be
naturally converted to a confidence score with a normalization method. In this
work, our confidence scores $C_{\mathrm{S/N}}$ of training data are calculated
by $C_{\mathrm{S/N}}=1-\exp(-0.15\ \mathrm{S/N})$. The
$C_{\mathrm{S/N}}\approx 0.6$ when the S/N is 6, and then rapidly increase to
1, matching our naked eye’s intuitive perception and previous research
thresholds (e.g., Kovács et al., 2002; Kunimoto et al., 2018). A different
normalization method is also acceptable for different requirements.

Then, we modify the original YOLOv3 loss function because of the different
number of anchor boxes and confidence scores. As shown in Equation 1, loss
function of each scale is a combination of three parts and is defined as

$\displaystyle L$ $\displaystyle=$ $\displaystyle
L_{\mathrm{coord}}+L_{\mathrm{obj}}+L_{\mathrm{noobj}}~{},$ (1)
$\displaystyle=$
$\displaystyle\lambda_{\mathrm{coord}}\frac{1}{N}\sum_{i=0}^{S^{2}}\mathbbm{1}_{i}^{\mathrm{obj}}[(x_{i}-\hat{x}_{i})^{2}+(y_{i}-\hat{y}_{i})^{2}]$
$\displaystyle+$
$\displaystyle\lambda_{\mathrm{coord}}\frac{1}{N}\sum_{i=0}^{S^{2}}\mathbbm{1}_{i}^{\mathrm{obj}}[(w_{i}-\hat{w}_{i})^{2}+(h_{i}-\hat{h}_{i})^{2}]$
$\displaystyle-$
$\displaystyle\frac{1}{N}\sum_{i=0}^{S^{2}}\mathbbm{1}_{i}^{\mathrm{obj}}[C_{\mathrm{S/N}}\log(C_{i})+(1-C_{\mathrm{S/N}})\log(1-C_{i}))]$
$\displaystyle-$
$\displaystyle\lambda_{\mathrm{noobj}}\frac{1}{N^{\prime}}\sum_{i=0}^{S^{2}}\mathbbm{1}_{i}^{\mathrm{noobj}}\log(1-C_{i})~{}.$

The sum of the first two terms is the coordinate mean squared error (squared
L2 norm) loss. $x_{i}$, $y_{i}$, $w_{i}$, $h_{i}$ denote the predicted
coordinates of boxes in the cell $i$, similarly, $\hat{x}_{i}$, $\hat{y}_{i}$,
$\hat{w}_{i}$, $\hat{h}_{i}$ denote the target coordinates of boxes. $S$ is
the scale size. $\mathbbm{1}_{i}^{\mathrm{obj}}$ equals to 1 when there is an
object in the cell $i$, otherwise 0; $\mathbbm{1}_{i}^{\mathrm{noobj}}$ equals
1 when there is no object in the cell $i$. $N$ is the number of object cells
for calculating the mean value. The third and fourth terms show the binary
cross-entropy losses of the object and no object cells. Binary cross-entropy
is a commonly used cross-entropy loss for binary classification applications.
$C_{i}$ is the predicted confidence score of whether there is an object or
not. Similar to $N$, $N^{\prime}$ is the number of cells without object.
$\lambda_{\mathrm{coord}}$ and $\lambda_{\mathrm{noobj}}$ are the constants to
increase the attention of certain parts. In our work, we choose
$\lambda_{\mathrm{coord}}=\lambda_{\mathrm{noobj}}=10$ to ensure the balance
of those three losses. Compared with original YOLOv3 loss function, we do not
have the class loss, because we only focus on transit signals.

###  2.3 Training Steps

There are two key metrics precision and recall. Precision is defined as
$\mathrm{TP/(TP+FP)}$, and recall is defined as $\mathrm{TP/(TP+FN)}$. The TP,
FP, and FN indicate true positive, false positive, and false negative. Average
precision (AP) is calculated by the area under the precision-recall (PR)
curve. $\mathrm{AP}_{50}$ means AP with the IOU threshold greater than 0.5,
$\mathrm{AP}_{70}$ means AP with the IOU threshold greater than 0.7, and so
does $\mathrm{AP}_{90}$. We choose the mean value of $\mathrm{AP}_{50}$,
$\mathrm{AP}_{70}$, and $\mathrm{AP}_{90}$ as our training evaluation metric
($\mathrm{AP}_{\mathrm{mean}}$). The $\mathrm{AP}_{50}$ is the baseline of
many algorithms because IOU larger than 0.5 is sufficient for most ordinary
target detection tasks. We consider $\mathrm{AP}_{70}$ and $\mathrm{AP}_{90}$
because we want the parameters of the box to provide better help for
subsequent applications.

After several trials based on the performance on the validation set, we choose
the Adam optimizer to optimize the total loss $L$. The initial learning rate
is $10^{-4}$, and the weight decay is $10^{-4}$. A lower learning rate would
increase the training steps significantly. We also reduce the learning rate by
a factor of 2 when our metric has stopped improving for two epochs. The epoch
is the number of passes of the entire training data set. In our work, each
epoch has 1265 iteration steps. To improve training efficiency, we input a
batch of training examples in each iteration step. Our batch size is 96, lower
batch size performs worse. The NMS threshold is 0.1, the IOU threshold is 0.5,
and the confidence score threshold is 0.6. Each epoch takes about 20 minutes
on an NVIDIA Tesla V100 GPU.

Figure 3 shows the losses, $\mathrm{AP}_{50}$, $\mathrm{AP}_{70}$,
$\mathrm{AP}_{90}$ and $\mathrm{AP}_{\mathrm{mean}}$, vary during the
training. Those APs are calculated on the validation data, and we can see them
increase and then keep constantly after longer epochs. Due to overfitting, the
loss on the training set continues to drop, but the metrics on the validation
set are no longer rising. Therefore, we choose the model with the largest
$\mathrm{AP}_{\mathrm{mean}}$ on the validation set as the best model. The PR
curve of the best model is shown in Figure 4. Our final best model has an
$\mathrm{AP}_{50}=0.9$, which is an excellent performance for an object
detection task. The best model file saves the states of the network trained
from Kepler data, which can be directly downloaded from DOI: 10.12149/101079.

Figure 3: Training steps on the Kepler light curves. The black line is the
loss and the APs are plotted as different colors for different IOU thresholds.
The AP values of the best model are marked and written on the figure. Figure
4: The precision-recall curve for IOU $>$ 0.5 of the best model. The blue
shaded area is the $\mathrm{AP}_{50}$. The sharp drop at 0.9 is due to our
relatively high confidence threshold (0.6), making recall only go up to 0.9.

###  2.4 Model Tests and Comparisons

After finishing the training, we need to test the performance of the model.
Although our high $\mathrm{AP}_{\mathrm{mean}}$ on the validation set gives us
confidence, we still want to know its performance on real detection when
facing transits with different S/Ns. To do so, we apply our best model on 100
previously reserved exoplanet hosts. Consistent with the real detection task,
we use the detection function implemented in our Deep-Transit package with
default parameters. To use more physical criteria, we treat a detected box as
a true positive when the time of the transiting midpoint locates in the box
range. We apply our test on different confidence levels and different S/Ns of
transits. Figure 5 and Figure 6 show the precision and recall matrices
accordingly. These two matrices present a comprehensive evaluation of the
detection ability of our model. We can also easily recognize that the trade-
off between precision and recall. For the detected confidence scores higher
than 0.6, our model’s precisions can achieve high to 0.9 in most cases. Higher
confidence scores allow precisions close to 1. For the confidence scores lower
than 0.5, the recalls are higher than 0.8 in most cases. Lower confidence
scores also make recalls close to 1 for S/Ns higher than 6. Therefore, our
model has high practicality and can control the precision and recall of
detection by adjusting the confidence threshold. To have a balance between
precision and recall, 0.6 can be a Goldilocks score in most cases. To perform
an extensive search, 0.5 is a practical choice.

Figure 5: Precision matrix of the best model trained from Kepler on the test
set. Figure 6: Recall matrix of the best model trained from Kepler on the test
set.

We also compare our model with two practically used baselines. Unlike other
commonly used exoplanet detection algorithms (periodogram-based algorithms
such as box-fitting least squares), our model lacks period information.
Therefore, a more appropriate baseline is the algorithm of single-transit
detection. One of the most straightforward and conventional single-transit
detection algorithms is a transit fitting with a moving window, which is
applied by most single-transit event searches of Kepler and K2 with only minor
differences (e.g., Foreman-Mackey et al., 2016; Osborn et al., 2016; Kawahara
& Masuda, 2019). We implement this algorithm by following Osborn et al.
(2016). Our transit model is an analytic model with quadratic limb darkening
(Mandel & Agol, 2002), and it is generated with the Batman, which is an open-
source Python package for fast calculation of exoplanet transit light curves.
The window sizes and transit durations are exactly adopted from Osborn et al.
(2016). We applied this algorithm to our test set. For transits with S/N $>$
6, the precision is 0.59 when the recall reaches 0.91 by tuning the reduced
$\chi^{2}$ threshold. The higher number of false positives is also confirmed
in Kawahara & Masuda (2019), this is also why single-transit detection usually
requires visual inspections to rule out false positives. Our network is based
on computer vision techniques, thus we have fewer false positives.

Another baseline is given by the naked eyes. Many exoplanet detection
pipelines require visual vetting to remove false positives (e.g., Batalha et
al., 2013; Huang et al., 2013; Guerrero et al., 2021). Also, as one of the
most famous citizen science projects, Planet Hunters
444https://www.zooniverse.org/projects/nora-dot-eisner/planet-hunters-tess
attracts volunteers to visually detect exoplanets. According to their papers
(Fischer et al., 2012; Eisner et al., 2021), the practical recovery rate of
Planet Hunters is about 67% for Kepler data, and 53% for TESS objects of
interest (TOI) data. The spurious transit boxes rate is about 10%. Therefore,
compared with the Planet Hunters, our network performs similar precision but
higher recall. The result is reasonable because our trained model is more
sensitive to transit signals compared to amateur volunteers.

##  3 Transfer Learning to TESS

As an ongoing survey, TESS provides higher duty cycle light curves than
Kepler. Therefore, we also wish to apply our well-trained model to the TESS
data. Usually, deep learning algorithms can only achieve high performance on a
specific task. It means that our model trained on Kepler can not be directly
applied to TESS. However, since the main difference between TESS and Kepler is
their photometric accuracy and sampling rates, we can create a well-performing
model for TESS data based on the pretrained Kepler model. To do so, a transfer
learning technique can be helpful.

Transfer learning inherits existing relevant data or models while building a
new machine-learning model. In practice, transfer learning is a standard way
when there are not enough training data, or the source and target domains have
some similarities but are not identical (Pan & Yang, 2009). Therefore, when
considering the short observation time of TESS, resulting in a small amount of
data, transfer learning is an appropriate choice. To achieve this, we need to
prepare the data set for TESS and then load the model already trained on the
Kepler data as a pretrained model. The training on TESS data only requires a
fine-tuning of the pretrained model.

###  3.1 Training and Evaluation

First, we create the TESS data set similarly as we create the Kepler data set
(described in Section 2.1). Since there are only over 100 confirmed exoplanets
detected by TESS, our training and validation data set are created from the
TOI catalog (Guerrero et al., 2021). The main difference in the data set
preparation is that the TESS images are plotted using scatter points with a 30
day moving average gray line overlaid. The points are 1 pixel points and the
line width is 1 point. Figure 7 shows two examples of the TESS training data
set.

Figure 7: Two examples of TESS training and validation data. The bounding
boxes are shown as green boxes. Note that the actual sample does not have a
border.

Then, we use the same network structure and load our pretrained Kepler model
to initialize the network without freezing any layers. As the model converges
quicker, we reduce the learning rate by a factor of 2 when our metric has
stopped improving for one epoch. The performance during training steps is
shown in Figure 8. Similarly, as described in Section 2.3, we choose epoch 6
as the best performance model. Also, the best model saves the states of the
network trained from TESS data, which can be directly downloaded from DOI:
10.12149/101079.

Figure 8: Training steps of transfer learning on TESS data. Same as Figure 3.

Finally, we test the performance of our model using light curves of confirmed
TESS exoplanet hosts. Due to the relatively small amount of data in TESS, we
manually adjust some parameters of the detrending method. Like with the
previous Kepler model, the test results are shown as two matrices (see Figure
9 and Figure 10). Their general performance is good but relatively lower than
Kepler, which might be caused by the lower data quality of TESS. The recall is
better than precision, makes it more suitable for a large-scale transit
search.

Figure 9: Precision matrix of the best model trained from TESS on the test
set. Figure 10: Recall matrix of the best model trained from TESS on the test
set.

##  4 Application

After the training and evaluation of the model is completed, our algorithm can
have some good practical applications.

Once the bounding boxes with confidence scores of transits are successfully
detected, the width and height of boxes can be used as a rough estimation for
the duration and depth of the transits. For example, Kepler-297 (KIC 11122894)
has three transit candidates and two of them are confirmed (Lissauer et al.,
2011; Rowe et al., 2014). Table 1 list some of their parameters from Rowe et
al. (2014).

Table 1: Table of parameters of transits of Kepler-297 from Rowe et al. (2014) Parameters |  |  |
---|---|---|---
KOI | 1426.01 | 1426.02 | 1426.03
Kepler ID | Kepler-297 b | Kepler-297 c |
Period (days) | 38.871826 | 74.920137 | 150.018303
$R_{\mathrm{p}}/R_{*}$ | 0.02876 | 0.06543 | 0.59798
Transit Depth | 940.4 | 4150.1 | 4437.6
Transit Duration | 6.125 | 4.794 | 4.400

We can apply our detection method to the 4 yr light curve of Kepler-297 to
reproduce its multiplicity. Figure 11 shows the flattened light curve and our
detected bounding boxes. The confidence threshold is 0.8. After we put the
width and height of the detected boxes into a scatter plot, we can see two
obvious clusters in Figure 12. If we look closely, the yellow points in the
upper-left corner also seem can be divided into two parts. Actually, they have
similar depth and width but different periods. By selecting the light curve in
the detected bounding boxes, we can perform a more focused period detection.
As shown in Figure 13, the periods of those three clusters can be estimated
from the middle time directly.

Figure 11: Light-curve and transit detection results of Kepler-297. The green
boxes are the detected bounding boxes, and their top square blue labels
indicate the confidence scores. Figure 12: Scatter plot of the width and
height of the detected boxes. The color of the points indicate the confidence
level. Figure 13: Selected light curves from our detected bounding boxes. (a)
is the selection of detected transiting events of Kepler297b. (b) is the
selection of detected transiting events of Kepler297c. (c) is the selection of
detected transiting events of KOI-1426.03.

Since our algorithm is complete and accurate to a single transit with high S/N
$>$ 7, it has a strong potential for single-transits detection. Kawahara &
Masuda (2019) yields 67 single-transit candidates in Kepler light curves from
previous work (e.g., Uehara et al., 2016; Foreman-Mackey et al., 2016). Nearly
all of them can be detected by our model with proper detrending (the default
window size 0.5 day will remove some long duration transits). Figure 14 shows
an example of KOI-1174, which has a single-transiting exoplanet candidate. Our
detection algorithm can easily find the only transit with a confidence score
higher than 0.8.

Figure 14: Light curve of KOI-1174. (a) is the 4 yr light curve of KOI-1174.
(b) is the zoom-in of the detected region. The green box is the detected
bounding box and the confidence score is written in the semitransparent blue
box.

Finding single transits becomes even more important due to the observation
mode of TESS (e.g., Cooke et al., 2018; Villanueva et al., 2019; Yao et al.,
2021). Our model can be a powerful tool for future single-transits detection
work.

##  5 Discussion

From the human perception, our method is intuitive and can help some visual
searches and inspections. For example, the individual vetting and group
vetting of TOIs could apply methods similar to ours to reduce manual effort
and increase efficiency. Also, some citizen science projects like Planet
Hunters and Citizen ASAS-SN
555https://www.zooniverse.org/projects/tharinduj/citizen-asas-sn/classify can
be partly replaced by our method. An alternative approach is to use the
results of citizen projects as the training set for the model.

From the view of data analysis, our methods of converting 1D time series into
2D images can be seen as a kind of feature engineering, which is widely used
in the field of signal processing. For example, the discrete cosine transform
converts data from time domain to frequency domain and it is commonly used in
many traditional and deep learning areas, such as computer vision (e.g., Zhang
et al., 2020), voice recognition (Bae et al., 2016) and electroencephalogram
signal detection (Rundo et al., 2019). Our approach implicitly includes some
prior and feature enhancements from our own visual perception. This provides
the direction for the optimization of our algorithm: clearer and more
intuitive illustration or presentation of data for humans can yield better
results on the computer.

Considering that the observation windows of ground-based telescopes are often
affected by weather, season, etc., the sampling interval is more complicated.
Our method can be seamlessly extended to the ground-based photometric survey,
like ASAS-SN, ZTF, Vera C. Rubin Observatory, etc. In particular, multiband
photometry can be naturally implemented as the multiple channels of an image,
providing richer information.

However, there are three known limitations in our current algorithm.

  1. 1.

Periodicity determination. Estimate period directly from the neural network is
a quite difficult problem. Traditional algorithms like the box-fitting least
squares are still more robust.

  2. 2.

The method of the detrending light curve has a significant impact on the S/Ns
of transits. Though Tukey’s biweight algorithm performs fairly robust in blind
searches, its parameters still need to be adjusted for different types of
trends and systematics.

  3. 3.

Technically, our algorithm only detects transit-like signals, we do not
introduce other possible false positives (e.g., eclipsing binaries) in our
data set. Considering that many works are using deep learning algorithms to do
the transit inspection (e.g., Astronet of Shallue & Vanderburg 2018; Exonet of
Ansdell et al. 2018), our results can be the inputs of their networks to
classify signals.

We have plans to solve the above difficulties in our future work. The lack of
period information, intrinsic and external noises of the light curve, mainly
limit the performance of our model for low S/N objects. An alternative
solution is to simultaneously include the result of the box least squares
periodogram and corresponding folded light curve. Some time-series modeling
methods such as autoregressive moving average and its extensions (e.g.,
autoregressive integrated moving average, autoregressive fractionally
integrated moving average; Carter & Winn, 2009; Feigelson et al., 2018;
Caceres et al., 2019) can be helpful for removing stellar variations and
systematics.

We will also extend our detection target to more classes of light-curve
signals (e.g., rotation, pulsation, and eclipsing binary). By modeling those
various types of detected objects, detrending algorithms can be more accurate,
and false positives can also be identified. In addition, the data within the
output box of the network can be used as the start of further detection and
inspection of the signal, resulting in a complete imitation of human
operation. However, some blended eclipsing binaries can be hardly
discriminated from planetary transits only based on light curves. Usually,
these contaminations are removed with some pixel-level inspection. Thus, we
are trying to add some pixel vetting modules in our network, and they can be
naturally applied in our 2D network. With the development of the computer
vision field, more advanced algorithms and structures (e.g., YOLOv5 and
Transformer) can be a promising choice in our future work.

We appreciate the suggestions and comments of referee and editors that make
this work more solid. We thank Masataka Aizawa for the helpful suggestions. We
acknowledge support from the National Key Research and Development Program of
China (NKRDPC) under grant numbers 2019YFA0405000, B-type Strategic Priority
Program of the Chinese Academy of Sciences, Grant No. XDB41000000, and from
the National Science Foundation of China (NSFC, Nos. 1193000332, 1197030174).
Data resources are supported by China National Astronomical Data Center (NADC)
and Chinese Virtual Observatory (China-VO). The authors acknowledge the
Beijing Super Cloud Computing Center (BSCC) and the Tsung-Dao Lee Institute
for providing GPU resources that have contributed to the research results
reported within this paper. This paper includes data collected by the Kepler
and TESS mission and obtained from the MAST data archive at the Space
Telescope Science Institute (STScI). Funding for the Kepler and TESS mission
is provided by the NASA Science Mission Directorate. STScI is operated by the
Association of Universities for Research in Astronomy, Inc., under NASA
contract NAS 5–26555.

## References

  * Ansdell et al. (2018) Ansdell, M., Ioannou, Y., Osborn, H. P., et al. 2018, ApJ, 869, L7, doi: 10.3847/2041-8213/aaf23b
  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33, doi: 10.1051/0004-6361/201322068
  * Astropy Collaboration et al. (2018) Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., et al. 2018, AJ, 156, 123, doi: 10.3847/1538-3881/aabc4f
  * Bae et al. (2016) Bae, H.-S., Lee, H.-J., & Lee, S.-G. 2016, in 2016 IEEE 11th Conference on Industrial Electronics and Applications (ICIEA), 1542–1546, doi: 10.1109/ICIEA.2016.7603830
  * Batalha et al. (2013) Batalha, N. M., Rowe, J. F., Bryson, S. T., et al. 2013, ApJS, 204, 24, doi: 10.1088/0067-0049/204/2/24
  * Caceres et al. (2019) Caceres, G. A., Feigelson, E. D., Jogesh Babu, G., et al. 2019, AJ, 158, 58, doi: 10.3847/1538-3881/ab26ba
  * Carter & Winn (2009) Carter, J. A., & Winn, J. N. 2009, ApJ, 704, 51, doi: 10.1088/0004-637X/704/1/51
  * Charnock & Moss (2017) Charnock, T., & Moss, A. 2017, ApJ, 837, L28, doi: 10.3847/2041-8213/aa603d
  * Chaushev et al. (2019) Chaushev, A., Raynard, L., Goad, M. R., et al. 2019, MNRAS, 488, 5232, doi: 10.1093/mnras/stz2058
  * Choi et al. (2019) Choi, J., Chun, D., Kim, H., & Lee, H.-J. 2019, in Proceedings of the IEEE/CVF International Conference on Computer Vision, 502–511
  * Cooke et al. (2018) Cooke, B. F., Pollacco, D., West, R., McCormac, J., & Wheatley, P. J. 2018, A&A, 619, A175, doi: 10.1051/0004-6361/201834014
  * Dattilo et al. (2019) Dattilo, A., Vanderburg, A., Shallue, C. J., et al. 2019, AJ, 157, 169, doi: 10.3847/1538-3881/ab0e12
  * Eisner et al. (2021) Eisner, N. L., Barragán, O., Lintott, C., et al. 2021, MNRAS, 501, 4669, doi: 10.1093/mnras/staa3739
  * Feigelson et al. (2018) Feigelson, E. D., Babu, G. J., & Caceres, G. A. 2018, Frontiers in Physics, 6, 80, doi: 10.3389/fphy.2018.00080
  * Feinstein et al. (2020) Feinstein, A. D., Montet, B. T., Ansdell, M., et al. 2020, AJ, 160, 219, doi: 10.3847/1538-3881/abac0a
  * Fischer et al. (2012) Fischer, D. A., Schwamb, M. E., Schawinski, K., et al. 2012, MNRAS, 419, 2900, doi: 10.1111/j.1365-2966.2011.19932.x
  * Foreman-Mackey et al. (2016) Foreman-Mackey, D., Morton, T. D., Hogg, D. W., Agol, E., & Schölkopf, B. 2016, AJ, 152, 206, doi: 10.3847/0004-6256/152/6/206
  * Girshick et al. (2014) Girshick, R., Donahue, J., Darrell, T., & Malik, J. 2014, in 2014 IEEE Conference on Computer Vision and Pattern Recognition, 580–587, doi: 10.1109/CVPR.2014.81
  * Goodfellow et al. (2014) Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., et al. 2014, Advances in Neural Information Processing Systems, 3, 2672
  * Guerrero et al. (2021) Guerrero, N. M., Seager, S., Huang, C. X., et al. 2021, ApJS, 254, 39, doi: 10.3847/1538-4365/abefe1
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., et al. 2020, Nature, 585, 357–362, doi: 10.1038/s41586-020-2649-2
  * Hinners et al. (2018) Hinners, T. A., Tat, K., & Thorp, R. 2018, AJ, 156, 7, doi: 10.3847/1538-3881/aac16d
  * Hippke & Heller (2019) Hippke, M., & Heller, R. 2019, A&A, 623, A39, doi: 10.1051/0004-6361/201834672
  * Hochreiter & Schmidhuber (1997) Hochreiter, S., & Schmidhuber, J. 1997, Neural Computation, 9, 1735, doi: 10.1162/neco.1997.9.8.1735
  * Huang et al. (2013) Huang, X., Bakos, G. Á., & Hartman, J. D. 2013, MNRAS, 429, 2001, doi: 10.1093/mnras/sts463
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science & Engineering, 9, 90, doi: 10.1109/MCSE.2007.55
  * Kawahara & Masuda (2019) Kawahara, H., & Masuda, K. 2019, AJ, 157, 218, doi: 10.3847/1538-3881/ab18ab
  * Kluyver et al. (2016) Kluyver, T., Ragan-Kelley, B., Pérez, F., et al. 2016, in Positioning and Power in Academic Publishing: Players, Agents and Agendas, ed. F. Loizides & B. Schmidt, IOS Press, 87 – 90
  * Kovács et al. (2002) Kovács, G., Zucker, S., & Mazeh, T. 2002, A&A, 391, 369, doi: 10.1051/0004-6361:20020802
  * Kreidberg (2015) Kreidberg, L. 2015, PASP, 127, 1161, doi: 10.1086/683602
  * Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., & Hinton, G. E. 2012, Advances in neural information processing systems, 25, 1097
  * Kunimoto et al. (2018) Kunimoto, M., Matthews, J. M., Rowe, J. F., & Hoffman, K. 2018, AJ, 155, 43, doi: 10.3847/1538-3881/aaa005
  * Lecun et al. (1998) Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. 1998, Proceedings of the IEEE, 86, 2278, doi: 10.1109/5.726791
  * Leung & Bovy (2019) Leung, H. W., & Bovy, J. 2019, MNRAS, 483, 3255, doi: 10.1093/mnras/sty3217
  * Lightkurve Collaboration et al. (2018) Lightkurve Collaboration, Cardoso, J. V. d. M., Hedges, C., et al. 2018, Lightkurve: Kepler and TESS time series analysis in Python, Astrophysics Source Code Library.  http://ascl.net/1812.013
  * Lissauer et al. (2011) Lissauer, J. J., Ragozzine, D., Fabrycky, D. C., et al. 2011, ApJS, 197, 8, doi: 10.1088/0067-0049/197/1/8
  * Liu et al. (2019) Liu, H., Liu, C., Wang, J. T. L., & Wang, H. 2019, ApJ, 877, 121, doi: 10.3847/1538-4357/ab1b3c
  * Maas et al. (2013) Maas, A. L., Hannun, A. Y., & Ng, A. Y. 2013, in ICML Workshop on Deep Learning for Audio, Speech, and Language Processing
  * Mandel & Agol (2002) Mandel, K., & Agol, E. 2002, ApJ, 580, L171, doi: 10.1086/345520
  * MegEngine Collaboration (2014) MegEngine Collaboration. 2014, MegEngine.  https://github.com/MegEngine/MegEngine
  * Mosteller & Tukey (1977) Mosteller, F., & Tukey, J. 1977, Data Analysis and Regression: A Second Course in Statistics, Addison-Wesley series in behavioral science (Addison-Wesley Publishing Company)
  * Muthukrishna et al. (2019) Muthukrishna, D., Narayan, G., Mandel, K. S., Biswas, R., & Hložek, R. 2019, PASP, 131, 118002, doi: 10.1088/1538-3873/ab1609
  * Olmschenk et al. (2021) Olmschenk, G., Ishitani Silva, S., Rau, G., et al. 2021, AJ, 161, 273, doi: 10.3847/1538-3881/abf4c6
  * Osborn et al. (2016) Osborn, H. P., Armstrong, D. J., Brown, D. J. A., et al. 2016, MNRAS, 457, 2273, doi: 10.1093/mnras/stw137
  * Osborn et al. (2020) Osborn, H. P., Ansdell, M., Ioannou, Y., et al. 2020, A&A, 633, A53, doi: 10.1051/0004-6361/201935345
  * Pan & Yang (2009) Pan, S. J., & Yang, Q. 2009, IEEE Transactions on knowledge and data engineering, 22, 1345
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., et al. 2019, in Advances in Neural Information Processing Systems 32, ed. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, & R. Garnett (Curran Associates, Inc.), 8024–8035
  * Pearson et al. (2018) Pearson, K. A., Palafox, L., & Griffith, C. A. 2018, MNRAS, 474, 478, doi: 10.1093/mnras/stx2761
  * Redmon & Farhadi (2018) Redmon, J., & Farhadi, A. 2018, arXiv e-prints, arXiv:1804.02767.  https://arxiv.org/abs/1804.02767
  * Ren et al. (2017) Ren, S., He, K., Girshick, R., & Sun, J. 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence, 39, 1137, doi: 10.1109/TPAMI.2016.2577031
  * Rosenfeld & Thurston (1971) Rosenfeld, A., & Thurston, M. 1971, IEEE Transactions on Computers, C-20, 562, doi: 10.1109/T-C.1971.223290
  * Rowe et al. (2014) Rowe, J. F., Bryson, S. T., Marcy, G. W., et al. 2014, ApJ, 784, 45, doi: 10.1088/0004-637X/784/1/45
  * Rumelhart et al. (1985) Rumelhart, D. E., Hinton, G. E., & Williams, R. J. 1985, Learning internal representations by error propagation, Tech. rep., California Univ San Diego La Jolla Inst for Cognitive Science
  * Rundo et al. (2019) Rundo, F., Rinella, S., Massimino, S., et al. 2019, Computation, 7, 13
  * Schanche et al. (2019) Schanche, N., Collier Cameron, A., Hébrard, G., et al. 2019, MNRAS, 483, 5534, doi: 10.1093/mnras/sty3146
  * Schawinski et al. (2017) Schawinski, K., Zhang, C., Zhang, H., Fowler, L., & Santhanam, G. K. 2017, MNRAS, 467, L110, doi: 10.1093/mnrasl/slx008
  * Shallue & Vanderburg (2018) Shallue, C. J., & Vanderburg, A. 2018, AJ, 155, 94, doi: 10.3847/1538-3881/aa9e09
  * Tian et al. (2019) Tian, Y., Yang, G., Wang, Z., et al. 2019, Computers and electronics in agriculture, 157, 417
  * Twicken et al. (2016) Twicken, J. D., Jenkins, J. M., Seader, S. E., et al. 2016, AJ, 152, 158, doi: 10.3847/0004-6256/152/6/158
  * Uehara et al. (2016) Uehara, S., Kawahara, H., Masuda, K., Yamada, S., & Aizawa, M. 2016, ApJ, 822, 2, doi: 10.3847/0004-637X/822/1/2
  * Villanueva et al. (2019) Villanueva, Steven, J., Dragomir, D., & Gaudi, B. S. 2019, AJ, 157, 84, doi: 10.3847/1538-3881/aaf85e
  * Yao et al. (2021) Yao, X., Pepper, J., Gaudi, B. S., et al. 2021, AJ, 161, 124, doi: 10.3847/1538-3881/abdb30
  * Yu et al. (2019) Yu, L., Vanderburg, A., Huang, C., et al. 2019, AJ, 158, 25, doi: 10.3847/1538-3881/ab21d6
  * Yurtsever et al. (2020) Yurtsever, E., Lambert, J., Carballo, A., & Takeda, K. 2020, IEEE Access, 8, 58443, doi: 10.1109/ACCESS.2020.2983149
  * Zhang et al. (2020) Zhang, J., Liao, Y., Zhu, X., Wang, H., & Ding, J. 2020, IEEE Signal Processing Letters, 27, 276, doi: 10.1109/LSP.2020.2966888
  * Zucker & Giryes (2018) Zucker, S., & Giryes, R. 2018, AJ, 155, 147, doi: 10.3847/1538-3881/aaae05


Iterative Sizing Field Prediction for Adaptive Mesh Generation From Expert Demonstrations

Niklas Freymuthalr
Philipp Dahlingeralr
Tobias Würthfast
Philipp Beckeralr
Aleksandar Taranovicalr
Onno Grönheimevago
Luise Kärgerfast
Gerhard Neumannalr

fastInstitute of Vehicle Systems Technology, Karlsruhe Institute of Technology, Karlsruhe
alrAutonomous Learning Robots, Karlsruhe Institute of Technology, Karlsruhe, Germany
evagoEVAGO GmbH, Leonberg, Germany

Niklas<EMAIL_ADDRESS>
Geometric Deep Learning, Graph Network Simulation, Meta-Learning

Many engineering systems require accurate simulations of complex physical systems.
Yet, analytical solutions are only available for simple problems, necessitating numerical approximations such as the Finite Element Method (FEM).
The cost and accuracy of the FEM scale with the resolution of the underlying computational mesh.
To balance computational speed and accuracy meshes with adaptive resolution are used, allocating more resources to critical parts of the geometry.
Currently, practitioners often resort to hand-crafted meshes, which require extensive expert knowledge and are thus costly to obtain.
Our approach, Adaptive Meshing By Expert Reconstruction (AMBER), views mesh generation as an imitation learning problem.
AMBER combines a graph neural network with an online data acquisition scheme to predict the projected sizing field of an expert mesh on a given intermediate mesh, creating a more accurate subsequent mesh.
This iterative process ensures efficient and accurate imitation of expert mesh resolutions on arbitrary new geometries during inference.
We experimentally validate AMBER on heuristic $2$D meshes and $3$D meshes provided by a human expert, closely matching the provided demonstrations and outperforming a single-step CNN baseline.

§ INTRODUCTION

Simulating complex physical systems is an integral part of most engineering disciplines.
These simulations build on fundamental laws of physics, such as conservation of mass and energy, and are often expressed through pde.
Due to their complexity, these pde are analytically intractable, requiring numerical approximations such as the fem <cit.>.
In the fem, a continuous problem geometry is partitioned into a mesh, i.e., a set of finite and simple elements.
Such a mesh allows efficient approximations of the pde solution, with cost and accuracy scaling with the number of elements.

For problems of realistic complexity, naively constructed meshes are often insufficient and adaptive meshes are required.
Those can have varying element sizes allowing for a finer resolution in regions of interest and coarser resolution in areas that are simpler to simulate.
To find appropriate meshes amr methods can refine existing meshes by allocating elements based on heuristics.
Similarly, amg creates meshes based on the problem geometry and initial conditions, often omitting the need for intermediate solutions.
Yet, both approaches are limited in efficiency and adaptability <cit.> and the required heuristics are often costly <cit.>.
Due to these challenges of existing tools, generating meshes for real-world applications still requires tedious manual labor and extensive domain knowledge.

Several recent approaches have explored amr with machine learning.
One popular paradigm uses rl <cit.> to frame amr as a sequential decision-making process <cit.>.
While promising, these rl approaches rely on application-specific and complex reward functions <cit.> or expensive, fine-grained uniform reference meshes <cit.>.
These factors, together with other constraints, such as a maximum refinement depth of the reference meshes <cit.> limit the practical applicability of these approaches.
Alternatively, recent supervised approaches aim to learn from expert meshes, using standard cnn <cit.> or mlp <cit.> to predict the target mesh.
However, these approaches are limited by the inherent properties of cnn, such as a fixed resolution or missing rotational equivariance.

In this work, we instead frame amg as an imitation learning problem, aiming to create a mesh that matches expert behavior by iteratively predicting a sizing field on an intermediate mesh to create a more accurate next mesh.
Our approach, amber, uses a gnn that consumes a graph representation of the current mesh and outputs a piecewise-constant sizing field on the mesh elements.
During inference, it then takes a geometry and a coarse initial mesh and iteratively improves this mesh by predicting a sizing field that is used to create the next mesh.
During training, this predicted sizing field is trained to be close to a sizing field that is projected from the expert mesh using a simple regression loss.
As the algorithm can create arbitrary intermediate meshes during inference, we maintain a replay buffer <cit.> during training, regularly adding meshes produced by our algorithm to this buffer and training on them.
Training on these buffered meshes instead of only using the initial meshes ensures there is no distribution shift between the simple regression during training and the iterative mesh generation procedure during inference.
We automatically label these intermediate meshes by projecting the expert sizing field onto them.
This process resembles the DAgger <cit.> in which an expert relabels out of distribution samples.
In  amber, we implement a similar approach, however, we use an oracle for re-labeling.
Figure <ref> provides a schematic overview.

We experimentally validate our approach against a cnn baseline on heuristic expert meshes generated on a series of $2$D Poisson Equations on randomly generated L-Shaped geometries.
Additionally, we experiment on a set of complex real-world $3$D geometries with meshes created by a human expert.
We find that amber is particularly well-suited for highly adaptive meshes and produces accurate imitations of both heuristic and human expert behavior, regardless of the underlying mesh resolution.
We further show the effectiveness of amber's individual design choices through a series of ablations.
[Code and datasets are available at

To summarize our contributions, we
amber, a novel imitation learning approach that generates expert-like meshes through a series of sizing field predictions,
challenging $2$D and $3$D datasets consisting of heuristic and human expert meshes and corresponding geometries, and
our method on these datasets, demonstrating high similarity to the provided expert meshes and outperforming a strong cnn-based baseline.

§ RELATED WORK

Meshing for Simulation.
The femfem is a well-established method to numerically approximate physical systems, especially on complex, irregular geometric problem domains <cit.>.
The fem solves a system of equations by dividing the geometry into a mesh consisting of smaller, simple elements.
Here, the simulation cost and accuracy directly depend on the mesh size, often necessitating an adaptive mesh that focuses more elements on important parts of the geometry for efficient simulations <cit.>.
Modern meshing approaches can be categorized into amramr <cit.>, which adapts a given mesh, and amgamg <cit.>, which generates a new mesh from an estimated sizing field or related properties of the geometry.
These traditional amr approaches rely on heuristics <cit.> or error estimates <cit.>, which can quickly become inaccurate, unreliable or computationally expensive <cit.>.

Learning Based Approaches for Meshing.
In recent years, gnn <cit.>, and particularly mpn, have become popular architectures for learning to simulate on meshes.
These gns have been applied to mesh-based deformable <cit.> and rigid <cit.> object simulations.
We also employ mpn acting on meshes to improve the speed and accuracy of physical simulations.
Yet, instead of directly learning to simulate, we generate application-specific efficient meshes for downstream classical fem solvers, which is more robust and safe as the actual simulation is done numerically.
Here, a recent suite of rl-based amr approaches has emerged for creating adaptive meshes <cit.>.
These methods use an rl policy to iteratively determine which mesh elements to subdivide.
They typically depend on complex and costly reward functions, which need careful design to handle complex problems <cit.>.
Despite this, the reward function requires a concrete underlying system of equations, restricts the maximum mesh resolution <cit.> or encodes a specific refinement criterion <cit.>.
These limitations are sub-optimal for real-world applications, where meshes often need to be highly locally refined, e.g., due to localized large gradients <cit.>.
We instead learn to refine based on expert meshes, omitting the need for a reward function and thus alleviating its limitations.

Other works consider supervised learning for mesh refinement.
These approaches use recurrent neural networks to learn mesh refinement strategies <cit.>,
optimize element stretching ratio and direction from a given error estimation <cit.>, and employ hand-crafted features to directly compute error estimates to solve an adjoint problem <cit.>.
Further work trains a surrogate model with supervised learning for predicting the solution error <cit.>, which in turn can be used for amg by setting the sizing field to the scaled inverse of the error estimation <cit.>.
Other approaches directly predict a sizing field using mlp for simulations of magnetic devices <cit.>, and cnn for fluid dynamics problem <cit.>.

Interactive Imitation Learning.

il is a common paradigm to learn behavior from expert demonstrations <cit.>, which is particularly applicable in scenarios where a reward function is difficult to define or can lead to undesired behavior <cit.>.
A major challenge in il occurs when the model is confronted with data absent from the expert data as it diverges from the expert behavior.
Interactive il methods, such as DAgger <cit.>, address this issue by allowing experts to intervene and label new data.
In some variants of this approach <cit.>, the model actively requests new data from the expert when needed to improve efficiency further.
Similarly, amber also actively acquires new data during learning.
However, instead of querying a human for new expert meshes, we maintain a static expert dataset and automatically generate and label new intermediate meshes based on the induced expert sizing fields.

§ ADAPTIVE MESHING BY EXPERT RECONSTRUCTION (AMBER)

amber views mesh generation as an iterative imitation problem aiming to match an expert mesh based on a series of intermediate meshes.
In contrast to previous amr methods <cit.>, we learn directly from meshes optimized for their respective downstream applications by experts.
This view greatly simplifies the mesh generation process, as our problem consists of matching an expert rather than optimizing for problem-specific and potentially complex adaptive mesh criteria <cit.>.
We approach this problem by training a mpn to iteratively predict sizing fields on a series of intermediate meshes, generating a new mesh from each prediction.
These projections are regressed against a projected expert sizing field, which is dynamically produced for meshes that are added to a replay buffer during training <cit.>.
Figure <ref> provides a schematic overview of amber, while the following sections describe its different aspects in more detail.

§.§ Preliminaries

We are given a training dataset $\mathcal{D}=\{(\Omega, M^*)\}_{n=1}^N$ consisting of pairs of a geometry $\Omega$ and a corresponding expert mesh $M^*$.
Each geometry describes a physical body in $2$D or $3$D, and the mesh discretizes this body into a number of simplical elements $M_j^*$.
We aim to generate a mesh $M^T$ that imitates the expert mesh $M^*$, i.e., that minimizes a distance $d(M^T, M^*)$ for unseen meshes during inference.

This generation is done using out-of-the-box approaches.
In this work, we use the Frontal Delaunay algorithm implemented in gmsh <cit.>.
The mesh generator consumes a sizing field and a geometry and returns a mesh that conforms to this sizing field.
It additionally optimizes the mesh with respect to certain element criteria, such as the element's aspect ratio, which results in a smooth transition between element sizes.
In turn, the sizing field is a function $\Omega\rightarrow\mathbb{R}$ that describes the average desired edge length of the mesh's elements in space.
We can thus precisely a given mesh by the piece-wise constant sizing field that is induced by its elements.
Given the volume $V(M_i)$ of the $d$-dimensional simplical element $M_i$, its corresponding sizing field is given as the average edge length

\begin{equation*}
f(M_i)=\left(V(M_i) \frac{d!}{\sqrt{d+1}}\right)^{\frac{1}{d}}\text{.}
\end{equation*}

To obtain a graph representation, we view the mesh elements as nodes $\mathcal{V}$ and their neighborhood relations as edges $\mathcal{E} \subseteq \mathcal{V} \times \mathcal{V}$.
We then construct a bidirectional graph $\mathcal{G}_{\Omega^t} = \mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathbf{x}_{\mathcal{V}}, \mathbf{x}_{\mathcal{E}})$ using node features $\mathbf{x}_{\mathcal{V}}$ and edge features $\mathbf{x}_{\mathcal{E}}$.
We process the graph using a mpn <cit.>, a type of gnn <cit.>.
mpn encompass the function class of several classical pde solvers <cit.>, making them a popular choice for learning representations on meshes <cit.>.
Appendix <ref> provides details for the mpn architecture.

§.§ Mesh generation with AMBER

When generating a mesh for a given geometry, amber starts with a coarse uniform initial mesh $M^t$ for $t=0$.
It then encodes the current mesh $M^t$ as a graph and feeds it through an mpn to predict a target size per mesh element.
The combination of these target sizes forms a piece-wise constant sizing field, which together with the underlying geometry is given to a mesh generator, which produces a new mesh $M^{t+1}$.
This process is iterated over $T$ steps, resulting in a final mesh $M^T$.

This iteration is necessary, as the current mesh limits the resolution of the predicted sizing field in every step.
As we can only learn one target size for each current element of $M^t$, the corresponding finer resolution elements in $M^{t+1}$ will all have roughly the same size.
However, by iterating this process, the final mesh converges to an appropriate mesh, even for problems with highly non-uniform meshes.

§.§ Training amber

amber's iterative mesh generation process necessitates a specialized training procedure.
To compute the target sizing field for an element $M_i^t\subseteq M^t$ of an intermediate mesh $M^t$, we first determine all expert elements $M_j^*\in M^*$ whose midpoints $p_{M_j^*}$ lie in $M_i^t$, and then aggregate over the sizing fields induced by these elements, i.e.,

\begin{equation*}
y(M_i^t) = \bigotimes_{j} \mathbb{I}(p_{M_j^*}\in M_i^t) f(M_j^*)\text{.}
\end{equation*}

Here, $\bigotimes$ is a permutation-invariant aggregation function and we consider using either a maximum or a mean operator.
This results in two variants, called amber (Mean) and amber (Max.).
Intuitively, we thus set each element's target element size to either the average or the maximum size of all of the expert elements it contains.
As the mean target size is always smaller or equal to the maximum target size, amber (Mean) tends to refine more aggressively than amber (Max) resulting in faster convergence while producing more and potentially unnecessary elements in the process.
In both cases, the target sizes for each element of the intermediate mesh will be smaller than their actual size, and iterating this process converges to the expert mesh.

If an element in $M^t$ to contains no midpoint $p_{M_j^*}$, we set its target sizing field to that of the element in the expert mesh that contains their midpoint.
Further, different meshes may approximate the same underlying geometry differently, leading to elements in one mesh that lie outside of the other.
We thus assign elements in the expert mesh that lie outside of the intermediate mesh to their nearest neighbor in the intermediate mesh.
This approximation ensures that all elements in the fine mesh contribute to the target sizing field of the intermediate mesh, and becomes more accurate the more closely the two meshes align.
Once the targets sizing field are obtained, we use them to train on a simple node-level mse loss.
This training scheme allows us to equally apply amber to both problem-specific meshes optimized for a given pde, and general-purpose meshes intended for different downstream analyses.

amber produces a series of intermediate meshes in an auto-regressive fashion during inference, but only trains on individual sizing field predictions.
To prevent distribution shifts during the iterative mesh generation, we thus maintain a replay buffer <cit.> over amber-generated intermediate meshes during training.
This replay buffer contains a fixed set of one coarse initial mesh per provided training sample.
Every $k$ training steps, we sample a mesh from the replay buffer, predict its projected sizing field, generate a new mesh from this prediction, project the expert field onto this new mesh to generate its labels, and store this new pair of meshes and labels in the buffer.
To stabilize the training process, we assign each intermediate mesh a `depth' that corresponds to the number of generating steps it has gone through, and store an equal amount of meshes for each depth up to a maximum.
Each training step consists of randomly drawing a number of meshes from the buffer.
Since the meshes greatly vary in size, we fill a batch until a number of nodes is reached, rather than using a fixed number of graphs.
Appendix <ref> provides details on the stratified replay buffer and the batch generation process.

§ EXPERIMENTS

To use an mpn for meshes, we set the node features to the corresponding element's volume and the edge features $\mathbf{x}_{\mathcal{E}}$ are to the Euclidean distance between element midpoints.
mpn are permutation-equivariant by design and as we only provide Euclidean distances as spatial information, we additionally obtain equivariance to the Euclidean group <cit.>.
Since the sizing field's values are always positive we apply a softplus transformation to the mpn outputs, which greatly stabilizes training.
Similarly, we normalize the input graph features using the statistics of the replay buffer.
Appendix <ref> provides further details of the architecture, training procedure, and hyperparameters.

We repeat all methods for $4$ random seeds.
We evaluate on a set of unseen test geometries, and evaluate the quality of each generated mesh by comparing it to a reference expert mesh on this geometry.
For this comparison, we use the dcd <cit.> over the sets of element midpoints of both meshes.
The dcd is defined as a symmetric, exponentiated Chamfer distance with normalization factors to account for elements in one set that match multiple elements in the other.
Intuitively, it measures the closest exponential distance of every midpoint in one mesh to the other mesh, weighting this distance by the number of matches.
It is bound in $[0,1]$, and $0$ if and only if both sets are the same. We provide a mathematical description of the dcd in Appendix <ref>.
We also explore an alternate metric that numerically integrates the difference in element volumes between the two meshes.

Poisson's Equation.
We consider problem-specific meshes using Poisson's Equation with a load function $f(\boldsymbol{x})$ and test function $v(\boldsymbol{x})$, given as

\begin{align*}
\int_\Omega \nabla u \cdot \nabla v\text{d}\boldsymbol{x} = \int_\Omega f v\text{d}\boldsymbol{x} \quad \forall v \in V\text{.}
\end{align*}

We enforce zero Dirichlet boundary conditions, i.e., .
As the training data, we randomly generate L-shaped geoemtries and employ a Gaussian Mixture Model with three components for the load function.
For this task, we evaluate the load function at each face's midpoint and use it as a node feature.
Additionally, we solve the equation for every intermediate mesh and use the mean and standard deviation of the solution on each element's vertices as additional node features.
We generate the problem-specific expert meshes using an error-based refinement heuristic <cit.>, which marks all elements for refinement for which $\text{err}(M_i)>\theta\cdot\max_j \text{err}(M_j)$ for $0<\theta<1$.
The error is estimated heuristically based on the load function and gradient jumps on the element facets.
We use $20$ randomly drawn systems of equations and corresponding expert meshes as the training data, and test on a disjoint, fixed set of $100$ unseen system of equations and expert meshes.
We vary the task's difficulty by adapting the number of refinement steps that the heuristic applies, considering easy, medium, and hard variants with $25$, $50$, and $75$ refinement steps respectively.
Appendix <ref> provides further details of the dataset creation and the expert heuristic.

Console Task.
In the Console task, we use data obtained from a real-world scenario in the automotive industry.
We have a parameterized family of $3$D geometries representing a car's seat crossmembers.
The geometries are obtained using Onshape[<https://www.onshape.com/>] and feature various sharp bends as well as up to $2$ circular holes.
Tetrahedral meshes for this task are generated by human experts using ANSA[<https://www.beta-cae.com/ansa.htm>].
The experts are initially presented with a coarse mesh, on which they iteratively select regions to refine, specifying the target element size of each selected region.
The resulting meshes are optimized for downstream strength and durability analyses, but our experiments are conducted solely on the meshes and their underlying geometry.
The middle of Figure <ref> provides an example close-up of an expert mesh.
The final dataset contains $22$ meshes and corresponding geometries.
The meshes range from $6,284$ to $41,856$ elements and are randomly split into fixed sets of $15$ training, $2$ evaluation, and $5$ test meshes.

Baselines and Ablations.
We compare amber to a cnn-baseline that takes a binary image mask of the geometry and outputs a pixel-wise sizing field on the geometry.
This baseline is conceptually similar to <cit.>, but with slight adaptations that improve the training stability and performance.
Instead of smoothing the expert sizing field and evaluating each pixel, we create an auxiliary mesh with roughly one element per pixel, calculate the target sizing field on this mesh as done in Section <ref>, and map the resulting labels to each pixel.
Since the cnn does no iterative generation, we use the mean label projection method for each pixel.
Further, we train the approach on the logarithm of the sizing field, i.e., set the loss to

\begin{equation}
\label{eq:cnn_loss}
\mathcal{L}_{\text{CNN}}=\frac{1}{N}\sum_{i=1}^N\left(\text{CNN}(x_i)-\log(y_i)\right)^2\text{,}
\end{equation}

and subsequentially exponentiate the cnn's output to recover the predicted sizing field.
This change prevents larger mesh elements from dominating the cnn's loss.
The cnn can only predict one target element size per pixel and we thus train separate cnn on different image resolutions.
We also provide the same input features and normalization as for amber.
Since the training sets are relatively small, we use all images for every training step, performing a total of $25,600$ training steps or equivalently epochs of training.

§ RESULTS

Qualitative Results.
Figure <ref> visualizes exemplary amber refinements trained on the Poisson task with $50$ oracle heuristic refinement steps.
On the top row, amber (Mean) converges to a relatively accurate mesh after only $2$ prediction steps.
However, the approach tends to locally over-refine some elements, causing potentially unnecessary refinements in intermediate meshes.
amber (Max.) is shown on the bottom row.
It produces fewer elements per step and thus requires more iterations until it converges to the expert mesh.
Figure <ref> shows amber (Mean) refinements for the same test example when trained on expert meshes that use $25$, $50$ and $75$ heuristic refinement steps.
Our approach closely matches the expert heuristic, producing meshes of varying granularity depending on the expert meshes that it sees during training.
Figure <ref> compares amber (Max) and the cnn baseline to an expert mesh on an exemplary test geometry on the Console task, finding that amber matches the expert's local element resolution well, while the cnn produces more uniform meshes and does not provide accurate sizing field predictions for the different geometric features.
Appendix <ref> provides visualizations for amber and the cnn baseline for all tasks.

Quantitative Results.
We measure mesh quality via the similarity to the expert mesh using the normalized dcddcd <cit.> for amber for both mean and max sizing field interpolations and the cnn baseline for different image resolutions.
Figure <ref> evaluates Poisson's Equation for $25$, $50$ and $75$ heuristic refinement steps of the expert mesh.
While the cnn baseline works well for $25$ and even $50$ steps, its fixed image resolution eventually causes issues as the expert meshes become more complex.
In contrast, amber directly acts on intermediate meshes of arbitrary resolution.
Here, amber works well on all $3$ task difficulties, as the method learns to adapt to the complexity of the expert's meshes.
This trend is more pronounced on the $3$D console task, as shown on the left side of Figure <ref>.
amber naturally generalizes to the more complex $3$D setting as it directly acts on meshes.
Thus, each amber forward pass is linear in the number of mesh elements.
In contrast, the cnn baseline quickly becomes computationally expensive due to cubic scaling with the number of voxels per dimension.
Finally, Figure <ref> shows the benefits of amber's iterative mesh generation process, finding that both amber (Mean) and amber (Max) benefit from up to $5$ intermediate mesh generation steps.
Appendix <ref> evaluates all tasks on an alternate metric that measures the difference in element volumes between the two meshes, yielding consistent results.

Further Experiments.
Figure <ref> shows that amber performs slightly worse when trained on the cnn loss (c.f. Equation <ref>), likely because amber already implicitly weights small elements more strongly as there are more of them.
In contrast, the cnn baseline performs worse on amber's mse loss, presumably because the same relative error makes a larger difference in mesh generation for pixels responsible for smaller elements when using this loss.
Omitting the softplus transformation of the output on this loss, which results in a method similar to that of <cit.>, further decreases stability and performance.
Finally, using a maximum operator to project the sizing fields for the cnn baseline leads to more conservative target estimation and thus too few elements, yielding worse results.
We explore additional variants and ablations of amber in Appendix <ref> and find that amber benefits from additional data but provides strong generalization performance to unseen scenarios from as few as $5$ expert meshes on Poisson's Equation.
Appendix <ref> validates the effectiveness of normalizing the input features, using a softplus transformation to predict the strictly positive sizing field, and maintaining an equal number of meshes for each intermediate generation step instead of randomly adding samples.

§ CONCLUSION

We introduce amberamber, an iterative adaptive mesh generation algorithm based on imitation learning.
amber combines a Message Passing Graph Neural Network and a replay buffer with automatically labeled intermediate meshes to learn how to imitate meshes produced by experts.
During mesh generation, amber takes an arbitrary problem geometry and generates a series of increasingly accurate intermediate meshes by predicting a sizing field on each mesh and feeding this sizing field to an out-of-the-box mesh generator.
Rather than optimizing any particular metric, amber learns to imitate an expert's meshing behavior from examples, enabling it to create suitable meshes for various engineering applications.

Experimental results on heuristic $2$D meshes on Poisson problems, and human expert meshes on real-world $3$D geometries demonstrate the strong performance of our approach.
amber is data efficient and generates highly accurate meshes, significantly outperforming a strong CNN baseline when evaluated on more complex task setups.

Limitations and Future Work.
amber requires a dataset with consistent expert behavior and always produces meshes that align with this single behavior.
We plan to extend amber to allow conditioning the mesh generation process on a desired expert behavior, e.g. producing varying numbers of final elements with the same model <cit.>.
Additionally, we aim to replace the current piece-wise constant sizing field over the elements with a piece-wise linear sizing field over the mesh's vertices.
This change reduces the size of the mesh's graph representation while increasing the information provided in each sizing field.

§ ACKNOWLEDGEMENTS

NF was supported by the BMBF project Davis (Datengetriebene Vernetzung für die ingenieurtechnische Simulation).
This work is also part of the DFG AI Resarch Unit 5339 regarding the combination of physics-based simulation with AI-based methodologies for the fast maturation of manufacturing processes. The financial support by German Research Foundation (DFG, Deutsche Forschungsgemeinschaft) is gratefully acknowledged. The authors acknowledge support by the state of Baden-Württemberg through bwHPC, as well as the HoreKa supercomputer funded by the Ministry of Science, Research and the Arts Baden-Württemberg and by the German Federal Ministry of Education and Research.

§ MESSAGE PASSING NETWORKS

mpn iteratively update latent features for the graph nodes and edges over $L$ message-passing steps.
Using linear embeddings $\mathbf{x}_v^0=\mathbf{x}_v$ and $\mathbf{x}_e^0=\mathbf{x}_e$ of the initial node and edge features, each step $l$ builds on the previous one, computing features

\mathbf{x}^{l+1}_{e} = f^{l}_{\mathcal{E}}(\mathbf{x}^{l}_v, \mathbf{x}^{l}_u, \mathbf{x}^{l}_{e}), \textrm{ with } e = (u, v)\text{,}

\mathbf{x}^{l+1}_{v} = f^{l}_{\mathcal{V}}(\mathbf{x}^{l}_{v}, \bigoplus_{e=(v,u)\in \mathcal{E}} \mathbf{x}^{l+1}_{e})\text{.}

The operator $\bigoplus$ is a permutation-invariant aggregation, such as sum, mean, or maximum operator.
Each $f^l_\cdot$ is parameterized as a learned mlp.
The final output of the mpn is a learned representation $\mathbf{x}^L_v$ for each node $v \in \mathcal{V}$.
We feed this representation into a decoder mlp to yield a scalar value per node that we train to approximate the target sizing field per mesh element.
Each $f^l_\cdot$ is a learned function, usually parameterized as a simple mlp. The final output of the mpn is a learned representation $\mathbf{x}^L_v$ for each node $v \in \mathcal{V}$.

§ AMBER TRAINING DETAILS

§.§ Replay Buffer

To prevent memory issues during training, we only add new meshes to the buffer if their total number of elements does not exceed a threshold of $1.2$ times the largest expert mesh.
Additionally, we count the number of generation steps each intermediate mesh has undergone, setting a maximum depth in the buffer to prevent the replay buffer from eventually filling up with too-fine meshes.
We use this depth parameter to ensure an even distribution over mesh depths in the buffer by adding new meshes with a stratified mesh generation procedure.
Here, we first select a target depth uniformly at random, and then choose a random mesh from this depth to select the next mesh to refine and add to the buffer.

§.§ Train Batch Construction

As different meshes may have very different sizes, we do not train on a fixed number of meshes in each batch.
Instead, we define our batch size over the maximum number of nodes in a batch of graphs and fill up batches by iteratively adding a graph without replacement to this batch until the maximum number of nodes is reached.
To prevent smaller graphs from being sampled disproportionally often, we count the number of training steps each graph has been used for, and prefer the least used graphs in the batch construction if possible.

§ METRICS

§.§ Density-Aware Chamfer Metric

This section describes our primary evaluation metric, the normalized dcddcd, in more detail.
The dcd is defined as a symmetric, exponentiated Chamfer distance with normalization factors to account for elements in one set that match multiple elements in the other.

Mathematically, we first define the closest point function

\begin{equation*}
\hat{z}(z, S) = \arg \min_{w \in S} \|z - w\|_2\text{.}
\end{equation*}

Using this function, we define $n_{\hat{z}}$ as the number of matches that $z$ has in the complementary set $\overline{S}$:

\begin{equation*}
n_{\hat{z}} = |\{w \in \overline{S} \mid \hat{w}(w, S) = z\}|\text{.}
\end{equation*}

Given a point $z\in S$, let

\begin{equation*}
C(S, z, \overline{S}) = 1 - \frac{1}{n_{\hat{z}}} e^{-\|z - \hat{z}(z, \overline{S})\|_2},
\end{equation*}

be the contribution function.
The full metric is then defined as

\begin{equation*}
d_{DCD}(S_1, S_2)=\frac{1}{2} \left( \frac{1}{|S_1|} \sum_{x \in S_1} C(S_1, x, S_2) + \frac{1}{|S_2|} \sum_{y \in S_2} C(S_2, y, S_1) \right)\text{.}
\end{equation*}

To accommodate the inaccuracies of the mesh generator, we report a normalized version of this metric as

\begin{equation*}
d_{NDCD}(M^t, M^*)=\frac{d_{DCD}(\{p_{M^t_i}\}, \{p_{M^*_j}\}) - d_{DCD}(\{p_{M^0_i}\}, \{p_{M^*_j}\})}{d_{DCD}(\{p_{\hat{M}_i}\}, \{p_{M^*_j}\}) - d_{DCD}(\{p_{M^0_i}\}, \{p_{M^*_j}\})}\text{,}
\end{equation*}

where $\hat{M}$ is the reconstruction of $M^*$ that we obtain by directly feeding the sizing field of $M^*$ into the mesh generator.

§.§ Volume Difference Metric

In addition to the normalized dcd of Section <ref>, we evaluate the difference in element volumes between the generated mesh $M^T$ and the expert $M^*$.
Let $V(M^T_i)$ be the volume of element $M^T_i$, and $p_{M^T_i}$ be its midpoint. For an index $i$, we define $j^*(i)$ as the index of the element $M^*_{j^*(i)}$ which contains the midpoint $p_{M^T_i}$. Analogously, we set $j^T(i)$ as the index of the element $M^T_{j^T(i)}$ which contains the midpoint $p_{M^*_i}$.
We then calculate the volume difference between two meshes as

\begin{equation}
\label{app_eq:vol_dif_metric}
d_{\text{VD}} = \sum_{\{i \mid M_i^T \subseteq M^T\}} | V(M^T_i) - V(M^*_{j^*(i)}) | + \sum_{\{i \mid M^*_i \subseteq M^*\}} | V(M^*_i) - V(M^T_{j^T(i)}) |
\end{equation}

Intuitively, this metric numerically approximates the integrated absolute difference in element volumes across the generated and the expert mesh using the elements of both meshes as integration points.

Figure <ref> evaluates the volume difference metric of Equation <ref> for all tasks.
While the results for the volume difference metric have more variance than the normalized dcd, they are consistent with that of Figure <ref> and Figure <ref>.

§ TASKS AND DATA COLLECTION

For Poisson's equation, we generate random L-shaped geometries, defined as $(0,1)^2\backslash (p_0\times(1,1))$, where the lower left corner $p_0$ is sampled from $U(0.2, 0.95)$ in $x$ and $y$ direction.
For the load function, we employ a Gaussian Mixture Model with three components.
The mean of each component is drawn from $U(0.1, 0.9)^2$ and re-sampled until it is within the geometry.
We then independently draw diagonal covariances from a log-uniform distribution $\exp(U(\log(0.0001, 0.001)))$ and randomly rotate them to yield a full covariance matrix.
Component weights are generated from $\exp(N(0,1))+1$ and subsequently normalized.
For this task, we evaluate the load function at each face's midpoint and use it as a node feature.
Additionally, we solve the equation for every intermediate mesh, and use the mean and standard deviation of the solution on each element's vertices as additional node features.

We generate the expert meshes using an error-based heuristic on a refinement threshold $\theta$ <cit.>, which marks all elements for refinement for which $\text{err}(M_i)>\theta\cdot\max_j \text{err}(M_j)$.
The error is estimated by a heuristic

$$\text{err}(\Omega^t_i) = h^2 || f ||^2 + h || \left[\left[ \nabla u \cdot \mathbf{n} \right]\right] ||^2\text{,}$$

which assumes high errors in areas with high values of the load function $f$ and with large gradients jumps on the element edges or facets, respectively.
Once elements with a high error are marked, the mesh is processed by a remesher, which compute a conforming refined mesh using the red-green-blue refinement method <cit.>.
We apply Laplacian smoothing after each refinement step.

§ NETWORK ARCHITECTURES AND HYPERPARAMETERS

All methods are trained on an NVIDIA A100 GPU, with each method given a computational budget of up to $48$ hours.
For mesh generation, we clip all predicted sizing field to the minimum sizing field of any expert mesh in the training data.

The mpn of amber consists of $10$ separate message passing steps.
Each message passing step uses separate $2$-layer mlp and LeakyReLU activations for its node and edge updates.
We independently apply residual connections <cit.> and layer normalization <cit.> to the node and edge updates.
The final node features are fed into a $2$-layer mlp decoder.
All mlp use a latent dimension of $64$.
We provide an overview of amber hyperparameters in Table <ref>.

For the cnn baseline, we use a unet <cit.> architecture with $32$ initial channels and $4$ down- and up-convolution blocks.
Each convolution block consists of $2$ convolutions with a kernel size of $3$, followed by batch normalization and a ReLU activation function.
After each down-convolution, we use max-pooling with a kernel size and stride of $2$ to halve the image resolution, and double the number of channels.
This process is reversed for the up-convolutions, and we add skip connections between the respective depths.
We use $2$D and $3$D convolutions, batch normalization and pooling operations for the $2$D and $3$D tasks, respectively.

AMBER hyperparameters and experiment configurations

Parameter     Value

Optimizer     ADAM

Learning rate     $3.0e$-$4$

Latent dimension     $64$

mpn aggregation function     mean

mpn steps     $10$

mpn activation function     Leaky ReLU

mpn edge dropout     $0.1$

mlp layers     $2$

Maximum replay buffer depth     $5$

Maximum buffer size     $1\,000$ meshes

Training steps     $128\,000$

Buffer addition frequency     every $8$ training steps

amber batch size     $400\,000$ graph nodes

amber inference steps     $5$

§ EXTENDED RESULTS

We explore the data efficiency of amber in Figure <ref>.
While the approach continuously improves its predictions when provided with more training data, as few as $5$ training meshes and corresponding geometries are sufficient for accurate mesh generation.
This generalization capabilities are likely a results of the local, per-node mse loss and the mpn network architecture.
Figure <ref> explores design choices of amber's training scheme.
Both input normalization and transforming the output predictions improve performance.
The stratified sampling of new training data for amber's replay buffer improves performance, presumably because this ensures an even distribution of training data across mesh generation iterations, whereas randomly adding new meshes eventually under-represents the early meshes in the training data.

§.§ Visualizations

We present the qualitative results of all methods applied to all tasks.
Figure <ref> displays the results for amber (Mean), Figure <ref> shows the results for amber (Max), and Figure <ref> showcases the predictions for the cnn baseline.