# Leveraging Generative Adversarial Networks to Create Realistic Scanning
Transmission Electron Microscopy Images

Abid Khan  Department of Physics, University of Illinois Urbana-Champaign,
Urbana, Illinois, 61801, United States  These authors contributed equally to
this work  Chia-Hao Lee  Department of Materials Science and Engineering,
University of Illinois Urbana-Champaign, Urbana, Illinois, 61801, United
States  These authors contributed equally to this work  Pinshane Y. Huang
<EMAIL_ADDRESS>Department of Materials Science and Engineering,
University of Illinois Urbana-Champaign, Urbana, Illinois, 61801, United
States  Materials Research Laboratory, University of Illinois Urbana-
Champaign, Urbana, Illinois, 61801, United States  Bryan K. Clark
<EMAIL_ADDRESS>Department of Physics, University of Illinois Urbana-
Champaign, Urbana, Illinois, 61801, United States  Institute for Condensed
Matter Theory and IQUIST and NCSA Center for Artificial Intelligence
Innovation, University of Illinois Urbana-Champaign, Urbana, Illinois, 61801,
United States

###### Abstract

The rise of automation and machine learning (ML) in electron microscopy has
the potential to revolutionize materials research through autonomous data
collection and processing. A significant challenge lies in developing ML
models that rapidly generalize to large data sets under varying experimental
conditions. We address this by employing a cycle generative adversarial
network (CycleGAN) with a reciprocal space discriminator, which augments
simulated data with realistic spatial frequency information. This allows the
CycleGAN to generate images nearly indistinguishable from real data and
provide labels for ML applications. We showcase our approach by training a
fully convolutional network (FCN) to identify single atom defects in a 4.5
million atom data set, collected using automated acquisition in an aberration-
corrected scanning transmission electron microscope (STEM). Our method
produces adaptable FCNs that can adjust to dynamically changing experimental
variables with minimal intervention, marking a crucial step towards fully
autonomous harnessing of microscopy big data.

## Keywords

Cycle Generative Adversarial Network (CycleGAN), deep learning, scanning
transmission electron microscopy (STEM), 2D materials, single-atom defects

## INTRODUCTION

Machine learning (ML) techniques have been widely applied in electron
microscopy for applications such as atom localization [1, 2, 3], defect
identification [4, 5, 6], image denoising [7, 8, 9], determining crystal tilts
and thickness [10, 11, 12], classifying crystal structures [13, 14],
optimizing convergence angles [15], identifying Bragg disks [16], visualizing
material deformations [17], automated microscope alignment [18], and many
others. Several recent reviews [19, 20, 21] provide an overview of new and
emerging opportunities at the interface of electron microscopy and ML.

One of the biggest challenges of ML in materials research is that it requires
a large amount of high quality training data, which must be paired with the
ground truth for supervised learning methods. For example, training a defect
identification network requires both a set of images and a known set of defect
positions in each image. Because manual labelling is extremely time consuming
and prone to human bias and error, the typical approach is to train ML models
using simulated data, which simultaneously produces both images and labels.
However, simulated scanning transmission electron microscopy (STEM) images
deviate non-trivially from experimental images because it is extremely
difficult to accurately reproduce complex experimental factors that impact the
image, including detector noise [22, 23], sample drift and scan distortions
[24, 25, 26], time-dependent alignment errors [27], radiation damage [28], and
surface contamination [29, 30]. These factors make it challenging to train
high-accuracy ML models with simulated data; often times such training
requires considerable manual optimization to generate a usable training set
for a specific experiment data set. Most of these experimental imperfections
can be simulated to certain degree [5, 3], but doing so accurately requires
significant parameter optimization for each set of experimental conditions. To
maximize their accuracy and precision, ML models are typically trained using a
small distribution of simulation parameters, which are carefully chosen to
cover most of the experimental variations. Such models must be re-trained each
time the effective resolution or contrast of the image changes, which may
occur several times a day, even during a single experimental session. Far from
realizing the promise of fully automated image processing, this lack of
generalizability of ML models represents a significant barrier for microscopy
data processing at the scale of big data.

Here, we construct a cycle generative adversarial network (CycleGAN) [31] to
minimize the difference between simulated and experimental STEM data,
producing realistic training data while simultaneously preserving the ground
truth for ML applications. CycleGANs are often used for image-to-image
translation, such as style transfer tasks like converting a photograph into a
Monet-like painting, or changing an image of a zebra into a horse [31]. We
utilize CycleGANs to generate a high quality, CycleGAN-processed training set
for defect identification tasks using a fully convolutional network (FCN) [1,
2, 4, 5]. We find that FCNs trained on CycleGAN-processed images show
comparable defect identification performance with the manually optimized
training set, while requiring much less human intervention.

## RESULTS AND DISCUSSIONS

### Generating realistic images with CycleGANs

Figure 1 shows an array of simulated (Figure 1a–c), CycleGAN-processed (Figure
1d–f), and experimental (Figure 1g–i) annular dark field (ADF) STEM images and
their power spectra for three different materials: monolayer graphene,
monolayer WSe2, and bulk SrTiO3 (see Methods for experimental details). The
simulated images are calculated with the ‘incoSTEM’ package in Computem [32,
33] with minimal parameter optimization. These out-of-the-box simulated images
are a poor qualitative and quantitative match for the experimental data for
two main reasons. First, in contrast with quantitatively accurate multislice
or Bloch wave approaches for STEM image simulation, incoherent approximations
such as incoSTEM are mainly used because they output images nearly
instantaneously, and are thus mainly used to generate initial, qualitative
image simulations. Second, images simulated using any method lack key aspects
of real experimental data, which contain detector noise, drift-induced
distortions, probe jittering, lens aberrations, thickness variations, and
surface contamination or damage.

Next, we generate realistic STEM images (Figure 1d–f) by feeding the simulated
images (Figure 1a–c) into the CycleGAN. We trained a separate CycleGAN for
each material system, because CycleGANs are typically unable to perform image
translation when large shape changes are needed [34]. The CycleGAN-processed
images are similar to the experimental ones in both real and reciprocal space,
including their atom size, noise level, image contrast, and contamination.
These CycleGAN-processed images can be used as high-quality ML training data.
The use of CycleGAN removes the need for manual parameter optimization in the
simulation process, but unlike conventional GAN, preserves the original labels
and associated ground truth. The CycleGANs are trained with both experimental
and out-of-the-box simulated images (see Methods for training details) and do
not require paired training data. This makes CycleGAN particularly useful for
this ‘simulation-to-experiment’ style transfer task, because a large number of
simulated images can be generated without manually matching all atom positions
with the experimental data.

### CycleGAN architecture and optimization

Figure 2 shows the ‘cycle’ structure of our CycleGAN. In our work, the
CycleGAN converts the images between experiment domain $X$ and simulation
domain $Y$. The conversion between domains is commonly called a ‘mapping.’ The
CycleGAN provides two mappings, $G:X\rightarrow Y$, which makes experimental
images look like simulations; and $F:Y\rightarrow X$, which makes simulated
images look experimental.

To implement this pair of mappings, the CycleGAN is built on two separate GANs
connected in a cyclic manner, where each GAN is responsible for converting
images from one domain to the other domain. The main components of our
CycleGAN are neural networks called the ‘generator’ and the ‘discriminator’.
The goal of a generator is to generate high-quality ‘fake’ images from inputs
of another domain, while the goal of a discriminator is to distinguish whether
the output from the generator is real or fake. Hence, the training of a
CycleGAN is essentially the competition between generators and discriminators.
At the end of the training, the final products are two well-trained generators
that act as the domain-to-domain mappings $G$ and $F$. Here, we mainly use the
CycleGAN to map simulated images into experiment-like images. However, as we
show in Supplemental Figure1, the generator $G$ can also be used to
effectively remove imperfections from experimental data to produce simulation-
like, high signal-to-noise ratio images that can be more easily interpreted
through other data-processing methods.

Our CycleGAN contains two generators ($G$ and $F$), two discriminators in
experiment domain $X$ ($D_{x,\text{img}}$ and $D_{x,\text{FFT}}$), and two
discriminators in simulation domain $Y$ ($D_{y,\text{img}}$ and
$D_{y,\text{FFT}}$) (Methods). While GANs typically operate in real space, we
add Fourier space discriminators ($D_{x,\text{FFT}}$ and $D_{y,\text{FFT}}$)
because certain information, such as low-frequency contamination and high-
frequency noise, are better encoded in Fourier space. The generators are then
forced to generate images that can fool not only the real-space image
discriminator, but also the Fourier space discriminator. The FFT
discriminators are critical for ensuring both the real- and k-space
information of the generated STEM images look realistic, especially for images
with lower signal-to-noise ratio. For example, a CycleGAN trained without the
FFT discriminator produces streak artifacts in graphene images and their FFTs,
as shown in Supplemental Figure2.

The optimization of the 6 neural networks ($G$, $F$, and $D$s) is done by
minimizing a set of loss functions (Methods). CycleGANs contain additional
machinery in the form of regularization terms to ensure the stability and
reversibility of the two mappings $F$ and $G$. The first is the cycle-
consistency loss $L_{\text{cyc}}(F,G)$, which regularizes the mappings so that
a full cycle of forward and backward operation returns the input image back to
its original domain and be similar to the initial input. Mathematically,
$L_{\text{cyc}}$ enforces $F(G(x))\approx x$, and $G(F(y))\approx y$ to
preserve major image information while transferring between domains.
Additionally, the identity loss $L_{\text{id}}$ aims to have $F(x)\approx x$
and $G(y)\approx y$, so that images would be mapped to themselves if they are
already in the target domain. Without these two constraints, it is likely that
a trained mapping will map a simulated image into a realistic experiment-like
image that will otherwise have no relation to the original input, i.e. it will
not preserve important features like atomic defects.

### Evaluation metrics of CycleGAN images

While Figure 1 shows that CycleGAN-processed images are a good qualitative
match with the experimental images, we also examined whether these image types
are quantitatively indistinguishable by calculating the Fréchet Inception
Distance (FID) [35] and Kullback–Leibler (KL) divergences [36] between
CycleGAN and experimental images (Figure 3). The FID score is extensively used
in the field of computer vision to evaluate the image quality and the
performance of generative models [37, 38, 39]. The FID score is evaluated by
first converting the images from different data sets into feature vectors with
a convolutional network (Inception v3) trained on the ImageNet [35], and then
calculating the Fréchet distance [40] between the feature vectors. Note that
the feature vectors are extracted through a non-local and non-linear mapping
using a convolutional network, specifically designed for computer vision
tasks. This process captures high-level, emergent features of the STEM images
and allows us to compare the similarity of two different image distributions.
In general, a lower FID score indicates higher similarity and better
performance of the generative model.

We calculate the FID scores of images generated from simulation without noise
(Figure 3b), simulation with manually optimized noise (Figure 3c), and
CycleGAN (Figure 3d). We choose experimental STEM images of WSe2 (Figure 3a)
as our reference for the FID score calculation because the ultimate goal is to
have a generator that outputs realistic, experiment-like images. We find that
the CycleGAN-processed images show the lowest (i.e. best) FID score (FID =
0.35) compared to simulations without noise (FID = 32) and simulations with
manually added noise (FID = 0.73). Table 1 shows the FID scores of graphene,
WSe2, and SrTiO3.

In addition to the FID score comparison, which involves an additional
Inception model, we calculate the KL divergence directly from the pixel
intensity histograms of different image data sets (Figure 3e–h). The KL
divergence ($D_{\text{KL}}$) is a type of statistical distance that measures
the difference between two probability distributions ($P$ and $Q$). It is
computed as $D_{\text{KL}}(P||Q)=\sum P\log(\frac{P}{Q})$, so
$D_{\text{KL}}\rightarrow 0$ when $P$ and $Q$ are similar. In other words,
$D_{\text{KL}}(P||Q)$ is the amount of lost information when approximating $P$
with $Q$, therefore, it is widely used as a similarity metric for
distributions. We find that the $D_{\text{KL}}$ of CycleGAN with respect the
experimental histogram is significantly lower than the others, indicating the
pixel intensity distribution is a better match to the experimental data than
the other data sets. We also analyze the power spectra of the images and find
that CycleGAN-processed images have the closest power distribution to
experimental images (Supplemental Figure3), again indicating the high
similarity between CycleGAN-processed and experimental images.

Finally, as a qualitative way to understand how the noise transfer works for
CycleGANs, we train CycleGANs on one material system and apply it to another
material system. This procedure allows us to separate the noise profile
learned from the underlying crystal structure and visualize it (see
Supplemental Figure 4).

### Defect identification with CycleGAN and FCN

We demonstrate the utility of the CycleGAN-processed, realistic STEM images
for defect identification in a two dimensional (2D) material, monolayer WSe2.
Identifying atomic defects in atomic-resolution STEM data is critical to
understanding the materials’ structure-property relations. 2D materials are
particularly well suited for this approach because they are atomically thin,
making it possible to image individual atoms across large areas [41, 5].
Several studies [42, 43, 44, 45] have employed quantitative analysis of ADF-
STEM image intensities, or so called Z-contrast, to identify atomic defects in
2D materials as the integrated intensity approximately scales with the atomic
number $Z^{1.5-1.8}$. However, the integrated intensity is very sensitive to
local contamination and microscope conditions, rendering this approach much
less reliable for data sets with varying imaging conditions.

Figure 4 shows our defect identification workflow. We utilize an FCN for
defect identification; this FCN is trained on the CycleGAN-processed,
realistic STEM images. The workflow has 6 main steps: (1) Acquire experimental
images using automated scripts, (2) simulate STEM images, (3) train the
CycleGAN with both experimental and simulated images, (4) use the CycleGAN to
post-process simulated images into realistic images, (5) train the FCN with
both CycleGAN-processed images from (4) and the corresponding defect ground
truth from (2), and (6) identify atomic defects in experimental images with
the FCN. Once set up, the workflow is relatively straightforward with steps
being automatically linked together. These steps, and quantitative
measurements of the resulting defect identification performance, are described
in detail below.

First, large-scale experimental STEM images are automatically acquired with
our custom-built script [46]. We acquire 2 experimental data sets of monolayer
WSe2 on 2 different days. These data are labeled as ‘Day A’ and ‘Day B’, which
include 107 and 211 images, respectively, with representative images shown in
Figure 5 (Methods). These data sets are not only massive (they contain 1.5 and
3 million atoms, respectively), but they also demonstrate the typical
variations of imaging conditions on different days. The irregular, bright
haziness on the atomic-resolution images corresponds to carbon-based surface
contamination, whose spatial distribution, thickness, and intensity vary
drastically between different regions in a single specimen. In addition,
microscope instabilities can cause degradation of image resolution as shown in
Figure 5d and h, where the resolution is degraded from 96 to 110 pm. In each
of these data sets, we manually label single Se vacancy defects in 3 images to
serve as the ground truth for FCN performance evaluation (Methods); this
manual labeling is for testing purposes only and would not be needed in a
typical workflow for data analysis. In total, 2 different test sets with 3
images each are constructed and labeled as ‘A’ and ‘B’. By combining the
experimental data sets (Day A and Day B) and test sets (A and B) separately,
we obtain the 3rd experimental data set (Day AB) and the 3rd test set (AB).
Note that ‘Day AB’ contains $107+211=318$ images, while the test set ‘AB’ has
$3+3=6$ images.

We then generate incoSTEM-simulated STEM images with defect labels, which
contain the same atomic defect species with the experimental STEM images. Note
that CycleGANs do not require that defect positions in the experimental and
simulated images be identical for training, because it is an unsupervised
learning technique for style-transfer between unpaired images [31]. Here, the
‘style’ being transferred is the ‘local texture’, or noise, rather than the
defects; the simulated images are generated with random defect distributions.
Using both the experimental and simulated STEM images, we train the CycleGAN
to generate realistic, high-quality simulated training data that matches the
experimental conditions, while keeping the initial defect positions unchanged.
Figure 6 shows the defect positions before and after the CycleGAN processing.
The defect positions (circled) are preserved by the CycleGAN because only the
‘local texure’, or noise, is being transferred to the simulated images. This
allows us to train an FCN with the combination of CycleGAN-processed STEM
images and corresponding defect labels that are produced in the initial
simulation. Once the CycleGAN is trained, it can generate unlimited amount of
experiment-like training data because the sole input is the labeled out-of-
the-box simulated data, which can be generated almost instantaneously.

### FCNs performance with different training sets

Lastly, we use the CycleGAN-generated images to train FCNs and evaluate their
performance for defect identification on experimental images (Methods). Table
2 shows the FCN performance with different FCN training sets. To demonstrate
the flexibility of our CycleGAN approach, we trained 5 FCNs with 5 different
training sets and evaluate their performance with 3 manually labeled
experimental test sets (A, B, and AB) as described above.

The 5 different FCN training sets are: (1) simulation without noise or any
processing ‘no noise’, (2) simulation with manually optimized noise ‘manual
noise’, as described by our previous work [5], (3) CycleGAN-A-processed images
‘CycleGAN-A’, (4) CycleGAN-B-processed images ‘CycleGAN-B’, and (5) CycleGAN-
AB-processed images ‘CycleGAN-AB’. In this naming convention, CycleGAN-A-
processed images means that the CycleGAN is trained with the experimental data
set from day A. The FCN training sets are kept the same size, and other
training details are described in the Methods. The FID scores of these
training sets are described in Supplemental Table 1. Conventional classifier
metrics including precision, recall, and F1 scores [47] are chosen to evaluate
the trained FCNs. Higher values indicate better FCN performance. While FCNs
are typically tested with simulated data, FCNs that perform well on simulated
data do not necessarily perform well on experimental data. For example, we
show in Supplemental Table 2 that all of our FCNs exhibit high precision,
recall, and F1 scores $(>96\%)$ when tested on simulated data. Here, we use a
more challenging test for the FCNs: evaluating their performance directly on
manually labeled experimental data.

When evaluated on experimental data, we observe excellent FCN performance for
the CycleGAN processed data. As shown in Table 2, the FCNs trained with
CycleGAN-processed images perform comparably well with the ones trained with
manually optimized simulated images, while requiring significantly less human
intervention. Note that precision and recall values above 90% are considered
exceptional, and often only achieved while training and testing with simulated
data.

We obtain the best results when we test the FCN on the same dataset of images
that the CycleGAN is trained on (gray cells in Table 2). This suggests the
CycleGAN is adapting to the microscopy conditions of that day. To further
validate this, we train the CycleGAN on one day (i.e. data set) and then test
the FCN using images from a different day. Here we should expect poor results
due to the differing microscopy conditions and this is often what we find (the
non-gray cells in Table 2) giving further evidence that the CycleGAN is
learning local-in-time microscopy conditions. In comparison, the performance
of the FCN trained with noise-free images (first row of Table 2) is
significantly worse than the other 4 FCNs . This suggests that adding proper
noise into the simulated images is necessary for the FCNs to correctly locate
defects in experimental STEM images. This is not surprising because one would
expect that FCNs perform the best when the input is well-described by the
training set.

While we use 107 (Day A), 211 (Day B), and 318 (Day AB) experimental images
for the CycleGAN training for the tests described above, we often achieve
comparable FCN performance using as few as only 6 experimental images, though
these small training sets occasionally lead to poor FCN performance
(Supplemental Table 3). Notably, this small number of experimental images
required for CycleGAN training opens the door to dynamically evolving ML
models. Although training a CycleGAN from scratch takes 6 hours in our case,
updating a pre-trained CycleGAN with a small set of new experimental images
may take significantly less time, which could enable nearly in-time adaptation
as the microscope conditions or sample properties evolve. While we chose FCN
and defect identification as a simple example to demonstrate the application
of CycleGAN, it is also possible to combine CycleGAN with more advanced
architectures such as ensemble learning [48] to achieve even higher
generalization capacity for real-time microscopy applications.

In conclusion, we developed a CycleGAN approach to generate high quality,
realistic STEM images and benchmarked its performance against manually
optimized simulated images. Using this approach, we show that an FCN trained
on CycleGAN-generated data achieves high precision for identifying atomic
defects in STEM images, enabling flexible, high throughput data processing for
large, automatically acquired experimental data sets. Our CycleGAN approach
could potentially be applied to most imaging techniques with a proper forward
model for generating realistic images for ML applications in microscopy, such
as autonomous collection and processing of atomic resolution materials data
[49, 50, 20].

## METHODS

### STEM sample preparation

The monolayer graphene (Grolltex) was wet-transferred from a Cu foil to an in
situ heating chip (Protochips). The monolayer WSe2 was first mechanically
exfoliated from bulk crystals (2D Semiconductors) using the Au-assisted
exfoliation method [51], and then was transferred onto perforated TEM grids
(Quantifoil) using a polymer free transfer [52] modified with subsequent KOH
and deionized water bath after the grid contact. The SrTiO3 sample was
fabricated using the standard focused ion beam lift-out procedures in an FEI
Helios 600i Dual Beam FIB-SEM system.

### STEM image acquisition

ADF-STEM image acquisition was conducted in a Thermo Fisher Scientific
Themis-Z aberration-corrected STEM operated at 80 kV. For atomic-resolution
ADF-STEM imaging, the point resolution was about 1 Å with 25 mrad convergence
semi angle, 35 pA probe current, 63 to 200 mrad collection semi angles for
WSe2 and SrTiO3, 14–20 pm pixel size and a total dwell time of 20 $\mu$s per
pixel using 10-frame averages. The STEM images for graphene were acquired with
100 pA, a 25 mrad inner collection angle, and were imaged after annealing at
1000 ∘C for a few minutes to reduce the contamination.

### STEM image simulation

The STEM simulation is done with the ‘incoSTEM’ package in Computem [32, 33]
with minimal parameter optimization. The ‘Simulation’, or ‘no noise’ images in
Figure 3b and Table 2 are simulated with the same experimental conditions
described above, except that no counting noise, source size, or aberrations
are included. The manually optimized parameters for the ‘manual noise’ data
set in both Figure 3c and Table 2 include surface contamination extracted from
experimental images, Poisson and Gaussian noise, image shear, source size,
aberrations up to 2nd order, and image brightness/contrast. All atom positions
are randomly perturbed (std = 0.01 Å) before simulation to remove a periodic
sampling artifact that can occur when atoms are not exactly located on a
pixel, which would show up as stripes in their FFTs.

### CycleGAN architecture

The generators follow a U-net architecture as described in the appendix of
Ref. [53]. The discriminators are patch-level [31], meaning that instead of
the output being a single number indicating whether an image is real or fake,
the discriminator outputs a $30\times 30$ array of outputs, where each element
in the array denotes the realness of a $70\times 70$ patch of the original
image. One difference between the architecture here and that in Ref. [53] is
that we use an instance normalization instead of a batch normalization in each
layer. The FFT discriminators are implemented using the log of the power
spectrum of the image, or $\log(|FT(I)|^{2})$, where $I$ is the image, and
$FT$ is the Fourier transform. Therefore, only the amplitude information are
being considered in the FFT discriminators. It is possible to extend the
approach to complex values for phase information [16], but we do not expect a
big difference in this case because the spatial frequencies of experimental
imperfections (noise and contamination) do not necessarily hold specific
phases in STEM images. Additionally, their spatial distributions are partially
regularized by image discriminators as well.

### CycleGAN training

The training set of CycleGAN is composed by both experimental images and
simulated STEM images. To prepare the CycleGAN training set, for each data
set, we normalize all the image intensities into [-1, 1] within their group by
their minimum and maximum value after a saturation step ($\pm\,3.5\sigma$). We
then cut them into $256\times 256$ patches. The batch-size is 42, however when
adding each patch into the batch, they are randomly augmented by a combination
of rotation (multiples of 90 degrees) and flipping (vertically or
horizontally). The out-of-the-box simulated images initially contain no noise,
which is detrimental to a CycleGAN because a sufficient amount of variability
is required to generate unique images. We add Gaussian noise (std = 0.1) to
the out-of-the-box simulated images to ensure enough variability across the
simulation data set. Note that this parameter does not require further manual
optimization. All networks were trained for 297 epochs, where in the first 148
epochs, the learning rate was 0.032, and from epochs 148–297, the learning
rate linearly decayed to zero. Supplemental Figure 5 shows the training loss
as a function of epochs. One key difference between traditional CycleGAN
training and here is that the FFT discriminator loss is also incorporated in
the generator loss alongside the real-space discriminator loss. For the
discriminators such as $D_{X,\text{img}}$, and $D_{Y,\text{img}}$, the loss
functions are

$\displaystyle\mathcal{L}(D_{X,\text{img}})$ $\displaystyle=\mathbb{E}_{y\sim
p_{\text{data}}(y)}||D_{X,\text{img}}(F(y))||_{2}+\mathbb{E}_{x\sim
p_{\text{data}}(x)}||1-D_{X,\text{img}}(x)||_{2},$ (1)
$\displaystyle\mathcal{L}(D_{Y,\text{img}})$ $\displaystyle=\mathbb{E}_{x\sim
p_{\text{data}}(x)}||D_{Y,\text{img}}(G(x))||_{2}+\mathbb{E}_{y\sim
p_{\text{data}}(y)}||1-D_{Y,\text{img}}(y)||_{2}$ (2)

and similarly for $D_{X,\text{FFT}}$, and $D_{Y,\text{FFT}}$. For the
generators $G:X\rightarrow Y$ and $F:Y\rightarrow X$, the loss functions are

$\displaystyle\mathcal{L}(G)$
$\displaystyle=\mathcal{L}_{\text{adv}}(G)+\lambda_{\text{cyc}}\mathcal{L}_{\text{cyc}}(F,G)+\lambda_{\text{id}}\mathcal{L}_{\text{id}}(G)$
(3) $\displaystyle\mathcal{L}(F)$
$\displaystyle=\mathcal{L}_{\text{adv}}(F)+\lambda_{\text{cyc}}\mathcal{L}_{\text{cyc}}(F,G)+\lambda_{\text{id}}\mathcal{L}_{\text{id}}(F)$
(4)

Here, the adversarial loss $\mathcal{L}_{\text{adv}}$ that is commonly used in
normal GANs [54] is the term that aims to fool its respective
discriminator(s):

$\mathcal{L}_{\text{adv}}(G)=\mathbb{E}_{x\sim
p_{\text{data}}(x)}||1-D_{Y,\text{img}}(G(x))||_{2}+\mathbb{E}_{x\sim
p_{\text{data}}(x)}||1-D_{Y,\text{FFT}}(G(x))||_{2}$ (5)

The cycle-consistency loss is given by

$\mathcal{L}_{\text{cyc}}(F,G)=\mathbb{E}_{x\sim
p_{\text{data}}(x)}||F(G(x))-x||_{1}+\mathbb{E}_{y\sim
p_{\text{data}}(y)}||G(F(y))-y||_{1}$ (6)

and the identity loss is given by

$\displaystyle\mathcal{L}_{\text{id}}(G)=\mathbb{E}_{y\sim
p_{\text{data}}(y)}||G(y)-y||_{2}.$ (7)

For the cycle-consistency loss, $\lambda_{\text{cyc}}$ was 10, and for the
identity loss, $\lambda_{\text{id}}$ was 5. Each CycleGAN in Table 2 has
different numbers of 1K-resolution STEM images as their training set.
CycleGAN-A has 107 images from Day A, CycleGAN-B has 211 images from Day B,
and CycleGAN-AB has 318 images from Day AB. All the models are constructed
with Tensorflow and ran on the Delta cluster at the National Center for
Supercomputing Applications (NCSA). There, training a CycleGAN model from
scratch takes around 6 hours to complete using a single NVIDIA A40 GPU.

### FCN architecture and training

We employ the same FCN architecture as in our previous report [5]. We used the
Adam optimizer with a categorical cross entropy loss function. For each
$1024\times 1024$ (1K-resolution) STEM image that is used for FCN training, we
cut the image into 16 $256\times 256$ patches with stride (256, 256). Each of
these patches is then rotated, flipped, and scale jittered, so that each
$256\times 256$ patch has 24 different augmentations. Hence, each
1K-resolution STEM image leads to $16\times 24=384$ smaller patches, which are
then divided into training set and simulated test set with a proportion of
10:1. In all FCN training, we use 107 1K-resolution STEM images. While
training the FCN, each epoch randomly chooses 1,000 $256\times 256$ patches
from the training set. We train for 500 epochs.

### FCN test sets

The FCNs are tested with simulated (Supplemental Table 2) and experimental
test sets (Table 2). The simulated test sets are generated along with the FCN
training sets but are picked out before the FCN training. The experimental
test sets are constructed by selecting 3 images from Day A (107 images) and
Day B (211 images) experimental data set, respectively. The images are
selected to roughly cover the entire experiment time span, which is usually
4–8 hours. The single Se vacancy positions are determined by the local
contrast, as single vacancy in monolayer WSe2 would produce only half of the
ADF-STEM pixel intensity comparing to a defect-free Se column. Each image is
slightly Fourier filtered (high pass and masking the Bragg reflections) to aid
the manually labeling process, while the final labeling is confirmed with the
raw image. Test images are $21\times 21$ nm2 large with $1024\times 1024$
pixels, while test set A and B contains 117 and 177 single Se vacancies per
image in average, respectively.

## DATA AVAILABILITY

The data sets are available on Zenodo: https://doi.org/10.5281/zenodo.7696721

## CODE AVAILABILITY

The codes are available on github: https://github.com/ClarkResearchGroup/stem-
learning

## ACKNOWLEDGEMENT

This material is based upon work supported by the U.S. Department of Energy,
Office of Science, Office of Basic Energy Sciences, Division of Materials
Sciences and Engineering under award number DE-SC0020190, which supported the
electron microscopy and related data analysis. We acknowledge Yue Zhang and
Prof. Arend van der Zande for the WSe2 sample fabrication. This work was
carried out in part in the Materials Research Laboratory Central Facilities at
the University of Illinois Urbana–Champaign. This research is also part of the
Delta research computing project, which is supported by the National Science
Foundation (award OCI 2005572), and the State of Illinois. Delta is a joint
effort of the University of Illinois Urbana–Champaign and its National Center
for Supercomputing Applications.

## AUTHOR CONTRIBUTIONS

Under supervision by B.K.C., A.K. built and evaluated the CycleGAN and FCN
models, and conducted the data set similarity analysis. Under supervision by
P.Y.H., C.-H.L. acquired and analyzed the experimental and simulated STEM
images, and conducted the data set similarity analysis. A.K. and C.-H.L.
contributed equally to this work and are considered as co-first authors. All
authors read and contributed to the manuscript.

## COMPETING INTERESTS

The authors declare no competing financial interest.

## ADDITIONAL INFORMATION

Supplementary information The online version contains supplementary material
including extra figures of CycleGAN denoising, CycleGAN without FFT
discriminators, cross-evaluated images, power spectra comparison, and training
losses per epoch.
Correspondence and requests for materials should be addressed to Pinshane Y.
Huang or Bryan K. Clark.

## References

  * [1] Maxim Ziatdinov et al.  “Deep Learning of Atomically Resolved Scanning Transmission Electron Microscopy Images: Chemical Identification and Tracking Local Transformations”  In _ACS Nano_ 11.12, 2017, pp. 12742–12752  DOI: 10.1021/acsnano.7b07504
  * [2] Jacob Madsen et al.  “A deep learning approach to identify local structures in atomic-resolution transmission electron microscopy images”  In _Advanced Theory and Simulation_ 1.8, 2018, pp. 1800037  DOI: 10.1002/adts.201800037
  * [3] Ruoqian Lin et al.  “TEMImageNet training library and AtomSegNet deep-learning models for high-precision atom segmentation, localization, denoising, and deblurring of atomic-resolution images”  In _Scientific Reports_ 11.1 Nature Publishing Group UK, 2021, pp. 5386  DOI: 10.1038/s41598-021-84499-w
  * [4] Artem Maksov et al.  “Deep learning analysis of defect and phase evolution during electron beam-induced transformations in WS2”  In _npj Computational Materials_ 5.1 Springer US, 2019, pp. 12  DOI: 10.1038/s41524-019-0152-9
  * [5] Chia-Hao Lee et al.  “Deep Learning Enabled Strain Mapping of Single-Atom Defects in Two-Dimensional Transition Metal Dichalcogenides with Sub-Picometer Precision”  In _Nano Letters_ 20.5, 2020, pp. 3369–3377  DOI: 10.1021/acs.nanolett.0c00269
  * [6] Yueming Guo et al.  “Defect detection in atomic-resolution images via unsupervised learning with translational invariance”  In _npj Computational Materials_ 7.1 Springer US, 2021, pp. 180  DOI: 10.1038/s41524-021-00642-1
  * [7] Tran Minh Quan et al.  “Removing Imaging Artifacts in Electron Microscopy using an Asymmetrically Cyclic Adversarial Network without Paired Training Data”  In _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_ Ieee, 2019, pp. 3804–3813  DOI: 10.1109/iccvw.2019.00473
  * [8] Jeffrey M. Ede and Richard Beanland  “Improving electron micrograph signal-to-noise with an atrous convolutional encoder-decoder”  In _Ultramicroscopy_ 202.November 2018 Elsevier B.V., 2019, pp. 18–25  DOI: 10.1016/j.ultramic.2019.03.017
  * [9] Feng Wang, Trond R. Henninen, Debora Keller and Rolf Erni  “Noise2Atom: unsupervised denoising for scanning transmission electron microscopy images”  In _Applied Microscopy_ 50.1 Applied Microscopy, 2020, pp. 23  DOI: 10.1186/s42649-020-00041-8
  * [10] W. Xu and J.. LeBeau  “A deep convolutional neural network to analyze position averaged convergent beam electron diffraction patterns”  In _Ultramicroscopy_ 188, 2018, pp. 59–69  DOI: 10.1016/j.ultramic.2018.03.004
  * [11] Chenyu Zhang, Jie Feng, Luis Rangel DaCosta and Paul M. Voyles  “Atomic resolution convergent beam electron diffraction analysis using convolutional neural networks”  In _Ultramicroscopy_ 210.October 2019 Elsevier B.V., 2020, pp. 112921  DOI: 10.1016/j.ultramic.2019.112921
  * [12] Renliang Yuan, Jiong Zhang, Lingfeng He and Jian Min Zuo  “Training artificial neural networks for precision orientation and strain mapping using 4D electron diffraction datasets”  In _Ultramicroscopy_ 231.December 2020 Elsevier B.V., 2021, pp. 113256  DOI: 10.1016/j.ultramic.2021.113256
  * [13] J.. Aguiar et al.  “Decoding crystallography from high-resolution electron imaging and diffraction datasets with deep learning”  In _Science Advances_ 5.10, 2019, pp. 1–10  DOI: 10.1126/sciadv.aaw1949
  * [14] Kevin Kaufmann et al.  “Crystal symmetry determination in electron diffraction using machine learning”  In _Science_ 367.6477, 2020, pp. 564–568  DOI: 10.1126/science.aay3062
  * [15] Noah Schnitzer, Suk Hyun Sung and Robert Hovden  “Optimal STEM Convergence Angle Selection Using a Convolutional Neural Network and the Strehl Ratio”  In _Microscopy and Microanalysis_ 26.5, 2020, pp. 921–928  DOI: 10.1017/s1431927620001841
  * [16] Joydeep Munshi et al.  “Disentangling multiple scattering with deep learning: application to strain mapping from electron diffraction patterns”  In _npj Computational Materials_ 8.1 Springer US, 2022, pp. 254  DOI: 10.1038/s41524-022-00939-9
  * [17] Chuqiao Shi et al.  “Uncovering material deformations via machine learning combined with four-dimensional scanning transmission electron microscopy”  In _npj Computational Materials_ 8.1 Springer US, 2022, pp. 114  DOI: 10.1038/s41524-022-00793-9
  * [18] Michael Xu, Abinash Kumar and James M. LeBeau  “Towards Augmented Microscopy with Reinforcement Learning-Enhanced Workflows”  In _Microscopy and Microanalysis_ 28.6, 2022, pp. 1952–1960  DOI: 10.1017/s1431927622012193
  * [19] Jeffrey M. Ede  “Deep learning in electron microscopy”  In _Machine Learning: Science and Technology_ 2.1, 2021, pp. 011004  DOI: 10.1088/2632-2153/abd614
  * [20] Sergei V. Kalinin et al.  “Machine learning in scanning transmission electron microscopy”  In _Nature Reviews Methods Primers_ 2.1 Springer US, 2022, pp. 11  DOI: 10.1038/s43586-022-00095-w
  * [21] Marc Botifoll, Ivan Pinto-Huguet and Jordi Arbiol  “Machine learning in electron microscopy for advanced nanocharacterization: current developments, available tools and future outlook”  In _Nanoscale Horizons_ Royal Society of Chemistry, 2022, pp. 1427–1477  DOI: 10.1039/d2nh00377e
  * [22] Takehito Seki, Yuichi Ikuhara and Naoya Shibata  “Theoretical framework of statistical noise in scanning transmission electron microscopy”  In _Ultramicroscopy_ 193.February Elsevier B.V., 2018, pp. 118–125  DOI: 10.1016/j.ultramic.2018.06.014
  * [23] L. Jones  “Quantitative ADF STEM: Acquisition, analysis and interpretation”  In _IOP Conference Series: Materials Science and Engineering_ 109.1, 2016  DOI: 10.1088/1757-899x/109/1/012008
  * [24] Nadi Braidy, Yann Le Bouar, Sorin Lazar and Christian Ricolleau  “Correcting scanning instabilities from images of periodic structures”  In _Ultramicroscopy_ 118 Elsevier, 2012, pp. 67–76  DOI: 10.1016/j.ultramic.2012.04.001
  * [25] Colin Ophus, Jim Ciston and Chris T. Nelson  “Correcting nonlinear drift distortion of scanning probe and scanning transmission electron microscopies from image pairs with orthogonal scan directions”  In _Ultramicroscopy_ 162 Elsevier, 2016, pp. 1–9  DOI: 10.1016/j.ultramic.2015.12.002
  * [26] Benjamin H. Savitzky et al.  “Image registration of low signal-to-noise cryo-STEM data”  In _Ultramicroscopy_ 191 Elsevier B.V., 2018, pp. 56–65  DOI: 10.1016/j.ultramic.2018.04.008
  * [27] S.. Schramm, S.. Molen and R.. Tromp  “Intrinsic Instability of Aberration-Corrected Electron Microscopes”  In _Physical Review Letters_ 109.16, 2012, pp. 163901  DOI: 10.1103/PhysRevLett.109.163901
  * [28] R.F. Egerton  “Control of radiation damage in the TEM”  In _Ultramicroscopy_ 127 Elsevier, 2013, pp. 100–108  DOI: 10.1016/j.ultramic.2012.07.006
  * [29] Simon Hettler et al.  “Carbon contamination in scanning transmission electron microscopy and its impact on phase-plate applications”  In _Micron_ 96 Elsevier Ltd, 2017, pp. 38–47  DOI: 10.1016/j.micron.2017.02.002
  * [30] Yin Min Goh et al.  “Contamination of TEM Holders Quantified and Mitigated With the Open-Hardware, High-Vacuum Bakeout System”  In _Microscopy and Microanalysis_ 26.5, 2020, pp. 906–912  DOI: 10.1017/s1431927620001762
  * [31] Jun-Yan Zhu, Taesung Park, Phillip Isola and Alexei A. Efros  “Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks”  In _2017 IEEE International Conference on Computer Vision (ICCV)_ 2017-Octob Ieee, 2017, pp. 2242–2251  DOI: 10.1109/iccv.2017.244
  * [32] Earl J. Kirkland  “Computem”, 2013  URL: http://sourceforge.net/projects/computem
  * [33] Earl J. Kirkland  “Advanced Computing in Electron Microscopy”  Springer Cham, 2020  DOI: 10.1007/978-3-030-33260-0
  * [34] Aaron Gokaslan et al.  “Improving shape deformation in unsupervised image-to-image translation”  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 649–665
  * [35] Martin Heusel et al.  “GANs trained by a two time-scale update rule converge to a local Nash equilibrium”  In _Advances in Neural Information Processing Systems_ 30, 2017  arXiv:1706.08500
  * [36] S. Kullback and R.. Leibler  “On Information and Sufficiency”  In _The Annals of Mathematical Statistics_ 22.1, 1951, pp. 79–86  DOI: 10.1214/aoms/1177729694
  * [37] Mario Lucic et al.  “Are GANs Created Equal? A Large-Scale Study”  In _Advances in Neural Information Processing Systems_ 31, 2018  arXiv: http://arxiv.org/abs/1711.10337
  * [38] Ali Borji  “Pros and cons of GAN evaluation measures”  In _Computer Vision and Image Understanding_ 179, 2019, pp. 41–65  DOI: 10.1016/j.cviu.2018.10.009
  * [39] Christopher Ren, Amanda K. Ziemann, James Theiler and Alice Durieux  “Deep snow: synthesizing remote sensing imagery with generative adversarial nets”  In _Algorithms, Technologies, and Applications for Multispectral and Hyperspectral Imagery XXVI_ Spie, 2020, pp. 33  DOI: 10.1117/12.2560716
  * [40] D.C Dowson and B.V Landau  “The Fréchet distance between multivariate normal distributions”  In _Journal of Multivariate Analysis_ 12.3, 1982, pp. 450–455  DOI: 10.1016/0047-259x(82)90077-x
  * [41] Pinshane Y. Huang et al.  “Imaging Atomic Rearrangements in Two-Dimensional Silica Glass: Watching Silica’s Dance”  In _Science_ 342.6155, 2013, pp. 224–227  DOI: 10.1126/science.1242248
  * [42] Ondrej L Krivanek et al.  “Atom-by-atom structural and chemical analysis by annular dark-field electron microscopy”  In _Nature_ 464.7288 Macmillan Publishers Limited. All rights reserved, 2010, pp. 571–574  DOI: 10.1038/nature08879
  * [43] Amin Azizi et al.  “Defect Coupling and Sub-Angstrom Structural Distortions in W 1– x Mo x S 2 Monolayers”  In _Nano Letters_ 17.5, 2017, pp. 2802–2808  DOI: 10.1021/acs.nanolett.6b05045
  * [44] Yu Jie Zheng et al.  “Point Defects and Localized Excitons in 2D WSe2”  In _ACS Nano_ 13.5, 2019, pp. 6050–6059  DOI: 10.1021/acsnano.9b02316
  * [45] Sujuan Ding, Fang Lin and Chuanhong Jin  “Quantify point defects in monolayer tungsten diselenide”  In _Nanotechnology_ 32.25 IOP Publishing, 2021  DOI: 10.1088/1361-6528/abeeb2
  * [46] Chia-Hao Lee et al.  “Automated Acquisition and Deep Learning of 2D Materials on the Million-Atom Scale”  In _Microscopy and Microanalysis_ 28.S1, 2022, pp. 3062–3063  DOI: 10.1017/s1431927622011424
  * [47] Kevin P. Murphy  “Machine Learning: A Probabilistic Perspective”  In _MIT Press_ Cambridge: MIT Press, 2012  DOI: 10.1007/978-94-011-3532-0“˙2
  * [48] Kevin M. Roccapriore et al.  “Probing Electron Beam Induced Transformations on a Single-Defect Level via Automated Scanning Transmission Electron Microscopy”  In _ACS Nano_ 16.10, 2022, pp. 17116–17127  DOI: 10.1021/acsnano.2c07451
  * [49] Steven R. Spurgeon et al.  “Towards data-driven next-generation transmission electron microscopy”  In _Nature Materials_ 20.3 Springer US, 2021, pp. 274–279  DOI: 10.1038/s41563-020-00833-z
  * [50] Matthew Olszta et al.  “An Automated Scanning Transmission Electron Microscope Guided by Sparse Data Analytics”  In _Microscopy and Microanalysis_ 28.5, 2022, pp. 1–11  DOI: 10.1017/s1431927622012065
  * [51] Matěj Velický et al.  “Mechanism of Gold-Assisted Exfoliation of Centimeter-Sized Transition-Metal Dichalcogenide Monolayers”  In _ACS Nano_ 12.10, 2018, pp. 10463–10472  DOI: 10.1021/acsnano.8b06101
  * [52] D. Pacilé, J.. Meyer, Ç.. Girit and A. Zettl  “The two-dimensional phase of boron nitride: Few-atomic-layer sheets and suspended membranes”  In _Applied Physics Letters_ 92.13, 2008, pp. 133107  DOI: 10.1063/1.2903702
  * [53] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou and Alexei A. Efros  “Image-to-Image Translation with Conditional Adversarial Networks”  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ 2017-Janua Ieee, 2017, pp. 5967–5976  DOI: 10.1109/cvpr.2017.632
  * [54] Ian J. Goodfellow et al.  “Generative Adversarial Networks”  In _Advances in Neural Information Processing Systems_ 27, 2014  arXiv: http://arxiv.org/abs/1406.2661%20https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf

## FIGURE LEGENDS

Figure 1: Representative images and their power spectra of simulated, CycleGAN-processed, and experimental images of graphene, WSe2, and SrTiO3. Each row displays the real space image along with its power spectrum. Power spectrum of the image is calculated by the fast Fourier transform (FFT) and displayed in log scale for clarity. (a–c) Semi-quantitative, simulated ADF-STEM images are generated by the Computem program [32, 33]. These simulated images are passed into pre-trained CycleGANs and resulted in highly realistic images in (d–f). Individual CycleGANs are trained for each materials system using both simulated and experimental data as the training set. The CycleGANs transfer experimental imperfections, including noise, jittering, distortion, and surface contamination to simulated images, making the simulated images appear realistic. (g–i) Experimental ADF-STEM images and their power spectra for comparison. Note that (h) has been selected to contain the same type of bright defect as in (e) so that images are more comparable. Scale bars are equal to 1 nm. Figure 2: Schematic of the major components in a CycleGAN. The CycleGAN translates images between two domains: (1) the experiment domain $X$ and (2) the simulation domain $Y$, and has two main components: generators and discriminators. The generator $G$ converts input experimental images $x$ to simulation-like images $G(x)$, and the generator $F$ converts input simulated images $y$ to experiment-like images $F(y)$. The generated images ($F(y)$ and $G(x)$) are then fed into four discriminators ($D_{x,\text{img}}$, $D_{x,\text{FFT}}$, $D_{y,\text{img}}$, $D_{y,\text{FFT}}$) along with the raw images ($x$ and $y$) to evaluate the quality of generated images. Both the input images and their FFTs are examined by the discriminators to calculate the adversarial losses ($L_{\text{adv}}$), which are used to optimize both generators $F$ and $G$ respectively. By passing the raw images ($x$ and $y$) with combinations of both generators, identity images ($F(x)$ and $G(y)$) and cycled images ($F(G(x))$ and $G(F(y))$) are also generated. The corresponding identity loss $L_{\text{id}}$ and cycle consistency loss $L_{\text{cyc}}$ are added to ensure the identity and cycle consistency mapping of the generators. Figure 3: Quantitative measurements of data set quality using the Fréchet inception distance (FID) and the Kullback–Leibler (KL) divergence of WSe2. Images are generated from (a) experiments, (b) simulation without noise, (c) simulation with manually optimized noise, and (d) CycleGAN. The FID score measures the dissimilarity between image data sets, so a smaller FID score implies higher similarity between image data sets. The FID scores with respect to the experimental data set are labeled at the bottom right corner of each representative image, with the lowest non-zero value marked in yellow. (e–h) Histograms of normalized pixel intensity are calculated for each image data set. Each histogram is normalized so that the probability distribution sums to unity. The KL divergence $D_{\text{KL}}(P||Q)$ of each data set with the experimental histogram are labeled as $D_{\text{KL}}$ at the top right corner of each histogram, with the lowest non-zero value marked in red. Note that both the FID score and KL divergence of intensity histograms are calculated for the entire data set with respect to the experimental data set, where each data set contains roughly 1,700 image patches with $256\times 256$ pixels. The CycleGAN generated image set exhibits the best FID score and lowest KL divergence, indicating it is the best match for experimental data.  | graphene |  WSe2 |  SrTiO3
---|---|---|---
Simulation | 32.69 | 31.87 | 19.60
CycleGAN | 1.66 | 0.35 | 0.47
Table 1: FID scores of the simulated data set without noise and the CycleGAN-processed data set with respect to the experimental data set for graphene, WSe2, and SrTiO3. Note that individual CycleGANs are trained for each materials system. Figure 4: The workflow of detecting single-atom defects using both a CycleGAN and a fully convolutional network (FCN). First, we acquire large data sets of experimental STEM images with a custom-build automated acquisition script. We then simulate STEM images using Computem with pre-determined defect types and positions, which constitute the ground truth of defect labels. The experimental images and the simulated images are then used to train a CycleGAN, whose main purpose is to train a generator that can produce realistic images from simulated input images without altering the defect ground truth. CycleGAN learns the noise distribution from experimental images and adds it to the input simulated images, making them higher quality training data for further supervised learning applications. These CycleGAN-processed images, along with the defect ground truth, are used to train a defect identification FCN. Lastly, the FCN takes in the same experimental images with atomic defects, and returns the locations of those defects. The black arrows represent the input and output data flow, and the gray arrows indicate the input data for neural network training. Figure 5: Example of variations within experimental STEM data sets taken on two different days, Day A and Day B. Experimental STEM images acquired on Day A (a–c) and B (e–g) showing variations within and between days due to sample contamination and microscope instabilities. (d and h) Power spectra obtained with FFT of (a and e) demonstrate the variation of image resolution, from 96 pm to 110 pm, determined by the highest transferred spatial frequency marked with white circles. The power spectra are displayed in log scale for visibility. Scale bars are equal to 1 nm. Figure 6: The CycleGAN-processed images preserve the defect types and positions from the input simulated images. (a) Atomic structure of WSe2 with atomic defects. The model is rotated by 10∘ for better visualization of the overlapping Se atoms and atomic defects. (b) Simulated images are generated by Computem with pre-determined defect types and positions, therefore the ground truth of defect labels are known. (c) By using the simulated images (b) as input, the CycleGAN-processed images show realistic, experimental-like noise distribution while keeping the defect label unaltered. Note that only 5 defects in the field of view are circled for better clarity. Scale bars are equal to 1 nm. | FCN test sets
---|---
A | B | AB
| FCN
---
training sets
P | R | F1 | P | R | F1 | P | R | F1
no noise | 66 | 84 | 74 | 15 | 83 | 25 | 22 | 83 | 35
manual noise | 91 | 89 | 90 | 66 | 95 | 78 | 74 | 92 | 82
CycleGAN-A |  [HTML]DADADA87 |  [HTML]DADADA91 |  [HTML]DADADA89 | 12 | 57 | 20 | 22 | 71 | 34
CycleGAN-B | 97 | 40 | 56 |  [HTML]DADADA85 |  [HTML]DADADA79 |  [HTML]DADADA82 | 87 | 63 | 73
CycleGAN-AB | 98 | 49 | 65 | 85 | 73 | 78 |  [HTML]DADADA89 |  [HTML]DADADA63 |  [HTML]DADADA74
Table 2: FCN performance metrics with different training sets. 5 different FCN
training sets are prepared, including simulation without noise, simulation
with manually optimized noise, and CycleGANs trained on 3 different sets of
experimental images (Day A, Day B, and Day AB). The FCN performance are tested
with manually labeled experimental test sets (A, B, and AB) and evaluated by
precision (P), recall (R), and F1 scores with units in percentage (%). In the
last three rows, the cells with matching CycleGAN training set and FCN test
set are colored in gray; using a matching set between CycleGAN and FCN would
be the standard approach for production runs. Unsurprisingly, running the
CycleGAN and FCN on different sets of data (white boxes), which one would not
do in practice, leads to poorer accuracy. The highest value of each column is
bolded.


2022

[1]Cong-Zhan Liu

[1]Key Laboratory of Particle Astrophysics, Institute of High Energy Physics,
Chinese Academy of Science, 19B Yuquan Road, Shijingshan District, Beijing,
100049, China

# In-orbit performance of HE onboard Insight-HXMT in the first 5 years

Xu-Fang Li<EMAIL_ADDRESS>Yi-Fei Zhang  Xiao-Bo Li  Zheng-Wei Li  Xue-Feng
Lu  Zhi Chang  Ming-Yu Ge  Juan Zhang  Yu-Peng Xu  Fang-Jun Lu  Li-Ming Song
Shuang-Nan Zhang  *

###### Abstract

Purpose: The High-Energy X-ray telescope (HE), one of the three main payloads
of the Insight-HXMT mission, is composed of eighteen NaI(Tl)/CsI(Na) phoswich
detectors, where NaI(Tl) serves as the primary detector covering 20–250 keV,
and CsI(Na) is used as an active shield detector to suppress the background of
NaI(Tl) and also serves as an all-sky gamma-ray burst monitor covering 0.2–3
MeV. In this paper, we review the in-orbit performance of HE in the first 5
years since Insight-HXMT was launched on June 15, 2017. Methods: The major
performances we concern include the gain and energy resolution of NaI(Tl) and
CsI(Na) detectors, the performance of pulse-shape-discriminator (PSD) and
system dead-time. In this work, we investigate these performances mainly using
the data of blank-sky observations and the data when the telescope in earth
occultation. Results: The overall performance of HE/NaI(Tl) is very stable in
the first 5 years, whereas the gain of HE/CsI(Na) shows a continuously
increasing trend and should be calibrated regularly. Conclusion: In general,
HE is still in good health and well-calibrated status after five-year’s
operation. The in-orbit performance of HE has no significant deviation from
expectation. HE is expected to be in operation healthily for another several
years of extended mission life.

###### keywords:

Insight-HXMT, hard X-ray telescope, scintillation detector, in-orbit

##  1 Introduction

The Hard X-ray Modulation Telescope (dubbed Insight-HXMT after launch),
China’s first X-ray astronomy satellite, was launched on June 15,
2017Zhang2020 ; Liu2020 . There are three main science instruments onboard
Insight-HXMT, covering the energy range from 0.7 keV to 3000 keV. The High
Energy X-ray telescope (HE), the biggest one among the three instruments,
consists of eighteen NaI(Tl)/CsI(Na) phoswich scintillation detectors
(HEDs)Liu2020 . Each detector has a collimator with an automatic gain control
detector (AGC) inside. The whole instrument is shielded by six anti-
coincidence detectors (HVT) on the top and twelve lateral anti-coincidence
detectors. The fluorescence photons produced in the two types of crystals of
HED are detected by a shared photomultiplier tube (PMT). Pulse height and
width of scintillation signals are analyzed by Pulse Height Analyzer(PHA) and
Pulse Shape Analyzer(PSA) respectively. Thanks to the pulse shape
discrimination (PSD) technology, the NaI events and CsI events can be
distinguished easily. For HED phoswich, the NaI(Tl) acts as the main science
detector covering an energy band from 20–250 keV, and the CsI(Na) detector
acts as an active shielding detector to reject Compton events which deposit
energies in both CsI(Na) and NaI(Tl) simultaneously, and to reject background
events incident from the backside within 2 $\pi$ solid angle. However, the
CsI(Na) detector can also be used as an all-sky Gamma-Ray Burst (GRB) monitor
covering the energy range from 40–800 keV in the normal mode and 0.2–3 MeV in
the low gain modeZhang2020 ; Liu2020 ; Song2022 .

Performances of HE have been derived from a combination of on-ground
calibration results and dedicated calibration observations during its in-orbit
operation. Detailed descriptions of its design, on-ground calibrations, and
in-orbit calibrations were previously reportedLiu2020 ; Li2019 ; Li2020 ;
Luo2020 . This work will focus on the evolution of the performance of the HE
phoswich detectors during the first 5 years in orbit.

##  2 Operation modes

To meet the requirements of the scientific observations, HE was designed to
have three operation modes:

(1) NOrmal Mode (NOM): In this mode, with the help of auto-gain control
system, the gain of a NaI(Tl) detector is continuously monitored and the
negative high-voltage (HV) of the detector’s PMT is automatically adjusted, so
that in NaI(Tl) detector the full-energy peak of the 59.5 keV photons (emitted
from the radioactive source 241Am in AGC ) can be fixed at PHA channel 50. In
this operation mode, HE can simultaneously observe sources within the Field of
View (FoV) restricted by collimators in 20–250 keV with NaI(Tl) and monitor
almost the entire sky in 40–800 keV with CsI(Na).

(2) Low Gain Mode (LGM): In this mode, the HE auto-gain control system is
turned off and the HVs of the HED PMTs are reduced to make the energy range of
CsI(Na) cover about 0.2–3 MeV. This mode is usually scheduled when the
Insight-HXMT runs into the earth occultation where the observed sources or
scanned regions are invisible and the normal observations have to be
interrupted.

(3) South Atlantic Anomaly (SAA) mode: In this mode, the HVs of HEDs are
switched off to protect the detectors from the damage of high-flux charged
particles. The instrument will not record any scientific event.

##  3 Performance evolution in orbit

###  3.1 High voltage monitoring

The HED was designed to work with negative HV (for convenience, we only use
the absolute value of HV hereafter). Figure 1 and Figure 2 show the daily
average absolute values of the PMT HVs in NOM and LGM over time, respectively.
The HVs changed obviously during the first three months because the telescope
was in commissioning phase. About three months later, the HED voltages tended
to change slowly. In the middle and right panels of Figure 1, we can see that
most of HED HVs declined in the last five years with different slopes. In the
left panel of Figure 1, HVs of DetID 0,3,5 and 15 are certainly stable, but
that of DetID 2 is peculiar with a slow increase. In LGM, the HVs can not be
adjusted automatically because the AGC system is turned off. According to the
calibration results, the HVs in LGM were updated in July 2017, Jun. 2020, and
Oct. 2021, respectively, resulting in the three discontinuities in Figure 2.
Most of HED HVs in LGM decreased but DetID 2. From the left panel of Figure 2,
we can clearly see that HV of DetID 2 has been progressively increasing, which
is coincident with the behavior in NOM.

In general, the gain of each HED detector system is determined by the
fluorescence yielding and collection efficiency, the HV applied to the PMT and
the gain of the electronics chain. In HE design, the gain of each HED detector
system is fixed by the corresponding AGC system. And the gain of electronics
chain is relatively stable because there is an active temperature control for
each HED to keep the detector temperature not changed too much. So we suppose
that the HV adjustment is a compensation for the change of fluorescence
yielding and collection efficiency. Before launch we found the HEDs slightly
deliquesced to some extent. The vacuum environment in orbit is beneficial to
the HED scintillators recovering from deliquescing. Then the fluorescence
yielding and collection efficiency improved slowly, leading to the gain
increasing and the HV decreasing at the same time. But the behavior of DetID 2
can not be understood very well, which indicates the nature is still
complicated.

Figure 1: Absolute values of HED HVs as a function of time in NOM in the first
five years. The HVs are automatically adjusted by AGC system.

Figure 2: Absolute values of HED HVs as a function of time in LGM in the first
five years.The voltages were adjusted by tele-commands in July 2017, Jun. 2020
and Oct. 2021, respectively.

###  3.2 Performance evolution of gain and energy resolution

For regular energy response calibration, we hope the background and source is
stable and observation frequency is suitable. So the data of blank-sky
observations are used to calibrate the energy responses of NaI(Tl) and CsI(Na)
detectors in NOM, and the data of earth occultation are used to calibrate
CsI(Na) in LGM. We select events with pulse width between PSA channels 54 to
70 as NaI events, and events with pulse width between PSA channel 80 to 120 as
CsI events in NOM. In LGM, we only select CsI events with pulse width between
PSA channel 78 to 120. Those events with veto tags by HVTs are rejected.

####  3.2.1 NaI(Tl) performance

Plenty of experimental studies on NaI(Tl) detector over many years have shown
that the NaI(Tl) crystal’s light output is not strictly proportional to the
deposited energy of X-ray and electronNLRofNaI1963 ; Wayne1998 ; Gardner1999 ;
Payne2011 . The non-proportional response (NPR) model of each HE/NaI(Tl)
detector to electrons was built by simulations based on on-ground calibration
campaignLi2019 ; Li2020 . According to the NPR model, the energy response of
the NaI(Tl) to X-ray was determined with the help of Monte Carlo simulations.
In orbit, four iodine activation lines (31, 56, 67, and 191 keV)zhangjuan2020
are used to modify the on-ground calibrated Channel-Energy relation and energy
resolution model. Then the NPR model and response matrix files of each
HE/NaI(Tl) detector can be produced. The in-orbit calibration progress and
results of the HE/NaI(Tl) were described in detail inLi2020  .

Regular blank-sky observations and the 241Am radioactive sources in AGCs are
used to monitor the long-term gain evolution of HE/NaI(Tl) detectors. Figure 3
shows the energy spectra of eighteen HE/NaI(Tl) detectors of two blank sky
observations. In each panel, the blue curve is the energy spectrum of 4.5
months after launch and the red curve is that of 5 years after launch. The two
spectra are almost identical except the intensity of the bump around
$\thicksim$67 keV. The bump intensity increased with time, which is attributed
to the in-orbit cumulative particle-induced activation of iodine isotopes in
NaI(Tl) and CsI(Na) detectors with long decay periods. In the first year, the
relative abundance of $\thicksim$67 keV line increased quickly and eventually
reached an equilibrium between activation and decayzhangjuan2020 .

In NOM, the gain of NaI(Tl) is automatically controlled by fixing the peak
centroid of 59.5 keV calibration events to channel 50. Figure 4 shows the
trend of the 59.5 keV line peak centriod values and energy resolutions in full
width at half maximum (FWHM) after launch. About 90 days after launch, the
performance of HE/NaI(Tl) became stable. The gain variations were even less
than 0.1$\%$ and the energy resolutions were getting better in the last five
years. However, the parameters derived from the 59.5 keV events emitted by AGC
sources can only represent a small local region of the corresponding NaI(Tl)
detector. Here, we also use a background line around 191 keV to monitor the
gain variation of the overall NaI(Tl) detector with time. Figure 5 and Figure
6 show the evolution of the peak position and resolution in FWHM at 191 keV,
respectively. The gain variations of all HE/NaI(Tl) detectors are less than
1$\%$ since launch. Also, for most of the HE/NaI(Tl) detectors the energy
resolutions are slightly getting better. The results derived from the 191 keV
line are highly consistent with that from 59.5 keV.

The spectra of multi-pointing observations of the Crab nebula were fitted by
using the current energy response matrices of HE/NaI(Tl), which were
calibrated in Oct. 2017, and if a residual larger than 2$\%$ in 28-120 keV was
observed, the energy response matrices would be re-calibrated.

Figure 3: Background PHA spectra of HE/NaI(Tl) detectors obtained from two
blank-sky observations. In each panel, one curve is for 4.5 months after
launch (blue) and another one is for 5 years after launch (red).

Figure 4: The 59.5 keV peak positions (left) and resolutions in FWHM (right)
as a function of time for 18 HE/NaI(Tl) detectors.(a) Evolutions of the 59.5
keV peak position for eighteen HE/NaI(Tl) detectors over time. (b) Evolutions
of the energy resolution of 59.5 keV peak for eighteen HE/NaI detectors over
time.

Figure 5: Peak centroid of 191 keV line for 18 HE/NaI(Tl) detectors as a
function of time. Compared with the values just after launch, the percentage
change of the peak positions from Sep.29, 2017 to Jun. 26, 2022 are within
1$\%$.

Figure 6: The 191 keV line resolution (in FWHM of PHA channel) for 18
HE/NaI(Tl) detectors as a function of time. Compared with the values just
after launch, the percentage changes of the FWHM from Sep. 29, 2017 to Jun.
26, 2022 are shown in the subplots directly, and for most HE/NaI(Tl) detectors
the energy resolutions are slightly getting better.

####  3.2.2 CsI(Na) performance

We can understand the evolution of the HE/CsI(Na) detector performances by
monitoring their background PHA spectra during the last five years (see Figure
7 in NOM and Figure 8 in LGM). We can see that almost all the characteristic
bumps of the spectra shift to the right over time, which indicates that the
gain of the HE/CsI(Na) detector has increased continuously. For clarity, we
plot the $\thicksim$692 keV peak positions in NOM as a function of time in
Figure 9 and the $\thicksim$2767 keV peak positions in LGM as a function of
time in Figure 10. The $\thicksim$2767 keV peak would shift out of the CsI(Na)
energy range if the detector gain increased to a certain extent. So we had to
adjust the PMT HVs in LGM in Jun. 2020 and Oct. 2021 respectively, to keep the
energy range of CsI(Na) basically unchanged. That is why there are two obvious
discontinuities in Figure 10. From the left panel of Figure 10, the gain of
the detector DetID 2 decreased with time before Jun. 2020 but turned to
increase after that. The abnormal evolution behavior of DetID 2 is confusing
and not understood well yet. Nevertheless, the data of DetID 2 can still be
used normally in scientific research.

Figure 7: Typical HE/CsI(Na) background spectra in NOM from blank-sky
observations in different periods. Figure 8: Typical HE/CsI background spectra
in LGM from earth occultation observations in different periods.

Figure 9: The positions of $\thicksim$692 keV peak centroid (in PHA channel)
in NOM as a function of time.

Figure 10: The poisitions of $\thicksim$2767 keV peak centroid (in PHA
channel) in LGM as a function of time.

Because the gain of the HE/CsI(Na) detector changes significantly over time,
we had to calibrate the channel-energy relation of the HE/CsI(Na) monthly. The
energy responses of the HE/CsI(Na) detectors are calibrated using several
background lines emitted by radioactive isotopes in the structure of the
satellite, induced by high energy particles (the energy of the lines are
around 67, 191, 478, 692, 1182, 1372, 1729 and 2767 keV)Luo2020 ;
zhangjuan2020 . A linear function was adopted to describe the channel-energy
relationship of the HE/CsI(Na) in both normal mode and low-gain operation
mode:

$E(x_{\rm c})=k\cdot x_{\rm c}+b,$ (1)

where $E$ is the line energy in keV, $x_{\rm c}$ is the line centroid position
in channel, and $k$ represents the gain of each detector system in keV/ch.
Figures 11 and 12 show the evolutions of the fitted slope index k with time in
the two operation modes, respectively. For most CsI(Na) detectors, the
coefficients of $k$ have been gradually decreasing in the first five years,
except DetID 2 in LGM (see that in Figure 12).

Figure 11: Long-term monitor of the HE/CsI(Na) channel-energy relations in
NOM. Figure 12: Long-term monitor of the HE/CsI(Na) channel-energy relations
in LGM.

###  3.3 Performance evolution of PSD

As described in section 1, HE employed NaI(Tl)/CsI(Na) phoswich detectors to
observe compact objects in hard X-ray band of 20–250 keV. The PSD technology
is essential to select NaI(Tl) events as the science data in NOM. The detailed
technology was described and a figure of merit (FoM) was defined to
characterize the discriminating capacity of a PSD used by HE inLiu2020 . The
on-ground FoM values calibrated with a 133Ba radioactive source distributed
between 2.6 and 3.0, showing a good discriminating capacity.

To investigate the performance evolution of these PSDs, we select one day of
background data of HE in earth occultation about every two months to make
pulse width spectra. As described inLiu2020 , the pulse width spectrum
consists of two obvious peaks corresponding to NaI and CsI events,
respectively. We fit the two peaks with Gaussian functions to get the centroid
values in PSA channel and the peak widths in FWHM. Then we can get the FoM of
each PSD with these parameters. Figure 13 shows that the performances of the
PSDs maintained high quality throughout the first five years. The peak
centroids of eighteen NaI(Tl) detectors (gray dots) and CsI(Na) detectors
(black squares) are fairly stable, as shown in Figure 13(a). In Figure 13(b),
the peak widths of NaI(Tl) detectors (gray dots) show a decreasing trend of
about 0.9$\%$ to 2.8$\%$ per year; the peak widths of CsI(Na) detectors (black
squares) also show a decreasing trend of about 1$\%$ to 2.7$\%$ per year.
According to the definition of FoM inLiu2020 , the stability of the two peaks’
centroids and the decrease of the two peaks’ widths will lead to the FoM of
PSDs getting better. We can see that the FoMs are getting better at a level of
about 1.2$\%$ to 2.8$\%$ per year in Figure 13(c). The in-orbit FoMs of PSDs
are distributed between 2 and 2.7, which are slightly smaller than the values
of on-ground calibration.

Theoretically, the pulse widths of NaI(Tl) and CsI(Na) events should be
independent of the energy of incident X-ray photons. But in the practical
phoswich detector system, the pulse widths will show an obvious spread due to
the fluctuations of fluorescence light output and also the noise of the
electronics system. So the two peaks in the pulse width spectrum will be
broadened into two Gaussian bumps rather than two lines. The peak widths must
be taken into account when we analyse the FoM of a PSD. According to the on-
ground calibration results, the peak widths of NaI events and CsI events
decreased along with energy increase (see Figure 14). The on-ground
calibration source 133Ba has an emission line at about 80 keV, whereas the in-
orbit background has a continuous energy spectrum covering 20–250 keV. So the
FoM of a PSD is expected to become slightly worse with the in-orbit background
data than that of on-ground calibration with 133Ba, as shown in Figure 13(c).
Here we consider that the peak spread in the pulse width spectrum is mainly
caused by the fluctuations of the fluorescence yielding and collection
efficiency in the HED detectors. So the in-orbit FoMs of PSDs have been
getting better as the gains of NaI(Tl) and CsI(Na) detectors have been
increasing in the first five years, as described in detail in section 3.2.

Figure 13: Long-term monitor performances of PSDs. (a) the peak centroid in
PSA channel evolution of NaI(Tl) (gray dots) and CsI(Na) (black squares)
detectors over time. (b) the FWHM in PSA channel evolution of NaI(Tl) (gray
dots) and CsI(Na) (black squares) detectors over time. (c) the FoM evolution
of PSDs over time. Figure 14: The peak width of NaI events as a function of
its pulse height, obtained from on-ground calibration on DetID 2 and all the
HEDs have the similar behavior. The blue dots are the measured data and the
red line is the fitting curve with a simple power function.

###  3.4 Performance evolution of dead-time

Dead-time is also an important parameter of the HED detector system. When HE
observes bright sources (compact objects or transient bursts), the dead-time
of the detector system must be corrected accurately to get the true light
curves of the sources. The on-ground calibration results of dead-time were
reported in detailLiu2020 . Xiao et al. (2020) proposed a parameterized model
of the dead-time of the HE detector system to estimate the dead-time in any
time scale. The method was successfully applied to bright transient source
observationsXiao2020 . According to the results of on-ground calibration, the
maximum dead-time of one NaI event is less than 8 $\mu s$Liu2020 . To
investigate the evolution of dead-time, we select the background data of HE in
earth occultation observed on Sept. 23, 2017, Feb. 29, 2020, and Oct. 1, 2022
as typical samples to represents the dead-time performance in the early,
middle and late phase of the mission in the first 5 years. The spectra of the
time interval of adjacent events follow negative exponential distribution
perfectly, as shown in Figure 15. The point deviating from the negative
exponential distribution indicates the maximum dead-time of a good NaI event.
From the inset in each panel of Figure 15, we can find the dead-time of one
NaI event kept unchanged at about 8 $\mu s$ during the first five years. It
indicates that the electronics system of HE is very stable.

We select the background data of HE in earth occultation about every two
months. The data of each day are divided into several segments with every
segment 2000 s long. We calculate the dead-time proportion of every segment.
Then the evolution curve of the average proportion of in-orbit dead-time is
plotted in Figure 16. The dots are the average proportion of dead-time on the
specific days. Each error bar shows the maximum and minimum dead-time
proportion in the day. It is easy to find that the accumulated proportion of
dead-time is about 2 % – 6 $\%$, which is a fairly low value. The excellent
dead-time performance makes HE capable of observing bright sources. Even so,
some bright transient sources, especially the ultra-bright bursts, can make
the HEDs saturatedLiCK2021 , because the accumulated proportion of dead-time
is indeed related to the in-orbit count rate of HED.

Figure 15: Statistics of the arrival time interval between adjacent photons
for HED with DetID=4 in different years. The insets show the details around
the deviation point to clearly find out the maximum dead-time of one NaI
event. Figure 16: Average proportion evolution of in-orbit dead-time with time
during the first five years. The dots are the average proportion of dead-time
in the specific days. Each error bar shows the maximum and minimum dead-time
proportion in that day.

##  4 Summary

Three months after launch, the in-orbit performances of HE became stable. In
NOM, the gain variations of all the HE/NaI(Tl) detectors are within 1$\%$
throughout the first 5 years, which should be credited to the AGC system.
However, for the HE/CsI(Na) detectors in both NOM and LGM, the gain slowly
changed with time and had to be calibrated monthly. The performances of PSDs
have been maintained at a high level. The figures of merit of all PSDs are
larger than 2 and show a slow increasing evolution. The performances of the
dead-time do not show significant evolution in the first five years. In
summary, HE has performed excellently and has obtained fruitful important
science outcomes in the first five yearsLiTP2018 ; LiCK2021 ; Ma2021 ;
Kong2022 . For now, HE is still in good health and well-calibrated status. We
have been getting a comprehensive understanding of HE in the last five years.
We therefore expect HE to achieve greater success in the era of multi-
messenger astronomy, together with other instruments.

## Declarations

  * •

This work used data from the Insight-HXMT mission, a project funded by the
China National Space Administration (CNSA) and the Chinese Academy of Sciences
(CAS). We gratefully acknowledge the support from the National Program on Key
Research and Development Project (Grant No.2021YFA0718500) from the Minister
of Science and Technology of China (MOST). The authors thank supports from the
National Natural Science Foundation of China under Grants 12273043, U1838201,
U1838202, U1938109, U1938102, U1938108. This work was partially supported by
International Partnership Program of Chinese Academy of Sciences (Grant
No.113111KYSB20190020))

  * •

Conflict of interest On behalf of all authors, the corresponding author states
that there is no conflict of interest.

## References

  *   * (1) Zhang, S.N., Li, T.P., Lu, F.J., et al: Overview to the hard x-ray modulation telescope (Insight-hxmt) satellite. Sci. China-Phys. Mech. Astron. 63(4:249502), 1–18 (2020). https://doi.org/10.1007/s11433-019-1432-6
  * (2) Liu, C.Z., Zhang, Y.F., Li, X.F., et al: The high energy x-ray telescope (he) onboard the Insight-hxmt astronomy satellite. Sci. China-Phys. Mech. Astron. 63(4:249503), 1–15 (2020). https://doi.org/10.1007/s11433-019-1486-x
  * (3) Song, X.-Y., Xiong, S.-L., Zhang, S.-N., et al: The first Insight-hxmt gamma-ray burst catalog: The first four years. Astrophys. J., Suppl. Ser. 259, 46 (2022). https://doi.org/10.3847/1538-4365/ac4d22
  * (4) Li, X.F., Liu, C.Z., Chang, Z., et al: Ground-based calibration and characterization of the he detectors for Insight-hxmt. JHEAp 24, 6–14 (2019). https://doi.org/10.1016/j.jheap.2019.09.003
  * (5) Li, X.B., Li, X.F., Tan, Y., et al: In-flight calibration of the Insight-Hard X-ray Modulation Telescope. JHEAp 27, 64–76 (2020). https://doi.org/10.1016/j.jheap.2020.02.009
  * (6) Luo, Q., Liao, J.Y., Li, X.F., et al: Calibration of the instrumental response of Insight-HXMT/HE CsI detectors for gamma-ray monitoring. JHEAp 27, 1–13 (2020). https://doi.org/10.1016/j.jheap.2020.04.004
  * (7) Devare, H.G., Tandon, P.N.: EFFECT OF THE NON-LINEAR RESPONSE OF NaI(TI) ON THE SINGLE CRYSTAL SUMMING SPECTRA. Nuclear Instruments and Metods 22, 253–255 (1963)
  * (8) Wayne, L.R., Heindl, W.A., Hink, P.L., Rothschild, R.E.: Response of NaI(T1) to X-rays and electrons. Nucl. Instr. and Meth. in Phys. Res. A 411(2-3), 351–364 (1998). https://doi.org/10.1016/S0168-9002(98)00193-4
  * (9) Gardner, R.P., Mayo, C.W.: NaI detector nonlinearity for PGNAA applications. Applied Radiation and Isotopes 51, 189–195 (1999). https://doi.org/10.1016/S0969-8043(98)00183-3
  * (10) Payne, S.A., Moses, W.W., Sheets, S., et al: Nonproportionality of scintillator detectors: Theory and experiment. II. IEEE Transactions on Nuclear Science 58(6 PART 2), 3392–3402 (2011). https://doi.org/10.1109/TNS.2011.2167687
  * (11) Zhang, J., Li, X., Ge, M., et al: Comparison of simulated backgrounds with in-orbit observations for HE, ME, and LE onboard Insight-HXMT. Astrophysics and Space Science 365(9), 1–12 (2020). https://doi.org/10.1007/s10509-020-03873-8
  * (12) Xiao, S., Xiong, S.L., Liu, C.Z., et al: Deadtime calculation method of the High Energy X-ray telescope (HE) onboard the Insight-HXMT satellite. JHEAp 26, 58–64 (2020). https://doi.org/10.1016/j.jheap.2020.02.003
  * (13) Li, C.K., Lin, L., Xiong, S.L., et al: HXMT identification of a non-thermal X-ray burst from SGR J1935+2154 and with FRB 200428. Nature Astronomy 5(1), 378–384 (2021). https://doi.org/10.1038/s41550-021-01302-6
  * (14) Li, T.P., et al (The Insight-HXMT team): Insight-HXMT observations of the first binary neutron star merger GW170817. Sci. China-Phys. Mech. Astron. 61(3), 031011 (2018). https://doi.org/10.1007/s11433-017-9107-5
  * (15) Ma, X., Tao, L., Zhang, S.N., et al: Discovery of oscillations above 200 keV in a black hole X-ray binary with Insight-HXMT. Nature Astronomy 5(1), 94–102 (2021). https://doi.org/10.1038/s41550-020-1192-2
  * (16) Kong, L.-D., Zhang, S., Zhang, S.-N., et al: Insight-HXMT Discovery of the Highest-energy CRSF from the First Galactic Ultraluminous X-Ray Pulsar Swift J0243.6+6124. The Astrophysical Journal Letters 933(1), 3 (2022). https://doi.org/10.3847/2041-8213/ac7711