# PeakNet: An Autonomous Bragg Peak Finder with Deep Neural Networks

Cong Wang  Linac Coherent Light Source, SLAC National Accelerator Laboratory,
Menlo Park, CA, USA.  Po-Nan Li  Department of Electrical Engineering,
Stanford University, Stanford, CA, USA.  Jana Thayer  Linac Coherent Light
Source, SLAC National Accelerator Laboratory, Menlo Park, CA, USA.  Chun Hong
Yoon  Linac Coherent Light Source, SLAC National Accelerator Laboratory, Menlo
Park, CA, USA.  Corresponding author<EMAIL_ADDRESS>
###### Abstract

Serial crystallography at X-ray free electron laser (XFEL) and synchrotron
facilities has experienced tremendous progress in recent times enabling novel
scientific investigations into macromolecular structures and molecular
processes. However, these experiments generate a significant amount of data
posing computational challenges in data reduction and real-time feedback.
Bragg peak finding algorithm is used to identify useful images and also
provide real-time feedback about hit-rate and resolution. Shot-to-shot
intensity fluctuations and strong background scattering from buffer solution,
injection nozzle and other shielding materials make this a time-consuming
optimization problem. Here, we present PeakNet, an autonomous Bragg peak
finder that utilizes deep neural networks. The development of this system 1)
eliminates the need for manual algorithm parameter tuning, 2) reduces false-
positive peaks by adjusting to shot-to-shot variations in strong background
scattering in real-time, 3) eliminates the laborious task of manually creating
bad pixel masks and the need to store these masks per event since these can be
regenerated on demand. PeakNet also exhibits exceptional runtime efficiency,
processing a $1920\times 1920$ pixel image around 90 ms on an NVIDIA 1080 Ti
GPU, with the potential for further enhancements through parallelized analysis
or GPU stream processing. PeakNet is well-suited for expert-level real-time
serial crystallography data analysis at high data rates.

##  1 Introduction

Serial crystallography with X-ray free electron lasers (XFEL) has enabled
radiation damage-free structural determination of macromolecules at room
temperature, an approach commonly known as diffraction-before-destruction
(Neutze et al., 2000; Chapman et al., 2006, 2011). Time-resolved serial
femtosecond crystallography at XFEL facilities, like the Linac Coherent Light
Source (LCLS), allows the investigation of biochemical reactions with
femtosecond-scale time resolution which has led to many significant structural
studies (Kupitz et al., 2014; Nango et al., 2016; Pande et al., 2016; Young et
al., 2016; Suga et al., 2017; Kern et al., 2018; Ibrahim et al., 2020; Suga et
al., 2020) since the first published work at LCLS (Aquila et al., 2012). In
the meantime, the data rate at XFEL facilities have also gradually increased
from hundreds of Hz to the MHz range, with the European X-ray Free Electron
Laser (EuXFEL) being the first MHz facility available to users and LCLS-II set
to launch in 2023.

At the MHz data rate, novel real-time data analysis approaches are needed to
tackle two major challenges that arise in SFX experiments, namely data
reduction and real time feedback. The standard approach for data reduction is
vetoing events where the x-ray pulse did not “hit” the sample. Typically, hit
finding in serial crystallography utilizes Bragg peak finding algorithms
implemented in such programs as Cheetah (Barty et al., 2014), DIALS (Winter et
al., 2018), and Psocake (Yoon, 2020). Additionally, a deep neural network-
based hit finder for serial crystallography has been reported to achieve a
processing rate of 1.3 kHz by analyzing downsized 180 $\times$ 180 images on a
single GPU (Ke et al., 2018). Real-time feedback in serial crystallography
experiments plays a crucial role by providing users with important information
about the experimental conditions. This enables them to make timely decisions
that optimize the data collection process. For instance, it is essential to
fine-tune the hit rate to maximize the utilization of samples. Additionally,
it is important to monitor factors such as the current diffraction limit,
crystalline mosaicity, reciprocal space coverage, and other relevant
parameters over time. Therefore, relying solely on hit finding is insufficient
to gain the necessary insights, making real-time Bragg peak finding vital for
delivering prompt and comprehensive feedback.

In this work, we introduce PeakNet, a solution that effectively addresses the
challenge of autonomously and efficiently identifying Bragg peaks in high data
rate scenarios. Our approach involves transforming the peak finding problem
into a semantic segmentation task, where different regions within diffraction
images are classified and segmented using a deep neural network. PeakNet
autonomously detects Bragg peaks while effectively identifying shot-to-shot
variations of strong non-Bragg scattering artifacts, eliminating the need for
human intervention. Subsequently, connected areas of pixels are examined and
treated as individual peaks, a task easily accomplished through connected
component analysis (Weaver, 1985). The coordinates of the peaks are determined
by calculating the center of mass of all pixels belonging to each identified
peak. Notably, the entire peak finding process is performed on GPUs, offering
a cost-effective solution for achieving low latency.

##  2 Related work

Reliable and automatic peak finding algorithms have undergone multiple
iterations of development. An early approach involved utilizing template
matching (Wilkinson et al., 1988) based on libraries of peak shapes, but this
method was found to be slow and inefficient in runtime, especially when
dealing with peaks with low signal-to-noise ratios (SNR). The main obstacle
lies in the localization of peaks and the tuning of user parameters for
pattern matching, compounded by the difficult task of developing a
comprehensive library of peak shapes. Another influential approach employed
region-growing techniques (Bolotovsky et al., 1995; Barty et al., 2014) to
detect pixel areas with high skewness. In the context of serial
crystallography with XFEL sources, the efficacy of this method was diminished
due to the lack of sufficient data to provide reliable statistics, rendering
it susceptible to outliers and less adept in analyzing weak peaks at higher
scattering angles, which are essential for achieving atomic resolution in
structure determination.

While template matching and region-growing techniques require users to provide
prior information like threshold values, Robust Peak Finder (RPF) (Hadian-Jazi
et al., 2017, 2021) has demonstrated proficient peak finding capabilities with
minimal user input. This technique utilizes the Modified Selective Statistical
Estimator (MSSE) method to segment background pixels and actual peaks.
Furthermore, the RPF method can also parallelize the processing of diffraction
patterns, making it ideal for real-time data analysis.

Psocake is a software program specifically designed to streamline high
throughput data reduction and analysis of XFEL experiments, and is routinely
used at SLAC National Accelerator laboratory (SLAC). The built-in peak-finding
algorithm (Shin et al., 2018) includes calibrating raw analog digital units
(ADUs); applying optional background correction and bad pixel masks;
identifying possible peaks; calculating their SNR; determining Bragg spot
sizes; and selecting peaks based on size, total intensity and SNR.

In recent years, neural network models have emerged as a promising avenue for
Bragg peak finding. BraggNet (Sullivan et al., 2019) is the first method that
demonstrates proficiency of neural network models, specifically U-Net
(Ronneberger et al., 2015), in accurately segmenting peak pixels, including
weak peaks, from background pixels in neutron crystallography data. BraggNet
employed simulated peaks to create training datasets, which underwent a number
of preprocessing steps, including centering and cropping to a specfic size and
adding Poisson noise. It should be noted that BraggNet works on a single peak
at a time within a small 32 by 32 window, which deviates significantly from
conventional Bragg peak finding tasks that require extracting peak positions
from images typically with hundreds or thousands of pixels along one
dimension. Furthermore, another neural network method for Bragg peak analysis
was detailed in the work of BraggNN (Liu et al., 2021), emphasizing the
refinement of Bragg peak positions without the need for explicit fitting of a
profile function, such as the pseudo-Voigt profile. Likewise, BraggNN works on
a single peak at a time within a 11 by 11 window, performing regression to two
variables representing the peak position. To the best of our knowledge, no
existing neural network based peak-finding models have demonstrated the
ability to detect multiple peaks, which is the primary goal we strive to
achieve with PeakNet.

##  3 Methods

The current peak finding process of PeakNet comprises of two steps, as
depicted in Fig. 1. First, a segmentation map is obtained from the input image
using a deep neural network. Subsequently, peak positions are determined based
on the generated segmentation map using connected component analysis (Weaver,
1985). Importantly, PeakNet has the capability to process inputs with
arbitrary image sizes in batches, enabling efficient analysis.

In this section, our focus is to provide a comprehensive description of the
underlying deep neural network components in PeakNet. We will specifically
discuss the network architecture, data augmentation techniques, and the loss
function employed during the training process. Furthermore, we will emphasize
the use of iterative data curation in addressing the outlier problem.

Figure 1: The peak finding process of PeakNet consists of two steps: (1)
obtaining a segmentation map from the input using the underlying deep neural
network; (2) identifying peak positions based on the generated segmentation
map using connected component analysis. It is worth noting that PeakNet is
capable of accommodating input with arbitrary image sizes and batch sizes.

###  3.1 Neural network architecture

The underlying neural network architecture of PeakNet is illustrated in Figure
2 (a). It employs a residual attention U-Net architecture, comprising three
main components: a feature extraction “backbone”, a feature fusion “neck”, and
a single prediction “head”. The feature extraction “backbone” utilizes
multiple residual double convolutional blocks to extract multi-resolution
features, as visualized in Figure 2 (b). In the feature fusion “neck”, low-
resolution features are merged with high-resolution features using gated
feature fusion blocks, as depicted in 2 (c). Each block incorporates an
attention gate that enhances shared features within both low and high-
resolution feature maps. The resulting combined features from each gated
feature fusion block are then rearranged by a subsequent residual double
convolutional block. Finally, to accomplish the segmentation task, the
prediction “head” leverages fused features from the “neck” and generates a
prediction map encompassing three distinct labels: background, Bragg peak, and
artifact scattering. The “head” itself consists of a 1 $\times$ 1
convolutional layer followed by a softmax function.

Our neural network architecture is primarily based on the original “Attention
U-Net” design (Oktay et al., 2018). However, we want to point out that the
three-component design, consisting of the “backbone”, “neck” and “head”,
offers the flexibility to replace these components with different
architectures. For instance, the “backbone” can be substituted with a ResNet
(He et al., 2016), while the “neck” can be replaced with other multi-
resolution feature fusion methods like feature pyramid networks (FPN) (Lin et
al., 2017) or bidirectional feature pyramid networks (BiFPN) (Tan et al.,
2020). Additionally, the “head” can be extended to perform additional tasks,
if desired, such as predicting the likelihood of the input image being
indexable. This modular approach provides flexibility and allows for
customization based on specific requirements.

Figure 2: Neural network architecture: Attention U-Net with residual
connections. (a) Overall end-to-end network architecture. (b) Residual double
convolutional block. (c) Gated feature fusion block for merging multi-
resolution features.

###  3.2 Focal loss as the loss function

One main technical issue in training neural networks for Bragg peak
segmentation is the extreme peak-background class imbalance (e.g., less than
$1:10^{3}$), resulting in reduced prediction accuracy. This may not pose as a
problem in previous peak finders like BraggNet, which operates on a much
smaller area of $32\times 32$, PeakNet is trained on larger detectors, such as
the Rayonix MX340-XFEL detector with a dimension of 1920 $\times$ 1920 pixels
even after $4\times 4$ pixel binning. To mitigate the label imbalance, we
employ a categorical focal loss to address the problem (Lin et al., 2018).

CFL $\displaystyle=-\sum_{i=1}^{N}\sum_{j=1}^{C}\alpha_{j}\cdot
y_{j}^{(i)}\cdot(1-\hat{p}_{j}^{(i)})^{\gamma}\cdot\log{\hat{p}_{j}^{(i)}},$
(1) $\displaystyle\hat{p}_{j}^{(i)}$ $\displaystyle=f_{\theta}(x_{j}^{(i)})$
(2)

where CFL stands for categorical focal loss, $\alpha_{j}$ is a balancing
factor for each class, $y_{j}^{(i)}$ is the ground truth label of the $j$-th
class for the $i$-th pixel. $p_{j}^{(i)}$ is the predicted probability of the
$j$-th class for the $i$-th pixel, calculated by the neural network $f$
parameterized by $\theta$ given input image $x_{j}^{(i)}$ with $N$ number of
pixels and $C$ categories. $\gamma$ is a parameter controlling the extent to
which the weight of well-classified examples, such as background pixels, is
reduced. In our neural network training, we chose $\gamma=2.0$, essentially,
this means $(1-\hat{p}_{j}^{(i)})^{\gamma}$ is solely responsible for down-
weighting easy examples, as $\hat{p}_{j}^{(i)}$ is usually large in this case.
Finding the ideal values for $\alpha$ can be a delicate process with only
marginal returns, as highlighted in the original focal loss study (Lin et al.,
2018). In order to fully utilize focal loss, we decided to use a fixed trivial
value of $\alpha=1.2$ for all examples, primarily for the sake of
completeness. This choice essentially treats $\alpha$ as a simple scaling
factor without significant impact. In most cases, the sum of all $\alpha_{j}$
values, denoted as $\sum_{j=1}^{C}\alpha_{j}$, sums to 1.

###  3.3 Data augmentation

We applied three data augmentation techniques to each diffraction pattern in
our dataset, including random in-plane rotation, random shifting in both
horizontal and vertical directions and random masking. Random in-plane
rotation and shifting augment the training data, reducing the model’s tendency
to memorize features at specific locations. This variability promotes more
generalized feature extraction and improved generalization performance. Random
masking involves randomly obscuring parts of the input data, forcing the model
to learn other relevant features that enhance its predictive capabilities
during training. 50 rectangular masks with random sizes between 80 and 120
were positioned randomly. The result of applying data augmentation to an
arbitrary example is shown in Fig. 3.

Figure 3: Illustration of data augmentation on an input image. (a) Augmented
image with random in-plane rotation, shift, and masking. (b) its corresponding
labeled segmentation map. (c, d) Zoomed-in views of (a) and (b), respectively.
The segmentation label distinguishes red pixels as peaks and blue pixels as
artifact scattering, attributed to scattering from metal shielding and
injection nozzle.

###  3.4 Dataset improvement and correction

PeakNet employs a data-driven approach to accurately identify Bragg peaks,
requiring dataset improvement and correction to enhance performance. This
contrasts with model-driven approaches that primarily entail direct model
modification to improve capabilities. Fig. 4 illustrates a feedback loop for
iterative dataset improvement and correction.

The process begins with the preparation of a source dataset, often
accomplished through a combination of algorithmic and manual data labeling.
Subsequently, the deep neural networks within PeakNet are trained on this
dataset to learn how to accurately detect peaks. Following training, the model
is deployed to predict peaks on previously unseen data, and its performance is
thoroughly evaluated. During this evaluation, instances that notably worsen
the performance of PeakNet are identified. To address these limitations, we
either correct low-quality labels predicted by the neural networks or,
alternatively, acquire additional data that closely resemble these challenging
instances and carefully annotate them. Finally, we merge these newly labeled
instances into the training set and retrain the model, thereby improving its
overall performance.

Interestingly, during the initial pass of this iterative process, PeakNet
showed limited performance when dealing with images containing relatively low
levels of photons. However, by obtaining and annotating these images, we were
able to enhance the training dataset and, as a result, significantly improve
the predictive performance of PeakNet.

Figure 4: The three step process of iterative dataset and model improvement.
(1) Model learns from training data. (2) Model predictions reveal potential
areas of improvement. (3) Correct low-quality training labels or acquire
additional examples of challenging instances.

##  4 Results

The primary goal of PeakNet is to autonomously detect Bragg peaks in
diffraction images, devoid of any user input, such as algorithmic parameter
tuning or manual artifact masking. We implemented a two-step approach for peak
detection, employing deep neural networks to perform semantic segmentation,
which was subsequently followed by connected component analysis to extract
image coordinates of Bragg peaks. In this section, we begin by introducing the
datasets used for training, validating, and testing PeakNet. Next, we present
the evaluation results that demonstrate the predictive performance of PeakNet.
Finally, we showcase its auto-masking capability and runtime efficiency.

###  4.1 Dataset

We obtained X-ray diffraction images from two experiments conducted at the
Macromolecular Femtosecond Crystallography (MFX) instrument within LCLS. The
primary dataset utilized for training and evaluating PeakNet is “mfx13016”
(Run number: 28,31-34, 36-38), which contains diverse biological samples, e.g.
thaumatin and proteinase, and this experiment was originally used for
automated drop dispensing research (Su et al., 2021). These diffraction images
were recorded on a Rayonix MX340-XFEL detector with a dimension of 1920
$\times$ 1920, downsized by a factor of 4 to accommodate a higher readout
rate.

The dataset “mfx13016” exhibits a notable ring-like background scattering
accompanied by bright Bragg peak-like spots located on the rings. This
characteristic presents two primary challenges for crystallography data
analysis: increased difficulty in crystal indexing due to false-positive peaks
and inaccurate integration results caused by artifact scattering. The current
best practice involves manual masking of these artifact scattering, enabling
data analysis to proceed as if they are absent, but this process is time-
consuming and can not adjust to shot-to-shot variations in the artifacts in
real-time. Additionally, the final outcomes often hinge on the expertise of
the individuals conducting the data processing, resulting in biases,
scalability limitations, lack of standardization, and the possibility of human
errors. Nevertheless, the inherent challenges presented by this dataset offer
an exceptional opportunity to showcase the autonomous peak finding capability
of PeakNet.

The neural network training dataset consists of 98 Rayonix images in run 28 of
“mfx13016”, which includes thaumatin samples, and an additional 37 images
sampled from an internal LCLS experimental dataset related to therapeutics
development associated with SARS-CoV-2. Incorporating the SARS-CoV-2 dataset
aims to educate PeakNet about the presence of images without artifact rings,
diversifying its understanding of different data characteristics. Nonetheless,
all these images used for neural network training have a dimension of 1920
$\times$ 1920 and they are divided into an 80% training set and a 20%
validation set. The final test dataset for evaluating the model comprises all
images in run 31-34 and run 36-38 of “mfx13016”.

###  4.2 Evaluating PeakNet’s predictive performance

The peak finding performance of peak finders is evaluated through three key
indicators from a user’s standpoint: the number of successfully indexed hits,
the indexing rate, and the merging statistics. For our evaluation, we utilized
the CrystFEL software suite (White et al., 2012) to perform crystal indexing,
integration, and merging tasks. We use Psocake’s peak finder as a baseline in
this evaluation.

####  4.2.1 Indexing results

PeakNet and manually fine-tuned Psocake exhibit comparable peak finding
performance in terms of the number of hits and the number of indexed images,
as shown in Table 1. However, we emphasize that while PeakNet operates
entirely autonomously, Psocake’s performance is contingent upon manual
parameter fine-tuning and manual artifact scattering masking. For example, an
earlier work (Su et al., 2021), which processed the same experimental data,
reported significantly lower hit count and indexed image count. Consequently,
this highlights PeakNet’s advantage in user-independent expert-level data
processing.

Moreover, while users might prioritize the absolute number of indexed images
as a more straightforward metric, the indexing rate provides valuable insights
into the method’s efficiency. Both metrics jointly offer a more comprehensive
evaluation of the peak finding method. In the case of PeakNet, both
consistently high number of indexed images and high indexing rate testify to
its effective and reliable autonomous peak finding process.

Table 1: Comparison of crystal indexing results for different methods on the
test dataset. The table shows the number of hits, number of indexed hits, and
indexing rates for PeakNet, Psocake (fine-tuned) and Psocake (Su 2021). The
“Run (Samples)” column lists the specific run numbers and biological samples
(TH: thaumatin; PK: proteinase).

Methods | Run (Samples) | 31 (TH) | 32 (TH) | 33 (TH) | 34 (TH) | 36 (TH) | 37 (PK) | 38 (PK)
---|---|---|---|---|---|---|---|---
PeakNet | Number of hits | 1205 | 3248 | 5797 | 3949 | 1184 | 1030 | 1223
Number of indexed | 860 | 2856 | 3433 | 2401 | 919 | 953 | 1169
Indexing rate | 0.71 | 0.88 | 0.59 | 0.61 | 0.78 | 0.93 | 0.96
Psocake (fine-tuned) | Number of hits | 1193 | 3087 | 5769 | 3926 | 917 | 1070 | 1212
Number of indexed | 849 | 2657 | 3375 | 2364 | 760 | 1005 | 1202
Indexing rate | 0.71 | 0.86 | 0.59 | 0.60 | 0.83 | 0.94 | 0.99
Psocake (Su2021) | Number of hits | 1083 | 2454 | 5604 | 3756 | 363 | 683 | 771
Number of indexed | 271 | 1094 | 963 | 770 | 266 | 569 | 709
Indexing rate | 0.25 | 0.45 | 0.17 | 0.21 | 0.73 | 0.83 | 0.92

####  4.2.2 Merging statistics

Merging statistics confirms PeakNet’s robust performance in terms of
consistency and quality in data processing. For the thaumatin (TH) sample,
PeakNet demonstrates marginally superior performance in terms of R-split
values, correlation coefficients, $I/\sigma(I)$ values, redundancy, and
completeness, achieving these results autonomously. Notably, the R-split value
for thaumatin is slightly lower for PeakNet (0.2605 vs. 0.2732), suggesting
slightly better internal consistency. Similarly, correlation coefficients for
PeakNet are marginally higher (CC${}_{1}/2$: 0.9434 vs. 0.9368; CC*: 0.9853
vs. 0.9835), indicating a minor advantage in the precision of measurements.

However, for the proteinase (PK) sample, the performance comparison is mixed.
Although PeakNet lags slightly behind manually fine-tuned Psocake in terms of
R-split and redundancy, it outperforms in CC${}_{1}/2$, CC*, $I/\sigma(I)$,
and completeness values. This reinforces the effectiveness of PeakNet in
achieving respectable results across varied sample types, without any manual
intervention or parameter fine-tuning.

Overall, PeakNet demonstrates comparable performance to a manually fine-tuned
Psocake for two distinct samples, showcasing its potential for autonomous peak
finding with consistent results. By reducing dependence on the expertise of
individuals, which can have considerable variation, PeakNet offers a promising
user-independent solution. Notably, despite primarily being trained on PK
samples, PeakNet maintains robust and versatile performance, indicating its
broad applicability and effectiveness in handling diverse datasets.

Table 2: Comparison of merging statistics between PeakNet and fine-tuned
Psocake. Thaumatin (TH) results were merged from run 31, 32, 33, 34 and 36.
Proteinase (PK) results were merged from run 37 and 38.

Sample | Methods | R-split | CC1/2 | CC* | $I/\sigma(I)$ | Resolution range (Å) | Redundancy | Completeness
---|---|---|---|---|---|---|---|---
TH | PeakNet | 0.2605 | 0.9434 | 0.9853 | 2.0590 | 1.45-27.93 | 76.05 | 0.9594
Psocake | 0.2732 | 0.9368 | 0.9835 | 2.0277 | 1.01-31.32 | 72.20 | 0.9367
|  |  |  |  |  |  |  |
Sample | Methods | R-split | CC1/2 | CC* | $I/\sigma(I)$ | Resolution range (Å) | Redundancy | Completeness
PK | PeakNet | 0.7711 | 0.2234 | 0.6043 | 1.135 | 1.45-26.98 | 16.36 | 0.8668
Psocake | 0.7589 | 0.1900 | 0.5650 | 0.974 | 1.45-30.04 | 17.83 | 0.8365

###  4.3 Auto-masking: predicting scattering artifacts

Auto-masking functionality in PeakNet represents a significant advancement in
the automated processing of diffraction images. The underlying neural network
architecture enables PeakNet to autonomously discern and segregate pixel
regions affected by scattering artifacts, such as distinct ring-like
structures produced by metal shielding used to reduce air scatter and/or
liquid jet nozzle scattering. This capability circumvents the need for manual
masking in the peak finding process and the need to store a binary mask used
for each event which is needed for crystal indexing/integration, thereby
optimizing the efficiency of crystallographic data processing.

Fig. 5 provides a comparative visualization of PeakNet’s effectiveness in
accurately segmenting regions with bright pixels induced by scattering
artifacts under various conditions. Images without ring artifacts are also
included in the visualization, reassuring that PeakNet does not generate or
“hallucinate” ring artifacts where they do not exist. This visual
representation serves as tangible evidence of PeakNet ’s adeptness in
discriminating and processing pertinent features, regardless of the contextual
complexities.

Furthermore, while evaluating PeakNet’s auto-masking feature on intended input
images, we explored its behavior on unintended inputs, providing insights into
its robustness and generalizability. Fig. 6 depicts illustrative examples of
PeakNet predicting artifact scattering patterns in silver behenate scattering
captured on various detectors. Silver behenate is a popular calibrant used to
determine the detector position relative to the beam based on the known ring
spacing. Surprisingly, PeakNet reliably identifies the characteristic
scattering ring structures present in silver behenate images. This observation
provides compelling evidence once again, further demonstrating the robust
predictive capability of PeakNet in accurately recognizing artifact scattering
patterns. As a result, it reinforces that effectiveness of PeakNet’s auto-
masking feature and the potential to automatically optimize the diffraction
geometry.

Figure 5: Examples of auto-masking using PeakNet. Each panel contains three
images: the original image, the segmentation map used for masking, and the
resulting mask after thresholding. Panels (a), (c) and (e) show three
different scenarios with artifact scattering rings. Panels (b), (d) and (f)
visualize the response of PeakNet on images without any artifact scattering
rings. Figure 6: Examples of identifying scattering rings in silver behenate
images using PeakNet.

###  4.4 Runtime efficiency

Peak finding can pose a significant computational demand, particularly in high
data rate experiments, highlighting the critical importance of an efficient
peak finding algorithm. A comparative analysis of runtime performance
demonstrates the greater efficiency of PeakNet compared to other commonly used
Psocake peak finder.

When processing Rayonix images with $1920\times 1920$ pixels, PeakNet achieved
a significantly reduced processing time of 90 milliseconds per event on an
NVIDIA 1080 Ti GPU. In contrast, Psocake required 176 milliseconds per event
when executed on an Intel E5-2620 CPU. Another peak finding method, RPF
(robust peak finder), operating on Intel E5-2698, is reported to require 120
milliseconds but on a smaller AGIPD 1M images (Hadian-Jazi et al., 2021). This
result clearly demonstrates that PeakNet not only delivers robust Bragg peak
finding results in an autonomous fashion, but also exhibits remarkable runtime
efficiency.

Table 3:  The runtime performance of diffraction image analysis algorithms
measured on Rayonix images containing 3.7 megapixels. We measured runtime
performance of Psocake and PeakNet directly using Rayonix images. For
reference purposes, we also included another peak finding method RPF (Hadian-
Jazi et al., 2017, 2021) and two classification methods for X-ray diffraction
data reduction. RPF runtime performance was originally measured on AGIPD 1M
(Allahgholi et al., 2019).

Methods | Hardware | Image size | Time (ms/event)
---|---|---|---
PeakNet | NVIDIA 1080 Ti | 1920 $\times$ 1920 | 90.0
Psocake | Intel E5-2620 | 1920 $\times$ 1920 | 175.5
RPF | Intel E5-2698 | 1024 $\times$ 1024 | 120.0

##  5 Conclusions

PeakNet exhibits exceptional capabilities in autonomous Bragg peaks finding
without requiring user input or intervention, such as algorithmic parameter
tuning or manual artifact masking. The predictive performance of PeakNet in
peak finding proves to be comparable to, if not better than, manually fine-
tuned Psocake with user-supplied masks. An integral feature of PeakNet, auto-
masking, significantly enhances its ability to autonomously identify and
process artifact scattering patterns. Moreover, the efficiency of PeakNet is
exemplified by its impressive runtime performance, surpassing commonly used
peak finders like Psocake and RPF. Parallel processing with GPU streaming can
potentially improve the runtime efficiency even further. In conclusion,
PeakNet offers a robust, versatile, and efficient solution that operates
independently of user input, thereby holding significant potential for
optimizing crystallographic data processing at high data rates.

## Acknowledgment

Earlier versions utilizing YOLO and U-Net models were built and tested by P.L
and C.H.Y. The experiments were designed by C.W with inputs from J.B.T and
C.H.Y. C.W implemented the residual attention U-Net, prepared the datasets,
labeled experimental data, and trained and evaluated the refined neural
network model. The manuscript was written by C.W and C.H.Y with input from all
authors. This material is based upon work supported by the U.S. Department of
Energy, Office of Science, Office of Basic Energy Sciences under Award Number
FWP-100643. Use of the Linac Coherent Light Source (LCLS), SLAC National
Accelerator Laboratory, is supported by the U.S. Department of Energy, Office
of Science, Office of Basic Energy Sciences under Contract No.DE-
AC02-76SF00515. C.W acknowledges the assistance provided by ChatGPT from
OpenAI in refining the language and enhancing the readability of this paper.

## References

  * Allahgholi et al. [2019] Aschkan Allahgholi, Julian Becker, Annette Delfs, Roberto Dinapoli, Peter Goettlicher, Dominic Greiffenberg, Beat Henrich, Helmut Hirsemann, Manuela Kuhn, Robert Klanner, Alexander Klyuev, Hans Krueger, Sabine Lange, Torsten Laurus, Alessandro Marras, Davide Mezza, Aldo Mozzanica, Magdalena Niemann, Jennifer Poehlsen, Joern Schwandt, Igor Sheviakov, Xintian Shi, Sergej Smoljanin, Lothar Steffen, Jolanta Sztuk-Dambietz, Ulrich Trunk, Qingqing Xia, Mourad Zeribi, Jiaguo Zhang, Manfred Zimmer, Bernd Schmitt, and Heinz Graafsma.  The Adaptive Gain Integrating Pixel Detector at the European XFEL.  _Journal of Synchrotron Radiation_ , 26(1):74–82, January 2019.  ISSN 1600-5775.  doi: 10.1107/S1600577518016077.
  * Aquila et al. [2012] Andrew Aquila, Mark S. Hunter, R. Bruce Doak, Richard A. Kirian, Petra Fromme, Thomas A. White, Jakob Andreasson, David Arnlund, Saša Bajt, Thomas R. M. Barends, Miriam Barthelmess, Michael J. Bogan, Christoph Bostedt, Hervé Bottin, John D. Bozek, Carl Caleman, Nicola Coppola, Jan Davidsson, Daniel P. DePonte, Veit Elser, Sascha W. Epp, Benjamin Erk, Holger Fleckenstein, Lutz Foucar, Matthias Frank, Raimund Fromme, Heinz Graafsma, Ingo Grotjohann, Lars Gumprecht, Janos Hajdu, Christina Y. Hampton, Andreas Hartmann, Robert Hartmann, Stefan Hau-Riege, Günter Hauser, Helmut Hirsemann, Peter Holl, James M. Holton, André Hömke, Linda Johansson, Nils Kimmel, Stephan Kassemeyer, Faton Krasniqi, Kai-Uwe Kühnel, Mengning Liang, Lukas Lomb, Erik Malmerberg, Stefano Marchesini, Andrew V. Martin, Filipe R.N.C. Maia, Marc Messerschmidt, Karol Nass, Christian Reich, Richard Neutze, Daniel Rolles, Benedikt Rudek, Artem Rudenko, Ilme Schlichting, Carlo Schmidt, Kevin E. Schmidt, Joachim Schulz, M. Marvin Seibert, Robert L. Shoeman, Raymond Sierra, Heike Soltau, Dmitri Starodub, Francesco Stellato, Stephan Stern, Lothar Strüder, Nicusor Timneanu, Joachim Ullrich, Xiaoyu Wang, Garth J. Williams, Georg Weidenspointner, Uwe Weierstall, Cornelia Wunderer, Anton Barty, John C. H. Spence, and Henry N. Chapman.  Time-resolved protein nanocrystallography using an X-ray free-electron laser.  _Optics Express_ , 20(3):2706, January 2012.  ISSN 1094-4087.  doi: 10.1364/OE.20.002706.
  * Barty et al. [2014] Anton Barty, Richard A. Kirian, Filipe R. N. C. Maia, Max Hantke, Chun Hong Yoon, Thomas A. White, and Henry Chapman.  _Cheetah_ : Software for high-throughput reduction and analysis of serial femtosecond X-ray diffraction data.  _Journal of Applied Crystallography_ , 47(3):1118–1131, June 2014.  ISSN 1600-5767.  doi: 10.1107/S1600576714007626.
  * Bolotovsky et al. [1995] R. Bolotovsky, M. A. White, A. Darovsky, and P. Coppens.  The ‘Seed-Skewness’ Method for Integration of Peaks on Imaging Plates.  _Journal of Applied Crystallography_ , 28(2):86–95, April 1995.  ISSN 0021-8898.  doi: 10.1107/S0021889894009696.
  * Chapman et al. [2006] Henry N. Chapman, Anton Barty, Michael J. Bogan, Sébastien Boutet, Matthias Frank, Stefan P. Hau-Riege, Stefano Marchesini, Bruce W. Woods, Saša Bajt, W. Henry Benner, Richard A. London, Elke Plönjes, Marion Kuhlmann, Rolf Treusch, Stefan Düsterer, Thomas Tschentscher, Jochen R. Schneider, Eberhard Spiller, Thomas Möller, Christoph Bostedt, Matthias Hoener, David A. Shapiro, Keith O. Hodgson, David van der Spoel, Florian Burmeister, Magnus Bergh, Carl Caleman, Gösta Huldt, M. Marvin Seibert, Filipe R. N. C. Maia, Richard W. Lee, Abraham Szöke, Nicusor Timneanu, and Janos Hajdu.  Femtosecond diffractive imaging with a soft-X-ray free-electron laser.  _Nature Physics_ , 2(12):839–843, December 2006\.  ISSN 1745-2473, 1745-2481.  doi: 10.1038/nphys461.
  * Chapman et al. [2011] Henry N. Chapman, Petra Fromme, Anton Barty, Thomas A. White, Richard A. Kirian, Andrew Aquila, Mark S. Hunter, Joachim Schulz, Daniel P. DePonte, Uwe Weierstall, R. Bruce Doak, Filipe R. N. C. Maia, Andrew V. Martin, Ilme Schlichting, Lukas Lomb, Nicola Coppola, Robert L. Shoeman, Sascha W. Epp, Robert Hartmann, Daniel Rolles, Artem Rudenko, Lutz Foucar, Nils Kimmel, Georg Weidenspointner, Peter Holl, Mengning Liang, Miriam Barthelmess, Carl Caleman, Sébastien Boutet, Michael J. Bogan, Jacek Krzywinski, Christoph Bostedt, Saša Bajt, Lars Gumprecht, Benedikt Rudek, Benjamin Erk, Carlo Schmidt, André Hömke, Christian Reich, Daniel Pietschner, Lothar Strüder, Günter Hauser, Hubert Gorke, Joachim Ullrich, Sven Herrmann, Gerhard Schaller, Florian Schopper, Heike Soltau, Kai-Uwe Kühnel, Marc Messerschmidt, John D. Bozek, Stefan P. Hau-Riege, Matthias Frank, Christina Y. Hampton, Raymond G. Sierra, Dmitri Starodub, Garth J. Williams, Janos Hajdu, Nicusor Timneanu, M. Marvin Seibert, Jakob Andreasson, Andrea Rocker, Olof Jönsson, Martin Svenda, Stephan Stern, Karol Nass, Robert Andritschke, Claus-Dieter Schröter, Faton Krasniqi, Mario Bott, Kevin E. Schmidt, Xiaoyu Wang, Ingo Grotjohann, James M. Holton, Thomas R. M. Barends, Richard Neutze, Stefano Marchesini, Raimund Fromme, Sebastian Schorb, Daniela Rupp, Marcus Adolph, Tais Gorkhover, Inger Andersson, Helmut Hirsemann, Guillaume Potdevin, Heinz Graafsma, Björn Nilsson, and John C. H. Spence.  Femtosecond X-ray protein nanocrystallography.  _Nature_ , 470(7332):73–77, February 2011.  ISSN 0028-0836, 1476-4687.  doi: 10.1038/nature09750.
  * Hadian-Jazi et al. [2017] Marjan Hadian-Jazi, Marc Messerschmidt, Connie Darmanin, Klaus Giewekemeyer, Adrian P. Mancuso, and Brian Abbey.  A peak-finding algorithm based on robust statistical analysis in serial crystallography.  _Journal of Applied Crystallography_ , 50(6):1705–1715, December 2017.  ISSN 1600-5767.  doi: 10.1107/S1600576717014340.
  * Hadian-Jazi et al. [2021] Marjan Hadian-Jazi, Alireza Sadri, Anton Barty, Oleksandr Yefanov, Marina Galchenkova, Dominik Oberthuer, Dana Komadina, Wolfgang Brehm, Henry Kirkwood, Grant Mills, Raphael de Wijn, Romain Letrun, Marco Kloos, Mohammad Vakili, Luca Gelisio, Connie Darmanin, Adrian P. Mancuso, Henry N. Chapman, and Brian Abbey.  Data reduction for serial crystallography using a robust peak finder.  _Journal of Applied Crystallography_ , 54(5):1360–1378, October 2021.  ISSN 1600-5767.  doi: 10.1107/S1600576721007317.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, Las Vegas, NV, USA, June 2016. IEEE.  ISBN 978-1-4673-8851-1.  doi: 10.1109/CVPR.2016.90.
  * Ibrahim et al. [2020] Mohamed Ibrahim, Thomas Fransson, Ruchira Chatterjee, Mun Hon Cheah, Rana Hussein, Louise Lassalle, Kyle D. Sutherlin, Iris D. Young, Franklin D. Fuller, Sheraz Gul, In-Sik Kim, Philipp S. Simon, Casper de Lichtenberg, Petko Chernev, Isabel Bogacz, Cindy C. Pham, Allen M. Orville, Nicholas Saichek, Trent Northen, Alexander Batyuk, Sergio Carbajo, Roberto Alonso-Mori, Kensuke Tono, Shigeki Owada, Asmit Bhowmick, Robert Bolotovsky, Derek Mendez, Nigel W. Moriarty, James M. Holton, Holger Dobbek, Aaron S. Brewster, Paul D. Adams, Nicholas K. Sauter, Uwe Bergmann, Athina Zouni, Johannes Messinger, Jan Kern, Vittal K. Yachandra, and Junko Yano.  Untangling the sequence of events during the S 2 → S 3 transition in photosystem II and implications for the water oxidation mechanism.  _Proceedings of the National Academy of Sciences_ , 117(23):12624–12635, June 2020.  ISSN 0027-8424, 1091-6490.  doi: 10.1073/pnas.2000529117.
  * Ke et al. [2018] Tsung-Wei Ke, Aaron S. Brewster, Stella X. Yu, Daniela Ushizima, Chao Yang, and Nicholas K. Sauter.  A convolutional neural network-based screening tool for X-ray serial crystallography.  _Journal of Synchrotron Radiation_ , 25(3):655–670, May 2018.  ISSN 1600-5775.  doi: 10.1107/S1600577518004873.
  * Kern et al. [2018] Jan Kern, Ruchira Chatterjee, Iris D. Young, Franklin D. Fuller, Louise Lassalle, Mohamed Ibrahim, Sheraz Gul, Thomas Fransson, Aaron S. Brewster, Roberto Alonso-Mori, Rana Hussein, Miao Zhang, Lacey Douthit, Casper de Lichtenberg, Mun Hon Cheah, Dmitry Shevela, Julia Wersig, Ina Seuffert, Dimosthenis Sokaras, Ernest Pastor, Clemens Weninger, Thomas Kroll, Raymond G. Sierra, Pierre Aller, Agata Butryn, Allen M. Orville, Mengning Liang, Alexander Batyuk, Jason E. Koglin, Sergio Carbajo, Sébastien Boutet, Nigel W. Moriarty, James M. Holton, Holger Dobbek, Paul D. Adams, Uwe Bergmann, Nicholas K. Sauter, Athina Zouni, Johannes Messinger, Junko Yano, and Vittal K. Yachandra.  Structures of the intermediates of Kok’s photosynthetic water oxidation clock.  _Nature_ , 563(7731):421–425, November 2018.  ISSN 0028-0836, 1476-4687.  doi: 10.1038/s41586-018-0681-2.
  * Kupitz et al. [2014] Christopher Kupitz, Shibom Basu, Ingo Grotjohann, Raimund Fromme, Nadia A. Zatsepin, Kimberly N. Rendek, Mark S. Hunter, Robert L. Shoeman, Thomas A. White, Dingjie Wang, Daniel James, Jay-How Yang, Danielle E. Cobb, Brenda Reeder, Raymond G. Sierra, Haiguang Liu, Anton Barty, Andrew L. Aquila, Daniel Deponte, Richard A. Kirian, Sadia Bari, Jesse J. Bergkamp, Kenneth R. Beyerlein, Michael J. Bogan, Carl Caleman, Tzu-Chiao Chao, Chelsie E. Conrad, Katherine M. Davis, Holger Fleckenstein, Lorenzo Galli, Stefan P. Hau-Riege, Stephan Kassemeyer, Hartawan Laksmono, Mengning Liang, Lukas Lomb, Stefano Marchesini, Andrew V. Martin, Marc Messerschmidt, Despina Milathianaki, Karol Nass, Alexandra Ros, Shatabdi Roy-Chowdhury, Kevin Schmidt, Marvin Seibert, Jan Steinbrener, Francesco Stellato, Lifen Yan, Chunhong Yoon, Thomas A. Moore, Ana L. Moore, Yulia Pushkar, Garth J. Williams, Sébastien Boutet, R. Bruce Doak, Uwe Weierstall, Matthias Frank, Henry N. Chapman, John C. H. Spence, and Petra Fromme.  Serial time-resolved crystallography of photosystem II using a femtosecond X-ray laser.  _Nature_ , 513(7517):261–265, September 2014\.  ISSN 0028-0836, 1476-4687.  doi: 10.1038/nature13453.
  * Lin et al. [2017] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature Pyramid Networks for Object Detection, April 2017.
  * Lin et al. [2018] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal Loss for Dense Object Detection, February 2018.
  * Liu et al. [2021] Zhengchun Liu, Hemant Sharma, Jun-Sang Park, Peter Kenesei, Antonino Miceli, Jonathan Almer, Rajkumar Kettimuthu, and Ian Foster.  BraggNN: Fast X-ray Bragg Peak Analysis Using Deep Learning.  _arXiv:2008.08198 [cs, eess]_ , June 2021.
  * Nango et al. [2016] Eriko Nango, Antoine Royant, Minoru Kubo, Takanori Nakane, Cecilia Wickstrand, Tetsunari Kimura, Tomoyuki Tanaka, Kensuke Tono, Changyong Song, Rie Tanaka, Toshi Arima, Ayumi Yamashita, Jun Kobayashi, Toshiaki Hosaka, Eiichi Mizohata, Przemyslaw Nogly, Michihiro Sugahara, Daewoong Nam, Takashi Nomura, Tatsuro Shimamura, Dohyun Im, Takaaki Fujiwara, Yasuaki Yamanaka, Byeonghyun Jeon, Tomohiro Nishizawa, Kazumasa Oda, Masahiro Fukuda, Rebecka Andersson, Petra Båth, Robert Dods, Jan Davidsson, Shigeru Matsuoka, Satoshi Kawatake, Michio Murata, Osamu Nureki, Shigeki Owada, Takashi Kameshima, Takaki Hatsui, Yasumasa Joti, Gebhard Schertler, Makina Yabashi, Ana-Nicoleta Bondar, Jörg Standfuss, Richard Neutze, and So Iwata.  A three-dimensional movie of structural changes in bacteriorhodopsin.  _Science_ , 354(6319):1552–1557, December 2016\.  ISSN 0036-8075, 1095-9203.  doi: 10.1126/science.aah3497.
  * Neutze et al. [2000] Richard Neutze, Remco Wouts, David van der Spoel, Edgar Weckert, and Janos Hajdu.  Potential for biomolecular imaging with femtosecond X-ray pulses.  _Nature_ , 406(6797):752–757, August 2000.  ISSN 0028-0836, 1476-4687.  doi: 10.1038/35021099.
  * Oktay et al. [2018] Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y. Hammerla, Bernhard Kainz, Ben Glocker, and Daniel Rueckert.  Attention U-Net: Learning Where to Look for the Pancreas, May 2018.
  * Pande et al. [2016] Kanupriya Pande, Christopher D. M. Hutchison, Gerrit Groenhof, Andy Aquila, Josef S. Robinson, Jason Tenboer, Shibom Basu, Sébastien Boutet, Daniel P. DePonte, Mengning Liang, Thomas A. White, Nadia A. Zatsepin, Oleksandr Yefanov, Dmitry Morozov, Dominik Oberthuer, Cornelius Gati, Ganesh Subramanian, Daniel James, Yun Zhao, Jake Koralek, Jennifer Brayshaw, Christopher Kupitz, Chelsie Conrad, Shatabdi Roy-Chowdhury, Jesse D. Coe, Markus Metz, Paulraj Lourdu Xavier, Thomas D. Grant, Jason E. Koglin, Gihan Ketawala, Raimund Fromme, Vukica Šrajer, Robert Henning, John C. H. Spence, Abbas Ourmazd, Peter Schwander, Uwe Weierstall, Matthias Frank, Petra Fromme, Anton Barty, Henry N. Chapman, Keith Moffat, Jasper J. van Thor, and Marius Schmidt.  Femtosecond structural dynamics drives the trans/cis isomerization in photoactive yellow protein.  _Science_ , 352(6286):725–729, May 2016.  ISSN 0036-8075, 1095-9203.  doi: 10.1126/science.aad5081.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  _arXiv:1505.04597 [cs]_ , May 2015.
  * Shin et al. [2018] Hocheol Shin, Seungnam Kim, and Chun Hong Yoon.  Data Analysis using Psocake at PAL-XFEL.  _Journal of the Korean Physical Society_ , 73(1):16–20, July 2018.  ISSN 0374-4884, 1976-8524.  doi: 10.3938/jkps.73.16.
  * Su et al. [2021] Zhen Su, Joshua Cantlon, Lacey Douthit, Max Wiedorn, Sébastien Boutet, Jan Kern, Chun Hong Yoon, and Daniel DePonte.  Serial crystallography using automated drop dispensing.  _Journal of Synchrotron Radiation_ , 28(5):1386–1392, September 2021.  ISSN 1600-5775.  doi: 10.1107/S1600577521006160.
  * Suga et al. [2017] Michihiro Suga, Fusamichi Akita, Michihiro Sugahara, Minoru Kubo, Yoshiki Nakajima, Takanori Nakane, Keitaro Yamashita, Yasufumi Umena, Makoto Nakabayashi, Takahiro Yamane, Takamitsu Nakano, Mamoru Suzuki, Tetsuya Masuda, Shigeyuki Inoue, Tetsunari Kimura, Takashi Nomura, Shinichiro Yonekura, Long-Jiang Yu, Tomohiro Sakamoto, Taiki Motomura, Jing-Hua Chen, Yuki Kato, Takumi Noguchi, Kensuke Tono, Yasumasa Joti, Takashi Kameshima, Takaki Hatsui, Eriko Nango, Rie Tanaka, Hisashi Naitow, Yoshinori Matsuura, Ayumi Yamashita, Masaki Yamamoto, Osamu Nureki, Makina Yabashi, Tetsuya Ishikawa, So Iwata, and Jian-Ren Shen.  Light-induced structural changes and the site of O=O bond formation in PSII caught by XFEL.  _Nature_ , 543(7643):131–135, March 2017.  ISSN 0028-0836, 1476-4687.  doi: 10.1038/nature21400.
  * Suga et al. [2020] Michihiro Suga, Atsuhiro Shimada, Fusamichi Akita, Jian-Ren Shen, Takehiko Tosha, and Hiroshi Sugimoto.  Time-resolved studies of metalloproteins using X-ray free electron laser radiation at SACLA.  _Biochimica et Biophysica Acta (BBA) - General Subjects_ , 1864(2):129466, February 2020.  ISSN 03044165.  doi: 10.1016/j.bbagen.2019.129466.
  * Sullivan et al. [2019] Brendan Sullivan, Rick Archibald, Jahaun Azadmanesh, Venu Gopal Vandavasi, Patricia S. Langan, Leighton Coates, Vickie Lynch, and Paul Langan.  BraggNet: Integrating Bragg peaks using neural networks.  _Journal of Applied Crystallography_ , 52(4):854–863, August 2019.  ISSN 1600-5767.  doi: 10.1107/S1600576719008665.
  * Tan et al. [2020] Mingxing Tan, Ruoming Pang, and Quoc V. Le.  EfficientDet: Scalable and Efficient Object Detection, July 2020.
  * Weaver [1985] James R. Weaver.  Centrosymmetric (Cross-Symmetric) Matrices, Their Basic Properties, Eigenvalues, and Eigenvectors.  _The American Mathematical Monthly_ , 92(10):711, December 1985.  ISSN 00029890.  doi: 10.2307/2323222.
  * White et al. [2012] Thomas A. White, Richard A. Kirian, Andrew V. Martin, Andrew Aquila, Karol Nass, Anton Barty, and Henry N. Chapman.  _CrystFEL_ : A software suite for snapshot serial crystallography.  _Journal of Applied Crystallography_ , 45(2):335–341, April 2012.  ISSN 0021-8898.  doi: 10.1107/S0021889812002312.
  * Wilkinson et al. [1988] C. Wilkinson, H. W. Khamis, R. F. D. Stansfield, and G. J. McIntyre.  Integration of single-crystal reflections using area multidetectors.  _Journal of Applied Crystallography_ , 21(5):471–478, October 1988.  ISSN 0021-8898.  doi: 10.1107/S0021889888005400.
  * Winter et al. [2018] Graeme Winter, David G. Waterman, James M. Parkhurst, Aaron S. Brewster, Richard J. Gildea, Markus Gerstel, Luis Fuentes-Montero, Melanie Vollmar, Tara Michels-Clark, Iris D. Young, Nicholas K. Sauter, and Gwyndaf Evans.  _DIALS_ : Implementation and evaluation of a new integration package.  _Acta Crystallographica Section D Structural Biology_ , 74(2):85–97, February 2018.  ISSN 2059-7983.  doi: 10.1107/S2059798317017235.
  * Yoon [2020] Chun Hong Yoon.  Psocake: GUI for Making Data Analysis a Piece of Cake.  In _Handbook on Big Data and Machine Learning in the Physical Sciences_ , pages 169–178. World Scientific Publishing Co Pte Ltd, May 2020.
  * Young et al. [2016] Iris D. Young, Mohamed Ibrahim, Ruchira Chatterjee, Sheraz Gul, Franklin D. Fuller, Sergey Koroidov, Aaron S. Brewster, Rosalie Tran, Roberto Alonso-Mori, Thomas Kroll, Tara Michels-Clark, Hartawan Laksmono, Raymond G. Sierra, Claudiu A. Stan, Rana Hussein, Miao Zhang, Lacey Douthit, Markus Kubin, Casper de Lichtenberg, Long Vo Pham, Håkan Nilsson, Mun Hon Cheah, Dmitriy Shevela, Claudio Saracini, Mackenzie A. Bean, Ina Seuffert, Dimosthenis Sokaras, Tsu-Chien Weng, Ernest Pastor, Clemens Weninger, Thomas Fransson, Louise Lassalle, Philipp Bräuer, Pierre Aller, Peter T. Docker, Babak Andi, Allen M. Orville, James M. Glownia, Silke Nelson, Marcin Sikorski, Diling Zhu, Mark S. Hunter, Thomas J. Lane, Andy Aquila, Jason E. Koglin, Joseph Robinson, Mengning Liang, Sébastien Boutet, Artem Y. Lyubimov, Monarin Uervirojnangkoorn, Nigel W. Moriarty, Dorothee Liebschner, Pavel V. Afonine, David G. Waterman, Gwyndaf Evans, Philippe Wernet, Holger Dobbek, William I. Weis, Axel T. Brunger, Petrus H. Zwart, Paul D. Adams, Athina Zouni, Johannes Messinger, Uwe Bergmann, Nicholas K. Sauter, Jan Kern, Vittal K. Yachandra, and Junko Yano.  Structure of photosystem II and substrate binding at room temperature.  _Nature_ , 540(7633):453–457, December 2016.  ISSN 0028-0836, 1476-4687.  doi: 10.1038/nature20161.


# Window to Wall Ratio Detection using SegFormer

Zoe De Simone
<EMAIL_ADDRESS>
MIT
Cambridge, MA
Sayandeep Biswas
<EMAIL_ADDRESS>
MIT
Cambridge, MA
Oscar Wu
<EMAIL_ADDRESS>
MIT
Cambridge, MA

###### Abstract

Window to Wall Ratios (WWR) are key to assessing the energy, daylight and
ventilation performance of buildings. Studies have shown that window area has
a large impact on building performance and simulation[1] [13] [8]. However,
data to set up these environmental models and simulations is typically not
available. Instead, a standard 40% WWR is typically assumed for all buildings.
This paper leverages existing computer vision window detection methods to
predict WWR of buildings from external street view images using semantic
segmentation, demonstrating the potential for adapting established computer
vision technique in architectural applications.

##  1 Introduction

Windows play a significant role in the performance and comfort of buildings,
transmitting daylight, heat, and wind. However, information about window
placement and window-to-wall ratio (WWR), which are necessary to set up
building performance simulation models (such as daylight, energy, and thermal
comfort models), is typically not publicly available, and instead, rules of
thumb of 40% WWR are applied to all buildings, regardless of location and
climate [11]. To improve the accuracy and data availability of building
characteristics, previous research has centered around façade geometry
extraction using image processing techniques or specialized equipment such as
3D laser scanning [3, 4, 9], which are time intensive and require dedicated
hardware.

Computer vision methods to process façade images to obtain WWR benefit from
scalability and availability of publicly labeled datasets of façade images
such as TSG-2002, TSG-6003, ZuBuDO4, CMP, and ECP [5, 6], which can be
augmented for increased accuracy in specific climates, through private
manually labeled images captured from Google Street View [12]. However, the
time-consuming nature of manually labeling building façades limits the number
of high-quality labeled data for WWR predictions. Fortunately, several
foundation models trained on much larger datasets such as ImageNet, ADE20K
[16], and more can be leveraged to address this data scarcity.

Image-based approaches revolve mainly around 3 types: 1) semantic parsing of
images with regular structure using Conditional random fields (CRF)[14], 2)
parsing and classification of image pixels based on their function such as
sky, wall, door, and roof, window [2], and 3) Convolutional Neural Network
(CNN) models using heatmaps to detect window corners and center points to then
form bounding boxes surrounding each window [6, 12]. There has been
significant progress in the field, in terms of model architectures and
datasets leading to better accuracies. However, there is still room for
further improvement as results from the recent paper by Tarkhan et al. have
only 72% of buildings detected within the 10% error range [12]. Furthermore,
certain subsets of building façades such as complete glass finishes (WWR
greater than 70-90%) have significantly worse predictions.

Another major obstacle spanning WWR approaches is related to the calculation
of the window and wall area. Generally, photos of buildings are taken at
street level and at an angle. In order to precisely calculate the WWR, and
accurately weigh each portion of the facade equally, images then have to be
re-projected such that they are frontal facing. Previous work by Tarkhan et
al. handled orthogonal transformations by computing edges, principal vanishing
points, computing homography, and warping the images to fronto-parallel view
with orthogonal axes. Lastly, images were cropped to the facade area of the
building, such that the image area is a proxy for the facade area [12]. This
methodology, however, presents its own challenges, only working on images that
are heavily skewed.

To address the aforementioned challenges of window and facade detection, the
efforts of this paper are three-fold: 1) we create an augmented semantic
segmentation dataset with window and facade labels, 2) we train several model
architectures, including Fully Convolutional Networks (FCNs) and SegFormer to
detect windows and facade areas, 3) we develop a corner point method to
reproject images of buildings to fronto-parallel views, which generalizes to
all skewed images.

While we did not develop novel ML architectures in this study, we demonstrate
the potential for adapting established computer vision techniques to address
research questions in architecture, an area where these methods have thus far
been underutilized. We note that the methods and results presented here are
part of an early stage research project and should be considered preliminary.
We share our opensource code in a github repository:
https://github.com/zoedesimone/wwr-semantic-segmentation.

##  2 Introduction

###  2.1 Dataset Preparation

The public datasets mentioned previously were compiled to create a combined
dataset of 2806 façade images. The raw dataset for each façade consists of a
JPEG and a JSON file. The JSON file contains the coordinates of the polygons
used to label the windows. This format is not amenable to the semantic
segmentation models which require a JPEG input and a PNG image as the label.
Therefore, the first step involved parsing the JSON files for each image to
generate the labels shown in Figure 1.

####  2.1.1 FCN

The next step for dataset preparation for the FCN architecture is to resize
and normalize the input images. The input size was chosen to be 520-by-520,
followed by a re-scaling to ensure pixel values were between 0 and 1. Next,
the RGB channels were normalized using the following parameters:
$[\mu_{R},\sigma_{R}]=[0.485,0.229],[\mu_{G},\sigma_{G}]=[0.456,0.224],\text{and
}[\mu_{B},\sigma_{B}]=[0.406,0.225]$. The parameters for resizing, scaling,
and normalizing were based on the dataset used to pre-train the FCN models.
The labels for FCN were obtained from the labeled PNGs as shown in Figure 1.
Simple logic was used to build a mask where 1 represented window pixels (red)
and 0 represented the background (black). Next, the obtained mask was resized
to also be 520-by-520.

####  2.1.2 SegFormer

The next step in data preparation for the SegFormer architecture is to
generate the labeled dataset containing pixel-by-pixel encodings from the
predefined ADE20K classes [16], including exterior windows and building
perimeters. The same normalized images as in the FCN were used.

The ADE20K dataset contained labels of building perimeters and windows,
however, only interior windows were labeled and exterior windows are
completely excluded. We obtain SegFormer model labels including exterior
windows, by using a pre-trained SegFormer with a ResNet50dilated + PPMdeepsup
encoder and decoder architecture to detect the building area in the images,
and overlaying window masks developed for the FCN architecture as shown in
Figure 2. Finally, we converted the output to greyscale masks encoding a
single number per pixel, representing the encoded semantic label in the ADE20K
semantic categories.

Lastly, we randomly crop, pad and resize both image and segmentation maps to
the same size 512-by-512.

Figure 1: Dataset preparation: JPEG of building façade (left) and PNG with
marked windows(right). Figure 2: Enhancing ADE20K labels by superimposing
using SegFormer predicted labels (Left) onto our labeled windows (Right).

###  2.2 Dataset Distributions

Analyzing our dataset, We can see that the mean WWR is 0.158 and mode WWR is
0.13, with nearly all of the images below 40% WWR, confirming our hypothesis
that existing 40% WWR rules of thumb are not error-free. From the distribution
of the percentage of window and wall pixels in each image as shown in Figure
3, we can see that photos are not cropped to the building’s façade. Instead,
photos contain building surroundings, and therefore the image area should not
be used as a proxy for the façade area.

For each image, we compute the WWR by calculating the façade area $A_{f}$ and
window area $A_{w}$:

$WWR=\frac{A_{w}}{A_{f}+A_{w}}$

The façade and window area are computed by counting the number of pink and
grey pixels in the image.

Figure 3: Window pixels % vs Building Pixel %

###  2.3 Model and Training

In this work, we showcase how these models can be trained with limited
resources, within the Google Colab computational constraints. We utilized a
randomly selected 10% subset consisting of 280 images to train our models.
This subset was randomly divided into a training, validation, and testing set
using an 80:10:10 split. After each epoch, the Intersection over Union (IoU)
metric is computed on the validation set. After completion of training, the
epoch with the highest IoU score on the validation set is chosen as the final
model. Further details specific to each architecture are discussed below.

$IoU=\frac{\text{Area of Overlap}}{\text{Area of Union}}$

####  2.3.1 FCN

The FCN models used in this work were imported from the torchvision library
and were developed by Long et al.. The models were pre-trained on a subset of
COCO, using only the 20 categories that are present in the Pascal VOC dataset.
For our application, we have a single-channel output for the windows compared
to the pre-trained model that has 21 channels including the background.
Therefore, the final convolution layer was modified to have the correct
dimensions. The outputs of these models were then passed through a sigmoid
layer to guarantee values between 0 and 1. The Adam optimizer with a binary
cross-entropy loss was used to train the model.

####  2.3.2 SegFormer

SegFormer consists of a hierarchical Transformer encoder and a lightweight
all-MLP decode head. The SegFormer models used in this work were imported from
HuggingFace and were developed by Xie et al.. The code was inspired from Rogge
developed using PyTorch by Rogge and Niels. We load the encoder of the model,
namely b0, b1, and b5 [15], with weights pre-trained on ImageNet-1k, and fine-
tune it together with the decoder head, which starts with randomly initialized
weights. The model was finetuned using the AdamW optimizer. The learning rate
of 0.00006 reported in the SegFormer paper was used. Cross entropy loss was
used for training and the window IoU was used to compute the validation and
test set performance.

##  3 Results

###  3.1 Semantic Segmentation

Table 1 shows a table summarizing our models’ performance. A preliminary study
was conducted without comprehensive hyperparameter optimization, primarily
using default settings. The FCN model with a ResNet-50 backbone achieved the
best performance, recording an IoU score of 0.66 on the test set with a 0.0005
learning rate.

Table 1: Summary of training details and performance of various FCN and
SegFormer(Seg) models varying model choice, pretraining, number of epochs, IoU
on testing Data and Learning Rate (LR).
Model | Pretrain | Epochs | IoU Test | LR
---|---|---|---|---
Seg-b0 | $\surd$ | 10 | 0.58 | 0.00006
Seg-b1 | $\surd$ | 10 | 0.62 | 0.00006
Seg-b5 | $\surd$ | 10 | 0.65 | 0.00006
Resnet-50 | $\surd$ | 10 | 0.61 | 0.005
Resnet-50 | $\surd$ | 100 | 0.66 | 0.0005
Resnet-50 | $\times$ | 100 | 0.60 | 0.005
Resnet-101 | $\surd$ | 100 | 0.59 | 0.005

Table 1 details the training and performance metrics of various models,
indicating the significance of model size for SegFormer, where a larger model
(b5) outperformed others. Conversely, for FCN models, a ResNet-50 backbone was
more effective than the larger ResNet-101. Learning rate emerged as a crucial
factor, with a lower rate significantly enhancing the FCN ResNet-50 model’s
IoU score by nearly 6%. The impact of pre-training was minimal, suggesting
that other hyperparameters and model choices might offer substantial
performance gains with dedicated optimization.

Figure 4 displays segmented images from our models, highlighting failure modes
such as misclassification due to glare on windows, doors mistaken for windows,
and issues with visible interiors and occlusions. These issues underscore the
potential benefits of larger datasets and data augmentation. Our approach used
a single-fold, which serves for a proof-of-concept; however, cross-fold
training and extrapolative data splits are recommended for improved results
and universal applicability.

Figure 4: Prediction generated by FCN (left) and SegFormer (right) models on
images in the test set

###  3.2 Robustness Assessment

We evaluated the robustness of our top FCN ResNet-50 model under different
times of day, zoom levels, and lighting conditions (Figure 5). The model shows
good resilience to lighting variations but struggles with varying zoom levels.
Specifically, it performs better on close-ups of building facades, reflecting
the training data’s composition of 80% building pixels (Figure 3). To enhance
zoom level robustness, incorporating more zoomed-out images or generating
synthetic images with artificial backgrounds into the training set could be
beneficial.

Figure 5: FCN ResNet-50 predictions on varying levels of zoom and lighting
conditions.

###  3.3 Perspective Correction

For accurate WWR calculations, correct estimation of window and wall areas is
essential. Street-level images often suffer from perspective distortion due to
oblique shooting angles, making parts of the facade closer to the camera
appear larger and distorting the true proportions. We address this by
reprojecting images to a fronto-parallel view using a four-point perspective
transformation in OpenCV, exemplified by adjusting an angled image of a
building (Figure 6). This method simulates a perpendicular view of the facade,
equalizing distances from the observer and correcting the proportions for more
precise WWR estimations. However, slight distortions remain, potentially
affecting WWR accuracy by compressing windows at the building’s top more than
those at the bottom.

Additionally, we investigated automatic building corner detection for the
transformation algorithm, exploring semantic segmentation and edge detection
techniques (Figure 6). While segmentation models are good for identifying the
bulk of objects, as highlighted by Zou et al. [17], they may not accurately
delineate precise boundaries for corner detection. Edge detection can provide
more exact corner locations but is sensitive to image noise. Therefore,
further exploration is warranted to investigate the potential benefits of
combining both semantic segmentation and edge detection methodologies to
enhance corner detection performance.

Figure 6: 1) Original image with building corners identified for the four
point perspective transform algorithm; 2)Semantic segmentation mask and 3)
Edge detection to identify building corners for perspective correction. 4)
Building image after perspective correction.

##  4 Future Work

Improving our models’ performance hinges on addressing training dataset
limitations, such as missing or inaccurately labeled windows. Future
strategies should include data cleaning for a refined dataset, employing
k-fold cross-validation to better evaluate model robustness across dataset
subsets, and multi-fidelity training, which leverages a tiered approach by
initially training on a broader, less precise dataset before fine-tuning on a
more accurately labeled subset.

##  5 Conclusion

Prior approaches to Window to Wall Ratio (WWR) detection have primarily
targeted window identification, using perspective correction and cropping to
estimate wall areas—a method prone to inaccuracies from including non-wall
elements. Our research introduces a semantic segmentation dataset with
detailed window labels and enhanced with ADE20K semantic categories, aiming to
accurately segment both windows and walls. We trained FCN and SegFormer models
on this dataset, achieving more precise window and wall detection. Despite
their effectiveness, challenges remain, including the misclassification of
reflective windows and doors as windows, underscoring the need for ongoing
refinement. We note that the methods and results presented here are part of an
early stage research project and should be considered preliminary.

##  6 Acknowledgements

We thank Nada Tarkhan for invaluable discussions on window-to-wall ratio and
urban extraction methods. This material is based upon work supported by the
National Science Foundation Graduate Research Fellowship under Grant No.
2141064. Any opinion, findings, and conclusions or recommendations expressed
in this material are those of the authors(s) and do not necessarily reflect
the views of the National Science Foundation.

## References

  * Bhandari and Tadepalli [2019] Nikhil Bhandari and Srinivas Tadepalli.  Influence of window-to-wall ratio on building energy load with daylight utilization for west facade in office building in hot and dry climate of india: A simulation-based approach.  In _In: Proceedings of the International Conference of Architectural Science Association_ , pages 615–624, 2019.
  * Cohen et al. [2014] Andrea Cohen, Alexander G Schwing, and Marc Pollefeys.  Efficient structured parsing of facades using dynamic programming.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3206–3213, 2014.
  * Früh and Zakhor [2003] Christian Früh and Avideh Zakhor.  Constructing 3d city models by merging aerial and ground views.  _IEEE computer graphics and applications_ , 23(6):52–61, 2003.
  * Früh and Zakhor [2004] Christian Früh and Avideh Zakhor.  An automated method for large-scale, ground-based city model acquisition.  _International Journal of Computer Vision_ , 60:5–24, 2004.
  * Isola et al. [2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1125–1134, 2017.
  * Li et al. [2020] Chuan-Kang Li, Hong-Xin Zhang, Jia-Xin Liu, Yuan-Qing Zhang, Shan-Chen Zou, and Yu-Tong Fang.  Window detection in facades using heatmap fusion.  _Journal of Computer Science and Technology_ , 35:900–912, 2020.
  * Long et al. [2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3431–3440, 2015.
  * Marino et al. [2017] C Marino, A Nucara, and M Pietrafesa.  Does window-to-wall ratio have a significant effect on the energy consumption of buildings? a parametric analysis in italian climate conditions.  _Journal of Building Engineering_ , 13:169–183, 2017.
  * Marston et al. [2015] Annie Marston, Eric Turner, Avideh Zakhor, Oliver Baumann, and Philip Haves.  _Testing rapmod: Can a portable scanner collect exisitng building data and create an energy model faster and more accurately than a human_.  eScholarship, University of California, 2015.
  * Rogge [2020] Niels Rogge.  Transformers Tutorials”, 2020.
  * Szcześniak et al. [2022] Jakub T Szcześniak, Yu Qian Ang, Samuel Letellier-Duchesne, and Christoph F Reinhart.  A method for using street view imagery to auto-extract window-to-wall ratios and its relevance for urban-level daylighting and energy simulations.  _Building and Environment_ , 207:108108, 2022.
  * Tarkhan et al. [2022] Nada Tarkhan, Samuel Letellier-Duchesne, and Christoph Reinhart.  Capturing façade diversity in urban settings using an automated window to wall ratio extraction and detection workflow.  In _2022 Annual Modeling and Simulation Conference (ANNSIM)_ , pages 706–717. IEEE, 2022.
  * Troup et al. [2019] Luke Troup, Robert Phillips, Matthew J Eckelman, and David Fannon.  Effect of window-to-wall ratio on measured energy consumption in us office buildings.  _Energy and Buildings_ , 203:109434, 2019.
  * Tyleček and Šára [2013] Radim Tyleček and Radim Šára.  Spatial pattern templates for recognition of objects with regular structure.  In _Pattern Recognition: 35th German Conference, GCPR 2013, Saarbrücken, Germany, September 3-6, 2013. Proceedings 35_ , pages 364–374. Springer, 2013.
  * Xie et al. [2021] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  _Advances in Neural Information Processing Systems_ , 34:12077–12090, 2021.
  * Zhou et al. [2017] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 633–641, 2017.
  * Zou et al. [2022] Xueyan Zou, Haotian Liu, and Yong Jae Lee.  End-to-End Instance Edge Detection, 2022.  arXiv:2204.02898 [cs] version: 1.