# Defect Analysis of 3D Printed Cylinder Object Using Transfer Learning
Approaches

Md Manjurul Ahsan
School of Industrial and Systems Engineering
University of Oklahoma
Norman, Oklahoma-73019
<EMAIL_ADDRESS>
Shivakumar Raman
School of Industrial and Systems Engineering
University of Oklahoma
Norman, Oklahoma-73019
<EMAIL_ADDRESS>
Zahed Siddique
School of Aerospace and Mechanical Engineering
University of Oklahoma
Norman, Oklahoma-73019
<EMAIL_ADDRESS>
###### Abstract

Additive manufacturing (AM) is gaining attention across various industries
like healthcare, aerospace, and automotive. However, identifying defects early
in the AM process can reduce production costs and improve productivity - a key
challenge. This study explored the effectiveness of machine learning (ML)
approaches, specifically transfer learning (TL) models, for defect detection
in 3D-printed cylinders. Images of cylinders were analyzed using models
including VGG16, VGG19, ResNet50, ResNet101, InceptionResNetV2, and
MobileNetV2. Performance was compared across two datasets using accuracy,
precision, recall, and F1-score metrics. In the first study, VGG16,
InceptionResNetV2, and MobileNetV2 achieved perfect scores. In contrast,
ResNet50 had the lowest performance, with an average F1-score of 0.32.
Similarly, in the second study, MobileNetV2 correctly classified all
instances, while ResNet50 struggled with more false positives and fewer true
positives, resulting in an F1-score of 0.75. Overall, the findings suggest
certain TL models like MobileNetV2 can deliver high accuracy for AM defect
classification, although performance varies across algorithms. The results
provide insights into model optimization and integration needs for reliable
automated defect analysis during 3D printing. By identifying the top-
performing TL techniques, this study aims to enhance AM product quality
through robust image-based monitoring and inspection.

_K_ eywords Additive manufacturing (AM) $\cdot$ Defect detection $\cdot$
Transfer learning (TL) $\cdot$ 3D printed cylinders $\cdot$ Machine learning
algorithms

##  1 Introduction

3D printing is a technology that has garnered considerable attention and
popularity in recent years. It involves creating three-dimensional objects
from a digital file by incrementally building up successive layers of material
until the desired shape is achieved. This method enables the production of
intricate and complex designs that would be challenging or impossible to
fabricate using conventional manufacturing techniques [1]. The range of
materials suitable for 3D printing extends from plastics to metals, ceramics,
and others. While initially employed primarily for prototyping and design
purposes, 3D printing has since permeated various industries, including
aerospace, automotive, medical, and fashion. The capacity to swiftly and
effortlessly generate customized and specific objects have endowed 3D printing
with significant utility in research and development, production processes,
and even artistic and design endeavors [2].

In recent years, the significance of 3D printing has been steadily growing,
primarily owing to its capacity to swiftly generate highly customized and
precise objects at a lower cost than traditional manufacturing methods [3].
This technological advancement introduces a novel dimension of creativity and
innovation by enabling the rapid prototyping and production of intricate
designs that were previously unattainable using conventional manufacturing
techniques. Furthermore, 3D printing possesses the potential to bring about a
revolutionary transformation in supply chain management and distribution by
obviating the necessity for transporting large and cumbersome components,
thereby promoting sustainability and environmental friendliness [4].
Additionally, it has found remarkable applications in developing prosthetics
and medical implants, leading to personalized solutions in the healthcare
domain [5]. Moreover, using 3D printing in space exploration has facilitated
the on-demand production of parts and tools, significantly reducing the
requirement for costly and time-consuming resupply missions [6]. As the
evolution of 3D printing persists, its transformative impact is anticipated to
extend across a broad spectrum of industries, reshaping conventional
approaches to product creation and consumption [7].

Despite the many advantages of 3D printing, the technology still faces some
challenges. One of the main challenges is the limited range of materials that
can be used in 3D printing, particularly for industrial applications where
specialized and high-performance materials are required [8]. Another challenge
is the limited print size of many 3D printers, making it difficult to produce
larger objects [9]. The printing speed of 3D printers is also a challenge, as
it can take hours or even days to print complex designs [10]. Additionally, 3D
printing requires a significant amount of energy and can generate a large
amount of waste material [11].

Moreover, the quality of the final product can be affected by the printing
process, including the formation of defects such as warping and layer
separation [12]. Despite these challenges, researchers and engineers are
working to overcome these limitations by developing new materials, improving
printing techniques, and optimizing the printing process. The continued
advancement of 3D printing technology is expected to address these challenges
and unlock new possibilities for this revolutionary technology.

Over the years, Machine Learning (ML) based approaches have shown promising
results in addressing many of the challenges faced by 3D printing. One key
application of ML in 3D printing is in predicting and preventing printing
defects. By training ML models on large datasets of 3D printing data, it is
possible to identify patterns and predict the likelihood of defects such as
warping, shrinkage, and layer separation. This can help optimize the printing
process and improve the quality of the final product [13]. Additionally, ML
can optimize printing parameters, such as speed, temperature, and layer
height, to reduce printing time and energy consumption while maintaining
quality. Furthermore, ML can be used to develop new materials optimized for 3D
printing, allowing for a broader range of materials to be used in 3D printing
[14]. ML can also be used to improve the accuracy of 3D scanning and modeling,
an essential step in designing and producing 3D printed objects. Overall, ML
has the potential to significantly enhance the capabilities of 3D printing and
overcome some of the challenges faced by this technology [15].

The current state of research on 3D printing defects is an active and growing
area of interest for researchers and engineers. Defects in 3D printing can
have significant implications for the quality and functionality of printed
parts, making it essential to identify and prevent them. Researchers have
investigated various defects, including warping, shrinkage, layer separation,
and porosity. Many studies have focused on identifying the root causes of
these defects, such as the printing parameters, material properties, and
environmental conditions [16]. Additionally, researchers have explored various
methods for defect detection and prevention, including ML process optimization
and real-time monitoring [16]. However, despite the significant progress made
in this field, many challenges remain to be addressed, such as the limited
range of materials that can be used in 3D printing and the difficulty of
detecting certain types of defects [17].

Several methods have been used in the past to identify 3D printing defects.
One common approach is visual inspection, where printed parts are visually
examined for defects such as warping, cracking, or uneven surfaces. However,
this method is subjective and time-consuming, making it impractical for large-
scale manufacturing [18] Another approach is using microscopy, such as
scanning electron microscopy (SEM) or optical microscopy, to examine the
microstructure of printed parts and identify defects such as porosity or
voids. This method provides high-resolution images that can reveal details not
visible to the naked eye, but it is often destructive and can be limited by
the size of the part being examined [19].

Other methods for defect identification include non-destructive testing
techniques such as X-ray computed tomography (CT) or ultrasound, which can
provide detailed internal images of printed parts to identify defects such as
voids or delamination. However, these methods can be expensive and time-
consuming, limiting their applicability for large-scale manufacturing [20].

Transfer Learning (TL) is an ML technique that involves using a pre-trained
model on one task to improve the performance of a related task. In TL, the
knowledge gained from the pre-trained model is transferred to a new model
being trained on a different but related task. This can significantly reduce
the training data required to achieve high performance on the new task [21].
In image classification, TL has been used to achieve state-of-the-art
performance on various image recognition tasks. Convolutional neural networks
(CNNs) are commonly used for image classification, and TL can be used to
improve their performance. Some popular TL methods include pre-trained CNN,
such as VGG, ResNet, or Inception, that has already been trained on a large
dataset, such as ImageNet, to extract features from images. The pre-trained
CNN is then fine-tuned on a new dataset for a different image recognition
task, such as recognizing specific objects in medical images [22]. However,
there are some limitations of the TL and CNN-based approaches in identifying
the defect from the images of 3d printed cylinders, such as:

  * •

The effectiveness of CNNs for 3D printing defect identification depends on the
quality of the training data, and the dataset must be carefully curated to
ensure that it is representative of the types of defects encountered in
practice [23].

  * •

3D printing defects can be challenging to detect, particularly internal
defects such as voids or porosity that may not be visible on the surface of
the printed part [24].

  * •

Transfer learning requires a pre-trained model relevant to the specific
application, and there may not be pre-trained models suitable for 3D printing
defect detection [25].

  * •

Interpreting the results of CNNs and transfer learning can be challenging,
particularly for complex models, and it may be difficult to determine which
features are most important for defect detection [26].

This study aims to solve the classification of 3D-printed cylinder objects as
either defective or non-defective using image classification techniques. The
dataset used in this research is balanced, imbalanced, and collected from the
Smart Materials and Intelligent Systems (SMIS) Laboratory, University of
Oklahoma. This imbalance can pose a challenge to classification accuracy and
can lead to biased results. Therefore, this research aims to accurately
classify the objects and ensure that the classification is interpretable and
explainable for researchers, practitioners, and users. This means that the
model should be able to provide insights into how it arrived at its
classification decisions and which features of the object are most important
for classification. By addressing these challenges, this research can
contribute to developing more reliable and interpretable image classification
techniques for 3D printing defect detection, which can have important
implications for improving the quality and efficiency of 3D printing
processes.

The technical contribution of this study can be summarized as follows:

##  2 Literature Review

Over the years, several Deep Learning (DL) researchers have proposed based
approaches to developing ML-based efficient 3d printing methods. For instance,
Saeed et al. (2019) propose an autonomous post-processor using CNN and DFF-NN
algorithms to detect defects and estimate thermogram depth. They use pre-
trained networks, fine-tuning, and pulsed thermography with CFRP samples. The
approach enables real-time, automatic post-processing without human
intervention, benefiting production lines. The study includes a comprehensive
review of CNN’s application to IR thermograms. Comparative results of
different CNN models and object detection frameworks are presented. The
proposed method demonstrates defect detection and depth estimation using
3D-printed CFRP samples. It could speed up inspections and integrate with
quality control systems in production facilities [25].

Jia et al. (2022) propose a fabric defect detection system using transfer
learning and an improved Faster R-CNN. The system addresses the limitations of
existing algorithms in detecting small target defects with low accuracy.
Transfer learning is employed by extracting pre-trained weights from the
Imagenet dataset. ResNet50 and ROI Align replace VGG16 and ROI pooling layer
for feature extraction. RPN and FPN are combined, using different IoU
thresholds to distinguish positive and negative samples. The softmax
classifier is utilized for image identification and predictions. The proposed
system demonstrates superior accuracy and convergence ability compared to
existing models [27].

Chen et al. (2023) propose a deep learning approach for detecting low-contrast
defects in 3D-printed ceramic curved surface parts. The method employs a
blurry inpainting network model and a multi-scale detail contrast enhancement
algorithm to improve image quality and enhance feature information. The
detection of defects is performed using the ECANet-Mobilenet SSD network
model. The proposed method achieves high accuracy in detecting crack and bulge
defects, with an average detection time of 0.78s. However, the study focuses
on only two types of defects and utilizes a specific dataset, which limits its
generalizability. Overall, the study contributes to the advancement of
intelligent defect detection in the advanced ceramic industry [28].

Li et al. (2022) present a comprehensive survey on Deep Transfer Learning
(DTL) techniques in Intelligent Fault Diagnosis (IFD). The study covers the
theoretical background of DTL and recent developments in IFD, offering
recommendations for selecting DTL algorithms in practical applications and
identifying future challenges. The survey aims to enhance researchers’
understanding of state-of-the-art techniques and facilitate the design of
effective solutions for IFD problems. However, empirical results or
experiments are not provided, and the survey is limited to DTL-based IFD
methods [29].

Ma et al. (2022) propose a transitive transfer learning CNN ensemble framework
for bearing surface defect detection and classification. The framework
addresses the limited dataset challenge in deep learning methods by
incorporating a transfer path and transfer method selection strategy to
enhance feature extraction. The proposed method achieves a high accuracy rate
of 97.51% and an average detection time of 155 ms per bearing, meeting
industrial online detection requirements. However, the study lacks comparisons
with other methods and focuses on only one type of defect. Overall, the
framework shows promise for improving the industry’s efficiency and accuracy
of bearing quality inspection [30].

Lu et al. (2023) propose a system for real-time defect detection and closed-
loop adjustment in additive manufacturing (AM) of carbon fiber reinforced
polymer (CFRP) composites. The system utilizes a deep learning model for
accurate real-time defect detection, classification, and evaluation. Defects
are controlled through closed-loop adjustment of process parameters. The
study’s focus on CFRP composites limits direct applicability to other
materials, and the system relies on accurate and accessible databases that may
not always be available. Nevertheless, the proposed system demonstrates the
potential of deep learning in real-time defect detection and closed-loop
adjustment for AM processes. It accurately identifies two types of CFRP
defects and quantifies their severity. The results highlight the effective
control of defects through process parameter adjustments, offering a promising
approach for enhancing the quality and efficiency of AM processes in CFRP
composites [31].

Westphal (2022) introduces a novel machine-learning approach for quality
assurance in Fused Deposition Modeling (FDM) additive manufacturing processes.
The study utilizes environmental sensor data (temperature, humidity, air
pressure, gas particles) and various state-of-the-art machine-learning
algorithms to classify 3D printing conditions. A new data preparation method
is developed, which sequences different sensor time series data. The
XceptionTime architecture is identified as the most effective and robust
against overfitting. A proof of concept comparing the ML analyses with 3D scan
examinations in quality assurance reveals the ML analyses to be a more
effective alternative in differentiating between various 3D printing
conditions. The study is limited to FDM processes and only considers
environmental sensor data for quality assurance. Overall, the findings present
a novel and effective approach that has the potential to enhance the trust and
acceptance of additive manufacturing processes in an industrial setting [32].

Caggiano et al. (2019) propose a machine learning approach using a bi-stream
deep convolutional neural network (DCNN) to identify defective conditions in
Selective Laser Melting (SLM) of metal powders. The DCNN analyzes in-process
images to recognize SLM defects, enabling adaptive process control and quality
assurance. The bi-stream DCNN effectively combines layer images to classify
defects caused by process non-conformities. Experimental evaluation
demonstrates accurate defect recognition and classification, with potential
applications in industrial quality assurance and adaptive process control for
SLM. Further validation is needed for different systems and materials [33].

Wang et al. (2020) present a comprehensive review of machine learning (ML)
applications in additive manufacturing (AM) across various domains. The review
explores ML’s potential in areas such as material design, topology design,
process parameter optimization, in-process monitoring, and product quality
assessment and control. Data security in AM is emphasized, and future research
directions are outlined. While current ML applications in AM research
primarily focus on processing-related processes, the review suggests a future
shift towards new materials and rational manufacturing [34].

Qi et al. (2019) conducted a review on the utilization of neural networks
(NNs) in additive manufacturing (AM) for complex pattern recognition and
regression analysis. NNs offer a promising approach to address the challenge
of constructing and solving physical models for the process-structure-
property-performance relationship in AM. The review encompasses diverse
application scenarios, including design, in-situ monitoring, and quality
evaluation. Challenges related to data collection and quality control are
identified, along with potential solutions. The authors emphasize the
significant potential of combining AM and NNs to achieve agile manufacturing
in the industry [35].

Kwon et al. (2020) applied a deep neural network to classify melt-pool images
in selective laser melting based on 6 laser power labels. The neural network
achieved satisfactory inference, even with images featuring blurred edges, by
reducing the number of nodes with increasing layers. The proposed neural
network demonstrated a low classification failure rate (under 1.1%) for 13,200
test images and outperformed simple calculations in monitoring melt-pool
images. The classification model could be utilized for non-destructive
inference of unexpected alterations in microstructures or the identification
of defective products. However, the study did not address potential
limitations of using deep neural networks for classification in selective
laser melting beyond its scope [36].

Aquil et al. (2020) discuss software defect prediction and the application of
machine learning techniques, including deep learning, ensembling, data mining,
clustering, and classification, for predicting potential bugs during software
maintenance. The study evaluates different predictor models using 13 software
defect datasets and concludes that ensembling techniques consistently achieve
high-accuracy predictions. The paper does not specify the types of software
defects being predicted, and it is unclear if the datasets used for evaluation
are representative of all software systems. The overall findings suggest that
ensembling techniques are reliable for software defect prediction, but further
research is needed to determine the optimal predictor model for different
software systems [37].

Dogan et al. (2021) conduct a comprehensive literature review on the
utilization of machine learning (ML) and data mining (DM) techniques in the
field of manufacturing. The review categorizes existing solutions into
scheduling, monitoring, quality, and failure topics. The paper highlights the
benefits of employing ML techniques in manufacturing, presents statistical
information on the field’s current state, and suggests potential areas for
future research. The knowledge discovery steps in the databases (KDD) process
for manufacturing applications are explained in detail. However, the paper
lacks a detailed analysis of the limitations and challenges associated with ML
implementation in manufacturing [38].

Majeed et al. (2021) propose the BD-SSAM framework, which combines big data
analytics, additive manufacturing, and sustainable smart manufacturing
technologies to aid additive manufacturing enterprises in decision-making
during the beginning of the product lifecycle. The framework is implemented in
the fabrication of AlSi10Mg alloy components using selective laser melting
(SLM) in additive manufacturing. The results indicate effective control over
energy consumption and product quality, promoting smart, sustainable
manufacturing, emission reduction, and cleaner production. However, the
study’s focus is limited to the beginning of the product lifecycle and
specific materials (AlSi10Mg alloy components with SLM). Further research is
needed to validate the framework across different materials and stages of the
product lifecycle. Overall, the paper presents a promising framework that
integrates big data analytics, additive manufacturing, and sustainable smart
manufacturing to support smart, sustainable manufacturing and cleaner
production, particularly during the beginning of the product lifecycle in the
additive manufacturing industry [39].

Azamfar et al. (2020) introduce a deep learning-based domain adaptation method
for fault diagnosis in semiconductor manufacturing. The method aims to
mitigate variations in manufacturing processes that can impact data-driven
approaches. It utilizes a deep neural network and the maximum mean discrepancy
metric to optimize high-level data representation. Experimental results on a
real-world semiconductor manufacturing dataset demonstrate the effectiveness
and generalization of the proposed method for quality inspection. However, the
paper does not discuss the limitations of the proposed method [40].

Qu et al. (2019) present an overview of smart manufacturing systems (SMSs),
which have gained significant attention from countries and manufacturing
enterprises. The paper covers the evolution, definition, objectives,
functional requirements, business requirements, technical requirements, and
components of SMSs. An autonomous SMSs model is also proposed driven by
dynamic demand and key performance indicators [41]. However, the paper lacks a
discussion on the challenges and limitations of implementing SMSs and
potential risks and concerns related to data privacy and security. The
findings provide a comprehensive understanding of SMSs and offer a reference
for the transformation of manufacturing enterprises to intelligent systems.

Based on the overall analysis, it can be observed that the majority of studies
have focused on employing machine learning (ML) based approaches to classify
or identify defects and faults in both the 3D printed objects and the 3D
printing process. While most of the studies have demonstrated the potential of
convolutional neural network (CNN) based approaches, only a limited number of
studies have explored transfer learning (TL) based approaches, which offer
computational efficiency and applicability in identifying defects across
various sizes and types of 3D printed objects. Furthermore, conducting an
analysis of TL-based approaches for defect analysis using images obtained from
3D-printed objects will enable us to determine the feasibility of using TL-
based approaches as viable solutions for developing ML-based defect analysis
on a large scale.

However, none of the existing studies offer sufficient contextual information
to assist future researchers and practitioners in comprehending whether there
is an ascertainable impact on their methodologies that can be elucidated.
Consequently, there arises a necessity for a comprehensive study capable of
delivering novel insights and facilitating an enhanced understanding of the
characteristics inherent in machine learning (ML)-based models and their
interpretability concerning the identification of defect analysis in additive
manufacturing.

##  3 Motivation

The motivation behind this study is to evaluate the performance of TL
approaches in the context of defect analysis in Additive Manufacturing (AM).
With the advancements in AM technology, detecting defects during the printing
process has become an essential issue to ensure the quality of the printed
product. Image classification using TL models has shown promising results in
various image-based applications. However, in the context of AM, challenges
include imbalanced data, variability in printing conditions, and complexity in
the defect patterns [32, 42]. Therefore, evaluating the performance of TL
models, such as VGG16, VGG19, ResNet50, ResNet101, and MobileNetV2, is
essential to determine their effectiveness in detecting defects in 3D printed
products.

Therefore, this study aims to evaluate and compare the performance of various
TL models on a dataset of 3D-printed product images to determine the best
approach for defect analysis in additive manufacturing. The study aims to
provide insights into the effectiveness of TL models and their potential
contribution to the field of additive manufacturing. By identifying the best
TL model for defect analysis, this research is expected to enhance the quality
of 3D-printed products and promote the advancement of the field.

The remainder of the paper is organized as follows: The methodology of the
experiment is described in detail in Section 4. The results of the experiment
are presented in Section 5. A brief discussion of the study’s findings is
provided in Section 6, and the overall conclusions of the research and
potential avenues for future research are summarized in Section 7.

##  4 Methodology

This section outlines the procedure for acquiring the 3D printing images, the
TL model used in this study, the experimental setup, and the evaluation
metrics employed to assess the model’s performance.

###  4.1 Data Collection

The dataset was generously provided by the Smart Materials and Intelligent
Systems (SMIS) Laboratory. It comprises a collection of 3D printed prototype
images of cylindrical objects, some of which contain defects while others are
non-defective. Figure 1 displays a selection of typical images of cylindrical
objects from the dataset, including both defective and non-defective examples.
Table 1 provides an overview of the allocation of data for training and
testing of each of the CNN models that were investigated. In both studies, six
different DL approaches were investigated: VGG16 [43], InceptionResNetV2 [44],
ResNet50 [45], MobileNetV2 [46], ResNet101 [47] and VGG19 [43].

Table 1: Allocation of data for training and testing of TL models used in this
study.

Dataset | Label | Train | Test | Total
---|---|---|---|---
Study One | Defect | 105 | 31 | 136
Non-defect | 112 | 24 | 136
Study Two | Defect | 736 | 211 | 947
Non-defect | 1677 | 393 | 2070

Figure 1: Representative samples of 3D-printed product images obtained from
SMIS, which have been utilized in this research.

###  4.2 Deep Learning Algorithms

####  4.2.1 VGG

The Visual Geometry Group (VGG), proposed by the Visual Geometry Group,
introduced two CNN models known as VGG16 and VGG19. These models are
characterized by their exceptionally deep architecture, consisting of 16 and
19 layers, respectively. To train these models, a dataset comprising one
million samples from the ImageNet dataset was utilized. Specifically, the
VGG16 model is designed to process input images with dimensions of 224 by 224
pixels.

During the forward pass, images are subjected to a series of Convolutional
layers featuring filters ranging from 64 to 512. The pooling layer is
implemented using a 2-by-2 filter with max-pooling. The Rectified Linear Unit
(ReLU) serves as the activation function with a stride size of 2.
Subsequently, the pooling layer output is transformed into a 1D vector within
the dense layer. Ultimately, the Softmax activation function is applied to the
output layer, facilitating the classification of samples into one of the 1000
available categories [43].

####  4.2.2 InceptionResNetV2

In 2017, a deep learning architecture known as InceptionResNetv2 was proposed
by Szegedy et al. This architecture combines the strengths of both the
Inception and ResNet architectures, resulting in synergistic advantages for
image classification tasks. With a total of 164 layers, IRv2 employs 1x1, 3x3,
and 5x5 convolutions, in addition to max pooling, to extract features from
input images. By incorporating residual connections, the architecture enables
the training of deeper networks while effectively addressing the issue of
vanishing gradients. Furthermore, batch normalization and the Rectified Linear
Unit (ReLU) activation function are integrated into the architecture, thereby
enhancing the speed and accuracy of the training process.

Significantly, IRv2 achieved remarkable performance on the ImageNet Large
Scale Visual Recognition Challenge dataset in 2016, attaining a top-5 accuracy
of 95.3% [44].

####  4.2.3 ResNet

ResNet, short for Residual Network, is a deep neural network architecture
introduced by researchers at Microsoft in 2015 to tackle the challenge of
vanishing gradients in extremely deep neural networks. ResNet50, a variant of
ResNet, implements the model with 48 convolution layers, incorporating one
layer each of max pooling, average pooling, and regular pooling. The
architecture of ResNet50 begins with a 7x7 convolutional layer consisting of
64 kernels, each with a stride size of 2. This is followed by a max-pooling
layer with a stride size of 2. The network then employs nine convolutional
layers with kernel filters of sizes 64, 64, and 256, respectively. The
subsequent nine levels of the network comprise kernel filters of sizes 512,
512, and 2048.

An FC (fully connected) layer with 1000 nodes, along with an average pooling
layer, is added, and the output layer employs the Softmax function as its
activation function. ResNet101 shares a similar architecture to ResNet50, but
it includes an additional three-block layer within the fourth block,
consisting of 256, 256, and 1024 filters [47, 48, 49].

####  4.2.4 MobileNetV2

MobileNetV2 is an advanced module that incorporates inverted residuals and a
linear bottleneck. This efficient model is specifically designed for
utilization in mobile phone applications, where it can accommodate low-
dimensional inputs, minimize computational operations, and consume less memory
while maintaining a high level of accuracy. MobileNetV2 leverages depthwise
separable convolution, which divides the convolution operation into two
distinctive layers.

The initial layer is the depth-wise convolution, which excels in efficiency by
performing a single filtering operation. The subsequent layer captures
additional features through the linear computation of the input. By employing
MobileNetV2, the computational cost of standard model layers has been
significantly reduced by a factor of $k^{2}$. In many cases, MobileNetV2 saves
approximately 8 to 9 times the computational resources required for a 3x3
depth-wise separable convolution compared to traditional models [46].

###  4.3 Using Pre-Trained Convet

A pre-trained network refers to a network that has been previously trained on
a larger dataset, enabling it to acquire a comprehensive hierarchy for feature
extraction. This attribute makes pre-trained networks particularly effective
when dealing with smaller datasets. An exemplary instance of such a network is
the VGG16 architecture, which was developed by Simonyan and Zisserman in 2014
[43]. Figure 1 showcases a sample architecture illustrating the procedure of
the pre-trained model. It is worth noting that all the models employed in this
study are readily available as pre-packaged options within the Keras framework
[50].

The fine-tuning process on the VGG16 network is carried out according to the
following steps:

  1. 1.

Initially, the models are initialized with a pre-trained network that lacks a
Fully Connected (FC) layer.

  2. 2.

Subsequently, a new connected layer is introduced, incorporating a pooling
layer and applying the "softmax" activation function. This newly created layer
is appended on top of the VGG16 model.

  3. 3.

Finally, during the training phase, the weights of the convolutional layers
are frozen to ensure that only the FC layer undergoes training for the
duration of the experiment.

The identical procedure was employed for all other deep learning techniques.
For each CNN architecture utilized in the experiment, the model was further
modified according to the following construction:

$AveragePooling2D(Poolsize=(4,4))\rightarrow Flatten\rightarrow
Dense\rightarrow Dropout(0.5)\rightarrow Dense(Activation=``softmax")$.

Figure 2: VGG16 architecture used during this experiment.

As is well known, most pre-trained models consist of multiple layers
associated with various parameters, such as the number of filters, kernel
size, hidden layers, and neurons [51]. However, manually fine-tuning these
parameters can be a time-consuming process [52, 53]. Taking this into
consideration, three parameters of the deep learning model are optimized in
this study: batch size (which determines the number of samples processed
before updating internal model parameters), epochs (indicating the number of
times the learning algorithm iterates over the entire dataset), and learning
rate (a hyperparameter controlling the magnitude of model adjustments when
updating model weights to calculate error) [54].

Inspired by prior works [55, 56], a grid search method commonly employed for
parameter tuning was applied in this study [57]. Initially, the following
parameters were randomly selected:

Batch size = $[4,5,8,10]$

Number of epochs = $[10,20,30,40]$

Learning rate = $[.001,.01,0.1]$

For Study One, better results were achieved using the grid search method with
the following parameters:

Batch size = $8$

Number of epochs = $30$

Learning rate = $.001$

Similarly, for Study Two, the best results were achieved with:

Batch size = $50$

Number of epochs = $50$

Learning rate = $.001$

The optimization algorithm employed for all models was Adam, an adaptive
learning rate optimization algorithm recognized for its robust performance in
binary image classification[58, 59]. Adhering to the customary procedure in
data mining techniques, 80% of the accessible data was designated for
training, while the remaining 20% was set aside for testing purposes [60, 61,
62]. Each study was replicated twice, and the ultimate outcome was established
by calculating the average of the outcomes from the two independent
experiments, aligning with the guidance put forth by Zhang et al. (2020) [63].

###  4.4 Experimental Setup

The entire experiment was conducted using an office-grade laptop equipped with
standard specifications, which encompassed a Windows 10 operating system, an
Intel Core i7-7500U processor, and 16 GB of RAM. The assessment of performance
was conducted utilizing diverse metrics, encompassing model accuracy,
precision, recall, and F1-score [64].

$\textrm{Accuracy}=\frac{t_{p}+t_{n}}{t_{p}+t_{n}+f_{p}+f_{n}}$ (1)

$\textrm{Precision}=\frac{t_{p}}{t_{p}+f_{p}}$ (2)

$\textrm{Recall}=\frac{t_{p}}{t_{n}+f_{p}}$ (3)

$\textrm{F1-score}=2\times\frac{\textrm{Precision}\times\textrm{
Recall}}{\textrm{Precision+Recall}}$ (4)

Where,

  * •

True Positive ($t_{p}$)= Defect cylinder classified as Defect

  * •

False Positive ($f_{p}$)= Normal cylinder classified as Defect

  * •

True Negative ($t_{n}$)= Normal cylinder classified as Normal cylinder

  * •

False Negative ($f_{n}$)= Defect cylinder classified as Normal cylinder.

##  5 Results

###  5.1 Study One

Table 2 presents the performance of six different DL algorithms, namely VGG16,
InceptionResNetV2, ResNet50, ResNet101, MobileNetV2, and VGG19, on the train
set. The highest performance in terms of all four metrics was achieved by
VGG16 and MobileNetV2, both with perfect scores. InceptionResNetV2 also
performed well, averaging 99% across all metrics. On the other hand, ResNet50
had the lowest performance among the models, with an average F1-score of 0.34.

Table 2: Computational results of the various Transfer Learning models used in
Study One on the train set.

Algorithm | Accuracy | Precision | Recall | F1-score
---|---|---|---|---
VGG16 | 1 | 1 | 1 | 1
InceptionResNetV2 | .99 | .99 | .99 | .99
ResNet50 | 0.51 | 0.25 | 0.5 | 0.34
ResNet101 | 0.86 | 0.87 | 0.85 | 0.85
MobileNetV2 | 1 | 1 | 1 | 1
VGG19 | 0.98 | 0.98 | 0.98 | 0.98

The confusion matrix on the train data, as shown in Figure 3 for Study One,
indicates the true positive, false positive, false negative, and true negative
values for each algorithm. The results show that VGG16, InceptionResNetV2, and
MobileNetV2 had perfect classification accuracy, with 109, 110, and 110 true
positive values, respectively. ResNet50 had the highest number of false
negative values, with 106, and a relatively low number of true positives, with
only 110. ResNet101 had 27 false negative values but a higher number of true
positives than ResNet50, with 106. Finally, VGG19 had the same number of false
negatives as ResNet50, with 0, but a higher number of false positives, with 4.
Overall, the confusion matrix on the train data shows that VGG16,
InceptionResNetV2, and MobileNetV2 performed the best, while ResNet50 and
VGG19 had lower accuracy due to higher false negative and false positive
values, respectively.

Figure 3: Confusion matrices of the different Transfer Learning models used
during Study One on the train set.

Table 3 presents the results of Study One model performance on the test set.
The table shows that VGG16, InceptionResNetV2, and MobileNetV2 achieved
perfect scores of 1 for all evaluation metrics, while ResNet50 had the lowest
accuracy, precision, and F1-score. ResNet101 achieved high performance with an
accuracy of 0.85 and an F1-score of 0.85, while VGG19 had a high accuracy of
0.98 and an F1-score of 0.98. These results demonstrate that different TL
algorithms perform differently on the same dataset.

Table 3: Computational results of the various Transfer Learning models used in
Study One on the test set.

Algorithm | Accuracy | Precision | Recall | F1-score
---|---|---|---|---
VGG16 | 1 | 1 | 1 | 1
InceptionResNetV2 | 1 | 1 | 1 | 1
ResNet50 | 0.47 | 0.24 | 0.50 | 0.32
ResNet101 | 0.85 | 0.87 | 0.86 | 0.85
MobileNetV2 | 1 | 1 | 1 | 1
VGG19 | 0.98 | 0.98 | 0.98 | 0.98

Figure 4 shows the confusion matrix on the test data for Study One. The
algorithms VGG16, InceptionResNetV2, and MobileNetV2 achieved perfect accuracy
with true positives (TP) of 26 and no false positives (FP) or false negatives
(FN), while ResNet50 had a TP of 26 but a high number of false negatives (FN)
at 29. ResNet101 had a slightly lower performance than VGG16,
InceptionResNetV2, and MobileNetV2, with a TP of 25 and 1 FP, and 7 FN. VGG19
had a TP of 25 and 1 FP but no FN.

Figure 4: Confusion matrices of the different Transfer Learning models used
during Study One on the test set.

Figure 5 demonstrates the top two DL models’ AUC scores on the test set.

Figure 5: AUC score of two well-performed TL models (a) VGG16 and (b)
InceptionResNetV2.

Figure 6 compares the In Figure 6, the performance of VGG16 and ResNet50
models during the training phase is compared. The results show that VGG16
continuously improved train accuracy and reduced train loss until epoch 20. At
the same time, ResNet50 stopped showing any further improvements in
performance after only ten epochs.

Figure 6: Training and validation performance throughout the training of (a)
VGG16 and (b) ResNet50 during Study One. TA – training accuracy; VA –
validation accuracy; TL – training loss; VS – validation loss.

###  5.2 Study Two

Table 4 presents the results of Study Two model performance on the train set.
The best-performing models on the train set in Study Two are VGG16,
InceptionResNetV2, MobileNetV2, and VGG19, which achieved perfect scores on
all evaluation metrics. The worst-performing model is ResNet50, which achieved
the lowest accuracy, precision, recall, and F1-score scores. ResNet101 showed
overall good performance, with a score of 0.85 for accuracy and F1-score and
precision and recall scores of 0.87 and 0.78, respectively.

Table 4: Computational results of the various Transfer Learning models used in
Study Two on the train set.

Algorithm | Accuracy | Precision | Recall | F1-score
---|---|---|---|---
VGG16 | 1 | 1 | 1 | 1
InceptionResNetV2 | 1 | 1 | 1 | 1
ResNet50 | .8 | .89 | .69 | .71
ResNet101 | 0.85 | 0.87 | 0.78 | 0.8
MobileNetV2 | 1 | 1 | 1 | 1
VGG19 | 1 | 1 | .99 | 1

Figure 7 shows the confusion matrix on the train data for Study Two. Among the
algorithms, InceptionResNetV2 and MobileNetV2 had the highest number of TP,
766 each, while ResNet50 had the lowest, with only 288 TP. InceptionResNetV2
had the fewest FP, with 0, while ResNet50 had the most, with 478. ResNet101
had the highest number of FN, with 45, while VGG16, InceptionResNetV2,
MobileNetV2, and VGG19 had none.

Overall, InceptionResNetV2 and MobileNetV2 achieved the best performance, with
high numbers of TP and TN and no FN or FP. ResNet50 had the lowest
performance, with a low number of TP and a high number of FP.

Figure 7: Confusion matrices of the different Transfer Learning models used
during Study Two on the train set.

Table 5 presents the results of Study Two model performance on the test set.
The MobileNetV2, VGG16, InceptionResNetV2, and VGG19 models achieved the best
results, with perfect accuracy, precision, recall, and F1-score. The ResNet50
and ResNet101 models showed lower performance in terms of accuracy, precision,
recall, and F1-score, with the ResNet50 performing the worst among all models.

Table 5: Computational results of the various Transfer Learning models used in
Study Two on the test set.

Algorithm | Accuracy | Precision | Recall | F1-score
---|---|---|---|---
VGG16 | 1 | 1 | 1 | 1
InceptionResNetV2 | 1 | 1 | 1 | 1
ResNet50 | 0.83 | 0.9 | 0.72 | 0.75
ResNet101 | 0.86 | 0.88 | 0.79 | 0.81
MobileNetV2 | 1 | 1 | 1 | 1
VGG19 | 1 | 1 | .99 | 1

Figure 8 shows the confusion matrix on the test data for Study Two. The figure
shows that MobileNetV2 had the highest TP and TN, equaling 181, indicating
that it correctly classified all positive and negative instances. VGG16 and
InceptionResNetV2 also performed well, with TP and TN equaling 180 for each
algorithm. ResNet50 had the highest number of FP, with 101, and the lowest
number of TP, with only 80. ResNet101 had a lower number of FP (72) but a
higher number of FN (12) than other models. VGG19 had the second-highest
number of TN, equaling 423, but had 2 FP. Overall, MobileNetV2 showed the best
performance, while ResNet50 and ResNet101 had relatively lower performance
than other algorithms.

Figure 8: Confusion matrices of the different Transfer Learning models used
during Study Two on the test set.

Figure 9 illustrates the AUC score of VGG16 and ResNet50 models during Study
Two. From the figure, it can be observed that the performance of VGG16 is
nearly perfect, whereas the performance of ResNet15 demonstrates significantly
poor performance.

Figure 9: AUC score of (a) VGG16 and (b) ResNet50.

Figure 10 compares the performance of VGG16 and ResNet50 models during the
training phase. The figure indicates that VGG16 constantly improved train
accuracy and reduced train loss until epoch 20, whereas ResNet50 ceased
showing further performance improvements after only ten epochs.

Figure 10: Training and validation performance throughout the training of (a)
VGG16 and (b) ResNet50 during Study Two. TA – training accuracy; VA –
validation accuracy; TL – training loss; VS – validation loss.

##  6 Discussions

The current study investigated how deep learning approaches impact imbalanced
data analysis. Six deep learning algorithms - VGG16, InceptionResNetV2,
ResNet50, ResNet101, MobileNetV2, and VGG19 - were evaluated on an imbalanced
data classification task. The results showed that VGG16 and MobileNetV2
achieved perfect scores on all four evaluation metrics, while
InceptionResNetV2 performed well, averaging 99% across all metrics. In
contrast, ResNet50 had the lowest performance among the models, with an
average F1-score of 0.34.

The confusion matrix analysis further supported these findings, showing that
VGG16, InceptionResNetV2, and MobileNetV2 had perfect classification accuracy
on the train data. In contrast, ResNet50 and VGG19 had lower accuracy due to
higher false negative and false positive values. Moreover, on the test data
for Study Two, MobileNetV2 demonstrated the best performance, correctly
classifying all positive and negative instances. ResNet50 had the highest
number of false positives and the lowest number of true positives, while
ResNet101 had a higher number of false negatives.

Overall, the findings suggest that deep learning algorithms can perform well
on imbalanced data classification tasks, but their performance may vary
depending on the specific algorithm and the imbalance level. While some
algorithms, such as MobileNetV2, can achieve high classification accuracy,
others, such as ResNet50, may struggle with such tasks due to higher false
positive and false negative rates. The limitations of this study include the
use of a specific dataset and an imbalanced data classification task, which
may not generalize to other datasets or tasks. Future research could
investigate the impact of various DL algorithms on different imbalanced data
classification tasks and explore ways to optimize their performance.

###  6.1 Models Prediction

Figure 11 illustrates the model’s failure to detect the defect in the image of
the 3D-printed cylinder. Based on the results presented in Figure 11, it is
evident that all three models failed to locate the defect in the 3D-printed
cylinder accurately. Although VGG16 and MobileNetV2 produced bounding boxes
that were relatively close to the defect, they were still unable to localize
it accurately. These models show little promise and could be used for further
investigation. However, similar poor performance was also observed for
ResNet101, InceptionResNetV2, and VGG19, indicating that these models may not
be well-suited for defect localization in 3D printed objects. These
differences in performance may be attributed to variations in the models’
architecture and how they handle spatial features. Nonetheless, it is
essential to note that despite these differences, all models failed to locate
the defect accurately, highlighting the challenges associated with defect
localization in 3D printed objects. These findings emphasize the need for
further investigation into the factors contributing to model failures and
suggest that optimizing model architecture and training strategies may improve
localization accuracy and develop more effective solutions for defect
detection and classification in additive manufacturing processes.

Figure 11: Failure to accurately locate the defect in the 3D printed cylinder
of (a) VGG16, (b) ResNet50, and (d) MobileNetV2.

##  7 Conclusion

The present study investigated the impact of DL-based approaches on imbalanced
data analysis using six popular TL-based DL methods, including VGG16,
InceptionResNetV2, ResNet50, MobileNetV2, ResNet101, and VGG19. The
experiments were conducted on both small balanced and large imbalanced
datasets containing images of the 3D-printed cylinder.

Based on the experiment, it can be answered that even though the performance
of TL-based methods on balanced and imbalanced data is very high, the existing
popular TL-based approaches failed to locate the potential area, such as
defect regions in cylinder images, which indicates that imbalanced data may
affect TL-based models’ performance. Additionally, VGG16 and MobileNetV2
showed promising results and are a good choice for further investigation.

However, additional experiments with larger and different datasets are
necessary to validate the findings. In future work, it is planned to conduct
experiments with various datasets and propose advanced TL-based approaches
that may help overcome the current challenges observed during this experiment.
Ultimately, the study highlights the need for further investigation into the
factors contributing to model failures and developing more effective solutions
for defect detection and classification in additive manufacturing processes.

## References

  * [1] OG Bhusnure, SV Gholve, BK Sugave, RC Dongre, SA Gore, and PS Giram.  3d printing & pharmaceutical manufacturing: opportunities and challenges.  Int. J. Bioassays, 5:4723, 2016.
  * [2] Ramin Javan and Merissa N Zeman.  A prototype educational model for hepatobiliary interventions: unveiling the role of graphic designers in medical 3d printing.  Journal of digital imaging, 31:133–143, 2018.
  * [3] Zhongfei Zou, Yuewei Chen, Sen Yuan, Nan Luo, Jiachun Li, and Yong He.  3d printing of liquid metals: Recent advancements and challenges.  Advanced Functional Materials, 33(10):2213312, 2023.
  * [4] Mahyar Khorasani, Jennifer Loy, Amir Hossein Ghasemi, Elmira Sharabian, Martin Leary, Hamed Mirafzal, Peter Cochrane, Bernard Rolfe, and Ian Gibson.  A review of industry 4.0 and additive manufacturing synergy.  Rapid Prototyping Journal, 28(8):1462–1475, 2022.
  * [5] Maziar Ramezani and Zaidi Mohd Ripin.  4d printing in biomedical engineering: Advancements, challenges, and future directions.  Journal of functional biomaterials, 14(7):347, 2023.
  * [6] Aamer Nazir, Ozkan Gokcekaya, Kazi Md Masum Billah, Onur Ertugrul, Jingchao Jiang, Jiayu Sun, and Sajjad Hussain.  Multi-material additive manufacturing: A systematic review of design, properties, applications, challenges, and 3d printing of materials and cellular metamaterials.  Materials & Design, page 111661, 2023.
  * [7] Shabnam Siddiqui, Srinath Surananai, Krishnamurthy Sainath, Mohammed Zubair Khan, Raghu Raja Pandiyan Kuppusamy, and Yalachigere Kempaiah Suneetha.  Emerging trends in the development and application of 3d printed nanocomposite polymers for sustainable environmental solutions.  European Polymer Journal, page 112298, 2023.
  * [8] Soyeon Park, Wan Shou, Liane Makatura, Wojciech Matusik, and Kun Kelvin Fu.  3d printing of polymer composites: Materials, processes, and applications.  Matter, 5(1):43–76, 2022.
  * [9] N Shahrubudin, P Koshy, J Alipal, MHA Kadir, and TC Lee.  Challenges of 3d printing technology for manufacturing biomedical products: A case study of malaysian manufacturing firms.  Heliyon, 6(4), 2020.
  * [10] Sajjad Rahmani Dabbagh, Misagh Rezapour Sarabi, Mehmet Tugrul Birtek, Siamak Seyfi, Metin Sitti, and Savas Tasoglu.  3d-printed microrobots from design to translation.  Nature Communications, 13(1):5875, 2022.
  * [11] Antreas Kantaros, Nikolaos Laskaris, Dimitrios Piromalis, and Theodore Ganetsos.  Manufacturing zero-waste covid-19 personal protection equipment: A case study of utilizing 3d printing while employing waste material recycling.  Circular Economy and Sustainability, 1:851–869, 2021.
  * [12] Mohammadreza Lalegani Dezaki, Mohd Khairol Anuar Mohd Ariffin, and Saghi Hatami.  An overview of fused deposition modelling (fdm): Research, development and process optimisation.  Rapid Prototyping Journal, 27(3):562–582, 2021.
  * [13] Muhammad Ayub Ansari, Andrew Crampton, and Simon Parkinson.  A layer-wise surface deformation defect detection by convolutional neural networks in laser powder-bed fusion images.  Materials, 15(20):7166, 2022.
  * [14] Carlos Belei, Jana Joeressen, and Sergio T Amancio-Filho.  Fused-filament fabrication of short carbon fiber-reinforced polyamide: parameter optimization for improved performance under uniaxial tensile loading.  Polymers, 14(7):1292, 2022.
  * [15] Abid Haleem, Pawan Gupta, Shashi Bahl, Mohd Javaid, and Lalit Kumar.  3d scanning of a carburetor body using comet 3d scanner supported by colin 3d software: Issues and solutions.  Materials Today: Proceedings, 39:331–337, 2021.
  * [16] Valentina Mazzanti, Lorenzo Malagutti, and Francesco Mollica.  Fdm 3d printing of polymers containing natural fillers: A review of their mechanical properties.  Polymers, 11(7):1094, 2019.
  * [17] Konstantinos Paraskevoudis, Panagiotis Karayannis, and Elias P Koumoulos.  Real-time 3d printing remote defect detection (stringing) with computer vision and artificial intelligence.  Processes, 8(11):1464, 2020.
  * [18] Shuhao Wang, Jinsheng Ning, Lida Zhu, Zhichao Yang, Wentao Yan, Yichao Dun, Pengsheng Xue, Peihua Xu, Susmita Bose, and Amit Bandyopadhyay.  Role of porosity defects in metal 3d printing: Formation mechanisms, impacts on properties and mitigation strategies.  Materials Today, 2022.
  * [19] Gabriel Demeneghi, Baxter Barnes, Paul Gradl, Jason R Mayeur, and Kavan Hazeli.  Size effects on microstructure and mechanical properties of additively manufactured copper–chromium–niobium alloy.  Materials Science and Engineering: A, 820:141511, 2021.
  * [20] Wenjuan Sun, Daniel R Symes, Ceri M Brenner, Michael Böhnel, Stephen Brown, Mark N Mavrogordato, Ian Sinclair, and Michael Salamon.  Review of high energy x-ray computed tomography for non-destructive dimensional metrology of large metallic advanced manufactured components.  Reports on Progress in Physics, 85(1):016102, 2022.
  * [21] Md Manjurul Ahsan, Kishor Datta Gupta, Mohammad Maminur Islam, Sajib Sen, Md Lutfar Rahman, and Mohammad Shakhawat Hossain.  Study of different deep learning approach with explainable ai for screening patients with covid-19 symptoms: Using ct scan and chest x-ray image dataset.  arXiv preprint arXiv:2007.12525, 2020.
  * [22] Md Manjurul Ahsan, Md Tanvir Ahad, Farzana Akter Soma, Shuva Paul, Ananna Chowdhury, Shahana Akter Luna, Munshi Md Shafwat Yazdan, Akhlaqur Rahman, Zahed Siddique, and Pedro Huebner.  Detecting sars-cov-2 from chest x-ray using artificial intelligence.  Ieee Access, 9:35501–35513, 2021.
  * [23] Hao Wen, Chang Huang, and Shengmin Guo.  The application of convolutional neural networks (cnns) to recognize defects in 3d-printed parts.  Materials, 14(10):2575, 2021.
  * [24] Nayan Dhakal, Xiaolong Wang, Cayetano Espejo, Ardian Morina, and Nazanin Emami.  Impact of processing defects on microstructure, surface quality, and tribological performance in 3d printed polymers.  journal of materials research and technology, 23:1252–1272, 2023\.
  * [25] Numan Saeed, Nelson King, Zafar Said, and Mohammed A Omar.  Automatic defects detection in cfrp thermograms, using convolutional neural networks and transfer learning.  Infrared Physics & Technology, 102:103048, 2019.
  * [26] Jiahuan Liu, Fei Guo, Huang Gao, Maoyuan Li, Yun Zhang, and Huamin Zhou.  Defect detection of injection molding products on small datasets using transfer learning.  Journal of manufacturing processes, 70:400–413, 2021.
  * [27] Zhao Jia, Zhou Shi, Zheng Quan, and Mei Shunqi.  Fabric defect detection based on transfer learning and improved faster r-cnn.  Journal of Engineered Fibers and Fabrics, 17:15589250221086647, 2022\.
  * [28] Wei Chen, Bin Zou, Chuanzhen Huang, Jinzhao Yang, Lei Li, Jikai Liu, and Xinfeng Wang.  The defect detection of 3d-printed ceramic curved surface parts with low contrast based on deep learning.  Ceramics International, 49(2):2881–2893, 2023.
  * [29] Sheng Li, JC Ji, Yadong Xu, Xiuquan Sun, Ke Feng, Beibei Sun, Yulin Wang, Fengshou Gu, Ke Zhang, and Qing Ni.  Ifd-mdcn: Multibranch denoising convolutional networks with improved flow direction strategy for intelligent fault diagnosis of rolling bearings under noisy conditions.  Reliability Engineering & System Safety, 237:109387, 2023.
  * [30] Jiajun Ma, Maolin Liu, Songyu Hu, Jianzhong Fu, Gui Chen, and Aixi Yang.  A novel cnn ensemble framework for bearing surface defects classification based on transfer learning.  Measurement Science and Technology, 34(2):025902, 2022.
  * [31] Lu Lu, Jie Hou, Shangqin Yuan, Xiling Yao, Yamin Li, and Jihong Zhu.  Deep learning-assisted real-time defect detection and closed-loop adjustment for additive manufacturing of continuous fiber-reinforced polymer composites.  Robotics and Computer-Integrated Manufacturing, 79:102431, 2023\.
  * [32] Erik Westphal and Hermann Seitz.  Machine learning for the intelligent analysis of 3d printing conditions using environmental sensor data to support quality assurance.  Additive Manufacturing, 50:102535, 2022.
  * [33] Alessandra Caggiano, Jianjing Zhang, Vittorio Alfieri, Fabrizia Caiazzo, Robert Gao, and Roberto Teti.  Machine learning-based image processing for on-line defect recognition in additive manufacturing.  CIRP annals, 68(1):451–454, 2019.
  * [34] Chengcheng Wang, XP Tan, Shu Beng Tor, and CS Lim.  Machine learning in additive manufacturing: State-of-the-art and perspectives.  Additive Manufacturing, 36:101538, 2020.
  * [35] Xinbo Qi, Guofeng Chen, Yong Li, Xuan Cheng, and Changpeng Li.  Applying neural-network-based machine learning to additive manufacturing: current applications, challenges, and future perspectives.  Engineering, 5(4):721–729, 2019.
  * [36] Ohyung Kwon, Hyung Giun Kim, Min Ji Ham, Wonrae Kim, Gun-Hee Kim, Jae-Hyung Cho, Nam Il Kim, and Kangil Kim.  A deep neural network for classification of melt-pool images in metal additive manufacturing.  Journal of Intelligent Manufacturing, 31:375–386, 2020.
  * [37] Mohammad Amimul Ihsan Aquil and Wan Hussain Wan Ishak.  Predicting software defects using machine learning techniques.  International Journal, 9(4):6609–6616, 2020.
  * [38] Onur Dogan, Sanju Tiwari, MA Jabbar, and Shankru Guggari.  A systematic review on ai/ml approaches against covid-19 outbreak.  Complex & Intelligent Systems, 7:2655–2678, 2021.
  * [39] Arfan Majeed, Yingfeng Zhang, Shan Ren, Jingxiang Lv, Tao Peng, Saad Waqar, and Enhuai Yin.  A big data-driven framework for sustainable and smart additive manufacturing.  Robotics and Computer-Integrated Manufacturing, 67:102026, 2021\.
  * [40] Moslem Azamfar, Xiang Li, and Jay Lee.  Deep learning-based domain adaptation method for fault diagnosis in semiconductor manufacturing.  IEEE Transactions on Semiconductor Manufacturing, 33(3):445–453, 2020.
  * [41] YJ Qu, XG Ming, ZW Liu, XY Zhang, and ZT Hou.  Smart manufacturing systems: state of the art and future trends.  The International Journal of Advanced Manufacturing Technology, 103:3751–3768, 2019.
  * [42] Alexander Zeiser, Bekir Özcan, Christoph Kracke, Bas van Stein, and Thomas Bäck.  A data-centric approach to anomaly detection in layer-based additive manufacturing.  at-Automatisierungstechnik, 71(1):81–89, 2023.
  * [43] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [44] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
  * [45] Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda.  Extremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes.  arXiv preprint arXiv:1711.04325, 2017.
  * [46] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [47] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [48] Md Manjurul Ahsan, Muhammad Ramiz Uddin, Md Shahin Ali, Md Khairul Islam, Mithila Farjana, Ahmed Nazmus Sakib, Khondhaker Al Momin, and Shahana Akter Luna.  Deep transfer learning approaches for monkeypox disease diagnosis.  Expert Systems with Applications, 216:119483, 2023.
  * [49] Md Manjurul Ahsan, Tareque Abu Abdullah, Md Shahin Ali, Fatematuj Jahora, Md Khairul Islam, Amin G Alhashim, and Kishor Datta Gupta.  Transfer learning and local interpretable model agnostic based visual approach in monkeypox disease detection and classification: A deep learning insights.  arXiv preprint arXiv:2211.05633, 2022.
  * [50] Francois Chollet.  Deep learning with python, 2017.
  * [51] Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando De Freitas.  Predicting parameters in deep learning.  In Advances in Neural Information Processing Systems, pages 2148–2156, 2013.
  * [52] Frank Hutter, Jörg Lücke, and Lars Schmidt-Thieme.  Beyond manual tuning of hyperparameters.  KI-Künstliche Intelligenz, 29:329–337, 2015.
  * [53] Basheer Qolomany, Majdi Maabreh, Ala Al-Fuqaha, Ajay Gupta, and Driss Benhaddou.  Parameters optimization of deep learning models using particle swarm optimization.  In 2017 13th International Wireless Communications and Mobile Computing Conference (IWCMC), pages 1285–1290. IEEE, 2017.
  * [54] Jason Brownlee.  What is the difference between a batch and an epoch in a neural network?  Machine Learning Mastery, 2018.
  * [55] Leslie N Smith.  A disciplined approach to neural network hyper-parameters: Part 1–learning rate, batch size, momentum, and weight decay.  arXiv preprint arXiv:1803.09820, 2018.
  * [56] Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le.  Don’t decay the learning rate, increase the batch size.  arXiv preprint arXiv:1711.00489, 2017.
  * [57] James Bergstra and Yoshua Bengio.  Random search for hyper-parameter optimization.  Journal of machine learning research, 13(2), 2012.
  * [58] Luis Perez and Jason Wang.  The effectiveness of data augmentation in image classification using deep learning.  arXiv preprint arXiv:1712.04621, 2017.
  * [59] Paweł Filipczuk, Thomas Fevens, Adam Krzyżak, and Roman Monczak.  Computer-aided breast cancer diagnosis based on the analysis of cytological images of fine needle biopsies.  IEEE transactions on medical imaging, 32(12):2169–2178, 2013.
  * [60] Sharada P Mohanty, David P Hughes, and Marcel Salathé.  Using deep learning for image-based plant disease detection.  Frontiers in Plant Science, 7:1419, 2016.
  * [61] Tim Menzies, Jeremy Greenwald, and Art Frank.  Data mining static code attributes to learn defect predictors.  IEEE Transactions on Software Engineering, 33(1):2–13, 2006.
  * [62] Salvatore J Stolfo, Wei Fan, Wenke Lee, Andreas Prodromidis, and Philip K Chan.  Cost-based modeling for fraud and intrusion detection: Results from the jam project.  In Proceedings DARPA Information Survivability Conference and Exposition. DISCEX’00, volume 2, pages 130–144. IEEE, 2000.
  * [63] Shufa Zheng, Jian Fan, Fei Yu, Baihuan Feng, Bin Lou, Qianda Zou, Guoliang Xie, Sha Lin, Ruonan Wang, Xianzhi Yang, et al.  Viral load dynamics and disease severity in patients infected with sars-cov-2 in zhejiang province, china, january-march 2020: retrospective cohort study.  bmj, 369, 2020.
  * [64] Md Manjurul Ahsan, Yueqing Li, Jing Zhang, Md Tanvir Ahad, and Munshi Md Shafwat Yazdan.  Face recognition in an unconstrained and real-time environment using novel bmc-lbph methods incorporates with dji vision sensor.  Journal of Sensor and Actuator Networks, 9(4):54, 2020.


\util_i^{\allocselectionrule}(\infty, \reportwealth_i\primed, \reportval_{-i}, \reportwealth_{-i})
\util_i^\allocselectionrule(\reportval_i, \reportwealth_i, \reportval_{-i}, \reportwealth_{-i})
\end{align*}

as desired.

§.§ Proof of

We prove this by contradiction. Suppose there exists a pure Nash equilibrium as desired.
Without loss of generality, we assume $\alloc_{11} = 1$ and $\alloc_{12} > 0$, Namely, agent 1 receives her favored item 1 entirely and a positive fraction of item 2.
Let $\price_1, \price_2$ be the per-unit prices of the inner FPPE. The “highest bang-per-buck” property implies $\frac{1}{2}\price_1 = \price_2 \triangleq \price$.
Moreover, let $\payment_i$ be the payments for each agent $i$ in the equilibrium. Below we consider three cases depending agents' payments.

Case (i) agent $1$ does not exhaust her true budget $\truewealth_1$, i.e., $\payment_1 < \truewealth_1$.
In this case, we know that agent 2 does not exhaust her true budget as well, i.e., $\payment_2 < \truewealth_2$.
Otherwise, the per-unit price $\price_1$ of item 1 is at least $\price_1 = 2 \price_2 \geq 2\truewealth_2$ and thus agent $1$ receives negative utility due to the violation of her budget constraint.
Since both agents do not exhaust their budgets, invoking <Ref>, we have

\begin{align*}
\frac{\val_1}{1} \leq \frac{\price_1 + \price_2}{(1-\alloc_{11})\ctr_{11} + (1-\alloc_{12})\ctr_{12}},
\quad
\frac{\val_2}{1} \leq \frac{\price_2}{(1 - \alloc_{22})\ctr_{22}}
\end{align*}

which can be simplified as

\begin{align*}
\price_{11} \geq 2\max\left\{\frac{2}{3}\alloc_{22}, 1 - \alloc_{22}\right\} \geq \frac{4}{5}
\end{align*}

where the second inequality by considering all possible $\alloc_{22}$.
Notice that this leads to a contradiction since agent 1 receives a negative utility due to the violation of her budget constraint.

Case (ii) agent 1 exhausts her true budget $\truewealth_1$, i.e., $\payment_1 = \truewealth_1$.
Since agent $1$ exhausts her true budget $\truewealth_1$, we have

\begin{align*}
\truewealth_1 = \payment_1
= \price_1\alloc_{11} + \price_2 \alloc_{12}
= (2 + \alloc_{12})\price_2
\end{align*}

Using the same argument as the one in case (i),
we know that agent 2 does not exhaust her true budget.
Using a similar argument as the one in <Ref>, we consider a deviation of agent 2 by
increasing her reported budget for a sufficiently small amount $\epsilon > 0$ and let reported value be infinity. Under this deviation, agent $1$ still wins strictly positive fraction for both items. Thus, the new per-unit prices of inner FPPE under the deviation is $\price_1\primed = \price_1 + \frac{2}{3}\epsilon$
and $\price_2\primed = \price_2 + \frac{1}{3}\epsilon$.
Given that this is not a profitable deviation, we have

\begin{align*}
\val_2 \ctr_{22}\alloc_{22} - \price_2 \alloc_{22}
\geq
\val_2 \ctr_{22} \frac{\price_2\alloc_{22} + \epsilon}{\price_2 + \frac{1}{3}\epsilon} - (\price_2\alloc_{22} + \epsilon)
\end{align*}

Letting $\epsilon$ approach zero, it can be simplified as

\begin{align*}
\price_2 \geq 1 - \frac{1}{3}\alloc_{22}
\end{align*}

Together with $\wealth_1 = (2 + \alloc_{12})\price_2$, we obtain
$\frac{1}{2(3-\alloc_{22})} \geq 1-\frac{1}{3}\alloc_{22}$ which is not satisfied for all $\alloc_{22} \in [0, 1]$ and thus leads to a contradiction as desired.

§.§ Proof of

The proof of <Ref> relies on the following concept and lemmas in <cit.>.

For budgeted agents with types $\{(\val_i, \truewealth_i)\}_{i\in[n]}$,
a set of budget-feasible first-price pacing multipliers (BFPM) is a tuple $(\pacescalar, \alloc)$, of
pacing multipliers $\pacescalar \in [0, 1]$ for each bidder $i \in N$, and factional allocation $\allocij\in  [0, 1]$ for agent
$i \in [n]$ and item $j \in [m]$ with the following properties:

* (Price) Per-unit price $\price_j = \max_{i} \pacescalar_i \val_{ij}$

* (Highest bid wins)
If $\alloc_{ij} > 0$, then $\pacescalar_i \val_{ij} = \price_j$

* (Budget feasible) $\sum_{j} \alloc_{ij}\price_j \leq \wealth_i$

* (Demanded goods sold completely)
If $\price_j > 0$, then $\sum_{i}\alloc_{ij} = 1$

* (No overselling) $\sum_{i}\alloc_{ij}  \leq 1$

As we mentioned in <Ref>, FPPE can be interpreted as a BFPM where the pacing multipliers are defined as $\pacescalar_i = \min\{\max_{j\in[m]}\sfrac{\vali\ctr_{ij}}{\price_{j}}, 1\}\in[0, 1]$ for each agent $i$.

For any set of budgeted agents, FPPE Pareto-dominates all BFPMs in pacing multipliers, per-unit prices, and revenue.

In an FPPE, adding an item (agent) weakly increases revenue.

Now, we are ready to prove <Ref>.

Let $(\pacescalar, \alloc)$/$(\pacescalar\primed, \alloc\primed)$
be the FPPE before/after
agent $i$'s budget increasing.
Let $\price$/$\price\primed$
be their induced per-unit prices respectively.
It can be verified that $(\pacescalar, \alloc)$
is still a valid BFPM after budget increasing.
By <Ref>, $\pacescalar\primed_\ell\geq \pacescalar_\ell$
for every agent $\ell$,
and thus per-unit price $\price_j\primed \geq \price_j$ for every item $j$.

Now consider an instance $\instance\doubleprimed$ where we exclude agent $i$
while holding all other parameters fixed.
By <Ref>,
the revenue of FPPE in this instance $\instance\doubleprimed$
is at most
the revenue of FPPE in the original instance where agent $i$ has budget $\wealth_i$,
i.e., at most $\sum_j \price_j$.
It is sufficient to show the revenue of FPPE in $\instance\doubleprimed$ is
at least $\sum_j \price_j\primed - \wealth_i\primed$.
To see this, consider the following pacing multipliers $\pacescalar\doubleprimed\triangleq \pacescalar\primed_{-i}$ and
allocations $\alloc\doubleprimed \triangleq \alloc\primed_{-i}$.
The tuple $(\pacescalar\doubleprimed, \alloc\doubleprimed)$ satisfies all
constraints of BFPM for instance $\instance\doubleprimed$ except
(demand goods sold completely).
Note that we can further decompose every items into smaller pieces and discard
pieces with no allocation under $\allocs\doubleprimed$.
In this way, $(\pacescalar\doubleprimed, \allocs\doubleprimed)$ becomes a valid
BFPM after removing those (fractional) items,
and its revenue is at least $\sum_j \price_j\primed - \wealth_i\primed$.
Invoking <Ref> and <Ref> finishes the proof.

§.§ Proof of

Note that <Ref> also serves as a lower bound for <Ref>.
For the upper bound, we use a similar argument as the one in <Ref>.
In this analysis, we introduce auxiliary notation $\type_i \triangleq (\valuefunc_i, \truewealth_i,\moneycost_i)$ to denote agent $i$'s realized type, and notation $\typeMinusI$ for other agents $-i$ similarly.

Fix an arbitrary Bayesian Nash equilibrium $\strategy$.
Fix an arbitrary agent $i$.
Let $\optalloc(\typei, \typeMinusI)$ be the optimal allocation that maximizes the liquid welfare when
agent $i$ has type $\typei$, and other agents have types $\typeMinusI$.
For every type $\typei$,
define $\optexpectalloc_i(\typei) \triangleq \expect[\typeMinusI \sim \typedist_{-i}]{\optalloc_i(\typei, \typeMinusI)}$,
For every type $\typei$,
let $\randomalloc(\typei),\randomprice(\typei)$ be the randomized allocation and per-unit prices of the inner FPPE when agent $i$ has type $\typei$ under the equilibrium.
In both $\randomalloc(\typei),\randomprice(\typei)$, the randomness is taken over agent $i$'s message given strategy $\strategy_i$, other agents' types $\typeMinusI$
and their messages given strategy $\strategy_{-i}$.
Similarly, let $\randomalloc\doubleprimed,\randomprice\doubleprimed$ be the randomized allocation and per-unit prices of the inner FPPE when agent $i$ reports $(\reportval_i = 0, \reportwealth_i = 0)$ and all other agents use their equilibrium strategy $\strategy_{-i}$.
By <Ref>, $\randomprice\primed$
is stochastically dominated by $\randomprice(\typei)$ for all $\typei$.

In the first step of our argument, we prove the following inequality
for every type $\typei$ for agent $i$:

\begin{align}
\label{eq:multi item bayes wtp bound}
\frac{1}{2}\wtp_i(\optexpectalloc_i(\typei)) \leq
\moneycost_i^{-1}(\util_i(\typei)) +
\sum_{j\in[m]}\expect[\pricej\sim\randompricej\doubleprimed]{\pricej}\optexpectallocij(\typei)
\end{align}

where $\util_i(\typei)$ is the expected utility of agent $i$ with type $\typei$ in the equilibrium.
We proof inequality (<ref>)
using the same argument as the one in <Ref>.
we construct a specific deviation based on $\optexpectalloc_i(\typei)$ for agent $i$ with type $\typei$.
Let $\tildePj\doubleprimed \triangleq \expect[\pricej\doubleprimed\sim\randompricej\doubleprimed]{\pricej\doubleprimed}$
for every item $j\in[m]$.
Consider the following deviation for agent $i$
where agent $i$ deterministically reports
$\reportval_i\primed = \infty,
\reportwealth_i\primed =
\sum_{j\in[m]}
\tildePj\doubleprimed
\optexpectallocij(\typei)$.
Let $\randomalloc\primed,\randomprice\primed$ be the randomized allocation and per-unit prices of the inner FPPE after this deviation of agent $i$.
Since agent $i$ reports $\val_i\primed = \infty$,
she always exhausts her reported budget$~\reportwealth_i\primed$.

In the remaining analysis,
we couple the reported message of all other agents except agent $i$ in the FPPE inducing randomized per-unit prices $\randomprice\doubleprimed$, and the FPPE inducing randomized
allocation $\randomalloc\primed$ and per-unit prices $\randomprice\primed$.
It is important to note that
<Ref> implies

\begin{align*}
\pricej\doubleprimed \leq \pricej\primed
\text{ for every $j\in[m]$, and }
\sum_{j\in[m]}\pricej\primed \leq
\sum_{j\in[m]}\pricej\doubleprimed + \reportwealth_i\primed
\end{align*}

for every realization of agents' message profile.

Define auxiliary random variable
$\boldsymbol\tau_i \triangleq
\frac{\reportwealth_i\primed}
\randompricej\doubleprimed\optexpectallocij(\typei)} \geq 0$.
By definitions of $\boldsymbol\tau_i$ and $\reportwealth_i\primed$, $\expect{\frac{1}{\boldsymbol\tau_i}} = 1$.
The expected utility $\util_i$
of agent $i$ in the equilibrium can be lowerbounded as her expected utility under this deviation. Namely,

\begin{align*}
\util_i
\expect
\valuefunc_i\left(
\left(
\max_{j\in[m]}
\frac{\ctrij}{\pricej\primed}
\right)
\reportwealth_i\primed
\right)
\moneycost_i\left(\reportwealth_i\primed\right)}
\\
\expect
\valuefunc_i\left(
\left(
\max_{j\in[m]}
\frac{\ctrij}{\pricej\primed}
\right)
\left(
\sum_{j\in[m]}
\frac{1}{\tau_i + 1}
\tau_i\pricej\doubleprimed\optexpectallocij(\typei)
\frac{\tau_i}{\tau_i+1}
\left(\pricej\primed - \pricej\doubleprimed\right)
\right)
\right)}
\moneycost_i\left(
\sum_{j\in[m]}
\tildePj\doubleprimed
\optexpectallocij(\typei)
\right)
\\
\expect
\valuefunc_i\left(
\left(
\max_{j\in[m]}
\frac{\ctrij}{\pricej\primed}
\right)
\left(
\sum_{j\in[m]}
\frac{\tau_i}{\tau_i + 1}
\pricej\primed
\optexpectallocij(\typei)
\right)
\right)}
\moneycost_i\left(
\sum_{j\in[m]}
\tildePj\doubleprimed
\optexpectallocij(\typei)
\right)
\\
\expect
\valuefunc_i\left(
\sum_{j\in[m]}
\frac{\tau_i}{\tau_i + 1}
\frac{\ctrij}{\pricej\primed}
\pricej\primed
\optexpectallocij(\typei)
\right)}
\moneycost_i\left(
\sum_{j\in[m]}
\tildePj\doubleprimed
\optexpectallocij(\typei)\right)
\\
\expect
\frac{\tau_i}{\tau_i + 1}
\valuefunc_i\left(
\sum_{j\in[m]}
\optallocij
\right)}
\moneycost_i\left(
\sum_{j\in[m]}
\tildePj\doubleprimed
\optalloc_{ij}\right)
\\
\frac{1}
+1 }
\valuefunc_i\left(
\sum_{j\in[m]}
\optexpectallocij(\typei)
\right)}
\moneycost_i\left(
\sum_{j\in[m]}
\tildePj\doubleprimed
\optexpectallocij(\typei)  \right)
\\
\frac{1}{2}
\valuefunc_i\left(
\sum_{j\in[m]}
\optexpectallocij(\typei)
\right)
\moneycost_i\left(
\sum_{j\in[m]}
\tildePj\doubleprimed
\optexpectallocij(\typei)  \right)
\end{align*}

where inequality (a) holds due to <Ref>;
inequality (b) holds since $\reportwealth_i\primed = \tau_i\sum_{j\in[m]}\pricej\doubleprimed\optexpectallocij(\typei)$
and $\wealth_i\primed \geq \sum_{j\in[m]}\pricej\primed - \pricej\doubleprimed$ for every realization of
inequality (c) holds since $\optexpectallocij(\typei) \leq 1$;
inequality (d) holds by algebra;
inequality (e) holds due to the concavity of
valuation function $\valuefunc_i$ and $\valuefunc_i(0) = 0$;
inequality (f) holds due to Jensen's inequality;
and equality (g) holds since $\expect{\frac{1}{\boldsymbol\tau_i}} = 1$
by definition.

We are ready to prove inequality (<ref>) as follows,[Similar to the proof of <Ref>, this analysis does not use the differentiability of money cost function $\moneycost_i$. Thus, without loss of generality we assume $\truewealth_i = \infty$ and allow $\moneycost:\reals_+ \rightarrow \realsinf$ to simplify the presentation.]

\begin{align*}
\frac{1}{2}\wtp_i(\optexpectalloc_i(\typei))
\frac{1}{2}\moneycost_i^{-1}\left(
\valuefunc_i\left(
\sum_{j\in[m]}
\optexpectallocij(\typei)
\right)
\right)
\\
\moneycost_i^{-1}\left(
\frac{1}{2}
\valuefunc_i\left(
\sum_{j\in[m]}
\optexpectallocij(\typei)
\right)
\right)
\\
\moneycost_i^{-1}\left(
\util_i(\typei) +
\moneycost_i\left(\sum_{j\in[m]}
\tildePj\doubleprimed
\optexpectallocij(\typei) \right)
\right)
\\
\moneycost_i^{-1}\left(
\util_i(\typei)  \right)
\sum_{j\in[m]}
\tildePj\doubleprimed
\optexpectallocij(\typei)
\end{align*}

equality (a) holds due to the definition of $\wtp_i$;
inequalities (b) and (d) hold due to the concavity of $\moneycost_i^{-1}$;
inequality (c) holds due to the monotonicity of $\moneycost_i^{-1}$ and the lower bound of the expected utility $\util_i$ obtained above.

Given inequality (<ref>), we are able to show the upper bound of the PoA.
First, for agent $i$ with type $\typei$,

\begin{align*}
\expect[\typeMinusI \sim \typedist_{-i}]{\wtp_i(\optalloc_i(\typei, \typeMinusI))}
\wtp_i\left(\expect[\typeMinusI \sim \typedist_{-i}]{\{\optalloc_i(\typei, \typeMinusI)}
\right)
\\
\wtp_i\left(\optexpectalloc_i(\typei)\right)
\\
\left(
\moneycost_i^{-1}(\util_i(\typei))
\sum_{j\in[m]}\tildePj\doubleprimed\optexpectallocij(\typei)
\right)
\end{align*}

inequality (a) holds due to the concavity of $\wtp_i$.

\begin{align*}
\expect[\type\sim\typedist]{\wtp(\optalloc(\type))}
\sum_{i\in[n]}\expect[\typei\sim\typedist_i]{\expect[\typeMinusI \sim \typedist_{-i}]{\wtp_i(\optalloc_i(\typei, \typeMinusI))}}
\\
\sum_{i\in[n]}\expect[\typei\sim\typedist_i]{
\left(
\moneycost_i^{-1}(\util_i(\typei))
\sum_{j\in[m]}\tildePj\doubleprimed\optexpectallocij(\typei)
\right)
\\
\sum_{i\in[n]}\expect[\typei\sim\typedist_i]{
\moneycost_i^{-1}(\util_i(\typei))
\sum_{i\in[n]}\sum_{j\in[m]}\expect[\typei\sim\typedist_i]{\tildePj\doubleprimed\optexpectallocij(\typei)}
\right)
\\
\expect[\typei\sim\typedist_i]{\wtp(\randomalloc_i(\typei)}
\end{align*}

where the last inequality hold
since $\moneycost_i^{-1}(\util_i(\typei))\leq \moneycost_i^{-1}
\left(\expect[\alloci\sim\randomalloc_i(\typei)]{\valuefunc_i\left(\sum_{j\in[m]}\allocij\ctrij\right)}\right)
\wtp_i(\randomalloc_i(\typei))$, and
= \sum_{j\in[m]}{\tildePj}
\leq \sum_{i\in[m]}\sum_{j\in[m]}\expect[\typei\sim\typedist_i]
\leq  \wtp(\allocs)$ where the last two inequalities hold due to the stochastic dominance between $\randomprice\doubleprimed$ and $\randomprice(\type_i)$ for all type $\typei$ and the non-negative utility of every agent in the equilibrium.

§.§ Proof of

For every agent $i$, we have

\begin{align*}
\allocLBi(\price) &\overset{(a)}{=} \left(1 \wedge \left(1 - \frac{\price}{\vali}\right)\right)\cdot \indicator{\vali \geq \price}
\left(1 - \frac{\price}{\vali}\right)\cdot \indicator{\vali \geq \price}
% \\
\allocUBi(\price) = \left(1 \wedge \frac{\truewealth_i}{\price}\right)\cdot \indicator{\vali\geq \price}
\end{align*}

where equality (a) holds since $\truewealthi > \frac{1}{4}\vali$.
Moreover, $\allocLBi(\price) < \allocUBi(\price)$ for every $\price \leq \vali$.
Thus, $\PriceL \subsetneq \PriceH$ and $\PriceL = \{\priceL\}$ is a singleton.

Below we argue that there exists high-price equilibrium with per-unit price in $\PriceH \backslash\PriceL$.
By the definition of $\PriceL$, there exists at least two distinct agents $i\primed, i\doubleprimed$ such that $\allocLB_{i\primed}(\priceL), \allocLB_{i\doubleprimed}(\priceL) > 0$, $\val_{i\primed}, \val_{i\doubleprimed} \geq \priceL$ and thus $\allocLB_{i\primed}(\priceL) < \allocUB_{i\primed}(\priceL)$, $\allocLB_{i\doubleprimed}(\priceL) < \allocUB_{i\doubleprimed}(\priceL)$.

\begin{align*}
\allocLB_{i\primed}(\priceL) + \sum\nolimits_{i\not=i\primed}\allocUBi(\priceL)
\allocLB_{i\primed}(\priceL) + \allocLB_{i\doubleprimed}(\priceL) +
\sum\nolimits_{i\not=i\primed,i\doubleprimed}\allocUBi(\priceL)
\geq
\sum\nolimits_{i\in[n]}\allocLBi(\priceL) = 1
\end{align*}

Since both $\allocLBi(\price)$ and $\allocUBi(\price)$ are continuous, for sufficiently small $\epsilon$, we have

\begin{align*}
\allocLB_{i\primed}(\priceL + \epsilon) + \sum\nolimits_{i\not=i\primed}\allocUBi(\priceL + \epsilon) > 1
\end{align*}

which is the condition for high-price equilibrium existence with per-unit price $\priceL + \epsilon \in \PriceH\backslash\PriceL$ as desired.

§.§ Proof of

By <Ref>, both $\allocLB_i(\price)$ and $\allocUB_i(\price)$ are weakly decreasing in $\price$.
Thus, $\PriceL$ and consequently low-price equilibrium in <Ref> can be computed (e.g., via binary search) in polynomial time. Similarly, $\PriceH$ can be computed (e.g., via binary search) in polynomial time. Moreover, price $\price\in\PriceH$ that satisfies conditions $|\lagent(\price)| \geq 1$ and $\sum_{i\in\hagent(\price)}\allocUB_(\price)\geq 1$ can also be identified in polynomial time. Therefore, the high-price equilibrium can be computed in polynomial time as well.

§ BEST RESPONSE COMPUTATION FOR SINGLE-ITEM INSTANCES

In this section, we focus on the metagame for single-item instances and explain how to compute the best response given other agents' reported message in polynomial time. Similar to <Ref>, we assume $\ctrij = 1$ for all agents and drop it as well as subscript index $j$ for the item without loss of generality.

In the metagame with a single item, given other agents' reported message $(\reportval_{-i}, \reportwealth_{-i})$,
there exists a polynomial time algorithm that
computes the best response $(\reportval_i, \reportwealth_i)$ of agent $i$.

We focus on the best response computation of agent $1$.
Without loss of generality, we assume $\reportval_2 \leq \reportval_3 \leq \dots \leq \reportval_n$.
By <Ref>, it suffices to consider reported message $(\reportval_1 = \infty, \reportwealth_1)$.
By definition, the per-unit price $\price$ of the inner FPPE is continuous, weakly increasing in $\reportwealth_1$ and agent $1$ has allocation $\alloc_1 = \sfrac{\reportwealth_1}{\price}$ and payment $\payment_1 = \reportwealth_1$. Though agent $1$'s utility $\util_1$ is not globally concave in $\reportwealth_1$, it can be divided into at most $2n-1$ pieces where each piece is concave in $\reportwealth_1$. See <Ref> for an graphical illustration. Specifically, in each piece,  price $\price$ either (i) increases linearly in $\reportwealth_i$, or (ii) stays constant and is equal to $\reportval_i$ for some agent $i$. In both cases, it can be verified that agent $1$'s utility is concave, since her valuation function $\valuefunction_1$ (money cost function $\moneycost_i$) is concave (convex). By utilizing this piecewise concavity, the best $\reportwealth_1$ in each piece can be solved efficiently. Since there are at most $2n-1$ pieces, the optimal $\reportwealth_1$ can be computed in polynomial time as desired.


# Advanced cryogenic process control and monitoring for the SPIRAL2
superconducting LINAC

A. Ghribi1,2<EMAIL_ADDRESS>M. Aburas1,3 P-E. Bernaudin1,3 P. Bonnay3,4
F. Bonne3,4
A. Corbel1,2 M. Di-Giacomo1,3 F. Millet3,4 A. Ngueguim Tsafak3,4 A. Trudel1,3
Q. Tura1,2
1Grand Accélérateur National d’Ions Lourds (GANIL), Caen, France
2Centre National de la Recherche Scientifique (CNRS - IN2P3), Caen, France
3Commissariat à l’Energie Atomique (CEA), Caen, France
4Univ. Grenoble Alpes, CEA, IRIG-DSBT, Grenoble, France

###### Abstract

SPIRAL2 is a superconducting accelerator for protons, deuterons and heavy ions
delivering a maximum beam power of 200 kW at 40 MeV (for deuteron beams). 26
superconducting quarter wave cavities are operated at 4.4 K, plunged in a
liquid helium bath with a drastic phase separator pressure control. Previous
years have seen the development of advanced process control for cryogenics
allowing to cope with high heat load dynamics thanks to an automatic heat
dissipation compensation and a model based control. The latter is based on
models, using the Simcryogenics library, optimized and linearised in the
Programmable Logic Controllers. The SPIRAL2 operation has demonstrated that
such control allows to keep the specified conditions for RF and beam operation
even at levels of heat load dissipation approaching the physical limits of the
cryogenic system. These developments allowed to synthesise a virtual observer
of the dynamic heat load dissipated by the cavities. The present paper
summarises the development of such observer based on the physical
thermodynamic model and on machine learning techniques.

##  1 INTRODUCTION

SPIRAL2 [orduz2022commissioning] at GANIL (Grand Accélérateur National d’Ions
Lourds) is a state of the art accelerator that aims at delivering some of the
highest intensities of rare isotope beams. Its driver should accelerate
protons, deuterons and heavy ions at intensities up to 5 mA and a maximum beam
power of 200 kW (see Table 1) [dubois2022radioactive]. The driver is composed
of ECR (Electron Cyclotron Resonance) sources, a RFQ (Radio-Frequency
Quadrupole) and a superconducting LINAC (LINear Accelerator). The latter is
made of 26 QWR (Quater Wave Resonating) independently phased superconducting
cavities. SPIRAL2 cavities are of two families: low beta ($\beta_{0}=0.07$)
and high beta ($\beta_{0}=0.12$) [marchand2015performances]; where $\beta_{0}$
is the relative velocity of the accelerated particles. The low beta cavities
are the first 12 cavities of the LINAC. They are made of bulk Niobium for
their upper part and copper for the bottom part. Each low beta cavity is
housed in a different cryostat and separated by room temperature quadrupole
magnets. The cryostats, that we call cryomodules, integrate passive RF, vacuum
and other components needed for the operation of the cavities. The last part
of the LINAC comprises 7 cryomodules, housing 2 high beta cavities each. These
cavities are made of bulk niobium. All cavities are operated at 88.0525 MHz.
To be operated, the superconducting cavities are plunged in liquid helium
baths with a stringent pressure control to avoid detuning. Every cryomodule
has a dedicated satellite valves-box that insures liquid helium distribution
and pressure/level regulations. These satellite cryostats form the
cryodistribution. The latter can be decomposed in two branches: A left branch
made of 12 valves boxes (for all low beta cryomodules) and a right branch made
of 7 valves boxes (for all high beta cryomodules). More details on the
cryoplant and the cryodistribution can be found in [GHRIBI201744].

The last years have seen the commissioning of the different parts of the
accelerators leading to the beam commissioning and the current ramp up to
achieve the target requirements [orduz2022commissioning]. Cryogenics proved
more challenging that initially expected [GHRIBI2020103126]. This pushed the
developments of advanced model based control bringing immunity to high dynamic
heat loads and uneven heat load distributions effects. One of the consequences
was the ability to introduce model based virtual observers which opened the
way to machine learning based diagnostics. This paper depicts these
developments. The first part details the specific challenges of the cryogenics
of the SPIRAL2 LINAC and the strategies to overcome them. The second focuses
on virtual heat load observers.

Table 1: SPIRAL2 Beam Specifications Particles | $H^{+}$ | ${}^{3}He^{2+}$ | $D^{+}$ | ions | [Units]
---|---|---|---|---|---
Q/A | 1 | 3/2 | 1/2 | 1/3 |
Maximum current | 5 | 5 | 5 | 1 | [mA]
Maximum beam power | 165 | 180 | 200 | 45 | [kW]

##  2 CRYOGENIC CONTROL:
constraints and strategies

The first purpose of the cryogenic system is to cool down the superconducting
cavities under their transition temperature and keep them in optimal operating
conditions. Apart from the cool down, which poses its own set of challenges
due to Q-disease (see [GHRIBI2020103126]), maintaining optimal operating
conditions can be difficult. Superconducting cavities operate around a center
frequency $f_{0}$ with an admissible bandwidth $\Delta f_{0}$. Micrometric
shape deformations due to surface forces such as Lorentz forces or liquid
helium pressure fluctuations can shift the cavities resonant frequency beyond
their admissible bandwidth. The link between the liquid helium pressure
variations $\delta P_{bath}$ and the frequency shift $\delta f$ can be
translated in term of pressure-frequency sensitivity $S_{p}$ with:

$S_{p}=\frac{\delta f}{\delta P_{bath}}$ (1)

In the case of SPIRAL2, $S_{p}$ is comprised between -1 and -3 Hz/mbar for low
beta cavities and between -5 and -7 Hz/mbar for high beta cavities.
Considering a cavity phase shift $\Delta\Phi=\pi/12$ and a loaded quality
factor $Q_{L}\sim 10^{6}$, this brings the admissible $\delta P_{bath}$ to
$\sim$ 10 mbars for low beta cavities, and $\sim$ 3 mbars for high beta ones.

The cryogenic system of SPIRAL2 is based on a cryoplant that produces up 120
g/s of bi-phasic helium at 1300 mbars. The design of the system makes the
pressure in the cavities helium baths (turbulence, two-phase flow behaviour)
particularly difficult to control with independently tuned PID (proportional
integral derivative) valves controllers and uneven helium flow distribution
along the cryodistribution. To overcome these challenges, two strategies have
been applied. The first one is to equally distribute the flows between the
LINAC cryodistribution branches and between the cryomodules of every branch.
This is done by using heaters located close to the cavities liquid helium
baths to apply a so-called dynamic heat load compensation. This dynamic heat
load compensation not only equally distributes the helium flows but also keeps
it constant with respect to accelerating fields variations (see Fig. 1). The
second strategy was to use model based MIMO (multiple inputs - multiple
outputs) controllers for simultaneous liquid helium level and pressure
control. These controllers were based on a cryogenic model of the SPIRAL2
LINAC [vassa2019] linearized around a given set of boundary conditions, kept
constant thanks to the dynamic heat load compensation. Tuned differently, the
controllers also allowed to operate abnormal cryomodules at the limit of what
the physical system allows.

Figure 1: Heat load distribution in the LINAC showing the dynamic heat load
compensation strategy.

##  3 State observers for advanced monitoring and diagnostics

###  3.1 Model based state observers

Figure 2: Model based heat load observer architecture. Figure 3: Up: density
distribution of the estimated heat load $Q_{h}^{obs}$ vs the measured heat
load $Q_{h}^{real}$. Bottom: density distribution of residual of estimated
heat load. Red: ideal behaviour. Green: DNN estimator. Blue: model based
linear PI estimator. Figure 4: Neural network architecture layout used for the
machine learning based heat load observer.

One by-product of having a model based control is the ability to estimate
hidden parameters of a physical system from existing instrumentation. In our
case, the cryogenic model of the LINAC has been used to generate a virtual
observer of the dynamic heat load of the cavities. The architecture of such
observer is described in Fig. 2. Starting from the instructions given to the
input and output valves controllers (pressure and liquid helium level
control), we extract the estimated internal energy of the system, its enthalpy
as well as the input and output helium mass flows. Filtered pressure and
liquid helium level measurements as well as estimated model parameters are
injected in a PI (Proportional Integral) observer. The latter uses a time
invariant linear system (obtained by linearization around the operation set-
point) and is described by the following equation:

$\begin{array}[]{c}x(k+1)=Ax(k)+Bu(k)+Ed(k)\\\ y(k)=Cx(k)\\\ \end{array}$ (2)

where $x=\left[\begin{array}[]{c}\rho\\\
u_{in}\end{array}\right]\in\mathbb{R}^{n_{x}}$ is the state vector, with
$\rho$ the helium density and $u_{in}$ the internal energy;
$y=\left[\begin{array}[]{c}PT\\\ LT\end{array}\right]\in\mathbb{R}^{n_{y}}$ is
the output vector, with $PT$ the measured helium bath pressure and $LT$ the
measure liquid helium level ; $u=\left[\begin{array}[]{c}\dot{m}_{in}\\\
\dot{m}_{out}\end{array}\right]\in\mathbb{R}^{n_{u}}$ the input vector, with
$\dot{m}_{in}$ and $\dot{m}_{out}$ being respectively the input and output
helium mass flows ; $d=\left[\begin{array}[]{c}Q_{h}\\\
\varepsilon_{\dot{m}}\end{array}\right]\in\mathbb{R}^{n_{d}}$ the unknown
input vector, with $Q_{h}$ the heat load and $\varepsilon_{\dot{m}}$ the error
on the estimated helium mass flow. Considering that the system is stable, the
estimated gain of the PI observer allows to recover the unknown inputs: the
heat load $Q_{h}$ and the mass flow error $\varepsilon_{\dot{m}}$. Results of
linear PI model based heat load observers are shown in Fig. 3. The system
being non linear, a linear estimator introduces an error that increases with
distance to the operation set-point. This limits its use for high dynamics
fault detection. To improve the estimation of the mass flow error, a non
linear PI observer is under development. Preliminary results show a clear
improvement with respect to the linear PI estimator but highlight other non
linearities due to measurement noise and non symmetric behaviour with regard
to a positive or negative heat load dynamic.

###  3.2 Machine learning state observers

To overcome the limitations of the model based PI heat load observer, a
different yet complementary approach based on machine learning techniques is
being explored. A 4 layers dense neural network, following the architecture
shown in Fig. 4, has been used. The network was trained during 100 epochs,
with a learning rate of $10^{-5}$ and a batch size of 64. A rectified linear
unit (ReLu) activation function has been chosen to account for the non-
linearity needed to reproduce the complex correlations underlying the features
set. Finally, MSE (mean squared error) has been chosen for the estimation of
the loss function with:

$\operatorname{MSE}={\frac{1}{n}}\sum_{i=1}^{n}\left(Y_{i}-{\hat{Y_{i}}}\right)^{2}$
(3)

where n is the size of the dataset, $Y$ the observed values and $\hat{Y}$ the
predicted values.

Results of the DNN model with respect to the linear PI observer can be seen in
Fig. 3. While the DNN observer outperforms the linear PI observer, it remains
subject to important improvements to account for measurement noise. Features
binning optimization and convolution networks are the next steps in improving
such observers. The model remains important for understanding specific
abnormal behaviours. In that respect, a physics informed learning is planned
for precise estimation of mass flow errors.

##  4 CONCLUSION

Previous years have seen the success of the cryogenic commissioning of the
SPIRAL2 LINAC. The road to operation poses a whole new set of challenges
linked to the reliability of the systems. Efforts to make the commissioning a
success gave birth to a robust thermodynamic model of the SPIRAL2 LINAC,
allowing an efficient model based control. As a result, knowledge of the
physical behaviour of the cryogenic system made possible the synthesis of
virtual model based dynamic heat load observers. However, the inherent reality
of everyday operation like valves mis-calibration, cold helium flow profile,
instrumentation measurement noises as well as internal non linearities limit
the use of a linear observer for diagnostics. A first and successful attempt
to use dense neural networks for heat load virtual observers has been made.
Several improvements on both machine learning and model based observers will
however be necessary to enable their final integration in the accelerator main
control system for fault detection and diagnostics.

##  5 ACKNOWLEDGEMENTS

This work has been funded by "Region Normandie" as well as the city of Caen,
CNRS and CEA. This work has been done in the frame of the SPIRAL2 project and
the GRAAL collaboration.

## References

  * [1] JACoW, http://www.jacow.org
  * [2] IEEE Editorial Style Manual, IEEE Periodicals, Piscataway, NJ, USA, Oct. 2014, pp. 34–52.
  * [3] https://woodward.library.ubc.ca/researchhelp/journal-abbreviations/