Paper Title: UMMAN: Unsupervised Multi-graph Merge Adversarial Network for Disease Prediction Based on Intestinal Flora.



















UMMAN: Unsupervised Multi-graph Merge Adversarial Network for Disease Prediction Based on Intestinal Flora
    Dingkun Liu , Hongjie Zhou , Yilu Qu , Huimei Zhang , Yongdong Xu
^ 0000-0001-1234-1234∗

        
Dingkun Liu, IEEE Student Member, is with the University of Huazhong University of Science and Technology, 430074, China. Email: clarkjs0817@gmail.com.
Yongdong Xu is with the University of Harbin Institute of Technology, 264209, China. Email: ydxu@hit.edu.cn.

(Corresponding author: Yongdong Xu.)
    Received ..... .., ....; accepted ......, ....
=================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
The abundance of intestinal flora is closely related to human diseases, but diseases are not caused by a single gut microbe. Instead, they result from the complex interplay of numerous microbial entities. This intricate and implicit connection among gut microbes poses a significant challenge for disease prediction using abundance information from OTU data. Recently, several methods have shown potential in predicting corresponding diseases. However, these methods fail to learn the inner association among gut microbes from different hosts, leading to unsatisfactory performance. In this paper, we present a novel architecture, Unsupervised Multi-graph Merge Adversarial Network (UMMAN). UMMAN can obtain the embeddings of nodes in the Multi-Graph in an unsupervised scenario, so that it helps learn the multiplex association. Our method is the first to combine Graph Neural Network with the task of intestinal flora disease prediction. We employ complex relation-types to construct the Original-Graph and disrupt the relationships among nodes to generate corresponding Shuffled-Graph. We introduce the Node Feature Global Integration (NFGI) module to represent the global features of the graph. Furthermore, we design a joint loss comprising adversarial loss and hybrid attention loss to ensure that the real graph embedding aligns closely with the Original-Graph and diverges from the Shuffled-Graph. Comprehensive experiments on five classical OTU gut microbiome datasets demonstrate the effectiveness and stability of our method. (We will release our code soon.)








§ INTRODUCTION






According to incomplete statistics, there are at least 500 trillion microorganisms in the human body, outnumbering human cells by more than tenfold <cit.>. Microorganisms are not only large in number, but also diverse in types, including viruses, bacteria, fungi, eukaryotes, etc. Collectively, these microbes constitute the human microbiome, an essential focus of contemporary biomedical research.

Some specific diseases are closely related to the abundance of microbial communities in the human body, with the vast majority of these microorganisms residing in the gut <cit.>. As the largest microbial habitat in the human body, the gut microbiome has a direct impact on human health and diseases. For example, researchers have found significant differences in the proportions of Bacteroides and Actinobacteria in the intestines of obese and lean individuals, with the proportion of Firmicutes also influencing obesity <cit.>. Various metabolites of gut microbes, such as short-chain fatty acid and aromatic amino acids, all directly or indirectly induce type 2 diabetes <cit.>. Patients with inflammatory bowel disease (IBD) exhibit a relatively disordered intestinal microbiota <cit.>, with microbial metabolites such as acetate and butyrate playing crucial roles in immune regulation. The concentration of butyrate is notably decreased in patients with ulcerative colitis, highlighting the influence of gut microbes on IBD. In recent years, with the rise of 16SrRNA sequencing technology <cit.>, new studies have shown that the number of lactic acid bacteria in the intestines of patients with irritable bowel syndrome decreased significantly, while the number of Veillonella in patients with constipation-predominant irritable bowel syndrome increased <cit.>. These findings further confirm the close connection between intestinal microbes and human health <cit.>.

With the rapid advancement of medical technology, obtaining relevant data has become increasingly convenient. 16SrRNA sequencing technology <cit.> can be used to obtain the abundance of human intestinal flora. Compared to whole genome sequencing, this technology is widely used due to its relatively low cost. However, processing this data to derive meaningful disease analysis results remains a significant challenge. Expert analysis of intestinal flora is complex and time-consuming, making it imperative to leverage artificial intelligence to enhance efficiency and reduce costs.

However, diagnosing diseases based on intestinal flora information has not achieved satisfactory results due to the complex and multiplex connections among the intestinal flora. Most existing approaches consider only a single relation-type even ignore the connection, fail to cover the intricate connection among the gut microbes of different hosts. Graph learning <cit.> are well-suited for handling graph-structured data with rich relationships <cit.>, and can represent information at various depths. Therefore, we utilize Graph Neural Networks (GNNs) to learn the connections among gut microbes from different hosts, effectively guiding disease prediction. It is worth noting that our method does not rely on the labels in the process of obtaining the embeddings of the nodes and the graphs, which can obtain satisfactory embeddings in an unsupervised scenario.

Considering all the above challenges, we summarize the contributions of this paper as follows:


   
  ∙ We are the first to introduce graph machine learning to the field of disease prediction based on gut microbiota. We propose a novel Unsupervised Multi-graph Merge Adversarial Network (UMMAN). It learns the intricate connection among intestinal flora of different hosts to guide the prediction of diseases.
   
  ∙ We use a variety of relation-types to build a Multi-graph, and shuffle the nodes of the corresponding graph to destroy the association among nodes. We introduce the adversarial loss and the hybrid attention loss as a joint loss in order to make the true embeddings agree with the Original-Graph and disagree with the Shuffled-Graph as much as possible.
   
  ∙ We propose the Node Feature Global Ensemble (NFGI) Descriptor, which includes node level stage and graph level stage to better represent the global embedding of a graph.
   
  ∙ Experiments on the benchmark datasets indicate that our UMMAN achieves state-of-the-art on the disease prediction task of gut microbiota, and also prove that our method is more stable than previous methods.






§ RELATED WORK


Benefiting from the rapid advancement of medical technology, many studies have been able to use OTU data for analysis <cit.>. Pasolli et al. <cit.> comprehensively evaluated the prediction tasks based on shotgun metagenomics and the method of microbial phenotype association evaluation, mainly using support vector machines and random forest models to predict diseases, and with the help of Lasso and elastic net (ENet) Regularized Multiple Logistic Regression. In their study, cirrhosis was the most predictive disease, and the model used in the study had good generalization ability for cross-stage data, but poor generalization ability for cross-datasets. Sharma et al. <cit.> proposed TaxoNN to predict the conn between gut microbes and diseases, but only used two datasets to test the effect.

Manandhar et al. <cit.> used fecal 16S metagenomics data to analyze 729 IBD patients and 700 healthy individuals through 5 machine learning methods. After identifying 50 microorganisms with significant differences, the prediction was obtained through the random forest algorithm. Based on the data of the gut project in the United States, Linares et al. <cit.> used the glmnet model and the random forest model to predict the country of origin. Wong et al. <cit.> investigated the possible gastrointestinal effects of neratinib in the treatment of breast cancer. By collecting stool samples from 11 drug-taking patients and classifying patients who may develop diarrhea by a tree-based classification method.

In general, at present, for data in the form of OTU datasets that are extended to table types, the traditional machine learning algorithm may be relatively more effective <cit.>. However, these algorithms are relatively fixed and there is not much space for improvement, reaching a bottleneck on the problem. More disappointing, basic CNNs perform well in many applications, but it can't beat the traditional machine learning algorithm in this case. This is because the basic CNN’s method cannot solve the problem of long-distance dependencies, nor can it adapt to data whose order can be changed at will. The exchange of rows and columns of the tabular datasets do not affect the results of pattern recognition, and more challengingly, diseases are not caused by a single gut microbe, but by a combination of a large number of microbial information. However, Graph Learning can help learn the intricate connection among the intestinal flora of different hosts. We propose the UMMAN unsupersived method, combining Graph Neural Network with gut microbiome for the first time, and it has achieved excellent performance on benchmark datasets.






§ METHOD





In this section, we will first introduce the overview of the architecture of our method — UMMAN. In the following section, we introduce how to construct the Original-Graph and the Shuffled-Graph. We then present the details of the two stages of the Node Feature Global Integration descriptor: node-level stage and graph-level stage. Then we introduce the attention block to derive the embedding of each node. We also propose the joint loss which consists of adversarial loss ℒ_adv and hybrid attention loss ℒ_h-attn in the end.



 §.§ THE OVERVIEW OF UMMAN ARCHITECTURE

The architecture of UMMAN we propose will be introduced detailedly in this part. As stated above, it exists close but extremely complex connection among gut microbes of different hosts, and such relation is implicit. Specifically, it is not the abundance of a single intestinal microbe that can establish a direct relationship with the final disease, but it is caused by the combined information of various microbes. For tabular datasets in the form of OTUs(A table with rows representing gut microbiota abundance and columns representing samples), their adjacent data are not correlated, so traditional convolutional neural networks are not applicable. Therefore, how to learn the connection among gut microbes of different hosts has become the biggest difficulty in this field, and we are the first to combine graph machine learning with gut flora disease prediction tasks which helps learn the connection. The following will introduce the architecture of UMMAN that we propose in detail. 

The overview architecture of UMMAN is shown in [figure1]Fig .1. Each node represents a host. We use multiplex indicators to measure the similarity among nodes to build the Original-Graph. In order to make UMMAN learn associations more accurately, we destroy the Original-Graph's association among nodes to get Shuffled-Graph. The Original-Graph and Shuffled-Graph are updated through the Graph Convolutional Network at the same time in order to obtain the embeddings of each node. We introduce the attention block to obtain the embeddings of the nodes in the Original-Graph and the embeddings of the nodes in the Shuffled-Graph respectively. In addition, we design a Node Feature Global Integration (NFGI) descriptor to denote the embedding of a graph. Then, in order to make the true embedding that includes the complex and implicit relationships among gut microbes of different hosts agree with the Original-Graph as much as possible and disagree with the Shuffled-Graph as much as possible, we design the total joint loss function consists of an adversarial loss and hybrid attention loss, and used unsupervised methods to extract features from the Original-Graph. Finally, the Original-Graph after being processed by the attention block is fed into a dense layer classifier to produce the classification result.





 §.§ CONSTRUCT MULTI-GRAPH AND NODE EMBEDDING

We find that the abundance data of a part of the intestinal flora in the dataset was extremely low in most hosts whose impact on the performance of the algorithm is detailed above. If all the data of the original dataset is retained, it may cause some intestinal flora to be almost completely absent in all hosts. We can consider such data as "dirty data", which will lead to training effects decline. Therefore, we removed the intestinal flora with low abundance in most hosts to improve the effect of feature extraction.

In order to make the motivation more convincing, we show the connection of the 10 most abundant flora in the cirrhosis dataset in the sample intestine in [figure2]Fig .2. The diagonal graph describes the distribution histogram of the flora, and the off-diagonal represents the relationships between the flora and the abundance in the intestinal tract of the sample, where the red dots represent healthy hosts, and the blue dots represent diseased samples. It can be found that the correlation is not obvious and appears to be disorganized, which proves that it is unreliable to measure the relation-type among hosts only based on the abundance of flora. To explore the implicit information of fused features in intestinal flora, we consider multiplex relation-types <cit.> to measure the relationships among vectors during the initialization of building the edges among nodes in the graph:


    𝒮_1(m, n) = ∑_i=1^d|m_i - n_i|/∑_i=1^dm_i + ∑_i=1^dn_i



    𝒮_2(m, n) = √(∑_i=1^d (m_i - n_i) ^ 2)



    𝒮_3(m, n) = ∑_i=1^d|m_i - n_i|/|m_i| + |n_i| ,









  where Bray Curtis Distance, Euclidean Distance and Canberra Distance are used to initialize a graph. We construct the original graph by measuring the multivariate discrepancy between nodes, two nodes are considered to be connected if the discrepancy among their embeddings is below a variable threshold θ. In order to make our model more robust, and to obtain the correlation among nodes more reliably, we design the adversarial control group, i.e., keep the position of the edges unchanged, randomly disrupt the nodes to get the Shuffled-graphs, and train the discriminator to get the adversarial loss. The specific process will be introduced in detail later.

  

  We introduce an encoder module for each relation type inspired by the Graph Convolutional Network<cit.>, aiming to obtain the embedding of each node in the graph. We define a conditional function ℱ as an update function between layers, the Shuffled-Graph is operated by the same process R^N× F↪R^N× D:


  
  
  


    𝒳_j^(l + 1)   = ℱ(𝒳_j^(l), Â | W^(l), b^(l)) 
       = σ( ∑_j ∈ N_i𝒟̂^- 1/2𝒜̂𝒟̂^- 1/2𝒳_j^(l)W^(l) + b^(l)),











  where 𝒳_j^(l), is the embedding of the l layer of the node whose index is j, N_i represents the set of nodes adjacent to 𝒳_j^(l), 𝒜̂ and 𝒟̂ are the adjacency matrix and degree matrix of a certain graph, W^(l) and b^(l) are the trainable parameters of the l-th layer's weight matrix and bias, σ is the nonlinearity layer which is designed as ReLU in our method. Through Graph Convolutional Network, X_j^(t) become a D-dimensional tensor representing the embeddings of the node with index j of the t-th relation-type.




 §.§ NODE FEATURE GLOBAL INTEGRATION DESCRIPTOR WITH TWO STAGES

In the NFGI module, we propose a novel contribution to characterize a graph 𝒥^(t)  / 𝒥̃^(t) with two stages(𝒥^(t) is the Original Graph constructed according to the t-th relation-type, and 𝒥̃^(t), is the corresponding graph after being shuffled, since the operations of the two graphs are equivalent, we use 𝒥^(t) uniformly in the following text). In the conventional method, only the features of each node are fused, but using this method to characterize a graph only considers the information of each node. Therefore, our NFGI module adopt a two-stage process, i.e., node-level stage and graph-level stage to jointly characterize the features of the graph to better obtain the embedding.

Node-Level stage. We first compute the node-level contribution to graph 𝒥^(t) based on the characteristics of each node. That is, to fuse the embedding of each node after the Graph Convolutional Network. We employ a function ℳ : R^N× D↪R^D:


    p^(t) = ℳ (X^(t)) = σ (1/N∑_i=1^NX_i^(t)).



p^(t) summarizes the node-level features of the graph 𝒥^(t). Among them, X^(t) is a matrix with n rows and d columns, X_i^(t) is the i-th row of matrix X^(t) and σ denotes the logistic sigmoid nonlinearity function. 

Graph-Level stage. In the graph-level stage, we concatenate the embedding of each node to build a tensor, and then flatten the tensor to get the total information of the whole graph which can form K bins. Let {x_j^*}_j = 1...n be the each center of the histogram bins. The value included in each bin is within the range of r centered on {x_j^*}. Given a certain feature value x_i, we define a function Ψ, the index Ψ(x_i^(t)) corresponding to the bin that the value belongs to. On the whole, the function 𝒢 denotes the embedding of the graph in the graph-level stage: R^N× D↪R^K:


    q̃_u^(t)  = 𝒢 (X^(t)) = ∑_i=1^N× DC(|x_i^(t)|)δ[Ψ(x_i^(t))-u]



    q^(t) = ℰ [q̃_u = 1...K^(t) ],


where δ is the Kronecker delta function. The normalization constant C is derived by imposing ∑_i=1^N ×Dx̃_i^(t) = 1, performing data smoothing on the graph-level results. The graph-level embedding result can describe the global feature of the graph. Therefore, after concatenating each graph-level stage feature using the symbol ℰ , the final graph-level embedding result is obtained: R^N× D↪R^D + K. Finally, we concatenate the node-level embedding and graph-level embedding by the axis 0.  


    G^(t) = ℰ [p̃^(t) , q̃^(t)]




 §.§ CONVERGE MULTI-GRAPH BY ATTENTION

Single-scale CNN fail to capture the intricate relationships among intestinal flora, while conventional multi-scale CNNs sometimes fall short in effectively integrating information from various scales using concatenation-based approaches.

The significance of self-attention CNNs in amalgamating multi-scale information is due to several factors. To begin with, the self-attention mechanism in CNNs improves classifying capability by selectively concentrating on significant regions or features in input data, consequently increasing classification accuracy and aiding in detecting nuanced patterns. Moreover, self-attention CNNs generate interpretable outcomes by indicating the regions that most influence the classification decision. Additionally, self-attention CNNs perform exceptionally well in managing high-dimensional data, allowing them to learn distinct features in multiple dimensions, thereby capturing intricate patterns and dependencies in gut microbiome abundance data. In general, self-attention is a notable subtype of attention mechanisms because of its ability to capture long-range dependencies, adaptively weigh input elements, manage variable-length sequences, ensure interpretability, and combine with other deep learning techniques. In this part, we utilize it to merge Original-graph and Shuffled-graphs that have been updated by Graph Convolutional Network.

After using Graph Convolutional Network to update the Original-graph and Shuffled-graphs, we get the embedding of each node. Each node in the Multi-graph 𝒥^(t) gets a feature vector to represent, and then we use  attention block to update the embeddings of the corresponding nodes <cit.> of the Multi-graph to an embedding result representing the node feature. The attention block is defined as follows:













    𝐗_attn   = 𝒜ttn(X_i^(t)| t ∈𝒯) 
       = ∑_t ∈ Texp(query^(t)X_i^(t))/∑_t' ∈ T'exp(query^(t')X_i^(t')) X_i^(t),





where query^(t)∈R^D + K is the feature vector of relation-type t. The Original Graphs and the Shuffled Graphs can obtain the embedding results x_i and x̃_i of each node after passing through the attention block. Finally, the embedding of nodes can be sent to the MLP for prediction.




 §.§ LOSS FUNCTIONS

The loss functions of our model consists of an adversarial loss ℒ_adv  and hybrid attention loss ℒ_h-attn.
The total loss function is as follows:


    ℒ = ℒ_adv + ὴℒ_h-attn,


where ὴ is a learnable coefficient for the loss terms.




  §.§.§ ADVERSARIAL LOSS

We not only use a variety of relation-types to construct Multi-graph and obtain the embedding of each Node through Graph Convolutional Network and Attention Block, but also the Original-Grpah Randomly shuffle to break the correlation among nodes. Therefore, we calculate the positive correlation loss between the global embedding obtained by NFGI (Node Feature Global Integration Descriptor) and the embedding of the nodes of the Original-Graph obtained by each relation-type. At the same time, on the converse side, we calculate the negative correlation loss between the global embedding and the embedding of the nodes of the Shuffled-Graph, and define this joint loss as Adversarial Loss inspired by <cit.>. In other words, we calculate the adversarial loss from the embedding obtained from the constructed Original-Graph (Positive) and the Shuffled-Graph (Negative) that destroys the correlation among nodes, which is defined as:






    ℒ_adv =    ∑_t ∈𝒯∑_i=1^Nlogσ((H^(t))^T W^(t)X_i^(t)) 
       + ∑_t ∈𝒯∑_j=1^Nlog(1 - σ((H^(t))^TW̃^(t)X̃_i^(t))).











  §.§.§ HYBRID ATTENTION LOSS

The Hybrid Attention Loss we proposed comprehensively considers the embedding after the nodes of the Original-Graph and the Shuffled-Graph pass through the attention block. The Hybrid Attention Loss ℒ_h-attn can make the global embedding matrix of real graph agree with X_attn^(t), and disagree with X̃_attn^(t), thereby improving the confidence of the attention block. The Hybrid Attention Loss function is defined as:


    ℒ_h-attn = (𝒫 - X_attn) ^ 2 - (𝒫 - X̃_attn) ^ 2.






§ EXPERIMENT




 §.§ SET UP

We performe extensive experiments on five real-world datasets to evaluate the performance of our UMMAN model. We also compare it to several state-of-the-art
machine learning and deep learning models. In pre-processing, we removed the intestinal flora which abundance lower than 0.01 in most hosts (set to 120). During the initialization of the graph, set the threshold θ to 0.6. We randomly divide each dataset into five folds, with four folds for the training set and one fold for the validation set, using the k-fold method for cross-validation. We randomly select five different random seeds for each validation dataset and take the average performance as the final result. We use a three-layer Graph Convolutional Network to extract the embedding for each node. In training, we use the Adam optimiser with an initial learning rate of 0.001 and epochs of 1000. We set the node embedding dimension to 256. 


 §.§ DATASETS

Five classical gut microbiota datasets are used for the experiments. We use five available disease-associated metage nomic datasets spanning four diseases: liver cirrhosis, inflammatory bowel diseases (IBD), obesity, and type 2 diabetes in Asia and Europe, respectively. These datasets have recorded the abundance of 1331 gut microbes in the sample intestine, as well as information such as gender, age, and region.

Cirrhosis <cit.>. Qin et al. extracted total DNA libraries from fecal samples of patients with cirrhosis and healthy controls, which can characterize the gut microbiome of patients with cirrhosis. Specifically, they used Illumina HiSeq 2000 for sequencing, which produced an average of 4.74 Gb of high-quality sequences per sample, and a total of 860 Gb of 16SrRNA gene sequence data.

IBD <cit.>.  Qin et al. collected stool samples from volunteers and performed Illumina GA sequencing. All Reads were assembled using Soapnovo19. Using BLAT36 to construct a non-redundant gene set for inflammatory bowel disease by pairwise comparison of all genes. Each sample yields an average of 4.5Gb of high-quality sequences.

Obesity <cit.>. Obesity is one of the most serious (proportional) gut microbiota-related diseases facing the world. Le et al. extracted the abundance of more than a thousand species of flora in the gut by sequencing fecal samples from Obesity patients and thinner stature.

T2D <cit.>. Stool samples are collected from patients with type 2 diabetes and healthy controls in order to obtain the DNA data of the flora in the samples, using the whole genome sequencing method, and then sequenced all the DNA samples, and analyzed the intestinal microbial DNA of 345 Chinese Two-stage MGWAS was performed for deep shotgun sequencing, with an average of 2.61 Gb per sample and a total of 378.4 Gb of high-quality DNA data.

WT2D <cit.>. Shotgun sequencing is used to analyze the whole genome sequence of fecal samples from European women, sequenced on Illumina HiSeq 2000, obtained an average of 2Gb of sequencing data per sample, and a total of about 449 Gb of data. Different from T2D, the target of T2D dataset is Chinese, while WT2D is sampled in Europe.




 §.§ COMPARISON WITH EXISTING WORK

We carry out quantitative comparisons on the results between our method and existing work on five datasets. The Random Forest and Support Vector Machine can be used in the disease prediction of intestinal flora <cit.>, although the results obtained are acceptable, the performance of such algorithms has reached the bottleneck so that it is difficult to improve. The basic CNN series methods are particularly effective at extracting closely located features, but they encounter difficulty in capturing attention over long distances. Additionally, due to the property of uncorrelated adjacent data in the OTU tabular dataset, traditional convolutional neural networks struggle to extract the intrinsic connections among gut microbiota. We innovatively combine a graph machine learning algorithm with the gut microbiota disease prediction task. We first compared the performance results of our method and the previous 8 methods on the Acc and AUC indicators on the benchmark datasets, as shown in [table1]Table 1 (bold indicates the best performance, underline indicates the second-best performance).




















Our method, UMMAN, achieves the best performance on the metrics across the five datasets on all indicators. From the experimental results in [table1]Table 1 and [figure3]Fig .3, it can be found that almost all the optimal and second-best results are derived from our method and Random Forest. This may be due to the robustness of traditional machine learning algorithms. Therefore, in order to prove that our method (UMMAN) not only achieves SOTA on the result value, but also surpasses the previous methods in all aspects in terms of stability and other indicators. [table2]Table 2 compares the performance of our method and traditional machine learning algorithms on five datasets in detail. We use K-fold cross-validation on five datasets of Cirrhosis, IBD, Obesity, T2D, WT2D, and use Accuracy, Precision, Recall, F1-Score, AUC indicators to compare with previous methods in an all-round way. [figure4]Fig .4 shows the visual comparison results of the four variants of the ablation experiment on five gut microbiome datasets. Each line represents a contour line, and the horizontal and vertical coordinates represent Accuracy and AUC metrics, respectively. In short, the method closer to the upper right region has better performance. The ablation experiments show that our method achieves the best results in almost all results, and the stability performs well, with minimal fluctuations.







 



























 







 §.§ ABLATION STUDY

In our method, Attention block, Node Feature Global Integration (NFGI) descriptor and Adversarial loss are three core components, which is to improve the performance on the OTU datasets. We conduct an ablation study on eight variants: a) Ours (Base), only with the framework of the UMMAN model; b) Ours (Adv), with Adversarial loss which is used to extract the features of the Original-Graph; c) Ours (Attn), with Attention block which is used to get the embedding of the Original-Graph and the Shuffled-Graph. In this study, we use the global average pooling of Multi-Graphs to replace Attention block, the global average pooling is defined as:

    𝐯 = GAP(ϕ_θ(𝐗))

d) Ours (NFGI), adopting Node Feature Global Integration(NFGI) which is designed to describe the global graph embedding, replace this module with the average value of each node in this part; e) Ours (Attn+Adv), with Attention block and Adversarial loss; f) Ours (NFGI+Adv), with NFGI and Adversarial loss; g) Ours (Attn+NFGI), with Attention block and NFGI. h) Ours (Full), with Attention block,  NFGI and Adversarial loss. The numeric comparisons on Cirrhosis, IBD, Obesity, T2D, WT2D are shown in [table3]Table 3. On the whole, the method with Attention block,  NFGI and Adversarial loss, i.e., Ours (Full) performs the best. It is worth mentioning that even if our method is ablated, the effect on most indicators is better than previous work.










§ CONCLUSION

In this paper, we propose a novel architecture UMMAN which combines GNN with disease prediction tasks based on intestinal flora for the first time. It helps learn the multiplex connection among gut microbes of different hosts. We construct the Multi-Graph and Shuffled-Graph with multiple relation-types, get the embedding of hosts through improved GCN. In addition, we introduce the Node Feature Global Integration (NFGI) to describe the global embedding of a graph with node-level stage and graph-level stage. Finally, we design a joint loss consisting of adversarial loss and hybrid attention loss as the final loss function. Extensive experiments show that our UMMAN performs well on the task of disease prediction based on intestinal flora and achieves the state-of-the-art. Our UMMAN model can be applied to assist in the diagnosis of gut microbiome-related diseases.

In the future, the main work may be to find a more suitable way to construct a gut microbiota, in order to avoid building microbial interaction networks incorrectly, which could negatively impact the accuracy of subsequent graph classification tasks. Lastly, the biggest challenge in this study is the data, as there are currently relatively few publicly available gut microbiome datasets for disease classification. However, with the development of sequencing technology, more datasets that can reflect sample features may emerge in the future. At that time, the model's performance may improve after extensive training.
























































































 






 
















































































































































































































































































































































































 
 






1
IEEEtran
































beals1984bray
E. W. Beals, “Bray-Curtis ordination: an effective strategy for analysis of multivariate ecological data,” in Advances in Ecological Research, vol. 14, pp. 1–55, Elsevier, 1984.

breiman2001random
L. Breiman, “Random forests,” Machine Learning, vol. 45, pp. 5–32, 2001.


cheng2023gnndelete
J. Cheng, G. Dasoulas, H. He, C. Agarwal, and M. Zitnik, “GNNDelete: A General Strategy for Unlearning in Graph Neural Networks,” arXiv preprint arXiv:2302.13406, 2023.

clemente2012impact
J. C. Clemente, L. K. Ursell, L. W. Parfrey, and R. Knight, “The impact of the gut microbiota on human health: an integrative view,” Cell, vol. 148, no. 6, pp. 1258–1270, 2012.


dethlefsen2008pervasive
L. Dethlefsen, S. Huse, M. L. Sogin, and D. A. Relman, “The pervasive effects of an antibiotic on the human gut microbiota, as revealed by deep 16S rRNA sequencing,” PLoS Biology, vol. 6, no. 11, p. e280, 2008.

dicksved2008molecular
J. Dicksved, J. Halfvarson, M. Rosenquist, G. Järnerot, C. Tysk, J. Apajalahti, L. Engstrand, and J. K. Jansson, “Molecular analysis of the gut microbiota of identical twins with Crohn's disease,” The ISME Journal, vol. 2, no. 7, pp. 716–727, 2008.


edgar2013uparse
R. C. Edgar, “UPARSE: highly accurate OTU sequences from microbial amplicon reads,” Nature Methods, vol. 10, no. 10, pp. 996–998, 2013.


frank2007molecular
D. N. Frank, A. L. St. Amand, R. A. Feldman, E. C. Boedeker, N. Harpaz, and N. R. Pace, “Molecular-phylogenetic characterization of microbial community imbalances in human inflammatory bowel diseases,” Proceedings of the National Academy of Sciences, vol. 104, no. 34, pp. 13780–13785, 2007.


hamilton2017inductive
W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” Advances in neural information processing systems, vol. 30, 2017.


jurman2009canberra
G. Jurman, S. Riccadonna, R. Visintainer, and C. Furlanello, “Canberra distance on ranked lists,” in Proceedings of Advances in Ranking NIPS 09 Workshop, pp. 22–27, Citeseer, 2009.


karlsson2013gut
F. H. Karlsson et al., “Gut metagenome in European women with normal, impaired and diabetic glucose control,” Nature, vol. 498, no. 7452, pp. 99–103, 2013.

kipf2016semi
T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” arXiv preprint arXiv:1609.02907, 2016.

krizhevsky2012imagenet
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems, vol. 25, 2012.


le2013richness
E. Le Chatelier et al., “Richness of human gut microbiome correlates with metabolic markers,” Nature, vol. 500, no. 7464, pp. 541–546, 2013.

ley2006human
R. E. Ley, P. J. Turnbaugh, S. Klein, and J. I. Gordon, “Human gut microbes associated with obesity,” Nature, vol. 444, no. 7122, pp. 1022–1023, 2006.

linares2021machine
J. Liñares Blanco, C. Fernández-Lozano, J. A. Seoane Fernández, and G. Lopez-Campos, “Machine Learning Algorithms Reveals Country-Specific Metagenomic Taxa from American Gut Project Data,” Studies in Health Technology and Informatics, vol. 281, pp. 382–386, 2021.

liu2020study
W. Liu, R. Zhang, R. Shu, J. Yu, H. Li, H. Long, S. Jin, S. Li, Q. Hu, F. Yao, and others, “Study of the relationship between microbiome and colorectal cancer susceptibility using 16SrRNA sequencing,” BioMed Research International, vol. 2020, 2020.


manandhar2021machine
I. Manandhar, A. Alimadadi, S. Aryal, P. Munroe, B. Joe, and X. Cheng, “Machine Learning of Gut Microbiome Composition for Diagnostic Classification of Inflammatory Bowel Diseases,” The FASEB Journal, vol. 35, 2021.

manichanh2006reduced
C. Manichanh, L. Rigottier-Gois, E. Bonnaud, K. Gloux, E. Pelletier, L. Frangeul, R. Nalin, C. Jarrin, P. Chardon, P. Marteau, and others, “Reduced diversity of faecal microbiota in Crohn’s disease revealed by a metagenomic approach,” Gut, vol. 55, no. 2, pp. 205–211, 2006.

mcdowell2022machine
A. McDowell, J. Kang, J. Yang, J. Jung, Y.-M. Oh, S.-M. Kym, T.-S. Shin, T.-B. Kim, Y.-K. Jee, and Y.-K. Kim, “Machine-learning algorithms for asthma, COPD, and lung cancer risk assessment using circulating microbial extracellular vesicle data and their application to assess dietary effects,” Experimental & Molecular Medicine, vol. 54, no. 9, pp. 1586–1595, 2022.


pajot2018unsupervised
A. Pajot, E. De Bézenac, and P. Gallinari, “Unsupervised adversarial image reconstruction,” in International Conference on Learning Representations, 2018.

pasolli2016machine
E. Pasolli, D. T. Truong, F. Malik, L. Waldron, and N. Segata, “Machine learning meta-analysis of large metagenomic datasets: tools and biological insights,” PLoS Computational Biology, vol. 12, no. 7, p. e1004977, 2016.




qin2010human
J. Qin et al., “A human gut microbial gene catalogue established by metagenomic sequencing,” Nature, vol. 464, no. 7285, pp. 59–65, 2010.

qin2012metagenome
J. Qin et al., “A metagenome-wide association study of gut microbiota in type 2 diabetes,” Nature, vol. 490, no. 7418, pp. 55–60, 2012.

qin2014alterations
N. Qin et al., “Alterations of the human gut microbiome in liver cirrhosis,” Nature, vol. 513, no. 7516, pp. 59–64, 2014.


rumelhart1986learning
D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, no. 6088, pp. 533–536, 1986.


sharma2020taxonn
D. Sharma, A. D. Paterson, and W. Xu, “TaxoNN: ensemble of neural networks on stratified microbiome data for disease prediction,” Bioinformatics, vol. 36, no. 17, pp. 4544–4550, 2020.

song2021review
Q. Song, Y. Wang, L. Huang, M. Shen, Y. Yu, Q. Yu, Y. Chen, and J. Xie, “Review of the relationships among polysaccharides, gut microbiota, and human health,” Food Research International, vol. 140, p. 109858, 2021.


tana2010altered
C. Tana, Y. Umesaki, A. Imaoka, T. Handa, M. Kanazawa, and S. Fukudo, “Altered profiles of intestinal microbiota and organic acids may be the origin of symptoms in irritable bowel syndrome,” Neurogastroenterology & Motility, vol. 22, no. 5, pp. 512–e115, 2010.

tena2022explainable
DJ Tena Cucala, B Cuenca Grau, EV Kostylev, and B Motik, “Explainable GNN-based models over knowledge graphs,” 2022, OpenReview.


vaswani2017attention
A. Vaswani et al., “Attention is all you need,” Advances in Neural Information Processing Systems, vol. 30, 2017.


wong2021analysis
C. W. Wong, S. E. Yost, J. S. Lee, J. D. Gillece, M. Folkerts, L. Reining, S. K. Highlander, Z. Eftekhari, J. Mortimer, and Y. Yuan, “Analysis of gut microbiome using explainable machine learning predicts risk of diarrhea associated with tyrosine kinase inhibitor neratinib: a pilot study,” Frontiers in Oncology, vol. 11, p. 604584, 2021.


xia2021graph
F. Xia, K. Sun, S. Yu, A. Aziz, L. Wan, S. Pan, and H. Liu, “Graph learning: A survey,” IEEE Transactions on Artificial Intelligence, vol. 2, no. 2, p. 109-127, 2021.


zhou2019review
Y.-H. Zhou and P. Gallins, “A review and tutorial of machine learning methods for microbiome host trait prediction,” Frontiers in Genetics, vol. 10, p. 579, 2019.

tuan2021gnn
Nguyen, Tuan et al., “Graph convolutional networks for drug response prediction,” IEEE/ACM transactions on computational biology and bioinformatics vol. 19, p. 146-154, 2021.

wang2024gnn
W. Wang et al., “LMGATCDA: Graph neural network with labeling trick for predicting circRNA-disease associations,“ IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2024.

sender2016estimates
R. Sender, S. Fuchs and R. Milo, “Revised estimates for the number of human and bacteria cells in the body,” PLoS Biology, vol. 14, no. 8, p. e1002533, 2016.
































 






















 Paper Title: A Scalable Tool For Analyzing Genomic Variants Of Humans Using Knowledge Graphs and Machine Learning.


[
    S.A.R. Horsley
    August 5, 2024
==================



§ ABSTRACT 
    The integration of knowledge graphs and graph machine learning (GML) in genomic data analysis offers several opportunities for understanding complex genetic relationships, especially at the RNA level. We present a comprehensive approach for leveraging these technologies to analyze genomic variants, specifically in the context of RNA sequencing (RNA-seq) data from COVID-19 patient samples. The proposed method involves extracting variant-level genetic information, annotating the data with additional metadata using SnpEff, and converting the enriched Variant Call Format (VCF) files into Resource Description Framework (RDF) triples. The resulting knowledge graph is further enhanced with patient metadata and stored in a graph database, facilitating efficient querying and indexing. We utilize the Deep Graph Library (DGL) to perform graph machine learning tasks, including node classification with GraphSAGE and Graph Convolutional Networks (GCNs). Our approach demonstrates significant utility using our proposed tool, VariantKG, in three key scenarios: enriching graphs with new VCF data, creating subgraphs based on user-defined features, and conducting graph machine learning for node classification.






§ INTRODUCTION



<cit.> aimed to sequence the entire human genome, resulting in an official gene map. The gene map has offered crucial insights into the human blueprint, accelerating the study of human biology and advancements in medical practices. This information has been represented in Variant Calling Format (VCF) files that store small-scale information or genetic variation data. 

Variants are genetic differences between healthy and diseased tissues or between individuals of a population. Analyzing variants can tremendously help prevent, diagnose, and even treat diseases. The process of analyzing these genetic differences or variations in DNA sequences and categorizing their functional significance is called variant analysis. RNA sequencing is similar to DNA sequencing but differs in its extraction. RNA is extracted from a sample and then reverse transcribed to produce what is known as copy or complementary DNA called cDNA. This cDNA is then fragmented and run through a next-gen sequencing system. Examining DNA provides a static picture of what a cell or an organism might do, but measuring RNA tells us precisely what the cell or organism is doing. Another advantage of RNA sequencing is that molecular features sometimes can only be observed at the RNA level.

Variant calling pipeline is the process of identifying variants from sequence data. To measure the deleteriousness of a variant, the Combined Annotation Dependent Depletion (CADD) <cit.> scores tool is used. CADD evaluates or ranks the deleteriousness of a single nucleotide, insertion, and deletion variants in the human genome. The COVID-19 genetic data discussed in this paper was collected from the European Nucleotide Archive (ENA) <cit.>. 

Knowledge graphs such as YAGO <cit.>, Wikidata <cit.>, DBPedia <cit.>, and Schema.org <cit.> are crucial for structuring and linking vast amounts of diverse data, to enable efficient information retrieval, enhance data interoperability, and provide a foundation for advanced applications in domains not limited to semantic search, natural language processing, and data integration. One such example is the work of <cit.> which focuses on constructing a comprehensive knowledge graph (KG) called ProductKG from Amazon's large-scale and diverse product catalog. ProductKG captures product attributes, categories, and relationships using graph mining and embedding techniques. This structured representation aims to improve understanding and retrieval of product information, thereby enhancing user experience and supporting various AI applications within Amazon's ecosystem. Another such example of KGs is FoodKG, introduced by <cit.> that demonstrates the importance of knowledge graphs in the Food domain through their tool, FoodKG, by integrating diverse datasets, using NLP to extract meaningful entities and state-of-the-art model to enhance and enrich graphs based on Food, Energy and Water (FEW) used by the tool. Their contribution highlights the significant role of knowledge graphs in managing and utilizing large-scale, heterogeneous data.  

Representing genomic data as knowledge graphs allows vast and diverse information from various sources to be integrated. These specialized graph structures, which model entities as nodes and relationships as edges, provide an ideal framework for integrating and organizing diverse biological information from multiple sources. Furthermore, it allows for efficient querying and indexing and supports inference and new knowledge discovery.

The key contributions of this work are:


    
  * We introduce a data collection pipeline to extract variant-level genetic information from RNA-seq IDs using the ENA browser. We also discuss adding additional information to the VCF files using the SnpEff tool.
    
  * We elaborate on the construction and enrichment of our knowledge graph. We utilize the vcf2rdf tool from Sparqling-genomics for the VCF files, in addition to our explicitly-defined ontologies. We translate the RDF-triples into an efficient and easily parsable format (NQuads). 
    
  * We demonstrate the use of our tool, VariantKG and present three scenarios which are Graph Enrichment to consume new VCF information or use the existing data from our knowledge base, Graph Creation to create subgraphs with user-defined features and Graph Machine Learning and Inference for node classification tasks using GraphSAGE and Graph Convolutional Network supported by Deep Graph Library.


The remainder of the paper is organized as follows: <ref> explores the previous works that integrate knowledge graphs with the vast genomic data; <ref> discusses the data collection pipeline, construction of our knowledge graph (KG) using ontology and enrichment of the KG with patient metadata; <ref> introduces the graph storage and database used in this research work and tool; <ref> discusses the use of Deep Graph Library (DGL) for node classification tasks and introduces our tool, VariantKG in which we demonstrate 3 scenarios, namely how a user can use data from our existing knowledge base, add new VCFs to enrich the dataset, and use a combination of the added files and the existing knowledge base to perform node classification tasks using two Graph Neural Networks - GraphSAGE and Graph Convolutional Networks (GCNs), using our knowledge graph database for graph machine learning tasks; <ref> concludes our work.



§ RELATED WORK



Knowledge graphs are widely used to integrate and analyze diverse genomic data, providing a comprehensive and contextual representation of genomic information. For instance, <cit.> extended the development of GenomicKB by creating a graph database that integrates human genome, epigenome, transcriptome, and 4D nucleome data. This extensive database, annotated from over 30 consortia and portals, includes 347 million entities, 1.36 billion relations, and 3.9 billion properties, covering comprehensive data on pancreases and diabetes' GWAS, disease ontology, and eQTL data. Another notable work, <cit.>, presented a knowledge graph, GenomicKB, that consolidates various sources of genomic information, including data from genomic databases, experimental studies, literature, and public repositories, into a single, unified framework. This integration facilitates efficient data analysis and knowledge discovery through a user-friendly web portal.

Knowledge graphs have been instrumental in understanding the COVID-19 virus and its treatment. For example, <cit.> developed a framework that integrates diverse COVID-19 drug resources to discover drug-drug interactions among COVID-19 treatments, utilizing RDF mapping language and Natural Language Processing (NLP) to extract relevant entities and relationships. Similarly, <cit.> proposed KG-COVID-19, a knowledge graph framework that integrates heterogeneous data on SARS-CoV-2 and related viruses, supporting downstream tasks such as machine learning, hypothesis-based querying, and user interface exploration. <cit.> used semantic web technology RDF to integrate COVID-19 data extracted from iTextMine, PubTator, and SemRep biological databases into a standardized Knowledge Graph (KG). This COVID-19 KG supports federated queries on the semantic web and is accessible through browsing and searching web interfaces, with a RESTful API for programmatic access and RDF file downloads.

Deep learning has significantly influenced a wide range of domains, with genomic studies being particularly impacted. For instance, <cit.> introduced DeepCDR, a method using deep learning to predict cancer cells' response to different drugs, facilitating effective cancer treatment. Another innovative approach was proposed by <cit.>, who developed ChromeGCN for predicting epigenetic states using sequences and 3D genome data. ChromeGCN leverages graph convolutional networks (GCNs) to predict the epigenetic states of genomic regions, representing genomes as graphs where nodes are genomic regions and edges represent relationships between them. This method's predictive power enables the identification of functional genomic elements and regulatory regions, providing insights into gene regulation and cellular function.

Additionally, <cit.> proposed constructing knowledge graphs from clinical data using the BERT (Bidirectional Encoder Representations from Transformers) model. This approach focused on creating biomedical knowledge graphs by leveraging BERT's contextual understanding capabilities to process biomedical text data, including clinical records and scientific literature, extracting meaningful and contextually rich information. <cit.> developed a multi-modal cause-and-effect COVID-19 knowledge model using Biological Expression Language (BEL) as a triple (i.e., source node-relation-target node) with metadata about nodes. This model utilized GraphML, NDEx, and SIF representations for network visualization and was made accessible through a web platform to enhance its visibility and utility.

Lastly, <cit.> focused on extracting and utilizing knowledge from COVID-19-related news articles, providing a platform for researchers, data analysts, and data scientists to investigate and recommend strategies to address global challenges. <cit.> proposed KGBSVM (Kernelized Generalized Bayesian Rule Mining with Support Vector Machines), a method to analyze high-dimensional genome data, aiming to improve classification accuracy on general tasks, whether binary or multi-class. This method enhances the accuracy and efficiency of genomic data classification, contributing to better data analysis and interpretation in the field.

These diverse efforts highlight the significant potential of integrating knowledge graphs with genomic data and deep learning, facilitating comprehensive data integration, efficient analysis, and innovative solutions to complex problems in genomics and beyond.



§ KNOWLEDGE GRAPH CONSTRUCTION



In this section, we explain how to construct knowledge graphs. First, we describe how the data was collected and the workflow processes. We also incorporate the patient metadata used to enhance the knowledge base. Then, we highlight the process of annotating the data, followed by the ontology description. Lastly, we discuss the representation of VCF files and CADD Scores in a knowledge graph.



 §.§ Data Collection



COVID-19 RNA sequence IDs were first collected from the European Nucleotide Archive. A total of 3,716 VCF files were collected till March 2023. The workflow has been shown in Figure <ref>.

 


    
  * FASTQ files (part A): These IDs were utilized to download the RNA sequences, which were in FASTQ <cit.> format. The FASTQ file format consists of a series of records, each of which contains four lines of text: the first line starting with `@' contains a sequence identifier, the second line contains the actual nucleotide sequence, the third line starts with `+' and may optionally contain additional information about the sequence, and the fourth line contains quality scores encoded as ASCII 10 characters. The quality scores indicate the confidence in the accuracy of each base call and are typically represented as Phred scores.
    
  * uBAM files (part B): The FASTQ files were then converted to unmapped BAM (uBAM) files for storing aligned sequencing data. A uBAM file contains unmapped reads, meaning reads that could not be confidently aligned to a reference genome. These reads can be used for downstream analysis, such as de novo assembly, quality control, and identification of novel sequences.
    
  * GATK workflow (part C): The uBAM files were passed through the Genomic Analysis Toolkit (GATK) workflow <cit.> that converts the files into Variant Calling Format (VCF) <cit.> files. It is a comprehensive toolkit developed by the Broad Institute that includes various tools and algorithms for processing genomic data, such as read mapping, local realignment, base quality score recalibration, variant calling, and variant filtering. 
    
  * The unannotated VCF files that were obtained as the result of the workflow have been shown in part D. For each VCF file, there is also a corresponding CADD Scores file that was obtained using GrCh37 through the workflow, as shown in part E.




 §.§ Data Annotations



Once the RNA sequencing was completed, two main files were generated for each RNA-seq ID – a VCF file and a CADD Scores file.

For further annotations, SnpEff <cit.>, a command-line, variant annotation, and effect prediction tool, was utilized. This tool annotates and predicts the effects of genetic variants. SnpEff takes the predicted variants (SNPs, insertions, deletion, and MNPs) as input and produces a file with annotations of the variants and the effects they produce on known genes. 

SnpEff classifies variants as single nucleotide polymorphisms (SNPs), insertions, deletions, multiple-nucleotide polymorphisms, or an InDel. While the original VCF file contains the INFO field, SnpEff adds additional annotations to this field to describe each variant further. In the process, it also updates the header fields. This field is tagged by `ANN', which is pipe symbol separated and provides a summary of the predicted effects of a genetic variant on each affected transcript. Figure <ref> shows the ANN field highlighted in bold.

 

A variant may have one or more annotations and multiple annotations are comma-separated. There are several fields within the ANN tag, mainly:

    
  * Allele (ALT): Information on alternate alleles that are predicted to cause a functional impact on a gene or protein
    
  * Annotation (effect): Type of effect caused by the variant on the transcript
    
  * Putative impact: Qualitative measure of the impact of the variant on the transcript
    
  * Gene name: Name of the affected gene
    
  * Gene ID: Unique identifier of the affected gene




 §.§ Metadata 



The metadata was downloaded from the SRA [<(https://www.ncbi.nlm.nih.gov/sra)>] web tool of NCBI. The metadata runs for each patient are stored in the Study section. We leveraged the SRA web tool over entrez to extract a wider range of metadata information such as disease, age, fatality. The obtained metadata was then converted to NQ triples to facilitate the use of named graph to link this information to the variant information obtained, as shown in Section <ref>. An example of the metadata obtained for the accessionID SRR12570493 is as follows:


SRR12570493,65 (Age),,RNA-Seq,119,1473875214,PRJNA661032,SAMN15967295,582422941,,GEO,Severe COVID,Alive,public,"fastq,sra,run.zq","gs,ncbi,s3","s3.us-east-1,gs.US,ncbi.public",SRX9058173,NextSeq 500,GSM4762164,,PAIRED,cDNA,Homo sapiens,TRANSCRIPTOMIC,ILLUMINA,2021-01-27T00:00:00Z,,2020-09-02T12:06:00Z,1,GSM4762164,male,6,Patient 14 blood,SRP279746,Patient 14,Blood,,,,,,,,,,,,


The NQ triple for the age attribute for the same accessionID run is as follows:


   <https://www.ncbi.nlm.nih.gov/sra/?term=SRR12570589> <https://www.wikidata.org/wiki/Q11904283> "61.0"^^<http://www.w3.org/2001/XMLSchema#float> <sg://SRR12570589> .




 §.§ Ontology



 

A knowledge graph is represented using an ontology, which can be represented using a formal language such as RDF (Resource Description Framework), OWL (Web Ontology Language), or another domain-specific language. The ontology in this work has been represented using RDF. Each node-edge-node is represented as a triple by RDF. In a triple, the subject defines the first node, and the object defines the second node. The predicate defines the edge or relation joining the two nodes. A triple always ends with a dot. 

An ontology mainly consists of classes, properties, and relationships. In Figure <ref>, the sub-classes are depicted by red nodes, the classes are by yellow nodes, and the relations by blue nodes. The description of the classes has been given in Table <ref> and the description of the properties has been given in Table <ref>.





In the defined ontology, chromosome and variant are both domain classes, and a chromosome has an associated chromosome number to connect all similar chromosomes as an extension, and a variant has an associated variant ID. A variant has a reference and alternate genome. 

The ontology also explicitly defines CADD as a class where a variant has CADD Scores represented by both raw score and Phred-scale score, as properties of the CADD class. The ontology description has been given in Table <ref>.





 §.§ Conversion of VCF files to Knowledge Graphs



To transform the data in VCF, SPARQLing Genomics <cit.> was utilized. SPARQLing Genomics is an open-source platform for querying and analyzing genomic data using the Semantic Web and Linked Data technologies. The platform provides an easy-to-use interface as well, that has been built to support SPARQL queries and various SPARQL query features, including sub-queries, filters, and aggregates. SPARQLing Genomics provides several in-built, ready-to-use tools, one of which is vcf2rdf that converts VCF data into RDF triples. 

The triples generated by the tool consist of uniquely identifiable names with symbolic and literal values like numbers or text. 


    #CHROM POS   ID REF ALT QUAL    FILTER    INFO sample
       1  16963  .   G   A  45.64 SnpCluster AC=1;AF=0.500;AN=2;BaseQRankSum=1.465;DP-8;ExcessHet=3.0103;FS=0.000;MLEAC=1;MLEAF=0.500;MQ=60.00;MQRankSum=0.000;QD=5.70;ReadPosRankSum=-0.366;SOR=0.169GT:AD:D:GQ:PL0/1:6,2:8:53:53,0,228


The following is an example of how a variant position (example shown above) is translated into a triple by the tool.


    <origin://4a37140cdc877d90ffe258a8151f27e@0> <http://biohackathon.org/resource/faldo#position> "16963"^^<http://www.w3.org/2001/XMLSchema#integer> .


As seen in the above example, the variant position is described with Feature Annotation Location Description Ontology (FALDO) <cit.>. For other features not defined by FALDO, the URI is customized to the tool.

Each VCF file eventually corresponds to one large knowledge graph that was originally stored in an N3 format. N3 format is one of several formats supported by RDF and can be considered a shorthand non-XML serialization of RDF models. However, to accommodate the accessionID that would map to an unidentified patient, the N3 serialization was converted to NQ format with the accessionID as the named graph. An example of a triple from an N3 file has been given here.


    <origin://4a37140cdc877d90ffe2b58a8151f27e@0> <sg://0.99.11/vcf2rdf/variant/REF> <sg://0.99.11/vcf2rdf/sequence/G>  .


The triple was then converted to NQ format, which yielded the following triple:


    <origin://4a37140cdc877d90ffe2b58a8151f27e@0>   <sg://0.99.11/vcf2rdf/variant/REF> <sg://0.99.11/vcf2rdf/sequence/G> <sg://SRR13112995>.


The ontology was extended to accommodate the new relations generated by the tool. 



 §.§ Conversion of CADD Score files to Knowledge Graphs



The SnpEff and vcf2rdf tools were useful for converting VCF files to triples. However, the CADD Scores obtained through the pipeline were in tab-separated (TSV) format. To enrich the knowledge graphs, the CADD Scores had to be translated to RDF triples as well. Therefore, the ontology for CADD Scores was explicitly defined. 

To visualize the graph, GraphDB has been utilized, and the ontology that was defined for CADD Scores has been shown in Figure <ref>



These scores have been represented with respect to the fields described in the VCF files, such as the chromosome, position, reference genome, and alternate genome. The raw scores and Phred-scale scores were obtained from the original TSV files.

The following is an example of a record in a TSV file for which the raw and phred scores map to chromosome 1 with position 16963, reference genome `G' and alternate genome `A' in the VCF file:



Each data record, like the above example, was converted to a Turtle triple (TTL), another format supported by RDF. A TTL format writes a graph in a compact textual form. There are only 3 parts to this triple – subject, predicate, and object. An example of the above data record converted to a TTL triple is given below:


   <http://sg.org/SRR13112995/1/variant1> a ns1:variant ;
        ns1:has_alt_genome "A" ;
        ns1:has_cadd_scores <http://sg.org/SRR13112995/1/variant1/cadd> ;
        ns1:has_pos 16963 ;
        ns1:has_ref_genome "G" .




Each VCF file eventually corresponds to one large knowledge graph that was originally stored in an N3 format. N3 format is one of several formats supported by RDF and can be considered a shorthand non-XML serialization of RDF models. However, to accommodate the accessionID that would map to an unidentified patient, the N3 serialization was converted to NQ format with the accessionID as the named graph. An example of a triple from an N3 file has been given here.


    <origin://4a37140cdc877d90ffe2b58a8151f27e@0> <sg://0.99.11/vcf2rdf/variant/REF> <sg://0.99.11/vcf2rdf/sequence/G>  .


The triple was then converted to NQ format, which resulted in the following triple:


    <origin://4a37140cdc877d90ffe2b58a8151f27e@0>   <sg://0.99.11/vcf2rdf/variant/REF> <sg://0.99.11/vcf2rdf/sequence/G> <sg://SRR13112995>.


The ontology was extended to accommodate the new relations generated by the tool. 



§ GRAPH STORAGE AND DATABASE



Each variant file represented a single knowledge graph, so to unify several knowledge graphs into one single large graph, BlazeGraph <cit.> has been leveraged. This large knowledge graph was then queried to create a dataset for a case study. It is important to note that the edges are homogeneous in nature.

BlazeGraph is a high-performance, horizontally scalable, and open-source graph database that can be used to store and manage large-scale graph data. It has been designed to provide efficient graph querying and supports RDF data model that allows it to store and process both structured and semi-structured data. BlazeGraph uses a distributed architecture that can be easily integrated with other big data tools, such as Hadoop and Spark, to perform complex analytics on large-scale graph data.

BlazeGraph has been leveraged for efficiently querying the knowledge graphs to generate the dataset for Graph Neural Network downstream tasks. Other tools such as RIQ <cit.> can be used to index and query RDF-named graphs.

The total number of triples in the knowledge graph, after aggregating only 511 VCF files on a single machine, is as large as 3.1 Billion.



§ KG INFERENCE: CASE STUDY



In this section, we will discuss the usage of the knowledge graph created and demonstrate the use of VariantKG. We aim to use the KG for a classification task using graph machine learning. For the classification task, we are leveraging the open-source graph-based library called Deep Graph Library <cit.>.

We packaged the several components discussed above in a user-friendly, robust and efficient tool called VariantKG, to empower users to train their own GNN models using our data or a combination of our data and new VCF files. We will further expand on the workings of our tool in Section <ref>.



 §.§ Deep Graph Library



To implement the task, Deep Graph Library (DGL), an open-source library supporting graph-based deep learning was utilized. DGL provides a set of high-level APIs for building scalable and efficient graph neural network models. With DGL, we can create, manipulate, and learn from large-scale graphs with billions of nodes and edges.

There are three main tasks supported by DGL:

    
  * Node Classification: Predict the class or label of a node in a graph based on its features.
    
  * Link Prediction: Predict if there is a connection or an edge between two nodes.
    
  * Graph Classification: Classify an entire graph into one or more classes or categories.


DGL represents a graph as a DGLGraph object, which is a framework-specific graph object. It requires the number of nodes and a list of source and destination nodes, where nodes and edges must have consecutive IDs starting from 0. Since DGL only accepts numeric input, all strings such as URI were mapped to integers. In this case study, node classification was used to classify variants into CADD Score categories based on their features.



 §.§ Node Classification Task



For this task, Graph Convolutional Network (GCN) <cit.> and GraphSAGE <cit.> have been used. For both models, each node is associated with a feature vector.

GCNs use node embeddings and adjacency matrices to compute new embeddings while training. Similar to CNN, the model weights and biases are first initialized to 1, and then a section of the graph is passed through the model. A non-linear activation function is used to compute predicted node embeddings for each node. Cross entropy loss is calculated to quantify the difference between the predicted node embeddings and the ground truth. Loss gradients are then computed to update the model using the ADAM optimizer<cit.> for this task. These steps are repeated until convergence.

GraphSAGE uses SAGEConv layers where for every iteration, the output of the model involves finding new node representation for every node in the graph. Mean is used as the aggregation function, and the ReLU activation function has been utilized. ADAM optimizer was used for this model as well. One of the most noted properties of GraphSAGE is its ability to aggregate neighbor node embeddings for a given target node. Through the experiments conducted, this property was observed. GraphSAGE also generalizes better to unseen nodes because of its ability to perform inductive learning on graphs. 





The architecture of GCN has been shown in Figure <ref>. The architecture of GraphSAGE, shown in Figure <ref>, differs in the property of message passing between the nodes. This was crucial as the nodes in the input graph relied on several pieces of information from their neighboring nodes.



 §.§ Experiments & Results





  §.§.§ Experiment Setup



The experiments were run on CloudLab <cit.>, a testbed for cloud computing research and new applications. CloudLab cluster at Clemson University was built in partnership with Dell. Clemson machines were used to carry out the standalone training experiments. They were carried out on nodes with 16 cores per node (2 CPUs), 12x4 TB disk drives in each node, plus 8x1 TB disks in each node. The nodes are configured with 256 GB of memory and 73TB of RAM. The operating system installed across all of them was Ubuntu 18.04.

Wisconsin machines were also used to carry out the standalone training experiments described in section 6.2. The CloudLab cluster at the University of Wisconsin was built in partnership with Cisco, Seagate, and HP. The cluster has 523 servers with a total of 10,060 cores and 1,396 TB of storage, including SSDs on every node. 

The nodes were chosen upon availability for the standalone node classification tasks.



 §.§ VariantKG as a tool



The architecture of our tool, shown in Figure <ref> has been designed to facilitate the extraction, processing, and analysis of variant-level genomic data using GNNs. The workflow integrates custom data inputs, feature selection, storage in a graph database, graph creation, and model training for inferencing genomic information. Given the rapidly expanding repository of genomic data, VariantKG offers a robust platform for researchers or users to extract relevant information and train Graph Neural Networks (GNNs) such as GraphSAGE for inferential purposes. Users have the flexibility to upload one or more VCF files, which may include associated CADD Scores for specific variants. Alternatively, users can select from pre-existing datasets that have been structured into a large-scale, comprehensive knowledge graph.



The first part of our tool, shown as A: Graph Enrichment handles data upload and pre-processing, where users can upload one or more VCF with the associated CADD Scores or select from the pre-existing database that contains genomic variants and annotations. If the user chooses to upload the files, the VCF and CADD Scores files are processed using SnpEff to provide additional variant-level annotation and SPARQLing-Genomics for transforming the data into RDF format using the vcf2rdf tool. The processed data is then stored along with the existing data on Blazegraph, which is a scalable graph database. The data from the VCF files is stored as NQ RDF-triples, and the data from the CADD Scores files is stored as TTL (turtle) RDF-triples. This data is essentially incorporated into our existing knowledge base, enriching the KG with new information. The ontology defined for the knowledge graphs allows for this extensibility. Users also have the option to select the accession IDs corresponding to de-identified patients from our existing large-scale knowledge graph. This feature has been provided by utilized an efficient, well-formed SPARQL query that fetches all the unique accession IDs that are available in the knowledge graph. This will be further discussed in <ref>

The second part in the workflow of our tool, shown as B: Graph Creation is the availability of feature selection and creation of the subgraph or graph suitable for model training. The user can select the features that are available in the NQ VCF files, which are passed through a well-formed SPARQL query that retrieves the selected features. The query results are temporarily stored in a columnar storage format optimized for analytical queries. The user can then assign a feature as the class label, facilitating supervised learning tasks such as node classification. Once the user selects one or more accession IDs from the knowledge graph, the user will then have an option to select the features based on the patient's variant-level information. The data along with the selected features and class labels is used to create a Deep Graph Library (DGL)-specific DGLGraph object that represents the graph as integers. This is essential as DGL necessitates that the input data be formatted into a frame-work specific graph object. This will be further discussed in <ref>

Once the graph has been constructed, users can then train the DGL GNN model, GraphSAGE and gain a better understanding of the model performance through inferencing, shown as C: Graph ML Training & Inference. As of now, we support only GraphSAGE but intend to extend the support to other GNN models in the future. The training process can be customized based on user-defined model hyperparameters. Existing data that have been prestructured as DGL graphs can also be loaded directly for model training. The user can observe the training process through the displayed charts and gain inferencing knowledge from the confusion matrices to assess the model performance. This will be further discussed in <ref>

The UI was developed using Gradio <cit.>, HTML5 and JavaScript, and the backend code was developed using Python 3.8 on Ubuntu 18.04.



  §.§.§ Scenario 1: Graph Enrichment



The first part of the VariantKG tool focuses on KG enrichment. A user can upload one or more VCF files containing variant-level information and CADD Scores in a TSV format as shown in Figures <ref> or the user can select patients from the existing KG. If the user uploads VCF files, the information in the files is first run through the SnpEFF tool. SnpEFF adds new annotated information to each variant in the VCF file and also updates the headers of the file to reflect the annotations. These annotations are functional information that is added to the `INFO' field using the `ANN' tag. This ANN field, in turn, consists of several bits of information, such as allele, annotation using Sequence and Ontology terms, putative impact, gene name, gene ID, feature type, feature ID, transcript biotype, exon or intron rank, cDNA position, protein position and several types of distances to the feature. 

 

Once the files have been annotated, we then utilize the vcf2rdf tool provided by SPARQLing-Genomics to efficiently translate the information on each variant into several triples in an RDF-suitable N3 format. These N3 triples were converted to NQ format for several reasons. Converting to NQ format allows for the use of a named graph, which provides a robust way to group triples into distinct sets, considerably enhancing data organization and management. Another advantage of using named graph is the support it provides to easily attach new metadata to specific data subsets. Additionally, named graphs also enable more precise and efficient SPARQL queries, improving data extraction quality and speed. 

   

The triples are stored in Blazegraph, a high-performance graph database, which also stores the larger knowledge graph. To prepare the data from ML tasks, users can select patients using age groups as shown in Figure <ref>, or using the accession IDs, as shown in Figure <ref>, which are internally fetched using another SPARQL query that is only executed when the user wishes to prepare data for downstream ML tasks for efficiency. 

The user can then select the features from a list consisting of the original headers from the VCF files and the annotated features by SnpEff. Once the user hits the `Fetch from KG' button, another SPARQL query is then executed in the backend. If the user is selected from an age group, the SPARQL query consists of a filter that fetches all the accession IDs, which is passed as a list to the final query. If the user selects the accession IDs, it is passed as a list, similar to the previous query and the final query, shown below, fetches all the features selected for those patients or accession IDs.


    PREFIX sg_biohackathon:<http://biohackathon.org/resource/faldo#>
    PREFIX sg_variant:<sg://0.99.11/vcf2rdf/>
    PREFIX sg_vcf:<sg://0.99.11/vcf2rdf/variant/>
    PREFIX sg_info:<sg://0.99.11/vcf2rdf/info/>
    PREFIX sg_format:<sg://0.99.11/vcf2rdf/format/>
    PREFIX sg_format_gt:<sg://0.99.11/vcf2rdf/format/GT/>
    PREFIX ns1:<http://sg.org/>
    
    SELECT DISTINCT ?accession_id ?origin (COALESCE(?variant_id, "None") AS ?variant_id) ?chromosome ?position ?ref_genome ?alt_genome ?quality ?ann ?ann_split_1 ?filter_status ?allele_count ?allele_frequency ?total_number_of_alleles ?baseqranksum ?depth ?excesshet ?fs ?mleac ?mleaf ?RMS_mapping_quality ?qd ?readposranksum ?sor ?combined_depth ?conditional_genotype_quality ?genotype ?raw_score ?phred_score
    WHERE 
      GRAPH ?accession_id 
        OPTIONAL  ?origin sg_variant:variantId ?variant_id . 
        BIND (COALESCE(?variant_id, "None") AS ?variant_id)
        ?origin sg_biohackathon:reference ?chromosome .
        ?origin sg_biohackathon:position ?position .
        ?origin sg_vcf:REF ?ref_genome .
        ?origin sg_vcf:ALT ?alt_genome .
        ?origin sg_vcf:QUAL ?quality .
        ?origin sg_info:ANN ?ann .
        BIND (IF(STRLEN(?ann) - STRLEN(REPLACE(?ann, ",", "")) = 0, ?ann, STRBEFORE(?ann, ",")) AS ?ann_split_1)
        OPTIONAL  ?origin sg_info:FILTER_STATUS ?filter_status . 
        OPTIONAL  ?origin sg_info:ALLELE_COUNT ?allele_count . 
        OPTIONAL  ?origin sg_info:ALLELE_FREQUENCY ?allele_frequency . 
        OPTIONAL  ?origin sg_info:TOTAL_NUMBER_OF_ALLELES ?total_number_of_alleles . 
        OPTIONAL  ?origin sg_info:BASEQRANKSUM ?baseqranksum . 
        OPTIONAL  ?origin sg_info:DEPTH ?depth . 
        OPTIONAL  ?origin sg_info:EXCESSHET ?excesshet . 
        OPTIONAL  ?origin sg_info:FS ?fs . 
        OPTIONAL  ?origin sg_info:MLEAC ?mleac . 
        OPTIONAL  ?origin sg_info:MLEAF ?mleaf . 
        OPTIONAL  ?origin sg_info:RMS_MAPPING_QUALITY ?RMS_mapping_quality . 
        OPTIONAL  ?origin sg_info:QD ?qd . 
        OPTIONAL  ?origin sg_info:READPOSRANKSUM ?readposranksum . 
        OPTIONAL  ?origin sg_info:SOR ?sor . 
        OPTIONAL  ?origin sg_info:COMBINED_DEPTH ?combined_depth . 
        OPTIONAL  ?origin sg_info:CONDITIONAL_GENOTYPE_QUALITY ?conditional_genotype_quality . 
        OPTIONAL  ?origin sg_info:GENOTYPE ?genotype . 
        OPTIONAL  ?origin sg_info:RAW_SCORE ?raw_score . 
        OPTIONAL  ?origin sg_info:PHRED_SCORE ?phred_score . 
      
      ?origin sg_biohackathon:position ?position .
      OPTIONAL  ?variant <http://sg.org/has_pos> ?position . 
      ?origin sg_vcf:REF ?ref_genome .
      OPTIONAL  ?variant <http://sg.org/has_ref_genome> ?ref_genome . 
      ?origin sg_vcf:ALT ?alt_genome .
      OPTIONAL  ?variant <http://sg.org/has_alt_genome> ?alt_genome . 
      OPTIONAL  ?variant <http://sg.org/has_cadd_scores> ?cadd_scores . 
      OPTIONAL  ?cadd_scores <http://sg.org/has_raw_score> ?raw_score . 
      OPTIONAL  ?cadd_scores <http://sg.org/has_phred> ?phred_score . 
      FILTER (?accession_id IN (accession_id_list))
     ORDER BY ?variant_id






  §.§.§ Scenario 2: Graph Creation



Once the data has been fetched in the backend, the user is redirected to `Graph Creation' where the user can select node features in the graph and select the class label from a subset of the features that are meaningful to be classified as a part of the classification task. The user is then given complete control of the graph construction where the user can select the edge type which can be using Gene Name (default) or fully connected. The weight of the edges can be 1 (default), the number of incoming edges to a node, or a user-defined value. The user can then select if the edges should be bidirectional and the train:val splits. The default is 80:10 with the remaining 10% calculated in the backend to reduce user clicks. Using this information, the graph is created for the node classification task. This is shown in Figure <ref>

    



  §.§.§ Scenario 3: Graph Machine Learning & Inference



The graph for the node classification task needs to be in a DGL-specific input format. The data is thus converted to a `DGLGraph' object that consists of only integers that are mapped to the actual string values. The user can view the summary of the graph once created. This summary consists of all the selected features, the class label, the number of classes, graph properties, and the number of nodes and edges. The user is given a choice to download the DGL graph or continue with graph machine learning. For the ML task, the user can select between GraphSAGE and GCN models for the classification task and set the hyperparameters to train the model as shown in Figure <ref>. This includes the number of layers other than the input and output layers that constitute the primary architecture of the model, the number of hidden layers, the dropout rate, the learning rate, and the number of epochs for training. 



Once the training begins, the user can view the training and validation loss plots, the validation accuracy plot, and the CPU memory usage as shown in Figure <ref>. 



If the user wants to infer the ML task, they will be navigated to the `Inference' tab shown in Figure <ref>, which displays the evaluation metrics and confusion matrix. The model is evaluated on Accuracy, macro-, and weighted- Precision, Recall, F1 score, and support, which is the number of samples for the given class. 





§ CONCLUSION



This work shows that representing genomic data as knowledge graphs allows vast and diverse information to be integrated from various sources. Modeling entities as nodes and relationships as edges provides an ideal framework for integrating and organizing diverse information. We first described the data collection pipeline, followed by the usage of the SnpEff tool to obtain additional annotations and the SPARQLing Genomics tool to convert the annotations to an RDF format. An ontology to collate information gathered from different sources is presented. Using this ontology, we described how the knowledge graph is created. This knowledge graph contains RNA sequencing information at the variant level from COVID-19 patients from different regions such as lung, blood, etc. Lastly, we demonstrate the usage of the created knowledge graph for a node classification task using the Deep Graph Library through our tool, VariantKG. As part of this, we described various scenarios that can be performed by a user. As part of our future work, we aim to expand the knowledge graph and explore more avenues to use the same to aid researchers working in this domain.



§ ACKNOWLEDGMENTS


This work was supported by the National Science Foundation under Grant Nos. 2201583 and 2034247.



§ DATA AVAILABILITY STATEMENT

The raw VCF and CADD Scores used in the knowledge graph creation for this study can be found in the IEEE Dataport <cit.>. The tool can be accessed via VariantKG [<https://github.com/MU-Data-Science/GAF>].






















































unsrtnat
  





























 Paper Title: The Energy Cost of Artificial Intelligence of Things Lifecycle.




IEEEexample:BSTcontrol




















    Shih-Kai Chou,
Jernej Hribar, Mihael Mohorčič and Carolina Fortuna

Department of Communication Systems, Jožef Stefan Institute, Slovenia

Email:{shih-kai.chou, jernej.hribar, miha.mohorcic, carolina.fortuna}@ijs.si

    August 5, 2024
============================================================================================================================================================================================================================

plain
plain





[MACHU]
  iot[IoT]Internet of Things
  cr[CR]Cognitive Radio
  ofdm[OFDM]orthogonal frequency-division multiplexing
  ofdma[OFDMA]orthogonal frequency-division multiple access
  scfdma[SC-FDMA]single carrier frequency division multiple access
  rbi[RBI] Research Brazil Ireland
  rfic[RFIC]radio frequency integrated circuit
  rf[RF]radio frequency
  sdr[SDR]Software Defined Radio
  sdn[SDN]Software Defined Networking
  su[SU]Secondary User
  ra[RA]Resource Allocation
  qos[QoS]quality of service
  usrp[USRP]Universal Software Radio Peripheral
  mno[MNO]Mobile Network Operator
  mnos[MNOs]Mobile Network Operators
  gsm[GSM]Global System for Mobile communications
  tdma[TDMA]Time-Division Multiple Access
  fdma[FDMA]Frequency-Division Multiple Access
  gprs[GPRS]General Packet Radio Service
  msc[MSC]Mobile Switching Centre
  bsc[BSC]Base Station Controller
  umts[UMTS]universal mobile telecommunications system
  Wcdma[WCDMA]Wide-band code division multiple access
  wcdma[WCDMA]wide-band code division multiple access
  cdma[CDMA]code division multiple access
  lte[LTE]Long Term Evolution
  papr[PAPR]peak-to-average power rating
  hn[HetNet]heterogeneous networks
  phy[PHY]physical layer
  mac[MAC]medium access control
  amc[AMC]adaptive modulation and coding
  mimo[MIMO]multiple input multiple output
  rats[RATs]radio access technologies
  vni[VNI]visual networking index
  rbs[RB]resource blocks
  rb[RB]resource block
  ue[UE]user equipment
  cqi[CQI]Channel Quality Indicator
  hd[HD]half-duplex
  fd[FD]full-duplex
  sic[SIC]self-interference cancellation
  si[SI]self-interference
  bs[BS]base station
  fbmc[FBMC]Filter Bank Multi-Carrier
  ufmc[UFMC]Universal Filtered Multi-Carrier
  scm[SCM]Single Carrier Modulation
  isi[ISI]inter-symbol interference
  ftn[FTN]Faster-Than-Nyquist
  m2m[M2M]machine-to-machine
  mtc[MTC]machine type communication
  mmw[mmWave]millimeter wave
  bf[BF]beamforming
  los[LOS]line-of-sight
  nlos[NLOS]non line-of-sight
  capex[CAPEX]capital expenditure
  opex[OPEX]operational expenditure
  ict[ICT]information and communications technology
  sp[SP]service providers
  inp[InP]infrastructure providers
  mvnp[MVNP]mobile virtual network provider
  mvno[MVNO]mobile virtual network operator
  nfv[NFV]network function virtualization
  vnfs[VNF]virtual network functions
  cran[C-RAN]Cloud Radio Access Network
  vran[V-RAN]Virtual Radio Access Network
  bbu[BBU]baseband unit
  bbus[BBU]baseband units
  rrh[RRH]remote radio head
  rrhs[RRH]Remote radio heads 
  sfv[SFV]sensor function virtualization
  wsn[WSN]wireless sensor networks 
  bio[BIO]Bristol is open
  vitro[VITRO]Virtualized dIstributed plaTfoRms of smart Objects
  os[OS]operating system
  www[WWW]world wide web
  iotvn[IoT-VN]IoT virtual network
  mems[MEMS]micro electro mechanical system
  mec[MEC]Mobile edge computing
  coap[CoAP]Constrained Application Protocol
  vsn[VSN]Virtual sensor network
  rest[REST]REpresentational State Transfer
  aoi[AoI]Age of Information
  lora[LoRa™]Long Range
  iot[IoT]Internet of Things
  snr[SNR]Signal-to-Noise Ratio
  cps[CPS]Cyber-Physical System
  uav[UAV]Unmanned Aerial Vehicle
  rfid[RFID]Radio-frequency identification
  lpwan[LPWAN]Low-Power Wide-Area Network
  lgfs[LGFS]Last Generated First Served
  wsn[WSN]wireless sensor network 
  lmmse[LMMSE]Linear Minimum Mean Square Error
  rl[RL]Reinforcement Learning
  nb-iot[NB-IoT]Narrowband IoT
  lorawan[LoRaWAN]Long Range Wide Area Network
  mdp[MDP]Markov Decision Process
  ann[ANN]Artificial Neural Network
  dqn[DQN]Deep Q-Network
  mse[MSE]Mean Square Error
  ml[ML]Machine Learning
  cpu[CPU]Central Processing Unit
  gpu[GPU]Graphics Processing Unit
  tpu[TPU]Tensor Processing Unit
  ddpg[DDPG]Deep Deterministic Policy Gradient
  ai[AI]Artificial Intelligence
  gp[GP]Gaussian Processes
  drl[DRL]Deep Reinforcement Learning
  mmse[MMSE]Minimum Mean Square Error
  fnn[FNN]Feedforward Neural Network
  eh[EH]Energy Harvesting
  wpt[WPT]Wireless Power Transfer
  dl[DL]Deep Learning
  yolo[YOLO]You Only Look Once
  mec[MEC]Mobile Edge Computing
  marl[MARL]Multi-Agent Reinforcement Learning
  aiot[AIoT]Artificial Intelligence of Things
  ru[RU]Radio Unit
  du[DU]Distributed Unit
  cu[CU]Central Unit
  api[API]Application Programming Interface
  cf[CF]Carbon Footprint
  ci[CI]Carbon Intensity
  oran[O-RAN]Open Radio Access Network
  flops[FLOPs]floating-point operations per cycle per core
  FLOPS[FLOPs]floating-point operations
  5g[5G]fifth-generation
  embb[eMBB]enhanced Mobile Broadband
  urllc[URLLC]Ultra Reliable Low Latency Communications
  mmtc[mMTC]massive Machine Type Communications
  hdd[HDD]hard disk drive
  ssd[SSD]solid state drive
  fspl[FSPL]free space path loss
  cav[CAV]Connected Autonomous Vehicle
  xr[XR]Extended Reality
  ap[AP]access point
  ble[BLE]Bluetooth low energy
  uwb[UWB]ultra wide band
  rss[RSS]received signal strength
  nn[NN]Neural Network
  v2x[V2X]vehicle-to-everything
  gai[GAI]Generative AI
  aiaas[AIaaS]AI as a service
  mlp[MLP]Multilayer Perceptron
  ble[BLE]Bluetooth Low Energy
  lorawan[LoRaWAN]Long Range Wide Area Network
  FPGAs[FPGAs]Field Programmable Gate Arrays
  llm[LLM]Large language model
  fl[FL]federated learning



 




§ ABSTRACT 
ai coupled with existing iot enables more streamlined and autonomous operations across various economic sectors. Consequently, the paradigm of aiot having ai techniques at its core implies additional energy and carbon costs that may become significant with more complex neural architectures. To better understand the energy and cf of some aiot components, very recent studies employ conventional metrics. However, these metrics are not designed to capture energy efficiency aspects of inference. In this paper, we propose a new metric, the Energy Cost of aiot Lifecycle (eCAL) to capture the overall energy cost of inference over the lifecycle of an aiot system. We devise a new methodology for determining eCAL of an aiot system by analyzing the complexity of data manipulation in individual components involved in the aiot lifecycle and derive the overall and per bit energy consumption. With eCAL we show that the better a model is and the more it is used, the more energy efficient an inference is. For an example aiot configuration, eCAL for making 100 inferences is 1.43 times higher than for 1000 inferences. We also evaluate the cf of the aiot system by calculating the equivalent CO_2 emissions based on the energy consumption and the ci across different countries. Using 2023 renewable data, our analysis reveals that deploying an aiot system in Germany results in emitting 4.62 times higher CO_2 than in Finland, due to latter using more low-ci energy sources.












AIoT lifecycle, Energy Consumption, Carbon Footprint, Metric, Methodology







§ INTRODUCTION



Next-generation networks aim to deliver unprecedented levels of connectivity and intelligence, facilitating seamless interactions between a massive number of devices, services, and applications. The trend is especially noticeable in iot where devices are becoming increasingly more intelligent <cit.>, from smart home appliances <cit.> to industrial machinery <cit.>, Smart City <cit.>, v2x <cit.>, and beyond <cit.>.
With the help of ai, we are moving away from conventional iot applications towards aiot systems <cit.> designed to achieve more streamlined and autonomous operations. For example, in future smart factories, the integration of ai and iot will enable real-time monitoring and optimization of production processes, predictive maintenance, and enhanced decision-making capabilities. 

While aiot systems enable unprecedented automation and agility, their reliance on ai implies additional energy and carbon costs <cit.>. Especially in the cases when  models with many parameters are employed, such as llms, such costs may be significant <cit.>. In light of climate challenges,  the general ai research community's effort to better estimate  the costs of training <cit.>, estimate the cost of ai <cit.>, the cfs of llms <cit.> and find ways to scale models efficiently <cit.> are intensifying. Furthermore, the networking and iot communities increasingly relying on ai techniques are also investigating pathways towards <cit.> net-zero carbon emissions, analyzing the cf of various learning techniques <cit.> and data modalities while also developing various approaches to optimizing energy efficiency through hardware-software optimization <cit.>, scheduling <cit.>, more efficient model design <cit.> and energy and carbon consumption testing <cit.>.


Depending on the scope of the study, various metrics are employed to understand energy and cf. For instance, <cit.> considers normalized energy as well as energy for buffering in [J], energy [Wh] and [CO_2eq] for the various phases involved in fl edge systems <cit.>, [J] and GFLOPS of the neural architectures <cit.> and energy [Wh] for the scheduled loads in  <cit.>. However, these metrics are not specifically designed to capture the energy efficiency of a system or part of. As well noted in <cit.>, the traditional ITU standardized Energy-per-Bit [J/b] metric 'will no longer be able to reflect the environmental impact of the modern mobile services, especially network AI-enabled smart services'. 

Recently, a number of environmentally friendly metrics, such as accuracy per consumption (APC), that capture the performance of a ai model together with its environmental impact <cit.>. Nevertheless we observe the following. First, while the metrics proposed in <cit.> are powerful in terms of assessing the energy/performance tradeoffs of models, these metrics are less suitable in assessing the energy efficiency of a system such as AIoT. Second, that a metric somewhat similar to the Energy-per-Bit [J/b] would be more suitable for measuring energy efficiency. Third, other efficiency metrics have been proposed in wireless in particular and engineering and economy in general <cit.>. For instance, spectral efficiency, measured in [b/s/Hz] <cit.> provides a means of calculating the amount of data bandwidth available in a given amount of spectrum, while lifecycle emissions for vehicles, measured in [g/km] <cit.>, enables computing the cf in grams per kilometer.


Following these observations, in this paper, we aim to find a suitable energy efficiency metric that would be simple, general and holistically quantify the environmental cost of inference carried out in AIoT systems. The contributions of this paper are as follows:



    
  * We propose a new metric, the Energy Cost of AIoT Lifecycle (eCAL), also measured in [J/b],  that captures the overall energy cost of generating an inference in an AIoT system. Unlike capturing the energy required for transmitting bits that is done by the Energy-per-Bit [J/b] metric, or [J] for ai model complexity, the proposed eCAL captures the energy consumed by all the data manipulation components in an AIoT system during the lifecycle of a trained model enabling inference. 
    
  * We devise a methodology for determining the  eCAL of an aiot system. The methodology breaks down the AIoT system into different data manipulation components, i.e. data collection, storage, and data preprocessing, training along with evaluation, and inference, and, for each component, it analyzes the complexity of data manipulation and derives the overall and per bit energy consumption. 
    
  * Using the proposed metric, we show that the better a model is and the more it is used, the more energy efficient an inference is. For an example aiot configuration, the energy consumption per bit for making 100 inferences is 1.33 times higher than for 1000 inferences. This can be explained by the fact that the energy cost of data collection, storage, training and evaluation, all necessary to develop an ai model are spread across more inferences.
    
  * We evaluate the cf of the aiot system by calculating the equivalent CO_2 emissions based on the energy consumption and the ci across different countries. Using 2023 renewable  data, our analysis reveals that deploying an aiot system in Germany results in emitting 4.62 times higher CO_2 than in Finland, due to the latter using more low-ci energy sources. 
    
    
    


The paper is structured as follows. We first review the relevant literature in Section <ref>, we then describe the lifecycle of an aiot system, define eCAL and the methodology for deriving it in Section <ref>. Subsequently, the derivation of the energy consumption formulas  for different data manipulation components involved in the aiot lifecycle are provided in Sections <ref> to <ref>. In Section <ref>, the proposed metric, eCAL, is derived and analyzed, demonstrating its utility in capturing the overall energy cost of AIoT systems over its lifecycle. whereas in Section <ref> we discuss the cf aspect of such a system, comparing emissions across different European countries. 
Finally, Section <ref> concludes the paper and outlines future research directions.







§ RELATED WORKS



While aiot systems enable unprecedented automation and agility, their reliance on ai implies additional energy and carbon costs <cit.> that in some cases that employ models with many parameters, such as llms, may be significant <cit.>. In light of climate challenges and the dazzling footprints of the latest generation of LLMs <cit.>,  the general ai research community's effort to better understand and then formalize methods to asses the environmental impact of the ai revolution is rising. Due to the complexity of state of the art neural architectures, in many cases the energy cost of training is computed after the training by measuring the performed computation through interfaces such as the performance application programming interface <cit.>. Furthermore, <cit.> shows that the energy footprint of inference is more studied than the one for training and highlights below 70% accuracy between predicted and measured energy consumption. 

However, a  number of approaches and tools capable of proactively estimating the energy and environmental cost of training have also emerged. LLMCarbon <cit.> is a very recent end-to-end cf projection model designed for both dense and mixture of experts LLMs.  It incorporates critical LLM, hardware, and data center parameters, such as LLM parameter count, hardware type, system power, chip area, and data center efficiency, to model both operational and embodied cfs of a LLM. Furthermore, <cit.> aims at providing guidelines for scaling ai in a sustainable way. They analyze the energy efficiency of new processing units, the cf of the most prominent LLM models since GPT-3 and analyze their lifecycle carbon impact showing that inference and training are comparable. They conclude that, to enable sustainability as a computer system design principle, better tools for carbon telemetry are required, large scale carbon datasets, carbon impact disclosure and more suitable carbon metrics.

The networking and iot communities, increasingly relying on ai techniques, are also investigating pathways towards <cit.> net-zero carbon emissions. The authors of <cit.> notice that in spite of improvements in hardware and software energy efficiency, the overall energy consumption of mobile networks continues to rise, exacerbated by the growing use of resource-intensive AI algorithms. They introduce a novel evaluation framework to analyze the lifecycle of network AI implementations, identifying major emission sources. They propose the Dynamic Energy Trading and Task Allocation framework, designed to optimize carbon emissions by reallocating renewable energy sources and distributing tasks more efficiently across the network. Similar as in <cit.>, they highlight the development of new metrics to quantify the environmental impact of new network services enabled by AI.  

The authors of <cit.> introduce a novel framework to quantify energy consumption and carbon emissions for vanilla FL methods and consensus-based decentralized approaches, identifying optimal operational points for sustainable FL designs. Two case studies are analyzed within 5G industry verticals: continual learning and reinforcement learning scenarios. Together with the authors of <cit.>, they consider 
energy [Wh] and [CO_2eq] in their evaluations. The solution proposed in <cit.> is a hardware/software co-design that introduces modality gating (throttling) to adaptively manage sensing and computing tasks. AMG features a novel decoupled modality sensor architecture that supports partial throttling of sensors, significantly reducing energy consumption while maintaining data flow across multimodal data: text, speech, images and video. More energy efficient task scheduling has been considered in  <cit.>, while more efficient neural architecture design and subsequent model development in <cit.>. The energy efficiency of various programming languages is studied in <cit.> while energy and carbon consumption testing for AI-driven IoT services is developed in <cit.>.

Novel metrics to evaluate dl models, considering not only their accuracy and speed but also their energy consumption and cost have been proposed in <cit.>. The  four metrics are: Accuracy Per Consumption (APC) and Accuracy Per Energy Cost (APEC) for inference, and Time To Closest APC (TTCAPC) and Time To Closest APEC (TTCAPEC) for training. These metrics aim to promote the use of energy-efficient and green energy-powered dl systems by integrating energy considerations into the performance evaluation. Nevertheless, while powerful in terms of assessing the energy/performance tradeoffs of models, these metrics are less suitable in assessing the energy efficiency of a system such as AIoT. The Energy-per-Bit [J/b] has been defined for assessing the efficiency of transmissions and therefore does not capture the ai aspects as also noted in <cit.>. Looking at the electric vehicle literature, we notice that, when comparing to traditional gas vehicles they employ lifecycle emissions as a metric, measured in [g/km] <cit.>. 

Inspired by the simplicity and effectiveness of metrics such as transmission efficiency through energy-per-bit and the lifecycle emissions, and responding to calls for new metrics <cit.> that capture the energy cost in the era of ai enabled systems, this work proposes the novel Energy Cost of AIoT Lifecycle (eCAL) metric and methodology to derive it.



§ ECAL DEFINITION AND METHODOLOGY



In our work, we consider an aiot communication and computing system whose lifecycle is  depicted in Fig. <ref>, consisting of the following data manipulating components: 


  
Data Collection 
This data-manipulating component, depicted in the lower part of Fig. <ref>, the collected data N_S, includes receiving telemetry data from sensors to the terminal computing infrastructure via wired or wireless technologies. The collected information can be turned into indicators such as signal strength, and for example, be interpreted by the ai models to predict the location of a user. To move this data from the sensor to the ap, E_T [J] is needed. 




  
Storage and Data Preprocessing 
Once the data from iot devices arrives at the computing infrastructure, it is first stored on the hard drive, and then fetched for preprocessing. Therefore, the energy consumption, E_storage, of this component arises from reading and writing data N_S into the storage units, which can vary depending on data volume and modality. 
To ensure accuracy and reliability during the training process, the data, N_S, must go through several preprocessing steps such as cleaning, feature engineering and transformation. The energy consumption for preprocessing E_pre, depends on the integrity of the ingested dataset. For example, datasets with a lot of invalid or missing data, require more extensive cleaning and error correction. 




  
Training and Evaluation 
The training component comprises the model development of the ai/ml using selected ai/ml techniques, such as neural architectures, and data. In this step of the model development process, the processed data, N_S,T is fetched and utilized to learn weights and biases in order to approximate the underlying distribution. For most neural architectures, the learning processes rely on gpu and tpu performing complex tensor processing operations and consuming E_train. Once the neural architecture weights are learned using the data in view of minimizing a loss function, the model is considered ready for evaluation and deployment. Subsequently, the quality of the learned model is evaluated on the evaluation datasets, N_S,E, a process that consumes E_eval energy. The data storage, preprocessing, training, and evaluation form the end-to-end training of the ai/ml model and cumulatively require E_D energy to complete.




  
Inference
Once the model is trained, it can be used by applications in inference mode. Various applications can send samples of data N_I,P and receive model outputs in the form of forecasts for regression problems or discrete (categorical) for classification problems. The energy consumption E_inf of a single inference is relatively small, however for high volumes of requests it can become significant.


Based on this lifecycle, we proposed eCAL as the ratio between all the energy consumed by all data manipulation components and all the manipulated application-level bits: 


    eCAL   =
       Total energy of data manipulation components  [J]/Total manipulated application level data  [b].


In the upcoming sections, we employ the following methodology in order to finally derive the proposed metric. We follow the telemetry data flow, highlighted with blue in Figure <ref>, across the above-mentioned components and derive the computational complexity and subsequently the absolute and per bit energy consumption, highlighted with green and orange respectively, in each component. For instance, in Section <ref>, we derive E_T and E_T,b for the data collection component while in Section <ref>, we do the same for preprocessing and storage E_pre and E_pre,b respectively. With this formalism in place, Section <ref> is concerned with deriving eCAL.

In addition to providing the first  theoretical framework for assessing and quantifying the energy consumption of each component of an aiot system in the data manipulation workflow, we also discuss and show differences across countries in view of aiot inference cf.



§ ENERGY COST OF DATA COLLECTION




In this section, we derive the energy cost of data collection in an aiot system. To calculate the cost, we first determine the number of bits that need to be transmitted from iot devices to the server via wireless technology. Then we calculate the energy consumption based on the given transmitting power and corresponding transmission rate. In the last step, we derive the total energy consumption for collecting data from iot devices. The energy consumption of wired communication compared to the wireless part is negligible <cit.>. Therefore, in this work, we only consider the communication from the iot device to the ap via wireless technologies.


Each wireless technology relies on different 
multi-layer protocol stack adding overhead bits, e.g., header, error correction, etc., to ensure collected data integrity and efficient transmission thus increasing energy consumption. Therefore, each packet is composed of payload, e.g., measurements collected by an iot device, and overhead, with overhead size depending on the selected technology. The size of the transmitted data in bits (B_T) can be expressed as defined in <cit.>:

    B_T [b]=α N_S_payload+⌈α N_S/F_U⌉·Ω_U_Overhead,



where α represents bit precision, i.e., the number of bits required to store value of a sample in floating number. Typically α equals 32 bits for single precision (float) or 64 bits for double precision. Moreover, we denote samples as 
N_S which represents the telemetry data that the application requests from the device in floating-point form, i.e., when an application requests 256 samples, the payload size can be calculated as 8192 and 16384 bits for single precision and double precision, respectively. The second part of the equation represents the overhead.  First, the number of packets is determined, by rounding up the payload divided with the maximal number of bits in 
 one packet (F_U) for the selected access technology. To obtain the total overhead the number of packets is then multiplied with the number of overhead bits per packet, that also depends on the selected access technology.



To illustrate the behavior of Eq. <ref>, we consider a scenario of a single device transmitting N_S=256 of double-precision samples to the application and alter the overhead percentages in one packet. More specifically, we consider a fixed packet size of 2000 bits. We consider overhead of 
1%, 30%, 50%, and 70%. 
Additionally, we consider overhead percentages of three mainstream wireless technologies <cit.> for iot systems, 
namely ble 5.0 <cit.>, ZigBee <cit.>, and lorawan <cit.>. In Table <ref>, we summarize the parameters of these three technologies based on iot protocol stack, in which the uplink data goes through the physical layer to the application layer. 
Fig. <ref> illustrates the relationship between the number of samples and the size of the transmitted data for different percentages of overhead. As seen in the figure, higher overhead percentages result in significantly more data being sent than when overhead is low. For example, when transferring 256 samples, the data transfer requirements for the scenarios Ω_U=1% F_U  and Ω_U=70% F_U are 16564 and 28984 bits, respectively, which means the latter has 1.7 times higher data transfer requirements than the former.















Thus, we can conclude that when the size of samples (N_S) is small, leveraging wireless technologies with lower overhead, such as ble, is advantageous. These technologies efficiently transmit fewer total bits. However, as the size of samples increases, the impact of overhead becomes less significant. In these scenarios, technologies with higher overhead but larger maximum packet sizes can become more beneficial. This shift emphasizes the importance of the trade-off between overhead, sample size, and maximum packet size, and highlights the importance of selecting the appropriate technology to achieve adequate energy efficiency in aiot deployments.










 
  
  
   



 





Next, we determine the energy consumption for a given technology, which we define as follows:

    E_T[J]=P_T[W]/R_T[b/s]· B_T[b].

where P_T represents the transmitting power. As a result, we can obtain energy consumption per bit for transmitting part of aiot system as follows: 

    E_T,b[J/b]=P_T[W]· T_T[s]/B_T[b]=P_T[W]/R_T[b/s] ,

where T_T is the corresponding transmitting time. T_T can be derived from the corresponding transmitting rate (R_T) and B_T, and is expressed as T_T=B_T/R_T.

Considering the previous example, if we transmitted the 256 samples with 1% overhead with P_T=10  [mW] and R_T=10^3 [b/s], the energy consumption [J] of 1% overhead would be 0.165, and 70% is 0.29. Moreover, the energy consumption per bit [J/b] for both 1% overhead and 70% overhead would be 10^-5. 

We can observe from the above equations that we are able to determine that when considering energy consumption per bit (E_T,b), this metric is exclusively related to P_T and R_T, which indicates that the energy required to transmit one bit of data depends on the power used for transmission and the speed of the transmission. Meaning, that energy consumption per bit is the inverse of energy efficiency <cit.>.








 We consider the wireless technologies listed in Table <ref> to illustrate per-bit energy consumption with the corresponding P_T and R_T and calculated E_T,b in Table <ref>. The comparison between E_T,b of different technologies is shown in Fig. <ref>. 
 It can be seen that ble exhibits the best performance due to its low power consumption and high transmission rate, while lorawan exhibits the highest energy consumption per bit despite its low overhead. This emphasizes the crucial role of transmission power and rate in determining the overall energy efficiency in aiot systems. Our results underscore the need to evaluate both power consumption and transmission rate, not just overhead, when selecting the communication technology for aiot deployments. Prioritizing lower power consumption and higher transmission rates can significantly improve energy consumption per bit.
 
 
 
 
Finally, the total energy consumption of the data transmission in an aiot system is illustrated in Fig. <ref>. We assume that the signals are transmitted at one-minute intervals and uniformly. These results showcase the total energy consumption over a period of 24 hours of continuous operation. ble 5.0 benefits from low power consumption, a high data rate, and low overhead, resulting in the lowest energy consumption per bit and very low overall energy consumption. Despite lower energy consumption per bit of ZigBee wireless technology, overall energy consumption rises with larger data sizes due to higher overhead percentage, whereas, lorawan has the highest energy consumption owing to its E_T,b is more dominate factor of the energy consumption. Moreover, these technologies excel in different aspects. For example, lorawan is optimal for long-range communication, whereas ZigBee is suitable for mesh networking. 
Consequently, both the total number of bits that devices transmit (B_T) and energy consumption per bit (E_T,b) play a role in determining energy cost of data transmission. Once the data reaches the ap is passed to the server for storage and preprocessing. 




§ ENERGY COST OF STORAGE AND DATA PREPROCESSING


Typically, data is first stored on a hard drive, subsequently preprocessed, and then used for the ml model development. In the following, we consider the dataset with N_S samples collected via wireless transmissions as depicted in Figure <ref> and discussed in Section <ref>. 




 §.§ Storage

The energy consumption of storage primarily depends on the type of hard drive. According to <cit.>, a typical hard drive consumes 0.65 Watt-Hour per Terabyte stored with a hdd-based technology. On the other hand, ssd-based hard drive consumes 1.2 Watt-Hour per Terabyte. For example, the energy consumption (E_storage [J]) of a dataset with N_S= 256 samples in the form of double-precision floating number consumes 4.79× 10^-6 and 8.85× 10^-6 Joules for hdd and ssd-based technology, respectively, whereas the corresponding energy consumption per bit (E_storage,b [J/b]) are 2.92× 10^-10 and 5.4× 10^-10. 


 §.§ Data preprocessing
















 We consider a data preprocessing approach consisting of data cleaning and data standardization steps, while we leave other more complex approaches such as feature engineering and transformations to future work. We assume that the data-cleaning process is performed in two steps. The first step is identifying and removing invalid samples from a list or array which  incurs 0 FLOP since no floating points additions and multiplications are performed, only comparisons and memory operations.
 The second step is to determine the complexity of data standardization (M_DS). We provide two examples, i.e., min-max scaling and normalization process, both representing a standard approach for data standardization.
 



  
Min-max scaling
In this method, which we detail in Algorithm <ref>, we first initialize two values for the minimum and maximum. Then, we iteratively search for minimum and maximum values by going through all the valid samples to update these values (lines <ref> to <ref>), which involve comparisons but no FLOP. After finding the minimum and maximum values, we calculate the range (line <ref>), which requires one FLOP. Finally, we scale the dataset (lines <ref> to <ref>) by performing a subtraction and division for each valid sample, costing 2 (N_S - N_NaN - 1) FLOPS. Therefore, the corresponding total number of FLOPS (M^minmax_DS) of min-max scaling can be expressed as: 

    M^minmax_DS = 2· (N_S-N_NaN)-1,

where N_NaN is the number of invalid samples.





  
Normalization
Normalization is performed in three steps, as described in Algorithm <ref>. The algorithm starts by calculating the mean value (lines <ref> to <ref>), followed by the calculation of the standard deviation (lines <ref> to <ref>). Finally, the algorithm iteratively normalizes each sample (lines <ref> to <ref>). Thus, the total operation (N^norm_DS) can be expressed as:

    M^normalization_DS = 6·(N_S - N_NaN - 1) + 3.









As a result, the total number of FLOPS for preprocessing can be expressed as:

    M_pre = 
    
        2· (N_DS - N_NaN) - 1,    if min-max scaling, 
    
        6·(N_DS - N_NaN) - 3,    if normalization.








We can observe that the computing complexity of both standardization methods decreases linearly as the number of invalid samples increases. This indicates that a higher number of invalid samples lowers the computing complexity, leading to reduced energy consumption. However, this also means that the application has to request additional transmissions from the device and perform extra operations to preprocess the data on the server.







 
 
 
  



















With the computing complexity in FLOPS formalized, we can obtain total energy consumption (E_pre) of the preprocessing (E_pre) by utilizing the power consumption of the processing unit and the executing time (T_pre). Thus, it is expressed as follows: 

    E_pre[J]= P_pre[W]· T_pre[s],

where T_pre can be calculated as the ratio between FLOPs needed for preprocessing and the computational power of a processor (N_PU). 

    T_pre[s] = M_pre[FLOPs]/M_PU[FLOPs/s].




 

Consequently, the energy consumption per bit for preprocessing can then be determined as: 

    E_pre,b[J/b]=E_pre[J]/α N_S[b].


 
 
  
   
    
     
    
    
    
    
    
    
    
    
    
    


For example, the E_pre and the corresponding E_pre,b for performing data preprocessing on a CPU with a performance of M_PU=10 GFLOPs/s and consume power of P_pre=140 W can be seen in the Fig. <ref> and Fig. <ref>, respectively. More specifically, Fig. <ref> shows that the energy consumption for both standardization methods decreases as the number of invalid samples increases, confirming that higher invalid sample rates lead to lower computational requirements. In addition, Fig. <ref> illustrates the energy consumption per bit for normalization and min-max scaling with N_S=256, where normalization consumes three times more energy per bit than min-max scaling. These results highlight the importance of selecting an appropriate preprocessing method based on both the energy efficiency requirements and AI/ML technique requirements. After processing the samples, the training process described in the next section can be carried out. 



 
  
   
   
  


 


















 
 






              



§ ENERGY COST OF TRAINING


While there are many ai/ml models available, in this work we focus on a fully connected mlp, as they form the foundation of all modern AI methods and point the reader to works such as <cit.> for a more comprehensive view on estimating energy consumption in machine learning. A filly connected MLP architecture is depicted in Fig. <ref> with the blue and red arrows illustrating the forward and backwards propagation taking place during the training process while the training is summarized in Algorithm <ref>. The architecture comprises of one input layer with N_I neurons, K hidden layers, in each layer it has M neurons and N_O neurons in the output layer. As shown in the Fig. <ref>, the training process includes two steps, namely, training and evaluation. Typically, input samples are split into certain ratios for training and evaluation. For example, with a 70/30 split, a dataset with 256 samples will have 179 samples for training, and 77 samples for evaluation. Therefore, we introduce β for such ratio. More specifically, the input samples can be expressed as:

    N_S=β N_S_N_S,T+(1-β)N_S_N_S,E,

where N_S,T and N_S,E represent the number of samples for training and evaluation, respectively. 

Model training of mlp involves two main phases as depicted in Fig. <ref> and Algorithm <ref>. First, the training samples undergo forward propagation where neurons compute linear combinations of weights and biases, apply activation functions, and calculate the loss between predicted and actual values. Subsequently, the model performs backward propagation where it calculates gradients of the loss with respect to each parameter, updates these gradients in reverse through the layers, and adjusts the weights and biases accordingly, which is described in Algorithm <ref> from line <ref> to line <ref>. This process completes one epoch. Consequently, we can calculate the energy costs associated with forward and backward propagation, leading to an understanding of the overall energy consumption for training, subsequently model evaluation, and inference.




 §.§ Computational Complexity of Forward and Backward Propagation

Assuming L layers where L=K+2, the total FLOPS for the forward propagation of a single input sample for an MLP can be expressed as: 

    M_FP=∑_l=1^L-1 (2 · (M_l-1· M_l)+2· M_l)















 
 
 
 


 
 
  
  



where L represents the overall number of layers in the MLP, the first term in the sum represents one  multiplication and one addition (=2 FLOPs) related to the weight and bias corresponding to an the edges times the number of edges on that layer (=M_l-1· M_l) while the second term represents the summation of all the values of the incoming edges and the application of the activation function (=2 FLOPs) times the number of nodes in that layer (M_l). As in our guiding example we assume all hidden layers having the same number of neurons, Eq. <ref> becomes M_FP=(2 · N_I· M + 2 · M) + (2 · (K-1)· M^2 + 2 · M) + (2 · MN_O + 2 · N_O) using the notations from Fig. <ref>. Considering an example mlp
with 3 hidden layers with 5 neurons on each hidden layer, 6 neurons on the input layer. Moreover, we aim to train the mlp model and utilize it to make predictions of three-dimensional coordinates based on the input data, thus, there will be 3 neurons on the output layer. Therefore, the total number of FLOPs can be calculated as M_FP= 2(6· 5+ 5) + 2(2· 25+5) + 2(5· 3+3) = 226. 



Training consists of multiple epochs and batches. Therefore, the total complexity of forward propagation is: 

    M_MLP,FP   = N_epochs· N_batch·batchsize· M_FP
       =N_epochs· N_S,T· M_FP.

Continuing with the example with 256 samples that goes through the aforementioned mlp model across 10 epochs, the computational complexity for forward propagation can be calculated M_MLP,FP=10· 256· 226=578560 FLOPS as also depicted in Fig. <ref>. When all hidden layers have the same number of neurons and their number M is increasing, it can be seen from Fig. <ref> 
that the FLOPs also increase quadratically. Adding more hidden layers K results in increases more than linearly due to the multiplicative nature of the connections between layers. As per Eq. <ref> and Fig. <ref>, the relationship between the increase in complexity and the augmentation of the number of samples and epochs is linear. These observations highlight the importance of considering the impact of architectural parameters on the computational complexity of MLP models. 



As we can see from Algorithm <ref> (line <ref> to line <ref>), the backward propagation consists of approximately twice the computational complexity of forward propagation, which is also confirmed in <cit.>. Thus, we can approximate the computing complexity of training a model as: 

    M_MLP≈  3M_MLP,FP.

As a result, we can compute the energy consumption for training, model evaluation, and the corresponding energy consumption per bit. 





 
  
  
  





    
    






 §.§ Energy Cost of Training

We can obtain the energy consumption of the entire training process as:

    E_train=3M_MLP,FP[FLOPs]/PU_performance[FLOPs/s/W],

where PU_performance represents the theoretical peak performance of a processing unit. Moreover, the corresponding energy consumption per bit can be obtained from Eq. <ref> and the number of input sample, thus, we can show it as: 

    E_train,b=3M_FP/α PU_performance.







 §.§ Energy Cost of Model Evaluation

The model evaluation starts once training is complete. This functionality tests the performance of the model on a separate dataset. During the evaluation, the model processes the data using forward propagation without any adjustments to its parameters, which can be seen from lines <ref> to <ref> in Algorithm <ref>. Therefore, the energy consumption and energy consumption of model evaluation is expressed as:

    E_eval=M_FP· N_S,E/PU_performance.

By utilizing the same technique as in Eq. <ref>, the energy consumption per bit of evaluating the model can be expressed as: 

    E_eval,b = M_FP/α PU_performance.
 


§ ENERGY COST OF INFERENCE


The complexity of making an inference depends on the number of FLOPS required for forward propagating with input sample size N_I,P, i.e., N_inf=M_FP· N_I,P. For example, when we adopt the aforementioned MLP model with N_I,P=77, the complexity is calculated as N_inf=226· 77 = 17402 FLOPS. Therefore, the energy consumption of the forward propagation and the corresponding inference can be calculated as: 

    E_inf = M_FP· N_I,P/PU_performance.
 
We can then observe from Eq. <ref> and  <ref>, the cost per bit of inference is the same as in evaluation. Thus, we can conclude that the computational complexity between finishing one training and inference is on the magnitude of 3N_epochsN_S,T/N_I,P. This indicates that the training process involves significantly more computational operations, particularly when the number of epochs or 
N_S,T>>N_I,P. On the other hand, the energy consumption per bit depends on the model complexity and hardware capability. Moreover, the factor of three increases in the energy consumption per bit during training versus inference highlights the additional computational load inherent to training. Finally, the calculation of end-to-end energy consumption of an aiot system is presented in the next section.  







§ ECAL: THE ENERGY COST OF AIOT LIFECYCLE


In this section, we describe the end-to-end energy consumption of an aiot system for making inferences with a mlp model. First, we consider the energy consumption of developing the model illustrated on the left hand side of  Fig. <ref> as follows:

    E_D=E_T+E_storage+E_pre+E_train+E_eval,

and the energy consumption per bit that can be  expressed as: 

    E_D,b=E_D/B_T+α (2N_S+N_S,T+N_S,E).


To compare the energy consumption of all the data manipulation components required to develop and deploy an aiot system, we continue with the guiding example from the paper where the ai model relies on an an MLP with 3 hidden layers (5 neurons each), whereas on the input layer and output layer there are 6, 3 neurons, respectively, and it requests 256 samples from a iot device via ble to train and evaluate the model with a 70/30 split as discussed in Section <ref>. In addition, we utilize hdd for storage and normalization as data standardization as discussed in Section <ref>. The energy consumption for this example is shown in Fig. <ref> and reveals that the total energy consumed in this scenario is 0.0264J. Moreover, the energy consumption of a training session is equal to 141 times collecting the same amount of samples from the iot device, or 71 times of inference. However, these numbers can be significantly higher if the complexity of the model increases or the size of input samples for training is significantly larger than that used for inference as the respective analyses in Sections <ref> and <ref> showed.

Once the model is developed, it is packaged and deployed in the aiot system and it can produce inference in the form of continuous or discrete outputs. During its operational lifecycle, the model will be presented some input data and requested to produce the corresponding inference, thus we can express the associated energy cost as:

    E_inf,p=E_T+E_storage+E_pre+E_inf,

it means that we consider the energy cost of inference before the current model needs to be updated. The corresponding energy per bit of inference once deployed is calculated as: 

    E_inf,p,b=E_inf,p/B_T+3α N_I,P.


Next, the entire energy cost over the lifetime of a model in aiot the energy consumed for the model development, E_D is summed with the energy required for performing an inference E_inf,p times the number of times γ the inference is performed with the currently deployed model, and γ∈[1, ∞), also referred to as eCAL absolute and measured in [J] as: 

    eCAL_abs=E_D+γ E_inf,p.


Moreover, the averaged end-to-end energy consumption for one inference can be expressed as:

    eCAL_abs=eCAL_abs/γ.

As we observe from Eq. <ref>, as the number of inferences increases, the average energy consumption decreases and eventually approaches E_inf,p, which is depicted in the Fig. <ref>. This finding can also be utilized to measure how well the model is resilient toward performance loss from an energy consumption point of view. More specifically, with the same hardware setup, a model that requires less frequent retraining due to performance degradation will consume less energy than one that needs frequent retraining.



Finally, the proposed metric computing the corresponding energy consumption per bit over the lifecycle of the aiot with the currently deployed model as:

    eCAL   =
       eCAL_abs/B_T+α (2N_S+N_S,T+N_S,E)+ γ (B_T+3α N_I,P).

To further quantify this relationship, we first compare between Eq. <ref>, Eq. <ref>, and Eq. <ref>, which is depicted in Fig. <ref>. It shows that the energy consumption per bit of developing the model is 2.59× 10^-7[J/b], which has approximately 26 times higher than the operational lifecycle (9.89× 10^-9[J/b]). Furthermore, we can observe from the figure that the energy consumption per bit decreases with more inferences, dropping from 1.49× 10^-8 [J/b] for 100 inferences to 1.04× 10^-8 [J/b] for 1000 inferences which is 1.43 times improvement. It is also confirmed in Fig.<ref>, where it follows a similar trend as in Fig.<ref>. More specifically, as the number of the inferences in the current model increases, the energy efficiency of the system also increases. 












 
 
 
  








§ CARBON FOOTPRINT OF AIOT INFERENCE


The cf is defined as the product of energy consumption and ci, where ci indicates the amount of carbon dioxide equivalents (CO_2eq) emissions associated with generating electrical energy, accounting for all greenhouse gases by weighting each gas by its Global Warming Potential (GWP). This relationship is expressed as:

    CF[gCO_2eq] = E[kWh] · CI [gCO_2eq/kWh].



Each country has its own ci, which varies depending on its energy production methods. For example, in 2023, Finland derived 40% of its electricity from hydro and wind sources, with only 10% originating from high ci sources such as coal or gas. This indicates a predominantly low-carbon energy grid. In contrast, Germany, despite utilizing 44% renewable energy, still recorded a relatively high ci <cit.>. This is attributed to the fact that 38% of its electricity production relied on burning gas, oil, and coal, which are substantial contributors to greenhouse gas emissions. This is shown in Table <ref>, which provides a summary of the average ci values for different countries in 2023, along with the corresponding cf for end-to-end training and inference in an aiot system. The energy consumption is calculated by utilizing the aforementioned setup, along with Eq. <ref> and Eq. <ref>. We can see that the table reflects the the significant impact of ci on the cf. For example, deploying the same model in Germany results in greenhouse gas emissions that are 4.62 times higher than those in Finland. This highlights the direct correlation between the ci of a country and the environmental cost of technological operations, especially with the integration of ai into the wireless communication systems. 

Although the cf for both end-to-end training and inference phases are relatively low, it is important to note that these figures pertain to less complex models and energy-efficient transmission methods. In more realistic scenarios, where models may be more complex and energy demands higher, the cf can substantially increase.





 

Additionally, by considering the total energy consumption in Eq. <ref>. We can determine how much CO_2 is emitted into the atmosphere in different locations as the number of inferences increases, this is shown in Fig. <ref>.

The results highlight the significant variation in the cf due to differences in average ci among countries. For example, Finland, benefiting from its lowest average ci, exhibits the lowest CO_2 emissions among the five countries analyzed. Therefore, by leveraging lower ci methods to power the system, such as wind or hydro, we can move towards a more sustainable aiot system.













§ CONCLUSION AND FUTURE WORK


In this work, we proposed a novel metric, namely, eCAL, unlike traditional metrics that focus only on the energy required for transmitting or ai models. It can capture the overall energy cost of generating an inference in an aiot system during the whole lifecylce of a trained model. We propose a detailed methodology to determine the eCAL of an aiot system by breaking it down into various data manipulation components, such as data collection, storage, preprocessing, training, evaluation, and inference, and analyzing the complexity and energy consumption of each component. Our proposed metric demonstrates that the more a model is utilized, the more energy-efficient each inference becomes. For example, considering a simple MLP architecture, the energy consumption per bit for 100 inferences is 1.43 times higher than for 1000 inferences. Moreover, we assess the cf of the aiot system by calculating the equivalent CO_2 emissions based on energy consumption and ci across different countries. Our analysis, using 2023 renewable data, shows that cf varies deploying an aiot system in different geographic locations, indicating the importance of considering regional energy profiles and their associated cis in the deployment of AIoT systems. 

Through the proposed eCAL metric, this study provides foundations for understanding the energy consumption of aiot systems. The assumptions are that the model is based on MLPs which are a type of feed-forward neural architectures. Future work may focus on removing that assumption and extend the study to more complex architectures such as recurrent, convolutional, recursive, attention based, etc. and provide a relative complexity comparison of these categories. Furthermore, as virtualization permeates 6G networks, the impact of network slices on the data collection part could be derived. Finally,  the development of  dynamic energy management algorithms to optimize the cf in view of a more sustainable operation is relevant. 








§ ACKNOWLEDGEMENTS


This work was supported in part by the HORIZON-MSCA-IF project TimeSmart (No. 101063721), the European Commission NANCY project (No.101096456), and by the 
Slovenian Research Agency under grants P2-0016 and J2-50071.











./templates/IEEEtran




 Paper Title: An Explainable Vision Transformer with Transfer Learning Combined with Support Vector Machine Based Efficient Drought Stress Identification.





[1,2]Aswini Kumar Patraaswinipatra@gmail.com / akp@nerist.ac.in

3]Ankit Varshney 

[2]Lingaraj Sahools@iitg.ac.in

[1]Dept. of CSE, NERIST, India

[2]Dept. of BSBE, IIT Guwahati, India

[3]Dept. of Physics, IIT Guwahati, India


Early detection of drought stress is critical for taking timely measures for reducing crop loss before the drought impact becomes irreversible. The subtle phenotypical and physiological changes in response to drought stress are captured by non-invasive imaging techniques and these imaging data serve as valuable resource for machine learning methods to identify drought stress. While convolutional neural networks (CNNs) are in wide use, vision transformers (ViTs) present a promising alternative in capturing long-range dependencies and intricate spatial relationships, thereby enhancing the detection of subtle indicators of drought stress. We propose an explainable deep learning pipeline that leverages the power of ViTs for drought stress detection in potato crops using aerial imagery. We applied two distinct approaches: a synergistic combination of ViT and support vector machine (SVM), where ViT extracts intricate spatial features from aerial images, and SVM classifies the crops as stressed or healthy and an end-to-end approach using a dedicated classification layer within ViT to directly detect drought stress. Our key findings explain the ViT model's decision-making process by visualizing attention maps. These maps highlight the specific spatial features within the aerial images that the ViT model focuses as the drought stress signature.  Our findings demonstrate that the proposed methods not only achieve high accuracy in drought stress identification but also shedding light on the diverse subtle plant features associated  with drought stress. This offers a robust and interpretable solution for drought stress monitoring for farmers to undertake informed decisions for improved crop management.







An Explainable Vision Transformer with Transfer Learning
Combined with Support Vector Machine Based Efficient
Drought Stress Identification
    *
    
===========================================================================================================================================





§ INTRODUCTION

Drought stress adversely affects the growth, development and yield of plants. The rise in global temperature coupled with the growing demand for water severely depletes soil water reserves. Water deficit and temperature stress can cause damage to the reproductive growth phase of crops, resulting in a substantial reduction in yield <cit.>. Furthermore, prolonged exposure to water deficit stress particularly in the reproductive stage can significantly affect grain quality. Ensuring sustainable crop yields in the face of escalating drought conditions has become a great challenge <cit.>. The plant's response to drought stress is influenced by a number of factors such as stress duration, severity, genotype and its developmental stage <cit.>. In plants, drought stress in early phase induces reduced stomatal conductance, transpiration and net carbon assimilation, whereas mid- and late-phases result in visible leaf senescence, stagnant growth and wilting. The early detection of drought induced phenotypic changes in plants, facilitates in preventing the adverse impacts of drought from irreversible damage through timely irrigation and other management practices. The point based, contact type and destructive methods based estimation of drought stress impact on crops are limited by good approximation of representative area as well as sample size and hence, difficult to implement over larger agricultural production systems. On the contrary, image-based, non-invasive and high-throughput analyses of subtle phenotypical changes in plants due to drought and other stresses, provides an efficient means of monitoring  stress levels in crops and good prediction of drought stress impact <cit.>. Various imaging techniques and sensors are utilized to precisely capture stress responses, including simple red-blue-green (RGB) , thermal, multispectral, hyperspectral, or fluorescence imaging methods <cit.>. However, traditional image processing generates significant amount of variation in light intensity, shading, occlusion, etc. leading to variation in quality thereby increases complexity of image processing <cit.>. Machine learning (ML) techniques, on the other hand, solve such complex problems which are nonlinear in nature by identifying key feature to be extracted from the acquired image in real time, based on knowledge and/or domain expertise of the user. Hence, ML aids in better decision-making and informed actions in real-world scenarios with minimal human intervention <cit.>. Deep learning (DL) <cit.> techniques however, help in extracting different information from the raw image data during input training, by means of various convolutions which allows larger learning capabilities, that result in the best classification/regression and higher performance and precision. Convolutional Neural Networks (CNNs) have been developed for providing a much-detailed representation of the image features, for a more discriminative and detailed analysis of plant images, leading to highly accurate classifications <cit.>. Many studies have used deep CNN models to detect drought stress in affected plants, mostly solved either by finely tuned CNNs or by training from scratch. 

 Early work by Zhuang et al. <cit.> utilized a combination of segmentation, color, and texture feature extraction, employing a gradient boosting decision tree (GBDT) method to detect water stress in maize. However, a subsequent study using the same RGB dataset demonstrated that a deep convolutional neural network (DCNN) significantly outperformed GBDT in terms of performance <cit.>.
Sankararao et al. <cit.> developed a comprehensive pipeline for detecting water stress in groundnut canopies using hyperspectral imaging. The pipeline included steps for image quality assessment, denoising, and band selection, followed by classification using Support Vector Machine (SVM), Random Forest (RF), and Extreme Gradient Boosting (XGBoost/XGB). Transfer learning with a pre-trained DenseNet-121 model has been utilized to classify soybean color images into four levels of drought severity <cit.>. Similarly, a controlled dataset is created for chickpea crops to identify water stress at different stages—control, young seedling, and pre-flowering—using a CNN-LSTM hybrid model where CNNs that served as feature extractors and LSTM models for predicting water stress levels <cit.>.
The evaluation of various CNN models, including AlexNet, GoogLeNet, and Inception V3, for distinguishing between stress and non-stress conditions in water-sensitive crops such as maize, okra, and soybean, revealed that GoogLeNet achieved the highest identification accuracy <cit.>.
Chlorophyll fluorescence images of wheat canopies have been studied employing a multi-step process involving segmentation and feature extraction with a correlation-based gray-level co-occurrence matrix (CGLCM) and color features, followed by classification using different ML classifiers. The tree-based methods, particularly random forest (RF) and extra trees classifier, is found giving superior performance <cit.>.
A dataset of aerial images of potato canopies has been studied using a deep learning-based model for drought stress identification, leveraging various imaging modalities and their combinations <cit.>. An explainable deep learning framework was later proposed using the same dataset, based on CNN architectures and transfer learning, achieving improved accuracy <cit.>.
A regression approach is applied to predict the drought tolerance coefficient using hyperspectral images of tea canopies; the comparison of SVM, RF, and Partial Least Squares Regression (PLSR) revealed that SVM was the most effective <cit.>. Machine learning and deep learning methods were compared, including DNN, SVM, and RF, for drought stress detection using full spectra and first-order derivative spectra <cit.>. Goyal et al. <cit.> created a dataset of RGB images of maize crops and proposed a custom-designed CNN model that outperformed existing state-of-the-art CNN architectures in early drought stress detection.  However, Vision Transformers (ViTs) offer notable advantages over CNNs in capturing long-range dependencies and global context due to their self-attention mechanism, whereas CNNs rely on local receptive fields and often struggle to capture relationships across various parts of an image <cit.>. Additionally, ViTs exhibit greater flexibility with input image sizes and handle complex patterns and data variations more effectively <cit.>.

The self-attention mechanism in ViTs enables more accurate classification by considering the entire image context, which enhances accuracy and robustness compared to traditional CNNs <cit.>. For example, Dosovitskiy et al. demonstrated that ViTs could outperform CNNs in image classification tasks, highlighting their potential for diversified applications [<cit.>]. Recent studies have also applied ViTs to plant disease identification <cit.>.  In this study, we applied ViTs to drought stress identification to (a) achieve higher accuracy in detecting drought stress, b) leverage the ViT’s superior ability to handle complex patterns and variations inherent in field images captured in natural settings, ensuring the model’s practicality and relevance to real-world conditions, c) interprete model by utilizing ViTs to generate attention maps, which provide insights into the model’s decision-making process and contribute to the transparency and explainability of the predictions.




§ MATERIALS AND METHODS

In this section, we begin by introducing the experimental dataset. Next, we present the drought stress classification model using two different approaches. In the first approach, we employ the vision transformer with transfer learning. In the second approach, we propose a framework that utilizes a vision transformer as a feature extractor, followed by the integration of an SVM as the classifier. Additionally, we investigate the interpretability of the model by generating and analyzing the attention maps. This comprehensive use of the ViT elucidates how the spatial features of drought stress can be precisely identified. Finally, we discuss the performance metrics for the proposed approaches.


 §.§ Preparing the Data

The potato crop aerial images utilized in this study have been sourced from a publicly accessible dataset that encompasses multiple modalities <cit.>. Collected from a field at the Aberdeen Research and Extension Center, University of Idaho, these images serve as valuable resources for training machine learning models dedicated to crop health assessment in precision agriculture applications. Acquired using a Parrot Sequoia multi-spectral camera mounted on a 3DR Solo drone, the dataset features an RGB sensor with a resolution of 4,608 × 3,456 pixels and four monochrome sensors capturing narrow bands of light wavelengths: green (550nm), red (660nm), red-edge (735nm), and near-infrared (790nm), each with a resolution of 1,280 × 960 pixels. The drone flew over the potato field at a low altitude of 3 meters, with the primary objective of capturing drought stress in Russet Burbank potato plants attributed to premature plant senescence.




The dataset consists of 360 RGB image patches in JPG format, each measuring 750×750 pixels. These patches were derived from high-resolution aerial images through cropping, rotating, and resizing operations. The dataset is divided into a training subset of 300 images and a testing subset of 60 images. Ground-truth annotations, provided in both XML and CSV formats, indicate regions of healthy and stressed plants marked by rectangular bounding boxes. These annotations were manually created using the  LabelImg software. An example RGB image and its annotated regions are shown in Fig. <ref>. The testing subset is independent of the training subset and comes from different aerial images. Additionally, the dataset includes corresponding image patches from spectral sensors with red, green, red-edge, and near-infrared bands, each sized 416×416 pixels. However, we are only utilizing the RGB images due to the limitations of the low-resolution monochromatic images.

We utilized the augmented dataset prepared by Butte et al., expanding the original set from 300 to 1,500 images for training. The test set comprised 60 images. From each augmented and test image, annotated windows (i.e., rectangular bounding boxes) were extracted, creating separate folders for "healthy" and "stressed" images. These folders served as input for our proposed model. The final count of training images for the "stressed" and "healthy" classes is 11,915 and 8,200, respectively. The test set, consisting of 60 images, yielded 401 healthy and 734 stressed images. We evaluated our method on this test set and further assessed the model performance through 5-fold cross-validation.



 §.§ Vision Transformer (ViT)
 



A vision transformer (ViT) is a type of neural network architecture that has revolutionized the field of computer vision <cit.>. Unlike traditional convolutional neural networks (CNNs), which process images pixel by pixel, ViT processes the input images in a sequential manner by dividing them into fixed-size patches, linearly embedding these patches, and then applying self-attention mechanisms for capturing global dependencies <cit.>. The VIT architecture used for our work is inspired by Dosovitskiy et al.<cit.> and depicted in Fig <ref>. It processes images by dividing them into fixed-size embedded patches, linearly transforming these patches, and treating them as sequences. The transformer architecture is then applied, incorporating multi-head self-attention mechanisms <cit.> that enable the model to capture long-range dependencies within the image. Layer normalization is applied before and after the multi-head attention, ensuring stable training, and a Multilayer Perceptron (MLP) head is added to the transformer's global representation for task-specific processing. 



  §.§.§ ViT Architecture
 The proposed ViT architecture is based on the layers in <cit.> and Dosovitskiy et al.<cit.> and comprises five main components: patch embedding, positional encoding, transformer encoder , normalization layer and a classification head. This is illustrated in Fig <ref>.

  
  * Patch Embedding: Input images are divided into fixed-size patches, which are then linearly embedded to create a sequence of embeddings.
  
  
  * Positional Encoding: To capture spatial information, positional encodings are added to the patch embeddings, allowing the model to understand the relative positions of different patches.
  
  
  * Transformer Encoder: The embedded patches are fed into a Transformer Encoder, which is the core component of the ViT architecture. This encoder consists of twelve identical encoder layers stacked together. Each attention layer analyzes the relationships between pairs of patches, allowing the model to understand how different image regions interact and influence each other. Each attention layer consists of the following:
        
            
  * Multi-Head Self-Attention: This mechanism allows the model to weigh the importance of different parts of the image. It captures global dependencies between the patches.
            
  * MLP (Multi-Layer Perceptron) Block: This block introduces non-linearity to the network and further processes the information from the attention layer.
            
  * Normalization Layers: Layer normalization is applied after the multi-head attention and MLP blocks to stabilize training.
            
  * Dropout: The model uses dropout at two levels: within the attention mechanism and after the MLP block. Dropout is used to prevent over-fitting by randomly dropping out neurons during training.           
        
   
  * Normalization Layer: Layer normalization applied to the output of the encoder serves several crucial purposes: Reduces internal co-variate shift, improves gradient flow, acts as regularization, handles varying input distributions and accelerates convergence.     
    
  
  * Classification Head: This head typically consists of a simple MLP layer that maps the feature representation to the desired output, such as class probabilities.






  §.§.§ Information Processing in ViT

At the core of the Vision Transformer (ViT) is the concept of a token, which plays a crucial role in the model's ability to process and understand images. A token is a fixed-size vector that represents a small patch of the input image. These tokens are at the core of the Vision Transformer model, forming the input sequence to the Transformer layers. This token-based approach enables the model to process and understand the image by focusing on the relationships between these patches through self-attention. The detailed explanation is given below.


    
  * Dividing the Image into Patches:
    
        
  * The input image is divided into smaller, fixed-size patches. For example, an image of size 224x224 pixels might be divided into patches of size 16x16 pixels, resulting in (224/16)^2 = 196 patches.
    
    
    
  * Flattening and Embedding:
    
        
  * Each image patch is then flattened into a one-dimensional vector. For instance, a 16x16 patch with 3 color channels (RGB) would be flattened into a vector of length 16 × 16 × 3 = 768.
        
  * These flattened vectors (patch representations) are then linearly embedded into a higher-dimensional space. This is typically done using a learnable linear projection, transforming each vector into a fixed-size embedding, say of dimension 768.
    
    
    
  * Tokens:
    
        
  * After linear embedding, each flattened and embedded patch becomes a "token." In this example, the image is transformed into a sequence of 196 tokens, each representing a 16x16 patch of the original image.
    
    
    
  * Adding Positional Encoding:
    
        
  * Since the transformer model does not inherently understand the order or position of tokens, positional encodings are added to each token to incorporate information about its original position in the image. This helps the model understand spatial relationships between patches.
    
    
    
  * Processing by Transformer Layers:
    
        
  * These tokens are then processed by the Transformer layers, which include self-attention mechanisms. The self-attention mechanism computes relationships between these tokens to understand how different parts of the image relate to one another.
    




 §.§ ViT with Transfer Learning

The proposed framework, as shown in Fig <ref> effectively combines transfer learning, the power of the Vision Transformer and attention-based interpretability to address the challenging task of drought stress identification in potato crop images captured in natural settings. A core component of this approach is the utilization of pre-trained weights. This technique, known as transfer learning, involves leveraging knowledge gained from solving one problem (often a large-scale image classification task) and applying it to a different but related problem. By employing pre-trained weights, the model can benefit from the rich feature representations learned from a massive dataset, accelerating training and potentially improving performance. Using this the ViT is trained over multiple epochs using a combination of binary cross-entropy loss, Adam optimizer and learning rate tuning. Experimenting with different learning rates is essential to find the optimal value for convergence and generalization. Eventually, the classification layer is responsible for making the final prediction. It takes the output of the ViT model and maps it to two classes: "healthy" and "stressed." This layer typically consists of a fully connected neural network with a sigmoid activation function for binary classification.

To enhance model interpretability, the architecture incorporates a mechanism to capture attention weights. Attention weights reveal which parts of the input image the model focuses on when making a decision. By visualizing these weights, researchers can gain insights into the model's decision-making process and identify key image features that contribute to the classification. 



  §.§.§ Attention Maps
 





Attention maps provide insights into how the model focuses on different parts of the image during the self-attention mechanism. This mechanism allows the model to selectively attend to specific areas while processing visual information. The image is first divided into patches. Self-attention then helps the model prioritize relevant patches and their relationships for effective feature extraction.

Attention maps act as a visual representation of the weights assigned by the self-attention mechanism to each patch. These maps can be visualized for each self-attention layer within the ViT model, as each layer progressively learns more intricate relationships between these image features. Higher weights indicate a greater focus on a specific patch and its connection to others. By analyzing these maps, we can essentially see the model's "thought process" during image understanding. We can identify which regions it prioritizes for information extraction. The following section highlights the computation behind the attention mechanism <cit.>. 

At its core, self-attention computes a weighted sum of the values (features) based on the similarities (attention scores) between different positions in the sequence. This is achieved through three learnable matrices: Query (Q), Key (K), and Value (V). 

Query Matrix (Q): The query matrix is responsible for capturing the information about the current token being processed. It learns to encode the features of the token in a format suitable for comparison with other tokens. Each token in the sequence is associated with a query vector, which represents its characteristics in the context of the entire sequence.

Key Matrix (K): The key matrix holds information about the relationship between the current token and other tokens in the sequence. It learns to encode the features that determine how relevant each token is to the current token.

Value Matrix (V):  

The key matrix holds information about the relationship between the current token and other tokens in the sequence. It learns to encode the features that determine how relevant each token is to the current token.

Given a sequence of tokens X = [x_1, x_2, ..., x_n], the attention scores A are computed as:


    A = softmax( QK^T/√(d_k))


where Q = XW_Q, K = XW_K, V = XW_V, and W_Q, W_K, W_V are weight matrices. d_k represents the dimensionality of the key vectors.

Once the attention scores are computed, they are used to compute a weighted sum of the values:


    Attention(Q, K, V) = A · V


where A is the attention matrix.


























To capture different relationships between tokens, ViT (and Transformers in general) often employ multiple attention heads. Each head learns different sets of Q, K, V weight matrices and computes separate attention scores and weighted sums. The results from all heads are concatenated and linearly transformed to maintain a consistent output dimension.
Since self-attention does not inherently consider the order of tokens, positional encoding is typically added to the token embeddings to provide positional information.
 
Algorithm for Capturing Attention Weights:
To enhance model interpretability, the architecture incorporates a mechanism to capture attention weights. The algorithm <ref> outlines the steps during the training of ViT to capture and utilize attention weights for the drought identification task. The key components of this class include the initialization, forward pass, attention weight capture, output size determination, and retrieval of attention weights. By visualizing these weights, researchers can gain insights into the model's decision-making process and identify key image features that contribute to the classification. 

Initialization
The VisionTransformerBinary class is initialized with a pre-trained Vision Transformer model passed as . During initialization, the model assigns the provided  to its own  attribute. Additionally, it sets up a fully connected (linear) layer () with a size appropriate to the output of the Vision Transformer. This layer is responsible for converting the output of the Vision Transformer into a format suitable for binary classification. An empty list  is also initialized to store the attention weights captured during the forward pass.

Forward Pass
The  function is central to the operation of the VisionTransformerBinary class. When an input image  is passed through the model, the function first clears any previously stored attention weights. This ensures that the attention weights list is fresh and only contains data relevant to the current input.

Next, the function registers hooks on the self-attention layers of each block in the Vision Transformer. These hooks are responsible for capturing the attention weights during the forward pass. For each block in the Vision Transformer's encoder layers, the self-attention layer is accessed, and a hook is registered to capture its attention weights using the  helper function. The hooks are stored in a list to facilitate their removal later.

With the hooks in place, the input image is passed through the Vision Transformer and then through the fully connected layer. This produces the model's output for the given input. After the forward pass is complete, the hooks are removed to free up memory, ensuring that they do not persist and interfere with future operations.

Attention Weights Capture
The  helper function is designed to capture the attention weights during the forward pass. It is triggered by the hooks registered on the self-attention layers. This function receives the module, input, and output as arguments. From the input, it extracts the query, key, and value components, which are essential for computing the attention weights. These components, along with the output, are appended to the  list. By capturing these components, the model can later compute and analyze the attention scores, which provide insights into the regions of the input image that the model focuses on during classification.

Output Size Determination
The  helper function determines the output size of the Vision Transformer model. This function performs a forward pass with a zero tensor of appropriate dimensions through the Vision Transformer. By doing so, it captures the shape of the output tensor produced by the Vision Transformer. The size of the last dimension of this output tensor is then returned, which is used to initialize the fully connected layer with the correct input size.

Retrieval of Attention Weights
The  function provides a simple interface to retrieve the captured attention weights. It returns the  list, allowing external components or functions to access the attention weights for further analysis or visualization.

The modular design of the class, with separate functions for initialization, forward pass, attention weight capture, output size determination, and retrieval, provides a clear and maintainable structure. This design facilitates the integration of attention-based insights into the classification process, enhancing the interpretability and performance of the model. 






 §.§ Integrating Vision Transformer (ViT) and Support Vector Machine (SVM)


This framework focuses on combining a vision transformer (ViT)  and a support vector machine (SVM) within a three-step approach for effective stress identification.

  
  * Feature Extraction: For each image in the dataset, extract the final hidden state or pooled representation from the ViT model to obtain a feature vector. Let 𝐗 represent the set of embedded features extracted from a dataset of images, and 𝐲 represent the corresponding class labels.
  
  
  * Training SVM: Train an SVM classifier using the extracted features 𝐗 and their corresponding class labels 𝐲.
  
  
  * Classification: For a new, unseen image, extract features using the pre-trained ViT model and use the trained SVM classifier to predict its class.



  §.§.§ Support Vector Machine (SVM)

Support Vector Machines <cit.> aim to find a hyperplane that best separates a given set of data points into different classes. Given a set of labeled data points (𝐱_1, y_1), (𝐱_2, y_2), …, (𝐱_n, y_n) where 𝐱_i is the feature vector of the i-th data point, and y_i is its corresponding class label (y_i ∈{1, 0} for binary classification), SVM seeks to find a hyperplane defined by the equation:


    𝐰·𝐱 + b = 0


where 𝐰 is the weight vector and b is the bias.

The goal is to maximize the margin, which is the distance between the hyperplane and the nearest data point from each class. The margin is computed as the perpendicular distance from a data point 𝐱_i to the hyperplane:


    margin = 1/𝐰· | 𝐰·𝐱_i + b |


Subject to the constraint that for all data points:


    y_i (𝐰·𝐱_i + b) ≥ 1


This constraint ensures that data points are correctly classified and lie on the correct side of the hyperplane.

The SVM optimization problem can be formulated as:


    Minimize  1/2𝐰^2


Subject to the constraints:


    y_i (𝐰·𝐱_i + b) ≥ 1   for all  i


This is the primal form of the optimization problem. However, the SVM problem is often reformulated in its dual form, which introduces Lagrange multipliers α_i to handle the constraints. The dual formulation is:


    Maximize  ∑_i=1^nα_i - 1/2∑_i=1^n∑_j=1^nα_i α_j y_i y_j 𝐱_i ·𝐱_j


Subject to the constraints:


    0 ≤α_i ≤ C   for all  i



    ∑_i=1^nα_i y_i = 0


The solution to the dual problem provides the values of α_i, and the weight vector 𝐰 and bias b can be obtained from these values.

In cases where the data is not linearly separable, SVM can be extended to handle non-linear decision boundaries using the kernel trick. The feature space is implicitly mapped to a higher-dimensional space, making it possible to find a linear separating hyperplane in that space.

Classification is performed based on the decision function derived from the trained model. Once the SVM model is trained with a set of support vectors, it identifies a hyperplane that best separates different classes in the feature space. The decision function for a new data point 𝐱 is given by:


    f(𝐱) = 𝐰·𝐱 + b


Here, 𝐰 is the weight vector, and b is the bias term. The sign of f(𝐱) determines the predicted class:


    Class 1,    if  f(𝐱) > 0 
    Class 0,    if  f(𝐱) < 0


The magnitude of f(𝐱) provides a measure of how far the data point is from the decision boundary. Larger magnitudes indicate greater confidence in the classification.

The key role of support vectors in this process is that they are the data points lying closest to the decision boundary. Support vectors effectively determine the position and orientation of the hyperplane. The optimization process in SVM aims to maximize the margin between the classes, and support vectors are the data points defining the edges of this margin. In practice, many data points do not significantly contribute to the definition of the decision boundary. Only the support vectors, with non-zero Lagrange multipliers α_i in the dual formulation, are crucial for determining the hyperplane. This property makes SVM memory-efficient and computationally faster, especially in high-dimensional spaces.`



  §.§.§ ViT+SVM Framework

The process begins with input images that are fed into the ViT. The ViT, pre-trained on a vast dataset, is adept at extracting meaningful features from the images. Once the features are extracted by the ViT, they are compiled into a feature matrix, which also includes the corresponding labels indicating whether the plants in the images are healthy or stressed. This matrix forms the input to the SVM, a robust classifier known for its effectiveness in handling high-dimensional data. The SVM is trained to discern between the two classes—healthy and stressed—based on the features provided. Finally, the trained SVM predicts the class of new images, categorizing them as either healthy or stressed. The entire approach is depicted in Fig <ref>.
The efficacy of this framework is evaluated using a designated test set and k-fold cross-validation. 


 §.§ Performance Evaluation Metrics

The model's performance underwent assessment using various evaluation metrics, including accuracy, precision, recall (sensitivity) and the Receiver Operating Characteristic (ROC) curve. These metrics are computed based on the counts of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN), which collectively form a 2x2 matrix known as the confusion matrix. In this matrix, TP and TN indicate the accurate predictions of water-stressed and healthy potato crops, respectively. FP, termed as type 1 error, denotes predictions where the healthy class is inaccurately identified as water-stressed. FN, referred to as type 2 error, represents instances where water-stressed potato plants are incorrectly predicted as healthy. The classification accuracy is a measure of the ratio between correct predictions for both stressed and healthy images and the total number of images in the test set.


    Accuracy = True Positive + True Negative/Total Population



    Precision = True Positive/True Positive + False Positive



    Recall = True Positive/True Positive + False Negative



    F1-score = 2 ·Precision·Recall/Precision + Recall



Cross-validation is crucial in model development for two key reasons: it helps prevent over-fitting by assessing a model's performance across different subsets of the data, and it ensures the model's generalization ability, providing a reliable estimation of its effectiveness under various conditions. The model was trained and evaluated using k-fold cross-validation, a robust technique for assessing the generalization performance of the model. In each iteration of the k-fold cross-validation process, the dataset was partitioned into k folds, and the model was trained on k-1 folds while being validated on the remaining fold. This process was repeated k times, ensuring that every fold had the opportunity to serve as the validation set. For each fold, the Receiver Operating Characteristic (ROC) curve was plotted, illustrating the trade-off between true positive rate and false positive rate at various thresholds. After completing the k-fold cross-validation, the individual ROC curves were aggregated, and the mean ROC curve was calculated and plotted. AUC (area under the curve) measures the entire two-dimensional area underneath the entire ROC curve. AUC provides an aggregate measure of performance across all possible classification thresholds. One way of interpreting AUC is as the probability that the model ranks a random positive example more highly than a random negative example. AUC ranges in value from 0 to 1. A model whose predictions are 100% wrong has an AUC of 0.0; one whose predictions are 100% correct has an AUC of 1.0.This comprehensive approach provides a more reliable estimation of the model's performance, capturing its consistency across different subsets of the data and enhancing the overall assessment of its predictive capabilities.



§ RESULTS AND DISCUSSION

In this section, we present the experimental results of our proposed model for identifying drought stress in potato crop field images. First, we distinguish between healthy and stressed images. Then, we identify the spatial features responsible for the stress. Our experiments with the proposed Vision Transformer (ViT) framework were conducted in three ways:

    
  * ViT with Transfer Learning: Leveraging pre-trained weights.
    
  * ViT+SVM with Pre-trained Weights.
    
  * ViT+SVM with Optimal Weights. The optimal weights are the ones at which the model performs best while executing the ViT with pre-trained weights in the first case.


The PyTorch library and it's various sub-packages are leveraged for deep learning functionalities. In addition to this, sklearn, pandas, numpy packages are used for developing, training, and evaluating deep learning models, as well as for performing data analysis and pre-processing. The data pre-processing involves resizing images to (224, 224) pixels and transforming them into tensors. 




 §.§ Performance of ViT with Transfer Learning

We relied on PyTorch, a core library for deep learning, which provides the essential tools for tensor computations and neural network construction. Specifically,  and  were used to build and train the neural network. To handle image data, we used Torchvision's  for accessing pre-trained architectures and  for preprocessing images. Additionally, TQDM was employed to create progress bars, enhancing visibility into the training process. For data analysis and manipulation, we leveraged scikit-learn for machine learning utilities and Pandas for managing structured data.

The dataset preparation involved mapping class names to numerical labels using , where 'healthy' images were assigned the label 0 and 'stressed' images were assigned the label 1. We applied image transformations to resize the images to 224x224 pixels and convert them into tensor format, ensuring compatibility with the ViT model. The datasets were loaded using  from Torchvision, which facilitated the organization of images into training and testing sets. Data loaders were then created to handle batching, shuffling, and efficient data retrieval during training and validation, with a batch size of 128.

To adapt the Vision Transformer (ViT) architecture (as depicted in Fig <ref>) for our specific task of binary classification, we began by configuring the model using the models.vit_b_16 variant, setting pretrained=False to start with a clean slate. This approach allowed us to build the model from scratch without any influence from pre-existing learned features. Subsequently, we loaded custom pre-trained weights into the ViT model to realize the approach as shown in Fig <ref>. This step was crucial as it transferred learned representations from a previously trained model to our current architecture, leveraging prior knowledge to enhance performance. A custom class was defined to extend the ViT model by adding a fully connected layer for binary classification. This class also included methods to capture attention weights, which are essential for understanding the model's focus on different image regions.

The training process involved setting up a binary cross-entropy loss function to evaluate the performance of the classification model. We used the Adam optimizer with a learning rate of 0.001 and 0.0009 to update the model's weights. The training loop consisted of multiple epochs, during which the model was trained on the training dataset and validated on the test dataset. Training and validation losses were computed to assess the model's performance and adjust parameters accordingly. Throughout the process, attention weights were captured to provide insights into the model's decision-making process.






The learning curve typically plots both the training loss and validation loss curves on the same graph.
This allows to analyze how the model is performing on both the data it's trained on (training loss) and unseen data (validation loss).
The plot shown in Fig <ref> compares the training losses over the first 10 epochs for VIT trained from scratch versus VIT trained with pre-trained weights. The Vision Transformer (ViT) with pre-trained weights demonstrates a notably faster and more consistent reduction in loss compared to the ViT trained from scratch. This indicates superior initial performance and quicker convergence. As a result, we chose to further experiment with pre-trained weights rather than training the model from scratch.

The training loss is plotted in Fig <ref> for 20 epochs when using a Vision Transformer (ViT) with pre-trained weights and a learning rate of 0.001. The goal of training a machine learning model is to minimize the loss on the validation set. In an ideal scenario, the training loss and validation loss will both decrease as the model trains. However, this is not always the case.
In this graph, the training loss decreases steadily over the course of training. This suggests that the model is learning the training data well. However, the validation loss starts to increase after a certain number of epochs, specifically after epoch 8. This is a sign of over-fitting. Over-fitting is a problem that occurs when a model learns the training data too well, and it is no longer able to generalize to unseen data.

The training loss is plotted in Fig <ref>  for 20 epochs using a Vision Transformer (ViT) with pre-trained weights. The learning rate is 0.001 for epochs 1 to 10 and 0.0009 for epochs 11 to 20. The training loss decreases steadily, indicating the model is learning the training data well. However, the validation loss fluctuates and increases after epoch 6, suggesting over-fitting. The training loss is plotted in Fig <ref> for 20 epochs using a Vision Transformer (ViT) with pre-trained weights. The learning rate is 0.001 for epochs 1 to 8, with no dropout. For epochs 9 to 20, the learning rate remains 0.001, with an attention dropout of 0.2 and an MLP dropout of 0.1. The training loss decreases steadily, but the validation loss fluctuates and increases after epoch 8, indicating over-fitting once dropout is applied. The training loss is plotted in Fig <ref> for 20 epochs using a Vision Transformer (ViT) with pre-trained weights. The learning rate is 0.001 for epochs 1 to 8 and 0.0009 for epochs 9 to 20. An attention dropout of 0.2 and an MLP dropout of 0.1 are applied for epochs 9 to 20. The training loss decreases steadily, but the validation loss fluctuates and increases after epoch 8, suggesting over-fitting as dropout is applied and the learning rate is adjusted. 

In summary, all figures show a decrease in training loss, indicating the model is learning from the training data. Validation loss generally starts to increase or plateau after a certain number of epochs, suggesting overfitting is a common issue in these experiments. The introduction of dropout in Figures <ref> and <ref> seems to have exacerbated the over-fitting problem. However, the optimal result for the specified test set, reaching an accuracy of 0.929, is attained with a learning rate of 0.001 and without applying attention dropout or MLP dropout. The Vision Transformer (ViT) with transfer learning demonstrated impressive performance, achieving a precision of 0.95 and recall of 0.94 for stressed images, as well as a precision of 0.89 and recall of 0.91 for healthy images. The F1-scores for stressed and healthy images were 0.95 and 0.90, respectively, highlighting the model’s strong capability in accurately classifying both categories.



  §.§.§ Analyzing Attention Maps

Visualizing attention weights provides insights into how the ViT focus on different parts of an input during processing. By examining these weights, researchers and practitioners can understand the model's decision-making process, diagnose potential biases, and improve interpretability.

The following pseudocode outlines a systematic approach to calculate and visualize attention weights from a Vision Transformer model. This process involves capturing attention weights during the forward pass, computing attention scores, and generating visual representations of these scores.

Initialization:
The process begins by initializing an empty list to store the attention weights that will be captured during the forward pass of the Vision Transformer model. This list will later be used to compute and visualize the attention maps.

Forward Pass and Capture Attention Weights:
The next step involves iterating through each layer of the Vision Transformer model. For each layer, a hook is registered to capture the attention weights. The input image is then passed through the Vision Transformer to compute the output features. This stage ensures that attention weights are collected during the forward pass for later analysis.

Calculate Attention Score:
After capturing the attention weights, the algorithm processes each weight to compute the attention scores. This involves extracting the query, key, and value tensors from the hook outputs. The attention score is computed as per the principles discussed in section <ref>. The attention scores are then normalized to ensure they are in a range suitable for visualization.

Visualize Attention Maps:
In the visualization phase, each normalized attention map is resized to match the dimensions of the input image. A colormap (e.g., 'hot') is applied to the attention map to highlight areas of high attention. The attention map is then overlaid on the original image to create a visual representation of where the model is focusing.

Display:
Finally, both the original image and the overlaid attention maps are displayed, allowing for an interpretation of how the Vision Transformer model is making its decisions based on different regions of the input image.







The stressed image along with the corresponding attention maps from the 12 encoder blocks of the Vision Transformer (ViT) is shown in Fig <ref>. Several key observations can be made from these attention maps, including spatial relevance, hierarchical processing, interpreting model decisions, visualization of learned features, and using them as a diagnostic tool for model improvement. Each attention map is resized and overlaid on the original image, with colors indicating the attention intensity:

    
  * Red/Hot Colors: Indicate areas of high attention.
    
  * Yellow/Warmer Colors: Show areas of moderate  attention.
    
  * Dark/Cold Colors: Represent areas of low attention.
    




Spatial Relevance: The spatial relevance can give insights into which parts of the image the model finds important for differentiating between classes. The observations from the Fig <ref> are as follow:

    
  * Layer 1: The model's attention is broadly spread with higher intensity in some central regions. This indicates the model initially focuses on larger areas, possibly capturing the general structure of the image.
    
  * Layer 2-3: Attention starts to become more localized, with specific regions of high attention emerging. The model begins to identify and focus on more distinct features.
    
  * Layer 4-7: Attention further refines, highlighting specific areas more sharply. These layers likely focus on intermediate features that are critical for further processing.
    
  * Layer 8-12: Attention becomes very concentrated on specific, smaller regions. The model is likely focusing on fine-grained details and specific objects or patterns.

In the initial layers, the model's attention is broadly spread, capturing the overall structure of the image. As we move to higher layers, the attention becomes more localized, focusing on specific features and objects.

Hierarchical Processing: As we move through the layers of the ViT, attention maps can show how the model progressively refines its understanding of the image. In the lower layers, attention is distributed across large regions, capturing global context. As we ascend through the layers, the attention narrows down to more specific features, highlighting finer details and important objects within the image. This hierarchical processing is crucial for the model to effectively balance global and local information.

Interpreting Model Decisions:
By visualizing attention maps, we can interpret why the model made certain predictions. For instance, if the attention maps highlight specific objects or patterns in an image, it suggests that those elements influenced the classification decision. This interpretability can help validate the model’s decisions and identify potential biases or weaknesses.

Visualization of Learned Features:
The attention maps provide a visual representation of the features learned by the ViT. Unlike abstract feature vectors, these maps directly relate model activations to spatial locations in the input image. This visualization helps in understanding how the model processes visual information and forms its internal representations.

Diagnostic Tool for Model Improvement:
Analyzing attention maps can serve as a diagnostic tool for improving model performance. By examining where the model attends and comparing it with ground truth or human perception, we can identify areas where the model might be lacking or where it might over-emphasize certain features. This feedback loop can guide model refinement and training strategies.

In summary, attention maps in image classification tasks with Vision Transformers provide a transparent view into the inner workings of the model, highlighting which parts of the input image contribute most to its decision-making process. 


 §.§ Performance of ViT+SVM

We used a pre-trained Vision Transformer (ViT) model to extract features from both the training and testing datasets. These extracted features were then utilized by an SVM to identify stress, aiming to distinguish between stressed and healthy images. The implementation was done using the PyTorch framework, leveraging both its core library and the torchvision module. Key libraries such as torch, torchvision, csv, pandas, and scikit-learn were imported to facilitate feature extraction, data transformations, file handling, and classification tasks.

The pre-trained ViT model weights were sourced from a specified path and loaded into an initialized ViT model. We used the ViT-B/16 model, whose architecture was previously discussed, and obtained it from torchvision.models. The model's weights were updated using torch.load and subsequently, the model was set to evaluation mode to facilitate feature extraction. Data transformations were defined to standardize the input images. Each image was resized to 224x224 pixels and converted into a tensor. Data loaders were created for both training and testing datasets , with a batch size of 32. The feature extraction process was encapsulated in a specific function. This function is called twice, once for the training data loader and once for the test data loader. This extracts features from each image in the respective datasets and saves them along with their corresponding labels to separate CSV (comma-separated values) files. Feature extraction was performed in a no-gradient context to optimize performance. The parameters used are mention in Table <ref>.




We extracted features from the ViT in two scenarios: firstly, using the pre-trained weights of ViT-B/16, which were trained on ImageNet; and secondly, by training the ViT from scratch as detailed in Section 3, starting with the ImageNet initial weights up to epoch 8. These extracted features were separately fed as inputs to an SVM to evaluate performance in both scenarios. We conducted 5-fold cross-validation to determine the ROC curve, mean AUC, and mean accuracy. Additionally, we examined both cases using the specified dataset provided in the original data repository.




Table <ref> shows the accuracy and AUC (Area Under the Curve) of approaches: ViT+SVM (Pre-trained Weights) and ViT+SVM (Optimal Weights). The models were evaluated on a specified test set and using 5-fold cross-validation. ViT+SVM (Optimal Weights) outperforms ViT+SVM (Pre-trained Weights) in terms of both mean accuracy (0.95 vs. 0.88) and mean AUC (0.96 vs. 0.95) with 5-fold cross-validation. Though, both the approaches achieved the same accuracy on the specified test set, but ViT+SVM (Optimal Weights) generalizes better to unseen data and suggests potentially less over-fitting as evidenced by its higher mean accuracy and AUC with 5-fold cross-validation. Upon examining the mean RoC curves given in Fig <ref>, it is evident that ViT+SVM (Optimal Weights) is performing consistently as the folds have a small variance in AUC (0.01) and each fold has a high AUC (> 0.95). On the other hand, ViT+SVM (Pre-trained Weights) have a mean AUC of 0.95 and mean variance of 0.02.





















 §.§ Comparison of the Models




Table <ref> presents the performance metrics of various models on a specific dataset. The comparison shows that each model has its strengths and is suitable for different scenarios.

Performance on Specified test set:
    ViT with Transfer Learning consistently performs best across most metrics, making it the most balanced and robust model for this task.
    ViT+SVM (Optimal Weights) excels in precision for the stressed class and recall for the healthy class, making it ideal where high precision in detecting stressed instances and high recall in identifying healthy instances are crucial.
    ViT (Pre-Trained Weights) provides a balanced performance but does not outperform the other models in any specific metric.



K-fold cross validation: 
The ViT with Transfer Learning model achieved an average accuracy of 86.8%, with F1-scores of 85.6% for the "Healthy" class and 88.8% for the "Stressed" class. The model demonstrated consistent performance across the five folds, although there was a slight dip in accuracy in the fifth fold (83%).

The ViT with SVM model achieved a significantly higher average accuracy of 95%, with F1-scores of 93.6% for Class 0.0 and 95.6% for Class 1.0. This model consistently performed well across all folds, with the lowest accuracy being 92.7% and the highest reaching 95.8%.




The comparative results, as shown in Fig <ref> indicate that the ViT with SVM model outperforms the ViT with Transfer Learning model in terms of both accuracy and F1-scores. The average accuracy of the ViT with SVM model is substantially higher (95% compared to 86.8%). Furthermore, the F1-scores for both classes are also higher in the ViT with SVM model, indicating better classification performance for both "Healthy" and "Stressed" classes.

The five-fold accuracies for the ViT with SVM model show less variability and higher consistency compared to the ViT with Transfer Learning model, suggesting that the integration of SVM helps in stabilizing the model's performance across different data splits.

In conclusion, integrating Support Vector Machine with Vision Transformer (ViT) significantly enhances the model's performance compared to using transfer learning alone. The ViT with SVM model demonstrates superior accuracy and F1-scores, making it a more reliable choice for classification tasks. The higher consistency across folds further reinforces the robustness of the ViT with SVM model. This comparative analysis highlights the benefits of combining advanced deep learning architectures with classical machine learning techniques for improved performance in identifying drought stress.


§ CONCLUSION

Drought stress represents a severe threat to crop yield and quality, disrupting normal plant growth and survival rates. Detecting early signs of drought stress is crucial for effective crop management and intervention. Traditional methods, primarily reliant on Convolutional Neural Networks (CNNs), have made significant strides in capturing spatial hierarchies in image data. However, Vision Transformers (ViTs) offer a compelling alternative by leveraging self-attention mechanisms to capture long-range dependencies and complex spatial relationships, thus enhancing the detection of subtle drought stress indicators.

Our study successfully addressed the challenge of stress identification using smaller datasets by harnessing the feature extraction capabilities of Vision Transformers combined with the classification prowess of SVM. Unlike conventional CNN architectures, Vision Transformers utilize self-attention mechanisms to effectively capture relationships across different parts of the image, enabling them to model long-range dependencies, which is particularly advantageous for complex datasets where traditional CNNs may struggle to capture global context. Moreover, Vision Transformers can handle images of varying resolutions without necessitating architectural modifications, enhancing their flexibility in handling diverse datasets. Our framework also emphasized explainability by generating attention maps, which provide insights into the model’s focus areas within the images, thereby offering transparency in its decision-making process.

However, our comparative analysis revealed that integrating SVM with ViT significantly enhances classification performance. The ViT+SVM model achieved a higher average accuracy of 95%, with F1-scores of 93.6% for stressed images and 95.6% for healthy images, outperforming the ViT with transfer learning approach. This integration not only improved accuracy but also demonstrated greater consistency across different data splits, highlighting the robustness of the ViT+SVM model.

In summary, our study illustrates the superior performance of combining Vision Transformers with Support Vector Machines over traditional transfer learning methods. This approach provides a more reliable and accurate solution for drought stress identification, while the attention maps offer valuable insights into the model’s decision-making process. The findings affirm the potential of advanced deep learning and machine learning techniques in enhancing agricultural practices and decision-making, paving the way for improved crop management strategies.
 

unsrt







 Paper Title: PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learning.


Cloud Collision Signatures in the Central Molecular Zone

    Felix D. Priestley^1
    July 2024
=========================================================

§ ABSTRACT 
Large Language Models (LLMs) encounter significant challenges in continual learning due to catastrophic forgetting, where new information overwrites previously acquired knowledge. This limitation leads to substantial environmental and economic waste. In this study, we introduce the PMoE, Progressive Mixture of Experts with Asymmetric Transformer, which aims to minimize forgetting by utilizing an asymmetric design with shallow layers dedicated to general knowledge and deep layers for new knowledge. 
PMoE incorporates progressively added experts in deep layers and a router that allocates new knowledge to the appropriate experts efficiently.
The router, positioned adjacent to the deep layers, utilizes deep features aggregating consolidated information. This enables the router to perform efficiently, allocating new knowledge to the appropriate experts, which progressively increase in the deep layers. Extensive experiments on TRACE datasets and general language understanding datasets demonstrate that the proposed PMoE outperforms previous state-of-the-art approaches.




§ INTRODUCTION


Although Large Language Models (LLMs) have advanced significantly <cit.>,  unlike humans, they often forget previously acquired knowledge during continuous learning <cit.>. This phenomenon, known as catastrophic forgetting <cit.>, results in substantial environmental and economic inefficiencies, particularly for generative LLMs that require extensive data for training according to scaling laws <cit.>. Consequently, research in continuous learning is crucial for the efficient recycling of language models. This paper focuses on developing techniques that preserve existing knowledge while simultaneously acquiring new information.

Existing continual learning approaches can be divided into three approaches: replay-based <cit.>, regularization-based <cit.>, and architecture-based <cit.>. Replay-based and regularization-based methods use a memory buffer with examples from previous tasks or add regularization constraints to penalize changes in important weights. However, they still suffer from forgetting due to continuous adjustments in parameters. Recently proposed architecture-based approaches <cit.> enhance performance in a parameter-efficient manner but still struggle with the trade-off between learning plasticity and memory stability <cit.> or require prior information such as task-ID to classify input text into task-specific parameters.

In this study, we propose the Progressive Mixture of Experts with Asymmetric Transformer (PMoE), which efficiently adapts to new tasks while minimizing the forgetting of existing knowledge within a continual learning framework. The key idea of PMoE is its asymmetric depth design: shallow layers retain general knowledge, while deeper layers acquire new task-specific knowledge. To achieve this, PMoE progressively adds experts <cit.> and employs a routing network to classify input text.

We take inspiration from <cit.>, which shows that LLMs aggregate information in shallow layers around important words and distribute it in deep layers. We hypothesize that the performance of the router is improved when using features from deep layers rather than shallow layers. In PMoE, specialized experts are absent in shallow layers but present in deep layers, enabling the model to benefit from parameter efficiency and robust routing performance.

The main contributions of our work are summarized as follows:



  * We propose PMoE, which efficiently adapts to new tasks for continual learning by introducing an asymmetric depth design.

  * Our method outperforms LoRA with replay-based and prior state-of-the-art methods on the TRACE benchmark.

  * Experimental results demonstrate that our asymmetric design is both effective and parameter-efficient in preserving prior knowledge while adapting to new knowledge.





§ BACKGROUND



 §.§ Problem Statement


In a continual learning setup, a model sequentially encounters datasets 𝒟_1, …, 𝒟_T, where each dataset 𝒟_t = (x^t, y^t) consists of data points specific to task t. Concretely, the model is provided only with 𝒟_t during training for task t. The primary challenge in continual learning is to enable the model to retain previously acquired knowledge while effectively learning new tasks. In this study, we adopt the general setup of continual learning, which involves maximizing the cumulative performance across all tasks as follows:


    max_θ∑_k=1^T ∑_x,y∈𝒟klog p_θ (y | x)


The overall performance (OP) and backward transfer score (BWT) are measured immediately following the completion of training for the t-th task, as follows:


    OP_t = 1/t∑_i=1^t R_t,i^D 
        BWT_t = 1/t∑_i=1^t (R_t,i^D - R_i,i^D)


where R_t,i^D refers to the score of the i-th task of the model after completing training on the t-th task (i ≤ t). Furthermore, we simulate a realistic setting by having the model generate text sequences without task ID at test time.



 §.§ LoRA


Recently, many Parameter Efficient Tuning (PET) approaches have been proposed and widely adopted in fine-tuning settings. These methods aim to efficiently adapt pre-trained models by updating a smaller subset of parameters than updating all. Among these approaches, Low-Rank Adaptation (LoRA) <cit.> hypothesizes that weight updates in pre-trained models have a "low intrinsic dimension." Instead of updating the entire pre-trained weight W_0 ∈ℝ^d × k, LoRA updates decomposed bottleneck weights B ∈ℝ^d × r and A ∈ℝ^r × k, where W_0 + Δ W = W_0 + BA and the rank r ≪min(d,k). Thus, the forward pass h = W_0 x becomes


    h = W_0 x + Δ W x = W_0 x + BA x.


In this paper, we utilize LoRA as the expert component of our model.




§ ASYMMETRIC MIXTURE-OF-EXPERTS



 §.§ Overall Architecture




To address the issue of information flow within transformer blocks, we propose an asymmetric Mixture-of-Experts (MoE) architecture as illustrated in Figure <ref>. By introducing a threshold τ, the overall network f_θ is divided into shallow layers f_θ_l=1,...,τ and deep layers f_θ_l=(τ+1),...,N. The shallow transformer blocks employ a single LoRA (BA)^l aligned with general human instruction-following, whereas deeper blocks utilize a multi-head LoRA (BA)_k^l aligned with specialized abilities, assigning experts k ∈1,...,T.

At the boundary between shallow and deep layers, the router distributes hidden features from the shallow layer to the deep layer. The routing network simply consists of a linear layer W_g ∈ℝ^h × T, and thus the distribution from routing G(x) ∈ℝ^T is represented as follows:


    G(x) = Softmax (x · W_g).


Many Mixture-of-Experts approaches adopt such routing networks to connect the input text to experts. However, there has been a major issue with the routers having fair connecting performance due to the overwhelming presence of a few experts, and various approaches have been proposed to overcome this drawback by introducing auxiliary loss <cit.>, token capacity <cit.>, or expert dropout <cit.>. Since PMoE uses deep features aggregating consolidated information, our asymmetric design does not require the application of such approaches. More details are provided in Section <ref>.

The forward pass of the self-attention layer can be expressed as:


    f_θ_l+1(h_l)=
    
    W^l · h_l,     l ≤τ
    
    W'^l · h_l,


where W^l = W_0^l + (BA)^l and
 W'^l = W_0^l + ∑_k=1^T G(h_τ)_k · (BA)^l_k.

To progressively allocate all experts with consolidated information, we progressively add the LoRA along with incremental tasks. This parameterization-based method avoids catastrophic forgetting by preserving the knowledge acquired from previous tasks.



 §.§ Experimental Setup



  §.§.§ TRACE Dataset for Continual Learning Evaluation


To evaluate fine-tuning as a scheme for continual learning, we adopt the TRACE dataset <cit.>, which comprises eight distinct datasets spanning multiple domains, including multilingual (C-STANCE <cit.>, 20Minuten <cit.>), code completion (Py150 <cit.>), mathematical reasoning (NumGLUE <cit.>), and domain-specific (FOMC <cit.>, MeetingBank <cit.>, ScienceQA <cit.>) datasets. Following Wang et al., we use a random sample of 5,000 training examples for each task, resulting in a total of 40,000 training examples, with the order always fixed as follows: C-STANCE, FOMC, MeetingBank, Py150, ScienceQA, NumGLUE-cs, NumGLUE-ds, 20Minuten.



  §.§.§ General Ability Evaluation


To assess the impact of domain-specific tasks on the generality performance of pre-trained models, we evaluate five well-known benchmark datasets: MMLU <cit.>, GSM <cit.>, BBH <cit.>, BoolQ <cit.>, and PiQA <cit.>. Following the experimental setup of previous work <cit.>, we measure the general ability delta, which represents the performance difference between pre-continual learning and post-continual learning, as follows:


    Δ R^G_i = 1/M∑_i=1^M(R_t,i^G - R^G_0,i),

where M refers to the total number of benchmarks, and R^G_t,i refers to the result on benchmark i after the t-th tuning.



  §.§.§ Implementation Detail

Our code implementation builds upon the PEFT <cit.> library. We employ Llama2-7b for our pre-trained weights and LoRA for our experts. We optimize our model using AdamW with a minibatch size of 128, a learning rate of 3e-4, and a cosine annealing scheduler. The hyper-parameter rank and τ are set to 4 and 24, respectively. At inference time, the model generates text sequences without knowing the task ID to which it belongs. All results are averaged over 3 runs.

Following the settings of previous studies <cit.>, we incorporate alignment data from LIMA <cit.> into the replay memory, replaying only 1% of historical data.



  §.§.§ Baselines

We compare PMoE with six baseline approaches. For consistency in experimental comparison, the foundation of all baselines is pre-trained Llama2-7b <cit.>. In Parameter Efficient Tuning (PET) comparisons, the number of trainable parameters is approximately consistent (≤ 0.1%).



  * Llama2-7b* <cit.>: uses in-context learning with 6-shot on the current task without fine-tuning.

  * Full-ft <cit.>: fine-tunes full parameters on sequential tasks.

  * LoRA <cit.>: fine-tunes fixed-size LoRA on sequential tasks.

  * O-LoRA <cit.>: incrementally learns new tasks in a direction orthogonal to the LoRA subspace of past tasks while fixing the previous parameters.

  * Full-ft-RE: fine-tunes full parameters with a memory buffer and replays samples from old tasks.

  * LoRA-RE: fine-tunes fixed-size LoRA with a memory buffer and replays samples from old tasks, following the same memory setting as Full-ft-RE.




 §.§ Main Result




The comparative analysis is presented in Table <ref>. The upper part of the table represents the general capabilities of models that have undergone continuous learning to completion, while the lower part represents the specialized performance of models using the evaluation set from the data employed in continual learning. Furthermore, the left side of the table presents models that utilize in-context learning and those without the replay-based approach, whereas the right side presents models that incorporate the replay-based method.

Utilizing replay generally enhances performance, even with only 1% of examples. Replay-based approaches show a BWT greater than zero, indicating not only no forgetting of past knowledge but also an improvement in performance due to recently acquired knowledge and memory. The PMoE outperforms LoRA-RE in both general and tuned abilities by 2.2% and 1.8%, respectively, and even exceeds the full-fine-tuning method, which has more than 1,000 times the trainable parameters, by 3.6% and 2.1% in general and task abilities, respectively.

The current state-of-the-art model, O-LoRA, demonstrates optimal general ability at 53.0%, indicating an improvement in previously acquired knowledge. However, its performance on new tasks is relatively lower at 41.2%. Notably, in the FOMC task, which was initially tuned, the PMoE demonstrated a 34.7% improvement. Consequently, the PMoE not only enhances the performance of tuned tasks but also retains a high level of generalization ability.



§ DISCUSSION




 §.§ Asymmetric Design


The core strategy of the PMoE architecture is characterized by its asymmetric design, where shallow layers are dedicated to general abilities and deeper layers are tailored for newly acquired abilities. To assess the efficacy of this asymmetric design, we analyze the performance as functions of varying τ. A lower τ results in the model possessing a greater number of task-specific parameters, with the routing mechanism utilizing shallow features. Conversely, a higher τ leads to a model enriched with parameters for general abilities, where the router utilizes deep features. As illustrated in Figure <ref>, the model demonstrates optimal average performance at a moderate τ, and a decrease in computational cost proportional to τ. Thus, if performance levels are comparable, it is advantageous to increase τ to reduce computational demands.





 §.§ Router Analysis



To investigate the router in PMoE, we show matrices representing the allocation probability by the router. As shown in Figure <ref>, when the router is located near the shallow layer (i.e., τ=6), the router mainly chooses only a few experts. This imbalance problem is a well-known phenomenon where input x is assigned to a specific superior expert, rather than being distributed across a diverse range of experts. Previous works have attempted to address the router imbalance problem by introducing auxiliary losses <cit.>, token capacity <cit.>, or expert dropout <cit.>. However, when the router is located near the deep layer (i.e., τ=24), the imbalance problem is automatically solved, well distributing all experts. As mentioned in the motivation of our study, this result supports the hypothesis that LLMs aggregate information in shallow layers and distribute it in deep layers, leading to better router performance when using features from deep layers rather than shallow layers.



To further investigate, we conducted a qualitative evaluation of the token allocation by the router, as illustrated in Figure <ref>. The evaluation utilized test prompts that include three distinct types: Python code, mathematical questions and answers, and lines from Shakespeare's Hamlet. With a τ of 6, most tokens are predominantly allocated to experts 2 or 8. In contrast, at τ of 24, the allocation patterns are more varied, indicating a comprehensive use of experts. Concretely, the router seems to allocate tokens to experts based on identifiable patterns related to the role of the tokens. For example, special characters are predominantly assigned to expert 3, elements of Python syntax are mainly assigned to expert 4, and plain text is typically assigned to expert 1.

Drawing from the aforementioned analysis, if τ is 24 and the task-ID is treated as the ground truth for the router during the training stage, PMoE is capable of employing each expert as a task-specific expert without the need for task-ID during the inference stage. We incorporate cross-entropy loss to guide the routing of input x to a suitable expert, as follows:


    ℒ(x, k) = - log G_k(x)


where G_k(x) denotes the router probability of assigning input x to its corresponding task-ID k. This loss provides a simple yet effective mechanism to prevent inputs from being pushed to a single expert by guiding the task assignment corresponding to input x.



The comparison with the application of Equation <ref> is shown in Figure <ref>. This auxiliary loss function experimentally demonstrates its effectiveness in making experts task-specific without task-ID in the inference stage. However, in Table <ref>, applying the auxiliary loss slightly degrades overall performance. This result suggests that a combination of experts rather than task-specific experts has a positive effect on overall performance. Future research should explore extending task-specific experts for asymmetric design.







 §.§ Comparison with Trainable Parameter Size



We assessed the detailed comparison between PMoE and LoRA with the replay method at different scales of adaptation parameters, as shown in Table <ref>. In our evaluation, we control the number of parameters by the rank of the adapter from under 0.1% to over 0.3%. In all parameter size settings, the general ability of PMoE is higher than that of LoRA by 0.7%-2.2%. In specialization performance, PMoE is higher than LoRA by 0.6%-0.8%. Interestingly, there is only a slight difference in performance along with the parameter size.




§ RELATED WORKS



 §.§ Parameter Efficient Fine Tuning


As larger pre-trained models become more prevalent, efficiently fine-tuning their parameters has gained significant importance. Parameter Efficient Fine Tuning (PEFT) approaches are a set of methods that update only a small portion of parameters while freezing the pre-trained parameters during the fine-tuning stage. Various approaches have been proposed, including inserting additional modules or parameters, such as Adapter <cit.>, Prompt Tuning <cit.>, and Low-Rank Adaptation (LoRA) <cit.>.

In this paper, we focus primarily on LoRA, which hypothesizes that weight updates in pre-trained models have a "low intrinsic dimension." Building upon LoRA, AdaLoRA parameterizes incremental updates in the form of singular value decomposition <cit.>, O-LoRA involves layering low-rank adapters on the key and value projection matrices <cit.>, SIRA leverages the Sparse Mixture of Experts to boost the performance of LoRA <cit.>, and PMoE employs an asymmetric design with shallow layers dedicated to general knowledge and deeper layers for new knowledge.



 §.§ Continual Learning


Existing continual learning approaches can be categorized into three main types: replay-based, regularization-based, and architecture-based.

Replay-based approaches<cit.> store examples from previous tasks for rehearsal. Although these approaches achieve strong performance, storing large samples can be inefficient and may not be feasible for data privacy settings.

Regularization-based approaches <cit.> update only a few weights trained on previous tasks. These approaches are memory efficient but can suffer from catastrophic forgetting, especially in long sequence tasks.

Architecture-based approaches isolate weights trained on previous tasks and progressively add weights for new tasks. Although the recently proposed progressive prompt <cit.> has achieved state-of-the-art results, it has the limitation of requiring the task-ID during the inference stage. Among the continual learning techniques that do not require task-ID, O-LoRA <cit.> effectively addresses catastrophic forgetting. However, its performance, as measured by the TRACE benchmark <cit.>, was lower. In this paper, PMoE achieves superior performance on the TRACE benchmark without requiring task-ID during the inference stage.



§ LIMITATIONS


In our study, we conducted experiments across eight distinct tasks. Expanding the number of tasks could enhance the robustness and generalizability of our findings. Our research focuses solely on generative models, particularly Llama2-7b. However, the foundational concepts of our model hold potential for broader applicability in task-agnostic settings. Further research will assess the efficacy of our proposed approach across diverse benchmarks, aiming to broaden its application to multimodal continual learning.



§ CONCLUSION

In this paper, we enhanced the performance of LLM in continual learning by using PMoE. The PMoE features an asymmetric architecture between shallow and deep layers. Specifically, the shallow part employs a single expert to enhance general ability, whereas the deep part progressively utilizes an increasing number of experts to focus on tuned performance. Empirical experiments on comprehensive continual learning benchmarks demonstrate that the proposed PMoE outperforms previous state-of-the-art approaches. Furthermore, to investigate the asymmetric structure, we experimented by varying the asymmetric hyperparameter τ, measuring the performance and efficiency of the model, and analyzing the router.







 Paper Title: A Simple Background Augmentation Method for Object Detection with Diffusion Model.





 



Background Augmentation with Diffusion Model






Y. Li, X. Dong, C. Chen, W. Zhuang, L. Lyu.




Sony AI

{yuhang.li, xin.dong, chen.chen, weiming.zhuang, lingjuan.lyu}@sony.com



A Simple Background Augmentation Method for Object Detection with Diffusion Model
    Yuhang Li10000-0002-6444-7253Xin Dong1 Chen Chen1 Weiming Zhuang1 Lingjuan Lyu1Corresponding Author. Work done during Yuhang Li’s internship at Sony AI.
    August 5, 2024
============================================================================================================================================================




§ ABSTRACT 
In computer vision, it is well-known that a lack of data diversity will impair model performance.
In this study, we address the challenges of enhancing the dataset diversity problem in order to benefit various downstream tasks such as object detection and instance segmentation. We propose a simple yet effective data augmentation approach by leveraging advancements in generative models, specifically text-to-image synthesis technologies like Stable Diffusion. Our method focuses on generating variations of labeled real images, utilizing generative object and background augmentation via inpainting to augment existing training data without the need for additional annotations. We find that background augmentation, in particular, significantly improves the models' robustness and generalization capabilities. 
We also investigate how to adjust the prompt and mask to ensure the generated content 
comply with the existing annotations. 
The efficacy of our augmentation techniques is validated through comprehensive evaluations of the COCO dataset and several other key object detection benchmarks, demonstrating notable enhancements in model performance across diverse scenarios. This approach offers a promising solution to the challenges of dataset enhancement, contributing to the development of more accurate and robust computer vision models.
  





§ INTRODUCTION


The quest for robust and accurate object detection <cit.> and instance segmentation <cit.> models is a cornerstone for contemporary computer vision research. These models underpin a variety of applications, from autonomous vehicles <cit.> to medical image analysis <cit.>, necessitating high levels of precision and reliability. Central to the development of these models is the availability of large, diverse, and accurately annotated datasets. However, creating such datasets is fraught with challenges, including the high costs of data collection and annotation, privacy concerns, and the potential for dataset bias. For example, collecting images of urban driving scenes requires physical car infrastructure, and labeling a segmentation annotation for a single urban image in Cityscapescan takes up to 60 minutes  <cit.>. 
Therefore, it is necessary to navigate new solutions to address these limitations, data augmentations and low-cost annotations. 
 
Recent advancements in generative models, particularly in the domain of text-to-image synthesis <cit.> (e.g., Stable Diffusion <cit.>), offer a promising avenue for addressing aforementioned challenges. 
While Stable Diffusion and similar models can create visually compelling images, generating synthetic images that are accompanied by accurate and detailed object annotations remains a formidable challenge. This complexity stems from the need for precise alignment between the generated visual content and its corresponding annotations. Although several works have 
generated the synthetic data and instances annotations simultaneously <cit.>, they all require finetuning the existing text-to-image on the current dataset and generate a significant amount of synthetic data for pre-training, which increases training complexity significantly. 


To this end, we explore generating variants of labeled real images for object detection and instance segmentation while reusing their annotations, in a way that augments the existing training data. Specifically, we explore generative object augmentation and background augmentation through inpainting, which allows for the enhancement of objects and backgrounds respectively within the dataset. 
Notably, we discover that background augmentation boosts the model performance, which suggests that modifying the 

background context of images plays a crucial role in improving model robustness and generalization.


We rigorously evaluate our augmentation methods on the MS COCO dataset <cit.> and PASCAL VOC <cit.> for both detection and segmentation tasks across multiple architectures. This comprehensive evaluation demonstrates the effectiveness of our proposed augmentation techniques in improving the performance of object detection and instance segmentation models under various challenging scenarios. We summarize our contributions as follows:

    
  * We introduce a simple data augmentation framework for object detection with the text-to-image model without finetuning. We find that augmenting the background brings more advantages than augmenting objects. 
    
  * We carefully craft the text prompt, mask region, and diffusion steps to ensure the generated content is effective for data augmentation purposes. The ablation studies show that they are essential to achieve superior performance.
    
  * We evaluate the background augmentation policies on various tasks including object detection and instance segmentation, significantly improving their performance. For example, our background augmentation increases mAP up to 5.3% when using only 10% of the COCO training data. 
    






§ RELATED WORK


This section provides literature reviews of object detection, data augmentation, and text-to-image generation for visual tasks.

Object Detection. 
Object detection aims to simultaneously predict the category and corresponding bounding box
for the objects in the images. Generally, object detectors <cit.> are trained on a substantial amount of
training data with bounding box annotations and can only
recognize a predetermined set of categories present in the
training data, MS COCO dataset <cit.>. 
Both backbone architecture and the detection neck/head framework are important for object detection performance. For backbone, Vision Transformer <cit.> has emerged as an effective backbone alternative compared to convolutional neural networks <cit.>. These transformer-based models leverage self-attention mechanisms to capture global dependencies within the image. 
Moreover, the development of efficient and scalable detectors, such as EfficientDet <cit.>, further underscores the importance of optimizing both accuracy and computational efficiency. 


Data Augmentation. 
Data augmentation <cit.> plays an indispensable role in the deep learning model, as it forces the model to learn invariant features, and thus helps generalization.
The data augmentation is applied in many areas of vision tasks including object recognition <cit.>, video perception <cit.>, and semantic segmentation <cit.>. 
Apart from learning invariant features, data augmentation also has other specific applications in deep learning. For example, adversarial training <cit.> leverages data augmentation to create adversarial samples and thereby improves the adversarial robustness of the model.
In terms of data augmentation in object detection, Zoph et.al <cit.> propose reinforcement learning to search for complex augmentation policies including color transformations, geometric transformations, and bounding box transformations. CutMix <cit.> proposes a spatial mixing strategy for augmenting the detection results. RandAugment <cit.> uses a reduced search space for searching both classification and detection augmentation policies. 



Text-to-Image Data for Visual Tasks. 
Traditional synthetic data are acquired through the renderings from the graphics engines <cit.>. However, this type of synthesis cannot guarantee the quality and diversity of the generated data, resulting in a large gap with real-world data. Recent success in generative models has made synthesizing photo-realistic and high-fidelity images possible, which could be used for training the neural networks for image recognition due to their unlimited generation. 
For example, early works explored Generative Adversarial Networks (GAN) <cit.> for image recognition tasks. <cit.> uses a class-conditional GAN to train the classifier head and <cit.> uses StyleGAN to generate the labels for object segmentation. <cit.> adopts the GAN as a generator to synthesize multiple views to conduct contrastive learning. 
Until recently, the text-to-images models <cit.> have been leveraged to synthesize high-quality data for neural network training due to their effectiveness and efficiency. 
<cit.> adopts GLIDE <cit.> to synthesize images for classifier tuning on the CLIP model <cit.>. However, simply tuning the classifier may not fully explore the potential of synthetic data. StableRep <cit.> proposes to use Stable Diffusion for generating pre-trained datasets for contrastive learning and leverages the synthetic data from different random seeds as the positive pairs. 
<cit.> explores synthesizing data under ImageNet label space and 
produces improved performance. Fill-up <cit.> balances the long-tail distribution of the training data using the synthetic data from text-to-image models.
<cit.> propose to leverage text-to-image models for segmentation tasks where they adjust the model to generate both masks and images. However, this type of method requires either finetuning the model on the target dataset or synthesizing a large amount of pre-trained data. 



§ GENERATIVE BACKGROUND AUGMENTATION


In this section, we introduce our generative background augmentation method using the text-to-image diffusion model to increase the diversity of the original dataset. We first discuss the principle of the Inpainting technique and then adapt this method to our background augmentation framework. 



 §.§ Generative Augmentation with Inpainting


Our primary data synthesis model is Stable Diffusion <cit.>. 
Formally, 

the Stable Diffusion generates images from a random Gaussian noise 𝐳∼𝒩(0, 1) by applying the denoising model ϵ(·) repeatedly. We denote the text language prompt embedding as 𝐬. In this work, we apply the widely-used classifier-free guidance <cit.> which linearly combines the conditional estimate and the unconditional estimate of noise, given by

    ϵ̃(𝐬, 𝐳_t) = wϵ(𝐬, 𝐳_t) + (1-w) ϵ(𝐳_t),

where w is the guidance scale and t is the step of the denoising process. After multiple denoising steps, Stable Diffusion employs a decoder 𝒟 to decode the latent to the image. 

Given an input image 𝐱, the inpainting task is to re-generate selective areas in the image indicated by a mask variable 𝐦 while keeping other pixels unchanged. Since Stable Diffusion conducts the generation process in the latent space, we consider inpainting in the latent space as well. First, the encoder projects input image to latent space 𝐱̂ = ℰ(𝐱). Similarly, the mask variable is also projected to the latent space by resizing it 𝐦̂=resize(𝐦) to have the same size of 𝐱̂. 

Now, the inpainting denoising step can be formulated as 

    ϵ̂(𝐬, 𝐱̂_t) = 𝐦̂⊙ϵ̃(𝐬, 𝐱̂_t) + (1-𝐦̂)⊙(𝐱̂_t - 𝐱̂_t-1).

Here, ⊙ denotes the element-wise multiplication operation. 


The inpainting process described above modifies only the masked region, denoted as 𝐦̂, within the original image's latent space. Owing to the localized nature of Stable Diffusion's decoding mechanism, this method can also provide a reasonable assurance that solely the region specified by 𝐦 will be regenerated within the image space.



 §.§ Object Augmentation or Background Augmentation?



Data augmentation serves the role of increasing data amount and diversity to optimize for generalization performance. Traditionally, data augmentation in object detection is conducted through image transformations. For example, Zoph et.al <cit.> define three kinds of data augmentation: (1) color operations, (2) geometric transformations, and (3) bounding box operations, which change the image illumination, position, and annotation respectively. However, these augmentation methods only change pixel values and keep the content of the object or background the same as the original images.


In this work, we explore another kind of augmentation method, using generative models like diffusion models to change the content of training images to increase the training data volume and diversity. 
Specifically, consider a training image 𝐱∈ℝ^3× H× W containing n different objects, where each object corresponds to a unique mask variable 𝐦_i∈{0, 1}^H × W. The background mask can be determined by the following formula, 

    𝐦_b = 1 - ∑_i=1^n 𝐦_i.

With different choices from {𝐦_b, 𝐦_i}, we are able to change the content of different regions of the original image. 

Specifically for dense vision tasks like object detection and instance segmentation, a natural question is whether we should augment objects, background, or both of them.
Surprisingly, we found object augmentation is not suitable in practice.

The reasons are multiple. 

First, generating a realistic object that can precisely fill the mask region is quite challenging for existing Stable Diffusion models. For example, if a generated object cannot adequately cover its corresponding mask region, reusing the mask as its annotation will mislead the training of the model. We visualize some exemplary failure cases of object augmentation in Fig. <ref>(a). Although some efforts have been investigated to improve the alignment between generated objects and their corresponding masks <cit.>, they either introduce prohibitive computational overhead for large datasets or achieve an unsatisfactory mask-object alignment level, 
which is harmful for enhancing task performance for object detection and instance segmentation.

In addition, we have observed that object augmentation can yield low-quality inpainting results for certain object classes, especially those involving human figures <cit.>. This can adversely affect the quality of the data and may even degrade the performance of vision-related tasks (as shown in <ref>(b)). Furthermore, augmenting small objects results in negligible improvements because they contain relatively few pixels (<ref>(c)). Consequently, the detection and segmentation of small objects do not benefit from object augmentation.


Finally, data generation costs of object augmentation is much higher than background augmentation.
While for background augmentation, only one inpainting is enough for the background area. For example, the averaged number of objects in the MS COCO training dataset <cit.> is 7.27, amounting to ∼7.3× generation cost when comparing object augmentation to background augmentation (<ref>(d)).










In the experiments section, we present a thorough ablation study demonstrating that object augmentation underperforms compared to background augmentation and can even lead to a degradation in downstream task performance when compared to a baseline without generative augmentation.




 §.§ Utility-Aware Background Augmentation


In this section, we concentrate on background augmentation techniques with high training utility. An optimal background augmentation approach must avoid introducing extraneous objects, as these can confound the training process due to the lack of corresponding annotations. Additionally, it is crucial for a background augmentation method to preserve the integrity of existing objects, thereby maintaining a high level of object-mask alignment. We have investigated several design strategies to fulfill these requirements.


Prompt Selection of Background Augmentation. 
A critical choice for data generation is the text prompt 𝐬 for guiding the inpainting process. For object augmentation, one can safely adopt the class label as the text prompt while for background augmentation a text description for the whole background is needed. 

Some datasets like MS COCO provide image caption annotations for each image, which could be a potential text prompt for background augmentation. 
However, applying an image caption as a text prompt directly brings two disadvantages in practice. 
First, the image caption usually contains object descriptions as well. Using this caption will generate additional objects in the background. Second, not all detection and segmentation datasets contain image captions. It would require other captioning models to generate the new caption for the image, which involves additional computation costs and data cleaning to get accurate captions. 

To this end, we use a simple text prompt “” for background augmentation. We found this choice is sufficient for Stable Diffusion to modify the background significantly without generating additional objects. 





Background Mask Erosion. 
During the process of background augmentation, we occasionally observed that objects would inadvertently extend into the background region. This phenomenon is illustrated in Fig. <ref>(b), where the giraffe appears larger than its corresponding mask, thereby disrupting the alignment with its initial annotation. To address this issue, we introduce the concept of background mask erosion. This technique employs a minimum filter to the background mask, specifically, 𝐦_b=MinimumFilter(𝐦_b, k). We empirically determined that a kernel size k of 7 is effective in preventing object extension. Fig. <ref>(a) demonstrates the process of eroding the background mask with the minimum filter, and Fig. <ref>(b) shows a much better result after adopting our background mask erosion.

Adaptive Augmentation Freedom. 

Considering the variation in background region sizes across different images, there is an increased likelihood of errors during background augmentation—such as the introduction of undesired objects or the unwarranted extension of objects into the background areas—for images with larger backgrounds, potentially leading to the generation of harmful synthetic data. In the Stable Diffusion inpainting process, the number of diffusion steps can be adjusted to calibrate the extent of modifications applied to the original input image, that is, the distance between the original and the generated image. Drawing inspiration from this aspect of the process, we propose tailoring the degree of background augmentation to the size of the background region.

To this end, we introduce an adaptive control method for the inpainting process. Specifically, we compute the ratio of the background area to the overall image area and utilize this metric to determine the appropriate number of diffusion steps for the original training image 𝐱. The relationship is formalized as follows:


    T̂ = T ×(1 - D ×sum(𝐦_b)/W × H).


In this equation, sum(·) denotes the summation of all elements within the input matrix, and W × H denotes the total area of the training image. 
D∈(0, 1) controls the range of diffusion steps variation.
T is the maximum diffusion step where T̂=T indicates background noise starts from random Guassian noise.  Technically, if the background-to-image area ratio is higher, we reduce the diffusion steps to preserve more information from the original data. 






Fig. <ref> illustrates the process of our overall background augmentation pipeline. The image is first diffused based on the area ratio calculated from the background mask and then denoised using the eroded mask to generate the new background.




 §.§ Training with Background Augmented Data

Unlike traditional data augmentation which modifies the image tensor on the fly <cit.>, our method generates the additional images and expands the dataset size. Hence, our data augmentation can be combined with conventional data augmentation methods to improve the generalization ability in object detection. 
We also utilize our method with semi-supervised learning. Specifically, we incorporate our augmented data in Soft Teacher <cit.> framework as part of the “strong augmentation” to optimize the student network.
The “weak augmentation” is applied to the unlabeled images on teacher networks to generate soft annotations for student learning. 




§ EXPERIMENTAL EVALUATION


In this section, we investigate the impact of the augmented data by using our simple background augmentation framework. We first visualize several examples of data augmentation and then report object detection/instance segmentation performance on MS COCO <cit.> and PASCAL VOC <cit.> datasets. 
We also conduct several ablation studies and investigations into different data regimes. 





 §.§ Visualization


We demonstrate a variety of synthesis results (as well as the original image and the background mask image) using our method on the MS COCO dataset <cit.>, as shown in Fig. <ref>.
Our approach can modify the background of training data to various extents. In certain instances, the new background remains within a similar scene as the original (e.g., images at the second row and first and second columns), yet exhibits different styles and content. Conversely, some augmented backgrounds may represent entirely new scenes. For example, in the fourth row and first column, the original background of a stop sign is transformed into a purple sky, while maintaining the original stop sign objects unchanged. In the fifth/sixth row second column, our method converts a background into a photo hanging on the wall.

These background augmentations enhance the diversity of the training dataset without introducing additional annotation burdens or compromising the quality of existing annotations.




 §.§ Object Detection Evaluation


We start with the evaluation of our method on the MS COCO 
dataset. We train a Faster-RCNN <cit.> with FPN <cit.>, using the ResNet-50 <cit.> backbone. We also provide results of other datasets and model architectures in <ref>. 

Implementation Details. 
Specifically, we sampled 1%, 5%, 10%, and 25% of the original training dataset. By default, background augmentation was applied to the selected training images, generating one augmented copy per image. Both the original and the augmented images were utilized during training. For the noise sampling schedule, we employed Stable Diffusion v2-1 <cit.> and DPMSolver <cit.>. We resized the image resolution to 512×512 for the inpainting process and subsequently restored the images to their original resolution post-inpainting.

For the training of the detection model, we adhered to the implementation and hyperparameters provided by MMDetection <cit.> and Soft Teacher <cit.>. We utilized anchors spanning 5 scales and 3 aspect ratios. During the training and inference phases, 2k and 1k region proposals were generated respectively with a non-maximum suppression threshold set at 0.7. During each training iteration, a subset of 512 proposals were sampled from the 2k available to serve as box candidates for training the RCNN. The model underwent training over 12 epochs on 8 GPUs, each processing 5 images. We opted for SGD as the training algorithm, initiating the learning rate at 0.01 and reducing it by a factor of 10 after 110k and 160k iterations. The weight decay was set at 0.0001, while the momentum was maintained at 0.9.

Comparison to Baselines. 
We compare our method against a baseline Faster-RCNN model that does not utilize generative background augmentation. Both the baseline model and our approach incorporate conventional data augmentation techniques, such as horizontal flipping with a 50% probability and multi-scale jittering, where images are randomly resized within the range of 800 to 1333 pixels. Furthermore, we contrasted our generative background augmentation with the state-of-the-art augmentation method, RandAugment <cit.>. RandAugment conducts an exhaustive search for complex augmentation policies across 14 different augmentation methods using grid search. Despite the simplicity and reduced search effort of our method compared to RandAugment, our approach consistently surpasses RandAugment in performance across various data regimes as shown in <ref>.





Table <ref> shows the results using our augmentation policy on the above procedures. The proposed background augmentation policy achieves systematic gains across different amounts of training data used, ranging from +2.7 mAP to +3.5 mAP. 
In comparison, a previous regularization technique applied to ResNet-50 Faster-RCNN achieves a gain ranging from +0.6mAP to +1.5mAP.


Combined with Semi-Supervised Learning. 
We then experiment with semi-supervised learning in object detection. In particular, we select Soft Teacher <cit.> and add our background augmentation data to the  dataset, which is used to train the teacher network to annotate the  dataset for knowledge transfer, which is the remaining data from MS COCO training set. The model (ResNet-50-based Faster-RCNN) is kept the same as in previous experiments. The other training hyper-parameters are based on the MMDetection implementation <cit.>. 

We compare Soft Teacher with or without our background augmentation as the augmentation strategy. The results are again shown in <ref>, from which we can find that our method also effectively improves the mAP. 




 §.§ Comparisons on Transformer Architecture


We also evaluate our background augmentation on Transformer-based architecture. Specifically, we use Swin Transformer small <cit.> as backbone, Mask-RCNN <cit.> as detection framework to train on the full MS COCO dataset. The training hyper-parameters are kept the same as the original configurations <cit.>, AdamW <cit.> optimizer (initial learning rate of 0.0001, weight decay of 0.05,
and batch size of 16), and 3x schedule (36 epochs with the
learning rate decayed by 10× at epochs 27 and 33). 
The standard augmentation includes horizontal flipping and multi-scale training (resizing the input such that the shorter side is between 480 and 800 while the longer side is at most 1333). 
We report the bounding box AP and instance mask AP metrics following the conventions of transformer evaluation.



<ref> summarizes the results on the Swin transformer. We can find that background augmentation effectively improves the detection and segmentation performance (+1.0 AP^box and +0.8 AP^mask on the Swin-T architecture). 









 §.§ Ablation Studies


In this section, we conduct ablation studies that verify the design choices of our background augmentation pipeline. In order to do this, we generate different sets of COCO training data and combine them with original data to train a ResNet-50 RetinaNet <cit.>. We use a similar training schedule and hyper-parameters with previous experiments. 
In particular, we compare several cases: (1) using object augmentation or background augmentation, (2) using background augmentation but with or without our proposed techniques. 
Specifically, for prompt input we either use the COCO caption as prompt or our proposed simple prompt method, and for background mask erosion as well as the adaptive diffusion steps, we choose to either apply them or not. 

Table <ref> summarizes the results of our ablation experiments, from which we can observe that applying the object augmentation will even decrease the mAP performance (-0.3 mAP). The same situation also applies to background augmentation if the prompt is not chosen correctly. This result indicates that augmenting the data in object detection is non-trivial. With background mask erosion and appropriate prompt choice, we can achieve better detection performance than the baseline (+1.3 mAP on the full COCO dataset). Furthermore, adding the adaptive strength control enhances the final performance to 38.3 mAP. 



 §.§ Augmenting More Data

In previous experiments, we only generate 1x of COCO data. This is still far from the ability of text-to-image models as it can generate a different image by randomly sampling a different 𝐳. In this section, we explore the limits of our background augmentation. Specifically, we experiment with two cases. In the first case, we repeatedly generate α times data for each image and study the effect of increasing α (called uniform sampling).
In the second case, we increase the ratio of augmenting small objects in the training data since detecting small objects often poses more challenges than detecting large objects <cit.>. 
Therefore, we increase the proportion of small object detection based on the background area ratio, which is called non-uniform sampling. 
In total, we also generate α× copy of COCO data but with more augmented data for small objects in this case. 




We evaluate these two scaling cases in the 10% COCO data regime. The α is set to 1, 2, 5, and 10 (Note that α=0 indicates no background augmentation applied). We use ResNet-50-based Faster RCNN <cit.> to evaluate different data regimes. 
The training hyper-parameters are kept the same with <ref>. We report four different metrics: mAP, mAP^small, mAP^medium, and mAP^big, as shown in <ref>.

From the figure, we find that uniformly increasing the number of augmented data does not consistently increase the mAP performance. 
Observing the results from different sizes of the objects, we find that the major cause is the small object performance, which is decreasing when α≥ 2. Instead, non-uniform sampling prioritizes the small objects in augmentation and ensures more diverse images are created in small object detection, leading to better-performed detectors. 






 §.§ Evaluation on PASCAL VOC

To evaluate the effectiveness of our proposed background augmentation on an entirely different dataset and another detection algorithm, we train a Faster R-CNN <cit.> model and a RetinaNet model <cit.> with a ResNet-50 backbone on PASCAL VOC dataset <cit.>.
We combine the training sets of PASCAL VOC 2007 and PASCAL VOC 2012, and test our model on the PASCAL VOC
2007 test set (4952 images). Our evaluation metric is the
bounding box mean average precision (mAP^box).
The training hyper-parameters are kept the same with previous experiments and the backbone model is initialized from the ImageNet pre-trained checkpoint <cit.>. 
We also include the performance of RandAugment in the PASCAL VOC evaluation. 

The results are summarized in <ref>, where we can find our method leads to a significant increase in terms of box mAP. 






§ CONCLUSION

In this work, we propose a simple yet effective background augmentation framework for object detection and instance segmentation. We directly utilize the existing mask annotation to perform Inpainting to create new content in the training data. Through a set of rigorous explorations, we confirm that background augmentation, instead of object augmentation, can achieve higher potential. We also propose several techniques to ensure the background contains no additional object so that the generated data can be directly used for augmentation. Results show that our method can be effectively used to improve the detection performance.
Our method has certain limitations: for images containing extremely large objects, background augmentation can hardly alter the content (as shown in <ref> the improvement of mAP^Large is relatively small). 
In this case, our method could be combined with an effective object augmentation (that may require finetuning of the text-to-image model) to enrich the diversity of this type of data.



§ ACKNOWLEDGEMENT

This work is sponsored by Sony AI.







splncs04



 Paper Title: HGOE: Hybrid External and Internal Graph Outlier Exposure for Graph Out-of-Distribution Detection.


















  IIP, ICT, CAS
      

 SCST, UCAS
    Beijing
  China
hejunwei22s@ict.ac.cn



[1]

  IIP, ICT, CAS
  
  Beijing
  China
xuqianqian@ict.ac.cn



  SCST, UCAS
  
  Beijing
  China
jiangyangbangyan@ucas.ac.cn



  IIP, ICT, CAS
  
  Beijing
  China
wangzitai@ict.ac.cn



  IIP, ICT, CAS
      

 SCST, UCAS
    Beijing
  China
sunyuchen22s@ict.ac.cn


[1]

  SCST, UCAS
    

  IIP, ICT, CAS
    

  BDKM, CAS
  
  Beijing
  China
qmhuang@ucas.ac.cn












§ ABSTRACT 
With the progressive advancements in deep graph learning, out-of-distribution (OOD) detection for graph data has emerged as a critical challenge. While the efficacy of auxiliary datasets in enhancing OOD detection has been extensively studied for image and text data, such approaches have not yet been explored for graph data. Unlike Euclidean data, graph data exhibits greater diversity but lower robustness to perturbations, complicating the integration of outliers. To tackle these challenges, we propose the introduction of Hybrid External and Internal Graph Outlier Exposure (HGOE) to improve graph OOD detection performance. Our framework involves using realistic external graph data from various domains and synthesizing internal outliers within ID subgroups to address the poor robustness and presence of OOD samples within the ID class. Furthermore, we develop a boundary-aware OE loss that adaptively assigns weights to outliers, maximizing the use of high-quality OOD samples while minimizing the impact of low-quality ones. Our proposed HGOE framework is model-agnostic and designed to enhance the effectiveness of existing graph OOD detection models. Experimental results demonstrate that our HGOE framework can significantly improve the performance of existing OOD detection models across all 8 real datasets.







<ccs2012>
   <concept>
       <concept_id>10010147.10010178.10010187</concept_id>
       <concept_desc>Computing methodologies Knowledge representation and reasoning</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010257.10010321</concept_id>
       <concept_desc>Computing methodologies Machine learning algorithms</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
   <concept>
       <concept_id>10002950.10003624.10003633</concept_id>
       <concept_desc>Mathematics of computing Graph theory</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Knowledge representation and reasoning
[300]Computing methodologies Machine learning algorithms
[500]Mathematics of computing Graph theory
























HGOE: Hybrid External and Internal Graph Outlier Exposure for Graph Out-of-Distribution Detection
    Qingming Huang
    August 5, 2024
=================================================================================================




[1]Corresponding authors.



§ INTRODUCTION

Nowadays, graph-structured data have shown significant success in handling non-Euclidean relationships, prevalent in multimedia systems such as social networks <cit.>, knowledge graphs <cit.>, citation networks <cit.>. These capabilities facilitate advanced applications like scene graph generation <cit.>, visual analysis <cit.>, anomaly detection<cit.> and recommender system <cit.>, 
by modeling complex relationships between heterogeneous data types<cit.>. 
However, a significant challenge arises from the i.i.d. assumption on which the mainstream graph learning methods depend, i.e., the training and testing graph data are from the same distribution. This assumption often fails in real-world scenarios, particularly in domains where data is complex and lacks sufficient labeling, such as in the case of drug molecules and proteins. For example, a new drug might not be any part of the already annotated data. Consequently, this gives rise to an interesting issue: how to determine whether such a new drug is present in the annotated drug library? This problem, known as Graph Out-Of-Distribution (GOOD) detection, is crucial in advancing the use of graph learning in real-world scenarios. 

Although several methods for graph OOD detection have been developed <cit.>, they only utilize ID graph data. When the full data distribution is complicated, merely modeling the ID data might be insufficient to capture the essential clues for OOD detection. In fact, auxiliary public OOD data is often accessible to help the detector discover such clues. Exposing such auxiliary OOD data to the model, also known as Outlier Exposure (OE), is widely studied for image data <cit.>. However, the application of OE for graph samples has not yet been explored. To bridge this gap, we attempt to investigate the incorporation of OE into graph OOD detection.



Unlike image data, introducing outliers to assist in the OOD detection of graph data presents two significant challenges. 

Firstly, since graphs are abstractions of patterns from the natural world, they exhibit inherent diversity. For instance, various non-Euclidean structures ranging from water molecules to complex social networks can be represented using graphs, yet these graphs can significantly differ in structure and properties. In contrast, images possess a Euclidean structure, which facilitates easier feature transfer. Therefore, utilizing existing graphs as outliers directly is often not effective.

Secondly, graph data exhibits less robustness to perturbations, resulting in the possibility of OOD samples existing within the boundary of ID classes. For example, as illustrated in Figure <ref>, phenol and benzene differ only by a hydrogen atom and an oxygen atom, yet exhibit vastly different properties; phenol is solid at room temperature, while benzene is liquid. This suggests that slight perturbations could transform ID graph sample into an OOD sample. In contrast, image data typically exhibit a more compact intra-class distribution and more well-separated classes (e.g., dogs and cats), where minor perturbations often do not change their categories. Therefore, merely incorporating some external outliers for exposure could be effective for image data. But considering that outliers may exist even within the same class of graphs, the external outlier exposure remedy is insufficient for graph OOD detection.

To address the first challenge, existing studies suggest that introducing diverse outliers closer to the in-distribution data is beneficial. Therefore, given the inherent diversity of graph data, we propose incorporating cross-domain external outliers into training. Moreover, in addition to these external outliers, we aim to synthesize outliers that are nearer to in-distribution data. Regarding the second challenge, since there exist subgroups within a class, and outliers also exist between these subgroups, we consider synthesizing some internal outliers between subgroups to assist in OOD detection. Because these samples are close to In-Distribution samples, if the model can accurately identify these samples, it would further enhance the OOD detection performance. 

Motivated by this, we propose a general hybrid graph outlier exposure (HGOE) framework for graph OOD detection which integrates both external and internal outliers. The external outliers are easily collected from public database. As for internal graph outliers, we design a graphon-based ID-mixup strategy to simulate the OOD region among subgroups and synthesize OOD samples. Given these outliers, we further propose a boundary-aware OE loss to adaptively learn from true outliers and prevent the unintended bias.

In summary, the contributions of this paper are as follows:

    
    
  * We propose a novel hybrid graph outlier exposure framework for graph OOD detection. It simultaneously utilizes external outliers and internal outliers to enhance the diversity. To the best of our knowledge, this is the first trial of outlier exposure in graph-level OOD detection tasks.
    
  * To synthesize internal outliers, we design an ID-mixup method that can effectively generate outlier graph samples between ID subgroups based on graphons. 
    
  * We further introduce a novel boundary-aware loss. By instantiating the HGOE framework with a SOTA detector, we have surpassed the competitors on 8 real-world graph datasets.




§ RELATED WORK

Graph Neural Networks. Graph neural networks (GNNs) have been widely adopted in various deep learning tasks due to their ability to process graph-structured data, which can effectively extract both the structural and attribute information of graphs <cit.>. GNNs have achieved remarkable results in various deep learning tasks in recent years, such as recommendation systems, natural language processing and computer vision <cit.>. GNNs can be broadly categorized into two distinct classes, which encompass spectral-based GNNs and spatial-based GNNs <cit.>. 

Existing Spectral-based Graph Neural Networks leverage spectral graph theory for graph analysis, offering the advantage of incorporating global graph topology information. However, they also exhibit certain limitations, including high computational complexity, challenges in handling dynamic or heterogeneous graphs, and a deficiency in local perception ability<cit.>. For instance, ChebNet approximates spectral graph convolutions using Chebyshev polynomials of arbitrary order, while GCN simplifies this by employing only the first two Chebyshev polynomials as the graph convolution, effectively creating a fixed low-pass filter<cit.>. 

Spatial-based Graph Neural Networks leverage the spatial information of nodes within the graph to perform graph convolutions<cit.>. These networks are rooted in message-passing mechanisms<cit.>, wherein each node updates its features based on the features of its neighboring nodes. Notably, Graph Attention Networks (GAT)<cit.> introduce a self-attention mechanism to compute node neighbor weights, allowing for dynamic and adaptive aggregation within neighborhoods. GAT further enhances model capacity by employing multi-head attention. Meanwhile, GraphSAGE<cit.> learns how to aggregate node features from various sources using diverse functions like mean, max-pooling, or LSTM, making it suitable for inductive learning tasks that involve new nodes or graphs during testing. Graph Isomorphism Networks (GIN)<cit.>, another class of graph convolutional network, employ sum and MLP operations to aggregate node features. 
Additionally, Graph Structural Neural Networks<cit.> provide a versatile solution for incorporating structural graph properties into the message-passing aggregation scheme of GNNs.

Out-of-distribution (OOD) Detection. 
Existing deep learning-based classification methods often exhibit overconfidence on unseen classes <cit.>. To address this issue, out-of-distribution (OOD) detection <cit.> involves the task of distinguishing test samples from distributions different from the seen training data. It comprises post-hoc and fine-tuning approaches <cit.>. Post-hoc methods <cit.> leverage the logit space and output scores of models that trained on in-distribution data to classify ID and OOD data. Fine-tuning approaches <cit.> introduce extra regularization terms during training or incorporate auxiliary training data, referred to as outlier exposure, which can be either real, synthetic, or sampled from the feature space. Outlier exposure has proven effective in enhancing OOD detection performance. 

However, these methods are typically applied to image or text data. OOD detection in graph data remains an under-explored area. Graph OOD detection <cit.> aims to determine whether test graphs originate from within the in-distribution relative to the training set or are out-of-distribution. Some studies <cit.> focus on graph anomaly detection, where the training data comprises both in-distribution and anomalous data, positioning it as a subset of graph OOD detection. OCGIN <cit.> utilizes a GIN as its encoder and capitalizes on an SVDD <cit.> objective for one-class graph anomaly detection. GLocalKD <cit.> employs joint random distillation to pinpoint anomalous graphs on both local and global scales. GOOD-D <cit.> differentiates between ID and OOD graph data through hierarchical contrastive learning and perturbation-free graph data augmentation, having been exposed only to ID data during training. AAGOD <cit.> introduces a novel learnable amplifier generator, designed to generate graph-specific amplifiers, thereby enhancing the detection of OOD graphs on trained GNNs. GraphDE <cit.> introduces a generative framework for debiased learning and OOD detection in graph data.

Graph Data Augmentation. 
Graph data augmentation, involving transformations to enrich or enhance information in the given graph, has been extensively explored. Non-learnable augmentation methods <cit.> achieve this through perturbation or random sampling of edges, nodes, or subgraphs. In contrast, learnable augmentation methods <cit.> train an Augmenter with learnable parameters using techniques like Decoupled Training, Joint Training, or Bi-level Optimization <cit.>. Among these methods, some utilize graph mixup <cit.> to generate new graphs. G-mixup <cit.> enhances classification performance by generating new graphs through mixup based on estimating graphons of the same class. S-Mixup <cit.> employs soft alignments for node matching to achieve instance-level graph mixup.

As an orthogonal direction to existing graph-level OOD detection approaches, our approach focuses on introducing outlier samples that assist in OOD detection training. Our framework relies on utilizing in-distribution graphons to generate internal outliers within them, rather than directly performing augmentations on the original graph structure.



§ PRELIMINARIES

In this section, we provide definitions for key terms and concepts utilized throughout this paper.



 §.§ Problem Setting

Denote a graph by G =(𝒱, ℰ, X), where 𝒱 denotes the set of nodes, ℰ denotes the set of edges, and X∈ℝ^n × d is the node feature matrix. The adjacency matrix is denoted as A∈{0,1}^n × n, where A_ij = 1 indicates a connection between nodes v_i and v_j and A_ij = 0 otherwise.

In this paper, we focus on the unsupervised graph-level OOD detection problem. Specifically, given a set of unlabeled in-distribution graphs 𝒟^in = {G_i}^N_i=1 drawn from the distribution 𝒫^in, the aim of unsupervised graph-level OOD detection is to learn a graph OOD scoring function f(·) based on the ID data 𝒟^in. A higher score s=f(G) indicates a higher probability to be an OOD graph. The scoring function is evaluated on a test set 𝒟_test= 𝒟_test^in∪𝒟_test^ood (𝒟_test^in∩𝒟_test^ood = ∅) where 𝒟^in_test∼𝒫^in and 𝒟^ood_test∼𝒫^ood. It should be emphasized that graph data sourced from 𝒫^in and 𝒫^out might fall into multiple categories. However, in the unsupervised graph-level OOD task, the model is not provided with any category-specific labels.



 §.§ Graphon

A graphon, denoted by the function 𝒲:[0,1]^2 → [0,1], is a continuous, bounded, and symmetric function in graph theory, extensively used to describe graph generation <cit.>. The value 𝒲(i,j) approximates the probability of an edge between nodes i and j in a specific graph. A graphon can be considered as a function that embodies the characteristics of a class of graphs, allowing for the sampling of graphs from the graphon. These sampled graphs share similar topological features. Distinct graphons represent different graph classes, enabling comparative analysis of their structural features. These graphons are estimated using step function approximations <cit.>. Graphons are particularly useful in synthesizing new graphs that reflect patterns found in real-world data, offering insights into the underlying structure of complex networks. However, there is no closed-form expression for graphons. Previous studies <cit.> have employed a two-dimensional step function to estimate graphons, which can be considered a matrix representing the probability of edge existence. In this paper, we denote it as W ∈ [0,1]^D × D, where D is the dimensionality of the graphon. This matrix W can then be used to generate graphs with a number of nodes less than D.



§ METHODOLOGY





 §.§ Overall Framework



In general, with a graph OOD score function f(G), the basic learning objective for unsupervised graph OOD detection can be described as:

    min_f 𝔼_G ∼𝒟_in [ℒ_ood(f(G))],

where ℒ_ood is the loss function.

By integrating the hybrid graph outlier exposure (HGOE) procedure, we hope the exposed OOD data have higher OOD scores than ID data. Subsequently, we simultaneously minimize the OOD score of ID data and maximize that of OOD data:

    min_f 𝔼_G ∼𝒟_in   [ℒ_ood(f(G)) + . 
       .     β·𝔼_G^'∼𝒟_OE [ℒ_GOE(f(G),f(G^'))]].


Apparently, such an introduction of HGOE is independent of the basic graph OOD model and thus can be applied to most existing models.

In deploying the GOE framework, we identify two pivotal challenges: (C1) How to acquire high-quality graph outlier data? (C2) How to design a ℒ_GOE to effectively utilize these OE data? To tackle (C1), we introduce both internal and external outliers. To address (C2), we develop a boundary-aware loss function. 
The overall pipeline of HGOE is illustrated in Figure <ref>. To facilitate the generation of internal outliers, we start with the division of graphs into distinct subgroups, wherein each subgroup is defined by the similarity in properties among its constituent graphs.  This is followed by the estimation of the graphon for each subgroup. Thereafter, we deploy an ID-mixup technique that combines the graphons from these subgroups to obtain internal outliers, incorporating node feature information from external outliers into this process. The resulted internal and external outliers are then fed into the OOD detector, which is further optimized through the implementation of an innovatively designed boundary-aware loss. In subsequent sections, we will introduce the HGOE framework in detail.



 §.§ Mixture Outlier Training Strategy

Our mixture outlier training strategy incorporates outliers from two sources: real-world external outliers and synthesized internal outliers. External outliers are derived from public graph datasets similar to the training graphs, while internal outliers are generated from ID graphs.

External Outliers.
In contrast to image data, graph data in the real world display significantly more complex structures, with notable gaps in characteristics between fields like social networks and protein networks. To investigate the role of real-world outliers, we sample from several graph datasets to create a real-world OE dataset. During the training of different ID data, we select graphs from the OE dataset that are consistent with them in feature dimensions, ensuring no overlap between the OE graphs and the training and test sets. This process results in an external outlier dataset, denoted as 𝒟^ext_oe.

Internal Outliers.
The quality and diversity of external outliers data are crucial for outlier exposure, but are limited by the scope and quality of the existing auxiliary dataset. Synthesizing additional outliers then emerges as a solution, offering tailored, abundant, and diverse graph data for training. However, the challenge lies in ensuring that the synthesized data resonates with real-world outlier scenarios and does not introduce unintended biases. As discussed in the introduction, there exist internal outliers among the in-distribution graph data, which are distributed near the boundaries of subgroups. To obtain these internal outlier samples, we design an internal outlier generation strategy, which will be described in the next subsection.



 §.§ Internal Outlier Synthesis





A straightforward way to generate graphs is to interpolate each pair of samples in the dataset directly. However, unlike Euclidean data such as images, different graphs usually have different sizes (e.g., different numbers of nodes) and have unique topology in Non-Euclidean spaces. Therefore, we propose to mix up the graph generators instead of graph samples themselves to synthesize more realistic graph data.

To find internal outliers for input graphs, it is imperative to first divide the graphs into subgroups. For the input ID graphs, we adopt graph contrastive learning techniques (e.g., GraphCL <cit.>) for feature extraction, and then perform k-means clustering to obtain k subgroups 𝒞_i, where 𝒟^in = ⋃_i=1^k𝒞_i. 
Since the members in each subgroup often share similar properties, we assume that each subgroup within the ID data is generated by a specific graphon. And we use the widely-used universal singular value thresholding (USVT) method <cit.> to estimate the graphon W_i for each 𝒞_i in 𝒟^in.
    
For each pair of non-overlapping subgroups 𝒞_i and 𝒞_j in 𝒟^in, we perform a mixup operation over the corresponding graphons W_i and W_j as follows:

    ℳ = λ W_i + (1 - λ) W_j,

where λ∈ [0,1] is the balancing hyperparameter. The mixup result ℳ combines the structural features of subgroups 𝒞_i and 𝒞_j, and can be considered a new graph generator positioned between the two subgroups. By sampling based on this graphon, we could obtain graphs lying in the interpolated regions between these subgroups, preserving the subgroups's topologies.


Random Size Based Sampling. 
The interpolated graphon ℳ∈ [0,1]^N × N has the capability to generate infinitely many graphs. However, the naive generated graphs are very likely to have a size around N. This limits the diversity of the synthesized outliers, which does not meet our goal in the HGOE framework to synthesize outliers broadly distributed near the original subgroups, with various sizes of graphs. In order to increase the diversity of outliers, we employ a random size based sampling strategy. We first randomly sample the size of the graph r ∈ [2, N] and then generate the graph from the sampled graphon ℳ'  ∈ [0,1]^r × r. The existence of an edge between nodes i and j is determined by sampling from a Bernoulli distribution with the parameter ℳ'(i, j).


External Feature Alignment.
Merely sampling from the interpolated graphons could only produce pure structure graph outliers with the structure information (denoted as A'). Therefore, we further propose to generate node features based on external outlier features. Specifically, we calculate the structural features of both the generated internal outlier structure A' and the external outliers. And then we construct the node features for the generated internal outliers based on that of the external outliers with the most similar structure features. Here, the structural features s^diff_i of the i-th node in a d_s-dimensional space is obtained through a d_s-step random walk-based diffusion process on the graph:

    s^diff_i = [T_ii, T^2_ii, …, T^d_s_ii] ∈ℝ^d_s,

where T = A'D^-1 denotes the transition matrix for the random walk on the graph, and D is the degree matrix of A'.  After computing the structural features of A', we compare them with the structural features of external outliers. For each node in the synthesized internal outlier G', we assign node features that are most closely aligned with the structural features of nodes in the external outliers. Note that our external feature alignment approach is training-free. By performing structural searches on nodes with existing features, we can effectively assign outlier features to nodes that initially lack features.

Through the aforementioned ID-mixup based procedure, we can generate an arbitrary number of internal outliers, denoted as 𝒟^int_oe. Compared to real-world outliers, these synthetic outliers are not limited in quantity or source. Furthermore, being located near in-distribution samples, they effectively compensate for the uncontrolled quality issue of external outliers.



 §.§ Boundary-Aware OE Loss
 


The introduced outliers further present two critical issues: How can we ensure that the introduced outliers do not fall within the in-distribution area, and how to find those more important outliers? Take the social graph ID data as an example. Intermixing social graphs could still result in a social graph, so the generated samples are not OOD data and should be excluded from the outlier dataset. At the same time, different outliers can have varying levels of importance, with points on the boundary potentially being critical points where a change in properties is about to occur.

To solve these problems, we design the following boundary-aware OE Loss ℓ_ba which is adaptively aware of whether the sample is in ID or OOD space. For an input outlier graph G', the loss is calculated as:

    ℓ_ba(s_G',τ)= -(l - s_G')^γmax(log(s_G'), τ),

where s_G'=sigmoid(f(G')) is the OOD score scaled by a sigmoid function, l and γ are hyperparameters. Note that τ is an ID-boundary threshold, which we adaptively set as the smallest s among the ID samples:

    τ=min_G ∈𝒟_insigmoid(f(G)).


To better understand Eq. (<ref>), we rewrite it in the following form:

    ℓ_ba(s_G',τ) = 
     
    -(l - s_G')^γlog(s_G'),    if log(s_G') > τ
    
    - τ (l - s_G')^γ ,    if log(s_G') ≤τ.

From this formulation, we have the following observations:

    
  * When log(s_G') > τ, it indicates that G' is a valid outlier. In this case, the smaller s_G' is, the closer it is to the in-distribution boundary. Apparently, such near-boundary outliers are important in guiding the detector to distinguish OOD samples from ID ones. Therefore, we weigh it by the term (l - s_G')^γ, giving higher weights to these boundary samples. As γ increases, the model pays more attention to samples nearer to the ID boundary.
    
  * When log(s_G') < τ, it means that the outlier G' is very likely invalid and fall into in-distribution space. Blindly treating it as an outlier might be harmful to the learning. Therefore, by minimizing - τ (l - s_G')^γ, the OOD score is reduced, which enhances the recognition ability for in-distribution samples.


Then, considering both external and internal outliers, the total graph outlier exposure loss is formulated as:

    ℒ_GOE=∑_G' ∈𝒟_OE^int∪𝒟_OE^extℓ_ba(s_G',τ).


Finally, we can instantiate the overall training objective ℒ as follows:

    ℒ   =  ∑_G ∈𝒟_in[ℒ_ood(f(G)) + β∑_G' ∈𝒟_OEℓ_ba(s_G',τ) ]  
       = ∑_G ∈𝒟_inℒ_ood(f(G)) + β∑_G' ∈𝒟_OE^int∪𝒟_OE^extℓ_ba(s_G',τ).

In practice, our ℒ_GOE can be added as a regularization term to most existing OOD detection models, without modifying their network architectures.



§ EXPERIMENTS
 

 
 






 §.§ Experimental Setup
 



  §.§.§ Datasets

We select three pairs of datasets from the TU dataset <cit.> and five pairs from the OGB dataset <cit.>. Each pair of datasets belongs to the same field and shares similar features, but exhibits distribution shifts between the two datasets in the pair. 90% of the In-Distribution (ID) samples are used for training, while 10% of the ID samples and an equivalent number of OOD samples are used for testing. For more detailed information about these datasets, refer to Table <ref>. 





For external outlier data, we grouped datasets with identical feature counts into a unified external dataset. Based on the feature counts of our selected datasets, as detailed in Table <ref>, this organization resulted in two distinct external dataset collections, with node feature numbers being 1 and 9, respectively. In this configuration, for each In-Distribution dataset, we utilized other datasets from the corresponding external dataset collection for training purposes, deliberately excluding both the OOD dataset for testing and the ID dataset itself.



  §.§.§ Competitors

We adopt the following three categories of graph OOD detection methods as our competitors:

    
  * Non-deep Two-step Methods.
    We use WL <cit.> graph kernels as feature extractors and employ local outlier factor (LOF) <cit.>, one-class SVM (OCSVM) <cit.>, and isolation forest (iF) <cit.> as detectors to perform OOD detection.
    
  * Deep Two-step Methods.
    The process is similar to the above methods, but the feature extractor is replaced with graph deep self-supervised methods InfoGraph <cit.> and GraphCL <cit.>, and the detectors are replaced with isolation forest (iF) and Mahalanobis distance (MD) <cit.>. Compared to graph kernels, self-supervised methods can extract features of graphs better.
    
  * End-to-end Methods.
    We utilize three popular end-to-end learning methods, including OCGIN <cit.>, which uses graph neural networks for feature extraction and is optimized with SVDD. GLocalKD <cit.> uses distillation learning for graph anomaly detection, and GOOD-D <cit.> employs multi-level contrastive learning for end-to-end OOD detection.

For the proposed framework, we instantiate it on the SOTA graph OOD detector baseline GOOD-D <cit.>. Besides, we also implement two ablated variants to show the impact of internal and external outliers. Specifically, HGOE w/o IO and HGOE w/o EO denote the variants without internal and external outliers, respectively.




  §.§.§ Implementation Details

For HGOE, the ratio of external to internal outliers was set to 1:1, with the total number equal to the number of in-distribution samples seen during training. The hyperparameter λ was set to 2, and γ was set to 2. The dimension of the structural features s_i^diff was set to 16, and the number of clusters for all datasets was determined to be 3. GraphCL was run for 50 iterations with an embedding dimension of 32, and the probabilities for node dropping, feature masking, and edge removing were all set to 0.1. During ID-mixup, λ was randomly chosen from the range [0.01,1]. AUC (Area under the ROC Curve) <cit.> is used as the performance metric. A higher AUC value indicates better detection performance.



 §.§ Main Results
 





We evaluate the performance of HGOE and other competitors, with the results presented in Table <ref>. Our findings include: 

(1) Two-step detection methods do not perform as well as end-to-end detection methods, especially where non-deep learning methods are inferior to deep learning-based feature extraction methods. Within self-supervised methods, GraphCL-MD demonstrates the best performance, showing GraphCL's capability in extracting features for graph OOD detection. This also explains why we chose to utilize GraphCL features for clustering. 

(2) For our HGOE framework, compared to GOOD-D without the use of graph outliers, there was an enhancement in performance across all 7 datasets. On the ENZYMES+PROTEIN dataset, the average performance improve from 60.15 to 64.44. On Tox21+SIDER, it increased from 64.98 to 68.24, and on FreeSolv+ToxCast, the performance increase from 78.79 to 83.36.

(3) The performance improvement on IMDB-M+IMDB-B was not substantial because both are social datasets, and the outliers we introduce were from molecular and protein datasets, which are biologically oriented. However, using these as outliers still contributed to enhanced detection on IMDB-M+IMDB-B. This indicates that selecting outliers similar to the ID distribution, as opposed to introducing outliers completely unrelated to the ID data, is more effective.



 §.§ Visualization




  §.§.§ ID-mixup Visualization.
 After obtaining the subgroups within the known distribution, we performed a mixup of the estimated graphons and visualized the resulting mixed graphons in the form of heatmaps, as shown in Figure <ref>. The graphon in the center represents the result of performing ID-mixup on the two adjacent graphons. It is evident that the graphon after ID-mixup retains the structures of the original graphons, forming a new graph generator. This demonstrates that our ID-mixup can effectively blend the distributions of subgroups to a certain extent, fulfilling our hypothesis.






  §.§.§ Score Distribution Visualization.
 Based on the OOD scores for samples in the test set, we visualize the frequency distribution of OOD scores for both ID and OOD samples using different colors. As shown in Figure <ref>, the left column depicts the score distributions without using HGOE, while the right column shows the results after applying the HGOE framework. The less the overlap and the greater the distance between the ID and OOD areas are, the better the corresponding model's performance is. It is observable that our HGOE method has decreased the overlap area between ID and OOD distributions, which also explains the reason for the performance improvement.










 §.§ Ablation Study
 



  §.§.§ Using Only One Type of Outliers.
 In Table <ref>, HGOE w/o IO and HGOE w/o EO represent the results of using only internal outliers and external outliers, respectively. We observe that using just one of these types yields better results than not using any OE (Outlier Exposure) samples at all. Moreover, in most cases, the performance of using only external outliers is superior to that of using only internal outliers, but inferior to using both types. This indicates that the combination of both types of outliers can more effectively enhance the performance of hybrid graph outlier exposure.



  §.§.§ The Strategy for Adaptive Allocation of τ.

In Section <ref>, we introduce an adaptive parameter τ in the  distribution-aware OE loss, which is set as the smallest normalized score among the ID samples. To explore the specific role of τ, we implement methods setting τ according to the maximum, average, and minimum normalized scores. The results, as shown in Figure <ref>, indicate that the min strategy for allocating τ performs the best. However, this strategy negatively affects performance on the IMDB-M+IMDB-B dataset, which we believe is due to its significant difference from the other datasets.






 §.§ Sensitivity Analysis






  §.§.§ ID-mixup Weight λ.
 We explore the sensitivity of HGOE with respect to λ by evaluating its performance during ID-mixup across different λ ranges. As seen in Table <ref>, we progressively narrow the range of λ from [0.01,1.0] to [0.4,0.6]. In this process, a performance decline was observed in most datasets. This suggests that for ID-mixup, blending two in-distribution samples at varying ratios can increase the diversity of internal outliers, thereby enhancing detection effectiveness. However, an opposite trend of performance increase, rather than decrease, was observed in the CLintox+LIPO dataset pair. This anomaly may be related to the characteristics of these datasets, both being molecular graphs from the OGB and possessing extremely similar average node and edge counts, resulting in closer distributions. If ID-mixup outliers falling within the in-distribution range, it could lead to a greater performance decrease compare to other datasets. So when λ is set to a middle value, the generated graphon lies between subgroups. This explains why choosing a λ value range [0.3,0.7] yields better results.



  §.§.§ Hyperparameter γ of Boundary-aware OE Loss.
 As described in our framework, γ plays a significant role in the boundary-aware loss. We vary the value of γ to {0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0}, and as observed in Figure <ref>, as γ increases, there is a performance improvement when γ∈ [0,2]. However, performance starts to decline for some datasets when it exceeds 2 and 2.5. This indicates that within an appropriate range, introducing γ can effectively weight the samples, and this effect improves with the increase of γ, up to a point where it begins to decrease. This phenomenon validates the rationality and effectiveness of our designed boundary-aware OE loss.









§ CONCLUSION

In this work, we investigate the enhancement of OOD detection performance on graph-level data through hybrid graph outlier exposure. We demonstrate that not only external outliers are crucial for graph OOD detection, but internal outliers also play a significant role. Based on this, we propose a carefully designed ID-mixup based method for synthesizing internal outliers by generating OOD samples between in-distribution subgroups and aligning these with external outlier features. Upon obtaining these synthesized internal outliers, we effectively utilize the characteristics of outlier samples by adaptively allocating learning weights to OE samples using a well-designed boundary-aware OE loss. Our HGOE framework provide substantial performance improvements for other graph OOD detection methods. Extensive experiments on 8 real-world datasets show its superior performance.



This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102000, in part by National Natural Science Foundation of China: 62236008, U21B2038, U23B2051, 61931008, 62122075 and 61976202, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No. XDB0680000, in part by the Innovation Funding of ICT, CAS under Grant No.E000000, in part by the China Postdoctoral Science Foundation (CPSF) under Grant No.2023M743441, in part by the Postdoctoral Fellowship Program of CPSF under Grant No.GZB20230732, and in part by the China National Postdoctoral Program for Innovative Talents under Grant BX20240384. 



ACM-Reference-Format




 Paper Title: Towards Scalable GPU-Accelerated SNN Training via Temporal Fusion.







Y. Li et al.

Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China 

ranchengcn@gmail.com


Towards Scalable GPU-Accelerated SNN Training 
via Temporal Fusion
    
    Yanchen Li0009-0002-2237-0451 Jiachun Li0009-0004-4182-9664Kebin Sun0009-0008-9213-7835 Luziwei Leng0000-0002-9344-8589 Ran Cheng()0000-0001-9410-8263

    August 5, 2024
================================================================================================================================================================



-1.5em

This work was supported in part by Guangdong Natural Science Funds for Distinguished Young Scholar under Grant 2024B1515020019.
Yanchen Li and Jiachun Li contributed equally to this work.
Corresponding author: Ran Cheng ()

§ ABSTRACT 
Drawing on the intricate structures of the brain, Spiking Neural Networks (SNNs) emerge as a transformative development in artificial intelligence, closely emulating the complex dynamics of biological neural networks. 
While SNNs show promising efficiency on specialized sparse-computational hardware, their practical training often relies on conventional GPUs. 
This reliance frequently leads to extended computation times when contrasted with traditional Artificial Neural Networks (ANNs), presenting significant hurdles for advancing SNN research.
To navigate this challenge, we present a novel temporal fusion method, specifically designed to expedite the propagation dynamics of SNNs on GPU platforms, which serves as an enhancement to the current significant approaches for handling deep learning tasks with SNNs.
This method underwent thorough validation through extensive experiments in both authentic training scenarios and idealized conditions, confirming its efficacy and adaptability for single and multi-GPU systems.  
Benchmarked against various existing SNN libraries/implementations, our method achieved accelerations ranging from 5× to 40× on NVIDIA A100 GPUs. Publicly available experimental codes can be found at <https://github.com/EMI-Group/snn-temporal-fusion>.








§ INTRODUCTION

Deep learning has firmly entrenched itself as a transformative field in artificial intelligence. 
Python, as a preferred programming language, has propelled this transformation. 
Notably, by leveraging the robust GPU computing supports of the Compute Unified Device Architecture (CUDA) <cit.>, PyTorch <cit.> stands out in the Python ecosystem, offering a robust platform for researchers working on traditional Artificial Neural Network (ANN) structures. 


As the deep learning landscape evolves, the integration of biological neural dynamics, manifested in the form of Spiking Neural Networks (SNNs), has emerged as a promising research direction. 
Presently, the spiking attribute is primarily viewed as a module or characteristic that can be integrated into existing ANNs. 
This perspective is largely influenced by studies focused on converting ANNs into SNNs <cit.>, as well as training SNNs directly with ANN methods <cit.>. 
Consequently, ANNs serve as foundational pillars for contemporary SNNs. 
Reflecting this trend, numerous SNN libraries have been developed atop the PyTorch framework, each distinguished by its unique focus <cit.>.

Despite the promising features of SNNs, a significant challenge arises with the incorporation of the temporal dimension, which often leads to slower training speeds. 
Many researchers have sought to mitigate this by truncating the network's temporal length <cit.>. While expediting training, this approach inadvertently suppresses its temporal characteristics and makes SNNs closely resemble ANNs <cit.>.
Moreover, the relatively slow execution speeds of SNNs on prevalent GPU platforms make it challenging to train and validate the models on large scales. 
Undoubtedly, such computational bottlenecks have impeded further exploration of the field. 


To address these challenges, various upper-level acceleration algorithms have been proposed, focusing on the integration of spiking neuron mechanisms with advanced training strategies <cit.>. 
In contrast, research on accelerating deep learning models has predominantly concentrated on lower-level computational improvements. 
These involve parallel and distributed processing techniques applied to both data and models, which have primarily been implemented in ANNs to boost efficiency in solving deep learning tasks <cit.>.
While these contributions are noteworthy, the lower-level computational infrastructures specifically designed for SNNs are yet to be further explored.


To bridge the gap, in this work, we seek to address the challenge of low-speed execution inherent in SNNs by introducing a theoretically sound solution, leveraging the temporal dynamics of SNNs during signal propagation.
This method effectively reduces the latency associated with the temporal aspects of processing, thereby facilitating the scalable training of SNNs across distributed GPU environments. 
Our contributions can be summarized as follows.

    
  * We propose a novel temporal fusion method specifically designed to expedite the training of SNNs. 
    Drawing inspiration from prior studies, our method decouples the spiking neuron model's propagation patterns along the temporal axis. 
    This strategy facilitates more efficient information flow and maintains computational precision without resorting to approximate substitutions, thereby ensuring that the model's outcomes are both simple and accurate.
         
    
  * We have extended the proposed temporal fusion method to leverage multiple GPUs, moving beyond traditional batch-based methods. 
    We adopt a pipeline parallelism framework, which is particularly suitable for the intrinsic temporal dynamics of spiking neurons, enabling the distribution of computational loads across GPUs in a temporal manner. 
    Theoretically, this design allows for scalable performance improvements that grow with the increase in time steps, offering significant acceleration benefits.
    
    
  * To empirically validate the acceleration capabilities of our method, we conducted experiments on deep learning tasks using widely adopted SNN architectures during both training and testing phases.
    Our implementation, developed on the PyTorch platform with CUDA optimization, is specifically engineered to reduce memory access latency, thereby enhancing efficiency across various temporal scales. 
    The results from our rigorous testing, devoid of extraneous computational burdens, demonstrate substantial performance improvements in both single and multi-GPU setups.





§ RELATED WORK




 §.§ GPU Acceleration for Deep Learning

In the realm of GPU acceleration for deep learning, NVIDIA's CUDA technology is particularly noteworthy for its high-level interface that enables direct interaction with GPUs. 
CUDA serves as a crucial software layer that allows developers to engage directly with the GPU's virtual instruction set and parallel computing components, thus facilitating efficient execution of kernel computations. 
This proximity to the hardware layer significantly enhances development flexibility and efficiency.

To further streamline the development process, CUDA is complemented by a suite of optimized standard routines.
A primary example is cuDNN <cit.>, which provides a comprehensive set of deep neural network operator implementations within the CUDA ecosystem, substantially simplifying the development of deep learning applications. 
These foundational technologies underpin many high-level deep learning frameworks, enabling robust and efficient experimentation.


There have also been significant advancements in parallel and distributed acceleration methods. 
Initial efforts, such as those by Krizhevsky et al. <cit.>, allocated neural network models across multiple GPUs, paving the way for processing large deep-learning models with multiple GPUs. 
Following this, more sophisticated distributed training methods have emerged.
For example, Megatron-LM <cit.> introduced an intra-layer model parallelism method for large language models, enhancing accuracy while maintaining performance. 
GPipe <cit.> explored serial operators in foundation models, implementing pipelined serial sub-layers and facilitating pipeline parallelism across multiple GPUs. 
These methods significantly boost model computation efficiency on supported platforms, with many integrated into existing deep learning frameworks to augment the efficiency of large-scale model development and research.



 §.§ Neuromorphic Computing Infrastructures

In light of the integration of spiking neuron properties into deep learning, several infrastructures tailored for SNNs have emerged, each focusing on different facets of the domain. 
BindsNET <cit.> emphasizes applications in machine and reinforcement learning. 
Norse <cit.> and snnTorch <cit.> expand on SNNs within the PyTorch ecosystem, prioritizing comprehensive functional support alongside an extensive documentation suite, praised for its user-centric design. 
SpikingJelly <cit.>, on the other hand, accentuates algorithmic advancements in SNNs, offering two backends and two SNN propagation mechanisms, thereby fostering both algorithmic enhancements and performance optimizations. 

In addition, certain simulation frameworks for neural systems have inspired the fundamental design of SNNs.  
Libraries such as cuSNN <cit.> and GeNN <cit.>, built directly on CUDA, aim to minimize runtime overhead through efficient, substrate-based designs. 
These frameworks facilitate a closer interaction with the hardware layer, enhancing the execution efficiency of SNN simulations.



 §.§ Spiking Neurons

As the fundamental units of the brain, neurons exhibit unique information transfer properties. 
To emulate these intricate behaviours in computational models, various spiking neuron models have been proposed. 
Notably, the Spike Response Model (SRM) <cit.> represents a broad category of spiking neurons, encompassing parameters like membrane potential decay, spike threshold, refractory period, etc.
Essentially, the membrane potential of a neuron undergoes continuous decay unless it receives an external stimulus. 
Upon receiving information, the potential increases until it hits a threshold, resulting in the generation of a spike, followed by an immediate potential drop. 

Spiking neuron models for deep learning are relatively complex in the early stages, e.g., Hodgkin-Huxley <cit.>, Izhikevich <cit.>. 
However, complex models, while accurate, often introduce implementation challenges. 
As the deep learning domain advances, the need for simplicity becomes paramount. 
Therefore, the Leaky-Integrate-and-Fire (LIF) model and its variants are preferred at present <cit.>. 
LIF emerges as a simplified version of SRM, preserving its core information transfer mechanisms while reducing the intricacies of neuronal information transmission. 
Recognizing the potential of this simplification, recent research has further refined the LIF model, yielding an iterable form <cit.>. 
This adaptation ensures that spiking neurons seamlessly integrate into the deep learning paradigm without compromising the salient features of SRM.





§ METHOD

We first analyze the properties of LIF neurons in forward and backward propagation, showcasing the potential for acceleration optimization. 
Then, we delve into the foundational aspects of the temporal fusion method and its theoretical implications when executed on a single GPU.
Finally, we explore the method's scalability across multiple GPUs.








 §.§ Parallelism of LIF Spiking Neurons

For a specific layer within the iterative LIF model of a SNN, let v_i^(t) denote the membrane potential of the i-th neuron at the t-th time step, and y_i^(t) represent its corresponding output. 
Initially, we assume the membrane potential of each neuron is v_i^(0) = V_rest, where V_rest indicates the resting potential. 
Additionally, k_τ is defined as the decay factor that influences the membrane potential's evolution over time.
Given these, the iterative dynamics of the LIF model for each neuron can be formulated as:

    v_i^(t) = k_τ· v_i^(t-1)·(1 - y_i^(t-1)) + V_rest· y_i^(t-1) + x_i^(t),

where the mechanism for spike generation is given by:

    y_i^(t) = H(v_i^(t) - V_th) =
        
        1,    if  v_i^(t) - V_th≥ 0, 
    
    
        0,    otherwise,

with H denoting the spike activation function characteristic of spiking neurons, and x_i^(t) being the integrated output from the preceding layer's neurons.

Eqs. (<ref>) and (<ref>) elucidate the forward propagation mechanism of the iterative LIF model, providing a detailed account of the spiking process while preserving the essential attributes of spiking neurons. 
Consequently, the discussions and methodologies presented in this paper are based on these formulations.

The structure of the LIF model's forward propagation inherently supports its backward propagation, which is essential for direct SNN training. 
This backward propagation is expressed as:

    ∇_x_i^(t)L    = k_τ·∇_v_i^(t+1)L·[1-y_i^(t)-v_i^(t)·δ(v_i^(t))] + ∇_y_i^(t)L·δ(v_i^(t)),

where L denotes the loss function. 
The spike function H, being non-differentiable as indicated in Eq. (<ref>), necessitates δ as an approximate surrogate gradient function.

Notably, both forward and backward computations in the LIF model are conducted on an element-wise basis, as opposed to the fully connected and convolutional operations. 
This means that each neuron functions autonomously, without reliance on a specific computational sequence or interaction with others. 
Such autonomy is pivotal, as it allows for parallel processing and optimization, significantly boosting computational efficiency.







 §.§ Temporal Fusion on a Single GPU


As given by Eq. (<ref>), the dynamics of the membrane potential in SNNs are intricately tied to both temporal and spatial factors. 
This entails that the activity of any given neuron is predicated on its preceding state at the last time step, as well as the input it receives from neurons within the previous layer. 

Drawing inspiration from prior research <cit.>, an intriguing observation emerges: Time-step-wise inference is fundamentally equivalent to layer-wise inference, which occurs after the completion of all time steps in each layer of the computation. 
This characteristic inherent to the latter opens a path for optimization across the temporal dimension, particularly when the initial states for all time steps are predetermined.


Notably, prioritizing the computation across each layer over all time steps yields two primary benefits. 
First, the element-wise computation inherent to the monolayer LIF model naturally facilitates parallelization at the neuron level, a feature that extends to temporal propagation.  
Second, executing computations across all time steps mandates the inclusion of data from all preceding time steps. 
Given that data for each time step retains a consistent format, strategic memory alignment post-concatenation can significantly reduce the overhead tied to batch memory operations, thereby elevating computational efficiency.

Building upon these insights, we introduce the temporal fusion method, which leverages the element-wise nature of spiking neuron layers. 
This method assigns each neuron's computation to an individual GPU thread, amalgamating memory operations for each layer across all time steps within the GPU kernel.  
Such fusions substantially reduce the overhead from multiple memory interactions and thus augment computational throughput. 
The overarching framework is illustrated in Fig. <ref>(a).




For detailed insight, Fig. <ref> illustrates the forward and backward propagation processes under the temporal fusion method, respectively. 
In these figures, each block containing neuronal information relies on preceding data within the unfolded temporal dimension, underscoring the interconnected nature of computations.

Theoretically, training with temporal fusion diverges from conventional serial training, which typically emphasizes sequential computation at the network layer. 
Instead, temporal fusion adopts a temporal-major order approach, computing all time steps within each layer to facilitate temporal operator fusion. 
Although reordering computations does not impact the end results, strategic utilization of this aspect can effectively minimize access losses. 
The contrast between serial and temporal fusion-based training is showcased in Fig. <ref>, illustrating how serial training's frequent, minor memory operations incur linearly scaling overhead; contrastively, temporal fusion training sidesteps such overhead, offering increased benefits with a rising number of time steps.



 §.§ Temporal Fusion across Multiple GPUs

As elaborated in the preceding subsection, the temporal fusion method enhances computational efficiency by minimizing memory access and leveraging operator fusion, even with small time steps. 
However, as time step sizes increase, a single GPU may encounter bottlenecks due to expanded temporal dimensions, potentially leading to storage constraints. 
The limited memory of a single GPU could necessitate halting computations or reverting to less efficient serial processing upon exceeding capacity. 
Furthermore, larger time steps contribute to increased computational latency. 
Utilizing multiple GPUs can mitigate these issues by distributing the computational workload. Consequently, we extend the temporal fusion method to a multi-GPU setup, as illustrated in Fig. <ref>(b).

In this expanded configuration, inter-GPU communication enables cross-device operator fusion and data exchange along the temporal dimension. 
Suppose k (k ≥ 1) identical GPUs are available. 
If the computational load for a specific SNN layer is enough to be divided into k segments along the temporal dimension on a single GPU, the completion time for k sub-tasks on these GPUs would be t^(k). 
Assuming the inter-GPU communication time is T_c. Ideally, we have:

    
    T_s   = k· t^(k), 
    
            T_m^(k)   = (k-1)· T_c + t^(k),


where T_s and T_m^(k) represent the total task time on a single GPU and k GPUs, respectively. 
The speedup rate μ for the multi-GPU setup is then defined as:

    μ = T_s/T_m^(k) = k· T_s/k(k-1)· T_c + T_s.



[16]l0.495
    -2.5em
    
    < g r a p h i c s >

    -0.8em
    The simulated relationship between the acceleration ratio and the GPU count under three task conditions, as per Eq. (<ref>).
    



Eq. (<ref>) reveals that the acceleration ratio μ depends on the GPU count k, influenced by T_s and T_c. 
When considering μ as a continuous function of k, its optimum is achieved at k = √(T_s/T_c). 
Thus, selecting a GPU count k close to √(T_s/T_c) maximizes acceleration for given T_s and T_c values in practical implementations.

Fig. <ref> demonstrates the simulated relationship between GPU count k and acceleration ratio μ for three distinct T_s/T_c ratios. 
An increase in μ is observed as k nears √(T_s/T_c). 
Given that T_c remains constant across LIF spiking neuron layer time steps, while T_s escalates with time step extension, it suggests that multi-GPU deployment becomes increasingly advantageous as SNN time steps expand.


[label=list:torch-cuda, language=Python, float=!t, belowskip=-2em, caption=Illustration of the programming model for our implementation in PyTorch with CUDA. The  represents the custom CUDA kernel package encapsulating the temporal fusion method. Within this context,  aggregates a tensor for all neurons i across each time step t, corresponding to x_i^(t) in Eq. (<ref>). Similarly,  denotes the aggregated gradient tensor, corresponding to ∇_y_i^(t)L as delineated in Eq. (<ref>). The  serves as a trainable and testable construct.]
      import torch
      import temporal_fusion_kernel

      # Define function to acquire kernel
      class FusedLIF(torch.autograd.Function):
        @staticmethod
        def forward(ctx, x, args):
          ctx.args = args
          return temporal_fusion_kernel.fusedForwardLIF(x, args)
        @staticmethod
        def backward(ctx, grad_y):
          args = ctx.args
          return temporal_fusion_kernel.fusedBackwardLIF(grad_y, args)

      # Define LIF layer module
      class LIFLayer(torch.nn.Module):
        def __init__(self, args):
          self.args = args
        def forward(self, x):
          return FusedLIF.apply(x, self.args)
        
      # Define hyper-parameters and model
      args  = ...  # hyper-parameters
      model = torch.nn.Sequential(..., LIFLayer(args), ...)





§ IMPLEMENTATION

This section provides an implementation developed with our temporal fusion method. 
Specifically, the implementation leverages the foundational architecture of PyTorch with CUDA. 
While PyTorch supports the definition and utilization of SNNs, CUDA accommodates both single and multiple GPU operator fusion configurations. 
We begin by introducing the programming models tailored for PyTorch and CUDA based implementation, with a discussion about the implementation schemes and an exploration of the design considerations for the associated functional modules.

To demonstrate the functionalities offered by our implementation, we present the programming model for defining spiking neuron layers with the temporal fusion method, as exemplified in Listing <ref>.
This example demonstrates how to define a temporally fused LIF neuron layer which can be used to design SNNs optimized for single and multi-GPU acceleration. 

Notably, our implementation primarily focuses on the aspect of designing high-performance spiking neurons. 
To this end, the pre-compilation by CUDA ensures high performance of the proposed temporal fusion, for both forward and backward propagations.
However, modern SNNs usually involve both ANN operators (e.g., convolution) and spiking neurons. 
Hence, from the practical point of view, we integrated our implementation into the PyTorch framework. 







§ EXPERIMENT

Our experiment comprises three parts.
Initially, we evaluated the temporal fusion method within various SNN architectures across a spectrum of time steps, utilizing both static image and event-based datasets. 
Subsequent experiments focused on assessing the temporal fusion method's performance using a LIF unit across different time steps, highlighting optimal conditions. 
Lastly, we present benchmark results from multi-GPU configurations, emphasizing the scalability and advantages of employing multiple GPUs.






 §.§ Experimental Setup

All experiments were meticulously carried out on NVIDIA A100 GPUs to encompass runtime comparisons across different platforms and implementations.
To ensure equitable comparisons, we aligned the hyperparameters as specified in Eq. (<ref>), setting V_rest=0, k_τ=0.2, and V_th=0.3. 
For libraries lacking the k_τ hyperparameter, we substituted with τ=1.25 based on the relation k_τ = 1 - 1/τ. 





 §.§ Main Results


In this experiment, we selected three architectures: Spiking-ResNet18, Spiking-ResNet34, and Spiking-ResNet50, as informed by existing research <cit.>. Referring to the SpikingJelly, we integrated these architectures with our acceleration method for comprehensive testing. 
The learning rate was set to 0.001 in the Adam optimizer <cit.>, with time steps fixed at 32 over 5 epochs, and the employment of cross-entropy loss. 
We adopted the sigmoid surrogate function as introduced in <cit.>:

    δ(x) = σ'(x) = α e^-α x/(1+e^-α x)^2,

with α=4.0. 
All tests across various datasets and architectures were conducted three times, averaging the results to ensure reliability.



Initially, we employed two static image datasets, MNIST <cit.> and CIFAR-10 <cit.>, to evaluate both the acceleration capabilities and the accuracy of our method. 
For these static datasets, the Poisson Sampling method was used during pre-processing, with a batch size of 100. Subsequently, we applied the same architectures to two event-based datasets, N-MNIST <cit.> and DvsGesture <cit.>, adjusting batch sizes to 100 and 10, respectively.
During pre-processing for these datasets, we utilized the Stacking Based on the number of Events (SBE) approach as implemented in SpikingJelly.

As summarized in Table <ref>, the results indicate that our method achieves comparable accuracy to existing implementations across different network architectures for both static and dynamic datasets, while also ensuring considerable speedups in both training and testing phases.





 §.§ Scalable Time Steps

This experiment assesses our method's scalability with respect to time steps in a single-GPU setting. 
To focus solely on our method's acceleration performance, all ANN operators were omitted, centering the experiment on a single-layer LIF model with 1,000,000 neurons. 
Results reflect the aggregated time from 1,000 independent trials.

Fig. <ref> showcases our temporal fusion method's remarkable acceleration performance, achieving up to a 40× speedup over traditional implementations, demonstrating its scalability concerning time steps.





 §.§ Multi-GPU Acceleration

The multi-GPU experiment involved deploying the temporal fusion method across 1 to 8 NVIDIA A100 GPUs, focusing on a monolayer LIF model with 1,000,000 neurons across various time steps. 
The reported outcomes are averages over 1,000 independent runs.
As shown in Fig. <ref>, a notable decrease in computational time was observed with increasing GPU count, highlighting our method's robust performance in a multi-GPU setting.






§ CONCLUSION

In this paper, we present a method, termed "temporal fusion", specifically designed for the efficient GPU-accelerated training of SNNs. To facilitate the practical application of this method, we have developed a CUDA-based implementation, complete with a detailed programming model. This implementation is seamlessly integrated with the widely-used deep learning framework, PyTorch, thereby enabling users to easily adopt and apply our method in their SNN research and development projects.
We have conducted a series of comprehensive benchmark tests on the temporal fusion method. These tests are meticulously designed to assess various aspects of the method, including its adaptability to different SNN architectures and training scenarios, as well as its impact on the overall efficiency of the training process. The results demonstrate the significant improvements in the training efficiency and wide applicability of our method.

While temporal fusion presents a promising avenue for scaling SNNs, certain challenges persist in our current framework. 
For instance, although our experimental findings confirm the viability of temporal fusion across both single and multi-GPU setups, the scalability of SNNs to larger dimensions may surpass the capacity of multi-GPU configurations within a single node. 
As such, extending support to multi-node systems for large-scale SNN acceleration emerges as an imminent challenge, awaiting resolution.
Moreover, the proposed solution is more focused on the forward and backward propagation acceleration of traditional deep learning-based training for SNNs. Despite the ongoing importance of this training method, it might lead to the loss of some temporal information, research on acceleration methods closer to the spiking neuron mechanism is one of the future works.

Since SNN research has started to focus more on larger temporal dimensions as they possess better temporal dynamics, we endeavour to research better utilization of these characteristics in future via temporal fusion. 
Importantly, harnessing the capabilities of large-scale SNNs could pave the way for advancements that edge closer to true intelligence, linking directly to the investigation of foundational support for large-scale models. 
The exploration of how to effectively synergize these two facets holds substantial promise as a future research direction.


splncs04





 Paper Title: Prompting Medical Large Vision-Language Models to Diagnose Pathologies by Visual Question Answering.



Einstein Probe discovery of a super-soft outburst from CXOU J005245.0-722844:
 a rare BeWD binary in the Small Magellanic Cloud
    [
    
===============================================================================================================================



§ ABSTRACT 
Large Vision-Language Models (LVLMs) have achieved significant success in recent years, and they have been extended to the medical domain. Although demonstrating satisfactory performance on medical Visual Question Answering (VQA) tasks, Medical LVLMs (MLVLMs) suffer from the hallucination problem, which makes them fail to diagnose complex pathologies. Moreover, they readily fail to learn minority pathologies due to imbalanced training data. We propose two prompting strategies for MLVLMs that reduce hallucination and improve VQA performance. In the first strategy, we provide a detailed explanation of the queried pathology. In the second strategy, we fine-tune a cheap, weak learner to achieve high performance on a specific metric, and textually provide its judgment to the MLVLM. Tested on the MIMIC-CXR-JPG and Chexpert datasets, our methods significantly improve the diagnostic F1 score, with the highest increase being 0.27. We also demonstrate that our prompting strategies can be extended to general LVLM domains. Based on POPE metrics, it effectively suppresses the false negative predictions of existing LVLMs and improves Recall by approximately 0.07.



    Medical Visual Question Answering, Large Vision-Language Models, Prompt Engineering




§ INTRODUCTION


Research on Large Language Models (LLMs) has yielded astonishing results in recent years. LLMs with billions of parameters have achieved outstanding abilities in a wide range of application scenarios <cit.>. The success of LLMs has quickly extended into the Vision-Language (VL) domain. Large Vision-Language Models (LVLMs) are built upon LLMs by training adapters that project visual features into features that can be interpreted by LLMs <cit.>. Visual Question Answering (VQA) is an essential skill of LVLMs, and VQA accuracy serves as a test metric for most of these models <cit.>. LVLMs have been pretrained on medical datasets <cit.> and they have been tested on medical VQA tasks <cit.>. These Medical LVLMs (MLVLMs) have been able to answer questions regarding the imaging modalities, organs, and abnormalities depicted by the input medical scans.

Unfortunately, however, “hallucination” has been a major problem for LVLMs. This refers to the generation of content that is contradictory to the input images. Hallucination can be measured via VQA. One may ask the model questions regarding the existence of objects in the input image(s) and the hallucination level is assessed as the percentage of correctly answered questions. VQA can also potentially serve for medical image diagnosis. Users pose questions regarding a pathology and the MLVLM responds based on its analysis of the medical scans. However, most of the available datasets involve simple questions such as “what is the modality of this image” and “what is the organ/tissue in this image”. MLVLMs have yet to be thoroughly evaluated on VQA accuracy across a broad spectrum of pathologies. Additionally, general VQA models are usually tested by the commonly known accuracy: the percentage of correctly answered questions, which is an unsuitable measure for medical VQA.
Medical image classification metrics such as the Precision, Recall, and F1 are more suitable for the evaluation of medical VQA models. Several strategies have been explored to enhance the question answering of LLMs/LVLMs, including chain-of-thought prompting <cit.>, self-consistency <cit.>, and retrieval-based augmentation <cit.>. All these methods involve fine-tuning the models, which is expensive. Training-free methods to improve the VQA accuracy of MLVLMs are desirable.

For MLVLMs, hallucination is exacerbated by imbalanced training data. Many pathologies are minority categories in medical datasets. Models trained on large-scale medical data may easily fail to learn the features of less common pathologies. Addressing data bias typically involves strategies such as including more data of better quality, but given the scarcity of medical data, significantly enlarging the dataset may not be feasible. Common remediations involve re-sampling the data such that the positive and negative cases are better balanced, but this poses challenges when the data involves multiple categories of pathology. Additionally, re-sampling may undermine the training needs of LVLMs, which generally require large quantities of data. These problems highlight the need of a cost-effective approach to navigate the problem of minority categories in datasets.

Our study focuses on the VQA abilities of MLVLMs. In particular, we test an existing MLVLM, LLaVA-Med <cit.> for chest X-ray VQA across 5 categories of pathologies. The results show that the model has low accuracy, especially on minority pathologies. To enhance its VQA accuracy, we propose two prompting strategies. The first involves enriching prompts with detailed explanations of the queried pathology. The explanations include how the queried pathology is defined and how it appears in images. Our second strategy involves introducing an auxiliary weak-learner model as another agent. We train a small image classifier and fine-tune it to identify negative images accurately. Then, the negative predictions of this classifier are appended to the prompt as a reference for the MLVLM.

We run our experiments on the MIMIC-CXR-JPG <cit.> and Chexpert <cit.> datasets. The results show that our prompt strategies improve the F1 score significantly in most pathology categories (highest +0.27). We also show that our weak-learner-prompting strategy is applicable to the general domain. It reduces the false negative predictions of general domain LVLMs and improves the Recall by around 0.07 according to POPE metrics <cit.>.

To summarize, our contributions include the following:

    
  * We improve the VQA accuracy of MLVLMs by prompting with detailed explanations of pathologies.
    
  * We introduce a low-cost weak learner model as a reference for LLaVA-Med, and this effectively reduces the false positive (FP) answers.
    
  * We show that our second prompting strategy can be extended to general domains to help models adapt to specialized accuracy needs.

<ref> reviews related work, <ref> describes our methodology, <ref> presents our empirical study and its results, and <ref> draws conclusions from our research.



§ RELATED WORK





  
LVLMs and VQA

LVLMs are built upon LLMs. A pretrained visual encoder extracts the visual features and an adapter module projects the extracted features to ones that can be understood by the LLM. Models of this type include those by <cit.>, <cit.>, and <cit.>. During training, the visual encoder and the LLM are usually fixed. VQA is an essential skill of LVLMs. Given an input image, the models should be able to answer questions correctly regarding that image.



  
Hallucination in LVLM VQA

The hallucination problem usually refers to the LVLM generating a response that is not consistent with the input image. For VQA, in their generated answers the models may make mistakes on object presence, location, attributes, or the mutual relationship between objects. <cit.> find that frequently occurring objects are easily hallucinated by LVLMs, in that they tend to mention such objects even if it they are absent in the image. <cit.> and <cit.> show that LVLMs sometimes presume the assumptions in questions are true and easily give wrong answers when asked about some objects not in the given image.



  
Causes of LVLM VQA Hallucination

Hallucination can result from bias in the training data, missing fine-grained visual features, and LLM decoding strategies <cit.>. For data bias, the imbalanced distribution of data is an important aspect. When most of the answers to a question in the training data are “Yes”, the model tends to answer “Yes” to that question. Missing fine-grained visual features usually result from the pretraining of the visual encoder. Most LVLMs use the visual encoder of CLIP trained through contrastive learning. The encoder mainly focuses on salient features and ignores fine-grained features <cit.>. LVLM decoding strategies mostly choose the next word as the one having maximum conditional probability given previous text and the input image. This can lead to hallucination when the model overly relies on the knowledge learned in its training texts. Other causes include model simplicity and insufficient attention <cit.>.



  
Mitigation of LVLM VQA Hallucination

Strategies to mitigate hallucination in LVLMs mainly fall into two categories: prompt engineering and model improvement. Regarding the former, <cit.> leverage visual instructions constructed from the bounding box information in the input image to prompt LLMs. <cit.> use a chain of thought scheme to prompt the models to perform step-by-step visual-language reasoning like humans, which eventually leads to the correct answers. <cit.> generate multiple chains of thought and use the one with the majority vote as the answer. <cit.> prompt the model with explanations of the terms in questions. With regard to the model improvement strategy for reducing hallucination, <cit.> improve the visual and text feature alignment through reinforcement learning. <cit.> propose a contrastive decoding strategy to reduce reliance on pretrained knowledge. <cit.> and <cit.> also focus on the inference stage and propose specialized decoding strategies to mitigate hallucination. Other strategies for reducing hallucination have been proposed. For example, <cit.> design a post-processing model to detect hallucinated objects and rephrase the generated answers, and <cit.> adapt a reinforcement learning strategy that uses human evaluation of the hallucination level to improve the model.



  
Assessment of LVLM Hallucination

There are two approaches to assessing hallucination in LVLMs. The first is VQA. The ground truth information of the input images is leveraged to construct questions regarding the existence of objects in the images (e.g., “Is there a black cat in the image?”), as well as questions about objects which do not exist in the images. The models are evaluated in terms of the percentage of correctly answered questions. Metrics of this type include POPE <cit.>, CIEM <cit.>, and NOPE <cit.>. The second  approach is to use pre-designed prompts from which the models produce various generations that are then evaluated. Examples include CHAIR <cit.>, which counts the hallucinated objects in generated image captions, and MMHAL-BENCH <cit.>, which uses GPT-4 <cit.> to compare the generations with human answers and determine the propensity toward hallucination.



  
VQA in MLVLMs

For MLVLMs, given a medical scan, models such as LLaVA-Med <cit.> and Med-PALM <cit.> are able to answer questions regarding the types of modalities, the scanned organs, and medical indicators such as opacity. They have demonstrated good performance on medical VQA datasets such as VQA-RAD <cit.>, SLAKE <cit.>, and Path-VQA <cit.>. However, most medical questions in existing datasets are simple. MLVLMs have not been tested on a broader range of complex pathologies.





§ METHODOLOGY



<ref> illustrates the structure of common LVLMs. They are based on a pretrained unimodal LLM such as Llama <cit.> and Vicuna <cit.>. A pretrained visual encoder, such as ViT <cit.> or conventional CNNs, is applied to extract image features that are projected to the text feature space by an adapter. The projected visual features are concatenated with the text prompt embeddings and fed to the LLM. The adapter usually consists of several linear layers with non-linear activations. The visual encoder and the LLM are usually frozen during training.

In our work, we choose the pretrained LLaVA-Med <cit.> as our model, which is a MLVLM built upon LLaVA <cit.>. The model structure resembles <ref>. It uses pretrained Vicuna <cit.> as the LLM and the pretrained ViT encoder from CLIP <cit.> as the visual encoder. The adapter is simply a trainable projection matrix. Both the visual encoder and LLM weights are frozen during training. LLaVA-Med fine-tunes LLaVA in two steps. First, it fine-tunes LLaVA to generate medical reports from input medical images. Second, it uses GPT-4 to generate various questions from the ground truth reports and fine-tunes the model to perform question answering. 

Most MLVLMs are currently trained by medical VQA such that medical diagnosis can be performed by asking questions related to various pathologies; e.g., “Does this image have lung lesion?”. To reduce model hallucination and improve VQA accuracy, we propose two prompting strategies at the inference stage: (1) providing the model with detailed explanations about the queried pathologies and (2) asking the model to consider the inferences of a weak learner.



 §.§ Prompting With Detailed Explanations


Given imbalanced training data, MLVLMs might not adequately be able to learn the features of the minority pathologies. To compensate for insufficient training, we provide a detailed explanation of the queried pathology as a prompt at the inference stage. The explanation briefly defines the pathology and lists several key findings in medical images that may indicate its existence. An example is shown in <ref>. The model is informed that Pulmonary Edema is defined as the accumulation of fluid in the lungs. Then several chest X-ray findings that may suggest its existence are provided. The model can determine if the given image has Pulmonary Edema by linking the given findings with the image features.

Prompt templates for a number of pathologies are listed in <ref>. 







 §.§ Prompting With Detailed Explanations and Weak Learners


Data re-sampling is a commonly-used strategy to deal with imbalanced datasets that are responsible for the tendency of traditional image classification models to return negative predictions for minority pathologies. Models trained on re-sampled datasets often exhibit improvements in Precision and Recall scores; however, this strategy may not be suitable to MLVLMs for two reasons. First, it is difficult to balance a dataset containing many categories of pathologies. Second, MLVLMs usually demand much larger datasets and fine-tuning is also expensive.

One can nevertheless enable MLVLMs to benefit by leveraging small models trained on re-sampled datasets. Our method resembles multiagent LLM systems, such as <cit.>, where multiple LLMs debate each other and hallucination can be corrected by referring to the generated outputs of other models. Given that traditional image classifiers are smaller, it is feasible to train multiple small classifiers each of which is trained on re-sampled datasets of a particular pathology. Those models can be further fine-tuned to optimize a single aspect, such as fewer false positives (FP) or fewer false negatives (FN). The classifiers are applied to the medical images and return preliminary predictions. These predictions are selectively included in the prompts as references for the MLVLM. Hence, MLVLMs can benefit indirectly from the nuanced understanding that these specialized models can provide. This method is meaningful because clinicians usually must balance the trade-off between overtreatment and undertreatment when making healthcare decisions. For instance, they may prefer models having a low FP rate if the cost of overtreatment is higher than that of undertreatment. 

 An example is shown in <ref>, which queries about the presence of Edema. We first provide the model with the detailed explanation of Edema. Then, we use the weak learner to suppress the FPs. The image is input to an Edema classifier that has been fine-tuned on a balanced dataset for high sensitivity and high true negative (TN) rate. If its prediction is negative, we append after the pathology explanation the prompt “For this image, another agent thinks the probability of Edema is 0.1”. Instead of using the actual predicted probability, the probability value is manually chosen because the decision threshold has been fine-tuned and is no longer 0.5. We do not use a zero probability value because we do not want the model overly to trust the weak learner. Although in this example our goal is only to reduce FPs, our strategy can also be applied to reduce FNs, simply by fine-tuning the classifier for a high true positive (TP) rate and applying the prompt in the case of positive predictions.




§ EMPIRICAL STUDY





 §.§ Datasets




LLaVA-Med is pretrained on the PMC-15M dataset <cit.>, which contains image-text pairs of multiple modalities; e.g., CT, MRI, X-ray, etc. In the first stage, 467,710 image-report pairs were selected for training. In the second stage, 56,708 question-answer pairs were created from the data of the first stage to fine-tune the model. <ref> shows the count of reports in the LLaVA-Med training data (second stage) that mention one of the five test pathologies as positive. Relative to the total amount of data, all five categories are minorities.



To assess the zero-shot performance of the MLVLM, we used the MIMIC-CXR-JPG <cit.> and Chexpert <cit.> chest X-ray test sets. They include 5,159 and 668 images, respectively. 
Neither dataset overlaps with PMC-15M.

MIMIC-CXR-JPG includes images and medical reports covering 13 categories of findings: Atelectasis, Cardiomegaly, Consolidation, Edema, 
Enlarged Cardiomediastinum, Fracture, Lung Lesion, Lung Opacity, Pleural Effusion, Pneumonia, Pneumothorax, Pleural Other, and Support Devices. The raw reports are parsed and rough image-level tags are automatically generated by a rule-based approach <cit.>. Each label contains four values: 1 (positive), 0 (negative), -1 (uncertain), and missing. For simplicity, we treat both uncertain and missing as negative. We also use the MIMIC-CXR-JPG training set, which contains 227,827 chest X-rays with reports, to train the weak learner models.

Chexpert covers the same 13 categories as MIMIC-CXR-JPG. However, it does not include medical reports and has only image-level labels. There is no overlap between MIMIC-CXR-JPG and Chexpert. 

<ref> shows the split of pathology categories (excluding normal) in the MIMIC-CXR-JPG and Chexpert test sets. Clearly, almost all pathology categories are minor classes with much fewer positive than negative occurrences.

For our main testing regimen, we selected the five pathologies in the Chexpert Competition <cit.>, Atelectasis, Cardiomegaly, Consolidation, Edema, and Pleural Effusion.



 §.§ Implementation Details




As was mentioned in <ref>, we use the pretrained LLaVA-Med MLVLM without any further fine-tuning. We convert the classification task into a VQA task by using the prompt template shown in Row 1 of <ref>, which we name Prompt Template 1 (PT1). We first run the pretrained LLaVA-Med with PT1. Next, we incorporate pathology explanations (Row 2 of <ref>), yielding Prompt Template 2 (PT2). Finally, we integrate the predictions of weak learners into the prompts (Row 3 of <ref>), resulting in Prompt Template 3 (PT3).

As will be justified by our experiments, our weak learner is designed to suppress FP predictions. To this end, we use the pretrained ResNet50 <cit.>. For each pathology, the training dataset was sampled such that the ratio of positive and negative cases is 2:1. The model was trained for 10 epochs with a 1e-4 learning rate. The training process was monitored using the AUC score and the one with the highest validation AUC was kept. Then, the decision threshold d was fine-tuned to optimize a weighted sum of Specificity and Negative Predictive Value (NPV); i.e.,

    d = w_1TN/TN+FP+w_2TN/TN+FN,

where weights w_1 and w_2 are preset to 0.2 and 0.8, respectively. The medical images were input to the weak learners to obtain preliminary predictions for each pathology and only the negative predictions were selected to craft the PT3 prompts. 

The responses returned by LLaVA-Med can take various forms, such as “This image has Edema”, “Edema is found”, “The fluid in the lung indicates Edema”, etc. An off-the-shelf Llama-7B <cit.> serves to summarize long responses into Yes/No answers such that accuracies could easily be computed.



 §.§ Results


To demonstrate the efficacy of our prompting strategies, starting from the PT1 baseline, the pathology explanations were provided first (strategy PT2) and then, based on the results, weak learners were introduced to improve performance on specific aspects, resulting in strategy PT3.





  
PT2: Adding Pathology Explanations
<ref> reports Precision, Recall, and F1 scores of the PT1 and PT2 strategies on the MIMIC-CXR-JPG and Chexpert test sets.[The AUC and ROC scores commonly reported in the literature to assess the performances of most medical image classification models on the MIMIC-CXR-JPG and Chexpert datasets are unsuitable in our context because MLVLMs output text rather than probabilities.] On MIMIC-CXR-JPG, after adding pathology explanations, the F1 scores increased for detecting Atelectasis, Cardiomegaly, Edema, and Pleural Effusion, albeit only minimally for Consolidation. On Chexpert, after adding pathology explanations, the F1 scores for detecting Atelectasis, Cardiomegaly, and Edema increased, whereas they did not for Consolidation and Pleural Effusion. The Precision and Recall scores reveal that adding explanations generally leads to a large increase in Recall, but only minimally influences Precision. For minority pathologies such as Consolidation whose F1 score is dominated by low Precision, improving the Recall would not have much effect. Thus, PT2's performance bottleneck is Precision.







  
PT3: Referring to Weak Learners
Going beyond our PT2 strategy, we applied our PT3 strategy to further improve diagnostic accuracy. <ref> provides the TP, FP, and FN prediction counts of LLaVA-Med on the Chexpert test set using the PT2 strategy. Note the large number of FP cases. Hence, we designed our weak learners to suppress FP predictions.
<ref> compares the performance on Chexpert before and after referring to the weak learner. It shows that the F1 prediction accuracy can be substantially increased by introducing weak learner predictions into the prompts. The F1 scores of Cardiomegaly, Edema, and Pleural Effusion increase by 0.115, 0.194 and 0.089, respectively. To further demonstrate the efficacy of our PT3 strategy, <ref> compares the FP predictions of the PT2 and PT3 strategies. The reduction of FP cases is noteworthy, especially on Edema, for which the FP count is reduced by 78.5% (322). 







  
Additional VQA Experiments
<ref> shows the results of applying the PT1, PT2, and PT3 strategies with LLaVA-Med on the MIMIC-CXR-JPG and Chexpert datasets across another five medical findings: Enlarged Cardiomediastinum, Lung Lesion, Lung Opacity, Pneumonia, and Pneumothorax. Providing pathology explanations (PT2) generally yields better results over the PT1 baseline, albeit inconsistently. Introducing weak learner references (PT3) yields only limited increases in Precision, but large decreases in Recall. Generally, it offers insignificant improvement. Enlarged Cardiomediastinum, Lung Lesion, Pneumonia, and Pneumothorax are minor categories and all our experimental settings, including for the weak learner, fail to learn them. Prompting is apparently unhelpful in such situations.







  
SOTA Benchmark
<cit.> report F1 scores for detecting Atelectasis, Cardiomegaly, Consolidation, Edema, and Pleural Effusion on the Chexpert dataset using their deep learning model, as well as for the performance of radiologists. Their work offers a state-of-the-art chest X-ray diagnosis benchmark. <ref> compares the F1 scores of radiologists, the model of <cit.>, and LLaVA-Med. It shows that LLaVA-Med's VQA performance of with the baseline PT1 strategy is unsatisfactory, rendering the model far from being deployable in clinical practice. However, while still underperforming radiologists, our PT3 strategy yields a significant improvement, especially on Atelectasis, Cardiomegaly, and Edema for which the F1 score increases by approximately 17% to 21%. 



  
Application to General Domain LVLMs
Our prompt strategies can also be applied to general domain LVLMs. We studied the performance of LLaVA <cit.> and MiniGPT-v2 <cit.> using POPE metrics <cit.>, which evaluate the hallucination of LVLMs by asking questions about the presence of objects. The POPE scores of LLaVA and MiniGPT-v2 have high Precision and low Recall. Hence, our weak learner strategy can be used to reduce the FN predictions. We selected an off-the-shelf Fast-RCNN <cit.> as the weak learner, fine-tuned the detection threshold of bounding box scores to achieve high Recall, and introduced the positive predictions of the weak learner into the prompts. The results in <ref> show that the Recall scores across three POPE categories increased by around 7% (Precision scores decrease slightly), thus improving the F1 scores.



§ CONCLUSIONS AND DISCUSSION



We have tested the visual question answering abilities of the LLaVA-Med medical large vision-language model when applied to the diagnosis of pathologies. Our results show that the model has unsatisfactory performance when asked questions regarding the presence of complex pathologies. We proposed two prompt engineering strategies to improve the visual question answering accuracy of the model: providing explanations of pathologies and referring to the predictions of weak learners. The first strategy helps the model understand minority pathologies that it does not learn well in the training stage. The second strategy can help improve diagnostic accuracy in specific ways; e.g., by suppressing false positives. This strategy can also be applied to LVLMs in other, non-medical domains.

However, our two strategies are not effective on pathologies with extremely scarce data. For example, providing text explanations for Consolidation, Fracture, Lung Lesion, Pneumonia, and Pneumothorax may not suffice since the visual encoder does not adequately learn meaningful visual features. Moreover, the data may not suffice to adequately train weak learners. A promising direction for future research would be to devise a strategy for handle these rare categories. Retrieval Augmented Generation (RAG) could be a potential solution. In addition to textual explanation of a pathology, for instance, typical example images can be provided to help the model make diagnostic decisions.





§ PATHOLOGY EXPLANATIONS



The explanations of the five pathologies are as follows:



  
Atelectasis: Atelectasis refers to the partial or complete collapse of a lung or a section of lung. The features of atelectasis on an X-ray can vary depending on the cause and extent of the collapse. Some common X-ray features include: 1. The affected area may appear denser or whiter than normal lung tissue due to the collapse, leading to increased opacity on the X-ray. 2. The affected portion of the lung may appear smaller or compressed compared to the surrounding healthy lung tissue. 3. Atelectasis can cause a shift or displacement of nearby structures, such as the trachea or heart, toward the affected area. 4. In obstructive atelectasis (caused by a blockage in the airways), there might be signs of hyperinflation in the unaffected areas of the lung and a visible blockage or narrowing in the affected bronchus. 5. Linear or band-like opacities may be visible, often referred to as plate or band atelectasis, which can occur due to the collapse of small airways. Given the information above, does this image have Atelectasis?



  
Cardiomegaly: Cardiomegaly is enlargement of the heart. The definition is when the transverse diameter of the cardiac silhouette is greater than or equal to 50% of the transverse diameter of the chest (increased cardiothoracic ratio) on a posterior-anterior projection of a chest radiograph or a computed tomography. Given the information above, does this image have Cardiomegaly?



  
Consolidation: Consolidation on an X-ray refers to the filling of the lung's air spaces with fluid inflammatory exudate, or cellular material. Typical X-ray findings suggesting consolidation include: 1. Areas of increased density in the lung tissue, appearing as an opaque or hazy patch on the X-ray. Given the information above, does this image have Consolidation?



  
Edema: Pulmonary edema is the accumulation of fluid in the lungs. Some common X-ray features include: 1. Increased density in the central lung fields resembling the shape of bat wings. 2. Thin, linear opacities at the lung periphery, often indicating interstitial edema. 3. Prominent blood vessel markings due to engorgement from increased pressure in the pulmonary vasculature. Given the information above, does this image have Edema?



  
Pleural Effusion: Pleural effusion is the accumulation of fluid in between the parietal and visceral pleura. Some common X-ray features include: 1. blunting of the costophrenic/cardiophrenic angle. 2. fluid within the horizontal or oblique fissures. 3. meniscus is seen. 4. mediastinal shift occurs away from the effusion.





 Paper Title: Quality Control for Radiology Report Generation Models via Auxiliary Auditing Components.









Error Detection for Report Generation Models




H. Warr et al.

Department of Engineering Science, University of Oxford, Oxford, UK 

{first_name.last_name}@eng.ox.ac.uk
Department of Oncology, University of Oxford, UK
Department of Medical Physics and Clinical Engineering, Oxford University Hospitals NHS FT, Oxford, UK
Department of Computing, Imperial College London, London, UK
School of Computer Science, University of Birmingham, Birmingham, UK








Quality Control for Radiology Report Generation Models via Auxiliary Auditing Components
    Hermione Warr1, 
Yasin Ibrahim1, 
Daniel R. McGowan2,3, 
 
Konstantinos Kamnitsas1,4,5
    August 5, 2024
==========================================================================================



§ ABSTRACT 
Automation of medical image interpretation could alleviate bottlenecks in diagnostic workflows, and has become of particular interest in recent years due to advancements in natural language processing. 
Great strides have been made towards automated radiology report generation via AI, yet ensuring clinical accuracy in generated reports is a significant challenge, hindering deployment of such methods in clinical practice. 
In this work we propose a quality control framework for assessing the reliability of AI-generated radiology reports with respect to semantics of diagnostic importance using modular auxiliary auditing components (ACs).
Evaluating our pipeline on the MIMIC-CXR dataset, our findings show that incorporating ACs in the form of disease-classifiers can enable auditing that identifies more reliable reports, resulting in higher F1 scores compared to unfiltered generated reports. 
Additionally, leveraging the confidence of the AC labels further improves the audit's effectiveness.
Code will be made available at: <https://github.com/hermionewarr/GenX_Report_Audit>








§ INTRODUCTION

 
Radiologists are facing mounting challenges managing the growing volume of medical imaging data under resource constraints, necessitating more efficient and innovative solutions for image analysis and interpretation. Automated (or partly automated) radiology report generation is one such solution.
Since the introduction of transformers <cit.>, which significantly enhanced language generation, numerous studies have investigated radiology report generation from medical images, particularly chest X-rays (CXR) <cit.>.
However, deployment of language models to medical applications faces a variety of challenges, with a key area being the need for factual accuracy. 
Various methods have been developed to address this - particularly useful have been the introduction of clinical metrics to supplement the standard natural language processing (NLP) metrics used to evaluate report generation quality <cit.>.
Previous efforts to improve the quality of the generated reports have incorporated extra information during training, such as bounding boxes for improved localisation, or have aimed to refine diagnostic accuracy by including loss functions that reward clinical accuracy <cit.>.
Additionally, more complex architectures, such as memory-augmented attention, have been developed to facilitate better image understanding <cit.>. 

Besides efforts to improve report generation accuracy, another line of work is error detection in language modeling. Most work in this area has targeted the natural language domain. The two main types of uncertainty estimation in large language models (LLMs) are statistical uncertainty (SU) and in-dialogue uncertainty (IDU) <cit.>. The latter relies on large pretrained models to admit when they don't know an answer <cit.>.
SU relates certainty to the model's output entropy, though in language modelling, it must address the coherence of entire sentences, as well as on a per token basis <cit.>. These methods often depend on extensive labelled datasets in the natural language domain, which do not adequately capture the specialised terminology and context of clinical settings <cit.>. 

Few works have investigated the possibility of error detection in reports written by radiologists <cit.>. 
To our knowledge, no previous work has investigated error detection for automated report generation, which is the focus of our study.


We introduce an auditing framework for identifying potential errors in AI-based radiology report generation, using modular auxiliary auditing components (AC) for quality control. Our study focuses on chest-Xray (CXR) imaging, defining ACs as image-based disease classifiers to extract semantics relevant to diagnosis. We develop a report generation model, GenX, which performs competitively with previous works and serves as basis for testing our auditing framework. Reports are audited using ACs that provide disease classification and confidence levels. Reports with consistent diagnoses across the pipeline meet our criteria, while inconsistencies flag potential errors. Additionally, by leveraging the ACs' classification confidence, we enforce stricter quality control, requiring consistency with the highest confidence predictions. Our evaluation on MIMIC-CXR shows a significantly higher F1 score for reports meeting the audit's criteria. Results demonstrate that consistency between report and auxiliary components is a promising auditing mechanism for automated report generation.



§ METHODOLOGY







 §.§ Report Generation Model - GenX

The proposed framework for auditing radiology report generation is shown in Fig. <ref>. The backbone of the framework is the report generation model. The generalised vision language modeling problem of generating a textual sequence T given an input image I can be framed as estimating the conditional distribution p(T|I), as a product of the conditional probabilities:

    p(T|I) = ∏_l=1^L p(T_l | T_<l, I; θ),

where T is modelled as a sequence of word tokens {T_1,...,T_L}, with T_<l being the set of tokens preceding T_l, where L is the number of tokens in the sequence, and θ the model parameters <cit.>.

An overview of our report generation model, GenX, is shown in Fig. <ref>. The model is trained to maximise the probability of generating the target sequence from the image, p(T|I) from eq <ref>.
Image features I_feat=f_I(I) ∈ℝ^D_I, with dimensionality D_I, are extracted by image encoder f_I.
A linear layer, f_I,emb, projects I_feat to the space of token embeddings that will be input to the language generator, giving I_emb={I_emb,1 ... I_emb,M}=f_I,emb(I_feat) ∈ℝ^D_T × M, a set of M image token embeddings with dimensionality D_T, similarly to LLaVA <cit.>. 
A tokeniser maps text to a set of L token vectors {T_l}_1:L∈ℝ^L × V, where V is the vocabulary size. Each T_l is embedded by a linear layer to T_l,emb=f_T,emb(T_l) ∈ℝ^D_T.
The concatenated embeddings of image and text tokens {I_emb,1 ... I_emb,M, T_emb,1 ... T_emb,L} are passed to a Transformer decoder (Fig. <ref>b) which predicts a probability distribution over V vocabulary tokens, expressing which is more probable in the sequence. The decoder has N_layers with Attention blocks that each has N_heads parallel heads <cit.>.
The model is trained using the common paradigm of diagonal attention masking <cit.>.
A report T_G is generated with auto-regressive inference, starting only with image token embeddings I_emb as input, then predicting the first and each following word token as the one with highest predicted probability.








 §.§ Auditing Generated Reports via Auxiliary Classifiers



While some types of mistakes in AI-generated radiology reports, such as grammatical errors, are merely inconvenient, semantic mistakes related to disease diagnosis can be detrimentalor patient care. 


Consider a semantic concept of interest, denoted c. We focus on the case where c represents a specific disease, with the class label C∈{1,0} indicating the presence or absence of that disease. 
To audit whether a generated report T_G is semantically correct about c, 
we need to extract the value C_T described in text T_G via a mapping function g_T: T_G → c. This can be any text-based classifier, or prompt-based LLM that can answer questions in the form "Does this text suggest evidence or not for disease c?", which have become widely available. Herein we use a pretrained CheXbert text-classifier <cit.> as g_T, obtaining C_T = g_T(T_G) (Fig <ref>b).

We then introduce to the framework an independent model g_I: I →c, which infers value C_I for concept of interest c from image I (Fig. <ref>c). We term such models auxiliary auditing components (AC). In the examined case when c is existence of specific disease, g_I is a classifier predicting C_I=1 if image I shows the disease or C_I=0 otherwise, with model confidence p_AC(c=C_I |I).
If multiple concepts c are of interest (e.g. different diseases), an independent AC model can be used per c, for ease of modular development, or a multi-label AC model.

Auditing is performed by comparing values C_T and C_I about the concept c of interest (e.g. disease) extracted from the report or the image respectively (Fig <ref>d). To consider the contents of the report reliable with respect to c, agreement between the report and AC is required, i.e. C_T = C_I. We further extend this by taking into consideration the confidence p_AC of the AC. We only consider the auditing successful if it satisfies the additional requirement that the value C_T=C_I has been inferred by the AC with confidence p_AC over a pre-determined threshold t. Formally, for a successfully completed auditing we require: 

    (C_I = C_T)  (p_AC(c = C_I | I) ≥ t),

where  is the logical and operator. 

The set of generated reports that pass the audit for a particular concept c is denoted by T_G,pass^(c)={T_G |  Eq. <ref> is  True}. The remainder, T_G,err^(c)={T_G |  Eq. <ref> is  False}, can be deferred to the human user for review as potentially inaccurate for that disease.

The framework is designed based on ACs that are modular, independent models. The design draws inspiration from N-Version programming <cit.> and the principle of redundancy for reliability in software engineering. Assuming independence of components (GenX, g_T, g_I) with label error rates e_C_T and e_C_I, the error rate of the auditing framework 
is e=e_C_T· e_C_I, which is lower than the individual error rates of C_T and C_I, as failure of auditing requires both components to fail simultaneously to incorrectly satisfy Eq. <ref>.
Modular ACs enable independent component training and validation prior to integration into the audit pipeline. 



§ EXPERIMENTS AND RESULTS



 §.§ Data



Experiments are performed using MIMIC-CXR <cit.>, containing chest X-rays (CXR), associated radiology reports, and disease labels for ∼228,000 studies of over 65,000 patients.
As common practice, we consider only the frontal-view images, and only the “findings” section of the reports, which describes evidence of pathologies and support devices. 
The labels 
include 14 classes (12 diseases, 1 support devices, 1 no findings) and
can be positive, uncertain, negative, or not mentioned. 
We report 5 out of 14 classes separately (Atelectasis, Cardiomegaly, Consolidation, Edema, Pleural Effusion) to facilitate comparison with literature <cit.>. We divide data into splits for training (226,261), validating hyper-parameters (1,864), and testing performance (3,595). 
To train and evaluate the report generation, we further exclude studies that do not include a “findings” section, resulting in 155,322 training, 1,231 validation and 2,607 testing samples. Finally, to help address class imbalance and noisy labels, following past works <cit.>,
we consider uncertain disease labels as positive and class no-mention as negative.




 §.§ Evaluating the Report Generation Model


Experimental settings: We here show that our report generation model, GenX, is of representative quality in comparison to previous work to serve as effective backbone for the study of our auditing framework. 
To build GenX we use ResNet-50 as the image encoder f_I, which we pre-train as a multi-label classifier of the 14 classes described above. Its penultimate layer of dimension D_I=512 is then projected to M=10 image token embeddings with D_T=512. These are concatenated with up to L=512 text embeddings, resulting from a GPT-2 tokeniser (V≈50,257) <cit.>, followed by a projection, D_T=512. The Transformer has N_layers=8, N_heads= 8, and dimension D_ff= 2048 of its feed-forward layer. GenX is then trained end-to-end (Fig. <ref>), learning parameters of the randomly initialised projections and Transformer, and fine-tuning the f_I encoder.



Results: 

Table <ref> reports test performance by our report generation model, along with previous works.

NLP metrics BLEU (BL) <cit.>, METEOR (M) <cit.> and  (R-L) <cit.> assess generated language quality. 
More important for this study is the assessment of diagnosis-related semantics in generated reports, using disease-classification metrics such as F1 score, by comparing agreement of class labels, C_T, extracted from generated and reference reports via CheXbert <cit.>.
Our report generator, GenX, achieves NLP metrics similar to  Ratchet <cit.>, which has the most comparable architecture. Importantly, GenX achieves a high F1 score in comparison to Ratchet, R2Gen <cit.>, even RGRG <cit.> that needs additional labelled bounding boxes for training. It approaches F1 score by ℳ^2 Trans <cit.> that trains using a more complex framework and reinforcement, and is surpassed by A_tTERG<cit.> which uses extra labels for training.
We conclude that GenX produces reports of sufficient quality to serve as effective backbone for the following study of our auditing method. Two examples of generated reports are shown in Table <ref>. The generated reports use realistic language, though they can occasionally miss a diagnosis. Thus a framework that can extract and audit report semantics is crucial.





 §.§ Evaluating the Auditing Framework for Generated Reports


Experimental settings: 
We analyse whether including auxiliary auditing components, ACs, enable us to quality control the generated reports and filter out those that are more likely to contain errors.
We evaluate on the 5 diseases most commonly reported in the literature (Sec. <ref>), from the CheXpert competition  <cit.>. For this, we first train 5 independent disease-specific AC image classifiers (AC_1), where each is a ResNet-18. 
We then apply the framework as described in Sec. <ref> to the test-split. For each image, the 5 AC_1 models predict 5 disease labels C_I^(c), one per disease c.
GenX generates a report for each test image and we extract 5 report-based class labels C_T^(c) via Chexbert, for each disease.
For a given scan, per disease c, we compare image and report based labels, C_I^(c) and C_T^(c), to check if they fulfil Eq. <ref>. We perform the check in two settings: First, without considering a confidence threshold for p_AC, satisfying only the first condition in Eq. <ref>; Second, with a confidence threshold t=0.8 where Eq. <ref> is fulfilled if p_AC≥0.8, to investigate whether stricter auditing criteria results in more accurate reports
For each disease, the images that fulfil the above criteria are considered to have successfully passed the audit. For these images, we calculate average F1 score by comparing their report-based labels C_T^(c) with the reference report labels, to determine whether the average score is higher than that of all unfiltered reports generated by GenX.
Finally, we repeat these experiments using a bigger ResNet-50 AC, trained as a multi-label image classifier of all 14 classes (AC_14). This is to evaluate whether it's beneficial to integrate a larger AC that learns from all 14 labels available per scan, or independent, disease-specific ACs. 



Results: Table <ref> shows the results. Class labels C_T extracted from reports generated by GenX achieve an average F1 score of 46.8 over the 5 diseases. Reports that agree with AC_1, fulfilling the auditing without the extra requirement for confidence p_AC≥t, achieve average F1 score 54.3. These benefits are realised over a significant percentage of the reports (72-80% across the diseases), while the rest would be deferred for user inspection and error correction in an actual workflow.
Reports that pass the stricter auditing and 
additionally fulfil the confidence requirement p_AC≥0.8 present even less errors, with average F1 score 58.4. This restricts the number of reports that pass the audit (30-65%), in return for higher reliability.
In comparison to the unfiltered reports of GenX, the reports that fulfil the audit show increased F1-scores for 4 out of 5 studied diseases. The exception is Consolidation, the class where both GenX and AC image classifiers have very low performance. Interestingly, for 3 out of 5 diseases, the reports that successfully pass the audit show F1-scores that are even higher than the auditing AC_1 classifiers.
These results show that measuring consistency between semantics of reports and auditing classifiers is a promising framework.

Finally, we find that a single, multi-label classifier, trained for all 14-classes as AC_14, achieves significantly lower F1 scores for image classification, which also leads to lower effectiveness of the auditing. Although AC_14 is a ResNet-50, larger than ResNet-18s used for the 5 AC_1 classifiers,

it is difficult to develop it at the level of disease-specific classifiers. This demonstrates the benefits of modularity, with independently developed AC for each concept of interest.





§ CONCLUSION

We introduced a framework to quality control AI-generated radiology reports with respect to semantics of interest via auxiliary auditing components. 
Our study focused on the task of report generation from chest-Xrays, detecting potential semantic errors in diagnosis-related report content, using auxiliary disease classifiers and assessing consistency between labels inferred from image and report.
Experiments show that reports fulfilling the auditing criteria exhibit fewer errors, with average F1 score that can even exceed the score of the auditing components. The framework is generic and future work could explore it for auditing semantic concepts beyond disease classification, such as for description of pathology location or volume, using regression models as ACs for such properties.

The limitation of the framework, as with any filtering pipeline for quality-control, is the inherent trade-off between reducing the number of reports that successfully pass the audit and can be used for down-stream workflow, in exchange for their higher reliability. Future work could improve performance in both aspects by integrating more potent models for inferring image-based and report-based labels, thereby ensuring greater consistency and reliability from their comparison.




§ ACKNOWLEDGEMENTS

Hermione Warr and Yasin Ibrahim are supported by the EPSRC Centre for Doctoral Training in Health Data Science (EP/S02428X/1). 








splncs04





 Paper Title: The Impacts of AI Avatar Appearance and Disclosure on User Motivation.







AI and Avatars






B. Visser et al.



Leiden Institute of Advanced Computer Science, Leiden University
 P.O. Box 9512, 2300 RA Leiden, The Netherlands



The Impacts of AI Avatar Appearance

and Disclosure on User Motivation
    Boele Visser10009-0002-7504-6495 Peter van der Putten10000-0002-6507-6896 Amirhossein Zohrehvand10000-0002-7473-6702
    August 5, 2024
========================================================================================================================



§ ABSTRACT 
  This study examines the influence of perceived AI features on user motivation in virtual interactions. AI avatars, being disclosed as being an AI, or embodying specific genders, could be used in user-AI interactions. Leveraging insights from AI and avatar research, we explore how AI disclosure and gender affect user motivation. We conducted a game-based experiment involving over 72,500 participants who solved search problems alone or with an AI companion. Different groups experienced varying AI appearances and disclosures. We measured play intensity. Results revealed that the presence of another avatar led to less intense play compared to solo play. Disclosure of the avatar as AI heightened effort intensity compared to non-disclosed AI companions. Additionally, a masculine AI appearance reduced effort intensity.






§ INTRODUCTION


With the rise of digital worlds <cit.>, humans are increasingly interacting with different forms of Artificial Intelligence (AI) in digital spaces, for example, in the form of the recommended system in social media, chatbots or avatars. Similar to the offline world, the perceived gender of the entity that we are interacting with shapes our behaviour and motivation <cit.>. Past research on motivation in offline settings argues that gender differences in motivation arise from the social and cultural expectations, experiences and roles that are associated with being male or female <cit.>. Gender roles and stereotypes can shape an individual’s beliefs, values and behaviours, including their motivation <cit.>. These predispositional features are almost immutable.

However, contrary to the offline world, the gender of the agent that we are interacting with, i.e. the AI agent, is customizable. The AI gender appearances could be chosen on a scale, allowing us to experiment with immutable characteristics and observe their impact on user behaviour. Building on social exchange theory <cit.> and trust and agency literature <cit.> we argue that users playing together with an avatar exhibit a higher degree of persistence in comparison with playing alone. Further, we argue that users are more motivated to play with an agent if they attribute more agency to or perceive the gender of the agent to be of a more cooperative type, they exhibit higher degrees of persistence. Therefore, we suggest that users play less intensely when they are playing with an AI in comparison to a human. Likewise, we argue that the disclosure that an avatar is actually AI-driven reduces the intensity which with the user plays, just as what happens when AI systems in the physical world are disclosed  <cit.>.

To research this, we performed an experiment in a game. In the game, published on Roblox  (a virtual world), players have to perform a simple task, which can be done together with others. Inside this game, over 72,500 participants played 15 levels of a simple search task. The participants either played alone or together with an AI avatar, from which the gender appearance was randomly chosen. For half of the participants playing together with an AI, it was not disclosed that they were playing with an AI. We measured how intense participants played the experiment by dividing the time before leaving the experiment by the number of levels completed. We used this measurement to compare the difference in motivation between different treatment groups, to see what effect the AI gender appearance has on motivation in this particular context. The findings of this study expand our understanding of social exchange theory in the online world <cit.> and trust in AI <cit.>  by showing how the perceived gender of a non-human agent influences human motivation and trust.



§ BACKGROUND

Motivation is defined as “the degree and type of effort that an individual exhibits in a behavioural situation” <cit.>. There are two types of effort: intrinsic motivation and extrinsic motivation <cit.>. Intrinsic motivation occurs when an individual finds something interesting or enjoyable, while extrinsic motivation occurs when an individual is motivated by the outcome of a task. While intrinsic motivation is more about the process, extrinsic motivation is more about the outcome. Past research shows that different demographics show categorically different motivational patterns or the demographics of others might influence their motivation. According to a literature review <cit.>, motivation drivers vary among age groups.



 §.§ Social interactions with AI

Interactions with artificial intelligence (AI) are becoming increasingly common.  Considering that users can give agency to the AI agent, the characteristics of the AI can influence how people interact <cit.> and trust AI based on their ability to think, plan and act <cit.>. Past research shows that even the features of how the AI agent is described to users can influence the interaction and the user <cit.>. Within the interaction itself, the form factor (i.e. written, voice, physical) of the AI changes how participants conform to the AI <cit.>. In these interactions, the perceived gender of the AI influences the compliance with the AI <cit.>. To sum, past research shows that various factors influence how people interact with an AI. Increasingly, virtual worlds are where many of these interactions with AI could take place.

Virtual spaces are expanding beyond gaming communities to offer courses, stores, and financial services, allowing consumers to discuss their interactions with firms, and for many more aspects of everyday life to take place online <cit.>.  Both intrinsic and extrinsic motivation, measured by factors such as escapism, visual attractiveness, perceived usefulness, and entertainment value, impact users' attitudes towards using virtual worlds <cit.>.

In virtual worlds, users are typically represented by avatars, which are graphical representations of themselves <cit.>. Users can design their avatars to represent their actual appearance or their desired identity in the virtual world <cit.>.
The appearance of an avatar can influence the behaviour and motivation of the user, especially when the user can identify with their avatar <cit.>. The appearance of an avatar can also influence how a user is perceived by others in virtual worlds <cit.>. Particularly, avatar gender can also influence how a user is perceived and behaves in virtual worlds <cit.>.

Users, similar to the offline world, can interact with others, represented by some avatar, in a virtual world. Past research suggests that cooperating with other players motivates players to use a virtual world: When an educational video game app added a feature where users could asynchronously cooperate, the app usage increased 217% <cit.>.
Synchronous cooperation also motivates players: players of video games are motivated to play an exergame again more when they play together (in cooperation) with another player in the same physical space than when they played alone <cit.>. When the AI behaves like a player, or when the player cannot distinguish an AI agent from a human player, there should be no difference between the interaction with a player and with an AI, thus the effects of cooperating with another human could carry over to cooperating with an AI. Therefore, we suggest the following:

H1: Participants who play with an AI avatar (treatment group, regardless of disclosure) have a higher intensity of playing than those who play alone (control group).



 §.§ Relationship between AI avatar gender and motivation

The gender of others who someone interacts with influences motivation <cit.>. Some research claims that men are motivated by competition and they believe they outperform others <cit.>, so men will want to keep ahead of others and are motivated to do so quickly. Other research claims that women behave more prosocially <cit.>, so women might want to support others and continue to keep interacting longer, which will make them play longer.

Since an AI is perceived to be a social actor by humans <cit.>, the characteristics of the AI determine how humans interact with the AI <cit.> and the perceived gender of an AI having an influence on people’s motivation during textual interactions <cit.>, the gender effects may carry on when interacting with an AI. We therefore suggest the following:

H2: Participants playing with an AI avatar that appears more masculine have a higher intensity of playing.



 §.§ Relationship between AI disclosure and motivation

When interacting with an AI, the user could know whether the one they are interacting with is an AI or not (which implies it is a human). This AI disclosure affects how users perceive an AI system <cit.>. Employees who get performance feedback created by an AI get a negative perception of this feedback when they know the feedback is created by an AI, which makes them less motivated by the feedback <cit.>. Additionally, Job candidates are less interested in an offer when their application is reviewed by an AI system and therefore are less motivated <cit.>. Note that the framing that is used when introducing the AI may have an influence on trust and conformance <cit.>). This lets us believe that AI disclosure causes a lower motivation of users, and therefore we suggest the following:

H3: Intensity of playing is negatively associated with participants knowing that they are playing with an AI avatar, in comparison to playing without the AI being disclosed.



§ METHODS

In this section, we will discuss the methodology of the experiment we performed. We preregistered this experiment at the Open Science Foundation.



 §.§ Organizational setting

The setting for this experiment is a game on Roblox. Roblox is an online gaming platform that allows users to create, share and play their own virtual experiences. This platform has over 150 million monthly active users. The game had more than 50.000 active daily users at the time of the experiment. The idea behind the game is simple: players get placed in a level (a room), and their task is to find a hidden object inside that room. Once they find the object they click on it, and they progress to the next level. 

The player base of the game is very diverse. Players come from around the world, with different age groups, languages and devices. This makes it a good fit to research the impact of an avatar’s gender on motivation in a very diverse player base.



 §.§ Participants

Participants are players of the Game X. Players could join the experiment by selecting a special mode called ‘Science Mode’. Only the data from plays during the first two weeks are used, although it is possible to join the ‘Science Mode’ after the experiment’s duration to be able to review the procedure. After participating in the experiment, players receive a cosmetic item and some in-game virtual currency.

Participants could participate in the experiment multiple times, but only their first participation will be taken into account for the analysis.

Due to the privacy regulations of Roblox, it is not possible to review the age and gender of the participant. However, data about age is available on a generic level of the game, where we could see the distribution of players in age groups. It is most likely that the participants follow the same demographics as the game itself. Players from the game come from around the world, which makes the participants very diverse and eliminates the WEIRD society bias from the data <cit.>.



 §.§ Procedure

When participants join the experiment, they get an information screen that they are going to participate in an experiment. One-third of the participants are in the control group and will participate alone. The other two-thirds are in the subject group and will participate in an AI avatar. Within the subject group, one-half was told they were playing with an AI (so one-third of all participants), while the other half (also one-third) was not told they were playing with an AI.

After the explanation, participants select their avatar. Participants were completely free to select the avatar they wanted; the question asked was ‘Select your avatar’. The avatars differ in gender appearance. The avatars were always displayed in the same order, from left to right (See Figure <ref>). Participants play with their selected avatar during the experiment, and participants are asked to rate this avatar on a 5-point scale ranging from ‘very feminine’ to ‘very masculine’.




         







        






After they select the avatar, they play the ‘Science Mode’, which consists of 15 stages. In this mode, players cannot see other participants, except the AI avatar controlled by the computer when participants are in the subject group. This AI avatar is also called a non-player character (NPC).

The appearance of the NPC is one of the 5 different avatars shown during the self-selection process. The appearance of the NPC is selected randomly out of the five preset avatars, regardless of the avatar chosen by the player. The NPC wanders around the room to preset positions and pretends to be searching for the button. When the participant finds the button, the NPC teleports to the next level within a few seconds. Therefore, the NPC is always playing at the same level as the participant, which gives the participant the feeling that the NPC is playing with them.

The difficulty and level design is the same for all participants, as is the behaviour of the NPC. The independent variables that are changed are playing alone vs. playing with an NPC, the disclosure of the NPC being an AI and the (gender) appearance of the NPC.

Before joining the experiment, participants are informed about its details. The experiment is non-intrusive and participation is voluntary, with the option to leave at any time without giving a reason. Participants are informed about the treatment type directly after participation. The AI is programmed to never harm participants, but rather to encourage and help them to complete the task while remaining friendly at all times. No personal identifier data was stored about the user (data collection has been fully anonymized).



 §.§ Measures

We proxy motivation based on how intensely players engage with the experiment and how likely it is that they complete all the stages. We measure the seconds spent per level by calculating the time before leaving the experiment divided by the number of levels completed. A lower value of seconds spent per level therefore implies that the participant spends less time on a level on average, and therefore engages more intensely with the experiment. The motivation of the participants is measured by using a performance-based behavioural measure. Performance-based behavioural measures indicate both a higher outcome-focused motivation (extrinsic motivation) and process-focused motivation (intrinsic motivation) <cit.>.

In addition, we use a dummy variable to indicate if a participant has completed all 15 stages. The time before ending the participation is also measured - participation can end by either completing the 15 stages or leaving earlier. The speed at which players complete the experiment therefore indicates if the avatar's appearance influences either the intrinsic process-focused motivation or extrinsic outcome-focused motivation.

To get a more reliable measure for the gender of the avatars, we asked each participant to rate their avatar on a 5-point scale, with 1 being ‘very feminine’  and 5 being ‘very masculine’. This gives an average score for each avatar between 1 and 5. To simplify the analysis, we calculated the average rating score of the avatar. Given that the average of our 1-5 scale is 3, we mapped the avatar to either feminine when the average score is below 3 or masculine when the average score is above 3. If a score is exactly 3, the avatar is treated as having no gender and is excluded from the analysis about gender.



 §.§ Analytical strategy

To test the hypotheses, we used Ordinary Least Squares (OLS) from the statsmodels package <cit.> in Python. For each variable, the dependent variable is the seconds spent per level, modelled using OLS. The independent variable is AI avatar presence, AI avatar masculinity and AI disclosure for respectively H1, H2 and H3.




§ RESULTS

During the 2-week time window, 126,013 participants started the experiment. From this group, 8488 participants (6.74%) completed all levels of the experiment.

In Figure <ref> the progress of participants over the experiment is displayed. In Figure <ref>, the time spent in the experiment before either leaving or completing the experiment is displayed. The mean time spent in the experiment is 249.4 (σ = 287.6), while the median time spent is 142 seconds. The mean time including the standard deviation is plotted in the error bar displayed in Figure <ref>.







53,099 (42.14%) participants left the experiment during the first level. This implies that participants did not finish a level. Since they did not finish a level, participants never completed a task. This data is left out of our analysis because there is no indication that participants did understand the task well enough or that their motivation was affected by the treatment they were subjected to.

Based on previous observations within the game, it is not possible to complete a stage within 5 seconds when you did not play the level before (based on loading times, time to search the button and the time to click on it). If a participant completed a stage within this 5-second timeframe, it is likely that the participant used some kind of cheating or did know the location of the button beforehand. Therefore, we left out the data of players from which the seconds spent per level (the average time spent per level) is below 5.0 seconds. This was the case for 11 (0.01%) participants. This makes the total sample for analysis consist of 72,903 participants, of which 8488 (11.64%) completed the experiment.

In Table <ref>, an analysis of the difference between participants who did and did not complete the experiment could be found. 11.6% of the participants completed all levels of the experiment, with the average participant completing 3.67 levels (when excluding participants who did not complete a single level).




As mentioned in the methodology, there is no information available about the age of individual participants. However, if we assume the participants in the experiment follow the same demographics as the game, nearly 50 per cent of the participants are 13 or younger, with an additional 17 per cent being aged between 13 and 17. There are no other game-level demographics available.

Out of the participants, 23,079 (31.66%) participants were in the control group, 24,894 (34.15%) participants played together with an AI and did not know they were playing with an AI and 24.930 (34.20%) participants played together with an AI and knew they playing with an AI. The distribution of both the self-chosen avatar and the assigned AI avatar of the groups who played with an AI are listed in Table <ref>. Over 50% of the participants chose the first avatar as their avatar.



To assess the gender of the avatars, we took the mean of all rankings given for that particular avatar. If the average avatar score was under 3, it was categorized feminine, while it was categorized masculine when the average avatar score was over 3. The categorization of all avatar types is listed in Table <ref>.





 §.§ Hypothesis Tests

We conducted an ordinary least squares regression analysis to estimate the relationship between the presence of an AI avatar (treatment group) and the seconds spent per level (H1: Participants who play with an AI avatar (treatment group) have a higher intensity of playing than those who play alone (control group)). In Figure <ref>, the seconds spent per level grouped by the treatment type is displayed.



We found a significant effect (R2 = 0.01, F(1, 72901) = 734.4, p < 0.001, 95% CI [12.033, 13.909]) for intensity of play. Participants’ seconds spent per level increased by 13.0 seconds per level when playing together. That is equivalent to 0.22 standard deviations. This means that participants who played together with another (AI) avatar played less intensely. This is the inverse of what we expected, and therefore we reject H2b.

To test the influence of the effect of the gender appearance of the AI avatar, we conducted an ordinary least squares regression analysis (H2: Participants playing with an AI avatar that appears more masculine have a higher intensity of playing).

We found a significant effect for seconds spent per level (R2 = 0.000, F(1, 72901) = 10.03, p = 0.002, 95% CI [0.841, 3.570]). Participants’ seconds spent per level increased by 2.2 seconds per level when playing together with a more masculine AI avatar. That is equivalent to 0.04 standard deviations. This means that participants who played together with an AI avatar that appeared more masculine played less intensely. This is the inverse of what we expected, and therefore we reject H2.

To test the hypothesis about the effect of participants knowing that they are playing with an AI avatar on their intensity and duration of play, we conducted an ordinary least squares regression analysis (H3: Intensity of play is negatively associated with participants knowing that they are playing with an AI avatar). In Figure <ref>, the seconds spent per level grouped by the treatment type is displayed.



We found a significant effect (R2 = 0.000, F(1, 49822) = 6.957, p = 0.008, 95% CI [-2.569, -0.379]) for seconds spent per level. Participants’ seconds spent per level decreased by 1.5 seconds per level when the other avatar was disclosed as AI. That is equivalent to 0.02 standard deviations. This means that participants who played together with an AI avatar that was disclosed as such, play more intensely. This is opposing to what we expected.

In addition, we did not find a significant effect (R2 = 0.000, F(1, 48007) = 0.803, p = 0.370, 95% CI [-0.003, 0.008]) that playing together with an AI avatar disclosed as such increases the value of seconds spent per level, compared to playing alone.




§ DISCUSSION

This study researched various independent variables that could be used by an avatar controlled by Artificial Intelligence (AI) to improve users’ motivation. Our findings show that in our experiment and game, the masculinity of the users’ avatar makes participants play more intensely. Furthermore, playing together with another avatar controlled by AI significantly affects users’ motivation. Besides that, the masculinity of the AI’s avatar significantly increases the intensity of play. When users learn that the other avatar is controlled by AI, users had a higher probability of not completing the experiment and played more intensely. Finally, the difference between the user’s gender appearance and AI’s avatar gender appearance moderates the effect of the AI’s avatar gender appearance and makes users play more intensely.



 §.§ Contribution to existing literature

Previous studies have shown that cooperating with others can increase motivation <cit.>. Our research shows that playing together with an avatar controlled by an AI makes players play more intense. Our findings suggest that the existing theory, which states that playing with someone else increases motivation, can be extended to include playing with an avatar controlled by AI. Future research in this field could focus on replicating our findings in other games and bringing these effects to non-gaming virtual worlds, to see if our findings are also applicable to other settings, such as virtual workspaces.



 §.§ Practical implications

Companies that create virtual worlds, such as Meta, could use this research when designing avatars in the virtual world. If the sole goal would be to maximize player intensity, it would be best to use a masculine avatar for the AI and to tell users that they are interacting with an AI, in order to motivate users in their virtual world. However, the results of this study also show the need for diversity in virtual worlds. We have seen that the similarity between the participants’ avatar gender appearance and the AI avatar gender appearance moderates the effects of the AI avatar gender appearance on motivation. Therefore, preset genders of both the own avatar gender and the AI avatar gender can limit the effect of the AI avatar on motivation. Diversity in both avatars, such as being able to choose the avatars from a diverse list or being able to freely create the avatars, could help the user in motivation within the virtual world. Beyond gender, ensure that the avatars communicate attributes players are looking for, especially if you want to promote gender diversity in AI collaborators, whilst still stimulating motivation. This requires more research to investigate what other attributes might affect the motivation of players, as well as a broader coverage of the gender continuum, for instance by introducing more androgynous characters.



 §.§ Limitations

Our AI avatar movement was simple. The avatar moved to preset points and did not interact with participants in another way, such as talking or following them. This could have limited the degree to which participants bestow agency to the AI. Furthermore, we don't theorize or study how the details of the interaction could change the user's perception regarding the agency of the AI  <cit.>. To address this limitation, future research could use a more naturally behaving AI, introduce different levels of intelligence, and observe the impact on the user's behaviour.

Besides the quality of the AI movement, the task and its setting might have influenced the outcomes of the study. The task was a simple search task, which could be seen as a more mathematical task, and the levels were themed around science. This particular task and setting could have led to a higher base motivation of some groups of participants, which could have worked through the measurements. Future research could be changing the task and setting, while keeping the rest of the methodology the same, to investigate if the task and setting influences the effects we found.

Finally, the selection of avatars was limited to only five avatars, which restricted the gender expressions in the avatars and diversity in skin colour. A participant raised this concern, which did not feel represented by the available avatars due to skin color. We decided to not include diverse avatars because participants should choose based on gender, not on other avatar features. However, this non-diverse set of avatars could have led to dissatisfaction with the experiment and decreased motivation regardless of the treatment type for some participants. To address this limitation, future research could include a wider range of avatars to choose from or allow players to use their avatar.



§ CONCLUSION

This study aimed to describe how AI avatars inside virtual worlds could be used to increase the motivation of a user. We conducted a large-scale field study inside a game where we were able to control variables in the virtual world. We found that playing together with an AI avatar makes users play less intensely. When this AI avatar appears more masculine, users play less intensely. The AI avatar being disclosed as an AI make users play more intensely. Further research could include replicating these results in other games & non-gaming virtual worlds and researching what the causes are for the effects we found.





  §.§.§ 

Boele Visser is co-owner of the game in which the research took place. The other authors have no competing interests.









splncs04









 Paper Title: Effective Black Box Testing of Sentiment Analysis Classification Networks.
















	Effective Black Box Testing of Sentiment Analysis Classification Networks
    Parsa Karbasizadeh1,
		Fathiyeh Faghih2, and Pouria Golshanrad3 

	College of Engineering, Department of Electrical and Computer Engineering 
 University of Tehran, Tehran, Iran 

		Email: 1parsa.karbasi@ut.ac.ir,
		2f.faghih@ut.ac.ir,
		3pouria.golshanrad@ut.ac.ir
    August 5, 2024
=============================================================================================================================================================================================================================================================================








§ ABSTRACT 
Transformer-based neural networks have demonstrated remarkable performance in natural language processing tasks such as sentiment analysis. Nevertheless, the issue of ensuring the dependability of these complicated architectures through comprehensive testing is still open. This paper presents a collection of coverage criteria specifically designed to assess test suites created for transformer-based sentiment analysis networks. Our approach utilizes input space partitioning, a black-box method, by considering emotionally relevant linguistic features such as verbs, adjectives, adverbs, and nouns. In order to effectively produce test cases that encompass a wide range of emotional elements, we utilize the k-projection coverage metric. This metric minimizes the complexity of the problem by examining subsets of k features at the same time, hence reducing dimensionality. Large language models are employed to generate sentences that display specific combinations of emotional features. The findings from experiments obtained from a sentiment analysis dataset illustrate that our criteria and generated tests have led to an average increase of 16% in test coverage. In addition, there is a corresponding average decrease of 6.5% in model accuracy, showing the ability to identify vulnerabilities. Our work provides a foundation for improving the dependability of transformer-based sentiment analysis systems through comprehensive test evaluation. 


Journal of IEEE Software
Shell et al.: Testing Smart Contracts Gets Smarter

	

		
AI Dependability, Test,Transformers, Coverage Criteria,NLP
	








§ INTRODUCTION


The impact of neural networks on various fields, including image recognition, natural language processing, and self-driving cars, has been significant <cit.>. In the context of Natural Language Processing (NLP), neural networks have gained notable recognition due to their ability to enhance accuracy <cit.>.
Sentiment analysis, also known as opinion mining or emotion AI, is a natural language processing (NLP) technique used to determine the emotional tone or subjective opinion expressed in text.  This procedure utilizes NLP methodologies to analyze and interpret human language, enabling diverse applications across various domains. For instance, in marketing, sentiment analysis can be leveraged to evaluate customer opinions about products and services, providing valuable insights for brand reputation management. 
Various tools, including NLTK <cit.>, are employed to interpret customer sentiment and capture embedded emotions. 

Traditional emotion recognition methods often struggle with the inherent complexities of human language. These complexities include word ambiguity, where a word's emotional connotation can shift depending on context (e.g., “scared” can be positive in an exciting situation or negative in a threatening one). In addition, sarcasm and negation pose challenges in identifying true sentiment.  Furthermore, the process of understanding emotional meaning must include evaluating words that are not directly close to each other in a sentence, which is a step away from traditional methods.

Transformer-based neural networks effectively handle the ambiguity in emotion sentiment analysis through their attention mechanisms and large parameters <cit.>. Parallel text processing allows them to analyze text data efficiently and capture the hidden emotional factors that can exist across longer distances within a sentence. Through pre-training on extensive datasets, transformers are capable of learning complex language patterns and the subtle details of human emotional expression <cit.>.

To guarantee the efficacy of these complex models, comprehensive testing protocols are essential. A significant challenge lies in evaluating the sufficiency of test suites specifically designed for Transformer-based emotional sentiment analysis networks. Given their complex architecture and the large number of parameters, conventional methods like neuron coverage <cit.> seem to be impractical. To address this limitation, we introduce in this work a novel set of coverage criteria designed to evaluate test suites for the assessment of Transformer-based emotion sentiment analysis networks. We employ the principle of input space partitioning, segmenting textual data into emotionally-relevant categories. This strategy enables us to concentrate on particular areas of the input space and generate targeted test cases.

To achieve high coverage with the least number of test cases, we employ the k-projection coverage method <cit.>. This technique utilizes dimensionality reduction by examining smaller feature sets at the same time. Concentrating on these smaller subsets allows for an efficient evaluation of the network's performance across diverse emotional categories within the input space. This strategy ensures comprehensive coverage with fewer tests, thereby eliminating the necessity for extensive test suites.
Additionally, we introduce a set of specialized features specifically designed for emotional sentiment analysis. These features capture the emotional details within text data, thereby aiding the network in accurately identifying the intended emotions.


Upon applying our emotion-specific k-projection coverage method to a dataset designed for emotional sentiment analysis, we observed a 16% average increase in coverage of the generated sentences. This enhanced coverage led to 6.5% decrease in the accuracy of the System-Under-Test (SUT) on the new test suites, demonstrating the effectiveness of our approach in uncovering additional faults.



Related Work:This paper explores concepts related to test generation, black-box testing, and model coverage in DNN models. It summarizes the current state of coverage and performance measures for DNN models and discusses previous research on utilizing diversity in black-box DNN testing to evaluate DNNs and conventional software systems <cit.>.


Transformers are preferred over traditional DNNs due to their superior accuracy, particularly in AI domains like sentiment analysis <cit.>. Evaluating the adequacy of test suites for emotional sentiment analysis networks poses a significant challenge. In software engineering, test suite effectiveness is assessed through black-box and white-box testing <cit.>. Applying white-box criteria to transformer-based NLP systems is limited by their complexity and internal access requirements. Alternative testing methods such as black-box strategies or input diversity metrics may be necessary to ensure their reliability <cit.>.

Input space partitioning is an effective approach for black-box testing, particularly in text data, where emotionally relevant features can be used to partition the input domain into smaller subsets for precise test case design <cit.>. 

k-projection coverage, employing dimensionality reduction, shows promise in image recognition for evaluating self-driving car perception systems using CNN-based neural networks and GAN-based models <cit.>. However, its potential in text-based emotional sentiment analysis remains unexplored.

The contributions of our study are as follows: 

    
  * We introduce novel coverage criteria based on k-projection for black-box testing of sentiment analysis systems. This metric serves a dual purpose: evaluating the effectiveness of existing test suites and guiding the generation of new test cases to augment them.

  * Our experimental results demonstrate that enriching test suites using this proposed coverage criterion leads to a significant reduction in model accuracy. This finding has two key implications:

    
  * The coverage criterion is a valid metric for assessing test suites in this context, as higher coverage scores correlate with a greater ability to uncover faults and decrease model accuracy.

  * The coverage criterion can be effectively utilized to generate effective test cases that reveal failures and thus improve the overall quality of test suites.




The remainder of this paper is organized as follows. We explain the proposed methodology in Section <ref>, followed by a presentation of the experimental results in Section <ref>.  Section <ref> concludes the paper by summarizing our findings and future work.




§ METHODOLOGY


The performance of a neural network is typically evaluated by calculating its accuracy on a test suite. However, accuracy alone is not a sufficient measure of model quality. The effectiveness of testing and the reliability of the results heavily depend on the adequacy of the test suite. Coverage criteria are metrics used to assess how effectively a test suite exercises different aspects of a system under test. In traditional software testing, common criteria include statement, branch, and path coverage <cit.>. However, these traditional criteria cannot be directly applied to learning models due to their fundamentally different architecture <cit.>. To overcome this challenge, we propose a set of black-box metrics for evaluating test suites specifically designed for sentiment analysis tasks.

Our approach originates from Input Space Partitioning (ISP), a black-box software testing technique. ISP involves dividing a system's input domain into distinct subsets or partitions, where each partition represents inputs expected to result in similar system behavior. Test cases are then strategically selected from each partition to ensure comprehensive coverage of the input space.



 §.§ Input Space Partitioning



The primary objective of this study is to assess the effectiveness of utilizing the input space partitioning technique to evaluate test suites for emotional sentiment analysis in Transformer-based neural networks. This technique utilizes the features of the input data to partition the input space into distinct regions, facilitating organization and understanding of the data. <cit.>

The architecture of the neural network model and the nature of the input data require the identification and selection of appropriate features for input space partitioning. Effective evaluation of a test suite requires developing a feature space that covers a wide range of scenarios and real-world instances. 
An effective feature space can be established by adherence to the guidelines proposed in Cheng et al.'s study <cit.>, which outlines specific autonomous driving scenarios that the test cases should encompass and which will be defined subsequently. 



To establish a clear foundation for the framework discussed in this study, we introduce the following notations. These notations are essential to understanding the subsequent mathematical definitions and analyses.



  * Input Domain:
The input domain includes all textual inputs the neural network can receive. All text variations the network must process and respond to are included.
    
    D = {d_1, d_2, …, d_n}

where d_i represents an individual input text.

  * Partition:
In order to understand the existing components and assess the test suite, it is crucial to establish specific partitions and map the input domain within those defined partitions. In this study, partitions are defined as follows:

    P = {P_1, P_2, …, P_k}

    where P_i ⊆ D and P_i ∩ P_j = ∅ for i ≠ j, and ⋃_i=1^k P_i = D.
    

  * Features:
Through the analysis of the input domain and the consideration of the functionality of the system under test (SUT), we can determine the important features that will be used to establish partitions.

    F = {F_1, F_2, …, F_m}

    

  * Test Suite: Representative tests should be selected from each partition to form a test suite.
    
    T = {t_1, t_2, …, t_l}

    where t_i ∈ P_j for some P_j ∈ P.





 §.§ Emotional Features


Input space partitioning relies on specific features to establish partitions, crucial for evaluating test suite effectiveness in neural networks. The upcoming section will outline the specific features employed in this study.

The meaning and sentiment of a sentence can be identified through crucial components such as verbs, adverbs, adjectives, and other linguistic elements. Therefore, features are determined according to these important elements of a sentence.

Verbs strongly influence a sentence's sentiment. For instance, in "She laughed when I saw her," the verb "laughed" implies joy, highlighting its importance in sentiment analysis.

Besides the verb, the adverb in a sentence is another influential feature affecting its sentiment. For example, in "He bites his nails nervously," the adverb "nervously" alone can convey fear or anxiety, making its presence significant for sentence categorization.

Adjectives shape a sentence's emotional tone by describing specific feelings. For example, "She received a joyful gift" conveys happiness, whereas "She received a disappointing gift" indicates sadness.

Nouns contribute significantly to a sentence's emotion. In "The old man sat alone, his eyes filled with tears, mourning the loss of his dearest friend," nouns like "tears" and "loss" contribute to its "sad" emotional label.

In this research, the emotional attributes of nouns were integrated as an additional feature alongside verbs, adjectives, and adverbs.
The sets of features presented in this paper are denoted as:



    F = {S_V, S_ADJ, S_ADV, S_N}


where:

    
  * S_V represents sentiment scores for verbs, categorized into emotional labels: 
    S_V : V →{joy, anger, sadness, fear, surprise}

    
  * S_ADJ represents sentiment scores for adjectives,  categorized into emotional labels: 
    S_ADJ : ADJ →{joy, anger, sadness, fear, surprise}

    
  * S_ADV represents sentiment scores for adverbs,  categorized into emotional labels: 
    S_ADV : ADV →{joy, anger, sadness, fear, surprise}

    
  * S_N represents sentiment scores for nouns, categorized into emotional labels, including a neutral category: 
    S_N : N →{joy, anger, sadness, fear, surprise, neutral}



When dealing with multiple features and their numerous potential values, the number of possible combinations can grow exponentially. This can make generating comprehensive input data, encompassing all features at once, extremely complex. k-projection coverage is a technique used in input space partitioning to overcome this challenge. It aims to systematically explore the input space by focusing on a subset of `k' parameters at a time, while keeping the remaining parameters fixed.

To illustrate this method, consider an example of four features {F_1, F_2, F_3, F_4}, each taking two values {0, 1}. resulting in a total of 2^4 = 16 possible combinations.
Applying k-projection coverage with different values of k results in the following:

    
  * k = 1 (Single Feature Projection): We focus on one feature at a time, keeping the others fixed.

    
  * Number of projections: 4 (one for each feature)
    
  * Number of cases per projection: 2 (the two possible values of the feature)


  * k = 2 (Pairwise Projection): We focus on pairs of features, keeping the other two fixed.

    
  * Number of projections: 6 (combinations of 4 features taken 2 at a time)

  * Number of cases per projection: 4 (2 values for each of the two features)


  * k = 3 (Three-way Projection): We focus on groups of three features, keeping the remaining one fixed.

    
  * Number of projections: 4 (combinations of 4 features taken 3 at a time)

  * Number of cases per projection: 8 (2 values for each of the three features)


  * k = 4 (All Features Projection): We consider all features simultaneously.

    
  * Number of projections: 1

  * Number of cases per projection: 16 (all possible combinations)




 Expanding on the previous example, this approach systematically selects sets of k features and strives to achieve comprehensive coverage within a newly formed reduced dimension. This technique of dimensionality reduction enables efficient test generation, as it requires fewer test cases to achieve complete coverage of the input space.
 
 Generally, to calculate the coverage rate for k = n, where n is the number of problem features, the following formula is employed:


    COV = C(D)/α^n

In the above equation, C(D) represents the coverage rate of the test data over possible states, where α is the range of each feature and n is the problem's dimensionality. In our context, α=6 and n=4, requiring 1296 unique test data points for full coverage. This approach, however, is computationally demanding and requires substantial test data.


 Alternatively, when considering the effect of k, the following formula is used:


    COV = C(D)/nkα^k


 Considering the inverse correlation of the denominator, it is evident that the denominator, which is affected by variables such as k, α, and n, greatly decreases the number of tests needed to achieve complete coverage. 

 Incorporating all four features in sentences can be challenging and costly, reducing sentence clarity. Thus, this approach is ineffective. In order to tackle this issue, the dimensions of the feature coverage space are reduced, with the variable k being set to values of 2 and 3 in this study.The values for the variable k are compared in Section <ref> to evaluate their individual effects.


After measuring the coverage achieved by the test suite to assess the effectiveness of the k-projection metric, we attempt to increase coverage by generating test cases for uncovered scenarios. Ultimately, we explore whether improving coverage aids in the detection of hidden flaws in existing sentiment analysis networks.



 §.§ Evaluation of the Coverage Criteria

Our approach to evaluating the proposed coverage criterion involves examining whether increasing a test suite's coverage score uncovers more failures in the sentiment analysis model. We introduce a method for augmenting test suites with additional test cases specifically designed to improve their coverage score. This evaluation method has two key advantages:

    
  * Validation of Coverage Criteria: Observing a decrease in model accuracy when using the augmented test suite confirms the effectiveness of the coverage criteria for evaluating sentiment analysis test suites. A higher coverage score indicates a greater ability to identify faults and lower model accuracy, thus validating the criteria.
    
  * Practical Test Case Generation: The proposed method offers practitioners a valuable tool for generating effective test cases in sentiment analysis tasks. By focusing on increasing coverage, practitioners can systematically identify and address potential weaknesses in their models.


To uncover unexplored regions within the test suite based on our proposed coverage criteria, we propose a strategy for generating novel sentences. Manual sentence creation, while a viable method for producing desired test cases, is both time-intensive and unsuitable for generating large quantities.

Our idea in this study is to use Large Language Models (LLMs) <cit.> to generate the required sentences. LLMs require training using huge quantities of textual data, which allows them to imitate human-like text production. Examples of such LLMs include GPT-3 <cit.> and LaMDA <cit.>. In this study, we employed Claude 3 Opus <cit.>, an LLM produced by Anthropic, which is recognized for its exceptional sentence generation capabilities.

When utilizing LLMs for test generation, explicit prompts play a crucial role in guiding the model to produce specific types of test data that align with the research objectives. By evaluating the datasets using defined features in <ref>, it is apparent that this dataset does not contain inputs with mixed emotions. For instance, after analyzing the test suites, we identified that the test suite lacks sentences where a verb is labeled with sadness and an adverb is labeled with joy. We generate such a test case using the following formulaic approach:

Let S represent the set of all sentences in the test suite, and define subsets V and A as follow:

    V = {s ∈ S |verb(s) = sadness}


    A = {s ∈ S |adverb(s) = joy}

The required test case would be s' ∈ V ∩ A. To generate the missing test case s', we utilize:

    s' = LLM("Generate a sentence with a verb labeled as sadness


    and an adverb labeled as joy")


This approach ensures that the generated sentence s' fills the identified gap in the test suite, thereby increasing the coverage of the test suite according to our proposed criteria. By adding s' to S, we obtain an augmented test suite S':

    S' = S ∪{s'}


The augmented test suite S' now includes sentences that exhibit the desired feature combinations, leading to a more comprehensive evaluation of the model's capabilities. Additionally, as stated in the evaluation report, these types of situations might occur in user experiences and are viewed as real-life scenarios <cit.>. Therefore, it is crucial to include these generated tests in the test set that showcases these real-life, uncovered scenarios.


For the purpose of generating test cases, we selected the Claude language model <cit.> due to its capacity to generate varied and innovative test sets that are in line with our research goals. This model's attributes enable the generation of test cases that meet research requirements and highlight the robustness and reliability of sentiment analysis.

The generated sentences lack an emotion label, making them unsuitable for use as a test. In order to assign labels to the generated tests, we utilized six of the most accurate sentiment analysis models that were identified in the research literature <cit.>. with their accuracy mentioned in table <ref>. The input sentences were fed into these models for labeling, and the resulting labels were multiplied by their accuracy on the dataset (coefficient). The sentiment with the highest score was assigned as the label for each generated sentence, a method known as “Differential Testing” <cit.>. The system under test was then evaluated using a combination of original and newly generated tests to assess its accuracy on the updated test set.




§ EXPERIMENTAL RESULTS AND DISCUSSION








The dataset utilized for assessing the results is denoted as CARER <cit.>. This dataset contains six primary emotions expressed through text, enabling a thorough assessment of the neural network's ability to accurately classify emotions.
The dataset is split into training and testing sets. The training set comprises 16,000 tweets, each associated with six output labels. Similarly, the test set consists of 2,000 tweets, each labeled with six output categories. A total of 18,000 tweets are chosen as the test set and assessed using the proposed coverage evaluation method. For this study, the dataset is divided into smaller subsets, each consisting of 200 tweets, in order to analyze the coverage capability of each subset. Additional tests will be set based on the uncovered features.

Our model is compared to the existing test dataset since, based on our findings, no other model conducts test generation for neural networks in text categorization in the same way. This study applied the Claude language model <cit.> for test generation and 6 sentiment analysis models for differential testing <cit.>. A distinct arrangement of the DISTILBERT model <cit.> was employed for word-level analysis.The process of learning and loading was executed on Google Colab, utilizing a central processing unit (CPU) with a capacity of 12.7 gigabytes of random access memory (RAM).

We tried different values of k in our implementation to evaluate the effect of k in our approach.
The results presented in Fig. <ref> illustrate the differences in coverage among various implementations of k. When k=2, the coverage increases on average by 14% (Fig. <ref>). However, when k=3, the average increase in coverage is 18% (Fig. <ref>).
The initial coverage for k=3 is 32%, while the initial coverage for k=2 is 62%. This indicates that the test suite lacks sufficient coverage for complicated sentences that involve three combinations of features. Although full coverage was not attained, the sentences generated noticeably raised the total test coverage in comparison to the original data. 

Figure <ref> demonstrates a decline in accuracy for both test generation implementations, suggesting that the generated tests expose previously unexplored areas and have the potential to improve the model's functionality.
When k=2 (Fig. <ref>), the accuracy experiences an average loss of 5%, whereas for k=3 (Fig. <ref>), there is an average decrease of 8%. These findings indicate that the tests created for k=3 demonstrate a higher number of defects in the model's decision-making capabilities.

The average accuracy after test generation for k=3 models is 85%, whereas for k=2 models it is 88%, which is lower than the model's accuracy on the base test suite, which is 94%. This indicates a decrease in the model's precision when tested on these more challenging data points. This drop results from intentionally pushing the model to its limits in order to uncover areas for enhancement.
Based on the experimental results for  k = 3 the augmented test suite exhibits a greater decrease in accuracy compared to  k = 2. This can be attributed primarily to the production of more intricate sentence structures. Nevertheless, when k = 3, there is a greater time complexity and increased costs involved in generating these sentences. The sentences produced for k = 3 exhibit a significant level of complexity and deviate considerably from real-life situations, thereby making it challenging to evaluate the model's performance in practical settings.




§ CONCLUSION AND FUTURE WORKS

This study introduced novel coverage criteria for evaluating test suites of transformer-based sentiment analysis neural networks. By using input space partitioning focused on emotionally relevant linguistic features and the k-projection coverage metric, we efficiently developed test cases and analyzed key feature subsets.
Our methodology demonstrated a 16% average increase in test coverage for all methods and a 6.5% decrease in model accuracy on augmented tests. 
The proposed criteria balance bug detection capability and computational efficiency, surpassing existing techniques. By emphasizing emotional dimensions through input space partitioning, our approach ensures thorough evaluation across diverse real-world scenarios.

Future work could extend our coverage criteria to other transformer-based NLP tasks like text summarization or machine translation. Expanding linguistic elements beyond verbs, adjectives, adverbs, and nouns could improve the detection of subtle emotional nuances, including negations and emojis. Finally, investigating the scalability and efficiency of our methodology on larger datasets and wider applications will be essential for real-world, industry-scale applications.

IEEEtran
 


 Paper Title: Securing the Diagnosis of Medical Imaging: An In-depth Analysis of AI-Resistant Attacks.



Article Title]Securing the Diagnosis of Medical Imaging: An In-depth
Analysis of AI-Resistant Attacks













1]Angona Biswasangonabiswas28@gmail.com

[2]MD Abdullah Al Nasimnasim.abdullah@ieee.org
These authors contributed equally to this work.

3]Kishor Datta Guptakgupta@cau.edu
These authors contributed equally to this work.

4]Roy George george@cau.edu
These authors contributed equally to this work.

5]Abdur Rashid A.Rashid.653@westcliff.edu
These authors contributed equally to this work.

[1]Research and Development Department, Pioneer Alpha,  Dhaka, Bangladesh

[2]Research and Development Department, Pioneer Alpha,Dhaka,Bangladesh

[3, 4]Department of Computer and Information Science, Clark Atlanta University, Atlanta, GA, USA

[5]Master’s of Engineering Management (MSEM), Westcliff University, Irvine, California, USA







Machine learning (ML) is a rapidly developing area of medicine that uses significant resources to apply computer science and statistics to medical issues. ML's proponents laud its capacity to handle vast, complicated, and erratic medical data. It's common knowledge that attackers might cause misclassification by deliberately creating inputs for machine learning classifiers. Research on adversarial examples has been extensively conducted in the field of computer vision applications. Healthcare systems are thought to be highly difficult because of the security and life-or-death considerations they include, and performance accuracy is very important. Recent arguments have suggested that adversarial attacks could be made against medical image analysis (MedIA) technologies because of the accompanying technology infrastructure and powerful financial incentives. Since the diagnosis will be the basis for important decisions, it is essential to assess how strong medical DNN tasks are against adversarial attacks. Simple adversarial attacks have been taken into account in several earlier studies. However, DNNs are susceptible to more risky and realistic attacks. The present paper covers recent proposed adversarial attack strategies against DNNs for medical imaging as well as countermeasures. In this study, we review current techniques for adversarial imaging attacks, detections. It also encompasses various facets of these techniques and offers suggestions for the robustness of neural networks to be improved in the future.



















[
    [
    August 5, 2024
==================





§ INTRODUCTION




 §.§ Adversarial Attack


Adversarial attacks encompass a set of techniques aimed at manipulating machine learning models by introducing well-crafted, often imperceptible alterations to input data. These modifications aim to deceive the model, leading to misclassifications or inaccurate predictions <cit.>. Adversarial attacks have the potential to compromise machine learning systems' dependability and security, which is concerning for crucial applications such as cybersecurity, autonomous vehicles, and medical diagnosis. Of all the adversarial attack types, "White-box attacks" and "Black-box attacks" are the most common.

Ongoing research in machine learning security centers on the analysis of adversarial attacks and the development of defense strategies. Researchers continually explore novel attack tactics to identify potential weaknesses in AI systems while concurrently striving to construct robust models less susceptible to adversarial perturbations. Artificial intelligence (AI) modern technology is widely successful in fields like computer vision, natural language processing, and automated driving; however, its application in vital safety sectors is hindered by its susceptibility to adversarial attacks. Consequently, enhancing the resilience of AI systems against such attacks has become paramount for the progress of AI <cit.>.

The rapid advancement of AI technologies has found diverse applications in numerous domains, from machine translation, speech recognition, and object identification to more intricate tasks like drug composition analysis <cit.>. Noteworthy applications include brain circuit construction <cit.>, particle accelerator data analysis <cit.>, <cit.>, and DNA mutation impact analysis <cit.>. Since Szegedy et al.'s seminal work <cit.> highlighting neural networks' vulnerability to adversarial attacks, research on adversarial technologies for artificial intelligence has steadily expanded. New techniques for adversarial attacks and strategies for mitigating them are consistently emerging.

The term "adversarial attacks at the training stage" refers to actions taken by adversaries to manipulate the training dataset, input characteristics, or data labels during the training phase of the target model. This manipulation of the training dataset involves actions such as adding or deleting training data, as demonstrated by Barreno et al.'s approach <cit.>. During the testing phase, adversarial assaults are categorized into white-box attacks and black-box attacks <cit.>. In white-box situations, attackers have access to the parameters, techniques, and structure of the target model, enabling them to craft adversarial samples based on this information.



 §.§ The Adversary's Objective: Evasion attack versus poisoning assault


"Poisoning attacks" are assault methods that allow an attacker to add or change a large number of fake samples to the training set of a DNN algorithm. The trained classifier may perform poorly as a result of these bogus data. They may have poor accuracy <cit.> conversely make imprecise predictions on a few samples for analysis <cit.>.
The classifiers used in evasion attacks are fixed and often work well on safe testing samples. The adversaries are unable to reform the classifier's settings or parameters, but they do generate some fictitious instances that the classifier is unable to distinguish.



 §.§ Deep Neural Networks: Adversarial Attacks

Regarding medical image processing use cases such as diagnosing cancer and lesion identification, deep neural networks, also known as DNNs, are rapidly gaining popularity <cit.>. Nonetheless, a new study suggests that adversarial examples/attacks with tiny, barely noticeable changes can weaken medical deep learning systems. There are presently safety concerns associated with the usage of these devices in clinical settings <cit.>.  Deep neural networks (DNN) are increasingly well-liked and effective at a variety of machine learning requisitions. They have been employed with surprising effectiveness in a variety of recognition issues in the fields of pictures, graphs, text, and voice <cit.>. They are able to identify things in images with accuracy that is almost human <cit.>, <cit.>. Additionally, they are employed in speech recognition <cit.>, natural language processing <cit.>, and gaming <cit.>. 

   
Regarding the data set provided by MNIST, Biggio et al.<cit.> produce adversarial instances first, focusing on traditional machine learning classifiers like SVMs and 3-layer neural networks with full connectivity which is shown in Figure <ref>. In order to trick the classifier, it optimizes the discriminant function. As an illustration, consider a linear SVM classifier on the MNIST dataset. 



 §.§ Examining examples of opposition in the real world


The research <cit.> put decals to traffic signs that pose a serious threat to autonomous vehicles' sign recognition technology. These hostile objects are especially detrimental to deep learning systems since they can interfere directly with numerous real-world DNN usages, such facial recognition and autonomous cars. By assessing the adversarial images (FGSM, BIM) produced to see if they are "robust" against changes in natural circumstances (e.g., shifting viewpoint, illumination, etc.), the authors of the work <cit.> investigate the viability of creating tangible adversarial objects. Robust in this context means that even after transformation, the produced pictures are still antagonistic. The experiment's findings show that many of these adversarial examples—particularly those produced by FGSM—remain antagonistic to the classifier even after transformation. The findings imply that antagonistic real-world objects could trick the sensor in many situations.



 §.§ Medical Image Under Artificial Intelligence Attack


Researchers have access to strong models of developing science and technology thanks to deep learning. Convolutional neural networks, also known as CNNs, are among the most significant categories of deep learning (DL) frameworks for the analysis and processing of images because of their exceptional ability to learn valuable information.  The processing of the human organism utilizing different picture modalities for therapeutic, diagnostic, and health surveillance purposes is known as medical image analysis <cit.>. The advancement of computer vision through the use of deep neural networks addresses issues that were not well-solved by traditional image processing methods. Beinfeld et al. <cit.> asserted that $385 spent on medical imaging results in a savings of almost $3000. MRI, CT scans, ultrasound (US), and X-rays are the most frequently used image modalities.  Figure <ref> shows how adversarial assaults may be used to arbitrary alter diagnosis outcomes in three different medical picture datasets: Dermoscopy <cit.>, X-ray of the chest <cit.>, and Fundoscopy <cit.>. 

   


Hu et al. <cit.> assert that in order to develop more reliable models, it is crucial to investigate the reasons why adversarial cases emerge and to comprehend deep learning models better. Depending on the knowledge of the enemy, attacks can be categorized into three kinds.   The saliency (or attention) map of a picture that was input shows the regions that significantly change the model's output based on the gradients of the categorization loss with regard to the input <cit.>. We can see that some medical photos have highly concentrated regions that are noticeably larger. This could indicate that the rich biological textures included in medical photos occasionally entice the DNN model to focus more on aspects unrelated to the diagnosis. Tiny adjustments in these high-focus areas can significantly affect the model's output. 

Due to several factors, including High-Dimensional Data, High-Dimensional Data, Transferability of Adversarial Examples, Complex Decision Boundaries, Black-Box Attacks, Safety-Critical Applications, and Safety-Critical Applications, Medical Image Deep Neural Network (DNN) models can be relatively easier to attack when compared to some other domains. Despite these difficulties, experts in the field are actively attempting to create more reliable and safe DNN models for medical images. To make these models more resistant to hostile attacks, strategies like adversarial training, input preprocessing, and defensive distillation are being investigated. To further enhance the security and dependability of AI systems in healthcare applications, consistent evaluation frameworks for adversarial robustness in medical imaging must be developed.

Byra et al. <cit.> proposed an attack strategy on ultrasonography (US) imaging for liver fatty tissue. Radio-frequency signals are used to rebuild US pictures, and the reconstruction technique was subjected to a zeroth-order optimization attack <cit.>. The InceptionResNetV2 model was used in the studies, and the assault resulted in a 48% loss in model accuracy. Ozbulak et al. <cit.> proposed an attack specifically designed for medical picture segmentation called the adaptive segmentation masking attack (ASMA). The suggested attack offers significant intersection-over-union (IoU) degradation and produces nearly undetectable samples for most portions. Because the U-Net framework is among the most well-known models for clinical picture segmentation, they employed it in the trials. The datasets used were for segmentation of glaucoma optic disk <cit.> and ISIC skin lesion segmentation <cit.>.  Chen et al. published a method for fabricating hostile cases to thwart medical picture segmentation <cit.>. In order to simulate anatomical and intensity fluctuations, geometrical deformations are used to create the adversarial examples. By attempting to partition organs from abdominal CT scans using a U-Net model, they tested the effectiveness of these examples. With reference to the Dice score measure, they successfully reduced it significantly across all organs. The kidneys and pancreas, however, require a higher level of disturbance and are more arduous to assault than the liver as well as spleen.

   


Tian et al. <cit.> examined the phenomena of bias field, which can arise from incorrectly capturing a medical image and jeopardize a DNN's efficacy, as shown in Figure <ref>. Motivated by adversarial attacks, the authors created an adversarial-smooth bias discipline approach to trick a model. The ResNet50, MobileNet, among others and DenseNet121 models were employed to fine-tune the chest X-ray dataset that was used in their study. They looked at this attack's transferability and white-box attacks. Comparing the suggested attack to other cutting-edge white-box attacks, it exhibited a greater attack accuracy on transferability.

This review's major organization is as follows: We offer some significant adversarial notions on medical images in Section 2. Additionally, it provides a detailed overview of earlier research. We explore a few strategies in Section 3 with regard to the picture classification scenario. We utilize Section 4 to quickly summarize some findings from earlier research that tries to explain the phenomena of antagonistic examples on medical data. The overview is concluded in Section 5.



§  BACKGROUND LITERATURE REVIEW


Adversarial instances are intentionally created data inputs that are intended to degrade a machine learning model's performance. When researchers examined strategies used by spammers to evade spam filters in 2004, they unofficially coined the term "adversarial inputs" <cit.>. These misleading examples are usually produced by deliberately faking real data, such spam advertising messages, in order to trick the computer system that analyzes it. Alterations can be applied to text data, like spam, by introducing innocuous text or modifying frequently used phrases in malicious communications with synonyms. To fortify algorithms against adversarial attacks, researchers have explored various strategies, including training algorithms on adversarial samples and employing sophisticated data processing techniques to minimize the susceptibility to manipulation. The quest for fully robust models in machine learning aims to accelerate the development of algorithms capable of making decisions based on consistent explanations, with promising early efforts in this direction <cit.>.

In the realm of healthcare, medical claims codes play a crucial role in determining the amount spent on a patient visit post-payer approval. Payers often assess these claims using automated fraud detectors increasingly driven by machine learning. Historically, healthcare providers have shaped payer records of patient visits, including the associated codes, to influence the algorithmic outputs of payers <cit.>. Medical fraud, a market valued at $250 billion, exemplifies the extreme end of this strategic tailoring of patient presentations. While some practitioners may overtly fabricate medical claims, patient data falsification often takes more covert forms. For instance, consistently submitting codes for billing services identical to, but more expensive than, those actually provided is known as intentional upcoding. Here, Table <ref> shows past exclusive past research works where authors did some fantastic work for medical images to defend adversarial attacks. Different researcher proposed different frameworks to overcome this problem and to suggest a new thought for revolution. 



 §.§ Elements of a Cyber Adversary's Assault


   


In some instances, subtle adjustments to billing codes blur the boundary between fraudulent activities and conscientious best practices. One notable illustration may be found in the guidelines on the Endocrine Society website, which advise medical professionals not to bill for metabolic syndrome, or International Classification of Diseases (ICD) code 277.77, in situations of obesity. Coverage denial is likely to occur with this particular code/condition combo in Figure <ref>. The Society recommends an alternative approach, suggesting billing for the individual codes related to specific disorders comprising the metabolic syndrome, such as hypertension. 










 §.§ The components of an adversarial attack


   


Presenting a thorough overview of the various adversarial attack strategies and defense techniques is the aim of the study <cit.>. We first discuss the theoretical underpinnings, practices, and practical uses of adversarial attack strategies. Then, a brief discussion of the research on defense strategies obscuring the boundary of the large field follows. A few selected articles are published year-by-year from 2012 to 2021, as seen in Figure <ref>. This work aims to provide thorough taxonomies, evaluations of harmful assaults, and protections against the full DL pipeline. In this context, the numerous attack and defense tactics that have been developed over the previous two years have been categorized, with a focus on the clinical deep learning systems that are susceptible to adversarial attacks. 

Several defensive strategies have been put forth in an effort to neutralize possible threats. A popular method in natural imaging called adversarial training adds adversarial images to the training dataset in order to make Convolutional neural network, or CNN, models more resilient. Nevertheless, this method is not the best for datasets pertaining to medical imaging, since the addition of different adversarial images could considerably reduce classification accuracy. A robust detection technique for malicious images is presented in the work of Li et al. <cit.>, successfully defending against attacks on deep learning-driven medical image labeling systems. In a different study <cit.>, scientists compared CNN performance with Vision Transformers (ViTs) to determine how resilient CNNs are to different types of attacks in computational pathology. The authors developed robust neural network models, evaluating their efficacy against both white- and black-box attacks. The structures of attacks for both models were scrutinized, with an exploration of the underlying factors influencing their performance. The study's findings were validated through two clinically relevant classification tasks involving distinct patient groups. Preceding Ma et al.'s research <cit.>, ambiguity surrounded medical image adversarial attacks (AAs), limiting adversarial machine learning analyses to natural images. Unlike natural images, medical images may contain domain-specific elements, impacting medical deep-learning systems susceptible to AAs. AAs possess the potential to manipulate diagnoses and outcomes, emphasizing the critical need for a robust healthcare infrastructure to mitigate potential risks from malicious attacks.






§ MATERIALS AND METHODS



 §.§ Medical Adversarial Defensive Strategies

Various defense models, including techniques such as input denoising, input gradients regularization, and adversarial training, have been devised. However, recent attacks often manage to evade or partially circumvent these protective measures. This paper undertakes a comprehensive exploration of adversarial attack and defense methods within the domain of medical image analysis. It introduces a novel taxonomy based on the application context for a family of techniques, as outlined in Dong et al.'s work <cit.>. The author establishes a unified theoretical framework encompassing diverse adversarial attack and defense strategies tailored for medical imaging applications.

The primary emphasis in most research efforts on adversarial attacks in medical image processing is on the white-box scenario. These studies, characterized by a comprehensive understanding of medical Deep Neural Networks (DNNs), concentrate on the susceptibility of computer-aided diagnosis models across various medical imaging applications. In the execution of adversarial attacks, attackers may utilize the target diagnosis DNN as a locally deployed model. In addition to white-box adversarial attacks for medical classification tasks, scholars have delved into vulnerabilities associated with various medical imaging tasks. Notably, these publications focus predominantly on medical segmentation, as evident in works such as Chen et al.'s <cit.> and Ozbulak et al.'s <cit.>.

For natural photos, semi-white-box (Gray-box) attacks have been extensively researched <cit.>. However, there are only a few academic publications <cit.>, <cit.> that address this attack scenario for medical image processing. The semi-white-box adversarial approach typically consists of two stages: 1) To create adversarial instances against the target DNN model, the attacker trains a generative model. The attacker has complete access to the target model during training, including backward propagation gradients. 2) Instead of needing to know anything about the target model, as would be the case in a completely black-box scenario, the adversary generator can directly obtain adversarial examples against the target model with the input of authentic photos during the application stage.

Presently utilized white-box adversarial attacks often require multiple backward gradients of the target model. In simpler terms, attackers generate comparable adversarial samples by treating the target Deep Neural Network (DNN) as if it were a locally deployed model. However, due to its reliance on an in-depth understanding of the DNN model to execute an attack, the white-box approach may be unreliable in real-world scenarios. Conversely, the general black-box scenario may offer a more suitable environment for simulating real-world adversarial attacks, with numerous proposals focusing on investigating black-box assaults for natural images.

The outlined solutions usually involve multiple queries to the black-box model or depend on having a comprehensive understanding of the desired diagnosis model. However, in many real-world situations, attackers may lack direct access to the target diagnosis model. The restricted black-box (no-box) configuration, specifically representing the most challenging scenario for real-world adversarial assaults, may be more covertly dangerous even without querying the target black-box DNN. The transferability <cit.> of adversarial instances across different DNN models fundamentally determines the success of no-box attacks. For example, a no-box attacker can generate adversarial images based on a locally deployed surrogate model, transferring them directly to target medical diagnosis systems. Restricted black-box adversarial attacks are deemed more subtly perilous for natural vision tasks according to a renowned study.

Considering the substantial risks to the healthcare sector, numerous defense strategies have been proposed to counter medical adversarial attacks. In order to overcome the weaknesses shown by adversarial examples, a number of methods have been developed to provide trustworthy deep learning-based systems for natural pictures <cit.>. The development of accurate computer-aided diagnosis models for clinical applications not only contributes to trustworthy healthcare services for millions but also underscores the importance of investigating adversarially robust models within the realm of medical image analysis.

In their work <cit.>, the authors succinctly summarize studies on adversarial defense in the context of medical image analysis. Various attack methods in the realm of medical images serve as robustness evaluation criteria for adversarial defense, complementing the extensive range of adversarial attack methods mentioned. The minimal separation between adversarial examples and the model's decision boundary signifies the model's resistance to disturbances. The goal of adversarial training is to mitigate subsequent losses. 

    
Minimize the 


    L(f(x_adv, y) + λ· L(x, x_adv))


where L(f(x_adv, y) is the classification loss, L(x, x_adv) is the perturbation size, and λ is a trade-off hyperparameter. 



Transforming the input data into an alternative feature space that diminishes the impact of adversarial perturbations on the model. A technique aimed at enhancing resilience against adversarial attacks involves introducing noise to the features or intermediate representations that characterize the input data.




 §.§ Adversarial Training

Most medical adversarial defense techniques focus on using adversarial training to create reliable diagnoses systems. A significant fraction of the works among them go beyond current techniques for natural image adversarial training to tasks relating to medical classification <cit.>, <cit.>, Vatian et al. <cit.> looked into opposing examples for medical imaging and tried a number of defense strategies to oppose these nefarious representations.



 §.§ Adversarial Detection

Adversary detection seeks to identify adversary cases from input examples during the application stage, as opposed to developing robustness during the training stage of computer-aided diagnosis models.Many adversarial detection techniques have been put forth in the field of medical image analysis to stop additional misdiagnosis caused by adversarial samples <cit.>. In particular, it is possible to think about medical adversarial detection as an anomaly detection issue that can be resolved by combining explainability approaches <cit.>.



 §.§ Image-level Pre-processing

A clean image and the related adversarial perturbation make up an adversarial image in general. Meanwhile, it has been shown that DNNs can perform well on clean images while still being vulnerable to adversarial examples <cit.>. Denoising the adversarial example to remove the perturbation component can therefore help make the subsequent network diagnostic easier. Image-level pre-processing can be useful and secure in the context of biomedical image analysis because it does not require re-training or modifying medical models. 



 §.§ Improvement of Features


The difference in robustness between human and machine vision is attributed to adversarial examples linked to non-robust features extracted from specific patterns in the data distribution <cit.>. Consequently, enhancing feature representation is crucial for the development of robust inference systems. In this investigation, we characterize feature augmentation as the modification of architectures or mapping functions. Various techniques aimed at improving features have significantly enhanced the robustness of medical classification models <cit.>.



 §.§ Distillation of Knowledge

In the field of machine learning, knowledge distillation is a helpful technique for transferring learned information from a complex (teacher) model to a simple (student) model. Therefore, the situation where the network structures for the teacher and learners are identical is specifically referred to as self-distillation. Furthermore, a great deal of research has been done on the natural imaging domain using adversarial knowledge distillation, which moves the adversarial resilience from a heavy teacher model to a view student model <cit.>.



§ RESULT AND DISCUSSION


This study utilizes four publicly accessible benchmark datasets, as detailed in <cit.>, to investigate adversarial attack and defense in the context of medical image processing (illustrated in Figure <ref>). Firstly, the Messidor1 dataset comprises 1,200 eye fundus color images for detecting diabetic retinopathy across four classes based on retinopathy grade. Secondly, the International Skin Imaging Collaboration (ISIC) dataset consists of 2,750 dermoscopic images categorized into three classes for skin lesion classification and segmentation. Thirdly, the ChestX-ray 14 dataset comprises 112,120 frontal-view X-ray images representing 14 thorax diseases. Lastly, the COVID-19 database incorporates 21,165 chest X-ray images accompanied by segmentation-capable lung masks.

   


The study conducted by the authors <cit.> employs experiments to demonstrate the following: 1) The SSAT module significantly enhances the adversarial robustness of the model without compromising the classification accuracy of clean images. 2) The UAD module effectively identifies and excludes a majority of successful adversarial examples. 3) In comparison to other existing AI systems, their medical imaging AI solution (UAD + SSAT) minimizes the risk of adversarial attacks. The research utilizes a publicly available dataset of retinal OCT images, employing the "white-box" setting as the most challenging threat for assessing class prediction performance. Across all white-box attack conditions, the authors establish that SSAT consistently outperforms alternative baselines while maintaining comparable or superior performance for clean image classification. Furthermore, the authors demonstrate that the combined approach of UAD complementing SSAT results in the lowest adversarial risk according to the new measure presented. Regardless of the training techniques employed, systems based on UAD consistently exhibit reduced risks compared to those that do not incorporate this defense mechanism.

   
An AI-CAD model depicted in Figure <ref> was initially trained to manipulate diagnosis-sensitive elements within images, such as adding or removing malignant tissue. The model's effectiveness was then assessed using adversarial images produced by a GAN model. To determine how well human specialists could visually distinguish images produced by the GAN, a reader study was carried out.

In a study by Zhou et al. <cit.>, the responses of an AI-CAD model to adversarial attacks on GAN-generated mammography images were investigated. This involved introducing malignant tissue into healthy images and removing cancerous tissue from images affected by cancer. The study also evaluated the proficiency of experienced radiologists in visually identifying such adversarial images, both before and after instructional intervention. The study cohort, provided by the University of Pittsburgh Medical Center, included 1284 women, and 4346 mammography images were collected. Among these, 366 patients had biopsy-proven malignant breast cancer, while 918 patients had breast cancer assessed as benign (including benign signs).

Following training, the researchers assessed the model's classification accuracy concerning both the original genuine test data and its equivalent GAN-generated adversarial counterparts. Two GAN-trained U-Net23 models were employed to generate adversarial images, flipping labels in the test set. The high-resolution (1728 x 1408) photographs indicated the categorization impact of the AI-CAD model on the test data. The AI-CAD model achieved an AUC of 0.82 on the test set, consisting of 364 genuine negative samples and 74 real positive samples. The adversarial GAN-generated images (with flipped labels) in the test set yielded an AUC of 0.94. These AUC values suggested that the adversarial collection of images effectively deceived the AI-CAD model. Additionally, 59.5% (44 out of 74 cases) of genuine positive images with a classification accuracy threshold of 0.5 were correctly labeled as positive, whereas 95.5% (42 out of 44 cases) of adversarial samples created by GANs successfully tricked the system by being classified as negative. 

   


Figure <ref> displays the mean SSIM values for FGSM and PGD assaults across all photos. The Supplementary Material includes the SSIM data for every model. As can be shown, the effects of the identical disturbance applied to various imaging modalities on human visual perceptibility with the observed SSIM varies. The radiology images most clearly showed adversarial disturbances, with = 0.02 producing an already apparent, but very modest perturbation. At the same perturbation level for the ophthalmology and pathology photos, perturbations were nearly undetectable and became apparent with larger epsilon values. The authors examine black-box adversarial attacks on deep learning in healthcare imaging. <cit.>. They investigate three medical imaging sectors where deep learning algorithms are vulnerable. In all three datasets, perturbations calculated by FGSM showed lower SSIM than those calculated using PGD. This is a predicted outcome given that PGD optimizes perturbations based on their size and influence on model predictions.We opted to document attacks in our subsequent analyses employing 0.02, as this represented the maximum perturbation level that remained effective across all applications and attack strategies. Additionally, it exhibited greater transferability than an epsilon value of 0.01 in the majority of the examined applications.

   


Figure <ref> shows examples taken from the ophthalmology collection that demonstrate the transferability of assaults when the target and surrogate are randomly started and when both are pre-trained on ImageNet. Whether the target and surrogate models had the same architecture or a different one did not affect the findings. 



§ CONCLUSION

Diagnostic imaging AI systems based on computer vision are increasingly being used as models for classifying and segmenting diseases. The scaled-up use of medical imaging AI systems has given rise to serious safety issues because to DNNs' susceptibility to adversarial samples. Recently, a number of ways have been put out to increase the efficacy of medical image defense tactics. Although numerous protection strategies have been put forth, there are still reservations over the use of medical deep learning techniques. This outcome arises from certain constraints within medical imaging, including a scarcity of high-quality image datasets and labeled data in comparison to more abundant datasets available for natural images. The efficacy of adversarial defenses may not be universally applicable in this context. The findings underscore the importance of exercising caution in the design of Deep Neural Networks (DNNs) for medical imaging and their subsequent real-world applications.


























§ DECLARATIONS






 §.§ Funding
 A portion of the funding for this research comes from DOEd Grant P116Z220008 (1) and NSF Grant No. 2306109. The author(s) expresses all opinions, findings, and conclusions; the sponsor(s) does not necessarily agree with them.


 §.§ Conflict of interest
 The authors do not declare any conflicts of interest.



 §.§ Ethics approval
 There is no original research involving human subjects, animal subjects, or sensitive data in this review publication. 



 §.§ Consent to participate
 The studies and data included in this review were sourced from publicly available and previously published literature. 


 §.§ Consent for publication
 All authors listed on this review paper have reviewed and approved the final manuscript for submission to Machine Learning. Each author has contributed significantly to the research, writing, and revision of the paper. All authors have read and agreed to the content presented in this review, and their consent for publication is hereby provided.


 §.§ Availability of data and materials
 The data and materials used can be found in the references of this work.


 §.§ Code availability
 The review is based on a comprehensive analysis of publicly available literature, and any references to specific methodologies or algorithms are attributed to the respective original sources. We have strived to provide clear citations and references to enable readers to locate the original works for further examination.


 §.§ Authors' contributions

 Conceptualization: A.B., M.A.A.N.; Methodology: A.B., M.A.A.N.; Formal analysis and investigation: A.B., M.A.A.N.; Writing - original draft preparation: A.B., M.A.A.N., R.G., A.R.; Writing - review and editing: M.A.A.N., A.R.; Funding acquisition: K.D.G; Resources: R.G.; Supervision: K.D.G., R.G. All authors have read and agreed to the published version of the manuscript.

















 Paper Title: Deceptive AI systems that give explanations are more convincing than honest AI systems and can amplify belief in misinformation.


Priority and Stackelberg Game-Based Incentive Task Allocation for Device-Assisted MEC Networks
    Yang Li2, Xing Zhang2 1, Bo Lei3, Zheyan Qu2, Wenbo Wang2 
2Key Laboratory of Universal Wireless Communications, Ministry of Education

2Beijing University of Posts and Telecommunications, Beijing 100876, China 

3Beijing Branch of China Telecom Co., Ltd., Beijing 100032, China 

1Email: zhangx@ieee.org

    August 5, 2024
=====================================================================================================================================================================================================================================================================================================================

§ ABSTRACT 
Advanced Artificial Intelligence (AI) systems, specifically large language models (LLMs), have the capability to generate not just misinformation, but also deceptive explanations that can justify and propagate false information and erode trust in the truth. We examined the impact of deceptive AI generated explanations on individuals' beliefs in a pre-registered online experiment with 23,840 observations from 1,192 participants. We found that in addition to being more persuasive than accurate and honest explanations, AI-generated deceptive explanations can significantly amplify belief in false news headlines and undermine true ones as compared to AI systems that simply classify the headline incorrectly as being true/false. Moreover, our results show that personal factors such as cognitive reflection and trust in AI do not necessarily protect individuals from these effects caused by deceptive AI generated explanations. Instead, our results show that the logical validity of AI generated deceptive explanations, that is whether the explanation has a causal effect on the truthfulness of the AI's classification, plays a critical role in countering their persuasiveness – with logically invalid explanations being deemed less credible. This underscores the importance of teaching logical reasoning and critical thinking skills to identify logically invalid arguments, fostering greater resilience against advanced AI-driven misinformation.









§ MAIN

Artificial Intelligence (AI) systems, such as large language models (LLMs), have the alarming capability to generate not only misinformation but also deceptive explanations that justify misinformation and make it seem logically sound.

Researchers have identified an increase in AI-generated disinformation campaigns <cit.> and the factors that make them disruptive to people' ability to discern true and false information <cit.>, as well as change people's attitudes <cit.>. These factors include authoritative tone <cit.>, persuasive language <cit.>, and targeted personalization <cit.>. However, little is known about the influence on people's beliefs when explanations are used as a tactic for misinformation.

In the broader context of the AI-generated misinformation landscape, we can identify three levels of sophistication, each posing unique challenges. The most basic level involves AI systems generating false news headlines without any accompanying justification. While such headlines can still mislead, they are more easily dismissed as baseless claims. The next level involves AI systems providing deceptive classifications, labeling false information as true or vice versa. This added layer of perceived authority can lend unwarranted credibility to the misinformation. However, the perhaps most subtle and deeper level is when AI systems generate deceptive explanations to justify and propagate false information.

One of the reasons why the topic of AI generated explanations and misinformation remains unexplored is that the use of explanations as a tactic for misinformation goes against the commonly held beliefs that explanations always make AI systems more transparent, trustworthy <cit.>, and fair 
 <cit.>. While researchers have shown that honest explanations can assist people in determining the veracity of information <cit.> and improve their decision-making outcomes <cit.>, as well as reduce human overreliance on AI systems <cit.>, research in psychology has demonstrated that even poor explanations can significantly impact people's actions and beliefs <cit.>. This implies that the mere presence of an explanation can lead to changes in beliefs and behavior, regardless of its quality or veracity. Researchers have also shown that people often do not cognitively engage with the content of the explanation unless they are forced to do so<cit.>.

Deceptive explanations can be used to exploit this vulnerability by connecting correct facts in misleading ways to justify false information or discredit real information and potentially making it more difficult to discern the truth. Disguised as political commentary, scientific explanations, or online discussions, AI-generated deceptive explanations from systems like LLMs could be weaponized at scale by bad actors to manipulate public opinion and decision making, even with safety measures built into the models <cit.>. 

The low cost and easy scalability of AI-generated explanations further compound the problem. With the ability to generate vast amounts of high quality content quickly and cheaply, malicious actors can flood online platforms with deceptive explanations, discrediting or drowning out legitimate explanations by human experts that take time to craft, making it harder for individuals to make sense of reliable information. 

This paper investigates the effects of AI-generated deceptive explanations on human beliefs. Through a comprehensive pre-registered online experiment with 23,840 observations from 1,192 participants, we find that not only are deceptive AI generated explanations more persuasive than deceptive AI classifications without explanations, they are even more persuasive than honest explanations (See Fig. <ref>). While personal factors such as cognitive reflection and trust in AI do not necessarily protect individuals from the effects of deceptive AI generated explanations, the logical validity, that is whether the truthfulness of the conclusions or classifications made by the AI system follows from the truth of the explanation, of AI generated deceptive explanations where found to counter their persuasiveness – with logically invalid explanations being deemed less credible. This research underscores the urgent need for vigilance and proactive measures to ensure the responsible use of AI technology.








§ METHODS






 §.§ Stimuli Curation



We created a dataset of headlines each with one honest and one deceptive explanation by prompting the text-generation model GPT-3 davinci 2 with 12 example explanations randomly sampled from the publicly available fact-checking dataset “liar-plus” <cit.>. This dataset consists of 12,836 short statements with explanation sentences extracted automatically from the full-text verdict reports written by journalists in Politifact (see Fig. <ref>). 


First, 5 honest and 5 deceptive explanations were generated for 40 true and false headlines by prompting GPT-3 (davinci, temp = .7) with the headline and making it complete the sentences “This is FALSE because…” or “This is TRUE because…” (see Fig. <ref>). We further curated the explanations by ranking them by highest semantic similarity and lowest repeated-word frequency. We picked the highest ranked explanations, confirmed the veracity and logical validity of each explanation. The truth veracity was confirmed independently by each of the authors after being instructed by a professional fact-checker following standard fact-checking procedures <cit.> and then aligned. The logical validity was also confirmed by each author independently by deconstructing the claims of each headline and explanation into premises, conclusions and inferences from which the logical validity could be determined in an almost mathematical fashion using a Fisher analysis <cit.>. Next, we then excluded explanations whose veracity did not match the veracity of the headline. Since the resulting dataset had an unequal distribution of veracity and logical validity, we randomly excluded generated explanations until we had somewhat equal distribution of true and false explanations and logically valid and logically invalid explanations for each condition (deceptive vs. honest explanations) ending with a stimulus set consisting of 28 headlines with 1 honest explanation and 1 deceptive explanation each (56 total). We tested for differences across four linguistic dimensions (word count, sentiment, grade level, and subjectivity) and found no statistical differences between conditions. To generate explanations for a Trivia stimulus set, we repeated the same procedure by prompting GPT-3 with 12 example explanations. The example explanations and Trivia statements were randomly sampled from online<cit.>, resulting in a stimulus set consisting of 28 trivia statements with 1 honest explanation and 1 deceptive explanation each (56 total). These were also tested for differences across four linguistic dimensions (wordcount, sentiment, grade level, and subjectivity) and were also found to have no statistical differences between conditions.






In real-life scenarios, large language models may generate explanations without the filtering specified above. However, in this experiment, the filtering algorithm serves to ensure the quality and validity of the explanations for the purpose of our experiment. The aim was to isolate the effects of deceptive explanations and see how they manipulate beliefs even when at their best. Hence, it was important to minimize other variables such as irrelevance or incoherence of the explanations to get a normalized sample. Nonetheless, since the filtering algorithm is automatic, it has no selection bias and could potentially reflect how LLMs could be used in a controlled manner, either by malicious actors crafting algorithms to filter the best explanations to use in misinformation campaigns or by organizations using AI for generating honest explanations for automated fact-checking. In either contexts, it is very likely for there to be a layer of curation or filtering applied to create more convincing or high-quality content.





 §.§ Participant Recruitment


We recruited 1,199 participants through Prolific, <https://prolific.co>. Participants were required to self-report as US citizens, fluent in English and rated their fluency in any other languages they spoke. 1192 of these individuals passed an initial attention check task and were allowed to proceed. Additionally, all participants were fluent in English and 142 had fluency in a second language. Our final sample had a mean age of 39%, was 50% female, and was 72% white. 



 §.§ Task Description



Participants were shown 20 statements during the main discernment task which were either true or false. Each participant saw the 20 statements in a random order, and rated the perceived truth of each statement (“Do you think the statement in the grey box is true or false?”) on a slider scale with 1 decimal from 1 (“Definitely False”) to 5 (“Definitely True”). After the rating, the participants would receive feedback from an AI system and be asked if they want to revise their rating (“Would you like to revise your estimate: Do you think the statement in the grey box is true or false?”) on a slider scale with 1 decimal from 1 (“Definitely False”) to 5 (“Definitely True”) with the default value being same as the previous rating. Participants also rated their knowledge on the topic (“How knowledgeable are you on the topic of [topic]”) on a slider scale with 1 decimal from 1 (“Not at all knowledgeable”) to 5 (“Very much knowledgeable”). The selection of statements and generation of AI feedback is further explained in section <ref> and the task interface can be seen in Fig. <ref>.






 §.§ Randomization



For the main discernment task, participants were randomly assigned to one of two conditions: (i) news headline statements or (ii) trivia item statements (between-subjects); and one of two conditions (i) no explanation (“This is true / false”), or (ii) explanation (“This true / false because…”) (between-subjects). The order of the stimuli being presented was also randomized. See Fig. <ref> for examples of items across conditions. 



 §.§ Post Task Survey



After the discernment task, participants were asked to complete post-test surveys to measure their critical thinking, and level of self-reported trust in the agent providing them with explanations. To measure the level of critical thinking of subjects, we used cognitive reflection test (CRT), a task designed to measure a person's ability to reflect on a question and resist reporting the first response that comes to mind <cit.>. For the CRT we randomly sampled three items from the extended CRT <cit.>. Finally, following Epstein et al. <cit.>, to assess trust in the AI agent participants answered a battery of six trust questions derived from Mayer, Davis, and Schoorman <cit.>’s three factors of trustworthiness: Ability, Benevolence and Integrity (ABI).




 §.§ Participants

A total of 1,199 individuals participated in the experiment. We used the Prolific platform to recruit individuals from the United States. We focus our analysis on the 1,192 of 1,209 recruited participants who passed the attention check. 589 participants rating news headlines and 610 participants rating trivia statements. Of the 589 participants rating news headlines, 289 received no explanation and 300 received an explanation. Of the 610 participants rating trivia statements, 299 got no explanation, and 311 got explanations. Each participant rated 20 statements, with an average of 51% of statements being true and 50% of explanations being deceptive.



 §.§ Approvals

This research complies with all relevant ethical regulations and the Massachusetts Institute of Technology’s Committee on the Use of Humans as Experimental Subjects determined this study to fall under Exempt Category 3 – Benign Behavioral Intervention. This study’s exemption identification number is E-3754. All participants are informed that “This is an MIT research project. All data for research is collected anonymously for research purposes. We will ask you about your attitudes towards information and AI systems. For questions, please contact vdanry@mit.edu. If you are under 18 years old, you need consent from your parents to continue.” Participants recruited from Prolific were compensated at a rate of $10.82 an hour. At the end of the experiment participants were made aware that they had received AI explanations that were sometimes deceptive in the experiment, being told that "In this study, you were asked to collaborate with an AI-system for rating the accuracy of statements. All feedback in this study was AI-generated. Some of the feedback from the AI system was simply deceptive". Future work must explore the limits, ethics and consequences of exposing participants to AI-generated content.



 §.§ Scoring Perceived Truthfulness and Perceived Logical Validity

To investigate the effects of semantic features of explanations, we used GPT-4 to split each explanation into individual propositions / premises and score the perceived truthfulness and logical validity of each premise. For instance, for the headline “Newsmax plans expansion to capitalize on Trump support, anger at Fox News”, the explanation “This is TRUE. Newsmax has been a vocal supporter of Trump, and the network has even hired Trump's former campaign manager.” was split by GPT-4 into the two premises: “Newsmax has been a vocal supporter of donald trump.” and “the network newsmax has hired donald trump's former campaign manager.”. To conduct the splitting GPT-4 was given the following prompt: “Split the following explanation into its containing claims. Fill in pronouns and references so that each claim can be verified by itself without any context. Separate them with a new line. Only give the answer and be as concise as possible. Example: example. Explanation:explanation”.

Each premise of each explanation was then scored by GPT-4 for its perceived truthfulness using the following prompt: “How likely is it that someone perceives this as true on a scale from 0.00-1.00 with decimals. 0.00 being extremely unlikely and 1.00 being extremely likely. Only give me the answer. Even if this is highly subjective how do you think some person might think this to be likely: premise”. Next, each premise was then scored by GPT-4 for its perceived logical validity with the deceptive classification of the headline using the following prompt: “How likely is it that the general public would believe this: 'premise' to support this: 'This is classification: headline'. Output a score from 0.00-1.00 with decimals. 0.00 being extremely unlikely and 1.00 being extremely likely. Only give me the answer:”. An average score for both truthfulness and logical validity was calculated for each explanation.



 §.§ Calculating Syntactic Features

To calculate word count and reading ease, we used the Natural Language Toolkit (NLTK) in Python. The word count was simply the total number of words in each explanation. Reading ease was estimated using the Flesch Reading Ease formula, which assesses text on a 100-point scale; the higher the score, the easier the text is to understand.

To calculate the grammatical correctiveness of the explanation, we used the LanguageTool Python library, which returned the count of grammatical errors it could detect in the text. 



 §.§ Analysis



In order to gain insights into whether explanations lead to more accurate beliefs, we first compare the belief ratings of user with AI-generated explanations (honest and deceptive) and no feedback. Accuracy was coded by subtracting the belief ratings from the ground truth per rating per participants.

To investigate the relationship between statement ground truth, the presence of explanations, and deceptive classifications (X1, X2, and X3) and belief and accuracy distribution (Y1, and Y2), along with their potential interactions and moderator variables, we employed an ordinary least squares (OLS) regression models.

We analyzed a total of 23,980 observations of belief ratings (12,200 trivia statement observations and 11,780 news headline observations) of true and false statements, collected through our experiment. The response variables, Y1 and Y2, represent the participant belief and accuracy distributions, while the predictor variables are as follows: X1 - statement ground truth, X2 - presence of AI explanation, and X3 - deceptive AI feedback. The moderator variables include the following z-scored variables: logical validity, self-reported prior knowledge, cognitive reflection test score <cit.>, and trust in AI systems<cit.>.

We chose an Ordinary Least Squares (OLS) regression model for its analytical rigor in assessing linear relationships and controlling for confounding factors and mediators, making it ideal to intricately dissect the effects of AI explanations on belief accuracy and to unravel the complex dynamics between statement truth, explanation presence, and deception effectively. We assume that errors are independent and normally distributed with constant variance. Interaction terms between the predictors (X1:X2, X1:X3, and X2:X3) were included in the model to examine any joint effects of the veracity and explainability factors on belief distribution (Y1), accuracy distribution (Y2), and discernment distribution. Belief refers to the participants' subjective judgments about the truthfulness of the statements, while accuracy represents the correctness of these judgments (i.e., whether participants correctly identified true and false statements). We pre-registered our analysis at <https://aspredicted.org/YLK_S3F>.

Our moderation analysis for CRT, need for cognition, trust, and prior knowledge was conducting by, for each moderating variable, re-running the main analysis model with the addition of the z-scored moderator and all interactions. Additionally, our moderation analysis also examined the 4-way interaction between veracity, explanation veracity, explanation type and the moderator.

We also conducting a moderation analysis for logical validity by re-running the main analysis model restricting to the classification + explanation condition, with the addition of the z-scored moderator and all interactions (limiting to only AI classifications with explanation) and examining the 3-way interactions between veracity, explanation veracity, explanation type and the moderator.

We then conducted a moderation analysis for number of premises, perceived truthfulness and perceived logical validity running the main analysis model restricted to deceptive explanations examining the interactions between headline veracity, number of premises, average perceived truthfulness of the premises and average perceived logical validity of the premises.

Lastly, we conducted a moderation analysis for the word count and reading ease by running the main analysis model restricted to deceptive explanations and examining the interactions between headline veracity, word count, and reading ease of the explanations.




§ RESULTS


In order to study the effects of deceptive AI explanations on human beliefs, we conducted a pre-registered online experiment with 23,840 observations from 1,192 participants rating their beliefs in true and false news headlines before and after receiving AI generated explanations. We designed our experiment to disentangle the influence of deceptive AI explanations from merely receiving a deceptive (inaccurate) classification of a news headline as true or false without explanation. This was done by randomizing participants to receiving true and false news headlines accompanied by either deceptive classifications or deceptive classifications with deceptive AI explanations (between-subjects). Moreover, we examine the impact of logical validity of the explanation, and how personal factors and the effects of syntactic and semantic features of the deceptive explanations may influence the outcomes.

For each true and false news headline, an LLM model (GPT-3) was used to generate an honest and a deceptive explanation by prompting the model to complete the following statements: “This is false because…” or “This is true because…”. For deceptive explanations, the model generated inaccurate explanations stating why true statements were false and why false statements were true. This was done for both news headlines and trivia statements. Examples of the generated classifications with explanations can be found in Appendix <ref>.



 §.§ Deceptive AI-generated Classifications with Explanations are more Persuasive than Honest AI-generated Explanations


To understand the persuasiveness of deceptive AI-generated explanations, we compared the relative persuasiveness of deceptive and honest explanations on belief change (i.e. the absolute difference in rating from before and after seeing an AI classification with explanation per item per participant). We find that deceptive AI-generated explanations are significantly more persuasive than honest explanations on both true and false news headlines (β = 0.40, p < 0.0001 and β = 0.29, p = 0.003, respectively, not pre-registered). Prior research has shown that fake news is shared more often that true news due to factors such as novelty and emotional content (fear, disgust, and surprise) <cit.>. It is likely similar factors are at play for deceptive explanations, potentially explaining why they were found to be more persuasive than honest explanations. The full regression table can be found in Table <ref> in the Appendix.





 §.§ Explanations Can Amplify Beliefs in False Information


To ensure our results were caused by explanations and not simply labeling information as true or false, we compared the influence of AI feedback with and without explanations (deceptive AI-generated explanations and deceptive AI generated classifications, respectively). 

Our results show that deceptive AI generated classifications without explanation, significantly increase belief in false news headlines (β = 0.71, p < 0.0001) and decreases belief in true news headlines (β = -1.72, p < 0.0001). In extension, when accompanied by deceptive AI-generated explanations, beliefs in false news headlines were further significantly increased (β = 0.32, p = 0.009) and beliefs in true news headlines where further significantly decreased as compared to just deceptive classifications (β = -0.72, p = 0.0001). These results suggests that the effects of deceptive AI systems amplifies beliefs in information beyond classifications without explanations (Fig. <ref>). The full results can be found in Table <ref> in the Appendix.







 §.§ Personal factors moderate the influence of deceptive AI explanations

In previous studies, cognitive reflection as measured by the Cognitive Reflection Test <cit.> has been found to associate with people's ability to correctly identify misinformation <cit.>. 

However, when receiving AI feedback on the truth of news headlines, our results revealed no significant interactions with cognitive reflection level and deceptive classifications both with and without explanations for false news headlines (β = -0.09, p = 0.20 and β = 0.13, p = 0.15, respectively) and for true news headlines (β = 0.16, p = 0.20 and β = 0.25, p = 0.15, respectively). This suggests that introducing an evaluative AI system framed as a fact-checking system could override the effects of cognitive reflection on truth discernment of news headlines. This may be attributed to several factors. First, the presence of information labeled as provided by “AI” might induce a reliance effect, where individuals defer judgment to the technology <cit.>, potentially undermining their reflective capacities <cit.>. This effect could be accentuated by the perceived authority or credibility of language based AI systems, which might attenuate the influence of cognitive reflection <cit.>. Additionally, the complexity or novelty of information such as new facts they have no knowledge of revealed in AI explanation could confuse or overwhelm users, reducing the effectiveness of their cognitive reflection in evaluating the information <cit.>. Lastly, it is also possible that individuals with high cognitive reflection are already near their maximum ability to discern truth from falsehood, resulting in a ceiling effect that leaves little room for AI generated explanations to induce cognitive reflection.

Trust in AI systems has in previous literature been highlighted as an important feature moderating the effects of explanations on people's beliefs <cit.>. Our results showed a significant effect of self-reported trust in AI systems on participants' belief ratings when getting deceptive classifications without explanations on true (β = -0.64, p = 0.0001) and false news headlines (β = 0.38, p = 0.0001). However, we did not find any increased effects of trust on participants' belief ratings when getting deceptive classifications with explanations on true (β = -0.11, p = 0.79) and false news headlines (β = -0.01, p = 0.59), indicating that trust in AI systems does not significantly increase the persuasion effects when receiving deceptive AI-generated explanations from just getting deceptive classifications.

Lastly, research has highlighted prior knowledge as a significant predictor of correctly identifying fake news <cit.>. However, it is unclear whether these effects extend to AI-generated deceptive explanations. To evaluate these effects, we had participants rate their own perception of their knowledge of a news headline after rating their belief in the news headline. When receiving AI classifications without explanations that were truthful (honest), self-reported prior knowledge was associated with increased beliefs in true news headlines (β = 0.44, p < 0.0001) and associated with decreased beliefs in false headlines (β = -0.11, p = 0.001). 

Conversely, when receiving deceptive classifications without explanations, prior knowledge was not found to have any significant effects on beliefs in false and true news headlines (β = -0.05, p = 0.54 and β = 0.02, p = 0.86, respectively). However, while there were no significant effects of deceptive AI-generated explanations on true news headlines (β = -0.19, p = 0.19), self-reported prior knowledge was associated with significantly increased beliefs in false news headlines when receiving deceptive AI-generated explanations (β = 0.18, p = 0.02). This suggests that individuals who report themselves as knowledgeable on a news headline might not necessarily be more resilient to deceptive AI classifications on true news headlines, and, in fact, might even be more susceptible to believing false news headlines when given deceptive AI explanations. One possible explanation for these results could be due to what is known as overconfidence bias <cit.>, where those who voice their perceived knowledge level to be high, could overestimate their ability to critically evaluate the AI system's outputs or to identify false information correctly. Future research should compare these results with an objective assessment of people's prior knowledge to more accurately detail the differences.
The complete results can be found in Table <ref> (CRT), Table <ref> (trust), Table <ref> (prior knowledge) in the Appendix.





 §.§ Deceptive AI-generated explanations that are logically invalid decrease people’s beliefs in false news headlines



Researchers have suggested that people's ability to identify logical flaws (or logical fallacies) could play an essential role in refuting misinformation <cit.>. In order to investigate the influence of logical validity of deceptive AI-generated explanations, we modeled the influence of logical validity of explanations on participants' belief rating using a linear regression, where logical validity is when the truth of the AI generated explanation necessarily implies the truth of the AI generated classification (logically valid) in comparison to where the truth of the classification is independent from the truth of the explanation (See Section <ref>). Limiting the data to only ratings where explanations were present, our results show that while logically invalid deceptive AI-generated explanations did not have any significant effects on participants' belief rating of true news headlines (β = -0.31, p = 0.13), logically invalid deceptive AI-generated explanations did significantly increase beliefs in false news headlines (β = -0.35, p = 0.02). This suggests that participants are more likely to reject deceptive explanations for false news headlines when the explanations are logically invalid. This demonstrates the potential for logical analysis and critical thinking skills to mitigate the influence of deceptive AI-generated explanations for false information. An overview of the results can be found in Table <ref> and Fig. <ref>.



 §.§ Limitations and Potential Generalization to Future AI Systems



As exemplified in this study, personal factors mediate the influence of explanations on humans beliefs. Extending upon these findings, research has shown that prior beliefs about AI systems can significantly influence how people integrate or reject AI-generated information <cit.>. Taking this into account, the impact of deceptive AI explanations might vary significantly across different cultural and contextual settings. Factors such as political climate, prevalent media literacy, and language around AI can influence how deceptive explanations are received and believed. For instance, research has shown that the choice of terminology significantly influences people's perceptions and reactions towards AI-generated content, with different terms leading to varying degrees of accuracy in identification and emotional response across different cultural contexts <cit.>. Moreover, while research has shown LLMs to be significantly more authoritative, persuasive, seemingly logically valid and even preferred over human-authored content, it is unclear to which extent the effects of deceptive explanations identified in this paper would transfer to human-authored deceptive explanations.



Second, our study utilized GPT-3, the most advanced language model available during the experimental period, to generate explanations. Although more sophisticated models have since surpassed its capabilities, our results indicate that even at the GPT-3 level, the model was able to produce deceptive explanations that adversely affected people's beliefs. It is important to recognize that while more advanced models generally exhibit less hallucination and incorporate more robust safeguards for the information they generate, researchers have repeatedly demonstrated that the safety measures implemented in large language models (LLMs) can be easily circumvented through techniques such as "jailbreaks" <cit.> or fine-tuning. For instance, an individual fine-tuned a readily available model on the HuggingFace platform using a dataset of posts from an online forum known for hosting harmful and offensive content, resulting in the generation of more than 30,000 posts on the platform <cit.>. As a result, even with better models becoming available, it likely that not only can these be exploited to generate misleading or deceptive explanations, even with safety measures in place, but they could even be misused to generate deceptive explanations far more persuasive and scalable than demonstrated here. As LLMs continue to advance and become more sophisticated, it is crucial for researchers and developers to remain vigilant in identifying and addressing potential vulnerabilities to ensure the responsible deployment of these technologies.



§ CONCLUSION

Our findings underscore the significant impact that deceptive AI-generated explanations can have on shaping public opinion and influencing individual beliefs. The ability of these explanations to not only present misinformation but also provide seemingly logical justifications makes them particularly potent tools for misinformation campaigns. This is especially concerning in the context of political discourse, scientific communication, and social media, where the rapid dissemination and acceptance of false information can have real-world consequences.

The persuasive power of deceptive explanations, as demonstrated in our study, highlights a critical vulnerability in the public's ability to discern truth from falsehood when interacting with AI-generated content. This is compounded by the finding that even individuals who consider themselves knowledgeable are not immune to the influence of these deceptive explanations. In fact, our results suggest that self-assessed knowledge may even increase susceptibility to believing false information when it is accompanied by a deceptive explanation. This could be due to a combination of overconfidence and the sophisticated nature of the explanations that make the misinformation seem credible.

Moreover, the role of logical validity in the effectiveness of deceptive explanations is particularly noteworthy. Our study found that logically invalid explanations were less effective in persuading individuals to believe false headlines, suggesting that enhancing critical thinking and logical reasoning skills could be a viable strategy to combat the influence of misinformation. This aligns with previous research emphasizing the importance of education in logical fallacies and critical thinking as tools for empowering individuals to better evaluate the information they encounter, particularly in digital environments where AI-generated content is prevalent.

While AI has the potential to bring about significant benefits, its capability to generate persuasive, deceptive explanations poses a serious risk to informational integrity and public trust. Our study highlights the urgent need for comprehensive strategies that address the dual aspects of enhancing public resilience against misinformation and ensuring responsible AI development and deployment. 





§ SUPPLEMENTARY INFORMATION


All data, including pre-registration, datasets, explanation prompts, and code generated and analyzed during the current study is available on GitHub  (<https://github.com/mitmedialab/deceptive-AI>), Zenodo (<https://zenodo.org/records/8172056>) and Research Box (<https://researchbox.org/1801   PEER_REVIEW_passcode=BDHVUP>).


unsrt  
 



§ EXTENDED DATA










 §.§ Additional Results




 §.§ Deceptive AI-generated Explanations Increase Beliefs in False Headlines and Decrease Beliefs in True Headlines

In order to gain insights into whether explanations lead to more accurate beliefs, we first compare how accurately participants rated true and false news headlines before (no feedback) and after getting deceptive AI-generated explanations. We coded accuracy by subtracting the belief ratings from the ground truth per rating per participant. Running an analysis of variance (ANOVA), we found that deceptive AI-generated explanations lead to a significantly lower accuracy than no feedback (14 percentage point difference, F(2, 10959) = -36, p < 0.0001, ANOVA Welch).

Breaking these results down into beliefs in true and false news headlines, we find that deceptive AI explanations significantly increase beliefs in false headlines and significantly decrease beliefs in true headlines (β = -2.44, p < 0.001 and β = 1.03, p < 0.001, respectively), suggesting that deceptive AI explanations significantly diminish people's ability to tell true news headlines from false news headlines. 



§ SEMANTIC AND SYNTACTIC FEATURES MODERATE THE INFLUENCE OF DECEPTIVE EXPLANATIONS

Beyond personal factors like cognitive reflection level, trust in AI, and prior knowledge, the manner in which information is structured and presented can greatly affect the way it is perceived and processed by the reader <cit.>. Notably, the influence of semantic and syntactic features on the effectiveness of persuasive communication has been acknowledged; aspects such as readability <cit.>, the amount of words used, the perceived truthfulness, and the perceived logical support of explanations for news headlines have the potential to shape a reader's beliefs and attitudes <cit.>. To understand what makes a deceptive explanation influence people to change their belief about a news headline, we conducted a post-hoc analysis of the semantic and syntactic structures of explanations such as their perceived truthfulness, perceived logical validity, the number of facts that was stated in the explanation, the explanation’s word count, and how easy the explanation was to read.

Our results revealed significant correlations between semantic structures of deceptive explanations and changes in belief about true and false news headlines. In particular, our results show that when an AI system gives deceptive explanations that are likely to be perceived as logically supporting a false news headline being true, people are more likely to update their initial belief and believe that the false news headline is true (β = 1.26, p < 0.001). Conversely, when the deceptive explanation are likely to be perceived as logically supporting that true news headline is false, people are more likely to believe that the true headline is false (β = -1.36, p < 0.01) than when it is not likely to be perceived as logically supporting it being false. We did not find any significant correlations between the perceived truthfulness of deceptive explanations and beliefs in true and false news headlines in our model (β = 0.81, p < 0.30 and β = -0.79, p < 0.10, respectively). The complete linear model with the semantic results can be found in Table <ref> in the Appendix.

For syntactic structures, our results show that when an AI system gives deceptive explanations on true headlines, stating that they are false, the longer the explanation is (i.e. its word count), the more likely people are to believe the true news headline than from before receiving an explanation (β = 0.03, p = 0.004). This indicates that deceptive explanations that are too long might have the opposite effects, and actually make people realize that a true headline is true, hence increasing their rating from before getting a deceptive explanation. Moreover, we also found that how easy a deceptive explanation was to read also significantly correlated with belief changes in true news headlines from before receiving an explanation. In particular, the easier an explanation was to read correlated with a significantly lower belief in true headlines (β = -0.01, p = 0.05). We did not find any significant correlations between word count of deceptive explanations and belief in false headlines (β = 0.006, p = 0.61), or reading ease of deceptive explanations and belief in false headlines (β = 0.0008, p = 0.88). The complete linear model with the syntactic results can be found in Table <ref>.

If we look at the mean perceived logical validity for deceptive explanations on true vs. false headlines, we found that deceptive explanations on true news headlines generally have a higher perceived logical validity than false headlines. This indicates that the LLM might be better at fabricating deceptive explanations with high perceived logical validity for true news headlines than false. Previous studies have found that false news is typically perceived as more novel than true news <cit.>. This might explain the model's capability of constructing more persuasive deceptive explanations for true news than false news, given that novel information might have less established sources to draw from and distort. Therefore, for making people belief true news less, simple tricks like making the deceptive explanations longer or easier to read might might not work as well for deceptive explanations on false news.




  §.§.§ The impact of Deceptive AI explanations on belief vary depending on the context and type of statements




To understand the generalizability of deceptive AI-generated explanations to other domains, we explore their impact on the belief rating of trivia statements. Overall, we find substantial differences between deceptive AI-explanations on trivia statements compared to news headlines. Compared to deceptive AI-generated explanations on false news headlines, our model does not reveal any significant effects of deceptive AI-generated explanations on participants' beliefs in false trivia statements (β = 0.01, p = 0.94, not pre-registered), nor on participants' beliefs about true trivia statements (β = -0.28, p = 0.19, not pre-registered). We do, however, still find significant effects of deceptive classifications without explanations on true and false trivia statements (β = -2.57, p = 0.0001 and β = 1.19, p = 0.0001, respectively, not pre-registered).

For the effects of logical validity on belief ratings, we also do not see any significant effects on false trivia statements (β -0.26, p = 0.50, not pre-registered) compared to the significance we found for news headlines.

Comparing the average self-reported prior knowledge across trivia items and news headlines, we find that participants on average are less knowledgeable on false trivia (mean = 2.25) statements than false news headlines (mean = 2.71) (β = -0.38, p = 0.003, ad hoc), whereas we found no significant differences in knowledge ratings between true trivia (mean = 2.28) and true news headlines (mean = 2.28) (β = 0.01, p = 0.97, ad hoc)). See Table <ref> in the Appendix for the full table. For differences in belief rating associated with self-reported prior knowledge, we found, similar to news headlines, no significant effects for deceptive classifications without explanations on true and false trivia (β = -0.08, p = 0.33 and β = 0.06, p = 0.23, respectively), nor was self-reported prior knowledge associated with differences in beliefs in true trivia statements when getting deceptive AI-generated explanations (β = -0.16, p = 0.33). However, contrary to news headlines, self-reported prior knowledge was not significantly associated with increases in beliefs for deceptive AI-generated explanations on false trivia statements (β = 0.16, p = 0.12). This difference could potentially be due to the significantly lower knowledge people report to have on false trivia statements.

While we observed that deceptive AI explanations have a stronger impact on people's belief compared to deceptive classifications in news headlines, we did not observe the same effect with trivia statements with the deceptive explanation not having significantly stronger convincing power than the deceptive classification by itself. Our hypothesis is that people might be more easily convinced about trivia items than news headlines because they are less high-stakes. Moreover, news headlines are also typically more personal and relevant to individuals, thus requiring additional information for people to be persuaded. On the other hand, trivia statements, which individuals have less personal investment in, might not require explanations to be believed. 

Furthermore, our findings indicate that when individuals receive deceptive AI explanations, they become less adept at discerning when they believe they possess prior knowledge. However, we did not find this to be the case for trivia statements. This might be due to our finding that people have more self-reported prior knowledge on news headlines than trivia statements. Overall, the observed differences between news headlines and trivia statement emphasize that the impact of deceptive AI explanations on belief may vary depending on the context and type of statements. 




 §.§ Results Graphs






 §.§ List of Stimuli






plaintableList of Stimuli shows the news headlines and trivia statements used in the experiment.  
[pages=1-,scale=0.8,pagecommand=]figures/appendix/stimuli.pdf



 §.§ Statistical Analysis


































