Paper Title: MambaCapsule: Towards Transparent Cardiac Disease Diagnosis with Electrocardiography Using Mamba Capsule Network.


1
.001


 





mode = title]MambaCapsule: Towards Transparent Cardiac Disease Diagnosis 
with Electrocardiography Using Mamba Capsule Network                      




























1]Yinlong Xu[
orcid=0009-0008-1987-7562
]






22321336@zju.edu.cn








[1]
    organization=College of Computer Science & Technology, Zhejiang University,
    city=Hangzhou,
    country=China
[2]
    organization=Department of Gastroenterology, First Hospital of Quanzhou Affiliated to Fujian Medical University,
    city=Quanzhou,
    country=China
[3]
    organization=School of Public Health, Zhejiang University,
    city=Hangzhou,
    country=China
[4]
    organization=State Key Laboratory of Transvascular Implantation Devices of The Second Affiliated Hospital and Liangzhu Laboratory, Zhejiang University School of Medicine,
    city=Hangzhou,
    country=China


[5]
    organization=Innovation Institute for Artificial Intelligence in Medicine, Zhejiang University,
    city=Hangzhou,
    country=China
    
[6]
    organization=School of Medicine, Stanford University,
    country=USA
    
[7]
    organization=School of Computer Engineering and Science, Shanghai University,
    city=Shanghai,
    country=China

[8]
    organization=College of Future Industry, Gachon University,
    country=Korea


  

2]Xiaoqiang Liu
liuxiaoqianghusina@163.com


1]Zitai Kong
kongzitai@zju.edu.cn

3]Yixuan Wu
wyx_chloe@zju.edu.cn

4]Yue Wang
ywang2022@zju.edu.cn


6]Yingzhou Lu

lyz66@stanford.edu

7,8]Honghao Gao
gaohonghao@shu.edu.cn
honghaogao@gachon.ac.kr

3]Jian Wu
wujian2000@zju.edu.cn

5]Hongxia Xu
Einstein@zju.edu.cn
[1]



[cor1]Corresponding author

















§ ABSTRACT 
Cardiac arrhythmia, a condition characterized by irregular heartbeats, often serves as an early indication of various heart ailments. With the advent of deep learning, numerous innovative models have been introduced for diagnosing arrhythmias using Electrocardiogram (ECG) signals. However, recent studies  solely focus on the performance of models, neglecting the interpretation of their results. This leads to a considerable lack of transparency, posing a significant risk in the actual diagnostic process. To solve this problem, this paper introduces MambaCapsule, a deep neural networks for ECG arrhythmias classification, which increases the  explainability of the model while enhancing the accuracy.Our model utilizes Mamba for feature extraction and Capsule networks for prediction, providing not only a confidence score but also signal features. Akin to the processing mechanism of human brain, the model learns signal features and their relationship between them by reconstructing ECG signals in the predicted selection. The model evaluation was conducted on MIT-BIH and PTB dataset, following the AAMI standard. MambaCapsule has achieved a total accuracy of 99.54% and 99.59% on the test sets respectively. 
These results demonstrate the promising performance under the standard test protocol.


















Mamba Capsule ECG Classification Explainable AI



[
    [
    August 5, 2024
==================





§ INTRODUCTION

Cardiac arrhythmia refers to the disruption of the cardiac rhythm caused by abnormal cardiac electrical activity <cit.>. For the reason that arrhythmia serves as an indicator to advent heart consequence, its accurate detection and classification are imperative for mitigating potential risks and complications. Electrocardiogram(ECG) is widely used for recording the electrical cardiac activity<cit.>. It provides critical information about the cardiac function and health by measuring the cardiac electrical signals during each heartbeat cycle, which makes it a reliable tool for diagnosing heart health.

Recent researches have shown great advantages of deep learning in arrhythmia classification<cit.>, leveraging the information contained in ECG signals. The development of powerful and effective deep learning methods has the potential to significantly improve the accuracy and efficiency of arrhythmia diagnosis, and ultimately leads to better patient outcomes.  

Although the numerous previous models have made substantial progress in arrhythmia classification, most of them solely focus on the performance  neglecting the interpretation of their results, which leads to a considerable lack of transparency and poses a low reliability in the actual diagnostic process<cit.>.

To alleviate this limitation, we proposed a encoder-decoder based neural network called MambaCapsule in this paper. Inspired by the mechanism of brain processing<cit.>, which tells what features exactly the eyes see and tells the reason why the brain chooses to classify one item to this label not the other one, we novelly designed our model to implement both processes by using the main architecture for prediction and a reconstruct part for explaination. Our evaluation was carried on MIT-BIH dataset<cit.> and PTB<cit.> dataset, reached 99.54% and 99.59% accuracy. By applying a reconstruct network, we could intuitively know what the model sees and why it makes that decision.

The main contributions of this paper are as follows:


=0pt

  ∙ We introduced a novel encoder-decoder based network architecture based on Mamba<cit.> and Capsule networks<cit.>, changing the form of the output from the probability sequence to capsule vector sequence, which contains both the classification and feature attribute.

  ∙ We applied a downstream network to reconstruct the ECG signal. The reconstructed signal can be used to explain what our model learns and the reason why our model makes its prediction.

  ∙ We applied a fusion states SSM architecture in process of feature extraction. The evaluation indicators show the feature extracted has considerable advantages on the MIT-BIH and PTB datasets.
  



§ RELATED WORK



 §.§ Mamba Nerual Network

Mamba has made excellent performance in numerous traditional fields since it was proposed by <cit.>. Different from traditional and the most popular structure Transformer, the advantages of Mamba are mainly reflected in the computational efficiency and the ability to deal with long sequences. For the reason above, many researches have been carried to replace traditional components with Mamba. <cit.> proposed Graph-Mamba, which improved the remote context modeling ability of attention mechanism by integrating Mamba block and node selection mechanism, and solved the problems of high computing cost and limited scalability of attention mechanism. 
<cit.> proposes a new model-based meta-reinforcement learning method, achieving higher returns and better sample efficiency (up to 15x) with little need for hyper-parameter tuning.
<cit.> proposes a multi-modal large language model VL-Mamba based on state space model, demonstrating competitive performance on a variety of multi-modal benchmark tasks and the great potential of state-space models in multi-modal learning.

Mamba also show its advantages in vision tasks <cit.>. <cit.> proposed a generic vision backbone with bidirectional Mamba blocks (Vim) to solve the location sensitivity and global context requirements of visual data. Vim performs well in various tasks, with significant improvements in compute and memory efficiency. <cit.> introduced VideoMamba into video understanding task. Its linear-complexity operator enables efficient long-term
modeling, which is crucial for high-resolution long video understanding.




 §.§ Capsule Neural Network

Capsule network was first proposed in a 2D handwritten digital image classification task and has excellent performance in the view-invariance <cit.>. Later on, it was widely applied to point cloud <cit.>, image processing <cit.>, and tabular learning <cit.>, graph learning <cit.>. Specifically, <cit.> proposed a multi-scale capsule network that is more robust and efficient for feature representation in image classification, demonstrating a competitive performance on FashionMNIST
and CIFAR10 datasets. <cit.> introduce
a capsule network that can detect various kinds of attacks. The model uses many fewer parameters than traditional convolutional neural networks with similar performance.

Capsule neural network also has a intrinsic advantage for explaining its results. For instance, <cit.> produced a sentiment capsule to identify the informative logic unit and the sentiment based representations in user-item level for rating prediction. By introducing capsules in different layers, <cit.> demonstrated the possibility of transforming deep networks into transparent networks. This work helps make DNNs' internal processes more understandable and provides the ability to explain their decisions.




 §.§ ECG based Cardiac Disease Diagnosis

Numerous researches have been carried in the field of ECG signal classification and the most challenging part of it is to extract the features from the signal. The feature extraction methods have experienced from the manual features and automatic features. Manually designed features extracting usually applied by traditional ML methods like Support Vetor Machine <cit.> and Random Forest <cit.>.
<cit.> proposed an effective methods to extract the abnormal beat eigenvectors of low-dimensional ECG and classify them using support vector machine. <cit.> proposed a ML model using multi-domain features combined with least-square support vector machine (LS-SVM), which extracts the time and frequency domain features and selects the most relevant ones.

With the success of deep learning in vision and NLP fields, its ability has been admitted and has been applied to many other fields, including the ECG signal classification <cit.>. 
<cit.> proposed a 1D residual CNN based on deep residual networks with a 31-layer one-dimensional residual convolutional neural network combined with 2-lead ECG signals for the classification.
<cit.> proposed a novel model combining residual networks, compressed activation blocks and bidirectional short-duration memory networks. It was evaluated on multiple databases and the results showed that the framework performed well especially for a small number of categories.

Although these methods have their own way to explain the process of the prediction, their lack of accuracy and inadequacy of features extracting hinder their practical applications. Some efforts also focused on the class imbalance problom on ECG datasets. For instance, <cit.> introduced a deep learning model called ECGTransForm ,which combines multi-scale convolution, channel calibration module and two-way Transformer mechanism to improve the ability to capture spatio-temporal features in ECG data and solves class imbalance problem by introducing context-aware loss function. However, these methods were employed through simple data reinforcement using statistical methods, the reinforced data benefits to model performance but has no specific physiological information through the process of the reinforcement.
 


§ METHODS




 §.§ Mamba

State space models (SSMs) <cit.> are defined as 1D-time-invariant systems, which serve as the predecessor of Mamba. These systems aim to map a 1D input sequence x(t)∈ℝ^L to a 1D output sequence y(t)∈ℝ^L through a hidden state h(t)∈ℝ^N, which can be represented as the following equations:

    h(t) = Ah(t)+Bx(t)
       y(t)=Ch(t)


where A∈ℝ^N× N, B∈ℝ^N× L, C∈ℝ^L× N. Considering that deep neural networks operate in discrete space, researchers applied a zero-order preservation on the matrix A and B, turning the equations to the form as:

    h_t=Ah_t-1+Bx_t
       y_t=Ch_t


Where A=exp(△ A) and B=(△ A)^-1(exp(△ A)-I)· (△ B) for△ represents the time step in the continuous space. SSMs also code the remote dependency by initializing matrix A as HiPPO<cit.> and allow training in the form of a convolution kernel to accelerate the training process. However, the matrix A, B and C do not change whatever the input data are, leading to problem of low sensitivity. To solve the problem, Mamba applies a selective method to SSMs, which makes the matrix B and C and △ adapt themself based on the input as follows:

    B = s_B(x)
       C = s_C(x)
       △ = τ_△(Parameter+s_A(x))


Where s_B, s_C and s_A are linear projections, and τ_△ is a SoftPlus function. The matrix A does not depend on the input directly, it is influenced by the △ in the process of the zero-order preservation, making all main parameter inputs adaptable<cit.>.






 §.§ Capsule Networks

The whole network contains three parts: the convolution network, the capsule network and the dynamic routing. The difference between the capsule network and the traditional neural network is that the unit of the neural network is a scalar neuron x∈ℝ, while the unit of the capsule network is a capsule, which is a vector x∈ℝ^N.

In a capsule network, there are many capsules in one layer, and each capsule contains two types of information: the given vector's orientation and the vector's length as the probability of the capsule, the next capsule first applies a linear transformation of the previous capsule output as:

    û_j|i,L+1=W_ij, Lu_i, L


where u_i,L∈ℝ^D is the i-th output capsule at layer L, W_ij,L∈ R^D̂× D serves as the transformation matrix and the û_j|i, L+1∈ℝ^D̂ represents the contributes from the parent i-th capsule to the j-th one at layer L+1. The model then gets the weight of each contributes through a clustering like routing algorithm. At present, the popular routing algorithms include dynamic routing algorithm and self-routing algorithm. This paper adopts dynamic routing algorithm, which obtains the final weight coefficient through several rounds of iteration. In each iterations, the weight coefficient c is:

    c_ij=exp(b_ij)/∑_k exp(b_ik)


In the first iteration the b_ij is initialized to 0 and will be updated at the end of each iteration. Then the coefficient is used to calculate the temp capsule:

    s_j=∑_i c_ijû_j|i


Where s_j∈ℝ^D̂ serves as a temp result for an iteration, the capsule network designs a special activation function to transform the length of the capsule into [0-1] as:

    v_j = ||s_j||^2/1+||s_j||^2s_j/||s_j||


Where v_j∈ℝ^D̂ serves as the real output capsule in each iteration. The equation <ref> separates the long and short length of the capsules in the dim of 0 to 1, making the long capsule close to 1 and short capsule close to 0. Then the b_ij is updated as:

    b_ij = b_ij + v_jû_j|i


The equation <ref> serves as an clustering-like algorithm for after each iteration the b_ij grows bigger in capsule whose position is  closest to the others. And after the setting iterations have done, the v_j will serve as the output of the capsule networks.



 §.§ Model Architecture


In this section, we present a classification model based on Mamba and Capsule networks. The overall framework, as shown in Figure <ref>, consists of three main parts: the feature network, the capsule network and the reconstruction network.  Before be fed into the feature network, the input ECG signal
is firstly up-projected to contain scope knowledge not a signal sequence, the input shape changes from x∈ℝ^batch, length to x̂∈ℝ^batch, length, dim.

Then N Mamba Layers constitute a pipeline to extract features from the x̂ and each Layer contains a Mamba Block and a Layer Norm. Previous work typically apply a single state Mamba as the encoder backbone, however, these fails to extract wider and narrower features. To solve this problem, we propose a fusion states SSM to construct the Mamba Block. For the i-th Block, it first applies a 1D-convolution on the upstream i-1-th feature sequence x_i, which does not change the dimension of the sequence. Then the sequence will be handled by m different SSM Blocks. The different SSM Blocks focus on different scales of time step and have different convolution shape to obtain more abundant and wider features. Then the features will be down sampled and then linked together to add the x_i in the same shape. The whole process can be formulated as:

    x̂_i = Conv_1D(x_i)
        u_j = DownSample_j(SSM_j(x̂_i))
        x_i+1 = Concat(∑_j^mu_j) + x_i


Where u_j ∈ℝ^batch, length, dim/mand the m is the factor of the dim to assure x_i+1∈ℝ^batch, length, dim.

The feature network extracts the signal features and sends the features to the capsule network. The capsule network outputs it in the form of capsules in the end. 




The loss of our model consists of two parts, the marginal loss of focusing on model output and the label, and the reconstruct loss focusing on the reconstructed signal and the original signal.

Since our output is in the form of a capsule network, we must first take the geometric length of the output as the probability of the classification label, and then the difference between the set positive label threshold and negative label threshold is made. The final form of margin loss function is as follows:


    Loss_margin =    ∑_k T_kmax(0, m^+-||v_k||)^2+
       λ(1-T_k)max(0, ||v_k||-m^-)^2


Where v_k∈ℝ^D is the k-th output capsule represents one class of the classification, T_k=1 if the label of the input is k, and m^+=0.9, m^-=0.1 at the beginning. λ is used to prevent the lengths of vectors from shrinking too much, which is set to 0.5 in the beginning.

Reconstruct loss needs to choose the label with the highest probability or we choose it for other reasons. Then we put the output into the reconstruct model, which is a 3 fully connection layers model with Relu activation. After that, calculate the MSEloss of the reconstructed signal output of the reconstruct model and the original input signal, the reconstruct loss is formulated as follows:

    x̂_reconstruct = Model(v_choose)
        Loss_reconstruct=MSELoss(x̂_reconstruct, x_origin)




§ MODEL SETUP AND EVALUATION



 §.§ Training Strategy

Since the output probability density of the capsule network is not the same as that of the traditional neural classification network, and the bias can directly affect the output, we adjusted bias by cosine schedule during the training process, so that the threshold of bias kept increasing. At the same time, due to the slow training speed of capsule network in the initial stage of training, the cosine annealing algorithm after linear increase is adopted to change the learning rate. The changes of bias and learning rate are shown in the figure.<ref>





 §.§ Performance Metrics

To evaluate the model, we applied five performance indexes: accuracy(ACC), sensitivity (SEN), F1-score(F1), precision (PPV) and specificity (SPEC). These indexes are defined as follows:


    ACC=TP+TN/TP+TN+FP+FN
       SEN=TP/TP+FN
       F1=2×ACC× SEN/ACC+SEN
       PPV=TP/TP+FP
       SPEC=TN/TN+FP


In these formulas, TP represents the number of true positives (correctly labeled positive); TN represents the number of true negatives (correctly labeled negative); FP represents the number of false positives (incorrectly labeled positive); and FN represents the number of false negatives (incorrectly labeled negative).


 

 §.§ Dataset

 Our training and testing datasets are preprocessed in the same routine and are randomly splited in a ratio of 0.8. The detailed description of the dataset is in Table<ref>:


  §.§.§ MIT-BIH Arrhythmia Dataset

The MIT-BIH arrhythmia dataset <cit.> is widely used for arrhythmia classification in recent researches. The dataset contains 48 half-hour two-channel ECG recordings digitized at 11 bit resolution in 10 mV range and a rate of 360 samples per second per channel. Two cardiologists annotated each record separately (about 110,000 annotations in total). MIT-BIH has annotated 41 categories which can be converted to five-classes classification problems following the ANSL/AAMI EC57: 2012 (The Association for the Advancement of Medical Instrumentation) standard. These distinctive classes are: Normal Sinus Rhythm (N), Supraventricular Premature or Ectopic Beat (S), Ventricular Premature or Ectopic Beat (V), Fusion of Ventricular and Normal Beat (F), Unknown Beats (Q). Some researches were carried on three or four classes due to imbalanced sample size, making the comprehensive evaluation difficult. In this paper, we chose to adapt to all the five classes.



  §.§.§ PTB Diagnostic ECG Database

The PTB diagnostic ECG database is a popular ECG recording database collected by the National Metrology Institute of Germany<cit.>. The dataset contains 549 records from 290 subjects (aged 17 to 87, mean 57.2; 209 men, mean age 55.5, and 81 women, mean age 61.6; ages were not recorded for 1 female and 14 male subjects). Each subject is represented by one to five records. Each record includes 15 simultaneously measured signals: the conventional 12 leads (i, ii, iii, avr, avl, avf, v1, v2, v3, v4, v5, v6) together with the 3 Frank lead ECGs (vx, vy, vz). Each signal is digitized at 1000 samples per second, with 16 bit resolution over a range of ± 16.384 mV. The PTB dataset consists of two main classes, i.e., (1) Normal (N), which represents normal ECG recordings, and (2) the Myocardial Infarction (M), which represents ECG recordings that exhibit signs of myocardial infarction, indicating the presence of a heart attack.









§ EXPERIMENTAL RESULTS



 §.§ Performance Evaluation


Many previous work focused on only four or less classes performance, aiming to reduce the perplexity of the problem or concentrate on specific arrhythmia types. However, it leads to great difficulties to compare the ability and universality of different models. In our work, we operate our evaluation on five full classes.



Table <ref> shows the confusion matrix of our best model prediction. And the detailed evaluations on each class of the MIT-BIH dataset are presented in Table<ref>. The result shows our model has excellent ability of discrimination, especially in class N, V and Q the Sensitive, Precision and Specificity reached almost 100%, which can be attribute the large training sample size. The performance of Supraventricular Premature or
Ectopic Beat (S) is not so outstanding, which can be the result of lack of training samples and the Fusion of Ven-tricular. The average performance of Normal Beat (F) can be the reason its original features resemble to the Normal class (N).





Table <ref> and Table <ref> also show our model great performance on PTB dataset. The confusion matrix indicates that our model predicts nearly all of positive and Negative samples, with an macro-avg accuracy of 99.59%.















The performance of our model is also compared with several previous studies, which is presented in Table <ref>, presenting comparative advantages over the other architectures. And the performance of each label can be seen in Table <ref>, which can be observed from the result that our model has considerable advantages compared with others in majority classes.



 §.§ Ablation Study

The ablation experiment was carried out to test the effectiveness of our model structure. Since Mamba was mainly used as the encoder and capsule networks as the decoder in our model, the ablation experiment was carried out in these two aspects on MIT-BIH dataset.



  §.§.§ Encoder Structure

This experiment aims to compare the performance of different model structures, including the number of stacked mamba layers, the number of mamba states, the steps per layer, whether to adopt the layernorm and dropout structures, and the use of different models as encoders.

In order to assess the number of mamba states and steps in each layer, we adjusted these parameters while keeping the encoder layers fixed at 4. The experimental results depicted in the figure<ref>demonstrate a significant impact of varying numbers of states and steps on the outcomes. Moreover, a layer composed of diverse mamba blocks outperforms a single layer due to its ability to focus on different ECG data features across various time scales. Consequently, this wider temporal coverage achieved by combined blocks leads to enhanced performance. However, if the states are too wide it will miss significant information.

To assess the impact of encoder layers on model performance, we held the parameters of each layer constant and varied the number of encoder layers from 2 to 8 while considering whether to implement layer normalization and dropout. The corresponding results are depicted in the figure <ref>. From our analysis of training time and memory usage for a four-layer encoder, it is evident that the metric increases with additional layers; however, model performance does not improve proportionally with increased depth. In fact, the optimal performance is achieved at six layers due to limitations in information capacity exceeding what can be learned from available data as well as increased computational demands associated with larger models.

In order to evaluate the effect of different types of encoder, we have also tried to use resnet, transformer and cnn as encoders. The experimental results are shown in the figure<ref>. The results show that mamba has the best effect when it is used as encoder.





  §.§.§ Decoder Structure

The purpose of this experiment was to compare the changes in the model training and the performance caused by using capsule instead of traditional decoder.

We tried to replace the capsule network with MLP after freezing encoder parameters. Experiments showed that there was no significant difference between the capsule network and MLP under the same encoder with F1-score 93.50 compared to 93.62, but the training time of capsule network was longer than that of MLP. This is determined by the routing structure of the capsule network and the number of parameters. 



 §.§ Explainable Study

The interpretability of this paper is unique from other recent papers on ECG diagnosis, mainly through the reconstruction of ECG signals in different ways to explain what knowledge the model has learned on MIT-BIH dataset. For the reason of the limitation of the memory, we only consider P-wave and R-wave in our reconstruction, discarding the T-wave which is far from the center of the signal.



  §.§.§ Feature Relation Mining


When processing the ECG graphs, the human brain not only attends to focus on individual wave peaks but also considers the interrelationship between them. The view invariance of capsule network aligns remarkably well with this characteristic, which is why we selected it as the decoder. In our experiment, we conducted forward and backward disturbance on the input data to observe the disturbance reconstruction of ECG signals using our model and DAE model. The experimental results are depicted in the figure <ref>, where left and right shift represent forward and backward shift respectively. Repeated reconstructions of ECG signals by MambaCapsule exhibit remarkable consistency, indicating that Mambacapsule has learned not only the intrinsic features but also their interrelationships. Conversely, this characteristic is absent in the reconstructed ECG signal produced by DAE model.



  §.§.§ Cardiac Disease Manifestation Type Inspection


When human brain categorize an item, it can discern the reasons for its classification and differentiation. Due to the inherent output characteristics of Capsule network, the output not only encompasses the probability density of each category but also encapsulates the distinctive features associated with that particular category. Consequently, by reconstructing ECG signals from unified inputs belonging to different categories, we are able to precisely comprehend why the model deems a certain category more probable. The experimental results are depicted in Figure <ref>. In Figure <ref>, both the ECG signal and model prediction are labeled as N, where P, R wave can be observed clearly and can be distinguished it represents a normal signal. However in Figure <ref>, which is reconstructed by the label S and the length of the capsule is adjusted to be equal to the capsule of label N. The false reconstruction represents a vanish in P-wave, which is a symbol of Supraventricular Premature or
Ectopic Beat. The reconstructions of N and P labels demonstrate an ablity of knowledge explaination and thus, each capsule can be seen as the information collection of different labels and the length of it is how much the information is concluded in the signal.



§ CONCLUSION


In this work, we have proposed an effective-while-explainable solution to ECG classification by introducing MambaCapsule, which emphasizes the capability of multi-scale Mamba in feature extracting and the interpretability of the form of Capsule network. By applying evaluation and reconstruction, the result of our model reveals great advantages in both classification score and mechanism explainablity. However, it's necessary to realize the limitation of our model, a the long training time and big memory occupied. In our future work, we aim to address these challenges by applying pre-trained parameters and optimize routing structure in Capsule network. As an automatic heart rhythm abnormality detection system, our model can offer doctors and patients accurate and reliable results, which have a wide range of application prospects.






cas-model2-names









 Paper Title: Cost-Effective Hallucination Detection for LLMs.









Cost-Effective Hallucination Detection for LLMs
    Simon Valentin^1[1]
        Jinmiao Fu^2[1]
        Gianluca Detommaso^3These authors contributed equally to this research. 
        Shaoyuan Xu^2
        Giovanni Zappella^1
        Bryan Wang^2
    
    ^1Amazon Web Services, Berlin, Germany 

    

    ^2Amazon, Seattle, USA 

    

    ^2Helsing, Berlin, Germany 

    


=======================================================================================================================================================================================================



§ ABSTRACT 
Large language models (LLMs)
can be prone to hallucinations — generating unreliable outputs that are unfaithful to their inputs, external facts or internally inconsistent.
In this work, we address several challenges for post-hoc hallucination detection in production settings.
Our pipeline for hallucination detection entails: first, producing a confidence score representing the likelihood that a generated answer is a hallucination; second, calibrating the score conditional on attributes of the inputs and candidate response; finally, performing detection by thresholding the calibrated score. We benchmark a variety of state-of-the-art scoring methods on different datasets, encompassing question answering, fact checking, and summarization tasks. We employ diverse LLMs to ensure a comprehensive assessment of performance.
We show that calibrating individual scoring methods is critical for ensuring risk-aware downstream decision making. 
Based on findings that no individual score performs best in all situations, we propose a multi-scoring framework, which combines different scores and achieves top performance across all datasets.
We further introduce cost-effective 
multi-scoring,
which can match or even outperform more expensive detection methods, while significantly reducing computational overhead. 





§ INTRODUCTION

Despite their impressive capabilities,  large language models (LLMs) can be prone to generating hallucinations — undesirable outputs that are incorrect, unfaithful, or inconsistent with respect to the inputs (or the output itself) <cit.>. 
These unreliable behaviors pose significant risks for adopting LLMs in real-world applications. 
Challenges in detecting hallucinations lie, among other things, in hallucinations taking different forms, being context-dependent and sometimes being in conflict with other desirable properties of generated text <cit.>. 
Hallucinations may be harmless in some contexts, but can be undesired or potentially dangerous in other applications (e.g., erroneous medical advice).
Detecting and quantifying hallucination risk is thus a critical capability to enable safe applications of LLMs and improve generated outputs.



Prior work has proposed various approaches for detecting and mitigating hallucinations in LLM-generated outputs, including verifying faithfulness to inputs <cit.>, assessing internal coherence <cit.>, consulting external knowledge sources <cit.>, and quantifying model uncertainty <cit.>.
However, deploying these methods in production settings is far from trivial due to several challenges:
First, there is limited comparative evaluation illuminating how different detection methods perform. 
Second, existing approaches for detecting hallucinations differ greatly in their computational demands, and guidelines are lacking on cost-effectiveness trade-offs to inform method selection for real-world applications with constraints.
Third, hallucination detection in the real world often requires careful consideration of risks and false positive/negative trade-offs, requiring methods to provide well-calibrated probability scores.
Fourth, many applications of LLMs take the form of calls to black-box APIs, which sometimes requires methods to employ workarounds to assess the model's confidence in its generated output. 

In this work, we provide a framework for detecting hallucinations in the outputs of any LLM in a model-agnostic manner.
Our approach relies on quantifying the probability that a generated answer is a hallucination. 
After computing initial scores, we employ state-of-the-art calibration techniques to provide calibrated probabilities of the generation containing a hallucination, which can subsequently be used for decision making or other downstream tasks. 
We evaluate a variety of scores proposed in the literature for hallucination detection on several metrics, across multiple datasets encompassing question answering, fact checking, and summarization tasks. We employ a range of different LLMs to ensure a comprehensive assessment of performance. 

Critically, as we show that no single score performs best across all datasets, 
we introduce
multi-scoring, 
a simple way of combining multiple calibrated scores, which achieves superior performance compared to 
any individual score alone.
Furthermore, 
we 
propose cost-effective multi-scoring, which 
finds the subset of best-performing scores for any fixed cost budget, and combines them in a multi-scoring fashion. Our empirical demonstrations reveal that cost-effective multi-scoring not only matches but often surpasses the performance of individual scores that incur significantly higher costs. Consequently, our proposed method achieves superior hallucination detection outcomes while maintaining a substantially lower cost footprint.


We summarize our contributions as follows: 1. We benchmark a variety of hallucination detection methods across the literature on several metrics, over different datasets and LLMs. 2. We introduce multi-scoring, a novel approach that aggregates multiple complementary scores and outperforms individual scores. 3. We further propose cost-effective multi-scoring, which optimally balances detection performance and computational constraints. 





§ DETECTING LLM HALLUCINATIONS




 §.§ Formalizing Hallucination Detection

We study the problem of quantifying the probability that a generated output from a language model contains hallucinations.
More formally, let 
x represent an input token sequence to a language model 𝒢, and let 
z represent a generated output text sequence from the model. 
We define a binary random variable y ∈{0,1} that indicates whether z is a “permissible” output (y=1) or contains a “hallucination” (y=0). 

Our goal is to develop a scoring function to model the probability that a given output text contains a hallucination conditioned on the input.[Note that while we are interested in whether the entire output contains any hallucination, one could easily adapt the methods at different granularities, such as the sentence or phrase level.]
This is critical, as in real-world scenarios, we need to set risk-aware thresholds, balancing false positive/negative rates to accommodate the production requirement.
Having access to scores allows us to flexibly set the threshold.
Conceptually, the key reason to model this probabilistically is that there is inherent uncertainty in determining whether a given text contains a hallucination or not. 
Even human raters may disagree on the assessment, based on their own knowledge and definition.
Some key contributors to this epistemic uncertainty are: First, that no system has complete world knowledge to perfectly assess factual correctness. 
Second, that there is ambiguity in whether something is a hallucination. 
Finally, any automatic scoring model may make occasional errors, so probabilistic scores reflect confidence. 
Therefore, while conditional on x and z the true hallucination label y is fixed, our estimate of y remains uncertain. The probabilistic score thus reflects this epistemic uncertainty — the degree of belief that z contains a hallucination given the available knowledge:
p(y = 0 |x, z; ψ), where ψ denotes parameters of the scoring function. We refer to this conditional probability function as the hallucination score, denoted s_ψ(x, z).
The hallucination score can then be applied to downstream tasks, including making risk-aware binary decisions.

We discuss concrete instances of hallucination scores in the following subsection.
Generally, the form of s_ψ can vary between hallucination detection approaches. 
In particular, s_ψ may depend on multiple candidate texts generated for the same input, as implemented by a number of hallucination detection methods proposed in the literature.
We denote K candidate generated outputs as Z = [z_1,...,z_K]. 
Then s_ψ(x, Z) could quantify inconsistencies within texts in Z using different metrics ψ, as discussed below.



 §.§ Scoring Methods

Many hallucination detection methods proposed in the literature make use of LLMs to “judge” the output of an LLM (either the same or a different one). 
For clarity, we thus distinguish between generator and detector LLMs. 
The generator is defined as the model used to generate the original response. 
The detector LLM is the model used to score a generated text for the presence of hallucinations. 
In general, the generator and detector LLMs may coincide. 
However, it may be more desirable to use different LLMs in some scenarios, e.g., when computational cost is a greater concern, where a smaller LLM may be used to judge outputs of a more expensive LLM, or when using hallucination scoring methods that require white-/grey-box access, while the generator is black-box. 

Today, many interactions with LLMs take the form of API-calls.
Typically, only the output tokens are returned with no access to the logits of the predicted tokens,  treating the LLM effectively as a black box. 
Sometimes, inference parameters may be accessible, allowing for setting different temperature (among others) values, thereby providing some (grey-box) access to the model. 
Generally, hallucination detection methods vary in their required model access, ranging from black-box APIs to full white-box access to the model weights.
That is, while some methods only require token-level outputs, other methods may need access to the logits of the generated tokens, or some control over inference parameters like temperature. 

In our experiments, we evaluate a comprehensive set of hallucination scoring methods.
Generally, we do not make any assumptions about the generator LLM being white-, grey- or black-box. 
We divide methods into single-generation methods, which require only one generated output, and multi-generation methods which are based on multiple alternative generations. 





 §.§ Single-generation

We first provide an overview over a set of hallucination detection methods that are based on scoring the hallucination from a single given generated output. 



  
Inverse Perplexity This method provides a prominent instance of using a model's logits over output tokens to score the confidence in the output. Computed as the inverse of the exponentiated average negative log-likelihood of the LLM's response <cit.>,

    Perplexity^-1(W) = exp(1/N∑_i=1^N log p(w_i|w_i-1,...,w_1)),

inverse perplexity thus provides a sequence-length normalized expression of the model's confidence. 
Here, the model may either be the generator LLM or a different detector LLM.
Using a different LLM from the generator as a plug-in estimator implies that the confidence estimate is detached from the generator. 
If our generator provides access to the model's logits we can directly use them to score hallucinations.
Alternatively, we can use another LLM to generate the logits of output tokens. 



  
P(True) This method works by prompting an LLM whether an answer is correct or not, then making use of the logits of the next token <cit.>. 
Given a prompt like the following: 


We compute P(True) by first taking the softmax over the first generated output token's logits, 
and then normalizing across the “True” and “False” tokens.[Note that alternatively, we can prompt the LLM to bind the True/False tokens to symbols like response options A or B <cit.>. However, this did not change results in our experiments. Further, if the detector LLM provides only black-box access, one can sample the corresponding tokens and use the empirical distribution to approximate, but thereby incurring a higher computational cost.]
That is, let 𝐳^(1) = (z^(1)_1, …, z^(1)_|V|) be the logit vector over the vocabulary V for the first generated token when asking the LLM to evaluate whether the answer is correct. We then take the softmax over these logits, 
p(w |𝐳^(1)) = exp(z^(1)_w)/∑_w' ∈ Vexp(z^(1)_w').
Let w_True, w_False∈ V be the “True” and “False” token ids. Then we can compute P(True) as:

    P(True|𝐳^(1)) = p(w_True|𝐳^(1))/p(w_True|𝐳^(1)) + p(w_False|𝐳^(1)).


In addition to checking whether the response is correct, we provide variations of P(True), where we check whether the output contradicts the input in P(InputContradict), the output contradicts itself in P(SelfContradict), or the output contradicts generally established facts in P(FactContradict), with the followings prompts: 








  
NLI Text Classification Natural language inference (NLI) models provide an alternative to assess the correctness of the model output. As hallucination detection requires checking for contradictions, we compute the score as 1-S_contradict, where S_contradict refers to the softmax probability of the output conflicting with the question. Specifically, we use a DeBERTa model fine-tuned on an MNLI task <cit.> as the underlying NLI model, which we refer to as NLI (DeBERTa).




  
Verbalized Probabilities Instead of analyzing a model's logits, scores are elicited by asking an LLM to provide a confidence verbatim, i.e., by generating tokens that indicate the numerical confidence <cit.> with the following prompt:

We note that moderated LLMs accessed through APIs may in practice decline to answer if the context does not provide sufficient information, rather than assigning a low confidence score.  



 §.§ Multi-generation
 Multi-generation methods are based on quantifying the consistency across multiple generated outputs from the generator LLM. 
This follows the notion of white-/grey-box uncertainty quantification via logits that if the LLM is confident in its response, multiple generated responses will probably be alike and contain compatible facts. 
Conversely, for fabricated information, sampled responses are more likely to differ and contradict one another. Crucially, this rests on the assumption that the model's confidence is calibrated, as we discuss and evaluate below.



  
SelfCheckGPT This method exploits a DeBERTa NLI model to assess whether the answer A_0 is consistent with K alternative generated answers A_1, ..., A_K <cit.>. Each sentence of A_0 is compared against the full set of answers A_1, ..., A_K. A consistency score per sentence is obtained via an NLI model, then the final score is obtained by averaging over the sentence-wise scores. 



  
Similarity Degree This method is based on computing pairwise similarities between the multiple responses using NLI models, and then quantifying uncertainty based on the distribution of similarities <cit.>. Here, we compute the pairwise similarities between responses via the contradict class probability of an NLI model and construct a degree matrix, where each diagonal element corresponds to the total (sum) similarity of the corresponding response to all other responses. We use the degree of the candidate response A_0 as the confidence estimate, following favourable results in prior work <cit.>, but note that other metrics have also been proposed in the original paper.



  
NeMO Guardrails: Hallucination Rail 
Unlike SelfCheckGPT or Similarity Degree, the score here is computed via one LLM call, where we check for agreement between the concatenated K additional generations and the candidate output and not averaged over sentences or computed on pairs of responses. Specifically, we use the softmax probability normalized over yes/no tokens using the following prompt:





 §.§ Calibration

Initial hallucination scores may not be properly calibrated, which can lead to poor downstream decisions. Seminal work by Guo et al. <cit.> has demonstrated that neural models tend to be miscalibrated, particularly in the form of models being overconfident. 
Later work, focusing on language models, has confirmed this across a wide range of tasks, models and datasets <cit.>. 
Notably, there is also research which demonstrates that given particular prompts and in-distribution tasks, LLMs can be well-calibrated <cit.>. However, this has been shown to be brittle and dependent on context <cit.>. 

Formally, our score outputs probabilities p_ψ(ŷ=0|x, z) that z contains a hallucination, parameterized by ψ. 
The score is calibrated if, for any probability level p ∈ [0,1], the average observed frequency of hallucinations matches the predicted probability:

    𝔼[y | p_ψ(ŷ=0|x, z) = p] = p


A naive approach for obtaining scores would be to compute the model's probabilities marginally, ignoring the context/question x and generated response z and directly estimating p(y=1). 
However, this does not allow for conditioning on individual inputs to obtain calibrated probabilities p(y=1 |x, z) for specific x, and z, which is, however, impossible to guarantee <cit.>. 
Common calibration methods include temperature scaling (Platt scaling) of logit outputs <cit.>, isotonic regression <cit.> or histogram binning <cit.>, which operate marginally, and can thereby not account for different confidence levels for different inputs (e.g., with an LLM being more confident on certain domains than others).

An alternative is to partition the inputs into G groups and compute calibration separately for each group g∈ G. However, this assumes the groups are disjoint and does not handle inputs belonging to multiple groups, which is often necessary <cit.>.
More advanced calibration methods, such as multicalibration, which we use in this work, allow defining G potentially overlapping groups  <cit.>. 
Prior work has shown the effectiveness of modern calibration techniques in scoring LLM confidence, albeit in a white-box setting <cit.>.
We describe our calibration approach in the experimental section.



 §.§ Multi-Scoring: Combining Scores

Different scoring methods capture different aspects of hallucinations, e.g., incorrect, non-factual, non-grounded, irrelevant, inconsistent with other answers, etc. As a result, some scores may work better on some data or for some specific models, and worse on others. Therefore, we design a multi-scoring method to combine the complementary information from individual scores into a single, strong predictor.

Denote each available score by s_n, for n=1, … ,N. To obtain an aggregated score, we run a logistic regression using as features the concatenations of the logit of each score, i.e. [logit(s_n(x, z)), …, logit(s_N(x, z)) ], and the labels of the calibration dataset as target variables. [Note that we also conducted experiments with alternative models, such as XGBoost <cit.> and Random Forest <cit.>, but results did not show significant improvements, so we opted for logistic regression for simplicity.]





 §.§ Cost-Effective Multi-Scoring

Scoring methods based on multiple generations incur the cost of additional generations as well as the cost of checking their consistency. 
Similarly, multi-scoring can be a viable choice when there are only few LLM generations to check for hallucinations, but incurs considerable computational cost, especially when using multi-generation methods. 
To avoid prohibitive computational costs in practice, we propose cost-effective multi-scoring, where we set a fixed computational budget and compute the best performing combination of scores that stay within the specified budget. 

Given an input text x and generated text z, we have N scoring functions s_1(x, z), ..., s_N(x, z) with associated costs c_1, ..., c_N (e.g., number of generations required). We are given a total computational budget B. Our goal is to find the optimal subset of scores S^* ⊆{1,...,N} that maximizes detection performance while staying within budget B:


    S^∗ = _S: S ⊆{1, …, N}ℒ(f(s_i(x, z))_i ∈ S)   s.t.  ∑_i ∈ S c_i ≤ B


where ℒ measures loss on a validation set. This is a constrained optimization problem over subsets of scoring functions. When B=min_i c_i, it reduces to selecting the single best score at the lowest cost. When B=∑_i c_i, it recovers full multi-scoring. In between, the optimal subset S^* provides the best trade-off between performance and cost. 
We note that in general, this problem is computationally challenging, given the exponential runtime complexity. 
However, given that there are only generally a small set of potential scores (N ∼ 10) and the logistic regression is very fast to fit, computing all combinations takes only 1.8 seconds on a single Intel Xeon processor (3.1 GHz), iterating over all candidate solutions sequentially. 
When this approach is not feasible, some exemplary alternatives include classic greedy forward-selection methods or regularising the model via an L1 penalty while scaling the regularisation term to accommodate score cost. 
Overall, cost-effective multi-scoring allows for flexibly balancing multiple scores under computational constraints.



  
Quantifying Cost
While the cost c_i of scoring function s_i is presented abstractly above, accurately quantifying computational cost can be challenging in practice. 
The actual runtime per method is a reasonable proxy, however the runtime depends on model architecture, hardware acceleration, batching, etc. 
One approach is to directly benchmark each scoring function's average runtime empirically on the target hardware. 
However, this overlooks nuances like caching effects and ignores runtime variability due to implementational differences.
To simplify our analysis, and as LLM calls are generally much more expensive than calling smaller NLI models based on parameter size, we leverage the number of LLM calls required per method as a proxy. See Table <ref> for an overview. 
If more precision is desired, we suggest empirically running different scores and computing their computational cost in the actual application, as the cost will depend on the precise setup and a range of factors.




§ EXPERIMENTS




 §.§ Experimental Setup



  §.§.§ Datasets




  
TriviaQA TriviaQA <cit.> is a commonly used factual open-respon­se QA dataset. Originally set up as a reading comprehension task, it is today often used without context as a closed-book (free-recall) task <cit.>. 
We use the validation fold, containing 17944 question-answer pairs. We use  <cit.> to generate candidate answers. To decide whether a given response should be labelled as positive or negative, we check whether the correct answer is contained in the generated answer after removing formatting, following the original evaluation script <cit.>.  



  
FEVER The closed-response Fact Extraction and VERification dataset <cit.> provides a comprehensive benchmark of factual hallucination detection. We take the test fold, containing 14027 labelled examples of source documents, claims and whether they are  ,  or contain . For the purpose of hallucination detection, we look at all claims that are either  or .



  
HaluEval We include the hallucination detection dataset HaluEval <cit.> and use the summarization task, which contains 10000 labeled examples of source documents, summaries and labels for whether the summary contains hallucinations. 



  
BIG-bench It provides an evaluation of LLMs on a diverse set of tasks <cit.>. We select 11 tasks suitable for hallucination detection, leading a total of 8664 labelled examples from the validation fold. 



  §.§.§ Metrics

Practical applications of hallucination detection require calibrated scores, but often also binary decisions over whether a given output is hallucinated or not. 
To this end we report the Brier score <cit.>, binary decision metrics F1 score and accuracy. 



  §.§.§ LLMs

We conduct experiments with  <cit.>,  <cit.>,  <cit.> and  <cit.>.
All models are used with default configurations via HuggingFace Transformers <cit.>.



  §.§.§ Calibration

The calibration step is performed via the following multicalibration approach, using the Fortuna library <cit.>:
To obtain groups, we compute embeddings of the input text x and the generated response z, such that our embedding is e[ (x), (z) ]. We obtain embeddings from Universal AnglE Embedding <cit.>, which is the SOTA in the MTEB benchmark <cit.> at the time of writing. 
We subsequently reduce the dimension of x via UMAP <cit.> and perform soft-clustering via Gaussian Mixture Models, as a simple off-the-shelf algorithm (fitting the number of cluster components via BIC <cit.>).
The calibration error is measured for each group g∈ G separately. 
The model is then iteratively patched on the group with the largest error until the calibration error drops below a threshold for all groups.[Note that alternative approaches are possible <cit.>.] 
This provably converges to a calibrated model with theoretical guarantees <cit.>.
We fit the calibrator to a random calibration fold of 80% of the data and report only held-out test results. 
For all binary predictions, we set the threshold to the 50th percentile to not impose preferences over false true/negative rates, but note that in practice any threshold could be applied. 



 §.§ Individual Scoring Methods

Table <ref> presents the results of different hallucination detection methods on all datasets. Multi-generation methods (SelfCheckGPT-NLI, HallucinationRail and SimilarityDegree) are employed only for TriviaQA, since the latter provides the true answer to each question, which can be compared to the alternative generated answers to assess their agreement. In contrast, for HaluEval, BIG-Bench and FEVER, the true answer is not provided. Instead, we compare the given candidate answer from the dataset against the provided binary label of correctness.
If we exclude Multi-Scoring methods, multi-generation methods such as SelfCheckGPT-NLI and SimilarityDegree (with 10 generations) achieve the best performance on TriviaQA, P(True) is the best on HaluEval and BIG-Bench, and P(InputContradict) performs best on FEVER. 
Thus, we find that there is no single best scoring method across all datasets. 
This is likely because different scoring methods capture different aspects of hallucination, supporting the notion that hallucination is a multi-faceted concept. 
Multi-generation methods measure hallucination based on the consistency of the generator LLM's responses and on the ability of the comparison method to identify whether multiple alternative responses are in agreement. 
We find that SelfCheckGPT-NLI performs best in our experiments in comparison to SimilarityScore and HallucinationRail.
While all of these methods follow similar approaches, there are subtle differences in how they assess the consistency between different responses.
More generally, as we discuss below, multi-generation methods are appropriate only if one can assume that there is exactly one correct response, but can fail otherwise. 

Other methods are based on different notions of hallucination, such as NLI (DeBERTa) measuring the entailment of the response given the input. 
Interestingly, variants of P(True) can be appropriate for detecting different kinds of hallucinations. 
P(True) explicitly asks the evaluator LLM to check whether the response is correct. 
In some contexts, the evaluator LLM may have the ability to directly evaluate this. 
However, in other situations, as we see with the FEVER dataset, other methods such as P(InputContradict) can be more appropriate, if we are trying to directly target a specific form of hallucination, which may not be subsumed under a general “correct or not?” prompt. 
Other applied scenarios may target even more different (or more specific) kinds of hallucinations, though the variants we include in this work are designed to cover the space in a reasonable manner. 
Similarly, while the datasets included in these experiments were selected to cover a wide range of different kinds of hallucinations, real-world applications may show even new kinds of hallucinations, for which no public datasets are available.
Overall, these findings highlight the need for our multi-scoring method which can absorb the strength of each individual method and can be easily applied to a concrete hallucination detection setting while only requiring a relatively small amount of labeled data. 

Overall, neither the inverse perplexity score nor the NLI (DeBERTa) scores emerge as the best scores for any of the datasets we consider. While they may add information that can be exploited in (cost-effective) multi-scoring, as reported below, individually they perform worse than some of the other scores. 






 §.§ Multi-Scoring

To evaluate the proposed multi-scoring method, we conduct experiments on all datasets combining the scores via logistic regression. 
As presented in Table <ref>, the multi-scoring ensemble achieves an F1 score of 0.9106 on TriviaQA, outperforming the best individual F1 score of 0.8614 (achieved by SelfCheckGPT-NLI).
Similarly, Brier and Accuracy also show the highest performance for multi-scoring.
For HaluEval, we also find that multi-scoring outperforms the best individual score (P(True)) in all metrics. 
BIG-Bench shows more nuance, as multi-scoring outperforms the best individual score (P(True)) on Brier and Accuracy, but not on F1. 
This reflects the fact that the metrics capture different properties, and practical considerations may require a decision as to which metric should be prioritized in a given application.
For FEVER, multi-score again outperforms the best individual score (P(InputContradict)) in all metrics. 

Thus we see that combining scores performs better than the top performing individual scores, showing that combinations of different scoring signals complement each other and can boost performance. 
This demonstrates that combining complementary signals enables more robust hallucination detection, while balancing the strength of each scoring method.
Crucially, while some settings may benefit from combinations of different signals (such as when trying to detect different kinds of hallucinations in a given generated output), in other settings the data-driven selection of an informative signal may be sufficient (such as when trying to detect a more narrowly defined notion of hallucination). 
These scenarios are covered by our use of multi-scoring, which allows for arriving at optimally combined hallucination scores (or binary decisions) for a given application.
Meanwhile, as computational cost can be a concern with scaling LLM applications, it may not be desirable to always use a full ensemble of scores, and we would rather compute the most performant score at a fixed computational cost.



 §.§ Cost-effective Multi-Scoring

Shown in Table <ref>, cost-effective multi-scoring methods are also amongst the top performers, but at considerably lower cost compared to multi-score. We see that at cost C=1, measured as the number of LLM calls (but see our discussion about measuring cost above), we recover the best individual scores, which varies across each dataset. As we increase the budget, the cost-effective multi-scores converge to the performance of the multi-score itself. Notice that in several instances, cost-effective multi-score with C=2 already performs as good as multi-score itself. 
For TriviaQA, we see that performance increases on all metrics as we increase the computational budget, with cost-effective multi-score at C=5 outperforming SelfCheckGPT-NLI as the most performant individual scores at half the computational cost in all metrics but Brier.
HaluEval shows no improvement beyond a budget of C=2, which is likely due to the kind of hallucination to detect being more narrowly defined and well-capture by a small number of signals, thereby not benefiting from additional scores. 
For BIG-Bench, as discussed for multi-score, we recover the best individual score at C=1, and match multi-score at higher budgets.  
FEVER results indicate that, again, we recover the best individual score at budget C=1, and can already recover the full multi-score at a budget of C=3.

We now take a closer look at cost-effect multi-scoring when including multi-generation scoring methods.
The overall cost budget is varied over the entire range. For each budget, we solve the constrained optimization in Eq. <ref> to find the optimal subset S^* of scores.
We compare the hallucination detection F1 score achieved by the cost-effective ensemble versus individual scoring functions and the full ensemble with all scores. Figure <ref> shows the results for TriviaQA for responses generated via . With a minimal budget of B=1, cost-effective selection recovers the best single method, as expected. As the budget increases, it selectively adds more expensive functions, gradually improving F1, though the gains are marginal at higher budgets per unit cost. At the maximum budget, it recovers the unconstrained full ensemble performance. In between, cost-effective multi-scoring incorporates both less and more and expensive methods to maximize detection within the computational constraints.



These results demonstrate that the proposed cost-effective multi-scoring approach can intelligently balance the trade-off between computational expense and hallucination detection effectiveness. It outperforms individual scoring functions and makes selective use of more costly scores to maximize detection performance under a fixed computational budget.

A key consideration in practice would be whether to include potentially more costly multi-generation methods, such as SelfCheckG­PT-NLI. 
In particular as implied by the results presented in Table <ref>, we may be interested in reducing the number of required multiple generations while matching performance at lower computational costs. 
To this end, we compare SelfCheckGPT (as the single best performing method for TriviaQA) alone with SelfCheckGPT combined with P(True) at different numbers of additional responses generated via . 
Here, we count the evaluation of P(True) as one additional LLM call, just as generating one additional output from the generator LLM. 
As presented in Figure <ref>, at one additional generation, SelfCheckGPT (in the degenerate case of only one generated output) combined with P(True) thus recovers P(True). 
As the number of additional generations increases, the combination quickly outperforms SelfCheck alone. 
In particular, we observe that cost-effective multi-scoring with 3 LLM calls is already as good as SelfCheckGPT with 9 LLM calls in our experiment.
This highlights how we can save costs by combining multi-generation methods with other scores while requiring fewer additional generated responses from the generator LLM than if we wanted to achieve the same performance with multi-generation methods alone. 





 §.§ Exploration of Relationships between Scores


As discussed above, different scoring methods can target different kinds of hallucations. 
To explore this empirically, we here explore their relationships via Spearman rank correlations. 
As presented in Figure <ref>, calibrated scores across all different individual scores are positively correlated. 
Meanwhile, the magnitude of their correlations is smaller than would perhaps be  expected if one were to consider hallucinations as a uniform phenomenon. 
At the same time, we can see certain “clusters” of more strongly inter-correlated scores emerge.
For example, multi-generation methods including SelfCheckGPT, HallucinationRail and SimilarityDegree, which check the consistency across multiple generations, show comparably high correlations. 
P(True) and P(True) Verbalised and P(InputContradict) emerge as a similarly correlated cluster of stronger correlations. 
Overall, this supports the idea that different scores can capture distinct information, and the need to empirically select appropriate (combinations of) scores for a given application, as we propose with cost-effective multi-scoring.






 §.§ Experiments across Different LLMs

Most of the scoring methods considered in this work rely on detector LLMs to compute scores for hallucination detection. 
In Table <ref> of Appendix <ref>, thus present additional data for different LLMs, namely ,  and .
Overall, we see that hallucination detection performance is correlated with performance on general LLM benchmarks[E.g., <https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard>]. 
Thus, more overall capable LLMs are likely to perform better at hallucination detection than less capable ones.
However, concrete applications may need to take the inference cost of LLMs into account, where one may not always be able to use the most expensive model. 
We note that cost-effective multi-scoring could here also make use of scores computed via LLMs with different computational costs to find cost-effective combinations. 



 §.§ The Importance of Calibration

To analyze the impact of calibration, we conduct an ablation study by evaluating model performance with and without calibrating the individual scores.
Table <ref> in Appendix <ref> indicates that model performance clearly drops in most of the datasets and metrics without calibration. This demonstrates the benefit of calibration for more accurate and risk-aware downstream applications.



 §.§ Exploring Multi-Generation Assumptions





Our results indicate that methods based on the uncertainty among multiple generations can provide strong signals when such multiple generations are available. 
However, such methods can also suffer from problems in practice. 
Methods based on multiple generations make use of the uncertainty of the generator, that is the distribution over generations z given the input x, for model 𝒢, i.e., p_𝒢(z|x) and the consistency between actual sampled generations z. 
We are interested in classifying whether the candidate output contains a hallucination or not, 
that is estimating and making use of p(y |x, z). 
The generating model's uncertainty p_𝒢(z|x) can be a useful proxy for p(y |x, z), but only under particular conditions. 

On a technical level, sampling multiple answers requires access to the generator's temperature parameter, as temperatures of zero collapse the generations to a single response, as is mentioned, e.g., in <cit.>. 
Therefore, these methods are technically grey-box models, as they require some access to the model's inference parameters. 

More conceptually, the generator's uncertainty (or confidence <cit.>) over p_𝒢(z|x) reflects uncertainty over generated tokens. 
All recently proposed multi-scoring methods we are aware of are based on the idea of scoring the consistency across multiple generations using model-based metrics. 
However, self-consistency across multiple generations is neither a necessary nor sufficient criterion for a hallucination to be present. 
Sufficiency is not given as a model may consistently provide an incorrect response. 
Self-consistency across multiple generations is also not necessary, as many tasks allow for multiple hallucination-free answers that are contradictory to each other. 

As a relatively harmless example, in tasking a model to generate cooking recipes for lunch, the model may generate, among other things, a recipe for a salad and a recipe for a soup. Clearly, the steps in preparing these dishes contain contradictory information, while otherwise being free of hallucinations in themselves. 
Thus, if there is more than one correct response, the self-consistency assessment in multi-generation methods may falsely score an output as likely to be hallucinated. 
As there are numerous situations where there exist multiple correct responses, assuming that there is only one could lead to worse LLM responses, also via a loss of diversity.

Practically, methods based on multiple generations can be costly due to added computational overhead, as we have seen above.
Finally, these methods are based on the assumption that LLMs are calibrated. 
As such, multi-generation methods do not allow for detecting cases where the generator LLM is confident yet wrong. 









§ CONCLUSION


In this work, we compared a comprehensive set of scoring methods to provide calibrated probability scores for the presence of hallucinations in generated LLM outputs.
Overall, we have observed that no single hallucination detection score performs best across all datasets.
Our experiments showed that combinations of scores, as suggested with multi-scoring, can effectively combine complementary signals to yield higher hallucination detection performance than any individual score.
Further, we demonstrate that cost-effective multi-scoring can find the highest performing scores at a given computational budget.

More generally, our findings support the notion that hallucinations can be rather multi-faceted than present a uniform phenomenon.
Thus, detecting hallucinations may require different methods.
In concrete settings, one may be interested in even more fine-grained detection of particular types of hallucinations, such as in specific domains like code generation <cit.>.

The approach outlined in this work requires only a small amount of labeled data to calibrate hallucination scores and combine scores via (cost-effective) multi-scoring.
However, it is important to acknowledge the limitations of the current work. Future research could explore the effectiveness of this approach on more diverse datasets, investigate alternative scoring methods, or extend the methodology to multi-modal models. 
While improvements in LLM performance can be expected to also lower their rate of occurrence, hallucinations are unlikely to go away completely <cit.>.
Thus, even as models improve further, detecting hallucinations is likely to remain relevant for applying LLMs in a reliable and trustworthy manner in future.

In summary, our work presents a promising approach for detecting hallucinations in LLM outputs by combining multiple scoring methods in a cost-effective way, offering a pathway towards more reliable and trustworthy language models that can be applied in real-world settings with more confidence.


















§ ADDITIONAL RESULTS














 Paper Title: Characterizing User Archetypes and Discussions on Scored.co.








Characterizing user archetypes and discussions on Scored.co 





Andrea Failla1,2*,
Salvatore Citraro2,
Giulio Rossetti2, Francesco Cauteruccio3






1 Department of Computer Science, University of Pisa, Pisa I56127, Italy


2 Institute of Information Science and Technologies “A. Faedo” (ISTI), National Research Council (CNR), Pisa I56127, Italy


3 DIEM, University of Salerno, Fisciano I84084, Italy


























*andrea.failla@phd.unipi.it






§ ABSTRACT

In recent years, the proliferation of social platforms has drastically transformed the way individuals interact, organize, and share information. 
In this scenario, we experience an unprecedented increase in the scale and complexity of interactions and, at the same time, little to no research about some fringe social platforms. 
In this paper, we present a multi-dimensional framework for characterizing nodes and hyperedges in social hypernetworks, with a focus on the understudied alt-right platform Scored.co. 
Our approach integrates the possibility of studying higher-order interactions, thanks to the hypernetwork representation, and various node features such as user activity, sentiment, and toxicity, with the aim to define distinct user archetypes and understand their roles within the network. 
Utilizing a comprehensive dataset from Scored.co, we analyze the dynamics of these archetypes over time and explore their interactions and influence within the community. 
The framework's versatility allows for detailed analysis of both individual user behaviors and broader social structures. 
Our findings highlight the importance of higher-order interactions in understanding social dynamics, offering new insights into the roles and behaviors that emerge in complex online environments.













§ INTRODUCTION



In the last decades, the proliferation of social platforms and the opportunities tied to each of them have drastically transformed the way individuals interact, share information, and form communities. This unprecedented growth in the scale and complexity of online interactions is largely due to social networks like X (formerly known as Twitter), discussion fora like Reddit, and even more niche platforms, like the now-defunct Voat, known for their explorations of censorship-free models from the point of view of specific political groups <cit.>.

Hence, understanding user behavior within these platforms has become a crucial area of research in computational social science <cit.>. 
By studying how users interact with each other and with others' content, researchers can uncover patterns of influence spread, opinion formation, and creation and dissolution of communities <cit.>.

These insights are not only academically interesting but also have practical implications for designing better social platforms, enhancing user experience, and mitigating issues such as misinformation and online harassment.
Moreover, understanding these behaviors can aid in developing strategies for marketing, public health campaigns, and even political mobilization, making this research area highly relevant in multiple domains. Furthermore, all of these aspects are even more critical in the analysis of understudied platforms. Indeed, platforms such as Scored.co <cit.>[<https://scored.co>], which have not been extensively explored, offer unique social dynamics that may differ significantly from “mainstream” platforms. Investigating these less-explored environments can uncover novel patterns of user behavior and mechanisms of information dissemination. 
Also, understudied platforms often cater to niche communities or specialized interests, thus providing insights into subcultures and micro-level social dynamics that might not be straightforwardly visible on larger social platforms.

Understanding these dynamics becomes particularly important when addressing the general challenges inherent in studying complex social platforms.
Among all of these, a significant one is the identification of roles that users play within complex social structures. Indeed, role identification is crucial for various applications, including targeted information dissemination <cit.> and community detection <cit.>. In traditional social network analysis, roles are often determined based on patterns of dyadic connectivity and interactions. 
Nevertheless, in a discussion on a social platform, two users can interact directly, e.g., by sharing comments, but also indirectly, i.e., through other users.
Hence, it is fair to assume that taking into account group dynamics and higher-order interactions could help in a more reliable identification of the roles played by users.

A promising approach to encapsulate group interactions is defined by the concept of a social hypernetwork <cit.>, which allows for the modeling of relationships among groups of nodes, generally called hyperedges. Such a representation is particularly useful for studying social platforms from a higher-order point of view. 

Despite its importance and the existing methodologies designed for pairwise graphs, current literature about the problem of role identification lacks comprehensive approaches for social hypernetworks.

To address this gap, in this work, we formally introduce an approach for defining higher-order roles in a social hypernetwork and characterize higher-order interactions.

We define the concept of an archetype, a general characterization of higher-order roles that serves as a “template” for identifying nodes representative of specific roles within the context of hyperedges. These archetypes provide a structured way to categorize and understand the diverse roles users play in group interactions. Then, we propose a general hyperedge characterization function, which allows one to characterize a hyperedge w.r.t. features of the nodes contained in it.

To validate the effectiveness of our proposed framework, we conduct an exhaustive experimental campaign on the understudied social platform Scored.co, which offers a rich, yet understudied dataset for analyzing user interactions in a hypernetwork context, making it an ideal testbed for our framework. Our experiments demonstrate the utility of our approach in characterizing user roles within social hypernetworks.


Summarizing, the main contributions of this work are as follows:


    
  * We define the concept of archetype, that is, a high-level description of user roles that serves as a “template” characterizing nodes representatives;
    
  * We introduce an approach to characterize higher-order entities, such as hyperedges, taking into account exhibited features and their higher-order dynamics;
    
  * We design an exhaustive experimental evaluation in which we assess the effectiveness of our framework on the understudied social platform Scored.co. Notably, this is also the first work studying the structural properties of Scored.co.
    


The outline of this paper is as follows: in the sec:related-literature section, we provide a detailed overview of related literature. In the sec:framework section, we provide the formalization of our proposed approach by defining the concept of archetype first, and the hyperedge characterization function after. In the sec:experiments section, we illustrate the experimental campaign we carried out to evaluate the effectiveness of our framework. Finally, in the sec:conclusion section, we draw our conclusions and delineate some future works.



§ RELATED LITERATURE






  §.§.§ Hypernetworks and Role Identification


The identification of roles within social networks stands as a fundamental aspect for understanding the dynamics and function of these complex systems <cit.>. In traditional social network analysis, roles are often determined based on patterns of connectivity that provide insights into the influence, responsibilities, and grouping of individuals within the network <cit.>. This analysis is crucial for applications ranging from the characterization of polarization <cit.> and user behaviors <cit.> to the analysis of biological networks <cit.> and animal populations <cit.>. In different cases, role identification is also framed within the task of discovering influential users in a social network. In light of this, we refer the interested reader to the comprehensive survey proposed in <cit.>.

Nevertheless, extending role identification into hypernetwork scenarios poses many challenges. Unlike traditional networks, hypernetworks consist of interactions, represented by hyperedges, that can encompass multiple nodes simultaneously. 
Higher-order interaction models can better represent complex real-world systems  <cit.>; however, the complexity of higher-order interactions can introduce difficulties in accurately defining and identifying roles, as traditional centrality metrics and methodologies (e.g., community detection algorithms) often fall short in capturing the multidimensional nature of hyperedges <cit.>.
Despite the relevance of such a task, to the best of our knowledge, there are no existing papers that explicitly discuss role identification within the context of hypernetworks.

Given this gap in the literature, the rest of the section will concentrate on providing an overview of the methodologies and findings from the conventional settings in dyadic interactions, which may serve as a baseline for future exploration into hypernetwork-specific role identification. 




  §.§.§ Role Identification in Pairwise Networks


The general task of role identification consists in capturing the multifaceted characteristics of a node's role in a pairwise network. The work in <cit.> identifies roles in undirected, unweighted networks by evaluating node importance using correlated indicators like degree and centrality measures. 
A node's role is then determined by comparing its indicator relationships to statistical correlations within the overall network. While this work shares the task of role identification with ours, the process does not involve group-based indicators leveraging the concept of “groups" such as communities or subgraphs.

The work in <cit.> frames the node role identification tasks as a node classification approach based on random walks.



Similarly, framed within a classification tasks, the authors in <cit.> focus on the so-called “answer-person” role on Reddit, characterized by users who predominantly respond to questions posed by others, with minimal engagement in broader discussions.


Both the works in <cit.> and <cit.> share a similar aim with ours. Nevertheless, they rely on supervised approaches on pairwise graphs. Also, the methods might not be straightforwardly generalized to higher-order interactions. 

The approach proposed in <cit.> focuses on the identification of node roles in dynamic social networks as a sequence of different types of activities, by leveraging pattern subgraphs and sequence diagrams.

Doing so, the authors are able to capture roles such as the “gossipmongers", i.e., users who replicate every received message at least three times. The work in <cit.> can be considered as orthogonal to ours, as an hypergraph mining approach could also be applied in our context to enhance role identification.
The authors in <cit.> outline a process for identifying user roles in Enterprise Social Networks (ESNs), using a mix of design science research and data mining, involving data collection, preparation, and evaluation, with user roles identified through statistical analysis, including PCA and clustering.
The work in <cit.> shares a similar focus with ours, although using a different methodology.
The identification of emergent leadership roles is addressed in <cit.>, where the authors examine collaboration patterns between teams and propose the so-called “leadership index", a combination of centrality measures including closeness and betweenness. 

In <cit.>, the authors propose a mixed-methods methodology for user role identification, focusing on dynamic user profiling. 
Several types of special users are defined and identified to support information dissemination.
The work in <cit.> proposes an approach based on computational methods and questionnaire-based evaluation to quantitatively describe user features. Such work parallels ours in the sense that it considers the network's features, too.

In <cit.>, the authors study community and user stereotypes in Reddit. Here, with the term stereotype, the authors refer to the tendency to classify people into groups and to associate each group with a general idea or a label. The authors propose a rule-based approach based on different quantitative views of the data, and define author stereotypes on the basis of two orthogonal taxonomies, namely, the number of posts, and the number of comments of an author. While the work in <cit.> and ours share some similarities, the employed methodologies are substantially different. The former defines stereotypes based on a single quantitative measure called score, which is inherent to the social platform itself. Instead, our approach proposes a general framework in which the characterization of users and the identification of their role are both defined by their feature-rich, higher-order surroundings. 
In <cit.>, the authors identify five distinct social roles in a specific community on Reddit. Among them, we cite the “knowledge broker”, i.e., a member who introduces knowledge to the community by sharing links, and the “translator”, i.e., a member who contributes academic knowledge. While the contribution in <cit.> is notable and similar to ours for considering various aspects of user characterization, these roles are specific and applicable only to the analyzed community.


Finally, it is worth mentioning embedding-based methodologies <cit.>, which could potentially be adapted to hypernetwork scenarios. For instance, in <cit.>, the authors introduce a technique that incorporates node attributes to generate embeddings that capture the similarities based on neighborhood structures. Instead, in <cit.>, the authors leverage both node and structural embeddings to detect nodes impersonating social bots. Indeed, node embeddings based on structural properties and exhibited attributes attracted substantial interest in the last years <cit.>. While these methods use embedding techniques that effectively summarize and exploit the structural information within traditional networks, their application to hypernetworks might not be straightforward.

Our approach diverges from embedding-based methods in the sense that it is specifically tailored to the unique features of nodes within hypernetworks, such as their exhibited features w.r.t. the hyperedges containing them.


By focusing on this, without relying on an embedding, our methodology seeks to offer a more direct and fitting analysis of node roles.




  §.§.§ “Understudied" Social Platforms


We conclude this bird's-eye view of the related literature by focusing on the last contribution of our paper, that is the study of an understudied platform. Indeed, the reasoning provided in the above discussion of related work is also valid for this aspect. In fact, there is a moderate number of works on understudied platforms, and generally these are limited to present a dataset collection about the considered platform <cit.>. For instance, a large-scale dataset of the social platform Scored.co has been presented in <cit.>, in which the authors studied aspects such as posting activity and user characterization, as well as the phenomenon of user migration from other platforms. A recent social platform, called Bluesky, has been studied in <cit.>. In the former work, the authors presented a comprehensive study of the social structure of the platform, as well as posting activity and content analysis. Also, the complete post history of over 4M users has been released. Similarly, the latter work studied users' political leaning and ideological polarization on BlueSky while also presenting a characterization of the network topology over time. Finally, in <cit.>, the authors released a dataset targeting the Indian microblogging platform Koo. The dataset consists of more than 72M posts and 75M comments, with related features such as shares and likes. Also, a thorough overview of the platform is presented, consisting of a discussion of the news ecosystem on the platform, hashtag usage, and user engagement.



§ MATERIALS AND METHODS



In this section, we introduce our framework to characterize nodes and hyperedges in a social hypernetwork, as well as the definition of roles, called archetypes, in such a network. We start by providing some background, which is useful to understand the frame of our context. Then, we introduce the definition of archetypes and our method to calculate them. Finally, we detail the proposed framework to characterize nodes and hyperedges.



 §.§ Background


A hypergraph H=(V,E), V = {v_1, …, v_n} is the set of nodes, E = {e_1, …, e_m} is the set of hyperedges, where e_j ⊆ V, for j = 1,…,m. 
A visual representation of a hypergraph is depicted in Fig. <ref>.
The order of H is n = |V|, while the size of H is m = |E|. We denote with E_v_i the set of hyperedges containing v_i, that is E_v_i = {e_j ∈ E : v_i ∈ e_j}. Given a hyperedge e_j ∈ E, its size is the number of nodes belonging to it, that is |e_j|. The degree of a node v_i, denoted as deg(v_i), is the number of neighbors of v_i; a node v_k ∈ V is the neighbor of node v_i if and only if there exists at least one hyperedge e_j which v_i and v_k both belong to. The hyperdegree of a node v_i, denoted as hdeg(v_i), is the number of hyperedges to which v_i belongs. A graphical depiction of a hypergraph is given in Fig. <ref>.

We use a hypergraph H to model a social hypernetwork. Here, nodes are users of the social hypernetwork, while a hyperedge represents a discussion between users. In what follows, we will refer to the social hypernetwork simply as hypernetwork.

Moreover, features can be associated with users, and these are generally based on the content the users interact with and the discussions they participate in. To formalize them, we employ a set of features F = {f_1,…,f_l} characterizing the nodes in H. Given a node v_i ∈ V, F_i = {f_1_i,…,f_l_i} represents the values of each feature of v_i, that is f_k_i indicates the value of the k-th feature of v_i. We assume features can be either numerical or categorical and that numerical ones are always normalized. Also, with H_F, we indicate the hypergraph equipped with the set of node features F.



 §.§ Archetypes for Nodes Characterization



The first aim of our approach is the definition of archetypes. An archetype serves as a “template” to represent nodes characterized by a certain subset of features from the set F. 

The concept of an archetype is pivotal in characterizing higher-order node roles based on subsets of hyperedges. Often, the study of a representation of nodes in a social network involves the definition of a taxonomy or a more thorough analysis of the behavior of such nodes <cit.>. Instead, our definition of archetype enables a more general characterization of possible behavioral dynamics occurring among nodes, and does not rely on a single attribute.

Let F_A = {f_1, …, f_p}⊆ F be a subset of features. Then, an archetype A is defined as a tuple of values based on the features indicated by F_A. Formally, let A = ⟨ a_1, a_2, …, a_p ⟩, where a_j is the value of the j-th feature from F_A for the archetype A. Then, each a_k, for k = 1,…, p, represents a particular value of the k-th feature from F_A that characterizes the nodes associated with this archetype. As an example, let F_A = {f_, f_}, where f_ (resp., f_) is the value of a quantitative feature indicating the average sentiment value (resp., average toxicity value) expressed by a user. Such features are extensively used in various data science-based studies and can be easily computed via classical methods such as VADER <cit.> and Detoxify <cit.>. Suppose f_∈ [-1,1], where -1 (resp., 1) indicates a mostly negative (resp., mostly positive) average sentiment, and f_∈ [0,1], with higher values indicating a high degree of toxicity. Then, various archetypes could be defined based on these features. For instance, we could define the archetype A = ⟨ -1, 0.5 ⟩ that describes a template for users that are extremely negative and are toxic on average, and we could name this archetype as Cinically Commenter; instead, the archetype A = ⟨ 1, 0.5 ⟩ would be a template for users that are extremely positive and exhibit moderate toxicity, and we could name it as Overzealous User. Also, it is noteworthy that, for the sake of presentation, here the definition of archetype is only given w.r.t. nodes. Nevertheless, in case of features being also available for hyperedges, then the same definition applies, and archetypes for the latter can be defined.

Essentially, an archetype A can be viewed as a prototypical example of nodes that share similar feature values. To simplify the analysis and categorization of archetypes, instead of taking into account the feature values directly, we can map them into categorical values, indicating the state of the feature for the particular archetype. This approach involves setting a series of thresholds T = ⟨ t_1, …, t_p ⟩, where the value t_j has the same domain of the feature f_j in F, a set of labels L and a labeling function ι. Then, given an archetype A = ⟨ a_1, …, a_p ⟩, by applying to each feature a_j the corresponding threshold t_j from T via the labeling function ι, we derive the archetype A_T = ⟨ a^t_1, …, a^t_2 ⟩, where ι(a_j,t_j) = a^t_j is a label in L. Practically speaking, A_T represents the same archetype as A, but its representation is now built over a specific set of labels. Let us take the aforementioned archetype A = ⟨ 1, 0.5⟩, defined over the set of features F_A={f_, f_}, and representing Overzealous Users. Suppose we set T = ⟨ 0, 0.75 ⟩, and we set L = {, }. Also, suppose we define the labeling function ι as


    ι(a_j,t_j) =    if a_j ≤ t_j 
       otherwise


Therefore, in this example, we derive A_T = ⟨, ⟩, which can be seamlessly interpreted as representing users exhibiting a high sentiment value and a low toxicity value. Note how the labeling depends on the thresholds T, thus allowing for flexibility that can accommodate different contexts of analysis.

Finally, we are now able to state when a node is represented by a given archetype. Given an archetype A=⟨ a_i, a_2, …, a_p ⟩, based on a subset of features F_A = {f_1, …, f_p}⊆ F, we want to effectively understand what nodes are represented by it. Let v_i ∈ V be a node, and let F_A(v_i) = ⟨ f_i_1, f_i_2, …, f_i_p⟩ be the feature vector of node v_i according to the features selected in F_A. The node v_i can be considered as represented by the archetype A if F_A(v_i) is sufficiently close to A, according to a predefined distance metric d. We express it as d(F_A(v_i), A) ≤ϵ where ϵ is a small positive threshold value that determines the acceptable distance between the node's feature vector and the archetype. Note that both A and F_A(v_i) can be considered as vectors of the same length p; thus, classical distance metrics, such as the cosine similarity <cit.>, can be used. Furthermore, as we will see in the experiments, in the simplest case also a feature-wise comparison can be used to assess when a node can be considered as represented by a given archetype.
















































  §.§.§ A Characterization for Archetypes



To comprehensively understand and categorize user archetypes, we decide to observe them through three different expressions, namely, (i) emotional, (ii) psycho-emotional, and (iii) moral expressions. By leveraging well-established psychological theories and lexicons, we aim to create detailed profiles that reflect the peculiar ways in which users interact and participate within the social platform. In what follows, we describe in detail these three expressions, which we will subsequently use in our experiments.



  
*Emotional Profiles We aim to characterize archetypes based on the emotions they express.
To do so, we refer to the psychological theory of emotions <cit.> developed in 1980 by the American psychologist Robert Plutchik.
This theory identifies eight basic emotions — joy, trust, fear, surprise, sadness, anticipation, anger, and disgust — and claims that all other emotions derive from a mixture of these primary ones.
To quantify feelings expressed by Scored users, we leverage the NRCLexicon (National Research Council Lexicon) <cit.>, a resource containing over 14,000
English words and their associated emotional ratings according to Plutchik’s theory.
This dictionary is further expanded in [<https://github.com/metalcorebear/NRCLex>] to include WordNet synonyms, reaching over 27,000 terms. 
For each of the users' texts, we compute emotion scores, and normalize them in [0, 1]. 
In this context, 0 indicates texts that do not elicit any emotion, while 1 signifies texts that strongly evoke the specified emotion.



  
*Psycho-emotional Profiles We also characterize how user archetypes relate to their surrounding social environments.
To do so, we refer to the PAD model (Pleasure, Arousal, Dominance) introduced by Mehrabian and Russell in 1974 <cit.>.
According to the PAD model, three dimensions characterize the perception an individual has of the environment in which she finds herself.
Pleasure (sometimes referred to as valence) concerns whether an individual perceives the environment as enjoyable or not. 
Arousal measures how stimulating the environment is for the individual. 
Dominance indicates whether the individual feels in control of the environment.
To operationalize these dimensions, we leverage the VAD Lexicon (Valence, Arousal, Dominance) <cit.>. 
This resource contains over 20,000 English
terms, along with their associated valence/pleasure, arousal, and dominance values. 
For each of the three dimensions, we associate each text with the total score of its words.
Then, we normalize results in [0,1], where 0 implies an absence of the corresponding dimension, and 1 implies the strong presence thereof.



  
*Moral Profiles
We aim to characterize archetypes based on the moral dimensions that emerge from the content they produce.
We rely on the Moral Foundations Theory, a psychological framing rooted in cultural anthropology that postulates the existence of five universal moral dimensions <cit.>: authority/subversion, care/harm, fairness/cheating, loyalty/betrayal, and sanctity/degradation.
Each dimension is composed of a virtue (e.g., loyalty), and a corresponding vice (e.g., betrayal).
Virtues can be understood as follows, while vices can be considered their opposites.
The concept of authority can be defined in relation to specific traits, such as deference to higher authorities, in order to maintain group cohesion. 
Similarly, the concept of care can be understood in terms of nurturing and protection. 
Fairness can be conceptualized in terms of equal treatment and reward. 
Loyalty can be understood in relation to the prioritization of one's group and alliances. 
Finally, sanctity can be defined in terms of the maintenance of the sacredness of the body and the avoidance of moral contamination.
We operationalize this framework via the eMFD (Extended Moral Foundations Dictionary), a lexicon containing more than 3,000 words <cit.>.
Each word has an associated score in [-1, 1] for each foundation, ranging from strong vice outage (-1) to strong virtue outage (1).



 §.§ Analyzing Higher-Order Entities



While archetypes are a particular yet effective way to analyze entities within the social platform, they mainly focus on the node features, whereas interactions are not taken into account.

Hence, we define here a general characterization function to characterize nodes and hyperedges according to their exhibited features and higher-order dynamics. 

Without loss of generality, we propose the definition of such a function w.r.t. hyperedges. The same can be also applied to nodes.

Let H_F = (V,E) be a hypergraph equipped with the node features set F. We denote with ω a function which we call the hyperedge characterization function. The need for a function such as ω addresses the challenge of characterizing a hyperedge w.r.t. the nodes contained in it.
Let us recall that our approach deals with analyzing higher-order entities. To do so, we exploit the representation of node relationships via hyperedges. While it effectively captures the higher-order structural interactions between nodes, it might not be sufficient in acquiring insights into the semantics of such interactions. Therefore, we focus on the latter aspect through a characterization of hyperedges that is based not only on the contained nodes but also on their features. Formally, the domain of our hyperedge characterization function is E. ω takes in input a hyperedge e ∈ E and returns a value ω(e_j), which we call its characteristic value. Such value depends on the actual implementation ω: in fact, ω is general, and different approaches can be exploited to accommodate the hyperedge characterization. Furthermore, to address the aforementioned challenge, there are cases in which ω should be defined to consider the values of each node's features contained in the considered hyperedge. Given a feature of interest f_k, we write ω^k to denote that the actual implementation of ω considers the feature f_k.

In the following, we propose various specializations of ω and a brief rationale for each of them. Some of these specializations are used in the sec:experiments section. We separate them in three families, namely, (i) numerical-only definitions, (ii) categorical-only, (iii) structural-based ones. In describing each of them, we assume we are interested in characterizing a hyperedge e ∈ E. Also, there are different specializations that are intended to be exploited when the analysis we are carrying out is feature-oriented. Therefore, in these cases, we assume having a feature f_k of interest.

















































  
*Numerical-only specializations The following specializations are intended to be used when numerical features are considered within the investigation. Therefore, here we assume the feature f_k of interest is numerical. Some specializations are:


    
  * Statistics Descriptors: common statistics descriptors such as mean, median, mode, variance, and standard deviation of the feature f_i among the nodes in e can be easily computed. For instance, the mean would be simply defined and denoted as ω_avg = ∑_u ∈ e f_i_u/|e|.
    
    
  * MAD: calculates the Mean Absolute Deviation of the feature f_i among the nodes in e, that is ω_mad(e) = ∑_u ∈ e|f_i_u - ω_avg|/|e|.
    
    
  * Gini Coefficient: employs the Gini Coefficient to compute the dispersion of the values of f_i among the nodes in e, that is ω_Gini(e) = ∑_u ∈ e∑_v ∈ e, u ≠ v|f_i_u - f_i_v|/2 f_i |e|^2. 




  
*Categorical-only definitions Different from the previous ones, the following specializations are intended to be used when categorical features are considered. Therefore, here we assume the feature f_k of interest is categorical. Some specializations are as follows:


    
    
  * Entropy: measures the cohesion of the values of f_i among the nodes in e. We define and denote it as ω_entr(e) = -∑_u ∈ er_i_ulog(r_i_u), where r_i_u denotes the proportion of the value of feature f_i_u over all nodes in e.
    
  * Gini Impurity: measures the probability of incorrectly classifying a randomly chosen element in the dataset if it were randomly labeled according to the distribution of all categories in the dataset. Here, we employ it on the feature f_i, thus we define and denote it as ω_GiniImp(e) = 1 - ∑_u ∈ e r_i_u^2, where r_i_u denotes the proportion of the value of feature f_i_u over all nodes in e.




  
*Structural-based specializations

The structural-based specializations focus on characterizing a hyperedge based on its exhibited structural properties rather than only focusing on the features of the contained nodes. Some proposed specializations are the following:


    
  * Hyperedge Size: the measure is simply given by the size of the hyperedge e, that is ω_size(e)=|e|, regardless of the feature f_i.
    
    
  * Purity: Here, the measure indicates the homogeneity of the values of f_i in e, that is ω_purity(e)=max(count(f_i_u) : u ∈ e)/|e|.

    

    

    
  * Cohesion: the characterization of the hyperedge e is given by the mean pairwise similarity of the value of the feature f_i for each pair of node u, v ∈ e. Formally, given a similarity function sim(·,·) defined for two features, we define ω_cohes(e) = 2/|e|(|e|-1)∑_u ∈ e∑_v ∈ e, v ≠ u sim(f_i_u, f_i_v).

    
  * Interaction potential: the hyperedge e is characterized by its capacity to create external connections, i.e., connections that span outside the hyperedge. It measures the extent to which the hyperedge facilitates interactions beyond its mere composition. Formally, ω_intpot(e) = |{ v ∈ V ∖ e : ∃ u ∈ e, v ∼ u }|/|e|, where v ∼ u indicates that v and u are neighbors within a hyperedge different from e. The denominator can be |e|, i.e., only the nodes in e are taken into account, or |V ∖ e|, i.e., ω_intpot is sensitive to the order of the hypergraph.

    

    

    

    

    

    






























§ EXPERIMENTS



In this section, we present an experimental evaluation carried out to assess the efficacy of our framework. In what follows, we first provide a characterization of the understudied dataset regarding the social platform Scored.co. We study archetypes based on platform-specific and linguistic features, and we illustrate how our proposed framework allows for a comprehensive analysis of them. Then, we focus on the constructed hypernetworks, and we provide a characterization of discussions, that is, hyperedges, through our proposed hyperedge characterization function.



 §.§ Dataset







We focus on the Scored.co social platform, which hosts a network of communities that users can create and join. The platform shares different similarities with the more common platform Reddit[<https://reddit.com>]. Users can join communities related to various topics, from politics to memes, and upvote content they like, and downvote content they dislike. The number of upvotes minus the downvotes constitutes a post's score, i.e., a value that indicates the content's usefulness and/or relevance to the discussion.

Interestingly, Scored.co acted as a shelter for many far-right/alt-right communities after the 2020 massive Reddit bans <cit.>. Some of these maintained their names and key figures, e.g., , , ultimately acting as a continuation of the original ones. This platform has allowed users to freely share their views without the risk of being banned. 
Scored.co is an emerging yet understudied platform, potentially hosting dangerous content. Despite its growing user base, it has received relatively little academic attention compared to other social media platforms, as we pointed out in the sec:related-literature section. The loosely moderated nature of some communities can lead to the spread of misinformation, hate speech, and radical ideologies. 
This makes Scored.co a platform of interest for researchers studying online extremism, digital sociology, and the impacts of social media on public discourse.

To build the dataset our experiments are carried out on, we collect data from the Scored.co platform via the official API[<https://help.scored.co/knowledge-base/getting-started-with-the-api/>]. To retrieve as much content as possible, we use a breadth-first search technique by first gathering all posts and comments authored by , one of the platform's admins, which is one of the most active users on the platform. Then, the content is parsed to retrieve other users and their related content. We repeat this procedure until no new user is found. Eventually, we find 207,554 unique users (referenced to by their username), 4,398,074 posts, and 36,978,685 comments. Our data collection process started on June 9th, 2024, and ended on June 22nd, 2024. The final dataset comprises higher-order interactions on Scored.co starting from the platform's launch on October 15th, 2019, and ending on June 1st, 2024.

  

To improve reproducibility and share our findings, we release the dataset on Zenodo <cit.>. The dataset is anonymized and provides: 

    
  * Higher-order interactions extracted from discussion threads. 
    Each thread/hyperedge is labeled with the community it is extracted from (e.g. ), and the year and month when the root post was created;
    
  * User profiles, namely a set of features characterizing each user at every time stamp.

A Python data collection script is also enclosed in the repository.
In the first release of this dataset, only user/thread data leveraged in this study is shared (i.e., 2023 data from the c/TheDonald community, see next section).
However, we plan to update it in the near future to include statistics and interactions for all users, leveraging Zenodo's versioning system.
Our dataset complies with the fair guidelines for data sharing <cit.>, which are as follows:

    
  * Findable. We release the data on Zenodo [<https://zenodo.org/>], a service managed by CERN that also assigns a DOI to the repository, ensuring findability. Moreover, we indexed it also on the SoBigData Research Infrastructure (URL: <http://sobigdata.eu/>), an EU-funded RI providing curated datasets and algorithms advocating open science;
    
  * Accessible. The dataset is freely accessible to anyone with an internet connection;
    
  * Interoperable. All data is released in CSV files, which can be read and manipulated with most programming languages;
    
  * Reusable. Files, fields, and data collection methods are described in a dedicated document released within the repository, ensuring reusability.






 §.§ Higher-order Interactions and Analysis


Given the previously described dataset, we are now able to construct the hypernetworks which constitutes the basis of the experiments.
We start by constructing a hypernetwork H = (V,E), where V denotes the set of Scored.co users who participated in at least a discussion, while E denotes the set of hyperedges encoding discussion threads. That is, a user v appears in V if they participate in at least one discussion. Also, we discard all hyperedges of size 2 due to the fact that we are interested in studying higher-order interactions. This means that, for all e ∈ E, |e| > 2.


Since discussion topics on Scored are varied, ranging from politics to sports, memes, and more, we decided to center our analysis on discussions taking place in  during 2023. 
This community is a controversial and highly active online group that originally formed on Reddit before migrating to platforms like Scored.co <cit.>. We chose it because it is the oldest and most active community on the platform, covering 74% of all collected posts and comments. 
This allows us to maintain topic continuity without sacrificing large portions of the data.

Note that H is a hypernetwork that represents the whole dataset. To enable analyses on the time dimension, we construct a hypergraph H_t = (V_t, E_t), for each month t = 1, …, 12, containing users and discussions relative to month t.

In Table <ref>, we report summary statistics of all the constructed hypernetworks. The table reports basic statics, such as nodes and hyperedges number |V| and |E|, respectively, the largest hyperedge size max_|e|, the average hyperdegree hdeg, and the average degree deg. Also, for each pair of adjacent timestamps 1 ≤ t, t_+1 < 12, we report the Jaccard similarity index between node sets. The last row of the table reports the same information for the hypernetwork H.



From the analysis of this table, we can observe different insights. First off, the aggregated hypernetwork ℋ includes interactions among roughly 20% of all discovered users. Indeed, on average, users participate in 113 discussions (hyperedges) throughout the year, and in 18-20 discussions each month. Scored.co users are exposed to ∼ 1K unique peers, with a monthly average of 300-400, thus showing locally dense patterns consistently over time. We can also observe how the largest hyperedge size exhibits some variations but remains in the same order throughout the whole period. This indicates the presence of a highly connected subgroup at certain timestamps. The average hyperdegree hdeg remains relatively stable across timestamps, indicating that on average, each node is part of ∼ 20 hyperedges. The average degree deg remains fairly consistent, too. To complement these statistics, we try observing with a more detailed look the distribution of hyperdegree and hyperedge size, which are depicted in Figures <ref> (a) and (b), respectively. From the analysis of these figures, we can see that both show the typical power-law behavior observed in many real-world hypernetworks <cit.>. We observe how a few users participate in a large number of discussions, while most users participate in just a few. The hyperedge size distribution (Fig. <ref> (b)) displays a similar shape, with mostly small discussions and a few large ones. We also note how these patterns are consistent over time, with 44% interactions of at least size 5, 15% interactions of at least size 10, and 2% interactions of at least size 50. Finally, in Table <ref>, we can observe that the overlap between adjacent timestamps, calculated via the Jaccard similarity index between subsequent node sets V_t and V_t+1, is large and coherent over time, stabilizing on 64% on average. Overall, the studied part of this platform emerges as a locally dense online community with temporally stable structural patterns and significant user engagement, which makes it interesting to study with our approach.








 §.§ Identification and Analysis of Archetypes



We recall that a core aspect of our approach relies on identifying archetypes, namely typical patterns that can give us approximate descriptions of individual element behaviors. As we also noted in sec:archetype section, archetypes can be easily defined for nodes and hyperedges. In this section, we aim to characterize user behaviors. Therefore, henceforth we refer to archetypes as related to nodes/users.

To define user archetypes, we combine platform-specific and linguistic features. Specifically, we outline eight archetypes based on these features and their values. The features we decide to use are, namely, (i) score, (ii) sentiment, and (iii) toxicity. As far as the score is regarded, we compute the average post score for each user, then we normalize values in [0,1]. For the sentiment feature, we use the well-known VADER algorithm <cit.>, a rule-based sentiment classifier that assigns each text positive, negative, neutral, and compound sentiment intensity values. Out of these, the compound sentiment is a score in [-1, 1] going from the most extreme negative (-1) to the most extreme positive (1) sentiment. For our purposes, we compute the average compound scores for each user and rescale values in [0, 1]. Moreover, for the toxicity feature, we compute its values with Detoxify <cit.>, a neural toxicity detection model that returns the probability that a text contains toxic language, and also in this case, we normalize the obtained values. Finally, to discern between high and low values for each feature, we set a threshold of 0.5, such that when a feature has a value higher than this threshold, it is considered high, and vice-versa.

The list of the eight archetypes defined is depicted in Table <ref>. Here, we can observe how setting the various combinations of the state of a feature (high or low) creates different archetypes. We defined these archetypes in order to encapsulate the inherent semantics of the discussions taking place in the studied dataset. For instance, the first row is the so-called Community Hero, which represents users who consistently contribute positively to the discussion by maintaining a positive and supportive attitude without engaging in toxic behavior. A detailed characterization of these archetypes is given in the following section.








  §.§.§ Archetype Characterization


Having defined the archetypes, we are now able to provide a characterization of them, based on multiple psychological and social dimensions. More in detail, we describe the behaviors of these types with respect to the different aspects highlighted in the <ref> section. We recall that these aspects are, namely, the (i) environment perception, the (ii) emotions, and (iii) morality. 



To control for class size imbalances, we compare average values for multiple features across the top 10 most archetypal users for each archetype, i.e., the most representative users for each category. We rank users according to their typicality, which is defined as follows:

    Typicality(u) = ∏_f α_f f(u)


Here, u is the target user; f is a function computing u's characteristic score, sentiment, or toxicity; alpha_f is a coefficient that equals 1 if u's archetype has a high value for f and, conversely, -1 if u's archetype has a low value for f. As an example, a Community Hero's typicality is maximized when its score is 1, its sentiment value is 1, and its toxicity value is 0. Thanks to the definition of typicality, we are now able to characterize each archetype with respect to the aforementioned aspects. Along with the characterization, in Fig. <ref>, we depict the profiles of each archetype. The top part of the figure represents the profiles according to Plutchik's wheel of emotions. The middle part, instead, represents the profiles according to the Pleasure/Arousal/Dominance model. Finally, the bottom part depicts the profiles according to the Moral Foundations theory.





  
*Benevolent Underdogs. With respect to their psychological features, Benevolent Underdogs are characterized by high trust (0.18), joy (0.14), and anticipation (0.11), low arousal (0.44) and high valence (0.66). 
These users have a positive and optimistic outlook, showing happiness and hopefulness in their interactions. 
From their feature values, a possible interpretation can suggest them as users focusing on creating a supportive environment and  emphasizing social cohesion.
It is important to note that, for this archetype as well as all the following ones, these descriptors can lead only to qualitative suggestions and interpretations of their behaviors.



  
*Community Heroes. Such users are characterized by high trust (0.17), joy (0.12), and anticipation (0.14), low arousal (0.45) and high valence (0.64). 
Thus, their psychological profile is similar to Benevolent Underdogs, with a generally positive outlook. 
From a moral focus perspective, they mostly emphasize loyalty and, to an extent, authority.
Community Heroes and Benevolent Underdogs essentially behave in the same way, although the former are more recognized.
Thus, we hypothesize that Community Heroes often provide support in more prominent and/or critical situations, gaining greater acknowledgement from the community. 



  
*Controversial Stars. With respect to their psychological features, Controversial Stars show signs of joy (0.13), trust (0.12), and anger (0.10), as well as high valence (0.60). 
This indicates that while they have a positive emotional tone and are capable of experiencing happiness and trust, they can also exhibit anger.
This leads to an interpretation of this archetype as an emotionally charged one, with a tendency to express both positive and negative tones.
Moreover, they mostly emphasize loyalty and authority moral dimensions.




  
*Infamous Celebrities. Infamous Celebrities show high signs of anger (0.14), fear (0.14), and disgust (0.10).
Arousal (0.53) and Dominance (0.50) are average, while Valence is slightly below average (0.48).
Their complex psychological profile hints at expressions of negative tones, perhaps toxic interactions.
High levels of fear could indicate a tendency to anticipate negative outcomes, which can make their discourse more alarmist. 
Disgust contributes to a critical and often harsh tone. 
Average arousal and dominance could suggest a moderate level of engagement with the environment, whereas their slightly below-average valence could be a sign of negative interactivity.
Moral dimensions of care, fairness, loyalty, authority, and sanctity display negative values.

This moral outlook, combined with their negative emotional profile, contributes to their provocative, even controversial, behavior.



  
*Malcontents.
Malcontents are characterized by high values for disgust (0.13), and anger (0.11).
This is paired with below-average values for Valence (0.39) and Dominance (0.45), and high Arousal (0.61).
This emotional profile could reveal a full negative profile.
The archetype shows high arousal as well as strong emotional responses and engagement, but also low valence and dominance.
This could mean the users belonging to it often display dissatisfaction, leading to discontented interactions. Morally, Malcontents use vice-oriented language, focusing on sanctity/degradation and care/harm.
Their low values in these areas could indicate strong criticism and a tendency to vent.




  
*Quiet Critics.
The Quiet Critics' psychological profile outlines an emotional landscape characterized by high levels of sadness (0.11), fear (0.14), and anger (0.12). 
These predominant features suggest a dissatisfaction with their environment.
Anger, although present, paired with low toxicity values, could suggest a profile expressing criticism in a non-confrontational way rather than being directly hostile.
With average levels of Valence, Arousal, and Dominance, their emotional responses are steady an moderated. 

From a moral perspective, Quiet Critics' language frequently involves vice-oriented terms, particularly focusing on sanctity/degradation, care/harm, and loyalty/betrayal. 
Their emphasis on sanctity could interestingly suggest a concern for moral purity, whereas the focus on care/harm could highlight preoccupation with injustice.


Their attention to loyalty/betrayal could indicate a sensitivity to perceived trust, which might align with their cautious-yet-critical nature. 



  
*Respected Critics.
The Respected Critics' psychological profile is similar to those of the Quiet Critics, mostly eliciting sadness (0.8), fear (0.14), and anger (0.12). 
Sadness and fear highlight their anticipation of negative outcomes, possibly making their critiques cautionary. 
Anger drives their behavior, moderated by average levels of valence, arousal, and dominance.
Their moral profile resembles that of the Quiet Critics. 







  
*Positive Provokers. Positive Provokers are characterized by high levels of trust (0.18) and joy (0.13). They show average Dominance (0.54) and Arousal (0.49) values, as well as high Valence (.60).
Their high trust and joy could suggest they generally expect positive interactions from others and have a positive disposition.
The high Valence reflects their overall positive emotional tone. However, despite these positive traits, their high toxicity scores reveal a tendency to a duality, that we could interpret as a confrontational/provocative communication style
This dichotomy of positivity and toxicity creates a complex profile where users intentions may be good, but their execution can be damaging to others.



  §.§.§ Archetypes Transitions


In this section, we aim to explore the temporal dimension of user interactions within the Scored.co community and understand whether significant transitions exist between the identified archetypes. By analyzing these transitions, we can gain insights into how user behaviors evolve over time. This analysis not only helps in understanding the fluidity of user roles but also potentially provides valuable information for targeted interventions and community management.

In order to carry out this analysis, we propose the following methodology. Given the imbalanced class sizes (see Table <ref>) throughout the observation periods, we employ a null model to test the transition probabilities between archetypes. This null model computes the expected transition probabilities on a copy of our dataset where archetype labels are shuffled. The detailed steps are as follows:


    
  * We generate N = 500 shuffled copies of the dataset, randomizing the archetype labels while preserving the overall distribution of interactions and activities;
    
  * For each archetype pair (A,B) in the shuffled copies we compute the transition probability P(B|A), which represents the likelihood of a user transitioning from archetype A to archetype B;
    
  * We calculate the mean and standard deviation of P(B|A) across all shuffled copies. These statistics are used to compute z-scores and p-values for the observed transition probabilities in the original dataset;
    
  * Finally, transitions with p-values less than 0.01 are considered statistically significant, indicating that these transitions occur at a higher-than-expected rate compared to the null model.


Fig. <ref> depicts the observed statistically significant transitions, letting us observe several insights from the transitions between archetypes.
One notable transition is from Benevolent Underdogs to Respected Critics, occurring at a significant rate of 39.23%. Benevolent Underdogs are characterized by positive sentiments and low post scores, typically engaging in supportive and encouraging behaviors. This transition can indicate that even users who start with low recognition can, through consistent positive contributions, build credibility and eventually gain influence within the community. Community Heroes, another archetype defined by high trust, joy, and engagement, show a tendency to transition to Benevolent Underdogs or, more rarely, to Positive Provokers. The slight shift to Benevolent Underdogs suggests that Community Heroes, despite their initially high engagement and recognition, may sometimes experience a reduction in visibility or engagement, leading them to adopt a more low-profile but still positive role. Also, we observe an occasional drift towards more toxic behaviors. Infamous Celebrities, which have high post scores and negative sentiment, exhibit a significant probability of remaining in their contentious roles (13.71%). However, there is also an 11.71% likelihood of transitioning to Malcontents. This shift reflects a potential decrease in engagement and influence, where high-profile negative behavior can lead to more discontent and a less active state. This transition from Infamous Celebrities to Malcontents suggests a trajectory where persistent negativity can diminish a user's central role within the community. Malcontents, who are driven by negative emotions and low post scores, present an interesting case with a substantial 33.29% chance of becoming Respected Critics. This transition implies that their critical nature can eventually earn them credibility, provided they moderate their toxic behaviors. Another significant transition is observed from Quiet Critics to Respected Critics, with a remarkably high probability of 43.29%. 







  §.§.§ Temporal Analysis of Archetype's Higher-order Interactions


Above, we proposed a thorough characterization of a series of archetypes. At this point, an interesting aspect to study is the dynamics of archetypes within the social platform, as well as their evolution in time. To do so, we now focus on temporal trends of archetypes by examining some features on a monthly basis. We focus on two key metrics, that is, the average hyperdegree hdeg and the average degree deg of all nodes represented by each archetype.

We consider the archetypes defined in the previous section, and we analyze the nodes each archetype represents in each month. More formally, for each archetype and for each node u, we compute its hyperdegree hdeg(u) and its degree deg(u), and we do this for each hypergraph H_t, t = 1,…,12. In Fig. <ref> we show the obtained results. The figure consists of 12 subplots, one for each hypergraph, corresponding to a different month. For each subplot, the x-axis represents the average hyperdegree hdeg of each archetype, while the y-axis represents the average degree deg of each archetype. From the analysis of this figure, different insights can be derived. First off, we can observe how the Respected Critics archetype has a higher hyperdegree and degree than all other archetypes. This is somewhat expected as their characterization encompasses a considerable number of nodes, as we have seen in the sec:archetype-characterization section. It is straightforward to observe how there are pairs of archetypes that are often collocated together w.r.t. these dimensions. This is the case of Infamous Celebrity and Quiet Critic, which appear very close in almost all monthly snapshots, although they are very different in size. The relatively small number of archetypes such as Controversial Star and Positive Provoker is also reflected in their average hyperdegree and degree values: indeed, they remain stable during the months and have little to no fluctuations. This is somewhat expected, considering that these archetypes are characterized by emotionally charged and often provocative interactions, leading to fluctuating levels of activity and influence. This variability suggests that their presence is more dynamic, potentially driven by specific events or contentious discussions that peak at different times. Also in this case, these results could be somewhat enhanced by a more qualitative-oriented approach: indeed, aspects such as the content shared between archetypes would be of great importance in a more in-depth analysis of their interaction. Nevertheless, we believe the provided observation already indicates different implications. For instance, archetypes with stable engagement patterns are likely to be reliable contributors, therefore they could be taken into account in systematic operations of the platforms, such as moderation activities. Also, archetypes that show fluctuating feature values might be highly responsive to specific events or topics, which could fuel critical discussions. 





 §.§ Characterization of Discussions


We recall that one of the objectives of our approach is to analyze higher-order entities. To do so, it offers the so-called hyperedge characterization function ω, which we presented in the sub:hyperedge-characterization-function section. We are now able to leverage it to describe the dataset with regard to some of the properties exhibited by the discussions.

First off, we focus on examining our dataset on a monthly basis, thus we consider each hypergraph H_t, t = 1, …, 12. We do not focus on the overall hypergraph H due to the fact that we are interested in observing how the structure and the interactions within Scored.co evolve over time. Also, this approach allows for a more granular understanding of user behavior and interactions. For each hypergraph H_t, we compute the s-betweenness hyperedge centrality <cit.>. This measure is a higher-order extension of classical graph betweenness centrality, and is computed on the line graph projection of the hypergraph to estimate the importance of hyperedges. We recall that the line graph of a hypergraph is a graph that represents the relationships between the hyperedges of the original hypergraph. In particular, there is a node in the line graph for each hyperedge in the hypergraph, and there is an edge between two nodes if the corresponding hyperedges share at least one node. After the computation of the centrality, we select the top 50 hyperedges in each hypergraph H_t, and for each of we compute some descriptive features based on the linguistic productions of the contained nodes. In particular, we focus on the following features, namely, (i) average word count, (ii) average unique word count, and (iii) average purity w.r.t. user archetypes. These values can be easily calculated via the ω function, specialized for each of them. For instance, the specialization for the average purity is given by ω_purity in the sub:hyperedge-characterization-function section.


In Fig. <ref>, we report the temporal trends of the average values of average word count, average unique word count, and average purity for the top 50 most prominent discussions. Note how the discussion here is another term for hyperedge: in fact, in our setting, a hyperedge represents a discussion between users in the social hypernetwork. From the analysis of this figure, we can derive different insights. First off, the average word count graph, in the left part of the figure, shows a relatively stable trend with slight fluctuations, indicating a consistent level of engagement in discussions over the year. There are some notable peaks, such as a slight increase around mid-year followed by a step-down and a subsequent rise towards the end of the year. Similarly, the average vocabulary size, which indicates the average unique word count, mirrors the previous measure but remains consistently lower, which is somewhat expected because of non-content words (e.g., articles, pronouns, etc.) appearing more frequently in texts. A slight increase in the middle of the year is also depicted here, suggesting periods of higher and lower lexical diversity in user interactions. The middle part of Fig. <ref> reports the trend for average subjectivity. We recall that this metric evaluates how subjective (as opposed to objective) the discussions are. It remains relatively stable throughout the year, which indicates that discussions typically contain a balanced mix of personal opinions and factual statements. Indeed, this calls for a more qualitative-oriented analysis of these discussions, an approach often used in the content-based investigation of social platforms <cit.>. Finally, the right part of the image depicts the graph of the average purity w.r.t. the user archetypes. This refers to the consistency of user behavior and content relative to the defined archetypes. We can observe how this trend shows more variability compared to the previous features. There are sharp declines in some months, suggesting periods of more inconsistent behavior among users, while the different peaks could indicate times when user behavior is more aligned with archetypes. While all these insights are interesting, and show the applicability of our proposed approach, more in-depth analyses could be performed, as also noted above. Nevertheless, the various specializations of ω enable a series of different lenses through social interactions can be studied, especially in understudied platforms.







§ CONCLUSION



The identification of node roles within complex networks is significant when analyzing their dynamics and function. To advance in this context, in this paper we proposed a multi-dimensional, general framework to characterize nodes and hyperedges in a social hypernetwork. The aim of our framework is two-fold: i) to characterize nodes and hyperedges taking into account their exhibited features and higher-order dynamics, and ii) to define “archetypes”, serving as a template to represent higher-order roles of nodes. 
Our framework consists of different components, namely, hyperedge ensembles, hyperedge- and node-based characterization functions. The combination of such components allows one to analyze the social hypernetwork from a more general point of view; each component can also be exploited in a standalone fashion, thus enabling a detailed analysis of the single aspect it captures. 
To assess the effectiveness of the framework, we carried out an exhaustive experimental campaign on Scored.co, an understudied social platform, focusing on different aspects such as the characterization of nodes as well as their behaviors and surroundings. 

This paper is a starting point for future research with various possible extensions.

First off, the framework can be applied as it is on other (possibly understudied) social platforms such as Bluesky <cit.>.
Second, it would be interesting to compare archetypes based on the set of features selected for defining them. Indeed, an approach to identify the archetype best representing a particular set of nodes could be defined as an optimization problem, and meta-heuristics solutions could be employed to solve it. 
Another future direction would be studying role dynamics in crisis situations. Indeed, we could apply the framework to study social hypernetworks in crisis situations, e.g., natural disasters and pandemics, to understand how roles and behaviors shift under stress and how information and support are mobilized <cit.>. 
Exploiting the temporal component, we would integrate this framework about higher-order roles in the recent multi-dimensional analysis of group evolution in temporal data <cit.>.
Finally, an interesting challenge would be the analysis of cross-platform role identification, and in particular the investigation of the consistency of identified roles across multiple social platforms. This could help in understanding whether certain archetypes are platform-specific or they can represent more universal roles in online social interactions.





§ ACKNOWLEDGMENTS


This work is supported by (i) the European Union – Horizon 2020 Program under the scheme “INFRAIA-01-2018-2019 – Integrating Activities for Advanced Communities”, Grant Agreement n.871042, ”SoBigData++: European Integrated Infrastructure for Social Mining and Big Data Analytics" (<http://www.sobigdata.eu>); (ii) SoBigData.it
which receives funding from the European Union – NextGenerationEU – National Recovery and Resilience Plan (Piano Nazionale di Ripresa e Resilienza, PNRR) – Project: ”SoBigData.it – Strengthening the Italian RI for Social Mining and Big Data Analytics" – Prot. IR0000013 – Avviso n. 3264 del 28/12/2021; (iii) EU NextGenerationEU programme under the funding schemes PNRR-PE-AI FAIR (Future Artificial Intelligence Research). 









 Paper Title: Artificial Intelligence Approaches for Energy Efficiency: A Review.



Ge-based Clinopyroxene series: first principles and  experimental local probe study
    João P. Araújo
    August 5, 2024
===================================================================================




§ ABSTRACT 
 set Sustainable Development Goals and this paper focuses on 7th (Affordable and Clean Energy), 9th (Industries, Innovation and Infrastructure), and 13th (Climate Action) goals.
Climate change is a major concern in our society; for this reason, a current global objective is to reduce energy waste.
This work summarizes all main approaches towards energy efficiency using Artificial Intelligence with a particular focus on multi-agent systems to create smart buildings.
It mentions the tight relationship between AI, especially IoT, and Big Data.
It also explains the application of AI to anomaly detection in smart buildings and a possible classification of Intelligent Energy Management Systems: Direct and Indirect.
Finally, some drawbacks of AI approaches and some possible future research focuses are proposed.





§ INTRODUCTION
 
According to estimates by the United States Energy Information Administration, 40% of the annual CO2 emissions are directly related to electricity consumption.

A way to tackle pollutant emissions is to reduce the amount of wasted energy.
Thanks to a more efficient use of energy, we can reduce energy consumption by more than 30% as shown in Figure <ref>.



The most energy-consuming part of a building is the Heating, Ventilation, and Air Conditioning (HVAC) system <cit.>, for this reason, a special focus is put on this system in the literature.

To improve awareness about energy waste habits and reduce it, consumers must first monitor their energy consumption and manage it after receiving appropriate advice <cit.>.
This can be achieved practically by using an Energy Management System (EMS). Many approaches use Artificial Intelligence (AI) tools to design Intelligent Energy Management Systems (IEMS), detect and minimize faults and increase energy efficiency. Here we review some possible approaches found in the literature.

AI in IEMS has a strong coupling with data collection from sensors, in this case, due to the amount of elaborated data, we can talk about "Big Data".



§ BIG DATA USAGE IN IEMS
 
Big Data (BD) refers to huge amounts of data, both structured and unstructured, that can be used to extract useful information.
BD usage has been increasing in recent years, and it is expected to continue to grow in the future because of the massive increase in global data creation<cit.>.

BD is characterized by the V's (Volume, Velocity, Variety, Veracity, and Value, and even more are being added to the list during the years) <cit.> and, regarding energy efficiency, the most relevant characteristics are <cit.>:

    
  Volume The amount of electricity consumption data collected by 1 million smart meters in 1 year (collecting once every 15 minutes) is about 3 TB; this presents a challenge for data storage and analysis.
    
  Velocity To analyze the data in real-time, the data processing speed must be fast enough to keep up with the incoming information, which means in a range that goes from sub-second to 15 minutes.
    
  Variety The data collected from smart meters is heterogeneous, including electricity consumption data, weather data, and customer information: each of them is useful in a different part of the industry. Also, this data is structured, semi-structured, and unstructured.


BD's primary focus is on collecting data and gaining insights, while AI aims to accomplish tasks.

The usage of BD for energy-efficient buildings is powered by the Internet of Things (IoT), the collection of local data from sensors and other services that provide local and global weather data, building information, and historical data provided by societies, groups, and commercial companies. It is also worth mentioning the potential usage of 3D topographic data which can help to assess the shadowing of surroundings, air flow, noise level, and other site-specific information even before the construction of a building  <cit.>.



Once the data is collected, it can be used to train (fine-tune) AI models to predict weather and inhabitant patterns together with the building's characteristics like insulation, heat transfer, household appliances energy consumption, etc. <cit.>.

Big Data can not only be used commercially, or for personal use, but also for research purposes thanks to the massive amount of geographically distributed data, which can be used for weather study and prediction or for AI training, etc..




§ ARTIFICIAL INTELLIGENCE FOR ENERGY EFFICIENCY
 
The most promising use of AI is in smart buildings equipped with sensors and actuators.
They can be components of a smart grid that uses a Demand Side Management system (DSM) to plan, control, and manage the loads of electricity.



AI systems can also be used for a statistical approach in order to make predictions about consumer energy use; the main 4 steps are:

    
  * Data collection
    
  * Data preprocessing
    
  * Model training
    
  * Model testing


 presented several AI techniques and models with their respective use cases:

    
  Decision trees to predict blackouts and to define energy storage planning and management in buildings.
    
  Random forest to predict energy consumption.
    
  Wavelet neural networks to predict time-series data for hybrid renewable residential micro-grids.
    
  Naive Bayes models to solve energy problems in buildings when having a prior probability, and the next consume guess is needed.
    
  Artificial neural networks to evaluate thermal comfort and to give suggestions to the user to save energy.
    
  Deep learning to estimate the power consumption of appliances in a distributed system.
    
  Regression to predict monthly heating and cooling demands.
    
  Genetic algorithms to minimize the total energy cost in a dynamic pricing scheme and it can be also used for finding the optimal energy consumption to provide a comfortable temperature.
    
  Fuzzy logic to forecast the demand load based on changes in indoor and outdoor conditions.
    
  Particle swarm optimization together with other techniques to find the optimal solution in problems like estimating the energy demand.
    
  K-nearest neighbor to analyze energy demands.
    
  Principal Component Analysis (PCA) to reduce the dimensionality of the data and then use another AI method to make the prediction.

Different models can be combined to make better predictions.












 §.§ Multi-Agent Systems
 
An agent is a computer system situated in some environment, and that is capable of autonomous action in such environment to meet its design objectives <cit.>. It can be described as the implementation of the "Belief-Desire-Intention" (BDI) model, where the agent has beliefs about the environment, desires to achieve some goals, and intentions to act to achieve them.


A Multi-Agent System (MAS) consists of an environment, objects, agents, and all their relations, a set of actions that can be performed by the entities to change the environment <cit.>; each agent cooperates with the others to achieve a common goal (but it can also have its own objectives).
This system makes it possible to emulate the actions and interactions of human organizations through the decomposition of problems into smaller parts; problems are solved by agents communicating.
An example of decomposition occurs when a group of agents monitor the comfort temperature set by the user and another group controls the HVAC system and communicates with a service agent that collects past data to help select a new ideal temperature for the residents.




MASs often rely on frameworks that provide the necessary tools to create and manage agents, such as JADE (Java Agent DEvelopment framework) <cit.> and PADE (Python Agent DEvelopment framework), the most used in the MAS community. Both of them are compatible with the FIPA (Foundation for Intelligent Physical Agents) <cit.> standard, which defines the communication protocols between agents.

In the past there has been a lack of standardization in the communication protocols; nowadays BACnet is the leader in the market for HVAC <cit.> systems; in the last years an open-source protocol supported by big players in the market, Matter, is arising and being established as the new standard for all the IoT devices <cit.>.
Optimization algorithms are used to lower the energy consumption of the building, two types of algorithms can be used: exact or heuristic. The first ones are based on mathematical models and can find the optimal solution; the second ones are not necessarily optimal, but they are faster and overall they are better suitable in problems with a human in the loop, like the energy efficiency of a building, in which a trade-off with users' comfort is needed. MASs have been proven to be promising as heuristic techniques for solving such problems whose domains are distributed, complex, and heterogeneous.

An interesting agent learning approach is the one described by  who use Reinforcement Learning (RL): the inhabitants of an Intelligent Building (IB) are modeled as agents that play a game in which they obtain a benefit if they lower their energy consumption and they are punished if they increase it. Agents acting as humans learn to optimize the trade-off between energy consumption and their comfort. In this way, the system can learn the best configuration for the IB.

MAS usage in the energy efficiency field can be divided into three main areas of application <cit.>:

    
  Demand Response (DR) the ability of the supplier-client system to react to the supplier's request to adjust energy consumption according to available supplies or to reduce the cost of energy. This is particularly useful in the case of demand peaks, failure, or maintenance in the grid or to exploit renewable energy sources at their best.
    
    
  Human Behavior Simulation the simulation ability, which is, as already mentioned, a strong potential of MASs. Simulations focus on humans present in the house: their habits and preferences, their interactions with the building (appliances, heating & cooling, lighting, etc.), and the consequences of their actions on energy consumption also taking into account all the external factors (weather, time of the day, price of energy, etc.).
    Each person can be modeled by a separate agent, allowing the evaluation of a different behavior for each inhabitant, taking into account the heterogeneity of human behavior in the comfort pursuit.
    
    The system aims to optimize energy consumption without reducing the user's comfort sensation.
     studied how the user's preferences can be perceived (asking the users or learning from manual adjustments made by users).
    Main parameters that influence energy consumption and need, a fundamental stage for improving optimization models, can be found using Principal Component Analysis (PCA) <cit.>.
    A review of other works can be found in tables 3,4 in .
    
  Wireless Sensor Networks (WSN) management the use of MASs to simulate real buildings and how sensors would act in it.
    A classification of the most relevant elements to be monitored: occupation, preferences, activities, temperature, etc. can be found in .
    The integration of Real-Time Localization Systems boosted MAS systems evolution in this field, achieving an average energy saving of 17-22% <cit.>.
    
    After a WSN simulation, which is especially useful in the research field, a real WSN can be deployed in situ.


Summing up a MAS-based system should focus on the following aspects <cit.>:

    
  Knowledge and learning the agents should know the user's preferences which affect energy consumption such as tastes in terms of comfort, habits, timetables, etc., also agents should know the environment characteristics like the building's structure, the thermal relation with the external environment, weather forecast, etc.
    
  Analysis, adaptation, and communication with the environment the agents should be able to analyze their prior knowledge and communicate the information that the MAS must use to adapt to the environment.
    To communicate with the environment, and send and receive information from it, it is necessary to deploy a WSN.
    
  Decision-making the agents should be able to make decisions based on the information they have, the environment, and the user's preferences. Here algorithms should give greater weight to the factors they consider appropriate; to identify the most important ones, a PCA can be used <cit.>.


The results in energy saving that can be achieved with MAS varies between 17% and 41% based on the specialization and the target human satisfaction of the system <cit.>




§ ANOMALY DETECTION
 
AI can be used to detect anomalous usage of electricity and its causes <cit.>.
Anomalies can be caused by different factors like:

    
  * Wrong behavior of the user that forgets to turn off some electrical appliances.
    
  * Some appliances not working like they are supposed to (faults).
    
  * Electricity thefts.

Moreover, an anomaly must be detected depending on the time of the day or the season, for example, it is not an anomaly if the heating system is on during the winter, but it is during the summer.

To determine if there is an anomalous electricity usage, the AI model must take care of a lot of aspects like the power consumption but also appliance-specific parameters, occupancy patterns, and weather conditions.

Many anomaly detection approaches can be found in the literature:

    
  * Unsupervised detection (clustering, dimensionality reduction)
    
  * Supervised detection (Neural Networks, regression, probabilistic models)
    
  * Ensemble method (boosting, bagging)
    
  * Feature extraction (distance-based, time-series analysis, etc.) 
    
  * Hybrid learning
    
  * Other techniques


Also, systems can be categorized based on the anomaly detection level:

    
  Aggregated level detect anomalous consumption based on data of the main building-wise meter but, doing so, the information about the responsible appliance is lost.
    
  Appliance level detect using individual sub-meters to have fine-grained tracking of abnormalities.
    
  Spatial-temporal level detect abnormal usage related to specific days and hours and, possibly, provide users some feedback to reduce energy waste.


A relevant challenge in this field is defining what exactly is an anomaly: it can be described as the event that occurs when the power usage is an outlier, both regarding a peak in power usage or a too-long electricity consumption; but this definition doesn't take in account the fact that if an appliance is running for a long time-span, for example, long cooking in the oven, then it can also not be an anomaly.
The aim is the production of systems that have good accuracy and consider all the possible abnormal usage.

A current problem is the lack of annotated datasets: the existent ones are mostly not labeled or contain only a small amount of anomaly samples, because of the rarity of anomalies in reality.




§ DIRECT VS INDIRECT CONTROL
 
 introduced a new distinction in IEMS classification:

    
  Direct Control IEMS (DCIEMS) includes all systems with the ability to modify the environment using actuators; these systems are able to process data, make decisions, and execute them, without the intervention of a human being.
    The advantages of Direct Control IEMS are that the occupants don't need to alter their daily routine and these systems can be involved in fault detection. Furthermore, they are better suited for people with reduced mobility.
    
    
  Indirect Control IEMS (ICIEMS) instead puts humans in the position of the actuator, forming a human-in-the-loop architecture.
    These systems aim to change the behavior of end-users to stop energy-wasting habits. To achieve this goal, suggestions are provided to the users through interfaces.
    This approach is cheaper since there is no need to buy actuators like smart sockets for every appliance; moreover, by inducing an eco-friendly behavior in the users, they will apply the acquired habits in all the other aspects of their lives, like at their workplace.


Both the IEMS types produce a lot of data: Direct Control IEMS through sensors, and Indirect Control IEMS through feedback from end users.
AI tools can be used to process this quantity of data as described in section "sec:big_data".
Machine learning tools can extract from such data the behavioral usage patterns of the users, predicting their future needs and personalizing the recommendations for each one of them.
Deep learning tools can monitor the energy profile of the environment and derive information about the energy consumption of appliances.
They are also able to make predictions about the next consumption in a determined time window.



§ FUTURE PROPOSALS
 
Until today only a few studies have been conducted on the use of Deep Learning in the energy efficiency field, in the authors' opinion this should be the focus of future research.
A possible approach could be to combine Deep Learning with Reinforcement Learning to enhance the learning done by Deep Learning Neural Networks in DCIEMS through both the refinements provided by a reward function based on the environment changes and the feedback provided by the human.

Although the aforementioned proposal, the authors recognize that a major issue in the implementation of DL systems is the computational power requirements that are needed in training/fine-tuning, but also at inference, such power is not always available in the buildings due to the high cost of the required hardware.

Using DL, the "Explainability" issue arises, since the user must understand the actions (DCIEMS) or suggestions (ICIEMS) from the system in order to be compliant with them. An optimal system should be able to explain the reasons behind its output in a way that the user can understand.

Another proposal could be to implement gamification experiences in ICIEMS architecture to engage users in the energy-saving process, making them more aware of their energy consumption and more willing to change their habits both at home and in other buildings.
An example implementation could be a mobile application that rewards the user with points for every energy-saving action taken; the user can then compete with other users globally or locally against occupants of the same building.




§ CONCLUSION
 
This paper has reviewed the literature on the use of AI and its different techniques in the energy efficiency field, focusing on Big Data produced by IoT, Anomaly Detection with AI, and the different approaches in interaction with humans and buildings.

Energy efficiency is a very important topic in current society since, as many studies pointed out, it can lead to a reduction in CO2 emissions and, consequently, to a reduction of climate change.
This also reflects on people's health:  found that for every dollar saved on energy expenditures by smart buildings, 77 cents are saved in health and climate benefits.

All the reviewed papers pointed out problems in AI systems, especially the privacy aspects because models know a lot of information about the users and the buildings, and such information can be used to harm inhabitants; in case of data leaks malicious people can harm users by knowing if they are at home.
A way of enforcing users' privacy in global models training is using anonymized data or, even better, using federated learning which helps to train algorithms over various de-centralized edge servers without sharing data.

In general, these systems are not even immune to attacks: in particular, DCIEMS are sensitive to attacks since they rely on IoT devices; a possible attack can be Denial of Service (DoS) and False Data Injection (FDI), which can affect electricity bills and load consumption.






 Paper Title: DisTrack: a new Tool for Semi-automatic Misinformation Tracking in Online Social Networks.


Enhancing Ethereum Fraud Detection via Generative and Contrastive Self-supervision
    Chenxiang Jin, Jiajun Zhou^∗, Chenxuan Xie, Shanqing Yu, Qi Xuan, Senior Member, IEEE, Xiaoniu Yang


This work was supported in part by the Key R&D Program of Zhejiang under Grants 2022C01018 and 2024C01025, by the National Natural Science Foundation of China under Grants 62103374 and U21B2001. (Corresponding author: Jiajun Zhou.)
C. Jin, C. Xie, S. Yu and Q. Xuan are with the Institute of Cyberspace Security, College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China, with the Binjiang Institute of Artificial Intelligence, ZJUT, Hangzhou 310056, China. E-mail:{jincxiang, 221122030330, yushanqing, xuanqi}@zjut.edu.cn.
J. Zhou are with the Institute of Cyberspace Security, College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou 310023, China, with the Binjiang Institute of Artificial Intelligence, ZJUT, Hangzhou 310056, China. E-mail: jjzhou@zjut.edu.cn.


X. Yang is with the Institute of Cyberspace Security, Zhejiang University of Technology, Hangzhou 310023, China, and also with Science and Technology on Communication Information Security Control Laboratory, the 36th Research Institute of China Electronics Technology Group Corporation, Jiaxing 314033, China. Email: yxn2117@126.com.










    August 5, 2024
===================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================

§ ABSTRACT 
Introduction: This article introduces DisTrack, a methodology and a tool developed for tracking and analyzing misinformation within Online Social Networks (OSNs). DisTrack is designed to combat the spread of misinformation through a combination of Natural Language Processing (NLP) Social Network Analysis (SNA) and graph visualization. The primary goal is to detect misinformation, track its propagation, identify its sources, and assess the influence of various actors within the network.

Methods: DisTrack's architecture incorporates a variety of methodologies including keyword search, semantic similarity assessments, and graph generation techniques. These methods collectively facilitate the monitoring of misinformation, the categorization of content based on alignment with known false claims, and the visualization of dissemination cascades through detailed graphs. The tool is tailored to capture and analyze the dynamic nature of misinformation spread in digital environments.

Results: The effectiveness of DisTrack is demonstrated through three case studies focused on different themes: discredit/hate speech, anti-vaccine misinformation, and false narratives about the Russia-Ukraine conflict. These studies show DisTrack's capabilities in distinguishing posts that propagate falsehoods from those that counteract them, and tracing the evolution of misinformation from its inception. 


Conclusions: The research confirms that DisTrack is a valuable tool in the field of misinformation analysis. It effectively distinguishes between different types of misinformation and traces their development over time. By providing a comprehensive approach to understanding and combating misinformation in digital spaces, DisTrack proves to be an essential asset for researchers and practitioners working to mitigate the impact of false information in online social environments.








§ INTRODUCTION


At present, the impact of misinformation on our societies is beyond question. Governments, companies, researchers, among many others, are putting a great deal of effort into providing solutions and limiting the damage caused by this problem. The challenge, however, lies in the multifaceted nature of misinformation, making a comprehensive approach to combat it elusive. Addressing misinformation is contingent upon numerous factors, notably the underlying intent. Thus, when we use the word “misinformation”, we are referring to information that is false by definition or unintentionally false information <cit.>, whereas “disinformation” is used when there is a deliberate intention to misinform.<cit.>, following the differences underlined by Karlova and Fisher in 2013 <cit.>. In addition to these terms, there is the concept of “malinformation”, which groups every content created as a weapon <cit.>.

In order to understand the phenomenon of misinformation in the current era, and beyond intentionality, it is essential to include the means of transmission in the equation. As such, social media, particularly Online Social Networks (OSNs), has risen as society's primary channel for accessing information. Social media has transformed information dissemination from direct peer-to-peer exchanges to expansive many-to-many propagations, altering societal content consumption habits and thereby easing the spread of false information <cit.>. Literature has demonstrated with specific cases how social media fosters falsehoods easily <cit.>. 

The combined effects of OSNs' dominance as information sources and the growing disenchantment with traditional news media amplify concerns about misinformation's influence on the public. According to the Reuters Institute Digital News Report, there's a notable decline in the practice of consulting a variety of news outlets and a general waning interest in news over the years. This trend includes the avoidance of topics like climate change and the Ukraine invasion by specific segments of the audience. This study underlines the contrast between worrying about misinformation and then depending more on social media to receive information <cit.>.  

Despite the lack of expert consensus on the recent escalation of misinformation <cit.>, academia has witnessed a significant surge in research articles tackling this issue, reflecting its growing impact <cit.>. In the years before the pandemic, these academic works had already increased exponentially <cit.>. Subsequent studies associate the surge in academic interest regarding misinformation, particularly from 2017, with events like the 2016 US elections <cit.>, suggesting these events catalyzed but did not solely trigger the focus <cit.>. After these contributions to research, the emergence of more relevant international events and their originated misinformation highlights the importance of tackling this problem. 

In the fight against this phenomenon, fact-checkers represent the foremost defense. They are responsible for verifying thousands of pieces of information daily, cross-checking them and issuing statements on social media to counter the spread of false information. The International Fact-Checking Network (IFCN) built a “Code of Principles” to establish the criteria for the task of debunking misinformation. During the coronavirus crisis, the then Associate Director Cristina Tardáguila highlighted an unprecedented volume of misinformation, presenting a novel challenge for fact-checkers compared to prior years <cit.>. Recent research on COVID-19 misinformation has shown the emergence of misinformation waves, with a plethora of hoaxes spreading rapidly, complicating efforts to address them all at once <cit.>. Evidence like this demands new automated mechanisms not only at the level of verification but also in the following steps of mitigation on OSNs for a coordinated response against emerging falsehoods.

Mitigating the problem of misinformation therefore requires a complex approach where all factors of the problem are considered. In addition to the verification of the content itself, it is necessary to take into account how it is spread, the actors involved in its distribution and, in general, any element that participates in the cascade of misinformation propagation. In Fig. <ref> the described cascade is represented, where several actors interact over time about a given falsehood. We want to find and appropriately characterize this misinformation cascade, allowing for a better understanding of online discourse. As a result of considering all these factors, we draw the following research question for our work: “Is it possible to track conversations around specific hoaxes on Twitter (X)?” which we sub-divide into four subquestions to answer it: 





    
  * Can we extract the conversation about a hoax on Twitter (X)?
    
  * Can we separate tweets related to the hoax in the extracted conversation from tweets not related to it?
    
  * Can we distinguish between hoaxes that propagate a hoax from those that contradict it?
    
  * Can we trace the movement of tweets related to a hoax with the users that spread them from beginning to end?













We integrate cutting-edge Language Models and graph generation techniques to explore content propagation within Online Social Networks (OSNs) through the lens of Social Network Analysis (SNA). The social platform chosen for this objective is Twitter (X), given its dominant use for information: in proportion, it is more used than other social media for consuming mainstream news but also smaller or alternative news sources, and politicians and political activists, too <cit.>. Furthermore, Twitter surpassed the other networks in this survey for the consumption of news about politics in the country and the Russia-Ukraine war, among others. This suggests the impact of false information and polarization could be disorders of these informative and political interests in this ecosystem and, thus, there would be a need to avoid them. 

This research enables not only the detection of misinformation but also the tracking of its journey from inception to conclusion within a Twitter (X) conversation, distinguishing between posts that propagate the falsehood and those that refute it. Our architecture is designed with two primary objectives: first, to combat misinformation by doing more than just verifying the veracity of a claim—we aim to identify every tweet that supports it; second, to monitor and trace the sources of misinformation. In the context of X, these sources of misinformation will be the tweets stating a false claim, not only the first tweets but all tweets that contribute to this, since all of them harvest this misinformation in the circles they belong to, whether the users inside them continue sharing it or not. This monitoring enables the analysis of the influence of various tweets and the identities of the individuals disseminating them.


Our methodology leverages a semi-automated fact-checking architecture <cit.> that avoids the pitfalls of relying solely on a fully automated system. The traditional automated method, which categorizes information as true or false based on a pre-defined database, suffers from two major issues: it relies on a static collection of data that may not keep pace with emerging hoaxes, and it tends to categorize new, unverified information based on existing patterns, possibly mislabeling them. In contrast, our semi-automated strategy mitigates these issues by utilizing a dynamic database curated by fact-checkers that specifically targets false claims and can be regularly updated with new misinformation. Additionally, this method prioritizes the evaluation of claims in the database that align with the content of new information being assessed, ensuring a more accurate and timely verification process.

This integration of fact-checking organizations with AI techniques enables the automated identification of posts that are part of a hoax, as flagged in the fact-checkers' database. Distrack's semi-automated method advances Social Network Analysis by innovatively creating graphs that categorize the elements that create a cascade of information on social media according to their relationship with misinformation or with its debunking. This approach refines the analysis of viral content flows by filtering out irrelevant data, a significant departure from prior methodologies that did not distinguish between relevant and unrelated content. 

This article presents the following contributions:


    
  * An integration of NLP and SNA to combat misinformation. The study integrates cutting-edge language models and graph generation techniques to explore content propagation within online social networks (OSNs), specifically focusing on Twitter. 
    
  * A graph generation architecture for tracking misinformation. The tool creates graphs that categorize the elements of information cascades on social media into three types: supporting a specific database claim, contradicting it, or being unrelated. This approach refines the analysis of viral content flows by filtering out irrelevant data and provides a comprehensive view of the spread of misinformation.
    
  * Case studies evaluation. The paper presents three case studies that illustrate the application of DisTrack in tracking different types of false information related to topics such as COVID-19 vaccines, political issues, and more. These case studies demonstrate the tool's versatility and effectiveness in misinformation tracking across various subjects.


The product of this research is conceived as a cognitive computation system, according to its purpose of replicating human problem-solving <cit.>. In this paper, it entails the human tasks of detecting false information, discovering all the posts related to it and analyzing the evolution of them, towards the final tracking of falsehoods. Through the computational methods presented, these manual processes are automated, assisted through AI.

More specifically, Natural Language Processing is cited as one of the areas in charge of cognitive systems and analyzing content in OSNs as one of the examples of computerized cognitive abilities. Likewise, the extraction of knowledge with Social Network Analysis is one of the techniques mentioned in Decision Support Systems (DSS), being these a combination of the outcomes of complex data analyses and machine learning <cit.>.

This paper continues as follows: Background section reviews the key concepts and models used in this research; the next section describes the three modules in the methodology, the tweets extraction procedure, Natural Language Inference and graph generation; after this, the following section disentangles three examples in which this described methodology has been set into practice. The answers to the research questions are derived from these evaluated use cases, composing the Results and Discussion section that substantiates the final conclusions in the last part and the future lines of work.



§ BACKGROUND



In this section, earlier contributions to the visualization of misinformation are explored. Discovering the veracity of a claim is crucial to visualizing the dissemination of misinformation because there are several lines of discourse surrounding any false-information claim, with fact-checkers on one side and misinforming actors on the other. It is expected that any information cascade about a hoax is surrounded by two different and opposing narratives.

Our research is motivated by earlier works performing semi-automated fact-checking using transformer-based language models, able to detect whether pieces of content (or claims) are either factually fake or have undetermined veracity. Following the previous rationale we motivate our techniques by exploring language models applied to fact-checking, as well as Social Network Analysis (or SNA) to accurately portray the dissemination of misinformation across online social networks (OSNs).





 §.§ Language Models








The development of machine learning and deep learning models in the field of Natural Language Processing has made it possible to deal with complex tasks related to Natural Language Understanding (NLU). One of the most important steps was the emergence of language models with the introduction of the attention mechanism, leading to the development of Transformer models like BERT <cit.>, RoBERTa <cit.>, and XML <cit.>. Unlike earlier embeddings (like word2vec <cit.> or Glove <cit.>), Transformer models generate vectorial representations using contextual information from neighboring words in the surrounding text, where each word is semantically informed by the sentence.

These advances opened the era of Language Models (LMs), architectures trained for tasks such as predicting the next word, but designed for multiple NLP problems. Among these, there are many scenarios where LMs can be deployed to fuel fact-checking processes, providing significant improvements over traditional machine learning methods <cit.>. For example, these models facilitate the automation of fact-checking by employing binary classification to discern false facts within the input. The state-of-the-art literature shows promising results in this line of work <cit.>. Furthermore, this approach can be refined beyond simple binary outcomes by using varied labels to provide more nuanced distinctions between types of information <cit.>.







 §.§ Automated fact-checking




Misinformation is an ever-shifting issue. New pieces of misinformation may emerge as time passes, new narratives may become misinformation whereas older known hoaxes become real after an unexpected world event happens. Automated models trained without information retrieval techniques will inevitably become obsolete within the span of a few months. Allowing a model to retrieve information from trusted sources allows for proper decision making <cit.>. In contrast to this, an alternative approach arises where the dataset is conceived as a knowledge base <cit.>. In this source, the data consists of textual statements containing verified falsehoods. Using these falsehoods, a model can compare an unverified claim against any verified falsehood and if there is any match, we can determine that the unverified claim also contains a falsehood. In this structure fact-checkers have a double role: they are responsible for the curation of the database, as well as the interpretation of the model output, giving complete control of the semi-automated model to fact-checkers and responsibility of its application <cit.>.


The process has two steps: information retrieval (IR) and Natural Language Inference (NLI). For IR, one of the most commonly used methods is the calculation of the semantic distance between semantic embeddings <cit.>. This approach does not depend on a preliminary dataset of posts on the social network chosen to assess the veracity of an unseen post on that platform, allowing the classification of texts that belong to messaging environments in which full datasets are rarely obtained, such as WhatsApp <cit.>. The advances in this type of pipelines in the era of coronavirus encouraged research to focus on refining that knowledge base for that specific context <cit.>. Nowadays, tools in the fact-checking process based on the cosine similarity of texts have already been implemented in newsrooms and show their success over other methods <cit.>.

The second step in this fact-checking process is determining the alignment between retrieved falsehoods and the original unverified content <cit.>. This alignment can be applied through the use of Natural Language Inference (NLI) as a subset of Natural Language. The task of NLI consists of checking if Sentence A (hypothesis) is inferred from Sentence B (premise) <cit.>. In essence, this involves demonstrating that both sentences make the same assertion. The relationships obtained from NLI are the following: entailment, when A and B refer to the same statement; neutral, when A and B are not related to each other, and contradiction, when A and B refer to the opposite statement <cit.>. 


The classification of statements in this task is performed by feeding the algorithms with datasets specifically designed for it. Stanford Natural Language Inference corpus (SNLI), with 570,000 pairs of sentences annotated with one of the three categories mentioned above <cit.>, stands as the main reference for NLI, but further datasets solve the drawbacks of SNLI: for example, MultiNLI, with texts extracted from images cutlines, presents a more enriched text <cit.>, or XNLI, with its cross-lingual approach, does not restrict NLI to one language <cit.>. The use of NLI datasets enhanced with Transformers facilitates the comparison among languages <cit.>.



 §.§ Social Network Analysis



Studying and mitigating the problem of misinformation involves detecting the misinformation itself, but also tackling the pathways by which it spreads. Thus, understanding how a piece of misinformation is disseminated on a social network is a vital tool. 

The flow of social media posts on Online Social Networks (OSNs) can be effectively represented as a graph. This graph-based structure organizes data into vertices linked by edges, providing a clear visualization of complex relationships <cit.>. Within this framework, vertices represent either users or their posts, while edges illustrate the myriad interactions or relationships between them. These connections encompass the explicit social interactions derived from the network's metadata but, additionally, the more subtle, latent properties that link posts together.

Social Network Analysis (SNA) is the area in charge of studying these graphs from social platforms. The directions in this discipline comprise both the extraction of the common features of networks and the identification of aspects from the users from the graph <cit.>. Algorithms can be trained with this information, using, for example, the dynamics of likes <cit.>, to distinguish between types of posts. Misinformation arises as one of the emerging domains of application of SNA, together with politics and multimedia, being fields such as marketing, tourism, healthcare or cybersecurity more settled in this sort of studies <cit.>.

Surveys in the field of misinformation have highlighted that the characteristics inside the post are one of the indicators to detect falsehoods, but the properties of the OSN itself play an important role. Sharma et al. <cit.> enumerated key elements such as the source/promoters, user responses and the information content parts. On the other hand, Parikh et al. <cit.> specified non-text cues-based methods in the fight against false information, with user behaviour analysis as one of the subareas covered. In 2018, different studies demonstrated how false pieces of information were disseminated much further on Twitter than those that were true, by looking at properties such as the depth of the cascades generated by the post, the accounts contributing to spreading them and the duration of the propagation <cit.>. However, in the COVID-19 context, graph analyses revealed this expansion was only in terms of vastness: both false and legitimate information have the same influence, but actors spreading false information post more than those publishing the true one <cit.>.

However, regardless of the temporal context, repetitive patterns can be found. Before the coronavirus, SNA showed that individuals' decisions related to vaccines could be influenced inside circles debating about vaccination <cit.>, indicating that the connections among users on these platforms matter to the extent of dangerous implications if the communities approached are anti-vaccine. These contagions from one group to another bring the issue of virality, taken from the propagation of viruses. The creation of a cascade responds to one of the models of infection, the viral model where infected vertices by others can exchange the virus too, in contrast to the broadcast models in which contagions derive from a main vertex <cit.>.

The role of the main spreaders in these cascades of misinformation among these circles has also been the focus of social-media-driven analyses. The change of information from one community to another through 'super-spreaders' and their characteristics were also disseminated with graphs <cit.>. Verified Twitter accounts (in the era before Elon Musk ownership and X) were shown to be 50 times more powerful in terms of propagating content about vaccines in comparison to non-verified profiles <cit.>.

To conclude, the use of NLP models without considering the dynamics of the social network allows us to visualize only part of the problem, leaving out key details <cit.>. Every instance of misinformation is surrounded by a community of users who interact with, share, comment on, support, or dispute it. Relying solely on content analysis limits our capabilities, overlooking the crucial task of unraveling the impact of false claims on Online Social Networks (OSNs), which extends beyond mere verification to include users' responses. The combination of NLP and SNA leads to a more realistic picture of the whole conversation of each falsehood, a technique yet to be sufficiently explored in the literature as we have seen in this section. 






§ TRACKING MISINFORMATION IN OSNS
 
NLP and SNA can represent an alternative map of misinformation in OSNs through all the posts about a claim spreading a falsehood. In it, false information appears from viralization, but also from messages of different shapes from a wide to a short range of interactions and from a variety of users, not necessarily with the same impact in terms of their popularity in the social network. As an example, this approach allows to model the contagion from one vertex to the rest <cit.> in two senses: on the one hand, NLP-driven research has demonstrated that there is not a unique message repeated in the propagation of misinformation, but many of them expressed differently; on the other hand, SNA-oriented studies show unconnected users outside the cascades also distribute falsehoods on social media.

Nevertheless, these studies contradicting the only focus on broadcast models <cit.> in their combination of NLP and SNA are limited to the analysis of the data of the properties from the social network chosen in their final goal. Graph generation is ignored and the possibility of representing these ecosystems of falsehoods is missed. This results in an obstacle between the theory that confirms how misinformation is not just a cascade and the practice of representing what it is instead. This mentioned practice, in contrast to the previous approaches, would go further than the demonstration of a different model of diffusion of misinformation to reveal a more realistic flux of the posts causing it, examining its origin, evolution and end, if applicable.



 §.§ The DisTrack architecture

The main goal of DisTrack is to create a complete representation of the propagation cascade of a piece of misinformation. To do so, it integrates different language models and SNA techniques that allow building a graphical representation of this cascade, providing information about the content and the actors in the social network that have played an essential role in the dissemination. DisTrack consists of three main sequential steps:

 

 
  * Information retrieval from OSNs: this module comprises the extraction of relevant keywords, the generation of queries through their possible combinations and the use of Twitter API to download all the tweets.
 
  * Semantic and Natural Language Inference: this module refers to the conversion of tweets into Transformer-based embeddings that capture their meaning and context and the extraction of metrics based on their inference (if a tweet supports a false claim, contradicts it or is unrelated. For this second step, we make use of FacTeR-Check <cit.>, which implements a semantic similarity filtering process followed by Natural Language Inference.
 
 
  * Graph generation: this module ends with the hydration of the tweets downloaded to extract the insights to be used in the graph and the creation of it. Its vertices will be the tweets extracted and whose edges will correspond to the interactions among them. After this, the particularity of DisTrack is the use of the NLP-related and Twitter-related metrics to show the rest of the properties (position, size or colour).
 



DisTrack leverages the concept of tracking by modeling a graph based on a set of tweets downloaded from the OSN and labeled according to the alignment with a specific claim. This modeling process includes information extracted from the OSN such as following between users, retweets, or replies. As a result, the mechanisms of DisTrack can be distributed in three modules (see Fig. <ref>).

 

The final output of these three modules is a visualization that acts as the operation center for the supervision of each piece of misinformation from the beginning to the end, assessing about the flux of a certain claim, the most influential spreaders and tweets with their connections or the periods in which this false information has had more impact.

This architecture allows us to characterize the discourse surrounding any piece of information. The visualization aims to highlight specific phenomena in social media discourse, as well as provide answers to questions for policymakers and content moderators. We provide some examples of these properties:

    
  * Proliferation (and decay) over time of falsehoods on social media. How does a piece of content propagate across time in an OSN?
    
  * Impact of fact-checkers on the discourse surrounding a falsehood. Do they contribute to the proliferation or the decay of the surrounding discourse?
    
  * Relationship between the influence of a social network actor and their impact on the discourse. How much do influential accounts dominate online discourse? Do smaller accounts have any impact on the evolution of the cascade?
    
  * Detection of possible astroturfing campaigns. Has the falsehood spawned from several disconnected accounts? Is a coordinated attack on the social network being performed?


This division into modules gives them independence in performing a task. For example, the extraction of Twitter- and NLI-related insights without the need to follow the trajectory of false information has already been applied, without any further implementation, to discover the type of tweets, according to their interactions, that represent the true volume of misinformation beyond viral posts. These modules are explained in the subsections below.

To enable our research we begin exploring Twitter [now X, however we will call it Twitter for the sake of simplicity and understandability.]. This will be useful to evaluate our method on a heterogeneous set of use cases, but it is important to note that our methods can be applied to any social network that enables API search of textual content.



 §.§ Retrieving Twitter Content


The process of extracting information from Twitter requires the execution of a series of searches for certain keywords. Since the Twitter API restricts the search to the exact keywords that are given as input, in order to retrieve a large representative sample of the relevant tweets and interactions related to one specific claim, it is necessary to build a set of multiple queries. By using different words and expressions in each of these queries it is possible to cover a large part of the tweets referring to the claim.





  §.§.§ Keyword search

The query is the input to search and download tweets through Twitter API. These downloaded posts will contain the keywords inside that query. Through logical operators, queries can request the API tweets with every keyword inside it or for every tweet that at least has one of the keywords. However, this process is manual and the downloaded tweets are the result of the subjective decision of typing a query with a set of keywords instead of another. How these keywords are distributed with the logical operators to optimize the search is also an arbitrary decision. Furthermore, thinking of all these steps of converting a false claim into a query to download as many tweets as possible requires time, slowing down the computer-based fight against misinformation.

To solve these drawbacks of the manual creation of a query, an information retrieval module to generate queries and automate this process is needed. This research follows FacTer-ChecKey <cit.>, a designed automated search to extract the most relevant keywords from a claim and concatenate them through the logical operator “AND” to generate the final query. For example, the sentence “Massive protest in France against the mandatory implementation of the COVID passport in public spaces” results in the concatenation of keywords “(protest AND france AND passport AND covid AND public)”. This development takes inspiration from the model KeyBERT <cit.> and it uses multilingual Transformer-based models to take the context from the meaning of a claim to extract the keywords from it, in addition to Name Entity Recognition (NER) to improve this extraction for those cases in Spanish, when needed.

This automated extraction faces the problem of the subjective decision of building the query manually, and they also deal with the subjective wording of the claim itself, from which the automated query is generated. In practice, including all the important keywords in these generated queries may not lead to the expected results because there can be tweets with some of the keywords from the claim but not necessarily all of them. Synonyms, abbreviations or word camouflage are examples of figures of speech contributing to a difficult keyword search.

We present a refined keyword extraction that tries to widen the search space and minimize the obstacles of this process, introducing three major add-ons: 

    
  * The introduction of the logical operator OR in the query, to allow multiple combinations of the keywords for the automated search
    
  * The inclusion of a parameter that indicates the number of keywords that will be discarded from each combination of words, being the result of the query a concatenation of all the possible mixtures excluding two different keywords in each of them. In the example previously seen, the query would be optimized in this way: “((protest AND france AND passport AND covid) OR (protest AND france AND passport AND public) OR (protest AND france AND covid AND public) OR (protest AND passport AND covid AND public) OR (france AND passport AND covid AND public))”. 
    
  * The conversion of numerical numbers into all the possible forms, also textual, to avoid ignoring pieces of misinformation that include figures cited in a different way (e.g., “10000 OR 10,000 OR 10.000 OR `10 thousand’ OR `ten  thousand’”)


In this case,  model is used in the query generator in parallel with FacTer-Check, to encode the meaning together with the context of the words, and includes the model  for Name Entity Recognition (NER). For this implementation, adverbs, conjunctions, adpositions and other stop words have been removed for the final composition with the multilingual Flair tagger and Spacy models.

Although the default settings could capture how a certain hoax has been diffused massively in a short time-lapse. For the sake of a better understanding of social media discourse, we are interested in extracting the maximum possible number of texts, actors and connections involved in the early steps and also the evolution of every falsehood, given the dimensions of misinformation on social media nowadays.



  §.§.§ Technical details

The query generated through the claim of each piece of misinformation constitutes the input of Twitter API, whose access is offered through a developer account on Twitter. 

The reduced limitations granted by the academic API access soften two restrictions: the maximum number of tweets extracted and the time of the start of the final download. This last aspect can be modified to obtain information from a certain timestamp. This becomes crucial in the field of misinformation because the search must be constrained after the birth of the topic that is referenced in each hoax (for example, claims about COVID will not be found prior to the emergence of coronavirus).

Overall, the created query, the credential to validate the permissions (the token found on the developer account) and the selected time-lapse and maximum number of posts as parameters are included for the automated request for the needed tweets about misinformation. 
The data of each downloaded tweet consists of a JSON with Twitter-based information structured as metadata in different fields. 

However, since this cannot be exploited enough, NLP-based features in the following step will contribute to filtering non-related hoaxes and developing the final tool. These technical details describe a fixed frame of the use of the API at the moment of the experiment, but the current and future restrictions of this developers' system do not distort the steps of this research if the API is not available. Twitter versions of the API are a hands-on automated solution based on X advanced search, available through the search bar on the interface. What the API returns in terms of content is the same as the results that the interface gives to the user, also with the same query as input. For this reason, parameters such as the chosen time-lapse can be selected on X advanced search too.

The difference involves the codification of the output: whereas the API offers the massive download of tweets in JSON format, the advanced search gives the visual integration of those posts in the interface, which would need further processing to transform them as raw data and to structure their metadata.



 §.§ Automated Verification



Once we have retrieved a sample of information from the OSN related to the input claim, each tweet is labeled according to the alignment with the original claim. We define alignment as the result of performing Natural Language Inference over a pair of content (the retrieved tweet from the last step) and falsehood (the piece of misinformation that is being tracked). After evaluating each retrieved Tweet we assign their labels.



  §.§.§ Natural Language Inference


Natural language inference (NLI) is crucial to distinguish if a tweet is aligned or contradicts a false claim. Again, Transformer-based architectures are applied to measure this alignment regardless of how different each tweet is formulated in comparison to the false claim selected.

The NLI task consists of discovering if a hypothesis h can be inferred from the premise p in a pair of texts (p, h). In the misinformation and fact-checking domains, p will be each tweet from the conversation downloaded from Twitter and h will be the piece of misinformation debunked by fact-checkers. Thus: h is h_f when stating that falsehood, and h is h_u when that factuality is undetermined.

The classification of posts according to their content through NLI is: 


  * Entailment (when the falsehood is enunciated): a post that entails a piece of misinformation is a post that supports it and spreads it.

  * Contradiction (when the negation of the falsehood is enunciated): a post that contradicts a piece of misinformation is a post that denies it and may act as a protective shield in the circles where it arrives.

  * Neutral (when the falsehood or its contradiction is not enunciated): a post that is neutral bears no relevance to the discourse whatsoever a secondary effect of widening the spread of the keyword search.











  §.§.§ Technical details

The semantic search uses the methodology proposed in FacTer-Check step by step without ad-dons or modifications. For the NLI task, a fine-tuned  <cit.> was used for this module. For the NLI task, the Machine Translated MultiNLI (MNLI-MT) <cit.> and XNLI <cit.> datasets were used. Additional datasets such as ANLI <cit.>, SNLI <cit.> and FEVER <cit.> for English have been also included, using two training processes (inspired by FacTeR-Check): one only with MNLI (for the cross-lingual texts) and one with all the mentioned datasets. The hyperparameters chosen are: 1024 as batch size, 2e-5 as the learning rate, with Adam <cit.> as the optimizer. Same for warmup and linear decay. The validation data in XNLI determines the optimal selected network after a manual hyper-parameter finetuning[Model available at https://huggingface.co/AIDA-UPM/xlm-roberta-large-snli_mnli_xnli_fever_r1_r2_r3AIDA-UPM/xlm-roberta-large-snli_mnli_xnli_fever_r1_r2_r3 ].



 §.§ Graph visualization



Our main contribution lies in the graph visualization module where we conceive how to translate an interconnected graph of tweets into a useful visualization of online discourse around a topic. Our DisTrack architecture makes use of all the properties stored in each tweet to make a readable composition, clearly showing how any piece of misinformation has evolved.



  §.§.§ Cascade graph building

The information stored for each tweet includes the preceding author and the preceding tweet from which it derives. This information is enough to generate a directed graph G = (V, E), with V the vertex set containing tweets published by an author. In this graph, the E edges represent a connection between vertices due to being either a reply, quote or retweet from another tweet, which will be the parent vertex. 

Each vertex has additional information contained by the contextual metadata of the author, likes among other details. Conveying this metadata to make it understandable is a non-trivial challenge that we address in the following subsections. In particular:

    
  * Time is contained within the metadata, but the information is so unevenly spaced over time that the interpretation of the cascade is compromised.
    
  * The cascade of misinformation is perpetrated mainly by actors that influence online discourse. Representing their influence accurately in the visualization aside from the vertices is crucial to understanding the actual impact of actors.
    
  * The information cascade of misinformation is twofold, containing usually two opposing narratives, the misinformation itself and the fact-checkers combating it. Greater insight can be achieved by using the results of NLI and integrating them into the visualization to evaluate the veracity of the evaluated claims.





  §.§.§ Non-linear time representation

The X-axis is defined by time. Each vertex will be placed according to the moment it has been published. However, time itself is not represented linearly in this axis. Falsehoods act in waves, the flow of information may be quick and sudden during a short time-lapse generating concentrated content, preventing the user from reading the evolution of that piece of false information properly when it explodes in terms of impact. Furthermore, if there is much distance in time from one vertex to the next one, again the space in the graph between these two points will create empty areas in the plot whereas the areas filled would be crowded with vertices (during the propagation of a hoax, there are moments of inactivity or a lower number of publications about it). Our approach just sorts posts chronologically, regardless of the time passed between one vertex and the next one, and sets different time stamps on the x-axis that improve the understanding of the propagation cascade.



  §.§.§ Author influence representation

Regarding the y-axis, it measures the degree of influence of the author of a vertex/tweet. The number of followers allows us to distinguish between the influence of the different actors responsible for the spread of falsehood or their contradiction. 

However, the achieved impressions by the tweet are also affected by the number of likes, in addition to retweets and quotes, which are already represented through the children vertices. For this reason, likes will affect the size of each vertex proportionately in the final visualization. Obtaining this would result in a visualization that shows the evolution of a conversation of the tweets extracted from a query but also the explanation of it through the impact that users, retweets and likes generate.



  §.§.§ Veracity representation

Finally, the outputs of NLI will serve as the colours to differentiate posts containing misinformation from those that contradict or are unrelated. This is the advance that allows us to check the beginning, transformation and current status of a hoax rather than just all the tweets downloaded through the main keywords appearing in a specific hoax, and that would culminate in the goal of tracing the conversations about misinforming posts, if the hypothesis is confirmed.





§ CASE STUDIES



Three use cases illustrate the application of DisTrack. They contain the beginning and evolution of a piece of misinformation on Twitter, all of them with a different topic to show the versatility of this tool. Firstly, an exploratory data analysis of the downloaded tweets is made to disentangle the types of tweets and of their authors according to their NLI- and Twitter-based metrics. After this, as the proof of the variety of tweets contributing to the expansion of content related to misinformation, the representation of the final graphs is made thanks to the final module of graph generation from DisTrack.




These three cases represent three different topics:

    
  * Case 1: “The 80 percent of Muslims living in Europe live from social welfare
and they refuse to work”. The first case is linked to the disbelief in institutions and hate against Muslims. It constitutes 32 original tweets and a positive balance of tweets involving entailment in contrast to contradiction (i.e., denying the hoax). The weight of entailment increases much more when every post is shared. Thus, this case contains a total of 84 representative posts, with all the retweets included.

    
  * Case 2: “RNA vaccines against coronavirus includes graphene oxid”. It focuses on COVID-19-related antivaccine statements. It also includes 32 original tweets and a balanced number between entailment and contradiction. This second case has a total of 128 representative posts, including retweets.

    
  * Case 3: “Zelensky sold 17 million hectares of land to Monsanto, Dupont and Cargill”. It is an example of misinformation around the Russia-Ukraine war. It departs from 26 original tweets and most of the total tweets represent entailment with the hoax (80%). This third case involves a total of 916 tweets.



 

The specific distribution of the number of tweets supporting or denying each false claim is represented in Figure <ref>. Cases 1 and 3 are examples of the general trend that can be observed in the dissemination of false information. The presence of posts that support misinformation is much higher than those that contradict the news, mostly due to the activity of fact-checker accounts. In contrast, during the pandemic, we could see how a large part of the social media community actively participated in countering false information, as in the example of case 2.

 












































In order to evaluate the impact of a post on a social network, the number of retweets or likes are two of the most commonly used indicators. As can be seen in Fig. <ref>, in terms of retweets, tweets from one to ten reposts are the main group, followed by those with zero retweets. Regarding likes, most of the posts in the flux of misinformation do not contain any interaction, in contrast with the lower percentages of tweets between one and ten likes. Tweets between one and ten likes are a small proportion except in the case of the hoax related to the vaccines, and only the propagation of the misinformation cited about the war between Russia and Ukraine has a tweet with more than 100 likes.

 

We also analyzed the number of followers of the user accounts involved in these three case studies (see Fig. <ref>, showing different types of users. The original tweets in the three fluxes about misinformation are mainly shared by users from 1000 or more followers. Whereas the propagation related to the hoax about discredit/hate has more users between 1001 and 10.000 followers than those that are below or above these numbers, the other two examples indicate that authors between 101 and 1000 correspond to the largest proportion. 









    
    
    
    
    

























































In the following subsections we show and describe in detail each of the three case studies. In the graphs, posts are rounded vertices, sized by the number of likes, whereas reposts are vertices with the shame of a rhombus. Posts and reposts tagged as entailment are orange, those tagged as contradiction are blue and neutral ones are grey. The x-axis represents the chronological order (with annotated dates each 100 days from the first post in the visualization as a guide) and the y-axis (log scale) shows the number of followers of the authors of each post and repost.



 §.§ Case 1: Discredit/hate

 








For this first case (see Fig. <ref>), we focus on a hoax that circulated on Twitter asserting that “80 percent of Muslims living in Europe live from social welfare and they refuse to work”. For this hoax, we retrieved a large pool of information from the social network, including mentions to other users, popular or not, fake attributions to researchers to make the text trustworthy and personal views and opinions reinforcing that hoax. 

Likewise, fact-checks announce the same sentence in the negative form to combat it, as seen in the graphs generated. The variations in this case mention the name of the accounts of fact-checking and/or add links to the news debunking the hoax. It is important to note that the same statement to negate the false information is shared repeatedly at different moments in the life of the hoax.

The most retweeted post affirming the hoax took place on April 2nd, 2022, with 22 retweets, however, we can see how this false information was originally posted around 10 months before. On June 8th, 2021, an account with 3,881 followers (at the time of the data extraction) published it and received a unique retweet. It took exactly five months up to that date to see the second most retweeted post supporting this claim, with 11 retweets, and this type of misinformation resists at least until October.

The most interesting fact about the two users with more than 10.000 followers who actively wrote the contradiction of the hoax is that both of them reproduce the same action several times: one of them (25,464 followers) does it three times; the other one (12,537 followers), four. Active spreaders posting misinformation by themselves have a lower number of followers. However, in the case of retweets, there are actors with more than 10,000 followers that spread this falsehood. Maldito Bulo, from the fact-checker Maldita, emerges as the user that exceeds that number of followers as a repost of its fact-checking through another user.

This confirms that our system shows the whole cascade of misinformation and not just the static picture of the most propagated tweet in April 2022. Furthermore, this visualization also demonstrates that any virality of a post, regardless of its impact, can be preceded by posts with zero or a few active interactions (retweet or like), as shown in this case in 2021. Finally, it also reveals how after the most retweeted post supporting the claim, the flux of false information continues with different users and in different ways up to the ones shown in the last trimester of 2022.

Regarding the involvement of the specific actors that spread misinformation (see Table <ref>), several accounts with an important number of followers posted the hoax. It is interesting to note, however, that the ones that had the most interactions, and therefore caused the most discussion around the falsehood, were some with a smaller number of followers.



 §.§ Case 2 - Antivaccine

 

The propagation of how falsely RNA vaccines against coronavirus include graphene oxide has also had an important impact on Twitter and other social media. This hoax has succeeded through paraphrasing in different periods. Whereas the previous case study showed a fixed structure and shape, this one discovers a range of posts expressing the same falsehood with different words and tones, from more declarative sentences to more aggressive ones.








Again, fact-checks are mostly composed of the same sentence, including small variations mentioning the fact-checking source and/or the link. In some specific cases, mainly in the first interactions of fact-checkers to contradict false information, we can observe longer explanations, substantiating the falsity of the fact. 

In this particular case study, we can see that the first tweets composing the propagation cascade deny the hoax. This surprising effect is present in those cascades where the origin is in other environments (i.e., chain messages on WhatsApp). The chain of propagation originates on 6 June 2021, and about a month later we can observe some tweets disseminating the hoax. It is noteworthy that the tweet that has the greatest effect on the dissemination of the hoax (16 September 2022) occurs after months in which the only references to the hoax are neutral. 





Analyzing in more detail the accounts involved (see Table <ref>), the two accounts that typed the contradiction of the claim have more than 10,000 followers, but none with this influence actively expresses the hoax itself (the most influential one has 9,547 followers). Close to these numbers, the last original tweet sharing this piece of misinformation had 9,547 followers. Regarding retweets, Maldito Bulo, from the fact-checker Maldita (300,081 followers at that moment) played an important role in this hoax.

This case study reveals the limitations of traditional methods, which might have analyzed this publication in August or its final iteration in September, without considering the origins or evolution of the misinformation involved. By adopting this new approach, we gain insight into the broader context of how misinformation spreads. It shows that the groups opposing a piece of false information represent just a segment of the overall dissemination of posts linked to misinformation, as it continues to spread among other individuals who either refute or challenge the assertion. The spread of the latest hoax on September 16, which occurred without any users debunking it, underscores this point. This example highlights the complexity of misinformation propagation and the necessity of considering its full lifecycle for effective analysis.





 §.§ Case 3: Russia-Ukraine



This third case, shown in Fig. <ref>, considers the hoax “Zelensky sold 17 million hectares of land to Monsanto, Dupont and Cargill”. Unlike the earlier cases, the visualizations here highlight variations in impact, illustrating the hoax's dissemination through a multitude of messages. Echoing the patterns seen in the previous example, there is not a single form of post; instead, a diversity of presentations emerge, ranging from the use of hashtags and user mentions to varying styles and tactics to engage the audience. The spread of this misinformation is primarily driven by reposts, yet these varied expressions of the same false claim also play a significant role, particularly at the initial stages of its spread. This multiplicity of formats and channels underscores the complex nature of misinformation propagation and the challenges in tracing and countering it.


This example showcases a viral post from September 19th, 2022, that stood out with 663 retweets and 1,000 likes. Remarkably, its retweets occurred not just on the day of the original tweet but continued sporadically until December 30th of the same year, although diminishing impact. This trend is visualized along the x-axis and highlights both the immediate impact of the hoax and its prolonged presence in the digital discourse.


Thanks to this visualization, abnormal activity is shown on the same user who writes the viral post. This suggests that this person retweets it several times, according to the download of Twitter API data. This continuous activity preserves the propagation of this false information and prevents it from dying on Twitter (X). In addition, there was another post with even more retweets whose initial user has been deleted or suspended (1.597 retweets), as there are no edges linking them to the original post. 








In this case, the virality of the post on September 19th already shows the vast propagation of this misinformation, how it arrives to other users throughout time, how it can be the continuation of other viral posts in the past, and even the strategies led by users to keep the impact of a tweet. This viral post has the same publication date as others with their own content that also spread content about the hoax. In Table <ref>, it can be seen how one user is the one leading the whole propagation cascade, spreading the hoax while being the center of a high number of interactions.










§ RESULTS AND DISCUSSION



In every analyzed example, DisTrack answers positively to the question “Can we extract the conversation about a hoax on Twitter?”. The conversations about the three chosen falsehoods have been visualized after downloading and processing and refining the data and metadata of the posts related to them. This demonstrates the success in generating automated queries to extract as many tweets as possible about false claims to later filter them depending on their type of alignment (entailment, contradiction or neutral).

This leads to the affirmative answer to the second subquestion “Can we separate tweets related to the hoax in the extracted conversation from tweets not related to it?”) and to the third one (“Can we distinguish between hoaxes that propagate a hoax from those that contradict it?”). The application of NLI leveraged with Transformers has been satisfactory at two levels: not only by separating tweets that are not related to each false claim, but also by separating the entailment posts (those that state misinformation) from contradiction posts (those that deny them).

In particular, the use of Transformers in the application of NLI to detect misinformation in this case has allowed DisTrack to identify falsehoods on X, regardless of the way they were exposed in the platform. In case 1, the posts were similar to the false claim taken as the reference, with subtle changes, but in cases 2 and 3, the posts paraphrased the content of the false claim together with hashtags, links or other elements, and their entailment was also guessed by the NLI model. In these downloaded conversations about misinformation, The existence of accounts reproducing exactly the same original content and of those that paraphrase it instead encourages research to unveil the dynamics of falsehoods and the existence of bots in contexts of crises <cit.>. 

Thanks to the whole process of searching tweets, aligning them with the validated claim and visual representation, what we present is a complete architecture that allows the automated and detailed analysis of the spread of disinformation on a social network. From the first references to the hoax, to the nodes that most influence its dissemination (thanks to their number of followers, for example) or the patterns observed in the cascade, DisTrack allows us to understand first-hand the dynamics of disinformation and how it spreads and generates a certain impact.



The fourth research question “Can we identify the users involved in the conversation of a hoax from beginning to end?” is also answered affirmatively. DisTrack shows a closer image of the whole life of a piece of misinformation: viral tweets matter, but all the participants in the conversation are relevant. In the three case studies analyzed, falsehoods appear before and after the spread of the most retweeted posts, in the shape of less viral tweets. This research reveals the characteristics of every actor inside the ecosystem of false information: the case studies unveil users with many followers as actors in the propagation of a hoax, those conceived as 'super-spreaders' <cit.>, but also accounts with different weights in terms of followers, not always the most followed ones <cit.>.

The results of the four subquestions allow us to answer the main research question: “Is it possible to track conversations around specific hoaxes on Twitter (X)?”. The three use cases confirm that misinformation can be traced on Twitter by modeling it, using the three presented modules: firstly, the tweets of the conversation about falsehoods were extracted through automated queries (subquestion 1); secondly, they were separated from each other according to their alignment to that false information (subquestions 2 and 3), and, finally, their authors and their characteristics were also identified (subquestion 4) through the final generation of graphs that show the evolution of that falsehood.

These outcomes follow recent research relying on tweet extraction and NLI-based classification in the field of misinformation. These studies show the orchestration of posts with different ranges of influence contributing to a more complex propagation, and also the different nature of users in the conversation about specific falsehoods, in accordance with the results provided by DisTrack. The exploratory data analysis of the three presented use cases was a demonstration of this variety of posts and users.

Nevertheless, DisTrack adds a layer of research through the generation of the graphs to exploit more the synergies between NLP and SNA as the fields in charge of combating misinformation <cit.>: a falsehood does not die progressively after its more viral publication and is not always born directly from it, and fact-checks are also repeated in the successive lives of each falsehood. In the cases analyzed, the conversation is reshaped and hoaxes arise again. Furthermore, the evolution of tweets from fact-checkers and of those that refuse verbally the selected hoaxes does not only occur with the virality of that type of misinformation at its best, but also in other periods, as observed in the final visualizations of this research. 

The combination of falsehoods and the posts that debunk them in a final visualization is also worth mentioning. DisTrack proposes an additional step to the precursory work: whereas previous experiments contribute to a better knowledge of the nature of the posts about misinformation, they do not offer any surveillance to monitor and mitigate it. Nevertheless, DisTrack brings back those steps of the tweets extraction and NLI filtering to put them at the service of graph generation for that desired supervision. This enables the coordinated response of fact-checkers and the control of their effects on false information.

Overall, DisTrack reinforces the need to study misinformation as a viral model <cit.> in a chain of infections by posts with different ranges of influence. Its opposed model covered in research, the broadcast model, would only have shown a part of misinformation because it only conceives contagion as a unique primary vertex infecting the rest. On the contrary, the examples made by Distrack print several versions of the vertex, various infections and, as a consequence, many propagations instead of one. 

This is also important in the combination of entailment and contradiction posts. Not only do broadcast models <cit.> isolate a cascade of misinformation from the rest of the generated false information, but they also put it apart from the posts that contradict it and from fact-checkers, preventing researchers from depicting their appearances in the ecosystem of that cascade. Likewise, a broadcast-model cascade of fact-checks without showing the rest of the conversation through the steps of DisTrack would remove them from the falsehoods they counteract.



§ CONCLUSIONS



All in all, our work releases a line of action through the shape of DisTrack, as the beginning of a tool able to merge Transformer-leveraged tweet extraction, NLI-driven tagging of misinformation from the posts retrieved and graph generation of Twitter-based and user-based properties in an output that shows chronologically the evolution of a conversation motivated by misinformation (spreaders, fact-checkers and other users) across the different publications and actors involved. 

With this proposed line of action, future experiments can study how DisTrack modules can be modified. With NLP as one of the core parts of this research, there is room for improvement given the advance of new Language Models (LM): on the one hand, with their application for new models that capture better the topics and, thus, the keywords for the query that enable the download of posts; on the other hand, with their use for NLI to increase the level of accuracy in the classification of posts as entailment, contradiction of neutral, the three tags used to colour the vertices in the final graphs to build the representative picture of misinformation.

Future work includes the application of DisTrack beyond misinformation. For instance, previous research shows how sentiment analysis has become relevant in the studies about aggressive discourse in the context of government elections  <cit.> and has stated Twitter as a “sentiment thermometer” through VADER  <cit.>. Although the first steps of these experiments evoque the implementation of DisTrack, involving the download of posts from X and the extraction of features for the study of specific behaviours, the part of monitoring polarization through the combination of NLP and SNA through graph generation is missing, unlike DisTrack. For this reason, future experiments could be oriented to develop the same modules as DisTrack but by substituting the task of detecting false information in favour of analyzing sentiment in this platform, a space with more proportion of politicians and political activists than others <cit.>.

Accordingly, this research opens the door to the development of the part of tracking in other scenarios, where NLI does not have to be necessarily excluded and can enrich the information in the graph (in the previous case, for example, to know the users stating the same political information and, thus, enunciating the same message to the audience for the elections). These other scenarios can include: a more general approach, beyond politics, to take more advantage of the advances of Transformers for hate speech detection  <cit.> and follow the trajectory of harmful posts on social media; an alternative approach in the domain of author profiling, where graphs can leverage recent research in LLM for this task <cit.> by tracing information and linking the vertices that are likely to have the same attribution, or the area of topic modeling, fueled by models such as BERTopic <cit.>, in which the concatenation of the modules proposed by DisTrack could result in edges tying the posts about the same issue, among other fields of study.

DisTrack also serves as an initiative to explore platforms beyond Twitter (X). The international survey developed by Reuters Institute Digital News Report in 2022 already revealed the prevalence of Facebook, YouTube, Instagram or TikTok as well as WhatsApp, Telegram or Facebook Messenger as ways of consuming news <cit.>, which has continued in 2023 <cit.>. Each of them has a different structure but DisTrack arises as a proposal to be adapted to other scenarios of misinformation on these ecosystems.

Although OSNs mutate or change, with Twitter, now X, as an example of that, users continue searching platforms to be in the circles of their ecosystems. Research about migrations from Twitter to Bluesky, Mastodon and Threads has already been covered <cit.>, showing the interest in the current microblogging social media and, thus, the necessity of fighting against their information disorders, motivated by tools and methodologies such as DisTrack.






§ DECLARATIONS




 §.§ Competing Interests


The authors declare no competing interests.



 §.§ Funding


This work has been funded by the project PCI2022-134990-2 (MARTINI) of the CHISTERA IV Cofund 2021 program, funded by MCIN/AEI/10.13039/ 501100011033 and by the “European Union NextGenerationEU/PRTR”; by the research project DisTrack: Tracking disinformation in Online Social Networks through Deep Natural Language Processing, granted by Mobile World Capital Foundation; by the Spanish Ministry of Science and Innovation under FightDIS (PID2020-117263GB-I00); by MCIN/AEI/10.13039/501100011033/ and European Union NextGenerationEU/PRTR for XAI-Disinfodemics (PLEC 2021-007681) grant, by European Comission under IBERIFIER Plus - Iberian Digital Media Observatory (DIGITAL-2023-DEPLOY- 04-EDMO-HUBS 101158511); and by EMIF managed by the Calouste Gulbenkian Foundation, in the project MuseAI.

elsarticle-num






 Paper Title: A Qualitative Study on Using ChatGPT for Software Security: Perception vs. Practicality.





















A Qualitative Study on Using ChatGPT for Software Security: Perception vs. Practicality
    
    M. Mehdi Kholoosi12, 
    M. Ali Babar12,
    Roland Croft12
    
    
    1 CREST - The Centre for Research on Engineering Software Technologies, The University of Adelaide, Adelaide, Australia
    2 Cyber Security Cooperative Research Centre, Australia
    
    Emails: {mehdi.kholoosi, ali.babar, roland.croft}@adelaide.edu.au
    August, 2024
==================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Artificial Intelligence (AI) advancements have enabled the development of Large Language Models (LLMs) that can perform a variety of tasks with remarkable semantic understanding and accuracy. ChatGPT is one such LLM that has gained significant attention due to its impressive capabilities for assisting in various knowledge-intensive tasks. Due to the knowledge-intensive nature of engineering secure software, ChatGPT's assistance is expected to be explored for security-related tasks during the development/evolution of software. To gain an understanding of the potential of ChatGPT as an emerging technology for supporting software security, we adopted a two-fold approach. Initially, we performed an empirical study to analyse the perceptions of those who had explored the use of ChatGPT for security tasks and shared their views on Twitter. It was determined that security practitioners view ChatGPT as beneficial for various software security tasks, including vulnerability detection, information retrieval, and penetration testing. Secondly, we designed an experiment aimed at investigating the practicality of this technology when deployed as an oracle in real-world settings. In particular, we focused on vulnerability detection and qualitatively examined ChatGPT outputs for given prompts within this prominent software security task. Based on our analysis, responses from ChatGPT in this task are largely filled with generic security information and may not be appropriate for industry use. To prevent data leakage, we performed this analysis on a vulnerability dataset compiled after the OpenAI data cut-off date from real-world projects covering 40 distinct vulnerability types and 12 programming languages. We assert that the findings from this study would contribute to future research aimed at developing and evaluating LLMs dedicated to software security.







security, software vulnerability, large language model




§ INTRODUCTION



Software security remains one of the most significant challenges facing modern software development. With the rise of agile development practices, a focus is being placed on developer-centric security assurance <cit.>. However, studies have shown that developers fail to keep up with the ever-growing list of required security knowledge and expertise <cit.>. Secure coding alone requires developers to learn and use thousands of security-related patterns, practices, and tools. Some areas of software security, such as Software Vulnerability Management (SVM), need specialised knowledge and expertise for detecting, assessing, patching, and disclosing vulnerabilities <cit.>. Consequently, developers tend to leverage recommended systems and/or social media platforms to gain and use knowledge about software security <cit.>. 

Chatbots, i.e., conversational agents, provide effective interfaces for enhanced information retrieval and task assistance <cit.>. Natural language comprehension offers a potentially game-changing interface facilitating human and computer interaction. Natural language can greatly improve the efficiency in which we perform complicated tasks and use complex technologies by allowing us to operate them with simple natural language queries. 

Like many other domains, such as education and marketing <cit.>, chatbots are gaining significant traction in the software development domain for obtaining knowledge/information related to specific software development tasks, such as API usages <cit.>. However, the use of chatbots for software security is still in its infancy stage <cit.> as current software security chatbots use simple rule-based technologies with limited capabilities <cit.>. This situation is expected to change as the advancements in Large Language Models (LLMs) enable much more powerful and capable chatbots, e.g., ChatGPT. 

ChatGPT is an Artificial Intelligence (AI) chatbot developed by OpenAI[https://openai.com/blog/chatgpt/] created using GPT (Generative Pretrained Transformer)<cit.>, a state-of-the-art LLM. These technical advancements allow ChatGPT to surpass traditional chatbots through its capability to provide answers and solutions to complex queries
<cit.>. 
Training and fine-tuning an LLM with billions of parameters (e.g., 175 billion in GPT-3<cit.> and 1,700 billion in GPT-4<cit.>) presents considerable challenges due to the extensive computational resources required <cit.>. Since ChatGPT was released to the public, this limitation has been addressed, and many users were able to experience LLMs for the first time. Whilst ChatGPT was not specifically developed for the software domain, it still capture intrinsic knowledge that enables software security tasking to a certain degree <cit.>. We believe exploring the capabilities and limits of this intrinsic knowledge is extremely beneficial for end-users as most organisations and practitioners lack the required resources to teach domain-specific knowledge to LLMs. Hence, we intend to investigate ChatGPT's intrinsic knowledge in software security to help enable the use of chatbots in this area in the future.


We conducted a multifaceted exploration of perceptions and practicality to investigate ChatGPT's potential for supporting software security tasks. First, we analysed public discussions of ChatGPT users regarding software security to gain insights into the successes and failures of LLM-based chatbots for secure development. In the subsequent phase of our study, we focused on vulnerability detection as it was the most anticipated security task among practitioners, according to the first step of our analysis. 
Through qualitative analysis of ChatGPT's responses, we obtained an overview of the types and manners in which information (e.g., descriptions and assessments) is provided and examined the responses for potential practicality issues. Using this two-staged comprehensive assessment, we uncovered not only the current capabilities and shortcomings of ChatGPT in the context of software security but also demonstrated how practitioners' expectations differ from the practical application of this technology in the real world. We assert that the findings from this study are expected to guide future endeavours that aim to develop specialised LLMs for security-related tasks.



§ RELATED WORK









 §.§ AI-Based Chatbots for Software Security.

Chatbots have been demonstrated to be suitable for software engineering tasks such as information retrieval <cit.> and API usage <cit.>. However, the use of chatbots for software security is limited. To date, the SKF-Chatbot (Secure Knowledge Framework) is one of the leading software security chatbots, which was developed by the OWASP Foundation[https://owasp.org/] to provide access to Software Vulnerability (SV) information via a chat interface. However, its capabilities are limited due to its lack of contextual understanding <cit.>. Recently, AI-based chatbots have been able to overcome these limitations via the LLMs that power them. ChatGPT has thus far demonstrated remarkable capabilities for semantic comprehension and providing tailored solutions for given tasks <cit.>. 




 §.§ Vulnerability Detection Using Transformer-based Language Models.

Prior studies have explored the use of transformer-based language models to identify software vulnerabilities using a variety of fine-tuning methodologies. For example, Thapa et al. <cit.> evaluated  transformer-based language models (i.e., BERTBase, GPT-2 Base) against recurrent neural network (RNN)-based models (i.e., BiLSTM, BiGRU) for vulnerability detection in software datasets featuring C/C++ source code. According to their findings, transformer-based language models outperformed RNN-based models on all evaluation metrics. In another study, Kalouptsoglou et al. <cit.> fine-tuned various transformer-based models (i.e., BERT variants, GPT-2, BART) and carried out a comparative analysis to determine which of these models is the most appropriate for vulnerability detection.
Furthermore, Chan et al. <cit.> created a vulnerability detection model targeted at detecting vulnerabilities in incomplete code snippets. They leveraged common learning approaches such as fine-tuning on three pre-trained LLMs, namely CodeBERT, code-davinci-002, and text-davinci-003. 



 §.§ Vulnerability Detection Using ChatGPT.

Several research efforts have explored the efficacy of ChatGPT in vulnerability detection. 
Chen et al. <cit.> exclusively probed ChatGPT's capability in identifying smart contract vulnerabilities. Szabó et al. <cit.> focused on identifying vulnerabilities associated with CWE-653 (Improper Isolation or Compartmentalization) using multiple GPT models. Ozturk et al. <cit.> assessed ChatGPT's effectiveness in detecting the top 10 OWASP vulnerability categories in web applications. Zhang et al., <cit.> through comprehensive testing on two Java and C/C++ 
vulnerability datasets, demonstrated that prompt-enhanced approaches could strengthen ChatGPT's vulnerability detection capability.
In a recent study, Fu et al.<cit.> compared the capabilities of ChatGPT (i.e., GPT-3.5 and GPT-4) against three other LLMs (e.g., CodeBERT<cit.>) across various vulnerability tasks. The LLMs used in this study were fine-tuned explicitly for code-related tasks.

There are two main ways in which our study differs from prior studies:


    
  * Prior research has primarily focused on assessing <cit.> and enhancing ChatGPT's vulnerability detection performance through prompt engineering methodologies <cit.> or the integration of supplementary elements <cit.> within the LLM pipeline.
    We instead strive to focus more on insights through qualitative analysis of existing real-world uses and outputs. This shift aims to illuminate the disparity between practitioners' perceived value and the practicality of this technology in real-world settings. We provide further details in Section <ref>.
    
    
    
    
    
    
  * Prior studies that utilised datasets from existing literature <cit.> or open-source projects <cit.> ignored the possibility of data leakage (i.e., overlap between training and test data). This factor could artificially inflate the reported performance of AI models' (e.g., ChatGPT) <cit.>. Consequently, we curated a high-quality vulnerability dataset tailored to our evaluation criteria from unseen data. Section <ref> discusses this in more detail.
    




§ RESEARCH DESIGN


Initially, we present our research questions in Section <ref>. This is followed by a detailed explanation of the data collection methodology in Section <ref>. Lastly, Section <ref> delves into the qualitative analysis processes.






Figure <ref> displays the overall workflow that we used to conduct this study.




 §.§ Research Questions.


The following Research Questions (RQs) motivated our empirical study: 

    
  * RQ1: What are users' perceptions of using ChatGPT for software security? We examine user demographics, topics, and types of discussions to determine how ChatGPT is being applied by users and its potential strengths and weaknesses.

    

    
    
    
    

    
  * RQ2: How practical are ChatGPT's outputs for the most desired software security task? We perform a qualitative analysis of ChatGPT's outputs for the most discussed topic of RQ1 to ascertain the practicality of this technology in real-world settings.



 §.§ Data Collection.




  §.§.§ Collecting Twitter Data and Curating a Sample
 For the first data source, we selected the popular social media site, Twitter[https://twitter.com], for our investigation due to its active use by developers to discuss emerging technologies <cit.> and its use in former research for uncovering public sentiments <cit.>.


We used the Twitter API to collect English language tweets that matched the following keyword search query: . We collected tweets that were posted between December 1, 2022, and February 28, 2023, the first three months of ChatGPT's public release. 

In total, we collected 7716 tweets.











To enable qualitative analysis of our data, we curated a sample of posts for manual examination. To ensure that our data sample was of sufficiently high quality, we selected the 700 most liked tweets (approximately 10% of the collected tweets) from our dataset. Twitter likes are given by users who appreciate or agree with a tweet. Hence, highly liked tweets are influential posts that reflect the viewpoints of a larger audience and represent broader public sentiments. We label these tweets T1-T700.



  §.§.§ Collecting Vulnerability Data and Curating a Sample


To perform the required analysis for RQ2, 
we needed to test ChatGPT's capabilities in real-world settings. This purpose was achieved by using the National Vulnerability Database (NVD) <cit.>, an established repository for SV management data maintained by the National Institute of Standards and Technology (NIST). This database holds an exhaustive array of information on identified SVs in the wild. 

As we planned to do an in-depth manual analysis for RQ2, 
we decided to work with a subset of 70 SVs.
This sample size was of statistical significance, providing 90% confidence and an error margin of 10% <cit.>. Previous researchers who needed to perform manual analysis over SVs also chose a similar sample size <cit.>. 

It is imperative to note that we considered the issue of data leakage (i.e., contamination of training and test data) in our data selection for RQ2. 
It is well known that ChatGPT has been trained using a great deal of publicly available data. Since NVD is a public database, it is possible that ChatGPT encountered NVD data during its training phase. Thus, to prevent data leakage, we only included SVs in our dataset with a published date (i.e., the official date on which a vulnerability is added to the NVD) after Sep 2021 (the knowledge cutoff date for ChatGPT at the time of conducting this study). Any newly discovered SVs after this date have not been seen by ChatGPT during training.

Additionally, three other factors were considered when curating the random sample of 70 SVs. First, as the fixed code was required for validation purposes in our analysis, the SV report on NVD must be directly linked to the GitHub commit that addresses the SV (i.e., fixing commit). The References section of each SV in NVD usually includes a link to the fixing commit. The second consideration is that the fixing commit should be focused and not tangled (i.e., a commit that consists of multiple unrelated changes). 

The last considered factor is that the vulnerability should be contained within a single function. The last two factors are because we planned to perform all analysis in the following steps manually, and tracing complicated commits is an error-prone process.

After incorporating the aforementioned factors into our SV selection criteria, we ended up with a dataset (containing 70 SVs) that covered 40 distinct CWEs (Common Weakness Enumeration) <cit.>, and its affected products (i.e., products and platforms considered vulnerable) were written in 12 different programming languages. 
































By utilising the provided fixing commits, we carefully collected the entire vulnerable and fixed (i.e., patched) versions of functions from the relevant GitHub repositories. The average number of lines of code among our 70 collected vulnerable functions was 32.



  §.§.§ Collecting ChatGPT Outputs
 

In this step, ChatGPT was fed the 70 collected vulnerable functions as input and asked about potential SVs within them. The GPT-4 model has demonstrated superior performance compared to the GPT-3 model in various benchmarks <cit.>. Considering the importance of delivering high-quality and up-to-date research insights in the critical domain of software security, all ChatGPT-related analysis in this paper were performed using the GPT-4 model. 

Prior research has shown that tailored prompts are essential to taking full advantage of LLM's capabilities in various software engineering tasks<cit.>. 
Following their prompt designs, for each of the SVs in our dataset, we utilised the following prompt template in our conversation with ChatGPT and carefully saved its outputs 
for further analysis. As ChatGPT remembers information from previous inputs, each query was made in a new chat. 





[colback=blue!5!white,colframe=blue!75!black,title= Prompt Template]
  Role: You are a software security expert.
  
  Instruction: Please analyze the following code snippet for potential security vulnerabilities. Provide a detailed explanation of the issues you find.
  
  Context: <code-snippet>




 §.§ Qualitative Analysis.




  §.§.§ Qualitative Analysis of Sampled Tweets


To understand user perceptions of ChatGPT for software security (RQ1), we qualitatively analysed our sample of 
700 tweets using thematic analysis <cit.>. This was an iterative and reflective process that we conducted over several phases. 

We started by immersing ourselves in the data by carefully reading each tweet and all surrounding data: other tweets in the thread, replies, attached media, linked articles, and associated user profiles. The first and third authors conducted this pilot data labelling process collaboratively for 30 tweets, and it typically took 15 to 20 minutes per tweet. During this process, we posed potential characteristics of the discussion that address RQ1: user demographics, topics and types of discussion. To determine the values of these characteristics, we performed open coding <cit.> to classify the common discussion points. We 
collaboratively identified keypoints as short summaries for each tweet. We then revised these into themes, which provide a one or two word description of the common topics. We conducted a second round of pilot data labeling on an additional batch of 30 randomly selected tweets to ensure thorough coverage of potential themes. In this iterative process, no new themes emerged, indicating that thematic saturation had been reached. 


Once we were satisfied with the identified themes, two authors independently analysed and categorised the remaining 640 
tweets in the sample. They then compared the annotated data to ensure data coding was conducted consistently. The two raters achieved a Cohen Kappa value of 0.813 <cit.>, which implies a high level of agreement. Any remaining disagreements between the first and third authors were resolved through discussion with the second author. During the labelling process, we excluded any tweets from our sample that were unrelated (434 tweets) to the use of ChatGPT for software security due to falsely matched keywords, e.g., tweets discussing the impact of ChatGPT on job `security'.

Ultimately, 266 tweets were included in the sample.

We also wanted to gain an understanding of general user sentiments toward ChatGPT's use for software security. E.g., whether the author of a tweet had a positive experience using it or was optimistic about its potential applications. We experimented with using the NLTK sentiment analysis tool <cit.> to classify our data automatically. However, we found this tool to be inaccurate from a manual review of 30 sample tweets. This is because security keywords such as vulnerability or security were often falsely interpreted as indicating negative sentiments. Furthermore, a tweet may have an overall positive sentiment but a negative opinion of ChatGPT, and vice versa. For example, “This article does a really good job of highlighting the weakness of ChatGPT.” Hence, we manually classified user sentiments of ChatGPT during qualitative analysis. For each tweet, we labelled the sentiment as either positive, negative, or neutral based on our interpretation of the user's opinion or experience of ChatGPT capabilities for software security. We reserved the neutral sentiment label for tweets that discussed positive and negative points or were ambiguous to identify. The nature of the sentiment of a tweet depends on the type of discussion (e.g., experiences vs. speculation). Hence, we consider the types of discussion whilst performing sentiment analysis, which we elaborate further on in Section <ref>.





  §.§.§ Qualitative Analysis of Sampled Vulnerabilities


To thoroughly investigate the practicality of ChatGPT 
in the vulnerability detection task (RQ2), we first needed to assess ChatGPT's efficacy in this task. This is because we were interested in exploring how characteristics (e.g., tone) of outputs differ when ChatGPT is accurate versus instances of inaccurate outputs.
To 
this end, we manually analysed the output of all 70 queries 
and provided an assessment as to whether the target SV was detected. This manual assessment was performed independently by the first and third authors of this paper who had a cumulative eight years of both academic and industrial software security experience.

For each SV within the dataset, we used the fixing commit as a baseline for our comparison and employed techniques from prior researchers <cit.> to manually analyse the code changes related to a SV. These techniques are explained step by step as follows. In the first step, we concentrated on the SV summary from the NVD Description section. Moreover, we focused on the information available in the source code (i.e., fixing commit). Aside from code changes, the title and message of the fixing commit were also examined, as they usually provide valuable information regarding the functional aspects of the code and the context in which it was altered. Having thoroughly understood the SV, we assessed 
its generated ChatGPT output. 
Accordingly, each output was labelled as either not found or found. Most often, ChatGPT provided a list of potential vulnerabilities within the input code, with the particular SV we were interested in being mentioned in one of the bullet points. These outputs were classified as found. If the output did not contain information about the vulnerability we sought, it would be marked as not found. This is even if the other potential vulnerabilities ChatGPT listed appeared genuine. This decision was because we wanted to maximise the reliability of our manual analysis and we did not have the ground truth (i.e., fixing commit) for comparison for the other detected vulnerabilities.



  §.§.§ Qualitative Analysis of ChatGPT Outputs


In response to RQ2, we conducted a thematic analysis <cit.> of ChatGPT's outputs regarding the vulnerability detection task. In particular, our primary goal was to demonstrate the breadth and depth of information that a conversational agent such as ChatGPT can offer and examine the nuances of its presentation.
For the thematic analysis of this step, we followed a similar approach as explained in Section <ref>. The first and third authors started by collaboratively performing a pilot data labelling of 10 ChatGPT outputs, which constitutes 14% of the 70 outputs in our dataset. During this process, we focused on the types of information ChatGPT provides and the way in which it is presented. After that, the two authors put more than 100 hours each into this qualitative evaluation and independently analysed the remaining 60 outputs in the dataset. In a supervisory role, the second author reviewed subsets of the analysis results every week to ensure their validity and suggested modifications to improve the assessment accuracy where deemed necessary. Implementing such supervision was necessary to mitigate potential biases and discrepancies in the analysis.

This qualitative analysis was performed with NVivo 12.0 software<cit.> to streamline theme finding. 



§ RESULTS





 §.§ RQ1: perceptions of using ChatGPT for software security


We firstly wanted to determine the occupations of users of ChatGPT for software security. Such insights provide essential context for the source of discussions and sentiments in our data. Furthermore, we reveal the invested stakeholders of AI-based chatbot solutions for software security. We analysed user profile descriptions of the tweet authors (n=202) and identified any occupation details that were listed. We then categorised user occupations using open coding as described in Section <ref>. The population of authors was analysed as a set not containing duplicates. This means that a user was only considered once in the population if they authored multiple tweets. We identified five main occupations in our sample data, which we describe below:


    
  * Security Practitioners are individuals (n=78) that conduct software security or cybersecurity jobs or tasks. This includes occupations such as security analysts, hackers, and vulnerability researchers. 

    
  * Security Companies are the Twitter profiles of organisations (n=27) that provide software security solutions and services. 

    
  * Software Practitioners are individuals (n=19) that perform software development or associated tasks for their occupation. 

    
  * Blogs are the profiles for online sites or organisations (n=46) that provide news outlets or blogging services. 
    
    
    
  * Other defines a collection of occupations which had too few samples to form individual categories. These include academics (n=6) and government employees (n=5). User profiles that contained insufficient or vague descriptions of occupation (n=21) were also placed in the Other category.











Table <ref> displays the occupations of Twitter users in our sample. Given that our dataset is comprised of software security discussions, security practitioners are expected to form the largest user demographic (39%). However, this provides evidence that security practitioners are interested in the potential applications of AI chatbots to their domain, which is supported by the noticeable demographic (13%) of commercial security companies that are also discussing the viability of such tools. Furthermore, the high percentage of security practitioners in our data indicates that our Twitter sample contains reliable information from users with background domain knowledge. 

We next wanted to examine the topics of discussion in our sample Twitter data to understand the areas of software security for which ChatGPT was being used. These insights were expected to help us determine the potential strengths and weaknesses of AI chatbots for different software security tasks. From our qualitative analysis of tweets described in Section <ref>, we discovered five main software security categories in which users utilise ChatGPT, which are also displayed in Table <ref> alongside their frequency. We go into detail for each category below. We provide quoted text from the sample tweets that we have analysed. However, many tweets provide associated media to provide further context to the discussion, which we cannot capture within our paper.





































Vulnerability Detection. Vulnerability detection is one of the most important components of SV management. Consequently, it is the most frequent discussion category (29%) within our sample of tweets. Many users were impressed that ChatGPT has even basic capabilities for vulnerability detection due to the difficulty of this task:






    “Experiment time. I fed ChatGPT the Damn Vulnerable Ethereum Smart Contract Migration code in it's entirety. It pointed to four potential vulnerabilities!!!” (T95)



Whilst many users demonstrated that ChatGPT could successfully identify vulnerabilities in simple code snippets, some users were sceptical about the validity of its use in real-world scenarios. 


    “I just went three rounds with a "security researcher" (who was chatGPT in disguise) making reports on our smart contracts. Plausible text and impacts wrapped in magnificent misunderstandings of the basics.” (T49)


ChatGPT produces plausible-sounding answers even for tasks which it cannot necessarily complete successfully. Vulnerability detection requires substantial expertise, so it is hard for many users to verify responses for these tasks. The accuracy of ChatGPT may also be limited as it was not trained to perform these tasks.  


    “Don’t use ChatGPT for security code review. It’s not meant to be used that way, it doesn’t really work (although you might be fooled into thinking it does)” (T80)


However, despite this lack of trust, users still found the vulnerability analysis capabilities helpful in speeding up code review. 


    “I had ChatGPT take code ... and then audit for security. Would I deploy the resulting code? No. But two weeks just became minutes.” (T150)


Vulnerability Exploits. Users have also explored leveraging ChatGPT to develop vulnerability exploits for penetration testing or offensive security purposes. These capabilities help security testers verify vulnerabilities, which exceeds the functionality of many traditional static analysis tools.  


    “Write an exploit code that abuse brute-force vulnerability on [One Time Password]. ChatGPT result is mind blowing. Half of the pen tester I know are not even close to that level of well/detailed reporting skills” (T67)


However, a serious concern is that ChatGPT can be similarly used to enable malicious and unethical hacking. 


    “In one instance, a hacker shared an Android malware code written by ChatGPT, which could steal desired files, compress them, and leak them online.” (T117)



    “...ChatGPT can turn anyone into a ransomware and malware threat actor.” (T163)


Such uses would have severe consequences for the state of software security due to the increased ease of cyber attacks and exploits. We observed that discussions about malicious hacking were primarily contained in the Speculation discussion category. Hence, further investigation is required to validate these fears and claims. 

Information Retrieval. Users have also used ChatGPT for software security information retrieval and education. 


    “Currently using ChatGPT to help me find and research security solutions for my organization. This has been the biggest game changer for my career since Python.” (T131)


However, similar to vulnerability detection, some security experts were sceptical about the quality of information returned by ChatGPT. A substantial concern is that it is challenging to detect false information provided by ChatGPT.


    “People are excited about using ChatGPT for learning. It's often very good. But the danger is that you can't tell when it's wrong unless you already know the answer. I tried some basic information security questions. In most cases the answers sounded plausible but were in fact BS.” (T1)


Code Analysis. ChatGPT can also be used for other secure code analysis tasks, such as understanding security patches, reverse engineering, malware analysis, or decompiling assembly. 


    “...We used ChatGPT to analyse the patch to understand the buffer overflow.” (T38)






Other. Finally, users have also theorised the potential utility of ChatGPT for other security tasks, such as threat modelling, vulnerability disclosure, secure configuration, and implementation of security controls. These topics appeared infrequently which implies that did not receive as much attention or interest, but users were still often positive. 


    “... we've not really explored its use-cases for Threat Modeling, which I think is game-changing...” (T161)



    “I had ChatGPT write the breach notification for... Pretty close to the real thing.” (T94)


We finally aimed to investigate the ways in which users were discussing the use of ChatGPT for software security. Our qualitative analysis uncovered five main types of discussion. We leveraged these types during our sentiment assessment of the tweets. We assessed user sentiments of ChatGPT for the tweets of each category to better understand the public sentiments about this emerging technology.


    
  * Article/Tutorial tweets promote an external article or media source containing information about using ChatGPT for software security. 

    
  * Speculation discusses the theoretical use of ChatGPT for software security tasks and topics, with any example usage or evidence of application. 

    
  * Example tweets present the authors findings when using example queries as proof of concept application for different software security tasks. Evidence of use is described through experiences or even screenshots of the use cases. 

    
  * Applied tweets are similar to Example tweets, except they discuss real-world industrial applications relevant to users' occupations. 
    
    
  * Tool tweets promote a tool that has been constructed using ChatGPT technologies. 




Figure <ref> displays the frequency and user sentiments for each category of discussion.
 Overall sentiments leaned towards positive (54% positive), which may indicate that most users are optimistic about the capabilities of ChatGPT for software security. 

Articles and tutorials demonstrating how ChatGPT could be used for software security was one of the most prominent discussion types. This demonstrates that the security community is interested in the potential use of AI chatbot technology. Furthermore, some users have begun to develop software security products using ChatGPT or related technologies, as shown in the Tool category. 

Sentiments were also primarily positive for Example tweets (77%) which indicates that ChatGPT may perform well when used for simple proof-of-concept applications. Similarly, articles that discuss or demonstrate ChatGPT's potential are also slightly positive (51%). However, applied use has split sentiments, which indicates that ChatGPT may not yet be transferable to real-world software security tasks. 





Furthermore, the sentiment in speculative discussions is generally negative (47%). Hence, many users are sceptical about the actual practical benefits of ChatGPT when used for software security tasks. As the majority of discussion on this topic is either speculation or untested example usage, it is clear that AI chatbots for software security require more investigation and rigorous evaluation. 

[right=1pt, left=1pt, top=1pt, bottom=1pt, colback=azure(web)(azuremist)]
RQ1 Findings: 


    
    
  * Vulnerability detection was the most discussed software security task among practitioners testing ChatGPT.
    
  * The overall sentiment of the discussions among users was positive.
    
    
  * In various security tasks, credibility and practicality of generated information were common concerns.
    
  * Sentiments were primarily positive for simple proof-of-concept use cases. 







 §.§ RQ2: practicality of ChatGPT's outputs when utilised for the most desired software security task











  

  



Using the qualitative analysis described in Section <ref>, we first recorded our observations regarding ChatGPT's vulnerability detection accuracy for each SV. 
A total of 43 out of 70 (61.42%) vulnerabilities were correctly identified by ChatGPT. This efficacy was achieved when ChatGPT was presented with vulnerable code snippets of moderate complexity— specifically, those that are relatively long (average of 32 lines of code per snippet) with focused fixing commits and not overly intricate.



































































We answer RQ2 in the following sections by describing the types of information ChatGPT generated and how that information is presented for the vulnerability detection task. This investigation can provide a basis for future enhancements in the design and functionality of tailored conversational agents for software security. We provide quoted text from the analysed ChatGPT outputs. Figure <ref> displays the overall structure of the provided information in ChatGPT outputs. 




  §.§.§ Assumption
 It has been observed that ChatGPT occasionally makes assumptions about tertiary code when analysing the source code inputted. This kind of information was presented tentatively because they all rest on assumptions.

    “For instance, we're making assumptions about functions like compose_sadb_supported() and pfkey_broadcast() which could have their own set of vulnerabilities.” (Output #30)








  §.§.§ Description
 There was a constant presence of descriptions in the ChatGPT responses. According to our thematic analysis, the descriptions can be divided into three major categories, Supplied Code, Vulnerability, and Mitigation.



    
  * Supplied Code: ChatGPT frequently begins its responses with a description of the supplied code. For instance, the functionality of the code or its programming language.

    “This code snippet is written in Java and seems to be responsible for handling requests to trigger a configuration reload through a security token system.” (Output #70)





    
  * Vulnerability: A large portion of each ChatGPT response is devoted to describing identified vulnerabilities. In addition to classifying these descriptions under the Vulnerability category, we further divided them into two themes for a more granular analysis: “Theoretical and Practical Implications". This is because we sought to distinguish between when ChatGPT delves into abstract security implications and when it explains the actionable aspects of potential exploits.


“Theoretical Implication" example:

    “...
4. Exposure of Sensitive Information: The password recovery link may contain sensitive information (like a reset token). The application must make sure to generate a unique and unpredictable token for each password recovery request. ...
” (Output #14)






“Practical Implication" example:

    “... There are a few places where objects are dereferenced without first ensuring they are not null. This could potentially lead to a segmentation fault and crash the program. ...
” (Output #26)


Based on our observations, out of 43 cases where ChatGPT identified the specific vulnerability of interest, 38 instances (88%) were classified under the “Theoretical Implications” theme, while only 5 instances  (12%) were categorised under “Practical Implications”.

    
  * Mitigation: ChatGPT infrequently provided specific information on how to address the particular vulnerability identified. This category contained two themes: “List of Resolutions" and “Strategy".
    
    With the “List of Resolutions" theme, when ChatGPT detects multiple potential vulnerabilities, it offers a numbered list with brief explanations of how to resolve them.

    “To mitigate these vulnerabilities: 
1. Use parameterized queries or prepared statements to avoid SQL injection. 
2. Perform comprehensive input validation and sanitization on all user inputs.
3. Implement proper access control and authorization checks to prevent unauthorized actions.
4. ...
” (Output #64)








The “Strategy" theme revealed that ChatGPT provides a 
detailed response 
when encountering a 
severe vulnerability. In addition, we noticed that the responses generated in these instances have an imperative tone. For instance, the phrase “you should" is commonly used in this theme.

    “In order to make this script more secure, you should: 
Sanitize and validate all input data to prevent injection attacks. Use a safer file handling technique, which does not allow manipulation of file names. ... ” (Output #51)






























  §.§.§ Security Guideline
 ChatGPT makes a persistent effort to make its responses as comprehensive as possible by including generic security guidelines. This phenomenon was observed in all 70 instances of outputs. We classified these guidelines into three categories: Advise Further Testing, Best Security Practices, and Pointers for Consideration.


    
  * Advise Further Testing: The first and most common category (33 instances) is when ChatGPT demands further security audits to be undertaken via experts and automated tools.


    “... It is always recommended to have your
code reviewed by a security expert and use static and dynamic analysis tools to uncover potential security vulnerabilities.” (Output #31)


    
  * Best Security Practices: In the second category, we identified that ChatGPT relied on best security practices (e.g., input validation) applicable to the particular vulnerability to deliver its response.

    “Misconfigurations in PAM can lead to security vulnerabilities, so ensure that you have properly configured PAM in accordance with best security practices. Always adhere to the principle of least privilege. ” (Output #34)





    
  * Pointers for Consideration: Lastly, the third category contains very basic security considerations.

    “you might want to consider using a pre-made, widely-reviewed authentication library or service, which will likely have these and other security issues resolved, rather than creating your own.” (Output #42)


Considering that these guidelines were all advice, all three security guideline categories were presented tentatively.



  §.§.§ Assessment
 The paramount section of ChatGPT responses pertains to its evaluation of potential vulnerabilities within the submitted source code. Our rigorous analysis identified three assessment categories: Vulnerable Classification, Security Issues and Concerns, and Non-vulnerable Classification.


    
  * Vulnerable Classification: This category emerged as the most prominent among the three assessment categories. Considering that all the codes supplied to ChatGPT were already vulnerable, this prevalence was unsurprising. However, a significant variation in the tone of the responses was discovered, which appears to be related directly to the severity of the vulnerability. To further understand, we separated the detected confidence levels into three distinct themes: “Very High", “High", and “Moderate".
    
Within the “Very High" theme, ChatGPT responses were characterised by an assertive tone. In contrast to the other two themes (High and Moderate), adjectives such as “significant" and “serious" were frequently used. Furthermore, ChatGPT repeatedly hinted at the presence of multiple vulnerabilities instead of identifying one individually.

    “While this code may seem harmless at first glance, it indeed has several serious security vulnerabilities. ...” (Output #9)




The tone is less assertive in the “High" theme compared to the tone in the previous theme. For instance, while ChatGPT continues to identify multiple vulnerabilities, the word “potential" is frequently used to indicate a degree of uncertainty.

    “I've found several potential security vulnerabilities in the code. Here's a detailed analysis: ...” (Output #53)




There is a noticeable drop in certainty at the “Moderate" theme compared to the “High" theme. Apart from the frequent use of the term “potential" ChatGPT recurrently cites an absence of contextual information as a rationale for its implied uncertainty.

    “Based on the provided code snippet, there are a few potential security vulnerabilities to consider: ... Please remember that a complete and accurate analysis would require more context...” (Output #52)




Out of all the correctly identified vulnerabilities (43) under investigation, 25 instances (58%) were classified into this theme. The responses were presented tentatively in this theme compared to the other two themes as ChatGPT was less confident about its outputs.
    
  * Security Issues and Concerns: There have been instances where ChatGPT identified potential security weaknesses but refrained from explicitly labeling them as security vulnerabilities. Alternatively, it has used more subtle terms such as “concerns" and “issues". We have categorised such instances under the Security Issues and Concerns category.

    “However, there are some potential issues to be aware of, more from the perspective of robust programming practices and potential misuse, rather than direct security vulnerabilities: ...” (Output #50)





    
  * Non-vulnerable Classification: ChatGPT made a non-vulnerable assessment in several instances. In these cases, ChatGPT consistently pointed out that the supplied code snippet may not necessarily be secure.

    “At a glance, this code does not seem to contain glaring security vulnerabilities such as buffer overflows, SQL injections, etc., that are commonly associated with insecure code. ...  - an innocent-looking piece of code can become a security hole if used improperly or in an unexpected way.” (Output #21)








  §.§.§ Lack of Context
 

We observed that in exactly %50 of the outputs (35 out of 70), ChatGPT mentioned that more context would be required for a concrete security analysis.

    “While the code snippet does not provide all of the details, we can still make several general observations based on the provided information.” (Output #13)


[right=1pt, left=1pt, top=1pt, bottom=1pt, colback=azure(web)(azuremist)]
RQ2 Findings: 


    
  * Even when ChatGPT correctly pinpoints the vulnerability of interest, it predominantly focuses on abstract rather than practical security implications.
    
  * Although ChatGPT responses appear comprehensive, they are heavily populated with generic security information (e.g., Description, Guideline).
    
  * The severity of the vulnerability appears to be directly related to the tone of the responses.
    
  * ChatGPT generally adopted a tentative tone to present the information.
    
    





§ DISCUSSION


In RQ1, our investigation has shown that software and security practitioners enthusiastically experiment with and share the pros/cons of using ChatGPT for assistance in various software security tasks, such as vulnerability detection and security information retrieval. However, the state of discussions mostly remains in example proof of concept use, whereas only a few cases of applied use have been discussed. Current security practitioners appear to think that ChatGPT is inaccurate as a standalone solution. Hence, ChatGPT may be better suited to secondary tasks such as vulnerability report writing. For instance, users can draft comprehensive and detailed vulnerability reports by providing context-specific prompts and formatting instructions. Nevertheless, security professionals should review and validate the contents to ensure accuracy and completeness.

In the second phase of our study (RQ2), we treated ChatGPT as an oracle for the vulnerability detection task. In our data sample, it achieved a detection accuracy of 61%, which is far from ideal. Furthermore, after qualitatively examining the generated responses, we determined the characteristics (i.e., type and presentation style of information) of this chatbot and found that it is unable to provide reliable information for this task. Our analysis revealed that ChatGPT responses suffer from two specific quality issues. The most prevalent issue we observed was that all responses (70 instances) contained generic security information, as indicated in the “Theoretical Implication” theme and the three themes under the “Security Guideline” information type. Throughout these themes, we found an abundance of unneeded information, resulting in a lack of clarity in the responses. This issue persisted even in cases where ChatGPT correctly identified the vulnerability. Another noteworthy issue we found was that ChatGPT regularly lacks assertiveness when delivering responses. Based on our analysis, the majority of responses were given with a tentative tone, as shown in Figure <ref>, and a consistent request for additional contextual information. 

We believe both of these quality issues may undermine the practicality of ChatGPT for vulnerability detection within industry settings. This is because when a fault localisation tool is not transparent about its limitations, developers may inadvertently spend valuable time and effort verifying potentially inaccurate outputs <cit.>.

However, in contrast to existing weakness detection tools (e.g., static code analyser), conversational agents such as ChatGPT allow users to interface with a tool with profound natural language comprehension abilities. This potentially can have positive effects on novice learners who want to start learning secure coding principles. 


Overall, we advocate the need for further development of domain-relevant LLMs trained using software security data. Although ChatGPT is trained for general use and not intended for software security, our qualitative study shows it can assist and augment processes to a certain degree. Hence, we expect domain-specific LLMs to have potent capabilities for software security in the future. These technologies ultimately need to undergo rigorous industrial evaluation for which the empirical software engineering community can provide leadership in collaboration with AI researchers and practitioners. 
























    
    

    

    
    

    













 §.§ Implications of Our Study


Programmers. Understanding the nuances of responses (RQ2) generated by ChatGPT can offer valuable benefits to programmers who leverage this technology for vulnerability detection. Notably, they may better assess the legitimacy of the generated information and make informed decisions about whether to trust and adopt the security suggestions. Furthermore, by understanding ChatGPT's strengths and limitations, programmers can better integrate this technology into their development workflows, using it as an adjunct to other weakness detection tools. 

AI practitioners. The investigation into ChatGPT's communication of software security information (RQ2) has explicit implications for AI researchers and developers. Insights of this nature may assist them in fine-tuning LLM-based chatbots for specific software security tasks. Specifically, they must strive to create security-tailored LLMs capable of conveying security information to users in a manner that is more straightforward, actionable, and less susceptible to misinterpretation. Such models are likely to perform better and be more practical when applied in real-world security settings.

Security researchers. We have identified some promising research directions for researchers. Our analysis in RQ1 unveiled that there is a lot of speculation (discussions in the Vulnerability Exploits category) regarding the potential misuse of ChatGPT for malicious purposes. Researchers need to identify whether technologies like ChatGPT can enable malicious hacking so that countermeasures can be applied to prevent using these technologies for such purposes. 
Additionally, our analysis in RQ2  demonstrated that ChatGPT alters its language tone when confronted with vulnerabilities of varying severity scores. Further research is required to thoroughly understand the capability of LLMs in estimating vulnerability severity. This capability is imperative in vulnerability prioritisation, where precise severity assessment is essential for effective mitigation strategies.










§ THREATS TO VALIDITY




Construct validity: Our manual analysis may be subject to bias or inaccuracies, as is the case for most qualitative research. We have attempted to minimise such threats to validity by ensuring that two annotators independently performed the required manual analysis.

Internal Validity: We acknowledge that the criteria we used for selecting vulnerability data could have influenced the outcome of our study. As explained in Section <ref>, our intention to manually trace code changes meant we could not trace tangled fixing commits. As a result, we decided to limit the number of line changes within fixing commits to 10 lines. This approach ensured we gathered code snippets that were neither simplistic nor too complex.


External Validity: Qualitative analysis enables us to examine our data in depth, but consequently, we can only engage with a small sample of the total dataset. We cannot generalise beyond the sampled tweets and vulnerabilities due to the effort required to examine the data manually. We obtained our tweet sample based on the most liked and influential tweets, so we expect our sample represents a larger audience. 


Furthermore, public opinions may be inaccurate or biased for emerging technologies, as users may be overly excited or not have had enough time to evaluate their use properly. Despite our desire to extend the three-month period for collecting tweets, we were unable to do so since Twitter's academia API has been deprecated following recent changes at the company.

It is also possible that our findings might not generalise to future ChatGPT versions enriched with updated data. However, we conducted this study using a single ChatGPT version with the same knowledge cut-off date (Sep 2021) for consistency.



§ CONCLUSION






We have conducted an empirical investigation to understand the potential use of ChatGPT for software security. Based on examining public discussions regarding this LLM, we found that practitioners were often optimistic about its utility for various security tasks. 
Our systematic evaluation further demonstrated that ChatGPT can identify vulnerabilities 
with roughly 61% accuracy. Furthermore, we qualitatively demonstrated the versatility and granularity of information that ChatGPT can offer in the vulnerability detection task. The results revealed that ChatGPT outputs in this task are often too generic and ambiguous to be of practical use at the industry level. A tailored security LLM could enhance this capability, enabling practitioners to leverage the abundance of knowledge embedded in LLMs.



§ DATA AVAILABILITY

We have made all our artefacts of this study available via a reproduction package<cit.>.



§ ACKNOWLEDGMENT

This work has been supported by the Cyber Security Cooperative Research Centre Limited whose activities are partially funded by the Australian Government’s Cooperative Research Centre Programme.


IEEEtran





 Paper Title: Learning to Embed Distributions via Maximum Kernel Entropy.




Floquet Engineering of Polaritonic Amplification in Dispersive Photonic Time Crystals
    Vladimir M. Shalaev
    August 5, 2024
=====================================================================================




§ ABSTRACT 
Empirical data can often be considered as samples from a set of probability distributions. Kernel methods have emerged as a natural approach for learning to classify these distributions. Although numerous kernels between distributions have been proposed, applying kernel methods to distribution regression tasks remains challenging, primarily because selecting a suitable kernel is not straightforward. Surprisingly, the question of learning a data-dependent distribution kernel has received little attention. In this paper, we propose a novel objective for the unsupervised learning of data-dependent distribution kernel, based on the principle of entropy maximization in the space of probability measure embeddings. We examine the theoretical properties of the latent embedding space induced by our objective, demonstrating that its geometric structure is well-suited for solving downstream discriminative tasks. Finally, we demonstrate the performance of the learned kernel across different modalities.




§ INTRODUCTION

Most discriminative learning methods conventionally assume that each data point is represented as a real-valued vector. In practical scenarios, however, data points often manifest as a 'set' of features or a 'group' of objects. A quintessential example is the task of predicting a health indicator based on multiple blood measurements. In this case, the single data point of a patient has multiple, or a distribution of, measurements. One approach to accommodate such cases involves representing each input point as a probability distribution. Beyond mere convenience, it is more appropriate to model input points as distributions when dealing with missing data or measurement uncertainty, as often encountered when facing the abundance of data, which commonly presents a challenge for data-rich fields such as genetics, neuroscience, meteorology, astrophysics, or economics.


The task of regressing a mapping of probability distributions to a real-valued response is known as distribution regression. Distribution regression has been successfully applied in various fields, such as voting behavior prediction <cit.>, dark matter halo mass learning <cit.>, human cancer cells detection <cit.>, brain-age prediction <cit.>, among others <cit.>. The versatility and effectiveness of this framework underscore its power in solving complex problems <cit.>. Kernel methods have become a widely used approach for solving distribution regression tasks by exploiting a kernel between distributions  referred to as a distribution kernel. Despite the multitude of proposed kernels, the practical application of kernel methods remains challenging due to the nontrivial choice of the appropriate kernel. While some efforts focus on identifying kernels with broad applicability and favorable statistical properties <cit.>, others aim to tailor kernels to the geometric characteristics of specific input spaces <cit.>. Remarkably, the question of learning data-dependent kernels has received limited attention. This study is thus driven by a fundamental question: What are the underlying principles that facilitate the unsupervised learning of an effective kernel, one that optimally encapsulates the data properties and is well suited for discriminative learning on distributions?

In this work, we leverage a key insight: an appropriate selection of the distribution kernel enables the embedding of a set of distributions into the space of covariance operators. Building on this theoretical idea, we claim that quantum entropy maximization of the corresponding covariance operator is a suitable guiding principle to learn data-dependent kernel. This, combined with a careful design of kernel parametrization, let us to devise a differentiable optimization objective for learning data-specific distribution embedding kernels from unlabeled datasets (i.e. unsupervised) (<Ref>). We show that the entropy maximization principle facilitates learning of the latent space with geometrical configuration suitable for solving discriminative tasks <cit.>. We empirically demonstrate the performance of our method by performing classification tasks in multiple modalities.

In summary, our unsupervised data-dependent distribution kernel learning framework introduces a theoretically grounded alternative to the common practice of hand-picking kernels. Such framework could be further leveraged for generalizing existing learning approaches <cit.> catalyzing the use of distribution-based representations within the broader scientific community.



§ PRELIMINARIES

We first introduce the main concepts necessary to formalize our learning framework: kernel mean embeddings and covariance operator embeddings.



 §.§ Kernel Embeddings of Distributions


Consider an input space 𝒳 and a positive-definite (p.d.) kernel k: 𝒳×𝒳→ℝ. Let ℋ be the corresponding reproducible kernel Hilbert space (RKHS) induced by such kernel. Consider a probability distribution P∈𝒫(𝒳). The kernel mean embedding map embeds the distribution P as a function in Hilbert space:


    μ_P≡μ(P) ∫_𝒳 k(x, ·)   dP(x) = ∫_𝒳ϕ(x)    dP(x) ,

where ϕ: 𝒳→ℋ is a feature map such that ϕ(x) = k(x, ·).

Importantly, if the kernel k is characteristic <cit.>, the mapping μ: 𝒫(𝒳) →ℋ is injective, implying that all information about the original distribution is preserved in ℋ. This last property underscores much of power under recent applications of kernel mean embeddings <cit.>. 
The natural empirical estimator for the kernel mean embedding approximates the true distribution with a finite sum of Dirac delta functions:

    μ̂_P1/N∑_i=1^N ϕ(x_i) ∈ℋ

where x_1, …, x_N ∼P are N empirical i.i.d. samples. The estimator has a dimension-free sample complexity error rate of 𝒪(N^-1/2) <cit.>.

Additionally, we denote d_k: 𝒳×𝒳→ℝ_≥ 0 a kernel metric induced by a kernel k,

    d_k(x, x^') = ϕ(x) - ϕ(x^') _ℋ  .

Note that d_k is a metric on 𝒳 if feature map ϕ is injective.



 §.§ Covariance Operators and Entropy

A second way of mapping a probability distribution to a Hilbert space can be defined by means of a covariance operators. For a given feature map ϕ(x) = k(x, ·): 𝒳→ℋ[Assuming k to be a continuous positive definite (p.d.) kernel and 𝒳 to be compact.], and a given probability distribution P∈𝒫(𝒳), the covariance operator embedding is defined as:

    Σ_P∫_𝒳ϕ(x) ⊗ϕ(x)   dP(x)

where ⊗ is a tensor product. Σ_P is a self-adjoint positive semi-definite (p.s.d.) operator acting on ℋ. Such operator can be seen as a mean embedding w.r.t. the feature map x ↦ϕ(x) ⊗ϕ(x) and therefore, for a universal kernel k, the map P↦Σ_P is injective (see <cit.>).

Similarly to <Ref>, the natural empirical estimator is:

    Σ̂_P = 1/N∑_i=1^Nϕ(x_i) ⊗ϕ(x_i)

where x_1, …, x_N ∼P are N i.i.d. samples.

For a translation-invariant kernel k(x, x^') = ψ(x - x^') normalized such that k(x, x) =  1, the covariance operator Σ_P is a density operator <cit.>. Henceforth, entropy measures can be applied to it, and the quantum Rényi entropy of the order α can be defined as:

    𝒮_α(Σ_P) 1/1 - αlogtr[ (Σ_P)^α]  = 1/1 - αlog∑_iλ_i^α

where {λ_i }_i are the eigenvalues of Σ_P.
The Von Neumann entropy can be seen as a special case of Rényi entropy in the limit α→ 1. However, in our work, we focus primarily on the second-order case of Rényi entropy, i.e. α = 2 (see <cit.> for an in-depth overview of the properties and theory of quantum entropies).



§ UNSUPERVISED DISTRIBUTION KERNEL LEARNING



 §.§ Distribution Regression


In this section, we discuss the key topics in distribution regression, including problem setup, the notion of a 2-stage sampling process, and the common solutions to regression employing kernel methods.

Distribution regression extends the common regression framework to the setup where covariates are given as probability distributions available only through samples. Formally, consider the task of finding a regressor f: 𝒫(𝒳) →𝒴 from the dataset of samples Γ_M = {(P_i, y_i)}_i=1^M where P_i ∈𝒫(𝒳) are distributions provided as a set of i.i.d. empirical samples x_1, …, x_N_i∼P_i (see <cit.> for a comprehensive analysis). A viable approach to solving this problem is to define a kernel K: 𝒫(𝒳) ×𝒫(𝒳) →ℝ that is universal in 𝒫(𝒳). By setting up such a kernel K, we can utilize kernel-based regression techniques, with SVM often being the preferred method for classification tasks <cit.> or Kernel Ridge Regression (KRR) when the space 𝒴 is continuous <cit.> [As pointed out in <cit.>, under mild conditions, KRR can also be used for classification problems.]. To this end, several kernels have been proposed over time (see details in <Ref>). 

One possibility, proposed by <cit.>, is to introduce a kernel in the input space, the so called embedding kernel k_emb: 𝒳×𝒳→ℝ and exploit the induced mean embeddings μ_emb: 𝒫(𝒳) →ℋ_emb to map input distributions to points in RKHS. Subsequently, to define a second level kernel, distribution kernel K_distr: ℋ_emb×ℋ_emb→ℝ between points in the RKHS ℋ_emb. The simplest choice for such a distribution kernel is the linear kernel:

    K_l(μ_P, μ_Q) ⟨μ_P, μ_Q⟩_ℋ_emb
    = ∬_𝒳×𝒳 k(x, x^')   d P(x)   d Q(x^') .

A standard alternative to the linear kernel is a Gaussian kernel with bandwidth parameter λ > 0:

    K_RBF(μ_P, μ_Q) exp( - λ/2‖μ_P - μ_Q‖_ℋ_emb^2 ) = exp( - λ/2 d_k_emb(μ_P, μ_Q)^2 )   ,

which was shown to be universal in 𝒫(𝒳) <cit.>. The Gaussian kernel K_RBF can be computed from the linear kernel K_l as d_k_emb(μ_P, μ_Q)^2 = K_l(P, P) + K_l(Q, Q) - 2 K_l(P, Q).

In practice, we only have access to a finite number of samples of each input distribution, thus the distribution kernel is approximated using the natural estimator for the kernel mean embedding. The related excess risk for the regression solution is analyzed in <cit.>.





 §.§ Dataset Embedding


Instead of using standard kernels designed to encapsulate the geometry of the input space, we consider learning a data-dependent kernel, tailored to the specific properties of the dataset. In a similar vein, <cit.> proposed learning an optimal kernel (or equivalently, a feature map) jointly with the classifier to address the text modality. In this work, we focus on an unsupervised problem, aiming to learn a data-dependent kernel between probability distributions without access to classification labels.

We first introduce proper parametrization to ensure both expressivity and robustness followed by the definition of the optimization objective. Leveraging the idea of 2-level kernel setup, we define the embedding kernel as

    k_θ: 𝒳×𝒳→ℝ = k_emb(f_θ(x), f_θ(x^')) .

where f_θ is a trainable encoder function f_θ: 𝒳→𝒵, 𝒵 is a latent encoding space, and k_emb is a kernel defined on the latent space k_emb: 𝒵×𝒵→ℝ. The encoder function f_θ transforms every input probability distribution P∈𝒫(𝒳) into a distribution over the latent space ℙ_θ∈𝒫(𝒵) [By the definition of the encoding process, P_θ is a push-forward measure. For an empirical probability distribution q=∑_iδ(x_i)∈𝒫(𝒳) and a measurable map f: 𝒳→𝒵, the push-forward measure f # q ∈𝒫(𝒵) is defined as ∑_iδ(f(x_i)).] (<Ref>). Furthermore, we denote RKHS corresponding to the kernel k_emb as ℋ_emb and the kernel mean embedding map as μ_emb (see <Ref>).

    P∈𝒫(𝒳)  P_θ∈𝒫(𝒵)  μ_emb(P) ∈ℋ_emb .

These transformations define mean embeddings for each input probability distributions through the first level, embedding kernel k_emb (<Ref>). The second level, distribution kernel K_distr: ℋ_emb×ℋ_emb→ℝ is defined over the mean embeddings μ_emb(P)'s. We can now consider the embedding dataset 𝒟_M = {P_i }_i=1^M as an empirical covariance operator (see <Ref>), i.e.

    𝒟_M Σ̂_𝒟 = 1/M∑_P∈𝒟_MK_distr(μ_emb(P), ·) ⊗ K_distr(μ_emb(P), ·)  .

As Σ̂_𝒟 encapsulates information about the entire dataset, we term it dataset embedding. With the assumption that the dataset 𝒟_M is sampled i.i.d. from the (unknown) true meta-distribution 𝒟, Σ̂_𝒟 is a natural estimator to approximate the true covariance operator Σ_𝒟. To simplify the notation we use Σ_𝒟 in place of Σ_𝒟 unless required by the context.

Both the embedding kernel k_emb and the distribution kernel K_distr remain fixed throughout the training, learning happens by adjusting the parametrization of the latent space encoder f_θ. Such separation ensures expressivity while conforming to all technical requirements for a distribution kernel. Throughout the paper, we make the following assumptions on the latent space 𝒵 and embedding kernel k.

Latent space 𝒵 is a compact subset of ℝ^d. Kernel k_emb: 𝒵×𝒵→ℝ is a p.d. characteristic translation-invariant kernel k_emb(z, z^') = f( z - z^'^2) such that -f^' is completely monotone on (0, ∞) (see Definition 2.2.4 of <cit.>) and ∀ z ∈𝒵:   k_emb(z, z) = 1.



To facilitate our optimization process (which will be explained shortly), we opt for 𝒵=𝕊^d-1 (the d-dimensional hypersphere) and Gaussian kernel both for k_emb and K_distr (see <Ref>). We retain other suitable choices as potential avenues for future research.



 §.§ Unsupervised Optimization Objective


This dataset level representation depends on the choice of first and second level kernels k, K and, in turn, on the trainable function f_θ parameterized by the set of parameters θ. In this work, we propose learning the parameters θ to maximize quantum entropy of the dataset embedding, i.e.,

    θ = {𝒮_2(Σ_𝒟)  - logtr[ (Σ_𝒟)^2 ] } .

As we will describe in brief, optimizing this target has clear benefits inherited from the underlying geometry of the setup. But, first, we show how to empirically compute 𝒮_2(Σ_𝒟). Building upon previous work <cit.>[See the proof of Proposition 6 in <cit.>.], we exploit the following property of the covariance estimator:

    tr[ (Σ_𝒟)^2 ] = tr[ ( 1/M K_𝒟)^2 ]

where K_𝒟∈ℝ^M × M is the distribution kernel matrix, with [K_𝒟]_ij = K_distr(μ_P_i, μ_P_j).  This equation follows directly from the fact that Σ_𝒟 and 1/M K_𝒟 share the same set of eigenvalues. Leveraging this relationship, we can define tractable unsupervised training loss, which we term Maximum Distribution Kernel Entropy (MDKE), with respect to the parameters of the encoder f_θ:

    ℒ_MDKE(θ)   - 𝒮_2(Σ_𝒟) = logtr[ ( 1/M K_𝒟)^2 ]  = log∑_i=1^Mλ_i^2 ( 1/M K_𝒟)  = log1/M K_𝒟_F^2

where the latter relies on the fact that the Frobenius norm A _F^2 = ∑_iλ_i^2(A), where λ_i(A) are eigenvalues A.

The MDKE objective is differentiable w.r.t. θ for commonly used kernels, provided that the encoder f_θ is differentiable as well. While the entropy estimator 𝒮_2(Σ_𝒟) is convex in the kernel matrix K_𝒟, the objective as a whole is generally not convex in θ. However, in practice, as we show in <Ref>, mini-batch Stochastic Gradient Descent (SGD) proves to be an effective method for optimizing this objective. The effectiveness of this optimization process is significantly influenced by the parameters of the Gaussian kernels. We elaborate on the methodologies for kernel bandwidth selection in <Ref>.

The Frobenius norm formulation in the loss <Ref> significantly reduces computational complexity. However, as we have observed in some of our experiments, it can lead to the collapse of small eigenvalues of K_𝒟, particularly near the optimal value of the objective. To address this challenge we introduced a regularized version of the loss ℒ_MDKE-R that incorporates optional regularization, based on the determinant K_𝒟 inspired by the connection with Fekete points (see details in <Ref>).



 §.§ Geometrical Interpretation
 

The optimization objective is specifically designed to minimize the variance within each distribution (inner-distribution variance) while simultaneously maximizing the spread of distributions over the compact latent space 𝒵=𝒮^d-1. This shaping of the distributions embeddings in the latent space facilitates easier separation in downstream tasks. In this section we show that the geometry of the optimal (w.r.t. the MDKE loss) configuration of mean embeddings in the RKHS attains describe properties. For doing so we leverage the notion of distributional variance  𝕍_ℋ (Definition 1 in <cit.>). 


For a set of M probability distributions 𝒟_M, distributional variance 𝕍_ℋ(𝒟_M) of the mean embeddings in the RKHS ℋ is given by

    𝕍_ℋ(𝒟_M) 1/Mtr[ G ] - 1/M^2∑_i=1^M∑_j=1^MG_ij ,

where G is the M × M Gram matrix of mean embeddings in ℋ, i.e. G_ij = ⟨μ_P_i, μ_P_j⟩_ℋ <cit.>.


Here we show that the distributional variance 𝕍_ℋ can be equally reformulated into two separate contributions:

    𝕍_ℋ(𝒟_M) ≡1/M∑_i=1^M‖μ_P_i‖_ℋ^2  - ‖μ_P̅‖_ℋ^2 ,

where P̅ denotes mixture distribution with elements of 𝒟_M being uniformly weighted mixture components (see proof in the <Ref>).

The relevance of distributional variance for MDKE objective is established by the following result. 


For a set of M probability distributions 𝒟_M, the second-order Rényi entropy 𝒮_2 of the empirical covariance operator embedding Σ̂_𝒟 induced by the choice of Gaussian distribution kernel K_RBF over points in the RKHS ℋ_emb, - as defined in <Ref>, - is upper bounded by the distributional variance 𝕍_ℋ_emb(𝒟_M), i.e.,


    1/2 γ𝒮_2(Σ̂_𝒟) ≤𝕍_ℋ_emb(𝒟_M)


where γ is the bandwidth of the distribution kernel K_RBF.

The proof of this proposition is provided in <Ref>. This result formalizes the fact that our objective increases distributional variance, pushing up the average squared norm of mean embedding of input distributions while minimizing squared norm of the mean embedding of the mixture. We further explore the geometrical implications of such optimization by formalizing connection between the variance of the distribution and the squared norm of the its mean embedding in RKHS.


Under <Ref>, the maximum norm of kernel mean embedding is attained by Dirac distributions {δ_z }_z ∈𝒵.

This result is trivial due to the fact that the set of mean embeddings is contained in the convex hull of { k_emb(z, ·) }_z ∈𝒵, and, under <Ref>, ∀ z ∈𝒵:  k_emb(z, ·) _ℋ_emb^2 = k_emb(z, z) = 1.


Under <Ref>, uniform distribution 𝒰(𝒵) is a unique solution of

    _P ∈𝒫(𝒵){μ_emb(P) _ℋ_emb^2 ≡∬_𝒵×𝒵 k_emb(z, z^')   d P(z)   d P(z^') }.


The key intuition here comes from the fact that minimization of the squared norm of mean embedding in RKHS could be seen as minimization of total interaction energy over the given surface where the potential is defined by the kernel k. Thus <Ref> is a special case of Theorem 6.2.1 of <cit.>. Similar setup was used w.r.t. Gaussian potential over the unit hypersphere in Proposition 1 from <cit.>. The reformulation in <Ref> together with <Ref> immediately suggests that the framework could be seen as an extension of the Hyperspherical Uniformity Gap <cit.> to infinite-dimensional spaces of probability distributions This extension maintains the goal of reducing variance among input distributions while maximizing the separability between their means. See <Ref> for a broader explanation of the connection.

More generally, utilizing the fact that under <Ref>, the kernel metric d_k_emb (see <Ref>) is a monotonically increasing function of Euclidean metric on the latent space, we establish a precise connection between the generalized variance <cit.> and the norm of the  mean embedding. Further details can be found in <Ref>.

An attempt to directly optimize 𝕍_ℋ_emb using SGD resulted in significantly weaker outcomes. While a thorough mathematical explanation necessitates further investigation, we contend that this issue aligns with the recurring challenge reported across various studies regarding direct optimization over Maximum Mean Discrepancy (MMD). We hypothesize that optimal solutions exhibit similar geometric configurations, while the entropy of the covariance operator providing a smoother objective. Nonetheless, 𝕍_ℋ_emb retains its value as an insightful and intuitive measure for describing the geometric configuration of the learned system.



 §.§ An Illustrative Example

We use a simple example to illustrate the connection between geometrical configurations of embedded distributions and distribution kernel entropy 𝒮_2(Σ_𝒟) (see <Ref>). We sample a number of points from 6 different Gaussian distributions and project on a sphere 𝕊^2 varying their projected variance γ. As γ decreases, the distributional variance of the overall distribution of Gaussians increases (<Ref>). For very small γ each distribution converges to a point (a Dirac distribution). This results in the entropy interpolating between lower and upper bounds, demonstrating how entropy behaves in response to changes in distribution variance. <Ref> showcases the behavior of two terms comprising distributional variance (<Ref>): the average kernel norm of the distributions alongside the kernel norm of the mixture. The increase in entropy and variance corresponds to a 'flattening' effect on the spectrum of the distribution kernel matrix. This example provides a simplified picture of how input distributions configurations influence kernel entropy.




 §.§ Limitations


Runtime complexity. The applicability of a data-dependent distribution kernel to solving discriminative tasks relies on the structure of the dataset being well-suited for distribution regression modeling. The model performs best when the number of input distributions is relatively small (e.g., thousands rather than millions), while the number of samples per distribution is large. It is crucial to note that the computational complexity of the proposed method, which is a common concern in practical applications, is most favorable for the tasks described. A detailed analysis of runtime complexity can be found in <Ref>.

Broader impact. We wish to emphasize that the distribution regression framework has emerged as a powerful tool for analysis and predictive modeling, especially in domains where traditional methods face challenges, including social science, economics, and medical studies. We urge researchers and practitioners applying distribution regression in these areas to give special consideration to issues such as bias, fairness, data quality, and interpretability, -  aspects that are currently under-researched in the context of distributional regression, largely due to the relative novelty.



§ RELATED WORK





 §.§ Distribution Regression

Distribution regression was introduced in <cit.>, while the seminal work of <cit.> provides a comprehensive theoretical analysis of this regression technique. A natural approach to solving distribution regression problems involves using kernels between measures. Notable examples include the Fisher kernel <cit.>, Bhattacharyya kernel <cit.>, Probability product kernel <cit.>, kernels based on nonparametric divergence estimates <cit.>, and Sliced Wasserstein kernels <cit.>. <cit.> proposed leveraging the mean embedding of measures in RKHS, and <cit.> provided theoretical guarantees for learning a ridge regressor from distribution embeddings in Hilbert space to the outputs. Distribution kernels have been successfully applied in various kernel-based methods, such as SVM <cit.>, Ridge Regression <cit.>, Bayesian Regression <cit.>, and Gaussian Process Regression <cit.>. They have also been adapted for different modalities like distribution to distribution regression <cit.>, sequential data <cit.>, and more. For an in-depth exploration of the diverse methodologies employed in distributional regression settings, we invite readers to consult <ref>.



 §.§ Matrix Information Theory

Quantum entropy, including Rényi entropy, is a powerful metric to describe information in a unique way (see <cit.> for foundational insights). <cit.> designed the measure of entropy using operators in RKHS to mimic Rényi entropy's behavior, offering the advantage of direct estimation from data. <cit.> applied von Neumann entropy of the density matrix to the covariance operator embedding of probability distributions, thereby defining an information-theoretic framework utilizing kernel methods. In machine learning, especially within self-supervised learning (SSL) setups, entropy concepts have recently found novel applications. Our study builds on most recent developments <cit.> by applying quantum Rényi entropy to the covariance operator in RKHS.



§ EXPERIMENTS


We here demonstrate that our proposed method successfully performs unsupervised learning of data-dependent distribution kernel across different modalities. The experimental setup is divided into two phases: unsupervised pre-training and downstream regression classification using the learned kernel.

For each dataset, we select a hold-out validation subset with balanced classes, while the remainder of the dataset is utilized for unsupervised pre-training. We use mini-batch ADAM <cit.> with a static learning rate of 0.0005. We report mini-batch based (instead of epoch based) training dynamics as our tasks do not require cycling over the entire dataset to converge to the optimal loss value. All experiments use Gaussian kernel both as an embedding kernel and distribution kernel, the hyperparameter selection is performed as described in <Ref>.

Once the samples encoder f_θ is learned, we employ it to compute distribution kernel Gram matrix, used as an input to the Support Vector Machine (SVM) for solving downstream classification tasks. A grid search with 5 splits (70/30) is conducted to optimize the strength of the squared l_2 regularization penalty C, exploring 50 values over the log-spaced range {10^-7, …, 10^5}. The best estimator is then applied to evaluate classification accuracy on the validation subset, which we report.

Additional experiments exploring the application of data-dependent distribution kernels in domains where distribution regression models are less common, such as image and text, are presented in <Ref>.



 §.§ Flow Cytometry



Flow Cytometry (FC) is a widely used technique for measuring chemical characteristics of mixed cell population. Because population-level properties are described through (randomized) sampling of cells, FC is used as a canonical setup of distribution regression. For this study we used a dataset <cit.> where more than 100.000 cells are measured per each patient (subject). For each cell a total of ten parameters are reported, hence, we treated each subject as an empirical distribution over ℝ^10. We considered downstream classification tasks on two different sets of labels. The first ('Tissue' classification) contains peripheral blood (pB) and bone marrow (BM) samples from N=44 subjects. The second ('Leukemia' classification) presents healthy and leukemia BM cell samples, N=50. Classes were balanced in both cases.
l8.2cm
 Distribution regression accuracy on Flow Cytometry datasets. Code available at https://github.com/hidden-for-review.






Model     2cTissue     2cLeukemia 
 2-5 
                   Acc.     Var.        Acc.     Var.    
 
GMM-FV             93.07%            ± 0.308            94.80%          ± 0.186                

SW1                87.10%            ± 0.530            95.07%          ± 0.111                

SW2                81.71%            ± 0.341            95.30%          ± 0.224                

MMD Linear         82.42%            ± 0.840            90.57%          ± 0.208                

MMD Gaussian       81.71%            ± 0.574            92.23%          ± 0.216                

MMD Cauchy         81.57%            ± 0.662            93.77%          ± 0.080                

MMD IMQ            82.89%            ± 0.698            91.43%          ± 0.217                

MDKE Rand        77.50%                ± 0.002                    89.50%                ± 0.003                

MDKE No Reg.    95.30%                ± 0.002                    92.46%                ± 0.002                
 
MDKE Reg.        98.89%                ± 0.010                    94.57%                ± 0.005                
 








We sampled 16 subjects for Tissue classification and 20 subjects for Leukemia for training. Unsupervised learning was performed over the entire dataset. The encoder f_θ was parametrized by a 2-layers neural network (NN) with ReLU nonlinearity and l_2 normalized output (on the unit hypersphere 𝕊^9). Per each subject we sampled a small percentage of cells, and we report performance for the sample size of 200. We repeated each training and testing phase for 100 times to track the variance induced by this aggressive subsampling.

To demonstrate the impact of unsupervised pre-training, we compared several methods across multiple configurations (<Ref>):

a) Kernels on distributions. This group includes Fisher kernels applied to parametric Gaussian Mixture Model (GMM) estimates, as suggested in <cit.>, along with Sliced Wasserstein-1 and Sliced Wasserstein-2 kernels <cit.>.

b) MMD kernels. Here, we employ a Gaussian embedding kernel for mean embeddings, marked as 'MMD' (Maximum Mean Discrepancy) in the table. This category includes various options for the distribution kernel, such as linear, Gaussian, Cauchy, and inverse multiquadrics.

c) MDKE. We explore various configurations of the encoder optimized with the MDKE objective. We report performance for randomly initialized encoder, and for unsupervised pre-trained encoder with and without regularization. Random initialization happens only once, and all subsequent accuracy measurements are taken using the same encoder.

The variance reported for each model is measured across multiple runs to demonstrate the effect of the sampling. Importantly, the optimization of the MDKE objective results in embedding kernels with significantly lower variance compared to non-data-specific kernels.




§ CONCLUSION

In this work, we presented an unsupervised way of learning data-dependent distribution kernel. While previous studies in distribution regression predominantly relied on hand-crafted kernels, our work, in contrast, demonstrates that entropy maximization can serve as a powerful guiding principle for learning adaptable, data-dependent kernel in the space of distributions. Our empirical findings show that this technique can not only serve as a pre-training step to enhance the performance of downstream distribution regression tasks, but also facilitate complex analyses of the input space. The interpretation of the learning dynamics induced by the proposed objective relies on a theoretical link between the quantum entropy of the dataset embedding and distributional variance. This theoretical link, which we have proven, enables us to approach the optimization from a geometrical perspective, providing crucial insights into the flexibility of the learned latent space encoding.

We hope that theoretically grounded way of learning data-dependent kernel for distribution regression tasks will become a strong alternative to the common practice of hand-picking kernels. More broadly, our results present a methodology for leveraging the distributional nature of input data along side the novel perspective on the encoding of complex input spaces. This highlights the potential to extend the application of more advanced learning methods, embracing the ever-increasing complexity of data by going beyond more conventional vector-based representations.



  
*Acknowledgments. 
We thank the Allen Institute for Brain Science founder, Paul G. Allen, for his vision, encouragement, and support.

abbrvnat







§ PROOFS


In this section, we present the proofs for the propositions outlined in our study. To ensure clarity, we will first restate the setup and introduce necessary concepts.

We define the input space as 𝒳, and 𝒫(𝒳) represents the space of probability distributions over 𝒳. Consider a dataset of M probability distributions, denoted as 𝒟_M = {P_i ∈𝒫(𝒳) }_i=1^M. With a p.d. characteristic embedding kernel k: 𝒳×𝒳→ℝ, the corresponding RKHS ℋ, and the feature map ϕ: 𝒳→ℋ, we define the mean embedding map μ: 𝒫(𝒳) →ℋ such that μ_P = μ(P) ∫_𝒳ϕ(x)   d P(x). A p.d. translation-invariant characteristic distribution kernel K: 𝒫(𝒳) ×𝒫(𝒳) →ℝ is defined using the mean embeddings of corresponding distributions. For simplicity, μ_i denotes μ(P_i).

Additional concepts essential for our proofs include the Gram matrix of mean embeddings G ∈ℝ^M × M, representing the inner products of mean embeddings in the dataset, i.e., G  [⟨μ_i, μ_j ⟩_ℋ]_ij. The kernel matrix K_𝒟∈ℝ^M × M with respect to the distribution kernel K is denoted as K_𝒟 [K(P_i, P_j)]_ij. We also recall the definition of distributional variance 𝕍_ℋ (see <Ref>):

    𝕍_ℋ(𝒟_M) 1/Mtr[G] - 1/M^2∑_i=1^M∑_j=1^MG_ij

These definitions and notations will be referenced throughout the proofs.



 §.§ Kernel Norms Gap and Distributional Variance


For both cases of input distributions being empirical probability distributions or continuous densities, we define mixture distribution P̅, with a slight abuse of notation:

    P̅(x) 1/M∑_i=1^M P_i(x)


For the mixture distribution P̅ and the Gram matrix G, the following relationship holds:

    μ_P̅_ℋ^2 = 1/M^2∑_i=1^M∑_j=1^MG_ij



Proof. We begin by recalling that the inner product between mean embeddings μ_i and μ_j in ℋ is given by:

    ⟨μ_i, μ_j ⟩_ℋ = ∬_𝒳×𝒳k(x, x^')   d P_i(x)   d P_j(x^')


Substituting this into the expression for the Gram matrix, we have:

    1/M^2∑_i=1^M∑_j=1^MG_ij   = 1/M^2∑_i=1^M∑_j=1^M∬_𝒳×𝒳k(x, x')    d P_i(x)   d P_j(x')
       = ∬_𝒳×𝒳 k(x, x')    d ( 1/M∑_i=1^MP_i(x) )   d ( 1/M∑_j=1^MP_j(x') ) 
       = ∬_𝒳×𝒳 k(x, x')    d P̅(x)   d P̅(x')  = μ_P̅_ℋ^2


This completes the proof of <Ref>.

By incorporating <Ref> into the definition of distributional variance (see <Ref>), and noting that the trace of G is the sum of squared norms of input distributions (i.e.,  tr[G] = ∑_i=1^M μ_P_i_ℋ^2), we obtain:

    𝕍_ℋ(𝒟_M) ≡1/M∑_i=1^Mμ_P_i_ℋ^2 - μ_P̅_ℋ^2

This result offers a new and intuitive perspective on distributional variance, conceptualizing it as the difference between the average squared norm of individual input distributions and the squared norm of the mixture distribution.



 §.§ Pairwise Distance and Distributional Variance




For a dataset 𝒟_M of M probability distributions, we define the average pairwise distance between kernel mean embeddings in ℋ as 𝕁_ℋ(𝒟_M), given by:

    𝕁_ℋ(𝒟_M) 1/M^2∑_i=1^M∑_j=1^M||μ_i - μ_j||_ℋ^2



For the distributional variance 𝕍_ℋ of the dataset 𝒟_M, the following relationship holds:

    𝕍_ℋ(𝒟_M) ≡1/2·𝕁_ℋ(𝒟_M)



Proof. Starting with the definition of 𝕁_ℋ(𝒟_M):

    𝕁_ℋ(𝒟_M)    = 1/M^2∑_i=1^M∑_j=1^M||μ_i - μ_j||_ℋ^2
       = 1/M^2∑_i=1^M∑_j=1^M⟨μ_i - μ_j, μ_i - μ_j ⟩_ℋ
       = 1/M^2∑_i=1^M∑_j=1^M( ⟨μ_i, μ_i ⟩_ℋ + ⟨μ_j, μ_j ⟩_ℋ - 2⟨μ_i, μ_j ⟩_ℋ) 
       = 1/M^2( 2 M ∑_i=1^M⟨μ_i, μ_i ⟩_ℋ - 2∑_i=1^M∑_j=1^M⟨μ_i, μ_j ⟩_ℋ) 
       = 2 ( 1/M∑_i=1^M⟨μ_i, μ_i ⟩_ℋ_diagonal of G - 1/M^2∑_i=1^M∑_j=1^M⟨μ_i, μ_j ⟩_ℋ)

Recognizing that ∑_i=1^M⟨μ_i, μ_i ⟩_ℋ = tr[G], we can equate expression in the brackets to 𝕍_ℋ(𝒟_M), leading to:

    𝕍_ℋ(𝒟_M) ≡1/2·𝕁_ℋ(𝒟_M)

<Ref> will be instrumental in further proofs, establishing a crucial link between distributional variance in ℋ and quantum entropy of the covariance operator embedding Σ_𝒟.



 §.§ Distribution Kernel Entropy Upper-Bound


In this section, we provide the proof for the key theoretical result stated in <Ref>.

Consider a dataset 𝒟 consisting of probability distributions {P∈𝒫(𝒳) }_i sampled i.i.d. from an unknown meta-distribution 𝔻. We assert that the second-order Rényi entropy 𝒮_2 of the empirical covariance operator embedding Σ_𝒟, induced by the choice of Gaussian distribution kernel K_RBF over points in the RKHS ℋ, is upper-bounded by the distributional variance 𝕍_ℋ(𝒟):

    1/2 γ𝒮_2(Σ̂_𝒟) ≤𝕍_ℋ(𝒟)

where γ is the bandwidth of the distribution kernel K_RBF.

Starting from the properties of 𝒮_2(Σ̂_𝒟) stated in  <Ref>:

    𝒮_2(Σ̂_𝒟) = 𝒮_2 ( 1/M K_𝒟) = -log∑_i=1^M∑_j=1^M( 1/M K_ij) ^2  = -log( 1/M^2∑_i=1^M∑_j=1^MK_ij ^2)


Applying Jensen's inequality, considering the concavity of the log, to <Ref>, we have:

    𝒮_2(Σ̂_𝒟)    ≤ - 1/M^2∑_i=1^M∑_j=1^MlogK_ij^2
       = - 1/M^2∑_i=1^M∑_j=1^Mlogexp(-γ/2μ_i - μ_j_ℋ^2)^2
       = - 1/M^2∑_i=1^M∑_j=1^Mlogexp(-γμ_i - μ_j_ℋ^2)
       =  γ( 1/M^2∑_i=1^M∑_j=1^Mμ_i - μ_j_ℋ^2) 
       = γ·𝕁_ℋ(𝒟) 
       = 2 γ·𝕍_ℋ(𝒟)

<Ref> uses the definition of K_RBF from <Ref>, <Ref> is derived from the definition of the average pairwise distance in <Ref>, and <Ref> follows from <Ref>. This completes the proof of <Ref>, establishing the upper bound for the second-order quantum Rényi entropy of the covariance operator embedding Σ_𝒟 in terms of the distributional variance in RKHS ℋ. 

Technical Note: It is important to note that 𝒮_2 is typically measured using the base-2 logarithm (log_2) rather than the natural logarithm. However, the proof remains accurate with a proper re-scaling to account for the change in the logarithmic base.



 §.§ Generalized Variance in RKHS



In this section we prove the connection between generalized variance and norm of the kernel mean embedding.

So far we established the squared norm of mean embedding is maximized by Dirac points (zero variance) and is minimized by uniform distribution (max variance). We now formalize the intuition that larger squared norm in embedding kernel RHKS corresponds to smaller variance in the latent space. We first note that under  <Ref>, the kernel metric d_k_emb (see <Ref>) induced by the choice of k_emb is a monotonically increasing function of Euclidean metric on the latent space, thus is representative of the geometry of encoded input distributions. Which let us use a generalized notion of variance here.



(<cit.>) Let X be a random variable which takes values in a Fréchet space ℱ equipped with seminorm ·_α. And suppose that X is square-integratable, in a sense that 𝔼 X _α^2 < ∞. Let μ∈ℱ be a Pettis integral of X (i.e. generalization of the mean). Generalized variance of X w.r.t. seminorm ·_α is defined as following

    Var_α[X] 𝔼 X - μ_α^2   .



Note that for a random variable in RKHS defined as a push-forward of a probability distribution P∈𝒫(𝒵) over the latent space, i.e. X = z ∼ϕ#P satisfies conditions of <Ref> with ℱ = ℋ, ·_α = ·_ℋ_emb, and a Pettis integral being a kernel mean embedding. Denote the described variance as Var_ℋ_emb[P]. We now show that


Under <Ref>, for every P∈𝒫(ℤ),

    Var_ℋ_emb[P] = 1 - μ_emb(P) _ℋ_emb^2.





  
Proof.

    Var_ℋ_emb[P]    = 𝔼_x ∼ Pϕ(x) - μ_P _ℋ_emb^2 
       = ∫ϕ(x) - μ_P _ℋ_emb^2   d P(x) 
       = ∫⟨ϕ(x) - μ_P, ϕ(x) - μ_P ⟩_ℋ_emb  d P(x) 
       = ∫( ϕ(x) _ℋ_emb^2 + μ_P _ℋ_emb^2 - 2 ⟨ϕ(x), μ_P ⟩_ℋ_emb)   d P(x) 
       = ∫ϕ(x) _ℋ_emb^2   d P(x)  + ∫μ_P _ℋ_emb^2   d P(x)  - 2 ∫⟨ϕ(x), μ_P ⟩_ℋ_emb  d P(x) 
       = ∫ 1   d P(x)  + μ_P _ℋ_emb^2 ∫ 1   d P(x)  - 2  ⟨∫ϕ(x)   d P(x) , μ_P ⟩_ℋ_emb
       = 1 + μ_P _ℋ_emb^2 - 2 μ_P _ℋ_emb^2 
       = 1 - μ_P _ℋ_emb^2

Which concludes the proof.

The proposition demonstrates a motivation behind training the encoder to minimize variance in the latent space by maximizing the (average) squared norm of the mean embeddings.



§ PRACTICAL ASPECTS OF LEARNING





 §.§ Kernel Hyperparameter Selection


In practice, the effectiveness of the optimization process is significantly influenced by the parameters of the Gaussian kernels. Setting the bandwidth parameter γ either too low or too high can hinder the model's ability to learn effectively. This issue is a well-known challenge in working with kernel methods, where determining the optimal kernel bandwidth has been an area of extensive study.

In our work, we have employed an empirical approach, which involves adjusting γ based on the idealized structure of the dataset after its projection onto the latent space. Specifically, we set γ such that 1/γ equals 10 times the average distance to the nearest neighbor in the set of points sampled uniformly on 𝕊^d-1 (inspired by the experimental approach in <cit.>). The number of points is chosen to be the number of distributions in the training set for the distribution kernel. For the embedding kernel, it is set to the number of distributions in a batch multiplied by the number of samples per distribution used for unsupervised training.



 §.§ Optional Regularization




As the theoretical maximum of 𝒮_2 entropy is attained when the spectrum of 1/M K_𝒟 is uniform, the optimal encoder with respect to ℒ_MDKE also tends to maximize the determinant of the kernel Gram matrix K_𝒟. This relationship is intuitive when considering that the determinant of a matrix is the product of its eigenvalues. The points that maximize such determinant K_𝒟 are theoretically known as Fekete points <cit.> and, in our case, are relative to the distribution kernel. Fekete points have been shown by <cit.> to be an optimal configuration for learning kernel interpolants, making them particularly suitable for downstream tasks framed as kernel regression. As such, encoders optimized under the ℒ_MDKE objective facilitate more accurate and robust performance in subsequent regression tasks.

In practice, we found that optimizing the MDKE objective posed certain numerical challenges, particularly due to the tendency of too small eigenvalues in the distribution kernel matrix K_𝒟 to collapse near the optimal value of the objective when using large batch size for training. To mitigate this issue and prevent undesirable optimization behavior, we have introduced a regularization term to the original objective. This term is inspired by the concept of Fekete points configuration, leading to the following loss function:

    ℒ_MDKE-R(θ)     -𝒮_2(Σ_𝒟) + ϵ·Ω(Σ_𝒟) 
       = log1/M K_𝒟_F^2 - ϵ·log| 1/M K_𝒟|


Here, ϵ serves as a hyperparameter to control the strength of regularization. The regularization term Ω(Σ_𝒟) is designed to stabilize the optimization process by counteracting the effects of the collapse of small eigenvalues. The empirical evidence supporting the effectiveness of regularization in stabilizing the training dynamics for the sentence representation learning experiment (as detailed in <Ref>) is showcased in <Ref>.



 §.§ Runtime Complexity


Scalability of the kernel methods is typically a key concern when it comes to practical applications. The runtime complexity of the proposed method could be decomposed into two components:


  * Computation of the distribution kernel Gram matrix. This involves computing O(N^2) inner products between distributions where N is a number of distributions. Each inner product involves computing matrix of pairwise distances between samples from each distribution, in the case of computing the Gaussian kernel over points on the hypersphere, the runtime complexity is the complexity of multiplying matrices ℝ^M × d where M is a number of samples. All computations could be efficiently parallelized.

  * Solving regression using distribution kernel Gram matrix. The complexity comes from matrix inverse and is typically estimated to be O(N^3) with a range of methods proposed to reduce the complexity <cit.>.

With this being said, we want to highlight that the runtime complexity of the proposed method is most favorable exactly for those tasks and datasets where distributional regression is an appropriate modeling approach. A significant  number of samples per  distribution ensures a high accuracy in approximating the kernel between a pair of distributions, and, at the same time, a relatively small number of distributions in the dataset alleviates issues related to storing the distribution kernel Gram matrix in memory and performing computations on it




§ CONNECTION TO OTHER FRAMEWORKS



 §.§ Distributional Regression Landscape



As the distributional regression task differs from common Machine Learning (ML) setups where inputs are given as vectors, the effective practical solution requires unique considerations.

The most obvious approach would be to ignore the fact that inputs are given as distributions and learn classifier on the space of samples, with a proper aggregation of posteriors (e.g. with a simple sum over histograms). This approach, while being simple, has been shown not to yield practically useful results, and was explicitly excluded from reported performance on different tasks by different authors <cit.>.


Viable approaches to solving distributional regression could be, approximately, split up into the following categories.



  
Discriminate generative models. The idea is to fit each input distribution to a parametric family, e.g. Gaussian Mixture Model (GMM), to use available closed-form solutions to compute kernel or similarity or distance between distributions. This group of methods originated from the work on hidden Markov models and it's applications to processing sequence modalities, like text, DNA, proteins, and more. Early work <cit.> on learning discriminative classifier for generative models leveraged the fact that parametric models forms a Riemannian manifold with local metric given by Fisher information, they derived kernel function termed Fisher kernel suitable for running SVM between generative models. Driving motivation was classification between hidden Markov models, with the developed method being applied to DNA and protein sequence analysis. Following the same modeling approach of analysing DNA sequences with discriminative models between Markov models, <cit.> proposed to use Bhattacharyya distance between distributions from exponential families to derived so-called Expected Likelihood Kernel. <cit.> explored the method of computing kernel between distributions as the integral of the product of pairs of distributions, termed Probability Product Kernel. A new family of kernels was applied to the same setup of discriminating task on text modality, hidden Markov models for biological data. The model was also successfully applied for analysis of linear dynamical systems for time series data. Critical advantage of this work was access to computationally effective way of computing kernel for the distributions without having access to analytical closed-form parametrization (only relying on samples). In <ref> we used fitting distributions to GMM with Fisher Kernel applied to learn parameters as a baseline for the performance on the task.



  
Point clouds. This group includes methods that model each input distribution as a set of points (also known as 'point cloud' or 'feature group' in computer vision) and use kernel or similarity function defined on sets. Large portion of the methods in this group arised in computer vision (CV) field when local features extractor were widely employed to pre-process images yielding either per-image histograms or sets of low dimensional vectors. The group includes kernels based on nonparametric divergence estimates, quantized set kernels, and so-called 'nearest-neighbor' set kernels to name a few <cit.>. Such kernel were employed to many CV tasks though the successfully application required not only a good kernel but also high-quality feature extractor. Special attention goes to methods leveraging  Wasserstein distance (including both kernel-based and similarity-based solutions). Wasserstein distance as a metric-aware discrepancy measure for probability distributions is a natural choice of deriving kernels or similarity functions between point clouds <cit.>. While being computationally problematic for large-scale problems, Sliced Wasserstein kernels were succesfully adopted <cit.> as they provided both reliable way for the point set comparison with a reduced cost leveraging sliced formulation. In <ref> we compared performance of Sliced Wasserstein-1 and Sliced Wasserstein-2 kernels, with the latter yielding significantly lower performance. Such a behavior is consistent with theoretical analysis stating that Sliced Wasserstein-1 has favoriable properties when compared to kernel based on Wasserstein-2 distance.



  
Kernel Mean Embeddidngs. <cit.> proposed to leverage  kernel mean embedding of measures in RKHS, so that the distributional regression could now be casted to a regression the corresponding Hilbert space. While the original work leveraged kernel between RKHS embeddings to train SVM, <cit.> provided theoretical guarantees for the  for learning Ridge regressor from distribution embeddings in the Hilbert space to the outputs. <cit.> applied the same approach in Bayesian regression settings, and <cit.> used it for Gaussian Process Regression. The setup gives a great deal of flexibility by choosing the kernel in RKHS, with a few being tested in practice. Gaussian kernel is a common choice, due to it's universality <cit.>. Among others, inverse multiquadric (IMQ) is a popular choice, typically paired with random features to improve runtime complexity of the algorithm. Linear kernel in RKHS, despite not being universal, was reported to produce competitive results in multiple practical setups. In <ref> we compared performance for 4 different kernels in RKHS, namely linear, Gaussian, Cauchy, and IMQ, -  all defined as functions of Maximum Mean Discrepancy (MMD), which gives as a way to compute the value of the kernel from finite number of samples with high accuracy. While all kernels demonstrated different performance in terms of the test accuracy, the difference reported is not that substantial.

Distribution kernels were successfully applied in different kernel-based methods, such as , as well as in different modalities, like distribution to distribution regression <cit.>, distribution regression for sequential data <cit.>, and more. In cases where input data is naturally represented as a probability distribution but not immediately applicable for existing distribution regression solutions, pre-processing or encoding of inputs is required. <cit.> proposed learning a latent space representation for input data to apply distribution regression in the text modality, with the encoder being trained jointly with the classifier. To our best knowledge, methods for learning data-dependent kernel for solving distributional regressions were not previously reported.



 §.§ Hyperspherical Uniformity Gap



The link between entropy maximization in the space of distribution, the gap between average norm and norm of the mixture distribution marginalized over the dataset (i.e. 'average' distribution), and properties of distributions that minimize and maximize kernel mean embedding norm (as described in <Ref>) unveils a subtle yet significant connection to the concept of the Hyperspherical Uniformity Gap (HUG), introduced in <cit.>. HUG has been developed to generalize the phenomenon of neural collapse observed in supervised classification settings. In our approach, working directly with samples from input distributions grants us explicit access to the 'grouping' of points in the input space. Even though the HUG framework setup does not have a notion of 'grouping', we can close the gap by noting that class labels provided with the dataset implicitly create 'groupings' of points, which can be interpreted as empirical samples drawn from latent probability distributions (one for each class).

However, a notable distinction lies in the dimensional aspect: HUG focuses on the distribution of points on the surface of a finite-dimensional hypersphere, whereas our work encompasses an infinite-dimensional hyperball setting, given the use of points in kernel induced RKHSs. Furthermore, the loss function introduced in our study presents a unified optimization objective that weaves together both the inner-group and inter-group dynamics while being derived from first principles. In the HUG framework these two aspects are addressed separately.

Establishing a more formal connection between these frameworks emerges as a promising direction for future research. Such an endeavor could offer a novel perspective on supervised classification, particularly by conceptualizing class prototypes as probability distributions rather than mere vectors. This exploration might bridge the gap between these distinct approaches, enriching our understanding of classification paradigms in high-dimensional spaces.




§ ADDITIONAL EXPERIMENTS



In the following section, we present additional experiments on learning data-dependent distribution kernels in the image and text domains. While representing text as empirical samples from the finite space of tokens (i.e., words from the dictionary) is quite common, the choice to model images as histograms over pixel positions is more subtle. We demonstrate that, in both scenarios, unsupervised pre-training of the encoder yields distribution kernel that achieves strong performance on downstream classification tasks, showcasing the versatility of the proposed learning framework in scenarios where distribution regression formulations are uncommon.



 §.§ Image Classification Tasks


MNIST <cit.> and Fashion-MNIST <cit.> consist of 28 × 28 pixel grayscale images divided into 10 classes. We considered each individual image to be a probability distribution (via rescaling pixel intensities so that l_1(image)=1) over the discrete space of pixel positions (i.e., histogram). Given this support space, the encoder f_θ is a discrete map which we implemented as a table lookup (i.e., embeddings) from pixel indices to the points on the hypersphere 𝒮^31. Embeddings were initialized by sampling points uniformly. Gradients of the MDKE objective with respect to the embeddings parameters were computed via automatic differentiation using projected gradient steps to ensure that the embeddings remain on the hypersphere. The small size of the support space enables the exact computation of the inner product between kernel mean embeddings of input distributions <Ref> and, subsequently, the distribution kernel <Ref> during both training and evaluation. This ensured a lower variance of the accuracy for the downstream classification.

Performing MNIST classification upon pre-training with our unsupervised encoder significantly improves the baseline (random initialization of latent embeddings) accuracy of 85.0% by reaching a plateau at 92.15%.

To understand the data-dependency of our encoding procedure, we analyzed the latent spaces of MNIST and Fashion-MNIST datasets.  The visualization of pixel-level interaction, computed using Gaussian kernel Gram matrix, reveals complex, dataset-specific interactions (<Ref>, <ref>). Spectral clustering using the kernel Gram matrix provides deeper insight into the pixel interaction landscape. The chart shows clusters of pixels with correlated intensities (<Ref>, <ref>), with the number of clusters set to 10 empirically.






 §.§ Text Classification Tasks


To assess our method's performance in a larger discrete support space, we utilized the "20 Newsgroups" <cit.>, a multi-class text classification dataset. We reduced the size of the dataset to 5 classes (resulting in 2,628 sentences and 38,969 unique words) by subsampling both training and test subsets. We treated sentences as empirical distributions over words, assuming word sets to be enough for topic classification, despite no positional info.

The encoder f_θ mirrored the setup used in the MNIST case (<Ref>), with l_2 normalized word embeddings on 𝕊^31.

However, while in the MNIST case embeddings computations were performed exactly, here considering the entire embedding kernel Gram matrix is impractical due to its large size. Instead, we optimized embeddings by randomly sampling 20 words per sentence, making the inner product between embeddings a stochastic approximation. This setup is meant to confirm that the optimization of the proposed MDKE objective yields a solution that is robust w.r.t. the excessive risk induced by first-level subsampling. 

During the unsupervised pre-training phase, we observed a steady decrease in the training loss (<Ref>) despite the small batch size (50 sentences with total of 1,000 words). Importantly, for this experiment we employed a regularized version of the objective (MDKE-R). We discuss the rationale and empirical evidence supporting the use of this regularization scheme in <Ref>. During evaluation, to reduce computational complexity, we sample 2,000 sentences for the train and 1,000 for the test split, keeping classes balanced. The maximum classification accuracy achieved was approximately 89.3%, while the random initialization performance averaged at 37. 5% (<Ref>). The framework’s high accuracy in downstream classification showcases its prowess in learning potent latent representations, even when dealing with large, finite input distributions.





§ IMPLEMENTATION DETAILS



In this section, we provide an example illustrating the implementation of the proposed method using the PyTorch framework. All experiments were performed on a single machine with 1 GPU and 6 CPUs.

Functions to compute distribution kernel Gram matrix:


def pairwise_kernel(x, gamma1):
    B, T = x.size(0), x.size(1)
    X_unroll = x.reshape(B*T, -1)
    dist = (X_unroll[:, None, :] - X_unroll[None, :, :])**2
    dist = torch.sum(dist, dim=2)
    G = F.avg_pool2d(
        dist[None, : , :],
        kernel_size=(T, T),
        stride=(T, T)
    )
    return torch.exp(-(gamma1/2.) * G.squeeze(0))

def distribution_kernel_gram(x, gamma1, gamma2):
    Gxy = pairwise_kernel(x, gamma1=gamma1)
    Gx = Gy = torch.diag(Gxy)
    G = Gx[:, None] + Gy[None, :] - 2*Gxy
    K = torch.exp(-(gamma2/2.) * G)
    return Gxy, K


Distribution kernel entropy estimator and the MKDE loss:


def distribution_kernel_entropy(K):
    B = K.size(0)
    Cov = (1/B) * K
    return -(Cov ** 2).sum().log2()

def mkde_loss(encoder, X, gamma1, gamma2):
    # X.shape is (n_distributions, n_samples, d_input)
    Z = encoder(X)
    # Z.shape is (n_distributions, n_samples, d_latent)
    _, K = distribution_kernel_gram(Z, gamma1, gamma2)
    # K.shape is (n_distributions, n_distributions)
    loss = -distribution_kernel_entropy(K)
    return loss


The code presented here follows the setup presented in <Ref> using Gaussian kernel both as embedding and as a distribution kernel. Other kernels could be used by adjusting implementation of both helper functions accordingly. When training on large datasets, <cit.> might be used to avoid memory overflow in average pooling.




 Paper Title: Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey.












Miyai et al.: 
Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey


§ ABSTRACT 
Detecting out-of-distribution (OOD) samples is crucial for ensuring the safety of machine learning systems and has shaped the field of OOD detection. Meanwhile, several other problems are closely related to OOD detection, including anomaly detection (AD), novelty detection (ND), open set recognition (OSR), and outlier detection (OD). To unify these problems, a generalized OOD detection framework was proposed, taxonomically categorizing these five problems. However, Vision Language Models (VLMs) such as CLIP have significantly changed the paradigm and blurred the boundaries between these fields, again confusing researchers. In this survey, we first present a generalized OOD detection v2, encapsulating the evolution of AD, ND, OSR, OOD detection, and OD in the VLM era. Our framework reveals that, with some field inactivity and integration, the demanding challenges have become OOD detection and AD. In addition, we also highlight the significant shift in the definition, problem settings, and benchmarks; we thus feature a comprehensive review of the methodology for OOD detection, including the discussion over other related tasks to clarify their relationship to OOD detection. Finally, we explore the advancements in the emerging Large Vision Language Model (LVLM) era, such as GPT-4V. We conclude this survey with open challenges and future directions.



Anomaly Detection, Novelty Detection, Open Set Recognition, Out-of-Distribution Detection, Outlier Detection, Vision Language Model, CLIP, Large Vision Language Model, Large Multi-modal Model


Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey
    Atsuyuki Miyai, Student Member, IEEE, Jingkang Yang, Jingyang Zhang, Yifei Ming, Yueqian Lin, 
 Qing Yu, Member, IEEE, Go Irie, Member, IEEE, Shafiq Joty, Yixuan Li, Member, IEEE, Hai Li, Fellow, IEEE, Ziwei Liu, Member, IEEE, Toshihiko Yamasaki, Member, IEEE,  Kiyoharu Aizawa, Fellow, IEEE

A. Miyai, Q. Yu, T. Yamasaki, and K. Aizawa are with the Department of Information and Communication Engineering, The University of Tokyo, Japan, 1138656. Q. Yu is also with LY Corporation, Japan, 1028282.


E-mail: {miyai,yamasaki}@cvm.t.u-tokyo.ac.jp, {yu,aizawa}@hal.t.u-tokyo.ac.jp
J. Yang and Z. Liu are with 
S-Lab, School of Computer Science and Engineering, 
Nanyang Technological University, Singapore, 639798.


E-mail: {jingkang001,ziwei.liu}@ntu.edu.sg
J. Zhang, Y. Lin, and Hai Li are with the Department of Electrical and Computer Engineering, Duke University, Durham, NC, United States, 27708.


E-mail: {jz288,yueqian.lin,hai.li}@duke.edu
Y. Ming and S. Joty are with Salesforce AI Research, Palo Alto, CA, United States, 94301. S. Joty is also with NTU, Singapore, 639798 (on leave).


E-mail: {yifei.ming, sjoty}@salesforce.com
G. Irie is with the Department of Information and Computer Technology, Tokyo University of Science, Japan, 1258585.


E-mail: goirie@ieee.org

Y. Li is with the Department of Computer Sciences, University of Wisconsin-Madison, Madison, WI, United States, 53706.


E-mail: sharonli@cs.wisc.edu

Resources: <https://github.com/AtsuMiyai/Awesome-OOD-VLM>.


    Received July 31, 2024; accepted XX XX, XXXX
=========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





§ INTRODUCTION

A reliable visual recognition system should not only accurately predict known contexts, but also identify and reject unknown examples <cit.>. In critical applications like autonomous driving, the system must alert and cede control to the driver upon encountering unfamiliar scenes or objects not seen during training. However, most existing machine learning models are trained based on the closed-world assumption <cit.>, where the test data is assumed to be drawn i.i.d. from the same distribution as the training data, known as in-distribution (ID). Therefore, the development of classifiers capable of detecting out-of-distribution (OOD) samples is a crucial challenge for real-world applications. This challenge is precisely the focus of research in the field of OOD detection.

While OOD detection primarily focuses on semantic distribution shift, several other tasks share similar goals and motivations, including outlier detection (OD) <cit.>, anomaly detection (AD)  <cit.>, novelty detection (ND)  <cit.>, and open set recognition (OSR) <cit.>. Subtle differences in the specific definitions among these sub-topics have caused confusion in the field, leading to similar approaches being proposed across them. To address this issue, the generalized OOD detection framework was introduced <cit.>. The taxonomy of the generalized OOD detection framework is shown in Fig. <ref>. The generalized OOD detection framework classifies these tasks as special cases or sub-tasks under a unified taxonomy. This framework provides clear definitions and fosters a deeper understanding of each field.



In recent years, the emergence of Vision Language Models (VLMs), represented by CLIP <cit.>, has rapidly accelerated research in the field of Computer Vision. This has changed the paradigm of the recognition field, allowing for zero-shot <cit.> or few-shot learning <cit.> in various domains. VLMs have significantly influenced the aforementioned five problems (OD, AD, ND, OSR, and OOD detection), and the application of VLMs has become a highly notable research field <cit.>. However, alongside this remarkable progress, the paradigm shift with the advent of the VLMs has blurred the boundaries between the five problems. Due to the difficulty of a clear understanding of the distinctions and interrelations between these tasks, each community within the fields is facing significant challenges in identifying the optimal direction to pursue in this VLM era.

In this survey, we introduce a novel unified framework termed generalized OOD detection v2, which extends the previous generalized OOD detection framework and summarizes the evolution of these five problems in the VLM era. To create it, we systematically review the use of VLMs across these five problem areas, tracing their development from the start to the present, and summarize the evolutionary trajectory of each problem. Importantly, our framework reveals that a paradigm shift has caused some fields to become inactive or integrate with others, and the demanding challenges in the VLM era become AD and OOD detection, which is a remarkable finding for each community. In addition to the inter-field evolution, we elaborate on the important shifts in the definition of OOD detection as well as the problem settings and benchmarks, with the contrast of those for related tasks. Then, we conduct a thorough review of the methodology for OOD detection and related tasks in the VLM era, intending to clarify their similarities and differences and inspire future research in OOD detection.

Finally, we introduce the evolution of these problems in the emerging Large Vision Language Model (LVLM) era, such as GPT-4V <cit.> or LLaVA <cit.> (also referred to as Large Multi-modal Models or Multi-modal Large Language Models <cit.>). 
We summarize the definition of each evolving problem, the findings so far, and future challenges.





To sum up, in this survey paper, we make three contributions to the research community:


  * Unified Framework in the VLM era:
We systematically review the evolution of five closely related topics of OD, AD, ND, OSR, and OOD detection in the Vision Language Model era and provide an updated unified framework termed generalized OOD detection v2.
Our framework reveals that the paradigm shift has led to some field inactivity or integration, and the demanding challenges are AD and OOD detection.
We hope that these observations highlight the demanding challenges in the VLM era and foster collaborative efforts among each community.


  * A Comprehensive Survey for OOD Detection in the VLM era:
While comprehensive surveys on OD, AD, ND, OSR, and OOD detection methodologies have been published in recent years <cit.>, this survey is the first to comprehensively overview OOD detection methods specifically in the VLM era.
By connecting with other related tasks, we aim to provide readers with a holistic understanding of the developments and interconnections within each problem, particularly in the context of OOD detection.


  * An Introduction to the Evolution in the LVLM Era:
We further introduce the evolution of each problem in the Large Vision Language Model era. Despite the infant stage of these fields, this survey offers an in-depth introduction to each problem, aiming to facilitate future advancements in this area.


  * Future Research Directions:
We draw readers' attention to the future work necessary for advancing the field in the VLM and LVLM era. We conclude this survey with discussions on open challenges and opportunities for future research.






The paper content is organized as follows. 
In Sec. <ref>, we introduce the new version of generalized OOD detection by summarizing the evolution of the five related fields in the VLM era.
We then overview the two key problems (OOD detection and AD) that have evolved and remain active in Sec. <ref>, with a detailed breakdown of existing methodologies being presented in Sec. <ref> (CLIP-based OOD detection) and Sec. <ref> (CLIP-based AD).
In Sec. <ref>, we introduce early advancements of OOD detection and AD in the LVLM era.
Sec. <ref> and Sec. <ref> feature discussions on potential challenges and future directions.
Finally, we conclude with Sec. <ref>.


§ GENERALIZED OOD DETECTION V2


In this section, we introduce a novel unified framework termed generalized OOD detection v2, which summarizes the evolution of the five related fields in the VLM era. We first revisit the previous generalized OOD detection framework in Sec. <ref>. Next, we introduce the evolution of each problem.



 §.§ Background: Generalized OOD Detection V1


We first briefly revisit a previous generalized OOD detection, which encapsulates five related sub-topics: anomaly detection (AD), novelty detection (ND), open set recognition (OSR), out-of-distribution (OOD) detection, and outlier detection (OD).
These sub-topics can be similar in the sense that they all define a certain in-distribution, with the common goal of detecting out-of-distribution samples under the open-world assumption.
Previously, subtle differences existed among the sub-topics in terms of the specific definition and properties of in-distribution (ID) and OOD data. 

To provide a clear definition, a generalized OOD detection framework was proposed <cit.>. The taxonomy for generalized OOD detection is shown in Fig. <ref>. It is based on the following four bases: (1) Distribution shift to detect: The task focuses on detecting either covariate shift (OOD samples from a different domain) or semantic shift (OOD samples from a different semantic). (2) ID data type: The in-distribution (ID) data contains either a single class or multiple classes. (3) Whether the task requires ID classification: Some tasks require classification of the ID data, while others do not. (4) Transductive vs. inductive learning: Transductive tasks require all observations (both ID and OOD), while inductive tasks follow the common train-test scheme.
According to the above taxonomy, these five problems can be clearly categorized as shown in Fig. <ref>: Anomaly detection is categorized into sensory anomaly detection, which deals with covariate shift, and semantic anomaly detection, which deals with semantic shift. Novelty detection falls under the same category as semantic anomaly detection.
When addressing a multi-class scenario that necessitates ID classification, both open-set recognition and out-of-distribution detection are encompassed within this category. The main difference between OSR and OOD detection was the benchmark setup <cit.> (Sec. <ref> (c)).
Outlier detection belongs to a different category from the other tasks, as this problem is transductive (it has access to all observations).

For the detailed definition of each task, we refer the readers to the previous generalized OOD detection survey paper <cit.>.



 §.§ Evolution of Each Problem in VLM Era


We review how each problem has evolved in the VLM era. To make a fair judgment, we comprehensively investigated papers that use VLMs from top venues and summarized them in Table <ref>. 
Our survey revealed that CLIP <cit.> is predominantly used as the VLM for OOD detection and other sub-tasks, and other VLMs <cit.> are rarely utilized. Therefore, we focus on CLIP as the target VLM in this survey and refer to OOD detection using CLIP as CLIP-based OOD detection. Similarly, we will prefix other tasks with “CLIP-based" (CLIP-based AD). As OOD detection research is primarily focused on the image domain, we conduct a survey of other tasks within the image domain that are common and have strong connections to OOD detection research. For instance, our survey does not cover video domain tasks <cit.> due to their limited connection to OOD detection.



(a) Sensory AD → CLIP-based AD
Sensory AD has continued to develop as a common problem setting for CLIP-based AD,
inheriting the challenges of traditional sensory AD <cit.>. As shown in Table <ref>, the first appearance in a top venue was at CVPR 2023, and since then, a total of six papers have been published in top venues. In addition, there are numerous other papers <cit.>. Moreover, in terms of benchmarks, in addition to the commonly used MVTec-AD <cit.>, the largest industrial anomaly detection dataset VisA <cit.> has also become a standard benchmark in the field. Therefore, it is evident that sensory AD has become a highly active and noteworthy field in the VLM era.



(b) Semantic AD/ND  → Inactive
Research on semantic AD/ND appears to become inactive in the VLM era. 
As shown in Table <ref>, there are only two papers, TMLR 2022 <cit.> and CVPR 2024 <cit.>. However, the CVPR 2024 work <cit.> aims to build a generalist anomaly detector that solves many AD tasks, including sensory AD and semantic AD, and is not primarily focused on semantic AD. 
The reasons for the inactivity include saturation of performance for one-class semantic AD/ND, and incompatibility of methods with CLIP for multi-class semantic AD/ND. As for one-class semantic AD/ND, TMLR <cit.> exists, but the performances with common CIFAR and ImageNet-30 datasets have already achieved around 99%. As for multi-class semantic AD/ND, a common approach is to treat ID classes as a single class, but treating ID classes as a single class is less compatible with CLIP's class-wise discriminative capability.



(c) OSR  → CLIP-based OOD Detection 
We consider that OSR has been integrated into CLIP-based hard OOD detection. According to Table <ref>, there are no top venue publications on OSR research in the VLM era. Originally, the main difference between OSR and OOD detection was the benchmark setup <cit.>. OSR typically divides the classes in the one dataset into some known (ID) classes and unknown (OOD) classes, as seen in MNIST-4/6 <cit.> CIFAR-4/6 <cit.>, CIFAR-50/50 <cit.>, and TinyImageNet-20/180 <cit.>. However, in recent years, some works on CLIP-based OOD detection incorporate the benchmark setup of OSR and create new benchmarks such as ImageNet-10/ImageNet-20 <cit.> and ImageNet-protocol <cit.> for hard OOD detection. Therefore, the boundary between OOD detection and OSR has effectively disappeared, and all research in the VLM era has been integrated into OOD detection. 

Nevertheless, while pure OSR research is declining, some studies have used the term “open-set" in the context of domain generalization <cit.>. 
These studies deviate from the original scope of OSR research and are rather closely aligned with the field of domain generalization <cit.>. Therefore, within our generalized OOD detection v2, we do not classify these studies as falling under OSR research. We will discuss them in the context of full-spectrum OOD detection, a research field combining generalization and detection, in Sec. <ref>. 




(d) OOD Detection  → CLIP-based OOD Detection
OOD detection is a highly active research area in the VLM era.
As shown in Table <ref>, there are many papers in top venues, indicating a high interest from the community. Additionally, as mentioned above, OSR has been integrated with OOD detection as a field of hard OOD detection <cit.>. Therefore, it is expected that OOD detection will continue to grow and develop further.



(e) OD  → Inactive
OD has become less active in the VLM era. Previously, OD was used for open-set semi-supervised learning <cit.>, learning with open-set noisy labels <cit.>, and novelty discovery <cit.>.
The reason for the inactivity is that the use of CLIP led to a reduction in training costs and only a small amount of data needs to be collected, eliminating the need for large amounts of unlabeled data and reducing the need to consider noisy data.
However, recently, Liang  <cit.> proposed Unsupervised Universal Fine-Tuning, a new problem setting for CLIP-based OD in ICML2024. Unsupervised Universal Fine-Tuning assumes a more realistic problem setting for unsupervised tuning of the downstream task with CLIP where some OOD samples are included in the unlabeled samples.
With this new problem setting, there is still a possibility that OD will become active in the future.
However, as OD is not currently an active area, we exclude OD from the main discussion of this survey. Unsupervised Universal Fine-Tuning is deeply related to OOD detection and will be discussed in detail in Sec. <ref>.





 §.§ Discussion

Through Sec. <ref>, we found that previously mixed fields have been correctly organized in the VLM era, and that the focus should be on OOD detection and sensory AD. These fields are still developing, with an increasing number of methodologies and benchmarks, and are expected to become more active in the future. Note here that this does not mean that other fields have come to an end. For example, one reason why one-class semantic AD/ND has not been studied is the saturation of performance <cit.>. If more fine-grained and challenging datasets could be constructed, the field could be reactive. We put this in out-of-scope for this survey paper, but this is an important future challenge.



§ OVERVIEW OF EACH PROBLEM IN VLM ERA


In addition to the above inter-field evolution, we emphasize that the advent of VLMs has significantly changed the field of OOD detection itself. In this section, we present an overview of CLIP-based OOD detection, highlighting the key changes in the problem definition, the problem setting, and benchmarks. In addition, we also present an overview of CLIP-based AD in the hope that the understanding of each field will lead to a deeper understanding of CLIP-based OOD detection. For items that remain unchanged from traditional problems, such as background, applications, and evaluation, we refer the readers to the original generalized OOD detection paper <cit.>.



 §.§ CLIP-based Out-of-Distribution Detection


Definition
The definition of CLIP-based OOD detection differs significantly from that of conventional OOD detection.
Conventional OOD detection aims to detect test samples drawn from a distribution that is different from the training distribution. As another definition, OOD detection is defined as a task to detect test samples that the model cannot or does not want to generalize <cit.>. 
However, for CLIP-based OOD detection, CLIP has a vast amount of knowledge, so the OOD sample is completely unrelated to the distribution of the CLIP's pretraining data or the CLIP's own generalization ability. Therefore, traditional definitions cannot adequately describe the definition of CLIP-based OOD detection.

Unlike the previous definition, CLIP-based OOD detection is defined as follows <cit.>: CLIP-based OOD detection aims to detect samples that do not belong to any ID class text provided by the user. Given a pre-trained model, a classification task of interest is defined by a set of class labels , 
which we refer to as the ID classes. The semantic distribution is represented by the distribution P(). CLIP-based OOD detection aims to detect test samples that come from the distribution with the semantic shift from the ID classes, P()≠ P(). 
Following the definition of the generalized OOD detection framework <cit.>, ideal OOD detectors should keep the classification performance on test samples from ID class space , and reject OOD test samples with semantics outside the support of .



Problem Setting
CLIP-based OOD detection focuses on solving the image classification task in a computationally efficient way. Unlike traditional OOD detection settings, which primarily involve training an ID classifier with whole ID data, CLIP-based OOD detection primarily focuses on a zero-shot <cit.> (without utilizing ID images) or few-shot <cit.> (utilizing only a few ID images) setting. Each detailed definition of both settings are described later in Sec. <ref>. The field is advancing towards greater computational efficiency, requiring minimal or no training data.



Benchmark
Most recent works in CLIP-based OOD detection use high-resolution and large-scale datasets such as ImageNet <cit.>. 
The common ImageNet OOD benchmark uses ImageNet as ID and other datasets <cit.> as OOD. However, in this common benchmark, the semantics between ID and OOD are far, which may allow easy distinction between the ID and OOD. Therefore, recent works use more challenging OOD benchmarks where they split ImageNet classes into ID and OOD categories for hard OOD detection <cit.>. The representative datasets are ImageNet-20 <cit.>, ImageNet-10 <cit.>, and the recently proposed ImageNet-protocol <cit.> created by dividing into multiple variations of ID/OOD pairs from ImageNet-1K. This creation strategy initially focused on OSR but has recently been repurposed for OOD detection. These changes in the datasets shift OOD detection closer to the real world and make it a more challenging and practical task.





 §.§ CLIP-based Anomaly Detection


Definition
Unlike OOD detection, the definition of anomaly detection (AD) has not changed between conventional AD and CLIP-based AD. AD is intended for use in specific circumstances (industrial inspection), where samples that deviate from predefined normality are considered an anomaly <cit.>. Whether a model can generalize is irrelevant to the definition of “Anomaly". Therefore, even with the emergence of CLIP, the definition has not changed. 



Problem Setting
CLIP-based AD focuses on solving anomaly classification and segmentation in a computationally efficient way. 
Anomaly classification, like conventional AD, is a binary classification task that distinguishes between normality and abnormality. Anomaly segmentation, also following conventional AD, involves segmenting the location of anomalies. Like CLIP-based OOD detection, CLIP-based AD also primarily focuses on a zero-shot <cit.> (without utilizing images in the target dataset) or few-shot <cit.> (using only a few normal images in the target dataset) setting. Each detailed definition of zero-shot and few-shot settings is described later in Sec. <ref>. As another shift, conventional AD created separate models for each category  <cit.>, while CLIP-based AD creates a single unified model across multiple categories <cit.>, which leads to a more computationally efficient approach.

One key difference from CLIP-based OOD detection is that CLIP-based OOD detection does not involve localization tasks, while these are mainstream in CLIP-based AD. This will be discussed in detail in Sec. <ref>. 



Benchmark
Most works on CLIP-based AD tackle industrial inspection <cit.>. 
As for the benchmarks, besides the common MVTec-AD dataset <cit.>, the more challenging dataset VisA <cit.> has been newly employed <cit.>. The VisA benchmark includes objects with complex structures such as printed circuit boards and multiple instances with different locations within a single view, making it one of the most challenging datasets currently available in the open datasets. Since the pioneering work in CLIP-based AD (WinCLIP <cit.>) used MVTec-AD and VisA, many subsequent works have also used these datasets <cit.>.








§ CLIP-BASED OOD DETECTION: METHODOLOGY


In this section, we introduce the methodologies for CLIP-based out-of-distribution (OOD) detection. Fig. <ref> presents the timeline for representative methodologies for CLIP-based OOD detection. Table <ref> presents representative methods. We introduce methods for zero-shot OOD detection in Sec. <ref>, few-shot OOD detection in Sec. <ref>, and other research directions in Sec. <ref>. For each methodology, we categorize them by the type of training and whether additional OOD prompts were employed. 



 §.§ Zero-shot Out-of-Distribution Detection


Zero-shot OOD detection was proposed in 2021 by Fort  <cit.>. Since then, a growing number of methods have been proposed year by year.



Definition of Zero-shot OOD Detection
In zero-shot OOD detection, the term “Zero-shot" refers to the non-use of ID images during both training and inference phases. For instance, the method with additional training with auxiliary datasets (non-use of ID images) can be regarded as a zero-shot method <cit.>. The method with the pre-processing of the ID class texts can also be regarded as a zero-shot method <cit.>.





  §.§.§ Training-free Methods


a. With OOD Prompts
CLIP-based OOD detection started in this setting. 
The earliest work is ZeroOE <cit.>. ZeroOE feeds the potential OOD labels to the textual encoder of CLIP. However, the method of using known OOD labels is infeasible for real-world applications.
To solve this problem, ZOC <cit.> proposed to train an OOD label generator based on the visual encoder of CLIP and use the generated pseudo-OOD labels for OOD detection.
However, when dealing with large-scale datasets encompassing a multitude
of ID classes, the label generator may not generate effective candidate OOD labels, resulting in poor performance.
Building on these early works <cit.>, recent works focus on how to obtain high-quality OOD labels through either (i) OOD label retrieval <cit.> or (2) OOD label generation <cit.>. (i) One of the representative retrieval-based methods is NegLabel <cit.>. NegLabel selects high-quality OOD labels from extensive corpus databases by calculating the distance between an extracted OOD label and ID label. (ii) One of the representative generation-based methods is EOE <cit.>.
EOE utilizes Large Language Models (LLMs) to produce high-quality OOD labels. 
By modifying the prompts given to the LLM, EOE can be generalized to a variety of tasks, including far and near OOD detection.



b. Without OOD Prompts
In zero-shot OOD detection, many methods utilize OOD labels, but the difficulty and cost of creating these labels pose challenges. To address these issues, Ming  <cit.> proposed MCM, which uses only ID labels to detect OOD. MCM is a simple approach that devises softmax scaling to align visual features with textual concepts for OOD detection. Despite its simplicity, MCM has high effectiveness and scalability, and it serves as a crucial baseline in CLIP-based OOD detection.
Building on the concept of MCM, Miyai  <cit.> proposed GL-MCM, which extends MCM by just adding a local MCM score to enhance the fine-grained detection capability in local regions. SeTAR <cit.> enhances MCM and GL-MCM by changing the model's weight matrices using a simple greedy search algorithm.
We consider these methods to be post-hoc methods for CLIP-based OOD detection in that they directly employ an ID classifier for OOD detection. Due to their simplicity and high scalability, these post-hoc methods can bring fundamental performance improvements for many subsequent methods <cit.>. Therefore, we expect that this field should be developed further in the future, reflecting the trajectory of the field before CLIP emerged <cit.>.





  §.§.§ Auxiliary Training-based Methods


CLIPN <cit.> is the only auxiliary training-based method for zero-shot OOD detection. CLIPN aims to empower the logic of saying “no" within CLIP, and it designs a novel learnable “no" prompt and an additional “no" text encoder to capture negation semantics within images. To create an additional text encoder, CLIPN needs to be pre-trained on the CC-3M dataset <cit.>.
While the extensive pre-training of CLIPN may indeed lead to intensive computations and lower scalability, once pre-trained, it performs zero-shot OOD detection across a wide range of domains with comparable performance to few-shot OOD detection methods <cit.>.
In the future, within this field, there is potential for zero-shot open-vocabulary OOD detection to further advance the realm of zero-shot OOD detection, which will be discussed later in Sec. <ref>.



 §.§ Few-shot Out-of-Distribution Detection


Few-shot OOD detection was concurrently proposed by Miyai <cit.> and Ming <cit.> in June 2023. Since then, it has become the most active research area in CLIP-based OOD detection.



Definition of Few-shot OOD Detection
CLIP-based few-shot OOD detection aims to detect OOD images using only a few labeled ID images.
In few-shot OOD detection, the term “Few-shot" refers to the use of a few ID images during training or inference phases. For instance, the method with additional training with a few ID images can be regarded as a few-shot method <cit.>. Even without training, if a method uses a few ID images as a reference, we regard it as a few-shot method <cit.>. 
Regarding the number of shots, it is common to experiment with 1-shot to 16-shot <cit.>, following the closed-set setting <cit.>.



  §.§.§ ID Training-based Methods


a. Without OOD Prompts
Few-shot OOD detection began in this setting. Ming  <cit.> proposed PEFT-MCM for CLIP-based OOD detection, which demonstrates the effectiveness of combining parameter-efficient tuning methods (prompt learning <cit.> or adapter <cit.>) and MCM <cit.>. Concurrently, Miyai <cit.> proposed LoCoOp, a pioneer prompt learning approach for few-shot OOD detection. 
LoCoOp enhances CoOp's <cit.> OOD detection capabilities by performing OOD regularization with local OOD features. LoCoOp is the simplest prompt learning method and serves as a crucial baseline in few-shot OOD detection. 
Unlike LoCoOp, which utilizes non-ID local regions for OOD regularization, GalLoP <cit.> proposes an approach that utilizes local ID regions to enable a more fine-grained distinction between ID and OOD samples. GalLoP learns a diverse set of prompts by utilizing both global and local visual representations, thereby enhancing the detection capabilities.



b. With OOD Prompts
Similar to zero-shot OOD detection, recent works in few-shot OOD detection utilize additional OOD prompts <cit.>.
As representative methods, LSN <cit.> and NegPrompt <cit.> were proposed concurrently. They state that the simple negative prompts added “not” (“not a photo of a [cls]”) fail to capture the dissimilarity for identifying OOD samples. 
Therefore, by preparing negative prompts and training with them, LSN and NegPrompt can learn suitable negative prompts, enabling more accurate detection of OOD samples.
The difference between LSN and NegPrompt lies in their approach to the use of negative prompts. LSN prepares unique negative prompts for each class and learns suitable negative prompts for each class. In contrast, NegPrompt prepares multiple negative prompts common to all ID classes and trains them to learn generic templates representing the negative semantics of any given class labels. Also, NegPrompt tested the performances in the hard OOD detection setting with ImageNet-protocol <cit.>, outperforming LoCoOp and CoOp. 
Alternatively, IDPrompt <cit.> takes a different approach by introducing ID-like prompts, which are designed for OOD features that are close to the ID features. It extracts ID-like OOD regions in ID training images and trains ID-like prompts with these extracted OOD data. In a unique direction, LAPT <cit.> proposes an automatic sample collection strategy that retrieves or generates training ID images only with ID class names, which achieves high performance without image collection and annotation costs. LAPT then performs distribution-aware prompt learning, which distinguishes between ID class and OOD class tokens. LAPT is positioned within the context of more efficient few-shot OOD detection in this survey paper since it requires generating or retrieving “ID images" for the data collection.

In the context of few-shot OOD detection, recently, Li  <cit.> proposed a new problem setting called open-vocabulary OOD (OV-OOD) detection. 
While common few-shot OOD detection involves training on images from all ID classes during training, OV-OOD detection involves training on images from just a small subset of ID classes and performing OOD detection using all ID classes at inference time. Formally, we define a subset of semantic labels ⊂, where  represents all ID labels. Based on this subset of labels, we define a corresponding subset dataset ⊂. During training, only  is used. Then, at inference time, all ID classes  are used, and the goal is to detect OOD from a combination of all ID test data  with  and OOD test data  with . For this setting, existing few-shot OOD detection methods <cit.> can be easily applied by simply combining the rest of the ID classes. In particular, NegPrompt <cit.> learns general negative prompts that are not specific to the training ID classes, so it achieves high performance in OV-OOD detection.




  §.§.§ Training-free Methods


Training-free few-shot OOD detection is a novel research field, and only Dual-Adapter <cit.> falls under this category. Dual-Adapter adopts a prior-based method Tip-Adapter <cit.>, which leverages both textual and
visual features with a cache model and enhances performance without training. To adapt this to few-shot OOD detection, Dual-Adapter employs the concept of dual cache modeling and constructs Positive-Adapter and Negative-Adapter, and identifies OOD samples with the prediction difference with both adapters. 






 §.§ Other Important Research Directions




  §.§.§ CLIP-based Full-spectrum OOD Detection

CLIP-based full-spectrum OOD (FS-OOD) detection is a crucial challenge <cit.>. FS-OOD detection was proposed by Yang  <cit.> in 2022 as an important setting that considers both OOD generalization <cit.> and OOD detection simultaneously. 
Unlike standard OOD detection, which only focuses on semantic shifts between training and test distributions, FS-OOD detection further considers non-semantic covariate shift by including covariate-shifted ID images. As for the benchmarks, OpenOOD v1.5 <cit.> provides two large-scale benchmarks based on ImageNet-200 and ImageNet-1K, incorporating ImageNet-C <cit.> with image corruptions, ImageNet-R <cit.> with style changes, and ImageNet-V2 <cit.> with resampling bias as ID. As for CLIP-based methods, LSA <cit.> uses a bidirectional prompt customization mechanism, which adjusts discriminative ID and OOD boundary. 




  §.§.§ Other Tasks with CLIP-based OOD Detection

Unsupervised Universal Fine-Tuning
CLIP-based OOD detection is useful for a new task called Unsupervised Universal Fine-Tuning (UUFT) <cit.>.
UUFT is a problem of unsupervised learning for outlier detection (OD). Existing studies for unsupervised learning assumed that all unlabeled images belong to one of the ID classes <cit.>, but they require prior knowledge of exact class names linked to ground truth labels, which restricts their usefulness in various real-world situations. For a more realistic setting, UUFT assumes that OOD images are included in the unlabeled images. To detect OOD images during training, they developed MCM <cit.> and proposed UEO, which leverages sample-level confidence to approximately minimize the conditional entropy of confident instances and maximize the marginal entropy of less confident instances.



Open-world Prompt Tuning
CLIP-based OOD detection is useful for a new task called Open-world Prompt Tuning <cit.>. 
Open-world Prompt Tuning is a task that evaluates the classification accuracy on a mix of known and novel ID classes while training the model with known classes. To solve this problem, Zhou  <cit.> proposed DeCoOp which incorporates OOD detection into the inference pipelines and improves the base-to-new separability, preventing performance degradation on new classes. 


§ CLIP-BASED AD: METHODOLOGY


In this section, we introduce methodologies for
CLIP-based anomaly detection (AD) in the hope that the contrast with OOD detection clarifies the similarities and differences between each task and facilitates a deeper understanding of CLIP-based OOD detection. 



 §.§ Zero-shot Anomaly Detection


CLIP-based zero-shot AD was proposed in 2023 by Jeong  <cit.>. Although it started about two years later than OOD detection, many methods have been proposed up to the present.



Definition of Zero-shot AD
The meaning of the term “Zero-shot" for zero-shot AD is similar to that for zero-shot OOD detection.
In zero-shot AD, the term “Zero-shot" refers to the non-use of the images in the target domain during both training and inference phases. For instance, the method with additional training with auxiliary datasets can be regarded as a zero-shot method <cit.>. The method with the pre-processing of the target class texts can also be regarded as a zero-shot method <cit.>.





  §.§.§ Training-free Methods


With Anomaly Prompts
In zero-shot AD, a common approach is to utilize anomaly prompts to detect anomalies. This hypothesis is supported by several observations from existing work <cit.>. Firstly, the concepts of normality and anomalies are context-dependent states <cit.> of an object, with language playing a crucial role in defining these states. Secondly, language provides additional insights that help differentiate defects from acceptable variations in normality. 

The simplest zero-shot AD methods are (i) to perform anomaly classification with CLIP using text prompts for normality and anomalies as classes (“” vs. “”) and (ii) to calculate the similarity to the normal prompt (“”) as the score. These methods are called CLIP-AC <cit.>. Jeong  <cit.> reported that CLIP-AC with both normal and anomaly prompts outperforms that with only normal text prompts, which indicates the importance of the use of anomaly prompts.
However, the performances for this naive method are not yet satisfactory due to the wide range of variations of anomalies.
To solve this issue, Jeong  <cit.> proposed WinCLIP.
WinCLIP performs a compositional ensemble on a large number of pre-defined normal and anomaly templates and efficient extraction and aggregation of window/patch/image-level features aligned with the text. WinCLIP outperforms CLIP-AC by a large margin. Because of its simplicity and pioneering work, WinCLIP has become an important baseline for CLIP-based AD.
AnoCLIP <cit.> follows WinCLIP's approach of using a large number of pre-defined normal and anomaly templates but modifies the templates to be domain-aware (industrial photo) and contrastive state for normal and anomaly (perfect and imperfect).
However, it is noteworthy that the performance of the ensemble strategies of previous methods heavily depends on the text descriptions <cit.>. 
Also, it is observed that more descriptions are not always better <cit.>, which makes the previous approaches <cit.> using a naive ensemble of large templates somewhat uncontrollable and random in their applications. Therefore, SDP <cit.> proposes RVS, a representative vector selection paradigm, which makes the mechanism of extracting representative vectors from large templates controllable, allowing for a more diverse selection of representative vectors.




  §.§.§ Auxiliary Training-based Methods


Unlike CLIP-based OOD detection, all auxiliary training-based methods in zero-shot AD are open-vocabulary AD methods that are trained with auxiliary AD datasets and tested on unseen target datasets by simply changing the category prompt. Existing methods perform supervised training on the test set of one dataset and perform zero-shot testing on the other dataset <cit.> (training with MVTec-AD and evaluation with VisA.)

In recent years, the development of auxiliary training-based methods has received more attention than training-free zero-shot methods. There are two main reasons why training is necessary for AD: (i) The first is the domain gap between semantics and anomalies. CLIP is pre-trained to understand the semantics of images, so when applied in a zero-shot manner, it captures the semantics of the image. However, actual anomalies are not semantics, but rather represent the state of an object and appear only in local areas of the image. Therefore, without training, this domain gap between semantics and anomalies cannot be bridged. (ii) The second reason is that there are limitations to relying on a large set of manually crafted anomaly prompts. This incurs prompt creation costs and also makes it difficult to respond to unknown anomalies. Therefore, by replacing the anomaly prompts with learnable parameters, they aim to solve the high costs and limited adaptability to new anomalies.

To address the above issue (i), APRIL-GAN (also known as VAND) <cit.> was proposed. APRIL-GAN tackles the domain gap between semantics and anomaly by adding additional linear layers in vision encoders. These linear layers project image features at each scale into the text space, creating and aggregating anomaly maps at each stage. 
Similarly, SDP+ <cit.> also incorporates additional linear layers into SDP <cit.> to effectively project image features into the text feature space, addressing the misalignment between image and text.
To solve both the issue (i) and (ii), AnomalyCLIP <cit.> was proposed. AnomalyCLIP is a prompt learning-based method similar to CoOp. By replacing anomaly prompts with learnable parameters, it eliminates the need to prepare a large number of manually pre-defined prompts such as those in WinCLIP <cit.>.
Furthermore, unlike CoOp which learns object semantics, AnomalyCLIP learns object-agnostic text prompts that capture generic normality and abnormality in an image regardless of its semantics. To achieve this, AnomalyCLIP introduces object-agnostic text prompt templates for both normal and anomaly and performs global and local context optimization. A more recent approach, FiLo <cit.> leverages Large Language Models (LLMs) to generate fine-grained anomaly descriptions for each object category. This method replaces generic abnormal descriptions with LLM-generated specific anomaly content for each sample. By adding learnable prompts before the generated anomaly prompts, FiLo performs global and local context optimization, enhancing the ability to detect anomalies. 
As a unique direction from these methods, RWDA <cit.> proposes a data augmentation approach by utilizing CLIP's text embeddings as training data. RWDA adds randomly generated words into normal and anomaly prompts to generate a diverse set of normal and anomaly training samples and trains a regular feed-forward neural network with diverse text embeddings. 





 §.§ Few-shot Anomaly Detection


CLIP-based few-shot AD was proposed in 2023 by Jeong  <cit.>, concurrently with the development of zero-shot AD <cit.>.
Traditional few-shot AD research focuses on modeling the normal distribution from a limited number of normal samples to detect anomalies <cit.>. However, these methods often struggle to generalize to new domains, as they typically require retraining with the target datasets. With the advent of CLIP, the field of few-shot AD is shifting towards using a few target images only for inference at test time without training.



Definition of Few-shot AD
CLIP-based few-shot AD aims to detect anomaly images using only a few images in the target domain. The meaning of the term “Few-shot" is similar to that of few-shot OOD detection.
In few-shot AD, the term “Few-shot" refers to the use of a few normal images in the target domain during training or inference phases. For instance, the method with additional training with a few normal images in the target domain can be regarded as a few-shot method <cit.>. Even without training, if a method uses a few normal images in the target domain as a reference, we regard it as a few-shot method <cit.>. 



  §.§.§ Training-free Methods


The earliest approach in CLIP-based few-shot AD is WinCLIP+ <cit.>, an improved method of WinCLIP. WinCLIP, a base zero-shot AD method, cannot identify certain defects that can only be defined visually rather than textually. For example, the “Metal-nut" category in MVTecAD has an anomaly type labeled “flipped upside down," which can only be identified relative to a normal image. To address this, WinCLIP+ incorporates a few normal reference images into a memory bank <cit.> and calculates the anomaly score with the cosine similarity between the query image and its most similar image in the memory bank.



  §.§.§ ID Training-based Methods


There are few training-based methods for CLIP-based few-shot AD research, although CLIP-based few-shot OOD detection actively explores the training-based methods. 
This is likely because detecting anomalies in unknown classes is highly valuable in practical applications, and training on target data may oversimplify the task of few-shot anomaly detection. This oversimplification concern arises because the anomaly space for known categories is much more limited compared to that for OOD detection. Training with the target data could therefore make the task oversimplified, reducing its difficulty. The only existing work in this area is PromptAD <cit.>, a prompt learning method for one-class AD (where the normal class consists of one class). In one-class AD, traditional prompt learning methods for multi-class classification (CoOp <cit.>) do not work well. To address this, PromptAD creates a large number of anomaly prompts by adding a learnable anomaly suffix to the normal prompt. It then learns to bring the visual features closer to the normal prompt and further away from the anomaly prompts, enabling prompt learning for one-class AD.




  §.§.§ Auxiliary Training- and Reference-based Methods


We explore the methods trained on auxiliary datasets and utilize the normal images in the target domain as references during inference.
An early work in this category is APRIL-GAN (few-shot) <cit.>, which uses a linear layer trained with auxiliary datasets. Similar to WinCLIP+ <cit.>, APRIL-GAN (few-shot) utilizes a few ID reference images with a memory bank-based approach <cit.>.
More recently, Zhu  <cit.> proposed an in-context-learning-based method called InCTRL. InCTRL trains a model to discriminate anomalies from normal samples by learning to identify residuals or discrepancies between query images and a set of few-shot normal images (in-context sample prompts) from auxiliary data. During inference, InCTRL identifies anomalies by measuring the discrepancy between the features of the query image and a few in-context normal samples from the target dataset.




 §.§ Other Research Direction



  §.§.§ Anomaly Detection with Localization Models

Some works <cit.> tackle AD using SAM <cit.> or DINO <cit.>, foundation models for localization. The representative work is SAA and SAA+ <cit.>. SAA is a simple baseline approach and utilizes Grounding-DINO <cit.> for the anomaly region generator and SAM <cit.> for the anomaly region refiner. SAA+ is an improved method of SAA and it incorporates domain expert knowledge and target image context into SAA. 
Given that AD necessitates localization, it is expected that the number of works employing localization foundation models such as SAM will continue to increase.



  §.§.§ Medical Anomaly Detection

While most works on CLIP-based AD focus on industrial AD, recent studies have begun to challenge medical anomaly detection (medical AD) <cit.>.
CLIP-based medical AD is a more challenging area than industrial AD due to the larger gap between different data modalities. A representative work on medical AD is MVFA <cit.>. MVFA is a method specifically tailored for medical AD. It incorporates multiple residual adapters into the CLIP's visual encoder to reduce the domain gap, enabling a stepwise enhancement of visual features across different levels. 
The future progression of medical AD and industrial AD offers an intriguing perspective, exploring whether these fields will develop independently or influence each other.
However, when considering practical applications, it should be noted that medical AD faces the challenge that anomalies are not always describable in the language. Therefore, the development of medical AD methods that do not use CLIP is also important.






 §.§ Discussion


We discuss the similarities and differences between CLIP-based OOD detection and CLIP-based AD to deepen our understanding of CLIP-based OOD detection.



  §.§.§ Difference between Each Methodology

Differing Scopes of OOD
OOD detection and AD differ significantly in the scope of OOD (anomaly) they cover, which leads to differences in methodologies, particularly in the use of OOD prompts. As explained in Sec. <ref>, sensory AD is intended for specific use cases like industrial inspection, where samples deviating from predefined normality (defective products) are considered anomalies <cit.>. In other words, in sensory AD, the anomaly space is limited to damaged objects with shared semantics, and anomalies like images of dogs are not expected. This limited anomaly space allows even simple prompts to achieve decent performance. Therefore, as shown in Table <ref>, all AD methods utilize anomaly prompts.
Conversely, in OOD detection, as explained in Sec. <ref>, anything semantically different from the ID class is considered OOD. Thus, utilizing naive manual OOD prompts is forbidden (even if it improves benchmark performance). This vastness of the OOD space is the key factor differentiating the methodologies between the two fields.



Essential Features Learned for Detecting OOD
There is a significant difference in the features that need to be captured between CLIP-based OOD detection and CLIP-based AD. In CLIP-based OOD detection, 
it is ideal to learn a more compact ID decision boundary that produces low uncertainty for the ID data, with high uncertainty for OOD data <cit.>.
On the other hand, CLIP-based AD aims to learn the anomalies, instead of ID decision boundary for detecting anomalies <cit.>. 
The learned features are entirely different, which is helpful in developing each training-based method.



Difficulty of Localization Task
CLIP-based OOD detection and CLIP-based AD differ significantly in the difficulty of OOD (anomaly) localization tasks. In CLIP-based AD, anomaly segmentation is a mainstream task, often performed alongside classification in many papers. However, in CLIP-based OOD detection, there has been no research on object-level OOD detection/segmentation. Object-level OOD detection aims to detect OOD objects <cit.>. The inactivity is related to the size of the OOD space, and the too-vast space of OOD makes it difficult to identify OOD objects with prompts effectively. To pave the way for future development, the foundation models for localization such as SAM <cit.>, which can segment individual objects, have the potential to address object-level OOD detection/segmentation. Object-level OOD detection/segmentation using SAM is a promising future research direction.





  §.§.§ Similarity between Each Methodology

Each Problem Setting
The existing problem settings for CLIP-based AD and CLIP-based OOD detection are similar. Both primarily focus on zero-shot and few-shot settings and can be categorized into training-free, auxiliary training-based, and ID training-based methods. By examining each problem setting more closely, we can observe, for instance, that while open-vocabulary AD is predominant in CLIP-based AD, it has not been deeply explored in CLIP-based OOD detection. This provides valuable insights into future directions that are crucial for OOD detection.



History of Approaches
The history of the progress of methods for CLIP-based AD and CLIP-based OOD detection is similar. For instance, both problems initially started with naive methods with manual OOD prompts (ZeroOE <cit.> for OOD detection, WinCLIP <cit.> for AD). To address the issues with these initial approaches, subsequent methods emerged that replaced OOD prompts with learnable parameters (LSN <cit.> and NegPrompt <cit.> for OOD detection, and AnomalyCLIP <cit.> for AD). Therefore, by carefully examining each other's fields, there is potential for mutual enhancement and interaction in the future.


§ EVOLUTION IN LVLM ERA


In this section, we introduce the early advances in OOD detection and AD in the Large Vision Language Models (LVLM) era. While previous sections focused on VLMs such as CLIP, this section shifts our focus to the more emerging topic of “Large" VLMs.
Recent advancements in computer vision have led to the emergence of LVLMs such as GPT-4V <cit.> and LLaVA <cit.>. 
Although these fields are still in their early stages with limited papers, this survey provides a deep introduction to each problem in the hope that our detailed review can help foster further advancements in this area.







 §.§ Change of Each Problem

i. Sensory AD → Sensory AD
Sensory AD has continued to develop in the LVLM era <cit.>. The use of LVLMs has made AD applicable in many domains and modalities <cit.>. 



ii. OOD Detection → Unsolvable Problem Detection
In the LVLM era, OOD detection has evolved into a new task termed Unsolvable Problem Detection (UPD) <cit.>. UPD evaluates the LVLMs' ability to recognize and abstain from answering unexpected or unsolvable input questions, effectively expanding the scope of OOD detection into the context of Visual Question Answering (VQA) tasks. This shift to the VQA task has significantly broadened the concept of OOD detection to a wider range of AI tasks involving LVLMs.




 §.§ Unsolvable Problem Detection



  §.§.§ Summary of Problem

Background
Following the recent revolutionary development of LLMs <cit.>, LVLMs <cit.> have demonstrated remarkable capabilities in diverse applications <cit.>. However, a significant concern has arisen regarding the reliability of these models, specifically their ability to generate accurate and trustworthy information. These models frequently produce incorrect or misleading information, a phenomenon referred to as “hallucination” <cit.>.
Among the various hallucination issues <cit.>, the challenge of identifying out-of-place questions is crucial for deploying LVLMs in safety-critical applications. This challenge extends the concept of OOD detection to the VQA tasks for LVLMs and represents a specific aspect of LVLMs' trustworthiness.



Definition
Unsolvable Problem Detection (UPD) is a task to measure the trustworthiness of LVLMs, which is designed to evaluate models' capacity to withhold answers when faced with unsolvable problems. The UPD task can be categorized into three distinct problem types: Absent Answer Detection (AAD), Incompatible Answer Set Detection (IASD), and Incompatible Visual Question Detection (IVQD).
The details of each setting are as follows:

      
  * Absent Answer Detection (AAD): AAD evaluates the model's capacity to determine when the correct answer is absent from the provided options.
      
  * Incompatible Answer Set Detection (IASD): IASD assesses the model's ability to discern answer choices that are completely irrelevant to the given question and image.
      
  * Incompatible Visual Question Detection (IVQD): IVQD evaluates the model's capacity to discern whether a question and image are unrelated or mismatched.




Benchmark
Miyai  <cit.> created MM-UPD Bench for the UPD challenge. MM-UPD encompasses MM-AAD, MM-IASD, and MM-IVQD benchmarks for each UPD problem. Each benchmarks are created on the top of MMBench (dev) <cit.>, which is a systematically designed objective benchmark for evaluating various abilities of LVLMs. Following the definition of each ability in MMBench (“Coarse Perception: Image Scene” and “Logic Reasoning: Future Prediction”), MM-UPD evaluates the trustworthiness of LVLMs from various abilities.

Although MM-UPD is the main benchmark, the adaptation cost of creating UPD problems is not high, making it highly applicable to other benchmarks. For instance, the recently proposed MuirBench <cit.>, a benchmark for multi-image understanding, has incorporated the concept of UPD by adding unsolvable problems. 



Application
UPD has a wide range of applications, from everyday use of LVLMs to robot manipulation. Especially when incorporating LVLMs into safety-critical domains such as robot manipulation <cit.> and autonomous driving <cit.>, there is a risk of significant problems if the LVLM fails to identify erroneous user questions and makes incorrect predictions. UPD serves as a task to ensure safety in such safety-critical scenarios.




Evaluation
UPD introduces new evaluation metrics that incorporate the concept of the evaluation protocols for OOD detection, taking into account the prediction distribution for both standard (ID) and UPD (OOD) samples. 
The rationale is that ideal LVLMs should not only give correct answers for the standard problems but need to withhold answering in the UPD scenario where the problem is unsolvable. 
To better reflect the ideal behavior of LVLMs, UPD measures several metrics:
(i) Standard Accuracy: 
The accuracy on standard problems where the image, question, and answer sets are all aligned, and the ground-truth answer is always present within the provided options. (ii) UPD Accuracy: The accuracy of AAD/IASD/IVQD problems. (iii) Dual Accuracy: Accuracy of standard and UPD pairs. We count success only if the model is correct for both standard and UPD problems.




  §.§.§ Findings

In the following, we briefly summarized the findings of the UPD challenge <cit.>.



1. Most LVLMs Hardly Hesitate to Answer.
Most LVLMs, especially open-source LVLMs, have significantly low UPD accuracies, which indicates the difficulty of the UPD challenge. For example, LLaVA-1.5 <cit.> and CogVLM <cit.>, which are state-of-the-art LVLMs, completely fail to withhold answering. GPT-4V achieves higher performances than other LVLMs due to its safety training process <cit.>. However, there is still a performance gap from the upper bound scores.



2. Performance Tendency Differs a lot by Each Ability in the Benchmark.
The performance of LVLMs differs in each ability in the MM-UPD Bench. For instance, GPT-4V has its limitation in attribute comparison and LLaVA-NeXT-34B has its limitation in object localization.



3. Effective Prompt Strategies Vary Across Different LVLMs 
Effective prompt strategies vary across different LVLMs. In the original paper, they experimented with an option-based prompt approach that adds an option of “None of the above" and an instruction-based approach that adds an instruction “If all options are incorrect, answer None of the above". As a result, the effectiveness of each approach differs significantly depending on the type of LVLMs. This highlights the difficulty of finding an effective prompt strategy for all LVLMs. 




 §.§ Anomaly Detection in LVLM Era



  §.§.§ Summary of Problem

Background
Anomaly detection is a crucial task in a variety of domains and data types. However, existing anomaly detection models are often designed for specific domains or modalities <cit.>. Also, current AD methods only provide an anomaly score for the test sample and require a manual threshold to distinguish between normal and anomalous instances for each sample <cit.>. To facilitate real-world applications, developing a system capable of expressing anomalies in natural language across various modalities and domains is crucial for ensuring accessibility to a wider range of users.



Definition
The definition of AD remains consistent with conventional and CLIP-based AD, aiming to identify samples that deviate from predefined normality. The key distinction lies in the output. While previous methods produced an anomaly score requiring a manual threshold, AD with LVLMs aims to recognize and describe anomalies using text, eliminating the need for manual thresholds and enhancing human interpretability.



Benchmark
Since the field of AD with LVLMs is in its infant stage, there are still no unified benchmarks. AnomalyGPT <cit.> focuses on industrial image anomaly detection/localization and uses the standard benchmarks MVTec-AD <cit.> and VisA <cit.>. 
More recently, Cao  <cit.> extend the domain and modality and demonstrate the applications in industrial image anomaly detection/localization (MVTec-AD <cit.>), point cloud anomaly detection (MVTec 3D <cit.>), medical image anomaly detection/localization (Chest X-ray <cit.>, Head CT <cit.>), logical anomaly detection (MVTec LOCO <cit.>), pedestrian anomaly detection (UCF-Crime Dataset <cit.>), traffic anomaly detection (Kaggle Accident Detection <cit.>), and time series anomaly detection (Outlier Detection Dataset <cit.>).



Evaluation
Evaluation in anomaly detection with LVLMs is an open challenge.  AnomalyGPT <cit.> asks LVLMs the question “Is there an anomaly in this image?" and determines anomaly or normal based on the simple rule-based approach of whether the response contains a “yes" or “no". However, this rule-based approach is not robust, as a response is considered correct even if the explanation following “yes" is completely incorrect. On the other hand, Cao  <cit.> conducted only qualitative evaluations and left quantitative evaluations as an open challenge. Therefore, the evaluation of anomaly detection by LVLM is a future challenge.



  §.§.§ Findings

Cao  <cit.> described the observations of GPT-4V in the paper, so we briefly summarized them here.



1. GPT-4V Excels in Zero/One-shot Settings across Various Modalities and Fields.
GPT-4V shows proficiency in identifying anomalies in multi-modality (images, point clouds, X-rays) and multi-field (industrial, medical, pedestrian, traffic, and time series anomaly detection). In addition, GPT-4V demonstrates strong performance in both zero-shot and one-shot settings. 



2. GPT-4V can Understand Both Global and Fine-grained Anomalies.
GPT-4V can recognize both global and local abnormal patterns or behaviors, which indicates the ability to understand global and fine-grained semantics.



3. GPT-4V can be Enhanced with Increasing Prompts.
By giving more context and information, the model significantly improves its ability to detect anomalies accurately. 



§ POTENTIAL CHALLENGES


This section discusses potential challenges for CLIP-based OOD detection that may be highlighted by the widespread adoption of our framework. Since similar or ambiguous problem settings currently exist, we present these challenges to avoid future confusion for readers.





 §.§ Open-vocabulary OOD Detection
 

The existing setting for open-vocabulary OOD (OV-OOD) detection <cit.> involves training with images from a small subset of ID classes and evaluating with all ID class names, as explained in Sec. <ref>. However, this problem definition has several issues: (i) The OOD detection performance on ID classes not included in the training subset is unclear, (ii) This contradicts the settings of the existing open-vocabulary settings, including CLIP-based AD (Sec. <ref>), which assume no explicit class overlap between training and test ID data <cit.>, leading to confusion in the fields.
Therefore, for clear evaluation and common understanding in the field, we redefine the OV-OOD detection as in-class open-vocabulary OOD detection (in-class OV-OOD detection) and zero-shot open-vocabulary OOD detection (zero-shot OV-OOD detection). The illustration for each setting is shown in Fig. <ref>. Detailed explanations for each setting are as follows:



In-class OV-OOD Detection
In-class OV-OOD detection is the same as the original setting proposed by  <cit.>, involving training with images from a small subset of ID classes and evaluating on all ID class names including training classes.
In-class OV-OOD detection should be evaluated in the few-shot OOD detection setting to assess how well these methods retain their performance when trained with limited class data. In this setting, it is preferable to employ the existing benchmark setup that randomly selects subset ID classes from all ID classes <cit.>.



Zero-shot OV-OOD Detection
Zero-shot OV-OOD detection shares the training setting with in-class evaluation, using a small subset of ID classes. However, it differs in the testing phase, where it exclusively utilizes classes not included in the training subset. 
This problem setting is consistent with the common open-vocabulary setting <cit.> and the open-vocabulary setting in CLIP-AD <cit.>.
Zero-shot OV-OOD detection can be positioned within the area of zero-shot OOD detection, which would facilitate the research of zero-shot OOD detection. When selecting the subset of classes for training, we should ensure semantic dissimilarity between the classes in the evaluation and training sets to guarantee the validity of the zero-shot problem setting. For instance, when splitting ImageNet classes into training and evaluation subsets for zero-shot OV-OOD detection, it would be preferable to consider the hierarchical classes of ImageNet to avoid similar semantics between training and evaluation subsets.




 §.§ CLIP-based Full-spectrum OOD Detection






CLIP-based full-spectrum OOD detection is a well-established task with existing research <cit.>. However, with the future prevalence of hard OOD detection, which incorporates the OSR setup, the distinction between this task and open-set domain generalization (OSDG)  <cit.> may become ambiguous. OSDG shares a similar motivation of realizing models capable of both generalization and detection. The illustration of both problem settings is shown in Fig. <ref> (a)(b). OSDG assumes that only data with covariate shifts will be input during testing, and aims to classify input samples into one of ID classes if their semantics align with ID classes and detect OOD that exhibit semantic shifts.
OSDG is a research area with a much smaller number of works compared to the main topics of this survey (OD, AD, ND, OSR, and OOD detection), but, in recent years, some works have tackled OSDG using CLIP <cit.>.

To eliminate potential ambiguity, this paper defines a hard full-spectrum OOD (FS-OOD) detection, following our generalized OOD detection v2. The illustration of the problem setting for hard FS-OOD detection is shown in Fig. <ref> (c).
Hard FS-OOD detection expands the scope of existing FS-OOD detection by utilizing the conventional OSR setup. 
It introduces OOD samples from different categories within the training domain and OOD samples that share the same covariate shift as the target domain but exhibit semantic shifts. This task can be regarded as an extensive version of both existing FS-OOD detection and OSDG tasks. Hard FS-OOD detection represents a developing challenge with promising potential for future research.




§ FUTURE DIRECTIONS


In this section, we discuss the future directions of OOD detection and UPD.
For OOD detection, we explore not only OOD detection for VLMs but also single-modal OOD detection, with a specific focus on emerging challenges as VLMs evolve. 
For a discussion of the long-standing challenges in OOD detection, we can refer the readers to the previous generalized OOD detection paper <cit.>.



 §.§ OOD Detectin for Vision Language Models

a. Hard OOD Detection
Hard OOD detection will become increasingly important in the future due to its high practicality and the challenging nature of the problem. Hard OOD detection utilizes the OSR benchmark setup, where some classes within a single dataset are designated as ID and others as OOD. In this field, not only small datasets such as ImageNet-10 and ImageNet-20 <cit.> but also datasets with a larger number of classes such as ImageNet-protocol <cit.> have been proposed. Many existing studies, such as LoCoOp <cit.> and LSN <cit.>, primarily use the common ImageNet OOD benchmark, so hard OOD detection has not yet been well studied. This field will develop further in the future.



b. Post-hoc Methods
To propose post-hoc methods is important for the fundamental performance improvement of CLIP-based OOD detection. The methods of directly employing an ID classifier such as MCM <cit.> are called post-hoc methods. Prior to CLIP, various approaches were proposed  <cit.>.
However, CLIP-based post-hoc methods often underperform methods with additional OOD prompts, so they are not extensively researched in zero-shot OOD detection. However, we should focus on the scalability of post-hoc methods. The post-hoc methods <cit.> can be easily applied to many subsequent methods <cit.>, bringing fundamental performance improvements. Furthermore, very recently, post-hoc methods specifically tailored for prompt learning methods have also emerged <cit.>.
Therefore, proposing post-hoc methods and demonstrating the improvements not only in zero-shot but also in subsequent few-shot settings <cit.> is crucial, even if they underperform methods using OOD prompts in the zero-shot setting. This field should continue evolving, mirroring its growth before the advent of CLIP.



c. Bridging the Gap with Closed-set Classifiers.
OOD detection ensures the safety of ID classifiers, so it is crucial to bridge the gap between the advancements in existing closed-set classifiers and OOD detection. 
Currently, the representative method for few-shot OOD detection is LoCoOp <cit.>, a text prompt learning method based on CoOp. However, in closed-set settings, few-shot learning methods based on text prompt learning have been proposed other than CoOp <cit.>. Furthermore, text-based prompt learning methods only train the text prompts, so they cannot handle differences in image domains. Therefore, adopting methods that can handle image domain differences <cit.> for OOD detection is essential for bridging the gap with closed-set ID classifiers.



d. Training-free Few-shot OOD Detection
The research direction of training-free few-shot OOD detection is still in its infant stage, with only one existing study <cit.>. Given the prevalence of training-based methods in few-shot OOD detection, proposing methods that do not require training is crucial. Considering the advancements of training-free methods in CLIP-based anomaly detection, we anticipate a similar trajectory for CLIP-based OOD detection.
Future directions include refining adapter-based methods or leveraging external knowledge such as retrieval augmentation <cit.>.
Addressing training-free few-shot OOD detection is a pivotal step towards realizing more computationally efficient OOD detection in the future.



e. Full-spectrum OOD Detection
CLIP-based full-spectrum OOD (FS-OOD) detection is a promising research area <cit.>. In practical applications, there is a strong motivation to create models that can not only detect semantically shifted OOD inputs but also generalize to covariate-shifted data <cit.>. Within CLIP-based methods, OOD detection and generalization are often discussed in separate contexts <cit.>, resulting in a trade-off between detection and generalization performance <cit.>. Furthermore, there exists potential ambiguity in the distinction between FS-OOD detection and open-set domain generalization.
To eliminate the potential ambiguity, we have formulated a potential problem setting called hard FS-OOD detection.
We hope this survey inspires further advancements and developments for FS-OOD detection.



f. Open-vocabulary OOD Detection
Open-vocabulary OOD (OV-OOD) detection has a high practical potential, but it is still in its infant stages <cit.>.
In particular, as explained in Sec. <ref>, zero-shot OV-OOD detection is a potential research field. 
We hope that this survey paper will inspire future efforts in OV-OOD detection.



g. Object-level OOD Detection
Object-level OOD detection remains an unexplored area. As discussed in Sec. <ref>, this is due to the vastness of the OOD space, which makes it difficult to identify OOD objects using texts. 
To pave the way for future advancements in object-level OOD detection/segmentation, foundation models for localization, such as SAM <cit.>, offer a promising solution. By integrating these models with methods such as MCM <cit.>, we can potentially achieve object-level OOD detection and segmentation, opening up a new frontier in OOD detection research.




 §.§ Single-modal OOD Detection

a. Leveraging Large Pre-trained Models
Leveraging large pre-trained models is important for single-modal OOD detection. Numerous methods for OOD detection conduct experiments using backbones trained from scratch and do not utilize pre-trained models <cit.>. 
In a recent study, Miyai  <cit.> systematically investigated the impact of pre-training on OOD detection from both the perspectives of the types of OOD data and pre-training algorithms <cit.>.
Dong  <cit.> explored parameter-efficient learning for single-modal OOD detection and proposed DSGF, which leverages both fine-tuned features and original pre-trained features. While leveraging large pre-trained models <cit.> with lightweight tuning is an active area of research in single-modal closed-set classification <cit.>, there have been limited studies for single-modal OOD detection, which presents a promising avenue for future research.



b. Real-world Benchmarks and Evaluations
Considering the future development of CLIP-based OOD detection, there should be increasing focus on expanding the scope of benchmarks to encompass real-world scenarios where CLIP is less applicable.
For instance, recently, Baek  <cit.> introduced ImageNet-ES, consisting of variations in environmental and camera sensor factors by directly capturing 202k images with an actual camera in a controlled testbed, which bridges the gap between the common benchmark and the real-world scenario. Besides, utilizing datasets such as WILDS <cit.>, which consider real-world data shifts, or datasets for medical OOD detection <cit.>, can provide valuable insights, especially in safety-critical applications like autonomous driving and medical image analysis.




 §.§ Unsolvable Problem Detection

a. Exploring Effective Solutions
It is important to propose effective solutions for UPD. One of the potential approaches is to adapt the methodologies of OOD detection to UPD. For example, the perplexity of the LVLM's response could be used as a score to identify unsolvable queries. 
Furthermore, proposing model-agnostic post-hoc methods is also important to enhance the reliability of many LVLMs.
Therefore, incorporating the concepts of OOD detection techniques for UPD is an important direction for future work.



b. Extension to Diverse Benchmarks
MM-UPD Bench consists of general QA datasets. However, UPD can be incorporated into more diverse benchmarks including domain-specific knowledge for advanced reasoning <cit.> and multi-image understanding <cit.>. For example, MuirBench <cit.> incorporated the concept of UPD as a metric for robust evaluation. 
Integrating the concept of UPD into benchmarks is essential for evaluating the robustness and trustworthiness of LVLMs in their target tasks.



c. Theoretical Understanding of UPD
Theoretically understanding the reasons behind the difficulty of UPD could provide the community with valuable insights. Theorizing the behavior of LVLMs poses a shared challenge in the field, highlighting the importance of collaborative efforts within the community.



§ CONCLUSION


In this survey, we comprehensively review the evolution of the five problems including AD, ND, OSR, OOD detection, and OD in the VLM era, and propose a framework of generalized OOD detection v2. Our framework identifies OOD detection and AD as the primary challenges in the VLM era, which highlights the demanding challenges and fosters collaborative efforts among each community. 
By articulating the shifts in the definitions, problem settings, and benchmarks, we encourage subsequent works to accurately understand their evolving target problems in the VLM era.
By sorting out the methodologies, we hope that readers can easily grasp the mainstream methods, identify important baselines and novel problem settings, and propose future solutions.
By shedding light on recent studies in the LVLM era, we hope that researchers within each community can identify promising research directions in this emerging era.
By providing future directions, we hope that our survey will clarify the tasks to be tackled by future works in the VLM and LVLM era, thereby facilitating future advances in the right direction.


§ ACKNOWLEDGMENT

This work was supported by JST BOOST, Japan Grant Number JPMJBS2418 and JST JPMJCR22U4. We thank Toyooka Mashiro (AYM Lab at UTokyo) for his valuable assistance in designing the figures, and Kazuki Egashira, Yuki Imajuku, Takubon Son, and Zaiying Zhao (AYM Lab at UTokyo) for valuable feedback on the paper. 

ieeetr






[
    < g r a p h i c s >
]Atsuyuki Miyai
(Student Member, IEEE)
received his B.E. degree in Information and Communication Engineering in 2022 and his M.E. degree in Interdisciplinary Studies in Information Science in 2024 from The University of Tokyo. He is currently a Ph.D. student with the Department of Information and Communication Engineering at The University of Tokyo. His research topic is the safety of vision language models and foundation models. He is an IEEE Student Branch Chair at The University of Tokyo.




[
    < g r a p h i c s >
]
    Jingkang Yang is currently a PhD student at the College of Computing and Data Science (CCDS) at Nanyang Technological University (NTU), Singapore, working in the MMLab@NTU under the supervision of Dr. Ziwei Liu. His research interests include visual generalist models and AI safety for foundation models. He has published over 20 papers in relevant fields at top-tier conferences and journals, including CVPR, ICCV, ECCV, NeurIPS, ICLR, and IJCV. He serves as a reviewer of CVPR, ICCV, ECCV, NeurIPS, ICLR, etc., and is an outstanding reviewer in ICCV 2021 and CVPR 2024.




[
    < g r a p h i c s >
]Jingyang Zhang received his B.E. degree in Electronic Engineering from Tsinghua University, Beijing, China, in 2019.
Since then, he has been pursuing the Ph.D. degree at Dept. of Electrical and Computer Engineering at Duke University, supervised by Dr. Yiran Chen and Dr. Hai (Helen) Li.
His research spans from robustness of deep learning-based vision systems to (more recently) generative AI and multi-modal LLMs.




[
    < g r a p h i c s >
]Yifei Ming received his Ph.D. in Computer Science from the University of Wisconsin-Madison in 2024, advised by Dr. Yixuan Li. He is currently a Research Scientist at Salesforce AI Research. His research interests encompass reliable machine learning, vision-language models, large language models, and responsible foundation models. He has a prolific publication record and serves on the program committees of leading machine learning conferences and journals, including NeurIPS, ICLR, ICML, EMNLP, and IJCV.




[
    < g r a p h i c s >
]Yueqian Lin
received a B.S. degree in Data Science from Duke Kunshan University in 2024, graduating summa cum laude with signature work distinction. He is currently pursuing a Ph.D. in Electrical and Computer Engineering at Duke University under the supervision of Dr. Yiran Chen and Dr. Hai (Helen) Li. His research focuses on robustness in generative models.




[
    < g r a p h i c s >
]Qing Yu
(Member, IEEE) 
received his B.E. degree in Information and Communication Engineering and the M.E. degree in Interdisciplinary Studies in Information Science from The University of Tokyo in 2018 and 2020, respectively. He completed his Ph.D. in Information Science and Technology from The University of Tokyo in 2023. He is currently a Research Scientist at LY Corporation, Japan. His research interests include multimodal recognition and generation.




[
    < g r a p h i c s >
]Go Irie
(Member, IEEE) received his B.E. and M.E. in System Engineering from Keio University, Japan, in 2004 and 2006, respectively. He completed his Ph.D. in Information Science and Technology from The University of Tokyo in 2011. 
He is currently an Associate Professor at the Department of Information and Computer Technology, Tokyo University of Science, Japan. He was a research scientist at NTT Corporation, Japan, from 2006 to 2022, and a Visiting Research Scholar at Columbia University from 2012 to 2013. His research interests include pattern recognition, machine learning and media understanding.




[
    < g r a p h i c s >
]Shafiq Joty is currently a Research Director at Salesforce Research (Palo Alto, USA), where he oversees the NLP group's efforts in large language modeling (LLM) and generative AI. He also holds the position of a tenured Associate Professor (currently on leave) in the School of Computer Science and Engineering (SCSE) at NTU, Singapore. He was a founding manager of the Salesforce Research Asia (Singapore) lab. His research has contributed to over 30+ patents and 145+ papers in top-tier NLP and ML conferences and journals. He has served as the Program Chair of SIGDIAL-2023, as a member of the best paper award committees for ICLR-23 and NAACL-22, and in the capacity of a (senior) area chair for many of the leading NLP and ML conferences.





[
    < g r a p h i c s >
]Yixuan Li (Member, IEEE) received the PhD degree from Cornell University, in 2017, advised by John E. Hopcroft. She is an assistant professor in the Department of Computer Sciences, the University of Wisconsin-Madison. Subsequently, she was a postdoctoral scholar in the computer science department, Stanford University. Her research focuses on the algorithmic and theoretical foundations of learning in open worlds. She has served as Area Chair for ICLR, NeurIPS, ICML, and Program Chair for Workshop on Uncertainty and Robustness in Deep Learning. She is the recipient of the AFOSR Young Investigator Program (YIP) award, NSF CAREER award, MIT Technology Review Innovator Under 35, Forbes 30 Under 30 in Science, and multiple faculty research awards from Google, Meta, and Amazon. Her works received a NeurIPS Outstanding Paper Award, and an ICLR Outstanding Paper Award Honorable Mention, in 2022.





[
    < g r a p h i c s >
]Hai (Helen) Li
(Fellow, IEEE) received the Ph.D. degree from Purdue University in 2004. 
Dr. Li is currently the Clare Boothe Luce Professor and Department Chair of the Electrical and Computer Engineering Department at Duke University.
Her current research interests include neuromorphic circuits and systems for brain-inspired computing, machine learning acceleration and trustworthy AI, conventional and emerging memory design and architecture, and software and hardware co-design.
Dr. Li is a recipient of the NSF Career Award (2012), DARPA Young Faculty Award (2013), TUM-IAS Hans Fischer Fellowship from Germany (2017), and ELATE Fellowship (2020). She received 9 best paper awards and additional 9 best paper nominations from international conferences. Dr. Li is a Distinguished Lecturer of the IEEE CAS Society (2018-2019) and a Distinguished Speaker of ACM (2017-2020). 





[
    < g r a p h i c s >
]Ziwei Liu
(Member, IEEE) is currently a nanyang assistant professor with Nanyang Technological University, Singapore. His research interests include computer vision machine learning and computer graphics. He has published extensively on top-tier conferences and journals in relevant fields, including CVPR, ICCV, ECCV, NeurlPS, ICLR, ICML, IEEE Transactions on Pattern Analysis and Machine Intelligence, ACM Transactions on Graphics and Nature - Machine Intelligence. He is the recipient of ICCV Young Researcher Award, HKSTP Best Paper Award, CVPR Best Paper Award Candidate, ICBS Frontiers of Science Award and MIT Technology Review Innovators under 35 Asia Pacific. He serves as an area chair of CVPR, ICCV, ECCV, NeurlPS and ICLR, as well as an associate editor of International Journal of Computer Vision.





[
    < g r a p h i c s >
]Toshihiko Yamasaki
(Member, IEEE) received the Ph.D. degree from The University of Tokyo. He is currently a Professor at the Department of Information and Communication Engineering, Graduate School of Information Science and Technology, The University of Tokyo. He was a JSPS Fellow for Research Abroad and a visiting scientist at Cornell University from 2011 to 2013. His current research interests are computer vision, multimedia, pattern recognition, machine learning. He is a pioneer of attractiveness computing.





[
    < g r a p h i c s >
]Kiyoharu Aizawa
(Fellow, IEEE) received the B.E., the M.E., and the Dr.Eng. degrees in Electrical Engineering all from the University of Tokyo, in 1983, 1985, 1988, respectively. He is a professor with the Department of Information and Communication Engineering, and director of VR center, University of Tokyo. He was a visiting assistant professor with the University of Illinois from 1990 to 1992. His research fields are multimedia, image processing, and computer vision, with a particular interest in interdisciplinary and cross-disciplinary issues. He received the 1990, 1998 Best Paper Awards, the 1991 Achievement Award, 1999 Electronics Society Award from IEICE Japan, and the 1998 Fujio Frontier Award, the 2002 and 2009 Best Paper Award, and 2013 Achievement award from ITE Japan. He received the IBM Japan Science Prize in 2002. He is on Editorial Board of ACM TOMM. He served as the Editor-in-Chief of Journal of ITE Japan, an Associate Editor of IEEE TIP, TCSVT, TMM, and MultiMedia. He has served a number of international and domestic conferences; he was a General co-Chair of ACM Multimedia 2012 and ICMR2018. He is a Fellow of IEEE, IEICE, ITE and a member of Science Council of Japan. 






 Paper Title: OmniParser for Pure Vision Based GUI Agent.



Extracting doubly-excited state lifetimes in helium directly in the time domain with attosecond noncollinear four-wave-mixing spectroscopy
    Stephen R. Leone
    August 5, 2024
==========================================================================================================================================




§ ABSTRACT 
   The recent success of large vision language models shows great potential in driving the agent system operating on user interfaces. However, we argue that the power multimodal models like GPT-4V as a general agent on multiple operating systems across different applications is largely underestimated due to the lack of a robust screen parsing technique capable of: 1) reliably identifying interactable icons within the user interface, and 2) understanding the semantics of various elements in a screenshot and accurately associate the intended action with the corresponding region on the screen. To fill these gaps, we introduce OmniParser, a comprehensive method for parsing user interface screenshots into structured elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface. 
   We first curated an interactable icon detection dataset using popular webpages and an icon description dataset. These datasets were utilized to fine-tune specialized models: a detection model to parse interactable regions on the screen and a caption model to extract the functional semantics of the detected elements.
   
   OmniParser significantly improves GPT-4V's performance on ScreenSpot benchmark. And on Mind2Web and AITW benchmark, OmniParser with screenshot only input outperforms the GPT-4V baselines requiring additional information outside of screenshot. 




§ INTRODUCTION

Large language models have shown great success in their understanding and reasoning capabilities. More recent works have explored the use of large vision-language models (VLMs) as agents to perform complex tasks on the user interface (UI) with the aim of completing tedious tasks to replace human efforts <cit.>. Despite the promising results, there remains a significant gap between current state-of-the-arts and creating widely usable agents that can work across multiple platforms, e.g. Windows/MacOS, IOS/Android and multiple applications (Web broswer Office365, PhotoShop, Adobe), with most previous work focusing on limiting applications or platforms.

While large multimodal models like GPT-4V and other models trained on UI data <cit.> have demonstrated abilities to understand basic elements of the UI screenshot, action grounding remains one of the key challenges in converting the actions predicted by LLMs to the actual actions on screen in terms of keyboard/mouse movement or API call <cit.>. It has been noted that GPT-4V is unable to produce the exact x-y coordinate of the button location, Set-of-Mark prompting <cit.> proposes to overlay a group of bounding boxes each with unique numeric IDs on to the original image, as a visual prompt sent to the GPT-4V model. By applying set-of-marks prompting, GPT-4V is able to ground the action into a specific bounding box which has ground truth location instead of a specific xy coordinate value, which greatly improves the robustness of the action grounding <cit.>. However, the current solutions using SoM relies on parsed HTML information to extract bounding boxes for actionable elements such as buttons, which limits its usage to web browsing tasks. We aim to build a general approach that works on a variety of platforms and applications. 

In this work, we argue that previous pure vision-based screen parsing techniques are not satisfactory, which lead to significant underestimation of GPT-4V model's understanding capabilities. And a reliable vision-based screen parsing method that works well on general user interface is a key to improve the robustness of the agentic workflow on various operating systems and applications. We present OmniParser, a general screen parsing tool to extract information from UI screenshot into structured bounding box and labels which enhances GPT-4V's performance in action prediction in a variety of user tasks. 


We list our  contributions as follows: 

    
  * We curate a interactable region detection dataset using bounding boxes extracted from DOM tree of popular webpages. 
    
  * We propose OmniParser, a pure vision-based user interface screen parsing method that combines multiple finetuned models for better screen understanding and easier grounded action generation.
    
  * We evaluate our approach on ScreenSpot, Mind2Web and AITW benchmark, and demonstrated a significant improvement from the original GPT-4V baseline without requiring additional input other than screenshot. 
    




§ RELATED WORKS



 §.§ UI Screen Understanding


There has been a line of modeling works focusing on detailed understanding of UI screens, such as Screen2Words <cit.>, UI-BERT <cit.>, WidgetCaptioning <cit.>, Action-BERT <cit.>. These works demonstrated effective usage of multimodal models for extracting semantics of user screen. But these models rely on additional information such as view hierarchy, or are trained for visual question answering tasks or screen summary tasks. 


There are also a couple publicly available dataset that on UI screen understanding. Most notably the Rico dataset <cit.>, which contains more than 66k unique UI screens and its view hierarchies. Later <cit.>  auguments Rico by providing 500k human annotations on the original 66k RICO screens identifying various icons based on their shapes and semantics, and associations between selected general UI elements (like icons, form fields, radio buttons, text inputs) and their text labels. Same on mobile platform, PixelHelp <cit.> provides a dataset that contains UI elements of screen spanning across 88 common tasks. In the same paper they also released RicoSCA which is a cleaned version of Rico. For the web and general OS domain, there are several works such Mind2Web <cit.>, MiniWob++<cit.>, Visual-WebArena <cit.>, and OS-World <cit.> that provide simulated environment, but does not provide dataset explicitly for general screen understanding tasks such as interactable icon detection on real world websites. 

To address the absence of a large-scale, general web UI understanding dataset, and to keep pace with the rapid evolution of UI design, we curated an icon detection dataset using the DOM information from popular URLs avaialbe on the Web. This dataset features the up-to-date design of icons and buttons, with their bounding boxes retrieved from the DOM tree, providing ground truth locations. 



 §.§ Autonomous GUI Agent


Recently there has been a lot of works on designing autonomous GUI agent to perform tasks in place of human users. One line of work is to train an end-to-end model to directly predict the next action, representative works include: Pixel2Act <cit.>, WebGUM<cit.> in web domain, Ferret <cit.>, CogAgent <cit.>, and Fuyu <cit.> in Mobile domain. Another line of works involve leveraging existing multimodal models such as GPT-4V to perform user tasks. Representative works include MindAct agent <cit.>, SeeAct agent <cit.> in web domain and agents in <cit.> for mobile domain. 
These work often leverages the DOM information in web browser, or the view hierarchies in mobile apps to get the ground truth position of interactable elements of the screen, and use Set-Of-Marks<cit.> to overlay the bounding boxes on top of the screenshot then feed into the vision-language models. However, ground truth information of interactable elements may not always be available when the goal is to build a general agent for cross-platforms and cross-applications tasks. Therefore, we focus on providing a systematic approach for getting structured elements from general user screens. 






§ METHODS

A complex task can usually be broken down into several steps of actions. Each step requires the model's (e.g. GPT-4V) ability to: 1) understand the UI screen in the current step, i.e. analyzing what is the screen content overall, what are the functions of detected icons that are labeled with numeric ID, and 2) predict what is the next action on the current screen that is likely to help complete the whole task. Instead of trying to accomplish the two goals in one call, we found it beneficial to extract some of the information such as semantics in the screen parsing stage, to alleviate the burden of GPT-4V so that it can leverages more information from the parsed screen and focus more on the action prediction. 


Hence we propose OmniParser, which integrates the outputs from a finetuned interactable icon detection model, a finetuned icon description model, and an OCR module. This combination produces a structured, DOM-like representation of the UI and a screenshot overlaid with bounding boxes for potential interactable elements. We discuss each component of the OmniParser in more details for the rest of the section. 






 §.§ Interactable Region Detection


Identifying interactable regions from the UI screen is a crucial step to reason about what actions should be performed given a user tasks. Instead of directly prompting GPT-4V to predict the xy coordinate value of the screen that it should operate on, we follow previous works to use the Set-of-Marks approach <cit.> to overlay bounding boxes of interactable icons on top of UI screenshot, and ask GPT-4V to generate the bounding box ID to perform action on. However, different from <cit.> which uses the ground truth button location retrieved from DOM tree in web browswer, and <cit.> which uses labeled bounding boxes in the AITW dataset <cit.>, we finetune a detection model to extract interactable icons/buttons. 

Specifically, we curate a dataset of interactable icon detection dataset, containing 67k unique screenshot images, each labeled with bounding boxes of interactable icons derived from DOM tree. We first took a 100k uniform sample of popular publicly availabe urls on the web <cit.>, and collect bounding boxes of interactable regions of the webpage from the DOM tree of each urls. Some examples of the webpage and the interactable regions are shown in <ref>.  

Apart from interactable region detection, we also have a OCR  module to extract bounding boxes of texts. Then we merge the bounding boxes from OCR detection module and icon detection module while removing the boxes that have high overlap (we use over 90% as a threshold). For every bounding box, we label it with a unique ID next to it using a simple algorithm to minimizing the overlap between numeric labels and other bounding boxes. 




 §.§ Incorporating Local Semantics of Functionality

We found in a lot of cases where only inputting the UI screenshot overlayed with bounding boxes and associated IDs can be misleading to GPT-4V. We argue the limitation stems from GPT-4V's constrained ability to simultaneously perform the composite tasks of identifying each icon's semantic information and predicting the next action on a specific icon box. This has also been observed by several other works <cit.>. To address this issue, we incorporate the local semantics of functionality into the prompt, i.e. for each icons detected by the interactable region detection model, we use a finetuned model to generate description of functionality to the icons, and for each text boxes, we use the detected texts and its label.

We perform more detailed analysis for this topic in section <ref>. To the best of our knowledge, there is no public model that is specifically trained for up-to-date UI icon description, and is suitable for our purpose to provide fast and accurate local semantics for the UI screenshot. Therefore we curate a dataset of  7k icon-description pairs using GPT-4o, and finetune a BLIP-v2 model <cit.>  on this dataset. Details of dataset and training can be found in Appendix <ref>. After finetuning, we found the model is much more reliable in its description to common app icons. Examples can be seen in figure <ref>. And in figure <ref>, we show it is helpful to incorporate the semantics of local bounding boxes in the form of text prompt along with the UI screenshot visual prompt. 




















§ EXPERIMENTS AND RESULTS

We conduct experiments on several benchmarks to demonstrate the effectiveness of OmniParser. We start by a motivating experiments showing that current GPT-4V model with set of mark prompting <cit.> is prone to incorrectly assigning label ID to the referred bounding boxes. Then we evaluate on Seeclick benchmark and Mind2Web to further showcase OmniParser with local semantics can improve the GPT-4V's performance on real user tasks on different platforms and applications.   



 §.§ Evaluation on SeeAssign Task


To test the ability of correctly predicting the label ID given the description of the bounding boxes for GPT-4v models, We handcrafted a dataset SeeAssign that contains 112 tasks consisting of samples from 3 different platforms: Mobile, Desktop and Web Browser. Each task includes a concise task description and a screenshot image. The task descriptions are manually created and we make sure each task refers to one of the detected bounding boxes, e.g. 'click on 'settings”, 'click on the minimize button'. During evaluation, GPT-4V is prompted to predict the bounding box ID associated to it. Detailed prompt are specified in Appendix.  
The task screenshot images are sampled from the ScreenSpot <cit.> benchmark, where they are labeled with set of marks using OmniParser. The tasks are further divided into 3 sub-categories by difficulty: easy (less than 10 bounding boxes), medium (10-40 bounding boxes) and hard (more than 40 bounding boxes). 

From table <ref>, we see that GPT-4V often mistakenly assign the numeric ID to the table especially when there are a lot of bounding boxes over the screen. And by adding local semantics including texts within the boxes and short descriptions of the detected icons, GPT-4V's ability of correctly assigning the icon improves from 0.705 to 0.938.  

From figure <ref>, we see that without the description of the referred icon in the task, GPT-4V often fails to link the icon required in the task and the ground truth icon ID in the SoM labeled screenshot, which leads to hallucination in the response. With fine-grain local semantics added in the text prompt, it makes it much easier for GPT-4V to find the correct icon ID for the referred icon. 
























 §.§ Evaluation on ScreenSpot

ScreenSpot dataset <cit.>  is a benchmark dataset that includes over 600 interface
screenshots from mobile (iOS, Android), desktop
(macOS, Windows), and web platforms. The task instructions are manually created so that each instruction corresponds to an actionable
elements on the UI screen. We first evaluate the performance of omniparser using the this benchmark.  In table <ref>, we can see across the 3 different platforms: Mobile, Desktop and Web, omniparser significantly improves the GPT-4V baseline. Noticeably, omniparser's performance even surpasses models that are specifically finetuned on GUI dataset including SeeClick, CogAgent and Fuyu by a large margin. 
We also note that incorporating the local semantics (omniparser w. LS in the table) further improves the overall performance. This corroborates with the finds in section <ref> that incorporating local semantics of the UI screenshot in text format, i.e. adding OCR text and descriptions of the icon bounding boxes further helps GPT-4V to accurately identify the correct element to operate on.  
Furthermore, our findings indicate that the interactable region detection (ID) model we finetuned improves overall accuracy by an additional 4.3% compared to using the raw Grounding DINO model. This underscores the importance of accurately detecting interactable elements for the success of UI tasks. Overall, the results demonstrate that the UI screen understanding capability of GPT-4V is significantly underestimated and can be greatly enhanced with more accurate interactable elements detection and the incorporation of functional local semantics.

  



 §.§ Evaluation on Mind2Web

In order to test how OmniParser is helpful to the web navigation secnario, We evaluate on <cit.> benchmark. There are 3 different categories of task in the test set: Cross-Domain, Cross-Website, and Cross-Tasks.

We used a cleaned version of Mind2Web tests set processed from the raw HTML dump which eliminates a small number of samples that has incorrect bounding boxes. In total we have 867, 167, 242 tasks in the test set from Cross-Domain, Cross-Website, and Cross-Tasks category respectively. During evaluation, we feed both the parsed screen results and the action history as text prompt, and SOM labeled screenshot to GPT-4V similar to the prompting strategy in <cit.>. Following the original paper, we perform offline evaluation focusing on the element accuracy, Operation F1 and step success rate averaged across the task.

In the first section of the table (row 1-3), We report numbers from a set of open source VL models as it appears in <cit.>. Here CogAgent and Qwen-VL are not finetuned on the Mind2Web training set. More detailed information about model settings can be found in the Appendix<ref>. 

In the second section of the table (row 4-9) we report numbers from Mind2web paper <cit.> and SeeAct <cit.>
paper. In this section, all of the approaches use the HTML elements selected by a finetuned element proposal model on Mind2Web training set which produces top 50 relevant elements on the HTML page based on the user task. Additionally, GPT-4V+SOM and GPT-4V+textual choices corresponds to the SeeAct with image annotation, and textual choices grounding methods respectively. In GPT-4V+SOM, the set of mark (SOM) boxes are selected from the element proposal model, and are labeled with the ground truth location extracted from HTML. In contrast, GPT-4V+textual uses DOM information of the selected relevant elements directly in the text prompt, rather than overlaying bounding boxes on top of screenshot. The better performance of textual choice corroborates with the experiment results in <ref>. 

In the last section (row 10-11), we report numbers from OmniParser. We observe GPT-4V augumented with local semantics of icon functionality and the finetuned interactable region detection model (w. LS + ID) performs better than the model with raw grounding DINO model (w. LS + GD) in all of the categories. 


Further, without using parsed HTML information, OmniParser is able to outperform GPT-4's performance that uses HTML in every sub-category by a large margin, suggesting the substantial benefit of the screen parsing results provided by OmniParser. 
Additionally, OmniParser outperforms the GPT-4V+SOM by a large margin. Compared to GPT-4V+textual choices, OmniParser significantly outperforms in Cross-Website and Cross-Domain category (+4.1% and +5.2%), while underperforming (-0.8%) slightly in the Cross-Task category, which indicates that OmniParser provides higher quality information compared ground truth element information from DOM and top-k relevant elemnt proposal used by the GPT-4V+textual choices set-up, and make the GPT-4V easier to make a accurate action prediction. 
Lastly, OmniParser with GPT-4V significantly outperform all the other trained models using only UI screenshot such as SeeClick and Qwen-VL.  








 §.§ Evaluation on AITW

In additional to evaluation on multi-step web browsing tasks, we assess OmniParser on the mobile navigating benchmark AITW <cit.>, which contains 30k instructions and 715k trajectories. We use the same train/test split as in <cit.> based on instructions, which retain only one trajectory for each instructions and no intersection between train and test. For a fair comparison, we only use their test split for evaluation and discard the train set as our method does not require finetuing.  

In table <ref>, we report the GPT-4V baseline in <cit.> paper, which corresponds to the best performing set up (GPT-4V ZS+history) that uses UI elements detected by IconNet <cit.> through set-of-marks prompting <cit.> for each screenshot at every step of the evaluation. The detected UI elements consist of either OCR-detected text or an icon class label, which is one of the 96 possible icon types identified by IconNet. Additionally, action history is also incorporated at each step's prompt as well.  We used the exact same prompt format in <cit.> except the results from the IconNet model is replaced with the output of the finetuned interactable region detection (ID) model.
Interestingly, we found that the ID model can generalize well to mobile screen. By replacing the IconNet with the interactable region detection (ID) model we finetuned on the collected webpages, and incorporating local semantics of icon functionality (LS), we find OmniParser delivers significantly improved performance across most sub-categories, and a 4.7% increase in the overall score compared to the best performing GPT-4V + history baseline. 










§ DISCUSSIONS


In this section, we discuss a couple of common failure cases of OmniParser with examples and potential approach to improve. 



Repeated Icons/Texts From analysis of the the GPT-4V's response log, we found that GPT-4V often fails to make the correct prediction when the results of the OmniParser contains multiple repeated icons/texts, which will lead to failure if the user task requires clicking on one of the buttons. This is illustrated by the figure <ref> (Left) in the Appendix. A potential solution to this is to add finer grain descriptions to the repeated elements in the UI screenshot, so that the GPT-4V is aware of the existence of repeated elements and take it into account when predicting next action. 



Corase Prediction of Bounding Boxes One common failure case of OmniParser is that it fails to detect the bounding boxes with correct granularity. In figure <ref> (Right), the task is to click on the text 'MORE'. The OCR module of OmniParser detects text bounding box 8 which encompass the desired text. But since it uses center of the box as predicted click point, it falls outside of the ground truth bounding box. This is essentially due to the fact that the OCR module we use does not have a notion of which text region are hyperlink and clickable. Hence we plan to train a model that combines OCR and interactable region detection into one module so that it can better detect the clickable text/hyperlinks. 


Icon Misinterpretation We found that in some cases the icon with similar shape can have different meanings depending on the UI screenshot. For example, in figure <ref>, the task is to find button related to 'More information', where the ground truth is to click the three dots icon in the upper right part of the screenshot.  OmniParser successfully detects all the relevant bounding boxes, but the icon description model interpret it as: "a loading or buffering indicator". We think this is due to the fact that the icon description model is only able to see each icon cropped from image, while not able to see the whole picture during both training and inference. So without knowing the full context of the image, a symbol of three dots can indeed mean loading buffer in other scenarios. A potential fix to this is to train an icon description model that is aware of the full context of the image. 






§ CONCLUSION

In this report, We propose OmniParser, a general vision only approach that parse UI screenshots into structured elements.  OmniParser encompasses two finetuned models: an icon detection model and a functional description models. To train them, we curated an interactable region detection dataset using popular webpages, and an icon functional description dataset. We demonstrate that with the parsed results, the performance of GPT-4V is greatly improved on ScreenSpot benchmarks. It achieves better performance compared to GPT-4V agent that uses HTML extracted information on Mind2Web, and outperforms GPT-4V augmented with specialized Android icon detection model on AITW benchmark. We hope OmniParser can serve as a general and easy-to-use tool that has the capability to parse general user screen across both PC and mobile platforms without any dependency on extra information such as HTML and view hierarchy in Android. 




§ ACKNOWLEDGEMENT

We would like to thank Corby Rosset and authors of ClueWeb22 for providing the seed urls for which we use to collect data to finetune the interactable region detection model. The data collection pipeline adapted AutoGen's multimodal websurfer code for extracting interatable elements in DOM, for which we thank Adam Fourney. We also thank Dillon DuPont for providing the processed version of mind2web benchmark.



alpha





§ APPENDIX



 §.§ Details of Icon-Description Dataset


In figure <ref>, we see that the original BLIP-2 model tend to focus on describing shapes and colors of app icons, while struggling to recognize the semantics of the icon. This motivates us to finetune this model on an icon description dataset. For the dataset, we use the result of parsed icon bounding boxes inferenced by the interactable icon detection model on the ScreenSpot dataset since it contains screenshots on both mobile and PC. For the description, we ask GPT-4o whether the object presented in the parsed bounding box is an app icon. If GPT-4o decides the image is an icon, it outputs one-sentence description of the icon about the potential functionality. And if not, GPT-4o will output 'this is not an icon', while still including this in the dataset. In the end, we collected 7185 icon-description pairs for finetuning. 

We finetune BLIP-2 model for 1 epoch on the generated dataset with constant learning rate of 1e^-5, no weight decay and Adam optimizer. We show a few of the qualitative examples of finetuned model vs the original model in figure <ref>. 





 §.§ Training details of Interactable Icon Region Detection Model

As introduced in <ref>, we train a YOLOv8 model on the interactable icon region detection dataset. We collect in total of 66990 samples where we split 95% (63641) for training, and 5% (3349) for validation. We train for 20 epochs with batch size of 256, learning rate of 1e^-3, and the Adam optimizer on 4 GPUs. We show the training curve in figure <ref>. 






 §.§ Details of SeeAssign Evaluation



  §.§.§ Prompt Used for GPT-4V

GPT-4V without local semantics: 

Here is a UI screenshot image with bounding boxes and corresponding labeled ID overlayed on top of it, your task is task. Which icon box label you should operate on? Give a brief analysis, then put your answer in the format of “`Box with label ID: [xx]“`

GPT-4V with local semantics: 

Here is a UI screenshot image with bounding boxes and corresponding labeled ID overlayed on top of it, and here is a list of icon/text box description: parsed_local_semantics. Your task is task. Which bounding box label you should operate on? Give a brief analysis, then put your answer in the format of “`Box with label ID: [xx]“`



 §.§ Details of Mind2Web Evaluation

Here we list more details of each baseline in table <ref>.


SeeClick, QWen-VL SeeClick is a finetuned version of Qwen-VL on the Mind2Web training set and we report both of their numbers in their paper. 

CogAgent CogAgent number is taken from the SEEAct paper <cit.>, where they report cogagent-chat-hf checkpoint that is not fine-tuned on Mind2Web for experiments. 

MindAct(Gen), MindAct, GPT-3.5-Turbo, GPT-4 The numbers for these baseline are taken from the Mind2Web <cit.> paper, where they use HTML information to augument the corresponding web agent.

GPT-4V+som This model corresponds to the image annotation grounding method in SeeAct paper, where the som boxes extracted from the selelcted HTML elements are provided to GPT-4V to make action prediction.

GPT-4V+textual choice This corresponds to the best performing scenario in SeeAct paper (except the Oracle), that uses the selected HTML elments information in a multi-choice question format as input to the GPT-4V agent. 



  §.§.§ Qualitative Examples

We list a few more examples to demonstrate local semantics of icon function description helps GPT-4V make better action prediction in figure <ref>.


























 Paper Title: Illustrating Classic Brazilian Books using a Text-To-Image Diffusion Model.


 

 









Mahlow et. al.: ILLUSTRATING CLASSIC BRAZILIAN BOOKS USING A TEXT-TO-IMAGE DIFFUSION MODEL.SKM: My IEEE article  

Illustrating Classic Brazilian Books using a Text-To-Image Diffusion Model
    Felipe Mahlow 0000-0001-9816-1440 , André Felipe Zanella 0000-0001-8828-6629 , William Alberto Cruz Castañeda 0000-0002-9803-1387 , 
        and Regilene Aparecida Sarzi-Ribeiro  0000-0001-6267-6549 
F.M and R.A.S.R are with São Paulo State University (Unesp), Bauru Campus. A.F.Z is with Maringá State University, Maringá Campus. W.A.C.C. is with Technologycal Federal University of Paraná, Guarapuava Campus.
Manuscript received xx xx, 202x; revised xx xx, 202x.
    August 5, 2024
====================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





§ ABSTRACT 
In recent years, Generative Artificial Intelligence (GenAI) has undergone a profound transformation in addressing intricate tasks involving diverse modalities such as textual, auditory, visual, and pictorial generation. Within this spectrum, text-to-image (TTI) models have emerged as a formidable approach to generating varied and aesthetically appealing compositions, spanning applications from artistic creation to realistic facial synthesis, and demonstrating significant advancements in computer vision, image processing, and multimodal tasks. The advent of Latent Diffusion Models (LDMs) signifies a paradigm shift in the domain of AI capabilities. This article delves into the feasibility of employing the Stable Diffusion LDM to illustrate literary works. For this exploration, seven classic Brazilian books have been selected as case studies. The objective is to ascertain the practicality of this endeavor and to evaluate the potential of Stable Diffusion in producing illustrations that augment and enrich the reader's experience. We will outline the beneficial aspects, such as the capacity to generate distinctive and contextually pertinent images, as well as the drawbacks, including any shortcomings in faithfully capturing the essence of intricate literary depictions. Through this study, we aim to provide a comprehensive assessment of the viability and efficacy of utilizing AI-generated illustrations in literary contexts, elucidating both the prospects and challenges encountered in this pioneering application of technology.




Image Generation, Diffusion Models, Illustration, Text-to-Image






§ INTRODUCTION



Generative Artificial Intelligence (GenAI) has revolutionized various tasks by integrating capabilities in text, audio, video, and image generation. GenAI excels in creating synthetic data that closely mimics real-world phenomena. For instance, text generation models such as OpenAI's GPT <cit.> have transformed the field of writing by demonstrating an exceptional understanding of context and coherence <cit.>. These models enhance natural language processing, content creation, and automated writing tasks <cit.>. In the realm of audio, models like Tacotron <cit.> and WaveNet <cit.> leverage deep neural networks to generate realistic speech and music, pushing the boundaries of audio synthesis <cit.>. Similarly, image generation has seen significant advancements with models such as DALL-E <cit.>, MidJourney <cit.>, and Stable Diffusion <cit.>, which can create intricate images from textual descriptions. Additionally, Generative Adversarial Networks (GANs) <cit.> have been pivotal in producing high-quality images for artistic endeavors and realistic face generation, significantly impacting computer vision and multi-modal tasks <cit.>. The ability of generative models to create human-like content has opened up new avenues for creative, automated, and innovative applications. However, these advancements also bring about concerns and challenges that need to be addressed in the future <cit.>. 

Text-to-image (TTI) models focus on developing methods and algorithms to create visual images from written text. A significant breakthrough in this field has been the rise of Latent Diffusion Models (LDMs) <cit.>, which build upon the foundational principles of Diffusion Probabilistic Models (DPMs) <cit.>. Diffusion models apply a series of random transformations progressively to an image's probability distribution <cit.>. This iterative process generates detailed and realistic images while providing greater control over the creation process.

The integration of GenAI into the creative process has shown a profound and multifaceted impact. For instance, research on the use of TTI models in craft education indicates that artificial intelligence (AI) can aid ideation and visualization but also raised concerns about skill gaps, lack of materiality, and ethical implications, including biases and the impact on creativity and copyright <cit.>. Similarly, GenAI can increase creativity in writing tasks, showing improved quality of text outputs with AI support, despite a greater uniformity in generated creations <cit.>. Additionally, the application of AI in independent publishing, reveals both the AI’s capability to enhance ideation and production, and the need for critical approaches to its role in preserving human craftsmanship <cit.>. AI use also has environmental implications, with research showing that text and illustration production with AI can generate significantly fewer carbon emissions compared to human methods <cit.>. However, the adoption of AI in creative fields is not without challenges, including ethical issues and the need for a careful balance between automation and human authorship. Analyzing these dynamics is crucial to understanding how AI can be a powerful tool in enhancing the creative process while addressing the challenges associated with its integration.

The extant literature offers a scant examination of the intersection between AI and book illustration. Traditionally, the illustration of literary works necessitated the intervention of human artists, a process often characterized by its time-consuming nature and subjectivity. However, recent advancements in AI present opportunities for automation, potentially enhancing this process by generating illustrations that encapsulate the essence and historical context of literary works with significantly reduced resources and time compared to traditional methodologies. An example of this is the GenAI methodology based on TTI for producing assisted art that allows dataset assembling, model training and fine-tuning, and content enhancement and post-processing within a historical or cultural setting <cit.>. Nevertheless, the endeavor of producing visually compelling and faithful images based on literary descriptions remains inherently complex. The quality and fidelity of these generated images are critically contingent upon the specificity of prompts and the efficacy of the models employed. Furthermore, challenges persist regarding data bias and the inherent limitations of AI models themselves. The training of these models is particularly contentious, as it often involves using copyrighted images without the explicit consent of the original artists <cit.>.

Our research explores the application of LDMs for book illustration, utilizing the Stable Diffusion model to generate visual representations based on textual prompts derived from seven distinct works of classic Brazilian literature. We employed a two-phase methodology that begins with initial image generation using Stable Diffusion XL Base 1.0, followed by a refinement process with Stable Diffusion XL Refiner 1.0. This iterative approach was designed to enhance the quality and fidelity of the generated illustrations, aligning them more closely with the literary descriptions. The aim of this work was not to achieve "perfect" illustrations but to demonstrate the feasibility of such an approach, providing a methodological framework and documenting the positive and negative aspects encountered, as well as the ethical biases inherent in GenAI.

Our findings underscore the critical importance of prompt specificity in generating high-quality visual content. By carefully designing concise prompts that capture the essence and context of the scenes described in the literature, we achieved accurate and engaging illustrations. However, our study also highlights challenges such as model biases, notably the tendency to produce predominantly white figures, as observed in the illustrations of characters from books like Senhora and O Triste Fim de Policarpo Quaresma. Quantitative evaluations using Contrastive Language-Image Pre-training (CLIP) and Inception Scores (IS) revealed varying success rates across different books, with better results achieved for works like Horto and O Triste Fim de Policarpo Quaresma, reflecting successful visualizations of the described concepts. These results offer valuable insights into the potential and limitations of using generative models for literary illustration and suggest areas for future improvements in prompt engineering and model training.

The paper is structured as follows: Section <ref> outlines the methodology employed, starting with <ref>, where we present the books, followed by <ref>, where we discuss the hardware specifications and configurations used for training and inference. In <ref>, we provide a detailed analysis of the image generation process, while in <ref> we address the techniques applied for refining these images. Further, in <ref>, we detail the large-scale generation process and the subsequent selection of the most relevant images. Section <ref> is dedicated to the quantitative evaluation of the results, focusing on the CLIP Score and the IS, discussed in subsections <ref> and <ref>, respectively. Section <ref> presents an extensive exploration of the experimental results obtained, with a qualitative approach (<ref>) illustrating the effectiveness and adaptability of our approach in generating diverse and visually appealing compositions in various contexts, as well as a quantitative approach (<ref>) assessing the performance of the generated images based on established metrics. Finally, Section <ref> synthesizes the main findings, highlights the effectiveness of the image synthesis pipeline, and discusses potential directions for future research and improvements to the proposed method.



§ METHODOLOGY



This section describes the systematic procedure adopted for generating and refining illustrations for seven classical Brazilian books using the Stable Diffusion model. 



 §.§ The Books



The selection of these texts was guided by the goal of respecting copyright constraints while providing a rich source for creative exercises. Therefore, only texts from Brazilian literature available in the public domain were chosen.

The selected books for this study are:


    
  * Senhora (1875) by José de Alencar
    
  * O Cortiço (1890) by Aluísio Azevedo
    
  * A Viúva Simões (1897) by Júlia Lopes de Almeida
    
  * Dom Casmurro (1899) by Machado de Assis
    
  * Horto (1900) by Auta de Souza
    
  * Os Sertões (1902) by Euclides da Cunha
    
  * O Triste Fim de Policarpo Quaresma (1915) by Lima Barreto


These works were chosen for their literary significance and their rich descriptive passages, which provide ample material for visualization through GenAI. Each text offers unique narratives and vivid descriptions that facilitate the generation of diverse and engaging illustrations. Senhora and O Cortiço, for example, are notable for their detailed portrayal of 19th-century Brazilian society, while Dom Casmurro and Os Sertões are renowned for their deep psychological and socio-political insights. A Viúva Simões and Horto contribute with their unique thematic and stylistic elements, and O Triste Fim de Policarpo Quaresma is recognized for its satirical and cultural critique. The choice of these works not only ensures a diverse range of illustrative challenges but also honors the literary heritage of Brazilian authors through modern technological means.



 §.§ Hardware Configuration



All computations were performed on a system equipped with an NVIDIA GeForce RTX 3090 GPU. This setup provided enough computational power and memory capacity, essential for handling the process involved in generating and refining the images.



 §.§ Image Generation



In the first stage, the Stable Diffusion XL Base 1.0[https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0] <cit.>  model was employed to generate the initial images. The model was fed with a comprehensive list of descriptive prompts, each crafted to depict specific scenes from the selected books. These prompts included detailed descriptions of characters, actions, and settings. For each prompt, the model performed the image generation in 40 inference steps. The inference process involved a denoising technique, crucial in diffusion models, which iteratively refined an image from an initial noisy state to a clear and detailed representation. In this case, the inference was configured to terminate at 0.8 of the denoising process, meaning that the image was generated to a certain level of detail before transitioning to the next stage.



 §.§ Image Refinement



In the second stage, the Stable Diffusion XL Refiner 1.0[https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0] model was used to enhance the images generated in the previous step. This refinement model received the latent images (partially processed images) and continued the denoising process from 0.8, completing the remaining steps to 1.0. This additional 40-step inference phase ensured that the final images reached a higher level of quality and detail. The refiner model utilized additional components, such as a second text encoder and the variational autoencoder (VAE) from the base model, to improve the accuracy and aesthetics of the images. The refiner corrected imperfections and added fine details, making the illustrations more vivid and true to the original descriptions.



 §.§ Large-Scale Generation and Selection



For each book, five prompts were generated, and for each prompt, 300 images, resulting in 1500 images per book. This large-scale generation ensured a wide variety of illustrations, allowing for a rigorous selection of the best representations for the project. These large quantities are important so that the calculation of quantitative metrics can be done more accurately. 



 §.§ Quantitative Evaluation Metrics



To evaluate the quality of the images generated we employed two evaluation metrics, the CLIP <cit.> and IS scores <cit.>.



  §.§.§ CLIP Score



The CLIP Score is a metric that evaluates the semantic quality and relevance of the generated images concerning the input prompts. We utilize the "clipq_score" function from the "torchmetrics.functional.multimodal"[https://lightning.ai/docs/torchmetrics/stable/multimodal/clip_score.html] library, which calculates the CLIP score of an image concerning a text prompt. The CLIP model was trained on a large number of text-image pairs to learn a joint representation of text and image. It is capable of assessing the semantic quality of the generated images concerning the provided prompts.



  §.§.§ Inception Score (IS)



IS is a metric that evaluates both the quality and diversity of the generated images. A higher IS indicates that the generated images are both diverse (capturing multiple classes) and of high quality (clearly recognizable by the classifier). The metrics were calculated through implementations based on the seminal paper[https://github.com/openai/improved-gan][https://github.com/w86763777/pytorch-image-generation-metrics/blob/master/pytorch_image_generation_metrics/inception.py]<cit.>.

By employing these metrics, we ensure a comprehensive evaluation of the generated images, considering both their semantic relevance to the prompts and their overall quality and diversity.



§ RESULTS



In this section, we present and analyze examples of the generated images, focusing on specific examples that highlight the process and outcomes of our methodology. Figures <ref> and <ref> showcase selected images, which facilitate a discussion on the creation process and the critical role of the prompt in generating visually captivating and contextually relevant illustrations.



While the AI system autonomously manages the image generation process, the formulation of the prompt remains a critical task for the user. Mere transcription of text from the book typically results in suboptimal outcomes. It is imperative to conceptualize the scene based on the textual descriptions provided in the book and devise a prompt that facilitates the creation of an image that is both visually compelling and faithful to the scene's intrinsic characteristics.



 §.§ Qualitative Approach



Figure <ref>a) exemplifies the representation of Capitu's eyes, often described as "olhos de ressaca" (sea surge eyes) in Machado de Assis's Dom Casmurro. These eyes are characterized as "gypsy eyes, oblique and dissimulated," as mentioned in chapters 13 and 32 of the book. The prompts used for this image were inspired by the physical and symbolic traits detailed in these chapters. Capitu is described as having brunette hair and light eyes, considering that the true color of her eyes is not explicitly revealed in the book. The prompt used was: 


"Painting of oblique and concealed eyes. Just eyes. Mysterious and energetic fluid, like the wave that retreats from the beach, on hangover days. Brunette, clear and large eyes, straight and long nose, had a thin mouth and wide chin."


Figure <ref>b) clearly illustrates the importance of imagining a way to represent a scene. This image was created to depict Bentinho's suspicion that his son was fathered by Escobar, as described in chapters 131 and 132 of Dom Casmurro, portraying Bentinho's obsession with Escobar's photograph, which he kept in his office. Bentinho frequently noted the resemblance between his son and Escobar, fueling his suspicions. The image positions the child in front of Escobar's photograph, highlighting the similarities between them and symbolizing Bentinho's growing distrust. The prompt used was:


"A painting of a young kid (4yo) in the center. In the background, there is a photograph of his father on the wall. The kid looks like the father, who was 40yo. 1800s."


Figure <ref>c) portrays the opulence of the character Ernestina, based on Júlia Lopes de Almeida's novel A Viúva Simões. The image captures the essence of a rich widow, embodying her social status and physical attributes as described in chapter I of the book. The prompt used for this illustration was:


"Concept art of a rich woman, widow, forty years old, bourgeoisie, a beautiful woman, tall, slender, beautiful black eyes, dark skin that was delicately feathery and soft, Rio de Janeiro. Digital artwork, illustrative, painterly, matte painting, highly detailed."


Figure <ref>d) depicts Aurélia at the window with her suitors, as described in part 2, chapter 4 of José de Alencar's novel Senhora. Despite the prompt not specifying skin color, all 300 images generated for "a young woman, very beautiful" depicted white women, indicating a bias in the algorithm. The image captures the essence of Aurélia's character as she stands at the window, attracting numerous suitors. The prompt used for this illustration was:


"Concept art of a young woman, very beautiful, stands in front of the window. She attracts a crowd of suitors who pass by in carriages and on foot. The eager looks and insinuating words of the suitors contrast with her cold impassivity; she remains at the window like a statue, fulfilling her duty, but without employing flirtatious tricks or seduction tactics."


Figure <ref>e) illustrates Seixas's room, highlighting the stark contrast between his luxurious lifestyle and the poverty of his family, as described in part 1, chapter 5 of José de Alencar's novel Senhora. The scene and specifically the room was chosen for illustration due to the detailed descriptions in the book, which reveal aspects of Seixas's personality. This disparity is further emphasized as Seixas changes marriages twice, driven by his desire to escape his circumstances and secure the dowry offered to him. The generated images, however, did not fully capture all the objects mentioned in the prompt, seemingly focusing primarily on the books. In this sense we emphasize that, to achieve optimal results, it is generally advisable to employ shorter and more succinct prompts that contain fewer elements within the image. The prompt used for this illustration was:


"Concept art of modest, worn-out study with faded blue wallpaper, old furniture. Iron bed with green mosquito net contrasts with surroundings. Luxurious items like a tailored black coat, elegant evening wear, a Parisian hat, quality gloves, and fine boots seem out of place. The embroidered blue satin pillow stands out. The disorderly alcove with books, inkwells, ashtrays, and assorted trinkets contrasts with the well-appointed dresser counter. Corner with umbrellas, canes, some valuable, alongside artistic curiosities."




Figure <ref>a) portrays a man from the Brazilian backlands, reflecting his determined nature and the harsh life he endures, as described in part 2, chapter III of Euclides da Cunha's Os Sertões. The prompt creation was based on: "The sertanejo is, above all, a strong man. He does not have the exhaustive rickets of the neurasthenic mestizos of the coast. His appearance, however, at first glance, reveals the opposite. He lacks the impeccable physique, the poise, the correct structure of athletic organizations.". Although the model faithfully represents the figure of the sertanejo, all 300 generated images show dense vegetation in the background, not reflecting the true scenery of the Brazilian backlands. The prompt used for this illustration was:


"A painting of a man from the Brazilian backlands. He is a determined person, yet his countenance reflects the harsh life in the backlands, 19th century, realism."


Figure <ref>b) illustrates a scene inspired by Auta de Souza's poem "À minha avó" from her book Horto. The generated image captures the warmth and comfort of the relationship between the grandmother and her grandson. The cozy environment, enhanced by elements such as a soft blanket, a cup of tea, and an old book, embodies the essence of comfort and security. However, it is important to note that the model also exhibited a bias towards generating white individuals, which is evident in the depiction of the characters. This particular book posed a challenge due to its abstract nature. The prompt used for the illustration was:


"A painting of a grandmother and her grandson (9yr) sitting together in a cozy environment, surrounded by elements that symbolize comfort and security, a soft blanket, a cup of tea, and an old book, 19th century, realism."


Figure <ref>c) illustrates the character Luciano, described in chapters I and II, from Júlia Lopes de Almeida's A Viúva Simões. The generated image portrays a man with gray hair, a thick build, consistent with the description provided. His expressive and friendly face, marked by dark circles under his eyes, captures the intended character traits. This image serves as a clear example of the model's tendency to create more stylized, cartoon-like illustrations when guided by prompts that include "concept art." The prompt used for this illustration was:


"Concept art of a man, gray hair, thick, virile physiognomy, not slender, rounded belly, expressive and friendly face, dark circles under his eyes, Rio de Janeiro. Digital artwork, illustrative, painterly, matte painting, highly detailed."


Figure <ref>d) depicts a scene from Aluísio de Azevedo's novel O Cortiço chapter XXI. This depiction highlights the model's ability to capture the specific details and atmosphere described in the prompt. The highly detailed, painterly style aligns with the intent to represent the character João Romão, a Portuguese owner of the collective housing, in a 19th-century Rio de Janeiro setting, showcasing the blend of period-appropriate elements with the everyday life of the character. The prompt used for this illustration was:


"Concept art of a man walking back in flip-flops and a nightgown in the bedroom, wide room lined in blue and white with little yellow flowers pretending to be gold, a rug at the foot of the bed, and on the bench a nickel alarm clock, Rio de Janeiro, 19th century. Digital artwork, illustrative, painterly, matte painting, highly detailed."


Figure 2e) presents the character Policarpo Quaresma from Lima Barreto's O Triste Fim de Policarpo Quaresma. The description comes from Chapter I and the image portrays a civil servant from the 19th century who is depicted as deeply valuing his country's culture. The illustration captures the essence of the character’s dedication and connection to the cultural heritage of Rio de Janeiro during that period. The prompt used for this illustration was:


"Concept art of a man, civil servant, values the country's culture, 19th century, Rio de Janeiro. Digital artwork, illustrative, painterly, matte painting, highly detailed."


Broadly speaking, from an ethical standpoint, three primary issues highlighted in <cit.> are also immediately discernible in our work: misrepresentation (e.g., the harmful stereotyping of minority groups), underrepresentation (e.g., the eradication of the presence of a particular gender within specific professional roles), and overrepresentation (e.g., the predominance of Anglocentric viewpoints) <cit.>.



 §.§ Quantitative Approach



In this section, we present a quantitative analysis of the generated images using two evaluation metrics: CLIP and IS scores. These metrics provide insights into the semantic relevance and overall quality of the images produced for various literary concepts. Table <ref> summarizes the evaluation results for each book concept. 



It should be observed that, despite the limited presentation of images — restricted to no more than two distinct prompts per book in Figures <ref> and <ref> — each book is characterized by five distinct prompts with 300 images each. The aforementioned metrics are thus computed based on an aggregate of 1500 generated images. The results indicate varying levels of performance across different book concepts. For the CLIP Score, Horto by Auta de Souza achieved the highest score of 21.05 (standard deviation 2.30), reflecting strong semantic alignment between the generated images and the text prompts. This is followed by A Viúva Simões with a CLIP Score of 20.44 (standard deviation 1.63) and Senhora by José de Alencar with a score of 20.28 (standard deviation 1.33). On the lower end, Dom Casmurro scored 17.96 (standard deviation 1.79), indicating comparatively weaker semantic relevance.

In terms of the IS, which assesses the image quality and diversity, O Triste Fim de Policarpo Quaresma achieved the highest score of 7.87 (standard deviation 0.36), suggesting superior image quality and variability. Horto follows with a score of 6.35 (standard deviation 0.34), indicating high-quality results but with less diversity compared to the previous. Conversely, O Cortiço and Senhora exhibited lower scores, 3.76 (standard deviation 0.23) and 3.86 (standard deviation 0.23) respectively, pointing to challenges in achieving high image quality and diversity.

Overall, these quantitative results highlight differences in the effectiveness of the generative models across various literary texts, reflecting the impact of both semantic alignment and image quality on the final outputs.



§ CONCLUSIONS



In this study, we explored the application of LDMs for the task of book illustration, utilizing the Stable Diffusion model to generate images based on prompts derived from seven classical Brazilian literary works. Our results indicate that the effectiveness of image generation is significantly influenced by the quality and specificity of the prompts provided. Prompts that were carefully crafted to encapsulate the essence of the scenes described in the books yielded more compelling and relevant visual results. Conversely, generic or poorly articulated prompts often led to suboptimal outcomes, highlighting the importance of prompt design in the generative process.

Through the illustrative examples provided, including figures representing characters and scenes from Dom Casmurro, A Viúva Simões, Senhora, O Cortiço, O Triste Fim de Policarpo Quaresma, and Os Sertões, we observed that the Stable Diffusion model could effectively capture and convey the thematic and visual elements of the source material. However, certain limitations were noted. For instance, the model's tendency to produce predominantly white individuals, despite the diversity described in the books, points to inherent biases in the training data that affect the generated results. This is exemplified by the generated images of characters from Senhora and O Triste Fim de Policarpo Quaresma. 

Quantitative evaluations using CLIP and IS scores revealed variations in the quality of the generated images across different concepts. The analysis demonstrated that images aligned with literary descriptions from Horto and O Triste Fim de Policarpo Quaresma scored high in both metrics, indicating successful visualizations of these concepts.

In conclusion, while the Stable Diffusion model is a powerful tool for generating illustrative content from textual prompts, its efficacy relies on precise prompt formulation and awareness of model biases. Future models should address these issues by incorporating more diverse datasets. AI research in illustration should also focus on advanced prompt engineering to enhance the quality and inclusivity of generated images. This study adds to the growing body of research on using generative models in creative contexts, offering insights into their potential and limitations in literary illustration.



§ ACKNOWLEDGMENTS

F. M. acknowledges support from Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), project number 88887.607339/2021-00. A. F. Z. acknowledges support from Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), project number 140935/2024-0.


IEEEtran  





[
    < g r a p h i c s >
]1st author
received a teaching degree in Physics and a bachelor's degree in Materials Physics from São Paulo State University (Unesp), in 2020 and 2023, respectively. He is currently pursuing a Ph.D. degree in Computer Science at São Paulo State University (Unesp), with a focus on classical and quantum machine learning and their applications to Quantum Information Science. His research encompasses the use of Machine Learning and Generative Artificial Intelligence, as well as Quantum Computing.




[

    < g r a p h i c s >

]
2nd author
Holds a Bachelor's degree in Mathematics (2022) and a Master's degree in Computer Science (2022), both from the State University of Maringá. Currently, he is pursuing a Ph.D. in Computer Science at the same institution (2024). His expertise lies in Computer Science, focusing on Computer Systems and Machine Learning. He is interested in research in optimization, machine learning, and TTI diffusion models.




[
    < g r a p h i c s >
]
3rd author
Researcher in AI holds a Bachelor's degree in Computer Science from Benemérita Universidad Autónoma de Puebla and a Bachelor's degree in Computer Engineering from the Federal University of Rio Grande do Sul. He completed his Master's and Ph.D. in Electrical Engineering with a focus on Biomedical Engineering at the Federal University of Santa Catarina. He holds a Postdoctoral degree in AI and Biomedical Engineering at the State University of Santa Catarina. Currently, he is a professor at the Federal Technological University of Paraná, Guarapuava Campus. His research interests include ubiquitous computing, machine learning, and GenAI.





[
    < g r a p h i c s >
]
4th author
Holds Postdoctoral degrees in Poetics and Cultures in Digital Humanities at UFG (2022) and Arts at UNESP/SP (2013). She is part of the Chair of Design, Art, and Science at Media Lab BR and coordinates the Media Lab UNESP. Research member of Red de Investigación de la Imagen at the University of Málaga, Spain. Holds a Ph.D. in Communication and Semiotics from PUC/SP (2012) and a Master's in Arts from UNESP/SP (2007). Graduated in Art Education with a specialization in Visual Arts from FAAC - UNESP, Bauru/SP (1994). Currently, faculty member at UNESP/Bauru, coordinates the Graduate Program in Media and Technology, and leads the labIMAGEM Research Group.




