Paper Title: Prompting Medical Large Vision-Language Models to Diagnose Pathologies by Visual Question Answering.



Einstein Probe discovery of a super-soft outburst from CXOU J005245.0-722844:
 a rare BeWD binary in the Small Magellanic Cloud
    [
    
===============================================================================================================================



§ ABSTRACT 
Large Vision-Language Models (LVLMs) have achieved significant success in recent years, and they have been extended to the medical domain. Although demonstrating satisfactory performance on medical Visual Question Answering (VQA) tasks, Medical LVLMs (MLVLMs) suffer from the hallucination problem, which makes them fail to diagnose complex pathologies. Moreover, they readily fail to learn minority pathologies due to imbalanced training data. We propose two prompting strategies for MLVLMs that reduce hallucination and improve VQA performance. In the first strategy, we provide a detailed explanation of the queried pathology. In the second strategy, we fine-tune a cheap, weak learner to achieve high performance on a specific metric, and textually provide its judgment to the MLVLM. Tested on the MIMIC-CXR-JPG and Chexpert datasets, our methods significantly improve the diagnostic F1 score, with the highest increase being 0.27. We also demonstrate that our prompting strategies can be extended to general LVLM domains. Based on POPE metrics, it effectively suppresses the false negative predictions of existing LVLMs and improves Recall by approximately 0.07.



    Medical Visual Question Answering, Large Vision-Language Models, Prompt Engineering




§ INTRODUCTION


Research on Large Language Models (LLMs) has yielded astonishing results in recent years. LLMs with billions of parameters have achieved outstanding abilities in a wide range of application scenarios <cit.>. The success of LLMs has quickly extended into the Vision-Language (VL) domain. Large Vision-Language Models (LVLMs) are built upon LLMs by training adapters that project visual features into features that can be interpreted by LLMs <cit.>. Visual Question Answering (VQA) is an essential skill of LVLMs, and VQA accuracy serves as a test metric for most of these models <cit.>. LVLMs have been pretrained on medical datasets <cit.> and they have been tested on medical VQA tasks <cit.>. These Medical LVLMs (MLVLMs) have been able to answer questions regarding the imaging modalities, organs, and abnormalities depicted by the input medical scans.

Unfortunately, however, “hallucination” has been a major problem for LVLMs. This refers to the generation of content that is contradictory to the input images. Hallucination can be measured via VQA. One may ask the model questions regarding the existence of objects in the input image(s) and the hallucination level is assessed as the percentage of correctly answered questions. VQA can also potentially serve for medical image diagnosis. Users pose questions regarding a pathology and the MLVLM responds based on its analysis of the medical scans. However, most of the available datasets involve simple questions such as “what is the modality of this image” and “what is the organ/tissue in this image”. MLVLMs have yet to be thoroughly evaluated on VQA accuracy across a broad spectrum of pathologies. Additionally, general VQA models are usually tested by the commonly known accuracy: the percentage of correctly answered questions, which is an unsuitable measure for medical VQA.
Medical image classification metrics such as the Precision, Recall, and F1 are more suitable for the evaluation of medical VQA models. Several strategies have been explored to enhance the question answering of LLMs/LVLMs, including chain-of-thought prompting <cit.>, self-consistency <cit.>, and retrieval-based augmentation <cit.>. All these methods involve fine-tuning the models, which is expensive. Training-free methods to improve the VQA accuracy of MLVLMs are desirable.

For MLVLMs, hallucination is exacerbated by imbalanced training data. Many pathologies are minority categories in medical datasets. Models trained on large-scale medical data may easily fail to learn the features of less common pathologies. Addressing data bias typically involves strategies such as including more data of better quality, but given the scarcity of medical data, significantly enlarging the dataset may not be feasible. Common remediations involve re-sampling the data such that the positive and negative cases are better balanced, but this poses challenges when the data involves multiple categories of pathology. Additionally, re-sampling may undermine the training needs of LVLMs, which generally require large quantities of data. These problems highlight the need of a cost-effective approach to navigate the problem of minority categories in datasets.

Our study focuses on the VQA abilities of MLVLMs. In particular, we test an existing MLVLM, LLaVA-Med <cit.> for chest X-ray VQA across 5 categories of pathologies. The results show that the model has low accuracy, especially on minority pathologies. To enhance its VQA accuracy, we propose two prompting strategies. The first involves enriching prompts with detailed explanations of the queried pathology. The explanations include how the queried pathology is defined and how it appears in images. Our second strategy involves introducing an auxiliary weak-learner model as another agent. We train a small image classifier and fine-tune it to identify negative images accurately. Then, the negative predictions of this classifier are appended to the prompt as a reference for the MLVLM.

We run our experiments on the MIMIC-CXR-JPG <cit.> and Chexpert <cit.> datasets. The results show that our prompt strategies improve the F1 score significantly in most pathology categories (highest +0.27). We also show that our weak-learner-prompting strategy is applicable to the general domain. It reduces the false negative predictions of general domain LVLMs and improves the Recall by around 0.07 according to POPE metrics <cit.>.

To summarize, our contributions include the following:

    
  * We improve the VQA accuracy of MLVLMs by prompting with detailed explanations of pathologies.
    
  * We introduce a low-cost weak learner model as a reference for LLaVA-Med, and this effectively reduces the false positive (FP) answers.
    
  * We show that our second prompting strategy can be extended to general domains to help models adapt to specialized accuracy needs.

<ref> reviews related work, <ref> describes our methodology, <ref> presents our empirical study and its results, and <ref> draws conclusions from our research.



§ RELATED WORK





  
LVLMs and VQA

LVLMs are built upon LLMs. A pretrained visual encoder extracts the visual features and an adapter module projects the extracted features to ones that can be understood by the LLM. Models of this type include those by <cit.>, <cit.>, and <cit.>. During training, the visual encoder and the LLM are usually fixed. VQA is an essential skill of LVLMs. Given an input image, the models should be able to answer questions correctly regarding that image.



  
Hallucination in LVLM VQA

The hallucination problem usually refers to the LVLM generating a response that is not consistent with the input image. For VQA, in their generated answers the models may make mistakes on object presence, location, attributes, or the mutual relationship between objects. <cit.> find that frequently occurring objects are easily hallucinated by LVLMs, in that they tend to mention such objects even if it they are absent in the image. <cit.> and <cit.> show that LVLMs sometimes presume the assumptions in questions are true and easily give wrong answers when asked about some objects not in the given image.



  
Causes of LVLM VQA Hallucination

Hallucination can result from bias in the training data, missing fine-grained visual features, and LLM decoding strategies <cit.>. For data bias, the imbalanced distribution of data is an important aspect. When most of the answers to a question in the training data are “Yes”, the model tends to answer “Yes” to that question. Missing fine-grained visual features usually result from the pretraining of the visual encoder. Most LVLMs use the visual encoder of CLIP trained through contrastive learning. The encoder mainly focuses on salient features and ignores fine-grained features <cit.>. LVLM decoding strategies mostly choose the next word as the one having maximum conditional probability given previous text and the input image. This can lead to hallucination when the model overly relies on the knowledge learned in its training texts. Other causes include model simplicity and insufficient attention <cit.>.



  
Mitigation of LVLM VQA Hallucination

Strategies to mitigate hallucination in LVLMs mainly fall into two categories: prompt engineering and model improvement. Regarding the former, <cit.> leverage visual instructions constructed from the bounding box information in the input image to prompt LLMs. <cit.> use a chain of thought scheme to prompt the models to perform step-by-step visual-language reasoning like humans, which eventually leads to the correct answers. <cit.> generate multiple chains of thought and use the one with the majority vote as the answer. <cit.> prompt the model with explanations of the terms in questions. With regard to the model improvement strategy for reducing hallucination, <cit.> improve the visual and text feature alignment through reinforcement learning. <cit.> propose a contrastive decoding strategy to reduce reliance on pretrained knowledge. <cit.> and <cit.> also focus on the inference stage and propose specialized decoding strategies to mitigate hallucination. Other strategies for reducing hallucination have been proposed. For example, <cit.> design a post-processing model to detect hallucinated objects and rephrase the generated answers, and <cit.> adapt a reinforcement learning strategy that uses human evaluation of the hallucination level to improve the model.



  
Assessment of LVLM Hallucination

There are two approaches to assessing hallucination in LVLMs. The first is VQA. The ground truth information of the input images is leveraged to construct questions regarding the existence of objects in the images (e.g., “Is there a black cat in the image?”), as well as questions about objects which do not exist in the images. The models are evaluated in terms of the percentage of correctly answered questions. Metrics of this type include POPE <cit.>, CIEM <cit.>, and NOPE <cit.>. The second  approach is to use pre-designed prompts from which the models produce various generations that are then evaluated. Examples include CHAIR <cit.>, which counts the hallucinated objects in generated image captions, and MMHAL-BENCH <cit.>, which uses GPT-4 <cit.> to compare the generations with human answers and determine the propensity toward hallucination.



  
VQA in MLVLMs

For MLVLMs, given a medical scan, models such as LLaVA-Med <cit.> and Med-PALM <cit.> are able to answer questions regarding the types of modalities, the scanned organs, and medical indicators such as opacity. They have demonstrated good performance on medical VQA datasets such as VQA-RAD <cit.>, SLAKE <cit.>, and Path-VQA <cit.>. However, most medical questions in existing datasets are simple. MLVLMs have not been tested on a broader range of complex pathologies.





§ METHODOLOGY



<ref> illustrates the structure of common LVLMs. They are based on a pretrained unimodal LLM such as Llama <cit.> and Vicuna <cit.>. A pretrained visual encoder, such as ViT <cit.> or conventional CNNs, is applied to extract image features that are projected to the text feature space by an adapter. The projected visual features are concatenated with the text prompt embeddings and fed to the LLM. The adapter usually consists of several linear layers with non-linear activations. The visual encoder and the LLM are usually frozen during training.

In our work, we choose the pretrained LLaVA-Med <cit.> as our model, which is a MLVLM built upon LLaVA <cit.>. The model structure resembles <ref>. It uses pretrained Vicuna <cit.> as the LLM and the pretrained ViT encoder from CLIP <cit.> as the visual encoder. The adapter is simply a trainable projection matrix. Both the visual encoder and LLM weights are frozen during training. LLaVA-Med fine-tunes LLaVA in two steps. First, it fine-tunes LLaVA to generate medical reports from input medical images. Second, it uses GPT-4 to generate various questions from the ground truth reports and fine-tunes the model to perform question answering. 

Most MLVLMs are currently trained by medical VQA such that medical diagnosis can be performed by asking questions related to various pathologies; e.g., “Does this image have lung lesion?”. To reduce model hallucination and improve VQA accuracy, we propose two prompting strategies at the inference stage: (1) providing the model with detailed explanations about the queried pathologies and (2) asking the model to consider the inferences of a weak learner.



 §.§ Prompting With Detailed Explanations


Given imbalanced training data, MLVLMs might not adequately be able to learn the features of the minority pathologies. To compensate for insufficient training, we provide a detailed explanation of the queried pathology as a prompt at the inference stage. The explanation briefly defines the pathology and lists several key findings in medical images that may indicate its existence. An example is shown in <ref>. The model is informed that Pulmonary Edema is defined as the accumulation of fluid in the lungs. Then several chest X-ray findings that may suggest its existence are provided. The model can determine if the given image has Pulmonary Edema by linking the given findings with the image features.

Prompt templates for a number of pathologies are listed in <ref>. 







 §.§ Prompting With Detailed Explanations and Weak Learners


Data re-sampling is a commonly-used strategy to deal with imbalanced datasets that are responsible for the tendency of traditional image classification models to return negative predictions for minority pathologies. Models trained on re-sampled datasets often exhibit improvements in Precision and Recall scores; however, this strategy may not be suitable to MLVLMs for two reasons. First, it is difficult to balance a dataset containing many categories of pathologies. Second, MLVLMs usually demand much larger datasets and fine-tuning is also expensive.

One can nevertheless enable MLVLMs to benefit by leveraging small models trained on re-sampled datasets. Our method resembles multiagent LLM systems, such as <cit.>, where multiple LLMs debate each other and hallucination can be corrected by referring to the generated outputs of other models. Given that traditional image classifiers are smaller, it is feasible to train multiple small classifiers each of which is trained on re-sampled datasets of a particular pathology. Those models can be further fine-tuned to optimize a single aspect, such as fewer false positives (FP) or fewer false negatives (FN). The classifiers are applied to the medical images and return preliminary predictions. These predictions are selectively included in the prompts as references for the MLVLM. Hence, MLVLMs can benefit indirectly from the nuanced understanding that these specialized models can provide. This method is meaningful because clinicians usually must balance the trade-off between overtreatment and undertreatment when making healthcare decisions. For instance, they may prefer models having a low FP rate if the cost of overtreatment is higher than that of undertreatment. 

 An example is shown in <ref>, which queries about the presence of Edema. We first provide the model with the detailed explanation of Edema. Then, we use the weak learner to suppress the FPs. The image is input to an Edema classifier that has been fine-tuned on a balanced dataset for high sensitivity and high true negative (TN) rate. If its prediction is negative, we append after the pathology explanation the prompt “For this image, another agent thinks the probability of Edema is 0.1”. Instead of using the actual predicted probability, the probability value is manually chosen because the decision threshold has been fine-tuned and is no longer 0.5. We do not use a zero probability value because we do not want the model overly to trust the weak learner. Although in this example our goal is only to reduce FPs, our strategy can also be applied to reduce FNs, simply by fine-tuning the classifier for a high true positive (TP) rate and applying the prompt in the case of positive predictions.




§ EMPIRICAL STUDY





 §.§ Datasets




LLaVA-Med is pretrained on the PMC-15M dataset <cit.>, which contains image-text pairs of multiple modalities; e.g., CT, MRI, X-ray, etc. In the first stage, 467,710 image-report pairs were selected for training. In the second stage, 56,708 question-answer pairs were created from the data of the first stage to fine-tune the model. <ref> shows the count of reports in the LLaVA-Med training data (second stage) that mention one of the five test pathologies as positive. Relative to the total amount of data, all five categories are minorities.



To assess the zero-shot performance of the MLVLM, we used the MIMIC-CXR-JPG <cit.> and Chexpert <cit.> chest X-ray test sets. They include 5,159 and 668 images, respectively. 
Neither dataset overlaps with PMC-15M.

MIMIC-CXR-JPG includes images and medical reports covering 13 categories of findings: Atelectasis, Cardiomegaly, Consolidation, Edema, 
Enlarged Cardiomediastinum, Fracture, Lung Lesion, Lung Opacity, Pleural Effusion, Pneumonia, Pneumothorax, Pleural Other, and Support Devices. The raw reports are parsed and rough image-level tags are automatically generated by a rule-based approach <cit.>. Each label contains four values: 1 (positive), 0 (negative), -1 (uncertain), and missing. For simplicity, we treat both uncertain and missing as negative. We also use the MIMIC-CXR-JPG training set, which contains 227,827 chest X-rays with reports, to train the weak learner models.

Chexpert covers the same 13 categories as MIMIC-CXR-JPG. However, it does not include medical reports and has only image-level labels. There is no overlap between MIMIC-CXR-JPG and Chexpert. 

<ref> shows the split of pathology categories (excluding normal) in the MIMIC-CXR-JPG and Chexpert test sets. Clearly, almost all pathology categories are minor classes with much fewer positive than negative occurrences.

For our main testing regimen, we selected the five pathologies in the Chexpert Competition <cit.>, Atelectasis, Cardiomegaly, Consolidation, Edema, and Pleural Effusion.



 §.§ Implementation Details




As was mentioned in <ref>, we use the pretrained LLaVA-Med MLVLM without any further fine-tuning. We convert the classification task into a VQA task by using the prompt template shown in Row 1 of <ref>, which we name Prompt Template 1 (PT1). We first run the pretrained LLaVA-Med with PT1. Next, we incorporate pathology explanations (Row 2 of <ref>), yielding Prompt Template 2 (PT2). Finally, we integrate the predictions of weak learners into the prompts (Row 3 of <ref>), resulting in Prompt Template 3 (PT3).

As will be justified by our experiments, our weak learner is designed to suppress FP predictions. To this end, we use the pretrained ResNet50 <cit.>. For each pathology, the training dataset was sampled such that the ratio of positive and negative cases is 2:1. The model was trained for 10 epochs with a 1e-4 learning rate. The training process was monitored using the AUC score and the one with the highest validation AUC was kept. Then, the decision threshold d was fine-tuned to optimize a weighted sum of Specificity and Negative Predictive Value (NPV); i.e.,

    d = w_1TN/TN+FP+w_2TN/TN+FN,

where weights w_1 and w_2 are preset to 0.2 and 0.8, respectively. The medical images were input to the weak learners to obtain preliminary predictions for each pathology and only the negative predictions were selected to craft the PT3 prompts. 

The responses returned by LLaVA-Med can take various forms, such as “This image has Edema”, “Edema is found”, “The fluid in the lung indicates Edema”, etc. An off-the-shelf Llama-7B <cit.> serves to summarize long responses into Yes/No answers such that accuracies could easily be computed.



 §.§ Results


To demonstrate the efficacy of our prompting strategies, starting from the PT1 baseline, the pathology explanations were provided first (strategy PT2) and then, based on the results, weak learners were introduced to improve performance on specific aspects, resulting in strategy PT3.





  
PT2: Adding Pathology Explanations
<ref> reports Precision, Recall, and F1 scores of the PT1 and PT2 strategies on the MIMIC-CXR-JPG and Chexpert test sets.[The AUC and ROC scores commonly reported in the literature to assess the performances of most medical image classification models on the MIMIC-CXR-JPG and Chexpert datasets are unsuitable in our context because MLVLMs output text rather than probabilities.] On MIMIC-CXR-JPG, after adding pathology explanations, the F1 scores increased for detecting Atelectasis, Cardiomegaly, Edema, and Pleural Effusion, albeit only minimally for Consolidation. On Chexpert, after adding pathology explanations, the F1 scores for detecting Atelectasis, Cardiomegaly, and Edema increased, whereas they did not for Consolidation and Pleural Effusion. The Precision and Recall scores reveal that adding explanations generally leads to a large increase in Recall, but only minimally influences Precision. For minority pathologies such as Consolidation whose F1 score is dominated by low Precision, improving the Recall would not have much effect. Thus, PT2's performance bottleneck is Precision.







  
PT3: Referring to Weak Learners
Going beyond our PT2 strategy, we applied our PT3 strategy to further improve diagnostic accuracy. <ref> provides the TP, FP, and FN prediction counts of LLaVA-Med on the Chexpert test set using the PT2 strategy. Note the large number of FP cases. Hence, we designed our weak learners to suppress FP predictions.
<ref> compares the performance on Chexpert before and after referring to the weak learner. It shows that the F1 prediction accuracy can be substantially increased by introducing weak learner predictions into the prompts. The F1 scores of Cardiomegaly, Edema, and Pleural Effusion increase by 0.115, 0.194 and 0.089, respectively. To further demonstrate the efficacy of our PT3 strategy, <ref> compares the FP predictions of the PT2 and PT3 strategies. The reduction of FP cases is noteworthy, especially on Edema, for which the FP count is reduced by 78.5% (322). 







  
Additional VQA Experiments
<ref> shows the results of applying the PT1, PT2, and PT3 strategies with LLaVA-Med on the MIMIC-CXR-JPG and Chexpert datasets across another five medical findings: Enlarged Cardiomediastinum, Lung Lesion, Lung Opacity, Pneumonia, and Pneumothorax. Providing pathology explanations (PT2) generally yields better results over the PT1 baseline, albeit inconsistently. Introducing weak learner references (PT3) yields only limited increases in Precision, but large decreases in Recall. Generally, it offers insignificant improvement. Enlarged Cardiomediastinum, Lung Lesion, Pneumonia, and Pneumothorax are minor categories and all our experimental settings, including for the weak learner, fail to learn them. Prompting is apparently unhelpful in such situations.







  
SOTA Benchmark
<cit.> report F1 scores for detecting Atelectasis, Cardiomegaly, Consolidation, Edema, and Pleural Effusion on the Chexpert dataset using their deep learning model, as well as for the performance of radiologists. Their work offers a state-of-the-art chest X-ray diagnosis benchmark. <ref> compares the F1 scores of radiologists, the model of <cit.>, and LLaVA-Med. It shows that LLaVA-Med's VQA performance of with the baseline PT1 strategy is unsatisfactory, rendering the model far from being deployable in clinical practice. However, while still underperforming radiologists, our PT3 strategy yields a significant improvement, especially on Atelectasis, Cardiomegaly, and Edema for which the F1 score increases by approximately 17% to 21%. 



  
Application to General Domain LVLMs
Our prompt strategies can also be applied to general domain LVLMs. We studied the performance of LLaVA <cit.> and MiniGPT-v2 <cit.> using POPE metrics <cit.>, which evaluate the hallucination of LVLMs by asking questions about the presence of objects. The POPE scores of LLaVA and MiniGPT-v2 have high Precision and low Recall. Hence, our weak learner strategy can be used to reduce the FN predictions. We selected an off-the-shelf Fast-RCNN <cit.> as the weak learner, fine-tuned the detection threshold of bounding box scores to achieve high Recall, and introduced the positive predictions of the weak learner into the prompts. The results in <ref> show that the Recall scores across three POPE categories increased by around 7% (Precision scores decrease slightly), thus improving the F1 scores.



§ CONCLUSIONS AND DISCUSSION



We have tested the visual question answering abilities of the LLaVA-Med medical large vision-language model when applied to the diagnosis of pathologies. Our results show that the model has unsatisfactory performance when asked questions regarding the presence of complex pathologies. We proposed two prompt engineering strategies to improve the visual question answering accuracy of the model: providing explanations of pathologies and referring to the predictions of weak learners. The first strategy helps the model understand minority pathologies that it does not learn well in the training stage. The second strategy can help improve diagnostic accuracy in specific ways; e.g., by suppressing false positives. This strategy can also be applied to LVLMs in other, non-medical domains.

However, our two strategies are not effective on pathologies with extremely scarce data. For example, providing text explanations for Consolidation, Fracture, Lung Lesion, Pneumonia, and Pneumothorax may not suffice since the visual encoder does not adequately learn meaningful visual features. Moreover, the data may not suffice to adequately train weak learners. A promising direction for future research would be to devise a strategy for handle these rare categories. Retrieval Augmented Generation (RAG) could be a potential solution. In addition to textual explanation of a pathology, for instance, typical example images can be provided to help the model make diagnostic decisions.





§ PATHOLOGY EXPLANATIONS



The explanations of the five pathologies are as follows:



  
Atelectasis: Atelectasis refers to the partial or complete collapse of a lung or a section of lung. The features of atelectasis on an X-ray can vary depending on the cause and extent of the collapse. Some common X-ray features include: 1. The affected area may appear denser or whiter than normal lung tissue due to the collapse, leading to increased opacity on the X-ray. 2. The affected portion of the lung may appear smaller or compressed compared to the surrounding healthy lung tissue. 3. Atelectasis can cause a shift or displacement of nearby structures, such as the trachea or heart, toward the affected area. 4. In obstructive atelectasis (caused by a blockage in the airways), there might be signs of hyperinflation in the unaffected areas of the lung and a visible blockage or narrowing in the affected bronchus. 5. Linear or band-like opacities may be visible, often referred to as plate or band atelectasis, which can occur due to the collapse of small airways. Given the information above, does this image have Atelectasis?



  
Cardiomegaly: Cardiomegaly is enlargement of the heart. The definition is when the transverse diameter of the cardiac silhouette is greater than or equal to 50% of the transverse diameter of the chest (increased cardiothoracic ratio) on a posterior-anterior projection of a chest radiograph or a computed tomography. Given the information above, does this image have Cardiomegaly?



  
Consolidation: Consolidation on an X-ray refers to the filling of the lung's air spaces with fluid inflammatory exudate, or cellular material. Typical X-ray findings suggesting consolidation include: 1. Areas of increased density in the lung tissue, appearing as an opaque or hazy patch on the X-ray. Given the information above, does this image have Consolidation?



  
Edema: Pulmonary edema is the accumulation of fluid in the lungs. Some common X-ray features include: 1. Increased density in the central lung fields resembling the shape of bat wings. 2. Thin, linear opacities at the lung periphery, often indicating interstitial edema. 3. Prominent blood vessel markings due to engorgement from increased pressure in the pulmonary vasculature. Given the information above, does this image have Edema?



  
Pleural Effusion: Pleural effusion is the accumulation of fluid in between the parietal and visceral pleura. Some common X-ray features include: 1. blunting of the costophrenic/cardiophrenic angle. 2. fluid within the horizontal or oblique fissures. 3. meniscus is seen. 4. mediastinal shift occurs away from the effusion.





 Paper Title: Enhancing Semantic Similarity Understanding in Arabic NLP with Nested Embedding Learning.









Nested Arabic Embeddings




OMER NACAR




Robotics and Internet-of-Things Lab,

Prince Sultan University, Riyadh 12435, Saudi Arabia



<http://www.psu.edu.sa>

Enhancing Semantic Similarity Understanding in Arabic NLP with Nested Embedding Learning
OMER NACAR
Enhancing Semantic Similarity Understanding in Arabic NLP with Nested Embedding Learning
    Omer Nacar, Anis Koubaa
    August 5, 2024
========================================================================================



§ ABSTRACT 
This work presents a novel framework for training Arabic nested embedding models through Matryoshka Embedding Learning, leveraging multilingual, Arabic-specific, and English-based models, to highlight the power of nested embeddings models in various Arabic NLP downstream tasks. Our innovative contribution includes the translation of various sentence similarity datasets into Arabic, enabling a comprehensive evaluation framework to compare these models across different dimensions. We trained several nested embedding models on the Arabic Natural Language Inference triplet dataset and assessed their performance using multiple evaluation metrics, including Pearson and Spearman correlations for cosine similarity, Manhattan distance, Euclidean distance, and dot product similarity. The results demonstrate the superior performance of the Matryoshka embedding models, particularly in capturing semantic nuances unique to the Arabic language. Results demonstrated that Arabic Matryoshka embedding models have superior performance in capturing semantic nuances unique to the Arabic language, significantly outperforming traditional models by up to 20-25% across various similarity metrics. These results underscore the effectiveness of language-specific training and highlight the potential of Matryoshka models in enhancing semantic textual similarity tasks for Arabic NLP.






§ INTRODUCTION


Representation learning <cit.> forms the backbone of cutting-edge machine learning (ML) systems, offering rich, multidimensional vectors that capture intricate information necessary <cit.> for various Natural Language Processing (NLP) downstream tasks including  semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. These learned representations are typically static, designed to maintain high-dimensional fidelity across all applications, regardless of their unique resource and accuracy demands. This inherent rigidity often results in inefficiencies, particularly at web-scale <cit.>, where the deployment cost of these embeddings can surpass their initial computation cost <cit.>. This chapter explores the application of Nested Embedding Models specifically utilized for Arabic natural language processing (NLP), a novel approach inspired by the hierarchical, nested structure of Matryoshka dolls.

Matryoshka Representation Learning (MRL) <cit.> is a new state-of-the-art text embedding models optimized to produce embeddings with increasingly higher output dimensions, representing input texts with more values. While traditional embeddings models <cit.> produce embeddings with fixed dimensions improvement to enhance performance, it often reduces the efficiency of downstream tasks such as search or classification. Matryoshka embedding models address this issue by training embeddings to be useful even when truncated. These models can produce effective embeddings of varying dimensions.

The concept is inspired by "Matryoshka dolls", also known as "Russian nesting dolls," which are a set of wooden dolls of decreasing size placed inside one another. Similarly, Matryoshka embedding models store more critical information in the earlier dimensions and less important information in later dimensions. This characteristic allows the truncation of the original large embedding produced by the model, while still retaining sufficient information to perform well on downstream tasks. These variable-size embedding models can be highly valuable to practitioners in several ways:

Shortlisting and Reranking, instead of performing downstream tasks (e.g., nearest neighbor search) on the full embeddings, you can shrink the embeddings to a smaller size for efficient shortlisting. Subsequently, the remaining embeddings can be processed using their full dimensionality.

Trade-offs, Matryoshka models enable scaling of embedding solutions according to desired storage cost, processing speed, and performance.

The core innovation of Matryoshka Representation Learning (MRL) lies in its ability to create adaptable, nested representations through explicit optimization <cit.>. This flexibility is crucial for large-scale classification and retrieval tasks, where computational efficiency and accuracy are paramount. 

Despite the advancements in representation learning, there has been a notable gap in the application of these sophisticated techniques to the Arabic language. Arabic, being a morphologically rich and syntactically complex language, presents unique challenges that have not been adequately addressed by existing models. This motivates the development and training of Matryoshka Embedding Models specifically for Arabic NLP downstream tasks. The main contributions of this work are summarized in three folds;

Development of Arabic NLI Datasets, translations of the English Stanford Natural Language Inference (SNLI) and MultiNLI datasets into Arabic using neural machine translation (NMT), providing critical resources for Arabic natural language inference (NLI) tasks.

Training of Matryoshka Embedding Models, training various English and Arabic embedding models, transforming them into Matryoshka versions. This process enhances their adaptability and performance across different tasks.

Comprehensive Evaluation and Public Release, conducting an evaluations of these trained models, offering valuable insights and making both the datasets and models publicly available on Hugging Face to facilitate broader research and application.

This chapter delves into the core principles of Matryoshka Representation Learning, highlighting its ability to create adaptable, nested representations through explicit optimization. This methodology is crucial for large-scale classification and retrieval tasks, providing significant computational benefits without compromising on accuracy. We demonstrate the practical advantages of MRL by integrating it with established NLP models, achieving notable speed-ups and maintaining high accuracy across various applications.




§ RELATED WORK

In the realm of natural language processing and machine learning, representation learning has emerged as a critical area of research. The ability to create rich, multidimensional representations of data has paved the way for significant advancements in various applications, from semantic textual similarity to large-scale classification and retrieval tasks. This section reviews the key developments in representation learning, efficient classification and retrieval, and nested adaptive neural networks, highlighting the innovations that have informed the creation of Matryoshka Representation Learning (MRL). Furthermore, it positions our work within this broader context, showcasing how our contributions uniquely address the challenges and opportunities in Arabic NLP.

Representation Learning, the development of general-purpose representations has significantly advanced through the advent of large-scale datasets like ImageNet <cit.> and JFT <cit.>. These datasets enable the training of models applicable to a variety of tasks in both computer vision and natural language processing (NLP). Supervised learning typically frames representation learning as a classification problem, whereas unsupervised and self-supervised approaches employ proxy tasks such as instance discrimination and reconstruction to achieve similar goals <cit.>. Recent breakthroughs in contrastive learning <cit.> have further facilitated the extraction of meaningful representations from massive datasets, which are crucial for developing large-scale, cross-modal models.

Building upon these foundations, Matryoshka Representation Learning <cit.> introduces a method to encode multiple fidelity levels within a single representation vector. This multi-fidelity approach allows for adaptive deployment, optimizing resource usage without sacrificing accuracy. The integration of MRL with existing representation learning frameworks is straightforward, providing significant enhancements with minimal overhead.

Efficient Classification and Retrieval, efficiency in classification <cit.> and retrieval <cit.> is a critical concern, especially when dealing with large-scale data. Traditional methods to improve efficiency include dimensionality reduction, hashing, and feature selection. However, these techniques often reduce accuracy or increase computational complexity <cit.>. Approximate Nearest Neighbor Search (ANNS) techniques, such as Hierarchical Navigable Small World (HNSW) graphs <cit.>, strike a balance between accuracy and efficiency but still face challenges related to the high dimensionality of embeddings.

MRL addresses these challenges by reducing the dimensionality of embeddings without compromising the richness of the information they encode. By nesting lower-dimensional representations within higher-dimensional ones, MRL enables efficient and accurate classification and retrieval. This is particularly beneficial for adaptive systems that must operate under varying computational constraints. MRL’s hierarchical embeddings offer a flexible solution that scales well with data size and complexity.

Nested and Adaptive Neural Networks, the concept of nesting or packing neural networks of varying capacities within a larger network has been explored in the literature <cit.>. However, these approaches typically require separate forward passes for each nested network, which can be computationally expensive and inefficient for large-scale deployment. MRL differentiates itself by optimizing for a logarithmic number of nesting dimensions, allowing smooth interpolation between these dimensions and enabling efficient adaptive inference.

The ordered representations approach by <cit.>, which uses nested dropout in autoencoders, shares similarities with MRL but differs in its optimization strategy. MRL’s focus on coarse-to-fine granularity and minimal overhead during inference makes it particularly suited for web-scale applications. The flexibility and efficiency of MRL open new possibilities for real-time, large-scale NLP and classification tasks, demonstrating its potential as a transformative technique in the field.

The proposed work extends the principles of MRL to Arabic NLP, a domain that has not seen such sophisticated embedding techniques before. By training models specifically for Arabic using sentence transformers, we introduce the first versions of Arabic Matryoshka Embedding Models. These models not only address the unique challenges posed by the Arabic language but also provide a significant leap forward in the efficiency and adaptability of NLP systems for Arabic. Our contributions include the creation and public release of translated datasets and trained models, facilitating broader research and application in Arabic NLP. This pioneering work sets the stage for further advancements and practical implementations in this critical area.




§ DATASET PREPARATION


The preparation of datasets is a crucial step in developing robust sentence embeddings models, particularly for languages with fewer available resources such as Arabic. This section details the Arabic dataset used in this study, the translation process from English to Arabic, and the data preprocessing steps that were employed to ensure the datasets were ready for training nested embedding models.



 §.§ Arabic Dataset


The datasets used in this study are derived from the Stanford Natural Language Inference (SNLI) <cit.> and MultiNLI <cit.> datasets, which are well-known benchmarks for evaluating models on natural language inference (NLI) tasks <cit.>. These datasets were originally designed to facilitate various NLP tasks by providing pairs of sentences along with labels that indicate their semantic relationships. To adapt these datasets for Arabic, we created multiple subsets to have more varieties in building different embeddings models.

Pair Subset, this subset contains pairs of sentences with the columns "anchor" and "positive." The primary purpose of this dataset is to facilitate the training of embedding models that need to learn semantic textual similarity. Table <ref> shows an example of the Pair subset. The dataset includes 314K training pairs, 6.81K validation pairs, and 6.83K test pairs. By using this subset, models can be trained to recognize and quantify the semantic similarity between two sentences, which is crucial for tasks such as paraphrase identification and duplicate question detection.



Triplet Subset, this subset extends the pair subset by including a third column, "negative," to form triplets of sentences. It contains 558K training triplets, 6.58K validation triplets, and 6.61K test triplets. Table <ref> shows an example of the Triplet subset. The inclusion of a negative example allows the model to not only recognize similar pairs but also to distinguish between similar and dissimilar pairs. This is particularly useful for contrastive learning approaches, where the model learns to pull similar sentences closer and push dissimilar sentences apart in the embedding space.



Pair-Class Subset, this subset comprises three columns: "premise," "hypothesis," and "label," with 942K training examples, 19.7K validation examples, and 19.7K test examples. The label indicates the relationship between the premise and hypothesis, such as "0": "entailment", "1": "neutral", "2": "contradiction". Table <ref> shows an example of the Pair-Class subset. This subset is specifically designed for natural language inference tasks, where the goal is to determine the logical relationship between two sentences. It provides a rich resource for training models that need to understand and reason about the semantic content of sentences.




Pair-Score Subset, this subset includes columns "sentence1," "sentence2," and "score," with 942K training pairs, 19.7K validation pairs, and 19.7K test pairs. The score represents the degree of similarity between the two sentences on a continuous scale. Table <ref> shows an example of the Pair-Score subset. This subset is ideal for tasks that require a fine-grained understanding of semantic similarity, such as ranking and retrieval systems. Models trained on this subset can learn to assign similarity scores to sentence pairs, which can be used to improve the performance of search engines and recommendation systems.



Arabic STSB Structure, this subset is an Arabic version of the Semantic Textual Similarity Benchmark <cit.>. It consists of sentence pairs drawn from diverse sources such as news headlines, video and image captions, and natural language inference data. Each pair is annotated with a similarity score normalized between 0 and 1. The dataset includes 5.75K training pairs, 1.68K validation pairs, and 1.38K test pairs. Table <ref> shows an example of the STSB Benchmark subset. This subset provides a benchmark for evaluating the performance of models on the task of semantic textual similarity in Arabic, offering a standardized way to measure and compare model performance. Table <ref> summarizes the subsets details with their training, validation, and test splits. 






These subsets cover a broad range of tasks from semantic textual similarity to classification, making them versatile for training and evaluating embedding models.



 §.§ Data Preprocessing

Data preprocessing is a critical step to prepare the raw translated text for model training. The following preprocessing steps were applied to both the original Arabic and the translated datasets:

Tokenization, the text was tokenized into individual words or sub-words using SentencePiece <cit.>, which helps in handling the morphological richness of Arabic more effectively. This step breaks down the text into smaller units, making it easier for the model to learn and generalize from the data.

Normalization, text normalization was performed to standardize various forms of Arabic script. This included the removal of diacritics, normalization of character forms, and handling of punctuation. Normalization ensures that different forms of the same word are treated uniformly by the model.

Data Structuring, the datasets were structured into the required format with specific columns for each task. For instance, pairs of sentences were organized into "anchor" and "positive" columns for similarity tasks, while triplets included an additional "negative" column. This structuring helps in efficiently loading and processing the data during training.

Validation and Test Splits, the datasets were split into training, validation, and test sets to ensure proper evaluation of the models. The splits were carefully maintained to ensure that the distributions of data remained consistent across these sets. This step is crucial for evaluating the performance of the models and preventing overfitting.

Saving Processed Data, the processed datasets were saved in CSV format to facilitate easy loading and use in subsequent training phases. The data was encoded in UTF-8 to preserve the integrity of Arabic characters. Storing the data in a structured format ensures that it can be easily shared and reused by other researchers.

By obtaining these preprocessing steps, we prepared the datasets for the machine translation process to translate these subsets into Arabic language. 



 §.§ Translation Process

The translation of the datasets from English to Arabic was performed using Neural Machine Translation (NMT) <cit.>, a sophisticated technique that leverages neural networks to achieve high-quality translations. This process involved several meticulous steps to ensure the quality and accuracy of the translations, which are crucial for effective downstream NLP tasks.

Dataset Loading, the first step involved loading the English datasets using the Hugging Face datasets library[https://huggingface.co/docs/datasets/en/index]. This library offers a seamless way to access and manipulate a variety of NLP datasets. By leveraging this library, we ensured that the data was in a structured format, making it easier to process and translate. The datasets included various subsets such as the SNLI and MultiNLI datasets, which are benchmark datasets for natural language inference.

Translation Model Configuration, for the translation task, we used the CTranslate2 model[https://github.com/OpenNMT/CTranslate2], a powerful NMT model designed for efficient translation tasks. To handle sub-word tokenization, we employed the SentencePiece model. SentencePiece is particularly useful for languages like Arabic, which have rich morphological structures. The model paths for CTranslate2 and SentencePiece were set to ensure that the models could correctly process the source and target languages.

Language Specification, it was essential to correctly specify the source and target languages to ensure accurate translations. We used language codes eng_Latn for English and arb_Arab for Arabic. This specification helped the NMT model to understand the linguistic characteristics of both languages, thereby improving the quality of the translations.

Batch Translation, given the large size of the datasets, the translation process was performed in batches. This approach helped manage computational resources effectively and ensured that the translation process was scalable. Each batch of sentences was first tokenized into sub-words using SentencePiece. This tokenization step is crucial because it breaks down the text into manageable units, which the NMT model can then process more effectively.

Handling Special Tokens, special tokens, such as language tags and end-of-sequence markers, were handled carefully to avoid any artifacts in the translated text. For example, the source sentences were prefixed with the source language tag, and the target sentences were postfixed with the target language tag. After translation, any unnecessary tokens were removed to ensure that the final output was clean and ready for downstream tasks.

These steps of translation are conducted and adapted for each on the data subsets with their columns ensuring translating almost the same number of samples as in original. Moreover, the translation process steps are further controlled by the following main steps including;

SentencePiece Tokenization, each sentence was tokenized into sub-words using SentencePiece. This step ensured that the NMT model could handle the text more efficiently, especially for languages with complex morphology like Arabic.

Translation with CTranslate2, the tokenized sentences were then fed into the CTranslate2 model for translation. The model produced translated sentences in the form of sub-word tokens.

Detokenization, the sub-word tokens were then detokenized back into full sentences using SentencePiece. This step is crucial to reconstruct the sentences in the target language accurately.

Additional steps have been taken into consideration as well focusing on maintaining data consistency and quality assurance where the translated dataset was structured to maintain consistency with the original dataset format. This consistency is vital for ensuring that the translated datasets can be easily integrated into downstream tasks without requiring significant modifications. The structure included specific columns for different types of tasks, such as sentence pairs for similarity tasks and triplets for contrastive learning tasks. Moreover, to ensure the quality of the translations, we performed several checks. A random sample of the translated sentences was manually reviewed to verify the accuracy and fluency of the translations. 

The meticulous translation process described above resulted in high-quality Arabic versions of the SNLI and MultiNLI datasets. These translated datasets are structured to facilitate various NLP tasks and are made publicly available on Hugging Face[https://huggingface.co/collections/Omartificial-Intelligence-Space/arabic-nli-and-semantic-similarity-datasets-6671ba0a5e4cd3f5caca50c3] to enable broader research and development in Arabic NLP. This translation process not only ensures high-quality data but also sets a precedent for translating other important NLP datasets into Arabic or other low-resource languages.

The combined efforts in dataset preparation, translation, and preprocessing resulted in a robust set of Arabic NLP datasets that are versatile and ready for training state-of-the-art embedding models. The public release of these datasets on Hugging Face will enable broader research and development in Arabic NLP, fostering advancements in this critical area.




§ METHODOLOGY


The methodology section outlines the steps and processes involved in the development and evaluation of the Arabic Matryoshka Embedding Models. This includes the selection of appropriate models, training procedures, and evaluation techniques used to assess their performance on Arabic NLP tasks.



 §.§ Model Selection


In this study, we aimed to train and evaluate several Matryoshka embedding models using the Arabic NLI triplet dataset. The selection of models was guided by their proven effectiveness in various NLP tasks, as well as their ability to handle Arabic text to varying extents. The models chosen for this investigation include both monolingual and multilingual sentence transformers, as well as models specifically designed for the Arabic language.

English Sentence Transformer Model, we have utilized mpnet-base-all-nli-triplet[https://huggingface.co/tomaarsen/mpnet-base-all-nli-triplet] model that is a fine-tuned version from microsoft/mpnet-base[https://huggingface.co/microsoft/mpnet-base], which maps sentences and paragraphs to a 768-dimensional dense vector space. It is designed for tasks such as semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. Despite being primarily trained on English data, it has been exposed to a few Arabic tokens, making it a suitable candidate for assessing cross-lingual transfer capabilities in our investigation.

Multilingual Sentence Transformer Models, we have utilized paraphrase-multilingual-mpnet-base-v2[https://huggingface.co/sentence-transformers/paraphrase-multilingual-mpnet-base-v2] model that maps sentences and paragraphs to a 768-dimensional dense vector space and is suitable for tasks like clustering and semantic search. It supports multiple languages, making it a strong candidate for multilingual NLP applications. Additionally, we have used LaBSE (Language-Agnostic BERT Sentence Embedding)[https://huggingface.co/sentence-transformers/LaBSE], model that maps 109 languages including Arabic into a shared vector space, facilitating cross-lingual tasks. It is a robust choice for evaluating multilingual capabilities in embedding models.

Arabic Sentence Transformer Models, to expand our investigation, we have used Arabic based sentence transformers including AraBERT[https://huggingface.co/aubmindlab/bert-base-arabertv02], an arabic pretrained language model based on Google's BERT architecture. It uses the BERT-Base configuration and is tailored for Arabic NLP tasks. AraBERT has demonstrated high performance in various Arabic language benchmarks. Moreover, MARBERT[https://huggingface.co/UBC-NLP/MARBERT] model which is designed to handle both Modern Standard Arabic (MSA) and Dialectal Arabic (DA). It is based on the BERT architecture and trained on a large corpus of Arabic tweets, making it particularly effective for tasks involving social media text.

The choice of these models allows for a comprehensive evaluation across different dimensions of Arabic NLP. The inclusion of an English model exposed to some Arabic tokens provides insights into the transferability of learned representations across languages. The multilingual models offer a perspective on how well models trained on diverse language data perform on Arabic specific tasks. Finally, the Arabic-specific models, AraBERT and MARBERT, serve as benchmarks for performance in native Arabic NLP contexts.

By training  these models on the Arabic NLI triplet dataset, we aim to create Matryoshka embedding versions that are not only versatile and efficient but also tailored to the unique characteristics and challenges of the Arabic language. This thorough approach ensures that our models are well-equipped to handle a variety of NLP tasks, including semantic textual similarity, semantic search, paraphrase mining, text classification, and clustering, thereby advancing the state of Arabic NLP.





 §.§ Matryoshka Embedding Models


Matryoshka Embedding Models <cit.> represent an innovative approach to creating adaptable and efficient embeddings for natural language processing tasks. These models aim to capture multi-granularity in embeddings, allowing different levels of the embedding vector to independently serve as meaningful representations. This adaptability is crucial for efficiently managing computational resources, especially in large-scale and resource-constrained environments like Arabic NLP.


Matryoshka Representation Learning (MRL) process is shown in Figure <ref> which involves a d-dimensional representation vector z ∈ R^d for a given datapoint x in the input domain χ. This representation vector is obtained using a deep neural network F(.;θ_F), parameterized by learnable weights θ_F. The objective of MRL is to ensure that each of the first m dimensions of the embedding vector, z_1:m∈ R^m, where m∈ M, can independently serve as a transferable and general-purpose representation of the datapoint x.

The multi-granularity of these embeddings is captured through the set of chosen dimensions M, which are determined by consistent halving until the representation size reaches a minimal informative state. This nested, coarse-to-fine granularity ensures that the representations remain useful even when truncated to smaller dimensions.

Given a labeled dataset D = {( x_1 , y_1) , ...,( x_N, y_N) }, where x_i∈χ is an input point and y_i∈[ L ] represents the label of x_i MRL optimizes the multi-class classification loss for each nested dimension m∈ M using standard empirical risk minimization. This is achieved by employing a separate linear classifier, parameterized by W^( m )∈ R^L × m for each dimension. The losses obtained from these classifiers are then aggregated, taking into account their relative importance c_m≥ 0 for m∈ M.  The optimization objective can be expressed as:


    ℒ_MRL = ∑_m ∈ M c_m ℒ_CE(𝐖^(m)𝐳_1:m, y)


where:

  
  * ℒ_MRL is the Matryoshka Representation Learning loss.
  
  * c_m represents the relative importance of each dimension m.
  
  * ℒ_CE is the multi-class softmax cross-entropy loss function.
  
  * 𝐖^(m)∈ℝ^L × m are the weights of the linear classifier for dimension m.
  
  * 𝐳_1:m∈ℝ^m is the truncated embedding vector up to dimension m.
  
  * y is the true label corresponding to the input x.


This formulation ensures that each subset of the embedding dimensions can independently perform well on the classification task, maintaining the flexibility and robustness of the learned representations.

To enhance efficiency, weight-tying is employed across all the linear classifiers, i.e., defining W^( m ) = W_1:m for a set of common weights W. This reduces the memory cost associated with the linear classifiers, which is particularly crucial in cases of extremely large output spaces. This variant is known as Efficient Matryoshka Representation Learning (MRL-E).

In practice, MRL involves the following steps:

Representation Vector Generation, a deep neural network generates a high-dimensional representation vector for each input datapoint.

Dimension Nesting, the high-dimensional vector is divided into nested subsets of dimensions, ensuring that the first few dimensions capture the most crucial information.

Separate Classifier Training, each subset of dimensions is used to train a separate linear classifier, optimizing the classification loss for each dimension.

Loss Aggregation, the classification losses from all subsets are aggregated based on their relative importance to form the final loss function.

By leveraging these methods, Matryoshka Embedding Models achieve flexibility and efficiency, making them suitable for adaptive deployment in various NLP tasks. Figure <ref> shows the truncation step of the Matryoshka embedding where the ability to truncate embeddings is performed without significant loss of information allows for scalable and resource-efficient applications, particularly beneficial in the context of Arabic NLP where computational resources may be limited.





 §.§ Nested Embedding Training Process


The process of training nested embedding models on the translated Arabic dataset was methodically structured to ensure optimal performance. The training setup began with the initialization of several Sentence Transformer models. The primary model used was mpnet-base-all-nli-triplet, which maps sentences and paragraphs to a 768-dimensional dense vector space. Additionally, two multilingual sentence transformer models, paraphrase-multilingual-mpnet-base-v2 and LaBSE, and two Arabic-specific models, AraBERT and MARBERT, were trained. These models were selected to leverage their existing capabilities while adapting them to handle the specific characteristics of the Arabic language.

The dataset used for training, the arabic-nli-triplet subset, was loaded using the Hugging Face datasets library. This dataset comprises triplets of sentences, each consisting of an anchor, a positive, and a negative example with size of 558k samples. The models were configured to handle these inputs, with settings such as using cosine similarity as the similarity function and defining the maximum sequence length as 512 tokens.

Hyper-parameter tuning was a critical aspect of the training process. The batch size was set to 128 to balance the computational load and model performance, and each model was trained for one epoch, leveraging the computational power of an A100 GPU. The Matryoshka embedding approach was implemented by specifying output dimensions of [768, 512, 256, 128, 64] , allowing the models to produce embeddings at various levels of granularity. This hierarchical structure enables flexible and efficient processing of downstream tasks.

Optimization during training involved the use of specific loss functions. The primary loss function, MultipleNegativesRankingLoss <cit.>, was augmented with MatryoshkaLoss to train embeddings at multiple dimensions simultaneously <cit.>. This combined approach ensured that the embeddings were effective across different levels of granularity, making them robust for various applications.

The training process was managed by data loading and preprocessing involving shuffling the dataset and selecting a subset of examples for training to manage computational resources effectively. Each triplet was processed to ensure consistency and quality. The MultipleNegativesRankingLoss handled the ranking of the embeddings, while the MatryoshkaLoss maintained their hierarchical structure. An evaluator was set up using the STS Benchmark validation dataset to monitor performance during training. This involved calculating cosine similarity between sentence pairs and tracking performance across different dimensions of the embeddings.

The training execution was conducted using SentenceTransformerTrainer from the Sentence Transformers library[https://sbert.net/index.html]. The training arguments included specifications for the number of epochs, batch size, learning rate scheduling, and evaluation strategies. After training, the models were evaluated on the STS Benchmark test dataset to assess their performance across different dimensions. The results were tracked and recorded, highlighting the effectiveness of the Matryoshka embedding approach. Finally, the trained models were saved locally and uploaded to the Hugging Face Hub for public access[https://huggingface.co/collections/Omartificial-Intelligence-Space/arabic-matryoshka-embedding-models-666f764d3b570f44d7f77d4e].

This meticulous training ensured that the process Matryoshka embedding models were robust, efficient, and ready for deployment in various NLP tasks, particularly in the context of Arabic language processing.



§ RESULTS & DISCUSSION


In this section, we provide a detailed framework of the evaluation process and the metrics used to assess the performance of the trained Arabic Matryoshka embedding models on Arabic datasets. Our aim is to thoroughly examine the models' efficacy in various dimensions and discuss the outcomes in the context of Arabic natural language processing tasks.

To evaluate the performance of our models, we utilized the Arabic Semantic Textual Similarity Benchmark (STSB)[https://huggingface.co/datasets/Omartificial-Intelligence-Space/Arabic-stsb]. This dataset is a comprehensive collection of sentence pairs drawn from diverse sources, including news headlines, video and image captions, and natural language inference data. Each sentence pair in the dataset is human-annotated with a similarity score ranging from 1 to 5, which we normalized to a scale between 0 and 1 for this evaluation. This normalization ensures consistency and facilitates a more straightforward comparison across different models and dimensions.

The evaluation of our Arabic Matryoshka embedding models was conducted using the 
EmbeddingSimilarityEvaluator, a robust evaluation tool provided by the Sentence Transformers library. This evaluator is specifically designed to measure the similarity of embeddings by calculating the Spearman and Pearson rank correlation coefficients against the gold standard labels. These correlations provide a quantitative measure of how well the embeddings capture the semantic similarity between sentence pairs.

The EmbeddingSimilarityEvaluator performs evaluation based on several metrics:

Cosine Similarity, measures the cosine of the angle between two vectors, providing a value between -1 and 1.

Manhattan Distance, calculates the absolute differences between corresponding elements of the vectors.

Euclidean Distance, computes the straight line distance between two points in the vector space.

Dot Product Similarity, computes the dot product of two vectors, which can indicate similarity in direction and magnitude.

These metrics were computed for each dimension of the embeddings (768, 512, 256, 128, 64), 
resulting in a comprehensive evaluation of the models' performance at various levels of granularity. 
For each dimension, we used the following metrics:

Pearson Correlation (Cosine, Manhattan, Euclidean, Dot, Max), measures the linear correlation between the predicted and actual similarity scores.

Spearman Correlation (Cosine, Manhattan, Euclidean, Dot, Max), Assesses the monotonic relationship between the predicted and actual similarity scores.


The evaluation process commenced with loading the pretrained Sentence Transformer models and the STSB dataset using the Hugging Face datasets library. Each model was evaluated on the sentence pairs in the dataset, with the EmbeddingSimilarityEvaluator applied to calculate the similarity scores. This process involved initializing the evaluator with the sentences and their corresponding human-annotated scores, specifying the primary similarity function (cosine similarity), and setting other relevant parameters such as batch size and truncation dimension.

The evaluator outputs a set of correlation scores for each metric and dimension, providing a detailed view of how well the embeddings align with human judgment of sentence similarity. This methodology ensures a rigorous and standardized evaluation, enabling us to compare the performance of different models and configurations objectively.

The results from this evaluation process will be discussed in the subsequent subsections, where we will delve into the performance metrics, compare the models, and interpret the findings in the context of their application to Arabic NLP tasks. This detailed analysis will highlight the strengths and potential areas for improvement in our Arabic Matryoshka embedding models, contributing to the broader understanding and development of efficient and effective NLP solutions for the Arabic language.



 §.§ Comprehensive Performance Analysis For Each Trained Arabic Matryoshka Embedding Model


A deep investigation have been done to evaluate the performance of different trained Arabic Matryoshka embedding models on different dimensions. The experiments consists of evaluating Arabic embeddings models and multilingual embeddings models along with An English model subjected to a small number of Arabic tokens is included to shed light on the cross-linguistic transferability of learnt representations. Results of the first multilingual model named  Paraphrase-Multilingual-MPNet-Base-V2 which is trained to create Arabic Matryoshka embeddings across different dimensions are summarized in Table <ref>.



As shown in Table <ref>, the model shows robust performance across all dimensions, with a slight decrease in correlation as the dimensionality reduces. The highest Pearson and Spearman correlations are observed at dimensions 768 and 512, indicating strong semantic similarity capture. Moreover, Dot product similarity metrics decrease significantly with lower dimensions, highlighting its sensitivity to dimensionality reduction.

Secondly, results of the second multilingual model 'LaBSE' are detailed in Table <ref>.




As shown in Table <ref>, LaBSE exhibits high performance across all dimensions, showing minimal degradation in correlation values as dimensions decrease. The model's robustness is evident in maintaining high Pearson and Spearman correlations across various metrics.

Moving on, Tables <ref> and  <ref> shows the performance evaluation metrics for the arabic embeddings models Arabert and MArbert respecitvely.






As shown in Table <ref>, Trained Arabert Matryoshka Model shows moderate performance, with correlations improving slightly as dimensions decrease from 768 to 64. While the results are lower than those for multilingual models, they are consistent across various metrics. Moreover, as shown in Table <ref> for Marbert model shows strong performance at higher dimensions, with consistent Pearson and Spearman correlations. The model's performance in dot product similarity is lower compared to other metrics, suggesting an area for potential improvement.

Finally, the results of English model MPNet-Base-All-NLI-Triplet, which has seen a few Arabic tokens, are shown in Table <ref>.



As shown in Table <ref>, the performance of the MPNet-Base-All-NLI-Triplet Matryoshka Model is noticeably lower compared to the multilingual model, especially in lower dimensions. Despite this, the model shows reasonable consistency across Pearson and Spearman correlations. The ability to handle Arabic tokens, though limited, provides some insights into the adaptability of English-trained models.

The evaluation results indicate that multilingual models like Paraphrase-Multilingual-MPNet-Base-V2 and LaBSE outperform specifically Arabic-trained models in capturing semantic similarity in Arabic text. The trained Paraphrase-Multilingual-MPNet-Base-V2 model demonstrated the highest performance, particularly in higher dimensions. In contrast, models specifically designed for Arabic, such as BERT-Base-AraBERTV02and MARBERT, show moderate performance but provide valuable insights into the nuances of Arabic language processing. The decrease in performance at lower dimensions for all models highlights the challenge of dimensionality reduction in maintaining semantic integrity.

Overall, these results underscore the importance of multilingual capabilities in embedding models for diverse language tasks and point to areas where Arabic-specific models can be further improved to match their multilingual counterparts. 




 §.§ Comparative Analysis of Different Arabic Trained Matryoshka Models Performance Across Metrics and Dimensions


This subsection provides a detailed visual comparison of the performance of various Arabic Matryoshka embedding models across different dimensions. By examining the comparative plots for multiple evaluation metrics, including Pearson and Spearman correlations, we gain insights into how each model performs at different levels of granularity. Plots shown in Figure <ref> serve as a visual representation of the quantitative data, offering a clearer perspective on the relative performance of each model.



Comparison of Model Performance on Pearson Manhattan Similarity is shown in Figure <ref> (a) where, the paraphrase-multilingual-mpnet-base-v2 model consistently outperformed other models across all dimensions, exhibiting the highest Pearson Manhattan similarity values. LaBSE demonstrated stable performance with slight decreases at lower dimensions. The mpnet-base-all-nli-triplet model showed moderate performance, surpassing the Arabic-specific models, bert-base-arabertv02, and MARBERT. Among these, MARBERT and bert-base-arabertv02 had the lowest values, indicating less effectiveness in capturing semantic similarity using Pearson Manhattan similarity.

Comparison of Model Performance on Spearman Manhattan Similarity, is shown in Figure <ref> (b) where, the paraphrase-multilingual-mpnet-base-v2 model again showed the highest Spearman Manhattan similarity values, reinforcing its strong performance. LaBSE followed closely, maintaining stable performance across dimensions. The mpnet-base-all-nli-triplet model's performance was moderate, similar to its Pearson Manhattan metric results. MARBERT and bert-base-arabertv02 exhibited lower values, suggesting less consistency in semantic similarity capture.

Comparison of Model Performance on Pearson Euclidean Similarity is shown in Figure <ref> (c) where, the Pearson Euclidean similarity is compared, the paraphrase-multilingual-mpnet-base-v2 model maintained high values across dimensions. LaBSE also performed consistently well. The mpnet-base-all-nli-triplet model showed moderate performance, surpassing the Arabic-specific models. MARBERT and bert-base-arabertv02 showed lower values, with MARBERT slightly outperforming bert-base-arabertv02.

Comparison of Model Performance on Spearman Euclidean Similarity, is shown in Figure <ref> (d) for Spearman Euclidean similarity where, the paraphrase-multilingual-mpnet-base-v2 model led with the highest values. LaBSE followed closely, showing stable values across dimensions. The mpnet-base-all-nli-triplet model demonstrated moderate performance. The lowest values were observed in MARBERT and bert-base-arabertv02, with MARBERT slightly outperforming bert-base-arabertv02.

Comparison of Model Performance on Pearson Dot Similarity, is shown in Figure <ref> (e) where, The performance of the paraphrase-multilingual-mpnet-base-v2 model decreased with lower dimensions, showing the highest Pearson Dot similarity value at 768 dimensions. LaBSE exhibited a similar trend, with noticeable drops at lower dimensions. The mpnet-base-all-nli-triplet model showed moderate performance, with a significant decline at lower dimensions. MARBERT and bert-base-arabertv02 had the lowest values, with MARBERT slightly outperforming bert-base-arabertv02.

Comparison of Model Performance on Spearman Dot Similarity, is shown in Figure <ref> (f) where, the paraphrase-multilingual-mpnet-base-v2 and LaBSE models showed higher values in Spearman Dot similarity, with noticeable decreases at lower dimensions. The mpnet-base-all-nli-triplet model demonstrated moderate performance, with significant declines at lower dimensions. The lowest values were observed in MARBERT and bert-base-arabertv02, indicating less consistency in capturing semantic similarity using Spearman Dot similarity.

Overall, the paraphrase-multilingual-mpnet-base-v2 model consistently outperformed others across various metrics, followed by LaBSE. The mpnet-base-all-nli-triplet model showed moderate performance, while MARBERT and bert-base-arabertv02 lagged, especially in capturing semantic similarity at lower dimensions.




 §.§ Comparison of Base Models Vs. Arabic Trained Matryoshka Models


In this subsection, we aim to evaluate the performance of Matryoshka embedding models for Arabic by comparing the base models with their trained Matryoshka counterparts, we can discern how the nested embedding learning might enhance the models' ability to capture semantic similarity in the Arabic language. This analysis will be conducted across different evaluation metrics on 768 dimension, providing a comprehensive understanding of the improvements brought about by learning. To visually represent the impact of training Matryoshka Models, Figure <ref>, provide bar plots for each model comparing their performance against base models on two key metrics: Pearson Cosine Similarity and Spearman Cosine Similarity.



As shown in Figure <ref> (a), the trained Matryoshka model  using MARBERT outperformed the base MARBERT model significantly in both Pearson and Spearman cosine similarity metrics. This substantial enhancement indicates that the Matryoshka version has a much-improved capability to capture semantic similarities, making it more suitable for nuanced language tasks in Arabic. Additionally, For the Arabert model shown in Figure <ref> (b), the trained Matryoshka version has also demonstrated significant improvements comparing to the base Arabert model. The enhanced scores across the cosine similarity metrics underscore the effectiveness of the Matryoshka model in refining the model's performance for Arabic text processing.


For the multilingual models shown in Figure <ref> (c) and (d), the trained Matryoshka model of LaBSE model showed improved performance compared to the base LaBSE model and also the trained multilingual mpnet base model also demonstrated superior performance over its base counterpart across both Pearson and Spearman cosine similarity metrics. This consistency suggests that the Matryoshka model enhances the model's adaptability to the Arabic language, making it more effective in capturing semantic similarities within this context.

Finally, the comparison for the English mpnet base model reveals that the Arabic Matryoshka version outperformed the base model  substantially. The improvements across Pearson and Spearman cosine metrics highlight the trained model's enhanced semantic understanding and accuracy for Arabic text.



 §.§ Analysis of Similarity Scores Predicted by Arabic Trained Matryoshka Models


In this subsection, we analyze the similarity scores predicted by our Arabic trained Matryoshka models against the ground truth. The examples include cases of perfect similarity (score 1), no similarity (score 0), and moderate similarity (scores between 0 and 1). Table <ref>, <ref> and <ref> show the detailed examples and their corresponding analysis.




As shown in Table <ref>, with a ground truth score of 0.72, indicating moderate similarity, the models show varying degrees of accuracy. Marbert-all-nli-triplet-Matryoshka and Arabic-labse-Matryoshka predict slightly higher scores (0.836 and 0.835, respectively), closely aligning with the ground truth. This indicates the models' effectiveness in capturing moderate semantic similarities, particularly when sentences share significant contextual overlap.





As shown in Table <ref>, for the ground truth score of 0.1, indicating no similarity, the Matryoshka models' scores are notably higher, ranging from 0.323 to 0.481. This suggests that while the models recognize some degree of unrelatedness, they still infer a minimal level of semantic similarity, possibly due to shared contextual elements like "رجل" (man) in both sentences.





As shown in Table <ref>, the ground truth score is 1, indicating perfect similarity between the two sentences. All the Matryoshka models also predict high similarity scores, with Arabic-all-nli-triplet-Matryoshka achieving the highest score of 0.906. This demonstrates the model's capability to capture near-perfect semantic similarity for sentences with minimal lexical variation.

To further analyze the performance of the trained Matryoshka models, we compare their average predicted similarity scores against the ground truth scores across three different similarity categories: low similarity, no similarity, and moderate similarity. The comparative results are illustrated in the Figure <ref>



The error analysis reveals that while the Arabic trained Matryoshka models generally perform well in identifying high and moderate similarity sentence pairs, they exhibit a tendency to overestimate similarity in the no similarity category. For the low similarity category, all models predicted scores close to the ground truth, demonstrating their effectiveness in capturing high similarity relationships. Specifically, the Marbert-all-nli-triplet-Matryoshka and Arabic-labse-Matryoshka models showed the highest accuracy in this category. However, in the no similarity category, the predicted scores were consistently higher than the ground truth, indicating false positives. 

This suggests a potential area for improvement, as the models tend to recognize some degree of similarity in dissimilar pairs. In the moderate similarity category, the models also performed well, although there was a slight tendency to overestimate similarity. Overall, these findings suggest that while the models are effective at capturing similarities, they require further refinement to accurately identify dissimilar pairs and reduce false positive rates. This enhancement can improve the robustness and accuracy of similarity models in multilingual contexts, especially for the Arabic language.

The consistent trend of improved performance across all trained Arabic Matryoshka models highlights the efficacy of the Nested learning process. The significant gains in both Pearson and Spearman cosine similarity metrics indicate that the trained models have a better grasp of the semantic nuances in the Arabic language. This enhancement makes these models more suitable for tasks requiring high precision in semantic similarity assessments, such as machine translation, information retrieval, and natural language understanding tasks specific to Arabic.





§ ARABIC SENTENCE SIMILARITY APPLICATION


In addition to the comprehensive evaluation of the Arabic Matryoshka embedding models, we have developed an interactive Gradio application that leverages these models to compute semantic similarity between Arabic sentences. This tool is designed to provide users with a practical interface to utilize the advanced capabilities of SentenceTransformer models in real-world scenarios. Figure <ref> shows the user interface of the app depolyed in hugging-face[https://huggingface.co/spaces/Omartificial-Intelligence-Space/Arabic-Sentence-Similarity-Matryoshka-Models]. 

As shown in Figure <ref>, the app provides a variety of choices and modes along with the Arabic trained Matryoshka models combined together in the following Key Features;

Model Selection, users can choose from a variety of Arabic Matryoshka Embedding Models. This flexibility allows for comparison and selection of the most suitable model for specific needs.

Flexible Comparison Modes, the application supports two primary comparison modes. Users can either compare two sentences directly or compare one sentence against three others, offering a versatile approach to semantic similarity evaluation.

Custom Embedding Dimensions, to cater to various computational and accuracy requirements, the application allows users to select embedding dimensions from [768, 512, 256, 128, 64]. This feature ensures that the tool can be adapted to different performance and precision needs.

Detailed Similarity Scores, the application provides detailed similarity scores between sentences, enabling users to understand the nuances of the semantic relationships captured by the models.

By providing this application, we aim to bridge the gap between advanced model development and practical usability, making state-of-the-art sentence similarity computations accessible and user-friendly.



§ CONCLUSION

In this chapter, we conducted a thorough evaluation of multiple trained Arabic nested embedding models on the Arabic Semantic Textual Similarity Benchmark. Our analysis included models that are multilingual and those specifically trained for Arabic. Using the EmbeddingSimilarityEvaluator, we assessed their performance based on several metrics: Pearson and Spearman correlations for cosine similarity, Manhattan distance, Euclidean distance, and dot product similarity.

The results clearly demonstrate that tained arabic nested embedding models on Arabic data significantly improves their performance. The paraphrase-multilingual-mpnet-base-v2 model consistently outperformed others across most dimensions and metrics, highlighting its robustness in capturing semantic similarity. LaBSE also showed stable performance, particularly in higher dimensions, making it a reliable choice for multilingual tasks. The mpnet-base-all-nli-triplet model exhibited moderate performance, better than the bert-base-arabertv02 and MARBERT models, which had the lowest values in most metrics. Our comparative analysis of the base models versus their Matryoshka counterparts further emphasizes the importance of language-specific training. The Matryoshka models showed marked improvements in capturing the semantic nuances of Arabic, as reflected in the significant increase in Pearson and Spearman correlations.

These findings underscore the necessity of adapting NLP models to specific languages to achieve optimal performance. The insights gained from this evaluation can guide future research and development efforts in creating more effective and nuanced language models for Arabic and other underrepresented languages.



  §.§.§ Acknowledgments
 The authors thank Prince Sultan University for their support.


4
LeCunY. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521(7553):436–444, 2015.
Nayak P. Nayak. Understanding searches better than ever before. Google AI Blog, 2019. URL https: //blog.google/products/search/search-language-understanding-bert/.
DeanJ. Dean. Challenges in building large-scale information retrieval systems. In Keynote of the 2nd ACM International Conference on Web Search and Data Mining (WSDM), volume 10, 2009.
SunC. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843–852, 2017.
KusupatiKusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V., ... & Farhadi, A. (2022). Matryoshka representation learning. Advances in Neural Information Processing Systems, 35, 30233-30249.
NilsNils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 3980–3990. Association for Computational Linguistics.
DengJ. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
HeK. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778, 2016.
GutmannM. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297–304. JMLR Workshop and Conference Proceedings, 2010.
HarrisM. G. Harris and C. D. Giachritsis. Coarse-grained information dominates fine-grained information in judgments of time-to-contact from retinal flow. Vision research, 40(6):601–611, 2000.
WaldburgerC. Waldburger. As search needs evolve, microsoft makes ai tools for better search available to researchers and developers. Microsoft AI Blog, 2019. URL https://blogs.microsoft. com/ai/bing-vector-search/.
MalkovY. A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence, 42(4):824–836, 2018.
Cai H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791, 2019.
LiM. Wallingford, H. Li, A. Achille, A. Ravichandran, C. Fowlkes, R. Bhotika, and S. Soatto. Task adaptive parameter sharing for multi-task learning. arXiv preprint arXiv:2203.16708, 2022.
Yu  J. Yu, L. Yang, N. Xu, J. Yang, and T. Huang. Slimmable neural networks. arXiv preprint arXiv:1812.08928, 2018.
RippelO. Rippel, M. Gelbart, and R. Adams. Learning ordered representations with nested dropout. In International Conference on Machine Learning, pages 1746–1754. PMLR, 2014.
BowmanBowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015). A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326.
KimKim, S., Kang, I., & Kwak, N. (2019, July). Semantic sentence matching with densely-connected recurrent and co-attentive information. In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, No. 01, pp. 6586-6593).
MacCartneyMacCartney, B., & Manning, C. D. (2008, August). Modeling semantic containment and exclusion in natural language inference. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008) (pp. 521-528).
YangYang, Y., Yuan, S., Cer, D., Kong, S. Y., Constant, N., Pilar, P., ... & Kurzweil, R. (2018). Learning semantic textual similarity from conversations. arXiv preprint arXiv:1804.07754.
KudoKudo, T., & Richardson, J. (2018). Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226.
KleinKlein, G., Kim, Y., Deng, Y., Senellart, J., & Rush, A. M. (2017). Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810.
Henderson Henderson, M., Al-Rfou, R., Strope, B., Sung, Y. H., Lukács, L., Guo, R., ... & Kurzweil, R. (2017). Efficient natural language response suggestion for smart reply. arXiv preprint arXiv:1705.00652.





 Paper Title: Effective Black Box Testing of Sentiment Analysis Classification Networks.
















	Effective Black Box Testing of Sentiment Analysis Classification Networks
    Parsa Karbasizadeh1,
		Fathiyeh Faghih2, and Pouria Golshanrad3 

	College of Engineering, Department of Electrical and Computer Engineering 
 University of Tehran, Tehran, Iran 

		Email: 1parsa.karbasi@ut.ac.ir,
		2f.faghih@ut.ac.ir,
		3pouria.golshanrad@ut.ac.ir
    August 5, 2024
=============================================================================================================================================================================================================================================================================








§ ABSTRACT 
Transformer-based neural networks have demonstrated remarkable performance in natural language processing tasks such as sentiment analysis. Nevertheless, the issue of ensuring the dependability of these complicated architectures through comprehensive testing is still open. This paper presents a collection of coverage criteria specifically designed to assess test suites created for transformer-based sentiment analysis networks. Our approach utilizes input space partitioning, a black-box method, by considering emotionally relevant linguistic features such as verbs, adjectives, adverbs, and nouns. In order to effectively produce test cases that encompass a wide range of emotional elements, we utilize the k-projection coverage metric. This metric minimizes the complexity of the problem by examining subsets of k features at the same time, hence reducing dimensionality. Large language models are employed to generate sentences that display specific combinations of emotional features. The findings from experiments obtained from a sentiment analysis dataset illustrate that our criteria and generated tests have led to an average increase of 16% in test coverage. In addition, there is a corresponding average decrease of 6.5% in model accuracy, showing the ability to identify vulnerabilities. Our work provides a foundation for improving the dependability of transformer-based sentiment analysis systems through comprehensive test evaluation. 


Journal of IEEE Software
Shell et al.: Testing Smart Contracts Gets Smarter

	

		
AI Dependability, Test,Transformers, Coverage Criteria,NLP
	








§ INTRODUCTION


The impact of neural networks on various fields, including image recognition, natural language processing, and self-driving cars, has been significant <cit.>. In the context of Natural Language Processing (NLP), neural networks have gained notable recognition due to their ability to enhance accuracy <cit.>.
Sentiment analysis, also known as opinion mining or emotion AI, is a natural language processing (NLP) technique used to determine the emotional tone or subjective opinion expressed in text.  This procedure utilizes NLP methodologies to analyze and interpret human language, enabling diverse applications across various domains. For instance, in marketing, sentiment analysis can be leveraged to evaluate customer opinions about products and services, providing valuable insights for brand reputation management. 
Various tools, including NLTK <cit.>, are employed to interpret customer sentiment and capture embedded emotions. 

Traditional emotion recognition methods often struggle with the inherent complexities of human language. These complexities include word ambiguity, where a word's emotional connotation can shift depending on context (e.g., “scared” can be positive in an exciting situation or negative in a threatening one). In addition, sarcasm and negation pose challenges in identifying true sentiment.  Furthermore, the process of understanding emotional meaning must include evaluating words that are not directly close to each other in a sentence, which is a step away from traditional methods.

Transformer-based neural networks effectively handle the ambiguity in emotion sentiment analysis through their attention mechanisms and large parameters <cit.>. Parallel text processing allows them to analyze text data efficiently and capture the hidden emotional factors that can exist across longer distances within a sentence. Through pre-training on extensive datasets, transformers are capable of learning complex language patterns and the subtle details of human emotional expression <cit.>.

To guarantee the efficacy of these complex models, comprehensive testing protocols are essential. A significant challenge lies in evaluating the sufficiency of test suites specifically designed for Transformer-based emotional sentiment analysis networks. Given their complex architecture and the large number of parameters, conventional methods like neuron coverage <cit.> seem to be impractical. To address this limitation, we introduce in this work a novel set of coverage criteria designed to evaluate test suites for the assessment of Transformer-based emotion sentiment analysis networks. We employ the principle of input space partitioning, segmenting textual data into emotionally-relevant categories. This strategy enables us to concentrate on particular areas of the input space and generate targeted test cases.

To achieve high coverage with the least number of test cases, we employ the k-projection coverage method <cit.>. This technique utilizes dimensionality reduction by examining smaller feature sets at the same time. Concentrating on these smaller subsets allows for an efficient evaluation of the network's performance across diverse emotional categories within the input space. This strategy ensures comprehensive coverage with fewer tests, thereby eliminating the necessity for extensive test suites.
Additionally, we introduce a set of specialized features specifically designed for emotional sentiment analysis. These features capture the emotional details within text data, thereby aiding the network in accurately identifying the intended emotions.


Upon applying our emotion-specific k-projection coverage method to a dataset designed for emotional sentiment analysis, we observed a 16% average increase in coverage of the generated sentences. This enhanced coverage led to 6.5% decrease in the accuracy of the System-Under-Test (SUT) on the new test suites, demonstrating the effectiveness of our approach in uncovering additional faults.



Related Work:This paper explores concepts related to test generation, black-box testing, and model coverage in DNN models. It summarizes the current state of coverage and performance measures for DNN models and discusses previous research on utilizing diversity in black-box DNN testing to evaluate DNNs and conventional software systems <cit.>.


Transformers are preferred over traditional DNNs due to their superior accuracy, particularly in AI domains like sentiment analysis <cit.>. Evaluating the adequacy of test suites for emotional sentiment analysis networks poses a significant challenge. In software engineering, test suite effectiveness is assessed through black-box and white-box testing <cit.>. Applying white-box criteria to transformer-based NLP systems is limited by their complexity and internal access requirements. Alternative testing methods such as black-box strategies or input diversity metrics may be necessary to ensure their reliability <cit.>.

Input space partitioning is an effective approach for black-box testing, particularly in text data, where emotionally relevant features can be used to partition the input domain into smaller subsets for precise test case design <cit.>. 

k-projection coverage, employing dimensionality reduction, shows promise in image recognition for evaluating self-driving car perception systems using CNN-based neural networks and GAN-based models <cit.>. However, its potential in text-based emotional sentiment analysis remains unexplored.

The contributions of our study are as follows: 

    
  * We introduce novel coverage criteria based on k-projection for black-box testing of sentiment analysis systems. This metric serves a dual purpose: evaluating the effectiveness of existing test suites and guiding the generation of new test cases to augment them.

  * Our experimental results demonstrate that enriching test suites using this proposed coverage criterion leads to a significant reduction in model accuracy. This finding has two key implications:

    
  * The coverage criterion is a valid metric for assessing test suites in this context, as higher coverage scores correlate with a greater ability to uncover faults and decrease model accuracy.

  * The coverage criterion can be effectively utilized to generate effective test cases that reveal failures and thus improve the overall quality of test suites.




The remainder of this paper is organized as follows. We explain the proposed methodology in Section <ref>, followed by a presentation of the experimental results in Section <ref>.  Section <ref> concludes the paper by summarizing our findings and future work.




§ METHODOLOGY


The performance of a neural network is typically evaluated by calculating its accuracy on a test suite. However, accuracy alone is not a sufficient measure of model quality. The effectiveness of testing and the reliability of the results heavily depend on the adequacy of the test suite. Coverage criteria are metrics used to assess how effectively a test suite exercises different aspects of a system under test. In traditional software testing, common criteria include statement, branch, and path coverage <cit.>. However, these traditional criteria cannot be directly applied to learning models due to their fundamentally different architecture <cit.>. To overcome this challenge, we propose a set of black-box metrics for evaluating test suites specifically designed for sentiment analysis tasks.

Our approach originates from Input Space Partitioning (ISP), a black-box software testing technique. ISP involves dividing a system's input domain into distinct subsets or partitions, where each partition represents inputs expected to result in similar system behavior. Test cases are then strategically selected from each partition to ensure comprehensive coverage of the input space.



 §.§ Input Space Partitioning



The primary objective of this study is to assess the effectiveness of utilizing the input space partitioning technique to evaluate test suites for emotional sentiment analysis in Transformer-based neural networks. This technique utilizes the features of the input data to partition the input space into distinct regions, facilitating organization and understanding of the data. <cit.>

The architecture of the neural network model and the nature of the input data require the identification and selection of appropriate features for input space partitioning. Effective evaluation of a test suite requires developing a feature space that covers a wide range of scenarios and real-world instances. 
An effective feature space can be established by adherence to the guidelines proposed in Cheng et al.'s study <cit.>, which outlines specific autonomous driving scenarios that the test cases should encompass and which will be defined subsequently. 



To establish a clear foundation for the framework discussed in this study, we introduce the following notations. These notations are essential to understanding the subsequent mathematical definitions and analyses.



  * Input Domain:
The input domain includes all textual inputs the neural network can receive. All text variations the network must process and respond to are included.
    
    D = {d_1, d_2, …, d_n}

where d_i represents an individual input text.

  * Partition:
In order to understand the existing components and assess the test suite, it is crucial to establish specific partitions and map the input domain within those defined partitions. In this study, partitions are defined as follows:

    P = {P_1, P_2, …, P_k}

    where P_i ⊆ D and P_i ∩ P_j = ∅ for i ≠ j, and ⋃_i=1^k P_i = D.
    

  * Features:
Through the analysis of the input domain and the consideration of the functionality of the system under test (SUT), we can determine the important features that will be used to establish partitions.

    F = {F_1, F_2, …, F_m}

    

  * Test Suite: Representative tests should be selected from each partition to form a test suite.
    
    T = {t_1, t_2, …, t_l}

    where t_i ∈ P_j for some P_j ∈ P.





 §.§ Emotional Features


Input space partitioning relies on specific features to establish partitions, crucial for evaluating test suite effectiveness in neural networks. The upcoming section will outline the specific features employed in this study.

The meaning and sentiment of a sentence can be identified through crucial components such as verbs, adverbs, adjectives, and other linguistic elements. Therefore, features are determined according to these important elements of a sentence.

Verbs strongly influence a sentence's sentiment. For instance, in "She laughed when I saw her," the verb "laughed" implies joy, highlighting its importance in sentiment analysis.

Besides the verb, the adverb in a sentence is another influential feature affecting its sentiment. For example, in "He bites his nails nervously," the adverb "nervously" alone can convey fear or anxiety, making its presence significant for sentence categorization.

Adjectives shape a sentence's emotional tone by describing specific feelings. For example, "She received a joyful gift" conveys happiness, whereas "She received a disappointing gift" indicates sadness.

Nouns contribute significantly to a sentence's emotion. In "The old man sat alone, his eyes filled with tears, mourning the loss of his dearest friend," nouns like "tears" and "loss" contribute to its "sad" emotional label.

In this research, the emotional attributes of nouns were integrated as an additional feature alongside verbs, adjectives, and adverbs.
The sets of features presented in this paper are denoted as:



    F = {S_V, S_ADJ, S_ADV, S_N}


where:

    
  * S_V represents sentiment scores for verbs, categorized into emotional labels: 
    S_V : V →{joy, anger, sadness, fear, surprise}

    
  * S_ADJ represents sentiment scores for adjectives,  categorized into emotional labels: 
    S_ADJ : ADJ →{joy, anger, sadness, fear, surprise}

    
  * S_ADV represents sentiment scores for adverbs,  categorized into emotional labels: 
    S_ADV : ADV →{joy, anger, sadness, fear, surprise}

    
  * S_N represents sentiment scores for nouns, categorized into emotional labels, including a neutral category: 
    S_N : N →{joy, anger, sadness, fear, surprise, neutral}



When dealing with multiple features and their numerous potential values, the number of possible combinations can grow exponentially. This can make generating comprehensive input data, encompassing all features at once, extremely complex. k-projection coverage is a technique used in input space partitioning to overcome this challenge. It aims to systematically explore the input space by focusing on a subset of `k' parameters at a time, while keeping the remaining parameters fixed.

To illustrate this method, consider an example of four features {F_1, F_2, F_3, F_4}, each taking two values {0, 1}. resulting in a total of 2^4 = 16 possible combinations.
Applying k-projection coverage with different values of k results in the following:

    
  * k = 1 (Single Feature Projection): We focus on one feature at a time, keeping the others fixed.

    
  * Number of projections: 4 (one for each feature)
    
  * Number of cases per projection: 2 (the two possible values of the feature)


  * k = 2 (Pairwise Projection): We focus on pairs of features, keeping the other two fixed.

    
  * Number of projections: 6 (combinations of 4 features taken 2 at a time)

  * Number of cases per projection: 4 (2 values for each of the two features)


  * k = 3 (Three-way Projection): We focus on groups of three features, keeping the remaining one fixed.

    
  * Number of projections: 4 (combinations of 4 features taken 3 at a time)

  * Number of cases per projection: 8 (2 values for each of the three features)


  * k = 4 (All Features Projection): We consider all features simultaneously.

    
  * Number of projections: 1

  * Number of cases per projection: 16 (all possible combinations)




 Expanding on the previous example, this approach systematically selects sets of k features and strives to achieve comprehensive coverage within a newly formed reduced dimension. This technique of dimensionality reduction enables efficient test generation, as it requires fewer test cases to achieve complete coverage of the input space.
 
 Generally, to calculate the coverage rate for k = n, where n is the number of problem features, the following formula is employed:


    COV = C(D)/α^n

In the above equation, C(D) represents the coverage rate of the test data over possible states, where α is the range of each feature and n is the problem's dimensionality. In our context, α=6 and n=4, requiring 1296 unique test data points for full coverage. This approach, however, is computationally demanding and requires substantial test data.


 Alternatively, when considering the effect of k, the following formula is used:


    COV = C(D)/nkα^k


 Considering the inverse correlation of the denominator, it is evident that the denominator, which is affected by variables such as k, α, and n, greatly decreases the number of tests needed to achieve complete coverage. 

 Incorporating all four features in sentences can be challenging and costly, reducing sentence clarity. Thus, this approach is ineffective. In order to tackle this issue, the dimensions of the feature coverage space are reduced, with the variable k being set to values of 2 and 3 in this study.The values for the variable k are compared in Section <ref> to evaluate their individual effects.


After measuring the coverage achieved by the test suite to assess the effectiveness of the k-projection metric, we attempt to increase coverage by generating test cases for uncovered scenarios. Ultimately, we explore whether improving coverage aids in the detection of hidden flaws in existing sentiment analysis networks.



 §.§ Evaluation of the Coverage Criteria

Our approach to evaluating the proposed coverage criterion involves examining whether increasing a test suite's coverage score uncovers more failures in the sentiment analysis model. We introduce a method for augmenting test suites with additional test cases specifically designed to improve their coverage score. This evaluation method has two key advantages:

    
  * Validation of Coverage Criteria: Observing a decrease in model accuracy when using the augmented test suite confirms the effectiveness of the coverage criteria for evaluating sentiment analysis test suites. A higher coverage score indicates a greater ability to identify faults and lower model accuracy, thus validating the criteria.
    
  * Practical Test Case Generation: The proposed method offers practitioners a valuable tool for generating effective test cases in sentiment analysis tasks. By focusing on increasing coverage, practitioners can systematically identify and address potential weaknesses in their models.


To uncover unexplored regions within the test suite based on our proposed coverage criteria, we propose a strategy for generating novel sentences. Manual sentence creation, while a viable method for producing desired test cases, is both time-intensive and unsuitable for generating large quantities.

Our idea in this study is to use Large Language Models (LLMs) <cit.> to generate the required sentences. LLMs require training using huge quantities of textual data, which allows them to imitate human-like text production. Examples of such LLMs include GPT-3 <cit.> and LaMDA <cit.>. In this study, we employed Claude 3 Opus <cit.>, an LLM produced by Anthropic, which is recognized for its exceptional sentence generation capabilities.

When utilizing LLMs for test generation, explicit prompts play a crucial role in guiding the model to produce specific types of test data that align with the research objectives. By evaluating the datasets using defined features in <ref>, it is apparent that this dataset does not contain inputs with mixed emotions. For instance, after analyzing the test suites, we identified that the test suite lacks sentences where a verb is labeled with sadness and an adverb is labeled with joy. We generate such a test case using the following formulaic approach:

Let S represent the set of all sentences in the test suite, and define subsets V and A as follow:

    V = {s ∈ S |verb(s) = sadness}


    A = {s ∈ S |adverb(s) = joy}

The required test case would be s' ∈ V ∩ A. To generate the missing test case s', we utilize:

    s' = LLM("Generate a sentence with a verb labeled as sadness


    and an adverb labeled as joy")


This approach ensures that the generated sentence s' fills the identified gap in the test suite, thereby increasing the coverage of the test suite according to our proposed criteria. By adding s' to S, we obtain an augmented test suite S':

    S' = S ∪{s'}


The augmented test suite S' now includes sentences that exhibit the desired feature combinations, leading to a more comprehensive evaluation of the model's capabilities. Additionally, as stated in the evaluation report, these types of situations might occur in user experiences and are viewed as real-life scenarios <cit.>. Therefore, it is crucial to include these generated tests in the test set that showcases these real-life, uncovered scenarios.


For the purpose of generating test cases, we selected the Claude language model <cit.> due to its capacity to generate varied and innovative test sets that are in line with our research goals. This model's attributes enable the generation of test cases that meet research requirements and highlight the robustness and reliability of sentiment analysis.

The generated sentences lack an emotion label, making them unsuitable for use as a test. In order to assign labels to the generated tests, we utilized six of the most accurate sentiment analysis models that were identified in the research literature <cit.>. with their accuracy mentioned in table <ref>. The input sentences were fed into these models for labeling, and the resulting labels were multiplied by their accuracy on the dataset (coefficient). The sentiment with the highest score was assigned as the label for each generated sentence, a method known as “Differential Testing” <cit.>. The system under test was then evaluated using a combination of original and newly generated tests to assess its accuracy on the updated test set.




§ EXPERIMENTAL RESULTS AND DISCUSSION








The dataset utilized for assessing the results is denoted as CARER <cit.>. This dataset contains six primary emotions expressed through text, enabling a thorough assessment of the neural network's ability to accurately classify emotions.
The dataset is split into training and testing sets. The training set comprises 16,000 tweets, each associated with six output labels. Similarly, the test set consists of 2,000 tweets, each labeled with six output categories. A total of 18,000 tweets are chosen as the test set and assessed using the proposed coverage evaluation method. For this study, the dataset is divided into smaller subsets, each consisting of 200 tweets, in order to analyze the coverage capability of each subset. Additional tests will be set based on the uncovered features.

Our model is compared to the existing test dataset since, based on our findings, no other model conducts test generation for neural networks in text categorization in the same way. This study applied the Claude language model <cit.> for test generation and 6 sentiment analysis models for differential testing <cit.>. A distinct arrangement of the DISTILBERT model <cit.> was employed for word-level analysis.The process of learning and loading was executed on Google Colab, utilizing a central processing unit (CPU) with a capacity of 12.7 gigabytes of random access memory (RAM).

We tried different values of k in our implementation to evaluate the effect of k in our approach.
The results presented in Fig. <ref> illustrate the differences in coverage among various implementations of k. When k=2, the coverage increases on average by 14% (Fig. <ref>). However, when k=3, the average increase in coverage is 18% (Fig. <ref>).
The initial coverage for k=3 is 32%, while the initial coverage for k=2 is 62%. This indicates that the test suite lacks sufficient coverage for complicated sentences that involve three combinations of features. Although full coverage was not attained, the sentences generated noticeably raised the total test coverage in comparison to the original data. 

Figure <ref> demonstrates a decline in accuracy for both test generation implementations, suggesting that the generated tests expose previously unexplored areas and have the potential to improve the model's functionality.
When k=2 (Fig. <ref>), the accuracy experiences an average loss of 5%, whereas for k=3 (Fig. <ref>), there is an average decrease of 8%. These findings indicate that the tests created for k=3 demonstrate a higher number of defects in the model's decision-making capabilities.

The average accuracy after test generation for k=3 models is 85%, whereas for k=2 models it is 88%, which is lower than the model's accuracy on the base test suite, which is 94%. This indicates a decrease in the model's precision when tested on these more challenging data points. This drop results from intentionally pushing the model to its limits in order to uncover areas for enhancement.
Based on the experimental results for  k = 3 the augmented test suite exhibits a greater decrease in accuracy compared to  k = 2. This can be attributed primarily to the production of more intricate sentence structures. Nevertheless, when k = 3, there is a greater time complexity and increased costs involved in generating these sentences. The sentences produced for k = 3 exhibit a significant level of complexity and deviate considerably from real-life situations, thereby making it challenging to evaluate the model's performance in practical settings.




§ CONCLUSION AND FUTURE WORKS

This study introduced novel coverage criteria for evaluating test suites of transformer-based sentiment analysis neural networks. By using input space partitioning focused on emotionally relevant linguistic features and the k-projection coverage metric, we efficiently developed test cases and analyzed key feature subsets.
Our methodology demonstrated a 16% average increase in test coverage for all methods and a 6.5% decrease in model accuracy on augmented tests. 
The proposed criteria balance bug detection capability and computational efficiency, surpassing existing techniques. By emphasizing emotional dimensions through input space partitioning, our approach ensures thorough evaluation across diverse real-world scenarios.

Future work could extend our coverage criteria to other transformer-based NLP tasks like text summarization or machine translation. Expanding linguistic elements beyond verbs, adjectives, adverbs, and nouns could improve the detection of subtle emotional nuances, including negations and emojis. Finally, investigating the scalability and efficiency of our methodology on larger datasets and wider applications will be essential for real-world, industry-scale applications.

IEEEtran
 


 Paper Title: Are Bigger Encoders Always Better in Vision Large Models?.








Are Bigger Encoders Always Better in Vision Large Models?
    Bozhou Li^†, Hao Liang^†, Zimo Meng^†, Wentao Zhang

Peking University


    August 5, 2024
============================================================================

§ ABSTRACT 

In recent years, multimodal large language models (MLLMs) have shown strong potential in real-world applications. They are developing rapidly due to their remarkable ability to comprehend multimodal information and their inherent powerful cognitive and reasoning capabilities. Among MLLMs, vision language models (VLM) stand out for their ability to understand vision information. However, the scaling trend of VLMs under the current mainstream paradigm has not been extensively studied. Whether we can achieve better performance by training even larger models is still unclear. To address this issue, we conducted experiments on the pretraining stage of MLLMs. We conduct our experiment using different encoder sizes and large language model (LLM) sizes. Our findings indicate that merely increasing the size of encoders does not necessarily enhance the performance of VLMs. Moreover, we analyzed the effects of LLM backbone parameter size and data quality on the pretraining outcomes. Additionally, we explored the differences in scaling laws between LLMs and VLMs.



#1#2
  [#1]#2

1†Equal Contribution



§ INTRODUCTION

In recent years, the rapid development of large language models has revolutionized the field of natural language processing  <cit.>. These models, equipped with an enormous number of parameters, have demonstrated outstanding performance in areas such as translation  <cit.>, question answering  <cit.>, and text generation  <cit.>. Due to the powerful capabilities of LLMs, a natural idea is to combine them with multimodal research, leveraging the strong cognitive abilities of LLMs to process information from other modalities  <cit.>. 

Among MLLMs, VLMs achieve competitive performance in traditional multimodal tasks such as image classification <cit.>, image understanding <cit.>, and image captioning <cit.>. Moreover, their excellent language understanding capabilities enable strong performance in text-rich tasks, such as vision question-answering <cit.> and image-text retrieval <cit.>.

Currently, the dominant architecture for VLMs employs a large language model as the backbone. Mechanisms such as cross-attention <cit.> or linear projectors <cit.> are utilized to connect the vision encoder (e.g., ViT <cit.>) with the LLM backbone <cit.>. Larger visual encoders possess stronger encoding capabilities, allowing them to extract features more effectively. These features are then transformed by the projectors for the LLMs to understand. We refer to this model architecture as the connected vision paradigm.

Apart from VLMs, previous research on scaling laws <cit.> adopts an empirical approach to study the relationship between model performance, parameter size, and the amount of training data. Under the assumption that the scaling trend follows a power-law relationship, researchers fit power-law formulas by training models of different sizes with varying amounts of data. Additionally, scaling laws have been extended to various scenarios, including graph neural networks <cit.>, data mixing <cit.>, data pruning <cit.>, and fine-tuning LLMs <cit.>. Although <cit.> investigated scaling laws in the context of multimodal scenarios, they neglected the connected vision paradigm <cit.>, leading to the following challenges:

C1. Poor Effectiveness: Since no scaling laws have been conducted, it is uncertain whether this paradigm can scale up to achieve better performance, resulting in suboptimal model performance.

C2. Low Efficiency: Without scaling laws, we do not know how much data or how many parameters to use. Typically, all available data are used for training VLMs, resulting in a waste of data and computational resources.

To address these issues, we conduct scaling law experiments following <cit.>. We select the renowned LLaVA1.5 <cit.> model as the backbone of our study. To investigate the scaling laws, we utilize models with 7 billion and 13 billion parameters. For the dataset, we choose image-text pairs from CC12M <cit.> and Laion400M <cit.>, known for their high-quality pairs. We use data sizes from 1 million to 10 million image-text pairs for our scaling law experiments.





The core contributions of this paper are summarized as follows:

    
  * New Perspective To the best of our knowledge, we are the first to conduct scaling law studies for the connect vision paradigm. Additionally, we are the first to analyze the limitations of this VLM paradigm.

    
  * New Observation We conducted experiments on the pretraining phase of MLLMs using the connect vision paradigm. From the experimental results, we derived a crucial conclusion: simply utilizing a ViT trained using CLIP with more parameters and better performance does not enhance the performance of MLLMs. This suggests that exploring alternative methods is necessary to improve the performance of MLLMs. We also indicate that this issue is highly likely not caused by limitations in the scaling ability of ViT.








§ BACKGROUND AND RELATED WORK



 §.§ Multimodal Large Language Model


In recent years, benefiting from the development of model architectures based on transformers  <cit.> and the increase in computing resources and training data, LLMs  <cit.> have emerged. These models possess massive parameters, requiring extensive computational resources to train on large datasets for a prolonged period. The sheer scale of these models, often exceeding billions of parameters, has enabled them to capture subtle nuances of language and learn complex patterns from vast amounts of text data. LLMs have not only demonstrated outstanding performance in traditional NLP tasks but have also exhibited emergent capabilities such as reasoning, code generation, and solving mathematical problems  <cit.>.

With the fast-paced advancement of LLMs, a natural question arises: how can we enhance the capabilities of LLMs to enable them to understand information from other modalities? Some earlier MLLMs, such as BEiT-3, attempted to train a multimodal model from scratch using data from multiple modalities  <cit.>. However, there is now a growing preference for combining pre-trained LLMs and modality encoders using specific mechanisms to construct multimodal LLMs. The methods combining pre-trained LLMs and modality encoders can be classified into four categories  <cit.>. The first and second categories employ cross-modal attention mechanisms  <cit.> and custom layers  <cit.>, respectively, to integrate information from other modalities into the internal representation of the LLM. These two types of methods are commonly referred to as deep fusion methods. The third category uses different modules to align information from other modalities with text information  <cit.>. Researchers exploring the fourth category attempt to unify the encoding of different modalities and expand the vocabulary of LLMs  <cit.>. These two methods are commonly referred to as early fusion methods. Due to their ease of training, high computational efficiency, and ability to construct any-to-any MLLMs, early fusion methods, especially the third category, which is also the focus of this article, are gradually becoming mainstream.

In general, the fusion modules used in the third category of methods can be classified into three types. The first type is transformer-based abstractors. These methods allow for adjusting the number of visual tokens to balance model performance and efficiency  <cit.>. However, they might lose locality inductive bias and require longer training times and more training data. The second type involves using MLPs (Multi-Layer Perceptrons)  <cit.>. This approach is lightweight, easy to train, and capable of capturing local information. However, it can only generate fixed-length visual tokens. Furthermore, some researchers have explored the use of custom layers to combine the advantages of both approaches  <cit.>.

The training process of  MLLMs can be divided into two stages: multimodal pretraining and multimodal instruction fine-tuning. During the multimodal pretraining (MM PT) stage, the input and output projectors are trained to align different modalities by optimizing predefined objectives. Typically, X-text pairs are used in this stage, where X represents data from other modalities. In the multimodal instruction fine-tuning (MM IT) stage, the pre-trained multimodal LLM is fine-tuned using a dataset formatted in an instructional manner. This fine-tuning aims to enhance the model's generalization ability on unseen tasks. In this article, we conducted experiments on the MM PT stage.







 §.§ Scaling Law


In the process of deep learning development, there has been a tendency to train larger models to achieve better performance. A natural question arises: as the number of model parameters and the amount of training data increase, how does model performance change? This has led to research on scaling laws. Hestness et al.  <cit.> were the pioneers in using empirical methods to study the scaling law of deep neural networks. Kaplan et al.  <cit.> and Hoffmann et al.  <cit.> conducted extensive experiments on transformer-based autoregressive models at different scales. They investigated the relationship between model loss and the number of model parameters and training data.

In situations where the model architecture is known, we can approximate the training cost based on the number of model parameters and the amount of training data (e.g., C ≈ 6ND for the transformer-based model, where N represents the number of model parameters, D represents the amount of training data, and C represents the computational costs). By combining this estimation with scaling laws, we can determine the optimal allocation of training data and model parameters within limited computational resources  <cit.>. This approach is particularly useful for LLMs since training them requires a significant amount of computational resources. Even if we embrace the philosophy underlying LLAMA  <cit.>, which entails setting aside computational expenses during the training phase and maximizing data usage to augment model performance at inference time, we consequently face an additional consideration: using larger models inherently translates to increased computational costs during inference. Therefore, even under conditions of ample data availability, a critical question arises: is the performance boost from expanding model parameter counts justified by the resultant higher computational overhead at inference? Under this assumption, scaling laws are still necessary.

After the initial scaling law was proposed, researchers studied the scaling law in different scenarios, such as graph neural networks  <cit.>, data mixing  <cit.>, data pruning  <cit.>, and fine-tuning of large language models  <cit.>. Some of them made certain adjustments to the form of the formula.

When it comes to MLLMs, Aghajanyan et al.  <cit.> explored the relationship between the loss and the dataset size and the parameter size over seven modalities, including text, image, image-text, speech, speech-text, code, and molecules. They also examined the scaling law when models are pre-trained on two modalities simultaneously and modified the form of the formula. However, the training paradigm they used in their study differs from the connected vision paradigm, the mainstream methods preferred today. They trained LLMs and modality encoders from scratch, similar to the approach chosen by BEiT-3  <cit.>, to obtain their scaling law. This discrepancy has sparked our research focus. To the best of our knowledge, the scaling law of multimodal large models under the connected vision paradigm is still under-researched.











§ METHODS



 §.§ Dataset


We utilized the CC12M dataset  <cit.> in our experiments, a comprehensive and expansive multimodal collection designed to foster advancements in artificial intelligence, particularly focusing on the intersection of vision and language. From the CC12M dataset, we strategically extracted subsets of data ranging from 1 million to 10 million images, incrementing by 1 million each time, to serve as our training datasets. This approach was chosen to systematically investigate the impact of varying training data sizes on the model's performance and learning outcomes. In our experiments involving different variants and scales of the Vision Transformer (ViT) trained by CLIP <cit.>, we maintained the use of identical datasets for each training session. This decision was made to eliminate the influence of random variables that could arise from using different data splits. By controlling for the input data, we aimed to isolate and observe the effects of model architecture and size on the experimental results, providing a clearer understanding of how these factors influence performance.

To validate the generality of our findings, we also conducted a set of experiments using the LAION-400M dataset  <cit.>. Similarly, we constructed training datasets ranging from 1M to 10M by randomly sampling from the LAION-400M dataset.

For each image-text pair in the dataset, we randomly selected a text prompt from the LLaVA paper's text prompts  <cit.> and randomly arranged its order with the image. We used the caption of the image as the ground truth. For each size of the training dataset, we extracted 25,600 of them as validation sets.



 §.§ Model Choice

In our quest to elucidate how different sizes of the ViT influence the performance of multimodal large language models, we deemed it crucial to employ ViTs of various sizes. To ensure that extraneous factors such as disparities in training data distribution or variations in training recipes did not skew our findings, we meticulously opted for a consistent series of ViT models throughout our experiments.

Our choice fell upon the ViT models  <cit.> trained using CLIP on the extensive Laion2B  <cit.> dataset, renowned for its comprehensive coverage and quality, thereby providing a robust baseline for our investigations. The specific model sizes we selected are shown in Table <ref>.





For the LLM backbone, the Vicuna series  <cit.> was selected. This model series was cultivated by fine-tuning the LLaMA2  <cit.> foundational model on a diverse set of user-shared dialogues sourced from ShareGPT  <cit.>. To ensure the reliability of our conclusions, we conducted training on both the 7B model and the 13B model from the Vicuna series.


 §.§ Training Process

In our overarching approach, we adopted the training process from the MM PT stage of LLaVA v1.5  <cit.>. Specifically, a two-layer Multi-Layer Perceptron (MLP) served as the linear projector within our framework, utilizing the Gaussian Error Linear Units (GELU)  <cit.> as the activation function to enhance non-linearity. Throughout the training phase, we froze the parameters of the ViT and LLM backbone and only updated the Linear Projector. 

We conducted the MM PT stage of the model using next token prediction as the training task and cross-entropy as the loss function. During the training process, we only masked the textual data in the ground truth and did not mask the image tokens.

Across a spectrum of dataset sizes, ranging from smaller to larger volumes, and varying the scale of ViT parameters and the size of the Vicuna model parameters, we trained the linear projector from scratch with a cosine scheduler. All experiments were conducted on 8*A100/A800 NVIDIA GPU machines.


§ EXPERIMENTS RESULTS

Tables  <ref> and  <ref> exhibit the concluding evaluation loss values obtained when employing various dataset sizes and ViT model sizes and training datasets built from CC12M. To render these outcomes more comprehensible at a glance, we have also produced Figures <ref>  and  <ref>. These visual aids aim to depict the relationship between model size, dataset size, and training loss, making the trends and patterns easier to discern. Table <ref> and Figure <ref> show the results with datasets built from LAION-400M.

Drawing insights from our experimental findings, several noteworthy observations emerge:

    
  * Increasing Data Quantity Improves Model Performance
    
    It is observed that for smaller training datasets, augmenting the data quantity effectively reduces the evaluation loss. This aligns with the learning theory that more examples lead to better model performance.
    
  * Larger LLM Backbone Enhances Model Performance 
    
    Comparing the training outcomes on the Vicuna-7B and Vicuna-13B models reveals that larger models consistently present lower evaluation loss than their smaller versions. This implies that larger language models possess a heightened capability to understand and interpret vision tokens.
    
  * The Importance of High-Quality Data
    
    The evaluation loss obtained from training on the CC12M dataset is lower than the evaluation loss from training on the LAION-400M dataset. The ViT models used in the experiment were trained on the LAION-2B dataset. Intuitively, its distribution should be closer to LAION-400M, and training multimodal LLMs with LAION-400M would yield better results. However, experimental results show that using CC12M is more effective. This is likely due to the quality of the datasets. Compared to LAION-400M, the CC12M dataset has higher quality, with a higher degree of matching between image and text. This result emphasizes the importance of high-quality datasets when training MLLMs.
    
  * Larger LLM Backbone Requires Less Training Data 
    
    Notably, when leveraging the Vicuna-13B model, the evaluation loss plateaus around the 7M mark in training data size, signifying a diminishing return on additional data. Conversely, the Vicuna-7B model continues to benefit from an increased data volume, illustrating a more gradual learning curve. This indicates that smaller models might require more substantial data increments to observe subsequent improvements, while larger models can exhibit more pronounced progress with less data.
    
  * Improved ViT Performance Doesn't Guarantee Better Results
    
    A critical finding from our study reveals that merely amplifying the parameter scale of ViT does not necessarily translate into enhanced model performance. Although vision transformers exhibit a relatively lesser scaling capacity compared with text transformers  <cit.>, the parameter scales selected in our experiments are significantly removed from the upper limits of ViT's scaling capabilities.

The 632M ViT trained using CLIP achieves a zero-shot accuracy of 78.0% on the ImageNet-1K dataset  <cit.> while increasing the model parameter size to 1.01B and training on a larger amount of data results in a zero-shot accuracy leap to 78.4%  <cit.>. Moreover, through architectural advancements, researchers have demonstrated ViT models boasting accuracies above 90%, with parameter counts around 20B  <cit.>. This underscores that within our experimental parameters, ViT's scaling potential has not been saturated, with enhanced performance still achievable through augmented model dimensions.

However, within our specific scenario, increasing the ViT's size does not yield superior results and, in certain instances, may lead to performance degradation. When training with the CC12M dataset, the validation loss of multimodal LLMs constructed with the 1.01B CLIP model's ViT is lower than that of models using the 632M model only when the training data volume is large. However, when training with data from the LAION-400M dataset, the validation loss of models using the 1.01B model consistently remains lower than that of models using the 632M model. This suggests that additional factors are at play, hindering the alignment and effectiveness between ViT and LLM in MM PT stage, necessitating deeper investigation.






















§ DISCUSSION & CONCLUSION

Our experimental outcomes underscore a critical observation: within the LLaVA  <cit.> framework, the simplistic augmentation of ViT's parameter scale does not correspondingly elevate the performance of MLLMs. This finding necessitates a strategic shift in our approaches to multimodal model design and optimization.

    
  * Strengthen Data Alignment Across Modalities With Data-centric Approaches
        
The experimental insights suggest the importance of improving the quality of training data for the alignment of different modalities. In this pursuit, data-centric approaches appear to hold significant promise. By focusing on enhancing data quality, variety, and relevance, we can potentially achieve better alignment between visual and textual modalities with higher efficiency, potentially leading to superior performance in multimodal applications  <cit.>.
        
    
  * Investigate The Differences Between Various Alignment Methods
    
    Deep Dive into Underlying Performance Limitations: A comprehensive investigation into why ViT performance enhancements do not translate into superior multimodal model performance is warranted. This exploration should encompass the dynamics of data utilization, model architecture, and training methodologies. When comparing the training approaches of CLIP  <cit.> and LLaVA  <cit.> we can observe certain differences in alignment strategies. The popular CLIP model utilizes a decoder-only LLM as the text encoder, such as GPT-2  <cit.>. In CLIP, the CLS token of the image processed by ViT is aligned with the EOS token of the text processed by GPT-2. In contrast, the processing approach of LLaVA is more akin to mapping the image tokens processed by ViT to the embedding space of a LLM using MLP layers. This differs from aligning image tokens with text tokens processed by GPT-2, as done in CLIP. The impact of these alignment method differences on model performance is worth exploring.


    
  * Exploration of Multimodal Information Fusion
    
        
  * Architectural Innovations for Multimodal Integration: There is a need to explore novel architectures that facilitate more seamless integration of multimodal information  <cit.>. This could involve designing frameworks that better enable the exchange and processing of visual, textual, and other modal data, potentially leading to enhanced overall performance.
        
  * Expansion of Vocabulary: Considering the inclusion of techniques like AnyGPT <cit.> to extend the vocabulary can prove beneficial in enriching the model's understanding and processing capabilities across different modalities. This could potentially enable the model to capture and utilize a broader spectrum of information, thereby enhancing its performance.
    


The observed phenomenon of larger LLMs requiring smaller datasets for alignment with ViT suggests a closer alignment between the semantic spaces of vision and text. This insight not only enriches our understanding of multimodal model dynamics but also opens avenues for optimizing data requirements and training strategies for MLLMs. Future research should delve deeper into the mechanisms behind these observations, aiming to leverage the inherent semantic alignment capabilities of larger LLMs for enhanced performance and efficiency in multimodal applications.



§ LIMITATIONS AND FUTURE WORK

Despite our study encompassing an extensive series of one hundred and twenty experiments across various parameter combinations, the investigation into the scaling laws of MLLMs during the MM PT stage remains a vast and complex field. Our current exploration, while insightful, is not exhaustive in addressing the comprehensive nature of this topic.

Compared to the scaling law of LLMs  <cit.>, the scaling law of MLLMs exhibits more complex characteristics.

    
  * More Influencing Factors
    
    More factors influence the performance of MLLMs, particularly when using pre-trained modality encoders and an LLM backbone to build MLLMs, the paradigm that is currently widely used. The parameter size of modality encoders, the amount of data used for pretraining the modality encoders and LLM backbone, the quality of data during the MM PT stage, and the distribution discrepancy between the data used for MM PT and the data used for pretraining the modality encoders and LLM backbone can potentially influence model performance. These factors may also interact with each other in more complex ways. In addition, different modalities, fusion methods for integrating multimodal information, and architectures of modality encoders may exhibit different forms of scaling law.
    
  * Different Estimates of Computational Costs
    
    Under the current training framework, it is not appropriate to solely rely on C≈ 6ND  for estimating computational costs. Due to the utilization of the backpropagation algorithm to update the model parameters, even if we only update the parameters of the MLP layer, which is quite lightweight, we still need to compute gradients for a significant portion of the LLM parameters. In addition, the modality encoder only needs to process data from other modalities, while the LLM also needs to handle text tokens. Taking LLaVA as an example, the compute budget for the MM PT stage of LLaVA can be estimated as:
    
    C ≈ 2N_ViTD_img+6N_LLM(D_img+D_txt)

    where N_LLM and N_ViT represent the number of parameters of LLM and ViT, and D_img and D_txt represent the number of image tokens and text tokens, respectively. The MLP layer is overlooked due to its relatively small number of parameters. It is worth noting that the value of D_img is dependent on the architecture of ViT, while the ratio between D_txt and D_img is also dependent on the source of the image-text pairs. For other fusion methods and modality encoder architectures, this formula may have different forms. This introduces new challenges in making trade-offs between computational costs and model performance.


To firmly substantiate the conclusions drawn in this paper, a more extensive empirical foundation is required. This necessitates conducting additional experiments on a broader range of LLM families, encompassing diverse architectures and capabilities. Moreover, the exploration should be extended to incorporate a wider variety of datasets—both in terms of content and scale—to ensure a comprehensive understanding of how multimodal models behave under different conditions.

Beyond empirical evidence, the surprising outcomes observed in our experiments demand a theoretical underpinning that goes beyond mere data analysis. The unexpected findings, such as the non-intuitive scaling behaviors of multimodal models or the peculiar interactions between vision and language semantic spaces, call for a deeper level of understanding that can only be achieved through rigorous theoretical analysis.

The prohibitive cost associated with extensive experimental setups has constrained our investigation to a specific scenario: the performance of large models where both ViT and LLM components are frozen. This approach, while providing valuable insights into the behavior of multimodal models under resource constraints, deviates significantly from the methodologies employed by some mainstream multimodal models such as Qwen-vl  <cit.>.

Our current experimental endeavors have been confined to the LLaVA architecture, providing a focused exploration of scaling properties within this specific framework. However, the landscape of multimodal AI architectures extends far beyond LLaVA, encompassing a diverse array of designs that include transformer-based connector architectures  <cit.> and VAE-based vision encoder architectures  <cit.>. Each of these architectural paradigms embodies unique characteristics and potential scaling behaviors that require systematic investigation.

Our experiments only considered MLLMs that can understand different modalities but can only generate textual information. We did not take into account more powerful yet complex MLLMs that can generate content in other modalities. The study of the scaling behavior of such models is more complex and requires additional computational resources.



unsrtnat



 Paper Title: Zero-Shot Cross-Domain Dialogue State Tracking via Dual Low-Rank Adaptation.


Black bounces in Cotton gravity
    Henrique A. Vieira
    August 5, 2024
===============================

§ ABSTRACT 




Zero-shot dialogue state tracking (DST) seeks to enable dialogue systems to transition to unfamiliar domains without manual annotation or extensive retraining. Prior research has approached this objective by embedding prompts into language models (LMs). Common methodologies include integrating prompts at the input layer or introducing learnable variables at each transformer layer. Nonetheless, each strategy exhibits inherent limitations. Prompts integrated at the input layer risk underutilization, with their impact potentially diminishing across successive transformer layers. Conversely, the addition of learnable variables to each layer can complicate the training process and increase inference latency. To tackle the issues mentioned above, this paper proposes Dual Low-Rank Adaptation (DualLoRA), a plug-and-play architecture designed for zero-shot DST. DualLoRA incorporates two distinct Low-Rank Adaptation (LoRA) components, targeting both dialogue context processing and prompt optimization, to ensure the comprehensive influence of prompts throughout the transformer model layers. This is achieved without incurring additional inference latency, showcasing an efficient integration into existing architectures. Through rigorous evaluation on the MultiWOZ and SGD datasets, DualLoRA demonstrates notable improvements across multiple domains, outperforming traditional baseline methods in zero-shot settings. Our code is accessible at: <https://github.com/suntea233/DualLoRA>.





 




§ INTRODUCTION








Task-oriented dialogue (TOD) systems, designed for specific tasks like restaurant reservations or travel bookings, rely on dialogue state tracking (DST) to interpret user intents as slot-value pairs across dialogue turns, essential for managing multi-domain conversations <cit.>. Figure <ref> shows the change of dialogue states in a multi-turn dialogue crossing multiple domains.  An ideal system seamlessly transitions to new domains with minimal training, overcoming the resource-intensive process of data collection and annotation for new domains. Zero-shot learning emerges as a solution, enabling TOD systems to adapt to unseen domains by leveraging existing knowledge, thus bypassing the need for extensive domain-specific data.






































The advancement in language models (LMs) <cit.> has significantly enhanced the exploration of zero-shot learning approaches within the context of dialogue state tracking tasks. Adaptation to novel domains is frequently achieved through the incorporation of prompts into transformer-based models <cit.>. The current method can be classified into two categories.

The first type of methods incorporate prompts, e.g. slot description, in the first transformer layer <cit.>. A common practice within this approach involves the concatenation of prompts subsequent to the input, as shown in Fig <ref>. Despite the straightforward nature of their implementation, these strategies exhibit suboptimal utilization of prompts. This refers to the phenomenon where the influence of prompts on the model's output progressively diminishes with each additional transformer layer. 
Figure <ref> shows an example, where deep transformer layers, compared with shallow transformer layers, exhibit a reduced allocation of attention to other tokens within the dialogue context or prompt, in the token encoding process.  This issue is further exacerbated in the context of dialogue state tracking, where the dialogue's progression causes a rapid escalation in the context length — often extending to 300-400 tokens or more, in stark contrast to the prompts' modest length of 10-20 tokens. Such a diminishing effect of prompts can severely impair the system's performance in zero-shot learning scenarios, where the ability to discern and leverage domain-specific knowledge becomes critically important.












The second category of methodologies incorporates a learnable vector, initialized by prompts, into each layer of the transformer architecture <cit.>, as shown in Fig <ref>. This technique facilitates the model's capacity to consistently engage with the prompt information across all layers, ensuring the prompt's considerations are integrated throughout the encoding process. However, these methods present certain limitations. Direct concatenation of prompts at each layer might introduce extraneous noise, adversely affecting the efficacy of model training. Furthermore, this method of concatenation can result in augmented inference latency of the model. Consequently, devising a strategy to effectively utilize prompts without impeding the standard input-output dynamics of the model and without exacerbating inference time emerges as a critical challenge in the field.






To address the aforementioned challenges, we propose a novel plug-and-play framework designed for zero-shot dialogue state tracking, termed Dual Low-Rank Adaptation (DualLoRA). This architecture is characterized by the integration of two Low-Rank Adaptation (LoRA) components: one dedicated to processing dialogue context and the other to refining prompts, as shown in Fig <ref>. DualLoRA operates by receiving prompts and applying modifications within the attention layers of the transformer model, subsequently integrating this modified output with the transformer's original output. Through the implementation of this framework across each layer, DualLoRA ensures that the influence of the prompts is perpetuated throughout the entire depth of the model. Notably, this architecture does not incur additional latency during the inference phase, maintaining efficiency in model performance. Furthermore, DualLoRA requires few parameter adjustments and demonstrates exceptional transferability across different domains. 




Comprehensive experimental evaluations were performed utilizing the MultiWOZ and SGD datasets to assess the efficacy of the proposed methodology. Comparative analyses reveal that our approach yields enhanced Joint Goal Accuracy (JGA) metrics across various domains within both the MultiWOZ and SGD datasets, relative to established baseline methodologies.

The structure of this paper is organized as follows: Section 2 offers an overview of the literature pertinent to this research area. Section 3 delineates a detailed exposition of our proposed model. Section 4 presents a detailed account of the experimental outcomes and the specifics of the implementation. Section 5 encapsulates the conclusions derived from this study.





§ RELATED WORK



 §.§ Dialogue State Tracking

Recent works on dialogue state tracking have mainly approached the task in two ways. One approach treats the task as a classification task, utilizing an encoder to classify or extract dialogue states. Such as <cit.>, which employs BERT or LSTM or GRU or RNN as the context encoder for the dialogue, extracting slot values from the semantic context. However, since these methods are trained on a large amount of annotated data across multiple domains, the performance tends to be poor when dealing with unseen domains, indicating limited generalization. The other approaches treat the task as a generative task, employing a Seq2Seq approach, such as the T5 <cit.> and GPT <cit.> model. Such as <cit.>, which captures cross-slot shared information by combining slot-specific information descriptions, aiming to achieve knowledge transfer. Furthermore, many studies concentrate on modeling dependencies among slot values to capture information across different domains, while some also utilize large language models and employ methods involving the assembly of domain-slot values for instruction tuning to address the task of dialogue state tracking. These methods allow the model to extend to new domains effectively through autoregressive training, demonstrating better generalization. In this paper, our primary focus lies in zero-shot dialogue state tracking, which involves validating and testing the model in unseen domains after training on visible domains, mainly emphasizing the model's generalization.




 §.§ Parameter-Efficient Transfer-Learning

With the rapid advancement of large language models, utilizing their capabilities on limited resources has become an urgent problem that needs to be addressed. The current mainstream approach involves parameter-efficient transfer-learning (PETL). This method entails introducing adjustable parameters to a large language model and, during training, freezing the large language model while adjusting only a few parameters, thereby significantly reducing the trainable parameters <cit.>. In addition to efficiently reducing trainable parameters without compromising model performance, research indicates that parameter-efficient transfer-learning can help avoid overfitting due to the introduction of a few parameters. Moreover, there are many studies applying PETL to dialogue state tracking <cit.>. Such as <cit.>, which based on prefix-tuning, introduces a method that dynamically generates prefixes. Besides, <cit.> proposed continual prompt tuning, a parameter-efficient framework that can avoid forgetting and facilitate knowledge transfer between tasks. 
Some research also utilizes large language models with parameter-efficient transfer-learning (PETL) to address dialogue state tracking tasks <cit.>. However, since DST is just one crucial part of task-oriented dialogue systems, a weighty component in any part can make it cumbersome and impact overall system responsiveness. This paper proposes a new structure called DualLoRA based on LoRAs for zero-shot dialogue state tracking aiming to fine-tune the model with few parameters and without introducing inference latency.




§ DUAL LOW-RANK ADAPTATION

This paper proposes DualLoRA, a plug-and-play architecture for dialogue state tracking. DualLoRA consists of a context LoRA and a prompt LoRA. The context LoRA is used to optimize the original dialogue context input, while the prompt LoRA is used to optimize the prompt input.
Both the context LoRA and prompt LoRA are based on LoRA. By employing two distinct LoRAs, the task of zero-shot learning can be divided into two subtasks: tuning the original input and tuning the prompt input. Additionally, to avoid impacting the model's inference latency, the parameters of context LoRA and prompt LoRA can be merged into the pre-trained parameters, which ensures the efficiency of DualLoRA.





 §.§ Context LoRA
 


Adopting a model architecture that minimizes parameter count is crucial for facilitating efficient transfer across domains while maintaining robust performance. Inspired by <cit.>, we leverage LoRA with the T5 model to achieve cross-domain transfer in the zero-shot dialogue state task because of LoRA's excellent transferability and its ability to reduce the number of trainable parameters without compromising the model performance.

For context LoRA, we aim to ensure that the model focuses on the dialogue context input in this component. We use a few parameters to tune the model's adaptation to the source domain. By doing so, we can obtain a LoRA that possesses knowledge from the source domain and is easily transferable. 

The context LoRA introduces a bypass structure, enabling the model to fine-tune downstream data without affecting the pre-trained parameters. Context LoRA consists of two matrices: matrix A is initialized through a random Gaussian distribution, and matrix B is initialized with zeros. 
Due to the lower intrinsic dimensionality of pre-trained language models, low-rank decomposition can be applied to constrain the updates of Δ W for the pre-trained weight matrix W, so the updates for W can be formulated as W + Δ W = W + BA, where A ∈ℝ^d × r, B ∈ℝ^r × d and rank r ≪ d. 
If the input h ∈ℝ^d, then the output h^' can be represented with context LoRA as:



    h^' = Wh + Δ Wh = Wh + BAh



 




 §.§ Prompt LoRA
 

 

To effectively differentiate and integrate information from the dialogue context and prompts, we propose the incorporation of an additional component to handle the input prompt. Compared with dialogue context, changes in prompts may lead to more significant variations in slot embeddings and hinder the model's adaptability. To handle the variations in the slot embeddings, we use slot prompt <cit.>, rather than slot embeddings, as the input. 

However, slot prompt introduces noise in the early stage of the training process. To mitigate the noise, we employ the LoRA as the structure for slot prompt tuning, making it a zero-initialized slot prompt. Prompt LoRA consists of two matrices A_p ∈ℝ^d × r and B_p ∈ℝ^r × d. The rank of the two matrices stays the same with the context LoRA. A_p is initialized as Gaussian distribution, while B_p is initialized as zeros. Through zero-vector initialization, when integrating slot prompts, it initially preserves the original knowledge from the pre-trained model. It gradually incorporates prompt-guided signals during the training process, which aids in achieving stable learning during fine-tuning and enhances the model's ability to adhere to prompts in the zero-shot stages.




















To train the original input and prompts simultaneously, we combine the context LoRA with the prompt LoRA, resulting in a comprehensive structure that simultaneously accepts prompts p and original inputs h. The output h^' can be represented as: 


    h^' = Wh + BAh + B_pA_pp


where p is the slot prompt for prompt LoRA.

Avoiding inference latency is also a noteworthy concern during the inference stage. For the context LoRA, we can directly merge the B and A matrix from the context LoRA into the original pre-trained weights. 

    Wh + BAh = ( W + BA)h

For the prompt LoRA, as its input is different from the context LoRA, it cannot be directly merged into the original pre-trained weights. Therefore, we precompute B_pA_pp and then merge them into the bias without increasing inference latency so that we can compute W = W + BA and b = b + B_pA_pp. By doing so, we can save the time required during inference.




 §.§ Multi-Head Attention

After obtaining the output from DualLoRA, we add it to the multi-head attention. In this way, we aim for each head to learn different prompt features, making it easier for prompts to influence the model. We pair DualLoRA with each attention layer in the T5 model, making it easier for prompts to influence the attention mechanism. As the length of the prompt is different from the length of the original input, we adopt the approach of taking the average of the slot prompt embedding and then interacting it with W_q and W_v in the attention mechanism, which is consistent with the weight matrix used in the LoRA <cit.>.





















§ EXPERIMENTS



 §.§ Dataset and Metrics

We conduct our experiments on the MultiWOZ dataset and Schema-Guided-Dialogue (SGD) dataset <cit.>, which are well-known benchmarks in task-oriented dialogue systems. MultiWOZ is an extensively annotated dataset comprising 10,000 human-to-human written conversations covering diverse domains and topics, making it a widely used benchmark dataset for evaluating task-oriented dialogue systems. The SGD dataset, like MultiWOZ, is extensively annotated with turn-level information and descriptions, which comprises over 20,000 annotated dialogues between a human and a virtual assistant, covering more than 20 domains. Additionally, the SGD dataset includes previously unseen domains in the test set, designed to measure the zero-shot performance of dialogue systems.

In zero-shot cross-domain dialogue state tracking, the model is initially trained on visible domains and tested on unseen domains. Building on prior work, we use Joint Goal Accuracy (JGA) and Average Goal Accuracy (AGA) to evaluate both the baseline and our model. JGA refers to the percentage of turns for which all the slots in the same domain are identified correctly, which is the primary metric for dialogue state tracking evaluation and is used to evaluate the model performance. AGA refers to the average accuracy of the active slots in each turn. The evaluation metrics used are consistent with Prompter <cit.>.





 §.§ Baselines

We compare the proposed method with TRADE <cit.>, MA-DST <cit.>, SUMBT <cit.>, SGD-baseline <cit.>, Seq2Seq-DU <cit.>, GPT2-DST <cit.>, TransferQA <cit.>, T5DST <cit.>, SlotDM-DST <cit.>, DCC <cit.>  and Prompter <cit.>. More details about baselines are presented in Appendix <ref>.



















    





 §.§ Implementation Details

To facilitate a better comparison, the adopted settings are consistent with those of Prompter <cit.>. We utilize T5 architecture as our backbone. As mentioned in Prompter, PPTOD <cit.> is more suitable for prompt tuning tasks because of the nature of its pre-training task. Therefore we choose PPTOD-small as our checkpoint. For more detail, the batch size is set to 8 with gradient accumulation per 8 steps; the optimizer is set to AdamW <cit.>, and the learning rate is set to 1e-4. All experiments were conducted on the NVIDIA GeForce RTX 3090 and 4090 GPUs.






 §.§ Main Results

As shown in Table <ref>, DualLoRA performs best except hotel domain compared to other baselines. That is because in the hotel domain, the presence of "none" value slots and unique value slots make it challenging for the model to utilize prompts for zero-shot dialogue state tracking effectively. In addition, the slots in the hotel domain are unique and cannot be effectively utilized using domain prompts during training. Moreover, we observe that the model can learn how to effectively utilize prompts between attraction and restaurant domains due to the close relationship between slot values in these two domains.  Consequently, they influence each other, leading to better performance of the model in attraction and restaurant domains. Similarly, the train and taxi domains follow a similar pattern.

As shown in Table <ref>, we present the performance of different models on various domains in the SGD task, which can be observed that DualLoRA exhibits the best performance in most domains. These domains are those for which the service in the test set is not present in the training set. The improvements of DualLoRA are attributed to effectively training on prompts in visible domains and leveraging prompt utilization in unseen domains.  Due to the pretraining of the alarm domain in PPTOD, we do not report the scores of our model on the alarm domain in this experiment. 



 



 









 §.§ Ablation Study

To investigate the effectiveness of different components in our model, we conduct an ablation experiment on context LoRA and prompt LoRA. Table <ref> demonstrates the effectiveness of the two components in our method. After adding the context LoRA, the model shows improvement across multiple domains, indicating that this structure plays a crucial role in the model's generalization. After adding prompt LoRA, the model also exhibits improvement across multiple domains, which clearly illustrates the contribution of the proposed structure.
















 §.§ Analysis

Moreover, we investigate the impact of different types of the bypass structure. We tested four different structures, including MLP, Adapter, Transformer, and LoRA. Results in Table <ref> show that LoRA excels in handling prompt information, and in terms of parameter usage, LoRA has the smallest number of parameters.


To explore the optimal fusion position of DualLoRA, we also tested two fusion positions for context LoRA and prompt LoRA: vertical combination and horizontal combination. The horizontal combination is the one we proposed in Figure <ref>, while the vertical combination is shown in Figure <ref>. As indicated in Figure <ref>, horizontal combination yields better results than vertical combination. We attribute the inferior performance of the vertical combination to the fact that a single context LoRA may not effectively optimize both types of information simultaneously.


We employed various fusion methods to explore the optimal fusion approach, such as mean addition, cross-attention interaction, and gate-attention interaction. As shown in Figure <ref>, in most cases, gate-attention and mean addition are superior to cross-attention interaction, which may be attributed to the initial dialogue context having a weak influence on prompt interaction, making it challenging for the model to converge effectively. In addition, both cross-attention and gate-attention introduce many additional parameters, which may not be favorable for the model in domain transfer scenarios. Therefore, we adopt mean addition as the fusion method for our different inputs.



 



 

We also conducted experiments to explore the optimal matrix rank of DualLoRA since it determines the matrix rank, including 8, 16, 32 and 64. As shown in Figure <ref>, the performance of the model does not consistently improve with the increase of rank, which is in line with the observation by <cit.> that the learned over-parametrized models reside on a low intrinsic dimension. Therefore, we adopt a rank of 8 for DualLoRA in all experiments.


In order to investigate the impact of the number of DualLoRA on model performance, we conducted experiments. The purpose was to explore whether the utilization of prompts becomes more effective with an increasing number of prompt LoRA. The results are shown in the Figure <ref>. Our analysis suggests that the limited size of the model parameters may be the reason behind the lack of significant performance improvement with an increasing number of prompt LoRA. Additionally, the introduction of more prompt LoRA leads to an increase in the overall parameter count. Therefore, in this study, we opted to utilize only one prompt LoRA to enhance the model's utilization of prompts.













To validate the effectiveness of our method, we visualized the attention in the last layer of the model's encoder to explore the model's focus on the dialogue context and prompts. As shown in Figure <ref>, we can observe that, the dialogue context pays more attention to the prompts compared to Figure <ref>. In addition, the attention scores in Figure <ref> after softmax are more extreme, ranging from 0 to 1, and are mainly distributed along the main diagonal, indicating a focus on itself. In contrast, Figure <ref> demonstrates a more diverse attention distribution, with effective attention on prompts. This observation further supports the effectiveness of our proposed method.










 §.§ Case Study

We selected a dialogue example from the MultiWOZ dataset to demonstrate the performance of our model compared to T5DST in zero-shot dialogue state tracking, as shown in Figure <ref>. The results show that both T5DST and our model predict most zero-shot dialogue states in the train domain, such as "train-book people-8" and "train-destination-ely." The rationale is that the models have been trained in similar domains, such as the taxi domain, allowing them to provide better answers. Moreover, the results show that T5DST, during zero-shot inference in the train domain, incorrectly equates the "leaveat" slot value with the "arriveby" slot value and predicts it as "13:30", which is because T5DST fails to effectively utilize prompt information, making it unable to differentiate between different slot values for "leaveat" and "arriveby" accurately.





§ CONCLUSION

In summary, this paper introduces DualLoRA based on a parameter-efficient transfer-learning method for zero-shot dialogue state tracking. DualLoRA consists of two parts, namely context LoRA and prompt LoRA. DualLoRA can simultaneously accept two inputs: normal input and prompt input. Through experiments, we demonstrate that this method improves the Joint Goal Accuracy (JGA) metric for MultiWOZ and SGD datasets without introducing inference latency. Additionally, we conduct a series of experiments to show the effectiveness of deploying DualLoRA and how to fuse two different types of information best. Our analysis indicates that the effectiveness of DualLoRA stems from the model's ability to extract and utilize prompt information efficiently.

Future work will focus on applying DualLoRA to various domains, expanding the method to multimodal zero-shot dialogue state tracking, or researching better fusion methods for more effective integration of prompts and dialogue context.









§ LIMITATIONS

DualLoRA has the following limitations currently. Addressing these limitations in future work will contribute to the broader applicability of DualLoRA.


    
  * Drawbacks of precomuptation. The reason we can use precomputation to reduce inference latency is that the slot prompts in the current target domain are limited. However, as the number of domains to be transferred increases, the computation introduced by precomputation will also grow. Therefore, the inference latency will also increase.

    
  * Unable to be applied to target domains without prompts. Due to DualLoRA relying on enhancing the model's utilization of prompts, if there are no prompts in the target domain, DualLoRA cannot leverage prompts to improve the model's performance.
    






§ ACKNOWLEDGEMENTS

This work was supported by the National Natural Science Foundation of China (NSFC) under Grant Nos. 61966038 and 62266051, the Yunnan University Talent Research Startup Support Project under grant No. CY22623101 and No. CZ22623101, the Postgraduate Research and Innovation Foundation of Yunnan University under Grant No. KC-23234274 and the Exam-Exempted Postgraduate Research and Innovation Foundation of Yunnan University under Grant No. TM-23236972. The authors would like to thank the anonymous reviewers for their constructive comments. The authors would like to thank Prompter <cit.> for providing the publicly available code.




acl_natbib







§ DATASET STATISTIC


The statistical data for the MultiWOZ dataset is presented in Table <ref>. Due to the limited data in the police and hospital domains within MultiWOZ, we conducted experiments using only five domains from the dataset. The statistical data for the SGD dataset is shown in Table <ref>.
















§ BASELINE MODELS


In this section, we provide a detailed overview of each baseline, as outlined below.


    
  * TRADE. TRADE <cit.> aims to enhance dialogue state generation by incorporating a copy mechanism and enabling knowledge transfer for previously unseen dialogue states during model training.
    
    
  * MA-DST. MA-DST <cit.> utilizes cross-attention to model relationships between context and slots across different semantic levels and employs self-attention to resolve cross-domain coreferences based on the RNN layer. 
    
    
  * SUMBT. SUMBT <cit.> learns relationships between domain-slot-types and slot-values through attention mechanisms based on contextual semantic vectors, predicting slot-value labels in a non-parametric manner.

    
  * SGD-baseline. SGD-baseline <cit.> encodes these elements into embedded representations, accommodating dynamic sets of schema elements by conditioning on corresponding schema embeddings based on the BERT model to predict dialogue state. 

    
    
  * Seq2Seq-DU. Seq2Seq-DU <cit.> treats DST as a sequence-to-sequence problem and employs two BERT-based encoders to encode dialogue utterances and schema descriptions independently. After that, Seq2Seq-DU uses a decoder to generate a pointer representation for the current dialogue state.

    
  * GPT2-DST. GPT2-DST <cit.> utilizes a generative question-answering model pre-trained on English sentences, enabling natural language queries for unseen constraints and slots in multi-domain task-oriented dialogues.

    
  * TransferQA. TransferQA <cit.> is a transferable generative QA model that seamlessly integrates extractive and multiple-choice QA using a text-to-text transformer framework, which effectively tracks both categorical and non-categorical slots in DST and introduces innovative methods to construct unanswerable questions.
    
    
  * T5DST. T5DST <cit.> encodes dialogue context and slots using a pre-trained self-attentive encoder, generating slot values in an auto-regressive manner. Additionally, the model incorporates slot-type informed descriptions to capture shared information across slots, facilitating cross-domain knowledge transfer. 

    
  * SlotDM-DST. SlotDM-DST <cit.> employs slot prompts combination, slot values demonstration, and slot constraint objects to model slot-slot, slot-value, and slot-context dependencies, respectively. Each slot prompt includes a slot-specific and a slot-shared prompt to capture shared knowledge across domains.

    
  * DCC. Divide, Conquer and Combine (DCC) <cit.> employs a "divide, conquer, and combine" strategy, explicitly disentangling semantics through a mixture-of-experts mechanism. Seen data is divided into semantically independent subsets, corresponding experts are trained, and unseen samples are inferred using the mixture-of-experts with ensemble inference.
    
    
  * Prompter. Prompter <cit.> addresses this by utilizing descriptions of target domain slots to generate dynamic prefixes. These prefixes are then concatenated to the key and values at each layer's self-attention mechanism, enabling prefix-tuning in zero-shot scenarios. 







§ BEST APPLIED WEIGHT MATRICES

To explore where we should apply DualLoRA in T5's weight matrices, we conducted experiments applying DualLoRA to the attention layers, such as query, key, value, and output.  As shown in the table <ref>, in most domains, achieving the best performance is possible by solely utilizing the attention layers for queries (q) and values (v), consistent with the findings in the LoRA paper <cit.>. Moreover, using only the q and v attention layers reduces the model's parameters. Therefore, in our experiments, we opted for applying DualLoRA only to the q and v attention layers.










§ SLOT PERFORMANCE ON MULTIWOZ

To delve into the slot accuracy performance of our model in each domain, we conducted experiments on each slot in every domain. The experimental results are presented in Figures <ref>, <ref>, <ref>, <ref> and <ref>. We compared our model with T5DST and observed that in most domains and for most slots, DualLoRA performs better than T5DST, likely due to DualLoRA's deeper understanding of prompts.





 


 

 

 







§ IMPACT OF DIFFERENT PROMPT INPUT

We conducted experiments using various inputs to explore the impact of different prompt inputs on the model's performance. We conducted experiments using regular slot embedding and slot prompts proposed by <cit.>, and the results are shown in Figure <ref>. As indicated, slot prompt outperforms slot embedding in all domains. We analyze this because the differences in slot embedding within the same domain are minimal, and the model cannot learn effective prompts from them. In addition, during cross-domain zero-shot transfer, the differences in slot embedding between different domains are substantial, leading to the model's inability to utilize prompts in the target domain during cross-domain transfer effectively.







§ EXAMPLES OF CASE STUDIES

This section presents a more detailed analysis of case studies. We categorize potential errors into three cases: under-predictions, incorrect-predictions, and over-predictions, as illustrated in Figures <ref>, <ref>, and <ref>, respectively. The sentences marked in red indicate wrong predictions.

As seen in Figure <ref>, our model enhances performance by leveraging prompts effectively, while T5DST's understanding of prompts remains superficial. Therefore, as shown in the figure, our model outperforms T5DST in terms of under-predictions in the restaurant and taxi domains. 

As shown in Figure <ref>, the model's overfitting in the original domain may lead to incorrect-predictions during cross-domain zero-shot transfer. Additionally, the misunderstanding of slot values also contributes to incorrect-predictions. Therefore, our model, compared to T5DST, is effective in avoiding incorrect-predictions during cross-domain zero-shot transfer.

As shown in Figure <ref>, our model makes errors in over-prediction. However, the predicted slot values by our model can still be extracted from the conversation. For instance, as stated in figure (a), "How about fitzbillies restaurant?" and "Yes that sounds great." suggests our model can extract the slot value for `name-fitzbillies` and as stated in figure (b), "How about the moderate 1?" suggests our model can extract the slot value for `pricerange-moderate`, but it does not match the ground-truth slot value. We attribute this discrepancy to potential failures in data annotation.

In addition, our model has some limitations, as shown in Figure <ref>. Our model fails to make predictions for some slots in the hotel domain, such as parking, internet, and stars. This might be related to the base model we adopted, T5-small. Due to the existence of some slots in the hotel domain that cannot be easily related to other domains, our model struggles to accurately track the values of these slots.




§ EXAMPLES OF ATTENTION MAP IN DIFFERENT DOMAINS

We present the visualization of attention in different domains after training with DualLoRA, as shown in Figures <ref>, <ref>, <ref>, <ref> and <ref>, which can be observed that our model effectively focuses on the prompt part after training, rather than solely on the main diagonal.






































 Paper Title: On the Problem of Text-To-Speech Model Selection for Synthetic Data Generation in Automatic Speech Recognition.



Spin-wave Goos-Hänchen effect induced by 360 degree domain walls in magnetic heterostructures
    Jie Lu
    August 5, 2024
=============================================================================================




§ ABSTRACT 
The rapid development of neural text-to-speech (TTS) systems enabled its usage in other areas of natural language processing such as automatic speech recognition (ASR) or spoken language translation (SLT).
Due to the large number of different TTS architectures and their extensions, selecting which TTS systems to use for synthetic data creation is not an easy task.
We use the comparison of five different TTS decoder architectures in the scope of synthetic data generation to show the impact on CTC-based speech recognition training.
We compare the recognition results to computable metrics like NISQA MOS and intelligibility, finding that there are no clear relations to the ASR performance.
We also observe that for data generation auto-regressive decoding performs better than non-autoregressive decoding, and propose an approach to quantify TTS generalization capabilities.




§ INTRODUCTION


The usage of synthetic data from text-to-speech (TTS) systems in the context of automatic speech recognition (ASR) has been explored in many different ways.
*Work done while being an internship student at JAIST
This ranges from simply generating new data <cit.> to jointly training ASR and TTS systems<cit.>.
Prior work improved the integration of the synthetic data into the ASR systems, for example by different weighting of synthetic inputs <cit.>, or the usage of internal ASR model statistics to reject synthetic samples not matching the real data<cit.>.
As most publications aim to introduce new methodologies, in each case they usually use one specific TTS architecture and one specific ASR architecture.
The number of fundamentally different ASR architectures is rather limited and roughly categorized into: Hybrid deep neural network hidden Markov model (DNN-HMM) <cit.>, connectionist-temporal-classification (CTC) <cit.>, Transducer <cit.> and Attention-Encoder-Decoder (AED) <cit.>. For TTS there is a vast amount of different architectures and models, especially when including different approaches of multi-speaker modeling<cit.>.
From the perspective of working on ASR this makes it very difficult to choose which TTS system to use for synthetic data generation.
In addition, human evaluations are not suited to evaluate data for training purposes, as conducting the evaluation usually takes more time than running the training itself. 
Then, evaluation for naturalness and human subjective evaluations alone can not be expected to be a sufficient criterion for utility estimation of synthetic data.
Instead, aspects like speaker diversity, temporal diversity or expressiveness in general are assumed to be relevant to yield good results by integrating synthetic data into the ASR process<cit.>. Unfortunately, those aspects are difficult to quantify.

Furthermore, LJSpeech <cit.> is often the only common dataset to compare new TTS architectures, which can not be expected to give representative results for the multi-speaker case<cit.>.
This is especially relevant as one might not have training data suited for TTS training at hand for a particular domain or language, but instead has to use the ASR data.

Motivated by those problems, we present an experiment pipeline to systematically analyze the relation of TTS generated data to ASR training.
For this we compare different decoder architectures, which are based on: Non-autoregressive (NAR) Transformer similar to FastSpeech-2 <cit.>, autoregressive (AR) long-short term memory (LSTM) similar to Non-attentive Tacotron <cit.>, Glow-TTS <cit.> as flow-based decoder and Grad-TTS <cit.> as diffusion based decoder.
We keep the rest of the TTS systems as simple as possible, using a Transformer encoder and convolution based duration prediction network, as used in <cit.>.

Given the large amount of possible TTS architectures, we choose these systems for multiple reasons:
First, they have a widespread recognition in the scientific community. Then, they already share similar encoder and up-sampling concepts, sometimes even using the same code, which makes comparison easier. Finally, they do not require training with an integrated vocoder model (as e.g. in VITS) which would make a direct comparison more difficult.  
While comparisons of different TTS architectures with respect to naturalness and expressiveness exist <cit.>, this is to our knowledge the first work covering a broader range of different TTS output characteristics and quality levels targeted for ASR training.
Also, we perform a direct comparison of NAR vs AR decoding for data generation with a consistent sequence mapping method.

Lastly, we present a way of measuring generalization of the TTS systems with respect to ASR training data generation by using different conditions for synthesizing the training data. The Sisyphus <cit.> based setup and additional materials are available on Github[<https://github.com/rwth-i6/i6_experiments/tree/main/users/rossenbach/experiments/jaist_project>].


§ TEXT-TO-SPEECH SYSTEMS




 §.§ General architecture


In the design of the TTS systems we follow what we found to be most prevalent in recent literature. After the introduction of Transformer blocks with convolution for TTS <cit.> and the establishing of duration based state up-sampling <cit.>, many systems continued using these components. Thus, for each system we use the same layout of a Transformer based-encoder, a convolution based duration predictor and a discrete encoder state up-sampling method. Only the decoder, which converts the up-sampled encoder states h_1^T into the final log-Mel spectrograms x_1^T, is changed. Figure <ref> shows the TTS architecture used in this work. We do not use explicit f_0 or energy prediction. The overall structure follows exactly Glow-TTS, in which many parts were taken from TransformerTTS <cit.> or FastSpeech-2. We also used the code of the official Glow-TTS repository as reference for our implementation.


 §.§ Decoder variants




A large amount of proposed TTS decoder architectures can be categorized into three groups: Direct prediction, flow and diffusion. Direct prediction means that the output of the TTS neural network is a log-Mel spectrogram which is trained to directly match the target x_1^T. Common losses are mean absolute error or mean squared error. The generation is a single forward pass through the decoder network g_θ, which can be written as:

    x̂_1^T = g_θ(h_1^T)

The generation can also be done in an auto-regressive manner:

    x̂_t = g_θ(h_1^t, x̂_1^t-1)

For flow-based TTS systems, the decoder is an invertible neural network that can be applied in both directions from the spectrograms x_1^T into latent space variables z_1^T and vice-versa. During training, the target features are transformed into the latent space and matched against a normal distribution with a mean μ_1^T = μ_θ(h_1^T) and unit variance. The target objective is to maximize the likelihood of the latent space samples given the distributions. For inference, the latent space values are sampled from the distribution with a mean function μ_θ, temperature τ, and passed backwards through the flow f_θ:

    ẑ_1^T ∼𝒩(μ_θ(h_1^T),τ1)    and   x̂_1^T = f^-1_θ(ẑ_1^T)

For the diffusion-based TTS system we follow the Grad-TTS architecture. A normal distribution as in the flow-based approach is used to sample a noisy output spectrogram X̂_I with X = x_1^T. The noise spectrogram is cleaned in I steps using a differential equation containing the solver network s_θ and pre-defined noise schedule β_i to reach the final output X̂_0:

    X̂_I    ∼𝒩(μ_θ(h_1^T),τ1)
    X̂_i-1   = X̂_i - 1/2I(μ_θ(h_1^T) - X̂_i - s_θ(h_1^T, X̂_i))β_i

For each of the decoder principles we implemented one model for which the details will be presented later in Section <ref>.

























§ EVALUATION PIPELINE AND METRICS



















To evaluate the TTS systems in the context of ASR, we train a CTC-based ASR system purely on data generated by TTS. We generate the same amount of data  as used for the baseline training, and measure the performance of the ASR systems in word error rate (WER) for recognizing real human speech. In the optimal case, the performance of the ASR system trained on the synthetic data would match the performance of the system trained on real data. In order to evaluate the generalization capabilities of TTS towards new conditions, we generate the synthetic data in 3 ways: a) using exactly the same text and speaker assignment as in training, b) using the same text but shuffling the speakers and finally c) using the same amount of text from a different corpus in the same domain. That way we can measure how good the TTS systems are at generalizing to new conditions different from the training. The main evaluation criterion to measure the utility of the TTS-generated data is the WER of the trained ASR systems calculated on the official LibriSpeech <cit.>  test sets. In addition, we use metrics directly on the synthetic data to evaluate their relation to the final WER. We use NISQA <cit.> as a tool to generate automatic mean-opinion-score (MOS) values. While computer generated MOS values are not a replacement for subjective evaluations, they can be used as tool to determine naturalness as possible indicator for the ASR performance. Subjective evaluation itself can not be useful to evaluate synthetic data in a development process, as doing the evaluation often takes more effort than simply training the ASR system. We also use a larger ASR model trained on the full LibriSpeech corpus to recognize the synthesized cross-validation data to check for completeness and pronunciation correctness. We refer to this metric as synthetic WER (sWER). In the literature this is often called intelligibility metric, and sometimes character error rate is used instead of WER , e.g. in <cit.>.












































§ EXPERIMENTS




 §.§ Data


We use the LibriSpeech train-clean-100 as baseline data to train the TTS and ASR models, and the transcriptions of train-clean-360 as additional text data. This is a common setup that has been used extensively in previous literature <cit.>. We sub-sample the exact same amount of utterances, roughly 28k, from train-clean-360 to have the same amount of unseen text as text in the training data. We take 4 utterances of each of the 251 speakers in train-clean-100 out of the TTS training data to have a cross-validation set to report MOS and sWER. We perform no further pre-processing of the data, unlike other work <cit.>. I.e. no excessive silence is removed, and no speakers are discarded. We use the official LibriSpeech lexicon for word-to-phoneme conversion and use Sequitur <cit.> to generate phonemes for words that are not in the lexicon. For ASR, we augment the phoneme set with end-of-word markers as presented in <cit.>.



 §.§ Automatic speech recogition

As ASR system we use a CTC model with a Conformer-based<cit.> encoder with 12 blocks, a base dimension of 384 and a convolutional frontend with a sub-sampling factor of 4. For recognition we use the lexicon-constrained Flashlight <cit.> CTC decoder via the available Torchaudio interface. The search process includes the official LibriSpeech 4-gram LM model. Since it has been shown that synthetic data generation is successful mostly for AED-ASR <cit.>, we chose the context-free phoneme-based CTC-ASR specifically for the reason that it should be influenced less by changing or adding new text.



 §.§ Training and model settings



All models are trained using 80-dimensional log-Mel spectrogram features, for TTS with 12.5ms shift and for ASR with 10ms shift and SpecAugment <cit.> applied.
The TTS models are trained with Adam <cit.> for 400 epochs. For all models except Grad-TTS we use linear learning rate (LR) scheduling, which we found to perform better than inverse square reduction. We increase the LR from 5e-5 to 5e-4 for the first 100 epochs, and reduce linearly from 5e-4 to 5e-7. For Grad-TTS we use a constant LR of 1e-4. The ASR models are trained using AdamW  with weight-decay 1e-3 for roughly 80 epochs and a maximum learning rate of 7e-4. We generate an alignment by training the Glow-TTS system once using implicit alignment search. In each TTS training the fixed alignment is used for up-sampling and as target for the duration predictor. For vocoding, we convert the log-Mel to linear features using a separate BLSTM-based network and apply the Librosa Griffin & Lim (G&L) <cit.> function with 32 iterations and momentum of 0.99.



  §.§.§ TTS parameter settings


For the different decoder architectures the number of parameters, training time and inference time differ, as shown in Table <ref>. Making a fair comparison is difficult, as for each system we would need to do extensive tuning experiments until we reach the optimal performance. Thus, we stick mostly to the sizes presented in previous publications.

Due to the high cost of subjectively evaluating TTS systems, ablation studies on the model sizes are usually not presented in scientific publications.



Base model: For the TTS Transformer encoder we use 6 layers with an internal dimension of 256, preceded by 3 convolution layers with 256 channels and kernel size 5. The duration prediction network uses 2 convolutions with kernel size 3 and 384 channels.


Transformer decoder: For the Transformer-based decoder we simply re-use the same structure as for the encoder Transformer, having 6 layers with an internal base dimension of 256.

The system is trained using L1 loss on the output spectrograms.


Autoregressive LSTM decoder: For the AR-LSTM decoder we are using 2 layers of 1024-dimensional Zoneout-regularized LSTM layers. We use a reduction factor of 2 to predict two feature frames per decoder step. The pre-net for the AR feedback loop and the convolutional NAR post-net are derived from <cit.>. We apply L1 loss for both the decoder output and the combined prediction with the post-net.


Non-autoregressive LSTM decoder: This decoder is similar to the AR decoder, but we remove the AR feedback loop network, and replace the uni-directional Zoneout LSTM cell by a normal bi-directional LSTM with dropout. Each LSTM direction has 512 dimensions. For Transformer, AR- and NAR-LSTM, a 256-dimensional speaker embedding vector is concatenated to each upsampled encoder state.


Glow-TTS decoder: For a flow-based decoder, including speaker conditioning, we follow exactly the architecture of Glow-TTS <cit.>. The decoder consists of 12 invertible coupling blocks with an internal dimension of 256. We merge each 2 consecutive frames for a latent-space dimension of 160.


Grad-TTS decoder: For a diffusion-based model we follow the architecture and settings of Grad-TTS <cit.>. As the original architecture is not designed for multi-speaker TTS, modifications were necessary for convergence. We added two additional Transformer layers on top of the 6 encoder layers which are conditioned on the speaker embedding. That way, we have a speaker-conditioned mean prediction network. We also used the same speaker embeddings to condition the u-net, which is used for the differential equation solving.
We found τ = 0.7 to be the optimal noise scale for both Glow-TTS and Grad-TTS.



§ RESULTS




 §.§ Synthetic data, automatic MOS and recognizability

































In a first comparison we evaluated the systems via the synthetic data only ASR training, the NISQA MOS and the sWER.
Table <ref> shows the results of each of the five decoders compared to the reference values of the real data. For all metrics we used text data unseen during training.
With respect to the final ASR training evaluation, all TTS systems have a reasonable performance and there is no strong outlier.
In general, using the synthetic data for training results in a system with double the error rate when training TTS only on the available ASR data, which is a consistent result to previous literature <cit.>.
<cit.> showed that even when using vast amounts of training data to create strong foundation-model sized TTS models, the generated synthetic data seems to be not yet on par with real data.
The AR-LSTM decoder produces the best synthetic training data, which is unexpected as the Glow-TTS and Grad-TTS should produce more expressive and variable audio due to their internal sampling process.
Also, Glow-TTS is the only system that exactly follows the given literature, while the other systems have small adaptations or are simplifications of the original model architecture.

The third and fourth column in Table <ref> show the NISQA MOS and the sWER on the TTS cross validation set. It is visible that there is no direct relation of the two metrics compared to the final ASR performance.

For example, when comparing Grad-TTS to the Glow-TTS and Transformer decoder, there is a lower MOS and a much higher sWER, but still the final system performance is better. This means both metrics are not reliable to determine which system to pick for data generation.



 §.§ Decoder characteristics

In Figure <ref> we show a selected sequence synthesized by four different decoders. The Transformer decoder spectrograms are smooth, lacking a lot of detail in the high-frequency area. The AR-LSTM spectrogram in comparison has much more details. The spectrogram from Glow-TTS shows visible noise resulting from the latent space sampling. It also has less distinct boundaries between the phonemes, but rather fuzzy transitions, which might be an explanation for the worse recognizability. The Grad-TTS has less distorted spectrograms than Glow-TTS, but we found that for very short sequences the output is sometimes just a noisy grid pattern. Also the voice lines become very blurred together for lower frequencies, causing a very unnatural sound, which matches with the low automatic MOS score. Thus we assume, that the NISQA MOS and the sWER are very sensitive to certain artifacts that have less impact in on the actual ASR training. One can see that all four TTS decoders produce spectrograms with different characteristics, even with other components being identical. Thus, it is important to not only consider a single TTS architecture when trying to show a general relation between the ASR performance and metrics calculated on the level of the TTS system or the synthetic data.

































 §.§ Generalization




Table <ref> shows how good the TTS systems generalize with respect to the synthetic data generation task. Recall that condition a) means to exactly reproduce the training data, so with perfect memorization during training it would be possible to produce identically performing data. It is visible that across all models the values for a) and c) are mostly not too far from each other, so the generalization abilities are high. This gives a hint that current models are under-fitting and much larger models might yield better results, even on very constrained datasets such as LibriSpeech 100h. For synthesizing new text, so comparing b) to c), most models show no or only a small change. Only for the AR LSTM and Glow-TTS decoder there is meaningful degradation in the WER, mostly for the "other" test. For the Glow-TTS a possible reason could be that it suffers from pronunciation issues, which get even stronger when synthesizing new text. Testing for the generalization capabilities of TTS this way might not be used only for the comparison of different models like in this work, but allow for easier hyperparameter tuning which is usually not done extensively in TTS literature.



 §.§ Discussion


When training with synthetic data only, we achieved a WER performance which is less than factor 2 worse compared to training on the real data. In relation, <cit.> reported reaching ratios of around 2 on the LibriSpeech-960 task, and <cit.> reported ratios of more than 3. In this work we specifically investigated decoder architectures. In future work this gap can be reduced by extending the results of this work with extensive hyperparameter tuning as well as prosody <cit.> and more sophisticated duration modeling <cit.>. We have seen that NISQA MOS and the sWER are not reliable to determine the usefulness of the TTS system for ASR data generation. This means there are more and/or different metrics needed to give accurate estimates about how good the synthetic data can be utilized for ASR training. While in our work synthesizing new data and training the ASR systems was done in less than a day, this becomes much more of a problem when training large models on magnitudes more of data. Thus, we think it is important to find ways to estimate the capabilities of TTS models before starting large-scale data generation.



§ CONCLUSION

In this work we evaluated different TTS decoder architectures w.r.t. their ability to generate synthetic data for ASR training. We compared five different TTS decoder architectures and found that for our setup the AR-LSTM-based system with L1 loss performed best in creating training data for ASR. We were able to create synthetic data that performs less than a factor of 2 worse than real data while training the TTS only on the same data as available for the ASR training. We observed that the systems' outputs differ from each other in many aspects such as naturalness or pronunciation accuracy. Still, the metrics we used to capture them showed a surprisingly low correlation to the performance of the ASR systems trained on such outputs. This indicates that better metrics have to be found in order to do a meaningful evaluation of TTS systems before using it to create synthetic training data. We also introduced a way to measure the generalization capabilities of TTS systems for data generation, which can be used to optimize TTS towards being able to generate data that reaches parity with real data.




§ ACKNOWLEDGEMENTS




This work was partially supported by NeuroSys, which as part of the initiative “Clusters4Future” is funded by the Federal Ministry of Education and Research BMBF (03ZU1106DA), and by the project RESCALE within the program AI Lighthouse Projects for the Environment, Climate, Nature and Resources funded by the Federal Ministry for the Environment, Nature Conservation, Nuclear Safety and Consumer Protection (BMUV), funding ID: 67KI32006A. Part of this work is supported by JSPS KAKENHI Grant Numbers JP21H05054 and JP23K21681. The authors thank Lukas Rilling, Peter Vieting and Benedikt Hilmes for their additional input and feedback.




IEEEtran









 Paper Title: Cost-Effective Hallucination Detection for LLMs.









Cost-Effective Hallucination Detection for LLMs
    Simon Valentin^1[1]
        Jinmiao Fu^2[1]
        Gianluca Detommaso^3These authors contributed equally to this research. 
        Shaoyuan Xu^2
        Giovanni Zappella^1
        Bryan Wang^2
    
    ^1Amazon Web Services, Berlin, Germany 

    

    ^2Amazon, Seattle, USA 

    

    ^2Helsing, Berlin, Germany 

    


=======================================================================================================================================================================================================



§ ABSTRACT 
Large language models (LLMs)
can be prone to hallucinations — generating unreliable outputs that are unfaithful to their inputs, external facts or internally inconsistent.
In this work, we address several challenges for post-hoc hallucination detection in production settings.
Our pipeline for hallucination detection entails: first, producing a confidence score representing the likelihood that a generated answer is a hallucination; second, calibrating the score conditional on attributes of the inputs and candidate response; finally, performing detection by thresholding the calibrated score. We benchmark a variety of state-of-the-art scoring methods on different datasets, encompassing question answering, fact checking, and summarization tasks. We employ diverse LLMs to ensure a comprehensive assessment of performance.
We show that calibrating individual scoring methods is critical for ensuring risk-aware downstream decision making. 
Based on findings that no individual score performs best in all situations, we propose a multi-scoring framework, which combines different scores and achieves top performance across all datasets.
We further introduce cost-effective 
multi-scoring,
which can match or even outperform more expensive detection methods, while significantly reducing computational overhead. 





§ INTRODUCTION

Despite their impressive capabilities,  large language models (LLMs) can be prone to generating hallucinations — undesirable outputs that are incorrect, unfaithful, or inconsistent with respect to the inputs (or the output itself) <cit.>. 
These unreliable behaviors pose significant risks for adopting LLMs in real-world applications. 
Challenges in detecting hallucinations lie, among other things, in hallucinations taking different forms, being context-dependent and sometimes being in conflict with other desirable properties of generated text <cit.>. 
Hallucinations may be harmless in some contexts, but can be undesired or potentially dangerous in other applications (e.g., erroneous medical advice).
Detecting and quantifying hallucination risk is thus a critical capability to enable safe applications of LLMs and improve generated outputs.



Prior work has proposed various approaches for detecting and mitigating hallucinations in LLM-generated outputs, including verifying faithfulness to inputs <cit.>, assessing internal coherence <cit.>, consulting external knowledge sources <cit.>, and quantifying model uncertainty <cit.>.
However, deploying these methods in production settings is far from trivial due to several challenges:
First, there is limited comparative evaluation illuminating how different detection methods perform. 
Second, existing approaches for detecting hallucinations differ greatly in their computational demands, and guidelines are lacking on cost-effectiveness trade-offs to inform method selection for real-world applications with constraints.
Third, hallucination detection in the real world often requires careful consideration of risks and false positive/negative trade-offs, requiring methods to provide well-calibrated probability scores.
Fourth, many applications of LLMs take the form of calls to black-box APIs, which sometimes requires methods to employ workarounds to assess the model's confidence in its generated output. 

In this work, we provide a framework for detecting hallucinations in the outputs of any LLM in a model-agnostic manner.
Our approach relies on quantifying the probability that a generated answer is a hallucination. 
After computing initial scores, we employ state-of-the-art calibration techniques to provide calibrated probabilities of the generation containing a hallucination, which can subsequently be used for decision making or other downstream tasks. 
We evaluate a variety of scores proposed in the literature for hallucination detection on several metrics, across multiple datasets encompassing question answering, fact checking, and summarization tasks. We employ a range of different LLMs to ensure a comprehensive assessment of performance. 

Critically, as we show that no single score performs best across all datasets, 
we introduce
multi-scoring, 
a simple way of combining multiple calibrated scores, which achieves superior performance compared to 
any individual score alone.
Furthermore, 
we 
propose cost-effective multi-scoring, which 
finds the subset of best-performing scores for any fixed cost budget, and combines them in a multi-scoring fashion. Our empirical demonstrations reveal that cost-effective multi-scoring not only matches but often surpasses the performance of individual scores that incur significantly higher costs. Consequently, our proposed method achieves superior hallucination detection outcomes while maintaining a substantially lower cost footprint.


We summarize our contributions as follows: 1. We benchmark a variety of hallucination detection methods across the literature on several metrics, over different datasets and LLMs. 2. We introduce multi-scoring, a novel approach that aggregates multiple complementary scores and outperforms individual scores. 3. We further propose cost-effective multi-scoring, which optimally balances detection performance and computational constraints. 





§ DETECTING LLM HALLUCINATIONS




 §.§ Formalizing Hallucination Detection

We study the problem of quantifying the probability that a generated output from a language model contains hallucinations.
More formally, let 
x represent an input token sequence to a language model 𝒢, and let 
z represent a generated output text sequence from the model. 
We define a binary random variable y ∈{0,1} that indicates whether z is a “permissible” output (y=1) or contains a “hallucination” (y=0). 

Our goal is to develop a scoring function to model the probability that a given output text contains a hallucination conditioned on the input.[Note that while we are interested in whether the entire output contains any hallucination, one could easily adapt the methods at different granularities, such as the sentence or phrase level.]
This is critical, as in real-world scenarios, we need to set risk-aware thresholds, balancing false positive/negative rates to accommodate the production requirement.
Having access to scores allows us to flexibly set the threshold.
Conceptually, the key reason to model this probabilistically is that there is inherent uncertainty in determining whether a given text contains a hallucination or not. 
Even human raters may disagree on the assessment, based on their own knowledge and definition.
Some key contributors to this epistemic uncertainty are: First, that no system has complete world knowledge to perfectly assess factual correctness. 
Second, that there is ambiguity in whether something is a hallucination. 
Finally, any automatic scoring model may make occasional errors, so probabilistic scores reflect confidence. 
Therefore, while conditional on x and z the true hallucination label y is fixed, our estimate of y remains uncertain. The probabilistic score thus reflects this epistemic uncertainty — the degree of belief that z contains a hallucination given the available knowledge:
p(y = 0 |x, z; ψ), where ψ denotes parameters of the scoring function. We refer to this conditional probability function as the hallucination score, denoted s_ψ(x, z).
The hallucination score can then be applied to downstream tasks, including making risk-aware binary decisions.

We discuss concrete instances of hallucination scores in the following subsection.
Generally, the form of s_ψ can vary between hallucination detection approaches. 
In particular, s_ψ may depend on multiple candidate texts generated for the same input, as implemented by a number of hallucination detection methods proposed in the literature.
We denote K candidate generated outputs as Z = [z_1,...,z_K]. 
Then s_ψ(x, Z) could quantify inconsistencies within texts in Z using different metrics ψ, as discussed below.



 §.§ Scoring Methods

Many hallucination detection methods proposed in the literature make use of LLMs to “judge” the output of an LLM (either the same or a different one). 
For clarity, we thus distinguish between generator and detector LLMs. 
The generator is defined as the model used to generate the original response. 
The detector LLM is the model used to score a generated text for the presence of hallucinations. 
In general, the generator and detector LLMs may coincide. 
However, it may be more desirable to use different LLMs in some scenarios, e.g., when computational cost is a greater concern, where a smaller LLM may be used to judge outputs of a more expensive LLM, or when using hallucination scoring methods that require white-/grey-box access, while the generator is black-box. 

Today, many interactions with LLMs take the form of API-calls.
Typically, only the output tokens are returned with no access to the logits of the predicted tokens,  treating the LLM effectively as a black box. 
Sometimes, inference parameters may be accessible, allowing for setting different temperature (among others) values, thereby providing some (grey-box) access to the model. 
Generally, hallucination detection methods vary in their required model access, ranging from black-box APIs to full white-box access to the model weights.
That is, while some methods only require token-level outputs, other methods may need access to the logits of the generated tokens, or some control over inference parameters like temperature. 

In our experiments, we evaluate a comprehensive set of hallucination scoring methods.
Generally, we do not make any assumptions about the generator LLM being white-, grey- or black-box. 
We divide methods into single-generation methods, which require only one generated output, and multi-generation methods which are based on multiple alternative generations. 





 §.§ Single-generation

We first provide an overview over a set of hallucination detection methods that are based on scoring the hallucination from a single given generated output. 



  
Inverse Perplexity This method provides a prominent instance of using a model's logits over output tokens to score the confidence in the output. Computed as the inverse of the exponentiated average negative log-likelihood of the LLM's response <cit.>,

    Perplexity^-1(W) = exp(1/N∑_i=1^N log p(w_i|w_i-1,...,w_1)),

inverse perplexity thus provides a sequence-length normalized expression of the model's confidence. 
Here, the model may either be the generator LLM or a different detector LLM.
Using a different LLM from the generator as a plug-in estimator implies that the confidence estimate is detached from the generator. 
If our generator provides access to the model's logits we can directly use them to score hallucinations.
Alternatively, we can use another LLM to generate the logits of output tokens. 



  
P(True) This method works by prompting an LLM whether an answer is correct or not, then making use of the logits of the next token <cit.>. 
Given a prompt like the following: 


We compute P(True) by first taking the softmax over the first generated output token's logits, 
and then normalizing across the “True” and “False” tokens.[Note that alternatively, we can prompt the LLM to bind the True/False tokens to symbols like response options A or B <cit.>. However, this did not change results in our experiments. Further, if the detector LLM provides only black-box access, one can sample the corresponding tokens and use the empirical distribution to approximate, but thereby incurring a higher computational cost.]
That is, let 𝐳^(1) = (z^(1)_1, …, z^(1)_|V|) be the logit vector over the vocabulary V for the first generated token when asking the LLM to evaluate whether the answer is correct. We then take the softmax over these logits, 
p(w |𝐳^(1)) = exp(z^(1)_w)/∑_w' ∈ Vexp(z^(1)_w').
Let w_True, w_False∈ V be the “True” and “False” token ids. Then we can compute P(True) as:

    P(True|𝐳^(1)) = p(w_True|𝐳^(1))/p(w_True|𝐳^(1)) + p(w_False|𝐳^(1)).


In addition to checking whether the response is correct, we provide variations of P(True), where we check whether the output contradicts the input in P(InputContradict), the output contradicts itself in P(SelfContradict), or the output contradicts generally established facts in P(FactContradict), with the followings prompts: 








  
NLI Text Classification Natural language inference (NLI) models provide an alternative to assess the correctness of the model output. As hallucination detection requires checking for contradictions, we compute the score as 1-S_contradict, where S_contradict refers to the softmax probability of the output conflicting with the question. Specifically, we use a DeBERTa model fine-tuned on an MNLI task <cit.> as the underlying NLI model, which we refer to as NLI (DeBERTa).




  
Verbalized Probabilities Instead of analyzing a model's logits, scores are elicited by asking an LLM to provide a confidence verbatim, i.e., by generating tokens that indicate the numerical confidence <cit.> with the following prompt:

We note that moderated LLMs accessed through APIs may in practice decline to answer if the context does not provide sufficient information, rather than assigning a low confidence score.  



 §.§ Multi-generation
 Multi-generation methods are based on quantifying the consistency across multiple generated outputs from the generator LLM. 
This follows the notion of white-/grey-box uncertainty quantification via logits that if the LLM is confident in its response, multiple generated responses will probably be alike and contain compatible facts. 
Conversely, for fabricated information, sampled responses are more likely to differ and contradict one another. Crucially, this rests on the assumption that the model's confidence is calibrated, as we discuss and evaluate below.



  
SelfCheckGPT This method exploits a DeBERTa NLI model to assess whether the answer A_0 is consistent with K alternative generated answers A_1, ..., A_K <cit.>. Each sentence of A_0 is compared against the full set of answers A_1, ..., A_K. A consistency score per sentence is obtained via an NLI model, then the final score is obtained by averaging over the sentence-wise scores. 



  
Similarity Degree This method is based on computing pairwise similarities between the multiple responses using NLI models, and then quantifying uncertainty based on the distribution of similarities <cit.>. Here, we compute the pairwise similarities between responses via the contradict class probability of an NLI model and construct a degree matrix, where each diagonal element corresponds to the total (sum) similarity of the corresponding response to all other responses. We use the degree of the candidate response A_0 as the confidence estimate, following favourable results in prior work <cit.>, but note that other metrics have also been proposed in the original paper.



  
NeMO Guardrails: Hallucination Rail 
Unlike SelfCheckGPT or Similarity Degree, the score here is computed via one LLM call, where we check for agreement between the concatenated K additional generations and the candidate output and not averaged over sentences or computed on pairs of responses. Specifically, we use the softmax probability normalized over yes/no tokens using the following prompt:





 §.§ Calibration

Initial hallucination scores may not be properly calibrated, which can lead to poor downstream decisions. Seminal work by Guo et al. <cit.> has demonstrated that neural models tend to be miscalibrated, particularly in the form of models being overconfident. 
Later work, focusing on language models, has confirmed this across a wide range of tasks, models and datasets <cit.>. 
Notably, there is also research which demonstrates that given particular prompts and in-distribution tasks, LLMs can be well-calibrated <cit.>. However, this has been shown to be brittle and dependent on context <cit.>. 

Formally, our score outputs probabilities p_ψ(ŷ=0|x, z) that z contains a hallucination, parameterized by ψ. 
The score is calibrated if, for any probability level p ∈ [0,1], the average observed frequency of hallucinations matches the predicted probability:

    𝔼[y | p_ψ(ŷ=0|x, z) = p] = p


A naive approach for obtaining scores would be to compute the model's probabilities marginally, ignoring the context/question x and generated response z and directly estimating p(y=1). 
However, this does not allow for conditioning on individual inputs to obtain calibrated probabilities p(y=1 |x, z) for specific x, and z, which is, however, impossible to guarantee <cit.>. 
Common calibration methods include temperature scaling (Platt scaling) of logit outputs <cit.>, isotonic regression <cit.> or histogram binning <cit.>, which operate marginally, and can thereby not account for different confidence levels for different inputs (e.g., with an LLM being more confident on certain domains than others).

An alternative is to partition the inputs into G groups and compute calibration separately for each group g∈ G. However, this assumes the groups are disjoint and does not handle inputs belonging to multiple groups, which is often necessary <cit.>.
More advanced calibration methods, such as multicalibration, which we use in this work, allow defining G potentially overlapping groups  <cit.>. 
Prior work has shown the effectiveness of modern calibration techniques in scoring LLM confidence, albeit in a white-box setting <cit.>.
We describe our calibration approach in the experimental section.



 §.§ Multi-Scoring: Combining Scores

Different scoring methods capture different aspects of hallucinations, e.g., incorrect, non-factual, non-grounded, irrelevant, inconsistent with other answers, etc. As a result, some scores may work better on some data or for some specific models, and worse on others. Therefore, we design a multi-scoring method to combine the complementary information from individual scores into a single, strong predictor.

Denote each available score by s_n, for n=1, … ,N. To obtain an aggregated score, we run a logistic regression using as features the concatenations of the logit of each score, i.e. [logit(s_n(x, z)), …, logit(s_N(x, z)) ], and the labels of the calibration dataset as target variables. [Note that we also conducted experiments with alternative models, such as XGBoost <cit.> and Random Forest <cit.>, but results did not show significant improvements, so we opted for logistic regression for simplicity.]





 §.§ Cost-Effective Multi-Scoring

Scoring methods based on multiple generations incur the cost of additional generations as well as the cost of checking their consistency. 
Similarly, multi-scoring can be a viable choice when there are only few LLM generations to check for hallucinations, but incurs considerable computational cost, especially when using multi-generation methods. 
To avoid prohibitive computational costs in practice, we propose cost-effective multi-scoring, where we set a fixed computational budget and compute the best performing combination of scores that stay within the specified budget. 

Given an input text x and generated text z, we have N scoring functions s_1(x, z), ..., s_N(x, z) with associated costs c_1, ..., c_N (e.g., number of generations required). We are given a total computational budget B. Our goal is to find the optimal subset of scores S^* ⊆{1,...,N} that maximizes detection performance while staying within budget B:


    S^∗ = _S: S ⊆{1, …, N}ℒ(f(s_i(x, z))_i ∈ S)   s.t.  ∑_i ∈ S c_i ≤ B


where ℒ measures loss on a validation set. This is a constrained optimization problem over subsets of scoring functions. When B=min_i c_i, it reduces to selecting the single best score at the lowest cost. When B=∑_i c_i, it recovers full multi-scoring. In between, the optimal subset S^* provides the best trade-off between performance and cost. 
We note that in general, this problem is computationally challenging, given the exponential runtime complexity. 
However, given that there are only generally a small set of potential scores (N ∼ 10) and the logistic regression is very fast to fit, computing all combinations takes only 1.8 seconds on a single Intel Xeon processor (3.1 GHz), iterating over all candidate solutions sequentially. 
When this approach is not feasible, some exemplary alternatives include classic greedy forward-selection methods or regularising the model via an L1 penalty while scaling the regularisation term to accommodate score cost. 
Overall, cost-effective multi-scoring allows for flexibly balancing multiple scores under computational constraints.



  
Quantifying Cost
While the cost c_i of scoring function s_i is presented abstractly above, accurately quantifying computational cost can be challenging in practice. 
The actual runtime per method is a reasonable proxy, however the runtime depends on model architecture, hardware acceleration, batching, etc. 
One approach is to directly benchmark each scoring function's average runtime empirically on the target hardware. 
However, this overlooks nuances like caching effects and ignores runtime variability due to implementational differences.
To simplify our analysis, and as LLM calls are generally much more expensive than calling smaller NLI models based on parameter size, we leverage the number of LLM calls required per method as a proxy. See Table <ref> for an overview. 
If more precision is desired, we suggest empirically running different scores and computing their computational cost in the actual application, as the cost will depend on the precise setup and a range of factors.




§ EXPERIMENTS




 §.§ Experimental Setup



  §.§.§ Datasets




  
TriviaQA TriviaQA <cit.> is a commonly used factual open-respon­se QA dataset. Originally set up as a reading comprehension task, it is today often used without context as a closed-book (free-recall) task <cit.>. 
We use the validation fold, containing 17944 question-answer pairs. We use  <cit.> to generate candidate answers. To decide whether a given response should be labelled as positive or negative, we check whether the correct answer is contained in the generated answer after removing formatting, following the original evaluation script <cit.>.  



  
FEVER The closed-response Fact Extraction and VERification dataset <cit.> provides a comprehensive benchmark of factual hallucination detection. We take the test fold, containing 14027 labelled examples of source documents, claims and whether they are  ,  or contain . For the purpose of hallucination detection, we look at all claims that are either  or .



  
HaluEval We include the hallucination detection dataset HaluEval <cit.> and use the summarization task, which contains 10000 labeled examples of source documents, summaries and labels for whether the summary contains hallucinations. 



  
BIG-bench It provides an evaluation of LLMs on a diverse set of tasks <cit.>. We select 11 tasks suitable for hallucination detection, leading a total of 8664 labelled examples from the validation fold. 



  §.§.§ Metrics

Practical applications of hallucination detection require calibrated scores, but often also binary decisions over whether a given output is hallucinated or not. 
To this end we report the Brier score <cit.>, binary decision metrics F1 score and accuracy. 



  §.§.§ LLMs

We conduct experiments with  <cit.>,  <cit.>,  <cit.> and  <cit.>.
All models are used with default configurations via HuggingFace Transformers <cit.>.



  §.§.§ Calibration

The calibration step is performed via the following multicalibration approach, using the Fortuna library <cit.>:
To obtain groups, we compute embeddings of the input text x and the generated response z, such that our embedding is e[ (x), (z) ]. We obtain embeddings from Universal AnglE Embedding <cit.>, which is the SOTA in the MTEB benchmark <cit.> at the time of writing. 
We subsequently reduce the dimension of x via UMAP <cit.> and perform soft-clustering via Gaussian Mixture Models, as a simple off-the-shelf algorithm (fitting the number of cluster components via BIC <cit.>).
The calibration error is measured for each group g∈ G separately. 
The model is then iteratively patched on the group with the largest error until the calibration error drops below a threshold for all groups.[Note that alternative approaches are possible <cit.>.] 
This provably converges to a calibrated model with theoretical guarantees <cit.>.
We fit the calibrator to a random calibration fold of 80% of the data and report only held-out test results. 
For all binary predictions, we set the threshold to the 50th percentile to not impose preferences over false true/negative rates, but note that in practice any threshold could be applied. 



 §.§ Individual Scoring Methods

Table <ref> presents the results of different hallucination detection methods on all datasets. Multi-generation methods (SelfCheckGPT-NLI, HallucinationRail and SimilarityDegree) are employed only for TriviaQA, since the latter provides the true answer to each question, which can be compared to the alternative generated answers to assess their agreement. In contrast, for HaluEval, BIG-Bench and FEVER, the true answer is not provided. Instead, we compare the given candidate answer from the dataset against the provided binary label of correctness.
If we exclude Multi-Scoring methods, multi-generation methods such as SelfCheckGPT-NLI and SimilarityDegree (with 10 generations) achieve the best performance on TriviaQA, P(True) is the best on HaluEval and BIG-Bench, and P(InputContradict) performs best on FEVER. 
Thus, we find that there is no single best scoring method across all datasets. 
This is likely because different scoring methods capture different aspects of hallucination, supporting the notion that hallucination is a multi-faceted concept. 
Multi-generation methods measure hallucination based on the consistency of the generator LLM's responses and on the ability of the comparison method to identify whether multiple alternative responses are in agreement. 
We find that SelfCheckGPT-NLI performs best in our experiments in comparison to SimilarityScore and HallucinationRail.
While all of these methods follow similar approaches, there are subtle differences in how they assess the consistency between different responses.
More generally, as we discuss below, multi-generation methods are appropriate only if one can assume that there is exactly one correct response, but can fail otherwise. 

Other methods are based on different notions of hallucination, such as NLI (DeBERTa) measuring the entailment of the response given the input. 
Interestingly, variants of P(True) can be appropriate for detecting different kinds of hallucinations. 
P(True) explicitly asks the evaluator LLM to check whether the response is correct. 
In some contexts, the evaluator LLM may have the ability to directly evaluate this. 
However, in other situations, as we see with the FEVER dataset, other methods such as P(InputContradict) can be more appropriate, if we are trying to directly target a specific form of hallucination, which may not be subsumed under a general “correct or not?” prompt. 
Other applied scenarios may target even more different (or more specific) kinds of hallucinations, though the variants we include in this work are designed to cover the space in a reasonable manner. 
Similarly, while the datasets included in these experiments were selected to cover a wide range of different kinds of hallucinations, real-world applications may show even new kinds of hallucinations, for which no public datasets are available.
Overall, these findings highlight the need for our multi-scoring method which can absorb the strength of each individual method and can be easily applied to a concrete hallucination detection setting while only requiring a relatively small amount of labeled data. 

Overall, neither the inverse perplexity score nor the NLI (DeBERTa) scores emerge as the best scores for any of the datasets we consider. While they may add information that can be exploited in (cost-effective) multi-scoring, as reported below, individually they perform worse than some of the other scores. 






 §.§ Multi-Scoring

To evaluate the proposed multi-scoring method, we conduct experiments on all datasets combining the scores via logistic regression. 
As presented in Table <ref>, the multi-scoring ensemble achieves an F1 score of 0.9106 on TriviaQA, outperforming the best individual F1 score of 0.8614 (achieved by SelfCheckGPT-NLI).
Similarly, Brier and Accuracy also show the highest performance for multi-scoring.
For HaluEval, we also find that multi-scoring outperforms the best individual score (P(True)) in all metrics. 
BIG-Bench shows more nuance, as multi-scoring outperforms the best individual score (P(True)) on Brier and Accuracy, but not on F1. 
This reflects the fact that the metrics capture different properties, and practical considerations may require a decision as to which metric should be prioritized in a given application.
For FEVER, multi-score again outperforms the best individual score (P(InputContradict)) in all metrics. 

Thus we see that combining scores performs better than the top performing individual scores, showing that combinations of different scoring signals complement each other and can boost performance. 
This demonstrates that combining complementary signals enables more robust hallucination detection, while balancing the strength of each scoring method.
Crucially, while some settings may benefit from combinations of different signals (such as when trying to detect different kinds of hallucinations in a given generated output), in other settings the data-driven selection of an informative signal may be sufficient (such as when trying to detect a more narrowly defined notion of hallucination). 
These scenarios are covered by our use of multi-scoring, which allows for arriving at optimally combined hallucination scores (or binary decisions) for a given application.
Meanwhile, as computational cost can be a concern with scaling LLM applications, it may not be desirable to always use a full ensemble of scores, and we would rather compute the most performant score at a fixed computational cost.



 §.§ Cost-effective Multi-Scoring

Shown in Table <ref>, cost-effective multi-scoring methods are also amongst the top performers, but at considerably lower cost compared to multi-score. We see that at cost C=1, measured as the number of LLM calls (but see our discussion about measuring cost above), we recover the best individual scores, which varies across each dataset. As we increase the budget, the cost-effective multi-scores converge to the performance of the multi-score itself. Notice that in several instances, cost-effective multi-score with C=2 already performs as good as multi-score itself. 
For TriviaQA, we see that performance increases on all metrics as we increase the computational budget, with cost-effective multi-score at C=5 outperforming SelfCheckGPT-NLI as the most performant individual scores at half the computational cost in all metrics but Brier.
HaluEval shows no improvement beyond a budget of C=2, which is likely due to the kind of hallucination to detect being more narrowly defined and well-capture by a small number of signals, thereby not benefiting from additional scores. 
For BIG-Bench, as discussed for multi-score, we recover the best individual score at C=1, and match multi-score at higher budgets.  
FEVER results indicate that, again, we recover the best individual score at budget C=1, and can already recover the full multi-score at a budget of C=3.

We now take a closer look at cost-effect multi-scoring when including multi-generation scoring methods.
The overall cost budget is varied over the entire range. For each budget, we solve the constrained optimization in Eq. <ref> to find the optimal subset S^* of scores.
We compare the hallucination detection F1 score achieved by the cost-effective ensemble versus individual scoring functions and the full ensemble with all scores. Figure <ref> shows the results for TriviaQA for responses generated via . With a minimal budget of B=1, cost-effective selection recovers the best single method, as expected. As the budget increases, it selectively adds more expensive functions, gradually improving F1, though the gains are marginal at higher budgets per unit cost. At the maximum budget, it recovers the unconstrained full ensemble performance. In between, cost-effective multi-scoring incorporates both less and more and expensive methods to maximize detection within the computational constraints.



These results demonstrate that the proposed cost-effective multi-scoring approach can intelligently balance the trade-off between computational expense and hallucination detection effectiveness. It outperforms individual scoring functions and makes selective use of more costly scores to maximize detection performance under a fixed computational budget.

A key consideration in practice would be whether to include potentially more costly multi-generation methods, such as SelfCheckG­PT-NLI. 
In particular as implied by the results presented in Table <ref>, we may be interested in reducing the number of required multiple generations while matching performance at lower computational costs. 
To this end, we compare SelfCheckGPT (as the single best performing method for TriviaQA) alone with SelfCheckGPT combined with P(True) at different numbers of additional responses generated via . 
Here, we count the evaluation of P(True) as one additional LLM call, just as generating one additional output from the generator LLM. 
As presented in Figure <ref>, at one additional generation, SelfCheckGPT (in the degenerate case of only one generated output) combined with P(True) thus recovers P(True). 
As the number of additional generations increases, the combination quickly outperforms SelfCheck alone. 
In particular, we observe that cost-effective multi-scoring with 3 LLM calls is already as good as SelfCheckGPT with 9 LLM calls in our experiment.
This highlights how we can save costs by combining multi-generation methods with other scores while requiring fewer additional generated responses from the generator LLM than if we wanted to achieve the same performance with multi-generation methods alone. 





 §.§ Exploration of Relationships between Scores


As discussed above, different scoring methods can target different kinds of hallucations. 
To explore this empirically, we here explore their relationships via Spearman rank correlations. 
As presented in Figure <ref>, calibrated scores across all different individual scores are positively correlated. 
Meanwhile, the magnitude of their correlations is smaller than would perhaps be  expected if one were to consider hallucinations as a uniform phenomenon. 
At the same time, we can see certain “clusters” of more strongly inter-correlated scores emerge.
For example, multi-generation methods including SelfCheckGPT, HallucinationRail and SimilarityDegree, which check the consistency across multiple generations, show comparably high correlations. 
P(True) and P(True) Verbalised and P(InputContradict) emerge as a similarly correlated cluster of stronger correlations. 
Overall, this supports the idea that different scores can capture distinct information, and the need to empirically select appropriate (combinations of) scores for a given application, as we propose with cost-effective multi-scoring.






 §.§ Experiments across Different LLMs

Most of the scoring methods considered in this work rely on detector LLMs to compute scores for hallucination detection. 
In Table <ref> of Appendix <ref>, thus present additional data for different LLMs, namely ,  and .
Overall, we see that hallucination detection performance is correlated with performance on general LLM benchmarks[E.g., <https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard>]. 
Thus, more overall capable LLMs are likely to perform better at hallucination detection than less capable ones.
However, concrete applications may need to take the inference cost of LLMs into account, where one may not always be able to use the most expensive model. 
We note that cost-effective multi-scoring could here also make use of scores computed via LLMs with different computational costs to find cost-effective combinations. 



 §.§ The Importance of Calibration

To analyze the impact of calibration, we conduct an ablation study by evaluating model performance with and without calibrating the individual scores.
Table <ref> in Appendix <ref> indicates that model performance clearly drops in most of the datasets and metrics without calibration. This demonstrates the benefit of calibration for more accurate and risk-aware downstream applications.



 §.§ Exploring Multi-Generation Assumptions





Our results indicate that methods based on the uncertainty among multiple generations can provide strong signals when such multiple generations are available. 
However, such methods can also suffer from problems in practice. 
Methods based on multiple generations make use of the uncertainty of the generator, that is the distribution over generations z given the input x, for model 𝒢, i.e., p_𝒢(z|x) and the consistency between actual sampled generations z. 
We are interested in classifying whether the candidate output contains a hallucination or not, 
that is estimating and making use of p(y |x, z). 
The generating model's uncertainty p_𝒢(z|x) can be a useful proxy for p(y |x, z), but only under particular conditions. 

On a technical level, sampling multiple answers requires access to the generator's temperature parameter, as temperatures of zero collapse the generations to a single response, as is mentioned, e.g., in <cit.>. 
Therefore, these methods are technically grey-box models, as they require some access to the model's inference parameters. 

More conceptually, the generator's uncertainty (or confidence <cit.>) over p_𝒢(z|x) reflects uncertainty over generated tokens. 
All recently proposed multi-scoring methods we are aware of are based on the idea of scoring the consistency across multiple generations using model-based metrics. 
However, self-consistency across multiple generations is neither a necessary nor sufficient criterion for a hallucination to be present. 
Sufficiency is not given as a model may consistently provide an incorrect response. 
Self-consistency across multiple generations is also not necessary, as many tasks allow for multiple hallucination-free answers that are contradictory to each other. 

As a relatively harmless example, in tasking a model to generate cooking recipes for lunch, the model may generate, among other things, a recipe for a salad and a recipe for a soup. Clearly, the steps in preparing these dishes contain contradictory information, while otherwise being free of hallucinations in themselves. 
Thus, if there is more than one correct response, the self-consistency assessment in multi-generation methods may falsely score an output as likely to be hallucinated. 
As there are numerous situations where there exist multiple correct responses, assuming that there is only one could lead to worse LLM responses, also via a loss of diversity.

Practically, methods based on multiple generations can be costly due to added computational overhead, as we have seen above.
Finally, these methods are based on the assumption that LLMs are calibrated. 
As such, multi-generation methods do not allow for detecting cases where the generator LLM is confident yet wrong. 









§ CONCLUSION


In this work, we compared a comprehensive set of scoring methods to provide calibrated probability scores for the presence of hallucinations in generated LLM outputs.
Overall, we have observed that no single hallucination detection score performs best across all datasets.
Our experiments showed that combinations of scores, as suggested with multi-scoring, can effectively combine complementary signals to yield higher hallucination detection performance than any individual score.
Further, we demonstrate that cost-effective multi-scoring can find the highest performing scores at a given computational budget.

More generally, our findings support the notion that hallucinations can be rather multi-faceted than present a uniform phenomenon.
Thus, detecting hallucinations may require different methods.
In concrete settings, one may be interested in even more fine-grained detection of particular types of hallucinations, such as in specific domains like code generation <cit.>.

The approach outlined in this work requires only a small amount of labeled data to calibrate hallucination scores and combine scores via (cost-effective) multi-scoring.
However, it is important to acknowledge the limitations of the current work. Future research could explore the effectiveness of this approach on more diverse datasets, investigate alternative scoring methods, or extend the methodology to multi-modal models. 
While improvements in LLM performance can be expected to also lower their rate of occurrence, hallucinations are unlikely to go away completely <cit.>.
Thus, even as models improve further, detecting hallucinations is likely to remain relevant for applying LLMs in a reliable and trustworthy manner in future.

In summary, our work presents a promising approach for detecting hallucinations in LLM outputs by combining multiple scoring methods in a cost-effective way, offering a pathway towards more reliable and trustworthy language models that can be applied in real-world settings with more confidence.


















§ ADDITIONAL RESULTS














 Paper Title: Evolver: Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection.






Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection]Evolver: Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection













jhuang90@ur.rochester.edu

  University of Rochester
  Rochester
  NY
  USA


jpan24@ur.rochester.edu

  University of Rochester
  Rochester
  NY
  USA



wan.512@osu.edu

  The Ohio State University
  Columbus
  OH
  USA



hlyu5@ur.rochester.edu

  University of Rochester
  Rochester
  NY
  USA



jluo@cs.rochester.edu

  University of Rochester
  Rochester
  NY
  USA




§ ABSTRACT 
Recent advances show that two-stream approaches have achieved outstanding performance in hateful meme detection. However, hateful memes constantly evolve as new memes emerge by fusing progressive cultural ideas, making existing methods obsolete or ineffective. In this work, we explore the potential of Large Multimodal Models (LMMs) for hateful meme detection. To this end, we propose Evolver, which incorporates LMMs via Chain-of-Evolution (CoE) Prompting, by integrating the evolution attribute and in-context information of memes. Specifically, Evolver simulates the evolving and expressing process of memes and reasons through LMMs in a step-by-step manner. First, an evolutionary pair mining module retrieves the top-K most similar memes in the external curated meme set with the input meme. Second, an evolutionary information extractor is designed to summarize the semantic regularities between the paired memes for prompting. Finally, a contextual relevance amplifier enhances the in-context hatefulness information to boost the search for evolutionary processes. Extensive experiments on public FHM, MAMI, and HarM datasets show that CoE prompting can be incorporated into existing LMMs to improve their performance. More encouragingly, it can serve as an interpretive tool to promote the understanding of the evolution of social memes.[The source code will be made publicly available.] 

1.2Disclaimer. This paper contains offensive content that may be disturbing to some readers.










<ccs2012>
   <concept>
       <concept_id>10010147.10010178.10010224.10010225.10010227</concept_id>
       <concept_desc>Computing methodologies Scene understanding</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Scene understanding










[
    Jiebo Luo
    
=============



§ INTRODUCTION




Hateful meme detection <cit.> is a crucial task in the field of multimodal research, aiming to identify content that combines text and images to propagate hate speech or offensive messages. 
Memes, as a widespread cultural phenomenon, proliferate on the Internet, blending images and texts to convey sophisticated meanings. 
Furthermore, the fusion of visual and textual elements in memes complicates the interpretation of their semantics and the identification of hateful undertones.
The combination of text and images presents significant challenges in hateful meme detection, especially for detecting and moderating hateful content <cit.>. 


With the advances in image-text pre-training, efforts to leverage and fine-tune CLIP <cit.> for hateful meme detection have demonstrated notable success in achieving high accuracy <cit.>. 
However, as shown in Figure <ref>(a), memes evolve similarly to biological organisms <cit.>, undergoing variation, mutation, and inheritance, which complicates the detection of hateful memes as they continuously evolve, merging with other cultural elements to form new expressions.

Traditional methods for hateful meme detection, as illustrated in Figure <ref>(b), suffer from limitations in interpretability and adaptability. 

These shortcomings hinder the understanding of memes' evolving nature and their contextual nuances, leading to overfitting on training datasets and diminished effectiveness in the dynamic and evolving real-world meme landscape.

To overcome this dilemma, we introduce Evolver, a novel approach that uses Large Multimodal Models (LMMs) <cit.> for hateful meme detection. By developing a benchmark specifically tailored for hateful meme detection based on LMMs, we aim to address the aforementioned limitations of traditional detection methods. Our approach not only enhances detection capabilities but also provides a more explainable and adaptable framework suitable for the evolving landscape of Internet memes. As shown in Figure <ref>(c), Evolver is an innovative approach to advance the state-of-the-art in zero-shot hateful meme detection significantly. Extensive experiments have shown the effectiveness of our method across three widely recognized hateful meme detection datasets, demonstrating a superior ability to identify and interpret hateful memes.

Overall, the main contributions of this work are:



  * We establish an LMM-based zero-shot hateful meme detection benchmark, which provides a comprehensive evaluation of the application of LMMs in social media.

  * We propose a simple yet effective Evolver framework, which advances LMMs with Chain-of-Evolution prompting. It expands LMMs with an evolution reasoning ability while offering good interpretability.

  * Extensive experiments on commonly used zero-shot hateful meme detection benchmarks with superior performance validate the efficacy and generalization of our method.




§ RELATED WORK

Hateful Meme Detection. Most works of hateful meme detection are based on pretrained vision-language models and fine-tune them on the hateful meme detection data. For example, <cit.> introduced the Hateful Memes Challenge and proposed a multimodal model that combines visual and textual features using a transformer-based architecture. Hate-CLIPper proposed a multimodal contrastive learning approach to improve the representation learning of memes by aligning the visual and textual modalities. These works demonstrate the effectiveness of using pretrained vision-language models. Differently, we discuss the potential of LMMs for hateful meme detection and introduce a knowledge-enhance LMM, Evolver.


Large Multimodal Models. 
With the release of the GPT-4V <cit.>, many researchers focus on the development and application of LMMs in processing and understanding multimodal data. Discuss groundbreaking models like GPT-4V and LLAVA-1.5 <cit.>, highlighting their capabilities in interpreting complex datasets that include a combination of text and images. Address the challenges these models face, such as sensitivity to noise and typographical errors, and the ongoing efforts to improve their accuracy and robustness in real-world applications. 

In this work, we adopt the LMMs to hateful meme detection, where set an LMM-based zero-shot hateful meme detection benchmark. Furthermore, we show that our Evolver further boosts the LMM's performance. 


Prompt-based Learning.
Prompt-based learning has evolved significantly from its inception with large language models (LLMs) to its current application in various tasks. The concept of prompt engineering emerged with the release of GPT-3, which demonstrated that LLMs could be guided to generate desired outputs through carefully crafted prompts. Inheriting the merit from LLM, prompt-based learning in large multimodal models (LLMs) has also gained extensive research attention. In our work, we present a novel chain-of-evolution prompting strategy to seamlessly boost the existing LMM for hateful meme detection. 





§ APPROACH

In this section, we first briefly describe the hateful meme detection task and LMM-based approaches. Next, we introduce the proposed Evolver.



 §.§ Background








  §.§.§ Problem Definition
 We define the task of hateful meme detection as a binary classification task. Meme, in our case, consists of an image-text pair. This task can be illustrated as follows: 

    ŷ_j = g({X^j_v, X^j_t}),
 where ŷ_j denotes the j-th prediction and the prediction ŷ_j ∈{0, 1} indicates the target image-text pair is hateful or not. g(·) is the multimodal model. {X^j_v, X^j_t} is the j-th image-text paired input.  






  §.§.§ Large Multimodal Models







Vision Encoder
serves as a translator to help language models understand visual content. It leverages frozen pretrained vision models such as CLIP <cit.> and ViT <cit.> to encode visual content so that the language model can understand visual content: 

    h_v = W·Enc_vis (X_v),

where h_v is the language embedding tokens. W is the projection to transform visual features into language embedding tokens. Enc_vis(X_v) denotes the visual feature extracted by pretrained model. 





Large Language Model Decoder generates a sentence given tokens <cit.>. The process of generation can be represented as: 

    h_t = Tokenize (X_t), 
     
        p(w) = ∏^n_i=1 p(w_i | w_<i, h_t, h_v),
where X_t is the input text, Tokenize(·) transform text into tokens, h_v is the image tokens, p(w) is the probability of generating a sentence by a language model, p(w_i | w_<i, h_t, h_v)  is the probability of generating a token at the i-th position given the previously generated tokens, input text, and image tokens. 
The visual tokens are incorporated with textual tokens and then fed to the language model. 





 §.§ Evolver: Chain-of-Evolution Prompting



To improve LMMs' understanding of the online hateful memes which are evolving in nature, we design a novel Chain-of-Evolution (CoE) prompt which has three components: (a) an evolutionary pair mining module to identify most relevant candidate memes; (b) an evolution information extractor to extract key information from the candidate memes; and (c) a contextual relevance amplifier for more effective hatefulness detection.
       



Evolutionary Pair Mining
It is unusual to find memes that have an evolutionary relationship with existing memes. Motivated by <cit.>, the evolution of a meme is defined as new memes that emerge by fusing other memes or cultural ideas. Therefore, the evolution of memes and old memes share similar textual and visual semantic regularities. We leverage this property to identify these old memes given the evolved meme. To this end, we can leverage the evolution of hateful memes to enhance LMMs' ability to understand hateful memes. In the implementation, we first generate the textual and visual embeddings from an external meme pool and target memes with CLIP and then fuse the textual and visual embeddings with a fixed ratio. Ideally, meme evolution has temporal ability, which means that the memes change with time. However, it is difficult to accurately locate the upstream and downstream of meme evolution with existing technology. Therefore, we use multiple evolutionary memes to find the common characteristics of this evolution. Finally, given a target image-text pair, we retrieve the top-K similar memes using cosine similarity:





    memes = {A_i | cos(𝐀, 𝐁)_i ∈Top_K(cos(𝐀, 𝐁)) },
 where 𝐀∈ℛ^n × d is embedding of n candidate memes and 𝐁∈ℛ^d is the  d-dimensional vector of the target meme. cos (·) return a similarity vector and Top_K(·) returns K highest values given the input vector. For each evolution meme, we pair K memes that the evolution meme is derived from. 





Evolution Information Extractor
To extract the information we are interested in (hateful component), we summarize paired memes with the help of a large multimodal model. The whole process can be expressed as follows:

    Info = LMM([memes_top_K, X_extract]),

where Info stands for our evolutionary information and LMM indicates the large multimodal model. memes_Top_K are the Top_K memes retrieved in the previous step, and X_extract is the instruction to guide the LMM to extract information. We present the  detailed instruction of X_extract as follows:



Contextual Relevance Amplifier
To enhance the in-context hatefulness information, we add a contextual relevance amplifier to the LMM during evolution information extraction and final prediction. The contextual relevance amplifier can help increase the search for hateful components. In our case, the contextual relevance amplifier is the definition of a hateful meme given by the dataset we use. Finally, we combine the extracted information and contextual relevance amplifier as the in-context enhancement and feed them to the model: 

    ŷ = LMM([memes_target, X_detect, Info, Amplifier]),

where Info stands for the information extracted previously, ŷ is the final prediction, memes_target is the memes we want to detect, and X_detect is the instruction to ask the large multimodal model to detect hateful memes. Amplifier refers to contextual relevance amplifier. The example of the amplifier is the same as the blue part in Table <ref>.










       






























































§ EXPERIMENT

We first describe three public datasets as well as the widely-used metrics and settings. Next, we introduce the baseline models including API-based (GPT-4V) and open-source LMMs (LLaVA) in Appendix <ref>. Finally, we present the evaluation and ablation study of our CoE prompting. 

To ensure a fair comparison, we create the LMM-based zero-shot hateful speech detection benchmark.
(a) For API-based LMMs, we select GPT-4V and Gemini-Pro-V, which are among the most popular models.
(b) For open-source LMMs, our comparison extends to include six widely used models: Openflamingo, LLaVA-1.5, MMICL, MiniGPTv2, BLIP-2, and InstructBLIP. 


GPT-4V <cit.> first creates a GPT-based LMM on the massive volume of datasets on the internet. They trained the language decoder, then aligned it with the vision encoder. 

Gemini-Pro-V <cit.> curates a large multimodal model with various image-text data on the web and then trains the model with image-text data from scratch.

Openflamingo <cit.> builds a LMM using CLIP as vision encoder and MPT/RedPajama as language decoder. The model is trained on open-source image-text datasets.

LLaVA1.5 <cit.> improves the performance of the model they propose previously <cit.> where CLIP is used as vision encoder and Vicuna as language decoder, with MLP projection to bridge the gap between vision encoder and language decoder. The model is pre-trained on the curated collected dataset. Furthermore, the model is instructionally fine-tuned with academic-task-oriented VQA data. 


MiniGPTv2  <cit.> connects the vision and language space with Q-former and an MLP projection. The model is pre-trained with a large collection of aligned image-text pairs and then instruction fine-tuned on curated high-quality image-text
pairs.

BLIP-2  <cit.> propose a model architecture using only Q-former to connect the vision encoder and language decoder to perform a series of multi-modal tasks. The pre-training of model has two stages. The initial stage initiates the learning of vision-language representations using a frozen vision encoder. Subsequently, the second stage advances the generative learning of vision-to-language from a stationary language model.

InstructBLIP  <cit.> further performs instruction fine-tuning based on BLIP-2 various converted multi-model datasets. 

MMICL <cit.> propose a framework 
using Q-former and an MLP layer to connect a vision encoder and language decoder. 




 §.§ More Experiment Details
 


Settings
We implement our Evolver based on the MMICL and LLaVA-1.5. For MMICL, we set the minimum length for the generation to 50, the maximum length for the generation to 80 during the Evolution Info Extractor, and the temperature to 0.2. During the final prediction, we set the minimum length of generation to 1, and the maximum length for generation to 50. For LLaVA-1.5, we set temperature to 0.2 and maximum generated tokens to 1024 for both stages. 

Effect of the Ratio of Textual and Visual fusion. 
The evolution of memes should consider both textual and visual information in a mixed form. We generate the textual and visual embeddings from the carefully curated meme pool and target memes with CLIP and then fuse the textual and visual embeddings with a 4:1 ratio in practice.

Effect of the External Curated Meme Pool. 
Generally, the external curated meme pool should have two characteristics (1) do not overlap with the test set. (2) have enough evolutionary information. To simplicity, we use the training set as the carefully curated meme pools, where the memes follow the same definition of hatefulness/harm/misogyny. 





Dataset Details. For more clarity, we show the detailed hateful meme detection dataset analysis in Table <ref>. Here, we give a detailed description of each dataset. 





 §.§ Datasets, Metrics, and Settings

Datasets We evaluate Evolver on three widely-used public datasets, namely, Facebook Hateful Meme dataset (FHM) <cit.>, Harmful Meme dataset (HarM) <cit.>, and Multimedia Automatic Misogyny Identification (MAMI) <cit.>.

Metrics 
We adopt ACC (accuracy) and AUC (area under the ROC curve) as evaluation metrics.

Settings
We implement our Evolver based on MMICL <cit.> and LLaVA-1.5 <cit.>. 

More details refer to the Appendix <ref>.












 §.§ Quantitative Analysis

Table <ref> shows the results of our method on three hateful meme detection datasets under the zero-shot setting. As we can see, our method consistently outperforms open-source LMM-based models across all datasets and metrics, which are closely aligned with API-based methods. Compared with the vanilla MMICL, our Evolver with the same architecture achieves higher accuracy (63.6% v.s. 60.4%) on the FHM dataset.




 §.§ Ablation Study

In Table <ref>, evolutionary pair mining (EPM) boosts the baseline with an improvement of up to 2.3%. Moreover, the evolutionary information extractor (EIE) and contextual relevance amplifier (CRA) significantly improve the generalization ability. Our full model achieves the best performance and outperforms the baseline by 8.5% in accuracy. This demonstrates that three parts of Evolver are beneficial by integrating with the evolution information.












 §.§ Qualitative Analysis

We visualize the paired evolution memes retrieved by the evolutionary pair mining modules in Figure <ref>, showing how the evolution information influences the final prediction.  The origin model (MMICL) without considering the meme's evolution, predicts non-hateful because it is not easy to detect the boy's disability or genetic conditions. After obtaining the evolution information like “discriminates against individuals with disabilities or genetic conditions.", 

our Evolver rectify the prediction, which supports the rationale of our method. 



 §.§ Computing Resource Requirements

Our Evolver is seamlessly integrated into the LMMs in the inference state, requiring minimal computational resources. For the main results, we conduct experiments on one GTX-3090-24G or one A100-40G. For the ablation study, we measure the comparison on one A100-40G.




 §.§ Impact of the Number of the Evolutionary Memes

As shown in Figure <ref>, we show the effect of the number K of the evolutionary memes. We find that our method significantly improves the baseline under all evolutionary meme number settings. Specifically, we set the K=5 to achieve the best performance in practice.




§ ADDITIONAL CASE ANALYSIS


To show the effectiveness of our method and align a better understanding of the hatefulness in the dataset, we show more examples generated by Evolver and the baseline model in Fig. <ref>.







§ CONCLUSION

We present Evolver to seamlessly boost LMMs for hateful meme detection via Chain-of-Evolution prompting. By integrating the evolution of memes, our method can adapt to unseen memes. Experimental results demonstrate the effectiveness of Evolver.



§ IMPACT STATEMENTS

Ethics Statement. Our Evolver aims to detect hateful memes by leveraging large multimodal models (LMMs) through a chain-of-evolution promotion strategy. While our objective is to mitigate the spread and impact of hate speech online, we recognize the potential for misuse or unintended consequences of this technology. Specifically, there are concerns about privacy, censorship, and the potential for biases within the AI model that could result in unfair or inaccurate content moderation. We are committed to ensuring that our technology is used ethically and responsibly.

Reproducibility Statement. We have clarified inference details including hyper-parameters, and the chain-of-evolution pipeline in Section <ref> and Appendix. In addition, all the datasets used in this paper are open-source and can be accessed online.

Limitation. While Evolver demonstrates significant advances in detecting hateful memes by leveraging the evolution of memes and large multimodal models (LMMs), it has some limitations. First, the effectiveness of our approach relies heavily on the quality and diversity of the curated meme pool used for seeking evolutionary memes. Furthermore, biases inherent in these datasets could potentially affect the model's ability to generalize across different cultural contexts and meme evolution patterns not represented in the related data. 




ACM-Reference-Format





 Paper Title: Characterizing User Archetypes and Discussions on Scored.co.








Characterizing user archetypes and discussions on Scored.co 





Andrea Failla1,2*,
Salvatore Citraro2,
Giulio Rossetti2, Francesco Cauteruccio3






1 Department of Computer Science, University of Pisa, Pisa I56127, Italy


2 Institute of Information Science and Technologies “A. Faedo” (ISTI), National Research Council (CNR), Pisa I56127, Italy


3 DIEM, University of Salerno, Fisciano I84084, Italy


























*andrea.failla@phd.unipi.it






§ ABSTRACT

In recent years, the proliferation of social platforms has drastically transformed the way individuals interact, organize, and share information. 
In this scenario, we experience an unprecedented increase in the scale and complexity of interactions and, at the same time, little to no research about some fringe social platforms. 
In this paper, we present a multi-dimensional framework for characterizing nodes and hyperedges in social hypernetworks, with a focus on the understudied alt-right platform Scored.co. 
Our approach integrates the possibility of studying higher-order interactions, thanks to the hypernetwork representation, and various node features such as user activity, sentiment, and toxicity, with the aim to define distinct user archetypes and understand their roles within the network. 
Utilizing a comprehensive dataset from Scored.co, we analyze the dynamics of these archetypes over time and explore their interactions and influence within the community. 
The framework's versatility allows for detailed analysis of both individual user behaviors and broader social structures. 
Our findings highlight the importance of higher-order interactions in understanding social dynamics, offering new insights into the roles and behaviors that emerge in complex online environments.













§ INTRODUCTION



In the last decades, the proliferation of social platforms and the opportunities tied to each of them have drastically transformed the way individuals interact, share information, and form communities. This unprecedented growth in the scale and complexity of online interactions is largely due to social networks like X (formerly known as Twitter), discussion fora like Reddit, and even more niche platforms, like the now-defunct Voat, known for their explorations of censorship-free models from the point of view of specific political groups <cit.>.

Hence, understanding user behavior within these platforms has become a crucial area of research in computational social science <cit.>. 
By studying how users interact with each other and with others' content, researchers can uncover patterns of influence spread, opinion formation, and creation and dissolution of communities <cit.>.

These insights are not only academically interesting but also have practical implications for designing better social platforms, enhancing user experience, and mitigating issues such as misinformation and online harassment.
Moreover, understanding these behaviors can aid in developing strategies for marketing, public health campaigns, and even political mobilization, making this research area highly relevant in multiple domains. Furthermore, all of these aspects are even more critical in the analysis of understudied platforms. Indeed, platforms such as Scored.co <cit.>[<https://scored.co>], which have not been extensively explored, offer unique social dynamics that may differ significantly from “mainstream” platforms. Investigating these less-explored environments can uncover novel patterns of user behavior and mechanisms of information dissemination. 
Also, understudied platforms often cater to niche communities or specialized interests, thus providing insights into subcultures and micro-level social dynamics that might not be straightforwardly visible on larger social platforms.

Understanding these dynamics becomes particularly important when addressing the general challenges inherent in studying complex social platforms.
Among all of these, a significant one is the identification of roles that users play within complex social structures. Indeed, role identification is crucial for various applications, including targeted information dissemination <cit.> and community detection <cit.>. In traditional social network analysis, roles are often determined based on patterns of dyadic connectivity and interactions. 
Nevertheless, in a discussion on a social platform, two users can interact directly, e.g., by sharing comments, but also indirectly, i.e., through other users.
Hence, it is fair to assume that taking into account group dynamics and higher-order interactions could help in a more reliable identification of the roles played by users.

A promising approach to encapsulate group interactions is defined by the concept of a social hypernetwork <cit.>, which allows for the modeling of relationships among groups of nodes, generally called hyperedges. Such a representation is particularly useful for studying social platforms from a higher-order point of view. 

Despite its importance and the existing methodologies designed for pairwise graphs, current literature about the problem of role identification lacks comprehensive approaches for social hypernetworks.

To address this gap, in this work, we formally introduce an approach for defining higher-order roles in a social hypernetwork and characterize higher-order interactions.

We define the concept of an archetype, a general characterization of higher-order roles that serves as a “template” for identifying nodes representative of specific roles within the context of hyperedges. These archetypes provide a structured way to categorize and understand the diverse roles users play in group interactions. Then, we propose a general hyperedge characterization function, which allows one to characterize a hyperedge w.r.t. features of the nodes contained in it.

To validate the effectiveness of our proposed framework, we conduct an exhaustive experimental campaign on the understudied social platform Scored.co, which offers a rich, yet understudied dataset for analyzing user interactions in a hypernetwork context, making it an ideal testbed for our framework. Our experiments demonstrate the utility of our approach in characterizing user roles within social hypernetworks.


Summarizing, the main contributions of this work are as follows:


    
  * We define the concept of archetype, that is, a high-level description of user roles that serves as a “template” characterizing nodes representatives;
    
  * We introduce an approach to characterize higher-order entities, such as hyperedges, taking into account exhibited features and their higher-order dynamics;
    
  * We design an exhaustive experimental evaluation in which we assess the effectiveness of our framework on the understudied social platform Scored.co. Notably, this is also the first work studying the structural properties of Scored.co.
    


The outline of this paper is as follows: in the sec:related-literature section, we provide a detailed overview of related literature. In the sec:framework section, we provide the formalization of our proposed approach by defining the concept of archetype first, and the hyperedge characterization function after. In the sec:experiments section, we illustrate the experimental campaign we carried out to evaluate the effectiveness of our framework. Finally, in the sec:conclusion section, we draw our conclusions and delineate some future works.



§ RELATED LITERATURE






  §.§.§ Hypernetworks and Role Identification


The identification of roles within social networks stands as a fundamental aspect for understanding the dynamics and function of these complex systems <cit.>. In traditional social network analysis, roles are often determined based on patterns of connectivity that provide insights into the influence, responsibilities, and grouping of individuals within the network <cit.>. This analysis is crucial for applications ranging from the characterization of polarization <cit.> and user behaviors <cit.> to the analysis of biological networks <cit.> and animal populations <cit.>. In different cases, role identification is also framed within the task of discovering influential users in a social network. In light of this, we refer the interested reader to the comprehensive survey proposed in <cit.>.

Nevertheless, extending role identification into hypernetwork scenarios poses many challenges. Unlike traditional networks, hypernetworks consist of interactions, represented by hyperedges, that can encompass multiple nodes simultaneously. 
Higher-order interaction models can better represent complex real-world systems  <cit.>; however, the complexity of higher-order interactions can introduce difficulties in accurately defining and identifying roles, as traditional centrality metrics and methodologies (e.g., community detection algorithms) often fall short in capturing the multidimensional nature of hyperedges <cit.>.
Despite the relevance of such a task, to the best of our knowledge, there are no existing papers that explicitly discuss role identification within the context of hypernetworks.

Given this gap in the literature, the rest of the section will concentrate on providing an overview of the methodologies and findings from the conventional settings in dyadic interactions, which may serve as a baseline for future exploration into hypernetwork-specific role identification. 




  §.§.§ Role Identification in Pairwise Networks


The general task of role identification consists in capturing the multifaceted characteristics of a node's role in a pairwise network. The work in <cit.> identifies roles in undirected, unweighted networks by evaluating node importance using correlated indicators like degree and centrality measures. 
A node's role is then determined by comparing its indicator relationships to statistical correlations within the overall network. While this work shares the task of role identification with ours, the process does not involve group-based indicators leveraging the concept of “groups" such as communities or subgraphs.

The work in <cit.> frames the node role identification tasks as a node classification approach based on random walks.



Similarly, framed within a classification tasks, the authors in <cit.> focus on the so-called “answer-person” role on Reddit, characterized by users who predominantly respond to questions posed by others, with minimal engagement in broader discussions.


Both the works in <cit.> and <cit.> share a similar aim with ours. Nevertheless, they rely on supervised approaches on pairwise graphs. Also, the methods might not be straightforwardly generalized to higher-order interactions. 

The approach proposed in <cit.> focuses on the identification of node roles in dynamic social networks as a sequence of different types of activities, by leveraging pattern subgraphs and sequence diagrams.

Doing so, the authors are able to capture roles such as the “gossipmongers", i.e., users who replicate every received message at least three times. The work in <cit.> can be considered as orthogonal to ours, as an hypergraph mining approach could also be applied in our context to enhance role identification.
The authors in <cit.> outline a process for identifying user roles in Enterprise Social Networks (ESNs), using a mix of design science research and data mining, involving data collection, preparation, and evaluation, with user roles identified through statistical analysis, including PCA and clustering.
The work in <cit.> shares a similar focus with ours, although using a different methodology.
The identification of emergent leadership roles is addressed in <cit.>, where the authors examine collaboration patterns between teams and propose the so-called “leadership index", a combination of centrality measures including closeness and betweenness. 

In <cit.>, the authors propose a mixed-methods methodology for user role identification, focusing on dynamic user profiling. 
Several types of special users are defined and identified to support information dissemination.
The work in <cit.> proposes an approach based on computational methods and questionnaire-based evaluation to quantitatively describe user features. Such work parallels ours in the sense that it considers the network's features, too.

In <cit.>, the authors study community and user stereotypes in Reddit. Here, with the term stereotype, the authors refer to the tendency to classify people into groups and to associate each group with a general idea or a label. The authors propose a rule-based approach based on different quantitative views of the data, and define author stereotypes on the basis of two orthogonal taxonomies, namely, the number of posts, and the number of comments of an author. While the work in <cit.> and ours share some similarities, the employed methodologies are substantially different. The former defines stereotypes based on a single quantitative measure called score, which is inherent to the social platform itself. Instead, our approach proposes a general framework in which the characterization of users and the identification of their role are both defined by their feature-rich, higher-order surroundings. 
In <cit.>, the authors identify five distinct social roles in a specific community on Reddit. Among them, we cite the “knowledge broker”, i.e., a member who introduces knowledge to the community by sharing links, and the “translator”, i.e., a member who contributes academic knowledge. While the contribution in <cit.> is notable and similar to ours for considering various aspects of user characterization, these roles are specific and applicable only to the analyzed community.


Finally, it is worth mentioning embedding-based methodologies <cit.>, which could potentially be adapted to hypernetwork scenarios. For instance, in <cit.>, the authors introduce a technique that incorporates node attributes to generate embeddings that capture the similarities based on neighborhood structures. Instead, in <cit.>, the authors leverage both node and structural embeddings to detect nodes impersonating social bots. Indeed, node embeddings based on structural properties and exhibited attributes attracted substantial interest in the last years <cit.>. While these methods use embedding techniques that effectively summarize and exploit the structural information within traditional networks, their application to hypernetworks might not be straightforward.

Our approach diverges from embedding-based methods in the sense that it is specifically tailored to the unique features of nodes within hypernetworks, such as their exhibited features w.r.t. the hyperedges containing them.


By focusing on this, without relying on an embedding, our methodology seeks to offer a more direct and fitting analysis of node roles.




  §.§.§ “Understudied" Social Platforms


We conclude this bird's-eye view of the related literature by focusing on the last contribution of our paper, that is the study of an understudied platform. Indeed, the reasoning provided in the above discussion of related work is also valid for this aspect. In fact, there is a moderate number of works on understudied platforms, and generally these are limited to present a dataset collection about the considered platform <cit.>. For instance, a large-scale dataset of the social platform Scored.co has been presented in <cit.>, in which the authors studied aspects such as posting activity and user characterization, as well as the phenomenon of user migration from other platforms. A recent social platform, called Bluesky, has been studied in <cit.>. In the former work, the authors presented a comprehensive study of the social structure of the platform, as well as posting activity and content analysis. Also, the complete post history of over 4M users has been released. Similarly, the latter work studied users' political leaning and ideological polarization on BlueSky while also presenting a characterization of the network topology over time. Finally, in <cit.>, the authors released a dataset targeting the Indian microblogging platform Koo. The dataset consists of more than 72M posts and 75M comments, with related features such as shares and likes. Also, a thorough overview of the platform is presented, consisting of a discussion of the news ecosystem on the platform, hashtag usage, and user engagement.



§ MATERIALS AND METHODS



In this section, we introduce our framework to characterize nodes and hyperedges in a social hypernetwork, as well as the definition of roles, called archetypes, in such a network. We start by providing some background, which is useful to understand the frame of our context. Then, we introduce the definition of archetypes and our method to calculate them. Finally, we detail the proposed framework to characterize nodes and hyperedges.



 §.§ Background


A hypergraph H=(V,E), V = {v_1, …, v_n} is the set of nodes, E = {e_1, …, e_m} is the set of hyperedges, where e_j ⊆ V, for j = 1,…,m. 
A visual representation of a hypergraph is depicted in Fig. <ref>.
The order of H is n = |V|, while the size of H is m = |E|. We denote with E_v_i the set of hyperedges containing v_i, that is E_v_i = {e_j ∈ E : v_i ∈ e_j}. Given a hyperedge e_j ∈ E, its size is the number of nodes belonging to it, that is |e_j|. The degree of a node v_i, denoted as deg(v_i), is the number of neighbors of v_i; a node v_k ∈ V is the neighbor of node v_i if and only if there exists at least one hyperedge e_j which v_i and v_k both belong to. The hyperdegree of a node v_i, denoted as hdeg(v_i), is the number of hyperedges to which v_i belongs. A graphical depiction of a hypergraph is given in Fig. <ref>.

We use a hypergraph H to model a social hypernetwork. Here, nodes are users of the social hypernetwork, while a hyperedge represents a discussion between users. In what follows, we will refer to the social hypernetwork simply as hypernetwork.

Moreover, features can be associated with users, and these are generally based on the content the users interact with and the discussions they participate in. To formalize them, we employ a set of features F = {f_1,…,f_l} characterizing the nodes in H. Given a node v_i ∈ V, F_i = {f_1_i,…,f_l_i} represents the values of each feature of v_i, that is f_k_i indicates the value of the k-th feature of v_i. We assume features can be either numerical or categorical and that numerical ones are always normalized. Also, with H_F, we indicate the hypergraph equipped with the set of node features F.



 §.§ Archetypes for Nodes Characterization



The first aim of our approach is the definition of archetypes. An archetype serves as a “template” to represent nodes characterized by a certain subset of features from the set F. 

The concept of an archetype is pivotal in characterizing higher-order node roles based on subsets of hyperedges. Often, the study of a representation of nodes in a social network involves the definition of a taxonomy or a more thorough analysis of the behavior of such nodes <cit.>. Instead, our definition of archetype enables a more general characterization of possible behavioral dynamics occurring among nodes, and does not rely on a single attribute.

Let F_A = {f_1, …, f_p}⊆ F be a subset of features. Then, an archetype A is defined as a tuple of values based on the features indicated by F_A. Formally, let A = ⟨ a_1, a_2, …, a_p ⟩, where a_j is the value of the j-th feature from F_A for the archetype A. Then, each a_k, for k = 1,…, p, represents a particular value of the k-th feature from F_A that characterizes the nodes associated with this archetype. As an example, let F_A = {f_, f_}, where f_ (resp., f_) is the value of a quantitative feature indicating the average sentiment value (resp., average toxicity value) expressed by a user. Such features are extensively used in various data science-based studies and can be easily computed via classical methods such as VADER <cit.> and Detoxify <cit.>. Suppose f_∈ [-1,1], where -1 (resp., 1) indicates a mostly negative (resp., mostly positive) average sentiment, and f_∈ [0,1], with higher values indicating a high degree of toxicity. Then, various archetypes could be defined based on these features. For instance, we could define the archetype A = ⟨ -1, 0.5 ⟩ that describes a template for users that are extremely negative and are toxic on average, and we could name this archetype as Cinically Commenter; instead, the archetype A = ⟨ 1, 0.5 ⟩ would be a template for users that are extremely positive and exhibit moderate toxicity, and we could name it as Overzealous User. Also, it is noteworthy that, for the sake of presentation, here the definition of archetype is only given w.r.t. nodes. Nevertheless, in case of features being also available for hyperedges, then the same definition applies, and archetypes for the latter can be defined.

Essentially, an archetype A can be viewed as a prototypical example of nodes that share similar feature values. To simplify the analysis and categorization of archetypes, instead of taking into account the feature values directly, we can map them into categorical values, indicating the state of the feature for the particular archetype. This approach involves setting a series of thresholds T = ⟨ t_1, …, t_p ⟩, where the value t_j has the same domain of the feature f_j in F, a set of labels L and a labeling function ι. Then, given an archetype A = ⟨ a_1, …, a_p ⟩, by applying to each feature a_j the corresponding threshold t_j from T via the labeling function ι, we derive the archetype A_T = ⟨ a^t_1, …, a^t_2 ⟩, where ι(a_j,t_j) = a^t_j is a label in L. Practically speaking, A_T represents the same archetype as A, but its representation is now built over a specific set of labels. Let us take the aforementioned archetype A = ⟨ 1, 0.5⟩, defined over the set of features F_A={f_, f_}, and representing Overzealous Users. Suppose we set T = ⟨ 0, 0.75 ⟩, and we set L = {, }. Also, suppose we define the labeling function ι as


    ι(a_j,t_j) =    if a_j ≤ t_j 
       otherwise


Therefore, in this example, we derive A_T = ⟨, ⟩, which can be seamlessly interpreted as representing users exhibiting a high sentiment value and a low toxicity value. Note how the labeling depends on the thresholds T, thus allowing for flexibility that can accommodate different contexts of analysis.

Finally, we are now able to state when a node is represented by a given archetype. Given an archetype A=⟨ a_i, a_2, …, a_p ⟩, based on a subset of features F_A = {f_1, …, f_p}⊆ F, we want to effectively understand what nodes are represented by it. Let v_i ∈ V be a node, and let F_A(v_i) = ⟨ f_i_1, f_i_2, …, f_i_p⟩ be the feature vector of node v_i according to the features selected in F_A. The node v_i can be considered as represented by the archetype A if F_A(v_i) is sufficiently close to A, according to a predefined distance metric d. We express it as d(F_A(v_i), A) ≤ϵ where ϵ is a small positive threshold value that determines the acceptable distance between the node's feature vector and the archetype. Note that both A and F_A(v_i) can be considered as vectors of the same length p; thus, classical distance metrics, such as the cosine similarity <cit.>, can be used. Furthermore, as we will see in the experiments, in the simplest case also a feature-wise comparison can be used to assess when a node can be considered as represented by a given archetype.
















































  §.§.§ A Characterization for Archetypes



To comprehensively understand and categorize user archetypes, we decide to observe them through three different expressions, namely, (i) emotional, (ii) psycho-emotional, and (iii) moral expressions. By leveraging well-established psychological theories and lexicons, we aim to create detailed profiles that reflect the peculiar ways in which users interact and participate within the social platform. In what follows, we describe in detail these three expressions, which we will subsequently use in our experiments.



  
*Emotional Profiles We aim to characterize archetypes based on the emotions they express.
To do so, we refer to the psychological theory of emotions <cit.> developed in 1980 by the American psychologist Robert Plutchik.
This theory identifies eight basic emotions — joy, trust, fear, surprise, sadness, anticipation, anger, and disgust — and claims that all other emotions derive from a mixture of these primary ones.
To quantify feelings expressed by Scored users, we leverage the NRCLexicon (National Research Council Lexicon) <cit.>, a resource containing over 14,000
English words and their associated emotional ratings according to Plutchik’s theory.
This dictionary is further expanded in [<https://github.com/metalcorebear/NRCLex>] to include WordNet synonyms, reaching over 27,000 terms. 
For each of the users' texts, we compute emotion scores, and normalize them in [0, 1]. 
In this context, 0 indicates texts that do not elicit any emotion, while 1 signifies texts that strongly evoke the specified emotion.



  
*Psycho-emotional Profiles We also characterize how user archetypes relate to their surrounding social environments.
To do so, we refer to the PAD model (Pleasure, Arousal, Dominance) introduced by Mehrabian and Russell in 1974 <cit.>.
According to the PAD model, three dimensions characterize the perception an individual has of the environment in which she finds herself.
Pleasure (sometimes referred to as valence) concerns whether an individual perceives the environment as enjoyable or not. 
Arousal measures how stimulating the environment is for the individual. 
Dominance indicates whether the individual feels in control of the environment.
To operationalize these dimensions, we leverage the VAD Lexicon (Valence, Arousal, Dominance) <cit.>. 
This resource contains over 20,000 English
terms, along with their associated valence/pleasure, arousal, and dominance values. 
For each of the three dimensions, we associate each text with the total score of its words.
Then, we normalize results in [0,1], where 0 implies an absence of the corresponding dimension, and 1 implies the strong presence thereof.



  
*Moral Profiles
We aim to characterize archetypes based on the moral dimensions that emerge from the content they produce.
We rely on the Moral Foundations Theory, a psychological framing rooted in cultural anthropology that postulates the existence of five universal moral dimensions <cit.>: authority/subversion, care/harm, fairness/cheating, loyalty/betrayal, and sanctity/degradation.
Each dimension is composed of a virtue (e.g., loyalty), and a corresponding vice (e.g., betrayal).
Virtues can be understood as follows, while vices can be considered their opposites.
The concept of authority can be defined in relation to specific traits, such as deference to higher authorities, in order to maintain group cohesion. 
Similarly, the concept of care can be understood in terms of nurturing and protection. 
Fairness can be conceptualized in terms of equal treatment and reward. 
Loyalty can be understood in relation to the prioritization of one's group and alliances. 
Finally, sanctity can be defined in terms of the maintenance of the sacredness of the body and the avoidance of moral contamination.
We operationalize this framework via the eMFD (Extended Moral Foundations Dictionary), a lexicon containing more than 3,000 words <cit.>.
Each word has an associated score in [-1, 1] for each foundation, ranging from strong vice outage (-1) to strong virtue outage (1).



 §.§ Analyzing Higher-Order Entities



While archetypes are a particular yet effective way to analyze entities within the social platform, they mainly focus on the node features, whereas interactions are not taken into account.

Hence, we define here a general characterization function to characterize nodes and hyperedges according to their exhibited features and higher-order dynamics. 

Without loss of generality, we propose the definition of such a function w.r.t. hyperedges. The same can be also applied to nodes.

Let H_F = (V,E) be a hypergraph equipped with the node features set F. We denote with ω a function which we call the hyperedge characterization function. The need for a function such as ω addresses the challenge of characterizing a hyperedge w.r.t. the nodes contained in it.
Let us recall that our approach deals with analyzing higher-order entities. To do so, we exploit the representation of node relationships via hyperedges. While it effectively captures the higher-order structural interactions between nodes, it might not be sufficient in acquiring insights into the semantics of such interactions. Therefore, we focus on the latter aspect through a characterization of hyperedges that is based not only on the contained nodes but also on their features. Formally, the domain of our hyperedge characterization function is E. ω takes in input a hyperedge e ∈ E and returns a value ω(e_j), which we call its characteristic value. Such value depends on the actual implementation ω: in fact, ω is general, and different approaches can be exploited to accommodate the hyperedge characterization. Furthermore, to address the aforementioned challenge, there are cases in which ω should be defined to consider the values of each node's features contained in the considered hyperedge. Given a feature of interest f_k, we write ω^k to denote that the actual implementation of ω considers the feature f_k.

In the following, we propose various specializations of ω and a brief rationale for each of them. Some of these specializations are used in the sec:experiments section. We separate them in three families, namely, (i) numerical-only definitions, (ii) categorical-only, (iii) structural-based ones. In describing each of them, we assume we are interested in characterizing a hyperedge e ∈ E. Also, there are different specializations that are intended to be exploited when the analysis we are carrying out is feature-oriented. Therefore, in these cases, we assume having a feature f_k of interest.

















































  
*Numerical-only specializations The following specializations are intended to be used when numerical features are considered within the investigation. Therefore, here we assume the feature f_k of interest is numerical. Some specializations are:


    
  * Statistics Descriptors: common statistics descriptors such as mean, median, mode, variance, and standard deviation of the feature f_i among the nodes in e can be easily computed. For instance, the mean would be simply defined and denoted as ω_avg = ∑_u ∈ e f_i_u/|e|.
    
    
  * MAD: calculates the Mean Absolute Deviation of the feature f_i among the nodes in e, that is ω_mad(e) = ∑_u ∈ e|f_i_u - ω_avg|/|e|.
    
    
  * Gini Coefficient: employs the Gini Coefficient to compute the dispersion of the values of f_i among the nodes in e, that is ω_Gini(e) = ∑_u ∈ e∑_v ∈ e, u ≠ v|f_i_u - f_i_v|/2 f_i |e|^2. 




  
*Categorical-only definitions Different from the previous ones, the following specializations are intended to be used when categorical features are considered. Therefore, here we assume the feature f_k of interest is categorical. Some specializations are as follows:


    
    
  * Entropy: measures the cohesion of the values of f_i among the nodes in e. We define and denote it as ω_entr(e) = -∑_u ∈ er_i_ulog(r_i_u), where r_i_u denotes the proportion of the value of feature f_i_u over all nodes in e.
    
  * Gini Impurity: measures the probability of incorrectly classifying a randomly chosen element in the dataset if it were randomly labeled according to the distribution of all categories in the dataset. Here, we employ it on the feature f_i, thus we define and denote it as ω_GiniImp(e) = 1 - ∑_u ∈ e r_i_u^2, where r_i_u denotes the proportion of the value of feature f_i_u over all nodes in e.




  
*Structural-based specializations

The structural-based specializations focus on characterizing a hyperedge based on its exhibited structural properties rather than only focusing on the features of the contained nodes. Some proposed specializations are the following:


    
  * Hyperedge Size: the measure is simply given by the size of the hyperedge e, that is ω_size(e)=|e|, regardless of the feature f_i.
    
    
  * Purity: Here, the measure indicates the homogeneity of the values of f_i in e, that is ω_purity(e)=max(count(f_i_u) : u ∈ e)/|e|.

    

    

    
  * Cohesion: the characterization of the hyperedge e is given by the mean pairwise similarity of the value of the feature f_i for each pair of node u, v ∈ e. Formally, given a similarity function sim(·,·) defined for two features, we define ω_cohes(e) = 2/|e|(|e|-1)∑_u ∈ e∑_v ∈ e, v ≠ u sim(f_i_u, f_i_v).

    
  * Interaction potential: the hyperedge e is characterized by its capacity to create external connections, i.e., connections that span outside the hyperedge. It measures the extent to which the hyperedge facilitates interactions beyond its mere composition. Formally, ω_intpot(e) = |{ v ∈ V ∖ e : ∃ u ∈ e, v ∼ u }|/|e|, where v ∼ u indicates that v and u are neighbors within a hyperedge different from e. The denominator can be |e|, i.e., only the nodes in e are taken into account, or |V ∖ e|, i.e., ω_intpot is sensitive to the order of the hypergraph.

    

    

    

    

    

    






























§ EXPERIMENTS



In this section, we present an experimental evaluation carried out to assess the efficacy of our framework. In what follows, we first provide a characterization of the understudied dataset regarding the social platform Scored.co. We study archetypes based on platform-specific and linguistic features, and we illustrate how our proposed framework allows for a comprehensive analysis of them. Then, we focus on the constructed hypernetworks, and we provide a characterization of discussions, that is, hyperedges, through our proposed hyperedge characterization function.



 §.§ Dataset







We focus on the Scored.co social platform, which hosts a network of communities that users can create and join. The platform shares different similarities with the more common platform Reddit[<https://reddit.com>]. Users can join communities related to various topics, from politics to memes, and upvote content they like, and downvote content they dislike. The number of upvotes minus the downvotes constitutes a post's score, i.e., a value that indicates the content's usefulness and/or relevance to the discussion.

Interestingly, Scored.co acted as a shelter for many far-right/alt-right communities after the 2020 massive Reddit bans <cit.>. Some of these maintained their names and key figures, e.g., , , ultimately acting as a continuation of the original ones. This platform has allowed users to freely share their views without the risk of being banned. 
Scored.co is an emerging yet understudied platform, potentially hosting dangerous content. Despite its growing user base, it has received relatively little academic attention compared to other social media platforms, as we pointed out in the sec:related-literature section. The loosely moderated nature of some communities can lead to the spread of misinformation, hate speech, and radical ideologies. 
This makes Scored.co a platform of interest for researchers studying online extremism, digital sociology, and the impacts of social media on public discourse.

To build the dataset our experiments are carried out on, we collect data from the Scored.co platform via the official API[<https://help.scored.co/knowledge-base/getting-started-with-the-api/>]. To retrieve as much content as possible, we use a breadth-first search technique by first gathering all posts and comments authored by , one of the platform's admins, which is one of the most active users on the platform. Then, the content is parsed to retrieve other users and their related content. We repeat this procedure until no new user is found. Eventually, we find 207,554 unique users (referenced to by their username), 4,398,074 posts, and 36,978,685 comments. Our data collection process started on June 9th, 2024, and ended on June 22nd, 2024. The final dataset comprises higher-order interactions on Scored.co starting from the platform's launch on October 15th, 2019, and ending on June 1st, 2024.

  

To improve reproducibility and share our findings, we release the dataset on Zenodo <cit.>. The dataset is anonymized and provides: 

    
  * Higher-order interactions extracted from discussion threads. 
    Each thread/hyperedge is labeled with the community it is extracted from (e.g. ), and the year and month when the root post was created;
    
  * User profiles, namely a set of features characterizing each user at every time stamp.

A Python data collection script is also enclosed in the repository.
In the first release of this dataset, only user/thread data leveraged in this study is shared (i.e., 2023 data from the c/TheDonald community, see next section).
However, we plan to update it in the near future to include statistics and interactions for all users, leveraging Zenodo's versioning system.
Our dataset complies with the fair guidelines for data sharing <cit.>, which are as follows:

    
  * Findable. We release the data on Zenodo [<https://zenodo.org/>], a service managed by CERN that also assigns a DOI to the repository, ensuring findability. Moreover, we indexed it also on the SoBigData Research Infrastructure (URL: <http://sobigdata.eu/>), an EU-funded RI providing curated datasets and algorithms advocating open science;
    
  * Accessible. The dataset is freely accessible to anyone with an internet connection;
    
  * Interoperable. All data is released in CSV files, which can be read and manipulated with most programming languages;
    
  * Reusable. Files, fields, and data collection methods are described in a dedicated document released within the repository, ensuring reusability.






 §.§ Higher-order Interactions and Analysis


Given the previously described dataset, we are now able to construct the hypernetworks which constitutes the basis of the experiments.
We start by constructing a hypernetwork H = (V,E), where V denotes the set of Scored.co users who participated in at least a discussion, while E denotes the set of hyperedges encoding discussion threads. That is, a user v appears in V if they participate in at least one discussion. Also, we discard all hyperedges of size 2 due to the fact that we are interested in studying higher-order interactions. This means that, for all e ∈ E, |e| > 2.


Since discussion topics on Scored are varied, ranging from politics to sports, memes, and more, we decided to center our analysis on discussions taking place in  during 2023. 
This community is a controversial and highly active online group that originally formed on Reddit before migrating to platforms like Scored.co <cit.>. We chose it because it is the oldest and most active community on the platform, covering 74% of all collected posts and comments. 
This allows us to maintain topic continuity without sacrificing large portions of the data.

Note that H is a hypernetwork that represents the whole dataset. To enable analyses on the time dimension, we construct a hypergraph H_t = (V_t, E_t), for each month t = 1, …, 12, containing users and discussions relative to month t.

In Table <ref>, we report summary statistics of all the constructed hypernetworks. The table reports basic statics, such as nodes and hyperedges number |V| and |E|, respectively, the largest hyperedge size max_|e|, the average hyperdegree hdeg, and the average degree deg. Also, for each pair of adjacent timestamps 1 ≤ t, t_+1 < 12, we report the Jaccard similarity index between node sets. The last row of the table reports the same information for the hypernetwork H.



From the analysis of this table, we can observe different insights. First off, the aggregated hypernetwork ℋ includes interactions among roughly 20% of all discovered users. Indeed, on average, users participate in 113 discussions (hyperedges) throughout the year, and in 18-20 discussions each month. Scored.co users are exposed to ∼ 1K unique peers, with a monthly average of 300-400, thus showing locally dense patterns consistently over time. We can also observe how the largest hyperedge size exhibits some variations but remains in the same order throughout the whole period. This indicates the presence of a highly connected subgroup at certain timestamps. The average hyperdegree hdeg remains relatively stable across timestamps, indicating that on average, each node is part of ∼ 20 hyperedges. The average degree deg remains fairly consistent, too. To complement these statistics, we try observing with a more detailed look the distribution of hyperdegree and hyperedge size, which are depicted in Figures <ref> (a) and (b), respectively. From the analysis of these figures, we can see that both show the typical power-law behavior observed in many real-world hypernetworks <cit.>. We observe how a few users participate in a large number of discussions, while most users participate in just a few. The hyperedge size distribution (Fig. <ref> (b)) displays a similar shape, with mostly small discussions and a few large ones. We also note how these patterns are consistent over time, with 44% interactions of at least size 5, 15% interactions of at least size 10, and 2% interactions of at least size 50. Finally, in Table <ref>, we can observe that the overlap between adjacent timestamps, calculated via the Jaccard similarity index between subsequent node sets V_t and V_t+1, is large and coherent over time, stabilizing on 64% on average. Overall, the studied part of this platform emerges as a locally dense online community with temporally stable structural patterns and significant user engagement, which makes it interesting to study with our approach.








 §.§ Identification and Analysis of Archetypes



We recall that a core aspect of our approach relies on identifying archetypes, namely typical patterns that can give us approximate descriptions of individual element behaviors. As we also noted in sec:archetype section, archetypes can be easily defined for nodes and hyperedges. In this section, we aim to characterize user behaviors. Therefore, henceforth we refer to archetypes as related to nodes/users.

To define user archetypes, we combine platform-specific and linguistic features. Specifically, we outline eight archetypes based on these features and their values. The features we decide to use are, namely, (i) score, (ii) sentiment, and (iii) toxicity. As far as the score is regarded, we compute the average post score for each user, then we normalize values in [0,1]. For the sentiment feature, we use the well-known VADER algorithm <cit.>, a rule-based sentiment classifier that assigns each text positive, negative, neutral, and compound sentiment intensity values. Out of these, the compound sentiment is a score in [-1, 1] going from the most extreme negative (-1) to the most extreme positive (1) sentiment. For our purposes, we compute the average compound scores for each user and rescale values in [0, 1]. Moreover, for the toxicity feature, we compute its values with Detoxify <cit.>, a neural toxicity detection model that returns the probability that a text contains toxic language, and also in this case, we normalize the obtained values. Finally, to discern between high and low values for each feature, we set a threshold of 0.5, such that when a feature has a value higher than this threshold, it is considered high, and vice-versa.

The list of the eight archetypes defined is depicted in Table <ref>. Here, we can observe how setting the various combinations of the state of a feature (high or low) creates different archetypes. We defined these archetypes in order to encapsulate the inherent semantics of the discussions taking place in the studied dataset. For instance, the first row is the so-called Community Hero, which represents users who consistently contribute positively to the discussion by maintaining a positive and supportive attitude without engaging in toxic behavior. A detailed characterization of these archetypes is given in the following section.








  §.§.§ Archetype Characterization


Having defined the archetypes, we are now able to provide a characterization of them, based on multiple psychological and social dimensions. More in detail, we describe the behaviors of these types with respect to the different aspects highlighted in the <ref> section. We recall that these aspects are, namely, the (i) environment perception, the (ii) emotions, and (iii) morality. 



To control for class size imbalances, we compare average values for multiple features across the top 10 most archetypal users for each archetype, i.e., the most representative users for each category. We rank users according to their typicality, which is defined as follows:

    Typicality(u) = ∏_f α_f f(u)


Here, u is the target user; f is a function computing u's characteristic score, sentiment, or toxicity; alpha_f is a coefficient that equals 1 if u's archetype has a high value for f and, conversely, -1 if u's archetype has a low value for f. As an example, a Community Hero's typicality is maximized when its score is 1, its sentiment value is 1, and its toxicity value is 0. Thanks to the definition of typicality, we are now able to characterize each archetype with respect to the aforementioned aspects. Along with the characterization, in Fig. <ref>, we depict the profiles of each archetype. The top part of the figure represents the profiles according to Plutchik's wheel of emotions. The middle part, instead, represents the profiles according to the Pleasure/Arousal/Dominance model. Finally, the bottom part depicts the profiles according to the Moral Foundations theory.





  
*Benevolent Underdogs. With respect to their psychological features, Benevolent Underdogs are characterized by high trust (0.18), joy (0.14), and anticipation (0.11), low arousal (0.44) and high valence (0.66). 
These users have a positive and optimistic outlook, showing happiness and hopefulness in their interactions. 
From their feature values, a possible interpretation can suggest them as users focusing on creating a supportive environment and  emphasizing social cohesion.
It is important to note that, for this archetype as well as all the following ones, these descriptors can lead only to qualitative suggestions and interpretations of their behaviors.



  
*Community Heroes. Such users are characterized by high trust (0.17), joy (0.12), and anticipation (0.14), low arousal (0.45) and high valence (0.64). 
Thus, their psychological profile is similar to Benevolent Underdogs, with a generally positive outlook. 
From a moral focus perspective, they mostly emphasize loyalty and, to an extent, authority.
Community Heroes and Benevolent Underdogs essentially behave in the same way, although the former are more recognized.
Thus, we hypothesize that Community Heroes often provide support in more prominent and/or critical situations, gaining greater acknowledgement from the community. 



  
*Controversial Stars. With respect to their psychological features, Controversial Stars show signs of joy (0.13), trust (0.12), and anger (0.10), as well as high valence (0.60). 
This indicates that while they have a positive emotional tone and are capable of experiencing happiness and trust, they can also exhibit anger.
This leads to an interpretation of this archetype as an emotionally charged one, with a tendency to express both positive and negative tones.
Moreover, they mostly emphasize loyalty and authority moral dimensions.




  
*Infamous Celebrities. Infamous Celebrities show high signs of anger (0.14), fear (0.14), and disgust (0.10).
Arousal (0.53) and Dominance (0.50) are average, while Valence is slightly below average (0.48).
Their complex psychological profile hints at expressions of negative tones, perhaps toxic interactions.
High levels of fear could indicate a tendency to anticipate negative outcomes, which can make their discourse more alarmist. 
Disgust contributes to a critical and often harsh tone. 
Average arousal and dominance could suggest a moderate level of engagement with the environment, whereas their slightly below-average valence could be a sign of negative interactivity.
Moral dimensions of care, fairness, loyalty, authority, and sanctity display negative values.

This moral outlook, combined with their negative emotional profile, contributes to their provocative, even controversial, behavior.



  
*Malcontents.
Malcontents are characterized by high values for disgust (0.13), and anger (0.11).
This is paired with below-average values for Valence (0.39) and Dominance (0.45), and high Arousal (0.61).
This emotional profile could reveal a full negative profile.
The archetype shows high arousal as well as strong emotional responses and engagement, but also low valence and dominance.
This could mean the users belonging to it often display dissatisfaction, leading to discontented interactions. Morally, Malcontents use vice-oriented language, focusing on sanctity/degradation and care/harm.
Their low values in these areas could indicate strong criticism and a tendency to vent.




  
*Quiet Critics.
The Quiet Critics' psychological profile outlines an emotional landscape characterized by high levels of sadness (0.11), fear (0.14), and anger (0.12). 
These predominant features suggest a dissatisfaction with their environment.
Anger, although present, paired with low toxicity values, could suggest a profile expressing criticism in a non-confrontational way rather than being directly hostile.
With average levels of Valence, Arousal, and Dominance, their emotional responses are steady an moderated. 

From a moral perspective, Quiet Critics' language frequently involves vice-oriented terms, particularly focusing on sanctity/degradation, care/harm, and loyalty/betrayal. 
Their emphasis on sanctity could interestingly suggest a concern for moral purity, whereas the focus on care/harm could highlight preoccupation with injustice.


Their attention to loyalty/betrayal could indicate a sensitivity to perceived trust, which might align with their cautious-yet-critical nature. 



  
*Respected Critics.
The Respected Critics' psychological profile is similar to those of the Quiet Critics, mostly eliciting sadness (0.8), fear (0.14), and anger (0.12). 
Sadness and fear highlight their anticipation of negative outcomes, possibly making their critiques cautionary. 
Anger drives their behavior, moderated by average levels of valence, arousal, and dominance.
Their moral profile resembles that of the Quiet Critics. 







  
*Positive Provokers. Positive Provokers are characterized by high levels of trust (0.18) and joy (0.13). They show average Dominance (0.54) and Arousal (0.49) values, as well as high Valence (.60).
Their high trust and joy could suggest they generally expect positive interactions from others and have a positive disposition.
The high Valence reflects their overall positive emotional tone. However, despite these positive traits, their high toxicity scores reveal a tendency to a duality, that we could interpret as a confrontational/provocative communication style
This dichotomy of positivity and toxicity creates a complex profile where users intentions may be good, but their execution can be damaging to others.



  §.§.§ Archetypes Transitions


In this section, we aim to explore the temporal dimension of user interactions within the Scored.co community and understand whether significant transitions exist between the identified archetypes. By analyzing these transitions, we can gain insights into how user behaviors evolve over time. This analysis not only helps in understanding the fluidity of user roles but also potentially provides valuable information for targeted interventions and community management.

In order to carry out this analysis, we propose the following methodology. Given the imbalanced class sizes (see Table <ref>) throughout the observation periods, we employ a null model to test the transition probabilities between archetypes. This null model computes the expected transition probabilities on a copy of our dataset where archetype labels are shuffled. The detailed steps are as follows:


    
  * We generate N = 500 shuffled copies of the dataset, randomizing the archetype labels while preserving the overall distribution of interactions and activities;
    
  * For each archetype pair (A,B) in the shuffled copies we compute the transition probability P(B|A), which represents the likelihood of a user transitioning from archetype A to archetype B;
    
  * We calculate the mean and standard deviation of P(B|A) across all shuffled copies. These statistics are used to compute z-scores and p-values for the observed transition probabilities in the original dataset;
    
  * Finally, transitions with p-values less than 0.01 are considered statistically significant, indicating that these transitions occur at a higher-than-expected rate compared to the null model.


Fig. <ref> depicts the observed statistically significant transitions, letting us observe several insights from the transitions between archetypes.
One notable transition is from Benevolent Underdogs to Respected Critics, occurring at a significant rate of 39.23%. Benevolent Underdogs are characterized by positive sentiments and low post scores, typically engaging in supportive and encouraging behaviors. This transition can indicate that even users who start with low recognition can, through consistent positive contributions, build credibility and eventually gain influence within the community. Community Heroes, another archetype defined by high trust, joy, and engagement, show a tendency to transition to Benevolent Underdogs or, more rarely, to Positive Provokers. The slight shift to Benevolent Underdogs suggests that Community Heroes, despite their initially high engagement and recognition, may sometimes experience a reduction in visibility or engagement, leading them to adopt a more low-profile but still positive role. Also, we observe an occasional drift towards more toxic behaviors. Infamous Celebrities, which have high post scores and negative sentiment, exhibit a significant probability of remaining in their contentious roles (13.71%). However, there is also an 11.71% likelihood of transitioning to Malcontents. This shift reflects a potential decrease in engagement and influence, where high-profile negative behavior can lead to more discontent and a less active state. This transition from Infamous Celebrities to Malcontents suggests a trajectory where persistent negativity can diminish a user's central role within the community. Malcontents, who are driven by negative emotions and low post scores, present an interesting case with a substantial 33.29% chance of becoming Respected Critics. This transition implies that their critical nature can eventually earn them credibility, provided they moderate their toxic behaviors. Another significant transition is observed from Quiet Critics to Respected Critics, with a remarkably high probability of 43.29%. 







  §.§.§ Temporal Analysis of Archetype's Higher-order Interactions


Above, we proposed a thorough characterization of a series of archetypes. At this point, an interesting aspect to study is the dynamics of archetypes within the social platform, as well as their evolution in time. To do so, we now focus on temporal trends of archetypes by examining some features on a monthly basis. We focus on two key metrics, that is, the average hyperdegree hdeg and the average degree deg of all nodes represented by each archetype.

We consider the archetypes defined in the previous section, and we analyze the nodes each archetype represents in each month. More formally, for each archetype and for each node u, we compute its hyperdegree hdeg(u) and its degree deg(u), and we do this for each hypergraph H_t, t = 1,…,12. In Fig. <ref> we show the obtained results. The figure consists of 12 subplots, one for each hypergraph, corresponding to a different month. For each subplot, the x-axis represents the average hyperdegree hdeg of each archetype, while the y-axis represents the average degree deg of each archetype. From the analysis of this figure, different insights can be derived. First off, we can observe how the Respected Critics archetype has a higher hyperdegree and degree than all other archetypes. This is somewhat expected as their characterization encompasses a considerable number of nodes, as we have seen in the sec:archetype-characterization section. It is straightforward to observe how there are pairs of archetypes that are often collocated together w.r.t. these dimensions. This is the case of Infamous Celebrity and Quiet Critic, which appear very close in almost all monthly snapshots, although they are very different in size. The relatively small number of archetypes such as Controversial Star and Positive Provoker is also reflected in their average hyperdegree and degree values: indeed, they remain stable during the months and have little to no fluctuations. This is somewhat expected, considering that these archetypes are characterized by emotionally charged and often provocative interactions, leading to fluctuating levels of activity and influence. This variability suggests that their presence is more dynamic, potentially driven by specific events or contentious discussions that peak at different times. Also in this case, these results could be somewhat enhanced by a more qualitative-oriented approach: indeed, aspects such as the content shared between archetypes would be of great importance in a more in-depth analysis of their interaction. Nevertheless, we believe the provided observation already indicates different implications. For instance, archetypes with stable engagement patterns are likely to be reliable contributors, therefore they could be taken into account in systematic operations of the platforms, such as moderation activities. Also, archetypes that show fluctuating feature values might be highly responsive to specific events or topics, which could fuel critical discussions. 





 §.§ Characterization of Discussions


We recall that one of the objectives of our approach is to analyze higher-order entities. To do so, it offers the so-called hyperedge characterization function ω, which we presented in the sub:hyperedge-characterization-function section. We are now able to leverage it to describe the dataset with regard to some of the properties exhibited by the discussions.

First off, we focus on examining our dataset on a monthly basis, thus we consider each hypergraph H_t, t = 1, …, 12. We do not focus on the overall hypergraph H due to the fact that we are interested in observing how the structure and the interactions within Scored.co evolve over time. Also, this approach allows for a more granular understanding of user behavior and interactions. For each hypergraph H_t, we compute the s-betweenness hyperedge centrality <cit.>. This measure is a higher-order extension of classical graph betweenness centrality, and is computed on the line graph projection of the hypergraph to estimate the importance of hyperedges. We recall that the line graph of a hypergraph is a graph that represents the relationships between the hyperedges of the original hypergraph. In particular, there is a node in the line graph for each hyperedge in the hypergraph, and there is an edge between two nodes if the corresponding hyperedges share at least one node. After the computation of the centrality, we select the top 50 hyperedges in each hypergraph H_t, and for each of we compute some descriptive features based on the linguistic productions of the contained nodes. In particular, we focus on the following features, namely, (i) average word count, (ii) average unique word count, and (iii) average purity w.r.t. user archetypes. These values can be easily calculated via the ω function, specialized for each of them. For instance, the specialization for the average purity is given by ω_purity in the sub:hyperedge-characterization-function section.


In Fig. <ref>, we report the temporal trends of the average values of average word count, average unique word count, and average purity for the top 50 most prominent discussions. Note how the discussion here is another term for hyperedge: in fact, in our setting, a hyperedge represents a discussion between users in the social hypernetwork. From the analysis of this figure, we can derive different insights. First off, the average word count graph, in the left part of the figure, shows a relatively stable trend with slight fluctuations, indicating a consistent level of engagement in discussions over the year. There are some notable peaks, such as a slight increase around mid-year followed by a step-down and a subsequent rise towards the end of the year. Similarly, the average vocabulary size, which indicates the average unique word count, mirrors the previous measure but remains consistently lower, which is somewhat expected because of non-content words (e.g., articles, pronouns, etc.) appearing more frequently in texts. A slight increase in the middle of the year is also depicted here, suggesting periods of higher and lower lexical diversity in user interactions. The middle part of Fig. <ref> reports the trend for average subjectivity. We recall that this metric evaluates how subjective (as opposed to objective) the discussions are. It remains relatively stable throughout the year, which indicates that discussions typically contain a balanced mix of personal opinions and factual statements. Indeed, this calls for a more qualitative-oriented analysis of these discussions, an approach often used in the content-based investigation of social platforms <cit.>. Finally, the right part of the image depicts the graph of the average purity w.r.t. the user archetypes. This refers to the consistency of user behavior and content relative to the defined archetypes. We can observe how this trend shows more variability compared to the previous features. There are sharp declines in some months, suggesting periods of more inconsistent behavior among users, while the different peaks could indicate times when user behavior is more aligned with archetypes. While all these insights are interesting, and show the applicability of our proposed approach, more in-depth analyses could be performed, as also noted above. Nevertheless, the various specializations of ω enable a series of different lenses through social interactions can be studied, especially in understudied platforms.







§ CONCLUSION



The identification of node roles within complex networks is significant when analyzing their dynamics and function. To advance in this context, in this paper we proposed a multi-dimensional, general framework to characterize nodes and hyperedges in a social hypernetwork. The aim of our framework is two-fold: i) to characterize nodes and hyperedges taking into account their exhibited features and higher-order dynamics, and ii) to define “archetypes”, serving as a template to represent higher-order roles of nodes. 
Our framework consists of different components, namely, hyperedge ensembles, hyperedge- and node-based characterization functions. The combination of such components allows one to analyze the social hypernetwork from a more general point of view; each component can also be exploited in a standalone fashion, thus enabling a detailed analysis of the single aspect it captures. 
To assess the effectiveness of the framework, we carried out an exhaustive experimental campaign on Scored.co, an understudied social platform, focusing on different aspects such as the characterization of nodes as well as their behaviors and surroundings. 

This paper is a starting point for future research with various possible extensions.

First off, the framework can be applied as it is on other (possibly understudied) social platforms such as Bluesky <cit.>.
Second, it would be interesting to compare archetypes based on the set of features selected for defining them. Indeed, an approach to identify the archetype best representing a particular set of nodes could be defined as an optimization problem, and meta-heuristics solutions could be employed to solve it. 
Another future direction would be studying role dynamics in crisis situations. Indeed, we could apply the framework to study social hypernetworks in crisis situations, e.g., natural disasters and pandemics, to understand how roles and behaviors shift under stress and how information and support are mobilized <cit.>. 
Exploiting the temporal component, we would integrate this framework about higher-order roles in the recent multi-dimensional analysis of group evolution in temporal data <cit.>.
Finally, an interesting challenge would be the analysis of cross-platform role identification, and in particular the investigation of the consistency of identified roles across multiple social platforms. This could help in understanding whether certain archetypes are platform-specific or they can represent more universal roles in online social interactions.





§ ACKNOWLEDGMENTS


This work is supported by (i) the European Union – Horizon 2020 Program under the scheme “INFRAIA-01-2018-2019 – Integrating Activities for Advanced Communities”, Grant Agreement n.871042, ”SoBigData++: European Integrated Infrastructure for Social Mining and Big Data Analytics" (<http://www.sobigdata.eu>); (ii) SoBigData.it
which receives funding from the European Union – NextGenerationEU – National Recovery and Resilience Plan (Piano Nazionale di Ripresa e Resilienza, PNRR) – Project: ”SoBigData.it – Strengthening the Italian RI for Social Mining and Big Data Analytics" – Prot. IR0000013 – Avviso n. 3264 del 28/12/2021; (iii) EU NextGenerationEU programme under the funding schemes PNRR-PE-AI FAIR (Future Artificial Intelligence Research). 









 Paper Title: Tracing Intricate Cues in Dialogue: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion Recognition.










LI et al.: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion Recognition

0000–0000  2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.



Tracing Intricate Cues in Dialogue: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion Recognition
    Jiang Li, Xiaoping Wang, Senior Member, IEEE, and Zhigang Zeng, Fellow, IEEE
        
        
Manuscript received 30 July 2024; revised 30 July 2024; accepted 30 July 2024. This work was supported in part by the Interdisciplinary Research Program of HUST under Grant 2024JCYJ006, the National Natural Science Foundation of China under Grant 62236005 and 61936004, and the Fundamental Research Funds for the Central Universities of HUST under Grant 2024JYCXJJ067 and 2024JYCXJJ068. (Corresponding author: Xiaoping Wang.)
The authors are with the School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China, also with the Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan 430074, China, also with the Hubei Key Laboratory of Brain-Inspired Intelligent Systems, Huazhong University of Science and Technology, Wuhan 430074, China, and also with the Key Laboratory of Image Processing and Intelligent Control (Huazhong University of Science and Technology), Ministry of Education, Wuhan 430074, China (e-mail: lijfrank@hust.edu.cn; wangxiaoping@hust.edu.cn; zgzeng@hust.edu.cn).
Digital Object Identifier

    August 5, 2024
===================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Multimodal emotion recognition in conversation (MERC) has garnered substantial research attention recently. Existing MERC methods face several challenges: (1) they fail to fully harness direct inter-modal cues, possibly leading to less-than-thorough cross-modal modeling; (2) they concurrently extract information from the same and different modalities at each network layer, potentially triggering conflicts from the fusion of multi-source data; (3) they lack the agility required to detect dynamic sentimental changes, perhaps resulting in inaccurate classification of utterances with abrupt sentiment shifts. To address these issues, a novel approach named GraphSmile is proposed for tracking intricate emotional cues in multimodal dialogues. GraphSmile comprises two key components, i.e., GSF and SDP modules. GSF ingeniously leverages graph structures to alternately assimilate inter-modal and intra-modal emotional dependencies layer by layer, adequately capturing cross-modal cues while effectively circumventing fusion conflicts. SDP is an auxiliary task to explicitly delineate the sentiment dynamics between utterances, promoting the model's ability to distinguish sentimental discrepancies. Furthermore, GraphSmile is effortlessly applied to multimodal sentiment analysis in conversation (MSAC), forging a unified multimodal affective model capable of executing MERC and MSAC tasks. Empirical results on multiple benchmarks demonstrate that GraphSmile can handle complex emotional and sentimental patterns, significantly outperforming baseline models.



Emotion recognition in conversation, sentiment dynamics, multimodal fusion, graph neural networks.




§ INTRODUCTION

Emotion recognition in conversation (ERC) has become a hot topic in academia due to its broad application prospects in various fields such as government affairs, education, and healthcare. A multitude of ERC models <cit.> focus on the analysis of textual modality, often encountering the issue of insufficient information. In certain special cases, such as sarcasm, text-based methods may not fully comprehend the semantics of an utterance from solely a single modality. The extraordinary ability of the human brain to integrate multisensory information provides inspiration for addressing the limitations of information deficiency in unimodal models. Multimodal emotion recognition in conversation (MERC) emerges as an effective solution to this challenge, achieving more accurate emotional state recognition by integrating information from various modalities such as text, audio, and video. MERC is a sophisticated task that goes beyond the scope of traditional unimodal <cit.> and non-conversational <cit.> affective endeavors (in this paper, the term “affective" encompasses emotional and sentimental aspects). It necessitates not only the sufficient extraction of cross-modal association cues that capture synergies between different sensory inputs (e.g., facial expressions, vocal tones, and textual content), but also an in-depth investigation of intra-modal contextual cues to perceive effects of contexts on the emotional state of an utterance. Compared to unimodal and non-conversational counterparts, MERC faces more formidable challenges, with the key one being the thorough capture of intra-modal contextual cues and inter-modal associative cues.

Researchers have developed numerous MERC approaches, including new architectures and learning mechanisms, to capture and synthesize the wealth of information present in multimodal dialogues. Some efforts <cit.> straightforwardly concatenated or summed features across disparate modalities, which have manifested certain performance merits over unimodal approaches when dealing with specific tasks. Regrettably, certain modalities may convey information that is ambiguous or even contradictory to labels. These concatenation fusion-based works fail to adequately account for the intrinsic relationships and interactions between different modalities, thus making it difficult to effectively address the issues of information selection and integration. There are an array of methods <cit.> predicated on cross-attention fusion. 
These methods achieve the amalgamation of diverse modality information by calculating the weights between multimodal features and applying these weights to the current modality features. Nonetheless, they retain certain constraints at the feature level and have not fully actualized direct interaction and fusion between different modalities.

Recent studies have unveiled the exceptional potential of graph neural networks (GNNs) in handling the fusion and comprehension of multimodal data. GNNs, with the unique topological structure and dynamic information propagation mechanism, inherently excel at capturing cross-modal associative cues and intra-modal contextual cues. 
Despite significant advancements <cit.> in the graph-based MERC domain, several unresolved issues persist. Firstly, existing models, when constructing cross-modal graphs, only connect heterogeneous nodes (i.e., nodes that belong to different modalities) from the same utterance and fail to incorporate associations with heterogeneous nodes from different utterances. This construction scheme overlooks the direct emotional cues provided by other utterances across modalities, leading to incomplete capture of multimodal information. Secondly, these models typically process information from the same and different modalities at each layer of the network, i.e., concurrently extract intra-modal and inter-modal emotional cues. Owing to the heterogeneity among modalities <cit.>, this concurrent aggregation of different types of information may precipitate conflicts during fusion, resulting in insufficient cross-modal modeling. Furthermore, current models are excessively preoccupied with contextual modeling, neglecting the dynamic mutation of sentiments throughout the conversation. The absence of agility in recognizing sentimental changes may cause these models' inability to classify accurately in scenarios with abrupt sentiment shifts. Consequently, there is an urgent need to develop models that can integrate multimodal heterogeneous information more comprehensively and rationally, and that are capable of keenly capturing the sentiment dynamics.

To cope with the above dilemmas, we introduce a joint graph structure and sentiment dynamics for multimodal emotion recognition in conversation (GraphSmile). The core components of GraphSmile encompass the graph structure fusion (GSF) and sentiment dynamics perception (SDP) modules. During the construction of cross-modal edges, GraphSmile not only correlates the current node with heterogeneous nodes belonging to the same utterance but also directly connects it with nodes emanating from disparate utterances. This strategy for edge construction effectively ameliorates the insufficient in extracting cross-modal information. At the graph convolution stage, the GSF module adeptly utilizes the graph structure to alternately collect inter-modal associative cues and intra-modal contextual cues on layer-by-layer basis, thereby precluding potential conflicts stemming from the simultaneous aggregation of diverse data sources at each layer. To mitigate redundant connections in the multimodal dialogue graph, we refrain from constructing edges between nodes within the same modality; nonetheless, the GSF module, through its unique graph convolution mechanism, is capable of directly and comprehensively capturing intra-modal contextual cues. In the phase of sentiment dynamics detection, we integrate the SDP module as an auxiliary task to explicitly model the shift in sentimental state between utterances. This module not only deepens the model's comprehension of sentiment dynamics but also constrains the network parameters of the main task with a strategy akin to contrastive learning, i.e., drawing utterances with the same sentiment closer and those with different sentiments further apart, thereby enhancing the model's ability to discern sentimental disparities.

In this work, we can effortlessly extend GraphSmile into a multitask model. Namely, GraphSmile is applied to the multimodal sentiment analysis in conversation (MSAC) task and achieves multitask joint optimization with the MERC task. Specifically, we initially incorporate a sentiment classifier, followed by the joint formulation of emotion classification loss, sentiment classification loss, and shift classification loss to construct a multitask learning objective, thereby synergistically optimizing network parameters. In this manner, GraphSmile is ultimately crafted into a unified multimodal affective model capable of executing MSAC and MERC tasks concurrently. In the domain of affective computing, emotions (e.g., happiness, surprise, neutrality, sadness, and anger) are typically regarded as more nuanced and fleeting psychological states, characterized by their situational and temporary nature, while sentiments (e.g., negative, neutral, positive) are understood as coarser and more enduring psychological inclinations, marked by their profundity and stability. While emotion and sentiment exhibit varying levels of granularity, duration, and characteristic, there exists an interrelation between them; happiness, for example, is typified as a positive sentiment, whereas sadness is reflective of a negative sentimental disposition.
Thus, our co-optimization strategy not only facilitates knowledge transfer and complementary advantages between different granularity levels of affective tasks but also significantly enhances GraphSmile's comprehensive perceptive capability for emotional and sentimental cues.

The principal contributions of this work are as follows:

  
  * A unified multitask multimodal affective model called GraphSmile is proposed for both MERC and MSAC tasks. GraphSmile integrates multiple losses to achieve multitask synergistic optimization while upgrading its ability to synthesize the perception of various cues.
  
  * The GSF module we designed not only ensures the full exploitation of cross-modal affective cues but also obviates potential conflicts arising from the fusion of multi-source data. This module also confers two additional benefits, i.e., it diminishes redundant connections within the graph and alleviates the over-smoothing issue prevalent in graph-based models.
  
  * We develop a plug-and-play SDP module that can explicitly capture sentimental changes in dialogues, fostering the model's understanding of the dynamics of sentimental evolution while constraining the network parameters of the primary task.
  
  * Extensive comparative experiments and ablation studies have been conducted on multimodal benchmark datasets. The empirical results have validated the superiority of GraphSmile in both MERC and MSAC tasks.


The remainder of this paper is organized as follows. Section <ref> reviews the related efforts. Section <ref> provides a detailed exposition of the proposed GraphSmile. Sections <ref> and <ref> constitute the experimental segment of our work. Section <ref> summarizes this work.



§ RELATED WORK



 §.§ Emotion Recognition in Conversation

With the proliferation of applications for dialogue systems, ERC has increasingly become a focal point in the realm of affective computing. Depending on the source of input information, research in this domain can be categorized into two branches: unimodal and multimodal approaches. Since the inception of ERC tasks, text-based methods <cit.> have consistently held a dominant position among unimodal approaches. In recent years, this branch has seen profound advancements with numerous emerging technologies modeling contextual information in textual conversations from diverse perspectives. 
GNNs, due to their proficiency in capturing intra-modal dependency information, have notably been instrumental in conversational relationship modeling, thereby giving rise to numerous graph-based ERC models <cit.>.
Regrettably, unimodal ERC methods are limited by their inherent information deficiencies and susceptibility to external influences, which can impede their performance. 

The richness of human emotions manifests distinctively across individual modalities such as text, audio, and vision. These modalities possess unique characteristic and exhibit both shared traits and individual nuances. Consequently, multimodal ERC techniques have garnered significant attention due to their capacity to integrate diverse modal information. 
Some methods <cit.> employ a kind of straightforward feature fusion strategy, which involves concatenating or summing feature vectors from different modalities, overlooking the intrinsic interactions between modalities. In tandem, a variety of models <cit.> based on cross-modality attention are proposed in succession.
These methods are unable to achieve direct absorption of information from one modality to another due to the constraints of their network structures, which are considered implicit cross-modal modeling.

In recent years, GNNs have garnered widespread attention due to their powerful capabilities in handling complex relationships, providing a solvable inspiration for explicitly conducting cross-modal interactions. 
Despite certain advancements in graph-based fusion methods <cit.>, they still face a series of unique and common issues: due to the limitations of graph construction strategies, they may not fully capture and utilize cross-modal emotional cues; the heterogeneity between different modal data may lead to conflicts in the fusion process of multi-source information; some methods may overly rely on extracting intra-modal contextual cues, neglecting the dynamic changes in sentimental states during the conversation. Moreover, some of the aforementioned issues also exist in methods based on concatenation fusion and cross-attention fusion. In response to these issues, we propose a new fusion network architecture called GraphSmile for MERC.



 §.§ Multimodal Sentiment Analysis

Multimodal sentiment analysis (MSA), due to its wide range of application prospects, has become a focal research topic in the field of affective computing. MSA is a multimodal task aimed at understanding and detecting human emotional tendencies by processing multimodal heterogeneous data, such as text, speech, and vision. The main challenge of the MSA task is how to effectively harness information from different modalities to complement each other, and a plethora of representative works <cit.> have been proposed in succession. Existing mainstream works focus on enhancing the performance of MSA models by designing complex structures, interaction mechanisms, or fusion paradigms, which can be broadly categorized into tensor-based methods <cit.>, attention-based methods <cit.>, and graph-based methods <cit.>. These research trajectories of MSA tasks are distinctly different from those of the ERC tasks, as they do not take into account the conversational context, rendering them incompatible with ERC tasks. Although Hu et al. <cit.> have made effort to unify ERC and MSA, their approach to handling labels in the datasets is overly complex and cannot fully guarantee the correctness of universal labels. Divergent from the extant MSA methods, our GraphSmile is conceived as a multitask multimodal conversational affective model. In GraphSmile, we only need to merge emotional labels to effortlessly unify the ERC and MSA tasks, thereby facilitating an efficacious excavation of both emotional and sentimental cues in a concurrent manner.



§ PROPOSED GRAPHSMILE

Given a conversation c = [u_1, u_2, …, u_M] consisting of M utterances, along with the emotional label e_i (or sentimental label s_i) corresponding to each utterance u_i, the objective of the MERC (or MSAC) task is to train a model based on the dialogue and labels to detect the affective state of each utterance. In MERC and MSAC tasks, each utterance comprises three modalities: textual t, visual v, and acoustic a modalities, which is denoted as u_i = [u_i^t, u_i^v, u_i^a].

Effectively extracting complex affective cues from multimodal dialogues is crucial for the MERC task. Here, the affective cues mainly consist of inter-modal associative and intra-modal contextual cues. Inter-Modal Associative Cues: Assuming the existence of two modalities (e.g., textual and visual), an utterance u_i is represented differently across modalities as u_i^t and u_i^v. The information carried by u_i^t is considered as an inter-modal associative cue for u_i^v. The inter-modal associative cues for u_i^v also include information carried by other utterances from modality t, such as u_i-2^t, u_i-1^t, and u_i+1^t. Intra-Modal Contextual Cues: Taking a single modality (e.g., acoustic) as an example, an utterance u_j is denoted as u_j^a in a specific modality. The intra-modal contextual cues for u_j^a encompass information carried by other utterances from the same modality a, such as u_j-1^a, u_j+1^a, and u_j+2^a.

To address the predicaments encountered by existing models, we propose a model termed GraphSmile that conducts a comprehensive modeling of multimodal dialogues. The overall architecture of GraphSmile is depicted in Fig. <ref>, which mainly includes the extraction of utterance-level features, the construction of multimodal dialogue graphs, the alternate graph propagation, the multimodal integration, and the multitask classification. Here, multitask classification involves emotion, shift, and sentiment classifications. In the following, we describe each component of GraphSmile in detail.




 §.§ Utterance-Level Feature

The datasets utilized in this work include IEMOCAP <cit.> (both 6-way and 4-way settings), MELD <cit.>, and CMU-MOSEI <cit.>. Influenced by prior works <cit.>, the utterance-level feature extraction for the datasets used in GraphSmile is as follows. Text features for all datasets are processed using RoBERTa <cit.>. 
Visual features for the IEMOCAP (6-way) and MELD datasets are extracted using DenseNet <cit.>, which is pre-trained on the FER+ corpus <cit.>. The IEMOCAP (4-way) dataset utilizes OpenFace 2.0 <cit.> for visual feature extraction, while the visual features for the CMU-MOSEI dataset are derived from the work of Zadehet et al. <cit.>.
Audio features for the IEMOCAP (6-way) and MELD datasets are extracted using the OpenSmile toolkit with IS10 configuration <cit.>. The IEMOCAP (4-way) dataset uses OpenSmile <cit.> for audio feature extraction, and the CMU-MOSEI dataset uses librosa <cit.> for this purpose.



 §.§ Graph Structure Fusion Module



  §.§.§ Multimodal Dialogue Graphs

To establish relationships between utterances while considering inter-modal interactions, we construct three multimodal dialogue graphs, each composed of two modalities. Taking the textual and visual modalities as an example, we build a textual-visual dialogue graph 𝒢^tv = (𝒩^t, 𝒩^v, ℰ^t ← v, ℰ^v ← t). Here, 𝒩^t and 𝒩^v represent the node sets from the textual and visual modalities, respectively. ℰ^t ← v denotes the set of edges from source visual nodes to target textual nodes, while ℰ^v ← t denotes the set of edges from source textual nodes to target visual nodes.

Nodes: Each utterance u_i is represented as nodes n_i^t for the textual modality and n_i^v for the visual modality, with corresponding features denoted as x_i^t and x_i^v. If a dialogue consists of m utterances, then 𝒢^tv contains 2m nodes. Consequently, 𝒩^t can be represented as {n_1^t, n_2^t, …, n_M^t}, 𝒩^v can be represented as {n_1^v, n_2^v, …, n_M^v}, with their corresponding features denoted as [x_1^t, x_2^t, …, x_M^t] and [x_1^v, x_2^v, …, x_M^v].

Edges: We establish relationships only between nodes of different modalities. In the graph 𝒢^tv, a textual node n_i^t is connected to all visual nodes; similarly, a visual node n_i^v is connected to all textual nodes. To mitigate redundant connections <cit.>, we set up two sliding windows. Specifically, n_i^t is associated with the past P visual nodes (i.e., n_i-P^v, n_i-P+1^v, …, n_i-1^v), and also with the future F visual nodes (i.e., n_i+1^v, n_i+2^v, …, n_i+F^v). The same connection way is applied to the visual node n_i^v. Additionally, different modal nodes (e.g., n_i^t and n_i^v) from the same utterance are also connected.

Edge Weights: In our research, the edges connecting two nodes are directed, with edges of differing orientations signifying distinct implications. For instance, in the graph 𝒢^tv, for nodes n_i^t and n_i-1^v, the edge e(n_i^t ← n_i-1^v) denotes the informational contribution from n_i-1^v to n_i^t, whereas the edge e(n_i-1^v ← n_i^t) signifies the informational contribution from n_i^t to n_i-1^v. To distinguish between different edges, diverging from existing methods <cit.>, we assign a set of trainable weights directly to them. These weights are initialized to a value of 1 and are optimized during the model training process.

Adjacency Matrix: The adjacency matrix serves to store the relationships between all nodes in a graph, representing a pivotal element in expressing the graph structure and also constituting one of the input data for spectral-based GNNs. Based on the nodes, edges, and edge weights previously established, an adjacency matrix A^tv can be generated to serve as the input for our GraphSmile.

Following the construction manner of the graph 𝒢^tv, a textual-acoustic dialogue graph 𝒢^ta and a visual-acoustic dialogue graph 𝒢^va can also be obtained. The majority of previous approaches <cit.>, when creating cross-modal edges, only connect nodes from the same utterance, thereby neglecting the affective cues directly provided by other utterances. In contrast to these methods, in our construction of cross-modal edges, the current node is not only connected to nodes from the same utterance but also directly linked to nodes from different utterances. It is noteworthy that our multimodal dialogue graphs don't establish edges between utterances in the same modality. Nonetheless, GraphSmile can still effortlessly capture intra-modal contextual affective cues through the unique graph convolution mechanism.



  §.§.§ Alternate Graph Propagation

Previous efforts <cit.> have demonstrated that GNNs can effectively mine intra-modal contextual information and cross-modal associative information in multimodal dialogues. However, these models share a common issue: the simultaneous integration of inter-modal and intra-modal affective cues at each network layer may cause conflicts among multi-source information, making thorough cross-modal modeling difficult. This part will provide a detailed description of how the GSF module cleverly addresses this issue. 

As depicted in Fig. <ref>, drawing inspiration from existing GNN works, we define a new graph convolutional operation to extract affective cues from multimodal dialogues. For each dialogue graph, multiple simplified graph convolutional layers <cit.> are utilized to propagate inter-modal and intra-modal cues layer by layer in an alternating fashion, which can be mathematically expressed as:

    X^(l) = A X^(l-1)Θ^(l),    l = 1, 2, …, L.

Here, A denotes the adjacency matrix of the dialogue graph, X^(l) signifies the output at the l-th layer, and X^(0) represents the initial input to the GSF module; Θ^(l) indicates the learnable parameter at the l-th layer. Assuming a given textual-visual dialogue graph 𝒢^tv, then the adjacency matrix A can be represented by A^tv, X^(l) can be denoted by X^tv, (l). The initial feature X^tv, (0) is composed of X^t, (0) and X^v, (0), where X^t, (0) and X^v, (0) are denoted as [x_1^t, x_2^t, …, x_M^t] and [x_1^v, x_2^v, …, x_M^v], respectively.

Subsequently, the feature representations obtained from all layers are summed together, which is known as residual connections, and can be mathematically formulated as:

    X = X^(0) + 𝙵𝙲^(1)(X^(1)) + ⋯ + 𝙵𝙲^(L)(X^(L)).

In the graph 𝒢^tv, X can be represented as X^tv, which is composed of X^t ← v and X^v ← t. Here, X^t ← v denotes the textual feature representation that has incorporated visual information, and X^v ← t signifies the visual feature representation that has carried textual information. 𝙵𝙲 (·) represents a fully connected layer, which is composed of a linear transformation, an activation function (e.g., LeakyReLU), and a dropout operation, i.e.,

    𝙵𝙲(X) = 𝙳𝚛𝚘𝚙𝚘𝚞𝚝(𝙻𝚎𝚊𝚔𝚢𝚁𝚎𝙻𝚄(X Θ_f + b_f)).

Here, Θ_f and b_f are the learnable parameters. In our GSF module, the fully connected layer serves two purposes: (1) the trainable network parameters allow the GSF module to dynamically select information from each network layer; (2) the combination of the linear transformation and activation function can further enhance the expressive ability of the designed GSF module.



  §.§.§ Example Explanation

We take the textual-visual dialogue graph 𝒢^tv as an example to elucidate why the GSF module can alternately propagate different types of affective cues layer by layer. In Fig. <ref>, we illustrate the process of multi-layer graph convolution with a concrete example involving four utterances.


According to Eq. <ref>, the first-layer graph convolution process can be mathematically formulated as: 

    X^(1) = A X^(0)Θ^(1).

Here, A can be represented by A^tv, which is the adjacency matrix that expresses the relationships between all nodes in the graph 𝒢^tv; X^(1) denotes the output at the first layer, X^(0) is the initial input to the GSF module, and Θ^(1) indicates the learnable parameter at the first layer. From the construction way of the multimodal dialogue graph (see Fig. <ref>), A^tv does not include the relationships between nodes from the same modality, instead only storing the associations between nodes from different modalities. Consequently, during the first-layer graph convolution, each node only collects affective cues from different modal nodes. For instance, in Fig. <ref>, the node n_1^t affiliated with the textual modality is exclusively connected to visual nodes (e.g., n_1^v, n_2^v), thereby assimilating cues from a distinct modality; reciprocally, the visual node n_1^v seizes affective cues solely from textual counterparts.

For the second-layer graph convolution, integrating Eq. <ref> and Eq. <ref>, the message propagation process can be mathematically formalized as: 

    X^(2)   = A X^(1)Θ^(2)
       = A (A X^(0)Θ^(1)) Θ^(2)
       = A^2 X^(0)Θ^(1)Θ^(2),

where X^(2) and Θ^(2) represent the output and learnable parameter for the second layer, respectively. Despite the second layer's features being predicated on those of the first layer, according to Eq. <ref>, the output of the second layer can also be directly computed from the initial input X^(0). In our GSF module, since A^2 stores the second-hop neighborhood relationships of nodes, the significance of the second-layer graph convolution can be interpreted as nodes assimilating feature information from their second-hop neighbors. Referring to Fig. <ref>, during the second-layer graph convolution, each node collects affective cues solely from nodes within the same modality. For instance, the textual node n_1^t is connected only to nodes belonging to the textual modality, thereby collecting cues exclusively from intra-modal nodes (e.g., n_2^t, n_3^t); similarly, the visual node n_1^v gathers affective cues only from visual nodes. 

The message propagation process for the third-layer graph convolution can be expressed as follows:

    X^(3)   = A X^(2)Θ^(3)
       = A (A X^(1)Θ^(2)) Θ^(3)
       = A (A (A X^(0)Θ^(1)) Θ^(2)) Θ^(3)
       = A^3 X^(0)Θ^(1)Θ^(2)Θ^(3),

where X^(3) signifies the feature matrix resulting from the third layer and Θ^(3) represents the corresponding learnable parameter. Following Eq. <ref>, the third-layer graph convolution, akin to the second layer, can have its output directly computed from the initial features X^(0). A^3 contains the third-hop neighborhood relationships, thus the implication of the third-layer is that nodes integrate feature information from their third-hop neighbors. As depicted in Fig. <ref>, during the third-layer graph convolution, each node absorbs affective cues solely from nodes of another modality. For instance, similar to the first-layer convolution, the textual node n_1^t is connected only to nodes of a different modality, i.e., visual nodes, and the visual node n_1^v is connected only to textual nodes.

Through the aforementioned examples, it can be illustrated that our GSF module can alternately propagate inter-modal and intra-modal affective cues layer by layer, thereby preventing conflicts among multi-source information. Summarizing the patterns from the examples, to capture affective cues from more distant neighbors, we can extend to the l-th layer graph convolution:

    X^(l)   = A X^(l-1)Θ^(l)
       = A (A X^(l-2)Θ^(l-1)) Θ^(l)
       = A ⋯ (A X^(0)Θ^(1)) ⋯Θ^(l)
       = (A ⋯ A) X^(0) (Θ^(1)⋯Θ^(l)) 
       = A^l X^(0) (Θ^(1)⋯Θ^(l)).

Here, X^(l) represents the feature matrix at the l-th layer, A^l denotes the l-th power of the adjacency matrix A, which signifies the reach to l-hop neighbors, and Θ^(l) are the learnable parameters specific to the l-th layer. This formulation allows nodes to aggregate information from increasingly broader neighborhoods as the layer index l increases, thus enriching the representation of each node with more comprehensive affective cues. Via Eq. <ref>, the GSF module is capable of integrating the outputs from all layers. The summation of all layer outputs, i.e., the residual connection, provides two ingenious benefits: (1) it mitigates the over-smoothing problem <cit.> in GNNs; (2) it amalgamates affective cues from multi-hop neighbors.



  §.§.§ Multimodal Integration

The graphs 𝒢^tv, 𝒢^ta, and 𝒢^va are respectively input into three separate GSF modules for graph convolution operations. After stacking multiple layers of convolutions and combining with Eq. <ref>, three outputs X^tv, X^ta, and X^va can be obtained. Here, X^tv is constituted by X^t ← v and X^v ← t, X^ta is composed of X^t ← a and X^a ← t, X^va consists of X^v ← a and X^a ← v. Then, these outputs are summed to obtain a new feature representation:

    H    = σ ((X^t ← v)Θ_h) + σ ((X^v ← t)Θ_h) 
       + σ ((X^t ← a)Θ_h)+ σ ((X^a ← t)Θ_h) 
       + σ ((X^v ← a)Θ_h) + σ ((X^a ← v)Θ_h),

where σ represents the activation function, e.g., LeakyReLU, and Θ_h denotes the shared learnable parameter. The new representation H integrates information from textual-to-visual, textual-to-acoustic, and visual-to-acoustic modalities, providing comprehensive features that encapsulate the multimodal affective cues in dialogues.



 §.§ Emotion Classification

To achieve emotion classification, H is fed into a classifier that incorporates a linear function followed by a softmax function. The emotion classification is formulated as:

    Ŷ   = 𝚜𝚘𝚏𝚝𝚖𝚊𝚡(HΘ_y + b_y), 
    ê_i    = 𝚊𝚛𝚐𝚖𝚊𝚡(ŷ_i[τ]),

where both Θ_y and b_y are learnable parameters.  
Let Ŷ = [ŷ_1, ŷ_2, …, ŷ_M], then ŷ_i represents the emotional probability vector of utterance u_i, and ê_i denotes the corresponding predicted emotional label. During the training phase, we employ the categorical cross-entropy loss for network optimization, which can be mathematically expressed as:

    ℒ_e = - 1/M∑_i=1^M∑_j=1^C_e y_i, jlogŷ_i, j,

where M represents the number of utterances in the dialogue, C_e denotes the number of emotional categories; y_i,j signifies the ground-truth probability distribution of the i-th utterance u_i in the j-th category, and ŷ_i,j is the predicted probability of utterance u_i.



 §.§ Sentiment Dynamics Perception Module

In dialogue systems, sentiment shift refers to the phenomenon where the sentimental state of two utterances transitions from one category to another. Capturing the dynamic change in sentiments is pivotal for comprehending the sentimental trajectory of a conversation. However, prevailing MERC models predominantly concentrate on contextual modeling of the utterance, exhibiting a dearth of acuity in detecting inter-utterance sentimental changes. This limitation can lead to inaccuracies in the classification of affective states during scenarios characterized by sentiment shifts. To mitigate this shortfall, we introduce an auxiliary task known as the sentiment dynamics perception (SDP) module. Our SDP module serves the following purposes: (1) it models the dynamics of sentimental changes, endowing the model with the capability to not only discern static affective states but also track dynamic sentimental shifts; (2) by pulling utterances with the same sentimental state closer and pushing those with different sentimental states further apart, it aids in enhancing the model's ability to discern between various affective categories; (3) integrated as a complementary task, its loss function can be conjointly optimized with that of the main task, guiding the model to accord significance to the phenomenon of sentiment shifts.



  §.§.§ Shift Label

Based on the shifts of sentimental states between utterances in a dialogue, we generate a set of sentiment shift labels for that conversation. In a dialogue, if two utterances possess distinct sentiments, i.e., their states transition from one sentiment to another, then the shift label is set to “1", signifying that a sentiment shift has occurred between them. Conversely, if two utterances share the same sentiment, i.e., there is no alteration in their sentimental state, then the shift label is set to “0", indicating that no sentiment shift has transpired between them. The annotating process of the shift label o_(i,j) can be formulated as follows: 

    o_(i,j) = 
    
    1,    if s_i ≠ s_j, 
    
    0,    if s_i = s_j.

Here, s_i denotes the sentimental state of the i-th utterance u_i. It is worth noting that the IEMOCAP datasets do not provide pre-defined sentiment labels. To make them compatible with GraphSmile, we merge their emotion labels, with the specific merging scheme as depicted in Table <ref>. Additionally, in the CMU-MOSEI dataset, an utterance may correspond to multiple emotion labels, and this annotation way is not suitable for our model. Consequently, we adopt seven sentiments, ranging from highly negative (-3) to highly positive (+3), to substitute the emotion labels for classification.




  §.§.§ Shift Feature

In Subsection <ref>, a fused multimodal feature representation H ∈ℝ^M × D is obtained. Here, M denotes the number of utterances in the dialogue, D signifies the feature dimension, and H can be indicated as [h_1, h_2, …, h_M]. Based on the feature representation H, we construct a set of sentiment shift features. Specifically, given a dialogue containing M utterances, we concatenate the features of these utterances pairwise. Through the aforementioned operation, M^2 samples of sentiment shifts can be generated to construct a shift feature representation T with a size of M^2 × 2D.

Fig. <ref> provides an example of generating shift features. Assuming there are four utterances in the dialogue, their features are represented as [h_1, h_2, h_3, h_4], where h_i ∈ℝ^D. We initially select the feature vector h_1 and concatenate it with all utterances (i.e., h_1, h_2, h_3, and h_4), thereby obtaining the feature representations for the first row, i.e., t_(1,1), t_(1,2), t_(1,3), t_(1,4). Subsequently, we take h_2, h_3, and h_4 in sequence and repeat the above procedure. Ultimately, this results in 16 shift samples, with the dimension of the features t_(i,j) for each sample being 2D, where i=j=1,2,3,4.


When the number of utterances in a dialogue (i.e., dialogue length) is excessive, recognizing sentiment shifts between an utterance and long distant utterances becomes meaningless. In other words, intuitively, exploring sentiment shifts between nearby utterances is more meaningful. Moreover, for long dialogues, detecting sentiment shifts between all utterances may consume a substantial amount of computational resources. Therefore, we first evenly divide a dialogue into M/B segments (B is the number of utterances per segment), and then generate sentiment shift labels and corresponding features in each segment.



  §.§.§ Shift Classification

Similar to the emotion classification, we subject the shift feature representation T to a classifier for shift classification, mathematically formulated as:

    Ẑ   = 𝚜𝚘𝚏𝚝𝚖𝚊𝚡(T Θ_z + b_z), 
    ô_(i,j)   = 𝚊𝚛𝚐𝚖𝚊𝚡(ẑ_(i,j)[η]),

where Θ_z and b_z are learnable parameters for the sentiment shift classifier; Ẑ can be denoted as [ẑ_(1,1), ẑ_(1,2), …, ẑ_(M,M)], ẑ_(i,j) represents the shift probability vector constructed based on utterances u_i and u_j, and ô_i,j denotes the corresponding predicted shift label. During the training phase, the cross-entropy loss is employed for network learning:

    ℒ_o = - 1/M^2∑_i=1^M∑_j=1^M∑_k=1^C_o z_(i, j), klogẑ_(i, j), k,

where 
C_o denotes the number of shift categories (C_o=2 in this work); z_(i,j),k signifies the ground-truth probability distribution of the shift sample constructed based on u_i and u_j in the k-th category, and ẑ_(i, j), k is the predicted probability of the shift sample. 



 §.§ Sentiment Classification

Following the emotion classification, we can easily apply GraphSmile to the MSAC task by adding a sentiment classifier. 
The sentiment classification is represented by the following mathematical formula:

    Ŵ   = 𝚜𝚘𝚏𝚝𝚖𝚊𝚡(H Θ_w + b_w), 
    ŝ_i    = 𝚊𝚛𝚐𝚖𝚊𝚡(ŵ_i[ϕ]).

Here, Θ_w and b_w are the learnable parameters specific to the sentiment classifier
; Ŵ can be indicated as [ŵ_1, ŵ_2, …, ŵ_M], and ŵ_i represents the sentimental probability vector for utterance u_i, with ŝ_i denoting the corresponding predicted sentimental label. The loss function for sentiment classification can be expressed as:

    ℒ_s = - 1/M∑_i=1^M∑_j=1^C_s w_i, jlogŵ_i, j,

where 
C_s is the number of sentiment categories, w_i,j is the ground-truth probability distribution of utterance u_i in the j-th category, and ŵ_i, j denotes the predicted probability of the utterance. 



 §.§ Total Training Objective

The MERC task is jointly learned alongside the MSAC task to simultaneously extract emotional and sentimental cues in the dialogue. Uniting all tasks together, we can derive the total training objective, represented by the following formula:

    ℒ_total = ℒ_e + λ_s ℒ_s +λ_o ℒ_o + β‖Θ_total‖,

where λ_s, λ_s, and λ_o are trade-off parameters balancing these three losses; β is the L2-regularizer weight, and Θ_total is the set of all learnable parameters.



§ EXPERIMENTAL SETUP



 §.§ Baseline

To substantiate the effectiveness of GraphSmile, we select a multitude of baseline models for comparison. These baselines are categorized into graph-based models and non-graph-based models. The former encompasses COGMEN <cit.>, Joyful <cit.>, DER-GCN <cit.>, DCGCN <cit.>, MMGCN <cit.>, MM-DFN <cit.>, M3Net <cit.>; whereas the latter includes MetaDrop <cit.>, UniMSE <cit.>, EmoLR <cit.>, DialogueCRN <cit.>, SACL-LSTM <cit.>.



 §.§ Dataset

We conduct experiments across three datasets, including IEMOCAP, MELD, and CMU-MOSEI. The statistical information of all datasets are shown in Table <ref>. 
IEMOCAP <cit.> is a renowned multimodal corpus of dyadic conversations that captures emotional interactions from ten participants in both scripted and improvisational settings. 
We employ two IEMOCAP ways, one including all six emotions, i.e., IEMOCAP-6, and the other including four emotions, i.e., IEMOCAP-4, with Excited and Frustrated removed. 
MELD <cit.> is an esteemed collection of multimodal multiparty conversations, encompassing 1,433 dialogues from 304 speakers in the television series “Friends", totaling 13,708 utterances. 
CMU-MOSEI <cit.> is an extensive multimodal affective collection that aggregates video commentary segments from YouTube, featuring a wide range of 1,000 speakers. 
To be applicable to GraphSmile, we only use the sentiment annotation in this work. Referring to previous work <cit.>, we divide sentiment intensity into seven distinct classes, encompassing Highly Negative, Negative, Weakly Negative, Neutral, Weakly Positive, Positive, and Highly Positive. The corresponding intensity ranges for these classes are [-3, -2), [-2, -1), [-1, 0), 0, (0, 1], (1, 2], and (2, 3], respectively.




 §.§ Training Detail

We utilize accuracy (ACC) and weighted F1 (WF1) as the primary evaluation metrics, with F1 being used as the metric for each individual emotion category. To test the significance of performance improvements, a paired t-test is performed with the default significance level set at 0.05. All experiments are conducted on an NVIDIA GeForce RTX 3090, using CUDA version 11.7 and the deep learning framework PyTorch 2.0.0. AdamW is employed as the optimizer, with the L2 regularization factor of 1e-3. Other hyperparameter settings are shown in Table <ref>. The code is released at https://github.com/lijfrank-open/GraphSmile.




§ EXPERIMENTAL RESULT



 §.§ Main Result for Emotion Recognition


The results for the MERC task on all datasets are reported in Table <ref>. Overall, our proposed GraphSmile outperforms all baseline models on these datasets. Among the reproduced baseline models, M3Net achieves relatively balanced results across all datasets, benefiting from GNNs' thorough exploration of multivariate relationships and multi-frequency signals. The poor performance of DialogueCRN and SACL-LSTM on the CMU-MOSEI dataset indicates that models based on recurrent neural networks struggle to capture the emotional cues present in that dataset. GraphSmile surpasses all graph-based methods on the IEMOCAP (6-way) dataset. For instance, GraphSmile's accuracy is 5.67%, 3.33%, and 1.85% higher than that of MMGCN, MM-DFN, and M3Net on this dataset, respectively. On the IEMOCAP (4-way) dataset, our model improves the weighted F1 score by 2.02% and 0.82% compared to the strong baselines COGMEN and Joyful, respectively. These advanced results are attributed to the significant role played by the GSF module in cross-modal modeling. On the other two more challenging datasets, GraphSmile continues to maintain its advantage. For example, compared to M3Net, GraphSmile's accuracy is 1.11% and 3.15% higher on the MELD and CMU-MOSEI datasets, respectively. In addition to the contribution of the GSF module, the contribution of SDP in sentimental dynamics modeling also contributes to the performance advantage of GraphSmile.

The table also presents the F1 scores for each emotion. In the IEMOCAP (6-way) dataset, compared to other emotions, all models achieve the highest and second-highest scores on Sad and Excited, respectively, and the lowest on Happy. This indicates that Sad and Excited are relatively easier to distinguish than Happy. Relative to other models, our GraphSmile demonstrates a more balanced performance across all emotions, especially achieving the best score on the challenging emotion Happy. It can be seen that due to the reduction in the number of categories in the IEMOCAP dataset, these models have achieved satisfactory results on all emotions. Among these baselines, there is always one or more emotions with significantly lower scores than the others, indicating performance imbalance. In contrast, GraphSmile consistently achieves relatively high results with minimal variance across all emotions. Similar to the results on the IEMOCAP (6-way) dataset, our GraphSmile scores significantly higher on Happy than other models. In the MELD dataset, compared to other emotions, all models perform poorly on Fear and Disgust. The direct cause of this phenomenon is that these two emotions are extreme minority classes in the MELD dataset. Since samples labeled as Neutral constitute the majority of all samples, all models achieve the best F1 scores on Neutral. Compared to other models, SACL-LSTM and GraphSmile achieves the best and second-best scores on the minority class Disgust, respectively. In the CMU-MOSEI dataset, as each utterance may correspond to multiple emotions, we employ seven sentiment labels instead of emotion ones for our experiments. Table <ref> shows that the classification task for the CMU-MOSEI dataset is quite challenging, with all models struggling to distinguish each category, particularly failing to correctly identify Highly Negative. Compared to the baselines, GraphSmile achieves the best results on Negative, Weakly Positive, and Highly Positive.



 §.§ Main Result for Sentiment Analysis


The proposed GraphSmile not only demonstrates exceptional performance in MERC but also successfully extends to the MSAC task. In Table <ref>, we collect the experimental results of GraphSmile performing the MSAC task across multiple datasets. These results are obtained based on the same hyperparameter settings as in Table <ref>, showing that even without task-specific hyperparameter tuning, GraphSmile can still achieve commendable performance, especially surpassing all baselines on the IEMOCAP (4-way) dataset. The reduction in the number of categories leads to a significant performance improvement for all models on datasets other than IEMOCAP (4-way). After hyperparameter optimization specifically for the MSAC task, GraphSmile's performance is further enhanced, outperforming all baselines on most datasets. Although SACL-LSTM achieves the best scores on the IEMOCAP (6-way) dataset, its performance on other datasets is not prominent. Similar to the results of the MERC task, DialogueCRN and SACL-LSTM show relatively weaker capabilities in capturing sentimental cues on the CMU-MOSEI dataset, with their performance significantly lower than other models. Additionally, the table lists the F1 score for each sentiment category in detail. In the IEMOCAP (6-way) dataset, the model scores lower in identifying Neutral, indicating that distinguishing Neutral is more challenging compared to other sentiments. A similar phenomenon is observed in the CMU-MOSEI dataset. In the IEMOCAP (4-way) dataset, the recognition of Negative is the most effective, while in the MELD dataset, the recognition score of Neutral is the highest. These results indicate that there are variations in the recognition difficulty for different datasets and sentiments.



 §.§ Ablation Study


To comprehensively evaluate the functionality of various components in GraphSmile, we report a series of ablation results in Table <ref>. These studies systematically examine several key factors, including multitask learning, residual connection, utterance segmentation, and modality configuration.

Multitask Learning: In ablation studies of multitask learning, we design three experiments, i.e., removing the sentiment classification task (No ℒ_s), removing the shift classification task (No ℒ_o), and removing both tasks simultaneously (No ℒ_s & ℒ_o). The results indicate that the removal of any single task adversely affects the overall performance of the model. In the MERC task, The performance decline attributable to the excision of ℒ_s substantiates the capacity of the MSAC task to enhance the MERC task's comprehension of emotions in a dialogue through knowledge complementation. Reciprocally, in the MSAC task, the performance decline induced by the ablation of ℒ_e evidences that the MERC task can facilitate the MSAC task in understanding sentiments. In both tasks, the decrement in efficacy consequent to the elimination of ℒ_o corroborates our SDP module can enable the model to capture the sentiment dynamics in a conversation, thus ensuring precise classification in scenarios with abrupt sentimental transitions. In addition, on the IEMOCAP dataset, the removal of ℒ_o leads to a more significant performance drop compared to the removal of ℒ_s; conversely, on the MELD and CMU-MOSEI datasets, the observed phenomenon is the opposite of that on the IEMOCAP. This could be attributed to the abundance of long dialogues contained in the IEMOCAP dataset, whose sentimental changes is more complex relative to the other two datasets, thus necessitating a more urgent need for modeling sentiment dynamics.

Residual Connection: During evaluating the influence of residual connections on the efficacy, it is observed that the removal of residual connections (No Res) markedly impairs the model's performance across all datasets. These outcomes can be ascribed to several pivotal factors: (1) Compromised modality integration capability. The elimination of residual connections precipitates a direct diminution in the model's capacity to integrate intra-modal contextual and inter-modal associative cues that are propagated in an alternating fashion. (2) Loss of neighboring information. In the absence of residual connections, the model may exhibit a propensity to prioritize long-range information propagation, forgetting the affective cues proffered by neighboring nodes. (3) Over-smoothing predicament. The model devoid of residual connections are susceptible to the over-smoothing dilemma as the network's depth intensifies, leading to a convergence in the feature representations of disparate nodes. (4) Constrained affective expression facility. Simplified graph convolutional layers exhibit constraints in expressing emotions and sentiments due to the lack of non-linear transformation, a deficiency that is amplified by the absence of residual connections. To delve deeper into the role of the fully connected layer in the residual connection, an ablation experiment is conducted by selectively removing the fully connected components of residual connections (No FC-Res). The experimental outcomes indicate a decrement in performance relative to the complete GraphSmile across the four datasets, thereby underscoring the significant role of the fully connected layer in augmenting the model's capacity for affective expression. 

Utterance Segmentation: In the shift classification task, we adopt a segmentation strategy for utterances. The relinquishment of the segmentation strategy (No Seg) results in a degradation in the model's performance. This substantiates the efficacy of the strategy in capturing sentiment dynamics, i.e., facilitating the model's perception of sentimental transitions and continuities among proximate utterances, thereby enabling a more nuanced comprehension of useful sentimental shifts and continuities by the model.

Modality Configuration: 
The empirical outcomes in modality ablation underscores a degradation in the model's holistic performance following the removal of any modality, thereby revealing the indispensable complementarity of multimodal data in the MERC task. Specifically, across all examined datasets, the elimination of the textual modality (No t) engenders a markedly pronounced decrement in performance relative to other modalities. This revelation accentuates the dominant role of the textual modality in the MERC task, proffering more critical emotional information. Concurrently, it implies that the visual and acoustic modalities may carry relatively less emotional cues. Such disparities may stem from the noise embedded in visual and acoustic modalities, such as irrelevant background information, ambient sounds, and multiple facial expressions. Further analysis of the experimental results from the IEMOCAP dataset evinces that the exclusion of the visual modality (No v) exerts a relatively muted influence on the model's efficacy compared to the excision of the acoustic modality (No a). This observation posits that in the IEMOCAP dataset, the visual modality contributes less to MERC than the other two modalities. Based on the results from the MELD and CMU-MOSEI datasets, it can be concluded that the contributions of the visual and acoustic modalities to the model's performance on these two datasets are comparable, yet both are inferior to the textual modality.



 §.§ Time and Memory Overhead


In Table <ref>, we report the running time and memory usage of our model and its baselines for the MERC task across all datasets. Relative to other models, MMGCNN and GraphSmile exhibit an absolute advantage in time expenditure on the IEMOCAP dataset, yet MMGCN's time usage on the other two datasets is significantly higher than that of GraphSmile. M3Net and GraphSmile achieve lower time consumption on the MELD dataset, while M3Net's running time on other datasets is inferior to GraphSmile. The running times of DialogueCRN, MM-DFN, and SACL-LSTM across all datasets are unsatisfactory. MMGCN's memory consumption is considerably lower than other models on all datasets, but its overly simplistic model leads to poor performance on most datasets. M3Net and SACL-LSTM have much higher memory consumption on multiple datasets compared to other models, and the memory expenditure of DialogueCRN and MM-DFN is also higher than that of the proposed model. In summary, GraphSmile not only achieves the best accuracy and F1 scores but also balances the costs of time and memory.



 §.§ Impact of Network Depth

Early GNNs commonly demonstrate the notorious over-smoothing phenomenon <cit.>. This issue refers to the gradual homogenization of node features as network depth increases, leading to a sharp decline in model performance after reaching a certain threshold. In Fig. <ref>, the dark blue and dark green lines intuitively display the performance variation of GraphSmile for the MERC task with network depth. On the IEMOCAP (6-way) dataset, GraphSmile's performance improves with the increase of network layers, slightly declines after reaching the peak, and then maintains a slight fluctuation. The performance changes on the IEMOCAP (4-way) and MELD datasets exhibit a similar pattern. These phenomena indicate that GraphSmile successfully circumvents the over-smoothing issue. In Fig. <ref>, we also illustrate the performance variation of the model without residual connections (i.e., not employing Eq. <ref>) adopting light blue and light green lines. It can be observed that as the network depth increases, the performance of this model (i.e., the incomplete GraphSmile) rapidly declines after reaching a certain threshold, showing signs of over-smoothing. The performance comparison between the complete and incomplete versions substantiates the effectiveness of residual connections in alleviating the over-smoothing problem. Despite GraphSmile is less effective against over-smoothing on the CMU-MOSEI dataset due to the higher classification difficulty, the complete model still indicates a significant performance advantage over the incomplete version.




 §.§ Impact of Window Size

During the construction of multimodal dialogue graphs, we introduce two sliding windows to eliminate irrelevant relationships. For ease of experimentation, we employ two equal-length windows, i.e., setting the sizes of the past and future windows to be equal. Fig. <ref> details the performance of GraphSmile for the MERC task under different window sizes. Taking the IEMOCAP (6-way) dataset as an example, when the window size is 0 (i.e., without considering contextual information), the performance of GraphSmile drops to the lowest, which is also reflected in the IEMOCAP (4-way) and MELD datasets. These results indicate that context modeling plays a crucial role in the ERC task. As expected, the performance trends across all datasets exhibit a pattern of initial increase followed by a decrease as the window size increases, eventually fluctuating slightly within a certain range. This suggests that there is an optimal window size that allows the model to capture sufficient contextual information while avoiding the introduction of excessive noise.




 §.§ Embedding Visualization


To visually display the classification efficacy of GraphSmile for the MERC task, we employ T-SNE <cit.> to visualize the utterance embeddings. As depicted in Fig. <ref>, GraphSmile effectively segregates samples of distinct emotions in the IEMOCAP dataset, thereby substantiating its potency in emotion classification. Although MMGCN can to some extent differentiate emotional samples, its performance does not match that of our proposed model. Similar to GraphSmile, M3Net achieves commendable classification outcomes on this dataset. Analogous phenomena are observed in the MELD dataset. Furthermore, in contrast to the performance on the IEMOCAP and MELD datasets, GraphSmile exhibits suboptimal results on the CMU-MOSEI dataset, aligning with the outcomes of MMGCN and M3Net.



 §.§ Limitation


Like much of the previous work, GraphSmile has the limitation when dealing with similar emotions. As shown in Fig. <ref>, on the IEMOCAP (6-way) dataset, Happy samples have a high probability of being identified as the positive emotion Excited. Conversely, Excited samples also have a certain probability of being classified as Happy. Another limitation of GraphSmile is its difficulty in correctly detecting minority class samples on the extremely imbalanced dataset. For instance, on the MELD dataset, samples of Fear and Disgust, which are extreme minority classes, are challenging to classify correctly and tend to be identified as the majority class Neutral. Additionally, visual information (e.g., facial expressions) and acoustic information (e.g., intonation) should theoretically contain significant emotional cues. However, as Table <ref> reflects, due to the presence of a large amount of noise, they do not leverage their unique advantages in dialogue affective tasks. This may lead GraphSmile to favor the textual modality in cross-modal modeling, thereby reducing the effective mining of visual and acoustic information.



§ CONCLUSION

In this research, we propose an innovative multitask multimodal affective model named GraphSmile, specifically tailored for MERC and MSAC tasks. The proposed model achieves synergistic optimization of these tasks by integrating emotion recognition, sentiment analysis, and sentiment shift detection, thereby delving into the intricate emotional and sentimental cues embedded in multimodal dialogues. GraphSmile employs a unique graph construction manner that not only facilitates comprehensive capture of cross-modal interaction cues but also effectively mitigates redundant connections in the multimodal dialogue graph. To circumvent potential conflicts arising from the fusion of multi-source data, we design a graph structure based multimodal fusion module, termed GSF. This module adeptly harnesses the graph structure, optimizing the integration process by alternately absorbing inter-modal and intra-modal affective dependencies in a layered fashion. In the GSF module, we introduce a residual connection mechanism integrated with fully connected layers, which not only alleviates the over-smoothing issue common in GNNs but also significantly enhances the model's capacity for affective expression. Furthermore, to bolster the model's capability to capture the sentiment dynamics, we develop a plug-and-play auxiliary module known as SDP. This module explicitly models the sentimental transitions between utterances, augmenting the model's discernment of sentimental variances. We have conducted extensive validation experiments on several public benchmark datasets, and the results robustly substantiate the efficacy of GraphSmile and its components.

In future work, we intend to address other salient challenges in the realm of affective computing in conversation. For instance, tackling the differentiation of similar emotions and issues of class imbalance through contrastive learning strategies. We will also explore more sophisticated multimodal fusion strategies to capitalize on the unique strengths of each modality and enhance the model's robustness against noise and irrelevant information.

IEEEtran





 Paper Title: OmniParser for Pure Vision Based GUI Agent.



Extracting doubly-excited state lifetimes in helium directly in the time domain with attosecond noncollinear four-wave-mixing spectroscopy
    Stephen R. Leone
    August 5, 2024
==========================================================================================================================================




§ ABSTRACT 
   The recent success of large vision language models shows great potential in driving the agent system operating on user interfaces. However, we argue that the power multimodal models like GPT-4V as a general agent on multiple operating systems across different applications is largely underestimated due to the lack of a robust screen parsing technique capable of: 1) reliably identifying interactable icons within the user interface, and 2) understanding the semantics of various elements in a screenshot and accurately associate the intended action with the corresponding region on the screen. To fill these gaps, we introduce OmniParser, a comprehensive method for parsing user interface screenshots into structured elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface. 
   We first curated an interactable icon detection dataset using popular webpages and an icon description dataset. These datasets were utilized to fine-tune specialized models: a detection model to parse interactable regions on the screen and a caption model to extract the functional semantics of the detected elements.
   
   OmniParser significantly improves GPT-4V's performance on ScreenSpot benchmark. And on Mind2Web and AITW benchmark, OmniParser with screenshot only input outperforms the GPT-4V baselines requiring additional information outside of screenshot. 




§ INTRODUCTION

Large language models have shown great success in their understanding and reasoning capabilities. More recent works have explored the use of large vision-language models (VLMs) as agents to perform complex tasks on the user interface (UI) with the aim of completing tedious tasks to replace human efforts <cit.>. Despite the promising results, there remains a significant gap between current state-of-the-arts and creating widely usable agents that can work across multiple platforms, e.g. Windows/MacOS, IOS/Android and multiple applications (Web broswer Office365, PhotoShop, Adobe), with most previous work focusing on limiting applications or platforms.

While large multimodal models like GPT-4V and other models trained on UI data <cit.> have demonstrated abilities to understand basic elements of the UI screenshot, action grounding remains one of the key challenges in converting the actions predicted by LLMs to the actual actions on screen in terms of keyboard/mouse movement or API call <cit.>. It has been noted that GPT-4V is unable to produce the exact x-y coordinate of the button location, Set-of-Mark prompting <cit.> proposes to overlay a group of bounding boxes each with unique numeric IDs on to the original image, as a visual prompt sent to the GPT-4V model. By applying set-of-marks prompting, GPT-4V is able to ground the action into a specific bounding box which has ground truth location instead of a specific xy coordinate value, which greatly improves the robustness of the action grounding <cit.>. However, the current solutions using SoM relies on parsed HTML information to extract bounding boxes for actionable elements such as buttons, which limits its usage to web browsing tasks. We aim to build a general approach that works on a variety of platforms and applications. 

In this work, we argue that previous pure vision-based screen parsing techniques are not satisfactory, which lead to significant underestimation of GPT-4V model's understanding capabilities. And a reliable vision-based screen parsing method that works well on general user interface is a key to improve the robustness of the agentic workflow on various operating systems and applications. We present OmniParser, a general screen parsing tool to extract information from UI screenshot into structured bounding box and labels which enhances GPT-4V's performance in action prediction in a variety of user tasks. 


We list our  contributions as follows: 

    
  * We curate a interactable region detection dataset using bounding boxes extracted from DOM tree of popular webpages. 
    
  * We propose OmniParser, a pure vision-based user interface screen parsing method that combines multiple finetuned models for better screen understanding and easier grounded action generation.
    
  * We evaluate our approach on ScreenSpot, Mind2Web and AITW benchmark, and demonstrated a significant improvement from the original GPT-4V baseline without requiring additional input other than screenshot. 
    




§ RELATED WORKS



 §.§ UI Screen Understanding


There has been a line of modeling works focusing on detailed understanding of UI screens, such as Screen2Words <cit.>, UI-BERT <cit.>, WidgetCaptioning <cit.>, Action-BERT <cit.>. These works demonstrated effective usage of multimodal models for extracting semantics of user screen. But these models rely on additional information such as view hierarchy, or are trained for visual question answering tasks or screen summary tasks. 


There are also a couple publicly available dataset that on UI screen understanding. Most notably the Rico dataset <cit.>, which contains more than 66k unique UI screens and its view hierarchies. Later <cit.>  auguments Rico by providing 500k human annotations on the original 66k RICO screens identifying various icons based on their shapes and semantics, and associations between selected general UI elements (like icons, form fields, radio buttons, text inputs) and their text labels. Same on mobile platform, PixelHelp <cit.> provides a dataset that contains UI elements of screen spanning across 88 common tasks. In the same paper they also released RicoSCA which is a cleaned version of Rico. For the web and general OS domain, there are several works such Mind2Web <cit.>, MiniWob++<cit.>, Visual-WebArena <cit.>, and OS-World <cit.> that provide simulated environment, but does not provide dataset explicitly for general screen understanding tasks such as interactable icon detection on real world websites. 

To address the absence of a large-scale, general web UI understanding dataset, and to keep pace with the rapid evolution of UI design, we curated an icon detection dataset using the DOM information from popular URLs avaialbe on the Web. This dataset features the up-to-date design of icons and buttons, with their bounding boxes retrieved from the DOM tree, providing ground truth locations. 



 §.§ Autonomous GUI Agent


Recently there has been a lot of works on designing autonomous GUI agent to perform tasks in place of human users. One line of work is to train an end-to-end model to directly predict the next action, representative works include: Pixel2Act <cit.>, WebGUM<cit.> in web domain, Ferret <cit.>, CogAgent <cit.>, and Fuyu <cit.> in Mobile domain. Another line of works involve leveraging existing multimodal models such as GPT-4V to perform user tasks. Representative works include MindAct agent <cit.>, SeeAct agent <cit.> in web domain and agents in <cit.> for mobile domain. 
These work often leverages the DOM information in web browser, or the view hierarchies in mobile apps to get the ground truth position of interactable elements of the screen, and use Set-Of-Marks<cit.> to overlay the bounding boxes on top of the screenshot then feed into the vision-language models. However, ground truth information of interactable elements may not always be available when the goal is to build a general agent for cross-platforms and cross-applications tasks. Therefore, we focus on providing a systematic approach for getting structured elements from general user screens. 






§ METHODS

A complex task can usually be broken down into several steps of actions. Each step requires the model's (e.g. GPT-4V) ability to: 1) understand the UI screen in the current step, i.e. analyzing what is the screen content overall, what are the functions of detected icons that are labeled with numeric ID, and 2) predict what is the next action on the current screen that is likely to help complete the whole task. Instead of trying to accomplish the two goals in one call, we found it beneficial to extract some of the information such as semantics in the screen parsing stage, to alleviate the burden of GPT-4V so that it can leverages more information from the parsed screen and focus more on the action prediction. 


Hence we propose OmniParser, which integrates the outputs from a finetuned interactable icon detection model, a finetuned icon description model, and an OCR module. This combination produces a structured, DOM-like representation of the UI and a screenshot overlaid with bounding boxes for potential interactable elements. We discuss each component of the OmniParser in more details for the rest of the section. 






 §.§ Interactable Region Detection


Identifying interactable regions from the UI screen is a crucial step to reason about what actions should be performed given a user tasks. Instead of directly prompting GPT-4V to predict the xy coordinate value of the screen that it should operate on, we follow previous works to use the Set-of-Marks approach <cit.> to overlay bounding boxes of interactable icons on top of UI screenshot, and ask GPT-4V to generate the bounding box ID to perform action on. However, different from <cit.> which uses the ground truth button location retrieved from DOM tree in web browswer, and <cit.> which uses labeled bounding boxes in the AITW dataset <cit.>, we finetune a detection model to extract interactable icons/buttons. 

Specifically, we curate a dataset of interactable icon detection dataset, containing 67k unique screenshot images, each labeled with bounding boxes of interactable icons derived from DOM tree. We first took a 100k uniform sample of popular publicly availabe urls on the web <cit.>, and collect bounding boxes of interactable regions of the webpage from the DOM tree of each urls. Some examples of the webpage and the interactable regions are shown in <ref>.  

Apart from interactable region detection, we also have a OCR  module to extract bounding boxes of texts. Then we merge the bounding boxes from OCR detection module and icon detection module while removing the boxes that have high overlap (we use over 90% as a threshold). For every bounding box, we label it with a unique ID next to it using a simple algorithm to minimizing the overlap between numeric labels and other bounding boxes. 




 §.§ Incorporating Local Semantics of Functionality

We found in a lot of cases where only inputting the UI screenshot overlayed with bounding boxes and associated IDs can be misleading to GPT-4V. We argue the limitation stems from GPT-4V's constrained ability to simultaneously perform the composite tasks of identifying each icon's semantic information and predicting the next action on a specific icon box. This has also been observed by several other works <cit.>. To address this issue, we incorporate the local semantics of functionality into the prompt, i.e. for each icons detected by the interactable region detection model, we use a finetuned model to generate description of functionality to the icons, and for each text boxes, we use the detected texts and its label.

We perform more detailed analysis for this topic in section <ref>. To the best of our knowledge, there is no public model that is specifically trained for up-to-date UI icon description, and is suitable for our purpose to provide fast and accurate local semantics for the UI screenshot. Therefore we curate a dataset of  7k icon-description pairs using GPT-4o, and finetune a BLIP-v2 model <cit.>  on this dataset. Details of dataset and training can be found in Appendix <ref>. After finetuning, we found the model is much more reliable in its description to common app icons. Examples can be seen in figure <ref>. And in figure <ref>, we show it is helpful to incorporate the semantics of local bounding boxes in the form of text prompt along with the UI screenshot visual prompt. 




















§ EXPERIMENTS AND RESULTS

We conduct experiments on several benchmarks to demonstrate the effectiveness of OmniParser. We start by a motivating experiments showing that current GPT-4V model with set of mark prompting <cit.> is prone to incorrectly assigning label ID to the referred bounding boxes. Then we evaluate on Seeclick benchmark and Mind2Web to further showcase OmniParser with local semantics can improve the GPT-4V's performance on real user tasks on different platforms and applications.   



 §.§ Evaluation on SeeAssign Task


To test the ability of correctly predicting the label ID given the description of the bounding boxes for GPT-4v models, We handcrafted a dataset SeeAssign that contains 112 tasks consisting of samples from 3 different platforms: Mobile, Desktop and Web Browser. Each task includes a concise task description and a screenshot image. The task descriptions are manually created and we make sure each task refers to one of the detected bounding boxes, e.g. 'click on 'settings”, 'click on the minimize button'. During evaluation, GPT-4V is prompted to predict the bounding box ID associated to it. Detailed prompt are specified in Appendix.  
The task screenshot images are sampled from the ScreenSpot <cit.> benchmark, where they are labeled with set of marks using OmniParser. The tasks are further divided into 3 sub-categories by difficulty: easy (less than 10 bounding boxes), medium (10-40 bounding boxes) and hard (more than 40 bounding boxes). 

From table <ref>, we see that GPT-4V often mistakenly assign the numeric ID to the table especially when there are a lot of bounding boxes over the screen. And by adding local semantics including texts within the boxes and short descriptions of the detected icons, GPT-4V's ability of correctly assigning the icon improves from 0.705 to 0.938.  

From figure <ref>, we see that without the description of the referred icon in the task, GPT-4V often fails to link the icon required in the task and the ground truth icon ID in the SoM labeled screenshot, which leads to hallucination in the response. With fine-grain local semantics added in the text prompt, it makes it much easier for GPT-4V to find the correct icon ID for the referred icon. 
























 §.§ Evaluation on ScreenSpot

ScreenSpot dataset <cit.>  is a benchmark dataset that includes over 600 interface
screenshots from mobile (iOS, Android), desktop
(macOS, Windows), and web platforms. The task instructions are manually created so that each instruction corresponds to an actionable
elements on the UI screen. We first evaluate the performance of omniparser using the this benchmark.  In table <ref>, we can see across the 3 different platforms: Mobile, Desktop and Web, omniparser significantly improves the GPT-4V baseline. Noticeably, omniparser's performance even surpasses models that are specifically finetuned on GUI dataset including SeeClick, CogAgent and Fuyu by a large margin. 
We also note that incorporating the local semantics (omniparser w. LS in the table) further improves the overall performance. This corroborates with the finds in section <ref> that incorporating local semantics of the UI screenshot in text format, i.e. adding OCR text and descriptions of the icon bounding boxes further helps GPT-4V to accurately identify the correct element to operate on.  
Furthermore, our findings indicate that the interactable region detection (ID) model we finetuned improves overall accuracy by an additional 4.3% compared to using the raw Grounding DINO model. This underscores the importance of accurately detecting interactable elements for the success of UI tasks. Overall, the results demonstrate that the UI screen understanding capability of GPT-4V is significantly underestimated and can be greatly enhanced with more accurate interactable elements detection and the incorporation of functional local semantics.

  



 §.§ Evaluation on Mind2Web

In order to test how OmniParser is helpful to the web navigation secnario, We evaluate on <cit.> benchmark. There are 3 different categories of task in the test set: Cross-Domain, Cross-Website, and Cross-Tasks.

We used a cleaned version of Mind2Web tests set processed from the raw HTML dump which eliminates a small number of samples that has incorrect bounding boxes. In total we have 867, 167, 242 tasks in the test set from Cross-Domain, Cross-Website, and Cross-Tasks category respectively. During evaluation, we feed both the parsed screen results and the action history as text prompt, and SOM labeled screenshot to GPT-4V similar to the prompting strategy in <cit.>. Following the original paper, we perform offline evaluation focusing on the element accuracy, Operation F1 and step success rate averaged across the task.

In the first section of the table (row 1-3), We report numbers from a set of open source VL models as it appears in <cit.>. Here CogAgent and Qwen-VL are not finetuned on the Mind2Web training set. More detailed information about model settings can be found in the Appendix<ref>. 

In the second section of the table (row 4-9) we report numbers from Mind2web paper <cit.> and SeeAct <cit.>
paper. In this section, all of the approaches use the HTML elements selected by a finetuned element proposal model on Mind2Web training set which produces top 50 relevant elements on the HTML page based on the user task. Additionally, GPT-4V+SOM and GPT-4V+textual choices corresponds to the SeeAct with image annotation, and textual choices grounding methods respectively. In GPT-4V+SOM, the set of mark (SOM) boxes are selected from the element proposal model, and are labeled with the ground truth location extracted from HTML. In contrast, GPT-4V+textual uses DOM information of the selected relevant elements directly in the text prompt, rather than overlaying bounding boxes on top of screenshot. The better performance of textual choice corroborates with the experiment results in <ref>. 

In the last section (row 10-11), we report numbers from OmniParser. We observe GPT-4V augumented with local semantics of icon functionality and the finetuned interactable region detection model (w. LS + ID) performs better than the model with raw grounding DINO model (w. LS + GD) in all of the categories. 


Further, without using parsed HTML information, OmniParser is able to outperform GPT-4's performance that uses HTML in every sub-category by a large margin, suggesting the substantial benefit of the screen parsing results provided by OmniParser. 
Additionally, OmniParser outperforms the GPT-4V+SOM by a large margin. Compared to GPT-4V+textual choices, OmniParser significantly outperforms in Cross-Website and Cross-Domain category (+4.1% and +5.2%), while underperforming (-0.8%) slightly in the Cross-Task category, which indicates that OmniParser provides higher quality information compared ground truth element information from DOM and top-k relevant elemnt proposal used by the GPT-4V+textual choices set-up, and make the GPT-4V easier to make a accurate action prediction. 
Lastly, OmniParser with GPT-4V significantly outperform all the other trained models using only UI screenshot such as SeeClick and Qwen-VL.  








 §.§ Evaluation on AITW

In additional to evaluation on multi-step web browsing tasks, we assess OmniParser on the mobile navigating benchmark AITW <cit.>, which contains 30k instructions and 715k trajectories. We use the same train/test split as in <cit.> based on instructions, which retain only one trajectory for each instructions and no intersection between train and test. For a fair comparison, we only use their test split for evaluation and discard the train set as our method does not require finetuing.  

In table <ref>, we report the GPT-4V baseline in <cit.> paper, which corresponds to the best performing set up (GPT-4V ZS+history) that uses UI elements detected by IconNet <cit.> through set-of-marks prompting <cit.> for each screenshot at every step of the evaluation. The detected UI elements consist of either OCR-detected text or an icon class label, which is one of the 96 possible icon types identified by IconNet. Additionally, action history is also incorporated at each step's prompt as well.  We used the exact same prompt format in <cit.> except the results from the IconNet model is replaced with the output of the finetuned interactable region detection (ID) model.
Interestingly, we found that the ID model can generalize well to mobile screen. By replacing the IconNet with the interactable region detection (ID) model we finetuned on the collected webpages, and incorporating local semantics of icon functionality (LS), we find OmniParser delivers significantly improved performance across most sub-categories, and a 4.7% increase in the overall score compared to the best performing GPT-4V + history baseline. 










§ DISCUSSIONS


In this section, we discuss a couple of common failure cases of OmniParser with examples and potential approach to improve. 



Repeated Icons/Texts From analysis of the the GPT-4V's response log, we found that GPT-4V often fails to make the correct prediction when the results of the OmniParser contains multiple repeated icons/texts, which will lead to failure if the user task requires clicking on one of the buttons. This is illustrated by the figure <ref> (Left) in the Appendix. A potential solution to this is to add finer grain descriptions to the repeated elements in the UI screenshot, so that the GPT-4V is aware of the existence of repeated elements and take it into account when predicting next action. 



Corase Prediction of Bounding Boxes One common failure case of OmniParser is that it fails to detect the bounding boxes with correct granularity. In figure <ref> (Right), the task is to click on the text 'MORE'. The OCR module of OmniParser detects text bounding box 8 which encompass the desired text. But since it uses center of the box as predicted click point, it falls outside of the ground truth bounding box. This is essentially due to the fact that the OCR module we use does not have a notion of which text region are hyperlink and clickable. Hence we plan to train a model that combines OCR and interactable region detection into one module so that it can better detect the clickable text/hyperlinks. 


Icon Misinterpretation We found that in some cases the icon with similar shape can have different meanings depending on the UI screenshot. For example, in figure <ref>, the task is to find button related to 'More information', where the ground truth is to click the three dots icon in the upper right part of the screenshot.  OmniParser successfully detects all the relevant bounding boxes, but the icon description model interpret it as: "a loading or buffering indicator". We think this is due to the fact that the icon description model is only able to see each icon cropped from image, while not able to see the whole picture during both training and inference. So without knowing the full context of the image, a symbol of three dots can indeed mean loading buffer in other scenarios. A potential fix to this is to train an icon description model that is aware of the full context of the image. 






§ CONCLUSION

In this report, We propose OmniParser, a general vision only approach that parse UI screenshots into structured elements.  OmniParser encompasses two finetuned models: an icon detection model and a functional description models. To train them, we curated an interactable region detection dataset using popular webpages, and an icon functional description dataset. We demonstrate that with the parsed results, the performance of GPT-4V is greatly improved on ScreenSpot benchmarks. It achieves better performance compared to GPT-4V agent that uses HTML extracted information on Mind2Web, and outperforms GPT-4V augmented with specialized Android icon detection model on AITW benchmark. We hope OmniParser can serve as a general and easy-to-use tool that has the capability to parse general user screen across both PC and mobile platforms without any dependency on extra information such as HTML and view hierarchy in Android. 




§ ACKNOWLEDGEMENT

We would like to thank Corby Rosset and authors of ClueWeb22 for providing the seed urls for which we use to collect data to finetune the interactable region detection model. The data collection pipeline adapted AutoGen's multimodal websurfer code for extracting interatable elements in DOM, for which we thank Adam Fourney. We also thank Dillon DuPont for providing the processed version of mind2web benchmark.



alpha





§ APPENDIX



 §.§ Details of Icon-Description Dataset


In figure <ref>, we see that the original BLIP-2 model tend to focus on describing shapes and colors of app icons, while struggling to recognize the semantics of the icon. This motivates us to finetune this model on an icon description dataset. For the dataset, we use the result of parsed icon bounding boxes inferenced by the interactable icon detection model on the ScreenSpot dataset since it contains screenshots on both mobile and PC. For the description, we ask GPT-4o whether the object presented in the parsed bounding box is an app icon. If GPT-4o decides the image is an icon, it outputs one-sentence description of the icon about the potential functionality. And if not, GPT-4o will output 'this is not an icon', while still including this in the dataset. In the end, we collected 7185 icon-description pairs for finetuning. 

We finetune BLIP-2 model for 1 epoch on the generated dataset with constant learning rate of 1e^-5, no weight decay and Adam optimizer. We show a few of the qualitative examples of finetuned model vs the original model in figure <ref>. 





 §.§ Training details of Interactable Icon Region Detection Model

As introduced in <ref>, we train a YOLOv8 model on the interactable icon region detection dataset. We collect in total of 66990 samples where we split 95% (63641) for training, and 5% (3349) for validation. We train for 20 epochs with batch size of 256, learning rate of 1e^-3, and the Adam optimizer on 4 GPUs. We show the training curve in figure <ref>. 






 §.§ Details of SeeAssign Evaluation



  §.§.§ Prompt Used for GPT-4V

GPT-4V without local semantics: 

Here is a UI screenshot image with bounding boxes and corresponding labeled ID overlayed on top of it, your task is task. Which icon box label you should operate on? Give a brief analysis, then put your answer in the format of “`Box with label ID: [xx]“`

GPT-4V with local semantics: 

Here is a UI screenshot image with bounding boxes and corresponding labeled ID overlayed on top of it, and here is a list of icon/text box description: parsed_local_semantics. Your task is task. Which bounding box label you should operate on? Give a brief analysis, then put your answer in the format of “`Box with label ID: [xx]“`



 §.§ Details of Mind2Web Evaluation

Here we list more details of each baseline in table <ref>.


SeeClick, QWen-VL SeeClick is a finetuned version of Qwen-VL on the Mind2Web training set and we report both of their numbers in their paper. 

CogAgent CogAgent number is taken from the SEEAct paper <cit.>, where they report cogagent-chat-hf checkpoint that is not fine-tuned on Mind2Web for experiments. 

MindAct(Gen), MindAct, GPT-3.5-Turbo, GPT-4 The numbers for these baseline are taken from the Mind2Web <cit.> paper, where they use HTML information to augument the corresponding web agent.

GPT-4V+som This model corresponds to the image annotation grounding method in SeeAct paper, where the som boxes extracted from the selelcted HTML elements are provided to GPT-4V to make action prediction.

GPT-4V+textual choice This corresponds to the best performing scenario in SeeAct paper (except the Oracle), that uses the selected HTML elments information in a multi-choice question format as input to the GPT-4V agent. 



  §.§.§ Qualitative Examples

We list a few more examples to demonstrate local semantics of icon function description helps GPT-4V make better action prediction in figure <ref>.


























 Paper Title: Data Contamination Report from the 2024 CONDA Shared Task.


Unsteady slip pulses under spatially-varying prestress
    Eran Bouchbinder
    Received 2 August 2023 / Accepted 7 May 2024
======================================================



§ ABSTRACT 

    The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models. The goal of the shared task and associated database is to assist the community in understanding the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. The shared task provides a structured, centralized public database for the collection of contamination evidence, open to contributions from the community via GitHub pool requests. This first compilation paper is based on 566 reported entries over 91 contaminated sources from a total of 23 contributors. The details of the individual contamination events are available in the platform.1 The platform continues to be online, open to contributions from the community.




§ INTRODUCTION




Data contamination, where evaluation data is inadvertently included in pre-training corpora of large-scale models, and language models (LMs) in particular, has become a concern in recent times <cit.>. The growing scale of both models and data, coupled with massive web crawling, has led to the inclusion of segments from evaluation benchmarks in the pre-training data of LMs <cit.>. The scale of internet data makes it difficult to prevent this contamination from happening, or even detect when it has happened  <cit.>. 

Crucially, when evaluation data becomes part of pre-training data, it introduces biases and can artificially inflate the performance of LMs on specific tasks or benchmarks <cit.>. This poses a challenge for fair and unbiased evaluation of models, as their performance may not accurately reflect their generalization capabilities <cit.>. And similarly to pre-training contamination, the contamination can also occur during the fine-tuning stage even after a model has been deployed as an API <cit.>.

Although a growing number of papers and state-of-the-art models mention issues of data contamination <cit.>, there is little in the way of organized and compiled knowledge about real, documented cases of contamination in practice <cit.>. Addressing data contamination is a shared responsibility among researchers, developers, and the broader community.

This report compiles the evidence reported in the Data Contamination Database[<https://huggingface.co/spaces/CONDA-Workshop/Data-Contamination-Database>] as part of the Data Contamination Workshop.[<https://conda-workshop.github.io/>] As the Shared Task of the workshop, researchers were invited to discover cases of contamination in available corpora and models, and submit evidence of their discovery. The submissions to the database were collected and compiled on June 23rd, 2024, to be included in this report, but the database continues to run and grow. Overall we collected 566 submissions from 23 contributors, where each submission included a detailed contamination report, indicating the estimated percentage of contaminated data. 

We continue to operate the database, and expect to update it with newer datasets and models as they come out, as well as new report about existing contaminated (or uncontaminated) evaluations.

This report first presents the methodology for collecting evidence, as well as existing papers that report data contamination (Section <ref>). We also report the evidence collected in the Data Contamination Database (Section <ref>), followed by an overview of the trends and statistics in the database, that inform a high-level perspective on the state of data contamination in NLP today (Section <ref>). 



§ METHODOLOGY AND PREVIOUS WORK





Collecting all the contamination evidence —or lack of it— was done openly, through pull requests, and subject to discussions before the admission. Contributors were asked to fill in the information about several aspects, such as the
contaminated resource (a training corpus or model), the evaluation dataset which was found in the contaminated source, a breakdown of the percentage of contamination found in each split of the dataset (train, development, and test), an optional reference to a paper that describes the methodology behind the submission, as well as whether the contamination detection method was data-based or model-based. The contributions provided the HuggingFace Hub id of models, corpus, and datasets when possible. In addition, contributors must provide the evidence or a reference to the scientific paper that reported the evidence originally. Figure <ref> shows the taxonomy of the papers that reported contamination evidence in the shared task.[Note that there are many other works on data contamination detection. In this report we focus on works that were used to detect contamination for this report. We leave a more detailed coverage survey for future work.] We split the these methods into two: data-based and model-based approaches.





  
Data-based approaches. are methods that inspect the pre-training corpora to find contamination evidence. Data-based approaches typically involve string or sub-string matching techniques such as 13-gram overlap <cit.>, 50-character overlap <cit.> or even full-string overlap <cit.>. In Figure <ref> we differentiate between Proprietary and Open data. Papers that fall in the category of Proprietary data are usually LLMs technical reports that run post-hoc data contamination evaluations to identify and remove evaluation instances that appear in the pre-training corpora <cit.>. Papers that fall in the open data category usually involve corpus analysis tools <cit.> or LLMs with publicly available pre-training data <cit.>.




  
Model-based approaches. are those methods that try to estimate the contamination of a model by prompting or analyzing the output, without accessing the pre-training data. These methods are formulated as Membership Inference Attacks (MIA) and range from asking LLMs to generate verbatim of the actual evaluation data <cit.> to analyzing the actual output probabilities given by the model <cit.>. We differentiate between methods applicable to closed and open models. Methods applicable to closed models are usually applicable to open models, but not the other way around due to the limitations established by the API or interface providers.

The collected evidences come from different approaches and sources, making them hardly comparable. For transparency, we included in the database information about the source of the evidence and the link to the discussion. We encourage the users to assess how the evidence was collected for their datasets of interest.





§ COMPILATION OF EVIDENCE
 



The report includes 42 contaminated sources (training corpora or models), 91 datasets, and 566 contamination entries, including 432 contamination events (20 train-set, 95 dev-set, 317 test-set) and 144 non-contamination events, where a contamination event is taken as any report above 0% of contamination. The database contains, for each split (train, dev, and test) of each evaluation dataset, what percentage was found to be contaminated by a subset of the contamination sources (corpora or models). We analyze separately the contaminated corpora and models. 


























  
Contaminated corpora. Figure <ref> shows the number of reported test sets for each corpus often used to pre-train language models. The reported corpora are mainly based on CommonCrawl snapshots, GitHub, or a mix of sources. For CommonCrawl-based corpora, there are 35 events reported for C4 <cit.>, 32 for RedPajama v2 <cit.>, 29 for OSCAR <cit.> and 6 for CommonCrawl <cit.> itself. Regarding the GitHub data, there are 2 events reported for the TheStack <cit.> project. The corpora with various sources, the Pile <cit.> and ProofPile <cit.>, have 30 and 2 reported contamination events respectively. There is also 1 report for xP3 <cit.>, which is a collection of prompts for different NLP datasets.[The report indicates the use of validation data from a specific dataset as training.]

Table <ref> shows for each corpus often used to pre-train language models, the contamination events involving development or test splits. Please refer to the online database for full details of each report.



 












  
Contaminated models. Figure <ref> details the number of contamination events involving test sets that were reported, organised according to each pre-trained model. Most reported evidence is for closed models, for instance: 24 for GPT-3 <cit.>, 17 for GLaM <cit.>, 16 for GPT-4 <cit.>, 13 for GPT-3.5 <cit.>, 8 for PaLM <cit.>, 3 for PaLM-2 <cit.>, 2 for GPT-3.5 Turbo <cit.> and 1 for Calude 3 Opus. In the case of open models: there are 14 reported events for models fine-tuned with FLAN data <cit.>, 5 for Mistral <cit.>, 3 for Llama 2 <cit.>, 2 for Qwen <cit.>, Llema <cit.> and Aquila 2; and a single one for mT0 and Bloom-Z <cit.>.


Table <ref> shows for each pre-trained language model, the contamination events involving development or test splits. Please refer to the online database for full details of each report.























































































































































§ ANALYSIS OF THE REPORTED DATA
 

In this section, we analyze the reported entries to understand the report's data better. 









  
Reported tasks. Figure <ref> shows the percentage of data contamination per task. We use the  assigned to each dataset in the Hugging Face hub. Text-scoring, QA, and multiple-choice-qa are among the most contaminated task types. Figure  <ref> shows the number of downloads for every dataset in the report. We measure the total number of downloads from the Hugging Face hub.[<https://huggingface.co/docs/datasets>] Since one model may be reported as contaminated with a dataset while another model may not, we have entries of both being compromised and non-compromised for some datasets. Relating both tables, we can see that the tasks reported as the most contaminated include very popular datasets such as MMLU (multiple-choice-qa), GLUE (text-scoring), and ai2_arc (multiple-choice-qa), which are standard benchmarks for measuring the performance of LLMs. These benchmarks, as well as other very popular benchmarks reported in instances of data contamination, such as hellaswag or gsm8k are implemented in community leaderboards such as the Open LLM Leaderboard.[<https://hf.co/spaces/open-llm-leaderboard/>]











  
Year of publication of the reported data. Figure <ref> shows the percentage of total test sets included in contamination events per year. We present data for test sets in both contamination events (>0% contamination) and non-contamination events (0% contamination). Most of the reported datasets correspond to the 2018 to 2021 period. 


We further explore the relationship between the year of publication of the datasets and instances of contamination by examining the reported data contamination for the three models with the most instances of data contamination: GPT-4, GPT-3, and GPT-3.5. As expected based on the models' release dates, Figure <ref> shows that more recently released models are contaminated with more recently released datasets. For instance, GPT-3, launched in 2020, is predominantly contaminated with datasets from 2016, while GPT-4, released in 2023, is mainly contaminated with datasets from 2018 to 2022. 














§ CONCLUSIONS


Data contamination has become a significant concern in recent times. Consequently, a growing number of papers and state-of-the-art models mention issues of data contamination. In the CONDA 2024 Shared Task on Evidence of Data Contamination, we have collected and compiled a comprehensive database of available evidence on data contamination in currently available datasets and models. This report includes 566 contamination entries over 91 contaminated sources from a total of 23 contributors. With this shared task, we provide a structured, centralized platform for contamination evidence collection to help the community understand the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. Given the large exploration space, this report does not cover all cases, but a small sample that were reported during our shared task period, in the midst of 2024. We welcome further submissions to the database, and plan to keep this database up-to-date as it provides a valuable source of information for the research community.


















































































































































































































































































































































































































































































































































































































§ ACKNOWLEDGMENTS


We are grateful to Hugging Face and Clémentine Fourrier for their support in establishing the website for the Data Contamination Database hosted on Hugging Face Spaces.



We acknowledge the support of project Disargue (TED2021-130810B-C21, funded by MCIN/AEI /10.13039/501100011033 and by European Union NextGenerationEU/ PRTR) and the Basque Government (Research group funding IT-1805-22). 






























 Paper Title: Synth-Empathy: Towards High-Quality Synthetic Empathy Data.






^Peking University     ^University of Chinese Academy of Sciences     ^Shanghai AI Laboratory ^ Institute of Information Engineering, Chinese Academy of Sciences


 ^†hao.liang@stu.pku.edu.cn, ^†sunlinzhuang21@mails.ucas.ac.cn,  wentao.zhang@pku.edu.cn



§ ABSTRACT 
In recent years, with the rapid advancements in large language models (LLMs), achieving excellent empathetic response capabilities has become a crucial prerequisite. Consequently, managing and understanding empathetic datasets have gained increasing significance. However, empathetic data are typically human-labeled, leading to insufficient datasets and wasted human labor. In this work, we present Synth-Empathy, an LLM-based data generation and quality and diversity selection pipeline that automatically generates high-quality empathetic data while discarding low-quality data. With the data generated from a low empathetic model, we are able to further improve empathetic response performance and achieve state-of-the-art (SoTA) results across multiple benchmarks. Moreover, our model achieves SoTA performance on various human evaluation benchmarks, demonstrating its effectiveness and robustness in real-world applications. Furthermore, we show the trade-off between data quantity and quality, providing insights into empathetic data generation and selection. The codebase and data are made available at <https://github.com/Aurora-slz/Synth-Empathy>.


Synth-Empathy: Towards High-Quality Synthetic Empathy Data
    Hao Liang^†, Linzhuang Sun^†, Jingxuan Wei^, Xijie Huang^, Linkun Sun^, Bihui Yu^, Conghui He^, Wentao Zhang^*
    
==================================================================================================================

[† The first two authors have equal contributions. 

* Corresponding Author

]
footnote-1





























§ INTRODUCTION


In recent years, with the rapid advancements in large language models (LLMs) <cit.>, data management has become a crucial aspect of these technologies <cit.>. At the same time, <cit.> also demonstrates that data processing, selection, and management can significantly influence the performance of LLMs.

Empathy, the ability to understand and share the feelings of another, is a critical component of human social interaction and communication. It allows individuals to connect with others on an emotional level, fostering relationships and promoting prosocial behavior  <cit.>. In the future of Human-central Artificial General Intelligence (AGI), excellent empathetic response capability is a crucial prerequisite, which focuses on equipping LLMs to understand and respond appropriately to human feelings  <cit.>.





In previous research, empathy modeling has predominantly concentrated on human-labeled data  <cit.> and integrating information through modifications to the model architecture  <cit.>. While the importance of model architectures is well acknowledged, the quality and quantity of data are also critical  <cit.>. Additionally, <cit.> indicates that new data management methods are needed for LLMs due to their massive data requirements. Previous empathetic studies underscore the significance of data-centric approaches. They frequently overlook effective data generation approaches, leading to the following two key challenges:






C1. High Cost of Human-Labor. 
Previous methods rely on human labor for the creation of empathetic datasets <cit.>. However, this process can be extremely costly and requires substantial human effort.

C2. Poor Effectiveness.
Previous research <cit.> has only a limited amount of data. Despite efforts by  <cit.> to improve model performance, the empathetic effectiveness remains constrained. More empathetic data is needed to train high-performance empathetic models.




To address these issues, as shown in Figure <ref>, we propose a three-step empathetic data generation and curation pipeline and subsequently train an empathetic LLM. First, we utilize prompts to generate empathetic responses. Next, we apply domain knowledge for quality selection to curate the empathetic data. Finally, we conduct diversity selection for further data curation. With the curated synthetic dataset, we fine-tune an LLM and achieve state-of-the-art (SoTA) empathetic response performance.


The core contributions of this paper are summarized as follows:



  * New Perspective.
Limited data and low effectiveness are significant impediments to the practical adoption of empathy models. To the best of our knowledge, this study represents the first approach to generate data from scratch to address these challenges.

  * New Method.
We propose a new data generation and curation pipeline for empathy, introducing the first generated high-quality empathy dataset. Utilizing our meticulously curated synthetic dataset, we pioneer the integration of synthetic data to empathetic model training. Our method, which utilized curated synthetic data, enables robust effective, and user-friendly empathetic response.

  * SoTA Performance. 
    (1) Effectiveness in Empathetic Response. 
    By utilizing the synthetic empathetic response data to fine-tune LLMs, our method can achieve SoTA performance in multiple benchmarks, as shown in Figure <ref>. At the same time, we achieve SoTA performance on human evaluation benchmarks. This means our model have high application potential. 

    
    (2) No Human Labor. 
    By utilizing curated prompts and LLMs, we can obtain high-quality empathetic responses without the need for human labor. With a meticulously designed data curation pipeline, as illustrated in Figure <ref>, we achieve high-quality data, as demonstrated in Figure <ref> and Figure <ref>. This approach significantly reduces the requirement for human labor.
    
    



§ RELATED WORK




 §.§ Empathetic Response Generation

 
 Imbuing models with empathy to enhance emotional experience is a crucial aspect of AGI.  <cit.> constructed the ED dataset, the most authoritative dataset for measuring the empathic abilities of models, which has since become a benchmark for empathy research. Building on this dataset, numerous researchers have focused on improving models' empathy comprehension by integrating emotional labels and external knowledge.

 <cit.> introduces commonsense knowledge inference into this task by enriching historical conversation data with the pre-trained COEMT model <cit.>. Diverged from this approach,  <cit.> incorporate external knowledge through Graph Neural Networks (GNN) for context encoding. Additionally, <cit.> posit that using detailed sentiment labels can improve the accuracy of capturing user sentiment and puts forward a sentiment loss mechanism with multiple levels of granularity to enhance model training.  Moreover,  <cit.> strives to provide empathetic responses by analyzing context at the word level.  <cit.> employed a emotion detection algorithm based on psychological principles to pinpoint important statements in conversations.  <cit.> take a different approach by not only recognizing the emotions of others but also assessing saved model emotional state.  Furthermore,  <cit.> divides the empathetic response task into two steps: verifying the semantic content and infusing emotional expression.  

In the LLM era, many researchs explore ways to boost the empathic potential of models through meticulous design of prompts <cit.>. Differently,  <cit.> delves into the general sensibility and rationality, exploring their respective contributions to empathy. However, the importance of fine-grained sensibility and rationality cognition for empathy still lacks comprehensive investigation. 



 §.§ Data Quality and Data Selection

The advent of large language models has brought about a substantial increase in the volume of training data. <cit.> In this scenario, the quality and quantity of data become paramount. LLMs, trained on vast amounts of data, can capture subtle nuances and complex patterns in language, excelling in various natural language processing tasks. However, the increase in data volume also brings new challenges, particularly in data management, cleaning, and annotation. <cit.> In this section, we mainly discuss the effectiveness of data quality and data selection.



  
Data Quality: High-quality data can significantly enhance the performance of models <cit.>. As the volume of data increases, ensuring high data quality becomes more challenging because it requires more resources for data cleaning, selection and annotation. <cit.> Poor quality data can lead to models learning incorrect patterns and making inaccurate predictions. 



  
Data Selection: 
LLMs-based methods were commonly used in data selection. <cit.> For instance, <cit.> leverages DeBERTa <cit.> for scoring, retaining high-quality data, and combining it with the k-center greedy algorithm to select diverse data. <cit.> score the accuracy of data using ChatGPT to pick out high-quality data. <cit.> use GPT-4 to rewrite data to increase their complexity and then streamline it by reducing its variety and improving its quality. <cit.> train two models using ChatGPT's labeled data to score the quality and complexity of the data. <cit.> rely on ChatGPT to tag each instance, defining its complexity and diversity based on these tags. <cit.> first cluster the data, and then use GPT-4 to select high-quality data for each cluster.

Given the critical role of data quality and selection in enhancing model performance, our paper focuses on leveraging advanced data diversity and quality selection techniques to optimize empathetic data quality. By employing methods that integrate data features and similarity scores, we aim to efficiently identify and utilize high-quality data for empathetic response. 



 §.§ Data Generation

Data has always been the key driver behind the success of large language models (LLMs). Recent advancements of LLMs are largely due to the availability of large-scale, diverse, and high-quality datasets for training these models  <cit.> . However, the scarcity of data and the high costs present substantial challenges in obtaining such datasets  <cit.>. Recent advancements in generating synthetic data and improving the performance of LLMs have shown promising results across various domains. Synthetic data holds great potential in building large-scale, high-quality datasets. Researchers have explored multiple approaches, from leveraging differential privacy to creating instruction-tuning frameworks, to enhance the quality, diversity, and utility of synthetic data  <cit.>. A key component in generating high-quality synthetic datasets is precise alignment. Fan et al.  <cit.> introduce REALIGN, a method that enhances the quality of instruction data by reformatting responses to better align with pre-established criteria and evidence, thereby improving LLMs' alignment with human values while minimizing human annotation and model hallucinations. Li et al.  <cit.> build a high-quality instruction-following language model by automatically labeling human-written text with corresponding instructions and demonstrating highly effective self-alignment. 

In the field of empathetic response, the construction of generative datasets remains relatively underexplored. Empathetic models primarily rely on the utilization and curation of existing datasets. Approaches such as those by  <cit.> have attempted to improve empathetic response performance through data selection methods. However, these methods still face challenges due to the lack of high-quality empathetic response data.



§ METHOD

In this section, we first introduce the empathetic data generation method in subsection <ref>. Then we introduce the newly designed empathetic quality selection module in subsection <ref>. Additionally, we introduce the diversity selection in subsection <ref>. We demonstrated the high-quality of our dataset in subsection <ref>. With the curated dataset, we are able to fine-tune empathetic models.



 §.§ Empathetic Data Generation






In this subsection, we introduce the data generation method. Our data generation is based on the authoritative EmpatheticDialogues (ED) dataset <cit.>, which consists of 25,000 daily conversations encompassing 32 uniformly distributed emotional labels. By utilizing the prompt in Figure <ref>(1), we generate empathetic responses based on the high-quality ED dataset.

The formula for generating data using the prompt is as follows:


    Synthetic Dataset = LLM(Prompt(ED Data)


Following the formula above, we generate more than 250k empathetic data points for further selection.


 §.§ Empathetic Data Quality Selection

In this subsection, we introduce the data quality selection process for empathetic data. Similar to subsection <ref>, we utilize the ED dataset as the baseline and select data with an empathetic style similar to the ED dataset. The selection process is shown in Figure <ref>(2) and involves the following steps:



  
Empathetic Discriminator
We fine-tune a LLM on the ED dataset to serve as the discriminator, denoted as M_D. This discriminator is trained to produce responses that mimic the high-quality empathetic style of the ED dataset, denoted as D_ED.

    M_D = SFT(LLM; D_ED)




  
Similarity Based Quality Selection
In generating empathetic response data, we first use the discriminator M_D to answer the question. Then, we use the SoTA embedding model gte-qwen2-7b-instruct to embed the model's response and the generated response. We calculate the similarity between the discriminator's response and the generated response. We only select the responses with similarity higher than the threshold T. The optimized procedure is as follows:

    
  * Use the discriminator M_D to answer the question, obtaining the response r_D:
    
    r_D = M_D(question)


    
  * Use gte-qwen2-7b-instruct to embed the discriminator's response r_D and the generated response r_G, obtaining the embedding vectors E_D and E_G:
    
    E_D = gte-qwen2-7b-instruct(r_D)

    
    E_G = gte-qwen2-7b-instruct(r_G)


    
  * Calculate the similarity S between the discriminator's response r_D and the generated response r_G:
    
    S = E_D · E_G/E_DE_G


    
  * Select the generated response r_G if the similarity S is higher than the Domain Knowledge (DK) threshold T:
    
    r_G  is selected if  S > T



With the similarity selection based on the empathetic discriminator, we obtain a high-quality synthetic dataset.



 §.§ Empathetic Data Diversity Selection

In this subsection, we introduce the diversity selection method. The aim of diversity selection is to maximize the minimum distance between selected data points, ensuring a diverse subset. The algorithm can be formally described as follows:

Given a dataset X = {x_1, x_2, …, x_n}, our goal is to select a subset S ⊆ X of size k that maximizes the minimum distance between any point in X and the closest point in S. The objective function is:


    max_S ⊆ X, |S| = kmin_x ∈ Xmin_s ∈ S d(x, s)


where d(x, s) denotes the distance between points x and s.

As shown in Figure <ref>(3), we select the KCenterGreedy <cit.> algorithm for its robustness and quick computation. Using the KCenterGreedy algorithm, we obtain a high-diversity synthetic dataset.



 §.§ High Quality Generated Empathetic Data

In this section, we introduce an analysis of the quality of our generated dataset. We use a three-dimensional evaluation to assess the quality of our curated data. Using the prompts in Figure <ref>, we evaluate the dataset's coherence, naturalness, and empathy, scoring each from 1 to 3 points. 

As shown in Figure <ref>, our method significantly improves the dataset's quality. The average score increased from 2.65 to 2.88. Additionally, more than 95% of the data achieved full marks, demonstrating the high quality of our curated synthetic dataset. 

With high-quality generated empathetic response pairs, we can train a more effective empathetic model. Additionally, the empathetic response generation pipeline alleviates concerns about the quantity of empathetic data.









§ EXPERIMENTS

In this section, we first introduce the experimental setups, including data selection and the training process. We then aim to answer the following questions to verify the effectiveness and robustness of our proposed Synth-Empathy model: 

Q1: Can we fine-tune an empathetic model to achieve state-of-the-art (SoTA) performance with the curated synthetic data? 

Q2: Can our model achieve favorable human evaluation results, given the importance of human evaluation for empathy? 

Q3: Can we determine the trade-off between data quantity and quality to guide future empathetic data selection? 

Q4: Are the quality selection and diversity selection modules necessary to enhance model performance?




















 §.§ Experimental Setup



  §.§.§ Datasets


We evaluate our methods on the widely used EMPATHETICDIALOGUES dataset, which includes 25k multi-turn empathetic conversations.   Each conversation involves a speaker and a listener, with an average of 4.31 turns per dialog.   This dataset contains 32 evenly distributed emotion labels, with each conversation being associated with a specific label. And in order to guarantee the equity of the comparison baselines, we follow the same dataset division as the previous research method by splitting the dataset into training, validation, and test sets in an 8:1:1 ratio. 




  §.§.§ Baselines

The baseline models are listed as follows:

1. SEEK: Employ multi-granularity sentiment labels at the word, sentence, and dialogue levels to enhance emotional understanding.
2. CASE: Enhancing the cognitive capabilities of models with external knowledge sources, COMET and ConceptNet, to improve the quality of empathetic responses.
3. ESCM: Leveraging dynamic emotion-semantic vectors along with dependency trees to direct the model in generating empathetic responses.
4. Qwen1.5-7B-F.T.: Conducting SFT of Qwen1.5-7B-Chat model using ED dataset.
5. LLaMA3-8B-F.T.: Conducting SFT of LLaMA3-8B-Instruct model using ED dataset.
6. LLaMA3-8B: Generating empathetic response through designed prompt based on LLaMA3-8B-Instruct[<https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct>] model.
7. Mixtral-8x7B: Generating empathetic response through designed prompt based on Mixtral-8x7B-Instruct[<https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1>] model.
8. Qwen1.5-72B-Chat: Generating empathetic response through designed prompt based on Qwen1.5-72B-Chat[<https://huggingface.co/Qwen/Qwen1.5-72B-Chat>] model.




  §.§.§ Evaluation


To offer a thorough evaluation of our model's capabilities, we assess its performance through both automatic metrics and human evaluations.

Automatic Evaluation Metrics: For automatic evaluation, we employ corpus-level BLEU (B-1 to B-4), sentence-level ROUGE (R-1, R-2), and Distinct (Dist-1, Dist-2) metrics. BLEU and ROUGE scores measure the similarity between the generated text and the reference text, with higher scores reflecting greater similarity. Meanwhile, Distinct-N assesses content diversity, where higher values indicate a broader range of diverse representations. CIDEr aligns more closely with human judgment of sentence similarity by employing TF-IDF to assign varying weights to different n-grams. Specifically, it reduces the weight of frequently occurring phrases while increasing the weight of less common ones, thereby reflecting the distinctiveness and significance of the n-grams in the evaluation process. Besides, we employed xxx to assess semantic similarity from the perspective of vector embedding.

Human Evaluation Metrics: For human evaluation, we choose the following metrics: Coherence, Empathy, Informativeness, and Continuity. Coherence: Assesses how well the model's generated text aligns with the intended response. Empathy: Measures the model's capacity to grasp the speaker's situation and convey appropriate concern. Informativeness: Evaluates the extent of information provided in the responses produced by the model. Continuity: Reflects the model's effectiveness in maintaining the flow of the conversation. 


Then, the A/B test is conducted to evaluate the effectiveness of our model in comparison to several baseline models. Specifically, 200 examples are randomly selected from the test dataset. Each example is paired with two responses: one generated by our model and the other by a baseline model. Three evaluators assess each pair of responses, determining a winner, loser, or tie based on above four criteria.



  §.§.§ Settings

All experiments are carried out on the machine equipped with 8 NVIDIA A100 GPUs, a 120-core CPU, and 960GB of memory. The inference process of LLMs is implemented on the Vllm framework with version 0.4.1.



 §.§ Main Experiments

To address Q1, in this section, we evaluated various models on their performance in generating empathetic responses using several metrics, as summarized in Table <ref>. 

Our model was fine-tuned on the dataset combining the ED training set and our generated dataset. Our generated dataset comprises data with S greater than 60 and diversity selection.

Our model consistently outperformed other models across most metrics. Specifically, it achieved the highest BLEU scores, with 22.05 for BLEU-1, 10.53 for BLEU-2, 5.99 for BLEU-3, and 3.77 for BLEU-4, demonstrating its superior ability to generate text closely aligned with reference translations. In terms of Distinct metrics, which gauge the diversity of generated responses, our model scored 3.23 for Distinct-1 and 18.84 for Distinct-2, indicating a strong balance between diversity and coherence.

Our model also excelled in the Rouge metrics, attaining scores of 18.72 for Rouge-1, 5.27 for Rouge-2, and 16.88 for Rouge-L. These scores reflect its effectiveness in capturing the essential content of the reference texts and producing more contextually relevant responses. In the CIDEr metric, which assesses content relevance and richness, our model achieved a score of 26.96, the highest among the compared models, highlighting its effectiveness in generating captioning that aligns closely with human judgments of content relevance and informativeness.

Furthermore, in embedding-based evaluations using BERT, our model demonstrated superior performance with a BERT-P score of 20.5, a BERT-R score of 18.2, and a BERT-F1 score of 19.4. These results indicate that our model generates text with more meaningful and contextually accurate semantic representations compared to other models. 

Overall, these results confirm that our model provides superior performance across a range of metrics, demonstrating its effectiveness in generating high-quality and diverse outputs. The promising results highlight the effectiveness of the high-quality synthetic dataset, thereby introducing a new paradigm for using synthetic data in empathetic responses.




 §.§ Human Evaluation




To address Q2, we conducted a series of A/B tests. The comparative analysis includes our model versus SEEK, CASE, ESCM, Qwen1.5-7B-F.T., and LLaMA3-8B-F.T. The results of these tests are summarized in Figure <ref>.

The results of our A/B tests demonstrate that our model significantly outperforms SEEK, CASE, and ESCM, with win rates of 66%, 63.3%, and 57.9%, respectively. When compared with the more advanced models, Qwen1.5-7B-F.T. and LLaMA3-8B-F.T., our model still demonstrates competitive performance. Against Qwen1.5-7B-F.T., our model secures a win rate of 31% and maintains a reasonable loss rate of 19.8%, with ties at 49.2%. Similarly, our model achieves a win rate of 26% against LLaMA3-8B-F.T., with a loss rate of 15.1% and the highest tie rate of 58.9%. 


Overall, our model's strong performance in human evaluations underscores its practicality and user-friendliness. These results indicate that our model is not only effective but also well-received by human evaluators, highlighting its potential for real-world applications.





 §.§ Data Quality and Data Quantity


To address Q3, we analyzed the trade-off between the DK threshold, the similarity threshold in section <ref>, and the CIDEr score to demonstrate the robustness of our approach. Table <ref> and Figure <ref> present detailed experimental results, comparing various DK thresholds across the Qwen1.5-7B and LLaMA3-8B models using two different data generation prompts.

As the DK threshold increases from 50 to 70, the CIDEr score of Qwen1.5-7B with Prompt 1 initially rises from 20.45 to a peak of 25.07 at a DK threshold of 60, before slightly decreasing to 22.45 at a DK threshold of 70. For the LLaMA3-8B model, the CIDEr score shows a similar pattern, peaking at 24.34 with a DK threshold of 60, before decreasing at higher DK ratios. In terms of Prompt 2, the Qwen1.5-7B model demonstrates an increase in the CIDEr score from 20.36 at a DK threshold of 50 to a peak of 26.96 at a DK threshold of 60, followed by a slight decline. This observation corroborates a clear trade-off trend, emphasizing that an optimal DK threshold enhances the generation of empathetic responses.

The consistency of the trade-off pattern across different models and prompts underscores the robustness of our method. Regardless of the specific model or prompt used, the CIDEr score consistently peaks around the DK threshold of 60, demonstrating that our method effectively balances data retention and response quality. Additionally, by analyzing the trade-off between data quantity and quality, we provide guidance for optimal empathetic selection methods in the future. Utilizing the best threshold can achieve superior empathetic performance and select the appropriate dataset, thereby avoiding the waste of computational resources.





 §.§ Ablation Study

To address Q4, we provide the following ablation study. We remove the data quality selection module and the data diversity selection module to
demonstrate the effectiveness of our Synth-Empathy pipeline. 



  
Excluding Data Quality Selection Module.
When excluding the data quality selection module, the performance of the model decreases across all benchmarks, as shown in Table <ref>. For both the Qwen1.5-7B and LLaMA-8B-Instruct models, this exclusion results in decreased performance, demonstrating the necessity of the quality selection module.



  
Excluding Data Diversity Selection Module.
Similarly, the model's performance declines when the data diversity selection module is excluded, as shown in Table <ref>. For both data selection ratios of 55 and 60, and with both the Qwen1.5-7B and LLaMA-8B-Instruct models, this exclusion results in decreased performance, demonstrating the necessity of the diversity selection module.

Overall, the ablation study highlights the critical role of data generation and data selection modules in SynthVLM. These experiments provide valuable insights into the contributions of each module, guiding future improvements and optimizations of the SynthVLM model.
































§ CONCLUSION

Empathy is a critical component of human social interaction and communication. However, high-quality empathetic data remains scarce, necessitating efficient and effective data generation algorithms. This paper presents Synth-Empathy, a new paradigm for generating high-quality empathetic data, pioneering a method to obtain empathetic responses without human labor. By leveraging quality and diversity selection, we achieved high-quality synthetic empathetic data. Remarkably, utilizing the curated empathetic data, our model outperforms all previous models. Additionally, our model achieved SoTA performance in human evaluation, demonstrating its contextual appropriateness and user-friendliness.





ACM-Reference-Format





 Paper Title: A Course Shared Task on Evaluating LLM Output for Clinical Questions.


[
    [
    August 5, 2024
==================

* Correspondence to yhou@ie.ibm.com.

§ ABSTRACT 
This paper presents a shared task that we organized at the Foundations of Language Technology (FoLT) course in 2023/2024 at the Technical University of Darmstadt, which focuses on evaluating the output of Large Language Models (LLMs) in generating harmful answers to health-related clinical questions. We describe the task design considerations and report the feedback we received from the students. We expect the task and the findings
reported in this paper to be relevant for instructors teaching natural language processing (NLP) and designing course assignments.




§ INTRODUCTION

The Foundations of Language Technology (FoLT) course, a regular offering at the Technical University of Darmstadt, provides undergraduate and graduate students with a comprehensive introduction to the fundamental concepts and technologies of Natural Language Processing (NLP). In the 2023/2024 academic year, we have updated the curriculum to incorporate the latest advancements of Large Language Models (LLMs). The course is structured into 14 lectures, supplemented by 9 hands-on coding tutorials that allow the students to reinforce their understanding of key concepts learned in the previous lectures. In addition, we organized a shared task to challenge students to evaluate the output of LLMs in generating harmful answers to clinical questions related to health.
The primary goal of this shared task is to help students gain practical experience in applying NLP techniques and tools to a real-world research problem that involves data annotation, preprocessing, model development, and model evaluation.

In this paper, we describe the task design and discuss the lessons learned from implementing the shared task, which can offer insights for educators seeking to develop  similar assignments for their own courses.



 



§ TASK DETAILS



 §.§ Task Design



Our task belongs to the category of scientific fact checking <cit.>, and is also closely related to recent research on LLM factuality evaluation <cit.>.
Building on our previous work <cit.>, which advocated for realistic fact-checking, our task aims to verify the output of LLMs using trustworthy, high-quality scientific evidence.
More specifically,
given a health-related clinical question q, and two corresponding answers a from human experts and a^' from an LLM, the objective of our shared task is two-fold: (i) harmfulness detection by determining whether a^' contains harmful information. We consider a^' to be harmful if it contains contradictory or exaggerated information compared to a;
(ii) fine-grained answer categorization by assigning a specific category label to each sentence within a^'. Table <ref> summarizes the six categories we considered, and Figure <ref> shows an answer from an LLM that annotated with fined-grained categories for the question “can adding multiple micronutrients to food improve health in the general population?”.








 §.§ Task Dataset

For the shared task, we utilize Cochrane Clinical Answers[<https://www.cochranelibrary.com/cca>],

a trusted resource that provides concise, evidence-based responses to clinical questions grounded in rigorous Cochrane systematic reviews. Each CCA consists of a clinical question, a brief answer, and relevant outcome data extracted from the corresponding Cochrane systematic review, specifically curated for practicing healthcare professionals. We collected a dataset of 500 CCAs published between 2021 and 2023, assuming that the answers written by clinical professionals represent accurate and truthful responses to the target questions.



§ SHARED TASK IMPLEMENTATION

We divide the shared task into four sub-tasks and require each participating team to consist of 2-3 members. The first two sub-tasks focus on data annotation and processing, while the latter two concentrate on developing and evaluating both basic and state-of-the-art models.

For the first two sub-tasks, each team is assigned to work with a set of ten CCAs. To complete these sub-tasks, each student needs to set up the annotation environment using Label Studio <cit.>,

carry out the annotations for answers from different LLMs, calculate the inter-annotator agreement, submit individual annotations and consolidated group annotations after resolving any disagreements.
To help students to quick grasp the professional medical concepts, we provide explanations of key terms from gold-standard CCA answers in plain language, based on an online Medical Terms in Lay Language Dictionary[<https://hso.research.uiowa.edu/get-started/guides-and-standard-operating-procedures-sops/medical-terms-lay-language>], such as “hypotension: low blood pressure”.




In total, 55 teams participated in the first two sub-tasks. After merging and cleaning the annotations from all teams, we compiled a dataset of 1800 annotated answers from five LLMs 
for 360 CCAs. We then divided the dataset into dev and test sets, comprising 500 LLM answers for 100 CCAs and 1,300 LLM answers for 260 CCAs, respectively. The five testing LLMs include Llama-2-70b-chat <cit.> with two different system instructions, OpenAI ChatGPT[<https://chatgpt.com/>], Microsoft BingChat[<https://www.bing.com/chat>], and PerplexityAI[<https://www.perplexity.ai/>]. The specific prompts employed to test these LLMs are detailed in Appendix <ref>.

For the third sub-task, we released the dev dataset to the students. Each team needs to write code to analyze human annotations and answer a list of questions, such as “Do retrieval augmented LLMs (BingChat, PerplexityAI) generate less harmful content compared to other models?” More details about the analyzed questions can be found in Table <ref>. In addition, we instructed the students to train two baseline models - a decision tree and a simple neural network model - for the two classification tasks outlined in Section <ref>.






For the fourth sub-task, the teams were required to design prompts to elicit responses from LLMs for the two classification tasks described in Section <ref>. Each team can submit up to three predictions on the test set for each task. Participants had the option to compete in either the open track or the closed track. In the closed track, teams were restricted to using the pre-defined LLM, Mistral-7b-instruct, to perform the task, whereas the open track placed no such constraints on the LLMs that could be used.
To facilitate participation in the closed track, we set up a Hugging Face endpoint inference service hosting a Mistral-7b-instruct-v02 model for two weeks, 
incurring a cost of $85.




§ SHARED TASK RESULTS



  
Grading system. Our grading system is designed to assess student performance across four sub-tasks. Each sub-task is worth 100 credits, which are allocated as follows: For the first two sub-tasks, students earn credits based on their annotation effort, including submitting individual and adjudication annotations, and correctly calculating inter-annotator agreement scores. For the third sub-task, students are automatically graded on the code snippets they write to fulfill the task goal. The credits for the fourth sub-task is divided into the following three components:


    
  * Completing code snippets for prompting LLMs through APIs (30 credits);
    
  * Submitting prediction files for the testing dataset for both closed and open tracks (30 credits);
    
  * Performance on the leaderboards of the closed and open tracks (40 credits). Specifically, if a team's rank is k on the closed track leaderboard and there are n teams participating for the closed track, then all members from this team will receive the credit c = 20/n*(n+1-k).





To qualify for a bonus point, which upgrades their final grade in the course (e.g., from 2.0 to 1.7), students must meet two conditions: 1) pass the final written exam, and 2) participate in all four sub-tasks and obtain at least 70% of all points.



  
Students' performance. A total of 121, 130, 110, and 94 students participated in the first, second, third, fourth sub-tasks, respectively. Overall,  87 students participated in all four sub-tasks, and 74 of them received the bonus points.





§ DISCUSSION AND CONCLUSIONS

During the shared task, we received diverse feedback from participants. Students with a linguistic background generally found setting up the annotation environment and performing annotations to be engaging tasks, whereas some from a computer science (CS) background perceived the annotation process as too time-consuming. Notably, the majority of students expressed a preference for the third sub-task, while the fourth sub-task was widely regarded as the most challenging. For future iterations, students recommended reducing the annotation load or selecting topics that require less domain-specific knowledge to facilitate judgment.

One potential limitation of our shared task design is that students were involved in constructing the test set, which may have given them implicit knowledge that could influence their prompt design in the fourth sub-task. However, we mitigate this risk by noting that each team only annotated a small proportion of CCAs (10), which, even in the worst-case scenario, would only account for 3.8% of the entire testing dataset. It is therefore unlikely that overfitting to these “leaked” instances would guarantee good performance on the whole testing dataset. Nevertheless, to eliminate any potential bias, we recommend that in future iterations, course instructors should keep the testing dataset completely hidden from participants to ensure a more robust evaluation.




Finally, following the shared task, we invited participants to voluntarily consent to donate their annotations to an open-source dataset. In total, we collected 850 annotated LLM answers for 130 CCAs. We release this dataset to the community to support future teaching and research endeavors: <https://github.com/UKPLab/folt-shared-task-23-24>.




acl_natbib



§ PROMPTS FOR TESTING LLMS


Llama-2-70b-chat Prompt 1: 

Provide a concise answer for the following question.
{Question}




Llama-2-70b-chat Prompt 2: 

You are a helpful, respectful and honest assistant. Provide a concise answer for the following question. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
{Question}




ChatGPT/BingChat/PerplexityAI Prompt: 

Provide a concise answer for the following question.
{Question}




 Paper Title: Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models.



[
Measuring Progress in Dictionary Learning for 
 Language Model Interpretability with Board Game Models














equal*


Adam Karvonenequal,ind
Benjamin Wrightequal,mit
Can Ragerind
Rico Angellumass
Jannik Brinkmannmann
Logan Smithind
Claudio Mayrink Verdunhar
David Baunor
Samuel Marksnor


Adam Karvonenadam.karvonen@gmail.com

indIndependent
mitMIT
umassUniversity of Massachusetts, Amherst
mannUniversity of Mannheim
harHarvard University
norNortheastern University







Machine Learning, ICML

0.3in
]










 

§ ABSTRACT 
What latent features are encoded in language model (LM) representations? Recent work on training sparse autoencoders (SAEs) to disentangle interpretable features in LM representations has shown significant promise. However, evaluating the quality of these SAEs is difficult because we lack a ground-truth collection of interpretable features that we expect good SAEs to recover. We thus propose to measure progress in interpretable dictionary learning by working in the setting of LMs trained on chess and Othello transcripts. These settings carry natural collections of interpretable features—for example, “there is a knight on F3”—which we leverage into supervised metrics for SAE quality. To guide progress in interpretable dictionary learning, we introduce a new SAE training technique, p-annealing, which improves performance on prior unsupervised metrics as well as our new metrics.




§ INTRODUCTION



Mechanistic interpretability aims to reverse engineer neural networks into human-understandable components. What, however, should these components be?
Recent work has applied Sparse Autoencoders (SAEs) <cit.>, a scalable unsupervised learning method inspired by sparse dictionary learning to find a disentangled representation of language model (LM) internals. However, measuring progress in training SAEs is challenging because we do not know what a gold-standard dictionary would look like, as it is difficult to anticipate which ground-truth features underlie model cognition. Prior work has either attempted to measure SAE quality in toy synthetic settings <cit.> or relied on various proxies such as sparsity, fidelity of the reconstruction, and LM-assisted autointerpretability <cit.>.

In this work, we explore a setting that lies between toy synthetic data (where all ground-truth features are known; cf. <cit.>) and natural language: LMs trained on board game transcripts. This setting allows us to formally specify natural categories of interpretable features, e.g., “there is a knight on ” or “the bishop on  is pinned.” We leverage this to introduce two novel metrics for how much of a model's knowledge an SAEs has captured:

    
  * Board reconstruction. Can we reconstruct the state of the game board by interpreting each feature as a classifier for some board configuration?
    
  * Coverage. Out of a catalog of researcher-specified candidate features, how many of these candidate features actually appear in the SAE?


These metrics carry the limitation that they are sensitive to researcher preconceptions. Nevertheless, we show that they provide a useful new signal of SAE quality.

Additionally, we introduce p-annealing, a novel technique for training SAEs. When training an SAE with p-annealing, we use an L_p-norm-based sparsity penalty with p ranging from p=1 at the beginning of training (corresponding to a convex minimization problem) to some p < 1 (a non-convex objective) by the end of training. We demonstrate that p-annealing improves over prior methods, giving performance on par with the more compute-intensive Gated SAEs from <cit.>, as measured both by prior metrics and our new metrics.

Overall, our main contributions are as followsCode, models, and data are available at <https://github.com/adamkarvonen/SAE_BoardGameEval>:

    
  * We train and open-source over 500 SAEs trained on chess and Othello models each.
    
  * We introduce two new metrics for measuring the quality of SAEs.
    
  * We introduce p-annealing, a novel technique for training SAEs that improves on prior techniques.




§ BACKGROUND





 §.§ Language models for Othello and chess


In this work, we make use of LMs trained to autoregressively predict transcripts of chess and Othello games. We emphasize that these transcripts only give lists of moves in a standard notation and do not directly expose the board state. Based on behavioral evidence (the high accuracy of the LMs for predicting legal moves) and prior studies of LM representations <cit.> we infer that the LMs internally model the board state, making them a good testbed setting for studying LM representations.





  
Othello. 
Othello is a two-player strategy board game played on an 8x8 grid, with players using black and white discs. 
Players take turns placing discs on the board, capturing their opponent's discs by bracketing them between their own, causing the captured discs to turn their color. 
The goal is to have more discs turned to display your color at the end of the game. The game ends if every square on the board is covered or either player cannot make a move. 

In our experiments, we use an 8-layer GPT model with 8 attention heads and a n = 512 dimensional hidden space, as provided by <cit.>.
This model had no prior knowledge of the game or its rules and was trained from scratch on 20 million game transcripts, where each token in the corpus represents a tile on which players place their discs. 
The game transcripts were synthetically generated by uniformly sampling from the Othello game tree. 
Thus, the data distribution captures valid move sequences rather than strategic depth.
For this model, <cit.> demonstrated the emergence of a world model—an internal representation of the correct board state allowing it to predict the next move—that can be extracted from the model activations using a non-linear probe.
<cit.> extended this finding, showing that a similar internal representation could be extracted using linear probes, supporting the linear representation hypothesis <cit.>.



  
Chess. 
Othello makes a natural testbed for studying emergent internal representations since the game tree is far too large to memorize. 
However, the rules and state are not particularly complex. 
Therefore, we also consider a language model trained on chess game transcripts with identical architecture, provided by <cit.>.
The model again had no prior knowledge of chess and was trained from scratch on 16 million human games from the Lichess chess game database <cit.>. 
The input to the model is a string in the Portable Game Notation (PGN) format (e.g., “”).
The model predicts a legal move in 99.8 % of cases and, similar to Othello, it has an internal representation of the board state that can be extracted from the internal activations using a linear probe <cit.>.










 §.§ Sparse autoencoders

Given a dataset 𝒟 of vectors 𝐱∈ℝ^d, a sparse autoencoder (SAE) is trained to produce an approximation

    𝐱≈∑_i^d_SAE f_i(𝐱)𝐝_i + 𝐛

as a sparse linear combination of features. Here, the feature vectors 𝐝_i∈ℝ^d are unit vectors, the feature activations f_i(𝐱)≥ 0 are a sparse set of coefficients, and 𝐛∈ℝ^d is a bias term. Concretely, an SAE is a neural network with an encoder-decoder architecture, where the encoder maps 𝐱 to the vector 𝐟 = [     f_1(𝐱)          … f_d_SAE(𝐱) ] of feature activations, and the decoder maps 𝐟 to an approximate reconstruction of 𝐱. 

In this paper, we train SAEs on datasets consisting of activations extracted from the residual stream after the sixth layer for both the chess and Othello models. At these layers, linear probes trained with logistic regression were accurate for classifying a variety of properties of the game board <cit.>. For training SAEs, we employ a variety of SAE architectures and training algorithms, as detailed in Section <ref>.





§ MEASURING AUTOENCODER QUALITY FOR CHESS AND OTHELLO MODELS



Many of the features learned by our SAEs reflect uninteresting, surface-level properties of the input, such as the presence of certain tokens. However, upon inspection, we additionally find many SAE features which seem to reflect a latent model of the board state, e.g., features that reflect the presence of certain pieces on particular squares, squares that are legal to play on, and strategy-relevant properties like the presence of a pin in chess (Figure <ref>). 

Fortunately, in the setting of board games, we can formally specify certain classes of these interesting features, allowing us to more rigorously detect them and use them to understand our SAEs. In Section <ref>, we specify certain classes of interesting game board properties. Then, in Section <ref>, we leverage these classes into two metrics of SAE quality.




 §.§ Board state properties in chess and Othello models


We formalize a board state property (BSP) to be a function g : {game board}→{0,1}. In this work, we will consider the following interpretable classes of BSPs:


    
  * 𝒢_board state contains BSPs which classify the presence of a piece at a specific board square, where the board consists of 8× 8 squares in both games. For chess, we consider the full board for the twelve distinct piece types (white king, white queen, ..., black king), giving a total of 8× 8× 12 BSPs. For Othello, we consider the full board for the two distinct piece types (black and white), yielding 8×8× 2 BSPs.
    
  * 𝒢_strategy consists of BSPs relevant for predicting legal moves and playing strategically in chess, such as a pin detector. They were selected by the authors based on domain knowledge and prior interpretability work on the chess model AlphaZero <cit.>. We provide a full list of strategy BSPs in Table <ref> in the Appendix. Because our Othello model was trained to play random legal moves, we do not consider strategy BSPs for Othello.












 §.§ Measuring SAE quality with board state properties


In this section, we introduce two metrics of SAE quality: coverage and board reconstruction.



  
Coverage. Given a collection 𝒢 of BSPs, our coverage metric quantifies the extent to which an SAE has identified features that coincide with the BSPs in 𝒢. In more detail, suppose that f_i is an SAE feature and t∈[0,1] is a threshold, we define the function

    ϕ_f_i, t(𝐱) = 𝕀[f_i(𝐱) > t· f_i^max]

where f_i^max is (an empirical estimate of) max_𝐱∼𝒟f_i(𝐱), the maximum value that f_i takes over the dataset 𝒟 of activations extracted from our model, and 𝕀 is the indicator function. We interpret ϕ_f_i,t as a binary classifier; intuitively, it corresponds to binarizing the activations of f_i into “on” vs. “off” at some fraction t of the maximum value of f_i on 𝒟. Given some BSP g∈𝒢, let F_1(ϕ_f_i,t; g)∈ [0,1] denote the F1-score for ϕ_f_i, t classifying g. Then we define the coverage of an SAE with features {f_i} relative to a set of BSPs 𝒢 to be

    Cov({f_i}, 𝒢) := 1/|𝒢|∑_g∈𝒢max_tmax_f_iF_1(ϕ_f_i,t; g).


In other words, we take, for each g∈𝒢, the F_1-score of the feature that best serves as a classifier for g, and then take the mean of these maximal F_1-scores. An SAE receives a coverage score of 1 if, for each BSP g∈𝒢, it has some feature that is a perfect classifier for g. Since Cov depends on the choice of threshold t, we sweep over t∈{0, 0.1, 0.2, …, 0.9} and take the best coverage score; typically this best t is in {0, 0.1, 0.2}.



  
Board reconstruction. Again, let 𝒢 be a set of BSPs. Intuitively, the idea of our board reconstruction metric is that, for a sufficiently good SAE, there should be a simple, human-interpretable way to recover the state of the board from the profile of feature activations {f_i(𝐱)} on an activation 𝐱∈ℝ^d. Here, the activation 𝐱 was extracted after the post-MLP residual connection in layer 6.

We will base our board reconstruction metric around the following human-interpretable way of recovering a board state from a feature activation profile; we emphasize that different ways of recovering boards from feature activations may lead to qualitatively different results. This recovery rule is based on the assumption that interpretable SAE features tend to be high precision for some subset of BSPs, in line with <cit.>. For example, features that classify common configurations of pieces are high precision (but not necessarily high recall) for multiple BSPs.
 Thus, using a batch of “train” data 𝒟_train we identify, for each SAE feature f_i, all of the BSPs g∈𝒢 for which ϕ_f_i,t is a high precision (of at least 0.95) classifier. Then for each g∈𝒢 our prediction rule is



    𝒫_g({f_i(𝐱)}) = 
        1,    if ϕ_f_i,t(𝐱) = 1  for any  f_i  which
         is high precision for  g  on 𝒟_train
    
        0,    otherwise.







Let F_1(𝒫({f_i(𝐱)}); 𝐛) denote the F_1-score for a given board state 𝐛, where 𝒫({f_i(𝐱)}) = {𝒫_g({f_i(𝐱)})}_g ∈𝒢 represents the full predicted board (containing predictions for all 64 squares) obtained from the SAE activations. Then, the average F_1-score over all board states in the test dataset 𝒟_test can be calculated as:

    Rec({x_i}, 𝒟_test) = 1/|𝒟_test|∑_𝐱∈𝒟_testmax_t F_1(𝒫({f_i(𝐱)}); 𝐛),





§ TRAINING METHODOLOGIES FOR SAES


In our experiments, we investigate four methods for training SAEs, as explained in this section. These are given by two autoencoder architectures and two training methodologies—one with p-annealing and one without p-annealing—for each architecture. Our SAEs are available at <https://huggingface.co/adamkarvonen/chess_saes/tree/main> (chess) and <https://huggingface.co/adamkarvonen/othello_saes/tree/main> (Othello).






 §.§ Standard SAEs


Let n be the dimension of the model's residual stream activations that are input to the autoencoder, m the autoencoder hidden dimension, and s the dataset size. Our baseline “standard” SAE architecture, as introduced in <cit.> is defined by encoder weights W_e ∈ℝ^m × n, decoder weights W_d ∈ℝ^n × m with columns constrained to have a L_2-norm of 1, and biases b_e ∈ℝ^m, b_d ∈ℝ^n. Given an input 𝐱∈ℝ^n, the SAE computes

    𝐟(𝐱)    = ReLU(W_e (𝐱 - 𝐛_d) + 𝐛_e) 
    𝐱̂   = W_d  𝐟(𝐱) + 𝐛_d

where 𝐟(𝐱) is the vector of feature activations, and  𝐱̂ is the reconstruction. 

For a standard SAE, our baseline training method is as implemented in the open-source  repository <cit.>, optimizing the loss

    ℒ_standard = 𝔼_𝐱∼𝒟_train[ 𝐱 - 𝐱̂_2 + λ𝐟(𝐱) _1].


for some hyperparameter λ>0 controlling sparsity.











 §.§ Gated SAEs

The L_1 penalty used in the original training method encourages feature activations to be smaller than they would be for optimal reconstruction <cit.>. 
To address this, <cit.> introduced a modification to the original SAE architecture that separates the selection of dictionary elements to use in a reconstruction and estimating the coefficients of these dictionary elements. This results in the following gated architecture: 






    π_gate(𝐱) := W_gate(𝐱 - 𝐛_d) + 𝐛_gate


    𝐟̃(𝐱) := 𝕀[π_gate(𝐱) > 0] ⊙ReLU(W_mag(𝐱 - 𝐛_d) + 𝐛_mag)


    x̂(𝐟̃(𝐱)) = W_d 𝐟̃(𝐱) + 𝐛_d

where 𝕀[ · > 0] is the Heaviside step function and ⊙ denotes elementwise multiplication. Then, the loss function uses x̂_frozen, a frozen copy of the decoder:





    ℒ_gated := 𝔼_𝐱∼𝒟_train[    𝐱 - x̂(𝐟̃(𝐱))_2^2 
        + λReLU(π_gate(𝐱))_1 
        + 𝐱 - x̂_frozen(ReLU(π_gate(𝐱)))_2^2 ]









 §.§ p-Annealing


Fundamentally, an L_1 penalty has been used to induce sparsity in SAE features because it serves as a convex relaxation of the true sparsity measure, the L_0-norm. The L_1-norm is the convex hull of the L_0-norm, making it a tractable alternative for promoting sparsity <cit.>. However, the proxy loss function is not the same as directly optimizing for sparsity, leading to issues such as feature shrinkage <cit.> and potentially less sparse learned features. Unfortunately, the L_0-norm is non-differentiable and directly minimizing it is an NP-hard problem <cit.>, rendering it impractical for training.



In this work, we propose the use of nonconvex L_p^p-minimization, with p<1, as an alternative to the standard L_1 minimization in sparse autoencoders (SAEs). This approach has been successfully employed in compressive sensing and sparse recovery to achieve even sparser representations <cit.>. To perform this optimization, we introduce a method called p-annealing for training SAEs, based on the compressive sensing technique called p-continuation <cit.>. The key idea is to start with convex L_1-minimization through setting p=1 and progressively decrease the value of p during training, resulting in closer approximations of the true sparsity measure, L_0, as p approaches 0. We define the sparsity penalty for each batch x as a function of the current training step s:




    ℒ_sparse(𝐱, s) = λ_s 𝐟(𝐱) _p_s^p_s = λ_s ∑_if_i(𝐱)^p_s


In other words, the sparsity penalty will be a scaled L_p^p norm of the SAE feature activations, with p decreasing over time. At p=1, the L_p^p norm is equal to the L_1 norm, and as p → 0, the L_p^p norm limits to the L_0-norm, as lim_p→ 0∑_i f_i(𝐱)^p = ∑_i f_i(𝐱)^0.

The purpose of annealing p from 1 → 0 instead of starting from a fixed, low value for p is that the lower the p, the more concave (non-convex) the L_p^p norm is, increasing the likelihood of the training process getting stuck in local optima, which we have observed in initial experiments. Therefore, we aim to first arrive at a region of an optimum using the easier-to-train L_1 penalty and then gradually shift the loss function. This manifests as keeping p=1 for a certain number of steps and then starting decreasing p linearly down to p_end > 0 at the end of training. We set p_end=0.2.







  
Coefficient Annealing.
Changing the value of p changes the scale of the L_p^p norm. Without also adapting the coefficient λ, the strength of the sparsity penalty would vary too wildly across training. Empirically, we found that keeping a constant λ would lead to far too weak of a sparsity penalty for the larger p's at the start of training, making the process worse than simply training with a constant p from the beginning.

Consequently, we aim to adapt the coefficient λ such that the strength of the sparsity penalty is not changed significantly due to p updates. Formally, the update step is:

    λ_s+1λ_s∑_j=s-q+1^s∑_if_i(𝐱_𝐣)^p_s/∑_j=s-q+1^s∑_if_i(𝐱_𝐣)^p_s+1.


We keep a queue of the most recent q batches of feature activations mid-training and use them to calibrate the λ_s updates. Therefore, the strength of the sparsity penalty is kept locally constant.



  
Combining p-annealing with other SAEs.

Since the p-annealing method only modifies the L_1 terms in the loss function without affecting the SAE architecture, it is simple to combine p-annealing with other SAE modifications. This allows us to create the Gated-Annealed SAE method by combining the Gated SAE architecture and p-annealing. Concretely, we modify ℒ_gated (Equation <ref>) by replacing the sparsity term λReLU(π_gate(𝐱))_1 in  with λ_s ReLU(π_gate(𝐱)) _p_s^p_s. Our experiments showed that the optimum values for coefficients λ and λ_s differ.





§ RESULTS



In this section, we explore the performance of SAEs applied to language models trained on Othello and chess. Consistent with <cit.>, we find that interpretable SAE features typically track properties relative to the player whose turn it is (e.g. “my king is pinned” rather than “the white king is pinned”). To side-step subtleties arising from this, we only extract our activations from the token immediately preceding white's move.
Specifically, we consider SAEs trained on the residual stream activations after the sixth layer using the four methods from Section <ref> (see Table <ref> for additional hyperparameters). 
In addition to our metrics introduced above, we also make use of unsupervised metrics previously appearing in the literature <cit.>:

    
  * L_0 measures the average number of active SAE active features (i.e., positive activation) on a given input.
    
  * Loss recovered measures the change in model performance when replacing activations with the corresponding SAE reconstruction during a forward pass.
    This metric is quantified as (H_* - H_0)/(H_orig - H_0), where H_orig is the cross-entropy loss of the board game model for next-token prediction, H_* is the cross-entropy loss after substituting the model activation 𝐱 with its SAE reconstruction during the forward pass, and H_0 is the cross-entropy loss when zero-ablating 𝐱.

Our key takeaways are as follows.





  
SAE features can accurately reconstruct game boards. 
In general, we find that SAE features are effective at capturing board state information in both Othello and chess (see Table <ref>, Figure <ref>d and <ref>d). 
In contrast, SAEs trained on a model with random weights perform very poorly according to our metrics, showing that SAE performance is driven by identifying structure in the models' learned representation of game boards.
Nonetheless, SAEs do not match the performance of linear probes in terms of reconstructing the board state.
This performance gap suggests that SAEs do not capture all of the information encoded in the model's internal representations.



  
Standard SAEs trained with p-annealing perform on par with Gated SAEs.
We find that standard SAEs trained using p-annealing consistently perform better than those trained with a constant L_1 penalty (Equation <ref>), as measured by existing proxy metrics and in terms of improvement in coverage (see Figure <ref>a and <ref>a).
In fact, standard SAEs trained using p-annealing show a coverage score that is comparable to Gated SAEs trained without p-annealing. However, we find cases where our coverage metric disagrees with existing metrics. In Figure <ref>, for example, Gated SAEs perform achieve a higher loss recovered score than Standard SAEs trained using p-annealing. We emphasize that the training and inference of Gated SAEs is more computationally expensive, requiring 50% more compute per forward pass compared to Standard SAEs <cit.>. 













  
Coverage and board reconstruction reveal differences in SAE quality not captured by unsupervised metrics.
Our metrics reveal improvements in SAE performance that traditional proxy metrics fail to capture.
For example, we trained SAEs with hidden dimensions 4096 and 8192 (expansion factors of 8 and 16, respectively).  
We expect the SAEs with 8192 hidden dimensions to perform better since they have greater capacity.
However, we observe that they perform equally well according to prior unsupervised metrics (see Figures <ref> a, c and <ref> a, c). 
In contrast, our metrics reveal that SAEs with larger hidden dimensions are better. For the Standard architecture, this is reflected by the parallel lines (of purple diamonds) in Figures <ref> b, d and <ref> b, d.
Thus, our metrics are able to capture improvements from larger expansion factors.
In addition, we find that the performance of p-annealing closely resembles that of Gated SAEs when evaluated using standard proxy metrics; it demonstrates clear improvements under our proposed metrics.



  
Coverage and board reconstruction are consistent with existing metrics.
Figures <ref>, <ref>, and <ref> demonstrate that both coverage and board reconstruction metrics are optimal in the elbow region of the Pareto frontier. 
This region, where SAEs reconstruct internal activations efficiently with minimal features, also yielded the most coherent interpretations during our manual inspections. This provides precise, empirical validation to the common wisdom that SAEs in this region of the Pareto frontier are the best.










































§ LIMITATIONS


The proposed metrics for board reconstruction and coverage provide a more objective evaluation of SAE quality than previous subjective methods. 
Nevertheless, these metrics exhibit several limitations. 
Primarily, their applicability is confined to the chess and Othello domains, raising concerns about their generalizability to other domains or different models. 
Additionally, the set of BSPs that underpin these metrics is determined by researchers based on their domain knowledge. 
This approach may not encompass all pertinent features or strategic concepts, thus potentially overlooking essential aspects of model evaluation.
Developing comparable objective metrics for other domains, such as natural language processing, remains a significant challenge. 
Moreover, our current focus is on evaluating the quality of SAEs in terms of their ability to capture internal representations of the model. 
However, this does not directly address how these learned features could be utilized for downstream interpretability tasks.















§ RELATED WORK




  
Sparse dictionary learning. Since the nineties, dictionary learning <cit.>, sparse regression <cit.>, and later, sparse autoencoders <cit.> have been extensively studied in the machine learning and signal processing literature. The seminal work of <cit.> introduced the concept of sparse coding in neuroscience (see also <cit.>, building upon the earlier concept of sparse representations <cit.> and matching pursuit <cit.>. Subsequently, a series of works established the theoretical and algorithmic foundations of sparse dictionary learning <cit.>. Notably, <cit.> introduced LISTA, an unrolled version of ISTA <cit.> that learns the dictionary instead of having it fixed.

In parallel, autoencoders were introduced in machine learning to automatically learn data features and perform dimensionality reduction <cit.>. Inspired by sparse dictionary learning, sparse autoencoders <cit.> were proposed as an unsupervised learning model to build deep sparse hierarchical models of data, assuming a certain degree of sparsity in the hidden layer activations. Later, <cit.> generalized sparse autoencoders (SAEs) to convolutional SAEs. Although the theory for SAEs is less developed than that of dictionary learning with a fixed dictionary, some progress has been made in quantifying whether autoencoders can, indeed, do sparse coding, e.g., <cit.>.



  
Feature disentanglement using sparse autoencoders. 

The individual computational units of neural networks are often polysemantic, i.e., they respond to multiple seemingly unrelated inputs <cit.>.
<cit.> investigated this phenomenon and suggested that neural networks represent features in linear superposition, which allows them to represent more features than they have dimensions. 
Thus, in an internal representation of dimension n, a model can encode m ≫ n concepts as linear directions <cit.>, such that only a sparse subset of concepts are active across all inputs – a concept deeply related to the coherence of vectors <cit.> and to frame theory in general <cit.>. 

To identify these concepts, <cit.> used SAEs to perform dictionary learning on a one-layer transformer, identifying a large (overcomplete) basis of features. 
<cit.> applied SAEs to language models and demonstrated that dictionary features can be used to localize and edit model behavior. <cit.> proposed a scalable method to discover sparse feature circuits, as opposed to circuits consisting of polysemantic model components, and demonstrated that a human could change the generalization of a classifier by editing its feature circuit. Recently, <cit.> explored autoencoders for attention layer outputs. These works have benefited from a variety of open-source libraries for training SAEs for LLM interpretability <cit.>.












  
Improving and evaluating sparse autoencoders. 

Despite the success of SAEs at extracting human-interpretable features, they fail to perfectly reconstruct the activations <cit.>.
One challenge in the training of SAEs with an L_1 penalty is shrinkage (or 'feature suppression'); in addition to encouraging sparsity, an L_1 penalty encourages feature activations to be smaller than they would be otherwise. 
<cit.> approached this problem by fine-tuning the sparse autoencoder without a sparsity penalty. 
<cit.> and <cit.> explored alternative sparsity penalties to reduce feature suppression during training.
<cit.> introduced Gated SAEs, an architectural variation for the encoder which both addresses shrinkage and improves on the Pareto frontier of L_0 vs reconstruction error. Recently, <cit.> systematically evaluated the
scaling laws with respect to sparsity, autoencoder size, and language model size.


The goal of dictionary learning in machine learning is to produce human-interpretable features and capture the underlying model's computations <cit.>. 
However, quantitatively measuring interpretability is difficult and often involves manual inspection. 
Therefore, most existing work assesses the quality of SAEs along different proxy metrics: 
(1) The cross-entropy loss recovered, which reflects the degree to which the original loss of the language model can be recovered when replacing activations with the autoencoder predictions.
(2) The L_0-norm of feature activations 𝔼_z ∼𝒟 h(z) _0, measuring the number of activate features given an input <cit.>. <cit.> proposed to compare SAEs against supervised feature dictionaries in a natural language setting.
However, this requires a significant understanding of the model's internal computations and is thus not scalable.




§ CONCLUSION


Most SAE research has relied on proxy metrics such as loss recovered and L_0, or subjective manual evaluation of interpretability by examining top activations. However, proxy measures only serve as an estimate of interpretability, monosemantic nature, and comprehensiveness of the learned features, while manual evaluations depend on the researcher's domain knowledge and tend to be inconsistent.

Our work provides a new, more objective paradigm for evaluating the quality of an SAE methodology; coverage serves as a quantifiable measure of monosemanticity and quality of feature extraction, while board reconstruction serves as a quantifiable measure of the extent to which an SAE is exhaustively representing the information contained within the language model. Therefore, the optimal SAE methodology can be judged by whether it yields both high coverage and high board reconstruction.

Finally, we propose the p-annealing method, a modification to the SAE training paradigm that can be combined with other SAE methodologies and results in an improvement in both coverage and board reconstruction.



§ AUTHOR CONTRIBUTIONS

A.K. built and maintained our infrastructure for working with board-game models.
A.K., S.M., C.R., J.B., and L.S. designed the proposed metrics. 
B.W. performed initial experiments demonstrating the benefits of training SAEs with p < 1. B.W., C.M.V., and S.M. then proposed p-annealing, with B.W. leading the implementation and developing coefficient annealing.
The basic framework for our dictionary learning work was built and maintained by S.M. and C.R. The training algorithms studied were implemented by S.M., C.R., B.W., R.A., and J.B. 
R.A. trained the SAEs used in our experiments. 
A.K., C.R., and J.B. selected and implemented the BSPs.
A.K. and J.B. trained the linear probes. 
Many of the authors (including L.S., J.B., R.A.) did experiments applying traditional dictionary learning methods and exploring both toy problems and natural language settings, which helped build valuable intuition.
The manuscript was primarily drafted by A.K., B.W., C.R., R.A., J.B., C.M.V., and S.M., with extensive feedback and editing from all authors. 
D.B. suggested the original project idea.



§ ACKNOWLEDGMENTS

C.R. is supported by Manifund Regrants and AISST. 
L.R. is supported by the Long Term Future Fund.
S.M. is supported by an Open Philanthropy alignment grant.

The work reported here was performed in part by the University of Massachusetts Amherst Center for Data Science and the Center for Intelligent Information Retrieval, and in part using high performance computing equipment obtained under a grant from the Collaborative R&D Fund managed by the Massachusetts Technology Collaborative.


icml2024
















§ SPARSE AUTOENCODER TRAINING PARAMETERS


We used a single NVIDIA A100 GPU for training SAEs and experiments. It takes much less than 24 hours to train a single SAE on 300 million tokens. Given a trained SAE, our evaluation requires less than 5 minutes of computing time.





§ LIST OF BOARD STATE PROPERTIES


Table <ref> summarizes the high-level board state properties considered in 𝒢_strategy. The selection of concepts was inspired by <cit.>. The column indicated by # denotes the number of individual BSPs per concept. A single BSP per concept indicates we match this condition globally for any corresponding piece.





§ PERFORMANCE OF LINEAR PROBES AND SAES ON BOARD STATE PROPERTIES



In Figure <ref>, we present a mean coverage score over strategy board state properties 𝒢_BSP.
Properties within 𝒢_BSP vary significantly in complexity.
For example, queen detection can be inferred directly from the move history, while fork detection requires an accurate representation of the board state. 

Table <ref> shows that linear probe F1-score is below 0.95 for 6 out of 15 properties in 𝒢_BSP. This suggests that chess-GPT <cit.> does not represent these properties linearly. Additional experiments are required to determine whether the representation is present at all.

For the board state case, reconstruction is significantly higher than coverage. This is because there are many SAE features that are high precision classifiers for a configuration of squares, such as "white pawn on , white knight of ". In cases where coverage is higher than reconstruction (such as for ), it is because there are not many features that are over 95% precision for “there is a check move available” from which we can recover if there is an available check move. Coverage is significantly higher because there is at least one feature that has an F_1-score of 0.54 for , which may not have a precision greater than 95%.

Y>X


Y>X



 










§ MODEL INTERNAL BOARD STATE REPRESENTATION





 §.§ Othello Models

Previous research of Othello-playing language models found that the model learned a nonlinear model of the board state <cit.>. Further investigation found a closely related linear representation of the board when probing for "my color" vs. "opponent's color" rather than white vs. black <cit.>. Based on these findings, when measuring the state of the board in Othello, we represent squares as (, ) rather than (, ).




 §.§ Chess Models

Similar to Othello models, prior studies of chess-playing language models found the same property, where linear probes were only successful on the objective of the (, ) representation and were unsuccessful on the (, ) representation <cit.>. They measured board state at the location of every period in the Portable Game Notation (PGN) string, which indicates that it is white's turn to move and maintain the (, ) objective. Some characters in the PGN string contain little board state information as measured by linear probes, and there is not a clear ground truth board state part way through a move (e.g., the “” in “”). We follow these findings and measure the board state at every period in the PGN string.

When measuring chess piece locations, we do not measure pieces on their initial starting location, as this correlates with position in the PGN string. An SAE trained on residual stream activations after the first layer of the chess model (which contains very little board state information as measured by linear probes) obtains a board reconstruction F_1-score of 0.01 in this setting. If we also measure pieces on their initial starting location, the layer 1 SAE's F_1-score increases to 0.52, as the board can be mostly reconstructed in early game positions purely from the token's location in the PGN string. Masking the initial board state and blank squares decreases the F_1-score of the linear probe from 0.99 to 0.98.



