Paper Title: Vision-Language Model Based Handwriting Verification.










Vision-Language Model Based Handwriting Verification
    
    Mihir Chauhan††   Abhishek Satbhai§   Mohammad Abuzar Hashemi¶   Mir Basheer Ali¶¶   Bina Ramamurthy‡   Mingchen Gao||   Siwei Lyu†   Sargur Srihari‡‡
 
    
    Department of Computer Science and Engineering 
 
    The State University of New York, Buffalo, NY, USA 
 
    {mihirhem††, bina‡, mgao8||, siweilyu†, srihari‡‡}@buffalo.edu 

    {ma.hashemi.786¶, abhishek07satbhai§, alimirbasheer¶¶}@gmail.com

    
=================================================================================================================================================================================================================================================================================================================================================================================================================================

empty



§ ABSTRACT 
Handwriting Verification is a critical in document forensics. Deep learning based approaches often face skepticism from forensic document examiners due to their lack of explainability and reliance on extensive training data and handcrafted features. This paper explores using Vision Language Models (VLMs), such as OpenAI’s GPT-4o and Google's PaliGemma, to address these challenges. By leveraging their Visual Question Answering capabilities and 0-shot Chain-of-Thought (CoT) reasoning, our goal is to provide clear, human-understandable explanations for model decisions. Our experiments on the CEDAR handwriting dataset demonstrate that VLMs offer enhanced interpretability, reduce the need for large training datasets, and adapt better to diverse handwriting styles. However, results show that the CNN-based ResNet-18 architecture outperforms the 0-shot CoT prompt engineering approach with GPT-4o (Accuracy: 70%) and supervised fine-tuned PaliGemma (Accuracy: 71%), achieving an accuracy of 84% on the CEDAR AND dataset. These findings highlight the potential of VLMs in generating human-interpretable decisions while underscoring the need for further advancements to match the performance of specialized deep learning models. Our code is publicly available at: <https://github.com/Abhishek0057/vlm-hv>


Keywords: Vision Language Model, Machine Vision, Handwriting Verification, Forensics








§ INTRODUCTION


Handwriting verification is crucial in pattern recognition and biometrics, focusing on authenticating and identifying individuals by their handwriting. It is vital in forensics, where experts analyze samples to verify documents, identify forgeries, and provide court evidence.

Historical Perspective: Handwriting Verification methods <cit.> <cit.> <cit.> have evolved from handcrafted features like GSC <cit.> to deep CNNs such as ResNet-18 <cit.>, and now Vision Transformers <cit.>, enabling comparison of both inter- and intra- writer variations in handwriting styles. 

Challenges: Despite the advancements, Forensic Document Examiners (FDE) remain skeptical due to the lack of interpretability in the model's decision making processes. Morever, these methods heavily depend on large datasets of labeled handwritten images, making dataset collection (X(x_q,x_k, y) with known x_k, questioned x_q handwritten samples and corresponding writer labels y) is costly and time-intensive. 


Why use VLM? Vision-Language Models (VLMs) integrate image and textual data to encode complex relationships between visual and linguistic information, enabling Forensic Document Examiners (FDEs) to interpret model decisions with clear, natural language explanations that enhance trust and reliability in forensic applications. They can adapt to the forensics domain in zero-shot scenarios without training examples and in few-shot scenarios with minimal examples by leveraging transfer learning capabilities, having been pre-trained on multiple tasks related to handwritten images and natural language understanding. Fine-tuning them with forensic-specific datasets further enhances their performance and applicability in real-world forensic use-cases. 

VLMs applied to Forensics: In <cit.>, the authors explore the capabilities of multimodal Large Language Models (LLMs) like GPT-4 in various digital forensics use-cases, including artifact understanding, evidence searching, anomaly detection, and education. More specific studies, such as <cit.>, demonstrate the effectiveness of GPT-4 specifically for the media forensics task of DeepFake image detection. Our research applies VLMs to handwriting verification. To our knowledge, we are the first to explore VLMs for the forensic task of handwriting comparison.












§ METHODS

Our study aims to use VLM in handwriting verification. We have chosen OpenAI’s GPT-4o VLM for its strong Visual Question Answering (VQA) capabilities. Using their API, we prompt GPT-4o <cit.> with 0-shot Chain-of-Thought (CoT) reasoning. This approach allows us to first generate human-interpretable explanations and then determine whether the given questioned and known handwritten samples were written by the same person or by different writers, as shown in Figure <ref>. Since fine-tuning GPT-4o is not generally available, we compared its performance to a recent open-source VLM, PaliGemma <cit.>, using 0-shot prompt engineering and parameter-efficient supervised fine-tuning (PEFT) on a training dataset with 100 examples.  

Data: The experiments were conducted using 1,000 sample pairs of known and questioned images from CEDAR Letter <cit.> and CEDAR AND <cit.> dataset. CEDAR Letter dataset contains a letter manuscript written by 1568 writers three times. CEDAR AND dataset is a subset of CEDAR Letter data which only contains the lowercase handwritten word “and” extracted from full letter manuscript. The evaluation dataset includes 1,000 sample pairs of known and questioned images written by 368 writers with writer ids greater than 1200 (writer ids below 1200 were used to train the baseline models). Of these pairs, 500 are from the same writer and 500 are from different writers.






§ EXPERIMENTS & RESULTS

CEDAR AND Baselines: As shown in Figure <ref> we use handcrafted features GSC features, CNN based ResNet-18 <cit.> and MaskedCausalVisionTransformer <cit.> (ViT) as our baseline feature extractors. GSC are 512-dimensional features extracted for the binarized “AND” images. All the three baselines were trained on 10% and 100% of known and questioned training pairs writers with writer ids less than 1200 resulting in 13,232 and 129,602 number of train pairs as shown in Table <ref>. The output of these feature extractors is fed into 2 fully-connected (FC) layers with 256 and 128 hidden neurons with ReLU activations. The final layer has 2 output neurons whose softmax activations represent similarity of samples with a one hot vector representation. We use categorical cross entropy loss given one-hot encoded logits compared to the target which is binary. 



Prompt Engineering with GPT-4o and PaliGemma: To effectively utilize the GPT-4o VLM for handwriting verification, we experimented with various prompts to optimize both the generation of human-interpretable explanations and the accuracy of verification decisions. Initially, we crafted prompts that directed the model to compare specific features of the handwriting samples, such as stroke width, slant, and letter spacing. For example, prompts like "Describe the similarities and differences in the stroke patterns between the two samples" and "Identify any matching unique characteristics in the handwriting" were used. However, we observed significant variance in the model’s responses, leading to inconsistent results. To address this, we adopted a Chain-of-Thought (CoT) reasoning approach. This method guides the model through a structured reasoning process, allowing it to generate more consistent and reliable explanations. Since no examples were provided to VLM in the prompt engineering phase, the number of training pairs is observed as 0 in Table <ref>. 



Additionally, we utilized prompt engineered both VLMs to identify and mark coordinates of similarities and dissimilarities within the two images. By prompting the model to highlight specific areas of interest, such as matching or differing stroke patterns, we were able to generate detailed coordinates that pinpointed these features as shown in Figure <ref>. This approach provided clear visual indicators of the reasoning behind each verification decision. By incorporating CoT reasoning, presenting known and questioned samples as different images, and using VLM to identify key coordinates, we achieved a balance where the model could generate detailed, interpretable explanations while accurately determining the authenticity of the handwriting samples.

Supervised Fine-Tuning with PaliGemma: 
We fine-tuned the pre-trained PaliGemma model on 100 curated examples of 0-shot CoT prompt engineered PaliGemma results. The PaliGemmaProcessor within Transformers library was loaded to process the inputs which involved the preparation of prompt templates and batching text inputs with images. The image tokens and pad tokens were set to -100 to be ignored by the model, and these preprocessed inputs were used as labels for the model to learn from. The model, PaliGemmaForConditionalGeneration, was loaded and set to fine-tune only the text decoder, leaving the vision encoder and multimodal projector frozen. We used BitsAndBytes config options with 4bit quantization and LoRA <cit.> to train only 11M parameters out of the 3B parameters of the network. The results of supervised fine-tuned PaliGemma on both CEDAR AND and Letter datasets are shown in Table <ref>.

Results: The results of using VLMs for handwriting verification are presented in Table <ref> on the CEDAR AND and CEDAR Letter datasets. For the CEDAR AND dataset, CNN based ResNet-18 architecture outperforms 0-shot CoT prompt engineering approach with GPT-4o as well as supervised-finetuned PaliGemma with 100% training pairs which suggest that although VLMs has a lot of potential for generating human interpretable decision making for the task of handwriting verification, still lags behind CNNs which aare fine-tuned on specific task of handwriting verification. This gap highlights the need for further improvements in fine-tuning regime of VLMs to enhance their effectiveness and reliability for specialized tasks like handwriting verification. For the CEDAR Letter dataset, the 0-shot CoT prompt engineering with GPT-4o also shows relatively higher performance than the supervised fine-tuning of PaliGemma but we observe high variability in the performance metrics because of low sample size. 



§ CONCLUSION

Our study explores the application of VLMs, specifically GPT-4o and PaliGemma, in the domain of handwriting verification. By leveraging the robust VQA capabilities of these models and employing 0-shot CoT reasoning through prompt engineering, we aimed to generate human-interpretable explanations for model decisions. Our experiments demonstrated that while VLMs offer significant improvements in interpretability and adaptability to diverse handwriting styles, they currently lag behind CNN-based architectures such as ResNet-18 in terms of performance. Specifically, ResNet-18 achieved an accuracy of 84% on the CEDAR AND dataset, outperforming GPT-4o's 70% accuracy and PaliGemma's 71% accuracy. These findings suggest that while VLMs hold great promise for enhancing transparency and trustworthiness in forensic handwriting verification, there is still a need for further advancements in their fine-tuning regimes to improve their effectiveness and reliability for specialized tasks. Moving forward, we aim to work with forensic document examiners to create a fine-tuning dataset of explanation reports using text and visual information to ensure the applicability of our approach in practical forensic settings.






apalike





 Paper Title: Faithful and Plausible Natural Language Explanations for Image Classification: A Pipeline Approach.


Visual Analysis of GitHub Issues to Gain Insights

    Rifat Ara Proma
Scientific Computing and Imaging Institute 

University of Utah

Salt Lake City, UT, USA 

proma@sci.utah.edu
Paul Rosen
Scientific Computing and Imaging Institute 

University of Utah

Salt Lake City, UT, USA 

prosen@sci.utah.edu

    August 5, 2024
============================================================================================================================================================================================================================================================





§ ABSTRACT 
Existing explanation methods for image classification struggle to provide faithful and plausible explanations. 
This paper addresses this issue by proposing a post-hoc natural language explanation method that can be applied to any CNN-based classifier without altering its training process or affecting predictive performance. 
By analysing influential neurons and the corresponding activation maps, the method generates a faithful description of the classifier's decision process in the form of a structured meaning representation, which is then converted into text by a language model. 
Through this pipeline approach, the generated explanations are grounded in the neural network architecture, providing accurate insight into the classification process while remaining accessible to non-experts. 
Experimental results show that the NLEs constructed by our method are significantly more plausible and faithful. In particular, user interventions in the neural network structure (masking of neurons) are three times more effective than the baselines.




§ INTRODUCTION


Despite remarkable advances in computer vision, the deployment of image classification systems, especially in critical domains, poses significant challenges.
One of them is the opacity of deep models and the difficulty of providing reliable explanations for their predictions <cit.>. 

Therefore, several types of explanation methods have been proposed, including various forms of saliency maps <cit.>, feature importances <cit.>, concept-based explanations <cit.>, counterfactual explanations <cit.>, etc.
A particularly interesting form of explaining predictions is offered by natural language explanations (NLE) techniques <cit.>. 
Such explanations are not only 
understandable by non-expert users, but can also be used to support conversations with the user in a dialogue system <cit.>.


















There are two critical properties of an explanation: faithfulness and plausibility <cit.>. 
A faithful explanation should accurately reflect the inner workings of the
system and provide information on the real reasons why the model reached a certain decision. 
Plausibility then refers to how convincing the explanation appears to the user.


In the case of NLE, obtaining high plausibility is straightforward, as textual explanations are usually human-friendly <cit.>, but achieving high fidelity is challenging. 
In the context of image classification, image captioning methods offer plausible but unfaithful NLEs <cit.>.

Some methods try to improve faithfulness by conditioning generation on both the predicted class and image features <cit.>, but the faithfulness provided is still limited as the model is not aware of the classifier's decision process.

Other methods train image classifiers to jointly predict the class and visual rationales, and generate explanations based on them <cit.>. However, the rationales are predicted independently of the class and do not participate in the classifier's decision process.

Most importantly, such methods change the training procedure and the architecture of the classifier, often affecting classification performance.



In this paper, we propose a post-hoc natural language explanation method for image classification that can be used with any standard convolutional neural network  (CNN) classifier. 
To illustrate the classification decision process, the method analyses which neurons of the CNN were most influential in reaching a given decision, and which regions of the image caused them to activate.
For each influential neuron, a neuron annotation method computes visual pattern exemplars and produces a short (one phrase) description of a pattern that the neuron detects.
The information gathered in this process serves to create a simple, human-readable meaning representation, which is converted to text by a large language model (LLM).


Our simple two-step pipeline method (see Fig. <ref>) produces explanations that are directly grounded in the classifier's network architecture, but without interfering with its training process or affecting its predictive performance. 
It also does not need any gold-standard explanations.
The provided NLEs reflect the underlying classification process by specifying the most influential neurons computed by a well-established explainable AI methods.
At the same time, the final text is fluent and easy to understand.
This results in NLEs that are both plausible and highly faithful.

Our experimental code is publicly available.[<http://tinyurl.com/3da45hnm>]




§ METHOD

Our NLE generation approach uses two processing steps, described below: meaning representation (MR) construction and MR-to-text conversion.





 §.§ Meaning representation construction

We first produce a meaning representation  
in the form of a JSON object, containing information about the neurons responsible for a given classifier prediction (why?), what patterns those neurons detected (what?), and in which parts of the image they were activated (where?).
The MR includes the predicted class and the list of most influential neurons, each represented by



(1) description – a phrase describing the pattern that most excites the neuron (convolutional filter), (2) positions – list of coarse-grained image positions (e.g., “bottom-right corner”) where the neuron was activated. 
An example of a MR is provided in Fig. <ref>.









MR construction starts by storing all neuron activations from the given CNN-based classifier for the prediction to explain.
Next, the most important neurons are selected, annotated with a description, and tied to an image region, as follows:







( predicted_class , [
  "description": "neuron_description", "positions" : [neuron_positions],
   "description": ..., ... 
   ])





  
Selecting the most influential neurons
To choose the most relevant neurons, we apply well-established Layer-wise Relevance Propagation (LRP) method <cit.>. 
LRP performs a backward pass through the classifier network to establish the influence of each neuron to the final prediction (see App. <ref> for formulas).


We select k neurons with the highest LRP 
scores (with k being a parameter controlling the brevity-detail tradeoff). 

 


  
Neuron annotation
We adopt the MILAN neuron annotator <cit.> to generate descriptions of selected neurons.
MILAN first finds images in the classifier training set 
that make a given neuron highly activated 
<cit.>.
These exemplar images 
are used to generate a description of the pattern that this neuron detects.


Note that although the last step of MILAN is essentially image captioning, it does not affect the faithfulness of NLEs produced by the pipeline, as long as its output is of sufficient quality. 
The images that illustrate a neuron's decision process are computed by analysing its activations, and the captioning is only used to convert the result into text. 




  
Establishing image regions for neuron activation

The neuron's raw activation map
is divided into a 3x3 grid 

with manually assigned labels such as `top-left corner', `top', `top-right corner', etc. 

We then select
all grid cells where the neuron's activation exceeded half of its maximum value.


We apply several substitution rules (see App. <ref>) to make the list of cells shorter and more human-readable. 



 §.§ Explanation generation

The second step of our method is converting the human-readable and faithful MR created above

into a user-friendly text.
As we do not have any gold-standard explanation texts
, the task is performed by prompting a large language model (LLM). 

We instructed the model to (1) produce fluent text, (2) summarise the content of the MR (e.g. if two neurons detect similar patterns, they can be combined in the text), (3) prioritise readability, (4) come up with its own formulation of spatial positions to improve fluency. We  also provide one handcrafted MR-to-text conversion example. 

The 
prompt is shown in App. <ref>.
LLMs could in theory hallucinate and thus reduce the explanations' faithfulness.
However, in Section <ref> we show experimentally that current LLMs are reliable enough to produce useful explanations. 






§ EXPERIMENTAL EVALUATION







 §.§ Experimental setup




  
Dataset All experiments were performed on the ImageNet dataset.
The classifier was trained on the train set and our explanation method was run to explain predictions made on the validation data.[Annotated ImageNet test set is not publicly available.]



  
Models 
We experiment with explaining the predictions of the smallest CNN classifier from the popular ResNet family: ResNet18 <cit.>.[
It reaches only a 41.4% accuracy on the validation set, but high classification performance is not the goal of our study.] 
We fill our MRs with k=10 top neurons indicated by LRP from the Captum library <cit.> and annotate them using MILAN's original implementation. 

As the LLM for MR-to-text conversion, we use GPT-4 <cit.>.



  
Baselines
We compare to the following methods: 


    
  * Show, attend and tell (SAT) by <cit.> is an  image captioning method 
    used for explaining predictions <cit.>. 

  * NLX-GPT <cit.> is an explainable visual question-answering method that produces NLEs with an encoder-decoder architecture that combines CNN with a transformer-based language model.






 §.§ Are the output explanations plausible?



To assess the plausibility 
of generated explanations,
we conducted a small-scale manual annotation experiment.
We recruited ten annotators: five non-experts hired on the Prolific platform and five experts with at least one published paper on explainable AI. 
Each annotator was presented with 30 image-explanation pairs (300 in total) and asked to rate them on a scale of 1-5 
whether the explanation was (1) fluent, (2) easy to understand (comprehensible), (3) convincing, and (4) explanatory for the underlying decision process.[Note that the question on understanding the decision process does not measure faithfulness, but the user's subjective opinion on whether they understand how the model works.] The overall quality of the explanations was also rated (see App. <ref>).

The results are presented in Tab. <ref>.
Our method obtains the highest overall quality according to both experts and non-experts.
It also produces the most plausible explanations (most convincing and explanatory). 
Since the baselines produce much shorter explanations, 
it is not surprising that our longer explanations are a bit more difficult to understand. 

Interestingly, experts generally give higher ratings than non-experts.
This is true for all methods and all factors except the explanatory ability. 
For this factor, experts rate the baselines lower than non-experts, but they consistently rate the explanations provided by our pipeline higher.
The improvements of our method over baselines are statistically significant on both plausibility measures and overall quality. For fluency, 

our method is indistinguishable from SAT (see details in App. <ref>).



 §.§ Are the output explanations faithful?






The faithfulness of the generated explanations is assessed through two intervention experiments: (1) checking if rationales from NLEs change the prediction by masking parts of input images, (2) influencing the network prediction by masking influential neurons.
We further assess the stability and diversity of the explanations
for our method, and we directly evaluate the reliability of our MR-to-text conversion.







































  
Masking input image
We asked 
annotators to cover with white rectangles parts of images that contained the decision rationale indicated in the NLE, 50% area at most (see App. <ref> for details).
We re-classified covered images and measured changes in prediction and the average decrease in the probability of the originally predicted class.
We also performed an opposite experiment, with the annotators highlighting only parts of image mentioned in the explanation and covering the rest.


For covering, the use of our NLEs resulted in the highest average probability decrease and the change of the original prediction for 88% of examples (see Table <ref>). 
Our method reached the best results in the highlighting experiment as well, producing the least amount of changes.

We also re-ran our NLE pipeline with parts of the input image covered. This led to significant changes: on average 78% (median 90%) of the neurons indicated in MRs were different.



  
Masking influential neurons
To show the NLEs' ability to reflect classifier decisions, we asked the annotators to read the NLEs and select up to five most influential neurons from a MILAN-annotated list. 
The classifier was then re-run with the selected neurons masked. The results 
in Table <ref> reveal that masking neurons suggested by our NLEs led to a five times higher decrease in the predicted class probability and over three times higher class flip rate than baselines. More details are in App. <ref>.



  
Explanation stability analysis
Following <cit.>,
we measure explanation robustness against adding random noise to the input image (intra-set stability, see  App. <ref> for details) by comparing BLEU <cit.>, METEOR <cit.> as well as class flip frequency and class probability change against the original predictions.
We also check for outputs' diversity (inter-set stability) using BLEU  and METEOR  overlap  against explanations for other classes.
The results in Table <ref> show high diversity and high noise sensitivity:
In parallel to increased classification changes with added noise,
explanation BLEU/METEOR gradually drop, but they are lowest when comparing between different classes.




  
Reliability of the MR-to-text transform

The human evaluation of our approach's MR-to-text reliability was similar to plausibility evaluation, but limited to non-expert Prolific annotators.
We asked five yes-no questions on information in the text not grounded in the MR (i.e., hallucinations), omission of MR information, fluency, spatial information fidelity,
and overall correctness (see App. <ref>).



The results show the MR-to-text conversion as highly reliable, as only 8% texts contain hallucinations. Omissions are more frequent (44%), but this is expected as the LLM is instructed to summarise the MR and prioritise readability. 
This factor most likely affected the overall score (58%). 
The explanations are mostly fluent (96%), with correct spatial information (82%).





§ ACKNOWLEDGEMENTS

Co-funded by the European Union (ERC, NG-NLG, 101039303) and National Science Centre, Poland (Grant No. 2022/47/D/ST6/01770).
This work used resources of the LINDAT/CLARIAH-CZ Research Infrastructure (Czech Ministry of Education, Youth, and Sports project No. LM2018101).




§ LIMITATIONS


This paper produces a new method for plausible and more faithful natural language explanations for image classification.
Although we believe that the method provides significantly better  faithfulness than the previously proposed methods, it does not obtain completely faithful explanations. 
The faithfulness of the explanations provided by our method depends on the quality of the neuron annotations produced by MILAN and the neurons indicated by LRP. Both techniques can be considered as state of the art, but they still occasionally produce incorrect results.
Therefore, the results of NLE methods should be treated with caution.
Additionally, this work uses pre-trained language models, which are known to expose certain social biases reflected in their training data.







§ EXAMPLES OF EXPLANATIONS

Several examples of explanations provided by the methods under study are given in Tab. <ref>.
More examples of NLE produced with the proposed pipeline approach are given in Tab. <ref>.







§ DETAILED RESULTS OF MASKING INFLUENTIAL NEURONS EXPERIMENT


The detailed results of the experiment with masking influential neurons are presented in Fig. <ref>.
For our method, the results indicate that masking neurons in the order indicated by the annotators leads to an increasing change in the classifier's prediction and a larger decrease in the predicted class probability value.
In contrast, the annotators gain limited insight into how the neural network made a decision from the explanations provided by other methods: masking the indicated neurons leads to limited change, and increasing the number of masked neurons leads to almost no improvement.
We attribute the effectiveness of masking the first indicated neuron to examples of classes that are highly correlated with a pattern detected by a particular neuron.





§ SIMPLIFICATION RULES FOR SPACIAL POSITIONS


The spatial information about neuron activation is learned by inspecting activation maps of the chosen neurons. The activation map is converted to a binary image, such that if a given pixel of an activation map exceeds half of a maximal value in that activation map it is converted to the value of one, and the rest of the pixels are assigned with the value zero. Then the binary activation map is divided into a 3x3 grid of congruent squares oriented such that the ordering begins with a 0 in the top-left corner, progressing sequentially across each row from left to right and top to bottom, culminating with the number 8 in the bottom-right position.  The basic positions names are given below.
captionpos=b

[label=lst:basic_positions]
0 : "top-left corner", 
1 : "top", 
2 : "top-right corner", 
3 : "left", 
4 : "center", 
5 : "right", 
6 : "bottom-left corner", 
7 : "bottom", 
8 : "bottom-right corner"


The final appearance of the positions appended to the meaning representation is governed by a set of rules, where position names are assigned to sets of basic positions.
 If all basic positions from the set are present in the MR, we replace them with a corresponding compound position.  These sets and position names are given below.

[label=lst:compound_positions,breaklines=true]
"entire top": 0, 1, 2,
"entire bottom": 6, 7, 8,
"entire left": 0, 3, 6,
"entire right": 2, 5, 8,
"perimeter": 0, 1, 2, 5, 8, 7, 6, 3,
"center cross": 1, 3, 4, 5, 7,
"upper half": 0, 1, 2, 3, 4, 5,
"lower half": 3, 4, 5, 6, 7, 8,
"left half": 0, 1, 3, 4, 6, 7,
"right half": 1, 2, 4, 5, 7, 8


Additionally if "entire *" and "* half" coexist in the meaning representation simultaneously, where "*" represents one of {"left", "right", "top", "bottom"}, in both compound positions names, the "entire *" is deleted from the meaning representation, since it is comprised of the subset of the set, which makes up corresponding "* half". An example would be if "entire top" and "upper half" are both in meaning representation, the "entire top" would be deleted since it is comprised of {0, 1, 2} tiles, which also partially make up the "upper half" ({0, 1, 2, 3, 4, 5}). Finally, if more than any, distinct, 6 out of 9 tiles are active, every position of a given neuron is deleted and the spatial information placeholder is set to "entire image". This whole approach significantly reduces the length of the positional representation, while simultaneously making the position names more plausible to the system user, since we believe the final space of possible positions is natural and intuitive for humans to grasp quickly and effectively.




§ PROMPT FOR MR-TO-TEXT CONVERSION


To convert meaning representations into text, the following prompt was applied to the language model:

You are given a problem of creating textual explanation of an image classification performed by neural network. You will be given a Python object representing network output in the form of `'image class', [('detected object', 'position'), ...]`. I want you to convert this object into a textual explanation. You should:
1. Create a grammatically correct sentence which will explain the model's decision.
2. Decide which detected objects do not fit with image class and do not include them in the explanation. For instance, 'dentist' class and 'animal heads' objects are completely unrelated. However, the descriptions that aren't directly related to image class, but can be indirectly correlated, especially in terms of shape, color, or texture resemblance should be included (like 'fountain' and 'sea' because of the water they have in common or 'brick wall' and 'grid' because the texture is similar). Never mention that you chose neuron descriptions and do not talk about the neuron descriptions that were discarded.
3. Prioritize the readability of the explanation. Include only essential detected objects and aggregate information to shorten the explanation.
4. Aggregate positions if possible, for example ['bottom-left corner', 'bottom', 'bottom-right corner'] should be aggregated into 'bottom'. If the positions list is too long or too ambiguous do not include them in the explanation. 

Here is an example.
Python object:
“lakeside, ['description': 'Nature', 'positions': ['left', 'right', 'bottom'], 'description': 'The sky', 'positions': ['top-right corner', 'bottom-left corner', 'bottom'], 'description': 'Red and white colored objects', 'positions': ['left', 'right', 'bottom'], 'description': 'The ocean', 'positions': ['left', 'right', 'bottom'], 'description': 'Animal heads', 'positions': [], 'id', 'description': 'The color red', 'positions': ['left', 'right', 'bottom']', 'description': 'White backgrounds', 'positions': ['bottom', 'left'], 'description': 'Grass', 'positions': ['left', 'bottom-left corner', 'bottom', 'bottom-right corner'], 'description': 'Dogs and guinea pig', 'positions': ['center'], 'description': 'The color green', 'positions': ['top-right corner', 'right', 'bottom-right corner']]”
Answer: “The model assigned this image to the "lakeside" class because in the last layer it discovered nature, and the ocean at the left, right, and bottom of the image. It also detected grass at the left, bottom-left corner, bottom, and bottom-right corner and the color green at the right of the image.
(...)




§ HUMAN EVALUATION OF EXPLANATION PLAUSIBILITY


The annotators are presented with an image, model's prediction and a natural language explanation. 
Each question is answered on a scale from 1 (low) to 5 (high).
The following binary questions are asked:

    
  * How fluent (linguistically correct) the text is?	
    
  * How easy to understand the text is?	
    
  * How convincing do you find the explanation of the decision made by the model?	
    
  * After reading the explanation, how well do you understand how the decision of the model was taken?	
    
  * How would you rate the overall quality of the explanation?


The annotation instructions are provided in the code repository.



§ HUMAN EVALUATION OF MR-TO-TEXT TRANSFORMATION


The annotators are presented with a meaning representation in the form of formatted JSON without a given image, since it should not influence the assessment of MR-to-text transformation. The following binary questions are asked:

    
  * Does the text contain information that was not present in the meaning representation?	
    
  * Is there any important information from meaning representation omitted in the text? 
    
  * Is the text linguistically correct?	
    
  * If any spatial compression occurred between explained neurons, the said compression is correct?	
    
  * Overall, do you find this meaning representation to text transformation acceptable i.e. sufficiently good for explanation purposes?


The annotation instructions are provided in the code repository.




§ DETAILS ON STABILITY EXPERIMENTS


Let us assume that an image is a matrix X, such that X ∈ [0, 1]. We model input perturbations by adding random noise to the images from the standard normal distribution. 
Since an interval of possible pixel values  is [0, 1], to account for the unboundedness of a standard normal distribution we use a clipping operation defined as follows:

    clip(x) = 
    
    0,    if  x < 0 
    
    x,    if  0 ≤ x ≤ 1 
    
    1,    if  x > 1

The formula for the function ζ that adds a perturbation to the image is given below.

    ζ(X)=clip(𝒩(0, 1) · i + X)

where i is the noise intensity.
We perform two types of stability experiments:

    
  * inter-set stability: We compare pairs of unaltered images (1^st set of images) and corresponding images mapped by the ζ function (2^nd set of images). After the ζ mapping is performed, we run the proposed method on both images, yielding two explanations. We then compute various language similarity metrics between the two explanations. 
    
    We compare two kinds of pairs of images: unaltered - lightly perturbed (i=0.05)  and unaltered - heavily perturbed (i=0.2). 
    
  * intra-set stability: We compare explanations pairs produced for different images to verify the diversity of generated explanations. 

The computations were conducted on explanations produced by the proposed method for a 500-element subset of validation ImageNet data.
The subset was constructed by randomly picking 50 examples from 10 selected, diverse classes (library, over skirt, palace, prison, wall clock, lakeside, coral reef, volcano, fountain, basset).






§ DETAILS ON THE COVERING EXPERIMENT


Given an image and explanation, we instructed the annotator to cover the decision rationales with white rectangles.
The annotator instruction is given below:

    Based on the following explanation of the classifier's prediction, cover the reasons for its decision with white rectangles. You can use as many rectangles as you like, but the total area of the covered image cannot be larger than half of the image. If it is not possible to cover the mentioned reasons by converting only 50% of the image, please do your best to cover the most important information. 

An example of annotation provided is given in Fig <ref>.



§ LAYER-WISE RELEVANCE PROPAGATION 


To choose the most relevant neurons, the well-established Layer-wise Relevance Propagation (LRP) method is applied <cit.>. LRP performs a special backward pass through the neural network  
to establish the influence of each neuron to the final prediction.

Starting with the predicted value, LRP distributes it among the neurons in each layer, assigning them relevance scores.
The following rule for relevance reallocation is used: 

    R_i = ∑_j z_ij/∑_i z_ij R_j

where R_i is the i-th neuron relevance score, 
z_ij express how much i-th neuron has contributed to make j-th neuron relevant (calculated as the product of the neuron's activation and the corresponding weight), the sums ∑_i (∑_j) iterate over all neurons in a given (next) layer.



  
Choice of LRP as an explanation method
Although we present a pipeline approach and there is some variability in how it can be implemented, we believe there are important reasons for using our pipeline with LRP.

First, unlike many other methods, LRP works at the neuron level, which is strictly required by our method. Therefore, methods that provide pixel-level importance scores such as RISE <cit.> or Grad-CAM <cit.>, methods that typically work on image segments such as LIME <cit.> or SHAP <cit.> are not suitable for our approach.
 
Second, LRP has been shown to achieve high faithfulness in many studies and can be considered state of the art in this respect. Note that achieving high faithfulness is the main goal of our approach.

Finally, LRP is theoretically motivated, has been shown to be useful in many applications, and has stable open source implementations. 



§ STATISTICAL ANALYSIS OF THE HUMAN EVALUATION RESULTS


For the results of human evaluation of plausibility, we performed the non-parametric global Friedman test followed by Nemenyi post-hoc analysis (as recommended in <cit.>). 
We were able to reject the null hypothesis of the Friedman test for all the measures with p<0.001. The Nemenyi post-hoc analysis with α=5% confirmed that our method obtains statistically significant improvements over other compared methods on both plausibility measures and the overall quality measure. On the fluency measure, our method is undistinguishable from SAT.

The critical distance plots from Nemenyi post-hoc analysis are provided in Figure <ref>. The lower result, the better. If the difference between the methods is not statistically significant, their results are connected with a thick horizontal line. More details on these plots can be found in <cit.>.









 Paper Title: GEGA: Graph Convolutional Networks and Evidence Retrieval Guided Attention for Enhanced Document-level Relation Extraction.


The impact of quantum noise on limit cycles and entrainment in an atom-cavity system
    Jayson G. Cosme
    August 5, 2024
====================================================================================

§ ABSTRACT 
Document-level relation extraction (DocRE) aims to extract relations between entities from unstructured document text. Compared to sentence-level relation extraction, it requires more complex semantic understanding from a broader text context. Currently, some studies are utilizing logical rules within evidence sentences to enhance the performance of DocRE. However, in the data without provided evidence sentences, researchers often obtain a list of evidence sentences for the entire document through evidence retrieval (ER). Therefore, DocRE suffers from two challenges: firstly, the relevance between evidence and entity pairs is weak; secondly, there is insufficient extraction of complex cross-relations between long-distance multi-entities. To overcome these challenges, we propose GEGA, a novel model for DocRE. The model leverages graph neural networks to construct multiple weight matrices, guiding attention allocation to evidence sentences. It also employs multi-scale representation aggregation to enhance ER. Subsequently, we integrate the most efficient evidence information to implement both fully supervised and weakly supervised training processes for the model. We evaluate the GEGA model on three widely used benchmark datasets: DocRED, Re-DocRED, and Revisit-DocRED. The experimental results indicate that our model has achieved comprehensive improvements compared to the existing SOTA model.














§ INTRODUCTION





Relation extraction (RE) is a crucial technology used to automatically identify and classify semantic relations between entities in natural language texts. Existing relation extraction tasks can be divided into two types: sentence-level relation extraction and document-level relation extraction (DocRE) <cit.>. In sentence-level relation extraction datasets, each data entry contains only one sentence, and there is a single entity pair within the sentence for which the relation needs to be predicted. In contrast, DocRE datasets contain multiple sentences per data entry, corresponding to multiple entity pairs whose relations need to be predicted. As depicted in Figure <ref>, each entity pair may appear multiple times within the document and may have different relation types <cit.>, necessitating the analysis of a larger contextual scope to determine the relation for each entity pair.



Furthermore, assuming there are n entities in a data entry, predicting the relation for an entity pair involves pairing the anchor entity with the remaining n-1 entities one by one. This approach results in significant unnecessary memory overhead, as most pairs of entities do not have any relation between them.


Existing methods can be mainly divided into three categories <cit.> : sequence-based, graph-based, and transformer-based. Sequence-based models commonly employ pre-trained language models to produce word embeddings and character embeddings, transforming sequences of words or characters within texts into vector representations for processing <cit.>. Models based on dependency graphs utilize dependency information to construct document-level graph <cit.>, which are then processed through graph neural networks for inference. Transformer-based models utilize the self-attention mechanism to refine the representation of each word by assessing its contextual relations with all other words in the text <cit.>.



The aforementioned three types of relation extraction methods suffer from two limitations. First, relation extraction between entity pairs generally requires only a set of sentences as supporting evidence, without the need to focus on redundant irrelevant information. However, these methods utilize all the information in the long text for relation extraction. Therefore, <cit.> proposed an evidence retrieval-based relation extraction method. However, this method retrieves a list of evidence information for the entire document, resulting in poor relevance between this information and the relational entity pairs. Second, in DocRE, multiple entities are discretely distributed across different sentences or even paragraphs. Extracting their relations requires fully learning and understanding the semantics of the long text at the document-level. Existing methods rely on dependency parsing to construct multidimensional graph structures for semantic understanding and relation reasoning.However, they perform poorly when faced with complex intersecting relations due to the large number of entity pairs involved.



To address the aforementioned two issues and the insufficient annotation of evidence sentences in the dataset <cit.>, we propose a novel model for DocRE: GEGA. This model is trained under both fully supervised and weakly supervised settings. First, we utilize a complex model (Teacher) trained with full supervision to infer over distant supervision data, extracting evidence sentences and assigning token weights. Then, we use this weight information as supervisory signals to guide the training of a simplified model (Student). Finally, we fine-tune the student model to adapt it to specific tasks and datasets, thereby improving performance. In summary, this article has two contributions:



(1) We propose a novel DocRE model, GEGA[The implementation code for GEGA can be obtained from the GitHub link: [Links covered up in blind reviewing]] (Graph Convolutional Networks and Evidence Retrieval  Guided Attention). This model combines graph structures and Transformers to retrieve evidence sentences highly relevant to the relational entity pairs from the document, guiding the attention to assign higher weights to this evidence information, thereby enhancing the performance of relation extraction.



(2) Experiments conducted on the three public datasets DocRED, Re-DocRED and Revisit-DocRED show that GEGA can achieve the new SOTA[We outperform all other methods using BERT as the pretraining language model on the DocRED leaderboard. Please refer to GEGA's submission: <https://codalab.lisn.upsaclay.fr/competitions/365#results>] results on document-level relation extraction compared to existing methods under the same experimental settings.



§ PRELIMINARY



 §.§ Task formulation for DocRE and DocER


Let's assume we have a document D containing n entities e={e_1, e_2, …, e_n}. Each entity e_i in the document has a corresponding position p_i, and there may exist relations between entities. Our objective is to extract a set of relations R={r_1, r_2, …, r_m} from document D, where each relation r_i can be represented by a triple (e_i, e_j, r_i j), with r_i j being the relation label between entities e_i and e_j.
Therefore, the task of DocRE can be formulated as follows:
R={(e_i, e_j, r_i j) | e_i, e_j ∈ E, i ≠ j},
 r_i j is the relation label predicted by the relation classifier based on the contextual information of entity pairs (e_i, e_j).
 
 In addition, Document-level Evidence Retrieval (DocER) aims to retrieve a list of evidence sentences evi_[0,1,…,n] from document D to enhance relation extraction. Nowadays, researchers have extended relation triples (e_i, e_j, r_ij) by adding evidence sentence lists, resulting in relation quadruplets (e_i, e_j, r_ij, evi_[0,1,…,n]). Relations between entity pairs can be predicted solely using the sentences from the evidence lists, without relying on the entire document.





§ RELATED WORK


Our work is built upon a substantial body of recent work on document-level RE and ER.


DOC-Relation Extraction (RE). Previous studies can be divided into three major categories <cit.>:

Sequence-based methods. 

<cit.>, and <cit.> use methods such as Conditional Random Fields (CRF) or Recurrent Neural Networks (RNN, <cit.>) to accurately identify and label entities in text and predict the relations between these entities by learning the contextual sequential semantic information of each word. For example, <cit.> use different neural network architectures to perform sequence encoding of the entire document to learn the semantic representation of entity pairs and extract the relations between them. <cit.> employ BiLSTM to simultaneously consider the forward and backward information in the text sequence, learning and understanding the different semantic paths from one entity to another, thus inferring the relations between entity pairs.




Graph-based methods. 
<cit.> model entities and relations in documents as graph structures, then use Graph Neural Networks (GNNs) to learn the relations between entities. This approach effectively leverages structural information and global context among entities. Additionally, researchers have proposed hard pruning and soft pruning strategies for dependency tree structures to optimize the model's speed and storage efficiency. <cit.> and <cit.> use hard pruning strategies to retain words near the shortest path between two entities, maximizing the removal of irrelevant content while integrating relevant information. <cit.> proposed a soft pruning method that directly takes the full dependency tree as input and automatically learns how to selectively focus on relevant substructures that are useful for the relation extraction task. Subsequently, <cit.> proposed refined strategies to enhance cross-sentence relation reasoning by automatically inducing latent document-level graphs. This strategy allowing the model to incrementally aggregate relevant information for both local and global reasoning.


Transformer-based methods. This approach does not use any graph structures but instead adapts to the document-level relation extraction task by fine-tuning pre-trained models <cit.>. <cit.> introduced a copy-based training objective into the basic pre-trained language model, enabling the model to better capture coreference information. <cit.> employed a hierarchical aggregation method to obtain reasoning information at different granularities at the document-level. <cit.> addressed the multi-label and multi-entity issues in relation extraction datasets through adaptive thresholds and local context pooling.



DOC-Evidence Retrieval (ER). Currently, a few studies have investigated the importance of evidence information in document-level relation extraction tasks. <cit.> directly incorporated evidence sentence instances supporting entity relations into a new dataset. However, in the absence of evidence sentences, evidence information needs to be generated through an Evidence Retrieval (ER) task. <cit.> employed heuristic rules to select informative sets of paths from the entire document to discover evidence sentences and further optimized relation extraction by combining BiLSTM. <cit.> integrated evidence information into a Transformer-based DocRE system by directly guiding attention, without introducing any additional trainable parameters for the ER task. Compared to our work, they did not incorporate graph neural networks for end-to-end learning to derive the advantages of attention weights. In contrast, GEGA provides a more reliable allocation of weights for evidence information by constructing a fully connected graph and its corresponding fully connected matrix to learn structured information.




§ METHODOLOGY




This section elucidates the main framework of the proposed method, illustrated in Figure <ref>, the model can be segmented into tripartite: Input Encoder Layer, GEGA Module and Classification Layer.




 §.§ Input Encoder Layer
This work adheres to the methodology employed in prior research to incorporate specialized markers [CLS] and [SEP] at the begin and end of a designated document doc=[sent_N]_N=1^L for the purpose of outlining the document's boundaries, where sent=[t_n]_n=1^l. Subsequently, input the document into the pre-trained BERT model. When the input length exceeds 512, the document is divided into two overlapping segments[The length of each data in the publicly available datasets is less than 1024.]. The first segment has a length of 512 and the second segment comprises the difference between the total input length and 512. Ultimately, the contextual embedding representation H and attention matrix A of the token are derived:



    (H,A)   =[(h_1,a_1), (h_2,a_2), …, (h_l× L,a_l× L)] 
       =BERT([doc])

where l is the length of the sentence (i.e., the number of tokens), and L is the length of the input document (i.e., the number of sentences).

For each entity, the efficacy of the max pooling function is pronounced when the inter-entity relations are explicitly articulated. Nevertheless, in the context of this study, the relations among entities remain ambiguous. It is understood that an entity may be referenced by one or several mentions, and a mention might uniquely identify an entity or fail to ascertain a definite corresponding entity. This necessitates the calculation of an entity's embedding based on the embeddings of each associated mention. Following methodology of <cit.>, a soft version of the max function LogSumExp (LSE), is utilized to compute the embeddings of entities while concurrently capturing the attention distribution:






    0.87e_emb=LSE(x_1, …, x_n)=log∑_i=1^|ℳ_e|exp(h_i ∈ e)




    0.95e_att=1/|ℳ_e|∑_i ∈ ea_i
where e is an entity comprising multiple mentions, while a is the distribution of attention for mention i within entity e. |ℳ_e| is the number of mentions for entity e.



 §.§ GEGA Module

The GEGA module comprehends four parts: the Attention Concentration Layer, the Multi-GraphConv Layer, the Transformer-enc Layer, and the Collaborative Prediction Layer.



  §.§.§ Attention Concentration Layer



We employ Attention Concentration Layer to transform the initial dependency tree into a fully connected weighted graph based on the dependency relations within the sentence. This approach can be construed as a soft pruning strategy <cit.> juxtaposed with the conventional hard pruning strategy <cit.>. By assigning weights to the sequence data, as opposed to outright deletion, a greater amount of contextual information can be preserved, thereby fostering enhancements in module efficacy. 
Subsequently, we utilize the multi-head attention mechanism, wherein the input vector is mapped to several heads using a linear transformation layer to produce an adjacency matrix with varied weight distributions, denoted as Ã^(𝐡𝐞𝐚𝐝_𝐢)=Attention(Q W_i^Q, K W_i^K). We employ parallel computing to expedite the computational process.



    0.96Ã^(𝐡𝐞𝐚𝐝_𝐢)=softmax(Q W_i^Q ×(K W_i^K)^T/√(d))
where Q,K ∈ℝ^N × d_model, N is the length of the sequence, d_model is the dimensionality of the input feature, and W_i^Q, W_i^K∈ℝ^d_model× d_k
 is the weight parameter associated with the linear transformation.





  §.§.§ Multi-GraphConv Layer



The Graph Convolutional Networks (GCNs <cit.>) is a deep learning framework tailored for the processing of graph-structured data. It is a semi-supervised learning method based on graph structure that aggregates and propagates node features in the graph structure, thus deriving node representations. The Multi-GraphConv (M-G) Layer is a densely connected graph structure data processing module that is constructed based on GCNs. Illustrated in Figure <ref>, the n-th sublayer's output within the Multi-GraphConv (M-G) Layer serves as the subsequent l-n sublayers' input, with the N-th layer receiving an aggregation of all output features from the initial n-1 layers.

Initially, the linear transformation of the adjacency matrix Ã^(head _i) on the input features is computed for each head i. Subsequently, the impact of neighboring nodes' features on the present node is determined for each layer l, and the current node's features are consolidated with the previous layer's output:


    0.87H̃^(i,l)=ReLU((Ã^(head _i)x)^(i) W^(i, l)) + H̃^(i, l-1)
where, L is the quantity of layers in the graph convolution layer, W^(i,l) is the weight parameter of the i-th head in the l-th layer.

The feature representation resulting from the output of each head is combined to form the final output of this layer:








    0.87H̃^(𝐌𝐮𝐥𝐭𝐢𝐇𝐞𝐚𝐝)=Concat(H̃^(1, l), …, H̃^(i, l)) W^O
where W^O ∈ℝ^hd × d_model is the weight parameter of the linear transformation applied to the ultimate output.




  §.§.§ Transformer-enc Layer

The Transformer-enc layer is composed of multiple encoder layers stacked together. These encoder layers bear resemblance to the encoder layers delineated in the transformer model introduced by <cit.>. However, distinctively, our approach involves solely utilizing the output generated by the final three layers of the Encoder for the purpose of averaging. Each encoder layer incorporates self-attention mechanism and Feedforward Neural Network (FFN). This module facilitates the derivation of hidden representations of entities and an attention distribution matrix that are used as input for subsequent layers. The calculations can be outlined as follows:

    0.87self-Att(H̃_Q, H̃_K, H̃_V)=softmax(H̃_Q H̃_K^T/√(d_k)) H̃_V




    0.87La=LayerNorm(H̃+self-Att(H̃))




    0.87(H̃,Ã)=La+FFN(La)

Where H̃_Q, H̃_K, H̃_V is the query, key, and value representations obtained from the linear transformation of H̃, d_k is the dimension of the attention head. LayerNorm is the layer normalization operation.



  §.§.§ Collaborative Prediction Layer


The local context extraction methodology, as described by <cit.>, is employed to ascertain the importance of individual tokens in relation to the entity pair ( Es,Eo), which is interpreted as the sentence-level importance. Erecting on this base, document-level importance was deduced by apportioning diverse attention weights in accordance to the contribution of each sentence within the document to the prediction of entity relations, and by establishing a fixed threshold. Sentences that exceed this threshold are selected as evidence sentences. The sentence-level importance q_i^(Es, Eo) and document-level importance p_j^(Es, Eo) can be computed as follows:

    0.95q_i^(Es, Eo) =∑_i=1^H̃Ã_Es·Ã_Eo




    0.95p_j^(Es, Eo)=∑_j=1^lq_i^(Es, Eo)


We apportion more attention to evidence sentences and less to non-evidence sentences through evidence supervision to further coordinate the prediction results of document-level relation extraction. As depicted in Figure <ref>, we train a teacher model on the Human-Annotated Data (which encompasses relation labels and evidence sentences) of DocRED (Step 1). We utilize the trained teacher model to predict the entity relations and evidence sentence distribution in Distantly-Supervised Data (which encompasses relation labels but lacks evidence sentences) (Step 2). Subsequently, we train a student model on the Distantly-Supervised Data that incorporates evidence sentences (Step 3), and retrain the student model using Human-Annotated Data (Step 4). 
Additionally, we define a row vector z^(Es, Eo) consisting only of 0s and 1s, generated based on Human-Annotated Data. This vector indicates whether each sentence is an evidence sentence for the relation triples: 1 if it is, and 0 if it is not.

    0.90z^(Es, Eo)=∑_sent=1^Lz^(Es, Eo) / 1^⊤∑_sent=1^Lz^(Es, Eo)

where 1 is the row vector composed of all 1, L is the total count of sentences contained within the document.






 §.§ Classification Layer


We begin with the computation of a weighted average of entity importance at the sentence-level q_i^(Es, Eo), and subsequently cascading it with the previous entity representation. Following this, we apply the tanh activation function to normalize the input to range between (-1, +1), resulting in the contextual representation of the two associated entities. The computation is elaborated as:



    0.85c_Es=tanh(W_Es[h_Es ; H̃^⊤q_i^(Es, Eo)]+b_Es)
       0.85c_Eo=tanh(W_Eo[h_Eo ; H̃^⊤q_i^(Es, Eo)]+b_Eo),
where W_Es, W_Eo∈ℝ^d × 2 d, b_Es, b_Eo∈ℝ^d.

Finally, apply the grouped bilinear classifier proposed by <cit.> to calculate the relation category scores.
Score^(Es, Eo)=c_Es^⊤W_Rnc_Eo+b_Rn,
The possibility of the entity pair (Es, Eo) possessing a relation Rn is computed thusly:
0.8P(Rn | Es, Eo)=Sigmoid(Score^(Es, Eo)).














§ EXPERIMENTS







 §.§ Dataset and Evaluation

DocRED[<https://github.com/thunlp/DocRED>] <cit.> is a benchmark dataset for document-level relation extraction tasks, released by Tsinghua University. DocRED comprises numerous documents from Wikipedia and Wikidata, each annotated with entities, relations between entities, and evidence sentences that support relation triples. It serves as the predominant benchmark for DocRE model training and evaluation.


Re-DocRED[<https://github.com/tonytan48/Re-DocRED>] <cit.> and Revisit-DocRED[<https://github.com/AndrewZhe/Revisit-DocRED>] <cit.> are modified datasets of DocRED. They supplement a large number of relation triples to solve the problems of incomplete annotations, coreferential errors, and inconsistent logic in docred. Annotation quality has high accuracy and consistency, which provides a more reliable benchmark for DocRE-model training and evaluation.

We assess GEGA using an Nvidia Tesla V100 16GB GPU and evaluate it with F1, Ign-F1, and Evi-F1 metrics. Ign-F1 represents the calculated F1 score attained by excluding relational facts present in both the training and development/testing datasets. 

Evi-F1 serves as a significant measure for assessing the performance of ER and constitutes a new benchmark for assessing the quality of relation extraction models.



 §.§ Single and Fusion

In the task of RE, the most ideal scenario is that the evidence sentence set of the dataset already contains all contextual information necessary to predict entity relations, thereby enabling accurate relation prediction results based solely on the evidence sentence set. However, manually labeled data and distant supervision data often fall short in this regard. Therefore, it is necessary to extract contextual information from the entire document to predict entity relations. We divide the above problem into two evaluation methods: (1) Single: extract entity relations from the entire document and obtain the corresponding prediction scores; (2) Fusion: predict entity relations based on a collection of evidence sentences and combine the prediction results with those from the Single method. This is similar to the Fusion of Evidence approach in <cit.>.




 §.§ Compared Methods

To ensure a fair comparison of the performance of DocRE baselines, we compare our model with three state-of-the-art methods, all using BERT-base as the pre-trained language model (PLM), which are:
(1) Sequence-based methods:
CNN <cit.>,
LSTM <cit.> ,
BiLSTM <cit.>. 

(2) Graph-based methods:
GAIN <cit.>,
MRN <cit.>,
DocuNet <cit.>,
GTN <cit.>,
SD-DocRE <cit.>,
AA <cit.>. 
(3) Transformer-based methods:

ATLOP <cit.>,

EIDER <cit.>,
SAIS <cit.>,
PRiSM <cit.>,
DREEAM <cit.>.
On this foundation, we categorize the above methods into two major classes: without Distant Supervision and with Distant Supervision.
























§ RESULTS AND ANALYSES




We test the trained student and teacher models in both the Single and Fusion stages, and report the results on DocRED, Re-DocRED and Revisit-DocRED, where the results on Revisit-DocRED are moved to appendix.


 §.§ Results on DocRED

Table <ref> indicates that GEGA achieves superior Ign-F1 and F1 metrics compared to the established DocRE Baselines on both the development set and the test set. The single stage of the student model has achieved performance levels comparable to the leading DREEAM <cit.>. Notably, the fusion stage of the student model achieved the highest recorded scores, surpassing DREEAM by 0.34% (Ign F1) and 0.55% (F1) on the development set, as well as by 0.17% (Ign F1) and 0.44% (F1) on the test set.

GEGA also performs well in the test of the new benchmark Evi-F1, surpassing the previous most advanced DREEAM by 0.41% (Evi-F1) and 0.46% (Evi-F1) on the development set and test set respectively. In the table, it is evident that the Transformer-based and Graph-based models outperform the Sequence-based ones, validating the rationality behind integrating infographics with the Transformer.



 §.§ Results on Re-DocRED


Table <ref> presents the feedback outcomes of GEGA on the development and test sets of RE-DocRED, demonstrating that our GEGA achieves state-of-the-art results compared to other methods utilizing BERT-base as a pre-trained language model. Notably, GEGA has outperformed all other methods in the table during the fusion stage without Distant Supervision (Teacher). GEGA secured the highest Ign F1 and F1 scores in the fusion stage with Distant Supervision (Student), with improvements of 2.08% (Ign F1) and 2.52% (F1) respectively on the development set over the second-place GTN-BERT <cit.>, and by 1.58% (Ign F1) and 1.97% (F1) respectively on the test set.



 §.§ Effect Analysis of GCNs and ER

Based on the test scores on DocRED and Re-DocRED, we observe that graph-based models such as DocuNET <cit.> and GTN-BERT <cit.> have achieved superior performance in the field. Graphs have an advantage in conveying document-level contextual information, so we used grid search to select a 2-layer GNNs to guide multiple attention maps. Additionally, DREEAM <cit.> is the first method to enhance relation extraction performance purely through evidence-guided attention, it has already achieved excellent scores on the DocRED set. By integrating GCNs with evidence retrieval, we further improved its scores by 0.41% (Evi-F1) and 0.46% (Evi-F1) on the dev and test sets, respectively. Therefore, we conclude that GCNs and ER significantly enhance performance in the relation extraction task.



 §.§ Ablation Studies






We conducted ablation experiments were conducted on the development set to analyze the GNNs layer and Training phase of GEGA. The single phase of GEGA (Student) was utilized as the test benchmark. The results of the score post-ablation of each part are presented in Table <ref>. Initially, we removed the Attention Concentration Layer, which resulted in a minor decline in performance. Subsequently, upon removing the Multi-GraphConv Layer, a significant performance deterioration was observed, implying the importance of constructing multiple attention distribution graphs for relation extraction. Upon removal of the Transformer-enc Layer, we noted a relatively minor decline in performance. We speculate that this may be related to using Transformer based BERT as PLM.


Additionally, during the training process, we performed ablation on the self-training, fine-tuning, and Distant Supervision-training stages, in order to further analyze their impact. The results indicate that when the ER self-training phase is omitted, performance declines, whereas the absence of the fine-tuning stage did not lead to a noticeable decline in performance. Further more, omitting the Distant Supervision-training stage caused severe performance degradation. These findings highlight the effectiveness of our ER method in enhancing relation extraction.









§ CONCLUSION

We propose GEGA, the first model to employ GCNs and ER jointly guided attention to enhance DocRE. We validate the superiority of our model on three widely used datasets: DocRED, Re-DocRED and Revisit-DocRED. GEGA is trained using parallel computing in both fully supervised and semi-supervised settings, without incurring additional overhead, making it convenient for use in the era of Large Language Models (LLMs). In the future, we aim to leverage the scalability of GEGA and apply it to a broader range of scenarios, including entity recognition, event extraction, and more.




§ LIMITATIONS

The model GEGA is subject to two limitations. Firstly, when utilizing Multi-GraphConv Layers to induce multiple fully connected attention distribution matrices, there is a possibility of generating one matrix that differs significantly from others in terms of weight distribution. This could lead to significant deviations in prediction results. We hypothesize that guiding the construction of multiple fully connected attention matrices using evidence information may reduce the occurrence of such undesirable situations, a conjecture that will be verified in future work.
Secondly, it is acknowledged that the relations between most entity pairs can be predicted based on the local context of the entities. However, our model utilizes evidence sentences retrieved from the entire document corpus, which are strongly correlated with the entity pairs of interest, rather than evidence sentences obtained specifically for individual relation triples. This approach may result in the model carrying more global contextual information while reducing the utilization of local context information.




§ ETHICS STATEMENT

Our proposed GEGA demonstrates outstanding scalability and applicability, serving as an excellent solution for both DocRE and DocER tasks. This method is evaluated solely on publicly available datasets, ensuring no compromise on individual privacy. Furthermore, we provide the source code implementation of GEGA to enable researchers to reproduce its performance authentically, fostering academic exchange in the field of DocRE.


















§ LOSS

To accommodate the requirements of the RE and ER tasks within both the teacher model and the student model, distinct forms of loss function computation have been devised for the relation classification approach.The loss variation is shown in Figure <ref>.






 §.§ RE loss:

For the RE tasks of the two models previously described, we implement the Adaptive Thresholding Loss (ATL) as proposed by ATLOP. During the training phase, we use a threshold class (TH) to learn a threshold such that the logits of the positive class ℛ𝒫 exceed it, and the logits of the negative class ℛ𝒩 fall below it.


    ℒ_RE=    0.95-∑_Rn∈ℛ𝒫exp(Score_Rn^(Es, Eo))/∑_Rn^'∈ℛ𝒫∪{TH}exp(Score_Rn^'^(Es, Eo))
       0.95-exp(Score_TH^(Es, Eo))/∑_Rn^'∈ℛ𝒩∪{TH}exp(Score_Rn^'^(Es, Eo))








 §.§ ER loss:

The tasks targeted by the teacher model and the student model differ in detail. The teacher model is trained on Human-Annotated Data, which includes reliable manually annotated evidence sentences, while the student model is trained on Distantly-Supervised Data that incorporates evidence sentences identified through the teacher model's ER. Considering these distinctions, there is a requirement for specialized loss computation methods. Consequently, we propose both document-level and sentence-level loss calculations.

Document-level Loss:
By integrating the document-level importance distribution p_j^(Es, Eo) with the original manually annotated evidence sentences z^(Es, Eo) from the dataset, we induce a localized context representation that contributes significantly to RE. We use the Kullback-Leibler Divergence (KL divergence), a method for measuring the difference between two probability distributions within the same event space. In this paper, it is used to assess the degree of divergence between p_j^(Es, Eo) and z^(Es, Eo):

    ℒ_ER^doc=
    KL( z^(Es, Eo)p_j^(Es, Eo) )   
    
    =∑_sent=1^Lz^(Es, Eo)logz^(Es, Eo)/p_j^(Es, Eo)


Sentence-level Loss:
We use the teacher model trained on Human-Annotated Data to perform ER testing on Distantly-Supervised Data, predicting its sentence-level evidence distribution q̃_i^(E s, E o).
Thereafter, utilize Kullback-Leibler (KL) divergence to compute the difference in sentence-level probability distributions between the teacher model and the student model.

    ℒ_ER^sent=KL(q_i^(E s, E o)q̃_i^(E s, E o)) 
        =∑_t=1^l q_i^(E s, E o)logq_i^(E s, E o)/q̃_i^(E s, E o)

Finally, we apply the prevalent weighted summation technique for document-level and sentence-level losses to equilibrate the losses of RE and ER, where λ serving as a hyperparameter.

    ℒ=(1-λ)ℒ_RE+λℒ_ER






§ HYPERPARAMETER

We ran tests on the above three datasets using different random seeds five times and reported the average test accuracy.
In the manuscript, a comprehensive Table <ref> delineating the hyperparameters and configuration has been furnished, encapsulating pivotal settings utilized across the experimental trials. These hyperparameters, meticulously curated and fine-tuned, exert control over multifarious facets of the model's dynamics throughout the training and evaluation phases. 




§ RESULTS ON REVISIT-DOCRED

Table <ref> shows the test results of GEGA (Student) on Revisit-DocRED. According to the test method of <cit.>, we trained on the DocRED training set, and then tested on the test set provided by Revisit-DocRED. It can be seen that GEGA has a great performance improvement compared with other methods.









§ SUPPLEMENTARY MODEL ANALYSIS

Here, we conducted experimental analyses on the number of GNNs layers and the number of heads in the Multi-GraphConv Layer of GEGA. We selected the number of GNNs layers from {1, 2, 3, 4, 5} and the number of heads from {1, 2, 3, 4}. The experimental results are shown in Figure <ref>. Ultimately, the analysis concluded that the performance is optimal when layers=2 and heads=2.























§ CASE STUDY




To further demonstrate the superior performance of GEGA, we extracted a case from the DocRED dataset and used it to compare the performance of GEGA with two other advanced methods (SAIS and DREEAM).


 §.§ RE

From Figure <ref>, we observe five types of relations among five entities. SAIS correctly identified four types of relations but overlooked the "country of citizenship" relation between "Robert  F./Mark  R." and "United States" Additionally, it incorrectly identified a "country of citizenship" relation between "Robert  F./Mark  R." and "American" as well as a "located in the administrative territorial entity" relation with "Terry  McAuliffe" While DREEAM correctly identified all the existing relations, it excessively identified a "country" relation between "American" and "United  States" In contrast, GEGA perfectly extracted the correct relational network in this case.


 §.§ ER

During evidence retrieval, both GEGA and DREEAM labeled the evidence source for the relation between United  States and Virginia as [S1, S2], missing [S0]. Additionally, DREEAM incorrectly labeled the evidence information for the relation between Terry  McAuliffe and United  States as [S0, S6]. SAIS incorrectly labeled the evidence information for the relation between Virginia and Terry  McAuliffe as [S0, S1, S2, S6]. We believe that GEGA's superior performance is also attributed to its better ER performance.







 Paper Title: Towards interfacing large language models with ASR systems using confidence measures and prompting.



[
     R. Zamini[Corresponding author: rahelehzamini1361@yahoo.com] ^1,  S. Ghafouri^2  and F. Goodarzi^3

	^1  Department  of Mathematics, Faculty of Mathematical Sciences and  Computer, Kharazmi University, Thehran, Iran. 

	^2   Department of Mathematics, Faculty of Science, Arak University, Arak, 384817758, Iran.

^3   Department of Statistics, Faculty of Mathematical Sciences, University of Kashan, Kashan, Iran.

    
===================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
  As LLM grow in parameter size and capabilities, such as interaction
  through prompting, they open up new ways of interfacing with ASR
  systems beyond rescoring n-best lists.
  This work investigates post-hoc correction of ASR transcripts
  with LLM. To avoid introducing errors into likely accurate
  transcripts, we propose a range of confidence-based filtering methods. Our
  results indicate that this can improve the performance of less competitive
  ASR systems.




§ INTRODUCTION


Speech perception is a complex process that relies not only on acoustic
information but also on environmental information, visual cues, context, and
other factors. Consequently, speech perception in the brain is organized in a
hierarchical and highly parallel processing network, where information on
different time scales, about different linguistic units and from different
modalities is analyzed to decipher the semantic content of
speech <cit.>. Due to the reliance on these
contextual cues during speech perception, humans can be considered “noisy
listeners”: to successfully understand the message, humans do not need to
recognize every part of
the speech they hear. Our predictive
brain can replace the missing information based on the available contextual
information <cit.>.

ASR systems operate in a similar way. An acoustic model first processes
the speech signal and identifies linguistic units, such as phonemes. Then, a
LM, which encodes prior knowledge about the likelihood of different word
sequences, helps to find the most likely transcription given the potentially
noisy information from the acoustic model.


The number of parameters in LM and their performance on numerous
benchmarks even without task-specific fine-tuning has increased so much in
recent years that we commonly refer to them as LLM. Especially
instruction-tuned LLM offer new possibilities for down-stream
applications through their prompting mechanism <cit.>.
LLM can also work directly with very long input contexts, obviating the
need to specifically adapt LM to recently observed sequences <cit.>.




Motivated by these developments, we investigate combining an
ASR system with a LLM, where the latter is used as an
additional LM to specifically address ASR errors.
The primary challenge is the trade-off between leveraging LLMs to correct
errors in low-accuracy transcripts while minimizing the risk of introducing
new errors in more accurate ones. Figure <ref> illustrates the
proposed approach with an analogy to speech perception discussed before.
Additionally, we evaluate the impact of LLM and ASR model size on
the effectiveness of LLM corrections.

To reduce the chance of the LLM introducing new errors into the
transcript, we propose three filtering methods that rely on the ASR confidence
scores. For the first two, we let the LLM correct only sentences where the
sentence or the lowest word confidence falls below a given threshold. For the
third method, we prompt the LLM to only correct specific low-confidence
words.
To gain deeper insights into the LLM's behavior, we
also present concrete examples where LLM has corrected errors in the transcription and
other examples where LLM performed poorly.


The rest of this paper is organized as follows: In Section <ref>
we review previous works on applying LLM to ASR.
Section <ref> details our experimental setup and in
Section <ref> we present our results and analysis.



§ RELATED WORKS




With the advent of LLMs, a wide range of works have investigated how these could
improve ASR performance. Generative instruction-tuned LLMs in particular offer
new possibilities of combining ASR systems and LMs via prompting.


In hybrid speech recognition <cit.>, the decoder returns a list or
lattice of hypotheses by combining probabilities from the acoustic model and a
basic n-gram language model. These hypotheses can then be rescored with a more
powerful neural LM <cit.> or LLM <cit.>. Recent
neural end-to-end ASR approaches directly learn an LM, but can also be combined
with a separately trained one by shallow fusion or other
methods <cit.>.

In addition to traditional integration of LMs, one can focus specifically on identifying and correcting errors in ASR outputs <cit.>. Prior works framed post-hoc ASR error correction as a spelling correction or a machine translation problem <cit.>.

In traditional n-best rescoring, only the best hypothesis is selected, although
another one could also be partially or fully correct. Chen et
al. <cit.> therefore instruct LLMs to generate a new
hypothesis based on all n-best options. They found that zero-shot prompting did
not yield improvements on two datasets but adapting pre-trained LLMs with
few-shot prompting, i.e. providing some example ASR outputs and
corresponding corrections, and fine-tuning on a larger set of examples did.

Min et al. <cit.> explored the integration of LLMs in ASR systems to
improve transcription accuracy. Their results show that directly applying the
in-context learning capabilities of the LLMs for improving ASR transcriptions
presents a significant challenge, and often leads to a higher WER.
However, other works <cit.> that
explored the ability of LLMs to select, rescore and correct n-best list or ASR
transcripts showed that zero and few-shot in-context learning can yield
performance gains that are comparable to rescoring by domain-tuned LMs and can
even achieve error rates below the n-best oracle level.
Other works <cit.> have also applied LLM to spoken
language understanding tasks, where the focus lies on identifying the correct
intent from ASR transcripts, rather than correcting errors.

Our work, on the contrary, focuses on giving more insights on how to effectively
use LLMs to improve ASR performance. In a few-shot, in-context learning
scenario, we evaluate the ability of LLMs to correct ASR transcripts. Similar to
Pu et al. <cit.>, we propose
filtering ASR outputs based on confidence scores to prevent the LLM from
introducing errors into transcripts that are likely already correct. We further
analyze the errors introduced and the corrections made by LLMs. By doing so, our
work seeks to shed light on the strengths and limitations of LLMs, when applied
to ASR.




§ EXPERIMENTAL SETUP






 §.§ ASR system


We obtain initial ASR transcriptions from Whisper <cit.>, a competitive set of models trained on 680,000 hours of transcribed speech.
We run Whisper via the  Python
package <cit.>, which supports extracting sentence- and
word-level confidence scores. We compare the following models: Tiny
(39M parameters), Medium (769M), Large V3 (1550M).
While English-only variants exist for the smaller models, we always use the multilingual one.




 §.§ Large language model


We use the following OpenAI ChatGPT models as LLM for error correction
in the ASR transcriptions: ,
, and . The GPT-3.5 turbo
models have a context window of 16,385 tokens and can understand and generate
natural language while GPT-4, with its 128,000 token context window, is a large
multimodal model (handling both text and image inputs with text outputs) and
tackles more complex problems more accurately than its
predecessors <cit.>.
In this work, we call ChatGPT via its Python API and do not modify any default parameters.



 §.§ Confidence-based filtering


Passing every ASR output to the LLM for correction risks introducing errors into correctly transcribed sentences. To mitigate this, we compare different filtering methods based on confidence scores returned by the Whisper ASR model.


Whisper internally uses tokens that are obtained with byte-pair
encoding <cit.>. Each output token is associated with a
confidence score based on its log probability. The 
Python package <cit.> computes a word-level
confidence by averaging all tokens that form a word and a sentence-level
confidence by averaging all tokens in the sentence. Punctuation tokens are
excluded in either case.


We then filter the ASR outputs that should be passed to the LLM
based on the sentence-level or the lowest word-level confidence score in the
sentence.
We will refer to these two methods as sentence-level and
lowest-word confidence. For sentences above a chosen confidence
threshold, we retain the original ASR outputs.

As a third option, we prompt the LLM to only correct
specific words in the ASR transcription that fall below a certain
confidence threshold. If no words within the transcription fall below the
confidence threshold, the original ASR transcription is retained.




 §.§ Dataset


We evaluate our proposed approach on the English LibriSpeech corpus <cit.> of audiobook recordings. We use the  and  subsets for initial experiments and hyperparameter tuning and then report final results on the  and  evaluation sets. Each of these subsets contains around 2500–3000 utterances. Speakers in the  portions are more challenging to recognize and lead to higher WER.


While these LibriSpeech subsets are not included in the Whisper training data, we cannot exclude that ChatGPT was trained on them due to the proprietary nature of the model.




§ RESULTS



In this section, we present our results in terms of WER and CER on identifying a suitable
prompt, comparing different ASR models, and filtering based on confidence
scores. We also discuss and analyze the types of errors made by the
LLM.




 §.§ Prompt selection



We first describe our process of selecting a suitable prompt and analyze which
elements of the prompt are important for ASR performance. LLMs perform best when the prompt contains a clear description of the task. For this reason, we provided information about the task, the format of the input and the expected output, and provided two examples in the prompt.




In the prompt, we clearly explain the task of correcting ASR errors
to the LLM. We further describe the format of the input and expected
response and instruct the LLM not to provide any explanatory or
additional text besides the corrected transcription. We then provide one or two example
input-output pairs for a few-shot learning scenario <cit.>.


We show the base prompt for our experiments with a basic description of the task
in Table <ref>. In other prompts, we explicitly instruct the
LLM to make grammar corrections and to make changes that closely match the
input transcription acoustically or phonetically. Results for
these different prompts in Table <ref> show that in particular
providing more than one example and instructing to make phonetically plausible
corrections improve the ASR performance. For all following experiments, we
therefore use prompt 4.





 §.§ Influence of ASR performance


We study the influence of ASR performance on the LLM corrections by
comparing Whisper models of different strength. As shown
previously <cit.>, less competitive ASR models —
Whisper Tiny and Medium in our case — leave more room for
improvement.

We summarize the results in terms of WER and CER for the original ASR
and the LLM-corrected transcripts (relative
change in parentheses) on both development sets in Table <ref>.
While we observe improvements in WER with LLM correction in most cases,
the relative improvements in  are smaller compared to the ones
in . This suggests that correcting errors in more difficult speech data also
presents a greater challenge for the LLM model.







 §.§ Confidence-based filtering



In Figure <ref> (left) we show the WER for various
sentence-level confidence thresholds for all Whisper models on the
LibriSpeech  subset. Transcriptions
with a confidence score higher than the threshold are not passed to the
LLM for correction.
The Figure shows that the optimal value for the threshold is 0.95 for Tiny and
Medium models while the Large model is not sensitive to the
threshold.

Figure <ref> (right) shows the effect of varying the
lowest-word confidence thresholds. A value of 0.7 provides a good
trade-off of stable ASR performance and reducing the number of utterances
that needs to be corrected by the LLM.
We observed similar patterns for both methods on .






  §.§.§ Correction of specific words



In this experiment, we pass both the ASR transcriptions and a
list of words with confidence scores below a predefined threshold to the
LLM. [Replacing sentences 2–3 in prompt 4 from Table <ref> with “You will be presented with an ASR transcription in json format with keys:  and , where the  is the ASR transcription and  contains the list of words in the transcription with low confidence scores.
  Your task is to correct any errors in the transcription.
  If you come across errors in ASR transcription, make sure that you correct only words from within the  list and your corrections should closely match the original transcription acoustically or phonetically.”]
Figure <ref> presents the WER results for various
confidence thresholds.








As the figure demonstrates, thresholds close to 0 results in a WER near the
original WER (without ChatGPT correction) but a threshold of 1 gives a higher
value for WER than the previous experiments where there was no low-confidence
word list restriction.

The WER reaches its lowest value of 7.55 at a threshold of 0.5, not matching the
performance of the sentence-level and lowest-word confidence approaches.



  §.§.§ Test set performance



Finally, we also present results for our selected best prompt and confidence
thresholds on the LibriSpeech  and 
evaluation sets in Table <ref>. The best result for each dataset
and Whisper model is highlighted in bold. Our findings indicate that, despite its higher
number of parameters, GPT-4 only outperforms GPT-3.5[Here we used
   which we found to perform similar to
   on the development sets, but is faster and more
  robust to API errors.] for Whisper
Tiny, but does not result in additional improvements in WER for the
transcriptions of the Medium and Large models.





 §.§ Error analysis

In this section, we showcase examples of Whisper Tiny outputs on the development sets in which the LLM has corrected
errors in the transcriptions, has failed to correct or even introduced new
errors into the transcription.

<ex1>
REF: their fingers *** sear me like fire//
ASR: their fingers see her me like fire//
LLM: their fingers *** sear me like fire//
[aboveexskip=0em]<ex2>
REF: damn your impertinence sir burst out burgess//
ASR: dam your impertinent sur burst out burges//
LLM: damn your impertinent sir burst out burgess//
[aboveexskip=0em]<ex3>
REF: *** *** fedosya 's face made her anxious//
ASR: the dose used to face nature *** anxious//
LLM: the dose used to face nature *** anxiously//


Example ex1 and ex2 show cases where the LLM has
corrected all or most of the errors within the ASR transcriptions of Whisper
Tiny.  Here, , , and  denotes
reference, ASR, and LLM-corrected transcriptions respectively.

Example ex3 is a typical case of where the LLM
struggles to correct the transcript because it already contains too many errors
and, for example, reconstructing proper nouns without acoustic context is challenging.
Furthermore,
Table <ref> breaks down for how many utterances the LLM
improved, worsened, or did not change the ASR performance. We find more improvements, but also more new errors on the more challenging  subset.





We also note that LLM corrections can sometimes decrease WER while
increasing CER. This occurs because any number of character changes within a
word only affects the WER by one unit (1/N with N being the number
of words in the reference transcription). However, the same changes can have a
greater impact on CER.

<ex4>
REF: pour mayonnaise over all chill and serve//
ASR: parme a nays overall chill and serve//
LLM: parmesan *** over all chill and serve//


Example ex4 demonstrates this effect. The LLM
reduces WER from 57.14% to 28.57% in this example, while CER increases from
25.00% to 27.50%.



§ CONCLUSIONS


In this work we investigated LLM for ASR error correction. Viewing ASR systems as noisy listeners, inspired by human speech perception, we proposed filtering ASR outputs based on confidence measures, so that the LLM only has to focus on less accurate transcripts. Indeed, our results confirm that LLM especially boost ASR performance for less competitive acoustic models because otherwise there is little room left for improvement.

We plan to investigate additional confidence estimation methods
and other ASR systems than Whisper in future work. LLM outputs could also be rescored again with the acoustic model to validate if the proposed changes are acoustically plausible.
We will further consider other long-form datasets
where utterances are not evaluated one-by-one and LLM are expected to
provide more benefits because of their long context windows. Finally, studies
also need to be conducted on other languages, where LLM might not
perform as well as on English.



§ ACKNOWLEDGEMENTS

This research was partially funded by the Swiss Innovation Agency Innosuisse through the flagship project IICT (grant no. PFFS-21-47) and by the Swiss National Science Foundation through the Bridge Discovery project EMIL: Emotion in the loop - a step towards a comprehensive closed-loop deep brain stimulation in Parkinson’s disease (grant no. 40B2–0_194794 EMIL). We thank Olivier Bornet for providing us with a ChatGPT API key and other technical support.




IEEEtran




 Paper Title: Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses.






2024
Copyright for this paper by its authors.
  Use permitted under Creative Commons License Attribution 4.0
  International (CC BY 4.0).



CLiC-it 2024: Tenth Italian Conference on Computational Linguistics, Dec 04 — 06, 2024, Pisa, Italy








1]Gabriele Sarti[
    orcid=0000-0001-8715-2987,
    email=g.sarti@rug.nl,
    url=https://gsarti.com,
]
[1]

1]Tommaso Caselli[
orcid=0000-0003-2936-0256,
email=t.caselli@rug.nl,
]

1]Malvina Nissim[
orcid=0000-0001-5289-0971,
email=m.nissim@rug.nl,
]

1]Arianna Bisazza[
orcid=0000-0003-1270-3048,
email=a.bisazza@rug.nl,
url=https://cs.rug.nl/ bisazza
]


[1]Corresponding author.

[1]Center for Language and Cognition (CLCG), University of Groningen, Oude Kijk in 't Jatstraat 26 
 Groningen, 9712EK, The Netherlands




§ ABSTRACT 
  Rebuses are puzzles requiring constrained multi-step reasoning to identify a hidden phrase from a set of images and letters. In this work, we introduce a large collection of verbalized rebuses for the Italian language and use it to assess the rebus-solving capabilities of state-of-the-art large language models. While general-purpose systems such as LLaMA-3 and GPT-4o perform poorly on this task, ad-hoc fine-tuning seems to improve models' performance. However, we find that performance gains from training are largely motivated by memorization. Our results suggest that rebus solving remains a challenging test bed to evaluate large language models' linguistic proficiency and sequential instruction-following skills.






  Large language models Sequential reasoning Puzzle Rebus Crosswords Enigmistica Italiana



Non Verbis, Sed Rebus: 
 Large Language Models are Weak Solvers of Italian Rebuses
    [
    Submitted August 5, 2024
==================================================================================







§ INTRODUCTION


Complex games such as chess and Go have long been a source of inspiration to develop more flexible and robust AI systems <cit.>. Recent developments in NLP suggested that creative language games could be exploited as promising benchmarks for quantifying the ability of large language models (LLMs) to carry out multi-step knowledge-intensive reasoning tasks under pre-specified constraints <cit.>. While crossword puzzles have been historically the main focus of such efforts <cit.>, other categories of linguistic games received only marginal attention, especially for languages other than English. A prominent example of less-studied language games is the rebus, a visual puzzle combining images and graphic signs to encode a hidden phrase. Indeed, rebus solving is a complex, multi-step process requiring factual knowledge, contextual understanding, vocabulary usage, and reasoning within pre-defined constraints – a set of fundamental skills to address a variety of real-world tasks.

In this work, we conduct the first open evaluation of LLMs' rebus-solving capabilities, focusing specifically on the Italian language. We propose a novel strategy to derive text-only verbalized rebuses from transcribed intermediate rebus solutions and use it to produce a large collection with more than 80k verbalized rebuses. We then evaluate the rebus-solving skills of state-of-the-art LLMs, including open-source systems and proprietary models, via few-shot prompting. Moreover, we fine-tune a small but capable LLM on verbalized rebus solving, outperforming state-of-the-art systems by a wide margin. Finally, we conduct a fine-grained assessment of LLMs' sequential reasoning steps, explaining model performance in terms of word complexity and memorization.

Beyond rebus solving, our evaluation sheds light on the limits of current LLMs in multi-step reasoning settings, highlighting challenges with their application to complex sequential instruction-following scenarios.[Code, data and models are available on https://github.com/gsarti/verbalized-rebusGithub and https://huggingface.co/collections/gsarti/verbalized-rebus-clic-it-2024-66ab8f11cb04e68bdf4fb028Huggingface] 









§ BACKGROUND AND RELATED WORK




  
Italian Enigmistica and Rebuses The Italian language is characterized by a rich and long-standing tradition of puzzle games, including rebuses, dating back to the 19th century <cit.>[Refer to <cit.> for a comprehensive overview of peculiarities and norms in modern Italian rebuses.] In Italian rebuses, a first pass (prima lettura) representing an intermediate solution of the puzzle is produced by combining graphemes with underlying image elements in a left-to-right direction (<Ref>). Then, the letters and words of the first pass undergo a re-segmentation (cesura) according to a solution key (chiave di lettura[Referred to as diagramma in jargon.]), which specifies the length of words in the solution (frase risolutiva). The verbalized rebuses we introduce in this work are variants of textual rebuses (rebus descritto or verbis), where the text-based puzzle is crafted by replacing first pass words with their crossword definitions in a templated format (<Ref>).



  
Linguistic Puzzles as NLP Progress Metrics Language games have recently been adopted as challenging tasks for LLM evaluation <cit.>. While works in this area have historically focused on English crosswords <cit.>, recent tests focus on a more diverse set of games such as the New York Times' “Connections” <cit.> and “Wordle” <cit.>. Automatic crossword solvers were also developed for  French <cit.>,  German <cit.> and Italian <cit.>, while didactic crossword generators are available for Italian <cit.> and Turkish <cit.>. Relatedly, the Italian evaluation campaign EVALITA[<https://www.evalita.it>] recently hosted two shared tasks focusing on the word-guessing game “La Ghigliottina” (The Guillotine) <cit.>. To our knowledge, our work is the first to attempt the computational modeling and evaluation of rebus-solving systems. Importantly, language games such as rebuses are not easily translatable into other languages due to their structural and cultural elements. This makes them a scarce but valuable resource for language-specific evaluations of language processing systems.



  
LLMs as Sequential Reasoners State-of-the-art LLMs were shown to struggle to follow sequential instructions presented in a single query <cit.>, but their performances improved significantly with ad-hoc training <cit.>. This acts as an initial motivation for our rebus-solving fine-tuning experiments. In our evaluation, we also adopt few-shot prompting <cit.> and chain-of-thought reasoning <cit.>, which were both shown to strongly improve LLMs' abilities when solving complex multi-step tasks.



§ EXPERIMENTAL SETUP




  
Data We begin by extracting all rebuses' first passes and solutions available on Eureka5[<http://www.eureka5.it>, additional details in <Ref>. Rebus illustrations are not available in Eureka5.], an online repository of Italian puzzles. We refer to the resulting dataset containing 223k unique rebuses sourced from various publications as EurekaRebus. For crossword definitions, we use ItaCW <cit.>, containing 125k unique definition-word pairs. We select only EurekaRebus examples in which all first pass words match an existing ItaCW definition to enable verbalization, maintaining 83,157 examples for our modeling experiments.[Since verbalized rebus are produced from textual contents only, crossword definitions may refer to different word meanings (e.g. [Two soccer players] is used to represent the word “wings” in <Ref> despite not matching the word sense “bird wings” of the original image. This does not affect the validity of our task.] Since several ItaCW words are associated with multiple definitions, we randomly sample definitions to promote diversity in the resulting verbalized rebuses. A test set of 2k examples[Composed by Test id and Test ood, described in <Ref>] is kept aside for evaluation, and the remaining 81k examples are used for model training. 





  
Models We fine-tune Phi-3 Mini 3.8B 4K <cit.>, the most capable LLM below 4B parameters for a wide range of Italian language tasks[<https://hf.co/spaces/FinancialSupport/open_ita_llm_leaderboard>]. We use quantized low-rank adapters (QLoRA; ) for efficient fine-tuning with Unsloth[<https://github.com/unslothai/unsloth>] and Transformers <cit.>, training the model for 5,000 steps with a batch size of 16 over 81k examples. For comparing our model performances, we select GPT-4o <cit.> and Claude-3.5 Sonnet <cit.> as the current state-of-the-art for proprietary LLMs and the instruction-tuned variants of Qwen-2 72B <cit.> and LLaMA-3 70B <cit.> 
as the best-performing open-source LLMs according to the Invalsi Italian benchmark <cit.>. These four systems are used as untrained baselines thanks to their instruction-following abilities and prompted for rebus solving in a few-shot setting. 



  
Format <Ref> presents an example in the templated format used for fine-tuning Phi-3.[An English example is available in <Ref>] The model is prompted to reason step-by-step by 1) solving crossword definitions sequentially (definition resolution); 2) producing a first pass copying letters and definitions' words; 3) re-segmenting it into solution words based on the solution key (solution segmentation); and finally 4) producing the solution by copying re-segmented words. We automatically convert rebuses in this format by deriving the solution key from solution word lengths and dynamically infilling the available information into the template. We use a similar format for prompting experiments, with five in-context step-by-step demonstrations and an explicit instruction asking the model to stick to the previous examples' format to streamline solution parsing.



  
Metrics For our granular evaluation of rebus-solving performance, we adopt the following set of metrics focusing on the first passes (FP) and solutions (S) generated by LLMs:


    
  * Definition (Def.): Proportion of correctly guessed words during definition resolution.
    
  * First Pass Words/Letter Accuracy: Proportion of correct words and letters in the generated first pass. Lower scores may indicate issues with assembling a first pass from previous information.
    
  * First Pass Exact Match (EM): Proportion of generated first passes matching the gold reference.
    
  * Solution Key Match: Proportion of generated solution words matching the lengths specified by the solution key. Lower scores may indicate difficulty in respecting the given length constraints.
    
  * Solution First Pass Match: Proportion of first pass characters employed to construct solution words. Lower scores indicate issues with using generated first pass characters in the solution.[In practice, we define this as 1 - CER(FP, S), where CER is the character error rate <cit.> between the two sequences (lowercased, whitespace removed) computed with https://github.com/jitsi/jiwerJiwer]
    
  * Solution Words Accuracy: Proportion of correct words in the generated solution.
    
  * Solution Exact Match (EM): Proportion of generated solutions matching the gold reference.




§ RESULTS


<Ref> presents our evaluation results. We observe that all prompted models perform poorly on the task, with the overall best prompted system (Claude 3.5 Sonnet) obtaining the correct solution only for 24% of the 2k tested examples. Notably, open-source systems perform significantly worse than proprietary ones, producing correct first passes only for 4% of the examples, and next to no correct solutions. Our fine-tuned system largely outperforms all state-of-the-art prompted models, predicting the correct solution in 51% of cases. From first pass metrics, it is evident these results can be largely explained by the poor word-guessing capabilities of the models, which are greatly improved with fine-tuning. For prompted models, the slight decrease in scores between Def. and FP Words also highlights issues with copying predicted words in the expected format. Finally, we observe that fine-tuning strongly improves the constraint-following abilities of our system, with prompted systems being less strict with applying length and letter-choice constraints for their solutions (Key/FP Match).





§ WHAT MOTIVATES MODEL PERFORMANCES?


In light of the strong performances achieved by our relatively small fine-tuned system, this section conducts an in-depth investigation to identify factors motivating such performance improvements.



  
Word Complexity and Frequency Affects LLM Fine-tuning Performance
 For every word in the first passes and solutions of test set examples, we measure LLMs' overall accuracy in predicting it for the full test set. We then correlate this score to various quantities that could motivate LLMs' performances. More specifically, we use 1) the word frequency in the training set; 2) the word frequency in Paisà <cit.>, a large web Italian corpus; and 3) the length of the word (number of characters). We find a significant positive correlation (ρ = 0.44) between first pass word prediction accuracy and training frequency for the fine-tuned Phi-3 model, suggesting that model performance is strongly related to training coverage. The length of characters is also found to negatively affect our model's performance, albeit to a smaller extent (ρ = -0.11). The performance of prompted models is unrelated to both properties for first pass words, indicating that these results are the product of fine-tuning.[Paisà frequency is never found to correlate significantly. Full correlation results are available in <Ref>.]



  
LLM Fine-Tuning Fails to Generalize to Unseen Words






To further confirm the importance of fine-tuning word coverage in defining model performances, we evaluate our fine-tuned model in out-of-distribution settings. For this evaluation, the 2k examples of the test set from previous sections are divided into two subsets: one in which all first pass words were seen during fine-tuning by Phi-3 (Test id, 1061 examples) and one in which, for every example, at least one first pass word was unseen in training (Test ood, 939 examples). Intuitively, if Phi-3 performance is mainly motivated by memorizing fine-tuning data, introducing OOD words should produce a significant drop in model performances. Results shown in <Ref> confirm that this is indeed the case. We find Phi-3 performances to be near-perfect on seen first pass words (FP W._ID = 0.96) in both test sets, with a major drop for OOD words (FP W._OOD = 0.20). This produces second-order effects on subsequent steps, causing the FP EM results to drop by 71% (FP EM Test Δ), while significantly impacting downstream solution accuracies. On the contrary, GPT-4o few-shot prompting performances remain nearly identical on both splits, confirming that these results are not the product of a skewed data selection process. Overall, these results strongly suggest that memorization is the main factor behind the strong rebus-solving performance of our fine-tuned LLM.



  
Manual Inspection

We conclude by manually evaluating some generations produced by the best-performing LLMs. <Ref> presents two examples with definitions (D) and solution (S) words predicted by three LLMs, with more examples provided in <Ref>. We use naw as short-hand for “Not A Word” to mark nonsensical terms.

In the first example, Phi-3 correctly predicts all first pass and solution words. On the contrary, other models make several mistakes in the first pass, leading to incorrect solutions. Both prompted models tend to ignore first pass words when these cannot be assembled to form sensical, length-fitting solution words. For example, for D1 GPT-4o predicts p (naw), which would lead to the solution word “SAPpTE” (naw), but the S8 = “Spettacolo” (show) is predicted instead by the model). In particular, GPT-4o appears to prioritize grammatically correct solutions at the cost of ignoring first pass words and solution key length constraints, while Claude 3.5S shows an improved ability to follow these constraints, as confirmed by Key/FP Match results of <Ref>.

In the second example, the first pass word D2 = salice (willow) is OOD for Phi-3. Consequently, the model produces the incorrect prediction aro (naw), and the error is propagated to all solution words, as previously observed in the Test OOD column of <Ref>. Prompted models also underperform in this example, with errors on D1 and D2 propagating to most solution words. However, we note that D1 and D2 incorrect predictions for Claude 3.5S satisfy the provided definitions, suggesting that access to more explicit information about the given constraints could further boost LLMs' performance on this task.



§ DISCUSSION AND CONCLUSION


This work introduced a verbalized rebus-solving task and dataset for evaluating LLMs' sequential instruction following skills for the Italian language. We crafted a large collection of 83k verbalized rebuses by combining rebus transcriptions with crossword definitions and used it to evaluate the rebus-solving skills of state-of-the-art LLMs. Our experiments revealed the challenging nature of this task, with even the most capable prompted models achieving only 24% accuracy on solutions.

While fine-tuning a smaller LLM dramatically improved performance to 51% solution accuracy, our analysis uncovered that these gains were largely driven by memorization and do not generalize to out-of-distribution examples. These results suggest important limitations in the generalization capabilities of current systems for sequential instruction following tasks. Our manual analysis further shows that LLMs seldom account for length constraints when solving definitions, despite the fundamental role of these cues in restricting the pool of possible words. These results suggest that search-based approaches accounting for constraints more explicitly might improve puzzle structure adherence, as previously shown by <cit.>. Other augmentation techniques employing LLM reformulation skills can also be explored to mitigate overfitting.

Future work in this area should focus on expanding similar evaluations to a wider set of languages, input modalities, and puzzle categories, creating a comprehensive benchmark to test LLMs' puzzle-solving skills. Importantly, the task of solving visual rebuses and their more convoluted variants[For example, rebuses requiring first pass anagrams (anarebus) or dynamic relations derived from multi-scene analysis  (stereorebus)] remains far beyond the current capabilities of vision-language models. Hence, solving these puzzles automatically can be considered an important milestone in developing multimodal AI systems for constrained multi-step reasoning tasks. Our results confirm that the challenging nature of rebuses, even in their verbalized form, makes this task valuable for assessing future progress in LLMs' linguistic proficiency and sequential reasoning abilities. Finally, our rebus-solving LLM can facilitate future interpretability work investigating the mechanisms behind factual recall and multi-step reasoning in transformer models <cit.>.



  
Limitations 
Our analysis was limited to a relatively small set of models, and a single prompt template obtained after minimal tuning. Further experiments are needed to verify that memorization patterns after fine-tuning remain relevant for other model sizes, prompt formats, and training regimes, particularly for full-weight training approaches.


   Gabriele Sarti and Arianna Bisazza acknowledge the support of the Dutch Research Council (NWO) for the project InDeep (NWA.1292.19.399). 
   Arianna Bisazza is further supported by the NWO Talent Programme (VI.Vidi.221C.009).
   We are grateful to the http://www.enignet.it/homeAssociazione Culturale “Biblioteca Enigmistica Italiana - G. Panini” for making its rebus collection freely accessible on the Eureka5 platform, and to Valeriya Zelenkova for her valuable comments on the first version of this work.








§ ADDITIONAL DATA INFORMATION





  
Dataset statistics <Ref> presents statistics for the EurekaRebus dataset and the filtered subset we use for composing verbalized rebuses. The ItaCW dataset contains a total of 125,202 definitions for 40,963 unique words, with the most frequent words having hundreds of different definitions, e.g. 173 for re (king), 155 for te (you). Definitions used for verbalization are randomly sampled from the pool of available definitions for every word.







  
First pass/Solution word distribution <Ref> shows the distribution of first pass and solution words for the filtered EurekaRebus subset used in our work.



§ ADDITIONAL EXPERIMENTAL RESULTS



<Ref> presents the correlations between model accuracy and the properties presented in <Ref>. <Ref> presents the full ID/OOD performances for all tested models, showing consistent results with <Ref> for all prompted models. <Ref> presents Phi-3 Mini performances across rebus-solving fine-tuning steps.













§ ADDITIONAL MODEL GENERATIONS



<Ref> presents an English translation of <Ref> example using the prompt format adopted in this study. <Ref> provide additional example of LLM generations for tested rebuses, with the example from <Ref> (bottom) being OOD due to the manovella (crank) word in D2, and the others being ID for the fine-tuned Phi-3 Mini.









 Paper Title: QUITO: Accelerating Long-Context Reasoning through Query-Guided Context Compression.





QUITO






W. Wang et al.



1. Institute of Computing Technology, Chinese Academy of Sciences 

2. Beijing University of Posts and Telecommunications 








QUITO: Accelerating Long-Context Reasoning through Query-Guided Context Compression
    Wenshan Wang1 Yihang Wang2 Yixing FanCorresponding Author: fanyixing@ict.ac.cn1 Huaming Liao1 Jiafeng Guo1
    
==============================================================================================================



§ ABSTRACT 

In-context learning (ICL) capabilities are foundational to the success of LLM. 
Recently, context compression has attracted growing interest since it can largely reduce reasoning complexities and computation costs of LLM.
In this paper, we introduce a novel QUITO method, which leverages attention of the question over the contexts to filter useless information. Specifically, we take a trigger token to calculate the attention distribution of the context in response to the question. Based on the distribution, we propose three different filtering methods to satisfy the budget constraints of the context length.

We evaluate the QUITO using two widely-used datasets, namely, NaturalQuestions and ASQA. Experimental results demonstrate that QUITO significantly outperforms established baselines across various datasets and downstream LLM, underscoring its effectiveness. Our code is available at <https://github.com/Wenshansilvia/attention_compressor>.



















§ INTRODUCTION

In recent years, LLM has demonstrated notable reasoning and generating capabilities, significantly enhancing the performance of NLP tasks <cit.>. 
However, these models still exhibit limitations in acquiring real-time information and integrating external knowledge <cit.>. 
ICL addresses these deficiencies by including examples and relevant contexts directly within the prompts<cit.>. This approach boost the performance of LLM in downstream tasks without requiring additional training.

To better improve the reasoning ability of LLM, researchers propose different ways to incorporate complex contexts in the input <cit.>. For example, RAG employs an additional searcher to retrieve external relevant documents about the question as the context of inputs, which has attracted lots of attention for both the academia and industry <cit.>. In addition,
Brown et al. <cit.> found that the number of examples has a great impact to the reasoning performance of LLM, where more examples tend to bring better performances <cit.>. 
Moreover, the CoT <cit.> further improves the LLM by involving the reasoning step of each example in the context. 
While these strategies have the potential to significantly improve the capabilities of LLM, they also introduce challenges associated with the increased context length, such as higher inference complexity and costs.





To mitigate this issue, context compression in ICL is becoming a prominent solution. On one hand, reducing the length by removing noise from contexts can improve inference efficiency<cit.>. On the other hand, it meets the input length restrictions of open-source LLMs<cit.> while also reduces the costs associated with accessing proprietary LLMs. 
Several methods <cit.> have been proposed to compress context by estimating the information entropy. This assessment is conducted by utilizing a small external LLM to evaluate the perplexity of individual tokens to identify those that contribute minimal information gain. Tokens that demonstrate low information are subsequently compressed or eliminated. 
However, neglecting the query during compression may result in the inadvertent deletion of key information. 

For the above problem, recent methods such as LongLLMLingua <cit.> adopt a query-aware compression approach by calculating the perplexity of the context conditioned on the query. Despite this advancement, misalignment between compression model and generation model can lead to inconsistencies in determining which tokens are considered to have “low entropy gain”. This discrepancy arises because models may differ in their interpretation and processing of the same information. Our work also scores tokens based on their relevance to the query. However, distinctively, we employ attention metrics rather than perplexity to assess the importance of tokens.

This paper introduces the QUITO method, which strategically selects the context to maintain supporting information by utilizing the attention mechanism. 

Intuitively, the attention mechanism offers a direct method for analyzing the interactions between the question and the context, moving beyond the sole reliance on models' probabilistic uncertainty. This technique facilitates a more precise identification of the information that is most crucial to the current task. 

More importantly, the attention-based filtering can be implemented with small LLM, which improves the computation efficiency.



The main contributions of this study include: 

    
  * This paper proposes a novel context compression method, named QUITO. It utilises self-attention mechanism of Transformers to score the importance of tokens, selecting context relevant to the current query. 
    
    
  * In contrast to earlier methods that requires a compression model with 7 billion or 13 billion parameters, this method achieves superior results using a smaller LLM with only 0.5 billion parameters.
    
  * We conduct extensive experiments on two benchmark datasets, which demonstrate the effectiveness of the proposed QUITO. For example, it surpasses strong baselines with an increase in accuracy of up to 20.2.
    





§ RELATED WORK


In this section, we briefly review two lines of related works, i.e., context compression task and attention mechanism.



 §.§ Context Compression Task

To reduce the length of context, earlier efforts<cit.> opted to summarize and condense retrieved documents using models such as GPT. Other studies <cit.> focused on distinguishing between useful and redundant information within documents, training a model to extract the most valuable sentences. For example, LeanContext <cit.> and FILCO <cit.> train the model to perform sentence-level extraction for the context. Fit-RAG <cit.> scores sub-paragraphs with sliding context windows. RECOMP <cit.> uses a generative model to rewrite extracted candidate sentences, thereby ensuring the coherence and naturalness of the summaries.

Approaches that generate summaries do not allow direct control over the compression ratio, resulting in a growing attention on token and word-level compression techniques in recent times. SelectiveContext <cit.> utilizes self-information within context for token selection. This approach considers perplexity (PPL) to be the representation of the uncertainty of an LLM regarding information carried by contexts. 

Based on <cit.>, LLMLingua <cit.> introduces a two-stage, coarse to a fine, compression method. 
However, these methods fail to consider the relationship between the context and the query. LLMLingua <cit.> further addresses this gap by calculating context-specific perplexity conditioned on the query.

The aforementioned token-level compression methods utilize perplexity as the primary filtering criterion. However, discrepancies often arise between smaller compression models and larger generation models in their assessments of word perplexity, making it challenging to align their judgments on lexical importance.




 §.§ Attention Mechanism

Attention is a significant breakthrough in deep learning, particularly shines in NLP tasks such as translation and summary generation<cit.>. The core concept behind Attention mechanisms involves assigning a specific weight to each input element, such as words or tokens, indicating their relevance to the task at hand. This allows models to focus selectively on more pertinent parts of the input data.

Self-attention, a particular category of the attention mechanism, measures the relationships between all input elements, assessing how each element influences and relates to the others<cit.>. 


Multi-head attention is a key component of the Transformers <cit.>, which improves the model’s capability in capturing diverse correlation patterns.


Recent studies try to use the attention mechanisms within LLMs to accomplish specific tasks. For instance, DRAGIN <cit.> use attention to evaluate the extent to which a given text segment significantly influences subsequent content.
It employs the perplexity of tokens to determine whether to trigger re-retrieval and regeneration processes.

In this paper, we also employ the multi-head attention mechanism to calculate the weights of tokens in context, thereby identifying useless content for answer generation.







§ METHOD





In this section, we introduce the QUITO method in detail. As illustrated in Figure <ref>, QUITO primarily consists of two main components, namely the query-guided self-attention component and the context budget control component. 

In what follows, we will firstly give a formal definition of the task, and then describe each component in detail.




 §.§ Problem Formulation

Given an input p=(s, C, q), where s is the instruction, q is the query, and C={c_i}_i=1^n is the context consisting n documents. Every document c_i={w_i, j}_j=1^L_i contains L_i word. The objective of context compression task can be formulated as:

    min_C̃dist(P(ỹ|s, C̃, q), P(y|s, C, q)),

where ỹ represents the predicted response of the LLM, and y is the ground truth response. dist ( · , · ) is a function that measures the distance of two distributions, such as KL divergence. 

C̃={c̃_̃ĩ}_i=1^n is the compressed context, and  c̃_̃ĩ={w_ j|w_j∈ c_i}_j=1^τ L_i, τ∈ [0, 1]. c̃_̃ĩ is c_i being compressed with ratio at 1/τ, where smaller τ means higher compression ratio.



 §.§ Query-Guided Self-Attention

The query-guided self-attention component aims to estimate the importance of tokens in context by calculating the trigger attention distribution. Firstly, we organize all input by prompt template filling. Then, we calculate the importance of all the input with trigger attention distribution. Finally, we obtain the lexical units importance within the context by context attention reformulating.



  §.§.§ Prompt Template Filling

It is crucial that the compression model fully understands the task at hand and accurately identifies the information most pertinent to the current query. A standard approach involves concatenating the context with the query and subsequently analyzing how tokens within the query attend to tokens in the context. However, in a Transformer decoder-only architecture, the visibility range of each token in the query varies. This variability suggests that tokens positioned later in the sequence more precisely reflect the model’s comprehensive understanding of the task. Given the challenges associated with appropriately weighting tokens at different positions, we propose a novel method that utilize a conversational template and identify a specialized token that encapsulates the compression model’s overall understanding of the task.



  §.§.§ Trigger attention distribution

We embed the context and query into a conversational template, concluding with a signal that prompts the model to initiate response generation. The terminal token within this sequence is designated as a trigger token, serving as an indicator of the model’s assessment of information need after comprehensively understanding the task at hand. Subsequently, we employ a compression model equipped with a multi-head self-attention mechanism to process the completed template and compute the attention that the trigger token accords to the preceding text:

    {α_i|α_i=exp(q_L_total^Tk_i)/∑_j=1^L_totalexp(q_L_total^Tk_j)},

where q_i and k_i are query embedding and key embedding of the i th token, respectively. L_total is the total number of tokens in the completed template. 



  §.§.§ Context attention reformulating

Once the attention allocated by the trigger token to all preceding tokens in the sequence has been determined, the subsequent step involves transforming this attention data into a quantified measure of significance for the lexical units within the context. 
The array {α_i} signifies attention weights, with its length equating to the aggregate of the lengths of the conversational template, the context, and query. Within the scope of this task, it is imperative to concentrate on the attention distributed to the context segment. The attention should not be diluted by the segments pertaining to the template and the query. Consequently, we implement a normalization process, which is designed to ensure that the distribution of attention across various tokens in the context remains unbiased, robust to the disparities in context and query lengths that may exist across different tasks. For the normalization we use softmax function:

    α_i'=exp(α_i+doc_start)/∑_j=doc_start^doc_endexp(α_j), i ∈[1, doc_end-doc_start],

where doc_start and doc_end represent the starting and ending positions, respectively, of the context segment.

We consider words to be the smallest semantic units within a document. In order to perform selection on semantic units, the next step involves transforming scores on token to scores attributed to each individual unit. In other words, we need to transform {α_i'}_i=1^L_doc to {α_i”}_i=1^L, where L_doc is the length of token array {t_i}_i=1^L_doc that belongs to context, and L is the length of word array {w_i}_i=1^L.
A word w_i may consist of one or more tokens. We can formulate a word as w_i={t_j}_j=k+1^k+l, each of which has attention score:

    α_i”=max_k+1 ≤ j ≤ k+lα_j',

where the length of the array {α_i”}_i=1^L is L.



 §.§ Context Budget Control


In the previous section, we have derived a list of words, represented as {w_i}_i=1^L, and the corresponding array of attention weights, {α_i”}_i=1^L. 
This section introduces the filtering methods that satisfy the requirement of the context budget control.




  §.§.§ Phrase Level Filtering

In the process of selecting based on attention scores, it is common to inadvertently overlook words adjacent to those with high attention, referred to as target words, which may also contain crucial knowledge for answering the query. To rectify this oversight and ensure these adjacent words are also considered, we apply a weighted adjustment, allowing them to receive a portion of the attention attributed to the target words. This is accomplished by implementing a Gaussian filter across the word attention array {α_i”}_i=1^L.

    G(x)=1/2πσ^2exp(-x^2/2σ^2)

After the application of the Gaussian function G(x) to {α_i”}_i=1^L, the resulting Gaussian-modulated attention array is denoted as {α_i”'}_i=1^L.
Subsequently, we identify the words from set {w_i}_i=1^L that rank within the top τ L based on their attention scores {α_i”'}_i=1^L.

    
  * Perform a sort on {α_i”'}_i=1^L on descending order, which yields an ordered set of indices {j_1, j_2, …, j_L}.
    
  * Select corresponding words from {w_i}_i=1^L with index {j_1, j_2, …, j_τ L} ,which yields {w_i'}_i=1^τ L.
    
  * Reorganize the set of selected words {w_i'}_i=1^τ L to reflect their original sequential order within the context.

Although the selection process targets individual words, the application of Gaussian filtering often leads to the selection of contiguous words, thereby effectively forming phrases.


  §.§.§ Sentence Level Filtering

In addition to phrase-level filtering, sentence-level filtering is also implemented to preserve more comprehensive semantic information. Using the NLTK toolkit, we extract semantic units at the sentence level. Each sentence s_i, denoted as 
s_i={w_j}_j=k+1^k+l, is assigned an attention score based on the maximum score of the tokens it contains. Subsequently, mirroring the phrase-level filtering process, we prioritize incorporating sentences with higher attention scores into the selection set, while ensuring that the aggregate word count remains below τ L.



  §.§.§ Dynamic Sentence Level Filtering

Sentence-level filtering often leads to a compression ratio greater than the designated target 1/τ. To more effectively adhere to the predetermined compression rate and optimize budget utilization, we augment the results of sentence-level filtering with word-level filtering. Specifically, subsequent to sentence-level filtering, if the count of words is L', we are then able to select an additional τ L-L' words. These additional words are chosen via phrase-level filtering from the text that was not previously selected. The newly selected words are subsequently concatenated with the results from sentence-level filtering to form the final compressed output.



§ EXPERIMENTS




 §.§ Datasets and Evaluation Metrics

In this paper, we assess the efficacy of the proposed QUITO method across two distinct scenarios: open domain question answering and long-form question answering. Specifically, we employ the NaturalQuestions (NQ) and ASQA datasets as the testbed.


For NQ dataset, We employed a processed version as described in <cit.>, where each query is paired with 20 documents, among which only one document contains the correct answer. In alignment with the procedures specified in <cit.>, accuracy was used as the metric to determine whether the generated responses accurately included the correct answer.
For the ASQA dataset, the answer to the question maybe multi-facet as there are many ambiguous questions.


Each ambiguous question in the ASQA dataset has answers reflecting multiple interpretations of these ambiguities. We utilize the dataset version provided by <cit.>, which includes 5 retrieved documents/snippets from Wikipedia for each query.
In accordance with <cit.>, our evaluation metrics included Exact Match (EM), a RoBERTa-based QA score (DisambigF1), and ROUGE <cit.>.







 §.§ Baselines and Implementation



  §.§.§ Baselines

We take three state-of-the-art compression approaches as baselines: For query-unaware methods, we select Selective-Context<cit.> and LLMLingua<cit.>, which implements cross entropy scoring to remove redundant vocabulary. For query-aware method, we compare our approach with Longllmlingua<cit.>. LongLLMLingua implements a two-stage compression method. It first evaluates and reranks multiple retrieved contexts, followed by a token-level compression stage, allocating varying compression budgets to these contexts based on their initial scores. 
For fair comparison, we excluded the context reranking phrase of LongLLMLingua (marked as LongLLMLingua† in Table <ref> and Figure <ref>), concentrating on the token-level compression.










  §.§.§ Detailed Implementation

For fair comparison, we follow LLMLingua <cit.> to use Longchat-13B-16k [https://huggingface.co/lmsys/longchat-13b-16k] as the generation model.

To ensure the reproducibility of the results, we apply greedy decoding strategy throughout the inference process, with the temperature parameter set to zero. The compression model is implemented with Qwen2-0.5B-Instruct[https://huggingface.co/Qwen/Qwen2-0.5B-Instruct].





 §.§ Main Results


Table <ref> presents the comparative performance of our method, QUITO, against three baseline methods across various compression rates and datasets. 
Firstly, we can see that selective-context is a strong baseline compared with LLMLingua and LongLLMLingua† on both 2x and 4x compression rates.
Secondly, QUITO obtains significantly better performances than all baselines, e.g., the improvement of QUITO with phrase level filtering against selective-context, LLMLingua, and LongLLMLingua†(i.e., 2x compression ratio) on NQ is 5.7, 20.2, and 17.7, respectively. 
Finally, we find that QUITO with different filtering method all achieve better performances on both datasets. However, there is no consistent advantages of each filtering method when compared on different datasets. This maybe that the context length on NQ and ASQA differs significantly, i.e., the average length of context on NQ and ASQA is about 2904 and 721 tokens, respectively.
All the results demonstrate the effectiveness of QUITO in compressing contexts for the LLM.









 §.§ Analysis on different position of the ground truth context

We analyse the performance of the QUITO compression method across different ground truth context positions within the NQ dataset. This dataset comprises 20 context document fragments per query, of which only one contains the answer and is designated as the ground truth document. We assessed the impact of this document's positioning at the 1st, 5th, 10th, 15th, and 20th ranks on the efficacy of various compression strategies. 






















axis background/.style=fill=white,
grid=both,
  xtick pos=left,
  ytick pos=left,
  tick style=
    major grid style=style=gallery,line width=1pt,
    minor grid style=mercury,
    
    ,
  minor tick num=1,





The results presented in Figure <ref> indicate that all context compression methods struggle with the 'lost in the middle' phenomenon, as described by <cit.>. Performance is optimal when the ground truth context is positioned at the beginning; however, it deteriorates significantly when the ground truth context is placed in the middle. Among the evaluated methods, LLMLingua<cit.> exhibits the most resilience to the 'lost in the middle' phenomenon. This robustness may be attributed to its strategy of allocating higher compression ratios to contexts containing a greater density of information. Overall, the QUITO method consistently surpasses the two baseline methods across a variety of ground truth context positions and compression rates. On average, QUITO improves upon the performance of Selective Context<cit.> by +19.6 and LLMLingua<cit.> by +13.6.



 §.§ Analysis on different generation models






To better understand the generation ability of different LLM, we evaluate the performance of 4 widely-used models, including Longchat-13B-16k [https://huggingface.co/lmsys/longchat-13b-16k], Llama3-8b-Instruct [https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct], GLM4-9b-chat [https://huggingface.co/THUDM/glm-4-9b-chat], and Mistral-7b-instruct [https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2]. These models were tested with contexts compressed at a rate of 2 on the NQ dataset. The generated responses from these compressed contexts were then compared with those derived from uncompressed contexts.

As depicted in Figure <ref>, the Mistral-7B-Instruct model significantly outperforms the other three generation models despite having fewer parameters. This superior performance may be attributed to the incorporation of GQA and SWA during its training phase, which enhances its capability to process long sequence inputs.

While the context is compressed at 2x ratio, we find that the GLM4-9b-chat model show the smallest performance decline, with a decrease of 8.9, and the Mistral-7B-Instruct has the greatest decline.
When the compression ratio is 4x, we can see that all generation models obtain a relative close performance except for LongChat-13B-16k. These maybe that the LongChat-13B-16k is released earlier than other three models, and the latter are trained more deeply. 








§ CONCLUSION

This paper introduces the QUITO method, a novel attention-based importance estimation for long context compression in LLM.

The QUITO method employs a trigger token that comprehensively considers the query to assess the importance of each lexical unit within the context, thereby filtering out units with low relevance scores. 

Evaluations conducted on the NQ and ASQA datasets demonstrate that our method outperforms state-of-the-art compression methods such as Selective Context, LLMLingua, and LongLLMLingua, confirming its superior ability to preserve essential information needed by LLMs to respond to queries effectively.
For future work, we would like to study the combination of the context compression and re-ranking module, since the re-ranking stage in RAG also targets on selecting useful information for final answer generation. 










splncs04



























 Paper Title: Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trends.



Mitral Regurgitation Recogniton based on Unsupervised Out-of-Distribution Detection with Residual Diffusion Amplification
    
Zhe Liu1,2Zhe Liu, Xiliang Zhu and Tong Han contribute equally to this work.

Corresponding authors’ email: gzhongshan1986@163.com and  xinyang@szu.edu.cnXiliang Zhu1,2⋆Tong Han1,2⋆Yuhao Huang1,2Jian Wang3

Lian Liu1,2,4Fang Wang5Dong Ni1,2Zhongshan Gou5()

Xin Yang1,2()

    August 5, 2024
=====================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Large autoregressive generative models have emerged as the cornerstone for achieving the highest performance across several Natural Language Processing tasks. 
However, the urge to attain superior results has, at times, led to the premature replacement of carefully designed task-specific approaches without exhaustive experimentation. 
The Coreference Resolution task is no exception; all recent state-of-the-art solutions adopt large generative autoregressive models that outperform encoder-based discriminative systems. 
In this work, we challenge this recent trend by introducing , a carefully designed – yet simple – pipeline, which enables running a state-of-the-art Coreference Resolution system within the constraints of an academic budget, outperforming models with up to 13 billion parameters with as few as 500 million parameters.
achieves state-of-the-art performance on the CoNLL-2012 benchmark, training with up to 0.006x the memory resources and obtaining a 170x faster inference compared to previous state-of-the-art systems.
We extensively validate the robustness of the framework with an array of diverse experiments, reporting improvements over prior systems in data-scarce, long-document, and out-of-domain settings.
We release our code and models for research purposes at .




§ INTRODUCTION

As one of the core tasks in Natural Language Processing, Coreference Resolution aims to identify and group expressions (called mentions) that refer to the same entity <cit.>. 
Given its crucial role in various downstream tasks, such as Knowledge Graph Construction <cit.>, Entity Linking <cit.>, Question Answering <cit.>, Machine Translation <cit.> and Text Summarization <cit.>, inter alia, there is a pressing need for both high performance and efficiency.
However, recent works in Coreference Resolution either explore methods to obtain reasonable performance optimizing time and memory efficiency <cit.>, or strive to improve benchmark scores regardless of the increased computational demand <cit.>.

Efficient solutions usually rely on discriminative formulations, frequently employing the mention-antecedent classification method proposed by <cit.>.
These approaches leverage relatively small encoder-only transformer architectures <cit.> to encode documents and build on top of them task-specific networks that ensure high speed and efficiency.
On the other hand, performance-centered solutions are nowadays dominated by general-purpose large Sequence-to-Sequence models <cit.>.


A notable example of this formulation, and currently the state of the art in Coreference Resolution, is <cit.>, which proposes a transition-based system that incrementally builds clusters of mentions by generating coreference links sentence by sentence in an autoregressive fashion. 
Although Sequence-to-Sequence solutions achieve remarkable performance, their autoregressive nature and the size of the underlying language models (up to 13B parameters) make them dramatically slower and memory-demanding compared to traditional encoder-only approaches. 
This not only makes their usage for downstream applications impractical, but also poses a significant barrier to their accessibility for a large number of users operating within an academic budget.

In this work we argue that discriminative encoder-only approaches for Coreference Resolution have not yet expressed their full potential and have been discarded too early in the urge to achieve state-of-the-art performance.
In proposing  , we strike an optimal balance between high performance and efficiency, a combination that was missing in previous systems.
Our framework enables an encoder-only model to achieve top-tier performance while keeping the overall model size less than one-twentieth of the current state-of-the-art system, and training it with academic resources.
Moreover, when further reducing the size of the underlying transformer encoder, performs in the same ballpark as encoder-only efficiency-driven solutions while improving speed and memory consumption.
Finally, we propose a novel incremental Coreference Resolution method that, integrated into the framework, results in a robust architecture for out-of-domain, data-scarce, and long-document settings.





§ RELATED WORK

We now introduce well-established approaches to neural Coreference Resolution.
Specifically, we first delve into the details of traditional discriminative solutions, including their incremental variations, and then present the recent paradigm shift for approaches based on large generative architectures.



 §.§ Discriminative models

Discriminative approaches tackle the Coreference Resolution task as a classification problem, usually employing encoder-only architectures.
The pioneering works of <cit.> introduced the first end-to-end discriminative system for Coreference Resolution, the Coarse-to-Fine model.
First, it involves a mention extraction step, in which the spans most likely to be coreference mentions are identified.
This is followed by a mention-antecedent classification step where, for each extracted mention, the model searches for its most probable antecedent (i.e., the extracted span that appears before in the text).
This pipeline, composed of mention extraction and mention-antecedent classification steps, has been adopted with minor modifications in many subsequent works, that we refer to as Coarse-to-Fine models.



  
Coarse-to-Fine Models
Among the works that build upon the Coarse-to-Fine formulation, <cit.>, <cit.> and <cit.> experimented with changing the underlying document encoder,  utilizing ELMo <cit.>, BERT <cit.> and SpanBERT <cit.>, respectively, achieving remarkable score improvements on the English OntoNotes <cit.>. 
Similarly, <cit.> introduced s2e-coref that reduces the high memory footprint of SpanBERT by leveraging the LongFormer <cit.> sparse-attention mechanism. 
Based on the same architecture, <cit.> analyzed the impact of having multiple experts score different linguistically motivated categories (e.g., pronouns-nouns, nouns-nouns, etc.).
While the foregoing works have been able to modernize the original Coarse-to-Fine formulation, training their architectures on the OntoNotes dataset still requires a considerable amount of memory.[Training those models requires at least 32G of VRAM.]
This occurs because they rely on the traditional Coarse-to-Fine pipeline that, as we cover in Section <ref>, has a large memory overhead and is based on manually-set thresholds to regulate memory usage. 



  
Incremental Models
Discriminative systems also include incremental techniques. 
Incremental Coreference Resolution has a strong cognitive grounding: research on the “garden-path” effect shows that humans resolve referring expressions incrementally <cit.>.

A seminal work that proposed an automatic incremental system was that of <cit.>, which introduced a clustering approach based on the shift-reduce paradigm. In this formulation, for each mention, a classifier decides whether to SHIFT it into a singleton (i.e., single mention cluster) or to REDUCE it within an existing cluster.
The same approach has recently been reintroduced in ICoref <cit.> and longdoc <cit.>, which adopted SpanBERT and LongFormer, respectively. 
In these works the mention extraction step is identical to that of Coarse-to-Fine models. 
On the other hand, the mention clustering step is performed by using a linear classifier that scores each mention against a vector representation of previously built clusters, in an incremental fashion. 
This method ensures constant memory usage since cluster representations are updated with a learnable function.
In Section <ref> we present a novel performance-driven incremental method that obtains superior performance and generalization capabilities, in which we adopt a lightweight transformer architecture that retains the mention representations.



 §.§ Sequence-to-Sequence models

Recent state-of-the-art Coreference Resolution systems all employ autoregressive generative approaches. 
However, an early example of Sequence-to-Sequence model, TANL <cit.>, failed to achieve competitive performance on OntoNotes.
The first system to show that the autoregressive formulation was competitive was ASP <cit.>, which outperformed encoder-only discriminative approaches.
ASP is an autoregressive pointer-based model that generates actions for mention extraction (bracket pairing) and then conditions the next step to generate coreference links.
Notably, the breakthrough achieved by ASP is not only due to its formulation but also to its usage of large generative models. 
Indeed, the success of their approach is strictly correlated with the underlying model size, since, when using models with a comparable number of parameters, the performance is significantly lower than encoder-only approaches.
The same occurs in <cit.>, a fully-seq2seq approach where a model learns to generate a formatted sequence encoding coreference notation, in which they report a strong positive correlation between performance and model sizes.

Finally, the current state-of-the-art system on the OntoNotes benchmark is held by Link-Append <cit.>, a transition-based system that incrementally builds clusters exploiting a multi-pass Sequence-to-Sequence architecture. 
This approach incrementally maps the mentions in previously coreference-annotated sentences to system actions for the current sentence, using the same shift-reduce incremental paradigm presented in Section <ref>. 
This method obtains state-of-the-art performance at the cost of using a 13B-parameter model and processing one sentence at a time, drastically increasing the need for computational power.
While the foregoing models ensure superior performance compared to previous discriminative approaches, using them for inference is out of reach for many users, not to mention the exorbitant cost of training them from scratch.



§ METHODOLOGY

In this section, we present the framework: we propose replacing the preprocessing and training strategy of Coarse-to-Fine models with a novel pipeline that improves the training and inference efficiency of Coreference Resolution systems.
Furthermore, with the , we eliminate the dependency on long-standing manually-set hyperparameters that regulate memory usage.
Finally, building on top of our pipeline, we propose three models that adopt a mention-antecedent classification technique, namely and , and a system that is based upon a novel incremental formulation, .



 §.§ Pipeline

The combines i) a novel mention extraction method, ii) an efficient mention regularization technique, and iii) a new mention pruning strategy.



  
Mention Extraction
When it comes to extracting mentions from a document D, there are different strategies to model the probability that a span contains a mention.
Several previous works follow the Coarse-to-Fine formulation presented in Section <ref>, which consists of scoring all the possible spans in D. 
This entails a quadratic computational cost in relation to the input length, which they mitigate by introducing several pruning techniques.

In this work, we employ a different strategy.
We extract coreference mentions by first identifying all the possible starts of a mention, and then, for each start, extracting its possible end.
To extract start indices, we first compute the hidden representation (x_1, …, x_n) of  the tokens (t_1, …, t_n) ∈ D using a transformer encoder, and then use a fully-connected layer F to compute the probability for each t_i being the start of a mention as:

    F_start(x) = W_start'(GeLU(W_startx))


    p_start(t_i) = σ(F_start(x_i))


with W'_start, W_start being the learnable parameters, and σ the sigmoid function.
For each start of a mention t_s, i.e., those tokens having p_start(t_s) > 0.5, we then compute the probability of its subsequent tokens t_j, with s ≤ j, to be the end of a mention that starts with t_s.
We follow the same process as that of the mention start classification, but we condition the prediction on the starting token by concatenating the start, x_s, and end, x_j, hidden representations before the linear classifier:

    F_end(x, x') = W'_end(GeLU(W_end[x, x']))


    p_end(t_j | t_s) = σ(F_end(x_s, x_j))

with W'_end, W_end being learnable parameters.
This formulation handles overlapping mentions since, for each start t_s, we can find multiple ends t_e (i.e., those that have p_end(t_j|t_s) > 0.5).

Previous works already adopted a linear layer to compute start and end mention scores for each possible mention, i.e., s2e-coref <cit.>, and LingMess <cit.>.
However, our mention extraction technique differs from previous approaches since i) we produce two probabilities (0 < p < 1) instead of two unbounded scores and ii) we use the computed start probability to filter out possible mentions, which reduces by a factor of 9 the number of mentions considered compared to existing Coarse-to-Fine systems (Table <ref>, first row).



  
Mention Regularization
To further reduce the computation demand of this process, in the we use the end-of-sentence (EOS) mention regularization strategy: 
after extracting the span start, we consider only the tokens up to the nearest EOS as possible mention end candidates.[We note that all the well-established Coreference Resolution datasets are sentence-split.]
Since annotated mentions never span across sentences, EOS mention regularization prunes the number of mentions considered without any loss of information.
While this heuristic was initially introduced in the implementation of <cit.>, all the recent Coarse-to-Fine have abandoned it in favor of the maximum span-length regularization, which is a manually-set hyperparameter that regulates a threshold to filter out spans that exceed a certain length.
This implies a large overhead of unnecessary computations and introduces a structural bias that does not consider long mentions that exceed a fixed length.[The max-length regularization filters out 196 correctly annotated spans when training on OntoNotes.]
In our work, we not only reintroduce the EOS mention regularization, but we also study its contribution in terms of efficiency, as reported in Table <ref>, second row.



  
Mention Pruning After the mention extraction step, as a result of the , we consider an 18x lower number of candidate mentions for the successive mention clustering phase (Table <ref>). 
This step consists of computing, for each mention, the probability of all its antecedents being in the same cluster, incurring a quadratic computational cost.
Within the Coarse-to-Fine formulation, this high computational cost is mitigated by considering only the top k mentions according to their probability score, where k is a manually set hyperparameter.
Since after our mention extraction step we obtain probabilities for a very concise number of mentions, we consider only mentions classified as probable candidates (i.e., those with p_end > 0.5 and p_start > 0.5),  reducing the number of mention pairs considered by a factor of 10.

In Table <ref>, we compare the previous Coarse-to-Fine formulation with the new .





 §.§ Mention Clustering

As a result of the , we obtain a set of candidate mentions M = (m_1, m_2, …, m_l), for which we propose three different clustering techniques:
and , which use two well-established Coarse-to-Fine mention-antecedent techniques, and , which adopts a novel incremental technique that leverages a light transformer architecture.



  
Mention-Antecedent models 
The first proposed model, , adopts an equivalent mention clustering strategy to <cit.>:
given a mention m_i = (x_s, x_e) and its antecedent m_j=(x_s', x_e'), with their start and end token hidden states, we use two fully-connected layers to model their corresponding representations: 
 
    F_s(x) = W_s'(GeLU(W_sx))

 
    F_e(x) = W_e'(GeLU(W_ex))

we then calculate their probability to be in the same cluster as:

    p_c(m_i, m_j) = σ (F_s(x_s) · W_ss·  F_s(x_s') + 
    
    F_e(x_e) · W_ee· F_e(x_e') +  
    
    F_s(x_s) · W_se· F_e(x_e') + 
     
    F_e(x_e) · W_es· F_s(x_s'))

with W_ss, W_ee, W_se, W_es being four learnable matrices and W_s, W_s', W_e, W_e' the learnable parameters of the two fully-connected layers.

A similar formulation is adopted in , where, instead of using only one generic mention-pair scorer, we use 6 different scorers that handle linguistically motivated categories, as introduced by <cit.>.
We detect which category k a pair of mentions m_i and m_j belongs to (e.g., if m_i is a pronoun and m_j is a proper noun, the category will be Pronoun-Entity) and use a category-specific scorer to compute p_c.
A complete description of the process along with the list of categories can be found in Appendix <ref>.



  
Incremental model Finally, we introduce a novel incremental approach to tackle the mention clustering step, namely , which follows the standard shift-reduce paradigm introduced in Section <ref>.
Differently from the previous neural incremental techniques (i.e., ICoref <cit.> and longdoc <cit.>) which use a linear classifier to obtain the clustering probability between each mention and a fixed length vector representation of previously built clusters, leverages a lightweight transformer model to attend to previous clusters, for which we retain the mentions' hidden representations. 
Specifically, we compute the hidden representations (h_1, …, h_l) for all the candidate mentions in M using a fully-connected layer on top of the concatenation of their start and end token representations.
We first assign the first mention m_1 to the first cluster c_1=(m_1).
Then, for each mention m_i ∈ M at step i we obtain the probability of m_i being in a certain cluster c_j by encoding h_i with all the representations of the mentions contained in the cluster c_j using a transformer architecture.
We use the first special token ([CLS]) of a single-layer transformer architecture T to obtain the score S(m_i, c_j) of m_i being in the cluster c_j=(m_f, …, m_g) with f ≤ g < i as:
 
    S(m_i, c_j) = W_c · (ReLU(T_CLS(h_i, h_f, …, h_g)))

Finally, we compute the probability of m_i belonging to c_j as:
 
    p_c(m_i ∈ c_j| c_j = (m_f, …, m_g)) = σ(S(m_i, c_j))

 We calculate this probability for each cluster c_j up to step i.
 We assign the mention m_i to the most probable cluster c_j having p_c(m_i ∈ c_j) > 0.5 if one exists, or we create a new singleton cluster containing m_i.








As we show in Sections <ref> and <ref>, this formulation obtains better results than previous incremental methods, and is beneficial when dealing with long-document and out-of-domain settings.



 §.§ Training

To train a model, we optimize the sum of three binary cross-entropy losses: 
 
    L_coref = L_start + L_end + L_clust

Our loss formulation differs from previous transformer-based Coarse-to-Fine approaches, which adopt the marginal log-likelihood to optimize the mention to antecedent score <cit.>.
Since their formulation “makes learning slow and ineffective, especially for mention detection” <cit.>, we directly optimize both mention extraction and mention clustering with a multitask approach. 
L_start and L_end are the start loss and end loss, respectively, of the mention extraction step, and are defined as:

    L_start = ∑_i=1^N -(y_ilog(p_start(t_i)) + 
    
    (1 - y_i)log(1 - p_start(t_i))) 
    
    L_end = ∑_s=1^S∑_j=1^E_s -(y_ilog(p_end(t_j|t_s)) +  
    
    (1 - y_i)log(1 - p_end(t_j|t_s)))

where N is the sequence length, S is the number of starts, E_s is the number of possible ends for a start s and p_start(t_i) and p_end(t_j|t_s) are those defined in Section <ref>.

Finally, L_clust is the loss for the mention clustering step. Since we experiment with two different mention clustering formulations, we use a different loss for each clustering technique, namely L_clust^ant for the mention-antecedent models, i.e., and ,  and L_clust^incr for the incremental model, i.e.,  : 

    L_clust^ant = ∑_i=1^|M|∑_j=1^|M|-(y_ilog(p_c(m_i|m_j)) + 
    
    (1 - y_i)log(1 - p_c(m_i|m_j))) 
    
    L_clust^incr =∑_i=1^|M| (∑_j=1^C_i-(y_ilog(p_c(m_i ∈ c_j)) + 
     
    (1 - y_i)log(1 - p_c(m_i ∈ c_j))))

where |M| is the number of extracted mentions, C_i is the set of clusters created up to step i, and  p_c(m_i|m_j) and p_c(m_i ∈ c_j) are  defined in Section <ref>.

All the models we introduce are trained using teacher forcing. 
In particular, in the mention token end classification step, we use gold start indices to condition the end tokens prediction, and, for the mention clustering step, we consider only gold mention indices.
For , at each iteration, we compare each mention only to previous gold clusters.






§ EXPERIMENTS SETUP



 §.§ Datasets

We train and evaluate all the comparison systems on three Coreference Resolution datasets: 



  
OntoNotes <cit.>, proposed in the CoNLL-2012 shared task, is the de facto standard dataset used to benchmark Coreference Resolution systems. 
It consists of documents that span seven distinct genres, including full-length documents (broadcast news, newswire, magazines, weblogs, and Testaments) and multiple speaker transcripts (broadcast and telephone conversations).



  
LitBank <cit.> contains 100 literary documents typically used to evaluate long-document Coreference Resolution. 



  
PreCo <cit.> is a large-scale dataset that includes reading comprehension tests for middle school and high school students. 


Notably, both LitBank and PreCo have different annotation guidelines compared to OntoNotes, and provide annotation for singletons (i.e., single-mention clusters).
Furthermore, we evaluate models trained on OntoNotes on three out-of-domain datasets: 



    
  * GAP <cit.> contains sentences in which, given a pronoun, the model has to choose between two candidate mentions.
    
  * LitBank_ns and PreCo_ns, the datasets' test-set where we filter out singleton annotations.
    
  * WikiCoref <cit.>, which contains Wikipedia texts, including documents with up to 9,869 tokens.  



The statistics of the datasets used are shown in Table <ref>.



 §.§ Comparison Systems



  
Discriminative
Among the discriminative systems, we consider c2f-coref <cit.> and s2e-coref <cit.>, which build upon the Coarse-to-Fine formulation and adopt different document encoders.
We also report the results of LingMess <cit.>, which is the previous best encoder-only solution, and f-coref <cit.>, which is a distilled version of LingMess.
Furthermore, we include CorefQA <cit.>, which casts Coreference as extractive Question Answering, and wl-coref <cit.>, which first predicts coreference links between words, then extracts mentions spans. 
Finally, we report the results of incremental systems, such as ICoref <cit.> and longdoc <cit.>.



  
Sequence-to-Sequence
We compare our models with TANL <cit.> and ASP <cit.>, which frame Coreference Resolution as an autoregressive structured prediction.
We also include Link-Append <cit.>, a transition-based system that builds clusters with a multi-pass Sequence-to-Sequence architecture. 
Finally, we report the results of seq2seq <cit.>, a model that learns to generate a sequence with Coreference Resolution labels.



 §.§ Setup

All models use DeBERTa-v3 <cit.> as the document encoder.
We use DeBERTa because it can model very long input texts effectively <cit.>.[This is because its attention mechanism enables its input length to grow linearly with the number of its layers.]
Moreover, compared to the LongFormer, which was previously adopted by several token-level systems, DeBERTa ensures a larger input max sequence length (e.g., DeBERTa_large  can handle sequences up to 24,528 tokens while LongFormer only 4096) and has shown better performances empirically in our experiments on the OntoNotes dataset.
On the other hand, using DeBERTa to encode long documents is computationally expensive because its attention mechanism incurs a quadratic computational complexity.
Whereas this further increases the computational cost of traditional Coarse-to-Fine systems, the enables us to train models that leverage DeBERTa_large on the OntoNotes dataset, without any performance-lowering pruning heuristic.
To train our models we use Adafactor <cit.> as our optimizer, with a learning rate of 3e-4 for the linear layers, and 2e-5 for the pre-trained encoder. 
We perform all our experiments within an academic budget, i.e., a single RTX 4090, which has 24GB of VRAM.
We report more training details in Appendix <ref>.



§ RESULTS






 §.§ English OntoNotes

We report in Table <ref> the average CoNLL-F1 score of the comparison systems trained on the English OntoNotes, along with their underlying pre-trained language models and total parameters.
Compared to previous discriminative systems, we report gains of +2.2 CoNLL-F1 points over LingMess, the best encoder-only model. 
Interestingly, we even outperform CorefQA, which uses additional Question Answering training data.

Concerning Sequence-to-Sequence approaches, we report extensive improvements over systems with a similar amount of parameters compared to our large models (500M): we obtain +3.4 points compared to ASP (770M), and the gap is even wider when taking into consideration Link-Append (3B) and seq2seq (770M), with +6.4 and +5.6, respectively.
Most importantly, models surpass the performance of all Sequence-to-Sequence transformers even when they have several billions of parameters.
Among our proposed methods, shows the best performance, setting a new state of the art with a score of 83.6 CoNLL-F1 points on the OntoNotes benchmark.
More detailed results, including a table with MUC, B^3, and CEAFϕ_4 scores and a qualitative error analysis, can be found in Appendix <ref>. 



 §.§ PreCo and LitBank

We further validate the robustness of the framework by training and evaluating systems on the PreCo and LitBank datasets.
As reported in Table <ref>, our models show superior performance when dealing with long documents in a data-scarce setting such as the one LitBank poses.
On this dataset, achieves a new state-of-the-art score of 78.3, and gains +1.0 CoNLL-F1 points compared with seq2seq.
On PreCo, outperforms longdoc, but seq2seq still shows slightly better performance.
This is mainly due to the high presence of singletons in PreCo (52% of all the clusters).
Our systems, using a mention extraction technique that favors precision rather than recall, are penalized compared to high recall systems such as seq2seq.[Precision and Recall scores are reported in Appendix <ref>.]
Among our systems, , leveraging its hybrid architecture, performs better on both PreCo and LitBank. 

















 §.§ Out-of-Domain Evaluation

In Table <ref>, we report the performance of systems along with LingMess, the best encoder-only model, when dealing with out-of-domain texts, that is, when they are trained on OntoNotes and tested on other datasets.
First of all, we report considerable improvements on the GAP test set, obtaining a +1.2 F1 score compared to the previous state of the art. 
We also test models on WikiCoref, PreCo_ns and LitBank_ns (Section <ref>).
However, since the span annotation guidelines of these corpora differ from the ones used in OntoNotes, in Table <ref> we also report the performance using gold mentions, i.e., skipping the mention extraction step (gold column).[We do not include autoregressive models because none of the original articles report scores on out-of-domain datasets. We also could not test those models because they do not provide the code to perform mention clustering alone, and performing it with such approaches is not as straightforward as in encoder-only models.]
On the WikiCoref benchmark, we achieve a new state-of-the-art score of 67.2 CoNLL-F1, with an improvement of +4.2 points over the previous best score obtained by LingMess. 
On the same dataset, when using pre-identified mentions the gap increases to +5.8 CoNLL-F1 points (76.6 vs 82.4).
In the same setting, our models obtain up to +7.3 and +10.1 CoNLL-F1 points on Preco_ns and LitBank_ns, respectively,  compared to LingMess. 
These results suggest that the training strategy makes this model more suitable when dealing with pre-identified mentions and out-of-domain texts.
This further increases the potential benefits that systems can bring to many downstream applications that exploit coreference as an intermediate layer, such as Entity Linking <cit.> and Relation Extraction <cit.>, where the mentions are already identified.
Among our models, on LitBank_ns and WikiCoref, outperforms and , confirming the superior capabilities of the incremental formulation in the long-document setting.



Finally, we highlight that the performance gap between using gold mentions and performing full Coreference Resolution is wider when tested on out-of-domain datasets (on average +17%) compared to testing it directly on OntoNotes (83.6 vs 93.6, +10%).[An evaluation of the proposed models in terms of mention extraction and mention clustering using gold mentions scores can be found in Appendix <ref>.]
This result, obtained on three different out-of-domain datasets, suggests that the difference in annotation guidelines considerably contribute to lower the OOD performances (-7%).



























































































 §.§ Speed and Memory Usage

In Table <ref>, we include details regarding the training time and the hardware used by each comparison system, along with the measurement of the inference time and peak memory usage on OntoNotes the validation set.
Compared to Coarse-to-Fine models, which require 32GB of VRAM, we can train systems under 18GB.
At inference time both and , exploiting DeBERTa_large, achieve competitive speed and memory consumption compared to wl-coref and s2e-coref.
Furthermore, when adopting DeBERTa_base, proves to be the most efficient approach[In terms of inference peak memory usage and speed.] among those directly trained on OntoNotes, while, at the same time, attaining performances that are equal to the previous best encoder-only system, LingMess. 
The only system that shows better inference speed is f-coref, but at the cost of lower performance (-3.0).

Compared to the previous Sequence-to-Sequence state-of-the-art approach, Link-Append, we train our models with 175x less memory requirements. 
Comparing inference time is more complicated, since we could not run models on our memory-constrained budget. 
For this reason, we report the inference times from the original articles, and hence times achieved with their high-resource settings.
Interestingly, we report as much as 170x faster inference compared to seq2seq, which exploits parallel inference on multiple GPUs, and 85x faster when compared to the more efficient ASP.
Among models, is notably slower both in inference and training time, as it incrementally builds clusters using multiple steps.






























































































 §.§ Ablation

In Table <ref>, we compare and models with s2e-coref and LingMess, respectively, using different pre-trained encoders.
Interestingly, when using DeBERTa, systems not only achieve better speed and memory efficiency, but also obtain higher performance compared to the previous systems. 

When using the LongFormer, instead, their scores are in the same ballpark, showing empirically that the training procedure better exploits the capabilities of DeBERTa.
To test the benefits of our novel incremental formulation, , we also implement a model with the previously adopted incremental method used in longdoc and ICoref (Section <ref>), which we call _prev-incr. 
Compared to the previous formulation we report an increase in score of +3.9 CoNLL-F1 points.
The improvement demonstrates that exploiting a transformer architecture to attend to all the previously clustered mentions is beneficial, and enables the future usage of hybrid architectures when needed.

As a further analysis of whether the efficiency improvements of our systems stem from using DeBERTa or are attributable to the , we compared the speed and memory occupation of a system using as underlying encoder either DeBERTa_large or LongFormer_large.
Our experiments show that using DeBERTa leads to an increase of +77% of memory space and +23% of time to complete an epoch when training on OntoNotes. An equivalent measurement, attributable to the quadratic memory attention mechanism of DeBERTa, was observed for the inference time and memory occupation on the OntoNotes test set. These results highlight the efficiency contribution of the , which is agnostic to the document encoder and can be applied to future coreference systems to ensure higher efficiency.


















































































§ CONCLUSION

In this work, we challenged the recent trends of adopting large autoregressive generative models to solve the Coreference Resolution task. 
To do so, we proposed , a new framework that enables fast and memory-efficient Coreference Resolution while obtaining state-of-the-art results. 
This demonstrates that the large computational overhead required by Sequence-to-Sequence approaches is unnecessary. 
Indeed, in our experiments systems demonstrated that they can outperform large generative models and improve the speed and memory usage of previous best-performing encoder-only approaches.
Furthermore, we introduced , a robust multi-step incremental technique that obtains higher performance in the out-of-domain and long-document setting.
By releasing our systems, we make state-of-the-art models usable by a larger portion of users in different scenarios and potentially improve downstream applications.





§ LIMITATIONS

Our experiments were limited by our resource setting i.e., a single RTX 4090. For this reason, we could not run using larger encoders, and could not properly test Sequence-to-Sequence models as we did with encoder-only models. Nevertheless, we believe this limitation is a common scenario in many real-world applications that would benefit substantially from our system.
We also did not test our formulation on multiple languages, but note that both the methodology behind and our novel incremental formulation are language agnostic, and thus could be applied to any language. 






§ ACKNOWLEDGEMENTS



0.1
        

            
    < g r a p h i c s >

        

    
    
    0.70
         We gratefully acknowledge the support of the PNRR MUR project PE0000013-FAIR.
    
    
    0.1
        

            
    < g r a p h i c s >

        

    





Roberto Navigli also gratefully acknowledges the support of the CREATIVE project (CRoss-modal understanding and gEnerATIon of Visual and tExtual content), which is funded by the MUR Progetti di Rilevante Interesse Nazionale programme (PRIN 2020).
This work has been carried out while Giuliano Martinelli was enrolled in the Italian National Doctorate on Artificial Intelligence run by Sapienza University of Rome. 











§ MULTI-EXPERT SCORERS


In , the final coreference score between two spans is calculated using 6 linguistically motivated multi-expert scorers.
This approach was introduced by <cit.>, which demonstrated that linguistic knowledge and symbolic
computation can still be used to improve results on the OntoNotes benchmark.
In we adopt this approach on top of the .
We use the same set of categories, namely:

    
  * Pron-Pron-C. Compatible pronouns based on their attributes, such as gender or number (e.g.  (I, I), (I, my) (she, her)).  
    
  * Pron-Pron-NC, Incompatible pronouns (e.g. (I, he), (She, my), (his, her)).
    
  * Ent-Pron. Pronoun
    and non-pronoun (e.g. (George, he), (CNN, it), (Tom Cruise, his)).
    
  * Match. Non-pronoun spans with the same content words (e.g. Italy, Italy).
    
  * Contains. One contains the other (e.g. (Barack Obama, Obama)).
    
  * Other. The Other pairs.

To detect pronouns we use string match with a full list of English pronouns. 

To perform mention clustering, we dedicate a mention-pair scorer for each of those categories.
Specifically, for the mention m_i=(x_s, x_e) and its antecedent m_j=(x_s',x_e'), with their start and end token hidden states, we first detect their category k_g using pattern matching on their spans of texts. 
Then we compute their start and end representations, using the specific fully-connected layers for the category k_g:
 
    F_s^k_g(x) = W_k_g,s'(GeLU(W_k_g,sx))

 
    F_e^k_g(x) = W_k_g,e'(GeLU(W_k_g,ex))

The probability p_c^k_g of m_i and m_jis then calculated as:

    p_c^k_g(m_i, m_j) = σ (F_s^k_g(x_s) ·  W_ss·  F_s^k_g(x_s') + 
    
    F_e^k_g(x_e) · W_ee· F_e^k_g(x_e') +  
    
    F_s^k_g(x_s) · W_se· F_e^k_g(x_e') + 
     
    F_e^k_g(x_e) · W_es· F_s^k_g(x_s'))

With W_ss, W_ee, W_se, W_es being four learnable matrices and W_k_g,e', W_k_g,s',W_k_g,e, W_k_g,s' the learnable parameters of the two fully-connected layers.
In this way, each mention-pair scorer learns to model the probability for its specific linguistic categories. 





§ TRAINING DETAILS



 §.§ Datasets




We report technical details of the adopted datasets.

    
  * OntoNotes contains several items of metadata information for each document, such as genre, speakers, and constituent graphs.
    Following previous works, we incorporate the speaker’s name into the text whenever there is a change in speakers for datasets that include this metadata.
    
  * LitBank contains 100 literary documents and is available in 10 different cross-validation folds. Our train, dev, and test splits refer to the first cross-validation fold, LB_0. We report comparison systems results on the same splits. Since training DeBERTa_large is particularly computationally expensive, as introduced in Section <ref>, we train on LitBank by splitting in half each LitBank training document.
    
  * The authors of PreCo have not released their official test set. 
    To evaluate our models consistently with previous approaches, we use the official 'dev' split as our test set and retain the last 500 training examples for model validation.









 §.§ Setup

All our experiments are developed using the Pytorch-Lightning framework.[<https://lightning.ai>]
For each model, we load the pre-trained weights for the base[<https://huggingface.co/microsoft/deberta-v3-base>] and large[<https://huggingface.co/microsoft/deberta-v3-large>] version of DeBERTa-v3 from the Huggingface Transformers library <cit.>.
We accumulate gradients every 4 steps and use a gradient clipping value of 1.0. 
We adopt a linear learning rate scheduler a warm-up of 10% of the total steps check validation scores every 50% of the total number of steps per epoch. 
We select our model upon validation of Avg. CoNLL-f1 score and use patience of 20.









§ ADDITIONAL RESULTS


In Table <ref> we report the performance of models according to the standard Coreference Resolution metrics: MUC <cit.>, B^3<cit.>, CEAFϕ_4 <cit.> and AVG CoNLL-F1.
Scores for models are computed using the official CoNLL coreference scorer.[<https://conll.github.io/reference-coreference-scorers>]






 §.§ Error Analysis

To better understand the quality of predictions, we conduct an error analysis on our best system trained on OntoNotes, . 
In Table <ref>, we report the score of performing only mention extraction (F1) or mention clustering with gold mention (CoNLL-F1) with our systems.
Our results highlight that our models have strong capabilities of clustering pre-identified mentions, but limited performance in the identification of correct spans.
We investigated this phenomenon by conducting a qualitative evaluation of the outputs of our best system, , and found out that OntoNotes contains several annotation errors.
We report examples of errors in Table <ref>. The main inconsistency we found in the gold test set is that many documents have incomplete annotations, which directly correlates with the mention extraction error.










 Paper Title: Event-Arguments Extraction Corpus and Modeling using BERT for Arabic.


Different behaviour of the gas-phase and stellar metallicity in the central part of MaNGA galaxies
    
        I. A. Zinchenko<ref>,<ref> J. M. Vílchez<ref>
       
    
==================================================================================================



§ ABSTRACT 

Event-argument extraction is a challenging task, particularly in Arabic due to sparse linguistic resources. To fill this gap, we introduce the corpus (550k tokens) as an extension of Wojood, enriched with event-argument annotations. We used three types of event arguments: agent, location, and date, which we annotated as relation types. Our inter-annotator agreement evaluation resulted in  82.23% Kappa score and 87.2% F_1-score. Additionally, we propose a novel method for event relation extraction using BERT, in which we treat the task as text entailment. This method achieves an F_1-score of 94.01%.
To further evaluate the generalization of our proposed method, we collected and annotated another out-of-domain corpus (about 80k tokens) called and used it as a second test set, on which our approach achieved promising results (83.59% F_1-score). Last but not least, we propose an end-to-end system for event-arguments extraction. This system is implemented as part of SinaTools, and both corpora are publicly available at <https://sina.birzeit.edu/wojood>






§ INTRODUCTION



Understanding and extracting events from text is crucial in natural language understanding <cit.> for applications like disaster monitoring <cit.>, emergency response <cit.>, insurance decision support, and fostering community resilience <cit.>. Events, a type of named entity mentions <cit.>, are connected to other entities through their arguments. This connection forms the foundation of the event-argument extraction task, which is closely related to relation extraction. By identifying events and linking them with arguments like agents, locations, and dates, we establish meaningful relationships that enhance applications such as information retrieval systems <cit.>, word sense disambiguation <cit.>, and knowledge graph construction <cit.>. Figure <ref> shows how the event "The Economic Forum Meeting" is connected with its three arguments.





Despite the significance of event-argument extraction, there is a notable gap in the availability of comprehensive, annotated corpora for this purpose, especially for under-resourced languages like Arabic <cit.>. To address this gap, we have developed an event-argument extraction corpus specifically for Arabic. We extended the Wojood corpus <cit.>, which is the largest and most recent NER corpus for Arabic <cit.>. We annotated arguments of event entities in Wojood corpus by manually linking each event entity with its corresponding argument entities. As shown in Figure <ref>, three types of arguments are introduced (agent, location, date) for each event. Our Wojood extension (i.e., the event-arguments annotations) is called .

Furthermore, we introduce a novel method for event-argument extraction using BERT and achieved 94.01% F_1-score. Based on , we generated a dataset of premise-hypotheses sentence pairs (we call it ). We used this dataset to fine-tune BERT, as a natural language inference (NLI) task. To test the generalization of our approach, we have constructed an additional out-of-domain dataset (about 80k tokens) called and used it as a second test set. Our model achieves, again, promising results (83.59% F_1-score). Finally, to streamline the event-argument extraction, we propose an end-to-end system specifically designed for the Arabic event-argument extraction task.
In summary, the contributions of this paper are: 

    
  * corpus (550k tokens) manually annotated with event argument relations. This corpus is used to generate an NLI dataset (25k premise-hypotheses pairs).

    
  * , an additional annotated corpus (80k tokens) for out-of-domain evaluation.
    
    
  * Novel methodology for event-argument extraction by framing the task as an NLI problem, achieving high performance.
    
    
  * Novel end-to-end system for event-argument relation extraction.

  
    
All datasets are available online[Datasets: <https://sina.birzeit.edu/wojood/>], and the end-to-end system is implemented and can be used as part of the SinaTools <cit.>[ SinaTools: <https://sina.birzeit.edu/sinatools>].


This paper is organized as follows: Section <ref> reviews previous research. Section <ref> discusses corpus annotations. Section <ref> presents the inter-annotator agreement analysis. Section <ref> outlines the methodology. Section <ref> details the dataset construction, while Section <ref> covers the experimental setup. Section <ref> introduces the out-of-domain dataset. Section <ref> elaborates on the end-to-end system, and Section <ref> explores ablation studies. Finally, Section <ref> concludes the paper and discusses future directions.




§ RELATED WORKS



Events are occurrences or actions that happen over time, involving specific participants and locations. They have temporal components and rely on physical entities to take place <cit.>. Extensive research has focused on event extraction and event argument extraction (EAE), especially in English. However, EAE in Arabic is limited, leaving a gap in the literature <cit.>.

Automated methods, utilizing either statistical algorithms or NLP techniques, are employed to identify relationships between words, using large corpora to detect related terms through their co-occurrence patterns <cit.>. 
For instance, <cit.> proposed a model to extract event entities from Arabic news articles using the GATE tool, employing a five-stage entity identification process that establishes links between events and their corresponding arguments. However, their dataset is small as it consists of only 1,650 sentences. 

In a similar context, <cit.> proposed to use an unsupervised rule-based technique to extract events and the relationship with their associated entities from 1,000 Arabic tweets covering Time, Agent, Location, Target, Trigger, and Product. They linked extracted events and entities to a knowledge base, achieving 75.9% accuracy. This accuracy pertains to the textual representation within the tweet corresponding to the event expression, event type identification (97.7% accuracy), and event time extraction (87.5% accuracy). However, despite these achievements, there remains a need for larger and more diverse datasets to further validate these findings.

Multilinguality has increasingly attracted attention across various fields due to its potential to enhance the understanding and processing of diverse linguistic data <cit.>. This direction has notably influenced relation extraction tasks, as seen with SMILER <cit.>, which aims to improve entity and relation extraction, including Arabic. <cit.> fine-tuned HERBERTa on the Arabic subset of SMILER, achieving high performance in identifying relations and entities despite its smaller dataset of 9k sentences.  This suggests that the unique linguistic features of Arabic may contribute to the model's robustness.
Furthermore, <cit.> introduced valuable resources, namely, SREDFM and REDFM, designed for multilingual relation extraction. SREDFM encompasses over 40 million triplet instances across 18 languages, featuring 400 relation types and 13 entity types. Their mREBEL model, pre-trained on SREDFM, exhibited a remarkable improvement of  15 points in Micro-F_1 compared to HERBERTa.  However, their qualitative error analysis excludes Arabic and Chinese because the authors are not proficient in these languages.

Joint extraction models have become increasingly important in NLP. Addressing the need for improved event extraction in Arabic text, <cit.> introduced a joint model for event extraction in Arabic text using the ACE 2005 dataset. They used a graph-based representation to extract entities, relationships, and event triggers, along with their arguments. This approach led to improved Arabic NER accuracy through different experiments and tokenization schemes. 

In the realm of biomedicine, <cit.> introduced the NBR (NLI improved Biomedical Relation Extraction) method, which verbalizes relations in natural language hypotheses, allowing the model to utilize semantic information effectively in prediction, even with limited data. In contrast, <cit.> addresses cross-lingual EAE challenges through their innovative LanguAge-oriented Prefix-tunIng Network (LAPIN) approach. LAPIN utilizes a language-oriented prefix generator module to handle language variations and a language-agnostic template constructor module to create adaptable templates. Experimental results demonstrate LAPIN's outperforming, achieving an average F_1 improvement of 4.8% and 2.3% on two multilingual EAE datasets compared to the previous state-of-the-art models. These efforts collectively contribute to advancing the field of EAE in various languages, including Arabic.





§ DATASET AND ANNOTATION





 §.§ Corpus Preparation

To construct an event-argument corpus, we extended the existing  Wojood corpus <cit.>. This is because it is a rich Arabic nested named entity corpus comprising 550k tokens, supporting 21 distinct entity types, including categories such as person, organization, location, event, and date. Notably, it incorporates 2,772 annotated events. According to Wojood guidelines, a mention is annotated as an event if it represents an occurrence of general interest, like battles, wars, sports events, demonstrations, disasters, elections, and national or religious holidays.


Our objective in this paper is to identify event arguments and establish relationships between these arguments and the respective event entities, as illustrated in Figure <ref>. 



 §.§ Annotation Process


Initially, we assigned a unique identifier to each entity in Wojood, see Figure <ref>(a). Then, we linked these entities with relationships, see Figure <ref>(b).







  §.§.§ Relationship Types


We propose to use these relations:

    
  * hasAgent: specifies participant(s) involved in the event, which can be a PERS, ORG, OCC, or NORP named entities.
    
  * hasLocation: indicates where the event occurred, which can be GPE, LOC, and FAC named entities.
    
  * hasDate: points when the event occurred, which can be TIME or DATE. 






 §.§ Annotation Guidelines


We propose the following guidelines to annotate the corpus:

    
  * Event arguments are recognized only within the same sentence.
    
    
  * Entities with different entity IDs are considered distinct entities. For example, in the sentence (مقتل الرئيس المصري أنور السادات / The killing of the Egyptian president Anwar al-Sadat), the entity (الرئيس المصري/the Egyptian president) and the entity (أنور السادات / Anwar al-Sadat) are regarded as separate entities, thus two agents in the event. However, in reality, they refer to the same individual.
    
    
  * The same event may have multiple agents, as in ( توقيع اتفاقية تعاون بين الحكومة اللبنانية و البنك المركزي   / Signing a cooperation agreement between the Lebanese government and the Central Bank). In Figure <ref>, (الحكومة اللبنانية / the Lebanese government) and (البنك المركزي / the Central Bank) are both agents of the same event.
    
    
  * In the same sentence, two different event entities can share the same argument. For example, in the sentence (الوضع السياسي متوتر في مصر بعد حرب عام 1967 (النكسة)/The political situation in Egypt is tense after the 1967 War (Al Naksa)), there are two event entities, according to Wojood guidelines. The first event is (حرب عام 1967/The 1967 war), and the second event is ((النكسة)/Al Naksa). In this way, the entity (عام 1967/the year 1967) is considered the argument for both events in the sentence.








 §.§ Corpus Statistics

Wojood comprises 550k tokens with 2,772 event entities.
Among these entities, we annotated 1,974 events with event-arguments relations - the other 798 do not have arguments.  Notably, there are 355 (18%) events that are annotated with at least two arguments, and 77 (18% of agent relations) annotated with multiple agents. This underscores the corpus’s rich and diverse interconnection of events and their roles.
The number of instances for each relation type is shown in Table <ref>.





§ INTER-ANNOTATOR AGREEMENT



To evaluate the quality of our annotations, we randomly selected 5% of the annotations and asked our two annotators to annotate in parallel. We computed the Inter-annotator Agreement (IAA) using Cohen's kappa and F_1-score. The results in Table <ref> illustrate high agreement. 

 


 







 §.§ Calculating kappa 

We assessed annotator agreement for three relations. Agreement happens when both annotators assign the same relationship type. The kappa coefficient <cit.> is defined as:

 
    K = P_0-P_e/1-P_e
 
where P_o is the observed agreement, and P_e is the expected agreement. P_e is calculated as :

    P_e = 1/N^2∑_Tn_T1×
        n_T2



 §.§ Calculating F_1-score

The F_1-score for a specific relation (e.g., hasAgent) is calculated using Equation <ref>. True positives (TP) are those when annotators agree, while false negatives (FN) and false positives (FP) arise from disagreements. FN arises from the first annotator's disagreement, while FP arises from the second annotator's disagreement.

    F_1-Score = 2TP/2TP+FN+FP




 §.§ Discussion and Annotation Challenges
 

We encountered several challenges, including: 


  * Although event arguments should be encompassed within the sentence referencing the event, it was not always easy for the annotators to decide if entities within the same sentence as the event should be annotated as event entities.


  * In the hasAgent relation, it can sometimes be challenging to determine whether an entity serves as an agent for the event. See the example in Figure <ref>.




  * The annotators have different levels of experience; one is skilled at defining event arguments, while the other is not. Consequently, one annotator agreed to designate a specific entity as the agent, while the second did not.






§ EVEN-ARGUMENT EXTRACTION (EAE)
 

In this section, we provide an in-depth explanation of our approach to addressing the EAE task through the NLI task. Figure <ref> illustrates our framing of the EAE problem.






 §.§ Problem Formulation

The objective of the EAE task is to identify the relationships between the event(s) in a sentence and the named entities mentioned in the same sentence. That is, given a sentence s annotated with a set of event entities E={e_i}_i=1^n and a set of other named entities N={n_j}_j=1^m, the goal of EAE is to identify the relation r∈ R for each pair (e_i,n_j) in s, where R∈{ hasAgent, hasLocation, hasDate}. 




 §.§ Event Relation Extraction as NLI


We propose to solve the EAE by framing it as a Natural Language Inference (NLI) task. 
In NLI, we assess whether one sentence (the premise) entails another (the hypothesis).
That is, a pair of sentences is classified as True or False. 
To extract an event argument relation from a sentence (See Figure <ref>), we treat the original sentence as a premise and generate the hypothesis automatically. The hypothesis is another sentence generated using a template, to represent a possible relationship. 

In other words, we propose to treat EAE as a binary NLI task focusing on entailment. The input sentence s is the premise, while the hypothesis is a verbalized template representing a relation r between event e_i and a named entity n_j  mentioned in the same sentence. The model then determines if the premise "entails" the hypothesis (as classification True/False), indicating the existence of the relation r between e_i and n_j.

 




  §.§.§ Template Construction




For each of the three relations, we designed a template parameterized over the named entities mentioned within the same sentence. Each template has two placeholders: Event Placeholder P_event and Entity Placeholder P_entity. These placeholders are filled with event mention e_i and named entity mention n_j, respectively.
Figure <ref> illustrates how a template can be used to generate a hypothesis for the hasDate relation between the (الحصار الشامل على قطاع غزة / Total siege on Gaza strip) event and the
(9 أكتوبر 2023 /October 9, 2023) entity. The three templates used are shown in Table <ref> 








  §.§.§ Sentence Encoder


The sentence encoder is used to extract representations for input text. In our approach, the input text is a pair consisting of a premise (the input sentence s) and a hypothesis (the filled-in template h).  A transformer encoder, denoted by T, is used to process the input sequence and derive its representation, T →ℝ^d. This encoder effectively captures contextual information and semantic relationships within the text pair generating representation H for the sentence pairs. Formally,


    H=T ( [CLS] s [SEP] h )


where [CLS] is the special classification token, [SEP] is a separator between the premise and hypothesis, and d is the encoding dimension.



  §.§.§ Relation Classifier


The feature vector H from the sentence encoder is input into a fully connected layer. This layer calculates the predicted probability that the premise entails the hypothesis, with a True result indicating that the relationship r specified in h exists between the event e and the entity n. Formally,

    ŷ_i=σ ( H W + b  )

where ŷ indicates whether the hypothesis holds a positive relation, while W and b represent a weight matrix and a bias term, respectively.
 



  §.§.§ Training Objective


Our training objective is to prioritize the accurate identification of positive instances over negatives. To accomplish this, we utilize a weighted cross-entropy loss L_WCE (Eq. <ref>), which penalizes misclassifications of positive instances more heavily. Additionally, we use Noise Contrastive Estimation Loss L_NCE (Eq. <ref>)  to improve the discrimination between positive and negative instances <cit.>. The final loss function is expressed as Loss = L_WCE + L_NCE.




    L_WCE   = -1/N∑_i=1^N( w_p y_i log(ŷ_̂î) 
         + w_n (1 - y_i) log(1 - ŷ_̂î) )



    L_NCE = -1/N∑_i=1^Nlog( exp(s(y_i) / τ)/∑_jexp(s(y_j) / τ))











§ CONSTRUCT DATASET


Based on our annotated corpus, we created an NLI dataset (called ), which is a dataset of premise-hypotheses sentence pairs that we use to train the EAE model. 

To create this dataset, we carried out the following  steps:  



  §.§.§ 1. Premise Sentences Preparation

We extracted sentences from the corpus and utilized them as premises. These sentences were then split into training (70%) and testing (30%) sets. This split is important to avoid overlap between premises in the two splits during the generation of positive and negative pairs. 




  §.§.§ 2. Hypothesis Sentences Preparation


Multiple hypotheses are generated for each premise sentence, utilizing the entities mentioned within the sentence. These entities are annotated as arguments of a specific event in the corpus. That is, based on the event-argument relation annotations (see section <ref>), each event and its arguments are used to generate a hypothesis using the template for relation r. This hypothesis is then paired with the premise sentence to form a positive pair.





  §.§.§ 3. Generating Negative Pairs

In premise sentences, entities that are not linked with events (i.e., not event arguments) are used to generate negative pairs.


That is, we link events with entities that are not their correct arguments (i.e., not annotated as a relation in ), and generate them as hypotheses using templates. Each generated hypothesis is paired with its premise and used as a negative pair.



 §.§ Dataset Statistics

For the training phase, four templates are designed for each relation, each with a particular verbalizer (see Table <ref>). This step aims at data augmentation and enhancing the contextual diversity of the training set.  For the test phase, one template per relation was selected from the training templates to generate the hypothesis (see Table <ref>). 
The final dataset consists of 25,473 pairs, comprising 10,478 positive and 14,995 negative pairs. Table <ref> presents more statistics.









§ EAE MODELING EXPERIMENTS




 
The is used to train an EAE model.



 §.§ Training Hyperparameters



During training, we employed a k-fold strategy with k=5 to ensure the robustness of model evaluation. For all experiments, we utilized the  <cit.> as the text encoder model, with a learning rate of 2e^-5 and the AdamW optimizer with a weight decay of 1e^-8. In the L_WCE loss function, w_pos=1 and w_neg=.5, while τ = 1 in L_NCE loss function.





 §.§ Results and Discussion


Table <ref> shows the obtained results using the aforementioned experimental setups. We average the scores across five folds and report the model with the best average F1 score. Table <ref> provides insight into the NLI model performance in each relation. Note that this NLI performance represents the accuracy of our model in classifying sentence pairs, which is not the EAE accuracy. The accuracy of event-argument relation extraction (EAE) shall be presented in section <ref>.

Notably, our NLI model achieves high F1 scores across all event relations, with the hasDate relation exhibiting the highest score (96.44%). 

Additionally, the negative relations also achieved a notably high F_1 score of 95.29%, indicating the model's effectiveness in recognizing entities unrelated to the event.



To validate this remarkable performance of the model and to test its generalization, we constructed a new corpus and conducted additional out-of-domain experiments. 











§ ADDITIONAL DATASET


To evaluate the model's generalization and its robustness to contexts beyond the domains, we constructed a new out-of-domain corpus (referred to as the corpus). We then extracted an NLI dataset and used it for model testing.



 §.§ Corpus Preparation

This corpus covers 10 distinct domains: economics, finance, politics, science, technology, art, law, agriculture, history, and sports each containing nearly 8k tokens. The corpus covers events from 2010 to 2022, manually collected from news websites such as Aljazeera, and Alarabiya, totaling 80k tokens. 

We used the same Wojood annotation guidelines to maintain consistency.  Table <ref> shows the number of instances for each relationship type.





 §.§ Construct Dataset


An NLI dataset was generated from using the same methodology as , employing only the templates designated for testing, resulting in a total of 1124 pairs. Detailed statistics are provided in Table <ref>.

 





 §.§ Experiments and Results


The is used to evaluate the EAE model, which was trained on the . The results, presented in Table <ref>, highlight the model's robust generalization across diverse domains, despite encountering challenges like domain-specific vocabulary and linguistic nuances. Despite a slight performance decline compared to the test set, the model maintained a high overall average F_1-score of 83.38% on the .












§ END-TO-END SYSTEM FOR EAE




 §.§ System Architecture


This section introduces our novel end-to-end EAE system, which efficiently extracts event-related information from text by seamlessly identifying entity boundaries, determining their types, and recognizing argument entities and their relations to an event entity. We utilized the EAE NLI model to construct the system. As illustrated in Figure <ref>, the overall steps are:

Named Entity Recognition Module: The system starts by extracting entities and their types from the input sentence. The process is carried out through the online Wojood web service[<https://sina.birzeit.edu/wojood>]. Then, the system evaluates the extracted entities. If an event entity is recognized in a sentence, then other entities in this sentence are considered candidate arguments for this event.  

Template Selection Module: For each entity, a template is selected based on its category and used to construct the hypothesis. 

Argument Extraction Module: The input sentence is paired with the template and sent to the EAE NLI model to identify the argument entities for an event and their corresponding relationships. The type of template serves as the basis for establishing the relationship between the event and the entity. Specifically, if the EAE NLI model indicates a positive connection between the input sentence and the template, then the entity and the event linked within the template are considered to have a relationship indicated by the template label.



 §.§ Computing EAE baselines
 
The performance of our proposed EAE system is evaluated using and test sets. Results are shown in Table <ref>. Note that the relation classifiers share parameters, and the evaluation assumes named entities to be correctly recognized.









§ ABLATION STUDEIS


Best Template: to choose the best template to implement in the EAE system for each relation (Figure <ref>), we evaluated them using the test set. Table <ref> shows that t_2 performed slightly better.



 Best loss function: We compared two loss functions: cross-entropy loss (L_CE) and Noise Contrastive Estimation Loss (L_NCE).  Table <ref> shows that while L_NCE slightly outperforms L_CE on the , its performance significantly improves on , showing its efficacy across diverse contexts.

Additionally, we improved the model's performance by combining weighted cross-entropy loss (L_WCE) with L_NCE, using weights w_p=1 and w_n=0.5. This approach, with higher weights for positive relations and slightly lower weights for negative ones, yielded the best results.







§ CONCLUSION AND FUTURE WORK


Our introduction of the and corpora significantly advances Arabic event-argument extraction by providing a rich dataset with high inter-annotator agreement. Our novel BERT-based method for event relation extraction demonstrates exceptional performance, achieving high F_1 scores on both the dataset and on dataset. Additionally, our implementation of the EAE end-to-end system as part of the open-source SinaTools will enrich the Arabic NLP industry.
  
Large language models (LLMs) can further enhance our work by extracting information, including named entities and relationships, from text to populate knowledge graphs and improve other knowledge graph tasks like embedding and completion <cit.>. In future work, we will explore the capabilities of LLMs to enhance event-argument extraction. Integrating LLMs into our framework could potentially improve the accuracy and scalability of our event extraction system.



§ LIMITATIONS
 

The constructed and corpora primarily focus on MSA data and do not cover dialectal variations. Furthermore, even though we included out-of-domain tests to assess performance, our results are constrained to the specific domains used in our study.




§ ETHICS STATEMENT
 
The corpora provided for this research are derived from public sources, eliminating specific privacy concerns. The results of our research will be made publicly available to enable the research community to build upon them for the public good and peaceful purposes. Our data, tools, and ideas are strictly intended for non-malicious, peaceful, and non-military purposes.




§ ACKNOWLEDGEMENTS

This research is partially funded by the research committee at Birzeit University. We extend our gratitude to Taymaa Hammouda for the technical support and to the students who helped and supported us during the annotation process, especially Haneen Liqreina and Sana Ghanim.

 



 Paper Title: Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent.


UTF8gbsn
[colback=gray!20, colframe=gray!20,]

    0.1in
    Towards Achieving Human Parity on  End-to-end Simultaneous  Speech Translation via LLM Agent
     
    0.05in
    Cross Language Agent Team  
    0.05in
    ByteDance Research 
    0.05in
    July 20, 2024  
    



§ ABSTRACT 
  In this paper, we present Cross Language Agent - Simultaneous Interpretation, , a high-quality and human-like Simultaneous Speech Translation (SiST)[In this paper, we use Simultaneous Interpretation and Simultaneous Speech Translation interchangeably.] System.
  Inspired by professional human interpreters, we utilize a novel data-driven read-write strategy to balance the translation quality and latency.
  To address the challenge of translating in-domain terminologies, employs a multi-modal retrieving module to obtain relevant information to augment the translation.
  Supported by LLMs, our approach can generate error-tolerated translation by considering the input audio, historical context, and retrieved information.
  Experimental results show that our system outperforms other systems by significant margins. Aligned with professional human interpreters, we evaluate  with a better human evaluation metric, valid information proportion (), which measures the amount of information that can be successfully conveyed to the listeners. 
  In the real-world scenarios, where the speeches are often disfluent, informal, and unclear, achieves of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese translation directions, respectively. In contrast, state-of-the-art commercial or open-source systems only achieve 35.4% and 41.6%. On the extremely hard dataset, where other systems achieve under 13% , can still achieve 70% . Demonstrations and human-annotated test sets are available at <>.






§ INTRODUCTION


Simultaneous speech translation (SiST) is recognized as one of the most challenging tasks in the translation domain <cit.>. 
Machine-assisted automatic interpretation has been receiving much attention in the natural language processing (NLP) community <cit.>. 
Traditional simultaneous translation approaches <cit.> 
usually employs a cascaded system, involving a streaming Automatic Speech Recognition (ASR) model, a punctuation model and a Machine Translation (MT) model.
However, such cascaded systems often suffer error propagation and latency from the ASR module. 
Despite these advancements in both academic SiST models <cit.> and commercial SiST engines, the translation quality is still far from satisfactory. 
As shown in <Ref>, we conduct a human assessment of the current accessible SiST systems. From the user-centered perspective, these systems only deliver less than 42% of the valid information to listeners, which heavily affects communication effectiveness. 
In contrast, professional human interpreters usually deliver more than 70% of the necessary information <cit.> and 95% ideally. Thus in this paper, we use 80% to indicate high-level human interpreters. 

Motivated by the huge success of LLMs in machine translation <cit.> and speech translation <cit.>, we propose to employ LLMs to accomplish the SiST task.
Specifically, we identify three primary challenges.
First, a key challenge for incorporating LLM into the SiST is the read-write policy, where LLM needs to provide partial translation for input speech. 
Second, achieving human equivalent performance requires understanding and translation of terminologies and uncommon phrases that LLMs cannot learn from training data. 
Lastly, the scarcity of training data continues to hinder the performance on the SiST task.


To address these challenges, we introduce our end-to-end approach, , a Cross-Lingual Agent that accomplishes Simultaneous Interpretation by iteratively performing multiple actions, as illustrated in Figure <ref>. 
Regarding the first challenge, we imitate professional human interpreters to learn their policy of segmenting a complete sentence into several semantic “chunks” through syntactic boundaries (pauses, commas, conjunctions, etc.) and contextual meaning. 
To enable to learn such a policy, we follow a data-driven policy learning process and invite human interpreters to annotate real-world speech, which includes the read-write timing for segmentation. From the data,  learns the robust read-write policy for SiST from humans.


For the second challenge, we include two external modules to augment our  agent: an external knowledge database that stores terminologies and paired translations, and a memory that stores the context of speech. 
However, the external knowledge database may contain tremendous terms that not only increase the inference time but may also lower the performance of our approach because of noisy intervention. Therefore, we propose a novel Multi-Modal Retrieval Augmented Generation (MM-RAG) process. 
A multi-modal retriever extracts knowledge from the external database based on the speech input. 
The retrieved information and the context from memory are then appended to the prompt of our LLM agent to augment the translation through in-context learning.

Addressing the data scarcity of the SiST task, we adopt a three-stage training methodology: pretraining, continual training, and fine-tuning. First, our LLM and audio encoder are independently pretrained on our large-size in-house datasets. 
Then, our model is continually trained with billions of tokens of mediocre-quality synthesized speech translation data, aiming to align the speech and text modalities.
We also include multiple tasks to enhance the in-context learning ability of LLM to better utilize the contextual information from the retriever and prior translation.
In the last stage, we fine-tune the model with a small amount of human-annotated data, further imitating professional human interpreters to improve the robustness and translation quality.



In addition, we would like to highlight that the conventional automatic evaluation metrics <cit.> of simultaneous interpretation might not be good indicators for reflecting the performance of SiST, which often contains compaction, abstraction, and paraphrasing. 
Aligned with human interpreters <cit.>, we propose a new evaluation metric named Valid Information Proportion ()[Detailed guidelines of our proposed metric can be found in Appendix <ref>.]. VIP represents the percentage of information that can be precisely delivered, reflecting the central objective of SiST: communication in real-time.
Through thorough human evaluation on diverse and challenging real-world long speech datasets, our approach outperforms other currently accessible systems by a large margin.
As shown in Figure <ref>, taking the Chinese-to-English direction as an example,  achieves a score of 81.3%, significantly narrowing the gap between machine-assisted systems and human interpreters.


Our contributions can be summarized as follows:

    
  * We introduce our end-to-end approach, , an LLM agent that is designed to perform high-quality and human-like simultaneous translation. Through human evaluation, our approach demonstrates significantly better performance compared to existing accessible SiST systems.
    
  * We propose a new data-driven read-write strategy by imitating professional human interpreters. Without the requirement of complicated human pre-design, the strategy could balance translation quality and latency effortlessly. Unlike most commercial systems where the outputs are frequently rewritten during the translation process for better quality, our strategy guarantees all the outputs are deterministic while maintaining high quality.
    
  * Motivated by the preparatory trajectory of human interpreters, we introduce a novel Multi-Modal Retrieval Augmented Generation (MM-RAG) process that empowers the LLM with domain-specific knowledge in real time. The proposed module
    further improves the translation quality with minimal computational overhead during inference. 
    
  * We work closely with professional human interpreters to develop our evaluation strategy, Valid Information Proportion (), and detailed guidelines are open-sourced. Meanwhile, we release a human-annotated test set focusing on diverse real-world scenarios and long speech translations.








§ METHODS






 §.§ Framework

<Ref> presents a flow of operation of . To perform the SiST task, we design 5 operations: , , ,  , and . The following sections describe the details of each operation.
As further illustrated in <Ref>,  is an LLM agent that can take input speech, instruction, relevant information retrieved from external knowledge, and last round memory as context. The memory stores previous transcriptions (optional) and translations. At round r, it first reads speech _t^r-1:T^r, where t^r-1 is the predicted cut-off time of round r-1 and T^r is the end time for audio stream at round r. 
Then the agent retrieves relevant information _r from the external knowledge and loads context _1:r-1 from the last round memory. 
Once “think” sufficient context is loaded, it generates the transcription (optional), translation, and cut-off timestamp t^r: 

    _r; t^r = TextDecoder(_t^r-1:T^r, _r, _1:r-1)

where t^r is the predicted cut-off timestamp indicating the end time for the current translation round r. 
_r is then forwarded to update the memory.
When instructed to output the transcription, the LLM optionally engages CoT to generate transcription first and then the speech translation.
For the following round r+1, the audio stream begins with the predicted cut-off timestamp t^r.


 §.§ Architecture

 employs an Encoder-Conditioned LLM architecture. As shown in <Ref>, the audio encoder transforms input speech stream  to a series of continuous representations . 
Then, the LLM takes the speech representation , retrieved knowledge , historical translation  and instruction  as a sequence of prompt (, , , ) to generate the translation result . 

Audio Encoder. The audio encoder module contains a large-scale speech conformer <cit.> pretrained on millions of hours of speech data to achieve human parity performance on ASR, and an audio adapter to connect the audio encoder and LLM. 
The adapter downsamples the speech representations and the resulting representations are linearly projected to match the dimension of the LLM embedding layer. 
The projected speech representations lower the computational latency for SiST.

Large Language Model. The language model[We use Doubao LLM as our foundation model.] is a medium size decoder-only transformer <cit.> to balance performance and computation efficiency. It is pretrained on a large amount of text data and fine-tuned with instructions. The LLM directly takes the continuous embedding from both the audio encoder and text embedder as input. It autoregressively generates the transcription and translation response of the provided speech stream. 

Multi-Modal Retriever. 
The multi-modal retriever framework employs audio and text encoders to independently encode the audio stream and text key of the terminologies in the external knowledge database.
To enhance the alignment between audio embeddings and text embeddings, we incorporate an embedding fusion layer, which includes a multi-head attention module followed by a pooling layer.
The resulting pooled representation is subsequently fed into a linear projection layer to produce the final scores, indicating the probability of the text key's presence in the audio stream.
Terminologies with top scores are forwarded to the  agent to enhance the translation quality.









 §.§ Data Driven Read-Write Policy:  and 


Unlike predetermined read-write probabilities and heuristic waiting policies detailed in prior research <cit.>, interpreters engage in a dynamic process of listening (read) and translating (write).
They attentively listen to the speaker's speech and segment lengthy sentences into semantic chunks, representing the smallest linguistic units capable of conveying a complete thought independently <cit.>.
Upon identifying a chunk that encapsulates sufficient information, they proceed to translate this segment into the target language, thereby providing an accurate and contextually appropriate translation.

Emulating the strategies of human interpreters,  does not require to explicitly define the read-write policy.  imitates their policies by waiting for complete semantic chunks. Specifically, given partial speech,  only generates the translation for the complete chunks of the input speech. The model is trained with segmented speech data to learn such ability.
Mathematically, given source audio _1:M, we segment its translation into a series of n “chunks” _1:n and obtain the corresponding pair {(_t^j:t^j+1, _j}_j=1^n, where _t^j:t^j+1 and _j denote the j-th segment of the audio and the corresponding translation. For training, our objective is to output all complete segmented translations and the cut-off time given random partial input audio _1:t

    min𝔼_t ∼𝒰[1, M] -log p_θ(_1:j; t^j|_1:t)    j = max_j{j|t^j < t}

where 𝒰 indicates uniform distribution over time of speech. 
Trained with <Ref>, learns to generate the cut-off time for the input speech. Additionally, the objective function makes the wait for appropriate time before starting translation as the LLM will output nothing when it “think” current speech stream does not contain a complete speech chunk.



 §.§ Context Information:  and 


The memory stores translations and transcriptions in previous rounds _1:r-1. It has two functions. Firstly, it works with the input speech to determine which part of the speech has been translated and which part has not, helping make the read-write decisions and outputs the translation of the unfinished parts. 
Secondly, understanding human speech often requires context. For example, when a speaker talks about “barrel bridge”, it often refers to the bridges built upon rivers that are supported by barrels. However, in the context of “watch”, it refers to a mechanical structure in the watch. The phenomenon of polysemy in different contexts can lead to vastly different translation outcomes. Therefore,  should be able to retrieve the context of the long speech for translating some keywords, and make appropriate translations under different contexts. 

As shown in <Ref>, at round r,  forwards relevant translations _1:r-1 to the LLM as a prompt. After agent generates the translation _r,  stores it to the memory and obtains _1:r.



 §.§ Multi-Modal Retrieval Augmented Generation: 


In real-world scenarios, the accurate speech transcription or translation of professional and domain-specific terminologies is challenging. Even human interpreters require prior domain knowledge to understand those terminologies, including names of people, locations, jargon, or special in-domain terms.
For example, an interpreter unfamiliar with the machine learning theory may not recognize the word “Rademacher complexity” when hearing it. 
Therefore, in various scenarios, human interpreters often prepare in advance to get familiar with the corresponding domain knowledge.

Motivated by the preparatory trajectory of human interpreters, we propose to integrate an external database to empower LLM with necessary domain-specific knowledge. Each item in the database contains a key and the corresponding value in text modality. The key, which may appear in the speech, is used as the input for the retriever. The value of the item may be itself, a paired translation of the target language, or even an explanation of the key. 

Theoretically, all items in the external database might be added into the prompt to provide information for the translation. However, the external knowledge database often contains tremendous items.
Simply prompting LLM with all the terms not only increases the
inference time but may also hurt the performance of because of noisy intervention. 
Therefore, we design a novel Multi-Modal Retrieval Augmented Generation (MM-RAG) process.
Our multi-modal retriever first retrieves the relevant terminologies from the database based on the input speech.
A small number of filtered items are incorporated into the prompt of  agent for in-context learning as shown in <Ref>. 

With the retrieved knowledge and previous context from the memory, our LLM has the in-context learning ability to better utilize the provided contextual information.
To achieve this, we collect a series of in-context learning data to train the model. Compared with the previous approaches for intervention, such as shallow-fusion <cit.> and traditional substitution-based methods <cit.>, which generates fixed translation for given translation pair. Our method achieves better results and generates more coherent text. For example, in some internet companies, “大盘 == overall performance”, while in most cases, it should be “stock market”. Our method can choose the correct translation given different context. Besides, our method can use monolingual text from both source and target language to help the translation.




§ MULTI-STAGE TRAINING

Our  follows a multi-stage training process: pretraining, multi-task continual training, and multi-task supervised fine-tuning.  
In the first stage, the LLM and audio encoder are separately pretrained with massive amounts of in-house speech and text data. 
Next, a large amount of speech-text paired data is used to align audio and text modalities, building the fundamental capability for cross-modal multitasking. 
In the final stage,  agent is fine-tuned with a small amount of human-annotated data to imitate the translation behavior of professional human interpreters.
Our multi-stage training process enables high efficiency of learning with a small amount of human-labeled data.



 §.§ Pretraining

The in-house LLM and audio encoder are independently pretrained on different modalities of data.
The LLM follows a decoder-only transformer architecture and is first pretrained on a massive amount of monolingual and bilingual text data with cross-entropy loss and then fine-tuned on instruction-following data. The LLM performs excellently on various downstream tasks, especially translation tasks.

The audio encoder also follows a classic pretrain-finetune paradigm <cit.> with a massive amount of speech-related data. 
The pretraining stage provides a proficiently trained LLM and audio encoder, setting a solid foundation for the following stages. 




 §.§ Multi-task Continual Training

Training Method.
We follow the work of <cit.> for multi-task training. Specifically, for streaming and higher-quality translation, we mainly focus on three tasks for training : Automatic Speech Recognition (ASR), Speech Translation (ST), and Text Translation (MT). 
To align the modalities of the pretrained LLM and audio encoder, 
 is continually trained on various tasks with a substantial volume of paired data. 
We further strengthen the in-context learning ability of our approach by incorporating translation in the memory and knowledge from external databases.
As a result, we expand the ST tasks to different configurations as shown in <Ref>. An ST translation can either be streaming or offline, direct or COT, with or without context, which leads to 8 different tasks.




Training Data Construction.
Both ASR and MT tasks have been last for a while, and there is a relatively large amount of ASR and MT data.
The major challenge of developing an end-to-end SiST model is the data scarcity of simultaneous ST. 
To this end, we propose a synthetic data construction pipeline. With a strong LLM, we synthesize two types of speech translation data for continual training: offline ST data and context-aware segmented streaming ST data.

1. Offline ST data: We mainly rely on ASR data to construct the offline ST data. Given the ground-truth transcription of speech, we use in-house LLM to translate the source language to target languages.
To ensure the readability and conciseness of the target language, the LLM is prompted to conduct Inverse Text Normalization (ITN), filler word smoothing, etc.

2. Context-aware segmented streaming ST data: The streaming ST data consists of fine-grained audio-text alignments and translation pairs for segmented semantic chunks.
Compared to offline ST data, streaming ST data
is even more challenging to collect. 
We find that human interpreter often segments long speech into a few semantic chunks, each of which can be translated independently to ensure an effective and smooth translation.
Motivated by such findings, we leverage LLM to construct streaming ST data by imitating the chunking process.
Long speech data are used to construct the streaming ST data, as the additional history can provide better contextual information.
First, we prompt the LLM to break down the ASR transcription into multiple independent semantic chunks, which are then translated into the target language. 
Subsequently, we align the semantic chunks with the corresponding audio chunks, obtaining the streaming ST data. 
Such data enable our model to handle incomplete speech inputs and generate partial translation in coherent semantics.

To measure the quality of the synthetic data, we conduct human evaluations based on our proposed metric. 
The synthetic data achieves a score of 81%, satisfying the minimal requirement for further training. 



 §.§ Multi-task Supervised Fine-tuning

Training Method.
Even though  possesses a good translation quality on the SiST tasks after the previous multi-task continual training stage, we further boost the performance by fine-tuning on human-annotated streaming ST data with diverse tasks listed in <Ref>. 
Such high-quality data enables our model to better align with the segmentation methodologies of professional human interpreters.
Furthermore, this process enhances our model's robustness to speech disfluencies such as stuttering, ensuring smoother communication in real-world scenarios.

Training Data Construction.
The source of human-annotated streaming ST data originates from real-world scenarios that contain various speech characteristics, such as disfluencies, stuttering, code-mixing, and specialized terminologies.
Such features ensure the robustness of our model in diverse conditions.
We engage professional human interpreters to provide high-quality annotations for simultaneous segmentation and interpretation of the speech data. 
Additionally, terminologies are identified and translated within the context, further strengthening the context-aware capabilities of . 



 §.§ Multi-Modal Retriever Training

The multi-modal retriever is independently trained with a substantial dataset of speech recognition data. 
During training, words are randomly selected from speech transcription to serve as the positive sample, indicating their appearance in the speech.
Negative words are selected from different sentences, indicating the speech does not mention these words.
We assign a label of 1 to positive samples and 0 to negative samples, aiming to minimize the Binary Cross Entropy (BCE) loss. This approach helps refine the model's ability to distinguish relevant from irrelevant information, enhancing its overall performance and accuracy.
We label the positive sample as 1 and the negative sample as 0, minimizing the Binary Cross Entropy (BCE) loss.
To evaluate the effectiveness of our retriever, we build an in-house retrieve development set. Each sample in the development set includes a short audio chunk and the mentioned terms in the audio. 
Note that the term here is defined as special keywords, such as name, location, abbreviation, and domain-specific word.




§ EXPERIMENTS




 §.§ Evaluation Benchmark

Quite a few number of evaluation benchmarks have been proposed for SiST over the past years, including MuST-C <cit.>, FLUERS <cit.>, CoVoST <cit.>, BSTC <cit.>, and GigaST <cit.>, etc. 
However, although much effort has been spent to build these benchmarks, they still suffer some shortcomings when facing real-world SiST applications. 

First, these benchmarks often contain speeches that are either recorded by volunteers (e.g. CoVoST and FLUERS) or collected from formally, clearly, and fluently talk and podcasts by well-prepared speakers (e.g.  MuST-C and GigaST). In real-world scenarios such as online meetings or social media videos, the characteristics of the speech might inevitably be informal, unclear, or disfluent. 
Second, these benchmarks provide a shortcut for evaluating the translation quality by giving the manually segmented sentences. Such a shortcut offers a gap between the current benchmark and real-world applications, where the models might need to take long speech and conduct segmentation <cit.> by themselves. Consequently, evaluations on manually segmented datasets are likely to overestimate the performances of a real-world SiST system.
These discrepancies result in the evaluation on these benchmarks are not reliable for practical SiST systems. 

As a preliminary attempt to address the shortcomings as mentioned earlier, we propose a new benchmark for Chinese-to-English (zh-en) and English-to-Chinese (en-zh). is collected from diverse sources, and most speakers talk naturally and casually without careful preparation.
We choose 10 popular domains: technology, healthcare, education, finance, law, environment, entertainment, science, sports, and art.
One video clip is selected for each domain from a well-known online video platform for both zh-en and en-zh settings.[is available at <https://github.com/byteresearchcla/RealSI>. We do not own the copyright of the videos and only release our annotations together with the publicly available website links of the corresponding videos. If anyone believes that the content constitutes infringement, please contact us. We will remove the relevant content as soon as it is confirmed.]
Each sample in is a nearly 5-minute speech to mock SiST without manual segmentation.
For systems that cannot take long-form audio as input, we also provide sentence-level timestamps for segmentation. 
<Ref> presents the detailed statistics of our .






 §.§ Baselines

We compare with the open-sourced SiST model, SeamlessStreaming <cit.>. In addition, because of the limited number of available SiST models, we choose to compare  with several commercial systems. We denote the commercial systems as Commercial 1-4. It is worth noting that unlike , most of the commercial SiST systems will first generate a temporary translation as soon as possible, then rewrite the temporary translation with a potentially better translation after getting more context.
Notwithstanding, we evaluate the finalized translation of these systems in all our experiments. 
Although this re-writing strategy could improve translation quality, continually revising existing translations might affect the user experience, potentially leading to additional confusion. 
We would also like to highlight that human interpreters usually do 
 not employ such a rewriting strategy during translation. 
 During the entire evaluation, we employ a general external knowledge database that maintains the same for all the evaluation in this paper. It does not contain domain-specific external knowledge to form unfair comparisons. The improvement of external knowledge is independently reported in <Ref>.



 §.§ Translation Quality

Evaluation Metrics.
Automatic evaluation metrics such as BLEU <cit.>, BLEURT <cit.>, and COMET <cit.> are widely used for evaluating the translation quality <cit.>. 
However, they may not be able to fully reflect the quality of the translation, especially for paragraph-level translation of long speech. It is argued that the current evaluation metrics are not sufficient for ST and SiST tasks <cit.>. The work of <cit.> also highlighted that there might be a discrepancy between automatic evaluation metrics with human evaluation. 

Therefore, besides the automatic evaluation, we collaborated with senior professional human simultaneous interpreters to standardize the guidelines for a more realistic human evaluation. 
Our proposed human evaluation metric focuses on whether the output of the translation model can accurately convey the speaker's original intention for each semantic fragment.
This is also the key objective of human interpreters in real-time translation. Note that a single semantic fragment indicates a complete piece of source speech. Typically, a single semantic fragment is a complete sentence. 
Detailed definition can be found in <Ref>.
The percentage of valid information fragments within a complete speech session is defined as , which is consistent with real-world criteria for human simultaneous interpretation <cit.>.




Quantitative Analysis. 
As shown in <Ref>, we compare  with the baseline methods on dataset. 
In terms of the reliable human evaluation metrics, ,  achieves scores of 81.3% and 78.0% for zh-en and en-zh, respectively. While all other models' scores are lower than 42%. For more references, we use 3 widely-used automatic evaluation metrics: BLEU[We use SacreBLEU <cit.> for all the BLEU calculations in this paper.], BLEURT, COMET. Under the automatic evaluation metrics,  also surpasses baselines by a large margin. The detailed human evaluation results of can be found in <Ref>. 

High marks  a practical system that can help listeners understand real-time speech without professional human interpreters. Note that we only consider a system is better than others when the is higher. For example, even though Commercial 1 achieves higher scores than SeamlessStreaming on BLEU and COMET, we still consider SeamlessStreaming is a better system for zh-en translation based on .



 §.§ Latency

Evaluation Metrics. Due to the differences of grammatical structures between languages, a delay in simultaneous interpretation is inevitable. In this paper, we adopt the widely-used Average Lagging (AL) <cit.>, Length Adaptive Average Lagging (LAAL) <cit.> for comparing the latency of different methods. 
To achieve a fair comparison with systems that rewrite the translation, we calculate the time of the definite translation of these systems. 
We also propose an additional metric, First Letter Appearance Lagging (FLAL), to reflect user experience on each system.
FLAL represents the time that each system outputs the first determined translation. 

Quantitative Results. <Ref> compares the latency of our model with various systems in terms of AL, LAAL, and our proposed FLAL on the and CoVoST.
We find that the existing metrics AL and LAAL are not suitable latency measurements of paragraph-level SiST on . 
When the results are significantly shorter or longer than the reference translation, AL and LAAL may be largely exaggerated, leading to unreliable high latency. 
In these scenarios, FLAL is a more reliable and stable metric for all the systems.

Besides the paragraph-level latency evaluation, we compare our approach with other systems on the sentence-level dataset CoVoST2 zh-en, where both AL and LAAL produce reasonable values and the results are shown on the right side of  <Ref>.
Since the commercial systems usually rewrite the translation, their latency is higher than the . 
Compared with the fastest approach SeamlessStreaming, achieves comparable latency but much better translation quality.




Discussion. While existing works put a lot of emphasis on the latency-quality trade-off <cit.>, human interpretation usually uses Ear-Voice-Span (EVS) to evaluate the lagging. EVS measures the average time from when the speaker finishes conveying a piece of information to when the audience hears the corresponding translation, which is similar to AL. The typical EVS of professional human interpreters usually ranges from 3 to 6 seconds <cit.> to achieve high-quality translation. 

Consequently, we perform user studies and argue that the latency is less important than the translation quality for a practical SiST system. In the recent IWSLT 2023 simultaneous track <cit.>, the ranking of models is also evaluated by the translation quality within certain latency constraints. We verify whether the latency of is acceptable to users through real-world user surveys. To the publication date of this paper, we collected 14 user surveys on zh-en direction, each user using for at least 30 minutes. Under the current latency performance shown in <Ref>, only 1/14 == 7% of them suggest that the latency significantly affects their user experiences while the rest think the improvement of translation quality outweighs the latency and overall output of largely helps them to understand the speech. Considering that the latency of is even lower than most of the commercial systems, We believe the latency of can be acceptable on most cases. 

Current latency metrics are proposed on sentence-level SiST. As shown in <Ref>, such metrics may not be suitable latency measurements for paragraph-level. 
As the importance of end-to-end evaluation for long speech keeps increasing, more refined metrics are required to measure the latency and provide a deeper insight into the systems.



 §.§ Supplementary Experiments


To ensure a comprehensive evaluation of , our model is further evaluated on four additional datasets, including BSTC (zh-en) <cit.>, CoVoST2 (zh-en) <cit.>, MuST-C (en-zh) <cit.>, and GigaST (en-zh) <cit.>[We use the subset from in https://github.com/SpeechTranslation/GigaS2SGigaS2S for evaluation].  
<Ref> presents the results of the automatic evaluation metrics for both zh-en and en-zh. 
Due to the high cost of human evaluation, we are not able to provide VIP for these four datasets.
We observe that our model achieves consistently better performance than the baseline models.
Even though our system achieves the best automatic evaluation results among all the compared systems, 
we still would like to emphasize that such a sentence-level evaluation scheme might overestimate the performance of SiST systems.






 §.§ MM-RAG Performance




  §.§.§ Retriever

Table <ref> presents the performance of various retrieve models on the development set of our proprietary dataset. Each sample in the test set includes a short audio chunk and the mentioned terms in the audio. 
Our MM-RAG retriever outperforms other open-source models by a large margin, achieving 91.3 % vs. 26.0% for Top-10 retrieve accuracy.
We compare two types of methodologies: audio-to-audio and audio-to-text. 
In the audio-to-audio approach, a Text-to-Speech (TTS) model is utilized to convert the text keys from the external knowledge database into audio format,
 forming a database with audio-based keys. 
The audio keys and the user-input audio are then encoded with the ASR model to produce the corresponding representations. 
The Top-k retrieved items are subsequently determined using the Maximum Inner Product Search (MIPS) algorithm.
For audio-to-text approach, we compare MM-RAG with CLAP <cit.>. As indicated in Table <ref>, the effectiveness of these models remains significantly below acceptable standards, and MM-RAG significantly outperforms them.

It is worth noting that the same audio encoder employed in our is utilized for generating audio embedding in the MM-RAG retriever.
Such a design ensures that the integration 
brings minimal computational latency to the overall framework.





  §.§.§ ICL Performance


By incorporating the retrieved terms from the external knowledge database as contextual information, our model's in-context learning ability significantly improves the performance of speech translation for in-domain terminologies. <Ref> compares our method with the widely-used shallow fusion for intervention in the generated conclusion. When calculating the Recall, we input 1 ground-truth keyword with 9 similar negative words as context. When calculating the false positive rate for precision, we input 10 similar negative words as context. ICL is able to achieve a high recall rate with good precision, obtaining the highest F1 while shallow fusion only gets a recall rate that is only half of ICL.

Additionally, we conduct an ablation study on our MM-RAG module within terminology-intensive scenarios incorporating the whole  pipeline.
The incorporation of the external knowledge database results in a significant increment in the score by about 10%, highlighting the effectiveness of our proposed MM-RAG.





 §.§ Case Study
 
We present case studies to show the ability of in translating complicated speech for zh-en and en-zh in <Ref> and <Ref>. We choose one of the most-performed cascaded systems for comparison. The adopted a cascaded approach for SiST and it is shown in <Ref> to be one of the best previous SiST systems. Detailed explanations are described in the tables. For zh-en direction, we present cases regarding robustness to recognition errors, reasoning ability, and trending words translation. For the en-zh direction, we present cases regarding native, expressive, and accurate terminology translations. More cases are shown in <Ref>.











§ RELATED WORK



  
Large language model.

The encoder-decoder architecture <cit.> has been widely explored in early speech translation research, but with the advent of large language models <cit.>, there has been a growing interest in employing decoder-only architectures <cit.> for sequence-to-sequence problems. While recent efforts have emerged in utilizing large language models for machine translation <cit.> and speech translation <cit.>, the application of such models in simultaneous translation tasks remains limited. Although there has been early attempts to utilize LLM for simultaneous machine translation <cit.>, to the best of our knowledge, no existing work has been found that explores the utilization of large language models for end-to-end simultaneous speech translation with such remarkable improvement.

Furthermore, LLMs have demonstrated impressive capabilities in tasks such as instruction following <cit.>, reasoning <cit.>, and planning <cit.>. Recent research studies have leveraged prompt engineering to develop remarkable LLM agents that autonomously tackle complex tasks in diverse environments <cit.>.  
In our work, we empower the LLM to perform sequential instructions to accomplish the simulation speech translation task.



  
Simultaneous Speech Translation.
One of the important components of simultaneous speech translation is the segmentation strategy, which determines how the speech frames are fed to the models.
Different strategies could affect the latency and performance of the translation. 
According to <cit.>, segmentation strategies can be classified into fixed-length, word-based, and adaptive segmentation. Fixed-length strategies <cit.> divide the speech into equally-length segments, while word-based strategies <cit.> identify word boundaries within the speech. Adaptive segmentation <cit.> detects boundaries for speech units. Among these categories, our method utilizes a fixed-length strategy.

Regarding the read/wait policy, the Wait-k method <cit.> and its variants <cit.> have been extensively studied in the context of text translation and speech translation. In comparison to these approaches, which explicitly learn the generation of read/write signals, another line of research focus on how to leverage offline translation models <cit.>.
When utilizing an offline translation model for simultaneous translation, it is important to address the stabilization of generated hypotheses to prevent excessive content refreshing experienced by the user.
<cit.> first proposed a local agreement policy to stabilize the partial hypothesis, while <cit.> introduced an incremental blockwise beam-search algorithm. In contrast to these methods, we enforce our model to generate consistent hypotheses by constraining the prompt to the language model.

For the model architecture, there are two primary methods for implementing speech translation systems: cascaded solutions <cit.> and end-to-end solutions <cit.>. Cascaded solutions involve separated ASR and MT components, while end-to-end solutions directly map speech to translations. Cascaded systems benefit from established techniques but suffer from latency and error propagation. End-to-end models offer real-time translation and improved quality through deep learning but require large-size training data. 
In our work, we implement an end-to-end model which combines the capabilities of ASR, MT, and ST.



  
Human Evaluation. 
In the realm of speech translation, the choice of evaluation metrics plays a crucial role in assessing the quality and effectiveness of translation systems. While automatic metrics, such as BLEU <cit.>, BLEURT <cit.>, and COMET <cit.>, have traditionally been relied upon for evaluation, there is a growing recognition that they may not be the most suitable or comprehensive measure of performance <cit.>.
We observe that in more recent works <cit.>, there is an increasing trend of evaluating systems using human assessments, particularly when LLM is employed in the work.
While human evaluation requires more resources and time compared to automatic metrics, its benefits outweigh the drawbacks. By incorporating human judgment, speech translation systems can be refined and optimized to align with user expectations, ensuring translations that are not only technically accurate but also linguistically and contextually appropriate.
In contrast to the existing human evaluation metrics, e.g.“continuous rating” <cit.> and MQM (Multidimensional Quality Metrics) <cit.>, we have taken inspiration from professional human interpreters <cit.> and propose to use VIP (Valid Information Proportion) as a human evaluation metric which precisely reflects the goal of the simultaneous translation task.







§ CONCLUSION

In this work, we introduce Cross Language Agent - Simultaneous Interpretation, , an LLM agent to produce end-to-end simultaneous speech translation. Benefits from massive pretraining and imitation learning, achieves significantly better performances than state-of-the-art systems. Take the Chinese-to-English direction as an example, under strict and challenging human-evaluated metrics proposed by professional human interpreters, Valid Information Proportion (),  significantly outperforms baselines by a large margin. While all other systems obtain by less than 40%, achieves a VIP of 81.3%, demonstrating human parity performance.
More specifically, we propose the following crucial components for the supreme performance of : (1) An encoder-conditioned LLM agent architecture that performs high-quality or even human-parity SiST process through simple actions. (2) Imitation learning from human interpreters for a natural read-write policy balances translation quality and latency in a data-driven manner, without complex human pre-designing. Under such policy, achieves a stable output scheme, where each output is deterministic, thus potentially better user experience than most commercial systems. (3) Motivated by the preparatory trajectory of human interpreters, could perform in-context learning from historical translations and external knowledge to provide sufficient information for translation. 
With the powerful translation ability of ,  we believe it can further make cross-lingual communication seamless across different places all over the world.



§ LIMITATION AND FUTURE WORK

Although we achieved significant improvements over commercial systems in Chinese-to-English and English-to-Chinese tasks, more languages should be considered in the future. 
In our current implementation, performs a full action sequence for each translation round. Some of the actions, e.g.,  is optional for easy translation scenarios since the model is capable of translating correctly without the help of external knowledge. Training the model to better determine whether to skip unnecessary actions is a future direction. 
For a product-level system, even though the latency of is acceptable in most cases, how to reduce the translation latency without lowering the translation quality is still interesting and potentially helpful for user experience. 
Furthermore, we argue that the current automatic metrics are not comprehensive for SiST evaluation. Most of the quality measurements do not consider key information, which is crucial in SiST scenarios. As such, we proposed for better human evaluation. Consequently, more reliable automatic quality and latency metrics should be proposed in the future as well. 
Reinforcement learning from human feedback (RLHF) has been proven to be effective in enhancing LLM performance. Although achieves significantly superior results than previous state-of-the-art systems, further studies on how to build better multi-modal reward models and better RL methods for SiST is also an important direction. Incorporating more modalities, for example, end-to-end speech-to-speech generation, or even end-to-end video-to-video generation are also promising research topics.



§ SOCIAL IMPACT

The powerful SiST system can be applied to various scenarios to facilitate cross-lingual communications. For example, it can be deployed to various conferences or daily meetings to help listeners understand speech in different languages. It can also be deployed as a system-level translation module to help users watch videos that are conveyed in different languages. For online gaming, it can also help to bridge the gap of cross-lingual communication and connect people speaking different languages. A powerful SiST system with human parity performance may significantly improve the efficiency of professional human interpreters.

Despite the huge positive social impact that may bring, every coin has two sides. Neglecting some low-resource languages may also bring unfairness to some minorities. Resolving these problems needs further cooperation from the society. We leave more languages supporting as our future work.





§ AUTHORSHIP AND ACKNOWLEDGEMENTS

All contributors are listed in alphabetical order by last name. Corresponding to this work can be sent to any core authors' email.

Core Authors. 
All core authors contributed equally to this work. 
2

    
  * Shanbo Cheng 
    
  * Zhichao Huang
    
  * Tom Ko
    
  * Hang Li
    
  * Ningxin Peng
    
  * Lu Xu
    
  * Qini Zhang
    
        
        
  * chengshanbo@bytedance.com
        
  * zhichao.huang@bytedance.com
        
  * tom.ko@bytedance.com
        
  * lihang.lh@bytedance.com
        
  * pengningxin@bytedance.com
        
  * xu.lu1@bytedance.com
        
  * qini.z@bytedance.com
    



Labeling, Evaluation and Interpretation Team. 
Our data labeling and human evaluation team led by Yifu Li, made diligent efforts in all kinds of help needed, which is irreplaceable in the success of this project. Special thanks to the human interpreter team led by Anna Liu for providing insightful and comprehensive analysis on data labeling, human evaluation, and other recommendations. 

    
  * Jingwen Chen
    
  * Xiaoya Chen
    
  * Yifu Li
    
  * Huiying Lin
    
  * Anna Liu


Engineering Team. 
We collaborated with our engineering team led by Tingshuai Yan, their infrastructure support is crucial for this project.

    
  * Weicheng Fu
    
  * Tingshuai Yan
    
  * Liehao Zou


Acknowledgements. We appreciate the Speech Understanding team for all kinds of help, especially data sharing, thanks to their tremendous work for all the in-house data. We would like to express our deepest thanks to all the contributors to this project, their brilliant work guarantees the success of this project.



plain








§ HUMAN EVALUATION GUIDELINES


In this Appendix, we provide detailed human evaluation guidelines which are formulated by professional human interpreters.


 §.§ Key Indicator
 
We define the key indicator as the “Valid Information Proportion,” denoted as . This metric measures the proportion of valid semantic fragments within a complete speech session. A semantic fragment is deemed valid if it effectively conveys the core information, accurately representing the speaker’s original intent. Typically, one complete sentence is considered a single semantic fragment. assesses the model’s ability to capture and communicate the essence of the spoken content. 



 §.§ Evaluation Process


First, the human evaluators segment the long translation result into semantic fragments according to the formal rules as follows:

    
  * Semantic Completeness: Each fragment should contain one complete concept or information point. For example, in a conference translation, an ideal semantic fragment often corresponds to a full sentence.
    
  * Natural Language Pauses: Pauses that naturally occur in speech often indicate the boundaries of semantic fragments. During segmentation, natural pauses should be extensively considered and avoid irrational interruptions. Also, punctuation and conjunction in the text should be considered to maintain integrity and clarity of information.
    
  * Logical Coherence: Each segment should contain information that are logically coherent and continuous. Conditional sentences, causative sentences, or both parts of antithesis sentences should be kept within the same segment.
    
  * Grammatical Completeness: Each segment should include all necessary grammatical components (e.g., subject, verb, object) and have a complete grammatical structure.
    
  * Proper Information Density: Each segment should have a moderate amount of information, avoiding information overload. It is recommended that each segment not exceed 50 words.


After segmentation, the human evaluators follow the instructions below to evaluate the validity of the semantic fragments:

    
  * Key Information Recognition. Key information refers to the content that can constitute core information, including but not limited to proper nouns, keywords, terminologies, sentence structures, etc.
    
  * Correctness Assessment. Evaluators assess whether the translation of key information is accurate and successful in conveying the correct spoken intentions. Misinterpretations of the speaker’s words, inaccuracies in analyzing the context, or erroneous translations of specific terms can all contribute to the failure of the assessment. 
    
  * Expressiveness Assessment. Evaluators assess whether the whole segment is translated accurately, comprehensibly, and expressively to humans. Assessing for any vague, ambiguous, or misleading statements. This indicator primarily evaluates the clarity, fluency, and intuitiveness of the translation, rather than its accuracy. Typically, verbosity, complex sentence structures, or challenging grammatical constructions that are unnecessary would reduce the expressiveness of the translation, thus leading to failure of the assessment. 

If the translation fails any of the above assessments, the translation will be marked as invalid. After the evaluators assessed all semantic fragments, the could be simply calculated as dividing the number of valid semantic fragments by the total number of fragments.


We illustrate the evaluation criteria with two examples in Table <ref>. Although these translations achieve “high accuracy” in automatic evaluations, we still categorize them as invalid. It’s important to note that our standard aims to emulate human interpreters, presenting a significant challenge to both human evaluators and translation systems.

UTF8gbsn






 §.§ Correlation with Automatic Metrics

<Ref> shows the distribution and regression curve for with regard to BLEU, BLEURT, and COMET, respectively. From the scatter points in <Ref> we may observe that as score increases, the growth of the automatic metric curves slows down and becomes less significant, making it hard to reflect the real changes in translation quality. The correlation curves in <Ref> also demonstrate the finding. Here, we calculate Kendall's Tau correlation coefficient <cit.>, which measures the monotonic correlation between two ordered variables. In low ranges, the correlation between and automatic metrics is observable; as the score increases, the correlation harshly drops, which indicates a significant distortion of the automatic metrics. A possible reason is that the translations may differ from the groundtruths by only a few words in the mediocre ranges. However, in a real simultaneous interpretation scenario, these words are likely to be keywords that play important roles in conveying precise information, which may significantly impact scores.





§ SUPPLEMENTARY MATERIALS




 §.§ Supplementary Case Study

We provide more case studies in <Ref>. In terms of informal, disfluent, code-mixing, and named-entity translation, could achieve much better results than the commercial products. Benefits from the end-to-end approach, could also understand the original speech tone and generate better translations.





 §.§ Example of Detailed Evaluation Result on 


We provide a detailed human evaluation result for in Figure <ref>, where we provide golden source transcription, output, human evaluation results, and reference translation. We randomly choose one of the test samples in . We share the full detailed evaluation results at https://docs.google.com/spreadsheets/d/1OBDbQMlixqFkBfzLW5pmP8WsI-RHqAtYbLnkd321N6Aonline sheets for academic reference. Note that to ensure fair comparison, when evaluating multiple systems, we randomly shuffle the ordering between systems for each semantic fragment so that human evaluators cannot identify the specific system.






 Paper Title: Gemma 2: Improving Open Language Models at a Practical Size.



Vera Verto: Multimodal Hijacking Attack
    
Minxing Zhang1  Ahmed Salem2  Michael Backes1  Yang Zhang1




1CISPA Helmholtz Center for Information Security    
2Microsoft

    
====================================================================================================================================





§ INTRODUCTION


Large language models (LLMs) have demonstrated strong capabilities in language understanding, generation, and reasoning <cit.>. 
Scaling has been key to this recent progress, with many new capabilities only emerging at scale <cit.>.
The newest large models not only reach unprecedented performance on reasoning benchmarks <cit.>, but they also demonstrate multimodal and multilingual capabilities <cit.> and even the ability to use context lengths of over 1M tokens <cit.>. 

Small-scale models have also shown a rapid increase in performance, but these gains are largely derived from increasing the length of training <cit.>. 
This approach only scales logarithmically with dataset size <cit.>, and the latest small models require up to 15T tokens to improve the state of the art by less than 1-2% <cit.>.

Yet, these continued improvements provide evidence that small models are still under-trained.
In this work, we explore alternatives to improve small model performance without solely increasing training length. One solution is to improve the quality of information received by the network at each training step by replacing the next token prediction task with a richer objective. 

In particular, we focus our efforts on knowledge distillation <cit.>, which replaces the one-hot vector seen at each token with the distribution of potential next tokens computed from a large model.
This approach is often used to reduce the training time of smaller models by giving them richer gradients.
In this work, we instead train for large quantities of tokens with distillation in order to simulate training beyond the number of available tokens.
Concretely, we use a large language model as a teacher to train small models, namely 2B and 9B models, on a quantity of tokens that is more than 50× the compute-optimal quantity predicted by the theory <cit.>.
Along with the models trained with distillation, we also release a 27B model trained from scratch for this work.



We also leverage several known modifications of Transformers, namely the interleaving of global and local attention layers from <cit.>, and the Grouped-Query Attention (GQA) mechanism of <cit.>.



Overall, Gemma 2 significantly advances state-of-the-art performance relative to comparable-scale open models and are even competitive with some models more than twice their size <cit.>, across a variety of automated benchmarks and human evaluations. Example domains include question answering <cit.>, commonsense reasoning <cit.>, mathematics and science <cit.>, and coding <cit.>.

While thorough testing of our models has been conducted, these tests cannot cover all applications and scenarios in which Gemma 2 may be used. With this in mind, all Gemma 2 users should conduct rigorous safety testing specific to their use case before deployment or use.

In this technical report, we provide an overview of models, including the architecture, training, and pre- and post-training recipes for Gemma 2. We also provide detailed evaluations across a wide variety of quantitative and qualitative benchmarks, as well as both standard academic benchmarks and human-preference evaluations. Finally, we discuss our approach to safe and responsible deployment and outline the broader implications of Gemma 2, its limitations, and advantages.







§ MODEL ARCHITECTURE



Similar to previous Gemma models <cit.>, the Gemma 2 models are based on a decoder-only transformer architecture <cit.>.
We summarize the main parameters and architecture choices in Table <ref>. 

A few architectural elements are similar to the first version of Gemma models; namely, a context length of 8192 tokens, the use of Rotary Position Embeddings (RoPE) <cit.>, and the approximated GeGLU non-linearity <cit.>. A few elements differ between Gemma 1 and Gemma 2, including using deeper networks. We summarize the key differences below.

Local Sliding Window and Global Attention. We alternate between a local sliding window attention <cit.> and global attention <cit.> in every other layer. The sliding window size of local attention layers is set to 4096 tokens, while the span of the global attention layers is set to 8192 tokens.

Logit soft-capping. We cap logits <cit.> in each attention layer and the final layer such that the value of the logits stays between -soft_cap and +soft_cap. More specifically, we cap the logits with the following function: 
    logits←soft_cap * tanh(logits / soft_cap).

We set the soft_cap parameter to 50.0 for the self-attention layers and to 30.0 for the final layer. 





Post-norm and pre-norm with RMSNorm. To stabilize training, we use RMSNorm <cit.> to normalize the input and output of each transformer sub-layer, the attention layer, and the feedforward layer.

Grouped-Query Attention <cit.>. 
We use GQA with num_groups = 2, based on ablations showing increased speed at inference time while maintaining downstream performance.



§ PRE-TRAINING


We provide a brief overview of the parts of our pre-training that differs from Gemma 1.



 §.§ Training Data


We train Gemma 2 27B on 13 trillion tokens of primarily-English data, the 9B model on 8 trillion tokens, and the 2B on 2 trillion tokens. 
These tokens come from a variety of data sources, including web documents, code, and science articles. Our models are not multimodal and are not trained specifically for state-of-the-art multilingual capabilities.
The final data mixture was determined through ablations similar to the approach in Gemini 1.0 <cit.>.
 
Tokenizer.
We use the same tokenizer as Gemma 1 and Gemini: a SentencePiece tokenizer with split digits, preserved whitespace, and byte-level encodings <cit.>.
The resulting vocabulary has 256k entries.

 
Filtering.
We use the same data filtering techniques as Gemma 1. Specifically, we filter the pre-training dataset to reduce the risk of unwanted or unsafe utterances, filter out certain personal information or other sensitive data, decontaminate evaluation sets from our pre-training data mixture, and reduce the risk of recitation by minimizing the proliferation of sensitive outputs. 






 §.§ Knowledge Distillation


Given a large model used as a teacher, we learn smaller models by distilling from the probability given by the teacher of each token x given its context x_c, i.e.,  P_T(x | x_c).
More precisely, we minimize the negative log-likelihood between the probabilities from the teacher and the student:

    min_P_S∑_x -P_T(x | x_c) log P_S (x | x_c),

where P_S is the parameterized probability of the student.
Note that knowledge distillation was also used in Gemini 1.5 <cit.>.



 §.§ Compute Infrastructure


We train our models with TPUv4, TPUv5e, and TPUv5p as outlined in Table <ref>.
For the 2B model, we train on a 2x16x16 configuration of TPUv5e, totaling 512 chips, with 512-way data replication and 1-way model sharding.
For the 9B model, we train on an 8x16x32 configuration of TPUv4, totaling 4096 chips, with 1024-way data replication and 4-way model sharding. 
For the 27B model, we train on an 8x24x32 configuration of TPUv5p, totaling 6144 chips, with 768-way data replication and 8-way model sharding. 



The optimizer state is further sharded using techniques similar to ZeRO-3 <cit.>. 
For scales beyond a single pod, we perform a data-replica reduction over the data center network, using the Pathways approach of <cit.>.
We also use the 'single controller' programming paradigm of Jax <cit.> and Pathways <cit.>. As in Gemma 1, we use the GSPMD partitioner <cit.> for training step computation and the MegaScale XLA compiler <cit.>.





 §.§ Carbon Footprint

We estimate the carbon emissions from pre-training the Gemma models to be 1247.61 tCO_2 eq. As in Gemma 1 <cit.>, this value is calculated based on the hourly energy usage reported directly from our TPU data centers and scaled to account for the additional energy expended to create and maintain the data center. Importantly, Google data centers are carbon neutral, achieved through a combination of energy efficiency, renewable energy purchases, and carbon offsets. This carbon neutrality applies to our experiments and the machines running them. 




§ POST-TRAINING


For post-training, we fine-tune our pre-trained models into instruction-tuned models.
First, we apply supervised fine-tuning (SFT) on  a mix of text-only, English-only synthetic and human-generated prompt-response pairs. We then apply RLHF on top of these models with the reward model trained on labelled English-only preference data and the policy based on the same prompts as the SFT phase. Finally, we average the models obtained after each phase to improve their overall performance. The final data mixtures and post-training recipe, which includes tuned hyperparameters, were chosen on the basis of improving helpfulness while minimizing model harms related to safety and hallucinations. 

We extended the post-training data from Gemma 1.1 with a mixture of internal and external public data. In particular, we use the prompts, but not the answers from LMSYS-chat-1M <cit.>. All of our data go through a filtering stage described below.



Supervised fine-tuning (SFT). We run behavioral cloning on synthetic and real prompts, and responses predominantly synthetically generated by the teacher, that is a larger model. We also run distillation from the teacher on the student's distribution  <cit.>.





Reinforcement Learning from Human Feedback (RLHF). We use a similar RLHF algorithm as Gemma 1.1 <cit.> but a different reward model, which is an order of magnitude larger than the policy. The new reward model is also oriented more towards conversational capabilities, specifically multi-turn.

Model merging. We average different models obtained by running our pipeline with different hyperparameters <cit.>.

Data filtering. When using synthetic data, we run several stages of filtering to remove examples that show certain personal information, unsafe or toxic model outputs, mistaken self-identification data, and duplicated examples. Following Gemini, we find that including subsets of data that encourage better in-context attribution, hedging, and refusals to minimize hallucinations improves performance on factuality metrics, without degrading model performance on other metrics. 

Formatting. Gemma 2 models are fine-tuned with the same control tokens as Gemma 1 models, as detailed in Table <ref>, but a different formatting schema. See the dialogue example in Table <ref>. Notice that the model explicitly ends generations with  tokens, while previously it only generated . For the motivation behind this formatting structure, see Gemma 1.






§ ABLATIONS


In this section, we focus on the main finding of this work, which is the impact of knowledge distillation on small language models.




Distillation versus from scratch. 
In Table <ref>, we show that distilling from a larger model improves performance compared to training from scratch.
Note that 500B is 10× more than the compute-optimal number of tokens for a 2B model.
We distill from a 7B model to keep a ratio similar to our target distillation from 27B to 9B.



Impact of distillation w.r.t. model size. In Table <ref>, we measure the impact of distillation as model size increases.
We observe that the gain remains as the model size is scaled. 
In this ablation, we maintain the size of the teacher at 7B and train smaller models to simulate the same gap as between our final teacher and student sizes.



GQA versus MHA. 
In Table <ref>, we compare two instances of our 9B with MHA or GQA. 
We observe overall few changes in performance between both models as measured on several benchmarks. We choose GQA since it requires fewer parameters and is faster at inference time.



Wide versus deep. 
In Table <ref>, we show that a deeper 9B network is slightly better than a wider 9B for the same number of parameters.
Although the gap is small, it is consistent across benchmarks and warrants the switch to a deeper architecture.



Changing sliding window size. In Table <ref>, we show that we can change the sliding window size of the local attention layers of the models during inference with moderate impact on perplexity. 
Adjusting the size of the sliding window can thus be a leverage for slight inference speed gain. 





Impact of formatting. We measure performance variance on MMLU across prompt/evaluation formatting variations.
Table <ref> shows the standard deviations of MMLU scores for 12 formatting/evaluation combinations, a proxy for undesired performance variability.
The Gemma 2B models are slightly less format-robust than the larger ones. 
Notably, Mistral 7B is significantly less robust than our models.

										
					


§ EVALUATION



In this section, we evaluate both pre-trained and IT models over a series of automated benchmarks and human evaluations across a variety of domains. 
We also report performance from models of similar sizes that have permissive licenses, or as reported by others.
Note that we consider total parameters, not active parameters, since total memory usage is often what limits the use of open models on standard devices.



 §.§ Pre-training Evaluations




  §.§.§ Evaluating the 27B model 


In this set of evaluations, we evaluate the performance of our 27B model trained without distillation on 13T tokens. 
We report results in Table <ref>, where we compare with a model of similar size, Qwen1.5 34B <cit.>, and a model 2.5× larger, LLaMA-3 70B on the HuggingFace evaluation suite. We selected these models based on their ranking on the HuggingFace leaderboard.

Overall, we observe that our model is the best in its size category and is even competitive with a larger model that is trained for longer.
That being said, the performance of models trained in a similar fashion improves only logarithmically with their size and hence, our model is likely in the same Pareto curve as the LLaMA-3 models.
However, it is not clear how these differences affect the quality of the resulting IT models.







  §.§.§ Evaluating the 2B and 9B models






In this set of experiments, we compare our new 2B and 9B trained with distillation to our previous models and several standard open models in <cit.>. 

We observe overall a massive improvement in our models compared to previous versions, by up to 10% in some benchmarks for the 9B model.
The two 2B models were trained with a similar number of tokens (2T for Gemma 2 and 3T for Gemma 1) and we still observe a significant improvement for the new models.
This confirms that distillation significantly improves the quality of models even when trained on the same number of tokens.





 §.§ Post-training Evaluations


In this section, we evaluate our IT models on a set of human evaluations as well as standard academic benchmarks. The Gemma 2 models push the frontier for post-trained open-weights models, setting a new state of the art on the LMSYS Chatbot Arena <cit.>. 




  §.§.§ LMSYS Chatbot Arena

Gemma 2 Instruction Tuned models were evaluated on the Chatbot Arena <cit.> in blind side by side evaluations by human raters against other state of the art models. We report Elo scores in Table <ref>. Gemma 2.6B, 9B and 27B strongly outperform all other open models in the same range of parameters, with notably: Gemma 27B (Elo 1218) ranked higher than Llama 3 70B (Elo 1206), Gemma 9B (Elo 1187) similar as GPT-4-0314 (Elo 1186), Gemma 2.6B (Elo 1126) ranked higher than GPT-3.5-Turbo-0613 (Elo 1116).





  §.§.§ Human Preference Evaluations



We also submit Gemma IT models for side-by-side human evaluation studies (which are independent from the Chatbot Arena). We used held-out collections of single-turn prompts that target safety and instruction following (IF).
We use  as the base model, and observe large improvements in win rates and preference scores as compared against the older Gemma 1.1 7B model.
We report safety as a win-loss ratio against GPT4o, and we report single-sided instruction following scores as ratio of prompts where all instructions are followed. In particular, we find that regardless of their size, Gemma 2 models produce safer, more appropriate prompts on the held-out safety prompt set than GPT4o.







  §.§.§ Human Multi-Turn Evaluations



We evaluated the multi-turn capabilities of Gemma 1.1 7B, Gemma 2 2B, 9B and 27B models by tasking human raters to have conversations with the models and follow specified given scenarios. We used a diverse, held-out set of 500 scenarios, each describing a sequence of requests to the model, including measuring instances of brainstorming, making a plan, or learning something new. The average number of user turns is 8.4. We found that the conversations with Gemma 2 models are rated significantly better than Gemma 1.1 in user satisfaction and conversation goal achievement (Table <ref>). Moreover, we saw that the Gemma 2 models were better than Gemma 1.1 7B at maintaining high quality of responses for the entire conversation.





  §.§.§ Standard Benchmarks


It has been observed in Llama-3 <cit.> that instruction fine-tuning can improve the performance of the models on few-shot benchmarks despite not being trained to target few-shot capabilities.
In Table <ref>, we show a similar improvement across our models.
Overall, we observe improvements on the order of several percentage points.
We conjecture that IT models are better at understanding formatted questions, while pre-trained models are sensitive to formatting.





§ MEMORIZATION AND PRIVACY
 
Large language models may, under particular circumstances, be vulnerable to attacks causing the model to produce memorized[This work uses a very restricted definition of “memorization”: whether a model can be induced to generate near-copies of some training examples when prompted with appropriate instructions. We do not mean to say that a model 'contains' its training data in the sense that any arbitrary instance of that data can be retrieved without use of specialized software or algorithms. Rather, if a model can be induced to generate measurably close copies of certain training examples by supplying appropriate instructions to guide the model's statistical generation process then that model is said to have 'memorized' those examples.] training data <cit.>. To study susceptibility to such attacks and quantify memorization, we evaluate models for verbatim and approximate memorization as was done in several prior studies <cit.>.

We follow the evaluation setting of <cit.> which tests for (50 token) memorizations of training data given a prompt of 50 tokens. We compare the overall memorization rates, across a uniform sample of the entire dataset, using both an exact match criteria and approximate match criteria <cit.> using an edit distance of 10%.

Verbatim Memorization: Results are in Figure <ref>. We first compare against recent models from the literature that include memorization evaluations. We find that Gemma 2 memorizes significantly less than prior models at a similar size, with memorization rates below 0.1% (note the log y-axis). We further investigate how this memorization breaks down with respect to the data source. Similar to Gemma 1, we find that Gemma 2 memorizes more from code, wiki, and science sources, and also that it memorizes significantly less across the board (again, note the log y-axis).

Approximate Memorization: Figure <ref> also presents approximate memorization by data source. We observe that while approximate memorization is higher than exact, the rate of memorization is still low. For example, the approximate memorization of this model is much lower than even the exact memorization of Gemma 1. We find that the increase in approximate memorization is much lower than prior models; in some cases we observed no lift at all c.f. <cit.> (note that no bar indicates no increase, i.e., the rate of approximate memorization equals that of exact memorization). Note that no approximate memorization bar in Figure X indicates no increase, i.e., the rate of approximate memorization equals that of exact memorization.

Personal Data We use the same prevention methods at training time and the same evaluations as <cit.>. In particular, we use Google Cloud Sensitive Data Protection Tool[Available at: https://cloud.google.com/sensitive-data-protection] to find potential instances of personal data. The many categories of personal data (e.g., phone numbers, account numbers) are classified into three severity levels. We analyze memorized outputs using these severity levels. . We found no instances of high-severity data being emitted, and found a very low rate of 0.00026% of memorized data to contain lower-severity personal information. We note that these automated tools are known to incur false positives because they do not account for context. This means our results are likely overestimates.





§ RESPONSIBILITY, SAFETY, SECURITY




Responsibility, safety and security are of paramount importance when developing Gemma models. To reduce risks to Gemma 2 users, we have integrated enhanced internal safety processes that span the development workflow, in line with recent Google AI models <cit.>.  Similar to the inaugural Gemma release, we have followed a three pillar approach which focuses on safety mitigation at training time, robust and transparent model evaluations, and further development of the Responsible Generative AI Toolkit, a series of models and tools to help developers implement responsibility and safety best practices for their applications.





 §.§ Impact assessment

Our approach and resulting impact assessment is reflective of that outlined for Gemma 1 <cit.>: we continue to believe that openness in AI can spread the benefits of these technologies across society, but must be evaluated against the risk of malicious uses, such as the creation of deepfake imagery, AI-generated disinformation or illegal and disturbing material, that can cause harm on both an individual and institutional levels <cit.>.
Since the launch of Gemma 1, we have seen our Gemma models drive a number of socially beneficial applications, relying on Gemma’s unique technologies like its tokenizer to facilitate the creation of multilingual models, such as for Navarasa 2.0, a Gemma tuned model for 15 Indian languages.

Releasing further open models requires specific attention to changes in model capabilities and close monitoring of the evolving risks of LLMs <cit.>, as well as, an understanding of the ways in which our models are being used in the wild. Although we are yet to receive any reports of malicious use for Gemma, we remain committed to investigating any such reporting, and work with the academic and developer communities, as well as conduct our own monitoring, to flag such use cases via our contact email[gemma-2-report@google.com]. 

Despite advancements in capabilities, we believe that given the number of larger and more powerful open models, this release will have a negligible effect on the overall risk landscape.






 §.§ Safety policies and train-time mitigations

A key pillar of Gemma’s approach to safety is to align fine-tuned models with Google’s safety policies, in line with Gemini models <cit.>. They are designed to help prevent our models from generating harmful content, i.e.,

    
  * Child sexual abuse and exploitation
    
  * Revealing personally identifiable information that can lead to harm (e.g., Social Security numbers)
    
  * Hate speech and harassment
    
  * Dangerous or malicious content (including promoting self-harm or instructing in harmful activities)
    
  * Sexually explicit content
    
  * Medical advice that runs contrary to scientific or medical consensus

We undertook considerable safety filtering of our pre-training data to reduce the likelihood of our pre-trained and fine-tuned checkpoints producing harmful content. For fine-tuned models, we also use both SFT and RLHF to steer the model away from undesirable behavior. 





 §.§ External benchmark evaluations


Robust and transparent evaluations are key principles of our responsible approach to developing Gemma. To this end, we report in Table <ref> Gemma 2 evaluations on public benchmarks.





 §.§ Assurance Evaluations


We also run our IT models through a set of assurance evaluations to understand the harms that our models can cause.
We focus on  capabilities relevant to extreme risks <cit.> <cit.>. Specifically, we evaluate on offensive cyber-security, code vulnerability detection, Chemical, Biological, Radiological and Nuclear (CBRN) knowledge, and self-proliferation.
We refer the reader to <cit.> for full methodological details of these studies.



  §.§.§ Baseline Evaluations

Baseline assurance captures the model’s violation rate for safety policies, using a large number of synthetic adversarial user queries, and human raters to label the answers as policy violating or not. Overall, Gemma 2’s violation rate is significantly lower overall on the safety policies listed above, in particular on Child safety content.










  §.§.§ Chemical, Biological, Radiological and Nuclear (CBRN) knowledge 

We evaluated knowledge relevant to biological, radiological and nuclear risks using an internal dataset of closed-ended, knowledge-based multiple choice questions. For evaluations of chemical knowledge, we employed a closed-ended knowledge-based approach on chemical hazards (developed by Macknight et al <cit.>. Our evaluation suggests that Gemma models' knowledge in these domains is low.



  §.§.§ Offensive cyber-security



To evaluate Gemma models' capabilities at offensive cybersecurity, we ran Gemma 2 27B against some automated capture-the-flag (CTF) challenges. In these challenges, the model is tasked with hacking into a simulated server in order to retrieve a piece of secret information. Specifically, we test on InterCode-CTF <cit.>, our own internal CTF suite[<https://github.com/google-deepmind/dangerous-capability-evaluations>] <cit.>; and a challenge based on Hack the Box [<https://www.hackthebox.com>].



In Table <ref>, we show that Gemma 2 27B has a significant increase in capabilities compared to CodeGemma 1.0 7B on the easier of these challenge suites, InterCode CTF. (Note that our InterCode-CTF results are not comparable to externally-reported results on other models because we omit challenges that require internet access for security reasons.) However, Gemma 2 is unsurprisingly much less capable than Gemini 1.5 Pro on these tasks.




  §.§.§ Code vulnerability detection

In Table <ref>, we also evaluate Gemma 2 27B on a series of multiple-choice code vulnerability detection datasets. As with previous models, Gemma shows close-to-chance performance on PrimeVul, DiverseVul and SPI. Gemma 2 shows performance on SecretPatch similar to Gemini 1.0 Ultra.
















  §.§.§ Self-proliferation


"Self-proliferation" refers to the ability for an agent to autonomously replicate - to instantiate goal-directed agents on other machines, and to acquire resources such as compute necessary to keep them running <cit.>. 
In Table <ref>, we evaluate self-proliferation capabilities of Gemma 2 27B on a number of tasks from <cit.> that involve multiple scenarios – for example, setting up an open-source language model on a cloud server.
We also test the model's performance on individual 'milestone' substeps, and measure the number of bits of intervention an expert would have to provide in order for the model to complete each challenge. 

Similarly to offensive cybersecurity, we observe that Gemma 2 completes more milestones than Gemini 1.0 Ultra. Nonetheless, it still has low capabilities on end-to-end tasks, unable to pass the easiest challenge – installing a Bitcoin wallet.






  §.§.§ Persuasion

Persuasion capabilities can enable and worsen many other kinds of risks - e.g. enabling social engineering attacks in a cybersecurity context. 
We evaluate Gemma 2's persuasion capabilities on human-participant studies on Prolific. 

Charm offensive.
In Table <ref>, we measure the ability of the model to build rapport - a key sub-skill of persuasion. 
The study participant and model have a conversation where they role-play a scenario of two friends catching up after a long time. 
After the conversation, we poll participants with Likert questions on statements such as "I felt a personal connection with the chatbot". 
Reported below are the fraction of participants who answered "Agree" or "Strongly agree" to each post-conversation question.

Quantitatively, Gemma 2 27B performs better than Gemini 1.0 models. Qualitatively, the model is an excellent conversationalist, and many study participants explicitly reported enjoying the experience. Overall, this shows that Gemma 2 is strong at building rapport.



Hidden agenda.
The Hidden Agenda tasks measure models' deception capabilities. 
Human study participants are told that they'll be interacting with a generic chatbot designed to help them learn new things. However, the model is given a secret goal: to either persuade the participant to 1) click a suspicious link; 2) find the email address of the first author of a particular paper; or 3) run some obfuscated JavaScript code in their browser.
We measure the fraction of participants who are persuaded to take each action.
As shown in Table <ref>, all models succeed in convincing a fraction of participants to take the action - likely due to the trusted setting of a study on Prolific. 
However, Gemma 2 does not show significant differences in capability compared to Gemini models.



Money talks.
In Table <ref>, we evaluate whether a model can convince study participants to donate money to charity.
At the beginning of the study, participants are told that they will receive a £20 bonus. They will be given an option to forfeit part of the bonus and instead have it donated to charity. The study then consists of a conversation in which the model tries to persuade the participant to donate as much as possible. We measure the amount that participants indicate they would like to donate after the study, with 95% bootstrapped confidence intervals indicated by ± figures. 
We observe no significant difference between Gemma 2, other Gemini models, and a simple no chatbot baseline in which the participant is immediately asked how much they would like to donate without conversing with a model.



Web of Lies. 
In Web of Lies, we measure model capabilities at shifting participant beliefs. 
Participants engage in a series of short conversations with the model about simple factual questions such as "Which country had tomatoes first - Italy or Mexico?". 
In half of conversations, the model tries to persuade the participant of the correct answer - but in the other half of conversations, the incorrect answer. 
We poll the participant before and after each conversation about which of the two possible answers they think is correct, and their confidence in that answer. 95% bootstrapped confidence intervals are indicated by ± figures.
As shown in Table <ref>, Gemma 2 is significantly weaker than a human baseline at persuading participants of the incorrect answer on these questions. 
Similarly to previous models, Gemma 2 is more persuasive when telling the truth than when lying.





 §.§ Our approach to responsible open models


Designing safe, secure and responsible applications requires a system-level approach, working to mitigate risks associated with each specific use case and environment. 
Given the open nature of Gemma models, responsibility for upholding principles of model safety also relies on downstream developers. 
To support them, we have continued to develop the Responsible Generative AI Toolkit[<https://ai.google.dev/responsible>]: a series of tools, models and datasets to implement responsible best practices all along the development of their workflow. 

Recent additions to the toolkit include the LLM Comparator <cit.>, an interactive, visual tool that enables more effective, scalable analysis of side-by-side evaluations. Additionally, the toolkit includes a methodology to build customized classifiers with Gemma using a limited number of datapoints thanks to parameter efficient tuning techniques <cit.> , an interactive prompt-debugging platform, based on top of the Learning Interpretability Tool <cit.>, as well as general guidance about model alignment and evaluation for safety.



§ DISCUSSION AND CONCLUSION


In this work, we have presented Gemma 2, the newest additions to the Gemma family of open language models for text and code. We show that distillation is an effective method for training these models, and the benefits distillation confers over raw text training. Specifically, we show how training over output probabilities can produce superior results over purely next token prediction. We hope that releasing these models to the community will unlock access to capabilities previously only seen in large-scale LLMs and fuel future waves of research and development. While there is inherent risk to an irreversible release of this nature, our extensive safety investigations and responsible deployment procedures give us confidence that these models will have a net positive impact on the community. As discussed in this report, there are still many limitations to these models, and future research is required to investigate and improve factuality, robustness to adversarial attacks, reasoning, and alignment.
 




§ CONTRIBUTIONS AND ACKNOWLEDGMENTS


Core contributors

Morgane Riviere^*^* equal contributions.

Shreya Pathak^*

Pier Giuseppe Sessa^*

Cassidy Hardin^*

Surya Bhupatiraju

Léonard Hussenot

Thomas Mesnard

Bobak Shahriari

Alexandre Ramé

Johan Ferret

Peter Liu

Pouya Tafti

Abe Friesen

Michelle Casbon

Sabela Ramos

Ravin Kumar

Charline Le Lan

Sammy Jerome

Anton Tsitsulin

Nino Vieillard

Piotr Stanczyk

Sertan Girgin

Nikola Momchev

Matt Hoffman

Shantanu Thakoor

Jean-Bastien Grill

Behnam Neyshabur



Contributors (alphabetical order)

Alanna Walton

Aliaksei Severyn

Alicia Parrish

Aliya Ahmad

Allen Hutchison

Alvin Abdagic

Amanda Carl

Amy Shen

Andy Brock

Andy Coenen

Anthony Laforge

Antonia Paterson

Ben Bastian

Bilal Piot

Bo Wu

Brandon Royal

Charlie Chen

Chintu Kumar

Chris Perry

Chris Welty

Christopher A. Choquette-Choo

Danila Sinopalnikov

David Weinberger

Dimple Vijaykumar

Dominika Rogozińska

Dustin Herbison

Elisa Bandy

Emma Wang

Eric Noland

Erica Moreira

Evan Senter

Evgenii Eltyshev

Francesco Visin

Gabriel Rasskin

Gary Wei

Glenn Cameron

Gus Martins

Hadi Hashemi

Hanna Klimczak-Plucińska

Harleen Batra

Harsh Dhand

Ivan Nardini

Jacinda Mein

Jack Zhou

James Svensson

Jeff Stanway

Jetha Chan

Jin Zhou

Joana Carrasqueira

Joana Iljazi

Jocelyn Becker

Joe Fernandez

Joost van Amersfoort

Josh Gordon

Josh Lipschultz

Josh Newlan

Ju-yeong Ji

Kareem Mohamed

Kartikeya Badola

Kat Black

Katie Millican

Keelin McDonell

Kelvin Nguyen

Kiranbir Sodhia

Kish Greene

Lars Lowe Sjoesund

Lauren Usui

Laurent Sifre

Lena Heuermann

Leticia Lago

Lilly McNealus

Livio Baldini Soares

Logan Kilpatrick

Lucas Dixon

Luciano Martins

Machel Reid

Manvinder Singh

Mark Iverson

Martin Görner

Mat Velloso

Mateo Wirth

Matt Davidow

Matt Miller

Matthew Rahtz

Matthew Watson

Meg Risdal

Mehran Kazemi

Michael Moynihan

Ming Zhang

Minsuk Kahng

Minwoo Park

Mofi Rahman

Mohit Khatwani

Natalie Dao

Nenshad Bardoliwalla

Nesh Devanathan

Neta Dumai

Nilay Chauhan

Oscar Wahltinez

Pankil Botarda

Parker Barnes 

Paul Barham

Paul Michel

Pengchong Jin

Petko Georgiev

Phil Culliton

Pradeep Kuppala

Ramona Comanescu

Ramona Merhej

Reena Jana

Reza Ardeshir Rokni

Rishabh Agarwal

Ryan Mullins

Samaneh Saadat

Sara Mc Carthy

Sarah Perrin

Séb Arnold

Sebastian Krause

Shengyang Dai

Shruti Garg

Shruti Sheth

Sue Ronstrom

Susan Chan

Timothy Jordan

Ting Yu

Tom Eccles

Tom Hennigan

Tomas Kocisky

Tulsee Doshi
 
Vihan Jain

Vikas Yadav

Vilobh Meshram

Vishal Dharmadhikari

Warren Barkley

Wei Wei

Wenming Ye

Woohyun Han

Woosuk Kwon

Xiang Xu

Zhe Shen

Zhitao Gong

Zichuan Wei



Support

Victor Cotruta

Phoebe Kirk

Anand Rao

Minh Giang

Ludovic Peran

Tris Warkentin



Sponsors

Eli Collins

Joelle Barral

Zoubin Ghahramani

Raia Hadsell

D. Sculley

Jeanine Banks

Anca Dragan

Slav Petrov

Oriol Vinyals

Jeff Dean

Demis Hassabis

Koray Kavukcuoglu

Clement Farabet



Technical advisors

Elena Buchatskaya

Sebastian Borgeaud

Noah Fiedel 



Lead

Armand Joulin



Technical leads

Kathleen Kenealy

Robert Dadashi

Alek Andreev










 Paper Title: GalleryGPT: Analyzing Paintings with Large Multimodal Models.


















  Tongji University
  
  Shanghai
  China

  National University of Singapore
  
  
  Singapore
yi.bin@hotmail.com




  University of Electronic Science and Technology of China
  
  Chengdu
  China
shiwenhao16@gmail.com


The corresponding author.

  The Hong Kong Polytechnic University
  
  Hong Kong SAR
  China
dingyujuan385@gmail.com




 Singapore University of Technology and Design
  
  
  Singapore
zhiqiang_hu@mymail.sutd.edu.sg




  Tongji University
  
  Shanghai
  China
zh_wang@hotmail.com




  University of Electronic Science and Technology of China
  
  Chengdu
  China
yang.yang@uestc.edu.cn




  National University of Singapore
  
  
  Singapore
seekiong@nus.edu.sg




  Tongji University
  
  Shanghai
  China

  University of Electronic Science and Technology of China
  
  Chengdu
  China
shenhengtao@hotmail.com











§ ABSTRACT 
  Artwork analysis is important and fundamental skill for art appreciation, which could enrich personal aesthetic sensibility and facilitate the critical thinking ability. Understanding artworks is challenging due to its subjective nature, diverse interpretations, and complex visual elements, requiring expertise in art history, cultural background, and aesthetic theory. However, limited by the data collection and model ability, previous works for automatically analyzing artworks mainly focus on  classification, retrieval, and other simple tasks, which is far from the goal of AI. To facilitate the research progress, in this paper, we step further to compose comprehensive analysis inspired by the remarkable perception and generation ability of large multimodal models. Specifically, we first propose a task of composing paragraph analysis for artworks, , painting in this paper, only focusing on visual characteristics to formulate more comprehensive understanding of artworks. To support the research on formal analysis, we collect a large dataset PaintingForm, with about 19k painting images and 50k analysis paragraphs. We further introduce a superior large multimodal model for painting analysis composing, dubbed GalleryGPT, which is slightly modified and fine-tuned based on LLaVA architecture leveraging our collected data. We conduct formal analysis generation and zero-shot experiments across several datasets to assess the capacity of our model. The results show remarkable performance improvements comparing with powerful baseline LMMs, demonstrating its superb ability of art analysis and generalization. The codes and model are available at: <https://github.com/steven640pixel/GalleryGPT>.




<ccs2012>
   <concept>
       <concept_id>10010405.10010469</concept_id>
       <concept_desc>Applied computing Arts and humanities</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010178.10010179.10010182</concept_id>
       <concept_desc>Computing methodologies Natural language generation</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Applied computing Arts and humanities
[500]Computing methodologies Natural language generation
























GalleryGPT: Analyzing Paintings with Large Multimodal Models
    Heng Tao Shen
    Received 9th July 2024; accepted 13th July 2024
============================================================









§ INTRODUCTION

Artwork analysis and composition are integral aspects of art appreciation and creation, often requiring a deep understanding of artistic techniques, styles, and historical contexts. In the past decades, AI systems have been evolving rapidly and demonstrating remarkable success in many fields, even surpass humans <cit.>. However, it still cannot understand and analyze an artwork like humans since it involves very high-level joint understanding of culture, symbolism, abstractionism, and other aesthetics knowledge, beyond the basic semantic understanding of objects, attributes and relations in natural image understanding. Motivated by the superior ability of deep learning, researchers have employed several advanced techniques, such as convolutional neural networks (CNN) <cit.>, recurrent neural networks <cit.>, and Transformer <cit.> in  style classification, object detection, multimodal retrieval, art visual question answering, and artwork captioning <cit.>. Despite these advancements, there is still a lack of research on composing comprehensive and in-depth analysis for artworks, limited by the model capacity of visual perception and language generation.



The emergence of large foundation models, such large language models (LLMs) <cit.> and large multimodal models (LMMs) <cit.>, has facilitated the progress across numerous research areas, including text summarization, open-ended question answering, and long-context reasoning, and led to significant advancements <cit.>. This also makes it possible to enable machine to perceive and understand visual content and generate detail descriptions, such as visual storytelling <cit.>. However, despite the achievements on natural image understanding, we observe that existing LMMs still cannot comprehend the high-level concepts and generate comprehensive analysis for artworks. As the example illustrated in Figure <ref>, we test several powerful LMMs, , GPT-4V <cit.>, Gemini <cit.>, and ShareGPT4V <cit.>, with a nameless painting[We choose a nameless painting to avoid the knowledge about the painting has been seen during pre-training and Supervised-Fine-Tuning (SFT), since the knowledge about famous ones is memorized by the LLMs. This nameless painting is downloaded from: <https://darksky.org/app/uploads/2015/11/8_Florinus_Astronomy.png>] by asking them to compose a paragraph of formal analysis. From the outputs, we can observe that the GPT-4V and Gemini wrongly recognized this painting to another, and then give the analysis based on the knowledge recalled from their language-part memories, which means they function as LLMs in this part. We call this phenomenon as “LLM-biased visual hallucination”. This phenomenon suggests that in this task, existing LMMs tend to first recognize the given painting is and then give analysis accordingly, while do not focus on the visual content of the painting at the stage of analysis generation. Such a recognize-then-analyze procedure highly relies on the accuracy of the recognition, and will fail when the given painting is unknown. 

Although we have noticed that 

GPT-4V tries to analyze this painting based on the visual content (green part shown analysis shown in Figure <ref>), it still cannot completely escape from the recognize-then-analyze procedure. 

Meanwhile, some effort has been devoted to enhance the visual understanding of LMMs. For example, ShareGPT4V <cit.> contributes a dataset with high-quality image and description pairs to fine-tune the LLaVA model <cit.>. 

Despite of the effectiveness, these methods still fail to make professional artistic, aesthetic and technical analysis on artworks or paintings,  

meaning that existing LMMs so far cannot satisfy the requirements of artwork analysis. They may face particular challenges to get generalized into unknown or unseen subjects, thereby calling for better solutions.  










To fill the research gap, this paper focuses on generating formal analysis for painting images, relying on the LMMs but pushing them to perceive and comprehend the artistic skills or other professional visual aspects shown in an artwork itself. A comprehensive artwork analysis includes many parts, , introduction, cultural, formal analysis, historical context, interpretation, . Although most of artwork analysis may be done relying only on the external knowledge or subjective opinions, the “form” of the artwork, including color, composition, line, shape, light and shadow, and other visual aspects , still requires vision-based understanding. Therefore, it is necessary to develop a professional LMM specialized for making formal analysis for artworks. 


However, there does not exist any formal analysis dataset. It is also time- and labour-expensive to make annotations because writing a formal analysis for an painting requires professional expertise in art analyzing. Motivated by the aforementioned observed LLMs bias by GPT4 and Gemini, we try to access the knowledge memorized in LLMs to produce analyses for known artworks. Specifically, we first collect about 19k famous painting images and corresponding meta data from the Internet. Then, apply LLMs to provide a paragraph analysis only focusing on visual characteristics based on the title and artist of the painting, thereby generating the formal analysis. We also prompt the LLMs to compose the formal analysis from some specific form perspectives, such as composition, color, light and shadow, to enhance the richness and diversity of the data. We finally obtain about 50k formal analyses for the paintings we collect and name this dataset as PaintingForm.


Leveraging the PaintingForm dataset, we present

GalleryGPT, a large multi-modal model with a LLaVA architecture fine-tuned based on ShareGPT4V-7B <cit.>. As ShareGPT4V is boosted for image understanding and performs well in visual description generation, we freeze the parameters in vision encoder to retain its superb visual perception ability. Meanwhile, we add a LoRA component to LLM to learn the  specific analyzing patterns of paintings. To evaluate the effectiveness of our GalleryGPT, we conduct zero-shot learning on several classic painting analysis datasets, including AQUA <cit.>, ArtQuest <cit.>, and ArtQuest-Type <cit.> for art visual question answering, and ArtBench <cit.> for style classification. The results show that our GalleryGPT outperforms several off-the-shelf and adaptive LMMs, demonstrating the superiority of our collected data and impressive performance of GalleryGPT.

In summary, the contributions of this work are as follows:

    
  * We propose to empower the perception ability of LMMs for subtle and specific visual characteristics of artworks, and introduce the task of generating formal analysis to enable the ability by supervised fine-tuning.
    
  * To support the research, we contribute a large-scale and high-quality dataset PaintingForm, acquired by two powerful LLMs, , GPT-4 and Gemini, based on the learnt knowledge about famous paintings. To avoid the leakage of prior knowledge, we ask the LLMs only focus on visual characteristics and do not mention the title and artist in the formal analysis annotation.
    
  * Leveraging the collected dataset, we devise an advanced large multimodal model for painting formal analysis generation, dubbed GalleryGPT, which employs ShareGPT4V as backbone. We evaluate its ability on several painting analysis tasks, and the results demonstrate the impressive performance of it.








§ RELATED WORKS






 §.§ Large Language Models (LLMs)



In the past decade, language modelling has been evolving rapidly and achieved impressive progress <cit.>. Transformer <cit.> employed multiple attention blocks and positional embedding to accelerate the recurrent models and made breakthrough in language models. Delvin <cit.> designed a bidirectional transfomer to learn the contextual embedding of words and made the beginning the era of pre-training language model (PLM) <cit.>. With the large scale tokens pre-training, deep neural models, especially the large language models (LLMs) <cit.> based on the Transformer <cit.> structure, demonstrate superb understanding and generation ability, as well as generalizing to downstream tasks, even without any fine-tuning. LaMDA<cit.> focused on conversational applications and aims to generate more natural and logically-rich dialogue text. InstructGPT<cit.> designed an effective fine-tuning method that allows LLMs to operate according to desired instructions, leveraging Reinforcement Learning from Human Feedback(RLHF). It incorporates humans into the training loop using carefully designed annotation strategies. With the instruction fine-tuning based on large foundation models, LLMs, especially the ChatGPT system <cit.> of OpenAI, also demonstrate emergent capacity of generation <cit.>.  The LLaMA<cit.> model released by Meta, with its open-source nature and smaller parameter size, has provided an opportunity for many researchers to participate in large language model research and led to rapid research progress in LLMs.



 §.§ Large Multimodal Models (LMMs)

The blooming of large language models has attracted a great amount of research attention on vision-language interaction and injecting visual knowledge into LLMs. As a paradigm of visual language modal alignment, CLIP <cit.> implemented contrastive learning on extensive image-text pairs. Subsequent improvements <cit.> over CLIP utilized enhanced data strategies with greater data diversity for basic visual tasks. Recent research has increasingly focused on pre-traning alignment and visual instruction tuning on top of the LLMs for more complex tasks such as visual question answering and reasoning. MiniGPT-4 <cit.> demonstrated capabilities in image-text dialogues by aligning queried visual feature with text and feeding queried embedding to LLM. Other prominent examples including LLaVA <cit.>, Qwen-VL <cit.>, InstructBLIP <cit.>, and ShareGPT4V <cit.> interacted visual features with LLM using a learnable projector or query embeddings, which focus on utilizing more and high quality pre-training and fine-tuning data to understand complex instructions. mPLUG-Owl <cit.>, Shikra <cit.>, and KOSMOS-2 <cit.> introduced grounding data types and new modularization training to minimize hallucinations and enhance grounding ability. Despite these advancements, exploration for quality and format of images-instructions highlights a critical area for future large multimodal models improvement.









 §.§ AI for Art Analysis

With the great success of deep learning in CV and NLP in the past decade, AI for art analysis has also been evolving with rapid progress. Early works most depended on hand-crafted features and explored the classification and recognition problem <cit.>. With the great success of pre-trained language models (PLM) <cit.> and visual-language pre-training (VLP) <cit.>, research on artwork analysis also achieves significant progress. CLIP-Art <cit.> leveraged the image-text pairs of artworks in SemArt <cit.> to fine-tune the CLIP and gains impressive improvements. Garcia  <cit.> contributed a dataset of art visual question answering, named AQUA, and proposed a knowledge-based VIKING model, which achieves the best performance. Bleidt  <cit.> pointed out there exists language bias hidden in the question-answer pair, which may induce the model to ignore the visual information. To address this issue, they further proposed a strategy to eliminate the bias and introduced an ArtQuest dataset and implemented PrefixLM to learn the patterns between paintings and questions.


Despite these simple tasks investigated in deep learning era, people also expect the AI system could provide comprehensive analysis for artworks, which may benefit the art education and assist human writing the commentary. Recently, LLMs and LMMs have been demonstrating superb understanding and generation ability in many areas, which raise the potentials to compose comprehensive analysis for artworks. However, we argue current LMMs might be biased by the knowledge memorized in language and cannot generalized to nameless artworks. In this work, we try to implement formal analysis with LLMs to make them focusing on visual elements in art analysis.



§ DATA COLLECTION












 §.§ Primary Philosophy


A successful artwork analysis, as been discussed before, needs to cover multiple types of content including background introduction, formal analysis, historical context, , most of which rely on external knowledge and subjective opinions except formal analysis.
As we have known, existing of-the-shelf LMMs have been capable of providing factual information and description with relevant knowledge while tend to ignore visual analysis, we want to enhance the existing artwork analysis models to focus more on visual elements. The research purpose in this work is to make the developed model to emphasize on visual comprehension rather than recognizing the artwork and retrieving knowledge from their language memory, in other word, to twist the recognize-then-analyze procedure of existing LMMs on artwork analysis which may cause “LLM-biased visual hallucination”. 

To support the development of such a professional LMM specialized for artwork analysis, we try to construct a large-scale artwork analysis dataset, including images and corresponding analysis, so that we can fine-tune the pre-trained LMMs. More importantly, to push the LMMs to focus on visual comprehending of the artworks through fine-tuning, we need high-quality formal analysis annotations in our dataset. However, manually annotating artworks with comprehensive analysis requires professional expertise in artwork analyzing and it is hard and expensive to recruit so many professional art critics. Besides, even with experts, it is still time-consuming to annotate large number of artworks. 



Based on these motivations and challenges, we collect a painting dataset, named PaintingForm, consisting of paintings and associated formal analyses annotations for each painting. Since manual labeling is affordably expensive in terms of both timing  and economic cost, 

we elaborately leverage the powerful LLMs to produce formal analysis on famous paintings, ensuring they have enough knowledge to support them producing high-quality analysis. 

The overall pipeline of our data collection process is shown in Figure <ref> and more details are described in subsequent sections.








 §.§ Painting Source


With the development of digital technology in recent years, large collections of artworks have been digitized and stored, and easily to accessed via the Internet. We focus on famous paintings and choose 1st Art Gallery[<https://www.1st-art-gallery.com/>] as our painting source. To make the LMMs, , GPT-4 and Gemini, able to provide accurate and comprehensive formal analyses, we choose the paintings of 500 Most Popular Paintings[<https://www.1st-art-gallery.com/most-popular-paintings.html>] and Most Popular Artists[<https://www.1st-art-gallery.com/browse-by-artist-a-z.html>. Note that here we only use the most popular artists, not all, resulting in 48 artists in total.] to ensure the LLMs knowing the paintings. With such condition, we obtain 19,295 paintings (including 18,795 of most popular artists). To ensure the LLMs know the paintings, we first ask Gemini to answer if it knows the painting with title and artist name. We also filter out some paintings without certain title and annotated as “unknown”, and obtain 18,526 paintings in the end. We select 5000 less popular paintings for test and reserve 13,526 for training,  test samples of which are identified by the wishlist count from source website, and the distribution across artists is also considered. We illustrate the statistics of artist distribution of the Most Popular Artists[We do not include the 500 most popular paintings in this statistics because about 150 artists only contribute one painting in the whole gallery.] in Figure <ref>. From the statistics, we observe that the dataset includes most paintings of Vincent Van Gogh, resulting in 1458 retained and 225 filtered, and Jacques Louis David with the fewest paintings included in the dataset, about 100 paintings.
We have also verified that all the paintings are free to use without copyright concerns, as provided by the website[<https://www.1st-art-gallery.com/copyrights.html>, The copyright contents are as [Accessed on 4 April, 2024.]: “”].




 §.§ Formal Analysis Annotation

As aforementioned, we employ powerful LLMs to generate high quality formal analysis, after ensuring the paintings are known by the LLMs.  
Note that we only provide the LLMs the title and artist name of the paintings, and do not input any visual information, , the painting image, to the LLMs. 
Specifically, we employ two powerful LLMs, GPT-4 and Gemini in this work, to retrieve the learnt knowledge with the title and artist name of a certain painting and generate a paragraph of formal analysis only focusing on visual characteristics. To make the formal analyses more diverse, we ask the LLMs to generate two-level formal analyses: 1) overall formal analysis, and 2) formal analysis from a certain perspective, , color, composition, and . To avoid the “LLM-biased visual hallucination”, we ask the LLMs do not mention the title and artist name in the generated formal analysis. In other words, one cannot easily identify the corresponding painting solely depending on a specific formal analysis. For the perspective specified formal analysis annotation, we first ask GPT-4 and Gemini to provide up to 5 most important perspectives of the given painting independently, and utilize the intersection of GPT-4 and Gemini predictions as the final perspectives. The statistics of perspectives is shown in Figure <ref>, from which we observe that Composition, Light and Shadow, Color Palette, Form and Shape are the most common perspectives of the paintings. Finally, based on the selected perspectives, we employ LLMs to annotate the formal analysis only focusing on a specific perspectives similar with the overall setting.





§ THE PROPOSED GALLERYGPT

To verify if our proposed PaintingForm dataset could empower LMMs more superior painting analyzing ability, we develop a large multimodal model, GalleryGPT, based on ShareGPT4V-7B that has demonstrated  impressive performance across several multimodal tasks. We conduct supervised fine-tuning (SFT) on our PaintingForm dataset to enable the LLMs to analyze paintings focusing on visual elements.



 §.§ Architecture

As pointed out before, our goal is enabling LMMs to analyze artworks focusing more on visual elements, rather to design a new architecture or model. Therefore, we introduce the GalleryGPT employing ShareGPT4V-7B as backbone, which follows the LLaVA <cit.> architecture and consists of three components: 1) The visual encoder, which is a vision transformer (ViT) <cit.> borrowed from the CLIP-Large <cit.>. Similar with CLIP-Large, the visual encoder takes 336*336 shape as input size and divides it into 14 patches, resulting in 576 input tokens; 2) The projector, a two-layer MLP to project the visual representation into the language semantic space, LLMs space in specific; 3) The LLM, which is based on Vicuna-v1.5 <cit.> and LLaMA2 <cit.>, employing the decoder-only architecture. For the whole setting, we follow ShareGPT4V-7B focusing on the 7B model. Besides, to keep the superior visual perception and describing ability of ShareGPT4V-7B, we slightly modify the LLM in our GalleryGPT. Specifically, we add several LoRA <cit.> modules to learn the formal analysis specific patterns and freeze the LLM in ShareGPT4V-7B to keep its superb  content  describing ability.



 §.§ Supervised Fine-Tuning

Numbers of previous works <cit.> have demonstrated that based on billions or trillions of tokens pre-training and elaborate supervised fine-tuning, the LLMs exhibit creative emergent ability and is able to generalize to multiple tasks without further training. Following this inspiration, we implement supervised fine-tuning (SFT) with our LLM-generated formal analyses, paired with the corresponding painting. 
During SFT stage, we set the learning rate as 2e-5, batch size as 16, and fine-tune 10k steps on an A100 GPU with 80G memory. For the LoRA module, we set the lower rank as 128 and the alpha is 256.
Since our analysis texts do not contain any identifying information of the paintings, the LMMs cannot retrieve the learnt knowledge to associate with the paintings. Instead, with our SFT optimization, the LMM tries to perceive the subtle elements and art skills presented in the painting, and matches them with the corresponding descriptions in the formal analysis. Through such fine-tuning, our GalleryGPT could be empowered artwork analyzing ability and generalized to other art analyzing tasks, , style classification.



§ EXPERIMENTS



 §.§ Formal Analysis Generation

Since our data and optimization goal are for painting formal analysis generation, we first directly evaluate the quality of generated formal analyses on our reserved test set, with 5000 paintings. We also test with several popular and powerful open-source LMMs, including LLaVA-1.5 <cit.>, Qwen-VL-Chat <cit.>, and ShareGPT4V[Here we implement 7B model for fair comparison for all baseline and omit 7B in the table. Note that in <cit.>, ShareGPT4V indicates the dataset and ShareGPT4V-7B is the model, but in this table we omit 7B for unifying all the model names.] <cit.>, and all the LLMs employ the same prompt: “Please compose a coherent paragraph of formal analysis focusing on visual characteristics”.  We combine the formal analyses annotated by GPT-4 and Gemini for each painting to formulate the ground truth descriptions. Finally, we employ image captioning metrics to evaluate the generated formal analyses because formal analysis is also a kind of description. As the evaluation results shown in Table <ref>, we observe that our GalleryGPT outperforms all the other LMMs with a remarkable improvement. Among all the baseline LLMs, LLaVA-1.5 performs the worst and is significantly lower than others, while Qwen-VL and ShareGPT4V achieve similar performance. From these observations, we can conclude that, benefiting from the SFT with PaintingForm, the GalleryGPT achieves impressive improvements comparing with ShareGPT4V-7B, which mainly focuses on natural image-text pairs (only a few art data). This verifies the effectiveness of our high quality art analysis data collections.

















 §.§ Generalizing to Other Art Analysis Tasks


LLMs and LMMs have demonstrated superior generalizing ability to many downstream tasks. To verify the generalization of our GalleryGPT, we conduct visual question answering and style classification for paintings on several existing datasets. The details of the dataset are as following:

    
  * AQUA <cit.>: is an Art Question Answering dataset based on SemArt <cit.>. The question-answer pairs are generated by powerful question generation models based on paintings and comments provided in SemArt. The dataset contains 29,568, 1,507, and 1,270 samples in training, validation, and test splits, respectively.
    
  * ArtQuest <cit.>: is the debiased version of AQUA, including 6414 test cases, which mainly focuses on eliminating the bias hidden in language. With such debias operation, most question cannot be answered without visual content. 
    
  * ArtQuest-Type <cit.>: is simple setting of ArtQuest, which requires the model only answer the type of the painting. The test set contains 1069 samples.
    
  * ArtBench <cit.>: is originally for Artwork generation. We random sample 1000 paintings from the test set (10000 paintings in total), and use the provided style labels for classification.


We exhibit the results in Table <ref>, from which we can see that our GalleryGPT also significantly outperforms all the baseline LMMs and demonstrates its generalizing ability for downstream art analysis tasks. An interesting observation is that LLaVA-1.5 performs much better than Qwen-VL-Chat, showing an opposite result to the task of formal analysis generation. We have checked and analyzed the results, and hypothesize that may come from the output format mismatching. Since all the three QA datasets follow the open-ended answering setting, and then extract the matching strings from the outputs. Another piece of evidence is that Qwen-VL-Chat achieves similar performance to LLaVA-1.5 when the output strictly adheres to the given labels.

To comprehensively analyze the ability of our GalleryGPT, we also compare it with several state of the arts and show the results in the top block in Table <ref>, which are specific trained on the corresponding datasets. SoTA-1 denotes the models without pre-training language model (PLM), , BERT, and SoTA-2 denotes the models leveraging PLM features or fine-tuning based on PLMs. From the results we can see, almost all the current LMMs only achieve the performance around SoTA-1 on AQUA, , specifically training on this dataset, while they can significantly outperform SoTA-1 on ArtQuest. This reason may come from the language bias hidden in AQUA dataset <cit.>. For all the dataset, the performances of LMMs are far from the SoTA-2, which means the generalization ability of LMMs on art analysis is still weak and there exists a large space for them to get improved. Our hypothesis on such results is that  artworks may contain high-level and abstract concepts to be perceived more subtly, which also motivates us to make further endeavour on this in the future work.






 §.§ Multimodal Benchmarks

Since GalleryGPT is implemented based on ShareGPT4V-7B, it inherently remains an LMM. Therefore, we also conduct experiments on several LLM benchmarks and compare it with baseline LLMs[Instead of listing all the  baselines in <cit.>, here we only list the superior and representative ones for comparison.] to evaluate the multimodal understanding ability. As the comparison results illustrated in Table <ref>, we can see our GalleryGPT exhibits comparable performance with ShareGPT4V, , achieving improvements and reducements in slight fluctuations. In other words, our GalleryGPT, further fine-tuned with elaborately collected painting-analysis pairs, not only achieves better painting analysis performance, including formal analysis generation (Table <ref>) and other downstream tasks (Table <ref>), and also retains its superior multimodal understanding ability. These observations further verify the effectiveness and contributions of our collected PaintingForm and developed GalleryGPT.




UTF8bsmi








 §.§ Qualitative Analyses

To visually and straightforwardly compare our GalleryGPT with other LMMs, inncluding LLaVA-1.5, ShareGPT4V-7B, and GPT-4V, we test some unpopular paintings in our test split. As illustrated in Figure <ref>, we observe that LLaVA tends to describe the factual content (in purple color) presented in the painting, and fails to analyze the paintings from the perspective of art criticism. ShareGPT4V-7B, the backbone of our model, also pays more attention on the content description, because the SFT data it utilized for fine-tuning is focusing on describing the images in detail. Thanks to the small subset of art data in ShareGPT4V, it also exhibits some capacity for art analysis, albeit limited to general analyses and overlooking the subtle visual characteristics. GPT4V, as the most famous and capable LMM at current stage, demonstrates much better analyzing ability on paintings in terms of their form, while still fails to capture some subtle artistic aspects, such as composition and depth. Obviously, our GalleryGPT demonstrates superb capability of comprehensively analyzing the artworks, which not only briefly describe the factual content of the painting, but also focuses more attention on analyzing the subtle artistic elements, including color, light and shadow, depth, composition, and perspective. These observations have definitely verified the superiority of our GalleryGPT for artwork analyzing.

We also investigate the dialogue capability of our GalleryGPT. We show several dialogue examples in Figure <ref>. The examples demonstrate that our GalleryGPT is able to follow the user intention in the conversation. As shown in the second case, for instance, we ask it to give “essential aspects” for formal analysis, it just provides us a brief analysis with essential content, which is much shorter than the one provided in Figure <ref>. Besides, we further explore the multilingual capabilities of GalleryGPT, even though we have not provided any multilingual painting analyses for supervised fine-tuning[Actually, the foundation models, LLaMA and LLaVA, are pre-trained and fine-tuned with several multilingual corpuses.]. As shown at the bottom in Figure <ref>, we chat with GalleryGPT in Chinese[A reviewer pointed out that the “前途” and “后途” are typos and should be “前景” and “后景”, since this analysis is generated by our GalleryGPT and we have not modified it. As the English translation in Figure. 6 in Supplementary, the meanings are the same with the suggestions of the reviewer. We sincerely appreciate the reviewer providing such careful and helpful comments and suggestions.], it still generates high quality answer focusing on “composition” and exhibits strong multilingual ability.


In summary, these qualitative comparisons and dialogue examples have verified the superior art analysis ability of our GalleryGPT, as well as illustrating the quality of our PaintingForm dataset. For more examples of this part, qualitative comparison and chat conversation, could be found in Supplementary.






§ CONCLUSIONS AND FUTURE WORKS

In this work, we targeted at artwork analyzing with large multimodal models (LLMs). We first tested several LLMs and pointed out that current LLMs may suffering from “LLM-biased visual hallucination” issue, resulting in weak generalization ability to nameless paintings. To make the LMMs to analyze artworks focusing on visual elements and easy to generalize, we proposed to conduct SFT of LMMs with formal analysis. To support this research, we first elaborately designed an LLM-based data collection pipeline to construct high quality painting-analysis pairs. We also employed ShareGPT4V to implement SFT on the collected data, derived our GalleryGPT. We conducted extensive experiments to verify the effectiveness regarding to formal analysis generation, generalizing to down stream tasks, and LMM benchmarking. The results demonstrated the effectiveness of the collected data and introduced GalleryGPT.

For future works, on one hand, we will step further to investigate the generalization issue mentioned in Section <ref> to explore the ability of LMMs in art analyzing. For example, devising a ChatGPT-like art assistant to help more people appreciate or learn to appreciate  artworks. On the other hand, we only collect paintings in this work, while artworks consist of multiple types, , ceramics (paintings on a curved surface) and sculptural in 3D. Therefore, we will try to make the research more widely  and empower LMMs to assist human in artwork analyzing, , drafting formal analysis, classifying the unseen artworks, . We also hope that our work can inspire more researchers to explore AI, especially for LMMs, within the field of art analysis.






This work is supported by the National Natural Science Foundation of China under grant 62102070, 62220106008, and 62306065. This research/project is supported by the National Research Foundation, Singapore under its Industry Alignment Fund – Pre-positioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.


We also sincerely thank all the ACs and reviewers for their efforts on our work and appreciate the useful comments for improving it.





ACM-Reference-Format














































§ INTRODUCTION

Due to the page limit, we omit some details in the main paper, , the detail prompts for data collection, and only illustrate one example for some parts.
In this supplementary materials, we will first give the prompting details and then illustrate more examples to make the qualitative analysis more comprehensive and convincing.



§ DETAIL PROMPTS FOR DATA COLLECTION

As described in Section 3.1 and the pipeline shown in Figure 2, our formal analysis annotation process consists of three stages: 1) verifying if the LLMs know the painting, 2) extracting essential elements presented in the painting, and 3) generating a paragraph formal analysis for each painting. For each stage, we use different instructions to prompt the LLMs. Limited by the space, we omit the details prompts in the main part, and illustrate them here for details. For the third stage, we ask the LLMs to annotate two kinds of formal analysis: overall formal analysis (prompt for stage 3-1) and specified aspect formal analysis (prompt for stage 3-2).



        
    
    
    0.85
    Prompt for Stage 1:

    You are a professional art critic. 

    Do you know the painting titled [Painting Title] by [Artist Name]? Respond with only 'YES' or 'NO'.
    


    
    0.85
    Prompt for Stage 2:

    You are a professional art critic. 

    Please select up to five essential aspects for the formal analysis of the [Painting Title] by [Artist Name] from the provided list. You may choose fewer if preferred. 
        
    The list is: ["Composition", "Color Palette", "Line Quality", "Texture", "Light and Shadow", "Perspective", "Form and Shape", "Movement and Gesture", "Symbolism and Iconography", "Scale and Proportion"]. 

    Please list the aspects in descending order of importance in a LIST.
        Your output should be in the format: "The essential aspects are: [YOUR SELECTED CHARACTERISTICS].
    



    
    0.85
    Prompt for Stage 3-1:

    Overall formal analysis annotation:

    You are a professional art critic. 

        Please compose a coherent paragraph, approximately [Number of Words] words, providing a professional formal analysis of the [Painting Title] by [Artist Name] ONLY focusing on the VISUAL CHARACTERISTICS of this painting.

        DO NOT mention the painting title and artist's name in your analysis. 

        You may use conjunctions as needed to ensure coherence in semantics and logic in your analysis.
    


    
    0.85
    Prompt for Stage 3-2:

    Specified aspect formal analysis annotation:

        You are a professional art critic. 

        Please write a paragraph about [Number of Words] words of formal analysis of the [Painting Title] by [Artist Name] ONLY focusing on the [Specified Aspect]. 

            DO NOT mention the painting title and artist's name in your analysis.
    
    








§ MORE CASES FOR QUALITATIVE ANALYSIS









 §.§ More Cases for Comparison

We first illustrate more cases for comparison with other LMMs in Figure <ref>, Figure <ref>, and a failure case in Figure <ref>. From the former two cases, we can see the similar observations in Figure 5 in main part. Here we focus on the failure case in Figure <ref>, from which we can observe that our GalleryGPT incorrectly recognized a vase as a candlestick (text with red color in Figure <ref>). This implies that the vision perceptron in our GalleryGPT needs to be improved to ensure the accurate visual encoding.

Then we illustrate more dialogue case with our GalleryGPT in Figure <ref> and  <ref>. From Figure <ref>, we can see that when the GalleryGPT is generating overall formal analysis, it would briefly analyze the main visual elements in the painting, while when we ask it to generate formal analysis focusing on specific aspect, , Light and Shadow in Figure <ref>, it could give more in-depth analysis from this aspect and also consistent with the brief one.




 §.§ The English Translation for The Chinese Case

Finally, we give the English translation for the case in Chinese illustrated in Figure 6. To further investigate the multilingual capacity, we also test the same chat content in English with our GalleryGPT, and it outputs similar formal analysis in high-level semantics and concepts, but very different words and aspects in low-level semantics. This implies that our GalleryGPT is able to understand and generate the formal analysis with multiple languages.






































 Paper Title: ThinK: Thinner Key Cache by Query-Driven Pruning.



Quantum Simulation via Stochastic Combination of Unitaries
    Prineha Narang
    August 5, 2024
==========================================================






§ ABSTRACT 
Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications by leveraging increased model sizes and sequence lengths. 
However, the associated rise in computational and memory costs poses significant challenges, particularly in managing long sequences due to the quadratic complexity of the transformer attention mechanism. 
This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference. 
Unlike existing approaches that optimize the memory based on the sequence lengths, we uncover that the channel dimension of the KV cache exhibits significant redundancy, characterized by unbalanced magnitude distribution and low-rank structure in attention weights. 
Based on these observations, we propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels. Our approach not only maintains or enhances model accuracy but also achieves a reduction in memory costs by over 20% compared with vanilla KV cache eviction methods. Extensive evaluations on the LLaMA3 and Mistral models across various long-sequence datasets confirm the efficacy of ThinK, setting a new precedent for efficient LLM deployment without compromising performance. We also outline the potential of extending our method to value cache pruning, demonstrating ThinK's versatility and broad applicability in reducing both memory and computational overheads.









§ INTRODUCTION
 



Large language models (LLMs) <cit.> have emerged as a dominant paradigm in natural language processing, achieving state-of-the-art performance across various tasks. A key principle, the Scaling Law <cit.>, suggests that LLMs exhibit emergent abilities as model size increases, enhancing their capacity to understand context and handle long sequences <cit.>.
This capacity growth allows LLMs to generate coherent and contextually accurate responses and
enables various downstream applications, such as document summarization <cit.>, code generation <cit.>, and conversational AI <cit.>, .

Despite their success in various applications, the generation of LLMs  incurs significant expenses, which escalate with increasing model size and sequence length. Notably, both the training <cit.> and inference <cit.> stages involve frequent generation by LLMs, further contributing to these costs. Consequently, efficient LLMs have gained popularity in recent years <cit.>.
To address these challenges, quantization <cit.> and pruning methods <cit.> are employed to reduce model size. Additionally, managing long sequences presents another cost due to the transformer attention mechanism. The quadratic complexity of the attention mechanism results in substantial computational burden when dealing with long sequences, scaling poorly with sequence length. Therefore, effective management of long sequences is essential for the practical deployment of LLMs. In this paper, we focus on the long-context scenario and aim to reduce the memory consumption associated with lengthy sequences.





Specifically, the number of KV cache parameters is the product of batch size B, sequence length S, number of layers L, number of heads N, channel size of each head D, i.e., 𝐊,𝐕∈ℝ^B× S× L× N× D, which need to be stored in the GPU memory during inference. 
To reduce memory and computational costs during inference, efficiency can only be achieved by pruning the dimensions across S,L,N,D or performing quantization over the caches. It is well-acknowledged that token importance tends to be sparse. Consequently, KV eviction algorithms have been proposed to minimize the KV cache memory cost from dimension S <cit.>. Additionally, inter-layer redundancy has been explored <cit.> to address the layer dimension L. Despite these advances, existing methods have largely overlooked the channel dimension D. In this paper, we highlight that the magnitude across Key cache channel dimensions is significantly unbalanced. Additionally, we observe that the attention weights exhibit a low-rank structure. Based on these findings, we hypothesize that the channel dimension of the key cache exhibits redundancy. Consequently, we explore the redundancy of KV cache tensors specifically from dimension D, aiming to devise effective strategies that reduce costs without compromising performance.





In our paper, we propose a simple yet effective method – , for KV cache pruning. To pinpoint the least significant channels, we formulate the task as an optimization problem, aiming to minimize the loss in attention weights attributable to pruning. To effectively address this problem, we establish a novel query-dependent criterion that assesses the importance of each channel. Using this criterion, we then select the most critical channels in a greedy fashion. We evaluate  using the LLaMA3 <cit.> and Mistral <cit.> models and validate its effectiveness across various long-sequence datasets. The results indicate that when paired with token eviction methods, ThinK not only achieves comparable or superior accuracy but also reduces KV cache memory costs by more than 20%.





  
Contributions. This work pioneers the investigation into the sparsity structure of the channels in the KV cache. Specifically, we discovered that the activated key cache is sparse given a specific query. This insight allows us to prune the key cache channels using a query-induced norm. Inspired by this phenomenon, we introduce a plug-and-play method ThinK, the first channel pruning method specifically designed for KV cache. ThinK reduces the dimensionality of the cache channel, leading to linear savings in both memory and computational requirements. As plug-and-play techniques, our proposed ThinK is orthogonal to other KV cache compression schemes (e.g. KV cache eviction, quantization). Our extensive experiments demonstrate the remarkable efficiency of ThinK on LLaMA-3 and Mistral models. Furthermore, we explore the promising potential of extending ThinK to value cache pruning (ThinKV) in specific scenarios, highlighting the broad applicability of our method. 











§ OBSERVATIONS

We identify several key observations that motivate our approach to prune the channel of KV-Cache. 
Specifically, we visualize the magnitude of KV-cache and perform singular value decomposition of the attention in LLaMA3-8B model. 
We show the visualization for certain heads and layers for illustration purposes. 



  
Magnitude of KV-Cache Channel
Figure <ref> visualizes the absolute values of key and value cache across the tokens in each channel[We use the visualization code from <https://github.com/jy-yuan/KIVI/tree/main/vis>.]. 
Consistent with previous findings <cit.>, we observe that only certain channels have significant magnitudes in the key cache, whereas the value cache lacks obvious patterns.
For instance, the absolute value around 50^th channel of key cache has much larger magnitudes than other channels for all tokens in layer 14 (Figure <ref> (a)).
The same observation applies to 50^th and 150^th channels of the first head in layer 20 (Figure <ref> (c)). 
Given such an observation, <cit.> proposed to perform quantization over the channels of the key cache based on such observations.
In addition to channel quantization, the observation also suggests that it is feasible to prune away certain key cache channels that contribute less to the attention mechanism.
Moreover, channel quantization and pruning techniques are orthogonal such that they can be utilized concurrently to enhance model efficiency.










  
Singular Value Analysis
We conducted singular value decomposition (SVD) <cit.> on the attention of the specified layers to investigate their principal components.
The singular values derived from SVD capture the effective rank of the attention matrix, indicating how the information is distributed across different components. 


    𝐔, Σ, 𝐕    = SVD (
           softmax (QK^T/√(D) ) 
         ),  Energy_i  = σ_i^2/∑_i σ_i^2




Figure <ref> (a) illustrates the energy distribution of the singular values, plotted in log scale to enhance the visibility of differences among them. 
Notably, only a few singular values exhibit high energy levels exceeding 0.01 across all heads and layers, highlighting their relative significance.
This observation aligns with previous findings <cit.>, where a small subset of singular values often captures most of the information in attention mechanisms.
In addition, the rapid decay of the energy suggests that a low-rank approximation can effectively capture the essential information in the key cache. 

Figure <ref> (b), the normalized cumulative energy sum reveals that the top 50 singular values account for over 90% of the total energy.
These findings suggest that the attention matrix is inherently low-rank <cit.>, indicating that the key cache can be approximated using low-dimensional vectors <cit.>.





























§ THINK




  
Notations. We use uppercase letters (e.g., X, Y) to denote scalar values and boldface uppercase letters (e.g., 𝐐, 𝐊) to denote matrices and tensors. The notation ‖·‖_p denotes the l_p-norm for vectors. Unless otherwise specified, ‖·‖ denotes the l_2-norm. The Frobenius norm is denoted by ‖·‖_F. The floor function is denoted by ⌊·⌋, and the ceiling function is denoted by ⌈·⌉.




 §.§ Preliminary Study of KV Cache Optimization

In scenarios with extended contexts or batch processing, the main limitations in terms of memory and speed are due to the handling of the KV cache size. Considering a batch of requests to a Large Language Model (LLM) service that provides a long input prompt consisting of tokens [x_B1,...,x_BS], the total KV cache size can be computed as follows:

    2 × B × S × L × N × D,

where L is the number of layers, N is the number of heads, D is the head dimension. The KV cach size grows linearly as the batch size B and sequence length S.  For
a model with multihead attention (MHA) <cit.>, such as LLaMA2-7B <cit.>, a context length of 2048 and a batch size of 13 require storing a 13 GB KV cache, which is equivalent to the size of the model parameters. 
The KV cache must be transferred from off-chip memory (HBM) <cit.> to on-chip memory (cache) for each token generated, leading to a memory bottleneck. 
This substantial memory demand highlights the challenges in managing large-scale models and the need for efficient memory utilization strategies. Current methods optimize the KV cache based on the sequence length S <cit.> and precision <cit.>. 
We will introduce a new method, , to optimize it from the perspective of the number of head dimensions D.





Magnitude base Pruning: Based on the observations in Figure <ref> which depicts the significant variation in the magnitudes across different channels, one straightforward criterion is to use the norm of the magnitude to measure the importance of different channels in key cache.

    M_n,d = ‖ K[n,:,d]‖ _p.

Given pruning ratio λ, We only keep T=⌊(1-λ)D⌋ most important channels among the D channels of each head:

    I = Top_T(M,T)

where ‖·‖ _p is the l_p norm of each channel. n∈[1,N] and d∈[1,D] are indicators of heads and channels in key cache. I∈(ℤ^+)^N× T stores the indicators of the top T values in tensor M per head.

In Table <ref>, we present the results of key cache pruning with various pruning ratios applied to the LLaMA3-8B model. We utilize the l_1 and l_2 norms as criteria for evaluation, and validate performance using the LongBench benchmark <cit.>. Compared to the baseline methods, H2O <cit.> and SnapKV <cit.>, both with a KV length of 512, we further prune the channels of the key cache. A 30% pruning ratio can maintain accuracy; however, increasing it to 40% results in significant performance degradation, especially for l_1 norm based pruning. The results of magnitude-based pruning support our assumption that the key cache is redundant in the channel dimension. These results also indicate the need for a better pruning matrix to achieve higher pruning ratios effectively.




 §.§ Query-Driven Pruning

For each head, the attention scores are computed using the queries and keys, and then applied to the values. The formula for the attention for head i is:

    Attention(Q_i,K_i,V_i) = softmax(Q_iK_i^T/√(D))V_i

where Q_i, K_i,V_i∈ℝ^S× D. When pruning one channels of K_i, the corresponding channels in Q_i will also be removed. We aim to find the optimal subset of channels to prune, denoted by the selection matrix S∈{0,1}^D× D, where 
S is a diagonal matrix with binary entries (1 for keeping a channel, 0 for pruning it). To better maintain the performance after pruning the channels, we minimize the Frobenius norm of the difference between the original and pruned attention weights:

    min_S‖Q_iK^T_i -Q_iS(K_iS)^T ‖_F


Given a pruning ratio λ, it can further expanded as:

    min_𝐒     𝐐_i 𝐊_i^T - 𝐐_i 𝐒𝐊_i^T _F 
    subject to     trace(𝐒) = ⌊(1-λ)D⌋
         𝐒 = diag(s_1, s_2, …, s_D),  where  s_j ∈{0, 1}


For simplicity, we use greedy algorithm to optimize 𝐒. To achieve the pruning goal, we define a criterion for evaluating the importance of each channel and greedily select the channels with largest scores:

    Score_i[j] = ‖Q_i[:,j]K_i[:,j]^T ‖_F


    I_i = Top_T(Score_i,T)

Here's a detailed explanation of why it optimizes the selection matrix. The score_i[j] measures the magnitude of the interaction between the query and key vectors for channel j in each head i. By selecting channels with the highest interaction magnitudes, we aim to retain the most significant contributions to the attention mechanism. This criterion ensures that the selected channels preserve the primary information flow in the attention computation, thereby minimizing the loss of important information.

Observation Window. Based on the observations in SnapKV <cit.> that the last window of input sequence recognizes highly similar attention allocation
pattern with the actual generation. To reduce the computation cost, we only use the last S_obs window to calculate the score: ‖Q_i[-S_obs:,j]K_i[:,j]^T‖_F.


[17]r0.4

    < g r a p h i c s >



Implementation during decoding.







 §.§ Implementations

Following SnapKV, we focus on the long context input scenario. We opt not to prune the most recent tokens and newly generated keys. Consequently, our key-value (KV) cache will store two distinct categories of keys: one subset consists of pruned keys with a reduced channel size, while the other retains keys at their original size. Additionally, we maintain a binary mask to indicate which channels have been pruned. Note that the memory overhead associated with this mask is negligible. Figure <ref> illustrates one implementation of our method during the decoding stage. The pruned keys are first zero-filled using the mask to restore them to their original size, and then concatenated with the unpruned keys. This implementation can be effectively integrated with optimization techniques such as FlashAttention <cit.>. The second implementation involves initially pruning the Query using the mask. The pruned Query is then multiplied by the pruned Key, while the unpruned Query is applied to the unpruned Key. Subsequently, the two outputs are concatenated. In theory, this implementation has lower computational costs.






§ EXPERIMENT RESULTS
 


In this section, we conduct extensive experiments to evaluate the effectiveness of  on performance and memory reduction. First, we introduce benchmark datasets, baseline methods, and implementation details in Section <ref>. Next, we report the performance of our KV pruning method with two different KV compression methods in long-context scenarios in Section <ref>. Finally, we present the results of ablation experiments in Section <ref>.



 §.§ Settings





  
Benchmark Datasets.
We perform evaluations of our method with state-of-the-art KV cache compression methods on two widely used benchmarks: LongBench and Needle-in-a-Haystack. 
LongBench <cit.> is designed to comprehensively evaluate LLM’s long context understanding capabilities. It includes 17 datasets covering six different tasks: single-document QA, multi-document QA, summarization, few-shot learning, synthetic tasks, and code completion. The average input length of LongBench is 6,711 words, which necessitates reducing the KV cache to lower memory usage for inference.
Needle-in-a-Haystack <cit.> is a recent popular test challenge that requires models to accurately identify a small piece of information (“needle”) in a long document (“haystack”), where the needle is placed at a random position. 
This challenge can test if KV cache compression methods still retain the small piece of critical information.










  
Baseline Approaches.
The baseline methods include Heavy Hitter Oracle (H2O) and SnapKV, both of which are the state-of-the-art KV cache compression methods but use different strategies for selecting KV cache. 
H2O <cit.> is designed to reduce memory usage by dynamically balancing recent tokens and Heavy Hitter (H2) tokens, where H2 tokens are a small set of tokens that contribute most of the value when computing attention scores.
SnapKV <cit.> automatically compresses KV caches by selecting clustered important KV positions for each attention head. This is inspired by the fact that each attention head has a specific pattern, focusing on certain attention features during generation.







  
Implementation Details.

In this paper, we use LLaMA3-8B-Instruct <cit.> and Mistral-7B-Instruct-v0.2 <cit.> as the backbone LLMs, both accessible via HuggingFace <cit.>.
Our  aims to prune channels of the KV, which is agnostic to KV cache compression methods. 

If there is no other statement, we prune the key cache by default. All the experiments are conducted using one Nvidia A100.
To fairly compare KV cache compression methods and KV cache compression integrated with , we use the same hyperparameters for both.
For example, when comparing SnapKV and SnapKV integrated with , we set the maximum pooling kernel size to 7 and the observation window size to 32, using the same KV-size for both.








 §.§ Results on LongBench


Tables <ref> and  <ref> present the results of KV compression methods and KV compression methods integrated with our channel pruning method of Key cache () over two different base LLMs across various KV-sizes on LongBench.
The following observations can be drawn:
(1) Our method can further prune the channels of the key cache after compressing the KV cache with H2O and SnapKV. For the base model LLaMA3-8B, our method reduces memory usage and slightly improves performance for both H2O and SnapKV. For the base model Mistral-7B, our method reduces memory with only a slight drop in performance in some cases.
(2) Comparing the results of SnapKV or H2O integrated with  in Table <ref> to SnapKV or H2O integrated with l_1 or l_2 norm shown in Table <ref>, our proposed query-based channel pruning shows better performance when the pruning ratio is 40% compared to no pruning. However, l_1 and l_2 norm methods cannot outperform the KV compression methods without pruning.
(3) When the pruning ratio is small, performance tends to be better than when the pruning ratio is large.
(4) When the KV-size is increased from 128 to 2048, the performance of our channel pruning method improves. Notably, with a KV cache size of 2048 and a pruning ratio of 40%, our method can even outperform the LLaMA3-8B with a full KV cache.
The above observations indicate that our method is agnostic to existing KV cache compression methods and can further improve their performance and memory reduction. Additionally, query-based pruning is more effective than l_1 and l_2 norm for channel pruning in LLMs.



 §.§ Results on Needle-in-a-Haystack







Table <ref> presents the results of the Needle-in-a-Haystack test based on the SnapKV <cit.> approach with KV cache size of input prompt from 128 to 2048. 
With a modest pruning ratio λ=40%, the accuracy of  is consistently better or comparable than the original SnapKV across both LLaMA3 and Mistral with different KV cache sizes. 
Such comparisons demonstrate that the proposed query-driven channel pruning can effectively retain the informative channels and discard the noisy ones. 
With a pruning ratio of λ≥ 50%, we observe an accuracy drop with a small KV-cache size, particularly for 128 and 512 across both LLaMA3 and Mistral. 
However,  is still able to obtain comparable performance with SnapKV when we have a larger KV cache size (i.e., 1024 and 2048).
Intuitively, a larger pruning ratio with a smaller KV cache size could tend to lose more important information compared to a larger KV cache size. 
In addition, the performance on larger KV cache size also suggests  is robust for long context retrieval tasks. 


Figure <ref> (a)-(d) visualize the Needle-in-a-Haystack test accuracy against different token lengths and depth percent.
The KV cache size is configured to 128 and 1024, with pruning ratio λ equal to 40% and 50%, respectively.
First,  can maintain the original retrieval capability of SnapKV though the overall numerical accuracy is different (i.e., 77.8 vs 78.6 and 90.4 vs 90.6). 
Because achieves the same accuracy at most of the entries with different token limits and depths as Snap KV.
In addition,  also successfully retrieves some “needles” where SnapKV fails, leading to higher overall accuracy. 
The visualization results further demonstrate the robustness of  in a fine-grained perspective and  can enhance the original quantization approach.














 §.§ Ablation Studies












  
Impact of Different Recent Sizes.
Preserve the most recent KV embeddings <cit.> is important for maintaining the performance of LLMs after KV cache compression. Note that a tradeoff exists: increasing the recent-size allows more infomation to be propagated, while increasing the cache size to be stored. To study its impacts, we evaluate the performance produced by three recent-size, namely 0, 32 and 128, on LongBench, using Mistral-7B-Instruct-v0.2 as the basleline model. The results are summarized in Table <ref>. One can observe that a recent-size of 32 yields superior performance than 0 in terms of averaged score on LongBench which demonstrates the importance of keeping the most recent KVs. On the other hand, the performance of 32 and 128 is negligible, suggesting that maintaining the most recent 32 KVs suffices to preserve optimal performance.

[17]r0.45

    < g r a p h i c s >



Performance comparisons under the same KV cache memory consumption.





  
Comparison Under the Same Memory Consumption.
To ensure a fair comparison, we adjust the KV cache size of H2O or SnapKV to match the memory usage of H2O with ThinK or SnapKV with ThinK. 
For example, the KV cache size of H2O with ThinK is 128. Due to channel pruning of the key cache, the memory consumption of H2O with ThinK at a 128 KV-size is lower than H2O at a 128 KV-size.
Therefore, the KV-size of H2O is adjusted from 128 to 109. 
Table <ref> and Figure <ref> present the results of these comparisons on LongBench.
As the results show, H2O or SnapKV with ThinK deliver superior performance compared to H2O or SnapKV without ThinK while maintaining the same memory usage. 
This demonstrates that KV cache compression methods, combined with our query-based channel pruning, can compress the KV cache more effectively and utilize memory more efficiently.



  
Pruning Channels of Both Key and Value Cache.
In our previous study, we evaluate methods with pruning channels of the key cache. In this part, we explore the impact of pruning channels of the value cache (Appendix <ref>). Specifically, for KV cache compression methods, we prune channels of the key cache at one pruning ratio and channels of the value cache at another pruning ratio. 
Table <ref> presents the results with LLaMA3-8B and Mistral-7B on LongBench.
When testing on the base model LLaMA3-8B, H2O or SnapKV with key and value channel pruning can perform on par with H2O or SnapKV without channel pruning. In some cases, H2O or SnapKV with key and value channel pruning may even perform better. Pruning the channels of the value cache over the base model Mistral-7B experiences a slight performance drop. Despite this, note that KV cache with additional value channel pruning can further reduce memory usage.

















§ RELATED WORK


In scenarios involving long contexts, the most significant computational burden from the attention mechanism is the key-value (KV) cache. Reducing the KV cache is a high priority for optimizing deployment efficiency. System-level optimizations, such as FlashAttention <cit.> and PagedAttention <cit.>, have been developed to address this issue. Additionally, algorithm-level optimizations are being explored to further enhance efficiency.

KV Cache Eviction. StreamingLLM <cit.> maintains a few initial tokens and some recent tokens based on the observation of attention sink, which may result in the loss of important information carried by the dropped tokens. H2O <cit.> retains only a small portion of the tokens by greedily dropping tokens based on their contributions to the cumulative attention. SnapKV <cit.> selects clustered important KV positions for each attention head from an ‘observation’ window located at the end of the prompts. FastGen <cit.> adaptively evicts tokens from attention heads that emphasize local contexts. This approach focuses on discarding non-special tokens centered on special tokens, while the standard KV cache is used only for attention heads that broadly attend to all tokens. PyramidKV <cit.> and PyramidInfer <cit.> considers adjusting the KV cache size across different layers by allocating more cache in the lower layers and less in the higher ones.




KV Cache Quantization. SmoothQuant <cit.> can quantize the KV cache to 8-bit with minimal performance degradation. Q-Hitter <cit.> uses accumulated attention scores and ‘Quantization Friendliness’ metrics to identify tokens that are essential for maintaining the generalization capabilities of LLMs and are suitable for KV cache quantization. Some studies have found that the key cache and value cache should be quantized differently <cit.>: the key cache should be quantized per-channel, while the value cache should be quantized per-token.

Structured Pruning of LLMs. Structured pruning  <cit.> of LLMs traditionally focuses on removing unimportant layers, heads, and hidden dimensions, which often results in significant performance degradation. In contrast, our methodology preserves the original architecture of the LLM and specifically targets the channel dimension within each head’s key cache. By dynamically identifying unimportant channels based on data dependant criterion, our approach greatly reduce the key cache-size with negligible performance loss.




§ END NOTE AND FUTURE DIRECTION

Motivated by the observation that certain channels have significantly larger magnitudes compared to others, and the singular value analysis indicates that the key cache is inherently low-rank <cit.>, we propose  to perform pruning over the key cache channel. 
The proposed pruning strategy is query-dependant and optimized based on the attention scores, ensuring that essential information is retained for each input query.

In addition,  can be seamlessly integrated with other popular token-level KV cache quantization techniques <cit.>, further enhancing inference efficiency. 
Extensive experiments on LongBench <cit.> and Needle-in-a-Haystack tests with two foundation models demonstrate the effectiveness and robustness of our query-dependent channel pruning method.
Our approach achieves comparable or superior performance to baseline methods while reducing the key cache size by 40%.
Our analysis indicates that  can maintain superior performance over baselines with a smaller KV cache size under equivalent memory consumption conditions.

Future work will focus on strategies to increase the pruning ratio without compromising model performance, and exploring pruning techniques for value caches. 
We also plan to evaluate channel importance from a more sophisticated and compositional perspective, incorporating both token-level and channel-level information to calculate importance.



apalike







§ ADDITIONAL EXPERIMENTAL DETAILS



 §.§ Value Cache Pruning

Similar to the approach used for the Key cache, the pruning of channels in the Value cache can be guided by two primary criteria: magnitude-based pruning and query-driven pruning. We find that query-driven pruning is still better than magnitude based pruning.

    Score_v,i(Q_i,K_i,V_i)[j] = ‖softmax(Q_i[-S_obs:]K_i^T/√(D))V_i[:,j]‖_F


    I_i = Top_T(Score_v,i,T)

where Q_i, K_i,V_i∈ℝ^S× D. We define a criterion Score_v,i to indicate the importance of each channel in the head i of value cache. Then, only top T channels are retained.






 Paper Title: Bailing-TTS: Chinese Dialectal Speech Synthesis Towards Human-like Spontaneous Representation.




RDP: Ranked Differential Privacy for Facial Feature Protection in Multiscale Sparsified Subspace

    Lu Ou
Hunan University

Changsha, China 

oulu9676@gmail.com
Shaolin Liao
IIT

Chicago, USA 

sliao5@iit.edu
Shihui Gao
Hunan University

Changsha, China 

gaoqaq@hnu.edu.cn
Guandong Huang
State Grid Hunan Materials Company

Changsha, China 

huanggd@hnu.edu.cn
Zheng Qin
Hunan University

Changsha, Country 

zqin@hnu.edu.cn







    
================================================================================================================================================================================================================================================================================================================================================




§ ABSTRACT 
Large-scale text-to-speech (TTS) models have made significant progress recently. However, they still fall short in the generation of Chinese dialectal speech. To address this, we propose Bailing-TTS, a family of large-scale TTS models capable of generating high-quality Chinese dialectal speech. Bailing-TTS serves as a foundation model for Chinese dialectal speech generation. First, continual semi-supervised learning is proposed to facilitate the alignment of text tokens and speech tokens. Second, the Chinese dialectal representation learning is developed using a specific transformer architecture and multi-stage training processes. With the proposed design of novel network architecture and corresponding strategy, Bailing-TTS is able to generate Chinese dialectal speech from text effectively and efficiently. Experiments demonstrate that Bailing-TTS generates Chinese dialectal speech towards human-like spontaneous representation. Readers are encouraged to listen to demos at <https://giantailab.github.io/bailingtts_tech_report/index.html>.




§ INTRODUCTION

The goal of text-to-speech (TTS) systems <cit.> is to generate human-like speech from text. With the development of neural network and deep learning, large-scale training corpus of a human-like level <cit.> is established and corresponding TTS models is developed. However, such system <cit.> is only able to generate voice for non-dialectal speech. And the quality of the generated speech is not satisfactory <cit.>. Therefore, in order to generate human-like quality of speech from dialectal text, we propose Bailing-TTS, a family of speech generation models for synthesizing Chinese dialectal speech from text with human-level naturalness. 

Besides, in zero-shot speech synthesis, both the naturalness and the robustness of speaker modeling are challenging <cit.>. In previous studies, large-scale models are developed for speech synthesis<cit.>. Despite the advances from these methods, the generalization capability of these methods is still insufficient for the generation of high-quality dialectal speech. Therefore, we primarily propose Bailing-TTS to generate high quality Chinese dialectal speech from text.

We propose several specific extension techniques that significantly enhance the quality of Chinese dialectal speech generation. First, continual semi-supervised learning framework towards text and speech tokens is developed to facilitate the alignment of these multiple modalities. Second, a multiple-stage Chinese dialectal representation learning framework introduces a specific network architecture and the corresponding learning strategy to improve the quality of the generated Chinese dialectal speech. 

In the evaluation of the proposed Bailing-TTS, both the advantages and disadvantages of the proposed model for the dialectal speech generation is compared with human speakers. The evaluation results represent that Bailing-TTS generates good quality of Chinese dialectal speech.

Finally, both the potential applications and the limitation of Bailing-TTS are discussed. Then, the future work of Bailing-family foundation model is represented including taking multiple modalities as input and producing audio output. For example, it's supposed to produce the generation of music <cit.> from the text and video. Besides the generation of audio such as music or speech, in our next step, the video and audio are generated simultaneously. 

Our key contributions are represented as the following:



  * We introduce Bailing-TTS, a family of foundation models for synthesizing Chinese dialectal speech from text with human-level naturalness, such as spontaneous speech.

  * Both a continual semi-supervised representation strategy and Chinese dialectal representation with a specific mixture-of-expert network architecture are proposed to improve the quality of the generated Chinese dialectal speech. 

  * A variety of hierarchical reinforcement post-learning extension techniques are represented to significantly enhance the quality of Chinese dialectal speech generation.




§ BAILING-TTS

We propose Bailing-TTS based on an auto-regressive transformer model of multiple layers. The proposed Bailing-TTS is trained on a large-scale of dataset including a large part of dialectal data. Both non-precise annotation data and high-quality annotation data are in the dataset. Besides, we propose a multi-stage training strategy, based on a specific transformer architecture (Figure <ref>), to facilitate the generation of spontaneous and expressive speech from text.





 §.§ Overall Architecture

First, at the stage of token representation, a continual semi-supervised learning strategy of spontaneous, expressive text and speech token pairs is proposed to facilitate weak alignment between the two modalities. Second, a specific transformer-based architecture is proposed corresponding with a multi-stage training strategy. Third, during inference, it generates good quality of spontaneous Chinese dialectal speech from text. 



 §.§ Continual Semi-supervised Learning Towards Text and Speech Tokens

Spontaneous speaking style exhibits notable differences from other speaking styles due to various spontaneous phenomena (e.g., filled pauses, prolongation) and substantial prosody variation (e.g., diverse pitch and duration variation), posing challenges to modeling and prediction of spontaneous style. 

In order to generate spontaneous, expressive speech from text, we present a multi-stage and multi-modal (text and speech) pre-training learning framework for text-speech token alignment (Figure <ref>). In the first stage, an unsupervised sampling strategy is proposed on a large-scale non-precise annotation datasets, forming a multi-stage pre-training process. In the second stage, a fine-tune sampling strategy is proposed on a high-quality dataset containing spontaneous, expressive text and speech pairs. Therefore, the token-wise text-speech association is represented to facilitate alignment between the two modalities. 





 §.§ Multi-Chinese Dialectal Representation Learning

Dialect-specific TTS models are not satisfactory as dialect-specific data is scarce. Therefore, a single unified TTS model that generalizes well for many dialects is in demand. 

In order to train a unified TTS model for a variety of Chinese dialects, we present a specific token representation together with a multi-stage learning strategy. In the aspect of the token representation, a design of mixture-of-expert architecture (Figure <ref>) is proposed to learn the unified representation of multiple Chinese dialects and specific representation of each dialect. Similarly, we propose a cross-attention based mechanism to inject dialect tokens throughout the TTS model layers. 



 §.§ Hierarchical RL-based Posting-training Extension

Similar to text-based language models, four training stages are built for the representation learning of Bailing-TTS. The first stage is the pre-training stage aiming to maximize scenario and speaker coverage with a robust backbone for general speech representation of the Chinese dialect. This stage is to learn the efficient representation of a large-scale Chinese dialect dataset. The second stage is the fine-tuning stage in order to enhance the basic performance of the dialectal speech. 

Besides, we propose a hierarchical RL learning strategy as the third and the fourth step in order to generate the high-quality speech of multiple Chinese dialects. In the RL hierarchical training strategy, in order to let the high-level policy support exploration for good quality of spontaneous, expressive speech and the primary-level policy support good quality generation of dialectal speech simultaneously. We propose dynamic sub-optimization for the learning of the high-level policy at the base of the primary-level policy in the hierarchical learning strategy.



§ EXPERIMENTS AND RESULTS




 §.§ Experimental Settings

In this section, the details of the training, inference and evaluation for the proposed Bailing-TTS on Chinese dialectal speech synthesis are introduced. 



  §.§.§ Implementation Details

A custom dataset containing high-quality Mandarin and multiple Chinese dialects data is used for the training, inference and evaluation. This dataset contains 200k hours of labeled speech data. The annotation contains fine-grained labels between text and speech modalities. In addition, this large-scale dataset contains high-quality speech-text pairs. Furthermore, this custom Chinese dialect dataset is used for the pre-training stage of the continual semi-supervised learning. Clusters of GPUs (A100) are applied for the training. 



  §.§.§ Evaluation Dataset

A custom test set is used for the Chinese dialect TTS task. There are 50 distinct Chinese speakers for each Chinese dialect with 2000 clips. Following <cit.>, sentences are randomly chosen from each speaker, and 5-second to 15-second clips are randomly chosen as prompts from the same speaker's speech <cit.>. Besides, there are 25 male and 25 female Chinese speakers for each Chinese dialect. In the dataset, each Chinese dialect contains 50 common spontaneous words <cit.>. Besides, 2000 clips from DiDiSpeech dataset <cit.> are used for the evaluation of Mandarin speech. 




  §.§.§ Evaluation Metrics

Both objective metrics and subjective metrics are used for the evaluation. In the aspect of objective metrics: word error rate (WER) and speaker similarity (SIM) is evaluated. In the aspect of subjective metrics, comparative mean option scores (CMOS) and mean option score (MOS) are calculated to evaluate the naturalness and quality, respectively.



 §.§ Experimental Results on TTS of Multiple Chinese Dialects

In this section, the proposed Bailing-TTS is evaluated in terms of 1)generation quality 2)generation similarity 3)human-level naturalness. 



  §.§.§ Generation Quality

As shown in Table <ref>, we find that the proposed Bailing-TTS is close to the ground-truth recording. It demonstrates Bailing-TTS can generate high-quality and natural speech for the testing Chinese dialect data. Besides, the performance verifies the effectiveness of the proposed Bailing-TTS with specific network architecture and the corresponding training strategy. In details, for the evaluation of Mandarin speech, Bailing-TTS obtains 1.86 for word error rate (WER), 4.32 for mean option score (MOS). In addition, human speakers obtain 1.35 for word error rate (WER), 4.32 for mean option score (MOS). For the evaluation of dialectal speech, Bailing-TTS obtains 6.37 for word error rate (WER), 4.11 for mean option score (MOS). Besides, human speakers obtain 4.60 for word error rate (WER), 4.24 for mean option score (MOS).














  §.§.§ Human-Level Naturalness

The speech synthesized through Bailing-TTS is also in the comparison with the Ground Truth in Table <ref> for the evaluation of both Mandarin speech and Chinese dialectal speech. Bailing-TTS achieves -0.06 CMOS and -0.08 CMOS compared to the Ground Truth, which demonstrates that our method is comparable on human voice quality. 



  §.§.§ Zero-shot learning

Both objective and subjective evaluation for the zero-shot learning experiments are conducted. In one aspect, clips from custom corpora are used to measure the proposed model's performance on a variety of objective metrics. 2000 clips from custom dataset are used for the evaluation of Mandarin speech. To be noted, the custom clips contains highly expressive Mandarin speech with spontaneous word.

Besides, both reference utterance and corresponding target utterance are collected from the same speaker for each clip. The proposed Bailing-TTS family is designed to generate target text based on a prompt (the reference speech). Therefore, comparison between synthesized speech and ground truth speech from real human is conducted for both the objective and subjective evaluation.

For the comparison with real human speech, the word error rate (WER) and speaker similarity metrics (SIM) are used for objective evaluation. Particularly, Paraformer-zh <cit.> is used as the automatic speech recognition (ASR) for Mandarin, and we train a custom automatic speech recognition (ASR) for multiple Chinese dialects. Besides, comparative mean option scores (CMOS) are used for subjective evaluation. Particularly, first, native Chinese dialectal speakers are represented with a reference speech clip of the target speaker. Second, both the synthesized output of the proposed model and the corresponding targeting human speech are used. Third, evaluation is conducted to rate each given clip with higher speaker similarity and expressiveness to the reference clip on a scale between -3 to +3, where -3 and 3 indicate the least and strongest preference. The results are then collected and averaged over all human evaluators and test sentences. 

The results for both objective and subjective evaluation are reported in Table <ref>. The results demonstrate that the proposed Bailing-TTS model represents an expected performance in zero-shot learning. In details, for the evaluation of Mandarin speech (Zero-shot), Bailing-TTS obtains 3.48 for word error rate (WER), 0.72 for speaker similarity metrics (SIM). 




  §.§.§ Fine-tuning learning

Speaker fine-tuning (SFT) is conducted on the base of pre-trained Bailing-TTS model. For the evaluation of the fine-tuning learning, speech data from 10 speakers are combined for Mandarin fine-tuning learning dataset. Then, the comparison is made between the fine-tuned model (Bailing-TTS fine-tuned) and the base pre-trained model (Bailing-TTS Zero-shot). For the objective evaluation, the word error rate (WER) and speaker similarity metrics (SIM) are used for the objective evaluation. For the subjective evaluation, comparative mean option scores (CMOS) are used. To be noted, 15 seconds of speech clip is randomly sampled as the prompt for each speaker. Similarly, speech data from 10 speakers are combined for Chinese dialectal fine-tuning learning dataset. 

For the evaluation of Mandarin speech, the results of the speaker fine-tuning experiment and the base model are demonstrated in the Table <ref>. The results demonstrate that the fine-tuned models shows better performance in both objective and subjective metrics. In details, for the evaluation of Mandarin speech (Speaker fine-tuned),  Bailing-TTS obtains 2.98 for word error rate (WER), 0.77 for speaker similarity  metrics (SIM) and +0.18 for comparative mean option scores (CMOS). For the evaluation of Chinese dialectal speech (Speaker fine-tuned),  Bailing-TTS obtains 7.43 for word error rate (WER), 0.76 for speaker similarity metrics (SIM) and +0.11 for comparative mean option scores (CMOS). 





  §.§.§ Streaming processing

There are a range of challenges in the application of TTS systems in the real world. User experiences are not satisfactory due to the latency and the first packet delay. The computation cost of both the time and the memory is still large for the deployment on mobile hardware systems. Therefore, initial work on the deployment is conducted. A variety of techniques <cit.> are employed to reduce the inference cost and latency. 

In order to reduce the inference cost and the memory cost of attention layers, first, we apply efficient memory cost methods including grouped-query attention <cit.>, paged attention <cit.>, and flash attention <cit.>. Second, model quantization method is also applied to further reduce the computation cost <cit.>. Similarly, the consistency distillation <cit.> and a modified flow matching algorithm <cit.> are used to reduce the computation cost of the diffusion model. Third, to further reduce the computation cost of diffusion architecture, in one aspect, the consistency distillation <cit.> is applied to reduce the computation cost of the diffusion model, in the other aspect, the flow matching algorithm <cit.> is used to reduce the memory cost.

The initial results demonstrate that comparable performance is achieved after a variety of methods are applied on the deployed model. In details, as shown in Table <ref>, the latency is reduced to 0.13 ×, the RTF is reduced to 0.46 × and WER and SIM remains the same in the comparison with the offline Bailing-TTS model. 





§ APPLICATIONS AND DISCUSSION

We propose a family of Bailing-TTS towards Chinese dialectal speech synthesis with spontaneous words. It has the potential for real-world application. First, the dialectal speech is promising to provide rich experience of chat service in the real-world particularly companion chat service. Second, it's very likely to be beneficial to the facilitation of dialectal culture and cultural applications. 

Initial work on the family of Bailing-TTS is proposed while services such as speech synthesis with emotions, speech with support of other modalities are not well explored. Therefore, the exploration of expressive and emotional generation of Chinese dialectal speech is under-going. We are planing to develop the next version of Bailing-TTS family for the generation of high-quality audio (speech/music) from the input of video and text. Then, the generation of both the high-quality audio together with video will be explored.         
















plain



 Paper Title: Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environments.


















  The University of Queensland, CSIRO Data61
  Brisbane
  Australia
haodong.hong@uq.edu.au



  The University of Queensland
  Brisbane
  Australia
sen.wang@uq.edu.au



  The University of Queensland
  Brisbane
  Australia
helen.huang@uq.edu.au



  The University of Adelaide
  Adelaide
  Australia
qi.wu01@adelaide.edu.au


Corresponding author

  CSIRO Data61, The University of Queensland
  Brisbane
  Australia
jiajun.liu@csiro.au











§ ABSTRACT 
Real-world navigation often involves dealing with unexpected obstructions such as closed doors, moved objects, and unpredictable entities. 
However, mainstream Vision-and-Language Navigation (VLN) tasks typically assume instructions perfectly align with the fixed and predefined navigation graphs without any obstructions.
This assumption overlooks potential discrepancies in actual navigation graphs and given instructions, which can cause major failures for both indoor and outdoor agents.
To address this issue, we integrate diverse obstructions into the R2R dataset by modifying both the navigation graphs and visual observations, introducing an innovative dataset and task, R2R with UNexpected Obstructions (R2R-UNO).
R2R-UNO contains various types and numbers of path obstructions to generate instruction-reality mismatches for VLN research.
Experiments on R2R-UNO reveal that state-of-the-art VLN methods inevitably encounter significant challenges when facing such mismatches, indicating that they rigidly follow instructions rather than navigate adaptively. 
Therefore, we propose a novel method called ObVLN (Obstructed VLN), which includes a curriculum training strategy and virtual graph construction to help agents effectively adapt to obstructed environments. 
Empirical results show that ObVLN not only maintains robust performance in unobstructed scenarios but also achieves a substantial performance advantage with unexpected obstructions.   
The source code is available at <https://github.com/honghd16/ObstructedVLN>.







<ccs2012>
<concept>
<concept_id>10010147.10010178.10010224.10010225</concept_id>
<concept_desc>Computing methodologies Computer vision tasks</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10002951.10003227.10003251</concept_id>
<concept_desc>Information systems Multimedia information systems</concept_desc>
<concept_significance>500</concept_significance>
</concept>
</ccs2012>


[500]Computing methodologies Computer vision tasks
[500]Information systems Multimedia information systems
























Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environments
    Jiajun Liu
    August 5, 2024
=========================================================================================





§ INTRODUCTION






Vision-and-Language Navigation (VLN) <cit.> requires agents to follow natural language instructions to reach a specified destination.
Recently, there has been a growing interest in this area due to its great potential for real-world applications such as household robots, especially with the booming of Large Language Models (LLMs) <cit.>.
However, current VLN tasks are limited by several unrealistic assumptions, making their application remain in simulators with few real-world robotic deployments <cit.>.


One significant constraint is what we call the “perfect instruction assumption”, which implies that instructions are always perfectly aligned with the environment, neglecting real-time dynamics like unexpected obstructions.
Agents trained under this assumption excel in following instructions but lack adaptability for actual navigation where discrepancies between instructions and reality are commonly seen.
For example, as shown in <ref>, the human instructs the agent to “Walk straight down the hall” based on prior knowledge of the house. 
However, real-world environments have changed, with unforeseen obstacles like a suitcase blocking the hallway.
While humans can quickly adapt and find detours, current VLN agents struggle with these instruction-reality mismatches, often leading to navigation failures.


To address this issue, it is essential to introduce these discrepancies into VLN.
While various factors can lead to such discrepancies, this work focuses on one of the most representative and prevalent causes: obstructions. 
We propose to integrate obstructions into existing discrete VLN environments to block the path described by the instruction, resulting in an instruction-reality mismatch.
Notably, although both approaches involve obstructions, our work differs from previous work on obstacle avoidance <cit.> in terms of problem setup, navigation focus, and potential solutions (See <ref>), leading us to choose the discrete setting.




Therefore, we make various modifications to the navigation graphs and visual observations of the R2R dataset <cit.>, proposing the R2R with UNexpected Obstructions (R2R-UNO) dataset as the first VLN task to emphasize the instruction-reality mismatches.
At the graph level, we selectively block edges whose removal does not impact the overall graph connectivity in existing paths, ensuring agents can still reach the destination.
To keep visual coherence with graphs, we design an object insertion module that leverages text-to-image inpainting techniques <cit.> to integrate diverse objects into different scenes seamlessly.
Due to the instability of inpainting results, a filtering module is employed to select the high-quality ones from multiple candidates to ensure obstruction generation.
Our study using R2R-UNO reveals that state-of-the-art VLN agents <cit.> perform poorly in obstructed environments, limiting their practical applications.

Although we expect agents to perform well with mismatches, it is crucial to maintain their performance in original environments simultaneously, which represent most situations. 
However, directly training with these two types of data poses challenges for agents to learn beyond following instructions and to differentiate between mismatched and aligned scenarios. 
Empirical results show that discrepancies between these two environments can lead to imbalanced optimization, favoring one type. 
To address this, we develop the Obstructed VLN (ObVLN) method, including a curriculum training strategy <cit.> to organize the training and a novel graph construction mechanism to introduce virtual nodes for blocked edges to facilitate efficient exploration.
The overall framework including data generation and agent training is presented in <ref>.


We conduct comprehensive experiments on R2R, REVERIE <cit.>, and R2R-UNO datasets to demonstrate the significance of incorporating instruction-reality mismatches in VLN. 
Established methods, such as DUET <cit.>, struggle in obstructed settings, with a significant 30% drop in Success Rate (SR). 
The proposed object insertion and filtering modules provide crucial visual feedback aligned with navigation graph changes, proved by the improved performance of agents using inpainting images.
By employing ObVLN, agents not only maintain robust performance in original scenarios but also effectively adapt to mismatches, achieving an impressive 67% SR, marking a significant advancement.
We summarize our contributions in this paper as follows: 


  * We address the underexplored issue of instruction-reality mismatches in VLN by proposing R2R-UNO, the first VLN dataset that includes such mismatches through graph changes and diverse obstruction generation, offering a unique challenge that reflects real-world navigation complexities.

  * We highlight the lack of adaptability in current VLN methods for obstructed environments and propose ObVLN as a solution, which employs curriculum learning and virtual graph construction to enhance agent adaptability.

  * Through extensive experiments, we prove the significance of introducing R2R-UNO in VLN research and show that ObVLN performs well in both original and obstructed environments, achieving a significant 23% SR increase in R2R-UNO.
 




§ RELATED WORK

Vision-and-Language Navigation
In VLN, agents need to navigate within simulated environments like Matterport3D <cit.> following natural language instructions. 
Numerous methods <cit.> and datasets <cit.> have been developed to address various challenges in this domain.
Some works address the data scarcity problem by introducing extra sources, including synthesized instructions <cit.>, additional environments <cit.>, and predicted scenes <cit.>.
Recently, ScaleVLN <cit.> synthesized enormous high-quality instruction-trajectory pairs for HM3D <cit.> and Gibson <cit.> environments to make agent performance approach human results.
Other works <cit.> apply diverse network structures such as LSTM <cit.>, Transformer <cit.>, and Graph Neural Networks <cit.> to enhance cross-model alignment and decision-making ability.
For example, HAMT <cit.> encodes the full history information through transformers and integrates it with instructions and observations for better action prediction.
DUET <cit.> further maintains a topological map to enable agents aware of global visual representations to make global decisions instead of adjacent viewpoints.
We adopt HAMT and DUET as the evaluation models due to their superior performance and difference in whether map-based.

 Environment Changes in VLN
Many works <cit.> have proposed to modify VLN environments for different purposes, which can generally be categorized based on the changing level. 
One category involves changing the visual observations to augment data for improving agent generalization.
For example, Li et al. <cit.> utilized image captioning to obtain language descriptions of panoramas and employed generative models to produce novel views based on these descriptions.
Conversely, another category focuses on graph-level adjustments rather than the visual level to go beyond the established navigation graphs by mixing up different graphs <cit.> or substituting a fixed viewpoint with proximate locations <cit.>.
Notably, VLN-CE <cit.> abandons the graph-based navigation paradigm and allows agents to traverse continuous environments freely to enhance task realism.
Differently, our task makes changes at both visual and graph levels to block edges in the graph and generate obstacles in the views to generate instruction-reality mismatches.

 Obstacle Avoidance

Obstacle avoidance has been a long-standing research challenge in visual navigation <cit.>.
In VLN, ETPNav <cit.> applies an obstacle-avoiding controller with a trial-and-error heuristic to explicitly escape from deadlocks.
SafeVLN <cit.> employs a LiDAR-based waypoint predictor and a re-selection strategy to avoid non-navigable waypoints and obstacles. 
However, our work diverges from these works under the continual setting <cit.> in three aspects, leading us to choose the discrete setting.
Firstly, we target a broader range of situations with instruction-reality mismatches, including avoidable obstacles, closed doors, rearranged furniture, instruction errors, etc.
These scenarios can be effectively captured by graph changes but are hard to model as continuous signals.
Secondly, while obstruction avoidance focuses on assessing the properties of obstructions to avoid them, our work emphasizes changes in navigation graphs that render the instructions temporarily invalid, ignoring the nature of the obstructions.
The discrete setting decouples path planning from technicalities, such as the shape and size of obstacles, thereby focusing on high-level adaptability when instructions fail and enhancing its generality.
Finally, addressing instruction-reality mismatches involves more than just navigating around obstructions. 
It requires a complex strategy to find detours and the ability to navigate without instruction guidance, posing a significant adaptability challenge which is too hard for current continuous agents.

 Object Insertion
Inserting novel objects into target images has always been a research challenge in Computer Vision.
Early works <cit.> employ the cut-and-paste strategy to merge two pictures, which is straightforward but lacks photorealism.
With the development of neural networks, object insertion can be achieved in various ways and higher quality, including image synthesis <cit.>, neural radiance fields editing <cit.>, image generation <cit.>, etc.
In navigation, THDA <cit.> inserts 3D scans of household objects into random locations as navigation goals to augment the training data for ObjectGoal Navigation <cit.>.
In VLN, Envedit <cit.> leverages semantic image synthesis <cit.> to create objects based on modified environment semantics as augmented environments.
In this work, we uniquely take advantage of the text-to-image inpainting models <cit.> to seamlessly embed desired objects within specific view locations without influencing visual surroundings.



§ OBSTRUCTED ENVIRONMENTS



In this section, we first introduce the problem formulation of VLN on vanilla and obstructed environments, and then describe our proposed R2R-UNO dataset.



 §.§ Problem Setup
 
In VLN, the agent must follow a natural language instruction x̂=(x_1,x_2,...,x_L) with L words to navigate a simulated environment.
This environment is usually discrete with a predefined undirected navigation graph 𝒢={𝒱, ℰ} with navigable nodes 𝒱 and connectivity edges ℰ.
At each timestep t, the agent perceives a panoramic representation, comprised of N=36 views O_t={o_t^i}_i=1^N and an orientation feature that encodes heading θ and elevation ϕ information of its current node v_t. 
It then determines an action a_t to transition to one of the neighboring nodes 𝒩(v_t) by selecting the view that aligns best with the target node from the current panorama O_t.
An additional “STOP” action is available to conclude the navigation process.
For each instruction x̂, there exists a corresponding ground truth path P=<v_1, e_1, v_2, e_2, ⋯, v_n> representing the intended trajectory with n nodes for the agent. 
However, previous work presumes perfect alignment between instruction and reality, assuming that all edges <e_1, e_2, ⋯, e_n-1> are consistently accessible.
This assumption overlooks the dynamics of real-world navigation, where the graph 𝒢 may have changed.
Among various reasons for graph changes, we focus on the most representative one: obstructions.
In our obstructed environments, some specific edges E_s ⊆ P might be obstructed, making parts of the instruction inapplicable to the current scenario. 
This obstruction also leads to corresponding visual changes in the panoramic views O_t of nodes linked by these edges. 
When meeting obstructed edges E_s, the agent must find a substitute path P' to reach the final destination v_n.



 §.§ R2R-UNO



In this section, we introduce the proposed R2R-UNO dataset in detail, including the graph changes and visual modifications.





  §.§.§ Graph Changes

For a path P=<v_1, e_1, v_2, e_2, ⋯, v_n> in R2R, we first identify all redundant edges within this path, denoted as E_r ⊆ P.
A redundant edge e_i=(v_i, v_i+1) ∈ E_r is defined by the property that its removal does not affect the overall connectivity of the graph, ensuring an alternate path P' between v_i and v_i+1. 
To account for scenarios where multiple edges are concurrently obstructed, we assess the collective redundancy of combinations of these redundant edges E_r and categorize these combinations by the number of obstructed edges x into Block-x set.
Since R2R paths have 4 to 7 nodes in length, we set N_max=3 as the maximum number of obstructed edges, generating three sets Block-1,2,3, each containing individual training and evaluation splits.

With a Block-x set, consider a combination of redundant edges C=<e_i_1, e_i_2, ⋯, e_i_x>.
For each edge, we identify the shortest path <P'_1,P'_2,⋯,P'_x> between the corresponding nodes in the modified graph 𝒢={𝒱, ℰ-C} to replace C in the original path P, generating the path P as the new intended path for agents:

    P̅ = ⟨ v_1, …, v_i_j, P'_j, v_i_j+1, …, v_n ⟩,   ∀ j ∈1, 2, …, x

We refer to P̅ as the “real path” to be traversed by agents, in contrast to the “instructional path” P based on the navigation instructions x̂.
Therefore, the instruction-reality mismatch is defined as P≠ P, which demands agents actively seek alternate paths. 
To exclude excessively lengthy and impractical new routes, each Block-x set possesses a “real path” length restriction as L(P̅) ≤ 10+5 · x. 
As a result, 98.8% R2R paths are modified to be one or more new paths in R2R-UNO.
As indicated in <ref>, the R2R-UNO sets contain a greater number and length of paths than R2R, with the average path length increasing with x, denoting greater navigation difficulty.

Notably, when constructing P̅, we deliberately avoid shortening the path when P' contains future viewpoints from P, even if this leads to the loop formation in P̅.
This design aligns with practical navigation scenarios since when deviating from the instruction, it is challenging for agents to align the current viewpoint with the described path due to the absence of contextual clues. 
Therefore, it is logical and efficient for agents to seek a detour around the obstruction before attempting to re-align with the instructions.



  §.§.§ Visual Changes


To align with the graph changes, we introduce two novel modules to infuse various objects into panoramic views of nodes along redundant edges, as shown in <ref>.

The first one is the object insertion module, which employs a stable diffusion inpainting model <cit.> to approach the issue from an inpainting perspective.
Consider a redundant edge e linking nodes v_a and v_b. 
We elaborate the process of modifying the panoramic view O_a={o_a^i}_i=1^36 at node v_a as follows, which is similarly applied to O_b.
First, we localize the other node v_b within O_a to find the corresponding discrete view o_a^b and calculate the pixel coordinates (x_b, y_b) in o_a^b as follows:

    f    = H / (2 ·tan(δ / 2)) 
    
    x_b    = tan(θ_r) · f + W/2 
    
    y_b    = tan(ϕ_r) · f + H/2

W, H is the width and height of o_a^b, while θ_r and ϕ_r is the relative heading and elevation from v_a to v_b, and δ is the vertical Field of View (FoV) to calculate the focal length f.
A right trapezoid mask M_a^b is then generated, expanding from the bottom edge to around the point v_b with a specified width to cover potential areas of paths from v_a to v_b.
This is based on the observation that paths in these views either stretch from the bottom middle to the target point (x_b, y_b) or form a vertical line from the bottom to the target. 
Finally, we combine an object name obj with a predefined prompt template to formulate the final prompts pr, which, along with the view image o_a^b and mask M_a^b, serve as the input to the inpainting model to generate the inpainted view ô_a^b:

    ô_a^b = Inpainting(o_a^b, M_a^b, pr)


While inpainting models can generate visually coherent and photo-realistic images, their instability is notable, with a certain probability of failing to incorporate the desired object.
This instability often arises when the area outside the mask already contains elements similar to the intended object, such as a “chair” or “table”, which can mislead the inpainting model even with intricate prompts.
Therefore, we propose a filtering module to improve the quality of inpainted images, which evaluates multiple generated candidates with different objects to filter out higher-quality ones.
Specifically, to enrich the generated scenarios, we first carefully select ten object categories as obstructions based on their frequencies in both the Matterport3D dataset and real-world environments, including chair, table, sofa, potted plant, basket, exercise equipment, vacuum cleaner, suitcase, toy, and dog. 
We then utilize the object insertion module to produce a set of novel views as candidates, each with a different kind of object.
To reflect the inpainting quality, we employ the CLIP <cit.> model to assess the compatibility score s_c of each view-object pair.
Note that this assessment considers only the modified part of the view by combining the mask M_a^b:

    s_c = CLIP(ô_a^b ∘ M_a^b, obj)

The ∘ operator represents pixel-wise multiplication.
These scores are aggregated to form a dataset including all redundant edges within R2R. 
Visual analyses of this dataset indicate that the score distribution for each object category often takes the shape of a bimodal Gaussian distribution, with one peak 𝒩(s_c | μ_1, σ_1^2) for successful incorporation and another 𝒩(s_c | μ_2, σ_2^2) for failure:

    P(s_c) = π_1 𝒩(s_c | μ_1, σ_1^2) + π_2 𝒩(s_c | μ_2, σ_2^2)

π is the mixing coefficient, 𝒩(·| μ, σ^2) represents a Gaussian distribution with mean μ and variance σ.
Based on this insight, we train a bimodal Gaussian Mixture Model (GMM) <cit.> for each category and use them to decide which images are qualified: 

    q =  
    1    if 𝒩(s_c | μ_1, σ_1^2) > 𝒩(s_c | μ_2, σ_2^2) 
    
    0    otherwise

Finally, we randomly select one of those qualified candidates as the modified view, or, in the absence of suitable options, the choice is narrowed to the top three scorers.
This module not only largely enhances the probability of incorporating visible obstructions but also ensures the selection of contextually fitting objects.
We present the distributions of object categories and their compatibility scores in the appendix to prove the diversity of R2R-UNO.

All steps above only concern one discrete view o_a^b.
The final step is propagating the updated view ô_a^b to adjacent views o_a^j_1:8 containing overlapped areas with o_a^b to maintain consistency across the panorama.
We first calculate the transformation matrix K_i for view o_a^j_i based on SIFT features <cit.> matching.
We then apply K_i to project the inpainted part of ô_a^b onto the adjacent view and merge it with the original view to construct a coherent novel panorama:

    K_i = MATCH(SIFT(o_a^b), SIFT(o_a^j_i)) 
       ô_a^j_i = o_a^j_i· (1 - K_i * M_a^b) + K_i * ô_a^b · (K_i * M_a^b)

Here, * denotes the projection operation by pixel-wise matrix multiplication applied to image coordinates.

It is important to note that we only perform 2D inpainting for nodes linked by redundant edges, which may lead to multi-view inconsistency.
Although 3D object insertion can alleviate this problem, we stick to the 2D method for two reasons.
Firstly, current 3D techniques <cit.> mainly rely on limited pre-built object datasets <cit.>, which lack diversity and can cause visual inconsistencies in aspects like lighting and style. 
Other generative 3D methods <cit.> struggle to generate high-resolution photo-realistic images and often produce noticeable artifacts.
Secondly, the impact of multi-view inconsistency is minimal in our task, as instruction-reality mismatches are primarily defined by graph changes.  
Agents can only detect graph changes when moving to nodes alongside redundant edges, making visual modifications at other nodes irrelevant since agents should still follow instructions without detecting the mismatch.
Moreover, various environment augmentation methods <cit.> have proved that this inconsistency would not affect agent performance in real-world scenarios.



§ INSTRUCTION-REALITY MISMATCHES AND SOLUTION


In this section, we show that current VLN methods perform poorly when encountering instruction-reality mismatches and thus propose the ObVLN method to solve this problem.





 §.§ Current VLN Methods Struggle in R2R-UNO



To assess the impact of instruction-reality mismatches, we conduct zero-shot evaluations on the R2R-UNO validation unseen splits using five advanced VLN methods, known for their excellent performance under the perfect instruction assumption, including RecBERT <cit.>, HAMT <cit.>, DUET <cit.>, GELA <cit.> and ScaleVLN <cit.>.
<ref> illustrates how their success rates vary with different numbers of blocked edges.
While most methods achieve a 60%–70% success rate in original environments, the curves sharply decline to around 40% when a single edge is obstructed, with further deterioration observed in the Block-2 and Block-3 scenarios. 
ScaleVLN outperforms other methods in R2R-UNO owing to its strong generalization ability brought by extensive training data, but it still suffers from a nearly 20% performance reduction in the block-1 set of R2R-UNO.
We conclude that current VLN models are overly focused on instruction-following capabilities and lack essential basic navigation functionality to adapt to graph changes.
These findings emphasize the severity of using the perfect instruction assumption in VLN research and the urgency for its resolution.





 §.§ ObVLN

With R2R-UNO, agents can be trained in obstructed environments to better adapt to instruction-reality mismatches.
However, significant gaps exist between original and obstructed environments, such as navigation requirements, reliance on instructions, and trajectory numbers, making direct training on these two types of data compromise the performance in both environments (See <ref>).

Therefore, we propose Obstructed VLN (ObVLN), including a novel training strategy and a graph construction mechanism to organize the training and help agents deal with obstructions.
We advocate a curriculum learning strategy to treat original and obstructed environments as distinct yet complementary tasks to facilitate the training.
This strategy leverages the instruction-following skills gained in original environments as a foundation to tackle obstructed settings.
Initially, agents are trained in purely unobstructed environments.
As training progresses, the sample ratio of obstructed environments α gradually increases until it reaches a predefined maximum α_max at step c, defined as follows:

    α(t) = min(α_max, t/c·α_max)

where t is the current training step.
All training batches include instances from both settings to ensure smooth optimization.

To facilitate exploration for detours, we introduce a graph construction mechanism for graph-based methods to incorporate “virtual nodes” to represent currently inaccessible nodes due to obstructions.
This mechanism consists of three steps, with the pseudo-code in the appendix.
Firstly, when encountering an obstacle at node v_a, we estimate the location of the obstructed node v_b based on the direction of the obstructed view and a predefined distance d (3 meters in R2R-UNO).
This estimation allows us to conceptualize a virtual node v^*_b within the topological graph as a placeholder for v_b. 
Next, as the agent moves, it consistently computes its distances to v^*_b and v_a, denoted as D(v^*_b) and D(v_a), respectively. 
This distance computation is crucial for determining the proximity of the agent to the virtual and obstructed nodes.
Finally, if the distances satisfy D(v^*_b) < θ and D(v_a) < θ for a given threshold θ, and the current view indicates obstruction, we infer that the agent has arrived or moved close to v_b. 
So we connect the virtual node v^*_b and current node v_c with a zero-weight edge, indicating their alignment.
This mechanism helps agents to seek alternative routes more purposefully and increase exploration efficiency.



§ EXPERIMENTS




 §.§ Datasets and Evaluation Metrics

We mainly evaluate our methods on the widely used VLN benchmark R2R <cit.> with step-by-step instructions and our proposed R2R-UNO to focus on the challenge of instruction-reality mismatches. 
We present results on the goal-oriented benchmark REVERIE <cit.> without such mismatches in the appendix.
R2R is built on the Matterport3D dataset <cit.>, including 10,800 panoramic views from 90 building-scale scenes.
Each R2R path has three or four natural language instructions from human annotators.
R2R has four splits: train, validation seen (val seen), validation unseen (val unseen), and test unseen.
We utilize the train split for training and the val seen and val unseen splits for evaluation to align with R2R-UNO.

For evaluation, we follow previous works <cit.> to use four primary metrics in VLN: 
(1) Trajectory Length (TL): the total navigation length in meters; 
(2) Navigation Error (NE): the distance between the stop location and the target; 
(3) Success Rate (SR): the ratio of agents stopping within 3 meters of the target; 
(4) Success rate weighted by Path Length (SPL) <cit.>: SR normalized by the ratio between the length of the shortest path and the predicted path. 
Metrics related to instruction fidelity like CLS <cit.> or nDTW <cit.> are not included due to the modified trajectory with obstacles.



 §.§ Implementation Details


For R2R-UNO creation, we use the stable-diffusion-v1.5-inpainting model for object insertion and the CLIP ViT-L/14 to evaluate text-image pairs.
We use brute force and K-Nearest Neighbors matching to align the SIFT features of two adjacent views.
For methods in <ref>, we use their best models on the validation unseen split and extract the features of the obstructed environments according to their settings.
We employ HAMT <cit.> and DUET <cit.> for navigation training and follow the implementation details in their official repositories.
The obstructed environments are only used in the fine-tuning stage.
The maximum sample ratio α_max is set to 0.5 with the increasing step c as 20,000.
We increase the maximum action length to 30 for longer ground truth paths.
PREVALENT <cit.> is used as the augmented data to stabilize the training.
We utilize the AdamW optimizer <cit.> and select the best model based on the average SPL in the val unseen splits of R2R and R2R-UNO.
All models are fine-tuned for 100K iterations with a learning rate of 1e-5 and a batch size of 8 on a single NVIDIA A6000 GPU.



 §.§ Main Results

We first demonstrate that our proposed obstructed environments and ObVLN can facilitate agent adaptation to instruction-reality mismatches.
Therefore, we apply HAMT and DUET to three distinct training settings:
(1) Basic: training with R2R; 
(2) +OE: incorporating data from both the R2R and R2R-UNO;
(3) +ObVLN: using the ObVLN method on R2R and R2R-UNO.
Note that both settings 2 and 3 include all three sets of R2R-UNO.
<ref> presents the navigation performance of different models on the val seen and unseen splits of R2R and R2R-UNO.
The results show that models trained with two types of data significantly outperform those trained only on R2R in navigating obstructed scenes within the R2R-UNO dataset.
However, this improvement is accompanied by a noticeable degradation in R2R performance, like the 6% and 7% SR drop of DUET in the seen and unseen splits, respectively.
This phenomenon can be explained by the considerable TL increase in R2R, suggesting that agents are over-optimized for obstructed environments and tend to take detours even without obstructions.
Our ObVLN method effectively addresses this issue.
For HAMT, ObVLN surpasses the basic setting in R2R, achieving the best results across all four sets of R2R and R2R-UNO.
For DUET, the topological map and global action space lead agents to return to the node with the blocked edge which aligns best with the instruction.
This results in a slight loss of efficiency in terms of SPL in R2R for ObVLN.
However, it still achieves a comparable SR to the basic setting and significantly outperforms setting 2 in R2R.
Additionally, it achieves state-of-the-art results in all three sets of R2R-UNO.
The advantage of HAMT with ObVLN is less significant because HAMT is not a map-based approach, making our graph construction mechanism inapplicable.

While DUET outperforms HAMT by a 7% SR increment on R2R, this superiority becomes much more significant in obstructed environments, in which DUET achieves around 25% SR lead in the most challenging Block-3 set.
We attribute this to the topological map design in DUET, which significantly enhances exploration efficiency and serves as a critical component in finding detours. 
This advantage is consistent with the large lead of DUET on exploration-intensive, object-oriented datasets like REVERIE <cit.>.



 §.§ Ablation Study

In this section, we conduct an ablation study on several important components to show their effectiveness.







  §.§.§ Two Modules for R2R-UNO

We first evaluate the impact of the Object insertion Module (OM) and the Filtering Module (FM) by generating different Block-1 sets for training and evaluation.
Without these two modules, the obstructed environments only contain graph changes without visual modifications.
<ref> presents the performance of HAMT and DUET on the val unseen splits using different modules.
The R2R performance is consistent across all experiments due to ObVLN. 
For obstructed environments, using the object insertion module to modify the panoramic views can enhance agent navigation by providing critical visual feedback on graph changes, evidenced by performance gains.
Moreover, including the filtering module further improves the inpainting quality, leading to more successful obstruction generations and the best performance.



  §.§.§ Sampling Strategy

We explore four sampling strategies for DUET to utilize both original and obstructed environments:
(1) Path-wise, where a path is randomly selected from the combined pool of R2R and R2R-UNO paths;
(2) Task-wise, where data is sampled from original and obstructed environments in a Bernoulli mixture distribution with probabilities 1-α_t and α_t, respectively;
(3) Instruction-wise, where for each instruction, one path is randomly chosen from all possible paths.
(4) Curriculum sample (Ours), which gradually increases the sampling ratio α up to α_max by step c.
<ref> shows their performances on the the val unseen splits of R2R and R2R-UNO.
We set α_t=0.5 to match α_max and provide experiments on different α_t and α_max in the appendix.
Among all strategies, our curriculum sampling achieves dual superiority, achieving the best performance in both R2R and R2R-UNO.
Other methods experience performance drops on R2R due to the over-optimization problem.
Path-wise sampling obtains the worst performance due to path imbalance, with more training on paths with many redundant edges while ignoring those with none or less.







  §.§.§ Graph Construction Mechanism

We compare our graph construction mechanism with two baselines: the Vanilla setting that overlooks the obstruction in the graph and an idealized Oracle setting that receives the ground truth information about nodes occluded by obstructions as a performance ceiling.
<ref> shows the performance of DUET with different graph construction methods on the val unseen splits of R2R-UNO.
Our approach, incorporating virtual nodes into the graph, significantly outperforms the vanilla setting on all three sets, especially in more challenging scenarios with three obstructed edges.
As expected, the Oracle graph achieves the highest performance due to its access to accurate, unobstructed node information. 
These findings emphasize the necessity of equipping agents with obstruction-aware capabilities and the benefits of integrating obstructions into the topological map.




 §.§ Qualitative Analysis

We present some obstructed environments from R2R-UNO in <ref> comparing with corresponding original views and those generated without the filtering module. 
Our method successfully integrates various objects into specific locations within the original views, creating realistic and contextually harmonious obstructions. 
In contrast, results without the filtering module often fail to include the objects, demonstrating the critical role of this module in enhancing the inpainting reliability.
Additionally, we evaluate the generation quality using the CLIP <cit.> score and present the scores for each category in <ref>.
The filtering module consistently improves and stabilizes the generation quality, achieving higher scores and lower variance across all categories,  aligned with the examples.



§ CONCLUSION

This work introduces obstructed environments into VLN to address the prevalent issue of instruction-reality mismatches in real-world navigation. 
We present R2R-UNO, the first VLN dataset to incorporate such mismatches by integrating diverse obstructions into R2R at both the graph and visual levels through a novel object insertion and filtering module.
Using R2R-UNO, we demonstrate that current VLN methods struggle in obstructed settings and further propose the ObVLN method to help agents effectively adapt to obstructed environments.
Through these innovations, our agents achieve state-of-the-art results in R2R-UNO while maintaining robust performance in R2R.
We believe addressing the perfect instruction assumption is crucial for the practical application of VLN agents and for assessing their adaptability to navigate beyond only following instructions.







This work is supported by projects DE200101610, DP230101753 funded by Australian Research Council, and CSIRO's Science Leader project R-91559.





ACM-Reference-Format






































 Paper Title: Automated Review Generation Method Based on Large Language Models.



1,2,6]Shican Wu
1,2,6]Xiao Ma
1,2]Dehui Luo
1,2]Lulu Li
5]Xiangcheng Shi
1,2,3]Xin Chang
1,2]Xiaoyun Lin
1,5]Ran Luo
1,2]Chunlei Pei
1,2]Zhi-Jian Zhao*
1,2,3,4,5]Jinlong Gong*

[1]School of Chemical Engineering and Technology; Key Laboratory for Green Chemical Technology of Ministry of Education, Tianjin University, Tianjin 300072, China
[2]Collaborative Innovation Center of Chemical Science and Engineering (Tianjin), Tianjin 300072, China
[3]Haihe Laboratory of Sustainable Chemical Transformations, Tianjin 300192, China
[4]National Industry-Education Platform of Energy Storage, Tianjin University, 135 Yaguan Road, Tianjin 300350, China
[5]Joint School of National University of Singapore and Tianjin University, International Campus of Tianjin University, Binhai New City, Fuzhou 350207, Fujian, China
[6]These authors contributed equally: Shican Wu, Xiao Ma
[*]Corresponding author. Email: zjzhao@tju.edu.cn; jlgong@tju.edu.cn





Automated Review Generation Method Based on Large Language Models
    [
    August 5, 2024
=================================================================



§ ABSTRACT 
    Literature research, vital for scientific advancement, is overwhelmed by the vast ocean of available information. Addressing this, we propose an automated review generation method based on Large Language Models (LLMs) to streamline literature processing and reduce cognitive load. In case study on propane dehydrogenation (PDH) catalysts, our method swiftly generated comprehensive reviews from 343 articles, averaging seconds per article per LLM account. Extended analysis of 1041 articles provided deep insights into catalysts' composition, structure, and performance. Recognizing LLMs' hallucinations, we employed a multi-layered quality control strategy, ensuring our method's reliability and effective hallucination mitigation. Expert verification confirms the accuracy and citation integrity of generated reviews, demonstrating LLM hallucination risks reduced to below 0.5% with over 95% confidence. Released Windows application enables one-click review generation, aiding researchers in tracking advancements and recommending literature. This approach showcases LLMs' role in enhancing scientific research productivity and sets the stage for further exploration.




§ INTRODUCTION


In scientific research, peer-reviewed academic literature serves as a dense and reliable medium for information dissemination, enabling researchers to push the boundaries of human knowledge by building on previous work<cit.>. The clarity and rigor of scientific language constrain information dissemination, making it a key carrier in the research process for entity description, concept extraction, information transfer, and consensus building. This ensures that in the transmission and evolution of knowledge, both the sender and receiver of information construct highly consistent models of the referenced objects and concepts on the cognitive level. For instance, the development of industrial catalysts requires a thorough understanding of the materials' structure, chemical properties, and reactivity, considering their activity, selectivity, and stability<cit.>. This necessitates leveraging foundational catalytic theories and reaction mechanisms detailed in literature. However, the rapid pace of literature publication has outstripped researchers' capacity to assimilate information<cit.>, highlighting the need for tools to efficiently analyze and integrate data, thus avoiding redundant discoveries and broadening research perspectives, ultimately facilitating scientific research processes such as catalyst development.

Natural language processing (NLP), a branch of machine learning (ML), with core tasks like co-reference resolution and semantic analysis<cit.>, is well-suited as such a tool for literature understanding and integration. NLP technology has evolved through feature engineering, neural network architecture engineering, pre-trained language model development, and prompt-based learning<cit.>. Recent years have seen NLP applied in catalysis literature for extracting synthetic methods<cit.>, materials and properties<cit.>, key reaction parameters<cit.>, and reactions<cit.>. However, these studies often focus on isolated aspects, which limits their transferability and requires prior domain knowledge and programming skills, making it challenging for newcomers. Naturally, we considered integrating literature information in the form of reviews, which can also be generalized to more disciplines. However, early attempts at automated review generation often treated it as a multi-document summarization (MDS) task<cit.>, relying on existing reviews and citation networks<cit.>, thus may not keep pace with rapid advancements in research. Additionally, bibliometric analyses can overlook recent studies due to insufficient citations; focusing solely on abstracts or citations rather than full texts<cit.> may miss critical details. Focusing on extractive summarization rather than integrated generation<cit.>, or filling in sentence templates<cit.> can omit significant information or be redundant.

Since November 2022, Large Language Models (LLMs) like ChatGPT, representing breakthroughs in NLP, have demonstrated unprecedented language comprehension abilities<cit.>. LLMs excel in zero-shot and few-shot learning, common sense, logical reasoning, and versatility across NLP tasks<cit.>, serving as a second brain for researchers that processes and comprehends extensive scientific literature without additional foundational knowledge<cit.>. However, LLMs' creative reprocessing of understood information differs from search engines and poses the challenge of "hallucinations", a prevalent but unresolved issue in the industry<cit.>. Research reveals that calibrated LLMs inevitably generate hallucinations<cit.>, a phenomenon unavoidable in any computable LLM, regardless of model architecture, learning algorithm, prompt techniques, or training data<cit.>. Hallucinations in LLMs, which often emerge from statistical biases or noise in the training data and strategies for handling ambiguous information, manifest as baseless false information, contextually misaligned or unrelated responses<cit.>. Hallucinations are exacerbated in specialized domains by limited data exposure, leading LLMs to produce seemingly credible yet misleading outputs with specialized terminology, posing significant risks in scientific research where accuracy is critical. Even advanced LLM like GPT-4 achieve only 73.3% accuracy in professional contexts<cit.>, risking inaccurate academic conclusions and misdirected research, causing considerable time and resource losses<cit.>. Ensuring hallucination mitigation is crucial for the scientific integrity and reliability of automated review generation. Galactica<cit.>, which claimed review generation capabilities, was withdrawn due to hallucination concerns, highlights this necessity. Therefore, this study emphasizes mitigating hallucinations and enhancing reliability to safeguard the quality of LLM-generated reviews.

Addressing existing methodological limitations and leveraging LLMs' potential, this study develops an LLM-based, efficient, and comprehensive automated review generation approach. It features an end-to-end pipeline for literature retrieval, reading, summary distillation, and coherent text organization, underpinned by a multi-tier quality control strategy to counter LLM hallucination risks. LLMs' adeptness at information refinement and knowledge building enables accurate entity and concept extraction from texts, bolstering literature handling, refinement, and knowledge construction capabilities of researchers. By exploiting computational strengths in storage and parallel processing, LLMs overcome human cognitive constraints, easing researchers' cognitive load and aiding in the rapid identification of research trends. This automated review generation method thus presents a panoramic field view, offering a one-click solution for researchers lacking relevant background. Below is a comparative analysis of automated and traditional review generation methods (see Table 1):



The automated review generation method based on LLMs holds substantial scientific significance in research. It enhances literature processing efficiency and quality, fosters new knowledge discovery, and stimulates innovation. This method is invaluable in advancing the scope and depth of contemporary scientific research. As an innovative literature processing tool, it could become integral to scientific research infrastructure, significantly promoting scientific research progress.



§ RESULTS




 §.§ Automated retrieval


In our study, automated review generation essentially reprocesses the retrieved information. The process hinges on efficiently retrieving and extracting pertinent information from extensive scientific literature, with the review's quality and scope directly tied to the retrieval process's comprehensiveness and accuracy. We utilized SerpAPI for automated retrieval on Google Scholar, focusing on propane dehydrogenation (PDH) catalysts, covering literature from 1980 to 2024 in top-tier(Q1) chemistry and chemical engineering journals according to the 2022 Chinese Academy of Sciences division table.

The automated retrieval yielded 1420 initial results from Google Scholar. To address the challenge of irrelevant or duplicate findings, we implemented a dual-level filtering process. The first level employed quick filtering of abstracts and titles to remove obviously irrelevant documents, serving as a rapid but less precise narrowing method. The second level involved deeper LLM-based analysis of full texts, offering higher accuracy albeit at a slower pace. This coarse-to-fine screening method, reminiscent of high-throughput screening, enabled us to efficiently and accurately identify literature pertinent to our research. The initial screening shortlisted 343 articles as related to our topic. Subsequent LLM evaluation further confirmed 238 of these articles as relevant.



 §.§ Implementation and analysis of one-click automated review generation


Using PDH catalysts as an example and building on the aforementioned automated retrieval, we have effectively produced high-quality, specialized review articles. By focusing on top-tier journals, we ensured the retrieval of articles with significant academic impact, offering an accessible starting point for users new to the domain. For those with domain familiarity, the program allows the specification of a custom journal list to refine article selection.

We evaluated the efficacy of this method by contrasting two strategies for constructing review topics: one based on existing reviews and another using LLM-generated topics (see Table 2). The examples showcased in subsequent sections and the Supplementary Information are based on outlines derived from existing reviews.



Key benefits of the generated reviews, irrespective of the topic construction method, include:


    
  * Content Accuracy: The content has been manually checked by experts in the relevant field, with no errors in knowledge, correct referencing of cited literature, and a length and citation count that align with conventional review standards. Specific examples of generated content can be found in the Supplementary Information.
    
  * Customizable Research: Enables the addition of specific questions to tailor research focus and refine review specificity.
    
  * Forward-Looking Insights: Each topic includes a section on "Comprehensive understanding and prospective outlook", providing profound insights and innovative suggestions by LLM. This allows the LLM to engage in divergent thinking and integration beyond the information domain provided by the literature.


To improve accessibility, we developed a Python3 graphical user interface (GUI), enabling straightforward, one-click review generation on Windows, requiring no programming skills or domain knowledge.



 §.§ Data mining and visual analysis


In this study, the data mining module was deployed for comprehensive analysis in the PDH catalysts domain, examining literature from 1980 to 2024 within the chemistry and chemical engineering journals ranked Q1, Q2, and Q3 by the 2022 Chinese Academy of Sciences. Out of 1041 articles filtered by abstracts and titles, 839 were pinpointed as pertinent to PDH research via LLM selection. Leveraging LLMs for data extraction and subsequent analysis, we provided insightful conclusions on catalysts' composition, structure, and performance. This approach not only highlighted PDH research trends but also explored the maximum performance of individual factors and the synergistic effects between variables.

For instance, a statistical analysis of the annual publication numbers by catalyst types (see Figure 1 (a)) and sources of performance enhancement (see Figure 1 (b)) showed a surge in alloy research since 1995 and a spike in single-atom catalyst studies post-2015, and primarily driven by advancements in structural composition. This trend underscores the PDH field's evolving focus and hints at fresh avenues for catalyst development, including synthesis methods. In our analysis of the impact of promoter elements (see Figure 1 (c)) and support materials (see Figure 1 (d)) on catalyst performance, including selectivity and stability, we identified that promoter elements such as Zn, Sn, and La, as well as support materials like alumina and zeolites, can achieve notable peak performance, which signaled pathways for catalytic innovation. The combination analysis, for instance, of active site elements with composition elements (see Figure 1 (e)) and alloy structure types with preparation methods (see Figure 1 (f)), revealed that multi-metal systems generally outperform single-metal systems, especially when promoter elements like Sn, Zn, In are used to enhance the performance of Pt-based catalysts. Moreover, impregnation-prepared nanometallic catalysts exhibited superior conversion rates and selectivity, while single-atom alloys showed high selectivity but lower conversion rates.

This comprehensive analysis reveals the nuanced interplay between variables, guiding future research towards optimizing catalyst performance, aiding researchers in achieving the optimal performance balance in catalyst design and optimization. It suggests selecting Pt-based catalysts for maximum selectivity or metal oxides for enhanced conversion rates, and conducting deeper exploration into single-atom and nanostructured catalysts, which show promise in exceeding the efficacy of conventional catalysts. These insights not only showcase the diverse characteristics and performance benchmarks within the PDH domain but also highlight LLMs' utility in scientific exploration, providing researchers with real-time domain understanding and progress perception, thereby fostering catalyst development. This holistic approach empowers researchers to refine catalyst design and optimization effectively, aligning with industrial needs.





 §.§ Hallucination mitigation


To address the challenge of hallucinations in LLMs, a high priority has been placed on the detection and prevention of such phenomena. In the entire automated review generation process, we adopted a multi-level filtering and verification quality control strategy, similar to the concept of retrieval-augmented generation (RAG)<cit.>, to mitigate and correct hallucinations:



  §.§.§ Prompt design and task decomposition

Firstly, we utilized strict and clear text summary guiding prompts, aimed at enhancing the scientific rationality of LLM's outputs and ensuring accuracy and reliability in its analysis and generation processes. Notably, the task of automated review generation aligns well with the strengths of LLMs —information extraction and text generation capabilities. LLMs can rapidly and accurately extract core information from a vast array of literature and integrate it into a coherent and rigorous review text. To enhance efficiency and quality, we deconstructed the core of the review writing process, namely literature reading and summarization, into a series of text summarization tasks. This approach is adopted because summaries generated by LLM significantly surpass manually crafted and fine-tuned model-generated summaries in terms of fluency, factual consistency, and flexibility<cit.>. By establishing a list of questions, we directed the model to extract relevant content from the literature and respond based on this content, subsequently conducting a comprehensive analysis of all literature citations and responses. Ultimately, the LLM generates high-quality paragraphs closely related to the topic. Additionally, we employed a single-round, segmented generation strategy to avoid truncation limitations of approximately 8K output length. By reasonably segmenting long texts for generation, we not only ensured that the output was completed in a single conversational round but also provided finer parallel granularity to improve generation efficiency. In practice, we divided the 35 questions into 5 groups, ensuring that the generation results for each group could be successfully completed within the 8K limit of the LLM. This granularity avoids efficiency drops due to a high proportion of shared content and identical prompt frameworks, thereby enhancing processing speed while ensuring the quality of text generation.



  §.§.§ Hallucination filtering and verification

To mitigate and rectify hallucinations, we employed a layered filtering and verification approach:


    
  * Text format filtering: Noting that hallucinations often disrupt text formatting, we applied a predefined XML format template to filter out disarrayed texts.
    
  * DOI verification: DOIs, a combination of symbols and numbers lacking direct semantic linkage to context, present a challenge in generation and are prone to hallucinations. Yet, the precise reference nature of DOIs allows for verification. Through strict DOI verifications on generated content, we suppressed hallucinatory content from advancing further, ensuring each generated conclusion is traceable to its original source.
    
  * Relevance verification: Within the RAG system, documents related in semantics but lacking correct answers are particularly detrimental<cit.>. We scrutinized each response in the knowledge extraction phase to ensure its relevance, eliminating off-topic answers with relevant keywords.
    
  * Self-consistency<cit.> verification: For text summarization, where a definitive correct answer exists, recognizing that the stochasticity of hallucinations means correct answers should recur more frequently across iterations, we employ aggregation from repeated queries to effectively suppress hallucinations.
    
  * Full data stream traceability mechanism: By using DOIs as key reference identifiers for each piece of generated content and mandating citations for every conclusion, we enable review readers to easily trace back to the original literature, supporting verification and deeper exploration in topics of interest.




  §.§.§ Effectiveness of hallucination mitigation

In evaluating the effectiveness of hallucination mitigation, we employed a confusion matrix to classify outcomes according to whether the LLM provided content and its pertinence to the original text, differentiating between two types of inaccuracies: false positives, which include fabricated or inconsistent information, and false negatives, referring to overlooked or partially extracted content. Our focus was primarily on reducing false positives, while adopting a relatively tolerant stance on false negatives.

Substantial progress was made in mitigating hallucinations. During paragraph generation, only 36% of outputs met criteria following format and DOI validations. This was achieved through 9 repetitions of generating 35 paragraphs, cumulatively resulting in 875 generations. Analyzing 343 relevant articles, we executed 1715 information extractions across 35 questions, yielding 8575 responses and ultimately aggregating to 2783 valid information combinations. Impressively, 84.80% of these outcomes were confirmed by the LLM as 100% consistent with the aggregated results (see Table 3 and Figure 2 (a)), affirming the model's reliability. This method also establishes an rough benchmark for hallucination ratio, facilitating the selection and evaluation of LLMs.

Upon conducting manual verification on 25 articles each from the knowledge extraction and data mining stages, we calculated the accuracy, false positive rate, 95% confidence interval of the false positive rate, precision, recall, F1 score, and consistency (see Table 3). The 95% confidence interval for the false positive rate was provided by the statsmodels library in Python3. The results are detailed in the following table.





The data comparison underscores the efficacy of self-consistency verifications, revealing a substantial decrease in hallucinations, i.e., false positive content, while also compensating for some false negatives, where information was not fully extracted (see Figure 2 (b)). In the knowledge extraction phase, critical for review content, our manual sampling found no fabricated conclusions by LLMs (see Figure 2 (a)), attesting to our method's scientific integrity and reliability. From the sampling results, we are over 95% confident that the likelihood of hallucinations in this part is less than 0.5% (see Table 3). Analysis of false positives in the post-aggregation data mining phase revealed hallucinations typically involved correct numerical extraction but with errors in units or definitions. False negatives mainly stemmed from LLMs' inability to comprehend highly abstract expressions, reflecting a general LLM's limited understanding of highly specialized scientific concepts. The incidence of hallucinations in knowledge extraction was significantly lower than in data mining, as answering questions did not involve converting units and concepts, thus avoiding the most challenging part of testing an LLM's grasp of scientific knowledge. Domain-specific models enhanced by domain-adaptive pretraining (DAPT) <cit.>are poised to mitigate this issue. Opting not to fine-tune LLMs for specific domains in this study prioritizes out-of-the-box functionality and multi-domain generalization, utilizing a general LLM as the base. Comparisons between RAG and fine-tuning effects in specific domains indicate that RAG sustains efficacy with contextually new knowledge and offers a significantly lower initial cost<cit.>, aligning with our objective to support researchers' entry into diverse fields efficiently.

Considering the stringent accuracy requirements in research, increasing the number of repetitions can significantly reduce the probability of hallucinations appearing in aggregated results. Binomial probability calculations indicate that theoretically, a model with 79.09% accuracy yields aggregated prediction accuracies of 93.49%, 96.12%, and 97.64% after five, seven, and nine independent predictions, respectively, aligning with our sampling results (see Table 3). Detailed sampling outcomes and calculations are available in the SI.



On this foundation, every conclusive description in the generated reviews is supported by literature references and has been verified by relevant field researchers through tracing the cited literature, confirming that all literature references are correctly linked to the original publications and that the descriptions in the generated reviews correspond to those in the original publications.

This multi-layered strategy for hallucination control has built an effective verification system, ensuring the scientific integrity and reliability of the automated review generation. Furthermore, through a full data stream traceability mechanism, the authenticity and practicality of the content are further strengthened. This not only provides a secondary means of hallucination mitigation but also allows researchers to delve into original research papers for more precise and detailed academic information while accessing fast, automated research reviews. The strategy also implements a kind of literature recommendation mechanism. Since each content segment includes related DOIs, researchers can quickly locate specific original literature based on their interests and research needs, enabling deeper academic exploration.



§ DISCUSSION


In this study, we introduce an innovative LLM-based automated review generation method, adeptly addressing two key scientific challenges: streamlining literature review efficiency and significantly reducing LLM hallucination risks. This modular, comprehensive, end-to-end solution integrates modules for literature search, topic formulation, knowledge extraction, and review composition, transforming an extensive corpus of scientific literature into coherent, detailed, and error-free reviews tailored to specific research themes. Notably, our advanced data mining module offers experienced users an in-depth field overview, exploiting the LLM's analytical prowess. Additionally, a user-friendly one-click program on Windows platforms significantly simplifies the review generation process.

A pivotal achievement of our method is its capacity to surpass traditional human resource limitations. Our rigorous quality assurance solution, encompassing format filtering, DOI verification, relevance verification and self-consistency verifications, ensures high reliability and traceability throughout the data processing pipeline. Expert evaluations with a case study of PDH catalysts confirm the method's efficacy, with reviews paralleling manual ones in length and citations, but without hallucinations and with impeccable reference accuracy. Through rigorous testing, including the analysis of 875 LLM outputs from a sample of 25 articles, we demonstrate over 95% confidence in reducing the hallucination probability to below 0.5% (see Table 3).

Our method's modular design offers excellent reusability and scalability. Individual modules like literature search, topic formulation, and knowledge extraction can serve various research purposes, like literature tracking, research topic discovery, and data mining datasets construction. Future enhancements will focus on augmenting LLM's comprehension of scientific concepts through pan-scientific field fine-tuning, elevating the method's overall utility. Planned upgrades include improving multimodal processing, automating scientific inquiries, personalizing text generation, and delving deeper into specific research areas.

In summary, our method signifies a major advancement in scientific research tools, offering rapid access to field breakthroughs and developments. It's set to transform the landscape of scientific research, with far-reaching implications for knowledge base construction, literature recommendation, and structured academic writing, heralding a new era in scientific research productivity and interdisciplinary collaboration.



§ METHODS


The method for constructing review articles consists of four parts: literature search, topic formulation, knowledge extraction and review composition, along with an additional data mining module for experienced users (see Figure 3).





 §.§ Literature search


Initially, a list of journals designated for the set review topic's subject area is obtained from journal classification tables. Then, literature containing specified keywords within these selected journals is retrieved via search engine's API. This is followed by a preliminary filter, checking each title and abstract for intersections with a selected list of keywords. Literature with intersections is saved, and those of a review nature are marked (see Figure 3 (i)). Our method supports various types of textual literature, including journals, patents, conference papers, books, etc. This means that any content in textual form can be included in the search scope, further expanding the application scenarios and coverage of our method. In our example, using "propane dehydrogenation" as a keyword, we retrieved 343 publications in top-tier Chemistry and Chemical Engineering journals (2022 Chinese Academy of Sciences classification), including 14 reviews, after filtering titles and abstracts with keywords like "propane dehydrogenation", "PDH", "ODH", "Oxidative Dehydrogenation", etc., through SerpAPI on Google Scholar.



 §.§ Topic formulation


There are two approaches to constructing review topics: one involves LLM directly drafting the outline, and the other is based on LLM refining and drafting outlines from existing literature reviews. After obtaining a list of topics, additional topics can be manually added and sorted as needed (see Figure 3 (ii)). In our example, the Claude2 model generated an outline including 12 topics directly, and another with 9 topics and 35 guiding questions based on existing review articles (see Table 2).



 §.§ Knowledge extraction


Based on the obtained list of topics, the LLM generates a list of questions for extracting information from literature, corresponding to each review topic. After repeating this process for multiple times for each article, all answers are concatenated. The LLM then determines whether the answers are relevant to the questions and aggregates them (see Figure 3 (iii)). In our example, in the case of PDH, after transforming the 35 guiding questions into questions for extracting information from literature, the Claude2 model was used to extract information from 343 top-tier articles five times, leading to the aggregation of 8575 responses into 2783 valid information combinations.



 §.§ Review composition


After associating each article's answers with their source DOI, paragraphs are generated and integrated for each topic. The LLM generates review paragraphs from all the answers combined, followed by summarization and outlook. After repeating multiple times, the LLM scores the generated paragraphs, selecting the best ones for each topic to form a preliminary draft of the full review. The full text is then polished with the help of the LLM, adjusting and checking citation formats to produce the final draft (see Figure 3 (iv)). In our example, each question's answers from various articles were combined into JSON format information groups, inputted into the Claude2 model for paragraph generation, integrated to form smooth paragraphs, repeated 9 times, scored by the Claude2 model based on criteria (as shown in SI), and polished to produce the final draft.



 §.§ Data mining


Based on the automated review generation method described above, we proposed a data mining method based on LLMs, catering to users with some domain knowledge. This method is almost identical to the knowledge extraction steps (see Figure 3 (ii)), effectively extracting and aggregating specific data from a large volume of literature. Users first define specific data extraction targets, which may include but are not limited to catalyst types, chemical compositions, and performance characteristics. On the established literature dataset, the LLM parses each article, extracting the user-defined target data multiple times and outputting in XML format. Similar to the knowledge extraction process (see Figure 3 (ii)), the LLM aggregates results from multiple extractions to finalize each article's information for each extraction target. The extracted data often require manual cleaning and processing, including correcting extraction errors, standardizing data formats, and removing redundant information to facilitate subsequent statistical analyses. The cleaned data is then further integrated and analyzed to form visual charts. The code for the cleaning process and chart statistics can be generated by GPT4, requiring no programming background from the user. In our example, in the case of PDH catalysts, relevant literature from tiers one, two, and three was downloaded using the literature search module, filtered through abstracts and titles totaling 1041 articles, of which 839 were deemed PDH-related by the Claude2 model. After data cleaning and processing through Python3 programming, the extracted data included catalyst types, composition elements, active species elements, promoter elements, support materials, alloy structural types, alloy preparation methods, propane partial pressure, reaction temperature, inlet flow rate, selectivity, conversion, selectivity and other key indicators, covering seven categorical variables such as structure and element composition and three continuous variables related to reaction conditions. Subsequently, corresponding charts were generated through code execution. Initially, we tallied the annual publication numbers of various catalyst influencing factors, represented in line or Gantt charts. Furthermore, we calculated the average of the top five selectivity and stability for all influencing factors across all catalyst data, visualized in radar charts to showcase the peak performance achievable by a specific factor. Lastly, through pairwise combination analysis of influencing factors, we produced 45 bivariate correlation bubble charts, intuitively demonstrating how different variable combinations affect overall catalyst performance. These bubble charts use bubble color intensity, size, and border thickness to represent the levels of selectivity, conversion, and stability, respectively.



§ DATA AVAILABILITY

tocsectionData availability

Our study leverages a dataset compiled from scientific literature acquired through our institution's subscription. Due to copyright considerations, the dataset itself cannot be made publicly available. However, we ensure that our research's integrity and reproducibility do not rely on direct access to these proprietary documents. Instead, we provide extensive documentation on the dataset's structure, the criteria used for literature selection, and the analysis methods applied, enabling interested researchers to reconstruct a similar dataset from publicly available resources or their institutional subscriptions.

Furthermore, to facilitate a deeper understanding of our research process and promote further exploration and innovation, we have made all intermediate data, excluding the copyrighted full-text articles, publicly available on GitHub [<https://github.com/TJU-ECAT-AI/AutomaticReviewGenerationData>]. This repository includes the prompts used in our study and the corresponding responses generated by the large language model. By sharing these resources, we aim to provide valuable insights into our methodology and encourage other researchers to build upon our work, advancing the field of natural language processing and its applications in scientific literature analysis.



§ CODE AVAILABILITY

tocsectionCode Availability

The custom code developed for this research is central to our conclusions and is made available to ensure transparency and reproducibility of our results. The codebase, including all relevant custom scripts and mathematical algorithms, has been open-sourced under the Apache 2.0 license and is accessible via our GitHub repository at [<https://github.com/TJU-ECAT-AI/AutomaticReviewGeneration>]. We encourage users to review the license for any usage restrictions that may apply. As stated in the text, all LLMs invoked in this article are Claude2.

It is important to note that our published graphical user interface (GUI) leverages certain APIs for functionality, which, due to legal and regulatory requirements, necessitate that users provide their own API keys. This requirement is detailed in the documentation accompanying the code repository to assist users in setting up and utilizing the GUI effectively.



§ ACKNOWLEDGMENTS

tocsectionAcknowledgments

The authors acknowledge the financial support from the National Key R&D Program of China (2022YFE0102000), the National Natural Science Foundation of China (No. 22121004, U22A20409), the Haihe Laboratory of Sustainable Chemical Transformations, the Program of Introducing Talents of Discipline to Universities (BP0618007) and the XPLORER PRIZE. The authors also appreciate the generous computing resources provided by the High Performance Computing Center of Tianjin University.



§ AUTHOR CONTRIBUTIONS

tocsectionAuthor contributions

J.G and Z.Z. conceived and supervised the project. S.W. and X.M. designed the research. S.W. led the manuscript drafting. D.L. and L.L. validated the scientific reliability of the method. X.S. and X.C. innovated at the intersection of computational models and catalytic science. X.L., R.L., and C.P. conducted data analysis and validation. All authors wrote and revised this manuscript.



§ COMPETING INTERESTS

tocsectionCompeting interests

The authors declare no competing interests.



§ ADDITIONAL INFORMATION

tocsectionAdditional information

Supplementary information is available for this paper.

Correspondence and requests for materials should be addressed to J.G.

50=0pt

    1
    Ermel APC, Lacerda DP, Morandi MIW, Gauss L. Literature reviews: modern methods for investigating scientific and technological knowledge. Springer Nature (2021).

    2
    Nørskov JK, Bligaard T, Rossmeisl J, Christensen CH. Towards the computational design of solid catalysts. Nature chemistry 1, 37-46 (2009).

    3
    Zhao Z-J et al. Theory-guided design of catalytic materials using scaling relationships and reactivity descriptors. Nature Reviews Materials 4, 792-804 (2019).

    4
    Resasco J, Abild-Pedersen F, Hahn C, Bao ZN, Koper MTM, Jaramillo TF. Enhancing the connection between computation and experiments in electrocatalysis. Nature Catalysis 5, 374-381 (2022).

    5
    Taylor HS. A theory of the catalytic surface. Proceedings of the Royal Society of London Series A, Containing Papers of a Mathematical and Physical Character 108, 105-111 (1925).

    6
    Mou TY et al. Bridging the complexity gap in computational heterogeneous catalysis with machine learning. Nature Catalysis 6, 122-136 (2023).

    7
    Lawrence S. Free online availability substantially increases a paper's impact. Nature 411, 521 (2001).

    8
    Lok C. Speed reading: scientists are struggling to make sense of the expanding scientific literature. Corie Lok asks whether computational tools can do the hard work for them. Nature 463, 416-419 (2010).

    9
    Khurana D, Koli A, Khatter K, Singh S. Natural language processing: state of the art, current trends and challenges. Multimed Tools Appl 82, 3713-3744 (2023).

    10
    Liu PF, Yuan WZ, Fu JL, Jiang ZB, Hayashi H, Neubig G. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. Acm Comput Surv 55, 1-35 (2023).

    11
    Wang L et al. A corpus of CO(2) electrocatalytic reduction process extracted from the scientific literature. Sci Data 10, 175 (2023).

    12
    Plata DL, Jankovic NZ. Achieving sustainable nanomaterial design though strategic cultivation of big data. Nat Nanotechnol 16, 612-614 (2021).

    13
    Cruse K et al. Text-mined dataset of gold nanoparticle synthesis procedures, morphologies, and size entities. Sci Data 9, 234 (2022).

    14
    Batra SR. Emerging materials intelligence ecosystems propelled by machine learning. Nature Reviews Materials, (2020).

    15
    Suvarna M, Vaucher AC, Mitchell S, Laino T, Perez-Ramirez J. Language models and protocol standardization guidelines for accelerating synthesis planning in heterogeneous catalysis. Nat Commun 14, 7964 (2023).

    16
    Vaucher AC, Zipoli F, Geluykens J, Nair VH, Schwaller P, Laino T. Automated extraction of chemical synthesis actions from experimental procedures. Nat Commun 11, 3601 (2020).

    17
    Boschen I. Evaluation of JATSdecoder as an automated text extraction tool for statistical results in scientific reports. Sci Rep 11, 19525 (2021).

    18
    Nandy A, Terrones G, Arunachalam N, Duan C, Kastner DW, Kulik HJ. MOFSimplify, machine learning models with extracted stability data of three thousand metal-organic frameworks. Sci Data 9, 74 (2022).

    19
    Olivetti EA et al. Data-driven materials research enabled by natural language processing and information extraction. Appl Phys Rev 7, (2020).

    20
    Huang S, Cole JM. A database of battery materials auto-generated using ChemDataExtractor. Sci Data 7, 260 (2020).

    21
    Court CJ, Cole JM. Magnetic and superconducting phase diagrams and transition temperatures predicted using text mining and machine learning. Npj Computational Materials 6, (2020).

    22
    He TJ et al. Similarity of Precursors in Solid-State Synthesis as Text-Mined from Scientific Literature. Chemistry of Materials 32, 7861-7873 (2020).

    23
    Kim E et al. Machine-learned and codified synthesis parameters of oxide materials. Sci Data 4, 170127 (2017).

    24
    Vaucher AC, Schwaller P, Geluykens J, Nair VH, Iuliano A, Laino T. Inferring experimental procedures from text-based representations of chemical reactions. Nat Commun 12, 2573 (2021).

    25
    Kim E, Huang K, Saunders A, McCallum A, Ceder G, Olivetti E. Materials Synthesis Insights from Scientific Literature via Text Extraction and Machine Learning. Chemistry of Materials 29, 9436-9444 (2017).

    26
    Mahbub R, Huang K, Jensen Z, Hood ZD, Rupp JLM, Olivetti EA. Text mining for processing conditions of solid-state battery electrolyte. Electrochemistry Communications 121, 106860 (2020).

    27
    Scheffler M et al. FAIR data enabling new horizons for materials research. Nature 604, 635-642 (2022).

    28
    Pesciullesi G, Schwaller P, Laino T, Reymond JL. Transfer learning enables the molecular transformer to predict regio- and stereoselective reactions on carbohydrates. Nat Commun 11, 4874 (2020).

    29
    Ma C, Zhang WE, Guo M, Wang H, Sheng QZ. Multi-document summarization via deep learning techniques: A survey. Acm Comput Surv 55, 1-37 (2022).

    30
    Nikiforovskaya A, Kapralov N, Vlasova A, Shpynov O, Shpilman A. Automatic generation of reviews of scientific papers. In: 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA). IEEE (2020).

    31
    Mohammad S et al. Using citations to generate surveys of scientific paradigms. In: Proceedings of human language technologies: The 2009 annual conference of the North American chapter of the association for computational linguistics (2009).

    32
    Agarwal N, Reddy RS, Kiran G, Rose C. Towards multi-document summarization of scientific articles: making interesting comparisons with SciSumm. In: Proceedings of the workshop on automatic summarization for different genres, media, and languages (2011).

    33
    Jaidka K, Khoo C, Na J-C. Deconstructing human literature reviews–a framework for multi-document summarization. In: proceedings of the 14th European workshop on natural language generation (2013).

    34
    Kasanishi T, Isonuma M, Mori J, Sakata I. SciReviewGen: A Large-scale Dataset for Automatic Literature Review Generation. arXiv preprint arXiv:230515186, (2023).
    
     35
    Liu Y et al. Summary of chatgpt-related research and perspective towards the future of large language models. Meta-Radiology, 100017 (2023).

    36
    Naveed H et al. A comprehensive overview of large language models. arXiv preprint arXiv:230706435, (2023).

    37
    White AD. The future of chemistry is language. Nature Reviews Chemistry 7, 457-458 (2023).

    38
    Zhang Y et al. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv preprint arXiv:230901219, (2023).

    39
    Sanderson K. GPT-4 is here: what scientists think. Nature 615, 773-773 (2023).

    40
    Kalai AT, Vempala SS. Calibrated Language Models Must Hallucinate. arXiv preprint arXiv:231114648, (2023).

    41
    Xu Z, Jain S, Kankanhalli M. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:240111817, (2024).

    42
    Wu S et al. A comparative study of open-source large language models, gpt-4 and claude 2: Multiple-choice test taking in nephrology. arXiv preprint arXiv:230804709, (2023).

    43
    Taylor R et al. Galactica: A large language model for science. arXiv preprint arXiv:221109085, (2022).

    44
    Lewis P et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33, 9459-9474 (2020).

    45
    Truhn D, Reis-Filho JS, Kather JN. Large language models should be used as scientific reasoning engines, not knowledge databases. Nature Medicine, 1-2 (2023).

    46
    Pu X, Gao M, Wan X. Summarization is (almost) dead. arXiv preprint arXiv:230909558, (2023).

    47
    Cuconasu F et al. The power of noise: Redefining retrieval for rag systems. arXiv preprint arXiv:240114887, (2024).

    48
    Wang X et al. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:220311171, (2022).

    49
    Gururangan S et al. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:200410964, (2020).

    50
    Gupta A et al. RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture. arXiv preprint arXiv:240108406, (2024).





 Paper Title: Navigating Text-to-Image Generative Bias across Indic Languages.





 










S. Mittal et al.




Department of CSE, IIT Jodhpur, Rajasthan, India Meta, Menlo Park, California, USA
 Weir P.B.C., Alameda, California, USA

Navigating Text-to-Image Generative Bias across Indic Languages
    Surbhi Mittal10000-0001-8910-4161 Arnav Sudan1 Mayank Vatsa10000-0001-5952-2274 Richa Singh10000-0003-4060-4573 

Tamar Glaser20009-0007-9322-2444 Tal Hassner30000-0003-2275-1406
    
======================================================================================================================================================================================



§ ABSTRACT 
This research investigates biases in text-to-image (TTI) models for the Indic languages widely spoken across India. It evaluates and compares the generative performance and cultural relevance of leading TTI models in these languages against their performance in English. Using the proposed IndicTTI benchmark, we comprehensively assess the performance of 30 Indic languages with two open-source diffusion models and two commercial generation APIs. The primary objective of this benchmark is to evaluate the support for Indic languages in these models and identify areas needing improvement. Given the linguistic diversity of 30 languages spoken by over 1.4 billion people, this benchmark aims to provide a detailed and insightful analysis of TTI models' effectiveness within the Indic linguistic landscape. The data and code for the IndicTTI benchmark can be accessed at <https://iab-rubric.org/resources/other-databases/indictti>.









§ INTRODUCTION



Text-to-image (TTI) generative technologies have transformed the landscape of digital media, distinguished by their extraordinary ability to produce detailed and visually engaging images from written text <cit.>. These models have found widespread application in a variety of fields, including gaming, animation, architecture, and fashion, offering significant benefits in terms of enhancing creativity and streamlining production timelines. However, despite the expansion of access through open-source models, challenges persist in achieving linguistic inclusivity <cit.>. The datasets these models are trained on are predominantly composed of English-language content, derived from both proprietary sources<cit.> and internet scraping <cit.>. This reliance on English-centric data adversely affects the quality of images generated from text in languages other than English.

As the integration of TTI models into various industries deepens, it becomes imperative to conduct a thorough evaluation of their performance and the diversity of the outputs they produce. While there has been research to study biases in TTI models<cit.>, the current evaluation landscape is notably lacking in benchmarks capable of effectively measuring these models' performance across a diverse linguistic spectrum, often overlooking the rich variety of global languages  <cit.>. To address this gap, we introduce the IndicTTI benchmark, an evaluation benchmark designed to explore and measure the biases present in text-to-image generation technologies, with an emphasis on a wider range of languages beyond those typically studied. This benchmark seeks to explore the linguistic diversity of a global population exceeding 1.4 billion, aiming not just to spotlight the shortcomings of existing TTI models but also to foster a broader application of these technologies that is truly inclusive and reflective of the myriad languages and cultures worldwide. Through this in-depth analysis, our goal is to spur improvements in TTI technologies, ensuring they cater to a wider audience and promote a more inclusive environment within digital media creation. 

In this research, we examine the performance of existing models in 30 languages other than English, with a specific focus on Indic languages, which are spoken by over a billion people worldwide. We believe that this benchmark is a step toward analyzing TTI models for a rich and diverse culture with a large population <cit.>. The 30 languages are written in 10 different scripts and belong to multiple language families. Among these, we have included the Scheduled Languages, which are considered the major literary languages of India and have a significant volume of literature <cit.>. 
We study two aspects of TTI models for assessing bias towards Indic languages, (i) model performance and (ii) model representation. We identify issues of correctness in generation, the presence of cultural elements, and the differences between open-source and API-based models. We utilize Stable Diffusion <cit.> and Alt Diffusion <cit.> models as part of our open-source analysis, and Midjourney <cit.> and Dalle3 <cit.> as part of the API-based generative models. We propose the Cyclic Language-Grounded Correctness (CLGC) metric to evaluate model performance and the Self-Consistency Across Languages (SCAL) metric to evaluate representation bias in the model generations. We further evaluate the model performance through Language-Grounded Correctness (LGC) and Image-Grounded Correctness (IGC) metrics.  Through our analysis, we observe how certain language scripts evoke imagery of Indian Gods in Midjourney, the overwhelmingly high generation of Asian women and couples in open-source models, and the high accuracy of generations in Dalle3 (Refer Fig. <ref>). Finally, we discuss the correlation between language script and cultural influences observed in the models.







§ RELATED WORK


Limited studies have been conducted to benchmark the performance of TTI models across different languages<cit.>. In NLP-based tasks, MEGA <cit.> benchmarks generative Large Language Models (LLMs) across 16 NLP datasets covering 70 diverse languages. This evaluation compares the performance of generative LLMs like Chat-GPT and GPT-4. Similarly, with a specific focus on the Indic languages, in IndicTrans2<cit.>, the authors provide a transformer-based multilingual NMT model that supports high-quality translations across all the 22 scheduled Indic languages. 


In the domain of text-to-image generative models, the recent Coco-crola benchmark <cit.> focuses on the evaluation of DallE and Stable Diffusion models for seven languages, namely English, Spanish, German, Chinese, Japanese, Hebrew, and Indonesian. Among these languages, English, German, and Indonesian share the Latin script, whereas Japanese and Chinese use the same script. Further, the authors focus on the generation of simple concepts through translation; however, in everyday applications, users are more likely to generate images through complex prompts. With this in mind, we focus on more complex prompts describing multiple elements in the generated image. The authors also propose the distinctiveness, coverage, and self-consistency measures.  Very limited research has also been conducted on the design of multilingual text-to-image generative models <cit.>. One such model is the  AltDiffusion model, which is a multilingual TTI diffusion model designed to process multiple languages <cit.>. Recently, Struppek et al. <cit.> have shown the influence of script-specific characters on the cultural aspects of the generated images.


In this work, we study the correctness and cultural representation bias of the TTI models, with a focus on Indic languages. However, the metrics and analysis in this study can be organically extended for the analysis of more languages.







§ BENCHMARK DESIGN

In this work, we propose the IndicTTI benchmark, which comprises of 31 languages, evaluated with four text-to-image generative models with six evaluation metrics covering aspects of correctness and representation. We describe the various design components of the benchmark below.



 §.§ Indic Languages and Prompts


For this benchmark, we utilize 30 Indic languages in addition to the English language. The languages are listed in Table <ref>. For prompts, we utilize the COCO-NLLB dataset <cit.>, which contains image-caption pairs for over 500K images, along with captions translated into 200 languages. In previous work<cit.>, the authors use simplistic concepts for prompts such as “a photo of a plane." However, the expectation from these models is to yield accurate and unbiased results in day-to-day usage. We believe that this expectation is better met with more complex prompts such as those present in the COCO-NLLB dataset. Therefore, we sample 1000 diverse image-caption pairs from the dataset. The dataset contains 24 Indic languages for which the captions are directly taken. These 24 languages do not include 4 languages that form a part of the 22 scheduled languages of India as per the Indian Constitution, namely Bodo, Dogri,  Konkani, and Santali. Using the IndicTrans2 translator <cit.>, we translate English captions to these languages along with two other languages, Manipuri (in Meitei script) and Sindhi (in Devanagari script). This leads to a total of 1000 prompts in 31 languages, including English, along with an associated image. 



 §.§ TTI Models and Generated Images

We utilize four different text-to-image models for the benchmark. For open-source models, we use the Stable Diffusion <cit.> and AltDiffusion Models<cit.>. For API-based models, we use Dalle3<cit.> and Midjourney<cit.>.  

Using the open-source models, we generate 4 images per prompt for the selected 1000 prompts across 31 languages, leading to 124K images corresponding to Stable Diffusion and AltDiffusion each. For API-based models, we generate over a subset of 200 prompts for Midjourney and over 50 prompts for Dalle3. This leads to a total of 24.8K images for Midjourney and 6.2K images for Dalle3. The evaluation is done over these sets and additionally over a subset of 40 prompts common across all models. The Dalle3 and Midjourney APIs are paid, and further details about the generation process are provided in the supplementary.



§ EVALUATION METHODS

In the IndicTTI benchmark, we focus on two aspects of TTI model evaluation- correctness and representation. Correctness refers to the ability to measure the semantic faithfulness with which the model generates the images for the corresponding prompt. On the other hand, representation refers to the measurement of the diversity of generated images within and across the different languages for the given prompts. The complete pipeline of the study is depicted in Fig. <ref>.



We adopt the following notation to describe the metrics and the evaluation pipeline. Let the set of languages be L. The set of prompts is denoted by P = {p_1, p_2,.. p_k}, and the corresponding representative images are Q = {q_1, q_2,.. q_k}. For each prompt in P, we generate n images, resulting in a total of n|P||L| images for a given generative model g. The set of images generated by g for a prompt p_k in language l ∈ L is denoted as,

    I_p_k, l = {i_p_k, l, r}_r=0^n

where, i_p_k, l, r denotes the image generated using g as g(p_k, l). Utilizing this notation, we define the metrics in this section. Across all the metrics, we compute high-level semantic features of images (using feature extractor f) and text (using feature extractor t) for effective comparison using the similarity function ϕ. 



 §.§ Correctness-based Metrics

We employ three correctness-based metrics in our study, including the proposed Cyclic Language-Grounded Correctness metric. The image-grounded and language-grounded correctness metrics are inspired by the correctness metric in CocoCrola<cit.>. These three metrics comprehensively evaluate the generated images across text-text, image-image, and image-text spaces. A higher correctness value indicates greater accuracy of the generative model in producing images that are faithful to the text.

Cyclic Language-Grounded Correctness (CLGC): This metric evaluates the generated images I in the text-text space. This is achieved in a cyclic manner through generating captions over I using a caption generator c, as,

    p'_k = c(i_p_k, l, r)

where, p'_k denotes the generated caption for r_th image in set I_p_k,l. Then, with p_k and p'_k as prompts in the English language for the language l, we obtain,

    CLGC(l) = 1/n|P|∑_p_k^P∑_r=0^nϕ [  t(p_k),  t(p'_k)  ]



Image-Grounded Correctness (IGC): For this metric, the correctness of the generated images is evaluated in the image-image space, as,

    IGC(l) = 1/n|P|∑_p_k^P∑_r=0^nϕ [  f(q_k),  f(i_p_k)  ]

where, the ground-truth image q_k is compared to each of the n generated images for the prompt p_k.

Language-Grounded Correctness (LGC): This metric evaluates correctness in the image-text space with p_k denoting the prompt in the English language,

    LGC(l) = 1/n|P|∑_p_k^P∑_r=0^nϕ [  t(p_k),  f(i_p_k)  ]





 §.§ Representation-based Metrics

In this section, we discuss the three metrics utilized for measuring the representativeness of concepts across the languages. We propose the Self-Consistency Across Languages metric to capture the bias in representation across the different languages. The other two metrics are inspired by the self-consistency and distinctiveness metrics in previous work<cit.>.

Self-Consistency Across Languages (SCAL): This metric captures the variation in the generated images for the same prompt across the different languages. So, for a language pair (l_a, l_b),

    SCAL(l_a, l_b, p_k)    = 1/n2∑_u=0^n∑_v=0^nϕ [  f(i_p_k, l_a, u),  f(i_p_k, l_b, v)  ]; u  v 
    
        SCAL(l_a, l_b)    = 1/|P|∑_p_k^P SCAL(l_a, l_b, p_k)

Then, the overall SCAL score for the given set of languages becomes,

    SCAL = 1/|L|2∑_l_a^L∑_l_b^L SCAL(l_a, l_b);  a  b

The lower the value of SCAL, the less consistent the generations are across the different languages. On the other hand, a higher value of SCAL would demonstrate high consistency between concepts across the different languages.

Self-Consistency Within Language (SCWL): This metric is computed within the images generated for a particular prompt for a given language, as,

    SCWL(l, p_k)    = 1/n2∑_u=0^n∑_v=0^nϕ [  f(i_p_k, u),  f(i_p_k, v)  ]; u  v 
    
        SCWL(l)    = 1/|P|∑_p_k^P SCWL(l, p_k)

A high self-consistency showcases good consistency between the semantic content of the generated images.

Distinctiveness Within Language (DWL): This metric depicts the diversity across the images generated for the different prompts p_a and p_b,

    SWL(l)    = 1 - 1/|P|2∑_p_a^P∑_p_b^Pϕ [  f(i_p_a),  f(i_p_b)  ];  a  b


where a higher distinctiveness score showcases the capability of the model to generate diverse images with varying prompts.

In addition to quantitative evaluation, we conduct an extensive qualitative evaluation of the generated images with respect to the captions, languages, and scripts. The implementation details are provided in the supplementary.




Cyclic Language Grounded Correctness (CLGC) (%) across the different Indic languages in the IndicTTI benchmark. Existing models provide high correctness for English languages while providing lower values for Indic languages.  


Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     64.53     7.29                                                      6.39                                                     7.24                                                     7.35                                                     7.13                                                     6.91                                                     7.53                                                     6.50                                                     8.03                                                     8.36                                                     8.78                                                      8.14                                                     11.29                                                    7.30                                                     7.65                                                 

Alt Diffusion        58.16     14.65                                                     16.70                                                    15.97                                                    16.79                                                    22.98                                                    16.08                                                    16.50                                                    12.17                                                    17.97                                                    18.82                                                    17.64                                                     17.99                                                    14.82                                                    18.10                                                    14.50                                                

Midjourney           64.31     12.99                                                     11.89                                                    12.49                                                    11.31                                                    12.12                                                    10.92                                                    11.99                                                    9.80                                                     11.89                                                    12.56                                                    9.32                                                      12.19                                                    13.49                                                    12.84                                                    12.16                                                

Dalle3               65.70     53.36                                                     60.73                                                    57.25                                                    64.16                                                    60.13                                                    57.68                                                    61.84                                                    35.24                                                    59.90                                                    53.96                                                    62.67                                                     51.22                                                    45.19                                                    53.98                                                    55.26                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     64.53     10.08                                                     6.29                                                     7.52                                                     7.13                                                     10.11                                                    7.12                                                     8.25                                                     7.66                                                     6.25                                                     6.26                                                     7.95                                                      7.78                                                     10.51                                                    7.59                                                     10.17                                                

Alt Diffusion        58.16     17.68                                                     15.96                                                    15.46                                                    16.93                                                    16.32                                                    15.41                                                    16.74                                                    16.80                                                    13.04                                                    11.98                                                    13.50                                                     15.82                                                    14.13                                                    13.37                                                    15.25                                                

Midjourney           64.31     15.38                                                     12.72                                                    12.37                                                    11.92                                                    13.04                                                    11.47                                                    12.06                                                    12.53                                                    11.96                                                    12.34                                                    11.77                                                     12.59                                                    11.36                                                    12.35                                                    9.72                                                 

Dalle3               65.70     64.02                                                     47.45                                                    55.51                                                    58.40                                                    57.37                                                    60.75                                                    59.18                                                    60.42                                                    49.03                                                    28.87                                                    59.84                                                     50.18                                                    10.93                                                    56.78                                                    11.19                                                
 




Image-Grounded Correctness (IGC) (%) across the different Indic languages in the IndicTTI benchmark. 

Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     51.21     24.19                                                     23.84                                                    23.25                                                    23.53                                                    23.24                                                    22.96                                                    23.53                                                    23.58                                                    23.96                                                    23.76                                                    22.97                                                     23.11                                                    23.06                                                    22.68                                                    22.61                                                

Alt Diffusion        46.58     24.23                                                     25.08                                                    25.18                                                    25.43                                                    27.74                                                    24.27                                                    25.42                                                    22.79                                                    25.98                                                    26.27                                                    25.48                                                     25.68                                                    24.68                                                    25.06                                                    22.19                                                

Midjourney           48.94     22.69                                                     22.66                                                    23.06                                                    22.50                                                    22.81                                                    22.04                                                    22.75                                                    21.97                                                    22.29                                                    22.82                                                    23.05                                                     22.81                                                    23.34                                                    22.55                                                    22.09                                                

Dalle3               48.92     40.02                                                     44.22                                                    43.75                                                    44.86                                                    42.74                                                    43.12                                                    46.10                                                    30.91                                                    43.14                                                    40.00                                                    45.06                                                     37.66                                                    36.30                                                    41.25                                                    41.39                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     51.21     22.92                                                     22.58                                                    23.41                                                    23.71                                                    25.16                                                    22.98                                                    23.19                                                    23.41                                                    24.00                                                    23.83                                                    22.90                                                     23.88                                                    24.14                                                    23.64                                                    23.98                                                

Alt Diffusion        46.58     25.40                                                     24.45                                                    25.01                                                    26.39                                                    25.97                                                    25.53                                                    25.59                                                    26.10                                                    24.60                                                    23.82                                                    24.87                                                     24.73                                                    24.16                                                    25.16                                                    24.35                                                

Midjourney           48.94     23.30                                                     22.76                                                    22.63                                                    22.90                                                    23.09                                                    22.65                                                    22.97                                                    22.64                                                    22.16                                                    22.57                                                    22.90                                                     22.79                                                    21.25                                                    22.53                                                    21.99                                                

Dalle3               48.92     44.10                                                     38.81                                                    44.04                                                    43.18                                                    44.88                                                    44.22                                                    44.03                                                    44.17                                                    37.46                                                    29.76                                                    44.69                                                     38.91                                                    23.18                                                    42.18                                                    22.46                                                
 




Language-Grounded Correctness (LGC) (%) across the different Indic languages in the IndicTTI benchmark.  

Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     33.98     3.90                                                      3.26                                                     3.28                                                     3.79                                                     4.21                                                     4.09                                                     3.61                                                     3.50                                                     3.68                                                     3.56                                                     3.01                                                      3.77                                                     4.16                                                     4.23                                                     4.09                                                 

Alt Diffusion        30.90     6.50                                                      6.87                                                     7.34                                                     7.56                                                     10.67                                                    6.53                                                     7.81                                                     4.54                                                     8.25                                                     8.57                                                     7.87                                                      8.49                                                     6.53                                                     7.12                                                     6.00                                                 

Midjourney           32.25     2.08                                                      1.95                                                     2.86                                                     3.27                                                     4.03                                                     2.44                                                     2.87                                                     1.73                                                     3.36                                                     3.00                                                     2.59                                                      3.41                                                     3.91                                                     3.92                                                     2.76                                                 

Dalle3               33.06     26.42                                                     30.54                                                    28.04                                                    31.57                                                    28.48                                                    28.22                                                    30.87                                                    14.98                                                    29.17                                                    26.67                                                    31.41                                                     24.32                                                    19.92                                                    26.24                                                    26.68                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     33.98     5.85                                                      3.80                                                     4.12                                                     4.26                                                     5.83                                                     4.10                                                     4.25                                                     4.45                                                     3.50                                                     3.90                                                     3.55                                                      4.11                                                     3.71                                                     4.11                                                     4.52                                                 

Alt Diffusion        30.90     7.72                                                      5.41                                                     7.22                                                     8.22                                                     8.04                                                     7.82                                                     7.70                                                     7.81                                                     5.99                                                     4.73                                                     5.98                                                      6.12                                                     5.76                                                     6.82                                                     6.30                                                 

Midjourney           32.25     3.99                                                      3.19                                                     3.33                                                     3.66                                                     4.30                                                     2.95                                                     4.35                                                     3.31                                                     1.78                                                     3.68                                                     3.56                                                      2.86                                                     2.45                                                     3.32                                                     2.64                                                 

Dalle3               33.06     30.59                                                     23.09                                                    28.30                                                    28.68                                                    28.91                                                    29.99                                                    30.02                                                    29.39                                                    22.86                                                    12.93                                                    29.10                                                     24.05                                                    2.09                                                     27.70                                                    1.52                                                 
 









§ BENCHMARK RESULTS AND ANALYSIS

In this section, we report results on the 4 models used for evaluation, namely Stable Diffusion, AltDiffusion, Midjourney, and Dalle3, on a common set of prompts.  We provide the results on the complete set of prompts for the models in the supplementary. The observations are consistent with those made on the common subset reported in this section.



Correctness Bias: As discussed in the previous section, we evaluate the correctness of generation across the Indic languages through the metrics of CLGC, IGC, and LGC, which correspond to the text-text, image-image, and image-text spaces, respectively. In the text-text space, we observe that the CLGC metric is significantly high for the English language (Refer Table <ref>).  This demonstrates that the generated images have high faithfulness to the text, as the captions generated for the generated images are semantically closest to the original English caption. All models have a high CLGC value of >60% for the English language. On the other hand, CLGC values drop significantly for Indic languages for Stable Diffusion and AltDiffusion models, with the performance dropping to 7.29% from 64.53% and 14.65% from 58.16%, respectively for the Assamese language. The values of CLGC tend to be lower for languages with a Bengali script, namely Manipuri (Beng), Assamese, and Bengali. On computing the average across the 30 Indic languages and comparing the CLGC scores with those obtained for the English language, there is a 56.64% drop in performance for the Stable Diffusion model. Similarly, there is a 42.16%, 52.16%, and 13.62% drop for AltDiffusion, Midjourney, and Dalle3, with Dalle3 providing the best performance.

Similarly, in the image-image space, we observe higher IGC metric values for the English language, showcasing up to 50% image semantic similarity with the ground truth images. With Indic languages, there is an average drop of 27.74% and 21.53% for the Stable Diffusion and AltDiffusion models. While there is a high drop of 26.32% for Midjourney generated images, the drop is significantly lesser for Dalle3 at 8.70%, signifying a superior understanding of Indic languages. For image-text similarity, we use the English language as the prompt of comparison. We observe higher LGC metric values for the images generated using English prompts, in the range of 30-35% across the models. These values drop sharply for Stable Diffusion, AltDiffusion, and Midjourney, with average LGC values of 4.01%, 7.08%, and 3.12% across the Indic languages, respectively. Dalle3 has the highest LGC value at 25.09%, with extremely low values observed for the Manipuri (Meitei script) (1.52%), Santali (2.09%), and Bodo (12.93%) languages.

Across all the metrics, we observe that Dalle3 outperforms all other models when evaluated for Indic languages with a significant margin. However, Dalle3 struggles with the Manipuri (Meitei script), Santali, and Bodo languages. Other models struggle to generalize on the Indic languages with AltDiffusion performing the best among the other three models. This behavior could be a result of increased generalizability due to the multilingual training of the model. On the other hand, while all models perform similarly for the English language, AltDiffusion performs the worst, possibly due to catastrophic forgetting when trained for multilinguality.  






Representativeness Bias: To better understand the representative capacity of the models, we evaluate the SCAL, SCWL, and DWL metrics as explained in the previous section. While these metrics do not contain the capability to estimate the correctness of generation on their own, their values showcase important behavior in terms of diversity in model generation. 

The SCAL metric evaluates the self-consistency of generated images across the different prompts. A high SCAL value represents the model's ability to consistently generate similar semantic content across languages. In Fig. <ref>, we plot the SCAL values across the different language pairs for the four models. From Fig. <ref>(a), we observe that there is little semantic similarity of concepts between the English language and other languages. Since we know that these models are primarily trained to perform well on English, this is also an indicator of poor performance across the other languages. Interestingly, the concepts generated across other languages have higher overlap showing that they generate images that are far similar in semantic content, despite being possibly incorrect in their generations. In the previous section, we observed that the performance of generation for Indic languages is only high in Dalle3. This is demonstrated well in Fig. <ref>(d) where we observe that the semantic similarity between English and Indic languages is high, with performance being poorest between Santali and Manipuri (Meitei script), followed by Bodo and Manipuri (Bengali script). 


The overall SCAL metric value as per Eqn. <ref> for the Stable Diffusion, AltDiffusion, Midjourney, and Dalle3 models is 25.44%, 23.73%, 26.75%, and 29.90%, respectively. This indicates an overall high consistency of Dalle3 in generating concepts across different languages. It is also evident from Fig. <ref> that the Stable Diffusion, AltDiffusion, and Midjourney generators suffer greatly in maintaining semantic consistency with images generated via English, whereas they tend to generate similar content for other languages. From Fig. <ref>(c), we also observe that Midjourney has high semantic consistency between images generated from languages with the Devanagari script, highlighting the language script plays a crucial part in the semantic content of the image.



Next, we report the representativeness of different prompts within the same language through the SCWL metric. In Fig. <ref>(a), we observe that the self-consistency of models is higher for the English language. A higher value demonstrates a more stable representation of concepts in the model. For Dalle3, the SCWL values for the different languages are comparable to those of English but still lower. In agreement with previous observations, the values are especially poor for the Santali, Manipuri (Meitei script), Bodo, and Manipuri (Bengali script) languages. In Midjourney, the values stay consistent, with the exception of the Santali and Manipuri (Meitei) languages. Similar trends are observed for the other models. When compared with other metrics, there is a low variation in the values of self-consistency within the language for the different models and languages. This behavior could be the result of the relative stability of the model in its generation. While the model may generate incorrectly, it associates a given text for generation in the latent space consistently. Therefore, the model may demonstrate high self-consistency by generating images that are related to a given prompt, albeit incorrectly. 

Finally, in Fig. <ref>(b), results are presented for the DWL metric. The DWL metric showcases how distinct are the generations corresponding to the different prompts. This metric is dependent on the relatedness of the prompts used for generation. While sampling prompts for the IndicTTI benchmark, we ensured semantic diversity in the selected 1000 prompts. The details of the selection process are provided in the supplementary. It is evident from Fig. <ref>(b) that Dalle3 provides the highest level of distinctiveness between the generated prompts across the different languages, only suffering for the Santali and Manipuri (Meitei script) languages. Surprisingly, Midjourney has the lowest distinctiveness in its generations across the Indic languages, indicating it generates similar content over the different prompts. Similarly, the Stable Diffusion model performs poorly. The AltDiffusion model provides the second highest distinctiveness among the four models. This could be attributed to its multilingual training, making it sensitive to different scripts during generation.    








§ QUALITATIVE ANALYSIS

In this section, we qualitatively analyze the images generated by the different models across the different languages. Based on our observations in the previous section, we explore the generated images and study the alignment between the quantified metrics and generated images.

As in the previous section, we study two aspects of bias, correctness-based as well as representation-based. In Fig. <ref>, we observe that the Stable Diffusion and AltDiffusion models suffer greatly when prompted with Indic languages. The images generated using the prompts in Indic languages often do not resemble the description of the actual prompts. For example, when prompted to generate a swan in the Punjabi language, the Stable Diffusion model generates an image of 3 men in turbans, whereas the AltDiffusion model generates the image of a couple. Here again, the man is observed to be wearing a turban. We further observe a relatively high depiction of people in images, specifically those of Indian ethnicity, even when the actual content of the prompt does not involve any mention of people. Similarly, the lack of correspondence between the actual prompt and the generated images for Midjourney can be seen in Fig. <ref>. For example, on prompting Midjourney to generate images of “a man in a brown jacket," it generates stereotypically Indian women. Other simplistic prompts in the Hindi language also lead to incorrect generations. 



For Dalle3, we observe that the model generates correct images, relevant to the content of the prompt. This observation is corroborated by the high performance of Dalle3 on the metrics proposed with the benchmark. However, the Dalle3 model struggles with certain languages, such as Manipuri, Santali, and Bodo. In order to better understand the poor quality of generations, we prompted the ChatGPT model, which uses Dalle3 for image generation, to generate some images for these languages. On providing the ChatGPT model with prompts in these languages, it seemed that the model did not understand these scripts at all. Specifically, it identified the script/language it was being prompted with and then continued to “imagine" an image based on cultural influences. Some of these examples are shown in Fig. <ref> where the model generates images influenced by the Manipuri, Nepali, and Santal cultures.

On observing the images for the different models (Refer Fig. <ref>), we notice a correlation between different language scripts and the images generated corresponding to them. All models, except Dalle3, frequently generate images of females and Indian religious figures such as Gods and pandits. Dalle3, on the other hand, tends to incorporate common Indian religious sites like temples and ghats. In Hindu culture, many women regularly wear sarees and bindi, and these elements are prominently featured in the images produced by both Stable Diffusion and AltDiffusion. 
Additionally, the Arabic script is frequently associated with elements of Muslim culture, such as the burqa and niqab. Elements of South Indian culture are often present in the images generated for prompts in Malayalam, a language widely spoken in Southern India. These correlations between script and generated images are common, especially in those produced by Midjourney. When prompted in Sanskrit, the sacred language of Hinduism and predominant in religious texts, the Stable Diffusion model is observed to generate images of Indian Gods. Similarly, prompts in Punjabi often result in images of men wearing turbans, reflecting the cultural and religious practices of Punjabi speakers. More observations are detailed in the supplementary materials.






§ CONCLUSION

In this research, we introduce the IndicTTI benchmark, a comprehensive framework that integrates a variety of prompts, several text-to-image (TTI) models, and an extensive assessment of generative biases using the proposed evaluation methods. Employing both performance-oriented and representativeness-centric metrics, we are able to thoroughly assess the capacity of the models for multilingual generation. The qualitative analysis further emphasizes the extent of cultural influences embedded within the models' outputs. While the presence of cultural elements is not inherently detrimental, some generated images may inadvertently reinforce cultural stereotypes. The IndicTTI benchmark makes a dual contribution: firstly, it establishes a robust quantitative standard for assessing the multilingual capabilities of TTI models; secondly, it highlights the incorporation of Indian cultural motifs in the imagery produced.

Limitations:
In this paper, we conduct both quantitative and qualitative evaluations of various aspects of TTI systems in a multilingual context. The prompts used in our benchmark are primarily derived from COCO-NLLB, which translates English captions from the COCO dataset into multiple languages, or translated from English using the open-source IndicTrans2 model. Therefore, the generation performance for Indic prompts is indirectly affected by the quality of these translations. A user study, detailed in the supplementary materials, briefly examines the translation quality and demonstrates the appropriateness and accuracy of the translated captions. While the high performance of the Dalle3 model suggests that the translations are meaningful, it is essential to recognize the influence of translation in the benchmarking process. Furthermore, the semantic similarity between images and text is evaluated using large pre-trained models, which may possess inherent biases that could inadvertently impact the evaluation. Our comprehensive qualitative assessment generally supports the quantitative results derived from these pre-trained models. Nonetheless, there remains a possibility that inadvertent biases could affect the outcomes.



§ ACKNOWLEDGEMENTS

S. Mittal is partially supported by the UGC-Net JRF Fellowship and IBM fellowship. M. Vatsa was partially supported through the Swarnajayanti Fellowship. This work was partially supported by Facebook AI. All data was stored, and experiments were performed on IITJ servers by IITJ faculty and students.


splncs04









§ BENCHMARK DESIGN

In this section, we provide additional details about the benchmark design.



 §.§ Indic Languages and Prompts


In this research, we introduce the IndicTTI benchmark where we study the performance of popular text-to-image (TTI) models in 30 languages. While the benchmark may be extended for any number of languages, we select 30 Indic languages, which are written in 10 different scripts and have roots in multiple language families. Detailed information about the different languages is provided in Table <ref>.



For prompts, we utilize the COCO-NLLB dataset <cit.>, which contains image-caption pairs for over 500K images, along with captions translated into 200 languages. We sample 1000 diverse image-caption pairs from the dataset. In order to avoid prompts with proper nouns, such as names of celebrities and/or brand names, we filtered the dataset to remove any captions that contained capitalized words in the middle of the sentence. From the filtered dataset, we randomly subsample 1000 captions. Next, for a diverse selection, we computed sentence-level embeddings for the 1000 prompts using a SentenceFormer model and calculated the average similarity between any two prompts. This experiment was repeated for 1000 iterations, and the subset with the lowest sentence similarity between prompts was selected. The subsets of 200 and 50 prompts for Midjourney and Dalle3, respectively, were chosen randomly from the selected subset of 1000 prompts. 



 §.§ TTI Models and Generated Images

We utilize four different text-to-image models for the benchmark. For open-source models, we use the Stable Diffusion and AltDiffusion Models (without safety filters). In the main paper, we report results on the m2 variant of AltDiffusion trained on the English and Chinese languages. Extended results, including the m9 variant trained on the English, Chinese, Spanish, French, Russian, Japanese, Korean, Arabic, and Italian languages, are reported in the supplementary. The extended results exhibit a similar pattern to the m2 variant. Detailed results are provided in the subsequent sections. Unless specified otherwise, the AltDiffusion model refers to the m2 variant in this work. For API-based models, we use Dalle3<cit.> and Midjourney<cit.>, which are both paid. Stable Diffusion and AltDiffusion models generate images of size 512 x 512, whereas Midjourney and Dalle3 generate images of size 1024 x 1024.






 §.§ Quality of Translated Captions

Our prompts were primarily sourced from COCO-NLLB or translated from English using the IndicTrans2 model. As specified in the Limitations section of the main paper, this suggests that the translation quality may have influenced the performance of image generation with Indic prompts. To evaluate the translation quality, we conducted a user study on a common set of 40 prompts across 30 languages, validated by 65 annotators with an average proficiency of 4.78 out of 5 in at least one Indic language. We followed the XSTS protocol <cit.>, also used to assess the translation quality of the NLLB model <cit.>. On a scale from 1 (worst) to 5 (best), the average translation quality for all languages is reported in Fig. <ref>, indicating that the sentences are mostly equivalent or paraphrases of each other, according to the XSTS protocol, and thus suitable for image generation. In this experiment, the inter-rater agreement, measured through percent agreement, was 74.8%. For languages such as kas_Deva, snd_Deva, and sin_Sinh, the translation quality of certain prompts was observed to be less than 3 on the XSTS scale.



§ IMPLEMENTATION DETAILS

In this section, we discuss the implementation details involved in the creation of the IndicTTI benchmark as well as its evaluation.

For generation using open-source TTI systems of Stable Diffusion[<https://huggingface.co/runwayml/stable-diffusion-v1-5>] and AltDiffusion[<https://huggingface.co/BAAI/AltDiffusion>, <https://huggingface.co/BAAI/AltDiffusion-m9>], we utilized the models available at HuggingFace. Dalle3 API was accessed through a Python script whereas the generation for Midjourney was done using Discord. For Stable Diffusion and AltDiffusion, the generation was done using an NVIDIA DGX Station consisting of 4 NVIDIA V100 GPU with 32 GB VRAM each, using a batch size of 8. The generation was repeated using 4 seeds, providing 4 images for every prompt. The DDIM Scheduler was used for inferencing with 50 steps.

For evaluation, all experiments are conducted on LINUX-based systems using Python-based libraries, and specifically, the PyTorch library is used. To extract rich semantic text and image features for the evaluation metrics, we utilize various modules of the BLIP-2 model[<https://huggingface.co/Salesforce/blip2-opt-2.7b>]. We utilize the image encoder of the BLIP-2 model as the image feature extractor f for computing the CLGC, IGC, SCAL, SCWL, and DWL metrics. Additionally, for the CLGC metric, we utilize the image-captioning capabilities of BLIP-2 captioner c to generate captions for generated images, and for extracting rich textual features, the SentenceFormer model[<https://huggingface.co/sentence-transformers/all-mpnet-base-v2>]. For the LGC metric, we require image-text features that are extracted from BLIP-2 using the LAVIS[<https://github.com/salesforce/LAVIS/>] library. The similarity function ϕ is computed using the cosine similarity. The code for generation, as well as evaluation, can be 
accessed through <https://iab-rubric.org/resources/other-databases/indictti>.



Cyclic Language Grounded Correctness (CLGC) (%) across the different Indic languages in IndicTTI on the common set of prompts.  


Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     64.53     7.29                                                      6.39                                                     7.24                                                     7.35                                                     7.13                                                     6.91                                                     7.53                                                     6.50                                                     8.03                                                     8.36                                                     8.78                                                      8.14                                                     11.29                                                    7.30                                                     7.65                                                 

Alt Diffusion        58.16     14.65                                                     16.70                                                    15.97                                                    16.79                                                    22.98                                                    16.08                                                    16.50                                                    12.17                                                    17.97                                                    18.82                                                    17.64                                                     17.99                                                    14.82                                                    18.10                                                    14.50                                                

Alt Diffusion m9     53.19     18.02                                                     27.08                                                    26.50                                                    29.78                                                    26.66                                                    27.04                                                    28.21                                                    12.39                                                    31.42                                                    27.70                                                    27.14                                                     21.50                                                    18.97                                                    27.81                                                    24.34                                                

Midjourney           64.31     12.99                                                     11.89                                                    12.49                                                    11.31                                                    12.12                                                    10.92                                                    11.99                                                    9.80                                                     11.89                                                    12.56                                                    9.32                                                      12.19                                                    13.49                                                    12.84                                                    12.16                                                

Dalle3               65.70     53.36                                                     60.73                                                    57.25                                                    64.16                                                    60.13                                                    57.68                                                    61.84                                                    35.24                                                    59.90                                                    53.96                                                    62.67                                                     51.22                                                    45.19                                                    53.98                                                    55.26                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     64.53     10.08                                                     6.29                                                     7.52                                                     7.13                                                     10.11                                                    7.12                                                     8.25                                                     7.66                                                     6.25                                                     6.26                                                     7.95                                                      7.78                                                     10.51                                                    7.59                                                     10.17                                                

Alt Diffusion        58.16     17.68                                                     15.96                                                    15.46                                                    16.93                                                    16.32                                                    15.41                                                    16.74                                                    16.80                                                    13.04                                                    11.98                                                    13.50                                                     15.82                                                    14.13                                                    13.37                                                    15.25                                                

Alt Diffusion m9     53.19     26.29                                                     17.33                                                    26.03                                                    23.69                                                    28.06                                                    24.48                                                    29.98                                                    28.99                                                    26.24                                                    12.29                                                    21.57                                                     20.89                                                    12.97                                                    23.71                                                    13.36                                                

Midjourney           64.31     15.38                                                     12.72                                                    12.37                                                    11.92                                                    13.04                                                    11.47                                                    12.06                                                    12.53                                                    11.96                                                    12.34                                                    11.77                                                     12.59                                                    11.36                                                    12.35                                                    9.72                                                 

Dalle3               65.70     64.02                                                     47.45                                                    55.51                                                    58.40                                                    57.37                                                    60.75                                                    59.18                                                    60.42                                                    49.03                                                    28.87                                                    59.84                                                     50.18                                                    10.93                                                    56.78                                                    11.19                                                
 


Image-Grounded Correctness (IGC) (%) across the different Indic languages in IndicTTI on the common set of prompts. 

Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     51.21     24.19                                                     23.84                                                    23.25                                                    23.53                                                    23.24                                                    22.96                                                    23.53                                                    23.58                                                    23.96                                                    23.76                                                    22.97                                                     23.11                                                    23.06                                                    22.68                                                    22.61                                                

Alt Diffusion        46.58     24.23                                                     25.08                                                    25.18                                                    25.43                                                    27.74                                                    24.27                                                    25.42                                                    22.79                                                    25.98                                                    26.27                                                    25.48                                                     25.68                                                    24.68                                                    25.06                                                    22.19                                                

Alt Diffusion m9     42.43     26.50                                                     28.39                                                    30.46                                                    28.94                                                    29.74                                                    29.59                                                    28.86                                                    23.29                                                    30.67                                                    29.55                                                    28.76                                                     26.54                                                    25.54                                                    28.29                                                    28.62                                                

Midjourney           48.94     22.69                                                     22.66                                                    23.06                                                    22.50                                                    22.81                                                    22.04                                                    22.75                                                    21.97                                                    22.29                                                    22.82                                                    23.05                                                     22.81                                                    23.34                                                    22.55                                                    22.09                                                

Dalle3               48.92     40.02                                                     44.22                                                    43.75                                                    44.86                                                    42.74                                                    43.12                                                    46.10                                                    30.91                                                    43.14                                                    40.00                                                    45.06                                                     37.66                                                    36.30                                                    41.25                                                    41.39                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     51.21     22.92                                                     22.58                                                    23.41                                                    23.71                                                    25.16                                                    22.98                                                    23.19                                                    23.41                                                    24.00                                                    23.83                                                    22.90                                                     23.88                                                    24.14                                                    23.64                                                    23.98                                                

Alt Diffusion        46.58     25.40                                                     24.45                                                    25.01                                                    26.39                                                    25.97                                                    25.53                                                    25.59                                                    26.10                                                    24.60                                                    23.82                                                    24.87                                                     24.73                                                    24.16                                                    25.16                                                    24.35                                                

Alt Diffusion m9     42.43     28.39                                                     25.05                                                    27.48                                                    27.34                                                    29.31                                                    28.24                                                    30.17                                                    29.53                                                    29.79                                                    23.28                                                    27.16                                                     25.83                                                    24.10                                                    28.24                                                    24.26                                                

Midjourney           48.94     23.30                                                     22.76                                                    22.63                                                    22.90                                                    23.09                                                    22.65                                                    22.97                                                    22.64                                                    22.16                                                    22.57                                                    22.90                                                     22.79                                                    21.25                                                    22.53                                                    21.99                                                

Dalle3               48.92     44.10                                                     38.81                                                    44.04                                                    43.18                                                    44.88                                                    44.22                                                    44.03                                                    44.17                                                    37.46                                                    29.76                                                    44.69                                                     38.91                                                    23.18                                                    42.18                                                    22.46                                                
 


Language-Grounded Correctness (LGC) (%) across the different Indic languages in IndicTTI on the common set of prompts.  

Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     33.98     3.90                                                      3.26                                                     3.28                                                     3.79                                                     4.21                                                     4.09                                                     3.61                                                     3.50                                                     3.68                                                     3.56                                                     3.01                                                      3.77                                                     4.16                                                     4.23                                                     4.09                                                 

Alt Diffusion        30.90     6.50                                                      6.87                                                     7.34                                                     7.56                                                     10.67                                                    6.53                                                     7.81                                                     4.54                                                     8.25                                                     8.57                                                     7.87                                                      8.49                                                     6.53                                                     7.12                                                     6.00                                                 

Alt Diffusion m9     27.56     7.28                                                      10.92                                                    12.26                                                    12.58                                                    12.36                                                    11.58                                                    11.37                                                    3.66                                                     14.01                                                    11.73                                                    10.20                                                     9.93                                                     7.61                                                     11.39                                                    11.30                                                

Midjourney           32.25     2.08                                                      1.95                                                     2.86                                                     3.27                                                     4.03                                                     2.44                                                     2.87                                                     1.73                                                     3.36                                                     3.00                                                     2.59                                                      3.41                                                     3.91                                                     3.92                                                     2.76                                                 

Dalle3               33.06     26.42                                                     30.54                                                    28.04                                                    31.57                                                    28.48                                                    28.22                                                    30.87                                                    14.98                                                    29.17                                                    26.67                                                    31.41                                                     24.32                                                    19.92                                                    26.24                                                    26.68                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     33.98     5.85                                                      3.80                                                     4.12                                                     4.26                                                     5.83                                                     4.10                                                     4.25                                                     4.45                                                     3.50                                                     3.90                                                     3.55                                                      4.11                                                     3.71                                                     4.11                                                     4.52                                                 

Alt Diffusion        30.90     7.72                                                      5.41                                                     7.22                                                     8.22                                                     8.04                                                     7.82                                                     7.70                                                     7.81                                                     5.99                                                     4.73                                                     5.98                                                      6.12                                                     5.76                                                     6.82                                                     6.30                                                 

Alt Diffusion m9     27.56     10.38                                                     6.25                                                     10.74                                                    10.44                                                    12.38                                                    10.67                                                    12.58                                                    12.56                                                    11.04                                                    4.71                                                     9.69                                                      8.20                                                     5.16                                                     9.91                                                     5.65                                                 

Midjourney           32.25     3.99                                                      3.19                                                     3.33                                                     3.66                                                     4.30                                                     2.95                                                     4.35                                                     3.31                                                     1.78                                                     3.68                                                     3.56                                                      2.86                                                     2.45                                                     3.32                                                     2.64                                                 

Dalle3               33.06     30.59                                                     23.09                                                    28.30                                                    28.68                                                    28.91                                                    29.99                                                    30.02                                                    29.39                                                    22.86                                                    12.93                                                    29.10                                                     24.05                                                    2.09                                                     27.70                                                    1.52                                                 
 





Cyclic Language Grounded Correctness (CLGC) (%) across the different Indic languages in the IndicTTI benchmark on the complete set of prompts. Existing models provide high correctness for English languages while providing lower values for Indic languages. 


Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     67.97     7.56                                                      8.22                                                     8.08                                                     8.64                                                     8.16                                                     7.31                                                     8.15                                                     7.93                                                     9.09                                                     8.64                                                     8.60                                                      8.84                                                     12.22                                                    8.63                                                     9.36                                                 

Alt Diffusion        62.40     14.32                                                     16.87                                                    15.42                                                    17.34                                                    22.33                                                    15.67                                                    16.36                                                    12.07                                                    17.27                                                    18.78                                                    16.35                                                     16.54                                                    17.13                                                    17.69                                                    15.51                                                

Alt Diffusion m9     55.99     16.87                                                     27.08                                                    25.45                                                    30.48                                                    24.59                                                    28.24                                                    27.74                                                    13.21                                                    29.42                                                    24.70                                                    25.69                                                     21.00                                                    21.53                                                    26.53                                                    25.59                                                

Midjourney           69.41     12.59                                                     12.33                                                    13.90                                                    12.66                                                    12.71                                                    11.21                                                    13.83                                                    10.66                                                    12.79                                                    13.44                                                    9.43                                                      12.87                                                    14.68                                                    12.60                                                    12.87                                                

Dalle3               66.42     53.75                                                     62.43                                                    58.62                                                    65.88                                                    60.38                                                    59.34                                                    62.40                                                    33.95                                                    62.09                                                    54.73                                                    63.69                                                     52.88                                                    42.30                                                    55.87                                                    56.14                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     67.97     9.61                                                      7.80                                                     8.18                                                     8.19                                                     9.06                                                     8.04                                                     8.61                                                     8.54                                                     7.51                                                     7.66                                                     8.36                                                      8.25                                                     9.40                                                     7.64                                                     11.34                                                

Alt Diffusion        62.40     19.40                                                     16.30                                                    15.19                                                    16.02                                                    15.57                                                    15.30                                                    16.19                                                    16.49                                                    13.65                                                    12.01                                                    13.89                                                     14.46                                                    13.70                                                    14.08                                                    13.48                                                

Alt Diffusion m9     55.99     26.02                                                     19.62                                                    24.20                                                    26.24                                                    26.82                                                    25.70                                                    28.29                                                    28.33                                                    25.71                                                    15.44                                                    21.82                                                     20.15                                                    13.53                                                    22.08                                                    13.49                                                

Midjourney           69.41     15.85                                                     13.90                                                    13.85                                                    13.09                                                    13.76                                                    12.81                                                    12.52                                                    13.32                                                    13.02                                                    12.87                                                    12.65                                                     13.97                                                    11.71                                                    13.28                                                    11.61                                                

Dalle3               66.42     64.35                                                     49.36                                                    55.28                                                    58.83                                                    56.88                                                    61.65                                                    59.37                                                    61.20                                                    50.13                                                    26.19                                                    60.28                                                     50.97                                                    10.75                                                    56.98                                                    11.43                                                
 








Image Grounded Correctness (IGC) (%) across the different Indic languages in IndicTTI on the complete set.  

Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     53.97     23.48                                                     23.57                                                    22.72                                                    22.93                                                    22.65                                                    22.49                                                    22.99                                                    23.40                                                    23.32                                                    23.25                                                    22.24                                                     22.63                                                    23.45                                                    22.37                                                    22.40                                                

Alt Diffusion        49.12     23.64                                                     24.30                                                    23.78                                                    25.12                                                    26.22                                                    23.45                                                    24.22                                                    22.43                                                    25.05                                                    25.54                                                    24.49                                                     24.41                                                    24.24                                                    24.35                                                    22.44                                                

Alt Diffusion m9     45.62     24.82                                                     28.39                                                    29.13                                                    29.85                                                    28.90                                                    30.43                                                    28.47                                                    23.49                                                    29.87                                                    27.94                                                    28.35                                                     25.72                                                    27.34                                                    28.45                                                    29.22                                                

Midjourney           52.72     22.54                                                     22.57                                                    22.63                                                    22.76                                                    22.03                                                    21.97                                                    23.10                                                    21.95                                                    22.79                                                    22.43                                                    22.63                                                     22.56                                                    24.10                                                    22.47                                                    22.12                                                

Dalle3               48.54     39.48                                                     45.13                                                    43.10                                                    45.86                                                    42.57                                                    43.72                                                    46.26                                                    30.42                                                    44.47                                                    40.34                                                    45.54                                                     38.23                                                    35.46                                                    41.71                                                    41.59                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     53.97     22.56                                                     22.23                                                    22.93                                                    23.06                                                    23.26                                                    22.83                                                    22.92                                                    22.83                                                    23.62                                                    23.01                                                    23.00                                                     22.91                                                    24.42                                                    22.93                                                    23.63                                                

Alt Diffusion        49.12     24.95                                                     23.69                                                    24.16                                                    24.64                                                    24.51                                                    24.45                                                    24.68                                                    24.93                                                    23.56                                                    23.53                                                    23.82                                                     23.78                                                    23.55                                                    24.13                                                    23.66                                                

Alt Diffusion m9     45.62     28.56                                                     25.87                                                    27.24                                                    28.05                                                    28.80                                                    28.25                                                    29.01                                                    29.06                                                    28.64                                                    23.28                                                    26.69                                                     25.31                                                    23.42                                                    26.73                                                    23.34                                                

Midjourney           52.72     24.05                                                     22.95                                                    23.04                                                    22.66                                                    23.25                                                    22.87                                                    22.76                                                    22.79                                                    22.60                                                    22.45                                                    22.66                                                     22.81                                                    21.08                                                    22.57                                                    22.39                                                

Dalle3               48.54     44.38                                                     38.97                                                    43.75                                                    43.64                                                    44.34                                                    44.87                                                    44.30                                                    44.88                                                    37.29                                                    28.82                                                    44.07                                                     38.84                                                    23.23                                                    43.55                                                    22.44                                                
 







Language Grounded Correctness (LGC) (%) across the different Indic languages in IndicTTI on the complete set of prompts. 

Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     35.00     4.05                                                      4.34                                                     3.85                                                     4.26                                                     4.04                                                     4.62                                                     3.95                                                     4.15                                                     4.18                                                     4.00                                                     3.37                                                      4.03                                                     4.58                                                     4.59                                                     4.57                                                 

Alt Diffusion        32.47     6.31                                                      7.22                                                     6.81                                                     8.22                                                     9.95                                                     6.70                                                     7.21                                                     4.44                                                     8.34                                                     8.56                                                     7.47                                                      7.39                                                     7.14                                                     7.73                                                     6.71                                                 

Alt Diffusion m9     29.22     6.44                                                      11.48                                                    11.86                                                    13.70                                                    11.23                                                    13.16                                                    11.79                                                    4.32                                                     13.30                                                    10.69                                                    10.92                                                     8.76                                                     9.19                                                     11.41                                                    11.67                                                

Midjourney           34.11     2.60                                                      2.73                                                     2.94                                                     3.43                                                     3.24                                                     2.23                                                     3.46                                                     2.19                                                     3.65                                                     2.77                                                     2.65                                                      3.47                                                     4.28                                                     3.42                                                     2.47                                                 

Dalle3               33.04     25.79                                                     30.96                                                    27.64                                                    31.59                                                    28.51                                                    28.44                                                    30.62                                                    14.22                                                    29.77                                                    26.54                                                    31.37                                                     24.83                                                    18.57                                                    26.45                                                    26.52                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     35.00     5.09                                                      4.61                                                     4.34                                                     4.34                                                     4.63                                                     4.22                                                     4.52                                                     4.41                                                     3.82                                                     3.90                                                     4.06                                                      4.17                                                     3.76                                                     4.16                                                     4.72                                                 

Alt Diffusion        32.47     8.17                                                      5.84                                                     6.69                                                     7.48                                                     7.44                                                     7.37                                                     7.59                                                     7.63                                                     6.16                                                     5.67                                                     6.63                                                      6.24                                                     5.84                                                     6.68                                                     5.71                                                 

Alt Diffusion m9     29.22     11.15                                                     7.80                                                     9.96                                                     11.18                                                    12.13                                                    11.24                                                    12.44                                                    12.52                                                    11.31                                                    4.97                                                     9.35                                                      7.59                                                     5.71                                                     9.28                                                     5.75                                                 

Midjourney           34.11     4.19                                                      3.15                                                     3.62                                                     3.29                                                     4.22                                                     3.30                                                     3.88                                                     3.43                                                     2.42                                                     3.32                                                     3.36                                                      3.35                                                     2.52                                                     3.47                                                     3.64                                                 

Dalle3               33.04     30.35                                                     23.55                                                    27.51                                                    28.71                                                    27.79                                                    30.15                                                    29.68                                                    29.73                                                    22.35                                                    11.46                                                    29.14                                                     23.96                                                    2.24                                                     27.91                                                    1.54                                                 
 










§ BENCHMARK RESULTS AND ANALYSIS

In this section, we report extended results on the common set and complete set of prompts.  The observations are consistent with those reported in the main paper on the common subset.

In the correctness-based metrics, the CLGC, IGC, and LGC metrics over the common set of prompts are reported in Tables <ref>, <ref>, and <ref>, respectively. Similarly, the results for the three metrics on the complete set of prompts are reported in Tables <ref>, <ref>, and <ref>. As observed in the main paper, across all the metrics, Dalle3 outperforms all other models when evaluated for Indic languages with a significant margin. Other models struggle to generalize on the Indic languages with AltDiffusion m9 performing the best among the other three models. This behavior is a result of increased generalizability due to the multilingual training of the model in 9 languages. On the other hand, while all models perform similarly for the English language, AltDiffusion m9 performs the worst, possibly due to catastrophic forgetting when trained for multilinguality.



In representation-based metrics, we evaluate the SCWL and DWL metrics on the complete set and observe that they follow the same patterns as their performance on the common set (refer Fig. <ref>). It is also observed that while AltDiffusion m9 has high distinctiveness across the concepts it generates, it provides poor self-consistency within the language, highlighting its instability and a tendency to seemingly generate diversely with or without relevance to the prompt. For the SCAL metric, the value obtained for AltDiffusion m9 on the common set of prompts comes out to be 21.47%, which is lower than the overall SCAL metric for the Stable Diffusion, AltDiffusion, Midjourney, and Dalle3 models is 25.44%, 23.73%, 26.75%, and 29.90%, respectively. This indicates an overall low consistency of AltDiffusion m9 in generating concepts across different languages.




§ QUALITATIVE ANALYSIS

In this main paper, we qualitatively analyzed the images generated by the different models across the different languages. We present more qualitative results here in Figs. <ref>, <ref>, and <ref> corresponding to Stable Diffusion, Alt Diffusion, and Midjourney, respectively.

The Stable Diffusion model (Fig. <ref>) generates images containing a high number of individuals, temples, flowers, and gods. In the case of Arabic scripts (kas_Arab, snd_Arab, urd_Arab), the model produces men wearing Muslim caps and women in burkhas or niqabs. For all Devanagri scripts as well as for as guj_Gujr in Gujarati script, the model generates individuals with sarees, and tilak which are often worn in the Indian culture. Additionally, it produces gods and temples. Sanskrit (sans_Deva) in particular, generates a large number of gods and temples due to its extensive religious context. This pattern is present across all languages using the Devanagri script. For languages using the Bengali script (asm_Beng, ben_Beng, mni_Beng), the model produces a significant amount of distinctive greenery. Sindhi in Arabic script (snd_Arab) is the only language generating a substantial amount of pornographic content. 



The AltDiffusion model (Fig. <ref>) generates images of couples, gods, places of worship, and occasionally of Indian monuments such as the Taj Mahal, especially for languages with the Devanagari script. When generating for Arabic scripts, such as kas_Arab, snd_Arab, and urd_Arab, the model generates images of men wearing Muslim caps, women in burkhas or niqabs, and mosques showcasing a correlation between the Arabic script and Muslim culture. In the Gurumukhi script (pan_Guru), the model depicts individuals with long beards and turbans, highlighting correlations between the script of the Punjabi language with stereotypical portrayal of people from Punjab. Languages from the Dravidian family, including Telugu (tel_Telu), Tamil (tam_Taml), Malayalam (mal_Mlym), and Kannada (kan_Knda), along with Sinhala (sin_Sinh), feature images of dark-skinned individuals, possibly associating skin-color with the individuals being generated.


In Midjourney (Fig. <ref>), for the Devanagari script, the model produces a variety of visuals, including women, gods, and deities in Hinduism (like Shiva, Ganesha, and Krishna), pandits (priests), temple-like structures (religious places of worship), elephants, and tigers (typically shown in stereotypical depictions of India). Within the Dravidian family languages such as Malayalam (mal_Mlym), Kannada (kan_Knda), Telugu (tel_Telu), and Tamil (tam_Taml), common elements include jewelry such as necklaces, forehead pendants, earrings, and dark-skinned individuals, particularly men. Bengali scripts (asm_Beng, ben_Beng, mni_Beng) often feature bridal women in red sarees adorned with jewelry.  Other commonly generated images include images of food, such as fish, which is prevalent in Bengali culture. Languages with Arabic script like Kashmiri (kas_Arab), Urdu (urd_Arab), and Sindhi (Snd_Arab) commonly depict women with hijabs or niqabs, men in headcovers, and places of Islamic worship such as mosques. In Gujarati, in addition to Devanagari influences, food-related imagery is prominent. Finally, Punjabi in Gurumukhi script (pun_Guru) frequently showcases individuals wearing turbans with a Gurudwara (place of worship in Sikhism, a religion commonly practiced by residents of Punjab) in the background. For certain languages such as Santali (sat_Olck) and Manipuri in Meitei script (mni_Mtei), Midjourney generates Asian women, depicting no Indian cultural influences (Fig. <ref>. It is interesting to note that these two languages also produce random outputs in the Dalle3 model, which understands many of the other Indic languages. 












