Paper Title: Tamper-Resistant Safeguards for Open-Weight LLMs.



[NO \title GIVEN]
    [NO \author GIVEN]
    August 5, 2024
======================




[1]Equal contribution. ^Equal advising. Correspondence to mailto:rishubt2@illinois.edurishubt2@illinois.edu.
[2]Part of work done during an internship at the Center for AI Safety.





§ ABSTRACT 
Rapid advances in the capabilities of large language models (LLMs) have raised widespread concerns regarding their potential for malicious use. Open-weight LLMs present unique challenges, as existing safeguards lack robustness to tampering attacks that modify model weights. For example, recent works have demonstrated that refusal and unlearning safeguards can be trivially removed with a few steps of fine-tuning. These vulnerabilities necessitate new approaches for enabling the safe release of open-weight LLMs. We develop a method, called , for building tamper-resistant safeguards into open-weight LLMs such that adversaries cannot remove the safeguards even after thousands of steps of fine-tuning. In extensive evaluations and red teaming analyses, we find that our method greatly improves tamper-resistance while preserving benign capabilities. Our results demonstrate that tamper-resistance is a tractable problem, opening up a promising new avenue to improve the safety and security of open-weight LLMs.






§ INTRODUCTION







The most capable open-weight large language models (LLMs) released over the past year now rival closed-source frontier models <cit.>. The availability of open-weight LLMs for anyone to download and use has yielded numerous benefits, including lowering costs for end users and enabling academic research on safety and security <cit.>. However, as these models become increasingly powerful, many have raised concerns that they could be repurposed by malicious actors to cause harm, motivating research on how to safeguard these models against malicious use.


Existing open-weight models often adapt safeguards designed for closed-weight models served through APIs <cit.>. These safeguards include refusal mechanisms and preference-based training, and they have provided substantial robustness against input-based jailbreaking attacks. However, recent work has demonstrated these safeguards are trivially defeated by attacks that edit model weights, breaking down after only a handful of fine-tuning steps <cit.>. This poses a serious problem for open-weight models, because adversaries have full access to model weights and can tamper with
built-in safeguards.

The vulnerability of open-weight models to tampering attacks poses risks for model developers as well. Under background tort law, AI developers must exercise reasonable care, meaning they have an obligation to take reasonable precautions to prevent foreseeable harm. If malicious actors can easily customize models to cause critical harm, model developers may inadvertently violate reasonable care standards and become open to liability under existing law. Thus, there is an urgent need for more robust safeguarding techniques that can withstand tampering attacks.

In this work, we study the problem of tamper-resistant safeguards for LLMs. This problem is depicted in Figure <ref>. Unlike existing research on LLM safeguards, we focus on attacks that modify model weights, which we refer to as tampering attacks. This problem has been considered very challenging and by some intractable, as no method has yet provided substantial robustness to these attacks. However, making progress on this problem would provide a valuable tool to regulators and model developers by ameliorating the dual-use dilemma of open-weight models <cit.>. 

To demonstrate that progress on this problem is possible, we develop the first LLM safeguards that obtain strong robustness against a wide variety of tampering attacks. Our approach allows developers to add a safeguard such that tampering attacks cannot easily remove the safeguard, while preserving the general capabilities of the LLM. We achieve this by performing adversarial training against tampering attacks, leveraging approaches from meta-learning. We identify various crucial factors that enable our method to work, including the choice of tamper-resistance loss, the selection of train-time adversaries, and the two-stage approach that we use for building in safeguards.



We apply our method to develop tamper-resistant unlearning and refusal safeguards. In experiments, we demonstrate that our safeguards are far more robust to tampering attacks than prior methods. We stress-test our safeguards with extensive red teaming evaluations against  test-time adversaries, demonstrating resistance to fine-tuning attacks up to 5,000 steps. We hope our results foster future work on this important problem. Our experiment code and models are available at <https://github.com/rishub-tamirisa/tamper-resistance>.





§ RELATED WORK



  
Adversarial attacks on LLMs. Due to the extensive pre-training distribution of modern LLMs, they are prone to generating harmful content <cit.>. To mitigate this, many LLMs undergo fine-tuning to implement safeguards <cit.>, using methods such as reinforcement learning from human feedback (RLHF) <cit.> and direct preference optimization (DPO) <cit.>. While effective for normal use, these safeguards have been shown to be brittle, breaking down under jailbreak attacks <cit.> or a handful of fine-tuning steps on “uncensored” data <cit.>. This suggests current techniques for LLM alignment are inadequate, raising security concerns after deployment.

r0.5
    

    
    < g r a p h i c s >

    Comparison of our method to 12 baseline safeguards. Unlike prior methods, provides far greater tamper-resistance at similar levels of general capability, measured via MMLU. Tamper-resistance is computed as the normalized error on WMDP Biosecurity, Chemical Security, and Cybersecurity questions <cit.>, averaged across up to  fine-tuning attacks.
    
    







  
LLM safeguards. Since the discovery of these attacks, many safeguards have been proposed to defend against them. Against jailbreak attacks, defenses include system-level defenses <cit.> that modify or filter model inputs or outputs and model-level defenses such as adversarial training <cit.>. Alternatively, some works explore machine unlearning as a way to remove harmful knowledge entirely with techniques such as influence functions <cit.>, maximizing loss on forget sets <cit.>, or modifying representations <cit.>. However, jailbreaking defenses are not fully robust to adaptive adversaries <cit.>, and existing unlearning methods are not robust to adversaries with access to model weights <cit.>.





  
Robust safeguards. Several works have explored the tamper-resistance of unlearning methods for image classification <cit.>. For bidirectional BERT-style models, <cit.> proposed a meta-learning approach for robustly preventing models from learning harmful tasks. In concurrent work, <cit.> proposed a method extending this approach to small-scale vision classifiers and diffusion models. Recently, <cit.> discussed the potential for robust unlearning in LLMs to improve the safety of open-source models, and <cit.> proposed evaluation metrics for robust unlearning in LLMs. To the best of our knowledge, no methods have been proposed for autoregressive LLMs that are robust to tampering attacks. 

Several concurrent works have explored ways of defending LLM refusal mechanisms against fine-tuning. <cit.> add a perturbation loss to keep new embeddings close to the original, while <cit.> regularize harmful representations to look random. Unfortunately, the authors acknowledge that these approaches have highly limited robustness. We corroborate this in our comparisons, finding that they lack robustness to the tampering attacks in our evaluations.


























§ TAMPER-RESISTANT SAFEGUARDS






 §.§ Threat Model


We assume the defender releases an LLM with weights θ_G and a safeguard G applied. The defender's goal is to design G such that θ_G obtains high values on (θ_G) and (θ_G). Moreover, the defender seeks to preserve a high value of (θ_G) after the adversary's move. We consider a compute-bounded adversary with unrestricted access to θ_G, enabling attacks that directly modify θ_G. We refer to these as “tampering attacks.” The adversary's goal is to obtain a model θ'_G that minimizes the safety metric given reasonable compute limits, such as fine-tuning for 1,000 to 5,000 steps. We assume the adversary will not spend a significant fraction of the compute required to pre-train the LLM, since at that point they could train their own model without safeguards.









 §.§ Problem Definition and Metrics



We describe a general notation for quantifying the tamper-resistance of safeguards. Define G, θ_G, , and  as in the threat model. Let  denote a compute-bounded adversarial attack that maps θ_G to θ'_G, with stronger attacks obtaining lower values of (θ'_G). We say that a safeguard G is tamper-resistant if its post-attack (θ_G') is high across a broad range of strong test-time adversarial attacks 𝒜_test.

Note that θ_G often modifies an underlying θ that lacks safeguards, often through a fine-tuning procedure. Additionally, strong tamper-resistance can be obtained if the safeguard simply overwrites θ with noise, but this model would no longer be useful. Thus, maintaining a high (θ_G) is crucial, and evaluation of a safeguard must consider both its tamper-resistance and how well it preserves general capabilities. 


We focus on two common safeguard domains: weaponization knowledge restriction and harmful request refusal. In each domain, we define safety and capabilities test metrics, which we use alongside test-time adversaries to evaluate tamper-resistant safeguards.



  
Weaponization knowledge restriction. In weaponization knowledge restriction, safeguards prevent the model from producing text about weaponization knowledge, while preserving capabilities for benign knowledge domains. Existing safeguards of this nature include representation engineering methods like circuit breaking <cit.>. The  is defined as error on a forget set, and the  is defined as accuracy on a retain set. Specifically, we consider the problem of restricting biosecurity, chemical security, and cybersecurity knowledge, and evaluate the resulting model on the Weapons of Mass Destruction Proxy (WMDP) benchmark <cit.>. WMDP contains 3,668 multiple-choice questions, spanning biosecurity, chemical security, and cybersecurity knowledge. Importantly, WMDP questions do not evaluate hazardous knowledge directly, but instead measure proxy expert-level knowledge for each hazardous domain, such that suppressing the expert-level knowledge would also suppress the hazardous knowledge. We define the forget set as the respective hazardous knowledge subject in WMDP, and retain set as the complement of the given subject in MMLU <cit.>, a multi-task question-answering benchmark spanning 57 tasks across a variety of knowledge domains.



  
Harmful request refusal. In the harmful request refusal setting, safeguards prevent the model from producing “harmful” outputs. We define the  as the complement of average Attack Success Rate (ASR) of various jailbreaking attacks, while the  captures the conversational abilities of θ_G. Specifically, we use a static set of test cases from HarmBench, an automated red-teaming framework for measuring prompt jailbreak robustness in LLMs, to evaluate jailbreak ASR <cit.> after tampering attacks. We use MT-Bench, a multi-turn question-answering benchmark graded by an LLM judge, to evaluate conversational abilities <cit.>.


    


 §.§ Red Teaming




To properly measure the robustness of tamper-resistant safeguards, we conduct red-teaming with up to  adversaries, including many that are unseen at training time. In our evaluations, we subject our method to adversaries with varying compute budgets, access to held-out datasets, and diverse hyperparameters. For fine-tuning adversaries, we vary the learning rate, learning rate scheduler, optimization algorithm, and batch size. Many of these adversaries were fixed during early experiments, with some added over time as we found attacks that broke intermediate versions of our method. Extensive stress testing of this nature is critical for obtaining confidence in a tamper-resistant safeguard. For research on developing these safeguards, extensive red teaming also allows measuring incremental progress, using the number and strength of existing attacks one can defend against as a robustness metric.












§ SAFEGUARD TAMPER-RESISTANCE TRAINING









To obtain tamper-resistant safeguards, we propose a new method outlined in Algorithm <ref> inspired by adversarial training and meta-learning to directly strengthen LLM safeguards against tampering attacks, called Tampering Attack Resistance (). We identify unique properties of this adversarial training regime and leverage them to improve robustness.

Our method for training tamper-resistant safeguards consists of two phases: (1) model safeguarding and (2) tamper-resistance training.



 §.§ Model Safeguarding
 The method begins by including an initial safeguard G into a base model θ. For example, initial safeguards for knowledge restriction can be drawn from a wide variety of existing methods, including circuit breaking <cit.> or constrained gradient ascent for a particular knowledge domain. Similarly, we can include a refusal safeguard by performing RLHF <cit.> or DPO <cit.> on refusal completions. Importantly, these initial safeguards do not need to be tamper-resistant. Empirically, we find that this safeguarding step is important for obtaining strong tamper-resistance.







 §.§ Tamper-Resistance Training






Starting from θ_G_0, we train the tamper-resistant θ_G using a novel adversarial training procedure. Namely, we train against a set of tampering attacks 𝒜_train, where the defender's objective is to maximize a proxy  after applying an adversarial attack ∼𝒜_train to θ. Since it may not be feasible to differentiate through , we draw on insights from prior work in meta-learning, defining (θ_G) = θ'_G = θ_G + '(θ_G) as a perturbation on top of initial parameters, where backpropagation through ' is approximated with a straight-through estimator <cit.>. 








We focus on supervised fine-tuning (SFT) adversaries where  applies several steps of optimization to θ_G, which allows straight-through estimation through ' to benefit from the setting and approximations of first-order MAML <cit.>. However, we note key differences in our approach from standard meta-learning and prior methods <cit.>. In particular, traditional meta-learning techniques seek to obtain a model initialization that is close to optimality on multiple test distributions. In our setting, we seek to obtain an initialization that is far from optimality on multiple adversaries' test distributions. Novel to our approach in this new setting is the use of a tamper-resistance loss in the “outer loop” that differs from the fine-tuning adversary's loss function and serves to maximize the proxy safety metric. We depict this structure in Algorithm <ref>, and explain the objective below.





  
Impeding the adversary's loss. The aim of tamper-resistance training is to prevent adversaries with large compute budgets from reducing the  at test-time. In adversarial training for tamper-resistance, we define a tamper-resistance loss ℒ_TR that counters . We operationalize our goal of avoiding adversary optimality as searching for θ such that ℒ_TR is minimized for (θ).

Empirically, we find that the choice of tamper-resistance loss ℒ_TR significantly affects this goal. Prior work <cit.> negates the loss of a fine-tuning adversary, in which the aim is to arbitrarily maximize the adversary's loss throughout fine-tuning. This formulation has two issues: (1) maximizing a cross-entropy loss can cause divergence; (2) empirically we observe that when using this objective against fine-tuning adversaries, the model learns to explode the adversary's loss for the first few inner loop steps, while loss at later steps remains low. In Figure <ref>, we show the difference in choosing ℒ_TR to be a clamped negative cross-entropy loss vs. negative entropy loss for weaponization knowledge restriction. For the latter, ℒ_TR is eventually satisfied for all inner loop steps. For harmful request refusal, we choose ℒ_TR to be the DPO loss <cit.>. We provide further detail on the choice of ℒ_TR in both settings in Appendix <ref>.



  
Tamper-resistance objective. We now describe the general proxy objective used for preventing a tampering attack from recovering weaponization knowledge or harmful behavior. For a given , let 𝒟_TR and ℒ_TR respectively be a dataset and loss function such that minimizing ℒ_TR(θ_G, 𝒟_TR) serves as a proxy objective for maximizing the (θ_G). We define 𝒟_retain and ℒ_retain correspondingly for (θ_G). The defender's objective is to solve the following optimization problem:

    min_θλ_TR·𝔼_∼𝒜_train[  ℒ_TR((θ); 𝒟_TR) ] + λ_retain·ℒ_retain(θ; 𝒟_retain),
 where ℒ_TR is a tamper-resistance loss that counters (θ). The ℒ_retain term is a representation engineering <cit.> inspired retain loss for preserving performance on the capabilities proxy dataset 𝒟_retain, given by

    ℒ_retain(θ; 𝒟_retain) = 𝔼_x∼𝒟_retain[ℒ_LM(θ, x) + ‖ h_θ(x) - h_θ_G_0
    (x) ‖_2^2]
 where h_θ(·) returns the residual stream hidden states for model parameters θ and ℒ_LM is the standard language modelling cross-entropy loss. Empirically, we find that pushing retain-set residual stream representations to be close to the base model θ_G_0 via the ℓ_2-norm loss in Equation <ref> maintains a high (θ_G). In Equation <ref> we include λ_TR and λ_retain as scalar weightings for the tamper-resistance loss and retain loss, respectively. We provide further details on the design of the tamper-resistance loss function in Appendix <ref> as well as an efficiency trick for sampling fine-tuning attacks for in Appendix <ref>.



























































§ EXPERIMENTS


We evaluate in weaponization knowledge restriction and harmful request refusal settings, with results shown in Table <ref> and Table <ref> respectively. We discuss the setup, baselines, and analysis for our results. In each setting, we use a specific set of training adversaries 𝒜_train and test adversaries 𝒜_test. Further experiment details are presented in Appendix <ref>.



 §.§ Weaponization Knowledge Restriction

We now describe the setup, baselines, and results for our weaponization knowledge restriction experiments, including the knowledge domains, optimizers, and evaluation details.



  
Setup.
 We focus on implementing tamper-resistant safeguards for restricting proxy weaponization knowledge about biosecurity, chemical security, and cybersecurity from Llama-3-8B-Instruct <cit.> that has been initially safeguarded via the Random Mapping method discussed in Appendix <ref>. For each weaponization domain, we assign 𝒟_𝒯ℛ to the corresponding forget set described in Appendix <ref>. We proceed to sample train-time 64-step fine-tuning attacks from different data distributions, detailed in Appendix <ref>. We use N = 750 outer loop steps, ScheduleFree AdamW <cit.> with a learning rate of 2×10^-5 as the outer loop tamper-resistance optimizer, and loss scales of λ_TR = 4.0, λ_retain = 1.0. Lastly, We evaluate Pre-Attack and Post-Attack accuracy on corresponding WMDP subjects <cit.> averaged across all adversaries in Appendix <ref>, and measure benign capabilities via the complement of subjects related to each proxy weaponization domain in MMLU <cit.>.







  
Baselines.
We evaluate two recently proposed knowledge restriction methods: RMU <cit.> and LLMU <cit.>. We also design two baseline methods for knowledge restriction: Min Posterior, which minimizes posterior loss on forget set tokens; Max Entropy, which maximizes entropy on forget set tokens. Two additional methods, MLAC <cit.> and SOPHON <cit.>, require substantial modifications for the LLM setting, so we show results on adapted versions of these baselines in <Ref>.









  
Results.
We show weaponization knowledge restriction safeguard results on Llama-3-8B-Instruct in <Ref> and Figure <ref>. These results are averaged across all 28 adversaries from Appendix <ref>. Our large-scale experiments corroborate the findings in recent work that existing LLM safeguards are extremely brittle to fine-tuning attacks. By contrast, maintains low post-attack forget accuracy across all three domains. We also observe that maintains non-negligible retain accuracy, although there is a modest trade-off, much like how adversarial robustness trades off benign accuracy. In Figure <ref>, we observe that is robust to significantly more fine-tuning attacks than all prior methods. While existing baselines break down under most attacks, obtains a post-attack forget accuracy near random chance for nearly all attacks, indicating a successful defense.

Overall, we find that provides significantly more robustness to realistic fine-tuning attacks than all prior methods, including attacks as strong as 5,000 steps of optimization on completely held-out data. These results demonstrate for the first time that obtaining strong tamper-resistance for open-weight LLMs may be possible.





 §.§ Harmful Request Refusal







 We now describe the setup, baselines, and results for our harmful request refusal experiments, including the datasets used and evaluation details. 


  
Setup.For harmful request refusal training, we seek to make existing refusal safeguards in Llama-3-8B-Instruct robust to tampering attacks. We sample train-time adversaries that perform 64-step SFT attacks using the Anthropic-HH-RLHF dataset <cit.>, following the methodology in Appendix <ref>. Similar to the weaponization knowledge restriction setting, we use N = 100 outer loop steps, ScheduleFree AdamW <cit.> with an LR of 6 × 10^-5 as the outer loop tamper-resistance optimizer, and loss scales of λ_TR = 0.1, λ_retain = 1.0. We evaluate the Post-Attack jailbreak attack success rate (ASR) on HarmBench <cit.> after the tampering attacks in Appendix <ref>, and measure benign capabilities preservation via MT-Bench <cit.>, which evaluates multi-turn conversation ability. 






  
Baselines. We consider 4 baselines alongside our model: Llama-3-8B-Instruct (Refusal Trained); Representation Rerouting (RR) <cit.> on Llama-3-8B-Instruct, which trains to push representations for harmful input prompts to be orthogonal to the original corresponding representations in Llama-3-8B-Instruct; R2D2 <cit.> on Zephyr-7B <cit.>, which performs adversarial training against GCG attacks <cit.>; and RepNoise <cit.> on Llama-2-7B <cit.>, which regularizes harmful representations to Gaussian noise.  





  
Results.
We show refusal results in <Ref>. While the Refusal Training, RR, and R2D2 baselines resist jailbreak attacks in HarmBench before tampering, we find that percentage attack success rate jumps up to above 77 after tampering, while our method only rises to 61.7. Since we apply our refusal safeguard to Llama-3-8B, it does reduce MT-Bench by 1.7. However, this exceeds the MT-Bench score of fairly capable open-weight models, indicating that benign capabilities are largely preserved. We leave the exploration of the full impact on capabilities to future work. Additional results are in <Ref>. In general, we find that our model refuses more Post-Attack jailbreaks than previous baselines, and demonstrates the flexibility of the tamper-resistance objective to accommodate the harmful request refusal setting.









 §.§ Analysis





  
Red teaming.
To stress-test the tamper-resistance of our models, we conduct an extensive suite of supervised fine-tuning attacks with  distinct adversaries. We vary the optimizer, number of optimization steps, learning rate, learning rate schedule, fine-tuning dataset, batch size, and overall fine-tuning method (e.g., full fine-tuning versus parameter-efficient fine-tuning). By default, our attacks use 1,000 fine-tuning steps, although some use 5,000 steps. Full details for these adversaries are provided in <Ref>.

We show red teaming results in Figures <ref> and <ref>. While baseline safeguards withstand fine-tuning attacks in a small number of cases, most adversaries succeed in removing the safeguards. By contrast, our safeguard is robust to a wide range of adversaries. This shows that tamper-resistance is a tractable problem on which progress can be made. However, our method is currently not robust to parameter-efficient fine-tuning (PEFT) attacks (adversaries 27 and 28), highlighting the importance of extensive red teaming when developing tamper-resistant defenses. We hypothesize that future work could easily address this issue, as we demonstrate in <Ref> that targeted patching of vulnerabilities is possible.


r0.5
    

    
    < g r a p h i c s >

    Our safeguard remains robust to fine-tuning attacks that greatly exceed the 64 steps used by train-time adversaries. For the LLMU safeguard, the adversary's loss quickly decreases into the recovery region. By contrast, maintains flatness at a high loss for all 1,000 steps.
    

    





  
Generalization to stronger test-time attacks. In Figure <ref>, we show a fine-tuning attack at an LR of 2×10^-5 that attempts to recover biosecurity knowledge on our model and a model safeguarded with LLMU. We find that the tamper-resistance of generalizes far beyond the 64 steps used by train-time adversaries. Surprisingly, we observe that the test-time adversary's cross-entropy loss does not decrease below 7 for all 1,000 steps. Moreover, the loss enters a plateau and does not decrease at all after 200 steps. In <Ref>, we find that the length and height of this plateau can be increased by increasing the number of inner loop steps during adversarial training. As a point of reference, we show the progression of the same attack on LLMU. In this case, the adversary's loss decreases to within the recovery region in under 20 steps. We note that the adversary's test loss decreasing into the recovery region does not always correspond with recovery on downstream metrics (e.g., WMDP), but often does.































    



























§ CONCLUSION


We introduced a novel method for implementing tamper-resistant safeguards for LLMs and explored applications in weaponization knowledge restriction and harmful refusal training. We compare our results to prior work in each setting, finding that our method is the first method robust under the rigorous red-teaming evaluation that we consider. More broadly, we demonstrate that progress on open-weight tamper-resistance is tractable. We believe this line of research is crucial for enabling ongoing deployment of robust, open-weight LLMs, ensuring their alignment with regulatory frameworks and preemptively addressing the risk of malicious use.





  
Acknowledgements. We thank Steven Basart and Luis Fernandez for providing valuable feedback for the paper, and Andriy Novykov for providing support for compute resources at the Center for AI Safety. We thank the Center for AI Safety for providing significant compute resources for this project. We also thank Volodymyr Kindratenko and the National Center for Supercomputing Applications (NCSA) and Illinois Campus Cluster Program (ICCP) for supporting our computing needs. This work used NVIDIA GPUs at NCSA Delta through allocations CIS230117 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by NSF Grants #2138259,
#2138286, #2138307, #2137603, and #2138296.











abbrvnat










§ LIMITATIONS




Our method for training tamper-resistant safeguards demonstrates considerable robustness against a wide range of tampering attacks, yet several avenues for improvement remain: (1) While we focus on supervised fine-tuning attacks, the broader spectrum of open-weight tampering techniques necessitates diverse future red-teaming efforts. (2) Scaling to larger models poses computational challenges that require optimization to reduce overheads. 



Tamper-resistance alone cannot fully mitigate the risks of malicious AI use. While it raises the initial costs for adversaries, it can eventually be circumvented. Once open-weight models are released, they cannot be “unreleased,” leaving any compromised defenses permanently vulnerable. Therefore, tamper-resistance should be considered a supplement to the broader effort of of improving the offense-defense balance of AI systems. Addressing these limitations will improve the robustness of LLMs to tampering and better support open-weight model developers.







§ METHOD DETAILS





 §.§ Initial Weaponization Knowledge Restriction Safeguard


Prior to tamper-resistance training, we install a safeguard that achieves surgical knowledge restriction on the target hazardous domain. Let h_θ(𝒟) denote the distribution of post-decoder layer residual stream activations for input sequences sampled from some data distribution 𝒟 and model weights θ. We define  (x) for some input sequence x, which returns fixed Gaussian-sampled vectors that are chosen via hashing the corresponding input token for each residual stream index of x in θ. As a proxy for scrubbing target representations according to downstream task labels, we propose a weaponization knowledge restriction safeguard termed Random Mapping, which maps h_θ(𝒟_TR) to random noise as follows:



    min_θ𝔼_x∼𝒟_TR[1 - | h_θ(x) ·(x)/‖ h_θ(x)‖‖(x)‖| ] + ℒ_LM(θ; 𝒟_retain)


The objective of Equation <ref> maximizes cosine similarity between row vectors in the residual stream in every layer of the LLM from h(𝒟_TR) and the hashed random vectors from (·). By providing each token's residual stream a unique random vector to push toward, the loss encourages a “re-mapping” of token representations from 𝒟_TR to the noised vectors. We include an additional term for preserving performance on 𝒟_retain via the language-modelling cross-entropy loss ℒ_LM. We show the performance of the raw Random Mapping safeguard as an ablation in Table <ref>, listed as “Excl. Adv. Training.”




 §.§ Designing the Tamper-resistance Loss




  
Weaponization knowledge restriction. For weaponization knowledge restriction, we summarize our intuition for ideal tamper-resistance loss design, corroborated by our empirical findings in Figure <ref>: we seek to flatten the adversary's loss at a high value, rather than simply raise its y-intercept.

We choose the tamper-resistance loss ℒ_TR as an entropy loss to be maximized during the adversary's cross-entropy fine-tuning trajectory, since maximizing entropy would impede the adversary's cross entropy loss from decreasing during fine-tuning. In other words, we wish to obtain θ such that after an adversary performs a fine-tuning attack on θ via a cross-entropy loss, entropy is still high. We find that this formulation achieves the desired flattening behavior, and we depict the difference in flattening between the choosing ℒ_TR to be a negative cross-entropy loss and negative entropy loss in Figure <ref>. In the lefthand plot, where ℒ_TR is a cross-entropy loss, loss only increases in the first inner loop step. In the righthand plot, where ℒ_TR is a negative entropy loss, entropy is eventually maximized in all inner loop steps. Figure <ref> also demonstrates the generalization of the flat adversary loss behavior beyond the length of the simulated fine-tuning trajectories during .



  
Harmful request refusal. For harmful request refusal, we choose ℒ_TR to be the DPO loss <cit.>, which works as follows. Given a DPO dataset containing pairs of rejected and refusal completions, the sampled  performs SFT on rejected completions, and the tamper-resistance loss ℒ_TR is a DPO loss computed on the pair chosen and rejected completions on parameter coordinates along the  trajectory. This encourages to find an initialization θ such that after a harmful fine-tuning attack, the model still prefers refusal completions over harmful completions when given an harmful prompt. While this does not necessarily encourage a flat adversary loss, we find empirically in Table <ref> that this formulation increases the average (θ_G) defined in Section <ref> after fine-tuning attacks on harmful data, detailed in Section <ref>.



We also observe that the length of the simulated adversary SFT trajectory during training affects test-time generalization in both Figure <ref> and Appendix <ref>. In particular, larger values of K result in increased tamper-resistance for longer SFT attacks. However, to maintain reasonable runtime efficiency, we need a more efficient sampling technique than simply running K independent trajectories of varying length for every outer-loop step in Algorithm <ref>, which we describe in Section <ref>.



 §.§ Efficiently Sampling Fine-tuning Attacks
 


Optimizing Equation <ref> with gradient descent requires simulating K tampering attacks for each tamper-resistance optimizer update, which is prohibitively expensive to run when the sampled  performs SFT and θ contains billions of parameters. Inspired by prior work on snapshot ensembles <cit.>, we leverage an efficiency trick: we can reuse the coordinates along steps of a single adversary fine-tuning trajectory of length K to obtain K-1 additional (though non-independent) trajectories of increasing length. Using this trick, we collect all K parameter coordinates along the trajectory into a single batch for computing the tamper-resistance losses, effectively sampling  from 𝒜_train non-IID. To further improve runtime efficiency, we do not compute the tamper-resistance loss ℒ_TR on all K steps and instead sub-sample coordinates along the trajectory for computing ℒ_TR within an adversary batch, for example every 4 adversary optimization steps. Additionally, we reduce variance in the tamper-resistance gradient by computing the tamper-resistance loss at each inner loop step on the same held-out batch, denoted as x_TR in Algorithm <ref>. 



 §.§ Implementation Details and Resource Requirements


We perform training on Llama-3-8B-Instruct <cit.> with 8 NVIDIA 80GB A100 GPUs, leveraging distributed training via FSDP <cit.>. We use ZeRO Stage 3 from DeepSpeed <cit.>, which shards optimizer states, gradients, and parameters during training. While the efficiency trick in Appendix <ref> improves runtime, we note additional considerations for conserving GPU memory. 

First, simulating fine-tuning attacks that require additional state (e.g., momentum) in the inner loop of Algorithm <ref> requires initializing a fresh optimizer for every outer loop iteration. Since we use an outer-loop optimizer that also requires maintaining state (ScheduleFree AdamW <cit.>), we move the outer loop optimizer to the CPU before instantiating inner-loop optimizers.

Second, first-order meta-learning in smaller models can typically be implemented by running multiple forward passes for each inner loop iteration, averaging losses, then backpropagating on the averaged loss term. However, because each inner-loop tamper-resistance loss term (ℒ_TR in Algorithm <ref>) is computed on a separate forward pass, this requires maintaining K computation graphs in memory. Since this is infeasible on reasonable hardware for LLMs with billions of parameters, we circumvent this inefficiency by accumulating tamper-resistance gradients in a separate data structure (g_TR in Algorithm <ref>). We note that this can be done without using additional  and  distributed operations, since tamper-resistance gradient accumulation and application to the pre-inner loop model parameters (θ_i-1 in Algorithm <ref>) can be computed solely on sharded gradients. 












§ ADDITIONAL EXPERIMENTS





 §.§ Benign Fine-Tuning






An important property of open-weight models is that they can be fine-tuned to improve performance on custom data or in specific domains. Thus, ideal tamper-resistant safeguards should allow continued fine-tuning of a model while preserving the safeguard. We evaluate whether models can be fine-tuned on data unrelated to the safeguard using economics as an example domain. Using models with biosecurity and cybersecurity safeguards, we perform supervised fine-tuning on the WMDP auxiliary economics corpora <cit.>. We fine-tune models for 800 steps using a learning rate of 2×10^-6 and a batch size of 64. For the test set, we evaluate average accuracy on the corresponding MMLU High School Macroeconomics and Microeconomics subjects before and after fine-tuning. To confirm that the safeguard remains tamper-resistant in this setting, we also evaluate accuracy on corresponding WMDP subjects before and after fine-tuning.

We show the results of this evaluation in <Ref>. We find that accuracy on economics questions can be improved by 7.5 percentage points without recovering significant hazardous knowledge. This illustrates that strong tamper-resistance can be compatible with benign model editing. Additional experiment details are discussed in Appendix <ref>.



 §.§ Varying the Train-time Inner-loop Length K



r0.5
    

    
    < g r a p h i c s >

    Comparison of a 1,000-step SFT attack against with the inner-loop length K varied between {8, 16, 32, 64}. Test-time loss plateau magnitude and duration tend to increase as K increases.
    




Recall that via the efficiency trick discussed in Appendix <ref>, a single inner loop trajectory of length K during returns the K sampled attacks in Algorithm <ref>.  We compare the test-time loss robustness as we vary the length of the inner loop K during , running fine-tuning attacks for 1,000 steps on a held-out forget dataset for biosecurity weaponization (Adversary 8 in Table <ref>). For each value of K, we observe a plateau in the test loss that drops off at later steps as K increases. This suggests that the robustness of improves as the inner loop length increases. Prior work also corroborates that increasing the inner-loop length during meta-learning increases test-time generalization <cit.>. We note the contrast to conventional meta-learning methods mentioned in Section <ref>, in which typical meta-learning applications seek optimality after as few test-time steps as possible <cit.>. Here, our results suggest that the objective is incentivized to run with as many inner loop steps as possible. We find that K=64 provides significant robustness to the range of adversaries we consider in Section <ref> and Appendix <ref>, while balancing computational efficiency as discussed in Appendix <ref>. 






 §.§ Targeted Patching of Vulnerabilities



In Section <ref>, we discuss the set of adversaries sampled during for each weaponization knowledge domain. However, we included the “Retain-set SFT followed by Weaponization-domain SFT” (R→F in Table <ref>) adversary to improve robustness to specific adversaries that broke the defense in earlier iterations of the method that did not sample this adversary. We find in Table <ref> that including the R→F adversary significantly improves robustness to more test-time attacks, indicated by an improvement in Post-Attack Forget error by over 20.0 percentage points. Importantly, we find that including the 64-step R→F adversary at train-time improves test-time robustness at for the 1000-step R→F adversary by nearly 40.0 percentage points, indicating that target patching of vulnerabilities is possible within .










 §.§ Ablations






  
Retain loss ℓ_2 representation engineering term. We observe in Table <ref> that retain-set MMLU performance is worse when excluding representation engineering <cit.> retain loss from . When including the term, retain MMLU improves by at least 5.0 percentage points. 



  
Including the initial safeguard. In Table <ref>, we examine the impact of incorporating the Random Mapping safeguard step prior to the adversarial training phase during . The Random Mapping safeguard in isolation achieves a near-random chance Pre-Attack Forget accuracy of 27.3. However, it is susceptible to fine-tuning attacks similar to other baselines in Table <ref>, indicated by a higher Post-Attack Forget accuracy of 59.4. When including the tamper-resistance adversarial training phase, we observe significantly increased tamper-resistance as the Post-Attack Forget accuracy decreases by nearly 28 percentage points. 

We also examine the impact of excluding the initial safeguarding step (“Excl. Initial Safeguard” in Table <ref>), finding that Pre-Attack and Post-Attack Forget accuracies are both higher 
without the initial safeguard. While the tamper-resistance adversarial training phase achieves moderate tamper-resistance without the initial safeguard step, we find that including the Random Mapping phase improves downstream tamper-resistance by 15.6 percentage points.





  
Varying the tamper-resistance loss scale λ_TR. We compare the downstream robustness of when varying the tamper-resistance loss weighting λ_TR between 1.0 and 4.0 in Table <ref>. We observe that when setting λ_TR = 1.0, maintains high retain MMLU accuracy at 62.5 percentage points, with moderate tamper-resistance indicated by a Post-Attack Forget accuracy of 40.8. Further increasing λ_TR to 4.0 in our final model results in a significantly improved Post-Attack Forget Accuracy of 31.3, with a partial decrease in Retain MMLU to 54.9. When varying λ_TR, we keep λ_retain constant; thus, our results indicate a clear way to increase downstream tamper-resistance by increasing the weighting of the tamper-resistance gradient during , reflecting a balance between tamper-resistance and capabilities similar to the robustness-performance tradeoff for adverarial robustness in vision models.





§ EXPERIMENT DETAILS





 §.§ Weaponization Domain Proxy Dataset Details




  
Biosecurity. We use a synthetically labeled partition of the Pile <cit.> that filters for relevance to biology and the Camel AI Biology dataset <cit.>. We generate synthetic labels for Pile token sequences using openchat-3.5 <cit.>, categorizing them as "Cellular Biology" or not. This process yields 49,984 samples: 7,558 for the forget-set (Pile-bio Forget) and 42,426 for the retain set (Pile-bio Retain). Concurrently, we pack entries from the Camel AI Biology dataset to the truncation-enabled 256 tokenization limit, resulting in 54,258 samples of about 188 words each (Camel-bio Forget). We apply the same procedure to our held-out hazardous biology dataset (identical to the WMDP biosecurity Forget-set), producing 598,933 samples of similar length (OOD Forget).



  
Chemical Security. We use a private forget dataset containing text sequences about hazardous chemical security content (Chem Forget).



  
Cybersecurity. We scrape CTF writeups on CTFtime <cit.> that are numbered between 1 and 39181, collecting cybersecurity writeups written as recently as 2024. We filter to keep writeups that contain more than 150 characters. As a result of filtering and HTTP errors while scraping, our resulting forget dataset contains slightly over 18k samples (Cyber Forget).




 §.§ Train-time Settings and Adversaries








  
Weaponization knowledge restriction. For each weaponization knowledge restriction domain, we have a corresponding retain dataset 𝒟_retain, comprised of a mix of data from the Pile-bio Retain set and Magpie-Align instruction-tuning dataset <cit.>, which we refer to as the Retain-set.  Referencing the datasets described in Appendix <ref>, we specify the adversary tamper-resistance datasets that constitute the data used by attacks sampled from 𝒜_train during training, as well as the attack setup for each domain as follows:


    
  * Biosecurity: We simulate six adversaries from the following three setups: Pile-Bio Forget-set SFT, Camel-Bio Forget-set SFT, and Retain-set SFT followed by Pile-Bio Forget-set SFT (R→F), where the switching point between Retain-set and Pile-Bio Forget-set SFT within the 64-step length trajectory is sampled from a beta distribution β(6.0,3.0). LRs are sampled from {2 × 10^-5, 4 × 10^-5}.
    
  * Chemical Security: We simulate nine adversaries from the following three setups: Chem Forget-set SFT, Retain-set SFT, and Retain-set followed by Chem Forget-set SFT, using the same switching-point sampling scheme as in the Biosecurity setting. LRs are sampled from {2 × 10^-6, 2 × 10^-5, 4 × 10^-5}.
    
  * Cybersecurity: We simulate four adversaries from the following two setups: Cyber Forget-set SFT, and Retain-set SFT followed by Cyber Forget-set SFT, using the same switching point sampling scheme as in the Chemical Security and Biosecurity settings. LRs are sampled from {2 × 10^-5, 4 × 10^-5}.


For each weaponization knowledge domain, we create 80-20 splits for adversary and held-out data of the corresponding forget sets, respectively. For Biosecurity, which uses multiple forget datasets, this involves creating 80-20 splits for each dataset, then combining the corresponding splits. The adversary data splits are used for sampled attacks from 𝒜_train, whereas the held-out split is used for computing tamper-resistance losses. The held-out splits for each domain correspond to 𝒟_TR in Section <ref>. We use minibatches from a held-out dataset for computing tamper-resistance losses rather than cycling through a single dataset, following the recommendation of <cit.>, in which first-order meta-learning without properly held-out minibatches caused a performance degradation.

All train-time adversary setups are tabulated in Table <ref>, where F-Pile, F-Chem, F-Cyber denote the respective datasets described in Appendix <ref>, and Retain denotes the mixed Pile-bio and Magpie-Align Retain-set described in Appendix <ref>. We use R→F to label the adversaries that perform Retain-set SFT followed by Forget-set SFT. The final column is an abbreviation for Finetuning Paradigm and indicates whether the SFT setup used full parameter finetuning or parameter-efficient fine-tuning (PEFT) via LoRA adapters <cit.>.





@lccccccc@
Train-time adversary setups for weaponization knowledge restriction of Biosecurity, Chemical Security, and Cybersecurity.
 


Adversary     Dataset     Opt. Steps (K)     Optimizer     LR     LR Schedule     Batch Size     FT Paradigm 



8c
  – continued from previous page 

Adversary     Dataset     Opt. Steps     Opt.     LR     LR Schedule     Batch Size     FT Paradigm 



8rContinued on next page 
 




8cBiosecurity Weaponization Restriction 

Adv 1     F-Pile     64     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 2     F-Pile     64     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 3     F-Camel     64     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 4     F-Camel     64     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 5     R→F     64     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 6     R→F     64     AdamW     4×10^-5     No Warmup     64     Full Parameter 

8cChemical Security Weaponization Restriction 

Adv 1     F-Chem     64     AdamW     2×10^-6     No Warmup     64     Full Parameter 

Adv 2     F-Chem     64     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 3     F-Chem     64     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 4     Retain     64     AdamW     2×10^-6     No Warmup     64     Full Parameter 

Adv 5     Retain     64     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 6     Retain     64     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 7     R→F     64     AdamW     2×10^-6     No Warmup     64     Full Parameter 

Adv 8     R→F     64     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 9     R→F     64     AdamW     4×10^-5     No Warmup     64     Full Parameter 

8cCybersecurity Weaponization Restriction 

Adv 1     F-Cyber     64     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 2     F-Cyber     64     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 3     R→F     64     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 4     R→F     64     AdamW     4×10^-5     No Warmup     64     Full Parameter 









  
Harmful request refusal. For harmful request refusal, we choose the retain dataset 𝒟_retain to be the Magpie-Align instruction-tuning dataset <cit.>. We sample train-time adversaries that perform K=64 steps of SFT on rejected completions of the Anthropic-HH-RLHF dataset <cit.> and vary the learning rate within {2 × 10^-6, 2 × 10^-5, 4 × 10^-5}. We depict this list of adversaries in Table <ref>. 

While the rejected completions from Anthropic-HH-RLHF constitute the data used for sampled attacks from 𝒜_train for harmful request refusal, we compute the tamper-resistance loss ℒ_TR as follows. Since ℒ_TR is the DPO loss <cit.> in this setting, we use the base model weights θ as the reference model and sample harmful and benign completions from a modified test split of Anthropic-HH-RLHF <cit.>, where rejected completions are replaced with refusals <cit.>. To avoid keeping the base model weights in memory and speed up training, we precompute the reference model DPO log-probabilities for the full Anthropic-HH-RLHF dataset before training. To summarize, we have that the sampled train-time adversaries perform SFT on the rejected completions from Anthropic-HH-RLHF, and tamper-resistance DPO loss is computed on the corresponding modified refusal completions; the modified refusal completions in this setting correspond to 𝒟_TR in Section <ref>.

In practice, we perform an additional 100 steps of supervised fine-tuning on the Magpie-Align dataset to improve the benign capabilities performance of the refusal model in Table <ref>.





@lccccccc@
Train-time adversary red-teaming setups harmful request refusal. “A-HH-Rejected” in the Dataset column corresponds to the adversary dataset comprised of rejected completions from the Anthropic-HH-RLHF dataset.
 


Adversary     Dataset     Opt. Steps (K)     Optimizer     LR     LR Schedule     Batch Size     FT Paradigm 



8c
  – continued from previous page 

Adversary     Dataset     Opt. Steps (K)     Optimizer     LR     LR Schedule     Batch Size     FT Paradigm 



8rContinued on next page 
 



8cHarmful Request Refusal 

Adv 1     A-HH-Rejected     64     AdamW     2×10^-5     No Warmup      64     Full Parameter 

Adv 2     A-HH-Rejected     64     AdamW     2×10^-6     No Warmup     64     Full Parameter 

Adv 3     A-HH-Rejected     64     AdamW     4×10^-5     No Warmup     64     Full Parameter 













§ RED TEAMING DETAILS






 §.§ Weaponization Knowledge Restriction


In Table <ref>, we list all test-time adversary setups for recovering Biosecurity, Chemical Security, and Cybersecurity Weaponization knowledge. 

For Biosecurity, we examine post-attack forget accuracy after fine-tuning for 1,000 steps on three data distributions: the Pile-bio Forget set and the Retain-set used during Random Mapping and , and an OOD-Forget set mentioned in Appendix <ref>, which is unseen during tamper-resistance training. We conduct an additional attack using 5,000 optimization steps on this unseen forget distribution to compose our strongest adversary. We leverage a consistent setup for Chemical Security and Cybersecurity, substituting the Pile-bio Forget set for the respective Chem and Cyber Forget sets. We exclude the unseen forget distribution adversaries for a total of 24 attacks, as we were unable to obtain an equally high quality held-out dataset for Chemical Security and Cybersecurity.

For all subjects, we compare the effect of different optimizers (AdamW, Adadelta, and Stochastic Gradient Descent with Nesterov Momentum, Schedule Free AdamW) <cit.>, learning rates (2 × 10^-6, 2 × 10^-5, 4 × 10^-5), and learning rate scheduling techniques (SGDR and 30 steps of linear warmup) <cit.>. 

Similar to Table <ref>, we use F-Pile, F-Chem, F-Cyber, OOD-F, and Retain in the Dataset Column to denote the respective datasets described in Appendices  <ref> and <ref>. Separate from the R→F train-time adversary, R→F at test-time represents the combination of 0.4 × Opt. Steps of Forget-set SFT followed by 0.6 × Opt. Steps of Retain-set SFT. In practice, we found that this fraction composed a potent attack which we sought robustness against as described in Appendix <ref>.

In the Optimizer column, Schedule Free is an abbreviation of AdamW Schedule Free and SGD Nesterov is an abbreviation of SGD with Nesterov Momentum. In cases where the adversary used parameter-efficient fine-tuning (PEFT) via LoRA adapters <cit.>, we used a LoRA config with an attention dimension, or rank, of 16, a LoRA alpha value of 32, a LoRA dropout of 0.05, on target linear modules:      

{, , , ,, , }.













@lccccccc@
Test-time adversary red-teaming setups for weaponization knowledge restriction of Biosecurity, Chemical Security, and Cybersecurity.
 


Adversary     Dataset     Opt. Steps (K)     Optimizer     LR     LR Schedule     Batch Size     FT Paradigm 



8c
  – continued from previous page 

Adversary     Dataset     Opt. Steps (K)     Optimizer     LR     LR Schedule     Batch Size     FT Paradigm 



8rContinued on next page 
 



8cBiosecurity Weaponization Restriction 

Adv 1     OOD-F     5000     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 2     OOD-F     5000     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 3     F-Pile     1000     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 4     F-Pile     1000     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 5     Retain     1000     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 6     Retain     1000     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 7     OOD-F     1000     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 8     OOD-F     1000     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 9     R→F     1000     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 10     R→F     1000     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 11     F-Pile     1000     Adadelta     2×10^-5     No Warmup     64     Full Parameter 

Adv 12     F-Pile     1000     Adadelta     4×10^-5     No Warmup     64     Full Parameter 

Adv 13     F-Pile     1000     Schedule Free     2×10^-5     No Warmup     64     Full Parameter 

Adv 14     F-Pile     1000     Schedule Free     4×10^-5     No Warmup     64     Full Parameter 

Adv 15     F-Pile     1000     SGD Nesterov     2×10^-5     No Warmup     64     Full Parameter 

Adv 16     F-Pile     1000     SGD Nesterov     4×10^-5     No Warmup     64     Full Parameter 

Adv 17     F-Pile     1000     AdamW     2×10^-6     No Warmup     64     Full Parameter 

Adv 18     F-Pile     1000     AdamW     2×10^-6     30 Steps Warmup     64     Full Parameter 

Adv 19     F-Pile     1000     AdamW     2×10^-5     30 Steps Warmup     64     Full Parameter 

Adv 20     F-Pile     1000     AdamW     4×10^-5     30 Steps Warmup     64     Full Parameter 

Adv 21     F-Pile     1000     AdamW     2×10^-5     SGDR     64     Full Parameter 

Adv 22     F-Pile     1000     AdamW     4×10^-5     SGDR     64     Full Parameter 

Adv 23     F-Pile     1000     AdamW     2×10^-5     No Warmup     32     Full Parameter 

Adv 24     F-Pile     1000     AdamW     4×10^-5     No Warmup     32     Full Parameter 

Adv 25     F-Pile     1000     AdamW     2×10^-5     No Warmup     128     Full Parameter 

Adv 26     F-Pile     1000     AdamW     4×10^-5     No Warmup     128     Full Parameter 

Adv 27     F-Pile     1000     AdamW     2×10^-5     No Warmup     64     PEFT 

Adv 28     F-Pile     1000     AdamW     4×10^-5     No Warmup     64     PEFT 

8cChemical Security Weaponization 

Adv 1     F-Chem     1000     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 2     F-Chem     1000     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 3     Retain     1000     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 4     Retain     1000     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 5     R→F     1000     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 6     R→F     1000     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 7     F-Chem     1000     Adadelta     2×10^-5     No Warmup     64     Full Parameter 

Adv 8     F-Chem     1000     Adadelta     4×10^-5     No Warmup     64     Full Parameter 

Adv 9     F-Chem     1000     ScheduleFree     2×10^-5     No Warmup     64     Full Parameter 

Adv 10     F-Chem     1000     ScheduleFree     4×10^-5     No Warmup     64     Full Parameter 

Adv 11     F-Chem     1000     SGD Nesterov     2×10^-5     No Warmup     64     Full Parameter 

Adv 12     F-Chem     1000     SGD Nesterov     4×10^-5     No Warmup     64     Full Parameter 

Adv 13     F-Chem     1000     AdamW     2×10^-6     No Warmup     64     Full Parameter 

Adv 14     F-Chem     1000     AdamW     2×10^-6     30 Steps Warmup     64     Full Parameter 

Adv 15     F-Chem     1000     AdamW     2×10^-5     30 Steps Warmup     64     Full Parameter 

Adv 16     F-Chem     1000     AdamW     4×10^-5     30 Steps Warmup     64     Full Parameter 

Adv 17     F-Chem     1000     AdamW     2×10^-5     SGDR     64     Full Parameter 

Adv 18     F-Chem     1000     AdamW     4×10^-5     SGDR     64     Full Parameter 

Adv 19     F-Chem     1000     AdamW     2×10^-5     No Warmup     32     Full Parameter 

Adv 20     F-Chem     1000     AdamW     4×10^-5     No Warmup     32     Full Parameter 

Adv 21     F-Chem     1000     AdamW     2×10^-5     No Warmup     128     Full Parameter 

Adv 22     F-Chem     1000     AdamW     4×10^-5     No Warmup     128     Full Parameter 

Adv 23     F-Chem     1000     AdamW     2×10^-5     No Warmup     64     PEFT 

Adv 24     F-Chem     1000     AdamW     4×10^-5     No Warmup     64     PEFT 

8cCybersecurity Weaponization Restriction 

Adv 1     F-Cyber     1000     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 2     F-Cyber     1000     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 3     Retain     1000     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 4     Retain     1000     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 5     R→F     1000     AdamW     2×10^-5     No Warmup     64     Full Parameter 

Adv 6     R→F     1000     AdamW     4×10^-5     No Warmup     64     Full Parameter 

Adv 7     F-Cyber     1000     Adadelta     2×10^-5     No Warmup     64     Full Parameter 

Adv 8     F-Cyber     1000     Adadelta     4×10^-5     No Warmup     64     Full Parameter 

Adv 9     F-Cyber     1000     ScheduleFree     2×10^-5     No Warmup     64     Full Parameter 

Adv 10     F-Cyber     1000     ScheduleFree     4×10^-5     No Warmup     64     Full Parameter 

Adv 11     F-Cyber     1000     SGD Nesterov     2×10^-5     No Warmup     64     Full Parameter 

Adv 12     F-Cyber     1000     SGD Nesterov     4×10^-5     No Warmup     64     Full Parameter 

Adv 13     F-Cyber     1000     AdamW     2×10^-6     No Warmup     64     Full Parameter 

Adv 14     F-Cyber     1000     AdamW     2×10^-6     30 Steps Warmup     64     Full Parameter 

Adv 15     F-Cyber     1000     AdamW     2×10^-5     30 Steps Warmup     64     Full Parameter 

Adv 16     F-Cyber     1000     AdamW     4×10^-5     30 Steps Warmup     64     Full Parameter 

Adv 17     F-Cyber     1000     AdamW     2×10^-5     SGDR     64     Full Parameter 

Adv 18     F-Cyber     1000     AdamW     4×10^-5     SGDR     64     Full Parameter 

Adv 19     F-Cyber     1000     AdamW     2×10^-5     No Warmup     32     Full Parameter 

Adv 20     F-Cyber     1000     AdamW     4×10^-5     No Warmup     32     Full Parameter 

Adv 21     F-Cyber     1000     AdamW     2×10^-5     No Warmup     128     Full Parameter 

Adv 22     F-Cyber     1000     AdamW     4×10^-5     No Warmup     128     Full Parameter 

Adv 23     F-Cyber     1000     AdamW     2×10^-5     No Warmup     64     PEFT 

Adv 24     F-Cyber     1000     AdamW     4×10^-5     No Warmup     64     PEFT 









 §.§ Harmful Request Refusal




For the harmful request refusal setting, we conduct 5 test-time adversary attacks that perform SFT for 10 epochs on a held-out toxicity dataset called Toxic-DPO v0.2, on each of the settings in Table <ref>. The dataset contains 541 user-assistant chat interactions where the assistant complies with harmful instructions. 




@lccccccc@
Test-time adversary red-teaming setups for harmful request refusal. “ToxicDPO” in the Dataset column refers to the ToxicDPOv0.2 dataset containing harmful chat completions.
 


Adversary     Dataset     Epochs     Optimizer     LR     LR Schedule     Batch Size     FT Paradigm 



8c
  – continued from previous page 

Adversary     Dataset     Epochs     Optimizer     LR     LR Schedule     Batch Size     FT Paradigm 



8rContinued on next page 
 



8cHarmful Request Refusal 

Adv 1     ToxicDPO     10     AdamW     1×10^-5     10 Steps Warmup      32     Full Parameter 

Adv 2     ToxicDPO     10     AdamW     1×10^-5     No Warmup     32     Full Parameter 

Adv 3     ToxicDPO     10     AdamW     1×10^-5     No Warmup     16     Full Parameter 

Adv 4     ToxicDPO     10     AdamW     2×10^-5     No Warmup     32     Full Parameter 

Adv 5     ToxicDPO     10     AdamW     4×10^-5     No Warmup     32     Full Parameter 

















 §.§ Additional Weaponization Knowledge Restriction Red-Teaming Results

In Figure <ref>, we characterize the post-tampering resistance of TAR and the baseline safeguards with respect to the Chemical Security and Cybersecurity domains. Similar to the Biosecurity domain, baseline safeguards break under the majority of fine-tuning attacks, while resists attacks from 23 out of 24 Chemical Security SFT adversaries and 21 out of 24 Cybersecurity SFT adversaries. 












§ BASELINE DETAILS





 §.§ Weaponization Knowledge Restriction



  
Max Entropy.

Let K be the set of all token-wise output probability distributions returned by a model θ, where k ∈ K corresponds to every position in the sequence. We maximize the average entropy of these discrete distributions in K as follows:





    ℒ_Max Entropy = ∑_k ∈𝒦 p_k log(p_k)


This is equivalent to minimizing the average Kullback-Leibler (KL) divergence <cit.> between each k and the discrete uniform distribution u(x) over the vocabulary 𝒱. For Llama-3-8B-Instruct, |𝒱| = 128256. Thus, this objective is upper-bounded by: 
    h(x) = -log(u(x)) = log(|𝒱|) ≈ 11.76

where h(x) measures the Shannon information or self-information and log(x) has base e. We apply this objective for all elements in the Forget-set and perform standard cross-entropy on the Retain-set.



  
Min Posterior.
The goal of the Min Posterior objective is to assign lower probabilities to true forget-set labels, essentially minimizing -log(1 - P(label)). Let p_i be the probability assigned to the true label for token i and 𝒱 be the model's vocabulary distribution. We define the Min Posterior objective function as follows:

    ℒ_Min Posterior = -1/|𝒱|∑_i ∈𝒱log(1 - p_i + ϵ) ·𝕀[log(p_i) ≥τ]
 
where τ is the threshold for masking out target label logits (which we set to the negative maximum entropy of the vocabulary distribution, -log|𝒱|) and 𝕀[·] is the corresponding indicator function (1 if the condition is true, 0 otherwise). We include an optional ϵ = 1×10^-12 to help with numerical stability. Similar to the Max Entropy objective, we apply this objective for all elements in the Forget-set and perform standard cross-entropy on the Retain-set.



  
RMU. We adapt RMU's implementation from <cit.> with a learning rate of 5×10^-5 and 250 unlearning steps. We use the released WMDP's unlearning datasets for Biosecurity (Bio) and Cybersecurity (Cyber) unlearning, and our private hazardous chemistry dataset for Chemical Security (Chem) unlearning. We use unlearning coefficients of 20, 30, and 50 for Bio, Cyber, and Chem respectively. We use a retain coefficient of 700 on Wikitext <cit.>.



  
LLMU.

We use a modified version of LLMU from <cit.>. Instead of computing the KL divergence to regularize retain-set logits towards the base frozen model, we employ a standard cross-entropy loss. This modification allows for memory-efficient execution on our hardware while maintaining comparable performance.




  
Hyperparameter tuning. Besides RMU, all baseline hyperparameters were chosen after a grid search across learning rates {3×10^-6, 5×10^-6, 8×10^-6, 1×10^-5}, optimization step count {600, 1000}, and warmup steps {0, 100}. We found that 600 optimization steps using the AdamW Schedule Free optimizer, at a learning rate of 1×10^-5, with 100 steps of linear warmup, and an effective batch size of 64 produced the best performance. For the Max Entropy, Min Posterior, and LLMU baselines, we train on the three corresponding forget datasets discussed in <ref>. For these baselines, we modify our Biosecurity forget corpus to be a mixture of the Pile-bio and Camel-bio Forget corpora. We use the Pile-bio Retain-set as a global Retain-set for baseline training.



 §.§ Harmful Request Refusal




  
Representation Rerouting. We use the Llama-3-8B-Instruct RR model from <cit.>, which uses a cosine distance loss to push representations for harmful inputs to become orthogonal to those of the base Llama-3-8B-Instruct model.



  
R2D2. We use the R2D2 model run on Zephyr-7B directly from <cit.>, which performs adversarial training against GCG attacks to increase jailbreak robustness.



  
RepNoise. We the RepNoise model run on Llama-2-7B directly from <cit.>, which uses a distributional loss to push representations for harmful inputs toward Gaussian noise.








 §.§ Additional Baseline Comparisons






  
MLAC-AR. Meta-Learned Adversarial Censoring (MLAC) <cit.> was originally proposed to prevent BERT-style models from learning binary classification for gender bias data. Since the approach is not immediately applicable to LLMs, we extend MLAC in a variant we call autoregressive MLAC (MLAC-AR). Since MLAC in its original formulation calls for “task-blocking” via negating the adversary's loss during the inner loop of meta-learning, we implement this by negating the cross-entropy loss of an LLM fine-tuning adversary. However, we found that this approach diverges in performance across a variety of hyperparameters, and opted to further improve performance of the MLAC-AR baseline by clamping the maximum cross-entropy loss at the value of the maximum entropy of the output vocabulary distribution, log(). We show results in Table <ref>, finding that MLAC-AR does not maintain sufficient benign capabilities performance nor uniform tamper-resistance across weaponization domains.



  
SOPHON-AR. In concurrent work, SOPHON <cit.> was introduced to prevent small diffusion models and image classifiers from learning specific data distributions. Similarly to MLAC-AR, we extend SOPHON to LLMs via SOPHON-AR, using the alternating retain loss and fine-tuning suppression loss formulation that the authors propose. Furthermore, we adapt the inverse cross-entropy loss from <cit.> , which aims to boost convergence of the fine-tuning suppression process. We find in practice that despite heavy tuning, SOPHON-AR does not converge well enough to yield a usable Pre-Attack model in Table <ref>.




 Paper Title: Improving Text Embeddings for Smaller Language Models Using Contrastive Fine-tuning.


[
    *
    August 5, 2024
==================

§ ABSTRACT 

While Large Language Models show remarkable performance in natural language understanding, their resource-intensive nature makes them less accessible. In contrast, smaller language models such as MiniCPM offer more sustainable scalability, but often underperform without specialized optimization. In this paper, we explore the enhancement of smaller language models through the improvement of their text embeddings. We select three language models, MiniCPM, Phi-2, and Gemma, to conduct contrastive fine-tuning on the NLI dataset. Our results demonstrate that this fine-tuning method enhances the quality of text embeddings for all three models across various benchmarks, with MiniCPM showing the most significant improvements of an average 56.33% performance gain. The contrastive fine-tuning code is publicly available at <https://github.com/trapoom555/Language-Model-STS-CFT>.





§ INTRODUCTION


Text embeddings are vector representations of text data that encodes 
semantic information, enabling machines to understand and process natural language. These embeddings are crucial for a variety of tasks including document classification, 

semantic similarity matching, 
and information retrieval. 


With the recent advancements of Large Language Models (LLMs),

LLMs such as GPT-4 <cit.>, LLaMA <cit.>, and Mistral <cit.> have demonstrated robust natural language understanding capabilities due to their large-scale training. Conversely, smaller models like Gemma <cit.>, Phi <cit.>, and MiniCPM <cit.>, while less resource-intensive, often lag in performance without specific optimizations.

This project is motivated by the lack of research focused on enhancing the text embedding capabilities of relatively smaller models. Notably, MiniCPM, a smaller-scale language model, underperforms in generating effective text embeddings without further fine-tuning. Addressing this gap, we aim to conduct experiments to improve the quality of text embeddings generated by MiniCPM, thereby making it a viable option for resource-constrained applications. This project could pave the way for using small scale models on the text embedding task. Furthermore, we compare MiniCPM's performance among other small language models including Gemma and Phi-2 after fine-tuning with the same dataset. 



§ RELATED WORKS




  
Text Embeddings Text embeddings are vector representations of text in a low-dimensional space. These embeddings are designed to encapsulate the semantic meaning of the text, facilitating various downstream tasks including information retrieval, document classification, and similarity matching. Traditional models such as SBERT <cit.> and Sentence T5 <cit.> have aimed to provide a universal framework for encoding the semantic significance of text across different tasks and domains. Moreover, methods such as Contriever <cit.>, E5 <cit.>, and SGPT <cit.> have implemented a multi-stage training strategy to enhance the effectiveness of text embeddings. To assess the performance of text embeddings within the proposed model, the benchmark MTEB <cit.> is utilized. This benchmark includes novel tasks to evaluate the text embedding quality of the models and their abilities to generalize across different domains. 





  
Contrastive Representation Learning Contrastive Representation Learning aims to learn effective representation by pulling semantically close neighbors
together and pushing apart non-neighbors <cit.>. Early works learn such representation by using contrastive loss <cit.> and triplet loss <cit.>, where only one positive and one negative sample are involved. On top of that, multi-class N-pair loss <cit.> and InfoNCE loss <cit.> include multiple positive and negative pairs in a batch to improve the semantic representation of the embeddings.



  
Lightweight LLMs Despite the remarkable performance of LLMs in text generation tasks, the cost of training, serving and maintaining such large scale model is huge. Therefore, several lightweight LLMs within the range of 1 to 3 billion parameters have been proposed to tackle this issue.  Notable examples include the Phi series <cit.>, Gemma <cit.> and MiniCPM <cit.>. These lightweight language models require fewer resources, thereby reducing the barriers to entry for conducting studies with smaller-scale models.



  
LLM Fine-tuning

In the recent year, Pre-training and Fine-tuning Paradigm has been successfully proven to be better than training from scratch. The foundational work, Bidirectional Encoder Representations from Transformers (BERT) <cit.> established the pre-training and fine-tuning paradigm, where models are first pre-trained on a large corpus of text and then fine-tuned on specific downstream tasks. This approach has become a cornerstone for many subsequent models, including GPT-3 <cit.>, which demonstrated that increasing model size and the amount of pre-training data can lead to significant improvements in performance across a wide range of tasks. Recent research has focused on making fine-tuning more efficient, particularly for large-scale models. Techniques such as Adapter modules, proposed by <cit.>, involve inserting small trainable layers within the pre-trained model, allowing for task-specific adaptation without modifying the entire model's parameters. This approach significantly reduces the computational cost of fine-tuning.
Another notable technique is Low-Rank Adaptation (LoRA), introduced by <cit.>, which adapts the weights of the model using low-rank matrix factorization. This method achieves substantial parameter efficiency, enabling the fine-tuning of large models with a fraction of the computational resources typically required.





§ METHODOLOGY


Our approach aims to address the problem of Semantic Textual Similarity (STS) in the English language by employing smaller language models, thereby ensuring that our solution is both efficient and scalable. 
To enhance the text embedding capabilities of these smaller models, we utilize a contrastive fine-tuning approach. Contrastive fine-tuning involves training the model to distinguish between similar and dissimilar pairs of text. This method helps the model to generate more accurate and contextually relevant embeddings, which are essential for assessing semantic similarity. The details of contrastive fine-tuning are described in section <ref>.

Additionally, we adopt a parameter-efficient fine-tuning technique. This technique is designed to achieve optimal performance while minimizing the computational resources required. By using the low-rank adaptation (LoRA) <cit.> method, we ensure that our approach remains computationally feasible even with limited hardware resources. This is particularly important for fine-tuning the models in the scenario where computational efficiency is a key concern.





 §.§ Dataset
 
The processed NLI dataset <cit.> is adopted as our training dataset. There are approximately 275k samples in the dataset. Each entry includes a premise x_i, along with its corresponding entailment x_i^+ and contradiction x_i^-, thus forming a triplet (x_i, x_i^+, x_i^-)



 §.§ Language Model Choices


We conduct our experiments on language models with fewer parameters up to 2B, including Gemma <cit.>, Phi-2 <cit.>, and MiniCPM <cit.> to explore their capabilities on the text embedding task.



 §.§ Contrastive Fine-tuning
 


In this study, we leverage the language understanding capabilities of language models (LMs) and enhance their text embedding quality using a contrastive fine-tuning paradigm. To achieve higher text embedding quality with limited computational resources, we adopt a parameter-efficient fine-tuning technique, LoRA <cit.>, as our fine-tuning method. The details of how text embeddings are extracted from the  Language Model (LM) and the training objective are described in the following sections.



  
Embedding Vector Extraction
Given a pretrained language model and a prompt, we augment the prompt by appending the <EOS> token. Subsequently, we input this modified prompt into the language model. The embedding vector can then be acquired by extracting the vector corresponding to the final <EOS> token from the last layer.



  
Training Objective
We utilize the standard InfoNCE objective <cit.> with in-batch negatives and hard negatives. The training objective can be described by the following expression.


    min  - loge^sim(h_i, h_i^+) / τ/∑_j=1^N ( e^sim(h_i, h_j^+) / τ+ e^sim(h_i, h_j^-) / τ)

where h_i denotes an embedding vector of a premise x_i, τ denotes a temperature and sim(h_i, h_i^+) computes the cosine similarity between embedding vectors h_i and h_i^+.
 






§ EXPERIMENTS

Given a pair of sentences, the STS task aims to measure the similarity score of the embeddings embeded by the models where higher scores signify greater similarity. Specifically, we measure the distance between embeddings using cosine similarity, then calculate the correlation with ground truth similarities using Spearman correlations. 




We compare the performance of MiniCPM and baseline models such as Gemma and Phi-2 on 9 benchmarks, namely STS12 <cit.>, STS13 <cit.>, STS14[<https://alt.qcri.org/semeval2014/tas
k10/>], STS15[<https://alt.qcri.org/semeval2015/tas
k2/>], STS16[<https://alt.qcri.org/semeval2016/tas
k1/>], STS17[<https://alt.qcri.org/semeval2017/tas
k1/>], STSBenchmark, BIOSSES[<https://tabilab.cmpe.boun.edu.tr/BIO
SSES/DataSet.html>] and SICK-R <cit.>. These datasets consist of pairs of sentences with similarity scores from 0 to 5. To ensure fairness, both MiniCPM and the baseline models are fine-tuned on the same processed NLI dataset.



  
STS12, STS13, STS14, STS15, STS16, STS17, STSBenchmark This set of STS  benchmarks incorporate sentences derived from various sources including image captions, news headlines, and user forums. These datasets range from 1,000 to 20,000 sentences. 





  
BIOSSES The BIOSSES benchmark contains 100 sentence pairs from the biomedical field.



  
SICK-R The Sentences Involving Compositional Knowledge (SICK) dataset comprises a substantial corpus of 100,000 sentence pairs, each characterized by its lexical, syntactic, and semantic richness.





 §.§ Main Experiment Results

Table <ref> presents the evaluation results of three models: Gemma, Phi-2, and MiniCPM across all 9 benchmarks, using Spearman correlations of cosine similarities post-supervised fine-tuning (SFT). The results demonstrate that MiniCPM consistently outperforms the other models on all benchmarks.

Specifically, MiniCPM achieves the highest Spearman correlations across all datasets, including notable results on STS12 (76.38%), STS13 (87.61%), and STS17 (89.96%). Gemma showed competitive performance but trailed slightly behind MiniCPM in every benchmark, with its highest correlation observed on STS17 (88.22%). Phi-2, on the other hand, exhibited the lowest performance among the three models, with its best result on STS17 (80.20%).

The overall performance metrics further illustrate this trend, with MiniCPM achieving an average Spearman correlation of 83.84%± 4.27, which is superior to Gemma's 81.56%± 4.83 and Phi-2's 70.89%± 6.91. These results indicate that while all models benefited from the fine-tuning process, MiniCPM's architecture and training regimen enabled it to achieve superior alignment with human judgment across diverse datasets.

In conclusion, the evaluation underscores MiniCPM's robustness and effectiveness in capturing semantic similarities, making it a reliable choice for tasks requiring nuanced understanding of textual data.

The fine-tuned versions of these three models have been published and are available for access here[<https://huggingface.co/collections/trapoom555/small-lms-text-embedding-663b3ec87527788a577f6852>].





 §.§ Ablation Studies




  §.§.§ Model performance before SFT

In this study, we aim to compare the performance of Gemma, Phi-2 and MiniCPM
before and after applying SFT (fine-tuning). The results are detailed in Tables <ref> and <ref>.

Table <ref> presents the spearman correlations of cosine similarities for the models prior to SFT across various benchmarks. Gemma consistently outperforms Phi-2 and MiniCPM on all benchmarks, achieving the highest overall correlation of 
56.63%± 7.89. Phi-2 and MiniCPM follow with overall correlations of 32.21%± 11.40 and 27.51%±12.76, respectively.

Comparing the results before and after SFT, we observe significant performance improvements on all three models. Notably, MiniCPM demonstrates the most substantial improvement, increasing its overall correlation by approximately 56 points. This indicates that SFT has a pronounced positive impact on MiniCPM’s performance.

In summary, while all models benefit from SFT, 

MiniCPM stands out as the top performer after SFT.





  §.§.§ Impact of learning rate

In this study, we aim to study the impact of learning rate during training. Results from Table <ref> shows that learning rate 5e-5 achieves the best result among the others, whereas learning rate 5e-3 achieves the lowest spearman correlations score, indicating the model suffers from training instability and is underfitting.





  §.§.§ How does prompting affect model performance?







In our main experiments shown in Tables <ref> and <ref>, we tested MiniCPM's embeddings without any additional prompts. 
Yet, it is known that employing prompting techniques may further leverage a language model's performance.
Hence, we conducted an ablation study on MiniCPM with prompt-tuning.

We designed 3 different prompts:



    
  * "This sentence: {original_sentence} means in one word: "
    
  * "This sentence {original_sentence} means: "
    
  * "{original_sentence} is: "


During testing, {original_sentence} is replaced by the sentence from the current test case.

We tested all 3 prompts on both our fine-tuned and original MiniCPM models. The results are shown in Table <ref>.



As the results show, the original MiniCPM model performs best with Prompt 1, reaching a +7.25% gain in average performance. The original model experiences performance enhancements with both Prompt 1 and Prompt 3. 
In contrast, our fine-tuned MiniCPM model only achieves a minor +0.48% performance gain with Prompt 3, and is negatively impacted by both Prompt 1 and Prompt 2. 
Such results suggest that prompt-tuning is effective in certain conditions but harmful in others, and it is more effective to the original MiniCPM model than to our fine-tuned MiniCPM model. 
We hypothesize that this may be due to our fine-tuned model's preference towards the distribution of the original sentences instead of sentences augmented with prompts, since the model has been exposed to more sentences of the original format during fine-tuning. 




  §.§.§ Training data efficiency




In this study, we aim to study training data efficiency by measuring how much training data is required to achieve convergences. To do so, we measure evaluate the every 20 checkpoints until the Spearman correlation coverged since lesser training step equivalent to lesser training data used. The results are recorded in Figure <ref>. 































Our results show that after 200 training steps, MiniCPM has already earned a average performance gain of +50% over our 9 test sets, indicating that it underwent the majority of its capability growth during the first 200 steps.
Furthermore, the model begins to converge at around 200 steps, reaching an average performance of 82.31% at Checkpoint-500. 
This shows that MiniCPM demonstrates a relatively high training efficiency with great convergence speed.



  §.§.§ Impact of hard negatives penalty

In the context of contrastive learning, the selection of hard negatives plays a critical role in enhancing the effectiveness of the training process. To study the impact of the hard negatives on the performance of MiniCPM after SFT, we remove the penalization on the hard negative samples by modifying the objective function in Equation <ref> to become:


    min  - loge^sim(h_i, h_i^+) / τ/∑_j=1^N ( e^sim(h_i, h_j^+) / τ)


Table <ref> compares the Spearman correlations of cosine similarities of the MiniCPM model with and without penalization on the hard negative samples. The results indicate that incorporating penalization generally improves performance across most benchmarks. Specifically, the MiniCPM model shows an increase in performance on all benchmarks except STS17 when hard negative samples are penalized.

For instance, the Spearman correlation for STS13 improves from 85.98% to 87.61%, which is a notable increase. Similarly, STS15 sees a rise from 85.66% to 87.33%, and STS14 improves from 78.52% to 81.55%. These results suggest that penalizing hard negative samples helps the model better distinguish between similar and dissimilar pairs, leading to higher correlation scores.

However, there is a slight decrease in performance for STS17, where the correlation drops from 90.23% to 89.96%. This anomaly might indicate that for some datasets, the penalization might not always lead to improvements, potentially due to the nature of the data or the specific characteristics of the benchmark.

Overall, the average performance across all benchmarks increases from 81.97% to 83.84%. This demonstrates that penalizing hard negative samples not only boosts the average performance but also leads to more consistent results, as evidenced by the lower standard deviation.







§ CONCLUSION


In this project, we enhanced the text embedding capabilities of MiniCPM through contrastive fine-tuning using the NLI dataset. 
Our results demonstrate that MiniCPM achieved a significant performance gain of 56.33% and outperformed 2 other models, Gemma and Phi-2, across 9 STS datasets. 
Additionally, we conducted multiple ablation studies to delve deeper into our method, examining aspects such as prompt-tuning, training efficiency, and the effects of incorporating a hard negatives penalty in the objective function. 
Our research contributes to the enhancement of text embedding qualities in smaller-scale LLMs, making them more robust and reliable for further applications.



§ ACKNOWLEDGEMENT


We would like to formally acknowledge the Natural Language Processing Spring 2024 course for providing us with the foundational knowledge and the opportunity to undertake this project. We also extend our sincere gratitude to Professor Huaping Liu and the Tsinghua Knowledge Engineering Group for supplying the essential computational resources for our work.



acl_natbib












§ FINE-TUNING DETAILS


In contrastive learning, a large batch size is required to improve training stability. In this work, we aim to enhance text embedding performance with limited computational resources. Therefore, we implement several training techniques to achieve this goal. We employ LoRA <cit.>, a parameter-efficient fine-tuning method, to significantly decrease the GPU VRAM requirements for a single batch. Consequently, more batches can fit into a single GPU. To further reduce GPU VRAM demand, we adopt the bf16 mixed-precision training technique, which also increases training speed while maintaining numerical stability. Additionally, we leverage multi-GPU processing to amplify batch size across GPUs using the distributed data parallelism (DDP) technique. Furthermore, we incorporate a learning rate scheduler with CosineAnnealingLR <cit.> and learning rate warm-up to optimize the training process.

All fine-tuning details listed in Table <ref>. With these settings, the model can be effectively fine-tuned using limited resources, ensuring efficiency and cost-effectiveness. The entire fine-tuning process involves 13,780 optimization steps to cover the entire dataset.

Figure <ref> illustrates the fine-tuning loss curves of the three selected models: Gemma, Phi-2, and MiniCPM. These curves show that the loss converges very quickly at the beginning of the fine-tuning process, indicating a rapid improvement in model accuracy during the initial stages. We observed this phenomenon and further examined the training data efficiency in the ablation study <ref>. 






 Paper Title: Improving Faithfulness of Large Language Models in Summarization via Sliding Generation and Self-Consistency.







§ INTRODUCTION


Abstractive summarization aims to generate summaries that are fluent, informative, and faithful to the source articles. Benefiting from the popularity and development of large language models (LLMs), abstractive summarization has achieved remarkable progress in fluency and coherence <cit.>. However, LLMs have propensity to generate content that contradicts or is not present in the source article <cit.>, which is commonly referred to as hallucination. Alleviating the hallucination of LLMs is a critical challenge for their reliability in real-world applications.











Many works <cit.> explore the mechanisms why LLMs exhibit hallucinations. 
<cit.> observe that the performance of LLMs significantly decreases as the length of input contexts increases, resulting in the hallucination phenomenon of LLMs being particularly serious in long document summarization. Furthermore, LLMs are sensitive to the order of context and are more likely to select information presented first or last, even for short contexts <cit.>. That is, the summaries generated by LLMs contain more content that occurs at the beginning and end of the source article, which has a detrimental effect on the summary quality of the entire article.



Recent works attempt to leverage post-processing models and the CoT (Chain of Thought) technique to improve factual consistency of summarization. Some works <cit.> employ LLMs as critic and editor models, the critic model generates editing instructions for the initial summary, and the editor model corrects factual errors by following the instructions. <cit.> propose a CoT-based method called SumCoT to elicit LLMs to generate summaries step by step. However, they do not mitigate position bias and performance degradation in long context scenarios. 

In this paper, we propose SliSum, a novel summary generation strategy that improves the faithfulness of LLMs in both short and long text summarization by sliding generation and self-consistency. As shown in Figure <ref>, our approach consists of the following three steps: (1) Sliding Generation: SliSum uses LLM to generate local summaries for overlapping windows, and local summaries contain multiple statements about the same event that may have contradictions (see Table <ref>).  
(2) Filtration: According to the principle of self-consistency, the statements generated more times by LLMs are more faithful and important to the source article <cit.>. 
Therefore, we obtain statements about the same event by lexical clustering over sentences of all local summaries and then filter out small sentence clusters and outliers so that the global summary contains only relatively more important information of the source article. (3) Aggregation: SliSum employs LLM to detect whether there are contradictions in statements about the same event and classify them into different categories based on their semantics. Finally, SliSum uses a majority voting algorithm to select the statements with the most proportion, and concatenates them to form a more faithful summary of the entire article. SliSum brings three major benefits: (1) Sliding window provides LLMs with more diverse and adequate information by splitting articles in an overlapping manner; (2) The filtration and aggregation based on self-consistency ingeniously mitigate the self-contradiction problem and further leverage the potential of LLMs to improve the faithfulness of them; (3) The combination of sliding windows and self-consistency impels LLMs to process the entire article more fairly and faithfully. Therefore, SliSum is capable of improving the faithfulness of LLMs in summarization without external resources and additional fine-tuning. 





We evaluate the effectiveness of SliSum applied to three advanced LLMs, LLaMA-2-13B <cit.>, Claude-2 <cit.> and GPT-3.5 <cit.>, on four popular summarization datasets. Extensive experiments have shown that SliSum significantly improves the faithfulness of three LLMs on the short news datasets CNN/DM and XSum, and the long scientific papers datasets arXiv and PubMed, respectively, while without sacrificing their fluency and informativeness. Besides, we conduct ablation studies to further investigate why SliSum works. We also perform quantitative investigations on hyperparameters in SliSum. Our contributions are summarized as follows.


    
  * We propose a novel summary generation architecture, SliSum, that improves faithfulness of LLMs by sliding windows and self-consistency without additional resources and fine-tuning. To the best of our knowledge, we are the first to apply overlapping context windows into LLMs for abstractive summarization.
    
  * We demonstrate that SliSum uniformly improves factual consistency of summaries generated by diverse LLMs while maintaining their fluency and informativeness, more importantly, SliSum is applicable to text of various lengths and styles.
    
  * We conduct extensive qualitative and quantitative experiments to validate the effectiveness of sliding generation and aggregation based on self-consistency and impacts of hyperparameters in SliSum on performance.






§ RELATED WORKS




  
Factual Consistency of Summarization The factual consistency in abstractive summarization has received increasing attention recently. Existing work has proposed various methods to improve the factual consistency, such as contrastive learning <cit.>, adversarial learning <cit.>, textual entailment <cit.> and post-editing <cit.>. However, these methods can not be directly applied in long document summarization due to the difficulty of modeling long texts accurately. Although recent studies have addressed this problem by leveraging Graph Neural Networks <cit.>, reinforcement learning <cit.> and structure information <cit.>, they do not improve the faithfulness of long document summarization systems. There are currently few works that focus on improving factual consistency of long document summarization. In contrast, we propose a unified architecture that consistently improves factual consistency of both short and long text summarization.





  
Mitigation of LLM Hallucination Recent studies have made several attempts to mitigate the hallucination of LLMs, including retrieval-augmented generation <cit.>, post-processing models <cit.>, prompt engineering <cit.> and self-supervised learning <cit.>. However, most of these works require training additional models or external knowledge, or lack sufficient evaluation on long contexts tasks. For example, LLM-Augmenter <cit.> acquires evidence by retrieving external knowledge for the LLM to generate candidate responses grounded in evidence. 
<cit.> propose a post-processing method that improves factuality of language models using multiagent debate. SELF-FAMILIARITY <cit.> has revealed excellent performance in short context tasks, but its advancement has not been demonstrated on long-context datasets. Different from previous works, SliSum improves faithfulness of LLMs in long context scenarios without additional fine-tuning and external resources.



  
Long Context for LLMs With the application fields of LLMs continue to expand, boosting the performance of LLMs in long context scenarios has aroused a surge of interest. <cit.> propose a decoding method, LM-Infinite, to maintain fluency and generation quality of LLMs on long sequences, which can only be applied to LLMs using relative positional encodings, but our approach is applicable to diverse LLMs, even black-box models, because it does not modify the original structure and implementation of the models. <cit.> and <cit.> compress long context by removing redundancy to enhance LLMs' perception of the key information, but there is no guarantee that compressed contexts contain faithful information to original contexts. 
<cit.> propose a parallel context windows method that LLMs process contexts located in these windows separately to mitigate the negative impact of long contexts on performance. However, parallel windows separate the semantic relationships within the article, which inevitably affects LLMs’ understanding and extraction of global information. 
We are the first to introduce overlapping sliding windows into LLMs for abstractive summarization tasks. Unlike traditional parallel windows, sliding windows overlap each other so that each segment of the source article can be located at the beginning of a certain window, allowing them to overcome their preference for position and process the entire context more fairly and faithfully. 
In previous works, the sliding window method is only applied to extractive  <cit.> and query-focused <cit.> summarization, because they do not solve the self-contradiction problem <cit.> between local summaries. SliSum addresses this problem by combining sliding windows and self-consistency and achieves better performance than existing methods.



§ APPROACH




 §.§ Sliding Generation




  
Sliding Window SliSum first divide the source article A into a list of sentences [C_1, C_2, ⋯, C_n]. The sliding window with predefined window size initially consists of several consecutive sentences [C_1, ⋯, C_t] at the beginning of the article. The number of words in the window only needs to be approximately equal to the window size. Figure <ref> shows that LLM generates a local summary for the content in the sliding window. SliSum restricts the attention of LLM within a window, alleviating the distraction issue caused by long contexts <cit.>. 
Then the sliding window moves downward by a distance of step size and LLM generates a local summary for current window again. Similarly, the distance moved does not need to be strictly equal to step size. That is, the starting index of the window advances a few sentences. Importantly, SliSum processes the source article at the sentence level. Finally, LLM iteratively generates summaries for the content in the current window until the sliding window traverses the entire article. Given an article A = [C_1, C_2, ⋯, C_n], the process of sliding generation can be described as follows:


    S_1    = SUM (W_1) = SUM ([C_p_1, ⋯, C_q_1])
    S_2    = SUM (W_2) = SUM ([C_p_2, ⋯, C_q_2])
       ⋮
    S_m    = SUM (W_m) = SUM ([C_p_m, ⋯, C_q_m])


where SUM(W_i) represents LLM summarizes the content in the window W_i, W_i consists of from p_i-th to q_i-th sentences. p_1  = 1, q_m = n, and


    p_i ≤ p_i + 1≤ q_i,    i = 1, ⋯, m - 1


Each text segment except the beginning and end can locate in different positions in the context window. SliSum balances the position distribution in the context window of text segments, eliminating the position preference of LLMs for source article.



  
Process Repeatedly Eventually SliSum will obtain a concatenation of summaries [S_1, S_2, ⋯, S_m] by sliding generation. In the sliding window method, step size must be less than or equal to window size, otherwise there will be gaps between windows. Figure <ref> shows that LLM processes the same part of the source article multiple times due to overlaps between windows. Generally, given window size L_w, step size L_s (L_s ≤ L_w) and an article whose length L_a is far larger than the window size, i.e, L_a ≫ L_w, the minimum number of times the middle part of the article is summarized by LLM:


    K = ⌊L_w/L_s⌋


It can be easily seen that if L_w is an integer multiple of L_s in sliding window method, LLM will read the middle part of the source article the same number of times, so we set L_w = K L_s in SliSum. In particular, for several windows at the beginning and end, we repeatedly generate local summaries for them so that all parts of the source article are summarized K times. 



 §.§ Events Filtering




  
Lexical Clustering The event is usually composed of entity, behavior, reason, result and other elements, so statements about different events have their own characteristic words that can be used to distinguish them. We observe that the statements generated by LLMs for the same event are usually lexically similar. Consequently, we can obtain a set of sentences about the same event by lexical similarity clustering. As shown in Figure <ref>, SliSum divides all summaries into sentences, then uses DBSCAN <cit.> algorithm to cluster them based on lexical similarity. Specifically, we use ROUGE-1 <cit.> F1 score to define the distance between two sentences C_1 and C_2:


    dist(C_1, C_2) = 1 - ℛ(C_1, C_2)


where ℛ(·) is ROUGE-1 F1 score, and is computed as follow:


    ℛ(C_1, C_2) = 2 * number of overlap words/length of  C_1 + length of  C_2


Obviously, ℛ(C_1, C_2) ∈ [0, 1]. There are two important parameters in the DBSCAN algorithm: distance threshold ε and minimum number of points (MinPts) in a cluster. Formally, let Ĉ represents a sentence, if exists a set of sentences (including Ĉ itself) ℂ = {C_1, C_2, ⋯, C_N} and the number of sentences |ℂ| ≥MinPts, every sentence C ∈ℂ satisfies


    dist(Ĉ, C) ≤ε


ℂ will be considered a cluster. A lower MinPts induces the algorithm to construct more clusters, while a higher MinPts will ensure more robust and consistent clusters. 




  
Filtering Noise Intuitively, if an event appears in the local summary of a window, it means that the event is important to the content in the window. We notice that some sentences are only important in minority windows and should not appear in the overall summary. Furthermore, LLMs occasionally generate few hallucination statements that deviate from the source article. These statements tamper with information in the source article or add information that cannot be inferred from the source article, thus they generally have low lexical similarity to factually consistent statements. These undesirable statements are treated as outliers and noise in DBSCAN algorithm. Due to MinPts determines the minimum number of points in a cluster, so we can filter out unimportant and hallucination statements by using an appropriate MinPts. 

As mentioned in Section <ref>, SliSum generate K summaries for the content of the source article by sliding generation, that is, sentences about the same event occur at most K times. Under normal circumstances, a cluster contains at most K sentences. In order to generate more concise and relative summaries, we select sentences that exist in as least MinPts local summaries, i.e, we only retains clusters whose size exceeds MinPts. Therefore, we can adjust MinPts in the range of [1, K] to filter noise and hallucination as well as improve self-consistency.




 §.§ Contradictions Detection and Sentences Aggregation




  
Sentences Selection As shown in Table <ref> and Figure <ref>, there may be contradictory statements in sentences about the same event. Due to LLMs have ability to detect self-contradictory statements without relying on additional knowledge <cit.>, SliSum first manipulates the LLM used in the sliding generation to divide sentences into different categories according to their semantics. Generally, the more times a certain response is output by LLMs, the higher the probability that the response is desirable <cit.>, since self-consistency reflects factual consistency. Consequently, SliSum can utilize majority voting algorithm to select sentences that are faithful to the source article. 

After distinguishing sentences that are high lexically similar but semantically different, sentences of the same category state the same fact. SliSum selects the sentence with the largest proportion to output to the final summary. When there are clusters of sentences with the same proportion or when it is necessary to choose sentence from a certain  cluster, the relatively last generated sentence is selected by SliSum. Because the last generated sentence is located at the beginning of the corresponding window, and LLMs have the highest factual consistency for the information occurs at the beginning of the input context <cit.>. For example, an event has five related statements [D_1, ⋯, D_5] generated sequentially, and LLM divides them into [D_2], [D_1, D_4], [D_3, D_5], SliSum selects D_5 as the most faithful statement to form the summary of the entire article.



  
Sentences Integration SliSum finally integrates all selected sentences to output the final summary of the entire article. SliSum uses LLM to generate connectives and concatenate these sentences in order, so that the order of the final summary is consistent with the source article. We determine the order of the sentences according to the order of the events they describe. In the integration stage, SliSum do not need to change any information of the selected sentences.


 


§ EXPERIMENTAL SETUP






 §.§ Datasets


We evaluate the performance of SliSum for short and long text summarization on four popular benchmark datasets: CNN/DM <cit.> and XSum <cit.> are two widely-used short news summarization datasets. PubMed and arXiv <cit.> are two scientific paper datasets for long document summarization, which are much longer than the common news articles. PubMed contains academic papers from the biotechnology domain, while arXiv contains papers from different scientific domains. 
Limited by the maximum number of requests and time cost, we randomly select 100 articles from test set of each dataset to construct our test set, respectively.









 §.§ Baselines


To validate the effectiveness of our proposed approach, we apply SliSum to three state-of-the-art instruction-tuned LLMs, including LLaMA-2 13B Chat[https://ai.meta.com/resources/models-and-libraries/llama/https://ai.meta.com/resources/models-and-libraries/llama/], Claude-2[https://www.anthropic.com/index/claude-2https://www.anthropic.com/index/claude-2] and GPT-3.5-Turbo-0613[https://openai.com/blog/chatgpthttps://openai.com/blog/chatgpt]. 







We also compare SliSum with recent faithfulness enhancement summarization models on CNN/DM and XSum, such as CLIFF <cit.> and FES <cit.>. We select GPT-3.5 with SumCoT <cit.> as LLM news summarization baseline. For arXiv and PubMed, we compare SliSum with two long document summarization baselines: FactorSum <cit.>, a factorized energy-based abstractive model that improves the performance and applicability by separate budget decisions from selecting important content in the document, and Lodoss <cit.>, an extractive architecture that learns robust sentence representations by performing summarization and segmentation simultaneously. We compare SliSum with Refine <cit.>, a LLM-based long document summarization method that joins the summary of the previous text segment with the next segment, and then iteratively generate summaries for them.







 §.§ Metrics


We evaluate the factual consistency, fluency and informativeness of summaries using four different metrics: (1) FactCC <cit.>, a weakly-supervised, model-based approach for verifying factual consistency and identifying conflicts between source documents and generated summaries, (2) SummaC <cit.> that enables natural language inference models to detect inconsistency, (3) ROUGE<cit.>, an automatic evaluation metric for the informativeness and fluency of a summary based on lexical overlap, and (4) BERTScore <cit.>, that computes a similarity score between candidate and reference summaries using contextual embeddings.



 §.§ Implementation


We run LLaMA-2-13B with Text Generation Inference[https://github.com/huggingface/text-generation-inferencehttps://github.com/huggingface/text-generation-inference] on 8 × 24GB NVIDIA GeForce RTX 3090 GPUs. For CNN/DM and XSum, we cluster and filter sentences using the DBSCAN algorithm with ε = 0.25 and MinPts = 2. The window size L_w is 150 words, and the step size L_s is 50 words. For arXiv and PubMed, we set distance threshold ε to 0.25, and MinPts to 3. The window size L_w is 750 words, and the step size L_s is 150 words. For SumCoT, we use the same prompts in paper. We set chunk size to 750 words in Refine. The prompts used for summary generation, sentences classification and sentences integration are listed in Table <ref>. Regarding other baselines used in the experiments, we use standard checkpoints provided by the authors and adopt the same configuration as in the corresponding papers, respectively.











































§ RESULTS




 §.§ Main Results
















  
Overall Performance.The experimental results of SliSum and baselines on four summarization datasets are reported in Table <ref>. 

In terms of short text summarization, Claude-2 with SliSum achieves a relative gain of 13.9% on FactCC and 15.8% on SummaC for CNN/DM respectively. Notably, Claude-2 with SliSum also performs the best on both ROUGE and BERTScore compared with other baselines, indicating that SliSum simultaneously improves informativeness and fluency of summaries. 

For long document summarization, Claude-2 with SliSum achieves a notable relative gain of 20.5% on FactCC and 22.2% on SummaC for arXiv respectively. Although FactorSum and Lodoss achieve close performance to SliSum on reference-based similarity metrics, they are significantly lower than SliSum on FactCC and SummaC. Regarding PubMed dataset, SliSum enables Claude-2 and GPT-3.5 to achieve almost the same excellent performance, and outperform other baselines on all metrics. 

Overall, LLM combined with SliSum uniformly outperforms base model itself with a wide margin on all metrics, which demonstrates the generality and effectiveness of SliSum. Specially, among three LLMs, SliSum reveals the most outstanding performance on Claude-2 and the greatest gain on LLaMA-2. Besides, SliSum applied to GPT-3.5 outperforms SumCoT and Refine on all four datasets. This shows that the superiority of our approach to other LLM-based baselines. We observe that SliSum achieves higher improvement on arXiv and PubMed than CNN/DM and XSum, and leads to greater performance gain on faithfulness metrics than on informativeness metrics. Therefore, our approach can substantially improves faithfulness of various LLMs in short and long text summarization, while maintaining their fluency and informativeness.





  
SliSum enhances the ability to process long contexts. We observe that SliSum has different performance gains on different datasets, which may be due to the style or length of the text. To eliminate the distractor of the text style and further demonstrate the effectiveness of our approach on news texts. We evaluate the factual consistency of GPT-3.5 on test samples of length ranges from 500 to 3000. We select 25 samples from CNN/DM and XSum with a length that is less than 5% different from the specified length to construct the test sets, respectively. 

As shown in Figure <ref>, compared to the original model, SliSum significantly improves the faithfulness of GPT-3.5 in news texts of various length, and the performance gain is positively correlated with the length. This is consistent with the results in Table <ref> and indicates that SliSum is capable of leading to larger benefits from increasingly long context. Besides, SliSum prevents the FactCC and R-1 score of the summaries from continually decreasing as source articles grow longer. For example, when the length of articles increases from 1000 to 3000, the FactCC score of summaries generated for CNN/DM by SliSum remains stable, while the scores of the baseline still decreases. We demonstrate that our approach enhances the ability to handle long texts in a variety of styles.



 §.§ Ablation Study


SliSum contains two essential modifications: sliding generation, filteration and aggregation based on self-consistency. In order to investigate the effect of modification designed in the SliSum, we conduct the following ablation studies by comparing SliSum with corresponding variation on GPT-3.5. 






















  
Sliding Windows vs. Parallel Windows. Parallel window(PW), a special case of sliding window(SW), partitions contexts in a non-overlapping manner. To investigate the contribution of sliding generation to SliSum, we compare comprehensive performance of SW and other baselines (also generate the summary K times to perform aggregation). Table <ref> shows that single window method is worse than others on PubMed, due to the distraction issue caused by long texts. In contrast, SW outperforms PW method on all three metrics, which demonstrates that sliding generation brings performance improvement to LLMs. This is because SW can take advantage of longer context to support generating summaries, whereas PW lack direct interaction between windows, resulting in the inability to adequately extract global important information. Specifically, benefiting from overlapping windows, SW can utilize information of  L_w + L_s(⌈ L_w / L_s ⌉ - 1) words during generating summary for each text segment, while PW can only utilize L_w words. Furthermore, Table <ref> shows that sliding windows facilitates LLMs to more fairly and faithfully process contexts of various length without position bias. The summaries generated by SliSum contain the content of each position range of the source article relatively evenly.




  
Filteration and Aggregation improve faithfulness. In order to understand the importance of filteration and aggregation based on self-consistency, we conduct ablation study by comparing SliSum to variation w/o aggregation. We remove two steps by directly generating global summaries for all local summaries (Map-Reduce <cit.>). As shown in Figure <ref>, the FactCC scores of both SW and PW w/o aggregation are lower than their counterparts. Notably, sliding windows w/o aggregation substantially diminish the factual consistency of summaries, even worse than parallel windows w/o aggregation. This indicates that aggregation is essential for SliSum, because sliding generation while providing more diverse information also leads to self-contradiction problem, which asks us to address this problem by filteration and aggregation based on self-consistency. 



 §.§ Impacts of Hyperparameters


We conduct quantitative experiments to investigate impacts of hyperparameters in SliSum on performance. Limited by computational resources and budget, we randomly select 25 articles from corresponding dataset to construct reduced test sets.





  
Ratio of L_w to L_s. Figure <ref> (left) shows that the FactCC score when varying in ratio K = ⌊ L_w / L_s⌋ from 1 to 10. As shown in Equation <ref>, K represents the minimum number of process content of the article. A large K can more adequately exploit the self-consistency of LLMs to improve performance. When the ratio grows from 1 to 5, the FactCC scores increased by 9.5% and 11.5% on arXiv and PubMed respectively, yet gradually converges when the ratio grows from 5 to 10, which indicates that processing articles too many times is extremely expensive and unnecessary. To save computational resources, we set the L_w / L_s to 5 in our experiments.











  
Window Size. Intuitively, shorter window means that LLMs can more accurately exploit information in context. But this has harmful effect on global understanding of entire article, resulting in the summary only contains locally important content. Nevertheless, a longer window will cause distraction issue and more uncertainty. As shown in Figure <ref> (right), the FactCC score of summaries initially raises as the window size grows. However, when window size increases from 750 to 1200, the performance remains stable or slightly drops. The quantitative experiments results indicate that LLMs require sufficient information to generate high-quality summaries from context, but a larger window will not further boost performance. 













  
Minimum Number of Clusters (MinPts). MinPts controls clustering and filtering of sentences about the same event. A higher MinPts forces SliSum to select sentences that are important in more windows. Table <ref> shows the FactCC and R-1 scores significantly increase with a bigger MinPts, but it will also lower R-1 score when MinPts greater than 3,  as a result of many relatively important sentences are filtered out. Therefor, we set MinPts to 3 for arXiv and PubMed. More generally, considering the balance between faithfulness and informativeness of summaries, the MinPts is set to 1/2 K for different parameter configurations.





  
Distance Threshold ε The choice of distance threshold ε depends on the data distribution. We count the average and the maximum distance between statements about the same event on the four datasets. Besides, we also calculate the average Hausdorff distance between sets of statements about different events. Given the set of statements X and Y for events A and B, the Hausdorff distance between X and Y is


    d_H(X, Y) = max{sup_x ∈ Xinf_y ∈ Yd(x, y), sup_y ∈ Yinf_x ∈ Xd(x, y)}


Table <ref> shows that the average distance between statements about the same event is around 0.25, which is much smaller than the distance between statements about different events. Hence, we set ε to 0.25 for all datasets. The results in table <ref> are consistent with our observation that statements generated by LLMs for the same event are lexically similar.



 §.§ Complexity Analysis




  
Theoretical Analysis SliSum reduces the computational complexity in the process of summary generation compared with the base model. For the input of length L, the computational complexity of most LLMs (such as LLaMA) is O(L^2). However, after applying SliSum to base models, the computational complexity of summary generation decreases to O(L). 
    
Generally, given window size L_w, step size L_s (L_s ≤ L_w) and an article whose length is L is far larger than the window size, K = L_w / L_s (see Equation <ref>). Our approach needs to process a total of (L / L_s + (K - 1)^2) text segments of length L_w, hence, the complexity of SliSum is 


    O(K × L_w × L + (K - 1)^2 × L_w^2) = O(L)


By solving the equation, we can see that when the input length is greater than 1.36 × K × L_w, the computational cost of summary generation of SliSum is less than base model.

Of course, SliSum also includes filteration and aggregation, which brings additional computational costs. However, the DBSCAN algorithm runs very fast. As for the final aggregation process, LLM usually only needs to process very few sentences. Importantly, SliSum splits the original task into smaller subtasks, so parallelization can be used to speed up inference. Overall, SliSum slightly increases the computational cost compared to the base model.





  
Quantitative Experiments In order to verify the above theoretical studies, we recorded the inference time of the LLaMA-2-13B on four datasets (including 100 articles) to evaluate the computational cost. Table <ref> indicates the LLaMA-2-13B with SliSum only took twice as long as the LLaMA-2-13B without SliSum. By contrast, as shown in Table <ref>, SliSum substantially improves faithfulness of various LLMs. Therefore, SliSum slightly increases the computational cost, but brings relatively high performance gains.



§ CONCLUSION


In this paper, we propose SliSum, a novel LLM summary generation architecture, which improves faithfulness of LLMs in both short and long text summarization without additional resources and fine-tuning. SliSum leverages sliding generation as instruments for exploiting the self-consistency of LLMs, enables LLMs to more fairly and faithfully process contexts of various length, and solve contradictions between local summaries by clustering and filtering. Extensive experiments demonstrate the effectiveness of SliSum in diverse LLMs. SliSum empowers LLMs to generate faithful summaries while maintaining their fluency and informativeness. Furthermore, we conduct ablation studies to justify the modification made in the SliSum and investigate the impact of hyperparameters in SliSum. We also demonstrate through theoretical analysis and quantitative experiments that SliSum only slightly increases the computational cost.



§ ACKNOWLEDGEMENTS

This work was supported by the Zhejiang Provincial Natural Science Foundation of China under Grant No. LZ23F020009, the NSFC project (No. 62072399), MoE Engineering Research Center of Digital Library, China Research Centre on Data and Knowledge for Engineering Sciences and Technology, and the Fundamental Research Funds for the Central Universities. We also express our sincere gratitude to anonymous reviewers for their invaluable feedback and constructive comments.




§ BIBLIOGRAPHICAL REFERENCES


lrec-coling2024-natbib















































































































    


 Paper Title: Advancing Vietnamese Visual Question Answering with Transformer and Convolutional Integration.



















































1,3]Ngoc Son Nguyen


1,3]Son Van Nguyen


2,3]Tung Lecor1
lttung@fit.hcmus.edu.vn

[cor1]Corresponding author

[1]organization=Faculty of Mathematics and Computer Science, University of Science,
            
            city=
Ho Chi Minh,
            
            
            country=Vietnam

[2]organization=Faculty of Information Technology, University of Science,
            
            city=Ho Chi Minh,
            
            
            country=Vietnam

[3]organization=Vietnam National University,
            
            city=Ho Chi Minh,
            
            
            country=Vietnam







§ ABSTRACT 
 Visual Question Answering (VQA) has recently emerged as a potential research domain, captivating the interest of many in the field of artificial intelligence and computer vision. Despite the prevalence of approaches in English, there is a notable lack of systems specifically developed for certain languages, particularly Vietnamese. This study aims to bridge this gap by conducting comprehensive experiments on the Vietnamese Visual Question Answering (ViVQA) dataset, demonstrating the effectiveness of our proposed model. In response to community interest, we have developed a model that enhances image representation capabilities, thereby improving overall performance in the ViVQA system. Specifically, our model integrates the Bootstrapping Language-Image Pre-training with frozen unimodal models (BLIP-2) and the convolutional neural network EfficientNet to extract and process both local and global features from images. This integration leverages the strengths of transformer-based architectures for capturing comprehensive contextual information and convolutional networks for detailed local features. By freezing the parameters of these pre-trained models, we significantly reduce the computational cost and training time, while maintaining high performance. This approach significantly improves image representation and enhances the performance of existing VQA systems. We then leverage a multi-modal fusion module based on a general-purpose multi-modal foundation model (BEiT-3) to fuse the information between visual and textual features. Our experimental findings demonstrate that our model surpasses competing baselines, achieving promising performance. This is particularly evident in its accuracy of 71.04% on the test set of the ViVQA dataset, marking a significant advancement in our research area. The code is available at <https://github.com/nngocson2002/ViVQA>.


















Visual Question Answering ViVQA EfficientNet BLIP-2 Convolutional.














§ INTRODUCTION


The rapid enhancement of multi-modal processing in recent years <cit.> has not only opened up vast potential but has also considered a range of remarkable challenges. Among these, the field of VQA emerges as a critical problem in comprehending multi-modalities. This task entails empowering computers to interpret and respond to queries based on the content of images, necessitating a synergistic integration of advancements in both image processing and natural language processing. Despite its complexities, VQA is an essential and challenging task in the ongoing evolution of multimedia data. It represents a significant step towards understanding and integrating diverse forms of information, a cornerstone in the development of more intelligent and versatile Artificial Intelligence (AI) systems.







However, a notable challenge is that the majority of datasets used to train current VQA models are in English. Most research efforts in this field have prioritized languages with abundant resources, leaving low-resource languages like Vietnamese largely overlooked. This presents significant issues regarding the applicability of VQA models to questions written in Vietnamese and their ability to comprehend Vietnamese culture. In this context, focusing on the ViVQA task is invaluable, as it plays an important role in driving the development of innovative approaches in this specific domain. To address this gap, an increasing number of VQA models for Vietnamese have been established, such as those proposed in recent studies <cit.>, significantly contributing to this field. Our work is also no exception, aiming to tackle the ViVQA challenge by developing a novel system for ViVQA that leverages models based on the Transformer architecture <cit.>, which have been pre-trained on various tasks related to both vision and language, such as UNITER  <cit.>, CLIP <cit.>, BLIP <cit.>, and Flamingo <cit.>.



A fundamental aspect of VQA systems is the effective extraction and representation of textual and visual features in meaningful ways. The development of pre-trained models in text and image processing plays a crucial role in this aspect. However, it is essential to recognize that each type of model brings unique strengths to the table and should be evaluated from diverse perspectives. In our model, multi-modal features are intricately connected and integrated through a specialized fusion module. Building on the successes of previous models, our approach focuses on refining the semantic representations of uni-modal.


Although the Transformer architecture is renowned for its ability to analyze complete images, it sometimes overlooks crucial small details. This oversight is a significant challenge in VQA, as these details often contain vital information for understanding and answering questions. Shen et al. <cit.> contended that a VQA model relied on the Transformer architecture, exclusively considering global dependencies, is unable to effectively capture image context information. Through studies, they also proved that using both local and global features in image processing improves performance. Clearly, it is essential to consider the contextual representation of images from both local and global perspectives. Therefore, with this consideration in mind, we propose an innovative integration of transformer-based and convolutional methods in image processing to enhance the feature extraction module in VQA. This integration aims to enhance the visual information by combining the detailed local features extracted by the Convolutional Neural Network (CNN) with the broad global features derived from Vision Transformer-based models. Our novel component has the potential to evolve alongside advancements in visual feature extraction models by incorporating current improvements into our sophisticated combination. The promising results on the Vietnamese VQA dataset demonstrate that our proposed approach is more efficient than existing competitive baselines and plays an important role in representing images in VQA processing.

This paper makes several pivotal contributions to the field of ViVQA, highlighted as follows:

    
  * We propose an innovative methodological advancement by integrating transformer-based and convolutional methods in image processing. Our novel representation consists of local and global consideration through the content of images.
    
  * Our novel visual feature extraction is integrated into the ViVQA approach, employing the latest fusion module to deploy the efficient system.
    
  * Through the detailed experimental analysis and comparative results, our  method demonstrates the potential performance over existing competitive baselines. This achievement is a significant stride in leveraging multi-view feature representation, enhancing the understanding of different modalities within ViVQA. 



§ RELATED WORKS



 §.§ Overview

VQA methods typically consist of three primary components: modules for feature extraction from textual and visual inputs, a multi-modal fusion module, and a classifier or generator module. Initially, Antol et al. <cit.> introduced the VQA task alongside a simple baseline, utilizing VGGNet <cit.> and Long Short-Term Memory (LSTM) <cit.> for visual and textual feature extraction, respectively. They then fused these features into a joint representation for classifying candidate answers. Subsequently, Yuling Xi et al. <cit.> introduced an approach focusing on object relationships within images to address VQA tasks. Their method utilized a question-based attention mechanism to highlight relevant object regions, enabling accurate responses to questions regarding image content. Additionally, Yirui Wu et al. <cit.> proposed a baseline method emphasizing both global and local object relationships, underscoring the significance of comprehensive relation reasoning. Their approach leveraged information from the entire image for global understanding while the local aspect focused on modeling relationships among multiple objects to facilitate answer generation. Notably, these baselines all share the commonality of employing pre-trained CNN models like ResNet <cit.>, VGGNet <cit.>, and FasterRCNN <cit.> for image feature extraction. In terms of text embedding, pre-trained word embeddings such as GloVe <cit.> or Word2Vec <cit.> are often used along with LSTM <cit.> or Gated Recurrent Unit (GRU) <cit.> networks to extract linguistic features. In recent years, researchers have increasingly turned to Transformer-based models such as ViLBERT <cit.>, X-LXMERT <cit.>, and VLMo <cit.>. Consequently, later works often rely on these models, leveraging the capabilities of pre-trained models.

The primary focus of this paper is the examination of monolingual ViVQA. In previous research, Tran et al. <cit.> introduced a model that utilized Hierarchical Co-Attention <cit.>, establishing the first baseline for ViVQA dataset. Their experiments demonstrated that their system outperformed two baseline models, LSTM <cit.> and BiLSTM, on the ViQA dataset. Following this, with the advent of transformers, Nguyen-Tran et al. <cit.> proposed an approach that incorporates a Bi-directional Cross-attention architecture. This model was designed to more effectively comprehend the complex interplay between visual and linguistic features, specifically in the Vietnamese context. Another notable contribution by Nguyen et al. <cit.> was the introduction of Parallel Attention Transformer (PAT), a novel multimodal learning scheme. This method effectively integrates the advantages of grammar and contextual understanding in Vietnamese. More recently, Tran et al. <cit.> proposed leveraging BEiT-3 <cit.> as a fusion encoder to intricately model the deep interactions between images and questions. Furthermore, they employed BARTpho <cit.>, a monolingual sequence-to-sequence model pre-trained for Vietnamese, to enhance question representations. Remarkably, this model has achieved the current state-of-the-art performance on the ViVQA dataset. This research significantly advances the field of VQA, particularly in the Vietnamese language context, showcasing the effectiveness of novel models in capturing the nuanced interactions between visual and linguistic elements.




 §.§ ViVQA Models in Previous Studies

In this section, we will briefly present two methods that currently achieve the best performance on the ViVQA task and point out their weaknesses, which motivates our study.


  §.§.§ Multimodal Contextual Attention Model

In 2022, Anh Duc Nguyen et al. <cit.> introduced the Multi-vision Contextual Attention (MCA) model, presenting an innovative approach to image feature extraction that seamlessly integrates both global and local features. This model excels in learning attentional embeddings, capturing diverse contexts from both images and questions while being mutually guided.



As depicted in Figure <ref>, the image initially extracts local features corresponding to specific regions or parts of objects in the image, acquired through the utilization of ResNet <cit.>, a convolutional neural network. Simultaneously, global features encapsulate fundamental information about the contextual surroundings within the image, including associations among objects, and are derived using the Vision Transformer (ViT) <cit.>. For question embedding, PhoBERT <cit.> is employed to extract features from questions. Based on the obtained visual and textual features, a multi-branch contextual attention fusion hierarchically integrates visual and textual attention features, guiding each other and fusing them into a multimodal feature representation. This pioneering model represents a significant advancement in feature extraction, showcasing the integration of diverse contextual elements for a more comprehensive understanding of multimodal data.

Despite the combination of CNN and a Transformer in this model, it's worth noting that ResNet is not the latest CNN model. In our proposed architecture, we aim to innovate by combining both local and global features inspired by the MCA model. Through this approach, we anticipate achieving better performance.


  §.§.§ BARTPhoBEiT

Tran et al. <cit.> have recently introduced an innovative methodology that integrates pre-trained sequence-to-sequence and image Transformers customized for the Vietnamese language. This pioneering approach draws inspiration from BEiT-3 <cit.>, serving as the fundamental framework for their model.



As shown in Figure <ref>, the image embedding module utilizes BEiT v2 <cit.> to extract global features from images, while for question embedding, BARTPho <cit.> is employed to extract question features. Subsequently, the visual and question features are fused using a Multiway Transformer architecture <cit.>. These Transformers dynamically route information to the appropriate expert for each modality, ensuring the capture of modality-specific details. This unique feature significantly enhances the model's efficacy in handling diverse data modalities, emphasizing its versatility in cross-modal applications. Furthermore, the robustness of this model relies on the Transformer architecture, enabling seamless integration of an extensive volume of pre-trained data across diverse domains, resulting in state-of-the-art performance on the ViVQA dataset.

Despite the commendable results of their work, we have identified the complexity of image processing and the absence of information about local features as areas for further exploration. Building on the insights from their study, we are motivated to focus on the development of methods that enhance the performance of image processing, particularly for the ViVQA task.



 §.§ Dataset for ViVQA

In the context of Vietnamese, the development of data for VQA is not as advanced as it is in English. The available resources are limited, presenting a scarcity of data specifically tailored for ViVQA. This dearth underscores the challenges and gaps in the field, as only a modest amount of data is currently accessible for research and improvement in ViVQA. It is challenging to create models that can effectively generalize to new inputs when there is insufficient data to capture the entire variety of text and images. Furthermore, a significant element influencing the models' performance is the dataset's quality. Predictions might be erroneous and ineffective generalization can result from incomplete and low-quality data. Within the scope of this study, our primary focus is on monolingual Vietnamese datasets. We will survey datasets that align with our research objectives to serve as the foundation for our model.


  §.§.§ OpenViVQA Dataset

In 2023, Nghia et al. <cit.> introduced the OpenViVQA dataset <cit.> aims to showcase the inherently open-ended nature of both questions and answers. In order to seamlessly align with the intricacies of the Vietnamese language, the OpenViVQA dataset thoughtfully incorporates a diverse array of images captured within the rich tapestry of Vietnam's landscapes. This deliberate inclusion not only serves to showcase the linguistic nuances but also aims to highlight the myriad characteristics that distinctly define Vietnam in comparison to other regions. By strategically curating scenes from Vietnam, the dataset offers a comprehensive representation of the country's unique cultural, environmental, and visual elements, contributing to a nuanced understanding within the realm of VQA research.

The dataset is divided into two distinct components: “Text QA" and “Non-text QA". Specifically, the non-text QAs are designed to extract information pertinent to objects, encompassing their attributes and relational aspects. On the contrary, it is categorized as Text QA if it effectively leverages textual information emanating from the images themselves or employs the words present within the images as a means to pose questions. As a result of the inclusion of Text QA within the dataset, consequently making Optical Character Recognition (OCR) <cit.> a more involved process, our primary focus does not lie in this domain. Our paramount focus is intricately tied to the detailed examination of large objects depicted in the images. Thus, the OpenViVQA dataset does not align with our preference for this particular study.








  §.§.§ ViVQA Dataset
 
The inaugural dataset in ViVQA, crafted by Tran et al. <cit.> in 2021, stands as a pioneering contribution to the field. Drawing inspiration from established English VQA benchmarks and COCO-QA <cit.>, this dataset has evolved into a widely embraced benchmark for related research endeavors. The creation process involved leveraging machine translation for the conversion of questions and answers, followed by meticulous manual validation to ensure correctness and linguistic fluency in the translated content. 

The ViVQA dataset <cit.> encompasses a collection of 10,328 images paired with 15,000 corresponding questions, intricately tailored to reflect the content within the images. To facilitate experimentation and evaluation, the dataset underwent a random partitioning into training and test sets, maintaining an 80:20 ratio. Noteworthy is the dataset's categorization of questions into four distinct types: Object, Number, Color, and Location, constituting 41.55%, 14.81%, 20.82%, and 22.82% of the total questions, respectively. 

This dataset is particularly pertinent to our research objectives as our paper predominantly delves into advanced image processing techniques. Our primary focus lies in extracting detailed information related to objects, encompassing their attributes, relational aspects, and the overall landscape depicted across the image. Consequently, we have deliberately chosen this dataset for our study, opting against OpenViVQA. This decision is attributed to the existence of questions related to textual content within images in OpenViVQA, which does not align with the specific aspects of image analysis and processing we are exploring in our research.















































































































§ METHODOLOGY

In this paper, we approach the ViVQA task as a classification problem. Specifically, given an image I and a question Q, the goal is to determine the most probable answer â from a predefined set of answer options A:

    â = max_a ∈ A P(a|I,Q,A)


In this section, we present the detail of our ViVQA model architecture, illustrated in Figure <ref>, drawing inspiration from BARTPhoBEiT <cit.> with specific modifications in the visual processing aspect. We also provide detailed descriptions of the main image processing steps employed in our model. Our proposal advocates for the implementation of visual processing with frozen parameters, effectively reducing both the model parameters and training costs.





 §.§ Image Embedding Module
 
Utilizing the capabilities of pre-trained models, our image embedding module proves to be both effective and powerful, capturing valuable features through pretraining tasks. We employ both BLIP-2 <cit.> and EfficientNet <cit.> to extract visual features from both local and global respectively, amalgamating them to derive the final representation for the image. A distinctive feature of our module is its separation from the overall architecture, ensuring that its parameters remain static throughout the training process. This design choice significantly reduces the overall number of parameters in our model and shortens the training duration.



  §.§.§ BLIP-2

The first model in the BLIP family, Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (BLIP), was proposed by Li et al. <cit.>. BLIP introduced a novel Vision-Language Pre-training (VLP) framework that supports a wider range of downstream tasks than existing methods. This is achieved by jointly training models on extensive language and image datasets, thereby leveraging the complementary strengths of both modalities. 

Following this, Li et al. <cit.> proposed BLIP-2, an enhanced version of the original model with several adjustments in both the training strategy and the architecture. BLIP-2 introduces a pioneering method that leverages the capabilities of pre-trained vision models and language models. Unlike BLIP, which involves end-to-end training and often incurs high computational costs and lengthy training durations, BLIP-2 opts to freeze these pre-trained models during the training process. This strategy results in improved performance at a reduced computational cost and addresses the issue of catastrophic forgetting. Moreover, it ensures that the model capitalizes on the rich representations learned by the visual and language model, thereby enhancing its overall performance and effectiveness in comprehending both image and text inputs. With these strengths of BLIP-2, we decided to adopt it to take advantage of global image-level feature extraction.

The central aim of BLIP-2 is to facilitate mutual understanding between the image and language. To achieve this objective, the Querying Transformer (Q-Former) module is specifically engineered to bridge the gap between vision and language. Following the Figure <ref>, the Q-Former architecture in BLIP-2 incorporates (left) an image transformer utilizing an encoder-only transformer and (right) a text transformer, capable of functioning as both a text encoder and a text decoder. Both the image transformer and text transformer share the same self-attention layers. Given our focus on obtaining image representations for the ViVQA system, we are particularly interested in the image transformer within the Q-Former, which returns the visual features. A specific number of learnable query embeddings serve as input for the image transformer. The queries, each comprising 32 learned query tokens with a dimension of 768, contextualize the query tokens through the cross-attention mechanism with the representations of the image patches encoded by a large (frozen) ViT <cit.>. The visual tokens serve as the output of the Q-Former.



Notably, a crucial stage involves representation learning, with the primary aim of pre-training within BLIP-2 being to refine the Q-Former's ability to enable queries to extract visual representations that are maximally informative regarding the accompanying text. Consequently, the features of the image generated by BLIP-2 encompass an ample amount of information, particularly in terms of global aspects, satisfying our ViVQA system's needs. The output query representation in ℝ^32 × 768, significantly smaller than the size of the frozen image features (e.g., 257 × 1024 for ViT-L/14). Thus, the image I representation after BLIP-2 feature extraction can be formulated as:

    V_G = BLIP-2(I) ∈ℝ^32 × 768




  §.§.§ EfficientNet

We integrate the EfficientNet architecture introduced by Tan et al. <cit.>, a sophisticated CNN, for local image-level feature extraction. Intuitively, CNN architectures inherently utilize kernels sliding over regions of the image, effectively capturing information locally. Previous CNN architectures commonly face a trade-off between model complexity and performance, wherein as performance improves, model complexity increases. This trade-off involves scaling, which can occur through depth scaling, width scaling, and resolution scaling.

Depth scaling entails increasing the depth of a neural network, typically resulting in improved performance. Width scaling involves increasing the number of channels (or neurons) in each layer, thereby enhancing the model's capacity to capture a broader range of features at each level and improving its representational power. Resolution scaling, meanwhile, requires resizing input images to higher resolutions, potentially leading to better performance as higher-resolution images contain more detailed information. 

EfficientNet <cit.> is an improvement version basing on CNN that implements the concept of compound scaling, which enhances performance by uniformly scaling all dimensions of depth, width, and resolution. Its power is in how these three dimensions are balanced and adjusted using a systematic methodology. The model employs Mobile Inverted Bottleneck (MBConv) <cit.> layers, combining depth-wise separable convolutions and inverted residual blocks. This bottleneck design enables efficient learning while preserving a high degree of representational power. The model's potency has been demonstrated through experiments by Tan et al. <cit.>, which showed that models in the EfficientNet family achieve performance equivalent to other models while maintaining low complexity. To leverage these capabilities, we employ an EfficientNet-B7 model, a pre-trained model based on adversarial training <cit.>. It is initialized with pre-trained weights on ImageNet <cit.>, providing a robust foundation for directly extracting features for our ViVQA system.

The input image I, denoted as ℝ^C × H × W, where C,H and W stand for channels, height, and width respectively, undergoes processing within the pre-trained EfficientNet model. The features are extracted from the output of the final head before undergoing average pooling and a fully connected layer. This process generates a representation capturing the local features of the image, expressed as ℝ^2560 ×H/32× W/32. It is noteworthy that our original image data is standardized to the dimensions of ℝ^3 × 224 × 224. Consequently, the image representation following EfficientNet feature extraction has dimensions ℝ^2560 × 7 × 7.

    V_L = EfficientNet(I) ∈ℝ^2560 × 7 × 7


This approach allows us to capture intricate local features from the images, providing a rich and informative representation essential for subsequent stages in our model architecture. 



  §.§.§ Image Embedding

In this stage, we will amalgamate the representations of BLIP-2 and EfficientNet to derive the final output of the image embedding module. Initially, an inconsistency was observed in the dimensions of BLIP-2 (<ref>) and EfficientNet (<ref>). Thus, we will implement several steps to transform the representation of EfficientNet to align it with the output of BLIP-2. This transformation is crucial to facilitate operations for seamlessly combining the local and global features.

The transformation involves employing Adaptive Average Pooling to downsample from ℝ^2560 × 7 × 7 to ℝ^2560 × 1 × 32:

    V_L = AdaptiveAveragePooling(V_L) ∈ℝ^2560 × 1 × 32


Subsequently, its dimensions are permuted to ℝ^32 × 1 × 2560, and another round of Adaptive Average Pooling is applied for downsampling to ℝ^32 × 1 × 768:

    V_L = Permute(V_L) ∈ℝ^32 × 1 × 2560


    V_L = AdaptiveAveragePooling(V_L) ∈ℝ^32 × 1 × 768


Through the flattening process, we derive the ultimate output of the image embedding module:

    V_L = Flatten(V_L) ∈ℝ^32 × 768


Finally, the fusion of local features and global features is computed as follows:


    V = ℱ(V_G, V_L) ∈ℝ^k × 768

where k relies on the operation ℱ. This equation represents the calculation for combining the global features, denoted as V_G, and the local features, denoted as V_L, through a fusion function ℱ.



 §.§ Text Embedding Module
 

Throughout Tran et al.'s study <cit.>, they observed the prevalence of grammatical errors and translation inaccuracies in ViVQA questions, even when validated by highly qualified annotators. Therefore, instead of relying on PhoBERT <cit.> to extract question information, which predominantly comprehends word meanings in diverse contexts, our preference is to employ BARTpho <cit.>. BARTpho stands out as a denoising autoencoder specifically crafted for pretraining sequence-to-sequence models geared towards Vietnamese language processing. The training regimen involves a two-step process: (1) corrupting the input text using a designated noising function and (2) learning to reconstruct the original text. This pretraining task empowers BARTpho with the capability to acquire robust representations resilient to noise, ambiguity, and variations in the input text. Hence, our approach leans towards leveraging the capabilities of BARTpho.

BARTpho offers two distinct variants, specifically BARTpho_syllyble, which is optimized for processing at the syllable level, and BARTpho_word, which is designed for handling language at the word level. In the case of BARTPho_word, the sentence must undergo preprocessing, being appropriately split into individual words and phrases. Subsequently, the input of complete sentences is fed into both the encoder and decoder, with the top hidden state of the decoder serving as a representation for each word. In our model, we use BARTpho_word and the textual features T can be articulated as:

    Q = BARTpho_word(T) ∈ℝ^(L+2) × 1024

where L + 2 with L is the max sequence length, adding 2 special token [CLS] and [SEP] representing the start-of-sentence and the end-of-sentence token.

Since the BLIP-2 in the image embedding module is predominantly trained on English data (specifically, images with English captions) and the text embedding module is tailored for Vietnamese, we set the parameters of the text embedding module to be trainable. This adaptability enables the text representation to dynamically adjust and align with the visual representation within the image embedding module.





 §.§ Multi-modal Fusion Module

Before forwarding the representations of visual and question features to BEiT-3 for multimodal fusion, a Feed-Forward Network (FFN) layer is applied on the textual features. The purpose is to diminish the dimensionality from 1024 (<ref>) to 768. This reduction is imperative due to the requirement of concatenating the embeddings of a given question and an image within this component. The dimension of the question is now given by:

    Q = FFN(Q) ∈ℝ^(L+2) × 768


This preprocessing step optimizes the dimensionality of the textual features, enhancing their compatibility for subsequent fusion with visual features. Subsequently, the visual (<ref>) and question (<ref>) features are concatenated:

    f_VQ = concatenate(V, Q) ∈ℝ^(k+L+2) × 768


Following this, the concatenated features are inputted into Multiway Transformers <cit.> to jointly encode the image-question pair. Each Multiway Transformer block is composed of a shared self-attention module and a set of feed-forward networks tailored for diverse modalities. Each layer includes both a vision expert and a language expert, each responsible for processing the input tokens directed to them. Leveraging a group of modality experts incentivizes the model to capture a broader range of modality-specific information. The shared self-attention module acquires the ability to discern alignments between diverse modalities, facilitating profound fusion for tasks involving multiple modalities, such as vision-language tasks.

The result will have the same dimension as the input f_VQ (<ref>). The first vector in the output fusion is earmarked for classification, denoted as f̂_VQ∈ℝ^1 × 768.
Prior to being input into the Classifier module, f̂_VQ undergoes processing through a pooler layer, involving the normalization, a FFN layer with hidden and output dimension of 768, and application of the hyperbolic tangent function as depicted below:

    f̂_VQ = Norm(f̂_VQ)


    f̂_VQ = FFN(f̂_VQ)


    f̂_VQ = Tanh(f̂_VQ)




 §.§ Classifier Module

We integrate a classification head to predict the answer, consisting of the following components: (1) a FFN layer with hidden and output dimensions of 768 and 1536, (2) a normalization layer, (3) the GELU activation function, and (4) an additional FFN layer that maps the input dimension of 1536 to the number of classes (353 candidate answers).

    l = FFN(f̂_VQ)


    l = Norm(l)


    l = GELU(l)


    l = FFN(l)


Following this, a softmax function is applied to convert the output into a distribution within the range of 0 to 1:

    p̂ = softmax(l)

The candidate with the highest probability is then chosen as the proper answer.


§ EXPERIMENTS

In this section, we conduct evaluations of our model using the ViVQA dataset <cit.>. Additionally, we compare our model's performance with that of previous ViVQA models that have achieved state-of-the-art results on the same dataset.

By formulating the VQA task as classification, we employ Cross-Entropy Loss as the loss function and assess our model's performance on the ViVQA dataset, utilizing accuracy as the main metric.



 §.§ Dataset

As we mentioned in section <ref>, the ViVQA dataset is the subject of our investigations. A training set of 11,999 Q-A pairs and a test set of 3,001 Q-A pairs comprise the dataset. The detail of the dataset is shown in Table <ref>.





 §.§ Evaluation Metrics

In formulating the ViVQA as a classification problem, we assess the model's effectiveness using key metrics: F1 score, Precision, Recall, and Accuracy. In alignment with our objective to compare against the baseline in our experiment, we prioritize accuracy over other metrics. This choice stems from the fact that all baseline models are evaluated based on accuracy, with some not utilizing additional metrics. Thus, in our experimental endeavor to evaluate the influencing factors of the model, accuracy will serve as the primary metric for a more straightforward analysis.


  §.§.§ Accuracy

Accuracy or exact match is defined as the ratio of the number of correct predictions to the total number of ground truth answers. Let N represent the total number of ground truth answers. The accuracy is calculated as follows:

    Acuracy = 1/N∑_i=1^N a_i


    a_i = 
            
          1,    if ŷ_̂î = y_i 
    
          0,    otherwise

where ŷ_̂î is the predicted answer, and y_i is the ground truth answer for question i.








  §.§.§ Recall

This metric, relying on tokenized versions, measures the proportion of correctly identified tokens in the ground truth answer that are also present in the predicted answer. Let P_i represent the tokens in the predicted answer and GT_i denote the tokens in the ground truth answer for the question i-th. The recall for the question i-th is determined as follows:

    r_i = P_i ∩ GT_i/GT_i

Let N represent the total number of ground truth answers, so the overall recall is computed as:

    Recall = 1/N∑_i=1^N r_i



  §.§.§ Precision

Precision, in the context of tokenized forms, evaluates how many of the predicted tokens match those in the ground truth answer. Let P_i represent the tokens in the predicted answer and GT_i denote the tokens in the ground truth answer for the question i-th. The precision for the question i-th is calculated as follows:

    p_i = P_i ∩ GT_i/P_i

Let N represent the total number of ground truth answers, so the overall recall is computed as:

    Precision = 1/N∑_i=1^N p_i



  §.§.§ F1 score

The F1 score is essentially computed from both recall (<ref>) and precision (<ref>) of each individual question. The F1 score for the question i-th is calculated as follows:




    f_i = 
            
          0,    if p_i = 0, r_i = 0 
    2 × p_i × r_i/p_i + r_i,    otherwise

Let N represent the total number of ground truth answers, so the overall F1 score is computed as:

    F1 = 1/N∑_i=1^N f_i




 §.§ Experimental Settings

In our experiments, the hyperparameters presented in Table <ref> were utilized. The training process spanned 20 epochs with a batch size of 65, a drop path rate of 0.3, a learning rate of 3e-5, weight decay set at 0.01, and a warmup ratio of 0.1. The input image resolution was 224 x 224. We employed AdamW <cit.> as the optimizer for updating parameters. The model training took place on a computing system equipped with NVIDIA TESLA P100-16GB GPUs and 32GB of RAM.



Remarkably, within the multi-modal fusion module, a crucial hyperparameter during training is the drop path rate, also known as stochastic depth <cit.>. The role of drop path is akin to dropout <cit.>, a regularization technique employed to counteract overfitting in neural networks. Both methods introduce randomness during training, yet they operate at different levels of network architecture. Drop path randomly omits individual samples within a batch during training, thereby impacting the depth of the network's computational graph. In contrast, dropout randomly eliminates units within layers, thereby altering the connectivity between layers. Drop path selectively bypasses specific connections within the neural network for each sample independently, while dropout affects all units within a layer. By randomly omitting paths (connections) during training, drop path encourages the network to acquire more robust features by discouraging reliance on specific paths. This diversification of learned features aids in mitigating overfitting by facilitating a broader exploration of the network's parameter space.


  §.§.§ Training Strategy

In our paper, we adopted a systematic approach to training our model to ensure optimal performance. Initially, we partitioned our dataset into a training set and a validation set in an 80:20 split. This division allowed us to fine-tune and determine the most effective parameters for our model, with a particular emphasis on identifying the most suitable hyperparameters, including the drop path rate, the number of encoder layers, and the count of encoder attention heads. For the rest of the hyperparameters, we closely followed the training settings used for fine-tuning the BEiT-3 model <cit.> on the VQAv2 dataset <cit.>.

To further refine our training process and ascertain the ideal number of training epochs necessary for model convergence, we employed a 5-fold cross-validation strategy on the training set. This approach enabled us to determine that our model reached convergence after an average of 20 epochs across all folds. Armed with this insight, we proceeded to train our model on the complete training set for the identified duration of 20 epochs.

After selecting the optimal hyperparameters, we tested each model ten times using ten different random seeds. This approach allowed us to determine the statistical significance between any two models using a two-sample t-test.



 §.§ Experimental Results



  §.§.§ Impact of Fusion Operation Types on the model

In the initial phase of our study, we conducted an exploration to identify the optimal fusion function ℱ, as defined in equation (<ref>), for our model. This investigation encompassed various strategies, including element-wise multiplication, element-wise addition, and concatenation, applied to integrate both global and local image features. Despite the simplicity of the fusion strategies employed, they demonstrated noteworthy effectiveness in enhancing the overall performance of our model.

Table <ref> presents the outcomes of our experimentation. For the element-wise operations of multiplication and addition, we have features of visual ∈ℝ^32 × 768, and for concatenation, we have features in ℝ^64 × 768. It is evident that the concatenation operation outperforms the other operations, achieving the highest level of performance. Consequently, we have chosen concatenation as the fusion function ℱ for subsequent experiments.

It is noteworthy that the outcome of the multiplication operation is markedly lower compared to concatenation and addition. Our investigation revealed that the diminished performance can be attributed to the multiplication of two features (local ∈ℝ^32 × 768 and global ∈ℝ^32 × 768) resulting in a new matrix with dimensions ∈ℝ^32 × 768 that exhibits a high degree of sparsity. The boxplots in Figure <ref> indicate that, on average, the elements in the matrices after addition and concatenation operations demonstrate elevated levels and substantial spreads. In contrast, the multiplication operation yields a sparse matrix characterized by elements that tend to approach zero closely, displaying a narrow range of spreads. This pattern suggests that the sparse matrix resulting from multiplication leads to information loss in the visual representation, thereby accounting for its diminished performance.







  §.§.§ Impact of CNNs on the model

In the subsequent phase, we conducted a meticulous investigation into the influence of various CNN architectures on the efficacy of our model. The choice of a CNN architecture plays a pivotal role in shaping our model's performance. We comprehensively evaluated diverse CNN models, including ResNet <cit.> and VGGNet <cit.>, alongside the EfficientNet family architecture. This assessment was rooted in accuracy metrics and an examination of the parameter count for each model.

Table <ref> presents a synthesis of accuracy and parameter count data for each CNN model. The findings reveal that the EfficientNet family not only surpasses other models in terms of accuracy but also, with a moderate parameter count, exhibits superior performance compared to the conventional ResNet and VGG architectures. As observed, VGG achieves an accuracy of over 70%, but it requires hundreds of millions of parameters. In contrast, EfficientNet achieves comparable accuracy with only tens of millions of parameters. We selected the large-scale model (EfficientNet-B7) as the optimal choice, offering elevated accuracy with reasonable model complexity for our model.





  §.§.§ Comparison with other Methods

Due to the limited research on VQA tasks in Vietnamese, we benchmarked our model against several existing approaches that have demonstrated promising results in this domain.  Specifically, we compared our model to strategies proposed by other methods. LSTM-based approaches <cit.> and pre-trained word embeddings (PhoW2Vec <cit.>) proved ineffective in handling the complexities of VQA. Conversely, methods based on transformers demonstrated excellent performance in addressing this challenging task.



The comparative results are presented in Table <ref>. Our model achieves a remarkable accuracy of 71.04% on the test set, surpassing all other methods by a significant margin. This represents a substantial improvement of 37.19% over the least accurate model (LSTM + PhoW2Vec) and a notable gain of 2.46% over the second-best approach (BARTPhoBEiT). These results unequivocally demonstrate the superiority of our proposed method in addressing ViVQA.

Overall, many previous baselines lack information about local features, with the exception of MCA. The effectiveness of incorporating both local and global features has proven to be crucial for the success of previous literature. We will conduct an ablation study in the next section to demonstrate their influence. Although MCA considers both local and global features in image extraction and achieves commendable accuracy, it utilizes ViT, which is trained solely on image datasets. In contrast, our method leverages BLIP-2, which is pre-trained on both vision and language data. This vision-language pre-training enables BLIP-2 to capture more comprehensive and contextually rich information than that obtained from ViT.

With promising results achieved on the ViVQA dataset, our model has demonstrated effectiveness in the VQA task, particularly in its ability to extract, represent, and comprehend the relationship between images and text. To ensure the model effectively understands Vietnamese, we propose leveraging not only a powerful pre-trained model like BARTPho but also integrating a multi-modal information synthesis module like BEiT-3 to create a joint representation containing information from both text and images. Additionally, our model not only relies on the existing knowledge encoded in the pre-trained BARTPho model for Vietnamese, but we also refine it to suit multi-modal tasks between Vietnamese images and texts, thereby establishing a foundation for similar multi-modal studies in Vietnamese.













 §.§ Ablation Study

We conducted an ablation study to thoroughly investigate the contributions of BLIP-2 and EfficientNet to the overall outcome. As depicted in Table <ref>, models relying solely on BLIP-2 or EfficientNet for extracting image features exhibited lower accuracy. Whilst combining both methods led to improved performance, surpassing the results achieved with a singular approach.


Furthermore, we systematically selected the number of encoder attention heads and encoder layers in the multi-modal fusion module. In our system, this module plays a pivotal role in the model’s performance for combining information from the visual and question modalities. Initially, we fixed the number of encoder layers at 6 and varied the number of encoder attention heads. The results, as illustrated in Figure <ref>, highlight the optimal performance of our model with 6 encoder attention heads.



Subsequently, maintaining 6 encoder attention heads constant, we adjusted the number of encoder layers. The outcomes, showcased in Figure <ref>, underscore that the optimal number of layers for the encoder is 6.



After obtaining the ultimate components of our model, we investigated the impact of freezing the visual extractor on model performance. Due to limited infrastructure in the experimental environment, we only used EfficientNet-B7 from our image embedding module to conduct the experiment. The results, depicted in Table <ref>, indicate that freezing the EfficientNet-B7 led to an accuracy of 71.04% within 20 epochs, whereas the visual extractor is not frozen, the accuracy is slightly lower at 70.44%, with a significantly longer training time of 935 seconds. From these findings, we conclude that freezing the visual extractor can enhance efficiency without compromising accuracy, as well as saving a lot of training time, rendering it a viable strategy for training deep learning models in similar contexts.






 §.§ Discussion



  §.§.§ Dataset Challenges

Despite the comprehensive validation of the dataset by professionals, certain samples exhibit ambiguity, grammatical issues in ViVQA questions, or a lack of relevance between questions and answers. This highlights the challenge posed by the dataset's overall low quality, a significant hurdle for VQA tasks in Vietnamese, primarily due to the scarcity of high-quality datasets. As depicted in Figure <ref>, we illustrate instances where the ground truth answers are inappropriate, unclear, or unrelated to the corresponding questions. Particularly, discrepancies arise, such as the misidentification of the bear's color as brown instead of black, or the color of the cows being described as brown while the ground truth indicates white. Moreover, descriptions indicating a sandwich topped with quality ingredients are contradicted by an irrelevant answer, `Dog'. Similarly, instances where the depicted scene clearly shows a man riding a horse but is misinterpreted as a `mountain', and a girl taking a selfie, potentially in a restroom, yet labeled as a `Cage', further underscore the poor quality of the dataset in this study. These findings highlight the poor quality of the ViVQA dataset in this study, presenting additional challenges for researchers interested in improving dataset quality.








  §.§.§ Experimental Analysis

To start, we present insightful examples to analyze the performance of our model. Through experimental analysis, strengths and weaknesses become readily apparent in practical scenarios. The specifics of these instances are illustrated in Figure <ref>. 

In Figure <ref>-(a), both BLIP-2 and EfficientNet made individual incorrect predictions, suggesting that each model lacks comprehensive information from the image. BLIP-2, emphasizing the global context of the image, may overlook local details, while EfficientNet may struggle to capture broader contextual information. 

In Figure <ref>-(b), the model solely relying on BLIP-2 for visual extraction erroneously predicted the color of the vase as white, likely due to BLIP-2's tendency to prioritize global features, where much of the image appears white. Consequently, EfficientNet provided a correct answer by considering local details and determining the color of the vase to be red. 

In Figure <ref>-(c), the model utilizing only EfficientNet for visual extraction predicted that the location where a woman is holding a sandwich is the kitchen. This is attributed to EfficientNet's capability to focus on intricate details such as the sandwich and the dashboard of a car, which might resemble oven buttons. Conversely, BLIP-2, taking into account the global context, accurately identified the location as a car. 

Through experimental evaluation, it became evident that our model, along with its constituent components BLIP-2 and EfficientNet, exhibits both strengths and weaknesses. BLIP-2 excels in capturing the global context of images but may overlook finer local details, while EfficientNet focuses on intricate local features but may struggle with broader contextual information. These insights highlight the importance of considering both global and local features in model development for visual understanding tasks.



Another aspect we wish to address is the ambiguity observed in the answers provided in the dataset utilized for evaluation, as well as the complexity of the linguistic aspects. In Figures <ref>-(a,b), ambiguity arises between the ground truth answer and the predicted answer. We believe that the predicted answer could potentially be correct regarding the location of the object in the image; specifically, the man and artwork could be situated in the bathroom. Furthermore, in Figure <ref>-(c), there is an excessive amount of redundant information contained in the question, making it difficult for our model to extract relevant information. Consequently, our model predicts the overall scene (Room) instead of a specific place (Class). This underscores the importance of addressing ambiguity and redundant information in question-answer datasets, as they can significantly impact model performance and interpretation. With a sufficiently robust dataset, we believe we could tackle this challenge, representing the motivation driving our future focus.



An additional noteworthy aspect we would like to address pertains to the distribution of incorrectly predicted question types by our model. As depicted in Figure <ref>, our model inaccurately predicted the question type `object' in approximately 43.6% of cases, which accounts for nearly half of all incorrect predictions. This observation suggests that our model may lack sufficient information regarding the specific details of objects within the image. Consequently, addressing this limitation stands as a motivating factor for our future work.





  §.§.§ Transfer Learning Considerations

Our architecture is capable of efficiently transferring knowledge from a source domain, where the model undergoes initial training, to a target domain characterized by limited data or a demand for rapid deployment. This capability is underpinned by the flexibility of our design, which facilitates the easy replacement and adaptation of the backbone components.

In the context of linguistic applications, while our primary focus has been on monolingual Vietnamese, the model architecture can be readily adapted to other monolingual languages with minimal modifications to the Text Embedding module presented in section <ref>. This adaptability ensures that the linguistic component of our model can handle a diverse array of languages, thereby broadening its applicability. Similarly, in the visual domain, the model's backbones presented in section <ref>  can be replaced with alternative architectures, providing the flexibility to optimize performance based on specific visual tasks or datasets. 

Moreover, the versatility of our model extends beyond typical visual-linguistic tasks to specialized domains, such as Medical Visual Question Answering (MedVQA). Although our current work primarily addresses everyday scenes and objects, the transition to the medical domain is facilitated by the consistent input format of text-image pairs. This consistency aligns seamlessly with our architecture, enabling the model to leverage pre-existing knowledge and adapt to the nuanced requirements of medical imagery and terminology.

The ability to transfer learning effectively across these varied domains underscores the robustness of our architecture. By allowing for the modular exchange of core components, such as the text and visual backbones, our model provides a scalable and adaptable solution that meets the demands of diverse applications. This not only enhances the model’s utility across different linguistic and visual tasks but also ensures that it can be efficiently repurposed for new, domain-specific challenges.


§ CONCLUSION AND FUTURE WORK

In this study, we introduced a ViVQA system with the key integration of convolutional and base-transformer methods for vision embedding. The convolutional methods capture local features, while the base-transformer methods address global features, resulting in a more comprehensive image representation. Our experiments demonstrated the effectiveness of these combined approaches in enhancing our architecture's performance. However, we also encountered challenges related to the quality of the ViVQA dataset. The dataset's low quality posed significant difficulties, as it included several incorrectly labeled samples, which affected the overall performance. Despite our model's correct predictions from a human-like perspective, these predictions were not reflected accurately in the performance evaluations due to the dataset's inaccuracies. Furthermore, our model's performance is limited when addressing open-ended questions. The current dataset only encompasses four types of questions, and questions that deviate from these predefined types are not handled effectively by our model.

Despite the promising results, we recognize the need to address several challenges in future work. Our plans include improving the quality of the ViVQA dataset by conducting thorough reviews to ensure the accuracy of both questions and answers. We also intend to employ data augmentation techniques to further enrich the dataset. Additionally, as analyzed, our model still faces difficulties with questions concerning objects. To overcome this limitation, we aim to enhance object information by incorporating object detection models into the visual extraction process.

Our objective is to significantly improve the overall performance of the ViVQA system and contribute to the advancement of more efficient and reliable natural language processing in Vietnamese. By addressing these challenges, we hope to develop a more robust and accurate ViVQA system capable of handling a wider range of queries and providing better support for Vietnamese language applications.


§ ACKNOWLEDGEMENT

This research is funded by University of Science, VNU-HCM, Vietnam under grant number CNTT 2024–12.












    





























elsarticle-num-names 















 Paper Title: Aligning Multiple Knowledge Graphs in a Single Pass.











Journal of  Class Files, August 2024
Yang et al.: Interpretable and Efficient Heterogeneous Graph Convolutional Network








§ ABSTRACT 
Entity alignment (EA) is to identify equivalent entities across different knowledge graphs (KGs), which can help fuse these KGs into a more comprehensive one. Previous EA methods mainly focus on aligning a pair of KGs, and to the best of our knowledge, no existing EA method considers aligning multiple (more than two) KGs. To fill this research gap, in this work, we study a novel problem of aligning multiple KGs and propose an effective framework named MultiEA to solve the problem. First, we embed the entities of all the candidate KGs into a common feature space by a shared KG encoder. Then, we explore three alignment strategies to minimize the distances among pre-aligned entities. In particular, we propose an innovative inference enhancement technique to improve the alignment performance by incorporating high-order similarities. Finally, to verify the effectiveness of MultiEA, we construct two new real-world benchmark datasets and conduct extensive experiments on them. The results show that our MultiEA can effectively and efficiently align multiple KGs in a single pass.



Knowledge Graphs, Entity Alignment, Graph Neural Networks




Aligning Multiple Knowledge Graphs in 
 a Single Pass
    Yaming Yang,
        Zhe Wang,
        Ziyu Guan^** Corresponding author,
        Wei Zhao,
        Weigang Lu,
        Xinyan Huang

Y. Yang, Z. Wang, Z. Guan, W. Zhao, and W. Lu are with the State Key Laboratory of Integrated Services Networks, School of Computer Science and Technology, Xidian University, Xi'an, China 710071.
E-mail: {yym@, zwang@stu., zyguan@, ywzhao@mail., wglu@stu.,}xidian.edu.cn
X. Huang is the Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, the School of Artificial Intelligence, Xidian University, Xi'an, China 710071.
E-mail: xinyanh@stu.xidian.edu.cn



    August 5, 2024
=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================











§ INTRODUCTION

knowledge graphs (KGs) are a special kind of graph that can store a wealth of structural facts (i.e. knowledge) about the real world. Each fact is usually structured as a triple (h,r,t), representing that head (subject) entity h and tail (object) entity t hold relation r between them. In recent years, KGs have successfully supported many web applications such as search engines <cit.>, question-answer systems <cit.>, recommender systems <cit.>, etc.

In practice, different KGs are constructed based on diverse data sources and different extraction approaches, and a single KG can usually cover only a specific aspect of structural facts. For example, an English KG usually contains more facts about the English-speaking society, while a Chinese KG contains more facts about the Chinese-speaking society. Considering that more KGs together can provide more comprehensive structural facts from various aspects, researchers have proposed many methods (Cf. surveys <cit.>) to fuse a pair of KGs into a unified one. The general process is to first identify equivalent entities between the candidate KGs, and then let them serve as the “bridge entities” to link the candidate KGs into a unified one.

Technically, existing mainstream methods typically solve this problem by embedding entities into a common latent representation space and minimizing the distance between pre-aligned entity pairs. This process is also referred to as entity alignment (EA). Depending on how to obtain the entity embeddings, they can be classified into two main categories: (1) Trans-based EA methods <cit.> learn entity embeddings by letting each triple satisfy some specific geometric properties in the embedding space. Most of these methods adopt TransE <cit.> as the translation module to preserve the property: h + r ≈ t. (2) GNN-based EA methods <cit.> adopt graph neural network (GNN) models such as graph convolutional network (GCN) <cit.> to learn entity embeddings by iteratively aggregating the embeddings of neighbor entities.



However, all the existing EA methods are specifically designed for aligning only a pair of candidate KGs, which can cover limited knowledge for downstream applications. To the best of our knowledge, none of the existing EA methods consider aligning multiple (more than two) KGs. For example, Figure <ref> shows a case where four candidate KGs need to be aligned, which triggers a more challenging research problem. 

There are three obstacles that prevent traditional pair-wise KG alignment methods from being applied to the multiple KG alignment problem studied in this work.
First, existing EA datasets are particularly constructed for pair-wise KG alignment. At present, there is no benchmark dataset that can support the study of aligning multiple KGs.
Second, if we trivially adapt existing methods to this task, they need to be separately executed multiple times since they can only utilize the local (pair-wise) alignment information in a single pass, which is inefficient.
Last and most importantly, they cannot capture some useful global (beyond pair-wise) alignment information, affecting the final alignment performance. For the example in Figure <ref>, when they align KG-en with KG-fr, and align KG-en with KG-zh independently, the entities of the three KGs will be projected into four separate embedding spaces. Consequently, it is easy to yield inconsistent results like e_1^(en)≡ e_1^(fr), e_1^(en)≡ e_1^(zh), and e_1^(fr)≡ e_2^(zh) (wrong), violating the transitivity of the alignment relationship.

To fill the research gap in this field, in this work, we innovatively study the problem of aligning multiple KGs in a single pass. First of all, to facilitate the study of this problem, we construct two new benchmark datasets that contain multiple KGs to be aligned. Then, we formulate a novel research problem of aligning multiple KGs concurrently. Finally, we propose an effective framework named MultiEA to solve the problem. As illustrated in Figure <ref>, our MultiEA can concurrently align the four candidate KGs in a single pass, which is more practical. In addition, it embeds the entities of all the candidate KGs into a unified space and thus can capture useful global alignment information, which helps achieve better alignment performance. Recall the example above, if MultiEA pulls e_1^(en) and e_1^(fr) closer and pulls e_1^(en) and e_1^(zh) closer. Then, e_1^(fr) and e_1^(zh) will definitely be pulled closer as well, leading to the consistent result: e_1^(en)≡ e_1^(fr), e_1^(en)≡ e_1^(zh), and e_1^(fr)≡ e_1^(zh) (right). 

The main contributions of this work are summarized as follows.


  * To the best of our knowledge, in the EA research community, we are the first to study the problem of concurrently aligning multiple (more than two) KGs. We are also the first to construct the benchmark datasets and define the evaluation metric for this problem.

  * We propose an effective framework named MultiEA to solve the problem. Three alignment strategies are explored to capture the global alignment information. An innovative inference enhancement technique is proposed to significantly boost the alignment performance.

  * We conduct extensive experiments on the two constructed benchmark datasets. The results demonstrate that MultiEA can effectively and efficiently align multiple KGs in a single pass. We will release our source codes and datasets to facilitate further research on this problem.






§ RELATED WORK

In this section, we comprehensively review existing methods that are related to ours.


 Trans-based EA Methods. Early EA methods exploit KG embedding methods, e.g., TransE <cit.>, to project a pair of candidate KGs into a common low-dimensional Euclidean space by letting the embeddings of head entity h, relation r, and tail entity h satisfy the triangular relationship: h + r ≈ t. Given a set of pre-aligned entity pairs (i.e., seed alignment labels), they minimize the distance between these equivalent entity pairs in the embedding space. Thus, more potential equivalent entity pairs can be discovered after model optimization. Representative Trans-based EA methods include JEwP <cit.>, JAPE <cit.>, MTransE <cit.>, IPTransE <cit.>, BootEA <cit.>, TransEdge <cit.>, MultiKE <cit.>, etc. 


 GNN-based EA Methods. Motivated by the extraordinary success of graph neural networks (GNNs) in extracting the structural features of graphs, GCN-Align <cit.> first introduces GCN <cit.> into EA, which has shown superior performance than Trans-based EA Methods, e.g., JEwP <cit.>, MTransE <cit.> and JAPE <cit.>. Further, AVR-GCN <cit.> deftly combines the graph convolutional operator in GNNs and the translation operator in Trans-based methods to extract better entity embeddings. Afterward, a series of following GNN-based EA methods, e.g.,  <cit.> are proposed. Recently, RREA <cit.> insightfully summarizes KG embedding-based EA methods and GNN-based EA methods into a unified framework consisting of two modules: shape-builder and alignment. The former constraints KGs into a specific distribution in the embedding space, and the latter minimizes the distance between the embeddings of pre-aligned entity pairs.


 EA Enhancement. In recent years, several works further enhanced the EA performance by exploiting attribute information and entity name information. HMAN <cit.>, BERT-INT <cit.>, and ACK-MMEA <cit.> can leverage the rich attributes of entities. Besides, PSR <cit.>, GMNN <cit.>, NMN <cit.>, HGCN-JE <cit.>, RDGCN <cit.>, LightEA <cit.> and SelfKG <cit.> can use the word embeddings of the entity names to effectively initialize entity embeddings, greatly improving the performance of EA.


 Alignment Methods in Other Fields.
In the social network analysis field, there are several methods that have made some efforts to align multiple social networks. COSNET <cit.> leverages multiple networks to enhance the alignment performance of two networks. It requires exponential time complexity to build a matching graph, which is impractical for many scenarios. ULink <cit.> learns a latent user space based on user attributes. It cannot exploit structural information and thus it is not a strict social network alignment approach. MC^2 <cit.> combines both structural information and attribute information to infer a common base by matrix factorization. It requires all social networks to have attributes, and its time computational complexity is square to the number of uses. MASTER <cit.> embeds multiple social networks in a common latent space through collaborative matrix factorization, which also requires square computational complexity. In the data integration field, a recent work <cit.> is proposed to align multiple tables based on Sentence-BERT <cit.>, table-wise hierarchical merging, and density-based pruning. These methods are specially designed methods for other fields and they generally require sophisticated optimization.

In summary, different from existing methods, in this work, we study the problem of aligning multiple (more than two) KGs based on the structural information. Accordingly, we develop a GNN-based neural network framework named MultiEA to solve the problem in an end-to-end optimization manner.





§ PRELIMINARIES


In this section, we first give the formal definition of knowledge graphs. Then, we formulate the novel problem of aligning multiple KGs.


Knowledge Graph (KG).
A knowledge graph is defined as 𝒢 = (ℰ, ℛ, 𝒯), where ℰ is the set of entities (nodes) and ℛ is the set of relations (edges). 𝒯⊆ℰ×ℛ×ℰ is the set of triples, and each triple t ∈𝒯 is represented as <e_i, r_k, e_j>, which means that head entity e_i ∈ℰ and tail entity e_j ∈ℰ hold relation r_k ∈ℛ between them. We use 𝐡_i∈ℝ^d to denote the representation of entity e_i in one model layer, where d is the dimensionality of embeddings. We use 𝐠_k∈ℝ^d to denote the embedding of relation r_k, which is a randomly initialized learnable parameter vector.


Multiple KG Alignment.
The input is a set of KGs {𝒢^(1), 𝒢^(2),  𝒢^(m), ..., 𝒢^(M)}, where M is the number of the input KGs, and M>2. The m-th KG is denoted by 𝒢^(m) = (ℰ^(m), ℛ^(m), 𝒯^(m)).
The problem is to identify equivalent entities that refer to the same real-world thing across all the M KGs, based on a set of seed alignment labels (i.e., pre-aligned entities), denoted as 𝒮 = { l_1, l_2, ..., l_n, ..., l_N }, where N is the number of labels. The n-th label is denoted as l_n = (e_n^(1), e_n^(2), ..., e_n^(m), ..., e_n^(M)), where entity e_n^(m)∈ℰ^(m), and all the M entities associated with l_n are equivalent.

As shown in Figure <ref>, KG-en, KG-zh, KG-fr, and KG-ja are four candidate KGs that need to be aligned, where the four entities marked with red dashed lines refer to the same object, i.e., “Sweden”. We treat the four equivalent entities as one of the seed alignment labels, to help our EA algorithm find more potential equivalent entities. Finally, we can link (merge) the four KGs based on these detected equivalent entities across the four KGs.






§ FRAMEWORK

In this section, we describe our proposed MultiEA framework in detail. Firstly, we design a GNN-like KG encoder to embed the entities of all the candidate KGs into a common vector space. Then, we propose three alignment strategies to measure the distances among pre-aligned entities, which guide the training of EA models. Finally, we develop an innovative inference enhancement technique to boost the alignment performance by incorporating high-order similarities.





















 §.§ KG Encoding


Given a set of KGs: {𝒢^(1), 𝒢^(2), ..., 𝒢^(m), ..., 𝒢^(M)}, we use a shared KG-oriented GNN encoder to compute their entity embeddings. In the following, we take the m-th KG, i.e., 𝒢^(m) = (ℰ^(m), ℛ^(m), 𝒯^(m)) as an example to elaborate on the details of our encoder. The superscript m is omitted for the sake of notation simplicity.

Firstly, we augment the original KG data to add a virtual self-relation r_self into the relation set ℛ, formally described as follows:

    ℛ = ℛ∪{ r_self}.

Then, for each entity, we add a virtual triple to describe that each entity e_i has the self-relation r_self between itself, which is added to the triple set as follows:

    𝒯 = 𝒯∪{ <e_i, r_self, e_i> | e_i ∈ℰ}.

To facilitate the aggregation of the GNN encoder, for entity e_i, we define the set of its neighbor relation-entity tuples as follows:

    𝒩_i = { <r_k, e_j> | <e_i, r_k, e_j> ∈𝒯, or <e_j, r_k, e_i> ∈𝒯}.

Although the relations in KGs are usually directed, following most previous works, we treat them as bi-directed (or undirected). This is reasonable because the inverse of each relation is usually intuitive as well. For instance, for the triple <Obama, wife, Michelle>, we can also rewrite it as: <Michelle, husband, Obama>. Note that due to the addition of self-connection, entity e_i will have a special neighbor relation-entity tuple: <r_self, e_i> ∈𝒩_i.

In a layer of our KG encoder, the output representation of entity e_i is computed as follows:

    𝐡_i' = σ( ∑_<r_k, e_j> ∈𝒩_iα_i,k,j·𝐖_k·𝐡_j),

where 𝐡_j is the neighbor entity e_j's feature vector output by the previous layer, 𝐖_k is a relation-specific projection matrix, α_i,k,j is the attention coefficient for aggregation, and σ is the non-linear activation function.

There are often thousands of relations in common KGs, and thus the relation-specific projection matrices 𝐖_k, ∀ r_k ∈ℛ may introduce too many trainable parameters, risking overfitting. Therefore, following previous studies <cit.>, we leverage the relation embedding vector 𝐠_k to define its related projection matrix, as follows:

    𝐖_k= 𝐈 - 2 ·𝐠_k·𝐠_k^T.

In this way, we no longer need to introduce additional parameters. In particular, the normalization constraint 𝐠_k_2 = 1 is imposed, so that the orthogonality of 𝐖_k is naturally guaranteed since we can easily derive: 
    𝐖_k^T·𝐖_k = (𝐈 - 2 ·𝐠_k·𝐠_k^T)^T (𝐈 - 2 ·𝐠_k·𝐠_k^T) = 𝐈.

The orthogonality of 𝐖_k is proven to be very beneficial for EA task <cit.>.

The attention coefficient is computed as follows. Given a neighbor tuple <r_k, e_j> ∈𝒩_i of entity e_i, we first compute the proximity among the head entity e_i, the relation r_k, and the tail entity e_i:

    β_i,k,j = σ( 𝐚_h^T ·𝐡_i + 𝐚_r^T ·𝐠_k + 𝐚_t^T ·𝐖_k·𝐡_j),

where 𝐚_h, 𝐚_r, and 𝐚_t are learnable parameter vectors for head entities, relations, and tail entities, respectively. Then, we compute the attention coefficient α_i,k,j by normalizing β_i,k,j across all the elements of 𝒩_i:

    α_i,k,j = exp( β_i,k,j)/∑_<r_x, e_y> ∈𝒩_iexp( β_i,x,y).

Note that by adding the self-relation r_self, each entity e_i itself acts as one of its own neighbors (Cf. Eqs. (<ref>, <ref>)). In addition, we use the attention mechanism to aggregate all the neighbors (Cf. Eqs. (<ref>, <ref>)). Thus, as proven by <cit.>, our encoder is able to automatically learn the importance of arbitrary hops of neighborhood information. This is more flexible than <cit.>, which directly concatenates all hops of neighborhood information.

Recall that the encoder is shared among all the KGs, and thus we can obtain the embeddings of the entities from every KG.





 §.§ Training for Alignment


In order to pull equivalent entities together in a unified embedding space, we adopt the following margin ranking loss as the training loss of our MultiEA model:

    L=∑_l ∈𝒮∑_l' ∈𝒮'max(d(l) - d(l') + λ, 0),

where 𝒮 is the set of positive examples, i.e., ground-truth alignment labels, 𝒮' is the set of negative examples, i.e., randomly generated false alignment labels, d(·) is a distance function, and  λ > 0 is a margin hyperparameter for separating positive and negative examples.

The negative examples in 𝒮' are generated as follows. For each positive example l = (e^(1), e^(2), ..., e^(m), ..., e^(M)) ∈𝒮, we fix one entity and replace the other entities by randomly sampling from their corresponding KGs, formally described as follows:

    η×{[ (e^(1), ẽ^(2), ..., ẽ^(m), ..., ẽ^(M)),; (ẽ^(1), e^(2), ..., ẽ^(m), ..., ẽ^(M)),;                                    ...,; (ẽ^(1), ẽ^(2), ..., e^(m), ..., ẽ^(M)),;                                    ...,;  (ẽ^(1), ẽ^(2), ..., ẽ^(m), ..., e^(M));                                         ]},

where ẽ^(m) is randomly sampled entity from ℰ^(m) to replace the original entity e^(m). Here, η is an integer, indicating that for each positive example, we generate η groups of negative examples in this way. Thus, each positive example corresponds to η× M negative examples.

In this work, we propose three strategies for minimizing the distance among equivalent entities. The ideas are intuitively shown in Figure <ref>. For a positive example l = (e^(1), e^(2), ..., e^(m), ..., e^(M)) ∈𝒮, we firstly obtain its associated entity embeddings from the KG encoder and denote them as: (𝐡^(1), 𝐡^(2), ..., 𝐡^(m), ..., 𝐡^(M)). Then, the distance d(l) is minimized by three strategies as follows.



 (1) Moving toward Mean.
This strategy lets all the equivalent entities approach their mean, as illustrated in Figure <ref>. We first compute the mean as follows:

    𝐮 = 1/M·∑_m=1^M𝐡^(m).

Then, we sum the Euclidean distances between each entity embedding and the mean:

    d(l) = ∑_m=1^M𝐡^(m) - 𝐮_2.



 (2) Moving toward Anchor.
In practice, the candidate KGs are usually unbalanced due to various realistic factors. For instance, on our constructed multi-lingual KG dataset DBP-4, the English KG contains many more triples than the other three KGs (Cf. Table <ref>). Therefore, as shown in Figure <ref>, we designate entities from KG DBP-4: en as anchor entities and let the equivalent entities from the other KGs approach these anchor entities. This idea is formally described as follows:

    d(l) = ∑_m=1^M𝐡^(m) - 𝐡^(a)_2,

where 𝐡^(a) is embedding of the anchor entity[Note that when 𝐡^(m) is designated as anchor entity, then 𝐡^(m) - 𝐡^(a)_2 = 0.].


 (3) Moving toward Each Other.
This strategy lets equivalent entities approach each other. As illustrated in Figure <ref>, we minimize the distances between all possible pairs of equivalent entities, formally described as follows:

    d(l) = ∑_m_1 ≠ m_2𝐡^(m_1) - 𝐡^(m_2)_2,

where m_1 and m_2 are in the set { 1, 2, ..., m, ..., M }.








 §.§ Training Time Complexity


Let |ℰ|, |ℛ|, and |𝒯| denote the total numbers of the entities, the relations, and the triples of all the M candidate KGs, respectively. Let d denote the dimensionalities of entity embeddings and relation embeddings. Let |𝒮| denote the number of alignment labels. Recall that each label corresponds to η× M negative examples.

In the KG encoding phase (Cf. Section <ref>), the time complexity involves three main computational operations. First, the aggregation operation described by Eq. (<ref>) has time complexity 𝒪( |ℰ| · d^2 ). Second, the computation of relation-specific projection matrices described by Eq. (<ref>) has time complexity 𝒪( |ℛ| · d^2 ). Finally, as described by Eqs. (<ref>, <ref>), the computation of attention coefficients has time complexity 𝒪( |𝒯| · (d + d + d^2) ). Considering that in practice, the embedding dimensionality d is a relatively small value, the time complexity of KG encoding is 𝒪( |ℰ| + |ℛ| + |𝒯| ). 

In the alignment phase (Cf. Section <ref>), as described by Eq. (<ref>) and Eq. (<ref>), both the mean strategy and the anchor strategy require time complexity 𝒪(M). As described by Eq. (<ref>), the each other strategy requires time complexity 𝒪(M^2). As described by Eq. (<ref>), taking the number of labels into consideration. The mean strategy and the anchor strategy have time complexity of 𝒪(|𝒮| ·η· M^2), and each other strategy has time complexity 𝒪(|𝒮| ·η· M^3).

In practice, |𝒮|, η, and M are much smaller than |ℰ|, |ℛ|, and |𝒯|. Therefore, the overall training time complexity of MultiEA is equal to 𝒪( |ℰ| + |ℛ| + |𝒯| ).




 §.§ Inference Enhancement


After training the model, we can obtain the embeddings of the entities from all the KGs. Based on these embeddings, we can compute the similarity between any two entities that are from different KGs. For example, given entity e_i^(m_1)∈ℰ^(m_1), and entity e_j^(m_2)∈ℰ^(m_2), we compute the similarity between them as follows[Since entity embeddings are normalized, i.e., 𝐡_i_2 = 1, ∀ i, the computed similarities lie in the range [0,1]. Thus, we can compose high-order similarities by matrix product operation later.]:

    𝐒_i,j^m_1 - m_2 = 1 - 𝐡_i^(m_1) - 𝐡_j^(m_2)_2/2.


Thus, we can obtain a matrix 𝐒^m_1 - m_2 to describe the entity similarities between 𝒢^(m_1) and 𝒢^(m_2). However, Eq. (<ref>) only captures the first-order similarity, which may not be sufficient since some higher-order similarity information is ignored. Therefore, in this work, we propose to further enhance the similarity matrix by incorporating higher-order similarities, which can be composed by matrix product operation. For example, if there are three KGs, 𝒢^(m_1), 𝒢^(m_2), and 𝒢^(m_3), we can enhance the similarity matrix 𝐒^m_1 - m_2 as follows:

    𝐒^m_1 - m_2 = γ_1 ·𝐒^m_1 - m_2 + γ_2 ·𝐒^m_1 - m_3·𝐒^m_3 - m_2,

where γ_1 and γ_2 are hyperparameters to balance the two terms. They are real numbers in the range [0,1] and satisfy γ_1 + γ_2 = 1. Similarly, if there are four KGs, we can enhance 𝐒^m_1 - m_2 as follows:

    𝐒^m_1 - m_2   = γ_1 ·𝐒^m_1 - m_2 + γ_2 ·𝐒^m_1 - m_3·𝐒^m_3 - m_2
       + γ_3 ·𝐒^m_1 - m_4·𝐒^m_4 - m_2,

where γ_1, γ_2, and γ_3 are hyperparameters in the range [0,1] and satisfy γ_1 + γ_2 + γ_3 = 1.

Here, we only incorporate the two-order similarities by the matrix product operation. In practice, higher-order similarities can be similarly incorporated through more matrix product operations. Finally, we can identify equivalent entities according to the enhanced similarity matrix. In this way, the model can utilize more comprehensive information, helping improve the inference performance.





§ EXPERIMENT


In this section, we construct the benchmark dataset and define the evaluation metric for the new task. Then, we conduct extensive experiments to verify the effectiveness of our MultiEA.



 §.§ Benchmark Datasets


As far as we know, there is no existing EA benchmark dataset that can support the task of aligning multiple KGs. To this end, we construct two novel benchmark datasets based on previously widely used real-world datasets <cit.>. One dataset DBP-4 contains four candidate KGs, and another dataset DWY-3 contains three candidate KGs to be aligned. The key statistics of the constructed datasets are listed in Table <ref>.




•DBP-4.
This dataset contains four KGs from different language sources, including English (en) KG, Chinese (zh) KG, Japanese (ja) KG, and French (fr) KG. It is constructed based on the DBP15K dataset, which was originally released by <cit.>. The original alignment labels describe the correspondence from English KG to the other three KGs, which can be described as: en-fr, en-zh, and en-ja. We treat the English KG as an intermediary and complement the alignment information among all four KGs, which can be described as: en-fr-zh-ja. Specifically, we filter entities from different KGs that correspond to the same English entity. Intuitively, if we have: e^(en) = e^(fr), e^(en) = e^(zh), and e^(en) = e^(ja). Then, we can infer: e^(en) = e^(fr) = e^(zh)= e^(ja), which means that all these four entities from different KGs are equivalent.

•DWY-3.
This dataset contains three KGs, i.e., DBpedia (dbp), Wikidata (wiki), and YAGO (yago). It is constructed based on the DWY100K dataset, which was originally released by <cit.>. Its original alignment labels describe the correspondence from DBpedia to the other two KGs, which can be described as: dbp-wiki, and dbp-yago. Similar to the construction of DBP-4, we treat the DBpedia KG as an intermediary to complement the alignment information among all three KGs, which can be described as: dbp-wiki-yago.

It is worth noting that the condition under which we construct labels is very strict. As a result, the number of labels is very small compared to the number of entities and triples. To address the issue, we further induce a smaller knowledge graph. Specifically, inspired by the previous work <cit.>, for each entity involved in labels, we select its popular neighbor entities whose degrees are larger than a threshold (we set 15 in this work). This target entity and its high-degree neighbor entries are added to a new entity set, and the involved relations are added to a new relation set. Then, we induce a new set of triples based on the selected entities and relations.







 §.§ Evaluation Metric

Traditional pair-wise KG alignment methods generally use Hits@K to evaluate the alignment performance. Specifically, given two KGs: 𝒢^(l) and 𝒢^(r), we first consider aligning 𝒢^(r) to 𝒢^(l), and define a temporary metric l_Hits@K to measure the proportion of the entities in 𝒢^(l) whose counterpart in 𝒢^(r) rank in top-K:

    l_Hits@K = ∑_n=1^N𝕀(e_n^(r)ranks in top-K)/N,

where 𝕀 is the indicator function that outputs 1 if the input fact is true and 0 otherwise. Then, we consider aligning 𝒢^(l) to 𝒢^(r), and similarly define a temporary metric r_Hits@K in the opposite direction. The final metric Hits@K is computed as the average of the two temporary metrics:

    Hits@K = 1/2· (l_Hits@K + r_Hits@K).


Unfortunately, Hits@K does not work for our task of multiple KG alignment. Hence, in the spirit of Hits@K, we define a novel evaluation metric. Given M KGs: {𝒢^(1), 𝒢^(2), ..., 𝒢^(m), ..., 𝒢^(M)} where M>2, we treat each KG 𝒢^(m) as the target KG, and consider aligning the other M-1 KGs to it. First, we count the number of entities from 𝒢^(m) whose all M-1 counterparts from the other M-1 KGs rank in top-K, described as follows:

    C = ∑_n=1^N𝕀(e_n^(1)...,e_n^(m-1),e_n^(m+1)...,e_n^(M)rank in top-K),

where 𝕀(·) is an indicator function that outputs 1 if all the input entities rank in top-K, and 0 otherwise. Then, a temporary metric m_Hits@K is defined to compute the proportion of:

    m_Hits@K = C/N.

The final metric is computed as the average of all the M temporary metrics: 

    M-Hits@K = 1/M·∑_m=1^Mm_Hits@K.

Obviously, our newly defined metric M-Hits@K is more challenging than the traditional metric Hits@K since the condition for counting a correct entity is much more strict. This can better reflect the model performance of aligning multiple KGs.




 §.§ Baselines and Variants

Regarding baselines, we select six representative supervised EA baselines that only leverage the structural information of candidate KGs, including two Trans-based EA methods (1), (2), and four GNN-based EA methods (3), (4), (5), (6). They are listed as follows:


  * MTransE <cit.> is a Trans-based EA method that uses TransE <cit.> to encode entities of each KG in a separated embedding space;

  * IPTransE <cit.> further uses PTransE <cit.> to encode KGs and propose an iterative alignment strategy;

  * GCN-Align <cit.> is the first GNN-based EA method that uses GCN <cit.> to encode two KGs;

  * NAEA <cit.> further uses GAT <cit.> to encode the structural information of KGs. Therefore, it learns entity embeddings by aggregating neighbor entities with different importance;

  * KECG <cit.> also uses GAT <cit.> to encode KGs, and additionally restricts the projection matrix to be a diagonal matrix, reducing the number of parameters and computations;

  * PSR <cit.> encodes KGs based on orthogonal projection matrix and attention aggregation like ours. Differently, (1) it dose not add self-connections, and (2) it directly concatenates all hops of neighborhood information.


Recall that in Section <ref>, we introduce three different alignment strategies. Here, we use +mean, +anchor, and +each to denote the three strategies, respectively. In Section <ref>, we propose the inference enhancement module. Here, we use +infer or -infer to indicate whether the module is equipped. By considering all the permutations, we set the following six variants for our MultiEA:



  * MultiEA+mean-infer;

  * MultiEA+anchor-infer;

  * MultiEA+each-infer;

  * MultiEA+mean+infer;

  * MultiEA+anchor+infer;

  * MultiEA+each+infer.










For anchor-based variants, we treat DBP-4: en and DWY-3: dbp as the anchor KGs on the two datasets, respectively.












 §.§ Implementation Details


For the constructed benchmark datasets of DBP-4 and DWY-3, we follow the convention to randomly select 30% elements of the seed alignment set 𝒮 to form the training set, and the rest are treated as the test set. We implement our MultiEA based on a PyTorch-based open-source EA toolkit named EAkit <cit.>. For baselines, we use their implementations provided in the EAkit package. All the experiments are conducted on our constructed benchmark datasets, and all the methods use the same dataset partitioning.

For our MultiEA, we use the same hyperparameter settings for both datasets. This is a challenging configuration since it can better reflect the sensitivity of hyperparameters w.r.t. datasets. All the model parameters are randomly initialized by the Xavier uniform distribution <cit.>. We adopt the Adam optimizer, and the learning rate is set to 0.01. The embedding dimensionalities of both entities and relations are set to 256. The non-linear activation function σ is set to the ELU function <cit.>. The margin hyperparameter λ is set to 1. The number of negative example groups, i.e., the hyperparameter η is set to 10. We adopt early stopping for model training, and the number of patience steps is set to 10.

All the experiments are conducted on an NVIDIA TITAN RTX GPU with 24GB GPU memory and 128GB main memory.










 §.§ Multiple KG Alignment


We first quantitatively evaluate the effectiveness of our MultiEA in aligning multiple (more than two) KGs. Specifically, we apply all six baseline methods as well as all six variants of MultiEA to the two constructed benchmark datasets, and use the metric M-Hits@K newly defined in Eq. (<ref>) to compare their accuracy. The results are reported in Table <ref>.


 Comparison with Baselines. It is worth noting that we cannot directly apply baseline EA methods to the multiple KG alignment task since they are specifically designed for the traditional pair-wise KG alignment task. To make them compatible, we split the concerned task into multiple pair-wise sub-tasks and separately align each pair of KGs. To reflect this, baselines are marked by ^* in the table. We can see that our MultiEA (especially the three variants with the inference enhancement module at the bottom of the table) can significantly outperform all the baselines on the two datasets. This may be due to that our MultiEA can effectively capture the global alignment information by projecting the entities of all the candidate KGs in a unified embedding space. Besides, GNN-based EA baselines show better performance than Trans-based EA baselines. This is consistent with the findings of many previous studies <cit.>.


 Abalation Studies. As we can see, our MultiEA+each+infer achieves the best overall results in most cases, showing its superior effectiveness in aligning multiple KGs. Overall, MultiEA+anchor+infer and MultiEA+mean+infer achieve the second-best and the third-best results, respectively. The situation is the same for the other three variants without the inference enhancement module. Therefore, we recommend the “each other" strategy as the default strategy for most cases. If users prefer higher efficiency and there is an obvious anchor KG, we recommend the “anchor" strategy, and otherwise, we recommend the “mean" strategy. From another dimension, we can observe that all three variants with the inference enhancement module significantly outperform the other three variants without this module. This indicates that the inference enhancement module is significantly beneficial for this task.





 §.§ Case Study


To intuitively show the effectiveness of our MultiEA, in Figure <ref> and Figure <ref>, we list ten groups of equivalent entities discovered by MultiEA (the best-performing variant MultiEA+each+infer) on the two datasets. As we can see, MultiEA can effectively discover equivalent entities across all the candidate KGs.

Further, we visualize the embeddings of these entities. Specifically, we utilize the t-SNE algorithm <cit.> to project them into the 2-dimensional Euclidean space. Figure <ref> and Figure <ref> show the results on DBP-4 and DWY-3, respectively, where different colors and different shapes mark the entities from different KGs, and the equivalent entities are connected by black lines. As we can see, on both datasets, the ten groups of equivalent entities are obviously gathered together, indicating the effectiveness of the alignment mechanism of MultiEA. On the other hand, a proper margin is maintained between different groups, avoiding the risk of model overfitting. The result on DWY-3 shows a relatively better distribution. This is consistent with the higher accuracy on DWY-3 as reported in Table <ref>.




 §.§ Efficiency Study


In this experiment, we demonstrate the efficiency advantage of our MultiEA in aligning multiple KGs. As shown in Table <ref> and Table <ref>, we first report the experimental results of our re-implemented GNN-based EA baselines as described in Section <ref>. That is, they align multiple candidate KGs by splitting this task into multiple separate pair-wise sub-tasks, as marked by ^* in the tables. Then, we apply the “each other” alignment strategy and the inference enhancement module of our MultiEA+each+infer to each baseline X, as denoted as MultiEA (X) in the tables. 

In addition to the M-Hits@K scores, we report the number of model parameters, the memory, and the time required by each method during its training. As we can see, our MultiEA can help each baseline achieve significantly better performance. Moreover, MultiEA can help each baseline significantly reduce the requirements of parameters, space, and time resources. This demonstrates the superior efficiency of our MultiEA.





 §.§ Pair-wise KG Alignment


We also apply our MultiEA (the variant MultiEA+each+infer) to the traditional pair-wise KG alignment task. Firstly, based on the two constructed benchmark datasets, we induce several pair-wise KG alignment sub-datasets. For DBP-4, we induce three sub-datasets, i.e., DBP-4: en-fr, DBP-4: en-ja, and DBP-4: en-zh. For DWY-3, we induce two sub-datasets, i.e., DWY-3: dbp-wiki and DWY-3: dbp-yago. For baselines, we reproduce their results on our induced datasets. Then, we re-implement the encoder of MultiEA as that used in a baseline X, denoted as MultiEA (X). This can eliminate the encoder's influence on the alignment results, and thus investigate whether our proposed “each other" strategy and inference enhancement module helps improve the performance of baselines on this task. By convention, we use the widely used metric Hits@K described by Eq. (<ref>) to evaluate the alignment accuracy.

Table <ref> and Table <ref> shows the final results. In addition to the original experimental results, we also compute the relative gain to show the improvement more intuitively. As we can see, our MultiEA can always help improve the baselines' performance on the traditional pair-wise KG alignment task. Besides, we have some other findings as follows. PSR, KECG, and NAEA perform better than GCN-Align, indicating the effectiveness of the attention mechanism used in their KG encoders. Moreover, MultiEA (PSR), MultiEA (KECG), and MultiEA (NAEA) outperform MultiEA (GCN-Align). This demonstrates that our MultiEA framework can effectively exploit the structural information captured by the graph attention mechanisms. Two Trans-based methods MTransE and IPTransE show worse performance than GNN-based methods, which is consistent with the conclusions of previous works <cit.>. 

The experimental results reported here are slightly different from the results reported in previous studies. The main reason lies in that our constructed datasets are different from existing datasets. For example, the label ratio on our DBP-4 is 2539 × 4/8901+3545+4326+3893≈ 0.4915, while the label ratio of DBP15K used in previous studies is (15000 × 2/19388+19572+15000 × 2/19780+19814+15000 × 2/19993+19661) ×1/3≈ 0.7614, which is much larger than ours.









 §.§ Hyperparameter Study




In this subsection, we study the impact of MultiEA's four key hyperparameters on multiple KG alignment accuracy.

First, we study the balance hyperparameters introduced in the inference enhancement module, as described by Eqs. (<ref>, <ref>). In practice, we only search the weight for the first term, and the weights for the other terms are set to the same value, thus compressing the hyperparameter search space. For one example of DBP-4 dataset, we can rewrite Eq. (<ref>) as: 

    𝐒^en - zh   = γ·𝐒^en - zh + 1 - γ/2·𝐒^en - fr·𝐒^fr - zh
       + 1 - γ/2·𝐒^en - ja·𝐒^ja - zh.

For another example of DWY-3 dataset, we can rewrite Eq. (<ref>) as: 

    𝐒^dbp - wiki = γ·𝐒^dbp - wiki + (1 - γ) ·𝐒^dbp - yago·𝐒^yago - wiki.
 
The sensitivity of γ on the two datasets is shown in Figure <ref> and Figure <ref>, respectively. We can see that on both datasets, our MultiEA achieves the best performance when γ = 0.2. It means that the importance of the first-order similarity is 0.2 and the importance of the other higher-order similarities is 0.8. This indicates that incorporating high-order similarities is very beneficial for the multiple KG alignment task, demonstrating the effectiveness of our proposed inference enhancement module. This finding is also consistent with the observation in Section <ref>.

Second, we study the sensitivity of the margin hyperparameter λ introduced in the loss function as described by Eq. (<ref>). The results are shown in Figure <ref> and Figure <ref>. As we can see, the performance is very poor when λ = 0, because the model cannot separate the positive examples and the negative examples, causing the underfitting issue. There is a clear inflection point when λ = 1, and thus in the other experiments, we set it to 1 by default. After that, the performance gradually declines, probably because the model suffers the overfitting issue when the margin is too large.

Third, we investigate the sensitivity of η, which has been introduced to describe the groups of negative examples. As shown in Figure <ref> and Figure <ref>, our model is not sensitive to this hyperparameter. Therefore, in practice, we set η to 10 to save computational resources.

Finally, we show the model performance w.r.t. the training ratio in Figure <ref> and Figure <ref>. As expected, the training ratio shows a clear positive effect on improving the model performance, which is consistent with the findings of many previous studies, e.g., <cit.>.





§ CONCLUSION

In this work, we note that existing methods mainly focus on aligning only a pair of candidate KGs, ignoring the multiplicity of the candidate KGs to be aligned. To fill the research gap in this field, we formulate a novel problem of aligning multiple (more than two) KGs and propose the MultiEA framework to effectively solve this problem. To enhance the inference performance, we innovatively propose to incorporate higher-order similarities. Last but not least, we construct two new benchmark datasets and define the evaluation metric for the problem. The experimental results show that our MultiEA framework can effectively and efficiently align multiple KGs in a single pass. We will make our source codes and the constructed benchmark datasets publicly available to facilitate further research on this problem, which we believe will inspire more interesting works in the future.





§ ACKNOWLEDGMENTS

This work was supported in part by the National Natural Science Foundation of China under Grants 62133012, 61936006, 62303366, and 62073255, in part by the Innovation Capability Support Program of Shaanxi under Grant 2021TD-05, and in part by the Fundamental Research Funds for the Central Universities under Grants QTZX23039, XJSJ23030.



IEEEtran


[
    < g r a p h i c s >
]Yaming Yang received the B.S. and Ph.D. degrees in Computer Science and Technology from Xidian University, China, in 2015 and 2022, respectively. He is currently a lecturer with the School of Computer Science and Technology at Xidian University. His research interests include data mining and machine learning on graph data.


[
    < g r a p h i c s >
]Zhe Wang received the B.S. degree in Management from Hefei University of Technology, China, in 2020. He is currently working towards a Ph.D. degree with the School of Computer Science and Technology, Xidian University, China. His research interests include data mining and machine learning on knowledge graph data.


[
    < g r a p h i c s >
]Ziyu Guan received the B.S. and Ph.D. degrees in Computer Science from Zhejiang University, Hangzhou China, in 2004 and 2010, respectively. He had worked as a research scientist in the University of California at Santa Barbara from 2010 to 2012, and as a professor in the School of Information and Technology of Northwest University, China from 2012 to 2018. He is currently a professor with the School of Computer Science and Technology, Xidian University. His research interests include attributed graph mining and search, machine learning, expertise modeling and retrieval, and recommender systems.


[
    < g r a p h i c s >
]Wei Zhao received the B.S., M.S. and Ph.D. degrees from Xidian University, Xi’an, China, in 2002, 2005 and 2015, respectively. He is currently a professor in the School of Computer Science and Technology at Xidian University. His research direction is pattern recognition and intelligent systems, with specific interests in attributed graph mining and search, machine learning, signal processing and precision guiding technology.




[
    < g r a p h i c s >
]Weigang Lu received the B.S. degree in Internet of Things from Anhui Polytechnic University, China, in 2019. He is currently working towards a Ph.D. degree with the School of Computer Science and Technology, Xidian University, China. His research interests include data mining and machine learning on graph data.




[
    < g r a p h i c s >
]Xinyan Huang received the B.S. degree from the School of Electronic Science and Technology, Shaanxi University of Science and Technology, Xi'an, China, in 2015, and the M.S. degree from the School of Physics and Information Technology, Shaanxi Normal University, Xi' an, in 2017. She is currently working towards a Ph.D. degree with the School of Artificial Intelligence, Xidian University, China. Her research interests include machine learning, computer vision, and pattern recognition.




 Paper Title: Generative Sentiment Analysis via Latent Category Distribution and Constrained Decoding.









Generative Sentiment Analysis via LCD and CD







Jun Zhou et al.




    Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, China 
    
    Beijing Rigour Technology Co., Ltd., China
    School of Cyberspace, Hangzhou Dianzi University, China
    North China Power Engineering Co., Ltd. of China Power Engineering Consulting Group, China

    
    
    
    
    dhji@whu.edu.cn
    
    
    

    
    



J. Zhou et al.









Generative Sentiment Analysis via Latent Category Distribution and Constrained Decoding
    Jun Zhou1 Dongyang Yu2 Kamran Aziz 1 Fangfang Su3 Qing Zhang 4 Fei Li 1 Donghong Ji 1()
    
    July 2024
================================================================================================






	
	F. Author et al.
	
	
	
	Princeton University, Princeton NJ 08544, USA Springer Heidelberg, Tiergartenstr. 17, 69121 Heidelberg, Germany
	lncs@springer.com

	<http://www.springer.com/gp/computer-science/lncs> ABC Institute, Rupert-Karls-University Heidelberg, Heidelberg, Germany

	{abc,lncs}@uni-heidelberg.de
	
	Generative Sentiment Analysis via Latent Category Distribution and Constrained Decoding
    First Author10000-1111-2222-3333 Second Author2,31111-2222-3333-4444 Third Author32222–3333-4444-5555
    July 2024
=========================================================================================================


	



§ ABSTRACT 

Fine-grained sentiment analysis involves extracting and organizing sentiment elements from textual data. However, existing approaches often overlook issues of category semantic inclusion and overlap, as well as inherent structural patterns within the target sequence. This study introduces a generative sentiment analysis model. To address the challenges related to category semantic inclusion and overlap, a latent category distribution variable is introduced. By reconstructing the input of a variational autoencoder, the model learns the intensity of the relationship between categories and text, thereby improving sequence generation. Additionally, a trie data structure and constrained decoding strategy are utilized to exploit structural patterns, which in turn reduces the search space and regularizes the generation process. Experimental results on the Restaurant-ACOS and Laptop-ACOS datasets demonstrate a significant performance improvement compared to baseline models. Ablation experiments further confirm the effectiveness of latent category distribution and constrained decoding strategy.















§ INTRODUCTION



Sentiment analysis, especially the fine-grained Aspect-Based Sentiment Analysis (ABSA), plays a vital role in Natural Language Processing (NLP) <cit.>. ABSA aims to convert unstructured text into structured records to achieve a profound comprehension of sentiment semantics.
For instance, in the comment “as far as gaming performance, the m370x does quite well”, the ABSA quadruple extraction task <cit.> seeks to extract aspect-level sentiment elements, including aspect, category, opinion, and sentiment polarity, represented as a structured quadruple ⟨m370x, GRAPHICS#OPERATION_PERFORMANCE, well, POS⟩. 


Given its ability to offer opinions and sentiments regarding specific aspects, this task has been widely used in domains such as marketing and conversational agents <cit.>.







Sentiment extraction models are currently categorized into two main groups: extractive methods and generative methods <cit.>. Extractive methods typically frame the task as a sequence labeling or multi-classification problem <cit.>, necessitating task-specific classification networks <cit.>. 
The introduction of generative pre-trained language models such as T5 <cit.> and BART <cit.> has introduced a new paradigm for information extraction, shifting tasks towards sequence generation <cit.>. This generative approach has also demonstrated promising results in sentiment quadruple extraction tasks <cit.>.




However, existing sentiment extraction models often overlook the challenges posed by category inclusion and overlap, which refer to the semantic inclusion or overlapping of categories within the dataset.
For example, the category “GRAPHICS” is semantically encompassed by categories such as “HARDWARE” and “LAPTOP” (where “GRAPHICS” is a semantic subset of “HARDWARE”, and “HARDWARE” is a semantic subset of “LAPTOP”). Similarly, categories like “MEMORY”, “KEYBOARD”, and “BATTERY” exhibit similar semantic inclusion relationships with “HARDWARE” and “LAPTOP”. Additionally, “GRAPHICS” and “DISPLAY” share a semantic overlap, as do “WARRANTY” and “SUPPORT”. 
These instances of semantic inclusion and overlap complicate the accurate extraction of elements, a concern often neglected by current models.








On the other hand, current generative models typically employ greedy decoding <cit.>, selecting the word with the highest probability from the entire target vocabulary at each time step. Nonetheless, this strategy fails to effectively utilize pattern knowledge inherent in the target sequence, which could otherwise constrain the search space <cit.>. To elaborate, datasets typically define restricted sets of categories, subcategories, and sentiment polarities. Greedy decoding based on the entire vocabulary may lead to the generation of invalid categories, misaligned subcategories, as well as invalid sentiment polarities. Despite these challenges, current generative models often neglect the significance of pattern knowledge in the target sequence.







Motivated by recent advances in generative frameworks, this study proposes a T5-based model for sentiment quadruple extraction. The model treats extraction as a generative process, producing sentiment elements in a uniform, natural language format. 
To address category semantic inclusion and overlap issues, a Latent Category Distribution (LCD) variable is introduced. 
This variable, learned through reconstructing the input of a Variational AutoEncoder (VAE) <cit.>, captures the relationship intensity between text and categories, supplementing target sequence generation.
Furthermore, to leverage the inherent pattern knowledge within the target sequence, a Constrained Decoding (CD) strategy based on a trie data structure <cit.> is adopted. 
This strategy introduces structural pattern knowledge during the decoding stage to reduce the search space and enhance sequence generation.
Evaluation on the Restaurant-ACOS <cit.> and Laptop-ACOS <cit.> datasets show that the proposed model outperforms baseline models, and demonstrate the efficacy of the LCD and CD.



The primary contributions of this study include:

	
  * Addressing category semantic inclusion and overlap challenges via a latent category distribution variable learning the relationship intensity between text and categories.
	
  * Regulating the generation process through a constrained decoding strategy leveraging structural pattern knowledge within the target sequence.
	
  * Experimental results on the Restaurant-ACOS and Laptop-ACOS datasets demonstrate the effectiveness of the proposed model.




§ RELATED WORKS












In recent years, there has been a growing interest in fine-grained sentiment analysis <cit.>, involving the extraction and organization of sentiment-related elements such as aspects, categories, opinions, and sentiment polarities from text. Initially focusing on individual elements such as aspect terms <cit.>, categories <cit.>, and sentiment polarities <cit.>. Early research has expanded to include paired extractions <cit.>. Subsequent studies have advanced the field by proposing tasks like triplet extraction <cit.>, involving aspects, opinions, and sentiment polarities. More recently, quadruple extraction <cit.> has been introduced to extract aspects, categories, opinions, and sentiment polarities concurrently, including implicit elements <cit.>. This task has garnered significant attention in the research community.







Quadruple extraction models can be broadly categorized into two main approaches <cit.>: extractive and generative methods. Extractive methods decompose the task into distinct subtasks, addressing them separately <cit.>. For instance, Cai et al. <cit.> employ either the Double Propagation (DP) <cit.> algorithm or the JET <cit.> model to extract aspect-opinion-sentiment triplets, and then predict the category of these triplets. Alternatively, they might use the TAS-BERT <cit.> model for the joint extraction of aspect-opinion pairs, followed by category and sentiment polarity prediction. Zhu et al. <cit.> capture local and contextual features and then utilize a grid labeling scheme along with its decoding method to extract quadruples. 
Extractive methods often transform the task into sequence labeling or multi-classification problems, requiring the development of task-specific classification networks <cit.>.



The emergence of generative language models <cit.> has introduced a novel paradigm to information extraction by unifying tasks into sequence generation problems, eliminating the need for task-specific network designs. 
This shift simplifies model design and facilitates the comprehensive use of correlation information among different subtask modules <cit.>. 
Generative extraction has been applied to various tasks <cit.>. Efforts have recently also focused on transforming fine-grained sentiment analysis tasks into sequence generation challenges. For example, the BARTABSA <cit.> model reformulates sentiment element extraction as a problem of generating indices for target elements. The Paragraph <cit.> model employs predefined templates to reframe quadruple extraction as a template-based generation problem. The GAS <cit.> model proposes annotation-based and extraction-based approaches to model the task as a text generation problem.




However, the models discussed overlook challenges posed by category semantic overlap and inclusion, and neglect to utilize structural pattern knowledge to constrain decoding. This study, inspired by the generative paradigm, adopts a generative model as its core framework. Unlike existing generative models, it addresses semantic overlap and inclusion by incorporating a latent category distribution, and utilizes pattern knowledge to guide decoding during generation. Experimental results validate the effectiveness of the proposed model.





§ TASK DEFINITION


Given a comment X = [x_1, …, x_N], the objective of sentiment quadruple extraction is to extract structured quadruples ⟨ a, c, o, s ⟩ from the comment <cit.>:
	
    X ⇒ [⟨ a_1, c_1, o_1, s_1 ⟩, ⟨ a_2, c_2, o_2, s_2 ⟩, …]

	Here, a represents the aspect described in the comment, c denotes the category to which the aspect belongs, o signifies the opinion on the aspect, and s indicates the sentiment polarity of the opinion. Given that a comment may encompass multiple aspects, the extraction may result in multiple quadruples being identified.


The objective of sentiment quadruple extraction is to extract structured quadruples ⟨ a, c, o, s ⟩ from a given comment X = [x_1, …, x_N] <cit.>:

    X ⇒ [⟨ a_1, c_1, o_1, s_1 ⟩, ⟨ a_2, c_2, o_2, s_2 ⟩, …]

where a represents the aspect, c denotes the category to which the aspect belongs, o signifies the opinion, and s indicates the sentiment polarity of the opinion. 


Since this study transforms the extraction task into a generative manner, specifically, given the comment X, this study utilizes a Sequence-to-Sequence (Seq2Seq) framework to produce a linearized augmented language sequence <cit.> Y:
	
    X ⇒ Y = [y_1, y_2, …]

	Here, the augmented language linear sequence Y comprises natural language words and encompasses the essential elements of the sentiment quadruples. Parsing the generated augmented sequence Y allows for the extraction of sentiment quadruples.
Since this study adopts a generative manner to the extraction task, using a Sequence-to-Sequence (Seq2Seq) framework to convert X into a linearized augmented language sequence Y <cit.>, the task is specifically transformed into:


    X ⇒ Y = [y_1, y_2, …]



where Y is a word sequence consisting of the essential elements of the quadruples, which can be extracted by parsing Y.






§ MODEL STRUCTURE







 §.§ Latent Category Distribution




Latent Category Distribution (LCD) module (refer to <Ref>) models category distribution within text following Fei et al. <cit.>, using a latent variable instead of explicit measurement. It reconstructs the variational autoencoder input to learn the LCD representation Z.



  §.§.§ Encoder

The input to the LCD module consists of the Category Bag of Words (CBoW) features X_CBoW∈ℝ^L, following the approach outlined by Fei et al. <cit.>. 
	The construction of the CBoW features involves the following steps: 
	1) Eliminating stopwords, meaningless words, and sentiment-related words from the training set sentences, while retaining words related to categories and aspects as much as possible, to construct a CBoW vocabulary of length L. 
	2) Calculating the frequency of words in the vocabulary that appear in the sentence to derive the CBoW features of the sentence, resulting in a L-dimensional vector. 
	The objective of the first step is twofold: firstly, to capture the latent category distribution, ensuring that the latent distribution representation contains comprehensive category information rather than general semantic information; secondly, reducing the size of the BoW vocabulary aids in training the variational autoencoder <cit.>.


The input to the LCD module is the Category Bag of Words (CBoW) features <cit.> X_CBoW∈ℝ^L. The construction of CBoW features involves: 
1) Removing stopwords, meaningless words, and sentiment-related words from texts while preserving category- and aspect-related words to construct a CBoW vocabulary of length L. This step aims to ensure that representation Z contains comprehensive category information rather than general semantic information, while decreasing the BoW vocabulary size is beneficial for VAE training <cit.>. 
2) Calculating the occurrence frequency of words in the vocabulary appearing in a sentence to obtain the sentence's CBoW features, forming a L-dimensional vector. 




The encoder of the LCD module comprises two Feedforward Neural Networks (FFNs), denoted as f_μ(·) and f_σ(·), respectively. Each of these networks includes a hidden layer with a dimensionality of 256, which is responsible for transforming the features X_CBoW into the prior parameters μ and σ:

    μ = f_μ ( X_CBoW )


    logσ = f_σ ( X_CBoW )


Following Bowman et al. <cit.>, a latent variable Z' = μ + σ·ϵ∈ℝ^K is defined to represent the latent category distribution, where K is the number of categories, ϵ is a Gaussian noise variable sampled from 𝒩(0,1). Z' is then normalized as follows:

    Z = softmax(Z')


Consequently, the latent category distribution can be reflected by variable Z.


  §.§.§ Decoder


The LCD module employs variational inference <cit.> to approximate a posterior distribution over Z. It begins by applying a linear transformation f_lcd(·) to Z:


    R^lcd = f_lcd(Z, W_lcd)

where W_lcd∈ℝ^K × dim, dim is the encoding dimension. R^lcd will be fed into the generative framework (<Ref>) with category distribution information to enhance the  generation.
Subsequently, a FFNs f_R(·) is employed to further decode R^lcd into R:

    R = f_R(R^lcd, W_R)

where W_R ∈ℝ^L × dim.
Finally, the CBoW features are reconstructed by:

    X̂_CBoW = softmax(R)




  §.§.§ Loss Function

Parameters of LCD module are learned through the optimization of the variational lower bound on the marginal log likelihood of features:

    log p_φ(X) = 𝔼_Z∼ q_ϕ(Z|X)[ log p_φ(X|Z) ] - KL(q_ϕ(Z|X)  p(Z))

where ϕ and φ are parameters of the encoder and decoder, respectively. The KL divergence term ensures that the distribution q_ϕ(Z|X) approaches the prior probability p(Z), while p_φ(X|Z) represents the decoding process.



 §.§ Generative Framework






  §.§.§ Encoder

The generative framework (refer to <Ref>) employs a T5 encoder <cit.> for input encoding. Given a comment X = [x_1,⋯ ,x_N], the encoder first converts the token sequence into a high-dimensional vector representation E∈ℝ^N × dim using an internal embedding layer, where N is the sequence length, and dim is the dimension of the representation.

Subsequently, E is passed through the T5 encoder which comprises multiple layers of Transformer <cit.> structure, to calculate context-aware representations:

    H^enc = T5Encoder(E)

where H^enc∈ℝ^N × dim.




  §.§.§ Category-Text Attention

After encoding the input sequence into representation H^enc, the model computes attention score of the latent category distribution over the input sequence, to establish the relationship between them. The attention score is calculated as follows:

    e_j = dot(R^lcd, H_j^enc)/√(dim)

where dot(·) is the dot product, and j ∈ [1, N]. Following this, e is fed to a softmax function to calculate the attention weights:

    a = softmax(e)

a is then applied as weights to the representation of the input sequence to obtain the output vector V of the encoder:

    V_j = a_j H_j^enc

The vector V, representing the input text and incorporating LCD information, is then fed into the decoder to provide contextual information for decoding.




  §.§.§ Decoder


The model employs a T5 decoder <cit.> to sequentially generate the output sequence Y=[y_1, …, y_i, …] in an auto-regressive manner. When predicting the output at step i, the decoder uses the context vector V and the previous outputs y_<i as inputs to calculate the current hidden state of the decoder:


    H_i^dec = T5Decoder(V, y_<i)


Here, each T5Decoder(·) layer comprises a Transformer structure and a cross-attention mechanism <cit.> for enhancing the representation of H_i^dec. The probability distribution over target vocabulary at step i is calculated as follows:

    P(y_i|y_<i,V) = Softmax(WH_i^dec + b)

where W∈ℝ^|𝒱| × dim, 𝒱 is the target vocabulary, and b is a bias vector.


The decoder repeats this procedure until it encounters the end token “⟨ eos ⟩” or reaches the maximum specified output length.



  §.§.§ Loss Function


The loss function of the generative framework is defined as the cross-entropy loss between the generated sequence and the target sequence:

    L(θ) = -∑_i=1^Mlog P(y_i|y_<i, V)

where θ represents the parameters of the generative framework, and M is the length of the target sequence.



 §.§ Constrained Decoding





  §.§.§ Target Sequence Pattern Knowledge




In generative models, a commonly used decoding approach is greedy decoding <cit.>, where the model calculates the probability distribution over the entire vocabulary at each decoding step, and selects the word with the highest predicted probability as the output.

However, this decoding strategy may lead to invalid structure patterns. 
For instance, 
1) due to the limited number of category types, greedy decoding using the entire vocabulary may yield invalid categories,
2) since each category corresponds to distinct subcategories, greedy decoding might generate subcategories that do not align with that category,
3) as sentiment polarity is finite, greedy decoding could result in invalid sentiment polarities,
4) incomplete sequence structures, such as the failure to generate pairs of brackets, may occur. 
Following Lu et al. <cit.>, information regarding the structural patterns of the target sequence, which can be used to regulate its generation process, is referred to as target sequence pattern knowledge in this paper.



  §.§.§ Constrained Decoding Strategy





Pattern knowledge of the target sequence can constrain the decoding process, reducing the search space and enhancing target generation. 
To leverage this knowledge, this study employs a trie-based constrained decoding strategy <cit.>. 
A trie, also referred to as a prefix tree, is a type of k-ary search tree that serves as a data structure for locating specific keys within a set <cit.>.
With this structure, constrained decoding strategy dynamically selects and prunes candidate vocabulary based on the current generation state. The candidate vocabulary 𝒱' includes the following three types:

	
  * Structure pattern: category set 𝒞_1 (see <Ref>), their respective subcategory set 𝒞_2 (see <Ref>), and sentiment polarities set 𝒮 (see <Ref>).
	
  * Text span: aspects 𝒜 and opinions 𝒪, representing text spans in the input.
	
  * Structure indicators: “[”,  “]”, “⟨”, and “⟩” (see <Ref>), utilized to combine structure patterns and text spans.





















<Ref> depicts constrained decoding using a trie of linearized sentiment quadruples. Starting with the initial token “⟨ bos ⟩”, candidate vocabulary 𝒱' at each step is confined to child nodes stemming from the previously generated node. 

For example, after generating the sequence “⟨ bos ⟩ [”, 𝒱' is limited to the child node set {“⟨”, “]”} of node “[”.






When generating the category, decoding involves searching the trie structure of category 𝒞_1 (<Ref>). Similarly, when generating the subcategory, decoding searches the corresponding subtree, constraining the search space based on the category-subcategory dependency. For example, when generating the subcategory of category “HARDWARE”, the candidate vocabulary 𝒱' is the child node set of “HARDWARE”, namely, {“PORTABILITY”, ⋯, “USABILITY”}.

To generate sentiment polarity, decoding is confined to searching the sentiment polarity trie 𝒮 (<Ref>). For aspect words and opinion words generation, the candidate vocabulary 𝒱' is the entire T5 vocabulary. This generation process repeats until the termination token “⟨ eos ⟩” is produced. Finally, the output sequence is converted into quadruples to serve as the final extraction results.



§ EXPERIMENTS



 §.§ Datasets





Two datasets, Restaurant-ACOS <cit.> and Laptop-ACOS <cit.>, are used to evaluate model performance. Each sample comprises a review text and annotated sentiment quadruples, as summarized in <Ref>.



These datasets also introduce the concept of implicit elements, where aspects or opinions within the review are considered implicit if they are not explicitly stated. 
For example, in the review “apparently apple isn't even trying anymore”, the annotated quadruple ⟨apple, COMPANY#GENERAL, Null, NEG⟩ indicates that the opinion towards the aspect “apple” is not explicitly stated in the review and is considered implicit, labeled as “Null” in the annotation.



The statistics of implicit elements is shown in <Ref>. Here, EA&EO represents quadruples with only explicit aspects and opinions, IA&EO indicates those with implicit aspects and explicit opinions, EA&IO covers quadruples with explicit aspects and implicit opinions, and IA&IO denotes quadruples with only implicit aspects and opinions.







 §.§ Experimental Setup










A GeForce RTX 3090 (24GB) was utilized for computation. T5-base model was employed with a training batch size of 16, and encoding and hidden layer dimensions set to 768. The AdamW optimizer was employed with a learning rate of 2e-5, and dropout was applied at a rate of 0.2. Experiments were initialized five times randomly, and the average result was considered the final outcome.







While it was possible to train the model solely with the generation framework loss (<Ref>), practical experiments revealed slow convergence and poor performance. To tackle this issue, the LCD module was initially trained with the reconstruction loss (<Ref>) until convergence. Subsequently, the entire model was trained using the generation loss (<Ref>). Once the generation loss approached convergence, further training was applied to the LCD module until convergence. This process was repeated until the model demonstrated improved performance.





Evaluation employed Precision, Recall, and F1 score, with a quadruple being deemed correctly predicted only if all its four elements were predicted accurately.



 §.§ Baselines


The subsequent models were employed for comparative analysis:

Extractive:
DP-ACOS <cit.>: Utilizes the DP algorithm to extract sentiment triplets ⟨ a,o,s ⟩ and then determines the category of each triplet.
JET-ACOS <cit.>: Employs JET <cit.> to extract triplets ⟨ a,o,s ⟩ and then predicts category of each triplet.
TB-ACOS <cit.>: Utilizes TAS-BERT <cit.> for aspect-opinion joint extraction based on category-sentiment, then filters out invalid pairs.
EC-ACOS <cit.>: Performs aspect-opinion joint extraction, then predicts category-sentiment pairs.
SGTS <cit.>: Captures high-dimensional features of sentences, then uses grid labeling scheme and its decoding method for extraction.

Generative:
BARTABSA <cit.>: A model based on BART <cit.>, generates sentiment quadruples by predicting the indices of target words and sentiment categories.
GAS <cit.>: Using T5, GAS models sentiment analysis and various subtasks as generative tasks.
Paraphrase <cit.>: Transforms the quadruple extraction task into a paraphrase generation process using predefined templates.
Seq2Path <cit.>: Formalizes the generated content as a tree, with sentiment tuples representing the paths of the tree.



§ EXPERIMENTAL RESULTS AND ANALYSES



 §.§ Main Results














<Ref> presents the experimental results of the proposed model and baseline models on two datasets. The results highlight the superiority of generative models over extractive models. This advantage can be attributed primarily to two factors: 1) treating the extraction task as an end-to-end sequence generation problem reduces error propagation; 2) uniformly modeling the generation of each quadruple element facilitates more effective utilization of semantic knowledge in pretrained models, enhancing semantic sharing and decision interaction.

Furthermore, the results in <Ref> suggest that the proposed model, utilizing LCD and CD, outperforms the comparing generative models on both datasets. Specifically, on the Restaurant-ACOS dataset, compared to the baseline model GAS, the proposed model shows a decrease of 0.004 in Precision, but an increase of 0.038 in Recall, and an increase of 0.017 in F1. On the Laptop-ACOS dataset, the proposed model exhibits an increase of 0.031 in Precision, 0.035 in Recall, and 0.033 in F1. These results highlight the superior performance of the proposed model in sentiment quadruple extraction.

Despite a notable improvement in performance compared to the baseline models, the overall performance remains modest. This can be attributed to two main factors. 
First, the inherent complexity of the task necessitates the extraction and precise matching of four distinct types of sentiment elements, thereby increasing the difficulty of the task. 
Second, special expressions in language such as the presence of implicit elements within the text presents a significant challenge, potentially impeding the model's comprehensive semantic understanding, and consequently leading to suboptimal performance. 
The following subsection will delve into the extraction of implicit elements.







 §.§ Implicit Element Extraction Results

As shown in <Ref>, both datasets contain a certain proportion of implicit elements <cit.>. Since these elements are not explicitly mentioned in the original text, the model lacks sufficient clues to extract them, thus requiring a deeper understanding of the text, which has also been a focus of attention in the field of sentiment analysis in recent years <cit.>.
	
	To evaluate the performance of the proposed model and the comparison models in extracting implicit elements, this section divides the test set into four subsets to examine the performance of the models on each subset. The four subsets are divided as follows: (1) EA&EO indicates that this subset contains quadruples with only explicit aspect words and opinion words; (2) IA&EO indicates that this subset contains quadruples with implicit aspect words and explicit opinion words; (3) EA&IO indicates that this subset contains quadruples with explicit aspect words and implicit opinion words; (4) IA&IO indicates that this subset contains quadruples with only implicit aspect words and opinion words. The proposed model and baseline models are experimented on these four subsets, and the results are shown in <Ref>, where the comparison model results are from the original literature.
	
	From <Ref>, it can be seen that on the Restaurant-ACOS dataset, the proposed model not only performs best in the EA&EO subset but also outperforms other models in subsets containing implicit elements, such as IA&EO and IA&IO. On the Laptop-ACOS dataset, the proposed model also achieves the best performance in the EA&EO subset and outperforms other models in subsets containing implicit elements, such as EA&IO and IA&IO. These results demonstrate that the proposed model's ability to deeply understand and generate implicit elements is superior to that of the comparison models.












As outlined in <Ref>, both datasets exhibit a notable presence of implicit elements <cit.>. Due to their non-explicit nature in the text, these elements present a challenge for models, requiring a deeper understanding of the text. This challenge has garnered significant attention in recent sentiment analysis studies <cit.>.
	
	To assess the capacity of models in extracting implicit elements, the test set was divided into four subsets, each targeting specific types of quadruples. These subsets, namely EA&EO, IA&EO, EA&IO, and IA&IO, are elaborated in <Ref>.
	<Ref> illustrates that the model's performance in extracting explicit elements is superior to its performance in extracting implicit elements. For instance, on the Restaurant-ACOS dataset, the proposed model achieves an F1 score of 66.12 on the subset EA&EO, which contains only explicit elements, outperforming the F1 scores of 54.35, 44.20, and 41.25 on the other three subsets containing implicit elements. Similar results are observed on the Laptop-ACOS dataset. This performance gap reflects the challenges associated with extracting implicit elements.
	
	The results presented in <Ref> indicate that, on the Restaurant-ACOS dataset, the proposed model excels not only on the EA&EO subset but also on subsets that include implicit elements (IA&EO and IA&IO). Similarly, on the Laptop-ACOS dataset, the proposed model shows superior performance on the EA&EO subset and surpasses on subsets containing implicit elements (EA&IO and IA&IO). These findings highlight the superior capability of the proposed model in deep semantic understanding and generating implicit elements.










	




As noted in <Ref>, both datasets include implicit elements <cit.>, which are not explicitly stated in the reviews. These elements necessitate a comprehensive understanding of the text and present a challenge for models, thus attracting considerable attention in recent sentiment analysis research <cit.>.

To assess the capacity of models in extracting implicit elements, the test set was divided into four subsets: EA&EO, IA&EO, EA&IO, and IA&IO, each targeting specific type of quadruples, whose meanings were introduced in <Ref>. Results in <Ref> indicate that models excel in extracting explicit elements compared to implicit ones. For instance, on the Restaurant-ACOS dataset, the proposed model achieves an F1 score of 0.661 on the EA&EO subset, which is comprised solely of explicit elements, significantly outperforming scores of 0.544, 0.442, and 0.413 on the other three subsets containing implicit elements. Similar trends are also observed on the Laptop-ACOS dataset. These results highlight the challenge of extracting implicit elements.

Moreover, <Ref> shows that, on Restaurant-ACOS, the proposed model not only outperforms baseline models on the EA&EO subset, but also on subsets containing implicit elements (IA&EO and IA&IO). 
Similarly, on Laptop-ACOS, the proposed model also outperforms on subsets comprising implicit elements (EA&IO and IA&IO). These findings highlight the capability of the proposed model in deep semantic comprehension and generation of implicit elements.












 §.§ Ablation Study





To assess the effectiveness of the proposed LCD module and CD strategy, ablation experiments were conducted on the datasets. Results in <Ref> indicate the impact of removing the LCD module (w/o LCD), excluding the CD strategy (w/o CD), and simultaneously excluding both components (w/o LCD&CD).

As shown in <Ref>, performance deteriorates when LCD and CD are excluded. Specifically, on the Restaurant-ACOS dataset, the removal of LCD alone resulted in an F1 decrease of 0.009, while excluding CD alone led to an F1 decrease of 0.012. Simultaneous removal of both components resulted in an F1 decrease of 0.014. Similar trends were also observed on the Laptop-ACOS dataset. 

These results suggest that the proposed LCD module enhances the extraction performance by capturing latent category features, while the CD strategy regulates and guides the generation process, mitigating the generation of erroneous patterns.






 §.§ Visualization of Latent Category Distribution





As introduced in <Ref>, the LCD module learns latent category distribution representation Z for a sample. To visually illustrate the effectiveness of this module, five samples were selected with their learned LCD representations visualized in <Ref>. The x-axis denotes the total 23 categories, while the y-axis indicates the sample indices. 
Each row in the figure corresponds to a LCD representation vector of a sample across categories, with color intensity reflecting the magnitude of the vector in respective category.
The five samples, along with their respective standard categories in parentheses, are as follows:

	
  * the product is great, but the customer support is horrible. (LAPTOP, SUPPORT)
	
  * second issue is with scaling of the ui. (SOFTWARE)
	
  * asus, has a horrible reputation. (COMPANY)
	
  * hdmi out doesn't work right. (PORTS)
	
  * now i have to deal with warranty stuff and sending it back. (WARRANTY)



 



 
As depicted in <Ref>, the LCD module effectively captures latent category distribution information for samples, including those with complex category inclusion (sample 4) or overlap (sample 5) relationships. 
For example, while the standard category of sample 4 is “PORTS”, it is highly predictable as “HARDWARE” or “LAPTOP” due to semantic inclusion relationships, where “PORTS” is semantically included by both “HARDWARE” and “LAPTOP”. 
For sample 5, although its standard category is “WARRANTY”, it is highly predictable as “SUPPORT” due to semantic overlap between the two categories. <Ref> demonstrates the accurate capture of latent category distributions by LCD.



§ CONCLUSION

This study introduces a generative model for extracting sentiment quadruples. To address challenges related to semantic inclusion and overlap among categories, the model incorporates a latent category distribution to capture the strength of the relationship between text and categories. Moreover, a constrained decoding strategy based on a trie structure is employed to exploit knowledge of target sequence patterns, thereby reducing the search space and enhancing the generation process. Experimental results demonstrate performance enhancements compared to baseline models, with ablation studies verifying the efficacy of the latent category distribution and constrained decoding strategy.
Despite advancements, performance remains moderate, primarily due to task complexity and special expressions in language, including implicit elements.

Future research could focus on enhancing feature extraction and representation, as well as improving contextual understanding, and integrating coreference resolution and background knowledge, to improve the extraction process, including implicit elements.















	


  §.§.§ Sample Heading (Third Level)
 Only two levels of
	headings should be numbered. Lower level headings remain unnumbered;
	they are formatted as run-in headings.
	
	

  
Sample Heading (Fourth Level)
	The contribution should contain no more than four levels of
	headings. Table <ref> gives a summary of all heading levels.
	
	
	
	
	Displayed equations are centered and set on a separate
	line.
	
    x + y = z

	Please try to avoid rasterized images for line-art diagrams and
	schemas. Whenever possible, use vector graphics instead (see
	Fig. <ref>).
	
	
	
	
	This is a sample theorem. The run-in heading is set in bold, while
	the following text appears in italics. Definitions, lemmas,
	propositions, and corollaries are styled the same way.
	
	
	
	
	
	
	Proofs, examples, and remarks have the initial word in italics,
	while the following text appears in normal font.
	
	For citations of references, we prefer the use of square brackets
	and consecutive numbers. Citations using labels or the author/year
	convention are also acceptable. The following bibliography provides
	a sample reference list with entries for journal
	articles <cit.>, an LNCS chapter <cit.>, a
	book <cit.>, proceedings without editors <cit.>,
	and a homepage <cit.>. Multiple citations are grouped
	<cit.>,
	<cit.>.






  §.§.§ 


This work was supported by the National Natural Science Foundation of China (No. 62176187), the National Key Research and Development Program of China (No. 2022YFB3103602, No. 2017YFC1200500), the Research Foundation of Ministry of Education of China (No. 18JZD015).




  §.§.§ 

	It is now necessary to declare any competing interests or to specifically
	state that the authors have no competing interests. Please place the
	statement with a bold run-in heading in small font size beneath the
	(optional) acknowledgments[If EquinOCS, our proceedings submission
	system, is used, then the disclaimer can be provided directly in the system.],
	for example: The authors have no competing interests to declare that are
	relevant to the content of this article. Or: Author A has received research
	grants from Company W. Author B has received a speaker honorarium from
	Company X and owns stock in Company Y. Author C is a member of committee Z.









splncs04


8
	ref_article1
	Author, F.: Article title. Journal 2(5), 99–110 (2016)
	
	ref_lncs1
	Author, F., Author, S.: Title of a proceedings paper. In: Editor,
	F., Editor, S. (eds.) CONFERENCE 2016, LNCS, vol. 9999, pp. 1–13.
	Springer, Heidelberg (2016). 10.10007/1234567890
	
	ref_book1
	Author, F., Author, S., Author, T.: Book title. 2nd edn. Publisher,
	Location (1999)
	
	ref_proc1
	Author, A.-B.: Contribution title. In: 9th International Proceedings
	on Proceedings, pp. 1–2. Publisher, Location (2010)
	
	ref_url1
	LNCS Homepage, <http://www.springer.com/lncs>, last accessed 2023/10/25
	





 Paper Title: Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances.


MIST: A Simple and Scalable End-To-End 3D Medical Imaging Segmentation Framework
    
Adrian Celaya1,2 Evan Lim1 Rachel Glenn1 Brayden Mi1 Alex Balsells1,2 Tucker Netherton1 Caroline Chung1 Beatrice Riviere2 David Fuentes1

    August 5, 2024
==============================================================================================================================================

§ ABSTRACT 
This paper introduces a novel approach to emotion detection in speech using Large Language Models (LLMs). We address the limitation of LLMs in processing audio inputs by translating speech characteristics into natural language descriptions. Our method integrates these descriptions into text prompts, enabling LLMs to perform multimodal emotion analysis without architectural modifications or addition. We evaluate our approach on two datasets: IEMOCAP and MELD, demonstrating significant improvements in emotion recognition accuracy, particularly for high-quality audio data. Our LLM-finetuning experiments show that incorporating speech descriptions yields a more than 2 percentage point increase in average weighted F1 score on IEMOCAP (from 70.111% to 72.596%). We also compare various LLM architectures and explore the effectiveness of different feature representations. Our findings highlight the potential of this approach in enhancing emotion detection capabilities of LLMs and underscore the importance of audio quality in speech-based emotion recognition tasks. We'll release the source code on Github. 






§ INTRODUCTION

Emotion detection in speech is a crucial component of human-computer interaction, with applications ranging from customer service to mental health monitoring. While Large Language Models (LLMs) have shown remarkable capabilities in processing and understanding text, they lack the ability to directly process audio inputs. This limitation has hindered their potential in multimodal emotion recognition tasks that involve both textual and audio data.

Our research addresses this challenge by introducing a novel approach that enables LLMs to leverage speech characteristics for emotion detection without requiring architectural modifications. We propose a method that translates audio features into natural language descriptions, which can be seamlessly integrated into text prompts for LLMs. This approach bridges the gap between audio and text modalities, allowing LLMs to perform multimodal emotion analysis.

In this paper, we evaluate our method using two widely recognized datasets in the field of emotion recognition: IEMOCAP <cit.> and MELD <cit.>. We explore the effectiveness of various feature representations and compare the performance of different LLM architectures. Our study aims to demonstrate the potential of this approach to enhance the emotion detection capabilities of LLMs and to provide insights into the factors that influence its success.

By enabling LLMs to consider both textual and audio-derived information, we open up new possibilities for more nuanced and accurate emotion recognition systems. This research not only contributes to the field of affective computing but also provides a framework for incorporating non-textual information into language-model-based systems, with potential implications for a wide range of multimodal AI applications.



§ RELATED WORK



 §.§ LLM for emotion recognition

Recent advancements in Emotion Recognition in Conversation (ERC) have leveraged Large Language Models (LLMs) to enhance performance and generalizability. Our work builds upon these developments, particularly the InstructERC framework <cit.>, which reformulated ERC as a generative task using LLMs. InstructERC introduced a retrieval template module and additional emotion alignment tasks, achieving state-of-the-art results on common ERC datasets. We extend this approach by incorporating speech descriptions into the template, aiming to capture crucial paralinguistic information.

Similar extensions to InstructERC have been explored in recent literature. BiosERC <cit.> enhances the model by extracting and injecting "biographical information" of speakers into the system, demonstrating improved performance across multiple datasets. CKERC <cit.> takes a different approach, incorporating commonsense knowledge about interlocutors' reactions and intentions to deepen the understanding of speaker information. DialogueLLM <cit.> incorporates  visual descriptions generated from GPT-4 into the prompt. All of these methods, like our approach, aim to enrich the input to LLMs with additional contextual information.

Our work differentiates itself by focusing specifically on the integration of speech characteristics, translated into natural language descriptions. This approach bridges the gap between audio and text modalities in LLM-based ERC systems, potentially capturing emotion cues that are present in speech but not in text alone. By doing so, we aim to combine the strengths of LLMs in processing textual data with the rich emotional information contained in speech, offering a novel perspective in the evolving landscape of ERC research.



 §.§ Speech Incorporation in LLM

The integration of speech features into LLM-based systems for emotion recognition and related tasks has been explored in various recent works, each offering unique approaches and insights. SECap <cit.> introduces a framework that uses LLaMA as a text decoder to generate coherent speech emotion captions instead of labels. While SECap demonstrates the potential of LLMs in describing speech emotions, it differs from our approach by requiring an additional audio encoder (HuBERT) and a bridging network (Q-Former) to process speech inputs. Similarly, <cit.> proposes an approach to integrate speech into LLMs for depression detection by adding acoustic landmark tokens to the input prompt. This method shares our goal of incorporating speech information directly into LLM inputs but necessitates additional parameters and pre-training steps to introduce audio information.

LanSER <cit.> takes a different approach by leveraging LLMs to infer weak emotion labels from speech transcripts, enabling the use of unlabeled data in SER tasks. Building upon this concept, <cit.> explores using LLMs for emotional speech annotation by incorporating both conversation context and textual acoustic feature descriptors. While this approach shares our interest in combining textual and acoustic information in the prompt, it primarily uses LLMs for weak label generation rather than direct emotion recognition training.

Recent research has also explored the application of LLMs in Automatic Speech Recognition (ASR), demonstrating the versatility of LLMs in speech-processing tasks. Seed-ASR <cit.> introduces an audio-conditioned LLM framework that incorporates continuous speech representations by developing a unique audio encoder. <cit.> proposes a speech prefix-tuning approach with RNNT loss to optimize speech prefixes for LLMs in ASR tasks. SLAM-ASR <cit.> takes a simpler approach, combining off-the-shelf speech encoders and LLMs with only a trainable linear projector, achieving state-of-the-art performance on the Librispeech benchmark.

Further extending the capabilities of LLMs in speech processing, AudioChatLlama <cit.> presents an end-to-end model that integrates general-purpose speech abilities into the Llama-2 model. This approach enables the model to handle audio prompts, perform spoken question answering, speech translation, and audio summarization, among other tasks. Similarly, Audio-LLM <cit.> introduces a method for encoding audio data into embedded representations that LLMs can comprehend, enabling tasks such as automatic speech recognition, emotion recognition, and music captioning. These advancements highlight the growing trend of integrating comprehensive speech-processing capabilities into LLMs.

Our work distinguishes itself from these approaches in several key aspects: (a) Direct Integration: We achieve seamless integration without requiring additional neural network components (e.g., audio encoders or projection layers) to process speech inputs, thereby reducing computational complexity. (b) End-to-End Training: We train the LLM directly on combined textual and speech description inputs, avoiding the need for weak label generation and additional training stages. (c) Flexibility and Generalizability: Our method of translating speech features into natural language descriptions allows for easy integration as a plug-in for other approaches and tasks, offering a versatile solution for incorporating speech information into LLM-based systems.





§ METHODOLOGY

Our novel approach incorporates speech characteristic descriptions in natural language into the prompt for LLMs. Previous LLM-based approaches have primarily focused on textual information, neglecting audio signals due to LLMs' inability to process direct audio inputs. This omission results in the loss of crucial emotional cues, as the same sentence can convey different emotions through variations in audio features such as pitch, volume, and intonation. Our method bridges this gap by translating audio signals into natural language descriptions of speech characteristics that are instrumental in emotion detection.

Our approach offers several advantages in the field of emotion detection. First, it enables multimodal analysis by seamlessly integrating speech characteristics into text prompts, allowing LLMs to process audio-derived information without requiring architectural modifications. This method is highly compatible with existing LLM infrastructures, facilitating easy implementation and integration into current systems. Moreover, by preserving paralinguistic information such as volume, pitch, and speaking
rate in our natural language descriptions, we capture crucial emotional cues that are 

often lost in text-only analysis. This enhanced contextual understanding empowers the LLM to grasp the full spectrum of the utterance, potentially leading to more accurate and nuanced emotion classification.



 §.§ Large Language Model Prompt Template




Our methodology utilizes a comprehensive LLM Prompt Template designed to enhance emotion detection capabilities. This template consists of four key components:


  
  * Instruction: This component sets the stage by positioning the LLM as an expert in sentiment and emotion analysis, priming it for the task ahead.
  
  * Context: A crucial element that provides the conversational background, allowing the LLM to understand the broader situation and interpersonal dynamics at play. We also add speech features behind the last three utterances to provide additional speech context.
  
  * Speech Descriptions: This innovative addition translates audio signals into natural language descriptions of speech characteristics. By including details such as volume, pitch, and speaking rate, we bridge the gap between audio and text modalities, enabling the LLM to consider paralinguistic features crucial for accurate emotion detection. 
  
  * Question: This component presents the specific task to the LLM, asking it to select an emotional label for a target utterance from a predefined set of options. Importantly, it explicitly instructs the model to consider both the contextual information and the audio features described.


This structured approach ensures that the LLM has access to both textual and audio-derived information, enabling a more nuanced and accurate emotion detection process. By incorporating speech characteristics into the prompt, we allow the LLM to perform multimodal analysis without requiring direct audio input processing capabilities.




 §.§ Audio Features

In our approach to emotion detection, we leverage five intuitive audio features that are easily interpretable by both humans and LLMs. These features capture essential aspects of speech that convey emotional information. volume is represented by two sub-features: average volume, which indicates the overall loudness of speech, and volume variation, which captures dynamic changes in intensity. Pitch, another crucial indicator of emotion, is similarly divided into average pitch and pitch variation, allowing us to detect both the general tone and the modulation of a speaker's voice. The fifth feature, speaking rate, represents the speed of speech, offering insights into the speaker's emotional state and urgency. By focusing on these fundamental yet comprehensive audio characteristics, we ensure that the extracted features are not only rich in emotional content but also readily understandable by LLMs when described in natural language. 



  §.§.§ Audio Feature Processing

In our approach to audio feature processing, we transform raw numerical features into categorical representations to enhance interpretability and facilitate natural language descriptions. This process involves four key steps:

a) Threshold Calculation: We compute thresholds for each audio feature (such as average volume, pitch variation, and speaking rate) using quantile-based segmentation. Depending on the desired granularity, we divide the feature space into 3, 4, 5, or 6 classes, with thresholds calculated using appropriate quantiles (e.g., 25th and 75th percentiles for 3 classes, 10th, 25th, 75th, and 90th percentiles for 5 classes).

b) Speaker-Specific Normalization: Recognizing the individual differences in speech patterns, we calculate speaker-specific thresholds, falling back to overall thresholds for less frequent speakers. This approach allows for a more accurate categorization of speech features relative to each speaker's typical patterns.

c) Categorization: Each numerical feature is then categorized based on these thresholds. For instance, in a 5-class system, a feature value might be classified as 'very low', 'low', 'medium', 'high', or 'very high'. This categorization provides a more intuitive representation of the audio features, bridging the gap between numerical values and human-interpretable descriptions.

d) Feature-Specific Descriptions: For each key audio feature (volume, pitch, speaking rate), we create descriptive phrases based on their categorical values. For example, "high volume with moderate variation" or "low pitch with high variation".




  §.§.§ Audio Impression



To make the audio features more accessible and meaningful for both human analysis and LLM processing, we generate natural language impressions based on the categorized features. Going beyond mere description, we generate interpretative impressions that suggest potential emotional or psychological states associated with the observed speech patterns. For instance, "speaking loudly with significant volume changes" might be interpreted as indicating "excitement, confidence, or urgency".

To account for the inherent uncertainty in interpreting speech patterns, we incorporate confidence-adjusting phrases (e.g., "likely", "may") based on the feature's proximity to category thresholds. This nuanced approach helps prevent overconfident interpretations of borderline cases.

The final impression synthesizes information about pitch, volume, and speaking rate into a coherent, flowing sentence. This integration provides a nuanced description of the speaker's vocal characteristics and their potential implications, offering valuable context for emotion detection tasks.



§ EXPERIMENTS




 §.§ Dataset

For our experiments, we utilized two widely recognized datasets in the field of emotion recognition: IEMOCAP and MELD.

IEMOCAP (Interactive Emotional Dyadic Motion Capture): IEMOCAP consists of dyadic conversations between actors, providing a rich set of emotional expressions in a controlled setting. It contains approximately 12 hours of audiovisual data from 10 actors, split into 5 sessions. Each utterance is annotated with one of nine emotion labels: anger, happiness, excitement, sadness, frustration, fear, surprise, neutral, and other. IEMOCAP is particularly valued for its high-quality audio recordings and motion capture data, making it ideal for multimodal emotion analysis. The dataset includes both scripted and improvised scenarios, offering a balance between controlled emotional expressions and more naturalistic interactions.

MELD (Multimodal EmotionLines Dataset): MELD is derived from the popular TV series "Friends," making it particularly suitable for analyzing emotions in conversational contexts. It contains 13,708 utterances from 1,433 dialogues, each annotated with one of seven emotion categories: anger, disgust, fear, joy, neutral, sadness, and surprise. MELD is multimodal, providing audio, visual, and textual data for each utterance. The dataset is pre-split into training (9,989 utterances), validation (1,109 utterances), and test (2,610 utterances) sets. Its multi-party conversational nature and situational diversity make it an excellent resource for evaluating emotion detection in complex, realistic scenarios.

For our experiments, we focused on the audio modality from both datasets, extracting and processing acoustic features as outlined in our methodology. In the case of the MELD dataset, we implemented speaker-based standardization of audio features to account for individual vocal characteristics. For the IEMOCAP dataset, we applied gender-based standardization, recognizing the physiological differences that can impact vocal features. Additionally, for IEMOCAP, we confined our analysis to six emotion categories (anger, happiness, excitement, sadness, frustration, and neutral), aligning with common practices in the field.

Our utilization of both MELD and IEMOCAP datasets was strategically designed to evaluate our approach across diverse audio recording environments. MELD, with its origin in television shows, presents more chaotic and naturalistic scenarios, while IEMOCAP offers a controlled, studio-based environment. This dual-dataset approach enables a comprehensive assessment of our emotion detection method across varied recording conditions.



 §.§ Primary Results




Our results from Table <ref> demonstrate that incorporating speech features into the text input yields performance improvements across both datasets, albeit to varying degrees. For the IEMOCAP dataset, we observe a substantial improvement of nearly 2 percentage points when adding speech descriptions to the text input (from 70.111% to 72.021%). By providing additional speech features for context, the model improves to 72.596%. (Detailed experiments are in Appendix <ref>)  This significant gain underscores the value of integrating acoustic information in emotion recognition tasks, particularly for high-quality audio recordings.

In contrast, the MELD dataset does not show improvemnet using speech descriptions and a modest improvement with the addition of speech impression, increasing from 67.44% to 67.604%. This disparity in performance between the two datasets can be attributed to the differences in audio quality and recording conditions. IEMOCAP, recorded in a controlled studio environment, provides clean and consistent audio, allowing for more accurate extraction of speech features. On the other hand, MELD, derived from the TV show "Friends," presents more challenging audio conditions. The presence of background laughter, varying audio quality, overlapping speakers, and short utterance durations (averaging 2-3 seconds) in MELD likely contribute to less reliable speech feature extraction, thereby limiting the potential benefits of speech descriptions.

Interestingly, the use of speech impressions does not yield further improvements over speech descriptions for either dataset. For IEMOCAP, the performance (71.542%) falls between that of text-only and text with speech descriptions, while for MELD, it performs similarly to the text-only baseline (67.023%). This suggests that our hard-coded impressions, despite being more interpretative, may introduce noise or inaccuracies that the model struggles to leverage effectively. The lack of improvement with speech impressions indicates that allowing the model to learn directly from more objective speech descriptions may be a more robust approach.

These findings highlight the potential of integrating speech characteristics into text-based emotion recognition models, particularly when working with high-quality audio data. However, they also underscore the challenges posed by real-world audio conditions and the need for careful consideration when designing multimodal emotion recognition systems.



 §.§ Model Comparison



To evaluate the effectiveness of our approach across different language models, we conducted experiments using various popular open-sourced LLMs, including LLaMA-2, LLaMA-3, and Phi-3, and a powerful closed-sourced model, Claude Sonnet 3.5. Table <ref> presents the emotion recognition accuracy on the IEMOCAP dataset for these models under different input conditions: text-only, text with speech descriptions, and text with speech impressions.

Our results demonstrate a consistent improvement in performance when incorporating either speech descriptions or impressions across all models. This underscores the value of integrating audio-derived features in emotion recognition tasks, regardless of the underlying language model. For most models, speech descriptions yield slightly better results than speech impressions, although the difference is often marginal. This suggests that more objective, feature-based descriptions may be more reliable for emotion detection tasks.

Interestingly, increasing model size does not necessarily lead to improved performance in this task. Despite their stronger performance on public benchmarks, larger models like LLaMA-3 and Phi-3 do not consistently outperform the smaller LLaMA-2 model when fine-tuned for emotion detection. In fact, Phi-3, despite being the largest model tested (13B parameters), performs worse than both LLaMA-2 and LLaMA-3 across all input conditions. This indicates that model size alone is not a determining factor for success in this specific task. While LLaMA-3-8b-instruct achieves the highest accuracy (72.098%) among all tested models when provided with speech descriptions, the improvement over LLaMA-2-7b-base (72.021%) is minimal. Additionally, we observe that using the instruct or base version of these models does not result in significant performance differences for this task. These findings suggest that while there may be some benefit to the architectural improvements in newer models, the gains are not as significant as one might expect given the increase in model size and the advancements claimed in public benchmarks. Certain features of the LLaMA architecture may be particularly well-suited for emotion recognition tasks, but the task-specific nature of emotion detection may limit the impact of general language model improvements.

The substantial performance gap between the zero-shot Claude model and the fine-tuned models highlights the importance of task-specific fine-tuning for emotion recognition. However, it is noteworthy that even in a zero-shot setting, Claude benefits from the addition of speech features, further emphasizing the value of multimodal inputs in emotion detection.

These findings emphasize the complexity of model selection for specific tasks like emotion recognition. While larger, more recent models often excel in general language understanding tasks, our results show that task-specific fine-tuning and the integration of multimodal features can sometimes yield superior results with smaller, specialized models. This underscores the importance of empirical testing and careful model selection in developing effective emotion recognition systems.






 §.§ Evaluation of Speech Features Using Large Language Model (LLM)






We compared the performance between numerical and categorical acoustic features using simple ML models in Appendix <ref>. Converting numerical features to categorical ones didn't significantly reduce performance and the speech features of IEMOCAP performed significantly better than random guessing. We found that audio quality is crucial for effective speech-based emotion recognition. High-quality audio (IEMOCAP) allows for extraction of informative speech features. Low-quality audio (MELD) makes speech features unreliable for emotion detection.

The results in Table <ref> demonstrate the potential of LLMs in leveraging speech features for emotion classification. As a reference point, recall that the 5-class categorical representation with ML models achieved a weighted F1 score of 31.333%, significantly outperforming the random guess baseline of 16.67%. When using LLMs with natural language descriptions of speech characteristics, we observe a slight decrease in performance compared to the ML approach. The 5-class categorical representation with LLMs yields a weighted F1 score of 27.895%, while the 3-class representation achieves 27.602%.

This performance gap between ML models and LLMs can be attributed to the inherent differences in how these models process information. ML models, designed specifically for discriminative tasks, can directly leverage the structured, one-hot encoded categorical features. In contrast, LLMs receive the information as natural language descriptions, which introduces a layer of abstraction and potential for ambiguity. Despite this, the LLM's performance remains substantially above the random guess baseline, indicating that it successfully extracts meaningful emotional cues from the speech descriptions.

The relatively small difference between 3-class and 5-class representations in LLM performance (27.602% vs. 27.895%) suggests that increasing the granularity of speech feature categorization does not significantly impact the LLM's ability to discern emotions. This could imply that LLMs are adept at interpreting broader emotional cues rather than fine-grained acoustic details.

Notably, we also examined the performance of LLMs using only speech impressions, which are derived from the 5-class categorical speech descriptions but provide a more interpretative representation of the audio features. This approach yields a weighted F1 score of 27.794%, slightly lower than 5-class categorical descriptions. This finding aligns with our earlier results when combining speech features with textual information, where speech impressions also showed a marginal decrease in performance compared to direct speech descriptions. The consistency of this pattern across different experimental setups suggests that, while speech impressions offer a more human-interpretable format, they may introduce a level of abstraction or subjectivity that slightly reduces the model's ability to accurately classify emotions. This underscores the delicate balance between feature interpretability and classification performance in emotion recognition tasks.

While there is some loss of information when transitioning from raw categorical features to natural language descriptions, the LLM's ability to maintain performance well above the baseline demonstrates its potential in multimodal emotion recognition tasks. This approach opens up possibilities for more flexible and interpretable emotion recognition systems that can integrate textual and acoustic information in a unified framework.



§ CONCLUSION


Our study demonstrates that integrating speech characteristics into LLM-based emotion detection systems significantly improves performance, especially for high-quality audio data. By translating audio features into natural language descriptions, we enable LLMs to perform multimodal emotion analysis without architectural changes.

Key findings show that incorporating speech descriptions enhances emotion recognition accuracy across different LLM architectures. However, the effectiveness of this approach heavily depends on audio quality, as evidenced by the contrasting results between IEMOCAP and MELD datasets.

This research opens up new possibilities for developing more flexible and interpretable emotion recognition systems that seamlessly integrate textual and acoustic information. Future work should focus on improving feature extraction methods for noisy, real-world audio data and exploring more sophisticated ways of representing speech characteristics in natural language.

Our approach not only enhances the emotion detection capabilities of LLMs but also provides a framework for incorporating non-textual information into language model-based systems, with potential implications for a wide range of multimodal AI applications.




§ ACKNOWLEDGMENTS

This research is supported in part by the Defense Advanced Research Projects Agency (DARPA), via the CCU Program contract HR001122C0034. The views, opinions and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.





§ LIMITATION

Our approach to enhancing LLM-based emotion detection shows promise, but it is important to acknowledge its limitations. A significant constraint is the method's heavy reliance on audio quality. As evidenced by the contrasting results between the IEMOCAP and MELD datasets, our system's performance degrades considerably with noisy or low-quality audio input.

Another limitation lies in the current feature extraction process. Our approach utilizes a limited set of audio features, which may not capture the full spectrum of emotional nuances present in speech. More complex emotional cues embedded in prosody, rhythm, or spectral characteristics could be overlooked, potentially limiting the depth and accuracy of our emotional analysis. This simplification of audio features may result in a loss of subtle emotional information that humans can easily discern.

The nature of our training datasets presents another challenge. Our study primarily relies on acted (IEMOCAP) or scripted (MELD) emotional expressions, which may not fully represent the complexity and subtlety of emotions in natural, spontaneous speech. This dataset bias could limit the generalizability of our findings to real-world scenarios where emotions are often more nuanced, mixed, or ambiguously expressed. The gap between performed emotions in our datasets and the diverse, spontaneous emotional expressions in everyday life may impact the real-world effectiveness of our emotion recognition system.

Lastly, the computational resources required for this approach pose a practical limitation. Fine-tuning large language models demands significant computational power, which may restrict the implementation of this method in resource-constrained environments. This requirement could limit the accessibility and widespread adoption of our approach, particularly in settings where high-performance computing resources are not readily available.

Addressing these limitations in future research will be crucial for developing more robust, versatile, and widely applicable emotion recognition systems that can handle the complexities of real-world emotional expressions across various audio conditions and computational environments.














§ APPENDIX





 §.§ Speech features for context





In our exploration of incorporating speech features into context utterances, we found that selective application of these features yielded better results than a blanket approach. As shown in Table <ref>, adding speech descriptions to only the last three utterances in the context proved more effective than applying them to all utterances. This suggests that recent context carries more weight in emotion recognition. Notably, pitch features outperformed volume features, with pitch information from the last three utterances providing a 0.6% improvement over non-contextual speech descriptions, achieving a weighted F1 score of 72.595%. Interestingly, combining all speech features for context utterances did not enhance performance, likely due to information overload. These findings highlight the importance of judicious feature selection and context consideration in multimodal emotion recognition tasks.




 §.§ Evaluation of Speech Features Using Simple Machine Learning Models


To evaluate the effectiveness of our extracted speech features in capturing emotion-related information, we conducted experiments using various machine learning models on both numerical and categorical representations of these features. Table <ref> and Table <ref> presents the performance of Random Forest (RF), Support Vector Machine (SVM), and Multi-Layer Perceptron (MLP) models on the IEMOCAP and MELD datasets.






Our analysis of the IEMOCAP dataset reveals that the simple five speech features contain substantial emotion-related information. Both weighted F1 and macro F1 scores (approximately 0.32 and 0.30, respectively) are nearly double the random guess baseline of 0.167. This significant improvement over the baseline indicates that our extracted speech features are effectively capturing emotional cues. Interestingly, the conversion of numerical features into categorical features does not result in a substantial loss of emotion-related information. Moreover, dividing the numerical features into five classes yields better performance than other granularity levels, with weighted F1 and macro F1 scores of 0.313 and 0.293, respectively. This suggests that a moderate level of discretization can reserve clear emotional patterns in speech features.

The results for the MELD dataset highlight one crucial aspects of speech-based emotion recognition: the importance of audio quality. While the weighted F1 scores (around 0.36) are higher than the random guess baseline of 0.143, the macro F1 scores (approximately 0.15) are only marginally better than the baseline. This discrepancy is crucial to understand: MELD is highly imbalanced, with neutral emotions comprising about 50% of the dataset. The high-weighted F1 score is misleading as it primarily reflects the model's ability to predict the dominant neutral class, rather than its capacity to discern emotions from speech features. In contrast, the near-baseline macro F1 score, which treats all classes equally, reveals that the extracted speech features provide minimal informative value across emotion categories. This lack of substantial improvement in macro F1 scores indicates that the key factor behind this poor performance is likely the low quality and challenging nature of the audio in MELD. This finding underscores the critical importance of audio quality in obtaining reliable speech features for emotion recognition tasks. It also explains the modest gains observed when incorporating speech descriptions in our earlier language model experiments on MELD, contrasting sharply with the significant improvements seen in IEMOCAP.

In conclusion, these results not only validate the informativeness of our extracted speech features for emotion detection but also emphasize the crucial role of audio quality in this process. The stark contrast between IEMOCAP and MELD results highlights that high-quality audio data, like that in IEMOCAP, allows for the extraction of informative speech features that significantly enhance emotion recognition. Conversely, challenging audio conditions, as in MELD, can render speech features unreliable and minimally informative. This underscores the need for robust feature extraction methods or pre-processing techniques when dealing with real-world, noisy audio data. Despite these challenges, the performance of categorical representations for IEMOCAP supports our strategy of using discretized speech descriptions in language model prompts for emotion recognition tasks, providing a foundation for future improvements in handling diverse audio qualities.































 Paper Title: Automated Software Vulnerability Static Code Analysis Using Generative Pre-Trained Transformer Models.



Extracting doubly-excited state lifetimes in helium directly in the time domain with attosecond noncollinear four-wave-mixing spectroscopy
    Stephen R. Leone
    August 5, 2024
==========================================================================================================================================



§ ABSTRACT 
Generative Pre-Trained Transformer models have been shown to be surprisingly effective at a variety of natural language processing tasks – including generating computer code. However, in general GPT models have been shown to not be incredibly effective at handling specific computational tasks (such as evaluating mathematical functions). 
In this study, we evaluate the effectiveness of open source GPT models, with no fine-tuning, and with context introduced by the langchain and localGPT Large Language Model (LLM) framework, for the task of automatic identification of the presence of vulnerable code syntax (specifically targeting C and C++ source code). This task is evaluated on a selection of 36 source code examples from the NIST SARD dataset, which are specifically curated to not contain natural English that indicates the presence, or lack thereof, of a particular vulnerability (including the removal of all source code comments). The NIST SARD source code dataset contains identified vulnerable lines of source code that are examples of one out of the 839 distinct Common Weakness Enumerations (CWE), allowing for exact quantification of the GPT output classification error rate. A total of 5 GPT models are evaluated, using 10 different inference temperatures and 100 repetitions at each setting, resulting in 5,000 GPT queries per vulnerable source code analyzed. 
Ultimately, we find that the open source GPT models that we evaluated are not suitable for fully automated vulnerability scanning because the false positive and false negative rates are too high to likely be useful in practice. However, we do find that the GPT models perform surprisingly well at automated vulnerability detection for some of the test cases, in particular surpassing random sampling (for some GPT models and inference temperatures), and being able to identify the exact lines of code that are vulnerable albeit at a low success rate. The best performing GPT model result found was Llama-2-70b-chat-hf with inference temperature of 0.1 applied to NIST SARD test case 149165 (which is an example of a buffer overflow vulnerability), which had a binary classification recall score of 1.0 and a precision of 1.0 for correctly and uniquely identifying the vulnerable line of code and the correct CWE number. 
Additionally, the GPT models are able to, with a rate quantifiably better than random sampling, identify the specific line of source that contains the identified CWE for many of the NIST SARD test cases. 









§ INTRODUCTION



Generative Pre-Trained Transformer (GPT) models, built using the attention architecture <cit.>, have been shown to be surprisingly effective at a variety of information processing tasks including computer code generation and natural language text summarization <cit.>. GPT models are intended to work very well with language, and are typically referred to as a class of Large Language Model (LLM). Notably however, GPT models can, in certain settings, also perform algorithmic tasks such as learning the greatest greatest common divisor algorithm <cit.>. The surprising capability of GPT models to perform well at tasks involving computer code leads to the natural question of whether GPT models could be used for determine security properties of static software samples. In this study, we examine whether current open source GPT models can be used to correctly identify software vulnerabilities and weaknesses in source code correctly. 


The evaluation of our study is based on a set of 36 labeled examples from the highly credible NIST SARD dataset <cit.>. Each source code example in this dataset is meticulously mapped to the Common Weakness Enumeration (CWE) framework <cit.>, providing a solid foundation for our research on software vulnerability detection. The Common Weakness Enumeration (CWE) is a categorization of known types of software weaknesses, hardware weaknesses, and software vulnerabilities. 



Static code vulnerability scanners are an important field of study because they speed up the process of identifying and thus patching software and even potentially hardware vulnerabilities and flaws that cause security problems in computer systems. There are a wide variety of static code analysis scanners that have been developed, usually focusing on specific domains or specific languages <cit.>. A challenge with static vulnerability analysis is that the outputs typically have high false positive rates because the tools usually use string pattern matching to identify known syntax representing an instance of a specific code flaw. The main advantage of static code analysis is that it is much faster and easier to apply than dynamic program analysis. 



Ref. <cit.> studies the task of repairing CWE examples using GPT models. Ref. <cit.> examined source code vulnerability detection using several code language models (namely BERT variants) on a large dataset of software examples. 


Our study introduces novel methods compared to existing research, particularly in the use of highly curated datasets that include the exact vulnerable line of source code. The structured GPT prompt we employ with these GPT models enables the automated extraction of vulnerability detection data from the output, facilitating large-scale, repeatable, accuracy analysis. This dataset combined with structured JSON prompting and parsing allows us to fully leverage the capabilities of GPT models for automated vulnerability detection, requiring no manual processing of the GPT output. In total, we generate and process 100 · 10 · 36 · 5 = 180000 discrete GPT text outputs (each of these being a separate, independent, and complete text inference computations. All of the GPT model computations are performed locally, on a local GPU compute server, using open source GPT models. 












§ METHODS



The five open-source GPT models used in this study are summarized in Table <ref>. These GPT models were selected for their general performance among available GPT models - and all of these models can generate and parse computer code. Although a large number of different GPT models could be utilized for this study - due to the high computation time required to perform these experiments, a small number of GPT models were selected. These GPT models are from the huggingface GPT repository <cit.> and the GPT models are run using the Python3 modules  <cit.> and , using a local compute server that has four Nvidia A100 GPU's <cit.> (each with 82 Gigabytes of memory), and CUDA Version 12.4. 


GPT models can be used to predict the most likely token, given the previous N tokens (where N is the context window for current GPT models on the order of 4096-32768). When used in this manner, the text produced by the GPT models is deterministic, given that the previous tokens are the same. However, GPT models can learn and generate probability distributions of the most likely subsequent tokens - and we can add stochasticity in the text generation process by sampling from that distribution, weighted by their respective probabilities. The parameter that controls this sampling is known as temperature. This stochastic text generation is part of what makes generative machine learning models able to produce a wide variety of outputs. In the context of investigating the usage of these models as code-scanning tools, these models will only perform somewhat well at this task. Moreover, we need outputs to vary somewhat to determine the possible range of characteristics of the output since the text generation process of GPT models can be stochastic. Therefore, we vary the text generation inference temperature from 0.1 to 1.0 in increments of 0.1. The huggingface <cit.> python 3 module used to execute this GPT model inference allows temperatures up to 100, where 0 denotes deterministic output (with no sampling), and 100 denotes closer to uniform token sampling. We choose relatively low temperatures because we need the output to conform to parsable data structures. Critically, because the outputs vary significantly, we view this task as a stochastic sampling problem, where we characterize the proportion of samples with specific required properties (the most basic being whether the data could be automatically parsed). 100 samples (GPT outputs) are generated for each inference temperature, GPT model, and source code test case. 










To ground the GPT model output in the definitions of the Common Weakness Enumeration (CWE) framework, we utilize a technique known as retrieval augmented generation <cit.>. Specifically, we use a codebase known as  <cit.> that uses langchain <cit.> to augment the capabilities of GPT models by using additional word embeddings of reference documents (not as part of the model pre-training). Specifically, the source documents are split into character maximum chunk sizes of 1000, with chunk overlap of 200 characters. 

The localGPT prompt template is a combination of context and then the user prompt question. No chat history was used, meaning that each text generation call is independent. The prompt construction uses a system prompt <cit.> followed subsequently by the context of the generated word embeddings and then the user prompt. The system prompt used in this study, which is from a version of the localGPT codebase <cit.>, is given below:



You are a helpful assistant, you will use the provided context to answer user questions.
Read the given context before answering questions and think step by step. If you can not answer a user question based on the provided context, inform the user. Do not use any other information for answering user. Provide a detailed answer to the question.



The reference documents that are used to provide context for the GPT model inference calls are the following: one text file for each of the 839 CWEs that are currently defined, where the text file contains the CWE number, the name of the software weakness, and the complete description of the CWE from Mitre [<https://cwe.mitre.org>]. Next is a text file with a brief definition of the CWE framework. And lastly, two books (PDF files of the books) on software, security, and vulnerabilities (specifically for the C family languages) <cit.>. 



Table <ref> outlines the exact code test cases that are analyzed in this study. Note that NIST SARD test case 148871 is the vulnerable code that was tracked by CVE-2009-3549, and test case 149199 is an example of an implementation of CVE-2002-1869. All of the selected NIST SARD test cases have a few specific properties that we manually selected these for. First, each test case is an example of a different CWE, which makes the results representative of a variety of different possible software weaknesses. Second, each test case contains exactly one line of source code that is an example of a particular CWE; this simplifies the parsing and accuracy metrics. Third, each test case contains no variable names, text, or function names that indicate whether the source code contains a vulnerability or not. In cases where there is a string or function name that indicates what type of example the source code is, we manually remove, or change, that string before parsing by the GPT models. Fourth, for no test cases does the vulnerable line of code occur more than once in the entire source code; this eliminates ambiguity for measuring the accuracy rate of the GPT models for identifying specific lines of source code. Any comments in the source code examples are removed before parsing by the GPT models so as to not give the GPT models natural language that describes the test cases. 

Appendix <ref> shows the exact source code for 10 of these NIST SARD test cases. The phrase vulnerable line of source will be used throughout the text to refer to the exact syntax of the vulnerable line of code – not line number. Line numbers can be somewhat ambiguous, especially for these C syntax test cases, and moreover having the GPT model output the exact source code makes checking the validity of that output much clearer. 


The exact prompt that is used for all of the GPT models is given in the text box below. Note the use of the triple backticks to delineate where the source code is, and where the JSON datastructure is. The use of the triple backticks in the prompt allows the GPT outputs to be automatically parsed by using the triple backticks, and possibly the markdown code identifier, as the delimiter. Note that this triple backtick parsing does not always work, in particular if the JSON is not correctly formatted or the triple backticks are not correctly used in the GPT output. The prompt is intended to be very source code agnostic; there are no details given of the source code in the prompt. The prompt is also intended to not bias the output towards specifying a specific number of vulnerabilities. In the case of the 36 NIST SARD test cases used in this study, the number of CWEs is always exactly 1, however the goal is for this method to more general than only identifying a single CWE and moreover we want the GPT model to be able to fail in a variety of ways in order to rigorously evaluate its accuracy (in this case, identifying more CWEs than are present would constitute a high false positive rate). 



I am seeking assistance to analyze provided source code for the purpose of detecting potential Common Weakness Enumeration (CWE) vulnerabilities. 
The key requirement is for the analysis results to be exclusively formatted in JSON enclosed by triple backticks, listing only the CWEs that are detected as 'Present' in the code, thereby omitting any 'Not Present' entries for a cleaner output. 
The exact line, or lines, of code which have this CWE vulnerability needs to be specified in the returned JSON structure, along with a succinct description of what the vulnerability category is for this source code. 
The JSON output should enumerate each identified CWE with its identification number, the exact source code syntax that causes the CWE, and a description of the vulnerability, as shown in the example structure below:




Source code:


Re-check that your output contains the required JSON data-structure before emitting text. 


The actual GPT prompt is generated by replacing the vulnerable source code in place of the text . 


The usage of requesting the vulnerable code syntax and the description of the vulnerability has two uses:


    
  * Ground the GPT response to only respond based on the provided source code
    
  * To check whether the provided syntax is correct. Because of potential ambiguity with line numbers, only exact syntax string matches are checked for, not integer line number responses. 


The description field is not parsed for accuracy or correctness, but the text contained in that field is an interesting indicator of how correct (or incorrect) the GPT output is. For this reason, when reporting results (in Section <ref>), we provide specific instances of GPT outputs, including the description fields. 


Some of the raw GPT output includes characters that are not ascii and can cause issues with saving and parsing the output. Therefore, the raw GPT output is encoded as utf-8 where character parsing errors are ignored and decoded into a parsable string using utf-8. The field in the JSON output  has repeated excess whitespace removed, as is done with the reference vulnerable line of source code from each of the NIST SARD samples; this ensures that when the string equality check is performed, the only relevant characteristic with respect to whitespace in the string that is being checked is that the whitespace is delineating characters (for example, repeated spaces, tabs, newlines do not impact this accuracy check). 

The GPT output is (attempted to be) parsed in several stages, as follows:


    
  * First, if there are triple backtick delimiters in the output, then the output is split at those characters and the first enclosed set of strings is attempted to be parsed as a valid JSON structure. Interestingly, occasionally the GPT model will add in markdown code identifiers immediately after the triple backticks - in these cases we automatically check for these and remove them if they occur so that the structure can be parsed. These markdown identifiers are given in Appendix <ref>. 
    
  * If the previous step failed, then we attempt an unstructured JSON parsing approach, where output strings are incrementally deleted from the beginning and from the end until a JSON structure can be parsed (or, not). 


Using this parsing, any valid JSON strings that are produced as part of the GPT output will be automatically parsed. This allows this study to perform a systematic evaluation of the accuracy rates of the GPT inference output. This type of data parsing is also highly scalable to a large number of static code samples - and importantly does not require any manual input, tuning, or processing. In other words, this analysis pipeline is entirely automated. 





 §.§ CWE Identification Accuracy Measures



This section lists several measures for quantifying the success rate of the GPT models for identifying known vulnerable source code. 


The first four metrics that are checked are defined as follows:


    
  * Count of parsable GPT outputs (being able to extract a JSON structure and parse it into Python, with not necessarily correct JSON fields. For example a JSON with an empty list is a valid parse. However, this step does require that there is at least one JSON key, which is correctly named as . The subsequent value is allowed to be empty or have any sort of incorrect structure (which could denote that the model is returning no found CWEs) but we need at least this key to validate that a JSON datastructure was parsed. 
    
  * The CWE field is labeled as  is correct (meaning, the correct CW number is listed), but the identified source code is not correct (meaning, it is not the labeled line of code that has the vulnerability) and does not contain the vulnerable line of code either. 
    
  * The CWE is correct and identified as  and the vulnerable code is contained within the identified code syntax but also includes other lines of code from the test case that are not themselves vulnerable as defined by the NIST SARD dataset. These instances can be interpreted as being partially correct. 
    
  * Count of fully correct instances where the CWE is correct (and identified as ) and the identified line of code is correctly identified as the vulnerable code. 


This last metric (metric number 4) is the most important accuracy measure because it counts out of the 1,000 GPT outputs how many correctly performed the computation that we are attempting to prompt the GPT model to perform. 

For none of these four metrics is the Description field, or any other additional field in the JSON, parsed for correctness or even considered. All of these measures are integers, where the larger the integer is the better the GPT model performed, but the absolute correct metric is measure 4. Note that the quantities measured in metrics 2, 3, and 4 (in the immediately preceding list of measures) are always disjoint; there are no instances that are counted in two or more of these measures. An important, and informative, aspect of these metrics is finding cases where the GPT output correctly identified the vulnerable line of code. However, since line numbers of the code are not checked or requested in the prompt (since they are somewhat ambiguously defined especially for C-family languages), a potential source of error in this quantification is if there exist multiple lines of code with the same syntax, but only one is vulnerable. This does happen, but only in one of the test cases. This is NIST SARD test case 149107, which is CWE-415 (Double Free) which by its definition contains two lines of memory free calls. Therefore, for this one test instance the identification of the vulnerable line of code is ambiguous, but for all others the unique identification is valid and correct. 


Another set of metrics that are measured are binary classification error rates. This is computed by considering the GPT output for identified CWEs as a vector of True or False entries, with length equal to the number of valid CWEs (which is currently 839). This measure disregards both the requested  and  fields (and in particular does not consider whether these entries even exist in the JSON structure. For all  entries (that identifies a valid CWE number), and whose  is exactly  (allowing for different capitalizations), that is measured as a binary state of True. All other entries (including instances of identified CWE numbers that are not valid) are measured as binary classifications of False. JSON dictionaries that are completely empty are a valid GPT output that is identifying no CWEs (meaning a binary classification vector of length 839 that is all False entries). These measures are only computed on the outputs that can be parsed; all outputs that can not be parsed are not counted towards any classification accuracy.[In the case where a JSON entry contains multiple entries with the same CWE number, that case is treated as the GPT model having identified that CWE number only once; in other words, repeated entries are ignored. This specifically works for this accuracy metric of binary classification since all other fields (except  are ignored. ] These binary classification measures allow rates of false negatives, false positives, true negatives, and true positives to be measured, and then standard machine learning classification measures such as recall and precision can thus be computed. Precision is intuitively defined as the classifier (which in this case is a GPT language model) being able to not label as positive a sample that is negative. Recall is intuitively defined as the classifier (which in this case is a GPT language model) being able to find all of the positive samples. These metrics being 0 corresponds to the highest possible error rate, and 1 is the best measure these metrics can be. 

All parsing steps require the field names in the JSON datastructure to be correct (correct spelling, no ancillary text), but differing capitalizations are always allowed. 









§ RESULTS



For each of the 36 NIST SARD test cases in Table <ref>, a total of 5,000 inference calls are performed using the methods described in Section <ref> for the 5 different open-sourced GPT models. The accuracy measures from these outputs are then summarized using several high-level metrics. Tables <ref> and <ref> present two different accuracy measure summaries. Table <ref> counts how many of the GPT outputs were parsable and correctly identified the CWE number and vulnerable line of code. Table <ref> shows the binary classification measures recall and precision. Table <ref> reports the absolute integer count of how many of the GPT outputs correctly identified the single line of vulnerable source code; importantly, these counts are independent of correct identification of the relevant CWE. These counts, as a proportion out of 1000 should be compared against uniform random sampling of the total lines of source code (shown in Table <ref>). Table <ref> shows that the correct identification of vulnerable lines of code is much higher than the correct identification of the CWE number and the vulnerable line of source code, as measured in Table <ref>. Notably in Table <ref>, only for a few of the NIST SARD test cases is the count of ever correctly identifying the syntax of vulnerable line of source code zero across all GPT models and settings; namely test cases 149203, 148871, and 2046. For some of the test cases, the proportion of correctly identified vulnerable line of source code is very high. The test cases where the vulnerable line of source was not uniquely identified in Table <ref> does not necessarily mean that the correct CWE identification was always 0 (when analyzing strictly whether the identified CWE was correct) - an example of this is test case 2046. 

Table <ref> shows that for the code test cases where the accuracy is 0, the 5 different language models do not perform uniformly bad – for some of the models the accuracy is always 0 for a given test case, but then when analyzed with a different model the accuracy is non-zero (an example of this is NIST SARD test case 500757). This means that some of the GPT models do perform better than other GPT models for specific test cases. And moreover, there is not consistently a single model that always outperforms the other models. 


Table <ref> shows the best performing GPT model and inference temperature in terms of recall and precision measures. Finally, Table <ref> counts the most frequently identified false positives. The best performing GPT model and inference temperature shows that there is no specific model, or temperature setting, that consistently performs the best - and in in particular, every one of the tested GPT models was the best performer for at least one of the CWE test cases. This shows that this task does require diversity of the model architecture, and very likely the type of text on which the models were trained. Lastly, even with these best performing GPT models and inference temperatures some of the CWE test cases were never correctly identified, meaning that the accuracy rates for all GPT models and settings were always 0 - that happened for 11 out of 36 of the test cases as denoted by asterisks in Table <ref>. 


Section <ref> then gives several examples of correct JSON output from the GPT models where the vulnerable line of source code was identified, and where the description is correct in some cases and incorrect in other cases. 



Table <ref> quantifies the exact counts of false positive CWE id occurrences for the full matrix of the 5 GPT models and NIST SARD test cases. Specifically, for each NIST SARD test case and GPT model, Table <ref> reports the most frequently mis-identified CWE number. Table <ref> demonstrates two important trends. First, the proportion of false positives is very high, and this is consistent across the different GPT models. Second, for most test cases and GPT models the distribution of incorrect CWE ids is not uniformly distributed across the possible CWE numbers - it is heavily biased towards a few specific CWE numbers. For example,  shows a clear bias towards predicting CWE-124 for multiple test cases. Similarly,  shows a consistent bias for (incorrectly) predicting CWE-123, and  shows a consistent bias for (incorrectly) predicting CWE-476. 



































 §.§ Example Correctly Identified Vulnerable Lines of Code



This section enumerates several parsed JSON outputs from the various GPT models and code test cases. These outputs were hand selected to be representative of cases that are correct (in the sense that the CWE number is correctly identified, as is the vulnerable line of source code), but whose description fields are in some cases correct and in other cases incorrect. Showing explicit examples of the output is intended to primarily show the variability of the generated Description field, and also to show concretely what successful static code analysis identification looks like from the GPT models. 

No text changes were made to these GPT outputs except for formatting to fit within the text boxes. Note that these outputs are specifically examples of text from the enclosed triple backtick fields - any additional prose or output from the model is not shown. 



[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 500843. Produced from the Turdus GPT model with temperature 0.8. Note that the generated description is partially correct - CWE-476 is NULL pointer dereference but the description also mentions Divison-by-Zero which is not correct. ]






[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 500843. Produced from the zephyr-7b-alpha GPT model with temperature 0.7. Note that the generated description is entirely incorrect.]



[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 500843. Produced from the Turdus GPT model with temperature 1.0. Note that the generated description is entirely incorrect. ]





[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 500843. Produced from the Llama-2-70b-chat-hf GPT model with temperature 0.8. Note that the provided description is mostly correct in that the correct vulnerability name was identified but there is not a function named f in the source code. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 1792. Produced from the Turdus GPT model with temperature 0.3. Note that the generated description is correct.]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 1792. Produced from the Turdus GPT model with temperature 0.4. Note that the generated description is correct.]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 1792. Produced from the Turdus GPT model with temperature 0.4. Note that the generated description is correct except that the identified print statement in the description text is incorrect.]





[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 1645. Produced from the Turdus GPT model with temperature 0.5. The generated Description field in this example is correct. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 1645. Produced from the zephyr-7b-alpha GPT model with temperature 0.9. The generated Description field in this example is correct. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149165. Produced from the Llama-2-70b-chat-hf GPT model with temperature 0.9. The generated Description field in this example is entirely incorrect. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149165. Produced from the Llama-2-70b-chat-hf GPT model with temperature 0.1. The generated Description field in this example is correct. ]



[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149165. Produced from the Mistral-7B-Instruct-v0.1 GPT model with temperature 0.6. The generated Description field in this example is correct. ]



[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149165. Produced from the zephyr-7b-beta GPT model with temperature 0.2. The generated Description field in this example is correct. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149165. Produced from the zephyr-7b-beta GPT model with temperature 0.4. The generated Description field in this example is not correct. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 500757. Produced from the Turdus GPT model with temperature 0.8. The generated Description field in this example is correct. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 1989. Produced from the Turdus GPT model with temperature 0.9. The generated Description field in this example is correct. ]






[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149111. Produced from the Llama-2-70b-chat-hf GPT model with temperature 0.3. The generated Description field in this example is correct. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149111. Produced from the Mistral-7B-Instruct-v0.1 GPT model with temperature 0.7. The generated Description field in this example is correct. ]



[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149241. Produced from the zephyr-7b-alpha GPT model with temperature 0.2. The generated Description field in this example is correct. ]



[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 313. Produced from the Llama-2-70b-chat-hf GPT model with temperature 0.6. The generated Description field in this example is correct. ]









§ DISCUSSION AND CONCLUSION




This study has examined the task of correctly identifying software weaknesses and vulnerabilities in software using open source Generative Pre-Trained Transformer (GPT) models. This was able to be examined because of two specific reasons - the first is the existence of the NIST SARD dataset, which is a very well labelled dataset that includes the vulnerable line of code and the relevant CWE. The second is the specific prompting template usage that allows fully automated extraction of JSON output for the vulnerability detection - and therefore error rates can be quantified for a large range of NIST SARD test cases, GPT models, and GPT model settings. Thus, this study reports a systematic analysis of automated vulnerability detection using GPT models. 

In general, our findings are that the correct CWE identification rate varies significantly depending on the specific code test case that is analyzed. For some test cases the correct CWE identification (and relevant line code of syntax) is surprisingly high, but for other test cases the identification accuracy rate is always 0 for all GPT models and temperature settings. In general, the error rates of this vulnerability detection is too high – in particular, the false positive rate is very high. However, it is notable that for some cases, seemingly the more popular classes of CWEs that are mentioned in the type of internet text on which these GPT models were trained, the identification rates are quite good. As specific examples of the lowest error rate results (from Table <ref>), we list several specific examples of CWE identification binary classification accuracy rates (without considering rates for correctly identifying the vulnerable line of source code). Test case 1989 had a best recall of 0.928, a best precision of 0.709 (with a parsability rate of 0.42) when analyzed with  GPT model and temperature 0.1. Test case 149165 had a best recall of 1.0, a best precision of 1.0 (with a parsability rate of 1.0) when analyzed with  GPT model and temperature 0.1. Test case 149111 had a best recall of 0.727, a best precision of 0.675 (with a parsability rate of 0.77) when analyzed with  GPT model and temperature 0.5. Test case 149241 had a best recall of 1.0, a best precision of 0.5 (with a parsability rate of 1.0) when analyzed with  GPT model and temperature 0.1. Given the high accuracy rates of at least some of these test cases, it seems plausible that some of the current open source GPT models, if thorough verification is used, could correctly identify certain classes of vulnerabilities, such as buffer overflows, with very low error rates. 


It should be emphasized that this particular software security analysis task requires a number of semantic connections to be made in the GPT model inference process (and of course, the correct specification of the JSON syntax) in order for the (automated) vulnerability identification to be correct. It is not clear to what extent a pre-trained language model can learn these semantic connections - but it is possible least for some of the CWEs, these connections were learned in the standard open source GPT models that we tested. This is is a non-trivial finding, and suggests that GPT models do have some capability to identify security vulnerabilities. However, it also needs to be emphasized that such usage of current GPT models should not be used in cases where the accuracy can not be tested. In other words, these GPT models and vulnerability identification prompting should not be used in software engineering tasks – they should be used in cases where careful and precise testing of the GPT output can be performed for the purpose of examining specific characteristics of source code of interest. The primary reason for this is that the false positive rate is high - and not only is it high but the false positives are biased towards incorrectly identifying specific CWEs. In other words, simply measuring which CWE is predicted most frequently in a GPT output distribution (for a non-zero inference temperature) will typically not be correct. 


The primary limitation with increasing the scope of more code test cases to more thoroughly test the capabilities of GPT models for software security analysis is the total amount of compute time that is required for performing these inference calls, especially in a systematic study of varying inference parameters. Future studies that increase the number of labeled source code examples (e.g., from the extensive amount of data available in the NIST SARD dataset) that are used for testing the accuracy GPT automated vulnerability identification would be very relevant. 


An aspect of this study that is unclear is what details of the source code are the GPT models using to infer relevant CWE's in the source code. In particular, what semantic details are used to parse whether a CWE is more likely or less likely. In some cases, it could be that context, such as what types of libraries are used or what types of strings are being constructed (such as SQL query strings), is sufficient to reduce the total number of possibly applicable CWEs so that random guessing among those feasible candidates accounts for some of the GPT model's performance. The metrics in this study do not determine what components of the source code are important; this would require quantifying the explainability of GPT models <cit.> for this task. We leave this study open to future research, and emphasize that this type of detailed explainability analysis is absolutely a critical component of study before these GPT models are incorporated into any type of vulnerability detection in production software development and testing. 






§ ACKNOWLEDGMENTS



Sandia National Laboratories is a multi-mission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC (NTESS), a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration (DOE/NNSA) under contract DE-NA0003525. This written work is authored by an employee of NTESS. The employee, not NTESS, owns the right, title and interest in and to the written work and is responsible for its contents. Any subjective views or opinions that might be expressed in the written work do not necessarily represent the views of the U.S. Government. The publisher acknowledges that the U.S. Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this written work or allow others to do so, for U.S. Government purposes. The DOE will provide public access to results of federally sponsored research in accordance with the DOE Public Access Plan.














§ MARKDOWN COMPUTER CODE LANGUAGE IDENTIFIERS











§ NIST SARD TEST CASES



This section shows the exact source code for 10 of the NIST SARD test cases that are used in this study. The single vulnerable line of source code is highlighted as red text for each test case. 


.49
[caption= NIST SARD Test case 500843, instance of CWE-476 ,captionpos=b,label=SARD_source_code:500843,language=C,showstringspaces=false,style=base,keywordstyle=]

struct Token 
    const Token* nextArgument() const;
    const Token* next() const;
    int varId() const;
    void str() const;;
void f(const Token *first) 
    first = first->nextArgument();
    if (first)
        first = first->next();
    @first->str();@



.49
[caption= NIST SARD Test case 1792, instance of CWE-79. Note that some of the strings in this test case have been modified so that terms related to cross site scripting are removed. ,captionpos=b,label=SARD_source_code:500843,language=C,showstringspaces=false,style=base,keywordstyle=]

#include <stdio.h>
#include <cgic.h>
#include <string.h>
#include <stdlib.h>

int cgiMain() 

	cgiHeaderContentType("text/html");
	 
	fprintf(cgiOut, "<html><head>");
	fprintf(cgiOut, "<title>Text: 1</title></head>");
	fprintf(cgiOut, "<body><h1>Text</h1>");
	 
	char q[4][1024];
	unsigned int i = 0;
	for (; i < 4; ++i)
		char name[2];
		sprintf(name,"q
		cgiFormString(name, q[i], sizeof(q[i]));
		if (strlen(q[i]))
		
			@fprintf(cgiOut, "Value number 
		
			
	 
	fprintf(cgiOut, "</body></html>");
	return 0;





.49
[caption= NIST SARD Test case 1779, instance of CWE-463. ,captionpos=b,label=SARD_source_code:1779,language=C,showstringspaces=false,style=base,keywordstyle=]

main ()

char *foo;
int counter;
foo=malloc(sizeof(char)*10);
for (counter=0;counter!=14;counter++)
@foo[counter]='a';@
printf("









.49
[caption=NIST SARD Test case 1645, instance of CWE-20. ,captionpos=b,label=SARD_source_code:1645,language=C,showstringspaces=false,style=base,keywordstyle=]

#include <stdio.h>
#include <stdlib.h>

#define	MAXSIZE		40

void test(char *str)
	char buf[MAXSIZE];
	snprintf(buf, sizeof buf, "/bin/echo 
	buf[MAXSIZE-1] = 0;
	@system(buf);@


int main(int argc, char **argv)
	char *userstr;
	if(argc > 1) 
		userstr = argv[1];
		test(userstr);
	
	return 0;









.49
[caption=NIST SARD Test case 149165, instance of CWE-121. ,captionpos=b,label=SARD_source_code:149165,language=C,showstringspaces=false,style=base,keywordstyle=]

#include <stdlib.h>
#include <stdio.h>
#include <time.h>
#include <string.h>

unsigned int getRand()

	unsigned int r;
	FILE *f;
	f = fopen("/dev/urandom", "rb");
	if(f == NULL)
	
		fprintf(stderr, "Error opening file");
		exit(-1);
	
	if(fread(   r, sizeof r, 1, f) != 1)
	
		fprintf(stderr, "Error reading file");
		fclose(f);
		exit(-1);
	
	if(fclose(f) != 0)
		fprintf(stderr, "Error closing file");

	return r;


unsigned plop() 
	return getRand() 


int main(int argc, char *argv[])

	char buffer[256];
	memset(buffer, 0, sizeof(buffer));
	@buffer[plop()] = '!';@		 
	printf("
	return 0;









.49
[caption=NIST SARD Test case 2015, instance of CWE-329. ,captionpos=b,label=SARD_source_code:2015,language=C,showstringspaces=false,style=base,keywordstyle=]

#include <openssl/evp.h>

main()
EVP_CIPHER_CTX ctx;
char key[EVP_MAX_KEY_LENGTH];
char iv[EVP_MAX_IV_LENGTH];
int b=8;
RAND_bytes(key, b);
memset(iv,0,EVP_MAX_IV_LENGTH);
@EVP_EncryptInit(   ctx,EVP_bf_cbc(), key,iv);@
return;









.49
[caption= NIST SARD Test case 500757, instance of CWE-787 ,captionpos=b,label=SARD_source_code:500757,language=C,showstringspaces=false,style=base,keywordstyle=]
#include <cstdlib>
void f()

  char *p; p = (char *)malloc(10);
  @p[10] = 7;@
  free(p);





.49
[caption=NIST SARD Test case 149135, instance of CWE-489. ,captionpos=b,label=SARD_source_code:149135,language=C,showstringspaces=false,style=base,keywordstyle=]

#include <stdio.h>
#include <stdlib.h>
#include <stdbool.h>
#include <string.h>

static bool debug = false;

void promote_root() 
	if (debug) 
		 
		printf ("# You are root now...");
	


int main(int argc, char *argv[])

	if (argc > 1)
	
		unsigned i;
		const unsigned nbArgs = argc;

		for (i = 1; i < nbArgs; ++i)
		@
			if (!strncmp(argv[i],"-debug",6)) @
			
				debug = true;
				printf("Move to debugging mode");
				 
				if (strlen(argv[i]) >= 11        !strncmp(argv[i]+6,":root",5))
				
					promote_root();
				
			
			
	
	else
	
		printf("No args...");
	

	return 0;










.49
[caption=NIST SARD Test case 149203, instance of CWE-416. ,captionpos=b,label=SARD_source_code:149203,language=C,showstringspaces=false,style=base,keywordstyle=]

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

typedef struct str_t str;
struct str_t 
	union 
		int   a;
		char *b;
	 foo;
;

int main()

	str container;
	char *str1;
	container.foo.a = 0;
	
	if ((container.foo.b = (char *)malloc(256*sizeof(char))) != NULL)
	
		strcpy(container.foo.b, "Falut!");
		container.foo.b[0] = 'S';
		printf("
		free(container.foo.b);
		str1 = malloc(65536); 
 		if (str1 == NULL)
        		printf("Memory allocation problem"); return 1;

		free(str1);
		
		if (container.foo.b)
			@container.foo.b[0] = 'S'; printf("
	
	return 0;









.49
[caption=NIST SARD Test case 149111, instance of CWE-134. ,captionpos=b,label=SARD_source_code:149111,language=C,showstringspaces=false,style=base,keywordstyle=]

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#define MAX_SIZE 255

typedef struct cont_t cont;
struct cont_t 
	char fmt[MAX_SIZE];
;

int main(int argc, char *argv[])

	if (argc > 1) 
		cont container = .fmt="";
		strncpy(container.fmt, argv[1],MAX_SIZE-1);
		container.fmt[MAX_SIZE - 1] = '';
		@printf(container.fmt);		@		 
	
	return 0;


















 Paper Title: AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation.





[NO \title GIVEN]
    [NO \author GIVEN]
    August 5, 2024
======================



§ ABSTRACT  




Large Language Model (LLM) based agents have garnered significant attention and are becoming increasingly popular. 


Furthermore, planning ability is a crucial component of an LLM-based agent, involving interaction with the environment and executing actions to complete a planning task, which generally entails achieving a desired goal from an initial state.

This paper investigates enhancing the planning abilities of LLM-based agents through instruction tuning, referred to as agent training.
Recent studies on agent training have demonstrated that utilizing expert-level trajectory data (sequences of action-observation pairs) for instruction-tuning LLMs effectively enhances their planning capabilities.
However, existing work primarily focuses on synthesizing trajectories from manually designed planning tasks and environments. The labor-intensive nature of creating these environments and tasks impedes the generation of sufficiently varied and extensive trajectories for agent training.
To address this limitation, this paper explores the automated synthesis of diverse environments and a gradual range of planning tasks, from easy to difficult. 
In response, we introduce a framework, , that leverages LLMs first to generate environments and subsequently generate planning tasks conditioned on these environments.


Specifically, to improve environmental diversity, we propose using an inspiration corpus composed of various domain-specific text segments as the context for synthesizing environments. 
Moreover, to increase the difficulty diversity of generated planning tasks, we propose a bidirectional evolution method, , that evolves planning tasks from easier and harder directions to synthesize a task set with a smoother difficulty curve, thereby enhancing the learning process of LLMs more effectively.
These methods collectively contribute to the generation of diverse trajectory data for instruction-tuning.


Based on , we greatly expanded the number of environments and planning tasks available for agent training.


The evaluation results derived from AgentBoard show that greatly improves LLMs' planning ability, e.g., the instruction-tuned Llama-3 8B surpasses GPT-3.5 in overall performance. Moreover, in certain tasks, it even outperforms GPT-4.




§ INTRODUCTION




Recently, owing to advancements in Large Language Models (LLMs) <cit.>, the LLM-based Agents have garnered widespread attention from the artificial intelligence community. 
Generally, an LLM-based agent refers to utilizing LLMs to perceive the environment, make decisions, and execute actions to substitute or help people accomplish some specific tasks <cit.>.
Furthermore, planning is often regarded as one of the most important applications of LLM-based agents, such as robotic planning <cit.>, travel planning <cit.>, etc. 
In this study, planning is conceptualized as the systematic process of identifying a sequence of executable actions within a given environment to complete a planning task, defined as the transition from an initial state to achieve specified goal conditions, considering constraints and available resources <cit.>.




Improving planning capabilities through instruction-tuning LLMs is a significant research problem, referred to as agent training.

As shown in Figure <ref>, similar to imitation learning <cit.>, a typical agent training process can be divided into three stages:
(i) Preparing environments and planning tasks.
(ii) Synthesizing expert-level trajectories (sequences of action-observation pairs) on these planning tasks. For example, utilizing state-of-the-art LLMs (e.g., GPT-4 <cit.>) as the agent and filtering trajectory based on reward score <cit.>.
(iii) Instruction-tuning LLMs with the synthesized trajectory data.
Recently, the effectiveness of enhancing the planning capabilities of LLMs through agent training has been demonstrated by many studies <cit.>.

Despite their success, one key limitation of these works is that they primarily rely on manually designed environments and planning tasks. 
The labor-intensive nature of creating environments and planning tasks hinders the generation of diverse and extensive trajectory data.

More explicitly, designing diverse environments requires defining a range of rich and practical scenarios, and implementing these environments typically involves the participation of human experts with programming skills.
Additionally, formulating tasks often demands creating a task set with a gradual difficulty progression.
Due to this constraint, existing agent training studies typically use only a few environments for data synthesis.





To address the aforementioned deficiencies, this paper introduces an automatic framework  that utilizes LLMs to construct diverse environments and planning tasks for agent training, expanding the available environments from a few to hundreds.




More specifically, is structured around two stages:

(1) Environment Generation: 
Achieving sufficient environmental diversity is essential for creating diverse planning tasks, which involves covering a broad range of scenarios and domains. 
To ensure this, we use an inspiration corpus composed of diverse text segments as context for generating environment specifications with LLMs, where actions, restrictions, and other details are defined using natural language.

For example, in Figure <ref>, we randomly selected a text segment from the inspiration corpus: “How to boost your diet with peanut butter powder?” This prompted the generation of a related environment specification: “You are a nutritionist tasked with creating a new healthy recipe book that incorporates peanut butter powder as a key ingredient”.
Subsequently, we prompt the LLM to produce the corresponding code based on this specification, which may be composed of Python, Planning Domain Definition Language (PDDL) <cit.>, or other domain-specific languages.
Furthermore, we constructed an environment library to serve as in-context examples and iteratively expanded it by incorporating high-quality newly generated environments.


(2) Task Generation: Conditioned on the generated environment, we aim to create multiple planning tasks. 
In this stage, it is crucial to have a gradual set of tasks ranging from easy to difficult, i.e., difficulty diversity.
To achieve greater difficulty diversity, we propose a bidirectional evolution method, , where the LLM first generates random planning tasks and then evolves these tasks by applying constraints towards both simplification and increased difficulty. 
The created task set with has a smooth difficulty curve that facilitates LLMs' smoother acquisition of planning skills.




















To verify the effectiveness of our method, we synthesized environments and planning tasks based on PDDL <cit.> and constructed a dataset comprising 592 environments, each with 20 tasks. 
We then used a domain-independent planner to obtain 7,246 high-quality trajectories.

Subsequently, we used this trajectory data for instruction-tuning a series of LLMs and demonstrated the trained model on AgentBoard <cit.>. 

Since our instruction-tuning dataset is composed of trajectory synthesized from PDDL-based planning tasks, we refer to evaluation tasks implemented in PDDL as in-domain tasks and tasks implemented in other programming languages as out-of-domain tasks. 

Importantly, this evaluation was conducted in a zero-shot manner without utilizing any trajectory data from these tasks. 

Experimental results show that achieved over five times improvement compared to the raw LLama3-8B on in-domain tasks (11.67 vs. 1.67), and overall performance surpassed GPT-3.5. Additionally, it outperformed GPT-4 in specific tasks.
In out-of-domain tasks, also demonstrated similar experimental outcomes. Specifically, it significantly improved success rates compared to the raw LLama3, achieving 29.1 and 4.47 improvement on Alfworld and BabyAI, respectively. On Alfworld, even surpassed the performance of GPT-3.5 (29.1 vs. 17.2).
In summary, the proposed environment and planning task generation method can help improve planning ability. Moreover, not only can in-domain tasks benefit from this, but out-of-domain tasks also improve, which confirms both the effectiveness and generalization.

Our contributions can be summarized as follows:

    
  * We introduce , which, as far as we know, is the first framework for automatically generating diverse planning tasks and environments targeted for LLM-based agent training.
    
  * We propose utilizing an inspiration corpus as the context for generating environments with LLMs, resulting in 592 diverse environments that encompass a broad range of scenarios.
    
  * We propose a bidirectional evolution method that evolves seed planning tasks in both simpler and more complex directions, thereby constructing a task set with a smoother difficulty curve.
    
  * We constructed an agent instruction-tuning dataset with 7246 high-quality trajectories through . LLMs instruction-tuned with this dataset achieved massive improvement in both in-domain and out-of-domain planning tasks, which validated the effectiveness and generalization of .







§ PRELIMINARY























 §.§ Planning Problems


We consider goal-directed deterministic planning problems <cit.>, which are formally defined as a tuple 𝒫 = (𝕋, 𝔼), where 𝔼 denotes the environment in which the agent interacts and 𝕋 denotes the task that the agent needs to complete.
Specifically, an environment 𝔼 typically models a world, encompassing the definitions of the action space 𝒜 and state space 𝒮, as well as the transition function 𝒯: 𝒮×𝒜→𝒮.
Task 𝕋 is further defined by the tuple 𝕋 = (𝒢, ℐ), where 𝒢 refers to the goal conditions and ℐ refers to initial states of the agent.
The initial states ℐ are a subset of the state space 𝒮_i that specifies the starting conditions of the agent.
The goal 𝒢 is a subset of the state space 𝒮_g that specifies the desired outcomes or conditions. Specifically, 𝒢 can be expressed as 𝒢 = { s ∈𝒮_g|ϕ(s) = true}. Here, ϕ(s) is a boolean-valued function representing conditions or propositions that must be satisfied for the state s to be considered part of the goal set. 





 §.§ Planning Problem Implementation





















A planning problem can be implemented with programming languages such as Python or domain-specific languages such as Planning Domain Definition Language (PDDL) <cit.>.
For example, in a PDDL-based planning problem, the domain PDDL file can be regarded as the environment 𝔼, defining states (predicates) and actions and specifying the transition function using preconditions and effects of each action. The problem PDDL file, on the other hand, can be seen as the task 𝕋. Both initial states and goal conditions are typically defined as combinations of predicates.
Another widely used programming language for constructing planning problems is Python.

For example, in OpenAI gym[<https://www.gymlibrary.dev/index.html>], a planning problem will be implemented as a Python class, where the transition function is implemented as a method of the class, usually named the "step" or "update" function. Meanwhile, the goal 𝒢 is typically represented as a reward function that indicates the objective of the task, and the initial states ℐ are defined in a method named "reset."







 §.§ Large Language Model based Agent

An LLM-based agent leverages a pre-trained language model to operate within the defined environment 𝔼 and complete the given task 𝕋.

Given an environment 𝔼, the LLM-based agent perceives its state 𝒮 and takes actions 𝒜 based on its understanding and processing of the input. 
The transition function 𝒯: 𝒮×𝒜→𝒮 remains consistent, where the LLM-based agent determines the next state by generating appropriate actions through natural language processing.
The goal G guides the LLM-based agent in selecting actions that maximize the reward. The agent utilizes the language model to interpret the task requirements and generate actions that align with achieving the specified goal.
In essence, the LLM-based agent forms a policy π: 𝒮→𝒜 using the LLM, where π(s) is the action taken in state s based on the LLM's understanding and processing of the task.



















































§ METHODOLOGY





  
Problem Definition


 
The process of generating planning tasks can be formalized as a function f: I → (𝕋, 𝔼), where I is the input space (e.g., instructions or prompts) and tuple (𝕋, 𝔼) is the space of all possible planning tasks and environments.
Based on the definition in Section <ref>, we can express this as f(i) = (𝕋_i, 𝔼_i),    i ∈ I, where 𝕋_i is the generated planning task and 𝔼_i is the generated environment for a given input i.
Our two-stage approach can be further decomposed as follows: 

i) Environment Generation (<ref>):
In the first stage, we generate the environment 𝔼_i based on the input instruction i. This can be represented as 𝔼_i = g_𝔼(i),  where g_𝔼 is the environment generation function that takes the instruction i as input and produces the environment 𝔼_i.
ii) Task Generation (<ref>):
In the second stage, we generate the task 𝕋_i, conditioned on the environment 𝔼_i generated in the first stage. This can be expressed as:
𝕋_i = g_𝕋(i, 𝔼_i),
where g_𝕋 is the task generation function that takes both the original instruction i and the generated environment 𝔼_i as inputs to produce the task 𝕋_i.
We will detail the implementation of these two stages in the following section.



 §.§ Environment Generation







  
Overview


As is shown in Figure <ref>, we propose a sophisticated framework for environment generation structured around three main components: (1) an environment specification generation module where an LLM first generates a specification of the environment, typically including a general overview of the environment, descriptions of the state space and action space, and definitions of the transition functions; (2) an environment implementation module that generates corresponding code based on the environment specification; and (3) an environment library that stores previously generated high-quality environments, serving as a comprehensive environment dataset and providing in-context examples for generating new environments. Each component will be elaborated on in the following paragraph.



  
Environment Specification
We initially prompt the LLM to generate an environment specification, which typically includes an overall depiction of the environment, specific actions and their corresponding preconditions and effects, and certain restrictions within the environment.
The environment specification will serve as the basis for generating specific environment codes. This two-stage approach, similar to the Chain-of-Thought <cit.>, can better assist the LLM in creating high-quality environments.
For generating environment specifications, One direct approach is to prompt LLMs to generate random environments. 
However, due to the inherent inductive bias of LLMs, they struggle to generate diverse environments in this way.
Therefore, to address this issue, we build an inspiration corpus D={t_0,t_1,·,t_n}, containing sufficiently diverse text segments used to serve as the "inspiration" for generating environment specification with LLMs. 
More specifically, when generating an environment, we first sample a text segment t_i from D, then prompt the LLM to generate a related environment based on t_i.
Taking the example in Figure <ref>, we first sample a text segment "How to boost your diet with peanut butter powder?" from D. Then we prompt an LLM to generate a related environment where the agent is defined as a nutritionist tasked with creating a new healthy recipe book that prominently features peanut butter powder as a key ingredient. 
This approach significantly enhances the diversity of generated environments, thereby empowering more generalized agent training.
The inspiration corpus can be implemented in various ways, such as using a large-scale pre-trained corpus like Common Crawl. Alternatively, a domain-specific corpus, such as a code generation dataset <cit.>, can be used to generate environments for a specific domain. This paper uses LIMA <cit.> as the inspiration corpus, an instruction-tuning dataset with sufficient diversity.



  
Environment Implementation
Conditioned on the generated environment specification, we generate its corresponding code, i.e., implementing the environment.
This can be formulated as a typical code-generation problem with LLMs.
We also introduce a validation tool capable of capturing syntax errors to provide feedback during the code generation process, thereby iteratively refining it.



  
Environment Library
We define the library at iteration t as: L_t = L_0 ∪⋃_k=1^t{𝔼_i | 𝔼_i = g_𝔼(i, L_k-1), i ∈ I_k, v(𝔼_i) = true},
where L_0 is the initial seed library, and the union represents all verified environments generated up to iteration t.
This iterative process allows continuous expansion and refinement of the environment library, potentially leading to increasingly complex and diverse environments over time.





 §.§ Task Generation








  
Overview
As depicted in Figure <ref>, conditioned on the generated environments, we prompt LLMs to generate corresponding planning tasks.
We employ a two-stage generation approach for creating a diverse range of planning tasks in terms of difficulty. 

We begin by prompting the LLM with a specific environment, enabling it to generate an initial set of planning tasks in a zero-shot way.
Subsequently, we adjust these tasks to make them simpler or more challenging, forming a comprehensive set of planning tasks.




  
Bidirectional Evolution
Many studies have proposed evolving instructions, primarily focusing on making instructions more difficult <cit.>. The effectiveness of this approach relies heavily on the assumption that LLMs inherently possess the ability to follow simple instructions. 
However, according to findings from some studies <cit.>, LLMs often exhibit poor performance even in simple planning tasks. 
Therefore, we propose , which introduces evolution in two directions: easy-evol and hard-evol. 
Easy-evol typically involves simplifying the goal conditions. The motivation is that easier tasks can facilitate learning when the agent performs poorly and cannot directly learn from typically difficult goals. 
Conversely, hard-evol usually involves making the goal conditions more complex, increasing the number of steps required for the agent to complete the task. This can further enhance the agent's capability to perform the planning task.
To our knowledge, we are the first to introduce bidirectional evolution in the agent data generation scenario.
The prompt examples are shown in Figure <ref>.












§ EXPERIMENTS





To evaluate the effectiveness of the proposed framework, we synthesize environments and planning tasks using the Planning Domain Definition Language (PDDL), a widely adopted programming language for planning.
Subsequently, we evaluate its performance across various unseen planning tasks in a zero-shot manner. To validate the effectiveness and generalizability of , we categorized the evaluated tasks into two distinct groups: 
i) In-Domain Tasks: Planning tasks implemented using PDDL.
ii) Out-of-Domain Tasks: These comprise tasks developed using other programming languages, such as Python.



 §.§ Experimental Setup




  
Evaluation Tasks

For In-Domain Tasks, we select four widely used PDDL-based planning tasks: Blocksworld, Gripper, Tyreworld, and Barman <cit.>.

More explicitly, Blocksworld requires an agent to achieve a target configuration by moving blocks, while Gripper involves moving objects between different rooms.
Tyreworld simulates changing a car tire, including removing the flat tire, replacing it with a spare, and installing the new tire. 
Barman emulates a bartender's tasks in mixing cocktails, which include combining various ingredients, using shakers, and garnishing drinks.

For Out-of-Domain Tasks, we select two challenging partial-observable planning tasks: Alfworld <cit.> and BabyAI <cit.>. 
Alfworld is an environment designed to test agents' abilities to perform everyday household tasks.
While in BabyAI, the agent interprets and executes natural language instructions in a grid-world setting.




  
Evaluation Metrics

We utilized two evaluation metrics to evaluate planning ability: success rate and progress rate <cit.>. 
During each interaction round, we assigned a progress rate, denoted as r_t, to measure the progression towards the goal state g. 
As the agent transitions through states s_t = [s_0, …, s_t], its progress is assessed using a matching score f(·, g) → [0, 1], which quantifies the similarity between the current state and the goal state. 
Initially, r_t is set to 0, indicating no progress. 
Only when the progress rate reaches 1 does the success rate attain 1; all other scenarios yield a 0 outcome. 
The success rate reflects the agent's capacity to complete a comprehensive task.



  
Baselines


We compare with a series of widely-used multipurpose foundation models that exhibit state-of-the-art performance, such as GPT-3.5 <cit.> (gpt-3.5-turbo) and GPT-4 <cit.> (gpt-4-turbo), CodeLlama-7B <cit.>, Mistral-7B <cit.>, Llama2-7B <cit.>, and Llama3-8B <cit.>. We use their instruct-tuned versions for all multipurpose foundation models (<ref>). Additionally, some models have undergone specialized training on agent trajectory data, such as AgentLM-7B <cit.>.
AgentLM is instruction-tuned using a combination of a lightweight trajectory dataset across 6 environments and a larger general instruction-tuning dataset.









  
Implementation Details

We followed the environment and task implementation of AgentBoard <cit.>. For the configuration of evaluation tasks, we employ act-only prompting <cit.>, setting the maximum step length for the LLM agent to 30.
We selected LIMA <cit.> as the text corpus D for generating environments, which leverages various data manipulation techniques to ensure a diverse range of instructions. 
For environment generation and task generation, we employ GPT-4 [We applied the gpt-4-20230321 API from Azure OpenAI service.], configuring the inference parameters with a temperature of 0 and a top_p value of 0.95. 

Based on , we generated a total of 592 environments. For each environment, we generated ten unconditioned tasks, which were then evolved into ten refined tasks using . To generate trajectory data for training, we utilized the domain-independent planner FastDownward[<https://www.fast-downward.org/>], ensuring optimal trajectory data. This process ultimately led to 7246 trajectories. 
More details of the dataset can be found in Appendix <ref> and <ref>.

Since the trajectory data is structured, such as "pickup(o1)", we employ GPT-4 to generate a natural language mapping, for example, "pick up object {arg1}", to transform structured actions into natural language actions. We detailed the generation of natural language mapping are presented in <ref>.

During the training process, we employed Llama3-8B [https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct] as our foundation model. The hyperparameters were configured as follows: batch size of 64, 10 epochs, a context length of 4096 tokens, no warmup steps. Checkpoints from epochs 5 through 10 were preserved and subsequently evaluated on In-Domain Tasks. The model exhibiting optimal performance was selected for further assessment on Out-of-Domain Tasks.
We conducted all experiments utilizing a cluster of eight V100 GPUs.



 §.§ Evaluation on In-Domain Tasks
























































































































Despite its relatively small size, overall outperforms GPT-3.5 in success rate (11.67 vs. 5.0). Furthermore, in the barman task, even surpassed GPT-4's performance (15 vs. 10). also achieved a comparable level to GPT-4 in tyreworld.
Compared to other models with similar parameter scales, consistently outperforms them across four distinct tasks.


Compared to Llama3, our model shows an overall success rate and progress rate improvement of 10 and 9.95, respectively. Notably, in several tasks where the success rate of Llama3 is zero (gripper, blockworld, tyreworld), achieves significant breakthroughs, further demonstrating the effectiveness of the dataset.






We can draw several conclusions from previous discussions:
i) surpasses GPT-3.5 in overall performance and achieves results that either exceed or are comparable to GPT-4 on certain specific tasks;

ii) fine-tuned Llama3 has achieved a significant improvement in success rate;
iii) consistently surpasses other models with similar parameter scales in performance.




 §.§ Robustness





















To validate the robustness of the constructed dataset with , we conducted a series of experiments to evaluate its performance across different foundation models. 
We selected several widely used 7-8B foundation models, including Llama3-8B, CodeLLama-7B, and Mistral-7B, to test the versatility and effectiveness of .
As is shown in Table <ref>, all three models exhibited significant improvements after training, with Llama3-8B showing the highest success rate increase of 10.0 and CodeLLama-7B demonstrating a maximum progress rate increase of 9.9. These experimental results prove that the dataset constructed with for agent training is highly effective across different models.



 §.§ Evaluation on Out-of-Domain Tasks










§ RELATED WORK




  
Large Language Model based Agent.

Large Language Models have demonstrated exceptional reasoning capabilities <cit.>. Owing to such abilities, over the past two years, LLM-based agents have experienced significant development <cit.>.
Unlike the traditional method of using LLMs for text-based reasoning, such as Chain-of-Thought <cit.>, LLM-based agents typically involve interaction with the environment, adjusting the output in a closed-loop manner based on environmental information.
These LLM-based agents, now fortified with capabilities like Memorizing <cit.>, Tool-use <cit.>, and Planning <cit.>, exhibit a marked enhancement in their overall efficacy. 
Although this paper mainly focuses on the planning capability of LLM-based agents, we believe has the potential to generalize to other scenarios of LLM-based agents.



  
Planning with Large Language Models.

Planning is one of the key applications of LLM-based agents, applicable in various scenarios such as robotic planning <cit.>, travel planning <cit.>, calendar scheduling <cit.>, code generation <cit.> and others <cit.>. 
It is typically defined as the process of systematically determining a sequence of actions or steps required to achieve a desired goal from an initial state, considering constraints and available resources. 
This definition primarily differentiates from studies that utilize LLMs to generate ungrounded plans as guidance for problem-solving <cit.>, rather than directly producing executable actions.
Planning can be categorized into two types: open-loop planning, where the LLM outputs an entire action sequence before execution <cit.>, and closed-loop planning, where the LLM-based agent decides the next action based on real-time environmental interaction after executing a previous action <cit.>. 
This paper mainly focuses on close-loop planning, which is more adaptable for error correction, human interaction, and environmental grounding. 
Recent studies on close-loop planning have integrated chain-of-thought reasoning into the planning process <cit.>. 
Additionally, some papers have explored the use of tree-search methods to enhance the performance of LLM planning <cit.>.
Instead of designing novel frameworks or engaging in prompt engineering, this paper explores how training can enhance the planning capabilities of LLM-based agents.




  
Agent Training.

Recently, numerous studies have aimed to enhance LLM-based agent capabilities by incorporating agent trajectory data into their training <cit.>. 
Advanced works such as AgentTuning <cit.> utilize GPT-4 to generate trajectory data across six distinct environments. Subsequently, this data is filtered and employed in training Large Language Models, enhancing the agent capabilities of base models.
Another work, FireAct <cit.>, proposes training with both CoT data and ReAct format data, enabling the model to discern when to use reasoning to solve problems and when to call external tools. 
Agent LUMOS <cit.> suggests separately training Planning and Grounding models, enabling LLM-based agents to learn to decompose complex problems before execution.
LLM-Modulo framwork <cit.> proposes to leverage LLMs generating candidate plans and verify them with an external verifier. Then, use the verified trajectories for fine-tuning LLMs.
Similarly,  <cit.> takes a generate-test loop to synthesize trajectories for LLM training.
Unlike previous papers on all agent training, goes beyond merely generating trajectory data using Large Language Models. Instead, we utilize Large Language Models to generate agent environments, which can be considered a more foundational application. As a result, we have constructed over 500 environments for training, whereas previous works typically use fewer than 10 environments to synthesize agent data.




  
Environment and Task Generation with Large Language Models.


The utilization of LLMs to generate environments and tasks is an emerging application.
Some studies have explored utilizing LLMs to generate layouts in robotic simulations, typically involving the creation of configuration files <cit.>. 
While these methods can construct numerous scene-level environments, they often struggle to achieve diversity at the underlying mechanism level. 
Agenttuning <cit.> employs a task generation approach similar to the Self-instruct <cit.> method, using the test set as seed data. This approach not only poses a risk of data leakage but also leads to insufficient diversity in task difficulty.
ByteSized32 <cit.> uses LLMs to generate Python-based games based on predefined task specifications automatically. 
Similarly, other works <cit.> leverage LLMs to automatically construct PDDL domains based on a task specification.
In contrast to these studies, this paper proposes using a diverse text corpus to generate environment code automatically. 
This approach facilitates the creation of a wide range of rich environments without predefined definitions.



§ CONCLUSION




In this paper, we explore using LLMs to automatically generate environment and planning tasks for LLM-based agent training. 
Specifically, for generating diverse environments, we propose utilizing an inspiration corpus composed of various domain-specific text segments as the context for environment synthesis. To enhance the difficulty diversity of generated planning tasks, we introduce a bidirectional evolution method, , which evolves planning tasks from both easier and more challenging directions to create a task set with a more gradual difficulty curve, thereby improving the effectiveness of LLM learning. Based on , we developed a dataset consisting of 592 environments and 7246 trajectories and trained it on a series of LLMs. The trained model outperformed GPT-3.5 across multiple tasks and, in certain specific instances, exceeded the performance of GPT-4.

plainnat








colframe = blue!50!black, 
colback = white, 
toptitle=1mm, 
bottomtitle=1mm, 
fonttitle=10pt12pt,  breakable, enhanced jigsaw, finish broken=colframe=white, bottom=0mm, bottomrule=0mm,before upper=colframe=white, toprule=0mm,after=colframe=blue!40!black, toprule=1mm, bottomrule=1mm,



§ MORE IMPLEMENTATION DETAILS




 §.§ Models



We applied the instruct version for models. Specifically, the detailed version for each model is presented in Table <ref>.





 §.§ Natural Language Mapping



We leverage GPT-4 to generate the natural language mapping that converts structured actions into its natural language format. When the mapping failed to yield, we heuristically serialized the structured actions.
The prompt for generating natural language mapping with GPT-4 is as follows:

[colback=gray!10!white,colframe=gray!50!white,title=Natural Language Mapping Generation]
I would like you to create natural language mapping for PDDL.   

The form of the natural language mapping is a Python dictionary, wherein

1. The key corresponds to the name of a predicate or action within the domain PDDL.

2. The value is its equivalent in natural language, with parameters presented in "{argn}", where n is the index of its parameters in the PDDL expression.

3. You must ensure that the number of "{}" corresponds precisely to the number of parameters in predicates or actions.  

4. You should very carefully check the order of {argn}. 



Your output must strictly follow the provided example.



Example:

PDDL Domain:

“`pddl

(define (domain hanoi)

  (:requirements :strips)

  (:predicates

  (clear ?x)

  (on ?x ?y)

  (smaller ?x ?y)

  )



  (:action move

    :parameters (?disc ?from ?to)

    :precondition (and (smaller ?to ?disc) (on ?disc ?from)

               (clear ?disc) (clear ?to))

    :effect  (and (clear ?from) (on ?disc ?to) (not (on ?disc ?from))

          (not (clear ?to))))

  )

“`

Specification:

Your goal is to solve the Tower of Hanoi puzzle, which involves moving a stack of discs from one peg to another, with the restriction that no disc may be placed on top of a smaller disc. The puzzle is solved when all the discs are moved to the target peg following these rules.



The actions defined in this domain include:

- move <disc> <from> <to>: This action allows moving a disc from one peg to another. The preconditions for this action are that the target peg is smaller than the disc being moved, the disc is on the source peg, and both the disc and the target peg are clear (i.e., there is no disc on top of them). The effect of this action is that the source peg becomes clear, the disc is now on the target peg, the disc is no longer on the source peg, and the target peg is no longer clear.



You have the following restrictions on your actions:

- A disc can only be moved if it is clear, meaning there is no other disc on top of it.

- A disc can only be placed on another disc or peg that is larger than itself.

- A disc can only be moved to a peg that is clear.

- Once a disc is moved from a peg, that peg becomes clear.

- Once a disc is placed on a peg, that peg is no longer clear.

 Natural Language Mapping:

“`python

{

    "clear": "{arg1} is clear.",

    "on": "{arg1} is on {arg2}.",

    "smaller": "{arg1} is smaller than {arg2}.",

    "move": "Move {arg1} from {arg2} to {arg3}."

}

“`



You need to generate the corresponding natural language mapping for the following pddl domain.

    

PDDL Domain:

{PDDL_Domain}

Specification:

{PDDL_Description}

Natural Language Mapping:






§ MORE STATISTICS ON ENVIRONMENT





 §.§ Environment Specification




We analyzed the token distribution within the environmental specifications. Among the 592 environmental specifications, the average token count is 473.55, with a median of 467.00. The minimum token count is 207, and the maximum is 934. As depicted in Figure <ref>, the number of specification tokens for the environment is predominantly concentrated within the range of 300 to 699.



 §.§ Environment Implementation




























The scale of action space and state space in an environment typically dictates its complexity, with a greater number of actions and states generally indicating a more complex environment. 
An environment library with a greater variety of difficulty levels is preferable for a training set. 
As shown in Figure <ref>, there is a significant diversity in the number of actions and predicates.






 §.§ Diversity Analysis



We evaluate the diversity of generated environments using cosine similarity. More specifically, we randomly sampled 100 environment specifications for better visualization and converted them into TF-IDF vectors. After calculating the cosine similarity matrix between all pairs of specifications, we visualize the matrix using heatmap as is shown in Figure <ref>.
The computed average cosine similarity of the sampled environment specifications is 0.176, indicating that the corpus exhibits a high degree of diversity, reflecting a rich tapestry of distinct semantic features and thematic elements.



§ EXAMPLES



In this section, we present the specific details of the cases depicted in Figure <ref> and Figure <ref>.



 §.§ Environment Specification


[colback=gray!10!white,colframe=gray!50!white,title=Environment Specification]
You are a nutritionist tasked with creating a new healthy recipe book that incorporates peanut butter powder as a key ingredient. Your environment consists of your office, a computer for research and writing, a kitchen for testing recipes, and a collection of ingredients including peanut butter powder. The computer can be used to write and save recipes, but it needs to be charged after every three hours of use.



The actions defined in this domain include:

- research_ingredient <location> <ingredient> <nutritionist>: This action allows the nutritionist to research peanut butter powder at the office. The effect of this action is that the nutritionist gains knowledge about the nutritional benefits and potential uses of peanut butter powder in recipes.

- develop_recipe <location> <computer> <nutritionist> <ingredient>: This action allows the nutritionist to create a new recipe using peanut butter powder. The precondition for this action is that the nutritionist has researched peanut butter powder. The effect of this action is a draft of a new recipe that incorporates peanut butter powder.

- test_recipe <location> <kitchen> <nutritionist> <recipe>: This action allows the nutritionist to test the drafted recipe in the kitchen. The preconditions for this action are that the nutritionist has developed a recipe and has all the necessary ingredients, including peanut butter powder. The effect of this action is feedback on the recipe's taste, nutritional value, and overall feasibility.

- finalize_recipe <location> <computer> <nutritionist> <tested_recipe>: This action allows the nutritionist to finalize the recipe after testing and receiving feedback. The preconditions for this action are that the nutritionist has tested the recipe and the computer is charged. The effect of this action is a completed recipe that is ready to be included in the healthy recipe book.



You have the following restrictions on your actions:

- You can only research peanut butter powder if you are in the office where your research materials are located.

- You can only develop a recipe after researching peanut butter powder and must have a charged computer.

- You can only test a recipe in the kitchen if you have developed a recipe and have all the necessary ingredients.

- You can only finalize a recipe after testing it in the kitchen and receiving feedback, and if your computer is charged.





 §.§ Environment Implementation


[colback=gray!10!white,colframe=gray!50!white,title=Environment Implementation, sharp corners=south]
[t].95
[commandchars=
{}]
(define (domain healthy-recipe-book)
  (:requirements :strips)
  (:predicates
    (in-office ?nutritionist)
    (in-kitchen ?nutritionist)
    (researched-peanut-butter ?nutritionist)
    (has-recipe-draft ?nutritionist ?recipe)
    (has-tested-recipe ?nutritionist ?recipe)
    (computer-charged)
    (has-ingredients ?recipe)
  )

  (:action research_ingredient
    :parameters (?nutritionist ?ingredient)
    :precondition (and 
                    (in-office ?nutritionist))
    :effect (and 
              (researched-peanut-butter ?nutritionist))
  )

  (:action develop_recipe
    :parameters (?nutritionist ?ingredient)
    :precondition (and 
                    (researched-peanut-butter ?nutritionist) 
                    (computer-charged))
    :effect (and 
              (has-recipe-draft ?nutritionist ?ingredient) 
              (not (computer-charged)))
  )

  (:action test_recipe
    :parameters (?nutritionist ?recipe)
    :precondition (and 
                    (in-kitchen ?nutritionist) 
                    (has-recipe-draft ?nutritionist ?recipe) 
                    (has-ingredients ?recipe))
    :effect (and 
              (has-tested-recipe ?nutritionist ?recipe))
  )

  (:action finalize_recipe
    :parameters (?nutritionist ?tested_recipe)
    :precondition (and 
                    (has-tested-recipe ?nutritionist ?tested_recipe) 
                    (computer-charged))
    :effect (and 
              (not (computer-charged)) 
              (not (has-tested-recipe ?nutritionist ?tested_recipe)))
  )






 §.§ Trajectory Data


[colback=gray!10!white,colframe=gray!50!white,title=Trajectory Data]
(User) You are a nutritionist tasked with creating a new healthy recipe book that incorporates peanut butter powder as a key ingredient. Your environment consists of your office, a computer for research and writing, a kitchen for testing recipes, and a collection of ingredients including peanut butter powder. The computer can be used to write and save recipes, but it needs to be charged after every three hours of use.



The actions defined in this domain include:

- research_ingredient <location> <ingredient> <nutritionist>: This action allows the nutritionist to research peanut butter powder at the office. The effect of this action is that the nutritionist gains knowledge about the nutritional benefits and potential uses of peanut butter powder in recipes.

- develop_recipe <location> <computer> <nutritionist> <ingredient>: This action allows the nutritionist to create a new recipe using peanut butter powder. The precondition for this action is that the nutritionist has researched peanut butter powder. The effect of this action is a draft of a new recipe that incorporates peanut butter powder.

- test_recipe <location> <kitchen> <nutritionist> <recipe>: This action allows the nutritionist to test the drafted recipe in the kitchen. The preconditions for this action are that the nutritionist has developed a recipe and has all the necessary ingredients, including peanut butter powder. The effect of this action is feedback on the recipe's taste, nutritional value, and overall feasibility.

- finalize_recipe <location> <computer> <nutritionist> <tested_recipe>: This action allows the nutritionist to finalize the recipe after testing and receiving feedback. The preconditions for this action are that the nutritionist has tested the recipe and the computer is charged. The effect of this action is a completed recipe that is ready to be included in the healthy recipe book.



You have the following restrictions on your actions:

- You can only research peanut butter powder if you are in the office where your research materials are located.

- You can only develop a recipe after researching peanut butter powder and must have a charged computer.

- You can only test a recipe in the kitchen if you have developed a recipe and have all the necessary ingredients.

- You can only finalize a recipe after testing it in the kitchen and receiving feedback, and if your computer is charged.

Goal: The goal is to satisfy the following conditions: The computer is charged. jordan has tested the recipe almond_butter_bars.

Observation:The computer is charged. The recipe almond_butter_bars has all the necessary ingredients. jordan has a recipe draft for almond_butter_bars. jordan has researched peanut butter. jordan is in the kitchen.

(Assistant) Action: jordan tests the recipe almond_butter_bars.

(User) Observation: The computer is charged. The recipe almond_butter_bars has all the necessary ingredients. jordan has a recipe draft for almond_butter_bars. jordan has researched peanut butter. jordan has tested the recipe almond_butter_bars. jordan is in the kitchen.

(Assistant) Action: jordan develops a recipe using almond_butter_bars.










 Paper Title: Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillation.



Extracting doubly-excited state lifetimes in helium directly in the time domain with attosecond noncollinear four-wave-mixing spectroscopy
    Stephen R. Leone
    August 5, 2024
==========================================================================================================================================




§ ABSTRACT 
This paper introduces a novel approach called sentence-wise speech summarization (Sen-SSum), which generates text summaries from a spoken document in a sentence-by-sentence manner. Sen-SSum combines the real-time processing of automatic speech recognition (ASR) with the conciseness of speech summarization. To explore this approach, we present two datasets for Sen-SSum: Mega-SSum and CSJ-SSum. Using these datasets, our study evaluates two types of Transformer-based models: 1) cascade models that combine ASR and strong text summarization models, and 2) end-to-end (E2E) models that directly convert speech into a text summary. While E2E models are appealing to develop compute-efficient models, they perform worse than cascade models. Therefore, we propose knowledge distillation for E2E models using pseudo-summaries generated by the cascade models. Our experiments show that this proposed knowledge distillation effectively improves the performance of the E2E model on both datasets.




§ INTRODUCTION

Automatic speech recognition (ASR) has undergone significant advancements in the past decades <cit.>,
 primarily aiming to produce word-for-word transcriptions,
 but such transcriptions can be difficult to read for humans 
 due to spoken-style and redundant expressions. 
On the other hand, speech summarization (SSum) condenses a spoken document 
 into a concise and written-style summary 
 offering informative and easily digestible summaries, 
 and has gained increasing interest for processing speech from various domains, such as meetings <cit.> and lectures <cit.>. 
However, unlike ASR, SSum is unsuitable for real-time applications 
 because it typically processes an entire spoken document at once,
 indicating the lack of technology to produce real-time and concise summaries of spoken content.

To address this issue, we propose sentence-wise speech summarization (Sen-SSum) to bridge the gap between ASR and SSum.
Figure <ref> compares ASR, SSum, and Sen-SSum with examples.
Sen-SSum goes beyond related technologies such as disfluency detection and removal <cit.> 
 and provides more concise and clearer outputs.
Furthermore, users can access summaries immediately,
 without waiting until the end of an entire meeting or lecture, 
 because Sen-SSum incrementally produces summaries after each speech sentence.
Therefore, as a substitute for personal notes, 
 Sen-SSum could help users review a meeting flow later 
 or catch up on the discussion when joining a lecture in the middle.
Despite its many promising applications, 
 the Sen-SSum task has not been well explored, partially due to the lack of publicly available datasets for Sen-SSum.



In this paper, we introduce a novel Sen-SSum dataset: Mega-SSum.
The Mega-SSum dataset is based on the Gigaword dataset <cit.> 
 and contains 3.8M English triplets of synthesized speech, transcriptions, and summaries.[https://huggingface.co/datasets/komats/Mega-SSumhttps://huggingface.co/datasets/komats/Mega-SSum]
We utilize a state-of-the-art multi-speaker text-to-speech model <cit.> to synthesize high-quality natural speech.
This dataset enables us to explore the impact of training data 
 availability on a large scale.
To increase the validity of our experiments with a more practical dataset, 
 we also use our internal Japanese Sen-SSum corpus, CSJ-SSum,
 which is based on the Corpus of Spontaneous Japanese <cit.>
 and contains 38k triplets with real speech.

We investigate two approaches for Sen-SSum: 1) cascade and 2) end-to-end (E2E) models.
The cascade model combines ASR and text summarization (TSum) models <cit.>
 and can output high-quality summaries thanks to the TSum model pre-trained on large-scale text data.
The E2E model directly generates text summaries from input speech with a single encoder-decoder model <cit.>.
This approach is promising in terms of parameter efficiency and potentially fast decoding,
 but it requires many expensive speech-summary pairs for training,
 and the lack of such large training dataset hampers the performance of E2E models.

To tackle this issue, 
 we propose knowledge distillation for the E2E models. 
We assume a practical scenario where a small subset of the training set (“core set”) 
 contains both summary and transcription labels,
 while the remaining samples lack either or both labels.
We increase the training data by creating pseudo-summaries from unlabeled speech
 using a cascade model trained on the core set. 
Subsequently, we train an E2E model using the core set and the pseudo-summaries.
We expect that the rich linguistic knowledge of the strong language model (LM), i.e., the TSum model, will be distilled into the E2E model via the pseudo-summaries.



The experimental evaluation showed that it significantly 
 improved the performance of E2E models on both datasets. 
Additionally, we found that the pseudo-summaries led to better summarization accuracies than the manual summaries in certain conditions.






§ RELATED WORK

Text sentence summarization has been widely studied in the natural language processing (NLP) field <cit.>, but little research has dealt with the audio modality.
<cit.> is most closely related to our work, 
 where they experimented with Sen-SSum using a synthesized Gigaword dataset.
However, they investigated only cascade modeling and did not publish Sen-SSum datasets.
Furthermore, their dataset is ten times smaller than ours
 and has high word error rates (WERs), indicating poor speech quality.

The Sen-SSum task is also related to disfluency detection and removal <cit.> 
 since it excludes less important words and enhances the comprehensibility of ASR transcriptions. 
However, the primary goal of disfluency removal is not to make the document shorter. 
In fact, after eliminating disfluencies, the compression rates were 86% for the Switchboard dataset <cit.> 
 and 92% for the CSJ dataset, according to <cit.>.
Other related studies, such as spoken-to-written style conversion <cit.>, subtitling <cit.>, and other comprehensive post-processing <cit.>, have similar compression rates, though they further involve paraphrasing to generate more user-friendly transcriptions.
On the other hand, our Sen-SSum datasets introduced in Section <ref> have lower compression rates of only 20% (Mega-SSum) and 40% (CSJ-SSum).
Thus, Sen-SSum is a significantly different task and more suitable for quickly grasping spoken content.



§ DATASETS FOR SEN-SSUM


The availability of high-quality datasets is crucial for advancing
 research and development on the Sen-SSum task.
We introduce a novel dataset, Mega-SSum, to investigate Sen-SSum. 
We also validate our experimental results on Mega-SSum using an in-house Japanese dataset, CSJ-SSum.
Table <ref> presents overviews and statistics of these datasets.

Mega-SSum is a large-scale English dataset for Sen-SSum, containing  3.8M synthesized spoken sentences and corresponding transcriptions and summaries.
It is based on the Gigaword dataset <cit.>,
 composed of the first sentences of news articles and their headlines,
 and is widely used for text sentence summarization studies <cit.>.
The DUC2003 dataset <cit.>, which is in a similar domain to Gigaword but with four manual summaries, rather than headlines, for each input sentence, is used for the evaluation set.
The summaries are fairly condensed, with a compression rate (CR)[The compression rate is defined as (#words in summary) / (#words in input text), 
 following <cit.>. Shorter summaries have lower rates.] of about 20%.
To synthesize high-quality and natural-sounding speech from the text sentences,
 a multi-speaker text-to-speech model VITS <cit.> was trained from scratch using the LibriTTS-R dataset <cit.>,
 which is a sound quality improved version of LibriTTS <cit.>.
The synthesized speech retains linguistic information, enabling us reasonable investigations for Sen-SSum.
For example, 
 when decoded with the Whisper (small.en) ASR model <cit.> without fine-tuning,
 the WER on the evaluation set is only 7.8%, which is as low as typical natural speech.
To simulate a realistic setting,
 the training set is split into a core set with the first 50k samples 
 and a remaining set with 3.75M samples.
The core set is used to investigate low-resource and practical situations.

We validate the results obtained on the Mega-SSum dataset using an 
 in-house Sen-SSum dataset named CSJ-SSum,
 which contains 38k real speech sentences, transcriptions, and summaries in Japanese.
It is based on the SPS subset of the CSJ <cit.>,
 consisting of spontaneous monologues and corresponding transcriptions 
 on daily general topics such as commentary on recent news.
In addition, it has an out-of-domain evaluation set based on 10 Japanese TED talks (“eval-TED”). 
Professional annotators are employed to provide summaries following two instructions:
 1) Segment the speech and its transcription into sentences by manually inserting periods.
 2) Give a concise and written-style summary for each transcribed sentence, 
 retaining the most essential information.
This dataset enables us to confirm the possibility of Sen-SSum on real data 
 and for a different language. 
Incidentally, since this dataset is much smaller, 
 it justifies the practical need for our proposed knowledge distillation scheme in Section <ref>.



§ METHOD



 §.§ Cascade and end-to-end modeling

We implement Sen-SSum with two distinct approaches: cascade <cit.> and E2E <cit.> speech-to-text summarization.

The cascade model combines the ASR and TSum models. 
The ASR model ASR(·) first transcribes the input speech 𝐱 
 into its corresponding transcription,
and then the TSum model TSum(·) predicts the target summary:

    ŝ = TSum(ASR(𝐱))

 where ŝ represents the summary hypothesis. 
We implement the ASR and TSum models using the Transformer-based encoder-decoder architecture <cit.>, trained with cross-entropy loss. 
The main advantage of cascade modeling lies in enabling the TSum model to acquire robust NLP capabilities by initializing it with a LM trained on a vast unlabeled text corpus.
Cascade modeling is a natural choice for Sen-SSum
 given that conventional SSum typically employs it <cit.>.

In E2E modeling, an encoder-decoder model E2E(·) directly predicts
 the target summary from the input speech 𝐱:

    ŝ = E2E(𝐱)

Note that we build the E2E model by fine-tuning the ASR model, following <cit.>.
E2E modeling is attractive with its compact and parameter-efficient structure,
 along with the potential for lower decoding latency.
Additionally, it mitigates the propagation of ASR errors
 and leverages the acoustic information in the input speech to predict summaries.
Despite its advantages, E2E modeling necessitates a substantial number of speech-summary pairs, which are expensive to collect and often limited, resulting in its poor summarization capabilities.



 §.§ Sequence-level knowledge distillation for E2E models


To mitigate the training data scarcity for E2E models, 
 we propose leveraging a cascade model to generate additional training data.

Specifically, 
 the cascade model, trained on a small available dataset (i.e., the core set), generates pseudo-summaries 
 from unlabeled speech sentences 
 by transcribing them with the ASR model 
 and then summarizing the transcriptions with the TSum model. 
These pseudo-summaries, along with the human summaries in the core set, are used to train an E2E model. 
We anticipate that the rich linguistic knowledge embedded in the TSum model will be distilled to 
 the E2E model through the pseudo-summaries. 
Although this method is motivated by the success in the E2E speech 
 translation field <cit.> and widely known as sequence-level knowledge distillation <cit.>, this is the first application to E2E SSum.











 §.§ Leveraging self-supervised models

We also investigte the impact of integrating WavLM large[https://huggingface.co/microsoft/wavlm-large] <cit.>, a state-of-the-art 
 speech self-supervised learning (SSL) model 
 renowned for its effectiveness in low-resource spoken language understanding tasks, 
 on enhancing our E2E model with the Mega-SSum dataset.
Specifically, 
 we first use WavLM as a feature extractor,
 and then fine-tune it alongside the downstream E2E model.
Speech SSL models hold promise as potential alternatives to our proposed method, given their inherent linguistic knowledge from pre-training on vast speech-only data <cit.>.



§ EXPERIMENT



 §.§ Setup



  §.§.§ Model architectures

We implemented three encoder-decoder models for ASR, TSum, and E2E Sen-SSum.
Both the ASR and E2E models comprised
 a 2-layer convolutional neural network with a sub-sampling rate of 4 for initial feature extraction,
 a 12-layer Conformer encoder with a model dimension of 512 and a kernel size of 31, 
 and a 12-layer Transformer decoder.
Both the encoder and decoder featured 2048-dimensional feed-forward (FF) layers.
Following <cit.>, 
 batch normalization layers in the Conformer blocks were replaced 
 with layer normalization layers,
 and learnable positional embedding was used for the decoder.
For the TSum model, 
 we used T5 models <cit.>, consisting of a 12-layer Transformer encoder and decoder
 with a model dimension of 768 and 3072-dimensional FF layers.
The number of parameters in the cascade and E2E models were 362M (=142M+220M) and 139M, respectively.



  §.§.§ Training details: datasets and hyper-parameters

For the experiments on Mega-SSum,
 we first trained an ASR model on the 960 hours of the Librispeech dataset <cit.> 
 and fine-tuned it using the 50k speech-transcription pairs in the core set.
We further fine-tuned the ASR model with the 50k speech-summary pairs to obtain an E2E model.
We denoted this baseline model as “”.
We prepared a TSum model by fine-tuning an English T5 model[https://huggingface.co/google-t5/t5-base] with 50k transcription-summary pairs. 
The combination of the ASR and TSum models were denoted as “”.
To investigate the impact of training data volume, 
 we also prepared ASR, E2E, and TSum models
 trained with the first 100k, 500k, 1M, or 3.8M samples.
We denoted them as “” and “” because they were trained with human summaries, in contrast to ones trained with pseudo-summaries in Section <ref>.
The WERs of the ASR models were 11.7%, 10.7%, 9.2%, 9.4%, and 7.7% for the evaluation set, respectively.

For the proposed knowledge distillation,
 we assumed that the core set with 50k samples  
 was fully available, and only the speech data of the first 50k, 450k, 950k, or 3.75M samples in the remaining set was additionally available.
The transcriptions and summaries were prepared as explained in Section <ref>.
The total number of training samples was 100k, 500k, 1M, or 3.8M since the 50k samples in the core set were added.
We denote these E2E models with the proposed knowledge distillation as  “”.
To assess the effect of ASR errors on the proposed method,
 we also investigate the E2E models trained with pseudo-summaries generated from the reference transcriptions, 
 denoted as “”.

For CSJ-SSum,
 we used the entire CSJ datasets <cit.> and 1M speech-transcription pairs from our in-house ASR datasets to train a ASR model.
The character error rates on the eval-CSJ/TED were 3.1% and 16.7%, respectively.
We then fine-tuned it with all 38k speech-summary pairs to obtain the baseline E2E model.
A Japanese T5 
 model[https://huggingface.co/sonoisa/t5-base-japanese] was fine-tuned with the 38k transcription-summary pairs to obtain the baseline TSum model.
For knowledge distillation, we leveraged the same 1M speech-transcription pairs used to train the ASR model.
We generated 1M pseudo-summaries from the reference transcriptions using the baseline TSum model,
 then used these to train the proposed E2E model alongside the 38k human summaries.
Note that the 1M pairs were selected from our in-house ASR dataset 
 to ensure that each reference transcription was longer than 10 characters and contained sentence-ending expressions, enabling the TSum model to produce reasonable summaries.

We adopted the same hyper-parameters for both Mega-SSum and CSJ-SSum unless specified otherwise.
To obtain an ASR model,
 we utilized the WarmupLR scheduler and the Adam optimizer, setting the maximum learning rate (LR) to 2x10^-3 
 and the number of warmup steps to 40k.
The average batch size was set to 168 for LibriSpeech and 350 for CSJ.
We also applied the connectionist temporal classification (CTC) auxiliary loss with a weight of 0.3, 
 SpecAugment <cit.>, and speed perturbation. 
The vocabulary size was set to 5,000 using byte-pair encoding <cit.> for Mega-SSum 
 and 3,262 with the character unit for CSJ-SSum.
For the fine-tuning, we adjusted the LR to 2x10^-4 and the number of warmup steps to 1k, reduced the batch size by one-fifth, and used the AdamW optimizer. 
CTC loss was omitted during E2E Sen-SSum model training.
To obtain a TSum model, we utilized a linearly decaying LR starting from 5x10^-5.
We used early stopping based on validation loss with a patience of 5 epochs. 




  §.§.§ Evaluation details

In the evaluation, we consistently set the beam width to 4 for the ASR, E2E, and TSum models.
For objective metrics, we used ROUGE-L (R-L) <cit.> and BERTScore (BScr) <cit.>, 
 which have been commonly used in previous TSum and SSum studies.
The R-L score measures superficial word matching, 
 while BScr leverages BERT embeddings to capture the semantic meanings of words.
Additionally, inspired by <cit.>, we conducted A/B tests using the ChatGPT API[https://platform.openai.com/docs/api-reference] for a more comprehensive assessment.
Specifically, GPT4-turbo (gpt-4-1106-preview) was instructed to select the better summary of the two, considering the reference transcription.
 



 §.§ Results



  §.§.§ Results on Mega-SSum


The first three columns in Table <ref> show 
 the 95% confidential intervals of R-L and BScr scores and CRs  
 by the cascade and E2E models trained on the core set.
As expected, the cascade model outperformed the E2E model
 due to the pre-trained TSum model's strong NLP capability.
In fact, the baseline E2E model occasionally generated only one or two words as a summary,
 likely due to its limited generalization capability, 
 resulted in lower CRs compared with the cascade model.
Although WavLM was also pre-trained on a large amount of unlabeled speech, 
 its combination with the E2E model did not improve the scores.
Its linguistic knowledge seemed insufficient to help the E2E model solve the summarization task.
The final column “_3.8M” shows the performance of the E2E model trained with the 50k human summaries from the core set and an additional 3.75M pseudo-summaries. 
This proposed method significantly enhanced the E2E model to achieve a performance level comparable to that of the cascade model.



Figure <ref> illustrates the R-L and BScr scores
 when the cascade and E2E models were trained on varying amounts of training data,
 including the scores of the E2E models trained with pseudo-summaries.
As evident from the results of  and ,
 the cascade and E2E models generated better summaries with more training samples.
Additionally, the pseudo-summaries effectively improved the E2E model 
 with its scores gradually approaching those of , which generated the pseudo-summaries.

Interestingly, training with the pseudo-summaries resulted in better scores
 for the E2E model up to 500k samples.
This was likely because the pseudo-summaries were easier for the E2E model to learn compared with the human summaries.
For instance, the R-L score, which considers the longest common sub-sequence of two sequences, 
 between the pseudo-summaries and transcriptions (34.6±0.94, on the evaluation set)
 was significantly higher than that between the human summaries and transcriptions (27.2±1.03).
This suggests that the pseudo-summaries made Mega-SSum a more extractive summarization task, which is more similar to the ASR task and easier to reproduce.













However, with 1M or more human summaries,  showed better scores than  because it could reproduce more human-like summaries,
 which cannot be completely learned with erroneous and simplified pseudo-summaries.



Figure <ref> shows the preference percentages of ChatGPT-conducted A/B tests
 comparing summaries generated by the baseline cascade model with those produced by the E2E models trained with varying amounts of pseudo-summaries.
As the dataset expanded, preferences gradually shifted towards the E2E model's summaries over those of the cascade model. 
However, even with 3.8M training samples,
 the E2E model's summaries were frequently deemed inferior to those of the cascade model.
This performance gap was not evident through conventional metrics such as R-L or BScr, indicating that more effective knowledge distillation methods may be required.

Figure <ref> shows the effect of knowledge distillation
 when the reference transcriptions of the remaining set were also available in addition to the speech.
Note that the baseline cascade and E2E models, i.e.,  and , were also improved 
 because the ASR model was trained on larger datasets.
While the speech-only data substantially improved,
 higher-quality pseudo-summaries derived from the reference transcriptions was important and resulted in better scores.





  §.§.§ Results on CSJ-SSum

Table <ref> shows the R-L and BScr scores on the CSJ-SSum dataset.
The scores were significantly higher than those for Mega-SSum
 because of more extractive nature and simpler structure of the summaries in the CSJ-SSum dataset.
Nevertheless, the proposed method significantly improved the scores on both evaluation sets,
 indicating its effectiveness in real-world scenarios.
We gained more improvements on the out-of-domain eval-TED set compared with the in-domain eval-CSJ set. 
This discrepancy may be attributed to the inherent generalizability of the proposed method, 
 which was better suited to adapt to diverse vocabularies in out-of-domain datasets. 






§ CONCLUSION

In this study, we introduced Sen-SSum along with two supporting datasets, Mega-SSum and CSJ-SSum. 
We demonstrated the potential of both cascade and E2E models 
 in Sen-SSum and the effectiveness of knowledge distillation 
 for E2E models using the cascade model. 
Future work could explore more efficient methods like <cit.>,
 which directly integrates LMs into E2E SSum models and does not rely on pseudo-summaries.
It is also important to develop 
 context-aware models <cit.>
 to consistently handle long speech documents in a sentence-by-sentence manner, which represents a more practical setting.



IEEEtran




 Paper Title: Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation.








xiaoye@hust.edu.cn

  Huazhong University of Science and Technology
  China



chenqiyuan1012@gmail.com

  Zhejiang University
  China



weiw@hust.edu.cn

  Huazhong University of Science and Technology
  China



gasolsun36@gmail.com

  Xiamen University
  China





dongjf24@gmail.com

  Zhejiang Gongshang University
  China







§ ABSTRACT 
Despite the remarkable ability of large vision-language models (LVLMs) in image comprehension, these models frequently generate plausible yet factually incorrect responses, a phenomenon known as hallucination. 
Recently, in large language models (LLMs), augmenting LLMs by retrieving information from external knowledge resources has been proven as a promising solution to mitigate hallucinations. 
However, the retrieval augmentation in LVLM significantly lags behind the widespread applications of LVLM. 
Moreover, when transferred to augmenting LVLMs, sometimes the hallucination degree of the model is even exacerbated. 
Motivated by the research gap and counter-intuitive phenomenon, 
we introduce a novel framework, the Active Retrieval-Augmented large vision-language model (ARA), specifically designed to address hallucinations by incorporating three critical dimensions: 
(i) dissecting the retrieval targets based on the inherent hierarchical structures of images. (ii) pinpointing the most effective retrieval methods and filtering out the reliable retrieval results. (iii) timing the retrieval process to coincide with episodes of low certainty, while circumventing unnecessary retrieval during periods of high certainty. To assess the capability of our proposed ARA model in reducing hallucination, we employ three widely used LVLM models (LLaVA-1.5, Qwen-VL, and mPLUG-Owl2) across four benchmarks. 
Our empirical observations suggest that by utilizing fitting retrieval mechanisms and timing the retrieval judiciously, we can effectively mitigate the hallucination problem. 
We hope that this study can provide deeper insights into how to adapt the retrieval augmentation to LVLMs for reducing hallucinations with more effective retrieval and minimal retrieval occurrences.



†The first two authors contribute equally.

*Corresponding author.

































Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation
    Jianfeng Dong
    August 5, 2024
============================================================================================





§ INTRODUCTION


Recently, Large Vision-Language Models (LVLMs)  <cit.> have gained widespread application across various scenarios due to their significant capacity to generate contextually appropriate text for visual inputs. 
Benefiting from the advancements in model architecture, training methods, and data diversity, LVLMs exhibit superior performance for various tasks such as visual question answering <cit.> and image captioning <cit.>.
Despite these advancements, LVLMs still suffer from a significant challenge termed “hallucination”, whereby the models produce semantically plausible but factually inaccurate text, misaligned with the ground-truth content of the associated image  <cit.>.
This problem damages the practical employment of LVLMs, particularly in fields that require accurate content generation, such as medical <cit.> and robotics <cit.>, potentially resulting in severe consequences. 
Hence, addressing the hallucination problem is paramount for bolstering the credibility of LVLMs across real-world scenarios.

To mitigate hallucinations in LVLMs, a series of attempts have been recently proposed and can be broadly classified into two categories. 
The first class of approach retrains the LVLMs with constructed hallucination-related datasets by
supervised fine-tuning (SFT)  <cit.>, or Reinforcement Learning from Human Feedback (RLHF) <cit.>.
Although effective, these methods introduce significantly additional training costs. 
The other solutions focus on designing more robust decoding strategies <cit.>. For example, VCD  <cit.> introduces visual contrastive decoding to contrast output distributions derived from original and distorted visual inputs. 
In this way, the LVLM reduces the over-reliance on statistical bias and unimodal prior.
Although these methods are training-free, they still suffer from the limitations of LVLMs’ static parametric capacity. 
Recently, in the realm of large language models (LMs), augmenting LLMs <cit.> by retrieving information from external knowledge resources has shown promise in reducing language hallucinations. Furthermore, this retrieval augmentation serves as a flexible way to extend beyond the model's inherent knowledge without the burden of extensive training costs.
In this paper, we aim to propose a novel framework to augment LVLMs with external knowledge by introducing an Active Retrieval-Augmented large vision-language model (ARA) for mitigating hallucinations. As shown in Figure <ref>, given an input image and query, the LVLM tends to produce hallucinated answers. Instead, our model can accurately identify the most relevant pairs and output the correct answer.



Concretely, our proposed ARA is grounded in three critical dimensions for mitigating hallucinations in LVLMs. 
Firstly, given the inherently hierarchical nature of images, simple full-image retrieval may result in noise and irrelevant outcomes, thus it is imperative to decompose the target object causing hallucination from the input image for more accurate retrieval. 
Secondly, in contrast to augmenting LLMs, LVLMs offer diverse retrieval modalities due to their multimodal inputs. 
In this way, discerning the most effective retrieval technique and securing trustworthy results is crucial for ensuring retrieving performance. 
Lastly, excessive retrieval activations may lead to undue time expenditure and unnecessary retrieval.
By initiating retrieval only during periods of low LVLM certainty and knowledge deficiency, one can circumvent unnecessary or improper information retrieval.


Based on the above analysis, given the input image and query, our proposed ARA first devises a coarse-to-fine retrieving paradigm. 
The target object is first extracted from the input query and subsequently positioned within a particular region of the input image. To build a robust retrieving system, we simultaneously engage in coarse-grained and fine-grained retrieval process which retrieves the full image as well as the specific regions pertaining to the target object. Recognizing the plurality of retrieval strategies due to the multimodal input, an extensive analysis of different methods is conducted to determine the most optimal retrieving approach. 
Subsequent to acquiring pertinent text and image pairs from external databases, a reranking strategy is employed to eliminate unreliable outcomes, thereby augmenting LVLMs with the appropriate external information.
Particularly, retrieval operations are circumvented when the LVLM exhibits high certainty, to prevent the inclusion of superfluous retrieval that may introduce redundant information. 
In this context, retrieval is activated using a difficulty metric that relies on the mutual information between multimodal inputs.

To assess the efficacy of ARA, empirical evaluations are performed using three prevalent LVLMs across four benchmarks related to hallucination challenges.
The promising results demonstrate that our framework can effectively mitigate hallucinations. 
To sum up, our contributions are summarized as follows:


    
  * Our research thoroughly investigates the three critical dimensions of augmenting LVLMs with external knowledge, including accurately dissecting the retrieval targets, pinpointing the most effective retrieval methods, and the sensible triggering of the retrieval process. 
    
    
  * We introduce an Active Retrieval-Augmented large vision-language model (ARA) for mitigating hallucination. 
    Our empirical evidence indicates that, by employing appropriate retrieval mechanisms and carefully timing the retrievals, we can significantly address the issue of hallucination while maintaining a moderate frequency of retrieval.

    
  * Through comprehensive experiments, we demonstrate the superior performance of our proposed ARA in alleviating hallucinations using three well-known LVLMs across four hallucination-related benchmarks. The results in our paper also suggest that with optimal retrieval settings, the potential of augmenting LVLMs can be more effectively harnessed.




§ RELATED WORK




 §.§ Large Visual-Language Models


Recently, large-scale models have attracted significant attention and gained widespread applications <cit.>. 
LVLMs significantly enhance the interaction between humans and AI in a more natural and intuitive
ways and demonstrate remarkable capabilities in understanding and generating multi-modal content.
With the aid of advanced Large Language Models like LLaMA <cit.> and Qwen <cit.>, a batch of LVLMs such as LLaVA-1.5 <cit.>, Qwen-VL <cit.>, and mPLUG-Owl2 <cit.> have emerged, which can comprehend and generate a wide array of content by utilizing information from distinct modalities like texts and images. 
The training of LVLMs is divided into two critical phases: feature alignment and instruction fine-tuning. The former aims to model visual and textual inputs coherently, while the latter strives to enhance the ability of LVLMs to comprehend user queries. 
Despite the advancements, these LVLMs continue to face significant challenges with hallucination issues. Thus, in this paper, we focus on solving hallucination problems to promote the use of LVLMs in practical scenarios.



 §.§ Hallucination in LVLMs


In LVLMs, hallucination refers to models that generate seemingly plausible outputs inclusive of objects,
attributes, or relations that do not correspond with the images <cit.>.
Approaches to address hallucinations in LVLMs have largely fallen into two camps. The first emphasizes model refinement through supervised fine-tuning or application of reinforcement learning techniques. For instance, LRV <cit.> seeks to diminish hallucinatory outputs by utilizing broad and varied supervised fine-tuning datasets. 
For methods based on reinforcement learning, LLaVA-RLHF <cit.> is the pioneer in applying Reinforcement Learning with Human Feedback (RLHF) to mitigate hallucination in LVLMs. 
This approach is further refined by RLHF-V <cit.>, which incorporates detailed corrective human feedback.
Considering the instability and training difficulty of RLHF, Zhao et al. <cit.> employ Direct Preference Optimization (DPO) and build a hallucination-aware dataset for alleviating hallucination. 
Although 
While these methods have yielded notable advancements, they also introduce significant training overheads and are prone to overfitting the training data. 
The second category consists of training-free strategies designed to circumvent hallucination without incurring additional training expenses.
VCD <cit.> endeavors to rectify the model's over-reliance on linguistic priors and statistical leanings by comparing the output distributions from both unaltered and visually perturbed inputs.
OPERA <cit.> mitigates overconfidence in model predictions by integrating a penalty term into the model logits during beam search decoding, further enhanced with a strategic rollback feature. 
Although these methods are effective, they still suffer from the limitations of LVLMs’ static parametric capacity. 
In this paper, different from the above two paradigms, we focus on augmenting LVLMs with external knowledge to mitigate hallucination in LVLMs.  



 §.§ Retrieval-Augmented Generation


In the realm of large language models (LLMs), Retrieval-Augmented Generation (RAG) has been widely used and shown promising results in mitigating hallucinations. RAG in LLMs retrieves relevant information from an external knowledge base before LLMs respond to a query <cit.>, thereby enabling them to collaboratively generate responses by leveraging the retrieved external non-parameterized knowledge alongside their internal parameterized knowledge. 
However, its application in LVLMs has not been thoroughly explored. Since LVLMs present a multimodal nature, findings pertinent to LLMs cannot be indiscriminately extrapolated to LVLMs. 
Predominantly, existing research on retrieval augmentation in multimodal tasks has been confined to image captioning <cit.> or image generation <cit.>, with a narrow focus that overlooks the broader implications of RAG in LVLMs, especially concerning its capacity for hallucination reduction.
In this paper, our research conducts an extensive evaluation of RAG's applicability in LVLMs to mitigate hallucination by tailoring retrieval strategies and optimizing active retrieval processes. 
It is worth noting that the active retrieving strategy has also been discussed in <cit.> in LLM. However, they propose a token-level confidence for triggering the retrieval, which is not applicable in LVLMs and we will investigate it in the ablation study.





§ METHOD


In this section, we first describe the generation procedure of the LVLMs, laying the groundwork for comprehending our approach. Subsequently, we will introduce our Active Retrieval-Augmentation large vision-language model (ARA) in detail. As shown in Figure <ref>, given the input image and query, the LVLM first analyzes them based on mutual information and determines whether to trigger the retrieval process. If the retrieval is necessary, our ARA will follow a coarse-to-fine paradigm to dissect the retrieval targets and conduct jointly coarse-grained and fine-grained retrieval. Following this, a reranking of the retrieval outcomes is performed for additional refinement. Finally, the LVLM harnesses this externally sourced knowledge to generate the final response.





 §.§ LVLM Input and Decoding

 
The input of LVLMs contains both image and text. Typically, the LVLMs use a vision encoder (e.g. CLIP) to extract visual tokens from the input images, and then map them into the input space of LLMs with a linear projection module. 
Subsequently, the mapped visual tokens are used as part of the LLM input, along with the text input. 
In this paper, we denote the visual tokens as 𝐱^v = {x_0,x_1,…,x_N-1}. Here N is the length of the visual tokens and it is a fixed number in most cases.  
Correspondingly, the input text is tokenized and we denote it as 𝐱^p = {x_0,x_1,…,x_M-1}.
The image and text tokens are concatenated as the final input sequence and we denote it as  {x_i}_t=0^T-1 that T=N+M.


During decoding, LVLM gets the probabilities for the next token prediction, formally:

    p(x_t|x_<t) = SoftMax[LVLM(h_t)]_x_t,    x_t∈𝒱,

where h_t denotes the output hidden states at time step t for the input image and query, x_<t to simplify the sequence {x_i}_i=0^t-1 and 𝒱 means the whole vocabulary set. For greedy decoding, the token with the highest probability will be chosen.




 §.§ Active Triggering Retrieval


As shown in Figure <ref>, given the input image and query, our ARA model first determines whether the input pair requires retrieval. In this way, we can circumvent unnecessary retrieval during periods of high certainty. 
In this section, we propose three active retrieval methods to explore the difficulty metrics that influence when to retrieve, including Confidence-aware Active Retrieval, 
Question-aware Active Retrieval, and Image-aware Active Retrieval. The first method relies on the confidence of the model output, and the latter two matrices depend on the mutual information between the inputs.
An ideal metric should effectively reduce the frequency of retrievals and can easily apply to different LVLMs.

Confidence-aware Active Retrieval. We feed the image and query into LVLMs to generate the answer. If the LVLM is confident about the answer, we accept it without retrieving additional information. Specifically, we actively trigger retrieval if any token of the answer has a confidence lower than the threshold θ. In this method, θ=0 indicates retrieval will never be used and θ=1 means that all inputs will use the retrieval. This strategy is inspired by <cit.> in LLM.

Query-aware Active Retrieval. To better evaluate the difficulty of queries from a view of language bias, we evaluate the degree of LVLMs answering relying on the input image. To this end, we define a difficulty metric using mutual information based on the difference below:


    M_ij = logP(a_ij | V_i, Q_i)/P(a_ij) - logP(a_ij | Q_i)/P(a_ij) = logP(a_ij | V_i, Q_i)/P(a_ij | Q_i).


where P(a_ij | Q_i) indicates the output probability with only query input to the LVLMs and P(a_ij | V_i, Q_i) means the output probability with both query and image as model input. The higher M_ij indicates more visual reasoning is needed to answer. M_ij<0 illustrates that the answer excessively depends on the input query. Thus, we define a threshold θ and trigger retrieval if M_ij<θ, where insufficient visual information is provided for answering the query. Incorporating retrieved information will enhance the confidence of LVLMs to answer.  

Image-aware Active Retrieval. Similar to the above query-aware active retrieval, we use the probability difference between the original and noisy image to evaluate the image information to answer the query. During the implementation, we use the method in <cit.> to add noise.


    D_ij = logP(a_ij | V_i, Q_i)/P(a_ij) - logP(a_ij | V'_i, Q_i)/P(a_ij) = logP(a_ij | V_i, Q_i)/P(a_ij | V'_i, Q_i).


where P(a_ij | V_i, Q_i) indicates the output probability with query and input image, P(a_ij | V'_i, Q_i) means the output probability with query and noisy image. Analogously, we define a boundary θ and trigger retrieval if D_ij<θ. 





 §.§ Coarse-to-Fine Hierarchical Retrieval


Given the inherently hierarchical nature of images, simple full-image retrieval may result in noise and irrelevant outcomes. As shown in Figure <ref>, using the full image for retrieval can not obtain the most relevant information for reasoning, namely “red shirt". Thus it is imperative to decompose the target object causing hallucination from the input image for more accurate retrieval. 
In the following, we will first describe the coarse-grained retrieval and then introduce the fine-grained retrieve. 

Coarse-grained Retrieve. 
Initially, based on the input image, we use the CLIP to extract the visual embedding. Similarly, we build the embeddings for the images and corresponding texts in the knowledge base. Then, we can retrieve the relevant images from knowledge bases with the similarity between images. Finally, the top-K image and text pairs will be fetched for the following reasoning. 
Benefiting from the multimodal encoding ability of CLIP, we can also retrieve the top-K pairs by matching the input image with texts in the database. We will compare different retrieving methods in the ablation study.


Fine-grained Retrieve. While full-image retrieval can incorporate valuable information, it may overlook fine-grained details, such as diminutive objects and intricate attribute data. Therefore, it is crucial to propose a fine-grained retrieval mechanism that concentrates on the distinct objects within an image. 
To conduct fine-grained retrieval, we initially extract key entities from the input query utilizing the large language model LLaMA2-7B <cit.>. This extraction is facilitated by in-context learning <cit.>, where demonstrations are presented to the LLM, which subsequently outputs the specific entities from the input query. 
Following this, Grounding Dino <cit.> is deployed to identify the objects in the image that correspond to the extracted entities. As illustrated in Figure <ref>, we crop the targeted object and then perform retrieval in a manner akin to the previously described coarse-grained retrieval.





 §.§ Reranking Retrieving Results


Throughout the above retrieving process, the similarity is computed by the CLIP embeddings between images. In this way, the visually similar but semantically different pairs may be recalled. 
To avoid noisy retrieving results which bring different semantic, after obtaining the retrieved pairs for both coarse-grained and fine-grained retrieval, we subsequently propose a reranking strategy designed to recalibrate the retrieved outcomes. 
Specifically, we adopt the LVLM to describe the input image and generate a detailed caption. For coarse-grained retrieval, the input full image is characterized, while describing the cropped image for the fine-grained retrieval.
In this way, we are able to derive the semantics of the input image.

Then, we extract the textual embeddings for both image captions and captions of retrieved images with the CLIP. Finally, we can rerank the retrieved pairs according to the semantic similarity between these captions. In this stage, we also explore other reranking methods, such as k-reciprocal encoding <cit.> or using an external visual entailment tool <cit.>. We will compare them in the ablation study. 






 §.§ Joint Coarse-grained and Fine-grained Decoding


After retrieving and ranking the pairs, we can conduct a joint coarse-grained and fine-grained decoding to yield the final responses. According to the different ways of using the retrieved results, we have two schemes including probability-level fusion and instance-level fusion. The former integrates the results of coarse-grained and fine-grained retrieval into a single prompt for inference, while the latter conducts separate inferences on the coarse-grained and fine-grained retrieval results, followed by a probabilistic-level weighting.

Probability-level Fusion. In this paradigm, we use the prompt for both coarse-grained retrieval and fine-grained retrieval: “Here are the image-caption pairs similar to the test image: ⟨Retrieval Pairs⟩. Based on these pairs and this image: ⟨Input Image⟩. Answer this question: ⟨Input query⟩". 
Here we denote the retrieval-augmented LVLMs as below:


    p^coarse(x_t|x_<t) = SoftMax[LVLM([R_I:h_t])]_x_t,    x_t∈𝒱,


where R_I indicates the hidden states of retrieved image-level pairs. For simplicity, we directly use [:] here. We use p^coarse to represent decoding with coarse-grained retrieval. Similarly, 


    p^fine(x_t|x_<t) = SoftMax[LVLM([R_O:h_t])]_x_t,    x_t∈𝒱,



where R_O indicates the retrieved fine-grained object-level pairs. We represent the decoding with object-level fine retrieval by p^fine(x_t|x_<t). For the probability-level fusion, we can combine them with a hyperparameter α in [0,1] to control the importance of two kinds of retrieval. 


    p^coarse&fine(x_t|x_<t) = α p^coarse(x_t|x_<t) + (1-α)p^fine(x_t|x_<t),



Instance-level Fusion. 
For this kind of fusion, we can integrate both coarse-grained retrieval and fine-grained retrieval with prompts: “Here are the image-caption pairs similar to the test image: ⟨Coarse-grained Retrieval Pairs⟩. Here are the image-caption pairs: ⟨Fine-grained Retrieval Pairs⟩ similar to the ⟨entity⟩ in the input image. Based on these pairs and this input image: ⟨Input Image⟩. Answer this question: ⟨Input query⟩." Here we extract the entity from the input query and fill it in the prompt. 
 

    p^coarse+fine(x_t|x_<t) = SoftMax[LVLM([R_I:R_O:h_t])]_x_t,    x_t∈𝒱,







§ EXPERIMENT


This section will elaborate on the evaluation of our proposed Active Retrieval-Augmented large vision-language model (ARA) across different LVLMs.



 §.§ Evaluation Metrics


POPE <cit.>, the Polling-based Object Probing Evaluation proposed a simplified method for evaluating object hallucinations. In this evaluation framework, the LVLM is tasked with determining the presence of specific objects in a given image, with a balanced distribution of queries for object presence and absence. The evaluation protocol encompasses three distinct sampling settings: random, popular, and adversarial, each employing unique strategies for constructing negative samples. In the random setting, objects not present in the image are chosen randomly. The popular setting selects missing objects from a high-frequency pool, whereas the adversarial setting prioritizes coexisting objects that could be easily mistaken for existing objects in the image. 
The POPE benchmark aggregates data from three diverse sources: MSCOCO <cit.>, A-OKVQA <cit.>, and GQA <cit.>. Within each dataset, every sampling setting comprises 500 images, with 6 questions posed for each image, resulting in a total of 27,000 question-answer pairs derived from the development sets of these datasets. The evaluation predominantly assesses metrics such as accuracy, precision, recall, and F1 score.

MME  <cit.> is a comprehensive benchmark designed specifically for evaluating LVLMs on multiple dimensions. It includes ten sub-tasks related to perception and four sub-tasks emphasizing cognition. According to previous studies <cit.>, we utilize the existence and count subsets for object-level hallucination evaluation, and the position and color subsets for attribute-level hallucination evaluation. Performance is quantified by accuracy and accuracy+, which is also the official evaluation method.


MMStar  <cit.>, foundational to vision-reliant multi-modal evaluation, consists of 1,500 challenge samples meticulously chosen by human experts. It is tailored to systematically benchmark six key capabilities alongside eighteen granular facets, aiming to rigorously assess the multi-modal proficiency of LVLMs using a selection of samples that have been carefully refined and equilibrated.


MMbench  <cit.> is collected from multiple sources, including public datasets and the Internet, and currently, contains 2974 multiple-choice questions, covering 20 ability dimensions. Similar to MME, we focus on hallucination-related ability. Thus, we select 4 ability subsets including object localization, attribute recognition, spatial relationship, and action recognition as our evaluation benchmarks. These subsets form a comprehensive assessment of hallucinations, addressing issues at the object-level, attribute-level, and relation-level comprehensively.





 §.§ Implementation Details
 

In the case of coarse-grained retrieval, we use the COCO dataset <cit.> as the retrieval database. For fine-grained retrieval, we use the VisualGenome <cit.> to build the fine-grained retrieval database as it provides a detailed caption for each grounding box in the image. CLIP-Large is used to extract image embeddings for retrieval.   
To decompose target objects from the input query, we extract key entities from the input query utilizing the large language model LLaMA2-7B <cit.>. 
Grounding Dino <cit.> is deployed to identify the objects in the image that correspond to the extracted entities. If the objects can not be located in the image, we only use coarse-grained retrieval for the following reasoning process.
In this paper, we implement our ARA model on three LVLMs, including LLaVA 1.5, Qwen-VL, and mPLUG-Owl2 to evaluate the effectiveness of our method. For LLaVA 1.5 and mPLUG-Owl2, we use the retrieved texts to augment, while we apply the retrieved image and text pairs to boost Qwen-VL. For these three LVLMs, we implement augmentation using 3, 4, and 5 retrieval components respectively. For active retrieval, we use the query-aware active retrieval. For the hyperparameter α presented in Equation 6, we apply a value of 0.8 for the POPE, MMstar, and the object-level subsets within the MME benchmark, as well as for OL, SR, and ACR in the MMbench. A value of 0.4 is utilized for the remaining evaluation subsets. For active triggering retrieval, we use query-aware active retrieval for its simplicity and effectiveness. The threshold of active triggering is determined by grid search. 

In our experiments, to have a strict comparison of our RAR and vanilla LVLMs, we consistently utilize greedy decoding instead of other probability sampling methods. With this strategy, the token with the highest probability from the post-softmax distribution is directly selected as the next token. 



 §.§ Experiment Results



Results on POPE. Experimental results on POPE under the random, popular, and adversarial settings are summarized in Table <ref>. This benchmark mainly focuses on the object-level hallucination. 
Specifically, the performances of our ARA consistently surpass the baseline results on all of the LVLMs. 
This suggests its pivotal role in augmenting LVLMs with retrieval, thereby reducing instances of object-level hallucination. 
In a more detailed model-specific analysis, RAR demonstrates varied effects across different LVLMs. For all three LVLMs, the F1 score elevation is predominantly driven by a recall boost (e.g., up to 10 points for mPLUG-Owl2), showcasing retrieving to external knowledge base can effectively help detect object presences.








Results on MME Hallucination Subset. 
The MME subset evaluations extend beyond POPE’s scope, encompassing both object-level and attribute-level hallucination. Results in Table <ref> show that our ARA leads to a uniform enhancement in addressing object-level hallucination for all models, except the count score of Qwen-VL. Meanwhile, our ARA demonstrates significant improvements in attribute-level Color scores, contributing to substantial overall performance gains. These improvements are attributable to our designed coarse-to-fine retrieval paradigms, which effectively focus on the target object.

Results on MMStar. In a more difficult benchmark MMStar which rigorously assesses the multi-modal proficiency, as depicted in Table <ref>, our method ARA secures a notable enhancement across all six subsets. For instance, LLaVA-1.5 realizes an average improvement of 8.8 points. Unlike POPE, which may include statistical bias, answering questions in MMStar necessitates reliance on image information. Hence, our retrieval augmentation method introducing external image and text knowledge significantly bolsters model performance.

Results on MMBench Hallucination Subset. 
This benchmark extends our hallucination evaluation to incorporate the relation level in addition to the above object and attribute levels, providing a more holistic assessment for reducing hallucinations. As evidenced in Table <ref>, benefiting from coarse-grained retrieval, our ARA manifests discernible enhancements in the object and relation-level subsets, such as OL, SR, and ACR. For instance, our ARA model surpasses the vanilla LLaVA-1.5 with 5.08 percent points on object localization (OL).  
Concurrently, the integration of fine-grained retrieval into our model facilitates superior performance in attribute-level subsets, namely attribute recognition (ATR). 












§ ABLATION STUDY


In this section, we undertake a series of ablation studies with the aim of identifying the critical elements integral to constructing a resilient and efficient retrieval-augmented large vision-language model.



 §.§ Ablation of Coarse-to-Fine Retrieval



In this section, in order to explore the effects of different factors on coarse-to-fine retrieval in a simple and effective way, we did not use the rerank method or active trigger strategy.


Which kind of retrieval is better? 
Due to the multimodal characteristics of the input, there are various retrieval methods at our disposal. As shown in Table <ref>, we attempt to use the input query and image to search for images and descriptions in the database. For this study, we only conduct coarse-grained retrieval and we use the COCO dataset as the database which contains a caption for each image. 
As a result, there are four retrieval methods available to us. 
As reflected in the results within this table, we adopt the input image as our primary retrieval modality for images in the database throughout our series of experiments.
Surprisingly, using the input query to search the database for image leads to substantially poorer outcomes, achieving only 51.6% accuracy. 
This lower performance may stem from the fact that, despite the presence of relevant keywords, the images retrieved this way tend to be surrounded by a significant degree of noise.

Additionally, we investigate the influence of different embedding models on the efficacy of image-to-image retrieval. 
In our comparison, we examine three models: CLIP-Base/Large <cit.> and DINOV2-Base <cit.>. Among them, we notice that CLIP-base performs better than DINO (86.33 vs 86.07) and this may be related to their different training methods. Moreover, when comparing CLIP models of various sizes, we observe that the larger-scale CLIP models contribute to more pronounced improvements in performance. Therefore, in the context of this study, CLIP-Large is selected as the image embedding model for retrieval purposes.







How to augment the LVLM? After retrieving, we can obtain retrieved images and paired captions. In this section, we explore how to augment the LVLM with this external information. For a fair comparison, in this study, we fix the number of retrieved pairs to 3 for both coarse-grained and fine-grained retrieval. As demonstrated in Table 6, when using coarse-grained retrieval, we observe that the image and caption pair perform worse than the single caption (86.93 vs 87.43). 
However, the opposite results are observed in fine-grained retrieval and the Fine (I+T) achieves 86.53 compared with 86.20 of Fine (T). These results imply that
retrieval augmentation is sensitive to specific configurations. In this experiment, we do not carefully compare with Coarse (I) or Fine (I) as we have found that the LVLMs we use in our experiments are still quite weak in multi-image reasoning.

In Section 3.5, we propose a joint coarse-grained and fine-grained decoding, designed to optimally utilize the two retrieval mechanisms. 
In Table <ref>, we also investigate the effectiveness of probabilistic-level fusion denoted as Coarse & Fine and instance-level fusion labeled as Coarse+Fine. 
As shown from the results, we observe that Coarse+Fine even performs than Coarse retrieval, signifying that simply integrating the coarse-grained and fine-grained retrieved results in a prompt fails to enhance performance, given the distinct granularity of the information retrieved through these two approaches. Instead, the probabilistic-level fusion Coarse & Fine achieves better accuracy than both two kinds of retrieval. 
Thus, in our paper, we use probabilistic-level fusion for the following experiments. It is worth noting that Coarse & Fine (T) performs better than Coarse & Fine (I+T) for LLaVA-1.5 but Coarse & Fine (I+T) achieves better for Qwen-VL (we will describe in Figure 3 later). This result further validates the necessity of carefully selecting retrieval configurations to obtain the most favorable experimental results.

How much external information do we need? 
With the above retrieval, we can obtain multiple image-caption pairs from the database. This research delves into the efficacy of varying retrieved pair configurations. 
Considering that different LVLMs may be sensitive to retrieval configurations, our study incorporates both (T) and (I+T) configurations for a thorough evaluation. The analysis is constrained to examining between one to five retrieved pairs due to the input token limitations inherent in contemporary LVLMs. 
As depicted in Figure <ref>, LLaVA (T) reaches peak accuracy with three retrieved texts, whereas mPLUG (T) begins to saturate when there are four retrieval texts.
Conversely, QwenVL (I+T) demonstrates improved outcomes with a greater number of retrieval pairs; however, it still lags considerably behind the performance achieved by the other two LVLM models. 












 §.§ The effectiveness of Reranking Method


To substantiate the efficacy and indispensability of reranking, we begin by providing a case study within the POPE benchmark. 
As depicted in Figure <ref>, in instances where an image portrays people in a pool, leveraging CLIP for image retrieval may inadvertently introduce irrelevant content due to its predominant emphasis on visual congruence.
For instance, it might retrieve an incongruent image captioned “fruit and a bowl of bread sit on the table." Therefore, an adept reranking methodology can proficiently filter and organize the retrieved images in a more relevant manner. In this particular scenario, our reranking strategy aligns the input image caption with those of the retrieved images. 
By assessing these similarity metrics, we are able to reorganize the pool of retrieved images into a more accurately ranked series, thereby enhancing the precision of the retrieval process.

In order to further quantify the effect of reranking and compare different reranking methods, we conduct experiments in Table <ref>. 
The result reveals that the omission of re-ranking precipitates a decline in accuracy, dropping from 87.17% to 86.93%
This suggests that re-ranking effectively mitigates noise, particularly when employing the top three re-ordered retrieval pairs. 
For comparative purposes, we executed two alternative methodologies. The k-Reciprocal approach <cit.>, a prevalent offline person re-identification method, re-ranks images using the k-Reciprocal nearest neighbor algorithm. While this method offers a swifter execution by circumventing the captioning process of input images, its efficacy slightly underperforms our strategy. Additionally, we examined the employment of the external tool VSR <cit.>, which determines the entailment between the caption of the input image and the retrieved images. As shown in Table 7, this method gains only a slight increase in accuracy (86.93 vs 86.97). This may be due to the visual entailment model not being sufficiently versatile.









 §.§ Ablation of Active Triggering Retrieval


As excessive retrieval activations may lead to undue time expenditure and unnecessary retrieval, in this paper, retrieval is activated using a difficulty metric. In this ablation study, we compare three kinds of active retrieval methods, including confidence-aware active retrieval, query-aware active retrieval, and image-aware active retrieval. As shown in Figure <ref>, we have drawn the proportion of queries that need to be retrieved to achieve peak performance. Firstly, it is quite intuitive, the fluctuations of confidence-aware active retrieval among different models are very significant. For mPLUG-Owl2, we even need to retrieve all the queries, which indicates that confidence-aware active retrieval is not a good indicator of whether to perform a retrieval. For both query-aware active retrieval and image-aware active retrieval, the LVLMs can better determine a trigger threshold. 
Considering that query-aware active retrieval is more concise and the process of adding noise is omitted, we use query-aware active retrieval in all our experiments. 
It is worth noting that the very slight decline in the model's performance after continuing to use retrieval beyond the trigger threshold is due to the fact that the model itself has a high level of confidence, but the retrieval has introduced some redundant noise information, which we cannot filter out through reranking.



 §.§ Comparison with State-of-the-Art Method



In this paper, we augment LVLMs with external knowledge to mitigate hallucinations. 
To further demonstrate the effectiveness of our method, we compare it with VCD <cit.> and the results are demonstrated in Table <ref>. VCD is a state-of-the-art training-free method that explores intra-model knowledge with visual contrastive decoding. From the results, we can notice that our method significantly surpasses VCD in all three settings. For instance, in popular settings, our method gains a 4.06 percent improvement. It is worth noting that VCD performs comparably with the vanilla LLaVA-1.5 when using greedy decoding. This phenomenon is also observed in <cit.>. Instead, our method significantly surpasses the vanilla LLaVA-1.5 model as presented in Tabe 1. This result further verifies the robust improvement of introducing external information when applying our ARA method.







 §.§ Text Generation Quality Evaluation


In this section, we evaluated the effectiveness of our method in improving text generation quality on the MS-COCO validation set[We use the official COCO evaluation package.]. As shown in Table 9, all models LLaVA, Qwen-VL, and mPLUG exhibited significant improvements after applying our method, with an average improvement of 7.7% across the three models. Notably, the improvement was most pronounced for Qwen-VL, with an average increase of approximately 14.9%.




 §.§ Qualitative Results


To qualitatively verify the effectiveness of our ARA method on downstream tasks, we presented five examples from the POPE MSCOCO dataset and MME benchmark. As shown in Figure <ref>, we present three image-caption pairs for both coarse-grained retrieval and fine-grained retrieval. In the above three examples of determining the existence of objects, fine-grained retrieval helps to increase the model's existence information about specific objects. For the last two examples, the coarse-grained retrieval contributed effectively to capturing the overall information of the images. Thus, with both kinds of retrieval, our RAR model can effectively increase the accuracy compared with vanilla LLaVA-1.5. 








§ CONCLUSION


In this paper, we propose a novel framework to augment LVLMs with external knowledge by introducing an
Active Retrieval-Augmented large vision-language model (ARA) for mitigating hallucinations. Specifically, our proposed ARA is grounded in three critical dimensions for mitigating hallucinations in LVLMs, including coarse-to-fine retrieval for dissecting the retrieval targets, pinpointing the most effective retrieval methods, and triggering the retrieval process properly. 
Our findings indicate that by utilizing fitting retrieval mechanisms and timing the retrieval judiciously, we can effectively mitigate the hallucination problem. Our extensive experiments across four benchmarks and three widely-used LVLM confirm ARA’s efficacy in reducing hallucination.

ACM-Reference-Format







 Paper Title: Defending Jailbreak Attack in VLMs via Cross-modality Information Detector.


An Explainable Vision Transformer with Transfer Learning
Combined with Support Vector Machine Based Efficient
Drought Stress Identification
    *
    
===========================================================================================================================================

§ ABSTRACT 

Vision Language Models (VLMs) extend the capacity of LLMs to comprehensively understand vision information, achieving remarkable performance in many vision-centric tasks. 
Despite that, recent studies have shown that these models are susceptible to jailbreak attacks, which refer to an exploitative technique where malicious users can break the safety alignment of the target model and generate misleading and harmful answers. This potential threat is caused by both the inherent vulnerabilities of LLM and the larger attack scope introduced by vision input. 
To enhance the security of VLMs against jailbreak attacks, researchers have developed various defense techniques. However, these methods either require modifications to the model's internal structure or demand significant computational resources during the inference phase.
Multimodal information is a double-edged sword. While it increases the risk of attacks, it also provides additional data that can enhance safeguards. Inspired by this, we propose Cross-modality Information DEtectoR (CIDER), a plug-and-play jailbreaking detector designed to identify maliciously perturbed image inputs, utilizing the cross-modal similarity between harmful queries and adversarial images.
This simple yet effective cross-modality information detector, CIDER, is independent of the target VLMs and requires less computation cost. Extensive experimental results demonstrate the effectiveness and efficiency of CIDER, as well as its transferability to both white-box and black-box VLMs.









§ INTRODUCTION





The remarkable advancements in Large Language Models (LLMs) have significantly improved performance benchmarks in various natural language processing (NLP) tasks <cit.>. To extend the capacities and open up the potentials of LLMs in comprehensively understanding diverse types of data, such as visual information, researchers have developed Vision Language Models (VLMs) that integrate visual modalities to handle vision-centric tasks. VLMs use LLMs as a core, complemented by modal-specific encoders and projectors, enabling them to process, reason, and generate outputs from multimodal data <cit.>. A typical VLM architecture is illustrated in Figure <ref>.



The widespread adoption of VLMs in various applications brings significant safety challenges, particularly due to inherited vulnerabilities from traditional LLMs, such as the susceptibility to jailbreak attacks <cit.>. Jailbreak attacks refer to an exploitative technique where malicious users can craft sophisticated-designed prompts to lead LLMs to answer misleading or harmful questions, effectively breaking the alignment and bypassing the model's safeguard. Various jailbreak attack algorithms targeting LLMs have been proposed, which can be categorized into template-based <cit.> and optimize-based <cit.> approaches. 




Additionally, VLMs not only inherit the vulnerabilities of LLMs but also become more susceptible to jailbreak attacks due to their integration of the visual modality. On the one hand, jailbreak attacks against VLMs can originate from both the textual and visual modalities, significantly broadening the scope of potential adversarial examples <cit.>. On the other hand, recent research indicates that fine-tuning VLMs to learn the vision modality can cause LLMs to disregard their previously learned safety alignment <cit.>. 




The existing jailbreak attacks on VLMs can be categorized into two strategies. One is white-box optimization-based attacks, which define a loss function to generate imperceptible perturbations in the image modality <cit.>. The other is black-box strategies including typographically transforming harmful queries into images such as FigStep <cit.> or adding related images containing harmful text such as QR <cit.>. 

From the defense perspective, optical character recognition (OCR) can serve as an effective detection tool for the second strategy but fails when defending against optimization-based adversarial attacks. In addition, <cit.> creates a vision-language dataset named VLGuard containing both safe and unsafe queries and images, which can be used to fine-tune VLMs for improved safety against jailbreak attacks. However, the effectiveness of VLGuard is only tested on FigStep attack and it requires the model to be white-box to fine-tune. <cit.> proposed a mutation-based jailbreaking detection framework named Jailguard. 

However, the performance of Jailguard heavily relies on the VLMs' original safety alignment, and it significantly increases computational costs during the inference phase.







Multimodal information is a double-edged sword: while it increases the risk of attacks, it also provides additional data that helps enhance safeguards. Inspired by this potential, we propose Cross-modality Information DEtectoR (CIDER), a plug-and-play jailbreaking detector designed to identify maliciously perturbed image inputs, specifically targeting optimization-based jailbreak attacks that are more imperceptible and susceptible. The intuition is that optimization-based perturbations break the VLM's safeguards by capturing the main harmful content in the malicious query. As a result, the semantic distance between a harmful query and an adversarially perturbed image is significantly smaller than that between a harmful query and a clean image. 

Directly utilizing the difference between clean and adversarial images on the semantic distance to harmful query is challenging, as the absolute value of the distance varies across different harmful queries. To address this issue, we incorporate a denoiser to preprocess the vision modality, using the relative shift in the semantic distance before and after denoising to reflect the difference between clean and adversarial images. As shown in Figure <ref>, the key insight of CIDER is to identify whether an image is adversarially perturbed based on the semantic similarity between image and text modality before and after denoising (⟨E_text, E_img(o)⟩ - ⟨E_text, E_img(d)⟩). If the image modality is not perturbed, the semantic similarity between text and image remains stable. However, the adversarially perturbed image designed for jailbreak will experience a significant drop.  By setting a threshold based on this change, we can effectively detect adversarially perturbed images aimed at jailbreaking VLMs. The detailed intuition is elaborated in Section <ref>.

As a pre-detection module encapsulated before any VLMs, the key advantage of CIDER
is its plug-and-play nature, making it independent of the target model.  Additionally, timely inference is crucial for safeguarding VLMs. CIDER achieves this by adding only denoiser procedures, ensuring efficient without introducing significant inference latency. 











In this work, we propose CIDER, an effective and efficient pre-detection module that denoises and inspects each input image.  For images identified as adversarially perturbed for jailbreak purposes (where the semantic shift exceeds a pre-defined threshold), the VLM will refuse to generate a response. Images deemed normal will be processed along with the text input for model inference by the VLM. The workflow of safeguarding VLMs against jailbreak attacks using CIDER is illustrated in Figure <ref>. Our contribution can be summarized as follows:


  * Based on the intuition that cross-modality information is a double-edged sword, we investigate the relationship between malicious queries and adversarial perturbed images in the semantic space. By incorporating a diffusion-based denoiser to uncover the potential of mitigating harmful information in adversarial images through denoising. 



  * We propose a plug-and-play jailbreaking detector, CIDER, which can effectively safeguard VLMs while incurring almost no additional computational overhead.


  * Extensive experiments validate that CIDER outperforms the baseline method, achieving a higher detection success rate while reducing the computational cost as well. Experimental results also demonstrate its strong transferability across both white-box and black-box VLMs and attack methods.





§ INTUITION: CROSS-MODALITY INFORMATION IS A DOUBLE-EDGED SWORD



While multimodal information aggravates model vulnerability to jailbreak attacks, it also provides additional information for defense. The design of CIDER is based on the intuition that optimization-based jailbreak attacks break the VLM's safeguards by sharing harmful content in the malicious query to the image modality. Consequently, the adversarially perturbed image is closer to the harmful query in the semantic space than the clean images. To support this intuition, we first explain the fundamentals of the optimization-based jailbreak attacks on VLMs. Then, we design a few experiments to explore how cross-modal analysis can help safeguard VLMs, and we analyze the semantic difference between clean and adversarial images relative to harmful queries, both before and after denoising.




 §.§ Preliminaries: Optimization-based Jailbreak Attacks on VLMs



Optimization-based VLM jailbreaking is similar to adversarial attacks on image classification tasks <cit.>, with the primary difference being the difference in the loss function. Specifically, given a dataset D={(q,a)} where q represents the harmful queries and a is the corresponding targeted answers, the attacker aims to find an adversarial image x_adv that can encourage the VLM ℱ to generate a when inputting q along with x_adv. The objective can be formulated as: 




    x_adv = x_adv∈ [0,1]^dargmin log(ℱ(a|q,x_adv))



where ℱ(a|q, x_adv) represents the likelihood that the VLM ℱ generate answer a when given the adversarial image x_adv and the query q.




 §.§ Experimental Setup
 


We design a series of experiments to explore how cross-modality information can help safeguard VLMs and to analyze the semantic difference between clean and adversarial images to harmful queries, before and after denoising. We utilize the image and text encoder of the state-of-the-art VLM LLaVA-v1.5-7B <cit.> to capture the semantic meanings. To measure the semantic similarity,  we employed cosine similarity which is a standard metric widely used in information retrieval and natural language processing <cit.>. In terms of denoiser, we incorporate a diffusion-based denoiser <cit.> to preprocess the image modality.










The inputs to the VLMs consist of two modalities: images and text queries. For malicious queries, we utilize the validation set proposed in the Harmbench framework <cit.>, which contains 40 textual harmful behaviors across 7 semantic categories. For images, we use 5 adversarial images generated by an optimization-based jailbreak attack <cit.> and 5 clean images from ImageNet <cit.>. As a result, we have 200 adversarial text-image pairs and 200 clean pairs.







 §.§ Findings

According to the results displayed in Figure <ref>, the key findings can be summarized as follows:





  §.§.§ Finding 1: Adversarial images indeed contain harmful information.



For each harmful query, we calculate the cosine similarity between the queries and both clean and adversarial images,  denoted as ⟨E^M_text, E^C_img(o)⟩ and  ⟨E^M_text, E^A_img(o)⟩ respectively. Figure <ref> shows the distribution of ⟨E^M_text, E^C_img(o)⟩ -⟨E^M_text, E^A_img(o)⟩. It can be observed that the distribution is almost entirely concentrated in the negative region, indicating that, for a specific harmful query, the semantic distance between it and an adversarial image is smaller than that between it and a clean image. Therefore, we can conclude that adversarial images indeed carry harmful information from queries.





  §.§.§ Finding 2: Directly utilizing the semantic difference between clean and adversarial images to harmful query is challenging



Figure <ref> shows the distribution of the absolute value of ⟨E^M_text, E^C_img(o)⟩ and  ⟨E^M_text, E^A_img(o)⟩. Although the distribution differs in the peak and concentration, distinguishing between adversarial and clean images based solely on the absolute value of the difference is challenging. This difficulty arises because the cosine similarity between different queries and adversarial images varies significantly, and the absolute value of the difference does not fully capture the characteristics of the images.






  §.§.§ Finding 3: Denoising can reduce harmful information but cannot eliminate



Subsequently, we applied denoising to each image 350 times, assessing cosine similarity with harmful queries every 50 iterations (visualization of the denoising is relegated to Appendix <ref>). Figure <ref> illustrates how cosine similarity between harmful query and adversarial images decreases as denoising progresses, indicating a reduction in harmful information. Despite this reduction, denoised adversarial images, when tested with harmful text inputs in the VLM, still enabled a significant number of queries to jailbreak. Thus, while denoising mitigates harmful information in images, it does not eliminate their adversarial properties. 




  §.§.§ Finding 4: Relative shift in the semantic distance before and after denoising can help detect adversarial images.



In Figure <ref>, we present the change of cosine similarity before and after denoising (⟨E^M_text, E_img(o)⟩ - ⟨E^M_text, E_img(d)⟩). The distribution of cosine similarity between harmful queries and images shifts notably after denoising, contrasting with the distribution in Figure <ref>. This observed shift supports our intuition that relative changes before and after denoising, rather than absolute differences, provide a method to distinguish adversarial images.  Figure <ref> further demonstrates this trend, showing a consistent decrease in cosine similarity between adversarial images and harmful queries, while the cosine similarity with clean images varies less.




§ METHOD



In this section, we first overview the defense pipeline and the components of CIDER, followed by introducing the threshold selection strategy.


 §.§ Overview

CIDER is implemented on top of the VLMs to defense optimization-based adversarial jailbreak attacks. Figure <ref> presents the overview of the CIDER pipeline. 
Specifically, given a text-image input pair, denoted as <text, img(o)>, CIDER calculates the embeddings of text and image modalities, denoted as E_text and E_img(o). Then, the built-in denoiser in CIDER will perform 350 denoising iterations on the image(o), calculating the denoised embeddings every 50 iterations, denoted as ℰ=E_img(d). 
The img(o) will be identified as an adversarial example if any E_img(d)∈ℰ satisfy the following condition: 



    ⟨E_text, E_img(o)⟩ - ⟨E_text, E_img(d)⟩ >τ

where ⟨·⟩ represents the cosine similarity and τ is the predefined threshold. Consequently, CIDER will directly refuse to follow the user's request by responding “I'm sorry, but I can not assist.” if the image modality is detected as adversarial. Otherwise, the original image and query will be fed into the VLM. The pseudo-code of CIDER is illustrated in Algorithm <ref>.


















 §.§ Threshold selection




The threshold is selected based on the harmful queries and clean images ensuring that the vast majority of clean images pass the detection. The selection of threshold τ can be formulated as: 


    r=∑𝕀(⟨E^M_text, E^C_img(o)⟩ - ⟨E^M_text, E^C_img(d)⟩ <τ) /#samples


where r represents the passing rate and E^M_text, E^C_img(o), E^C_img(d) stand for the embeddings of input query, the input image and denoised image respectively. The threshold τ is determined by controlling the passing rate r. For example, using the τ when setting r to 95% as the threshold indicates allowing 95% percent of clean images to pass the detection.

The selection of the threshold significantly impacts the effectiveness of CIDER: a threshold that is too high will cause many adversarial examples to be classified as clean samples, resulting in a low true positive rate (TPR); conversely, a threshold that is too low will lead to a high false positive rate (FPR), affecting the model's normal performance. 






The ablation study is conducted to determine the optimal threshold.  By treating adversarial pairs as positive samples and clean pairs as negative samples, we plot the TPR-FPR curve with thresholds ranging from 80% to 100% in 1% increments, as shown in Figure <ref>. Ideally, we expect high TPR and low FPR (the upper left corner of the plot). Therefore, we selected τ when r equals 95% as the detection threshold of CIDER.




§ EXPERIMENT



In this section, we begin by outlining the configurations of our experiments, including the models, datasets, baselines, and evaluation metrics. We then evaluate the effectiveness and efficiency of CIDER, comparing with the baseline methods. Next, we discuss the trade-off between robustness and utility. Finally, we demonstrate the generalization of our method.




 §.§ Configurations



Models
Note that CIDER is an auxiliary model that is independent to the VLMs. We use LLaVA to capture the semantic meaning of each modality, but CIDER can be plugged into any other VLMs. To demonstrate the effectiveness of CIDER, we test the detection and defense performance on four open-source VLMs, including LLaVA-v1.5-7B <cit.>, MiniGPT4 <cit.>, InstructBLIP <cit.>, and Qwen-VL <cit.>, as well as the API-access VLM, GPT4V <cit.>. 




Datasets Similar to the dataset used in Section <ref>, we generate 800 adversarial text-image pairs utilizing the 160 harmful queries in Harmbench <cit.> and adversarial images provided by <cit.>. To further demonstrate CIDER will not destroy the original utilities on the normal queries, we also evaluate the utility of CIDER protected VLMs on MM-Vet benchmark <cit.>, which examines 6 core vision language capabilities, including recognition, optical character recognition (OCR), knowledge, language generation, spatial awareness, and math. 

Baseline and evaluation metrics We use Jailguard <cit.> as a baseline, which is a SoTA mutation-based jailbreak detection strategy that protects the VLMs at the inference stage. We involve four evaluation metrics to demonstrate the performance of defending methods from different aspects. From the perspective of the effectiveness of CIDER, we incorporate detection success rate (DSR) and Attack success rate (ASR). DSR represents the proportion of adversarial examples 𝒟 that can be successfully detected: 





    DSRdef=1/|𝒟|∑_(q,x_adv)∈𝒟𝕀_adv((q,x_adv))




ASR is a standard evaluation metric indicating the proportion of samples that can successfully jailbreak VLM ℱ and generate harmful contents, which can be stated as: 



    ASRdef=1/|𝒟|∑_(q,x_adv)∈𝒟𝕀_harm(𝒢(ℱ(q,x_adv)))


𝒢 refers to an LLM classifier <cit.> that determines the harmfulness of a response. 𝕀_harm and 𝕀_harm represent the adversarial and harmful indicator. In terms of efficiency, we measure the time cost of processing a single query. In addition, to evaluate the model utility on regular tasks, responses, we incorporate an online evaluator <cit.> provided along with MM-Vet benchmark, which utilizes GPT-4 to generate a soft grading score from 0 to 1 for each answer.

  



 §.§ Effectiveness

  

DSR
We first demonstrate the overall DSR that CIDER can achieve and compare it with the baseline method, Jailguard. Table <ref> shows that CIDER achieves a DSR of approximately 80%, while the DSR of Jailguard varies, depending on the target VLMs. Note that CIDER is independent of the VLMs, thus the DSR does not vary with the choice of VLMs. However, Jailguard's detection capability relies heavily on the model's safety alignment, so the DSR also varies. VLMs with good alignment achieve high DSR (e.g., GPT4V), while poorly aligned VLMs have relatively low DSR (e.g., InstructBLIP). In other words, Jailguard does not significantly enhance VLM robustness against adversarial jailbreak attacks, whereas CIDER does. Nonetheless, CIDER achieves a higher DSR than most of the Jailguard results, except Jailguard on GPT4V.



ASR
To evaluate the effectiveness of CIDER, we measure the decline in ASR after applying CIDER. Figure <ref> compares the original ASR without defense (red bar), ASR after CIDER (blue bar) and ASR after Jailguard (yellow bar). Note that, Jailguard is solely designed to detect jailbreak input. To ensure a fair comparison, we add an output module following Jailguard's detection. Specifically, if Jailguard detects a jailbreak, it will refuse to respond, similar to CIDER. Otherwise, the original input will be processed by the VLM.

Across all models, defending with CIDER significantly reduces the ASR, yielding better results than the baseline. This indicates that CIDER effectively enhances the robustness of VLMs against optimization-based jailbreak attacks. The most notable improvements are seen in LLaVA-v1.5-7B, where ASR drops from 60% to 0%, and in MiniGPT4, from 57% to 9%. For VLMs with initially low ASRs, such as InstructBLIP and Qwen-VL, ASR is reduced to approximately 2% and 1% respectively. Another notable disadvantage of Jailguard is observed in models like GPT4V, InstructBLIP, and Qwen-VL, which already had strong safety alignment and resistance to adversarial attacks. In these cases, the use of Jailguard resulted in a slight increase in ASR.




 §.§ Efficiency



Timely inference is crucial for safeguarding VLMs in real-world applications. Table <ref> shows the time required to process a single input pair and generate up to 300 tokens with different VLMs, comparing no defense, CIDER, and Jailguard.



CIDER surpasses Jailguard in efficiency, adding only 1.02 seconds per input pair on average, which is relatively acceptable compared to the original inference time. In contrast, Jailguard requires 8-9 times the original processing time. Additionally, CIDER detection is irrelevant to the number of generated tokens in the query answers. Therefore, CIDER does not cause additional overhead when increasing the number of generated tokens, ensuring the stability of CIDER's efficiency.

  



 §.§ Robustness-utility trade-off




To further demonstrate CIDER's influence on the original utilities on normal queries, we also evaluate the utility of CIDER protected VLMs on MM-Vet benchmark, including recognition, OCR, knowledge, language generation, spatial awareness, and math. As shown in Figure <ref>, employing CIDER leads to an approximate 30% overall performance decline on normal tasks. Specifically, CIDER mostly affects the VLM's recognition, knowledge, and language generation capabilities, while it has minimal impact on OCR, spatial awareness, and math skills. We hypothesize that CIDER's stringent decision-making process, which outright rejects tasks once an image is identified as adversarial, hampers the model's overall performance. To further illustrate the robustness-utility trade-off, we conducted an ablation study using denoised images as inputs for the adversarial images, termed CIDER-de. The result is relegated to Appendix <ref>.





To find the optimal balance between safety and utility, we could design a more flexible rejection strategy, such as implementing multi-level thresholds for different types of content. This approach could reduce the negative impact on the model's functionality and we leave it to our future work.





 §.§ Generalization

 

In the previous sections, we evaluated the ASR and DSR against adversarial examples generated by <cit.>. To assess the generalization of our defense method, which is crucial for its applicability to other attacks, we evaluate CIDER against another optimization-based jailbreak attack. We generated 800 adversarial pairs using ImgJP, as proposed by <cit.>. Table <ref> presents the drop in ASRs on four open-source VLMs. The ASR for all VLMs dropped to below 4%, with Qwen reaching 0%. Additionally, CIDER achieves a DSR of 93.87% against ImgJP. These results demonstrate that CIDER effectively generalizes in defending against optimization-based adversarial attacks, highlighting its practical utility for real-world applications.






§ RELATED WORK




Vision Language Model
A typical Vision Language Model (VLM) consists of an image encoder <cit.> to extract feature maps, a projector to align image modality information with text modality, and a Large Language Model (LLM) to integrate textual and visual input for generating responses. The impressive multimodal capabilities of these models have spurred significant research interest, leading to contributions from both academia and industry <cit.>.

Jailbreaking VLMs
Incorporating visual information into the LLM framework significantly broadens its range of applications but also introduces new security vulnerabilities, complicating the security issues of VLMs. Besides transferring text jailbreak templates from LLMs to VLMs <cit.>, effective strategies for jailbreaking VLMs include using gradient-based methods to generate adversarial images <cit.>, and submitting screenshots containing harmful instructions <cit.> or related images <cit.>. This paper focuses on safeguarding VLMs against gradient-based adversarial image attacks, aiming to fortify VLMs against such sophisticated threats and ensure their robustness and reliability in practical applications.

Safeguarding VLMs
Various defense mechanisms have been proposed to address vulnerabilities in VLMs and enhance their security and robustness. These mechanisms can be categorized into proactive and reactive defenses based on their preventive and responsive nature. Proactive defenses aim to prevent attacks through techniques like adversarial training <cit.> and reinforcement learning <cit.> during the training phase. In contrast, reactive defenses focus on detecting attacks during the inference phase using methods such as <cit.>. However, many of these methods require access to internal model parameters or rely on additional large models for implementation. Our approach prioritizes a reactive defense strategy for its practicality and ease of implementation. Notably, Jailguard <cit.> is closely related to our work, as it detects jailbreak queries by analyzing variations in responses to perturbed inputs. However, Jailguard's detection success heavily depends on the safety of the underlying LLM and involves significant computational costs.





§ CONCLUSION



In this work, we propose a plug-and-play cross-modality information detector, CIDER, which can effectively and efficiently defend against adversarial jailbreak attacks. Compared to previous methods, CIDER achieves superior defense performance, as evidenced by higher DSR and a significant decline in ASR, while greatly reducing processing time. We also evaluate the transferability of CIDER to other optimization-based adversarial attacks and demonstrate the robustness-utility trade-off in VLMs. In future research, we aim to improve CIDER by reducing the negative impact on VLM utilities to normal tasks. Additionally, it would be useful to develop defense mechanisms against non-optimization-based jailbreak attacks.




§ LIMITATIONS



We outline the limitations of our study as follows:

1. While CIDER is an effective, efficient, and user-friendly defense mechanism, it does impact VLM performance to some extent. We believe this is due to CIDER's stringent handling of adversarial examples. In future work, we plan to implement multi-level thresholds to process adversarial examples with varying degrees of rigor, aiming to maintain robust defense capabilities without compromising VLM performance.

2. CIDER is specifically designed to defend against optimization-based adversarial jailbreak attacks, and its effectiveness against other types of jailbreak attacks is uncertain. Future research will explore CIDER's effectiveness against these alternative attacks and develop corresponding defense strategies, aiming to enhance the overall security and resilience of VLMs against a wider array of adversarial threats.



§ ETHICS STATEMENT



Ensuring the security of Vision Large Language Models (VLMs) is crucial as they become more widely used in various applications. This paper introduces CIDER, a simple yet effective cross-modality information detector designed to defend against adversarial jailbreak attacks in VLMs. Our work significantly contributes to the field by providing a tool that mitigates known vulnerabilities and lays the groundwork for future improvements in safety measures.
While CIDER marks significant progress, it doesn't make VLMs immune to all threats. Continuous evaluation and updates are crucial as VLMs evolve. By sharing CIDER and our findings, we aim to encourage ongoing research and collaboration, promoting advanced and secure VLMs.
We are committed to addressing the ethical implications of VLM deployment, ensuring transparency, and prioritizing the responsible use of these technologies for societal benefit.








§ VISUALIZATION OF DENOISING

Figure <ref> presents an example of an adversarially perturbed image, showing the effects of denoising it after 100, 200, and 300 iterations.




§ ABLATION STUDY ON ROBUSTNESS-UTILITY TRADE-OFF

To further illustrate the robustness-utility trade-off, we perform an ablation study using denoised images as inputs for adversarial images, referred to as CIDER-de. Figure <ref> shows the ASR of CIDER-de and Figure <ref> shows the MM-Vet score of it. It can be observed that using CIDER-de hardly impacts the utility of the VLM. However, this comes at the expense of greatly diminished defensive effectiveness.







