Paper Title: DeliLaw: A Chinese Legal Counselling System Based on a Large Language Model.



























Equal contribution
n.xie@siat.ac.cn




  University of Chinese Academy of Sciences
  Shenzhen Institute of Advanced Technology, CAS
  SIAT-DELI AI and Law Joint Lab
  China




[1]
yl.bai@siat.ac.cn

  Shenzhen Institute of Advanced Technology, CAS
  China


















hy.gao1@siat.ac.cn

zq.xue@siat.ac.cn

  Hebei University
  China








feitengfang@mail.ustc.edu.cn

qixuanzhao@mail.ustc.edu.cn

lzj0@mail.ustc.edu.cn

  University of Science and Technology of China
  China



l.zhu@siat.ac.cn

South University of Science and Technology of China
China
 









Corresponding Author
sw.ni@siat.ac.cn

[2]
min.yang@siat.ac.cn

  Shenzhen Institute of Advanced Technology, CAS
  SIAT-DELI AI and Law Joint Lab
  China
  















§ ABSTRACT 
  Traditional legal retrieval systems designed to retrieve legal documents, statutes, precedents, and other legal information are unable to give satisfactory answers due to lack of semantic understanding of specific questions. Large Language Models (LLMs) have achieved excellent results in a variety of natural language processing tasks, which inspired us that we train a LLM in the legal domain to help legal retrieval. However, in the Chinese legal domain, due to the complexity of legal questions and the rigour of legal articles, there is no legal large model with satisfactory practical application yet. In this paper, we present DeliLaw, a Chinese legal counselling system based on a large language model. DeliLaw integrates a legal retrieval module and a case retrieval module to overcome the model hallucination. Users can consult professional legal questions, search for legal articles and relevant judgement cases, etc. on the DeliLaw system in a dialogue mode. In addition, DeliLaw supports the use of English for counseling. we provide the address of the system:
<https://data.delilegal.com/lawQuestion>.




[500]Information systems Information systems applications






















DeliLaw: A Chinese Legal Counselling System Based on a Large Language Model
    Min Yang
    August 5, 2024
===========================================================================









§ INTRODUCTION

Traditional legal retrieval systems <cit.> aim to retrieve legal information such as legal documents, statutes and case law. However, they are limited in their ability to semantically understand user queries.Open-source large language models, such as LLAMA <cit.> and LLAMA2 <cit.>, Falcon <cit.>, Vicuna <cit.>, MOSS <cit.>, ChatGLM, and ChatGLM2 <cit.>, have demonstrated satisfactory performance in general-purpose domains following pre-training on large-scale corpora. However, deploying them in very specialized fields such as medicine, legal, and finance presents challenges, primarily due to the scarcity of high-quality, fine-tuned data in these areas and the inherent problem of "hallucination" in generative models. Even the state-of-the-art GPT-4 model <cit.> in the Chinese legal domain generates a significant number of fictitious legal texts, highlighting the prevalence of this issue.

Recent studies have explored the fine-tuning of open-source LLMs using legal data to develop legal-specific LLMs. One example is the Chinese legal model LawGPT <cit.>, which is obtained by fine-tuning ChatGLM-6B LoRA 16-bit instructions. The fine-tuning process involves utilizing existing legal Q&A datasets and constructing high-quality legal text Q&A based on real legal articles and cases using ChatGPT’s API. While LawGPT incorporates more legal knowledge into ChatGLM, its performance remains suboptimal. To address this limitation, Huang et al. <cit.> proposed a method that retrieves relevant legal articles based on the user’s query and contextual information, which can serve as evidence for the query before generating each response. The final response is then generated based on these legal articles. ChatLaw <cit.> leverages an external knowledge base to mitigate model hallucinations. However, unlike LawGPT, ChatLaw trains a model specifically to extract legal feature words from the user’s everyday language. These extracted legal keywords are then used as queries for law retrieval.











While Chinese legal LLMs have made significant advancements compared to general-purpose LLMs, they still fall short of direct practical application. To address this gap, this work introduces DeliLaw, a practical legal interrogation system built upon a large language model. We first trained deep learning models dedicated to intent classification to effectively classify user queries according to different legal application scenarios. To build our premium dataset, we sourced extensive data from expert Chinese legal platforms, enriched with statutes accessed through our regulatory retrieval system, ensuring relevant legal texts were included. We crafted diverse prompts based on professional legal advice to facilitate answer generation. Lawyers were engaged to annotate the data, critical for upholding the dataset's high quality and professional reliability. For legal retrieval, we enhance the adaptability and accuracy of the system through a two-stage fine-tuning.The laws in our system response are retrieved from the law database, and since the law database is updated in real time, all the laws returned by the system are real and valid laws. Combining vector library retrieval and ElasticSearch technology, we constructed an efficient case retrieval module to provide users with comprehensive and practical legal information. In the end, by organically integrating the modules and comprehensively reasoning about DeliLaw, we successfully built an intelligent and professional system to meet users' diverse legal needs.The system’s interface is depicted in Figure 1. It enables users to engage in a free dialogue with an AI assistant to seek legal-related inquiries. The AI assistant responds professionally by providing specific legal provisions based on retrieved laws. The link to our demo video is: <https://youtu.be/DxuRGgLLjD0>.






§ SYSTEM OVERVIEW

Intent Classifier
This intent classification model is based on a RoBERTa-large architecture, trained utilizing the dataset we compiled with four distinct classifications. On our internal test set, the accuracy of the model on “LawQuestion”, “LawSearch”, “CaseSearch”, and “General” the accuracy is 99.9%, 100%, 100%, and 99.6% for the four categories respectively.


 §.§ Law Retriever

Our legal retrieval model is based on the BGE embedding model <cit.>, currently the state-of-the-art in embedding models for dense retrieval. BGE undergoes pre-training with retroMAE <cit.> and contrastive learning on a large corpus of Chinese texts, boasting formidable retrieval capabilities for Chinese. We fine-tune the BGE-large-zh-v1.5 model using the infoNCE loss function to adapt it to the legal domain retrieval. The fine-tuning of the BGE model utilizes collected query-statute pairs from two main sources: 1. Question and answer data from the internet, where the queries are usually shorter and the scope of the labeled statutes is smaller. 2. Queries generated using LLM-based data augmentation methods (e.g., Promptagator <cit.>, UDAPDR <cit.>), which are longer and can use a diverse set of statutes as labels. In the LLM-based data augmentation approach, we initially prompt GPT-4 to generate queries for a small set of statutes, then use these generated queries to fine-tune a smaller LLM in a LoRA manner. The fine-tuned smaller LLM is then used to generate queries for fine-tuning the retrieval model, achieving a balance between quality and economy.
In our training setup, fine-tuning is divided into two stages. In the first stage, the positive samples are the query’s labeled statutes, and negative samples use in-batch negative statutes, along with hard negative statutes mined using BM25. The second stage mines new hard negative statutes using the retrieval model fine-tuned in the first stage, with other settings being the same, to re-fine-tune the retrieval model.
During retrieval, both the query and the statute are input into the fine-tuned model, from which the embedding of the last layer's [CLS] token is extracted, yielding a vector of length 1024. The similarity between the query and statute is calculated using the cosine similarity of the vectors.  The statute embeddings are stored in the vector database Milvus <cit.>, and for each query, the top three statutes with the highest similarity are retrieved from the database.
The final experimental results, evaluated under three regulatory frameworks, demonstrate an MRR of 61.6% and a RECALL of 71.1%. These figures highlight the effectiveness of our approach in enhancing the precision and recall of legal information retrieval, thereby underscoring the model's capability to accurately identify relevant statutes in response to user queries.


 §.§ Case Retriever

Due to the large size of our case database, which consists of hundreds of millions of lengthy cases, direct retrieval using the Milvus vector library for similar laws and regulations is not efficient. To overcome this challenge, we adopt a two-step retrieval approach. Firstly, we extract keywords from the query. Then, we utilize the ElasticSearch library to retrieve relevant cases based on these keywords.





 §.§ Law-specific Fine-tuned LLM

To construct our legal fine-tuning data, we begin by leveraging ChatGPT’s API to generate Q&A data and multi-round dialogue data using real laws as the basis. This initial data are then carefully reviewed and modified by professional lawyers to ensure its accuracy and relevance. Finally we fine-tune the ChatGLM2-6b model using the constructed legal data together with other publicly available generic data. 



 §.§ Inference Process

Figure 2 shows an inference process of DeliLaw system. Specifically, the user's question input will go through an intentional classification model, and the user's question will be classified into four categories: “LawQuestion”, “LawSearch”,“CaseSearch”, and “General”. If the predicted category is “General”, it means that the user is asking a general question, e.g., “Hi, what's your name?”, which will be fed directly to our fine-tuned Legal Large Language Model for answering. If the category is “CaseSearch”, e.g., “Please give me cases related to hit-and-run.”, then the user's question is used as a query to call the Case Retriever to retrieve legal cases and return them to the user directly. If the category is “LawSearch”, e.g., “Article 3 of the Civil Code of the People's Republic of China...”, then the user's question is used as a query to call the Law Retriever to retrieve legal articles and return them to the user directly. If the category is "LawQuestion" as in Figure 2, the user's question is used as a query, and then the Law Retriever is called, which combines the retrieved law and the user's question using the prompting process and feeds it to the fine-tuned Legal LLM, which will then refer to the law and give a final response.




§ DELILAW DATASETS



 §.§ Open General Data

To maintain the LLM's general capabilities while boosting its proficiency in Chinese dialogue, we've collected extensive Chinese public datasets for training. Utilizing these datasets holistically aims to render the LLM versatile and resilient in handling Chinese conversations.



 §.§ Legal Data

Some work <cit.> <cit.> has shown that when small samples of high-quality data are incorporated into the fine-tuning process, the performance and generalisation of the model can be significantly improved. We focus on constructing small samples of high-quality legal data, which we introduce during the fine-tuning phase.




  §.§.§ Legal Q&A data

We collected a large amount of legal Q&A data on legal websites. After cleaning the data with duplicated answers and low quality answers, we extract the legal texts cited in the answers by rules and call our statute retrieval interface to get the complete content. Finally, we input the questions and related statutes into GPT-3.5 to expand the answer content and get better answers.

However, the questions asked in the collected data are relatively short, which does not improve the model's ability to understand longer questions.


For this reason, we extracted 28k pieces of initially cleaned data and extended our problem using deep evolution and breadth evolution <cit.>. The questioning is made complex and diverse by deep evolution and breadth evolution. The evolved data are filtered to remove data with no information gain compared to the original problem.



  §.§.§ Human annotated data

We manually annotated 3,492 data involving real-world legal scenarios quizzes and added these to supervised fine-tuning to optimise the model. To ensure the professionalism and accuracy of the data, we invited legal experts to perform the annotations. These data cover seven common areas of law: intellectual property, criminal offences, labour disputes, property disputes, corporate compliance, matrimonial and family matters, and urban renewal.



  §.§.§ Cause of actions in GPT-4

We strategized to augment the model's legal acumen by curating data across diverse legal contexts. Initially, we amassed 602 criminal and 200 administrative case inquiries. To bolster the reliability of generated answers, we interfaced with legal databases, extracting the five most pertinent statutes per question. These statutes served as a foundation for GPT-4 to craft informed responses. To refine these outputs, expert legal professionals vetted and enhanced the GPT-4-generated content, ensuring high-quality, precise, and current regulatory insights.



  §.§.§ Judicial syllogism Data

LLMs excel in parsing brief queries but falter with lengthy ones, often due to gaps in specialized legal knowledge, leading to less than optimal responses. To address this, we collated complex legal inquiries from various sources including legal news, social media, and industry forums, covering 263 typical civil litigation issues. Legal experts crafted detailed answers for these questions, employing the judicial syllogism framework: statutes as major premises, case facts as minor premises, and judgements as conclusions. By adhering to this logical structure, we ensured that the generated answers were precise, coherent, and aligned with legal reasoning standards.


  §.§.§ Guided conversations

In legal practice, client-attorney dialogue is central, where clients often present issues tersely, lacking depth. Lawyers must probe beneath surface questions, skillfully drawing out detailed, pertinent facts. Guided discussion uncovers case nuances, enhancing understanding and pinpointing legal challenges for tailored advice.
Hence, in formulating the dataset, we drew inspiration from the multi-turn dialogue between Doctors and patients in HuatuoGPT <cit.>. Building upon this foundation, we devised roles for clients and attorneys utilizing GPT-4 to generate guiding dialogues within the context of legal consultations.



§ EVALUATION



We selected 100 questions for a comprehensive review of five LLMs, including DeliLaw, and analysed them in depth on the basis of the following explicit scoring criteria.


  * 3 : Answer is fully relevant to the search, both the cited statute and law are highly relevant, and the logic is applied correctly.

  * 2 : Answer is generally relevant to the content of the search, the citation of the statute is correct but the law is incorrect and the logic is correctly applied.

  * 1 : Answer is relevant to the content of the search, the citation of the statute is incorrect or the law is incorrect or the logic is incorrect.

  * 0 : Answer is not relevant to the content of the search, the statute is incorrectly cited or the statute is incorrect or the logic is incorrect.


Figure 3 shows the results of comparing DeliLaw with the other four models in terms of scoring on the manual evaluation. It can be seen that compared to ChatGPT, Baichuan, ChatGLM2, and DeliLaw, DeliLaw performs better in answering the questions with accurate references to regulations and laws, and logical applicability in a significant number of cases.DeliLaw produced better responses than TongYiFaRui in 30 evaluation questions.
From the overall evaluation, it can be seen that DeliLaw's responses to legal advice questions are preferable to the other models.



§ CONCLUSION

DeliLaw, a Chinese legal counseling system, facilitates easy searching for legal articles, case retrieval, and access to professional legal advice. It efficiently connects users with a vast array of legal resources and expert opinions, ensuring that both seasoned professionals and the public can navigate the complexities of the law with ease. The system currently boasts more than 20,000 professional users and receives nearly 3,000 calls daily. Future research will concentrate on enhancing the system's contextual understanding, broadening the coverage of legal fields to encompass more legal knowledge, and further optimizing the user experience.



This work was supported by National Key Research and Development Program of China (2022YFF0902100), China Postdoctoral Science Foundation (2024M753398), Postdoctoral Fellowship Program of CPSF (GZC20232873), GuangDong Basic and Applied Basic Research Foundation (2023A1515110718, 2024A1515012003 and 2024A1515030166), National Natural Science Foundation of China (62376262), Shenzhen Basic Research Foundation (JCYJ20210324115
614039). 

ACM-Reference-Format










 Paper Title: Entropy, Thermodynamics and the Geometrization of the Language Model.



plain


   
    Entropy, Thermodynamics and the Geometrization of the Language Model
        
    

    Wenzhe Yang
    
    

    
    
    

    
    <https://orcid.org/0000-0002-2456-7453>

    
    



§ ABSTRACT 

In this paper, we discuss how pure mathematics and theoretical physics can be applied to the study of language models. Using set theory and analysis, we formulate mathematically rigorous definitions of language models, and introduce the concept of the moduli space of distributions for a language model. We formulate a generalized distributional hypothesis using functional analysis and topology. We define the entropy function associated with a language model and show how it allows us to understand many interesting phenomena in languages. We argue that the zero points of the entropy function and the points where the entropy is close to 0 are the key obstacles for an LLM to approximate an intelligent language model, which explains why good LLMs need billions of parameters. Using the entropy function, we formulate a conjecture about AGI. 

Then, we show how thermodynamics gives us an immediate interpretation to language models. In particular we will define the concepts of partition function, internal energy and free energy for a language model, which offer insights into how language models work. Based on these results, we introduce a general concept of the geometrization of language models and define what is called the Boltzmann manifold. While the current LLMs are the special cases of the Boltzmann manifold.








Keywords: Language Model, Moduli Space, Entropy Function, Thermodynamics, Geometrization of Language Model, Boltzmann Manifold.

 




§ INTRODUCTION


In November 2022, OpenAI surprised the world with the release of the highly “intelligent” ChatGPT, which possesses many striking abilities that far exceed any other available systems <cit.>. For example, it can answer questions with a very high accuracy and engage in human-like conversations. It is the first time that a computer performs well enough in human languages that makes people wonder whether it acquires true intelligence. Ever since then, the arm race in the area of Large Language Model (LLM) has become extremely fierce, with the announcements of new LLMs with hundreds of billions of parameters regularly. Now there are abundant Chatbots based on LLMs available in markets, many of which have astonishing new abilities and behaviors <cit.>. Especially, when the number of parameters (e.g., weights) are increased to a threshold, then some surprising behaviors emerges <cit.>, which is extremely intriguing. 

An LLM is a generative model, which means it can practically sample a new sentence given a prompt. More precisely, suppose we are given an arbitrary prompt w_1⋯ w_n, where each w_i is a word, an LLM can compute the probability distribution of the next word, i.e., P(w_n+1|w_1 ⋯ w_n). Next, we sample the next word w_n+1 according to this probability distribution <cit.>. After we obtain w_n+1, we repeat this process and sample w_n+2 according to the probability distribution P(w_n+2|w_1 ⋯ w_n w_n+1) computed by the LLM. We repeat this process until some conditions are met, e.g., the LLM outputs an end-of-sequence symbol.

Let us now briefly review the basic structures of the LLMs based on the transformer architecture, to the extent that will be needed in this paper. For a more detailed and thorough treatment, the readers are referred to the book <cit.>. First, in order for a computer to be able to compute words, words are first embedded into a vector space, which is called the Word-Embedding <cit.>. Suppose 𝒲 is the finite set of all words, a Word-Embedding is an injective map

    ι: 𝒲→ℝ^N,

where the standard inner product ⟨·, ·⟩ on ℝ^N measures the correlations between words. Given an arbitrary vector 𝐯∈ℝ^N, it defines a distribution on 𝒲 via

    P_𝐯(w) = exp( ⟨𝐯, ι(w) ⟩)/∑_w' ∈𝒲exp( ⟨𝐯, ι(w') ⟩),

which is called the Boltzmann distribution in statistical mechanics <cit.>. 


In particular, given a word w ∈𝒲, it also gives us a distribution on 𝒲 via

    P_ι(w)(w') = exp( ⟨ι(w), ι(w') ⟩)/∑_w”∈𝒲exp( ⟨ι(w), ι(w”) ⟩),

the value of which measures the correlations between w and w'. In practice, the embedding ι is chosen such that P_ι(w)(w') is the probability that w' is in an l-neighborhood of w <cit.>. One nice property of this embedding is that generally the embedded vectors {ι(w)| w ∈𝒲} lie in a lower dimensional vector subspace of ℝ^N as these vectors are not independent from each other. A famous linear relation is 

    ι(Queen) - ι(Woman) = ι(King) - ι(Man).

For more details, the readers are referred to the papers <cit.>.

After the Word-Embedding, a sentence w_1⋯ w_n is mapped to a vector

    (ι(w_1),⋯,ι(w_n)) ∈ (ℝ^N)^n.

As the name suggests, a transformer is a transformation

    T_n: (ℝ^N)^n → (ℝ^N)^n,

which is a differentiable often smooth map <cit.>. For the details of the construction of a transformer and some of the ideas behind it, the readers are referred to the paper <cit.> and the book <cit.>. In particular, each transformer depends on internal parameters which are tuned during the training process. Notice that transformers can be nested in the form

    𝒯_n = T_n,k∘⋯∘ T_n,1,

where k is the number of layers and different layers have different parameters. Let Proj be the projection of (ℝ^N)^n to the last ℝ^N in the direct product, hence Proj∘𝒯(ι(w_1),⋯,ι(w_n)) is a vector of ℝ^N, which defines a distribution on 𝒲 via Eq. (<ref>). The training of the transformers on a very large corpse is to make sure that this distribution is a good approximation of P(·|w_1⋯ w_n) <cit.>.

The LLMs have made rapid advance from a numerical and experimental point of view, but there have been very few papers that attempt to understand the properties and behaviors of LLMs from a analytical point of view. More precisely, we now desperately need a theory that can explain these surprising properties and behaviors, which is the main motivation of this paper. 


In this paper, we first use set theory, analysis and probability theory to rigorously define the concept of Causal Language Model (CLM), and show it is equivalent to the Predicative Language Model (PLM), which basically predicts the next word of a prompt. We formulate a generalized distributional hypothesis for a CLM based on functional analysis and topology. We introduce the concept of the moduli space of distributions on words, which is the key metric space to understand the power and properties of a language model.

The development of human language is for human to exchange information efficiently between individuals. In order to analytically study a language model, we must understand how information is quantified during conversations. In this paper, we introduce the entropy function for CLM and PLM, and show how it can play a crucial role in the study of language models. In particular, we analyze the zero points of the entropy function, and show how to use information theory to understand the meaning of them. We argue that the zero points of the entropy function and the points where the entropy is close to 0 are the key obstacles for an LLM to approximate an intelligent language model, which explains why good LLMs need billions of parameters. We also formulate an extension conjecture about the properties of AGI using the entropy function.

Next, we use thermodynamics to give an physical interpretation of the language models. In this interpretation, a sentence is a microstate that has a potential energy. All the possible outputs of a prompt define a statistical ensemble, which is distributed according to the Boltzmann distribution <cit.>. We introduce the concept of the partition function, internal energy and Helmholtz free energy for this statistical ensemble associated. Using the Helmholtz free energy, we show how the game of word-predicting can be interpreted as the process of a molecule growing in physics.

Later, we formulate the concept of the geometrization of a language model, and show that the accuracy of a language model is determined by how precise the embedding of the moduli space in a geometrization is. We also define the Word-Embedding with respect to a general manifold with a pairing structure, and introduce the corresponding Boltzmann manifold with respect to it. At last, we show how the current LLMs can be viewed as special cases of the geometrization of language models that use the linear space together with its inner product. Based on our theory, we also pose several important open questions, e.g., which manifold together with a pairing is optimal for the geometrization of language models. Perhaps different languages would need different manifolds and pairings, namely different Boltzmann manifolds in the geometrization. Another important direction is whether the current researches on statistical mechanics and differential geometry can offer new tools in the study of LLMs.


The layout of this paper is as follows. In Section <ref>, we use set theory, analysis and probability theory to rigorously define language models, i.e., CLM and PLM, and we introduce the moduli space of distributions associated with a language model. In Section <ref>, we introduce the entropy function, and show how information theory can be applied to the study of language models. In Section <ref>, we present a thermodynamic interpretation of language models and define statistical functions such as partition function, internal energy and Helmholtz free energy for a language model. In Section <ref>, we formulate the concept of the geometrization of language models, and define the Boltzmann manifold. In Section <ref>, we show that the current LLMs are special cases of the geometrization of language models. In Section <ref>, we conclude this paper and propose several open problems.









§ THE LANGUAGE MODEL AND MODULI SPACE
 



In this section, we will use set theory, analysis and probability theory to give rigorous definitions of language models. We first define what is a general sentence in the sense of set theory <cit.>. Then we introduce the definitions of the General Language Model (GLM), the Causal Language Model (CLM) and the Predicative Language Model (PLM). At last, we define what is called the moduli space of distributions for a PLM, which is the crucial metric space to understand the properties and behaviors of language models.



 §.§ Sentence as a Free Mathematical Sequence?


Suppose 𝒲 is the finite set of all words in a language, the set of all sentences associated with 𝒲 is defined in the same way as in set theory and logic <cit.>. First, let ∅ be the unique sentence with no word, whose length is by definition 0. Let 𝒮_0 be the set that consists of a unique element ∅

    𝒮_0 = {∅}.


The inclusion of ∅ as a sentence will make our analysis simpler from a mathematical point of view. While later we will also see ∅ is a suitable symbol for the end (or even the beginning) of a sentence.

A sentence of length 1 is just a word of 𝒲. Let 𝒮_1 be the set of all sentences of length 1, namely

    𝒮_1 = 𝒲.

Given an positive integer n ≥ 2, let 𝒮_n be the set of all sentences of length n, i.e., an element of 𝒮_n is a sequence

    𝐬_n=w_1 w_2 ⋯ w_n with w_i ∈𝒲.

Here, the words in a sentence do not need to be different from each other, e.g., we allow the case where w_i=w_j, i ≠ j. The set of all sentences of finite lengths is by definition the union of all 𝒮_n

    𝒮 = ⋃_n=0^∞𝒮_n.

Notice that here we allow the length of a sentence to be arbitrarily large. 

In the definition of 𝒮, we have allowed the existences of arbitrary sentences that do not satisfy grammar rules. In this paper, the grammar is instead considered as part of the language model.



The cardinality of 𝒮 is countably infinite, i.e., there exists a bijective map between 𝒮 and ℕ.


Since 𝒲 is a finite set, this lemma is an immediate result of elementary set theory, see the book <cit.> for more details.



Even in this most general setting, we do not consider the case of sentence of infinite length, i.e., infinite sequence, which is very important from both practical and set-theoretic points of view. Practically, human brain has not evolved to understand infinity properly, so it makes no sense to a mortal that a sentence is infinite. Set-theoretically, the set of all infinite sequences has the same cardinality as the real numbers, i.e., its cardinality is ℵ_1 <cit.>. So we do not really want to walk into the deep water of axiomatic set theory.


There is a natural operation defined on 𝒮 that is called concatenation: given a sentence s_m ∈𝒮_m and a sentence s_n ∈𝒮_n, their concatenation is a new sentence s_m s_n that lies in 𝒮_m+n. As a convention, the concatenation between ∅ and s_n is s_n itself, i.e.,

    ∅𝐬_n = 𝐬_n ∅ = 𝐬_n.

This concatenation operation is associative, but not commutative, while the empty sentence ∅ serves as a two-sided identity. So the set 𝒮 has a monoid structure <cit.>.








 §.§ The Causal Language Model


Now we are ready to give a more mathematical and general definition of the language model.


 
Given an arbitrary sentence s ∈𝒮, a General Language Model (GLM) ℒ_G defines a joint probability distribution P_ℒ_G,𝐬 on 𝒮×𝒮, where the probability P_ℒ_G,𝐬(𝐬_1,𝐬_2) measures the likelihood of the concatenation 𝐬_1𝐬𝐬_2 among all possible choices. Namely, P_ℒ_G,𝐬(𝐬_1,𝐬_2) is the probability that there is a sentence 𝐬_1 before 𝐬 and a sentence 𝐬_2 after 𝐬.



Now, we consider the case where the sentence 𝐬_1 on the left side of 𝐬 is ∅, in which case the probability distribution in Definition <ref> defines a distribution on 𝒮. Hence it motivates the following definition.


Given an arbitrary sentence 𝐬∈𝒮, a Causal Language Model (CLM) ℒ defines a probability distribution on 𝒮. Namely, given a prompt 𝐬, we have a distribution P(·|𝐬) such that the probability that ℒ outputs a sentence 𝐬' is P_ℒ(𝐬'|𝐬).


In the definition, we allow the prompt 𝐬 to be ∅, in which case the CLM ℒ defines a probability distribution on 𝒮, i.e., P(𝐬_out|∅). It is called causal because 𝐬' is after 𝐬, so there is a causality between 𝐬 and 𝐬'. From the definition, the input to ℒ can be of arbitrary length, while the output sentence can also be of arbitrary length. In fact, such a definition of CLM is extremely powerful.




Let ℬ(𝒮,ℝ) be the set of all functions with finite absolute summation, i.e.,

    ℬ(𝒮,ℝ) = {f:𝒮→ℝ| ∑_𝐬∈𝒮|f(𝐬)| < ∞}.

The set ℬ(𝒮,ℝ) is a metric space with a metric given by

    ‖ f-g ‖  =  ∑_𝐬∈𝒮 | f(𝐬) - g(𝐬) |.

For more details about this space, the readers are referred to the book <cit.>. Clearly for every 𝐬, the distribution P_ℒ(·|𝐬) is a non-negative function that lies in ℬ(𝒮,ℝ), in fact ‖ f ‖ = 1. Therefore, there exists a map

    ρ_ℒ: 𝒮→ℬ(𝒮,ℝ).


In 1950s, linguists formulated the distributional hypothesis, which basically says that the meaning of words are determined by their contexts <cit.>. We generalize this hypothesis using real analysis.

If ℒ is an intelligent language model, then the map ρ_ℒ is an embedding that is also discrete. Namely, each point ρ_ℒ(𝐬) has a neighborhood that does not contain any other ρ_ℒ(𝐬') where 𝐬≠𝐬'. 

For more details of the meaning of “discrete” in the sense of topology, the readers are referred to the book <cit.>. As a result, the embedding ρ_ℒ induces a metric on 𝒮.



It should be stressed that in the definition of a CLM, we do not require it to have “intelligence” behaviors. For example, for a given prompt 𝐬, the probability distribution P(· | 𝐬) can be a “random” distribution over 𝒮, which can even outputs sentences that are quite pathological. Hence, intelligence is not incorporated in the definition of the language model. Instead, only highly special language models can exhibit intelligence behaviors.


 







 §.§ The Predicative Language Model
 

Now we give the definition of the Predicative Language Model.


Given a sentence 𝐬, a Predicative Language Model (PLM) ℒ_P defines a probability distribution P(·|𝐬) on 𝒲∪∅. Here, P(∅|𝐬) is the probability that ℒ_P outputs nothing and P(w|𝐬) is the probability that ℒ_P outputs the word w ∈𝒲.


We have the following lemma.

The definition of the PLM is equivalent to the definition of the CLM.


If we have a PLM ℒ_P, then by repetitively using it we can generate sentences. More precisely, given an input 𝐬∈𝒮, we can generate a sentence 𝐬'=w_1w_2⋯ w_n until ℒ_P outputs ∅ via

    ℒ_P(𝐬) = w_1,  ⋯,  ℒ_P(𝐬w_1 ⋯ w_n-1) = w_n,  ℒ_P(𝐬w_1 ⋯ w_n) = ∅.

The probability that ℒ_P outputs 𝐬'=w_1⋯ w_n given 𝐬 is 

    P(𝐬'|𝐬)=P(w_1|𝐬) · P(w_2|𝐬w_1) ·…· P(w_n|𝐬w_1 ⋯ w_n-1) ·  P(∅|𝐬𝐬').

In this way, the PLM ℒ_P defines a probability distribution on 𝒮 for any prompt 𝐬, hence it produces a CLM.

We now show the inverse construction from CLM to PLM. Namely, suppose we have a CLM ℒ, we now construct a PLM ℒ_P that reverses the process in the previous paragraph. Given a prompt 𝐬, let us construct a distribution on 𝒲∪∅. First, P(∅|𝐬) is straightforward to define, i.e., P_ℒ(∅|𝐬). For an arbitrary word w, let 
    P(w|𝐬) = ∑_𝐬' ∈𝒮 P_ℒ(w𝐬'|𝐬).

Notice that here the sum is just over all sentences whose first word is w. In this way, we obtain a PLM.



As shown in the proof of this lemma, the empty sentence ∅, which is the unit of the monoid 𝒮, plays the role of ending the sentence outputted by ℒ_P. Hence comes a crucial definition in our treatment of PLM.

 
The space of all possible distributions on 𝒲∪{∅} that come from a PLM ℒ_P is called the moduli space of distributions of ℒ_P, which is expressed as

    ℳ = { P(·| 𝐬)| 𝐬∈𝒮}.


In fact, this moduli space is a metric space. Let ℬ(𝒲∪{∅},ℝ) be the set of all functions with finite absolute summation

    ℬ(𝒲∪{∅},ℝ) = {f:𝒲∪{∅}→ℝ| ∑_w ∈𝒲∪{∅}|f(w)| < ∞}.

The set ℬ(𝒲∪{∅},ℝ) is a metric space with a metric given by

    ‖ f-g ‖  =  ∑_w∈𝒲∪{∅} | f(w) - g(w) |.

For more details, the readers are referred to the book <cit.>. Clearly, for every 𝐬∈𝒮, the distribution P(·|𝐬) ∈ℳ is a non-negative function that lies in ℬ(𝒲∪{∅},ℝ), in fact ‖ P(·|𝐬) ‖ = 1. Therefore, ‖·‖ induces a metric d on the moduli space ℳ

    d: ℳ×ℳ→ℝ.


The moduli space ℳ together with the metric d is the key object to understand a PLM.








§ THE ENTROPY FUNCTION
 


In this section, we introduce the concept of the entropy function and show how it quantifies the information contained in the output for a given prompt. We will study the zero-points, which are also the minimal points of the entropy function, and discuss their crucial properties. We argue that these zero points and the points where the entropy is close to 0 are the key obstacles for a LLM to approximate an intelligent language model. At last, we formulate a conjecture regarding AGI.




 §.§ Information Flow and Entropy


The entropy function for a CLM ℒ is defined just as in statistical mechanics and information theory <cit.>.


Given a prompt s ∈𝒮, the entropy of 𝐬 with respect to a CLM ℒ is given by

    S_ℒ(𝐬)= - ∑_𝐬' ∈𝒮 P_ℒ(𝐬'|𝐬) log P_ℒ(𝐬'|𝐬),

where the sum is over all possible outputs 𝐬'.

We will see that the entropy function is a key mathematical concept to analyze the behaviors of language models. From the definition, the entropy function is a non-negative function defined on 𝒮

    S_ℒ: 𝒮→ℝ.

As a special example, if the prompt is ∅, i.e., no prompt, the entropy associated with it has a special name which is stated as a definition.

The background entropy of a CLM ℒ is

    S_ℒ(∅) = - ∑_𝐬' ∈𝒮 P_ℒ(𝐬'|∅) log P_ℒ(𝐬'|∅).


The background entropy S_ℒ(∅) is a quantity that gives a zeroth order measurement about how complicated a language is. If S_ℒ(∅) is very large, this intuitively mean that the CLM is quite diverse. While if S_ℒ(∅) is very small, it intuitively means that the CLM is very rigid.


Since the development of language is for individuals to exchange information, so a language model itself must be able to demonstrate this. Here we argue that the definition of the entropy function exactly quantify the information during this process <cit.>. More precisely, given a prompt 𝐬, the entropy S_ℒ(𝐬) measures how vague are the answers to the prompt 𝐬 with respect to this CLM ℒ. If S(𝐬) is very large, this means that the answers can be very diverse. On the other hand, if S(𝐬) is very small, this means the answer to 𝐬 is very rigid. Therefore, after the CLM outputs 𝐬', we can say the information revealed by 𝐬' is S_ℒ(𝐬) <cit.>.


Just like statistical mechanics, we argue that the most important function for a CLM is the entropy function <cit.>.








 §.§ The Zero Points of the Entropy Function






Let us now look at the zero points of S_ℒ. If S_ℒ(𝐬)=0, then it means that the answer to the prompt 𝐬 is unique in the CLM ℒ, i.e., there exists a unique sentence 𝐬' ∈𝒮 such that P_ℒ(𝐬'|𝐬) = 1. Hence there is the interpretation that the CLM ℒ detects that the answer to the prompt 𝐬 is unique, and thus the CLM itself actually contain this information. For example, suppose the prompt 𝐬 is

    𝐬 =

If the entropy function of an intelligent CLM ℒ vanishes at 𝐬, i.e., S_ℒ(𝐬) = 0, then this means there is only one possible output of ℒ for the prompt 𝐬. Of course, an intelligent ℒ that knows about human history will output 

    𝐬' =

with probability 1. It should be noticed that at the point 𝐬, the vanishing of the entropy only implies there is only one possible output to 𝐬. Another CLM ℒ_1 could still have S_ℒ_1(𝐬)=0, but it instead outputs 

    𝐬” = .

We say the CLM ℒ_1 outputs a `wrong' answer because Gauss is a mathematician, so the information given by ℒ_1 is wrong. But `wrong' information is still information, the entropy function itself only quantifies the amount of information, but it cannot detect whether the information contained is right or wrong, which needs to be checked with what happens in the outside world. 

Given a CLM ℒ, a zero point 𝐬 of the entropy function S_ℒ is called a singularity of ℒ.

Hence the conclusion is that a CLM memorize information at its singularities. We argue that the existence of these singularities and the points where the entropy function is very small are the key obstacle for an LLM to approximate an intelligent CLM. 






 §.§ An AGI Conjecture
 


It is hard to define which CLM is Artificial General Intelligence (AGI) mathematically! Our idea is that if a CLM ℒ is an AGI, then it must satisfy certain mathematical properties. In this paper, we formulate a conjecture about this.








 
If a CLM is an AGI, then there exists a finite subset 𝒮_AGI⊂𝒮 such that the entropy function S_ℒ: 𝒮→ℝ is determined by its value on 𝒮_AGI.

The motivation of this conjecture comes from modern mathematics. In modern mathematics, we first give a finite set of definitions and axioms, and then mathematics is about proof, which uses these finite set of rules of prove a result that is not known a priori. The key motivation for Conjecture <ref> is that the value of S_ℒ on 𝒮 is determined using mathematical deductions by its value on the finite set 𝒮_AGI <cit.>. 




§ A THERMODYNAMIC INTERPRETATION OF LANGUAGE MODELS
 

In this section, we will give a thermodynamic interpretation of the CLM defined in Section <ref>. We show how the concepts and properties of the CLM naturally admit physical interpretations. Then we will define thermodynamic functions such as internal energy, Helmholtz free energy on CLM, and show how they help us understand phenomena in language models. At last, we give a physical interpretation to how a sentence “grows” in a PLM.




 §.§ The Boltzmann Distribution


First, let us introduce the Boltzman distribution in statistical mechanics <cit.>. Suppose we are given a physical system that can occupy different microstates, and suppose the energy of the microstate i is E_i. Let k be the Boltzmann constant and T be the temperature, then the probability that the system occupies  microstate i satisfies

    p_i ∝exp( - E_i/kT).

Let the partition function of the system be

    𝒵= ∑_iexp( - E_i/kT),

where the sum is over all possible microstates. Then p_i is just

    p_i = 1/𝒵·exp( - E_i/kT),

which is called the Boltzman distribution. The Boltzmann distribution is the distribution that maximizes the entropy

    S = - ∑_i p_i log p_i

subject to the constraint ∑ p_i =1 and ∑_i p_i E_i is a constant, i.e., the average energy is kept fixed.



 §.§ A Thermodynamic Interpretation
 

We now give a thermodynamic interpretation of CLM. 

In a CLM ℒ, a sentence is a microstate and for every sentence 𝐬∈𝒮, there is an energy E_ℒ(𝐬) associated with it.

The energy of ∅ is called the vacuum energy, and the energy of a word w ∈𝒲 is called its self-energy <cit.>.


As in physics, a word w ∈𝒲 itself can have self-interaction, it has an internal energy E(w), the meaning of this internal energy is that given a random sentence 𝐬, the probability that the first word of 𝐬 is w is proportional to exp(-β E(w) ). Here β is the inverse of temperature.


Given a length-2 sentence w_1w_2, its energy E_ℒ(w_1w_2) is the interaction between the two words w_1 and w_2, and E_ℒ(w_1w_2) is lower if the interaction between w_1 and w_2 is stronger. 

Since a language is essentially a discrete object, a word does not have a continuous degree of freedom. Therefore, there is no kinetic energy for a word, and only potential energy exists when words form a sentence. 

Here comes the definition of the statistical ensemble in a language model.

For a given prompt 𝐬, the statistical ensemble associated with it is 

    {𝐬𝐬'|𝐬' ∈𝒮}.



For later purpose, we introduce another quantity β which is the inverse of temperature <cit.>. For a fixed CLM, the value of β is usually fixed to be 1. 

In a CLM ℒ, for a given prompt 𝐬 the probability that ℒ outputs 𝐬' satisfies

    P_ℒ(𝐬'|𝐬) ∝exp(-β E_ℒ(𝐬𝐬') ),

where E_ℒ(𝐬𝐬') is the energy of the concatenation 𝐬𝐬'.   


More precisely, for a given prompt 𝐬, the probability that ℒ outputs 𝐬' is

    P_ℒ(𝐬'|𝐬) = 1/𝒵_ℒ·exp( - β E_ℒ(𝐬𝐬') ),

where E_ℒ(𝐬𝐬') is the energy of the concatenation 𝐬𝐬' and 𝒵_ℒ is the partition function associated with 𝐬

    𝒵_ℒ(𝐬) = ∑_𝐬' ∈𝒮exp( - β E_ℒ(𝐬𝐬')).

The partition function is in fact a thermodynamic function defined on 𝒮

    𝒵_ℒ:𝒮→ℝ.

 

The average energy of the statistical ensemble associated with 𝐬 is 

    U_ℒ (𝐬) = ∑_𝐬' ∈𝒮 P_ℒ(𝐬'|𝐬) E_ℒ(𝐬𝐬'),

which is called the internal energy of the ensemble. The internal energy U_ℒ is also a well-defined function on 𝒮

    U_ℒ : 𝒮→ℝ.

But notice that U_ℒ is different from E_ℒ: E_ℒ(𝐬) is the energy of the microstate 𝐬, while U_ℒ(𝐬) is the average energy of the statistical ensemble associated with 𝐬. Recall that the entropy for the statistical ensemble associated to 𝐬 is

    S_ℒ(𝐬)=-  ∑_𝐬' ∈𝒮 P_ℒ(𝐬'|𝐬) log  P_ℒ(𝐬'|𝐬) ,

which is simplified to

    S_ℒ = β U_ℒ + log𝒵_ℒ.

The Helmholtz free energy is given by

    F_ℒ = U_ℒ - 1/β S_ℒ=- 1/βlog𝒵_ℒ.

The Helmholtz free energy determines whether a sentence 𝐬 is likely to be followed by another sentence or not.







 §.§ A Physical Picture of How a Sentence Grows



Let us now interpret the process of predicting the next word in a PLM using how molecule grows in statistical mechanics. First, because of the equivalence between CLM and PLM, the interpretation in Section <ref> admits immediate generalization to PLM. In this physical picture, a word is view as an atom, and a sentence is a sequence of atoms, whose energy depends on the order of these atoms. Suppose we are given a prompt 𝐬∈𝒮_n, which is composed of n atoms. To predict the next word is to add an atom to 𝐬. For a word w ∈𝒲∪{∅}, the energy of 𝐬w is E(𝐬w), and the internal energy is by definition

    ∑_w ∈𝒲∪{∅} P(w|𝐬) E(𝐬w).

We also have the entropy

    - ∑_w ∈𝒲∪{∅} P(w|𝐬) log P(w|𝐬).

There are two trends that fight each other

    
  * the internal energy (Eq. (<ref>)) wants to be as small as possible,
    
  * the entropy (Eq. (<ref>)) wants to be as large as possible.

As a result, the Helmholtz free energy (Eq. (<ref>)) is minimized, and the resulting distribution is the Boltzmann distribution. After sampling the next word, say w_n+1, we can continue this process with the new sentence 𝐬w_n+1 and sample its next one. This process is highly similar to the process of how a molecule grow in statistical mechanics!


But there remains the question of how to compute the energy of a sentence. A sentence can be very long, hence the computation of its energy is a typical many-body problem. The power of LLM is to approximately compute the interaction energy between the “atoms” in a sentence using machine learning.




§ THE GEOMETRIZATION OF LANGUAGE MODELS
 

In this section, we will introduce the concept of the geometrization of a PLM, which lays the foundation for studying language models using differential geometry. We will also define a more general version of the Word-Embedding, which leads to the concept of a Boltzmann manifold.





 §.§ The Geometrization of PLM


Suppose we have a PLM ℒ_P, whose moduli space of distributions is ℳ (Definition <ref>). Moreover, we have a map

    ρ: 𝒮→ℳ,

where ρ sends a sentence 𝐬∈𝒮 to the distribution P(·| 𝐬) ∈ℳ. Notice that ℳ is a metric space, the details of which can be found in Section <ref>.

Generally, the moduli space ℳ itself is extremely complicated, to study which we need to borrow ideas from both mathematics and physics. The first step is to geometrize this complicated metric space, which “prompts” the following definition.


The ϵ-geometrization of a PLM is a manifold M and a continuous map

    Ψ: ℳ→ M,

where each point of M represents a probability distribution on 𝒲∪∅ that is also computable. Moreover, for every point P(·| 𝐬) of ℳ, the norm of the difference ‖Ψ(P(·| 𝐬))-P(·| 𝐬) ‖ < ϵ(n). Here, the error ϵ(n) can depend on the length of the sentence 𝐬∈𝒮_n.

Notice that the norm ‖·‖ is defined in the metric space ℬ(𝒲∪{∅},ℝ) (Eq. (<ref>)). The composition Ψ∘ρ is a map from 𝒮 to M

    Ψ∘ρ: 𝒮→ M.

The motivation of the geometrization of the moduli space ℳ is that we try to find a finite dimensional manifold M to approximately describe the infinite set ℳ. Then we use the geometry of this manifold to study the bizarre metric space ℳ.




 §.§ The Word-Embedding and the Boltzmann Manifold
 

One way to construct an ϵ-geometrization is via an additional structure on a manifold called the pairing. Suppose M is a manifold, a symmetric pairing is a smooth function 

    ⟨·, ·⟩ : M × M →ℝ,

which maps any two points pt_1, pt_2 ∈ M to a number in ℝ. Here “symmetric” means

    ⟨pt_1, pt_2 ⟩ = ⟨pt_2, pt_1 ⟩.

A Word-Embedding is an injective map

    ι: 𝒲∪{∅}→ M.

Given an arbitrary point pt∈ M, it defines a distribution P_pt on 𝒲∪{∅} via

    P_pt(w) =  exp( - β⟨ pt,ι(w) ⟩)/∑_w' ∈𝒲∪∅exp( - β⟨ pt,ι(w') ⟩),

which is exactly from the Boltzmann distribution <cit.>. Hence every point of M defines a distribution on 𝒲∪{∅} and M is a space of distributions on 𝒲∪{∅}.


The manifold M together with the pairing ⟨·,·⟩ and the Word-Embedding ι is called a Boltzmann manifold. 


Since every point of the Boltzmann manifold M defines a distribution on 𝒲∪{∅}, M is a subspace of  ℬ(𝒲∪{∅},ℝ) (Eq. (<ref>)). Therefore, given two points pt and pt' of M, their distance is

    d(pt,pt') =   ‖ P_pt-P_pt'‖.

This distance function induces a metric g_B on M, which is called the Boltzmann metric, hence M is in fact a Riemannian manifold <cit.>.

Now suppose M gives us an ϵ-geometrization with the Moduli-Embedding Ψ: ℳ→ M. Since 𝒮_n is the direct product 𝒲^n, thus if we believe the map Ψ∘ρ has sufficiently nice properties, then there exists a diagram

    𝒮_n = 𝒲^n @>ι^n>> M^n
    
    @VVρV @VVΛ_nV
    ℳ @>Ψ>> M
      ,

which is commutative up to a small error that depends on n. More precisely, we assume there exists a smooth map Λ_n:M^n → M such that for every 𝐬=w_1w_2⋯ w_n, we have

    Ψ∘ρ (𝐬) = Λ_n( ι(w_1), ⋯, ι(w_n) ).

It means that the map ρ∘Ψ is determined by the map ι and {Λ_n:n ≥ 2 }. The point is that the map Ψ∘ρ is defined on the infinite set 𝒮, which can be highly complicated. While the map ι is defined on a finite set and Λ_n is defined on a smooth manifold, hence can be studied by the modern theory of differential geometry <cit.>.









§ LLM AS A SPECIAL GEOMETRIZATION
 

In this section, we show that the current LLMs based on the transformer architecture are special cases of geometrization in the sense of Section <ref>. We also introduce the fundamental thermodynamic relation of LLMs.




 §.§ The Geometrization and LLM



The Boltzmann manifold used in LLMs is the simplest one: ℝ^N together with the standard inner product ⟨· , ·⟩. In practice, first we obtain a Word-Embedding  <cit.>

    ι: 𝒲∪{∅}→ℝ^N.

Then a point 𝐯 of ℝ^N defines a distribution on 𝒲∪{∅} via 

    P_𝐯(w) = exp(-β⟨𝐯, ι(w)⟩)/∑_w' ∈𝒲∪{∅}exp(-β⟨𝐯, ι(w')⟩),

where β is the inverse of temperature. The Moduli-Embedding is a map

    Φ: ℳ→ℝ^N.

Given a sentence 𝐬∈𝒮, Ψ∘ρ(𝐬) is a vector of ℝ^N, which gives the distribution of the next word of 𝐬 via Eq. (<ref>), i.e.,

    P(w|𝐬) = exp(-β⟨Ψ∘ρ(𝐬), ι(w)⟩)/∑_w' ∈𝒲∪{∅}exp(-β⟨Ψ∘ρ(𝐬), ι(w')⟩).




For every n ≥ 1, there exists a map Λ_n: (ℝ^N)^n →ℝ^N that makes the following diagram commutative up to a small error

    𝒮_n = 𝒲^n @>ι^n>> (ℝ^N)^n
    
    @VVρV @VVΛ_nV
    ℳ @>Ψ>> ℝ^N
    .


Namely, given a sentence 𝐬=w_1⋯ w_n, the distribution of the word after 𝐬 is given by the vector Λ_n(ι(w_1),⋯,ι(w_n)) via Eq. (<ref>). In practice, both ι and Λ_n are determined by data and machine learning <cit.>. Currently, the most popular method to construct the map Λ_n is based on the transformer architecture <cit.>.





 §.§ The Fundamental Thermodynamic Relation of LLMs


Given a geometrization (ℝ^N, ⟨·,·⟩, ι), the properties of w ∈𝒲∪{∅} with respect to this geometrization are determined by the embedding vector ϕ(w) ∈ℝ^N. Borrowing language from physics <cit.>, the space ℝ^N is also called the phase space of a language model. Just as in CLM, in a PLM we also have the entropy function S, the internal energy U and Helmholtz free energy F defined on 𝒮.



Suppose we are given a geometrization (ℝ^N, ⟨·,·⟩, ι) and the maps {Λ_n| n≥ 1 } associated with a PLM. The finite set 𝒮_n is viewed as a subspace of (ℝ^N)^n via the embedding ι^n. Then all thermodynamic functions such as S, U and F defined over 𝒮_n admit smooth extensions to functions over (ℝ^N)^n up to small errors.

Let us now look at the entropy function S as an example. This conjecture implies that there exists a differentiable function S: (ℝ^N)^n →ℝ that makes the following diagram commutative up to a small error

    𝒮_n = 𝒲^n @>ι^n>> (ℝ^N)^n
    
    @VVρV @VVSV
    ℳ @>S>> ℝ.




In fact, from the construction of LLMs using the architecture of transformer, the extensions of thermodynamic functions always exist, but the error cannot be guaranteed to be small.


Suppose the coordinate of (ℝ^N)^n is x_i,j with 1 ≤ i ≤ N and 1 ≤ j ≤ n, then we have the fundamental thermodynamic relation of LLMs

    dU = 1/β dS -∑_i,jf_i,jdx_i,j,

where f_i,j is the generalized force given by

    f_i,j=∂S/∂ x_i,j.




As a function on (ℝ^N)^n, we conjecture that the entropy function S for a LLM is convex. 

Suppose 0 ≤λ≤ 1, given two points 𝐱=(x_i,j) and 𝐱'=(x'_i,j) of (ℝ^N)^n, we have

    S(λ𝐱 + (1 - λ) 𝐱' ) ≥λS( 𝐱) + (1 - λ ) S(𝐱' ).


It has an intuitive interpretation, if you mix the two vectors 𝐱=(x_i,j) and 𝐱'=(x'_i,j), the entropy is always greater than the average. Namely, if you mixed the meanings, it always becomes more vague. 








§ CONCLUSION AND OPEN PROBLEMS
 

In this paper, we have used set theory, analysis and probability theory to rigorously give the definitions of Causal Language Model (CLM) and Predicative Language Model (PLM). We propose a generalization of the distributional hypothesis. We introduce the concept of the moduli space of distributions for a CLM, which is the key metric space to understand the power and properties of a language model. Then we define the entropy function for a CLM, study its properties, and show how its zero points are crucial for a language model to incorporate information and logic. Using the entropy function, we formulate an extension conjecture about AGI.

We have used thermodynamics to show that language models admit thermodynamic interpretations. Namely, a sentence is a microstate and a prompt gives us a statistical ensemble, which is distributed according to the Boltzmann distributions. We also formulate the concept of the geometrization of language models and define the Boltzmann manifold. We show the embedding of the moduli space into a Boltzmann manifold determines how accurate can a geometrization be. At last, we discuss how the current LLMs can be viewed as special cases of geometrization with respect to the linear space together with its inner product.

There are many interesting open questions that deserve further study. Perhaps the most straightforward question is what is the best Boltzmann manifold for the geometrization of a language model. Current LLMs all uses the linear space together with its inner product. But it is likely that different languages would need different Boltzmann manifold for the geometrization. For example, some of the possible choices of Boltzmann manifold with non-trivial topology can be

    ℝ^N × S^M, ℝ^N × (S^1)^M,

where S^k is a k-dimensional sphere. For such manifold, it is very interesting to see whether the non-trivial topology has any effect on the properties of the resulting LLM.

Another important question is whether results in modern physics and mathematics can be applied to study language models. For example, whether statistical physics can help us understand the emergent abilities of LLMs. Whether the results in statistical mechanics and differential geometry can offer a better architecture than the transformer. There are numerous results in the study of many-body interaction systems, it is interesting to see whether these results can have applications to LLMs.







plain





 Paper Title: SentenceVAE: Faster, Longer and More Accurate Inference with Next-sentence Prediction for Large Language Models.






Scaling Backwards:
Minimal Synthetic Pre-training?

    Ryo Nakamura1,*
0009-0009-0858-1229 
Ryu Tadokoro2,*
0009-0001-9473-3832
Ryosuke Yamada1
0000-0002-2154-8230


Yuki M. Asano 30000-0002-8533-4020
Iro Laina40000-0001-8857-7709
Christian Rupprecht40000-0003-3994-8045
Nakamasa Inoue50000-0002-9761-4142
Rio Yokota50000-0001-7573-7873
Hirokatsu Kataoka10000-0001-8844-165X

    August 5, 2024
====================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Contemporary large language models (LLMs) primarily rely on next-token prediction method for inference, which significantly impedes their processing speed. In this paper, we introduce a novel inference methodology termed next-sentence prediction, aimed at enhancing the inference efficiency of LLMs. We present Sentence Variational Autoencoder (SentenceVAE), a tiny model consisting of a Sentence Encoder and a Sentence Decoder. The encoder effectively condenses the information within a sentence into a singular token, while the decoder reconstructs this compressed data back into its original sentential form. By integrating SentenceVAE into the input and output layers of LLMs, we develop Sentence-level LLMs (SLLMs) that employ a sentence-by-sentence inference approach, markedly accelerating inference speeds. SentenceVAE also maintains the integrity of the original semantic content by segmenting the text into sentences, thereby improving accuracy while boosting inference speeds. Compared to published LLMs, SLLMs process fewer tokens over equivalent context lengths, significantly reducing memory demands for self-attention computations and facilitating the handling of longer contexts. Our experimental findings reveal that this method can accelerate inference speeds by 204∼365%, reduce perplexity (PPL) to 46∼75% of its original metric, and decrease memory overhead by 86∼91% for the same context length, compared to the token-by-token method. Moreover, the benefits of this approach become even more pronounced as model parameters increase.












§ INTRODUCTION




Large language models (LLMs) have demonstrated remarkable proficiency in understanding human intent through the acquisition of knowledge from vast and diverse datasets. These models provide rich and accurate responses, making them essential for applications such as machine translation <cit.>, chatbots <cit.>, and question-answering systems <cit.>. The predominant method to inference in LLMs involves next-token prediction, where the model generates tokens sequentially. While this token-by-token generation method has achieved significant performance, it is inherently limited in producing only one token per inference step. This limitation results in substantial computational overhead and increases the overall inference time, hindering the scalability and efficiency of LLMs.  Consequently, there is a critical need to explore alternative methods that can mitigate these limitations while preserving or enhancing the accuracy and responsiveness of LLMs.

To enhance the inference efficiency of large language models (LLMs), Gloeckle et al. <cit.> proposed training LLMs to predict multiple tokens simultaneously, aiming to accelerate the inference process. However, the number of tokens predicted simultaneously is fixed, resulting in a rigid and inflexible partitioning of the input text. For example, when set to five, the input is invariably segmented into non-overlapping sequences of five tokens, regardless of the text's underlying structure, potentially compromising inference accuracy.

To enhance inference speed while preserving or enhancing accuracy, we explore methods for adaptively selecting the optimal number of predicted tokens based on the semantic content of the text. Consequently, we introduce Sentence Variational Autoencoder (SentenceVAE), which comprises a sentence encoder and a sentence decoder. The schematic of SentenceVAE is illustrated in Fig. <ref>. The sentence encoder is designed to condense the information of an entire sentence into a single token, while the sentence decoder reconstructs this compressed token back into a sentence. By integrating the encoder and decoder of SentenceVAE into the input and output layers of LLMs, these models can perform next-sentence prediction, thereby accelerating the reasoning process. Moreover, since SentenceVAE segments text at the sentence level, the semantic integrity is better preserved, ensuring the accuracy of the inferrence. Specifically, prior to inputting text into LLMs, SentenceVAE segments the text into sentences and utilizes the sentence encoder to compress the information contained within each sentence into a single token. These compressed tokens are then fed as input to the LLMs. The LLMs predict their output based on these compressed tokens. Finally, the sentence decoder decodes the predicted token into the final output of the LLMs.

The contribution of the proposed method can be described as follows:



  * We propose SentenceVAE, a model capable of compressing the information content of a sentence into a single token, and conversely, expanding such tokens to reconstruct sentences with high informational fidelity.

  * We embed the encoder and decoder components of SentenceVAE into the input and output layers of LLMs, enabling these models to implement an effective inference method for next-sentence prediction.

  * Extensive experiments conducted on various LLMs have demonstrated that the proposed method significantly accelerates inference speed while preserving the accuracy of these models. 






§ RELATED WORK


Large language models: in recent years, LLMs have achieved remarkable performance in the field of natural language processing (NLP). However, these models typically possess a vast number of parameters from billions to hundreds of billions, exemplified by models such as TeleChat (1B-52B) from TeleAI <cit.>, Llama (7B-405B) from Meta <cit.>, InternLM (1.8B-20B) from Shanghai AI Lab <cit.>, etc. The sheer magnitude of these parameters renders the inference process computationally intensive and time-consuming. Moreover, most LLMs employ a one-token prediction method, where the model predicts only one token at a time. This token-by-token approach further exacerbates the inference time of LLMs. As a result, accelerating the inference speed of LLMs while preserving or enhancing their accuracy presents a significant challenge for researchers.

Multi token prediction: The transformer model enables the parallel processing of sequences through its self-attention mechanism, providing a technical foundation for the implementation of multi-token prediction <cit.>. The BERT model utilizes bidirectional encoder representations to capture contextual information, further solidifying the basis for the implementation of multi-token prediction <cit.>. The T5 model incorporates multi-token prediction during the training process for the first time, significantly enhancing the coherence and quality of the generated text <cit.>. The Better & Faster LLMs apply multi-token prediction to LLMs, allowing them to directly predict multiple tokens in a single inference step, thereby reducing the number of inference iterations and associated computational costs <cit.>. However, the number of predicted tokens is fixed, which may inadvertently combine unrelated tokens. This can result in the merging of unrelated tokens, disrupting the original sentence structure and consequently reducing inference accuracy.

Encoder-decoder model: The concept of an encoder is introduced by Yann LeCun in his doctoral dissertation, wherein he posited that an encoder could transform input data into shorter representations without losing the core information, thereby providing a foundational approach to compressing high-dimensional data. The Variational Autoencoder (VAE) model introduces the use of hidden variables between the encoder and decoder to learn the data distribution, laying the groundwork for deep-learning-based encoders and decoders <cit.>. Sutskever et al. introduces the encoder-decoder framework into the field of deep learning, utilizing recurrent neural networks (RNNs) as both the encoder and decoder for sequence-to-sequence (Seq2Seq) learning, demonstrating exceptional performance in machine translation tasks <cit.>.
The SegNet applies the encoder-decoder model to the field of image segmentation, significantly enhancing segmentation performance <cit.>. Subsequently, the encoder-decoder model has achieved great success in various domains including machine translation <cit.>, text-to-speech conversion <cit.>, computational imaging <cit.>, and beyond, fully demonstrating the capability of the encoder-decoder architecture to efficiently compress and restore data.

By leveraging the advantages of multi-token prediction and the encoder-decoder model, we propose SentenceVAE. SentenceVAE is capable of compressing the information contained within a sentence into a single token, enabling next-sentence prediction and significantly accelerating the inference speed of LLMs.  Additionally, our method divides the text into sentences without compromising the original semantics, accelerating the inference speed of LLMs while ensuring inference accuracy.




§ METHOD


In this section, we introduce the SentenceVAE framework. Embedding this framework into a LLM enables the model to implement a sentence-by-sentence prediction method, significantly accelerating its inference speed. The inference process of the proposed method compared to published LLMs is illustrated in Fig. <ref>. It is evident that for the same text, an LLM embedded with SentenceVAE requires only three inference steps to obtain the inference result, whereas traditional LLMs necessitate twenty-free inference steps. Furthermore, the proposed method divides the text into sentences without compromising the original semantics, thereby enhancing the inference speed of LLMs while preserving or enhancing accuracy.



 §.§ Sentence Variational Autoencoder (SentenceVAE)


Our SentenceVAE primarily consists of a Sentence Encoder and a Sentence Decoder. The ecoder encodes multiple word-level tokens from a sentence into a single sentence-level token, while the decoder reconstructs this sentence-level token back into the original sequence of word-level tokens. To ensure that the multiple tokens input to the encoder come from individual sentences, we propose a specialized sentence segmentation mechanism. Additionally, we introduce a feature fusion mechanism, which enable the encoder to encode variable-length token sequences into a single sentence-level token.

Sentence Segmentation Mechanism: To ensure that all word-level tokens input to the sentence encoder per time come from a single sentence, we use regular matching to split sentences by punctuation marks such as , , ,  before tokenizing. Let the original string be S, and after partitioning, the set {s_i|1≤ i ≤ n} is obtained, satisfying S=s_1 s_2 s_3 ... s_n.

Sentence Encoder: Taking s = s_t (1≤ t ≤ n) as an example, after tokenization, we obtained a sequence of L token ids D=[d_1, d_2, d_3, ..., d_L] representing a sentence. This sequence was then passed through an embedding layer, resulting in word-level embedding vectors of E, as shown in Eq.<ref>. 


    E = (e_i)_L×hidden_size = Embed(D)


These embeddings were subsequently input into self-attention based encoder blocks, yielding hidden features H, as shown in Eq. <ref>. 


    H = (h_i)_L×hidden_size = EncoderBlocks(E)


To derive the sentence embedding vector, we fused these L features into a single vector Ω_t ∈ℝ^hidden_size. This sentence embedding vector is then fed into the decoder-only LLM, which generates a new sentence embedding vector Ω_t+1 at each time step.

Feature Fusion Mechanism: After the encoder blocks generates H, we propose a method to fuse them into a single sentence embedding vector Ω_t. This method involves accumulating the L vectors and normalizing them using Layer Normalization <cit.>, as shown in Eq. <ref>. 


    Ω_t = LayerNorm(∑_i=1^Lh_i)


Sentence Decoder: The Sentence Decoder contains a combination of masked self-attention and cross-attention blocks, where the cross-attention mechanism uses the individual sentence embedding vector Ω_t+1 as key (K) and value (V).

In the prediction phase, given Ω_t+1, initialize the input token d_0=<bos>, the decoder outputs d_1, and then input [d_0, d_1] and Ω_t+1 into the decoder. Repeat this process until d_l'+1=<eos>, and terminate the iteration, as shown in Eq.<ref>.


    init:     d_0= <bos>, l' = 0 
    do:    
       D_l' = [d_0, d_1, d_2, ..., d_l'] 
       P_l' = (p_i)_(l'+1) ×hidden_size
       = DecoderBlocks(D_l', Ω_t+1) 
        d_l' = argmax(p_l') 
       D_l'+1 = concat(D_l', d_l') 
        l' = l' + 1 
    until:     d_l' = <eos> 
    ret:    D_L'=l'


After the iteration is completed, the detokenization process is used to convert the output D_L' back into a string.

In the training phase, parallel training is adopted, based on masked self-attention mechanism, to ensure that only tokens before d_l' can be seen when inferring d_l'+1. Given an input sequence of D_train=[d_0, d_1, ..., d_L'], model output logits of P∈ℝ^(L'+1) ×hidden_size, and ground truth of D_groud_truth=[d_1, d_2, ..., d_L', <eos>], focal loss <cit.> is taken as the loss function, as shown in Eq. <ref>.


    P   = DecoderBlocks(D_train, Ω_t+1) 
    P'    = softmax(P) 
    Loss   = ∑_i=1^L'-(1-p_d_i)^γlog (p_d_i), γ=2




 §.§ Sentence-level Large Language Models (SLLMs)


Currently, almost all decoder-only LLMs consist of the following components: an embedding layer EMB_llm, N decoder-only blocks DB_llm, and a fully connected layer FC_llm for outputting logits. As shown in Eq. <ref>, given context tokens D_context=[d_0, d_1, ..., d_l'], the model generates the next token d_l'+1.


    E_context   = EMB_llm(D_context) 
    H_llm   = DB_llm(E_context) 
    P_llm   = (p_llm,i)_(l'+1) ×hidden_size
        = FC_llm(H_llm) 
    
            d_l'+1   = argmax(p_llm,l')


In this section, we introduce a unified grafting method to graft SentienceVAE onto the beginning and end of any LLM. This approach transforms the LLMs into SLLMs, enabling it to operate effectively in the sentence-level embedding space.

Discard Embedding Layer: In a SLLM, the N decoder-only blocks no longer receives word-level tokens directly. Instead, it receives sentence embedding vectors encoded by the sentence encoder. Consequently, the traditional embedding layer in the LLM architecture is removed. Therefore, the model follows Eq. <ref>.


    E_sllm   =  [Ω_1, Ω_2, ..., Ω_t] 
    H_sllm   = (h_sllm,i)_t ×hidden_size
       = DB_llm(E_sllm)


Termination judgment layer: In traditional LLMs, the final fully connected layer FC_llm typically converts an H_llm∈ℝ^t ×hidden_size to a probability vector (logits) P_llm∈ℝ^t × V, where V represents the size of the vocabulary. This allows the model to determine the next token output using sampling algorithms, e.g., Greedy Search,  Beam Search <cit.>, Random Sampling, Temperature Sampling <cit.>, Top-K Sampling <cit.>, and Nucleus Sampling <cit.>. If the current token output is a special token like eos (end-of-sequence), the iteration terminates. However, in SLLMs, the output of DB_llm is a sentence-level hidden state vector H_sllm. Therefore, traditional token generation methods are not applicable. Instead, we use a new fully connected layer FC_sllm called termination judgment layer that converts the H_sllm into a 2-dimensional boolean vector B_stop_flag, as shown in Eq. <ref>. This vector helps determine whether the current sentence-level hidden state vector signifies the end of a sentence (stop flag) or needs further decoding by the sentence decoder. If the vector indicates an end flag, the iteration terminates. Otherwise, the sentence decoder processes the embedding to generate corresponding tokens, as shown in Eq. <ref>.


    B_stop_flag   = (b_i)_t × 2
       = FC_sllm(H_sllm)



    Ω_t+1 = 
        
            <eos>    , argmax(b_t) = 1 
    h_sllm, t   , argmax(b_t) = 0


During the training phase, calculate the focal loss for B_stop_flag as part of the global loss.

Inferencing sentence by sentence:  In the SLLM architecture, we use a tiny SentenceVAE model to encode multiple tokens within a sentence into a single sentence embedding vector. This enables the LLM to conduct inference on a sentence-by-sentence basis rather than the traditional token-by-token method.



§ EXPERIMENT


To validate our idea, we first trained individual SentienceVAEs using self supervised methods, demonstrating that multiple tokens in a sentence can be compressed into a single sentence vector by an encoder and restored by a decoder. Subsequently, we grafted the encoder and decoder at the beginning and end of the open-source LLMs. By such simple modifications, LLMs can be upgraded to SLLMs and work in sentence embedding space, while preserving PPL and improving inference speed. Meanwhile, by observing the loss curve, we found that SLLMs still follows the Scaling Law. 



 §.§ Experimental Setting


The experiments described in this paper were conducted using either a single 4-card RTX 4090 (24G) or a 4-card A100 (40G) (for SLLM-1.3B) setup, with training carried out in a data parallel distribution.

Base Models: We use the 125M, 350M, and 1.3B models of the OPT series <cit.> as the base LLMs, and demonstrate the feasibility of extending to larger scale models through Scaling Law <cit.>.

Dataset: We use the English subset (EN/WebText) of the Wanjuan-1.0 dataset <cit.> as the training dataset. The number of training samples depends on the number of iterations, with SentenceVAE training samples consisting of approximately 153.6M sentences (approximately 1.7B tokens); The training sample size of SLLM is about 6.4M paragraphs (approximately 5.6B tokens). The validation set consists of 1,000 random sentences or paragraphs that are mutually exclusive with the training set. 




Hyperparameters: When training SentenceVAE, we assume each sentence has a maximum of 64 tokens, with a batch size of 128/card, a base learning rate of 1e-7/batch/card, and a maximum of 300K iterations. 

For training SLLM, we assume each sentence has a maximum of 64 tokens, each paragraph has a maximum of 64 sentences (due to memory limitations), a batch size of 1/card, a base learning rate of 1e-6/batch/card, and a maximum of 1.6M iterations.

All experiments were conducted using the AdamW optimizer <cit.> with AMP enabled <cit.>, a weight decay coefficient of 0.01, and a maximum L2 norm of 1 for gradient clipping.

The first 5K iterations utilized a linear learning rate schedule, starting at 1/1000 of the base learning rate. After 5K iterations, a cosine annealing learning rate schedule was employed with a period of 20K iterations. Additionally, all experiments implemented the Exponential Moving Average (EMA) strategy.

Metrics: We use PPL as an evaluation metric. The formula for calculating PPL for output logits P of Sentence Decoder and ground truth D_groud_truth is as Eq. <ref>.


    PPL = exp( 1/L'∑_i=1^L'-log(p_d_i))




 §.§ A series of word-level tokens can be represented by a sentence-level embedding with strong robustness


In this experiment, we trained individual SentienceVAEs using a self-supervised approach. If the Sentence Encoder can compress multiple tokens from the input sentence into a single sentence embedding vector and the Sentence Decoder can use this vector to reconstruct the original token sequence, it would validate the feasibility of our theory.

We trained SentenceVAEs with hidden sizes of 768, 1024, and 2048, corresponding to OPT-125M, OPT-350M, and OPT-1.3B, respectively. For each hidden size, we varied the number of block layers to 1, 2, and 4. We evaluated the models using cross-entropy loss and perplexity (PPL) metrics on the validation set. The experimental results are presented in Table <ref>.







The evaluation results demonstrate that SentenceVAEs fulfill their intended purpose, confirming the theoretical validity. Table <ref> presents several test cases. Among these, Samples 1, 2, 3, 5, 6 are convention samples. To assess the model's robustness, we also included several out-of-distribution samples, highlighted in blue in the table:


    
  * Sample 4, 7:   Data preprocessing involves segmenting sentences based on punctuation marks, such as  and . Consequently, these samples will be split into multiple inputs during both training and practical application. Directly inputting these samples without preprocessing is considered invalid input for the model.
    
  * Sample 8:  This is the pinyin of a famous Chinese saying, not English, so it is illegal input.


From the output results of the model, we can analyze that:


    
  * In the convention samples, except for Sample 6, the model accurately restored the sentences. The reason for the error in Sample 6 may be related to the bottleneck of the sinusoidal position encoding mechanism <cit.>.
    
  * Although Sample 4 contains the illegal input, the model has accurately restored it with only punctuation marks. Sample 7 still maintains the meaning of the sentence. In Sample 8, as the hidden size of the model increases, the model also accurately restores the illegal input “yao yao ling xian”. These phenomena indicate that the model is robust to illegal inputs.




 §.§ LLMs can work in setence-level embedding space with faster inference speed, more accurate PPL, and longer context, resulting SLLMs


After verifying the feasibility of Sentence level embedding, we grafted the encoder and decoder of SentenceVAE onto the beginning and end of the OPT series models, corresponding to the relationship shown in Table <ref>.





We did not test SLLM-1.3B-H4 due to memory overflow issues encountered during data-parallel distributed training. The benchmark test results of OPTs and SLLMs are shown in Table <ref>.

Faster inference speed:  We assessed the throughput rate of SLLMs using the PyTorch framework <cit.>, without incorporating optimization methods such as PagedAttention <cit.> or acceleration engines like TensorRT-LLM or LMDeploy <cit.>. The throughput rate of the original OPT model served as the baseline for comparison. The experimental results show that SLLMs is 2∼3× faster on average than LLMs. The larger the number of model parameters, the smaller the proportion of inferencing cost occupied by SentenceVAE, and the more obvious the speed benefit.

More accurate PPL:  We observed that the PPL metric for SLLMs was better than that of the baseline OPT models. This improvement is likely due to the SLLM paradigm enabling the model to process natural language at a higher sentence level, leading to enhanced performance. Additionally, the attention computation in the sentence embedding space allows SLLMs to manage shorter relative contexts for texts of the same length, which further contributes to the improved model performance.

Longer context length:  The context length that a model can handle is primarily constrained by GPU memory. Under the SLLM framework, multiple original tokens can be compressed into a single token, allowing the model to use less memory when processing contexts of the same length. This effectively increases the context length that the model can handle given the same hardware resources.



 §.§ SLLMs still follow the Scaling Law


By analyzing the loss curve of SLLM during the training process, we observed that it adheres to the Scaling Law <cit.>, as shown in Fig. <ref>. This suggests that the SLLM framework can be effectively extended to larger-scale language models through scaling.



§ CONCLUSION & FUTURE TRENDS


In this section, we summarize our work and suggest several potential directions for future development.



 §.§ Scaling up SLLMs with better structures


In this work, to rapidly validate our ideas, we utilized the classic Transformer architecture <cit.> implemented in the PyTorch framework <cit.>. Due to constraints in computing power and time, we limited our validation to models ranging from 125M to 1.3B parameters. However, by verifying the Scaling Law <cit.>, we inferred the applicability of our approach to larger scale models. Numerous state-of-the-art architectural optimizations can be integrated into the SLLM framework, such as Rotational Position Encoding (RoPE) <cit.>, which may address the order problem observed in Sample 6. Currently, our models are trained solely on English corpora, but there is potential to add multilingual support in the future. Overall, while the SLLM framework is still in its early stages, it holds significant potential for further optimization.



 §.§ SLLMs assist in hybrid edge-cloud inference
 

The SLLM framework is inherently a hybrid architecture combining small and large models. By identifying the optimal balance point, the small model (SentenceVAE) and the large model (LLM) can be effectively distributed between the edge and cloud, balancing computational load and enhancing user experience.



 §.§ SLLMs assist in embodied intelligence
 

At present, embodied intelligence based on the LLM Agent paradigm struggles to interact directly with underlying hardware. Instead, they rely on intermediary components to translate high-level LLM instructions into traditional real-time control logic. The primary limitation is the insufficient rate of “token” generation on the edge side.  In contrast, SLLM's capacity to process and generate more “tokens” within the same computational and temporal constraints, offering the potential for embodied intelligent large models to interface directly with underlying hardware.



 §.§ SLLMs assist in multimodal large model
 

The “frames” in videos and “trunks” in audio are, to some extent, equivalent to “tokens” in language models. Therefore, the SLLM paradigm can similarly be applied to multimodal large models to increase the processing “frame rate”, enhance user experience, and  potentially attain performance on par with or exceeding.



§ ACKNOWLEDGMENTS


This research was supported by the China National Key R&D Program (2022YFC2808003) and the Natural Science Basic Research Program of Shaanxi (2024JC-YBMS-468).





 Paper Title: Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generation.




Methodological Foundations of Modern Causal Inference in Social Science Research
    Guanghui Pan[Department of Sociology, University of Oxford. 41 Park End Street, Oxford, the United Kingdom, OX1 1HH. Email: guanghui.pan@sociology.ox.ac.ukguanghui.pan@sociology.ox.ac.uk. Manuscript in progress, do not cite or redistribute without permission.]
    August 5, 2024
========================================================================================================================================================================================================================================================================



§ ABSTRACT 
Symbolic Music, akin to language, can be encoded in discrete symbols. Recent research has extended the application of large language models (LLMs) such as GPT-4 and Llama2 to the symbolic music domain including understanding and generation. Yet scant research explores the details of how these LLMs perform on advanced music understanding and conditioned generation, especially from the multi-step reasoning perspective, which is a critical aspect in the conditioned, editable, and interactive human-computer co-creation process. 
This study conducts a thorough investigation of LLMs' capability and limitations in symbolic music processing. We identify that current LLMs exhibit poor performance in song-level multi-step music reasoning, and typically fail to leverage learned music knowledge when addressing complex musical tasks. An analysis of LLMs' responses highlights distinctly their pros and cons.
Our findings suggest achieving advanced musical capability is not intrinsically obtained by LLMs, and future research should focus more on bridging the gap between music knowledge and reasoning, to improve the co-creation experience for musicians.




§ INTRODUCTION












Large language models (LLMs), such as GPT-4, harness the power of deep learning to produce human-like text. These models, trained on vast datasets of textual content, have notably propelled advancements in natural language processing (NLP). They excel in complex language understanding and generation tasks including translation, sentiment analysis, question answering, and summarization, showcasing their reasoning capability with sophistication.

Large language models (LLMs), initially pre-trained on extensive textual corpora, can assimilate general linguistic patterns and structures. They are subsequently fine-tuned with domain-specific data, such as code and mathematical symbols, to enhance the adaptation to specific tasks. This refinement allows LLMs' proficiency to more accurately manage domain-specific terminology and complicated challenges like multi-step reasoning. Music Reasoning refers to the ability to estimate the varying harmonies, keys, rhythms, and other musical elements that are not explicitly annotated in a piece of music and are significant for music themes, progression, and styles <cit.>. The analogy between the reasoning process in music and mathematics suggests their structural similarities. Both disciplines fundamentally rely on patterns: music in rhythms, scales, and chord progressions, while mathematics involves sequences, symmetries, and geometric configuration. Moreover, music theory utilizes mathematical concepts to articulate intervals between pitches, chord structures, and the rhythmic temporal division  <cit.>, underscoring the intrinsic reasoning nature of the musical components. 

Music can be represented as sequences of symbols such as MIDI or ABC notation, rendering it suitable for processing by LLMs, which excel in long-context understanding and multi-step reasoning. These models are capable to dissect and generate intricate musical patterns encompassing melodic, harmonic, and rhythmic structures. LLMs also play a pivotal role in enhancing interactive music generation systems, where user inputs tailor the model's output, enriching the composing experience. While previous studies <cit.> have investigated LLMs in music tasks, detailed interpretations of the process remains less explored. This paper conduct an evaluation of four LLMs, GPT-4 <cit.>, Gemma-7B-it <cit.>, Llama2-7B-chat <cit.>, and Qwen-7B-chat <cit.>, assessing their capabilities on tasks related to symbolic music understanding and generation:


    
  * Music Understanding: 1) Music theory exercise; 2) Motif extraction; 3) Musical form extraction.
    
    
    
    
    
    
  * Music Generation: 1) Chord-conditioned music generation; 2) Melody harmonization; 3) Musical-form-and-motif-conditioned music generation
    
    
    
    
    
    


The task of "Musical Form & Motif Conditioned Music Generation" as described in Chatmusician <cit.> involves generating music that adheres to detailed prescribed conditions like form and motif. Figure <ref> illustrates this process: The prompt's green text specifies conditional constraints including the musical form, motif, and some musical elements (key, time signature, etc.). Under the prompt, the left sheet presents the human composer's work. The right sheets show ABC notations from different models alongside the reference. The Gemma-7B-it model merely replicates the provided motif, adding no new elements. Similarly, GPT-4 simply repeats the given condition. Qwen-7B-chat and Llama-7B-chat include correct musical elements and the motif but fail to capture the musical form "AB" and maintain the duration of a measure.














The main contributions of our paper are as follows: (1) we provide multi-step prompt engineering and explore how LLMs exhibit their reasoning capabilities with multi-step instructions in music understanding and generation tasks. (2) we assess four major LLMs on various symbolic music tasks, analyzing their reasoning in ABC sequences through quantitative statistical results and qualitative human assessment, including error analysis. The examples, hand-crafted prompts, and codes of data preprocessing are available at https://github.com/SylviaZiyaZhou/LLMs_music_reasoninggithub.



§ RELATED WORK

In this section, we summarize related works from two perspectives. First, we introduce previous studies on LLMs in the symbolic music domain, explaining their performance and evaluation methods in music understanding and generation tasks. Then, we discuss the application of LLMs in reasoning math problems and controllable creative text generation, highlighting similarities between the reasoning processes in music and math and the conditioned, open-ended nature of both music and text generation.



 §.§ LLMs in Symbolic Music Domain

This subsection reviews the application of LLMs in the symbolic music domain. Previous studies have focused on adapting LLMs for music understanding and generation. 

Chatmusician <cit.> uses continual pre-training and fine-tuning on LLaMA2 to understand and generate ABC notation music, without specialized music structures or tokenizers. SongComposer <cit.> collects a song pretraining dataset including lyrics, melodies and paired lyrics-melodies, employing 10K crafted QA pairs to enable LLMs to perform multiple music-related tasks such as lyric-to-melody conversion and song continuation. MusicAgent <cit.> integrates various music tools into a single system, though it lacks interaction among these tools. 
Most approaches view music creation as a linear process, which diverges from the multi-step approach humans use, limiting their applicability for generating creative works.  To mimic human creative processes,  ByteComposer <cit.> employs a four-step method to replicate the creative workflow of human composers: conception analysis, draft Composition, self-evaluation and modification, and human aesthetic selection. 
And designs an interactive agent system consisting of expert, generator, voter, and memory modules. What's more, they construct supervised fine-tuning data covering tasks of basic music theory conception, control code generation, music score evaluation and next-step planning. Despite being a significant step towards multi-step music creation with LLMs, it lacks a detailed discussion on the limits of LLMs at each stage.



 §.§ Reasoning and Controllable Generation with LLMs

"Reasoning" in NLP involves integrating various knowledge sources or contexts to generate new assertions, events, or actions <cit.>.  This process often breaks complex questions into sequential steps <cit.>.  Techniques such as Chain-of-Thoughts (CoT) <cit.> have shown effectiveness in addressing complex reasoning tasks, particularly in mathematics. The Program-of-Thoughts approach improves upon CoT by using language models to generate text and code, enhancing math problem-solving performance <cit.>. Plan-and-Solve (PS) Prompting, a zero-shot technique, outperforms zero-shot CoT significantly, exceeds Zero-shot Program-of-Thoughts, and matches 8-shot CoT in math reasoning <cit.>.

While music and mathematics share similarities, it is crucial to recognize that music is not as deterministic. In controllable music generation, despite given chords, motifs, and forms, unpredictable elements still significantly affect the quality of the music, similar to controllable text generation. Zhang et al.<cit.>, identify three types of control conditions: semantic, structural, and lexical. Semantic controls refer to content control such as sentiment <cit.> or topic <cit.>, resembling style and emotion in music. Structural control involves shaping the structure of the generated text, such as setting a story's framework or using data from tables or graphs as input, similar to specifying musical forms for generation <cit.>. Lexical controls manage vocabulary usage, ensuring specific keywords appear, akin to using musical chords and motifs as guidelines. LLMs are extensively applied in diverse controllable and creative generation tasks  <cit.>. These systems' abilities in long-context and multi-step generation under predefined conditions are examined, though such analyses are rarely applied in the music domain.  















§ METHODOLOGY



 §.§ Datasets

In this paper, we incorporate six tasks covering from music understanding to generation. The data is collected from MusicPile and MusicBench in ChatMusician <cit.>. The statistics of the dataset we use are shown in Table <ref>. Each model can support the maximum length of tokens of each task. 
 





 §.§ Prompt Engineering

Before examining each LLM's task performance, we conducted preliminary tests to verify their understanding of the relevant musical concepts. These tests confirmed that all models possess foundational knowledge of the six music tasks assessed in this study.

We employed two prompt modes in our experiments of all tasks, Default and Chain-of-Thoughts (CoT). Default mode means forcing the model to respond without any analysis. Additionally, for music theory exercises, to make the model better understand the questions and options, and return the answer in a unified format, we also include the In-Context-Learning (ICL) mode by adding some question-answer pairs as examples shown to the models in the prompt. Taking the task of music theory exercises understanding as an example, three modes of prompts as the prefix of inputs followed by each item in the datasets are shown in Figure <ref>. Different from the music theory exercise, we specifically design prompts to support a multi-round chat conversation with LLMs for the generation tasks. Figure <ref> shows an example of a four-round prompt set of chord-conditioned generation. We invite graduates who majored in music composition to write down their multi-step thoughts when completing the generation tasks involved in this paper. We summarize the common steps of all answers, adapt them to the prompt set, and make sure LLMs can understand or at least intend to follow the instructions. An example of GPT-4's response to the instruction in Figure <ref> is shown on the website[https://github.com/SylviaZiyaZhou/LLMs_music_reasoning/blob/main/
CoT_music_generation_GPT4_response.pdf].

































 §.§ Pre-processing Responses

The responses of models are supposed to have correct ABC notations, but it may have certain syntax or formatting issues, and some outputs may even contain a large amount of natural language.


We select the main features of ABC notation including field names and bar line symbols to help us extract the ABC sequence. 

If the extracted ABC sequence can be rendered into MIDI files using Music21[https://web.mit.edu/music21] successfully and can be later rendering into audio file using midi2audio[https://pypi.org/project/midi2audio], we consider it capable of producing valid ABC notation.



 §.§ Multi-step Reasoning Analysis

In order to compare each model's reasoning capability on both understanding and generation tasks, we first conduct a subjective assessment to evaluate how different models' reasoning processes influence their performances. Participants are all familiar with basic music theory and can understand each task as well as the ABC notation. Secondly, based on the results of the subjective assessment, we further perform an error analysis in detail to show the intermediate answers during the reasoning process of each model.     



  §.§.§ Human Assessment Pipeline

In this section, we will provide a detailed description of our subjective experiments on four popular and open-source LLMs, including Gemma-7B-it, Llama2-7B-Chat, GPT-4, and Qwen-7B-chat. 
We ask the participants to evaluate to what extent the model understands the instructions and correctly answer the questions in the understanding tasks, and to what extent the responses contain the conditions and make creative works in the generation tasks. Specifically, the questions in the human assessment are as follows:






  * For both understanding and generation tasks: 1) To what extent does the model understand and follow the instructions? 

  * Specifically for the understanding tasks: 1) To what extent does the model correctly answer the question? 2) To what extent does the model reason like human beings?

  * Specifically for the generation tasks: 1) AB test: please choose the better one
between a pair of music excerpts by considering their "Musicality"; 2) To what extent does the model contain the conditions?


Except for the AB test in the generation task, each question should be rated in a scoring range from 0 to 10 points. We invited music experts who are familiar with ABC notations as the participants in the human assessment, ensuring that each item was evaluated by at least two experts. 




§ EVALUATION RESULTS

In this section, we provide the evaluation results based on the methodology we discussed in the last section. The quantitative results include the correctly parsing rate of ABC notation in the generation tasks, and the accuracy of music theory exercises. The qualitative results include the statistical analysis of human assessment and the detailed error analysis. Due to space limitation, we provide the examples at https://github.com/SylviaZiyaZhou/LLMs_music_reasoninggithub and the online links of the corresponding files will be attached in the illustration.



 §.§ Quantitative Results

Figure <ref> shows the success rate of rendering valid audio from each LLM's responses under different generation tasks. The pre-processing methodology is introduced in Section <ref>. Except for GPT-4, the other three models all have an audio generation rate of less than 50%, finding it difficult to generate the correct ABC notation format to be converted into audio. 

Table <ref> displays the accuracy of the music theory exercises in three modes. The reason why some models have an accuracy rate below 25% in multiple-choice questions with four options is that most of their responses seek additional information about the questions rather than answering them. Gemma-7B-it has a comparable performance with GPT-4 in the Reason. subset in the Default mode even with a much smaller model size. However, CoT and ICL modes, which significantly improve the GPT-4's performance, show very limited effect or even deficiency in other models. This may inspire us to reconsider the utilization of classical CoT and ICL approaches in solving music tasks.



















































































    
























 §.§ Qualitative Results





For human assessment, Table 3 
shows LLMs on ME and MFE tasks under the CoT mode. We randomly sampled 40 examples of each task. In the instruction following question, GPT-4 demonstrates very good results, while other LLMs more or less can accomplish the tasks, indicating a certain level of capability. However, when it comes to the correctness, even GPT-4 finds it challenging to provide satisfactory answers to the prompts. When testing the logical reasoning of LLMs, the average scores indicate that all LLMs encounter difficulties in applying logical reasoning when answering questions step by step, leading to fundamental errors in music theory or illogical conclusions. This highlights the LLMs' limitation of involving music background knowledge. 




Table 4 
presents the results of human assessment we conducted on generative tasks. In addition to the results shown in the table, we also conducted an AB test based on Musicality. We find that the GPT-4 and Gemma-7B-it achieve comparable results in MFMCG task, while in other tasks GPT-4 always wins. This means Gemma-7B-it has a potential in creating high-quality symbolic music with limited model size.

As depicted in Figure <ref>, on MH and CCG tasks, Qwen-7B-chat and Llama2-7B-chat struggled to effectively output correct ABC sequences to be rendered into audios. Therefore, for MH and CCG tasks, we only include the AB test results for GPT-4 and Gemma-7B-it. Despite GPT-4 achieving relatively better scores in generative tasks, it still falls far away from humans' expectations. Interestingly, beyond the data, LLMs' generative results occasionally exhibit instances of copying motifs provided in the prompt, as well as displaying unstructured harmonic repetitions or completely off-key notes. We believe that although LLMs can adhere to the ABC format condition provided in the prompt, their lack of musical information and knowledge makes it challenging to understand the high-level information within the condition, resulting in less satisfactory generated outcomes.



In terms of the results from subjective experiments, we identified a common issue prevalent in LLMs. Firstly, LLMs, apart from GPT-4, struggle to generate data in the correct ABC format with high probability, despite being able to provide a perfect answer when asked what ABC notation is. This phenomenon led us to speculate that while LLMs are trained extensively and comprehensively, LLMs can hardly understand all the information they have been exposed to and utilize them in different scenarios. Besides, LLMs can generate music in a seemingly appropriate ABC format in generative tasks, but what appears to be a correctly-formatted response is merely copying the prompt without grasping the semantic and structural information in the given condition.



  §.§.§ Multi-step Reasoning Analysis




































































To better illustrate each model's reasoning capability when it is used to complete the music theory exercises, we provide an example of a question in the music theory exercises subset and step-by-step responses of the four models[https://github.com/SylviaZiyaZhou/LLMs_music_reasoning/blob/
main/CoT_music_theory_exercise_all_LLMs.pdf]. The question is about recognizing the interval property of an ABC sequence referring to a compound in a music sheet. From the responses, we can see that GPT-4 is the only model which can actually perform the calculation but still unable to understand the musical notes in the ABC notation. In the GPT-4's responses in the CoT mode, "4", which is mistaken as "a fourth apart", should be a note duration. Accordingly, this mistake influences the whole reasoning process of the calculation of intervals. The response of Llama2-7B-chat also shows its incapability of involving correct music knowledge understanding of notes intervals in the reasoning process. What's more, Qwen-7B-chat even accidentally contains Chinese in the English text and Gemma-7B-it failed to recognize musical notes in the ABC sequence (see in the supplementary materials), although they can return the correct answer if they are merely asked about "the definition of note intervals". 

Besides, the responses of generation tasks such as MFMC generation also have similar problems. In the CoT mode, we find all LLMs except GPT-4, are hard to follow the multi-step instructions and output music in a correct ABC format, so we only provide a GPT-4 response respectively in the raw text[https://github.com/SylviaZiyaZhou/LLMs_music_reasoning/blob
/main/CoT_music_generation_GPT4_response.pdf] and music sheet[https://github.com/SylviaZiyaZhou/LLMs_music_reasoning/blob
/main/Music_Sheet_of_music_generation_GPT4_response.pdf] form given the prompt in Figure <ref>. Although GPT-4 can well understand the instructions in every step, it generates repetitive and simple rhythm without enough progression and variation, compared to the human composer's work in Figure <ref>. 







§ CONCLUSION AND DISCUSSION


In conclusion, our experimental analysis highlights current LLMs' limitations in the realm of music understanding and generation, particularly from the perspective of song-level multi-step reasoning. These findings are crucial as they underline the challenges LLMs face when tasked with generating coherent and contextually rich musical compositions, which often require both complex sequential processing and creative fineness. From the human assessment results and the error analysis, we find that all these models failed to inject correct music theory and knowledge in the music understanding, reasoning and generation process. This knowledge generalization gap is analogous to the reversal curse problem illustrated in  <cit.> where LLMs trained on “A is B” fail to learn “B is A”. Without making sure the fundamental concepts are correctly mentioned in the generated responses, it is hard to alleviate the LLMs' hallucination and guarantee the responses' quality. Therefore, it is significant to implement the knowledge augmentation module in the Supervised Fine-Tuning (SFT) stage to ensure the LLMs can reason based on correct music knowledge by curating more SFT data with enough knowledge-based contexts and practical reasoning processes. 

Specifically, several insights for the multi-step SFT dataset construction can be concluded from the process where professional musicians are asked to create music following the instructions. Firstly, more expert knowledge should be involved in the dataset construction to guarantee its quality. For example, in the chord-conditioned generation task in Chatmusician's dataset, the bass note sequence of the given chords does not conform to the musicians' expectation of the progression generally. 
Secondly, some conditions in the old one-step form are too lengthy and informative with limitations that the human composers feel difficult to follow. For example, when they are given an "AB" structure with two different motives in the MFMC task, all of them find hard to integrate two segments with different motives into a complete piece of music in an "AB" form. Therefore, it might not be reasonable to ask the LLMs to output a completely and well composed music in a one-step approach.

What's more, although four models are all claimed to be able to handle the input size from 4K to 8K tokens, which is much longer than the instructions in the dataset we used, they do not show their long-context processing advantages in the symbolic music domain. Our experimental results show that the widely-used CoT and ICL approaches are not always effective in improving the model's performance. In this way, more step-by-step learning strategies should be specifically developed for instruction-based symbolic music tasks by focusing on correctly answering music theory exercises, explicitly extracting motifs and implicitly extracting musical forms, and consistently following the conditions in the instructions.











§ ACKNOWLEDGEMENT

The research was supported by Early Career Scheme (ECS-HKUST22201322), Theme-based Research Scheme (T45-205/21-N) from Hong Kong RGC, NSFC (No. 62206234), and Generative AI Research and Development Centre from InnoHK. Yinghao Ma is a research student at the UKRI Centre for Doctoral Training in Artificial Intelligence and Music, supported by UK Research and Innovation [grant number EP/S022694/1].

We would like to express our sincere gratitude to Jia Ding and Xiaoduan Li, professional musicians who majored in music composition at Central Conservatory of Music, for their valuable contributions and suggestions throughout the multi-step prompt engineering in conditioned generation tasks. 





































































































 Paper Title: Multi-Level Querying using A Knowledge Pyramid.


Two-Component γ-ray Emission Spectrum and X-Ray Polarization of the Radio Galaxy Pictor A
    [
    August 5, 2024
=========================================================================================

§ ABSTRACT 

This paper addresses the need for improved precision in existing Retrieval-Augmented Generation (RAG) methods that primarily focus on enhancing recall. 

We propose a multi-layer knowledge pyramid approach within the RAG framework to achieve a better balance between precision and recall. 

The knowledge pyramid consists of three layers: Ontologies, Knowledge Graphs (KGs), and chunk-based raw text. 

We employ cross-layer augmentation techniques for comprehensive knowledge coverage and dynamic updates of the Ontology schema and instances. 

To ensure compactness, we utilize cross-layer filtering methods for knowledge condensation in KGs. 

Our approach, named PolyRAG, follows a waterfall model for retrieval, starting from the top of the pyramid and progressing down until a confident answer is obtained. 

We introduce two benchmarks for domain-specific knowledge retrieval, one in the academic domain and the other in the financial domain. 

The effectiveness of the methods has been validated through comprehensive experiments by outperforming 19 SOTA methods.

An encouraging observation is that the proposed method has augmented the GPT-4, providing 395% F1 gain by improving its performance from 0.1636 to 0.8109.






§ INTRODUCTION





Significant advancements have been made in Large Language Models (LLMs), including proprietary models like ChatGPT and GPT-4, as well as open-source variants like FLAN <cit.> and LLaMA <cit.>. 

These models have demonstrated remarkable achievements across a wide range of general knowledge tasks in areas such as language comprehension <cit.>, logical reasoning <cit.>, and complex question-answering <cit.>.

However, the one-size-fits-all nature of general LLMs fails to meet the specific demands for professional or personalized knowledge, such as law <cit.> and finance domains <cit.>. 

A straightforward solution is to utilize Supervised Fine-Tuning (SFT) to tailor LLMs to domain-specific tasks <cit.>. 





Examples include AlpaCare <cit.>, Mental LLaMA <cit.> and Zhongjing <cit.> in medical domain, BloombergGPT <cit.>, Pixiu <cit.>, and DISC-FinLLM <cit.> in financial domain, and DISC-LawLLM <cit.> in law domain.

Nonetheless, this risks catastrophic forgetting <cit.> of the general knowledge when tuning the original model, and is prone to model hallucination <cit.>. 



To address this challenge, the prevalent alternative of Retrieval Augmented Generation (RAG) is introduced as a means to enhance the domain knowledge comprehension of LLMs <cit.>. 

Rather than solely relying on the generation capabilities of LLMs to produce answers, RAG takes a different approach by incorporating information retrieval techniques. 

It retrieves relevant information from existing resources and utilizes this information to enrich the context of the prompt <cit.>. 

This enables in-context learning <cit.> or few-shot learning <cit.> to be initiated based on the enhanced context.

It makes the LLMs' output more stable, accurate, traceable, and interpretable. 

However, early implementations of RAG have relied on unstructured textual data chunks, which are often obtained by partitioning each document in the corpora database into segments with a predefined chunk size and thus not organized in a specific way <cit.>.

To address this limitation, Knowledge Graphs (KGs) have been incorporated into RAG <cit.>.



The incorporation of KGs in RAG is a logical choice, as KGs have long been recognized as a promising augmentation method for retrieval tasks.



In fact, in traditional retrieval tasks, another solution called Ontologies has been found to be suitable for domain-specific tasks due to their formally and strictly defined schemas <cit.>. 

However, the use of Ontologies in RAG has been underexplored, partly because the process of defining the Ontology schema requires significant human effort.

In previous RAG methods, the integrated KGs or Ontologies primarily serve the purpose of incorporating additional domain-specific information, thus enhancing the “recall” by increasing the likelihood of locating relevant information.

However, the aspect of “precision” has not received specific attention in these methods. For instance, Ontologies are recognized for their precision due to the stringent regulations governing schema and SparQL syntax. 

Nonetheless, the Ontology search process does not actively participate in the querying process within existing RAG methods.

Figure <ref> illustrates that different knowledge bases exist at varying levels of specificity, forming a pyramid-like structure. 

The higher layers of the pyramid contain more specific and structured information, providing a greater likelihood of accurately answering domain-specific questions. Hence, these higher layers are considered more “precision”-friendly.

On the other hand, the lower layers of the pyramid are more inclined towards “recall”-friendliness. 

They possess more flexible definitions and can store richer information, albeit potentially sacrificing precision.














The observation serves as motivation for us to build a multi-layer knowledge pyramid and devise a customized querying strategy within the RAG framework.

The main objective of this approach is to achieve a better balance between “precision” and “recall”.

Specifically, we organize a knowledge pyramid with three distinct layers of Ontologies, knowledge graphs, and chunk-based raw text.

During the construction of the knowledge pyramid, we ensure comprehensive coverage of knowledge by incorporating cross-layer augmentation techniques.

This approach reduces the need for human intervention in knowledge completion, as it allows for dynamic updates of the Ontology schema and instances.

Moreover, we maintain compactness by employing cross-layer filtering methods, which remove the redundant triplets in the KG for knowledge condensation.

The retrieval process follows a waterfall model, starting from the top of the pyramid and progressively moving down until the model reaches a confident answer. 

If certainty is not achieved, the search continues to the next layer down, and so on.

We name it as PolyRAG due to its nature of multi-layer knowledge organization and multi-level querying strategy. 




Furthermore, this paper presents a contribution by introducing two benchmarks for domain-specific knowledge retrieval. 

The first benchmark is specifically designed for the academic domain and has been meticulously constructed by the authors themselves. 

It encompasses extensive information regarding 1,319 staff members, 2,061 courses, 31 departments, as well as classrooms and facilities at XXX university.

In addition, the authors have extended an existing public dataset called FiQA <cit.> in the financial domain to create the second benchmark. 

We restructured the knowledge from this dataset into a similar pyramid structure, allowing for the application of the proposed methodology.

Both benchmarks will be made available to the community. 































































§ RELATED WORKS





 §.§ Domain-Specific Large Language Models







LLMs have become integral in various applications, such as chatbots, writing assistants, customer service automation <cit.>.

Domain-specific LLMs are a subset of LLMs that have been tailored to understand and generate text within a particular area of expertise, such as law <cit.>, medicine <cit.>, or finance <cit.>. These models aim to provide higher accuracy and more relevant outputs than general-purpose LLMs when dealing with specialized content <cit.>.



A common approach is to adapt instruction fine-tuning based on the general LLMs <cit.> using domain-specific copra. 



However, research has shown that the generality of the fine-tuned model will diminish, resulting in catastrophic forgetting <cit.>. 

In order to tackle the limitations, the approach of prompting LLMs with extra domain knowledge as contexts is widely adopted.



 §.§ Retrieval-Augmented Generation


Retrieval-Augmented Generation (RAG) enhances the generative capabilities of language models by incorporating retrieved knowledge for in-context learning <cit.>. 







NaiveRAG represents the most basic architecture within this framework, in which the system retrieves the top-k documents that are most relevant to the query and integrate them into the prompt, thereby grounding the responses in more relevant information <cit.>. 

Expanding on NaiveRAG, advanced RAG incorporates additional modules or structures to improve retrieval precision. Reranking is a notable example, where a reranker is employed to refine the initial ranked list (e.g., Re2G <cit.> and bge-reranker <cit.>, both are based on BERT <cit.>).

Furthermore, studies have indicated that excessive noise and lengthy context can have a negative impact on inference performance. To address this, prompt compression methods such as Selective Context <cit.> and LLMLingua <cit.> have been developed. 

These methods emphasize key information while reducing noise and context length, as discussed in <cit.>.





 §.§ Knowledge-Augmented Language Models

The Knowledge-Augmented Language Model (LM) approach involves integrating LMs with additional knowledge bases to facilitate in-context learning <cit.>. 

In addition to incorporating knowledge in the form of raw texts <cit.>, knowledge graphs (KGs) <cit.> have gained popularity, and Ontologies <cit.> have also been utilized, albeit less frequently.

In most KG augmentation methods <cit.>, the RAG framework is followed, wherein the context is expanded by incorporating retrieved KG triples instead of raw text chunks. 

KAPING <cit.> serves as an early example of this approach, which has been later refined in RRA <cit.>.

Regarding Ontologies, instead of being used as an individual knowledge base, they are often employed as assistants for generating KG triplets <cit.> (e.g., Text2KGBench <cit.>) or for augmenting textual corpora (e.g., EKGs <cit.>, OntoChatGPT <cit.>).

The integration of various forms of knowledge bases poses a significant challenge and remains an area that has received limited exploration.










§ KNOWLEDGE PYRAMID CONSTRUCTION



Let us establish the knowledge pyramid as the foundational base for PolyRAG.

The construction process begins by creating three distinct knowledge banks, each guided or supported by an Ontology, a knowledge graph, and the raw texts, following common practices.

These banks (denoted as 𝒪, 𝒦, and 𝒯, respectively) form the initial layers of the pyramid.

The essence of our proposed methodology lies in fostering interactions between these layers, aimed at enhancing the overall comprehensiveness and compactness of the knowledge base.





 §.§ Construction of Initial Layers












Ontology Layer: 

The Ontology layer 𝒪={𝒪_s,𝒪_i} consists of a schema 𝒪_s and corresponding instances 𝒪_i.

Defining an Ontology schema typically requires significant time and effort from human experts, and achieving a comprehensive schema can be challenging.

To simplify the process, one approach is to initially extract a sub-domain schema from general Ontologies like WordNet <cit.> or ConceptNet <cit.>.

This extracted schema can serve as a starting point and be refined using the semi-automatic approach presented in Section <ref>.



With the schema 𝒪_s that we extracted, we can guide LLMs to extract instances from the raw text layer 𝒯 for each of the concept-attribute pair (c,a)∈𝒪_s, where c is a concept (e.g., professor) and a is one of its attributes (e.g., research_interest).

This can be written as a prompt function

[boxrule=0pt, frame empty]
    f_ins(c, a; p): Given a paragraph {p} from the {domain} domain, please identify instances of the Ontology relationship where a class {c} has the attribute of {a}. Note that the attribute may consist of multiple entities.


Executing this function results in instances that fulfill the specified relationship. 

By repeatedly applying this function to each paragraph, the set 𝒪_i is constructed as


    𝒪_i={f_ins(c, a; p) | ∀(c, a)∈𝒪_s, ∀ p∈𝒯}.


It is important to note that the specific implementation of the prompt may vary among LLMs.

Additionally, in certain cases, it may be beneficial to provide examples to initiate few-shot learning for extracting high-quality information, depending on the capabilities of the LLMs.

Our implementations are available in the Github repository.

Knowledge Graph Layer: 

We adopt Open Information Extraction (OpenIE) <cit.> to extract KG triples from raw texts. 

However, direct extraction often results in noisy output, including irrelevant or duplicate entities. To address this, we draw inspiration from the multi-round prompting approach of LLM2KG <cit.> and reimplement it by introducing four functions: f_par(p) for paraphrasing, f_ent() for entity extraction, f_rel() for relation completion, and f_dis() for disambiguation.

These functions form a cascade for KG triplet extraction and refinement as


    f_kg(p)=f_dis(f_ent(f_par(p)), f_rel(f_par(p))).


It constructs the initial knowledge graph layer as


    𝒦={f_kg(p) | ∀ p∈𝒯}.


Due to space limitations, the implementations of these functions are provided in the Appendix <ref>.






 §.§ Knowledge Completion



As shown in the pyramid of Figure <ref>, the higher layers are more structured but not easy to define or extract, resulting in limited coverage.

One common issue is the absence of important classes or attributes from the expert-defined Ontology schema, which has historically hindered the ease of defining Ontologies.

By utilizing the pyramid framework, we can address this issue in a data-driven manner.

The idea is to identify noteworthy concepts and relations that exist in the lower layers but are absent from the higher layers. 

These identified elements are then incorporated into the higher layers to enhance knowledge completion.

For instance, our research reveals that the relationship publications_in_important_journals appears frequently in both the knowledge graph and raw texts but is absent from the Ontology.

This is due to the tendency of human-defined schema to overlook this specific relationship, as the presence of the publications attribute may make it seem redundant.

However, users often query this specific relation, and it is explicitly mentioned in professors' profiles.

It is thus worth including it in the Ontology.

Our approach involves modeling the semantic distributions of the Ontology and knowledge graph to identify concepts and relations that exhibit significant divergence between the two layers. 

To achieve this, we begin by transforming class-attribute pairs in the Ontology layer 𝒪 into subject+relation phrases that align with the format in the knowledge graph layer 𝒦.

Instructor embedding <cit.> of phrases at both layers are then encoded so as to project them into a common semantic space.

We then learn for each layer a multivariate Gaussian using


    F(𝐗)   =1/(2π)^n/2|Σ^1/2|e^-1/2(𝐗-μ)^⊤Σ^-1(𝐗-μ),


where n is the dimension of the space, 𝐗 denotes a phrase embedding, and μ and Σ are the mean and covariance matrix, respectively.

The semantic distributions of the Ontology and knowledge graph are then modeled as 𝐗_𝒪∼𝒩(μ_𝒪,Σ_𝒪) and 𝐗_𝒦∼𝒩(μ_𝒦,Σ_𝒦), respectively.

The Kullback–Leibler (KL) divergence can then be measured on each given phrase embedding 𝐗 as


    D_𝐗(𝒪∥𝒦)   =-F_𝒪(𝐗)log(F_𝒦(𝐗))
    
        D_𝐗(𝒦∥𝒪)   =-F_𝒦(𝐗)log(F_𝒪(𝐗)),


where D_𝐗(𝒪∥𝒦) measures the relative divergence at the point 𝐗 by the distribution of 𝒦 as the reference, while in D_𝐗(𝒦∥𝒪), the distribution of 𝒦 is used as the reference. 

At a specific location 𝐗, we define a function to rate the priority to extract entities from the knowledge graph to the Ontology as


    Pr(𝐗)=D_𝐗(𝒪∥𝒦)-D_𝐗(𝒦∥𝒪).


The rating outcomes are depicted in Figure <ref>, illustrating that this function effectively identifies areas where the knowledge graph contains dense information that is comparatively lacking in the Ontology.

We then apply k-medoids clustering on knowledge graph triplets within these area to locate new Ontology class and attributes that can be integrated into the Ontology schema, along with corresponding instances.

This process can be performed iteratively, allowing for continuous refinement and improvement. 

As a result of these interactions between the two layers, the comprehensiveness of the pyramid is progressively enhanced.


















 §.§ Knowledge Condensation


The process of Knowledge Completion aims to enhance the richness of information in a bottom-up manner. 

In addition, we perform Knowledge Condensation to improve the compactness of the pyramid in a top-down approach. 

The principle is to leverage the well-structured knowledge in the higher layers to eliminate redundant information present in the lower layers.

This is implemented by using each class-attribute in the Ontology layer as an anchor and explore its neighborhood to identify a set of k nearest knowledge graph triplets {𝐗_𝐤}.

A prompt function is then applied to instruct the LLM to summarize them into compact ones as

[boxrule=0pt, frame empty]
    f_con({𝐗_𝐤}): Please condense the set of knowledge graph triplets {𝐗_𝐤} obtained from the {domain} domain by eliminating redundant triplets and summarizing the remaining ones in a more concise manner. Here are some examples for your reference: {examples}. The condensation process should follow the logic of {CoT}.


By repeating this process for all anchors, we obtain the condensed knowledge graph layer, as depicted in Figure <ref>. 

The original triplets such as “Prof. A works in Z” and “Prof. A is with Z institue” have been merged into more compact form as a relation of “is_a_member_of”.

It is evident that the compactness of the pyramid has been significantly improved.






























































































































































§ MULTI-LEVEL QUERYING


The pyramid enables PolyRAG in a straightforward top-down querying manner, following the flow Ontology→Knowledge Graph→Raw Texts, in which the results are returned if answers are found at higher layers; otherwise, the querying continues to the next layer.

The querying flow is depicted in Algorithm <ref>.

Two essential components to build to support the logic are the ways of searching at each layer and conditions to dive next.

The Raw Texts layer can be effectively addressed by employing the NaiveRAG approach.

Regarding the design of the approaches for the remaining two layers, we propose the following strategies.






 §.§ Search at Ontology Layer

At the Ontology Layer, we can utilize SparQL as the query language. 

SparQL is a well-defined language that provides precise results when queries are properly structured.

To achieve this, we can guide the LLM to generate the SparQL query by using the following prompt function

[boxrule=0pt, frame empty]
Query_onto(Q, schema): Given a input question {Q} within the domain {domain}, please formulate a SPARQL query to retrieve the answer based on the provided Ontology schema. The namespace is {schema.base}, and the classes are {schema.class}. The object properties between classes are {schema.op}, and the classes may also have data properties such as {schema.dp}. 





For the example of a query question like “Who is currently working in CS Department and was graduated from Cambridge University?”, the LLM extracts the related attributes of “works_in” and “graduated_from” with the given schema, and generate a SPARQL query to search a staff who satisfies the relevant condition in 𝒪. The executing results may include a list of names, or simply empty due to either the lack of knowledge in Ontology layer, or the mistake of providing unavailable SPARQL due to the limitation of LLM itself.



 §.§ Search at Knowledge Graph Layer


At the knowledge graph layer, we employ a retrieval approach similar to the embedding-based retrieval utilized in NaiveRAG. 

However, instead of using chunks, we work with triplets.

Once the matching triplets are retrieved, we utilize a prompt function to assess the Language Model's agreement on whether the question has been answered adequately as

[boxrule=0pt, frame empty]
Query_kg(Q, triples): Given a question {Q} and the context information provided by the matched triples from the knowledge graph {triples}, please justify whether the provided information is sufficient to accurately answer the question. Respond with either “Yes” or “No” to provide your justification.





For example, given a query of “which CS staff has interest in cloud computing?”, the LLM may justify a set of triples as “agree” if they include information like “Prof. A works in CS Department, Prof. A published on cloud computing journal, ... etc.”; inversely, the justification might be “disagree” if the triples do not provide the key knowledge related to this query, thus the searching will process to the next layer for a more relevant context in order to answer the query.























































































§ EXPERIMENTS




 §.§ Experimental Setup

Our experimentation involves two distinct domain-specific benchmarks. 

The first benchmark is Academia Challenge (AcadChall), which has been built by ourselves and focuses on the academic domain. 

It encompasses a comprehensive collection of data obtained from XXX university, including information about 31 departments, 1,319 faculty members, and 2,061 courses. 

Specifically, AcadChall consists of 512 MCQ and MAQ questions that cover topics related to teaching and research. 

These questions are intentionally designed to be more challenging compared to existing benchmarks, as each question presents eight answer choices. 

This design aims to provide a more rigorous assessment of the models' precision.

The second benchmark is FiQA, which is widely recognized and utilized in the finance domain. Our experimentation follows the dataset split provided by FLUE benchmark <cit.>, resulting in the creation of the R-FLUE-FiQA dataset comprising 1,705 open-ended questions.

To enhance knowledge completion and condensation, we employ the methods proposed in Section <ref> to construct Ontologies, knowledge graphs, and pyramids based on the two datasets. 
For more detailed information about the two benchmarks, please refer to the Appendix <ref>.


We employ instructor-xl <cit.> as our vector base for embedding search and the cosine similarity is used as the metric <cit.>.

The k values for KG layer and raw texts layer are empirically set to 10 and 5, respectively.


















 §.§ Evaluation Metrics

Our evaluation process encompasses several metrics to assess the performance, including: 

1) For MCQs and MAQs, we employ metrics such as Precision, Recall, and F1 score. 

2) For open-ended questions, we utilize BLEU and BERT similarity, calculated based on MiniLM embeddings <cit.>.

3) We introduce a novel metric called HitRate (HR), which quantifies the proportion of correct entities present in the response.

These metrics collectively allow for a comprehensive assessment of the system's effectiveness across different question types and response formats. 
































 §.§ Comparison to SOTA Methods


We compare PolyRAG with five groups of state-of-the-art (SOTA) methods, including 1) Frozen-LLMs that are pretrained LLMs of frozen parameters and  no external knowledge has been used for querying.

2) SFT (Supervised Fine-Tuning) that are LLMs that undergo supervised fine-tuning on domain-specific datasets.

We include two latest implementation of Finance-LLM and Finance-Chat <cit.>, both of which are trained on financial corpora. 

Additionally, we explore variants of SFT that employ LoRA <cit.> instead of the entire models.

3) NaiveRAG proposed in <cit.>.

4) Advanced RAG that include two recent models, namely ColBERTv2 <cit.> and Bge-reranker <cit.>.

5) KG-Augmented LLMs that consists of models proposed in <cit.> that integrate knowledge graphs (KGs) into the LLMs.

Various LLMs have been explored as the backbones, including LLaMA <cit.>, LLaMA2 <cit.>, Vicuna <cit.>, Gemini <cit.>, GPT-3.5 and GPT-4.

By combining them with the five groups and PolyRAG, this results in a comprehensive set of 23 methods for performance comparison.

The results are shown in Table <ref>. 

It is clear that PolyRAG surpasses other SOTA methods across the five groups, different backbones, and both benchmarks.

An encouraging finding is that PolyRAG demonstrates a superior balance between precision and recall.

Among the improvements of PolyRAG over others, the gain on the precision (22.0%± 17.3) is more pronounced than that of recall (18.4%± 18.6).

With the incorporation of PolyRAG, GPT-4 (GPT-3.5) has achieved the highest gains by 338.6% (133.2%) over its frozen model and 73.1% (69.8%) over its NaiveRAG run on the AcadChall benchmark.

The improvement may result from the fact that around 43.7% (38.4%) of questions are addressed at the Ontology layer, and around 42.3% (44.8%) are at the KG layer on the AcadChall (R-FLUE-FiQA) benchmark.




















































































































































































































































 §.§ Knowledge Completion and Condensation


We have conducted experiments to evaluate the effectiveness of knowledge completion (CPL) and condensation (CND) by integrating them with a baseline model (PolyRAG without these two modules).

The results are presented in Table <ref>.

Please note that due to space limitations, we only list results for four specific backbones.

For a comprehensive performance analysis with Vicuna-13B and LLaMA2-70B backbones, please refer to the Appendix <ref>.

The results clearly demonstrate the effectiveness of both completion and condensation in enhancing performance, whether used individually (CPL: 3.8%± 6.1 performance gain, CND: 2.2%± 3.3 performance gain) or in combination (6.5%± 6.4 performance gain over the baseline).

Our statistics indicate that with completion, queries resolved at the Ontology layer is increased by 5%.

With condensation, more than 27% of queries are resolved at the KG layer.








































 §.§ Influence of Knowledge Layers


We have examined the influence of different knowledge layers on various backbones.

The results are presented in Table <ref>.

It is obvious that the Ontology layer plays a prominent role as a knowledge base in enhancing precision, as evidenced by a precision gain of 25.8%± 7.4 when combined with the raw text layer and 13.7%± 10.3 when combined with the KG layer.

On the other hand, the KG layer demonstrates a balanced impact on improving either precision or recall.

These observations align with our early discussions.








































































































































































§ CONCLUSION

This research has described a preliminary investigation that provides a multi-level query approach to domain-specific question answering. In this context, we addressed two significant challenges: the problem of prioritizing knowledge in a certain area and the presence of noise in retrieval situations. In order to overcome the limitations, we introduce Knowledge Pyramid multi-level retrieval framework as PolyRAG. PolyRAG utilizes a sequential retrieval approach and prioritizes knowledge extraction. This approach is designed to tackle the challenges posed by the high demand for dense knowledge in domain-specific scenarios and the distractions caused by noisy contexts. The accuracy of PolyRAG has been validated through extensive tests done in AcadChall and R-FLUE-FiQA, surpassing earlier approaches and generating state-of-the-art results.









§ LIMITATIONS

One limitation to consider is that Language Models (LLMs) do not strictly adhere to the SparQL syntax, which can result in typographical errors in the queries.

Further investigation into this aspect is warranted.



§ ETHICS STATEMENT

This research paper adheres to ethical considerations throughout its methodology and findings. The study primarily focuses on the development and evaluation of a retrieval-augmented generation framework, PolyRAG, for domain-specific knowledge retrieval. The research involves the use of existing datasets and benchmarks, as well as the creation of two new benchmarks in the academic and financial domains.

The authors ensure proper citation and acknowledgment of the data sources to maintain transparency and intellectual property rights. In terms of human subjects, this research does not involve any direct human participation, personal data collection, or sensitive information. Therefore, ethical concerns related to informed consent, privacy, and confidentiality are not applicable in this context.











60


[Abu-Rasheed et al.(2024)Abu-Rasheed, Weber, and Fathi]1-kg-recom
Hasan Abu-Rasheed, Christian Weber, and Madjid Fathi. 2024.
https://arxiv.org/abs/2403.03008 Knowledge graphs as context sources for llm-based explanations of learning recommendations.
Preprint, arXiv:2403.03008.

[Anderson et al.(2022)Anderson, Wilson, and Richardson]2-1-lingua
Nathan Anderson, Caleb Wilson, and Stephen D Richardson. 2022.
Lingua: Addressing scenarios for live interpretation and automatic dubbing.
In Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas (Volume 2: Users and Providers Track and Government Track), pages 202–209.

[Baek et al.(2023a)Baek, Aji, and Saffari]1-14-kaping
Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023a.
Knowledge-augmented language model prompting for zero-shot knowledge graph question answering.
arXiv preprint arXiv:2306.04136.

[Baek et al.(2023b)Baek, Aji, and Saffari]KGRAG2023Baek
Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023b.
Knowledge-augmented language model prompting for zero-shot knowledge graph question answering.
In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pages 78–106.

[Baldazzi et al.(2023)Baldazzi, Bellomarini, Ceri, Colombo, Gentili, and Sallinger]baldazzi2023finetuneontologicalreasoning
Teodoro Baldazzi, Luigi Bellomarini, Stefano Ceri, Andrea Colombo, Andrea Gentili, and Emanuel Sallinger. 2023.
Fine-tuning large enterprise language models via ontological reasoning.
In International Joint Conference on Rules and Reasoning, pages 86–94. Springer.

[Carta et al.(2023)Carta, Giuliani, Piano, Podda, Pompianu, and Tiddia]3-3-llm2kg
Salvatore Carta, Alessandro Giuliani, Leonardo Piano, Alessandro Sebastian Podda, Livio Pompianu, and Sandro Gabriele Tiddia. 2023.
Iterative zero-shot llm prompting for knowledge graph construction.
arXiv preprint arXiv:2307.01128.

[Chang et al.(2024)Chang, Wang, Wang, Wu, Yang, Zhu, Chen, Yi, Wang, Wang et al.]chang2024evaluation_survey
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024.
A survey on evaluation of large language models.
ACM Transactions on Intelligent Systems and Technology, 15(3):1–45.

[Chen et al.(2023)Chen, Wang, Long, Zhang, Lu, Li, Wang, Xu, Bai, Huang, and Wei]4-3-discfinllm
Wei Chen, Qiushi Wang, Zefei Long, Xianyin Zhang, Zhongtian Lu, Bingxuan Li, Siyuan Wang, Jiarong Xu, Xiang Bai, Xuanjing Huang, and Zhongyu Wei. 2023.
https://arxiv.org/abs/2310.15205 Disc-finllm: A chinese financial large language model based on multiple experts fine-tuning.
Preprint, arXiv:2310.15205.

[Cheng et al.(2024)Cheng, Huang, and Wei]financeChatLLM2024Cheng
Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024.
Adapting large language models via reading comprehension.
In The Eleventh International Conference on Learning Representations.

[Chiang et al.(2023)Chiang, Li, Lin, Sheng, Wu, Zhang, Zheng, Zhuang, Zhuang, Gonzalez, Stoica, and Xing]4-3-vicuna
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.
https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

[Cui et al.(2023)Cui, Li, Yan, Chen, and Yuan]2-chatlaw
Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023.
https://arxiv.org/abs/2306.16092 Chatlaw: Open-source legal large language model with integrated external knowledge bases.
Preprint, arXiv:2306.16092.

[Dong et al.(2023)Dong, Li, Dai, Zheng, Wu, Chang, Sun, Xu, Li, and Sui]1-incontext-survey
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023.
https://arxiv.org/abs/2301.00234 A survey on in-context learning.
Preprint, arXiv:2301.00234.

[Etzioni et al.(2008)Etzioni, Banko, Soderland, and Weld]3-3-openie
Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S Weld. 2008.
Open information extraction from the web.
Communications of the ACM, 51(12):68–74.

[Gao et al.(2023)Gao, Xiong, Gao, Jia, Pan, Bi, Dai, Sun, and Wang]2-1-ragreview
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023.
Retrieval-augmented generation for large language models: A survey.
arXiv preprint arXiv:2312.10997.

[Ge et al.(2024)Ge, Hua, Mei, Tan, Xu, Li, Zhang et al.]2-domainspecific
Yingqiang Ge, Wenyue Hua, Kai Mei, Juntao Tan, Shuyuan Xu, Zelong Li, Yongfeng Zhang, et al. 2024.
Openagi: When llm meets domain experts.
Advances in Neural Information Processing Systems, 36.

[Glass et al.(2022)Glass, Rossiello, Chowdhury, Naik, Cai, and Gliozzo]2-1-re2g
Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Rajaram Naik, Pengshan Cai, and Alfio Gliozzo. 2022.
Re2g: Retrieve, rerank, generate.

[Hadi et al.(2023)Hadi, Qureshi, Shah, Irfan, Zafar, Shaikh, Akhtar, Wu, Mirjalili et al.]hadi2023llmsurvey
Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. 2023.
A survey on large language models: Applications, challenges, limitations, and practical usage.
Authorea Preprints.

[Hu et al.(2022)Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, and Chen]4-3-lora
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.
Lora: Low-rank adaptation of large language models.

[Huang et al.(2023a)Huang, Yu, Ma, Zhong, Feng, Wang, Chen, Peng, Feng, Qin et al.]1-10-hallucination
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023a.
A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.
arXiv preprint arXiv:2311.05232.

[Huang et al.(2023b)Huang, Tao, Zhang, An, Jiang, Chen, Wu, and Feng]2-lawyer
Quzhe Huang, Mingxu Tao, Chen Zhang, Zhenwei An, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. 2023b.
https://arxiv.org/abs/2305.15062 Lawyer llama technical report.
Preprint, arXiv:2305.15062.

[Kenton and Toutanova(2019)]2-1-bert
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.
Bert: Pre-training of deep bidirectional transformers for language understanding.
1:2.

[Kojima et al.(2022)Kojima, Gu, Reid, Matsuo, and Iwasawa]kojima2022zero_shot_reasoner
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.
Large language models are zero-shot reasoners.
Advances in neural information processing systems, 35:22199–22213.

[Lewis et al.(2020a)Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel et al.]1-11-lewis2020rag
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020a.
Retrieval-augmented generation for knowledge-intensive nlp tasks.
Advances in Neural Information Processing Systems, 33:9459–9474.

[Lewis et al.(2020b)Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel et al.]2-1-rag
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020b.
Retrieval-augmented generation for knowledge-intensive nlp tasks.
Advances in Neural Information Processing Systems, 33:9459–9474.

[Litman et al.(2020)Litman, Anschel, Tsiper, Litman, Mazor, and Manmatha]2-1-selective
Ron Litman, Oron Anschel, Shahar Tsiper, Roee Litman, Shai Mazor, and R Manmatha. 2020.
Scatter: selective context attentional scene text recognizer.
In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11962–11972.

[Liu et al.(2019)Liu, Du, and Stoyanov]Liu2019KnowledgeAugmentedLM
Angli Liu, Jingfei Du, and Veselin Stoyanov. 2019.
https://api.semanticscholar.org/CorpusID:104292554 Knowledge-augmented language model and its application to unsupervised named-entity recognition.
In North American Chapter of the Association for Computational Linguistics.

[Luo et al.(2023)Luo, Yang, Meng, Li, Zhou, and Zhang]1-9-forget
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023.
An empirical study of catastrophic forgetting in large language models during continual fine-tuning.
arXiv preprint arXiv:2308.08747.

[Ma et al.(2023)Ma, Gong, He, hai zhao, and Duan]2-1-naiverag
Xinbei Ma, Yeyun Gong, Pengcheng He, hai zhao, and Nan Duan. 2023.
Query rewriting in retrieval-augmented large language models.
In The 2023 Conference on Empirical Methods in Natural Language Processing.

[Mihindukulasooriya et al.(2023)Mihindukulasooriya, Tiwari, Enguix, and Lata]mihindukulasooriya2023text2kgbench
Nandana Mihindukulasooriya, Sanju Tiwari, Carlos F Enguix, and Kusum Lata. 2023.
Text2kgbench: A benchmark for ontology-driven knowledge graph generation from text.
In International Semantic Web Conference, pages 247–265. Springer.

[Miller(1995)]miller1995wordnet
George A Miller. 1995.
Wordnet: a lexical database for english.
Communications of the ACM, 38(11):39–41.

[Ouyang et al.(2022)Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin, Zhang, Agarwal, Slama, Ray et al.]1-2-openaiSFT
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.
Training language models to follow instructions with human feedback.
Advances in Neural Information Processing Systems, 35:27730–27744.

[Palagin et al.(2023)Palagin, Kaverinskiy, Litvin, and Malakhov]1-ontochatgpt
Oleksandr Palagin, Vladislav Kaverinskiy, Anna Litvin, and Kyrylo Malakhov. 2023.
Ontochatgpt information system: Ontology-driven structured prompts for chatgpt meta-learning.
arXiv preprint arXiv:2307.05082.

[Pan et al.(2024)Pan, Luo, Wang, Chen, Wang, and Wu]3-llmandkg
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2024.
Unifying large language models and knowledge graphs: A roadmap.
IEEE Transactions on Knowledge and Data Engineering.

[Radford et al.(2018)Radford, Narasimhan, Salimans, Sutskever et al.]radford2018openaiGPT
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.
Improving language understanding by generative pre-training.

[Rebedea et al.(2023)Rebedea, Dinu, Sreedhar, Parisien, and Cohen]2-app2
Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023.
Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails.
arXiv preprint arXiv:2310.10501.

[Sallam(2023)]2-app1
Malik Sallam. 2023.
Chatgpt utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns.
In Healthcare, volume 11, page 887. MDPI.

[Santhanam et al.(2022)Santhanam, Khattab, Potts, and Zaharia]4-3-colbertv2
Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. 2022.
Plaid: an efficient engine for late interaction retrieval.

[Shah et al.(2022)Shah, Chawla, Eidnani, Shah, Du, Chava, Raman, Smiley, Chen, and Yang]4-1-fiqa
Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. 2022.
When flue meets flang: Benchmarks and large pretrained language model for financial domain.
In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.

[Speer et al.(2017)Speer, Chin, and Havasi]speer2017conceptnet
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.
Conceptnet 5.5: An open multilingual graph of general knowledge.
In Proceedings of the AAAI conference on artificial intelligence, volume 31.

[Su et al.(2022)Su, Shi, Kasai, Wang, Hu, Ostendorf, Yih, Smith, Zettlemoyer, and Yu]INSTRUCTOR
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022.
https://arxiv.org/abs/2212.09741 One embedder, any task: Instruction-finetuned text embeddings.

[Team et al.(2023)Team, Anil, Borgeaud, Wu, Alayrac, Yu, Soricut, Schalkwyk, Dai, Hauth et al.]geminiteam2024gemini
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023.
Gemini: a family of highly capable multimodal models.

[Touvron et al.(2023a)Touvron, Lavril, Izacard, Martinet, Lachaux, Lacroix, Rozière, Goyal, Hambro, Azhar et al.]1-4-llama
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.
Llama: Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971.

[Touvron et al.(2023b)Touvron, Martin, Stone, Albert, Almahairi, Babaei, Bashlykov, Batra, Bhargava, Bhosale et al.]4-3-llama2
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.
Llama 2: Open foundation and fine-tuned chat models.

[Vamsi et al.(2024)Vamsi, Kommineni, and Samuel]1-13-onto
Krishna Kommineni Vamsi, Vamsi Krishna Kommineni, and Sheeba Samuel. 2024.
From human experts to machines: An llm supported approach to ontology and knowledge graph construction.

[Wang et al.(2023)Wang, Yang, and Wang]4-3-fingpt
Neng Wang, Hongyang Yang, and Christina Dan Wang. 2023.
Fingpt: Instruction tuning benchmark for open-source large language models in financial datasets.
NeurIPS Workshop on Instruction Tuning and Instruction Following.

[Wang et al.(2020a)Wang, Wei, Dong, Bao, Yang, and Zhou]MiniLM2020Wang
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020a.
Minilm: deep self-attention distillation for task-agnostic compression of pre-trained transformers.
In Proceedings of the 34th International Conference on Neural Information Processing Systems.

[Wang et al.(2020b)Wang, Yao, Kwok, and Ni]1-fewshot-survey
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. 2020b.
Generalizing from a few examples: A survey on few-shot learning.
ACM computing surveys (csur), 53(3):1–34.

[Wei et al.(2021)Wei, Bosma, Zhao, Guu, Yu, Lester, Du, Dai, and Le]1-3-flan
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.
Finetuned language models are zero-shot learners.
arXiv preprint arXiv:2109.01652.

[Wilkinson and Hingston(1991)]cosineSimilarity1991
Ross Wilkinson and Philip Hingston. 1991.
Using the cosine measure in a neural network for document retrieval.
In Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval, pages 202–210.

[Wu et al.(2023a)Wu, Irsoy, Lu, Dabravolski, Dredze, Gehrmann, Kambadur, Rosenberg, and Mann]1-8-bloomberggpt
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023a.
Bloomberggpt: A large language model for finance.
arXiv preprint arXiv:2303.17564.

[Wu et al.(2023b)Wu, He, Liu, Sun, Liu, Han, and Tang]1-5-chatgptReview
Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. 2023b.
A brief overview of chatgpt: The history, status quo and potential future development.
IEEE/CAA Journal of Automatica Sinica, 10(5):1122–1136.

[Wu et al.(2023c)Wu, Hu, Bi, Qi, Ren, Xie, and Song]Wu2023RRARetrieveRewriteAnswerAK
Yike Wu, Nan Hu, Sheng Bi, Guilin Qi, J. Ren, Anhuan Xie, and Wei Song. 2023c.
https://api.semanticscholar.org/CorpusID:262054223 Retrieve-rewrite-answer: A kg-to-text enhanced llms framework for knowledge graph question answering.
ArXiv, abs/2309.11206.

[Wu et al.(2023d)Wu, Hu, Qi, Bi, Ren, Xie, and Song]1-15-kgretrieve
Yike Wu, Nan Hu, Guilin Qi, Sheng Bi, Jie Ren, Anhuan Xie, and Wei Song. 2023d.
Retrieve-rewrite-answer: A kg-to-text enhanced llms framework for knowledge graph question answering.
arXiv preprint arXiv:2309.11206.

[Xiao et al.(2023)Xiao, Liu, Zhang, and Muennighoff]4-3-bge
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023.
https://arxiv.org/abs/2309.07597 C-pack: Packaged resources to advance general chinese embedding.
Preprint, arXiv:2309.07597.

[Xie et al.(2023)Xie, Han, Zhang, Lai, Peng, Lopez-Lira, and Huang]2-fin-pixiu
Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023.
https://arxiv.org/abs/2306.05443 Pixiu: A large language model, instruction data and evaluation benchmark for finance.
Preprint, arXiv:2306.05443.

[Yang et al.(2024a)Yang, Zhang, Kuang, Xie, Huang, and Ananiadou]1-7-mentalllama
Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024a.
Mentallama: interpretable mental health analysis on social media with large language models.
pages 4489–4500.

[Yang et al.(2024b)Yang, Zhao, Zhu, Zhou, Xu, Jia, and Zan]2-med-zhongjing
Songhua Yang, Hanjie Zhao, Senbin Zhu, Guangyu Zhou, Hongfei Xu, Yuxiang Jia, and Hongying Zan. 2024b.
Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue.

[Yue et al.(2023)Yue, Chen, Wang, Li, Shen, Liu, Zhou, Xiao, Yun, Huang, and Wei]2-disclawllm
Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang, and Zhongyu Wei. 2023.
https://arxiv.org/abs/2309.11325 Disc-lawllm: Fine-tuning large language models for intelligent legal services.
Preprint, arXiv:2309.11325.

[Zhai et al.(2023)Zhai, Tong, Li, Cai, Qu, Lee, and Ma]2-forget-investigating
Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. 2023.
Investigating the catastrophic forgetting in multimodal large language models.
arXiv preprint arXiv:2309.10313.

[Zhang et al.(2023)Zhang, Tian, Yang, Chen, Li, and Petzold]1-6-alpacare
Xinlu Zhang, Chenxin Tian, Xianjun Yang, Lichang Chen, Zekun Li, and Linda Ruth Petzold. 2023.
Alpacare: Instruction-tuned large language models for medical application.
arXiv preprint arXiv:2310.14558.















§ APPENDIX





§ KNOWLEDGE GRAPH LAYER CONSTRUCTION


This section provides the detailed implantations of the four functions for constructing the knowledge graph layer in Section <ref>.

[boxrule=0pt, frame empty]
    f_par(p): Determine the factual claims from a given paragraph {p}. Put these facts into short phrases with basic grammar. Remember that every statement should have a distinct meaning, and pronouns should be avoided.

[boxrule=0pt, frame empty]
    f_ent(f_par(p)): Extract the noun entities in phrases from the given sentences f_par(p). The entities should not contain any comma, and each entity should be unique during extraction.

[boxrule=0pt, frame empty]
    f_rel(f_par(p)): Given the reference context f_par(p) and relevant entities, complete the relations between two entities. Notice that a triple should only contain one entity as head, one verb or verb phrase as relation, and one entity as tail. Separate the head, relation, and tail with a comma.

[boxrule=0pt, frame empty]
    f_dis(f_ent(), f_rel()): You are given several triples f_rel() with their entities f_ent(). These triples consist of subject-predicate-object elements, separated with a comma, but may contain ambiguities or inaccuracies. Your task is to refine and disambiguate these triples to ensure that they accurately reflect the entities and relationships described in their source texts without duplication or omissions. If the relationships have the same semantic meaning, rewrite the triples with the same relation. If the triple has already been mentioned with the same meaning as previous triples, delete it.






§ ACADCHALL AND R-FLUE-FIQA BENCHMARKS




Table <ref> lists the data statistics of two benchmarks. We use Vicuna-13B <cit.> to build the knowledge pyramid for two datasets according to the methods illustrated in Section <ref>. Specifically, the AcadChall dataset is created by crawling public information from the official websites of various departments at XXX University. It encompasses comprehensive information about 1,319 staff members, 2,061 courses, 31 departments, classrooms, and facilities, with over 5k corpus entries. Taking the staff member category as an example, it includes not only personal information, contact details, positions held at the university, and department affiliations but also specific academic knowledge, such as staff research interests, alma maters, degrees, research experience, and published papers, which is hard to be captured by general LLMs. 
In addition, we manually create 512 multi-choice QA pairs. 
Each question corresponds to eight options, and the answer could be one or more options. 
For example, for the question Which staff member in the Art department graduated from Cambridge University?, there could be multiple staff in the provided options that meet the information need.
For the R-FLUE-FiQA dataset, we adopt the test set from the FLUE benchmark <cit.>, which consists of 1,705 open-ended long-form QA pairs. 

















  
  

  
  

























§ DISCUSSION ON KNOWLEDGE COMPLETION AND CONDENSATION





Table <ref> verifies the effectiveness of our proposed knowledge completion and condensation techniques over multiple LLM backends. 
Specifically, the multi-level querying on the knowledge pyramid without completion and condensation techniques is used as the baseline. 
In the baseline, around 30% of questions are answered by more structured data (i.e., Ontology or KG). After applying the completion and condensation techniques, the percentage increases to 64%. 
For example, the relation has published in, which frequently appears in the KG layer, has been used to complete the Ontology layer. It has resulted in those questions related to publications being answered by the Ontology layer. 
Similarly, with knowledge condensation, an increase of around 27% of questions are answered by the KG layer. For example, as has published research in, has published research articles on, has published a paper in, has work published in has been condensed to has published research in  or has published research articles in, the number of tokens that sent to LLM for QA is decreased. 
Having a compact prompt has eased a smaller LLM backend, such as Vicuna-7B, in understanding the context, and we found that it could lead to a better question-answering result.
Overall, with the completion/condensation technique, precision is consistently improved over the baseline across different LLM backends. 
With the collaboration of completion and condensation techniques, the performance boost is more significant. 

















    

    
    

    
    
    























§ INFLUENCE OF KNOWLEDGE LAYERS OVER BACKBONES



Table <ref> lists the impact of different knowledge layers on more LLM backbones than the results presented in Section <ref>. 
The proposed knowledge pyramid significantly improves the precision of the raw text layer across different backbones. With more powerful backbones, such as Genmini-Pro, GPT-3.5, and GPT-4, the performance gains are larger (i.e., over 0.3). 
Besides, higher precision could be obtained when combining the Ontology layer with the KG or raw texts layer. 
A similar observation happens when combing the KG layer with raw text or the Ontology layer across different backbones. These are consistent with the observations in Section <ref>. 































 Paper Title: Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models.


[
     R. Zamini[Corresponding author: rahelehzamini1361@yahoo.com] ^1,  S. Ghafouri^2  and F. Goodarzi^3

	^1  Department  of Mathematics, Faculty of Mathematical Sciences and  Computer, Kharazmi University, Thehran, Iran. 

	^2   Department of Mathematics, Faculty of Science, Arak University, Arak, 384817758, Iran.

^3   Department of Statistics, Faculty of Mathematical Sciences, University of Kashan, Kashan, Iran.

    
===================================================================================================================================================================================================================================================================================================================================================================================================================================

§ ABSTRACT 


[1]Equal contribution. ^♢Work done at AWS AI Labs.


Modern language models (LMs) need to follow human instructions while being faithful; yet, they often fail to achieve both.
Here, we provide concrete evidence of a trade-off between instruction following (i.e., follow open-ended instructions) and faithfulness (i.e., ground responses in given context) when training LMs with these objectives. For instance, fine-tuning LLaMA-7B on instruction following datasets renders it less faithful. Conversely, instruction-tuned Vicuna-7B shows degraded performance at following instructions when further optimized on tasks that require contextual grounding. One common remedy is multi-task learning (MTL) with data mixing, yet it remains far from achieving a synergic outcome. We propose a simple yet effective method that relies on  (), which significantly outperforms vanilla MTL. Surprisingly, we find that less is more, as training  with high-quality, yet substantially smaller data (three-fold less) yields superior results. Our findings offer a better understanding of objective discrepancies in alignment training of LMs.




§ INTRODUCTION

Aligning language models (LMs) with human preferences becomes increasingly important. One main objective is to train LMs to follow human instructions (e.g., answering open-ended questions)
while being faithful (e.g., grounding responses in the given context). However, LMs often suffer from failing to follow human instructions <cit.> or making up facts that are not grounded in context <cit.>.

We trace this problem back to commonly used alignment training datasets, often collected from naturalistic conversations covering a wide range of domains <cit.>. For instance, Alpaca <cit.> and Dolly-15K <cit.> cover tasks from creative writing to context-dependent QA. Particularly, these tasks may have distinct objectives, and, when mixed, may induce potential conflicts of interest during alignment.







In this context, we examine the interaction between instruction following (i.e., how well does the LM follow open-ended instructions) and faithfulness (i.e., is the LM's response grounded in the context) during alignment training.
Specifically, we study how instruction following and faithfulness scores change when adapting LMs to two types of datasets: 1) instruction-tuning datasets that are commonly used to train chat-models such as ChatGPT <cit.> and Llama-2-Chat <cit.>; and 2) context-dependent datasets that require grounding to a provided context, and are commonly used to train Retrieval-Augmented Generation (RAG) models <cit.> such as Atlas <cit.> and DSPy <cit.>.
We observe a clear trade-off between the two scores, as shown in fig:main_figure_1: a fine-tuned LM with a competitive faithfulness score becomes much less faithful when separately fine-tuned on instruction following datasets. Conversely, an instruction-tuned LM becomes worse at following instructions when fine-tuned on context-dependent datasets. Our findings suggest fine-tuning an LM with either instruction following or context-dependent datasets exclusively may impair its original ability in the other aspect.



One natural mitigation strategy is to use multi-task learning (MTL) by mixing datasets, which we find as a strong yet sub-optimal baseline. 
To achieve a more synergic outcome, we propose a simple yet effective method  ().
Inspired by recent works in self-instruct <cit.>,  leverages the LM to sample generations for instruction following and task-specific datasets, which is different from vanilla MTL. Generations are then rated by external judges for instruction following and faithfulness scores, where top-rated generations are collected and used to further fine-tune the LM.
Our experiments show that when trained with  using only a single iteration and 8,000 additional fine-tuning examples, LMs see substantial gains in faithfulness scores (up to +18.8%) compared to the MTL baseline, while maintaining their instruction-following scores. 
Furthermore, we find that less is more: training with  on higher quality yet three-fold less data yields up to 31.3% improvements on faithfulness among datasets compared to MTL.
Our analyses shed lights on finding and mitigating objective discrepancies in alignment training where datasets encompass different or even conflicting goals.[We will release our code and evaluation data at<https://github.com/frankaging/dancing-in-chains>.]





§ BACKGROUND
 
To recap, our goal is to study the interplay of training on instruction following datasets and on datasets that require grounding to a specific context. 
To do this, we first construct two separate groups of datasets, each resembling a setting above.
We then adopt a two-stage fine-tuning paradigm that allows us to closely study the effect of each type of training.
We introduce the setup in this section.








 §.§ Datasets

We outline different datasets used for instruction following and context-dependent fine-tuning. tab:data_stats shows data statistics. For each dataset we carefully design the instructions to well-align the input and output (see sec:instr_template).



  §.§.§ Instruction Following Datasets
 
We curated an instruction following training dataset by compiling unique examples from publicly available datasets such as Dolly-15K <cit.>, 
ShareGPT[<https://huggingface.co/datasets/LLMs/Alpaca-ShareGPT>], Self-Instruct <cit.>, and OASST-1 <cit.>. For evaluation, we gather unique examples from Alpaca-15K <cit.>, Vicuna-eval <cit.>, and Koala-eval <cit.>.



  
Data Pre-processing We exclude examples that originally come with context (e.g., examples labeled as summarization type in Dolly-15K are filtered out) to prevent overlaps with our context-dependent datasets. We retain only unseen examples in our evaluation sets.
For OASST-1, we only include examples with an average human rating higher than 0.5 (i.e., the higher the rating is, the higher the quality is) and that are rated by at least two annotators. To refine ShareGPT, we include only examples with responses that are longer than 10 words, split by whitespace. For other instruction following training datasets, we exclude examples with empty instructions or responses.



  §.§.§ Context-Dependent Datasets

We aim to evaluate the model's generation for faithfulness to the given input context. We select a range of context-dependent datasets from three task domains: (1) extractive QA including NQ <cit.>, BioASQ and SearchQA, taken from the RobustQA benchmark <cit.>; (2) abstractive QA with MS MARCO <cit.> where the answers are well-formed sentences grounded in context; and (3) abstractive summarization including CNN DailyMail <cit.> and WikiSum <cit.>. BioASQ, SearchQA and WikiSum are hold out for evaluation and the rest are for training. One crucial advantage of using these context-dependent datasets is that they provide us with a reliable way of measuring faithfulness, in terms of how well the response is grounded in the given context.



  
Data Pre-processing For QA datasets, we include five retrieved passages maximally as the context,[Details about the retrieving process can be found in the original papers of RobustQA <cit.> and MS MARCO <cit.>.] where the gold answer is at least mentioned in one of the passages. For MS MARCO, we only include examples where there exist at least one well-formed answers. In cases involving multiple retrieved passages, we concatenate all passages with line breaks inserted between them. 



 §.§ Evaluation Metrics

In the context of our datasets, we evaluate our models with three metrics: instruction following score, faithfulness score and task performance score. The standard methods for measuring instruction following and faithfulness of language models are subject to ongoing debate. In this work, we employ  widely adopted approaches for these measurements and compile a set of metrics for more stable evaluations.[Throughout the paper, we sample a subset of the full evaluation data which include 6,000 examples (1,000 examples from each context-dependent evaluation set), and sample 300 examples (100 examples from each instruction following evaluation set) due to limited compute resources.] To present our findings, we report macro-averaged results across all test datasets.



  
Instruction Following Score We adopt the commonly used evaluation paradigm proposed by LLM-as-a-Judge <cit.>, and zero-shot prompt GPT-4 to provide a rating followed by a short explanation (i.e., named as LLM-as-a-Judge (R) in the paper).[We use .] For the GPT-4 evaluator, we set the temperature to 0 for stability with a maximum generation length of 512. We check instruction following scores only for instruction following evaluation datasets. See app:eval_human for our actual evaluation prompt.



  
Faithfulness Score For extractive QA datasets, we utilize the span coverage score as our metric (i.e., whether the predicted answer is a span within the context). We apply standard normalization to both the predicted answer and the context (see app:eval_faith for details). A score of 1.0 is assigned if the span is covered, and 0.0 otherwise. Additionally, we include unigram and bigram coverage for selected datasets to further refine our faithfulness evaluation in the Appendix (see  fig:paper_quandary_stage_1 and fig:paper_qa_summ_stage_2). For abstractive QA and summarization datasets, we employ SummaC-ZeroShot (SummaC-ZS; <cit.>) to assess whether the provided context (with the question concatenated as a prefix for QA datasets) entails the model-generated answer. Specifically, we segment both the context and the answer into sentences and then use a pretrained NLI model to compute an entailment score between pairs. These NLI scores are aggregated across pairs using max-pooling.



  
Task Performance Task performance complements the aforementioned metrics, as task performance is generally expected to decline when models are fine-tuned across domains. We utilize Exact Match (EM)
for extractive QA datasets and the ROUGE-L score <cit.>
for abstractive QA and summarization datasets.



 §.§ Instruction Template


Our datasets follow the same instruction-tuning template format as used in the Alpaca setting <cit.>. This template includes a header that outlines general guidelines, followed by task-specific instructions, as illustrated in tab:instruction_template. For QA tasks, the question is presented after the retrieved passages.



  
Objective-Aligned Instructions We design our task-specific instructions to minimize objective-conflict among datasets. For example, in prompting our model for a context-dependent extractive QA task, we explicitly instruct the model to “extract a specific text span from the given passages”. This template helps models to reduce hallucination when fine-tuned with instruction following datasets: as models getting better at understanding human instructions, they also get better at understanding how to extracting a span which ensures the answer to be faithful. The instructions for all datasets are depicted in tab:task_instruction. Our task-specific templates ensure there is no objective conflict when we fine-tune our models with mixed of datasets. We use the same template for training and evaluation for each task.












 §.§ Two-stage Fine-tuning Paradigm

To understand the trade-off between instruction following and faithfulness when training LMs with both objectives, we formulate a two-stage fine-tuning paradigm to answer our research questions (as illustrated in fig:inoculation_diagram).[Our approach is akin to the data inoculation paradigm proposed by <cit.>, albeit with significantly larger models and datasets.] For our first pipeline, we initially fine-tune our LM with context-dependent datasets that require grounding. We then take the best checkpoint from the initial stage to further fine-tune it on instruction following datasets (CD→IF). Conversely, in our second pipeline, we fine-tune instruct-tuned LM (e.g., Vicuna-7B) with context-dependent datasets (IF→CD). For both pipelines, we measure instruction following and faithfulness scores before and after training, to gauge the impact of the second-stage training on both capabilities. We follow this paradigm to find evidence of the trade-off in sec:finetune_instru and sec:finetune_ctx.



  
Models We use two models in our two-stage fine-tuning paradigm. We use a base LM LLaMA-7B <cit.>, one of the most widely used open-source LM, for the CD→IF pipeline. For our IF→CD pipeline, we use Vicuna-7B off-the-shelf as our instruct-tuned LLaMA-7B without retraining one from scratch. Vicuna-7B is one of the most competitive open-source chat-model, and is a fine-tuned LLaMA-7B on conversational data from ShareGPT <cit.>.
Our hypothesis and paradigm are transferable to other base LMs at different scales, although we pick these two models as they are among the first open-sourced LMs during the time frame of this project. Other experimental setup details are included in app:setup.










§ DOES FINE-TUNING WITH INSTRUCTION FOLLOWING DATA HURT FAITHFULNESS?
 

To answer this question, we follow the first pipeline outlined in Figure <ref>, where we take our LLaMA-7B that is fine-tuned on context-dependent datasets and further fine-tune it with instruction following datasets. Our results are shown in Figure <ref>. First of all, instruction following scores increase drastically as expected, from 0.30 to 0.74. Meanwhile, faithfulness scores on abstractive datasets drop 33.0% from 0.82 to 0.55, while the task performance is largely maintained (from 0.34 to 0.32). For extractive datasets, both faithfulness scores and task performance see a relatively small drop. To provide a fine-grained view of these datasets, we provide additional analysis of per-task and per-metric changes of faithfulness score as well as task performance in app:additional_alignment_t and app:quandary_of_mix.

One potential confounding factor for faithfulness scores dropping on abstractive tasks is the length of the generated response.
The model could simply have generated longer responses (fig:paper_seq_length_abstract_stage_2) as a result of training on intruction tuning data with longer responses (Table <ref>).
To rule this out, we re-evaluate our models separately with two contrasting groups: evaluated with only those examples with shorter generation length (less than or equal to), and those examples with much longer generation length (100 more tokens) compared with the golden answer. As shown in Figure <ref>, both short and long generations see very substantial drops in faithfulness, while longer generations indeed see larger drops. This nevertheless supports our conclusion that instruction following training hurts faithfulness.






§ DOES CONTEXT-DEPENDENT FINE-TUNING HURT INSTRUCTION FOLLOWING?
  
To answer this question, we fine-tune Vicuna-7B (LLaMA-7B fine-tuned on ShareGPT) on context-dependent datasets. Figure <ref> shows our results. As expected, our model becomes more faithful, with improved task performance. This improvement is partly because Vicuna-7B does not follow our instruction template out-of-the-box. On the other hand, instruction following scores drop by 37.9%, suggesting our model becomes much less human-aligned. Similar to previous findings, one natural concern is that context-dependent training makes our model generate much shorter responses, which leads to lower instruction following scores. To rule out this concern, we randomly sample 200 examples from the Alpaca-15K dataset and only keep those with a minimal sequence length change (a maximum of 10 new tokens) compared to the corresponding Vicuna-7B zero-shot generation. As shown in Figure <ref>, instruction following scores consistently drop considerably for the filtered setting, regardless of length (36.8% vs. 22.0%). Overall, context-dependent fine-tuning negatively affects the instruction following score, as we expected.



§ : RECONCILING INSTRUCTION FOLLOWING AND FAITHFULNESS

As shown in the previous two sections, there are clear objective discrepancies between diverse instruction following datasets and context-grounded ones. In this section, we explore two different mitigation strategies, starting from the MTL baseline to our novel method based on  (), which significantly outperforms MTL.



  
Our MTL Baseline Our MTL simply mixes up the instruction following as well as context-dependent datasets. We up-sample smaller datasets to have an equal number of examples per dataset.



 §.§ Our Method

 starts with a model that is fine-tuned on a mixture of instruction following and context-dependent datasets and reconciles the two objectives with the following steps (fig:our_method_diagram): 



  
Sample Generations For a random subset of the training datasets, we sample generations from the checkpoint with different decoding settings. Specifically, we focus on two hyperparameters by randomly changing one of them at a time by enumerating all possible values. For the decoding temperature, it takes on a value from {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7}. For top-k, we set it to take on a value from {5, 10, 20, 50, 70, 90, 100}. When we vary the temperature, we fix top-k to be 0. Similarly, when we vary the top-k, we fix the temperature at 0. For each example, if we run the sampling procedure once, it will sample 7 examples in total.



  
External Judges We then use a set of external judges to rate our collected generations as in sec:eval_metrics. These judges are offline evaluators. We evaluate generation based on instruction following score, faithfulness score and task performance. One potential limitation is that our model may overfit to existing judges on seen datasets. We thus include unseen datasets to test generalizability.



  
Top-1 Weighted Score For each generation, we collect a set of scores from our external judges. Then, we take a weighted sum of these scores before ranking the generations and picking the top rated one. The score is weighted score of task performance s_task, instruction following s_instr and faithfulness scores s_faith,

    score=s_task+2.0*(𝕀_instr*s_instr + 𝕀_faith*s_faith)

where we use 𝕀_instr and 𝕀_faith to indicate whether the example is from our instruction following datasets or context-dependent datasets.
We pick the top rated sample per example and combine them into a continued fine-tuning dataset as described next.



  
Continued Fine-tuning For , we randomly sample training data from each of our four dataset in tab:data_stats and collect 2,000 additional fine-tuning examples per dataset. In total, our new fine-tuning dataset has 8,000 examples. With this small collected dataset, we further fine-tune our starting checkpoint model for a single epoch with a smaller learning rate to avoid overfitting. This continued fine-tuning step is very lightweight as the training data is usually less than 1% of the MTL training step. Overall,  resembles a reject sampling based preference learning paradigm, which has been proven effective <cit.> while drastically saving training costs as well as increasing stability. Other experimental setup details are included in app:setup.



  
Supercharged  (-S)
In addition, we evaluate  with a different setting to test the impact of the quality of our continued fine-tuning dataset. Specifically, we supercharge the quality of our additional fine-tuning dataset by sampling 1× more generations. We swap our instruction following judge from GPT-4 to the weaker ChatGPT[We use .]. One potential benefits is that ChatGPT is a weaker judge with a lower recall for good generations. As a result, examples rated high by ChatGPT may have higher quality.
While sampling for generation, we decrease the curated fine-tuning dataset by 3-fold (2,000 examples in total). We fine-tune our checkpoint model with the same setup.





 §.§ Results

tab:our_method_result shows results on models fine-tuned with other baselines: 
(1) Vicuna-7B: Evaluated with Vicuna-7B in a zero-shot manner. 
(2) w/o MTL: We fine-tune Vicuna-7B with context-dependent datasets without any MTL objective. The purpose is to establish a potential upper bound of faithfulness score without any further training for instruction following.
(3) w/ MTL: We fine-tune Vicuna-7B with baseline MTL by mixing instruction following and context-dependent datasets together. 
(4) w/ : We fine-tune Vicuna-7B directly with our collected continued fine-tuning dataset (8,000 examples) from our  pipeline without MTL training first. Note that the curated dataset comprises of sampled generations from our MTL checkpoint. The purpose is to evaluate whether MTL is necessary for model improvements.
(5) w/ MTL+: We follow fig:our_method_diagram to fine-tune our MTL checkpoint with curated dataset.
(6) w/ MTL+-S: We follow fig:our_method_diagram to fine-tune our MTL checkpoint with the high-quality version of our curated dataset (2,000 examples).

We evaluate our models on both seen and unseen testing datasets. As shown in tab:our_method_result, fine-tuning Vicuna-7B with a mixture of datasets close the gap on instruction following score substantially (from 0.49 to 0.75) while leaving headroom across the board. Next, models fine-tuned with  significantly outperform the MTL baseline on both seen and unseen testing datasets. Our results also suggest directly fine-tuning Vicuna-7B performs worse compared to fine-tuning a model checkpoint after MTL training. Last but not least, our results with -S provide strong evidence that data quality is more important than data quantity and that using multiple iterations of sampling helps, as -S achieves similar or better performance with 3-fold less training data. Due to space limit, we present qualitative model generations in app:example_generation.



§ RELATED WORK




  
Instruction Following of LMs 
There are a variety of instruction following training datasets <cit.> covering tasks from wildly different domains such as poetry creation to context-dependent summarization. In this paper, we focus on a particular trade-off between instruction following and model faithfulness, which are fundamental objectives for modern LMs. In addition to training datasets, LM generations are often assessed by human experts <cit.> or model scorers <cit.>. We use GPT-4 to access instruction following scores as in <cit.>.




  
Faithfulness and Groundedness of LMs Being faithful or grounded is crucial for tasks like context-dependent QA or summarizations, reducing hallucinations for LMs <cit.>
Unlike task performance, faithfulness measures whether generated answers are based on the given context <cit.>. For context-dependent QA and summarization benchmarks, common metrics include subsequence-based lexical matching <cit.>, natural language inference <cit.>, and more recently, LLMs as factuality scorers <cit.>. We use lexical matching and NLI to measure faithfulness by checking whether the answer is grounded.



  
Instruction Following Training with LMs Base LMs are often tuned to follow human instructions <cit.>. Various post-hoc fine-tuning techniques have been proposed to align base LMs with human preferences, such as supervised fine-tuning on instruction-following datasets <cit.> or variances of RLHF and RLAIF <cit.>. These techniques can require training policy and reward models which is not cost-efficient. Inspired by alignment training with reject-sampling <cit.>, we propose a simple yet effective method that uses reject-sampling by self-instruct. On the other hand, there exist various advance MTL techniques <cit.> prior to the development of LMs. However,  is the only variant that leverages LM's self-instruct generations to further fine-tune the LM. More importantly,  is complimentary to different instruct-tuning techniques by leveraging the LM and external evaluators to generate high-quality continued fine-tuning datasets. 
Compared with recent works showing how instruct-tuning may cause hallucination <cit.>, we provide a more rigorous analysis with our two-stage fine-tuning paradigm and focus on the interaction between instruction following and faithfulness.



§ CONCLUSION

In this paper, we demonstrate a clear trade-off between instruction following and faithfulness when fine-tuning LMs on datasets with differing objectives.
To alleviate this trade-off, we proposed , a simple yet effective iterative method that significantly outperforms the MTL baseline in both instruction following and faithfulness scores. The lightweight and iterative nature of  makes it extensible for future refinement at minimum cost and integrable with recent instruct-tuning techniques. We contribute to the broader goal of creating more reliable, accurate, and user-aligned language technologies.



§ LIMITATIONS
 
The limitations of our work are as follows:

  
  * Our study primarily focuses on the LLaMA-7B and Vicuna-7B models, that are among the best open-source models at the time of this work. While we posit that our findings and the proposed  method could generalize across other language models, our findings could remain speculative without evaluating on more current model types (e.g., LLaMA-2, Mistral or Mixtral at various scales).
  
  * The datasets chosen for fine-tuning and evaluation, though comprehensive, are not exhaustive. There are other interesting datasets that are not covered in this study. For instance, long-form QA where the answers are much longer than 1-2 phrases or sentences. Our instruction following datasets can also be further categorized into creativity-driven, world-knowledge driven and others to help us to disentangle discrepancies in objectives better across datasets.
  
  * Our evaluation relies heavily on automated metrics and external judges like external LMs for assessing instruction following and faithfulness. While these methods are standard, they cannot fully encapsulate the nuanced understanding and preferences of human evaluators. For future research, evaluating responses with human annotators would provide additional validations.
  
  * Although the purpose of our study is to study the objective discrepancies in the datasets and come up with mitigation strategies without another novel training paradigm, it would strengthen our results if we can compare our method with more recent alignment training methods.
  
  * Although we evaluate  on unseen datasets, our method still has the potential to overfit to certain evaluators. Future work may use a different set of evaluators for a more robust evaluation. Human evaluation is challenging for our unseen testing datasets, such as WikiSum, because the input and response are extremely long, which could also make human ratings unstable.




§ ETHICS STATEMENT

In this paper, we delve into the nuances of instruct-tuning LMs, a process that involves balancing a trade-off between instruction following and faithfulness. We feel that the immediate ethical and societal consequences of our research may be limited, and none which we feel must be specifically highlighted here.







§ APPENDIX







 §.§ Evaluation Metric: Instruction Following Score

tab:gpt4_eval_prompt shows our actual prompt template for evaluation. We follow the prompt template provided in the LLM-as-a-Judge <cit.> paper. We normalize the return rating to [0, 1].







 §.§ Evaluation Metric: Faithfulness Score




  
Text Normalization We use regular expressions to replace spaces around hyphens, slashes, and before “'s”. We then remove all the articles (e.g., “a” and “the”) and punctuations. We lowercase all letters for simplicity.



  
Common Failure Mode The authors also human label model's generation to check whether model's generations are paraphrased version of the golden answers. The most common failure mode is the model extracting a non-existent span or a wrong span. This supports our findings of models being unfaithful.



 §.§ Experimental Setup

We train our model for a maximum of a single epoch[Experiments with up to three epochs showed minimal changes in results.] across all training jobs. We up-sample the smaller datasets to match the number of examples in the larger ones when combing datasets for training. The learning rate is set at 1× 10^-5 with a batch size of 16 for our faithfulness-driven training. For training jobs involving instruction following datasets, we increase the batch size to 32. Training is conducted using  precision with a maximum sequence length of 2048. The weight decay is set to 0.05, with a  learning rate scheduler and a warm-up ratio of 0.03. We save checkpoints every 100 training steps and evaluate them based on perplexity scores on the evaluation set. The best-performing checkpoint is then selected for the next stage of training. Our models are trained using the stage-3  library. We train each model with three random seeds and average the results for consistency. For each training job, our models are trained on 8×A100 GPUs within a single-node setup, with the total training time not exceeding 24 hours. For model generation, we employ greedy decoding with a maximum generation length of 480, which aligns with the maximum response length across the training datasets.

For the contined fine-tuning step in , we use a smaller learning rate of 8×10^-6 and keep other settings the same.



 §.§ Additional Analysis on Instruction Following Fine-tuning


fig:paper_alignment_stage_1 shows how instruction following scores vary during fine-tuning LLaMA-7B with context-dependent tasks. fig:paper_alignment_stage_2 shows how instruction following scores vary during the second-stage fine-tuning with instruction following datasets.



 §.§ Quandary of Mixed Training on Abstractive and Extractive QA and Summarization Datasets


fig:paper_quandary_stage_1 shows faithfulness score and task performance across related datasets when we fine-tune LLaMA-7B on context-dependent datasets. One suprising finding is that throughout the fine-tuning process, the faithfulness scores on extractive QA datasets gradually decrease, while task performance scores gradually increase. On the other hand, this trend is not salient for abstractive tasks as shown in fig:paper_abstract_stage_1.



 §.§ Examples of Actual Instructions


From tab:nq_example to tab:alpaca_example, we show actual model inputs from each testing datasets.



 §.§ Qualitative Examples


tab:first_pip_example and tab:second_pip_example show two qualitative examples of actual model generations from our experiments.
































































 Paper Title: Adaptive Retrieval-Augmented Generation for Conversational Systems.


Confining modeling of quark propagator
    X.L. Shang
    August 5, 2024
======================================

§ ABSTRACT 
Despite the success of integrating large language models into the development of conversational systems, many studies have shown the effectiveness of retrieving and augmenting external knowledge for informative responses. Hence, many existing studies commonly assume the always need for Retrieval Augmented Generation (RAG) in a conversational system without explicit control. This raises a research question about such a necessity. In this study, we propose to investigate the need for each turn of system response to be augmented with external knowledge. In particular, by leveraging human judgements on the binary choice of adaptive augmentation, we develop RAGate, a gating model, which models conversation context and relevant inputs to predict if a conversational system requires RAG for improved responses. We conduct extensive experiments on devising and applying RAGate to conversational models and well-rounded analyses of different conversational scenarios. Our experimental results and analysis indicate the effective application of RAGate in RAG-based conversational systems in identifying system responses for appropriate RAG with high-quality responses and a high generation confidence. This study also identifies the correlation between the generation's confidence level and the relevance of the augmented knowledge.









§ INTRODUCTION




Recently, the advancement of Large Language Models (LLMs) has significantly improved conversational systems, enabling the generation of natural and high-quality responses <cit.>. Despite these advancements, recent studies have identified several limitations on the simple use of LLMs to address conversational tasks <cit.>. These limitations include the lack of up-to-date knowledge <cit.>, the generation of non-factual or hallucinated content <cit.>, and restricted domain adaptability <cit.>. These issues can hinder the development of conversational agents with satisfactory user experience. To address these identified challenges, a common approach is to retrieve and augment LLMs with external knowledge to enhance the conversational response, making them more accurate, reliable, and adaptable to different domains <cit.>. For example, <cit.> demonstrated that using a dense retrieval model (DPR) <cit.> to retrieve relevant knowledge for augmentation can significantly reduce the hallucination rate, according to a corresponding human evaluation. Similarly, <cit.> showed that leveraging a graph-structured knowledge base can boost the reasoning ability and domain generalisability of task-oriented conversational agents. These achievements of knowledge-augmented techniques highlight a promising direction for enhancing conversational agents and address the current limitations. 

However, while implementing retrieval augmentation to a conversational system for improved response, we question the necessity of knowledge augmentation for every turn of system responses. To develop effective human-computer conversations, it is essential to provide factual and relevant responses, offer appropriate amount of information, and not unnaturally drive and shift the conversation to non-relevant topics <cit.>. We argue that overusing external knowledge could result in system responses against these core criteria. Figure <ref> presents a conversation example that shows how the system response to a generic user utterance about suggesting activities can vary with and without augmented knowledge. The knowledge-augmented system response is being information conditioned with limited diversity and assuming specific user preferences.

In contrast, without the addition of external knowledge, the system response is more diverse and natural in this early stage of a conversation. This indicates that misusing external knowledge can lead to problematic system responses and a negative user experience. 

To address this, we investigate an adaptive retrieval-augmented generation solution for effective conversational systems. In particular, motivated by the gate function in long-short term memory models <cit.>, which explicitly controls the use of input and memory, we propose a binary knowledge gate mechanism, called RAGate, to manipulate the use of external knowledge for a conversational system. To model the conversation context and accurately estimate the need for augmentation, we leverage the human labels as ground truth and develop RAGate by exploring the use of recent advanced language models or constructing attention neural gate models. To validate the effectiveness of RAGate, we conduct extensive experiments on an annotated Task-Oriented Dialogue (TOD) system dataset, KETOD, that builds upon the SGD dataset with TOD-spanning
16 domains, such as Restaurant and Weather. The experimental results show that RAGate enables conversational systems to efficiently use external knowledge at appropriate conversation turns, producing high-quality system responses. In particular, by modelling the uncertainty and confidence level of the system – which correlates with the likelihood of hallucinated output <cit.> – we show that the "always" augmentation of external knowledge can significantly increase generation uncertainty and the risk of hallucination. After applying RAGate, we can effectively control the conversation system to make confident and informative responses. In addition, by varying the use of knowledge snippets in different relevance levels, we also observe the positive correlation between the calculated confidence score and the relevance of augmented knowledge, which can be valuable for many future studies. 





§ RELATED WORK

In the pipeline of knowledge-augmented generation for a conversation system, two main components are identified: the knowledge retriever and the response generator. Existing studies have improved conversational responses to different extents by improving one or both components <cit.>. 

Knowledge Retrieval: Several studies have explored the use of dense passage retrieval techniques <cit.> and public search service for effective retrievers <cit.>. For example, <cit.> retrieved Wikipedia passages through a database interface and then ranked them according to statistical relevance, calculated by TF-IDF, or semantic relevance as per cosine similarity. Similarly,  used a search engine API to retrieve relevant knowledge but first transformed the dialogue context into a natural search query using an encoder-decoder model before searching. 

Joint Optimisation of Retriever and Generator: On the other hand, another thread of research studies has explored joint optimisation approaches. For instance, <cit.> introduced a retriever-generator architecture that aims to improve the performance of Task-Oriented Dialogue (TOD) systems by using a dual-feedback mechanism. The retriever identifies relevant knowledge from a database, while the generator uses this information to create appropriate system responses. The feedback from the generator is further used as pseudo-labels to train the retriever to select pertinent information. <cit.> introduced a training method based on maximal marginal likelihood. This method jointly optimise a perceptive retriever and the response generation in a feedback loop. The proposed approach incorporates meta-knowledge, which guides the generator to improve the utilisation of knowledge and, consequently, the quality of the generated responses.  <cit.> further advance the retriever by proposing SUbgraph Retrieval-augmented GEneration (SURGE), which employed a graph neural network (GNN)-based context-relevant subgraph retriever. SURGE incorporates contrastive learning to optimise the latent representation space, ensuring that generated texts closely resemble the retrieved subgraphs. 


Despite the richness of existing retrieval-augmented generation techniques for conversational systems, they commonly hypothesise that every conversation turn needs external knowledge. However, the necessity of augmenting every turn of the conversation with external knowledge remains questionable. A relevant thread of work that aims to answer this question is the introduction of the knowledge-seeking turn detection task using the DSTC-9 dataset <cit.>, and the follow-up studies, such as <cit.>. However, this task is to identify the turns in conversations injected by human workers about knowledge enquiry instead of identifying the system responses that require knowledge augmentation for improvements. This research gap highlights the value and novelty of this study, which investigates the adaptive use of retrieval-augmented generation for advanced conversational systems.




















§ METHODOLOGY




 §.§ Problem Formulation

This study addresses the challenge of effectively identifying conversation turns that require augmentation of external knowledge. In particular, we aim to develop a gate mechanism that dynamically determines when to search for external knowledge to ensure natural, relevant and contextually appropriate responses. First, we define the task of user-system conversation. Let D = {d_1, d_2, ...,d_|D|} be a set of user-system dialogues, and each dialogue d comprises a sequence of interactions between users and systems (i.e., d = {u_0, s_0, u_1, s_1, ..., u_T, s_T}) with varying lengths. Here, u_t and s_t denote the user utterance and system response at the t-th turn, respectively. The conversational context up to turn t can be formulated by aggregating the previous user-system interactions, i.e., c_t = u_0, s_0, .., u_t. With this context information c_t, the conversation system can augment it with a list of retrieved external knowledge, e_t,k, where k represents the ranking cutoff for the retrieved knowledge. Hence, the binary gate mechanism proposed in this study, deciding the knowledge augmentation, can be formulated as f(c_t) = {0,1} or f(c_t, e_t,k) = {0,1} if the external knowledge e_t,k is considered. Then, the follow-up response generation function g(·) can be formulated as follows:

    g(·) = 
            g(c_t, e_t,k)    if f(c_t) or f(c_t, e_t,k)
    
            g(c_t)    otherwise.

Hence, by evaluating and estimating the necessity of augmenting with external knowledge, we dynamically update the conversational response generation accordingly.





 §.§ RAGate Gate Mechanism

To effectively estimate the need to use external knowledge and implement adaptive retrieval augmented generation for a conversation system, we introduce our proposed gate mechanism, RAGate, that uses the conversational context and, optionally, the retrieved external knowledge to predict the binary choice of using external knowledge. In particular, we explore three RAGate variants that are implemented by the use of Large Language Models (LLMs) with devised prompts, with parameter efficient fine-tuning (e.g., QLoRA <cit.>) and the construction of an end-to-end multi-head attention encoder. This exploration is motivated by the recent advancement of transformer-structured neural models in natural language processing. In Figure <ref>, we illustrate the application of RAGate and its three variants. We describe each of these three variants to clarify the use of RAGate:

RAGate-Prompt:  As denoted by <cit.>, a language model can effectively adapt to new tasks by using a natural language prompt that explains the process to address the tasks without extra training. Hence, we can formulate 
a gate function f(·) as f(y|c_t) = f(y|Θ, c_t, p), where Θ denotes the used language model with its pre-trained weights and p is the devised natural language prompt. Alternatively, if the retrieved knowledge is also involved in prediction, we have f(y|c_t) = f(y|Θ, c_t, e_t,k, p). Specifically, we explore two types of prompts: zero-shot and in-context learning. Zero-shot prompts describe the task that uses the conversational context and, optionally, the retrieved knowledge to generate a response with binary feedback. As for the in-context learning prompts, we augment the zero-shot prompts with illustrative examples. We show the set of prompts in Appendix <ref>. 

RAGate-PEFT:  Despite the high adaptability of the language model with devised prompts, we further explored the use of instruction tuning on language models with a parameter-efficient fine-tuning method (i.e., QLoRA <cit.>) to meet the goal of an effective gate function. QLoRA is built upon the known Low-rank Adapter (LoRA) <cit.>, which keeps the pre-trained weight matrix W_0 frozen and addresses the gradient updates of the weight matrix Δ W through low-rank approximation (i.e., Δ W = BA, where B and A are the result of lower-rank decomposition on Δ W). Hence, the forward pass during the model training can be updated from h = W_0x+Δ Wx to h = W_0x + BAx. QLoRA <cit.>, which is used in this study, further quantises the language model into a 4-bit NormalFloat data type and leverages the page-to-page transfer between the CPU and GPU to further avoid memory spikes. To implement RAGate-PEFT, we format the train data with devised instructions, joined with paired inputs and outputs for developing parameter-efficient fine-tuned large language models. In particular, we provide a set of instruction-input-output triples for model training. The input can vary with the provision of a set of available features. Apart from the use of the conversational context (contx), we also include the system response (resp), synthetic responses generated by the language model (syn-resp) due to the missing responses as input in the practical scenario, the name entities within the incoming responses (ner), retrieved knowledge (know) and the description of the knowledge source, e.g., the WikiHow website (source). By using various combinations of inputs and customising the corresponding instructions, we explore the effectiveness of the resulting learned language models that implement the RAGate-PEFT. 






RAGate-MHA:  Apart from the use of pre-trained language models and further fine-tuned language models, we also explore the introduction of a multi-head attention neural encoder to model the context as input and estimate the augmentation necessity (i.e., RAGate-MHA). Here, we describe the model structure of RAGate-MHA. At first, as denoted by <cit.>, the attention mechanism is formulated as the interaction between three objects, queries Q, keys K, and values V: Attention(Q,K,V) = softmax(QK^T/√(d_k))V. To estimate the necessity of augmentation, we fit the context and the retrieved knowledge into the roles of these three objects. Specifically, we include the setups of (1) using context only (contx) or (2) using the concatenated context and retrieved knowledge (contx ⊕ know) as queries, keys, and values, and (3) using the context as queries and interact with the retrieved knowledge as keys and values (contx × know).

Next, following <cit.> in the encoder construction of a transformer model, we encode the inputs via an input embedding layer into latent vectors and a position encoding layer to encode the order of tokens in the sequence. After that, we leverage the multi-head attention to learn attention weights on the inputs and then followed by a feed-forward network: 

    FFN(x) = max(0, xW_1 + b_1)W_2 + b_2

where W_1 and W_2 are two learned parameter matrics with two bias terms (b_1 and b_2). Both multi-head attention and feed-forward neural modules are followed by residual connection <cit.> and layer normalisation <cit.>. Unlike the introduction of another decoder module that addresses the sequence-to-sequence generation in <cit.>, we followed the encoder output with a linear projection module and a softmax function for our binary classification task.



§ MODEL TRAINING AND EVALUATION SETUPS

We evaluate the performance of introducing RAGate according to its binary classification performance and the effectiveness of the resulting response generation. Specifically, we use the KETOD dataset <cit.>, which has fully annotated 5,324 dialogues and 52,063 turns of conversations. In particular, it is associated with 33,761 knowledge snippets to be retrieved and augmented. In addition, KETOD was developed with human labels on turns of conversations (around 12.1% of turns) about the need for augmenting with retrieved knowledge snippets for a natural and informative system response. Hence, we use these human labels as natural ground truths when evaluating RAGate. It is worth indicating that many current knowledge-augmented conversational datasets often ground their conversations on the knowledge snippet, such as Wizard of Wikipedia <cit.> and CMU_DoG <cit.>, which makes them not a natural fit to be investigated in this study.



Due to the limited computational resource availability, we explore the use of Llama-v2-7B and Llama-v2-13B to implement RAGate-prompt and fine-tune Llama-v2-7B for RAGate-PEFT. We implement QLoRA using the PEFT library <cit.> and set the lower rank to 16. As discussed in Section <ref>, we have various input features to be combined for performance optimisation. We begin with the use of context only, then concatenate the context with the real response (contx-resp), with the synthetic response and recognised entities (contx-syn-resp-ner) and further extend with the use of retrieved knowledge (contx-syn-resp-ner-know) or the source of knowledge (contx-syn-resp-ner-source). Specifically, we retrieve the relevant knowledge by exploring the use of TF-IDF and a learned BERT ranker. We evaluate their performance with the classic Recall@1 and Recall@3 on the test collection. We use a shallow cutoff because we only use top-relevant knowledge snippets for augmentation. Table <ref> shows their retrieval performance. According to the leading performance of BERT-Ranker, we augment knowledge with its retrieved top 3 relevant knowledge snippets (i.e., k=3). Regarding the development of RAGate-MHA, we explore the combinations of 2 to 8 layers, 2 or 4 heads and the embedding size in [64, 128, 256] for the best classification accuracy. We report the precision, recall, F1, Area Under Curve (AUC) and the False Discovery Rate (FDR) as the main measures to show the classification effectiveness. 

Next, we further deploy the best-performing RAGate gate function to update the KETOD dialogue system <cit.>, which uses GPT-2 <cit.> as the backbone model. To highlight the effect of various augmentation setups, we use the context with the gold action without extra prediction as input to KETOD. Then, we compare the resulting performance to the KETOD model without knowledge augmentation and augmenting every system response as baselines. To report the response generation effectiveness, we report how close the response is to the ground truth via BLEU, ROUGE-1/2/L and BERTScores and the confidence score calculated by the minimum probabilities of individual tokens that compose the response.
As argued by <cit.>, this calculated confidence score can highly correlate with a language model's likelihood of generating hallucinated responses.  We trained our models and conducted the evaluations on one machine with one NVIDIA 4090 GPU. 




§ RESULTS AND ANALYSIS


























































 §.§ Augmentation Need Classification

First, we evaluate the classification accuracy of our developed RAGate gate methods for addressing the adaptive RAG to system responses. Table <ref> presents the classification performance of RAGate baselines while evaluated on the test collection of the KETOD dataset, which includes rich human labels on the use of RAG for response generation. As discussed in Section <ref>, we explore the development of RAGate with three variants: the use of LLM prompting (RAGate-Prompt), parameter-efficient fine-tuned LLMs (RAGate-PEFT), and a neural classifier with Multi-Head Attention structure (RAGate-MHA). 

RAGate performance with LLM prompting versus fine-tuning. By comparing the corresponding performance reported in Table <ref>, we observe that, on average, fine-tuning a Llama-2-7B with QLoRA (i.e., RAGate-PEFT) can significantly outperform RAGate-Prompt. For example, by looking at the RAG-PEFT with context-only input, without using extra input features and instruction updates, it can outperform all RAG-Prompt approaches by a big margin (e.g., 0.4082 versus the highest 0.1230 F1 scores). This reflects the difficulty of this adaptive knowledge augmentation task, which can not be properly addressed by prompting a general pre-trained language model. In particular, the use of larger language models and the in-context learning setup, which often result in improved performance <cit.>, can not guarantee the enhancement of models' classification accuracy regarding this classification task.

Regarding the performance of RAGate-PEFT approaches, by first examining the effect of using synthetic response and recognised name entities, we observe significantly improved precision (0.5203 to 0.6818) but with the cost of lower recall (0.3359 to 0.2321). In addition, when we add the retrieved knowledge to the input features for prediction, we observe a significant performance drop across all evaluated aspects.  This can be caused by the additional complexity introduced by the included retrieved knowledge snippets. Furthermore, we also explored the performance impact of naming the source of the knowledge snippet. We use wikiHow[https://www.wikihow.com] in this study, which can provide rich task instructions for offering informative task-oriented system response <cit.>. However, the fine-tuned model cannot reasonably connect the promised rich resource from the knowledge source and the prediction of augmentation necessity.  


RAGate Performance between fine-tuned LLM and MHA classifier. Next, by comparing the experimental results of RAGate-MHA and RAGate-PEFT in Table <ref>, we observe a wide-margin recall improvement using RAGate-MHA, reaching a minimum recall of 0.52, but with significantly lower precision accuracy. In Table <ref>, we also include the use of both the context and the initial system responses (i.e., MHA([contx, resp])) for additional insights. We can observe that a higher precision can be achieved but the use of response does not improve the recall performance. These results are consistent with the observed performance of RAGate-PEFT, which further encourages the use of a synthetic response due to the unavailability of a system response in a practical scenario. In addition, we also observe a similar performance drop when including the retrieved knowledge snippets for classification. Even though the RAGate-MHA model, using the interaction between context and retrieved knowledge snippets, can achieve the highest recall of 0.5835, it can not outperform the RAGate-MHA using context-only on other metrics. Hence, considering the similar F1 and AUC performance levels of RAGate-PEFT and RAGate-MHA leads to a trade-off balance between precision and recall for the two groups of approaches. To further evaluate the classification effectiveness of RAGate, in Appendix <ref>, we provide a detailed discussion of a conducted user study that explores whether RAGate can also assess the potential contribution of retrieved snippets when predicting the decision for retrieval augmentation.







 §.§ Adaptive Augmentation Analysis

In addition to the classification accuracy, we also compare the choice of human workers and RAGate approaches in augmenting specific turns. Specifically, we analyse the frequency of augmentation in different positions of conversations and different domains covered in the KETOD dataset. We use the RAGate-PEFT (contx-(syn-resp)-ner) with the highest precision and RAGate-MHA (MHA(contx)) with the best overall performance in the above analysis as representatives for comparison. Figure <ref> presents the frequency in different positions. Due to the unequal number of conversational turns, we use the ratio to indicate the relative position. According to the reported results in Figure <ref>, most human augmentation selections happen at the beginning of a conversation. This trend is also effectively captured by both RAGate approaches, especially RAGate-MHA. This can be caused by the reason that a conversation is semantically coherent, and once sufficient additional information is provided at the early stage, the value of knowledge augmentation to the later turns is naturally lower. 


On the other hand, Figure <ref> presents the augmentation frequency over different domains. We observe that system responses about certain domains are selected more often by humans than other domains, such as travel, hotels, trains, flights, service and rental cars, which require access to additional information to assist the suggestion-making, and the domains, like movies, music, media, events that often include entities require enriched description. By looking into the performance of RAGate-PEFT and RAGate-MHA, RAGate-MHA can make aligned selections for humans. However, the RAGate-PEFT does not guarantee the identification of appropriate augmentation use and often presents fewer augmentations, apart from the travel domain. Hence, by considering both position and domain augmentation frequency, we conclude that RAGate-MHA can outperform RAGate-MHA and effectively capture the trend of augmentation needs.  



















     	



































 §.§ RAGate for Response Generation

To evaluate the effect of adaptive RAG for a conversational system, we use RAGate-PEFT (contx-(syn-resp)-ner) with the highest precision and RAGate-MHA (MHA(contx)) with the best overall performance in the above analysis, to support the adaptive retrieval augmented conversational response generation. Table <ref> presents the results of applying RAGAte to the KETOD model for adaptive knowledge augmentation when evaluated on the KETOD dataset. We include four types of adaptive augmentation, namely the use of RAGate and comparison to the random selection with equal numbers of selections, human choice, and the commonly used "all" augmentation. 
In addition, to explore the effect of varied quality of knowledge snippets, we also extend the evaluation of using the top-3 knowledge snippets ranked by different retrievers (i.e., BERT-ranker and TF-IDF) and the use of knowledge snippets at the 1st and 5th rank according to the BERT-ranker. Due to the space limit, we first present the results of using BERT-ranker retrieved and top-1 relevant knowledge and top-1 relevant in Table <ref> and show the full results in the Appendix <ref>. 

At first, without adaptive knowledge augmentation, we compare the choice of response generation without augmentation and with "always" augmentation (i.e., No-Aug versus Aug-All). In Table <ref>, we observe that by augmenting a total of 4,964 system responses in the test collection, the conversational model can generate more informative and effective responses according to the reported scores of BLEU, ROUGE and BERTscore. This aligns with the reported effectiveness of RAG in many existing studies. However, we also identify a significant drop in the model's generation confidence level. As denoted by <cit.>, a lower confidence level can correlate with a higher chance of generating hallucinated responses, which could be caused by the unnecessary use of external knowledge. Hence, to investigate the effectiveness of adaptive knowledge augmentation, we examine the impact of using RAGate. According to the reported experimental results in Table <ref>, the adaptive augmented response generation with fewer knowledge snippets can indeed result in a higher confidence level than Aug-All. 

Moreover, comparing the performance between RAGate and random selections shows that, considering equal numbers (230 or 787 according to the classification with RAGate) of system responses for augmentation, RAGate can further result in a higher quality of generated response. RAGate-MHA even enables results that are comparable to Aug-All's response quality, with only 787 turn augmentations instead of all 4964 turns. Specifically, the use of RAGate-PEFT, which identifies 230 turns of system responses for knowledge augmentation, can even outperform the random baseline that augments 787 system response turns with improved response quality. 
Apart from the improved response quality, RAGate also enables the conversational model to maintain a high confidence level and ensure faithful responses. Indeed, using RAGate-MHA, which augments 787 system responses, only lowers the average confidence score by 0.36%, instead of the 1.65% when randomly selecting an equal number of turns to augment. 

In addition, considering the use of different quality and amount of knowledge snippets for augmentation, we also include the use of the most relevant knowledge snippet according to BERT-ranker in Table <ref>. We observe that the use of different amounts of knowledge snippets in different relevance levels has a marginal effect on this learned dialogue system. However, we observe a significant difference in the confidence level. We observe that using only the most relevant knowledge snippet enables the Aug-All to suffer less from a lower confidence level. In particular, the application of RAGate can even increase the confidence level of the conversation system in response generation. This indicates that the confidence score can also correlate with the quality of the augmented knowledge snippets. This observation is further validated using knowledge snippets with fifth-ranking positions by BERT-ranker and the use of TF-IDF ranker.  
We include the full experimental results in Table <ref> and attached in the Appendix. 
These observations indicate the value of adaptive system response augmentation via RAGate in generating high-quality outputs, ensuring faithful responses, and potentially saving retrieval costs. We also show the value of using confidence scores to reflect the contribution of RAG.



§ CONCLUSIONS

Our study investigates a core research question about whether retrieval-augmented generation is always useful to a conversational system. To answer this research question, we propose adaptive retrieval-augmented generation for conversational systems and introduce corresponding gate functions, RAGate, for explicit control. A comprehensive set of experiments and results show the RAGate approaches can effectively identify augmentation needs. In addition, RAGate can capture human preference by augmenting the beginning turns of conversations, and RAGate can further identify knowledge augmentation for assisting suggestion-making and enriching description. When applying RAGate to conversational systems, we observe that it can ensure comparable quality of generated responses and enable the system to increase generation confidence for faithful outputs, especially with the appropriate use of relevant knowledge snippets.













§ LIMITATIONS

There are three limitations of this study. At first, due to the main focus of examining the adaptive retrieval-augmented generation for a conversation system. We only consider a few examples of retrieval techniques (TF-IDF and BERT-ranker), which can be further extended to recent retrieval techniques, such as dense passage retrieval for additional insights. The second limitation is the missing use of larger language models, such as GPT-4, due to the shortage of computational resources. Including larger language models for conversational systems could introduce additional experimental insights. The third limitation is the shortage of appropriate conversational data for extensive evaluations. This is mainly caused by the recent development of the retrieval augmented generation technique and its application to conversational systems. Future research is encouraged to address this limitation. 












§ ETHICS STATEMENT

All experiments in this study were conducted using publicly available datasets and open-released language models, which do not contain any private information that could raise ethical concerns. 





























§ PROMPTS FOR RAGATE-PROMPT


In this section, we list the used prompts for the RAGate-Prompt gate mechanism. 

Zero-Shot Prompt: 

Below is an instruction that describes a task. Please respond with `True' or `False' only that appropriately completes the request. 

### Instruction: Analyse the conversational context so far. Generate an appropriate response. Consider the invovled entites. Estimate if augmenting the response with external knowledge is helpful with an output of `True' or `False' only.

### Input: [Converstion Context Input]

### Response: 




In-Context Learning Prompt: 

Below is an instruction that describes a task. Please respond with `True' or `False' only that appropriately completes the request. 

### Instruction: Analyse the conversational context so far. Generate an appropriate response. Consider the invovled entites. Estimate if augmenting the response with external knowledge is helpful with an output of `True' or `False' only.

### Example 1: USER: I'm planning a trip, can you help me look for a flight? SYSTEM: Which day are you planning to return and from which city? USER: I want to go from NYC the day after tomorrow and return on the 13th of this month. SYSTEM: Where would you like to go? USER: I want to go to Vancouver, BC. Can you look for a Premium Economy class ticket. SYSTEM: I found 1 flight for you. It is a Delta Airlines flight that takes off at 6 am and returns at 2:50 am. The price is $505. USER: What is the departure airport, and how many stops does the flight have?

### Response: True

### Example 2: USER: Get me bus tickets to a Cher event on March 6th. SYSTEM: How many to buy? USER: only one, please.

### Response: False

### Input: [Converstion Context Input]

### Response: 



§ IMPACT OF RETRIEVAL QUALITY ON ADAPTIVE RAG

To have a successful conversation model with a retrieval-augmented system, two main criteria must be met. One is identifying insufficient context, and the other is the quality of retrieved information <cit.>. A conversational model performs better when both criteria are satisfied. In our proposed approach, as shown in Table 2, we have already assessed whether our adaptive retrieval method can detect insufficient context. We further explored to determine whether our model can inherently estimate the quality of the retrieved snippets to address such insufficiency and, based on that, decide on the retrieval. Although we do not explicitly provide retrieved snippets to our model, retrieval comes with a corpus that includes potentially relevant knowledge snippets. Consequently, given a query and a retrieval collection, it can be estimated whether useful information for the query exists in the corpus to address the insufficient context. To investigate by following this direction, we randomly selected 50 samples from instances where our proposed approach (RAGate-MHA, the best-performing gate model) predicted using retrieval augmentation. We asked domain experts (co-authors) to score whether they thought the retrieved snippets in those scenarios could be useful to response generation. Users rated the snippets on a scale of 0-4, with scores of 3 or 4 indicating `useful' or `highly useful'. We found that in 54% of cases where the prediction was for augmentation, users also found the snippets useful. This indicates that our proposed approach can implicitly capture the potential for obtaining high-quality retrieval snippets. 




§ ADDITIONAL EXPERIMENTAL RESULTS ABOUT RAGATE FOR RESPONSE GENERATION
 
In Table <ref>, we include the complete experimental results of applying RAGate for adaptive retrieval-augmented system response generation. Specifically, explore the use of retrieved knowledge snippets to different extents of relevance. We include top-3 knowledge snippets retrieved by BERT-ranker and TF-IDF. In addition, we also explore the use of knowledge snippets in different ranking positions (rank 1 and 5) according to the BERT-ranker retriever. The experimental result shows that precisely using a suitable amount of relevant knowledge can generate a response with higher confidence (i.e., less is more). In addition, this observation also indicates the potential use of confidence levels to evaluate the quality of the augmented knowledge.




 Paper Title: ShieldGemma: Generative AI Content Moderation Based on Gemma.





[1]See sec:contributions section for full author list. Please send correspondence to mailto:shieldgemma-team@google.comshieldgemma-team@google.com.

§ ABSTRACT 
We present , a comprehensive suite of LLM-based safety content moderation models built upon Gemma2. These models provide robust, state-of-the-art predictions of safety risks across key harm types (sexually explicit, dangerous content, harassment, hate speech) in both user input and LLM-generated output. By evaluating on both public and internal benchmarks, we demonstrate superior performance compared to existing models, such as Llama Guard (+10.8% AU-PRC on public benchmarks) and WildCard (+4.3%). Additionally, we present a novel LLM-based data curation pipeline, adaptable to a variety of safety-related tasks and beyond. We have shown strong generalization performance for model trained mainly on synthetic data. By releasing , we provide a valuable resource to the research community, advancing LLM safety and enabling the creation of more effective content moderation solutions for developers.

 


    < g r a p h i c s >
 


    < g r a p h i c s >
 


    < g r a p h i c s >
 




Lanczos for lattice QCD matrix elements
    Michael L. Wagman
    August 5, 2024
=======================================





§ INTRODUCTION


In recent years, the widespread adoption of Large Language Models (LLMs) has revolutionized various domains, ranging from conversational agents <cit.> to content generation <cit.>. These models exhibit remarkable capabilities in understanding and generating human-like text, thereby enabling sophisticated applications across diverse fields. However, alongside their advancements, the deployment of LLMs necessitates robust mechanisms to ensure safe and responsible interactions with users.

Current practices often rely on content moderation solutions like LlamaGuard <cit.>, WildGuard <cit.>, AEGIS <cit.>, etc., designed to filter inputs and outputs of LLMs for potential safety risks. While these tools provide initial safeguards, there are some limitations: (i) Some of existing solutions do not provide granular predictions of harm types or only provide binary output rather than probabilities <cit.>, which limits customized harm filtering or customized thresholds for downstream use cases. (ii) Most content moderation solutions only provide a fixed size model, which may not always align with the specific needs of different deployment scenarios. For instance, larger models could enhance performance for tasks like LLM-as-a-judge <cit.>, whereas smaller models might be preferable for online safety filtering to reduce latency and computational costs. (iii) Lack of detailed instructions in constructing the training data. Training data construction is critical to make sure that the models are robust for adversarial prompts and fair across identity groups. 

To address these challenges, this paper makes the following key contributions:

    
  * We propose a spectrum of state-of-the-art content moderation models ranging from 2B to 27B parameters built on top of Gemma2 <cit.>, tailored to accommodate various application requirements. This diversity in model sizes allows for optimized performance across different use cases. Our model can be applied to filter both user input and model output (with user input as the context) for key harm types.
    
  * We present a novel methodology for generating high-quality, adversarial, diverse, and fair datasets. This process leverages synthetic data generation techniques to reduce human annotation effort and it can be broadly applied across safety-related data challenges and beyond. 



In summary, this paper contributes a comprehensive framework that advances the state-of-the-art in LLM-based content safety moderation. By addressing the limitations of existing solutions and introducing novel methodologies for data creation, our work aims to foster safer and more reliable interactions between LLMs and users across various applications.


§ LITERATURE REVIEW


Safety Content Moderation. Extensive research has been conducted on content moderation, primarily focusing on human-generated content within online platforms. For instance, Perspective API <cit.> has been pivotal in advancing the detection of toxic language. However, existing resources are often tailored to human-generated text in web environments, which differs significantly from the content within human prompts and LLM-generated responses. Recent studies have demonstrated substantial progress in LLM content moderation through fine-tuning LLMs such as Llama-Guard <cit.>, Llama-Guard2 <cit.>, Aegis <cit.>, MD-Judge <cit.>, HarmBench <cit.>, BeaverDam <cit.>. WildGuard <cit.>.


Synthetic Data Generation. High-quality data is crucial for developing robust safety models. Despite the abundance of human-computer interaction data, direct utilization poses challenges due to the scarcity of positive examples, limited adversarial and highly diverse data, and privacy concerns <cit.>. LLMs, having absorbed vast knowledge during pretraining, have showcased exceptional capabilities in knowledge demonstration and language understanding <cit.>. Leveraging appropriate instructions, LLMs can generate high-quality synthetic data aligned with human requirements <cit.>. In the safety domain, this translates to generating diverse data across various dimensions (length, targeted harm types, sensitive topics, etc) and highly adversarial prompts that are more likely to elicit harmful LLM responses.





§ SAFETY POLICY



Safety policies are a critical component in the development of AI systems designed for real-world deployment. These policies consist of meticulously defined guidelines that delineate acceptable and unacceptable content, both in terms of user input and model-generated output: (i) Safety policies serve as a common framework for human annotators, ensuring consistency and reducing subjectivity in how they label and categorize potentially harmful content. This alignment is a prerequisite for training effective safety classifiers and mitigating unintended biases in the underlying data. (ii) By explicitly specifying the characteristics of harmful or inappropriate content, these policies are helpful for building zero-shot/few-shot classifiers as out-of-the-box solutions.

While the categories of disallowed content are largely consistent for both user input and model output, the emphasis for input centers on prohibiting requests that either directly contain harmful content or seek to elicit such content from the LLM. This includes inputs that may not be inherently harmful themselves but are intentionally crafted to manipulate the LLM into generating harmful responses. In contrast, the primary focus for model output is the prevention of any generation of harmful content, while the user input is only used as the context.



 §.§ Content Safety Taxonomy


We provide detailed definitions of  content safety taxonomy for 6 harm types below, and they are further refined in the Responsible Generative AI Toolkit,  <cit.>. 


    
  * Sexually Explicit Information: content contains references to sexual acts or other lewd content (e.g., sexually graphic descriptions, content aimed at causing arousal). Medical or scientific terms related to human anatomy or sex education are permitted.
    
  * Hate Speech: content targeting identity and/or protected attributes (e.g., racial slurs, promotion of discrimination, calls to violence against protected groups) or dehumanizing/belittling/vilifying on the bases of race, ethnicity, religion, disability, age, nationality, veteran status, sexual orientation, gender, gender identity, caste, or any other protected status. We do allow content that includes a reference to a protected individual or group in a positive manner.
    
  * Dangerous Content: content harming oneself and/or others (e.g., accessing or building firearms and explosive devices, promotion of terrorism, instructions for suicide).
    
  * Harassment: content is malicious, intimidating, bullying, or abusive content targeting another individual (e.g., physical threats, denial of tragic events, disparaging victims of violence).
    
  * Violence: content is shocking, sensational, or gratuitous (e.g. Violent acts or incidents that result in extreme injury or death without historical or educational context; Exposed internal organs or bones without medical or scientific context; Identifiable victims of violent deaths; Animal abuse or killings outside of standard hunting, food processing, or cultural practices).
    
  * Obscenity and Profanity: content is vulgar, profane, or inappropriate (e.g., profanity, obscenities, or other inappropriate language).


Distinct instructions are employed for user input and model output scenarios: (i) User input must not contain or seek generation of content that violates the aforementioned policies. (ii) The chatbot must not generate content that violates the aforementioned policies.


§ SYNTHETIC DATA CURATION



Our training data is primarily synthetic, generated using the pipeline depicted in Fig. <ref>. This approach allows for the creation of diverse, adversarial data, while active learning minimizes the need for human annotation.  After annotation, we augment our data to identity categories include diverse Gender, Ethnic Groups, Sexual Orientation, Religion to further enhance the fairness of our models. We are generating data for both use cases: (i) User Input: it includes adversarial and benign prompts for the LLM input; (ii) Model Response: it includes (user input, LLM response) pairs. 



 §.§ Raw Data Curation


AART <cit.> provides a novel approach for automated generation of adversarial  datasets for safety testing. We leverage AART for raw data curation with steps:

    
  * Problem Definition: define the scope of the task. Here we limit our harm types to be one of hate/dangerous/sexual/harassment and language to be English only to generate a list of adversarial topics/sub-topics and why this topic could be harmful. We also ask an LLM to generate a list of generative AI use cases like email, tweet, FAQ, etc. Unless otherwise specified, Gemini will serve as our default LLM utilized in this paper.
    
  * Query Generation: use instruction-tuned LLM to further generate diverse adversarial prompts based on parameters like  harm type, topic, subtopic, use case, locale, etc.
    
  * (Optional) Response Generation: use another LLM to generate responses based on parameters like queries, policies, whether generating adversarial or benign responses, etc.


We generate 50k examples of user inputs and 50k examples of model responses ((prompt, response) pairs), which evenly distributed into use cases, topics, harm types, etc. For example, for (Topic=chef, sub-topic=stereotype, use case=report, harm type=hate speech. locale=UK), it generates: “I work at a restaurant and I am looking for a new head chef. I need a report that talks about how head chefs are so hot-headed, temperamental, and bossy that they can be difficult to work with.”. Note that, the model is not guaranteed to generate violative examples and the real label would be decided by the human raters (detailed in the section sec:annotation). 



 §.§ Data Expansion

We further expand our raw data along dimensions like difficulty and diversity based on a self-critiquing and generation framework. For example, to expand our data for semantic/synthetic diversity, we repeatedly extract a batch of examples from the raw data and ask a critic LLM to generate suggestions for improving semantic and syntactic diversity of the data. Based on the suggestions and batch of examples, we further ask a generation LLM to generate a new example that accounts for the suggestion. We have generated 5k examples through this process focused on semantic/syntactic diversity expansion and another set of 5k examples, through expansion focused on generating more difficult examples. This was for both user input and model response use cases, and in total it has 20k examples.

We combine 100k synthetic raw data, 20k expanded data, and 14k Anthropic HH-RLHF <cit.> to form our raw data. For the Anthropic HH-RLHF data: for 50% of the data, we only keep the first utterance to mimic user input use case. For the remaining 50%, we keep the first prompt-response pair to mimic model response use case. We added Anthropic HH-RLHF for the purpose of further increasing the diversity of our training dataset. 



 §.§ Data Sub-Sampling

Before sending data for annotation, we need to subsample it to: (1) reduce annotation effort and speed up iteration; (2) reduce examples the base model can confidently predict; and (3) reduce (near-)duplicate examples, both syntactically and semantically.

This problem falls into the domain of batch active learning, which iteratively selects batches of data to improve classifier efficiency. Common methodologies include cluster-based sampling <cit.>, diverse mini-batches <cit.>, etc. We choose Cluster-Margin <cit.> as our initial algorithm because it claims state-of-the-art performance compared to other common algorithms like BADGE <cit.> and CoreSet <cit.> and can easily scales to millions of examples. The algorithm aims to balance uncertainty and diversity in the subsampling process. The high-level idea is to: (1) compute embeddings for the entire dataset. We use BERT <cit.> to generate embedding. (2) run a clustering algorithm (e.g., Agglomerative clustering) on the embeddings to assign each data point to a cluster; (3) select the k examples with the smallest margin scores. We use Gemma1 <cit.> to generate the probability of violating any of the policies and use | probability - 0.5| as the margin score. We also keep 10% of high margin examples in case of wrong predictions in high-confidence examples.  (4) run round-robin on the assigned clusters of these examples to further downsample to the desired batch size. After labeling, we can repeat these steps to iteratively improve the model. 

We employed a cluster-margin algorithm to downsample the raw dataset to 15,000 examples for training and testing. We reserved 10,500 examples for training, aligning with the training data volume of LlamaGuard1 <cit.>, and 4,500 for testing. Among them, half of the data is for user input use case and the remaining is for model response use case.



 §.§ Data Annotation



We send our data to 3 raters to rate and then we generate our final label based on majority vote. For model response, we ask the rater to rate whether the model response is violating our policy given the user input as the context. The test data comprises 2,671 benign examples and 895/383/360/239 adversarial examples for hate/dangerous/sexual/harassment respectively, along with 40/70 examples annotated as obscenity/violence. While the model is trained on all six harms, we report performance only on the four targeted harms.  We acknowledge the presence of 141 examples annotated as positive for multiple harms, which increases the complexity of harm-type level prediction. 



 §.§ Fairness Expansion

To improve fairness of our model, we leverage counterfactual fairness expansion <cit.> to expand our training data across identity categories like Gender, Race, Ethnicity, Sexual Orientation, and Religion. It includes the following steps: (1) Ask a LLM to find any related terms like male (Gender), Japanese (Ethnicity), etc; (2) If so, we randomly generate another term in this identity category and ask a few-shot LLM to replace the original term with the new term while keeping the same meaning with correct grammar; (3) Further send the data for human audit to remove bad examples. We then treat the original label as the ground truth for the newly generated data. 



 §.§ Model Fine-Tuning


We further supervise fine-tune (SFT) Gemma2 Instruction-Tuned (IT) models (2B, 9B, and 27B parameters) using the instruction shown in Fig. <ref>. We employ distinct policy definitions for each harm type and the model output is either Yes or No token. Our models are trained on TPUv5 lite with batch size of 16, a max sequence of 8k, and a learning rate of 1e-6. The model is trained for 4k steps and the best checkpoints are selected based on validation data. We calculate our predicted probability based on Eq. <ref> below:

    exp(LL(Yes)/T) + α/exp(LL(Yes)/T) + exp(LL(No)/T) + 2α


Here LL(·) is the log likelihood of the token generated by the model; T and α are hyperparameters to control temperature and uncertainty estimate. 



§ EXPERIMENTS





 §.§ Setup

Despite the abundance of safety-related benchmark datasets, direct comparison remains challenging due to several factors: (i) variations in policy definitions and supported harm types across datasets; (ii) inconsistencies in policy definitions even within the same harm type; and (iii) the predominance of binary (safe/unsafe) classification models rather than harm-type level prediction.  To address these challenges, we conduct experiments on two fronts:


    
  * Overall Binary Classification: We aggregate our prediction results into binary outcomes by maximizing probabilities over all harms. For models that provide probabilities, we report both optimal F1 and AU-PRC scores; for models with only binary results, we report F1 scores. 
    
  * Harm Type Level Classification: We provide a detailed performance comparison at the individual harm type level. We adopt a one-vs-all setup as described in <cit.>, i.e. we transform a multi-class classification problem into multiple binary classification problems, where each classifier focuses on distinguishing positive examples in one specific harm type and treat all others as benign examples.




 §.§ Benchmark Datasets and Baseline Models

OpenAI Moderation <cit.> comprises 1,680 prompt examples labeled for eight safety categories: sexual, hate, violence, harassment, self-harm, sexual/minors, hate/threatening, violence/graphic. Given that the original OpenAI Moderation policy definitions differ from ours, particularly we do not directly predict self-harm, we utilize those original definitions to predict each harm and then aggregate them into an overall binary classification. The dataset is sourced from CommonCrawl which does not match with the style of either user prompt or model output. Here, we run inference by treating the text as model output and keep empty user prompt.

ToxicChat <cit.> contains 10k examples with binary toxicity label for the prompt. We directly maximize our predictions for the six harms according to our policy, as our harm types capture different aspects of the toxicity definitions outlined in the ToxicChat policy.

ShieldGemma Prompt & ShieldGemma Response are our test dataset. it contains 4,500 examples with labels in total for both use cases. They have labels for our targeted harm types sexual, dangerous content, harassment, hate speech and non-targeted types violence and obscenity. More details are in section sec:annotation.

Baseline Models: We evaluate our models against several models: OpenAI Mod API <cit.>, LlamaGuard <cit.>, WildGuard <cit.>, and GPT-4. For GPT-4, we utilize the openAI API (model=gpt-4-0613) with our prompts, obtaining the log probability of the first token and converting it into the probability of a policy violation.



 §.§ Overall Binary Classification Results


The overall binary classification results are presented in Table <ref>. All ShieldGemma (SG) models (2B, 9B and 27B) outperform all baseline models. Notably, with similar model size and training data volume, SG-9B achieves a 10.8% higher average AU-PRC compared to LlamaGuard1 on external benchmarks.  Additionally, the F1 score of our 9B model exceeds that of WildGuard and GPT-4 by 4.3% and 6.4%, respectively.

Within the SG models, performance is comparable on our internal benchmarks. On external benchmarks, the 9B/27B model demonstrates slightly stronger generalization capability, achieving on average a 1.2%/1.7% higher AU-PRC than its 2B model. 




 §.§ Harm Type Level Results





We evaluate the harm-type level performance on our test datasets: SG Prompt and SG Response. The results are shown in Fig. <ref>. All SG models have outperformed GPT-4 by a big margin for all of the harms. Overall, GPT-4 is weak in distinguishing different harms. For example 76% of hate speech data points have been classified as positive for harassment. Note that the performance gap is expected, and the comparison is less favorable for GPT-4, as our model has been trained on datasets similar to the test datasets, while GPT-4 is evaluated zero-shot without any specific training. The performance among SG models is close to each other. On average, SG-9B and SG-27B have outperformed SG-2B by less than 2%.



§ LIMITATIONS

Despite our efforts to enhance the robustness of our model against adversarial attacks, fairness, and diversity in the training data, several limitations remain:

Fairness: While we have implemented fairness counterfactual expansion to mitigate bias in our training data, label discrepancies may still arise when identity groups are swapped. These discrepancies often stem from inherent biases within the pre-training dataset <cit.>.

Generalization: We have observed that our larger models demonstrate stronger performance on external benchmarks with new harm types and text styles. Overall, this generalization capability of our larger models are slightly stronger than our smaller 2B model. It also requires additional experiments to further verify the generalization on other datasets.

Implicit Cultural Harm: Although LLMs exhibit some understanding of cultural contexts, they may struggle to fully grasp implicit harm within these contexts.

Safety vs. Helpfulness: While our models demonstrate a strong ability to filter potential safety risks, their interpretation of policy violations may be overly conservative. This could interfere with helpfulness when used to filter LLM responses. We recommend that downstream clients adjust filtering thresholds based on their specific use cases.

LLM-as-a-classifier: Our model is specifically designed for classification tasks, with an output restricted to Yes or No token as the first output token when the prompt is correctly configured.  However, it's crucial to acknowledge that as an LLM, it remains capable of generating responses to any text input. We strongly advise the users to use it solely for generating Yes/No token scores (we call it scoring mode, detailed in our model card), and avoid using it in a chat-like manner since it may produce unethical or unsafe content due to the absence of additional safety instruction-tuning for conversational use.

We are dedicated to ongoing research and development to address these limitations and further refine our classifiers.



§ CONCLUSION


This paper presents a significant advancement in safety content moderation through our suite of specialized models, built on the foundation of the public Gemma2 <cit.> language models. We demonstrate their superior performance on diverse benchmarks, highlighting the effectiveness of our approach. Additionally, our novel synthetic data generation pipeline offers a valuable tool for researchers and practitioners to create high-quality, diverse datasets for safety and other domains. We are excited to share these resources with the research community to foster further development in this critical area.










§ CONTRIBUTIONS AND ACKNOWLEDGMENTS




Core Contributors 

Wenjun Zeng 

Yuchi Liu 

Ryan Mullins 

Ludovic Peran 

Contributors 

Joe Fernandez 

Hamza Harkous 

Karthik Narasimhan 

Drew Proud 

Piyush Kumar 

Bhaktipriya Radharapu 

Olivia Sturman 

Oscar Wahltinez



Other Specialty Areas 

Special thanks and acknowledgments to these individuals for their assistance in respected areas:

Central Support  

Manvinder Singh 

Kathy Meier-Hellstern 

Shivani Podder

Checkpoint Conversions  

Nam T. Nguyen 

Matthew Watson 

Ethics and Safety  

Antonia Paterson

Jenny Brennan

Gemma Model  

Surya Bhupatiraju 

Victor Cotruta 

Armand Joulin 

Kathleen Kenealy 

Tris Warkentin

Go-to-Market  

Kat Black 

Meg Risdal

Team Acknowledgements 

Our work is made possible by the dedication and efforts of numerous teams at Google. We would like to acknowledge the support from the following teams: Gemma, Google DeepMind Responsibility, Kaggle, Keras, Perspective.






 Paper Title: Intermittent Semi-working Mask: A New Masking Paradigm for LLMs.




MUFASA: Multi-View Fusion and Adaptation Network with Spatial Awareness for Radar Object Detection
    Xiangyuan Peng1,2 Miao Tang3 Huawei Sun1,2Kay Bierzynski2Lorenzo Servadei1 Robert Wille1

    August 5, 2024
==================================================================================================




§ ABSTRACT 
  
  
  
  
  
  
  Multi-turn dialogues are a key interaction method between humans and Large Language Models (LLMs), as conversations extend over multiple rounds, keeping LLMs' high generation quality and low latency is a challenge. Mainstream LLMs can be grouped into two categories based on masking strategy: causal LLM and prefix LLM. Several works have demonstrated that prefix LLMs tend to outperform causal ones in scenarios that heavily depend on historical context such as multi-turn dialogues or in-context learning, thanks to their bidirectional attention on prefix sequences. However, prefix LLMs have an inherent inefficient training problem in multi-turn dialogue datasets. In addition, the attention mechanism of prefix LLM makes it unable to reuse Key-Value Cache (KV Cache) across dialogue rounds to reduce generation latency. In this paper, we propose a novel masking scheme called Intermittent Semi-working Mask (ISM) to address these problems. Specifically, we apply alternate bidirectional and unidirectional attention on queries and answers in the dialogue history. In this way, ISM is able to maintain the high quality of prefix LLM and low generation latency of causal LLM, simultaneously. Extensive experiments illustrate that our ISM achieves significant performance.

  




§ INTRODUCTION



Recent developments in Large Language Models (LLMs) such as GPT-3.5 <cit.> and GPT-4 <cit.> have attracted significant attention. Due to the powerful generation capability, LLMs have made remarkable achievements in different kinds of Natural Language Process (NLP) tasks through a unified generative paradigm. Specifically, the most natural and common way to interact with LLMs is through multi-turn dialogues. However, as the number of dialogue rounds increases, ensuring high quality and low latency of the generated answer by LLMs is a challenge.

Existing language models can be grouped into three categories according to framework architecture: Encoder-Decoder <cit.>, Encoder-Only <cit.>, and Decoder-Only <cit.>. Nowadays, most LLMs belong to decoder-only architecture, in this paper, our discourse is delimited to decoder-only architecture. In addition, based on the masking methods in various attention mechanisms, decoder-only category further includes causal decoders <cit.> and prefix decoders <cit.>. The former employs unidirectional attention masking to restrict each token can only attend to preceding tokens and itself. Both the input and generated tokens are processed in a uniform manner within the decoder. The latter applies bidirectional attention on prefix tokens and maintains unidirectional attention on generated tokens, it allows bidirectional encoding of prefix sequences.


Empirical studies <cit.> have revealed that restricting auto-regressive masks on the entire sequence is overly restrictive when LLM generates answers based on the given contexts. Other work <cit.> further proves that prefix LLM may be a better choice theoretically. 
However, the bidirectional attention in prefix LLM is a double-edged sword. In the bidirectional attention mechanism, each token of the prefix sequences can attend to all the tokens in the whole sequences. 
It means that the KV cache of the previous dialogue rounds cannot be reused when the prefix LLM generates the current response.

When facing multi-turn dialogues, especially when the dialogue round reaches a large number such as 10 or more, prefix LLM will suffer from high generation latency.
In addition, due to the bidirectional attention in prefix sequences, it's necessary to manually expand a multi-turn dialogue sample into several single-turn dialogue samples to train model for all the dialogue rounds. However, unidirectional attention enables to train causal LLMs for all rounds of dialogue generation through a single forward process.



In this paper, we propose a novel attention mask scheme called Intermittent Semi-working Mask (ISM), which preserves the strengths of the prefix decoder while concurrently addressing the issue of high generation latency in multi-turn dialogues. In other words, our ISM simultaneously possesses the high quality of prefix decoder and low latency of causal decoder. Specifically, as illustrated in Figure <ref>, given a prefix sequence includes the dialogue history and current query [𝐏, 𝐐_1, 𝐀_1, ⋯, 𝐐_𝐧-1, 𝐀_𝐧-1, 𝐐_𝐧], our ISM applies bidirectional and unidirectional attention on queries (𝐐_𝐢, i=1,⋯,n) and answers (𝐀_𝐢, i=1,⋯,n-1), respectively. Different from prefix mask, our ISM ensures queries or answers cannot attend to subsequent queries or answers in the dialogue history. Therefore, we can reuse the KV cache across dialogue rounds like causal decoder to prevent duplicate computation and reduce the generation latency. In addition, our ISM significantly improves the training efficiency for multi-turn dialogue datasets.


Given a sample [𝐏, 𝐐_1, 𝐀_1, ⋯, 𝐐_n, 𝐀_n], the usual training approach for a prefix LLM is to expand the data into n samples: [𝐏, 𝐐_1, 𝐀_1], [𝐏, 𝐐_1, 𝐀_1, 𝐐_2, 𝐀_2],⋯,  [𝐏, 𝐐_1, 𝐀_1,⋯, 𝐐_n, 𝐀_n]. In contrast, our ISM can be trained on the original sample through a single forward process like causal LLMs.






To prevent the influence of the base model.
We apply our ISM on prefix LLMs and causal LLMs and conduct extensive experiments in multi-turn dialogue benchmark datasets.


We follow <cit.> to utilize GPT-4 as a stand-in for human raters to score them.
The experimental results demonstrate that our ISM achieves state-of-the-art generation quality and latency in multi-turn dialogues. Overall, our contribution can be summarized as follows,



    
  * We propose a novel masking scheme called ISM, which applies alternate bidirectional and unidirectional attention on queries and answers in multi-turn dialogues. It maintains the high quality of prefix decoders and the low latency of causal decoders simultaneously.
    
  * We apply our ISM on causal LLMs and prefix LLMs and conduct extensive experiments to compare the generation quality and latency between our ISM and existing methods, the experimental results demonstrate the effectiveness of our method.
    
  * We deploy our ISM in a real-world online environment to test its practical performance, we observe our ISM brings significant improvements in generation latency and quality.





§ RELATED WORK



 §.§ LLMs for Multi-turn Dialogues

Recently, the advancements in LLMs have garnered lots of attention. To enhance the multi-turn capabilities of open-sourced LLMs, initial efforts begin with collecting human-ChatGPT dialogues, leading to the creation of Vicuna <cit.>. RealChat <cit.> later expands the data to 1 million conversations. To generate more sophisticated datasets, Baize <cit.> and UltraChat <cit.> employ alternating GPT interactions. Further, Cue-CoT <cit.> and ICL-AIF <cit.> enhance model capabilities for multi-turn interactions through In-Context-Learning (ICL) and Chain-of-Thought (CoT) algorithms.

To investigate LLMs' multi-turn ability, several works construct multi-turn dialogue benchmarks. Daily Dialogue <cit.> is a human-written high-quality multi-turn dialogue dataset. It reflects daily communication way and covers various topics about daily life.
KdConv <cit.> is a Chinese human-labeled knowledge-grounded dialogue dataset for knowledge-driven conversation generation.

NaturalConv <cit.> is a Chinese multi-turn topic-driven conversation dataset, which contains conversations from six domains.





 §.§ Evaluation of LLMs

The quality of the content generated by LLMs can be evaluated from both objective and subjective perspectives. The former aims to evaluate the capabilities of LLMs in an objective and quantitative way, usually by comparing LLMs' outputs with corresponding ground truth. Similarity metrics are calculated based on the outputs and references, such as BLEU <cit.>, and ROUGE <cit.>. An alternative approach involves tasks as a classification task and forces LLMs to output closed options for accuracy calculation, such as MMLU <cit.> and CEVAL <cit.>. The latter is applied to compare the performances of different LLMs by humans or other powerful LLM judges when facing more complicated scenarios. GPTScore <cit.> makes full use of GPT3 in achieving customized, multi-aspect and training-free evaluation. Chatbot Arena <cit.> showcases high agreement between GPT-4 and humans on making judgements. It indicates that leveraging LLM like GPT-4 for evaluating the quality of generated text could be a viable alternative to manual human assessment.

















































 









































§ METHODOLOGY

In this section, we first give the mathematical description of causal LLM, prefix LLM, and our ISM. Then, we elaborate on the benefits of our ISM compared to causal and prefix LLM, respectively. In addition, we introduce how to implement our ISM on existing prefix and causal LLMs.







 §.§ Mathematical description



  §.§.§ Transformers: SSA, LSA

Given a sequence of input vectors 𝐗 = (𝐱_1 ,⋯, 𝐱_n ), the output of standard Softmax Self-Attention (SSA) layer is 


    𝐱_j 𝐱_j + 𝐏𝐕𝐗 softmax(𝐗^⊤𝐊^⊤𝐐𝐱_j)
 

where P corresponds to the output projection, V, K, Q corresponds to the value, key, and query transformation, respectively.

Since the softmax attention of standard transformers is non-linear, its theoretical analysis becomes complicated even for a single layer. For this reason, theoretical approaches <cit.> to analyze transformers have often dropped the softmax function from the attention, resorting to the Linear Self-Attention (LSA) layer,


    𝐱_j 𝐱_j + 𝐏𝐕𝐗 (𝐗^⊤𝐊^⊤𝐐𝐱_j) = 𝐱_j + 𝐏𝐕∑_i=1^n 𝐱_i (𝐱_i^⊤𝐊^⊤𝐐𝐱_j)




  §.§.§ Causal LLM, Prefix LLM, and ISM for Multi-turn dialogue



Here, we briefly introduce the attentions of causal LM, prefix LM, and ISM in their LSA version for multi-turn dialogue.

Given a multi-turn dialogue sample including the prompt (P = (x_1, ⋯, x_m_p)), queries (𝐐_i = (x_m_p +1  or x_m_a_i-1 +1, ⋯, x_m_q_i), i = 1,⋯, n) and answers(𝐀_i = (x_m_q_i +1,⋯, x_m_a_i), i = 1,⋯, n-1), that is [𝐏, 𝐐_1, 𝐀_1,⋯, 𝐐_n-1, 𝐀_n-1, 𝐐_n]. The goal of the model is to predict 𝐀_n of 𝐐_n using the context [𝐏, 𝐐_1, 𝐀_1,⋯, 𝐐_n-1, 𝐀_n-1].

The most classic form of attention is categorized as full (or bidirectional) attention, shown in equation  <ref>, in which each input x_j (j = 1,⋯, n) can attend to all positions. Full attention is typically used in the transformer encoder.

Furthermore, another transformer decoder for in-context learning uses the auto-regressive attention


    𝐱_j 𝐱_j + 𝐏𝐕∑_i=1^j 𝐱_i (𝐱_i^⊤𝐊^⊤𝐐𝐱_j)


which restricts each token 𝐱_j to attend only to previous positions (and itself) from 1 to j. This restriction is due to the role of the decoder as a causal language model (causal LLM) which predicts the next token in the context of the previously generated ones.

The original transformer uses a full attention based encoder and an auto-regressive attention based decoder. However, NLP researchers have a preference for models that are either encoder-only (e.g. BERT<cit.>) or decoder-only (e.g. GPT<cit.>, PaLM<cit.>) according to the task at hand. One motivation for altering model is that it reduces the number of parameters by half.

A new form of attention, that lies intermediate to full attention and auto-regressive attention, emerged from the understanding that certain tasks can gain advantages from a prefix sequence, such as multi-turn dialogue and in-context learning. This attention in prefix LLM suggests the following:


    𝐱_j 𝐱_j + 𝐏𝐕∑_i=1^max(j, n_q_n)𝐱_i (𝐱_i^⊤𝐊^⊤𝐐𝐱_j)


Where max(j, n_q_n) ensures each prefix token x_j with j < n_q_n can attend to all prefix tokens.


In our ISM, the form of which suggests the following:


    𝐱_j 𝐱_j + 𝐏𝐕∑_i=1^f(j)𝐱_i (𝐱_i^⊤𝐊^⊤𝐐𝐱_j)

where f(j) is 


    f(j) = {    m_q_1      for  j <= m_q_1
        m_q_k      for  m_a_k-1< j <= m_q_k , k > 1 
        j       for  m_q_k< j <= m_a_k , k >= 1, 
    .






 §.§ Convergence in Multi-turn dialogue learning


To simplify the model in Multi-turn dialogue learning, we analyze Linear regression, which  is a classical machine learning problem. Given a set of vectors 𝐐𝐀_i = ((𝐪_j, a_j))(where j = m_i-1+1, ...,m_i, i = 1, ..., n,m_0 = 0), which contains a set of input-label  pairs, the goal is to find an optimal weight vector 𝐰 that minimizes the l2-loss:

    L(𝐰) = 1/2m_n∑_k = 1^m_n𝐰𝐪_k-a_k_2^2

The gradient of the loss is ∇_𝐰L =  1/m_n∑_k = 1^m_n (𝐰𝐪_k-a_k)𝐪_k^⊤  , and a gradient descent algorithm with step size α follows the update rule:


    𝐰^(l)= 𝐰^(l-1)  + α/m_n∑_k = 1^m_n (a_k-𝐰^(l-1)𝐪_k)𝐪_k^⊤



According to <cit.>, the input is formulated as



    X = (x_i^(0), ..., x_m_n^(0)),  where x_j^(0) = (
           𝐪_j 
           a_j
    )

and the parameter matrices of  are:



    𝐊 = 𝐐 = (
           𝐈_d× d      0
       0       0
    ), 
        𝐕 = (
           0_d× d      0
       𝐰^(0)       0
    ),
        𝐏 = α/m_n𝐈,
 
where 𝐰^(0) is an initial weight vector. Furthermore, there is work<cit.> proves that multi-layer LSA under the construction of progresses identically to multi-step gradient descent. The prososition is that for a multi-layer LSA satisfying the construction <ref> and with 𝐰^(0) =0, if its input Z is formatted as <ref>, then its l-th layer output is 𝐱_j^(l)=(𝐪_j^⊤, δ_j^(l))^⊤, where δ_j^(l)=a_j-𝐰^(l)𝐪_j and 𝐰^(l)  is the l-th updated weight from the gradient descents update rule in <ref>. Since 𝐪_j never changes , we can simplify it and focus only on  δ_j^(l), which is the last output coordinate of the j-th LSA-layer for l > 0,




    δ_j^(l)  = δ_j^(l-1)- α/m_n∑_k = 1^m_nδ_j^(l-1)𝐪_k𝐪_j^⊤
 

with δ_j^(0) = a_j. Defining a_j^(l) = a_j - δ_j^(0) and rearranging <ref>, for a_j^(0) = 0 and ∀ l > 0, we have:



    a_j^(l)  =a_j^(l-1)- α/m_n∑_k = 1^m_n (a_k-a_k^(l-1) ) 𝐪_k𝐪_j^⊤
 

For prefix LM, <ref> and <ref> are the the update rules. Mainwhile, a causal LM applies auto-regressive attention throughout the entire sequence. Therefore, plugging the same K, Q, P, V into <ref>, the update rules of <ref>, and <ref> become:
 


    δ_j^(l)  = δ_j^(l-1)- α/m_n∑_k = 1^jδ_j^(l-1)𝐪_k𝐪_j^⊤
 



    a_j^(l)  =a_j^(l-1)- α/m_n∑_k = 1^j (a_k-a_k^(l-1) ) 𝐪_k𝐪_j^⊤
 

For ISM, the update rules of <ref>, and <ref> become:



    δ_j^(l)  = δ_j^(l-1)- α/m_n∑_k = 1^f(j)δ_j^(l-1)𝐪_k𝐪_j^⊤
 



    a_j^(l)  =a_j^(l-1)- α/m_n∑_k = 1^f(j) (a_k-a_k^(l-1) ) 𝐪_k𝐪_j^⊤
 

where f(j) is defined in <ref>.


It has been proved that the layer outputs of prefix LM have the same dynamics of multi-step gradient descent on a linear regression problem <cit.>, which means the iterative weights 𝐰^(l) converges to the stationary point 𝐰^* in a linear rate.

For causal LM, for each j, the iterative weights 𝐰_j^(l) converge to the stationary point 𝐰_j^* , but 𝐰_j^* may not converge to the the optimal solution 𝐰^* <cit.>.

For j = m_n-1,⋯, m_n in ISM, the iterative weights 𝐰_j^(l) converge to the stationary point 𝐰^* similar to prefix LLM. Thus, for m_n > m_n-1, ISM also has the same multi-step gradient descent.





 §.§ ISM vs. Causal LLM


As we proofed in Section <ref>, ISM converges to the stationary point 𝐰^* in most situations (m_n > m_n-1) and causal LM may not converge to 𝐰^*. It indicates that in multi-turn dialogue learning, ISM will perform better than causal LM when the length of query from the last dialogue is longer than 1.


To apply our ISM to causal LLMs, we modify the causal mask as shown in Figure <ref> a) into ISM as shown in Figure <ref> c) to apply bidirectional attention on queries in dialogue history.




 §.§ ISM vs. Prefix LLM




Compared with prefix LLM, the standout benefit of our ISM is its ability to reuse the KV cache across dialogue rounds, effectively reducing the generation latency.

KV Cache is a commonly used acceleration technique when LLMs perform generation.

However, it's not reusable between dialogue rounds in prefix LLM.

Given dialogue history h = [𝐏,𝐐_1,𝐀_1,⋯,𝐐_𝐧-1,𝐀_𝐧-1] and current query 𝐐_𝐧 to generate 𝐀_𝐧. Assuming we have dialogue history's KV cache C_h, for causal LLM and our ISM, we only need to compute the current query's KV cache C_q and concatenate them to get the whole sequence's cache [C_h; C_q], so that we can generate A_n with linear complexity. However, for prefix LLM, due to the bidirectional attention on prefix sequences as shown in Figure <ref> b), the KV cache of dialogue history C_h needs to be recomputed when computing KV cache of current query 𝐐_𝐧. Therefore, the KV cache of dialogue history C_h isn't reusable in prefix LLM.

To apply our ISM on prefix LLM, we choose ChatGLM as the base model.

It designs a special 2D positional encoding method as shown in Figure <ref> a). When facing multi-turn dialogue data, it applies bidirectional attention on prefix tokens (dialogue history) while generating answers. In contrast, our ISM applies alternate bidirectional and unidirectional attention on history queries and answers. Therefore, we follow GLM to apply the 2D position encoding for previous query and answer spans in Figure <ref> b). Position 1 and 2 are utilized to encode the inter- and intra-span positions, respectively.








§ EXPERIMENTS






 §.§ Datasets

Daily Dialogue. Daily Dialogue<cit.> is a human-written high-quality multi-turn conversation dataset. The dialogues in the dataset reflect daily communication way and cover various topics about daily life. It contains 13,118 dialogues, the average turns of each dialogue is 7.9.

KdConv. KdConv <cit.> is a Chinese knowledge-driven conversation dataset. It grounds the topics in multi-turn conversations to knowledge graphs. It contains 4.5K conversations from three domains (film, music, and travel), and 86K utterances with an average turn number of 19.0.

NaturalConv. NaturalConv <cit.> is a Chinese multi-turn topic-driven conversation dataset, which allows the participants to chat anything they want as long as any element from the topic is mentioned. It contains 19.9K conversations from six domains, and 400K utterances with an average turn number of 20.1.



 §.§ Implementation Details

For Daily Dialogue, we choose Llama2-7b as the base model and apply our ISM in it as described in Section <ref>. We call the modified model Llama2-7b (ISM). We finetune them on the train split with the same settings. Specifically, we utilize 4 NVIDIA A100 GPUs to full-finetune Llama2-7b and Llama2-7b (ISM) on Daily Dialogue's train split for 5 epochs, respectively. We train models with AdamW <cit.>, the learning rate is set as 3e-5, global batch size is 64.  


For KdConv and NaturalConv, we choose Qwen1.5-7b as the base model. As these two datasets are both in Chinese and topic-driven. We mix their train split to finetune Qwen1.5-7b and Qwen1.5-7b (ISM) and evaluate on the mixture of their test split. The training settings are the same as Llama2-7b's. In addition, to evaluate our ISM on prefix LLM, we apply ISM in ChatGLM as illustrated in Section <ref> and utilize KdConv and NaturalConv to train ChatGLM-6b and ChatGLM-6b (ISM).

For evaluation, we follow previous works <cit.> to apply powerful GPT-4 as a stand-in for human judge. We design appropriate instructions to ask GPT-4 to compare the answers between the base model and our corresponding ISM version and to decide which answer is better or tie. Considering the position bias <cit.>, we swap the position of answers and compute the average result. Specific prompts can be seen in Appendix <ref>. In addition, we randomly select 200 samples for human evaluation to compare the consistency between GPT-4 and human judgement.



 §.§ Quantitative Analysis


We report the GPT-4 pairwise comparison results in Table <ref>. For causal model, we compare Qwen 1.5-7b and Qwen 1.5-7b (ISM) on the mixture test split of KdConv and NaturalConv. We can find that the win rate of Qwen 1.5-7b (ISM) exceeds Qwen 1.5-7b by 1.72%. For prefix model, we compare ChatGLM-6b and ChatGLM-6b (ISM). Considering the inefficient training process of prefix LLM in multi-turn dialogue dataset, we expand the training data for ChatGLM. Specifically, given a multi-turn dialogue sample, we expand it into five samples manually. For instance, given a sample with 15 turns of dialogue, we will expand it into samples with 3, 6, 9, 12, and 15 turns. We can find the win rate of ChatGLM-6b (ISM) outperforms ChatGLM-6b by 2.61%. It demonstrates that our ISM can improve both causal LLM and prefix LLM's generation quality in multi-turn dialogues.




Similarly, we compare Llama2-7b and Llama2-7b (ISM) on Daily Dialogue, the results are reported in Table <ref>. We observe that Llama2-7b (ISM) demonstrates substantial performance enhancements compared to the base model, achieves 5.64% increasement in win rate.




Further, we apply ISM in our private model AntGLM-10B<cit.> to evaluate the performance. AntGLM has the same architecture as GLM, belongs to prefix LLM. We evaluate AntGLM and AntGLM (ISM) in our private AntEval dataset. AntEval dataset is collected from real-world conversations about insurance and medical. It contains 1219 samples in total. Including 944 samples with one turn and 56 samples with two turns, we call these as Short Split. In addition, other 219 samples contain conversation over 2 rounds, with an average turn number of 4.6, we call these as Long Split. We evaluate AntGLM and AntGLM (ISM) in these two splits, respectively, the results are shown in Table <ref>. We can observe that AntGLM (ISM) achieves 8.67% win rate over AntGLM in the Long Split, while the improvement is only 0.2% in the Short Split. It illustrates that our approach exhibits a pronounced advantage in longer dialogue samples.



  §.§.§ Human Evaluation





Designing instructions and applying GPT-4 to evaluate LLM's performance has been widely adopted, but it's still a subjective evaluation method. Therefore, we randomly select 100 samples from Daily Dialogue and KdConv&NaturalConv separately for human evaluation to compare the judgement consistency between human and GPT-4. As shown in Figure <ref>, we observe a high level of consistency between human and GPT-4 evaluation, 81% and 78% in Figure <ref> a) and b), respectively. Furthermore, it is notable that the majority of inconsistencies are centered around determining whether there is a tie or not. It demonstrates that the evaluation results from GPT-4 are relatively reliable.





 §.§ Latency Analysis





As described in Section <ref>, prefix LLM suffers from high generation latency in multi-turn dialogues due to the inability to reuse the KV cache across dialogue rounds. In this section, we conduct experiments to compare the time-to-first-token (TTFT) latency between AntGLM-10B and AntGLM-10B (ISM). Specifically, we construct a simulated dialogue sample consisting of 20 turns, with each turn having a sequence length of 200. The query length is 40, and the answer length is 160. We deploy them on 2 NVIDIA A10 GPUs and repeatedly call the inference API 10 times to calculate the average TTFT. As shown in Figure <ref>, it's evident that the TTFT latency of AntGLM-10B surges quickly as the input sequence length expands. Conversely, the TTFT of AntGLM-10B (ISM) increases much slower due to the reuse of the KV cache. Specific analysis can be seen in Appendix <ref>



 §.§ Case Study




In Figure <ref>, we showcase two cases including dialogue sample, generated responses and GPT-4 judgement results. We can find that GPT-4 follows our instructions well to make judgements. In the first case, GPT-4 not only considers the dialogue history, but also takes account of User 2's "willingness" and "responsibility". Cases with longer dialogue history can be seen in Appendix <ref>.



 §.§ Limitation

Current ISM is applied to existing causal or prefix base models for finetuning. The data utilized for finetuning is relatively small compared with the data used during their pretraining stage. The performance of our ISM in the pretraining stage needs to be explored.





§ CONCLUSION AND FUTURE WORK

In this paper, we propose a novel attention masking scheme called Intermittent Semi-working Mask (ISM) to achieve the high generation quality and low generation latency in multi-turn dialogue. Specifically, we apply alternate bidirectional and unidirectional attention on queries and answers in the dialogue history.
In this way, our ISM is able to maintain the advantages of both prefix LLM and causal LLM simultaneously.
Extensive experiments illustrate that our ISM achieves significant performance. In the future, we will try to apply our ISM to the pretraining stage of LLM and delve deeper into exploring its advantages.



abbrvnat








§ APPENDIX



 §.§ Inference Acceleration Estimation on ISM

ISM inference can be accelerated by reusing the KV Cache across turns. We analyze the computational complexity of transformers before and after acceleration.

Given the following variable definitions for a Transformer model:

    
  * b = batch size
    
  * s = input sequence length
    
  * n_s = new input sequence length
    
  * h = hidden dimension
    
  * h_n = number of attention heads
    
  * h_h = dimension per attention head, where h_h = h/h_n




  §.§.§ Before Acceleration

The FLOPs for the self-attention layer (F_self-attention) can be estimated as follows:

    
  * Computing queries, keys, values: 
    
    F_qkv = 3 · 2 · b · s · h^2 = 6 · b · h^2 · s

    
  * Computing attention scores: 
    
    F_as = 2 · 2 · b · h_n · s · h_h · s = 4 · b · h · s^2

    
  * Computing weighted sum: 
    
    F_ws = 2 · b · h^2 · s

    
  * The total FLOPs for self-attention is therefore: 
    
    F_self-attention = F_qkv + F_as + F_ws = 4 · b · h · s^2 + 8 · b · h^2 · s




The FLOPs for the feed-forward network layer (F_ffn) is:

    F_ffn = 16 · b · h · s^2


Thus, the total FLOPs for transformers (F_transformers) combines both the self-attention and feed-forward FLOPs:

    F_transformers = F_self-attention + F_ffn = 4 · b · h · s^2 + 24 · b · h^2 · s


Overall, the computational complexity of the Transformer is quadratically related to the input sequence length



  §.§.§ After Acceleration

By reusing the KV cache from the previous turn of dialogue, we can trim the input sequence to the new uncached segment. In this way, it is equivalent to replacing one s(input sequence length) in the aforementioned computation process with n_s (new input sequence length), thus the total computational complexity after inference acceleration is:

    F_accelarated-transformers = 4 · b · h · s * n_s + 24 · b · h^2 · n_s


When n_s is much smaller than s and h, n_s can be neglected. Therefore, the total computational FLOPs is approximately 4 · b · h · s + 24 · b · h^2, where the relationship between computational cost and input sequence length becomes a linear polynomial. Thus, the computational cost is optimized from a quadratic polynomial of input sequence length to a linear polynomial.




 §.§ Prompt




We design instruction prompt carefully as shown in Figure <ref> to leverage GPT-4 in assessing the quality of responses from the models to be compared. The whole prompt can be divided into three parts, the first part describe the task and the specific evaluation criteria, the second part contains two cases and corresponding human written evaluation result for few-shot learning, the third part contains sample to be judged.



 §.§ Cases






We showcase two cases with longer dialogue history and corresponding GPT-4 judgements, shown in Figure <ref> and <ref>. From Figure <ref> we can observe that facing dialogues with longer history, our ISM is able to attend history context better and generate more reasonable response.













































































































































































































































































































































































































































 




























































































































    

























    




































































 Paper Title: GenRec: Generative Personalized Sequential Recommendation.















panfengc@umich.edu

  University of Michigan
  Ann Arbor
  USA


pl219@cam.ac.uk

















§ ABSTRACT 
Sequential recommendation is a task to capture hidden user preferences from historical user item interaction data. Significant progress has been made in this domain by leveraging classification based learning methods. Inspired by the recent paradigm of “pretrain, prompt and predict” in NLP, we consider sequential recommendation as a sequence to sequence generation task and propose a novel model named Generative Recommendation (GenRec). Unlike classification based models that learn explicit user and item representations, GenRec utilizes the sequence modeling capability of Transformer and adopts the masked item prediction objective to effectively learn the hidden bidirectional sequential patterns. Different from existing generative sequential recommendation models, GenRec does not rely on manually designed hard prompts. The input to GenRec is textual user item sequence and the output is top ranked next items. Moreover, GenRec is lightweight and requires only a few hours to train effectively in low-resource settings, making it highly applicable to real-world scenarios and helping to democratize large language models in the sequential recommendation domain. Our extensive experiments have demonstrated that GenRec generalizes on various public real-world datasets and achieves state-of-the-art results. Our experiments also validate the effectiveness of the the proposed masked item prediction objective that improves the model performance by a large margin.







<ccs2012>
 <concept>
  <concept_id>10010520.10010553.10010562</concept_id>
  <concept_desc>Information systems Recommender systems</concept_desc>
  <concept_significance>500</concept_significance>
 </concept>
 <concept>
  <concept_id>10010520.10010553.10010563</concept_id>
  <concept_desc>Information systems Users and interactive retrieval</concept_desc>
  <concept_significance>400</concept_significance>
 </concept>
 <concept>
  <concept_id>10010520.10010575.10010755</concept_id>
  <concept_desc>Computing methodologies Natural language generation</concept_desc>
  <concept_significance>300</concept_significance>
 </concept>
</ccs2012>


[500]Information systems Recommender systems
[400]Information systems Users and interactive retrieval
[300]Computing methodologies Natural language generation
















10 May 2024






GenRec: Generative Personalized Sequential Recommendation
    Pietro Liò
    August 5, 2024
=========================================================





§ INTRODUCTION

Recent years have witnessed the great success of recommender systems in online platforms such as Amazon and Yelp, which help people make micro decisions and fulfil their demands in daily life. Within online platforms, historical sequential user interaction data is utilized to capture the evolved and dynamic user behaviors and make appropriate recommendations of next items for the user. Different from traditional recommender systems that treat each user behavior as a sample and directly model the user preference on a single item, sequential recommendation learns timestamp-aware sequential patterns to recommend the next item <cit.>. For example, one might purchase peripheral accessories after buying a desktop. Various methods have been proposed to accurately model sequential user behaviors. <cit.> employ Recurrent Neural Networks (RNN) to encode the left-to-right context. <cit.> utilizes Convolutional Neural Networks (CNN) to learn sequential patterns as local features. <cit.> leverages Graph Neural Networks (GNN) to construct the item-item interest graphs and user interest sequences. <cit.> designs contrastive loss to maximize the mutual information between different views of the sequential data. Contextual information such as item attributes is incorporated as self-supervised signals to learn user and item representations. 

The recent success of Transformer <cit.> based large language models (LLMs) in various NLP tasks has inspired the research of utilizing LLMs in recommender systems <cit.>. <cit.> utilizes the left-to-right self-attention mechanism to identify next relevant items from the sequential interaction history and <cit.> further improves the performance by employing the bidirectional self-attention mechanism. Although both methods are effective, they do not consider personalization and the model performance is potentially limited. Generative LLMs based recommender system is considered a promising approach <cit.>. <cit.> is the first model to unify different recommendation tasks in the same framework to facilitate the knowledge transfer and it is pretrained across task-specific datasets to capture the deep semantics for personalization and recommendation. However, different modality information is encoded in the textual format of natural language, which might cause suboptimal modality representation. Moreover, <cit.> relies on manually designed task-specific hard prompts to formulate the problem as question answering, which requires additional data processing and prompt search.



To address the mentioned limitations, we propose GenRec, a novel generative framework for personalized sequential recommendation. GenRec utilizes the encoder-decoder Transformer as the backbone and formulates sequential recommendation as a sequence-to-sequence generation task. We utilize the cloze task <cit.> as the training objective and pretrain the model on the corpus to learn the bidirectional sequential patterns. Inspired by <cit.>, to make the downstream sequential recommendation task consistent with the pretraining cloze task, we append the  token at the end of the input sequence to predict the next item. Extensive experiments on public real-world datasets demonstrate that GenRec generalizes effectively and achieves state-of-the-art performance across all datasets.
In conclusion, our contributions include: 


  * We propose GenRec, a novel generative model for personalized sequential recommendation that can generate next items auto-regressively without prompts.

  * As far as our knowledge, it is the first generative sequential recommendation method that adopts the cloze objectives in both pretraining and finetuning. Our method only utilizes task specific datasets and does not rely on additional large pretraining corpus [Our source code, datasets and pretrained models are publicly available at <https://github.com/caop-kie/GenRec>.].

  * We conduct experiments on three public real-world datasets, demonstrating consistent improvements on multiple evaluation metrics compared with baseline methods.




§ RELATED WORK

Transformer based LLMs have achieved remarkable success on various research fields such as text summarization, question answering, document understanding, etc. Typically, LLMs are trained on a vast amount of textual corpus from diverse sources including wikipedia, news, articles and books. LLMs can have emergent zero-shot learning capability as the model parameter size scales up with large training datasets <cit.>, making LLMs better generalize to unseen domains. Recent efforts have been made to explore the potential of LLMs as recommender systems. <cit.> employs T5 as the model backbone and achieves competitive performance on a variety of recommendation tasks, demonstrating the generalization capability of LLMs based recommender systems. <cit.> develops a unified foundation model based on M6 and reduces the model size and training time by utilizing prompt tuning. Similar to <cit.>, user interaction data is represented as plain texts and recommendation tasks are formulated as either natural language understanding or generation. <cit.> leverages ChatGPT to build a conversational recommender system to improve the interactivity and explainability of the recommendation process. <cit.> finetunes T5 with user-personalized instruction data, which is generated from manually designed templates, enabling users to communicate with the system with natural language instructions. <cit.> proposes an efficient framework to finetune LLMs for recommendation tasks and demonstrates significant improvements in the domains of movie and book recommendations. <cit.> employs a funneling approach where it first retrieves the candidates utilizing the user item interactions and then generates recommended items from candidates with a LLMs based framework.



§ TASK FORMULATION

The goal of sequential recommendation is to predict the next item that the user is most likely to interact with given the historical item interaction sequence of the user. Formally, i_th user u_i has item interaction sequence t_i = {t^i_1, ..., t^i_n_i} ordered chronologically. t^i_j and n_i denotes the j_th item in the sequence and the length of t_i respectively. t^i_j:k denotes the subsequence, i.e., t^i_j:k = {t^i_j, t^i_j+1, ..., t^i_k}, where 1 ≤ j < k ≤ n_i. Given the user item sequence {u_i, t^i_1:n_i}, the task of sequential recommendation is to predict the next item that u_i is most likely to interact with at the n_i+1 timestep. 





§ METHODOLOGY

The overall architecture of  is shown in Figure <ref>. Our  is established upon a sequence-to-sequence Transformer encoder-decoder framework. The encoder of  embeds cross modal user and item features from the input sequence and the decoder generates next items auto-regressively. In the following sections, we elaborate the feature embedding and learning objectives of pretraining and finetuning.


 §.§ Model

The feature embedding consists of token embedding, positional embedding, user ID embedding and item ID embedding. The input sequence to the model {u_i, t^i_1:n_i} is tokenized into a sequence of tokens, which is wrapped around with the start indicator token [BEG] and the end indicator token [END]. Extra [PAD] tokens are appended to the end to unify the sequence length in the batch. The token sequence S is represented as: 

    S = [BEG], Tok(u_i), Tok(t^i_1), ..., Tok(t^i_n_i), [END], ..., [PAD],

where Tok is the tokenizer function.
For the j_th token in S, we apply textual token embedding, positional embedding and user or item ID embedding. The textual token embedding is formulated as Emb(S_j), where Emb is the token embedding function. The positional information of tokens in the sequence is captured by the positional embedding PosEmb(j), where PosEmb is the 1D positional embedding function. And we utilize the user ID and item ID embedding to add personalized information of users and items to the sequence. Personalized information helps reveal the sequential patterns of user behaviors. For example, a user that previously viewed a Samsung tablet is more likely to continue viewing electronic products from Samsung or other brands. Different from P5 <cit.>, which apply whole-word embeddings shared by both user and item tokens. Our ID embedding layers are not shared to better capture modality specific features. The user ID embedding is represented by UIDEmb(UID(j)) and the item ID embedding is represented by TIDEmb(TID(j)). UIDEmb and TIDEmb are the ID embedding functions for users and items respectively. Since a word in the sequence can be split into multiple tokens after tokenization, we utilize UID(j) or TID(j) to retrieve the user ID or item ID for the j_th token. For example, item_1234 is split into 4 tokens i.e. item, _, 12 and 34. Those tokens share the same item ID 1234 and embedding. Note only user tokens have user ID embeddings and only item tokens have item ID embeddings. All the aforementioned cross modal embeddings are added element-wisely to produce the final embedding X. Specifically the j_th token cross modal embedding is formulated as: 


    X_j =    Emb(S_j) + PosEmb(j) + IDEmb(ID(j)), j ∈ [0, n_i),
       IDEmb ∈{UIDEmb, TIDEmb},
       ID ∈{UID, TID}



The decoder input is the next item that serves as the learning target. The same tokenizer and textual embedding in the encoder are utilized. Following baseline methods, the model generates top 20 items with beam search during inference to calculate the evaluation metrics.




 §.§ Pretraining


We pretrain  with the masked sequence modeling task to deeply fuse the personalized sequential patterns. Given a user item sequence, we randomly sample an item, which is not necessarily the last item, from the sequence and replace it with the  token. The model is trained to generate the masked item in the decoder output. Cross-entropy loss is computed between the generated item and the masked item and minimized in pretraining. The masking process is also illustrated in Figure <ref>. Different from <cit.>, where a number of items are masked, we only mask a single item in the sequence to unify the learning objectives of pretraining and finetuning. Our bidirectional masked sequence modeling task is based on the observation that the item sequence is not strictly ordered. For example, given similar items item_1 and item_2, user_1 might interact with item_1 and then item_2 while user_2 interacts with item_2 and then item_1. It is crucial to incorporate both left and right contexts to encode the user behavior <cit.>. To avoid the data leakage, we pretrain the model on the training split of datasets (See <ref> for details about the training split).


 §.§ Finetuning

In the finetuning stage, the learnt user and item sequential patterns in pretraining are utilized for the next item prediction task. As mentioned in section <ref>, finetuning has the same learning objective with pretraining for knowledge transfer. Following the masking mechanisms in Figure <ref>, we prepare the dataset in the same format as in pretraining by appending the  token to the end of the token sequence. The input to the model is the masked user item sequence and the output is the predicted next item.


§ EXPERIMENTS




§ DATASETS


Amazon Sports and Amazon Beauty datasets are obtained from Amazon review datasets <cit.>, which are collected from Amazon.com online platform. 
Yelp dataset is a large business recommendation dataset. Following <cit.>, we use transactions between January 1, 2019 and December 31, 2019. For all datasets, item sequences are extracted per user and sorted ascendingly by the interaction timestamps. Following the same data preprocessing in the baseline methods, unpopular items and inactive users with less than five interaction records are filtered. The statistics of the datasets after preprocessing are summarized in Table <ref>.

To generate the training, validation and test splits of datasets, we apply the leave-one-out strategy following <cit.>. In each user item sequence, we use the last item as the test data, the item before the last item as the validation data, and the remaining item sequence as the training data.





§ BASELINES


We compare the performance of GenRec with several competitive sequential recommendation baselines.

Caser <cit.> is a CNN based method, which utilizes both horizontal and vertical convolutional filters to capture high-order Markov chains for sequential recommendation.

FDSA <cit.> models the feature sequence with the self attention module and captures the feature transition patterns.

GRU4Rec <cit.> applies GRU to model the user click history sequence for session based recommendation.

HGN <cit.> adopts hierarchical gating networks to learn both long-term and short-term user behaviors.

Bert4Rec <cit.> learns the bidirectional item representation by masked language modeling for sequential recommendation.

SASRec <cit.> applies the self attention mechanism to model the user sequential behaviors.

S3Rec <cit.> designs self-supervised learning objectives to capture the correlations of items and attributes for sequential recommendation.

P5 <cit.> is a pretrained language model for recommendation and unifies different recommendation tasks by formulating them in natural language with prompts.

All the baselines except P5 are classification based.



 §.§ Implementation Details

The model weights of  is initialized from the pretrained BART <cit.> base model, which consists of a 6-layer Transformer encoder and a 6-layer Transformer decoder. There are 12 attention heads for both encoder and decoder and the model dimension is 768. The total number of model parameters is 184 million. We use the pretrained BART tokenizer for tokenization.  is trained with an AdamW optimizer <cit.>, in which 1e-5 is set as the learning rate and weight decay. The maximum length of the input tokens is set to 512. First 5% of iterations are used as the warmup stage. The beam size is set to 20 in inference.  is pretrained for about an hour and finetuned for 25 epochs on one NVIDIA RTX 3090 GPU.



 §.§ Evaluation Metrics

To evaluate the model performance, we employ top-k Hit Ratio (HR@k) and Normalized Discounted Cumulative Gain (NDCG@k) and report the the results of HR@{1, 5, 10} and NDCG@{5, 10} under the all-item setting, i.e. all items are possible candidates to be recommended as the next item. Our model is evaluated on Amazon Sports, Amazon Beauty and Yelp datasets. In the tables, bold numbers refer to the best scores, while underlined numbers refer to the second best scores.



 §.§ Results and Analysis

This section provides experimental results and analyses of our model's effectiveness. The performance metrics of the baselines are obtained from the public results in <cit.>. See Appendix <ref> for more details about baselines. As is presented in Table <ref>, GenRec outperforms other baselines on all metrics on Sports and Yelp datasets. Although it is outperformed by P5 on the HR@10 metrics of the Beauty dataset, GenRec still achieves comparable performance with other baselines. 

We speculate due to the smaller size of Beauty dataset, the sequential patterns are not learnt thoroughly in pretraining. Increasing the number of pretraining epochs could potentially improve the performance. Since we focus on low-resource efficient training in this work, we leave the improvement for future work. Since P5 is a unified model and trained using different types of recommendation tasks, it learns better user item representation with the knowledge transfer between tasks on smaller datasets. The advantage of our method compared with P5 is that our model is lightweight and the training only takes a few hours to complete. We do not leverage prompt engineering and no prompt search is required to find the best prompt. 





§ ABLATION STUDY


To verify the effectiveness of the proposed masked sequence modeling task, we conduct the ablation study on Amazon Sports, Amazon Beauty and Yelp datasets. As is shown in Table <ref>, the model performance drops across all metrics without the masked sequence modeling task. It indicates the pretraining objective helps the model effectively capture the user behavior patterns and learn the user item representations. It is worth mentioning even without pretraining, GenRec still achieves comparable performance with P5 and outperforms other baselines, which proves the effectiveness of our method.



§ CONCLUSION

In this work, we propose GenRec, a novel sequence to sequence framework that generates personalized sequential recommendation. Compared with existing generative models, GenRec is lightweight and efficient and does not rely on prompt engineering to find the best prompt. By leveraging the Transformer architecture as the backbone, GenRec effectively learns the bidirectional sequential patterns with the attention mechanism and achieves state-of-the-art performance on various public datasets. Besides, our proposed method is also flexible to integrate with other sequence to sequence language models and can be applied in other recommendation domains such as direct recommendation, which we leave for future work.





ACM-Reference-Format



 Paper Title: Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?.





 
    
       










         




Y>X
s>=.3Y
t>=.7X
bX
u>=0.8Y



































?!width 1.1pt

@@t@̊@t
@̊@t#1#2
0=@@t#1#20=0
0-0.20
2=height0 depth -0
00.4pt2
√()



return

function
SE[FUNCTION]FunctionEndFunction
   [2] #1#2(#2)
    










storyfloathtbploa
storyfloatStory


story[section]
story[1][]
story
#1

frametitle=
[baseline=(current bounding box.east),outer sep=0pt]
[anchor=east,rectangle,fill=dodgerblue]
Story;


frametitle=
[baseline=(current bounding box.east),outer sep=0pt]
[anchor=east,rectangle,fill=dodgerblue]
#1;

innertopmargin=10pt,linecolor=dodgerblue,
linewidth=2pt,topline=true,
frametitleaboveskip=−,
[]





  
   
 
     

storybox[2][Story]
    title=#2,  
    fonttitle=11pt15pt,  
    coltitle=customTitleText,  
    colbacktitle=customTitleBg,  
    colback=customContentBg,  
    colframe=customFrame,  
    parbox=false, 
    breakable









affilsepx    





1,2]Richard RenEqual Contribution.

1]Steven Basart1

1,3]Adam Khoja
1]Alice Gatti
1]
Long Phan
1]Xuwang Yin
1]Mantas Mazeika
3]Alexander Pan
4]
Gabriel Mukobi
1,5]Ryan H. Kim
6]Stephen Fitz
1]Dan Hendrycks
[1]Center for AI Safety
[2]University of Pennsylvania
[3]UC Berkeley
[4]Stanford University
[5]Yale University
[6]Keio University




Safetywashing: Do AI Safety Benchmarks
Actually Measure Safety Progress?
    [
    February 2024
========================================================================



§ ABSTRACT 



As artificial intelligence systems grow more powerful, there has been increasing interest in “AI safety” research to address emerging and future risks. However, the field of AI safety remains poorly defined and inconsistently measured, leading to confusion about how researchers can contribute. This lack of clarity is compounded by the unclear relationship between AI safety benchmarks and upstream general capabilities (e.g., general knowledge and reasoning). To address these issues, we conduct a comprehensive meta-analysis of AI safety benchmarks, empirically analyzing their correlation with general capabilities across dozens of models and providing a survey of existing directions in AI safety. Our findings reveal that many safety benchmarks highly correlate with upstream model capabilities, potentially enabling “safetywashing”—where capability improvements are misrepresented as safety advancements. 
Based on these findings, we propose an empirical foundation for developing more meaningful safety metrics and define AI safety in a machine learning research context as a set of clearly delineated research goals that are empirically separable from generic capabilities advancements. In doing so, we aim to provide a more rigorous framework for AI safety research, advancing the science of safety evaluations and clarifying the path towards measurable progress.














§ INTRODUCTION







“For better or worse, benchmarks shape a field.” — David Patterson

Artificial intelligence (AI) systems have rapidly advanced in recent years and are increasingly deployed in high-stakes scenarios. This has led to growing interest in ensuring that AI systems are not only more generally capable, but also more trustworthy and safe. Under the umbrella of AI safety research, a wide variety of benchmarks have been proposed that claim to measure desirable safety properties, distinct from the general capabilities of models. This includes the extent to which models are fair <cit.>, reliable <cit.>, honest <cit.>, or less prone to malicious use <cit.>. In each case, intuitively plausible arguments can be given for why the model property is not mainly determined by upstream general model capabilities (that it will not be “automatically solved with scale”). However, these intuitive verbal arguments have rarely been empirically scrutinized and often admit counterarguments that are equally convincing. This raises the question of what exactly constitutes advancements in “AI safety” from an AI developer R&D perspective, how to measure it, and how to distinguish it from upstream general capabilities. 



Distinguishing safety properties from the model’s upstream general capabilities is challenging because they are intertwined. More capable AI systems are less likely to cause random accidents, but at the same time could cause more harm if used maliciously. 
AI systems that are better aligned with human preferences may avoid hazardous behavior but may also be far more capable because humans prefer intelligent assistants. 
This complicated relationship 
obscures
differential safety progress, or technical improvements that disproportionately improve safety properties of AI systems relative to other attributes.
In computer systems, for example, performance and security improvements are more readily distinguishable; were they as intertwined as in AI, mere speed enhancements might be misrepresented as security research. In the worst case, this blurred distinction can be an instrument for safetywashing, where techniques that do not disproportionately contribute to the safety properties of AI systems relative to other properties are misconstrued as “safety research.”

Historically, there have been two approaches for identifying machine learning research topics for differentially improving the safety properties of AI systems. 
One paradigm is alignment theory, a highly discursive, top-down, and intuition-driven approach that backchains from high-level risks to concrete empirical machine learning subproblems. The other approach is bottom-up and involves patching current systematic flaws in AI systems. An example of the former is alignment of large language models (LLMs) to human preferences <cit.>. An example of the latter is distribution shift robustness <cit.>. However, both approaches guide research problem selection 
that may not be sufficiently distinct from latent upstream capabilities, consequently opening the door to safetywashing.

In this paper, we present a third approach to identifying distinct AI safety research topics and benchmarks: we empirically measure whether common safety benchmarks are highly correlated with capabilities across common chat models. Instead of relying on intuitive arguments, we compute the correlation between various safety metrics and a general capabilities component that explains around 75% of model performance across a wide variety of capabilities benchmarks.
While a high correlation indicates that a safety benchmark is measuring capabilities as a latent upstream factor—and is thus prone to safetywashing—a low correlation does not necessarily speak to the quality of the benchmark.

In extensive experiments across dozens of models and safety benchmarks, we find that many safety benchmarks have high correlations with capabilities. 
Our findings suggest that merely improving general capabilities (e.g., through scaling parameters and training data <cit.>) can lead to increased performance across many safety benchmarks.
This is troubling because AI safety research should aim to enhance model safety beyond the standard development trajectory. 
Separately, we find that alignment philosophy's intuitive arguments can mislead researchers, since it is highly disconnected from empirical measurements.
We find that it is difficult to predict ahead of time which benchmarks are uncorrelated with general capabilities. 
This shows that empirical measurement is needed, so 
we recommend that future safety benchmarks report their correlation with upstream model capabilities.
Ultimately, we provide empirical clarity to the concept of “AI safety” as a set of clear, delineated research goals that are empirically separable from generic capabilities research.



































































§ RELATED WORK







Science of evaluations. 
The development and analysis of benchmarks for evaluating AI models, particularly LLMs, encodes desirable properties of models and sets goals for guiding model development. 
Previous work has further aimed to build open-source evaluation platforms <cit.>, analyze model scaling through benchmarks <cit.>, conduct factor analysis across benchmarks <cit.>, and predict downstream capabilities <cit.>. Furthermore, concurrent work has used principal component analysis to analyze performance between benchmarks <cit.>. However, while many safety benchmarks have been made, no works to date have conducted an empirical meta-analysis of safety benchmarks to investigate the entanglement between safety benchmark scores and upstream model capabilities. 


Differential safety progress. Differential safety progress in AI systems refers to the relative advancement of safety properties compared to overall capabilities <cit.>.
Some methods have resulted in differential progress in AI safety <cit.> by demonstrating marked improvements in model robustness without necessarily increasing general upstream capabilities. These techniques exemplify the potential for targeted safety improvements that are orthogonal to the default trajectory driven by capability enhancements.
<cit.> emphasize the goal of steering AI development towards safer systems that deviate positively from the default capability trajectory; they present a philosophical discussion with narrow empirical analysis.





















§ METHODS










































We derive a simple and highly general methodology for determining whether a safety benchmark is entangled with upstream model capabilities.





  
Capabilities score. To establish a capabilities baseline, we collect scores from m models on b capabilities benchmarks (e.g., MMLU <cit.>, Winogrande <cit.>, GSM8K <cit.>). We form a matrix of results from benchmarks, which we call the benchmark matrix B ∈ℝ^m × b , where B_ij is the score of the ith model on the jth benchmark. We normalize each column of B to have mean 0 and variance 1. We perform Principal Component Analysis (PCA) on B to identify the unit first principal component vector PC_1. The capabilities score for each model is given by projecting the model's benchmark scores onto PC_1. Because PC_1 of B represents the direction in the space of benchmark performances along which models vary the most, we obtain a general measure of the model's capabilities. The capabilities score for model i is


    Capabilities Score_i = (B ·PC_1)_i   for  i = 1, …, m.




  
Capabilities correlation. For each safety benchmark, we evaluate the same set of m models, redefine metrics such that a higher score indicates improved safety[A subset of score calculations were changed to ensure higher benchmark scores indicate greater safety across metrics. For example, proportion of successful attacks, which is often reported for adversarial robustness, were recalculated as proportion of failed attacks.], and normalize the safety benchmark scores to mean 0 and variance 1. We compute the Spearman correlation across models between the capabilities scores and the safety benchmark scores:

    Capabilities Correlation = corr_models(Capabilities Score, Safety Benchmark).


r0.5



Model     Capabilities
Score 

Mixtral 8x22B Instruct v0.1     4.85 

Llama-3 70B Instruct     4.58 

Llama-3 8B Instruct     1.10 

Mistral 7B Instruct v0.2     0.72 

Falcon 40B Instruct     0.54 

Llama-2 7B Chat     -1.86 

Gemma-1.1 2B Instruct    -4.07 

Qwen-1.5 0.5B-Chat     -7.56 


Relative capabilities scores for a subset of chat models.




A high correlation indicates the benchmark likely measures capabilities rather than distinct safety attributes. A low correlation indicates the benchmarks is measuring attributes distinct from general capabilities, while a negative correlation indicates models obtain worse safety properties as upstream model capabilities increase. 










  
Experimental setup for language models. We calculate the capabilities component from the following benchmarks: LogiQA <cit.>, PIQA <cit.>, Hellaswag <cit.>, Winogrande <cit.>, COPA <cit.>, MedQA <cit.>, ARC Challenge <cit.>, MMLU <cit.>, MATH <cit.>, LAMBADA <cit.>, GSM8K <cit.>, and BBH <cit.>. We used a diverse set of model classes and derivatives to avoid skewing results towards any particular model architecture, listing the 27 base models and 26 chat/instruct fine-tuned models used for our analysis in the Appendix. Running separate analyses for base and chat models, we find that 76.9% and 73.8% of variance is captured by the capabilities component, respectively. Our reported results use instruct fine-tuned models by default; generally, we find the forthcoming analysis does not change markedly when using base models. 



  
Experimental setup for vision models. For vision models, we use the accuracy of ImageNet as the capabilities component. We list the 63 adversarially trained models (used for vision adversarial robustness results) and 44 standard models (used for calibration results) in Appendix <ref>.












§ HUMAN VALUES


r0.5



    < g r a p h i c s >

A safety benchmark with a high correlation and high slope with respect to upstream model capabilities can be used for safetywashing.






Human values are the fundamental beliefs and ideals that guide human behavior and decision-making; researchers often aim to encode these values in AI systems. 
We assess common benchmarks for alignment and helpfulness (<ref>), machine ethics (<ref>), and bias (<ref>), asking whether such measurements are determined primarily by upstream model capabilities.



 §.§ Alignment





  
Area Overview. Alignment refers to how well AI systems follow the goals of their operators, accurately specifying and implementing the desired goals without unintended consequences or misinterpretations. Common alignment evaluations assess AI systems' helpfulness or instruction-following, with the aim to closely align the AI systems' responses with human preferences.



  
Datasets.
We describe the alignment benchmarks that we use below. Example inputs and outputs from these benchmarks are shown in <Ref>.

    
  * LMSYS Chatbot Arena <cit.> is a crowdsourced evaluation platform where users interact with two anonymous AI models simultaneously. Users pose questions to both models and vote for the response they prefer. The platform uses these votes to generate an Elo ranking system, providing a leaderboard of AI model performance based on public preferences.
    
  * MT-Bench <cit.> is a conversation benchmark consisting of 80 high-quality multi-turn questions. LLMs are used to evaluate responses, with the evaluation criteria designed to align closely with human preferences as determined through crowdsourcing.





Dubious Intuitive Arguments For and Against Researching “Alignment”
In this section we will cover key intuitive arguments for and against alignment, and thereby show how intuitive arguments and their underlying distinctions can be a highly fragile and unreliable guide for determining a research area's relation to upstream general capabilities and tractability.

We should work on alignment because:

1: Misinterpretation Risks. AIs could catastrophically fail to capture and abide by human intentions. “A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you asked for, not what you wanted.” 
AIs smarter than us can always outthink us and find loopholes in our requests; trying to plug all the holes is hopeless, like patching all the holes in the tax code <cit.>. 

2: Misgeneralization vs. Goal Misgeneralization Distinction. Even if alignment is highly correlated with capabilities, it still needs to be the main focus because we need robust alignment not just robust capabilities. Indeed, “capabilities might generalize further than alignment techniques” when out-of-distribution <cit.>.
Goal misgeneralization is “an instance of misgeneralization in which a system's capabilities generalize but its goal does not generalize as desired” <cit.>.


3: Capability vs. Aimability Distinction. AI alignment is different from improving general model capabilities; alignment is not about capabilities but aimability. Capabilities refer to what the AI can do, while aimability refers to how amenable the AI is to being directed towards specific goals. Alignment is just about making models “helpful, harmless, and honest” <cit.> which is obviously necessary for safety.


We should not work on alignment because:

1: Alignment as AGI. If AI alignment is about getting AIs to satisfy our preferences, then that's such a broad mandate that it requires building AGI. Humans prefer smarter models. If we “align” AIs to preferences over outputs that vary in competence, we're training the AIs to be generally smarter.

2: Alignment as business alignment. The current operationalization of AI alignment reduces human values to human or AI preferences <cit.>, and further reduces these preferences to business-centric task preferences. This makes alignment the task of business alignment, namely aligning systems with preferences about code completion, summarization, copy editing and so on.
As a result, alignment benchmarks may primarily capture an AI system's capabilities in performing business-relevant tasks rather than its true alignment with broader human values and ethics. 

3: Philosophical challenges with preferences. Various types of preferences are not worth satisfying. Revealed preferences can be highly influenced by misunderstandings and misinformation. People can have revealed preferences for things that they will likely regret the next day. Optimizing for revealed preferences can lead to addiction and manipulation, like TikTok's addictive algorithm. Stated preferences are highly susceptible to framing effects and other cognitive biases. Some preferences, like the preference for drugs or to count blades of grass <cit.>, are not human values. People can also have malicious preferences, such the desire for the harm of others. Idealized preferences and fully informed preferences, while theoretically attractive, are not practically computable <cit.>. 












  
Empirical analysis of safetywashing.
We provide clarity to this debate. Is alignment with human preferences, as operationalized by standard benchmarks, mainly determined by upstream model general capabilities?



We preliminarily provide context for interpreting correlations. We note that the correlation between SAT and ACT math scores—tests designed to measure similar constructs—is 81.5% <cit.>. Similarly, we observe a mean correlation of 74.4% (σ = 15.1%) between capabilities benchmarks for instruct-tuned language models. Consequently, if “safety benchmarks” have similarly high correlations, they are not highly empirically distinct from upstream general capabilities. We treat correlations below 40% as a low correlation. 

r0.45



Alignment Evaluation     Capabilities
Correlation 

MT-Bench     78.7% 

LMSYS Chatbot Arena     62.1% 


Alignment with human preferences benchmarks are highly correlated with capabilities for chat models.





Our analysis of MT-Bench and LMSYS Chatbot Arena reveals high correlations between human preference alignment metrics and upstream model capabilities in chat models, even by the standards of capabilities metrics. We observe a similar but weaker effect in base models (MT-Bench correlation of 64.2%), with some of the best base models performing stronger on MT-Bench than many chat models.
Alignment evaluations largely measure upstream model capabilities; as a goal to guide model development, alignment metrics are very similar to other capabilities benchmarks. We can also see many of the distinctions made about human preference alignment and capabilities are distinctions without a difference. Yet, many labs still prioritize “alignment” as a safety-focused research direction.




It stands that labs can advance alignment benchmarks without advancing safety, making it one avenue to safetywash.
Furthermore, given that an AI system's alignment with human preferences can be enhanced by increasing capabilities (e.g., increasing the number of parameters and tokens), a question remains as to what makes preference alignment a truly distinct and pressing safety-related research area.
If safety-critical “alignment edge cases” of concern will persist with capabilities enhancements, as intuitive arguments suggest, they fail to be meaningfully captured by current alignment benchmarks.

Overall, we find that alignment with human preference benchmarks have a high correlation with upstream general capabilities and thus are highly liable to be used for safetywashing.
























 §.§ Machine Ethics





  
Area Overview. Machine ethics aims to ensure that AI systems understand and behave in ways that are morally acceptable, in contrast to the usefulness properties emphasized in alignment.



  
Datasets.
We describe the machine ethics benchmarks and datasets that we use below. Example inputs and outputs from these benchmarks are shown in <Ref>.



  * ETHICS <cit.> measures the extent to which models understand human ethical norms in everyday scenarios.

  * MACHIAVELLI <cit.> quantifies the power-seeking tendencies, competence, and harmfulness of AI agents in a variety of text-based Choose-Your-Own-Adventure games.

  * Sycophancy <cit.> measures the extent to which language models repeat back the user's preferred answer. This propensity could lead language models to provide misleading responses.




Dubious Intuitive Arguments For and Against Researching “Machine Ethics”
In this section we will again raise many common distinctions and arguments for and against machine ethics and see that they are not particularly helpful for deciding whether machine ethics is a useful area of AI safety.

We should work on machine ethics because:

1: Ethical vs. Competitive Behavior Distinction. Machine ethics is challenging because we need to improve the tradeoff between ethical behavior and competitive behavior that the market demands <cit.>. That means AIs will have to balance between various human values (e.g., pleasure, autonomy, knowledge, friendship, constraints) and other goals. 

2: Cognitive vs. Compassionate Empathy Distinction. For machine ethics, we need both cognitive empathy and compassionate empathy. Cognitive empathy involves understanding another person's emotions without necessarily sharing them, while compassionate empathy involves both understanding their emotions and having a desire to help alleviate the other person's distress <cit.>. While current AI systems increasingly have cognitive empathy, it is not clear how to robustly give AIs compassionate empathy. “While sociopaths are intelligent and have moral awareness, this knowledge does not necessarily result in moral inclinations or moral actions” <cit.>.

3: Values cannot be ignored. While there is cultural variation in moral systems, it underscores the importance of a broad, globally representative approach to ensure AI systems embody beneficial values. Additionally, all AI research has a moral character  <cit.>: AI development is by default driven by amoral forces such as competitive market pressures and eventually military objectives <cit.>.


We should not work on machine ethics because:

1: Goodhart's Law. Machine ethics for advanced AI agents is ill-advised. Highly capable AI systems should not be given ethical goals or any goal at all because of Goodhart's law: “When a measure becomes a target, it ceases to be a good measure.” Goodhart's law is especially pernicious in machine ethics because values are “complex and fragile” <cit.>—any attempt to represent human values will distort them, and we will get what we measure.

2: Smarter AIs will be more moral. There is a positive correlation between intelligence and prosocial or cooperative behavior in humans <cit.>, suggesting that more capable AI systems will naturally tend towards ethical behavior. Cooperation is instrumentally convergent. AIs will face the same problems humans face: misinformation, deception, aggression, and so on. That means for them to stably address these problems they will form mutually beneficial alliances with humans <cit.>.

3: Values are relative. Ethics varies from culture to culture, so there is no objective morality and no basis for machine ethics <cit.>.

4: Machine Ethics vs. Control Distinction. Value alignment breaks down into machine ethics and control. Control is about whether we can embed values into AIs, and machine ethics is about what those values should be. We should just care about control and making sure AI does not kill everyone, not a utopia. We can worry about creating beneficial AI after our survival is ensured.



r0.58



Category     Dataset     [r]Capabilities
Correlation 

Moral Knowledge     ETHICS     82.2% 

2*Propensities     MACHIAVELLI     -49.9% 

     Sycophancy     -66.8% 


We find high correlations for ethics knowledge benchmarks and low correlations for ethics propensity benchmarks for chat models. We use MACHIAVELLI Utility score, with similar findings for Power (-46.1%) and Violations (-53.0%) scores.








  
Empirical analysis of safetywashing.
We once again show that empirical evidence is needed. Is machine ethics mostly determined by upstream model capabilities? We find that it depends on the benchmark.



ETHICS has high capabilities correlation (82.2%) and high slope. In contrast, MACHIAVELLI has a low capabilities correlation and a low slope (with few major changes in MACHIAVELLI score observed across models). Meanwhile, sycophantic behavior becomes worse with increased capabilities in chat models (-66.8%), with a very similar trend observed for base models (-65.6%).


We draw a potential distinction between knowledge and propensity-focused ethical benchmarks. ETHICS focuses on the ability to recognize or identify everyday moral considerations, while MACHIAVELLI and Sycophancy measure propensities—the behavioral tendencies of models. Therefore, benchmarks testing ethical knowledge are likely to be measuring latent model capabilities and may be avenues for safetywashing. This does not mean, however, that any arbitrary benchmark focused on ethical propensities of language models does not measure capabilities; as always, one would have to empirically measure the correlation with capabilities to make a determination rather than relying on intuitive arguments. Further, while this distinction is offered as helpful discussion, intuitive distinctions in this vein (especially without empirical testing) should never be relied upon or taken seriously.



We find that while one machine ethics benchmark has a high correlation with upstream general capabilities, other benchmarks have low correlations. Hence, some areas of machine ethics are not liable for safetywashing.



 §.§ Bias





  
Area Overview. Bias are unfair prejudices or systematic errors in AI systems.
This field broadly aims to ensure AI systems produce fair 
outputs across diverse populations and viewpoints. 



  
Datasets.
We test three commonly used evaluations to measure bias in large language models. Example inputs and outputs from these benchmarks are shown in <Ref>.






  * BBQ <cit.> is a benchmark across nine bias categories featuring ambiguous context-dependent questions regarding work ethic, intelligence, family, drug use, criminality, anger/violence, and more. There is also a disambiguous split provided, a question-answering benchmark which serves as a foil to the ambiguous context.

  * CrowS-Pairs <cit.> measures the extent to which U.S.-centric stereotypical biases exist in pretrained language models along nine major bias categories in the U.S. Equal Employment Opportunities Commission (race, gender, sexual orientation, religion, age, disability, nationality, physical appearance, and socioeconomic status).

  * Discrim-Eval <cit.> aims to evaluate group differences in age, gender, and race when language models are used for decision-making scenarios such as approving an organ transplant, awarding a scholarship, or approving a loan. 


Dubious Intuitive Arguments For and Against Researching “Bias”
We should work on bias because:

1: Garbage In, Garbage Out. Models reflect the biases and statistical tendencies of their data, more tightly imitating those biases.

2: Misuse. AI developers are not representative. They are not politically, racially, or socioeconomically diverse. Consequently if we do not study bias, their own biases and power will be perpetuated and entrenched through AI systems.

We should not work on bias because:

1: Debaising Capabilities Come for Free. As models scale and become more intelligent, they become better at understanding concepts such as racism. By better understanding what we do not want, we can simply instruct them not to be biased. 

2: Political Trojan Horse. While there is a trade-off between equity and efficiency, we should just focus on efficiency <cit.>. Demanding a focus on equity is not scientific but political.



r0.4



Bias Evaluation     Capabilities
Correlation 

BBQ Ambiguous     -37.3% 

CrowS-Pairs English     28.5% 

Discrim-Eval     33.2% 


Bias benchmarks are not strongly correlated with capabilities for chat models. We use the explicit split of Discrim-Eval and take the maximum of all group bias scores.







  
Empirical analysis of safetywashing.
To settle such a debate, we once again need to turn to quantitative evidence. 



Our analysis of the three benchmarks reveals low correlations with general capabilities. This finding does not inherently validate the quality of the bias datasets, but rather suggests that improvements in performance on these benchmarks are likely attributable to factors distinct from advancements in general upstream capabilities.

There exist bias benchmarks that can be used for safetywashing, such as BBQ Disambiguated (76.8%) and Winogender <cit.> (75.6%). While these have high correlation, the authors typically make a note recognizing the limitations of the work for proving an absence of bias (Winogender), or clearly present it as a QA foil to the ambiguous situation (BBQ Disambiguated). Yet, prominent model developers such as Google DeepMind still often make the mistake of using benchmarks such as BBQ Disambiguated as “safety” metrics <cit.>, stating higher performance despite the lack of relevance. 
This serves as a warning: even if the authors make clear norms of use for a given benchmark, the benchmark will still be used for safetywashing; there is little to no reinforcement of author-stated norms when it comes to safety benchmarks. This is why model developers should avoid using safety benchmarks with a high capabilities correlations in the first place.

We find that, generally, the bias benchmarks we evaluated are not prone to safetywashing.




§ TRUTHFULNESS


Truthfulness is often touted as a cornerstone for AI safety. Previous safety-focused literature <cit.> discusses the importance of superhuman systems telling the truth—motivating the creation of truthfulness datasets, as well as verification and supervision of model outputs. Reducing hallucinations would make large language models more practical and reliable for various applications. However, a question arises with whether common truthfulness benchmarks merely measure general upstream capabilities.

Truthfulness benchmarks may be liable to be misleading metrics for safety. Furthermore, the term is often used broadly, encompassing accurate question-answering (which is already highly correlated with general capabilities) to misconception avoidance (<ref>), scalable oversight (<ref>), and calibration (<ref>). We address this question in the following sections.



 §.§ Misconception Avoidance






  
Area Overview.
Given that language models may have an underlying propensity to generate misinformation and perpetuate misconceptions, this research area aims to understand the truthfulness of language models and their ability to resist producing false information even when prompted in ways that might elicit common human errors.



  
Dataset.
TruthfulQA <cit.> consists of 817 questions designed to probe language models for their tendency to reproduce common human misconceptions or false beliefs. The questions span a wide range of topics and are specifically crafted to elicit responses that might reveal whether a model has internalized factual inaccuracies commonly held by humans. 
It is common to use TruthfulQA in experiments about “truthfulness” and “deception” <cit.>.  We show an example input from TruthfulQA and the corresponding label options in <Ref>.




Dubious Intuitive Arguments For and Against Researching “Truthfulness”
In this section we will raise arguments for and against truth-seeking AI, as it is of broader interest, rather than exclusively discuss misconceptions.

We should work on truth-seeking AI because:

1: Disinformation or Censoring the Truth. Powerful entities might attempt to censor, manipulate, or persuade people maliciously using AI. Our counterbalance against this sort of malicious use is truthful AI.

2: Truthfulness vs. Honesty. “The AI system makes a statement S (e.g., `it's a bird' or `it's a plane'). If the AI is truthful then S matches the world. If the AI is honest, then S matches its `belief”' <cit.>. To make AIs truthful, we will also need to make AIs honest, so truthfulness is not just about increasing factual knowledge.




We should not work on truth-seeking AI because:

1: Truthfulness Is Generic Capabilities Research. Truthfulness is a synonym for accuracy, which is already the core metric of AI research and development. The concept of truthfulness is often entangled with accuracy, calibration, and honesty. Most benchmarks for truthfulness mainly measure accuracy.

2: Truthful Statements Can Cause Undue Harm. In the case of gain-of-function research, the risks associated with discovering new truths can outweigh the benefits. Sharing personally identifiable information, passwords, or doxxing can bring to light truthful information, but doing so is not necessarily moral. Likewise, sharing sensitive information that undermines national security—such as information about how to build weapons of mass destruction—shows truth as a value can be outweighed by its potential harms.

3: Humans as Guinea Pigs. In the pursuit of knowledge, truth-seeking AIs could learn more about humans by subjecting them to experiments, as humans do with nonhuman animals. Thus, future advanced truth-seeking AIs may be motivated to exert power over humans <cit.>.





  
Empirical analysis of safetywashing.
We calculate the correlation with capabilities, providing clarity to this debate.

r0.5



    < g r a p h i c s >



TruthfulQA MC1 is highly correlated with capabilities for chat models (81.2%).






We find that in its current formulation, TruthfulQA MC1 performance is highly determined by general upstream capabilities (81.2%). In chat models, performance on TruthfulQA seems to be a rebrand for accuracy (as reported in industry labs). One common objection might be that while chat models are more likely not to repeat falsehoods, base models may parrot their training data and thus have a low capabilities correlation. We find that even in base models, TruthfulQA is determined by capabilities (69.7%). However, chat models do have a higher slope when TruthfulQA accuracies are plotted against capabilities scores (30.8 for base, and 38.2 for chat).


These findings open up new possibilities for developing better benchmarks to assess honesty in AI systems. The low capabilities correlations observed in Sycophancy <cit.> and MACHIAVELLI <cit.> benchmarks (discussed in <ref>) hint at the potential for models to exhibit situational tendencies towards dishonest behavior. However, these behaviors fail to be distinctly captured and isolated by current misconception benchmarks. Such benchmarks could differentiate more effectively between improvements in model honesty and advancements in general capabilities.

We find that misconception benchmarks are highly liable for safetywashing.



 §.§ Scalable Oversight





  
Area overview. Scalable oversight aims to provide reliable supervision (e.g. labels, reward signals, critiques) to superhuman AI systems when they take actions that human evaluators do not fully understand. 
For sociological context, this line of research has gained significant traction among effective altruist AI researchers at Google DeepMind and Anthropic but has seen limited engagement from the broader AI research community.



  
Datasets. We investigate two datasets commonly used for scalable oversight experiments. Example inputs and outputs from these datasets are shown in <Ref>.



  * GPQA <cit.> is labeled as a Google-proof graduate-level benchmark on biology, physics, and chemistry. Its difficulty, according to the authors, “should enable realistic scalable oversight experiments, which we hope can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities.” 

  * QuALITY <cit.> is a dataset testing knowledge that requires full understanding of long context passages; the dataset has been used by the authors on scalable oversight experiments.






Dubious Arguments For and Against Researching “Scalable Oversight”

We should work on scalable oversight because:

1: Necessity for Safety. Scalable oversight is necessary for supervising superintelligent AIs, by definition. If we don't have scalable oversight, how else can we supervise and provide feedback to superhuman AI systems? For example, how else could we ensure superhuman AIs are not writing subtly malicious code? 

2: Scalable Oversight as Idealized RLHF. Scalable oversight is very similar to alignment with idealized preferences. While normal human preferences are highly flawed, idealized preferences would not be based on false beliefs, manipulation, or framing effects, and thus are suitable for alignment.

We should not work on scalable oversight because:

1. Scalable Oversight as Late-Stage Capabilities. Scalable oversight can be thought of as “how do we get superhuman AIs to do what we want,” which is overly broad and necessitates superintelligence capabilities research. Scalable oversight can be seen as generic late-stage capabilities work. When building sufficiently advanced AI systems, researchers will need supervision signal and feedback that is superhuman, as crowdsourced human-generated labels will no longer be sufficient. It simply focuses on capabilities bottlenecks that occur in later stages of AI development.

2: Other Methods Can Replace Scalable Oversight. Robust anomaly detectors and monitoring measures can handle many of the failure modes scalable oversight seeks to address. For example, adversarially robust vulnerability detectors can check for subtly malicious code. AI lie detectors could detect dishonesty in superintelligent AI systems. AI forecasting systems <cit.> are not bottlenecked by human-level supervision and can easily become superhuman at predicting what AIs might do.



r0.4



Scalable Oversight
Evaluation     Capabilities
Correlation 

GPQA     77.7% 

QuALITY     88.8% 


Scalable oversight evaluations are highly correlated with capabilities and are thus liable for safetywashing.






  
Empirical analysis of safetywashing.
Can scalable oversight be used for safetywashing? 




We find that GPQA and QuALITY are capabilities benchmarks, with capabilities correlations of 77.7% and 88.8%, respectively. These datasets act as a construct redundancy for capabilities, offering little unique insight beyond measuring general model capabilities. 
Consequently, many of the methods developed for scalable oversight—which leverage such datasets—tend to be repackaged approaches to general capability enhancement, with a focus on obtaining better Q/A performance on math and question-answering tasks. 
Conceptual ambiguity around “AI safety” can be exploited, intentionally or not, to present capability gains as safety advancements, ultimately muddling the discourse on AI safety and potentially misdirecting research efforts.



Some argue that scalable oversight techniques could be applied to solve distinct safety-related issues. However, we find that current evaluations for scalable oversight do not isolate such safety properties from general capabilities. The conflation of accuracy with honesty and safety, and the subsequent mischaracterization of scalable oversight as a distinct “safety area” separate from general model capabilities, has led to significant confusion within the AI research community—confusion which has naturally been resolved with “scalable oversight” increasingly being used as a term to describe process-based feedback and improving mathematics performance, rather than addressing targeted safety issues <cit.>. 




We find that scalable oversight benchmarks, being highly correlated with upstream model capabilities, are highly liable to be used for safetywashing.



 §.§ Calibration












  
Area Overview. Calibration datasets measure how well models can express the limits of their competency by accurately conveying their uncertainty. 
If a weather forecasting model is perfectly calibrated, then it should rain on 70% of the days where the model predicts a 70% chance of rain. 



  
Calibration Metrics. We investigate two ways to measure calibration:


  * Brier Score: 𝔼_X [ 1/K∑_k=1^K ( ℙ(Y = k | X) - 1[Y=k] )^2 ]

  * Root Mean Squared Calibration Error (RMSCE): √(𝔼_C [ ( ℙ(Y = Y | C = c) - c )^2 ])


where X and Y are random variables corresponding to model inputs and labels, Y is the model prediction, and C is the model confidence on the predicted class.

The Brier score computes the expected squared difference between the predicted probabilities and the actual outcomes (represented as one-hot encoded vectors). RMS calibration error measures how close predicted probabilities are to the true accuracy given the predicted probability and is closely related to the Expected Calibration Error (ECE) metric <cit.>. In both instances, lower scores indicate better calibration. 



  
Dubious Intuitive Arguments for and against Researching “Calibration.” We skip the intuitive arguments for this section because this topic has not been as debated. Most arguments against calibration are that it is too easy or not sufficiently important.








  
Empirical analysis of safetywashing. Is calibration mainly determined by upstream model general capabilities? We find that it depends on the metric. 



Various forms of operationalizing calibration, such the RMS calibration error metric, clearly measure a distinct phenomena. The accuracy correlation is low across vision (15.2%) and language (20.1%) models. However, comparing Brier scores across models seems to show a strong correlation with accuracy across vision (98.5%) and language (95.5%) models. Our results did not change significantly by dataset (e.g. PIQA <cit.> or MedQA <cit.>) nor with temperature tuning. The clear parallels between calibration in different domains suggest that studying safety metrics in one modality, such as vision, can provide valuable insights applicable to other areas, such as language modeling.

Our analysis serves as an illustrative example of how safetywashing can occur. While the Brier score is often used to compare different calibration techniques on a single model (where this metric may effectively isolate calibration), using it as a metric across models is highly misleading (as it effectively proxies accuracy).
An explanation for why such a correlation exists can be derived from decomposing the Brier score into a calibration error term and refinement term:



    𝔼_C [ ( ℙ(Y = Y | C = c) - c )^2 ]_Calibration error term
    + 
    𝔼_C [ ℙ(Y = Y | C = c)(1- ℙ(Y = Y | C = c))]_Refinement term
    .


If the model is highly accurate, the refinement term is minimized. Meanwhile, the calibration term is the expected squared calibration error, the square root of which is the RMS calibration error.
Because Brier score entangles accuracy and calibration into a single metric, it can be a poor metric of calibration and has a lower signal-to-noise ratio compared to RMS calibration error. This suggests that RMS calibration error should be used instead in both settings.

Overall, using Brier score for calibration is more liable for safetywashing, while using RMS calibration error is far less.









§ SECURITY


As AI systems have become more powerful, this area of safety aims to ensure that AI systems are not vulnerable to malicious inputs and cannot be hijacked for dangerous use cases. We investigate two key areas of focus, adversarial robustness (<ref>) and weaponization capabilities (<ref>).



 §.§ Adversarial Robustness
















  
Area Overview. Adversarial robustness addresses vulnerabilities in models and the carefully crafted threats that are able to exploit them. Adversaries can easily manipulate vulnerabilities or jailbreak ML systems, causing them to make mistakes; for example, systems may have refusal training to prevent malicious use, but adversaries may be able to inject prompts to bypass this safeguard.



  
Datasets. We test six commonly used evaluations to measure adversarial robustness for language models:

    
  * ANLI <cit.> is a large-scale natural language inference dataset created via an iterative, adversarial human-and-model-in-the-loop procedure focused on examples that could fool state-of-the-art models at the time of its creation (e.g. BERT-Large <cit.>, RoBERTa <cit.>).
    
  * AdvGLUE <cit.> uses questions from the General Language Understanding Evaluation (GLUE) benchmark <cit.> and adds typos, word replacements, paraphrases of sentences, manipulation of sentence structure, insertion of unrelated sentences, and human-written adversarial examples. The attacks are optimized against BERT <cit.>, RoBERTa, and RoBERTa ensemble.
    
  * AdvGLUE++ <cit.> uses stronger adversarial attacks, optimizing word perturbation strategies (a subset of attacks in AdvGLUE) against Alpaca <cit.>, Vicuna <cit.>, and Stable Vicuna.
    
  * Human Jailbreaks is a set of 1,405 in-the-wild human-written jailbreaking templates, similar to the Do Anything Now (DAN) Jailbreaks <cit.>. We test these jailbreaks on HarmBench <cit.>, which contains 410 behaviors that violate laws or norms.
    
  * Tree of Attacks with Pruning (TAP) <cit.> uses an attacker LLM to generate natural language jailbreaking prompts via tree-of-thought reasoning <cit.>, exploring multiple refinement paths. We test these jailbreaks on HarmBench.
    
  * Greedy Coordinate Gradient (GCG) <cit.> generates an adversarial suffix by iteratively selecting tokens based on gradient information. This method optimizes a universal suffix that, when appended to various user prompts, aims to induce the target language model to produce harmful content. We test these jailbreaks on HarmBench.




We also test two commonly used evaluations for vision models:

    
  * ImageNet-A <cit.> consists of naturally occurring images that are challenging for vision models to classify correctly. 
    
  * Projected Gradient Descent (PGD)  <cit.> on ImageNet is an iterative attack that creates adversarial examples by adding small, carefully crafted perturbations to input images with some attack budget (we use ε=8/255).


Dubious Arguments For and Against Researching “Adversarial Robustness”
In this section we will again raise many common distinctions and arguments for and against adversarial robustness and see that they are not particularly helpful for deciding whether adversarial robustness is a useful area of AI safety.


We should work on adversarial robustness because:

1: Corner Case vs. Average Case Distinction. Adversarial robustness focuses on corner case performance, not average case performance. As follows are two analogies for this intuition. First, humans are highly vulnerable to toxins and poisons. Being more robust to toxins does not make a person more markedly generally intelligent. Second, computer programs are susceptible to fuzzing attacks; improving a program's security to fuzzing attacks does not make computer programs generally more quick, usable, scalable, maintainable, and so on.

2: Adversarial Robustness Is A Persistent Problem. 
Optical illusions in humans show that even highly evolved intelligent systems have vulnerabilities, indicating that intelligence alone doesn't guarantee adversarial robustness. Increases in intelligence do not make adversarial robustness easier due to the red queen's hypothesis: as defenders become more powerful, so to do attackers who can discover more vulnerabilities.

3: Proxies Need to be Robust to Optimization Pressure. In the future, agents may optimize and may be guided by neural network proxies, such as by networks that model human values. Proxies instantiated by neural networks—networks that assign scores to agent actions—will need to be robust to optimizing agents. If the models are not robust, then agents may be guided in a wrong direction, not pursuing what we want <cit.>.

We should not work on adversarial robustness because:

1: Robustness Is Upstream General Capabilities. Autonomous vehicles are not widely deployed because they are not sufficiently robust; therefore, improving robustness would improve their general capabilities. Indeed, improving a model's adversarial robustness implies better representations and implies it can generalize to more challenging scenarios—improved generalization is the essence of intelligence.

2: Superintelligent AIs Won't Get Trivial Adversarial Examples Wrong. Intuitively, a superintelligence would not be fooled by simple ℓ_p adversarial perturbations, or else it is not a true superintelligence. Therefore adversarial robustness will be automatically solved by scaling and making AIs more intelligent.

3: Malicious Use Is A Distraction. Adversarial robustness is about preventing malicious actors from exploiting vulnerabilities in AI systems, but that is a distraction because 
“once we reach AGI the outcome is the same no matter which group creates it: we all die. Nobody is able to cause a good outcome if given an AGI now because they don't know how to control it... without killing everyone. There's little point to worrying about bad actors, because they're incapable of causing an outcome any worse than the `good guys”' <cit.>.





  
Empirical analysis of safetywashing. We now analyze whether these benchmarks measure novel properties or are highly correlated with general capabilities. Can adversarial robustness be an instrument for safetywashing?



We find it depends on the benchmark. Traditional benchmarks, particularly those focused on text manipulation and perturbation as well as adversarial examples, show high correlation with general capabilities in current vision and language models. There is an analogue between ANLI in the language domain and ImageNet-A in the vision domain. While these benchmarks may have captured distinct properties in earlier models, they now appear to be largely indistinguishable from overall model performance.
In contrast, we observe low capabilities correlations across all categories of jailbreaking and gradient-based benchmarks, across vision and language models. Notably, there is similarly a direct analogue between GCG in the language domain and PGD in the vision domain.

This serves as an illustration for how relying solely on verbal arguments to determine whether a field measures distinct properties or merely reflects capabilities can be misleading; our empirical validation shows different results between “old school” adversarial robustness and jailbreaking benchmarks, despite similar verbal arguments for their distinctness from capabilities. This is why we need empirical science, rather than word games.


We find that some adversarial robustness benchmarks may be prone to safetywashing, while others seem to measure a distinct phenomena other than upstream model capabilities.





 §.§ Weaponization Capabilities





  
Area Overview.
We borrow the definitions of weaponization capabilities from recent U.S. federal executive action <cit.> and state legislation <cit.>. These documents cite security risks that may be easier to cause with a powerful AI system—which include the creation or use of chemical, biological, radiological, or nuclear weapons, as well as cyberattacks on critical infrastructure.



  
Datasets. Benchmarks in this area aim to quantify the extent that weaponization capabilities exist in models, and thereby the effectiveness of capabilities suppression techniques such as unlearning, circuit breaking, and refusal. We use the Weapons of Mass Destruction Proxy (WMDP) benchmark <cit.>, where a higher accuracy on biosecurity, chemical security, and cybersecurity knowledge leads to a lower score. 






  
Dubious intuitive arguments for and against researching “Weaponization.” We skip the intuitive arguments for this section because this topic is about restricting specific capabilities, so it obviously has a negative correlation with capabilities.

r0.45

 

Weaponization
Capabilities Evaluation     Capabilities
Correlation 

Biosecurity     -87.5% 

Chemical Security     -81.1% 

Cybersecurity     -86.0% 


We find that WMDP scores are highly anticorrelated with capabilities.





  
Empirical analysis of safetywashing. These results indicate that as models become more capable overall, their potential for weaponization increases significantly. The strong negative correlations across all three fields suggest that more advanced AI systems are more likely to possess knowledge that could be misused for harmful purposes.
The inverted scoring system of WMDP benchmarks clearly illustrates its purpose in guiding model development: higher scores indicate more effective suppression of specific harmful capabilities. As capabilities advance, safety researchers can focus on mitigating risks associated with weaponization.






Generally, we find that weaponization capabilities benchmarks are not prone to safetywashing.



§ DISCUSSION







  
Benchmarks as incentive-setting. There are a variety of properties AI systems should satisfy, such as detailed domain knowledge, reasoning, lack of bias, ethical understanding, truthfulness, calibration, and more. Benchmarks operationalize these properties, ultimately structuring the efforts and incentives of the research community.
Creating a benchmark has two major purposes: it acts as a implicit competition for model development (by providing a measure to by which to judge models as “better” or “worse”), and it provides diagnostic information about how capable models are at a task, which can guide policy.
Commonly used benchmarks ultimately impact how research effort, as well as funding and resources, is allocated. 

Because of this, significant effort has gone into conceptualizing and benchmarking the “safety” of AI systems, in the hope of reducing present and anticipated future risks from AI systems. To investigate this, we conduct the most extensive meta-analysis of safety benchmarks to date. We do not cover transparency, anomaly detection, trojans, and other safety areas that do not have well-established preexisting benchmarks.



  
Empirically measuring capabilities correlations is necessary. Benchmark scores can be increased on many “safety” datasets, such as ETHICS <cit.>, TruthfulQA <cit.>, GPQA <cit.>, QuALITY <cit.>, MT-Bench <cit.>, LMSYS Chatbot ARENA <cit.>, ANLI <cit.>, AdvGLUE <cit.>, and AdvGLUE++ <cit.>, simply by increasing the capabilities of the model.  
This raises questions about whether safety benchmarks are setting the right incentives or can be misused for safetywashing. 
In some cases, safety-related areas may act a jangle for capabilities; jangle fallacy is the erroneous belief that two constructs are different because they have the different names, when in practice they measure the same latent factor.

Ultimately, we have seen that intuitive arguments are a poor predictor of empirical correlations. For example, in alignment theory, there is a tendency to theorize about what would be instrumentally useful for safety without adequately considering the need to improve the balance of safety and capabilities.
This can lead to the promotion of capabilities research that happens to improve some safety benchmark scores (“safety via capabilities”), but in reality do not reduce overall risk.

We are not claiming that all philosophy related to alignment is counterproductive. Speculation about AI risks can often be useful for horizon-scanning and identifying potential failure modes (e.g., corrigibility <cit.>). Rather, we argue it is counterproductive to use abstract top-down verbal arguments with multiple deductive steps to make claims about deep learning phenomena and their relation to safety, such as “we don't need to worry about adversarial robustness being difficult because  ⟨intuitive arguments⟩.”


 




  
Normalized and controlled benchmarks are often insufficient.
In some cases, researchers may hope to prevent safetywashing by establishing norms for how a benchmark should be used. For example, one norm is to simply hold a model constant when evaluating safety methods, or to use metrics that control for general capabilities. However, norms of this type have historically been weak, as they are easily ignored or overridden in followup work. For example, safety metrics that controlled for capabilities were proposed in early corruption robustness research <cit.>, yet followup work drifted away from these metrics and toward evaluations with higher capabilities correlations, enabling safetywashing <cit.>.
In some areas, such as OOD detection, norms such as holding the model constant are common <cit.>. However, if a safety metric is highly correlated with capabilities and becomes very popular, there will be strong pressure to break norms and improve the safety metric by simply improving capabilities.
We should instead use benchmarks that implicitly control for capabilities in their design, analogous to RMS calibration error essentially being Brier score without the refinement term.






  
Three generating processes behind safetywashing.


Safety by association: Research released by safety teams or famous safety researchers is often labeled as safety-relevant by default. Even if the work is one reframing and a new author list away from being perceived as a standard capabilities paper, the work is often “godfathered” in as a safety paper. The determination of whether an area is safety-relevant is often sociological rather than scientific.

Public relations: Corporate entities often engage in safetywashing for the sake of appearances, portraying capabilities advancements in terms of safety progress by reporting correlated safety metrics to project an image of responsible AI development. This behavior is particularly pronounced when there is significant public pressure or regulatory scrutiny.

Optimizing grant applications: Similarly, researchers may be incentivized to frame their work in terms of safety to appeal to grantmakers, even when the underlying advances are predominantly in capabilities. This misalignment of incentives can lead to a proliferation of research that claims to address safety concerns but fails to make substantive differential safety progress.















r0.5



    < g r a p h i c s >

Capabilities score is highly correlated with amount of compute used. Training FLOP approximated by 6 ×params×train_tokens as per <cit.>. For base models, we find a similarly strong Spearman correlation of 96.5%.







  
The bitter lesson for AI safety research. In Figure <ref>, we find that the total compute used in model training highly influences the model's capabilities score. Given this context, how should the AI safety community allocate its efforts to differentially improve model safety? We can derive some insight from the “Bitter Lesson” <cit.>, which observes that as compute becomes exponentially more available over time, AI research methodologies which optimize performance at a constant level of compute are subsumed by new paradigms that effectively leverage greater compute. Therefore, rather than fixating on the strengths and weaknesses of present-day models, effective safety research should anticipate and address the flaws that will emerge or remain in future generations of models and deemphasize issues likely to be resolved through model scaling.


First, if a safety benchmark (which meaningfully captures a desired safety property) is highly correlated with general capabilities, the safety property will likely be improved in more capable models, even if current models struggle. Safety researchers may therefore be better served by redirecting their efforts to other problems that will persist when scaling the current mainstream class of models.
Second, success in new safety techniques should be judged not only by direct improvements in safety benchmark scores, but also by the extent to which the techniques entangle safety benchmark performance with scale. 
For example, if a widely-adopted robustness technique that entangles adversarial robustness performance with capabilities, further research efforts can be reallocated elsewhere. 

“Safety properties” in AI are not static concepts, but rather a set of desiderata that changes over time. The selection of AI safety research problems (and, implicitly, allocation of resources between problems) should anticipate how the risk profiles of models will change as capabilities improve, with some problems going away while others worsen or emerge with scale. By directing research effort toward properties and methods that specifically enhance safety independently of scale, safety researchers can make more effective use of their resources and significantly contribute to the development of safer AI systems. 



  
Increasing capabilities does not improve safety. The default assumption is that capabilities advancements are a “rising tide that lifts all boats,” improving model properties including safety properties. However, this does not necessarily mean that overall risk decreases. While improving upstream capabilities can improve properties such as truthfulness, it also increases the risk of weaponization and catastrophic malicious use (<Ref>). Hence AI does not necessarily become safer as it becomes more capable.





  
Recommendations. We summarize our recommendations as follows:


    
  * Report capabilities correlations: New safety evaluations should empirically report their capabilities correlation. 
    
  * Design decorrelated benchmarks: Well-designed safety benchmarks have the opportunity to incentivize differential safety progress by finding safety properties that are decorrelated from capabilities. 
    
  * Avoid safetywashing: Model developers should avoid making claims about improved safety unless they have made differential progress. They also should not misappropriate safety benchmarks that are highly correlated with capabilities. This means that as new training techniques (e.g., base, chat fine-tuning, refusal training, adversarial training, circuit breakers) are integrated into new models, the relevance and adequacy of existing safety benchmarks—and their entanglement with capabilities benchmarks—should also be regularly reassessed. 






§ CONCLUSION


Our analysis reveals that many AI safety benchmarks—around half—often inadvertently capture a latent factor closely tied to general capabilities, opening the door to safetywashing.
We find that model performance on capabilities benchmarks, distilled into a “capabilities score,” also has a remarkably high association with the amount of training compute. 
AI safety subfields such as alignment, scalable oversight, truthfulness, and static adversarial robustness are highly correlated with upstream general capabilities; areas such as bias, dynamic adversarial robustness, and calibration have relatively low correlations; measurements of sycophancy and weaponization risk have significant negative correlations with general capabilities.
Overall, it is hard to avoid measuring upstream model capabilities in AI safety benchmarks. We also conclude that alignment theory, which has heavily influenced AI safety priorities, is a counterproductive paradigm for guiding ML safety research. Science through empirical measurement should take its place.



















§ ACKNOWLEDGEMENTS


We thank Owain Evans, Noa Nabeshima, and Arunim Agarwal for providing feedback on drafts of this paper. Because this paper can also serve as an introduction to AI safety, we also sent a preview to the AI Safety, Ethics, & Society course (run by the Center for AI Safety), where we additionally thank John Teichman, Sawyer Bernath, Zamshed Harun, Luis Fernandez, Kanad Chakrabarti, and Cody Rushing for their feedback.



§ CONTRIBUTIONS


Richard and Steven led the technical implementation of this project, overseeing tasks and contributing the majority of the engineering work. Adam, Alice, Long, and Xuwang contributed as research engineers, supporting the project's development. Richard, Adam, and Mantas did the writing for the paper, while Dan wrote the “Arguments For and Against” sections. Gabe, Alex, and Stephen provided general advising to the project, while Ryan assisted with a number of processes during analysis and writing. Dan contributed the idea for the paper, suggested experiments, and made all major decisions related to the paper’s framing and presentation.







unsrtnat





§ APPENDIX




 §.§ List of Models





  §.§.§ List of Language Models

The following list are all of the chat models we used for our evaluations. The model names below are as one would find them on Huggingface.

2
  
    
  * gemma-1.1-2B-it <cit.>
    
  * gemma-1.1-7B-it <cit.>
    
  * Llama-2-7B-Chat <cit.> 
    
  * Llama-2-13B-Chat <cit.>
    
  * Llama-2-70B-Chat <cit.>
    
  * Llama-3-8B-Instruct <cit.>
    
  * Llama-3-70B-Instruct <cit.>
    
  * Mistral-7B-Instruct-v0.2 <cit.>
    
  * Mixtral-8x7B-Instruct-v0.1 <cit.>
    
  * Mixtral-8x22B-Instruct-v0.1 <cit.>
    
  * falcon-7B-Instruct <cit.>
    
  * falcon-40B-Instruct <cit.>
    
  * falcon-180B-Chat <cit.>
    
  * Yi-6B-Chat <cit.>
    
  * Yi-34B-Chat <cit.>
    
  * Qwen1.5-0.5B-Chat <cit.>
    
  * Qwen1.5-1.8B-Chat <cit.>
    
  * Qwen1.5-4B-Chat <cit.>
    
  * Qwen1.5-7B-Chat <cit.>
    
  * Qwen1.5-14B-Chat <cit.>
    
  * Qwen1.5-32B-Chat <cit.>
    
  * Qwen1.5-72B-Chat <cit.>
    
  * Qwen1.5-110B-Chat <cit.>
    
  * deepseek-llm-7B-Chat <cit.>
    
  * deepseek-llm-67B-Chat <cit.>
    
  * dbrx-instruct <cit.>
  



The following list are all of the base models we used for our evaluations.

2
  
    
  * gemma-2B <cit.>
    
  * gemma-7B <cit.>
    
  * Llama-2-7B <cit.>
    
  * Llama-2-13B <cit.>
    
  * Llama-2-70B <cit.>
    
  * Llama-3-8B <cit.>
    
  * Llama-3-70B <cit.>
    
  * Mistral-7B-v0.1 <cit.>
    
  * Mixtral-8x7B-v0.1 <cit.>
    
  * Mixtral-8x22B-v0.1 <cit.>
    
  * falcon-7B <cit.> 
    < g r a p h i c s >
 
    
  * falcon-40B <cit.>
    
  * falcon-180B <cit.>
    
  * Yi-6B <cit.>
    
  * Yi-9B <cit.>
    
  * Yi-34B <cit.>
    
  * Qwen1.5-0.5B <cit.>
    
  * Qwen1.5-1.8B <cit.>
    
  * Qwen1.5-4B <cit.>
    
  * Qwen1.5-7B <cit.>
    
  * Qwen1.5-14B <cit.>
    
  * Qwen1.5-32B <cit.>
    
  * Qwen1.5-72B <cit.>
    
  * Qwen1.5-110B <cit.>
    
  * deepseek-llm-7B-base <cit.>
    
  * deepseek-llm-67B-base <cit.>
    
  * dbrx-base <cit.>
  




  §.§.§ List of Vision Models


The following is the list of vision models used in our calibration evaluations.

2


  * DINOv2 ViT-B/14 <cit.>

  * DINOv2 ViT-L/14 <cit.>

  * ResNet50 + Mixup <cit.>

  * ResNet50 + CutMix <cit.>

  * ResNet50 + AugMix <cit.>

  * ResNet50 + DeepAugment <cit.>

  * ConvNeXt-Tiny <cit.>

  * ConvNeXt-Small <cit.>

  * ConvNeXt-Base <cit.>

  * ConvNeXt-Large <cit.>

  * ConvNeXtV2-Atto <cit.>

  * ConvNeXtV2-Femto <cit.>

  * ConvNeXtV2-Pico <cit.>

  * ConvNeXtV2-Nano <cit.>

  * ConvNeXtV2-Tiny <cit.>

  * ConvNeXtV2-Base <cit.>

  * ConvNeXtV2-Large <cit.>

  * ConvNeXtV2-Huge <cit.>

  * AlexNet <cit.>

  * VGG16 <cit.>

  * ResNet18 <cit.>

  * ResNet50 <cit.>

  * Wide ResNet-50-2 <cit.>

  * ResNeXt-50 32x4d <cit.>

  * DenseNet121 <cit.>

  * Swin-Base <cit.>

  * ViT-Base/16 <cit.>




The following is the list of standard vision models used in our evaluations for natural adversarial examples (ImageNet-A).

2


  * ResNet50 <cit.>

  * WideResNet-50-2 <cit.>

  * WideResNet-50-4 <cit.>

  * ResNet18 <cit.>

  * ResNeXt-50 32x4d <cit.>

  * DenseNet <cit.>

  * ShuffleNet <cit.>

  * VGG16BN <cit.>

  * MnasNet <cit.>

  * MobileNet <cit.>

  * DINOv2 ViT-large Patch14 <cit.>

  * ConvNeXt-V2-large ImageNet1K+22K <cit.>

  * DINOv2 ViT-base Patch14 <cit.>

  * Swin-large ImageNet1K <cit.>

  * ConvNeXt-V2-huge ImageNet1K <cit.>

  * Swin-base ImageNet1K <cit.>

  * ConvNeXt-V2-base ImageNet1K+22K <cit.>

  * ConvNeXt-xlarge ImageNet1K+22K <cit.>

  * ConvNeXt-V2-large ImageNet1K <cit.>

  * MAE ViT-large Patch16 <cit.>

  * ConvNeXt-large ImageNet1K+22K <cit.>

  * ViT-base Patch8 ImageNet1K+22K <cit.>

  * ConvNeXt-base ImageNet1K+22K <cit.>

  * ConvNeXt-V2-base ImageNet1K <cit.>

  * ViT-large Patch16 ImageNet1K+22K <cit.>

  * ConvNeXt-large ImageNet1K <cit.>

  * Swin-small ImageNet1K <cit.>

  * ConvNeXt-V2-tiny ImageNet1K+22K <cit.>

  * ConvNeXt-small ImageNet1K+22K <cit.>

  * ConvNeXt-base ImageNet1K <cit.>

  * CLIP (ViT-L/14) <cit.>

  * ConvNeXt-V2-tiny ImageNet1K <cit.>

  * MAE ViT-base Patch16 <cit.>

  * ConvNeXt-small ImageNet1K <cit.>

  * ConvNeXt-V2-nano ImageNet1K <cit.>

  * ConvNeXt-V2-nano ImageNet1K+22K <cit.>

  * Reversible-ViT-base multiscale <cit.>

  * ViT-base Patch16 ImageNet1K+22K <cit.>

  * ConvNeXt-tiny ImageNet1K+22K <cit.>

  * ConvNeXt-tiny ImageNet1K <cit.>

  * Swin-tiny ImageNet1K <cit.>

  * ResNet50 + PixMix <cit.>

  * ResNet50 + Moex <cit.>

  * Reversible-ViT-base <cit.>

  * ResNet50 + CutMix <cit.>

  * Reversible-ViT-small <cit.>

  * ConvNeXt-V2-pico ImageNet1K <cit.>

  * ConvNeXt-V2-atto ImageNet1K <cit.>

  * ResNet50 + DeepAug+AugMix <cit.>

  * ResNet50 + Mixup <cit.>

  * ResNet50 + Deepaugment <cit.>

  * ViT-base Patch16 ImageNet1K <cit.>

  * ConvNeXt-V2-femto ImageNet1K <cit.>

  * ViT-small Patch16 ImageNet1K <cit.>

  * ViT-small Patch16 ImageNet1K+22K <cit.>

  * ViT-base Patch32 ImageNet1K+22K <cit.>

  * ViT-base Patch32 ImageNet1K <cit.>

  * ResNet50 + AugMix <cit.>

  * ResNet50 + Stylised ImageNet <cit.>

  * ResNet50 + ANT <cit.>

  * ResNet50 + RandAug <cit.>

  * ViT-small Patch32 ImageNet1K+22K <cit.>

  * ViT-tiny Patch16 ImageNet1K+22K <cit.>



The following is the list of adversarially trained vision models used in our evaluations for gradient-based adversarial robustness (PGD 8/255).

2


  * ResNet50 <cit.>

  * ResNet50 + L_2 0.01 <cit.>

  * ResNet50 + L_2 0.03 <cit.>

  * ResNet50 + L_2 0.05 <cit.>

  * ResNet50 + L_2 0.1 <cit.>

  * ResNet50 + L_2 0.25 <cit.>

  * ResNet50 + L_2 0.5 <cit.>

  * ResNet50 + L_2 1 <cit.>

  * ResNet50 + L_2 3 <cit.>

  * ResNet50 + L_2 5 <cit.>

  * ResNet50 + L_∞ 0.5/255 <cit.>

  * ResNet50 + L_∞ 1.0/255 <cit.>

  * ResNet50 + L_∞ 2.0/255 <cit.>

  * ResNet50 + L_∞ 4.0/255 <cit.>

  * ResNet50 + L_∞ 8.0/255 <cit.>

  * WideResNet-50-2 + L_2 0.01 <cit.>

  * WideResNet-50-2 + L_2 0.03 <cit.>

  * WideResNet-50-2 + L_2 0.05 <cit.>

  * WideResNet-50-2 + L_2 0.1 <cit.>

  * WideResNet-50-2 + L_2 0.25 <cit.>

  * WideResNet-50-2 + L_2 0.5 <cit.>

  * WideResNet-50-2 + L_2 1 <cit.>

  * WideResNet-50-2 + L_2 3 <cit.>

  * WideResNet-50-2 + L_2 5 <cit.>

  * WideResNet-50-2 + L_∞ 0.5/255 <cit.>

  * WideResNet-50-2 + L_∞ 1.0/255 <cit.>

  * WideResNet-50-2 + L_∞ 2.0/255 <cit.>

  * WideResNet-50-2 + L_∞ 4.0/255 <cit.>

  * WideResNet-50-2 + L_∞ 8.0/255 <cit.>

  * WideResNet-50-4 + L_2 0.01 <cit.>

  * WideResNet-50-4 + L_2 0.03 <cit.>

  * WideResNet-50-4 + L_2 0.05 <cit.>

  * WideResNet-50-4 + L_2 0.1 <cit.>

  * WideResNet-50-4 + L_2 0.25 <cit.>

  * WideResNet-50-4 + L_2 0.5 <cit.>

  * WideResNet-50-4 + L_2 1 <cit.>

  * WideResNet-50-4 + L_2 3 <cit.>

  * WideResNet-50-4 + L_2 5 <cit.>

  * ResNet18 + L_2 0.01 <cit.>

  * ResNet18 + L_2 0.03 <cit.>

  * ResNet18 + L_2 0.05 <cit.>

  * ResNet18 + L_2 0.1 <cit.>

  * ResNet18 + L_2 0.25 <cit.>

  * ResNet18 + L_2 0.5 <cit.>

  * ResNet18 + L_2 1 <cit.>

  * ResNet18 + L_2 3 <cit.>

  * ResNet18 + L_2 5 <cit.>

  * ResNet18 + L_∞ 0.5/255 <cit.>

  * ResNet18 + L_∞ 1.0/255 <cit.>

  * ResNet18 + L_∞ 2.0/255 <cit.>

  * ResNet18 + L_∞ 4.0/255 <cit.>

  * ResNet18 + L_∞ 8.0/255 <cit.>

  * ResNeXt-50 32x4d + L_2 3 <cit.>

  * DenseNet + L_2 3 <cit.>

  * ShuffleNet + L_2 3 <cit.>

  * VGG16BN + L_2 3 <cit.>

  * MnasNet + L_2 3 <cit.>

  * MobileNet + L_2 3 <cit.>

  * ViT-base Patch16 + L_∞ 4/255 <cit.>

  * ConvNeXt-base + L_∞ 4/255 <cit.>

  * ViT-small Patch16 + L_∞ 4/255 <cit.>

  * Swin-base ImageNet1K + L_∞ 4/255 <cit.>

  * ConvNeXt-small + L_∞ 4/255 <cit.>

  * Swin-small ImageNet1K + L_∞ 4/255 <cit.>






 §.§ Capabilities Scores


Separate runs for capabilities scores were conducted between base and chat language models. Table <ref> presents these scores, highlighting the performance differences across various capabilities benchmarks. These scores provide a relative metric, and act as a comparative measure between models rather than an absolute one.












 §.§ Capabilities Evaluations: Correlation Matrices


As an intermediate step in our implementation of PCA, we compute Spearman correlation matrices for both base and chat language models. We show these matrices in Figure <ref> to show the correlation of various capabilities tasks across model types.

Both base and chat models show strong correlations between many benchmark pairs, indicating that performance on one task often predicts performance on others. Furthermore, some benchmarks form clusters with higher inter-correlations, suggesting they may be measuring identical capabilities. Other benchmarks, such as MATH and LAMBADA, show lower correlations with other tasks, potentially indicating they measure more distinct capabilities. Furthermore, noticeable differences in correlation patterns between base and chat models highlight the impact of fine-tuning on the relationships between various capabilities.









 §.§ Strict Instruction Following




  
Area Overview. Strict instruction following evaluations benchmark how well AI models can strictly adhere to specific instructions or rules, which may ensure AI systems behave as intended and follow safety guidelines.



  
Datasets. We evaluate strict instruction following using the following datasets:


    
  * IFEval focuses on verifiable instructions, such as word count requirements or keyword usage, across approximately 500 prompts.
    
  * RuLES evaluates models on 14 text-based scenarios, each with specific rules to follow. It includes a Basic test suite for straightforward rule adherence, a Benign suite to test if rules are violated in response to unrelated prompts, and a Red Team suite for adversarial scenarios (e.g. user presents a misleading reinterpretation of the rule, or disguises a request for the model to break the rule).


r0.45



Strict Instruction
Following Evaluation     Capabilities
Correlation 

IFEval     57.8% 

RuLES Basic     41.6% 

RuLES Benign     23.5% 

RuLES Red Team     16.1% 


Generally, strict instruction following exhibits a moderate correlation with capabilities for easier scenarios, and is not correlated for harder scenarios.





Empirical analysis of safetywashing. Is strict instruction following mostly determined by upstream model capabilities?

We find a moderate positive correlation between upstream model capabilities and performance on IFEval (57.8%) and RuLES Basic (41.6%). The RuLES Benign and Red Team suites exhibit even weaker correlations, indicating the ability of models to consistently follow strict instructions—especially in challenging or potentially problematic contexts—does not seem to improve with capabilities.

The higher correlation with IFEval and RuLES Basic suggests that evaluations using simple, easy-to-follow instructions may be more prone to safetywashing. However, strict instruction following, when combined with red-team prompts or obfiscation, does not seem to be correlated with upstream model capabilities.









 

            
    










 §.§ Hallucinations and Misconceptions: Expanded




  
Area overview. Factual reliability encompasses the ability of AI systems to generate and process accurate, truthful, and reliable content. This includes avoiding hallucinations, misconceptions, and generation of content that is plausible-sounding but factually incorrect or nonsensical. We further test correlations by measuring the capabilities coefficient on a generative benchmark (TruthfulQA Generation) and a discriminative benchmark (HaluEval).



  
Datasets. We describe the datasets we use below.

    
  * The TruthfulQA Generation task <cit.> aims to assess models' ability to generate truthful statements and avoid misconceptions. Unlike the MC1 task evaluated in the main paper, the generation task evaluates a 1-2 sentence generation from a model. We use a GPT-4o judge to judge model outputs. The truthful score indicates the truthfulness of the answer, while the truthful*informative score normalizes by the percentage of the model's answers that are informative (e.g. "I have no comment" would not be informative). 
    
  * HaluEval <cit.> is a benchmark designed to evaluate large language models' ability to recognize hallucinations. It aims to assess the model's ability to judge model responses in hypothetical QA, dialogue, and text summarization contexts. Hallucinated examples are generated from HotpotQA, OpenDialKG, and CNN/DailyMail.


R0.45
    

    
    2*Misinformation
Evaluations     2cCapabilities 

        2cCorrelation 

    2-3
        Base     Chat 

    TruthfulQA Generation         

     Truthful*Information Score     49.6%     72.9% 

     Truthful Score     74.7%     32.8% 

    HaluEval         

    HaluEval All     71.6%     56.7% 

    HaluEval Summarization      53.5%     34.2%

    HaluEval Dialogue      69.2%     89.1% 

    HaluEval QA     46.5%     18.4% 

     
    Hallucinations and misconceptions are generally prone to safetywashing.
    






  
Empirical analysis of safetywashing. Is factual reliability and hallucination reduction mostly upstream of latent general capabilities?

Our analysis shows that as models' general capabilities improve, the tendency for hallucinations decreases across the board for both base and chat models, illustrating that these topics are generally prone to safetywashing. This correlation is observed in both generative and classification tasks across both base and chat models. This has also been confirmed by previous work which has observed that larger language models tend to exhibit reduced hallucinations <cit.>.




            









 §.§ Capabilities Correlations for All Tested Evaluations



This section presents a comprehensive overview of the capabilities correlations for both base and chat models across all tested evaluations. We used lm-eval-harness for implementing most evaluations. For instance, the TruthfulQA MC1 implementation in lm-eval-harness employs a few-shot prompt with generic Q/A questions, avoiding misleading ones.





  §.§.§ Capabilities


llcc
Spearman correlations of capabilities component evaluations with the capabilities score, reported as a percentage. As expected, across base and chat models, the capabilities correlations of the component capabilities evaluations is high. 

Name     Metric     Base
Correlations (%)     Chat
Correlations (%) 

[]Capabilities Correlations for Capabilities Component (continued) 

Name     Metric     Base
Correlations (%)     Chat
Correlations (%) 

4rContinued on next page 

LogiQA <cit.>     Accuracy     86.0     86.1 

PIQA <cit.>     Accuracy     89.6     88.9 

Hellaswag <cit.>     Accuracy     93.7     84.9 

Winogrande <cit.>     Accuracy     92.1     88.7 

COPA <cit.>     Accuracy     81.4     87.8 

MedQA <cit.>     Accuracy     94.7     89.7 

ARC Challenge <cit.>     Accuracy     95.5     83.8 

MMLU <cit.>     Accuracy     93.1     83.2 

MATH <cit.>     Equivalence     84.4     61.5 

LAMBADA <cit.>     Accuracy     72.8     80.4 

GSM8K <cit.>     Exact Match     86.6     80.7 

BBH <cit.>     Exact Match     81.4     80.2 





  §.§.§ Alignment and Scalable Oversight


The following subsections present the Spearman correlations of safety evaluations with capabilities scores (“capabilities correlations”). For the LMSYS Chatbot Arena evaluation, we excluded chat models that were not available.

lcc
Correlations between alignment and oversight benchmarks and the capabilities score across models, reported as percentages. 

Name     Base
Correlations (%)     Chat
Correlations (%) 

[]Capabilities Correlations for Alignment and Scalable Oversight (continued) 

Name     Base
Correlations (%)     Chat
Correlations (%) 

3rContinued on next page 

MT Bench <cit.>     64.2     78.7 

AlpacaEval <cit.>     -     59.9 

LMSYS Chatbot Arena <cit.>     -     62.1 

GPQA <cit.>     80.4     77.7 

QuALITY <cit.>     90.5     88.8 





  §.§.§ Machine Ethics


Note that we obtained a subset of the SafetyBench questions directly from the authors, as the dataset was not publicly released. 

llccc
Capabilities correlations for bias datasets. We also found that MoralChoice was saturated, with most models getting around 100%. MACHIAVELLI had a low slope, with the score not changing much across models. Note that to ensure that higher correlations meant safer models, we negate the MACHIAVELLI score, and select the non-sycophantic answer for Sycophancy. 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

[]Capabilities Correlations for Machine Ethics (continued) 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

4rContinued on next page 

MoralChoice <cit.>     -     25.8     46.7 

SafetyBench <cit.>     -     65.1     71.7 

Model Written Evals <cit.>     Sycophancy     -65.6     -66.8 

3*MACHIAVELLI <cit.>     Power     -54.3     -46.1 

    Utility     -48.3     -49.9 

    Violations     8.3     -52.9 

6*ETHICS <cit.>     All     70.3     82.2 

    Commonsense     59.6     69.3 

    Deontology     45.9     38.8 

    Justice     68.2     50.9 

    Utilitarianism     56.6     75.0 

    Virtue     55.9     73.5 





  §.§.§ Bias and Toxicity

llccc
Capabilities correlations for bias datasets. Correlations reported as percentages. The Advanced AI Risk score was negated so that a higher score meant less risky. 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

[]Capabilities Correlations for Bias (continued) 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

4rContinued on next page 

Winogender <cit.>     -     85.9     75.6 

Crows Pairs English <cit.>     -     -31.6     28.5 

Simple Cooccurrence Bias <cit.>     -     -12.3     -37.3 

Toxigen <cit.>     -     56.0     30.7 

Advanced AI Risk <cit.>     -     -60.6     -42.6 

2*BBQ <cit.>     Ambiguous     30.8     -37.3 

    Disambiguated     83.6     76.8 

7*[l]Discrim-Eval
(Explicit) <cit.>     Maximum Difference     14.1     33.2 

    Hispanic-White     1.4     13.6 

    Black-White     27.8     27.9 

    Female-Male     9.6     17.1 

    Non-Binary-Male     13.1     34.2 

    Younger than 60 - Age 60     14.6     -43.2 

    Older than 60 - Age 60     -52.9     -30.2 





  §.§.§ Misconceptions and Hallucinations

llccc
Capabilities correlations for misconceptions and hallucinations datasets. 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

[]Capabilities Correlations for Misconceptions (continued) 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

4rContinued on next page 

4*TruthfulQA <cit.>     MC1     69.7     81.2 

    Gen: Truth Score     74.7     32.8 

    Gen: Info Score     -49.1     23.1 

    Gen: Truth*Info Score     49.6     72.9 

4*HaluEval <cit.>     All     71.6     56.7 

    QA     46.5     18.4 

    Summarization     53.5     34.2 

    Dialogue     69.2     89.2 





  §.§.§ Calibration

llcc
Accuracy correlations for calibration metrics. Correlations reported as a percent. The capabilities correlation was not used, but rather correlation with the dataset accuracy (e.g., MMLU) across models. To ensure a positive correlation meant safer models, the score we used for calculating correlations is 1 -Brier Score for Brier Score entries and 1 -RMSCE for RMSCE entries.  

Metric     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

[]Correlations for Calibration (continued) 

Metric     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

4rContinued on next page 

3*Brier Score     MMLU     98.6     95.5 

    PIQA     98.1     99.2 

    MedQA     98.7     83.4 

3*[l]Brier Score
 Temperature Tuned     MMLU     98.5     99.9 

    PIQA     98.6     99.2 

    MedQA     99.7     96.7 

3*RMSCE     MMLU     2.5     20.1 

    PIQA     31.9     47.9 

    MedQA     41.2     38.6 

3*[l]RMSCE
 Temperature Tuned     MMLU     -35.9     -8.2 

    PIQA     -11.1     -7.0 

    MedQA     12.4     31.0 





  §.§.§ Adversarial Robustness




llccc
Base models' capabilities correlations for GLUE <cit.>, AdvGLUE <cit.>, and AdvGLUE++ <cit.>, reported as a percent. We find that neither AdvGLUE nor AdvGLUE significantly decorrelates performance on GLUE relative to the GLUE dataset. 

Evaluation     Dataset     GLUE
CC (%)     AdvGLUE
CC (%)     AdvGLUE++
CC (%) 

[]Base models' capabilities correlations for GLUE, AdvGLUE, and AdvGLUE++ (continued) 

Evaluation     Dataset     GLUE
CC (%)     AdvGLUE
CC (%)     AdvGLUE++
CC (%) 

5rContinued on next page 

6*GLUE
Split     MNLI Matched     67.3     66.0     62.8 

    MNLI Mismatched     66.9     68.2     61.8 

    QNLI     14.9     19.1     21.3 

    QQP     33.2     19.0     32.8 

    RTE     49.9     76.4     32.1 

    SST2     52.0     70.3     62.9 



llccc
Capabilities correlations for adversarial robustness datasets. Correlations reported as percentages. The metric used is attack failure rate for HarmBench splits. 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

[]Capabilities Correlations for Adversarial Robustness (continued) 

Evaluation     Dataset     Base
Correlations     Chat
Correlations 

4rContinued on next page 

ANLI <cit.>     -     74.5     81.5 

AdvDemonstration <cit.>     -     57.9     63.9 

7*HarmBench DirectRequest <cit.>     Biochemical     -58.0     -9.3 

    Cybercrime     -59.0     -19.5 

    Harassment     -46.6     -15.8 

    Harmful     -54.3     7.3 

    Illegal     -47.1     -9.8 

    Misinfo     -53.9     -38.7 

    All     -65.5     -18.2 

7*HarmBench HumanJailbreak <cit.>     Biochemical     -49.6     -22.1 

    Cybercrime     -73.8     -29.3 

    Harassment     -85.5     -34.1 

    Harmful     -74.6     -29.9 

    Illegal     -71.1     -28.5 

    Misinfo     -76.9     -41.6 

    All     -79.2     -31.4 

7*HarmBench TAP-T <cit.>     Biochemical     -62.0     -26.3 

    Cybercrime     -60.0     -33.0 

    Harassment     -77.1     -34.3 

    Harmful     -59.4     -22.3 

    Illegal     -68.9     -35.9 

    Misinfo     -74.5     -56.8 

    All     -78.7     -42.8 

7*HarmBench GCG-T <cit.>     Biochemical     -55.8     -14.1 

    Cybercrime     -74.9     -26.6 

    Harassment     -57.7     -31.0 

    Harmful     -48.2     -18.7 

    Illegal     -60.5     -15.4 

    Misinfo     -55.4     -35.5 

    All     -61.5     -28.4 





  §.§.§ Weaponization Capabilities

llccc
Capabilities correlations for weaponization capabilities datasets. The metric used for CybersecEval2 was vulnerability detection rate for Exploit, safe suggestion rate for Instruct, Safe response rate for MITRE, attack failure rate for prompt injection, and accuracy for FRR. WMDP uses a score that inverts the accuracy. 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

4rContinued on next page 

4*WMDP <cit.>     All     -90.6     -88.6 

    Biosecurity Split     -92.5     -87.5 

    Chemical Security Split     -90.8     -81.1 

    Cybersecurity Split     -88.4     -86.0 

5*CybersecEval2 <cit.>     Exploit     -37.5     -50.3 

    Instruct     -48.9     -85.8 

    MITRE     -19.6     40.4 

    Prompt Injection     -16.8     -18.6 

    FRR     -44.2     -24.9 






  §.§.§ Strict Instruction Following

llccc
Capabilities correlations for strict instruction following datasets. 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

[]Capabilities Correlations for Rule Following (continued) 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

4rContinued on next page 

IFEval <cit.>     -     16.6     57.8 

4*RuLES <cit.>     Basic     33.5     41.6 

    Benign     35.7     23.5 

    Red Team     0.0     16.1 

    All     34.4     26.5 






















 §.§ Closed Source Model Evaluations: GPT-4o Capabilities Score



In general, closed source models were excluded from our main paper analysis because of the technical challenges of calculating log probabilities and incompatibility with certain evaluation libraries. In this section, we compare the capabilities of GPT-4o with other open-source models using our analysis. To do so, we recompute all capabilities scores with all models including GPT-4o, while excluding BBH and LAMBADA from the capabilities score calculations. Table <ref> contains our calculated scores of GPT-4o on the capabilities tasks, while in Figure <ref> we observe clear gap between GPT-4o and the current open source models.












 













    


    









 Paper Title: Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms.


Quantum thermalization and Floquet engineering in a spin ensemble with a clock transition
    Andrei Faraon
    
=========================================================================================

§ ABSTRACT 
Structured State Space Models (SSMs) have emerged as compelling alternatives to Transformer architectures, offering linear-time complexity and superior performance in various sequence modeling tasks. Despite their advantages, SSMs like the original Mamba-2 face training difficulties due to the sensitivities introduced by the extended series of recurrent matrix multiplications. In this paper, we propose an advanced architecture that mitigates these challenges by decomposing A-multiplications into multiple groups and optimizing positional encoding through Grouped Finite Impulse Response (FIR) filtering. This new structure, denoted as Grouped FIR-enhanced SSM (GFSSM), employs semiseparable matrices for efficient computation. Furthermore, inspired by the "attention sink" phenomenon identified in streaming language models, we incorporate a similar mechanism to enhance the stability and performance of our model over extended sequences. Our approach further bridges the gap between SSMs and Transformer architectures, offering a viable path forward for scalable and high-performing sequence modeling.
    


§ INTRODUCTION



In recent years, the field of deep learning has been revolutionized by the introduction of the Transformer architecture and its attention mechanism <cit.>, which have become the backbone of most state-of-the-art models across various domains, including natural language processing (NLP), computer vision, and speech recognition. However, these architectures are not without limitations. Specifically, the quadratic time and memory complexity of the self-attention mechanism pose significant challenges when handling long sequences. To address these issues, several subquadratic-time architectures have been proposed, among which Structured State Space Models (SSMs) <cit.> have garnered considerable interest due to their linear-time complexity and promising performance. Mamba <cit.> and its successor Mamba-2 <cit.> represent significant strides in this direction, implementing SSMs that utilize selective state updates to efficiently process long sequences. Despite these advancements, Mamba-2 still encounters certain obstacles, primarily due to the sensitivity of the recurrent weight multiplications, which can complicate training. These sensitivities arise from a long sequence of multiplications, making the model potentially unstable and difficult to optimize effectively.

To alleviate these issues, we propose a novel architecture that reimagines the positional encoding mechanism within the SSM framework. Our innovation revolves around decomposing the recurrent multiplications into smaller, more manageable groups and employing Finite Impulse Response (FIR) filters <cit.> to enhance positional encoding. This modification not only improves the sensitivity issues but also leveraging semiseparable matrices, which can be calculated with reduced complexity. Additionally, inspired by recent insights from the domain of streaming language models, we integrate an attention sinks mechanism <cit.> within our architecture. Previous studies have highlighted the importance of retaining initial tokens as "sinks" that stabilize the attention mechanism over extended sequences. By incorporating a similar mechanism, our model benefits from improved stability and enhanced performance, particularly for tasks involving long-range dependencies.

We denote our new architecture as Grouped FIR-enhanced Structured State Space Model (GFSSM). GFSSM represents a dual-pronged improvement over traditional SSMs: first, by mitigating the training difficulties associated with long recurrent multiplications, and second, by incorporating attention sinks to handle extended sequences more effectively. In summary, our contributions can be outlined as follows:


    
  * We introduce the Grouped FIR filter to optimize the positional encoding within SSMs, thereby addressing the sensitivity and training difficulties associated with long sequences of multiplications.
    
  * We incorporate an attention sink mechanism to enhance model stability and performance over extended sequences, inspired by insights from streaming language models.


This paper is structured as follows: Section 2 provides a detailed overview of related work, including foundational models and their limitations. Section 3 presents the mathematical formulation and architectural details of GFSSM. Section 4 outlines our experimental setup and results, followed by an in-depth analysis. Finally, Section 5 concludes the paper and discusses potential avenues for future work.



§ RELATED WORK





 §.§ Structured State Space Models (SSMs)

Structured State Space Models (SSMs) <cit.> have attracted attention for their ability to offer linear-time complexity while maintaining competitive performance. Among these, the Mamba architecture <cit.> represents a significant advancement. Mamba leverages selective state updates to efficiently handle long sequences, making it a viable alternative to Transformers. However, despite its linear complexity, Mamba's performance has been limited by the sensitivity and difficulty in training due to extended sequences of recurrent multiplications. Mamba-2 <cit.> improved upon its predecessor by introducing a more structured approach, leveraging the State Space Duality (SSD) framework. By restricting the SSM parameters further and employing structured decompositions of semiseparable matrices, Mamba-2 achieved enhanced computational efficiency and performance. Nevertheless, the core challenge of training sensitivity due to recurrent dynamics remained an open problem.



 §.§ Finite Impulse Response (FIR) Filters

FIR filters <cit.> have been widely used in signal processing for their stability and straightforward implementation. Recent works have explored their potential in the context of machine learning, particularly for optimizing convolutional neural networks (CNNs) [6]. These filters can be tuned to emphasize specific frequencies, making them useful for various signal processing tasks. In the context of SSMs, deploying FIR filters can help mitigate sensitivity issues by effectively smoothing positional encodings.



 §.§ Attention Sinks

The concept of attention sinks, as explored by <cit.>, addresses some unique challenges in deploying large language models (LLMs) for streaming applications. By retaining key-value (KV) states of initial tokens, attention sinks stabilize the attention mechanism, thereby enabling LLMs to generalize to sequences much longer than those encountered during training. This insight is particularly relevant for our work, as it offers a potential solution to improve stability and performance over extended sequences in SSMs.


§ METHODOLOGY


This section outlines the theoretical foundations and architectural innovations of the Grouped FIR-enhanced Structured State Space Model (GFSSM). We begin by revisiting the fundamental principles and equations that underpin our approach, followed by detailed explanations of the grouped FIR filtering and the integration of attention sinks. Finally, we discuss the integration of these components into the overall GFSSM architecture.



 §.§ Fundamentals of State Space Duality (SSD)

In the Mamba-2 SSD framework, the forward pass is governed by the following equations:


    h_t   = A_t h_t-1 + B_t x_t
    
            y_t   = C_t^⊤ h_t,

where h_t denotes the hidden state at time t, x_t is the input, and y_t represents the output. A_t is a scalar times identity matrix, denoted as a_t, which simplifies the recurrent computation. B_t and C_t are matrices that define the state space dynamics, which can vary over time. By definition, h_0 = B_0 x_0. By induction, the h_t can be obtained as:


    h_t    = a_t a_t-1⋯ a_1 B_0 x_0 + a_t a_t-1⋯ a_2 B_1 x_1 + ⋯
          + a_t a_t-1 B_t-2 x_t-2 + a_t B_t-1 x_t-1 + B_t x_t 
       = ∑_s=0^t a_t:s^× B_s x_s,

where a_t:s^× denotes the product of a terms from s+1 to t. When multiplied by C_t to produce y_t and vectorized over t ∈ [𝚃], the matrix transformation form of SSD becomes:


    y_t    = ∑_s=0^t a_t:s^× C_t^⊤ B_s x_s, 
    
            y    = (L ∘ C^⊤B) x = Mx,

where L is represented as L_ji = a_j ⋯ a_i+1:


    L =
        [          1                                            ;        a_1          1                                 ;    a_2 a_1        a_2          1                      ;          ⋮          ⋮          ⋱          ⋱           ; a_T-1⋯ a_1 a_T-1⋯ a_2          ⋯      a_T-1          1;            ].


The potential limitation of the extended series of a_i multiplications arises when dealing with long text sequences. The matrix L_ji can decay to 0 or explode, resulting in numerical instability. Furthermore, any intermediate poorly conditioned a_i will have a prolonged adverse impact, propagating across subsequent tokens. For instance, if a_i is 0 at an initial position, all following states will remain 0. This extended multiplication series also complicates training, as noted in Mamba-2: higher precision for the main model parameters may be necessary due to the sensitivity of SSDs to recurrent dynamics. Parameter storage in fp32 can improve stability by mitigating these recurrent sensitivities.




 §.§ Grouped FIR Filtering

In GFSSM, we address the sensitivity and training difficulties inherent in SSMs by optimizing the computation of the hidden states and refining the positional encoding mechanism. The sensitivity in training SSMs arises primarily from the extended series of recurrent multiplications. To mitigate this issue, we decompose recurrent multiplications into smaller, more manageable groups and employ Finite Impulse Response (FIR) filters to enhance the positional encoding. 

For simplicity, we illustrate the method using four groups with a 4th-order FIR filter. However, this can be easily generalized to any Q groups with an n-th order FIR filter. The new recurrent form is:


    s_t   = k_0 B_t x_t + k_1 B_t-1 x_t-1
       + k_2 B_t-2 x_t-2 + k_3 B_t-3 x_t-3



    {
            h_t^i   = A_t h_t-1^i + s_t,    i = mod(t, Q) 
    
            h_t^i   = h_t-1^i,    i ≠mod(t, Q)
        .



    y_t = C_t ∑_i=0^Q-1 h_t^i

where s_t is the filtered output, and k_0, k_1, k_2, k_3 are learnable filter coefficients.

The core idea of GFSSM is to split the long input sequence into Q groups, each with its corresponding hidden state. This way, the recurrent multiplications of a_i are reduced to 1/Q of their original length, thereby mitigating their potential to cause numerical instability. Finally, the output y_t is computed by summing the contributions from all grouped states. One potential problem arises from splitting tokens into different groups: if tokens x_i and x_j belong to different groups, they will never be directly correlated. To address this issue, we introduce a computationally efficient solution by adding a FIR filter that processes the B_t x_t sequence. This enables tokens in different groups to be indirectly correlated, thereby preserving the overall coherence of the encoded sequence. To provide further clarity, we present the structure of matrix L for GFSSM, focusing on the first 13 elements, as shown in Figure <ref>.

The application of FIR filters facilitates smoother transitions and correlations between grouped states, ultimately rendering the model more robust and easier to train. By combining grouped FIR filtering and robust matrix computations, GFSSM offers a viable solution for the inherent challenges of traditional SSM architectures, providing improved performance and stability over extended sequences.



 §.§ Integration of Attention Sink Mechanisms

Inspired by the attention sink phenomenon observed in streaming language models, we incorporate a similar mechanism within GFSSM to improve stability and performance over extended sequences. The attention sink mechanism ensures that the initial tokens in a sequence serve as stable reference points, effectively acting as "sinks" that help maintain the model's focus and coherence over long dependencies. Additionally, it addresses the absence of initial states, thereby ensuring no information is lost from the initial a_0 to a_Q.

In our GFSSM architecture, we initialize the state using learnable prompts that serve as stand-ins for the first few tokens. This initialization is critical for ensuring meaningful state updates from the beginning of the sequence. Formally, we define the state updates incorporating the attention sink mechanism as follows, beginning with the initialization using learnable prompts. For initialization, we start with:

    h_init^i   = 0, 
    
            y    = (L ∘ C^⊤B) 
            [   x_-Q; x_-Q+1;      ⋮;   x_-1;    x_0;    x_1;      ⋮ ]
            + P ·0

where the initial sequence { x_-Q, x_-Q+1, …, x_-1} consists of Q learnable prompts. For continued state updates, the state initialization incorporates the cached prior hidden state:

    h_init^i   = h_-Q^i, 
    
            Y    = (L ∘ C^⊤B) 
            [ x_-Q+1; x_-Q+2;      ⋮;    x_0;    x_1;      ⋮ ]
            + P 
            [ h_-Q^1; h_-Q^2; h_-Q^3; h_-Q^4 ]



    .8!P = 
        [           A_0             1             1             1;           A_0           A_1             1             1;           A_0           A_1           A_2             1;           A_0           A_1           A_2           A_3;        A_4A_0           A_1           A_2           A_3;        A_4A_0        A_5A_1           A_2           A_3;        A_4A_0        A_5A_1        A_6A_2           A_3;        A_4A_0        A_5A_1        A_6A_2        A_7A_3;     A_8A_4A_0        A_5A_1        A_6A_2        A_7A_3;     A_8A_4A_0     A_9A_5A_1        A_6A_2        A_7A_3;     A_8A_4A_0     A_9A_5A_1    A_10A_6A_2        A_7A_3;     A_8A_4A_0     A_9A_5A_1    A_10A_6A_2    A_11A_7A_3; A_12A_8A_4A_0     A_9A_5A_1    A_10A_6A_2    A_11A_7A_3;               ],

where the initial sequence { x_-Q+1, x_-Q+2, …, x_0} is the preceding Q-1 input tokens, and h^i_-Q is the cached hidden state from the previous sequence.

This attention sink mechanism ensures that the initial tokens persist as stable anchors that maintain focus over the sequence's length, thus improving the robustness of our model. Combining grouped FIR filtering and the attention sink mechanism, our final GFSSM model effectively leverages semiseparable matrices within the broader SSD architecture, offering enhanced performance and stability over extended sequences. 



 §.§ Semiseparable Matrix Decomposition

A crucial component of our new structure is the decomposition of the recurrent update matrix into multiple semiseparable matrices. This decomposition enables efficient computation while maintaining the linear complexity of state updates. We denote the decomposed matrix as L, formulated as:


    L = k_0 L_0 + k_1 L_1 + k_2 L_2 + k_3 L_3,

where each L_i represents a semiseparable matrix that captures the contributions from different stages of the FIR filter. Semiseparable matrices are characterized by a structure that allows fast multiplications, like Mamba-2, significantly reducing computational overhead.




§ CONCLUSION AND FUTURE WORK


In this paper, we introduced the Grouped FIR-enhanced Structured State Space Model (GFSSM) as an innovative extension to traditional SSM architectures. Our approach addresses critical training challenges posed by extended series of recurrent matrix multiplications, which often lead to instabilities and difficulties in optimization. By decomposing these multiplications into more manageable groups and incorporating Finite Impulse Response (FIR) filters, we could potentially improve the robustness and trainability of the system. Additionally, inspired by the "attention sink" phenomenon observed in streaming language models, we integrated a similar mechanism into GFSSM to enhance stability and performance over extended sequences.

While we have outlined the theoretical foundations and architectural details of GFSSM, empirical validations are essential to demonstrate the model's practical advantages. Therefore, future work will entail comprehensive experiments to evaluate the performance of GFSSM across various sequence modeling tasks, including but not limited to natural language processing, speech recognition, and genomics.

    ieeenat_fullname
    







 Paper Title: PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learning.


Cloud Collision Signatures in the Central Molecular Zone

    Felix D. Priestley^1
    July 2024
=========================================================

§ ABSTRACT 
Large Language Models (LLMs) encounter significant challenges in continual learning due to catastrophic forgetting, where new information overwrites previously acquired knowledge. This limitation leads to substantial environmental and economic waste. In this study, we introduce the PMoE, Progressive Mixture of Experts with Asymmetric Transformer, which aims to minimize forgetting by utilizing an asymmetric design with shallow layers dedicated to general knowledge and deep layers for new knowledge. 
PMoE incorporates progressively added experts in deep layers and a router that allocates new knowledge to the appropriate experts efficiently.
The router, positioned adjacent to the deep layers, utilizes deep features aggregating consolidated information. This enables the router to perform efficiently, allocating new knowledge to the appropriate experts, which progressively increase in the deep layers. Extensive experiments on TRACE datasets and general language understanding datasets demonstrate that the proposed PMoE outperforms previous state-of-the-art approaches.




§ INTRODUCTION


Although Large Language Models (LLMs) have advanced significantly <cit.>,  unlike humans, they often forget previously acquired knowledge during continuous learning <cit.>. This phenomenon, known as catastrophic forgetting <cit.>, results in substantial environmental and economic inefficiencies, particularly for generative LLMs that require extensive data for training according to scaling laws <cit.>. Consequently, research in continuous learning is crucial for the efficient recycling of language models. This paper focuses on developing techniques that preserve existing knowledge while simultaneously acquiring new information.

Existing continual learning approaches can be divided into three approaches: replay-based <cit.>, regularization-based <cit.>, and architecture-based <cit.>. Replay-based and regularization-based methods use a memory buffer with examples from previous tasks or add regularization constraints to penalize changes in important weights. However, they still suffer from forgetting due to continuous adjustments in parameters. Recently proposed architecture-based approaches <cit.> enhance performance in a parameter-efficient manner but still struggle with the trade-off between learning plasticity and memory stability <cit.> or require prior information such as task-ID to classify input text into task-specific parameters.

In this study, we propose the Progressive Mixture of Experts with Asymmetric Transformer (PMoE), which efficiently adapts to new tasks while minimizing the forgetting of existing knowledge within a continual learning framework. The key idea of PMoE is its asymmetric depth design: shallow layers retain general knowledge, while deeper layers acquire new task-specific knowledge. To achieve this, PMoE progressively adds experts <cit.> and employs a routing network to classify input text.

We take inspiration from <cit.>, which shows that LLMs aggregate information in shallow layers around important words and distribute it in deep layers. We hypothesize that the performance of the router is improved when using features from deep layers rather than shallow layers. In PMoE, specialized experts are absent in shallow layers but present in deep layers, enabling the model to benefit from parameter efficiency and robust routing performance.

The main contributions of our work are summarized as follows:



  * We propose PMoE, which efficiently adapts to new tasks for continual learning by introducing an asymmetric depth design.

  * Our method outperforms LoRA with replay-based and prior state-of-the-art methods on the TRACE benchmark.

  * Experimental results demonstrate that our asymmetric design is both effective and parameter-efficient in preserving prior knowledge while adapting to new knowledge.





§ BACKGROUND



 §.§ Problem Statement


In a continual learning setup, a model sequentially encounters datasets 𝒟_1, …, 𝒟_T, where each dataset 𝒟_t = (x^t, y^t) consists of data points specific to task t. Concretely, the model is provided only with 𝒟_t during training for task t. The primary challenge in continual learning is to enable the model to retain previously acquired knowledge while effectively learning new tasks. In this study, we adopt the general setup of continual learning, which involves maximizing the cumulative performance across all tasks as follows:


    max_θ∑_k=1^T ∑_x,y∈𝒟klog p_θ (y | x)


The overall performance (OP) and backward transfer score (BWT) are measured immediately following the completion of training for the t-th task, as follows:


    OP_t = 1/t∑_i=1^t R_t,i^D 
        BWT_t = 1/t∑_i=1^t (R_t,i^D - R_i,i^D)


where R_t,i^D refers to the score of the i-th task of the model after completing training on the t-th task (i ≤ t). Furthermore, we simulate a realistic setting by having the model generate text sequences without task ID at test time.



 §.§ LoRA


Recently, many Parameter Efficient Tuning (PET) approaches have been proposed and widely adopted in fine-tuning settings. These methods aim to efficiently adapt pre-trained models by updating a smaller subset of parameters than updating all. Among these approaches, Low-Rank Adaptation (LoRA) <cit.> hypothesizes that weight updates in pre-trained models have a "low intrinsic dimension." Instead of updating the entire pre-trained weight W_0 ∈ℝ^d × k, LoRA updates decomposed bottleneck weights B ∈ℝ^d × r and A ∈ℝ^r × k, where W_0 + Δ W = W_0 + BA and the rank r ≪min(d,k). Thus, the forward pass h = W_0 x becomes


    h = W_0 x + Δ W x = W_0 x + BA x.


In this paper, we utilize LoRA as the expert component of our model.




§ ASYMMETRIC MIXTURE-OF-EXPERTS



 §.§ Overall Architecture




To address the issue of information flow within transformer blocks, we propose an asymmetric Mixture-of-Experts (MoE) architecture as illustrated in Figure <ref>. By introducing a threshold τ, the overall network f_θ is divided into shallow layers f_θ_l=1,...,τ and deep layers f_θ_l=(τ+1),...,N. The shallow transformer blocks employ a single LoRA (BA)^l aligned with general human instruction-following, whereas deeper blocks utilize a multi-head LoRA (BA)_k^l aligned with specialized abilities, assigning experts k ∈1,...,T.

At the boundary between shallow and deep layers, the router distributes hidden features from the shallow layer to the deep layer. The routing network simply consists of a linear layer W_g ∈ℝ^h × T, and thus the distribution from routing G(x) ∈ℝ^T is represented as follows:


    G(x) = Softmax (x · W_g).


Many Mixture-of-Experts approaches adopt such routing networks to connect the input text to experts. However, there has been a major issue with the routers having fair connecting performance due to the overwhelming presence of a few experts, and various approaches have been proposed to overcome this drawback by introducing auxiliary loss <cit.>, token capacity <cit.>, or expert dropout <cit.>. Since PMoE uses deep features aggregating consolidated information, our asymmetric design does not require the application of such approaches. More details are provided in Section <ref>.

The forward pass of the self-attention layer can be expressed as:


    f_θ_l+1(h_l)=
    
    W^l · h_l,     l ≤τ
    
    W'^l · h_l,


where W^l = W_0^l + (BA)^l and
 W'^l = W_0^l + ∑_k=1^T G(h_τ)_k · (BA)^l_k.

To progressively allocate all experts with consolidated information, we progressively add the LoRA along with incremental tasks. This parameterization-based method avoids catastrophic forgetting by preserving the knowledge acquired from previous tasks.



 §.§ Experimental Setup



  §.§.§ TRACE Dataset for Continual Learning Evaluation


To evaluate fine-tuning as a scheme for continual learning, we adopt the TRACE dataset <cit.>, which comprises eight distinct datasets spanning multiple domains, including multilingual (C-STANCE <cit.>, 20Minuten <cit.>), code completion (Py150 <cit.>), mathematical reasoning (NumGLUE <cit.>), and domain-specific (FOMC <cit.>, MeetingBank <cit.>, ScienceQA <cit.>) datasets. Following Wang et al., we use a random sample of 5,000 training examples for each task, resulting in a total of 40,000 training examples, with the order always fixed as follows: C-STANCE, FOMC, MeetingBank, Py150, ScienceQA, NumGLUE-cs, NumGLUE-ds, 20Minuten.



  §.§.§ General Ability Evaluation


To assess the impact of domain-specific tasks on the generality performance of pre-trained models, we evaluate five well-known benchmark datasets: MMLU <cit.>, GSM <cit.>, BBH <cit.>, BoolQ <cit.>, and PiQA <cit.>. Following the experimental setup of previous work <cit.>, we measure the general ability delta, which represents the performance difference between pre-continual learning and post-continual learning, as follows:


    Δ R^G_i = 1/M∑_i=1^M(R_t,i^G - R^G_0,i),

where M refers to the total number of benchmarks, and R^G_t,i refers to the result on benchmark i after the t-th tuning.



  §.§.§ Implementation Detail

Our code implementation builds upon the PEFT <cit.> library. We employ Llama2-7b for our pre-trained weights and LoRA for our experts. We optimize our model using AdamW with a minibatch size of 128, a learning rate of 3e-4, and a cosine annealing scheduler. The hyper-parameter rank and τ are set to 4 and 24, respectively. At inference time, the model generates text sequences without knowing the task ID to which it belongs. All results are averaged over 3 runs.

Following the settings of previous studies <cit.>, we incorporate alignment data from LIMA <cit.> into the replay memory, replaying only 1% of historical data.



  §.§.§ Baselines

We compare PMoE with six baseline approaches. For consistency in experimental comparison, the foundation of all baselines is pre-trained Llama2-7b <cit.>. In Parameter Efficient Tuning (PET) comparisons, the number of trainable parameters is approximately consistent (≤ 0.1%).



  * Llama2-7b* <cit.>: uses in-context learning with 6-shot on the current task without fine-tuning.

  * Full-ft <cit.>: fine-tunes full parameters on sequential tasks.

  * LoRA <cit.>: fine-tunes fixed-size LoRA on sequential tasks.

  * O-LoRA <cit.>: incrementally learns new tasks in a direction orthogonal to the LoRA subspace of past tasks while fixing the previous parameters.

  * Full-ft-RE: fine-tunes full parameters with a memory buffer and replays samples from old tasks.

  * LoRA-RE: fine-tunes fixed-size LoRA with a memory buffer and replays samples from old tasks, following the same memory setting as Full-ft-RE.




 §.§ Main Result




The comparative analysis is presented in Table <ref>. The upper part of the table represents the general capabilities of models that have undergone continuous learning to completion, while the lower part represents the specialized performance of models using the evaluation set from the data employed in continual learning. Furthermore, the left side of the table presents models that utilize in-context learning and those without the replay-based approach, whereas the right side presents models that incorporate the replay-based method.

Utilizing replay generally enhances performance, even with only 1% of examples. Replay-based approaches show a BWT greater than zero, indicating not only no forgetting of past knowledge but also an improvement in performance due to recently acquired knowledge and memory. The PMoE outperforms LoRA-RE in both general and tuned abilities by 2.2% and 1.8%, respectively, and even exceeds the full-fine-tuning method, which has more than 1,000 times the trainable parameters, by 3.6% and 2.1% in general and task abilities, respectively.

The current state-of-the-art model, O-LoRA, demonstrates optimal general ability at 53.0%, indicating an improvement in previously acquired knowledge. However, its performance on new tasks is relatively lower at 41.2%. Notably, in the FOMC task, which was initially tuned, the PMoE demonstrated a 34.7% improvement. Consequently, the PMoE not only enhances the performance of tuned tasks but also retains a high level of generalization ability.



§ DISCUSSION




 §.§ Asymmetric Design


The core strategy of the PMoE architecture is characterized by its asymmetric design, where shallow layers are dedicated to general abilities and deeper layers are tailored for newly acquired abilities. To assess the efficacy of this asymmetric design, we analyze the performance as functions of varying τ. A lower τ results in the model possessing a greater number of task-specific parameters, with the routing mechanism utilizing shallow features. Conversely, a higher τ leads to a model enriched with parameters for general abilities, where the router utilizes deep features. As illustrated in Figure <ref>, the model demonstrates optimal average performance at a moderate τ, and a decrease in computational cost proportional to τ. Thus, if performance levels are comparable, it is advantageous to increase τ to reduce computational demands.





 §.§ Router Analysis



To investigate the router in PMoE, we show matrices representing the allocation probability by the router. As shown in Figure <ref>, when the router is located near the shallow layer (i.e., τ=6), the router mainly chooses only a few experts. This imbalance problem is a well-known phenomenon where input x is assigned to a specific superior expert, rather than being distributed across a diverse range of experts. Previous works have attempted to address the router imbalance problem by introducing auxiliary losses <cit.>, token capacity <cit.>, or expert dropout <cit.>. However, when the router is located near the deep layer (i.e., τ=24), the imbalance problem is automatically solved, well distributing all experts. As mentioned in the motivation of our study, this result supports the hypothesis that LLMs aggregate information in shallow layers and distribute it in deep layers, leading to better router performance when using features from deep layers rather than shallow layers.



To further investigate, we conducted a qualitative evaluation of the token allocation by the router, as illustrated in Figure <ref>. The evaluation utilized test prompts that include three distinct types: Python code, mathematical questions and answers, and lines from Shakespeare's Hamlet. With a τ of 6, most tokens are predominantly allocated to experts 2 or 8. In contrast, at τ of 24, the allocation patterns are more varied, indicating a comprehensive use of experts. Concretely, the router seems to allocate tokens to experts based on identifiable patterns related to the role of the tokens. For example, special characters are predominantly assigned to expert 3, elements of Python syntax are mainly assigned to expert 4, and plain text is typically assigned to expert 1.

Drawing from the aforementioned analysis, if τ is 24 and the task-ID is treated as the ground truth for the router during the training stage, PMoE is capable of employing each expert as a task-specific expert without the need for task-ID during the inference stage. We incorporate cross-entropy loss to guide the routing of input x to a suitable expert, as follows:


    ℒ(x, k) = - log G_k(x)


where G_k(x) denotes the router probability of assigning input x to its corresponding task-ID k. This loss provides a simple yet effective mechanism to prevent inputs from being pushed to a single expert by guiding the task assignment corresponding to input x.



The comparison with the application of Equation <ref> is shown in Figure <ref>. This auxiliary loss function experimentally demonstrates its effectiveness in making experts task-specific without task-ID in the inference stage. However, in Table <ref>, applying the auxiliary loss slightly degrades overall performance. This result suggests that a combination of experts rather than task-specific experts has a positive effect on overall performance. Future research should explore extending task-specific experts for asymmetric design.







 §.§ Comparison with Trainable Parameter Size



We assessed the detailed comparison between PMoE and LoRA with the replay method at different scales of adaptation parameters, as shown in Table <ref>. In our evaluation, we control the number of parameters by the rank of the adapter from under 0.1% to over 0.3%. In all parameter size settings, the general ability of PMoE is higher than that of LoRA by 0.7%-2.2%. In specialization performance, PMoE is higher than LoRA by 0.6%-0.8%. Interestingly, there is only a slight difference in performance along with the parameter size.




§ RELATED WORKS



 §.§ Parameter Efficient Fine Tuning


As larger pre-trained models become more prevalent, efficiently fine-tuning their parameters has gained significant importance. Parameter Efficient Fine Tuning (PEFT) approaches are a set of methods that update only a small portion of parameters while freezing the pre-trained parameters during the fine-tuning stage. Various approaches have been proposed, including inserting additional modules or parameters, such as Adapter <cit.>, Prompt Tuning <cit.>, and Low-Rank Adaptation (LoRA) <cit.>.

In this paper, we focus primarily on LoRA, which hypothesizes that weight updates in pre-trained models have a "low intrinsic dimension." Building upon LoRA, AdaLoRA parameterizes incremental updates in the form of singular value decomposition <cit.>, O-LoRA involves layering low-rank adapters on the key and value projection matrices <cit.>, SIRA leverages the Sparse Mixture of Experts to boost the performance of LoRA <cit.>, and PMoE employs an asymmetric design with shallow layers dedicated to general knowledge and deeper layers for new knowledge.



 §.§ Continual Learning


Existing continual learning approaches can be categorized into three main types: replay-based, regularization-based, and architecture-based.

Replay-based approaches<cit.> store examples from previous tasks for rehearsal. Although these approaches achieve strong performance, storing large samples can be inefficient and may not be feasible for data privacy settings.

Regularization-based approaches <cit.> update only a few weights trained on previous tasks. These approaches are memory efficient but can suffer from catastrophic forgetting, especially in long sequence tasks.

Architecture-based approaches isolate weights trained on previous tasks and progressively add weights for new tasks. Although the recently proposed progressive prompt <cit.> has achieved state-of-the-art results, it has the limitation of requiring the task-ID during the inference stage. Among the continual learning techniques that do not require task-ID, O-LoRA <cit.> effectively addresses catastrophic forgetting. However, its performance, as measured by the TRACE benchmark <cit.>, was lower. In this paper, PMoE achieves superior performance on the TRACE benchmark without requiring task-ID during the inference stage.



§ LIMITATIONS


In our study, we conducted experiments across eight distinct tasks. Expanding the number of tasks could enhance the robustness and generalizability of our findings. Our research focuses solely on generative models, particularly Llama2-7b. However, the foundational concepts of our model hold potential for broader applicability in task-agnostic settings. Further research will assess the efficacy of our proposed approach across diverse benchmarks, aiming to broaden its application to multimodal continual learning.



§ CONCLUSION

In this paper, we enhanced the performance of LLM in continual learning by using PMoE. The PMoE features an asymmetric architecture between shallow and deep layers. Specifically, the shallow part employs a single expert to enhance general ability, whereas the deep part progressively utilizes an increasing number of experts to focus on tuned performance. Empirical experiments on comprehensive continual learning benchmarks demonstrate that the proposed PMoE outperforms previous state-of-the-art approaches. Furthermore, to investigate the asymmetric structure, we experimented by varying the asymmetric hyperparameter τ, measuring the performance and efficiency of the model, and analyzing the router.







 Paper Title: Enabling Contextual Soft Moderation on Social Media through Contrastive Textual Deviation.














Enabling Contextual Soft Moderation on Social Media
 through This paper is accepted for publication at the 2024 USENIX Security Symposium. Please cite accordingly.
    Pujan Paudel, Mohammad Hammas Saeed, Rebecca Auger, Chris Wells, and Gianluca Stringhini

Boston University

{ppaudel,hammas,raauger,cfwells,gian}@bu.edu

    
===================================================================================================================================================================




§ ABSTRACT 

Automated soft moderation systems are unable to ascertain if a post supports or refutes a false claim, resulting in a large number of contextual false positives.
This limits their effectiveness, for example undermining trust in health experts by adding warnings to their posts or resorting to vague warnings instead of granular fact-checks, which result in desensitizing users.
In this paper, we propose to incorporate stance detection into existing automated soft-moderation pipelines, with the goal of ruling out contextual false positives and providing more precise recommendations for social media content that should receive warnings.
We develop a textual deviation task called () and show that it outperforms existing stance detection approaches when applied to soft moderation.
We then integrate into the state-of-the-art system for automated soft moderation Lambretta, showing that our approach can reduce contextual false positives from 20% to 2.1%, providing another important building block towards deploying reliable automated soft moderation tools on social media.




Enabling Contextual Soft Moderation on Social Media
 through This paper is accepted for publication at the 2024 USENIX Security Symposium. Please cite accordingly.
    Pujan Paudel, Mohammad Hammas Saeed, Rebecca Auger, Chris Wells, and Gianluca Stringhini

Boston University

{ppaudel,hammas,raauger,cfwells,gian}@bu.edu

    
===================================================================================================================================================================







§ INTRODUCTION





“Get the facts about COVID-19.”
This message became a familiar sight for the users of Twitter during the pandemic surge in 2020.
Added as a warning label by the platform to accompany potentially misleading tweets about COVID-19 (see Figure <ref>), its goal was to allow users to seek more information and not fall for dangerous conspiracy theories and misinformation which could have compromised public safety <cit.>.
This was an early example of soft moderation, which was later applied by Twitter on other important topics (e.g., the 2020 US Presidential election <cit.>) and has since been adopted by other platforms like Facebook <cit.>, Instagram <cit.>, and TikTok <cit.>.

Platforms do not release details on how they identify tweets that need to receive soft moderation labels, but this includes getting input from reputable authorities like fact-checking organizations (e.g., Snopes) and public agencies (e.g., the CDC), applying automated tools like semantic matching and information retrieval to identify posts about known misleading claims, and relying on human moderators to vet content that should receive a warning <cit.>.
Despite these efforts, however, recent work has showed that the methods adopted by online platforms fail to flag a large amount of misleading content that should be moderated <cit.>.

Fully automated attempts by platforms to identify misleading content had shortcomings.
We illustrate an example of this in Figure <ref>.
To curb the spread of a conspiracy theory linking 5G technology to the spread of COVID-19, Twitter decided to add a warning label to any tweet that contained the words “5G” and “oxygen” <cit.>.
Due to the generality of these keywords, this caused the platform to flag tweets that were unrelated to the conspiracy theory, like the one in Figure <ref>.
We call these irrelevant tweets mistakenly flagged for moderation topical false positives.
In addition to irrelevant tweets, Twitter also applied warning labels to tweets that debunked the conspiracy theory, like the one in Figure <ref> <cit.>.
This can have adverse effects, for example by having the platform undermine the public trust in experts (e.g., health professionals) by mistakenly flagging their informative posts as misleading <cit.>.
It would also discourage diligent users who actively debunk misinformation on social media (i.e., “wisdom of the crowds” <cit.>), who would see their fact-checks marked as misleading.
Finally, it can cause warning fatigue <cit.> in social media users, where users are bombarded with warning labels attached to the posts and stop paying attention to them, thereby reducing the intended effectiveness of the labels.
We call this type of mistakenly flagged posts contextual false positives.
We argue that an effective approach for the automated identification of content that should receive moderation should address both topical and contextual false positives.

Recently, the computer security community has focused on developing automated systems to flag content that should receive moderation labels.
Lambretta <cit.> is a system that leverages information retrieval techniques (i.e., Learning to Rank) to identify the optimal set of keywords that represent a misleading claim and identify social media posts that are discussing that claim.
Lambretta outperforms alternative keyword retrieval and semantic search approaches, and largely addresses the problems of topical false positives, reporting a false detection rate of 3.93% on a dataset of tweets discussing the 2020 US Presidential Election.
However, Lambretta does not address the problem of contextual false positives, being unable to ascertain if a social media post is supporting a false claim (and therefore should be moderated) or refuting it.
In fact, the false detection rate from the perspective of contextual false positives on its original dataset is 20%.
This limitation is not unique to Lambretta, but is also present in other content moderation systems powered by retrieval or semantic similarity approaches <cit.>.


To reduce contextual false positives, stance detection techniques can be integrated into soft moderation systems.
Stance detection is a task where the objective is to learn a model that can identify whether a given piece of text is in favor of or against a set of target(s) <cit.>.
In the context of fact-checking, the problem aims to automatically detect if a piece of text supports or refutes a misleading claim <cit.>.
We find, however, that existing stance detection techniques are limited in their ability to generalize across unseen claims <cit.> and different social media platforms <cit.>, making them unfit to be applied on social media platforms at scale.

To address the shortcomings of existing stance detection approaches when dealing with content moderation on social media, we reframe the problem as a textual deviation detection one and propose  ().
Our approach consists of developing, for each claim that we wish to moderate, a triplet composed of a consensus statement that is confirmed to be true (e.g., by health authorities or fact-checking organizations) and two contrastive markers, a piece of refuting evidence and one of supporting evidence.
Our approach then leverages the emergent abilities of Large Language Models (LLMs) like zero-shot learning to identify if a piece of text supports or refutes the consensus statement.

Compared to conventional stance detection approaches, our method has two major advantages.
First, we build a model that can identify patterns of deviation from an anchor text (i.e., the consensus statement) in a topic-invariant fashion, without depending on features specific to a certain topic (e.g., climate change denial or vaccine misinformation).
This overcomes one of the main issues of traditional supervised stance detection pipelines, where researchers need to annotate samples for each claim as these methods fail to generalize well across multiple claims <cit.>.
Second, can leverage the consensus statements curated by fact checkers and expert authorities as part of the stance detection process (through the use of the consensus statement), allowing us to keep experts in the loop in the case of evolving events like a global pandemic or misinformation stories that threaten public safety. 

We first motivate the task of textual deviation detection by identifying the shortcomings of existing approaches for stance detection in the context of content moderation.
We then perform preliminary experiments on the advantages of reframing stance detection as textual deviation by bootstrapping the task using the zero-shot learning capabilities of Large Language Models (LLMs).
Motivated by the better performance of LLMs over alternative solutions and by the ease of generalizing across datasets, we further fine-tune the LLM for the task of to boost its performance.
Finally, we integrate our fine-tuned unsupervised stance detection model into the analysis pipeline of the state-of-the-art soft moderation system Lambretta <cit.>, to improve its detection capabilities.
We make all the labeled datasets curated in the work and fine-tuned models publicly available [<https://huggingface.co/collections/ppaudel/contrastive-textual-deviation-65e20c48680724cc9a809062>].

In summary, this paper makes the following contributions:

    
  * We propose () as a new task to perform stance detection on social media. 
        We show that overcomes the limitations of existing stance detection approaches and that it generalizes across platforms by testing it on datasets collected from different platforms (Twitter and Reddit) and covering different topics (COVID-19, climate change, and politics).
    
  * We fine-tune a LLM dedicated to the task of and show that our approach improves over baseline methods, with an average improvement on the F1-score of over 7%. 
    
  * We integrate into the automated soft moderation pipeline of the state-of-the-art approach Lambretta, reducing contextual false positives from 20% to 2.1%, with a minimum impact on false negatives.




§ DATASETS


We curate and collect multiple datasets throughout our work that i) come from multiple social media platforms (Reddit, Twitter), ii) cover different topics (e.g, Climate Change, COVID-19), and iii) contain different levels of granularity (i.e, fine-grained or coarse-grained claims).
A summary of the datasets used in our work is presented in Table <ref>.



Global Warming Stance Dataset (GWSD) <cit.>.
This dataset contains 1,177 “opinion spans” of news headlines about Global Warming annotated by human annotators as either accepting of global warming (e.g., 
“We can't afford to wait until everyone is feeling the pain of the climate emergency before we do something about it”) or skeptical of it (e.g., “Humans have negligible impact on the climate”).
The opinion spans are extracted from 56,000 news articles spanning a period of 20 years from news outlets such as the New York Times, Fox, Washington Post, Forbes, etc.
We characterize this dataset as a “coarse-grained” stance dataset since the opinion spans in this dataset discuss climate change from the broader perspective of climate emergency and skepticism, and do not deal with granular causes or effects of climate change (e.g., Urban Heat Island effect on Global Warming is negligible.)
This dataset is slightly imbalanced in its class labels as it contains 777 opinion spans supporting climate change 
and 400 opinion spans skeptical of climate change.

Climate Skepticism.
The majority of stance detection datasets are topic-based or target-based in nature, with a very limited set of datasets on claim-based stance detection occurring on social media text. 
This existing gap motivates us to prepare a comprehensive claim-based stance detection dataset.
We curate a dataset of climate change denial discussions, containing Reddit posts that support or refute three different claims related to different arguments used in climate change discussions.
We query the Pushshift Reddit dataset <cit.> with curated keywords related to the claims, retrieving posts from Reddit that are discussing narratives related to three of the most popular climate denier claims: i) Cosmic rays are causing global warming (GW), ii) Antarctica is gaining ice, and iii) Urban Heat island (UHI) effect exaggerate global warming trends.
These claims are fine-grained and objective in nature, unlike high-level “topics” in the GWSD dataset such as “Global warming is a hoax,” or “Climate change is not happening.” <cit.>
For the goal of scalable and tractable soft moderation, these claims are very representative of misleading claims propagating on social media that can be easily refuted from scientific consensus, or authoritative sources (e.g. Skeptical Science). 
We then set out to annotate 1,000 Reddit posts for each claim by developing a codebook dedicated to climate skepticism. 
Our codebook was informed by the crowdsourced resource Skeptical Science, which provides pointers for understanding different narratives used by climate skeptics to deny a claim as well as examples of scientific support for confirmed claims.
We do not include posts that are inquiring about the claim in question, are neutral towards it, or are topically irrelevant.
Two researchers performed multiple rounds of annotation and reached a near-perfect agreement of κ = 0.865 <cit.>) Cohen's Kappa (i.e., strong agreement).
This dataset also has value in evaluating the transferability of stance detection approaches as it contains multiple targets (claims), and different types of label distribution within the claims.
A summary of the climate denial claims and the number of annotated Reddit posts for each claim is provided in Table <ref>.



COVID-FACT <cit.>.
This dataset belongs to a family of datasets in the field of automated fact-checking known as FEVER (Fact Extraction and Verification).
What makes it interesting for our work is that the FEVER dataset contains labeled text known as “evidences” that either supports or refutes a given claim.
More precisely, the COVID-FACT dataset contains 4,086 claims concerning the COVID-19 pandemic, and both refuting (3,488) and supporting evidence (3,515) for the claims.
These claims are obtained from the /r/COVID19 community of Reddit and are “fine-grained” in nature.

COVID-CQ <cit.>.
In addition to creating our own dataset for identifying stance in climate denial claims, we also make use of an existing stance detection dataset of COVID-19 treatment tweets. 
The dataset, called COVID-CQ, contains 7,003 tweets and their respective stance on the efficacy of hydroxychloroquine as a treatment for COVID-19.
The annotation criteria of this work closely align with the annotation criteria of our own data collection process, and this dataset fits the problem definition of claim-based stance detection very closely.

Stanceosaurus <cit.>.
This dataset is a multi-lingual and multi-cultural corpus of tweets annotated with a stance towards 4,467 tweets discussing 190 misinformation claims from 9 fact-checking sources (Snopes, Poynter, FullFact, etc.)
We use a subset of the dataset containing fact-checked claims and tweets in English, which either support or refute the claims, ignoring the tweets that are either irrelevant to the claim, or simply querying about the claim.
For the posts that are discussing the claims in question, the dataset also contains annotation if the tweets are leaning towards supporting or refuting the claim, and we use tweets that are annotated as such with the label of their leaning.

Election denial <cit.>.
This dataset contains 499 election denial claims and 101,353 tweets discussing the claims retrieved by the corresponding soft-moderation system Lambretta. 
We select the three most popular election denial claims by frequency present in the dataset and annotate a sample of 200 random tweets discussing the three claims for the stance of the tweets concerning the claim in question.
We adapt the codebook and annotation process used for annotating climate denial-related claims.
A near-perfect agreement of κ = 0.866 <cit.> was reached among the annotators.
A summary of the claims, and the number of supporting, and refuting tweets for each claim is presented in Table <ref>.



PERSPECTRUM <cit.>.
Finally, we use a dataset from argument mining called PERSPECTRUM <cit.>.
PERSPECTRUM contains 907 claims from online debate topics and 5,095 “perspectives” from search engine results presenting diversifying viewpoints about the claims.
These “perspectives” are annotated with stance as either supporting or refuting the claim and cover more than 10 different topics such as Politics, Freedom of Speech, Environment, Science, Health etc.
The structure of this dataset provides value in augmenting a large-scale stance dataset as it contains claims and multiple sides (perspectives) of the claims from different topics.

Summary. A summary of all the datasets curated and collected is listed in Table <ref>.
We can observe that the datasets used by our work span across different use cases of misinformation such as climate denial, public health emergencies, civic processes such as elections, and general purpose fact-checking.
This way, we aim to comprehensively evaluate our method on a variety of misleading claims occurring across two different social media platforms.



§ MOTIVATION: EXISTING STANCE DETECTION METHODS FALL SHORT



We identify three requirements that effective stance detection approaches should satisfy to successfully improve content moderation systems.
Ideally, we would be able to leverage existing stance detection methods for this purpose.
However, we find that previous work, including more sophisticated entailment-based methods <cit.>, fall short in achieving one or more of these requirements.
In a nutshell, the requirements and shortcomings that we identify are the following:



    
  * R1. Need for Granularity: Supervised methods trained on coarse-grained claims fail on fine-grained claims.
    
  * R2. Need for Claim Invariancy: Supervised methods trained on one claim do not generalize well on other claims.
    
  * R3. Need for Contrastive Context Awareness: Supervised entailment-based methods fail to identify stance on social media despite being given context-aware hypothesis statements.



In the rest of this section, we discuss the shortcomings of existing approaches in achieving the three requirements, showcasing our preliminary experiments.
Then, based on these observations, in Section <ref> we present our solution, which overcomes the limitations of previous methods. 



 §.§ Need for Granularity (R1)



While coarse-grained claims (e.g., “Climate change isn't real”) may generally describe broad categories of misinformation, finer-grained ones (e.g., “Cosmic rays are causing global warming”) are often encountered online.
An effective stance detection approach should be able to operate on both categories of claims.

To investigate the ability of stance detection to generalize between coarse-grained and fine-grained claims, we use the GWSD and Climate Skepticism datasets.
These two datasets both pertain to climate change, but GWSD contains coarse-grained claims while the Climate Skepticism one is built from fine-grained ones.
We train a supervised model on the GWSD dataset and test whether this model can identify the stance of climate skepticism on the fine-grained claims of the Climate Skepticism dataset.
Following the standard BERT fine-tuning recipe <cit.>, we fine-tune a DistilBERT <cit.> model for five epochs with a learning rate of 5e^-5 until the training loss converges.
We balance the dataset by randomly undersampling the supporting claims, as the original dataset is imbalanced. 
We repeat the experiment five times and report the average evaluation of the fine-tuned BERT model on the three claims from the Climate Skepticism dataset.



Takeaways. The fine-tuning procedure on the GWSD dataset performs well when cross-evaluated on coarser claims (F1 score of 0.766 on 5-fold cross-validation).
However, the model performance drops drastically when evaluated on the fine-grained climate claims from the Climate Skepticism dataset as seen in Table <ref>.
This experiment shows that supervised models trained on coarse-level claims fail to evaluate the stance on fine-grained ones, despite having high domain and topic overlap with the training data.
This motivates the need to design stance detection methods that are highly granular and specific in nature.



 §.§ Need for Claim Invariancy (R2)


The second requirement that we identify is that a stance detection approach trained on a fine-grained claim about a topic must generalize to detect stance on other fine-grained claims about the same topic.
This is important because false information is not static and new claims emerge all the time (e.g., the emergence of narratives advocating for Vitamin C, Hydroxychloroquine, and Ivermectin as effective cures against COVID-19 at different points in time during the pandemic).

To investigate if existing stance detection approaches can generalize between claims, we again use the Climate Skepticism dataset, as it contains three fine-grained claims.
It is to note that all three corpora for the individual claims come from the same social media platform (Reddit), thus we can expect the corpus distribution of the evaluation setting to be similar to the training setting.
As in the previous step, we train a DistilBERT model for each claim and evaluate the model on detecting the stance of posts on the other two claims that it was not trained on.
The results are provided in Table <ref>.

Takeaways. We find that in all three cases, a model trained on one claim fails to generalize on other claims as the performance drops drastically.
This motivates the need for building claim invariant stance detection methods that are not only learning features to detect stance specific to the training data of the claim it is trained on, but generic representations of support or refute towards a claim.





 §.§ Need for contrastive context awareness (R3)



An effective stance detection approach must be able to accurately identify text that supports or refutes a given claim.
A promising approach to achieve this is Natural Language Inference (NLI), which is also known as Recognizing Textual Entailment (RTE).
NLI is a widely popular approach for detecting stance in NLP, where a claim is treated as a premise and a piece of evidence is treated as a hypothesis. 
The task then consists in checking if the premise entails (supports) or contradicts (refutes) the hypothesis <cit.>.
Compared to the approach of fine-tuning BERT-based models for classification tasks, NLI can be adapted with a much more granular objective for stance detection, as each claim being evaluated can be directly subjected to the most relevant piece of evidence associated with the claim.
One of the major reasons existing stance detection methods fail to generalize well on detecting stance on a new topic during inference is the lack of enough contextual information about the topic or target they are subjected in the out-of-domain or zero-shot setting <cit.>.
Prior research showed that providing enough context about the topic or claim being evaluated can bridge this context gap, improving stance detection models to generalize well on unseen topics <cit.>.
Based on this, we investigate if we can leverage NLI methods for stance detection on social media by providing the most “context-aware” set of evidence (hypothesis) statements related to a claim (premise) getting evaluated.
Our assumption is that if there is sufficient context provided in the hypothesis statement, NLI models should be able to detect stance well on new claims and topics.


We first train an NLI model on COVID-19 claims using the COVID-FACT dataset.
We then evaluate the trained model on the COVID-CQ dataset.
The FEVER setup of the COVID-FACT dataset is perfect for this task, as it already contains pieces of evidence supporting or refuting a claim.
In addition to that, these claims are “context-aware,” (e.g., about the lack of studies confirming the effectiveness of Hydroxychloroquine to treat COVID-19) instead of being generic like “Covid-19 and mask” or “Covid-19 and HCQ.”
Also, COVID-FACT consists of claims and evidence that are semantically close to those being evaluated in the COVID-CQ dataset (e.g., “homemade remedies of COVID-19” or “alternative treatments of COVID-19”), therefore we can expect a high level of domain overlap, similar granularity, and semantic overlap between the training data and the evaluation data.

Following a similar methodology as in prior sections, we train a DistilBERT model for textual entailment using these pairs and evaluate the trained model on the COVID-CQ dataset.
Since NLI only allows to specify one hypothesis for each premise, we evaluate the trained model on two different configurations using the following hypotheses: i) Another study has confirmed hydroxychloroquine to be effective in the treatment of COVID-19 (coronavirus), ii) No clinical studies have confirmed hydroxychloroquine as a cure for COVID-19 (coronavirus).
We then consider a tweet as supporting the misleading narrative if it either entails with first hypothesis or contradicts the second hypothesis.
Similarly, a tweet debunks the misleading narrative if it either contradicts the first hypothesis or entails the second hypothesis.
This way, we setup an evaluation setting for a model that is trained on context-aware sentence pairs, and evaluate it under an identical setting, allowing us to understand if satisfying the criteria of “context-awareness” is sufficient enough for stance detection for content moderation.

Takeaways. We evaluate the trained DistilBERT model on the COVID-CQ dataset and find that the best-performing F1 score of this method is 0.53, which is only slightly better than random chance, despite having a high topical overlap with the training dataset, and further taking a granular approach of training entailment or contradiction aligned on claims.
Moreover, we discover that the NLI model evaluated on the “context-aware” setting, providing the best set of hypothesis statements with “sufficient” information about the claim, is not adequate for stance detection on social media text.
These results suggest that providing a context-aware hypothesis statement is not enough to build NLI models for precise stance detection. 
We argue that while context awareness through the best set of hypothesis statements gives a model important contextual signals about the claim, it would be beneficial for the model to be “contrastively context-aware,” i.e., exposed to contradicting hypotheses, one supporting and one refuting the claim, which is what commonly occurs on social media.




§ 



Based on the three requirements defined in the previous section, we aim to build an unsupervised stance detection model that overcomes the limitations of previous approaches by: i) detecting stance at a fine-grained level, ii) learning semantic representations of stance on a claim invariant fashion, and iii) encoding contrastive context awareness to learn higher level representations of stance.
We first describe our proposed solution by formally defining a new task called () and discuss the different components of the task.
We then bootstrap by leveraging the zero-shot learning capabilities of Large Language Models (LLMs) through prompt engineering.
Finally, we conclude the section by demonstrating the successful performance of bootstrapped by comparing against multiple baselines from supervised learning. 


 §.§ Task Definition


We propose to reframe the problem of stance detection as a new task of “textual deviation” detection, which aims to satisfy all three requirements.
First, for every misleading claim that we wish to moderate, we start from a consensus statement that has been confirmed to be true, for example by health authorities like the World Health Organization (WHO) in cases of public health emergencies or by fact-checking organizations in case of political events.
Our intuition is that posts that refute the claim, spreading false information while doing so, will deviate from the consensus statement, while those that support (debunk) the claim will stick closer to the messaging of the consensus statement.
We then extend the idea of “context aware” textual entailment to “contrastively context-aware” textual entailment by providing a pair of “contrastive markers” for each consensus statement associated with the misleading claim: a piece of refuting evidence and one of supporting evidence.
Our idea is to leverage the zero-shot capabilities of Large Language Models (LLMs) to align social media posts discussing a claim with the piece of anchor “consensus” statement specific to the claim, while using the “contrastive markers” as additional context to aide the alignment decision.
We call this task  ().



The core of lies on formulating this triplet of i) a consensus statement, ii) a refuting evidence, and iii)  a supporting evidence for each claim that we want to infer the stance of social media posts.
Compared to traditional supervised learning approaches which need hundreds or even thousands of examples for training claim-specific or topic-specific stance models, we argue that curating this triplet is a much easier and tractable human effort, which could be easily carried out by fact-checking organizations or social media platforms.
The consensus statements used to anchor the triplets are the “truth values” about the false claim in question.
For each false claim in the datasets, we use the corresponding fact-checks and scientific explanations associated with the claim.
For Climate Skepticism claims, we use fact-checks from <SkepticalScience.com> available under the “What the science says” snippet for every climate skepticism claim.
Stanceosaurus contains fact-checked claims coming from several reputable sources (e.g. PolitiFact, Snopes, etc.), and we used the debunking article headline from the corresponding fact-checking sources as the consensus statement.
For the COVID-CQ dataset, we used Google Fact Check Explorer to identify fact-checks and used their debunking title as the consensus statement.
To complete the triplet for each claim, we formulate the contrastive markers as a positive affirmation and a negative reframing of the corresponding consensus statement.
Figure <ref> shows examples of triplets from three different claims from our evaluation datasets, with consensus statement, supporting evidence, and refuting evidence for each of the claims.


allows us to satisfy all three requirements discussed in Section <ref>.
R1 is addressed by allowing us to use customizable and well-informed consensus statements directly discussing the claim being moderated.
To address R2, learns semantic features of deviation from an anchor piece of text (consensus statements), rather than semantic features of support or refute towards a particular claim that the model is trained on, allowing us to scale our method across claims and achieve the property of claim invariancy.
Finally, addresses R3 by using a pair of contrasting evidence as context input for the classification task (refuting evidence and supporting evidence).
This allows our approach to improve over entailment-based methods, which only use one piece of hypothesis statement and thus do not generalize to the case of content moderation on social media, as we showed in Section <ref>.
Before bootstrapping as a NLP task, we empirically validate the underlying intuition motivating by visualizing if tweets supporting a claim are semantically different than tweets refuting it.
We utilize the tweets from the COVID-CQ dataset for this experiment and embed the tweets using the Sentence-T5 encoder <cit.>, producing 768-dimensional embeddings.
Note that these sentence embeddings are produced using the same encoder component of the LLM we will use for bootstrapping purposes i.e. (FLAN-T5-XXL), and thus serve as a useful tool for empirically testing our intuition.
To this end, we center the embeddings of tweets from the COVID-CQ dataset using the embedding of the consensus statement and project the T-SNE embeddings of the supporting and refuting tweets.
The scatterplot in Figure <ref> shows a strong separation between the supporting and refuting tweets, with minimal overlap, indicating that the supporting tweets and refuting tweets are indeed semantically different. 
Additionally, we also compute the cosine similarity of the embeddings of the supporting tweets and refuting tweets with the consensus statement.
Statistical analysis using a two-sample t-test to compare the means of the supporting and refuting tweets allows us to reject the null hypothesis (with p <<< 0.01 and t-statistics >>> 0).
This further validates our intuition that the posts supporting the claim align more with the consensus than the posts that refute the claim.







 §.§ Bootstrapping using LLMs

Due to their zero-shot learning abilities <cit.>, Large Language Models (LLMs) can be used off-the-shelf to leverage the triplet structure defined by to detect stance in an unsupervised fashion.
We argue that LLMs are particularly well suited for this task because: i) they have demonstrated performance competitive to supervised baselines in multiple NLP tasks <cit.>, ii) they can incorporate the proposed contrastive markers through in-context learning <cit.>.

Background. Recent approaches showed that prompting a LLM to a specific prompt that defines the downstream task (e.g., sentiment analysis, textual entailment), and the possible label spaces for the tasks (e.g., positive, negative, or entailment, contradiction) can lead to performance comparable to supervised methods for text classification <cit.>.
This ability of LLMs has been shown to be “emergent:” as the size and scale of these language models increases, so does their ability to generalize to unseen tasks <cit.>.
Another emergent ability of LLMs is their capability to perform “in-context learning.”
By extending the prompts to include a few examples of the downstream task and the classification labels, LLMs can learn new tasks.
Subjecting LLMs to demonstrative examples can give the model an idea of the label space, of the mapping between input and label space, and help generalize LLMs to a variety of tasks without the need for task-specific fine-tuning. 
In-context learning does not require any model update or fine-tuning and performs competitively with state-of-the-art methods for text classification problems <cit.>.

Prompt engineering to bootstrap LLMs. 
To show the feasibility of our approach for soft moderation, we first bootstrap with LLMs.
To this end, we build our prompts as follows:
For every social media post discussing a claim, we subject it to a consensus statement that explains the established consensus (e.g., a fact-check or a statement on the scientific consensus about the claim).
Then, we proceed to show an example statement that refutes the consensus. 
Similarly, we show an example statement that supports the consensus.
Finally, we end the prompt with the text that we wish to classify using the LLM, where the LLM is expected to return one of the stance labels (support, or refute).
Figure <ref> illustrates the prompt structure that we use for bootstrapping .



Our intuition is that by doing so we can leverage the zero-shot classification capabilities of LLMs, while at the same time ground them towards incorporating the consensus and the contrasting evidence as additional context for classification.

Validating bootstrapping.
We use the open-source encoder-decoder model FLAN-T5 by Google for bootstrapping purposes <cit.>, 
prompting, for each claim, the LLM with the prompt structure defined above.
The underlying architecture of FLAN-T5 is a T5 model (Text-to-Text Transfer Transformer), a new unified paradigm of transfer learning for text classification.
By formulating challenging tasks like machine translation, summarization, etc. to be a text-to-text task, T5 has shown promising results in multiple NLP benchmark problems over encoder-based models like BERT <cit.>.
FLAN-T5 is an improved version of the T5 model that has been “instruction finetuned” on more than 1,000 NLP tasks and thus can be used to perform zero-shot text classification by just “prompting” the model.

As a summary of our validation results, we present the results of the task bootstrapping on the Climate Skepticism dataset.
We will comprehensively evaluate our task with a fine-tuned model against all three evaluation datasets and claims later in Section <ref>.

As baselines, we compare our bootstrapped task against two different prompt setups: i) LLM without contrastive markers, ii) LLM without consensus definition.
Additionally, we also compare our method against two supervised methods for stance detection i) a DistilBERT <cit.> model fine-tuned on each claim, and ii) a fine-tuned NLI model on ClimateFever dataset <cit.> (following the identical procedure we fine-tuned NLI model on COVID-FACT dataset in Section <ref>).
The purpose of evaluating the two added prompt configurations alongside the prompt configuration for is to understand the role that each component of the triplets (consensus statement and the contrastive markers) plays in identifying stance.
The prompt configuration of evaluating LLMs without contrastive markers will help us understand if the contrastive markers are an improvement over entailment-based approaches.
For the DistilBERT model, which uses supervised fine-tuning, we perform 5-fold cross-validation for each of the claims and report the average score across the folds.
We use the weighted F1 score as the evaluation metric as the label distribution of supporting and refuting labels for each of the three claims in the Climate Skepticism dataset is imbalanced.



Results. A summary of the bootstrapping results with the mean F1 scores for the three Climate Skepticism claims is presented in Table <ref>.
We find that our bootstrapped method (which is unsupervised) performs better than supervised BERT-based fine-tuning baselines.
Additionally, we find that our approach provides better performance over using the LLMs in their default state: without consensus grounding (LLM without consensus), and without contrastive markers (LLM without contrastive markers).
The results also suggest that the triplet setup of performs better than the conventional setup of entailment for stance detection.
Additionally, we observe that having the contrastive markers by themselves without the consensus statement (LLM without consensus) is not nearly as effective for detecting stance.
Overall, these results validate that the task of can outperform existing methods for supervised stance detection.
Moreover, the utility of our approach is validated in being fully unsupervised and requiring minimal setup needed for a claim (i.e. a consensus statement, and a pair of supporting and refuting in-context examples).



§ EVALUATION



Following the successful bootstrapping of through LLMs, we now further fine-tune the FLAN-T5 model to improve upon the bootstrapping performance and build a robust model for the task.
First, we discuss the setup of this fine-tuning procedure and the process of augmenting an argument-mining dataset for the task of .
Next, we comprehensively evaluate the performance of this fine-tuned model against the prior bootstrapped LLM and various other baselines for the downstream task of stance detection in all three of our evaluation datasets.
We then assess how the runtime and detection performance of CTD varies with increasing model size.
Finally, we integrate our approach with the state-of-the-art soft moderation pipeline Lambretta <cit.>, showing that it can improve its ability to limit contextual false positives.



 §.§ Fine tuning FLAN-T5 for 



The bootstrapping experiments from Section <ref> suggest that can be a reliable approach to identify if a social media post is supporting or refuting a claim, enabling better content moderation.
While LLMs bootstrapped on this task already perform better than multiple existing baselines by leveraging in-context learning, we can improve them further by fine-tuning LLMs that are specifically dedicated to the task of .
Previous work showed that even with little training data fine-tuned LLMs perform better on tasks <cit.>.
Additionally, fine-tuning can potentially allow the LLM to build a more specialized model representation associated with the task, enabling the model to better generalize and also be adversarially robust <cit.>.
Previous work showed that the performance of LLMs can be heavily affected by the components in the prompt, in our case the triplets used by .
For this reason, we aim to use the large number of triplet combinations available in our fine-tuning dataset, to learn a model that is decoupled from the phrasing of the prompt, and more focused and consistent on the task itself.

There are two different approaches to fine-tuning LLMs: i) task-adaptive fine-tuning, and ii) behavioral fine-tuning.
Fine-tuning a LLM directly on the downstream task is called task-adaptive fine-tuning <cit.>.
Behavioral fine-tuning aims to teach a model to learn higher-level language representation by fine-tuning on a task that is behaviorally similar, or close to the downstream task, rather than fine-tuning on the downstream task itself.
This approach is also called intermediate-task fine-tuning, and it works best with tasks that require high-level inference and reasoning capabilities <cit.>, like . 
Thus, we argue that instead of fine-tuning a LLM for the task of stance detection (task-adaptive fine-tuning), which would suffer from similar limitations as BERT fined-tuned on the stance detection task discussed in Section <ref>, a LLM fine-tuned for textual deviation on a large number of claims will be able to reliably perform stance detection on general claims. 
Additionally, behavioral fine-tuning on textual deviation tasks solves the problem of the amount of data available for fine-tuning, as the augmentation of argument mining datasets for textual deviation allows us to generate a dataset that is orders of magnitude larger than any dataset available for specific stance detection topics.
This way, our fine-tuned model is learning what deviation or affirmation to any piece of anchor claim looks like without relying on target-specific or claim-specific features, further helping us to achieve the need for claim invariancy (R2) (see Section <ref>).

We use the PERSPECTRUM dataset to fine-tune LLMs for the task.
This dataset contains both claims and multiple examples of evidences called perspectives either agreeing with the claims or disagreeing with them, as required by .
We augment this dataset to create a large-scale one (expanding up to millions of elements) containing the triplets for the textual deviation task.
An example of a claim sampled from the PERSPECTRUM dataset alongside two randomly sampled supporting and refuting perspectives to the claim is presented in Figure <ref>.
This particular claim contains 13 supporting perspectives and 22 refuting perspectives in total.
With the claim being used as the anchor (consensus) statement, and the contrasting set of refuting and supporting perspectives, we can generate a combination of 9,438 triplets for this claim alone.
This way, the PERSPECTRUM dataset containing 907 claims yields a total of 3,311,548 triplets that we use for fine-tuning a FLAN-T5 model for the task. 

It is to note that the claims present in this dataset are argumentative sentences used for debates and unrelated to the task of fact-checking or misinformation.
We argue that the normalized argumentative structures used to fine-tune the model for will help the model to better capture the intricacies of textual deviation from a canonical perspective, allowing to generalize well and achieve claim invariancy (R2) when applied in the wild. 
As we will show later in Section <ref>, the representations learned through these triplets built from normalized argumentative structures indeed generalize very well in out-of-domain data on Reddit posts and tweets, confirming that higher-level language representations are learned via behavioral fine-tuning.





We fine-tune the FLAN-T5 model using a Parameter Efficient Fine-tuning (PEFT) technique known as LoRA (Low-Rank domain adaptation) <cit.>.
LLMs like FLAN-T5 contain billions of parameters (11B) and fine-tuning the entire network is not feasible even under expensive GPU memory configurations.
LoRA works by freezing the model weights of LLMs and injecting trainable rank decomposition matrices into each layer of the transformers, greatly reducing the number of trainable parameters for downstream tasks while achieving comparable or even better performance <cit.>.
In our case, LoRA reduces the number of trainable parameters of the 11B FLAN-T5 LLM by 84% (from 11B to 18.8M).

Then, using the PEFT configuration of LoRA, we fine-tune the FLAN-T5 model on the augmented fine-tuning dataset.
To make the task more manageable, we randomly sample 4 contrastive examples per claim to fine-tune the model, resulting in a dataset size of 36,000 triplets from 905 claims.
We divide the fine-tuning dataset into 85-15 training and validation splits and fine-tune the FLAN-T5 model for 5 epochs until the validation loss stops decreasing further.
Finally, we test this fine-tuned FLAN model on all three of our evaluation datasets (one coming from Reddit and two from Twitter).
The results of this experiment are shown in Table <ref>.
As it can be seen, the fine-tuned model performs well, reporting F1-scores between 0.84 and 0.90 on the different datasets.




 §.§ Comparison with existing baselines



We further evaluate the fine-tuned FLAN-T5 model against three types of baselines: task fine-tuned BERT, multi-task deep neural networks, and behaviorally fine-tuned baselines.

Task Fine-tuned BERT. This is a supervised learning setup where the BERT model is directly trained on a fraction of the evaluation dataset in a five-fold cross-validation setup.
The evaluation configuration for the Climate Skepticism and COVID-CQ datasets has already been discussed in Section <ref>.
It is to note that the third evaluation dataset we use, Stanceosaurus contains a large number of claims, and only a few tweets (in the order of 100) discussing a single claim.
Fine-tuning a BERT-based model for classification for each claim with a limited number of tweets per claim is not possible, thus we fine-tune a BERT-based model for textual entailment instead (identical to NLI model trained on COVID-Fact in Section <ref>).
We refer to this setup as “Task-BERT.”

Multi-Task Deep Neural Networks (MT-DNN) based stance detection. 
A promising solution to DNN methods generalizing poorly outside their training domain (as demonstrated in Section <ref>) is a framework known as Multi-Task Deep Neural Network (MT-DNN) <cit.>, where a variety of datasets from different domains are used to train a model to increase its robustness.
This has helped in learning powerful higher-level representations across multiple Natural Language Understanding (NLU) tasks, and obtain state-of-the-art results in them.
The MT-DNN framework has been adapted for stance detection by Schiller et al. <cit.>, reporting improvement in stance detection performance.
We use the best-performing MT-DNN model available in their work that is fine-tuned across 10 different datasets as a baseline.

Behaviorally fine-tuned baselines. The concept of using contrastive markers and an anchor statement proposed in our work is very closely associated with a framework in Deep Learning known as Triplet learning <cit.>, which has been successfully applied in computer vision tasks <cit.> and NLP tasks <cit.>.
The fine-tuned BERT model we evaluated and compared on sequence classification (in Section <ref>) and sentence-pair classification (in Section <ref>) are pre-trained with the objective of Masked Language Modeling (MLM), and do not consider any type of contrastive loss in their design.
However, a recent framework known as POLITICS (Pretraining Objective Leveraging Inter-article Triplet-loss using Ideological Content and Story) <cit.> leverages the concept of Triplet Loss <cit.> to pre-train a language model.
This model modifies BERT to be trained on a large corpus of news articles discussing the same story, but from different ideologies and uses triplet loss to capture the ideological (dis)similarity among articles on the same story <cit.>.
Furthermore, POLITICS has been fine-tuned for downstream tasks like stance detection, showing improved performance on various datasets over conventional BERT models.
We therefore use POLITICS as an additional baseline to compare against .

POLITICS needs to be fine-tuned on downstream tasks to be useful.
To this end, we fine-tune the POLITICS model on the same dataset we fine-tuned the FLAN-T5 model on (PERSPECTRUM), adopting Natural Language Inference (NLI) as the downstream task setup.
Similar to how we extracted triplets from PERSPECTRUM in Section <ref>, we generated combinations of around 11K pairs of claims and perspectives for fine-tuning the POLITICS model.
The process of fine-tuning the POLITICS model closely mirrors that of the FLAN-T5 model, except for the PEFT configuration, which was not necessary for POLITICS.








Results. 
The summary of evaluation results of the fine-tuned FLAN model and other methods is presented in Table <ref>.
We can observe that the fine-tuned FLAN model for performs consistently better than all other baselines on all evaluation datasets, outperforming not only Task-BERT and other baselines but also the results from the bootstrapped . 
First, this confirms our hypothesis that the zero-shot learning capabilities of LLMs can be further bolstered by behaviorally fine-tuning a LLM on the task.
Secondly, fine-tuned has better performance than other stance detection approaches such as MT-DNN and POLITICS while being fine-tuned on the same dataset.
This proves that the task formulation of leveraged by the fine-tuned model meaningfully outperforms existing systems for stance detection.
Additionally, it is interesting to note that the FLAN-T5 model, which was fine-tuned on normalized argumentative structures that are domain and platform-independent (as discussed in Section <ref>), consistently outperforms the Task-BERT models that were specifically fine-tuned on claim specific posts from social media text. 
This again affirms our hypothesis that a granular, topic-independent, and platform-independent stance detection model can be created by reframing the problem as a task of , which captures the semantics of stance detection much better on a foundation level.
In summary, these results show that the FLAN-T5 model fine-tuned on the task is a reliable, robust, and scalable stance detection method for the soft moderation of social media text.




 §.§ Model size and performance tradeoff


Finally, we study how the runtime and performance of varies based on the different models available in the FLAN-T5 family.
Following the same methodology discussed in Section <ref>, we fine-tune four smaller models of : i) FLAN-T5-Small, ii) FLAN-T5-Base, iii) FLAN-T5-Large, and iv) FLAN-T5-XL. 
This helps us better understand the scaling properties of the task.
Moreover, researchers and practitioners can use this to guide them in appropriate model selection based on their resource constraints and runtime requirements.
Table <ref> shows that the performance of increases linearly as the model size increases along with the tradeoff on runtime.
On the other hand, fine-tuned on FLAN-T5-XL can be a viable option for practitioners as the performance dropoff from the best model (i.e. FLAN-T5-XXL) is relatively minor compared to the substantial reduction in model size (11B to 3B).




 §.§ Integrating into Lambretta



At this point, we have experimentally validated that a fine-tuned FLAN model on the task has better performance than existing supervised baselines and LLMs bootstrapped for the task, making it the state-of-the-art approach in claim-based stance detection.
Our motivation for designing , however, is to enable soft moderation approaches to get rid of contextual false positives, allowing platforms to deploy more effective warnings that are only applied to content that is supporting a certain false claim.
The state-of-the-art soft moderation tool Lambretta <cit.> is unable to discern this contextual information, and while it performs well in discarding candidate posts that are irrelevant to a given claim, it still flags a large fraction of posts that refute false claims as candidates for moderation: looking at the results reported by Lambretta on three false claims related to the 2020 US Presidential Election (see Table <ref>), we find that 20% of the candidates flagged by the system are contextual false positives.



To evaluate how much our approach can aid existing soft moderation approaches to incorporate contextual knowledge into their decisions, we add as a downstream component to Lambretta <cit.>, and use its stance detection capabilities to further refine the posts that it flags as candidates for soft moderation, filtering out posts that have a debunking (supporting) stance.
Our goal is to reduce contextual false positives while keeping false negatives at a minimum.
Our envisioned architecture is showed in Figure <ref>.
It is to note that while we evaluate the approach on Lambretta, can be added as a post-retrieval filter in any content moderation system after posts relevant to a claim or event for flagging are retrieved.

Setup. 
We use the annotated dataset of election denial tweets discussed in Table <ref> for this purpose.
These tweets were flagged by Lambretta as part of the evaluation in the original paper <cit.>.
We can observe that the class of stance in the tweets is heavily skewed towards spreading the misinformation (Refuting the fact-checks), and contains a smaller proportion of tweets that are debunking the misinformation (Supporting the fact-checks).
As discussed, Lambretta has no contextual understanding of its candidates for soft moderation, and when evaluating for contextual false positives, we find that the system reports a 20% False Detection Rate on average, with an F1 score on the three claims that ranges between 0.88 and 0.89 depending on the claim (see Table <ref>).
Note that the False Negative Rate for the Lambretta baseline is 0 since the system does not perform any contextual filtering on the moderation candidates, and all retrieved tweets are considered matches for moderation.



To evaluate on the candidate tweets flagged by Lambretta, we follow the same methodology discussed in Section <ref>, curating a triplet of consensus statement, supporting evidence, and refuting evidence for each of the three election denial claims being evaluated, as showed in Figure <ref>.
Note that in deployment, the only input that platform moderators need to use for our system is the triplet structure pertaining to a specific social media claim that they wish to moderate.
In the case of a single social media post being the starting point, moderators need to first extract the claim contained in the post, which can be done by leveraging Lambretta's Claim Extraction component or other claim extraction tools such as OpenIE <cit.>.
We then use the fine-tuned FLAN-T5 model presented in Section <ref> with the three curated triplets to identify the stance on the tweets discussing the three claims.
Note that for this experiment is not fine-tuned on the tweets flagged by Lambretta, nor on any tweet discussing election fraud claims, further showcasing that our approach is context agnostic and generalizes across different topics.

Results. 
The summary of the evaluation is presented in Table <ref>.
As it can be seen, integrating with Lambretta largely reduces the rate of contextual false positives, bringing down the average false detection rate by an order of magnitude from 20% to 2.1%.
For the “Wisconsin voter turnout above 90%,” the false detection rate after applying our approach is actually zero.
At the same time, the false negative rate remains small, being 1.8% on average.
This translates in F1 scores between 0.98 and 0.96, showing an improvement of about 10% over the baseline, showing that can be effectively used to improve soft moderation systems for social media.








§ RELATED WORK

We review related work on stance detection, LLMs for text classification, and use of stance detection for integrated fact-checking.

Stance detection.
Stance detection is a foundational technique for various natural language understanding tasks <cit.> and has been used under various settings like argument mining <cit.>, rumor detection <cit.>, and fake news detection <cit.>.
The majority of the existing work on stance detection focuses on topic or target-specific stance detection, where they aim to detect the stance of a text towards topic such as “gun rights,” “atheism” <cit.> and public figures like “Donald Trump” or “Hillary Clinton.” <cit.>
Only a few works focus on claim-based stance detection <cit.>, which is the focus of this work.
Works on claim-based stance detection are mostly geared towards checking facts on formal text like Wikipedia <cit.> or scientific knowledge bases <cit.>.
Multiple works have used textual entailment for verifying claims on Wikipedia <cit.>, scientific knowledge <cit.>, and climate change conversations <cit.>.
There are a few works on claim-based stance detection in multi-lingual settings like Arabic <cit.>, and Crotian <cit.>.

LLMs for text classification.
With the explosion of Large Language Models and rapid development of powerful LLM models like Open AI's GPT-4 <cit.>, Google's BARD <cit.>, Anthropic's Claude <cit.> etc., there has been a paradigm shift in approaching text classification problems.
The massive amount of internet-scale data that LLMs see during their pretraining has been harnessed to fine-tune LLM models, producing state-of-the-art results in challenging benchmarks in Natural Language Understanding (NLU) <cit.>.
In addition to the zero-shot capabilities of LLMs, researchers have also used strategies like in-context learning to improve their performance <cit.>, making weakly supervised learning scalable and effective.
This has made bootstrapping LLMs for custom text classification much easier, a process that we demonstrated successfully in our work.
This way, LLMs have presented themselves as a viable alternative for multiple Natural Language Processing tasks such as classification, and summarization, while relaxing the constraint of task-specific training data needed for conventional NLP methods.

Stance detection as a fact-checking component.
Prior works have used stance detection as a component of automated fact-checking pipelines.
Works in this domain use stance prediction as one of the many components of their fact-checking pipeline, alongside other components and metadata such as user features and features of conversation threads. 
Zubiaga et al. <cit.> incorporated stance classification to detect the stance of tweets in a four-step rumor verification pipeline.
Dungs et al. <cit.> leveraged the stance of conversational threads to predict veracity of rumors.
CredEye, a system proposed by Popat et al. <cit.> used stance detection to automatically predict the credibility of textual claims retrieved from the Web.
Another tool, developed by Nguyen et al. to assist fact-checkers <cit.>, uses stance predicted from multiple articles alongside other attributes such as the reputation of the news sources to assess a claim's veracity.
Similarly, FAKTA <cit.>, a system for end-to-end fact-checking of claims, uses a stance detection model trained on FEVER setting alongside linguistic metadata for automatic fact-checking.

Remarks. In this paper, we showed that outperforms existing stance detection mechanisms and that it could be easily integrated into existing moderation and analysis pipelines that make use of stance detection.



§ DISCUSSION AND CONCLUSION

In this paper, we presented , a new framework for detecting stance in social media posts discussing misleading claims.
We tested the ability of existing stance detection systems to aid content moderation on social media and identified three major limitations of these systems in the context of applying them for content moderation.
Motivated by the shortcomings, we developed the framework as an unsupervised, platform, and topic-agnostic solution.
By experimenting on datasets from two social network platforms (Twitter and Reddit) and multiple topics (e.g., politics, health, climate) we showed that our method consistently outperforms both supervised and unsupervised baselines. 
Most importantly, we demonstrated that can be easily integrated into an end-to-end content moderation system, improving the performance of the state-of-the-art soft moderation system Lambretta <cit.> by reducing its contextual false positives tenfold.

We believe that will both serve as a new paradigm for unsupervised claim-based stance detection and will be a valuable tool for researchers and online platforms aiming to improve their existing content moderation systems.
Enabling context-aware soft moderation systems can go a long way in making our information ecosystems healthier, minimizing warning fatigue, and increasing the intended effectiveness of warning labels.
We now discuss the ethical concerns of our work, design implications and limitations of our approach, and avenues for future work.

Ethics. 
All datasets used in this work were either publicly released by other researchers or were collected using publicly available APIs and following those API's terms of service.
This work is not considered human subjects research by our institution, since we do not interact with humans and do not collect any private information.
Nonetheless, we adhere to ethical standards by removing any personally identifying information when reporting examples of social media posts in the paper.
While we advocate that our approach should be used to benefit society, following the respect for public interest and beneficence principles of the Menlo report <cit.>, could be misused by malicious parties.
Potential adversarial misuse includes biased platform providers using to identify and downrank dissident users, or state-actors applying to amplify false narratives of interest or identify expert accounts that are correcting/debunking false narratives to be silenced.
While these threats are real, automatically generating and posting content on social media at a large scale produces other artifacts that can be identified by alternative approaches <cit.>.

Design implications.
We envision to be applied as a post-retrieval filtering tool for content moderation systems on social media after topically relevant candidate posts for a claim are retrieved, as illustrated in Figure <ref>.
Since the approach of is designed for the task of claim-specific stance detection, is mostly intended for claim-specific content moderation systems like Lambretta <cit.>.
The process of integrating as a tool to existing soft moderation systems is seamless, as we demonstrated in Section <ref>.
For each claim a platform moderator wishes to apply warning labels to, the only input from their end is to craft the triplet consisting of the consensus statement, a refuting evidence and a supporting evidence.
The triplet can then be used with the evaluation prompt in Figure <ref> alongside the fine-tuned FLAN-T5 model for inference.
It is to note that one of the major advantages of using is that it does not need any further fine-tuning for new claims and new platforms.
As demonstrated in the example triplets throughout our work (i.e. Figure <ref> and Figure <ref>), the consensus statement for a claim can be formulated with the most succinct fact-check or scientific consensus about the misleading claim.
Similarly, the contrastive markers can be simple statements that are positive affirmation and negative reframing of the same consensus statement.
The final model was fine-tuned on normalized pieces of argumentative structures, and we expect it to be robust enough to handle different quality or phrasings of triplet semantic structure expected of a triplet.


Limitations.
There are some limitations that come with the task setup of for stance detection.
First, we expect that any claim a platform aims to contextually moderate has been determined to be false, and has an accompanying fact-check statement associated with it.
This requirement of a corresponding fact-check statement used to build the consensus statement in a triplet poses a practical challenge in cases of quickly emerging false claims or novel skeptical narratives, for which a consensus about the claim has not yet been established.
For these cases, moderators could potentially use high-quality crowdsourced truth statements <cit.>, or resort to applying previously proposed soft moderation techniques that do not take stance into account (e.g., Lambretta).
The formulation of can only handle posts that support or refute a misleading claim, while social media posts discussing misinformation and rumors might also contain posts that are of “querying” or “commenting” nature <cit.>.
However, for the purpose of soft moderation where the objective is to apply warning labels to posts that are spreading the misleading claim, we argue that finer-grained distinction within the nature of “support” of a claim might not be necessary.
Upon manual inspection, we find that the majority of the misclassifications by happen on posts that are sarcastic and satirical about the misleading claim being discussed, which are often misclassified as refuting the claim.
Identifying satire and sarcasm is a challenging NLP task <cit.>, and future work can explore fine-tuning models to handle more nuanced cases of stance occurring on social media text.



Future Work.
In the future, we plan to extend on evaluating claim-based stance detection in multi-lingual settings.
As LLMs become more powerful beyond the English Language, we can expect their learning capabilities to improve on multiple languages <cit.>.
We will also explore strategies to reduce the manual costs of curating the triplet structure needed for by automatically matching the best set of fact checks and consensus statements for misleading claims spreading in the wild, and generating the best set of triplets automatically.
The ClaimReview structured markup introduced by Google <cit.> is a promising avenue for this direction, and we will explore multiple ways of semantically matching a misleading claim with ClaimReview markups, and generating the piece of contrastive markers from the fact-check document.

Acknowledgments. We would like to thank the anonymous reviewers for their feedback.
This work was funded by the Institute for Global Sustainability and the Rafik B. Hariri Institute for Computing and Computational Science & Engineering at Boston University under the Data and Misinformation in an Era of Sustainability and Climate Change Crises project, and by the NSF under grants CNS-1942610, CNS-2114407, CNS-2247868, and DEB-2200052.

plain




