Paper Title: Contrastive Graph Representation Learning with Adversarial Cross-view Reconstruction and Information Bottleneck.

















































address2]Yuntao Shou^a ,
shouyuntao@stu.xjtu.edu.cn

address3]Haozhi Lan
haozhilan1@gmail.com

address2]Xiangyong Cao^a ,cor1
caoxiangyong@mail.xjtu.edu.cn
[cor1]Corresponding author

[address1]School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China

[address2]Ministry of Education Key Laboratory for Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an, China

[address2]School of Artificial Intelligence, Xi'an Jiaotong University





§ ABSTRACT 
Graph Neural Networks (GNNs) have received extensive research attention due to their powerful information aggregation capabilities. Despite the success of GNNs, most of them suffer from the popularity bias issue in a graph caused by a small number of popular categories. Additionally, real graph datasets always contain incorrect node labels, which hinders GNNs from learning effective node representations. Graph contrastive learning (GCL) has been shown to be effective in solving the above problems for node classification tasks. Most existing GCL methods are implemented by randomly removing edges and nodes to create multiple contrasting views, and then maximizing the mutual information (MI) between these contrasting views to improve the node feature representation. 
However, maximizing the mutual information between multiple contrasting views may lead the model to learn some redundant information irrelevant to the node classification task. To tackle this issue, we propose an effective Contrastive Graph Representation Learning with Adversarial Cross-view
Reconstruction and Information Bottleneck (CGRL) for node classification, which can adaptively learn to mask the nodes and edges in the graph to obtain the optimal graph structure representation. Furthermore, we innovatively introduce the information bottleneck theory into GCLs to remove redundant information in multiple contrasting views while retaining as much information as possible about node classification. Moreover, we add noise perturbations to the original views and reconstruct the augmented views by constructing adversarial views to improve the robustness of node feature representation. Extensive experiments on real-world public datasets demonstrate that our method significantly outperforms existing state-of-the-art algorithms.




Graph Neural Networks, Contrastive Learning, Mutual Information, Information Bottleneck, Adversarial Learning







§ INTRODUTION

Graph Neural Networks (GNNs) have attracted extensive attention from researchers due to the increase in large amounts of real-world graph-structured data <cit.>. Meanwhile, GNNs are widely used in intelligent recommender systems, and social media fields because they provide a practical way to aggregate high-order neighbor information <cit.>.




Although GNNs have achieved reliable performance on node classification tasks, we argue that most of the node classification models based on GNNs suffer from the following two problems. i) Popularity Bias. As shown in Fig. <ref>, different categories of papers have different numbers, and the number of citations of papers also varies, and this unbalanced learning can lead to the popularity bias problem in GNNs learning. In most node classification tasks, the categories and degrees of nodes follow a long-tail distribution, which means that popular categories have many papers, while most papers have few citations. In other words, most of the nodes have less interaction, which hinders the information update of the nodes. The above-mentioned popularity bias problem cause GNNs to tend to learn node representations that are popular and have many interactions, which hinders the representation learning of GNNs. ii) Noise Interference. There may be miscitations in the citation process of papers (i.e., there is a citation relationship between two unrelated papers in different fields), which leads to noise in the information contained in the data. Studies have shown that the feature extraction ability of GCN is closely related to the quality of the input graph, which means that the input image with noise may cause the model to learn poor solutions. In response to the above problems, existing graph contrastive learning (GCL) methods <cit.>, <cit.>, <cit.>, <cit.>, <cit.> propose an effective solution mechanism to alleviate popular bias and improve the robustness of the GCN model.

Nonetheless, the above-mentioned methods suffer from two limitations. 1) Most GCL methods perform data augmentation to optimize the graph structure by randomly masking nodes or perturbing edges. However, the strategy of randomly masking nodes and edge perturbations is too random, which may cause serious damage to the semantic information of the graph structure. For example, in the functional prediction of molecular structural properties, if edges are randomly perturbed, the structural properties of the molecule will change greatly. In addition, the interpretability that can alleviate popular paranoia and improve model robustness through the above methods is relatively poor. 2) The purpose of existing GCL methods <cit.>, <cit.>, <cit.> to generate multiple views through data augmentation is to maximize the mutual information between views, which may cause the model to capture task-irrelevant feature information. Inspired by the information bottleneck theory <cit.>, we believe that a good GCL method should reduce as much redundant information as possible while retaining as much task-related information as possible.

To tackle the aforementioned issues, we propose a novel method called Contrastive Graph Representation Learning with Adversarial Cross-view
Reconstruction and Information Bottleneck (CGRL) for node classification. CGRL consists of two key components: i.e., adaptive automatic generation of graph-augmented views and graph contrastive learning via information bottlenecks.

First, this paper designs an automatic graph augmentation that adaptively learns node masks and edge perturbations to optimize the original graph into relevant views. In addition, SCGCL employs a joint training strategy to train an adaptive learnable view generator and node classifier in an end-to-end manner, thereby generating augmented views with structural heterogeneity but semantic similarity. As a result, the generated augmented view can undersample the popular nodes in the original graph while retaining the majority of isolated nodes to alleviate the model's popularity bias problem. Intuitively, random masking nodes or perturbed edges do not consider the distribution probabilities and neighborhood information of different types of nodes in the original graph, but dropout them randomly. However, GCN based on message passing is difficult to reconstruct the information of isolated nodes and it is easier to optimize the semantic information of popular nodes. Therefore, the model may achieve better classification results on popular nodes and poor classification results on isolated nodes. The method CGRL proposed in this paper takes the augmented views of debias information and inputs them into GCN for node classification, which improves the model's ability to resist popular bias.

Second, we integrate multiple views that are semantically similar and contain complementary information into a shared feature space for compact representation, which can improve the robustness of CGRL. The intuition behind is that when different views contain complementary semantic information, the model can obtain more prior knowledge to improve the performance of node classification tasks <cit.>. However, we argue that maximizing mutual information between different views forces the model to learn redundant information that is irrelevant to downstream tasks. Inspired by the information bottleneck (IB) theory <cit.>, it obtains optimal solutions by maximizing label information relevant to downstream tasks and minimizing mutual information between different views. Based on the IB strategy, an automatic graph augmenters learns to generate augmented views that remove noise information and contain semantically similar and complementary views. In addition, when calculating the contrastive loss, we not only use the node feature representations of the two augmented views, but also introduce the node feature representations of the original view perturbed in an adversarial manner as a third view. This additional adversarial view introduces perturbations that force the model to not only accurately distinguish the semantic features of the augmented views, but also to maintain an understanding of the semantic integrity of the original graph in the face of perturbations. Through multi-view adversarial reconstruction, we further improve the robustness of the feature representations.

Compared with the previous methods, the contributions of this paper method are summarized as:

Firstly, we propose a Contrastive Graph Representation Learning with Adversarial Cross-view Reconstruction and Information Bottleneck (CGRL) for node classification. The CGRL method can alleviate the popularity bias and interaction noises problem of the existing GNNs in aggregating neighbor node information.

Secondly, The CGRL approach provides a learnable approach to adaptively mask nodes and edges for multiple graph contrastive views in an end-to-end manner. In addition, redundant information irrelevant to node classification is discarded by innovatively introducing information bottleneck theory into multi-view graph contrastive learning.

Thirdly, we introduce a cross-view adversarial reconstruction strategy to further improve the robustness of node feature representation. 

Finally, extensive experiments also show that the proposed CGRL method outperforms the state-of-the-art methods on seven real-world publicly available datasets. 



§ RELATED WORK


Graph Representation Learning   Early work <cit.> have made great progress in representation learning tasks (e.g., node classification, entity alignment, and link prediction, etc). Specifically, GNNs on graph representation learning usually follow the information aggregation mechanism to update the feature representation of nodes, i.e., stimulating connected neighbor nodes to have similar semantic information. Inspired by GNNs, several works (e.g., AROPE <cit.>, DeepWalk <cit.>, and GraphSAGE <cit.>, etc) on graph representation learning usually follows the label propagation mechanism to update the feature representation of nodes, i.e., stimulating connected neighbor nodes to have similar labels. In recent years, GNNs (e.g., GraphSMOTE <cit.>, Nodeformer <cit.>, DisGNN <cit.>, and GraphFL <cit.>, etc) have applied GNN algorithms to learn discriminative latent representations on node classification tasks <cit.>.


Contrastive Learning   Contrastive learning (CL) <cit.>, which is originally widely used in computer vision to obtain better image feature representations, has received extensive research attention in graph learning. Specifically, graph contrastive learning (GCL) learns discriminative node representations by maximizing mutual information (MI) between multiple graph views. For instance, DGI <cit.> learns more discriminative node representations by contrasting node embeddings of local and global graph views. GIC <cit.> maximizes the MI between node clusters with high similarity to make full use of coarse-grained and fine-grained information between nodes. CMC <cit.> maximizes MI by contrasting feature representations from different views. GMI <cit.> estimates and maximizes the MI between the input graph and the feature representation from two aspects of node features and network topology. Some recent self-supervised graph learning (SGLs) methods (e.g., MGAE <cit.>, GraphMAE <cit.>, and GAE <cit.>) generate multiple graph views of nodes and edges and force the consistency between different graph views. On the one hand, all these methods generate multiple contrasting graph views by randomly masking nodes or edges, which may cause some important structural and semantic information to be lost. On the other hand, these methods maximize mutual information between different graph views, which may force the model to learn some task-independent semantic information. To sum up, existing self-supervised graph learning for node classification suffer from insufficient information utilization, i.e., information associated with the label.



Information-Bottleneck Representation Learning   The information bottleneck (IB) theory <cit.> argues that if the feature representation learned by the model from the input data discards information that is not useful for the given task while retaining as much as possible the information relevant to the given task, it will increase the generalization of downstream tasks. Formally, IB needs to construct multiple views for feature representation learning. Motivated by the effectiveness of the IB, several works have considered transferring information bottleneck theory to graph representation learning tasks. For instance, MIB <cit.> designs an unsupervised multi-view method and uses the information bottleneck theory to minimize the representation of redundant information.  DeepIB <cit.> reduces redundant information irrelevant to a given task by minimizing mutual information between multiple views and input data. CMIB <cit.> combines information bottleneck theory to capture the complementarity of intrinsic information between different views and balance the consistency of multi-view latent representations. Nevertheless, almost all the above-mentioned methods aim to obtain a discriminative graph view to replace the original input graph, which may cause some semantic information and topological information of the original graph to be lost. In conclusion, existing information bottleneck methods for node classification suffer from insufficient information utilization, i.e., input graph information and multi-view information.






§ PRELIMINARIES


Notations.
Suppose 𝒢={𝒱, ℰ,ℛ,𝒲} represents a graph where 𝒱={v_1, v_2, …, v_M} is the nodes set, M is the number of nodes, ℰ⊆𝒱×𝒱 represent the edges set, r_ij(r_ij∈ℰ) represents the connection relationship between node i and node j, and ω_ij (ω_ij∈𝒲,0≤ω_ij≤ 1) the weight of the edge ℰ_ij. The feature matrix and degree matrix of nodes are expressed as X= {x_i}_i=1^M amd A={a_ij}∈{0, 1}^M × M, respectively, where x_i represent the features of the node v_i, and if (v_i, v_j) ∈ℰ then a_ij=1 otherwise a_ij=0. 𝒢 = G(x) is regarded as the process of graph processing.

GCN Processing. For input features X ∈ℝ^M × D, a graph 𝒢 = F=G(X) is constructed based on the features X. A GCN layer is utilized to update features representations between nodes by aggregating information from their neighbor nodes. Specially, GCN operates as follows:

    𝒢^'    =F(𝒢, 𝒲) 
        = Update(Aggregate(𝒢, W_agg), W_update) 
        = W_update(∑_r ∈ℛ∑_j ∈ N_i^r1/|𝒩_i^r |(ω_ij W_agg𝒢+ω_ii W_agg𝒢))

where 𝒩_i^r is the set of neighbor nodes of node i under the edge relationship r ∈ℛ, and W_agg and W_update represent the learnable weights of the nodes in aggregating surrounding neighbor nodes information and updating the aggregated features, respectively.

In addition, we also introduce a multi-head attention mechanism in the GNN layer to capture node feature information and topology information in a more fine-grained manner. The feature vectors x_i^” after being aggregated and updated is divided into N heads, i.e., h^1, h^2, …, h^N, and each head is assigned a learnable parameter. Therefore, the feature vectors x is finally updated as follows:

    𝐱_i^'=[h^1 W_update^1, h^2 W_update^2, …, h^N W_update^N],



Information Bottleneck (IB). IB is an information theory-based strategy that describes the information in the data that is relevant to downstream tasks. IB argues that if the obtained feature representation excludes semantic information in the original input that is irrelevant to a given downstream task, it improves the robustness of the model. Specifically, for a given input data x, with associated label information is y, using the IB strategy for model optimization can obtain a compact feature representation z. The optimization goals of IB are as follows:

    max _𝐙 I(𝐘, 𝐙, θ)-β I(𝐗, 𝐙, θ),

where β is an adjustment factor, θ is a learnable parameter.

Combining I(Y,Z) and mutual information theory in Eq. <ref>, we can get:

    I(𝐘, 𝐙)     =∫ d 𝐲 d 𝐳 p(𝐲, 𝐳) logp(𝐲, 𝐳)/p(𝐲) p(𝐳)
        =∫ d 𝐲 d 𝐳 p(𝐲, 𝐳) logp(𝐲|𝐳)/p(𝐲).


However, directly calculating p(y|z) is quite difficult. Inspired by <cit.>, we use q(y|z) as the variational approximation of p(y|z). 

Since the KL divergence is always greater than or equal to 0, we can get:

    𝐊 𝐋[p(𝐲|𝐳), q(𝐲|𝐳)] ≥ 0 ⇒∫ d 𝐲 p(𝐲|𝐳) logp(𝐲|𝐳)/q(𝐲|𝐳)≥ 0 
    ⇒   ∫ d 𝐲 p(𝐲|𝐳) log p(𝐲|𝐳) ≥∫ d 𝐲 p(𝐲|𝐳) log q(𝐲|𝐳).


Therefore we can know:

    I(𝐘, 𝐙)    ≥∫ d 𝐲 d 𝐳 p(𝐲, 𝐳) logq(𝐲|𝐳)/p(𝐲)
        =∫ d 𝐲 d 𝐳 p(𝐲, 𝐳) log q(𝐲|𝐳)+H(𝐲) 
       ≥∫ d 𝐲 d 𝐳 p(𝐲, 𝐳) log q(𝐲|𝐳) 
        =∫ d 𝐲 p(𝐲) ∫ d 𝐳 p(𝐳|𝐲) log q(𝐲|𝐳) .


For I(X,Z) in Eq. <ref>, we can get:

    I(𝐙, 𝐗)   = 	∫ d 𝐳 d 𝐱 p(𝐱, 𝐳) logp(𝐳 , 𝐱)/p(𝐳), p(𝐱)
       = ∫ d 𝐳 d 𝐱 p(𝐱, 𝐳) logp(𝐳|𝐱)/p(𝐳).


Similarly, it is quite difficult to calculate p(z). We use r(z) as the variational approximation to estimate p(z). Since 𝐊 𝐋[p(𝐳), r(𝐳)] ≥ 0 ⇒∫ d 𝐳 p(𝐳)log p(𝐳) ≥∫ d 𝐳 p(𝐳)log r(𝐳), we can get an upper bound:

    [ I(𝐙, 𝐗) ≤∫ d 𝐱 d 𝐳 p(𝐱) p(𝐳|𝐱) logp(𝐳|𝐱)/r(𝐳);     =∫ d 𝐱 p(𝐱) ∫ d 𝐳 p(𝐳|𝐱) logp(𝐳|𝐱)/r(𝐳) . ]


Combining the above analysis and inequalities, we can obtain the lower bound of the information bottleneck theory:

    I(𝐘, 𝐙)-∑_v=1^V β I(𝐙, 𝐗) 
       ≥∫ d 𝐲 p(𝐲) ∫ d 𝐳 p(𝐳|𝐲) log q(𝐲|𝐲) 
        - β∫ d 𝐱 p(𝐱) ∫ d 𝐳 p(𝐳|𝐱) logp(𝐳|𝐱)/r(𝐳).




§ METHODOLOGY

In this section, we introduce our proposed Contrastive Graph Representation Learning with Adversarial Cross-view Reconstruction and Information Bottleneck (CGRL) method in detail.



 §.§ Automatically Generated Multi-view Augmentation


Adversarial View. By adding perturbed adversarial examples to the original image views, the robustness of the model is significantly enhanced. This improvement can be attributed to a possible explanation that there is still non-predictive redundant information in the information shared between the two augmented views. When adversarial examples are introduced, this redundant information is weakened or eliminated, forcing the model to focus on more important and discriminative features. We define adversarial view as follows:

    𝒢_adv^(l)=𝒢 + δ^*


    δ^*   =δ_∞⩽ϵargmaxℒ_adv(𝒢_E D,𝒢_N D,𝒢+δ) 
       =δ_∞⩽ϵargmaxmax_δ^*[ℒ_CL(𝒢_N D,G+δ^*) .
       .+ℒ_CL(𝒢_E D,G+δ^*)]

where 𝒢_adv^(l) = {𝒱', ℰ',ℛ',𝒲'}, 𝒱={v_1, v_2, …, v_M}, δ is the randomly initialized Gaussian noise, ℒ_CL is the infoNCE loss, ϵ is the radius. Inspired by recent work <cit.>, we add a perturbation δ to the output of the first hidden layer. It has been empirically shown that it can more effectively perturb the intermediate representation of the model than adding perturbations to the initial node features, allowing the model to learn and predict in more complex environments.

Node-Masking View.
As shown in Fig. 1, both the category of nodes and the degree of nodes in citation data show data imbalance, which hinders GCN from learning the feature representation of minority class nodes. Therefore, we perform automatic learnable node masking before each information aggregation and feature update of GCN to alleviate the shielding effect of influential nodes on minority class nodes. The node-masking view we created is formulated as follows:

    𝒢_N D^(l)={{v_i' ⊙η_i^(l)| v_i' ∈𝒱'}, ℰ', ℛ', 𝒲'}

where η_i^(l)∈{0,1} is sampled from a parameterized Bernoulli distribution Bern(ω_i^l), and η_i^(l) = 0 represents masking node v_i, η_i^(l) = 1 represents keeping node v_i.

Randomly removing some nodes and their connections in the graph may result in a large loss of minority class node information, thereby affecting the information aggregation of minority class nodes and leading to unsatisfactory classification results. Therefore, instead of directly removing the selected nodes from the graph, we replace the selected nodes by sampling the representation of the local subgraph using a random walk strategy to obtain a local representation of the node.

Edge Perturbation View.
The goal of perturbing edges is to generate an optimized graph structure that filters noisy edges and alleviates the problem of popularity bias. The edge perturbation view is formulated as follows:

    𝒢_E D^(l)={𝒱',{e_i j' ⊙η_i j^(l)| e_i j' ∈ℰ', ℛ', 𝒲'}}

where η_
ij^(l)∈{0,1} is also sampled from a parameterized Bernoulli distribution Bern(ω_ij^l), and η_ij^(l) = 0 represents perturbating edges e_ij, η_i^(l) = 1 represents keeping edge e_ij.

To enable the model to automatically learn whether to mask nodes and perturb edges, we formally define the learnable parameter ω^l_i and ω^l_ij as follows:

    ω_i^(l)=Linear(𝐞_i'^(l)) ;   ω_i j^(l)=Linear([𝐞_i'^(l) ; 𝐞_j'^(l)])


To ensure that the model can automatically optimize and generate augmented multi-views in an end-to-end learning method, we use reparameterization technology <cit.> to convert the discretized η into a continuous function. The formula is defined as follows:

    η_i=exp((log(π_i)+g_i) / τ)/∑_j=1^m exp((log(π_j)+g_j) / τ),    for  i=1, …, m

where g_i=-log(-log(ϵ_i)), ϵ_i ∼ Uniform(0, 1), τ∈ℝ^+ means annealing temperature, τ represents the class probability, and m represents number of categories.

After obtaining the masked node and edge perturbed views, we input them into GCN for feature representation to obtain optimized multi-views. The formula is defined as follows:

    𝐄_N D^(l)=GraphConv(𝐄_N D^(l-1), 𝒢_N D^(l)) 
       𝐄_E D^(l)=GraphConv(𝐄_E D^(l-1), 𝒢_E D^(l))

where GraphConv represents the graph convolution operation, and we choose GAT as our graph encoder. 𝐄_N D and 𝐄_E D represent the node feature representations of node-masking view and edge perturbation view respectively, 𝐆_N D and 𝐆_E D represent node-masking view and edge perturbation view respectively.



 §.§ Contrastive Learning via IB

Although CGRL combines an automatic learnable view augmentation and a node classification process for model optimization, we argue that relying solely on the classification objective does not well guide the node masking and edge perturbation process to create optimal multi-views. Therefore, we follow the information bottleneck strategy <cit.> to retain sufficient semantic information relevant to downstream tasks in the augmented node mask and edge perturbation views. Specifically, unlike traditional CL strategies, we encourage maintaining topological heterogeneity between the augmented view and the original graph while maximizing the information relevant to the node classification task. Based on the above strategy, we can obtain topologically heterogeneous but semantically similar enhanced multi-view representations and effectively remove noise information in the graph. Therefore, the objective in Eq. <ref> is summarized as:

    min_(E, Ẽ)ℒ_CLS + I(𝐄, 𝐄̃) 
        = -∑_i=1^ny_ilog(ŷ) + I(𝐄, 𝐄̃)

where L_CLS represents the cross-entropy loss, I(𝐄, 𝐄̃) represents the mutual information between the original view and the augmented view.

Following <cit.>, minimizing the lower bound of mutual information (i.e., Eq. 9) is equivalent to maximizing the InfoNCE loss <cit.>. Therefore, we adopt negative InfoNCE to optimize the feature representation between the augmented view and the original graph. Specifically, we treat the same node representations in the original graph and the automatically generated view as positive pairs (i.e., {(e_i', ẽ_i')|v_i' ∈𝒱'}), and different node representations as negative pairs (i.e., {(e_i', ẽ_j')|v_i', v_j' ∈𝒱'}, i ≠ j).

    I(𝐄, 𝐄̃)     = ∫ d 𝐲 p(𝐲) ∫ d 𝐳 p(𝐳|𝐲) log q(𝐲|𝐲) 
        - β∫ d 𝐱 p(𝐱) ∫ d 𝐳 p(𝐳|𝐱) logp(𝐳|𝐱)/r(𝐳)
        = ∑_v_i' ∈𝒱'logexp(sim(𝐞_i', 𝐞̃_i') / τ)/∑_v_j ∈𝒱exp(sim(𝐞_i', 𝐞̃_j') / τ).

where s(·) is used to measure the similarity between positive and negative sample pairs.



 §.§ Adversarial Cross-view Reconstruction

To further achieve feature disentanglement, we propose a cross-view reconstruction mechanism. Specifically, we hope that the representation pairs within and across enhanced views can recover the original data. More specifically, we define (𝐄_ND, 𝐄_ED), (𝐄, 𝐄_ED), (𝐄, 𝐄_ED) as a cross-view representation pair, and repeat the reconstruction process on it to predict the original view, aiming to ensure that 𝐄_ND, 𝐄_ED, 𝐄 are optimized to approximately disentangle each other. Intuitively, the reconstruction process is able to separate the information of the shared feature set from the information in the unique feature set between the two enhanced views. We formally define the reconstruction process as:

    ℒ_recon=   1/2N[𝐄-𝐄_ND_2^2+𝐄-𝐄_ED_2^2]

wherer N represents the number of nodes.



 §.§ Model Training

We train the model with the goal of optimizing the node mask view, edge perturbation view, and node classification:

    ℒ   =ℒ_CLS+αℒ_CLS ^N D
    		+(1-αℒ_CLS ^E D)
       +β(I(𝐄, 𝐄_N D)+I(𝐄, 𝐄_E D))
    		+ℒ_recon+λΘ_2^2

where λ and β represent the learning weights of L2 regularization and information bottleneck contrast learning respectively.




§ THEORETIC ANALYSIS
 
Theorem 1. Specifically, we regard the augmented view as 𝒢̃, the noisy view as 𝒢^', and the node label information as Y_CLS. We assume that 𝒢̂ is not related to the classification information Y_CLS. Therefore, the upper bound of I(𝒢̂ ; 𝒢̃) is defined as follows:

    I(𝒢̂ , 𝒢̃) ≤ I(𝒢 , 𝒢̃)-I(Y_CLS , 𝒢̃)


Proof. We assume that the clean graph 𝒢 is defined by 𝒢^' and label information Y, and we can get (𝒢^', Y_CLS) →𝒢→𝒢̃ according to the Markov chain <cit.>. Therefore, we can get:

    I(𝒢 , 𝒢̃)    ≥ I((Y_CLS, 𝒢̂) , 𝒢̃) 
        =I(𝒢̂ , 𝒢̃)+I(Y_CLS , 𝒢̃|𝒢̂) 
        =I(𝒢̂, 𝒢̃)+H(Y_CLS|𝒢̂)-H(Y_CLS|𝒢̂, 𝒢̃)


Because there is no correlation between 𝒢̃ and Y_CLS, H(Y_CLS|𝒢̃) = H(Y_CLS). Furthermore, H(Y_CLS|𝒢̂, 𝒢̃) ≤ H(Y_CLS). Therefore, we simplify the Eq. <ref> as follows:

    I(𝒢 , 𝒢̃)    ≥  I(𝒢̂ , 𝒢̃)+H(Y_CLS)-H(Y_CLS|𝒢̃) 
       =   I(𝒢̂, 𝒢̃)+I(Y_CLS , 𝒢̃)

Therefore, we prove that Eq. <ref> holds. In summary, we provide a theoretical basis to ensure that graph contrastive learning via information bottlenecks can achieve noise invariance by reducing redundant and interfering information in augmented views.

Theorem 2. We optimize the reconstruction by minimizing the entropy H(𝐄|𝐄_ND, 𝐄_ED). Ideally, when H(𝐄|𝐄_ND, 𝐄_ED) - 𝔼_𝐄, 𝐄_ND, 𝐄_ED[log p (𝐄|𝐄_ND, 𝐄_ED)]=0, we achieve the best feature disentanglement. However, in practice, the estimation of conditional probability p(𝐄|𝐄_ND, 𝐄_ED) is very tricky and complicated. Therefore, we use an approximate variational distribution q(𝐄|𝐄_ND, 𝐄_ED) to simplify the calculation and optimization process. We provide a theoretical upper bound on H(𝐄|𝐄_ND, 𝐄_ED) as follows:

    H(𝐄|𝐄_ND, 𝐄_ED)⩽max{𝐄-𝐄_ND_2^2, 𝐄-𝐄_ED_2^2}


Proof. For a given original view 𝐄 and two augmented views 𝐄_ND and 𝐄_ED, we have:

    p(𝐄_ND,𝐄_ED)   =p(𝐄_ND)p(𝐄_ED)
    p(𝐄_ND,𝐄_ED|𝐄)   =p(𝐄_ND|𝐄)p(𝐄_ED|𝐄)


Lemma 1. For three given random variables a, b, c, if p(b, c) = p(b)p(c) and p(b, c| a) = p(b| a)p(c| a), then I(a, b| c) = I(a, b). Based on the definition of mutual information, we deduce:

    I(a;b| c)= 
       =∑_a∑_b∑_cp(a,b,c)logp(a,b,c)p(c)/p(a,c)p(b,c)
       =∑_a∑_b∑_cp(a)p(b,c| a)logp(b,c| a)p(c)/p(c| a)p(b)p(c)
       =∑_a∑_b∑_cp(a)p(b| a)p(c| a)logp(b| a)p(c| a)/p(c| a)p(b)
       =∑_a∑_bp(a)p(b| a)logp(b| a)/p(b)
       =∑_a∑_bp(a,b)logp(b| a)/p(b)
       =I(a;b)


According to Lemma 1, we can derive the theoretical bound of I(𝐄;𝐄_ND,𝐄_ED) as follows:

    I(𝐄;𝐄_ND,𝐄_ED)
    		
        =I(𝐄;𝐄_ND|𝐄_ED)+I(𝐄;𝐄_ND)+I(𝐄;𝐄_ED) 
       ⩾ I (𝐄_ND, 𝐄_ED;𝐄) 
       =I(𝐄_ND;𝐄)+I (𝐄_ED;𝐄) 
       ⩾ I(𝐄_ND;𝐄_ED)


Assuming q is a Gaussian distribution, the reconstruction process can be equivalent to minimizing the information entropy and its theoretical upper bound is formally defined as follows:

    H(𝐄|𝐄_ND, 𝐄_ED)⩽max{𝐄-𝐄_ND_2^2, 𝐄-𝐄_ED_2^2}


Proof. The estimation of conditional probability p(𝐄|𝐄_ND, 𝐄_ED) is very tricky and complicated. Therefore, we use an approximate variational distribution q(𝐄|𝐄_ND, 𝐄_ED) to simplify the calculation and optimization process. Therefore, we have,

    H(𝐄|𝐄_ND, 𝐄_ED)
       =-𝔼_p(𝐄|𝐄_ND, 𝐄_ED)[log p(𝐄|𝐄_ND, 𝐄_ED)] 
       ≤-𝔼_p(𝐄|𝐄_ND, 𝐄_ED)[log q(𝐄|𝐄_ND, 𝐄_ED)] 
       - D_KL(p(𝐄|𝐄_ND, 𝐄_ED)q(𝐄|𝐄_ND, 𝐄_ED))


Assume q(𝐄|𝐄_ND, 𝐄_ED) is a Gaussian distribution 𝒩(𝐄|𝐄_ND, 𝐄_ED,σ^2𝐈):

    H(𝐄|𝐄_ND, 𝐄_ED)
       ⩽-𝔼_p(𝐄|𝐄_ND, 𝐄_ED)[log q(𝐄|𝐄_ND, 𝐄_ED)] 
       =-𝔼_p(𝐄|𝐄_ND, 𝐄_ED)[log(1/√(2π I)σe^-1/2(𝐄-𝐄_{ND,ED})^2/(σ^2𝐈))] 
       =-𝔼_p(𝐄|𝐄_ND, 𝐄_ED)[log(1/√(2π I)σ)-(𝐄-𝐄_{ND,ED})^2/2σ^2𝐈]

Therefore, we get the upper bound of H(𝐄|𝐄_ND, 𝐄_ED).



§ EXPERIMENTS





 §.§ Experimental Setup


Datasets Description   In our experiments, seven publicly available benchmark datasets are used including two Amazon items datasets <cit.> (i.e., Computers, and Photo), five citation network datasets <cit.> (i.e., Citeseer, Pubmed, DBLP, CoraFull, and Cora), three large-scale datasets <cit.> (i.e., Ogbn-arxiv, Ogbn-mag, and Ogbn-products), page network datasets <cit.> (i.e., Wiki-CS and Croco).







Evaluation Metrics   We use classification accuracy to evaluate the performance of our method CGRL and other comparative methods.

Comparison Methods   We compare our method CGRL with twelve state-of-the-art deep learning-based algorithms, including two traditional graph embedding algorithms (i.e., raw features <cit.>, and DeepWalk <cit.>), three semi-supervised algorithms (i.e., GCN <cit.>, NIGCN<cit.>, and GAT <cit.>), and nine self-supervised algorithms (i.e., GAE <cit.>, VGAE <cit.>, DGI <cit.>, GCA <cit.>, MVGRL <cit.>, GIC <cit.>, GRACE <cit.>, GMI <cit.>, and CRLC <cit.>).


Setting-up 
All the experiments in this paper are implemented on a server with 2 A100 (total 160GB memory). For each experiment, we run the code five times with a random seed to obtain the final mean and corresponding standard deviation to avoid experimental chance. Futhermore, for some parameter settings of the model, we set epochs to 1000/300, batch-size to full-batch/mini-batch, learning rate to 0.005, mask rate to 0.5, α to 0.5, β is 0.5, the activation function to GELU, dropout is 0.2, the weight decay to 1e-4, learning rate scheduling to cosine, warmup epochs to 100, and hidden_size to 128. We utilize the Adam optimization algorithm to update parameters.



 §.§ Results and Analysis






Node classification.
Tables 1 and 2 summarize the node classification accuracy of the baselines and the proposed method CGRL on twelve real graph-structured data sets. Specifically, our approach leverages traditional graph embedding algorithms (i.e., raw features and DeepWalk). For example, our method CGRL improves the average accuracy by 20.31% and 10.76% compared to raw features and DeepWalk methods respectively. Compared with self-supervised methods (e, g., DGI, and GCA, etc.), CGRL also achieves better performance. In addition, CGRL also outperforms semi-supervised algorithms (i.e., GCN , NIGCN, and GAT). The performance improvement may be attributed to the design of the multi-view contrast learning strategy for adaptive masking nodes and perturbed edges, which enables the model to automatically learn whether to mask nodes and perturbed edges. To ensure that our model can reconstruct nodes and edges, we use a random walk strategy to sample the node's subgraph structure to blur its representation. However, most other baselines perform multi-view contrastive learning by randomly dropping nodes and edges, which will seriously destroy the semantic information of the graph. In addition, we also introduce the information bottleneck theory to ensure that the augmented views are structurally heterogeneous but semantically similar. The intuition behind this is that maximizing the mutual information between multiple views will lead to a consistent representation of the augmented views learned by the model, which leads to overfitting of the model. 



Effect of noise. In order to verify the anti-interference ability of our model CGRL, we perform experiments to demonstrate the performance by varying the noise rate in {0, 0.1, 0.2, 0.3, 0.4, 0.5} and the β in {0.1, 0.2, 0.3, 0.4, 0.5
,0.6, 0.7, 0.8, 0.9}. The experimental results are shown in Fig. <ref>, We can find that when the parameter β is 0, CGRL has the worst effect on the Cora, Citeseer and PubMed data sets. When β is greater than 0, CGRL's anti-interference ability is significantly reflected, and the best effect is in the range of 0.2 to 0.6. The experimental results show that the introduction of information bottleneck theory can enhance the noise invariance ability of the model.


Hyper-parameter analysis.
We set two hyperparameters α and β during the model optimization process. Their settings will have a relatively large impact on the performance of the model. Therefore, we also investigated the impact of hyperparameters on CGRL. We performed the node classification task by varying α and β from 0.1 to 0.9 and visualized the accuracy in Fig. <ref>. When α and β are set to relatively large values, the node classification effect of the model is better, and when set to smaller values, the performance is poor. The difference in performance can be attributed to the inability of smaller parameter settings to eliminate redundant information in the graph structure and to obtain optimal augmented multi-view.




 §.§ Ablation Study

We performed three ablation experiments to verify the effectiveness of our proposed node-masking view, edge perturbation view, and information bottleneck theory.





  §.§.§ Effectiveness of adaptive graph contrastive learning via IB.


We perform ablation studies to analyze the impact of node-masking view, edge perturbation view, and IB criterion on experimental results. The experimental results are shown in Table <ref>. Firstly, when CGRL includes all modules, the accuracy of node classification on seven data sets is the highest. Secondly, We find that the node-masking view has a greater impact on the experimental results than the edge perturbation view. Experimental phenomena show that node classification mainly depends on the features of nodes. Thirdly, node-mask view and edge perturbation view combined with IB accuracy can further improve the accuracy of node classification. The performance improvement may be attributed that the IB criterion can promote multi-views to obtain structurally heterogeneous but semantically similar graph structures.



  §.§.§ Impact of GNN variants.
 
We explore the impact of different graph neural network variants on experimental results. As shown in Table <ref>, GAT has the best effect on seven graph structure data sets, with node classification accuracy rates of 86.3%, 75.4% and 84.5% on the Cora, Citeseer, and PubMed data sets, respectively, followed by GraphSAGE, with node classification accuracy rates of 83.6%, 72.8% and 82.0% , respectively, and GCN has the worst effect, with node classification accuracy rates of 82.7%, 70.7% and 81.4%, respectively. Therefore, we choose GAT as our encoder in our experiments.



  §.§.§ Effectiveness of multi-view optimization.

We perform ablation experiments to verify the impact of multi-view augmentation on the optimization process of model training. The experimental results are shown in Fig. <ref>. On Cora, Citeseer, and PubMed datasets, we found that in the absence of node-masking view or edge perturbation view, the loss value of the model cannot converge to the optimal value, and the edge perturbation view has the worst convergence effect. When CGRL combines the node-masking view, edge perturbation view and information bottleneck criterion, the loss value of the model can converge to close to 0. The experimental results show that the performance of the augmented node-masking view is better than the augmented edge perturbation view, but worse than the augmented multi-view that combines information bottlenecks, which shows that the information bottleneck theory can further enhance the utilization of information.



  §.§.§ Visualization.

We use the T-SNE method to project the high-dimensional node features into a 2-dimensional space and visualize them. The visualization results are shown in Figs. <ref> and <ref>. In the Cora data set, GCN and GAT have more overlaps between different node categories and the class boundaries are not clear enough, while CGRL has clearer class boundaries between different node categories. In the Citeseer data set, the feature embeddings of GCN and GAT are more scattered among the same node categories, and there is excessive overlap between different categories, while CGRL is more densely distributed for the same category of nodes and there is less overlap between different categories.










§ CONCLUSIONS

In this paper, we propose a Contrastive Graph Representation Learning with Adversarial Cross-view Reconstruction and Information Bottleneck for node classification to automatically generate structurally heterogeneous but semantically similar multi-views. Specifically, CGRL can adaptively learn to mask nodes and perturb edges in the graph to obtain optimal graph structure representation. Furthermore, we innovatively introduce the information bottleneck theory into GCL to eliminate redundant information in multiple contrasting views while retaining as much information about node classification as possible. Moreover, we add noise perturbations to the original views and reconstruct the augmented views by constructing adversarial views to improve the robustness of node feature representation. Extensive experiments on real-world public datasets show that our approach significantly outperforms existing the SOTA algorithms.






elsarticle-num


















 Paper Title: Human interaction classifier for LLM based chatbot.


Black bounces in Cotton gravity
    Henrique A. Vieira
    August 5, 2024
===============================



§ ABSTRACT 

This study investigates different approaches to classify human interactions in an artificial intelligence-based environment, specifically for Applus+ IDIADA's intelligent agent AIDA. The main objective is to develop a classifier that accurately identifies the type of interaction received (Conversation, Services, or Document Translation) to direct requests to the appropriate channel and provide a more specialized and efficient service. Various models are compared, including LLM-based classifiers, KNN using Titan and Cohere embeddings, SVM, and artificial neural networks. Results show that SVM and ANN models with Cohere embeddings achieve the best overall performance, with superior F1 scores and faster execution times compared to LLM-based approaches. The study concludes that the SVM model with Cohere embeddings is the most suitable option for classifying human interactions in the AIDA environment, offering an optimal balance between accuracy and computational efficiency. 





§ ACKNOWLEDGEMENTS

To Karim Belaid, to review and help us make clear movements towards this paper submission. Your expertise and insights will be invaluable in refining our research, strengthening our arguments, and ensuring the quality of our work meets the highest standards. We greatly appreciate your time and dedication in supporting this important academic endeavor.




§ INTRODUCTION


Artificial intelligence has emerged as a transformative force, reshaping the landscape of modern society. Its influence extends to various sectors, including the automotive industry where Applus+ IDIADA operates. Within this environment, AI has taken the form of an intelligent chatbot baptized as AIDA, an LLM-powered virtual assistant.

An LLM-based chatbot is an artificial intelligence system that uses a Large Language Model (LLM) to generate human-like responses in conversations. These chatbots are trained on vast amounts of text data, allowing them to understand and produce natural language across a wide range of topics. They can engage in more complex, context-aware dialogues compared to traditional rule-based chatbots, often demonstrating impressive language understanding and generation capabilities. 

This digital aide lends its capabilities to a multitude of tasks, serving as a versatile companion to IDIADA's workforce. From addressing inquiries to tackling intricate technical challenges spanning code, mathematics, and translation, AIDA's versatility knows no bounds.

At a certain point, the need to establish a clear criterion for the categorization of the various interactions that this intelligent agent receives on a daily basis arises. The main aim is to offer a more in-depth service through the creation of dedicated pipelines for various contexts (conversation, document translation, services).

By categorizing the interactions into these three main groups, the system can better understand the user's intent and respond accordingly. The Conversation class encompasses general inquiries and exchanges, the Services class covers requests for specific functionalities or support, and the Document Translation class handles text translation needs.

In this paper, we will discuss the research process carried out to develop a classifier for human interactions in this AI-based environment. The objective of this classifier is to accurately identify the type of interaction received by the intelligent agent AIDA, in order to route the request to the appropriate pipeline and provide a more specialized and efficient service.

Through this classification model, the AIDA assistant can optimize its responses, allocate resources more effectively, and enhance the overall user experience. The research and development of this LLM-based classifier is an important step in the continuous improvement of the intelligent agent's capabilities within the Applus IDIADA ecosystem.



§ THE DATA


We are going to use a set of 1668 examples of pre-classified human interactions. These will be divided into 666 for training and 1002 for testing. A 40-60 has been applied, thus giving a lot of importance to the test set.


The following figure shows some of the examples:





All samples have been gathered from real AIDA users interactions. All the samples remain in a RAW state, unadulterated and without preprocess. They have been manually labeled by the Datalab team. There are no mislabeled cases, despite the complexity in some cases given the abstractness of the interaction itself.





§ METHODOLOGY


In this section three different methods are presented, two of them based in LLM models and another one based in machine learning classic algorithm. The aim is to understand which approach it is the most suitable to face the presented problem.

NOTE: in all cases the prompt or the implementation can be seen in the Annex section.




 §.§ LLM Based Classifier: Simple prompt


In this case, we present an LLM-based classifier to discern between three classes: Conversation, Services, and Document Translation. This initial approach is based on the principle of understanding a set, through which the nature of the cases that compose it is defined.

Therefore, three different definitions have been given to the model, one for each one of the classes (Conversation, Services, Document_translation). It is important to note that maybe, the definition that It could be expected for some class, It is not exactly the one It could be held in mind. For instance, in case of Conversation, we can see that we englobe tasks about text summarisation.





 §.§ LLM Based Classifier: Example Augmented Inference


This approach uses RAG (retrieval augmented generation) techniques to feed the model response, not only with a definition by compression, but also a quasi-definition by extension. That is, in this case we show the model different examples of each class with the aim that the model is able to learn the inherent characteristics of each one in order to complement the initial description given.



The total amount of examples given for each class is:

 
        
  * Conversation: 500 examples. This is the most common class, only 500 samples are given to the model given the Huge amount of information held which causes overflow of the infrastructure (very high delays, throttling, connections shutouts).

        This is something very important to note, because it is a big stopper. Giving more examples this approach could perform better, but the question is ¿How many examples? Surely a huge amount of them. 
            
        
  * Services: 26 examples. This is the less common class, In this case all train data have been used.

        
  * Translate_Document: 140 examples. In this case all train data have been used.


This approach encounters a challenge absent in the previous one: Scalability. While the model's performance benefits from additional examples, the mounting computational demands render it incompatible with the current infrastructure. The sheer volume of data becomes unwieldy, leading to quota issues with Bedrock services and very long time responses. This is not acceptable, there is a need of rapid response times to maintain a satisfactory user experience.




 §.§ KNN Based Classifier: Titan Embeddings


On this occasion we approach the problem taking into account that despite the multitude of different interactions that may exist, at the end of the day these are very repeatable or similar to each other. So, perhaps, when making a new estimation for an input, a distance-based approach could be applied, such as KNN (K-NEAREST-NEIGHBORS) <cit.>, assigning the class of the most similar examples surrounding the input.

To achieve this, it is imperative to transform the interactions, which are in textual form, into a space where algebraic operations can be applied. To this end,a technique known as Embeddings has been applied, which facilitates the derivation of a coordinate vector for a text within a space. Embeddings are vector representations of text that capture semantic and contextual information. Consequently, by comparing two vectors, the semantic similarity between two different interactions is analyzed, thereby determining "their proximity."



    
  * Embeddings: Amazon Titan 1 has been used. This model generate vectors of 1536 dimensions, it is trained to accept several languages and to retain the semantic meaning of the phrases embedded.
    
    
  * Classifier: a classic machine learning algorithm (knn) has been applied by using the sklearn python module. This method takes a parameter, which has been set to 3.

    

    This figure shows the F1 scores for each one of the classes vs the Number of neighbors used. As it shows, the best point is k=3 (highest value for Document_Translation, 3rd highest value for Services, but Document_Translation is much more common than Services).






 §.§ KNN Based Classifier: Cohere Embeddings


In the previous section, we used the Amazon Titan 1 model to generate text embeddings due to its popularity. However, there are other models available in the market. In this section, we repeat the same process but using the COHERE multilingual model. We chose this model because of its excellent ability to work with various languages without affecting the vectorization of phrases. As we will demonstrate shortly, this model does not cause significant differences in the generated vectors, thus making it more suitable for use in a multilingual environment such as AIDA.


    
  * Embeddings: COHERE model has been used. This model generate vectors of 1024 dimensions, it is trained to accept several languages and to retain the semantic meaning of the phrases embedded.

    
  * Classifier: a classic machine learning algorithm (knn) has been applied by using the sklearn python module. This method takes a parameter, which has been set to 11.
    

    This figure shows the F1 scores for each one of the classes vs the Number of neighbors used. As it shows, the best point is k=11 (highest value for Document_Translation and 2nd for Services). This time the Documents_Translation / Services trade-off worth's it.






 §.§ Titan Embeddings VS Cohere Embeddings


In this section we go a little deeper into the embeddings generated by both models, with the aim of understanding their nature and thus being able to understand the results obtained with the models. To achieve this, a dimensionality reduction has been carried out with the aim of being able to visualize the vectors obtained in both cases in 2D.

One point to keep in mind is that the cohere model has a limitation on the size of the text that it is capable of vectorizing, this being an important limit. Therefore, in the implementation shown in the previous section you can see the application of a filter to only have interactions of 1500 characters (which leaves out all cases that exceed the limit).

In the next figure, the vector spaces generated in each case can be observed:

    

As we can see, the spaces generated are relatively similar, it could seem that they are very similar spaces with a rotation between one and the other but looking closely it can be seen that the direction of maximum variance in the case of cohere is different (it can be deduced through from observing the relative position/shape of the different groups).

This type of situation, in which a high class overlap is observed, is a good case for applying algorithms such as KNN. As said in the introduction "most human interactions with AI, at the end of the day, are very similar to each other of the same class" This would explain why KNN-based models outperform LLM-based models.





 §.§ SVM Based Classifier: Titan Embeddings


In this scenario, it is likely that user interactions belonging to the three main categories (Conversation, Services, and Document Translation) form distinct clusters or groups within the embedding space. This is because each category has particular linguistic and semantic characteristics that would be reflected in the geometric structure of the embedding vectors. Remember that in the previous visualization of the embeddings space only 2D transformation of this space is displayed. This does not mean that clusters could not be highly separable in a higher dimension.

Classification algorithms like SVMs <cit.> are especially well-suited to leverage this implicit geometry of the data. SVMs seek to find the optimal hyperplane that separates the different groups or classes in the embedding space, maximizing the margin between them. This ability of SVMs to exploit the underlying geometric structure of the data makes them an intriguing option for this user interaction classification problem.

Furthermore, SVMs are a robust and efficient algorithm that can effectively handle high-dimensional data sets, such as text embeddings. This makes them particularly suitable for this scenario, where the embedding vectors of the user interactions are expected to have a high dimensionality.




    
  * Embeddings: Amazon Titan 1 <cit.> has been used.  This model generate vectors of 1536 dimensions, it is trained to accept several languages and to retain the semantic meaning of the phrases embedded.

    
  * Classifier: a classic machine learning algorithm (SVM) has been applied by using the sklearn python module. This method takes several parameters, in order to get the optimal ones a grid search with 10-fold cross validation based in f1 multi class score has been applied resulting in the following parameters selected:

    
        
  * C: 1. This parameter controls the trade-off between allowing training errors and forcing rigid margins. It is a regularization parameter. A higher value of C (e.g., 10) indicates a higher penalty for miss-classification errors, resulting in a more complex model that tries to fit the training data more closely. This can be useful when the classes in the data are well-separated, as it allows the algorithm to create a more intricate decision boundary to accurately classify the samples. A C of 1 indicates a reasonable balance between fitting the training set and the model's generalization ability which could mean the data has a simple structure and that a more flexible model is not necessary to capture the underlying relationships.
        
        
  * class_weight: None. This parameter adjusts the weights of each class during the training process. Setting 'class_weight' to 'balanced' automatically adjusts the weights inversely proportional to the class frequencies in the input data. This is particularly useful when dealing with imbalanced datasets, where one class is significantly more prevalent than the other(s). A value of None suggest that the minnor clases do not have much relevance which in the same manner suggest that the implicit geometry not be good enough in order to separate the different classes.
        
        
  * kernel: 'linear'. This parameter specifies the type of kernel function to be used by the SVC algorithm. The 'linear' kernel is a simple and efficient choice, as it assumes that the decision boundary between classes can be represented by a linear hyperplane in the feature space. This value suggest that in a higher dimension vector space the categories could be linearly separated by an hyperplane.
    





 §.§ SVM Based Classifier: Cohere Embeddings



    
  * Embeddings: COHERE <cit.> model has been used. This model generate vectors of 1024 dimensions, it is trained to accept several languages and to retain the semantic meaning of the phrases embedded.
    
    
  * Classifier: a classic machine learning algorithm (SVM) has been applied by using the sklearn python module. This method takes several parameters, in order to get the optimal ones a grid search with 10-fold cross validation has been applied resulting in the following parameters selected:

    
        
  * C: 10. A C of 10 may suggest that the data has a more complex structure and that a more flexible model is necessary to capture the underlying relationships in the data.
        
        
  * class_weight: 'balanced'.In this case 'balanced' suggest that the minor classes may have more relevance than in the previous case which in the same manner suggest that the implicit geometry may be better in order to separate the different classes.
        
        
  * kernel: 'linear'. This parameter suggest that in a higher dimension vector space the categories can be successfully linearly separated by an hyperplane.
    





 §.§ ANN Based Classifier: Titan/Cohere Embeddings


We have seen in the final results that SVM can provide good solutions, and therefore we decided to explore another method based on geometry. An Artificial Neural Network (ANN) <cit.> approach has been applied.

In this case, a normalization of the input vectors has been performed to obtain the advantages of normalization when using neural networks. The normalization of the input data is a crucial step when working with ANNs, as it can help improve the performance and stability of the model. By ensuring that the input features are on a similar scale, the neural network can learn more effectively and avoid issues such as vanishing or exploding gradients during training. A min-max scaling has been applied.


The use of an ANN-based approach provides the ability to capture complex non-linear relationships in the data, which may not be easily modeled using traditional linear methods like SVM. The combination of the geometric insights and the normalization of inputs can potentially lead to improved predictive accuracy compared to the previous SVM results.

 
    
  * Model Definition: A sequential deep learning model is defined using the Keras library from TensorFlow. 
  * Model Architecture: The model consists of three densely connected layers. The first layer has 16 neurons and uses the ReLU activation function. The second layer has 8 neurons and also uses the ReLU activation function. The third layer has 3 neurons and uses the softmax activation function. 
    
  * Model Compilation: The model is compiled using the categorical_crossentropy loss function, the Adam optimizer with a learning rate of 0.01, and the categorical_accuracy metric. An EarlyStopping callback is used to stop the training if the categorical_accuracy metric does not improve for 25 epochs. 
    
  * Model Training: The model is trained for a maximum of 500 epochs using the training set and validated on the test set. The batch size is set to 64. The performance metric used is the maximum classification accuracy (categorical_accuracy) obtained during the training. 



It is important to note that in order to avoid ordinal assumptions an OHE (one hot encoding) representation have been chosen for the output of the net. One-Hot Encoding does not make any assumptions about the inherent order or hierarchy among the categories. This is particularly useful when the categorical variable does not have a clear ordinal relationship, as the model can learn the relationships without being biased by any assumed ordering.





§ RESULTS




A comparative analysis have been conduced using the code and the data previously presented. The models were assessed based on their F1 scores for the Conversation, Services, and Document Translation tasks, as well as their execution times. This procedure has been conduced several times in order to ensure the results by averaging the obtained metrics for each one of the approaches.

As shown in Figure <ref>, the SVM and ANN models leveraging COHERE Embeddings demonstrated the strongest overall performance. The SVM with COHERE Embeddings achieved the highest F1 scores in two out of three tasks, reaching 0.99 for Conversation, 0.80 for Services, and 0.93 for Document Translation. Similarly, the ANN with COHERE Embeddings also performed exceptionally well, with F1 scores of 0.99, 0.77, and 0.96 for the respective tasks.

In contrast, the LLM model exhibited relatively lower F1 scores, particularly for the Services (0.22) and Document Translation (0.46) tasks. However, the performance of the LLM improved when provided with examples, with the F1 score for Document Translation increasing from 0.46 to 0.68.

Regarding execution time, the KNN, SVM, and ANN models demonstrated significantly faster inference times compared to the LLM. The KNN and SVM models with both Titan and COHERE Embeddings had execution times around 0.3-0.35 seconds, while the ANN models were even faster, with execution times of approximately 0.15 seconds. In contrast, the LLM required approximately 1.2 seconds for inference, and the LLM with examples took around 18 seconds.

These results suggest that the SVM and ANN models leveraging COHERE Embeddings offer the best balance of performance and efficiency for the given tasks. The superior F1 scores of these models, coupled with their faster execution times, make them promising candidates for application. The potential benefits of providing examples to the LLM model are also noteworthy, as this approach can help improve its performance on specific tasks.



§ CONCLUSIONS


In this study, we explore different approaches to classify human interactions in an AI-based environment. Our main objective was to develop a classifier that accurately identifies the type of interaction received by the AIDA intelligent agent, to direct the request to the appropriate channel and provide a more specialized and efficient service.

The comparative analysis of various models conducted in this study has yielded valuable insights into the performance and efficiency trade-offs for the given tasks. The key findings and conclusions are as follows:

 
    
  * The SVM and ANN models leveraging COHERE Embeddings demonstrated the strongest overall performance, achieving the highest F1 scores across the Conversation, Services, and Document Translation tasks. 
    
    
  * The SVM with COHERE Embeddings achieved exceptional results, with F1 scores of 0.99, 0.80, and 0.93 for the respective tasks, making it a promising candidate for real-world applications. 
     
  * The ANN model with COHERE Embeddings also exhibited excellent performance, with F1 scores of 0.99, 0.77, and 0.96, further reinforcing the benefits of the COHERE Embedding approach. But this models introduce the need of preprocessing the input vectors in order to normalize them and it may introduce a bias (which min/max values must be used?).
    
  * While the LLM model exhibited relatively lower F1 scores, particularly for the Services and Document Translation tasks, providing examples to the LLM was shown to improve its performance on the Document Translation task, increasing the F1 score from 0.46 to 0.68. 
    
  * In terms of execution time, the KNN, SVM, and ANN models demonstrated significantly faster inference times compared to the LLM, with execution times around 0.3-0.35 seconds for the KNN and SVM models, and approximately 0.15 seconds for the ANN models. 
    
  * The superior performance and efficiency of the SVM and ANN models with COHERE Embeddings make them promising choices for real-world applications that require both high accuracy and low latency. 


These findings contribute to the understanding of the trade-offs between model performance, task-specific capabilities, and computational efficiency. The insights gained from this study can inform the selection of appropriate models and preprocessing techniques for various applications, ultimately enhancing the effectiveness and practicality of the deployed systems.

The findings of this study clearly demonstrate the superior performance and efficiency of the SVM model with COHERE Embeddings, making it the most suitable choice for the classification of human interactions in the AIDA intelligent agent environment. The SVM model's exceptional F1 scores across the Conversation, Services, and Document Translation tasks, coupled with its fast inference time, position it as a highly promising candidate for real-world applications.

The ability of the SVM with COHERE Embeddings to accurately identify the type of interaction and direct the request to the appropriate channel will enable AIDA to provide a more specialized and efficient service, ultimately enhancing the overall user experience. Furthermore, the insights gained from this comparative analysis can guide the selection of appropriate models and preprocessing techniques for similar classification tasks, contributing to the development of more effective and practical AI-based systems.




§ ANNEX



 §.§ Simple prompt reproducibility




    
  * Libraries: The programming language employed in this code is Python, complemented by the Langchain module, which is specifically designed to facilitate the integration and utilization of large language models (LLMs). This module provides a comprehensive set of tools and abstractions that streamline the process of incorporating and deploying these advanced artificial intelligence models.
 

    To leverage the power of these language models, the Amazon Bedrock service has been utilized. Bedrock is a managed service offered by Amazon Web Services (AWS) that is specifically tailored for the deployment and scaling of large language models. The integration with Bedrock is achieved through the boto3 Python module, which serves as an interface to the AWS ecosystem, enabling seamless interaction with the Bedrock service and the deployment of the classification model.

    

    
  * Prompt: The task at hand is to assign one of the three classes (Conversation, Services, or Document_Translation) to a given sentence, represented by question.
    
        
  * Conversation Class: This class encompasses casual messages, summarization requests, general questions, affirmations, greetings, and similar types of text. It also includes requests for text translation, text summarization, or explicit inquiries about the meaning of words or sentences in a specific language.
        
  * Services Class: Texts belonging to this class consist of explicit requests for services such as room reservations, hotel bookings, dining services, cinema information, tourism-related inquiries, and similar service-oriented requests.
        
  * Document_Translation Class: This class is characterized by requests for the translation of a document to a specific language. Unlike the Conversation class, these requests do not involve summarization. Additionally, the name of the document to be translated and the target language are specified.
 
    
    The prompt suggests a hierarchical approach to the classification process. First, the sentence should be evaluated to determine if it can be classified as a Conversation. If the sentence does not fit the Conversation class, then one of the other two classes (Services or Document_Translation) should be assigned.
 

    The priority for Conversation class yields in the fact that the 99% of the interactions are actually simple questions regarding some matters.
 
    

    
  * Model invocation: 
        
            
  * Model: Anthropic Claude 3 Sonnet model has been used <cit.>, which it is the language model to be used for the natural language processing task. This LLM model has a context window of 200.000 tokens, it is able to manage different languages and to retrieve highly accurate answers.
            
  * Parameters: max_tokens: This parameter limits the maximum number of tokens (words or subwords) that the language model can generate in its output to 50.
            
  * Parameters: temperature: This parameter controls the randomness of the language model's output. A temperature of 0.0 means that the model will produce the most likely output according to its training, without any randomness introduced.

            
        

    
  * Output parser: another important part is the output parser, which allow us to gather the desired information in JSON format. In order to achieve that, langchain output_parsers has been used.









 §.§ Example Augmented Prompt Inference reproducibility



    
  * Prompt: in this case the prompt is modified to include all the examples in json format under "Here you have some examples" section.


The next figure shows the changes applied to the first version of the classifier.







 §.§ KNN Based Classifier: Titan Embeddings reproducibility


In the next figure the implementation is presented:







 §.§ KNN Based Classifier: Cohere Embeddings reproducibility



In the next figure the implementation is presented:







 §.§ SVM Based Classifier: Titan Embeddings reproducibility


In the next figure the implementation is presented:





 §.§ SVM Based Classifier: Cohere Embeddings reproducibility


In the next figure the implementation is presented:





 §.§ ANN approach: Titan/Cohere Embeddings reproducibility


In the next figure the model implementation is presented:




In both cases (Titan/Cohere) the same preprocessing has been applied:









 Paper Title: Enhancing Partially Spoofed Audio Localization with Boundary-aware Attention Mechanism.



Spatial Transformer Network YOLO Model for Agricultural Object Detection


 



    Yash Vivek Zambre^1, Ekdev Rajkitkul^2, Akshatha Mohan^1, Joshua Peeples^1
 ^1Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA

	^2Department of Computer Science and Engineering, Texas A&M University, College Station, TX, USA

		yashzambre@tamu.edu, ekdev@tamu.edu, akshatha.mohan@tamu.edu, jpeeples@tamu.edu
    
=========================================================================================================================================================================================================================================================================================================================================================================




§ ABSTRACT 
    
   The task of partially spoofed audio localization aims to accurately determine audio authenticity at a frame level. Although some works have achieved encouraging results, utilizing boundary information within a single model remains an unexplored research topic. In this work, we propose a novel method called Boundary-aware Attention Mechanism (BAM). Specifically, it consists of two core modules: Boundary Enhancement and Boundary Frame-wise Attention. The former assembles the intra-frame and inter-frame information to extract discriminative boundary features that are subsequently used for boundary position detection and authenticity decision, while the latter leverages boundary prediction results to explicitly control the feature interaction between frames, which achieves effective discrimination between real and fake frames. Experimental results on PartialSpoof database demonstrate our proposed method achieves the best performance. The code is available at <https://github.com/media-sec-lab/BAM>.



[Corresponding author.]





§ INTRODUCTION


With the rapid advancement of Artificial Intelligence Generated Content (AIGC), technologies like text-to-speech (TTS) <cit.> and voice conversion (VC) <cit.> can generate realistic human voices. Partially spoofed audio, where synthesized speech segments are inserted or spliced into genuine utterances, pose significant threats. An attacker can alter the meaning of sentences by manipulating small and specific units (e.g., words, characters, or even phonemes), deceiving both machines and humans.

In recent years, numerous methods and datasets have been developed for Partially Spoofed Audio Detection (PSAD). Yi et al. <cit.> constructed the first partially spoofed audio dataset named HAD, which replaces some nature segments with synthesized speech segments having different semantic content. The Audio Deep Synthesis Detection (ADD) challenge 2022 <cit.> involves a detection track containing partially spoofed audio. The organizers provided only real and entirely fake data for training, and the task was target to detect fake at utterance level. In this challenge, notable detection performance was achieved by fine-tuning pretrained self-supervised learning (SSL) models with different back-end classifiers <cit.>. Boundary detection task was only introduced as a proxy task to achieve utterance-level detection <cit.>.

Beyond merely detecting the presence of spoofing in audio, the Partially Spoofed Audio Localization (PSAL) task has become an emerging topic for better analyzing deepfake audio. Zhang et al. <cit.> developed a dataset named PartialSpoof, which includes segment-level labels. They then investigated an approach to detect spoofing at both segment and utterance levels within a multi-task learning framework <cit.>. Subsequently, they extended the PartialSpoof dataset and used Wav2vec2 (W2V2) <cit.> as a front-end to detect segment-level and utterance-level fakes simultaneously <cit.>. Moreover, the ADD 2023 <cit.> introduced a PSAL track, which has resulted in numerous promising works <cit.>. Among them, an effective approach was proposed to achieve the best localization score by integrating the decisions from three countermeasure (CM) systems including boundary detection, frame-level fake detection, and utterance-level fake detection <cit.>. Recently, an approach incorporating a contrastive learning  module and temporal convolution was proposed to effectively capture better features for localization <cit.>.



Some aforementioned methods <cit.> utilize boundary information solely for PSAD at utterance-level detection. However, leveraging boundary features to aid in pinpointing the exact locations of spoofing within a single CM system is still an unexplored topic. As we know, all frames within a segment delineated by boundaries are assigned the same label. Once the correct boundary position is identified, the authenticity of a frame can be determined by referencing other frames within the same segment. Thus, boundary information serves as vital auxiliary knowledge that can steer authenticity decisions at the frame level. To exploit this idea, in this paper, we have designed a new attention mechanism that leverages boundary information, as illustrated in Figure <ref>. Message passing between those frames that belong to different authenticity classes is explicitly weakened by the attention mechanism. Our proposed method is named Boundary-aware Attention Mechanism (BAM). Specifically, pre-trained W2V2 <cit.> or WavLM <cit.> is used as front-end feature extractor. Subsequently, a Boundary Enhancement (BE) module extracts inter-frame and intra-frame features to effectively detect the boundary frame position. Finally, a Boundary Frame-wise Attention (BFA) module leverages the boundary prediction results to make reliable  authenticity decisions at frame level. In this way, discriminative features are learned to better distinguish between real and fake frames, leading to improved localization performance. In summary, the contributions of this paper are as follows:


  * We propose a novel approach called BAM that takes boundary information as auxiliary attentional hint to guide localization. To the best of our knowledge, this work is the first attempt to leverage boundary information for PSAL within a single CM.

  * We develop a boundary enhancement module containing two branches for better exploiting intra-frame and inter-frame information to obtain discriminative boundary features.

  * We conducted experiments to demonstrate the effectiveness of the proposed method. Compared with existing methods, our method achieves the best localization performance on PartialSpoof dataset.




§ PROPOSED METHOD




The flowchart of our BAM is depicted in Figure <ref>. Firstly, we use a pre-trained SSL model to extract the features of speech and apply an attentive pooling layer <cit.> to make each frame represents a specific temporal resolution (e.g., 160). The output of the pooling layer is fed into the Boundary Enhance (BE) module to enhance boundary feature representation and identify the boundary frames with a simple fully connected layer. Then, the Boundary-aware Frame-wise Attention (BFA) module takes the boundary prediction results and the outputs of the pooling layer as inputs to capture the correlation information among frames. Finally, the outputs of the BFA module and the BE module, are concatenated and fed into a fully-connected layer to make frame-level authenticity decisions.



 §.§ Pretrained self-supervised front-end

We start by processing raw speech using a pretrained SSL speech model, W2V2 <cit.> or WavLM <cit.>, to extract effective front-end features. Compared to traditional handcrafted features (e.g., LFCC <cit.>, MFCC <cit.>), the SSL-based front-end features exploit extensive volumes of unlabeled data for pre-training, significantly improving data representation capabilities and facilitating the identification of intricate patterns within audio data. In our implementation, we utilize WavLM-large <cit.> or XLS-R-300M <cit.> and fine-tune their weights in conjunction with other modules, and use their last hidden states of the transformer as front-end features.



 §.§ Boundary enhancement module


In the context of PSAL, frames at the boundaries that contain both spoofed and genuine samples are labeled as spoofed. During training with binary authenticity labels, boundary frames (especially those with a smaller proportion of spoofed data) are close to fully genuine frames in the feature space, even though they have opposite labels. This situation can lead to training instability and performance degradation. To mitigate this issue, we have developed a boundary enhancement module specifically to model and distinguish boundary frames from non-boundary frames.

As shown in Figure <ref>, the boundary enhancement module transforms the front-end feature F_g ∈ℝ^T × D into the boundary feature F_b ∈ℝ^T × 2D, where F_g is the output feature of a pooling layer, T is the number of frames, and D is the feature dimension. The boundary feature serves dual purposes.. Firstly, it is fed into a fully-connected layer with a sigmoid function to obtain the boundary prediction probability b̂∈ℝ^T × 1 as following:

    b̂=sigmoid(ϕ(F_b)),

where the linear mapping function ϕ with a learnable weight matrix W and a bias term b is given by: 

    ϕ(x)=Wx+b,

Then we conduct binarization with a fixed threshold to make binary boundary prediction B̂. Secondly, the boundary feature is processed by a fully-connected layer with an activation function and concatenated for the final frame-level authenticity decision. 

The detail of BE module is illustrated in the middle of Figure <ref>. Specifically, two different branches are designed to respectively extract the intra-frame and inter-frame features. For the inter-frame branch, a Frame-wise Attention Block (FAB) is employed to capture inter-frame correlations using an attention mechanism. The right part of Figure <ref> illustrates the details of the FAB. We start by computing the attention score s ∈ℝ^T × T × D through the element-wise multiplication of each frame feature with others. Then, a learnable attention weight W_a ∈ℝ^D× H is defined to weigh the attention score. This process can be written as:

    A_t=tanh(ϕ(s))W_a,

where A_t∈ℝ^T× T× H represents the attention map. Subsequently, each frame within F_g aggregates information from other frames, utilizing the attention map as following:

    F_a=softmax(A_t)F_g,

where F_a ∈ℛ^H× T× D represents updated frame features. The residual structure is employed to stabilize training. The output of FAB is obtained from F_a and F_g according to:

    F_inter=𝒮(ℬ𝒩((ϕ(F_a) ⊕ (ϕ(F_g)))),

where ⊕ represents element-wise addition and ℬ𝒩 represents 1-D batch normalization, and 𝒮 is SELU <cit.> activation function. For the intra-frame branch, each frame is individually fed into a 1D-ResNet <cit.>, followed by a fully-connected layer to learn intra-frame features. Then, the intra-frame feature F_intra and inter-frame feature F_inter are concatenated to obtain the boundary feature F_b. 



 §.§ Boundary frame-wise attention module



As shown in Figure <ref>, the BFAM module consists of two stacked Boundary Frame-wise Attention Blocks (BFAB). Specifically, the structure of BFAB is similar to FAB but is equipped with a boundary masking component, which takes binary boundary prediction B̂ and F_g to weaken the message passing between those frames that belong to different genres. A boundary adjacency matrix A_b∈{0,1}^T × T is constructed based on the boundary prediction B̂. Its element is defined as follows:

    A_b_i,j=
        
        1   if  i = j, 
    ∏_n=i^j(1-B̂[n])   if  i<j, 
    ∏_n=j^i(1-B̂[n])   if  i>j,

where B̂[n] represents the n^th element of sequence B̂, and i,j∈[0,1,…,T-1] represent the indices of frames. In other words, equation <ref> determines whether there is a boundary between the i^th and the j^th frame; if there is, the value in the i^th row and the j^th column of A_b is set to 0, otherwise, it is set to 1. Once the adjacency matrix is obtained, the attention map (refer to equation <ref>) can be updated to a boundary attention map Â_̂t̂, as illustrated in Figure <ref>, and given as:

    Â_̂t̂=A_t⊗ A_b,

where ⊗ denotes element-wise multiplication.



 §.§ Loss function

We employ two loss functions to constrain the model for supervised training: boundary loss ℒ_b and frame-level authenticity loss ℒ_s as shown in Figure <ref>. For boundary loss, binary cross-entropy loss is used and the ground-truth boundary labels B can be derived from the segment-level authenticity labels Y. It is worth noting that we only set the boundary frames as label 1 and all other frames as label 0, which differs from previous methods <cit.> where they also set the label of frames near the boundary frames to 1. This labeling strategy <cit.> may introduce additional noise in our case and lead to diminished localization performance. For authenticity loss, we follow prior work and utilize the standard cross-entropy loss. In summary, the overall loss can expressed as:

    L = ℒ_s(ŷ,Y)+λℒ_b(b̂,B),

where ŷ∈ℝ^T× 1 represents the authenticity prediction result of model, and λ is set to 0.5 in practice.



§ EXPERIMENTS AND RESULTS




 §.§ Dataset and implementation details

To validate the effectiveness of our method, we conducted experiments on the PartialSpoof <cit.>. The dataset is constructed by segmenting the fully spoofed and genuine utterances from the ASVspoof2019 <cit.> LA database using Voice Activity Detection (VAD) and then concatenating those segments with basic digital signal processing. Four evaluation metrics are used to compare model performance: Equal error rate (EER), precision, recall and F_1 score.   

In the training phase, for experiments conducted at a 160 temporal resolution, we fix the length of training samples at 4s. Samples exceeding this length are randomly trimmed, while those shorter than are padded with zero. Then raw data is padded to correspond with the 20 shift of W2V2 and WavLM. Therefore, the attentive pooling stride is set to 8, the number of frames T is 25, the feature dimension D is 1024, block number N is 2, and the learnable weight head number H is 1. For experiments conducted at a 20 temporal resolution, the number of frames T is 200, and we use a linear transformation to reduce the feature dimension D to 256. Other hyperparameters remain consistent with those used in 160 experiments. In the test phase, variable-length test samples are directly fed into the model to obtain the prediction result. we use Adam optimizer with a learning rate of 10^-5 and halved every 10 epochs, and all of model are trained for 50 epochs.



 §.§ Comparison with existing methods

To ensure a fair comparison with existing methods <cit.>, we conducted experiments with 160 ms temporal resolution. The results are shown in Table <ref>. Based on our observation, WavLM achieves better results compared with W2V2-XLSR. Moreover, Our method gets EER as 3.58% and F1 score as 96.09%, reaching the best single CM performance on PartialSpoof dataset.






 §.§ Ablation study


We conducted ablation experiments to assess the effectiveness of each modules within our framework under two settings: 1) localization task and 2) boundary detection task. Table <ref> presents the results for localization task. The baseline model, denoted as 1 , is equipped with a WavLM front-end, succeeded by a max pooling layer and a fully-connected layer. The models 2 and 3 represent simplified versions of the BAM. Model 2 incorporates only the right branch as illustrated in Figure <ref>, while model 3 includes both the left and right branches but omits the middle pathway from the BE module to the BFA module. Model 4 is actually the proposed BAM. It can be seen that each module contributes to the enhancement of localization performance. Specifically, compared to the baseline model, our method reduced the EER by 2.21% and increased the F1 score by 1.73%.

Table <ref> presents the results for boundary detection task. It is clear that the inter-frame feature is more effective than intra-frame. The BE module combined with both inter-frame and intra-frame achieves the best boundary detection performance with EER as 3.33% and F1 score as 92.25%.




 §.§ Finer-grained resolution experiment




Extra experiment was conducted at a 20 temporal resolution. Specifically, we trained the BAM without pooling directly on the front-end feature with a 20 shift for 50 epochs as the base model. Subsequently, we removed the last fully-connected layer from pretrained base model and replaced it with an attentive pooling layer succeeded by a new fully-connected layer. We finetuned this model for 10 epochs to predict the authenticity at specific temporal resolutions. The results are shown in Table <ref>. The base model achieves remarkable localization performance at 20 resolution with EER as 5.20% and F1 score as 95.82%. As the temporal resolution decreases, the F1 score remains almost constant while the EER continually increase. We explain the reason behind this trend is that with the decreases in resolution, the total number of frames and the proportion of genuine frames both rise, and EER is particularly sensitive to changes in class proportions while the F1 score is relatively robust. Moreover, there is a marginal decrease in localization performance compared to BAM executed on frames of 160 (3.66% v.s 3.58%). We attribute this decline in performance to the increased challenge of boundary detection at finer resolutions. The base model gets EER as 5.12% and F1 score as 82.97% at 20 resolution for boundary detection task, which is worse than the boundary detection performance at 160 (refer to Table <ref>).
    


§ CONCLUSION

In this paper, we propose a novel partially spoofed audio localization method. Our method simultaneously conduct boundary detection and frame-level authenticity determination tasks within a single CM model. The boundary information is utilized to enhance the accuracy of localization. Experimental result show that the proposed method achieves the best performance on PartialSpoof dataset. The finer-grained resolution experiment demonstrates that accurately detecting boundary position at a finer resolution is a more challenging task and will be a direction of our future work.



§ ACKNOWLEDGEMENTS

This work was supported in part by the National Natural Science Foundation of China (Grant U23B2022, U22B2047, 62322120), and in part by the Guangdong Provincial Key Laboratory (Grant 2023B1212060076).

IEEEtran




 Paper Title: Can Developers Prompt? A Controlled Experiment for Code Documentation Generation.














Can Developers Prompt? A Controlled Experiment for Code Documentation Generation
    

Hans-Alexander Kruse 0009-0006-5568-7841,
Tim Puhlfürß 0000-0001-8421-8071,
Walid Maalej 0000-0002-6899-4393


Universität Hamburg
  
Hamburg, Germany

hans-alexander.kruse@studium.uni-hamburg.de,
tim.puhlfuerss@uni-hamburg.de,
walid.maalej@uni-hamburg.de

    August 5, 2024
======================================================================================================================================================================================================================================================================



§ ABSTRACT 

llm bear great potential for automating tedious development tasks such as creating and maintaining code documentation.

However, it is unclear to what extent developers can effectively prompt llm to create concise and useful documentation.

We report on a controlled experiment with 20 professionals and 30 computer science students tasked with code documentation generation for two Python functions.
The experimental group freely entered ad-hoc prompts in a ChatGPT-like extension of Visual Studio Code, while the control group executed a predefined few-shot prompt.

Our results reveal that professionals and students were unaware of or unable to apply prompt engineering techniques.
Especially students perceived the documentation produced from ad-hoc prompts as significantly less readable, less concise, and less helpful than documentation from prepared prompts.
Some professionals produced higher quality documentation by just including the keyword Docstring in their ad-hoc prompts.
While students desired more support in formulating prompts, professionals appreciated the flexibility of ad-hoc prompting.
Participants in both groups rarely assessed the output as perfect.
Instead, they understood the tools as support to iteratively refine the documentation.

Further research is needed to understand which prompting skills and preferences developers have and which support they need for certain tasks.



Software Documentation,
Large Language Model,
Program Comprehension,
Developer Study,
AI4SE


Can Developers Prompt? A Controlled Experiment for Code Documentation Generation
    

Hans-Alexander Kruse 0009-0006-5568-7841,
Tim Puhlfürß 0000-0001-8421-8071,
Walid Maalej 0000-0002-6899-4393


Universität Hamburg
  
Hamburg, Germany

hans-alexander.kruse@studium.uni-hamburg.de,
tim.puhlfuerss@uni-hamburg.de,
walid.maalej@uni-hamburg.de

    August 5, 2024
======================================================================================================================================================================================================================================================================










§ INTRODUCTION




Developers often overlook or ignore software documentation and generally assign it a low priority <cit.>. 
Yet, carefully documenting code is an essential task in software engineering. 
Up-to-date and high-quality documentation facilitates program comprehension <cit.>, accelerates developer onboarding <cit.>, and mitigates technical debt <cit.>. 
Software documentation is also central to software maintenance, as documentation often requires updates and should evolve with software  <cit.>.


Researchers and tool vendors have thus investigated different ways to automate documentation <cit.>. 
In particular, llm designed to model and generate human language <cit.> hold great potential in automating documentation tasks <cit.>.
Among popular llm use cases, code summarization has recently gained much attention <cit.>.
However, the output of llm usually depends on the user input, called prompt. 
Slightly changing the prompt can lead to different results regarding conciseness, language style, and content of generated text <cit.>. 


Several prompt engineering techniques have emerged to optimize interactions with llm <cit.>.
Few-shot prompting is one of the most prominent techniques and comprises adding example outputs to a predefined prompt to specify output requirements <cit.>.
This technique allows users to optimize llm responses while minimizing the number of input messages sent to the model <cit.>.


Previous research has primarily focused on benchmarking the performance of various models and prompt engineering techniques for generating source code documentation <cit.>.
Recently, Ahmed and Devanbu concluded that common prompt engineering techniques perform better than ad-hoc prompting according to popular benchmark metrics <cit.>.
However, studies focusing on the perspectives of developers when using llm, e.g., to generate documentation, are still sparse.
In fact, a recent study showed that common metrics to evaluate llm do not align with human evaluation to assess the quality of generated documentation <cit.>.
Hence, it remains uncertain whether prompt engineering techniques meet developers' requirements for generating and using code documentation.
Moreover, it is unclear whether developers prefer a flexible, iterative interaction using chatbot-like ad-hoc prompting or a rather transaction-like execution of predefined prompts.

This study takes a first step towards filling this gap, focusing on two research questions:



  RQ1: How well can developers prompt llm to generate code documentation compared to a predefined prompt?

  RQ2: How is the developer experience for ad-hoc prompting compared to executing predefined prompts?



To answer these questions, we conducted a randomized controlled experiment with 20 professional developers and 30 computer science students.
The experiment involved generating code documentation using an llm-powered ide.
The first group utilized a vscode extension based on GPT-4, enabling participants to enter ad-hoc prompts for selected code.
The second group used a similar vscode extension that executed a predefined few-shot prompt to generate code documentation via GPT-4 with a single click.



Overall, we observed that developers with less experience require more guidance in phrasing an ad-hoc prompt, whereas more experienced developers can generate fairly good documentation by including specific keywords like Docstring in the prompts.
Overall, participants preferred transactional or guided llm interactions to create code documentation while enjoying the flexibility of ad-hoc and iterative prompting.
Our results show that predefined prompts deliver code documentation of significantly higher quality than ad-hoc prompts, especially due to a consistent documentation format.

We present the design of our study in Section <ref> and report on the results in Section <ref> (RQ1) and Section <ref> (RQ2). 
We then discuss the implication of our findings in Section <ref> and potential threats to validity in Section <ref>. 
Finally, Section <ref> summarizes related work, and Section <ref> concludes the paper.
We share our experiment results, including the ide extension, in our replication package <cit.>.







§ METHODOLOGY





 §.§ Experimental Design




We designed a controlled experiment in a laboratory setting to compare developer usage of and experience with two vscode extensions employing different prompts for code documentation generation.
We followed a between-subject design, with one participant testing only one of both tools to avoid learning effects <cit.>.
The ad-hoc prompt group created prompts in an already available GPT-4-powered ide extension, while the predefined prompt group interacted with our extension.
We randomly assigned participants to the groups, with adjustments made to balance the occupations. 
Participants were unaware of their group.


We offered offline and online versions of the experiment.
In the offline version, participants used our computer with vscode and the digital questionnaire displayed on two screens.
In the online version, we utilized Zoom for video conferencing, sharing a questionnaire hyperlink, and screen-sharing our vscode window with external input control.
Participants employed Thinking Aloud to express thoughts during the task <cit.>.
To ensure a calm environment, we put a participant in a room empty of people.
The questionnaire included questions about the code and the generated documentation.
Additionally, we collected the ad-hoc prompts for a later comparison with the predefined prompt.


The experiment comprised three parts, as shown in Figure <ref>.
In the first part, participants answered questions about occupation, as well as Python and vscode experience to assess their ability to understand complex functions and use the ide.
Furthermore, we could check whether these variables have any effect on the observed behavior.

The second part focused on code comprehension and documentation generation.
It consisted of two rounds, each involving a pre-selected Python function (Listing <ref>).
Per round, participants (1) first attempted to comprehend the code without any documentation.
This enabled us to analyze the effect of the llm-generated documentation on understanding the code functions.
Subsequently, (2) participants rated their comprehension on a scale from 1 (very low) to 5 (very high), and answered True/False questions about the code to further check their understanding (Table <ref>).
Participants could also state that they had insufficient information to answer a question.
Furthermore, (3) participants manually created a function comment to externalize their understanding and better assess the quality of the comment subsequently generated by the tool.
To guide the participants and shorten the completion time, we provided the participants with a comment template adhering to guidelines for clean Python code <cit.> and instructed them to add only the function description without the explanations of parameters.
Participants then used the respective llm-powered ide extension to generate code documentation.
Afterward, (4) as documentation should serve to facilitate code comprehension, they studied the generated comment and revisited the True/False questions.
This step provided insights if the documentation supported or even hindered the comprehension.
Finally, (5) they rated the generated documentation on six quality dimensions based on the human evaluation metrics by Hu et al. <cit.>.

The documentation quality dimensions each provided a question and a five-point answer scale, addressing grammatical correctness, readability, missing relevant information, unnecessary information presence, usefulness for developers, and helpfulness for code comprehension. 
We made minor adjustments to the original answers of two dimensions as we found them overloaded for our goal.
In particular, the answers for grammatical correctness originally also included a rating of fluency, and the original readability answers also concerned the comment's understandability.
The lowest point of each scale represents the negative pole, like very low readability, while the highest point represents the positive pole, like very low amount of unnecessary information.

In the third part of the experiment, participants assessed the usability of the respective ide extension.
They completed the standardized ueq with 26 items, each associated with a seven-point answer scale and one of six usability categories <cit.>.
Finally, participants provided comments on tool strengths and areas for improvement in two free-text fields.



 §.§ Ethical Considerations



We followed the standard procedure of the ethics committee of our department for assessing and mitigating risks regarding ethics and data processing.
Upon welcoming participants, we explained the privacy policy, study purpose, upcoming tasks, and data processing procedures.
We designed the study to last for approx. 30 minutes per participant to prevent exhaustion and instructed participants not to seek perfect solutions to reduce stress. 
We offered clarifications on tasks and questionnaire content without providing solutions.
We minimized observational bias by facing away while participants answered the questionnaire and also encouraged negative feedback <cit.>.
To express our gratitude for the participation, we raffled vouchers among participants.
We pseudonymized all published data as far as possible while maintaining our study objective.



 §.§ Technical Setup



[float, caption=Python functions that participants had to comprehend and for which they generated documentation., label=lst:exp_fct]list=no
Function 1:
def string_from_vector_bool(data):
    return ",".join(str(int(i)) for i in data)
—————————————————-
Function 2:
def _parse_date(date):
    if date is None:
        date = Timestamp()
    if isinstance(date, Timestamp):
        date = date.toLocal()
    d = dateutil.parser.parse(date)
    if d.tzinfo is None:
        d = d.replace(tzinfo=dateutil.tz.tzlocal())
    return d.astimezone(dateutil.tz.tzutc())
    .replace(tzinfo=None).isoformat()


[float, caption=Predefined few-shot prompt. Few-shots 2 and 3 are comparable to 1 and included in the replication package <cit.>., label=lst:const_prompt, xleftmargin=0pt]list=no
For the following prompt, take into account these 
three input/output pairs of functions and 
corresponding appropriate comments:

Function 1:
def elapsed_tid(cls, reference, new):
    time_difference = new.toTimestamp() - reference.toTimestamp()
    return np.int64(time_difference * 1.0e6 // cls._period)
Comment 1:
"""
    Calculate the elapsed trainId between reference and newest timestamp

    :param reference: the reference timestamp
    :param new: the new timestamp

    :type reference: Timestamp
    :type new: Timestamp

    :returns: elapsed trainId's between reference and new timestamp
"""
Function 2:
...

Generate a comment for the following function: 
FUNCTION_CONTENT. Fill in this template: 
TEMPLATE. Adhere to the appropriate comment 
syntax for multi-line comments.




  §.§.§ Selection of Experiment Tasks



We selected the Python source code for the experiment tasks from the open-source control system Karabo <cit.>.
Karabo is maintained by the scientific facility euxfel, whose developers partially participated in the experiment.
By choosing this project and involving project insiders, we could assess how their programming and domain experience supports their code comprehension and documentation rating, especially in comparison to outsiders.

Two authors conducted a thorough screening of available Karabo functions and identified six candidates.
All six were utility functions, which we deemed comprehensible for experiment participants not associated with euxfel.
Our selection criteria also included code documentation quality aspects such as conciseness, completeness, and usefulness especially relevant for the few-shot examples <cit.>.
We independently analyzed the candidates and categorized them into the complexity levels easy, medium, and hard.
Afterward, we resolved categorization conflicts by discussion.

We selected one easy-, and one medium-rated function for the tasks of the experiment (Listing <ref>).
The first function converts a boolean vector into a comma-separated string.
We considered the code easy to understand as only standard Python operations were used within this short function.
The second function converts a given date to the ISO format.
The complexity of this function lies in the usage of multiple classes and functions with abbreviated identifiers, and the lack of inline comments.
Hence, participants must make assumptions based on identifier names and usage.

Initially, we also selected a third function rated hard but removed it after a pilot study with three software engineering researchers, who reported difficulties with this function and far exceeded the time limit.
While we acknowledge that llm might require more sophisticated prompts and context information to generate appropriate documentation for more complex code, our user study focused on how differently skilled developers can and prefer to interact with an llm for the creative process of documentation generation.
Therefore, within the experiment's time limit, all participants should be able to phrase ad-hoc prompts that generate appropriate comments for these functions.



  §.§.§ LLM Selection and Prompt Construction



Research and industry have introduced multiple llm with individual strengths and limitations. 
For our study, we focused solely on the GPT-4 model by OpenAI (version gpt-4-0613). 
We did not aim to benchmark different models but rather to study developers prompting skills particularly for code documentation generation.
OpenAI's models have been prominent in recent research on documentation generation <cit.>. 
Furthermore, during our study period, GPT-4 demonstrated general superiority over other models <cit.>.

We defined the predefined few-shot prompt (Listing <ref>) based on prompt engineering guidelines <cit.> and tested it in a pilot study.
Given GPT-4's limited context window at the time of the study, we applied a three-shot approach to specify the required output format.
We chose three distinct function-comment pairs from the remaining previously selected Karabo code candidates.
Hence, these functions were from the same domain and of similar complexity as the two used in the experiment.
They adhered to a structured documentation format with precise and concise content.
We instructed GPT-4 to generate a comment for the selected source code that we passed as input (FUNCTION_CONTENT).
To further enforce the clean code format <cit.>, we explicitly defined an output template within the prompt (TEMPLATE) and requested adherence to Python's syntax for multi-line comments (Docstrings).
Listing <ref> only contains example 1 of the few-shot prompt. 
Examples 2 and 3 are available in our replication package <cit.>.
We acknowledge that the prompt can be further optimized and generalized to multiple programming languages and application domains.
We considered its performance and complexity sufficient for our research goal.



  §.§.§ Tool of the Ad-hoc Prompt Group





We focused on ide extensions as developers usually use ide for generating code documentation.
We chose vscode for its extensive customization capabilities through various extensions maintained by an open-source community. 
During our study, the extension marketplace already offered multiple ChatGPT-like tools that we examined.
Additionally, vscode supports multiple programming languages, including Python.

We selected the ad-hoc prompt tool based on two criteria.
First, it needed to provide a user interface similar to the ChatGPT interface, allowing users to engage in a conversation with the software. 
This approach involves at least one self-written message to the model and copying and pasting the generated comment from the chat interface to the code.
This criterion was crucial as we aimed to compare the user experience for this classic Q&A-based interaction with the one for the predefined prompt tool, which generates and inserts code documentation with a button click.
Second, the tool had to support and behave like the original GPT-4 api to handle any confounding factors related to the use of different llm models.

After a thorough testing of available extensions, we opted for ChatGPT by zhang-renyang <cit.> in version 1.6.62. 
This extension met all specified requirements.
It offers a small prompt field (Figure <ref>) and a larger ChatGPT-like chat window within the ide.
Our test showed that it mimics GPT-4.
It was one of the most popular GPT extensions in the marketplace during our study period.



  §.§.§ Tool of the Predefined Prompt Group




Our design objective for the predefined prompt group was to minimize the steps required for developers to generate a high-quality Docstring comment for a specific code section. 
We determined that marking the code within the ide's code editor and selecting the Generate a comment action from the context menu should suffice, requiring three clicks to generate documentation (Figure <ref>).
Alternatively, the developer can choose this action through vscode's command field. 
Hence, inexperienced and experienced vscode users can use the primary feature of the tool with their preferred workflow.
A secondary feature allows developers to customize the predefined prompt via the Edit default prompt action, enabling them to tailor the tool output to their preferred format.


We followed the vscode guidelines to create the TypeScript-based extension <cit.> and utilized the axios package to implement calls to the OpenAI api.
Based on the input prompt, the api returns a code comment generated by GPT-4.
The extension automatically inserts this documentation above the previously selected code.
To set up the extension, developers must enter their OpenAI api token into a configuration field.
This token is stored locally on the developer's machine.
For the experiment, we pre-configured the tool with our token.


We used GPT-4 version gpt-4-0613 for both tools. 
Generating code documentation with this model took, on average, four seconds.
The vscode version was 1.85.
The predefined prompt tool is included in our replication package <cit.> and also available in the vscode marketplace <cit.>.



 §.§ Experiment Execution and Participants




The experiment took place in October and November 2023, involving 50 participants: 25 per ad-hoc and predefined prompt group. 
20 participants were professionals, including 12 software engineers, four data scientists, and four with other software development roles. 17 worked in the domains of scientific computing and three in finance.
These participants took the expert perspective due to their professional experience with Python programming and, eventually, the application domain.
The remaining 30 participants were students in computer science or related subjects, taking the newcomer perspective.
We recruited participants through personal contacts in academia and industry to ensure that the participants had at least some experience with Python and vscode. 
We conducted 33 experiments online and 17 offline at the euxfel campus in Schenefeld, Germany, with the first author leading the experiment and the second author assisting.

On a scale from 1 (very low) to 5 (very high), the mean value and standard deviation of the Python experience among professionals were 3.7 (1.06) in the ad-hoc and 4.5 (0.71) in the predefined prompt group.
The values among students were notably lower, with 2.53 (0.83) in the ad-hoc and 2.47 (1.19) in the predefined prompt group.
This difference confirms the expert/newcomer setting.

All participants were familiar with vscode, which was important to mitigate biases due to the tool setup.
Mean values and standard deviations in the ad-hoc/predefined prompt group among professionals were 2.9 (0.99) and 3.5 (1.18); among students they were 3.07 (1.03) and 3.07 (1.28).

At the beginning of each session, we trained each participant in using the respective tool to mitigate learning bias.
We introduced them to the tool features, generated documentation for an example function, and answered tool-related questions.
Afterward, we started the experiment.
Due to manually writing prompts, ad-hoc prompt group participants completed the experiment after 23:53 minutes on average, while predefined prompt group participants required only 19:58 minutes.



 §.§ Data Analysis




Our experiment comprised quantitative and qualitative data analyses.
We report totals, mean values, and standard deviations where applicable.
Furthermore, we employed inferential statistics to test for significant differences between the groups' ratings of the six documentation quality dimensions. 
Depending on the statistical assumptions fulfilled by the respective variables, we utilized the parametric Welch's t test or the non-parametric mwu test <cit.>.

The assumptions for both tests were that the collected data points were quantitative and independent.
The first assumption was fulfilled as we provided a five-point answer scale for each question related to a quality dimension.
The second assumption was fulfilled as we conducted the experiment with each participant individually, and we asked participants to not share information with others to avoid external influences.
The third assumption of the Welch's t test is a normal distribution of data points, which we tested with the Shapiro-Wilk test <cit.>.
If a normal distribution was not met for a specific quality dimension, we applied the mwu test instead.
Besides reporting the p-values for statistical significance, we also report the effect sizes to indicate practical significance.
We particularly applied Cliff's δ <cit.>, which describes the overlap of two groups of ordinal numbers.
Values between 0.0 and 0.8 indicate a low to medium effect size, whereas values from 0.8 represent a large effect size, meaning a large difference between both groups.


To quantify the experience of developers using the experiment tools, we utilized the 26 seven-point scales of the ueq <cit.> and its official data analysis tool <cit.>.
This tool provides descriptive (mean values and standard deviations) and inferential statistics (t test) to compare user experience ratings for the two studied ide extensions.

Each of the 26 scales is related to one of six quality categories.
The first category, Attractiveness, represents the pure valence of and overall satisfaction with a tool.
Efficiency, Perspicuity, and Dependability are the pragmatic, goal-directed categories, and Stimulation and Novelty are hedonistic, not goal-directed categories.
We chose Efficiency, Perspicuity, Dependability, and Novelty to measure tool-related experience, as these categories focus on tool interactions.

To evaluate participants' assessment of the quality of generated documentation, we used the Stimulation category as it focuses on the output of a tool.
We enhanced this analysis with qualitative insights made by manually analyzing participants' free-text answers to the questions What did you like most about the tool? and  What would you change about the tool?, placed at the end of the questionnaire. 
Two authors independently conducted Open Coding for all answers and discussed their codes to achieve consensus <cit.>.
Furthermore, we report relevant remarks expressed by participants during the experiment.










§ PROMPTING FOR CODE DOCUMENTATION (RQ1)





 §.§ Patterns in Ad-hoc Prompts




The prompts entered by the ad-hoc prompt group varied in detail and keywords. 
About half of the participants had to re-enter a different prompt after their first attempt to generate a code comment.
Among the failed attempts were prompts such as Explain the function and Describe what this function does, which summarized the code in longer prose text.


We categorized the 50 prompts that led to a successful generation of code comments for both code functions into five prompt patterns, listed in Table <ref>.
These patterns commonly included documentation-specific keywords, i.e., comment (34x), Docstring (12), and Python-conform description (1).
Also, slightly longer prompts that included the comprehension-focused terms explain (2) and summarize (1) led to appropriate function comments.
Professionals used the Docstring term more often (9x) than students (3x), which is in line with the higher Python experience of most participating professionals.


Successful prompts were generally short, with a mean length and standard deviation of 7.1 (2.09) words among professionals and 6.7 (2.62) words among students.
The shortest prompt was Generate docstring.
None of the participants enhanced their prompts with examples or templates.



 §.§ Perceived Quality of Generated Documentation




Participants rated the documentation generated by the ad-hoc and predefined prompt tools based on six five-point-scale questions.
All participants were aware of the expected format of a Python Docstring, as they were familiar with Python. 
Moreover, before using the tool, they manually created a function comment based on a Docstring template.
Table <ref> displays the ratings per group, subgroup (student vs. professional), code function, and quality dimension, along with the p-values <cit.> and effect sizes <cit.>.


We performed several trials to determine the appropriate statistical test (see section <ref> for details). 
We rejected the null hypothesis of the Shapiro-Wilk test for all groups and dimensions, indicating a non-normal data distribution, likely due to small sample sizes. 
We chose the mwu test for all comparisons due to its suitability for non-normally distributed data <cit.>.
The null hypothesis of this test indicates no statistically significant difference between the mean values of both groups for a specific dimension. 
We rejected this null hypothesis if the p-value was lower than 0.05.
We noted that in all cases when the null hypothesis was rejected, the effect size δ was large (0.8 or higher), indicating practical significance <cit.>.




Comparing all participants in the ad-hoc and predefined prompt groups, we observed several statistically significant differences.
For function 1, we found significant differences in the dimensions of Readability (4.12/4.72) and Unnecessary Information (3.28/4.32).
The predefined prompt tool consistently provided concise Docstrings that participants perceived as more readable and contained less non-informative content compared to the output of the ad-hoc prompt tool.
For function 2, significant differences were found in four dimensions, with the predefined prompt tool consistently receiving higher ratings than the ad-hoc prompt tool.
The differences in Readability (3.52/4.68) and Unnecessary Information (2.84/4.68) reinforced the conclusions drawn from function 1.

We attribute the ratings of Usefulness (3.52/4.6) to the often excessive output length of the ad-hoc prompt tool.
Participants noted that longer, prose documentation could impede development workflows, as these comments require more time to read while providing little additional information.
Helpfulness was rated high in both groups (4.0/4.6), with some participants suggesting that the explanations provided by both tools were either too complicated or lacked the required level of detail.
Across both groups, participants rated Grammatical Correctness and Missing Information very high.


When we compared documentation ratings between students and professionals, we observed similar statistically significant differences in the ad-hoc and predefined prompt groups for students, while professionals did not exhibit significant differences.
Among students, we observed significant differences for function 1 in the dimensions Readability (4.0/4.66), Unnecessary Information (3.33/4.86), and Usefulness (3.93/4.6).
The different expectations regarding the documentation content contributed to the disparity in Usefulness, with some students preferring the longer summarizations by the ad-hoc prompt tool, while most others expected a structured comment as taught in university.

For function 2, students in the predefined prompt group consistently provided significantly better ratings for Readability (3.2/4.6), Unnecessary Information (2.66/4.86), Usefulness (3.13/4.73), and Helpfulness (3.66/4.6) compared to students in the ad-hoc prompt group.
Unnecessary Information had the highest differences, likely due to the unstructured and lengthy comments generated via the ad-hoc prompts.

Among the professionals, we found no statistically significant differences between the ad-hoc and predefined prompt groups.
However, three dimensions showed large effect sizes (≥ 0.8), indicating a practical significance.
These differences concerned Readability for functions 1 (4.3/4.8) and 2 (4.0/4.8), and Unnecessary Information for function 2 (3.1/4.4).
This indicates that professionals also found the comments of the ad-hoc prompt tool to contain non-informative content, impairing readability.
Conversely, the ad-hoc prompt group provided a more positive rating for the dimension Missing Information (4.7/4.0) for function 1.
Professionals expected more detailed descriptions, which the concise comments by the predefined prompt tool did not provide.
Furthermore, as professionals used the Docstring keyword more often than students in their prompts (Table <ref>), the ad-hoc prompts in these cases resulted in similar documentation to the predefined prompt tool.



 §.§ Code Comprehension




We analyzed how the llm-generated documentation influenced the comprehension of the code.
Initially, participants rated their perceived comprehension of the respective code function without available documentation, on a scale from 1 (not at all) to 5 (very well).
For function 1, the mean and standard deviation for these ratings were 3.6 (0.96) by the ad-hoc and 3.84 (1.11) by the predefined prompt group, indicating a similar good understanding.
The assessment for function 2 was lower, with 2.4 (0.96) by the ad-hoc and 2.92 (0.81) by the predefined prompt group.
Hence, the predefined prompt group expressed slightly higher confidence in the comprehension than the ad-hoc prompt group, despite similar stated Python experience.
Across both groups and functions, professionals stated a higher comprehension than students, which aligns with their higher Python experience.


We assessed participants' actual comprehension through True/False questions about the code (Table <ref>).
We counted the number of participants who correctly answered these questions before and after generating the documentation. 
Participants could also choose the option Not enough information available to answer the question, which we treated as an incorrect answer.

Before using the tools, most participants in both groups answered the two questions for function 1 correctly (Q1: 19+23; Q2: 23+24), as expected for this short and easy-categorized function.
After using the tool, the correct answers for question 1 increased (Q1: 22+24), especially among students, while the ones for question 2 slightly decreased (Q2: 21+20) for both students and professionals.
Thinking-aloud observations revealed that many participants were confused by the less technical documentation generated by both tools regarding this question, as it did not include the term iterable.

The number of correct answers to function 2 before using the tool was lower (Q1: 10+14; Q2: 15+13), reflecting its higher complexity. 
In both groups, correct answers increased after using the tools (Q1: 22+18; Q2: 21+21). 
Students had the highest increases: in the ad-hoc prompt group for Q1 from 5 to 13, and in the predefined prompt group for Q2 from 5 to 14. 
The increase in comprehension among professionals was marginal, except for the correct Q1 answers in the ad-hoc prompt group, which increased from 5 to 9.
This data shows that the availability of documentation helped inexperienced developers to better comprehend the code, while the effect on professionals was mixed.


Answering RQ1, we conclude that developers were able to generate code documentation using ad-hoc prompts without prompt engineering techniques, often by providing relevant documentation keywords.
However, particularly less experienced students initially did not use such keywords and generated lengthy, less readable, and less useful code explanations instead.

This is reflected in participants rating the quality of generated documentation.
We observed statistically significant differences exclusively among students, who rated the readability, conciseness, usefulness, and helpfulness of comments generated via the predefined few-shot prompt higher than those created via ad-hoc prompts.
Although we observed large effect sizes for some quality dimensions among professionals, this subgroup was generally less critical as they often generated documentation of the required quality and format by using specific keywords.

We also conclude that documentation generated by both tools helped (particularly students) comprehend more complex code.
The generated documentation could also lead to confusion, even for professionals.









§ DEVELOPER EXPERIENCES WITH PROMPTING (RQ2)





 §.§ Overall Experience



Participants rated both tools positively in all six categories of the standardized ueq <cit.>, with values ranging from -3 to +3 (Table <ref>).
However, students found the predefined prompt tool more positive in all user experience categories than the ad-hoc prompt tool, with statistically significant differences and large effect sizes in five categories.
Professionals also rated the predefined prompt tool more positively in five categories, without significant differences but a large effect size in one category.

For the pure valence category Attractiveness (1.01 ad-hoc / 1.99 predefined prompt group), the predefined prompt tool received higher ratings than the ad-hoc prompt tool, with a high difference among students (0.64/1.94). 
This indicates an overall better developer experience with the predefined prompt tool.



 §.§ Tool-Related Experience



The most significant difference between the groups was in the category Efficiency (0.7/1.98), with the predefined prompt tool receiving higher ratings, especially among students (0.42/2.12).
Participants praised the ad-hoc prompt tool for its practicality of not having to open ChatGPT in the browser (two students, one professional), flexibility in selecting relevant code (one professional), and custom prompts (one professional).
They noted issues with copy-pasting the tool output to the code panel (six students, four professionals), multiple tries sometimes required for sufficient results (one student), the need to phrase a prompt (one professional), missing keyboard shortcuts (one professional), and the slow output generation, which is primarily caused by the performance of GPT-4 (one student).
Participants who interacted with the predefined prompt tool also noted the slow response (one student, two professionals), but expressed no further positive or negative comments.


The ratings for Perspicuity (1.25/1.87) were higher for the predefined prompt tool, with large effect sizes for students and professionals.
While some participants of the ad-hoc prompt group participants found their tool easy to use (four students, one professional), others reported that they would find it easier to enter the prompt in a separate ide panel instead of the pop-up at the top of the screen (one professional) and that the right-click context menu, which listed the tool features besides non-related ide features, was obscure to them (one professional).
Multiple predefined prompt group participants also found their tool easy to use (eight students, four professionals), highlighting the ease of generating comments without the need for further prompt input besides the code and a short command (one professional).
Nevertheless, they assessed that the selection of relevant code could be facilitated (one professional).
Overall, this assessment aligns with our observation that participants quickly understood both ide extensions but tended to prefer the simplicity of the predefined prompt tool without having to create an effective prompt.


Ratings for Dependability (1.05/1.84) showed strongly diverging views among students (0.85/1.95).
Participants praised the ad-hoc prompt tool for supporting the comment creation workflow (six students, two professionals) and code comprehension (three students, four professionals).
They criticized the lack of prompt templates (three students), that the tool did not always generate code comments (two students, one professional), and the missing opportunity for iterative improvements caused by the limitations of the OpenAI api at the time of our study (one student, one professional).
The predefined prompt tool was also noted for facilitating the workflow (three students, one professional) and code comprehension (four students), as well as for its reliability (two professionals).
However, participants suggested adding a progress bar (one professional), previewing generated comments within the function (one professional), enabling the configuration of the required comment content (one student), adding a disclaimer that the output might be incorrect (one student), implementing a linter that alerts when generated comments became outdated (one professional), and using a locally deployed api for data security (one professional).
Besides these improvement comments, the predefined prompt group was overall satisfied regarding the Dependability aspects, whereas the ad-hoc prompt group, and especially the students, required more enhanced tool support.


Both tools achieved similar positive results concerning Novelty (1.1/1.18), without significant differences.
Interestingly, the ad-hoc prompt tool received a higher rating from professionals (1.38/1.18) for its chat-like interaction within an IDE, which they considered more novel than clicking a button for documentation generation.
Participants expressed no further comments regarding this ueq category.



 §.§ Documentation-Related Experience



For Stimulation (0.82/1.62), predefined prompt tool ratings were higher, especially among students (0.6/1.6).
Concerning the output of the ad-hoc prompt tool, participants noted the detailed explanations (two students, two professionals), the mitigation of personal biases (one student), inspirational value (one professional), and overall high quality (three professionals).
Other participants criticized the output for containing unnecessary information (five students, three professionals), lack of detail regarding complex code (three students), and inconsistent format due to using prompts of varying qualities (two students, one professional).
The predefined prompt tool output received positive ratings for its conciseness (four students, one professional), overall high quality (three professionals), and similar structure across all generated comments (one student, one professional).
Improvements were deemed necessary for the parameter descriptions (one student, one professional), line breaks (one professional), consistent punctuation (one student), and lack of detail for complex code (one student, one professional).
These insights show that students were especially unsatisfied with the ad-hoc prompt tool output, while professionals were overall satisfied with both tools.


Answering RQ2, we conclude that the participants overall preferred a tool that automates code documentation generation with a few clicks while offering options to configure this process.
On average, professionals and students preferred executing predefined prompts over creating ad-hoc prompts.
The simplicity and efficiency of a single button click to receive consistently high-quality documentation were key factors for this assessment.
While the predefined prompt group praised their tool, some missed the flexibility to adjust the documentation depth.
Users of the ad-hoc prompt tool appreciated this flexibility, which often unintentionally resulted in longer and more explanatory comments that may assist code comprehension.








§ DISCUSSION



We summarize the results of our experiment and discuss potential implications for research and practice.



 §.§ Developers are not Necessarily Prompting Experts


Our study indicates that developers are generally neither skilled (RQ1) nor inclined (RQ2) to effectively prompt an llm in a way that it generates concise, readable code documentation. 
This partly aligns with previous studies showing that optimized, predefined prompts outperform ad-hoc prompts <cit.>. 
But it also partly contradicts the underlying assumption that effective optimized prompts, such as few-shot prompts, are trivial and can intuitively be created by developers in their daily work. 
We observed in our experiment that participants interacted with the provided tools intuitively via natural language queries.
However, they were often disappointed by the results as the tools provided outputs that were aligned with their unspecific prompts.
In fact, we even observed that the generated documentation can partly be misleading during comprehending code. 

The lack of prompting skills among developers is unsurprising, as this is not a standard skill taught in universities or commonly practiced in software development.
Therefore, researchers should explore ways to learn effective prompting and to intuitively support conversations with llm, particularly in domains like software engineering where the llm outcome has an impact on other users.
One approach is to compile a catalog of evaluated prompt templates for different well-defined tasks (e.g., documentation of Python functions).
Recently, Torricelli et al. also found such templates beneficial for llm users <cit.>.
Tool vendors could integrate the templates into their tools.
Each template could include instructions for the llm to ask clarifying and contextual questions, e.g., related to the expected documentation style <cit.>.
Additionally, educators could use these prompt templates, together with underlying ideas and hints, to teach prompting skills in schools and universities.
Such templates should only provide guidance and should not hinder human creativity in asking out-of-the-box and follow-up questions to an llm-powered tool.



 §.§ Generating Documentation With LLMs is an Iterative Task


Our results show that the participants rated the quality of documentation generated via the predefined prompt as high.
However, many were not completely satisfied with the documentation quality after this initial iteration.
The first iteration helped to comprehend the code, formulate the initial comment version, and gather ideas to alter and extend the documentation during further iterations.
This indicates that generating code documentation is rather an iterative task, not to be fully automated without the feedback of developers.

To better understand and assist this task, additional research should be conducted to identify and satisfy the preferences of code documentation providers and consumers <cit.>. 
Personal aspects include the depth of explanations, tone of the text, and lyrical style. 
Project aspects include documentation style used <cit.> and knowledge needs of code users.
This research can help vendors of ai-powered code documentation tools align the generated comments with the requirements of their users.
Thus, vendors can focus on applying and optimizing predefined prompts, which participants in our experiment preferred, while also offering additional iterations to customize the output based on preferences and needs.
Furthermore, generating multiple documentation versions at once might also stimulate developer creativity in exploring possibilities to document code <cit.>.



 §.§ Evaluating Code Documentation Quality Requires Human-Centered Quality Metrics


As Hu et al. <cit.> pointed out, current metrics used to assess the quality of ai-generated content do not align with human evaluation regarding code documentation generation.
For instance, the usefulness of generated content <cit.> should be crucial in practice but is barely evaluated in common llm benchmarks. 
Our results suggest that the assessment of code documentation quality depends on multiple factors <cit.>, such as the preferences of documentation consumers, the ratio between documentation conciseness and code complexity, and the availability of certain information and style that developers expect to be included in the documentation.
Current metrics for automatically assessing the quality of generated texts do not incorporate these aspects.
Hence, while automatically generating code documentation is fairly straightforward with llm, an expert assessment is still required to ensure good quality, e.g., during a code review.
It remains unclear how far this assessment can be fully automated, e.g., by other agents or other llm.
Future research should focus on creating and testing quality metrics for generated documentation through documentation analyses, expert surveys, and developer and benchmark studies.



 §.§ Better Code Comprehension Requires Support by Personalized LLM-Generated Explanations On The Fly


The results of our code comprehension tasks show that llm can support developers in understanding code.
Especially, the longer explanations provided by the ad-hoc prompt tool in response to vague prompts helped participants understand the given code function.
However, such explanations can also mislead developers if the generated output does not consider the information needs <cit.> or provide the particular knowledge nuggets needed for the comprehension task <cit.>.

Therefore, similar to the work of Nam et al. <cit.>, we propose that vendors of ide-integrated code comprehension tools should offer multiple code-related aspects that tool users can explore during the comprehension process.
Examples include the concepts mentioned in the code or the typical code usage <cit.>, but could also extend to explaining the code in a personalized way.
Researchers should survey developers to identify additional aspects and test their relevance in practice.



 §.§ Practitioners May Have Specific LLM Requirements


Participants in our study expressed multiple requirements for an llm-powered documentation generation tool.
For example, participants expressed concerns regarding security risks associated with using closed-source, externally hosted models and highlighted the extended time required for generating comments via the llm model we used.

To address these issues, tool vendors should support a variety of llm within a single tool, including smaller, open-source models, to better meet the needs of organizations operating with sensitive data and under specific constraints.
One of the research challenges is that different models may respond variably to the same prompt, impacting the consistency and reliability of generated outputs.






§ THREATS TO VALIDITY





 §.§ Construct Validity





(T01) Our study exclusively employed human evaluation techniques to assess code documentation quality, introducing a mono-method bias <cit.>.
Although automated metrics are used in other studies, we opted for human evaluation due to its limited correlation with automated metrics <cit.>, and considered it beneficial for a more comprehensive understanding.



 §.§ Internal Validity



(T02) We unintentionally excluded the Understandable scale from our ueq, which may compromise the Perspicuity category's accuracy. 
Remedying this, we calculated Perspicuity scores with and without the scale's median value, finding only marginal differences. 
Despite this omission, participant feedback indicated an adequate understanding of the tool.
(T03) Customizability of the predefined prompt could have positively influenced participant satisfaction and tool assessment. 
However, allowing customization could have affected the comparability of assessments among the predefined prompt group. 
(T04) Lack of significant differences in professional ratings may be attributed to the small sample size of ten professionals per group. 
Nevertheless, descriptive statistics reveal fewer differences among the professionals' ratings compared to students, likely due to their more targeted use of keywords in ad-hoc prompts.
(T05) A potential language barrier existed given the non-native English-speaking background of most participants. 
However, this was deemed a minor threat as all were able to produce coherent prompts in English.



 §.§ External Validity




(T06) Our controlled lab setup, where developers documented unfamiliar code, may not accurately represent typical documentation practices. 
However, this setting was intended to eliminate variability and enable statistical analysis. 
(T07) The professionals in our study predominantly came from euxfel, with specific expertise in Python, which might influence the results due to familiarity with the experimental code. 
This was seen as potentially beneficial, providing insight into how domain knowledge of one experiment group might improve their prompt creation.
(T08) The findings' generalizability is limited, yet they corroborate and broaden the insights from similar studies that involved different methodologies.



 §.§ Conclusion Validity




(T09) The selection of utility functions as code candidates might limit the applicability of our results to more complex coding tasks.
We chose these functions based on a pilot study confirming the difficulty of comprehending challenging code under time constraints.
(T10) Variations in llm outputs due to inherent randomness <cit.> may have influenced assessments in the predefined prompt group, though differences in outputs were marginal.
(T11) Dividing the experimental and control groups into professionals (experts) and students (newcomers) reduced overall sample sizes, weakening statistical power but allowing for a more detailed analysis of differing perceptions between these groups.
The separation between experts and newcomers was confirmed by the participants' self-perceived Python experience.
Hence, we believe our findings can be generalized to a broader population.
Nevertheless, developers with different expertise in conversational agents might behave differently.







§ RELATED WORK





 §.§ Prompting Skills of Practitioners



Prompt engineering has been pivotal since the introduction of few-shot prompting <cit.>. 
Ahmed et al. demonstrated its effectiveness in code summarization <cit.>.
They highlighted that applying a few-shot prompt can outperform traditional models like CodeBERT <cit.>, while zero- and one-shot prompts were less effective, aligning with findings of Geng et al. <cit.>. 
Meanwhile, subsequent techniques, like chain-of-thought prompting <cit.> and active prompts <cit.>, have emerged.

Prompt engineering remains a skill requiring practice <cit.>.
llm providers and researchers have published guidelines and prompt pattern catalogs for effective llm interaction across various domains, including healthcare <cit.>.
Our study is among the first to examine if and how well developers utilize prompt-related knowledge in software engineering tasks.



 §.§ LLM-Powered Developer Assistants



Recent studies on ai programming assistants show that tools like GitHub Copilot <cit.> boost productivity, reduce keystrokes, and assist in syntax recall <cit.>.
However, users desire tool enhancements for direct feedback, personalized output, and improved code context understanding <cit.>.
Currently, they often avoid these tools due to unmet requirements and lack of control.
For example, Copilot often fails to grasp instructions for code adjustments unless precisely specified <cit.>.
However, tool shortcuts to optimize an initial prompt can also nudge users to explore fewer prompts and lead to less diverse output <cit.>.
We can confirm these findings with our study.
While our participants overall preferred the interaction with and the output of the predefined prompt tool, the more exploratory ad-hoc prompts often led to more extensive output that supported code comprehension.



 §.§ Code Documentation Generation in the IDE




In the last decade, researchers have developed various methods for generating code documentation <cit.>, including template-based <cit.>, information retrieval <cit.>, and, more recently, deep learning techniques <cit.>.
Especially deep learning approaches trained on large datasets have shown high performance in code documentation generation and enabled better scalability and more complete results than the other techniques <cit.>.

We found many available ide extensions that use these ai models to facilitate developer workflows.
However, they were conceptualized as general-purpose tools and lacked code documentation features.
In the academic realm, the vscode extension Divinator was the most relevant tool to our project <cit.>. 
It provides short code summaries in multiple programming languages but has limited performance and interactivity.
With our study, we aimed to counter this lack of ide features related to code documentation generation.



 §.§ Evaluating Documentation Quality



Empirical research on software documentation quality is an active field that focuses on various artifacts, like api reference documentation <cit.> or README files <cit.>, and the perspectives of documentation writers <cit.>.
Studies on the evaluation of ai-generated documentation usually focus on automated metrics like BLEU, ROUGE, and METEOR <cit.>.
Hu et al. compared such automated metrics with human evaluations on six documentation quality dimensions <cit.> and found that automated metrics often misalign with human judgment <cit.>. 
Consequently, our study benefited from their human-centered metric by capturing nuanced aspects in the generated code documentation that automated approaches may have missed.







§ CONCLUSION




Recently, many llm have emerged, offering the potential to automate developer tasks, including creating code documentation.
However, it remained unclear how effectively developers could prompt llm to generate useful documentation.

We studied the interactions of developers with and their perception of llm-powered ide extensions during a controlled experiment with professionals and computer science students.
To generate documentation for two Python functions, the experimental group freely prompted an llm and the control group applied a predefined few-shot prompt.

Our results revealed that students, who had relatively low Python experience, preferred the guidance of the few-shot prompt over ad-hoc prompting.
They rated the documentation generated by predefined prompts significantly higher in quality, particularly regarding readability, conciseness, usefulness, and helpfulness.
Professionals were more adept than students at including Python-specific keywords in their ad-hoc prompts, resulting in the generation of higher-quality documentation.
Consequently, they enjoyed the flexibility of ad-hoc prompting more than students, even if they did not apply prompt engineering techniques.
Overall, both types of llm interactions improved the code comprehension of the study participants, but the participants often viewed the generated documentation as an intermediate result that needed iterative improvement.

We hope our findings encourage researchers to replicate this study <cit.> in diverse settings, aiming to improve developers' prompting skills and ai-powered tool support.




§ ACKNOWLEDGEMENT

We thank all participants of our experiment.
We acknowledge the support by DASHH (Data Science in Hamburg - Helmholtz Graduate School for the Structure of Matter) with the Grant-No. HIDSS-0002.

IEEEtran




 Paper Title: Deduction Game Framework and Information Set Entropy Search.

















[] 979-8-3503-5067-8/24/$31.00 2024 IEEE  
[] 



Deduction Game Framework and Information Set Entropy Search
    
Fandi Meng


Queen Mary University of London

London, United Kingdom 

f.meng@qmul.ac.uk
Simon Lucas


Queen Mary University of London

London, United Kingdom 

simon.lucas@qmul.ac.uk


    August 5, 2024
==============================================================================================================================================================================================



§ ABSTRACT 
We present a game framework tailored for deduction games, enabling structured analysis from the perspective of Shannon  entropy variations. Additionally, we introduce a new forward search algorithm, Information Set Entropy Search (ISES), which effectively solves many single-player deduction games. The ISES algorithm, augmented with sampling techniques, allows agents to make decisions within controlled computational resources and time constraints. Experimental results on eight games within our framework demonstrate the significant superiority of our method over the Single Observer Information Set Monte Carlo Tree Search(SO-ISMCTS) algorithm under limited decision time constraints. The entropy variation of game states in our framework enables explainable decision-making, which can also be used to analyze the appeal of deduction games and provide insights for game designers.





Information Set, Deduction Games, Information Entropy, Fun of Games, Game Design, Search Algorithm




§ INTRODUCTION


Deduction games represent an important category of games with a substantial user base engaging in regular play. Examples include single player games such as Black Box<cit.>, Mastermind, and Wordle, and multiplayer games such as Cluedo, Scotland Yard, and The Resistance. While some poker games feature deduction elements, the term “deduction games” specifically denotes those where deductive reasoning is the fundamental mechanism<cit.><cit.>. The structure of single-player deduction games typically involves the concealment of a `secret code' at the game's start, prompting players to engage in a sequence of queries (actions). Following each player action, feedback is provided by the in-game `Oracle.' Players must continually engage in action and reasoning processes until they identify the secret.

Our longer term goal is to better understand single and multi-player
deduction games, and provide agents to play the games and tools to
analyse the games.  In this paper we focus on single-player games.  The main contributions are:


    
  * A framework for implementing single-player deduction games and different kinds of information sets.
    
  * Information Set Entropy Search - ISES: a family of algorithms to play the games.
    
  * Entropy variation as measure of a game's challenge or potential interest.
    
  * Results on eight deduction games.


 



§ RELATED WORKS


Deduction games are a class of imperfect information games, in which Information Set is a key concept: it
is the set of all game states that are consistent
with an agent's knowledge at a point in the game. The Information Set Monte Carlo tree search(ISMCTS) algorithm specializes in sampling potential states within an information set, effectively tackling the complexities and strategic subtleties inherent in games where complete information is not universally accessible <cit.>. It exhibits superior performance in deduction games, as demonstrated by Reinhardt's utilization of the SO-ISMCTS agent in Secret Hitler, outperforming rule-based approaches <cit.>.

Our work is closely related to Downing et al (2019) 
<cit.>, who
used entropy considerations to derive 
a deduction game policy.  Although the core
principles of entropy-based policy are similar,
the methods are significantly different.  

Downing et al used a Domain-Specific Language (DSL) to 
describe deduction games.  

In our work we offer an alternative approach that has significant advantages:

    
  * No need to learn a DSL.
    
  * Fast game solving in many cases, though this depends on the game and the variant of the algorithm used.
    
  * The anytime variants of our algorithm offer extremely rapid game solving which often remains close to optimal. 
    
  * We envisage that ISES can be more easily adapted to multi-player and/or impure deduction games i.e. games with a deduction element combined with other mechanics.


Besides, methods similar to ours have also been applied in research on specific games. For instance, Neuwirth and Kooi have used entropy-based methods in Mastermind<cit.><cit.>. However, our work focuses more on the versatility of the method and provides a deeper analysis of deduction games. Our final goal is to explore how entropy-based methods can be used to evaluate game states in information-based gameplay to optimize existing search algorithms. This application of entropy differs completely from some past studies that used entropy to optimize policy iteration<cit.>.






§ DEDUCTION GAME FRAMEWORK





 §.§ Structure of Single-player Deduction Games


Single-player deduction games usually start with a hidden 'secret', and players ask questions and receive feedback until they discover the truth. Unlike typical partially observable Markov processes represented by a 6-tuple (S,A,T,R,Ω,O)<cit.>, in these games, the 'Oracle' is usually truthful, and both feedback and state transitions are deterministic. Thus, transition probabilities can be ignored, and the reward form differs from other games since the goal is often to uncover the secret in the fewest steps possible.

A simpler model can be used to represent this process, as shown in Fig. <ref>, where the game's decision-making information is represented by a 2-tuple. The information set is the collection of all possible states at a decision point, and the action space is all legal actions at that point. Without termination conditions, players must continually choose actions (make queries) and exclude possibilities contradicted by observations to update the information set. Termination conditions vary by game, usually ending when the information set has only one possibility, though different rules may apply to some games.






 §.§ Two representations of information sets


In our framework, the information entropy of game information sets is used to evaluate the degree of uncertainty in an agent's incomplete information state within a game. Meanwhile, the goal of players in deduction games is to obtain certain state information, that is, to reduce the information entropy of the information set to a target value. We note that the information state of some games can be represented in table form, whereas for some games, due to the nature of combinatorial mathematics, their information states cannot be updated in a tabular form. The following two examples of games illustrate this point.




  * Fake Coin Game: In this game, among n identical-looking coins, one coin is slightly lighter or heavier. Players can infer the identity and weight of the fake coin by placing coins on both sides of a balance scale and observing its tilt. The game terminates when the fake coin and its weight are determined<cit.>.



  * Mastermind: In its classic form, the secret comprises a code of four colored pegs, with each peg chosen from a set of six colors. Players receive feedback by proposing their own four-color code. The number of black pegs in the response indicates the number of pegs with correct colors and positions, while white pegs represent correct colors but incorrect positions. The game terminates when the correct code is provided<cit.>.





As shown in Fig. <ref>, the information state of the fake coin game can be represented by a table, and the table can be accurately iterated at each step of the game to reflect the information currently held by the player. However, as an NP-complete problem, Mastermind cannot represent its information state as a two-dimensional matrix that can be iterated step-by-step due to its combinatorial nature<cit.>. Consequently, obtaining its optimal solution cannot be transformed into a directly solvable polynomial computation problem<cit.>.

This leads to the need in our framework to calculate the entropy of information states in two ways. Based on Shannon's entropy formula (1) <cit.>, for games like the fake coin game, it is only needed to calculate the entropy of the probability table of the information state. For games like Mastermind, however, it is necessary to know the number of all possible codes in the current information set. Since the probability of all possible codes is equal, the calculation formula is as shown in formula (2) and n is the number of possible codes.



    H(X)    = -∑_i=1^n P(x_i) log P(x_i)




    H(X) = -∑_i=1^n1/nlog1/n
    = -n ·1/nlog1/n
    = log n





 §.§ Entropy analysis of games


The use of state information entropy to evaluate game states not only confers a high level of interpretability upon strategies but also manifests the strategic depth of a game. For example, in Fig. <ref>, we demonstrate the mean value of the change in state information entropy induced by various actions at the initial state across all possible starting configurations in Mastermind and Simple Mastermind—games characterized by three pegs and colors. 



The bar graph for Mastermind shows starting with two colors optimally reduces entropy and increases certainty. In Simple Mastermind, unlike the standard version of Mastermind, the feedback for guesses includes only black pegs, indicating correct colors in the correct positions, but no white pegs for correct colors in incorrect positions. Consequently, all first moves are similar, lacking strategic depth. Fig. <ref> illustrates this difference, explaining why Mastermind is more intriguing.

Is strategic superiority of some actions sufficient for an engaging deduction game? We visualized state information entropy changes through successive query rounds in Treasure Hunting Game and Mastermind Fig. <ref>. Results assume an agent invariably adopting the optimal average strategy, with algorithmic details in the next section.









In Treasure Hunting Game, players select cells to locate a hidden treasure. The game ends when treasure's location is narrowed to one cell. Fig. <ref> shows entropy range over game iterations. For Treasure Hunting Game, entropy change with optimal strategy is a straight line, while Mastermind has a range. Binary search optimally locates treasure in 8 cells, consistently halving uncertainty. But Mastermind, an NP-complete problem<cit.>, has uncertain outcomes even with the policy of minimizing entropy on average, lacking a polynomial solution like Treasure Hunt.

This explains why Mastermind is more interesting despite both being strategic games: Mastermind has uncertain entropy changes and no definitive resolution, while Treasure Hunt loses appeal once binary search is known. In conclusion, two factors make deduction games interesting: 1) Spatial - players' choices at decision points aren't equivalent. 2) Temporal - an optimal action's outcome remains uncertain after exploration, as Mastermind's entropy range in Fig. <ref> shows. Same initial move could reduce entropy to zero or require more steps.







§ INFORMATION SET ENTROPY SEARCH


In the previous section, we introduced a methodology for assessing game states using information entropy and discussed that the optimal action in single-player games should maximize average information gain. This idea has been expanded into a generalized search algorithm called the ISES algorithm.

Due to the exhaustive nature of the original ISES algorithm, which requires significant computational resources, we propose a variant that allows for controlled sampling scale. The pseudocode below shows that state and action space sampling scales can be manually set. By establishing a termination time, sampling can be halted upon reaching the predefined time limit, enabling immediate decision-making. This ensures suboptimal yet advantageous decisions within a manageable timeframe.













§ RESULTS AND CONCLUSIONS





To assess the ISES algorithm's effectiveness, we tested it on eight games and compared it with two agents: the Random Agent and the SO-ISMCTS, which is effective in many imperfect information games. The goal of these games is to decipher the game's secret efficiently, so we designed the final reward to be inversely related to the number of steps taken. 



For the eight games within our framework, we designed experiments across varying scales, with each of the four agents undergoing 500 trials for each variant. The results displayed in Fig. <ref> encapsulate their average performance across these games. It is evident that the ISES algorithm consistently outperformed the others, achieving game completion in fewer average steps across all scales. 

The sampled version of ISES, even with a 0.1s decision time limit, closely mirrored the original ISES's efficacy, particularly in games with lower computational complexity. This indicates that for smaller-scale games, partial random sampling is sufficient to maintain performance because many actions and states are equivalent. However, as game complexity escalates, the sampled approach may falter, as seen in the Fake Coin Game results, where exponential complexity growth at nine coins challenged the sampled ISES algorithm.

The experimental findings also highlight the limitations of the ISMCTS algorithm, particularly in games with extensive search spaces where performance diminishes due to insufficient simulation within the constrained decision time. In our studies, the ISMCTS agent, like the sampled ISES agent, was allotted a 0.1s decision time cap, yet its performance was significantly inferior. This gap widened with increasing game scale due to the exponential growth of the search space, as clearly shown in the Treasure Hunt Game and Low-Middle-High. 



In future we aim to expand our framework to include multiplayer deduction games such as Cluedo. The dynamics of multiplayer deduction games involve maximising one's own
information while minimising the information
gain of our opponents, and entropy-based methods seem
a natural way to approach this. 


IEEEtran




 Paper Title: AI-Assisted Generation of Difficult Math Questions.

































 
    
       
            
       
       
       
      
         






breakable



































































breakablebox[1][]
    breakable,
    colback=mybackcolor2, 
    colframe=myframecolor2, 
    fonttitle=,
    title=#1,


genbox[1][]
    breakable,
    colback=mybackcolor, 
    colframe=myframecolor, 
    fonttitle=,
    title=#1,


example
  Example:  
  




AI-Assisted Generation of Difficult Math Questions
    
  Vedant Shah*^1,2   Dingli Yu^3   Kaifeng Lyu^3   Simon Park^3

  Nan Rosemary Ke^1   Michael Mozer^4   Yoshua Bengio^1,2

  Sanjeev Arora^3   Anirudh Goyal*^1 

  ^1Mila – Quebec AI Institute   ^2Université de Montréal   ^3Princeton University 
   ^4University of Colorado, Boulder

    August 5, 2024
=================================================================================================================================================================================================================================================================================================



























§ ABSTRACT 
Current LLM training positions mathematical reasoning as a core capability. With publicly available sources fully tapped, there is an unmet demand for diverse and challenging mathematics questions. Relying solely on human experts is both time-consuming and costly, while LLM-generated questions often lack the requisite diversity and difficulty.  We present a design framework that combines the strengths of LLMs with a human-in-the-loop approach to generate a diverse array of challenging math questions. Initially, leveraging LLM metacognition skills <cit.>, a strong LLM is used to extract core “skills” from existing math datasets.  These skills serve as the basis for generating novel and difficult questions by prompting the LLM with random pairs of core skills that must be utilized in the question. The use of two very different skills within each question makes finding such questions an “out of distribution” task for both LLMs and humans. Our pipeline employs LLMs to iteratively generate and refine questions and solutions through multi-turn prompting. Human annotators then verify and further refine the questions, with their efficiency enhanced via further LLM interactions. Applying this pipeline on skills extracted from MATH dataset <cit.> resulted in MATH^2 - a dataset of higher quality  math questions,

as evidenced by:
(a)  Lower performance of all models on MATH^2 than on MATH
(b) Higher performance on MATH when using MATH^2 questions as in-context examples. 
Although focused on mathematics, our methodology seems applicable to other domains requiring structured reasoning, and potentially as a component of scalable oversight.

Also of interest is a striking relationship observed between models' performance on the new dataset: the success rate on MATH^2 is the square on MATH. This suggests that successfully solving the question in MATH^2 requires a nontrivial combination of two distinct math skills.





§ INTRODUCTION



Significant improvement in the capabilities of LLMs <cit.> to understand and generate complex mathematical content has been achieved by leveraging all the public data and a fair bit of private data. Sources of high-quality, varied, and difficult mathematical questions are drying up. Even finding new questions for evaluation is getting difficult since newly-released human exams are somewhat similar to past exams,  which are potentially present in the LLMs' training datasets. Hence, there is a pressing need for innovative methods to create new, diverse, and challenging 
questions.


Expert mathematicians and educators possess the deep understanding required to create questions that not only test a wide range of skills but also push the boundaries of what the learners, and by extension, the models, can handle. However, relying solely on human experts is not scalable.

Generating synthetic questions using LLMs is feasible at scale <cit.>, but often fall short in terms of the necessary difficulty. <cit.> employs a similar approach as ours where they extract topics and corresponding keypoints from a set of seed problems using GPT-4, and then combine the topic to generate new questions, again using GPT-4). However, the generated data is meant to be used for the finetuning of models as compared to serving as an evaluation set in our case. As a result, the questions generated in <cit.> are not sufficiently difficult. Similarly, limited work exists on ensuring the necessary diversity in the generated synthetic data. <cit.> proposes prompting frontier models to generate questions where each question is generated in the context of a persona as a way of ensuring diversity. They use 1M different personas to generate questions, which are then used for finetuning models, leading to significant improvements.  This dichotomy between the quality of human-generated questions and the scalability of LLM-generated questions presents a significant challenge <cit.>.













 §.§ Evaluation Saturation Phenomenon




LLM  evaluations getting saturated is a well-known issue. 
Some of the saturation is driven by across-the-board improvements arising from better training and more extensive/better datasets. 

But a lot has to do with evaluation-specific enhancements that optimize model performance on standard evaluations through techniques like supervised fine-tuning (SFT) on synthetic question-answer pairs. These synthetic pairs can be generated by leading proprietary models when provided with a few examples from the dataset or by filtering the model’s own responses <cit.>. Such methods can dramatically boost performance; for example, just 1 million synthetic examples can elevate Llama2 7B’s performance on the MATH dataset to levels comparable to GPT-4 <cit.>.

The distinction between general and evaluation-specific improvements is crucial. The latter may lead to overfitting to particular evaluations rather than a genuine acquisition of mathematical skills. This issue was highlighted when a new version of the GSM8K dataset revealed performance drops in many models, indicating overfitting to the previous dataset version <cit.>. Similarly, leading LLMs performed significantly worse on newer versions of the Chinese GaoKao exam compared to older exams, raising fundamental questions about the depth of their mathematical understanding. 







 §.§ Proposed Framework: AI-assisted Generation of Difficult Math Questions



 
Traditional methods of question generation, which rely heavily on human expertise, are increasingly unsustainable due to their resource-intensive nature, especially for academic research. Purely synthetic questions generated by large language models (LLMs) often fail to meet the desired standards of complexity and novelty. To address this challenge, we propose a design framework that leverages the strengths of both LLMs and human expertise in a synergistic manner.


At first glance, it may seem counterintuitive to use an AI model to generate and correct novel questions that it is unable to solve itself. 
However, recent research <cit.> demonstrated that top LLMs possess a robust understanding of mathematical skills, including the capability to identify the skills required to solve given questions <cit.>. This naturally raises the question: can LLMs operate in the reverse direction, i.e., generate math problems when given a list of skills that have to be tested? Our initial attempts yielded mixed results. While leading models could produce creative math questions when provided with a list of skills, the majority of these questions exhibited one or more of the following shortcomings: too similar to existing questions in datasets;  have errors or nonsensical elements; are too tedious or mechanical to be engaging for human annotators. (See Section <ref>.) 
Moreover, they often conflate “difficulty” with tedious calculations, which actually would play to the strength of machines to leverage external tools such as calculators or Python interpreters.




Nevertheless, there were promising instances where LLMs generated interesting and correct questions that they were unable to solve,  due to incomplete or incorrect reasoning. This observation led us to the concept of AI-assisted creation of evaluation datasets. Our process may also be of interest for human pedagogy since it begins with the extraction of core "skills" from existing math datasets, which serve as the foundational elements of mathematical questions. 
The current paper focuses on the MATH dataset <cit.>, a mainstay of  LLM evaluation in recent years.

Using state-of-the-art LLMs, we generate new, targeted questions that require random pairs of these skills.  The involvement of LLMs in creating questions based on these skills introduces a level of creativity and novelty that human efforts alone may not achieve. Finally, the proposed human-in-the-loop approach ensures that the generated questions meet high standards of quality and challenge, benefiting from human oversight and refinement.



Starting with a list of mathematical skills extracted from the MATH dataset using recently discovered methods <cit.>, we focused on creating questions that involve one skill from pre-algebra and algebra portions of the MATH dataset and one other skill randomly sampled from different sections of MATH. 
 Our generation pipeline uses carefully crafted prompts and multi-turn interactions with leading models to significantly improve the generation of high-quality questions and candidate answers.



In our AI-assisted process, human experts played a crucial role. Using the (question, answer) pairs generated by LLMs and leveraging API access to leading models, experts identified promising questions—-often those incorrectly answered by the LLMs but containing many correct ideas. Experts then refined these questions to enhance their engagement value and provided gold-standard answers. The AI-assisted process not only boosted human productivity but also resulted in high-quality, novel questions distinct from those in existing datasets. 

Importance of using random pairs of skills: We think that  this is key driver of improved diversity and difficulty among generated questions. Recall that MATH dataset is neatly partitioned into sub-areas such as “Geometry” and “Number theory.”  Requiring generated questions to combine skills from two subareas (e.g., a question linking area-and-perimeter calculations with prime number knowledge) necessitates “out of distribution” thinking; some examples appear in Section <ref>. Such questions challenged all LLMs  as well as our human raters. Our new dataset is much harder than MATH for all models. For instance, if a model has success rate 0.5  on MATH, then  success rate on our new dataset is closer to 0.5^2 =0.25.  This trend is more general, and Section <ref>  suggests that  this is a indication that the average question of MATH^2 indeed requires nontrivial use of two distinct underlying skills, which is the reason for naming this new dataset as MATH^2.
We believe that our methodology could also introduce fresh perspectives into math instruction for  AI and human learners. 










Connection to Scalable Oversight: This notion <cit.> looks ahead to how humans might supervise and check AI systems that potentially outperform humans in many relevant skills. While typically discussed in the context of alignment and safety, the concept is pertinent here. How can human experts reliably evaluate LLMs' understanding of high-school or freshman-level math when these models have already been trained on all available exams and textbooks? Could human-AI collaboration lead to more novel evaluations?


Paper organization: Sections <ref> describes our design methodology and generation pipeline for MATH^2. 


Section <ref> discusses the performance of many open-source and proprietary models  on MATH<cit.> as well as on the new MATH^2 dataset of 180 questions (see Table <ref> ).  Section <ref> discusses the interesting relationship between MATH and MATH^2 scores.

Section <ref> shows that MATH^2 questions are more useful than MATH questions when used as in-context exemplars for various LLMs. Section <ref> sheds some light on interesting behaviors and failure modes of leading LLMs that we observed during the question generation process. 















§ PIPELINE FOR AI-ASSISTED QUESTION GENERATION



We present a structured approach to generating challenging mathematics questions by combining the capabilities of large language models (LLMs) and human expertise. Given below is a high-level overview of the process before delving into the details of each step.

We begin our pipeline with skill extraction - identifying and cataloging distinct mathematical skills from a dataset, as described in <cit.>. This step creates a repository of skills linked to specific questions. The motivation behind this is to systematically generate and analyze questions that require specific skills, ensuring a comprehensive evaluation framework.

Next, we focus on generating questions that combine pairs of distinct skills to increase their difficulty. By using advanced models like GPT-4 and Claude, and incorporating in-context examples of multi-way interactions between AI and humans, we enhance the models' performance in generating complex questions. This step aims to produce challenging questions that robustly assess problem-solving abilities.

The final step involves screening and validation to filter out invalid or flawed questions. This rigorous process includes evaluating and solving the questions to identify hidden flaws, such as computational intractability or logical inconsistencies. Advanced techniques like in-context exemplars and self-consistency further ensure the accuracy and quality of the solutions. This step is crucial for maintaining the integrity and reliability of the generated questions and their solutions. Overall, each step in the pipeline is designed to systematically enhance the quality and difficulty of questions, providing a robust and comprehensive evaluation of mathematical skills.

Motivated by these challenges, we employ a five-step approach to generate difficult math questions using advanced models. For each round of generation, we randomly sample a pair of skills and three sample question-solution pairs corresponding to each skill from the skill repository. These reference examples are sourced from the MATH dataset.

Step 1: Skill Pair Validation.
We begin by asking the LLM (GPT-4 or Claude) to validate the skill pair by assessing the qualitative similarity of the two skills. Reference examples are provided in-context to enrich the model's understanding of the skills. If the model deems the skills too similar, they are flagged and excluded from question generation, as similar skills might lead to simpler questions.

Step 2: Question Generation.
Next, we prompt the LLM to generate a question and a brief solution requiring the application of both skills in the sampled pair. We specify two conditions to ensure high-quality questions: the question should either require an exact answer or specify that an approximate answer is acceptable, and it should ask for only a single final result. In-context, we provide two multi-turn conversations between a human and an AI assistant. These conversations demonstrate the human providing feedback on the AI-generated questions, which the AI then refines. This helps the model anticipate and avoid practical issues, such as insufficient involvement of skills or logical inconsistencies. Appendix <ref> provides examples of the responses of different models in the question generation step. 

Step 3: Solution Attempt.
The model then attempts a solution to the generated question, adopting an adversarial approach to identify flaws such as insufficient information, ambiguity, self-contradiction, or excessive computation. If any issues are found, the model stops solving and clearly states the problems. Otherwise, it completes the solution. During this step, the model does not receive the skill names or reference examples to ensure unbiased problem-solving.

Step 4: Question Validation.

We give LLM the generated question and its solution for validation against a fixed rubric consisting of seven criteria:


    
  * Single Answer Requirement: The question should ask for only one final answer.
    
  * Exact Answer Requirement: There should be only one exact answer, unless approximations are explicitly stated.
    
  * Dual Answer Requirement: The question must necessarily and sufficiently involve the application of both skills, with difficulty comparable to or greater than the reference examples.
    
  * Clarity and Completeness: The question should be clear and contain all necessary information.
    
  * Computational Tractability: The question should not require overly complex computations.
    
  * Realism and Logic: The scenario should be realistic and logically consistent.
    
  * Syntax and Grammar: The question should be grammatically correct and clearly written.


The model uses reference examples and validation exemplars - model generated examples of validating questions, to facilitate this step. We employ majority voting (maj @ 4) to enhance robustness.

Step 5: Final Solution and Re-validation.
For questions classified as valid, we ask the LLM to re-solve the question to obtain a final solution. Reference examples are provided in-context to improve the model's understanding. We use majority voting (maj @ 4) to ensure consistency. If all the answers obtained in this step are unique, indicating potential ambiguity, the question is discarded.

The questions obtained from the above pipeline are further screened by humans. This structured approach not only generates challenging and novel math questions but also ensures their quality through rigorous validation, effectively combining the strengths of AI and human oversight. For detailed examples of prompts used at each step, refer to Appendix <ref>.


    
  * Skill Pair Validation. Firstly, we ask GPT-4 / Claude Opus to validate the skill pair by determining the qualitative similarity of the two skills. We supply the reference examples to the model in-context, to enrich its understanding of the two skills. If the model deems a pair of skills to be similar, they are flagged and not used for question generation. The rationale behind eliminating pairs with similar skills is that such pairs might lead to simple questions. 
    
  * Generation. Next, we prompt GPT-4 / Claude to generate a question and a brief solution such that the question requires the application of both skills in the sampled pair, to be solved. In order to ensure high-quality generations, we explicitly give the model a list of two conditions that the generated question should satisfy: Firstly, the answer to the question should either be exact, or it should be specified in the question that the answer is expected to be only approximately correct. Second, the question should ask for only a single final result. It should not ask for multiple things. We also provided in-context, two multi-turn conversations between a human and an AI assistant of the following form.  The conversation shows the human prompting the AI assistant to come up with a challenging math question requiring the application of two mathematical skills (which may not be similar to the current skills at hand). The AI assistant comes up with a question and the human gives feedback, pointing out the problems with the generated question. Then the AI assistant incorporates the feedback and modifies the question. This goes on till the human is satisfied with the generated question. Using these conversations in context helps the models to understand some of the practical issues that arise in the process and possibly avoid them, e.g.,  insufficient involvement of the skills in the generated question, logical inconsistencies in the questions, etc. Similar to the previous step, we also provide the reference in context to help the model understand the skills and their applications better.
    
  * Solution Attempt. In the next step, we ask the model to attempt a solution to the generated question, but to approach this adversarially: look for flaws such as insufficient information in the question,  ambiguity, self-contradiction, need for excessive computation, final answer being too complicated for autograding,  etc.  If the model encounters any of the above problems in any step of the solution, it is told to not solve the question any further and clearly state the issues it found. Else, it should solve the question to completion. Note that we do not supply the model with the skill names or the reference examples in context in this step.
    
  * Question Validation. In this step, we give GPT-4 / Claude the question generated in the first step and the solution obtained in the second step and ask it to validate the question by evaluating it according to a fixed rubric. If the question does not meet any of the criteria in the rubric, the question is classified as invalid and discarded. The model is given the reference examples in-context to facilitate this validation. The rubric is a set of 7 criteria, each described briefly below:
    
        
  * Single Answer Requirement - The question should ask for only final answer
        
  * Exact Answer Requirement - There should be only one possible exact answer to the question, without the need to make any assumptions or approximations whatsoever, unless explicitly states in the question.
        
  * Dual Answer Requirement - The question must necessarily and sufficiently involve the application of both the skills. To ensure sufficiently rigorous application of both skills, we ask the model to ensure that the difficulty of the application of each skill should be similar to or greater than that in the reference examples.
        
  * Clarity and Completeness - The question should be unambiguous and contain all the information necessary to complete the solution.
        
  * Computational Tractability - Solving the question should not involve very complicated computations. The computations involved should be manageable by a human withing a reasonable amount of time.
        
  * Realism and Logic - The question should involve scenarios that are realistic, plausible, and logically consistent with common sense.
        
  * Syntax and Grammar - The question must be grammatically correct and clearly written to prevent misinterpretation.
    

    A question needs to satisfy all the above criteria to be valid. Further, we provide the model with two types of exemplars to facilitate this step. Firstly, we give the model the reference examples for each skill. Secondly, we also give the model a set of 6 question validation outputs which are manually annotated with human feedback on how the model performed while validating a given question. These validation outputs are not human crafted but instead were extracted from the output of older iterations of the validation prompt. We use majority voting (maj @ 4) <cit.> to increase the robustness of the validation step.
    
  * Final Solution and Re-validation. If the question is classified as valid in Step 4, we ask GPT-4 / Claude to (re-)solve the question to get a final solution. The reference examples for both the skills are provided in-context to improve the model's understanding of the skills. We use majority voting (maj @ 4) to improve the model's performance. Additionally, we investigate the final answer obtained by the model in all 4 trials. If all the answers obtained are unique (i.e. if no single answer has a majority), we discard the question since it hints at the possibility of some ambiguity in the question which was not detected in the validation step.






§ EXPERIMENTS AND FINDINGS



Through our experiments, we demonstrate the difficulty and quality of the MATH^2 while also analyzing the behavior of different models on this task of compositional generalization. Firstly, we evaluate a wide range of models spanning a large range of parameter counts on MATH^2 and compare against their performance on MATH <cit.> which is the base dataset used for extracting skills, showing that the MATH^2 is necessarily harder than MATH. Next, we further demonstrate the difficulty and quality of questions in MATH^2 by showing that they are better in-context exemplars as compared to standardly used exemplars. We describe the experimental setup below.





 §.§ Experimental Setup
 

We follow the pipeline proposed in <cit.> to extract skills from the MATH dataset <cit.>. The MATH dataset encompasses seven high-level topics, allowing us to identify and extract finer-grained skills within each topic and label each question accordingly. At the end of the skill-extraction process, we identify a set of 114 skills. We then remove a few simple skills, such as  and , before using the remaining set to generate questions using the proposed approach. We generate and verify 180 difficult questions to create the MATH^2 dataset. Figure <ref> shows the distribution of skills in MATH^2.


r9.76cm


Comparison of Zero-Shot CoT Performance (Accuracy) on the Generated Dataset vs. MATH Test Set:

GPT-4 Omni demonstrates the least drop in percentage terms (13.42%) whereas MAmmoTH-7B shows the highest relative drop (92.91%). 







Model      MATH^2 (Y)     MATH (X)     % Drop   



GPT-4 Omni          66.85%              77.21%        13.42%  

Claude 3.5 Sonnet     47.78%     73.54%     35.45% 

GPT-4 Turbo             57.22%            73.27%          21.90%    

Gemini-1.5-Pro     43.34%     67.70%     35.98% 

Claude 3 Opus                39.66%            61.20 %       35.20%  

Llama-3-70B-Instruct         22.77%           47.89%       52.45% 

MetaMath-70B       6.11%           26.27%      76.74% 

MAmmoTH-70B         5.00%            19.31%      74.11% 

Mixtral-8×7B-Instruct            12.78%           31.52%      59.45%  

MetaMath-13B       2.79%              21.32%      86.91% 

MAmmoTH-13B         2.23%             10.99%      79.71% 

Deepseek-math-7b-instruct     17.88%     45.05%     60.31% 

Llama-3-8B-Instruct        9.45%          28.62%       66.98% 

Gemma-1.1-7B-Instruct       7.78%          23.36%      66.69% 

MetaMath-7B         2.23%            18.69%        88.07% 

MAmmoTH-7B            0.56%             7.90%      92.91%  

Phi-3-mini-128k-instruct     22.78%     45.14%     49.53% 

Gemma-1.1-2B-Instruct           2.78%             7.52%      63.03% 






 


Table <ref> presents details of the changes made to the questions during the human verification process. Out of 180 question-solution pairs included in MATH^2, 79 underwent some form of modification by the human annotators before being included in the dataset. Out of the 41 questions modified, 3 were minor modifications to improve the clarity of the question. Another 24 modifications were minor modifications (up to 3 words), which nevertheless affected the meaning of the question and changed the final answer. But 14 modifications were significant;  either making the given questions harder, or correcting them, or making them more interesting (i.e., less tedious) for humans.

As for the solutions,  62 out of the 180 solutions originally generated by the model were modified to correct them or improve their clarity. 

In total,  56% of the question-answer pairs in MATH^2 appear exactly as phrased by their LLM creator. 

We evaluate the generated set of questions on a variety of language models, both small and large. Specifically, we assess the MetaMath <cit.>, MAmmoTH <cit.>, Gemmma <cit.>, and Llama-3 series, Phi-3, deepseek-math as well as one Mixture-of-Experts model Mixtral-8×7B-Instruct. Additionally, we include evaluations of larger proprietary models such as GPT-4o, GPT-4 Turbo[Points to  at the time of writing] <cit.>, Gemini-1.5-Pro, Claude 3.5 Sonnet [Points to  at the time of writing] and Claude 3 Opus[Points to  at the time of writing]. We compare the performances of these models on our generated questions against their performance on the MATH dataset <cit.>.

For generating responses, we use the MAmmoTH <cit.> evaluation suite. The responses are graded using a GPT-4 grader, where GPT-4 Omni checks the correctness of a solution response against the ground truth solution. During response generation, we set the temperature to 0 and top_p to 1 for all models. All necessary compute details are discussed in Appendix <ref>



 §.§ Performance across the two datasets: A surprising pattern 
 




Table <ref> shows that all tested models   have significantly lower performance on MATH^2 than on the original MATH dataset. Denoting Y as the performance on MATH^2 and X as the performance on MATH, the percentage drop 100(X- Y)/X
for frontier models — GPT-4 Omni, GPT-4 Turbo, Gemini-1.5-Pro, Claude-3.5-Sonnet and Claude 3 Opus — ranges from 13.42% to 35.45%. 
MAmmoTH-7B, a specialist math model, shows the largest drop at 92.91%. 




The fact that performance drops for all models should not be too surprising, since as noted, the 
MATH^2 questions, by combining skills from different subareas of MATH, could be seen as  “out of distribution (OOD).” This makes it tempting to interpret the percentage drop as a measure of a model's (lack of) “OOD-resilience.” For instance,
very large percentage drops seen with open-source models MetaMath and MAmmoTH feel understandable since their training used synthetic data generated using seed questions from MATH and GSM-8k. Lack of diversity in such synthetic data is known to cause overfitting to the  dataset being imitated.  Similarly, GPT-4o and Claude Sonnet 3.5 are suspected to also have been extensively trained with synthetic data.  Although their MATH performance is similar, Sonnet 3.5 has worse MATH^2 performance,  which might suggest lower quality/diversity in its synthetic data.

However,  in our opinion, the overall pattern among proprietary models of similar size does fit with the OOD story.


A much simpler explanation pops out when we plot Y vs X^2 (Figure <ref> and Figure <ref>): we find a linear relationship Y ≈ X^2!

This implies that the relative drop in performance of the models is well-predictable from just their performance on MATH, and does not require taking their training details into account! 

Why should the two scores be expected to have this relationship? Here is a natural (albeit heuristic) explanation.  Suppose there are N skills and s_i denotes the success rate of the model at correctly applying the ith skill. Then,  its X value should reflect the average[With perhaps a small correction factor if the skills are not evenly distributed among the questions] of the s_i's. Furthermore, on a random question using the ith and jth skill, the probability that the model correctly answers it should be  s_i s_j, since it has to successfully apply both skills. 
If the questions are created using  pairs of skills chosen randomly and independently, then the Y value will be the average value of  s_is_j's, which by independence will be roughly X^2. 

This reasoning in fact  suggests that our pipeline has created  questions
that genuinely required applying two very distinct skills (as opposed to, say,  requiring primarily skill i, and mildly using  skill j). The discovered relationship suggests further that if we could create questions where each combines k skills, we might see the relationship Y ≈ X^k, which would tend to further magnify performance differences between models. 







 







 §.§ Generated Questions are Effective In-Context Exemplars for MATH.
 


A possible test for the quality of a Q&A pair on similar topics as MATH dataset is whether performance on MATH improves when using these as in-context exemplars. 

We test as follows. Recall that MATH has 7 sections. Exemplars for a section are chosen from the section area.  However, by design, our new questions cross section boundaries.  We implemented a new procedure to retrieve in-context exemplars from MATH^2 based on the skill requirements of the current question.

Since MATH^2 is limited in size, it does not cover all the skills extracted during the skill extraction process, containing 97 out of 114 skills. Figure <ref> shows the distribution of different skills in the dataset. We filtered the MATH test set to remove examples requiring skills not present in the generated dataset, resulting in the removal of 809 test examples. During evaluation on the filtered MATH test set, for each question Q labeled with skill a (a ∈𝒮, where 𝒮 is the set of extracted skills), we retrieved in-context exemplars from the MATH^2, ensuring each exemplar involved skill a. We used four such exemplars per question (i.e., 4-shot CoT <cit.>). To handle skills represented by fewer than four examples in MATH^2, we run two experiments: (A) Proposed 4-shot CoT: If a given skill is represented by n examples in the MATH^2, where n < 4, we use n in-context examples instead of 4 exemplars. (B) Proposed + Skill Based 4-shot CoT: If a given skill is represented by n examples in MATH^2, where n < 4, we supplement 4-n exemplars for that skill from MATH training set. The relevant in-context exemplars in MATH training set are determined by following the methodology proposed in <cit.>. We compared the performance of models using these targeted prompting strategies against two baselines: (C) MAmmoTH 4-shot CoT: The 4 in-context exemplars are taken from the MAmmoTH evaluation suite <cit.>. (D) Skill Based 4-shot CoT: We use skill-based prompting as proposed in <cit.>, where the in-context exemplars are selected from the MATH training set, in accordance to the skill required by the question at hand, as determined by GPT-4. 

Table <ref> presents the results of this comparison. The two prompting strategies using questions from MATH^2 as in-context exemplars, clearly outperform the two baselines. We conclude that the MATH^2 questions, due to their difficulty and skill relevance, serve as effective in-context exemplars. Performance gains would likely be more significant with larger datasets generated using our approach, reducing the need to supplement with external exemplars.

















§ OBSERVATIONS FROM THE QUESTION GENERATION PROCESS




The question generation pipeline described in Section <ref> was developed through an iterative process of refining prompts and design choices, and evaluating their impact on the quality of the final questions and solutions. Notably, the inclusion of the attempted solution and question validation steps significantly enhanced the pipeline's effectiveness. Despite the sophistication of the pipeline and prompts, we still observe instances where models fail to follow the given instructions. This section highlights prominent failure modes at various stages of the pipeline, which human raters need to be aware of. Additionally, we explore some intriguing behaviors of the models where they successfully create interesting and creative questions. Section <ref> details the role of human raters in improving these questions.





 §.§ Creative questions: Examples of Synergy from Human-AI interaction 
 




The models frequently produced interesting and creative questions, although they often failed to generate correct solutions. In these cases, the incorrect solutions usually contained enough correct ideas for a human to quickly complete them.

Human annotators were tasked with verifying the validity of the questions and the correctness of the solutions. They were instructed to look out for any failure modes discussed in Section <ref>. Their responsibilities included ensuring that the created questions actually employed the intended math skills, and improving the questions in terms of readability, quality, and difficulty when possible. They were encouraged to suggest changes that would make the problems harder for automated tools to solve while allowing easier or more elegant solutions for humans. The following examples illustrate this process:





Our human team had not encountered such questions before. It requires recognizing that k^3 - k = k(k-1)(k+1) is always divisible by 2 and 3. Thus, k must be such that k(k-1)(k+1)/6 is divisible by 15 (both 3 and 5). Additionally, the sum of the digits of k must be a prime number, and ensuring such conditions is challenging even for powerful LLMs.





The original question combined elements in a novel way. The human rater modified the question to change the sequence size from 20 to 6 beads, maintaining the essential difficulty while making it more elegant for humans. All tested models failed on the modified question.







This was one of many questions the models created about exponential growth and geometric series, possibly similar to standard math test questions. The human slightly altered it to simplify calculations by hand and substituted a different condition that the models found challenging, while humans could easily estimate an approximate answer and then verify.







An LLM can solve the original question through simple computation. The modified question, however, requires understanding an underlying pattern.







Models tend to adopt a brute-force approach to the original question by calculating 15^4 + 16^4. After rephrasing, the number 23^17 + 17^17 is too large for direct computation, requiring understanding of arithmetic modulo a prime.

These examples highlight the essential role of human oversight in refining and improving the questions generated by LLMs, ensuring they are challenging, creative, and suitable for advanced mathematical problem-solving.








 §.§ Failure Modes





Despite the sophistication of our pipeline, models frequently exhibit several failure modes: (a) Insufficient Involvement of Skills: Models often generate questions that either miss one of the skills completely or require a very shallow application of one or both skills. For example, a geometry question may fail to involve ratio and proportion adequately, (b) Insufficient Information: Questions may lack essential details needed for solving, making them incomplete or ambiguous. For instance, a trigonometry question might omit necessary angles or distances, (c) Unsolvable or Computationally Intractable Questions: Some questions generated are either unsolvable or require excessive brute-force calculations, which are impractical for evaluating reasoning abilities, (d) Nonsensical Questions: Models sometimes produce questions that are logically inconsistent, confusing, or ambiguous, such as a probability problem with unclear parameters or an impossible geometry scenario, (e) Deceitful Solutions: Occasionally, models fabricate solutions to nonsensical or unsolvable questions, presenting incorrect logic as plausible reasoning and (f) Finding a Needle in the Haystack: Long and complex validation prompts sometimes cause models to confuse or overlook the specified skills, leading to incorrect evaluations. For a more detailed discussion and examples of questions in the various categories listed above, refer to Appendix <ref>.
 















§ CONCLUSIONS






We introduced a framework that leverages the complementary strengths of humans and AI to generate new, challenging mathematics questions. Building on recent insights into LLM metaknowledge, we use LLMs to extract and name key skills necessary for solving math problems. Using these insights, we developed a pipeline that employs named skills from the well-known MATH dataset, and leverages multi-turn interactions with advanced LLMs to generate questions that combine pairs of skills. These questions were subsequently reviewed and refined by human raters. The proposed pipeline produced questions with greater novelty and difficulty compared to those in the original MATH dataset.

This framework also resulted in a new math evaluation MATH^2, that assesses the same skills as the MATH dataset but is significantly more challenging for leading models because each question involves two skills from different parts of MATH.   GPT-4-T and GPT-4-O exhibited the smallest performance drops, which aligns with the subjective evaluations of our human raters. Additionally, we demonstrated that providing the newly generated questions as in-context examples improved GPT-4-O's performance on the MATH dataset more effectively than examples sourced directly from the MATH dataset. This finding further validates the quality of the questions produced by the proposed pipeline.

Open-source models performed disappointingly on these new questions, but this weak performance also suggests that their performance should be improveable using  novel questions of only moderate difficulty —- which our proposed pipeline can generate at scale. 

We plan to release detailed information about our pipeline to encourage further research and development in the field of open-source math models. 

Limitations and Future Work.  Our pipeline incurs moderately high costs due to  extensive API-based use of frontier models as well as significant human verification. To improve efficiency, future work should focus on optimizing prompting strategies to produce higher-quality questions initially, thereby reducing the need for extensive filtering. Additionally, reducing human verification through the development of automated validation tools is crucial. Enhancing our pipeline by integrating a training-based feedback loop, where the model is trained on  the questions that pass human verification, could further streamline the process by progressively improving question quality. 

These measures will reduce dependency on expensive proprietary models, lower overall operational costs, and maintain or even enhance the quality of the generated math evaluation benchmarks.

Looking ahead, an even more exciting prospect is the potential application of the proposed framework to efficiently produce high-quality data in domains beyond mathematics. 




§ ACKNOWLEDGEMENTS

This research used compute resources provided by Mila (mila.quebec) and  GPT4 access as well as compute resources provided by Princeton Language and Intelligence (PLI).  The Princeton participants were funded by NSF, DARPA, and PLI.  VS would like to thank Aniket Didolkar for helpful discussions throughout the project and for proof reading the paper. AG would like to thank Melvin Johnson, James McClelland and Yoram Bachrach for helpful discussions and useful feedback. AG would also like to thank Daan Wierstra, Melvin Johnson, Siamak Shakeri, Murray Shanahan, John Quan, Theophane Weber, Olivier Tieleman, David Silver, Charles Blundell, Behnam Neyshabur, Ethan Dyer and Nicolas Heess for support and guidance.  










abbrvnat












§ APPENDIX


Here we further analyze the quirks of the question generation pipeline and provide additional experimental details and results. In  Appendix <ref>, we discuss several failure modes of the models that we notice during the question generation process as well as interesting behaviors exhibited by the models throughout the pipeline and interesting creative questions that the models came up with. Appendix <ref> discusses the different considerations that human annotators were instructed to take into account while annotating and verifying the questions generated by the proposed pipeline. Appendix <ref> provides details about the compute used for running our experiments as well as some further analysis of the model evaluations. In Appendix <ref> we provide examples of questions generated by different models in the Question Generation step (Section <ref>). Appendix <ref> gives a detailed description of the prompts used for each step in the question generation pipeline as well as evaluation of the models. It also provides a link to the skill exemplar repository and in-context exemplars used in the question generation process.



 §.§ Failure Modes and Interesting Behaviors





  
Insufficient involvement of skills. Despite clearly specifying that solving the question should necessarily require a rigorous application of both skills, the models often generate questions that either miss one of the skills completely or require a very shallow application of one (while the other one is sufficiently involved) or both skills. This is the most prominent failure mode of the models in the context of question generation. This leads to potentially easy questions, defeating the purpose of skill composition. Consider the question given below which was generated by Claude Opus when asked to combine the skills  and . 



    


Upon careful examination of the question, we note that although the question tests , the involvement of  is practically non-existent. Further, the question validation step in some cases also fails to identify these flaws. Supplying multi-turn human-AI interactions where the user prompts a chatbot to generate a question combining two skills, in-context during the generation step helps the models to avoid such questions to a certain extent. Further, to make the question validation step more robust to such questions, we prompt the model to ensure that the complexity of each skill application in the question being validated in similar to or more than the complexity of these skills in the reference examples present in the skill descriptions. The combination of these two techniques helps us nearly eliminate questions where the absent one of the skills is absent completely and reduce questions involving shallow application of skills to a significant extent.



  
Insufficient information in the questions. Another common failure mode of the pipeline is the generated questions missing information or details essential for solving the question. For example in the question given below which is supposed to combine the skills  and , lacks sufficient detail about the inclinations and elevations of the paths relative to the streetlight's position which is necessary to answer the question.


    


To screen such questions, we include and explicit clause in the question validation prompt as described in Section <ref>. Moreover, we also notice that the inclusion of the solution attempt step improves the chances of detecting such errors since the missing information may not always be apparent from just the question itself. In such cases, attempting a solution (with a defeatist approach) can help detect such flaws.





  
Unsolvable or Computationally Intractable Questions. There are instances when the model generates questions which are unsolvable. For example the question given below has no solution which satisfies all three constraints (i.e., the area of the rectangle being 360 and the sides belonging to the two arithmetic progressions defined in the question.) 


    


In other instances, the model generates questions that are computationally intractable or require manually and tediously iterating through a long sequence of values. For example, solving the question given below requires manually calculating the first 100 terms of the sequence to find the sum 


    


While technically not wrong, such questions are not ideal for evaluating the reasoning abilities of the models since they mostly involve brute force calculations. Further, in cases where the sequence of calculations is very long, the LLM's performance may be bottlenecked by other limitations such as the context length of the model. 

Thus, we strive to filter such questions out. We add an explicit condition to check for computational tractability and solvability of the generated questions in the verification prompt. This check is assisted by the solution attempt produced by the model which will potentially point out any such problems. 



  
Nonsensical Questions. In several cases, the model comes up with questions which are nonsensical - confusing, incomprehensible, logically inconsistent or ambiguous. Consider the question given below.

Given below is an example of a question which is logically inconsistent. More concretely, a square plot of land whose side length is equal to the radius cannot fit inside the quarter-circle.


    


We add checks for such cases in the question validation prompt. Further, at the end of the final solution step (maj @ 4), we further check for cases where the final answer produced in all the 4 self-consistency trials are unique. If all answers are unique, we discard the question. The rationale behind this being that it is highly likely that the model produces a different answer every time due to some inherent ambiguity in the question which was not detected in the solution attempt and the question validation checks. 





  
Deceitful Solutions. Although rare, we encounter cases where the model makes up solutions even though the question is nonsensical or cannot be solved with the amount of information provided in the question. This happens very commonly in the solutions which are generated in the question generation prompt. Thus, we do not use these solutions and include the final solution step where the model is asked to solve the question again. Although most of such solutions and thus questions are screened out in the question validation step and consistency check at the end of the final solution check, in rare cases we see this behavior in the solution produced after the final solution step as well. Given below is one such example. 


    


While solving this question, the model arrives at the conditions sin(θ) = 2 or sin(θ) = -2. Clearly, these conditions have no solutions since -1 ≤ sin(θ) ≤ 1, ∀θ∈ [0, 2π]. However, the model goes on to argue that the the closest values to 2 and -2 in the range of sin(θ) are 1 and -1, and thus, goes on to solve the question for sin(θ) = 1 or sin(θ) = -1.



  
Finding a needle in the haystack. In an attempt to make the question validation step robust to as many failure modes as possible, we come up with a long and sophisticated prompt (see Appendix <ref> for an example of this prompt). To elaborate, the validation prompt comprises of 1.) skill descriptions consisting of 3 exemplars for each of the 2 skills, 2.) 6 very long validation exemplars 3.) rest of the prompt consisting of the 7 conditions described in Section <ref>. In prior iterations of the validation prompt, the names of the two skills which are supposed to be combined in the question, were mentioned only once throughout the prompt. In such a case, we observed that more often than not, model confused the pair of skills at hand, with skills mentioned in the validation exemplars. We also observed cases where the model was unable to locate the original skill names, came up with its own skill names depending on the skill exemplars provided in the skill descriptions. More specifically, while checking the question for Dual Skill Requirement in Section <ref>, the models would check the question for the presence of skills distinct from the pair of skills at hand. We attribute this phenomenon to the the lost in the middle or finding a needle in the haystack challenges faced by LLMs. We provide an example of the phenomenon below. Notice how in Point 3 of the reasoning trace the model checks the question for the presence of two completely unrelated skills. We were able to nearly eliminate this problem by replacing referential terms (such as above given skills, these skills, etc.) to the two skills by the skill names.


    

    

     

    

    

    

    

    
        
        

    

    

    

    

    

    

    

    


Note that none of the above failure modes are completely eliminated in the pipeline described in Section <ref>. Thus, human verification is required.

Despite struggling with the failure modes described above, there also exist cases where the models exhibit positively surprising and creative behaviors. We talk about some of them below.



  
Thinking out of the box. Although rare, we observe instances where the models get creative while validating the question. Consider the question below


    


When validating this question using prior iterations of the question validation prompt, which did not consist of the computational tractability check, the model output while validating the question consists of the following excerpt.


    ...This might introduce a significant challenge not solely due to the methodology's complexity but also due to the potential computational requirement, which may not be feasible in a standard test environment without tools. Furthermore, while the connection to practical geometry (reflective and rotational symmetry) and combinatorics (color patterning and adjacency constraints) is strong, the depth of understanding required to manually adjust for these symmetry considerations in a test question might be too intense or require more guided learning than a single evaluation question could provide....


i.e, the model takes into consideration the fact that the question involves a lot of brute force computation, despite there being no explicit check for computation complexity in the prompt, and classifies the question as invalid. We attribute such out of the box thinking behavior to the role-playing nature of our prompts. Our prompts consist of a math teacher evaluating the the fitness of the given question for being used for testing students' reasoning and analytical skills in a math exam. This leaves room open for the model to detect potential problems not explicitly accounted for in the prompts which might make the question unfit for being used for evaluation.




















































    











    
















 §.§ Considerations for human-annotaters



Human annotators were tasked with double checking the validity of the question and the correctness of the solution. They were asked to look out for any of the failure modes discussed in Section <ref>.
They were asked to check that the created question actually used the math skills it was supposed to exhibit and to improve the question with respect to readability, quality and difficulty. They were encouraged to suggest changes that  make the problem harder to solve using automated tools while retaining easiness for the humans. We illustrate with an examples. 

GPT-4 created the following question given the skill-tags 
 and   :


    


An LLM can solve this by simple computation. The human modified the question so that solving the problem requires understanding the underlying pattern. 


    


For the modified question, one leading model mentioned calculation difficulties for the inability to give any answer, and another resorted to an incorrect numerical approximation that led to an incorrect answer. 

Human annotators were also asked to go through the solutions carefully and correct or improve the solution for good questions if necessary. They were also asked to look out for questions that contain lot of enumeration, i.e. questions which are tedious and require significant amount of brute force computation. For such questions, the annotators were encouraged to reword them such that enumeration is not a feasible strategy below. For example, given below is an example of an enumerative question which was modified to avoid enumeration.


    

    


 Models tend to adopt brute force approach on the original question calculating 15^4 + 16^4. After rephrasing the models cannot use brute force on 23^17 + 17^17, instead being forced to check the divisors more analytically, in particular understanding of arithmetic modulo a prime.









 §.§ Further Experimental Details and Results


For open source LLMs, we use 2 80GB A100 GPUs and 72GB of RAM to run inference facilitated by vLLM <cit.>. For evaluating GPT-4 Omni and GPT-4 Turbo, we use 25 parallel workers to make the API calls, and for Claude-3 Opus we use 2 workers.

Analysis of performance degradation on smaller models. Figure <ref> 
shows that smaller models generally experience larger relative performance declines compared to larger models in the same family.

It is known that larger models are better at composing skills or concepts (a phenomenon known as "compositional generalization") than smaller models. This finding aligns with conclusions drawn in <cit.>.

Skill Representation in the Generated Dataset. Figure <ref> shows the distribution of different skills in the generated set of questions. Figure <ref> shows the comparison in relative performance drops between models of different sizes belonging to the same family.



 §.§ Skill composition using different models


In this section, we provide examples of the responses of different models when asked to generate a question combining two different skills. We prompt GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Llama-3-70B-Instruct and Llama-3-8B-Instruct to generate a question combining the skills  and . We use the same prompt used in the Question Generation step (Step 2) of the proposed pipeline described in Section <ref> (see Appendix <ref>). Given below are the responses of each model to the prompt.

[GPT-4 Omni]
Sure! Here is a challenging math question that tests both area calculation skills and algebraic manipulation skills, formatted as requested.


# QUESTION

A rectangle ABCD has vertices A(0, 0), B(a, 0), C(a, b), and D(0, b). Inside this rectangle, there is a right triangle PQR where P is at vertex A, Q is on BC, and R is on CD. The area of triangle PQR is given by 1/6 of the area of rectangle ABCD. Given that the coordinates of Q and R are (a, kb) and (ka, b) respectively, where k is a constant, find the value of k.


# SOLUTION

First, let's find the area of the rectangle ABCD:


    Area of rectangle  ABCD = a · b



The area of the triangle PQR is given as 1/6 of the area of the rectangle ABCD:


    Area of triangle  PQR = 1/6· (a · b) = ab/6



Next, we need to find the area of triangle PQR using the coordinates of P, Q, and R.

The vertices of triangle PQR are P(0, 0), Q(a, kb), and R(ka, b).


The formula for the area of a triangle with vertices at (x_1, y_1), (x_2, y_2), and (x_3, y_3) is:


    Area = 1/2| x_1(y_2 - y_3) + x_2(y_3 - y_1) + x_3(y_1 - y_2) |



Substitute the coordinates of P, Q, and R into the formula:


    Area of  PQR = 1/2| 0(kb - b) + a(b - 0) + ka(0 - kb) |



    Area of  PQR = 1/2| 0 + ab - ka^2b |



    Area of  PQR = 1/2| ab(1 - k) |



    Area of  PQR = ab/2 |1 - k|



We know that the area of triangle PQR is ab/6:


    ab/2 |1 - k| = ab/6



To solve for k, we can divide both sides by ab/2:


    |1 - k| = 1/3



This gives us two possible equations:


    1 - k = 1/3  or   1 - k = -1/3



Solving these equations for k:


    1 - k = 1/3  ⇒   k = 1 - 1/3 = 2/3



    1 - k = -1/3  ⇒   k = 1 + 1/3 = 4/3



Therefore, the possible values for k are 2/3 and 4/3.


# DETAILS

The question involves area calculation by requiring students to find the area of a triangle within a rectangle, and algebraic manipulation by solving an equation involving a variable. The problem complexity is similar to the examples provided, ensuring that students are sufficiently challenged. The final answer consists of the value(s) of k, making it straightforward to evaluate the students' answers.


[Claude 3 Opus]
Here is my attempt at generating 1 (non multiple choice) difficult question which tests both area_calculation_skills and algebraic_manipulation_skills:


# QUESTION

Consider an equilateral triangle ABC with side length 2a. Let points D, E, F be the midpoints of sides BC, CA, AB respectively.
 
Now construct squares ACDE, ABFG, BCHI externally on sides AC, AB, BC respectively. 

Express the area of the shaded region (consisting of the equilateral triangle DEF and squares ACDE, ABFG, BCHI) in the form ka^2, where k is a constant. Find the value of k.


# SOLUTION

Let the side length of the equilateral triangle ABC be s = 2a.

The area of equilateral triangle ABC is √(3)/4s^2 = √(3)/4(2a)^2 = √(3)a^2.


In an equilateral triangle, the distance from a vertex to the midpoint of the opposite side is √(3)/2 times the side length. So DE = EF = FD = √(3)/2s = √(3)a.


The area of equilateral triangle DEF is √(3)/4(√(3)a)^2 = 3√(3)/4a^2.


The side length of each of the squares ACDE, ABFG, BCHI is equal to the side length of the equilateral triangle ABC, which is s = 2a.
 
So the area of each square is s^2 = (2a)^2 = 4a^2.


The total area of the shaded region is:

Area of DEF + 3 * Area of each square

= 3√(3)/4a^2 + 3(4a^2) = 3√(3)/4a^2 + 12a^2 = (12 + 3√(3)/4)a^2


Therefore, the area of the shaded region is ka^2, where k = 12 + 3√(3)/4≈ 15.1981.


# DETAILS

This question combines area calculation of equilateral triangles and squares with algebraic manipulation of side lengths and areas. Key steps involve:


1) Calculating areas of equilateral triangles ABC and DEF in terms of side length s and a respectively. This tests area_calculation_skills.


2) Algebraically manipulating the expressions for side lengths DE, EF, FD and square side lengths in terms of s and a. This requires algebraic_manipulation_skills.


3) Adding up the component areas algebraically and simplifying the expression in the form ka^2 to solve for constant k. 


The question difficulty is calibrated to be similar to the provided examples. The solution is a single numerical constant k which avoids ambiguity in student answers. The equilateral triangle and square constructions provide an interesting geometric visualization.


Let me know if you would like me to modify anything in the question or solution to better suit your needs! I'm happy to refine it further.




[Gemini 1.5 Pro]
# QUESTION

A circle is inscribed in equilateral triangle ABC, and a smaller circle is externally tangent to this circle and to sides AB and AC.  If the side length of triangle ABC is 4, what is the area of the smaller circle?


[asy] size(150); defaultpen(linewidth(0.8)); pair A=(0,2*sqrt(3)), B=(-2,0), C=(2,0), O=(0,sqrt(3)/3), P=(0,sqrt(3)); draw(A–B–C–A); draw(circle(O,sqrt(3))); draw(circle(P,sqrt(3)-1)); [/asy]


# SOLUTION

Let O be the center of the larger circle, and let P be the center of the smaller circle.  Let r be the radius of the smaller circle.


[asy] size(150); defaultpen(linewidth(0.8)); pair A=(0,2*sqrt(3)), B=(-2,0), C=(2,0), O=(0,sqrt(3)/3), P=(0,sqrt(3)); draw(A–B–C–A); draw(circle(O,sqrt(3))); draw(circle(P,sqrt(3)-1)); draw(O–P); draw(O–C); draw(P–C); label("A",A,N); label("B",B,SW); label("C",C,SE); label("O",O,S); label("P",P,NE); label("2",(1,0),S); label("r",(P+C)/2,E); label("1",(C+O)/2,SE); [/asy]


Since O is the centroid of equilateral triangle ABC, OC = 2/3· 2 √(3) = 4 √(3)/3.  Also, PC = r + 1, so by the Pythagorean Theorem on right triangle POC,


    r^2 + 2r + 1 = PC^2 = OC^2 - OP^2 = 16/3 - (r + √(3))^2.
This simplifies to 2r^2 + (2 + 2 √(3)) r - 13/3 = 0.  By the quadratic formula,


    r = -(2 + 2 √(3)) ±√((2 + 2 √(3))^2 - 4 · 2 · -13/3)/4 = -1 - √(3)±√(40)/4.
Since r is positive,


    r = -1 - √(3) + 2 √(10)/4.
Then the area of the smaller circle is


    π r^2 = π( -1 - √(3) + 2 √(10)/4)^2 = 27 π/8 - 13 √(3)π/8 + 5 √(30)π/4.



# DETAILS

This question tests both area calculation skills (area of a circle) and algebraic manipulation skills (solving a quadratic equation with radicals).  The problem is challenging because it involves nested radicals, but the steps are all fairly standard. 




[Llama-3-70B-Instruct]
Here is a difficult question that tests both area calculation skills and algebraic manipulation skills:


# QUESTION

A square with side length 4 is inscribed in a circle. A semicircle is drawn inside the square, with its diameter being one of the sides of the square. Find the area of the shaded region.


[asy]

size(100);

draw(unitcircle);

draw((-2,0)–(2,0)–(2,2)–(-2,2)–cycle);

filldraw((-2,0)–(2,0)–(2,2)–(-2,2)–cycle,lightgray);

[/asy]


# SOLUTION

The area of the shaded region is equal to the area of the square minus the area of the semicircle. The area of the square is 4^2 = 16. The area of the semicircle can be found using the formula 1/2π r^2, where r is the radius of the semicircle. Since the diameter of the semicircle is 4, the radius is 2, so the area of the semicircle is 1/2π(2)^2 = 2π. Therefore, the area of the shaded region is 16 - 2π.


# DETAILS

This question requires students to calculate the area of a square and a semicircle, and then subtract one from the other to find the area of the shaded region. It also requires algebraic manipulation skills to simplify the expression for the area of the semicircle. The answer is not a simple numerical value, but rather an expression involving π, which adds an extra layer of complexity to the question.



[Llama-3-8B-Instruct]
Here is a question that tests both area calculation skills and algebraic manipulation skills:


# QUESTION

Find the area of the region bounded by the curve y=x^2-4x+3 and the x-axis, to the nearest hundredth.


# SOLUTION

The area of the region can be found by integrating the function y=x^2-4x+3 with respect to x from 0 to 2, which is ∫_0^2 (x^2-4x+3) dx = [x^3/3 - 2x^2 + 3x]_0^2 = (8/3) - 8 + 6 - [0] = 2. The area can also be found by using the formula for the area of a region bounded by a curve and the x-axis, which is ∫_0^2 y dx = ∫_0^2 (x^2-4x+3) dx = 2. Therefore, the area is approximately 2 square units, to the nearest hundredth.


# DETAILS

This question tests both area calculation skills, as it requires the student to find the area of a region bounded by a curve and the x-axis, and algebraic manipulation skills, as it requires the student to integrate a function with respect to x.


Note: I've also reviewed the question and solution to ensure that they are accurate and clear for students.





 §.§ Prompt Examples


We detail the prompts used in the various stages of question generation and evaluation in the following sections. We present the skeleton of the prompts with placeholders for skill names, in-context exemplars, etc. in favor of keeping the prompt brief. The in-context exemplars and the skill exemplar repository are made available https://anonymous.4open.science/r/ai_assisted_questions-131A/here.



  §.§.§ Skill Pair Validation



[Prompt for validating a pair of skills]
I have two basic mathematical skills described below, each followed by some sample question - solution pairs such that solving the questions involves the use of the particular skill. 


SKILL 1: <skill_1>


Question 1: <skill_1_question_1> 

Answer 1: <skill_1_solution_1>


Question 2: <skill_1_question_2>

Answer 2: <skill_1_solution_2>


Question 3: <skill_1_question_3>

Answer 3: <skill_1_answer_3>


SKILL 2: <skill_2>


Question 1: <skill_2_question_1>
 
Answer 1: <skill_2_solution_1>


Question 2: <skill_2_question_2> 

Answer 2: <skill_2_solution_2>


Question 3: <skill_2_question_3>

Answer 3: <skill_2_solution_3>


I am going to use these two skills for framing a new question such that the question requires an expertise in both the skills in order to be solved, i.e. the question will compose these two skills. However, I do not want the two skills to be very similar, i.e., they should not mean the same thing. Go through the descriptions of the skills carefully. Based on your understanding of the skills, can you please tell me whether the two skills are essentially entirely the same or not? Think step by step and give a detailed explanation of your answer. The answer should begin with a prefix '# EXPLANATION
'. Note that your understanding of the skills should not be restricted to the sample questions provided previously. They are just example questions. Use your own prior knowledge as well. End your response with a 'Yes' or 'No' answer to whether the skills are similar or not. This final answer should be on a new line and preceded by the prefix '# FINAL ANSWER
'. Thank you very much!









  §.§.§ Question Generation



[Prompt for question generation]
I am a math teacher trying to create challenging math questions for smart students. I was wondering if you could give me 1 (non multiple choice) question which tests both the following skills:
 (<skill_1>, <skill_2>)
Please also provide a brief solution. Then please look over the question and the solution, and fix any issues so that my students do not get frustrated. This being a math exam, the answers should either be exact, or if not possible, then the question should clearly say the answer is only expected to be approximately correct. Further, for ease of evaluating the students' answers, the question should ask for a single final result.This process is difficult so I am attaching two sample conversations where (Agent) is an AI agent and (Query) is teacher feedback. The conversations revolve around framing such mathematical reasoning questions and using them for evaluating students. These should give you some idea of the expectations and the potential difficulties involved in this task. I am also giving three example question - answer pairs for both <skill_1> and <skill_2> skills, such that the example questions test the corresponding skill. Please ensure that the complexity / difficulty of application of <skill_1> and <skill_2> skills in the generated question is similar to the complexity / difficulty of the skills in the example questions. Please format your output as
 

'# QUESTION

<question>


# SOLUTION

<solution>


# DETAILS

<all other text>'


SKILL 1: <skill_1>


Question 1: <skill_1_question_1>
 
Answer 1: <skill_1_solution_1>


Question 2: <skill_1_question_2>

Answer 2: <skill_1_solution_2>


Question 3: <skill_1_question_3>

Answer 3: <skill_1_solution_1>


SKILL 2: <skill_2>


Question 1: <skill_2_question_1>
 
Answer 1: <skill_2_solution_1>


Question 2: <skill_2_question_2>
 
Answer 2: <skill_2_solution_1>


Question 3: <skill_2_question_3>

Answer 3: <skill_2_solution_3>


# CONVERSATION 1

<agent_convo_1>


# CONVERSATION 2

<agent_convo_2>





    




  §.§.§ Attempted Solution



Prompt for solution attempt. Note that we instruct the model to take a defeatist approach towards solving the question

[Prompt for solution attempt]
You are a professional math teacher and you are given a question which is supposed to test the analytical and mathematical reasoning abilities of your students. You are supposed to provide a solution to the given question. However, the question may be flawed. For example, it might have problems like question being unsolvable using the information provided, question being self-contradictory, the final answer being computationally intractable, the question being ambiguous and confusing, question having multiple possible interpretations, etc., which you may encounter while solving the problem. This question being used for evaluating students in math, the question should ideally have a single, exact answer, with no room for any deviations due to factors such as approximations, rounding errors, etc., unless explicitly specified in the question. Problems such as the ones described above, would prevent the students from solving the question properly, and thus, any question with either of these problems is unfit for testing the students. If you encounter any such problems, stop the solution right there and explain the problems. For example, if you encounter the need to make any approximations or rounding which is not specified in the question, stop solving the question along with the reason. You do not need to solve the question further once you encounter any such problem. If you do not encounter any such problem, solve the question to achieve the single exact answer which the question asks for.


# QUESTION

<question>








  §.§.§ Question Validation


Note that how in the first paragraph, the names of the two skills are mentioned even time instead of using referential phrases. This is done to address the lost in the middle problem

[Prompt for validating the questions]
You are a professional math teacher. You want to evaluate the analytical and mathematical reasoning abilities of your students in a math exam. The students are supposed to sit in an examination hall and solve the questions within a given time limit, without access to any computational devices. The evaluation is designed to test the students' expertise in using two given mathematical skills simultaneously, namely <skill_1> and <skill_2>. This is achieved by asking them to solve a question that necessitates expertise in both <skill_1> and <skill_2> skills, to be solved completely. Since evaluating the students is a critical task allowing very little margin for any error in the process, it is very important to ensure that the questions used for evaluating are high quality and fit for being used to evaluate the students. You need to carefully review the question and a given attempt at solving it, and ensure that the question is of high quality and fit to assess students. In order to do this, you should check the quality of the question with respect to several criteria, such as:


- Single Answer Requirement: The question should ask for one and only one final result. It should not request multiple distinct answers or pieces of information.

- Exact Answer Requirement: It should be possible to achieve one,exact answer to the question, without the need of making any approximations or assumptions whatsoever, unless explicitly specified in the question. There should be no margin for the students to arrive at any other possible answer due to things like rounding errors, etc.

- Dual Skill Requirement: The question must require rigorous expertise in both a: a) '<skill_1>' and b) '<skill_2>', for resolution. Application of both <skill_1> and <skill_2> and their subskills should be, necessary and contribute directly to obtaining the final answer; <skill_1> and <skill_2> skill should be applicable separately and critically during the problem-solving process. You are also given three example question - answer pairs for both <skill_1> and <skill_2> skills in order to help you better understand the meaning of each skill. Please carefully review the question and its attempted solution, paying close attention to how well it aligns with the examples provided for each skill. Consider the depth and breadth of knowledge demonstrated in the examples. The complexity / difficulty of application of both <skill_1> and <skill_2> in the question should be similar or greater than the complexity / difficulty of <skill_1> and <skill_2> in the example question-answers given for that respective skill. 

- Clarity and Completeness: The question should be unambiguous and contain all the information necessary to complete the solution. Any required assumptions not common knowledge should be explicitly stated. Check for any ambiguity that might confuse students. Carefully go through the solution to check if it makes any assumption or approximation in order to solve the question.

- Computational Tractability: Since the students are supposed to solve the questions within a given time limit and without access to any computational devices such calculators, computer, mobile phones, etc., you must ensure that the question is computationally tractable and all the computations involved can be done by hand in a limited amount of time.

- Relevancy of Information: The question should not have any extra details that do not contribute to the solving of the problem.

- Realism and Logic: The question should involve realistic scenarios or hypotheses with logically consistent data. The specified operations and the contextual setup should reflect plausible mathematical situations. (e.g., positive amounts for transactions, integers for counts).

- Syntax and Grammar: The question must be grammatically correct and clearly written to prevent misinterpretation.

- etc. (any other problems which you think make the question not fit for being used for evaluating the students)

Your task is to give a 'Yes' or 'No' assessment, indicating whether the question is high quality and suitable for evaluating the students on simultaneous application of the skills <skill_1> and <skill_2>. Provide thorough reasoning for your assessment based on the conditions mentioned above and any other relevant analytical points concerning mathematical reasoning and problem-solving. Your response should be structured as follows:


# REASONING

<Your detailed analysis justifying your decision>


# FINAL ANSWER

<'Yes' or 'No'. No other text should be present in this section>


Ensure to review the combination of skills intended for assessment, and check the logical flow and mathematical correctness from the question's setup to the solution's conclusion. Look out for any problems in the question which are pointed out in the attempted solution. Account for all the potential pitfalls such as logical inconsistencies, unnecessary complexity, or insufficient detail that may obstruct the clarity or solvability of the question. Given below are the two skills and some example question-answer pairs for the two skills. This process is difficult so I am attaching a few sample conversations where (agent) is an AI agent who is trying to verify the questions and (query) is teacher feedback. This should give you some idea of potential difficulties in this task. This is followed by the question which you need to check (preceded by '# QUESTION TO BE CHECKED') and its attempted solution (preceded by '# SOLUTION ATTEMPT'). 


SKILL 1: <skill_1>


Question 1: <skill_1_question_1> 

Answer 1: <skill_1_solution_1>


Question 2: <skill_1_question_2>

Answer 2: <skill_1_solution_2>


Question 3: <skill_1_question_3>

Answer 3: <skill_1_solution_3>


SKILL 2: <skill_2>


Question 1: <skill_2_question_1> 

Answer 1: <skill_2_solution_1>


Question 2: <skill_2_question_2> 

Answer 2: <skill_2_solution_2>


Question 3: <skill_2_question_3>

Answer 3: <skill_2_solution_3>


# CONVERSATION 1

<validation_exemplar_1>


# CONVERSATION 2

<validation_exemplar_2>


......


# CONVERSATION 6

<validation_exemplar_6>


# QUESTION TO BE CHECKED

<question>


# SOLUTION ATTEMPT

<solution>


Thank you very much!









  §.§.§ Final Solution


For the final solution, we make use in-context exemplars from MATH <cit.> as opposed to the attempted solution step.

[Prompt for the final solution]
I have two basic mathematical skills described below, each followed by some sample question - solution pairs such that solving the questions involves the use of the particular skill in order to be solved.


SKILL 1: <skill_1>


Question 1: <skill_1_question_1>
 
Answer 1: <skill_1_solution_1>


Question 2: <skill_1_question_2>

Answer 2: <skill_1_solution_2>


Question 3: <skill_1_question_3>

Answer 3: <skill_1_solution_3>


SKILL 2: <skill_2>


Question 1: <skill_2_question_1>
 
Answer 1: <skill_2_solution_1>


Question 2: <skill_2_question_2>
 
Answer 2: <skill_2_solution_2>


Question 3: <skill_2_question_3>

Answer 3: <skill_2_solution_3>


Go through the descriptions of the skills carefully. Now, here is a new question such that the question requires an expertise all both the skills in order to be solved. That is, the question composes these two skills


QUESTION: <question>


Based on your understanding of the skills, can you please solve the question accurately? Think step by step and explain the solution. Finally, end your response by stating the final numerical answer obtained using the solution. Note that your understanding of the skills should not be restricted to the sample questions provided in their description. They are just example questions. Use your own prior knowledge as well. The explanation of your solution and the final numerical answer should each be on a new line, and should be preceded by the prefixes '# SOLUTION
' and '# ANSWER
' respectively. Thus, your response should be in the format:


'# SOLUTION

<solution>


# ANSWER

<final_answer; no other text should be present in this section>'.


Thank you very much!









  §.§.§ Evaluation



[Prompt given to the GPT-4 for evaluating the model's solution]
You are a professional math teacher and are tasked with evaluating your students on a math exam. You are will be given a question, the correct solution to the question and the student's solution. You need to tell me whether the student solved the question correctly, thus matching the answer obtained by the correct solution. Think step-by-step and give a detailed explanation of your answer. At the end, give a 'Yes' or 'No' answer to whether the student's solution is correct. Your output should be in the following format:


# STEP BY STEP EXPLANATION

<detailed explanation of your thought process>


# CORRECTNESS

<'Yes' if the student's solution is correct. 'No' otherwise. This section should not contain any other text>


Here are the question, correct solution to the question and the student's solution:

QUESTION: <question>


CORRECT SOLUTION: <correct_solution>


STUDENT'S SOLUTION: <student's_solution>





    







 Paper Title: Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models.



[
Measuring Progress in Dictionary Learning for 
 Language Model Interpretability with Board Game Models














equal*


Adam Karvonenequal,ind
Benjamin Wrightequal,mit
Can Ragerind
Rico Angellumass
Jannik Brinkmannmann
Logan Smithind
Claudio Mayrink Verdunhar
David Baunor
Samuel Marksnor


Adam Karvonenadam.karvonen@gmail.com

indIndependent
mitMIT
umassUniversity of Massachusetts, Amherst
mannUniversity of Mannheim
harHarvard University
norNortheastern University







Machine Learning, ICML

0.3in
]










 

§ ABSTRACT 
What latent features are encoded in language model (LM) representations? Recent work on training sparse autoencoders (SAEs) to disentangle interpretable features in LM representations has shown significant promise. However, evaluating the quality of these SAEs is difficult because we lack a ground-truth collection of interpretable features that we expect good SAEs to recover. We thus propose to measure progress in interpretable dictionary learning by working in the setting of LMs trained on chess and Othello transcripts. These settings carry natural collections of interpretable features—for example, “there is a knight on F3”—which we leverage into supervised metrics for SAE quality. To guide progress in interpretable dictionary learning, we introduce a new SAE training technique, p-annealing, which improves performance on prior unsupervised metrics as well as our new metrics.




§ INTRODUCTION



Mechanistic interpretability aims to reverse engineer neural networks into human-understandable components. What, however, should these components be?
Recent work has applied Sparse Autoencoders (SAEs) <cit.>, a scalable unsupervised learning method inspired by sparse dictionary learning to find a disentangled representation of language model (LM) internals. However, measuring progress in training SAEs is challenging because we do not know what a gold-standard dictionary would look like, as it is difficult to anticipate which ground-truth features underlie model cognition. Prior work has either attempted to measure SAE quality in toy synthetic settings <cit.> or relied on various proxies such as sparsity, fidelity of the reconstruction, and LM-assisted autointerpretability <cit.>.

In this work, we explore a setting that lies between toy synthetic data (where all ground-truth features are known; cf. <cit.>) and natural language: LMs trained on board game transcripts. This setting allows us to formally specify natural categories of interpretable features, e.g., “there is a knight on ” or “the bishop on  is pinned.” We leverage this to introduce two novel metrics for how much of a model's knowledge an SAEs has captured:

    
  * Board reconstruction. Can we reconstruct the state of the game board by interpreting each feature as a classifier for some board configuration?
    
  * Coverage. Out of a catalog of researcher-specified candidate features, how many of these candidate features actually appear in the SAE?


These metrics carry the limitation that they are sensitive to researcher preconceptions. Nevertheless, we show that they provide a useful new signal of SAE quality.

Additionally, we introduce p-annealing, a novel technique for training SAEs. When training an SAE with p-annealing, we use an L_p-norm-based sparsity penalty with p ranging from p=1 at the beginning of training (corresponding to a convex minimization problem) to some p < 1 (a non-convex objective) by the end of training. We demonstrate that p-annealing improves over prior methods, giving performance on par with the more compute-intensive Gated SAEs from <cit.>, as measured both by prior metrics and our new metrics.

Overall, our main contributions are as followsCode, models, and data are available at <https://github.com/adamkarvonen/SAE_BoardGameEval>:

    
  * We train and open-source over 500 SAEs trained on chess and Othello models each.
    
  * We introduce two new metrics for measuring the quality of SAEs.
    
  * We introduce p-annealing, a novel technique for training SAEs that improves on prior techniques.




§ BACKGROUND





 §.§ Language models for Othello and chess


In this work, we make use of LMs trained to autoregressively predict transcripts of chess and Othello games. We emphasize that these transcripts only give lists of moves in a standard notation and do not directly expose the board state. Based on behavioral evidence (the high accuracy of the LMs for predicting legal moves) and prior studies of LM representations <cit.> we infer that the LMs internally model the board state, making them a good testbed setting for studying LM representations.





  
Othello. 
Othello is a two-player strategy board game played on an 8x8 grid, with players using black and white discs. 
Players take turns placing discs on the board, capturing their opponent's discs by bracketing them between their own, causing the captured discs to turn their color. 
The goal is to have more discs turned to display your color at the end of the game. The game ends if every square on the board is covered or either player cannot make a move. 

In our experiments, we use an 8-layer GPT model with 8 attention heads and a n = 512 dimensional hidden space, as provided by <cit.>.
This model had no prior knowledge of the game or its rules and was trained from scratch on 20 million game transcripts, where each token in the corpus represents a tile on which players place their discs. 
The game transcripts were synthetically generated by uniformly sampling from the Othello game tree. 
Thus, the data distribution captures valid move sequences rather than strategic depth.
For this model, <cit.> demonstrated the emergence of a world model—an internal representation of the correct board state allowing it to predict the next move—that can be extracted from the model activations using a non-linear probe.
<cit.> extended this finding, showing that a similar internal representation could be extracted using linear probes, supporting the linear representation hypothesis <cit.>.



  
Chess. 
Othello makes a natural testbed for studying emergent internal representations since the game tree is far too large to memorize. 
However, the rules and state are not particularly complex. 
Therefore, we also consider a language model trained on chess game transcripts with identical architecture, provided by <cit.>.
The model again had no prior knowledge of chess and was trained from scratch on 16 million human games from the Lichess chess game database <cit.>. 
The input to the model is a string in the Portable Game Notation (PGN) format (e.g., “”).
The model predicts a legal move in 99.8 % of cases and, similar to Othello, it has an internal representation of the board state that can be extracted from the internal activations using a linear probe <cit.>.










 §.§ Sparse autoencoders

Given a dataset 𝒟 of vectors 𝐱∈ℝ^d, a sparse autoencoder (SAE) is trained to produce an approximation

    𝐱≈∑_i^d_SAE f_i(𝐱)𝐝_i + 𝐛

as a sparse linear combination of features. Here, the feature vectors 𝐝_i∈ℝ^d are unit vectors, the feature activations f_i(𝐱)≥ 0 are a sparse set of coefficients, and 𝐛∈ℝ^d is a bias term. Concretely, an SAE is a neural network with an encoder-decoder architecture, where the encoder maps 𝐱 to the vector 𝐟 = [     f_1(𝐱)          … f_d_SAE(𝐱) ] of feature activations, and the decoder maps 𝐟 to an approximate reconstruction of 𝐱. 

In this paper, we train SAEs on datasets consisting of activations extracted from the residual stream after the sixth layer for both the chess and Othello models. At these layers, linear probes trained with logistic regression were accurate for classifying a variety of properties of the game board <cit.>. For training SAEs, we employ a variety of SAE architectures and training algorithms, as detailed in Section <ref>.





§ MEASURING AUTOENCODER QUALITY FOR CHESS AND OTHELLO MODELS



Many of the features learned by our SAEs reflect uninteresting, surface-level properties of the input, such as the presence of certain tokens. However, upon inspection, we additionally find many SAE features which seem to reflect a latent model of the board state, e.g., features that reflect the presence of certain pieces on particular squares, squares that are legal to play on, and strategy-relevant properties like the presence of a pin in chess (Figure <ref>). 

Fortunately, in the setting of board games, we can formally specify certain classes of these interesting features, allowing us to more rigorously detect them and use them to understand our SAEs. In Section <ref>, we specify certain classes of interesting game board properties. Then, in Section <ref>, we leverage these classes into two metrics of SAE quality.




 §.§ Board state properties in chess and Othello models


We formalize a board state property (BSP) to be a function g : {game board}→{0,1}. In this work, we will consider the following interpretable classes of BSPs:


    
  * 𝒢_board state contains BSPs which classify the presence of a piece at a specific board square, where the board consists of 8× 8 squares in both games. For chess, we consider the full board for the twelve distinct piece types (white king, white queen, ..., black king), giving a total of 8× 8× 12 BSPs. For Othello, we consider the full board for the two distinct piece types (black and white), yielding 8×8× 2 BSPs.
    
  * 𝒢_strategy consists of BSPs relevant for predicting legal moves and playing strategically in chess, such as a pin detector. They were selected by the authors based on domain knowledge and prior interpretability work on the chess model AlphaZero <cit.>. We provide a full list of strategy BSPs in Table <ref> in the Appendix. Because our Othello model was trained to play random legal moves, we do not consider strategy BSPs for Othello.












 §.§ Measuring SAE quality with board state properties


In this section, we introduce two metrics of SAE quality: coverage and board reconstruction.



  
Coverage. Given a collection 𝒢 of BSPs, our coverage metric quantifies the extent to which an SAE has identified features that coincide with the BSPs in 𝒢. In more detail, suppose that f_i is an SAE feature and t∈[0,1] is a threshold, we define the function

    ϕ_f_i, t(𝐱) = 𝕀[f_i(𝐱) > t· f_i^max]

where f_i^max is (an empirical estimate of) max_𝐱∼𝒟f_i(𝐱), the maximum value that f_i takes over the dataset 𝒟 of activations extracted from our model, and 𝕀 is the indicator function. We interpret ϕ_f_i,t as a binary classifier; intuitively, it corresponds to binarizing the activations of f_i into “on” vs. “off” at some fraction t of the maximum value of f_i on 𝒟. Given some BSP g∈𝒢, let F_1(ϕ_f_i,t; g)∈ [0,1] denote the F1-score for ϕ_f_i, t classifying g. Then we define the coverage of an SAE with features {f_i} relative to a set of BSPs 𝒢 to be

    Cov({f_i}, 𝒢) := 1/|𝒢|∑_g∈𝒢max_tmax_f_iF_1(ϕ_f_i,t; g).


In other words, we take, for each g∈𝒢, the F_1-score of the feature that best serves as a classifier for g, and then take the mean of these maximal F_1-scores. An SAE receives a coverage score of 1 if, for each BSP g∈𝒢, it has some feature that is a perfect classifier for g. Since Cov depends on the choice of threshold t, we sweep over t∈{0, 0.1, 0.2, …, 0.9} and take the best coverage score; typically this best t is in {0, 0.1, 0.2}.



  
Board reconstruction. Again, let 𝒢 be a set of BSPs. Intuitively, the idea of our board reconstruction metric is that, for a sufficiently good SAE, there should be a simple, human-interpretable way to recover the state of the board from the profile of feature activations {f_i(𝐱)} on an activation 𝐱∈ℝ^d. Here, the activation 𝐱 was extracted after the post-MLP residual connection in layer 6.

We will base our board reconstruction metric around the following human-interpretable way of recovering a board state from a feature activation profile; we emphasize that different ways of recovering boards from feature activations may lead to qualitatively different results. This recovery rule is based on the assumption that interpretable SAE features tend to be high precision for some subset of BSPs, in line with <cit.>. For example, features that classify common configurations of pieces are high precision (but not necessarily high recall) for multiple BSPs.
 Thus, using a batch of “train” data 𝒟_train we identify, for each SAE feature f_i, all of the BSPs g∈𝒢 for which ϕ_f_i,t is a high precision (of at least 0.95) classifier. Then for each g∈𝒢 our prediction rule is



    𝒫_g({f_i(𝐱)}) = 
        1,    if ϕ_f_i,t(𝐱) = 1  for any  f_i  which
         is high precision for  g  on 𝒟_train
    
        0,    otherwise.







Let F_1(𝒫({f_i(𝐱)}); 𝐛) denote the F_1-score for a given board state 𝐛, where 𝒫({f_i(𝐱)}) = {𝒫_g({f_i(𝐱)})}_g ∈𝒢 represents the full predicted board (containing predictions for all 64 squares) obtained from the SAE activations. Then, the average F_1-score over all board states in the test dataset 𝒟_test can be calculated as:

    Rec({x_i}, 𝒟_test) = 1/|𝒟_test|∑_𝐱∈𝒟_testmax_t F_1(𝒫({f_i(𝐱)}); 𝐛),





§ TRAINING METHODOLOGIES FOR SAES


In our experiments, we investigate four methods for training SAEs, as explained in this section. These are given by two autoencoder architectures and two training methodologies—one with p-annealing and one without p-annealing—for each architecture. Our SAEs are available at <https://huggingface.co/adamkarvonen/chess_saes/tree/main> (chess) and <https://huggingface.co/adamkarvonen/othello_saes/tree/main> (Othello).






 §.§ Standard SAEs


Let n be the dimension of the model's residual stream activations that are input to the autoencoder, m the autoencoder hidden dimension, and s the dataset size. Our baseline “standard” SAE architecture, as introduced in <cit.> is defined by encoder weights W_e ∈ℝ^m × n, decoder weights W_d ∈ℝ^n × m with columns constrained to have a L_2-norm of 1, and biases b_e ∈ℝ^m, b_d ∈ℝ^n. Given an input 𝐱∈ℝ^n, the SAE computes

    𝐟(𝐱)    = ReLU(W_e (𝐱 - 𝐛_d) + 𝐛_e) 
    𝐱̂   = W_d  𝐟(𝐱) + 𝐛_d

where 𝐟(𝐱) is the vector of feature activations, and  𝐱̂ is the reconstruction. 

For a standard SAE, our baseline training method is as implemented in the open-source  repository <cit.>, optimizing the loss

    ℒ_standard = 𝔼_𝐱∼𝒟_train[ 𝐱 - 𝐱̂_2 + λ𝐟(𝐱) _1].


for some hyperparameter λ>0 controlling sparsity.











 §.§ Gated SAEs

The L_1 penalty used in the original training method encourages feature activations to be smaller than they would be for optimal reconstruction <cit.>. 
To address this, <cit.> introduced a modification to the original SAE architecture that separates the selection of dictionary elements to use in a reconstruction and estimating the coefficients of these dictionary elements. This results in the following gated architecture: 






    π_gate(𝐱) := W_gate(𝐱 - 𝐛_d) + 𝐛_gate


    𝐟̃(𝐱) := 𝕀[π_gate(𝐱) > 0] ⊙ReLU(W_mag(𝐱 - 𝐛_d) + 𝐛_mag)


    x̂(𝐟̃(𝐱)) = W_d 𝐟̃(𝐱) + 𝐛_d

where 𝕀[ · > 0] is the Heaviside step function and ⊙ denotes elementwise multiplication. Then, the loss function uses x̂_frozen, a frozen copy of the decoder:





    ℒ_gated := 𝔼_𝐱∼𝒟_train[    𝐱 - x̂(𝐟̃(𝐱))_2^2 
        + λReLU(π_gate(𝐱))_1 
        + 𝐱 - x̂_frozen(ReLU(π_gate(𝐱)))_2^2 ]









 §.§ p-Annealing


Fundamentally, an L_1 penalty has been used to induce sparsity in SAE features because it serves as a convex relaxation of the true sparsity measure, the L_0-norm. The L_1-norm is the convex hull of the L_0-norm, making it a tractable alternative for promoting sparsity <cit.>. However, the proxy loss function is not the same as directly optimizing for sparsity, leading to issues such as feature shrinkage <cit.> and potentially less sparse learned features. Unfortunately, the L_0-norm is non-differentiable and directly minimizing it is an NP-hard problem <cit.>, rendering it impractical for training.



In this work, we propose the use of nonconvex L_p^p-minimization, with p<1, as an alternative to the standard L_1 minimization in sparse autoencoders (SAEs). This approach has been successfully employed in compressive sensing and sparse recovery to achieve even sparser representations <cit.>. To perform this optimization, we introduce a method called p-annealing for training SAEs, based on the compressive sensing technique called p-continuation <cit.>. The key idea is to start with convex L_1-minimization through setting p=1 and progressively decrease the value of p during training, resulting in closer approximations of the true sparsity measure, L_0, as p approaches 0. We define the sparsity penalty for each batch x as a function of the current training step s:




    ℒ_sparse(𝐱, s) = λ_s 𝐟(𝐱) _p_s^p_s = λ_s ∑_if_i(𝐱)^p_s


In other words, the sparsity penalty will be a scaled L_p^p norm of the SAE feature activations, with p decreasing over time. At p=1, the L_p^p norm is equal to the L_1 norm, and as p → 0, the L_p^p norm limits to the L_0-norm, as lim_p→ 0∑_i f_i(𝐱)^p = ∑_i f_i(𝐱)^0.

The purpose of annealing p from 1 → 0 instead of starting from a fixed, low value for p is that the lower the p, the more concave (non-convex) the L_p^p norm is, increasing the likelihood of the training process getting stuck in local optima, which we have observed in initial experiments. Therefore, we aim to first arrive at a region of an optimum using the easier-to-train L_1 penalty and then gradually shift the loss function. This manifests as keeping p=1 for a certain number of steps and then starting decreasing p linearly down to p_end > 0 at the end of training. We set p_end=0.2.







  
Coefficient Annealing.
Changing the value of p changes the scale of the L_p^p norm. Without also adapting the coefficient λ, the strength of the sparsity penalty would vary too wildly across training. Empirically, we found that keeping a constant λ would lead to far too weak of a sparsity penalty for the larger p's at the start of training, making the process worse than simply training with a constant p from the beginning.

Consequently, we aim to adapt the coefficient λ such that the strength of the sparsity penalty is not changed significantly due to p updates. Formally, the update step is:

    λ_s+1λ_s∑_j=s-q+1^s∑_if_i(𝐱_𝐣)^p_s/∑_j=s-q+1^s∑_if_i(𝐱_𝐣)^p_s+1.


We keep a queue of the most recent q batches of feature activations mid-training and use them to calibrate the λ_s updates. Therefore, the strength of the sparsity penalty is kept locally constant.



  
Combining p-annealing with other SAEs.

Since the p-annealing method only modifies the L_1 terms in the loss function without affecting the SAE architecture, it is simple to combine p-annealing with other SAE modifications. This allows us to create the Gated-Annealed SAE method by combining the Gated SAE architecture and p-annealing. Concretely, we modify ℒ_gated (Equation <ref>) by replacing the sparsity term λReLU(π_gate(𝐱))_1 in  with λ_s ReLU(π_gate(𝐱)) _p_s^p_s. Our experiments showed that the optimum values for coefficients λ and λ_s differ.





§ RESULTS



In this section, we explore the performance of SAEs applied to language models trained on Othello and chess. Consistent with <cit.>, we find that interpretable SAE features typically track properties relative to the player whose turn it is (e.g. “my king is pinned” rather than “the white king is pinned”). To side-step subtleties arising from this, we only extract our activations from the token immediately preceding white's move.
Specifically, we consider SAEs trained on the residual stream activations after the sixth layer using the four methods from Section <ref> (see Table <ref> for additional hyperparameters). 
In addition to our metrics introduced above, we also make use of unsupervised metrics previously appearing in the literature <cit.>:

    
  * L_0 measures the average number of active SAE active features (i.e., positive activation) on a given input.
    
  * Loss recovered measures the change in model performance when replacing activations with the corresponding SAE reconstruction during a forward pass.
    This metric is quantified as (H_* - H_0)/(H_orig - H_0), where H_orig is the cross-entropy loss of the board game model for next-token prediction, H_* is the cross-entropy loss after substituting the model activation 𝐱 with its SAE reconstruction during the forward pass, and H_0 is the cross-entropy loss when zero-ablating 𝐱.

Our key takeaways are as follows.





  
SAE features can accurately reconstruct game boards. 
In general, we find that SAE features are effective at capturing board state information in both Othello and chess (see Table <ref>, Figure <ref>d and <ref>d). 
In contrast, SAEs trained on a model with random weights perform very poorly according to our metrics, showing that SAE performance is driven by identifying structure in the models' learned representation of game boards.
Nonetheless, SAEs do not match the performance of linear probes in terms of reconstructing the board state.
This performance gap suggests that SAEs do not capture all of the information encoded in the model's internal representations.



  
Standard SAEs trained with p-annealing perform on par with Gated SAEs.
We find that standard SAEs trained using p-annealing consistently perform better than those trained with a constant L_1 penalty (Equation <ref>), as measured by existing proxy metrics and in terms of improvement in coverage (see Figure <ref>a and <ref>a).
In fact, standard SAEs trained using p-annealing show a coverage score that is comparable to Gated SAEs trained without p-annealing. However, we find cases where our coverage metric disagrees with existing metrics. In Figure <ref>, for example, Gated SAEs perform achieve a higher loss recovered score than Standard SAEs trained using p-annealing. We emphasize that the training and inference of Gated SAEs is more computationally expensive, requiring 50% more compute per forward pass compared to Standard SAEs <cit.>. 













  
Coverage and board reconstruction reveal differences in SAE quality not captured by unsupervised metrics.
Our metrics reveal improvements in SAE performance that traditional proxy metrics fail to capture.
For example, we trained SAEs with hidden dimensions 4096 and 8192 (expansion factors of 8 and 16, respectively).  
We expect the SAEs with 8192 hidden dimensions to perform better since they have greater capacity.
However, we observe that they perform equally well according to prior unsupervised metrics (see Figures <ref> a, c and <ref> a, c). 
In contrast, our metrics reveal that SAEs with larger hidden dimensions are better. For the Standard architecture, this is reflected by the parallel lines (of purple diamonds) in Figures <ref> b, d and <ref> b, d.
Thus, our metrics are able to capture improvements from larger expansion factors.
In addition, we find that the performance of p-annealing closely resembles that of Gated SAEs when evaluated using standard proxy metrics; it demonstrates clear improvements under our proposed metrics.



  
Coverage and board reconstruction are consistent with existing metrics.
Figures <ref>, <ref>, and <ref> demonstrate that both coverage and board reconstruction metrics are optimal in the elbow region of the Pareto frontier. 
This region, where SAEs reconstruct internal activations efficiently with minimal features, also yielded the most coherent interpretations during our manual inspections. This provides precise, empirical validation to the common wisdom that SAEs in this region of the Pareto frontier are the best.










































§ LIMITATIONS


The proposed metrics for board reconstruction and coverage provide a more objective evaluation of SAE quality than previous subjective methods. 
Nevertheless, these metrics exhibit several limitations. 
Primarily, their applicability is confined to the chess and Othello domains, raising concerns about their generalizability to other domains or different models. 
Additionally, the set of BSPs that underpin these metrics is determined by researchers based on their domain knowledge. 
This approach may not encompass all pertinent features or strategic concepts, thus potentially overlooking essential aspects of model evaluation.
Developing comparable objective metrics for other domains, such as natural language processing, remains a significant challenge. 
Moreover, our current focus is on evaluating the quality of SAEs in terms of their ability to capture internal representations of the model. 
However, this does not directly address how these learned features could be utilized for downstream interpretability tasks.















§ RELATED WORK




  
Sparse dictionary learning. Since the nineties, dictionary learning <cit.>, sparse regression <cit.>, and later, sparse autoencoders <cit.> have been extensively studied in the machine learning and signal processing literature. The seminal work of <cit.> introduced the concept of sparse coding in neuroscience (see also <cit.>, building upon the earlier concept of sparse representations <cit.> and matching pursuit <cit.>. Subsequently, a series of works established the theoretical and algorithmic foundations of sparse dictionary learning <cit.>. Notably, <cit.> introduced LISTA, an unrolled version of ISTA <cit.> that learns the dictionary instead of having it fixed.

In parallel, autoencoders were introduced in machine learning to automatically learn data features and perform dimensionality reduction <cit.>. Inspired by sparse dictionary learning, sparse autoencoders <cit.> were proposed as an unsupervised learning model to build deep sparse hierarchical models of data, assuming a certain degree of sparsity in the hidden layer activations. Later, <cit.> generalized sparse autoencoders (SAEs) to convolutional SAEs. Although the theory for SAEs is less developed than that of dictionary learning with a fixed dictionary, some progress has been made in quantifying whether autoencoders can, indeed, do sparse coding, e.g., <cit.>.



  
Feature disentanglement using sparse autoencoders. 

The individual computational units of neural networks are often polysemantic, i.e., they respond to multiple seemingly unrelated inputs <cit.>.
<cit.> investigated this phenomenon and suggested that neural networks represent features in linear superposition, which allows them to represent more features than they have dimensions. 
Thus, in an internal representation of dimension n, a model can encode m ≫ n concepts as linear directions <cit.>, such that only a sparse subset of concepts are active across all inputs – a concept deeply related to the coherence of vectors <cit.> and to frame theory in general <cit.>. 

To identify these concepts, <cit.> used SAEs to perform dictionary learning on a one-layer transformer, identifying a large (overcomplete) basis of features. 
<cit.> applied SAEs to language models and demonstrated that dictionary features can be used to localize and edit model behavior. <cit.> proposed a scalable method to discover sparse feature circuits, as opposed to circuits consisting of polysemantic model components, and demonstrated that a human could change the generalization of a classifier by editing its feature circuit. Recently, <cit.> explored autoencoders for attention layer outputs. These works have benefited from a variety of open-source libraries for training SAEs for LLM interpretability <cit.>.












  
Improving and evaluating sparse autoencoders. 

Despite the success of SAEs at extracting human-interpretable features, they fail to perfectly reconstruct the activations <cit.>.
One challenge in the training of SAEs with an L_1 penalty is shrinkage (or 'feature suppression'); in addition to encouraging sparsity, an L_1 penalty encourages feature activations to be smaller than they would be otherwise. 
<cit.> approached this problem by fine-tuning the sparse autoencoder without a sparsity penalty. 
<cit.> and <cit.> explored alternative sparsity penalties to reduce feature suppression during training.
<cit.> introduced Gated SAEs, an architectural variation for the encoder which both addresses shrinkage and improves on the Pareto frontier of L_0 vs reconstruction error. Recently, <cit.> systematically evaluated the
scaling laws with respect to sparsity, autoencoder size, and language model size.


The goal of dictionary learning in machine learning is to produce human-interpretable features and capture the underlying model's computations <cit.>. 
However, quantitatively measuring interpretability is difficult and often involves manual inspection. 
Therefore, most existing work assesses the quality of SAEs along different proxy metrics: 
(1) The cross-entropy loss recovered, which reflects the degree to which the original loss of the language model can be recovered when replacing activations with the autoencoder predictions.
(2) The L_0-norm of feature activations 𝔼_z ∼𝒟 h(z) _0, measuring the number of activate features given an input <cit.>. <cit.> proposed to compare SAEs against supervised feature dictionaries in a natural language setting.
However, this requires a significant understanding of the model's internal computations and is thus not scalable.




§ CONCLUSION


Most SAE research has relied on proxy metrics such as loss recovered and L_0, or subjective manual evaluation of interpretability by examining top activations. However, proxy measures only serve as an estimate of interpretability, monosemantic nature, and comprehensiveness of the learned features, while manual evaluations depend on the researcher's domain knowledge and tend to be inconsistent.

Our work provides a new, more objective paradigm for evaluating the quality of an SAE methodology; coverage serves as a quantifiable measure of monosemanticity and quality of feature extraction, while board reconstruction serves as a quantifiable measure of the extent to which an SAE is exhaustively representing the information contained within the language model. Therefore, the optimal SAE methodology can be judged by whether it yields both high coverage and high board reconstruction.

Finally, we propose the p-annealing method, a modification to the SAE training paradigm that can be combined with other SAE methodologies and results in an improvement in both coverage and board reconstruction.



§ AUTHOR CONTRIBUTIONS

A.K. built and maintained our infrastructure for working with board-game models.
A.K., S.M., C.R., J.B., and L.S. designed the proposed metrics. 
B.W. performed initial experiments demonstrating the benefits of training SAEs with p < 1. B.W., C.M.V., and S.M. then proposed p-annealing, with B.W. leading the implementation and developing coefficient annealing.
The basic framework for our dictionary learning work was built and maintained by S.M. and C.R. The training algorithms studied were implemented by S.M., C.R., B.W., R.A., and J.B. 
R.A. trained the SAEs used in our experiments. 
A.K., C.R., and J.B. selected and implemented the BSPs.
A.K. and J.B. trained the linear probes. 
Many of the authors (including L.S., J.B., R.A.) did experiments applying traditional dictionary learning methods and exploring both toy problems and natural language settings, which helped build valuable intuition.
The manuscript was primarily drafted by A.K., B.W., C.R., R.A., J.B., C.M.V., and S.M., with extensive feedback and editing from all authors. 
D.B. suggested the original project idea.



§ ACKNOWLEDGMENTS

C.R. is supported by Manifund Regrants and AISST. 
L.R. is supported by the Long Term Future Fund.
S.M. is supported by an Open Philanthropy alignment grant.

The work reported here was performed in part by the University of Massachusetts Amherst Center for Data Science and the Center for Intelligent Information Retrieval, and in part using high performance computing equipment obtained under a grant from the Collaborative R&D Fund managed by the Massachusetts Technology Collaborative.


icml2024
















§ SPARSE AUTOENCODER TRAINING PARAMETERS


We used a single NVIDIA A100 GPU for training SAEs and experiments. It takes much less than 24 hours to train a single SAE on 300 million tokens. Given a trained SAE, our evaluation requires less than 5 minutes of computing time.





§ LIST OF BOARD STATE PROPERTIES


Table <ref> summarizes the high-level board state properties considered in 𝒢_strategy. The selection of concepts was inspired by <cit.>. The column indicated by # denotes the number of individual BSPs per concept. A single BSP per concept indicates we match this condition globally for any corresponding piece.





§ PERFORMANCE OF LINEAR PROBES AND SAES ON BOARD STATE PROPERTIES



In Figure <ref>, we present a mean coverage score over strategy board state properties 𝒢_BSP.
Properties within 𝒢_BSP vary significantly in complexity.
For example, queen detection can be inferred directly from the move history, while fork detection requires an accurate representation of the board state. 

Table <ref> shows that linear probe F1-score is below 0.95 for 6 out of 15 properties in 𝒢_BSP. This suggests that chess-GPT <cit.> does not represent these properties linearly. Additional experiments are required to determine whether the representation is present at all.

For the board state case, reconstruction is significantly higher than coverage. This is because there are many SAE features that are high precision classifiers for a configuration of squares, such as "white pawn on , white knight of ". In cases where coverage is higher than reconstruction (such as for ), it is because there are not many features that are over 95% precision for “there is a check move available” from which we can recover if there is an available check move. Coverage is significantly higher because there is at least one feature that has an F_1-score of 0.54 for , which may not have a precision greater than 95%.

Y>X


Y>X



 










§ MODEL INTERNAL BOARD STATE REPRESENTATION





 §.§ Othello Models

Previous research of Othello-playing language models found that the model learned a nonlinear model of the board state <cit.>. Further investigation found a closely related linear representation of the board when probing for "my color" vs. "opponent's color" rather than white vs. black <cit.>. Based on these findings, when measuring the state of the board in Othello, we represent squares as (, ) rather than (, ).




 §.§ Chess Models

Similar to Othello models, prior studies of chess-playing language models found the same property, where linear probes were only successful on the objective of the (, ) representation and were unsuccessful on the (, ) representation <cit.>. They measured board state at the location of every period in the Portable Game Notation (PGN) string, which indicates that it is white's turn to move and maintain the (, ) objective. Some characters in the PGN string contain little board state information as measured by linear probes, and there is not a clear ground truth board state part way through a move (e.g., the “” in “”). We follow these findings and measure the board state at every period in the PGN string.

When measuring chess piece locations, we do not measure pieces on their initial starting location, as this correlates with position in the PGN string. An SAE trained on residual stream activations after the first layer of the chess model (which contains very little board state information as measured by linear probes) obtains a board reconstruction F_1-score of 0.01 in this setting. If we also measure pieces on their initial starting location, the layer 1 SAE's F_1-score increases to 0.52, as the board can be mostly reconstructed in early game positions purely from the token's location in the PGN string. Masking the initial board state and blank squares decreases the F_1-score of the linear probe from 0.99 to 0.98.



 Paper Title: Vision-Language Model Based Handwriting Verification.










Vision-Language Model Based Handwriting Verification
    
    Mihir Chauhan††   Abhishek Satbhai§   Mohammad Abuzar Hashemi¶   Mir Basheer Ali¶¶   Bina Ramamurthy‡   Mingchen Gao||   Siwei Lyu†   Sargur Srihari‡‡
 
    
    Department of Computer Science and Engineering 
 
    The State University of New York, Buffalo, NY, USA 
 
    {mihirhem††, bina‡, mgao8||, siweilyu†, srihari‡‡}@buffalo.edu 

    {ma.hashemi.786¶, abhishek07satbhai§, alimirbasheer¶¶}@gmail.com

    
=================================================================================================================================================================================================================================================================================================================================================================================================================================

empty



§ ABSTRACT 
Handwriting Verification is a critical in document forensics. Deep learning based approaches often face skepticism from forensic document examiners due to their lack of explainability and reliance on extensive training data and handcrafted features. This paper explores using Vision Language Models (VLMs), such as OpenAI’s GPT-4o and Google's PaliGemma, to address these challenges. By leveraging their Visual Question Answering capabilities and 0-shot Chain-of-Thought (CoT) reasoning, our goal is to provide clear, human-understandable explanations for model decisions. Our experiments on the CEDAR handwriting dataset demonstrate that VLMs offer enhanced interpretability, reduce the need for large training datasets, and adapt better to diverse handwriting styles. However, results show that the CNN-based ResNet-18 architecture outperforms the 0-shot CoT prompt engineering approach with GPT-4o (Accuracy: 70%) and supervised fine-tuned PaliGemma (Accuracy: 71%), achieving an accuracy of 84% on the CEDAR AND dataset. These findings highlight the potential of VLMs in generating human-interpretable decisions while underscoring the need for further advancements to match the performance of specialized deep learning models. Our code is publicly available at: <https://github.com/Abhishek0057/vlm-hv>


Keywords: Vision Language Model, Machine Vision, Handwriting Verification, Forensics








§ INTRODUCTION


Handwriting verification is crucial in pattern recognition and biometrics, focusing on authenticating and identifying individuals by their handwriting. It is vital in forensics, where experts analyze samples to verify documents, identify forgeries, and provide court evidence.

Historical Perspective: Handwriting Verification methods <cit.> <cit.> <cit.> have evolved from handcrafted features like GSC <cit.> to deep CNNs such as ResNet-18 <cit.>, and now Vision Transformers <cit.>, enabling comparison of both inter- and intra- writer variations in handwriting styles. 

Challenges: Despite the advancements, Forensic Document Examiners (FDE) remain skeptical due to the lack of interpretability in the model's decision making processes. Morever, these methods heavily depend on large datasets of labeled handwritten images, making dataset collection (X(x_q,x_k, y) with known x_k, questioned x_q handwritten samples and corresponding writer labels y) is costly and time-intensive. 


Why use VLM? Vision-Language Models (VLMs) integrate image and textual data to encode complex relationships between visual and linguistic information, enabling Forensic Document Examiners (FDEs) to interpret model decisions with clear, natural language explanations that enhance trust and reliability in forensic applications. They can adapt to the forensics domain in zero-shot scenarios without training examples and in few-shot scenarios with minimal examples by leveraging transfer learning capabilities, having been pre-trained on multiple tasks related to handwritten images and natural language understanding. Fine-tuning them with forensic-specific datasets further enhances their performance and applicability in real-world forensic use-cases. 

VLMs applied to Forensics: In <cit.>, the authors explore the capabilities of multimodal Large Language Models (LLMs) like GPT-4 in various digital forensics use-cases, including artifact understanding, evidence searching, anomaly detection, and education. More specific studies, such as <cit.>, demonstrate the effectiveness of GPT-4 specifically for the media forensics task of DeepFake image detection. Our research applies VLMs to handwriting verification. To our knowledge, we are the first to explore VLMs for the forensic task of handwriting comparison.












§ METHODS

Our study aims to use VLM in handwriting verification. We have chosen OpenAI’s GPT-4o VLM for its strong Visual Question Answering (VQA) capabilities. Using their API, we prompt GPT-4o <cit.> with 0-shot Chain-of-Thought (CoT) reasoning. This approach allows us to first generate human-interpretable explanations and then determine whether the given questioned and known handwritten samples were written by the same person or by different writers, as shown in Figure <ref>. Since fine-tuning GPT-4o is not generally available, we compared its performance to a recent open-source VLM, PaliGemma <cit.>, using 0-shot prompt engineering and parameter-efficient supervised fine-tuning (PEFT) on a training dataset with 100 examples.  

Data: The experiments were conducted using 1,000 sample pairs of known and questioned images from CEDAR Letter <cit.> and CEDAR AND <cit.> dataset. CEDAR Letter dataset contains a letter manuscript written by 1568 writers three times. CEDAR AND dataset is a subset of CEDAR Letter data which only contains the lowercase handwritten word “and” extracted from full letter manuscript. The evaluation dataset includes 1,000 sample pairs of known and questioned images written by 368 writers with writer ids greater than 1200 (writer ids below 1200 were used to train the baseline models). Of these pairs, 500 are from the same writer and 500 are from different writers.






§ EXPERIMENTS & RESULTS

CEDAR AND Baselines: As shown in Figure <ref> we use handcrafted features GSC features, CNN based ResNet-18 <cit.> and MaskedCausalVisionTransformer <cit.> (ViT) as our baseline feature extractors. GSC are 512-dimensional features extracted for the binarized “AND” images. All the three baselines were trained on 10% and 100% of known and questioned training pairs writers with writer ids less than 1200 resulting in 13,232 and 129,602 number of train pairs as shown in Table <ref>. The output of these feature extractors is fed into 2 fully-connected (FC) layers with 256 and 128 hidden neurons with ReLU activations. The final layer has 2 output neurons whose softmax activations represent similarity of samples with a one hot vector representation. We use categorical cross entropy loss given one-hot encoded logits compared to the target which is binary. 



Prompt Engineering with GPT-4o and PaliGemma: To effectively utilize the GPT-4o VLM for handwriting verification, we experimented with various prompts to optimize both the generation of human-interpretable explanations and the accuracy of verification decisions. Initially, we crafted prompts that directed the model to compare specific features of the handwriting samples, such as stroke width, slant, and letter spacing. For example, prompts like "Describe the similarities and differences in the stroke patterns between the two samples" and "Identify any matching unique characteristics in the handwriting" were used. However, we observed significant variance in the model’s responses, leading to inconsistent results. To address this, we adopted a Chain-of-Thought (CoT) reasoning approach. This method guides the model through a structured reasoning process, allowing it to generate more consistent and reliable explanations. Since no examples were provided to VLM in the prompt engineering phase, the number of training pairs is observed as 0 in Table <ref>. 



Additionally, we utilized prompt engineered both VLMs to identify and mark coordinates of similarities and dissimilarities within the two images. By prompting the model to highlight specific areas of interest, such as matching or differing stroke patterns, we were able to generate detailed coordinates that pinpointed these features as shown in Figure <ref>. This approach provided clear visual indicators of the reasoning behind each verification decision. By incorporating CoT reasoning, presenting known and questioned samples as different images, and using VLM to identify key coordinates, we achieved a balance where the model could generate detailed, interpretable explanations while accurately determining the authenticity of the handwriting samples.

Supervised Fine-Tuning with PaliGemma: 
We fine-tuned the pre-trained PaliGemma model on 100 curated examples of 0-shot CoT prompt engineered PaliGemma results. The PaliGemmaProcessor within Transformers library was loaded to process the inputs which involved the preparation of prompt templates and batching text inputs with images. The image tokens and pad tokens were set to -100 to be ignored by the model, and these preprocessed inputs were used as labels for the model to learn from. The model, PaliGemmaForConditionalGeneration, was loaded and set to fine-tune only the text decoder, leaving the vision encoder and multimodal projector frozen. We used BitsAndBytes config options with 4bit quantization and LoRA <cit.> to train only 11M parameters out of the 3B parameters of the network. The results of supervised fine-tuned PaliGemma on both CEDAR AND and Letter datasets are shown in Table <ref>.

Results: The results of using VLMs for handwriting verification are presented in Table <ref> on the CEDAR AND and CEDAR Letter datasets. For the CEDAR AND dataset, CNN based ResNet-18 architecture outperforms 0-shot CoT prompt engineering approach with GPT-4o as well as supervised-finetuned PaliGemma with 100% training pairs which suggest that although VLMs has a lot of potential for generating human interpretable decision making for the task of handwriting verification, still lags behind CNNs which aare fine-tuned on specific task of handwriting verification. This gap highlights the need for further improvements in fine-tuning regime of VLMs to enhance their effectiveness and reliability for specialized tasks like handwriting verification. For the CEDAR Letter dataset, the 0-shot CoT prompt engineering with GPT-4o also shows relatively higher performance than the supervised fine-tuning of PaliGemma but we observe high variability in the performance metrics because of low sample size. 



§ CONCLUSION

Our study explores the application of VLMs, specifically GPT-4o and PaliGemma, in the domain of handwriting verification. By leveraging the robust VQA capabilities of these models and employing 0-shot CoT reasoning through prompt engineering, we aimed to generate human-interpretable explanations for model decisions. Our experiments demonstrated that while VLMs offer significant improvements in interpretability and adaptability to diverse handwriting styles, they currently lag behind CNN-based architectures such as ResNet-18 in terms of performance. Specifically, ResNet-18 achieved an accuracy of 84% on the CEDAR AND dataset, outperforming GPT-4o's 70% accuracy and PaliGemma's 71% accuracy. These findings suggest that while VLMs hold great promise for enhancing transparency and trustworthiness in forensic handwriting verification, there is still a need for further advancements in their fine-tuning regimes to improve their effectiveness and reliability for specialized tasks. Moving forward, we aim to work with forensic document examiners to create a fine-tuning dataset of explanation reports using text and visual information to ensure the applicability of our approach in practical forensic settings.






apalike





 Paper Title: Segment anything model 2: an application to 2D and 3D medical images.


Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation
    Yixiao Wang1 Chen Tang1,2 Lingfeng Sun1 Simone Rossi3 Yichen Xie1Chensheng Peng1Thomas Hannagan3Stefano Sabatini3Nicola Poerio4Masayoshi Tomizuka1Wei Zhan1
    August 5, 2024
===============================================================================================================================================================



§ ABSTRACT 
Segment Anything Model (SAM) has gained significant attention because of its ability to segment a variety of objects in images given a prompt. The recently developed SAM 2 has extended this ability to video inputs. This opens an opportunity to apply SAM to 3D images, one of the fundamental tasks in the medical imaging field. In this paper,



we provide an extensive evaluation of SAM 2's ability to segment both 2D and 3D medical images.
We collect 18 medical imaging datasets, including common 3D modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) as well as 2D modalities such as X-ray and ultrasound.
We consider two evaluation pipelines of SAM 2: (1) , where prompts are provided to one or multiple slice(s) selected from the volume, and (2) , where prompts are provided to each slice. The former is only applicable to 3D modalities, while the latter applies to both 2D and 3D modalities.
We learn that SAM 2 exhibits similar performance as SAM under , and has variable performance under depending on the choices of slices to annotate, the direction of the propagation, the predictions utilized during the propagation, etc. 










§ INTRODUCTION

Medical image segmentation is a crucial task for multiple clinical applications such as disease diagnosis and clinical analysis <cit.>. Despite the advancements in medical imaging technologies, segmentation remains a challenging task due to the labor-intensive nature of data annotation and the complexity of medical images <cit.>. 



Segment Anything Model (SAM) addresses these challenges in multiple directions. On one hand, SAM has demonstrated impressive zero-shot segmentation performance with prompt inputs, significantly reducing the need for extensive manual data annotation <cit.>. On the other hand, several works fine-tune SAM to specific tasks and demonstrate improvements over standard segmentation techniques <cit.>, such as nn-UNet <cit.>.

Despite these advancements, SAM's limitation to 2D images restricts its applicability to scenarios that require three-dimensional understanding <cit.>. 

Existing work addresses this challenge by introducing extra components to SAM to enable its 3D segmentation capability. For example, SAM3D <cit.> combines the SAM encoder with a lightweight 3D CNN decoder; 3DSAM-A <cit.> modifies the original prompt encoder and mask decoder to operate in 3D; SAM-Med3D <cit.> introduces an additional 3D convolution before the image encoder and replaces 2D positional encoding layers with 3D one. On the other hand, the recently introduced SAM 2 <cit.> solves this limitation fundamentally by extending the backbone of SAM to 3D. Specifically, SAM 2 proposes a memory bank that retains information from past predictions and allows it to make predictions on slices without prompts based on the information.

This feature motivates us to examine SAM 2's ability to segment 3D medical images since video segmentation can be transferred to 3D segmentation seamlessly, i.e., each slice is treated as a frame. Note that we will use the terms “slice” and “frame” interchangeably throughout the paper.

In this paper, we extend the previous evaluation experiments on SAM <cit.> to SAM 2, aiming to explore the model's effectiveness in a more complex, three-dimensional context. Specifically, we consider two evaluation pipelines: and . In <cit.>, prompts are provided to the object of interest on each slice, whereas in , prompts are provided on one or a few slices selected from the volume. We further consider some unique challenges in , such as the selection of the slice(s) to be annotated, the direction of propagation, i.e., predicting slices without prompts based on ones with prompts, and the selection of prediction during propagation. Our experiments are conducted across 18 datasets, covering 5 modalities (magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), X-ray, and Ultrasound) and various body locations. We observe several trends of SAM 2 under both evaluation pipelines and summarize them in Section <ref>.






§ METHODS




In this section, we discuss the two evaluation pipelines, and , in detail. 
Intersection over Union (IoU) is the evaluation metric throughout the paper. To have a comparable performance between 2D and 3D segmentation, IoU is only computed over non-empty slices.



 §.§ Evaluation of Single-Frame 2D Model Testing on 2D Images

During , SAM 2 exhibits the same behavior as SAM in segmenting the object of interest based on prompts.
For datasets with 2D modalities, we run SAM 2 naturally at the image level. For datasets with 3D modalities, we simulate prompts for each slice of the volume.

Following previous work <cit.>, we design in a non-iterative manner in which all prompts are determined without feedback from any prior predictions. 

Specifically, the following four 2D prompting modes (P-Mode) are used:

    
  * P-Mode 1: One point prompt placed at the center of the largest connected region of the object of interest.
    
  * P-Mode 2: One point prompt placed at each separate connected region of the object (up to three points).
    
  * P-Mode 3: One box prompt placed at the center of the largest connected region of the object of interest.
    
  * P-Mode 4: One box prompt placed at each separate connected region of the object (up to three boxes).

These strategies cover the most efficient and straightforward annotation strategy that provides one point or box prompt to each connected region of the object of interest. 


    





 §.§ Evaluation of Multi-Frame 3D Model Testing on 3D Volumes

SAM 2 differs from SAM mostly in its new ability to segment videos, which can be seamlessly transferred to the 3D image segmentation task.

In this section, we are mostly interested in SAM 2's semi-supervised segmentation ability, where we only provide prompts for one or a few frames in advance and use SAM 2 to predict other frames. 

Initial Frame Selection. In video segmentation, the initial frame to be annotated is typically the first frame of video as it arrives first in the time stream and the object of interest usually does not change shape or size dramatically between consecutive frames <cit.>.<cit.> 
This is also the strategy used by SAM 2 for the semi-supervised video object segmentation task. However, such a strategy is usually infeasible in 3D medical imaging because, in most scenarios, the boundary of the object of interest appears first, and the change between different frames can be significant. 

Instead, when annotating medical volumes, annotators tend to start with the slice in which the object of interest is most salient.
To approximate this behavior, we consider the following three frame modes (F-Mode):

    
  * F-Mode 1: Choose the slice with the largest object of interest.
    
  * F-Mode 2: Choose the slice at the center of the object of interest.
    
  * F-Mode 3: Uniformly select 3 slices from the entire object of interest.













Prompts Simulation. When the slice to be annotated is selected, we only utilize the same point-based strategies (P-Mode 1 and 2) for simplicity because SAM 2's video predictor does not have the function of adding boxes yet. 





Propagation Direction. When the first frame is annotated, SAM 2 supports propagation to predict other non-conditioned frames (frames without prompts). 
This propagation step involves the newly introduced memory structures: memory attention, memory encoder, and memory bank. Memory attention conditions the current slice's features on features from previously predicted slices and prompts. The memory encoder compresses these predictions into smaller embeddings and stores this information in the memory bank.







Note that since new prediction relies on the output from existing predictions, the order of propagation matters. 

Although the order of propagation might have less impact on video predictions in which the consecutive frames are similar and the object of interest appears consistently from beginning to end, it can have a more significant impact on 3D medical volumes in which the change between slices can be large and the object of interest does not occupy the full volume.
To take the order of propagation into account, we consider the following 2 direction modes (D-Mode):

    
  * D-Mode1: Front-to-End Propagation: start from the first slice of the volume and propagate forward through the volume.
    
  * D-Mode2: Start from the first annotated slice and propagate backward to the first slice; then restart from the annotated slice and propagate forward to the end of the volume.

    
    
    
    
    
Predicted Mask Selection. Due to the ambiguity of providing click prompts, it is common to predict various levels of objects based on a single or a few clicks. For example, if we place a point prompt on a tumor inside the brain, the object of interest can either be the tumor or the whole brain.

In the 2D setting of both SAM and SAM 2, the mask predictor can create multiple mask outputs from which humans can select the one closest to their needs. If there is no human involvement in the selection step, we can simply pick the first prediction channel, which, according to previous SAM empirical results, tends to represent the smallest area among the three channels <cit.>. Alternatively, we can select the channel with the highest confidence based on the IoU calculated by SAM's mask decoder.

In 3D mode, to prevent propagating ambiguity across frames, SAM 2 chooses to pick the prediction with the highest confidence and only utilizes this prediction when estimating the object of interest on other frames. However, based on previous experiments with SAM <cit.>, selecting the predictions with the highest confidence can be sub-par when compared to selecting the first channel. This observation motivates us with the following predicted mask selection strategies:

    
  * Selecting the predicted mask with the largest estimated IoU, i.e., confidence. 
    
  * Selecting the first channel.



In summary, under , we cover 3 slice selection modes, 2 prompt-simulation modes, 2 predicted mask strategies, and 2 propagation modes, resulting in a total of 24 different experimental designs. Notably, we do not enumerate all possible choices. For some options, when we identify a consistently better choice for one stage compared to others, we trim some subsequent combinations applying this choice to reduce the total number of experiments.
























§ DATASET


Consistent with the previous experimental study on SAM <cit.>, this study utilizes 18 diverse medical datasets to evaluate the performance of SAM 2. Specifically, all datasets are evaluated during , and twelve datasets with 3D modalities (MRI, CT, and PET) are evaluated during . The “US-Nerve” dataset<cit.>, which was included in the previous SAM evaluation paper<cit.>, has been removed from this project due to licensing regulations. We also updated the MRI-Breast dataset to include more annotated volumes. The pre-processing steps are consistent with the previous study, except that for SAM 2 3D mode, we have to convert the input images to JPG format instead of PNG. We also provided visual representations of the annotations masks for each dataset in Fig. <ref>.



 §.§ 2D Datasets

We included 6 2D datasets, 2 X-rays, and 4 ultrasounds, covering 7 different anatomical objects. Specifically, the X-ray datasets cover chest and hip joint segmentation, and the ultrasound datasets encompass a broader range of regions, including breast, kidney, muscle, and ovarian tumor segmentation. Detailed information on these datasets can be found in Table <ref>. To keep the format consistent with 3D datasets, the images of 2D datasets are also converted to JPG using the same pipeline. 





 §.§ 3D datasets

We included 12 3D datasets, 5 MRI, 6 CT, and 1 PET-CT, covering 20 different anatomical objects. The structures of the datasets are modified to fit the input requirements of SAM 2 3D, where each individual volume has its own folder, and each slice is numbered according to its position within the volume. Table <ref> demonstrates more detailed information.












§ EXPERIMENTAL RESULTS



 §.§ Results of SAM 2 on Single-frame 2D Segmentation testing compared with previous SAM

In this section, we present the performance of SAM 2 under scenario across four prompting modes (refer to Section <ref>). Fig. <ref> and Fig. <ref> show the results for 2D and 3D datasets, respectively. The findings indicate that SAM 2's single-frame 2D segmentation capability is comparable with that of SAM (a comparison of single-frame 2D segmentation results between SAM and SAM 2 is provided in Fig. <ref>). Similar to SAM, the performance of SAM 2 varies significantly across different datasets. For instance, SAM 2 achieves an impressive IoU of 0.907 on the Xray-Hip dataset for ilium but performs poorly with an IoU of 0.277 on the MRI-Spine dataset for GM.

Comparing the performance for different prompting modes, both the results on SAM and SAM 2 highlight the superiority of box prompts over point prompts. Moreover, prompts that indicate each separate part of the object individually are generally superior to those that indicate only one part or outline all parts in a single box, which is also consistent with the findings in previous SAM evaluations <cit.>. 






 §.§ Results compared with 3D model's Prediction Selection Strategy


We investigate the impact of prediction selection strategies across three initial frame selection modes: F-Mode 1, F-Mode 2, and F-Mode 3, all within the single point click setting (P-Mode 1) and using the front-to-end propagation direction (D-Mode 1). The results, depicted in Figure <ref>, demonstrate that for the annotated slice, opting for the first channel's prediction achieves performance that is comparable to or better than SAM 2's default method of choosing the most confident slice. Notably, for the MRI-Spine: SC dataset, selecting the first channel leads to an approximate 4% improvement in DSC when starting from the largest slice, and a 14% improvement when starting from the center slice and 17% when starting from 3 uniform slices.

Thus, we find that selecting the first channel from the annotated slices is a better choice compared to the highest confidence selection. Consequently, we will primarily focus on this selection method in the following sections.





 §.§ Results compared with different Initial frame selection and Prompt simulation strategies

As shown in Figure <ref>, we compare the results of providing prompts on the largest slice, the center slice, and three uniformly distributed slices.
We find that providing only one prompt for the entire volume results in unsatisfactory performance. For the front-to-end propagation (subplot row 1, first two plots), selecting the largest slice yields an average DSC of 6% across all datasets, while selecting the center slice yields an average DSC of 14%. For the bidirectional propagation (subplot row 2, first two plots), these numbers are 8% and 19%, respectively. These results show a dramatic decrease in performance compared to the previous 2D evaluation results across all datasets.

For different prompt modes, we do not observe a significant performance change. Providing one or more prompts (as shown by the different colors of the bars in each plot) yields consistent results. This is expected because additional prompts are only provided when there is more than one disconnected region. In most medical image datasets, the target object usually appears as a single connected region.

Comparing the performance of starting with annotations on three slices versus one slice (the three plots in the first row), we observe a slight improvement: around 10% compared to annotating the largest slice and 2% compared to annotating only the center slice. These results are surprising at first glance. However, upon deeper examination, we find that when annotating three uniformly distributed slices, the IoU from the center slice is usually the highest, while the IoUs from the first and last slices are lower (likely because the objects around the 1/4 and 3/4 quarters are smaller). Thus, the center slice's prediction mainly contributes to the final performance. Through these findings, we conclude that placing prompts on a few slices is not significantly helpful.

The low performance of SAM 2 raises some concerns since expanding the predictions from a single slice to a full volume is a common task in both the medical and natural image fields. 

To further explore whether the low performances stem from the propagation step (i.e., the annotated slice's performance not transferring well to other slices), we calculate the average IoU for the prompted slices as well on F-Mode 2 on two different propagation ways, resulting in 41.2% and 37.1% average IoU. We find that the segmentation performance on the prompted slices is quite consistent with previous 2D segmentation results. Therefore, the low results are more likely caused by the errors accumulated during the propagation. 

We hypothesize that there are two reasons for the unsatisfying behavior of SAM 2 during propagation: (1) SAM 2 was trained on a frame width of 8 frames, and thus the memory attention is most effective when the propagated frame is within 8 frames of the annotated frame. Medical volumes range from 10 to a few hundred slices. Consequentially, the performance will be significantly affected when the volumes are deeper. (2) SAM 2's primary objective is to segment any videos. Although this function can be transferred seamlessly between video segmentation and 3D segmentation, videos and medical volumes can differ significantly. Namely, the change between consecutive slices can be large for medical volumes due to the nature of medical imaging technology. In this case, we believe that direct usage of SAM 2 for 3D medical imaging segmentation tasks is not yet feasible. Further fine-tuning and redesigning of the memory strategy are needed.






 §.§ Impact of the propagation mode

Comparing the first and second rows in Figure <ref>, we find that applying bi-directional propagation can slightly improve the performance. When selecting the initial frame with F-Mode 1, i.e., the slice with the largest object of interest, the performance of SAM 2 improves around 2% IoU; and under F-Mode 2, i.e., the middle slice of the object of interest, SAM 2's performance improves around 5% IoU. 

These findings suppose that, though SAM 2 supports prompting on any frame as a condition to predict new frames, it is better to start propagating from adjacent slices as they are first added to the memory bank. Though the original SAM 2 was mainly examined on 1-N propagation mode in their experimental settings, the performance of reverse propagation (saying from K to K-1) can still work well given that the order of medical images is not relevant to the time order. 



 §.§ Results compared with 2D model and 3D model

When evaluating the 3D datasets, we can compare it on testing on 2D mode and testing on 3D mode. 
this should be done last and is optional. We can just describe the comparison verbally in 4.3






§ CONCLUSION

In this work, we conduct an initial investigation of the performance of SAM 2 in the medical imaging field. 
In addition to evaluating the 2D segmentation task, we investigate its ability to do the 3D segmentation task thanks to SAM 2's ability to segment videos. 
We find that under :

    
  * SAM 2 exhibits similar performance to SAM.
    
 
Under , we observe that:

    
  * When deciding the initial frame selection, selecting the center slice of the object of interest is better than selecting the slice with the largest object of interest.
    
  * Providing 1 or 3 points to the selected frame has little effect on the final performance. 
    
  * During propagation, it is better to utilize the first predicted mask instead of the one with the largest confidence.
    
  * Propagating predictions bidirectionally, starting from the annotated slice, works better than propagation from beginning to end.
    
  * The best average performance of SAM 2 in the 3D segmentation is 0.19 IoU when only one point prompt is provided to one volume.



splncs04



 Paper Title: Faithful and Plausible Natural Language Explanations for Image Classification: A Pipeline Approach.


Visual Analysis of GitHub Issues to Gain Insights

    Rifat Ara Proma
Scientific Computing and Imaging Institute 

University of Utah

Salt Lake City, UT, USA 

proma@sci.utah.edu
Paul Rosen
Scientific Computing and Imaging Institute 

University of Utah

Salt Lake City, UT, USA 

prosen@sci.utah.edu

    August 5, 2024
============================================================================================================================================================================================================================================================





§ ABSTRACT 
Existing explanation methods for image classification struggle to provide faithful and plausible explanations. 
This paper addresses this issue by proposing a post-hoc natural language explanation method that can be applied to any CNN-based classifier without altering its training process or affecting predictive performance. 
By analysing influential neurons and the corresponding activation maps, the method generates a faithful description of the classifier's decision process in the form of a structured meaning representation, which is then converted into text by a language model. 
Through this pipeline approach, the generated explanations are grounded in the neural network architecture, providing accurate insight into the classification process while remaining accessible to non-experts. 
Experimental results show that the NLEs constructed by our method are significantly more plausible and faithful. In particular, user interventions in the neural network structure (masking of neurons) are three times more effective than the baselines.




§ INTRODUCTION


Despite remarkable advances in computer vision, the deployment of image classification systems, especially in critical domains, poses significant challenges.
One of them is the opacity of deep models and the difficulty of providing reliable explanations for their predictions <cit.>. 

Therefore, several types of explanation methods have been proposed, including various forms of saliency maps <cit.>, feature importances <cit.>, concept-based explanations <cit.>, counterfactual explanations <cit.>, etc.
A particularly interesting form of explaining predictions is offered by natural language explanations (NLE) techniques <cit.>. 
Such explanations are not only 
understandable by non-expert users, but can also be used to support conversations with the user in a dialogue system <cit.>.


















There are two critical properties of an explanation: faithfulness and plausibility <cit.>. 
A faithful explanation should accurately reflect the inner workings of the
system and provide information on the real reasons why the model reached a certain decision. 
Plausibility then refers to how convincing the explanation appears to the user.


In the case of NLE, obtaining high plausibility is straightforward, as textual explanations are usually human-friendly <cit.>, but achieving high fidelity is challenging. 
In the context of image classification, image captioning methods offer plausible but unfaithful NLEs <cit.>.

Some methods try to improve faithfulness by conditioning generation on both the predicted class and image features <cit.>, but the faithfulness provided is still limited as the model is not aware of the classifier's decision process.

Other methods train image classifiers to jointly predict the class and visual rationales, and generate explanations based on them <cit.>. However, the rationales are predicted independently of the class and do not participate in the classifier's decision process.

Most importantly, such methods change the training procedure and the architecture of the classifier, often affecting classification performance.



In this paper, we propose a post-hoc natural language explanation method for image classification that can be used with any standard convolutional neural network  (CNN) classifier. 
To illustrate the classification decision process, the method analyses which neurons of the CNN were most influential in reaching a given decision, and which regions of the image caused them to activate.
For each influential neuron, a neuron annotation method computes visual pattern exemplars and produces a short (one phrase) description of a pattern that the neuron detects.
The information gathered in this process serves to create a simple, human-readable meaning representation, which is converted to text by a large language model (LLM).


Our simple two-step pipeline method (see Fig. <ref>) produces explanations that are directly grounded in the classifier's network architecture, but without interfering with its training process or affecting its predictive performance. 
It also does not need any gold-standard explanations.
The provided NLEs reflect the underlying classification process by specifying the most influential neurons computed by a well-established explainable AI methods.
At the same time, the final text is fluent and easy to understand.
This results in NLEs that are both plausible and highly faithful.

Our experimental code is publicly available.[<http://tinyurl.com/3da45hnm>]




§ METHOD

Our NLE generation approach uses two processing steps, described below: meaning representation (MR) construction and MR-to-text conversion.





 §.§ Meaning representation construction

We first produce a meaning representation  
in the form of a JSON object, containing information about the neurons responsible for a given classifier prediction (why?), what patterns those neurons detected (what?), and in which parts of the image they were activated (where?).
The MR includes the predicted class and the list of most influential neurons, each represented by



(1) description – a phrase describing the pattern that most excites the neuron (convolutional filter), (2) positions – list of coarse-grained image positions (e.g., “bottom-right corner”) where the neuron was activated. 
An example of a MR is provided in Fig. <ref>.









MR construction starts by storing all neuron activations from the given CNN-based classifier for the prediction to explain.
Next, the most important neurons are selected, annotated with a description, and tied to an image region, as follows:







( predicted_class , [
  "description": "neuron_description", "positions" : [neuron_positions],
   "description": ..., ... 
   ])





  
Selecting the most influential neurons
To choose the most relevant neurons, we apply well-established Layer-wise Relevance Propagation (LRP) method <cit.>. 
LRP performs a backward pass through the classifier network to establish the influence of each neuron to the final prediction (see App. <ref> for formulas).


We select k neurons with the highest LRP 
scores (with k being a parameter controlling the brevity-detail tradeoff). 

 


  
Neuron annotation
We adopt the MILAN neuron annotator <cit.> to generate descriptions of selected neurons.
MILAN first finds images in the classifier training set 
that make a given neuron highly activated 
<cit.>.
These exemplar images 
are used to generate a description of the pattern that this neuron detects.


Note that although the last step of MILAN is essentially image captioning, it does not affect the faithfulness of NLEs produced by the pipeline, as long as its output is of sufficient quality. 
The images that illustrate a neuron's decision process are computed by analysing its activations, and the captioning is only used to convert the result into text. 




  
Establishing image regions for neuron activation

The neuron's raw activation map
is divided into a 3x3 grid 

with manually assigned labels such as `top-left corner', `top', `top-right corner', etc. 

We then select
all grid cells where the neuron's activation exceeded half of its maximum value.


We apply several substitution rules (see App. <ref>) to make the list of cells shorter and more human-readable. 



 §.§ Explanation generation

The second step of our method is converting the human-readable and faithful MR created above

into a user-friendly text.
As we do not have any gold-standard explanation texts
, the task is performed by prompting a large language model (LLM). 

We instructed the model to (1) produce fluent text, (2) summarise the content of the MR (e.g. if two neurons detect similar patterns, they can be combined in the text), (3) prioritise readability, (4) come up with its own formulation of spatial positions to improve fluency. We  also provide one handcrafted MR-to-text conversion example. 

The 
prompt is shown in App. <ref>.
LLMs could in theory hallucinate and thus reduce the explanations' faithfulness.
However, in Section <ref> we show experimentally that current LLMs are reliable enough to produce useful explanations. 






§ EXPERIMENTAL EVALUATION







 §.§ Experimental setup




  
Dataset All experiments were performed on the ImageNet dataset.
The classifier was trained on the train set and our explanation method was run to explain predictions made on the validation data.[Annotated ImageNet test set is not publicly available.]



  
Models 
We experiment with explaining the predictions of the smallest CNN classifier from the popular ResNet family: ResNet18 <cit.>.[
It reaches only a 41.4% accuracy on the validation set, but high classification performance is not the goal of our study.] 
We fill our MRs with k=10 top neurons indicated by LRP from the Captum library <cit.> and annotate them using MILAN's original implementation. 

As the LLM for MR-to-text conversion, we use GPT-4 <cit.>.



  
Baselines
We compare to the following methods: 


    
  * Show, attend and tell (SAT) by <cit.> is an  image captioning method 
    used for explaining predictions <cit.>. 

  * NLX-GPT <cit.> is an explainable visual question-answering method that produces NLEs with an encoder-decoder architecture that combines CNN with a transformer-based language model.






 §.§ Are the output explanations plausible?



To assess the plausibility 
of generated explanations,
we conducted a small-scale manual annotation experiment.
We recruited ten annotators: five non-experts hired on the Prolific platform and five experts with at least one published paper on explainable AI. 
Each annotator was presented with 30 image-explanation pairs (300 in total) and asked to rate them on a scale of 1-5 
whether the explanation was (1) fluent, (2) easy to understand (comprehensible), (3) convincing, and (4) explanatory for the underlying decision process.[Note that the question on understanding the decision process does not measure faithfulness, but the user's subjective opinion on whether they understand how the model works.] The overall quality of the explanations was also rated (see App. <ref>).

The results are presented in Tab. <ref>.
Our method obtains the highest overall quality according to both experts and non-experts.
It also produces the most plausible explanations (most convincing and explanatory). 
Since the baselines produce much shorter explanations, 
it is not surprising that our longer explanations are a bit more difficult to understand. 

Interestingly, experts generally give higher ratings than non-experts.
This is true for all methods and all factors except the explanatory ability. 
For this factor, experts rate the baselines lower than non-experts, but they consistently rate the explanations provided by our pipeline higher.
The improvements of our method over baselines are statistically significant on both plausibility measures and overall quality. For fluency, 

our method is indistinguishable from SAT (see details in App. <ref>).



 §.§ Are the output explanations faithful?






The faithfulness of the generated explanations is assessed through two intervention experiments: (1) checking if rationales from NLEs change the prediction by masking parts of input images, (2) influencing the network prediction by masking influential neurons.
We further assess the stability and diversity of the explanations
for our method, and we directly evaluate the reliability of our MR-to-text conversion.







































  
Masking input image
We asked 
annotators to cover with white rectangles parts of images that contained the decision rationale indicated in the NLE, 50% area at most (see App. <ref> for details).
We re-classified covered images and measured changes in prediction and the average decrease in the probability of the originally predicted class.
We also performed an opposite experiment, with the annotators highlighting only parts of image mentioned in the explanation and covering the rest.


For covering, the use of our NLEs resulted in the highest average probability decrease and the change of the original prediction for 88% of examples (see Table <ref>). 
Our method reached the best results in the highlighting experiment as well, producing the least amount of changes.

We also re-ran our NLE pipeline with parts of the input image covered. This led to significant changes: on average 78% (median 90%) of the neurons indicated in MRs were different.



  
Masking influential neurons
To show the NLEs' ability to reflect classifier decisions, we asked the annotators to read the NLEs and select up to five most influential neurons from a MILAN-annotated list. 
The classifier was then re-run with the selected neurons masked. The results 
in Table <ref> reveal that masking neurons suggested by our NLEs led to a five times higher decrease in the predicted class probability and over three times higher class flip rate than baselines. More details are in App. <ref>.



  
Explanation stability analysis
Following <cit.>,
we measure explanation robustness against adding random noise to the input image (intra-set stability, see  App. <ref> for details) by comparing BLEU <cit.>, METEOR <cit.> as well as class flip frequency and class probability change against the original predictions.
We also check for outputs' diversity (inter-set stability) using BLEU  and METEOR  overlap  against explanations for other classes.
The results in Table <ref> show high diversity and high noise sensitivity:
In parallel to increased classification changes with added noise,
explanation BLEU/METEOR gradually drop, but they are lowest when comparing between different classes.




  
Reliability of the MR-to-text transform

The human evaluation of our approach's MR-to-text reliability was similar to plausibility evaluation, but limited to non-expert Prolific annotators.
We asked five yes-no questions on information in the text not grounded in the MR (i.e., hallucinations), omission of MR information, fluency, spatial information fidelity,
and overall correctness (see App. <ref>).



The results show the MR-to-text conversion as highly reliable, as only 8% texts contain hallucinations. Omissions are more frequent (44%), but this is expected as the LLM is instructed to summarise the MR and prioritise readability. 
This factor most likely affected the overall score (58%). 
The explanations are mostly fluent (96%), with correct spatial information (82%).





§ ACKNOWLEDGEMENTS

Co-funded by the European Union (ERC, NG-NLG, 101039303) and National Science Centre, Poland (Grant No. 2022/47/D/ST6/01770).
This work used resources of the LINDAT/CLARIAH-CZ Research Infrastructure (Czech Ministry of Education, Youth, and Sports project No. LM2018101).




§ LIMITATIONS


This paper produces a new method for plausible and more faithful natural language explanations for image classification.
Although we believe that the method provides significantly better  faithfulness than the previously proposed methods, it does not obtain completely faithful explanations. 
The faithfulness of the explanations provided by our method depends on the quality of the neuron annotations produced by MILAN and the neurons indicated by LRP. Both techniques can be considered as state of the art, but they still occasionally produce incorrect results.
Therefore, the results of NLE methods should be treated with caution.
Additionally, this work uses pre-trained language models, which are known to expose certain social biases reflected in their training data.







§ EXAMPLES OF EXPLANATIONS

Several examples of explanations provided by the methods under study are given in Tab. <ref>.
More examples of NLE produced with the proposed pipeline approach are given in Tab. <ref>.







§ DETAILED RESULTS OF MASKING INFLUENTIAL NEURONS EXPERIMENT


The detailed results of the experiment with masking influential neurons are presented in Fig. <ref>.
For our method, the results indicate that masking neurons in the order indicated by the annotators leads to an increasing change in the classifier's prediction and a larger decrease in the predicted class probability value.
In contrast, the annotators gain limited insight into how the neural network made a decision from the explanations provided by other methods: masking the indicated neurons leads to limited change, and increasing the number of masked neurons leads to almost no improvement.
We attribute the effectiveness of masking the first indicated neuron to examples of classes that are highly correlated with a pattern detected by a particular neuron.





§ SIMPLIFICATION RULES FOR SPACIAL POSITIONS


The spatial information about neuron activation is learned by inspecting activation maps of the chosen neurons. The activation map is converted to a binary image, such that if a given pixel of an activation map exceeds half of a maximal value in that activation map it is converted to the value of one, and the rest of the pixels are assigned with the value zero. Then the binary activation map is divided into a 3x3 grid of congruent squares oriented such that the ordering begins with a 0 in the top-left corner, progressing sequentially across each row from left to right and top to bottom, culminating with the number 8 in the bottom-right position.  The basic positions names are given below.
captionpos=b

[label=lst:basic_positions]
0 : "top-left corner", 
1 : "top", 
2 : "top-right corner", 
3 : "left", 
4 : "center", 
5 : "right", 
6 : "bottom-left corner", 
7 : "bottom", 
8 : "bottom-right corner"


The final appearance of the positions appended to the meaning representation is governed by a set of rules, where position names are assigned to sets of basic positions.
 If all basic positions from the set are present in the MR, we replace them with a corresponding compound position.  These sets and position names are given below.

[label=lst:compound_positions,breaklines=true]
"entire top": 0, 1, 2,
"entire bottom": 6, 7, 8,
"entire left": 0, 3, 6,
"entire right": 2, 5, 8,
"perimeter": 0, 1, 2, 5, 8, 7, 6, 3,
"center cross": 1, 3, 4, 5, 7,
"upper half": 0, 1, 2, 3, 4, 5,
"lower half": 3, 4, 5, 6, 7, 8,
"left half": 0, 1, 3, 4, 6, 7,
"right half": 1, 2, 4, 5, 7, 8


Additionally if "entire *" and "* half" coexist in the meaning representation simultaneously, where "*" represents one of {"left", "right", "top", "bottom"}, in both compound positions names, the "entire *" is deleted from the meaning representation, since it is comprised of the subset of the set, which makes up corresponding "* half". An example would be if "entire top" and "upper half" are both in meaning representation, the "entire top" would be deleted since it is comprised of {0, 1, 2} tiles, which also partially make up the "upper half" ({0, 1, 2, 3, 4, 5}). Finally, if more than any, distinct, 6 out of 9 tiles are active, every position of a given neuron is deleted and the spatial information placeholder is set to "entire image". This whole approach significantly reduces the length of the positional representation, while simultaneously making the position names more plausible to the system user, since we believe the final space of possible positions is natural and intuitive for humans to grasp quickly and effectively.




§ PROMPT FOR MR-TO-TEXT CONVERSION


To convert meaning representations into text, the following prompt was applied to the language model:

You are given a problem of creating textual explanation of an image classification performed by neural network. You will be given a Python object representing network output in the form of `'image class', [('detected object', 'position'), ...]`. I want you to convert this object into a textual explanation. You should:
1. Create a grammatically correct sentence which will explain the model's decision.
2. Decide which detected objects do not fit with image class and do not include them in the explanation. For instance, 'dentist' class and 'animal heads' objects are completely unrelated. However, the descriptions that aren't directly related to image class, but can be indirectly correlated, especially in terms of shape, color, or texture resemblance should be included (like 'fountain' and 'sea' because of the water they have in common or 'brick wall' and 'grid' because the texture is similar). Never mention that you chose neuron descriptions and do not talk about the neuron descriptions that were discarded.
3. Prioritize the readability of the explanation. Include only essential detected objects and aggregate information to shorten the explanation.
4. Aggregate positions if possible, for example ['bottom-left corner', 'bottom', 'bottom-right corner'] should be aggregated into 'bottom'. If the positions list is too long or too ambiguous do not include them in the explanation. 

Here is an example.
Python object:
“lakeside, ['description': 'Nature', 'positions': ['left', 'right', 'bottom'], 'description': 'The sky', 'positions': ['top-right corner', 'bottom-left corner', 'bottom'], 'description': 'Red and white colored objects', 'positions': ['left', 'right', 'bottom'], 'description': 'The ocean', 'positions': ['left', 'right', 'bottom'], 'description': 'Animal heads', 'positions': [], 'id', 'description': 'The color red', 'positions': ['left', 'right', 'bottom']', 'description': 'White backgrounds', 'positions': ['bottom', 'left'], 'description': 'Grass', 'positions': ['left', 'bottom-left corner', 'bottom', 'bottom-right corner'], 'description': 'Dogs and guinea pig', 'positions': ['center'], 'description': 'The color green', 'positions': ['top-right corner', 'right', 'bottom-right corner']]”
Answer: “The model assigned this image to the "lakeside" class because in the last layer it discovered nature, and the ocean at the left, right, and bottom of the image. It also detected grass at the left, bottom-left corner, bottom, and bottom-right corner and the color green at the right of the image.
(...)




§ HUMAN EVALUATION OF EXPLANATION PLAUSIBILITY


The annotators are presented with an image, model's prediction and a natural language explanation. 
Each question is answered on a scale from 1 (low) to 5 (high).
The following binary questions are asked:

    
  * How fluent (linguistically correct) the text is?	
    
  * How easy to understand the text is?	
    
  * How convincing do you find the explanation of the decision made by the model?	
    
  * After reading the explanation, how well do you understand how the decision of the model was taken?	
    
  * How would you rate the overall quality of the explanation?


The annotation instructions are provided in the code repository.



§ HUMAN EVALUATION OF MR-TO-TEXT TRANSFORMATION


The annotators are presented with a meaning representation in the form of formatted JSON without a given image, since it should not influence the assessment of MR-to-text transformation. The following binary questions are asked:

    
  * Does the text contain information that was not present in the meaning representation?	
    
  * Is there any important information from meaning representation omitted in the text? 
    
  * Is the text linguistically correct?	
    
  * If any spatial compression occurred between explained neurons, the said compression is correct?	
    
  * Overall, do you find this meaning representation to text transformation acceptable i.e. sufficiently good for explanation purposes?


The annotation instructions are provided in the code repository.




§ DETAILS ON STABILITY EXPERIMENTS


Let us assume that an image is a matrix X, such that X ∈ [0, 1]. We model input perturbations by adding random noise to the images from the standard normal distribution. 
Since an interval of possible pixel values  is [0, 1], to account for the unboundedness of a standard normal distribution we use a clipping operation defined as follows:

    clip(x) = 
    
    0,    if  x < 0 
    
    x,    if  0 ≤ x ≤ 1 
    
    1,    if  x > 1

The formula for the function ζ that adds a perturbation to the image is given below.

    ζ(X)=clip(𝒩(0, 1) · i + X)

where i is the noise intensity.
We perform two types of stability experiments:

    
  * inter-set stability: We compare pairs of unaltered images (1^st set of images) and corresponding images mapped by the ζ function (2^nd set of images). After the ζ mapping is performed, we run the proposed method on both images, yielding two explanations. We then compute various language similarity metrics between the two explanations. 
    
    We compare two kinds of pairs of images: unaltered - lightly perturbed (i=0.05)  and unaltered - heavily perturbed (i=0.2). 
    
  * intra-set stability: We compare explanations pairs produced for different images to verify the diversity of generated explanations. 

The computations were conducted on explanations produced by the proposed method for a 500-element subset of validation ImageNet data.
The subset was constructed by randomly picking 50 examples from 10 selected, diverse classes (library, over skirt, palace, prison, wall clock, lakeside, coral reef, volcano, fountain, basset).






§ DETAILS ON THE COVERING EXPERIMENT


Given an image and explanation, we instructed the annotator to cover the decision rationales with white rectangles.
The annotator instruction is given below:

    Based on the following explanation of the classifier's prediction, cover the reasons for its decision with white rectangles. You can use as many rectangles as you like, but the total area of the covered image cannot be larger than half of the image. If it is not possible to cover the mentioned reasons by converting only 50% of the image, please do your best to cover the most important information. 

An example of annotation provided is given in Fig <ref>.



§ LAYER-WISE RELEVANCE PROPAGATION 


To choose the most relevant neurons, the well-established Layer-wise Relevance Propagation (LRP) method is applied <cit.>. LRP performs a special backward pass through the neural network  
to establish the influence of each neuron to the final prediction.

Starting with the predicted value, LRP distributes it among the neurons in each layer, assigning them relevance scores.
The following rule for relevance reallocation is used: 

    R_i = ∑_j z_ij/∑_i z_ij R_j

where R_i is the i-th neuron relevance score, 
z_ij express how much i-th neuron has contributed to make j-th neuron relevant (calculated as the product of the neuron's activation and the corresponding weight), the sums ∑_i (∑_j) iterate over all neurons in a given (next) layer.



  
Choice of LRP as an explanation method
Although we present a pipeline approach and there is some variability in how it can be implemented, we believe there are important reasons for using our pipeline with LRP.

First, unlike many other methods, LRP works at the neuron level, which is strictly required by our method. Therefore, methods that provide pixel-level importance scores such as RISE <cit.> or Grad-CAM <cit.>, methods that typically work on image segments such as LIME <cit.> or SHAP <cit.> are not suitable for our approach.
 
Second, LRP has been shown to achieve high faithfulness in many studies and can be considered state of the art in this respect. Note that achieving high faithfulness is the main goal of our approach.

Finally, LRP is theoretically motivated, has been shown to be useful in many applications, and has stable open source implementations. 



§ STATISTICAL ANALYSIS OF THE HUMAN EVALUATION RESULTS


For the results of human evaluation of plausibility, we performed the non-parametric global Friedman test followed by Nemenyi post-hoc analysis (as recommended in <cit.>). 
We were able to reject the null hypothesis of the Friedman test for all the measures with p<0.001. The Nemenyi post-hoc analysis with α=5% confirmed that our method obtains statistically significant improvements over other compared methods on both plausibility measures and the overall quality measure. On the fluency measure, our method is undistinguishable from SAT.

The critical distance plots from Nemenyi post-hoc analysis are provided in Figure <ref>. The lower result, the better. If the difference between the methods is not statistically significant, their results are connected with a thick horizontal line. More details on these plots can be found in <cit.>.









 Paper Title: MaskUno: Switch-Split Block For Enhancing Instance Segmentation.


[


    
    
    
    
    
    MaskUno: Switch-Split Block For Enhancing Instance Segmentation 

    
    
    


Jawad Haidar, Marc Mouawad, Imad Elhajj, Daniel Asmar




Vision and Robotics Lab (VRL), Maroun Semaan Faculty of Engineering and Architecture, American University of Beirut, Beirut, Lebanon


   



]


§ ABSTRACT 
Instance segmentation is an advanced form of image segmentation which, beyond traditional segmentation, requires identifying individual instances of repeating objects in a scene. Mask R-CNN is the most common architecture for instance segmentation, and improvements to this architecture include steps such as benefiting from bounding box refinements, adding semantics, or backbone enhancements. In all the proposed variations to date, the problem of competing kernels (each class aims to maximize its own accuracy) persists when models try to synchronously learn numerous classes. In this paper, we propose mitigating this problem by replacing mask prediction with a Switch-Split block that processes refined ROIs, classifies them, and assigns them to specialized mask predictors. We name the method MaskUno and test it on various models from the literature, which are then trained on multiple classes using the benchmark COCO dataset. An increase in the mean Average Precision (mAP) of 2.03% was observed for the high-performing DetectoRS when trained on 80 classes. MaskUno proved to enhance the mAP of instance segmentation models regardless of the number and type of classes or the type of architecture in which it was applied.




instance segmentation, competing kernels




§ INTRODUCTION
 
Instance segmentation is a fundamental task in computer vision, which combines mask segmentation and object detection, and whose goal is to perform a per-pixel classification of different objects in a scene. Instance segmentation has a wide range of applications such as in medical diagnostics <cit.>, agricultural analysis <cit.>, and autonomous driving <cit.>. Its abundant use in practical scenarios necessitates accurate and robust predictions. The problem of instance segmentation is difficult and faces challenges such as occlusion, scale changes, and cluttered backgrounds  <cit.>.   

With an eye toward enhancing the performance of instance segmentation, some researchers adopted the cascade methods that rely on progressive refinement <cit.>. For example, one could pipe the output of the the bounding box prediction from a Stage 1 cascade into a region extractor (Stage 2), and so on, such that the progressive flow of the refined bounding boxes guarantees better-localized feature extraction. As a result, a higher accuracy is achieved in delineating masks and bounding boxes. 
    
Another research direction concentrates on designing backbone networks that can detect features that are better suited for the downstream task of instance segmentation. There exists a trade-off between the depth of a backbone model and the spatial resolution of features. Contrary to image classification, instance segmentation requires high spatial resolution of features, and so research on this front is concerned in architecting backbone networks that can identify features with high spatial resolutions
     <cit.>.  



Most previous work in instance segmentation adopts the pipeline of Mask R-CNN <cit.>: after extracting features using a backbone, a Region Proposal Network (RPN) localizes the regions that encompass the sought-after objects. The localized regions are then extracted using a region of interest block and fed to detection and mask generation branches. To avoid competition among classes, Mask R-CNN replaces the multi-nomial cross entropy with a per-class sigmoid and binary loss <cit.>, keeping in mind that the weights are common between different classes given that they belong to the same branch.      
To tackle this gap, our method works on top of most instance segmentation methods by simply isolating the branches corresponding to different classes. Specifically, our pipeline consists of two main parts: detection and per-class segmentation. The output of the detection is fed to a switch that directs the bounding boxes to their corresponding segmentation branch. Consequently, the optimization of each branch is done independently to guarantee non-competing kernels between different classes. These simple modifications significantly improve the performance of the baseline Mask R-CNN <cit.> and the high performing CNN based model Detectors <cit.>.

The remainder of this paper is organized as follows. In the next section, we will delve deeper into the recent methods in the field of instance segmentation. In Section <ref>, the MaskUno proposed method is introduced. In Section <ref>, experiments are conducted and results demonstrate the enhancements obtained by MaskUno when applied to various instance segmentation models, trained and tested on the standard COCO dataset benchmark. Finally, Section <ref> concludes the paper.




§ RELATED WORK


The architecture of an instance segmentation model contains different blocks, each contributing in a different manner to the end result. First, the backbone is responsible for extracting high-level features of the image. One can usually add a Feature Pyramid Network (FPN) that can help the network gain multi-scale context, which is crucial for detecting objects of different sizes. Next, comes the RPN, which generates from a feature map potential region candidates that might contain objects. The ROI-Align layer ensures that all the regions of interest are of similar size. Finally, the regions of interest are fed to a terminal block which classifies them and predicts bounding boxes and binary masks for each instance. In the following section, we will review the various interventions at the level of the backbone, object detection, and cascade. 



 §.§ Backbones
 

The pillar of every deep learning model is the feature extraction block (named backbone) and most of them are built around a CNN. The downside of such models is that spatial resolution decreases with depth, which is problematic for instance segmentation models that require high spatial resolution for pixel-wise mask prediction. To address this issue, SpineNet <cit.> uses Neural Architecture Search to learn scale permuted features as well as cross-scale connections. DetectoRS <cit.> proposes a recursive FPN  that mimics thinking twice at a macro level; more specifically, the outputs of the FPN are fed to the bottom-up backbone. Additionally, they introduce the looking twice concept at the micro level, which is achieved by adding Switchable Atrous Convolutions.



 §.§ Object detection
 
Object detection is an essential block for instance segmentation as it delineates the bounding box surrounding each of the detected objects in a scene. 

Object detection methods can be based either on one stage<cit.>, or two stages <cit.>, where the former achieves better inference speed and the latter provides higher accuracy. The high inference speed in one-stage detectors is ascribed to their simple architecture; for instance, YOLO <cit.> divides the image into a grid and places various anchors having different scales and aspect ratios. Then, the anchors are directly classified and adjusted after a feature extraction stage. For the sake of predicting small-scale objects, SSD <cit.> performs detection at different stages of the feature extraction. RetinaNet <cit.> proposes the focal loss to attenuate the effect of the abundant easy negatives, resulting in improved accuracy over the standard cross entropy loss. To get rid of anchors, CornerNet <cit.>  expresses the bounding boxes by two points, the top-left and bottom-right corners. CenterNet <cit.> uses the corners as proposals, and the center to verify the class of the object. A more accurate object representation is proposed by ExtremeNet <cit.> that replaces the corners with the objects' extreme points.  


On the other hand, two-stage object detectors include an RPN, whose role is to first propose regions with high objectiveness probability, and output class-agnostic bounding boxes of possible objects. These bounding boxes are then used by the region of interest extractor.

R-CNN <cit.> uses a non-learning-based proposal generator to propose possible objects, and then each of these regions of interest is passed through a CNN to extract corresponding features. This is a slow operation as each box is passed independently. On the other hand, Fast R-CNN <cit.> applies a feature extractor to the entire image and then extracts the corresponding features. Finally, Faster R-CNN <cit.>, replaces the non-learning-based proposal method with a learnable RPN. Mask R-CNN <cit.> adds a mask prediction head to support instance segmentation, and additionally proposes a region of interest alignment block that takes into account the neighboring pixels surrounding the bounding box.
 



 §.§ Cascade
 

Cascade is the process of iterating the output of a model to achieve progressive refinement. Cascade R-CNN <cit.> proposes to add a simple cascade architecture on top of Faster R-CNN. A Stage 1 bounding box prediction is fed to a region of interest extractor and has the effect of superior localized bounding boxes that can contribute to enhanced feature extraction. Likewise, Cascade Mask R-CNN applies this concept to enhance mask predictions. Hybrid Task Cascade (HTC) introduces direct connections between the different stages of mask branches, thereby strengthening the flow by integrating complementary features across stages <cit.>.






To summarize, the most significant enhancements to instance segmentation in the literature include interventions at the level of the cascade, loss function, or backbone. In this paper, we propose to intervene at the level of the last prediction layer, in a manner to mitigate the problem of competing gradients between different classes. As far as we know, none of the previous methods do this. 

The contributions of this paper include the following:  


  * Proposing a  modular Switch-Split block that can replace multi-class prediction heads in most instance segmentation methods. 

  * Ensuring no competing kernels between the different classes, which leads to richer representations as no trade-off between classes is permitted.

  * Enhancing the accuracy of instance segmentation models on the standard COCO dataset benchmark.





§ PROPOSED SYSTEM
 



Recalling from the previous section, at the output of an instance segmentation model, detected regions of interest are streamed into a terminal block consisting of three different branches: the first one is usually a classifier indicating the presence of an object in this region, and if so classifies it. The second branch predicts refined bounding box coordinates for each proposed region, adjusting the initially generated region proposals. The third branch takes the ROI-aligned features as input and predicts a binary mask for each object instance within the proposed regions. These masks indicate the pixel-level segmentation of the objects.

To our knowledge, all instance segmentation architectures in the literature include the aforementioned terminal block (i.e., including the three branches).  An interesting observation we came to realize is that this block in its current form suffers from what we refer to as competing kernels. The competing kernels problem arises when a model tries to synchronously learn multiple classes having different features. During training, the kernels learned for each class compete with others from another class in an attempt to maximize their own segmentation accuracy. 

The main intuition driving our proposed MaskUno system is that each class should be learned on its own in a block we name the Switch-Split block.



 §.§ Switch-Split block

The idea behind adding a switch and split block is to first benefit from the bounding box refinement and then specialize the learning per class. In other words, we first pass the bounding box head output through the ROI Align layer, then use it as an input to the Switch-Split block (see Fig.<ref>). The output of the classifier is given as an input to the switch, then the classifier, based on the input, classifies the refined ROI which then turns the switch to the corresponding class inside the splitting block. Finally, each ROI is assigned to a specific one-class mask head. 

The split block is a set containing N+1 blocks, each trained to classify and predict the bounding box and the mask of one specific class.

This idea applies to any architecture based on Mask-RCNN, and to any class. In this section, we will show how to adapt it to the baseline Mask-RCNN, the Cascade Mask-RCNN, and the hybrid task cascade. 



  §.§.§ Mask-RCNN

instead of having the bounding boxes and the masks predicted in parallel as is usually done in Mask-RCNN, in MaskUno it is done sequentially. In other words, the bounding boxes are first predicted, and then based on these refined boxes the mask is estimated. Figure <ref> presents an example of the methodology applied to this architecture.




  §.§.§ Cascade Mask-RCNN and Hybrid Task Cascade


this architecture already includes bounding box refinements from one layer to the other, meaning that the output of the refined bounding box of Layer i+1, is fed as an input to the Switch-Split block of Layer i.
The split block is a set containing N+1 blocks, each trained to classify, and predict the mask of one specific class. Figure <ref> shows how each block is represented: they all include a mask head, all having the same functionality as the branches of the basic Mask-RCNN pipeline. In terms of hybrid-task cascade, we also replace every mask-head with a Switch-Split block, taking as input the refined classified bounding boxes as shown in Fig. <ref>.






 §.§ Loss functions


Every one of these architectures is allocated a set of three loss functions, one for every sub-part of this block. Hence, there is the categorical cross-entropy loss defined by (<ref>), where y_i is the true class label, and ŷ_i is the predicted class label, and N is the number of classes. 


    L_cls = - ∑_i=1^N y_i log(ŷ_i)


The second loss function is the smooth L1 loss defined by (<ref>) where x represents the difference between the predicted and the true value.


    L1(x) = 
    0.5x^2,    if  |x| < 1 
    
    |x| - 0.5,    otherwise


For the mask prediction part, the loss function is a pixel-wise binary cross entropy defined by (<ref>).


    L_mask = - ∑_i,j [ y_true(i, j) log(y_pred(i, j)) + 
            
    (1 - y_true(i, j)) log(1 - y_pred(i, j))

where y_true is the true pixel label, and y_pred is the predicted pixel label. 



 §.§ Training

A typical model contains its own set of loss functions [L_cls,  L1,  L_mask] that it will try to minimize, while eliminating the issue of competing kernels. The mask head of class i will have its own L_mask_i representing the pixel-wise loss function that will be minimized. Having different loss functions for every class is also essential to solve the issue of competing kernels as it makes the weights of every mask head independent of one another also in terms of cost.

The training schedule and hyper-parameters are kept the same for every block to avoid overfitting on a specific class. 



 §.§ Generalizing to other pipelines for instance segmentation

The switch split method should be applicable to any architecture, by first identifying the blocks where classification, bounding box regression, and mask prediction, are implemented, and then splitting the segmentation block into multiple ones, each specialized for the concerned class.

As a proof of concept, one can even, perform sequential training for every architecture, meaning train the model on Class 1 to obtain a specialized mask-head for this class. Then, train a new mask-head using the original checkpoint of the imported model on Class 2, up until all needed classes have a specialized mask-head. 

In Section IV, we perform experiments on models in the literature to validate our proposed method.










§ EXPERIMENTATION AND DISCUSSION
























 §.§ Datasets and metrics



  §.§.§ Data used

Experiments are performed on the Common Object in Context (COCO) dataset <cit.> because it is challenging due to the numerous classes it contains, and its high number of samples, but also to make a fair comparison with the state-of-the-art models trained using this benchmark. We use the train2017 and val2017 data which contain a total of 115,000 and 5,000 images respectively. The data is split per-class for both training and validation. For the bounding box and mask training, instance annotations are used, and for the semantics, the COCO-stuff annotations.





  §.§.§ Evaluation


to evaluate the model performance, we use the mean average precision taken at different IOU thresholds (at 0.5 and 0.75). Additionally, different scales are taken for the data: small, medium, and large.  We report the AP, AP_50, AP_75, AP_s, AP_m, and AP_l.


 §.§ Experimentation


To validate the generalization of MaskUno we test it on various classes and various architectures. The process starts by removing the multiclass mask prediction head from a pre-trained model with a plateau in its accuracy, followed by adding the Switch-Split block and finally training the new weights on the sought classes. Afterward, we compare its mean Average Precision (mAP) with the one obtained before training the model. The hypothesis is that its mean average precision increases for every class using the MaskUno method.
The experimentats are divided accordingly. First, using the COCO dataset, and the library MMDetection <cit.>, a similar experiment is conducted on four different models: the baseline Mask-RCNN <cit.>, the Cascade-Mask-RCNN <cit.>, the Hybrid Task Cascade <cit.>, and the DetectoRS backbone <cit.> on a list of 10 classes. Since the pre-trained models have more classes compared to the MaskUno models, we separate the validation data into different sub-datasets, each responsible for validating a specific class. 
Furthermore, we experiment on a total of 80 classes and compare the results for Mask-RCNN and DetectoRS. Mask-RCNN is used since it is the baseline model for instance segmentation. DetectoRS is an architecture that contains a change in backbone, as well as a hybrid task cascade architecture. 
Note that, given the methodology, we care about the mask prediction results for all models and classes.



 §.§ Results and discussion


We discuss in this section the results obtained after applying our method to different models and a variety of classes. Table <ref> shows the results of the four models on every class. The experiments consist of training each of these previously assumed saturated models, on a list of ten classes, shown in the column of the table. The models are chosen in a way to include the different techniques found in the literature, including (1) a change in the backbone (DetectoRS), (2) a cascade architecture (Cascade mask-RCNN), and (3) a hybrid task (HTC). It is worth mentioning that DetectoRS is currently the best-performing CNN-based model for instance segmentation. 
Moreover, we can see comparative results of the metrics before and after training, and for all architectures, there is a significant increase in the mAP which varies from 0.5% to 5%.   


Furthermore, for some classes, we did not observe a significant increase in the metrics. A possible explanation would be the number of samples available. One example is the class 'sheep' which contains a small number of samples (765 samples for training). Additionally, very few entries show a slight decrease after applying our method.


This experiment supports the fact that MaskUno can be applied to a wide set of methods for instance segmentation. 
We extend this experiment by training 80 classes for both the baseline Mask-RCNN and DetectoRS. For both models, we observe an average increase in the mAP (Fig.<ref>).  This enhancement infers a different conclusion for every model. For Mask-RCNN, the mAP increases by 4.8%, which confirms that MaskUno is not affected by a specific choice of class. As for DetectoRS, the mAP increases by 2.03%, which validates that solving the competing kernels is complementary to a cascade architecture or a backbone enhancement. 
 
Figure <ref> shows a few examples at inference time, for multiple classes. Besides successful detection and segmentation, one can note that the degree of confidence for every instance is significantly high.







 










§ LIMITATIONS AND FUTURE WORK



  §.§.§ Limitations

Our experiments demonstrate that MaskUno outperforms previous models when evaluated on a per-class basis, with each class evaluated exclusively on its positive images. However, it's important to note that this experimental setup does not account for negative frames where false positives may cause the switch to misdirect certain images to incorrect branches.


  §.§.§ Further experiments


Another experiment would be to investigate the effect of splitting the bounding box regression block on the mAP, compare it to classical bounding box refinement, and merge both together. A new version of MaskUno where splitting is done on the branch responsible for bounding box prediction, followed by the mask split. This would not only enhance bounding box refinements but also the segmentation branch benefiting from this refinement. 



  §.§.§ Transformer-based models


Instance segmentation models based on transformers proved to surpass CNN-based models. They are usually used as a backbone for various computer vision downstream tasks. A similar approach could be implemented, where the bounding box regression and segmentation branches of such models would be split into multiple parts to specialize the training per-class. 



§ CONCLUSION


To address the issue of conflicting gradients, we offer MaskUno, a Switch-Split block that can be used for most instance segmentation algorithms. It enhances the accuracy of instance segmentation models based on Mask-RCNN that use multi-class segmentation methods and has a complementary effect to tasks such as cascade or hybrid task. MaskUno provided a 2.03% improvement over one of the best-performing CNN-based instance segmentation models "Detectors" on the standard COCO dataset. Moreover, for future work, applying MaskUno on transformer-based models, while benefiting from a split of both bounding box regression and mask prediction, has the potential to reach a new state-of-the-art accuracy.



§ ACKNOWLEDGMENT

This work was supported by the Didymos Horizon Europe project, grant number 101092875–DIDYMOS-XR.

ieeetr



 Paper Title: Dynamic Object Queries for Transformer-based Incremental Object Detection.




Kramers-Kronig relations via Laplace formalism and L^1 integrability
    Leonardo Ricci
    August 5, 2024
====================================================================

 
[1]These authors contributed equally to this work. 
[2]Corresponding authors. 


§ ABSTRACT 
Incremental object detection (IOD) aims to sequentially learn new classes, while maintaining the capability to locate and identify old ones. As the training data arrives with annotations only with new classes, IOD suffers from catastrophic forgetting. Prior methodologies mainly tackle the forgetting issue through knowledge distillation and exemplar replay, ignoring the conflict between limited model capacity and increasing knowledge. In this paper, we explore dynamic object queries for incremental object detection built on Transformer architecture. We propose the Dynamic object Query-based DEtection TRansformer (DyQ-DETR), which incrementally expands the model representation ability to achieve stability-plasticity tradeoff. First, a new set of learnable object queries are fed into the decoder to represent new classes. These new object queries are aggregated with those from previous phases to adapt both old and new knowledge well. Second, we propose the isolated bipartite matching for object queries in different phases, based on disentangled self-attention. The interaction among the object queries at different phases is eliminated to reduce inter-class confusion. Thanks to the separate supervision and computation over object queries, we further present the risk-balanced partial calibration for effective exemplar replay. Extensive experiments demonstrate that DyQ-DETR significantly surpasses the state-of-the-art methods, with limited parameter overhead. Code will be made publicly available.





§ INTRODUCTION





Humans inherently possess the ability to incrementally learn novel concepts without forgetting previous ones, capable of acquiring and accumulating knowledge from past experiences. Traditional object detection models <cit.> rely on supervised learning with fixed data, where all classes are predefined and known beforehand. However, real-world data continuously evolve over time, leading to non-stationary distributions. Due to the stability-elasticity dilemma, fine-tuning models directly on new class data leads to catastrophic forgetting <cit.>, whereas joint training is expensive in both computation and storage. Therefore, incremental object detection (IOD) has attracted increasing attention in both research and practical applications. 


Recent advances for IOD adopt knowledge distillation and exemplar replay to address forgetting. Knowledge distillation-based methods <cit.> typically involve distillation on the non-background predictions of the old model to circumvent the imbalance issue caused by an excessive number of background predictions. Exemplar replay-based methods <cit.>, on the other hand, operate by preserving a small subset of samples from past training data (exemplars), replaying them in subsequent phases to mitigate the forgetting of old data. Despite the progress, there still exist limitations. First, due to the fixed model capacity, severe conflicts between preserving knowledge of old classes and incrementally learning new ones exist. This not only impedes the accommodation of new class knowledge but also leads to the overwriting and forgetting of old class knowledge. Second, exemplars with incomplete labels are inadequately equipped for replay. Even if pseudo labeling is an intuitive way, the low-quality supervision inherited from old knowledge will hamper the adaption.


To address the above concerns, we propose Dynamic object Query Assembly based DEtection TRansformer, abbreviated as DyQ-DETR, for incremental object detection. Inspired by dynamic networks for incremental classification <cit.> , we motivate to investigate the dynamic architecture for expanding model capacity in IOD. Specifically, the object queries serve as the class-specific representative essence for memorization. The object queries from old classes can be assembled with learnable new queries for dynamically expandable representations with limited memory overhead and acceptable time overhead. As demonstrated in Fig.<ref>, we first obtain the visual features of the whole image with a CNN attached Transformer encoder. Then the set of object queries from previous steps are assembled with learnable ones corresponding to new classes in each incremental step. Moreover, we disentangle the self-attention and isolate the bipartite matching to remove the interaction between object queries from different steps. Besides, we propose the risk-balanced partial calibration to tackle incomplete annotations for effective exemplar replay.


Generally, we circumvent the catastrophic forgetting in IOD from the perspective of dynamic networks, with the inspiration of incrementally adding class tokens has been proven effective in incremental learning <cit.>. Notably, for object detection, multiple objects of a single class may co-exist in one image (sometimes even many). We adopt a many-to-many rather than one-to-one matching of dynamic queries, for which sparse object queries implicitly associate with the content and reference positional information of one or multiple seen old classes. We propose to memorize the set of object queries for sequentially arrived classes of data. An isolated group of queries is responsible for detecting the objects from a set of classes arriving at the same time step. By incrementally aggregating the category-wise object queries from previous steps and learning new class embedding, we decouple the representations for old and new class knowledge with lightweight query embeddings, simultaneously maintaining the stability and plasticity. As for exemplar replay, we propose to reserve images with moderate matching losses as exemplars, further perform partial calibration on the outputs of corresponding queries with only the annotated classes. Such risk-balanced partial calibration avoids over focusing on classes from any particular stage and eliminates the reliance on low-quality pseudo labels. Through dynamic object queries and risk-balanced partial calibration, our proposed DyQ-DETR significantly alleviates the forgetting in IOD.  
The main contributions are three-fold: 





  * We propose a novel approach to integrate dynamic object queries into Detection Transformer for incremental learning. By dynamically incorporating object queries into DETR, our approach provides a simple yet effective way to expand the model capability compatible with new knowledge adaption and old knowledge preserving. 

  * We propose the disentangled self-attention for dynamic object queries. For dynamic knowledge expansion, the isolated bipartite matching over the object queries from different phases further decouples the representation learning of old classes and new ones.

  * A risk-balanced selection mechanism is proposed to explore informative and reliable exemplars. The partial calibration is further associated to tackle the incomplete annotations for incremental detection with exemplar replay.


Extensive experiments on public benchmarks demonstrate the superiority of our proposed DyQ-DETR, outperforming the state-of-the-art methods by a large margin. It achieves the average 4.3% AP improvement in non-exemplar scenarios and 2.9% AP improvement with exemplar replay.




§ RELATED WORK



Incremental Learning (IL). 

Prevailing IL methods can be broadly divided as regularization-based, distillation-based and structure-based ones. Regularization-based methods<cit.> estimate parameter importance and penalize updating of crucial parameters to maintain previous knowledge. Distillation-based methods build the mapping between the old and new model by matching logits<cit.>, feature maps<cit.>, or other information<cit.>, which leverage the knowledge transfer to prevent forgetting. Structure-based methods<cit.> dynamically expand the representative network, e.g., backbone, prompt, to fit the evolving data stream. 



Incremental Object Detection (IOD).  As the typical extension of incremental learning, IOD involves multiple objects belonging to the old and new classes appearing simultaneously. This co-occurrence makes knowledge distillation an inherently effective strategy for IOD, since it allows for the utilization of old class objects from new training samples to minimize the discrepancies in responses between the previous and current updating model. As a pioneering work, ILOD<cit.> distills the responses for old classes to counteract catastrophic forgetting on Fast R-CNN<cit.>. The idea of knowledge distillation is then extended to other detection frameworks, such as CenterNet<cit.> (SID<cit.>), RetinaNet<cit.> (RILOD<cit.>), GFLV1<cit.> (ERD<cit.>), Faster R-CNN<cit.> (CIFRCN<cit.>, Faster ILOD<cit.>, DMC<cit.>, BNC<cit.>, IOD-ML<cit.>) and Deformable DETR<cit.> (CL-DETR<cit.>). Built on Deformable DETR instead of conventional detectors such as Faster R-CNN, DyQ-DETR can efficiently expand queries rather than inefficiently enlarging the backbone or specific convolutional layers. Note that DyQ-DETR also uses knowledge distillation techniques. 
As for exemplar replay,<cit.> proposes maintaining an exemplar set and fine-tuning the model on the exemplars after each incremental step.<cit.> proposes an adaptive sampling strategy to achieve more efficient exemplar selection and <cit.> proposes distribution-preserving calibration, which selects exemplars to match the training distribution. They usually finetune directly using the exemplar set with incomplete annotations and overlook the amount of information and reliability of the annotated objects. 




Transformer-based Object Detection.
The infusive work DETR (DEtection TRansfomer)<cit.> formulates object detection as a set prediction problem, with an elegant transformer-based architecture. It captures global context and reasons object relations with attention mechanism. With a small set of learnable object queries and Hungarian bipartite matching<cit.>, it eliminates the need for the complex non-maximum suppression and many other hand-designed components in object detection while demonstrating good performance. Deformable DETR<cit.> introduces sparse attention on multi-level feature maps, thereby accelerating the convergence of DETR and improving the performance, particularly for small objects. There also exists many other DETR variants<cit.> designed to accelerate convergence speed and enhance detection performance. Without loss of generality, we build our method on the commonly used Deformable DETR.




§ METHODOLOGY



Preliminaries
In the paradigm of IOD, object detection is performed in multiple steps from sequentially arrived training data to recognize and localize objects of all seen classes in test images. Let D be a dataset with samples (x,y), where x is the image, y is the set of object class labels and the associated bounding boxes. Suppose there are T steps. At time step t, the incoming dataset is denoted as D_t=(x_t, y_t) and the objects belong to seen classes in C_t. Specially, the images in D_t are only annotated for objects in C_t. Moreover, the class sets at different time steps are mutually exclusive from each other, i.e., C_t∩ C_τ=∅, t≠τ. Since D_t only contains the annotations with classes in C_t, each class can be learned only once in a specific step. Due to the missing annotations of old classes {C_1, C_2, ⋯, C_t-1} in current data D_t, detection models are prone to forget the knowledge of old classes and bias towards new ones. To overcome  the catastrophic forgetting, we introduce the dynamic object queries into Detection Transformer and propose DyQ-DETR for IOD.


We construct the DyQ-DETR based on Detection Transformer for incremental object detection. In particular, we elaborate the details of dynamic object queries with the structure design and training strategy in Sec. <ref>. The risk-balanced exemplar selection and partial calibration are in Sec. <ref>.







 §.§ Transformer-based Incremental Object Detection



We build up the DyQ-DETR based on the DETR architecture <cit.>. Other than the backbone, a DETR consists of an encoder, a Transformer decoder, and the predictor to generate object classes and locations. The encoder takes the images as inputs and outputs visual features. Then those visual features and learnable object queries are fed into Transformer decoder for prediction. Notably, we propose to aggregate the object queries from up-to-far learning steps to resist the forgetting of old class knowledge. Due to the absence of annotations from previous seen classes in incremental step t, knowledge distillation is applied for preserving the class-specific old knowledge. As in Fig.<ref>, we select the non-background predictions of old classes by thresholding. Pseudo labels ỹ_τ (1 ≤τ<t) are generated by the last previous model. Instead of mixing the pseudo labels ỹ_τ (1 ≤τ<t) with the ground-truth labels y_t, we use ỹ_τ and y_t separately to supervise the model training. The pseudo labels for old classes and the real ground-truths are used to guide the learning of object queries being Q_τ and Q_t, respectively. Besides, object queries from different groups share the weight parameters of the Transformer decoder. To allow for computational complexity to grow linearly rather than quadratically, the self-attention over different groups of object queries are eliminated.


As for IOD with exemplar replay, where a small number of exemplar images ϵ_t from dataset D_t in different time steps are stored, we introduce a risk-balanced selection mechanism. At time step t, we use the trained model Φ_t to score the annotated objects from images in D_t. The computed loss from the partial bipartite matching is considered as the risk score for exemplar selection. We choose the sample images with moderate risk score for the tradeoff between the annotation importance and quality. Specifically, to build the exemplar dataset ϵ_1:t, we select the samples falling into the middle part after sorting, because they are informative and reliably annotated. Considering the image in ϵ_1:t is incompletely annotated for specific classes, we adapt the partial calibration. We leverage the incomplete annotations to calibrate the output for corresponding object queries in each group. Since ϵ_1:t is balanced, the partial calibration will prohibit the model being biased towards certain classes.
 




 §.§ Dynamic Object Query Assembly




Existing DETR models employ a fixed set of object queries (i.e., learnable embedding) as the inputs of Transformer decoder. These object queries are progressively optimized to map into object instances in images. Despite various designs, the learnable object queries are highly relevant with the specific classes. For IOD, the object queries are expected to be associated with objects belonging to sequentially arrived classes. Since new classes are considered as backgrounds in previous time steps, the preservation of old knowledge naturally contradicts the knowledge updating from new data learning, especially from the perspective of object queries. Moreover, the conflict between a fixed network and the continually emerged class-specific information severely undermines the performance of incremental learning, especially in non-exemplar scenarios. 


To cope with incremental classes without extra modules in network architecture, we focus on tackling the forgetting issue with dynamic object queries. At time step t, for a set of new classes C_t, a new set of learnable object queries Q_t is added. The newly added object queries Q_t are aggregated with previous sets of queries Q_τ, 1 ≤τ<t. The assembly of object queries {Q_1, Q_2, ⋯, Q_t} serves as the input of Transformer encoder in step t. The visual embeddings e_t corresponding to the newly expanded queries Q_t are used to predict the objects of new classes C_t, and the old classes C_τ (1 ≤τ<t) is detected with the embeddings e_τ of old queries Q_τ. By dynamically expanding object queries, the new and old classes are segregated by class embeddings, significantly alleviating the conflict between the old knowledge and continuously evolving new knowledge.


Based on the dynamic assembly of object queries, we further investigate the decoder design to restrict the computational burden. In standard DETR, object queries interact visual features with cross-attention for refinement. Besides, those object queries interact with each other by self-attention. 
Through self-attention, duplicated detections can be removed, but the computational complexity increases quadratically with the number of object queries. Considering that the object instances from different class sets rarely overlap, we disentangle the self-attention in Transformer decoder. The self-attention is computed among separate set of object queries, as:

    a_i,j=
        
        Softmax(Q_m,i· Q_n,j^T/√(d))   m=n
    
        0   m≠ n,

where a_i,j denotes the attention weight between two queries Q_m,i and Q_n,j from the query set Q_m and Q_n, respectively. By eliminating the attention interaction between queries of different groups as shown in Eq.(<ref>), we achieve a linear growth in computational complexity almost for free. Upon the addition of new queries, the capability of old queries to detect old classes is fully preserved. We perform the decoder forward passes as many as the time steps, obtaining the embedding vectors from different sets of queries as: 

    e_τ=decoder(f_t, Q_τ), 1 ≤τ≤ t,

where f_t denotes the visual feature extracted from image x by the CNN and Transformer encoder. Each decoder forward pass is executed with a different set of queries Q_τ, resulting in different task-specific embeddings e_τ to obtain detection predictions of the corresponding classes C_τ.


We further adapt the knowledge distillation for incremental detector training. The foreground predictions with the pseudo labels generated from old model are kept for distillation and used for supervision. As in <cit.>, the foreground predictions with high confidence from the old model are selected. A probability threshold θ_p (typically 0.4) is set over the prediction scores. An additional IoU threshold θ_IoU (typically 0.7) is used to restrict the predictions not too close to the ground-truth bounding boxes of new classes. It helps filter out incorrect predictions about new class objects that are misclassified as old classes. The highly-confident predictions after filtering are used as pseudo labels ỹ_τ (1 ≤τ < t), which contain two parts of annotations (i.e., the predicted object labels and bounding boxes). Notably, 
instead of merging the pseudo labels with real grounding truths, we compute the separate bipartite matching losses with different set of object queries in an independent way. The loss for retaining old class knowledge ℒ_τ^DETR (1 ≤τ < t), as well as the loss for learning new class knowledge ℒ_t^DETR can be formulated as in Eq.(<ref>):

    ℒ_τ^DETR = ℒ^DETR(e_τ,ỹ_τ), 1 ≤τ < t ; 
      ℒ_t^DETR = ℒ^DETR(e_t, y_t),

Note that the specific embeddings e_τ, 1 ≤τ≤ t are only supervised with the corresponding pseudo labels or real annotations, leading to decoupled representations. The total loss is the weighted sum as:

    ℒ_total=∑_τ=1^tw_τℒ_τ^DETR.

To tackle the varying number of classes in each step and prevent the model from being biased towards class sets with fewer classes, we set the weight w_τ of the class set C_τ to |C_τ|/|C_1:t|.


r0.5
  

  
    < g r a p h i c s >

  Illustration of risk-balanced exemplar selection. We choose the middle part with moderate risk score to serve as the exemplars.
  

  




 §.§ Risk-balanced Partial Calibration



For exemplar replay based IOD paradigm, an exemplar memory is formed to store a small number of samples for subsequent incremental learning. At time step t, the exemplar set ϵ_t is a subset of the entire dataset D_t. Let ϵ_1:t=ϵ_1 ∪⋯ϵ_t. The exemplars in ϵ_t are used to represent the samples with objects in C_t. We intend to choose the class annotated images which are substantial for detector training. An intuitive way is to directly compute the loss between the model output and the real label y_t for sample selection. However, since the image x ∈ D_t only contains annotations for the classes C_t of interest, this will result in the loss being dominated by the absence of old classes, posing challenges to foreground-background balancing. 

Benefiting from the internal decoupling of dynamic object queries, we are able to exclusively detect new classes C_t using the corresponding queries Q_t. Considering that the image is only annotated for the specific classes, the partial loss is more reliable. For computing the partial loss, the old classes are considered as backgrounds by Q_t, which is compatible with the incomplete real labels y_t. After each incremental step of training, the partial loss from Eq.(<ref>) is considered as the risk score to guide the subsequent selection of exemplars.

    Υ←ℒ_partial=ℒ_t^DETR(e_t, y_t).









The risk score can measure the quality of incomplete class labels and bounding boxes. Additionally, it also takes the number of annotated objects like in <cit.> into account. As in Fig. <ref>, images with low risk, which constitute a high proportion, provide limited information for optimization, while those with high risk are likely to be outliers with incorrect annotations. Based on the risk estimation, we construct the exemplar set ϵ_t by sorting and selecting the middle part of risk-balanced samples in D_t. The exemplar set ϵ_t is merged with ϵ_t-1 to form a set ϵ_1:t for partial calibration of the model.


In incremental step t, the exemplar sets ϵ_1:t are used to finetune the model after training with D_t. Previous IOD methods typically finetune the model directly with dataset ϵ_1:t, without addressing the issue of missing labels. That is, an image x in the balanced exemplar set ϵ_1:t is only annotated for a specific class subset from {C_1, ...,C_t-1, C_t}, with the absence of annotations for other classes. The confusing, even contradictive supervision hinders the process of prediction calibration. An intuitive way is to use pseudo labels, yet the quality of pseudo labels is hard to guarantee. Thanks to the dynamic object queries and associative disentangled computation, we propose to perform partial calibration that relies only on the incomplete real labels. Specifically, we calculate the partial loss between the outputs of the corresponding queries and the ground-truth annotations in ϵ_1:t. This type of partial calibration further mitigates the forgetting.




§ EXPERIMENT







 §.§ Experimental Setup


Dataset and evaluation metrics. Following<cit.>, we conduct experiments on the widely-used COCO 2017 dataset<cit.>, which consists of images from 80 object categories in natural scenes. The standard COCO metrics as AP, AP_50, AP_75, AP_S, AP_M, AP_L are used for performance evaluation.



Protocols. We evaluate DyQ-DETR with two protocols: 1) traditional protocol <cit.> (Tab.<ref>-left) and 2) revised protocol proposed by<cit.> (Tab.<ref>-right). Protocol 2) avoids observing the same images at different stages. Therefore, we adopt protocol 2) for all subsequent experiments, and the details of protocol 1) can be found in the appendix. For protocol 2), we adopt both two-phase and multiple-phase settings, which can be formulated in the form of c_1+c_2+...+c_T, where c_t represents the number of new classes in incremental step t (c_t=|C_t|) and the sum of c_t is denoted as c (c=|C_1:T|). At time step t, we observe a fraction c_t/c of the training samples with c_t new categories annotated. We test settings c_1+c_2+...+c_T = 40+40, 70+10, 40+20 × 2, and 40+10 × 4. Following <cit.>, we also set the total memory budget for the exemplars to be 10% of the total dataset size.


























































































Implementation details. Following <cit.>, we build DyQ-DETR on top of Deformable DETR <cit.> without iterative bounding box refinement and the two-stage variant. The backbone is ResNet-50<cit.> pretrained on ImageNet<cit.> and the training configurations for the initial stage are consistent with<cit.> to maintain uniform performance in the initial phase. We denote the initial number of queries for the detector as N (typically 300). For both two-phase and multi-phase settings, we dynamically expand by N queries at each incremental step t and the initial parameters of Q_t are inherited from Q_t-1. At step t, old queries Q_1:t-1 are frozen during the incremental training and unfrozen in subsequent exemplar replay. 
We train the model for 50 epochs, and for an additional 20 epochs during fine-tuning. All the experiments are performed on 8 NVIDIA GeForce RTX 3090, with a batch size of 8.




 §.§ Quantitative Results


Two-phase setting. We compare our DyQ-DETR with LwF<cit.>, iCaRL<cit.>, RILOD<cit.>, SID<cit.>, ERD<cit.>, and the previous SOTA method CL-DETR<cit.>. For each setting, we provide the performance of different methods with/without Exemplar Replay (ER). The metrics by joint training are also presented as the upper bound for reference. Tab.<ref> shows that, in the two-phase settings, our proposed DyQ-DETR consistently outperforms the aforementioned methods under different protocols with significant margins. For protocol 2), with exemplar replay, the DyQ-DETR achieves the AP of 39.7% and 41.9% under 40+40 and 70+10 settings. It surpasses CL-DETR by 2.2% AP and 1.8% AP under the 40+40 and 70+10 settings, respectively. Compared with the upper bound, the DyQ-DETR obtains an average performance gap of 1.4%, which is much smaller than the 3.4% gap of CL-DETR. To evaluate the model's ability to preserve old knowledge and learn new knowledge, we conduct a comparison with CL-DETR in the 40+40 setting, where both capabilities are equally important. CL-DETR achieves an old class AP of 39.7% and a new class AP of 36.3%, while our method achieves much better results with 41.3% AP for old classes and 38.6% AP for new classes, respectively. This validates that our proposed method achieves better stability and plasticity, thus addressing the catastrophic forgetting effectively.



Additionally, Tab.<ref> include a performance comparison without exemplar replay (w/o ER) to show the effectiveness of dynamic object queries. In non-exemplar scenarios, DyQ-DETR demonstrates a more significant advantage, outperforming CL-DETR by 2.9% AP and 5.6% AP in the 40+40 and 70+10 settings, respectively. It is noteworthy that the performance of our method without ER is comparable or even exceeds that of the existing method with ER. For example, the performance of our DyQ-DETR w/o ER is 1.6% AP higher than that of CL-DETR w/ ER in the 40+40 setting.

Multiple-phase setting. We conduct experiments in the more challenging 40+20×2 and 40+10×4 settings. The changing AP and AP_50 along with time steps are presented in Fig.<ref>. Our DyQ-DETR consistently outperforms other IOD methods. Moreover, in both settings, the AP improvements of DyQ-DETR become more pronounced with the increase of incremental steps.


r0.5
  

  
    < g r a p h i c s >

  
  Comparison of parameter (left) and complexity overhead (right) with the addition of 100 queries at each step. DSA denotes disentangled self-attention.
  

  


Scalability. As shown in Fig.<ref>-left, by expanding queries rather than the network structure, the additional parameter overhead of our method is almost negligible. As illustrated in Fig.<ref>-right, the computational overhead of our method grows linearly because of the removed inter-group query interaction. Since the computational load of the decoder (excluding the portion shared by different query groups) constitutes a small portion (∼6%) of the entire model's computation, our computational complexity increases at a slow linear rate. Specifically, with 20 stages and an increment of 100 queries per stage (note that the standard Deformable DETR has 300 queries), DyQ-DETR only increases the parameters and GFLOPs by 2% and 39% respectively compared to the standard Deformable DETR, confirming its scalability.







 §.§ Ablation Study


Effect of dynamic object queries. Tab.3 illustrates the ablation study over dynamic object queries in the 70+10 setting. Compared to the baseline, the naive way of expanding queries (+Nat Query) increases the AP to 34.5% and AP_50 to 52.7%, by 1.1% and 4.2%, respectively. This can be attributed to model capacity. By equipping the dynamic object queries (+Dy Query) with isolated matching on independent query set, we further obtain 3.6% AP and 3.5% AP_50 improvements. It validates that dynamic object queries is effective to retain old knowledge during the learning of new knowledge. Furthermore, freezing the task-specific old queries (+DyFro Query) during the incremental training leads to slightly more improvement of 0.4% AP.
  

Components for exemplar replay. We introduce the risk-balanced exemplar selection (RS) and partial calibration (PC) for exemplar replay. The ablative results of the two components in the 70+10 setting are provided in Tab.<ref>. We compare the RS and PC with the distribution-preserving selection (DS) and direct calibration (DC) proposed in CL-DETR <cit.>, respectively. 
By separately applying RS and PC, we increase the AP by 0.5% and 0.9%, respectively. Applying the whole risk-balanced exemplar selection and partial calibration obtains 41.9% AP, exceeding the baseline by 1.5%.








Effect of disentangled self-attention. The comparison in Tab.<ref> under both the 40+40 and 70+10 settings shows that removing the self-attention interaction between different query groups has almost no impact on performance. This is reasonable because the class sets detected by different query groups do not overlap, thus eliminating the need for self-attention interaction to remove duplicate predictions. Combined with Fig.<ref>-right, by disentangling the self-attention computation, the computational complexity can be reduced from quadratic to linear growth without performance drop.

Effect on the number of expanded object queries. Tab.<ref> presents the APs with changing number of object queries. It indicates that the expanded queries number has a trivial impact. By expanding fewer queries, we can decrease complexity with negligible harm to performance.


Visualization of decoupled queries. In Fig.<ref>, we visualize the decoupling behavior of  dynamic queries on the test set, based on the 40+20×2 setting. The classes "bicycle", "person", and "car" appear in phases 1, 2, and 3, respectively. They are detected by queries in Q_1, Q_2 and Q_3 accordingly, in a decoupling manner. It can be observed that Q_1 detects the "bicycle" class accurately, while Q_2 and Q_3 consider it as background. Once a query group Q_t learns to detect the class set C_t at step t, its class-specific knowledge remains unchanged thereafter, requiring only the maintenance of old knowledge, which significantly improves the performance of IOD.







§ CONCLUSION



In this paper, we propose DyQ-DETR for incremental object detection. Distinct from the mainstream methods focus on the distillation mechanism, we address the catastrophic forgetting by taking inspiration from dynamic networks for model capability expansion. In particular, we propose the dynamic object queries with incremental assembly of new object queries, disentangled self-attention computation and isolated bipartite matching over object queries from different time. The DyQ-DETR can alleviate the conflict between outdated background knowledge and continually emerged classes, thereby achieving stability-plasticity tradeoff. Benefiting from the isolated supervision of dynamic object queries, we further propose risk-balanced partial calibration for effective exemplar replay, with the idea to select exemplars based on risk and partially finetunes the model without relying on low-quality pseudo labels. Extensive experiments demonstrate that our proposed DyQ-DETR surpasses existing IOD methods by a large margin, with quite limited memory overhead. Besides of dynamic query, we hope that more various ways of model expansion can be explored in IOD.












§ APPENDIX



In Sec.<ref>, we present the algorithmic pipeline of DyQ-DETR. In Sec.<ref>, we introduce a simple but strong baseline with only exemplar replay to emphasize the importance and rationale of the non-exemplar setting. In Sec.<ref>, we provide more detailed experimental results. This includes details of the traditional protocol (Sec.<ref>), more fine-grained results about old classes and new classes (Sect.<ref>), and more ablation results (Sec. <ref>). In Sec.<ref>, we show more visualization results, including images with different levels of risk, behavior of decoupled queries, and performance comparison with CL-DETR. Please note that all experiments, except for those in Sec.<ref>, are based on the revised protocol, using Deformable DETR on COCO 2017, and we will not specify this further.



§ ALGORITHMIC PIPELINE OF DYQ-DETR












To get a more comprehensive understanding of the overall framework, we describe the training pipeline of DyQ-DETR in Algorithm <ref>. In the τ-th phase, we dynamically add a new set of queries, which are responsible for detecting and recognizing new classes. We disentangle self-attention modules in the decoder, ensuring that queries between different groups remain relatively independent. In the incremental training, we use a score threshold to filter out background predictions, resulting in the pseudo-labels y^pseudo. Instead of simply merging pseudo-labels y^pseudo and incomplete real labels y, we divide the completed annotations of an image according to the incremental phase/class set. For an image, annotations corresponding to different class sets act on the outputs of respective queries, thus resulting in the decoupled loss ℒ_total^DETR for training.

After the incremental training, we utilize the trained model to calculate the partial loss as the risk score for images in D_τ. We sort D_τ based on the risk score and simply select the middle 10% of these samples to serve as exemplars. Then, for an image in the exemplar set ϵ_1:τ, we identify the specific 
stage corresponding to its annotation, and we use the respective output and real labels to compute the partial loss ℒ_partial for calibration.




§ BASELINE WITH ONLY EXEMPLAR REPLAY



We introduce a simple baseline, which only combats forgetting by saving exemplars of the same size. As shown in Tab.<ref>, it completely forgets past knowledge after the incremental training, but its performance significantly improves after the simple exemplar replay. The total memory budget for exemplars is typically set as 10% of the total dataset size. This relatively large proportion may obscure issues in the incremental training and potentially diminish the advantages of superior methods. Therefore, we believe that comparisons in non-exemplar settings are more suitable for evaluating the performance of different methods. 




§ MORE DETAILED EXPERIMENTAL RESULTS






 §.§ Traditional protocol and results

















































The main difference between the traditional protocol<cit.> and the revised protocol<cit.> lies in data partitioning. In the traditional protocol, the model can observe all images containing at least one object of the currently interested classes, which may result in an image appearing across multiple phases. Formally, let D={(x,y)} denotes a dataset with images x and corresponding object annotations y. First, we divide the total class set into non-overlapping parts {C_1, C_2,..., C_T}, one for each incremental phase. For each phase τ, all samples in D retain annotations of C_τ and drop others. The incremental training dataset in phase τ consists of images that contain at least one annotation for C_τ. Additionally, the revised protocol shuffles the class order, while the traditional one does not, leading to different class partitions.

Tab.<ref> shows that our proposed DyQ-DETR consistently outperforms the state-of-the-art (SOTA) method in the traditional IOD protocol, especially in the non-exemplar settings. Specifically, without exemplars, our DyQ-DETR surpasses the SOTA by 2.2% AP and 3.7% AP in the 40+40 and 70+10 settings, respectively. With exemplars, the advantage decreases to 0.4% AP and 2.0% AP, respectively. The substantial advantage of our DyQ-DETR in non-exemplar settings demonstrates the effectiveness of dynamic query, which is the core of our method. 







 §.§ Results about old classes and new classes


Tab.<ref> presents fine-grained results in the 40+40 setting about old classes and new classes, which respectively represent the model's stability and plasticity. The results show that, compared to the SOTA, our DyQ-DETR achieves improvements of 1.6% AP for old classes and 2.3% AP for new classes, indicating enhanced stability and plasticity in our method. 



 §.§ More ablation results



Effect of the score threshold. In the 40+40 setting, we set the score threshold to 0.6, while for other settings, it is set to 0.4. This is because, in other settings, there is a higher proportion of old classes where stability is preferred, and more old class pseudo-labels help combat forgetting. The ablation study on the score threshold is shown in Fig.<ref>(a), where the optimal threshold for the 40+40 setting is 0.6. We also find that when the score threshold is low, the pseudo-label noise is significant, leading to poor performance in non-exemplar settings. However, after exemplar replay, the performance greatly improves, further illustrating that exemplar replay may mask issues in the incremental training.



Effect of the selected risk interval. After sorting the images by risk in ascending order, the results of changing the selection interval are illustrated in Fig.<ref>(b). It can be observed that selecting images with moderate risk is optimal. This selection strategy is straightforward, and our contribution lies more in demonstrating the feasibility of using the model to optimize the selection.



§ MORE VISUALIZATION RESULTS




Images with different levels of risk. As illustrated in Fig.<ref>, images with low risk are considered as simple samples containing only a few annotated objects, while images with high risk are characterized by a dense distribution of objects, accompanied by severe occlusion or inaccurate annotations. Samples in the middle part are considered informative and reliable, 
and such an active selection strategy promotes more effective exemplar replay.









Behavior of decoupled queries. As depicted in Fig.<ref>, our designed dynamic query set achieves the goal of only detecting the corresponding class set. The old query set is responsible for memorizing knowledge of old classes, while the new query set focuses on learning new knowledge. This decoupling significantly improves the performance in non-exemplar settings.

Performance comparison with CL-DETR.
Fig.<ref> shows the visualized detection results of CL-DETR and our DyQ-DETR in non-exemplar and exemplar-based settings. In both settings, DyQ-DETR presents more robust performance for remaining old knowledge and learning new knowledge, thus greatly mitigating catastrophic forgetting.





§ DISCUSSION OVER OPEN VOCABULARY OBJECT DETECTION



Open vocabulary object detection (OVD) leverages visually related language data as auxiliary supervision to bridge the gap between base categories and novel label spaces<cit.>. By using image-caption pairs, it acquires an extensive vocabulary of concepts, allowing the model to learn object detection with annotations only for certain base categories. Consequently, the model can detect both base classes and novel classes that were not present in the training annotations. OVR-CNN<cit.> is the pioneering work that introduced the concept of open vocabulary object detection and established a comprehensive framework to address this problem. Subsequently, with the introduction of large-scale visual language pretraining models such as CLIP<cit.>, novel approaches have emerged that utilize these pretrained models to enhance the capability of open vocabulary object detection. Some works<cit.> employ knowledge distillation, using large model text encoders to align visual features with the large model and transfer new class information. Other works address the gap in object detection region localization by converting image-text level pre-training to region-text level<cit.>, while some complete this during the detector training stage after pre-training<cit.>.

The difference between IOD (Incremental Object Detection) and OVD (Open Vocabulary Detection) lies in their focus. IOD emphasizes the continuous learning and updating process of the model, while OVD focuses on the model's generalization ability. IOD aims to improve the model through multiple learning steps, whereas OVD aims for the model to generalize perfectly to other problems with just training once. Specifically, after training, OVD can detect some objects in a new category space, even if these objects have not been seen in the annotations of base categories, due to the rich semantic concepts obtained during the image-caption alignment process. However, despite the richness of the learned semantic concepts, OVD is difficult to handle newly generated concepts due to the dynamic nature of the environment and requires re-alignment of image-caption pairs to learn these new concepts. In contrast, IOD simplifies this process by allowing incremental learning of new concept based on newly collected and annotated data, thus avoiding the need for retraining the entire model.

IOD and OVD are not mutually exclusive but complementary. They are both important for open-world practical applications. As mentioned above, OVD still shows deficiency in handling the emergence of new concepts without retraining, whereas IOD simplifies this process, saving time and computational resources. However, IOD struggles to resist catastrophic forgetting of old class semantics during continuous model updates. OVD relies on pre-learned rich semantic concepts for novel classes, without consideration of forgetting issue.




§ LIMITATIONS AND BROADER IMPACTS


We propose a dynamic object queries-based detection transformer (DyQ-DETR) to address catastrophic forgetting in incremental object detection. While it scales well to tasks with an incremental step of 20, DyQ-DETR may face challenges in scaling to tasks with longer incremental steps.

High-performance incremental object detection systems have tremendous potential to have a significant impact on various fields and inspire innovative research approaches in robotics, autonomous driving, and beyond. For example, our proposed DyQ-DETR can progressively enhance the recognition and manipulation capabilities of robots, thereby improving efficiency and productivity in industries and healthcare. Furthermore, considering the significant overhead of joint training, DyQ-DETR helps detection models avoid retraining from scratch, providing an effective and efficient approach for incremental updates of large visual models. As for potential widespread impact, our work has not shown any negative social consequences.




















§ NEURIPS PAPER CHECKLIST







  * Claims
    
   Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
    
   Answer:  
    
   Justification: We have clearly claimed the scope, contribution and related experiments in the abstract and introduction sectoin.
    


  * Limitations
    
   Question: Does the paper discuss the limitations of the work performed by the authors?
    
   Answer:  
    
   Justification: We have discussed at the end of the appendix.
    


  * Theory Assumptions and Proofs
    
   Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
    
   Answer:  
    
   Justification: We do not include theoretical results.
   

    
  * Experimental Result Reproducibility
    
   Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
    
   Answer:  
    
   Justification:  We have included details in the beginning of the experiments section and in the appendix.
    



  * Open access to data and code
    
   Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
    
   Answer:  
    
   Justification: We are currently organizing the code and we guarantee that it will be open-sourced.
    



  * Experimental Setting/Details
    
   Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
    
   Answer:  
    
   Justification: We have included details in the beginning of the experiments section and in the appendix.
    


  * Experiment Statistical Significance
    
   Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
    
   Answer:  
    
   Justification: The experimental results are quite stable, so error bars are not necessary.
    


  * Experiments Compute Resources
    
   Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
    
   Answer:  
    
   Justification: We have included details in the beginning of the experiments section and in the appendix.
    
    

  * Code Of Ethics
    
   Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics <https://neurips.cc/public/EthicsGuidelines>?
    
   Answer:  
    
   Justification: We have read and conformed the the NeurIPS Code of Ethics.
   



  * Broader Impacts
    
   Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
    
   Answer:  
    
   Justification: We have discussed at the end of the appendix.
    
    

  * Safeguards
    
   Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
    
   Answer:  
    
   Justification: We will include this in the open-sourced code in the future.
    


  * Licenses for existing assets
    
   Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
    
   Answer:  
    
   Justification: We have cited all the original papers.
   


  * New Assets
    
   Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
    
   Answer:  
    
   Justification: We do not release new assets.
    


  * Crowdsourcing and Research with Human Subjects
    
   Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? 
    
   Answer:  
    
   Justification: We do not involve crowdsourcing nor research with human subjects.
    


  * Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
    
   Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
    
   Answer:  
    
   Justification: We do not involve crowdsourcing nor research with human subjects.
    






 Paper Title: GEGA: Graph Convolutional Networks and Evidence Retrieval Guided Attention for Enhanced Document-level Relation Extraction.


The impact of quantum noise on limit cycles and entrainment in an atom-cavity system
    Jayson G. Cosme
    August 5, 2024
====================================================================================

§ ABSTRACT 
Document-level relation extraction (DocRE) aims to extract relations between entities from unstructured document text. Compared to sentence-level relation extraction, it requires more complex semantic understanding from a broader text context. Currently, some studies are utilizing logical rules within evidence sentences to enhance the performance of DocRE. However, in the data without provided evidence sentences, researchers often obtain a list of evidence sentences for the entire document through evidence retrieval (ER). Therefore, DocRE suffers from two challenges: firstly, the relevance between evidence and entity pairs is weak; secondly, there is insufficient extraction of complex cross-relations between long-distance multi-entities. To overcome these challenges, we propose GEGA, a novel model for DocRE. The model leverages graph neural networks to construct multiple weight matrices, guiding attention allocation to evidence sentences. It also employs multi-scale representation aggregation to enhance ER. Subsequently, we integrate the most efficient evidence information to implement both fully supervised and weakly supervised training processes for the model. We evaluate the GEGA model on three widely used benchmark datasets: DocRED, Re-DocRED, and Revisit-DocRED. The experimental results indicate that our model has achieved comprehensive improvements compared to the existing SOTA model.














§ INTRODUCTION





Relation extraction (RE) is a crucial technology used to automatically identify and classify semantic relations between entities in natural language texts. Existing relation extraction tasks can be divided into two types: sentence-level relation extraction and document-level relation extraction (DocRE) <cit.>. In sentence-level relation extraction datasets, each data entry contains only one sentence, and there is a single entity pair within the sentence for which the relation needs to be predicted. In contrast, DocRE datasets contain multiple sentences per data entry, corresponding to multiple entity pairs whose relations need to be predicted. As depicted in Figure <ref>, each entity pair may appear multiple times within the document and may have different relation types <cit.>, necessitating the analysis of a larger contextual scope to determine the relation for each entity pair.



Furthermore, assuming there are n entities in a data entry, predicting the relation for an entity pair involves pairing the anchor entity with the remaining n-1 entities one by one. This approach results in significant unnecessary memory overhead, as most pairs of entities do not have any relation between them.


Existing methods can be mainly divided into three categories <cit.> : sequence-based, graph-based, and transformer-based. Sequence-based models commonly employ pre-trained language models to produce word embeddings and character embeddings, transforming sequences of words or characters within texts into vector representations for processing <cit.>. Models based on dependency graphs utilize dependency information to construct document-level graph <cit.>, which are then processed through graph neural networks for inference. Transformer-based models utilize the self-attention mechanism to refine the representation of each word by assessing its contextual relations with all other words in the text <cit.>.



The aforementioned three types of relation extraction methods suffer from two limitations. First, relation extraction between entity pairs generally requires only a set of sentences as supporting evidence, without the need to focus on redundant irrelevant information. However, these methods utilize all the information in the long text for relation extraction. Therefore, <cit.> proposed an evidence retrieval-based relation extraction method. However, this method retrieves a list of evidence information for the entire document, resulting in poor relevance between this information and the relational entity pairs. Second, in DocRE, multiple entities are discretely distributed across different sentences or even paragraphs. Extracting their relations requires fully learning and understanding the semantics of the long text at the document-level. Existing methods rely on dependency parsing to construct multidimensional graph structures for semantic understanding and relation reasoning.However, they perform poorly when faced with complex intersecting relations due to the large number of entity pairs involved.



To address the aforementioned two issues and the insufficient annotation of evidence sentences in the dataset <cit.>, we propose a novel model for DocRE: GEGA. This model is trained under both fully supervised and weakly supervised settings. First, we utilize a complex model (Teacher) trained with full supervision to infer over distant supervision data, extracting evidence sentences and assigning token weights. Then, we use this weight information as supervisory signals to guide the training of a simplified model (Student). Finally, we fine-tune the student model to adapt it to specific tasks and datasets, thereby improving performance. In summary, this article has two contributions:



(1) We propose a novel DocRE model, GEGA[The implementation code for GEGA can be obtained from the GitHub link: [Links covered up in blind reviewing]] (Graph Convolutional Networks and Evidence Retrieval  Guided Attention). This model combines graph structures and Transformers to retrieve evidence sentences highly relevant to the relational entity pairs from the document, guiding the attention to assign higher weights to this evidence information, thereby enhancing the performance of relation extraction.



(2) Experiments conducted on the three public datasets DocRED, Re-DocRED and Revisit-DocRED show that GEGA can achieve the new SOTA[We outperform all other methods using BERT as the pretraining language model on the DocRED leaderboard. Please refer to GEGA's submission: <https://codalab.lisn.upsaclay.fr/competitions/365#results>] results on document-level relation extraction compared to existing methods under the same experimental settings.



§ PRELIMINARY



 §.§ Task formulation for DocRE and DocER


Let's assume we have a document D containing n entities e={e_1, e_2, …, e_n}. Each entity e_i in the document has a corresponding position p_i, and there may exist relations between entities. Our objective is to extract a set of relations R={r_1, r_2, …, r_m} from document D, where each relation r_i can be represented by a triple (e_i, e_j, r_i j), with r_i j being the relation label between entities e_i and e_j.
Therefore, the task of DocRE can be formulated as follows:
R={(e_i, e_j, r_i j) | e_i, e_j ∈ E, i ≠ j},
 r_i j is the relation label predicted by the relation classifier based on the contextual information of entity pairs (e_i, e_j).
 
 In addition, Document-level Evidence Retrieval (DocER) aims to retrieve a list of evidence sentences evi_[0,1,…,n] from document D to enhance relation extraction. Nowadays, researchers have extended relation triples (e_i, e_j, r_ij) by adding evidence sentence lists, resulting in relation quadruplets (e_i, e_j, r_ij, evi_[0,1,…,n]). Relations between entity pairs can be predicted solely using the sentences from the evidence lists, without relying on the entire document.





§ RELATED WORK


Our work is built upon a substantial body of recent work on document-level RE and ER.


DOC-Relation Extraction (RE). Previous studies can be divided into three major categories <cit.>:

Sequence-based methods. 

<cit.>, and <cit.> use methods such as Conditional Random Fields (CRF) or Recurrent Neural Networks (RNN, <cit.>) to accurately identify and label entities in text and predict the relations between these entities by learning the contextual sequential semantic information of each word. For example, <cit.> use different neural network architectures to perform sequence encoding of the entire document to learn the semantic representation of entity pairs and extract the relations between them. <cit.> employ BiLSTM to simultaneously consider the forward and backward information in the text sequence, learning and understanding the different semantic paths from one entity to another, thus inferring the relations between entity pairs.




Graph-based methods. 
<cit.> model entities and relations in documents as graph structures, then use Graph Neural Networks (GNNs) to learn the relations between entities. This approach effectively leverages structural information and global context among entities. Additionally, researchers have proposed hard pruning and soft pruning strategies for dependency tree structures to optimize the model's speed and storage efficiency. <cit.> and <cit.> use hard pruning strategies to retain words near the shortest path between two entities, maximizing the removal of irrelevant content while integrating relevant information. <cit.> proposed a soft pruning method that directly takes the full dependency tree as input and automatically learns how to selectively focus on relevant substructures that are useful for the relation extraction task. Subsequently, <cit.> proposed refined strategies to enhance cross-sentence relation reasoning by automatically inducing latent document-level graphs. This strategy allowing the model to incrementally aggregate relevant information for both local and global reasoning.


Transformer-based methods. This approach does not use any graph structures but instead adapts to the document-level relation extraction task by fine-tuning pre-trained models <cit.>. <cit.> introduced a copy-based training objective into the basic pre-trained language model, enabling the model to better capture coreference information. <cit.> employed a hierarchical aggregation method to obtain reasoning information at different granularities at the document-level. <cit.> addressed the multi-label and multi-entity issues in relation extraction datasets through adaptive thresholds and local context pooling.



DOC-Evidence Retrieval (ER). Currently, a few studies have investigated the importance of evidence information in document-level relation extraction tasks. <cit.> directly incorporated evidence sentence instances supporting entity relations into a new dataset. However, in the absence of evidence sentences, evidence information needs to be generated through an Evidence Retrieval (ER) task. <cit.> employed heuristic rules to select informative sets of paths from the entire document to discover evidence sentences and further optimized relation extraction by combining BiLSTM. <cit.> integrated evidence information into a Transformer-based DocRE system by directly guiding attention, without introducing any additional trainable parameters for the ER task. Compared to our work, they did not incorporate graph neural networks for end-to-end learning to derive the advantages of attention weights. In contrast, GEGA provides a more reliable allocation of weights for evidence information by constructing a fully connected graph and its corresponding fully connected matrix to learn structured information.




§ METHODOLOGY




This section elucidates the main framework of the proposed method, illustrated in Figure <ref>, the model can be segmented into tripartite: Input Encoder Layer, GEGA Module and Classification Layer.




 §.§ Input Encoder Layer
This work adheres to the methodology employed in prior research to incorporate specialized markers [CLS] and [SEP] at the begin and end of a designated document doc=[sent_N]_N=1^L for the purpose of outlining the document's boundaries, where sent=[t_n]_n=1^l. Subsequently, input the document into the pre-trained BERT model. When the input length exceeds 512, the document is divided into two overlapping segments[The length of each data in the publicly available datasets is less than 1024.]. The first segment has a length of 512 and the second segment comprises the difference between the total input length and 512. Ultimately, the contextual embedding representation H and attention matrix A of the token are derived:



    (H,A)   =[(h_1,a_1), (h_2,a_2), …, (h_l× L,a_l× L)] 
       =BERT([doc])

where l is the length of the sentence (i.e., the number of tokens), and L is the length of the input document (i.e., the number of sentences).

For each entity, the efficacy of the max pooling function is pronounced when the inter-entity relations are explicitly articulated. Nevertheless, in the context of this study, the relations among entities remain ambiguous. It is understood that an entity may be referenced by one or several mentions, and a mention might uniquely identify an entity or fail to ascertain a definite corresponding entity. This necessitates the calculation of an entity's embedding based on the embeddings of each associated mention. Following methodology of <cit.>, a soft version of the max function LogSumExp (LSE), is utilized to compute the embeddings of entities while concurrently capturing the attention distribution:






    0.87e_emb=LSE(x_1, …, x_n)=log∑_i=1^|ℳ_e|exp(h_i ∈ e)




    0.95e_att=1/|ℳ_e|∑_i ∈ ea_i
where e is an entity comprising multiple mentions, while a is the distribution of attention for mention i within entity e. |ℳ_e| is the number of mentions for entity e.



 §.§ GEGA Module

The GEGA module comprehends four parts: the Attention Concentration Layer, the Multi-GraphConv Layer, the Transformer-enc Layer, and the Collaborative Prediction Layer.



  §.§.§ Attention Concentration Layer



We employ Attention Concentration Layer to transform the initial dependency tree into a fully connected weighted graph based on the dependency relations within the sentence. This approach can be construed as a soft pruning strategy <cit.> juxtaposed with the conventional hard pruning strategy <cit.>. By assigning weights to the sequence data, as opposed to outright deletion, a greater amount of contextual information can be preserved, thereby fostering enhancements in module efficacy. 
Subsequently, we utilize the multi-head attention mechanism, wherein the input vector is mapped to several heads using a linear transformation layer to produce an adjacency matrix with varied weight distributions, denoted as Ã^(𝐡𝐞𝐚𝐝_𝐢)=Attention(Q W_i^Q, K W_i^K). We employ parallel computing to expedite the computational process.



    0.96Ã^(𝐡𝐞𝐚𝐝_𝐢)=softmax(Q W_i^Q ×(K W_i^K)^T/√(d))
where Q,K ∈ℝ^N × d_model, N is the length of the sequence, d_model is the dimensionality of the input feature, and W_i^Q, W_i^K∈ℝ^d_model× d_k
 is the weight parameter associated with the linear transformation.





  §.§.§ Multi-GraphConv Layer



The Graph Convolutional Networks (GCNs <cit.>) is a deep learning framework tailored for the processing of graph-structured data. It is a semi-supervised learning method based on graph structure that aggregates and propagates node features in the graph structure, thus deriving node representations. The Multi-GraphConv (M-G) Layer is a densely connected graph structure data processing module that is constructed based on GCNs. Illustrated in Figure <ref>, the n-th sublayer's output within the Multi-GraphConv (M-G) Layer serves as the subsequent l-n sublayers' input, with the N-th layer receiving an aggregation of all output features from the initial n-1 layers.

Initially, the linear transformation of the adjacency matrix Ã^(head _i) on the input features is computed for each head i. Subsequently, the impact of neighboring nodes' features on the present node is determined for each layer l, and the current node's features are consolidated with the previous layer's output:


    0.87H̃^(i,l)=ReLU((Ã^(head _i)x)^(i) W^(i, l)) + H̃^(i, l-1)
where, L is the quantity of layers in the graph convolution layer, W^(i,l) is the weight parameter of the i-th head in the l-th layer.

The feature representation resulting from the output of each head is combined to form the final output of this layer:








    0.87H̃^(𝐌𝐮𝐥𝐭𝐢𝐇𝐞𝐚𝐝)=Concat(H̃^(1, l), …, H̃^(i, l)) W^O
where W^O ∈ℝ^hd × d_model is the weight parameter of the linear transformation applied to the ultimate output.




  §.§.§ Transformer-enc Layer

The Transformer-enc layer is composed of multiple encoder layers stacked together. These encoder layers bear resemblance to the encoder layers delineated in the transformer model introduced by <cit.>. However, distinctively, our approach involves solely utilizing the output generated by the final three layers of the Encoder for the purpose of averaging. Each encoder layer incorporates self-attention mechanism and Feedforward Neural Network (FFN). This module facilitates the derivation of hidden representations of entities and an attention distribution matrix that are used as input for subsequent layers. The calculations can be outlined as follows:

    0.87self-Att(H̃_Q, H̃_K, H̃_V)=softmax(H̃_Q H̃_K^T/√(d_k)) H̃_V




    0.87La=LayerNorm(H̃+self-Att(H̃))




    0.87(H̃,Ã)=La+FFN(La)

Where H̃_Q, H̃_K, H̃_V is the query, key, and value representations obtained from the linear transformation of H̃, d_k is the dimension of the attention head. LayerNorm is the layer normalization operation.



  §.§.§ Collaborative Prediction Layer


The local context extraction methodology, as described by <cit.>, is employed to ascertain the importance of individual tokens in relation to the entity pair ( Es,Eo), which is interpreted as the sentence-level importance. Erecting on this base, document-level importance was deduced by apportioning diverse attention weights in accordance to the contribution of each sentence within the document to the prediction of entity relations, and by establishing a fixed threshold. Sentences that exceed this threshold are selected as evidence sentences. The sentence-level importance q_i^(Es, Eo) and document-level importance p_j^(Es, Eo) can be computed as follows:

    0.95q_i^(Es, Eo) =∑_i=1^H̃Ã_Es·Ã_Eo




    0.95p_j^(Es, Eo)=∑_j=1^lq_i^(Es, Eo)


We apportion more attention to evidence sentences and less to non-evidence sentences through evidence supervision to further coordinate the prediction results of document-level relation extraction. As depicted in Figure <ref>, we train a teacher model on the Human-Annotated Data (which encompasses relation labels and evidence sentences) of DocRED (Step 1). We utilize the trained teacher model to predict the entity relations and evidence sentence distribution in Distantly-Supervised Data (which encompasses relation labels but lacks evidence sentences) (Step 2). Subsequently, we train a student model on the Distantly-Supervised Data that incorporates evidence sentences (Step 3), and retrain the student model using Human-Annotated Data (Step 4). 
Additionally, we define a row vector z^(Es, Eo) consisting only of 0s and 1s, generated based on Human-Annotated Data. This vector indicates whether each sentence is an evidence sentence for the relation triples: 1 if it is, and 0 if it is not.

    0.90z^(Es, Eo)=∑_sent=1^Lz^(Es, Eo) / 1^⊤∑_sent=1^Lz^(Es, Eo)

where 1 is the row vector composed of all 1, L is the total count of sentences contained within the document.






 §.§ Classification Layer


We begin with the computation of a weighted average of entity importance at the sentence-level q_i^(Es, Eo), and subsequently cascading it with the previous entity representation. Following this, we apply the tanh activation function to normalize the input to range between (-1, +1), resulting in the contextual representation of the two associated entities. The computation is elaborated as:



    0.85c_Es=tanh(W_Es[h_Es ; H̃^⊤q_i^(Es, Eo)]+b_Es)
       0.85c_Eo=tanh(W_Eo[h_Eo ; H̃^⊤q_i^(Es, Eo)]+b_Eo),
where W_Es, W_Eo∈ℝ^d × 2 d, b_Es, b_Eo∈ℝ^d.

Finally, apply the grouped bilinear classifier proposed by <cit.> to calculate the relation category scores.
Score^(Es, Eo)=c_Es^⊤W_Rnc_Eo+b_Rn,
The possibility of the entity pair (Es, Eo) possessing a relation Rn is computed thusly:
0.8P(Rn | Es, Eo)=Sigmoid(Score^(Es, Eo)).














§ EXPERIMENTS







 §.§ Dataset and Evaluation

DocRED[<https://github.com/thunlp/DocRED>] <cit.> is a benchmark dataset for document-level relation extraction tasks, released by Tsinghua University. DocRED comprises numerous documents from Wikipedia and Wikidata, each annotated with entities, relations between entities, and evidence sentences that support relation triples. It serves as the predominant benchmark for DocRE model training and evaluation.


Re-DocRED[<https://github.com/tonytan48/Re-DocRED>] <cit.> and Revisit-DocRED[<https://github.com/AndrewZhe/Revisit-DocRED>] <cit.> are modified datasets of DocRED. They supplement a large number of relation triples to solve the problems of incomplete annotations, coreferential errors, and inconsistent logic in docred. Annotation quality has high accuracy and consistency, which provides a more reliable benchmark for DocRE-model training and evaluation.

We assess GEGA using an Nvidia Tesla V100 16GB GPU and evaluate it with F1, Ign-F1, and Evi-F1 metrics. Ign-F1 represents the calculated F1 score attained by excluding relational facts present in both the training and development/testing datasets. 

Evi-F1 serves as a significant measure for assessing the performance of ER and constitutes a new benchmark for assessing the quality of relation extraction models.



 §.§ Single and Fusion

In the task of RE, the most ideal scenario is that the evidence sentence set of the dataset already contains all contextual information necessary to predict entity relations, thereby enabling accurate relation prediction results based solely on the evidence sentence set. However, manually labeled data and distant supervision data often fall short in this regard. Therefore, it is necessary to extract contextual information from the entire document to predict entity relations. We divide the above problem into two evaluation methods: (1) Single: extract entity relations from the entire document and obtain the corresponding prediction scores; (2) Fusion: predict entity relations based on a collection of evidence sentences and combine the prediction results with those from the Single method. This is similar to the Fusion of Evidence approach in <cit.>.




 §.§ Compared Methods

To ensure a fair comparison of the performance of DocRE baselines, we compare our model with three state-of-the-art methods, all using BERT-base as the pre-trained language model (PLM), which are:
(1) Sequence-based methods:
CNN <cit.>,
LSTM <cit.> ,
BiLSTM <cit.>. 

(2) Graph-based methods:
GAIN <cit.>,
MRN <cit.>,
DocuNet <cit.>,
GTN <cit.>,
SD-DocRE <cit.>,
AA <cit.>. 
(3) Transformer-based methods:

ATLOP <cit.>,

EIDER <cit.>,
SAIS <cit.>,
PRiSM <cit.>,
DREEAM <cit.>.
On this foundation, we categorize the above methods into two major classes: without Distant Supervision and with Distant Supervision.
























§ RESULTS AND ANALYSES




We test the trained student and teacher models in both the Single and Fusion stages, and report the results on DocRED, Re-DocRED and Revisit-DocRED, where the results on Revisit-DocRED are moved to appendix.


 §.§ Results on DocRED

Table <ref> indicates that GEGA achieves superior Ign-F1 and F1 metrics compared to the established DocRE Baselines on both the development set and the test set. The single stage of the student model has achieved performance levels comparable to the leading DREEAM <cit.>. Notably, the fusion stage of the student model achieved the highest recorded scores, surpassing DREEAM by 0.34% (Ign F1) and 0.55% (F1) on the development set, as well as by 0.17% (Ign F1) and 0.44% (F1) on the test set.

GEGA also performs well in the test of the new benchmark Evi-F1, surpassing the previous most advanced DREEAM by 0.41% (Evi-F1) and 0.46% (Evi-F1) on the development set and test set respectively. In the table, it is evident that the Transformer-based and Graph-based models outperform the Sequence-based ones, validating the rationality behind integrating infographics with the Transformer.



 §.§ Results on Re-DocRED


Table <ref> presents the feedback outcomes of GEGA on the development and test sets of RE-DocRED, demonstrating that our GEGA achieves state-of-the-art results compared to other methods utilizing BERT-base as a pre-trained language model. Notably, GEGA has outperformed all other methods in the table during the fusion stage without Distant Supervision (Teacher). GEGA secured the highest Ign F1 and F1 scores in the fusion stage with Distant Supervision (Student), with improvements of 2.08% (Ign F1) and 2.52% (F1) respectively on the development set over the second-place GTN-BERT <cit.>, and by 1.58% (Ign F1) and 1.97% (F1) respectively on the test set.



 §.§ Effect Analysis of GCNs and ER

Based on the test scores on DocRED and Re-DocRED, we observe that graph-based models such as DocuNET <cit.> and GTN-BERT <cit.> have achieved superior performance in the field. Graphs have an advantage in conveying document-level contextual information, so we used grid search to select a 2-layer GNNs to guide multiple attention maps. Additionally, DREEAM <cit.> is the first method to enhance relation extraction performance purely through evidence-guided attention, it has already achieved excellent scores on the DocRED set. By integrating GCNs with evidence retrieval, we further improved its scores by 0.41% (Evi-F1) and 0.46% (Evi-F1) on the dev and test sets, respectively. Therefore, we conclude that GCNs and ER significantly enhance performance in the relation extraction task.



 §.§ Ablation Studies






We conducted ablation experiments were conducted on the development set to analyze the GNNs layer and Training phase of GEGA. The single phase of GEGA (Student) was utilized as the test benchmark. The results of the score post-ablation of each part are presented in Table <ref>. Initially, we removed the Attention Concentration Layer, which resulted in a minor decline in performance. Subsequently, upon removing the Multi-GraphConv Layer, a significant performance deterioration was observed, implying the importance of constructing multiple attention distribution graphs for relation extraction. Upon removal of the Transformer-enc Layer, we noted a relatively minor decline in performance. We speculate that this may be related to using Transformer based BERT as PLM.


Additionally, during the training process, we performed ablation on the self-training, fine-tuning, and Distant Supervision-training stages, in order to further analyze their impact. The results indicate that when the ER self-training phase is omitted, performance declines, whereas the absence of the fine-tuning stage did not lead to a noticeable decline in performance. Further more, omitting the Distant Supervision-training stage caused severe performance degradation. These findings highlight the effectiveness of our ER method in enhancing relation extraction.









§ CONCLUSION

We propose GEGA, the first model to employ GCNs and ER jointly guided attention to enhance DocRE. We validate the superiority of our model on three widely used datasets: DocRED, Re-DocRED and Revisit-DocRED. GEGA is trained using parallel computing in both fully supervised and semi-supervised settings, without incurring additional overhead, making it convenient for use in the era of Large Language Models (LLMs). In the future, we aim to leverage the scalability of GEGA and apply it to a broader range of scenarios, including entity recognition, event extraction, and more.




§ LIMITATIONS

The model GEGA is subject to two limitations. Firstly, when utilizing Multi-GraphConv Layers to induce multiple fully connected attention distribution matrices, there is a possibility of generating one matrix that differs significantly from others in terms of weight distribution. This could lead to significant deviations in prediction results. We hypothesize that guiding the construction of multiple fully connected attention matrices using evidence information may reduce the occurrence of such undesirable situations, a conjecture that will be verified in future work.
Secondly, it is acknowledged that the relations between most entity pairs can be predicted based on the local context of the entities. However, our model utilizes evidence sentences retrieved from the entire document corpus, which are strongly correlated with the entity pairs of interest, rather than evidence sentences obtained specifically for individual relation triples. This approach may result in the model carrying more global contextual information while reducing the utilization of local context information.




§ ETHICS STATEMENT

Our proposed GEGA demonstrates outstanding scalability and applicability, serving as an excellent solution for both DocRE and DocER tasks. This method is evaluated solely on publicly available datasets, ensuring no compromise on individual privacy. Furthermore, we provide the source code implementation of GEGA to enable researchers to reproduce its performance authentically, fostering academic exchange in the field of DocRE.


















§ LOSS

To accommodate the requirements of the RE and ER tasks within both the teacher model and the student model, distinct forms of loss function computation have been devised for the relation classification approach.The loss variation is shown in Figure <ref>.






 §.§ RE loss:

For the RE tasks of the two models previously described, we implement the Adaptive Thresholding Loss (ATL) as proposed by ATLOP. During the training phase, we use a threshold class (TH) to learn a threshold such that the logits of the positive class ℛ𝒫 exceed it, and the logits of the negative class ℛ𝒩 fall below it.


    ℒ_RE=    0.95-∑_Rn∈ℛ𝒫exp(Score_Rn^(Es, Eo))/∑_Rn^'∈ℛ𝒫∪{TH}exp(Score_Rn^'^(Es, Eo))
       0.95-exp(Score_TH^(Es, Eo))/∑_Rn^'∈ℛ𝒩∪{TH}exp(Score_Rn^'^(Es, Eo))








 §.§ ER loss:

The tasks targeted by the teacher model and the student model differ in detail. The teacher model is trained on Human-Annotated Data, which includes reliable manually annotated evidence sentences, while the student model is trained on Distantly-Supervised Data that incorporates evidence sentences identified through the teacher model's ER. Considering these distinctions, there is a requirement for specialized loss computation methods. Consequently, we propose both document-level and sentence-level loss calculations.

Document-level Loss:
By integrating the document-level importance distribution p_j^(Es, Eo) with the original manually annotated evidence sentences z^(Es, Eo) from the dataset, we induce a localized context representation that contributes significantly to RE. We use the Kullback-Leibler Divergence (KL divergence), a method for measuring the difference between two probability distributions within the same event space. In this paper, it is used to assess the degree of divergence between p_j^(Es, Eo) and z^(Es, Eo):

    ℒ_ER^doc=
    KL( z^(Es, Eo)p_j^(Es, Eo) )   
    
    =∑_sent=1^Lz^(Es, Eo)logz^(Es, Eo)/p_j^(Es, Eo)


Sentence-level Loss:
We use the teacher model trained on Human-Annotated Data to perform ER testing on Distantly-Supervised Data, predicting its sentence-level evidence distribution q̃_i^(E s, E o).
Thereafter, utilize Kullback-Leibler (KL) divergence to compute the difference in sentence-level probability distributions between the teacher model and the student model.

    ℒ_ER^sent=KL(q_i^(E s, E o)q̃_i^(E s, E o)) 
        =∑_t=1^l q_i^(E s, E o)logq_i^(E s, E o)/q̃_i^(E s, E o)

Finally, we apply the prevalent weighted summation technique for document-level and sentence-level losses to equilibrate the losses of RE and ER, where λ serving as a hyperparameter.

    ℒ=(1-λ)ℒ_RE+λℒ_ER






§ HYPERPARAMETER

We ran tests on the above three datasets using different random seeds five times and reported the average test accuracy.
In the manuscript, a comprehensive Table <ref> delineating the hyperparameters and configuration has been furnished, encapsulating pivotal settings utilized across the experimental trials. These hyperparameters, meticulously curated and fine-tuned, exert control over multifarious facets of the model's dynamics throughout the training and evaluation phases. 




§ RESULTS ON REVISIT-DOCRED

Table <ref> shows the test results of GEGA (Student) on Revisit-DocRED. According to the test method of <cit.>, we trained on the DocRED training set, and then tested on the test set provided by Revisit-DocRED. It can be seen that GEGA has a great performance improvement compared with other methods.









§ SUPPLEMENTARY MODEL ANALYSIS

Here, we conducted experimental analyses on the number of GNNs layers and the number of heads in the Multi-GraphConv Layer of GEGA. We selected the number of GNNs layers from {1, 2, 3, 4, 5} and the number of heads from {1, 2, 3, 4}. The experimental results are shown in Figure <ref>. Ultimately, the analysis concluded that the performance is optimal when layers=2 and heads=2.























§ CASE STUDY




To further demonstrate the superior performance of GEGA, we extracted a case from the DocRED dataset and used it to compare the performance of GEGA with two other advanced methods (SAIS and DREEAM).


 §.§ RE

From Figure <ref>, we observe five types of relations among five entities. SAIS correctly identified four types of relations but overlooked the "country of citizenship" relation between "Robert  F./Mark  R." and "United States" Additionally, it incorrectly identified a "country of citizenship" relation between "Robert  F./Mark  R." and "American" as well as a "located in the administrative territorial entity" relation with "Terry  McAuliffe" While DREEAM correctly identified all the existing relations, it excessively identified a "country" relation between "American" and "United  States" In contrast, GEGA perfectly extracted the correct relational network in this case.


 §.§ ER

During evidence retrieval, both GEGA and DREEAM labeled the evidence source for the relation between United  States and Virginia as [S1, S2], missing [S0]. Additionally, DREEAM incorrectly labeled the evidence information for the relation between Terry  McAuliffe and United  States as [S0, S6]. SAIS incorrectly labeled the evidence information for the relation between Virginia and Terry  McAuliffe as [S0, S1, S2, S6]. We believe that GEGA's superior performance is also attributed to its better ER performance.







 Paper Title: Lyapunov weights to convey the meaning of time in physics-informed neural networks.









Université Paris Dauphine - PSL

CEREMADE, 
 Place du Marechal de Lattre de Tassigny, Paris 75016, FRANCE



[url]https://turinici.com

Gabriel.Turinici@dauphine.fr



§ ABSTRACT 
Time is not a dimension as the others. In Physics-Informed Neural Networks (PINN) several proposals attempted to adapt the time sampling or time weighting to take into account the specifics of this special dimension. But these proposals are not principled and need guidance to be used. We explain here theoretically why the Lyapunov exponents give actionable insights and propose a weighting scheme to automatically adapt to chaotic, periodic or stable dynamics. We characterize theoretically the best weighting scheme under computational constraints as a cumulative exponential integral of the local Lyapunov exponent estimators and show that it performs well in practice under the regimes mentioned above.




 physics-informed neural networks; causal weighting in PINNs; Lyapunox exponent








§ TIME IN PINNS : INTRODUCTION AND PREVIOUS WORKS


Physics-informed Neural Networks introduced in <cit.> proved to be a very successful paradigm invoked to solve complex mathematical equations such as time evolutions 
<cit.> <cit.> possibly in high dimensions <cit.>, 
 partial differential equations <cit.>
 or control systems <cit.>; this framework exploits the power of neural networks (including automatic differentiation) to represent the main unknown function which is the solution of some equation involving its derivatives. The PINNs can also incorporate different other elements such as (possibly noisy) measurements on the system or further data such as controls.

To improve PINN performance several leads have been followed; 
Sharma and Shankar
 <cit.> studied meshless discretization, Cho et al. <cit.> consider separable network architecture while Wang et al. <cit.> analyze 
best norm to express the loss. We borrow here from all these perspectives but we focus on time-dependent problems and more specifically on how the sampling of time points (or their weighting in the loss) impacts the overall performance. The time dimension has been long recognized to possess particular characteristics that deserve special attention. In a highly influential work on 
Allen-Cahn and Cahn-Hilliard  
phase field equations <cit.> Wight and Zhao introduce various sampling strategies in both space and time while McClenny and Braga-Neto
<cit.> make weights trainable which requires additional computational time; in the latter work the influence of the weights on the training loss is mediated by a function that becomes a hyper-parameter to be specified by the user.
In a related approach, 
<cit.> Wang et al. remarked that, especially for chaotic or near chaotic systems, the causality of the time dimension has to be respected and enforced to obtain good results. In particular they proposed that earlier times be given more weight in the loss functional, proportional to the exponential of the cumulative sum of the accumulated errors (up to that point in time, see formula (<ref>) below); a hyper-parameter denoted ϵ remains to be chosen and in practice it is iterated in a prescribed list. The procedure also requires some special termination criterion.  
On the other hand 
<cit.> Penwarden et al. also studied the time dimension and propose a scalable framework for causal sweeping strategies in PINNs. This and many other contribution recognized the necessity to combine PINNs sequentially in order to solve a problem set on a large time interval. While we agree with this perspective, we search here only for the best time weighting scheme on each of these individual intervals ; as such, our procedure can be combined with any other time sweeping protocol. Moreover, as the previous protocols have been rather prescriptive and proposed ad-hoc choices of time weighting, we focus here on the design of principled approaches to orient the choice of  weights.  Our procedure is on one hand flexible enough to recognize the main regime the evolution takes place into (stable, chaotic, periodic) and on the other hand it does not require additional hyper-parameters to adjust, which was a more laborious part of the previous approaches.

The balance of the paper is the following: in section <ref> we introduce the general framework and in section <ref> we give the main theoretical insights that motivate our procedure. The numerical experiments are the object of section <ref>, followed in sections <ref> and <ref> by concluding remarks. 




§ PRESENTATION OF THE FRAMEWORK
 
To set notations let us recall very briefly how PINNs operate. Suppose that the evolution equation is to be solved is written in the form :

    ∂_t u    = _t u
    
    
    u(0,x)     =u_0(x),  ∀ x ∈Ω
    
    u(t,x)    = u_bc(t,x),  ∀ t∈[0,T], x ∈∂Ω.

where u(·,·) is the main unknown, t is the time variable and x the spatial variable, 
_t is some operator possibly involving the 
spatial derivatives of u (e.g., for the heat equation _t = ∂_xx), u_0(·) and u_bc(·,·) are the initial and boundary conditions respectively. When the equation is set in finite dimension such as the Lorenz system in section <ref> then Ω will be discrete and ∂Ω = ∅. The solution u will be searched in the form of a neural network (NN) taking two inputs t and x and outputting an approximation _θ(t,x) of u(t,x) ; θ are the
NN parameters, see figure <ref>.
In order to simplify the presentation we will moreover assume as in <cit.> and related works, that we can construct the network _θ such that _θ satisfies exactly the initial and boundary conditions. For instance for the initial conditions one can simply shift the output of the NN :
_θ(t,x) ↦_θ(t,x)-_θ(0,x)+u_0(x)· t
as in <cit.> or  

    _θ(t,x) ↦_θ(t,x)-_θ(0,x)+u_0(x).

 as in <cit.>.
In order to train the NN a loss is formulated that includes as main ingredient the equation error w(t,x) defined as w(t,x) = ∂_t _θ(t,x) - _t _θ(t,x) or equivalently :

    ∂_t _θ(t,x) = _t _θ(t,x) + w(t,x),  _θ(0,x) = u_0(x).

Our use of this last equation is slightly different from the literature, see also figure <ref> : since the derivatives are here exactly computed using the automatic differentiation capabilities of the NN, the equation 
(<ref>) is also satisfied exactly. So we can consider the error w as being the main output and _θ as the solution of 
(<ref>) that is available at no additional cost.


The NN is trained to minimize a loss functional; the original proposal in <cit.> is 
∫_0^T ∫_Ω | w(t,x)|^2 dx dt but, later on, particular weighting schemes appeared that proposed a loss of the form :

    (θ) = ∫_0^T ∫_Ωρ(t) | w(t,x)|^2 dx dt,

with the weight ρ(t) ≥ 0 to be chosen in order to speed up the convergence and improve the quality of the output. For instance <cit.> use 
a discrete form of 

    ρ(t) = e^-ϵ∫_0^t w(t,x)^2_L^2_x

for some ϵ >0 ; such a choice will give more weight to low values of t. Another proposal is 
<cit.> who sets  
 ρ(t) = e^-ϵ t for some ϵ∈; the intuitive reason is that errors at earlier times will accumulate into larger errors in the final output so  the solution at initial times should be computed more precisely. Although these proposals are inducing better numerical properties, they remain somehow ad-hoc. We investigate below in a principled way 
 how the weights ρ contribute to the quality of the result and set it accordingly.





§ THEORETICAL RESULTS
 

To design time weighting schemes we need to know how the equation error w(t,·) at time t will impact the accuracy of the final result _θ(T,·). There is no general answer to this question but we will use a close concept, namely the Lyapunov exponent which describes  quantitatively 
the rate of separation of infinitesimally close trajectories. The Lyapunov exponent originates from the dynamic system analysis and, for some 1D ordinary differential equation z'(t) = f(t,z(t)) describes how two solutions, starting from z^1(0) and z^2(0) diverge in the limit of large times t; more precisely the Lyapunov exponent is the coefficient λ∈ such that | z^1(t)-z^2(t)| ∼ e^λ t | z^1(0)-z^2(0)| (for large t). The determination of the exact value of the Lyapunov exponent is a difficult task in dynamical systems and ever more so in evolution PDE, but we will retain this idea. This allows to state :

Let u, _θ as in  (<ref>) and (<ref>) respectively. Let λ(t) be such that 

    ⟨_t(u(t,·))- _t(_θ(t,·)), u(t,·)- _θ(t,·) ⟩/u- _θ^2≤λ(t).

Then

    u(T,·)- _θ(T,·)≤∫_0^T e^∫_t^Tλ(s)dsw(t,·) dt.

Here the norm is euclidean when Ω is finite and L^2 norm otherwise.

 Denote e(t) = u(t,·)- _θ(t,·). Using (<ref>) and (<ref>)  we obtain:

    1/2d/dte(t)^2 = ⟨d/dt e(t), e(t)⟩
    =  ⟨_t(u)- _t(_θ)- w(t,·), e(t)⟩
       ≤λ (t) ⟨  e(t), e(t)⟩ -
    ⟨ w(t,·), e(t)⟩≤λ (t)  e(t)^2 + w(t,·)· e(t)

This means that d/dte(t)≤λ (t)e(t) + w(t,·), and, using the notation Λ(t) = e^-∫_0^t λ(s) ds
we obtain d/dt(Λ(t)  e(t) ) ≤Λ(t) w(t,·) ; hence, since e(0) = 0, by integration from 0 to T we get 
Λ(T)  e(T) ≤∫_0^T Λ(t)  w(t,·) dt which gives the conclusion.


The result already gives qualitative insight into how the error at the final time T depends on errors at intermediary times t≤ T, which is the main idea of Lyapunov exponents. Let us take the simple case _t(u) = λ u with λ∈ constant; in this case 
(<ref>) is in fact an equality.
As in <cit.> we note that for a chaotic or divergent system, which is characterized by a strictly positive Lyapunov exponent i.e., Re(λ)>0 the trajectories diverge exponentially with respect to error in the equation (<ref>) or in the data. In this case the precision has to be large for t near zero because the factor e^λ (T-t) will multiply it resulting in large error in the final state. On the contrary, for values of t close to T the factor e^λ (T-t) is relatively small and a larger error in the equation can be tolerated because it will be multiplied by a smaller factor. What is interesting to note is that this also works the other way around for Re(λ)<0 which characterize  systems converging to stable equilibria. In this case initial errors in the equation will be "washed away" by the stable dynamics and are less important and estimation (<ref>) says that we can manage errors increasing exponentially with T-t as long as the exponent is smaller than |Re(λ)| !
Finally, for the (quasi-) periodic dynamics |λ|=1, all times t≤ T contribute equally to the final error and no special weighting should be enforced. 

To add more actionable quantitative details to the discussion above we 
will see now how this translates into weighting schemes for ρ(t); changing the weight ρ(t) indicates to the training algorithm that some values of t should be given priority when minimizing the error loss w(t,·). Of course, more computational resources are available better the algorithm will converge, so we will make our reasoning assuming a given computational level of error i.e., will assume that we can employ numerical algorithms that  make ∫_0^T ρ(t)  w(t,·)^2 dt small enough (but not zero), say equal to C_p and inquire which is the best weighting scheme for this class of errors. The idea is to see how well the choice of ρ guarantees a good behavior of the final error at time T. Note that the training will only minimize the loss without regards as to which of the possible functions w are selected, as long as 
 ∫_0^T ρ(t)  w(t,·)^2 dt reaches the desired level C_p. So here we will assume worse case scenario and see what is the ρ
 whose worse case scenario is the best. This will be formalized as looking for the ρ such that 
 {max_∫_0^T ρ(t)  w(t,·)^2 dt ≤ C_p∫_0^T e^∫_t^T λ(s)ds  w(t,·) dt }
is smaller, see proposition below.
To gain in generality we will accept weighting schemes with irregular densities and write for a law η on [0,T] with density ρ (maybe not a proper function) _τ∼η w(τ,·)^2 dt instead of 
 ∫_0^T ρ(t)  w(t,·)^2 dt.
 For technical reasons we will need the law η to have finite second order moment which we write η∈^2([0,T]) and we assume w is bounded.
The following result gives precise information on the best weighting to be used :

Let λ(s):[0,T] → with 
∫_0^T e^∫_0^t λ(s)ds  dt < ∞ and w such that 
t↦ w(t,·) is bounded. 


Then the  problem

    min_η∈^2([0,T]){max_w; _τ∼η[ w(τ,·)^2 ] ≤ C_p∫_0^T e^∫_t^T λ(s)ds  w(t,·) dt }

admits an optimum which is attained in some η^opt. Moreover the minimum η^opt has a density ρ^opt(t) which is proportional to 
e^- ∫_0^t λ(s) ds that is :

    ρ^opt(t) = e^- ∫_0^t λ(s) ds/∫_0^T e^- ∫_0^t λ(s) ds dt.



We first suppose that ρ is regular enough and look for the solution.
Denote Λ(t) = e^- ∫_0^t λ(s) ds and write 
∫_0^T e^∫_t^T λ(s)ds  w(t,·) dt  = 
1/Λ(T)∫_0^T Λ(t)  w(t,·) dt. Dismissing the constant 
1/Λ(T) we are thus maximizing ∫_0^T Λ(t)  w(t,·) dt under the constraint _τ∼η[ w(τ,·)^2 ] ≤ C_p.
The Cauchy's inequality informs that 

    ( ∫_0^T Λ(t)  w(t,·) dt )^2 ≤_τ∼η[ w(τ,·)^2 ] 
     _τ∼η[(Λ^2(τ)/ ρ^2(τ) ) ]

with equality only when Λ(τ)/ √(ρ(τ)) and √(ρ(t)) w(τ,·) are proportional i.e. 
w(τ,·) is proportional to  Λ(t)/ρ(t). In this case the maximal value is ∫_0^T Λ^2(t) / ρ(t) dt. 
So for a given ρ we cannot guarantee a final error better than
∫_0^T Λ^2(t) / ρ(t) dt (up to a multiplicative constant depending on C_p and Λ(T) but not on ρ).
Now the choice of ρ should set this integral to the smallest value possible under the constraint ∫ρ(t) dt = 1  and ρ≥ 0. Use again the Cauchy inequality to see that 

    ∫_0^T Λ^2(t) / ρ(t) dt  = 
    ∫_0^T ρ(t) dt ∫_0^T Λ^2(t) / ρ(t) dt ≥(∫_0^T Λ(t) dt)^2,

which shows that we cannot do better than (∫_0^T Λ(t) dt)^2, 
with equality when Λ^2(t) / ρ(t) and ρ(t) are proportional i.e.
ρ(t) is proportional to Λ(t), which is the conclusion. 

For any other, possibly non smooth, choice of ρ one can follow the same reasoning by regularization : take smooth approximations of ρ which, by the arguments above turn out to not be better than ρ^opt so passing to the limit one obtains that no non-smooth ρ can do better. 

So the best weights ρ(t) are proportional to 
e^- ∫_0^t λ(s) ds; this quantity is calculated from the values of λ(t). Ideally, one would like to have :

    λ(t) = ⟨_t(_θ(t,·)) - _t(u(t,·)), _θ(t,·)- 
        u(t,·)⟩/_θ(t,·)- u(t,·) ^2,

i.e., the λ that is realizing equality in (<ref>). But the right hand side depends on the exact solution which is the unknown and has to be approximated.  
We will estimate the λ(s) by taking simply :







    λ_n(t) ∼⟨_t(_θ_n(t,·))-_t(0), _θ_n(t,·)
        ⟩/_θ_n(t,·)  ^2

where θ_n is the value of the NN parameters after the n-th training step (_θ_n is the solution candidate at this iteration). 
This amounts to approximating the exact solution (which is unknown) to zero. Although this approximation seems rough, it the most universal to be made and in practice it gives good results because we do not want exact values for λ_n(t) but some general information e.g., on its sign and monotonicity. Of course, if additional information on the exact solution u(t,x) is available this could be incorporated into  (<ref>). 
Furthermore, note  that for the simple situation when 

_t(u) = _t(u) + ξ_t + α_t u,

with _t a anti-symmetric operator, ξ_t independent of u and α_t ∈ the choice (<ref>) results in λ_n(t) = α_t (for all n) which is exact and optimal.



§ NUMERICAL EXPERIMENTS
 

The code for the numerical experiments will be provided on Github https://github.com/gabriel-turinici. 



 §.§ Lorenz system
 

The Lorenz system is a set of ordinary differential equations that describes a simplified model of atmospheric convection. It exhibits chaotic behavior and has been widely studied in the field of dynamical systems; it is described by the following set of ordinary differential equations:

    x'(t) = σ (y - x), 	 
    y'(t) = x (ρ - z) - y, 	 
    z'(t) = x y - β z,







where x, y, and z are the state variables, and σ, ρ, and β are parameters.
Coherent with the literature <cit.> we will take final time T=0.5[In the literature time goes up to T=20 but in practice it is implemented as sequential solves on 40 time intervals of size 0.5.] and use the classical parameter set (initially studied by Lorenz) 
(σ, ρ, β) = (10, 28, 8/3) and the initial state 
(x, y, z)(0)=(1,1,1). This test case is notoriously difficult to solve with PINN  (the default scheme does not even converge, see below) because of its very high sensitivity with respect to errors in the data and equation.










The reference solution, considered exact, is the one obtained with  routine with default settings. 
For the initial condition we used the shift  in (<ref>). 
The NN has 5 hidden fully connected layers of 20 neurons each ('tanh' activation) and a final FC layer with output dimension 3 (to match the unknown dimension) with no activation. All layer initializations are Glorot Uniform. The NN has 1'783 trainable parameters (which compares very favorably with 1'054'723 training parameters used by <cit.> who employ 5 hidden FC layers with 512 neurons). We used a uniform time grid of 256 points (as in <cit.>) and 2000 iterations of the Adam optimizer; initially we tested with the default learning rate (0.001) but to accelerate convergence we then chose everywhere a learning rate of 0.01 (only exception being  figure <ref>  where we compare these two learning rates). All other optimizer parameters are the TensorFlow version 2.15 defaults. A run of the procedure takes 80 seconds on a T4 GPU.

We first show convergence results in figure <ref>. It is seen that the loss reaches low values (and will improve if more iterations are allowed); at the same time we plot (right figure) also the error at final time T between the reference solution and the PINN solution, that also behave well and shows that convergence occurred. This is to be compared with figure <ref> where the numerical solution shows high quality match with the baseline solution. We also checked (not shown here) that given more iteration the quality improves further (but then we cannot distinguish graphically the two).

We now pass to the inner workings of our procedure. First in figure <ref> (right) we note that the equation error is indeed of intuitive form: since the system is very sensitive to errors, it is best to set error to be lower for small t and larger at final times. 
This is coherent with the weights selected (automatically) by our algorithm, depicted in figure <ref> left, that are indeed decreasing. The weight assignment is the result of the estimators (<ref>) plotted in figure <ref> right. We plot both our estimator and the ground truth (<ref>) (that requires knowledge of the exact solution). The agreement is remarkable and it is seen that our estimator (<ref>) captures the good order of magnitude and sign while retaining a more smooth behavior.

As discussed above, we also investigated the influence of the learning rates; we plot in figure <ref> left 
a comparison of the losses during the optimization with two different learning rates, 0.001 (TensorFlow Adam optimized default) and 0.01. As expected from the general theory of stochastic optimization it is seen that the larger learning rate gives faster convergence but also larger final variance (oscillations) at the solution. 

Finally, in figure <ref> right we plot a confidence interval for the loss (learning rate set to 0.01 for all) after 1000 independent runs of the procedure. The variability is due to the initialization of the NN layers and is normal to have runs taking longer time. For all quantiles the decrease in loss is systematic. 








We now turn to comparing these results with equivalent results for two other time weighting algorithms, <cit.> (cumulative exponential error loss (<ref>)) and <cit.> (exponential weighting ρ(t) ∼ e^-λ t). 
We do not even compare with uniform time-weighting (which is the default PINN protocol) because this is known to not work well on Lorenz (and most chaotic) systems. The interested reader can nevertheless find in the supplementary material section <ref> this result.

For both we used exactly the same NN and optimizer settings.
We seeded the random number generation of numpy and TensorFlow with the same seed as in the previous case so all procedures start from the same initial network weights.
The 'causal' procedure seem to not have converged yet cf. figure <ref> even if we gave it 5 times more iterations. We did so because this procedure needs to change the value of ϵ in a schedule involving 5 preset levels in the list 0.01,0.1, 1, 10, 100. To each level we gave 2000 iterations (which is the total number of iterations used by our procedure). We checked that if given even more time the protocol does indeed converges. Obviously it attempts to exploit the same idea of using larger weights for t small resulting in lower equations errors at t small and larger errors for t large (cf. figure <ref> rightmost plot).
In figure <ref> left we plot the convergence of the procedure (the 5 ϵ levels are visible on left plot). Note that, since the Lorenz system is close to chaotic, significant equation errors of the early stages of the evolution equation destroy completely any confidence in the solution for final time T as is shown in figure <ref> right, where the final error between the exact and numerical solution does not appear to improve much. The loss is indeed decreasing but it does not convey enough relevant information to orient the convergence. See supplementary material for further explanation of the weights behavior in this procedure, which appear sensitive to the precise number of iterations run for each  value of ϵ.




We compare now with the exponential time weighting scheme in  <cit.>. The procedure has a hyper-parameter λ  that give the exponential decay of the weights. We tested several choices of parameters 
but will only show results for two of them,
λ=10 (which has the good order of magnitude) and 
λ=13.8; this latter value was chosen to have a posteriori the same average decay as our procedure (and thus should give comparable results). The results in figure  <ref> for λ=10 show that the procedure is sensitive to the choice of the λ parameter; when this parameter is given by some 'oracle' then the solution is much improved. When λ is farther away from the optimal value the results degrade.




 §.§ The Burgers' equation
 
The Burgers' equation models the one-dimensional flow of viscous  fluid and reads :

    ∂ u/∂ t + u ∂ u/∂ x = ν∂^2 u/∂ x^2,  t∈ ]0,T],  x∈]-1,1[ 
        u(0,x) = -sin(π x) 
        u(t,± 1)=0,  ∀ t ∈ [0,T].

where u(x, t) is the fluid velocity, ν=0.01/π is a real positive constant called the viscosity coefficient, x and t are spatial and temporal variables respectively. Total time is T=1.0.
These are supplemented by the initial 
(<ref>) and boundary (<ref>) conditions. For the initial condition we used the shift described in  (<ref>); on the other hand we did nothing special for the boundary conditions which were implemented by adding a corresponding term in the loss as it is classic in these cases <cit.>.

For the 'exact' baseline, as this problem is known to be difficult to solve, a first proposal was obtained with a  
finite difference scheme combining two half-step  Crank-Nicholson propagators for the viscosity term  with a Lax-Friedrichs  propagator for the transport part. A different computation can be proposed through the exact formula (2.5) used by <cit.> combined with Hermite polynomial quadrature ; see the supplementary material  <ref> for all scheme details and analytic formula. We checked that both agree up to 1.0e-4 and used quadrature analytic formula if precision beyond this was required.

As in <cit.> and <cit.> 
the NN architecture uses 8 fully connected hidden layers of 20 neurons each (3021 parameters to train) and learning rate of 0.01; we define a uniform grid of 50 points in time and 25 in space (which gives a total number of interior collocation points of 1250, almost an order of magnitude less than in the references cited). Procedure is run for 3000 iterations in 387 seconds on a T4 GPU.

We plot in figure <ref> the results for our procedure, the causal procedure <cit.> and the standard PINN <cit.> approach. Compared with the Lorenz system this is a non-chaotic case and all algorithms will eventually converge and give good results. The plot confirms that our procedure has some merits in using insights from the system during the initial iterations and improve efficacy. Data in supplementary material shows that this is realized by some counter-intuitive behavior of over-weighting final instants when t is close to T because at this point that the discontinuity is created (and detected by our procedure).







§ LIMITATIONS
 
Although the theoretical and empirical results appear encouraging, further work could improve some aspects of the procedure. The most apparent is the estimation of the Lyapunov exponents, which is now done in a very crude way; this could be refined or, if not possible, at least construct some trust regions to inform when these estimations are reliable or not.

Another aspect is  the interaction between the weights dynamic and the solution. This is common to all adaptive weights procedures but it may concern some cases where the weights given by (<ref>) induce instabilities in the solution preventing convergence (in the same vein as the well-known GAN "mode collapse" problem). An analysis of whether this can happen or how to prevent it could be required.

Finally, for the problems where the canonical, simple implementation of the PINN procedure already works well this algorithm may cause slight numerical computational overhead.



§ CONCLUSION
 

In this work, we presented a new approach to treat the time dimension within the framework of Physics-Informed Neural Network. We started from the observation that time instants are  not permutable and depending on the regime (chaotic, periodic, or stably convergent) errors in earlier instants affect differently the quality of the outcome. We proposed a theoretical explanation based on Lyapunov exponents and then a new algorithm built from this insight.

The procedure was then tested on two different cases, one chaotic (Lorenz system) and one stable but that induces singularities (Burgers' equation), both known to require careful numerical treatment. The proposed algorithm not only showcases robustness and practical efficiency but also addresses some limitations of earlier works that lacked a principled and theoretically grounded foundation. Of course, while our procedure is robust across various benchmarks, it is designed as a complementary addition to the existing methods rather than a one-size-fits-all solution. We believe this work contributes meaningfully to the field, offering a solid foundation for future research and application development.












§ SUPPLEMENTARY MATERIAL




 §.§ Standard PINN (no time weighting) results for the Lorenz system
 

As announced in section <ref> we plot in figure <ref> the solution of the procedure for the situation when the weights are not optimized at all but left all uniform. This case was allowed 30'000 iterations (10 times more than our standard setting) but did not even start to converge.





 §.§ Exact solution for Burgers' equation

To compare the PINN solution with a reference ground truth we solved Burger's equation with a finite difference scheme and compared with an analytic formula with Hermite-Gauss integration.



  §.§.§ Finite differences scheme for Burgers' equation

The finite difference scheme works on a spatial domain
discretized with nx = 400+1 spatial grid points (counting both segment extremities ± 1), 
i.e. Δ x =2/400 
and nt = 700 time steps (Δ t = 1/700);  each time step is implementing the following split-operator technique :

- first a Crank-Nicholson (CN) scheme over a Δ t/2 time step (taking into account the boundary conditions) for the diffusion part i.e. the solution by CN of the equation ∂_t u = ν∂_xxu 

- then a Δ t step of a Lax-Friedrichs scheme for the viscosity term, i.e. for the equation
∂_t u + u ∂_x u = 0.

- then a final Crank-Nicholson (CN) scheme over a Δ t/2 time step (taking into account the boundary conditions) for the diffusion part

The splitting technique is known to be of high order in time <cit.> 
which leaves the Crank-Nicholson part (second order in time) and Lax-Friedrichs scheme (first order in time and  space) as limiting factors. As for stability it is known that CN is unconditionally stable for the heat diffusion and since the matrix for the Lax-Friedrichs scheme is tridiagonal by Gersgorin theorem it is stable when max(|u|) ·Δ t/ Δ x<1.

In practice, if V∈^nx+1  is the current solution with V_j the component  representing the solution at point x=-1 + j·Δ x), the CN propagation for a Δ t/2 time step means solving the linear system in V^next : 

    V^next_j- V^next_j/Δ t/2 = ν/2(
         V_j+1+ V_j-1-2 V_j/Δ x^2  +
        V^next_j+1+ V^next_j-1-2 V^next_j/Δ x^2), 
     V^next_0=0=V^next_nx.


The  Lax-Friedrichs scheme means replacing V with :

    V^next_j- V_j+1+ V_j-1/2/Δ t +  (
         V_j+1^2/2- V_j-1^2/2/2 Δ x)=0, 
     V^next_0=0=V^next_nx.

Once this is solved the resulting 2D solution matrix is used as input to a 2D interpolation routine to obtain a linear interpolation function that is able to output values for any other point not necessarily on the space-time grid used by the finite difference resolution.



  §.§.§ Analytic formula for the solution of the Burgers' equation


Another way to find a solution is to used the analytic formula (2.5) from <cit.>. It reads

    u(t,x) = ∫_ u_0(x-η) ζ(x-η) e^-η^2/(4ν t) d η/∫_ζ(x-η) e^-η^2/(4ν t) d η,  where ζ(y) = e^-cos(π y) / (2 πν).

The integrals can be computed using Hermite-Gauss quadrature and in practice we used 50-th order formulas after a change of variables η = η' √(4 ν t ). For t < 10^-10 we used a first order Taylor development with respect to t using the exact formula for the time derivative from the Burgers' equation.




 §.§ Behavior of weights of the causal procedure (Lorenz system)

 
To explore the convergence properties of the causal protocol <cit.> we looked at the history of the weights used by the procedure during the resolution of the Lorenz system, see comments in section <ref>; these are plotted in figure <ref>.
The algorithm's protocol iterates the ϵ parameter ; at each change the weights associated with latter times are smaller which causes deterioration of the quality there. So the algorithm can only advance to the next value of ϵ only after the equation is satisfactory solved at initial times.





 §.§ Weights for Burgers' equation

We compare in figure <ref> the final weights of the three procedures considered for the Burgers' equation, we note that our proposal over-weights final times which is when the discontinuity appears. This is not the case of the other procedures ; this behavior explains the results in last columns of figure <ref> (rows 2 and 3) that clearly lack precision at that point in space.





